It's Elven territory since the it had always been part of Zin-Ashari, which was Night Elves' lands. It doesn't matter if the NE considered the land cursed and left. If, because of a natural disasters, US citizens all leave Washington DC., are you saying that it would no longer be considered US land any longer? The same logic hold here. It's not newly discovered land, it's a land that has always been there, one that the NE lived and built a city on.
It doesn't matter which attacks come first, as the Goblin settled in Azshara - which, again, was NE's land and started terraforming it to a Horde port. The NE simply defended their land - one that they left and considered cursed, but technically their land nonetheless. Not to mention that canonically, it might not even be true that Azshara attacks come before Ashenvale (any longer, at least). Chronicle 3 stated that it was Garrosh's maneuvers on Ashenvale that raised the tension between two factions, and the Goblin's settling on Azshara, reshaping the land and creating Bilgewater Habour was only told few pages after.