It wasn't just that the High Elves settled in Amani lands, they actually constructed Silvermoon City on a sacred site of the Amani (likely a nexus of Ley lines suitable for establishing the Sunwell). So they were viewed not only as invaders but also as desecrating an important locale, which was probably the reason they were attacked so quickly and ferociously.