Why would it be a retcon? Do you even know what that word means? They never stated that Sauron/Halbrand did not manipulate them. So it wouldn't be retroactively changing canon in order to be a thing. It just happened off-screen. The scene does have him using "magic" which could have been the point of manipulation.
And yet they still grew to dislike the elves. Their two factions divided along "liking elves" and "not liking elves". It wasn't one elf showing up. It was a human, that arrived with the elf, trying to get a guild badge with out having earned it. A human that had special status because he was a friend of the elf. A human that then beat up the guy who gave the speech. You don't see why he would be angry and worried about his job being taken if the elf-friend human can just waltz in and disrupt the "natural order" of society