The Highborne didn't have any side effects? The Blood Elves? The Nightborne? Hell, the Trolls who did stay near areas saturated with Arcane energy even mutated into an entirely different species because of it. You're ignoring the affliction and effects of Arcane magic to suit your argument.
Except every member of the Council of the Black Harvest who didn't show signs of mutation, as well as most prolific Warlocks who don't explicitly allow the corruption to take hold as they hope for empowerment, almost as though there's a difference between using Fel magic and recklessly saturating an environment with it. Moreover, the magnitude of magic they used on Draenor even reached a fever pitch where it tore the planet apart, which has never been seen on Azeroth. This sort of comparison doesn't look good for an argument which wants to overlook that the mutations caused by the most reckless use of magic that's seen in Warcraft has similar effects to mortals as simply being in proximity to wells of Arcane energy.Not just magic by proximity, just magic that is high ly corruptive, fel is more corruptive than the other, period.
The void and the whispers affect your mind, this he think a strong will can surpass, unlike fel magic, that no matter how strong of will you are, it will eventually corrupt both body and mind.