Right ... Let say 25% of horde BE players switch sides. Now the Horde is less populated, raiding guild have more trouble staying afloat, cries of doom and gloom, more players switch because they can't find a guild, and the Horde faction enters a death spiral.
Lore doesn't matter.And if you want to get technical Blood Elves are the imbalancing ones. Horde population is naturally lower than Alliance, the lore supports this and it is reflected in the number of players who choose the traditional Horde races.