I am all for immersion, but that there is just a debuff for being somewhere at a certain time. It is too blatant and obvious to not see through immediately.
However, I think that you could improve on the ideas of a straight up debuff. So for example if you are in a snowy zone, your character lift his legs higher up to traverse through the snow (and has a small debuff), in rain they'll cover their head and maybe in combat there is 1 % for a disarm because of the water making your hands slippery, in hot and warm areas your character stacks fatigue (not the ingame invisible wall form) and the longer you stay out the less mana, stamina, focus etc. you have at your disposal.
See that would be a little more immersive, but I still don't like the idea. A game like WoW I feel it needs to be a constant speed and not debuff you for immersion.