Do you think when early humans migrated north they just all turned white as they passed a certain latitude? Do you think the mutations that led to skin lightening in the early peoples of Europe popped up across the continent all at once?
If you knew anything about these things you’d realize that it took thousands of years of migrations and passing on genetic mutations to get to what YOU seem to think is required for a “grounded” fantasy setting.
Tolkien set Middle-earth in a prehistoric time, and his peoples didn’t evolve or migrate the way actual Homo sapiens did. The idea that for it to be “realistic” the setting must mirror a relatively modern, highly segregated version of our world is simply your own prejudices talking.