Other then Mabye hobbits the darker members of each race don’t break common sense at all.
If the god of LoTR just made the races fully formed with various Colours it could take many generations to fully eliminate any colour, something you wouldn’t see in the elfs given that they are immortal, and something you likely wouldn’t see in the dwarfs as they are also longer lived and seemingly breed slowly. The human town is also all people related to humans who joined Morgoth which could easily be any number of different races and cultures of middle earth jammed together at Morgoths fall which could take hundreds of years to end up a single skin tone.
The only real way for it to break common sense would be to say that god just can’t/won’t make blacks elf's or dwarfs.