But it is, it's called genotype. Skin color is a physical trait of hereditary information and has been developed through evolution under certain environmental factors. Native populations in the tropical and tropical to temperate equatorial zones have darker or "black" skin tones as a natural means to protect from the sun rays. Whereas native populations in the temperate and temperate to polar equatorial regions have lighter or "white" skin tones which tend to produce vitamin D at a much higher rate than "black" skin under less intense sunlight, again, this is nature's way for providing for these populations, so to speak.
The "skin color that doesn't matter" means there are equatorial dwarves, elves, humans and... harfoots, who have moved out of their arid/desert/jungle equatorial regions to Middle Earth and have established themselves to become part of the societies there. As we can see, however, they are rare in between largely "white" populations, yet have retained very distinctly darker skin colors, aka their genotype is largely unchanged. The only way for this to occur is if over however long they've spent in Middle Earth they have reproduced only exclusively with each other, however, the show presents to us the fact that they have not reproduced with each other and have "mixed" parentage, for example the queen's white father and others. This would change the backgrounds of numerous characters and societies, the entire world in fact.
So we reach three roadblocks:
1. The above population thing never happened in Tolkien's work.
2. It is genetically impossible to retain strongly expressed physical traits of a minor genotype over thousands of generations
3. Even if we know nothing about how life works, we should have a minimal reasoning to deduct that different race individuals don't just randomly materialize here and there among a population.
Therefore the only conclusion is to assume the showrunners expect their nonsensical world to be believable only if the viewers are stupid.