You misunderstand. Currently, there are many differences. But in say 1,000 years, if you were raised then, you may not think that way as there may not be multiple skin pigments. Or if there are, they may only vary slightly.
Current you can absolutely have that preference, but if you didn't live in a world with such differences, I think you would have a different opinion.