I think it's making too many assumptions that I never move around. Not once have my colour been a point of discussion. If you end up to frequently in a situation where people harass you it seems to me you are seeking those situations out for yourself. In which case, be smarter and don't waste your time.
Idiots will allways exist, you'd do better to avoid them. They cannot help you, ever.