The funny thing at least where I live is that the people who aren't affected by racism take it as one of the highest offence you can do. And those people actually make it that those words stay as taboo and because its taboo people keep using it in a wrong way.
I don't think racism in behaviour should be ever tolerated but a silly word standing on its own that CAN be used in a different context as a offensive word? That is just silly. There are loads of words out there that can be like that. Are you going to get banned on the Chinese realms if you call yourself Yellow?