It's evidence of privilege that you can live your life oblivious to the hardships of others. A white person thinks they're not privileged, in part, because they think that racial discrimination does not exist. They think it doesn't exist because they've never experienced it or do not experience it regularly. They don't experience it because they're white.
I'm white, and I have no problem admitting that I've avoided adversity because of it. I've had other hardships, but I don't pretend that I haven't had it easier than if I were a racial minority.