The statement wasn't racist or sexist. The only thing that can make a statement racist or sexist is context. The exact same words, in a different context, not necessarily bigoted.
The easiest and most obvious example is to take a statement like "X are all shit". If the context is someone using that because they think all X really are shit, that's bigoted. If I, however, quote that person, saying "that person was a bigot because their saying 'X are all shit' was motivated by hatred", that's not bigoted. Even though I said exactly the same words.
Because context is all that matters. It's what gives language meaning.