I don't think orange goes very well with clothing, and it makes some garments less appealing to me. I don't like purple that much when it's too high on saturation level.
I think my main issue is saturation, all the colours can seem unpleasant to me when their saturation levels aren't moderate.