I don't mind being whispered, and generally speaking I tend to politely whisper people to avoid making a mess of party/instance/raid chat. And if a person I whisper has a problem with that I assume they have some sort of defect or something and move on.