I don't know how common the fear is, but it is fairly common for gay men to feel they don't have to respect some touch and personal space preferences, because to then it's not predatory because they're not attracted to your gender. The basis for the fear is real.
Do you think it's common to feel this way for women?