Caring about whether someone wants sex and having sex with someone without their consent are two very different things. One is rape, the other is not.
Secondly, unless these men are raping the women they want to have sex with, they care about consent. And because they care about consent, they care about whether or not she wants it.