"Casual" is never as pejorative term as "Elitist".
I think most of us/you, when thinking about someone being casual is prone to forgive him his casualness.
Casual is more of a not caring, easy going (about the game), free... But you rarely mean "casual" as a bad player. If you want to point out that trait you say "noob", or "clueless", "idiot" or something like that.
"Elitist" on the other hand is much more of a negative term. Because it implies thinking of self as an "elite", which is almost equal to being very conceited.
Elitist as a term defines your attitude towards other players, implying it is not cool.
Casual as a term defines your attitude towards the game, being indifferent in reference to others.
Therefore I must say that the title of this thread is not constructed logically well enough to prevent misunderstanding.
Elistist is not opposed to casual. "hardcore player" is more proper as an opposition to casual.
You could be a casual and still elitist in your attitude.
You could be hardcore but still not elitist.
You could be also modest, easy going and lenient - either as a hardcore player, or casual.
Problem is, human's ego is more prone to think better of itself than of others.
An this ego really gets pumped up when you know you can do something better than others, in this matter - playing wow better than others.
Therefore I think hardcore players are more prone to being elitists, cos they usually play better, and have progress/gear/logs to show for it.
OMG this is long post , but I think it clears up the subject.