Crime rate and lack of assimilation.
Take in mind media presents crime rates more then what good those people did, so crime rates are a bit skewed.
However, lack of assimilation, when you live in a free society and then you see someone who considers their women to be some sort of objects (just an example), you can't accept such a person. And as such you tolerate them and their kind less.
However, in general those that do assimilate we have no problems with. The most of us at least, there's of course the old cranky people who consider every immigrant evil.