You can refer to people in many ways, including ways that don't distinguish them from others. When you choose to refer to them with a word that focus on how they are different, you put emphasis on that, and others can assume that what makes them different matters more to you than anything else. And this sentiment is often associated with prejudice.