That anti-Irish sentiment stems from the Irish being Papists, it is a religious thing.
Notice how the overwhelmingly Protestant nations (Anglo-Teutonic, i.e. England and Germany) were classed as white, but the Catholic ones (Irish Iberian, i.e. Ireland and Spain) weren't?
It is about divisions within Christianity and looking down on your enemy.