If you were honest, you'd acknowledge that the idea you're talking to was a suggestion by a sociologist to create a jargon that distinguished between prejudice by groups without power to effect change on behalf of their prejudices, and "racism" by those groups who did have such power.
It was never an argument that groups that lack social power couldn't be prejudiced. It was a suggestion about drawing a distinction between prejudice backed by social influence, and prejudice without such, nothing more.