Quite simple really: instead of ignoring them, you kill them. Have about half your DPS on it after 3-4 spawns and you should be fine (depends on your DPS if more or less are needed), always have everyone with the anti-fear / 60% buff kill them. When boss gets to like 20% ignore them, blow hero and loot boss.
We assigned all DPS to either left or right side and only DPS'd adds when they were up, no focus on boss unless the side you were on is dead. Also they go down fast when people who got cackled come back with their buff and focus the adds.