It's a long fight and basic Fury gen should never be an issue with adds in all phases. So mathematically, imp swarm is just inferior. And imps don't really provide burst damage, they're primary function is to generate fury. In transition phases as noted, fury gen is through the roof anyway, imp swarm doesn't significantly alter the timeframe to build up to a DB dump.
That said, I've tried both extensively (with and w/o) glyph. I don't really notice a difference overall, in damage or ability to burst on mages. My advice is do it based on what you think benefits your raid the most, I don't think it matters that much.