Basically, an AoE damage cap is a HARD cap on the amount of damage any single aoe spell can do. At low numbers of mobs, you get the max damage capable of the spell including spell damage gear, pots, trinkets, etc. However, once the mob numbers increase to the point you start hitting the cap, the Damage cap number is divided evenly among the number of targets. That is why you will see Seed explode and hit for a range of 1300 to 1400, but once you hit the damage cap EVERY hit will be very close, like 849, or even as low as 600 as I have seen sometimes in Illhoof.
It is important to note that once you hit the damage cap, it doesn't matter if you trinket, pot, pop arcane power, or get naked. The damage cap prevails over all and is inflexible. The exceptions are crits---(the extra crit damage does NOT go towards the damage cap) and any debuffs that might be on the mob like Curse of elements.
Cone of Cold ~ 6500
Maxed DPS without crits ~ 4330
Arcane Explosion ~ 6800
Maxed DPS without crits ~ 4530 (12.5 DPM)
Flamestrike ~ 7800 (*no clarification given if Dot effect is itself effected)
Maxed DPS without crits ~ 2600
Seed of Corruption - near 12,700
Maxed DPS without crits ~ 6350 (14.4 DPM...plus whatever you get on the DoT effect--usually zero in a raid situation)