Ghostcrawler delivers the formula for the Armor Pen cap in the following bit of pseudo code (I've cleaned it a bit for readability sake):
Let C be the Armor Penetration constant.
if ( level < 60 )
C = 400 + ( 85 * targetlevel )
C = 400 + ( 85 * targetlevel ) + ( 4.5 * 85 * (targetlevel – 59))
Thus where a level 80 mob is being attacked, C = 15232.5. For a level 83 mob C = 16635.
The cap for Armor Penetration is then: (Armor + C) / 3
A quick example, used originally by Ghostcrawler but extrapolated upon here:
Assume a level 80 death knight NPC (note: non player character) has 9729 armor. C for level 80 mobs is 15232.5. The Armor Penetration cap is (9729 + 15232.5) / 3 = 8320.5. At this point the game will choose the lowest of two values, either 8320.5 or 9729, to represent the most armor which can be ignored. We'll call this value X. So X = 8320.5.
A person is attacking with 30% Armor Penetration. The game will then compute the amount of armor to be ignored against the previous value X, which means the amount of armor ignored on our level 80 death knight NPC is 2496.15 (30% of 8320.5), which we'll call M. M is then subtracted from the death knight's actual armor, which means the attack goes through against 9729 – 2496.15 = 7232.85 armor present.