Some mobs(mostly bosses) can't be taunted or become immune to taunts after several taunts. It was added in BC so people didn't ping pong bosses between tanks.
If you understand the way diminishing returns (DRs) in PvP works, it should make more sense.
So i'm facerolling about on my mage, i polymorph someone, lasts 8 seconds. I don't feel like destroying them *just* yet, so i poly them again, oh, lasts 4 seconds, lame. Buff mages pl0x. I'm still not quite ready to make this person feel the need to re-roll, so i chain up another poly, lasts 2 seconds. Hmm. Gonna need to engage at least 10% of my brain here and start fighting. But not quite yet. Another poly should do it, oh, only lasts 1 second, crap, quick, hit it again!, WHAT?! HE'S IMMUNE?!
LOL JK. I'm a frost mage, I'll just kill him now. Whats your game?
Where was i?
Oh yeh, taunt has a similar sort of DR effect on the target, overspam it, and the target will eventually become immune.
I would assume this immunity drops after 20 seconds like CCs in PvP, but i could be wrong there.
Oh, and some bosses are just straight up immune to taunt (see; Shannox), so have to manually build up threat, it's a mechanic thing usually, to make the fight interesting for tanks, who generally just sit about doing fuck all.