As other have said, it was bloody exclusive, which always help polish the rose tinted glasses.
The barrier-to-entry required your character to pretty much have slogged it all the way through Scholo/Strat/LBRS/UBRS -> (ZG) -> MC -> (AQ20) -> BWL -> AQ40 and actually having had the luck to get geared from that trek (remember, 40 players in the raid ~20 items per raid - and +Spelldamage / +Healing were exclusive to their particular role). And of course, at least for the first while, stats on gear was atrocious to the point you might as well DE half of it (Plate with strength and spirit? Why, of course!). And during that trek, you had to continually refresh the pool of raiders, as your raiders would inevitably be poached once they got a bit of gear.
You could do PVP, but in order to get the same quality items, you had to put in a brutal number of hours.
Then there was the, for the time, rather harsh levels of coordination required. I may be wrong, but Naxx hit at a time when server instability was still a bit of an issue (See http://www.youtube.com/watch?v=lzsgD6N5Yow for how Ravencrest EU was at the time), so fights like, oh say, Thaddius and 4HM became instant wipes. And speaking of 4HM, that only upped the amount of poaching that raiding guilds were exposed to. Got a 4pc T3 equipped tank? Good luck holding on to him.
And of course, the amount of preparation time for raids. This was the days before fish feasts, cauldrons, etc. and having x number of resistance potions on you was mandatory (Hello Loatheb). Farming stuff took a while, unless you were lucky enough to rule the AH and could buy the mats.
And finally, it was released so late in the cycle, that very few raiding guilds had the time and skill to clear the place before BC hit. Patch 1.11 hit June 20 2006 and 5 months later, we saw Patch 2.0.1 and the raiding scene more or less collapsed. I've heard of realms which didn't see a Server-first Kel'thuzad kill until after Burning Crusade had been released.