Well I actually agree with the difficulty being about where it is (up or down a little in either direction would be fine) as once you're in full 5.2 lfr / rep gear it should be much, much easier. And if you're a raider, once you get to above 510 or so it should be kinda simple. I mean, I know if I could shave just a couple seconds even off how long it takes me to kill the pups I would probably already have this done.
I think the big part of the problem is the bugged nature of looting the rares on the island, and that bug is easily conflated with the rarity of the drop. I know for a fact more folks would have it if the full nature of the looting bug had been clear sooner. The time it dropped for me, I couldn't see it on the boss initially even though I'd gotten a valor tome off it. If I wasn't grouped with guildies I wouldn't have known it was there.
edit/update for pvpers:
It'll take longer because of how the pvp gear is ilvled, but even with that it won't be long before this is outgeared. There are only so many ways they could gate access to this, and I think rarity of drop and the difficulty of the boss ... and considering people got it with 5.0/5.1 ilvls, by the time 5.2 is over and especially when whatever 5.3 brings around (possibly upgrading again, likely new rep gear) and certainly when 5.4 is out ... yeah. It'll be cake. Doing it now is a mark of skill, as people in the 480s have gotten it.