Well the syndicate are annoying, and there is little incentive to work for them. But I personally find Ravenholdt the most annoying. Lore-wise, they are a bunch of smug ninjas and grinding their reputation offers little in the way of incentives and rewards. Also the higher reputation you gain, the more the guards start to recognize you, including insulting you for being in the lower orders.
Waterlords. They don't even count towards the Insane or anything like that. Nowadays there's no reward at ANY rep level, hell you can't even see these dudes around anymore! There's not even any decent amount of lore behind them and getting Exalted is a pain since only raid bosses in MC give rep after Revered.
...and yet I did it anyway. There's something seriously wrong with me.
An interesting question, since it is a combination of two. The most pointless faction isn't necessarily the most annoying one, and a faction doesn't have to be annoying to be pointless. Out of the list, I'd have to say...
Most pointless: Syndicate. No rewards, and you can't hit exalted. Pretty pointless in my book.
Most annoying: Nat Pagle. There are rewards - good ones too - but damn. That's all I can say. Damn. You can't even fish the little bastards out of pools.
Most pointless and annoying:
Ravenholdt. In my view. There is a point, but not much of one, and bag space limitations combined with the fact that you need a rogue make this one the perfect balance in my eyes.
The older reps that I didn't get on my shaman from vanilla and BC (since the character was created at the start of BC and was an alt up until middle-end of T6). I ended up going back during Wrath and Cata to complete many of those reps...urgh. I remember Keepers of Time being particularly annoying.