I'm playing desto aswell, and enjoying it ALOT more than Affliction which is the most boring spec ever made in my honest opinion.
I'm so lucky that I don't have the time to raid in a guild anymore, so I'm just having fun DPS'ing as Destro in LFR.
OT:
If you going Affliction would make your guild kill whatever boss you are stuck on, I would say they are right in benching you, if you refuse to do so.
BUT - If they are benching you, and you are doing fine DPS compared to the other DPS' in the raid, and you have no problem killing stuff, I would say, that they seem like some jerks.
Summary:
If your spec is holding back your guild, then go Affliction.
If not, then they seem like some jerks.