Perhaps instead of doing something you dont want to do for a slight edge if there is one would be to ya know just say no. I myself dont raid and i literally pug my own KSM ( this season was elemental / Resto shaman first season it was all 3 flavors of Druid and next will be Monk ) was it expected of me to have shards? no. It did not stop me from getting invites at all. Tell the people you raid with that you dont have time or a want to do it, its that simple.
I do not know why people always assume the absolute worst all the time.