The more quiet a subjugated people are, the easier it is to subjugate them. Silence in the face of oppression is NEVER the right answer. Because when the people who are affected fail to bring light to the situation, then only the hateful side is ever heard, only ignorance is spread, and things only become worse.
It might annoy you, but your annoyance over a discussion of an issue is nothing compared to the bigotry, the fear, and the violence that those loud "LGBT flag humpers" face.