When the majority in the groups have read that you write "chill" they probably read is as "I am a bad player" and react accordingly.
In an automated grouping system you either go with the majority or you have to man up and make your own groups with like-minded people.
Blizzard clearly states that the majority in the group is always right:
"Groups are empowered to manage their own members through the vote-kick system. If the majority of the group feels a player should be removed, they can initiate a vote-kick to remove them. The system is entirely under the control of our players. Players may choose to remove any other player from the group for any reason, provided the vote to kick passes." (My bolding)
https://eu.battle.net/support/en/article/52363
Remember that you might feel that you have the right to demand that other people play at your pace, but the 4 others in the group also feel that they have the right to demand that you play at their pace. At the end of the day the majority wins as they can kick the minority