One of the guys might be getting the legendary, this should be more than enough for him to listen to you... Remove him from the next run you guys do, and let him feel the legendary slip away. I will actually promise you, that he either listens or leaves.
The legendary is simple too juicy not to raid for...
And having 2 people that doesnt listen will halt your progress on Raggy... So your choice is simple, they either listen or you recruit 2 new people and kick the other two.