Why would you want to be in a group that doesn't want you? When you could make your own group and invite whoever you like?
Players focused on M+ score are wanting to play the game a very particular way (group stacking, highly competitive, key pushing) and want to group with others of the same mindset. It's not their fault you have other demands on your time, and it's not their 'job' to carry you.
Group Finder is set up so that anyone can make whatever kind of group they like. It works best when the group description makes clear the experience they are looking for and the kind of players they hope to group with. Groups of people with different goals/level of commitment are groups that end up with drama. There is absolutely nothing wrong with being a casual player who wants to "just play the game"; in fact it's great and you should be looking to group with other casual players who want to just play the game.
Personally, I play the game differently ways on different nights. This week of pugging some regular mythics on my various alts for the weekly cache has reminded me how important it is to group with players who want the same experience. I wanted to "pull all the things", see how fast I could push it, and wasn't fussed about some hiccups along the way. First group I joined did NOT want that experience, and was so offended by my chain pulling that they all rage quit lol. After that I started my own groups and put in description "FAIR WARNING: I am pulling all the things!" The groups I got with that approach knew what they were getting into, wanted the same thing, and we all had fun together.
TL;DR: Group with players who share your goals and level of commitment. Stop trying to force yourself on groups looking for a different experience.