There's a good GDC video on cursed problems in game design:
https://www.youtube.com/watch?v=8uE6-vIi1rQ
This is a cursed problem. There's two conflicting paradigms here.
The first is a desire to make content that is engaging and challenging for players at any skill level. Content that is appropriately challenging, that is, not so easy that it is boring, and not so hard that you don't enjoy yourself or learn anything by doing it, is fun. This is an achievable goal generally speaking, although the gap in player skill is pretty wide, it could probably be done pretty effectively with a typical Normal, Heroic, Mythic system.
However, it's made impossible by the second paradigm, and that is extrinsic reward and motivation through increase of player power. Everything in the game is driven by a desire to increase player power. It's what makes WoW what it is. It's what makes it compelling, and has always been the case, though maybe not to the same scale as it is currently. Content that is challenging generally has a chance to fail, and takes a time investment, and is intrinsically rewarding. However, players will avoid it unless they have a "reason" to engage in it. When they do engage in it, and it's at the right challenge level, it's fun even if there's a risk of failure, and if there's absolutely no risk of failure, it's not an appropriate challenge to be as fun, it's just a boring grind. The reason that WoW has always given is a noticeable increase in player power. They've done a lot of work to determine what is noticeable. This is generally getting a piece of equipment that is 13 item levels higher than what you had previously. They want to make sure a single item upgrade is a difference you can typically see. This is why tiers are 13 ilvls apart, and why the ilvl formula has really not changed fundamentally since late wrath, early cataclysm. If you know you will get stronger, you will do the harder content, and if you do the harder content and succeed, it's fun.
The problem here is these two systems are incompatible. If you get noticeably more powerful, and each of your 14ish items can be upgraded into something noticeably more powerful, the content that was appropriately scaled for your skill level will become trivial before you get much more skillful. To continue to provide a challenge, they need to provide more content at a more difficult skill level, or scale it to player level. However, if they scale it with player level (which they tried in a limited capacity in BfA) players lose the feeling of progression that makes them feel justified in challenging themselves. Players want to get stronger, and noticeably so, but also want to be challenged. You also have the problem of skill differentials. A player with ilvl 200 gear in season 1 Shadowlands might struggle with a +5, but another group with the same gear could complete a +15. Gear lets players scale in both these dimensions, a player with higher gear can do content that might be scaled for a player who is more skilled through the gear power differential.
Because of this, you can't make content that remains challenging without taking away gear scaling, which they've tried to do from time to time, templates and challenge modes for example. But the fundamental basis of the game is about meaningful player power progression. Other games have tried to use less player power progression, but these games have all missed out on the fun of doing the content to get the upgrades to make your character incredibly powerful, which actually feels really good. Some of those games have done well in their own way, but they aren't the same scale as WoW. A game like Final Fantasy XIV takes this approach and is similar to WoW, but the endgame is way more about socializing, cosmetics, and prestige than it is about the actual gameplay. The challenges they need to put into boss encounters have to be more technical. The rewards they have to give need to be cosmetic or prestigious. But another big problem is that player skill will always outgrow the challenges. Finishing an ultimate encounter might be fun and rewarding and hard to get through, but then you've mastered it, and you might want to feel that challenge again, but you can't. You need to wait until more content releases.
So both need to exist for WoW to be what it is, but both can't exist without being in conflict. Mythic+ was a good compromise to bridge the gap. It allows for gear progression, it allows for skill progression, it allows for a certain level of challenge tolerance. But from this there's another problem. You have individual differentials in skill level and challenge tolerance, and when you try to match strangers together without knowing this information, you have more conflict. Some people like to do things that are a bit of a long shot, I am eager to try a key that's a bit too hard for me and I'm happy when I just barely fail it because it means I'm really close. Some people hate not making the timer. Some people hate even the possibility that they might be close to not beating the timer. Some people are able to do content that they're far undergeared for because they just really understand the requirements and are focused and can execute. Some people need to greatly overgear the content before they can do it. And finally, past success doesn't translate into evidence of capability because people can get carried.
Mythic+ is totally a lot of fun in a group of friends, even if your personal skills and preferences are a bit different, because you move as a group. Maybe a group of people like you can do +17s easy, but with your friends you're doing +14s and struggling, but because you're kind of carrying the group, it's still a demanding challenge, you're still maybe seeing gear upgrades, still getting valor points, maybe a bit disappointed in your progress, but maybe playing with your friends is incentive enough.
But in a pick up group, you run into issues. You like to punch above your skill level and go for bit of a long shot, you invite a person who has good gear but isn't very good, but has been carried by his friends and HATES losing. You have no way with what is presented to you in the UI to know this person's history. You see his good gear, his good score, you start the run, something goes wrong on the second trash pack, you are running at max mental capacity dealing with what you can but you don't avoid a frontal while you're trying to keep up with damage, you die, the guy you invite swears and calls you a mean word and leaves the group, your key depletes.
It's an imperfect solution. They're improving it a bit by showing things like rating, ilvl, but these are imperfect too. Some of the 9.1.5 changes to the group finder will also probably help, but also not be perfect. There's too many variables and tolerance levels and skill levels that it makes it hard. There's other things like, they could take away key depletion, but then people would just bash their head against the hardest difficulty forever instead of sitting in the sort-of sweet spot. I think a good compromise would be to not deplete a key that gets completed without going over time by more than 10 minutes, and to reward 2 full ilvl items, but not upgrade the key. That way if you're kind of close, but maybe wipe on a tyrannical boss, you still get the rewards, you just don't get pushed into a harder level with the next key. It would also stop the yo-yo feeling, where if you're good enough to do a +15 but getting better to do a +16, you won't have to struggle against a +16, then do a +15 which you've already kind of mastered, then do a +16 again just to have to do a +15, and then when you do get your +17 key, when you've just barely timed the +16, you totally won't be ready for it. If you had a range where you didn't get a key upgrade, but also didn't get a downgrade, you would go from repeating 16s until you could time one, and then get to do 17s which you would also much more likely be able to complete without downgrade but also not upgrading, for some time.
Your solution to resolve the problem is to cut out the player power progression or make it far more flat. This basically unmakes WoW. WoW has always been a game about doing challenging things to make your character stronger and able to do things that were hard for you in the past. Other games have flatter progression. They miss some of the compelling parts of WoW that come from the player power progression. But that player power progression also will always cause problems relating to trivializing content, power inflation, and a difficulty in separating character power from player skill. These situations need to be mitigated, but can't be resolved, because trivializing content, power inflation, and compensating for skill with character power are some of the fundamental reasons that power progression is compelling.
It's fun to power up your character and completely obliterate the content that was hard before. At the same time, this will fundamentally mean that content that was hard before can not be hard now. If you correct this, say, by scaling the content, then you lose the fun of powering up your character and obliterating that content. Mythic+ mitigates this, because you still get to power up your character, you still obliterate the lower level keys, you still have a challenge in higher level keys, yet fundamentally the +10 and the +20 are identical, apart from a scale factor for health and damage.
There are already just a few actual difficulty levels in M+, you have the 1-3 range, which has a single additional affix, the 4-6 range which has a second mechanic, the 7-9 range which has a third, and the 10+ range, which has a fourth. The rest are simply numbers. This is similar to raids, where you have LFR with limited mechanics, Normal with an additional mechanic per boss typically, Heroic with another, and then Mythic where they take out the rest of the stops. Similarly, in raids, there's a progression. The first boss is generally the one with the lightest requirements on player power and execution. Subsequent bosses demand more and more. It's not always linear, and sometimes the 7th boss is easier than the 4th, and it's not completely unheard of for the last boss to be easier than the second last. But just like Mythic+ doesn't just force you to go from 1 to 5 to 10, raids don't force you to go from a single normal boss to a single heroic boss to a single mythic boss. You progress, if you're in a group who is kind of normal level, you do normal tarragrue, then you do normal eye who is a bit harder, then you do normal nine, which is a bit harder, up through to sylvanas, where you might decide to do heroic tarragrue after, and now you deal with the new mechanic of soaking orbs on top of what was before. By the time your group has gone through normal, they're going to be better geared and more comfortable with their class and characters as things have gotten progressively harder. Five Mythic+ levels multiplies health and damage of all of the content by about 150%. That's enough to take a group that feels really good and crush them. Give a group who just got comfortable with +15s and wants a new challenge only the option to have +20 keys and it will not be that fun for most people. One M+ level is enough of a difference to be hard, but small enough of a difference to feel potentially possible until you're really playing at 100%.