"Challenging" isn't a binary scale, it's relative to character power, group teamwork and player ability - it also applies to both difficulty and complexity. It doesn't necessarily need to require min/maxing, flasks, potions, voice comm, etc. Nor does it necessarily mean players need to repeatedly try the same dungeon/content for 3 hours just to get perfect choreography. The point of the game is to be fun for different players and the word takes a different context for 5 players from a heroic raiding guild compared with 5 players in a casual/social guild.
It doesn't necessarily need to mean 1-shot abilties, although those are valid too. Challenging can also mean "you or your party can make 1 or 2 mistakes, but you can't make 3".
In any case, look back over the years for all the different kinds of mechanics Blizz have used to make players use their brain; sometimes on their own, sometimes with different kinds of 'punishments' for mistakes, but usually mixing them up together:
- Disruptive mob abilities - e.g. mobs casting AoE fear, MC'ing a player, sheeping the healer, healing the mobs to 100% hp, etc.
- Situational awareness - Needing to watch for incoming patrols, summoning adds, or mobs which "run away in fear" on trash.
- Higher damage - gear requirements, need to use defensive cooldowns, need for healers to conserve mana, need for healers to react quickly.
- Coordination on-the-fly with other players - e.g. needing to run to/from a particular player or needing to click a player to release them from a trap.
- Reactions to mechanics (e.g. mobs which cast uninterruptable abilities, moving to/from things on the ground).
- Require non-combat abilities and class/role-limited abilities (dispels, stuns, slows, interrupts, CC, speed boosts, etc).
- Kill order/priorities (used to be common for TBC heroics) - e.g. Mobs which get more powerful as they take longer to die or keep summoning adds.
- Ensuring that control is not exclusively the responsibility of the tank (e.g. mobs which need kiting because they 1-shot the tank, or mobs which don't have aggro tables).
- Forcing players to change their play style (e.g. requiring constant movement so limiting to instant-cast abilities).
- Abilities which require good mob/boss positioning (e.g. Valiona's dragon breath in Grim Batol).
Some challenges are a little role/class-dependant - e.g. movement is easier for melee than some ranged dps, but "punishing" doesn't have to mean the same thing as "1 shot death", failing to deal with mechanics might just mean taking extra damage, players being stunned for 8sec, knocked off a wall and needing to run back up, etc.
RNG is pretty standard too - in fact, mobs and bosses without any RNG at all just get boring and formulaic really quickly. I can't really think of many dungeons or bosses which have zero RNG aside from Patchwerk. Even if the RNG is just a random player being targeted with something or a random chance for CC to break etc.