It is great, and yes the challenge is the same or possibly greater than any of the other modes if you set the difficulty as such (For example, setting difficulty to 2.99 so youre fighting things at Master difficulty but without getting the accessory slot). It also has a lot of QoL functions if you want them, but you don't have to use any of it.
The difficulty slider works really well. As an example, once you set that slider to Expert (2x), the enemy AI gets upgraded, the loot gets upgraded, the enemies stats get upgraded to Expert mode. If you set it to Master (3x), you gain that extra accessory slot you get from Master mode, and the enemies stats increase, etc.
You can really fine tune the game experience, which I like. Say you want to fight bosses on Master mode, but not every little enemy you come across, etc.
Technically, I suppose Journey mode has the most potential difficulty, since you can set the difficulty to Master, and the spawn rate to 10x.