I think it was pretty clear that he knew he was in over his head. In my opinion the goal was to last long enough to goad her into exposing her true stance on things. Which obviously he succeeded in doing.
What I don't understand about Sylvanas' character arc is, at the start of Legion she generally seemed like she was about doing the right thing: she saved Varian's ass once during the assault on the Broken Shore, and retreated not out of cowardice but out of desperation when Vol'jin was gravely injured. The fact that she sounded the retreat on his behalf - however awful it worked out for the alliance - was largely a benevolent act. And even when Vol'jin chose Sylvanas as warchief, there was reluctance on her part, not 'NYAH HAHA, AT LAST THE POWER IS MINE'. I know she tried to enslave Eyir, but that could be seen as an act of desperation as she was running out of Valkyr.
So what happened between then and BFA's prepatch to turn her into a hope-killing 'mad queen'?
.
Its mostly called different writers with different ideas on where to take her character.