It's the same reason the pandaren stuck with the Horde after Garrosh went on a racist rant and then threw them into a deathmatch against massive beasts from Northrend without warning. There's a point where you're in too deep; if Sylvanas just incinerated the night elves' main civilian population center (seemingly) out of spite because a night elf mildly sassed her, dollars to donuts says Thalyssra had no illusions the Horde would march on Suramar next if the Nightborne pulled out.
As for why they joined the Horde, it's a simple matter of diplomacy. The Nightborne wanted allies so they had places to fall back on while spreading back into the world (again, like the Huojin pandaren). Their choices were the Alliance, represented to them by Tyrande, who was outspoken about her lack of trust and intention to use the Nightfallen as cannon fodder, or the Horde, represented by Liadrin, who spent the whole war forging personal connections with leaders in the Nightfallen, empathizing with their plight, and drawing parallels to the blood elves' situation before joining the Horde.
That Sylvanas marched to war just as the ink was done drying and the Nightborne were officially part of the Horde is completely separate from their decision to join the Horde beforehand, and afterwards, leaving the Horde would have left them without allies and having just made enemies of someone who torched a city without a second thought and her massive army besides.
edit: Now, if you want to discuss actions that run counter to Thalyssra's established personality, we can discuss her and Lasan's complicity in the burning of Stormwind while they escaped with Talanji and Zul, instead of Thalyssra showing off her talents as First Arcanist by breaking the teleportation barriers Jaina and the other mages had put in place, or by setting up one of those relocation grids they used all the time in the Nightfallen storyline to circumvent said barriers.