Another explanation (for alliance anyway)
1) Garrosh and Kairoz need a place with tons of magic to block their time travelling being detected / to enable the initial jump back in time. Kara is the best bet for this sort of thing, as it's space-time is messed up already. They appear at Chronal Spire, which Kairoz makes into his lair / raid instance. Note - this spire is completely inaccesible without a flying mount, so good luck getting there.
2) Eventually, Kairoz fixes the dark portal at both ends so it does time travel, not just teleportation. Horde starts charging through from Draenor.
3) Khadgar shows up in SW, needs some help to fix this malfunctioning dark portal. Needs some help clearing kara for some spellbook or something. You know - "LFG Kara need all, spellbook reserved"....
4) On arriving, discover that the inhabitents of Kara have been already been smashed by Garrosh. He even looted his weapon off the last boss - after Jaina ninjad his last one. Eventually find where Garrosh and Kairoz made the hole in space-time on the upper floors, but it's closed already, and Khadgar can't re-open it.
5) Khadgar tell us he can't close the dark portal from this side - he tried at the end of warcraft 2 and failed. But he can adjust the portal back a few years, so you'll appear where the portal's still being built. (kill 100 orcs outside the portal as Khadger does it)
Your mission: to stop the portal being opened, or otherwise secure it so that Khadger can close it at the other end. Lore-wise, it's a 1-way trip, as the portal isn't finished at the other side, but after the first quest at the other end, Khadgar / Jaina pops through and gives you a portal back to SW
TLDR: Garrosh and Kairoz use Kara to go back to Chronal Tower. We just use a 1-way dark portal.