This whole expansion, and some of WoD, has been showing glimpses of what's to come: space, the final frontier.
I mean, the Great Dark Beyond, the final frontier*
The Great Dark is just that. Space. Much like our physical space (everything outside of Earth's atmosphere)... the Great Dark Beyond is the physical universe that Azeroth and many other planets inhabit.
The Twisting Nether is a separate plane of existence; a separate space to dwell, but instead of physical it is ethereal (to my knowledge) which occasionally "bleeds" into the physical universe as evident by the green, glowing streams of whatever you see in the skies of Outland.
So are we going into space? Most likely. We just traveled to another planet (albeit through a portal rather than a trek through space) on a spaceship. We are visiting other planets through the Invasion Points.
Hell, even our creators were planets before they birthed.