Do we know that?
"Infinite afterlives"... Tons of gateways floating around the arbiter... Seems to suggest that there's room for more out there without it being an "asspull."
It could very well be that these four particular realms have a connection to Oribos specifically because of their specific roles in the Shadowlands: Bastion to ferry souls from their places of origin. Revendreth to rehabilitate or condemn souls on the brink. Ardenweald where gods from the various realms are reborn. Maldraxxus to serve as the military. We don't need to hear about all the "infinite" others that may exist since they don't serve these functions and so don't require any official representation within Oribos itself.
Looking at this map and player mechanics as some indication of the actual state of things is like looking at the map of Kalimdor and thinking that a character in that setting could walk from Darkshore to Silithus in a day.