I had already hoped for this but with the advent of trade chat inside of Garrisons with the latest beta push I made a video to show what I hope for.
EDIT:
For clarification:
You are phased to the party leader's garrison by right clicking your own player frame and selecting "View Leader's Garrison". Then it ports you to the entrance anyway.
As well, their garrison may be set up differently than yours, so you cant say meet me at the barracks, maybe they do not even have a barracks or have nothing built.
Cheers!
EDIT: Followup video: