I never understood this myself. I had more interaction with people in my garrison than I do in any city. In cities I see people mostly just standing around doing a lot of nothing, the only time I ever saw interaction was on RP servers. Order halls are more anti social than garrisons were. In order halls I have literally zero interaction with people. The best you get is Warlocks and DH doing that demon kill thing.
People went outside the garrison for PvP duels if they didn't just go outside Org for it. People often called out dropping a blingtron just outside the garrison for everyone to use. Garrisons had invasions and bosses that required grouping. I had a couple friends who would come see my garrison or I would go to theirs to see the statues they had set up, to see their followers walking around.