Just got it last night, and after opening 15-20 boxes or so, you actually get LESS follower gear and more grey items. Upgrading a building shouldn't be detrimental in terms of progression. Sure, you get more items, but also a lot more green/blue sub 600 ilevel trash items. My guildies are experiencing the same thing.
If you're only in it for the follower enhancements, I suggest sticking at level 2 for the time being.