What you should do:
-Keep the lumber mill until you've got level 3 on all your buildings, or until you've banked enough GR to upgrade the remaining buildings.
-Then replace it with the barn, you want it for savage bloods! You need 30 of them to upgrade a crafted epic to 670, with a maximum of 3 crafted epics that are able to be equipped (Thats a boatload of savage bloods that are in demand).
-Keep the inn forever, the follower recruitment makes it the best building. Keep trying to recruit followers with double racial traits (human/draeni etc) until you can stack 3 followers to make every mission 100% success chance.
Also, get a salvage yard!