The garrison wasnt the problem though, the problem was the lack of content to do on your own. You only had your Garrison and nothing else to do for almost 2 patches, apart from raiding.
But if you decide to not return to play the game because of something you see in the alpha version of the next expansion then you shouldnt return, because it almost seems like your looking for a reason not to return anyways