Only done ToT once on my hunter so don't know about the flys, but when fighting panda rares you need to call your pet back when they do the spinning crane kick. Those rares were trivial as a hunter, blizzard didn't like.
Panda rares were "hotfixed" since they were too easy to kill as a pet class.
I find it darkly humorous that I have a far easier time soloing Warbringers now than I do any Pandaren rare. That kick hits harder than a Vengeful Spirit. You can use Master's Call in conjunction with /petfollow to get your pet quickly away from both, though.
The Pandas are suppose to be hard I believe, I've died a number of times on my tank to that kick too. As a hunter I deal with them by scatter shotting the kick as soon as he does it and silencing shot the heal. As far as the flys, they hit like trucks too, pet might just have had aggro? Overall though, hunter pets, even in tenacity spec feel much less tankier than in previous expansions.