Not a problem at all. In fact, I would love it. But the general public would complain that is yet another expansion in Draenor, even if the setting, the ogre continent, would be drastically different and flavorful.
The thing is that Azeroth settings had more chances to be liked, because most of the lore has been semi establised, and most of the stories are, in fact, more of the same favourite theme, like elven civilization, Illidan and even Karazhan, putting Legion as an example.
With the ogre empire, they'll have to pull another MoP. The lore is almost inconclusive and most of it comes from Chronicles Volume II. If they decide to do an expansion in this setting, the new lore has to compete with the more everyday, establised, expected outcomes of many of the ongoing plotpoints and stories of Azeroth.
But if they pull the expansion like MoP, it will be amazing. That's for sure.