But Cho is correct, there haven't been any Wandering Isle Pandaren on Pandaria for thousands of years. There's a difference between Pandaria Pandaren and the ones that come from the Wandering Isle, it's just that the ingame model doesn't reflect this. (as it'd only be minor differences, at most.)
The ones on Pandaria (mainly Tushui) thought the ones that wanted to explore the world beyond the mists (mainly Huojin) to be crazy.
They left Pandaria on the back of the Turtle about 10,000 years ago. The ones on the Isle have been living a different life for a very long time, compared to the ones on Pandaria.
Unless you forgot that your Pandaren character has never been to Pandaria.
Though, I do agree that the Wandering Isle Pandaren lore has been lackluster.