First, to answer your question, while you might lose a little bit in stats, the 4pc is usually worth it. You'll get to use Ascendance roughly every 2 minutes instead of 3, which might put it out of sync with Skullbanner, but that's about it.
Second, definitely, DEFINITELY swap UVLS for Wushalay's. UVLS is just about the bottom of the list for us except on AOE, because it does absolutely nothing for us during Ascendance. Wushalay's is 2nd best in slot, after Breath of the Hydra.
Third, sim your character to make sure, but you can probably safely start swapping your yellow slot gems for pure haste, red for int/haste, and blues for haste/spirit. 2xHaste pulls ahead of int at a point. Again, sim to make sure.
ETA: Also, put Pandaren's Step on your boots. Yes, it's mastery, but you really want that extra movement speed. And you can probably use the int/crit spellthread on your legs, especially with Wush. In fact, if you do sim and 2xhaste pulls ahead of int, you can swap some of your purple gems for pure haste, too (such as in your belt and weapon), because the slight loss in int will be made up for by the tons of extra haste, and you won't need the spirit with Wush.