Except for a few niche fights (Malkorok, Norushen for instance) haste is extremely overrated. You're looking at a marginally (0.2 seconds maybe) faster HR cast which also means you OOM faster, and 1 tick of EF HoT which will most likely just overheal and not trigger any mastery anyway, absolutely pointless.
And you are far, far better off until at least 560+ (with amp/mana trinkets, LMG, cloak) dropping spirit for mastery instead of dropping mastery for haste (both decrease your time to OOM, but spirit for mastery is overall better).
Nobody, in my "humble" opinion, should even think about 3506 or 7170 haste until you their regen level is comfortable to the point where they've dropped all spirit gems for mastery gems. Before that point, you'll most likely naturally have 3506 haste anyway (even reforging all mastery) just from amp trinket and pure item level.