Haste should be your primary stat yes, but I'd say don't go overboard with it. Keep your mastery up to standard so you can still pull good numbers on fights where destruction reigns supreme. At least that will be my approach for when I get the meta gem (it will push my affliction past destruction on single target).