Another thought would be having external movement not start the timer. So things like disengage wouldn't disrupt the timer, nor would knockbacks like thunderstorm.
Could also have time get banked while standing still instead of a fixed 6 second timer. Stand still for 10 seconds, have a 16 second movement window. Encourages player to stand still whenever possible but comes with less risk of downtime (and therefore a 0% benefit from mastery.)