Yeah let's ask ourselves something. Legendary has to mean something. Would it be a good idea to require 9/9 gold ? Personnaly I think so.
Afking in LFR requires no skill, whereas getting 9/9 gold means you kinda know how to play. The casual excuse doesnt matter : you dont need time, hell you dont even need gear.
I always found the challenge mode rewards highly lackluster. I think gating the legendary quest to them would be a good idea. The bads wont have a legendary, at least