Stop using the word casual like that. Casual players are the players who cannot turn this game into a formal hobby such as having an organized raiding team. Casuals can do flex, because it is not a major time commitment. What you are talking about are scrubs.
Second, whether or not I use LFR is irrelevent because it still takes up just as much of the devs time. Pumping out hotfixes/nerfs to LFR right after a patch as other stuff is missed, having the patch take longer because they need to fly in some monkeys from Africa to make sure LFR is easy enough, stuff like that. Everything that exists takes dev time, and I would rather them not waste it making Raiding for Retards. Also, because of the requirement of 25 people, Blizzard is pushing it into as many parts of the gearing path as possible. They know that the moment they have dungeons give even close to LFR gear the place will be dead. See the content is a load of crap for 95% of the LFR playerbase. As long as LFR exists, I also know I am not playing optimally if I am missing possible upgrades from the mode. Lastly it just contributes to inflation, one of the reasons pug requirements right now seem high is because the legendary cloak was earn able by every brain dead LFR Hero, and it has a ridiculously high item level. LFR also causes far more gear to be disenchanted into crystals, which messes up sales.
I can ignore LFR all I want, but it still has an impact on the game.