I guess you are right.
I doubt any casual players would enjoy legion, as most casual players have better things to do outside of the game. I think the only people who would enjoy legion are the same people who enjoys doing the same mindless things over and over again, because they got nothing better to do with their lives. If you are the type of person who enjoys grinding for the sake of grinding without asking yourself why you are doing it, then legion was your expansion. Most casual players though have probably quit the expansion a long time ago, which would explain why the servers are so empty.
Id rather have an expansion with quality content thats worth playing, instead of an expansion with an overload of boring, repetitive and mindless singleplayer content mixed with RNG legendaries, which was what legion was all about if you look back at it.
Hopefully they can make a comeback with BfA so the game will be worth paying for again.