I have seen quite a few people complaining about anything they don't personally enjoy. Any one of the raid difficulties might not be for YOU, but if you have a look around, there are plenty of people who participate in all 4 of the difficulties.
Seriously, you've contributed absolutely zero reason to make a change. It's a fact that all 4 difficulties are heavily attended. None of the people with your line of reasoning ever explains why any of them need to be removed. Same exact thing with people who want other aspects of the game to be removed even though any player can simply ignore what they don't like.