Originally back in the burning crusade, you could get them from killing bosses in heroic dungeons and use them to buy gear with. Later they also dropped in raids.
They where meant as a sort of consolation price, so that everybody got something out of doing heroics or raids.
With wrath new kind of badges came into the game and with each tier new badges where introduced. So that you got "tokens" that allowed you only to buy consolation price gear from current raids. There where also a dayly heroic quest that would grant you 2 badges each day upon completion.
The system stayed in untill last tier of wrath where the current justice point+ valor point system was introduced. With a cap on maximum valor.
Back in the days of TBC and wrath imo there was a certain point that you could reach where you didn't care about current content badges anymore and where surplus badges could be used for BoE's or heirlooms. That was the point where I as a player would feel, that i could stop the grind for badges. And focus on other things. Strangely enough i will say that personally historically I first began really to try capping badges with wrath.
With cataclysm the justice and valor system was fully implemented. And it was relatively easy to cap your valor each week. By raiding and doing the 7 heroics each week that gave you valor points.
Now a days it is harder to cap your points each week and with the upgrade system you will practicly never reach a point where you can stop the grind for badges. Personally I think that this is a bad situation. And in my mind in the long run it leads to burnout. Sure some will say well just do not grind valor then, but if you are raiding one often wants to give your best, and to give your best you want to prepare as best you can. So you kinda have to do this chore.
Therefor I think that it would be prudent to turn back to look at what used to work.
-Introduce old and new ways to burn of Valor that piles up, like reintroducing BoE's or other things like special food that you can send to alts that allows them to gain more xp, or allow people to but elder charms perhaps even coins for Valor
-increase the amount of valor gained from dungeons, raids and LFR.
-remove the upgrade system. The system while nice as a way to upgrade your gear. Is unfortunatly a system that continually forces you to cap your valor.
-remove coins and elder charms although I did write at my first point that maybe you schould be able to buy coins and charms for valor. The whole coin and charm system unfortunatly makes people do daylies not out of choice but because they have to do them to get their coins.