You can use the Broken Crown to speed up the farming. Aside from that, maybe goblin farming can help? I doubt that its more efficient than rifting/doing bounties though.
As ophion1990 said use a broken Crown (even if its cubed) with the gem you want to collect in your helm. The higher the torment the better. And any puzzle rings keep em so you can make a portal to the vault! Which is a lot of gold/gems. Theres not too much else you can do other than farm farm farm. they will build up eventually.
An easy way to target farm the gems is to do the bounties where you kill a boss. The chest that spawns always has several gems (can just switch broken crown to loot the chest and back to your normal helm).
So create a game, do those boss killing bounties in each act that pop up (killing bosses without the bounty doesn't give you the chest) then exit game and repeat while swapping in broken crown (or just wearing or cubing as others said).
Dont forget the cube. With some gold input you can change the gem colours you dont need to those you actually need. I never ran out of gems, just of my favorite color. Oh and of course it simply takes time to get all gems at max quality. But no rush, at some point later you have too many gems....
Not sure how long you have been playing but I have around 1200 of each gem at Imperial Quality and 2,2 billion gold on S4. And I played for about 5 weeks on Hardcore.
Just plow through rifts. Pick up every gem you see. I still pick up gems. But that is just because of old habbits. Also cube Puzzle Rings, they are great for both gold and gems.