In short.. HBM is a memory stacking technology that lets you stack multiple memory chips in a very small package. It enables you to do stuff like add graphics card memory right next to the core(same die), dramaticly decreasing distances and shrinking the overall space needed.
Wiki link
And the difference between AMD and NVIDIA is price points mostly. For a good NVIDIA card you have to go up to a GTX 970 atm, while you can buy something from AMD for quite a bit less. Now NVIDIA is probably better than AMD for most people after 970, but for mainstream gaming it's a bit too pricy for my taste(currently). In the end it depends on your monitor more than anything else. 1080p, I would keep your 770 till next year, 1440p --> GTX 970 or higher.