I've been doing video editing for a few years now and it took a long time for me to get things right.
Here's what I use in vegas for 2560x1440 60FPS on YouTube;
- Mainconcept AVC/AAC
- 60FPS
- Progressive scan
- 1.00 Pixel Aspect Ratio with 2 reference frames
- 50Mbps constant bit rate
As an example of the quality this produces;
https://www.youtube.com/watch?v=gLzAE-GyXZY
Obviously, that's game footage but I use the same settings for other projects (including video shot on a DSLR). You should also allow YouTube at least 24 hours to make sure that it's encoded across the various quality options.