It doesn't matter how much time a manlet spends at the gym. He'll never be as tough or as big as taller men. Taller men can put on more weight which means they can lift more.
Someone who is 6'4 that doesn't work out can literally beat the shit out of some 5'6 manlet that goes to the gym all the time. Height 100% matters.