only times i dont like physical contact with men would be when sitting and the legs(knees/thighs) touch even though there is no need to with ample space.
and about hugs...i dont do it normally with girls either (unless really close friends+) so no way with men.