In general i agree, but there are some exceptions.
My height is 1.90 meters, and i would feel myself uncomfortable being with a girl who is shorter than 1.70. In addition to the obvious inconvenience going with such a gap, there is an awkward feeling like you're dating with a kid or something.