The optics do matter for physical attraction and arousal, so the initial physical part of a relationship, sure.
I'd have a hard time getting turned on by a woman with unkempt hair dressed in sweats. I mean, she'd have to be ridiculously attractive naturally to overcome how ugly the whole sweats/pajamas in public thing is for me.