In my experience, it's not the muscles that attract women so much as confidence does. There are thin, nerdy-looking guys who are awesome, know that they are awesome, and have little trouble attracting women. And there are really ripped guys who are obviously trying to cover up their poor self-image, and they are still as alone as they were before they started working out.
The fact is, however, that confident people tend to take care of themselves better than people with poor self esteem. Which means that, on average, they'll be somewhat fitter than their counterparts. Even the people who are thin, nerdy, and awesome are usually not sickly-looking thin, they're just not musclebound particularly.