I'm pretty sure that Gerudos used some Hylian men to reproduce. But they only give birth to females as it is known now, a male borns one hundred years. After OoT and Link warning about the consecuenses of trusting Ganondorf to Zelda. The King orders his execution. But but the race was not erradicated was in fact banished. If you remember Naboru was the Spirit Sage and second in command to Ganondorf. According to lore, she was pretty old before Ganondorf was born so their lifetime is vastly longer than Hylian. Also Koume and Kotake were pretty old when they brainwashed Naboru and fought Link (Seven years between both scenes) but there some disagreement since is not known if their fused form was in fact they true form. If that is correct, then they actually probe they live longer than most Hylian races.
After hundred years of banishment they retreated further to the Desert and was considered an extinct race.
There's a lot of speculation about the cursed swordman identity. I do not think is the MM link since he went to the woods searching for Navi but he never came back (It is explained in WW intro)
P.S. Anyone know about ALTP remake? I actually hope there is! Was my 1st Zelda game I played in my childhood and the one I think is the best of all.