Okay. First of all, a sinus infection can also be caught with 100% sterile water. The bacteria responsible for it are basically everywhere. Usually, your body deals with them easily on its own. However, when you get your sinus clogged with water (they are ususally supposed to have air in them), you give the bacteria better conditions to grow. Chlorine also causes the production of mucus, because it inflamates sensitive tissue, which can also lead to bacterial infections.
That being said, you had swimming lessons, with other kids I assume? One out of five kids usually has a cold. Not he harsh "have to stay in bed" kind, but runny nose, etc. You can easily catch one and have a sinus infection as a direct result.
Sinus infections have dozens of reasons. Wet environments help most of them.
And as to the geometry: Again, size isn't the only relevant factor. Handling and age of the clothes is, aswell. Do you have a pet sleeping on it? How regularily do you wash it? How old is it? What is your state of health? What do you think is more likely to carry contaminants, a new pair of jeans, or a pair of hotpants that you hiked the outback in?
Then there is the quesiton of the fabric. Cotton carries more dirt than silk. Etc. You are trying to simplify a complicated subject to further your beliefs.
Do larger clothes potentially carry more contaminents then smaller ones? Yes. Do they always? No. Does it matter? Not at all. Chlorine will usually deal with all relevant bacteria.