They were also nomads. Aang had visited both the earth kingdoms and the fire nation by the time he was twelve, remember? The exact details aren't clear, but it's likely that, while all children are raised in the temples, the adults travel the world and the elderly act as monks.
If a single gene was responsible, it could still be genetic, but it probably wouldn't allow an energy-bender to survive for hundreds of generations either.