In real life our ears are 90% finished growing by age six from what a quick google search indicates. So it is possible that Elven ears start to change at a later age. Or it simply could be longer hair obscuring differences. Turin, a human, was mistaken for an Elf at times. So even fully grown, when differences would be more apparent, they could still be confused.
At least we know hobbits had slightly pointed ears since Tolkien explicitly corrected an artists depiction. It is strange at times what details Tolkien left out despite having such a detailed work.