Kinda hard to do that when there's a heckin huge sword still stuck in the planet, eh?
From what I understand the pillars were said to have been given to the Keepers by the Titans after their creation to assist in their task. The Titans had already left Azeroth by the time the well was made, since the Titans never came back, they had to have given them the pillars by then.Plus I'm not sure the keepers even had the pillars at that point. They might have been made during different ages even.