You can just pick them all up while leveling. Even if you do every quest in every zone you'll hit 100 at the end of spires and from there there's very few followers left to get by just running around in the world. The only thing that is shared is level 3 blueprint access.