There two big variables that play a role and they clash together to make an even bigger mess. Especially the second variable is completely uncontrollable by the player hence it will never be fixed without another design. Namely,
1) The abilities of the followers are unclear, they get random traits, random extra traits (quality), with an uncertainty of how they may be used in the future
but most importantly,
2) The missions are completely random. You have no idea if the followers you are making are going to be any good or not.
All in all, it's a total absolute mess to organize or predict.
It may end up being just "let's level them all anyway, and then deactivate/activate as we go along".
But ending up to that is frankly a bad design since it takes away the whole point of organizing it.