As far as I understand, there's only three sources of these: missions, bunker/mill, and salvage. I have been making maximum possible use of all three since day one and I simply don't get anywhere near enough upgrade items to "keep up."
Most of my friends have multiple ilvl 655 followers and a bunch around 640. I have two at 645, one at 640, and then some in the 620-30ish range. I simply have not received any more upgrade items than that.
Missions that reward follower upgrades, I get maybe one every three days. Salvage crates contain them maybe one in twenty times. The bunker gives iron horde scrap 95% of the time. I don't feel like I'm failing to take advantage of some source of them, nor am I playing too little (trust me).
What the hell is going on?