See that's the one thing that does bug me. I don't care much about the exact intricacies of time travel because that's pretty much always bullshit and the movie obviously is designed so that you just roll with it, much like Strange's magic or Stark's ridiculous super hologram-nanotech nonsense invented by one guy in his basement.
But the movie pays specific attention to the fact that the alternate timelines must have their stones to continue functioning. All well and good... except the prime timeline lost theirs. They're gone, poof, rekt, destroyed, 360 noscoped. That should have had an impact if you ask me.
As someone else said, maybe dealing with reality without the Stones is going to be a plot point moving forward, in which case I retract my criticism but still think at least one character should have went "erm, guys, does that mean nobody has a Soul now or something?".