There's a couple reasons why I have a problem with time travel when I can more easily suspend disbelief for the other issues you bring up. One is because of how often it is used. When you have a story that's not about time travel, then time travel is introduced, it's almost always (if not always) the answer to the writer admitting either, "I don't know how to fix this situation" or "I have no more ideas for where this story goes next". It's an overused trope. You can see movies like Superman I and Superman II where Richard Donner used it almost identically to fix both movies. Secondary, it's the ultimate story mulligan, and ends up removing tension, because you know someone somewhere can just time travel us to the point before the point and fix whatever was unfixed or refixed, by which point you might as well just be watching Stewie and Brian trying to re-fix their mistake over and over again.