This isn't really true - Logic is a fundamental component of any proper Philosophical degree, and it gets very mathy. Physics isn't necessary to philosophy, but physics isn't the entirety of maths.
Philosophy doctorates and professors are usually very careful about what is possible through speculation. What you are more likely railing against, is people uneducated in philosophy, who wax philosophic about theoretical physics.
The problem isn't that they are Philosophers talking about physics, it's that they are Not-Philosophers talking about physics.
If a Not-Aircraft Mechanic started giving you financial investment advice, by way of reference to aircraft components - you wouldn't conclude - "Aircraft Mechanics give shitty Financial Advice".
The real underlying problem is more so that Philosophy is central to the human experience - we're all armchair philosophers (and have a right to be) - but without proper training every stoner is compelled to still ask the big questions of their existence, but lacks the rigor to state assumptions and apply logic.
What we need, is philosophy taught in public education - particularly logic and critical thinking (where philosophy always begins).