The difference is that while the imperial units are arbitrary, the only SI unit that's arbitrary is the kilogram. But it's being worked at!
Every other has a specific variable to let us discover it again. If the exact lenth of an inch was lost to humans. It'd never be discovered again.
If the exact lenth of the meter was lost, but not the sorce (Which depends on us still having the same definition of a second).
The celsius is again recreational at observing water and grading from there at exact points of entalpy. (The only reason celsius is still used is that it's a bloody mouthful saying that it's threehundred and two degrees outside, while twenty five degrees is easy)