There are a couple of fights that come to mind where the damage is physical. Grand Empress hard mode comes to mind. Switching an offhand there for a lower ilvl shield increased my survivability a lot. However I played with a 522 staff until I got a decent replacement this tier, and if there is a hard hitting physical boss ability I just did a macro weapon swap.
i.e. a shaman should have a shield in this backpack to swap to on progression