Hey All,
I was perusing the WoW sub-reddit and came across this post today and I confirmed myself what the OP described. I made a demo video below demonstrating the bug, but a TL;DR: is that it looks like at least PBoI is only calculating it's contribution to your stats when it is equipped and only then. When the amount of Haste/Mastery/Spirit on your gear changes it's contribution isn't updated until it's re-equipped.
The workaround is just to re-equip the trinket every time your raw Haste/Mastery/Spirit stat amounts change due to your gear.
FYI!
-Brusalk