I personally wouldn't. Most people won't take it wrong, but maybe some would. For me, an article of clothing isn't worth potentially offending a soldier over. I'm not saying this is likely, but it's likely enough that I'd feel odd about wearing it.
The only times I get steamed is when posers are actually in military regalia, and BAD ones at that. You'd think if they go through all this trouble to present themselves they would get simple things right.