Ahem.Yes when you enter a CRZ you enter a "neutral" zone of that which several different servers zone into, but if you're on a PvP server only people from other PvP servers will be able to zone into it. Someone from a PvE realm cannot zone into a PvP realm unless invited by someone on a PvP realm over real ID, and even then you are just a "guest" on their server and still have to live up to their rules.
So even if someone who's on a PvE realm invites someone from a PvP realm, that person will still not be able to gank you.
Make your mind up...