Well, the safest thing for you is to let the driver pass. That puts the most distance between the two of you in the least amount of time. It'll also keep him from engaging in more dangerous activities like passing in the oncoming lane or on the right hand shoulder.
Also, you're not the one breaking the law....he is.
And generally, it's just less stressful just letting that maniac pass you and continuing on with your day than to have him behind you honking their horn, tailgating you, looking for any opportunity to get past you, etc. And people drive better when they aren't stressed.
- - - Updated - - -
The laws may be different...but the general safe practice guideline is to always keep right unless you are passing. Some states and "sovereign citizens" would probably consider it "tyranny" to make such a thing a law though.