You're confusing the macro system.
Code:
/cast [target=mouseover]Righteous Defense
This is correct, but you should make it
Code:
[target=mouseover,exists,nodead]
to prevent getting error messages in the heat of combat (usually not an issue in the current game but it never hurts unless you're hitting the character limit ceiling).
Code:
/cast [target=mouseover][mod:shift] Hand of Reckoning
Here is the problem. This part of the macro tells WoW the following:
1. If my mouse cursor is over a valid target, cast Hand of Reckoning.
2. If it isn't, then check if I'm holding Shift down; if so, cast Hand of Reckoning anyway. Also, it's "mod:shift" not "mod=shift". The equal sign is only used for target instructions.
What you want is:
Code:
/cast [target=mouseover,mod:shift] Hand of Reckoning
Which tells WoW:
1. If my mouse cursor is over a valid target, and I'm holding down Shift, cast Hand of Reckoning.
But now you have a problem: your mouse cursor is over a valid target in both cases. However, Righteous Defense is only checking if your target has mouseover in order to go ahead and trigger. That means it will still trigger, even if shift is held down.
Solve this by changing the first line to:
Code:
[nomod:shift,target=mouseover,exists,noharm,nodead]
Which tells WoW to only use RD if Shift isn't down, and only use HoR if Shift is down.
The final macro would look like (assuming error catchers):
Code:
/cast [nomod:shift,target=mouseover,exists,nodead]Righteous Defense;[mod:shift,target=mouseover,exists,nodead] Hand of Reckoning
... Because you can sequence multiple branches into one instruction (eg, one "/cast") as long as you separate them with a semicolon.