Tab targeting has always been pretty badly designed. I'm a "tabber" myself and I've always struggled with it, but lately it's beyond all reason. They keybind literally says "target nearest enemy", but when you press it you sometimes end up targeting something 20 yards behind you when you're getting hit in the face by a bunch of enemies. Clearly not doing what it says it should be doing.
There are some commands you can fiddle around with and see if you can get it working somewhat to your liking:
Code:
/TargetNearestUseOld 0-1
0=new style (default), 1=old style
/TargetPriorityAllowAnyOnScreen 0-2
0=no, 1=if no 100% correct target is available, allow selecting any valid in-range target (default), 2=also out of range
/TargetPriorityIncludeBehind 0-1
0=no, 1=yes (default)
There are also these commands you can try out, I haven't noticed any difference with them though, at least not that sorts the problem.
Code:
/TargetPriorityCombatLock 0-2
0=off, 1=lock target to in-combat tagets (default), 2=restrict to in-combat with player
/TargetPriorityValueBank x
set a scoring valuye bank for calculating target priority
Another tip is to try out the new "combat scan" feature that came with a recent update. If you hold tab, you'll see a targeting circle under the target that you're facing with your camera. Once you release tab you'll select that target. I think you need to set /TargetNearestUseOld to 0 for it to work.
That works fine when you're at range and got time to choose your target, but when you're inside a snakepit of mobs running around all over you ain't got time for that!
As for macros selecting targets based on current HP: that's not a functionality thats possible as far as I know.