If you want to get technical, if his ex-wife was court-ordered to turn their kids over for his custody time, and she was refusing (or unable; it amounts to the same thing here), then what was occurring in that moment was a kidnapping. So yeah; the father is in the legal right to push the issue against the people kidnapping his kids.
That's what refusing to turn children over for custodial rights is. It's kidnapping. Literally the legal definition.
You're probably right that the better angle would have been to back off, call the cops, and have both his ex-wife and her boyfriend arrested and charged with kidnapping. Would pretty much guarantee his ex-wife loses custody, too.
Regardless, the ex-boyfriend retreated to arm himself, rather than calling the police, so that makes the shooting first-degree murder. I could care less if you're gonna whine about a minor case of trespass at best when the father's kids are being kidnapped, it doesn't remotely justify the murder.