No, this is not the entirety of the problem, but it's a necessary precondition of covering what the appropriate response to "Nazis" is.
The resistance to even bothering to define what's meant by "Nazi" points out why there's significant resistance to the idea that "punch-a-Nazi" is appropriate. While I might condemn the violence in some abstract sense, I don't actually care if someone assault punches an honest-to-god, gas-the-kikes, swastika-tattoo having Neo-Nazi. The core of the issue is that giving moral permission to do so immediately provides an incentive for people to declare their political opponents "Nazis" and to start feeling morally licensed to attack them physically.