Outside of the Charlottesville have we really seen the neo nazi's of today committing violence? Bearing in mind at that event many left wingers were inciting violence as well. If it was one thing where these people were routinely assaulting others and the victims were merely acting in self defense, violence would be understood. Or if this was a time of war or something. But that's clearly not the case.
Resorting to violence in all reality is just going to strengthen their resolve because it will show to them that their "problems are legitimate". To beat that ideology you got to point out the flaws and engage in a meaningful discussion.