Not only is it not cowardly to walk away, its also the most intelligent option as it:
1. In the short term, saves yourself from injury.
2. Gives you the opportunity to remember their faces, catch one of them when they are alone and not so "tough" and break their f**ing legs.