If you increase the penalty in leaving, people will be more motivated to get out of the instance without leaving. Yes, the debuff is ineffective, but a harsher penalty definitely won't be a quality of life improvement for those who don't leave dungeons. There shouldn't be a penalty harsh enough to keep someone in a group they don't want to be in. The person who wants to leave gets a negative attitude and the group has to deal with it. Nobody wins.