A quick update to the people who care what happend.
So I decided to leave and migrate to join some other guys I used to raid with. They fit me better so I'm happy i did it.
However the guild I left ended up dissbanding after I left (from what I've heard so far) and most people understod why I decided to change guild. But there was some people that couldn't handle it and just started raging and giving me guilt to come back.