Like so many have said, just stand up for yourself. You can only be bullied when you allow yourself to be. People will always persue the path of least resistance, as long has it isn't you anymore you won't have anything to stress about. Also make sure you contact the cops if the brother does step in, he may be stupid, but jail time comes across pretty clear to most people.