What happens next?

How to stop your brother from annoying you
Ignore your brother rather than responding.
If your brother is being difficult, you may have to try temporarily ignoring him. This isn't a very effective strategy for long-term sibling relationships, but if you want to avoid lashing out in anger, the best response is to just ignore him.
Not responding is not a sign of weakness. If anything, it takes more strength and willpower to resist lashing out at your brother or sinking to his antics.