Question: are you upset with me or at me?
Answer: The key is who the anger is aimed at. "Angry at" is not really aimed at anyone. So in the example you give, "Are you angry with me?" is correct.