The more annoyed he got, the more unnerved I was.
I was getting fed up with the man.