It must, in fact, have been distinctly annoying to the pair of schemers."
There were two schemers, and they must have been annoyed by something.