  When humans judge the affective content of texts, they also implicitly assess
the correctness of such judgment, that is, their confidence. We hypothesize
that people's (in)confidence that they performed well in an annotation task
leads to (dis)agreements among each other. If this is true, confidence may
serve as a diagnostic tool for systematic differences in annotations. To probe
our assumption, we conduct a study on a subset of the Corpus of Contemporary
American English, in which we ask raters to distinguish neutral sentences from
emotion-bearing ones, while scoring the confidence of their answers. Confidence
turns out to approximate inter-annotator disagreements. Further, we find that
confidence is correlated to emotion intensity: perceiving stronger affect in
text prompts annotators to more certain classification performances. This
insight is relevant for modelling studies of intensity, as it opens the
question wether automatic regressors or classifiers actually predict intensity,
or rather human's self-perceived confidence.
