  Appraisal theories explain how the cognitive evaluation of an event leads to
a particular emotion. In contrast to theories of basic emotions or affect
(valence/arousal), this theory has not received a lot of attention in natural
language processing. Yet, in psychology it has been proven powerful: Smith and
Ellsworth (1985) showed that the appraisal dimensions attention, certainty,
anticipated effort, pleasantness, responsibility/control and situational
control discriminate between (at least) 15 emotion classes. We study different
annotation strategies for these dimensions, based on the event-focused enISEAR
corpus (Troiano et al., 2019). We analyze two manual annotation settings: (1)
showing the text to annotate while masking the experienced emotion label; (2)
revealing the emotion associated with the text. Setting 2 enables the
annotators to develop a more realistic intuition of the described event, while
Setting 1 is a more standard annotation procedure, purely relying on text. We
evaluate these strategies in two ways: by measuring inter-annotator agreement
and by fine-tuning RoBERTa to predict appraisal variables. Our results show
that knowledge of the emotion increases annotators' reliability. Further, we
evaluate a purely automatic rule-based labeling strategy (inferring appraisal
from annotated emotion classes). Training on automatically assigned labels
leads to a competitive performance of our classifier, even when tested on
manual annotations. This is an indicator that it might be possible to
automatically create appraisal corpora for every domain for which emotion
corpora already exist.
