Hell Hath No Fury? Correcting Bias in the NRC Emotion Lexicon (2021)

Abstract: There have been several attempts to create an accurate and thorough emotion lexicon in English, which identifies the emotional content of words. Of the several commonly used resources, the NRC emotion lexicon (Mohammad and Turney, 2013b) has received the most attention due to its availability, size, and its choice of Plutchik’s expressive 8-class emotion model. In this paper we identify a large number of troubling entries in the NRC lexicon, where words that should in most contexts be emotionally neutral, with no affect (e.g., ‘lesbian’, ‘stone’, ‘mountain’), are associated with emotional labels that are inaccurate, nonsensical, pejorative, or, at best, highly contingent and context-dependent (e.g., ‘lesbian’ labeled as Disgust and Sadness, ‘stone’ as Anger, or ‘mountain’ as Anticipation). We describe a procedure for semi-automatically correcting these problems in the NRC, which includes disambiguating POS categories and aligning NRC entries with other emotion lexicons to infer the accuracy of labels. We demonstrate via an experimental benchmark that the quality of the resources is thus improved. We release the revised resource and our code to enable other researchers to reproduce and build upon results.

## introduction emotion detection is an nlp task that has long been of interest to the field @xcite @xcite , and is usually conceived as a single-or multi-label classification in which zero (or more) emotion labels are assigned to variously defined semantic or syntactic subdivisions of the text. the importance of this task has only grown as the amount of available affective text has increased: social media, in particular, has made it especially convenient 1 https://doi.org/10.34703/gzx1-9v95/ po3ygx for people around the world to express their feelings and emotions regarding events large and small. there are generally two ways to express emotions in textual data @xcite . first, emotions can be expressed using emotive vocabulary: words directly referring to emotional states (surprise, sadness, joy). second, emotions can be expressed using affective vocabulary: words whose emotional content depends on the context, without direct reference to emotional states, for example, interjections @xcite . an emotion lexicon is a specific type of linguistic resource that maps the emotive or affective vocabulary of a language to a fixed set of emotion labels (e.g. plutchik's eight-emotion model), where each entry in the lexicon associates a word with zero or more emotion labels. because this information is difficult to find elsewhere, emotion lexicons are often used as one of the key components of affective text mining systems @xcite . however, as is usual with linguistic resources, creating an emotion lexicon is a time-consuming, costly, and sometimes impractical part of the task. the difficulty is only accentuated when one considers the many affective uses of words, in which the emotional content is context dependent. such context dependency underlines the utility of general-purpose (context-independent) emotion lexicons (gpels), which captures the mostly fixed emotive content of words, and which can serve as a foundation for more context-dependent systems. in this paper, we analyze and improve one of the most commonly used gpels, namely, the nrc lexicon (national research council of canada; also known as the emolex emotion lexicon @xcite . the nrc used macquarie's thesaurus @xcite as the source for terms, retaining only words that are repeated more than 120,000 times in google n-gram corpus @xcite . the nrc maps each word to zero or more labels drawn from plutchik's 8-emotion psychological model @xcite , and provides labels for 14,182 individual words. while the nrc has been used extensively across the emotion mining literature @xcite @xcite @xcite , close inspection reveals a large number of incorrect, non-sensical, pejorative, or otherwise troubling entries. while we provide more examples later in the paper, to give a flavor of the problem, the nrc provides emotion labels for many generic nouns (@xmath0, common verbs (@xmath1, colors (@xmath2, places (@xmath3, relations (@xmath4, and adverbs (@xmath5. furthermore, the nrc suffers from significant ambiguity because it does not include part of speech categories for the terms: for example, while console implies sadness in its most common verb sense (as the nrc indicates), in its most common noun sense means a small side table, which probably should have no emotive content. in our analysis, many of these problematic entries seem to stem from a conflation of emotive (context-independent) and affective (context-dependent) emotion language use: it is as if, during the annotation of shakespeare's macbeth, the annotators of the nrc marked @xmath6: while it is true that this statement is often cited to support an assertion that women are angry people in general, and such a lexicon entry would help in correct marking of the affective implication of this specific statement in this particular context, it does not generalize to all, or even most, uses of the word woman. therein lies the rub. we begin the paper with a brief review of psychological models of emotion, available emotion lexicons, and datasets of emotion labeled text ( §2). we then discuss in detail the deficiencies of the nrc, giving a variety of problematic examples, and speculating as to how these entries were included ( §3). next we describe a semi-automatic procedure designed to filter out many of these deficiencies ( §4), after which we evaluate the effectiveness of the filtering procedure by integrating the corrected version of the nrc into an emotion detection system ( §5). we conclude with a list of our contributions ( §6). 2 literature review ## problems with the nrc in our close inspection of the entries in the nrc, we noted three main problems. first, the nrc does not indicate the part of speech of terms labeled with emotion. this obviously causes a great deal of ambiguity as to whether a particular emotion label should apply to a particular use of a word form. second, the nrc contains numerous incorrect, inaccurate, nonsensical, or pejorative associations, most of which can be ascribed to an apparent conflation of the distinction between emotive and affective emotional language, i.e., ignoring the importance of context for emotional semantics. third, and finally, there are emotion markings in the lexicon for which we can find no support in keywordin-context (kwic) databases for any sense; we count these as simple errors. ## semi-automatic correction of the nrc the nrc includes 14,182 entries made up of a unigram (single token wordforms) associated with a term label term label term label arm neutral diversity neutral office neutral buy neutral endpoint neutral road neutral carrier neutral flat neutral weather neutral clothes neutral filter neutral yeast neutral ## evaluation of the corrected resource in order to compare and evaluate the outcome of the correction procedure, we ran the emotion detection model developed by @xcite using nrc.v1, nrc.v2, and nrc.v3 as the emotion lexicon. we chose this model because the code was helpfully provided in full, and the model uses a single emotion lexicon with wordform-pos pairs to drive its emotion detection. in this section, we discuss the details of this comparison. the emotion detection system of zad and finlayson originally used wna as the emotion lexicon (leveraging wordform+pos pairs), and tested on alm's fairy tale dataset @xcite . while the system is convenient as an experimental testbed because the full code is available, alm's dataset uses only six emotions (anger, fear, sadness, surprise, disgust, and joy), as opposed to plutchick's eight used by the nrc. this means we needed to trim our nrc versions down to six labels for compatibility (we dropped anticipation and trust). this makes the evaluation of the nrc using this experimental setup at best an approximation for the quality of our procedure. one would imagine that, if we had an experimental testbed that used all eight of plutchik's emotions, performance would be correspondingly higher. as described below, we also experimented with reducing the number of labels, following the experimental procedure outlined in @xcite . further, following the same procedure, we conducted our emotion detection comparisons on the subset of alm's dataset which represented "high agreement", namely, only sentences for which the annotators fully agreed with each other. ## contributions we noted three categories of error in the popular nrc emotion lexicon, including a large number of seemingly biased entries. we developed and applied a semi-automatic procedure to generate three different corrected version of the nrc, and showed via experiment that these new versions improved the performance of an existing emotion-lexiconbased emotion detection system. this work shows the utility of careful error checking of lexical resources, especially with attention to correcting for unintended biases. finally, we release the revised resource and our code to enable other researchers to reproduce and build upon results foot_2 .