Systematic Evaluation of a Framework for Unsupervised Emotion Recognition for Narrative Text (2020)

Abstract: Identifying emotions as expressed in text (a.k.a. text emotion recognition) has received a lot of attention over the past decade. Narratives often involve a great deal of emotional expression, and so emotion recognition on narrative text is of great interest to computational approaches to narrative understanding. Prior work by Kim et al. 2010 was the work with the highest reported emotion detection performance, on a corpus of fairy tales texts. Close inspection of that work, however, revealed significant reproducibility problems, and we were unable to reimplement Kim’s approach as described. As a consequence, we implemented a framework inspired by Kim’s approach, where we carefully evaluated the major design choices. We identify the highest-performing combination, which outperforms Kim’s reported performance by 7.6 F_1 points on average. Close inspection of the annotated data revealed numerous missing and incorrect emotion terms in the relevant lexicon, WordNetAffect (WNA; Strapparava and Valitutti, 2004), which allowed us to augment it in a useful way. More generally, this showed that numerous clearly emotive words and phrases are missing from WNA, which suggests that effort invested in augmenting or refining emotion ontologies could be useful for improving the performance of emotion recognition systems. We release our code and data to definitely enable future reproducibility of this work.

## introduction emotion is a primary aspect of communication, and can be transmitted across many modalities including gesture, facial expressions, speech, and text. because of this importance, automatic emotion recognition is useful for many applications, including for automated narrative understanding. a narrative is "a representation of connected events and characters that has an identifiable structure, is bounded in space and time, and contains implicit or explicit messages about the topic being addressed" @xcite , and narratives are often used to express the emotions of authors and characters, as well as induce emotions in audiences. for many narratives-one need only consider romances such as romeo and juliet or the movie titanic-it is no exaggeration to say that lacking an understanding of emotion leads to a seriously impoverished view of the meaning of the narrative. emotion recognition is a challenging problem on account of the complex relationship between felt emotion and linguistic expression. this includes not only standard natural language processing challenges, such as polysemous words and the difficulty of coreference resolution @xcite , but also emotion-specific challenges such as how context can subtly change emotional interpretations @xcite . these technical challenges are exacerbated by a shortage of quality labeled data addressing this task. there has been much prior work on emotion recognition. with regard to narrative specifically, @xcite reported a high-performing approach to emotion recognition on a corpus of fairy tales texts @xcite . this approach involved an unsupervised learning framework for emotion recognition in textual data, using a modified form of ekman's psychological theory of emotion (joy, anger, fear, sadness; @xcite . in that work, they used the wordnetaffect (wna) and anew (affective norm for english words) emotion lexicons to construct a semantic space. each sentence is placed in the space using tf-idf weights for emotion words found in the lexicons. they then tested three methods-non-negative matrix factorization (nmf), latent semantic analysis (lsa), probablistic latent semantic analysis (plsa)for compressing the space to extract features of the constructed vector space model, reduce noise, and eliminate outliers. finally, the framework used cosine-similarity to label sentences by evaluating how similar they are compared to standard vectors generated based on wna entries strongly associated with emotion lexicon (more specifically an extension of wna). the best performing method was nmf, which they reported achieved an average emotion recognition f 1 of 0.733. close inspection of the work, however, revealed significant reproducibility problems. despite our best efforts we were unable to reproduce results anywhere near kim's reported performance; indeed, our best attempt yielded only roughly 0.25 f 1 . this was due to several reasons. first, the paper lacked information on model hyper-parameters. second, the paper omitted descriptions of key nmf steps, including how to identify representative features and what features should be removed before semantic space compression. third, the paper did not explain how to adapt nmf to deal with the sparse matrices that occur in textual nmf models. fourth, certain resources associated with wna either were not correctly identified, or are no longer available. these omissions prevented us from reproducing their models to any degree of accuracy. therefore, we undertook to do a systematic exploration of the design space described in @xcite . we examined the highest performing vector space compression techniques reported by kim et al. (nmf), as well as principle component analysis (pca) and latent dirchelet allocation (lda) which were reported as high-performing techniques in other work. we show that nmf indeed performs the best, and we clearly explain our experimental setup including methods for identifying relevant features and handling sparse text matrices. the pca and nmf methods implemented in this paper are based on the works of @xcite and @xcite respectively which have implemented mechanisms that works for a large sparse matrix (in our case, 1, 090 × 2, 405). this work resulted in an improvement of performance of roughly 7.6 points of f 1 over kim's reported results. we release our code and data to facilitate future work foot_0 . the rest of this paper is structured as follows. we briefly review psychological models of emotions, describe several key emotion language resources, and outline a number of well-known emotion recog-nition models ( §2). we then describe our adapted unsupervised emotion recognition method, giving detailed descriptions of all steps, parameters, and resources needed ( §3). we next describe the performance of our method on alm's corpus of fairy tales @xcite , which was annotated for emotion on a per-sentence level ( §4). finally, we identify some unsolved challenges that point toward future work ( §5), and summarize our contributions ( §6). ## emotion recognition framework we now describe an unsupervised system for emotion recognition modeled on that reported by kim citation corpus lexicon # emotions method f1 kim et al. (2010) fairy tales wna 4 nmf 0.73 bandhakavi et al. (2017) tweets umm+dsel 6 lexicon only 0.64 aman and szpakowicz (2008) blog -6 unigrams 0.57 cherry et al. (2012) suicide notes -15 svm+ls 0.55 strapparava and mihalcea (2008) headlines -6 lsa 0.17 tokuhisa et al. (2008) "ep" events jee dict. 10 svm+knn 0.5-0.8 acc. augmenting wna as mentioned before, wna 1.1 assigns an emotion label to 1,471 synonym sets (synsets) of wordnet. this corresponds to a lexicon of nearly 3,495 affective lemma-pos pairs. careful inspection of wna revealed both incor- ## unsolved challenges and future work as already discussed, one challenge regarding automatic emotion recognition is the context dependency of emotional semantics. for instance, i'm over the moon! is an expression of extreme happiness but does not use any explicitly happy or joyful words (or, indeed, any emotion word at all). another obstacle is polysemous words, when words have both an emotional and non-emotional senses; recognizing which sense of the word is being used is challenging and remains an open problem. aside from these fundamental issues, there is a serious lack of high-quality annotated data, not just for narrative text but for all discourse types. annotated corpora use a wide variety of sometimes incompatible emotion theories and are often poorly annotated, with low inter-annotator agreements and many errors. given these considerations, there are many possible directions for future work, for example: ## contributions we identified a high performing approach to emotion recognition in narrative text @xcite and carefully reimplemented and characterized the technique, exploring a design space of three different noise cancellation or dimension reduction techniques (nmf, pca, or lda), exploring various hyper-parameter settings. our experiments indicated that nmf performed best, with an overall f 1 of 0.809. in the course of our investigation we clarified numerous implementational issues of the work reported by @xcite , as well as made some improvements to wordnet affect (wna), one of the language resources used in the system, by adding new terms manually and using wordnet similarity relations. this work suggests several promising future directions for improving the work, including careful annotation of a larger corpus, and augmenting wna or similar lexicons to provide improved coverage of emotion terms. we release our code and data to enable future work foot_2 .