Towards a new Benchmark for Emotion Detection in NLP : A Unifying Framework of Recent Corpora (2024)

Abstract: Emotion recognition in text is a complex and evolving field that has garnered considerable interest. This paper addresses the pressing need to explore and experiment with new corpora annotated with emotions. We identified several corpora presented since 2018. We restricted this study to English single-labeled data. Nevertheless, the datasets vary in source, domain, topic, emotion types, and distributions. As a basis for benchmarking, we conducted emotion detection experiments by fine-tuning a pretrained model and compared our outcomes with results from the original publications. More importantly, in our efforts to combine existing resources, we created a unified corpus from these diverse datasets and evaluated the impact of training on that corpus versus on the training set for each corpus. Our approach aims to streamline research by offering a unified platform for emotion detection to aid comparisons and benchmarking, addressing a significant gap in the current landscape. Additionally, we present a discussion of related practices and challenges. Our code and dataset information are available at https://github.com/a-koufakou/EmoDetect-Unify. We hope this will enable the NLP community to leverage this unified framework towards a new benchmark in emotion detection.

## introduction detecting emotions in language, such as anger, joy, or sadness, is a powerful application of natural language processing (nlp) with significant interest, especially in recent years @xcite @xcite @xcite . emotion detection is sometimes confused with sentiment analysis, a much simpler task that focuses on detecting polarity of sentiments or opinions @xcite . auto-mated emotion detection is considerably more nuanced and complex due to the subjective and intricate nature of emotions. nlp-based emotion detection uses datasets annotated with emotions. there is great variability in emotion annotation, including differences in annotation levels (e.g., basic vs. detailed) and labeling schemes (e.g., single vs. multi-label). an even more important challenge is which emotions to use in order to annotate data. various emotion taxonomies or theories have been presented. @xcite provided 6 basic emotions: anger, disgust, fear, joy, sadness, surprise. @xcite proposed a wheel of 8 emotions, adding trust and anticipation to ekman's, also presenting dyads (feelings composed of two emotions). @xcite identified 6 basic emotions: love, joy, anger, fear, sadness, surprise, on which they also provided secondary and tertiary levels in a tree-like structure, later refined in @xcite . the appraisal theory @xcite linked emotions to a persons interpretation of a situation or event. recently, @xcite identified 27 distinct categories based on videos, facial expressions etc., revised by @xcite for text-based emotion recognition. despite the variety of theories available, many efforts have concentrated on single-labeled corpora with a limited set of basic emotions such as @xcite , likely because these are typically easier for nlp models to handle. nevertheless, the presence of multiple theories allows different approaches to emotion annotation, making it complicated to unify different datasets for comparisons and benchmarking. in recent years, numerous emotion-annotated corpora have been introduced from diverse sources and domains, such as social media posts given specific tags or essays on specific topics. any such available corpora are found in separate repositories and articles, making it challenging for re-searchers to be aware of all available resources in order to fully investigate their use. it is noteworthy that a few corpora are well-known, e.g., goemotions @xcite or tweeteval @xcite with more than 700 citations each, in contrast to others such as @xcite with around 20 citations. 1 as a result, many studies have relied on a subset of available resources, and there has been limited work towards benchmarking. the work by oberländer and klinger (2018) stands out: they analyzed and aggregated 14 popular emotionannotated corpora into a unified framework, in 2018. they used their unified corpus for benchmark results with in-corpus and cross-corpus experiments. the unified framework available online 2 facilitated comparisons of the different corpora. recent surveys @xcite @xcite @xcite do not cover many of the corpora we present in this paper. very recently, an excellent recent paper by plaza-del arco et al. (2024) reviewed over 150 acl papers @xcite @xcite @xcite @xcite @xcite , and offered a detailed overview of practices, gaps, and guidelines for emotion analysis in text. still, their paper did not provide a unified framework or experimentation results. to address this gap, our paper introduces a unifying framework of text corpora annotated with emotions, as presented in the literature since 2018. we chose 2018 because: (a) it was the most recent year when a unifying framework was presented @xcite , and (b) it marked a notable increase in the number of related studies @xcite . specifically, we identified 11 publicly available emotion-annotated text corpora: we focused our experimentation to english and single-labeled data. 3 while this may seem limited, it serves as a good representation for a significant portion of existing datasets in this area @xcite . more importantly, our primary aim is to explore how to unify (combine) various datasets annotated with different emotions into a single framework, which is not as straightforward. we conducted classification experiments with these datasets, while comparing our results with the results reported in the original articles. based on these corpora, we introduce a unified corpus built by mapping original emotions in the corpora to a common set of emotions. finally, we present baseline benchmarking results for emotion classification with our unified corpus. the ultimate goal is to aid researchers in the field of text-based emotion recognition by providing a unified resource built on a comprehensive set of recent data, which they can access in one repository. our secondary goal is to furnish a classification baseline benchmark with valuable insights they can use while conducting their own experiments. the following sections provide descriptions of the corpora (section 2), details of the unified corpus we created (section 3), results of our emotion classification experiments (section 4), and a discussion of findings and observations (section 5), followed by our concluding remarks and future research directions (section 6). ## corpora carer @xcite collected tweets with a set of hashtags they constructed, e.g. #depressed, #grief for sadness, or #fear, #worried for fear. these hashtags were used to annotate the data (distant supervision). the dataset posted on hugging face is labeled with shaver, and it is a variant of the dataset presented in the article. covid-worry this dataset contains survey responses collected in uk over 2020-22, starting with the first covid-19 lockdown @xcite . participants wrote short and long texts, along with demographic data and selfratings for several emotions. they also chose one emotion among anger, anxiety, disgust, desire, fear, happiness, relaxation, sadness. in 2023, the authors presented a 3-year dataset (van der vegt and @xcite . emoevent plaza-del-arco et al. (2020) collected tweets related to events in 2019, and then followed certain steps to select a subset of affective tweets. the resulting tweets in english and dataset source # emotions size reference avail. carer tweets 6 417* (saravia et al., 2018) hg covid-worry essays 8 5.2 (van der vegt and kleinberg, 2023) g,o emoevent tweets 6+1 7.3 (plaza-del-arco et al., 2020) g enisear self-written 8 1 (troiano et al., 2019) o github-love github 6 1.7 (imran et al., 2022) hg goemotions reddit 27+1, 6+1 58* (demszky et al., 2020) g goodnews headlines 15+1 5 (oberländer et al., 2020) uni stackov-gs stack overflow 6 4.8 (novielli et al., 2018) g tweeteval tweets 4 5 (barbieri et al., 2020) g universal joy facebook 5 284* (lamprinidis et al., 2021) g wassa-21 essays 6+1 2.6 (tafreshi et al., 2021) cd, rq github-love imran et al. ( 2022 ) collected github comments on pull requests/issues for popular repositories, annotated by the authors using shaver. besides these basic emotions, they also used detailed levels of emotions @xcite , where they added some of the emotions presented by @xcite , e.g. approval or confusion. note that the dataset available online has basic emotion labels, not the detailed levels in the paper. goemotions demszky et al. ( 2020 ) collected reddit comments with crowdsourced annotations for 27 emotions or neutral, revised from @xcite . they also provided an ekman mapping from their detailed emotions. this dataset is multi-labeled and we transformed it into single-labeled for the purposes of this study (see section 3 for details). goodnews oberländer et al. ( 2020 ) collected english news headlines and annotated them via crowdsourcing (named goodnewseveryone in the original article). annotations were provided for emotions (extended plutchik) and their intensity, as well as semantic roles (such as experiencer or cause), and reader interpretation of the headline. stackov- @xcite collected stack overflow questions, answers and comments for the 'stackoverflow gold standard'. they were annotated by volunteers with shaver. tweeteval @xcite created a unified twitter dataset with seven heterogeneous twitter-specific classification tasks. among those, they included affect in tweets @xcite only keeping single-label records and dropping rare emotions. this resulted in records labeled with anger, joy, optimism, sadness. universal joy @xcite presented a dataset with anonymized public facebook posts that were originally collected in 2014 in 18 languages. the authors labeled the records with anger, anticipation, fear, joy, sadness. wassa-21 this dataset was part of a shared task in the 11th workshop on computational approaches to subjectivity, sentiment & social detection and emotion classification (wassa), summarized by @xcite . it contains essays written to express the authors empathy and distress in reaction to news articles related to harm. the emotion labels @xcite were first predicted by neural networks and then post-annotated by crowdsourcing workers and a phd student. as far as size, most corpora are small; the two smallest are enisear and github-love. there are 3 larger datasets: carer and universal joy (hundreds of thousands) and goemotions (about 58k). finally, all corpora follow basic emotion annotation, except goemotions and goodnews. based on finally, the distribution of emotions varies across the corpora. some corpora exhibit a range of dominant emotions versus very low representation of certain emotions. for instance, carer is primarily dominated by joy, followed by sadness, and it has a very low sample of surprise. universal joy is heavily dominated with anticipation and then joy, while low on anger and fear. covidworry is led by anxiety and fear, with joy trailing behind, and it has a very low number of records with disgust or anger. stackov-gs is led by love, followed by anger. emoevent is predominantly neutral ('other'), followed by joy. both emo-event and stackov-gs are very low in fear and surprise. finally, disgust has very low representation in most datasets. as an exception, enisear is balanced as crowdsourcing workers were asked to write a certain number of statements for each of the emotions. ## creating a unified corpus first, for any corpus we downloaded, we spent effort reading instructions and exploring file formats, features, labeling schemes, etc. for example, some data had integers as labels, which we had to map to emotions per author instructions; some data came with many features so we had to extract text/labels. many sets were well-organized and documented, with a couple of exceptions that were harder to understand and transform. in short, we spent significant effort to integrate diverse corpora into the unified corpus with the goal to save other researchers time and effort. based on the emotions in we mapped original emotions from the data to the emotions in the common set as shown in ## discussion in this paper, we started by selecting 11 recently introduced datasets with emotion-annotated records in order to introduce a new unified framework for benchmarking emotion detection in nlp. we described the characteristics of these datasets, which vary in size, topic, source, emotions and distributions. nevertheless, one could question the selection of the specific datasets. it would be benefi-cial to explore earlier datasets or datasets we did not include, and how they compare/connect to corpora in this work: we leave this for future work. moreover, we focused on english corpora, unfortunately, a common limitation in nlp: a multilingual study is thus needed, e.g. see a multilingual sentiment analysis study by @xcite . specifically for emotion detection, one should consider linguistic and cultural differences for emotions @xcite . we combined the datasets into one unified framework by mapping to a common set of emotions. we followed a simple emotion scheme for this: each record gets assigned a single label out of n emotions, similar to earlier work @xcite . many of the available corpora only have a few basic emotions to start with @xcite . in reality, though, emotions are complex and an individual's writings may encapsulate multiple emotions. this is even more prevalent in essays with multiple sentences @xcite . to more accurately reflect human emotions, finegrained emotion annotations are often preferred over coarse ones @xcite . our review of the datasets and related literature revealed several limitations and issues similar to those identified in previous work, such as inconsistencies in annotation practices or inadequate reporting of the annotation process @xcite . these issues underscore the need for a more thorough analysis of practices and benchmarking within available corpora. good examples in other areas include a systematic review in hate speech detection by @xcite . also, we found research in generalized offensive language identification by @xcite : they used relatively basic labels, e.g. offensive or non-offensive. in contrast, as we have shown in this paper, emotion-annotated data involves more complexity and thus additional challenges. additionally, automated emotion recognition carries ethical considerations as shown by @xcite , who proposed an ethics sheet outlining 50 ethical considerations. for instance, the need to account for both the speaker's and the reader's perspectives, which can vary significantly from one individual to another. while this paper represents considerable effort, there is still more work to match the underlying complexity of this study. our classification experiments offer a baseline rather than a complete benchmark. our published code and list of dataset links will enable anyone to recreate and further utilize the unified , and even possibly extend it. the research community is welcome to use these resources and employ various models and/or explore the effect of different hyper-parameters on the results. ## conclusions this paper answers the imperative need for studying recent text-based corpora annotated for emotion detection. our investigation into diverse corpora sourced from various domains and introduced since 2018 summarizes and gives insights into their characteristics, such as source, topic, size, emotion and distributions, etc. furthermore, we constructed a unified framework built from these corpora by mapping their emotions to a common set of labels. we used these resources to conduct emotion detection experiments, and compared the effect of fine-tuning a pretrained model to the train set of each corpus versus to the unified train set. this consolidated platform will be a valuable resource for researchers, streamlining efforts and providing the basis for a practical emotion classification benchmark. while this paper represents considerable effort, there is still more work to match the underlying complexity of this study. future directions include expanding this work to additional datasets, including multi-lingual and multi-label settings, while also conducting additional experiments (e.g. crosscorpus or various classifiers) and delving deeper into annotation practices and methodologies.