G o E motions: A Dataset of Fine-Grained Emotions (2020)

Abstract: Understanding emotion expressed in language has a wide range of applications, from building empathetic chatbots to detecting harmful online behavior. Advancement in this area can be improved using large-scale datasets with a fine-grained typology, adaptable to multiple downstream tasks. We introduce GoEmotions, the largest manually annotated dataset of 58k English Reddit comments, labeled for 27 emotion categories or Neutral. We demonstrate the high quality of the annotations via Principal Preserved Component Analysis. We conduct transfer learning experiments with existing emotion benchmarks to show that our dataset generalizes well to other domains and different emotion taxonomies. Our BERT-based model achieves an average F1-score of .46 across our proposed taxonomy, leaving much room for improvement.

## introduction emotion expression and detection are central to the human experience and social interaction. with as many as a handful of words we are able to express a wide variety of subtle and complex emotions, and it has thus been a long-term goal to enable machines to understand affect and emotion @xcite . in the past decade, nlp researchers made available several datasets for language-based emotion classification for a variety of domains and applications, including for news headlines @xcite , tweets @xcite , and narrative sequences @xcite , to name just a few. however, existing available datasets are (1) mostly small, containing up to several thousand instances, and (2) cover a limited emotion taxonomy, with coarse classification into ekman @xcite or plutchik @xcite emotions. recently, @xcite have aggregated 14 popular emotion classification corpora under a unified framework that allows direct comparison of the existing resources. importantly, their analysis suggests annotation quality gaps in the largest manually annotated emotion classification dataset, crowdflower (2016), containing 40k tweets labeled for one of 13 emotions. while their work enables such comparative evaluations, it highlights the need for a large-scale, consistently labeled emotion dataset over a fine-grained taxonomy, with demonstrated high-quality annotations. to this end, we compiled goemotions, the largest human annotated dataset of 58k carefully selected reddit comments, labeled for 27 emotion categories or neutral, with comments extracted from popular english subreddits. we include a thorough analysis of the annotated data and the quality of the annotations. via principal preserved component analysis @xcite , we show a strong support for reliable dissociation among all 27 emotion categories, indicating the suitability of our annotations for building an emotion classification model. we perform hierarchical clustering on the emotion judgments, finding that emotions related in intensity cluster together closely and that the toplevel clusters correspond to sentiment categories. these relations among emotions allow for their potential grouping into higher-level categories, if desired for a downstream task. we provide a strong baseline for modeling finegrained emotion classification over goemotions. by fine-tuning a bert-base model @xcite , we achieve an average f1-score of .46 over our taxonomy, .64 over an ekman-style grouping into six coarse categories and .69 over a sentiment grouping. these results leave much room for improvement, showcasing this task is not yet fully addressed by current state-of-the-art nlu models. we conduct transfer learning experiments with existing emotion benchmarks to show that our data can generalize to different taxonomies and domains, such as tweets and personal narratives. our experiments demonstrate that given limited resources to label additional emotion classification data for specialized domains, our data can provide baseline emotion understanding and contribute to increasing model accuracy for the target domain. ## goemotions our dataset is composed of 58k reddit comments, labeled for one or more of 27 emotion(s) or neutral. ## data analysis figure 1 shows the distribution of emotion labels. we can see a large disparity in terms of emotion frequencies (e.g. admiration is 30 times more frequent than grief ), despite our emotion and sentiment balancing steps taken during data selection. this is expected given the disparate frequencies of emotions in natural human expression. ## modeling we present a strong baseline emotion prediction model for goemotions. ## transfer learning experiments we conduct transfer learning experiments on existing emotion benchmarks, in order to show our data generalizes across domains and taxonomies. the goal is to demonstrate that given little labeled data in a target domain, one can utilize goemotions as baseline emotion understanding data. emoint @xcite is part of the semeval 2018 benchmark, and it contains crowdsourced annotations for 7k tweets. the labels are intensity annotations for anger, joy, sadness, and fear. we obtain binary annotations for these emotions by using .5 as the cutoff. emotion-stimulus @xcite contains annotations for 2.4k sentences generated based on framenet's emotion-directed frames. their taxonomy is anger, disgust, fear, joy, sadness, shame and surprise. ## conclusion we present goemotions, a large, manually annotated, carefully curated dataset for fine-grained emotion prediction. we provide a detailed data analysis, demonstrating the reliability of the annotations for the full taxonomy. we show the generalizability of the data across domains and taxonomies via transfer learning experiments. we build a strong baseline by fine-tuning a bert model, however, the results suggest much room for future improvement. future work can explore the cross-cultural robustness of emotion ratings, and extend the taxonomy to other languages and domains. data disclaimer: we are aware that the dataset contains biases and is not representative of global diversity. we are aware that the dataset contains potentially problematic content. potential biases in the data include: inherent biases in reddit and user base biases, the offensive/vulgar word lists used for data filtering, inherent or unconscious bias in assessment of offensive identity labels, annotators were all native english speakers from india. all these likely affect labeling, precision, and recall for a trained model. the emotion pilot model used for sentiment labeling, was trained on examples reviewed by the research team. anyone using this dataset should be aware of these limitations of the dataset.