Uncovering the Limits of Text-based Emotion Detection (2021)

Abstract: Identifying emotions from text is crucial for a variety of real world tasks. We consider the two largest now-available corpora for emotion classification: GoEmotions, with 58k messages labelled by readers, and Vent, with 33M writer-labelled messages. We design a benchmark and evaluate several feature spaces and learning algorithms, including two simple yet novel models on top of BERT that outperform previous strong baselines on GoEmotions. Through an experiment with human participants, we also analyze the differences between how writers express emotions and how readers perceive them. Our results suggest that emotions expressed by writers are harder to identify than emotions that readers perceive. We share a public web interface for researchers to explore our models.

## introduction identifying emotional signals is key to a series of downstream tasks. for instance, emotion detection is necessary for empathetic chatbots that can respond to the emotional needs of their users @xcite . distinguishing emotionally charged content is required to study viral @xcite , educational @xcite , political @xcite , or incendiary (brassard-gourdeau and khoury, 2019) interactions on social media. capturing the evolution of user-provided emotional content can help preventing harassment @xcite or developing early indicators for depression @xcite . facial expressions @xcite , speech @xcite , body movements @xcite ) and text @xcite are sources from which emotions may be automatically extracted. emotion analysis contrasts sentiment analysis, which characterizes text in terms of polarity (positive, negative or neutral), by involving a larger set of classes, often influenced by aspects such as ambiguity, misunderstandings, irony or sarcasm @xcite . recent progress in the field has been enabled by the success of pre-trained language models, such as bidirectional encoder representations from transformers (bert) @xcite , and the release of high-quality large-scale annotated datasets. in this work, we analyze the limits of text-based emotion detection on the two largest now-available corpora: goemotions @xcite and vent @xcite . goemotions contains 58k reddit comments tagged with possibly multiple labels out 28 emotions, annotated by thirdperson readers. the raw vent corpus @xcite includes 33m messages tagged with one out of 705 emotions by their original firstperson writers. the unprecedented volume of these datasets makes them suitable to study textual emotion detection at scale from different perspectives. we first focus on the predictive performance of several feature spaces and learning algorithms. our benchmark includes simple models that outperform previous baselines, including a strong bert baseline on goemotions. we find that statistical methods such as tf-idf outperform more complex word-level embeddings such as fasttext. finally, we release a web interface showcasing our models. second, we analyze the hierarchical structure of the label space for models trained on vent. our models capture the cluster structure defined by emotion categories to a large extent, despite not explicitly observing emotion categories during training. finally, we design an experiment with human participants to evaluate our model and the differences between emotions provided by writers and those perceived by participants (readers). our models outperform readers at predicting emotions intended by writers, and they also predict the emotions annotated by readers even more accurately, with important implications for emotion analysis. the landscape of human emotion has been represented by several different taxonomies and approaches. ekman @xcite proposed 6 basic emotions expressed through facial expressions across cultures: joy, sadness, anger, surprise, disgust and fear. independently, plutchik @xcite introduced a similar taxonomy that added anticipation and trust, characterizing emotions through his wheel of emotion @xcite .finer-grained emotion taxonomies have been recently proposed, capturing high-dimensional relationships of up to 600 different emotions, clustering emotion concepts using machine learning techniques @xcite . these taxonomies show the multifaceted nature of emotions across cultures in vocalization @xcite , music @xcite , or facial expressions @xcite . beyond these discrete categorizations, other models such as the affective-circumplex model of emotions @xcite , have captured emotions as proportions on three dimensions rather than discrete categories: valence (positive or negative), activation (active or passive), and dominance (dominant or submissive). finally, a small number of works extend these taxonomies to include the perspective of senders and receivers of emotional communication @xcite @xcite . in this work, we focus on categorical approaches with recent emotional taxonomies covering a rich spectrum of emotions from the perspectives of senders and receivers. ## emotion detection in text a text snippet can have several associated emotions in cases of ambiguity, or when expressing multiple feelings at once, as seen in text emotion wow. i just read the synopsis, and thats really what happens. surprise what do you think? if you look at my question above? last thing i should do is to say sorry? confusion and then everyone clapped and cheered. ## experimental design here we briefly describe the goemotions and vent datasets, our multi-label classification benchmark design, and the representation and modelling approaches that serve as the building blocks for our emotion detectors. we include extensive details on the experimental design, hyper-parameters, additional results, and data analyzes in the supplemental materials. ## experimental results in this section, we study the results of our benchmark, evaluate the behaviour of the trained models, and analyze the differences between writer and reader emotions on vent. ## conclusions we present a principled analysis of emotion detection techniques on the two largest available datasets to date: goemotions and vent. our thorough benchmark shows how different models behave, with bert-based architectures consistently achieving the best performance across datasets. we release emotionui, a web interface for researchers to explore our models 23 , and share our code, tools, annotations and experiment data foot_2 . on goemotions, our best performing model shows an improvement over the previous baseline by 11.8% relative micro-f1. on vent, we train a model that outperforms readers at predicting emotions provided by writers by 19.9% relative micro-f1. surprisingly, the same model shows better performance when evaluated against emotional labels provided by readers by 35.9% relative micro-f1 despite not being trained on this task. our findings show that models trained on vent outperform readers in the task of predicting writer-provided emotions, even though the model performs better on reader-provided emotions rather than the original ground truth labels provided by vent users (writers). these findings open new research directions on the automatic detection of emotions between readers and writers in discrete emotion taxonomies, and its implications for emotion detection systems. our work suggests that the task of predicting the emotions that writers aim to express is harder than detecting those perceived by readers. however, the majority of current annotated emotion detection studies focus on reader emotions.