MASIVE : Open-Ended Affective State Identification in E nglish and S panish (2024)

Abstract: In the field of emotion analysis, much NLP research focuses on identifying a limited number of discrete emotion categories, often applied across languages. These basic sets, however, are rarely designed with textual data in mind, and culture, language, and dialect can influence how particular emotions are interpreted. In this work, we broaden our scope to a practically unbounded set of affective states, which includes any terms that humans use to describe their experiences of feeling. We collect and publish MASIVE, a dataset of Reddit posts in English and Spanish containing over 1,000 unique affective states each. We then define the new problem of affective state identification for language generation models framed as a masked span prediction task. On this task, we find that smaller finetuned multilingual models outperform much larger LLMs, even on region-specific Spanish affective states. Additionally, we show that pretraining on MASIVE improves model performance on existing emotion benchmarks. Finally, through machine translation experiments, we find that native speaker-written data is vital to good performance on this task.

## introduction in the field of emotion analysis, much nlp research focuses on identifying a limited number of discrete emotion categories, typically using basic emotion sets from the field of psychology @xcite . these basic emotion sets are rarely designed with textual expression in mind (e.g., @xcite , whose model defines basic emotions by the recognizability of facial expressions), and very little research examines the validity of adapting these sets to textual data. emotion analysis furthermore relies on largely the same emotion categories across languages, including, in some cases, translating resources such as lexicons, fine-tuning data, or evaluation data "...i'm glad we spoke. now i @xmath0, and i know now that though it may be hard, we can have those conversations and that type of relationship..." ## affective state identification (asi) in contrast to traditional emotion detection, we propose a novel task, asi. our goal is to capture the broad set of ways in which humans describe their own feelings in text. we refer to these expressions as affective states @xcite ; this is an umbrella term incorporating multiple kinds of feelings such as emotions and moods. we highlight multiple implications of models that are capable of accurately inferring affective states. first, asi enables the identification of more nuanced feelings derived from textual expressions of affect (e.g., distinguishing despair and grief which may be similarly described by basic emotions). additionally, asi-capable models can be adapted to multiple theories of emotion. whereas a model specifically trained on one set of emotions (e.g., ekman) must be fine-tuned for each label set (e.g., 27 emotions of @xcite , asi models can be restricted to an arbitrary subset of affective state labels. finally, asi is grounded in expressions of feelings by the author in contrast to perceived emotion labels generated by annotators. these perceived emotion labels may train models to encode spurious factors affecting human emotion perception, such as cultural differences. ## conclusion in this work, we introduce the novel task of asi, a language generation task prioritizing the authors' natural expressions of their feelings rather than using a prescribed set of emotion labels. for this task, we automatically collect and publish two datasets of reddit posts in english and spanish, both containing over 1,000 unique affective state labels. we use this dataset to benchmark multilingual generative models, and find that (takeaway #1) small fine-tuned t5 and mt5 models outperform zero-shot llms. results specifically show that (takeaway #2) t5 outperforms mt5 in english on asi, suggesting that monolingual models may be more capable. additionally, we show that (takeaway #3) models fine-tuned on our corpora transfer knowledge that generalizes to existing emotion detection benchmarks. in analyzing model performance on unseen affective states and spanish regionalisms, we argue that (takeaway #4) generalization to a broader set of affective states, including those from underrepresented dialects, is an important avenue for future work. with respect to grammatical gender in spanish and negations, (takeaway #5) fine-tuning on masive improves on specific linguistic constructions unique to generative asi. finally, we quantify the observed performance differences when using machine-translated data at fine-tuning or inference time, finding that in contrast to prior work, (takeaway #6) machine translation leads to large performance drops. we hope these results spark future work into asi to enable prediction of more nuanced feelings in a variety of languages and contexts, and ultimately, enable prediction of an unbounded set of labels.