My Heart Skipped a Beat! Recognizing Expressions of Embodied Emotion in Natural Language (2024)

Abstract: Humans frequently experience emotions. When emotions arise, they affect not only our mental state but can also change our physical state. For example, we often open our eyes wide when we are surprised, or clap our hands when we feel excited. Physical manifestations of emotions are referred to as embodied emotion in the psychology literature. From an NLP perspective, recognizing descriptions of physical movements or physiological responses associated with emotions is a type of implicit emotion recognition. Our work introduces a new task of recognizing expressions of embodied emotion in natural language. We create a dataset of sentences that contains 7,300 body part mentions with human annotations for embodied emotion. We develop a classification model for this task and present two methods to acquire weakly labeled instances of embodied emotion by extracting emotional manner expressions and by prompting a language model. Our experiments show that the weakly labeled data can train an effective classification model without gold data, and can also improve performance when combined with gold data. Our dataset is publicly available at https://github.com/yyzhuang1991/Embodied-Emotions.

## introduction most people experience emotions every day. when emotions arise, we not only feel them mentally but we also experience them physically via our body. sometimes an emotion evokes a visible physical reaction. for instance, we may clench our fists or stomp our feet when we feel angry, or raise our hands in the air and dance when we feel happy. we may also have physiological responses when we experience an emotion. for example, we may feel our heart racing or feel a chill down our spine when we get scared. or we may feel our cheeks flush when we are embarrassed. in general, the john's was beet red when he realized he forgot his pants. ## related work in nlp, emotion recognition has been extensively studied. researchers have worked on creating emotion resources @xcite @xcite and analyzing emotion in texts across different genres @xcite @xcite . some research has focused on identifying implicit emotion/affect, including implicit sentiment analysis @xcite , good-for/bad-for events @xcite and affective event recognition @xcite @xcite . another relevant line of work is the study of non-verbal communication signals for expressing emotions (kim and klinger, 2019), such as physical appearance, facial expressions and movements of body as a whole. the non-verbal communication signals are a superset of embodied emotions, and kim and klinger (2019) only performed manual analyses and did not propose an automated task for recognizing these non-verbal signals. another similar line of work @xcite , inspired by the emotion component process model @xcite , focuses on recognizing emotion components that are often used for expressing emotions. casel et al. (2021) identifies five emotion components, including cognitive appraisal, neurophysiological symptoms, motivational action tendencies, motor expressions and subjective feelings. similarly, cortal et al. (2023) identifies four emotion components, including behavior, feeling, thinking and territory. this line of work differs from ours in two aspects. first, the emotion components are fundamentally different from embodied emotions. for example, @xcite includes all behaviors not evoked by emotion during an emotional event (e.g., giving a lecture) and @xcite includes goal-oriented physical movements (e.g., recover the stolen horse), while ours does not. in addition, their work focuses on teasing apart different emotion components from each other. in essence, it assumes the text to classify is emotional, while ours does not. researchers have found that figurative language is commonly used to express emotion @xcite , including metaphor @xcite , sarcasm @xcite and rhetorical questions @xcite . we observe that embodied emotion expressions are sometimes metaphorical. for example, the phrase "butterflies in my stomach" refers to a true physiological reaction to an emotion (so it is an embodied emotion) but the physical sensation is described metaphorically. however, many metaphorical expressions that mention a body part do not corre-spond to an actual physical response. for example, the phrase "my heart melted" is often used emotionally, but it does not indicate a real physical change of the heart. in psychology, most emotion theories acknowledge that the body plays a role in the emotion experience @xcite . different theories have been proposed for the role of the body when emotion arises. some studies suggest that changes in the body cause emotion @xcite . on the other hand, some researchers propose that emotion results in bodily changes @xcite @xcite . and some modern theories of embodied emotion @xcite present a different view that the mind and the body interact and shape emotion. our work draws heavily from the view that emotion results in bodily changes. ## evaluating emotion classifiers we conducted experiments to investigate how well existing emotion classifiers recognize embodied emotion. we evaluated several classifiers that achieved state-of-the-art performance on emotion or affect recognition tasks. the implementation details of these classifiers can be found in appendix b. the first model is spanemo (alhuzali and ananiadou, 2021), which is based on bert @xcite and trained on the affective tweets in semeval-2018 @xcite . the second model, which we will refer to as ge-bert, is a bert-base model fine-tuned with the goemotions data in @xcite . we also evaluated seq2emo @xcite , a bi-lstm trained on the goemotions dataset. all these models take a text snippet as input and generate multi-label emotions. finally, we evaluated aff-bert @xcite , an affective event classifier that takes an event phrase as input and identifies its affective polarity. these models were trained on different types of input, so we experimented with four strategies for applying each classifier to instances in our cheer data. consider the instance below with the underlined "eyes" as the targeted body part: preceding context: every step he took echoed throughout the room. he stood in front of me, empty eyes locked into mine. ## producing weakly labeled data for embodied emotions our goal was to create a classifier for recognizing embodied emotion expressions. we created gold training data, but its amount is relatively small as human annotation is time-consuming. in this section, we introduce two methods to automatically produce a large amount of weakly labeled instances. we will later show that this weakly labeled data can be used to train an effective classifier without any gold data at all, or used in combination with gold data to further improve classification performance. ## experimental results we conducted experiments with classification models trained on weakly labeled data, gold labeled data, or both. we also present results for zero-shot prompting with llms as a baseline comparison. for the evaluation metric, we report the macroaveraged f1 score over the test set, as well as precision, recall and f1 for each class. our classification model is based on fine-tuning the pretrained bert model @xcite . given an input instance, we concatenate the preceding sentences and the sentence that mentions the body part, and insert the cls token between them. we pass this to the bert-base-uncased model and get its last-layer token embeddings. finally, we produce an embedding for the body part word by averaging the embeddings of its leftmost and rightmost tokens, and then feed it through a linear classification layer to predict the label. for the sake of brevity, we will refer to the classification model as the embodied emotion classifier (eec). ## analysis we present several analyses to better understand the behavior of our embodied emotion classifier. ## conclusion our work presents the first study on recognizing expressions of embodied emotion in natural language. we created a dataset that contains 7,300 body part mentions with human annotation for this task, which can be found at https://github. com/yyzhuang1991/embodied-emotions . we performed extensive experiments to show that this task is challenging for existing emotion recognition methods. we also presented two methods to automatically produce a large set of weakly labeled instances, one pattern-based method that extracts manner expressions with explicit emotion words, and one prompting method that exploits a large language model. we showed that the weakly labeled data can be used to train an effective embodied emotion classifier, and that combining it with gold data yields a better classifier than using gold data alone.