Modeling Label Semantics for Predicting Emotional Reactions (2020)

Abstract: Predicting how events induce emotions in the characters of a story is typically seen as a standard multi-label classification task, which usually treats labels as anonymous classes to predict. They ignore information that may be conveyed by the emotion labels themselves. We propose that the semantics of emotion labels can guide a model’s attention when representing the input story. Further, we observe that the emotions evoked by an event are often related: an event that evokes joy is unlikely to also evoke sadness. In this work, we explicitly model label classes via label embeddings, and add mechanisms that track label-label correlations both during training and inference. We also introduce a new semi-supervision strategy that regularizes for the correlations on unlabeled data. Our empirical evaluations show that modeling label semantics yields consistent benefits, and we advance the state-of-the-art on an emotion inference task.

## introduction understanding how events in a story affect the characters involved is an integral part of narrative understanding. @xcite introduced an emotion inference task on a subset of the rocstories dataset @xcite , labeling entities with the emotions they experience from the short story contexts. previous work on this and related tasks typically frame them as multi-label classification problems. the standard approach uses an encoder that produces a representation of the target event along with the surrounding story events, and then pushes it through a classification layer to predict the possible emotion labels @xcite . this classification framework ignores the semantics of the emotions themselves. each emotion label (e.g., joy) is just a binary prediction. however, consider the sentence, "danielle was really short on money". the emotional reaction is fear of being short on money. first, if a model had lexical foreknowledge of "fear", we should expect an improved ability to decide if a target event evokes fear. second, such a model might represent relationships between the emotions themselves. for example, an event that evokes fear is likely to evoke sadness and unlikely to evoke joy. when previous models frame this as binary label prediction, they miss out on ways to leverage label semantics. in this work, we show that explicitly modeling label semantics improves emotion inference. we describe three main contributions foot_0 . first, we show how to use embeddings as the label semantics representation. we then propose a label attention network that produces label-informed representations of the event and the story context to improve prediction accuracy. second, we add mechanisms that can make use of label-label correlations as part of both training and inference. during training, the correlations are used to add a regularization loss. during inference, the prediction logits for each label are modified to incorporate the correlations, thus allowing the model's confidence on one label to influence its prediction of other labels. third, we show that the label correlations can be used as a semi-supervised signal on the unlabeled portion of the rocstories dataset. our empirical evaluations show that adding label semantics consistently improves prediction accuracy, and produces labelings that are more consistent than models without label semantics. our best model outperforms previously reported results and achieves more than 4.9 points absolute improvement over the bert classification model yielding a new state-of-the-art result for this task. ## emotion inference the emotion inference task introduced by rashkin et al. ( 2018 ) is defined over a subset of short stories from the rocstories dataset @xcite . it infers the reactions that each event evokes in the characters of the story, given the story context thus far. for each sentence (i.e. event) in a story, the training data includes annotations of eight emotions. given a sentence x s denoting a single event in a story, the task is to label the possible emotional reactions that an event evokes in each character in the story. since an event can evoke multiple reactions, the task is formulated as a multi-label classification problem. the standard approach to this task has been as follows. for a given character c and the target sentence x s , collect all previous sentences x c in the story in which the character c is mentioned as the character context. encode the target sentence, and the character context to obtain a single representation, and use it as input to a multi-label classification layer for prediction. rashkin et al. (2018) benchmark the performance of multiple encoders (see section 5). we extend this previous work to integrate label semantics into the model by adding label embeddings (section 3) and explicitly representing labellabel correlations (section 4). ## label semantics using embeddings a simple strategy to model label semantics is to explicitly represent each with an embedding that captures the surface semantics of its label name. since the emotion labels correspond to actual words (e.g., joy, fear, etc.), we can initialize them with their corresponding word embeddings (learned from a large corpus). we then use these label embeddings in two ways as detailed below. ## results a further increase of 2 points in f1 is achieved by tracking label-label correlations through training loss and inference logits. in addition, adding semi-supervision yields the best gain of more than 4.9 points in f1 over basic bert, providing a significant advance in state-of-the-art results for emotion inference in this dataset. we also checked the statistical significance of the semi-supervision model ( sentence ground truth ls nols and nobody could give him any direction sad, disgust sad, disgust sad surprise anger she said mark can come for free joy, trust joy ,trust joy, anticipation anticipation anticipation he is relieved that it was not harmed joy, surprise joy, surprise fear, surprise anticipation anticipation the marshmallows were totally smooshed anger, sad anger, sad joy, anticipation ## related work one of the most widely-used work in narrative understanding introduced rocstories, a dataset for evaluating story understanding @xcite . on a subset of these stories @xcite added annotations for causal links between events in stories and mental states of characters. they model entity state to predict emotional reactions and motivations for causing events occurring in rocstories. additionally, they also introduce a new dataset annotation that tracks emotional reactions and motivations of characters in stories. other work looked at encoding external knowledge sources to augment motivation inference @xcite on the same dataset. both treat labels as anonymous classes, whereas this work explores modeling the semantics of the emotion labels explicitly. recent work in multi-label emotion classification has shown that using the relation information between labels can improve performance. @xcite use the label co-occurrence information in the final layer of the neural network to improve multi-label classification. correlationbased label representations have also been used for music classification styles @xcite . our work builds on these and adds a similar result showing that label correlations can have significant impact for emotion label inference. ## conclusions we present new results for the multi-label emotion classification task of rashkin et al. (2018), extending previous reported results by 10.7 f1 points (55.1 to 65.8). the multi-label nature of emotion prediction lends itself naturally to use the correlations between the labels themselves. further, we showed that modeling the class labels as semantic embeddings helped to learn better representations with more meaningful predictions. as with many tasks, bert provided additional context, but our integration of these label semantics showed significant improvements. we believe these models can improve many other nlp tasks where the class labels carry inherent semantic meaning in their names.