Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts (2022)

Abstract: Emotion Recognition in Conversations (ERC) is an important and active research area. Recent work has shown the benefits of using multiple modalities (e.g., text, audio, and video) for the ERC task. In a conversation, participants tend to maintain a particular emotional state unless some stimuli evokes a change. There is a continuous ebb and flow of emotions in a conversation. Inspired by this observation, we propose a multimodal ERC model and augment it with an emotion-shift component that improves performance. The proposed emotion-shift component is modular and can be added to any existing multimodal ERC model (with a few modifications). We experiment with different variants of the model, and results show that the inclusion of emotion shift signal helps the model to outperform existing models for ERC on MOSEI and IEMOCAP datasets.

## introduction humans are complex social beings, and emotions are indicative of not just their inner state and feelings but also their internal thinking process @xcite . to fully understand a person, one needs to understand their inherent emotions. recent research has witnessed colossal interest in including artificially intelligent machines as conversable companions for humans, e.g., personal digital assistants. however, communication with ai systems is quite limited. ai systems do not understand the inherent emotions expressed implicitly by humans making them unable to comprehend the underlying thought processes and respond appropriately. consequently, a wide variety of approaches have been proposed for developing emotion understanding and generation systems @xcite @xcite @xcite @xcite . ## related work emotion recognition using multiple modalities is an active area of research leading to the development of widely popular benchmark datasets, e.g., cmu-mosei @xcite , and iemocap @xcite . recent works have highlighted the crucial aspects of self, and interpersonal dependencies in the emotional dynamics of the conversations @xcite . another essential feature is the role of the local and global context for emotion recognition systems. some notable works like dialogue rnn @xcite try to capture these properties by modeling each speaker with a party state and the emotion of each utterance by an emotional state. furthermore, a context state is maintained to model the global conversation context. another work multilogue-net @xcite highlights the limitation of the fusion mechanism used in dialogue rnn @xcite and tries to solve it using a party, context, and emotion grus for each modality. it uses a pairwise attention mechanism proposed by @xcite to fuse the emotion states for all the modalities effectively. however, dialoguernn highlights the poor performance in predicting the emotions with the utterances where the emotion shifts from positive to negative sentiments. our work considers the emotion shifts present in the dialogues and tries to leverage them for improving emotion recognition. another line of work includes transformer-based joint-encoding (tbje) @xcite ) that achieves the state-of-the-art results on the sentiment task for the mosei dataset using a multimodal transformer-based model for combining multiple modalities. however, in the emotion task, tbje is outperformed by the multilogue-net model. the possible reason highlighted by the paper is the lack of context-awareness in the architecture, as tbje neither uses the previous nor next utterance to predict the emotion for the current utterance. some of the other works in multimodal emotion recognition include the memory fusion network (mfn) @xcite , which aligns multimodal sequences using multi-view gated memory, graph-mfn @xcite which uses dynamic fusion graph (dfg) and learns to model the n-modal interactions dynamically, and bc-lstm @xcite which uses an lstm-based model to capture contextual information. cesta @xcite captures the emotional consistency in the utterances using crf model @xcite for boosting the performance of emotion classification and comes close to our idea of leveraging emotion shifts. ## the proposed model during a conversation, speakers tend to maintain an emotional flow of affective states. these states majorly rely on the context of the entire conversation; for example, if the overall gist of the speaker about the topic is positive, the emotions like happiness, joy, and surprise can be seen more often than negative emotions like anger and sadness. moreover, a speaker's emotions are often affected by the past emotions present in the conversation. hence, an emotion prediction model should not only take into account the context but should also be able to maintain the speaker-level information along with the emotions present in the past utterances. considering these assumptions, we propose the primary component of our emotion recognition model: the emotion classification component. the emotions classification component predicts an utterance's emotion label using information from the current speaker, emotions of the previous utterances, and the entire conversation context. another significant insight about the emotions in a conversation is the sudden shift of emotional states. many of the existing state-of-the-art approaches highlight this in their error analysis, where the model fails to capture the sudden shifts in emotional states leading to a misclassified emotion prediction. to incorporate the effect of a sudden shift in the emotion, we introduce a separately trained component called the emotion shift component. the emotion shift component explicitly models the probability (p shif t t ) of a shift in emotion between the utterances u t-1 and u t . this shift in emotion can be expressed as moving from a positive (e.g., happy) to negative emotion (e.g., sad) or vice-versa between consecutive utterances. the emotion shift component being independent of the primary architecture is pretrained, and helps control the information flow from past to future during a sudden change. the signal from the pretrained emotional shift component is added to the emotion classification component to control the flow of emotions from past to future. figure 2 shows detailed architecture of the proposed model. emotion classification component: for modeling the underlying emotions in a conversation, we maintain a party state, emotion state and context state. the party state is maintained for each speaker and helps to keep track of the participant specific aspect in a conversation. the context state is global (common across each participant) and helps to encode the entire conversation context, thereby capturing inter-utterance dependencies. akin to the context state, the emotion state is also global and helps to leverage the emotion information flow between utterances. moreover, the emotion shift signal between the current and previous utterance is used to update the global emotion state. the emotion label for each utterance is then predicted by decoding the emotion state. in our model each of the party, context and emotion states are modality specific and are updated using a modality specific gru @xcite network for each modality @xmath0, a, v} (indicated by the superscript m). we employ late fusion to combine the emotion states from different modalities. next, we explain different gru networks used in the model. for each modality @xmath1, a, v}, q t 's party state s qt,m t-1 is updated to s qt,m t using an attention vector x m t and modality specific feature m t (eq. 1), ⊕ denotes concatenation operation. here x m t is calculated using a simple dot product attention mechanism over the context states (c m t ). note that for all speakers other than q t , the party state at t -1 and t remains the same. context state update (gru c ): global conversation context is modeled using the context state update gru c . for each modality @xmath2, a, v}, the global context state c m t-1 is updated to c m t (eq. 2) using the q t 's party state s qt,m t and the corresponding modality feature m t . context states (c m 1 , • • • , c m t-1 ) are used for calculating the attention vector x m t for each modality @xmath3, a, v} as follows: emotion state update (gru arc ): for each modality @xmath4, a, v}, the global emotion state e m t-1 is updated to e m t (eq. 3) using the current party state s qt,m t and modulated by the emotion shift component (p shift t ). the emotion states for all the three modalities are fused together @xcite to create e t using a pairwise attention mechanism @xcite ). e t is later used to decode the emotion class for an utterance. the emotion classification component is a context-aware model similar to that of previous works like multilogue-net (shenoy and sardana, 2020) but with a few key differences. firstly, instead of modelling an emotion state for each participant, we introduce global emotion state for each conversation. this is done to make use of the flow of emotion between utterances. secondly, the emotion shift signal between the current and previous utterance (p shift t ) is used to update the global emotion state using a gru arc which aims to model the emotion arc in the conversation. emotion shift component: to capture the emotional arc across the conversation, we explicitly model probability of emotion shift (p shif t t ) between successive utterances (u t-1 and u t ). we use a siamese network @xcite to model the emotional shift present across utterances. a siamese network generally consists of two or more identical subnetworks having the same configuration with shared parameters and weights. the proposed emotion shift architecture takes the textual features of the current (l t ) and previous (l t-1 ) utterances and outputs the probability of maintaining emotional inertia (p inertia t ). the architecture of the emotion shift prediction network is shown in lower half of figure 2 . we use sentence-bert (sbert) @xcite embeddings as textual features. sbert is a modification of the pretrained bert @xcite network that uses the siamese network to derive semantically meaningful sentence embeddings for transfer learning tasks. the emotion shift prediction network makes use of only the text modality for two reasons. firstly, it has often been found empirically that among the text, audio, and video modalities, text modality carries more information for erc tasks @xcite . secondly, early fusion techniques to combine the three modalities can suffer in a siamese-type architecture due to difficulty in mapping the fused modality vector to a vector space in which similar vectors are close. we also experimentally verify this ( §6). the emotion shift prediction network (between u t and u t-1 ) takes in text features corresponding to utterances (l t and l t-1 ) and their element wise differences to output the probability of a shift as given by (eq. 7). here, p inertia t is calculated using siamese network (eq. 8, 9). here, h t is the siamese hidden state, @xmath5. for the siamese network, we use binary cross entropy loss (l s ) over the distribution p shift t . the emotion shift component modulates the emotion state gru arc via p shift t and hence controls the flow of information during the conversation. the emotion shift component captures the emotional consistency in the utterances and can act as an independent modular component that can be pretrained and added to any existing multi-modal erc framework with a few modifications for improving emotion recognition in conversations. overall architecture: the motivation for the proposed architecture follows from the intuition that we need to weigh down the contribution of the previous emotion state in case of an emotion shift. in other words, we need to reduce the influence of e m t-1 in the calculation of e m t when there is a high p shif t t . to do so, we modify the reset and update gates in the gru modelling the emotional arc of the conversation i.e. gru arc . a gru has gating units (reset and update gates) that modulate the flow of information inside the unit. @xcite mention the usefulness of reset gate in scenarios where significant discontinuities are present in the sequence, thereby indicating its crucial role to forget information. their work also finds a redundancy in the activations of the reset and update gates when processing speech sequences. motivated by this, and the intuition that we need to forget more information when there is a higher probability of an emotional shift, we directly use the value of (1 -p shif t t ) for both the reset and update gates. the updates for gru arc unit are given by eqs. 10, 11. eq. 10 calculates a candidate emotion state ẽm t in which the prior emotion state's (e m t-1 ) is controlled by the emotion shift signal. the output e m t is a linear interpolation between ẽm t-1 and e m t-1 . again, p shif t t controls the influence of e m t-1 (eq. 11). therefore, a higher value of p shif t t will limit the contribution of the previous emotion state. in the absence of the emotion shift component, the gru gates are learned using only the classification data, much like the rest of the parameters in the model. if the total number of parameters in a model is huge (as is the case with most deep learning models), the gates might be unable to learn well. we verify that the modeling of the shift in emotion encourages better learning of these gates ( §6). for prediction at time t, the emotion vector e t (formed from fusion of e m t as described in (eq. 4) ) is passed through a final classification layer w c (∈ r de×k ) where k is the number of emotion or sentiment classes. this is used to obtain probability distribution over emotion labels via the softmax activation: @xmath6. we use the cross-entropy loss over this distribution to train the weights. ## analysis and ablation studies due to a wide variety of components, it becomes vital to perform a detailed analysis of the architecture to understand the importance of various choices. feature and design choices: for understanding the importance of features used for different modalities, we choose two different sets of features for text and visual modalities. in one setting, we use averaged glove embeddings @xcite ) for text, opensmile features @xcite for audio and facet features @xcite for video. whereas in another setting, for text modality, we make use of a pre-trained bert @xcite model's output layer. we calculate the average of the output layer to get a fixed-sized vector. for visual modality, we use the features provided by openface2.0 @xcite , which are useful in performing facial analysis tasks such as facial landmark detection, head-pose tracking, and eye-gaze tracking. the results in ## discussion the presence of emotion shifts in human-to-human conversation is prominent in the conversational datasets. the existing works based on sequential modeling often suffer from these shifts, leading to poor performance for utterances with emotion shifts. in this work, we try to control the effect of previous utterances using an independent emotion shift module. as highlighted in tables 8 and 9 , the proposed architecture performs significantly better on emotion shift cases when compared to multilogue-net (20% improvement in negativepositive and 4% improvement on positive-negative shifts). the novel design of the emotion shift-based gating mechanism in the gru unit helps boost the prediction performance for utterances with emotion shifts. as noticed in fig. 3 , the reset and update gates provide a significant signal when there is an emotional shift in conversation. modularity: the modular design and idea of the proposed emotion shift component can further be used to improve any emotion prediction systems that have poor performance in emotion shift cases. moreover, the designed emotion shift component works considering only the textual modality, mak-most of the time in an online course the student to content interactivity is assumed but --the first thing to think about here is that they are three interactivity types. you have student to student interactivity, student to content interactivity, and student to instructor interactivity mostly --assumed. student to student relationships and student to instructor relationships aren't always assumed. one of the opportunities you can do is to have a hallway conversation area i know that some .. emotion unchanged emotion shift emotion shift 0.87 0.23 0.33 1.00 o t r t 0.83 0.75 0.87 0.84 o t r t reset gate with emotion shift component reset gate without emotion shift component video id (m7sjs73sf8w) figure 3: reset gate activations on the dialogue 'm7sjs73sf8w" from cmu-mosei test dataset application to real-time systems: a notable limitation of all the existing emotion recognition state-of-the-art systems often comes from the incapability of their implementations for real-time use cases as they require the entire context to be given in the form of multiple utterances to the model. for future approaches where the models will target the real-time setting, the proposed emotion shift component can be handy as it only uses two consecutive utterances to predict the emotion shift. ## conclusion and future directions in this paper, we proposed a deep learning based model for multimodal emotion recognition in conversations. we proposed a new emotion shift component (modeled using the siamese net) that captures the emotional arc in a conversation and steers the main emotion recognition model. we performed a battery of experiments on two main emotion recognition datasets. results and analysis show the importance of the emotion shift component. currently, the emotion shift component uses only the text modality for predicting the shift and we plan to explore more sophisticated ways of using information from multiple modalities.