COGMEN : CO ntextualized GNN based Multimodal Emotion recognitio N (2022)

Abstract: Emotions are an inherent part of human interactions, and consequently, it is imperative to develop AI systems that understand and recognize human emotions. During a conversation involving various people, a person’s emotions are influenced by the other speaker’s utterances and their own emotional state over the utterances. In this paper, we propose COntextualized Graph Neural Network based Multi- modal Emotion recognitioN (COGMEN) system that leverages local information (i.e., inter/intra dependency between speakers) and global information (context). The proposed model uses Graph Neural Network (GNN) based architecture to model the complex dependencies (local and global information) in a conversation. Our model gives state-of-the- art (SOTA) results on IEMOCAP and MOSEI datasets, and detailed ablation experiments show the importance of modeling information at both levels.

## introduction emotions are intrinsic to humans and guide their behavior and are indicative of the underlying thought process @xcite . consequently, understanding and recognizing emotions is vital for developing ai technologies (e.g., personal digital assistants) that interact directly with humans. during a conversation between a number of people, there is a constant ebb and flow of emotions experienced and expressed by each person. the task of multimodal emotion recognition addresses the problem of monitoring the emotions expressed (via various modalities, e.g., video (face), audio (speech)) by individuals in different settings such as conversations. emotions are physiological, behavioral, and communicative reactions to cognitively processed stimuli @xcite . emotions are often a result of internal physiological changes, and these physiological reactions may not be noticeable by others and are therefore intra-personal. for example, in a conversational setting, an emotion may be a communicative reaction that has its origin in a sentence spoken by another person, acting as a stimulus. the emotional states expressed in utterances correlate with the context directly; for example, if the underlying context is about a happy topic like celebrating a festival or description of a vacation, there will be more positive emotions like joy and surprise. consider the example shown in figure 1 , where the context depicts an exciting conversation. speaker-1 being excited about his admission affects the flow of emotions in the entire context. the emotion states of speaker-2 show the dependency on speaker-1 in u 2 , u 4 and u 6 , and maintains intra-personal state depicted in u 8 and u 10 by being curious about the responses of speaker-1. the example conversation portrays the effect of global information as well as inter and intra dependency of speakers on the emotional states of the utterances. moreover, emotions are a multimodal phenomenon; a person takes cues from different modalities (e.g., audio, video) to infer the emotions of others, since, very often, the in-formation in different modalities complement each other. in this paper, we leverage these intuitions and propose cogmen: contextualized graph neural network based multimodal emotion recog-nition architecture that addresses both, the effects of context on the utterances and inter and intra dependency for predicting the per-utterance emotion of each speaker during the conversation. there has been a lot of work on unimodal (using text only) prediction, but our focus is on multimodal emotion prediction. as is done in literature on multimodal emotion prediction, we do not focus on comparison with unimodal models. as shown via experiments and ablation studies, our model leverages both the sources (i.e., local and global) of information to give state-of-the-art (sota) results on the multimodal emotion recognition datasets iemocap and mosei. in a nutshell, we make the following contributions in this paper: ## related work emotion recognition is an actively researched problem in nlp @xcite . the broad applications ranging from emotion understanding systems, opinion mining from a corpus to emotion generation have attracted active research interest in recent years @xcite @xcite @xcite @xcite @xcite . availability of bench-mark multimodal datasets, such as cmu-mosei @xcite , and iemocap @xcite , have accelerated the progress in the area. broadly speaking, most of the existing work in this area can be categorized mainly into two areas: unimodal approaches and multimodal approaches. unimodal approaches tend to consider the text as a prominent mode of communication and solve the emotion recognition task using only text modality. in contrast, multimodal approaches are more naturalistic and consider multiple modalities (au-dio+video+text) and fuse them to recognize emotions. in this paper, we propose a multimodal approach to emotion recognition. nevertheless, we briefly outline some of the prominent unimodal approaches as some of the techniques are applicable to our setting. unimodal approaches: cosmic @xcite ) performs text only emotion classification problem by leveraging commonsense knowledge. dialogxl @xcite uses xlnet @xcite as architecture in dialogue feature extraction. cesta @xcite ) captures the emotional consistency in the utterances using conditional random fields @xcite for boosting the performance of emotion classification. other popular approaches parallel to our work use graph-based neural networks as their baseline and solve the context propagation issues in rnn-based architectures, including dialoguegcn @xcite , @xcite , congcn @xcite , and sumagg-gin @xcite . some of the recent approaches like dag-erc @xcite combine the strengths of conventional graph-based neural models and recurrence-based neural models. ## proposed model in a conversation involving different speakers, there is a continuous ebb and flow in the emotions of each of the speakers, usually triggered by the context and reactions of other speakers. inspired by this intuition, we propose a multimodal emotion prediction model that leverages contextual information, inter-speaker and intra-speaker relations in a conversation. in our model, we leverage both the context of dialogue and the effect of nearby utterances. we model these two sources of information via two means: 1) global information: how to capture the impact of underlying context on the emotional state of an utterance? 2) local information: how to establish relations between the nearby utterances that preserve both inter-speaker and intra-speaker dependence on utterances in a dialogue? global information: we want to have a unified model that can capture the underlying context and handle its effect on each utterance present in the dialogue. a transformer encoder @xcite architecture is a suitable choice for this goal. instead of following the conventional sequential encoding by adding positional encodings to the input, in our approach, a simple transformer encoder without any positional encodings leverages the entire context to generate distributed representations (features) efficiently corresponding to each utterance. the transformer facilitates the flow of information from all utterances when predicting emotion for a particular utterance. ## experiments we experiment for the emotion recognition task on the two widely used datasets: iemocap @xcite and mosei @xcite . the dataset statistics are given in iemocap is a dyadic multimodal emotion recognition dataset where each utterance in a dialogue is labeled with one of the six emotion categories: anger, excited, sadness, happiness, frustrated, and neutral. in literature, two iemocap settings are used for testing, one with 4 emotions (anger, sadness, happiness, neutral) and one with 6 emotions. we experiment with both of these settings. mosei is a multimodal emotion recognition dataset annotated with 7 sentiments (-3 (highly negative) to +3 (highly positive)) and 6 emotion labels (happiness, sadness, disgust, fear, surprise, and anger). note that the emotion labels differ across the datasets. we use weighted f1-score and accuracy as evaluation metrics (details in appendix c). implementation details: for iemocap, audio features (size 100) are extracted using opensmile @xcite , video features (size 512) are taken from @xcite , and text features (size 768) are extracted using sbert @xcite baselines: we do a comprehensive evaluation of cogmen by comparing it with a number of baseline models. for iemocap, we compare our model with the existing multimodal frameworks ( with number of utterances present in a dialogue (more details on effect of window size in appendix g.2). this experiment helps understand the importance of context in a dialogue. moreover, it points towards challenges in developing a real-time system (details in §6). we test the local information hypothesis by removing the gnn module and directly passing the context extracted features to the emotion classifier. ## discussion comparison with baselines: emotion recognition in a multimodal conversation setting comes with two broadly portrayed research challenges @xcite , first, the ability of a model to capture global and local context present in the dialogues, and second, the ability to maintain self and interpersonal dependencies among the speakers. all the popular baselines like dialogue-gcn @xcite , dialoguernn @xcite , bc-lstm @xcite af-can @xcite , etc., try to address these challenges by proposing various architectures. bc-lstm (bi-directional contextual lstm @xcite ) uses lstm to capture the contextual information and maintain long relations between the utterances from the past and future. another contemporary architecture af-can @xcite utilizes recurrent neural networks based on contextual attention to model the interaction and dependence between speakers and uses bi-directional gru units to capture the global features from past and future. we propose to address these issues using a unified architecture that captures the effect of context on utterances while maintaining the states for self and interpersonal dependencies. we make use of transformers for encoding the global context and make use of graphtransformers to capture the self and interpersonal dependencies. our way of forming relational graphs between the utterances comes close to dialoguegcn (unimodal architecture). we further use a shared emotion classifier for predicting emotions from all the obtained utterance level features. moreover, our unified architecture handles multiple modalities effectively and shows an increase in performance after adding information from other modalities. limitations (offline setting): a noteworthy limitation of all the proposed emotion recognition approaches (including the current one) is that they use global context from past and future utterances to predict emotions. however, baseline systems compared in this paper are also offline systems. for example, bc-lstm (bi-directional contextual lstm) and af-can use utterances from the past and future to predict emotions. other popular baselines like dialoguegcn and dialoguernn (bidi-aloguernn) also peek into the future, assuming the presence of all the utterances during inference (offline setting). all such systems that depend on future information can only be used in an offline setting to process and tag the dialogue. an emotion recognition system that could work in an online setting exhibits another line of future work worth exploring due to its vast use cases in live telecasting and telecommunication. a possible approach to maintain the context in an online setting would be to take a buffer of smaller context size, where the model can predict emotions taking not the complete dialogue but a smaller subset of it as input in real-time. we tried exploring this setting for our architecture with an online buffer of maintaining a smaller context window. for experimenting with it, we created a sub-dataset using the iemocap (4-way) setting by splitting each dialogue into n utterances and training our architecture. our results in ## conclusion and future work we present a novel approach of using gnns for multimodal emotion recognition and propose cogmen: contextualized gnn based multimodal emotion recognition. we test cogmen on two widely known multimodal emotion recognition datasets, iemocap and mosei. cogmen outperforms the existing state-of-the-art methods in multimodal emotion recognition by a significant margin (i.e., 7.7% f1-score increase for iemo-cap (4-way)). by comprehensive analysis and ablation studies over cogmen, we show the importance of different modules. cogmen fuses information effectively from multiple modalities to improve the performance of emotion prediction tasks. we perform a detailed error analysis and observe that the misclassifications are mainly between the similar classes and emotion shift cases. we plan to address this in future work, where the focus will be to incorporate a component for capturing the emotional shifts for fine-grained emotion prediction. we use pytorch @xcite for training our architecture and pyg (pytorch geometric) @xcite for the gnn component in our architecture. we use comet (comet.ml, 2021) for logging all our experiments and its bayesian optimizer for hyperparameter tuning. our architecture trained on the iemocap dataset has 55,932,052 parameters and takes around 7 minutes to train for 50 epochs on the nvidia tesla k80 gpu. comparison of the model with baselines in terms of the number of parameters is challenging, as the baselines parameters vary depending on the hyperparameter setting. moreover, many baselines do not provide information about the number of parameters.