D ialogue TRM : Exploring Multi-Modal Emotional Dynamics in a Conversation (2021)

Abstract: Emotion dynamics formulates principles explaining the emotional fluctuation during conversations. Recent studies explore the emotion dynamics from the self and inter-personal dependencies, however, ignoring the temporal and spatial dependencies in the situation of multi-modal conversations. To address the issue, we extend the concept of emotion dynamics to multi-modal settings and propose a Dialogue Transformer for simultaneously modeling the intra-modal and inter-modal emotion dynamics. Specifically, the intra-modal emotion dynamics is to not only capture the temporal dependency but also satisfy the context preference in every single modality. The inter-modal emotional dynamics aims at handling multi-grained spatial dependency across all modalities. Our models outperform the state-of-the-art with a margin of 4%-16% for most of the metrics on three benchmark datasets.

## introduction with the development of conversational agents, e.g., apple siri, google assistant, microsoft cortana, etc., there emerges pressing needs for emotion recognition in conversations (erc). different from conventional emotion recognition @xcite that treats emotions as stable traits, erc involves emotion dynamics @xcite in conversations. existing studies propose methods for modeling vanilla emotion dynamics by capturing self and inter-personal dependencies @xcite . the two dependencies are methodologically considered as modeling individual and conversational context using variants of context-dependent models @xcite . bidirectional contextual lstm @xcite is a straightforward approach but suffers from inadequacy of long-range summarization. to overcome the shortcoming, attention mechanism @xcite and memory network figure 1 : an example of multi-modal conversation. @xcite are introduced. besides, variants of hierarchical recurrent neural networks @xcite @xcite are proposed to model self and inter-personal dependencies simultaneously. for better context modeling, pre-training techniques are employed to erc @xcite . despite the progress of existing studies in modeling vanilla emotion dynamics in conversations, the temporal and spatial dependencies within multiple modalities are ignored. thus, we extend the concept of emotion dynamics to multi-modal settings, which takes account of the intra-modal and intermodal emotion dynamics, or multi-modal emotion dynamics for short. the intra-modal emotion dynamics is an emotional influence that one modality received from itself during a conversation. it needs temporal modeling in each modality. the intermodal emotion dynamics is another emotional influence that one modality received from the other modalities at each conversation turn. it requires spatial modeling across all modalities. the interplays between intra-modal and inter-modal emotion dynamics produce final emotional predictions. for intra-modal emotion dynamics, the temporal dependency of one modality can be captured through modeling the self and inter-personal dependencies as it is done in vanilla emotion dynamics. however, multi-modal expressions exhibit different dependence on context information. such charac-teristic is ignored by existing studies on erc. intuitively, spoken words are highly semantic that require inferences from the context to understand the emtions @xcite , while facial attributes or tones of voice are relatively concrete in which emotions are instantly burst in a short time, i.e., within an utterance period @xcite . the phenomenon is illustrated in figure 1 . here, the 7th-turn utterance "my sandwich" does not exhibit any anger unless looking back to infer that a is angry because b ate his sandwich. on the contrary, the anger is directly shown up in the frown faces or loud intonations at the 7th utterance period. thus, the modeling of intra-modal emotion dynamics should satisfy the context preferences of different modalities. for inter-modal emotion dynamics, the spatial dependency can be captured by interactive weighting across multi-modal features. existing studies on erc @xcite use concatenation to learn the linear weights, which lacks the interactions between modalities. many studies on multi-modal learning @xcite @xcite apply interactive weighting to fuse information from multiple modalities. however, most of them consider only one granularity of feature interaction. we argue that interactive weighting should consider both prototype and representation dependencies. the prototype dependency relates to position-wise neuron-grained feature interactions that allocate different weights to neurons in a vector. the representation dependency handles vectorgrained feature interactions that allocate a single weight to all neurons in a vector. the modeling of inter-modal emotion dynamics should consider the two granularities of dependencies. in this paper, we propose a dialoguetrans-former (dialoguetrm) that models the intramodal and inter-modal emotion dynamics simultaneously. for intra-modal emotion dynamics, we facilitate transformers for temporal modeling that satisfies the context preferences of different modalities. for inter-modal emotion dynamics, we design a multi-grained interactive fusion (mgif) to deal with the prototype and representation dependencies across modalities. finally, by incorporating the intra-modal and inter-modal emotion dynamics, our dialoguetrm achieves more accurate emotional predictions than state-of-the-art (sota). we highlight our contribution as follows: ## related work emotions are hidden mental states associated with thoughts and feelings @xcite . without physiological signals, they are only perceivable through human behaviors like spoken words, tones of voice, and facial attributes. emotion recognition is an interdisciplinary field that spans psychology, cognitive science, machine learning, natural language processing, and others @xcite . it involves handling multimodal data. early studies on emotion recognition are usually single-modal oriented @xcite @xcite . pioneers have explored the advantages of combining facial expressions and speech signals to predict emotions @xcite @xcite . recent studies @xcite @xcite have considered all the three modalities, whose primary focus is on fusion strategy while ignoring the emotion dynamics in a conversation. notice that @xcite take account of the intra-modal and cross-modal interactions between modalities, however, they ignore the context preference for each modality. emotion recognition in conversations is different from traditional emotion recognition due to emotion dynamics in conversations. by comparing with the recent proposed erc approaches @xcite @xcite , poria et al. discovered that traditional emotion recognition approaches @xcite @xcite @xcite failed to work well on erc datasets, because the same utterance within different context may exhibit different emotions @xcite . erc is advancing in the recent few years. sclstm @xcite is an rnn-based approach that captures the self-dependency using a bidirectional lstm. cmn @xcite and icon @xcite distinguish the self and inter-personal dependencies by leveraging memory network. dialoguernn @xcite uses multiple grus with global attention and further develops erc to multi-party conversations. dialoguegcn @xcite uses the graph convolutional network (gcn) to model complex interactions between interlocutors. bieru @xcite focus on the partyignorant transferring of emotion in a conversation. recently, several pieces of work, e.g., transfer learning erc @xcite , and commonsense knowledge erc @xcite , have employed pre-training models to the task of erc. however, those approaches ignore the multimodal emotion dynamics in conversations. our dialoguetrm is specially designed to model such kinds of emotion dynamics. multi-modal fusion seeks to generate a single representation to boost a specific task involving multiple modalities when building classifiers or other predictors. many surveys @xcite @xcite have investigated the strategies of multi-modal analysis with different kinds of clues. we divide fusion techniques into two groups. the first is combination approaches, including concatenation @xcite , hadamard product @xcite , summing up @xcite , differential operation @xcite , gate @xcite , attention @xcite . according to whether there are interactions between features, those approaches can be categorized into linear weighting fusion (first three) and interactive weighting fusion (latter three). the second is learning approaches. according to the learning objective, approaches can be categorized as task-oriented and self-learning fusion. taskoriented fusion @xcite @xcite is for supervised learning, whose hidden states are the learned features. self-learning fusion @xcite @xcite is often unsupervised learned by structures like restricted boltzmann machines @xcite or autoencoders @xcite . the strategy is to reconstruct source representation to target representation. both source and target representations could be one or any combination of the multiple modalities @xcite . our mgif has a similar idea with the sub-view attention (sva) mechanism @xcite . the main differences are, 1) our mgif considers both prototype and representation granularities of feature interactions while sva considers only the subview granularity; 2) our mgif can deal with multiple modalities while sva is dyadic fusion. and visual x λ i i,(v) modalities. according to the speakers that are involved, we define two types of context within a sliding window of k, which are indi-context,