M 3 ED : Multi-modal Multi-scene Multi-label Emotional Dialogue Database (2022)

Abstract: The emotional state of a speaker can be influenced by many different factors in dialogues, such as dialogue scene, dialogue topic, and interlocutor stimulus. The currently available data resources to support such multimodal affective analysis in dialogues are however limited in scale and diversity. In this work, we propose a Multi-modal Multi-scene Multi-label Emotional Dialogue dataset, M ^3 ED, which contains 990 dyadic emotional dialogues from 56 different TV series, a total of 9,082 turns and 24,449 utterances. M ^3 ED is annotated with 7 emotion categories (happy, surprise, sad, disgust, anger, fear, and neutral) at utterance level, and encompasses acoustic, visual, and textual modalities. To the best of our knowledge, M ^3 ED is the first multimodal emotional dialogue dataset in Chinese.It is valuable for cross-culture emotion analysis and recognition. We apply several state-of-the-art methods on the M ^3 ED dataset to verify the validity and quality of the dataset. We also propose a general Multimodal Dialogue-aware Interaction framework, MDI, to model the dialogue context for emotion recognition, which achieves comparable performance to the state-of-the-art methods on the M ^3 ED. The full dataset and codes are available.

## introduction emotion recognition in conversation (erc) aims to automatically identify and track the emotional status of speakers during a dialogue @xcite . it is a crucial component to improve natural human-computer interactions and has a wide range of applications in interaction scenarios, including call-center dialogue systems @xcite , conversational agents @xcite and mental health diagnoses @xcite , etc. different from traditional multimodal emotion recognition on isolated utterances, multimodal erc is a more challenging problem, because there are many influencing factors that affect the speakers' emotional state in a dialogue, including the dialogue context from multi-modalities, the scene, the topic, and even the personality of subjects, etc. @xcite @xcite . it has been proved in recent works @xcite @xcite that contextual information plays an important role in erc tasks and brings significant improvements over baselines that only consider isolated utterances. dialoguernn @xcite uses recurrent networks to model global and speaker-specific temporal-context information. di-aloguegcn @xcite and mmgcn @xcite use graph-based networks to capture conversational dependencies between utterances in dialogues. dialogxl @xcite applies a strong pre-trained language model xlnet @xcite to erc and proposes a dialogaware self-attention method for modeling the context information. the iemocap @xcite and meld @xcite are two multimodal emotional dialogue benchmark datasets, which are widely used in the above-mentioned works and promote research in the affective computing field. however, both of them are limited in size and diversity. the videos in meld are collected only from the friends tv series, and the videos in iemocap are recorded in laboratory environments from ten actors performing scripted and spontaneous dialogues. these limitations not only affect the investigation of generalization and robustness of the models, but also limit the exploration of other important influencing factors in dialogues, such as dialogue scene, dialogue topic, emotional influence from interlocutors, and so on. in this work, we construct a large-scale multimodal multi-scene and multi-label emotional dialogue dataset, m 3 ed, which consists of 990 emotional dyadic dialogue video clips from 56 different tv series (about 500 episodes), ensuring that there are various dialogue scenes and topics. we also consider the blended annotations of emotions, which are commonly observed in real-life human interactions @xcite . m 3 ed contains 24449 utterances in total, which are more than three times larger than iemocap and almost two times larger than meld. there are rich emotional interaction phenomena in m 3 ed dialogues, for example, 5,396 and 2,696 inter-turn emotion-shift and emotioninertia scenarios respectively, and 2,879 and 10,891 intra-turn emotion-shift and emotion-inertia scenarios respectively. to the best of our knowledge, m 3 ed is the first large-scale multi-modal emotional dialogue dataset in chinese, which can promote research of affective computing for the chinese language. it is also a valuable addition for cross-cultural emotion analysis and recognition. we further perform the sanity check of the dataset quality. specifically, we evaluate our proposed m 3 ed dataset on several state-of-the-art approaches, including dialoguernn, dialoguegcn, and mmgcn. the experimental results show that both context information and multiple modalities can help model the speakers' emotional states and significantly improve the recognition performance, in which context information and multi-ple modalities are two salient factors of a multimodal emotion dialogue dataset. furthermore, motivated by the masking strategies of self-attention used in dialogxl @xcite , we propose a general multimodal dialogue-aware interaction (mdi) framework which considers multimodal fusion, global-local context modeling, and speaker interactions modeling and achieves state-of-the-art performance. all in all, m 3 ed is a large, diverse, high-quality, and comprehensive multimodal emotional dialogue dataset, which can support more explorations in the related research directions, such as multi-label learning, interpretability of emotional changes in dialogues, cross-culture emotion recognition, etc. the main contributions of this work are as follows: ## proposed framework a dialogue can be defined as a sequence of utterances @xmath0, utt 2 , ..., utt n }, where n is the number of utterances. each utterance consists of textual (l), acoustic (a) and visual (v) modalities. we denote u a t @xcite as the utterance-level feature of utterance utt t from speaker a with the textual, acoustic and visual modality respectively. the task aims to predict the emotional state for each utterance in the dialogue based on all existing modalities. figure 2 illustrates our proposed multimodal dialogue-aware interaction (mdi) framework, which contains three main modules: 1) multimodal fusion module aims to generate the utterance-level multimodal representation from different modalities. 2) dialog-aware interaction module aims to model the interactions in the dialogue; 3) interaction fusion and classification module fuses the different interaction information from the outputs of the dialog-aware interaction module, and then makes the emotional state prediction based on the fused interaction information. multimodal fusion module: based on the modality-specific feature representations from different modalities, we apply early fusion of these modalities features to produce the multimodal feature representation: figure 3 : illustration of the four masking strategies corresponding to the four interaction sub-modules respectively. a i denotes the i-th utterance of the speaker a. the yellow blocks denote the current utterances. utterances that can be accessed by the current utterance are marked as green, while those can not be accessed are marked as white. dialog-aware interaction module: in order to adequately capture the contextual information in the dialogue, we propose the dialog-aware interaction module which consists of l dialog-aware interaction blocks (gray block in figure 2 ). in each block, we adopt four sub-modules, global interaction, local interaction, intra-speaker interaction and inter-speaker interaction, to model the global, local, intra-speaker and inter-speaker interactions in the dialogue respectively. we implement these four types of interactions in one transformer layer by skillfully changing the masking strategies of self-attention @xcite ) as illustrated in figure 3 . ## future directions the m 3 ed dataset is a large, diversified, highquality, and comprehensive multimodal emotional dialogue dataset. based on the characteristics of the dataset and the analysis from the extensive experiments, we believe that m 3 ed can support a number of related explorations in affective computing field. ## conclusion in this work, we propose a multi-modal, multiscene, and multi-label emotional dialogue dataset, m 3 ed, for multimodal emotion recognition in conversations. compared to meld, the currently largest multimodal dialogue dataset for emotion recognition, m 3 ed is larger (24,449 vs. 13,708 utterances), more diversified (56 different tv series vs. only one tv series friends), with higher-quality (balanced performance across all three modalities), and containing blended emotions annotation which is not available in meld. m 3 ed is the first multimodal emotion dialogue dataset in chinese, which can serve as a valuable addition to the affective computing community and promote the research of cross-culture emotion analysis and recognition. furthermore, we propose a general multimodal dialog-aware interaction framework, which considers multimodal fusion, temporal-context modeling, and speaker interactions modeling, and achieves the state-of-the-art performance. we also propose several interesting future exploration directions based on the m 3 ed dataset. 5706 this work was partially supported by the national key r&d program of china (no. 2020aaa0108600), the national natural science foundation of china (no. 62072462), large-scale pre-training program 468 of beijing academy of artificial intelligence (baai), a*star rie2020 advanced manufacturing and engineering domain (ame) programmatic grant (no. a1687b0033), nrf centre for advanced robotics technology innovation (cartin) project and china scholarship council.