M 3 HG : Multimodal, Multi-scale, and Multi-type Node Heterogeneous Graph for Emotion Cause Triplet Extraction in Conversations (2025)

Abstract: Emotion Cause Triplet Extraction in Multimodal Conversations (MECTEC) has recently gained significant attention in social media analysis, aiming to extract emotion utterances, cause utterances, and emotion categories simultaneously. However, the scarcity of related datasets, with only one published dataset featuring highly uniform dialogue scenarios, hinders model development in this field. To address this, we introduce MECAD, the first multimodal, multi-scenario MECTEC dataset, comprising 989 conversations from 56 TV series spanning a wide range of dialogue contexts. In addition, existing MECTEC methods fail to explicitly model emotional and causal contexts and neglect the fusion of semantic information at different levels, leading to performance degradation. In this paper, we propose M3HG, a novel model that explicitly captures emotional and causal contexts and effectively fuses contextual information at both inter- and intra-utterance levels via a multimodal heterogeneous graph. Extensive experiments demonstrate the effectiveness of M3HG compared with existing state-of-the-art methods. Codes are available at https://anonymous.4open.science/r/M3HG-6B34.

## introduction emotion cause analysis in conversations (ecac) aims at identifying emotions and their causes in conversations, which is a crucial research field in natural language processing @xcite . however, most of ecac research @xcite @xcite only focuses on the textual contexts, overlooking other modalities @xcite . to address this limitation, @xcite proposed a new task called multimodal emotion cause triplet extraction in conversations (mectec). the task aims to simultaneously identify the emotion utterance, the corresponding cause utterances, and the emotion category (i.e., the utter-causeemotion triplet) from a conversation containing three modalities: text, audio, and video. figure 1 illustrates a multimodal conversation between a mother and daughter. in this example, there are six non-neutral utterances, and consequently, six uttercause-emotion triplets are identified. mectec differs from ecac in 1) multimodal contexts (i.e., text, audio, and video) resulting in more complex emotional expression, and 2) multi-scale semantic information from overall conversation and utterance features like intonation and facial expressions, which pose significant challenges. another major challenge in mectec is the scarcity of datasets. while numerous text-based datasets exist for ecac, only one dataset, namely the ecf dataset @xcite , is specifically designed for mectec. however, the videos in ecf are all from the friends tv series with restricted speakers and scenarios, hindering mectec model development. therefore, in this work, a new multimodal, multi-scenario mectec dataset, namely mecad, is constructed. to the best of our knowledge, it is the first of its kind and will greatly facilitate research in this field. constrained by the limited dataset, existing mectec models have various deficiencies. @xcite proposed a two-stage architecture that predicts emotion and cause utterances separately. however, this approach is computationally intensive and prone to error accumulation. therefore, recent studies @xcite @xcite propose one-stage architectures using graph neural networks or prompt engineering to extract utter-cause-emotion triplets. however, these methods do not explicitly extract specific contexts 11416 actually, this is pretty nice too. ## proposed mecad dataset to facilitate the research in mectec and other related fields, we constructed a multi-scenario mectec dataset called mecad. compared with ecf @xcite , mecad has more diverse conversation scenarios. in addition to labeling emotion categories and their causes for each utterance, we also categorized the types of emotion causes (e.g., event, expression) and the modality of annotation (i.e., text, audio, or video) to support future studies in multimodal emotion cause analysis. we selected the publicly available m 3 ed @xcite dataset as our data source, which contains 990 segments from 56 chinese tv series. however, m 3 ed dataset only contains conversation scripts, audios, and screenshots, lacking corresponding videos. therefore, we endeavored to collect the corresponding video segments based on the conversation timestamps provided by m 3 ed. we concatenated sentences to form 989 multimodal conversations with 10,516 full utterances. we invited 10 chinese graduate students majored in psychology to annotate the corresponding cause utterances, the types of emotion causes and the modal cues of annotations in the conversations. to obtain high-quality annotations, we designed detailed guidelines based on previous studies @xcite , trained the volunteers, and tested them with annotation cases. only those passing the test participated in the final annotation process. each volunteer was paid $50 for their annotations. then, we randomly assigned three qualified annotators for each conversation. if divergence exists among annotations from different volunteers, the final annotation for the utterance is determined by majority voting. two strategies were used to review and revise incorrect annotations: 1) annotation consistency among the three annotators for each tv series is calculated. for series with low consistency, the annotators rechecked and revised their labels as needed. 2) if disagreements remained, a fourth annotator was invited to relabel the utterances and make the final decision. to enhance annotation efficiency and accuracy, we developed an online multimodal conversation emotion cause annotation tool. the interface of the annotation tool is shown in figure 3 in appendix b. this tool is highly reusable and user-friendly, making it ideal for related research in the future. we use cohen's kappa @xcite to assess pairwise agreement and fleiss's kappa @xcite for overall consistency among annotators. the cohen's kappa results are in appendix a, and the fleiss's kappa score of 0.6932 exceeds the threshold of 0.61 @xcite , confirming the statistical reliability of our annotations. the dataset statistics and detailed analysis of mecad are presented in figure 4 in appendix a. mecad provides solid support for assessing the performance and generalization capabilities of mectec models in broader scenarios. ## conclusion in this work, we propose the first multimodal and multi-scenario chinese emotion-cause analysis dataset, mecad, for mectec and related emotion cause analysis tasks. compared to ecf, the only existing dataset for multimodal emotion-cause analysis, mecad offers more diverse conversation scenarios. it helps to enhance the generalizability and applicability of mectec models in complex social media environments. moreover, mecad is a valuable resource for cross-cultural emotion analysis and recognition. furthermore, we propose a generalized mectec framework named m 3 hg, which deeply extracts emotional and causal contexts, while effectively integrating semantic information across multiple granular levels. extensive experiments on the ecf dataset and the mecad dataset demonstrate the superiority of our method compared to the existing state-of-the-art methods.