MIPS at S em E val-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models (2024)

Abstract: This paper presents our winning submission to Subtask 2 of SemEval 2024 Task 3 on multimodal emotion cause analysis in conversations. We propose a novel Multimodal Emotion Recognition and Multimodal Emotion Cause Extraction (MER-MCE) framework that integrates text, audio, and visual modalities using specialized emotion encoders. Our approach sets itself apart from top-performing teams by leveraging modality-specific features for enhanced emotion understanding and causality inference. Experimental evaluation demonstrates the advantages of our multimodal approach, with our submission achieving a competitive weighted F1 score of 0.3435, ranking third with a margin of only 0.0339 behind the 1st team and 0.0025 behind the 2nd team.

## introduction emotion-cause pair extraction in conversations has garnered significant attention due to its wideranging applications, such as optimizing customer service interactions and tailoring content recommendations based on user emotions @xcite . however, a fundamental challenge lies in identifying the causal determinants of emotional states. recent research emphasizes the exploration of causes triggering emotions from multimodal data @xcite , followed by further generation tasks based on multimodal emotional cues @xcite . practical conversations often exhibit multimodal cues through facial expressions, vocal changes, and textual content. recognizing the significance of multimodal information, @xcite proposed the task of multimodal emotion-cause pair extraction in conversations (mecpe) as a critical step towards understanding the fundamental elicitors of emotions. the emotion-cause-in-figure 1 : an example of an annotated conversation from the ecf dataset. dashed lines connect each emotion label to its corresponding cause utterance, illustrating the emotion-cause pairs present in the conversation. the image modality provides additional context and cues for understanding the expressed emotions. friends (ecf) dataset, introduced in semeval 2024 task 3 @xcite , incorporates additional modalities such as images and audio alongside the original textual data, enabling a more realistic and comprehensive approach to emotion understanding. figure 1 illustrates an example of an annotated conversation in the ecf dataset, where variations in facial expressions directly mirror the emotions expressed by the characters. to address the mecpe task, we propose the multimodal emotion recognition-multimodal cause extraction (mer-mce) framework, building upon the two-step approach introduced by @xcite . our method adopts a two-stage process to predict emotions and identify emotion causes in multimodal conversations. in the first stage, mer-mce leverages text, audio, and image modalities for emotion prediction, utilizing state-of-the-art models specifically designed for capturing emotional cues from each modality. this approach sets our work apart from the first-place team in the competition, who relied solely on the textual modality for emotion recognition, and the second-place team, who employed a general-purpose model, im-agebind @xcite , for extracting visual and audio features. in the second stage, considering the complexity of analyzing emotion causes for each utterance, we employ a multimodal large language model (llm) to dissect the visual and textual modalities and discern the origins of each emotion. by leveraging the power of multimodal llms, our approach can effectively capture the intricate relationships and dependencies present in real-world conversations, enabling a more nuanced and accurate identification of emotion causes. the mer-mce framework achieved notable results in subtask 2 of semeval 2024 task 3, ranking third with a weighted f1 score of 0.3435, only 0.0339 behind the first-place team and 0.0025 behind the second-place team. we evaluate the two stages of our model separately to analyze their efficacy and limitations, providing valuable insights into the inherent challenges of the mecpe task. the main contributions of this paper are as follows: ## system overview in this work, we propose a multimodal emotion recognition and multimodal cause extraction (mer-mce) framework for the task of multimodal emotion cause prediction and extraction (mecpe) in conversational settings. as illustrated in figure 2 , our mer-mce model comprises two key modules: a multimodal emotion recognition (mer) module and a multimodal cause extraction (mce) module, designed to work in tandem to tackle the intricate challenge of identifying emotions and their underlying causes from multimodal conversational data. following we describe the structure of the entire system in detail. ## conclusion this paper introduces the mer-mce model for emotion cause analysis in conversations, developed for semeval 2024 task 3. our model leverages multimodal information and language models (llms) to identify emotion causes in conversational data, considering textual, visual, and audio modalities. mer-mce achieves a weighted f1 score of 0.3435, ranking third in the competition. the results demonstrate the effectiveness of multimodal approaches in capturing emotional dynamics. future work will focus on enhancing generalizability and robustness by exploring additional modalities and advanced techniques. we plan to investigate the incorporation of pose estimagesture recognition and facial expression analysis to improve the model's ability to detect emotions.