QFNU _ CS at S em E val-2024 Task 3: A Hybrid Pre-trained Model based Approach for Multimodal Emotion-Cause Pair Extraction Task (2024)

Abstract: This article presents the solution of Qufu Normal University for the Multimodal Sentiment Cause Analysis competition in SemEval2024 Task 3.The competition aims to extract emotion-cause pairs from dialogues containing text, audio, and video modalities. To cope with this task, we employ a hybrid pre-train model based approach. Specifically, we first extract and fusion features from dialogues based on BERT, BiLSTM, openSMILE and C3D. Then, we adopt BiLSTM and Transformer to extract the candidate emotion-cause pairs. Finally, we design a filter to identify the correct emotion-cause pairs. The evaluation results show that, we achieve a weighted average F1 score of 0.1786 and an F1 score of 0.1882 on CodaLab.

## introduction the competition of multimodal emotion cause analysis @xcite involves not only understanding linguistic content but also recognizing and comprehending various forms of information such as emotional expressions, sounds, and images. the significance of this competition lies in its ability to comprehensively understand and interpret emotions and motivations in human communication. by analyzing various forms of information in conversations, we can more accurately identify the sources and reasons for emotions, thereby enhancing our understanding of human behavior and communication methods. this holds importance across various fields including psychology, human-computer interaction, and affective computing, aiding in the development of more intelligent and human-centric technologies and systems, improving communication efficiency and quality, and promoting better understanding and communication among individuals. this paper details our contribution to semeval-2024 task 3: multimodal emotion cause analysis in conversations @xcite , encompassing two sub-tasks: extracting emotion-cause pairs @xcite ) from text-only dialogues and from multimodal dialogues that include text, audio, and video modalities. in this task, we place a particular emphasis on implementing sub-task 2. for sub-task 2: multimodal emotion-cause pair extraction, we aim to extract emotion-cause pairs from dialogues that contain representations in text, audio, and video. each pair includes an emotional utterance, its emotion category, and a cause utterance. the challenge lies in integrating information from multiple modalities to accurately identify emotional expressions and their related causes. in our approach to task 2, we first preprocessed the dataset, mapping the text, audio, and video data of the emotion cause in friends (ecf) dataset to a unified feature space. then, we utilized a baseline model with a two-stage training scheme: emotion recognition and cause pair extraction. this approach focused on utilizing modalities, selecting models such as bert @xcite and lstm @xcite , and adjusting parameters for two phases of model training. after that, we predicted on test data in two stages using the trained models and evaluated the results through codalab to obtain corresponding f1 scores. our best-performing solution involved using bert for emotion recognition followed by lstm for cause pair extraction across all three modalities, achieving an f1 score of 0.1882. this methodological progression demonstrates our systematic approach to tackling the complexities of multimodal emotion cause analysis, highlighting our efforts in dataset preprocessing, model experimentation, and performance evaluation, while also proving the effectiveness of the baseline model @xcite . ## methodology in this section, we describe the e-mecpe method in depth. in general, this method is divided into three main parts: multimodal fusion, emotion and cause extraction through multi-task learning and emotion-cause pairing and filtering. the methodology of this paper is summarized in fig. 1. ## conclusion in this paper, we present effective multimodal emotion-cause pair extraction (e-mecpe) method. we used this method to perform emotional cause analysis on the emotion-cause-in-friends (ecf) dataset. ablation experiments were conducted for text unimodal and multimodal under different text encoders, respectively, and the relevant parameters associated with the experiments were tuned. the experimental results show that bert encoding-based text representation and multimodal joint representation help in the extraction of emotional cause pairs, and that the parameter settings are crucial for the performance enhancement of this task. this finding not only validates the effectiveness of our method, but also points out an important direction for future research in the field of sentiment analysis kresearch by pointing out an important direction.