UWBA at S em E val-2024 Task 3: Dialogue Representation and Multimodal Fusion for Emotion Cause Analysis (2024)

Abstract: In this paper, we present an approach for solving SemEval-2024 Task 3: The Competition of Multimodal Emotion Cause Analysis in Conversations. The task includes two subtasks that focus on emotion-cause pair extraction using text, video, and audio modalities. Our approach is composed of encoding all modalities (MFCC and Wav2Vec for audio, 3D-CNN for video, and transformer-based models for text) and combining them in an utterance-level fusion module. The model is then optimized for link and emotion prediction simultaneously. Our approach achieved 6th place in both subtasks. The full leaderboard can be found at https://codalab.lisn.upsaclay.fr/competitions/16141#results

## introduction the semeval-2024 task 3: the competition of multimodal emotion cause analysis in conversations @xcite is aimed at extracting emotion-cause pairs (ecps) in conversations. the main data source to tackle this task is recordings from the sitcom friends in the english language -emotion-cause-in-friends dataset @xcite . the detection of emotions and a deeper analysis of what causes them is one of the interesting and important tasks that have recently been tackled within the nlp community. previously, researchers focused their efforts on text-only emotion-cause extraction @xcite @xcite . however, representing dialogues solely through the text (speech transcription) is not entirely adequate, as people use different intonations and other prosodic features. moreover, the fact that something happens during the conversation (e.g. someone walks in or something breaks) affects the dialogue in terms of emotions and their causes. we can also mention different facial expressions for different emotions. so, the fact that a conversation in its natural form is multimodal (text, audio and video) opens a big space for research. our approach is targeted at both subtasks of the above-mentioned semeval 2023 task. the main difference between them is the number of different modalities used for predicting ecps. for subtask 1, the prediction of ecps is solely based on the text transcription without recordings. the goal is to provide text spans along with ecps: i.e. the segment of an utterance primarily responsible for emotion-cause. subtask 2, does not require extracted text spans (in some cases it is impossible because the emotion-cause is not expressed in the textual form), but it is required to use other modalities embedded in available mp4 video files to extract emotion-cause pairs together with a target emotion. our approach uses all modalities and encodes them into a common utterance-level representation, which is then used for link and emotion prediction. the objective is to learn both prediction tasks with two loss functions that are combined. the main idea behind this is that emotions might positively influence the links (pairs) and vice versa. another point we would like to highlight is that the textual input is a whole dialogue, so the context is taken into consideration. this improves the link results significantly, as shown further. after validating our codes by the semeeval task organizers, the final implementation will be released foot_0 . ## task and background the goal of the emotion-cause pair extraction (ecpe) task @xcite is to extract potential pairs of emotions and corresponding causes in a conversation/document and/or other source of dialogue. it is an extension of the emotion-cause extraction -ece task @xcite , where the goal is to decide if a clause/utterance is the corresponding cause, given the annotation of emotions. this semeval task does not require only the extraction of corresponding pairs but also the prediction of emotions. in other words, extracted pairs must be complemented by the prediction of the target utterance emotion. so, for the evaluation, we have a triplet (a source utterance, a target utterance, an emotion of the target utterance). e.g., 3_joy, 2 which means that the emotion joy in utterance 3 is caused by utterance 2. our team participated in both competition subtasks. we aimed to extract not only the pairs (as illustrated in the previous example) but also text spans -the exact parts of utterances/clauses primarily responsible for the emotion-cause (e.g. 3_joy, 2_you made up! -meaning that the emotion joy in utterance 3 is caused by the text span you made up! in utterance 2) hence, we work with all input data (i.e., multimodal -text, sound, image sequence/video) of the dataset emotion-cause-in-friends that serves as the competition dataset. @xcite presented a text-based approach for the ecpe task. they created a rule-based system and tested it on a chinese dataset created from the sinica corpus. ## system overview we decomposed the main objective into emotion and link prediction (the estimation of pairs) tasks. the final result then consists of source and target utterances provided by the link and emotion of the target utterance. the architecture is depicted in figure 1 . first, we encode the different modalities at the utterance or dialogue level to incorporate more context. next, we fuse the representations at the utterance level. once we have representations of all individual utterances in a dialogue, we predict links and emotions (subtask 2). based on this output, we employ our separate model for text span prediction, which is necessary for subtask 1. we have followed the competition rules and used pre-trained language models (plms), but no additional training data. text audio video modality-dependent text encoder video encoder audio encoder utterance level fusion link prediction emotion prediction ## experimental set-up in the data provided, there is a mapping of video names on train/dev/test splits. according to this information, we dedicated 9,966 utterances (1,373 dialogues) for training, and the remaining dialogues/utterances have been used as development (validation) data. our preliminary experiments, as well as the experiments resulting in the final system, have been evaluated on this development part of the dataset. due to the memory limitations, we fine-tuned audio and video encoders separately. after that, the whole pipeline is trained end-to-end with frozen audio/video encoders and, therefore, constant audio/video representations. we made this choice based on the preliminary experiments, where we obtained the best results with textual modality so we take audio/video features as an auxiliary input. if not stated differently, we used the adamw optimizer with a learning rate of 5e-5. categorical and binary cross-entropy loss was used for emotion and link prediction, respectively. we started with 2 "warmup" epochs with frozen encoders to limit "forgetting" of the pre-trained knowledge. further, we continued with 50+ epochs until convergence. to increase the importance of positive links, the positive link weight is set to 5. the fusion size is set to 1024 or 1536. the batch size ranges from 2 to 24. due to memory limitations, we adapted gradient accumulation technique. the number of samples used for weight update is 8, 12, or 24. these hyperparameters are further studied in appendix a. in tables 1 , 2 , and 3, we report the combination of these hyper-parameters that obtained the best result among runs. ## results in this section, we present and analyse the results from multiple perspectives. ## conclusion we have participated in two tasks in the semeval-2024 task 3: the competition of multimodal emotion cause analysis in conversations and obtained 6th place overall. our model incorporates all modalities (text, audio and video features). the main information lies within the text since the models based solely on textual modality are consistently among the best ones (see tables 1 2 3 ). we proposed and implemented several strategies for the fusion of modalities at the utterance level. to benefit more from video features, we mean that better preprocessing might be helpful (e.g., detecting a main speaker and focusing on her/his face). a possible bottleneck is in the fixed representation of the audio/video features. optimizing them during the learning process might improve their positive impact and, subsequently, the overall task success rate. moreover, we can benefit from the usage of a bigger model. our experiments have shown that the one multitask model may not be ideal since optimal hyperparameters differ for link prediction and emotion detection tasks. therefore, our final submission is the composition of two models. we have provided a good starting point and a set of analyses for further research. as a future work, one of our ideas is to use a single learning objective. in such a model, it would not be necessary to have an emotion module since everything would be managed by the link module with a multi-head self-attention matrix where each head would represent a link of one emotion. the training objective should be simpler since it uses single-label classification across components of attention matrices of individual heads. in this way, we can prevent the prediction of neutral links.