JMI at S em E val 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models (2024)

Abstract: This paper presents our system development for SemEval-2024 Task 3: “The Competition of Multimodal Emotion Cause Analysis in Conversations”. Effectively capturing emotions in human conversations requires integrating multiple modalities such as text, audio, and video. However, the complexities of these diverse modalities pose challenges for developing an efficient multimodal emotion cause analysis (ECA) system. Our proposed approach addresses these challenges by a two-step framework. We adopt two different approaches in our implementation. In Approach 1, we employ instruction-tuning with two separate Llama 2 models for emotion and cause prediction. In Approach 2, we use GPT-4V for conversation-level video description and employ in-context learning with annotated conversation using GPT 3.5. Our system wins rank 4, and system ablation experiments demonstrate that our proposed solutions achieve significant performance gains.

## introduction emotion cause analysis (eca) is centered around the extraction of potential cause clauses or pairs of emotion clauses and cause clauses from human communication, enabling a deeper understanding of communication dynamics. by incorporating multimodal cues like visual scenes, facial expressions, and vocal intonation, it facilitates a comprehensive and technically robust analysis of the factors that trigger diverse emotional reactions @xcite @xcite . despite the considerable amount of research conducted using diverse audio, visual, and text modalities @xcite @xcite @xcite , there has been a noticeable † equal contribution gap in the exploration of multimodal eca in natural settings (human conversations). in this context, @xcite introduce multimodal emotion cause analysis in conversations (ecac) task and provide emotion-cause-in-friends (ecf) dataset, which incorporates text, audio, and video modalities. this task consists of two sub-tasks: textual emotion-cause pair extraction in conversations (subtask 1) and multimodal emotion cause analysis in conversations (subtask 2). a detailed description of these sub-tasks can be found in the task description paper @xcite . in our submission to subtask 2 of multimodal ecac, this paper presents two distinct approaches to address the ecac problem, giving competitive results. drawing inspiration from the effectiveness of llms in diverse downstream tasks @xcite @xcite , including emotion recognition, we propose two llm-based approaches that decompose the emotion-cause pair extraction process into two steps. the first step involves predicting the emotions of the utterances in the conversation. in the next step, we utilize these emotion labels to guide cause extraction. approach 1 involves instruction-tuning two separate llama 2 models for emotion and cause prediction, while approach 2 leverages the in-context learning (icl) capabilities @xcite of the gpt-3.5 model. additionally, we introduce an efficient technique using the gpt-4v model to extract conversation-level descriptions from video modality. during the evaluation, our team ranked 4th on the leaderboard competing against more than 25 teams with a weighted-f1 score of 0.2816. ## conclusion we tackled the multimodal ecac task with a two-step framework of recognizing emotions first and then predicting their causes using llms. we implemented two approaches: a llama-2 model which has been fine-tuned with instructions and a gpt model which solves the task by learning from demonstration examples in context. conversationlevel video captions were extracted to provide more context to llms. our second approach was our best submission for the task, placing us at rank 4 with our first approach being placed at rank 6. our results were under cost constraints and further investigation with larger llama-2 models and gpt-4 with more sophisticated icl approaches are a clear follow-up of our work. peixiang zhong, di wang, and chunyan miao. 2019. knowledge-enriched transformer for emotion detection in textual conversations. in proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pages 165-176.