NUS -Emo at S em E val-2024 Task 3: Instruction-Tuning LLM for Multimodal Emotion-Cause Analysis in Conversations (2024)

Abstract: This paper describes the architecture of our system developed for participation in Task 3 of SemEval-2024: Multimodal Emotion-Cause Analysis in Conversations. Our project targets the challenges of subtask 2, dedicated to Multimodal Emotion-Cause Pair Extraction with Emotion Category (MECPE-Cat), and constructs a dual-component system tailored to the unique challenges of this task. We divide the task into two subtasks: emotion recognition in conversation (ERC) and emotion-cause pair extraction (ECPE). To address these subtasks, we capitalize on the abilities of Large Language Models (LLMs), which have consistently demonstrated state-of-the-art performance across various natural language processing tasks and domains. Most importantly, we design an approach of emotion-cause-aware instruction-tuning for LLMs, to enhance the perception of the emotions with their corresponding causal rationales. Our method enables us to adeptly navigate the complexities of MECPE-Cat, achieving an average 34.71% F1 score of the task, and securing the 2nd rank on the leaderboard. The code and metadata to reproduce our experiments are all made publicly available.

## introduction emotion cause analysis is a critical component of human communication and decision-making, offering substantial applications across diverse fields. it enables a deeper and more detailed understanding of sentiments. the introduction of emotioncause analysis in textual conversations by @xcite ; @xcite has paved the way for advancements in understanding emotional dynamics within dialogues. however, textual analysis alone does not fully capture the complexity of human emotional expression, as emotions and their causes are often conveyed through a blend of modalities @xcite @xcite . subtask 2 of semeval-2024 task 3, referred to as mecpe-cat, seeks to expand this analysis into the multimodal domain, focusing on english-language conversations. the task draws inspiration from the seminal work of @xcite , which sets out to jointly extract emotions and their corresponding causes from conversations across multiple modalities, including text, audio, and video, and it also encompasses the identification of the corresponding emotion category for each emotion-cause pair. in our system, we leverage llms such as gpt-3 @xcite , flan-t5 @xcite , and glm @xcite known for their exceptional performance in various natural language processing tasks. we employ parameter-efficient fine-tuning, specifically lora @xcite , to efficiently fine-tune llms, enhancing their performance with minimal computational overhead. additionally, we harness emotion-cause-aware prompt-based learning and instruction-tuning to enhance model performance such that the llms can more accurately perceive the emotions with their corresponding causal rationales. prompt-based learning guides llms to generate contextually relevant outputs, while instruction-fine-tuning models for our specific tasks by improving their response to explicit instructions. in this paper, we investigate the optimal llm for the mecpe-cat task, selecting chatglm based on its superior zero-shot performance. we further refine chatglm through instruction-tuning, using carefully crafted prompts to enhance its taskspecific accuracy. our fine-tuned model achieves the second-highest score on the official test set for subtask 2, with a weighted average of 34.71% f1, figure 1 : an example of an official task and annotated dataset. each arc points from the cause utterance to the emotional triggers. the cause spans have been highlighted in yellow. background: chandler and his girlfriend monica walked into the casino (they had a quarrel earlier but made up soon), and then started a conversation with phoebe. ## methodology in this section, we first conduct preliminary experiments to determine which llm to select as a backbone reasoner. we then elaborate on how we design the system and emotion-cause-aware instructions for tuning our chosen llm. ## experiments this section will quantify the effectiveness of our systems via experiments and also show more analyses to gain more observations. ## conclusion in this work, we explore the llms for solving the multimodal emotion-cause pair extraction with emotion category (mecpe-cat) task. through a pilot study, we first select an llm, chatglm, that assists in achieving optimal task performance. the backbone chatglm receives textual dialogue, and also perceives the multimodal information via the imagebind vision encoder. lastly, we devise an emotion-cause-aware instruction-tuning mechanism for updating llms, which enhances the perception of the emotions with their corresponding causal rationales. our system achieves a weighted average f1 score of 34.71%, securing second place on the mecpe-cat leaderboard.