S em E val-2024 Task 3: Multimodal Emotion Cause Analysis in Conversations (2024)

Abstract: The ability to understand emotions is an essential component of human-like artificial intelligence, as emotions greatly influence human cognition, decision making, and social interactions. In addition to emotion recognition in conversations, the task of identifying the potential causes behind an individual’s emotional state in conversations, is of great importance in many application scenarios. We organize SemEval-2024 Task 3, named Multimodal Emotion Cause Analysis in Conversations, which aims at extracting all pairs of emotions and their corresponding causes from conversations. Under different modality settings, it consists of two subtasks: Textual Emotion-Cause Pair Extraction in Conversations (TECPE) and Multimodal Emotion-Cause Pair Extraction in Conversations (MECPE). The shared task has attracted 143 registrations and 216 successful submissions.In this paper, we introduce the task, dataset and evaluation settings, summarize the systems of the top teams, and discuss the findings of the participants.

## introduction understanding emotions is crucial to achieve human-like artificial intelligence, as emotions are intrinsic to humans and significantly influence our cognition, decision-making, and social interactions. conversation is an important form of human communication and contains a large number of emotions. furthermore, given that conversation in its natural form is multimodal, many studies have explored multimodal emotion recognition in conversations (erc), using language, audio and vision modalities @xcite @xcite @xcite . however, emotion recognition alone is not sufficient to fully understand the intricacies of hu-man emotions. emotion cause analysis (eca), the process of identifying the potential causes behind an individual's emotion state, has broad application scenarios such as human-computer interaction, commerce customer service, empathetic conversational agents, and automatic psychotherapy. for example, conversational agents equipped with emotion cause analysis can better understand the user's emotional state, offer empathetic responses, and provide more personalized services. by identifying the cause of the emotional state of a patient, a psychotherapy system can provide more accurate and customized treatments. eca has gained increasing attention both in academic and practical fields @xcite @xcite @xcite @xcite . however, to our knowledge, there has not been any evaluation competition conducted specifically for emotion cause analysis in conversations. to promote research in this direction, we organize a shared task in semeval-2024, named multimodal emotion cause analysis in conversations. our task consists of two subtasks: subtask 1 (textual emotion-cause pair extraction in conversations, tecpe) focuses on extracting emotion and textual cause spans solely based on text; subtask 2 (multimodal emotion-cause pair extraction in conversations, mecpe) involves extracting emotion-cause pairs at the utterance level considering three modalities. for this shared task, we provide a multimodal emotion cause dataset ecf 2.0 sourced from the sitcom friends. this dataset contains 1,715 conversations and 16,720 utterances, where 12,256 emotion-cause pairs are annotated at the utterance level, covering three modalities (language, audio, and vision). specifically, in our preliminary work @xcite , we have constructed a benchmark dataset, emotion-cause-in-friends (ecf 1.0), which contains 1,374 conversations and 13,619 utterances. on this basis, we have furthermore annotated an extended test set as the evaluation data and provided the span-level annotations of emotion causes within the textual modality. our task has attracted 143 registrations and a total of 216 successful submissions during the 16-day evaluation phase. participants tended to decompose our task into emotion recognition and cause prediction, proposing numerous well-designed pipeline systems. moreover, many teams applied advanced large language models (llms) for emotion cause analysis and achieved promising results. after the evaluation, 18 teams finally submitted system description papers. ## task we clarify the definitions of emotion and cause before introducing the task and dataset. emotion is a psychological state associated with thought, feeling, and behavioral response @xcite . in computer science, emotions are often described as discrete emotion categories, such as ekman's six basic emotions, including anger, disgust, fear, joy, sadness and surprise @xcite . in conversations, emotions were usually annotated at the utterance level @xcite @xcite . cause refers to the objective event or subjective argument that triggers the corresponding emotion @xcite . the goal of our shared task, named multimodal emotion cause analysis in conversations, is to extract potential pairs of emotions and their corre-sponding causes from a given conversation. figure 1 illustrates a typical multimodal conversation scenario, which involves multiple emotions and their corresponding causes. under different modality settings, we define the following two subtasks: ## evaluation our semeval task runs on codalab foot_1 . we released the training data in september 2023, and notified participants to commence model development. the evaluation phase began on january 16, 2024, and ended on january 31, 2024. we mixed the extended test set (consisting of 341 conversations with emotion and cause annotations; the labels are not publicly available) with some noise data (containing 324 conversations, not intended for evaluation) and released them together. each team is allowed to submit their results up to three times a day. ## conclusions in this paper, we describe the semeval-2024 task 3 named multimodal emotion cause analysis in conversations, which aims to extract all potential pairs of emotions and their corresponding causes from a conversation. the shared task has attracted 143 registrations and 216 successful submissions. we provide detailed descriptions of task definition and data annotation, summarize participating systems, and discuss their findings. as an important direction of affective computing, multimodal emotion cause analysis in conversation plays an important role in many real-world applications. we hope that our research and resources can contribute towards the design of future systems in this direction.