U ni MEEC : Towards Unified Multimodal Emotion Recognition and Emotion Cause (2024)

Abstract: Multimodal emotion recognition in conversation (MERC) and multimodal emotion-cause pair extraction (MECPE) have recently garnered significant attention. Emotions are the expression of affect or feelings; responses to specific events, or situations – known as emotion causes. Both collectively explain the causality between human emotion and intents. However, existing works treat emotion recognition and emotion cause extraction as two individual problems, ignoring their natural causality. In this paper, we propose a Unified Multimodal Emotion recognition and Emotion-Cause analysis framework (UniMEEC) to explore the causality between emotion and emotion cause. Concretely, UniMEEC reformulates the MERC and MECPE tasks as mask prediction problems and unifies them with a causal prompt template. To differentiate the modal effects, UniMEEC proposes a multimodal causal prompt to probe the pre-trained knowledge specified to modality and implements cross-task and cross-modality interactions under task-oriented settings. Experiment results on four public benchmark datasets verify the model performance on MERC and MECPE tasks and achieve consistent improvements compared with the previous state-of-the-art methods.

## introduction recently, multimodal emotion recognition in conversations (merc) and multimodal emotion-cause pair extraction (mecpe) have attracted increasing attention @xcite . both task play crucial roles in dialog systems, especially in empathetic response generation in a conversation @xcite @xcite . merc detects the emotion category of each utterance in a conversation, while mecpe finds the reasons that trigger a certain emotion for the utterance. both tasks are tightly related in practice and theory (baumeister good job joe! well done! top notch! response event causality you liked it? you really liked it? emotion praised joe joy mecpe: merc: u 4 : u 3 : cause emotion cause figure 1 : illustration of the causal inference between emotion and emotion cause, which unifies mecpe and merc tasks. "response" denotes the speaker's reaction to the event and "event" denotes the event that triggers emotion. @xcite @xcite . however, the existing works treat merc and mecpe as two separate tasks and ignore their causality. on the one hand, emotions are responses to emotion causes (e.g., specific events) @xcite . on the other hand, emotion and its emotion causes are interdependent and mutually influential @xcite . the two serve as reflections for each other and together provide a causal story of human behavior and intents. figure 1 illustrates the causal alignment between emotion category and emotion cause @xcite . for example, the emotion causes of "happiness" generally are positive events, such as "being praised". similarly, the emotion causes of "sad" generally are negative events, such as "being criticized". we view the mapping between the specific events (e.g., emotion cause) and response (e.g., emotion label) as the emotion-cause causality. from the causal perspective, @xcite proposes the idea of causal prompts, which are prompts that describe the causal story behind the sentiment rating and reviews, further demonstrating that pretrained language model (plm) is able to be aware of the underlying causality. a natural question arises: how should we perform causality between emotions and their causes in a unified architecture? recently, the unification of related but different tasks into a framework has achieved significant progress @xcite @xcite . for example, unimse @xcite unifies emotion and sentiment into a single architecture to share complementary knowledge between them. different from unimse which focuses on the unification of emotion and sentiment in a generative way, we propose a multimodal causal prompt to unify merc and mecpe tasks, thereby capturing the causal nature between emotion and emotion cause. in this paper, we propose a unified multimodal emotion recognition and emotion-cause pair extraction framework (unimeec) to explore the causality between emotion and emotion cause. as @xcite illustrated, plm can capture the causal stories with the causal prompts. starting from this perspective, unimeec reformulates merc and mecpe as two mask prediction tasks and unifies the two tasks using a causal prompt, aiming to capture the understanding of plm to emotion-cause causlity. in order to differentiate the modal effects, unimeec probes modal features from plm using the multimodal causal prompt, and meanwhile, unimeec captures the emotion-specific, cause-specific, and utterance-specific contexts in a hierarchical way. the main contributions are summarized as follows: ## conclusion this paper presents a unified multimodal emotion recognition and emotion-cause analysis framework, which aims to explore the emotion-cause causality by jointly modeling multimodal emotion recognition and emotion-cause pair extraction. unimeec reformulates merc and mecpe tasks as two mask prediction problems, tunes plm via multimodal causal prompts specific to uni-modality, and aggregates task-specific context in a conversation. experiments on iemocap, meld, con-vecpe, and ecf consistently gain significant improvements on most metrics compared to the previous sota, further demonstrating the effectiveness of unimeec in addressing merc and mepce.