Listen, Watch, and Learn to Feel: Retrieval-Augmented Emotion Reasoning for Compound Emotion Generation (2025)

Abstract: The ability to comprehend human emotion using multimodal large language models (MLLMs) is essential for advancing human-AI interaction and multimodal sentiment analysis. While psychology theory-based human annotations have contributed to multimodal emotion tasks, the subjective nature of emotional perception often leads to inconsistent annotations, limiting the robustness of current models. Addressing these challenges requires more fine-grained methods and evaluation frameworks. In this paper, we propose the Retrieval-Augmented Emotion Reasoning (RAER) framework, a plug-and-play module that enhances MLLMs’ ability to tackle compound and context-rich emotion tasks. To systematically evaluate model performance, we introduce the Stimulus-Armed Bandit (SAB) framework, designed to benchmark emotional reasoning capabilities. Additionally, we construct the Compound Emotion QA dataset, an AI-generated multimodal dataset aimed at strengthening emotion understanding in MLLMs. Experimental results demonstrate the effectiveness of RAER across both traditional benchmarks and SAB evaluations, highlighting its potential to enhance emotional intelligence in multimodal AI systems.

## introduction emotion is a multifaceted phenomenon encompassing subjective experiences, physiological responses, and context-dependent behaviors, shaped by both internal states and external stimuli. emotions play a critical role in human cognition and interaction, influencing decision-making, directing attention, and shaping social relationships. their complexity and impact underscore the significance of emotions as a core component of human experience and behavior. recent advancements in neural network methods have highlighted the effectiveness of special-* corresponding author. figure 1 : overall of our proposed method and evaluation framework, where the retrieval-augmented emotion reasoning (raer) method is introduced as a plugand-play module to enhance the capability of mllms in handling compound and ambiguous emotions. the stimulus-armed bandit (sab) evaluation framework is used to assess the model's emotional capabilities, especially for tasks that are difficult to quantify. ized models for emotion tasks. these models, which predict labels within a constrained range, have achieved impressive performance, particularly in tasks such as dynamic facial emotion recognition (dfer) @xcite @xcite and multimodal emotion recognition (mer) @xcite @xcite . however, widely adopted annotation standards within a constrained range-such as the "big six" discrete label system @xcite and the vad (valence-arousal-dominance) dimensional label system @xcite -have proven effective in capturing emotional expression, they may not fully align with the more nuanced emotional interactions required for ai systems, particularly in the era of large models that demand more human-like interaction for emotionrelated tasks. to address these limitations, ap-proaches based on multimodal large language models (mllms) have emerged @xcite . tasks such as multimodal empathetic response generation (merg) @xcite and other emotion-related tasks @xcite @xcite have shown strong performance on mllms, with excellent generalization capabilities. despite these advancements, significant challenges remain in handling compound and ambiguous emotions, especially in tasks involving compound and context-rich emotional scenarios. in this paper, as shown in figure 2 , we draw inspiration from recent advances in preference-based learning methods-such as reinforcement learning with human feedback (rlhf) @xcite , reinforcement learning from ai feedback (rlaif) @xcite , direct preference optimization (dpo) @xcite , and inverse preference optimization (ipo) @xcite -to explore a more fluid and subjective approach to evaluating emotion tasks. however, a major challenge in constructing emotion preference datasets lies in the labor-intensive process of manually drafting labels, which is not only timeconsuming but also prone to inconsistencies. these inconsistencies arise from both variations in linguistic descriptions and differences in human preferences, making it difficult to disentangle the specific preference signals required for ai model training @xcite . as shown in figure .1. instead of relying on manually curated label drafts and preference annotations, we propose a retrieval-augmented emotion reasoning (raer) framework, a rag-based module that integrates chain-of-thought (cot) reasoning. raer can be easily applied to mllms, enhancing their emotional reasoning and generalization capabilities to tackle compound emotional scenarios. to evaluate mllms' emotion task capabilities and gather human preferences, we introduce the stimulus-armed bandit (sab) evaluation framework, which uses ai-generated stimuli to test a broad range of emotion tasks. this approach not only collects human preferences but also benchmarks model performance in dynamic and compound emotional contexts. by combining raer-generated responses with sab-collected human preferences, we construct the compound emotion qa dataset, a multimodal dataset that captures nuanced emotional reasoning aligned with human preferences. this methodology bridges the emotions is inherently open-ended, suggesting that label systems with predefined boundaries are limited in the context of large models. to enhance the emotional capabilities of these models, it is essential to adopt more human-like, nuanced approaches that allow for a broader range of emotional expression. ## retrieval-augmented emotion reasoning emotional reasoning refers to the process of deriving conclusions based on emotional responses, even when empirical evidence may suggest otherwise. this concept has proven effective in large models for addressing compound emotion-related tasks @xcite , particularly those involving ambiguous or context-dependent emotional content. building on this foundation, as shown in figure 3 , we propose retrieval-augmented emotion reasoning (raer), a framework designed to enhance multimodal large language models (mllms) by integrating emotional reasoning into a structured chain-of-thought (cot) process @xcite . ## stimulus-armed bandit the stimulus-armed bandit (sab) framework is a novel evaluation method designed to assess the compound emotion capabilities of multimodal large language models (mllms). the name is inspired by the classic multi-armed bandit optimization problem, as emotion tasks similarly involve balancing between different emotional responses. sab introduces a preference-based ranking system that combines multimodal stimuli with emotion tasks. through pairwise comparisons, sab dynamically ranks models based on their emotional reasoning, while also collecting human preferences and corresponding task labels. the sab framework integrates three key components to enable comprehensive and scalable evaluation: ## experiments in this section, we present a comprehensive evaluation of the retrieval-augmented emotion reasoning (raer) framework. the experiments are conducted across two main categories: traditional datasets and metrics, encompassing various tasks in multimodal emotion recognition (mer) and multimodal empathetic response generation (merg), and the novel stimulus-armed bandit (sab) evaluation framework, which systematically evaluates both visual-language and audio-language generated samples. ## conclusion in this paper, we introduced the retrieval-augmented emotion reasoning (raer) framework, which enhances multimodal large language models (mllms) by combining emotional reasoning with retrieval-augmented processes. our experiments on standard benchmarks and the stimulus-armed bandit (sab) evaluation demonstrate raer's effectiveness in handling compound emotion tasks. we also contributed the compound emotion qa dataset, an ai generated dataset designed to further improve emotional reasoning. the results highlight raer's potential in advancing multimodal sentiment analysis and enhancing human-ai interaction.