GRPO -Guided Modality Selection Enhanced L o RA -Tuned LLM s for Multimodal Emotion Recognition (2025)

Abstract: Multimodal emotion recognition in conversation (MERC) aims to identify speakers’ emotional states by utilizing text, audio, and visual modalities. Although recent large language model (LLM)-based methods have demonstrated strong performance, they typically adopt static fusion strategies that integrate all available modalities uniformly. This overlooks the fact that the necessity of multimodal cues can vary significantly across utterances. In this work, we propose an adaptive modality selection framework for MERC. The core of our approach is a modality selection module based on Group Relative Policy Optimization (GRPO), which enables a LoRA-tuned LLM to reason about the necessity of multimodal input via chain-of-thought (CoT) generation. This process does not require manually labeled modality selection data and is trained in a fully unsupervised manner. The selected modality configuration is then provided as input to a downstream emotion classifier, which is also implemented using a LoRA-tuned LLM and trained to predict emotional states. Experimental results on benchmark multimodal dialogue datasets show that our method consistently outperforms strong baselines, demonstrating the effectiveness of adaptive modality selection in improving recognition accuracy. Our code is available at https://github.com/youflyaway/Modality-Selection-Enhanced-LoRA-Tuned-LLMs .

## introduction multimodal emotion recognition in conversations (merc) aims to identify the speaker's emotion by combining text, audio, and visual signals. existing studies @xcite @xcite have primarily focused on multimodal feature fusion and alignment, yielding better results than unimodal baselines. recently, large language models (llms) * corresponding author have achieved significant results in a variety of natural language processing tasks due to their powerful context modeling and reasoning capabilities. some researchers have applied llms to emotion recognition tasks. instructerc @xcite formulates emotion recognition as a generative task, introducing retrieval-based templates and emoticon alignment mechanisms to model speaker interactions. voiceerc @xcite transforms speech features into natural language descriptions in a way that enables multimodal emotion analysis without architectural changes. dia-loguellm @xcite implements merc using an end-to-end supervised instruction finetuning method by converting conversation videos into textual descriptions that will be used as complementary knowledge to construct high-quality instructions. similarly, textmi @xcite and @xcite converts audio and visual inputs into structured textual descriptions, effectively reducing model complexity while preserving task performance. although llm-based methods show advantages in multimodal emotion recognition tasks, they mainly use a fixed multimodal usage strategy, ignoring that different utterances may require different combinations of modalities. when there are semantic breaks or cryptic expressions in the conversation text, other modal cues in the historical conversation can effectively complement the key contextual information; while when the text contains sufficient emotion cues (e.g., explicit emotion words, intention indicators, or contextual markers), the introduction of multimodal features may cause the model to focus excessively on redundant information. as illustrated in figure 1a , joey's utterance in the friends dataset is emotionally ambiguous in text alone, but visual and acoustic cues from earlier turns help clarify the emotional state. conversely, in figure1b, the teacher's statement already provides sufficient semantic signals, allowing accurate joey visual audio text 𝑢1 𝑢2 𝑢3 𝑢4 𝑢5 𝑢6 push! [joy] push 'em out, push 'em out, harder, harder. push'em out, push'em out, way out! @xcite let's get that ball and really move, hey, hey, ho, ho. text audio visual 𝑢1 𝑢2 𝑢3 𝑢4 𝑢5 how would you characterize the theme of this book, uh let's see here , rachel green? [neutral] umm, well i would have to say that it's a, it's tragic love story. [neutral] well, that's sort of a given, but yes. anyone else? [neutral] oh-oh-oh, symbolism! and uh, the-the uh, wildness of the mores, which [joy] excellent! what rachel has shrewdly observed here [joy] recognition based solely on the textual modality. to address the limitations of fixed modality usage, we propose a merc framework centered on adaptive modality selection. the goal is to dynamically determine whether to incorporate multimodal information based on contextual needs, thereby improving modality utilization. the method adopts a two-stage structure: in the first stage, the model learns in an unsupervised setting whether the current utterance requires multimodal cues from the historical conversation. we introduce a reinforcement learning mechanism based on group relative policy optimization (grpo) @xcite to train a lora-tuned @xcite llm as a modality selector. instead of using an explicit value function, the model leverages the average reward from multiple outputs for the same input as a learning signal. in the second stage, the selected unimodal or multimodal inputs are used to construct prompts for lora-based fine-tuning of the llm on supervised emotion-labeled data, enabling the final emotion classification. the main contributions are as follows: ## method the proposed method follows a two-stage framework, as illustrated in figure 2 . first, the grpoguided modality selector determines whether multimodal cues from the historical context are necessary for the current utterance in an unsupervised manner. second, a lora-tuned llm performs emotion classification based on the selected input configuration (text-only or multimodal). prior to these two stages, we transform raw acoustic and visual inputs into structured textual descriptions, ensuring compatibility with the llm-based prompting paradigm. ## conclusion in this paper, we propose a novel two-stage framework for merc, which integrates structured multimodal descriptions and adaptively determines their necessity for each input. unlike prior works that apply fixed fusion strategies, our method selectively incorporates audio and visual cues based on contextual relevance, improving the robustness and interpretability of emotion recognition. experiments on two benchmark datasets demonstrate that our approach achieves competitive performance. moreover, the framework is simple and readily applicable to a wide range of dialogue understanding tasks.