EMO - RL : Emotion-Rule-Based Reinforcement Learning Enhanced Audio-Language Model for Generalized Speech Emotion Recognition (2025)

Abstract: Although large audio-language models (LALMs) have demonstrated remarkable capabilities in audio perception, their performance in affective computing scenarios, particularly in emotion recognition, reasoning, and subtle sentiment differentiation, remains suboptimal. Recent advances in reinforcement learning (RL) have shown promise in improving LALMs’ reasoning abilities. However, two critical challenges hinder the direct application of RL techniques to speech emotion recognition (SER) tasks: (1) convergence instability caused by ambiguous emotional boundaries and (2) limited reasoning ability when using relatively small models (e.g., 7B-parameter architectures). To address these challenges, we propose EMO-RL, a novel framework incorporating reinforcement learning with two key innovations: Emotion Similarity-Weighted Reward (ESWR) and Explicit Structured Reasoning (ESR). Built upon pretrained LALMs, our method employs group-relative policy optimization with emotion constraints. Comprehensive experiments demonstrate that our EMO-RL training strategies can significantly enhance the emotional reasoning capabilities of LALMs, achieving state-of-the-art performance on the MELD and IEMOCAP datasets, and cross-dataset experiments prove the strong superiority of generalization.

## introduction speech emotion recognition (ser) is a significant research direction in the field of affective computing, aiming to map speech signals to corresponding emotional labels through computational analysis. it plays an important role in various applications, including intelligent customer service @xcite , mental health assessment @xcite , and human-computer interaction @xcite . ## conclusion this paper introduces emo-rl, a reinforcement learning framework that improves the emotionalreasoning capacity of large audio-language models for speech-emotion recognition. by incorporating emotion similarity-weighted reward, which integrates psychological prior knowledge into rl, and explicit structured reasoning into our framework, emo-rl effectively overcomes the challenges of convergence instability and limited reasoning ability in speech emotion recognition tasks. comprehensive experiments demonstrate that emo-rl not only improves the emotional reasoning capa-bilities of lalms on the meld and iemocap datasets (compared with sota, achieving an ua improvement of 25.1% and 18.9%, respectively), but also shows excellent generalization across different datasets. this work signifies a step forward in applying reinforcement learning and large audiolanguage models to speech emotion recognition, paving the way for future speech affective computing research. moreover, emo-rl shows potential for enhancing multi-modal llms' emotion perception, bringing us closer to building truly emotional llms.