A lign C ap: Aligning Speech Emotion Captioning to Human Preferences (2024)

Abstract: Speech Emotion Captioning (SEC) has gradually become an active research task. The emotional content conveyed through human speech are often complex, and classifying them into fixed categories may not be enough to fully capture speech emotions. Describing speech emotions through natural language may be a more effective approach. However, existing SEC methods often produce hallucinations and lose generalization on unseen speech. To overcome these problems, we propose AlignCap, which Aligning Speech Emotion Captioning to Human Preferences based on large language model (LLM) with two properties: 1) Speech-Text Alignment, which minimizing the divergence between the LLM’s response prediction distributions for speech and text inputs using knowledge distillation (KD) Regularization. 2) Human Preference Alignment, where we design Preference Optimization (PO) Regularization to eliminate factuality and faithfulness hallucinations. We also extract emotional clues as a prompt for enriching fine-grained information under KD-Regularization. Experiments demonstrate that AlignCap presents stronger performance to other state-of-the-art methods on Zero-shot SEC task.

## introduction the identification and description of speech emotions play a crucial role in improving communication efficiency. it also aids in understanding the speaker's intentions. previous work usually treats emotion acquisition as a classification task, such as speech emotion recognition (ser) @xcite @xcite , which assigns speech to different emotion categories based on the emotions such as sadness, anger, and happiness contained within the speech. however, there may be a mixture of emotions within one utterance, and classifying speech into a single emotion ## background and discussion the section describes the speech-text distribution gap of traditional sec methods. to explore this gap, we conducted preliminary experiments to analyze its potential impact on train-inference mismatch and performance degradation. furthermore, we discuss the impact of modal alignment position on downstream sec performance. ## conclusion we proposed aligncap, achieving speech-text alignment and human preference alignment. to minimize the distribution gap between llm's response to speech input and those to corresponding text inputs, we design kd-regularization to achieve speech-text alignment. additionally, we align emotion captions to human preference by po-regularization. this process eliminates the factuality and faithfulness hallucinations of align-cap on unseen speech. experiments demonstrate aligncap's superiority in both zero-shot and crossdomain scenarios.