Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation (2025)

Abstract: We present an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. We explore multiple visual encoders and multimodal fusion strategies during pre-training to identify the most effective integration approach. Subsequent fine-tuning on emotion recognition and expressive dialogue tasks yields substantial gains over speech-only baselines (e.g., +5 F1 in emotion recognition). AVLM highlights the value of expressive visual information in guiding speech generation and offers a foundation for end-to-end multimodal conversational systems.

## introduction expressive speech generation are supported by high-quality speech tokenization/codec @xcite @xcite , emotionally-nuanced speech datasets @xcite @xcite , and effective emotionaware representation learning @xcite . while recent advances in speech modeling have greatly improved the quality and controllability of synthesized speech, most methods rely solely on audio input and overlook the rich expressive cues available in the visual modality. human communication, however, is inherently multimodal. visual signals-such as facial expressions, head gestures, and eye movements-are tightly intertwined with vocal expression, offering critical paralinguistic information that reflects emotion and intent. this raises a central question: can we enhance expressive speech generation by incorporating these visual cues, enabling the model to produce responses that are not only fluent but also emotionally resonant and contextually adaptive? to address this question, we propose developing an emotion-aware audio-visual perceptual model for expressive speech generation. although prior work has explored lip reading, audio-visual speech recognition (avsr), and emotion recognition @xcite @xcite @xcite , these efforts are often task-specific. avsr methods, for example, improve transcription in noisy settings by focusing on the lip region, but they generally ignore non-semantic yet affective visual signals. similarly, emotion recognition benchmarks like iemocap @xcite are designed for classification rather than generation, limiting their utility in expressive synthesis tasks. consequently, current systems fall short in capturing the full emotional dynamics present in audio-visual communication. in this work, we propose an audio-visual language model (avlm) that integrates full-face visual cues into a pre-trained expressive speech language model (speechlm) to support emotionally rich speech generation. our training follows a twostage framework. first, we introduce a modality fusion module to align visual representations with speechlm's latent space, systematically evaluating visual encoders and fusion strategies during pre-training. second, we fine-tune the avlm for emotion recognition and dialogue generation using synthetic data derived from iemocap. to construct expressive conversations, we rewrite dialogue responses using @xcite and synthesize audio with @xcite , an expressive tts system supporting voice cloning and natural language instruction. our approach effectively incorporates a wider range of visual information beyond lip movements. it achieves lower perplexity during pre-training and improves avsr performance by reducing word error rate (wer) over 1 point under clean and noisy conditions. when fine-tuned for expressive speech generation and emotion recognition, our visually enhanced avlm consistently outperforms the speech-only counterpart (by more than 5 f1score), demonstrating higher classification accuracy and generating more expressive outputs. in summary, we strengthen the connection between visual and audio modality for expressive generation and show that our novel avlm could be a strong foundation for building emotionally intelligent endto-end conversational agents. ## motivate the integration of visual cues: audio-visual emotion recognition before diving into expressive audio-visual language modeling, we first motivate the study by showing why visual cues, besides its semantic correlation to speech in lip-reading, could be useful. we choose the emotion recognition benchmark, iemocap @xcite , and compare the performance of existing audio-only baselines and our simple audio-visual classification model. iemocap is made up of videos of two speakers' conversation with strong emotions. we follow prior benchmark emobox @xcite to train and evaluate models with their released data splits foot_1 . we follow the findings of emobox to use whisper @xcite as the audio encoder. for visual information, we directly encode frames with pre-trained open-magvit2 @xcite ) encoder foot_2 , which has achieved impressive image and video reconstruction performance. model (modality) ua (%) ↑ wa (%) ↑ f1 (%)↑ emobox (speech) 73.5 72.9 73.1 sensevoice-l (speech) 73.9 75.3 73.2 ours (visual) 77.9 79.1 78.7 ours (speech+visual) 88.2 89.2 88.5 to train our classifier, we encode visual and speech input into features, adapt them through lightweight encoders, and feed them into a transformer-encoder model @xcite . the features are then pooled and passed into a feedforward network to predict the emotion label. for details of our model setup, please refer to appendix b. as shown in we perform an ablation study by grouping samples based on the prediction outcomes of each modality. as shown in these findings indicate that the visual modality provides complementary non-semantic information. effectively leveraging such visual signals holds promise for enhancing the expressiveness and emotional alignment of speech generation systems. ## audio-visual language modeling to support expressive speech generation, we propose audio-visual language modeling and explore modality fusion strategies in §4.1. we then discuss our fine-tuning strategy for emotion recognition and expressive speech generation in §4.2. ## experiments we begin with avlm pre-training experiments ( §5.1), comparing different visual encoders and modality fusion strategies. next, we fine-tune the avlm on avsr tasks ( §5.2) to validate the ef-fectiveness of our pre-training. finally, we finetune avlm for emotion recognition and expressive speech generation ( §5.3), achieving superior performance over speechlm by utilizing visual cues. ## conclusion we present a framework for audio-visual language modeling that effectively incorporates visual cues into speechlm. following empirical exploration of visual encoders and modeling architectures, we employ a prefix-based query-former with smirk visual features for avlm. our system, through leveraging visual modality, outperforms speechonly baselines in both audio-visual speech recognition and expressive speech generation.