SSR : Alignment-Aware Modality Connector for Speech Language Models (2025)

Abstract: Fusing speech into a pre-trained language model (SpeechLM) usually suffers from the inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR outperforms existing mechanisms for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.

## introduction large language models @xcite @xcite @xcite @xcite have demonstrated remarkable performance across various tasks and extending pretrained abilities from llms to other modalities has sparked interest in multimodal llms @xcite @xcite @xcite . in this work, we focus on integrating speech into pretrained language models (speechlms). a straightforward approach is to transcribe speech into text and use these transcriptions as prompts for large language models @xcite ; however, such cascaded systems suffer from error propagation, higher latency, and cannot leverage raw speech information like emotion, speaker identity, and other paralinguistic cues @xcite @xcite . con-* work was done during an internship at meta ai. sequently, developing end-to-end speechlms that directly fuse speech or audio input has gained popularity, where various approaches have been explored to encode speech and align its representation with pre-trained language models @xcite @xcite @xcite @xcite . ## methodology we develop an alignment-aware speech representation connector to foster modality fusion between speech and pre-trained language model. we introduce our connector design in §3.1 and present our two-stage training pipeline in §3.2. ## stage 2: speech language model fine-tuning in stage 1 ( §4), we freeze the pre-trained llm and distill its text embeddings into our alignment-aware connector. in this section, we fine-tune speechlm by freezing the connector and updating the llm. this process enhances the model's spoken language understanding (slu) performance by fitting speechlm on the aligned speech-text data, albeit at the expense of degrading its pre-trained text capabilities. in the following sections, we compare various methods to mitigate catastrophic forgetting and demonstrate their trade-offs between speech and text understanding. ## conclusion we propose ssr-connector to inject speech representation into pre-trained llms. through explicitly leveraging speech-text alignment, our connector compresses long and sparse speech information to the same granularity as text tokens. with our proposed two-stage training pipeline for modality fusion, ssr-connector-based speechlm achieves better speech understanding while retaining its pre-trained text ability.