L ego SLM : Connecting LLM with Speech Encoder using CTC Posteriors (2025)

Abstract: Recently, large-scale pre-trained speech encoders and Large Language Models (LLMs) have been released, which show state-of-the-art performance on a range of spoken language processing tasks, including Automatic Speech Recognition (ASR). To effectively combine both models for better performance, continuous speech prompts and ASR error correction have been adopted. However, these methods are prone to suboptimal performance or are inflexible. In this paper, we propose a new paradigm, LegoSLM, that bridges speech encoders and LLMs using the ASR posterior matrices. The speech encoder is trained to generate Connectionist Temporal Classification (CTC) posteriors over the LLM vocabulary, which are used to reconstruct pseudo-audio embeddings by computing a weighted sum of the LLM input embeddings. These embeddings are concatenated with text embeddings in the LLM input space. Using the well-performing USM and Gemma models as an example, we demonstrate that our proposed LegoSLM method yields good performance on both ASR and speech translation tasks. By connecting USM with Gemma models, we can get an average of 49% WER reduction (WERR) over the USM-CTC baseline on 8 MLS testsets. The trained model also exhibits modularity in a range of settings – after fine-tuning the Gemma model weights, the speech encoder can be switched and combined with the LLM in a zero-shot fashion. Additionally, we propose to control the decode-time influence of the USM and LLM using a softmax temperature, which shows effectiveness in domain adaptation.

## introduction with the advancement of self-supervised and semisupervised learning, large-scale pre-trained speech and text models have been released in recent years @xcite . today, speech encoders * work done during internship at google. are pre-trained on extensive datasets that cover a wide range of spoken languages @xcite @xcite . these models have achieved state-of-the-art performance in various spoken language processing tasks, including automatic speech recognition (asr) and automatic speech translation (ast). in the field of natural language processing (nlp), large language models (llms) aim to capture general world knowledge in the network parameters through the task of next-word prediction @xcite . after being pre-trained on vast text corpora, llms have demonstrated remarkable capabilities in complex language understanding tasks facilitated by prompt engineering. to enhance the performance of spoken language processing tasks, several studies have focused on integrating speech encoders with llms. in asr error correction (aec), a cascaded system is built where the decoded hypotheses from the asr system are given as input to the llms for correction @xcite . such a method does not require deep access to the asr system and has the advantage of being modular. previous work has shown that an llm trained on the outputs from one asr encoder can be reused to correct the outputs of other speech encoders without any retraining @xcite . however, the aec performance is constrained by the limited contextual information accessible to llm @xcite . another popular approach to equipping llms with speech processing ability is to prompt llms with vectors transformed from the speech encoder output @xcite @xcite . usually, a mapping network is inserted and trained to match the embedding space of the speech and text modalities @xcite . by directly passing the continuous encoder outputs, this approach largely mitigates the information loss encountered in cascaded aec systems. while speech prompts demonstrate strong performance on various datasets, this approach sacrifices some flexibility. the llm is bonded with a specific speech encoder and fails to perform the task when prompted with outputs from a different speech encoder. in this paper, we present legoslm, a new paradigm to bridge pre-trained speech encoders and llms, as shown with the red line in figure 1 . first, the pre-trained speech encoder is fine-tuned with connectionist temporal classification (ctc) loss using the same vocabulary as the llm. afterward, we multiply the output ctc posteriors with the llm embedding table to reproduce pseudospeech embeddings as the llm input. compared to traditional aec methods, where the asr hypotheses are decoded and passed, much more information is preserved with our proposed approach. at the same time, our approach maintains flexibility. the ctc posteriors, as opposed to the continuous asr outputs, are acquired, which helps to protect the speaker's privacy. moreover, after fine-tuning the llm weights, new speech encoders can be plugged in a zero-shot fashion. using the usm @xcite and gemma @xcite models as an example, we design comprehensive experiments to study the system performance on asr and ast tasks. experiments on asr demonstrate that legoslm outperforms aec and achieves performance comparable to the speech prompt method, where the encoder weights are kept frozen. on ast, the proposed legoslm model shows improved performance over all baseline systems. asr error correction aec is a widely used post-processing approach to enhance the overall performance of an asr system. it takes the asr hypotheses as input and is trained on the reference text to automatically detect and correct recognition errors @xcite . since it only requires decoding hypotheses from the asr system, aec can be applied to api-based services without requiring in-depth access @xcite . several studies leverage asr n-best lists as input instead of using the top-1 hypothesis, as these provide richer information and have been shown to enhance the system performance @xcite . recent works build aec models using powerful llms to leverage their superior language understanding and reasoning capabilities @xcite @xcite . ## conclusions in this work, we propose a novel approach to combine a pre-trained speech encoder and llm. extensive experimental results show that legoslm achieves competitive performance compared to prior approaches. since ctc posteriors are used to bridge the two modules, after training the llm, the asr encoder can be switched in a zero-shot manner. additionally, we propose using the temperature value in softmax to adjust the relative emphasis placed on the speech encoder and the llm components. furthermore, several model variants are introduced and evaluated. the results presented in this paper indicate that legoslm has potential for broader applications, such as speech summarization and spoken language understanding.