When Large Language Models Meet Speech: A Survey on Integration Approaches (2025)

Abstract: Recent advancements in large language models (LLMs) have spurred interest in expanding their application beyond text-based tasks. A large number of studies have explored integrating other modalities with LLMs, notably speech modality, which is naturally related to text. This paper surveys the integration of speech with LLMs, categorizing the methodologies into three primary approaches: text-based, latent-representation-based, and audio-token-based integration. We also demonstrate how these methods are applied across various speech-related applications and highlight the challenges in this field to offer inspiration for future research.

## introduction in recent years, the field of natural language processing (nlp) has been greatly reshaped by the development of large language models (llms) @xcite @xcite @xcite . these models have not only shown excellent ability in understanding and generating text but have also sparked interest in their potential applicability across other modalities, including speech. the integration of speech and llms offers a wide range of potential applications, including speech translation, conversational chatbots, and enhanced human-computer interaction in robotics. survey papers have reviewed speech language models @xcite , as well as audio language models @xcite and multimodal language models @xcite . however, there still lacks a survey specifically on the integration approaches of speech and llms, posing a challenge for researchers seeking to address this complex problem. distinct from other survey papers regarding speech and llms, this paper provides insight into इनपु ट… (speech output) ## text-based integration this approach primarily utilizes text as both input and output for llms. one of the most direct implementations of this method is through cascaded integration, which is highly adaptable to various tasks. within tasks like asr, llms are not limited to working with the final recognition results but can also operate on intermediate hypotheses. this allows for sophisticated operations like llm rescoring, and llm ger (generative error correction). ## latent-representation-based integration in this integration approach, a speech encoder is used to process the speech input and generate latent representations that are directly fed into the llm, bypassing the embedding layer. the speech encoder is usually a transformer-based model, which can be pre-trained on large-scale speech data or trained from scratch for the speech-llm integration. a critical issue of this approach is the sequence length gap between the speech and text modalities. speech features, often sampled at rates of 50 to 100 frames per second, result in longer sequences compared to text tokens. consequently, some form of modality adaptation mechanism is necessary to bridge these modalities, ensuring that the latent representations align with the llm's embedding space. ## audio-token-based integration as explained in section 2.2, there are two types of audio tokens: semantic tokens and acoustic tokens. studies use either or both, treating them as independent tokens like text or converting them back to latent representations. when semantic tokens are used for the output, a separate model to con-vert them into filterbanks and to waveform signals similar to tts models is often used. @xcite introduced the concept of generative spoken language modeling, which uses semantic tokens as both the input and output of a language model. @xcite and @xcite extended the idea by incorporating prosodic information. these models only used audio, and although they could output temporarily coherent contents, semantic understanding ability, and audio quality remained as challenges. ## challenges 7.1 text-based integration text-based integration preserves the original llm input modality, which is text. because the model is already optimized for textual inputs, this approach typically requires the least adaptation from the llm. however, transforming speech into text before feeding it into the llm inevitably introduces a layer of abstraction and potential information loss such as prosody and emotion, which limits the downstream performance. therefore, the main challenge is how to convey the rich information of source speech through text. in current practice, the intermediate text is often generated based on the highest probability @xcite @xcite . although this strategy maximizes intermediate accuracy, it may not be optimal for the llm and its final outputs. one possible direction is to inject controlled randomness during decoding to ensure additional diversity. however, degrading accuracy could also have a negative effect, creating a trade-off. how to balance diversity and accuracy remains an open research question that requires future investigation. ## conclusion in this survey, we systematically categorize the integration of llms with speech modality, outlining three primary approaches: text-based, latentrepresentation-based, and audio-token-based integration. each method demonstrates unique strengths in performing different speech-related tasks. however, numerous challenges remain in this evolving field, presenting significant opportunities for future research.