LLMV o X : Autoregressive Streaming Text-to-Speech Model for Any LLM (2025)

Abstract: Recent advancements in speech-to-speech dialogue systems leverage LLMs for multimodal interactions, yet they remain hindered by fine-tuning requirements, high computational overhead, and text-speech misalignment. Existing speech-enabled LLMs often degrade conversational quality by modifying the LLM, thereby compromising its linguistic capabilities. In contrast, we propose LLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS system that generates high-quality speech with low latency, while fully preserving the capabilities of the base LLM. Our approach achieves a significantly lower Word Error Rate compared to speech-enabled LLMs, while operating at comparable latency. By decoupling speech synthesis from LLM processing via a multi-queue token streaming system, LLMVoX enables seamless, infinite-length dialogues. Its plug-and-play design also facilitates extension to various tasks with different backbones. Furthermore, LLMVoX generalizes to new languages with minimal dataset adaptation, attaining a low Character Error Rate on an Arabic speech task. Evaluations demonstrate that LLMVoX matches or surpasses existing speech-enabled LLMs in both speech quality and latency, while maintaining the original linguistic strengths of the LLM. Additionally, we have integrated LLMVoX with a Vision-Language Model to create an omni-model with speech, text, and vision capabilities, without requiring additional multimodal training.

## introduction large language models (llms) have excelled in the new era of conversational ai, transforming how machines understand, generate, and interact with humans. while most llms were initially designed for text-based interactions, there are some recent efforts toward more natural and intuitive speech-tospeech dialogue systems, allowing users to engage with ai models through spoken language. alternatively, an llm-agnostic approach is to leverage a cascaded pipeline, where speech is converted to text via automatic speech recognition (asr), processed by an llm to generate a textual response, and finally passed through a text-tospeech (tts) module for speech output. this cascaded approach offers several advantages, including the availability of diverse off-the-shelf asr @xcite , llm @xcite , and tts @xcite ) models, the preservation of base llm capabilities, improved speech quality, and an llm-agnostic design that allows seamless adaptation to any base llm in a plug-andplay manner, without the need for computationally expensive model retraining. however, such cascaded approaches often introduce high latency (see cascaded-xtts in figure 1 ), making real-time interactions challenging. the primary reason for this high latency is the incompatibility between the autoregressive nature of llm-based text generation and conventional tts models, which typically process text inputs collectively, despite the text being available incrementally from llm. this prevents speech generation from starting until the entire text response, or a large chunk of it, has been generated by the llm. furthermore, many existing tts models rely on non-streaming speech decoders, leading to a larger delay between text and speech generation. to address the aforementioned limitations of existing speech-enabled llms, we propose llmvox, an autoregressive, llm-agnostic streaming framework. it aims to preserve the underlying llm's capabilities by completely decoupling speech synthesis from the llm, while enabling high-quality, low-latency speech generation (figure 1 ) in an autoregressive setting, running in parallel with the llm's text generation. ## methodology our proposed llmvox system in figure 2 is a fully autoregressive text-to-speech (tts) framework designed to convert text outputs from an upstream large language model (llm) into highfidelity streaming speech. the central motivation behind our design is to decouple the speech synthesis component from the text-generation process so that the inherent reasoning and expressive ca- in our approach, the speech signal is represented as a sequence of discrete tokens drawn from a fixed vocabulary of 4096 entries. these tokens are generated by a neural audio codec, and the speech token is predicted token-by-token in an autoregressive manner. figure 2 provides an overview of the overall architecture, where phoneme-aware embeddings derived from grapheme-to-phoneme (g2p) @xcite model are combined with previous acoustic context and processed by a decoder-only transformer to predict the next speech token. ## streaming inference we adopt a low-latency streaming inference pipeline (figure 3 and algorithm 1) for realtime speech dialogue system. given the user's speech input x user , we first transcribe it using an asr model (e.g., whisper) to obtain t @xmath0. an llm then generates a stream of words { t1 , t2 , . . . , tn } = llm(t query ), which are alternately enqueued into two fifo queues, q 1 and q 2 , based on sentence boundaries. two replica tts modules, llmvox 1 and llmvox 2 , concurrently dequeue words from q 1 and q 2 and predict speech tokens {s 1 , s 2 , . . . , s t } = llmvox i (q i ) for @xmath1, 2}. every n speech token generated is then decoded into speech by wav-tokenizer decoder and placed in producer queues p 1 and p 2 accordingly which is then streamed to the user immediately ensuring uninterrupted playback. the initial chunk size is n tokens, and after each segment is decoded, the chunk size doubles, leveraging the playback interval of previous speech to allow extra processing time as decoding larger chunks gives better speech output. this toggling mechanism seamlessly handles long or continuous text without requiring models with an extended or large context window. we evaluate llmvox on five key tasks: general qa capability assesses the model's ability to generate coherent and informative responses to general queries, reflecting the preservation of the llm's reasoning; knowledge retention measures the accuracy on fact-based questions to ensure robust information; speech quality examines the naturalness and clarity of the generated speech; speech-text alignment verifies the consistency between the synthesized speech and corresponding text generated by the llm. latency is defined as the total elapsed time from when a query is submitted to when the model begins speaking. ## conclusion we introduce llmvox, an llm-agnostic autoregressive streaming tts that decouples speech synthesis from text generation. leveraging a lightweight transformer and multi-queue streaming, llmvox delivers high-quality, continuous speech with minimal latency while preserving llm reasoning. experiments on english and arabic tasks show that llmvox outperforms or matches other speech-enabled llms, offering a scalable solution for real-time multimodal ai.