V oice T ext B lender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuning (2025)

Abstract: Recent studies have augmented large language models (LLMs) with speech capabilities, leading to the development of speech language models (SpeechLMs). Earlier SpeechLMs focused on single-turn speech-based question answering (QA), where user input comprised a speech context and a text question. More recent studies have extended this to multi-turn conversations, though they often require complex, multi-stage supervised fine-tuning (SFT) with diverse data. Another critical challenge with SpeechLMs is catastrophic forgetting, where models optimized for speech tasks suffer significant degradation in text-only performance. To mitigate these issues, we propose a novel single-stage joint speech-text SFT approach on the low-rank adaptation (LoRA) of the LLM backbone. Our joint SFT combines text-only SFT data with three types of speech-related data: speech recognition and translation, speech-based QA, and mixed-modal SFT. Compared to previous SpeechLMs with 7B or 13B parameters, our 3B model demonstrates superior performance across various speech benchmarks while preserving the original capabilities on text-only tasks. Furthermore, our model shows emergent abilities of effectively handling previously unseen prompts and tasks, including multi-turn, mixed-modal inputs.

## introduction large language models (llms) have demonstrated impressive success in natural language processing @xcite @xcite , sparking a surge of research into multimodal foundation models that extend beyond text. recent studies in speech processing have focused on augmenting pre-trained llms with speech capabilities, giving rise to a new class of models known as speech language models (speechlms) @xcite @xcite @xcite @xcite @xcite . initially, speechlms were primarily designed for single-turn speech-based question answering (sqa) tasks @xcite @xcite , where the input consists of an audio clip and a text question, with the model expected to generate a text answer. while these models perform well on training tasks such as automatic speech recognition (asr), automatic speech translation (ast), and sqa, they often struggle with general-purpose textual or spoken instructions and are not capable of handling multi-turn mixed-modal conversations. recent speechlms aim to support multi-turn conversations where user input can be entirely audio @xcite . developing these models requires aligning speech features with text embeddings using large amounts of carefully curated data. however, no well-established public methodology-including data generation scripts, data specifications, or model training details-currently exists for building such models. in this work, we present voicetextblender (or vt-blender in short), a voice-text language model that supports multi-turn, mixed-modal conversations, where user turns may contain both speech and text (see figure 1 for example). we will publicly release our data generation scripts, provide comprehensive details about our training data and process, and make our pre-trained model weights available. another challenge in building speechlms is maintaining the original text-only performance while enhancing speech capabilities, which is crucial for creating a truly multi-modal model. some approaches, such as llama 3.1 @xcite , give one more suggestion. here's another suggestion: 4. rest and get enough sleep. your body needs time to recover from illness. remember, it's always better to be safe than sorry. if you're unsure about your symptoms or need further assistance, don't hesitate to reach out to a healthcare professional. here are some resources to help you find doctors near santa clara: our contributions are summarized below. speechlms on most evaluated benchmarks. ## conclusion we propose a novel single-stage joint speech-text sft approach for training speechlms using lora adapters. this method simplifies the training process, preserves the text-only performance of the llm backbone, and achieves excellent speech understanding capabilities. specifically, we combine multi-turn text-only sft data with singleturn speech-related sft data during training. to extend beyond speech-based qa tasks, we propose a novel data generation method that can create mixed-modal interleaving speech-text inputs. our model achieves excellent performance across various speech benchmarks while retaining performance on text-only benchmarks. our 3b model even outperforms previous 7b or 13b speechlms on most evaluated benchmarks. furthermore, our model exhibits emergent capabilities in handling previously unseen instructions and multi-turn mixed-modal conversations. we will publicly release our codebase and pre-trained models to advance research in speechlms.