S pi R it- LM : Interleaved Spoken and Written Language Model (2025)

Abstract: We introduce SpiRit-LM, a foundation multimodal language model that freely mixes text and speech. Our model is based on a 7B pretrained text language model that we extend to the speech modality by continuously training it on text and speech units. Speech and text sequences are concatenated as a single stream of tokens, and trained with a word-level interleaving method using a small automatically curated speech-text parallel corpus. SpiRit-LM comes in two versions: a Base version that uses speech phonetic units (HuBERT) and an Expressive version that models expressivity using pitch and style units in addition to the phonetic units. For both versions, the text is encoded with subword BPE tokens. The resulting model displays both the semantic abilities of text models and the expressive abilities of speech models. Additionally, we demonstrate that SpiRit-LM can learn new tasks in a few-shot fashion across modalities (i.e., ASR, TTS, Speech Classification). We make available model weights and inference code.1,2

## introduction prompting large language models (llms) has become a standard in natural language processing (nlp) since the release of gpt-3 @xcite . scaling language models to billions of parameters with massive datasets helps to achieve general-purpose language understanding and gen- ,b,c equally contributed as co-first, co-second, and co-last authors, resp. speech language models (speechlms), i.e., language models trained directly on speech, have been introduced @xcite @xcite and have recently become an active field of research @xcite @xcite . these models are either trained on speech-only datasets or datasets of specific tasks, e.g., text-to-speech (tts), automatic speech recognition (asr) or translation, making the lms focus on certain modality or tasks and potentially loose their generalization capabilities. given the increasing quality of text-only llms @xcite , one successful approach to generate speech has been to build pipelines that first transcribe input speech with asr, then generate text using a text-only llm and finally synthesize the generated text into speech with tts. however, with such pipelines, modeling and generating expressive speech is constrained out of the language model, leading to poor generation from an expressive point of view. in this work, we aim to combine the generative abilities and pretrained knowledge of text llms with the expressive capacities of speech-language models. we show that llms trained on interleaved speech and text can learn speech and text cross-modally and are able to generate language content in either modality. we evaluate the models with comprehension tasks in both speech and text, and extend few-shot prompting to speech-text tasks such as asr, tts or speech classification. we further extend the phonetic speech tokens with expressive tokens that capture the pitch and style of the speech, and evaluate the models with newly introduced sentiment modeling tasks. our contributions are the following: (i) we introduce spirit-lm, a single language model that can generate both speech and text. spirit-lm is based on continuously pretraining llama 2 with interleaving speech and text data. (ii) similarly to text llms, we find that spirit-lm can learn new tasks in the few-shot setting in text, speech and in the cross-modal setting (i.e., speech to text and text to speech). (iii) to evaluate the expressive abilities of generative models, we introduce the speech-text sentiment preservation benchmark (noted stsp) that measures how well generative models preserve the sentiment of the prompt within and across modalities for both spoken and written utterances. 3 (iv) we propose an expressive version of spirit-lm (spirit-lm-expressive). using stsp, we show that spirit-lm is the first lm that can preserve the sentiment of text and speech prompts both within and across modalities. (v) finally, we quantify the potential added toxic content in the generation of our model for both speech and text. as all pretrained base models @xcite , spirit-lm can generate harmful content. for these reasons, all user-facing applications using our work should integrate the necessary red-teaming work and implement safety instruction-tuning to meet safety standards @xcite . 4 2 related work textless nlp recent progress in self-supervised speech representation learning (ssl) @xcite @xcite has made it possible to learn from raw audio speech representations that are good for a variety of downstream tasks @xcite . in addition, these methods can be used to derive discrete tokens that operate as a kind of pseudo-text and can be used to learn a language model from raw audio @xcite which is able to capture both the linguistic content and the prosody @xcite , giving rise to a host of applications: emotion conversion @xcite , dialogue generation @xcite , and speech classification @xcite . even though these models are good at capturing expressivity, they trail text models in capturing semantics when trained with comparable amounts of data (see @xcite . in this work, we use phonetic speech tokens extracted from hubert @xcite , possibly combined with pitch and style tokens (as in @xcite , and supplement the model with textual bpe-units. speech and speech+text lms there has been an increasing number of speechlms since gslm @xcite . audiolm @xcite utilizes two types of discrete speech tokens with phonetic tokens foot_2 @xcite , and acoustic tokens @xcite to capture phonetic and acoustic information from speech respectively. vall-e @xcite ) models speech with acoustic tokens @xcite and perform tts task by translating phonemes to tokens using an autoregressive lm. @xcite found that fine-tuning pretrained textlms helps boost the performance of speechlms. speechgpt @xcite further fine-tunes speechlms on cross-modal tasks (asr, tts) and chain-of-modality question-answering (qa) tasks. similar to speechgpt, spectron @xcite ) utilizes text as a proxy for spoken qa and speech continuation tasks. unlike previous work, they represent speech using a spectrogram with pre-trained speech encoder from @xcite . in the same spirit, @xcite adapted llama 2 for speech generation tasks. audiopalm @xcite and viola @xcite both train autoregressive language models on text and speech in a multi-task fashion. most recently, voxtlm @xcite and sutlm @xcite jointly trained speech and text lms on asr, tts, and speech/text continuation tasks. our work is similar to @xcite in the training tasks but with the capacity of performing cross-modal generation and expressive speech and text generation. we also study larger models and evaluate their zero-shot and in-context learning capabilities. ## spirit-lm training recipe spirit-lm models are based on continuously pretraining a text-pretrained language model on a combination of text and speech (figure 1 .a). following @xcite , we continuously pretrain llama 2 (touvron et al., 2023b) using a collection of text-only datasets, speech-only datasets and aligned speech+text datasets fed to the model with interleaving. spirit-lm comes in two versions: base and expressive. spirit-lm-base models speech using hubert tokens @xcite while spirit-lm-expressive uses the concatenation of hubert pitch and style tokens. ## speech and text understanding as illustrated in in addition to speech, these benchmarks are also available in the text modality. we therefore measure the text-modeling abilities of spirit-lm on these. in addition, we evaluate spirit-lm on mmlu @xcite , a popular evaluation benchmark for text-based llms, using a 5-shot setting. all the tasks are reported with the accuracy metrics. cross-modal speech-to-text and text-to-speech tasks spirit-lm is trained in both speech and text. for this reason, it has the ability to model tasks that require both text and speech modeling. based on the text and speech versions of storycloze, we build speech to text (@xmath0.g., speech) and the hypothesis is in the other modality (e.g., text). they are evaluated similarly to other comprehension tasks by comparing the log-likelihood given by the model and are performed in the zero-shot setting. we also evaluate spirit-lm in-context learning capability with few-shot generation tasks: asr, tts, and speech intent classification. for asr, we prompt the model with examples of speech-text transcription pairs along with a new speech segment for the model to generate the text transcriptions. we report the word-error-rate baselines we compare our results with previously published generative speech systems. all these methods use one or several transformer @xcite decoder-only models trained on speech units. they differ in how they are trained (pretrained from scratch or fine-tuned), the types of speech units they model, and their amount of training data. we compare with gslm @xcite , twist @xcite based on llama-13b, and audiolm @xcite . in contrast with spirit-lm, the approaches mentioned above only rely on speech units during training, making them speech-only models (i.e., they do not support text understanding nor generation). we also compare our models to voxtlm @xcite , a concurrent work on speech and text language modeling. we report the best scores from the original published papers for all the mentioned methods. as a top-line comparison, we compare our models with cascade models that use llama 2 as a text generative model. for text-to-text (@xmath1, we only rely on llama 2 -7b. for speech-to-speech (@xmath2, we utilize the cascade model, asr from whisper-medium(radford et al., 2023), followed by llama 2, synthesized by mms-tts @xcite . ablation experiments finally, we ablate the several components of the spirit-lm training recipe. we compare spirit-lm-base to a llama 2 model continuously pretrained with two parallel data training settings. first, the asr+tts-only model consists of training with pairs of semantically equivalent sequences of speech and text (e.g., ''[text] the cat jumped by the window [tts][hu12]..[hu54]'' or ''[speech][hu12]..[hu54][asr] the cat jumped by the window'' foot_8 ). second, the word-level transcription model consists of training on sequences of pairs of textual and spoken words (e.g., ''[text] the [speech][hu12]..[hu34] [text] cat [speech][hu454]..[hu90]...[text] window [speech][hu15]..[hu54]''). additionally, we compare spirit-lm-base to models trained on a single modality (speech or text) and with speech+text but without any interleaving data (cf. noted ''no interleaving''). ## expressivity modeling one of the core contributions of this work is the modeling of expressivity. to measure the expressivity of our model we first evaluate the quality of the introduced pitch and style tokens ( § 5.1). second, we evaluate our spirit-lm models on the newly introduced speech-text sentiment preservation benchmark ( § 5.2). ## responsible ai in speech and text this section discusses and evaluates responsibility aspects from spirit-lm. speechlms have the potential to bring the same benefits as text-based lms and potentially increase their reach to lowresource languages that are mainly spoken. quantifying and working on user safety is a key aspect from generative model development. these models can inadvertently generate content that is harmful, offensive, or inappropriate is essential for generative language models @xcite @xcite . while safety is a broad concept, we focus on the specific problem of added toxicity in the generation of the spirit-lm. inspired by previous studies @xcite , we define added toxicity as a toxicity increase in the generation compared to the initial source utterance. ## conclusion we introduced spirit-lm, a language model based on llama 2 that can generate both speech and text in a cross-modal manner. we showed that by alternating speech and text in the input sequence during training, the model can generate the content fluidly by changing from one modality to another. we evaluated our models on a collection of speech and text metrics. we plan to make future improvements both in the area of model capability and in transparency and safety. @xmath3:stop>, the spoken utterance ''stop'' tokenized into speech tokens. 18 for models trained with parallel tts data (e.g., spirit-lm-base+asr+tts), the token [speech] is replaced with @xcite .