From Tower to Spire: Adding the Speech Modality to a Translation-Specialist LLM (2025)

Abstract: We introduce Spire, a speech-augmented language model (LM) capable of both translating and transcribing speech input from English into 10 other languages as well as translating text input in both language directions. Spire integrates the speech modality into an existing multilingual LM via speech discretization and continued pre-training using only 42.5 K hours of speech. In particular, we adopt the pretraining framework of multilingual LMs and treat discretized speech input as an additional translation language. This approach not only equips the model with speech capabilities, but also preserves its strong text-based performance. We achieve this using significantly less data than existing speech LMs, demonstrating that discretized speech input integration as an additional language is feasible during LM adaptation. We make our code and models available to the community.

## introduction large language models (llms) have demonstrated remarkable success on various text-based natural language processing tasks @xcite @xcite @xcite , motivating research into extending them to other modalities. this has led to the development of multimodal lms capable of processing speech, audio, images, and video @xcite @xcite @xcite @xcite @xcite . however, the integration of new modalities often comes at the cost of existing capabilities @xcite . for speech-llm integration, a simple approach is to link the output of an automatic speech recognition (asr) system to a text-only llm @xcite . this solution, however, is prone to error propagation and depends largely on individual model quality. more popular are solutions that investigate equipping llms natively with speech processing capabilities through modality projection @xcite @xcite @xcite . typically, a speech foundation model generates speech representations that are mapped to the embedding space of the llm, following which the model is then fine-tuned along with a projector on speech-to-text tasks to equip the llm with speech processing capabilities. in this setting, key challenges include prompt overfitting and high training costs, as tuning these multimodal llms requires the adaptation of the speech projector module on vast amounts of raw speech data @xcite . an alternative approach for integrating speech into a text-only llm is to use speech discretization, where continuous speech features are transformed prior to training into sequences of "discrete speech units" (dsus), which can be processed similarly to text @xcite @xcite @xcite @xcite . this approach simplifies training by eliminating the need for additional parameters beyond extended embedding matrices. finally, while both projector-based and discretization-based solutions have shown promising results on text-tospeech and speech-to-text tasks, their development has prioritized speech-centric tasks at the expense of textual performance. furthermore, limited research has focused on integrating speech while preserving the llm's original capabilities in textual tasks @xcite . in this work we present spire, a speechaugmented llm built from the open-weight multilingual model tower (alves et al., 2024). spire can perform english asr and from-english speech translation (st) while maintaining tower's strong performance on machine translation (mt) across all 10 languages 1 supported by tower. spire encodes speech via hubert-based (hsu et al., 2021) adapting llms previous approaches involve training from scratch with task-and domainspecific data @xcite , performing cpt with a diverse training data mix designed to broadly extend the model's knowledge @xcite , or it on use-case-specific data @xcite . recent work has explored combining the latter two approaches @xcite @xcite . in our approach to integrating dsus into tower, we take inspiration from @xcite in adopting a two-step cpt+it process. our work differs in that we focus on adding the speech modality, whereas alves et al. (2024) focused on increasing the multilingual capabilities of an llm. continuous and discrete speech representations self-supervised speech representation models produce contextualized high-dimensional speech vectors directly from raw audio @xcite @xcite , largely outperforming statistical speech features on downstream tasks @xcite . these continuous representations can be used to derive dsus that capture both linguistic content and prosody through clustering @xcite . dsus provide better alignment with textual data, facilitating the transfer of successful training settings from the text domain @xcite . building on @xcite , which demonstrated that hubert @xcite is a powerful feature extractor, several studies have adopted this approach, incorporating a k-means clustering step for discretization @xcite @xcite @xcite . @xcite study the optimal settings to obtain dsus in terms of cluster size and feature extraction layer. we use their findings to inform our initial choices. ## spire: a speech-to-text llm we introduce spire, whose goal is to equip an llm with speech capabilities while preserving its preexisting text capabilities. as our base llm we choose tower @xcite , which was developed from llama-2 @xcite with a two-step approach: cpt on a mixture of monolingual and parallel data (towerbase), followed by it on translation-related tasks (towerinstruct). we use an approach similar to tower to extend the model to the speech modality. first, we perform cpt with a combination of text-only and aligned speech-to-text datasets, followed by it using both text-only general-purpose and task-specific data curated in towerblocks, 3 alongside task-specific speech-to-text datasets. we choose tower in particular due to its competitive performance compared to other open alternatives. tower-based models were among the best participating systems in the wmt24 general translation task @xcite . tower's usage of open source data during the cpt phase along with the release of the towerblocks dataset, used in the it phase, further motivates our choice. ## experiments we evaluate our models across three tasks: asr, mt, and st. first, we present our results for asr ( §4.1), confirming the new capabilities spire has in the speech domain. we then present mt results ( §4.2), demonstrating that the speech performance does not come at the expense of the original model's mt performance. finally, we present results for st ( §4.3) to investigate model performance on a task that requires both asr and mt capabilities. evaluation setup across models and tasks, we perform inference with greedy decoding with a maximum of 256 generated tokens. for the tower and spire models, we decode with vllm @xcite . however, since vllm does not support all of our baselines, we use alternative libraries (transformers; @xcite where necessary. unless specified otherwise, we use zeroshot prompts for all models and tasks. ## analysis the key innovation of our approach is the application of the cpt followed by it paradigm to dis- we report additional datasets in appendix d. effectiveness of cpt and it our previous results demonstrated that using both cpt and it was the most effective strategy. the performance gap between spirefull and the towerfull on asr (5.3 points in ls test-clean) further shows that it alone is also not as effective. however, for st we observe that performing only it leads to a strong model that is capable of performing speech translation. this contrasts from spirebase, for which we also attempted direct st but the model failed to produce output in the target language, even when given few-shot prompts. despite the impressive results from towerfull, we still observe the best performance by spirefull showing that while the effect of cpt is not as drastic as in the case of asr, we still observe gains with a speechcentric cpt phase. ## conclusion in this work we presented spire, a simple and effective recipe for adapting a text-based, translationspecialist llm to the speech modality while preserving the original performance on text-based tasks. we investigated the impact of speech integration on two stages of llm adaptation, cpt and it, finding that both contribute to the final model's performance on speech tasks. our results demonstrate that we are able to successfully integrate a new modality without compromising the original model's capabilities. spire achieves competitive performance on asr, while its mt abilities remain on par with the original tower model. finally, for the st task, we find that the leveraging asr and mt data does not directly transfer to st performance. nonetheless, the model achieves promising performance with both direct and selfcascaded st. to benefit the community, we only used publicly available and licensed data to train our models, making our results reproducible. as future work, we intend to extend this recipe to multilingual settings by replacing our english hubert speech component by the multilingual mhubert-147 @xcite .