Instituto de Telecomunicações at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning (2025)

Abstract: This paper presents Instituto de Telecomunicações’s submission to the IWSLT 2025 Shared Task on Instruction Following Speech Processing. We submit results for the Short Track, i.e., speech recognition, translation, and spoken question answering. Our model is a unified speech-to-text model that integrates a pretrained continuous speech encoder and text decoder through a first phase of modality alignment and a second phase of instruction fine-tuning. Crucially, we focus on using small-scale language model backbones (< 2B) and restrict to high-quality, CC-BY data along with synthetic data generation to supplement existing resources.

## introduction this paper presents our submission to the iwslt 2025 instruction following track for the tasks of automatic speech recognition (asr), speech translation (st), and spoken question answering (sqa) for english, chinese, and german. our work builds upon a long line of previous research equipping lms with additional multimodal capabilities, aligning an lm's semantic spaces with that of a pretrained speech encoder @xcite @xcite @xcite . our contribution is particularly motivated by efficiency, i.e., the goal of achieving strong performance using small-scale (< 2b) models. recent work has explored audio quantization techniques @xcite , quantized input mel spectrograms @xcite , or extreme compression of input data over the time dimension through convolutional kernels paired with strong small-scale lm backbones @xcite . we take stock of such advancements and propose a model for even smaller scales. we use established methods for speech integration in lms @xcite using pretrained base models of up to 1.5b learnable parameters, finding empirically that with highly filtered and synthetic data, we can enable similar results at a fraction of the cost of larger lms. the main contributions of our system are as follows: ## results our results for all three short-track tasks are in our model obtains reasonably good asr scores for english. this result is particularly relevant, considering that the test data originate from the technical domain, exhibit high speaker variability, and consist primarily of spontaneous speech. however, while the model successfully performs asr, sqa and st prove to be more complex. through manual inspection, we observed poor quality outputs for st. at times, the model repeats the same word or ignores the task tag and transcribes the audio segment rather than translating it. this finding aligns with prior work that has found asr data dominates the multitask capabilities of models @xcite . moreover, it emphasizes the importance of a more carefully designed training curriculum, where sqa, st, and asr data are more evenly distributed. lastly, due to the audio length cutoff-set to 120 seconds due to technical limitations-we were unable to use all of the available sqa data. at test time, when prompted to perform the sqa task, the model sometimes generates the question itself, rather than the answer. we believe that by utilizing a combination of more data, enhanced base models with stronger multilingual capabilities, and extended context support, we will be able to improve upon these results substantially. ## conclusions we have presented our submission for the iwslt 2025 instruction following short track. we explored the usage of a small-scale lm in modality adaptation through a continuous speech encoder. in particular, we equip an existing text model, qwen 2.5 1.5b, with the speech modality for a joint multilingual and multitask model. we used standard modality alignment approaches, including building on pretrained speech encoders and autoregressive text decoder mod-els, and a two-stage curriculum learning. in future work, we plan to support longer contexts, better filtered data, and further push small-scale lms to be fully multimodal. we will incorporate more high-quality multilingual data to enhance the model's language identification capabilities. additionally, we will extend the evaluation beyond standard performance-oriented benchmarks, e.g., by accounting for safety @xcite and fairness @xcite .