CUNI - NL @ IWSLT 2025: End-to-end Offline Speech Translation and Instruction Following with LLM s (2025)

Abstract: This paper describes the CUNI-NL team’s submission to the IWSLT 2025 Offline Speech Translation and Instruction Following tasks, focusing on transcribing the English audio, and translating the English audio to German text. Our systems follow the end-to-end approach, where each system consists of a pretrained, frozen speech encoder, along with a medium-sized large language model fine-tuned with LoRA on three tasks: 1) transcribing the English audio; 2) directly translating the English audio to German text; and 3) a combination of the above two tasks, i.e. simultaneously transcribing the English audio and translating the English audio to German text.

## introduction end-to-end speech translation (st) is a growing research direction that aims to ignore the intermediate speech recognition (asr) step to directly translate the audio input into the corresponding text in another language. this approach simplifies the overall architecture and has been shown to match the performance of the cascaded counterpart @xcite @xcite . large language models (llms) have demonstrated their good performance in a large number of complex natural language tasks, including machine translation @xcite @xcite . with the ever-improving potential of llms, researchers have been trying to integrate different components used for other modalities, in order to extend their abilities to go beyond text-only tasks @xcite @xcite @xcite . motivated by recent contributions in speech representation learning and llms, to participate in the iwslt 2025 offline speech translation and instruction following tasks, we aim to investigate an end-to-end architecture that can perform both asr and st. this architecture combines the high-quality audio representation from the pre-trained acoustic models with the excellent performance of llms to serve as an end-to-end speech translation system, while still having the ability to transcribe from the audio signal. our systems, after being fine-tuned with the low-rank adaptation (lora; @xcite technique, achieve a solid performance in both speech recognition and translation. the paper is structured as follows: ## future work for the iwslt 2025 offline speech translation and instruction following tasks, we have only conducted experiments for the english-to-german di- in addition, we have some ideas to improve the pipeline: ## conclusion in this paper, we leveraged pre-trained speech encoders and llms and connected them into an endto-end architecture to participate in the iwslt 2025 offline speech translation and instruction following tasks. our primary goal was to develop a system that could perform both asr for english audio, and st from english audio to german text. in our experiments, the model with seamless-m4t-v2-large as the speech encoder and eurollm-9b-instruct as the llm yielded the best results in evaluation of both asr and st tasks, suggesting that this pair could be a promising combination for end-to-end models. computational resources were provided by the e-infra cz project (id:90254), supported by the ministry of education, youth and sports of the czech republic. nam luu has been supported by the erasmus mundus program in language and communication technologies (lct). ondřej bojar has received funding from the project op jak mezisektorová spolupráce nr. cz.02.01.01/00/23_020/0008518 named "jazykověda, umělá inteligence a jazykové a řečové