KIT ’s Offline Speech Translation and Instruction Following Submission for IWSLT 2025 (2025)

Abstract: In this paper, we present the submissions for the Offline ST and Instruction Following (IF) tracks, where we leverage LLMs to enhance performance across all tasks. For the Offline ST track, we propose a pipeline that employs multiple automatic speech recognition systems, whose outputs are fused using an LLM with document-level context. This is followed by a two-step translation process, incorporating additional contextual refinement step to improve translation quality. For the IF track, we develop an end-to-end model that integrates a speech encoder with an LLM to perform a wide range of instruction-following tasks. We complement it with a final document-level refinement stage to further enhance output quality by using contextual information.

## introduction this paper provides an overview of the systems submitted by the karlsruhe institute of technology (kit) to the offline speech translation (st) and the constraint long instruction-following (if) tasks of iwslt 2025. for the offline track, we participate in the unconstrained setting for the en-@xmath0. for the if task, we participate in the constrained-long track, aiming to perform automatic speech recognition (asr), speech translation (st), spoken question answering (sqa), and speech summarization (ssum) across various languages. a growing research trend in the field is the application of large language models (llms) to speech processing tasks @xcite @xcite , leveraging their strong general knowledge and natural language understanding capabilities. these strengths make llms particularly relevant to both the offline st and if tracks. accordingly, in our submissions, we explore strategies for effectively integrating llms into speech processing pipelines. there are multiple approaches to leveraging llms in speech systems. one strategy involves incorporating llms as an additional step within a cascaded architecture @xcite , where they can perform task-specific refinement. this modular approach allows each component to be trained independently, benefiting from specialized data. alternatively, llms can be integrated in an end-to-end fashion @xcite @xcite , allowing for better information flow and potentially improving generalization to unseen tasks. although both the offline and if tasks fall under the umbrella of speech processing, they differ significantly in nature. in the offline setting, speed and adaptability to unseen tasks are not primary concerns. in contrast, the if task demands flexibility and generalization, as the system must handle a variety of instructions. this has an impact on the architectures we choose for the different tracks. for the offline track, we utilize llms specialized on a specific task as refinement modules within a cascaded architecture. this is common practice; all systems submitted to iwslt 2024 for this track employed a cascaded architecture @xcite , underlining its practical advantages in training the system due to availability in data, e.g for low-resource languages @xcite , and simplicity by decomposing into smaller tasks. seamless-m4t qformer llama-3.1 we use bertscore and bleu score. we use bertscore and bleu score. ## offline track the goal of the offline st track is to generate highquality translations across diverse domains without latency constraints. recent work has highlighted the potential of llms for this task @xcite . building on these insights, we integrate llms at multiple stages of our speech translation pipeline. below, we present a high-level overview, with each component detailed in the following sections. we begin with long-form audio inputs, which may span several minutes to hours. due to memory limitations and the lack of training data for such durations, our asr and mt systems cannot handle these directly. thus, we first segment the audio into manageable chunks using a voice activity detection (vad)-based method, which is effective even in noisy conditions. the segmented audio is then transcribed into en-glish using asr. rather than relying on a single model, we adopt a fusion strategy, combining outputs from multiple asr systems-including both pre-trained models and a fine-tuned variant. this approach, akin to model ensembling, leverages the complementary strengths of different systems to reduce errors. we fuse the asr outputs using an llm, which processes the combined hypotheses at the document level. this allows for the incorporation of broader context, resulting in more coherent and accurate transcriptions. the english text is then segmented into sentences using the nltk tokenizer and translated into german. for this, we fine-tune a translation llm on high-quality parallel data. to ensure quality, we use a quality estimation model to filter out noisy sentence pairs, keeping only high-confidence examples. finally, both the source transcript and the machine-translated output are passed to an automatic post-editing (ape) model. this model refines the translations, producing polished final outputs. ## openai/whisper-large-v3 since we participated in the language direction en-@xmath1, the audio needs to be transcribed in english, a high-resource language. many publicly available pre-trained models excel at english transcription, and we first evaluated several of them individually. specifically, we considered the whisper variants v2 foot_0 and v3 foot_1 @xcite , as well as the recently developed multimodal llm phi-4 foot_2 @xcite . as shown in ## instruction following long track the instruction-following (if) speech processing track in the scientific domain aims to benchmark foundation models that can follow natural language instructions-an ability well-established in textbased llms but still emerging in speech-based counterparts. the track covers four tasks: automatic speech recognition (asr), speech translation (st), spoken question answering (sqa), and spoken summarization (ssum). asr is evaluated on english, st on @xmath2, chinese, and italian (@xmath3, it, zh}), and sqa/ssum across all four directions (@xmath4, de, it, zh}). we participate in the constrained long track, which focuses on long-form speech inputs (5-10 minutes). this track enforces limitations on both model selection and training data. specifically, only seamlessm4t-large 13 @xcite and llama-3.1-8b-instruct 14 @xcite are permitted as base models. our approach employs an end-to-end speech model trained under these constraints, enhanced with a post-editing stage for improved output quality similar to the offline track. ## conclusion this system paper presents kit's submissions to the offline and the if long tracks. by inte-grating llms into both cascaded and end-to-end architectures for speech processing, we demonstrate their potential in handling a range of spoken language tasks. for future work, we aim to explore a unified architecture capable of producing high-quality translations while also supporting instruction-following capabilities.