HITSZ ’s End-To-End Speech Translation Systems Combining Sequence-to-Sequence Auto Speech Recognition Model and I ndic Large Language Model for IWSLT 2025 in I ndic Track (2025)

Abstract: This paper presents HITSZ’s submission for the IWSLT 2025 Indic track, focusing on speech-to-text translation (ST) for English-to-Indic and Indic-to-English language pairs. To enhance translation quality in this low-resource scenario, we propose an end-to-end system integrating the pre-trained Whisper automated speech recognition (ASR) model with Krutrim, an Indic-specialized large language model (LLM). Experimental results demonstrate that our end-to-end system achieved average BLEU scores of 28.88 for English-to-Indic directions and 27.86 for Indic-to-English directions. Furthermore, we investigated the Chain-of-Thought (CoT) method. While this method showed potential for significant translation quality improvements on successfully parsed outputs (e.g. a 13.84 BLEU increase for Tamil-to-English), we observed challenges in ensuring the model consistently adheres to the required CoT output format.

## introduction speech-to-text translation plays a vital role in overcoming language barriers in multilingual and international contexts, such as real-time translation during online meetings. although translation systems for high-resource language pairs have achieved impressive performance, low-resource language pairs, particularly those involving indic languages, continue to face significant challenges @xcite . this paper presents hitsz's submission to the indic track of iwslt 2025, covering bidirectional speech translation between english and three major indic languages: hindi, bengali, and tamil. an overview of the end-to-end system is illustrated in figure 1 . ## related work recent advances in end-to-end speech translation (st) have demonstrated the effectiveness of combining large pre-trained models with task-specific adaptation @xcite @xcite @xcite , especially in lowresource and multilingual settings @xcite @xcite . among them, several works stand out for their innovative training paradigms and architectural choices that have directly influenced our approach. these include nict's submission to iwslt 2024 @xcite , which leverages decoder-side fine-tuning of whisper with pseudo-labels from indictrans2 @xcite ; zeroswot, which introduces an encoder-centric alignment method for zero-shot st @xcite ; and salmonn, a multimodal framework that uses a lightweight training pipeline to adapt frozen encoders and llms through cross-modal instruction tuning @xcite . in what follows, we briefly review each of these works and highlight their relevance to our system design. ## data in this section, we present the statistics of the initial corpora and describe our methods for preprocessing the raw data. we rely solely on the corpus provided by the organizers, with its statistics detailed above. although we did not incorporate any supplementary data, our model remains unconstrained by leveraging the pre-trained whisper asr model, the krutrim large language model, and adapters trained specifically for the spoken language translation task based on these two models. the audio segments corresponding to textual sentences are extracted from the original files based on the given offset and duration details. post-segmentation, each dataset entry includes an audio clip in the source language, its transcription, and a translation in the target languages. ## method our model builds upon the dhwani model @xcite , which is trained for speech translation tasks in indic languages and is itself derived from the salmonn architecture. to effectively process and align multimodal audio data with textual outputs, the architecture integrates several specialized components. for speech signals, it leverages the whisper speech encoder (whisperse) to extract robust linguistic representations. in parallel, non-speech audio inputs, such as environmental sounds and music, are processed using the beats encoder, which is optimized for general audio understanding. these two audio streams are subsequently bridged to the language model via a window-level query transformer (q-former), which acts as a connection module to transform modalityspecific features into a unified representation space. the transformed tokens are then passed to the krutrim llm, a 7-billion-parameter dense transformer model built on a multilingual foundation and optimized for indic language tasks. trained on a corpus of 2 trillion tokens with extensive coverage of native indian languages, krutrim demonstrates strong performance across multilingual benchmarks in both indic and english, despite being relatively lightweight in terms of training compute. to enable efficient domain-specific adaptation without retraining the entire model, low-rank adaptation (lora) is employed during fine-tuning. this technique aligns the llm's outputs with the semantics of the input audio, facilitating robust and adaptable performance. ## experiments and results this section details the experimental setup and presents the results for our monolingual speech translation models, trained individually for each translation direction. we follow the settings of the dhwani model, employing the whisper-large-v2 model as the speech encoder and the krutrim-1instruct model as the text decoder branch. ## conclusion this paper presented hitsz's submission to the iwslt 2025 speech-to-text translation task in the indic track. we leveraged recent advancements in indic llm by integrating the whisper model and the krutrim model into our end-to-end system. future work will primarily focus on two key directions: first, enhancing the instructionfollowing capability of the specialized llm for indic languages to facilitate the development of a spoken language translation system utilizing the chain-of-thought (cot) method; and second, improving its generation capabilities in indic languages to boost performance in english-to-indic translation tasks.