IWSLT 2025 I ndic Track System Description Paper: Speech-to-Text Translation from Low-Resource I ndian Languages ( B engali and T amil) to E nglish (2025)

Abstract: Multi-language Speech-to-Text Translation (ST) plays a crucial role in breaking linguistic barriers, particularly in multilingual regions like India. This paper focuses on building a robust ST system for low resource Indian languages, with a special emphasis on Bengali and Tamil. These languages represent the Indo-Aryan and Dravidian families, respectively. The dataset used in this work comprises spoken content from TED Talks and conferences, paired with transcriptions in English and their translations in Bengali and Tamil. Our work specifically addresses the translation of Bengali and Tamil speech to English text, a critical area given the scarcity of annotated speech data. To enhance translation quality and model robustness, we leverage cross-lingual resources and word level translation strategies. The ultimate goal is to develop an end-to-end ST model capable of real-world deployment for under represented languages.

## introduction speech-to-text translation (st) has seen significant progress in recent years, driven by advancements in deep learning and large-scale multilingual datasets. however, the benefits of these advancements have not equally reached low-resource languages. many indian languages, despite being spoken by millions, lack sufficient parallel speechtext corpora to train high-performing supervised models. this paper addresses this gap by focusing on st systems for bengali and tamil speech to english text, two major indian languages that are often underrepresented in current st research. india's linguistic diversity presents both a challenge and an opportunity for speech translation research. in multilingual communities, there is a growing demand for st systems that can facilitate communication across different linguistic groups, especially in education, healthcare, and public services. however, the shortage of human translators and limited digital resources for many indian languages hamper efforts to build such systems. this work is motivated by the need to create languageinclusive st models that cater to low resource indian languages such as bengali and tamil. these languages are widely spoken but lack the largescale annotated datasets available for high-resource languages like english, german, or mandarin. by focusing on bengali and tamil audio-to-english text translation, this paper aims to fill a critical gap in current st research. additionally, the paper explores methods to overcome data scarcity, such as leveraging related language resources, incorporating multilingual pretraining, and utilizing wordlevel translation dictionaries. the broader goal is to build a scalable and adaptable st pipeline that not only improves translation accuracy but also supports the integration of more languages in the future. the work leverages a curated dataset of public speeches, including ted talks and conference recordings, to build a system that can handle the unique linguistic and acoustic features of these languages. our approach explores both end-to-end and cascaded architectures, aiming to strike a balance between performance and scalability. by addressing the linguistic diversity and resource limitations of these languages, we aim to contribute towards more inclusive language technology in india and beyond. speech-to-text translation (st) research has traditionally been concentrated on high-resource languages, with significant advancements in languages such as english, german, and mandarin. however, languages such as bengali and tamil, spoken by millions in multilingual regions like india, remain underrepresented in research due to the scarcity of parallel speech text corpora. the absence of large-scale annotated datasets for these languages presents a significant barrier to training high-performance models. in recent years, there has been growing interest in leveraging existing resources from related languages and innovative techniques to overcome these challenges. for instance, the iwslt shared tasks have provided valuable insights into the effectiveness of both cascaded and end-to-end (e2e) models for speech-to-text (st) tasks. a cascaded architecture typically involves a two-step process: first, automatic speech recognition (asr) converts speech to text, and then machine translation (mt) translates the recognized text from the source language into the target language. this approach has been shown to perform effectively in the handling of complex speech data. in the context of low-resource languages, the cascaded model approach has demonstrated robustness, especially when training data is limited. recent work has explored combining asr models like openai's whisper @xcite , trained on multilingual data, with neural mt systems such as helsinki-nlp/opus-mt-bn-en. this combination has proven effective in addressing the challenges posed by limited resources, especially for languages such as bengali and tamil. these models leverage the strengths of both components to improve translation accuracy and ensure scalability in real-world applications. the 2023 iwslt evaluation campaign, for example, evaluated offline slt systems for translating speech from english to german, japanese, and chinese, using both cascaded and e2e models. the campaign highlighted the importance of combining asr and mt components, as well as the performance improvements that could be achieved by integrating large-scale language models, data augmentation, and ensemble methods @xcite . although e2e models are more direct, cascaded systems are particularly well-suited for lowresource languages, as they allow for leveraging pre-existing, powerful models for both asr and mt tasks. for bengali and tamil, this cascaded approach has shown promise by using pre-trained asr models (such as whisper) followed by fine-tuned mt models (like helsinki-nlp/opus-mt-bn-en) to handle the translation from these languages to english. by employing this two-step architecture, the system benefits from the specific strengths of both asr and mt, improving overall translation accuracy and ensuring adaptability to the challenges presented by these languages. the work conducted in the iwslt2024 indic track system description paper @xcite focuses on speech-to-text translation for multiple indian languages, including bengali and tamil, and follows a similar cascaded approach to tackle the challenges of low-resource languages. this aligns closely with the goal of our paper, which aims to build a robust and scalable st system that can effectively handle the translation of bengali and tamil speech to english text. by incorporating these strategies, our work contributes to advancing speech-to-text translation for low-resource languages, filling a critical gap in the research landscape, and offering a path forward for scalable models that can support the diverse linguistic landscape of india. ## dataset description the iwslt 2025 indic track @xcite focuses on speech-to-text (st) translation between english and three low-resource indian languages: hindi (hi), bengali (bn), and tamil (ta) and vice-versa. these languages belong to two major language families-indo-aryan (hindi and bengali) and dravidian (tamil)-and are widely spoken across south asia. the dataset for this task specifically supports speech-to-text translation from indic languages to english, where the source is audio in a lowresource indian language, and the target is english text. it includes: each language dataset is carefully aligned. every english transcript line has a corresponding line in the target language (hindi, bengali, or tamil), along with metadata in yaml format. this metadata provides information such as file name, offset, duration, and speaker id. the corpus is divided into training, validation, and test subsets. each audio file corresponds to a talk by a single speaker, contributing to diverse speaking styles and accents. while the number of segments is consistent across the aligned files, token counts may differ across languages due to linguistic variations. in our work, we focused specifically on the bengali-to-english and tamil-to-english translation directions. we used bengali and tamil audio files aligned with their english translations. the actual dataset contains significantly more than 50,000 samples for each language pair, providing a rich and diverse resource for training, evaluation, and fine-tuning. this well-structured, multilingual dataset serves as a strong foundation for building effective speech-to-text translation systems for lowresource indian languages like bengali and tamil. ## system overview the system integrates advanced speech-to-text (asr) and machine translation (mt) models to transcribe bengali audio files and translate the transcriptions into english. the architecture consists of several interconnected modules, each playing a crucial role in ensuring accuracy, efficiency, and robustness. the key components of the system include an input module that accepts audio files in wav format, which is a standard format for audio processing due to its lossless nature. along with the audio file, a yaml metadata file is provided ,which contains the following information: the data validation and preprocessing module validates the provided metadata, ensuring that the offset, duration, and speaker id align correctly with the audio file. audio segmentation is then performed based on the provided offset and duration to ensure precise transcription. the audio processor and transcription module consists of two sub-modules, namely, the audio chunk extraction and the model integration. in the audio chunk extraction module, librosa @xcite and soundfile libraries were used to extract precise segments from the original audio file based on the metadata. these libraries are efficient in processing and manipulating audio data. the whisper-small model is loaded via the hugging face transformers library @xcite . whisper is a robust, multilingual asr model that can handle diverse languages and dialects with zero-shot capabilities @xcite . the model is fine-tuned using a quantized low-rank adaptation (qlora) @xcite technique on a custom dataset of 10, 000 bengali-english audio pairs. qlora is a parameterefficient fine-tuning technique that allows the model to adapt to new tasks with minimal computational overhead while retaining its generalization ability and quantization greatly reduces memory usage by reducing precision of floating points. the extracted audio chunks are then passed to the model, which transcribes the bengali audio to bengali text or tamil audio to tamil text . we have used the kaggle free resources for all our task which provides us with 2 tesla p100 gpus due to which we faced computational constraints and used qlora as an alternative. in the translation module, the system uses the helsinki-nlp/opus-mt-bn-en model, a state-ofthe-art model pre-trained for bengali-to-english translation tasks. the model is fine-tuned using csv-aligned bengali-english pairs. this alignment ensures the model learns the appropriate context, improving translation accuracy. the model is also trained with the seq2seqtrainer framework, which is highly effective for sequence-to-sequence tasks such as translation. this method optimizes the model for better handling bengali syntax and semantics complexities during translation. after the completion of the transcription and translation phase, the system merges the filename, transcription, and translated output into a unified output. the results are stored in both csv (for structured data) and txt (for easy reading and further processing) formats. this allows for easy extraction and post-processing of results. to evaluate the translation quality, our system uses sacre-bleu and chrf++ metrics, which are standard in machine translation tasks. we have achieved a bleu score of 8.6945 and a chrf++ score of 35.5653 for the bengali-english pair. these scores suggest that the system provides a reasonably highquality translation, with strong character-level accuracy. in addition to the bengali-english translation system, the architecture supports tamil-to-english (ta-en) translation using the facebook / nllb-200-distilled-600m model @xcite . this is a multilingual, distilled version of the nllb-200 model by meta ai, designed to handle translations across 200 languages with enhanced efficiency. for tamil (language code ta_taml) to english (language code eng_latn), the system takes tamil transcriptions (e.g., from speech recognition output or manually curated corpora), tokenizes them using the nllb tokenizer, and then applies the sequenceto-sequence model for translation. the translation is done using forced bos (beginning-of-sentence) tokens to ensure the output is directed towards english. this translation pipeline is implemented using hugging face transformers and evaluated using standard machine translation metrics. the system achieves a bleu and chrf++ score of 13.3904 and 39.0237 on the tamil-english test set. these results reflect a strong translation performance, especially given the morphological richness of tamil. like the bengali-english pipeline, the tamil-english system operates in an unconstrained setting, where no limitations are placed on the type of data or preprocessing methods used. this allows maximum flexibility in improving performance through data augmentation, custom preprocessing, or enhanced model assembling techniques. the logic of fine-tuning the transformers are mentioned below: ## limitations while we acknowledge the significant challenges ahead, such as the shortage of multilingual individuals and insufficient data for certain languages, we are determined to find innovative solutions. some of the limitations of our current approach include: despite these limitations, we remain committed to enhancing system performance. ongoing research focuses on expanding language support, improving inference speed, and increasing the quality of both transcription and translation outputs. these efforts are part of a broader goal to make speech-sync system a reliable and efficient multilingual speech-to-text translation system, particularly for underrepresented indian languages. ## future work while the current version of the speechsync system demonstrates strong performance in bengalito-english and tamil-to-english speech translation, there remain several promising directions for future improvement and expansion. one important enhancement is the incorporation of speaker diarization and multi-speaker handling. this would allow the system to differentiate between individual speakers in a single audio stream. this feature is essential for accurately processing meetings, interviews, or conversational datasets. by integrating diarization models, the system could associate transcription segments with specific speaker labels, improving readability and structure. another potential development is real-time streaming transcription and translation. this would significantly expand the system's usability in live scenarios such as conferences, classrooms, and emergency response settings. achieving this would involve optimizing the current pipeline to minimize latency and memory usage, allowing for faster and more efficient processing. currently, the asr outputs include only basic punctuation, which can hinder readability. to address this, future iterations will aim to integrate advanced punctuation and formatting. this includes accurate sentence boundaries, speaker turn indicators, and proper capitalization. these enhancements would make both transcriptions and translations more natural and easier to follow. further improvement could come from multimodal integration, where additional visual cues such as lip movements or gestures are used to aid transcription accuracy, especially in noisy or acoustically challenging environments. this would position the system for use in richer, context-aware applications like video subtitling or assistive communication. ## conclusion in summary, our key contributions lie in the rigorous experimentation conducted to identify effective models for speech translation, especially for lowresource languages like bengali and tamil. we perform extensive preprocessing of data to ensure quality and suitability for training. the proposed solution establishes a robust pipeline, including code development and workflow setup, allowing for efficient transcription and translation tasks. the training and experimentation were focused on bengali to english translation for an in-depth analysis, which included fine-tuning whisper for transcription tasks using lora and helsinki-nlp/opus-mtbn-en for translation. in addition, we extended our work to tamil to english translation using the facebook/nllb-200-distilled-600m model, which was fine-tuned on tamil-english parallel data to improve translation quality and generalization. this enabled the system to support multilingual speechto-text translation more broadly. close monitoring of performance metrics, including bleu and chrf++ scores, was carried out to assess model performance and guide future improvements. this paper is committed to advancing speech translation (st) technology for low-resource languages. through the creation of dedicated datasets and the development of robust models for both bengali and tamil, our aim is to facilitate seamless communication and accessibility across diverse linguistic communities, ultimately promoting inclusivity and empowerment.