IWSLT 2024 I ndic Track system description paper: Speech-to-Text Translation from E nglish to multiple Low-Resource I ndian Languages (2024)

Abstract: Our Speech-to-Text (ST) translation system addresses low-resource Indian languages (Hindi, Bengali, Tamil) by combining advanced transcription and translation models for accurate and efficient translations. The key components of the system are: The Audio Processor and Transcription Module which utilizes ResembleAI for noise reduction and OpenAI’s Whisper model for transcription. The Input Module validates and preprocesses audio files. The Translation Modules integrate the Helsinki-NLP model for English to Hindi translation and Facebook’s MBart model for English to Tamil and Bengali translations, fine-tuned for better quality. The Output Module corrects syntax and removes hallucinations, delivering the final translated text. For performance evaluation purpose, SacreBLEU scores were used and attained the following values: English-to-Hindi: 24.21 (baseline: 5.23); English-to-Bengali: 16.18 (baseline: 5.86); English-to-Tamil: 10.79 (baseline: 1.9). The solution streamlines workflow from input validation to output delivery, significantly enhancing communication across different linguistic contexts and achieving substantial improvements in SacreBLEU scores. Through the creation of dedicated datasets and the development of robust models, our aim is to facilitate seamless communication and accessibility across diverse linguistic communities, ultimately promoting inclusivity and empowerment.

## introduction multi-language speech-to-text translation (st) is indispensable for facilitating communication across diverse linguistic contexts. while recent advancements have shown remarkable progress, many dialects and low-resource languages still lack sufficient parallel data for effective supervised learning. creative approaches are essential to overcome this challenge, such as leveraging resources from related languages or utilizing word-level translation resources and raw audio. this work aims to address these gaps by developing an end-to-end (e2e) or cascaded st model for low-resource indian languages, including hindi, bengali, and tamil. ## motivation the scarcity of translators proficient in multiple languages, especially in low-resource settings, highlights the urgent need for st systems supporting multiple languages. in regions like india, characterized by a multitude of languages, the development of dedicated models for indian languages is essential for effective communication. this task aims to advance st technology for a wide range of languages. our ultimate goal is to foster inclusively and accessibility through the creation of robust st models. this research work is fueled by a strong commitment to address significant challenges in speech translation, with a particular focus on languages spoken in india. in modern interconnected society, the capacity to communicate across various languages is crucial. however, the shortage of translators who can handle multiple languages in resource-constrained areas, presents a major obstacle. ## related work prior research in st has primarily focused on highresource languages, leaving many dialects and lowresource languages underserved. the lack of parallel data poses a significant challenge in training supervised learning models for these languages. however, recent efforts have demonstrated the effectiveness of leveraging existing resources from related languages and employing innovative approaches to enhance translation accuracy @xcite . the 20 th international conference on spoken language translation (iwslt) organized shared tasks targeting nine scientific challenges in spoken language translation (slt). these tasks covered a wide spectrum. this encompasses simultaneous and offline translation, automatic subtitling and dubbing, speech-to-speech translation, multilingual translation, translation of dialects and low-resource languages, and formality control. the conference witnessed substantial interest with a total of 38 submissions from 31 teams, evenly distributed between academia and industry @xcite . the focal point of the 2023 iwslt evaluation campaign was offline slt, which involved translating audio speech from one language to text in another language without time constraints. it com-1 359 prised three sub-tasks for translating english into german, japanese, and chinese. participants were given the flexibility to utilize either cascade architectures, which combine automatic speech recognition (asr) and machine translation (mt) systems, or e2e approaches that directly translate input speech @xcite . principal objectives were twofold: firstly, to gauge the performance disparity between cascade and end-to-end systems, and secondly, to evaluate slt technology's competence in handling intricate scenarios like simultaneous overlapping or concurrent speakers. the introduction of new test sets, encompassing acl presentations and press conferences/interviews, aimed at a comprehensive assessment of system efficacy @xcite . training data conditions spanned from constrained to unconstrained, offering varying levels of access to training resources. development data encompassed ted talks, acl presentations, and interviews from the european parliament multimedia centre. system evaluations were conducted employing bleu and comet metrics, supplemented by human assessment of the top-performing entries @xcite . ten teams partook in the offline task, collectively submitting 37 runs. a plethora of techniques were employed across these submissions, including cascade and direct models, leveraging large language models, multimodal representations, data augmentation, ensemble methods, and advanced training strategies. evaluation criteria emphasized the attainment of high translation quality across diverse language pairs and challenging scenarios @xcite . ## workflow figure 2 shows basic workflow of the application. at first, there is input processing, where users upload audio files or provide input through supported channels. this serves as the gateway for input data, ensuring its integrity and validity. the input module undertakes the crucial task of verifying and preprocessing the audio data, preparing it for subsequent processing stages by addressing any inconsistencies. once the transcription and translation processes are complete, the output delivery stage takes over, presenting the translated text to users through the output module. this enables seamless access and utilization of the translated content, offering users the flexibility to download the text or integrate it directly into their workflows. by providing a userfriendly interface and facilitating easy dissemination of translated content, the application empowers users to overcome language barriers and engage in effective cross-cultural communication. ## baseline vs. results the baseline sacrebleu scores is provided by in-dic track. for each language pair we have a different baselines. ## limitations while we acknowledge the significant challenges ahead, such as the shortage of multilingual individuals and insufficient data for certain languages, we are determined to find innovative solutions. our input module currently supports only one language, so if the audio file contains multiple languages, the application ignores languages other than primary language. currently, the other limitation is the time taken by the models to produce output. we may try out various optimisations and configurations to achieve faster results. for language pair -english to bengali, we are barely crossing the baseline, so our primary goal is to achieve better score for bengali language.