NICT ’s Cascaded and End-To-End Speech Translation Systems using Whisper and I ndic T rans2 for the I ndic Task (2024)

Abstract: This paper presents the NICT’s submission for the IWSLT 2024 Indic track, focusing on three speech-to-text (ST) translation directions: English to Hindi, Bengali, and Tamil. We aim to enhance translation quality in this low-resource scenario by integrating state-of-the-art pre-trained automated speech recognition (ASR) and text-to-text machine translation (MT) models. Our cascade system incorporates a Whisper model fine-tuned for ASR and an IndicTrans2 model fine-tuned for MT. Additionally, we propose an end-to-end system that combines a Whisper model for speech-to-text conversion with knowledge distilled from an IndicTrans2 MT model. We first fine-tune the IndicTrans2 model to generate pseudo data in Indic languages. This pseudo data, along with the original English speech data, is then used to fine-tune the Whisper model. Experimental results show that the cascaded system achieved a BLEU score of 51.0, outperforming the end-to-end model, which scored 19.1 BLEU. Moreover, the analysis indicates that applying knowledge distillation from the IndicTrans2 model to the end-to-end ST model improves the translation quality by about 0.7 BLEU.

## introduction and related work speech-to-text translation is crucial for breaking the language barriers during international activities, such as translating diverse languages in online meetings. although high-resource language pairs often achieve excellent results, the performance for low-resource language pairs remains unsatisfactory @xcite , such as english to indic languages. this paper presents nict's submission to the indic track of iwslt 2024, which includes translation directions from english to hindi, bengali, and tamil. an overview of the cascade and end-to-end systems is illustrated in figure 1 . data scarcity is a significant challenge for the english to indic languages st task due to its lowresource scenario and we are using data-driven neural models. data augmentation on speech and text data is an efficient way to address this challenge @xcite . assisting information such as phonetic information @xcite and spectral features @xcite , or knowledge transferred from related languages @xcite @xcite can also enhance the performance. to this end, we use data combined data from three directions rather than using them separately. cascade and end-to-end (e2e) systems are two popular paradigms in st with their advantages. in general, cascaded systems show higher translation quality @xcite and end-to-end systems usually show lower latency and less modeling burden @xcite . to maximize the translation quality, we adopt the cascaded way and attempt to make full use of the recent advancements in asr and mt fields @xcite . following preliminary experiments, we decided to participate in the unconstrained setting, where we leverage pre-trained models such as whisper and indic-trans2 to develop our cascaded and e2e systems. although additional datasets like indicvoices are available for indic languages @xcite , we refrain from using them due to concerns about test set overlap. we use whisper @xcite as our asr system. unlike previous work @xcite who prompt whisper without fine-tuning, we fine-tune whisper-medium on the training data. our results demonstrate significant improvements through fine-tuning. although other asr systems such as hubert @xcite , wav2vec 2.0 @xcite and others @xcite ) exist, we adopt whisper for its ease of use and its ability to deliver whisper automatic speech recognition indictrans2 text-to-text translation speech in english text in english thank you text in indic language ध"यवाद cascade inference separate training indictrans2 text-to-text translation fine-tune whisper automatic speech recognition input: speech in english output: text in english fine-tune thank you so much input: text in english output: text in indic language thank you so much आपका बह ु त-बह ु त ध"यवाद (a) cascaded system training and inference process. training of the speech-to-text translation system indictrans2 text-to-text translation fine-tune input: text in english output: text in indic language thank you so much आपका बह ु त-बह ु त ध*यवाद ① indictrans2 text-to-text translation text in english thank you so much pseudo translation in indic language आपका बह ु त ध*यवाद ② whisper speech-to-text translation input: speech in english output: pseudo translation in indic language fine-tune आपका बह ु त ध*यवाद ③ end-to-end inference whisper speech-to-text translation speech in english text in indic language ध*यवाद generate pseudo translation generation high-quality transcriptions of english speech. we then cascade whisper with indictrans2 @xcite as our mt system. it supports high-quality translations across 22 popular indic languages and outperforms the mbart50 model @xcite and the m2m-100 model @xcite in directions involving indic language. additionally, we explore the potential of the e2e system by employing knowledge distillation from the indictrans2 model into the whisper model. experimental results show that our cascaded systems are about 32 bleu better than the e2e systems. furthermore, e2e systems trained with distilled translations, which are obtained by translating english transcripts to indic languages via indictrans2, tend to be about 0.7 bleu points better than those using the originally provided gold standard translations. the remainder of this paper is structured as follows: section 2 describes the datasets and data preprocessing. section 3 introduces our cascade and end-to-end models. section 4 presents the ex-perimental settings, results, and analysis. lastly, section 5 concludes the paper. ## data we show the statistics of the original corpora and how we pre-process the raw data in this section. we use only the corpus provided on the official site, with statistics shown in ## method we describe the training and inference processes of the cascaded and e2e systems. ## conclusion this paper presented nict's submission to the iwslt 2024 english to indic speech-to-text translation task. we took advantage of the advancements in asr and mt where we combined the whisper model and indictrans2 model in our cascaded system. in our end-to-end system, we further utilize the pseudo translation data technique, also known as knowledge distillation, to empower the whisper model. future work will focus on combining whisper with indictrans2 jointly to train an even stronger speech translation system.