BUT Systems for IWSLT 2023 M arathi - H indi Low Resource Speech Translation Task (2023)

Abstract: This paper describes the systems submitted for Marathi to Hindi low-resource speech translation task. Our primary submission is based on an end-to-end direct speech translation system, whereas the contrastive one is a cascaded system. The backbone of both the systems is a Hindi-Marathi bilingual ASR system trained on 2790 hours of imperfect transcribed speech. The end-to-end speech translation system was directly initialized from the ASR, and then fine-tuned for direct speech translation with an auxiliary CTC loss for translation. The MT model for the cascaded system is initialized from a cross-lingual language model, which was then fine-tuned using 1.6 M parallel sentences. All our systems were trained from scratch on publicly available datasets. In the end, we use a language model to re-score the n-best hypotheses. Our primary submission achieved 30.5 and 39.6 BLEU whereas the contrastive system obtained 21.7 and 28.6 BLEU on official dev and test sets respectively. The paper also presents the analysis on several experiments that were conducted and outlines the strategies for improving speech translation in low-resource scenarios.

## introduction a typical end-to-end (e2e) speech translation model is trained with the help of data triplets (x, y, z), i.e., the speech signal (x) in source language, along with its transcription (y), and, text translation (z) in target language. in usual lowresource scenarios, the transcriptions in source language are unavailable and moreover the speech signal and the translation pairs (x, z) are also limited, which is the case for the iwslt 2023 marathi to hindi low-resource speech translation task @xcite . in such cases, one can rely on transfer learning, where models trained on relatively large amounts of data (possibly on a related task such as automatic speech recognition) are transferred (adapted) to the target task/scenario (such as speech translation) using little amounts of labelled data @xcite . to be specific, we train automatic speech recognition (asr) systems on relatively large amount of transcribed speech data (2790 hours), and transfer the model for speech translation task by fine-tuning it on relatively small amount (16 hours) of iwslt marathi-hindi training data. this paper describes the systems submitted for the aforementioned task. while building the systems, we mainly focused on end-to-end systems, which resulted in our primary submission. we have also put some efforts in building a cascade pipeline that was submitted as a contrastive system. both the systems come under the unconstrained category, i.e., we relied on external, publicly available data to train models. these models, which we refer to as back-bone models, mainly comprise automatic speech recognition (asr), machine translation (mt) and language models (lm). the section 2 describes the various datasets used for training the back-bone models, and section 3 presents the details of each individual back-bone models (asr, mt, lm), followed by description of transfer learning for actual speech translation systems in section 4. the section 5 gives the results and analysis, quantifying the effect of various factors on the target translation task. finally, we conclude in section 6 and discuss directions for future works. ## datasets for training here we describe the details and present the statistics of various datasets used for training the backbone models. these datasets come under various categories, i.e., paired speech data for training asr, parallel text data for training mt and monolingual data for training lms. all the data we considered for training covers only hindi and marathi languages. both these share the same devanagari script (unicode block) but there a few set of charac-ters that are mutually exclusive. ## back-bone models here, we describe the architecture and training details of various backbone models. ## speech translation systems here, we briefly describe both the end-to-end and cascade systems. ## results and analysis here, we present the performance of various backbone models, along with analysis showing the effectiveness of various factors such as initializations, data augmentation, auxiliary objectives and joint decoding. ## conclusions in this paper, we presented the systems submitted to the iwslt 2023 marathi hindi low resource track. our main efforts were along the end-to-end direct speech translation system, initialized from a bilingual asr. the model was jointly trained with ctc and attention objective directly for translation. the joint decoding provided additional benefits. these strategies combined with speed perturbation for data augmentation and re-scoring the n-best hypotheses using external lm provided further significant improvements. we also submitted a cascade system which uses the same bilingual asr as the backbone, followed by an mt system. both systems performed competitively, while the one based on end-to-end provided superior results in terms of bleu. it is yet to be investigated, if the large pre-trained mt systems would close the gap between cascade and end-to-end systems.