SRI - B ’s Systems for IWSLT 2023 Dialectal and Low-resource Track: M arathi- H indi Speech Translation (2023)

Abstract: This paper describes the speech translation systems SRI-B developed for the IWSLT 2023 Evaluation Campaign Dialectal and Low-resource track: Marathi-Hindi Speech Translation. We propose systems for both the constrained (systems are trained only on the datasets provided by the organizers) and the unconstrained conditions (systems can be trained with any resource). For both the conditions, we build end-to-end speech translation networks comprising of a conformer encoder and a transformer decoder. Under both the conditions, we leverage Marathi Automatic Speech Recognition (ASR) data to pre-train the encoder and subsequently train the entire model on the speech translation data. Our results demonstrate that pre-training the encoder with ASR data is a key step in significantly improving the speech translation performance. We also show that conformer encoders are inherently superior to its transformer counterparts for speech translation tasks. Our primary submissions achieved a BLEU% score of 31.2 on the constrained condition and 32.4 on the unconstrained condition. We secured the top position in the constrained condition and second position in the unconstrained condition.

## introduction speech translation (st) is the task of automatically translating a speech signal in a given language into text in another language. while rapid strides have been made in speech translation in recent times, this progress has been restricted to a small number of high resource languages. this progress excludes sizable sections of people who speak languages that have very little speech data available. so, for these speech systems to be beneficial and impactful in the real world, they have to be developed and shown to work on low-resource languages as well. in order to mitigate these issues and encourage research on low-resource languages, iwslt propose a dialectal and low-resource speech translation track @xcite as a part of their 2023 shared tasks evaluation campaign. while this track includes various low resource languages, we focus our efforts on the marathi-hindi language pair. the goal of this task is to translate marathi speech to it's corresponding hindi text. marathi and hindi are both indo-aryan languages used in india. even though there were 83 million people across india speaking marathi as per the 2011 census of india, it lacks sufficient speech data to support modern speech translation systems. this paper discusses our work and submissions on the marathi-hindi low-resource speech translation task. our experiments in this paper focus only on end-to-end architectures. we begin our experiments with a simple end-to-end transformer and build on this approach with the following key contributions that significantly better our final performance: ## related work traditionally, speech translation was performed using cascaded systems @xcite ) @xcite ) @xcite of asr and machine translation (mt) models. in this approach, speech was first transcribed using an asr model and then the transcriptions were translated to text in the target language with the help of a mt model. this approach however possessed several key drawbacks like error propagation, increased latency, and architectural complexity due to multiple models. the first attempt towards building an end-to-end speech translation system was by @xcite , where they built a system that eliminated the need for source language transcriptions. similarly, @xcite proposed an attention based encoder-decoder architecture for end-to-end speech translation that exhibited improved performance over cascaded systems. @xcite perform a detailed comparison between the paradigms of cascaded and end-to-end speech translation. developing speech translation systems for lowresource scenarios are especially challenging given the scarcity of training data. speech translation systems submitted in iwslt 2019 @xcite tended to prefer cascaded approaches for low-resource tracks. the cascaded approach which was favoured in @xcite , used a hybrid asr system with wav2vec features followed by a mt model for two low-resource language pairs. recently, as system trained with joint optimization of asr, mt and st @xcite exhibited good performance. also, usage of self-supervised learning based pre-trained models such as xlr-s @xcite and mbart @xcite have been shown to be effective, especially for low-resource scenarios. ## data description the challenge data consists of marathi speech to hindi text translation data from the news domain for the model training and development which we shall henceforth refer to as st (speech translation) data. the details of this dataset has been mentioned in the organizers shared additional marathi audio data along with its transcripts which can be used for the constrained condition, the details of which have been mentioned in for the unconstrained condition, in addition to the aforementioned datasets, iiit-h voices (prahallad et al., 2012) @xcite and iitm indic tts @xcite were also utilized, both of which were designed for building tts systems. ## system description all the models we trained for this challenge are end-to-end speech translation (st) systems. for the purposes of this challenge, we tried two architectures: listen, attend and spell (las) @xcite style transformer @xcite and the same model with its encoder replaced with conformer @xcite layers. both the models were implemented using the fairseq s2t toolkit @xcite . the conformer model consists of a 16-layer conformer encoder paired with a 6-layer transformer decoder. the transformer model comprises of a 12-layer transformer encoder and a 6-layer transformer decoder. in all the cases where pretraining is involved, the encoder blocks are pretrained @xcite using marathi asr data mentioned in for speech inputs, 80-channel log mel-filter bank features (25ms window size and 10ms shift) were extracted with utterance-level cmvn (cepstral mean and variance normalization) applied. specaugment @xcite is applied on top of this feature set. we experimented with character vocabulary and a 1000 bpe (byte pair encoding) vocabulary and found that the former performs better for our task. adam (kingma and ba, 2014) with a learning rate of 2 × 10 -3 was the optimizer of choice for all the experiments. inverse square-root scheduling available in the toolkit was used with a warm-up of 1000 steps. label-smoothed-cross-entropy with 0.1 as label smoothing was used as the criterion across all the experiments. we set dropout @xcite 0.1 during st training. we pre-train on the asr data for 6000 steps and then train on the st data for 2250 steps. after st training, we average the last 10 checkpoints to create the final model. we used a beam size of 10 for decoding. ## results the results for all the models we trained can be seen in ## conclusion in this paper we present our approaches to the iwslt 2023 evaluation campaign dialectal and low-resource track: marathi-hindi speech translation which secured the first and second places in the constrained and unconstrained conditions respectively. we start off with a simple end-toend approach with transformers and then apply a gamut of ideas like replacing the encoder blocks with conformers, encoder pre-training, etc., to drastically improve our dev bleu score from 1.02 to 20.22. through our results, we also quantitatively demonstrate how much of an impact each of our ideas bring forth and sincerely hope that some of these ideas might be useful for researchers and practitioners alike working on low-resource speech translation problems.