NVIDIA N e M o Offline Speech Translation Systems for IWSLT 2023 (2023)

Abstract: This paper provides an overview of NVIDIA NeMo’s speech translation systems for the IWSLT 2023 Offline Speech Translation Task. This year, we focused on end-to-end system which capitalizes on pre-trained models and synthetic data to mitigate the problem of direct speech translation data scarcity. When trained on IWSLT 2022 constrained data, our best En->De end-to-end model achieves the average score of 31 BLEU on 7 test sets from IWSLT 2010-2020 which improves over our last year cascade (28.4) and end-to-end (25.7) submissions. When trained on IWSLT 2023 constrained data, the average score drops to 29.5 BLEU.

## introduction we participate in the iwslt 2023 offline speech translation task @xcite for @xmath0, @xmath1, and @xmath2. this year, we focus on an endto-end model, which directly translates english audio into text in other languages. in contrast to automatic speech recognition (asr) and text-to-text neural machine translation (nmt), the data for direct speech translation (st) is scarce and expensive. thus, to train a highquality end-to-end st model, we heavily rely on a number of auxiliary models for which the amount of available data is enough. specifically, we train the following models: our constrained end-to-end st model consists of a fastconformer encoder and a transformer decoder. we initialize the encoder with the corresponding component from asr and train our st model on a mix of speech-to-text and text-to-text data. we replace all ground truth translations (wherever available) with synthetic ones generated with the nmt model and voice the english portion of parallel text corpora with tts. our systems will be open-sourced as part of nvidia nemo foot_0 framework @xcite . ## data in this section, we describe the datasets used for training ( ## system in this section, we describe the essential components of our end-to-end submission. asr we trained 17-layer large conformertransducer @xcite with fastconformer @xcite encoder and rnn-t loss and decoder @xcite . the prediction network consisted of a single layer of lstm @xcite , and the joint network is an mlp. all the hidden sizes in the decoder were set to 640. unigram senten-cepiece @xcite with 1024 tokens was used for tokenization. the asr models were trained for 45 epochs, starting with a checkpoint pre-trained on lib-rispeech. we used adamw @xcite optimizer and noam annealing @xcite with 10k warmup steps and a maximum learning rate of 1.15. weight decay of 0.001 on all parameters was used for regularization. the effective batch size was set to 1200, and we could fit larger batch sizes via batch splitting for the rnn-t loss. time-adaptive specaugment (park et al., 2020) with 2 freq masks (@xmath3. we also used dropout of 0.1 for both the attention scores and intermediate activations. nmt we trained our nmt models (transformer, 12 × 6 layers, d @xmath4, d @xmath5, n @xmath6.5k warmup steps and a maximum learning rate of 10 - foot_2 . the models were trained for a maximum of 75k steps with a dropout of 0.1 on intermediate activations and label smoothing with α = 0.1. our @xmath7. after training, we did checkpoint averaging and fine-tuned all our base nmt models on ted talks for 3 epochs with an initial learning rate of 2×10 -5 , inverse square root annealing, and a warmup of 10% steps. finally, we ensembled 2 models trained with different initializations for each language direction. tts our tts model was multi-speaker fast-pitch @xcite text-to-mel-spectrogram generator. training vocoder was not necessary for our setup as the parameters of spectrograms matched ones for st models following the approach described in @xcite . ttsgenerated spectrograms were fed directly into the fastconformer encoder when training the st model. our tts model was trained for 200 epochs on ted talks with restored speakers from ted-lium v3 @xcite . segmentation we used supervised hybrid audio segmentation (shas) approach following tsiamas et al. (2022). as using speech representation pre-trained wav2vec 2.0 @xcite goes beyond the scope of constrained submission, we replaced it with conformer asr encoder, pretrained on librispeech. st our end-to-end model consisted of fastconformer encoder followed by transformer trained on pairs of english audio and transcripts in other languages (17-layer fastconformer encoder, 6 × 6 transformer, both with d @xmath8, d @xmath9: word error rate (wer) of the english asr model evaluated on ted talks from must-c v2 and past test sets from iwslt. all predictions and ground truths transcripts were normalized for wer computation. ## conclusion we present nvidia nemo group's offline speech translation systems for @xmath10, @xmath11, and @xmath12. our primary end-to-end models that translate english speech directly into german, chinese, and japanese texts, consist of fastconformer encoder and transformer decoder. to alleviate the problem of direct st data scarcity, we capitalized on a number of auxiliary asr, tts, and nmt models, and their ability to generate hiqh-quality audio and translations. the resulting models achieve competitive performance without using any amount of direct st data. although we participated in constrained scenario, our pipeline can be easily scaled to arbitrarily large amounts of asr and nmt data.