NVIDIA N e M o Offline Speech Translation Systems for IWSLT 2022 (2022)

Abstract: This paper provides an overview of NVIDIA NeMo’s speech translation systems for the IWSLT 2022 Offline Speech Translation Task. Our cascade system consists of 1) Conformer RNN-T automatic speech recognition model, 2) punctuation-capitalization model based on pre-trained T5 encoder, 3) ensemble of Transformer neural machine translation models fine-tuned on TED talks. Our end-to-end model has less parameters and consists of Conformer encoder and Transformer decoder. It relies on the cascade system by re-using its pre-trained ASR encoder and training on synthetic translations generated with the ensemble of NMT models. Our En->De cascade and end-to-end systems achieve 29.7 and 26.2 BLEU on the 2020 test set correspondingly, both outperforming the previous year’s best of 26 BLEU.

## introduction we participate in the iwslt 2022 offline speech translation task @xcite for @xmath0. due to the limited amount of direct speech translation (st) data, we mostly focused on building a strong cascade pipeline structured as follows: we also trained end-to-end models capitalizing on the pre-trained asr encoder and synthetic translations obtained with the ensemble of nmt models. our best end-to-end model consisting of conformer encoder and transformer decoder lags behind the best cascade by 2.7 bleu on average, however, it might be preferred for some scenarios of limited resources or latency requirements. our systems are open-sourced as part of nvidia nemo foot_0 framework @xcite . ## data in this section, we describe the datasets used for training ( asr for training our asr model, we used lib-rispeech @xcite , mozilla common voice v6.1 @xcite , ted-lium v3 @xcite , voxpopuli v2 @xcite , all available speech-to-english data from must-c v2 @xcite en-de/zh/ja datasets, st-ted @xcite , and clean portion of europarl-st @xcite . pc for training our punctuation-capitalization (pc) model, we combined 268m sentences from europarl @xcite , rapid @xcite , ted @xcite , newscrawl, news-commentary english corpora used in wmt 2021 @xcite and wikipedia dump from wmt 2020. after that, we split the data into segments of up to 128 words ignoring sentence boundaries and removed all punctuation and capitalization. mt for training our nmt models, we used all available bitext from wmt 2021 @xcite , as well as its right-to-left distillation and back-translated monolingual data (for @xmath1, following @xcite . after training, we fine-tuned our models on bitexts from must-c v2 dataset. st for training our end-to-end st models, we used must-c v2, covost v2 @xcite , st-ted, and clean portion of europarl-st. in addition, we translated english transcripts from asr datasets with unnormalized transcripts (all datasets, except for librispeech and ted-lium v3) to obtain more speech-to-german data. ## system in this section, we describe the essential components of our cascade and end-to-end submissions. segmentation we relied on voice activity detection (vad) to transform long ted talks from the evaluation datasets into smaller segments. specifically, we used webrtc foot_1 toolkit with frame duration, padding duration, and aggressive mode set to 30ms, 150ms, and 3, respectively. following @xcite , we then merged multi-ple short segments into longer chunks until there were no two segments shorter than a threshold m @xmath2. we also experimented with other hyperparameters in the vicinity of these values but the resulting average bleu score on iwslt test datasets from previous years was lower. asr we transcoded all audio data to monochannel 16khz wav format and normalized all the transcripts by removing capitalization and all punctuation marks except for apostrophe. we also discarded samples shorter than 0.2s and longer than 24s. as a result, our training dataset contained 1.9m audio segments with the total duration of 3800 hours. we then trained a large version of conformertransducer @xcite with roughly 120m parameters, which uses rnn-t loss and decoder @xcite . the prediction network consists of a single layer of lstm @xcite and the joint network is an mlp. all the hidden sizes in the decoder were set to 640. pc our punctuation-capitalization (pc) model consists of transformer encoder initialized with pre-trained t5 @xcite and two classification heads -one for predicting punctuation and another for predicting capitalization. capitalization head has two labels which correspond to whether the corresponding token needs to be upper-cased. punctuation head has four labels for period, comma, question mark, and no punctuation which correspond to whether the corresponding token needs to be followed by a particular punctuation mark. to do inference on the text of arbitrary length, we split it into segments of equal segment length and compute a sliding window (with a step step) product of token probabilities. to reduce prediction errors near the segment boundaries, we discard probabilities of margin tokens near the segment boundaries except for the left boundary of the first segment and the right boundary of the last segment. nmt our @xmath3. we discarded all examples where a sentence in either language is longer than 250 tokens and where the length ratio between source and target exceeds 1.3. we also applied langid and bicleaner filtering following @xcite . after such aggressive filtering, we ended up with 60m parallel sentences and 250m monolingual sentences for back-translation. we then trained four 24 × 6 nmt transformers using different combinations of bitext, its right-to-left forward translation, and back-translated monolingual data. our @xmath4, instead of moses based tokenization and normalization. we used sentencepiece @xcite tokenizer with shared vocabulary trained on a combination of english, chinese and japanese. we also did not do ensembling. after training with news-only data, we additionally fine-tuned all our models on must-c v2 dataset which resulted in nearly 4 bleu score boost on iwslt test sets for @xmath5. the ensemble of four such models was used to generate synthetic translations for end-to-end st model training. to better adapt our cascade nmt models to possible punctuation-capitalization model artifacts, we altered the source side of fine-tuning dataset by normalizing it and running through the pc model. end-to-end our end-to-end model is conformer encoder followed by transformer decoder trained on pairs of english audio and german translation. after discarding all segments longer than 24s, we ended up with 740k segments with the total duration of 1180 hours. adding synthetic translations of asr datasets with unnormalized transcripts resulted in 2.06m segments with the total duration of 3450 hours. time-adaptive specaugment (park et al., 2020) with 2 freq masks (@xmath6. we also used dropout of 0.1 for both the attention scores and intermediate activations. all predictions were made with greedy decoding and no external language model. pc we trained our pc model for up to 400k updates using adam optimizer @xcite and inverse square root annealing @xcite with 12k warm-up steps and a maximum learning rate of 6 × 10 -5 . dropout of 0.1 was used for regularization. despite significant imbalance between no punctuation / capitalization and other classes, we trained with cross-entropy loss which showed to perform well in prior work @xcite . we then computed f1 scores for both classification heads on iwslt tst2019 dataset. our high mean punctuation f1 score of 84.6 and capitalization f1 score of 92.6 suggest that the model does not suffer from the class imbalance inherent in the training data. nmt we trained our nmt models (transformer, 24 × 6 layers, d @xmath7, d @xmath8, n @xmath9. the models were trained for a maximum for 450k steps with a dropout of 0.1 on intermediate activations and label smoothing with α = 0.1. after training, we finetuned all our base nmt models on must-c v2 for 3-4 epochs with an initial learning rate of 2 × 10 -5 , linear annealing and no warmup. end-to-end our end-to-end models (17-layer conformer encoder, 6-layer transformer decoder, both with d @xmath10, d @xmath11, n @xmath12. our vocabulary consists of 16384 youtokentome foot_2 byte-pair-encodings trained on german transcripts of st corpus. ## conclusion we present nvidia nemo group's offline speech translation systems for @xmath13. our primary cascade system consists of conformer rnn-t asr model, followed by transformer-based pc and nmt models. to improve over the baseline, we utilize checkpoint averaging, in-domain fine-tuning, adaptation to pc artifacts, and ensembling. the resulting submission outperforms the last year's best @xcite by 3.7 bleu on iwslt 2020 test dataset. however, it is worth noting that this year more data was available for training. our contrastive end-to-end model consists of conformer encoder and transformer decoder and translates speech directly into the text in target language. the performance of such model trained on available st data was almost 10 bleu worse comparing to cascade. we managed to shrink this gap to 2.7 bleu by capitalizing on strong asr and nmt components of our cascade via pre-training and synthetic data generation. due of its size and simplicity this model may be preferred for some scenarios, such as simultaneous speech translation.