NVIDIA N e M o’s Neural Machine Translation Systems for E nglish- G erman and E nglish- R ussian News and Biomedical Tasks at WMT 21 (2021)

Abstract: This paper provides an overview of NVIDIA NeMo’s neural machine translation systems for the constrained data track of the WMT21 News and Biomedical Shared Translation Tasks. Our news task submissions for English-German (En-De) and English-Russian (En-Ru) are built on top of a baseline transformer-based sequence-to-sequence model (CITATION). Specifically, we use a combination of 1) checkpoint averaging 2) model scaling 3) data augmentation with backtranslation and knowledge distillation from right-to-left factorized models 4) finetuning on test sets from previous years 5) model ensembling 6) shallow fusion decoding with transformer language models and 7) noisy channel re-ranking. Additionally, our biomedical task submission for English \leftrightarrow Russian uses a biomedically biased vocabulary and is trained from scratch on news task data, medically relevant text curated from the news task dataset, and biomedical data provided by the shared task. Our news system achieves a sacreBLEU score of 39.5 on the WMT’20 En-De test set outperforming the best submission from last year’s task of 38.8. Our biomedical task Ru-En and En-Ru systems reach BLEU scores of 43.8 and 40.3 respectively on the WMT’20 Biomedical Task Test set, outperforming the previous year’s best submissions.

## introduction we take part in the wmt'21 news shared task for english ↔ german, english ↔ russian, and the biomedical shared task for english ↔ russian. our systems are implemented in the nvidia nemo foot_0 framework @xcite . they build on baseline sequence-to-sequence transformer models @xcite in the following ways: 1) checkpoint averaging, 2) model scaling up to 1b parameters, 3) data augmentation with large-scale backtranslation @xcite of monolingual newscrawl data and sequence-level knowledge distillation from a right-to-left factorized model @xcite , 4) finetuning models on in-domain news data from wmt test sets made available in previous years, 5) ensembling models trained on different subsets of the overall data 6) shallow fusion decoding with transformer language models @xcite 7) noisy channel re-ranking of beam search candidate hypotheses @xcite . overall, we find each of these components results in a small improvement in bleu scores with backtranslation results being mixed depending on the language direction and whether the test data contains translationese inputs. using a combination of these techniques, we achieve 39.5 sacre-bleu scores on the @xmath0, outperforming the best bleu scores from last year's competition of 38.77. training our en ↔ ru biomedical task submission from scratch using a biomedical vocabulary and similar model improvements to those used for our news task submission, we report a sacrebleu score of 40.3 on @xmath1.8 on @xmath2. this improves over the best submissions from last year's competition foot_1 of 39.6 and 43.3 on @xmath3. ## datasets we participated in the constrained data track at this year's news and biomedical competitions and used all the parallel corpora provided by the wmt shared tasks for both en ↔ de and en ↔ ru. we used the provided english, german, and russian monolingual newscrawl data for backtranslation and training our autoregressive transformer language models. we filter out monolingual newscrawl data only based on minimum and maximum length criteria, but perform more aggressive filtering of our parallel data described in section 2.1. ## system overview our systems build on the transformer sequenceto-sequence architecture @xcite . in the subsequent subsections, we discuss model scaling, checkpoint averaging, data augmentation with backtranslation and right-to-left distillation, model finetuning, ensembling, shallow fusion decoding with lms, and noisy channel re-ranking. ## news task submission in this section, we present results for our news shared task submission. tables 2 and 3 contain ablations for en ↔ de and while tables 4 and 5 has ablations for en ↔ ru. each of the components we describe improves bleu scores except for backtranslation and scaling our models to 1b params. both show mixed results on en ↔ de -scores improve significantly on wmt'14 and wmt'18 test sets when adding backtranslated data (possibly because these test sets contain translationese inputs) but hurts or does not improve performance on wmt'19 and wmt'20 test sets. our 1b parameter model does significantly better on wmt'14, but worse on wmt'19 and is comparable to the 500m parameter model on wmt'18 and wmt'20. we found optimization @xmath4.6 33.0 37.7 36.6 0 (2) (1) + transformer-xlarge + ckpt avg 38.7 34.3 38.2 37.2 0.9 (3) (2) + finetuning (wmt'14-16) 40.7 35.4 40.5 37.1 2.2 (4) (3) + ensemble (x3) 40.7 35.5 41 37.7 2.5 (5) (4) + shallow fusion 40.9 35.9 40.8 37.5 2.6 (6) oracle bleu with beam size 4 --46.4 -- with adam to be unstable and used adamw with a weight decay of 0.01 instead. our final @xmath6.5 on the wmt'20 test set, which improves over the submission with the best bleu score from last year's competition of 38.8. we however do not do as well on @xmath7, with a final bleu score of 42, compared to last year's best submission of 43.8. backtranslation significantly hurt performance in initial experiments on en ↔ ru so we exclude it from our ensemble. the impact of ensembling, finetuning, and shallow fusion are fairly similar to en ↔ de. additionally, we also report an "oracle bleu" score in tables 4 and 5 where we compute bleu scores by cheating and picking the translation on our beam that has the highest sentence bleu score with respect to the reference. this is a useful indicator of how much there is to gain by re-ranking the beam search candidates. ## biomedical translation task submission we present our biomedical shared task submission in this section. building on lessons learned from our news task ablation studies, we opted to use the transformer-xlarge architecture, and average all of the intermediate model checkpoints which helps reduce model variance as a consequence of finetuning. tables 6 and 7 show our results as we iterated on improving our models. we trained our bpe tokenizer on biomedical data to mitigate character-level segmentation of words unique to the biomedical domain. we found this had a minimal effect on model performance. this could be because the majority of our parallel biomedical data was selected from news task training data, and thus biomedical words were already adequately accounted for by the news task model's tokenizer. we found that up-sampling in-domain biomedical data hurt performance compared to concatenating out-of-domain and in-domain data with no up-sampling. for the @xmath8, including any biomedical domain data during initial model training showed improvements over training on exclusively news task data. up-sampling indomain data for the @xmath9. unsurprisingly, finetuning base models on biomedical domain data improved bleu scores for all models. in-domain finetuning helped models initially trained on news task data overcome performance gaps between themselves and models that had seen a higher amount of biomedical data during training. neither shallow fusion nor noisy channel re-ranking improved model performance after ensembling for the @xmath10. both techniques individually improved @xmath11. ensembling our models led to an additional performance boost and allowed us to reach our maximum @xmath12.3 and @xmath13.8. these scores show a 0.9 and 0.5 improvement over last year's best score of 39.4 and 43.3 @xcite respectively. ## conclusion we present neural machine translation systems for the en ↔ de news task and en ↔ ru news and biomedical shared tasks implemented in the nemo framework @xcite . our systems build on the transformer sequence-tosequence model to include backtranslated text and data from right-to-left factorized models, ensembling, finetuning, mining biomedically relevant data using domain classifiers, shallow fusion with lms, and noisy channel re-ranking. these achieve competitive performance to submissions from previous years.