The Volctrans Machine Translation System for WMT 20 (2020)

Abstract: This paper describes our submission systems for VolcTrans for WMT20 shared news translation task. We participated in 8 translation directions. Our basic systems are based on Transformer (CITATION), into which we also employed new architectures (bigger or deeper Transformers, dynamic convolution). The final systems include text pre-process, subword(a.k.a. BPE(CITATION)), baseline model training, iterative back-translation, model ensemble, knowledge distillation and multilingual pre-training.

## introduction we participated in the wmt2020 shared news translation task in 14 directions: english↔chinese, english↔german, french↔german, english↔polish, english↔tamil,english↔pashto,english↔khmer, covering language pairs from high to low resources. in this year's translation task, we mainly focus on exploiting self-supervised and unsupervised methods for nmt to make full use of the monolingual data @xcite . we aims at building a general training framework which can be well applied to different translation directions. our models are mainly based on the transformer @xcite . techniques used in the submitted systems include iterative back-translation, knowledge distillation. we also employed several tricks to improve in-domain bleu scores, typically in-domain transfer learning. we also experimented with a multilingual pretraining technique which we proposed recently @xcite . ## baseline models we apply two different nmt skeletons for the shared news translation as our baseline systems. ## settings and results for all news tasks, all paracrawl corpus is cleaned by the script proposed by @xcite . we trained the baseline using the sampling method described in section 3.1 with different architectures showed in section 2. for the low resource language pair (english↔pashto), as showed in section 3.1, we pretrained three multilingual models with different model architectures (dlcl 25layers, transformer 15000ffn and dynamic convolution 25e6d) on all parallel data available in wmt20 except the english↔chinese to avoid a large dictionary 1 and fine-tuned the pre-trained models on the their own parallel data with different data sampling strategies to get 9 baseline models 2 . then we applied the tag back-translation, joint training, knowledge distillation and random ensemble methods as described in section 3 to get the final translation system. all bleu scores were reported with sacrebleu @xcite . ## conclusion this paper describes volctrans's nmt systems for the wmt20 shared news translation task. for all directions, we almost adopted the same strategies, except for low-resource language pairs, we employed multilingual pre-training to boost the baseline models. we found that splitting the monolingual data into disjoint parts is an effective way to increase data diversity among single models, which is an important premise for building strong ensemble models. our final systems achieved significant improvements, usually 3 to 5 bleu scores, over baseline systems by integrating techniques such as tagged back-translation, iterative back-translation, random ensemble, knowledge distillation.