The N iu T rans Machine Translation Systems for WMT 20 (2020)

Abstract: This paper describes NiuTrans neural machine translation systems of the WMT20 news translation tasks. We participated in Japanese<->English, English->Chinese, Inuktitut->English and Tamil->English total five tasks and rank first in Japanese<->English both sides. We mainly utilized iterative back-translation, different depth and widen model architectures, iterative knowledge distillation and iterative fine-tuning. And we find that adequately widened and deepened the model simultaneously, the performance will significantly improve. Also, iterative fine-tuning strategy we implemented is effective during adapting domain. For Inuktitut->English and Tamil->English tasks, we built multilingual models separately and employed pretraining word embedding to obtain better performance.

## introduction this paper describes the niutrans submissions to the wmt20 news tasks, including @xmath0, @xmath1, @xmath2. some useful methods in the wmt18 @xcite and wmt19 @xcite submissions are also reused this time, such as model ensemble, knowledge distillation (kd) et al., and we explore some novel approaches this year. for this participation, we experimented with some deeper and wider transformer @xcite architectures to get reliable baselines, nucleus sampling @xcite in backtranslation to generate more suitable pseudo bilingual sentences, more effectively fine-tuning strategy to adapt domain. particularly in the lowresources tasks, {ta,iu}→en, we built multilingual neural machine translation by using some similar language to get better performance and further replaced decoder's word embedding by an english pretraining transformer language model's which trained by two monolingual in-domain data corpora. furthermore, we presented a new fine-tuning pattern which could significantly improve the bleu score on the test set, and it worked well on all five tasks whether it is a low or rich resource. we carefully rethought this strategy and found the main gain came from domain adaptation and improved inferior translations. our systems and this paper followed six main steps:1) data preprocessing and filter, 2) iterative back-translation to generate pseudo bilingual data, 3) using different model architectures to enhance the diversity of translation, 4) iterative knowledge distillation by in-domain monolingual data, 5) iterative fine-tuning with in-domain using small training batch, 6) translation post-process. ## conclusions this paper introduced our submissions on wmt20 five tasks and our main exploration is using more diversified architectures, improving a iterative finetuning strategy and utilizing several similar languages to build a multilingual model on lowresource tasks. and we experimented with iterative back-translation by different decoding strategies, using pre-trained embeddings in multilingual models. on the whole, all of our systems performed competitively and ranked 1st on ja↔en both sides.