The N iu T rans Machine Translation Systems for WMT 21 (2021)

Abstract: This paper describes NiuTrans neural machine translation systems of the WMT 2021 news translation tasks. We made submissions to 9 language directions, including English2Chinese, Japanese, Russian, Icelandic and English2Hausa tasks. Our primary systems are built on several effective variants of Transformer, e.g., Transformer-DLCL, ODE-Transformer. We also utilize back-translation, knowledge distillation, post-ensemble, and iterative fine-tuning techniques to enhance the model performance further.

## introduction our niutrans team participated in the wmt 2021 news translation shared tasks, including english↔chinese (en↔zh), english↔japanese (en↔ja), english↔russian (en↔ru), english↔icelandic (en↔is) and @xmath0, nine submissions in total. all of our systems were built with constrained data sets. we adopt some effective models and useful methods, which have been witnessed the success in previous papers @xcite @xcite @xcite @xcite @xcite . to enhance the performance of the single model, we choose pre-normalized transformer-dlcl @xcite and ode-transformer @xcite as the backbone. all systems are built upon the relative position representation @xcite due to its strong performance when models are deep @xcite . for the system combination, we adopt the post-ensemble @xcite to find the most similar hypothesis among several ensemble outputs, which could be regarded as a reranking technique without pre-training. previous works have emphasized the importance of diversity when building ensemble systems. besides the architecture diversity, we also adopt iterative ensemble knowledge distillation leveraging the source-side monolingual data to enlarge the diversity. more details please refer to @xcite . our data preparation pipeline consists of threefold: (i) for the data filtering. we use a stricter cleaning process than last year @xcite ). details will be discussed in section 2.1. (ii) for the data augmentation, both iterative backtranslation @xcite ) method, and iterative knowledge distillation @xcite method are employed to take the full advantage of monolingual data provided by the wmt organization. in the back-translation stage, we leverage target-side monolingual sentences to generate source-side pseudo sentences and use a nucleus sampling @xcite decoding strategy to improve the generalization ability. furthermore, we leverage in-domain source-side monolingual data by applying iterative knowledge distillation. (iii) for data selection, it's hard to find massive in-domain data for low-resource languages to train a neural language model, so we use a statistical n-gram language model (xenc toolkit3 foot_0 ) instead. domain finetuning is quite essential to improve the translation system given a certain target domain. we use domain adaptation to migrate the models from the general domain to the news domain by iterative finetuning. after in-domain finetuning, we use multiple ensemble combinations by the postensemble method. this paper is structured as follows: in section 2, we introduce several effective techniques, including data preprocessing, deeper and wider transformer models, iterative back-translation, iterative knowledge distillation, fine-tuning and postensemble. in section 3, we show the experiment settings and report the experimental results of the validation set (newstest2020). finally, we draw the conclusion in section 4. ## conclusion this paper introduced our submissions on wmt21 nine tasks. our main exploration is using a new effective architectures ode transformer and utilizing post-ensemble technology to enhance the system. and we experimented with iterative backtranslation by different decoding strategies, iterative knowledge distillation, iterative fine-tuning, model ensembling and post-ensemble.