The N iu T rans Machine Translation Systems for WMT 22 (2022)

Abstract: This paper describes the NiuTrans neural machine translation systems of the WMT22 General MT constrained task. We participate in four directions, including Chinese→English, English→Croatian, and Livonian↔English. Our models are based on several advanced Transformer variants, e.g., Transformer-ODE, Universal Multiscale Transformer (UMST). The main workflow consists of data filtering, large-scale data augmentation (i.e., iterative back-translation, iterative knowledge distillation), and specific-domain fine-tuning. Moreover, we try several multi-domain methods, such as a multi-domain model structure and a multi-domain data clustering method, to rise to this year’s newly proposed multi-domain test set challenge. For low-resource scenarios, we build a multi-language translation model to enhance the performance, and try to use the pre-trained language model (mBERT) to initialize the translation model.

## introduction we participate in the wmt22 general mt task, including @xmath0, @xmath1, and livonian↔english (liv↔en) in four directions. all of our systems are built with constrained data sets. we adopt some methods that have been proven to work well in wmt over the past few years @xcite @xcite . at the same time, we also adopt some new model structures @xcite , data clustering @xcite , initialization @xcite , and training methods @xcite , which are described in detail below. for data preparation and augmentation, since filtering data could hurt the model performance on the general domain machine translation task, we apply several soft data filtering rules to preserve as much data as possible @xcite . to obtain the in-domain data, we use the open-source toolkit xenc @xcite and specially try a domain clusters method based on the bert @xcite and roberta @xcite model in the @xmath2. we also use back-translation @xcite and knowledge distillation @xcite iteratively to increase the size of in-domain data, which has been proved effective in recent years @xcite . for model architectures, our system is built on several transformer variants, including transformer-rpr, transformer-dlcl @xcite , transformer-ode @xcite , transformer-umst @xcite , and transformer-based model with domain mixing @xcite . we build a wide and deep model based on the pre-norm structure @xcite and relative position representation(rpr) @xcite , inspired by the effectiveness of the deep model. furthermore, we select four single models to build the ensemble model for better performance. particularly, in the en↔liv direction, we build a multilingual machine translation system @xcite based on the above models. for model initialization, training, and decoding strategies, we use nucleus sampling(top-p) (holtzman et al., 2020), top-k sampling(top-k), and beam search as decoding methods in all languages. at the same time, we adopt scheduling sampling @xcite in @xmath3. furthermore, we attempt to initialize the translation model with the pre-trained language model based on lightweight adapter @xcite in the en↔liv direction. based on the softer filtering rules and appropriate hypo-parameter settings, we achieve better results on the deep model than last year. in the @xmath4, fine-tuning with the normal training and the scheduling sampling also obtain good results. furthermore, we use an unsupervised multi-domain data clustering method and some simple domain classification methods. however, we find no significant domain differences in the constrained data. initializing the translation model with the pre-trained model leads to poor performance in the en↔liv direction. it may be due to the sensitivity to the size of the training set. the rest of the paper is organized as follows: in section 2, we describe our system in detail, including the data preprocessing and filtering, model structure, back-translation and knowledge distillation, fine-tuning, and post-editing. in section 3, we introduce our experimental settings and results according to different tasks and give a brief analysis. in section 4, we summarize our work. ## system overview in figure 1 , we describe the whole process of our system. we use three different colors to represent the different translation tasks. at the data preparation stage, we perform several data processing methods to obtain the training set. then, we train several models with different structures and use back-translation(bt) and knowledge distillation(kd) iteratively based on ensemble model. finally, we obtain our final submission based on fine-tuning and post editing. ## conclusion this paper introduces our submissions on wmt22 in four directions. we train our system with constrained data in all directions. the system is constituted by the ensemble model based on multiple deep models. for training data, we use a softer data filtering method to obtain more data and make the model more robust in the general domain. based on this data, model performance is better than our last year's systems. we use iterative back-translation and knowledge distillation methods which have been proven to be very effective in the past. in addition, fine-tuning using both normal training and scheduling sampling also achieves good results. in the @xmath5, we mainly build the news domain translation model. also, we try the multi-domain data clustering method and multi-domain adaptation method to build the multidomain model. however, because the sentences in constrained data have no noticeable domain difference, the performance of the above method is not satisfactory. in the en↔liv direction, we try the multilingual model and initialization method, which initialize the translation model with the pretrained model. we find that the multilingual model show more considerable potential than the model initialized with mbert, even under minimal data.