Manifold’s E nglish- C hinese System at WMT 22 General MT Task (2022)

Abstract: Manifold’s English-Chinese System at WMT22 is an ensemble of 4 models trained by different configurations with scheduled sampling-based fine-tuning. The four configurations are DeepBig (XenC), DeepLarger (XenC), DeepBig-TalkingHeads (XenC) and DeepBig (LaBSE). Concretely, DeepBig extends Transformer-Big to 24 encoder layers. DeepLarger has 20 encoder layers and its feed-forward network (FFN) dimension is 8192. TalkingHeads applies the talking-heads trick. For XenC configs, we selected monolingual and parallel data that is similar to the past newstest datasets using XenC, and for LaBSE, we cleaned the officially provided parallel data using LaBSE pretrained model. According to the officially released autonomic metrics leaderboard, our final constrained system ranked 1st among all others when evaluated by bleu-all, chrf-all and COMET-B, 2nd by COMET-A.

## introduction this report describes manifold's machine translation system submitted to wmt 22 @xmath0. the general domain translation task of wmt is a new task set up this year, replaces the time-honored news translation task. however, as the newstest datasets released previously were created by professional translators manually, they were considered to have better quality compared with the web crawled dataset. therefore, generally our strategy is selecting data from the provided datasets according to their similarity between past wmt test sets, and training big models with different architectures or training methods based on the selected data. ## data preprocessing for officially released bilingual data, we merged paracrawl v9, news commentary v16, wiki titles v3, un parallel corpus v1.0, ccmt corpus and wikimatrix corpus together, then applied multiple rules to preprocess and filter the officially released bilingual data, including • normalizing punctuation. ## basic models after applying the aforementioned preprocessing and filtering rules, 30m segment pairs were kept foot_3 . for the kept data, we adopted two different selection strategies: config item value dropout 0.3 learning rate 0.0005 max tokens 4096 warmup init lr 1e-7 warmup steps 4000 label smoothing 0.1 num max updates 300,000 update frequency 8 we applied bpe-subword @xcite foot_5 to divide tokens into subwords. bpe codes were learned jointly with 32k merge operations but dictionaries for the source language and target language are generated separately. basic models were trained to generate synthetic, pseudo bilingual segment pairs for the next step. as is discovered by @xcite , sub-model diversity is a key factor to enhance the performance of ensemble model. therefore, we trained various transformer models @xcite applying different architectures. for very deep transformers, we followed the suggestion given by dlcl @xcite to use pre-norm. main configurations for the architectures we trained are listed in for each architecture x listed in we selected 12m sentences from officially released english and chinese monolingual datasets respectively foot_7 , also applying xenc algorithm on them. the selected monolingual data was preprocessed by the similar pipeline presented in section 2, with the difference on skipping the steps to filter unaligned bilingual data. the reason behind selecting 12m sentences is we expect that the data from each monolingual dataset has the same size compared with the bilingual training data, as @xcite indicated. the preprocessed chinese monolingual data was no. method data size newstest 2020 newstest 2021 1 deep baseline model 12m 42.7 32.4 2 1 + forward-translation (fort) 24m 44.7 (+2.0) 33.3 (+0.9) 3 1 + top-p back-translation (topp bt) 24m 45.8 (+3.1) 32.9 (+0.5) 4 1 + r2l kd + r2l fort 36m 44.2 (+1.5) 33.5 (+1.1) 5 sequence kd 12m 42.7 (-) 32.5 (+0.1) 6 5 + fort + topp bt 36m 45.3 (+2.6) 33.7 (+1.3) 7 4 -1 + 6 60m 45.5 (+2.8) 33.8 (+1.4) the preprocessed english monolingual data was forward-translated into chinese, leading to another 12m english-pseudo chinese dataset. the ensemble model used to generate data has the same architecture with the back-translation model introduced above, the only difference is it is trained by the parallel xenc @xmath1. we further applied sequence-level knowledge distillation @xcite to distill this ensemble model by translating english sentences of the parallel xenc corpus into pseudo chinese. furthermore, we also trained a deep model with the reversed parallel xenc data to get a right-to-left (r2l) model, and generated forward-translation and the results of knowledge distillation using reversed monolingual and parallel english corpora. after having acquired these different synthetic datasets, we trained deep models by various combinations of them. the concrete model performance is shown in ## fine-tuning and bigger models after having acquired extra data by back/forwardtranslation, knowledge distillation, and r2l data augmentation, we experimented on several other methods to further improve our system. fine-tuning. we fine-tuned our deep model using the combination of newsdev2017, newstest2017, newstest2018, and newstest2019 datasets. as the dataset used for fine-tuning is quite small, we applied decoder steps based scheduled sampling @xcite as a means of regularization. we set k to 0.99. with this step, a 0.7 bleu gain has been brought on the validation set and 0.8 bleu gain on the test set. ensemble of bigger models. we trained three bigger models on the 60m xenc dataset (configuration no. 7 in ## conclusion in this report, we describe our manifold @xmath2. the core idea of our system is to train various big transformer models utilizing in-domain (actually news domain) data, based on which an ensemble model is created. although most training data belongs to a special domain, we still achieved compelling results in the final submission, i.e. our final system ranked first among the constrained systems, evaluated by bleu score based on the two references.