SJTU - NICT ’s Supervised and Unsupervised Neural Machine Translation Systems for the WMT 20 News Translation Task (2020)

Abstract: In this paper, we introduced our joint team SJTU-NICT ‘s participation in the WMT 2020 machine translation shared task. In this shared task, we participated in four translation directions of three language pairs: English-Chinese, English-Polish on supervised machine translation track, German-Upper Sorbian on low-resource and unsupervised machine translation tracks. Based on different conditions of language pairs, we have experimented with diverse neural machine translation (NMT) techniques: document-enhanced NMT, XLM pre-trained language model enhanced NMT, bidirectional translation as a pre-training, reference language based UNMT, data-dependent gaussian prior objective, and BT-BLEU collaborative filtering self-training. We also used the TF-IDF algorithm to filter the training set to obtain a domain more similar set with the test set for finetuning. In our submissions, the primary systems won the first place on English to Chinese, Polish to English, and German to Upper Sorbian translation directions.

## introduction our sjtu-nict team participated in the wmt20 shared task, including supervised track, unsupervised, and low-resource track. during the participation, we placed our attention on polish (pl) → english (en) and english (en) → chinese (zh) on the supervised track, while on the unsupervised and low-resource track, the german (de) ↔ upper sorbian (hsb) both directions are focused. our baseline system in supervised track is based on the transformer big architecture proposed by @xcite , in which its opensource implementation version fairseq @xcite ) is adopted. in the unsupervised and low-resource track, we draw on the successful experience of the xlm framework @xcite , and used the two-stage training mode of masked language modeling (mlm) pre-training + back-translation (bt) finetune to obtain a very strong baseline performance. marian @xcite toolkit is utilized for training the decoder in reranking using machine translation targets instead of common gpt-style language modeling targets. in order to better play the role of wmt evaluation in polishing the methods proposed or improved by our team @xcite @xcite @xcite @xcite , we divided the three language pairs we participated in into three categories: 1. traditional language pair with rich parallel corpus: en-pl, 2. language pair with document-level information: en-zh, 3. language pair with no or low parallel resources: de-hsb. in the supervised @xmath0, we based on the xlm framework to pre-train a polish language model using common crawl and news crawl monolingual data, and proposed the xlm enhanced nmt model inspired from the idea of incorporating bert into nmt @xcite . besides, we trained a bidirectional translation model of en-pl based on the parallel corpus and further finetuned it to the @xmath1. in the supervised @xmath2, we propose a document enhanced nmt model based on longformer @xcite . the training of our proposed document enhanced nmt model is split into three stages. in the first stage, we pre-train the longformer document encoder with mlm target on the document text in wikipedia dumps, un news, and news commentary monolingual corpus. a conventional transformer-big nmt model is trained in the second stage. in the final stage, the longformer encoder and conventional transformer big nmt model are used to initialize the full document-enhanced nmt model parameters, in which the longformer encoder is adopted to extract representations for the document of an input sequence, and then the document representations are fused with each layer of the encoder and decoder of the nmt model through attention mechanisms. in the unsupervised machine translation track on de-hsb, we experimented with the reference language based unmt (runmt) @xcite framework we proposed recently. under this framework, we choose english as the reference language, and use the europarl parallel corpus of en-de to enhance the unsupervised machine translation between de and hsb. specifically, we adopted reference language translation (rat), reference language back-translation (rabt), and cross-lingual back-translation (xbt) three training targets with the help of the cross-lingual agreement provided by the en-de parallel corpus to enhance the unsupervised translation performance. due to the introduction of more explicit supervision signals brought by parallel corpus in the low-resource machine translation track on de-hsb, we discarded the use of the weaker agreement provided by the reference language, conducted joint training on the unsupervised back-translation and the supervised (forward-)translation directly, and introduced bt-bleu based collaborative filtering technology for further self-training. in addition, inspired by our previous work @xcite , we also use mlm and translation language modeling (tlm) to continue pre-training the model while machine translation training. in addition, in all basic nmt models, we empower the training process with our proposed data-dependent gaussian prior objective (d2gpo) @xcite , so that the model can maintain the diversity of the output. when the main model training is finished, the tf-idf algorithm is employed to filter the training set according to the input of the test set, a training subset whose domain is more similar to the test set is obtained, and then used to finetune the model for reducing the performance degradation caused by domain inconsistency. for the final submission, an ensemble of several different trained models outputs the n-best predictions, and used the decoder trained with marian toolkit to performs reranking to get the final system output. ## data preprocessing and model setup before model training, we preprocessed the data uniformly and customized the processing according to the requirements of each model. we normalized punctuation, remove non-printing characters, and tokenize all data with the moses tokenizer @xcite except for the chinese. for chinese, we removed the segmentation space in some training data and then use pkuseg @xcite toolkit to cut all chinese sentences, so as to obtain unified word segmentation annotations. we use joint byte pair encodings (bpe) with 40k split operations for subword segmentation @xcite . in xlm-enhanced nmt and documentenhanced nmt, we first train a basic nmt (transformer big) model on the sentence-level data until convergence, then initialize the encoder and decoder of the xlm-enhanced nmt and document-enhanced nmt full model with the obtained model. the plm-encoder attention attn p and plm-decoder attention attn pc are randomly initialized. en-pl on the language pair en-pl, we explored performance in two training data settings. the first is base data, including europarl v10, tilde rapid corpus, and wikimatrix bitext data, whose raw data is on the sentence-level. in the second setting base data + paracrawl, we converted the paragraph-level alignment data in paracrawl to sentence-level alignment and incorporated it with the base data. in the conversion process, we adopted the method and program proposed by @xcite for aligning sentences based on a simple statistical model of character lengths, which uses the fact that longer sentences in one language tend to be translated into longer sentences in the other language, and that shorter sentences tend to be translated into shorter for the polish pre-trained xlm language model, we used all newscrawl monolingual data and some commoncrawl monolingual data. since the commoncrawl data is very large and noisy and can potentially decrease the performance of lm if it is used in its raw form. we apply language identification filtering (langid; @xcite ), keeping sentences with correct languages. in order to filter out the sentences shorter than 5 words or longer than 150 words more precisely, we re-split sentences using spacy @xcite toolkit. en-zh in en-zh, the pre-training of longformer as a document encoder is unique. as described in @xcite , the longformer needs a large number of gradient updates to learn the local context first; before learning to utilize longer context. in the first phase of the staged training procedure, an initial roberta @xcite model implemented in fairseq @xcite repository was trained on the sentence-level text available. in each subsequent phase, we trained the model on the paragraph text, doubled the window size and the sequence length, and halve the learning rate. for the paragraph text, the wikidumps and newscommentary v15 have document intervals and can be used directly, while un v1.0 has no document intervals but the sentence order is not interrupted. therefore, we use the bert next sentence prediction (nsp) classification model provided by google for document interval prediction to recover the documents. ## results and analysis results and ablations for @xmath3, @xmath4, unsupervised de↔hsb in in @xmath5, the introduction of paracrawl data improves the baseline performance on the dev dataset by about 4.2 bleu. +d2gpo, xlm-enhanced nmt, bidirectional nmt, and ensembling outperforms our strong baseline by 2 bleu point. finally, finetuning and reranking further gives another 0.5 bleu. for @xmath6, as with @xmath7, we see similar improvements with +d2gpo, xlm-enhanced nmt, ensembling and reranking. we also observe that the addition of document-enhanced nmt is much more substantial, improving single model performance by over 1.5 bleu. in the unsupervised track, we compared clm, mlm, and explicit sentence compression (esc) pre-training approaches joint trained with bt in the second stage of unmt, respectively, and found that mlm and esc had similar effects and were stronger than clm. moreover, the pretraining baseline of mlm was stronger than that of mass. the combination of unsupervised training of de-hsb and supervised training of en-de achieves the purpose of transfer learning, and the improvement is greater than 3 bleu. based on the conclusion of mlm and bt joint training on the unmt baseline, we also got a similar in the low-resource track, the model in the unsupervised track is used as the pre-trained model, and de-hsb nmt and bt are jointly trained. due to the de-hsb parallel corpus, we can not only use mlm for monolingual pre-training, but also use tlm for cross-lingual pre-training. the addition of cfst and d2gpo further improves the effect of the model, indicating that these contributions are orthogonal. in addition, comparing unmt with munmt given a parallel corpus, we found that although munmt used more data, it did not bring about a large enough effect improvement, so we will leave it for future research. ## conclusion this paper describes sjtu-nict's submission to the wmt20 news translation task. for three typical scenarios, we adopt different strategies. in this work, we not only study the pre-trained language model to enhance mt, but also consider the impact of document information on translation. we considered both the way of converting document alignment into sentence alignment and the use of bert's nsp to recover the structure of documents. in addition, transfer learning from supervision is taken into account in unsupervised translation, and various means are used to enhance low-resource translation. our systems performed strongly among all the submissions: we ranked 1st in @xmath8, @xmath9, and @xmath10, and stayed top-3 for the @xmath11.