D i D i’s Machine Translation System for WMT 2020 (2020)

Abstract: This paper describes the DiDi AI Labs’ submission to the WMT2020 news translation shared task. We participate in the translation direction of Chinese->English. In this direction, we use the Transformer as our baseline model and integrate several techniques for model enhancement, including data filtering, data selection, back-translation, fine-tuning, model ensembling, and re-ranking. As a result, our submission achieves a BLEU score of 36.6 in Chinese->English.

## introduction we participate in the wmt2020 news translation shared tasks in @xmath0. for this translation direction, we train several variants of transformer @xcite models on the provided parallel data enlarged with synthetic data from monolingual data. we experiment with several techniques proposed in the past translation tasks and adopt effective ones as components of our system. our data preparation pipeline consists of data filtering, data augmentation, and data selection. for data filtering, we filter sentence pairs based on language model scoring, alignment model scoring, etc. for data augmentation, we experiment with iterative back-translation @xcite methods and iterative knowledge distillation @xcite methods. we leverage source-side monolingual data by applying iterative knowledge distillation, and target-side monolingual data by back-translation methods, including greedy search, beam search, and noised beam search. for data selection, we select an in-domain corpus with n-grams language models and binary classifiers. a tri-gram token-level language model and a bi-gram character-level language model are introduced for english and chinese respectively. out-of-domain sentences which have similar scores as in-domain sentences are chosen. we also treat data selection as a text classification problem, and use bert @xcite as the basic classifier. in this way, we collect a corpus of high-quality in-domain training data, which improves translation performance significantly. to enhance a single model, we use several variants of transformer, including transformer with relative position attention @xcite , transformer with larger feedforward inner (ffn) size (8, 192 or 15, 000), and transformer with reversed source. we then ensemble these models with adequate model diversity and data diversity to further improve the performance. domain conflicts influence the translation performance significantly. for example, there exist differences between written english and spoken english. usually, a model cannot do the best in all domains due to the conflicts. in this work, we propose to obtain domain information with unsupervised clustering and exploit this information for translation. specifically, we partition the training data, dev data, and test data into different clusters, and translate each cluster part of the test set with the model fine-tuned on the corresponding training set. exploiting domain information helps improve the translation significantly. details will be discussed in section 3. this paper is structured as follows: section 2 describes variants of transformer we used in the competition. in section 3, we introduce several techniques for model enhancement, including data filtering, back-translation, fine-tuning, model ensembling. section 4 presents experimental settings, results and analysis. finally, in section 5 we draw a brief conclusion of our work in the wmt2020. ## conclusion in this paper, we present our nmt systems for wmt2020 news translation shared tasks in @xmath1. our final system achieves substantial improvement over baseline systems by integrating the following techniques: