GTCOM Neural Machine Translation Systems for WMT 20 (2020)

Abstract: This paper describes the Global Tone Communication Co., Ltd.’s submission of the WMT20 shared news translation task. We participate in four directions: English to (Khmer and Pashto) and (Khmer and Pashto) to English. Further, we get the best BLEU scores in the directions of English to Pashto, Pashto to English and Khmer to English (13.1, 23.1 and 25.5 respectively) among all the participants. Our submitted systems are unconstrained and focus on mBART (Multilingual Bidirectional and Auto-Regressive Transformers), back-translation and forward-translation. Also, we apply rules, language model and RoBERTa model to filter monolingual, parallel sentences and synthetic sentences. Besides, we validate the difference of the vocabulary built from monolingual data and parallel data.

## introduction we participated in the wmt shared news translation task and focus on the bidirections: english and khmer, english and pashto. we applied fairseq @xcite as our develop tool and use transformer @xcite as the main architecture. the primary ranking index for submitted systems is bleu @xcite , therefore we apply bleu as the evaluation matrix for our translation system. for khmer, we use polyglot foot_0 as the tokenizer before evaluation. for data preprocessing, the basic method includes punctuation normalization for all language. further, according to the different language characteristics. tokenization, truecase and byte pair encoding (bpe) @xcite are applied for english, and sentencepiece @xcite is applied for khmer and pashto. besides, human rules, language model and roberta model @xcite are also involved to clean parallel data, monolingual data and synthetic data. regard to the techniques on model training, backtranslation @xcite and forwardtranslation are applied to verify whether these techniques could improve the translation performance especially in low-resource condition. we all know that it is more difficult to train a model in low-resource condition, because it suffers from data sparsity and out-of-vocabulary problem. normally knowledge distillation @xcite is a good way to generate synthetic data. but in this task we suppose that knowledge distillation can only generate 100 thousand to 1 million parallel sentences due to the size of provided data. therefore, we use forward-translation with monolingual data to generate more synthetic data. here forword-translation refers to translate the source sentences to target language, and clean synthetic data. this paper is arranged as follows. we firstly describe the task and show the data information, then introduce how we do data filtering, including human rules, language model and roberta model. after that, we describe the techniques on low-resource condition and show the conducted experiments in detail of all directions, including data preprocessing, model architecture, back-translation and forwor-translation. at last, we analyze the results of experiments and draw the conclusion. ## task description the task focuses on bilingual text translation in news domain and the provided data is show in ## data filtering the methods of data filtering are mainly the same as we did in last year @xcite , including human rules and language model. further, another methods we used this year are as follows: clean repeated translation sentences in synthetic data. for example, we often see the translation like: i want to eat an apple apple apple apple apple, when translating a source language sentence with repeated words until the end of the sentence. in this task, we made a simple clean strategy which is to remove the sentences that repeat one word four times, two words three times or three words two times. clean synthetic data by roberta model. in order to clean synthetic data, especially from forwordtranslation, we represent the source and target sentences by roberta model and calculate the cosine distance. remove the sentences with low score or without translation (the source sentence and target sentence are same). ## forward-translation in low-resource condition, out-of-vocabulary is a problem. there is a difference between the test scenario and training scenario, which means the words appear in test set may be not existed in training vocabulary. back-translation is a common way to extend the word vocabulary. however, with the generated synthetic data from back-translation, only target vocabulary can be enriched.