OPPO ’s Machine Translation System for the IWSLT 2020 Open Domain Translation Task (2020)

Abstract: In this paper, we demonstrate our machine translation system applied for the Chinese-Japanese bidirectional translation task (aka. open domain translation task) for the IWSLT 2020. Our model is based on Transformer (Vaswani et al., 2017), with the help of many popular, widely proved effective data preprocessing and augmentation methods. Experiments show that these methods can improve the baseline model steadily and significantly.

## introduction machine translation, proposed even before the first computer was invented @xcite , has been always a famous research topic of computer sci ence. in the recently years, with the renaissance of neural network and the emergence of atten tion mechanism @xcite , the old area has stepped into a new era. furthermore, the transformer architecture, after being published, has immediately attracted much attention nowadays and is dominating the whole field now. although transformer has achieved many sota results, it has tremendous amount of param eters so is hard to be fit on small datasets, there fore it has a high demand on good and large data source. despite of the lack of high quality data of parallel corpus, the document level comparable data is relatively easy to be crawled, so exploring an effective and accurate way of mining aligned sentence pairs from such large but noisy data, to enrich the small parallel corpus, could benefit the machine translation system a lot. besides, cur rently, most open access big volume machine trans lation datasets are based on english, and many of them are translated to/from another european lan guage -as many popular european languages are fusional languages, most corpus are composed by two fusional languages together. to understand whether the existing model architectures and train ing skills can be applied on the translation between asian languages and other type of languages , such as between an analytic language like chinese and an agglutinative language like japanese, interest us. in this paper, we demonstrate our system ap plied for the iwslt 2020 open domain text transla tion task, which aims to translate chinese from/to japanese foot_0 . besides describing how we trained the model that is used to generate the final result, this paper also introduces how do we mine extra paral lel sentences from a large but noisy data released by the organizer, and several experiments inspired by the writing systems of chinese and japanese. ## data preprocessing however, per our investigation, even the sen tences in the existing parallel file are actually not fully aligned. for example, a sentence "1994 年 2 月、ジャスコはつるまいの全株式を取得。" (means "jasco bought all shares in february, 1994" ) in the corpus is translated into "次年 6 月乌迪 内斯买断了他的全部所有权。", which means "udinese bought out all his ownership in june in the next year", so here is clearly a noise. since deep neural network demands high quality input data, we combined the filtered file and the existing parallel file into a 20 million pairs dataset (noted as combined dataset afterwards) , and made a further data preprocessing, including two main steps: ## main task experiments taking the 5.4m corpus in the hand, we further di vided the words in the text into subwords @xcite . bpe code is trained on the chinese and japanese corpus jointly, with 32,000 merging operations, but the vocabulary is extracted for each language individually, so for chinese the size of its vocabulary is 31k and for japanese it is 30k. vocab ularies for both two directions (ja-zh and zh-ja) are shared. we trained 8 heads transformer big models with facebook fair's fairseq @xcite using the following configuration: the first model trained on the filtered genuine parallel corpus (i.e. the 5.4m corpus) is not only seen as the baseline model of the consequent exper iments, but also used as a scorer foot_2 . we rescored the alignment scores of the sentences using this model, and again filtered away about one quarters data. the model trained on the refined data improved the bleu score by 1.7 for zh-ja and 0.5 for ja-zh. as many works proved, backtranslation (bt) @xcite is a common data augmen tation method in the machine translation research. besides, @xcite ) also provides some other ways to backtranslate. we applied both of them and in our experiments, top10 sampling is effective on zh-ja direction and for ja-zh tradi tional argmaxbased beam search is still better. firstly, 4m data in the original corpus is se lected by the alignment score and translated by the models (models for the different directions) got in the previous step to build synthetic corpus, then for each direction a new model is trained on the augmented dataset (contains 5.4m + 4m + 4m = 13.4m pairs). to get a better translation result, we used ensemble model to augment the dataset. one more thing could be clarified that, in this aug mentation step we not only introduced 4m back translated data, but also generated 4m synthetic tar get sentences by applying knowledge distillation (kd) @xcite . on this genuinebtkd mixture dataset, we tried one more round of backtranslation and knowledge distillation, but just saw a minor im provement. afterwards we trained language model on the 5.4m parallel corpus for each lan guage using kenlm @xcite . with the help of the language model, 3m chinese sentences and 4m japanese sentences with the highest scores are selected from the unaligned monolingual cor pus as the new input of bt models, augmented the mixture dataset to 20.4m pairs (noted as final augmented dataset, which will be referenced later in the report), and we did another round of back translation and knowledge distillation. after these three rounds iterative bt @xcite and kd, several best single models are further com posed together to an ensemble model. in the last step, following @xcite , we use both back ward model (for zh-ja task, model from ja-zh is its backward model, and vice versa) and trans former language model to rerank the nbest candi dates of the output from the ensemble model, to generate the final results. detailed results on the dev dataset of each inter mediate step is shown in ## branched task experiments besides the main task experiments demonstrated in the previous section, as the introduction part says, we are also interested in how to mine or extract par allel data from such huge but noisy datasets, and explore some special skills on translating from chi nese to japanese (and also vice versa). this section will mainly discuss our work on these two parts. ## final results from the evaluation results provided by the orga nizer officially, our bleu score for jazh direc tion is 32.9, for zhja is 30.1. however, per our observation on the dev dataset, we found most of the numbers and latin words are styled in full width characters, so we made an extra step in postprocessing to convert all splitting method zh-ja bleu ja-zh bleu zhja baseline 39.0 35.1 split cjk chars only 37.9 (1.1) 35.4 (+0.3) + katakanas 37.9 (1.1) 34.9 (0.2) + hiraganas 38.2 (0.8) 35.3 (+0.2) scores shown in the table are characterlevel bleu calcu lated by multibleu ( https://github.com/moses smt/mosesdecoder/blob/master/scripts/generic/multi bleudetok.perl). our results are styled in bold and the contrastive one is marked with an additional asterisk symbol * . at the date of submitting the cameraready version report, the leaderboard hasn't marked which system(s) is/are unconstrained the numbers and latin words in our final submis sion of zhja to full width characters. for exam ple, "2008" was converted to "２００８" 7 . our contrastive result, in which all the numbers and latin words are composed by half width characters (and this is the only difference compared with the primary submission we made), was scored 34.8, gained an improvement of nearly 5 points. the contrastive result is generated by the same model we trained on the constrained dataset. all the re sults reported above is shown in in this report, we demonstrate our work for the chinesejapanese and japanesechinese open do main translation task. the system we submitted is a neural mt model based on transformer ar chitecture. during the experiments, many tech niques, such as backtranslation, ensemble, rerank ing are applied and are proved to be effective for the mt system. parallel data extraction, noisy data filtering methods and characterbased models are also experienced and discussed, although currently they are not integrated into our systems, there will be still a lot work on them to find out proper ways to optimize the procedure and models, or to prove their limitations.