CASIA ’s System for IWSLT 2020 Open Domain Translation (2020)

Abstract: This paper describes the CASIA’s system for the IWSLT 2020 open domain translation task. This year we participate in both Chinese→Japanese and Japanese→Chinese translation tasks. Our system is neural machine translation system based on Transformer model. We augment the training data with knowledge distillation and back translation to improve the translation performance. Domain data classification and weighted domain model ensemble are introduced to generate the final translation result. We compare and analyze the performance on development data with different model settings and different data processing techniques.

## introduction neural machine translation(nmt) has been introduced and made great success during the past few years @xcite @xcite @xcite @xcite . among those different neural network architectures, the transformer, which is based on self-attention mechanism, has further improved the translation quality due to the ability of feature extraction and word sense disambiguation @xcite . in this paper, we describe our transformer based neural machine translation system submitted to the iwslt 2020 @xmath0. our system is built upon transformer neural machine translation architecture. we also adopt relative position @xcite and dynamic convolutions @xcite to investigate the performances of advanced model variations. for the implementation, we extend the latest release of fairseq 1 @xcite . to further improve the translation quality, we utilize data augmentation techniques of backtranslation with a sub-selected monolingual corpus to build additional pseudo parallel training data. sentence level knowledge distillation is used to strengthen the performance of student model from multi-policy teacher models including @xmath1, @xmath2, @xmath3. we also investigate the domain information of the large training data by using a bert based domain classifier, which is a masked language model and has been shown effective in large scale text classification tasks @xcite . with the in-domain data, we transfer the model of general domain to each specific domain, and use weighted domain model ensemble as decoding strategy. ## system description figure 1 depicts the whole process of our submission system, in which we pre-process the provided data and train our advanced transformer models on the bilingual data together with synthetic corpora from back-translation and knowledge distillation. with domain classification and fine tuning techniques, we obtain multiple models for ensemble strategy and post-processing. in this section, we will introduce each process step in detail. ## data preparation this section introduces the methods we employ to prepare the provided parallel data (18.9m web crawled corpus and 1.9m existing parallel sources) and monolingual sentences (unaligned web crawled data). we also describe how to prepare domain specific data to facilitate translation. the provided parallel corpus existing parallel for the two translation directions consists of around 1.9m sentence pairs with around 33.5m characters (chinese side) in total. furthermore, a large, noisy set of japanese-chinese segment pairs built from web data web crawled is also provided, which con-sists of around 18.9m sentence pairs with around 493.9m characters (chinese side) in total. we use the provided development dataset as the validation set during training, which consists of 5,304 sentence pairs. the average length and length ratio of the provided parallel corpus and development dataset is shown as in ## conclusion we present the casia's neural machine translation system submitted to iwslt 2020 @xmath4. our system is built with transformer architecture and incorporating the following techniques: