An Iterative Knowledge Transfer NMT System for WMT 20 News Translation Task (2020)

Abstract: This paper describes our submission to the WMT20 news translation shared task in English to Japanese direction. Our main approach is based on transferring knowledge of domain and linguistic characteristics by pre-training the encoder-decoder model with large amount of in-domain monolingual data through unsupervised and supervised prediction task. We then fine-tune the model with parallel data and in-domain synthetic data, generated with iterative back-translation. For additional gain, we generate final results with an ensemble model and re-rank them with averaged models and language models. Through these methods, we achieve +5.42 BLEU score compare to the baseline model.

## introduction this paper describes our submission to the wmt20 news translation task in english to japanese direction. in this year, english-japanese directions have newly established in news translation shared task. the english-japanese translation is not easy to deal with because of the difference in word order and the rich morphological characteristics of japanese. nevertheless, recent architectures for neural machine translation (nmt), such as transformer @xcite , show reasonable results when we have enough parallel data. unfortunately, however, there is not much in-domain parallel data provided for english-japanese task. to solve this issue, in this paper, we suggest the iterative knowledge transfer system which pre-trains the model with in-domain monolingual data. our system is based on transformer architecture. we pre-train the model to transfer linguistic characteristics and domain knowledge of monolingual data. although there are various pre-training methods for nmt, mass @xcite ) is adopted in our system since mass pre-trains the encoder and the decoder jointly and uses both labeled data and unlabeled data as the training data. to supplement insufficient in-domain parallel data, we generate synthetic data by back-translation from in-domain monolingual data. we also add some noise to the synthetic data. we then pre-train the model with the synthetic parallel data for supervised method and the monolingual data for unsupervised way. in fine-tuning step, we train the model with parallel corpus and perform the backtranslation with in-domain data for iterative finetuning. in addition, we adopt an ensemble and averaging methods which are simple but very effective to improve performance in deep learning. with ensemble and average models, we apply noisy channel re-ranking which shows higher performance compared to r2l re-ranking @xcite . through these methods, we achieve +5.42 bleu score @xcite compared to the baseline model. ## approach our system aims to encourage knowledge extraction of domain knowledge and linguistic characteristics by iteratively performing pre-training and fine-tuning. in this section, we explain techniques we use in each step. ## conclusions in this paper, we describe our submission to the wmt20 news translation task in english to japanese direction. our main approach is based on transferring knowledge from large amount of monolingual data by pre-training the model itera-