POSTECH - ETRI ’s Submission to the WMT 2020 APE Shared Task: Automatic Post-Editing with Cross-lingual Language Model (2020)

Abstract: This paper describes POSTECH-ETRI’s submission to WMT2020 for the shared task on automatic post-editing (APE) for 2 language pairs: English-German (En-De) and English-Chinese (En-Zh). We propose APE systems based on a cross-lingual language model, which jointly adopts translation language modeling (TLM) and masked language modeling (MLM) training objectives in the pre-training stage; the APE models then utilize jointly learned language representations between the source language and the target language. In addition, we created 19 million new sythetic triplets as additional training data for our final ensemble model. According to experimental results on the WMT2020 APE development data set, our models showed an improvement over the baseline by TER of -3.58 and a BLEU score of +5.3 for the En-De subtask; and TER of -5.29 and a BLEU score of +7.32 for the En-Zh subtask.

## introduction automatic post-editing (ape) is a subtask of mt, which aims to improve mt outputs by directly modifying machine-translated sentences @xcite . using ape systems to correct such errors that are automatically detectable can greatly reduce human effort compared to correcting machine-translated sentences manually from scratch @xcite . given that neural-network systems require a large quantity of training data, creating ape triplets, which each consist of a source sentence (src), a machine-translated sentence (mt), and a manually post-edited sentence (pe), requires a lot of human labor. furthermore, because neural ape is a recently minted field of study, only a few smallsized training data sets are available at present. to mitigate such data shortage, several methods are proposed such that 1) create artificial ape triplets @xcite ; and 2) apply 'transfer learning' @xcite . we believe that pre-trained models such as elmo @xcite , openai gpt @xcite , and bert @xcite helped ape models learn rich language representations that compensated for the performance loss caused by using an insufficient quantity of training data. ape is a task that handles both src and mt simultaneously, and learning a joint representation of these two inputs requires an understanding of both languages. although previous works that used bert have shown that transfer learning is effective in ape @xcite , adopting bert as a pre-trained language model may restrict to properly model the relation between two different languages because bert is trained only on monolingual data sets. therefore, following the recent trend of adopting transfer learning to various nlp tasks, we propose a new method that adopts a cross-lingual language model as a pre-trained langauge model for ape. ## model description our ape system is built on top of transformer's encoder-decoder structure @xcite . in the following subsections, we describe the main features of the encoder and decoder, respectively. figure 2 illustrates the overall structure of our model. ## conclusion for the wmt2020 ape shared task, we propose ape systems that adopt cross-lingual pre-trained language models. to better apply transfer learning to the ape task, we trained tlm in addition to using the original mlm models and initialized the decoder's weights in the same way as the encoder. furthermore, we created new synthetic triplets to augment the training data and used the ensemble technique to build our final model. experimental results show that our proposed model achieved significant improvements on the wmt2020 development and test data sets in terms of ter and bleu scores for both en-de and en-zh language pairs.