A libaba’s Submission for the WMT 2020 APE Shared Task: Improving Automatic Post-Editing with Pre-trained Conditional Cross-Lingual BERT (2020)

Abstract: The goal of Automatic Post-Editing (APE) is basically to examine the automatic methods for correcting translation errors generated by an unknown machine translation (MT) system. This paper describes Alibaba’s submissions to the WMT 2020 APE Shared Task for the English-German language pair. We design a two-stage training pipeline. First, a BERT-like cross-lingual language model is pre-trained by randomly masking target sentences alone. Then, an additional neural decoder on the top of the pre-trained model is jointly fine-tuned for the APE task. We also apply an imitation learning strategy to augment a reasonable amount of pseudo APE training data, potentially preventing the model to overfit on the limited real training data and boosting the performance on held-out data. To verify our proposed model and data augmentation, we examine our approach with the well-known benchmarking English-German dataset from the WMT 2017 APE task. The experiment results demonstrate that our system significantly outperforms all other baselines and achieves the state-of-the-art performance. The final results on the WMT 2020 test dataset show that our submission can achieve +5.56 BLEU and -4.57 TER with respect to the official MT baseline.

## introduction and related work even machines can approach and achieve parity with human translations @xcite empowered by a sequence-to-sequence fashion @xcite , postediting is still an important and necessary step in the translation process, especially in scenarios where extremely high-quality translation results are essentially required such as business legal documents, technical product guides, medicine instructions and so on. it is the process whereby humans * * indicates equal contribution. however, post-editing, while improving, that can match human understanding of meaning, nuance, tone, humor-the list goes on, it's often worth paying extra more. the time spent on translation mistake corrections by humans remains substantial to the extent @xcite so that it even occasionally offsets the efficiency gained from the neural machine translation (nmt) systems. in this paper, we explore automatic post-editing (ape) in a deep learning framework where a two-stage training pipeline is engaged. the goal of ape task is to examine automatic methods of correcting translation mistakes produced by a black-box machine translation engine to improve the mt results. human efforts are correspondingly reduced in the later editing process @xcite if our ape system can approach human translations as much as possible. traditionally, ape is a supervised learning task, requiring sufficient training data in the triplet of source (src), machine translation (mt) and post editing (pe) that are usually expensively available. due to the limited number of such ape data released officially in this year's ape tasks and the specific domain, wikipedia, which is quite different from the previous years'(it domain), we adopt an imitation learning to mine wmt corpora, escape @xcite , opus wikipedia corpus @xcite and our own english-german corpus to augment ape training data. however, pseudo data strategy is far from enough to train the state-of-the-art ape system. inspired by the masked language model objective in the encoder bert @xcite , we introduce our bertlike cross-lingual training objective to the encoderdecoder framework by adapting the decoder to become a memory encoder @xcite , allowing us to pre-train the target language model similar to bert but conditioned on the source language text. knowledge learned from the pre-training can be extensively transferred to many secondstep downstream tasks, including but not limited to translation quality estimation, parallel corpus filtering and of course, automatic post-editing. the overall framework of our ape model is the same with the generative automatic post-editing model's structure in @xcite . similar training mechanism is applied in the winner system of wmt 2019 ape shared task @xcite , that wisely takes full advantage of the pre-trained multilingual bert (mbert) @xcite and achieves top performances.they concatenate the source and machine translation sentences to feed into the encoder mbert and then fine tune the encoder and a transformer decoder where the context attention block is initialized by the self-attention weights of mbert as well. we examine our approach on the public english-german dataset from wmt 2017 ape shared task. our system outperforms the top ranked methods in both bleu @xcite and ter @xcite metrics. for this year's wmt ape task, we finally submitted two ensemble english-german ape models according to different model selection methods and accomplish +5.56 increase in bleu and -4.57 decline in ter on 2020 test set. ## methodology in this section, we will introduce our ape model in terms of general structure and some computation details together with our data augmentation strategy. ## experiment we conduct our experiments on two different datasets: first, to make a fair comparison with other top-ranked systems on wmt ape tasks in recent years, we perform a single model evaluation on the wmt 2017 english-german ape shared task without any other pseudo data except the artificial dataset @xcite and escape @xcite . considering the application domain for this year's task changes from it to wikipedia and the size of the official in-domain training set is quite small (only 7000 samples), we generate about 20m in-domain pseudo data for our model training as follows: pre-processing. in all of our experiments, we apply truecasers trained independently for english and german seperately @xcite and process our data into subword units @xcite with a 32k shared vocabulary. triplets with more than 70 subword units in any one of the srcs, mts or pes are removed. evaluation metrics. we mainly evaluate our systems with the metrics, translation edit rate (ter) @xcite and bilingual evaluation understudy (bleu) @xcite , since they are standard and widely employed in evaluation of the wmt ape tasks. ## conclusion this paper describes our automatic post-editing system for the wmt 2020 english-german ape shared task. we introduce a cross-lingual bertlike conditional model with an innovative memory encoder which can capture the deep semantic information of machine translations conditional on the source sentences. in addition, efforts on data augmentation strategies, corpus filtering and imitation learning, are able to overcome the scarcity of real ape data and further improve the model performance together with the ensemble strategy. our single ape model outperforms all winner systems of recent years' wmt ape shared tasks on the wmt 2017 english-german test set and achieves impressive performances on the wmt 2020 english-german ape test set.