Cross-Lingual Transformers for Neural Automatic Post-Editing (2020)

Abstract: In this paper, we describe the Bering Lab’s submission to the WMT 2020 Shared Task on Automatic Post-Editing (APE). First, we propose a cross-lingual Transformer architecture that takes a concatenation of a source sentence and a machine-translated (MT) sentence as an input to generate the post-edited (PE) output. For further improvement, we mask incorrect or missing words in the PE output based on word-level quality estimation and then predict the actual word for each mask based on the fine-tuned cross-lingual language model (XLM-RoBERTa). Finally, to address the over-correction problem, we select the final output among the PE outputs and the original MT sentence based on a sentence-level quality estimation. When evaluated on the WMT 2020 English-German APE test dataset, our system improves the NMT output by -3.95 and +4.50 in terms of TER and BLEU, respectively.

## introduction automatic post-editing (ape) is the task of automatically correcting errors in the output of a machine translation (mt) system by learning from human corrections @xcite . ape can be viewed as a cross-lingual sequence-to-sequence task, which takes a source sentence and the corresponding mt output as inputs and generates the post-edited (pe) output. our work is inspired by xlm-roberta (xlm-r) @xcite , a cross-lingual language model, which shows the state-of-the-art performance for a wide range of cross-lingual tasks. xlm-r takes a concatenation of two sentences in different languages as an input to generate crosslingual representations. similarly, we propose a transformer @xcite architecture for ape in which the encoder uses the same architecture as xlm-r. in addition, we use xlm-r-based translation quality estimation (qe) @xcite to further improve the pe output of the transformer. qe is the task of estimating the quality of the mt output when only the source text is provided @xcite . we use two granularity levels of qe: word-level and sentence-level. based on the wordlevel qe, we try to correct the wrong words or insert the missing words in the pe output. through the sentence-level qe, we select the best translation among pe outputs and the original mt sentence to prevent over-correction (i.e., one of the ape models rephrases an already correct mt output). our contributions are summarized as follows: in the experiment using the wmt 2020 english-german ape test set, our system achieves -3.95 ter and +4.50 bleu improvement over the baseline (nmt output). ## methodology our approach for ape comprises three components: 1) a cross-lingual transformer, 2) word masking based on word-level quality estimation (qe) and xlm-r-based mask prediction, and 3) output selection based on sentence-level qe. ## conclusion in this paper, the bering lab's submission to the wmt 2020 english-german ape shared task is described. a cross-lingual transformer architecture is proposed for ape in which a single encoder takes the concatenation of a source and a mt sentences as an input to generate intermediary cross-lingual representations, and then a decoder outputs postedited results. in addition, methods to improve the ape performance through translation qe are proposed. first, the incorrect or missing words in the post-edited output are masked based on a word-level qe. then, the actual word for each mask is predicted based on the fine-tuned xlm-r using the translation language modeling (tlm) objective. finally, a sentence-level qe-based output selection method is proposed to prevent over-correction. the experimental results show that our ape system significantly improves the nmt output in terms of both ter and bleu.