Noising Scheme for Data Augmentation in Automatic Post-Editing (2020)

Abstract: This paper describes POSTECH’s submission to WMT20 for the shared task on Automatic Post-Editing (APE). Our focus is on increasing the quantity of available APE data to overcome the shortage of human-crafted training data. In our experiment, we implemented a noising module that simulates four types of post-editing errors, and we introduced this module into a Transformer-based multi-source APE model. Our noising module implants errors into texts on the target side of parallel corpora during the training phase to make synthetic MT outputs, increasing the entire number of training samples. We also generated additional training data using the parallel corpora and NMT model that were released for the Quality Estimation task, and we used these data to train our APE model. Experimental results on the WMT20 English-German APE data set show improvements over the baseline in terms of both the TER and BLEU scores: our primary submission achieved an improvement of -3.15 TER and +4.01 BLEU, and our contrastive submission achieved an improvement of -3.34 TER and +4.30 BLEU.

## introduction there has been a surge of interest in developing automatic post-editing (ape) models, which is capable of automatically correcting errors produced by a machine-translation (mt) system, and thus is an attractive way to improve the quality of the mt output. currently, sequence-to-sequence modeling has become a dominant approach to constructing ape models @xcite , which requires a large quantity of training samples. however, ape data 1 -comprising triplets of three texts: source (src), a machine-translation (mt) of src, and a human-crafted post-edited sentence (pe) of mt -is too small and costly to acquire. consequently, the lack of ape data becomes a great 1 http://www.statmt.org/wmt20/ape-task.html obstacle to a satisfactory performance of sequenceto-sequence models. to reduce such data scarcity, there have been several attempts at constructing synthetic ape data @xcite . most notably, @xcite proposed a simple but effective way to construct a large-scale synthetic ape data set escape (src, mt, ref ), of which src and ref is the source and target text of freely available parallel corpora, respectively, and mt is a translation of src produced by the mt system that had been trained on those parallel corpora. as escape has shown to be beneficial in training ape models @xcite , it has become feasible to train deep ape models and also what most recent works have been relying on so far. nevertheless, the availability of a limited quantity of parallel corpora may not only be insufficient, but also vary depending on the language pair, that is, while some language pairs have plenty of resources, some others have relatively few resources. we thus argue that further works to supply additional resources should be needed to mitigate the potential data scarcity. in this work, we introduce a noising scheme by which corrupted texts (ref noise ) are produced from ref of parallel corpora, resulting in additional ape triplets (src, ref noise , ref ), where src and ref is the source and target text of parallel corpora, respectively. during post-editing, certain editing operations including the word insertion, deletion, substitution, and shifting are applied to translated texts (noisy texts) for error correction. thus, we applied such operations to target texts (clean texts) of parallel corpora to inject errors in reverse. moreover, to simulate the quantity of errors that the target mt system produces, we refer to the distribution of "translation error rate" (ter) @xcite occurring in the actual ape data to determine the quantity of errors to be injected. while we trained our models using the noising module, we supplied to the models synthetic ape data in addition to the wmt'20 ape data set as training data. the synthetic data was produced by using the escape method, which uses parallel corpora and a trained nmt model. we observed that models with noising module improved up to about -0.7 ter and +1.45 bleu on the english-german (en-de) wmt'20 ape validation set compared to models without noising. finally, our primary and contrastive submission to the wmt'20 ape shared task respectively recorded 28.41 ter and 54.22 bleu, and 28.22 ter and 54.51 bleu on the blind en-de wmt'20 ape test set. ## related work noise injection to input sentences has become a popular method to let auto-encoders @xcite or pre-trained language models @xcite learn how to reconstruct the original input. because post-editing is a process of reconstructing corrupted translations, simulating corrupted mt outputs by injecting noise to the target sentence is a way to get synthetic ape training samples. in the ape task, @xcite employed a data noising technique that incorporates a noise vector generated from a gaussian or uniform distribution into the word embedding vector. however, their noising process has an effect on all tokens in a sequence, whereas only certain tokens in a given mt output are to be corrected in the ape process. ## conclusion we propose a noising scheme to supply ape models with synthetic ape triplets during the training phase. our noising scheme is designed based on the error types that are defined in the ape task, and the quantity of noise that are injected during the training phase are determined in consideration of the distribution of those error types in the official training data. according to the experimental results, applying our noising scheme to ape models showed an improvement of the post-editing quality in terms of both ter and the bleu scores, which indicates that our noising scheme was effective in training ape models although there may be differences between the synthesized errors and the actual mt errors. therefore, in future work, we will aim to reduce those gaps caused by our noising scheme.