Adaptation of Back-translation to Automatic Post-Editing for Synthetic Data Generation (2021)

Abstract: Automatic Post-Editing (APE) aims to correct errors in the output of a given machine translation (MT) system. Although data-driven approaches have become prevalent also in the APE task as in many other NLP tasks, there has been a lack of qualified training data due to the high cost of manual construction. eSCAPE, a synthetic APE corpus, has been widely used to alleviate the data scarcity, but it might not address genuine APE corpora’s characteristic that the post-edited sentence should be a minimally edited revision of the given MT output. Therefore, we propose two new methods of synthesizing additional MT outputs by adapting back-translation to the APE task, obtaining robust enlargements of the existing synthetic APE training dataset. Experimental results on the WMT English-German APE benchmarks demonstrate that our enlarged datasets are effective in improving APE performance.

## introduction automatic post-editing (ape) seeks to automatically correct errors included in the output of a blackbox machine translation (mt) system to improve the final translation quality, thereby reducing the effort required for manual post-editing @xcite @xcite . in general, ape can be considered as a task of sequence-to-sequence supervised learning, which requires a considerable amount of human-annotated data. however, constructing an ape corpus-a set of triplets ( to overcome the lack of genuine ape corpora, several previous studies have proposed methods to construct synthetic training datasets @xcite @xcite , and they appear to be partially helpful in mitigating the data scarcity problem. one such study is escape @xcite , which has been shown to be effective in training deep models and adopted in a number of ape works @xcite . utilizing parallel corpora, which comprise pairs of a source sentence (src) and a reference sentence (ref ), es-cape was constructed as a set of synthetic ape triplets in the form of (src, mt, ref ) where mt is a machine translation of src, and ref serves as an alternative to pe of a genuine ape triplet. despite the effectiveness of escape, we argue that it may have two major drawbacks: (1) es-cape's method relies heavily on parallel resources, so its scalability is restricted to the quantity of available parallel resources and can be even more limited in low-resource scenarios; (2) the relation between mt and ref may not thoroughly reflect the actual relation between mt and pe because ref is not guaranteed to be a minimally edited revision of mt, potentially leading to the discrepancy between the distribution of translation errors in genuine data and that of translation errors in the synthetic data (figure 1 ). in this paper, we propose two automatic methods that are inspired by back-translation from the mt task @xcite , employing the ape process in the forward direction and the backward direction with applying the escape resource to them to create additional synthetic mt. our approach not only extends the existing resource, but also aims to better simulate the characteristics of real ape data by making our synthetic mt better approximate the error distribution of the wmt ape benchmark dataset. ## method beyond the escape corpus, to yield a more convincing error distribution as well as to supply ape models with more ape resources made out of limited parallel resources, we propose synthetic-data generation methods that can be seen as adaptations of back-translation to the ape task in terms of creating synthetic mt, which is one of the two sources of ape. we produce new synthetic mt so that ref can better act as its minimally post-edited text, whereas this ref may not do so for the original mt. specifically, we suggest two strategies, both of which apply the ape process: 'forward generation' and 'backward generation'; each one of them performs ape in the forward direction and the backward direction, respectively. as described in figure 2 , the former partially corrects mt to reduce the distance between mt and ref, while the latter injects the right quantity of translation errors into ref. ## experiments metric. following the evaluation setting used in the wmt ape shared task, we adopt ter @xcite as the primary metric to measure the distance between the model's prediction and the reference text; and bleu @xcite as the secondary metric to measure the degree of ngram match. in addition, all evaluations in our experiments are case-sensitive. dataset. we use two kinds of ape datasets: human-made ape datasets, which are provided by wmt, and escape. both are english-german (en-de) ape corpora; they are further categorized according to their subtask depending on whether the target mt system is a phrase-based statistical mt (pbsmt) system or a neural mt (nmt) system. the wmt datasets are in the it domain, whereas escape was made out of domaingeneral parallel corpora. detailed data statistics are presented in ## results and discussion nevertheless, we found that when we augment escape with escape fg and/or escape bg , the trained ape model shows consistent improvements in its ape performance and most of the improvements upon the escape baseline are statistically significant. moreover, the results also surpass current state-of-the-art (except the ensemble models) ape models @xcite , which are built on top of bert @xcite , thus contain more model parameters, and exploit a huge amount of monolingual data. we expect that these results are because, in addition to an increase in the total quantity of training samples, the integration of multiple synthetic datasets, each of which focuses on different aspects of ape from the other-escape contains actual mt outputs; on the other hand, synthetic triplets better satisfy the minimal-edit criterion-appears to have an effect on the models' ape performance. we found that our proposed methods derive a large number of new mt ( ## conclusion and future work in this paper, we tried to alleviate the drawbacks of escape by suggesting two new methods that adapt back-translation to the ape task, consequently increasing the data quantity and address the minimum editing characteristic. according to our experimental results, although ape models trained on each one of our two synthetic datasets show just comparable performances to the escape baseline, those trained on integrations of multiple synthetic datasets show consistent improvements over the baseline, implying that our new synthetic datasets are beneficial enlargements of escape. however, we manually selected the optimal stop points for both of our proposed generation schemes, so we will automate these selection processes in our future work.