Exploring the Importance of Source Text in Automatic Post-Editing for Context-Aware Machine Translation (2021)

Abstract: Accurate translation requires document-level information, which is ignored by sentence-level machine translation. Recent work has demonstrated that document-level consistency can be improved with automatic post-editing (APE) using only target-language (TL) information. We study an extended APE model that additionally integrates source context. A human evaluation of fluency and adequacy in English–Russian translation reveals that the model with access to source context significantly outperforms monolingual APE in terms of adequacy, an effect largely ignored by automatic evaluation metrics. Our results show that TL-only modelling increases fluency without improving adequacy, demonstrating the need for conditioning on source text for automatic post-editing. They also highlight blind spots in automatic methods for targeted evaluation and demonstrate the need for human assessment to evaluate document-level translation quality reliably.

## introduction neural machine translation (nmt) has significantly improved the state of the art in mt @xcite @xcite on the sentence level. however, accurate translation requires looking at larger units than individual sentences @xcite , and context-aware nmt has recently become a popular research direction @xcite @xcite . one approach to discourse-level processing in nmt is automatic post-editing of the output of a sentence-level system. docrepair @xcite is a monolingual sequence-to-sequence model to correct inconsistencies in groups of adja-cent sentence-level translations, showing improvements for specific discourse-level phenomena such as the generation of inflections in elliptic sentences. the hypotheses explored in this work are (1) that the coherence of the translation can be further improved by exploiting context in the source language, and (2) that the omission of source context disproportionately affects adequacy in a way that is not measured adequately by the existing automatic evaluation procedures. our post-editing model is a document-level adaptation of transference @xcite , a successful three-way transformer architecture from the wmt 2019 automatic post-editing (ape) task @xcite . to keep the model from over-correcting the hypothesis, we use data weighting @xcite and a conservativeness penalty @xcite . we evaluate on the same training and evaluation sets as @xcite , including a general test set validated by bleu score and contrastive sets for several discourse phenomena. our experimental results confirm both hypotheses. despite similar bleu, human evaluation demonstrates that our transference model significantly outperforms docrepair in terms of adequacy, whilst both models show a comparable improvement in fluency over a baseline without ape. the automatic evaluation on discourse-specific test sets suggests that source-side information is particularly useful for predicting omitted verb phrases; however, even the targeted discourse-specific evaluation does not reflect the adequacy gain found by human evaluators. this is especially true since some of the discourse-specific test sets of @xcite have a very narrow focus on problems for which source context is unlikely to help. ## transference transference @xcite (figure 1 ) is a multisource transformer @xcite architec-ture which exploits both source src and the mt output mt to predict the reference ref. it is composed of (1) a source encoder (enc src ) to generate the src representation, (2) a second encoder (enc @xmath0, and (3) a decoder (dec ref ) which captures the final representation from enc @xmath1. if document-level ape is trained on a small subset of the parallel data, or only synthetic data, and therefore presumably weaker as a general model of translation than the sentence-level main model, we need to control how aggressively ape can modify mt to prevent over-correction. we adopt two strategies from the ape literature to achieve this. a conservativeness penalty @xcite , denoted c, penalises the score of each prediction that is not in src or mt. formally, let v @xmath2. given a |v |-sized vector of candidates h t at time step t, the score of each candidate v is defined as: second, similar to lopes et al. (2019), we apply a data weighting strategy during training. we assign each training sample a weight that is defined as bleu smooth (mt, ref ) @xcite to upweight samples that require little post-editing. ## data and preprocessing we use all of the english-to-russian data released by @xcite al. (2019a) the four contrastive test sets have a narrow focus on specific discourse-level phenomena. the "deixis" set targets consistent use of formal and informal second-person pronouns (t-v distinction) in russian (however without regard to the social acceptability of the selected form). "lexical cohesion" targets the consistent transliteration of proper names into cyrillic script. these two sets are independent of source context by design, as the model is only evaluated on the generation of consistent repetitions of a form it has committed to, regardless of its adequacy in the context. the "ellipsis vp" set targets elliptic verb phrases, where russian requires the production of a lexical verb form not found in english. the "ellipsis inflection" set tests the generation of noun inflections in sentences where the governing verb has been elided. the training data is tokenised and truecased with moses @xcite , and encoded using byte-pair encoding @xcite with source and target vocabularies of 32000 tokens. like @xcite , we report lowercased, tokenised bleu @xcite with multibleu.perl from the moses toolkit. the sentence-level baselines (@xmath3. for document-level ape, docrepair is a transformer base model that operates on groups of adjacent sentences, mapping from mt to ref. we use the nematus toolkit @xcite for docrepair and our implementation of the transference architecture, using the same configuration as @xcite . 2 detailed hyperparameters are listed in appendix a. we train our document-level models on the 30m pairs of synthetic data. for some models, we also include the subset of the parallel data (1.5m pairs) for which context sentences are available, referred to as pardata. the mt part of pardata is generated by randomly sampling 20 translations with our @xmath4. in preliminary experiments, adding noise to the training data improved model generalisation. we generated noise with two strategies. following @xcite , mt in both synthetic data and pardata is randomly selected from 20 translations, and noise is added by making random token substitutions with probability of 10%. following @xcite , noise is added to the src in synthetic data by three operations: (1) replacing a token; (2) deleting a token; (3) swapping adjacent token pairs, with a probability of 10%. ## automatic evaluation on the general test set, trained on only synthetic training data, transference achieves about 2 bleu points less than docrepair. we suspect that this derives from the mismatch of the training and test data for transference. specifically, during training, the "source" seen by transference is the result of noisy back-translation from russian, whereas at test time, the source is an original english sentence. when pardata is included, transference and docrepair achieve comparable bleu. in accuracy on the test sets for t/v pronouns ("deixis") and transliteration consistency ("lexical cohesion"), transference does not improve over docrepair, which is unsurprising considering how those test sets are constructed. however, adding source knowledge does improve results on both ellipsis test sets, for vp ellipsis even without adding the pardata data. the improvement is generally greater for vp ellipsis than for noun inflection. ## human evaluation to gain a better picture of the merits of the different systems, we conducted a manual evaluation. we randomly selected 720 sentences from the general test set and 100 sentences from the discourse test set and had them evaluated separately for adequacy and fluency by two native speakers of russian. to avoid priming between the fluency and adequacy conditions, the test set was split between the annotators, and no sentence was annotated for adequacy and fluency by the same annotator. to determine the inter-annotator agreement, there are 100 overlapping sentences for two annotators. the sentences were presented to the annotators in random order along with 3 sentences of preceding context. the sentence to be evaluated was highlighted, and the russian translations of the three systems (baseline, docrepair (+pardata) and transference (+pardata)) were displayed next to each other, ordered randomly. in the adequacy condition only, the english source text was also shown. the annotators received instructions according to ## related work our work draws on two strands of research: automatic post-editing and context-aware mt. automatic post-editing has a long history in mt @xcite , with regular shared tasks @xcite @xcite . neural multi-source ape systems as first proposed by @xcite and @xcite , some of them including source language information @xcite @xcite , have come to dominate ape. we take inspiration from the top-performing systems at the wmt19 shared task for architectures and training/decoding tricks @xcite , and make heavy use of synthetic training data @xcite @xcite . neural context-aware mt can be achieved by integrating context into the main translation model @xcite @xcite . two-stage models with a sentence-level first pass and document-level second pass have been explored for scenarios with asymmetric training data. @xcite introduces a two-pass model where, unlike in ape, the second-pass is tightly integrated with the first-pass model, reusing its hidden representations. apart from @xcite , the model closest to ours is by @xcite , who explored document-level ape, but only manually evaluated its efficacy as part of a large model ensemble. ## conclusion our human evaluation shows that monolingual ape oriented towards consistency beyond the sentence level improves fluency, but not adequacy, while multi-source ape with source context improves both adequacy and fluency. this shortcoming of monolingual ape in terms of adequacy was not easily visible with a consistency-focused automatic evaluation, highlighting the need for human evaluation to avoid such blind spots and reinforcing earlier findings about the inadequacy of automatic evaluation methods for discourse-level mt @xcite . clearly, a two-stage process with sentence-level translation and multi-sentence ape is a viable approach in asymmetric data settings with little document-level parallel data. however, we still required some actual document-level parallel data, and were unable to match the success of monolingual repair when using only synthetic data. exploring the data requirements of document-level ape, and devising ways to reduce them, are worth further study.