Incorporating Terminology Constraints in Automatic Post-Editing (2020)

Abstract: Users of machine translation (MT) may want to ensure the use of specific lexical terminologies. While there exist techniques for incorporating terminology constraints during inference for MT, current APE approaches cannot ensure that they will appear in the final translation. In this paper, we present both autoregressive and non-autoregressive models for lexically constrained APE, demonstrating that our approach enables preservation of 95% of the terminologies and also improves translation quality on English-German benchmarks. Even when applied to lexically constrained MT output, our approach is able to improve preservation of the terminologies. However, we show that our models do not learn to copy constraints systematically and suggest a simple data augmentation technique that leads to improved performance and robustness.

## introduction automatic post-editing (ape) aims to improve the quality of the output of an arbitrary machine translation (mt) system by pruning systematic errors and adapting to a domain-specific style and vocabulary @xcite . although previous work has shown the usefulness of ape to prune errors by focusing on improving the translation error rate (ter), few have studied the effect of incorporating lexical constraints. there are several use cases where such a system would be beneficial. for example, content providers meticulously curate lists of terminologies for their domains that indicate preferred translations for technical terms. lexically constrained ape would also be useful for cross-lingual information retrieval. when displaying snippets from retrieved documents, the query term should appear in the translation output (if it does in the source) as it can make relevance clear to the end user. here, the query serves as the term. while recent approaches allow inference time adaptation of nmt systems using these terminologies @xcite , postediting translations with a generic ape system may lead to dropped terms. a constraint-aware ape system would allow to fix systematic translation errors, while keeping the terminologies intact. inspired by @xcite , we consider a range of representations which augment input sequences with constraint tokens and factors for use in an autoregressive transformer (at) ape model. using this approach, the constraints are explicitly represented in the encoder input sequence, and the model learns to prefer translations that contain the supplied terminologies during decoding. we also explore the use of the levenshtein transformer (levt) @xcite , a non-autoregressive transformer (nat) model. the levt model applies neatly to the ape task since the decoder can be initialized with an incomplete sequence to be refined. additionally, multiple corrections can be made simultaneously, yielding a decoding speedup over autoregressive models. we then show that constrained ape improves translation quality and terminology preservation on top of both unconstrained and constrained mt. while both constrained and unconstrained ape models perform similarly on reducing systemic errors in the mt output, they differ in their ability to preserve terminology constraints. when applying unconstrained ape on top of constrained mt, we find a 12.6% relative drop of supplied terminology constraints as compared to a fully constrained mt to ape pipeline. we experiment extensively with variations of both at and levt models, testing on both pbmt and nmt english to german wmt ape tasks @xcite . under all scenarios, the model performs post-editing while satisfying terminology constraints when supplied. our evaluation of both constrained at and nat models on pbmt and nmt ape benchmarks shows that both models correctly translate more than 95% of terminology constraints, with the nat model achieving the highest coverage of terminologies at the expense of post-editing quality. finally, using constraints constructed with synonyms and antonyms, we show that our models do not learn to copy constraints systematically, and introduce a simple data augmentation strategy to improve the preservation of unusual constraints. to summarize, our contributions are as follows: ## constrained ape the task of ape is to correct systematic errors in an mt system output. an ape model takes as input two sequences: the source language sentence to be translated and the translation of this sentence into the target language by an mt system. the intended output is a corrected version of the mt system's initial translation @xcite . constrained ape allows for the specification of terminology constraints: a translation for one or more phrases in the source language input may be pre-specified as additional input. the constrained ape model must use the supplied terminology constraints when performing the ape task. formally, let @xmath0, . . . , x m ] and γ = [γ 1 , . . . , γ n ] be the source language sentence and the initial mt translation respectively. the tokens x i and γ i are drawn from the source and target language vocabularies s and t respectively. the target post-edited sentence is a sequence @xmath1, . . . , y o ], with tokens y i also drawn from t . we are also given a series of t translation constraints, @xmath2, y(1) , , . . . , x(t) , y(t) , , where each constraint x(i) , y(i) ∈ s * ×t * is a tuple of source language phrase, x(i) = [x 1 , . . . , xj ] , and its desired translation, y(i) = [y 1 , . . . , yk ], into the target language. the goal of the constrained ape task is to learn a mapping of x, γ, and c to the target post-edited translation y. crucially, when a source side constraint x(i) matches a sub-sequence in x, it is required that the sub-sequence be translated as y(i) . see figure 1 for an example. ## models while there are existing models to address the ape task, and the lexical constrained mt task, it is not clear how to represent lexical constraints for ape models which, unlike mt models, take two sequences as input. we propose several techniques to incorporate constraints as additional inputs to the ape encoder by combining the input sequence modification used in constrained mt @xcite with the mst method of @xcite . for decoding, we experiment with both the at and the levt decoders. the levt decoder can additionally take advantage of different decoder initialization strategies for constrained decoding. we first briefly show how we encode terminology constraints in the input sequence, before de- figure 2: setup for the models by the input and initialization at inference. scribing how they are incorporated into the mst and levt ape models specifically. ## experiments we use the fairseq toolkit @xcite for implementing the mst and extending the levt. 4 we evaluate the models on translation error rate (ter) @xcite and bleu @xcite using the official evaluation script foot_4 for analyzing the post-editing performance. we also compute the percentage of target language term constraints present in the output (term %) to measure the performance of the constrained models. ## analyzing constraint translation behavior terminology constrained ape aims to add some degree of user control over the ape process without destabilizing the general post-editing behavior of the decoder. however, the imposition of rare or unusual terminology constraints will necessar-ily be in conflict with the decoder language model, which will give higher probabilities to terminology translations found frequently in the training data. in practice, a user may specify a terminology constraint that is not well represented in the training distribution. for example, a user may want a product description translated using location specific brand names or marketing copy. ideally, a terminology constrained model would reliably produce these terms and use them appropriately even if they do not rank highly by the decoder. additionally, it is desirable that the addition of terminology constraints does not lead to large changes in the model's output. since terminology constraints are only bound to a word or phrase, the model should only need to make minimal changes between the unconstrained and constrained output. large changes in output may make it harder for a user to anticipate the effects of a constraint which may make constrained ape less useful in practice. we refer to this behavior as systematic copying, i.e. the model should behave in a transparent and stable way, enforcing terminology constraints even when they strongly disagree with the decoder language model, while only making minimally necessary changes in the output to do so. by harvesting terminology constraints from the training data, we run the risk that the model simply learns to draw some translation hints from the supplied terminology constraints, but does not actually learn this systematic copying behavior. that is, it never truly sees an out-of-sample constraint that is extremely unlikely from the perspective of the decoder language model. to test whether our proposed models indeed learn this systematic copying behavior we perform a qualitative experiment, comparing model outputs when supplying different constraints for a source word, by varying whether the target language constraint was (a) the original target language constraint specified in the test set, (b) a target language synonym of the original constraint term, (c) a target language antonym of the original constraint term, or (d) a totally random term in the target language. while antonym and random term constraints might not seem to correspond to realistic use cases, they let us to examine the effects of specifying a constraint where the source and target language terms are extremely semantically divergent. additionally, they simulate scenarios where translations for names vary dramatically by region. for example, the cleaning product called "mr. clean" in the u.s. is called "meister proper" in germany. as can be seen in figure 3 , our qualitative exploration reveals that synonym, antonym, and random terminology constraints are frequently not included in the output. for example the mst model fails to generate the antonym verkleinerung. this suggests that target side constraints that are unseen during training may be ignored by the model, and that the models are not learning to systematically copy arbitrary constraints. ## conclusion and future work this work introduces the terminology constrained ape task and several mst and levt model variants for incorporating lexical constraints during post-editing. furthermore, we show that constrained ape is necessary for preserving lexical constraints in a mt to ape pipeline. evaluations on standard ape benchmarks show that terminology constraints are satisfied while improving the original mt quality. finally, we show that the constrained ape models do not learn a robust systematic copying behavior, and propose a data augmentation method to help mitigate this issue. in future work, we hope to explore ways of modifying model architecture or training algorithms to further improve the systematic copying behavior.