Learning Non-Monotonic Automatic Post-Editing of Translations from Human Orderings (2020)

Abstract: Recent research in neural machine translation has explored flexible generation orders, as an alternative to left-to-right generation. However, training non-monotonic models brings a new complication: how to search for a good ordering when there is a combinatorial explosion of orderings arriving at the same final result? Also, how do these automatic orderings compare with the actual behaviour of human translators? Current models rely on manually built biases or are left to explore all possibilities on their own. In this paper, we analyze the orderings produced by human post-editors and use them to train an automatic post-editing system. We compare the resulting system with those trained with left-to-right and random post-editing orderings. We observe that humans tend to follow a nearly left-to-right order, but with interesting deviations, such as preferring to start by correcting punctuation or verbs.

## introduction neural sequence generation models have been widely adopted for tasks such as machine translation (mt) @xcite and automatic post-editing of translations @xcite @xcite . these models typically generate one word at a time, and rely on a factorizac 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. * work partly done during a research visit at new york university. while prior work has focused on learning generation orderings in an unsupervised manner, in this paper we ask the question of whether human generation orderings can be a useful source of supervision. one such possible source lies in the keystrokes of humans typing. it is known that edit operations performed by human translators are not arbitrary @xcite . but it is not known how the orderings preferred by humans look like, or how they compare to orders learned by models. to investigate this question, we propose a model that learns generation orderings in a supervised manner from human keystrokes. since a human is free to move back and forth arbitrarily while editing text, the chosen order of operations can be used as an additional learning signal. more specifically, we do this in the context of automatic postediting (ape) @xcite . ape consists in improving the output of a blackbox mt system by automatically fixing its mistakes. the act of post-editing text can be fully specified as a sequence of delete (del) and insert (ins) actions in given positions. furthermore, if we do not include redundant actions in a sequence, that sequence can be arbitrarily reordered while still producing the same output. for instance, in we compare a model trained with human orderings to others trained with left-to-right and random orderings. we show that the resulting nonmonotonic ape system learned from human orderings outperforms systems learned on random orderings and performs comparably or slightly better than a system learned with left-to-right orderings. ## model inspired by recent work in non-monotonic generation @xcite @xcite , we propose a model that receives a src, mt pair and outputs one action at a time. when a new action is predicted, there is no explicit memory of previous time-steps. the model can only observe the current state of the mt, which may have been changed by previous actions of the model. this model is based on a transformer-encoder pre-trained with bert @xcite . after producing one hidden state for each token, a linear transformation outputs two values per token: the logit of deleting that token and of inserting a word to its right. out of all possible del or ins positions, the most likely operation is selected. a special operation is reserved to represent end-of-decoding. if an ins operation is chosen, we still need to choose which token to insert. another linear transformation is applied to the hidden state of the chosen position. we obtain a distribution over the vocabulary and select the most likely token. figure 3 illustrates this procedure. after a del or ins is applied, we repeat this procedure using the updated mt. decoding can end in three different ways: once decoding ends, the model outputs the final post-edited mt. model details. we use bert's implementation from @xcite together with opennmt @xcite , both based on pytorch @xcite . the pretrained bert-encoder contains 12 layers, embedding size and hidden size of 768. we begin with an input sequence x. this sequence is the concatenation of: to represent the end-of-decoding operation, we use the action @xmath0. all unavailable actions are masked: @xmath1, ins-@xmath2, and edit-operations on src. when the model predicts an ins, a token is then predicted for that position. let i be the chosen position, h @xmath3, and @xmath4: the predicted action is applied and we repeat this procedure using the updated x. since no history of previous actions is kept, this opens the possibility of entering a loop. to handle loops, when we re-visit a state x we stop decoding. alternatively, we tried applying the n th most likely action on the n th visit to a given x, but this degraded performance slightly. ## training during training, the model may have several correct actions to choose from, even if we only consider actions following a minimum edit distance path. we compare different ground-truth action sequences based on minimum edit actions, all arriving at the same pe: minimum edit actions are generated using the dynamic programming algorithm to compute levenshtein distance. the algorithm is set to output left-to-right actions, but since its output contains no redundant actions, they can be arbitrarily reordered. one simple way to re-order the actions is by randomly shuffling them. a more sophisticated alternative consists in matching each minimumedit action to a human action, as described in §2.2. we also experimented with unfiltered human actions. however this resulted in significantly inferior performance, possibly due to the hesitations made by humans typing, who may add and delete words unnecessary for the final pe. training details. we train the model by maximizing the likelihood of the action sequences provided as ground truth. following @xcite we use adam @xcite with a triangular schedule, increasing linearly for the first 5,000 steps until 5 × 10 -5 , applying a linear decay afterwards. bert components have ℓ 2 weight decay of 10 -4 . we apply dropout @xcite with p @xmath5.1 and, for the loss of vocabulary distribution, label-smoothing @xmath6.1 @xcite . we use batch size of 512 tokens and save checkpoints every 10,000 steps. ## experiments we compare the effect of using different action orders on the development set and test sets of wmt 2018 ape shared task @xcite . by using only training data overlapping with wmt's training sets (as described in §2.1), we are able to use wmt's development and test sets for evaluation. this allows to compare the performance of our model with that of previous submissions. note however that our systems are in disadvantage, due to being trained on fewer data: out of the original 23,000 training samples we only found 16,068 in @xcite . ## conclusions in this work we explored different ways to order the edit operations necessary to fix mistakes in a translated sentence. in particular, we studied which orderings are produced by humans, and whether they can be used to guide the training of a non-monotonic post-editing system. we found that humans tend to use nearly left-toright order, although with exceptions, such as preferring to fix punctuation and verbs first. we then proposed a transformer-based model pre-trained with bert that learns to automatically post-edit translations in a flexible order. we learned this model in three different ways: by supervising it with orderings performed by humans, by using a left-to-right ordering, or with random orderings. in all three settings, the model outperformed the uncorrected machine translation baseline and a previous system also designed to predict actions @xcite . training the model with human orderings achieved performance equivalent to left-to-right, or even superior. the random order consistently yielded slightly lower results. the model learned to mimic the proposed orders in all three cases.