Enhancing Supervised Learning with Contrastive Markings in Neural Machine Translation Training (2023)

Abstract: Supervised learning in Neural Machine Translation (NMT) standardly follows a teacher forcing paradigm where the conditioning context in the model’s prediction is constituted by reference tokens, instead of its own previous predictions. In order to alleviate this lack of exploration in the space of translations, we present a simple extension of standard maximum likelihood estimation by a contrastive marking objective. The additional training signals are extracted automatically from reference translations by comparing the system hypothesis against the reference, and used for up/down-weighting correct/incorrect tokens. The proposed new training procedure requires one additional translation pass over the training set, and does not alter the standard inference setup. We show that training with contrastive markings yields improvements on top of supervised learning, and is especially useful when learning from postedits where contrastive markings indicate human error corrections to the original hypotheses.

## introduction due to the availability of large parallel data sets for most language pairs, the standard training procedure in neural machine translation (nmt) is supervised learning of a maximum likelihood objective where reference tokens constitute the target history in the conditional language model, instead of the model's own predictions. feeding back the reference history in model training, known as teacher forcing @xcite , encourages the sequence model to stay close to the reference sequence, but prevents the model to learn how to predict conditioned on its own history, which is the actual task at inference time. this lack of exploration in learning has been dubbed exposure bias by @xcite . it has been tackled by techniques that explicitly integrate the model's own prediction history into training, e.g. scheduled sampling @xcite , minimum risk training @xcite , reinforcement learning @xcite , imitation learning @xcite , or ramp loss @xcite , amongst others. in most of these approaches, feedback from a human expert is simulated by comparing a system translation against a human reference according to an automatic evaluation metric, and by extracting a sequence-or token-level reward signal from the evaluation score. in this paper, we present a method to incorporate contrastive markings of differences between the model's own predictions and references into the learning objective. our approach builds on previous work on integrating weak human feedback in form of error markings as supervision signal in nmt training @xcite . this work was conceptualized for reducing human annotation effort in interactive machine translation, however, it can also be used on simulated error markings extracted from an automatic evaluation score. it allows the model to extract a contrastive signal from the reference translation that can be used to re-inforce or penalize correct or incorrect tokens in the model's own predictions. such a reward signal is more fine-grained than a sequence-level reward obtained by a sequence-level automatic evaluation metric, and less noisy than token-based rewards obtained by reward shaping @xcite . our hypothesis is that such contrastive markings should be especially useful in learning setups where human postedits are used as reference signals. in such scenarios, contrastive markings are likely to indicate erroneous deviations of machine translations from human error corrections, instead of penalizing correct translations that happen to deviate from independently constructed human reference translations. we confirm this hypothesis by simulating a legacy machine translation system for which human postedits are available by performing knowledge distillation @xcite on the stored legacy machine translations. we define a "legacy" machine translation system as a system which was previously used in production and produced translations for which human feedback was gathered, but which is no longer productive. knowledge distillation is required because the legacy system is a blackbox system that is unavailable to us, but its outputs are available. for comparison, we apply our framework to standard parallel data where reference translations were generated from scratch. our experimental results show that on both datasets, combining teacher forcing on postedits with learning from error markings, improves results with respect to ter on test data, with larger improvements for the knowledge-distilled model that emulates outputs of the legacy system. a further novelty of our approach is the true online learning setup where new error markings are computed after every epoch of model training, instead of using constant simulated markings that are pre-computed from fixed machine translation outputs as in previous work @xcite @xcite . online error markings can be computed in a light-weight fashion by longest common subsequence calculations. the overhead incurred by the new training procedure is one additional translation pass over the training set, whereas at inference time the system does not require additional information, but can be shown to produce improved translations based on the proposed improved training setup. ## related work most approaches to remedy the exposure bias problem simulate a sentence-level reward or cost function from an automatic evaluation metric and incorporate it into a reinforcement-or imitationlearning setup @xcite @xcite @xcite @xcite . methods that are conceptualized to work directly with human postedits integrate the human feedback signal more directly, without the middleman of an automatic evaluation heuristic. the standard learning paradigm is supervised learning where postedits are treated as reference translations (see, for example, @xcite ). most approaches to learning from error markings adapt the supervised learning objective to learn from correct tokens in partial translations @xcite @xcite . the quickedit @xcite approach uses the hypothesis produced by an nmt system and token-level markings as an extra input to an automatic postediting system (ape), and additionally requires markings on the system output at inference time. this requires a dual encoder architecture with the decoder attending to both the source and hypothesis encoders. in this case, convolutional encoders and decoders of @xcite are used. our approach builds upon the work of @xcite and @xcite who incorporate token-level markings as learning signal into nmt training. in contrast to grangier and auli (2018), who compute markings offline before training and require them for inference, we only require them during training and calculate markings online. furthermore, instead of presenting markings to the system as an extra input, they are integrated into the objective function as a weight. while @xcite simulate markings from reference translations by extracting deletion operations from longest common subsequence calculations, @xcite show how to learn from markings solicited from human annotators. in contrast to these approaches, we integrate markings to enhance supervised learning in a true online fashion. ## data we use the wmt17 en-de dataset foot_0 for pretraining. our data is pre-processed using the moses tokenizer and punctuation normalization for both english and german implemented in sacremoses foot_1 . we first test our ideas on the iwslt14 en-de dataset foot_2 @xcite . we download and pre-process the data using joey scripts foot_3 . the en-de dataset consists of transcribed ted talks and volunteer provided reference translations into the target languages. the ape dataset is from the wmt automatic postediting shared task 2021 @xcite . the legacy system that produced the original mt outputs is based on a standard transformer architecture @xcite and follows the implementation described by @xcite . this system was trained on publicly available mt datasets, including paracrawl @xcite and europarl @xcite data consists of source, mt output, and postedit triples. the source data was selected from english wikipedia articles. the mt outputs were provided by the legacy system and were postedited by professional translators. the sizes of the datasets are given in ## discussion our experimental results in furthermore, this result has implications for how to best use postedits. postedits are often treated as new reference translations for the sources and used to train new systems, whereas the original mt outputs are discarded. however, fine-tuning the original system on the postedits may yield larger improvements than training a new, unrelated model on the source and postedit alone. lastly, we believe that our results can be interpreted as the effect of mitigating exposure bias. the pre-trained model is exposed not only to reference translations, but to its own trajectories. even if the model's trajectory is far from the gold reference and multiple tokens in its history are incorrect, it will be rewarded if it predicts a token that is in the output. this may enable it to return to a more rewarding trajectory. ## conclusion in this work we present a way to combine postedits and word-level error markings extracted from the edit operations between the postedit and the mt output to learn more than what the postedit alone is able to provide. experimentally, we try this on systems unrelated to the legacy system, whose outputs were originally postedited, and on a simulation of the legacy system we create via knowledge distillation. we show that these contrastive markings are able to bring significant improvements to ter scores and we hypothesize this is because they are able to target insertion errors that contribute to higher ter scores. additionally, learning from the model's own output may allow it to learn how to correct itself after making an error if it is later rewarded for correct outputs. source the superstructure was armored to protect the bases of the turrets , the funnels and the ventilator ducts in what he termed a breastwork . postedit der überbau wurde gepanzert , um die fundamente der türme , der trichter und der ventilatorkanäle in dem bereich zu schützen , den er als brustwehr bezeichnete . epoch 0 system hypothesis a die superstruktur wurde getrieben , um die stützpunkte der turm-, der funn ¤ rn-und der ventilator die herde in dem , was er als die bruststbesteigung bezeichnet hatte zu schützen . b die superstruktur wurde getrieben , um die stützpunkte der turm-, der funn ¤ rn-und der ventilator die herde in dem , was er als die bruststbesteigung bezeichnet hatte zu schützen . c der überbau wurde gewagelt , um die fundamente der türme , die funnels und die ventilatorenkanäle in einem brustwerk zu schützen . d der überbau wurde gewagelt , um die fundamente der türme , die funnels und die ventilatorenkanäle in einem brustwerk zu schützen . epoch 1 system hypothesis a die superstruktur wurde gezeichnet , um die stützen der turrets , der funnels und der ventilator in seiner art brustwork zu schützen . b die überbauung war gepanzert , um die grundstücke der turrets , der funnels und der vaterfunkanten in dem , was er als brustwerk nannte , zu schützen . c der superbau wurde gepanzert , um die stützpunkte der turrets , der funnels und der ventilatorentötungen in einer so genannten brustarbeit zu schützen . d der überbau wurde gepanzert , um die fundamente der türme , der trichter und der ventilatorkankanäle zu schützen , was er als brustwerk nannte . although " überbau" is a part of " überbauung", the subwords used to construct them differ, leading to "bau" in " überbauung" being marked as incorrect.