Correct Me If You Can: Learning from Error Corrections and Markings (2020)

Abstract: Sequence-to-sequence learning involves a trade-off between signal strength and annotation cost of training data. For example, machine translation data range from costly expert-generated translations that enable supervised learning, to weak quality-judgment feedback that facilitate reinforcement learning. We present the first user study on annotation cost and machine learnability for the less popular annotation mode of error markings. We show that error markings for translations of TED talks from English to German allow precise credit assignment while requiring significantly less human effort than correcting/post-editing, and that error-marked data can be used successfully to fine-tune neural machine translation models.

## introduction successful machine learning for structured output prediction requires the effort of annotating sufficient amounts of gold-standard outputs-a task that can be costly if structures are complex and expert knowledge is required, as for example in neural machine translation (nmt) @xcite . approaches that propose to train sequenceto-sequence prediction models by reinforcement learning from task-specific scores, for example bleu in machine translation (mt), shift the problem by simulating such scores by evaluating machine translation output against expert-generated c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. our work is the first to investigate learning from error markings in a user study. error corrections and error markings are collected from junior professional translators, analyzed, and used as training data for fine-tuning neural machine translation systems. the focus of our work is on the learnability from error corrections and error markings, and on the behavior of annotators as teachers to a machine translation system. we find that error markings require significantly less effort (in terms of key-stroke-mouse-ratio (ksmr) and time) and result in a lower correction rate (ratio of words marked as incorrect or corrected in a post-edit). furthermore, they are less prone to over-editing than error corrections. perhaps surprisingly, agreement between annotators of which words to mark or to correct was lower for markings than for postedits. however, despite of the low inter-annotator agreement, fine-tuning of neural machine translation could be conducted successfully from data annotated in either mode. our data set of error corrections and markings is publicly available. foot_1 ## related work prior work closest to ours is that of marie and max (2015); @xcite ; @xcite , however, these works were conducted by simulating error markings by heuristic matching of machine translations against independently created human reference translations. thus the question of the practical feasibility of machine learning from noisy human error markings is left open. user studies on machine learnability from human post-edits, together with thorough performance analyses with mixed effects models, have been presented by @xcite ; @xcite ; @xcite . albeit showcasing the potential of improving nmt through human corrections of machine-generated outputs, these works do not consider "weaker" annotation modes like error markings. user studies on the process and effort of machine translation postediting are too numerous to list-a comprehensive overview is given in @xcite . in contrast to works on interactive-predictive translation @xcite @xcite @xcite , our approach does not require an online interaction with the human and allows to investigate, filter, pre-process, or augment the human feedback signal before making a machine learning update. machine learning from human feedback beyond the scope of translations, has considered learning from human pairwise preferences @xcite , from human corrective feedback @xcite , or from sentence-level reward signals on a likert scale @xcite . however, none of these studies has considered error markings on tokens of output sequences, despite its general applicability to a wide range of learning tasks. ## user study on human error markings and corrections the goal of the annotation study is to compare the novel error marking mode to the widely adopted machine translation post-editing mode. we are interested in finding an interaction scenario that costs little time and effort, but still allows to teach the machine how to improve its translations. in this section we present the setup, measure and compare the observed amount of effort and time that went into these annotations, and discuss the reliability and adoption of the new marking mode. machine learnability, i.e. training of an nmt system on human-annotated data is discussed in section 4. ## machine learnability of nmt from human markings and corrections the hypotheses presented to the annotators were generated by an nmt model. the goal is to use the supervision signal provided by the human annotation to improve the underlying model by machine learning. learnability is concerned with the question of how strong a signal is necessary in order to see improvements in nmt fine-tuning on the respective data. definition. let @xmath0. . . x s be a sequence of indices over a source vocabulary v src , and @xmath1. . . y t a sequence of indices over a target vocabulary v trg . the goal of sequence-tosequence learning is to learn a function for mapping a input sequence x into an output sequences y. for the example of machine translation, y is a translation of x, and a model parameterized by a set of weights θ is optimized to maximize p θ (y | x). this quantity is further factorized into conditional probabilities over single tokens p θ (y | x) = t @xmath2; @xmath3, where the latter distribution is defined by the neural model's softmaxnormalized output vector: there are various options for building the architecture of the neural model nn θ , such as recurrent @xcite , convolutional @xcite or attentional @xcite encoder-decoder architectures (or a mix thereof @xcite ). learning from error corrections. the standard supervised learning mode in human-in-theloop machine translation assumes a fully corrected output y * for an input x that is treated similar to a gold standard reference translation @xcite . model adaptation can be performed by maximizing the likelihood of the user-provided corrections where learning from error markings. a weaker feedback mode is to let a human teacher mark the correct parts of the machine-generated output ŷ (marie and @xcite @xcite . as a consequence every token in the output receives a reward δ m t , either δ + t if marked as correct, or δ - t otherwise. @xcite proposed a model with δ + @xmath4, but this weighting schemes leads to the ignorance of incorrect outputs in the gradient and the rewarding of correct tokens. instead, we find it beneficial to penalize incorrect tokens, with e.g. δ - @xmath5.5, and reward correct tokens δ + @xmath6.5, which aligns with the findings from @xcite . the objective of the learning system is to maximize the likelihood of the correct parts of the output where ## conclusion we presented the first user study on the annotation process and the machine learnability of human error markings of translation outputs. this annotation mode has so far been given less attention than error corrections or quality judgments, and has until now only been investigated in simulation studies. we found that both according to automatic evaluation metrics and by human evaluation, finetuning of nmt models achieved comparable gains by learning from error corrections and markings. however, error markings required several orders of magnitude less human annotation effort. in future work we will investigate the integration of automatic markings into the learning process, and we will explore online adaptation possibilities.