Automatic Correction of Human Translations (2022)

Abstract: We introduce translation error correction (TEC), the task of automatically correcting human-generated translations. Imperfections in machine translations (MT) have long motivated systems for improving translations post-hoc with automatic post-editing. In contrast, little attention has been devoted to the problem of automatically correcting human translations, despite the intuition that humans make distinct errors that machines would be well-suited to assist with, from typos to inconsistencies in translation conventions. To investigate this, we build and release the Aced corpus with three TEC datasets (available at: github.com/lilt/tec). We show that human errors in TEC exhibit a more diverse range of errors and far fewer translation fluency errors than the MT errors in automatic post-editing datasets, suggesting the need for dedicated TEC models that are specialized to correct human errors. We show that pre-training instead on synthetic errors based on human errors improves TEC F-score by as much as 5.1 points. We conducted a human-in-the-loop user study with nine professional translation editors and found that the assistance of our TEC system led them to produce significantly higher quality revised translations.

## introduction despite recent progress in machine translation (mt), a tremendous amount of translated content in the world is still written by humans @xcite . humans are often assumed to produce trusted, high-quality translations. in reality, they do make errors, including spelling, grammar, and translation errors @xcite . this paper introduces the task of translation error correction (tec). given a source sentence s and a humangenerated translation t, the goal of tec is to produce an improved translation t ′ by correcting all errors in t. using this error analysis, we propose an approach for tec that pre-trains on synthetic corruptions more similar to errors made by humans, outperforming models that were developed for the related tasks of mt, grammatical error correction, and ape on all aced datasets. the task of tec is often currently performed by humans, e.g. translators hired to review and edit translations ("reviewers"). can a tec system help reviewers edit faster, or produce higher quality final translations than they would have without assistance? we ran a human-in-the-loop user study with nine professional translators using our bestperforming tec model. we found that the reviews produced when assisted with a tec system were rated as higher quality than those produced without, and produced with less manual effort. qualitatively, users commented that trust and consistency of the suggestions were critical. they speculated future automated assistance could be helpful for onboarding to new content, spotting technical errors, and improving their own awareness of errors to catch. looking forward, a natural question arises of whether the research community should focus on learning to revise model outputs (ape) or human outputs (tec). with recent improvements in mt, it has been increasingly difficult for ape models to improve model output that is already high quality @xcite . on the other hand, we should expect that humans will continue to make errors. tec models will continue to provide benefit as a way of assisting humans, whether for professional translation work or everyday language learners. tec is synergistic with continuing advancements in mt: improved mt will lead to improved error correction for human-generated translations. while ape pits models against models, tec is an opportunity to combine the best of humans and models because humans and models make different errors. in sum, this paper revisits the notion of "translation correction" conceived narrowly as the mtcentered task of ape, with an empirical investigation of translation error correction (tec), the task of learning to correct human translations. our contributions are: ## the ♠ aced corpus for tec given a source language sentence s and a humangenerated translation t, the goal of tec is to produce a corrected target language sentence t ′ . we introduce the aced corpus, a set of three tec datasets: asics, emerson, and digitalo-cean (do), each consisting of english-german sentence triples (s, t, t ′ ) from varying domains. aced is a real-world benchmark, containing naturalistic data from a task humans perform, rather than manually annotated data. all translations were created by professional translators working with lilt, a localization services provider. all translators have at least 5 years of professional translation experience and experience working with the customer and domain. each document was translated from scratch (i.e. not post-edited) by a human translator using an interactive neural mt system. each translated document was then reviewed by a reviewer, who lilt selects as one of the more senior translators. as a result, the examples in our corpus exhibit real errors that translators make, and the corrected translations are publication quality. secondly, aced is diverse, with the three datasets from varying domains exhibiting different error distributions and difficulty for initial work on the tec task. information for each dataset is shown in duplicate sentences with the same source s were removed. a portion of sentences were rewritten by the reviewer rather than being edited (a relative edit distance of more than 25% and a minimum of two edited words). we replace t in these sentences with t ′ in the corpus, so that training and evaluation focuses on local edits rather than re-translations. the train, dev, and test splits were constructed by splitting along document boundaries. ## approaches to tec we propose a tec model and compare it to several models designed for related tasks to determine whether they are also effective for tec. an overview of the differences between the models is shown in figure 1 . all models use the transformer neural architecture @xcite that generates the target sequence t ′ from left to right. all are pre-trained on 36m sentences from the wmt18 asics emerson do train total edits 606 1436 1212 unique edits 418 486 940 dev total edits 246 381 1004 % in train 14 63 21 test total edits 287 364 766 % in train 23 60 21 all models have 6 encoder and decoder layers, model dimension of 256, feed-forward dimension of 512, and 8 attention heads. we use a joint english-german vocabulary with 33k byte pair encoding subwords @xcite . during pre-training, we set the dropout to 0.1 and use the adam optimizer (kingma and ba, 2015) with a learning rate of 0.0002. during fine-tuning, we decrease the learning rate to 0.0001 and reset the adam momentum parameters. we select the best fine-tuning checkpoint with edit-level f 0.5 score on our dev set. we use greedy inference. ## results & discussion the primary metric for tec is maxmatch scores (m 2 ) @xcite computed with the errant toolkit @xcite . m 2 is a standard metric for gec that aligns t and t ′ to extract discrete "edits." we choose to follow the gec evaluation practice of up-weighting precision by comparing f 0.5 , since the original translation is mostly correct: it is better to suggest few correct edits than potentially introduce new errors. both the mt model (which ignores t) and the gec model (which ignores s) underperform tec. the mt model's high edit recall can be attributed to the fact that it proposes many edits, greatly trading off precision. without conditioning on t, direct mt translations of the source diverge from the reference. the gec model obtains high precision but underperforms on recall. conditioning on s not only makes it possible to propose bilingual edits, but also provides additional information to correct monolingual edits, as we show in section 4.1. can ape models be directly adapted for tec? since our task is structurally identical to ape, a natural question is whether models that are trained on the ape objective can be directly adapted for tec. the ape and tec models differ only in pretraining, but the performance difference between them is substantial, indicating that the more geclike data synthesis procedure is a better fit for tec than ape-style data synthesis via mt. even more, the bert-ape model, which is first fine-tuned to achieve state-of-the-art on ape before fine-tuning on aced, achieves a particularly low f 0.5 score because it makes too many edits (low precision). although future work may find insights in ape, our results emphasize that models that excel at correcting machine errors cannot be assumed to work well on human translations. ## user study: assisting professional translators with tec we are ultimately interested in whether any tec system is indeed useful in practice. presently, tec is done manually by humans. to investigate whether tec systems can already be useful to humans-improving the quality, speed, or ease of human review-we performed a human-in-theloop user study with our tec model. ## conclusion & future work we introduced the task of translation error correction (tec) and released the aced corpus to study automatic correction of human translations, consisting of three tec datasets across varying domains. in our analysis of tec data, we showed how the errors that humans make differ from those made by mt systems, suggesting that this task warrants different approaches from those previously studied in the task of automatic post-editing. we confirm this empirically by proposing a synthetic data generation procedure that more closely matches the distribution of human translation errors and showing that our tec model, pre-trained on this data, consistently outperforms models developed for ape, as well as those for mt and gec. finally, we showed how our tec system is helpful to real humans, assisting professional reviewers and leading them to produce higher quality reviewed translations. future work may improve on our tec system by investigating how to leverage the strengths of recent mt systems (e.g. for initializing systems or proposing edits) or developing more sophisticated synthetic data generation techniques (e.g. using the source sentence or linguistic knowledge). beyond our benchmark, it would be interesting to apply tec systems to other settings in which human translation errors appear, e.g., to correct translations written by language learners, denoise mt training sets, or clean up mt evaluation sets. from the perspective of human-ai interaction, tec presents a real-world use case and testbed to study how to assist experts with modern nlp systems, hinting at the opportunity to combine the best of humans and machines.