Machine Translation Quality and Post-Editor Productivity (2016)

Abstract: We assessed how different machine translation (MT) systems affect the post-editing (PE) process and product of professional English–Spanish translators. Our model found that for each 1-point increase in BLEU, there is a PE time decrease of 0.16 seconds per word, about 3-4%. The MT system with the lowest BLEU score produced the output that was post-edited to the lowest quality and with the highest PE effort, measured both in HTER and actual PE operations.

## introduction and related work there is a relatively fair amount of empirical research on post-editing machine translation output (pe) focusing on assessing potential time and quality improvements over human translation with or without translation memories (tm). a common finding is that, despite differences among participants, pe is on average faster than unassisted or tm-assisted translation. some examples of studies finding such speed benefits are those by @xcite , @xcite , @xcite , @xcite and @xcite . in terms of pe quality, studies have shown, through the use of human judgments, that pe leads to quality comparable to @xcite or better than other types of translation @xcite @xcite @xcite . there is therefore strong empirical evidence pointing at the speed and quality benefits of pe. while many factors may affect productivity in a pe workflow, mt quality is one that is relatively easy to measure. previous studies to have investigated how mt quality affects pe speed are those by @xcite and @xcite . they found different levels of correlations between mt quality, measured on a sentence level with automatic metrics, and pe speed. @xcite and de sutter (2012) too found that human judgments of mt quality correlated to pe speed. @xcite found their worst mt system entailed 20% more editing activity than their best one. to the best of our knowledge, this is the first study to attempt to assess how different mt systems, of the same type but with different quality levels, affect the pe productivity of professional translators. by investigating how mt quality affects both pe time and pe quality, we aim at providing mt users and researchers with another approach to examining the usefulness of mt for pe purposes. nine translators were hired through proz foot_0 , self-described as the biggest online translation workplace. selected participants were offered a fixed compensation based on the standard, general, english-spanish translation rates displayed on the website. each translator post-edited four news texts of about 650 words each. texts had similar complexity levels and were presented to translators in randomized order to dilute possible familiarization or fatigue effects. all four texts were translated into spanish with nine mt systems (cf. section 4.1). the output of all nine systems was assigned randomly to participants. as in @xcite and @xcite , to deal with between-participant variability, the following restrictions were implemented: translators were asked to post-edit to full, human-like quality. they worked remotely on the open-source, web-based, computer aided translation (cat) tool casmacat (cognitive analysis and statistical methods for advanced computer aided translation) foot_1 @xcite . ## translator profile all participants were native spanish, professional translators, with at least 2 years' experience using cat tools. all were educated to college level. except for tr4 and tr5, all participants had degrees in translation. ## variables of interest 4.1 machine translation quality initiatives providing free resources for mt development and evaluation help boost mt research collaboration and efforts. one of these efforts is the wmt evaluation campaign of the association for computational linguistics (acl). among the data they freely provide are training and test data sets. as part of their 8 th workshop on statistical machine translation (wmt 2013) 3 , the organizers released a test set comprised of 3,000 source english sentences and their corresponding spanish human reference translations. we trained nine mt systems with training data from the european parliament proceedings, news commentary, common crawl, and united nations. the systems are phrase-based moses systems @xcite with hierarchical lexicalized reordering @xcite , operation sequence model @xcite , and sparse lexical features, using all available language model data (target side of the full parallel corpus, plus the provided monolingual news corpus and ldc gigaword). the best system reaches comparable quality to the best system participating in the wmt 2013 evaluation campaign. we iteratively halved the parallel training corpus to obtain systems of inferior quality. the quality of the mt systems was measured with case-sensitive bleu @xcite on the official wmt 2013 test set. ## analysis and results sentences with a logged period of inactivity of 2 minutes or more were excluded from analysis as such long pauses are likely not indicative of difficulties posed by the underlying mt system. also excluded were sentences inadvertently skipped by translators and sentences with unreliable measurements 5 . in total, out of 1233 observations, 1202 were considered valid and submitted for analysis.