Neural Machine Translation Quality and Post-Editing Performance (2021)

Abstract: We test the natural expectation that using MT in professional translation saves human processing time. The last such study was carried out by Sanchez-Torron and Koehn (2016) with phrase-based MT, artificially reducing the translation quality. In contrast, we focus on neural MT (NMT) of high quality, which has become the state-of-the-art approach since then and also got adopted by most translation companies. Through an experimental study involving over 30 professional translators for English -> Czech translation, we examine the relationship between NMT performance and post-editing time and quality. Across all models, we found that better MT systems indeed lead to fewer changes in the sentences in this industry setting. The relation between system quality and post-editing time is however not straightforward and, contrary to the results on phrase-based MT, BLEU is definitely not a stable predictor of the time or final output quality.

## introduction machine translation is increasingly utilized in the translation and localization industry. one of the most common use cases is mt post-editing where human translators use mt outputs as a starting point and make edits to obtain the final translation. this process has been shown to be more efficient than translating from scratch, i.e. it and can lead to reductions in cost and delivery time @xcite . while various studies have looked at how mt post-editing affects translation quality and speed in general, few have attempted to measure how the quality of mt outputs affects the productivity of translators. in this work, we attempt to answer the following questions: we have carried out a large-scale study on one language pair that involved over 30 professional translators and translation reviewers, who worked in two stages to post-edit outputs of 13 different sources (11 mt engines, 1 raw source, 1 human reference). this allowed us to collect not only the post-editing times but also to estimate the quality of the produced results. based on this data, we present an in-depth analysis and provide observations and recommendations for utilizing mt in localization workflows. while the task for humans for both rounds is the same (improve a given translation by freely editing it), we strictly distinguish the first and second rounds, using the term "posteditor" in the first round and "reviewer" in the second round. we make the code along with all collected data (including all translations) publicly available. 1 2 related work one of the earliest experiments that noticed a significant correlation between various automatic evaluation metrics and post-editing speed was performed by @xcite . the survey of @xcite briefly covers the history of postediting and pinpoints the two main topics: effort and final output quality. the authors conclude that post-editing improves both compared to translating the original text from scratch, given suitable conditions (good mt quality and translator experience with post-editing). experiments were done by @xcite and @xcite show that purely monolingual post-editing leads to results of worse quality than when having access to the original text as well. finally, koponen (2013) comments on the high variance of post-editors, which is a common problem in postediting research @xcite . interactive mt is an alternative use case of computer-assisted translation and it is possible that effort or behavioural patterns in interactive mt could be used as a different proxy extrinsic measure for mt quality. post-editor productivity has also been measured in contrast to interactive translation prediction by @xcite . similar experiments. our work is most similar to sanchez-torron and @xcite and @xcite , which served as a methodological basis for our study. while most of the previous works focused on statistical mt, we experiment solely with neural mt (nmt) models. many studies have shown that nmt has very different properties than older mt models when it comes to post-editing @xcite . for instance, nmt outputs tend to be very fluent which can make post-editing more cognitively demanding and error-prone as suggested by @xcite . @xcite showed that in a specific setting, the adequacy of nmt is higher than that of human translators. we believe that the relationship between nmt system quality and pe effort is not a simple one and that older results based on statistical mt may not directly carry over to nmt. the first of the six challenges listed by @xcite suggests that fluency over adequacy can be a critical issue: nmt systems have lower quality out of domain, to the point that they completely sacrifice adequacy for the sake of fluency. additionally, our focus is state-of-the-art nmt systems, which was not true for sanchez-torron and @xcite , who constructed 9 artificially severely degraded statistical phrase-based mt systems. the experiment by @xcite used only 4 mt systems. our focus is motivated by the industry's direct application: considering the cost of skilled staff and model training, what are the practical benefits of improving mt performance? in contrast to the previous setups, we evaluate two additional settings: post-editing human reference and translating from scratch, corresponding to a theoretical foot_1 bleu of 100 and 0, respectively. we also consider the quality of the pe output and not only the process itself. sanchez-torron and @xcite found a linear relationship between bleu and pe effort: for each 1-point increase in bleu, there is a pe time decrease of 0.16 seconds per word, about 3-4%. the performance of the mt systems they use is, however, close to uniformly distributed between 24.85 and 30.37. the observed linear relationship can then be partially attributed to the lower mt performance of artificially uniformly distributed mt systems. ## experiment design in this section, we thoroughly describe the design of the study, including the used data, mt engines and the translation process. ## post-editing effort to measure the post-editing effort, we examine first the differences between provided translation and the post-edited output (section 4.1). we then focus on the time spent post-editing, which is an extrinsic evaluation of this task (section 4.2).