Quantitative Analysis of Post-Editing Effort Indicators for NMT (2020)

Abstract: The recent improvements in machine translation (MT) have boosted the use of post-editing (PE) in the translation industry. A new machine translation paradigm, neural machine translation (NMT), is displacing its corpus-based predecessor, statistical machine translation (SMT), in the translation workflows currently implemented because it usually increases the fluency and accuracy of the MT output. However, usual automatic measurements do not always indicate the quality of the MT output and there is still no clear correlation between PE effort and productivity. We present a quantitative analysis of different PE effort indicators for two NMT systems (transformer and seq2seq) for English-Spanish in-domain medical documents. We compare both systems and study the correlation between PE time and other scores. Results show less PE effort for the transformer NMT model and a high correlation between PE time and keystrokes.

## introduction the use of machine translation (mt) systems for the production of drafts that are later post-edited has become a widespread practice in the translation industry. research has concluded that postediting of machine translation (pemt) is usually more efficient than translating from scratch @xcite @xcite . thus, it has been included in the translation workflow because it increases productivity c 2020 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. when compared with human translation @xcite and reduces costs @xcite without having a negative impact on quality @xcite . post-editors "edit, modify and/or correct pre-translated text that has been processed by an mt system from a source language into (a) target language(s)" @xcite . in recent years, neural machine translation (nmt) has produced promising results in terms of quality, for example in wmt 2019 @xcite . this has increased the interest in this new paradigm for the translation industry, which has begun to substitute its corpus-based predecessor, statistical machine translation (smt), with new nmt models. it has also boosted the incorporation of pemt in many translation workflows. in the 2018 language industry survey, foot_0 37% of the respondents reported an increase of mt postediting and an additional 17% indicated that they had started implementing this practice. given the improved-quality performance of nmt and its widespread use in industrial scenarios, it is necessary to study the potential this approach can offer to post-editing. one of the main problems is that automatic scores give a general idea of the mt output quality but do not always correlate to post-editing effort @xcite . however, many professional translators state that if the quality of the mt output is not good enough, they delete the remaining segments and translate everything from scratch @xcite . one of the main goals both of industry and research is to establish a correlation between the quality measurements of the mt output and translators' performance. regarding post-editing ef-fort, all research uses the three separate but interrelated, dimensions established by krings (2001): temporal, technical and cognitive. temporal effort measures the time spent post-editing the mt output. technical effort makes reference to the insertions and deletions applied by the translator and is usually measured with keystroke analysis, hter @xcite or levenshtein distance (edit distance). cognitive effort relates to the cognitive processes taking place during post-editing and has been measured by eye-tracking or think-aloud protocols. @xcite claimed that post-editing effort could be determined as a combination of all three dimensions. even though no current measure includes them all, cognitive effort was found to correlate with technical and temporal pe effort in a study by @xcite . in this paper we present a preliminary comparative quantitative analysis of different postediting effort indicators (technical and temporal) for two nmt systems for english-spanish indomain medical documents. first of all, we trained a transformer and seq2seq model and compared them with google translate and an smt engine (check section 4.1 for further detail on the results). as the nmt systems produced better quality results, we used them to translate three english-to-spanish medical texts. then, two different translators post-edited each version with posedion, foot_1 a post-editing tool developed mainly to collect information on different direct and indirect effort indicators (technical and temporal effort). in section 2 we analyse some of the previous work on post-editing effort. we explain the different nmt architectures in section 3. in section 4 we detail the mt systems and corpora used. we explain the experimental settings in section 5 and we present the results in section 6. ## previous work nmt is not a new architecture, but it can only be applied once the computational limitations have been solved @xcite . the promising results obtained in automatic metrics such as bleu @xcite have been paired with excellent scores in human evaluation of nmt @xcite @xcite when compared to smt, which has been the predominant mt architecture so far. once the improvement in quality has been determined, it was necessary to analyse its benefits for post-editing. one of the first complete papers studying the impact of smt and nmt in postediting was @xcite . they carried out a small scale study on post-editing nmt and smt outputs of english to german translated ted talks. they conclude that nmt in general terms decreases the post-editing effort, but degrades faster than smt with sentence length. one of the main strengths of nmt is reordering of the target sentence. @xcite increase the initial scope of the study by @xcite by increasing the language combinations and the metrics. one of the main conclusions is an improvement in quality when using nmt, although it is not the same for all the language combinations. @xcite report on a comparative analysis of phrase-based smt (pbsmt) and nmt. they compare four language pairs and different automatic metrics and human evaluation methods. general results show a quality increase for nmt, although it also highlights some of the weaknesses of this new system. it focuses on post-editing and uses the pet interface @xcite to compare educational domain outputs from both systems using different metrics. nmt is shown to reduce word order errors and improve fluency. however, even if keystrokes are reduced, temporal pe effort exhibits no significant reduction. @xcite present a comparison of pe changes performed on nmt, rule-based mt (rbmt) and smt output for the english-finnish language combination. a total of 33 translation students participate in this english-to-finnish pe experiment. it outlines the strategies participants adopt to post-edit the different outputs, which contributes to the understanding of nmt, rbmt and smt approaches. it also concludes that pe effort is lower for nmt than for smt. in industrial scenarios, @xcite show that nmt systems obtain higher rankings by human reviewers than phrased-based smt in all cases. they highlight that automatic measures such as bleu, f-measure @xcite are traditionally post-edited with rbmt systems, nmt systems with worse automatic metrics show better results in human evaluation @xcite . regarding pe effort indicators, pe time is one of the most commonly-used elements to study mt quality, although research shows considerable variation among translators @xcite . hter is another measure frequently used in the industry due to its theoretical correlation to pe effort @xcite . however, research has shown it does not always correspond to translators' perception of quality @xcite . in fact, some authors suggest new ways of measuring pe effort taking into account different scores @xcite or a multidimensional approach that combines some of the currently existing measures @xcite . given the undeniable improvements in quality nmt offers for post-editing, we study two different nmt systems and how they affect different indicators of post-editing effort. we also analyse the correlation of pe time with different direct and indirect measures of technical effort (keystrokes, hbleu, hter and edit distance). as far as we are aware, there are no studies comparing how two different nmt outputs affect post-editing for english to spanish in-domain texts. ## nmt architectures the basic architecture of nmt models @xcite consists of an encoder and a decoder. first of all, each word included in the input sentence is introduced as a separate element into the encoder so that it can encode it into an internal fixed-length representation called the context vector. it contains the meaning of the whole sentence. then, the decoder decodes the context vector and predicts the output sequence. instead of encoding the input sequence into a single fixed context vector, attention @xcite is proposed as a solution to the limitation of the encoder-decoder model encoding the input sequence to one fixed length vector. it develops a context vector that is filtered specifically for each output time step. transformer @xcite follows mainly the encoder-decoder model with attention passed from encoder to decoder. it employs a selfattention mechanism that allows the encoder and decoder to account for every word included in the entire input sequence. transformer proposes to encode each position, apply self-attention in both decoder and encoder, and enhance the idea of selfattention by calculating multi-head attention. this improves performance expanding the model's ability to focus on different positions and gives the attention layer multiple sets of weight matrices. there are no recurrent networks, only a fully connected feed-forward network. ## experiment we used the two nmt systems (transformer and s2s) trained with the corpora described above to translate from english into spanish three texts (1468, 631 and 2247 words respectively) from the medical domain. four professional translators with at least one year of post-editing experience carried out the task: two of them post-edited the s2s output (t1 and t2) and the other two, the transformer output (t3 and t4). they were asked to produce publishable quality translations. as we wanted to reduce the external variables as much as possible, they all used posedion foot_14 , a computer-assisted translation tool specifically designed for assessing postediting effort, which logs both post-editing time and edits (keystrokes, insertions and deletions, that is, technical effort). the main characteristics of the post-editing tool were also explained to them before starting the task. in order to avoid any bias, translators never postedited the same text twice. however, they were told that an nmt system was used to produce the output. they received previous information on the tool and a three day period to test it before doing the task. they were paid their usual rate and had a two-week deadline. two of them expressed concerns about the tool, as they preferred to work with their usual tools. however, they did not think it would affect the final quality of their job or their usual working speed. while postediting, they could search for all the required information in order to produce the final translation. they could also pause the post-editing task whenever they wanted. ## concluding remarks there is a need for reliable metrics to evaluate mt quality in order to produce outputs which translators can post-edit without too much effort. our experiments have shown that no single pe indicator can provide all the information necessary to assess the quality of the mt output. pe time provides a useful measure, even though it does not always correspond with other pe metrics and includes a great variation among translators. the only score that seems directly related to temporal effort are keystrokes (technical effort), but not hter or hbleu. in industrial scenarios, the quality of a certain mt output is usually linked to pe time. the results of our experiments suggest that the analysis of temporal effort can indicate the quality of the mt output, but we believe a multidimensional approach that includes different effort indicators would be a safer path to assess to convenience of post-editing a certain mt output. our future work will study further indicators of mt quality for post-editing in depth, mainly the characterization of source text to assess pe effort.