Assessing Post-editing Effort in the E nglish- H indi Direction (2021)

Abstract: We present findings from a first in-depth post-editing effort estimation study in the English-Hindi direction along multiple effort indicators. We conduct a controlled experiment involving professional translators, who complete assigned tasks alternately, in a translation from scratch and a post-edit condition. We find that post-editing reduces translation time (by 63%), utilizes fewer keystrokes (by 59%), and decreases the number of pauses (by 63%) when compared to translating from scratch. We further verify the quality of translations thus produced via a human evaluation task in which we do not detect any discernible quality differences.

## introduction translation workflows that are based on postediting of machine translation output are being increasingly adopted in the industry @xcite . gains that accrue from a post-editing based workflow, measured over multiple post-editing effort indicators, have been reported to be considerably significant by a number of previous studies over multiple language combinations @xcite ; c. m. de @xcite . but to extend post-editing beyond its current silos it is imperative to put new and less-studied language pairs under the lens to make a case for wider adoption via empirically backed evidence. 1 post-editing effort is often quantified across three different dimensions, each focusing in turn on a different aspect of post-editing behaviour @xcite . the dimensions studied are the following: temporalunderstood as the time taken to complete a translation task, often reported per segment or word; technicalestimate of the physical labour of the translation activity, measured in terms of keystrokes logged or edit operations performed; and cognitivean indirect estimate of the extent of cognitive processes underlying the translation task, inferred from keylogging pause or eye-tracking data as it is not possible to observe these directly @xcite . if it can be shown that post-editing machine translation output is temporally efficient, technically less laborious, and cognitively less demanding, then it can be recommended as the default workflow for large translation jobs. but this first calls for a comparison between machine translation based post-editing behaviour (henceforth pe) and unaided human translation from scratch (henceforth ht). thus, the research questions that we pose are the following: most of this paper will focus on answering the first question in some detail. we are equally interested in the other two as well, but will only be presenting some initial results from a first attempt at tackling them. the rest of this paper is organized as follows: section 2 discusses some past studies including those that have previously studied the english-hindi pe direction. in section 3 we detail our experimental setup. section 4 presents our results and analysis and in section 5 we draw our conclusions and sketch the outlines of our future work. ## related work we now take a more detailed look at some of the past efforts towards contrasting the two settings. @xcite compared ht and pe when translating from english into 4 european languages (french, italian, german, and spanish) and reported an overall productivity gain of 74% which converted into time savings of 43%. they also observed a 70% reduction in keyboard time and 31% in pause time for the pe setting. c. m. de @xcite also report pe to be 40% faster than ht in the english-portuguese direction when translating movie subtitles. other studies however @xcite , have reported more modest gains, with estimated time savings of 15-20% when translating between a european language pair (german-french) within a realistic translation environment. 2 garcia (2011) also finds only marginal productivity gains when studying the english-chinese pair and additionally reports an impact of directionality when source and target languages are switched. all of these earlier experiments, however, were based on the output of phrase based statistical machine translation (pbsmt) systems. with neural machine translation (nmt) and its subsequent iterations being the current state of the art and outperforming pbsmt @xcite @xcite , this shift in technology paradigm from pbsmt to nmt must then be addressed in post-editing studies as well. @xcite conduct such a study, this time utilizing the output of an nmt system to compare pe with ht in the german-french and german-italian translation directions. 3 they report significant overall productivity gains, but with marked differences between the pairs: 59.74% for the former and only 9.26% for the latter. another interesting comparison of ht, pbsmt, and nmt post-editing settings performed on a literary text (chapter from a novel) reports an increase in productivity by 36% for the nmt based setting over ht @xcite . we have seen in previous studies that throughputs vary considerably depending on the language pair under the lens @xcite . we now discuss some earlier efforts that have included an indian language in their experiment. @xcite conducted an experiment where students post-edited parts of a specialized english language textbook on bioelectromagnetism into 7 languages, 3 of them being indian languages including hindi. they reported an increase in postedit time by a factor of 3-5 when the target language was an indian language. they put this down to greater terminological distance between english and indian languages compared to other languages in their experiment. they do not study and compare against the ht condition, or report on technical or cognitive indicators. carl et al. (2016) also report results on hindi (amongst 6 languages) comparing the ht and pe conditions. their english-hindi results are based on an existing multilingual translation database that contains experimental data around translators' activities in both conditions. they find in favour of the pe condition across all languages when measured on temporal indicators, but report translating into hindi to be the slowest amongst the 6. they do not quantify average throughput gain or time savings. meetei et al. (2020) compare pe behaviour when translating from english into 3 indian languages (manipuri, mizo and hindi). they conduct light post-editing and report hindi to be the fastest to post-edit amongst the three languages. 4 they ascribe it to the availability of relatively mature mt systems in the english-hindi direction compared to mizo and manipuri, which are low-resource languages. they use student volunteers and do not investigate cognitive indicators. @xcite present an industry perspective and claim a 2-3 fold increase in productivity for users using their tools in combination with mt. however, they base this on longitudinal tracking of their users. while all these past studies have certainly helped in providing insights into post-editing behaviour in indian languages in general and in hindi in particular, we sense a need for a more in-depth look along all pe effort indicators within at least one indian-language setting. with our current work, we seek to address this gap. we see our contribution differentiated from previous related work as follows: (i) we present indepth results from all three primary indicators of pe effort (temporal, technical and cognitive) for the english-hindi direction; (ii) we account for per translator and per item variation with the use of mixed-effects models; (iii) we utilize professional translators in order to accurately gauge the impact of contrasting conditions; (iv) we conduct a human quality-rating exercise comparing target text produced in both conditions; (v) we present correlations of automatic mt metrics with pe effort indicators. ## experimental setup we conducted this experiment under a 2 (translation conditions) × 200 (source segments) mixed design. all subjects saw both factor levels (ht and pe), but only one combination for each level as having been exposed to a source segment in one condition would have affected their translation in the other. the study was conducted online over 5 consecutive days with 2 sessions per day. the sessions were not time bound. subjects translated 2 files of 10 segments each in alternating conditions in each session. all subjects participated in a warm-up translation exercise a week prior to the start of the actual task. this was done to establish familiarity with the interface used in the study. we chose to adapt an existing beta version of a web-based translation workbench by adding extensive keystroke logging features along with some other minor tweaks. 5 the ui itself was kept clean and uncluttered, serving one segment at a time to the translators. this meant that while translators had previous context of the text under translation, they could not navigate ahead for context. a timer was displayed once a translator navigated to each new segment. this was meant to prompt the translator to focus on the activity at hand. figure 1 shows the workbench interface as seen by translators under the pe condition. we instructed the participants to aim for publishable translation quality. they were free to conduct web searches and consult online or offline dictionaries, but were discouraged from spending too much time doing so. 6 it was deemed acceptable to transliterate any technical terms or terminology into hindi if they could not find its translation even with the aid of resources available to them. however, they were strictly prohibited from consulting any online mt engines during the task. subjects were encouraged to complete each task (consisting of 10 segments) in a single sitting without a break. previous studies, such as those discussed earlier, have noted the impact of a number of different variables (language pairs, mt paradigms, text domains, translation environments, translator competencies) on translation throughputs. this calls for not only careful experiment design, but also utilization of techniques that can help with the testing and inference of results. @xcite utilized one of the first such designs for post-editing productivity studies and deployed mixed-effects models @xcite to account for inter-language, inter-subject, and inter-item variability. mixed-effects models are able to model this variability in two ways: (i) through random intercepts, that can account for the differences between translators seen in their differing throughputs (or differences between linguistic items due to the features inherent to them); (ii) and through a random slope that accounts for how different subjects may experience the change of condition differently. accounting for these variabilities allows us to isolate the effect of condition, generalize our findings beyond our sample, and avoid the "language-as-fixed-effect fallacy" @xcite . in fitting our mixed-effects models we follow a methodology similar to the one described by @xcite and followed by @xcite and later @xcite . maximal models were fit when possible @xcite case of convergence failure, a less complex model was fit by successively removing the random slopes of the by-subject and by-segment random effects component. models thus obtained were compared via likelihood ratio tests. we also refit our final models after filtering data points with residuals deviating more than 2.5 standard deviations. this helps check for the influence of any atypical outliers @xcite . we verify the residual plots for normality and homoscedasticity. we utilized the lme4 package in r @xcite for all mixed-effects models related analyses. ## conclusion we conducted a post-editing effort assessment study and presented detailed analysis of effort indicators along the temporal, technical and cognitive dimensions. we observed that in the temporal dimension, post-editing reduced translation time by 63%; in the technical dimension it reduced number of key strokes by 59%; and in the cognitive dimension, it reduced the frequency of pauses by 63%. however, it increased average pause duration by 12% and average initial pause duration by 5%. we then compared the quality of translations generated in each condition and found them to be similar. and finally, we detected moderate to strong correlations for 3 automatic mt evaluation metrics across all pe effort indicators, with technical effort most strongly correlating with automatic mt metrics. the last two observations regarding human quality judgement, and mt metrics and their correlations demand a closer look, which was not possible owing to time and space constraints. we expect to undertake this as part of our future work. we also propose to extend this study by including a third condition in future, either as an additional mt engine to check if mt quality differences show up in pe effort indicators @xcite , or by the use of translation aids (tm) to gauge their impact in a similar manner @xcite . we also intend to study other language pairs, especially those within the multilingual indian context.