Introducing Quality Estimation to Machine Translation Post-editing Workflow: An Empirical Study on Its Usefulness (2025)

Abstract: This preliminary study investigates the usefulness of sentence-level Quality Estimation (QE) in English-Chinese Machine Translation Post-Editing (MTPE), focusing on its impact on post-editing speed and student translators’ perceptions. The study also explores the interaction effects between QE and MT quality, as well as between QE and translation expertise. The findings reveal that QE significantly reduces post-editing time. The interaction effects examined were not significant, suggesting that QE consistently improves MTPE efficiency across MT outputs of medium and high quality and among student translators with varying levels of expertise. In addition to indicating potentially problematic segments, QE serves multiple functions in MTPE, such as validating translators’ evaluation of MT quality and enabling them to double-check translation outputs. However, interview data suggest that inaccurate QE may hinder the post-editing processes. This research provides new insights into the strengths and limitations of QE, facilitating its more effective integration into MTPE workflows to enhance translators’ productivity.

## introduction in a typical machine translation post-editing (mtpe) workflow, translators still need to spend a certain amount of time and effort on evaluating the quality of machine translation (mt) outputs to determine the cost-effectiveness of mtpe. to be more specific, if the mt output is of acceptable quality, post-editing is feasible; otherwise, translating from scratch may be more efficient. however, this process can be time-consuming, especially when the mt outputs are ultimately deemed unsuitable for post-editing. in order to achieve a quick turnaround, it is therefore necessary to speed up or even automate the process of evaluating whether mtpe is worthwhile @xcite . the cost-effectiveness of mtpe can be evaluated from two different but interrelated perspectives: predicting the mt quality @xcite , to see whether translators are going to work with "good" mt or "bad" mt, or predicting mtpe effort @xcite , to see how much effort, such as post-editing time and editing distance, is required by the pe task. in the field of computer science, both approaches are considered as quality estimation (qe) 1 . in contrast to traditional referencebased metrics such as bleu @xcite , qe estimates mt quality without requiring reference translation @xcite , making it particularly relevant for real-world translation scenarios. in the current research, we focus on the qe method that provides mt quality scores, rather than the one that estimates mtpe effort. the latter may be less straightforward for translators when making post-editing decisions, since a threshold that sets a point from which post-editing becomes translating from scratch @xcite ) has yet to be widely established. the possible advantages of adopting qe to facilitate the mtpe workflow extend beyond streamlining the initial assessment of mtpe's costeffectiveness. by providing information about the estimated quality of mt outputs, qe may help translators to allocate their efforts more effectively and focus on the outputs that deserve editing. on the one hand, they can spend minimal time on the 1 however, since pe effort is a complex, multidimensional concept influenced by various factors -including but not limited to mt quality -and is not necessarily linearly related to mt quality @xcite , we argue for a clear distinction between the tasks of predicting mt quality and those of predicting pe effort, rather than grouping them into the same category. despite the potential benefits and advancements of qe, it seems that qe has yet to be widely integrated in the real post-editing settings (gilbert, 2022). one possible reason for this is the scarcity of cat tools that can effectively incorporate qe. while some cat tools, such as trados, have recently started offering qe information, it is usually not freely accessible, posing a challenge to the widespread adoption of qe. moreover, there is limited empirical evidence supporting the usefulness of qe in enhancing mtpe workflow, which makes it challenging for translators to embrace qe, as they may be uncertain about its practical value and impact on their work. in real-world applications, various factors could influence the effectiveness of qe, such as translators' attitudes towards qe, the accuracy of qe information, and the actual quality of mt outputs. therefore, a critical question arises: to what extent and under what conditions can qe facilitate the mtpe processes? in light of the above, this study investigates the usefulness of sentence-level qe foot_0 in the context of english-chinese mtpe, taking both productivity and users' perceptions into consideration. it is expected that the current research can provide a more detailed understanding of qe's application in aiding post-editing tasks, shedding new lights on its strengths and limitations. this insight will contribute to a more effective integration of qe into the mtpe workflow, enhancing efficiency for translators. specifically, it focuses on the following three research questions: 1. what is the impact of sentence-level qe on post-editing time? 2. is the impact of sentence-level qe on post-editing time consistent across different conditions, in particular, varying levels of mt quality and translation expertise? 3. what are users' perceptions of the usefulness of sentence-level qe in post-editing? in addition to productivity, it is essential to explore how translators perceive the usefulness of qe and the challenges they encounter when interacting with it. while earlier surveys revealed translators' interest in using qe information @xcite @xcite , few studies have investigated the perceptions of translators after letting them actually work with qe during post-editing, and the findings have been inconclusive. parra @xcite collected translators' opinions on qe and revealed generally negative attitudes towards its usefulness. however, the reasons behind these negative results were not examined in this study. by contrast, most of the participants in @xcite expressed positive views on qe, particularly regarding its usefulness for proofreading purposes, such as doublechecking the potential translation errors. apart from investigating the general impact of qe, efforts have also been made to consider additional factors and examine whether the usefulness of qe varies under specific conditions. for instance, given that qe was found to contribute only slight and insignificant global productivity gains in @xcite study, the authors conducted an additional analysis to explore whether these marginal gains might become more pronounced under certain conditions. the analysis incorporated the length of source text (st) and the quality of mt outputs, and the results suggested that qe led to significant productivity gains when the sentences were of medium length and had hter foot_1 values between 0.2 and 0.5. the accuracy of qe has also been examined, with somewhat conflicting results. parra @xcite found that qe, especially good qe that provided a predicted quality score close to the actual score, significantly decreased post-editing time. however, @xcite reported that no significant effect was introduced by qe, even when it was accurate. in addition, while not explicitly addressed as a variable of interest, @xcite presented data pertaining to translation experience. in this study, despite varying levels of experience, all translators, with only one exception, increased their post-editing speed when qe information was provided. in conclusion, there is a notable lack of empirical research on the effectiveness of presenting sentence-level qe information within the mtpe context, and the findings to date have been inconsistent. while considering additional factors has provided a more nuanced understanding of qe's impact on post-editing efficiency, more research is warranted. it should be noted that most previous studies have relied on basic statistical analyses, which may not fully capture the true impact of qe. additionally, while professional translators have been the focus of these studies, the way student translators utilise qe information in the mtpe workflow has yet to be investigated, which can provide valuable insights into translation education. ## conclusions and future work motivated by the potential benefits of qe in streamlining mtpe workflow, the current study preliminarily explored the usefulness of sentence-level qe in increasing post-editing speed and gathered student translators' views about its application. three major findings emerged. first, qe significantly reduced post-editing time, and no significant interaction effects were found between qe and mt quality or between qe and translation expertise. in other words, the impact of qe remained consistent across mt outputs of medium and high quality and among students with varying levels of translation expertise. this stability implies that the advantages of qe in reducing post-editing time are likely to be broadly applicable. second, the benefits of using qe in post-editing extend beyond highlighting problematic mt segments, it can also validate translators' own evaluations of mt quality and assist in quality checking. these findings shed new light on how translators can integrate qe information into the mtpe workflow to enhance overall efficiency. finally, although this study did not explicitly examine the impact of qe's accuracy, interview data indicate a potential detrimental effect of inaccurate qe on post-editing processes. however, this finding requires further empirical validation. this study has several limitations that open avenues for future research, particularly regarding the number and diversity of texts and participants, the range of factors considered, and the indicators used to measure mtpe efficiency. in future research, we aim to expand the sample size by including more participants and a wider variety of text types to better understand the conditions under which qe proves most beneficial. comparisons between student and professional translators will also be views n main reasons increasing quality but not necessarily the speed 4(33.3%) validating translators' own evaluation of mt quality; assisting quality check increasing both quality and speed 4(33.3%) saving time and effort for more difficult translations; assisting quality check no impact 2(16.7%) a firm belief in translators' own evaluation of mt quality increasing speed 1(8.3%) the ability of qe to highlight potentially erroneous mt decreasing speed 1(8.3%) low accuracy of qe; distrust of qe in conclusion, this study provides preliminary evidence for the usefulness of sentence-level qe in the mtpe context. instead of simply saying "no" to qe, we should embrace its potential and investigate how to optimise its integration into mtpe workflows. as one interviewee aptly remarked, "if we have access to such information, why not use qe?" this perspective encapsulates the pragmatic value of qe and underscores the need for further exploration into its role in enhancing mtpe efficiency.