Validating Quality Estimation in a Computer-Aided Translation Workflow: Speed, Cost and Quality Trade-off (2021)

Abstract: In modern computer-aided translation workflows, Machine Translation (MT) systems are used to produce a draft that is then checked and edited where needed by human translators. In this scenario, a Quality Estimation (QE) tool can be used to score MT outputs, and a threshold on the QE scores can be applied to decide whether an MT output can be used as-is or requires human post-edition. While this could reduce cost and turnaround times, it could harm translation quality, as QE models are not 100% accurate. In the framework of the APE-QUEST project (Automated Post-Editing and Quality Estimation), we set up a case-study on the trade-off between speed, cost and quality, investigating the benefits of QE models in a real-world scenario, where we rely on end-user acceptability as quality metric. Using data in the public administration domain for English-Dutch and English-French, we experimented with two use cases: assimilation and dissemination. Results shed some light on how QE scores can be explored to establish thresholds that suit each use case and target language, and demonstrate the potential benefits of adding QE to a translation workflow.

## introduction quality estimation (qe) for machine translation (mt) predicts how good or reliable automatic translations are without access to gold-standard references @xcite @xcite . this is especially useful in real-world settings, such as within a translation company, where it can improve post-editing efficiency by filtering out segments that require more effort to correct than to translate from scratch @xcite , or select high-quality segments to be published as they are @xcite . however, while the utility of mt is widely accepted nowadays, thus far no research has looked into validating the utility of qe in practice, in a realistic setting. to address this gap, in this paper we ask ourselves the following questions: 1) can qe make the translation process more efficient (i.e. faster and cheaper)? 2) what is the impact of a qe-based filter on the quality of the final translations? and 3) how does varying the threshold for this filter affect these two competing goals (efficiency and quality)? in the ape-quest project for automated post-editing and quality estimation @xcite ), foot_0 we set up a proof-of-concept environment combining mt with qe. this quality gate was integrated within the workflow of the two companies in the consortium (crosslang and unbabel), specialized in computer-aided translation: predicted qe scores are used to decide whether an mt output can be used as-is (predicted as acceptable quality) or should be post-edited (predicted as unacceptable quality). it is expected that this quality gate speeds up the translation workflow and reduces costs since not all mt outputs would require human post-edition, but having humans read translations to make this decision is time-consuming. however, without a good understanding of the effects of qe-based filtering, there is a risk that the workflow becomes biased towards maximising throughput, i.e. towards selecting more low-quality translations as acceptable, and thus compromising the quality of the final translations. we propose a simple approach to studying the trade-off between speed, cost and quality, and show how important it is in allowing the quality gate to provide sufficiently-good mt while employing humans to only post-edit "difficult" sentences. we also show that this varies depending on the intended use of the translations. our experiments with the quality gate use state-of-the-art neural mt (nmt) and qe models with texts in the public administration domain, and translation use cases with different quality requirements (section 3). to elaborate a realistic trade-off model, stakeholder input is important. as such, we collected human post-edits (along with post-editing time) and end-user acceptability judgements (binary scores) for two use cases (assimilation and dissemination) and two language pairs (english-dutch and english-french) to evaluate the quality gate in different scenarios (section 4). this data served to analyse how varying thresholds of qe scores affect post-editing time, overall cost and end-user acceptability, where we compare the quality gate against a human-only translation workflow (all mts are checked and post-edited) and an mt-only translation workflow (all mts are used as-is). results (section 5) show that qe scores can be used to establish thresholds that reduce cost and time, while maintaining similar quality levels as the human-only workflow, for all use cases and target languages. the gains are even greater when using oracle scores instead of predicted scores, signalling the benefits of improving this type of technology. this trade-off methodology for establishing qe thresholds proved helpful to demonstrate the benefits of incorporating qe in real-world computer-aided translation workflows (section 6). ## related work previous studies on the benefits of qe in translation workflows compared translators' productivity when post-editing selected mt outputs (based on qe scores) versus translating from scratch. @xcite found that significant gains depend on the length of the source sentences and the quality of the mt output. similarly, parra @xcite showed that translators spent less time post-editing sentences with "good" qe scores, i.e. scores that accurately predicted low pe effort. different from these studies, we do not investigate impact on post-editor productivity, but rather whether it is possible to rely on qe scores to selectively bypass human post-edition and still achieve similar levels of translation quality. in addition, we experiment with state-of-the-art neural qe systems instead of feature-based ones. the applicability of neural qe was investigated by @xcite when translating software ui strings from microsoft products. the authors compared three systems in terms of business impact (using microsoft's business metrics, such as throughput), model performance (using standard metrics, such as pearson correlation), and cost (in terms of training and inference times). different from theirs, our work relies on end-user translation acceptability as primary evaluation metric. finally, some work attempted to determine thresholds for metrics' scores to identify ranges where post-editing productivity gains can be obtained @xcite , or improvement in the quality of the raw mt output can be expected @xcite . however, they were based on post-hoc computations of ter (translation edit rate) or edit distance, respectively, instead of predicted qe scores as in our case. in addition, we experiment with thresholds of qe scores that benefit the overall translation workflow for different use cases and language pairs. ## quality gate we describe the technical components of the quality gate, the translation workflows that it compares to, and the translation use cases we considered. ## evaluation protocol our trade-off model should help to answer the following questions: in addition, we investigate how the answers to these questions vary for: (1) different thresholds on the predicted quality of translations; (2) each of the two use cases (assimilation and dissemination); (3) different target languages; and (4) different quality of the qe scores (predicted vs oracle). ## experiments and results we evaluate the performance of the quality gate workflow depending on different values of qe score thresholds. we present the three evaluation metrics: quality, cost and time. for better visualisation, we normalized time as a percentage with respect to the traditional workflow. we set up thresholds from 0 to 1 in 0.05 increments, and computed the evaluation metrics under the assumption that sentences whose qe score was below the threshold required hpe. more specifically, for these sentences we took their post-editing time and quality judgement after hpe to calculate the metrics. for the rest (i.e. sentences not "requiring hpe") their time is 0 and their quality judgement is that of the mt output. we first use the predicted qe scores to evaluate the current performance of the quality gate (section 5.1). then, we experiment with an oracle scenario where the qe scores are perfect, in order to measure the potential best-case-scenario performance of the quality gate workflow (section 5.2). ## conclusions in this paper, we provided evidence of the benefits of introducing qe into the computer-aided translation workflow of a company. in the framework of the ape-quest project, we implemented a quality gate that decides, based on predicted qe scores, whether mt outputs can be used as-is (acceptable quality) or if they require post-edition (unacceptable quality). we performed a trade-off study to establish thresholds on the qe scores that allow reducing time and cost, while keeping translation quality more or less stable. we collected human post-edits and acceptability ratings from real use case scenarios and real end-users, and demonstrated that the quality gate can obtain similar levels of quality to the current human-only workflow, for all use cases and target languages explored. in addition, when the predicted qe scores are changed to oracle ones, the gains are higher, illustrating the potential benefits of improving the predictive abilities of the qe models.