Investigating the Helpfulness of Word-Level Quality Estimation for Post-Editing Machine Translation Output (2021)

Abstract: Compared to fully manual translation, post-editing (PE) machine translation (MT) output can save time and reduce errors. Automatic word-level quality estimation (QE) aims to predict the correctness of words in MT output and holds great promise to aid PE by flagging problematic output. Quality of QE is crucial, as incorrect QE might lead to translators missing errors or wasting time on already correct MT output. Achieving accurate automatic word-level QE is very hard, and it is currently not known (i) at what quality threshold QE is actually beginning to be useful for human PE, and (ii), how to best present word-level QE information to translators. In particular, should word-level QE visualization indicate uncertainty of the QE model or not? In this paper, we address both research questions with real and simulated word-level QE, visualizations, and user studies, where time, subjective ratings, and quality of the final translations are assessed. Results show that current word-level QE models are not yet good enough to support PE. Instead, quality levels of > 80% F1 are required. For helpful quality levels, a visualization reflecting the uncertainty of the QE model is preferred. Our analysis further shows that speed gains achieved through QE are not merely a result of blindly trusting the QE system, but that the quality of the final translations also improves. The threshold results from the paper establish a quality goal for future word-level QE research.

## introduction advances in machine translation (mt) have made mt a key component in many professional translation workflows, where human post-editors identify and correct mistakes in raw mt output. overall, post-editing (pe) saves time and reduces errors @xcite @xcite , however, for sentences of low mt quality, pe can take longer than manually translating from scratch @xcite . in order to better support and guide posteditors, it would therefore be helpful to indicate mt quality and flag potential errors. quality estimation (qe) focuses on this goal by predicting how good mt output is based on factors such as fluency, comprehensibility, and adequacy @xcite . in contrast to standard mt evaluation methods such as bleu @xcite and ter @xcite , qe does not rely on reference translations for comparison. instead, qe can be framed as a supervised machine learning task, predicting the quality of an mt output for a source sentence @xcite . qe models can be created at the wordlevel, sentence-level, and document-level @xcite . sentence-level scores can be leveraged for the basic decision whether to post-edit a sentence or to translate from scratch. word-level quality estimates provide more fine-grained output that allows post-editors to quickly identify incorrect words, thereby aiming for faster translations of higher quality. however, if a qe prediction is incorrect, translators might waste time on already correct mt output or overlook errors. apart from the quality of the qe system, the way its output is presented to the user can also have an impact on its usefulness. in particular, should the uncertainty of the model be communicated to the user or is a classification into correct/incorrect more suitable? this paper focuses on word-level qe, which holds great potential in supporting pe but is also a very difficult machine learning task: even the best models currently only achieve f1 scores in the range 60% to 63% @xcite depending on the text domain and the underlying mt system used to generate the data. while publicly available datasets and shared tasks are rapidly advancing qe research, we are not aware of any research that investigated the level of quality required to make word-level qe useful in prac-tical applications. in this paper, we address this research question in terms of a well-controlled experiment with professional translators who are presented with word-level qe output of varying quality, based on state-of-the-art qe systems and simulated qe output, with the goal of determining minimum quality thresholds required to support the pe process, thereby setting a target for future qe model developers. since visualization of qe output might also have an impact on the helpfulness in pe, we compare two visualization techniques in our study. the results indicate that current stateof-the-art word-level qe models are not yet good enough to support pe, but that quality levels of at least 80% f1 are required. for the desired qe quality levels, a visualization showing the model's uncertainty is preferred. ## related work word-level qe attempts to automatically mark words in an mt proposal, such that words requiring post-editing are labelled as 'bad', while all others are tagged as 'ok' @xcite . this is usually cast as a binary classification task for each word (and gap) in an mt output. qe research is strongly advanced through shared tasks organized by the workshop/conference for machine translation (wmt) every year since 2012 @xcite . until recently, wmt used the target f1 score as performance metric for word-level qe. in 2019 @xcite , the matthews correlation coefficient @xcite ) was introduced in addition to the primary metric (f1), and became the new main metric in 2020 @xcite due to its usefulness on unbalanced classes, since in word-based qe, ok tags are much more common than bad tags. however, as works prior to 2019 did not report mcc, we will focus on f1 on the remainder of the paper. the word-level model of @xcite won the wmt 2018 qe shared task @xcite by achieving an f1 score of 62.46%. their "qe-brain" uses a pre-trained neural bilingual expert model @xcite , extracting semantic features from both the source and translation output for estimating translation quality with a bidirectional lstm @xcite . in particular, three important strategies were utilized, namely incorporating human-crafted features, artificial qe data augmentation @xcite for more diversified train-ing data, and a model ensemble with a greedy algorithm @xcite . @xcite won the wmt 2019 word-level qe task @xcite by combining linear, neural, and predictor-estimator systems with new transfer learning approaches using bert @xcite and xlm @xcite pretrained models, achieving an f1 score of 47.5%. in the most recent word-level qe shared task of 2020 @xcite , lee (2020) proposed an xlm-r-based neural network architecture, which was trained in two phases. first, the model was trained with a very large artificially generated dataset based on a parallel corpus with opennmt-py @xcite and the ter tool @xcite . then, the model was fine-tuned with a humanlabeled dataset. this approach achieved the highest f1 score of 61.89% among the submissions. note that the scores across the years are not directly comparable because the datasets used are different: as mt gets better, qe models need to adapt to work well on this changed output. even though there has been considerable progress in qe and detailed analyses on datasets are done, to date there is very little research on the usability and impact of word-level qe in the pe process. in particular, no research has investigated whether current word-level qe models are already good enough to reduce translators' pe effort and increase translation quality and throughput, and if not, which level of qe quality is required to achieve this. as discussed in section 1, the way qe output is visualized to the human translator is also important for the pe process. @xcite propose to simply visualize the ok and bad labels predicted by their state-of-the-art word-level qe system as either red or green words. additionally, a gap indicating missing words is depicted by a red-colored underscore. similarly for the sentence level, @xcite use binary color-coded labels for visualizing mt quality information and assess whether this can lead to noticeable gains in translators' productivity: indeed, the authors show a statistically significant pe speedup, however, only for source sentences containing 5-20 words and a mt quality of more than 0.1 hter. inspired by the work of @xcite , parra escartín et al. (2017) modified the post-editing tool (pet, @xcite ) to present translators with a traffic light system of sentence-level qe visualizations that indicates whether they need to translate the source text from scratch or post-edit the mt. their results indicate that good and accurate mt qe, is vital to the efficiency of the translation workflow, and can cut translation time and effort significantly. @xcite explore the impact of expressing the scores from different sentence-level qe models as percentages between 20% and 99%. their study with 20 professional translators that post-edited four pieces of text each indicates that just displaying sentence-level percentage scores is not enough. instead, one should also visualize word-level qe predictions. to summarize, while word-level qe research has considerably advanced, with novel word-level qe systems emerging on a year by year basis, existing studies usually focus on detailed technical analyses on the test data, without user studies to investigate if the quality levels achieved by the qe systems are already useful for enhancing translation throughput or reducing errors in practical pe. today, our best word-level qe models achieve f1 scores in and around 60% to 63%. in this paper we test whether this is sufficient for practical use, and if not, what quality levels are required. we do this through a user study with professional translators, comparing state-of-the-art qe model output with simulated qe output of higher quality levels created by manipulating ground truth data. furthermore, we compare the binary red/green visualization proposed by @xcite to a color gradient-based visualization showing the uncertainty of the model. ## concept and implementation state-of-the-art word-level qe models roughly attain a quality level of 60% to 63% f1. this quality level may not yet be good enough to guide posteditors well: if the qe fails to detect errors, a posteditor relying on the qe might miss them more easily than without qe. furthermore, correct output marked as incorrect could lead to wasted time trying to figure out what the mistake might be. hence, incorrect predictions can have a severe impact on productivity and translation quality. to ascertain which quality level would be sufficient to start helping, we artificially generate data at quality levels higher than what can be achieved by current state-of-the-art word-level qe models. the process of simulating qe output, as well as integrating their visualization into a translation environment, is explained in the following sections. ## evaluation using our implementation, we conducted a user study to assess the quality threshold when wordlevel qe starts facilitating and stops hindering the pe process. we further want to find out which word-level qe visualization is preferred. ## discussion our results show that existing state-of-the-art wordlevel qe systems are not yet good enough to be helpful during pe. instead, all our analyses agree that qe systems need an f1 score of at least 80% to support pe in terms of subjective helpfulness, editing duration, and quality of the final translations. this establishes a target for future qe research. in terms of visualization, the word-level quality scores should be visualized using gradient-based visualization which also shows uncertainties of the model, since the binary approach was considered superior only in cases where qe was not helpful. this preference is interesting as the exact color chosen for the gradient was randomly sampled in the red/green range for bad/ok ratings for the artificial qe output. a reason for the preference could be a stronger involvement in the decision process and hence a stronger feeling of control. as expected, with increasing qe quality, pe becomes more efficient, where in particular the higher quality levels of 85% and 95% require less editing time than the lower quality levels. lastly, we found that translation speed gains obtained by qe are not merely a result of blindly trusting the qe system, but indeed help producing higher quality translations. to sum up, a qe quality level of at least 80% f1 sets the approximate boundary where word-level qe starts helping translators, and for these qe quality levels, a gradient-based visualization is preferred. ## conclusion and future work the goal of this paper was to estimate how accurate a word-level qe system would need to be in order to support the pe process. furthermore, we also test how to best visualize the output of such a system. since we hypothesized that stateof-the-art qe systems may not yet be good enough to aid the pe process, we developed an approach to generate higher quality levels artificially. we performed a user study where the output from a state-of-the-art qe model and three artificial qe models were presented to users in a cat tool using either a binary or gradient-based red/green visualization, where the latter also shows uncertainties of the qe model. the results of the evaluation show that current word-level qe models are not yet good enough to guide post-editors; instead, quality levels of at least 80% f1 serves as a reasonable first approximate boundary required to aid the pe process. for these quality levels, gradient is the preferred visualization scheme. in the future, we plan to explore the impact of qe models' performance on different language pairs and text domains. furthermore, we will investigate how sentence-level and word-level qe can best be combined for efficient pe. as our current quality analysis only verified that a qe hinting towards a certain reference indeed leads to translations close to that reference, we would like to extend our analysis with a manual human translation quality analysis that is not bound to a comparison to a single reference. besides, we want to explore further non-color-based visualization schemes. finally, in addition to having a word quality prediction task, we want to also explore gap prediction and how this can be best visualized.