Bad MT Systems are Good for Quality Estimation (2023)

Abstract: Quality estimation (QE) is the task of predicting quality of outputs produced by machine translation (MT) systems. Currently, the highest-performing QE systems are supervised and require training on data with golden quality scores. In this paper, we investigate the impact of the quality of the underlying MT outputs on the performance of QE systems. We find that QE models trained on datasets with lower-quality translations often outperform those trained on higher-quality data. We also demonstrate that good performance can be achieved by using a mix of data from different MT systems.

## introduction quality estimation (qe) involves predicting the quality of a machine-translated text based on the original text and the machine translation (mt) output @xcite . this can be done at the word, sentence, or document level. in this paper, we focus on sentence-level qe, where the goal is to predict a score that a human assessor would attribute to the sentence. depending on the manual evaluation process used to gather data, we can talk about different variations of the task. these include direct assessment qe @xcite , which aims to estimate the perceived quality of translation, post-editing qe, which measures the effort required to edit the translation, and mqm qe @xcite , which identifies critical errors in the translation. evaluating a qe system means checking how closely its predictions match manual scores on a held-out set. qe systems are closely tied to mt systems in many ways. their performance can vary greatly depending on the mt system on which they are being evaluated. the current high-performing solutions for quality estimation are based on supervised methods, which in turn makes these qe systems dependent on the specifics of the mt systems used to create the training data. it is not clear which mt system should be used to create a qe system with the best performance. the contributions of this experimental work are as follows: ## proposed approach we investigate the impact of mt system quality on qe system performance by training qe models on datasets consisting of a fixed set of source sentences and differing in the target side which is translated by mt systems of varying quality. as there are no existing qe datasets that have the same source sentences translated by different mt systems of known performance, we create our own datasets by training mt systems and translating the same source sentences. due to the lack of human annotators and a large amount of work required, we approximate the manual quality scores, i.e. our targets for qe are assigned automatically. the scores are assigned by calculating the similarity between the translations and reference translations available in a parallel dataset. note that for qe itself, reference translations are not needed, only the quality judgments. we explore the use of two automatic reference-based metrics of mt quality, namely ter @xcite and comet @xcite , as the golden truth for qe training. we select these metrics because they mimic the manual targets typically used in qe tasks, and each highlights a distinct aspect of translation quality. specifically, comet has been trained to predict sentence-level direct assessment scores, while ter is a proxy for hter @xcite , which measures post-editing effort. additionally, we conduct the evaluation of the models trained on comet scores on available data with direct assessment scores to demonstrate that the relationship that holds for proxy targets also applies to real targets. comet is a metric based on sentence embeddings and designed to predict the quality score that a human annotator would assign. this leads us to believe that comet reflects the overall meaning match. as a pre-trained metric, it has a high correlation with human-based scores. however, its training to directly predict da scores is also a limitation. comet may contain a bias towards the mt systems on which it was trained, which is the exact bias that we are trying to evaluate in our qe systems. while comet is available in qe mode with multiple releases, it is not suitable for our purposes, since they differ in various aspects like training procedures, source data, and mts used in training. our focus, however, is solely on understanding the impact of the mt used in translation and using qe comet models would not allow us to separate the mt's impact from other factors affecting the qe evaluation. ter, on the other hand, is focused on string editing, which means a rather superficial similarity of the candidate and the reference translation. it uses the same mechanism of string comparison as hter, so we use it as a proxy for hter-measured post-editing effort. ter is known for having a lower correlation with translation targets. however, it is not trained on translations of any kind, so the risk of any bias towards some training data is avoided. ## experiments our experimental approach involves training qe systems on translations of varying quality, and then evaluating their performance on datasets with different target types, namely comet and ter targets, as well as da targets. in this section, we provide a detailed description of our experimental setup, including information on how we trained the mt and qe systems, as well as the datasets used for training and evaluation. ## results we evaluate the performance of our qe models by computing the pearson correlation between their predictions and the corresponding targets. to determine whether there is a statistically significant difference in correlation between the models, we use a z-test on fisher z-transformed correlation coefficients. ## conclusion our study investigated the impact of mt quality used to train qe systems on the performance of the qe systems. we trained qe models on the datasets that consist of the same source data but different translations produced by mt systems of varying quality. the findings revealed that qe models trained on lower-quality mt translations tended to perform better than those trained on higher-quality mt outputs. additionally, the study suggests that mixing the better and worse mt outputs for training qe models may not necessarily lead to improved qe performance, and the results may vary depending on the specific application or usage scenario.