Quality Estimation with k -nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation (2024)

Abstract: Providing quality scores along with Machine Translation (MT) output, so-called reference-free Quality Estimation (QE), is crucial to inform users about the reliability of the translation. We propose a model-specific, unsupervised QE approach, termed k NN-QE, that extracts information from the MT model’s training data using k -nearest neighbors. Measuring the performance of model-specific QE is not straightforward, since they provide quality scores on their own MT output, thus cannot be evaluated using benchmark QE test sets containing human quality scores on premade MT output. Therefore, we propose an automatic evaluation method that uses quality scores from reference-based metrics as gold standard instead of human-generated ones. We are the first to conduct detailed analyses and conclude that this automatic method is sufficient, and the reference-based MetricX-23 is best for the task.

## introduction machine translation (mt), due to its currently advanced stage in research, has been widely adopted in real-life use cases @xcite . in many application domains such as health care or lawsuits, errors in translation could be tremendously harmful to the users. therefore, it is important to inform the user whether to rely on a certain translation, by providing some kind of quality assessment along with each translation output. this task is referred to as quality estimation (qe). more specifically, quality estimation is assigning quality scores to mt output, without using gold-standard human translation. common qe approaches train a standalone qe module that takes in the source sentences and the mt outputs to produce quality scores. these qe modules are usually model-agnostic, i.e., they can work with the output of any mt model. however, they often require training on human-labeled quality data, which can be costly to obtain. another line of research is on model-specific qe, where they exploit or modify the mt model for self-quality assessment, thus not requiring training a separate qe module. following this line of work, we propose knn-qe -an unsupervised qe approach that exploits the information of inference-time output's k-nearest neighbors found in the mt model's training data. we hypothesize that the closer the inference-time sample output is to the training data, the better the quality of the translation, since it is an indication that the model has learnt about such samples. the qe scores obtained using our method can also be interpreted as the confidence scores of the mt model. unlike model-agnostic qe approaches which can take any mt translation as input, evaluating model-specific qe approaches like knn-qe is not as straightforward. public qe test sets are generated using human quality scores on pre-made mt output, thus not always suitable for qe approaches that perform self-evaluation on their own mt output by design. many previous works on model-specific qe perform human evaluation on their own mt output to be used as gold standard to evaluate qe metrics @xcite @xcite . however, for faster development, it would be useful to automatically evaluate qe metrics and not relying on human resource. therefore, we propose using quality scores generated by reference-based metrics as the gold standard to automatically evaluate reference-free qe metrics. our motivation is that reference-based metrics, by making use of reference translation, tend to be better than reference-free qe metrics @xcite , thus can be used as gold standard. to the best of our knowledge, we are the first to perform a detailed analysis on whether reference-based metrics are sufficient to evaluate qe, and which reference-based metric is best suited. for this analysis, we make use of different qe submissions to public shared tasks: wmt22 metrics @xcite and wmt22 quality estimation @xcite . we investigate whether our automatic qe evaluation method can produce similar qe rankings compared to using human-labeled quality data in these shared tasks. in summary, our contribution is in two folds: model-specific qe researchers have also been looking into integrating quality estimation into mt models. these approaches exploit information or modify white-box mt model to measure the translation quality, rather than training a separate qe module relying completely on human quality data. @xcite uses the attention distributions from the mt model as a qe metric. @xcite uses the attention distribution and the output probabilities from the mt model for qe. @xcite propose qe learnt jointly with the training of the mt model. in their approach, the mt model can ask for hints to improve its translation, and the more hints it asks for, the lower the confidence. @xcite extends the mt model with a self-estimator module for qe, which examines whether it can reconstruct the source sentence' semantics using the information from the decoding procedure. the work by @xcite is the closest to our knn-qe, where they measure the similarity of the test sentence with sentences from the training data to estimate translation quality. the difference between this work and ours is that they focus on evaluating source side rather than target side; they use encoder output similarity rather than decoder output similarity; and they do not analyze different metrics derived from the nearest neighbors. evaluating these model-specific qe approaches is not straightforward, as will be discussed below. ## quality estimation with knn motivation we propose knn-qe -a modelspecific quality estimation method that exploits information from the mt model's training data using k-nearest neighbors. our method is unsupervised, thus does not require human quality scores for training. generally, if the hidden representation of a translation sample generated during in-ference is similar to ones generated on the training data, then it is an indication that this sample is indomain, thus more likely to have higher quality. ## conclusion in this paper, we proposed knn-qe -a modelspecific, unsupervised quality estimation approach which exploits the information from the mt model's training data. we also propose an automatic qe evaluation method for such model-specific qe approaches, which make use of reference-based metrics. our experiments show that this automatic evaluation method is sufficient, and that the reference-based metricx-23 xl is the most suitable. using this automatic qe evaluation method, we found that knn-qe performs better than the mt probability baseline, but still falls behind the supervised qe approach. we also find that our approach works with a small number of retrieved neighbors and a small portion of the training datastore, making it more memory-and timeefficient to be used in practice. for future work, we can explore whether this method is applicable to other types of generative models, such as the currently prominent large language models.