Quality Estimation Using Minimum B ayes Risk (2023)

Abstract: This report describes the Minimum Bayes Risk Quality Estimation (MBR-QE) submission to the Workshop on Machine Translation’s 2023 Metrics Shared Task. MBR decoding with neural utility metrics like BLEURT is known to be effective in generating high quality machine translations. We use the underlying technique of MBR decoding and develop an MBR based reference-free quality estimation metric. Our method uses an evaluator machine translation system and a reference-based utility metric (specifically BLEURT and MetricX) to calculate a quality estimation score of a model. We report results related to comparing different MBR configurations and utility metrics.

## introduction the task of quality estimation (qe) is to assign a sentence-or word-level quality score to a machine translation (mt) output without the use of a reference translation. in this paper, we describe the methodology used in our sentence-level qe metric submission to the 2023 workshop on machine translation's metrics shared task. minimum bayes risk (mbr) decoding has been widely used in machine translation to address the limitation of map decoding @xcite @xcite . @xcite showed applying mbr decoding using bleurt @xcite as a utility function can out-perform beam search decoding. mbr decoding can be viewed as a method for reranking candidate outputs from an mt system. it first samples a set of hypothesis translations from the model, scores each hypothesis against a set of pseudo-references (generally, the same set of sample hypotheses) with a utility metric, then selects the hypothesis with the highest average score to be the final translation. central to mbr is assigning a quality score to a hypothesis translation without the use of a reference. because this decoding procedure has been successful in improving the quality of translations from an mt system, in this work, we explore how mbr could be repurposed as a qe metric. our proposed metric uses an mt system in conjunction with a utility metric to assign a quality score to a translation without using a reference. the metric assigns a score to a hypothesis translation by using the utility metric to evaluate the hypothesis against a set of pseudo-references that are sampled from the mt model. in this work, we experiment with creating a metric that uses different mt systems, utility functions, and different pseudo-reference pool sizes. our experiments demonstrate that (1) a better utility function results in better mbr-qe scores, (2) the choice of mt system can have significant impact on qe metric performance, and (3) the size of the pseudo-reference pool does not have a significant impact on overall metric quality. based on our experiments, we chose our primary mbr-qe submission to be an in-house encoderdecoder model with metricx @xcite as the utility function with a pseudo-reference pool size of 256. ## metric descriptions mbr decoding has two components: an mt system and a utility function. the mt model p model (y|x) estimates the probability of target segments y given a source segment x. the utility function estimates the quality of a translation h given a reference translation r. the best hypothesis is selected using the expected utility with respect to a finite sample generated by the model. the underlying assumption is that the model provides good approximation for the true distribution of human translations. we adopt that assumption to develop an mbr-based quality estimation metric. the mbr-qe metric uses an mt system p to generate the set of pseudo-references, denoted r. then, the utility function defines the quality of a translation h given r as the average score over all of r: this methodology has multiple potential pitfalls. first, because the distribution p is used to substitute for the distribution of human translations, any significant divergence between these two distributions will lead to the qe score becoming inconsistent because the pool of pseudo-references will not resemble human references. this can be mitigated by using a high quality mt system. arguably, the mt system should have better performance compared to the mt models that are being evaluated. second, our qe metric is dependent on the quality of the utility function. if it has limitations or biases, they will affect the predicted quality scores and introduce inconsistencies between the qe score and ground-truth human quality scores. we next discuss the experimental setup for analyzing our proposed qe metric. ## experimental results the main experimental results are shown in tables 1 and 2. comparing utility functions. for both mt systems and all pseudo-reference pool sizes, the mbr-qe metric that uses metricx as a utility function in general has higher correlations than when bleurt is used. this result is expected since metricx was the best performing metric in the wmt'22 evaluation. this is evidence that the quality of the utility function is important for the quality of the mbr-qe score. comparing mt systems. when comparing whether the encoder-decoder mt system or palm 2 is used to generate the pool of pseudo-references, there is no clear winner between the two. the mbr-qe score has a higher correlation at the segment-level with the encoder-decoder model, but the correlations are higher at the system-level with palm 2. it is not clear why this is the case. ## related work incorporating evaluation metrics into reranking the outputs from mt systems has been very successful. for example, the @xcite showed that reranking translations with bleurt as part of mbr produced higher-quality translations. this work served as the inspiration for our qe metric submission. research on quality estimation focuses on predicting word-and sentence-level quality scores @xcite . the most successful approaches to predicting sentence-level scores are learned regression-based metrics that are trained to predict ground-truth quality scores, like cometkiwi @xcite or unite-src @xcite . our metric is quite different from these approaches in that it is not directly trained to predict quality scores, but rather it leverages a reference-based metric combined with an mt system to score a translation. to the best of our knowledge, ours is the first metric that uses mbr to build a qe metric. ## conclusion in this report, we proposed a new qe metric called mbr-qe that repurposes an mt system in combination with mbr to score a translation without ac