Uncertainty-Aware Machine Translation Evaluation (2021)

Abstract: Several neural-based metrics have been recently proposed to evaluate machine translation quality. However, all of them resort to point estimates, which provide limited information at segment level. This is made worse as they are trained on noisy, biased and scarce human judgements, often resulting in unreliable quality predictions. In this paper, we introduce uncertainty-aware MT evaluation and analyze the trustworthiness of the predicted quality. We combine the COMET framework with two uncertainty estimation methods, Monte Carlo dropout and deep ensembles, to obtain quality scores along with confidence intervals. We compare the performance of our uncertainty-aware MT evaluation methods across multiple language pairs from the QT21 dataset and the WMT20 metrics task, augmented with MQM annotations. We experiment with varying numbers of references and further discuss the usefulness of uncertainty-aware quality estimation (without references) to flag possibly critical translation mistakes.

## introduction evaluation of machine translation (mt) quality is a key problem with several use cases: it is needed to compare and select mt systems, to decide on the fly whether a translation is ready for publication or needs to be post-edited by a human, and more generally to track progress in the field @xcite . even when reference translations are available, the increasing quality of neural mt systems has made traditional lexical-based metrics such as bleu @xcite or chrf @xcite insufficient to distinguish the best systems. this fostered a line of work on neural-based metrics, with recent proposals such as bleurt @xcite , comet @xcite and prism @xcite . metrics for quality estimation (qe; when references are not available) have also been devel- mt da comet ua-comet она сказала, -0.815 0.586 0.149 'это не собирается [-0.92, 1.22] работать. gloss: "she said, 'that's not willing to work" она сказала: 0.768 1.047 1.023 это не сработает. gloss: "she said, «that will not work" in this paper, we propose a simple and effective method to obtain uncertainty-aware quality/metric estimation systems, by representing quality as a distribution, rather than a single value. to this end, we make use of and compare two well-studied techniques for uncertainty estimation: monte carlo (mc) dropout @xcite and deep ensembles @xcite . in both cases, our method is agnostic to the particular metric estimation system, as long as it can be ensembled or perturbed. in our experiments we use comet @xcite , and we call our uncertainty-aware version ua-comet. 1 our method allows using the same system with a varying number of references. we show that confidence intervals tend to shrink as more references are added, which matches the intuition that mt evaluation systems should become more confident as they have access to more information. we evaluate our approach using data from the wmt20 metrics task @xcite , including its recent extension with google mqm annotations @xcite , and the qt21 dataset @xcite . the results show that our uncertainty-aware systems exhibit better calibration with respect to human direct assessments (da; @xcite , multi-dimensional quality metric scores (mqm; @xcite , and human translation error rates (hter; @xcite ) than a simple baseline, while their average quality scores achieve similar or better correlation than the vanilla comet system. finally, we illustrate a potential quality estimation use case enabled by our approach: automatically detecting low-quality translations with a risk-based criterion. confidence estimation in mt a related line of work is confidence estimation of sentence-level mt outputs @xcite @xcite . the work that relates the most with ours is the one by @xcite , who propose an unsupervised glass-box approach to qe, extracting uncertainty-related features from the mt system via mc dropout. they show that the more confident the decoder (as measured by the lower variance of its output), the higher the quality of the mt output. our work builds upon this perspective to propose uncertainty estimation of the qe systems themselves, rather than uncertainty of mt. performance prediction in nlp a related problem is that of predicting the performance of an nlp system without having to train it @xcite . recent approaches perform such predictions by adding confidence intervals @xcite and measuring calibration error. we take inspiration from these works to improve the calibration of our methods @xcite and to evaluate how good our uncertainty estimates are with a suite of performance indicators. ## evaluating uncertainty having described our framework, we now turn to the problem of verifying the effectiveness and informativeness of the proposed uncertainty quantification method. two crucial aspects to take into account when evaluating uncertainty-aware systems are: (i) the system should not harm the predictive accuracy compared to a system without uncertainty and (ii) the uncertainty estimate should reflect the failure probability of the system well, meaning that the system "knows when it does not know." in what follows, we assume a test or validation set calibration error one way of understanding if models can be trusted is analyzing whether they are calibrated @xcite @xcite , that is, if the confidence estimates of its predictions are aligned with the empirical likelihoods @xcite . in classification tasks, this is assessed by the expected calibration error (ece; naeini et al. 2015), which has been generalized to regression by @xcite . it is defined as: we use this metric with @xmath0. negative log-likelihood to evaluate parametric methods that represent the full distribution pq (q), we can use a single metric that captures both accuracy and uncertainty, the average negative loglikelihood of the ground truth quality scores according to the model: ## conclusions we introduced uncertainty-aware mt evaluation and showed how mt-related applications can benefit from this approach. we compared two techniques to estimate uncertainty, mc dropout and deep ensembles, across several performance indicators. through experiments on three datasets with different human quality assessments encompassing several language pairs, we have shown that the resulting confidence intervals are informative and correlated with the prediction errors, leading to slightly more accurate predictions with informative uncertainty. our uncertainty-aware system can take into account multiple references and becomes more confident (and accurate) when more references are available; it can so perform quality estimation without any human reference by relying on pseudo-references from other mt systems (prism). we show that uncertainty-aware mt evaluation is a promising path. as a future direction, we aspire to further explore uncertainty predicting methods that tackle the different kinds of aleatoric and epistemic uncertainty described in §3.2 and are better tailored to the specifics of this task.