Disentangling Uncertainty in Machine Translation Evaluation (2022)

Abstract: Trainable evaluation metrics for machine translation (MT) exhibit strong correlation with human judgements, but they are often hard to interpret and might produce unreliable scores under noisy or out-of-domain data. Recent work has attempted to mitigate this with simple uncertainty quantification techniques (Monte Carlo dropout and deep ensembles), however these techniques (as we show) are limited in several ways – for example, they are unable to distinguish between different kinds of uncertainty, and they are time and memory consuming. In this paper, we propose more powerful and efficient uncertainty predictors for MT evaluation, and we assess their ability to target different sources of aleatoric and epistemic uncertainty. To this end, we develop and compare training objectives for the COMET metric to enhance it with an uncertainty prediction output, including heteroscedastic regression, divergence minimization, and direct uncertainty prediction.Our experiments show improved results on uncertainty prediction for the WMT metrics task datasets, with a substantial reduction in computational costs. Moreover, they demonstrate the ability of these predictors to address specific uncertainty causes in MT evaluation, such as low quality references and out-of-domain data.

## introduction trainable neural-based metrics, such as comet or bleurt @xcite , hold great promise for mt evaluation @xcite . for system comparison, they surpass or complement traditional lexical metrics such as bleu @xcite , and at a segment level, they show higher correlations with human judgments, with and without access to references @xcite @xcite ). 1 our code and data is available at: https://github.com/ deep-spin/uncertainties_mt_eval 0.00 0.25 0.50 0.75 1.00 1.25 ## improving uncertainty-aware mt evaluation a limitation of ua-comet is its reliance on model variance techniques that often produce poor estimates of uncertainty and conflate aleatoric and epistemic uncertainty, making it hard to accurately represent uncertainty related to out-of-distribution samples @xcite . we therefore examine alternate methods to learn aleatoric and epistemic uncertainty directly from the available data. we assume that for each of the training scenarios and learning objectives described in the following sections, we can learn to predict the uncertainty of quality estimates q either as the noise variance σ in the case of aleatoric uncertainty, or as the generalization error ϵ in the case of epistemic (and total) uncertainty. ## experiments the main focus of our experiments is to investigate how the uncertainty estimators we explore in this paper compare to each other and against proposed variance-based methods. our comparisons address the accuracy of uncertainty predictions on mt evaluation datasets ( §5.2) as well as more specific concerns such as the performance on out-of-domain data ( §5.2), the ability to detect low quality references ( §5.3), and the computational costs ( §5.4). ## conclusions we assessed the potential of different uncertainty predictors to capture different sources of uncertainty in mt evaluation. we demonstrated that methods modeling heteroscedasticity can detect noisy references as a source of aleatoric uncertainty, and that the direct epistemic prediction method reflects well the increased epistemic uncertainty under a domain shift. besides providing more informative uncertainty estimates than mc dropout and deep ensemble methods, our proposed predictors are also computationally cheaper. overall, our work provides insight about which uncertainty predictors to choose for mt evaluation depending on the uncertainty source(s) to be addressed. the proposed uncertainty predictors that are able to target specific types of uncertainty are the first step towards mitigating the sources of such uncertainty, i.e. removing noisy instances from training to reduce aleatoric uncertainty, or identifying informative instances that would allow adapting to a new domain to reduce epistemic uncertainty. in future work, we are planning to further explore their properties and potential in improving mt and mt evaluation performance.