Conformalizing Machine Translation Evaluation (2024)

Abstract: Several uncertainty estimation methods have been recently proposed for machine translation evaluation. While these methods can provide a useful indication of when not to trust model predictions, we show in this paper that the majority of them tend to underestimate model uncertainty, and as a result, they often produce misleading confidence intervals that do not cover the ground truth. We propose as an alternative the use of conformal prediction, a distribution-free method to obtain confidence intervals with a theoretically established guarantee on coverage. First, we demonstrate that split conformal prediction can “correct” the confidence intervals of previous methods to yield a desired coverage level, and we demonstrate these findings across multiple machine translation evaluation metrics and uncertainty quantification methods. Further, we highlight biases in estimated confidence intervals, reflected in imbalanced coverage for different attributes, such as the language and the quality of translations. We address this by applying conditional conformal prediction techniques to obtain calibration subsets for each data subgroup, leading to equalized coverage. Overall, we show that, provided access to a calibration set, conformal prediction can help identify the most suitable uncertainty quantification methods and adapt the predicted confidence intervals to ensure fairness with respect to different attributes.1

## introduction neural models for natural language processing (nlp) are able to tackle increasingly challenging tasks with impressive performance. however, their deployment in real-world applications does not come without risks. for example, systems that generate fluent text might mislead users with fabricated facts, particularly if they do not expose their confidence. high performance does not guarantee an accurate prediction for every instance-for example, the degradation tends to be more severe when instances are noisy or out of distribution. this makes uncertainty quantification methods more important than ever. while most work on uncertainty estimation for nlp has focused on classification tasks, uncertainty quantification for text regression has recently gained traction, with applications in machine translation (mt) evaluation, semantic sentence similarity, or sentiment analysis @xcite . this line of work builds upon a wide range of methods proposed for estimating uncertainty @xcite @xcite . however, current uncertainty quantification methods suffer from three important limitations: or under-estimated) results @xcite . hence, choosing a suitable method for a dataset can be complicated. to address the shortcomings above, we propose conformal prediction ( §2) as a means to obtain more trustworthy confidence intervals on textual regression tasks, using mt evaluation as the primary paradigm. we rely on the fact that given a scoring or uncertainty estimation function, conformal prediction can provide statistically rigorous uncertainty intervals @xcite @xcite . more importantly, the conformal prediction methodology provides theoretical guarantees about coverage over a test set, given a chosen coverage threshold. the predicted uncertainty intervals are thus valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions @xcite , and they also allow for an intuitive interpretation of the confidence interval width. we specifically show ( §3) that previously proposed uncertainty quantification methods can be used to design non-conformity scores for split conformal prediction @xcite . we confirm that, regardless of the initially obtained coverage, the application of conformal prediction can increase coverage to the desired-user defined-value (see figure 2 ). to this end, we compare four parametric uncertainty estimation methods (monte carlo dropout, deep ensembles, heteroscedastic regression, and direct uncertainty prediction) and one non-parametric method (quantile regression) with respect to coverage and distribution of uncertainty intervals. additionally, we introduce a translation-inspired measure for referenceless quality estimation (qe) that uses the distance between quality estimates of translated and back-translated text to estimate non-conformity. we show that the estimated quantiles over each non-conformity score are indicative not only of the coverage but also of the overall suitability of the non-conformity score and the performance of the underpinning uncertainty quantification method (e.g. it aligns well with error correlation computed over the test set). our experiments highlight the efficacy of quantile regression, a previously overlooked method for the mt evaluation task. moreover, we investigate the fairness of the obtained intervals ( §4) for a set of different attributes: (1) translation language pair; (2) translation difficulty, as reflected by source sentence length and syntactic complexity, (3) estimated quality level and (4) uncertainty level. we highlight unbalanced coverage for all cases and demonstrate how equalized conformal prediction @xcite @xcite can address such imbalances effectively. ## conformal mt evaluation we now apply the machinery of conformal prediction to the problem of mt evaluation, which is a regression task, aiming to predict a numeric quality score over an (automatically) translated sentence. the input is a triplet of source segment s, automatic translation t, and (optionally) human reference r, x := s, t, r , and the goal is to predict a scalar value ŷ(x) that corresponds to the estimated quality of the translation t. we can also consider a reference-less mt evaluation scenario where the input is simply x := s, t . 4 the ground truth is a quality score y manually produced by a human annotator, either in the form of a point on a quality scale called direct assessment (da; @xcite ) or in the form of accumulated penalties called multidimensional quality metrics (mqm; @xcite . we use da scores that are standardized for each annotator. an example instance is shown in figure 3 . subsequently, to apply conformal prediction we need to determine suitable non-conformity metrics, that can capture the divergence of a new test point x test with respect to the seen data. to that end, we primarily experiment with a range of uncertainty quantification heuristics to generate δ(x) (or δ -(x) and δ + (x) in the non-symmetric case). with the symmetric parametric uncertainty methods, described in §3.1.1, we obtain heuristics to compute δ which we use to obtain non-conformity scores via (2), leading to the confidence intervals c q (x) in (4), for each x triplet. alternatively, in §3.1.2 we describe a non-symmetric and non-parametric method which returns ŷ, δ -, and δ + , and which we will use to compute the non-conformity scores (5) and confidence intervals (6). finally, in 3.1.3 we describe a heuristic non-conformity score which is inspired by the mt symmetry between s and t. ## conditional coverage the coverage guarantees stated in theorem 1 refer to marginal coverage-the probabilities are not conditioned on the input points, they are averaged (marginalized) over the full test set. in several practical situations it is desirable to assess the con- in fact, evaluating the conditional coverage with respect to different data attributes may reveal biases of the uncertainty estimation methods towards specific data subgroups which are missed if we only consider marginal coverage. in the next experiments, we follow the feature stratified coverage described in angelopoulos and bates (2021); we use conformal prediction with mc dropout as our main paradigm. we demonstrate five examples of imbalanced coverage in figure 8 and we can see that, coverage varies significantly across groups, revealing biases towards specific attribute values. for example, the plots show that into-english translations are under-covered for most uncertainty quantifiers (@xmath0.9), i.e., we consistently underestimate the uncertainty over the predicted quality for these language pairs. more importantly, we can see that examples with low predicted quality are significantly undercovered, as coverage for quality scores where @xmath1.5 drops below 50%. for mcd-based uncertainty scores on the other hand, the drop in coverage seems to be related to the low uncertainty scores, indicating that due to the skewed distribution of uncertainty scores, the calculation of the q quantile is not well tuned to lower uncertainty values (i.e., higher non-conformity scores). instead, our two proxies for source difficulty reveal better balanced behaviour, with small deviations for very small sentences or high syntactic complexity. similar patterns for these dimensions are also ## conclusions in this work, we apply conformal prediction to the important problem of mt evaluation. we show that most existing uncertainty quantification methods significantly underestimate uncertainty, achieving low coverage, and that the application of conformal prediction can help rectify this and guarantee coverage tuned to a user-specified threshold. we further show that the estimated quantiles provide a way to choose the most suitable uncertainty quantification methods, aligning well with other metrics such as ups @xcite . we also use conformal prediction tools to assess the conditional coverage for five different attributes: language pairs, sentence length and syntactic complexity, predicted translation quality, and estimated uncertainty level. we highlight inconsistencies and imbalanced coverage for the different cases, and we show that equalized conformal prediction can correct the initially unfair confidence predictions to obtain more balanced coverage across attributes. overall, our work aims to highlight the potential weaknesses of using uncertainty estimation methods without a principled calibration procedure. to this end, we propose a methodology that can guarantee more meaningful confidence intervals. in future work, we aim to further investigate the application of conformal prediction across different data dimensions as well as different regression tasks in nlp.