Useful Conﬁdence Measures: Beyond the Max Score
Abstract
An important component in deploying machine learning (ML) in safety-critic applications is having a reliable measure of conﬁdence in the ML model’s predictions.
For a classiﬁer f producing a probability vector f ( x ) over the candidate classes, the conﬁdence is typically taken to be max i f ( x ) i .
This approach is potentially limited, as it disregards the rest of the probability vector.
In this work, we derive several conﬁdence measures that depend on information beyond the maximum score, such as margin-based and entropy-based measures, and empirically evaluate their usefulness, focusing on NLP tasks with distribution shifts and Transformerbased models.
We show that when models are evaluated on the out-of-distribution data “out of the box”, using only the maximum score to inform the conﬁdence measure is highly suboptimal.
In the post-processing regime (where the scores of f can be improved using additional in-distribution held-out data), this remains true, albeit less signiﬁcant.
Overall, our results suggest that entropy-based conﬁdence is a surprisingly useful measure.
Introduction
As machine learning (ML) is increasingly deployed in high-stakes decision-making applications, it becomes increasingly important that practitioners have access to reliable measure of how conﬁdent the ML model is in its various predictions.
This becomes especially crucial in settings where the predictions are made in conditions signiﬁcantly different than the ones present during development.
In these cases accuracy may unavoidably degrade but a useful conﬁdence measure can at least ensure practitioners “know” when the ML model “doesn’t know”.
In this work we assume classiﬁers f output a probability vector f
( x ) over the candidate classes Y and treat conﬁdence as a scalar quantity c ( f ( x )) ∈ [0 , 1] that represents how conﬁdent f is in its prediction, with scores near 1 representing highly conﬁdent predictions.
Intuitively, a good conﬁdence measure should give rise to scores that correlate well with the accuracy of f .
In Section 3 we show that this objective can be decomposed into two familiar terms from the literature on forecasting (Murphy, 1973; Dawid, 1982): a calibration error term (encouraging that whenever we output a conﬁdence value of e.g. 0.7, then on average, 70% of the time the model makes a correct prediction) and a sharpness term (encouraging the conﬁdence values to also be varied).
Given a classiﬁer f , what should we choose as our conﬁdence measure c ?
One natural choice is to use the maximum class probability, c ( f ( x )) =
max i f ( x ) i .
When f is itself calibrated, this will give rise to a calibrated conﬁdence measure.
However, we don’t necessarily expect models to be well-calibrated “out of the box”, especially not in the presence of distribution shifts.
While it is common to post-process predictions to improve their calibration, this approach is not always feasible as it requires additional data, is observed to have limited success in settings of distributions shifts (Desai and Durrett, 2020; Dan and Roth, 2021), and may come at the cost of sharpness (Kumar et al., 2018).
The above discussion suggests that in the presence of model miscalibration, using the tail of the predictions to inform the conﬁdence score could be beneﬁcial.
Indeed, the literature on active learning (AL) has long since considered uncertainty scores that employ the rest of the probability vector (Settles, 2009).
In the AL context, these are used to greedily select examples from a large pool of unlabeled examples for which labels will be requested.
E.g., it is common to use the gap between the ﬁrst and second largest entries of f ( x ) , and in some cases even the entropy of f ( x ) , to inform this selection. Our contributions.
In this work, we consider such uncertainty scores in the context of conﬁdence measures, and perform a systematic evaluation of these measures in the presence of distribution shifts.
We focus on large pre-trained Transformer-based language models like BERT (Devlin et al., 2018) for multi-class NLP tasks, which have observed to be well-calibrated on in-distribution data (Desai and Durrett, 2020).
We use the Amazon reviews dataset from the WILDS benchmark (Koh et al., 2021), in which the out-of-distribution (OOD) test set consists of a set of reviewers that is disjoint from the training set and in-distribution (ID) validation set.
We consider models trained on this task with different objectives (regular risk minimization, but also approaches that are designed to handle distribution shifts), and evaluate the different conﬁdence measures on the OOD test data.
Our key ﬁndings are:
When the conﬁdence measures are evaluated “out of the box” (with no further tuning based on a validation set), using max i f ( x ) i is highly sub-optimal.
Margin-based conﬁdence measures perform better for most of the models considered (and by a signiﬁcant gap), and the entropy-based conﬁdence measure is consistently better.
We derive a variant of temperature scaling (TS), a popular post-processing technique for improving calibration, and show that it can be used to consistently improve the calibration for all the conﬁdence measures we consider.
In the post-processing regime (namely, after applying TS), the entropy-based conﬁdence measure Pareto dominates the max-based measure for most of the models (and is otherwise incomparable - has a marginally larger calibration error but is sharper). Additional related work.
(Desai and Durrett, 2020) evaluate the calibration of pre-trained Transformer models in both ID and OOD settings.
Their results demonstrate that Transformer-based models tend to well-calibrated ID but that the calibration error can decrease signiﬁcantly OOD.
(Dan and Roth, 2021) empirically evaluate the relationship between scale and calibration, showing that
OOD, smaller Transformer models tend to have worse calibration than larger models, even after applying TS.
These works, together with earlier works (Guo et al., 2017), all conﬂate a model’s conﬁdence with the probability of the predicted label. One recent exception is (Taha et al., 2022), which consider conﬁdence measures based on the margin and kurtosis of the logits. Their work is signiﬁcantly different from ours as they do not directly evaluate calibration of these proposed measures and also do not consider distribution shifts.  Preliminaries
Setup.
We consider a multi-class classiﬁcation problem with feature space X and label space Y , where |Y| = k ≥ 2 and ∆( Y ) denotes the simplex over R k .
Let P denote a joint distribution over X × Y and X, Y random variables w.r.t P .
A classiﬁer is a mapping f : X → ∆( Y ) and its predicted label is arg max i f ( x ) i .
We use err to denote the 0-1 error of f : err ( x, y )
= 1 [arg max i f ( x ) i (cid:54) =
y ] .
Calibration.
A binary classiﬁer f : X → [0 , 1] is said to be calibrated if ∀ v ∈ [0 , 1] , Pr[ Y = 1 | f ( X ) = v ] = v .
For classiﬁcation problems with multiple classes ( k > 2 ), there are different ways to deﬁne calibration (Widmann et al., 2019; Zhao et al., 2021).
Arguably the simplest and most popular approach is to restrict the attention only to the most likely prediction (Guo et al., 2017).
According to this deﬁnition, a classiﬁer f : X → ∆( Y ) is calibrated if ∀ v ∈ [0 , 1] ,
Pr[ Y = arg max i f
( X ) i | max i f ( X ) i = v ] = v .
Post-hoc calibration.
Since many ML models are not typically calibrated “out of the box”, it is a common practice to post-process the model outputs in a way that improves their calibration.
Methods include Histogram Binning (Zadrozny and Elkan, 2001), Isotonic Regression (Zadrozny and Elkan, 2002) and Platt scaling (Platt et al., 1999).
In the context of modern ML models, the method of
Temperature Scaling (TS) has demonstrated to be both simple and effective (Guo et al., 2017).
For classiﬁers of the form f ( x ) = σ ( z ) (where z is the logit vector and σ is the softmax operator), TS rescales the logit vector z by a factor of T before applying softmax σ .
The hyperparameter T is called the temperature as it has the effect of “softening” the softmax (i.e. increasing the entropy of the output probability vector) when T > 1 .
In particular, T = 1 recovers the original model output and as T → ∞ the model output approaches 1 /K .
T is optimized by minimizing the Negative Log
Liklihood on a labeled validation set.
Conﬁdence measures
Notation.
The entropy of a vector v ∈ ∆( Y ) is H ( v ) = − (cid:80) ki
=1 v i · log( v i ) .
We use ˜ H ( v ) =log k · H ( v ) to normalize the entropy to be in [0 , 1] .
We also use ˜ v to denote the sorted version of v (in descending order),
so that ˜ v
i is the i -th largest value in v .
Conﬁdence measures.
A conﬁdence measure is a mapping c : ∆( Y ) → [0 , 1] .
The conﬁdence of a classiﬁer f on input x ∈ X according to c is then c ( f ( x )) , where values near 1 represent high-conﬁdence predictions and values near 0 represent low-conﬁdence predictions.
In this work, we consider the following measures: max : v (cid:55)→ ˜ v 1 ; margin12 : v (cid:55)→ ˜ v 1 − ˜ v 2 ; margin123 : v (cid:55)→ ˜ v 1 − (0 .
5˜ v 2 + 0 .
5˜ v 3 ) ; and entropy : v (cid:55)→ ˜ H ( v ) .
See Figure 2 in Appendix A for a comparison between these measures in a classiﬁcation problem with k = 3 classes.
3.1 Evaluating conﬁdence measures: calibration and sharpness
A good conﬁdence signal c should give rise to calibrated conﬁdence predictions; Namely, the binary classiﬁer c ◦ f ∈ [0 , 1] should be calibrated.
However, calibration by itself does not guarantee that the conﬁdence measure is useful.
For example, the conﬁdence measure that always output the marginal accuracy c ( x ) =
Pr[1 − err ( X, Y )] will be perfectly calibrated but useless for practical purposes.
Thus, a good conﬁdence measure should also give rise to a variety of conﬁdence values; this requirement is referred to as sharpness (Gneiting et al., 2007) 3 .
Letting T ( x ) =
E [ err ( X, Y ) | c ( X )
=
c ( x )] , we can decompose the (cid:96) 2 “loss” of a conﬁdence measure c as: [Equation] )) 2 ] = Var [ err
[Equation]
X
[Equation]
Y )] − [Equation] [Equation] The decomposition is a direct application of a similar decomposition for binary classiﬁers 4 .
Consider the three terms in the decomposition of (1).
The ﬁrst term, from the perspective of the choice of the conﬁdence measure, is irreducible.
The sharpness term measures the variation in the error across conﬁdence predictions.
The calibration term measures how closely the conﬁdence predictions track the error.
Overall, this suggests that a good conﬁdence measure should strike a balance between minimizing miscalibration and maximizing sharpness.
Estimation from ﬁnite samples.
In principle, estimating T ( x ) accurately for every value c ( x ) ∈ [0 , 1] requires an inﬁnite amount of data.
To estimate both calibration and sharpness from ﬁnite data, employ we use discretized versions of both notions.
Speciﬁcally, let B be a partitioning (“binning”) of the interval [0 , 1] , where B : [0 , 1] → B maps any v ∈ [0 , 1] to the bin B ( v ) that contains it.
Then, we redeﬁne T ( x ) as follows: T B ( x ) =
E [ err ( X, Y ) | B ( c ( X ))
=
B ( c ( x ))] .
Binning strategy.
Fix a granularity parameter n representing the number of target bins.
We distinguish between a ﬁxed binning strategy – in which B is formed by partitioning [0 , 1] interval into n equally-spaced bins; and an adaptive binning strategy – in which B is formed in a way that depend on the classiﬁer f .
Speciﬁcally, such that each bin has equal mass under c ◦ f .
In terms of measuring the calibration error, this is the different between the Expected Calibration Error (ECE) (Naeini et al., 2015) and the Adaptive Calibration Error (ACE) (Nixon et al., 2019).
Since our focus is on comparing different models in terms of their calibration error 5 , we prefer to use adaptive binning.
3.2 Post-processing to improve the calibration of arbitrary conﬁdence measures
In principle, a post-processing approach such as temperature scaling (TS) is designed to improve the calibration error of max .
To generalize this idea to general conﬁdence measures, instead of tuning the temperature value T to minimize the NLL, we simply perform a line search over the relevant region (e.g. [0 , 2] ) to choose the value T that minimizes the calibration error of the desired conﬁdence signal over a validation set.
From this point, when we refer to TS, we mean this approach.
Evaluation
To empirically study optimal conﬁdence signals under realistic and naturally occurring distribution shifts, we use the Wilds benchmark (Koh et al., 2021).
We continue to our evaluation procedure.
Data.
We use the Amazon Wilds dataset, which is a multi-class sentiment classiﬁcation task derived from the Amazon Reviews dataset (Ni et al., 2019).
The input x is the test of a review, the label y is a corresponding star rating (from 1 to 5), and the domain d is the identiﬁer of the reviewer who wrote the review.
The validation and test splits are comprised of both an in-distribution (ID) set and an out-of-distribution (OOD) set.
The ID sets consist of reviews that are not in the training set, but are written by reviewers that are in the training set.
The OOD sets consists of reviews written by a collection of reviewers that are disjoint from the training set and ID set reviewers.
Models.
We evaluate the calibration of DistilBERT-base-uncased models (Sanh et al., 2019) that were ﬁnetuned on the Amazon Wilds training set 6 .
All models were ﬁne-tuned using a similar setup (grid search over learning rate and objective-speciﬁc parameters, with the rest of the hyperparameters set to standard/default values; see Appendix E in (Koh et al., 2021) for a detailed description).
The models are standard ERM, IRM (Arjovsky et al., 2019) and GroupDRO (Sagawa et al., 2019).
Evaluation and results.
For each model, we measure the performance of each conﬁdence measure ( max , margin12 , margin123 and entropy ) via its sharpness and calibration errors.
Results are means over three independent training runs.
We distinguish between “out of the box” performance and performance with temperature scaling, with the temperature value T (per conﬁdence measure) optimized on the ID validation set.
We visualize the results in Figure 1; see Appendix A for numerical results.
We see that in the OOB regime, max is never optimal ( entropy is always preferrable, and margin123 is better for ERM and IRM type models).
In the TS regime, entropy Pareto-dominates max for both ERM and IRM model (for groupDRO, they are incomparable).
Discussion
In this work we deﬁne and empirically evaluate several conﬁdence measures, and show that measures beyond the max score achieve favourable results.
These observations join other recent work (Rothblum and Yona, 2022) in demonstrating that that in the presence of model miscalibration, additional attention should be given to otherwise trivial choices, in this case the conﬁdence measure.
Our empirical ﬁndings raise several natural directions for future exploration.
First, the striking efﬁcacy of using the entropy as a conﬁdence measure seems surprising, and it is interesting to explore 3.00 sharpness    c a li b r a t i o n e rr o r ERM 3.075 3.100 3.125 3.150 3.175 3.200 3.225 3.250 3.275 sharpness4 610 12 c a li b r a t i o n e rr o r IRM 2.
3.2 sharpness4812 c a li b r a t i o n e rr o r GroupDRO max margin margin123 entropy OOB TS whether this ﬁnding generalizes to additional settings and whether there are cases in which it can be justiﬁed theoretically. It is also natural to consider arbitrary conﬁdence measures, beyond those we chose to evaluate here.
In principle, the problem of ﬁnding the optimal conﬁdence measure could itself be cast as an ML problem and solved using ML tools.
Our preliminary ﬁndings suggest that this type of “conﬁdence meta-learning” is a promising direction for future work.
A Additional Figures
In Figure 2 we show heatmaps illustrating the behaviour of each conﬁdence measure in a classiﬁcation problem with three classes.
The high conﬁdent predictions are marked in green (e.g. the simplex corners) and low conﬁdent predictions are in red (e.g., for all signals, conﬁdence is minimized at the simplex center where the distribution over classes is uniform).
In Tables 1 - 4 we report results of the calibration errors for the various conﬁdence measures, with and without applying temperature scaling on the in-distribution validation data, and when calibration is measured both with adaptive binning (ACE) and with ﬁxed binning (ECE).