E MPIRICAL CONFIDENCE ESTIMATES FOR CLASSIFICATION BY DEEP NEURAL NETWORKS
Anonymous authors
Paper under double-blind review
A BSTRACT
How well can we estimate the probability that the classiﬁcation predicted by a deep neural network is correct (or in the Top 5)?
It is well-known that the softmax values of the network are not estimates of the probabilities of class labels.
However, there is a misconception that these values are not informative.
We deﬁne the notion of implied loss and prove that if an uncertainty measure is an implied loss, then low uncertainty means high probability of correct (or top k ) classiﬁcation on the test set.
We demonstrate empirically that these values can be used to measure the conﬁdence that the classiﬁcation is correct.
Our method is simple to use on existing networks: we proposed conﬁdence measures for Top k which can be evaluated by binning values on the test set.
I NTRODUCTION
Despite lots of effort to build conﬁdence measures for classiﬁcation by deep neural networks, there is still a lot of confusion about the value and applicability of these measures.
In this article we present a simple method for estimating conﬁdence based on implied loss values which leads to results which are empirically more accurate than benchmarks on test sets.
We prove that high conﬁdence values imply a high probability of correct classiﬁcation on test sets.
Many have observed that used blindly, the maximum softmax probability of a network does a poor job of predicting uncertainty (Nguyen & O’Connor, 2015;
Provost et al., 1998; Nguyen et al., 2015;
Yu et al., 2011; Lakshminarayanan et al., 2017).
However, Zaragoza & d’Alché Buc (1998) showed in the 1990s that on on shallow networks the maximum softmax probability and the (negative) entropy of the probabilities strongly correlate with model conﬁdence on in-distribution images.
More recently, in the deep setting, Hendrycks & Gimpel (2017) showed empirically that the maximum softmax probability can be used to predict network conﬁdence.
We demonstrate that both the maximum softmax and the model entropy are uncertainly measures.
We extend the uncertainty metric to Top k predictions.
We show that, in conjunction with binning, simple uncertainty statistics outperform common approaches like MC-dropout as a measure of conﬁdence, at a fraction of the computational cost.
Using this simple idea, we make the following contributions.
We estimate the probability that the classiﬁcation of the model on a test set is correct which works for existing models (no need to retrain), using a simple tabular form (see Table 2 for Imagenet).
We give a simple deﬁnition of uncertainty, which applies to previously proposed methods, and leads to a proof that low uncertainty (high conﬁdence) implies high probability of correct classiﬁcation.
It applies to both Top 1 and Top k uncertainty.
We can discover mislabelled data, see Figure 1(c) and we can detect off manifold data and adversarial examples. We advocate evaluating model uncertainty via expected Bayes factors (Kass & Raftery, 1995), which provide a rigorous probabilistic approach to evaluating uncertainty, and are widely used for hypothesis testing in other scientiﬁc ﬁelds, see for example (Good, 1979) and (Jeffreys, 2003).
Compared to other methods (such as AUROC or Brier scores) Bayes factors better distinguish improvements to
P RIOR WORK
As neural networks are adopted into safety critical systems, the need for neural network uncertainty estimates has become abundantly clear.
Indeed, any accident adverse system must by design incorporate notions of uncertainty (Amodei et al., 2016).
Real-world examples abound: uncertainty measures are needed in autonomous vehicles (Feng et al., 2018), robotics (Richter & Roy, 2017), medical imaging (Ching et al., 2018; DeVries & Taylor, 2018b) and medical decision making (Begoli et al., 2019), and semantic understanding (Kendall et al., 2017).
Much effort has been dedicated to addressing this deﬁciency.
Many works have placed neural networks within a Bayesian probabilistic framework.
Initial work placed Bayesian priors on model weights (MacKay, 1992a; Neal, 1996), leading to Bayesian neural networks, however this has proven difﬁcult to implement in practice.
Many techniques have been developed to overcome this difﬁculty (MacKay, 1992b; Neal, 1996; Graves, 2011;
Hasenclever et al., 2017; Li et al., 2015; Balan et al., 2015; Welling & Teh, 2011; Springenberg et al., 2016).
One promising approach in the deep learning setting is to perform approximate posterior inference (Louizos & Welling, 2016; Hernández-Lobato & Adams, 2015; Blundell et al., 2015; Sun et al., 2017).
Due to its simplicity, dropout is widely used as a surrogate for uncertainty.
Dropout (Srivastava et al., 2014) was interpreted in a Bayesian setting by Gal & Ghahramani (2016) and Kingma et al.
(2015), however, there are problems with this interpretation, see (Hron et al., 2018) for a recent discussion.
Dropout involves evaluating an ensemble of models at test time, which can be both memory and computationally intensive for very large networks.
Non-Bayesian model ensembles have also been developed (Dietterich, 2000), for a recent survey see (Li et al., 2018).
Lakshminarayanan et al.
(2017) train an ensemble of adversarially robust models and empirically showed an improvement in uncertainty estimates over dropout based methods.
Geifman et al.
(2018) proposed using an early stopping criteria to collate an ensemble of models.
Kristiadi &
Fischer (2019) use mixture modeling to chose ensemble weights.
Several deep learning speciﬁc approaches have been proposed in recent years, especially in the context of detecting out-of-distribution samples.
Oliveira et al.
(2016) suggest detecting outliers via an anomaly detector.
Lee et al.
(2018) generator out-of-distribution images through a GAN; the classiﬁer is trained to assign the equal weight probability vector to these images.
Hendrycks et al.
(2018) train networks on two distributions: the in-distribution samples, and out-of-distribution samples.
Liu et al.
(2018) develop PAC-style guarantees on detection of out-of-distribution samples.
Several recent works (Jiang et al., 2018; Papernot & McDaniel, 2018; Mandelbaum & Weinshall, 2017) have suggested using nearest neighbour distances, in feature space, for outlier detection and conﬁdence measures.
DeVries & Taylor (2018a) suggest training an additional network to predict uncertainty;
Malinin & Gales (2018) speciﬁcally model prediction probabilities with a Dirichlet distribution, which implicitly describes model uncertainty.
Platt (2000) proposed scaling SVM predictions to better match the validation set; this has been generalized to neural networks and multiclass classiﬁcation (Niculescu-Mizil & Caruana, 2005; Guo et al., 2017).
Other scaling approaches, such as changing the softmax temperature, have shown promise (Guo et al., 2017; Liang et al., 2018).
Another popular approach to calibration is based on binning model probabilities, developed by Zadrozny & Elkan (2001).
Each bin is assigned a probability of being correct, which is obtained by minimizing the Brier score of the bins (Brier, 1950).
Bins edges may be optimized as well (Zadrozny & Elkan, 2002); and can be extended to the Bayesian setting by assigning a prior on binning schemes (Naeini et al., 2015).
C ONFIDENCE MEASURES AND UNCERTAINTY ESTIMATES
Suppose a model f ( x ) , generalizes well, so that it has a high probability, p , of a correct prediction on an image x sampled from the same underlying distribution.
Write I k ( f )
= { indices of the k largest components of f } for the top k indices.
The classiﬁcation of the vector f is given by the largest component, C ( f ) =
I 1 ( f ) .
Deﬁne the random variables
X k = 1 { y ( x ) ∈ I k
( f ( x )) } to be the Bernoulli random variables with expected value
p k = E
[ X k ] , the probability that the correct label is in the Top k .
Our goal is to estimate p k .
We do this by deﬁning random variables, U k , which we call uncertainties , whose statistics allow us to better estimate p k .
The histogram of the uncertainty variables will result in an estimate of the conditional probability that the classiﬁcation is correct, given the uncertainty value, Prob ( X k ( [Equation] =
[Equation]
U k
[Equation]
) =
[Equation]
Deﬁnition 3.1. Given (cid:15) > 0 , and the uncertainty measure U ( x ) , deﬁne the set [Equation] = {
[Equation]
≤ [Equation] and y (cid:54)∈ I k ( x ) } [Equation]
[Equation]
The uncertainty measure U k ( x ) is an implied loss if the event S (cid:15)k has high expected loss.
Example 3.2 . For the Kullback-Leibler loss, the (negative) entropy of the probabilities is an uncertainty measure. Write f sort for the indices of f sorted in decreasing order. Deﬁne
[Equation]
Example 3.3 . For general losses, L , the top 1 uncertainty can be given by U 1 ( x ) = {L ( f ( x ) , y ) | y =
C ( f ( x )) } , which is the loss, given that the classiﬁcation was correct . Also U k ( x ) = L ( f, y w ) is an implied loss, for top k , where y w is the ( k + 1) -th ranked label.
3. I LLUSTRATION OF UNCERTAINTY RANDOM VARIABLES
The histogram of U 1 is plotted on the test set in Figure 1(a). Note that for small values of U 1 , the images have a very high probability of being correct. In fact, we can use U 1 to detect incorrectly classiﬁed images: we visualized the images which smallest value of U 1 (i.e. highest conﬁdence), which correspond to the few isolated points in the upper left of the ﬁgure.
It turned out that all of these were either incorrectly labelled, or were ambiguous images, see illustrations in Figure 1(c).
For example, in the second image, the animal is a wallaby, not a wombat.
In the fourth image, a paintbrush is a kind of plant, but there is also a pot in the image.
In the second part of Figure 1(b) we illustrate more quantitatively the Top 1 (green) and Top 5 (green or blue) probabilities conditioned on the 100 histograms bins of − log( p max ) on test set for ResNet152 on ImageNet.
The Topprobability conditioned on the lower bins is very close to 100%.
The Top 5 probability is no better than 50% on the last few bins.
The intermediate bins are less informative.
3. T OP 1 UNCERTAINTY ESTIMATES
The next theorem shows that if the uncertainty is small, then the probability of correct classiﬁcation must be high.
Theorem 3.4 (Conﬁdence estimate) .
Deﬁne U 1 ( x ) by (2) and deﬁne S (cid:15) by (1) , and let L KL be the
Kullback-Leibler loss.
Then
[Equation]
Proof.
Claim: Let (cid:15) > 0 be small.
By assumption, − log f sort≤ (cid:15) .
Thus f sort≥ exp( − (cid:15) ) .
Let e k be the correct label.
Then f k ≤ f sort, so f k ≤ 1 − exp( − (cid:15) ) and − log( f k ) ≥ − log(1 − exp( − (cid:15) ))
≥ log(1 /(cid:15) ) . Thus for x ∈ S (cid:15) , L KL ( f ( x ) , y ( x )) ≥ log(1 /(cid:15) ) .
Apply Markov’s inequality (13) to the random variable L ( x ) =
L ( f ( x ) , y ( x )) to obtain the result.
Remark 3.5 (Neural Networks are always overconﬁdent) .
Note that the uncertainly is always less than the loss,
[Equation]
with equality when C ( f ( x ))
=
y ( x ) .
[Figure]
log( f (1) )
[Figure]
101010log( f (1) ) 0.
P ( C o rr e c t | X ) [Figure] 3. T OP k UNCERTAINTY ESTIMATE
In the next result we show that if the top k uncertainty is small, then the probability that the correct labels is in the top k must be high.
The result can also be proven in the case of general losses, and uncertainly measures satisfying (1).
Consider the event S (cid:15)k (1) for a given k ≥ 1 .
If the correct label is not in the top k , then the probability of the correct label, f c , must satisfy
[Equation]
with
[Equation]
Thus
[Equation]
Then, by an argument similar to the one for Top 1 error, we see that
[Equation]
[Equation]
E MPIRICAL R ESULTS
The previous section proved that, under fairly general conditions, we can deﬁne uncertainty measure which ensure that the top k classiﬁcation is correct with high probability.
The theory applies to uncertainties used in the literature, such as the negative entropy of the probabilities, and negative log softmax.
In practice, once we have an uncertainty measure, the method is simple
Compute the statistics on the test set of the uncertainty estimates.
Divide the test set into bins, based on uncertainty values.
Estimate the conditional probabilities based on the bin populations.
4. T HE B AYES FACTOR
The Bayes factor is a way to measure the value of new information, in terms of how much the expected winnings of a fair bet increase, when the information is available. Unlike other measures of conﬁdence, which are additive, the Bayes factor is multiplicative .
On a model which is correct 95% of the time, there still a lot of value in knowing when the probability correct increases to 99.5%.
In this case, the Bayes factor is close to 10. On the other hand, going from 50% to 54.5% gives a Bayes factor close to 1.2. On the other hand, additive scoring methods give equal weight to both improvements.
As an example, we show in Table 5 that the Brier scores of eight different measures of conﬁdence all lie close together, between .033 and .076.
On the other hand, the expected Bayes factors range more widely, from 1.3 to 16.6.
Consider a Bernoulli random variable X
= B ( p X ) .
The odds for X are given by O ( p X ) =
p X 1 − p X .
Now consider a test, Y = B ( p Y ) , for which
[Equation]
Then the odds, given the test succeeds, are O ( p X,Y ) =
p X,Y 1 − p X,Y .
If the odds have increased, we deﬁne the Bayes Factor to be BF ( X | Y )
=
O ( p X,Y )
O
( p X ) .
On the other hand, if the odds have decreased, then the value of the information provided by Y is to bet against, so we deﬁne the Bayes factor to be
BF ( X | Y )
=
O ( p X ) O
( p X,Y ) .
Combining these possibilities, deﬁne the Bayes factor by
[Equation]
4. E XPECTED B AYES FACTOR
We can also deﬁne a metric for measuring the quality of an uncertainty random variable, such as the value of the loss (or another random variable) for predicting the probability of correct classiﬁcation.
In practice, we will deﬁne the tests based on bin values for some uncertainty variable.
If we have more data, we can use more bins.
Deﬁnition 4.1 (Histogram random variables) .
Given a random variable U ( x ) ∈ [ a, b ] and a partition of [ a, b ] into bins
[Equation]
Deﬁne the histogram (or bin) random variables B i , corresponding to each interval
[Equation]
so that
[Equation]
Each Bayes factor measures the value of information that x lies in each quantile.
The value of the test itself is deﬁned to be the expected value of the Bayes factors.
Deﬁnition 4.2 (Histogram Bayes Factors) .
Given X , U and the histogram random variables B i , deﬁne the conditional probabilities
[Equation]
The predictive value for X of the random variable U with respect to the histogram, is given by
[Equation]
When the number of bins is large, we deﬁned the bins to be quantiles, so that they have an equal number of examples in each bin.
In addition, in order to make the histogram Bayes factors ﬁnite, we require that each bin have at least one correct and one incorrect example.
See Appendix C for a detailed example of the expected Bayes factor.
Next we present an example based on actual conﬁdence measures for networks.
[Table]
Example 4.3 . This example follows closely the conﬁdence bins for top 5 on ImageNet-1K, using the Model Entropy, as in the ﬁrst row of Table 2. Consider a model with p X = . In this the odds are 94 to 6, so O ( p X )
= 15 .
Deﬁne three bins for Model Entropy with bin edges 0 .
Then with probability .55, data is in the ﬁrst bin, in which case the probability correct is .99.
So the Bayes factor in this bin is ( . 99 /. 94)( . 06 /. 01)
= 6 . 3 .
Thus knowing the Model Entropy is less than .
31 tell you that you are 6.3 times more likely to be correct than on average.
The second bin consists of data with with Model Entropy between . 31 and .
14 , which occurs with probability .31, in this case, the
Bayes Factor (95 / 94)(6 / 5) = 1 . 2 is nearly one, so there is little additional value to knowing data is in this bin. Finally, when the Model Entropy is greater than .14, which occurs with probability .14, the probability correct is only .8. In this case, the relative probability to correct is worse, to the Bayes Factor is given by (94 / 80) / (6 / 20) = 3 . 9 .
The expected Bayes factor is the weighted average of the Bayes factor of each bin, weighted by the probability of the bins
[Equation] So the expected value of the Model Entropy, for the chosen bins, is 4.4.
By ﬁne graining the bins we can capture relatively small and relatively large values of the Model Entropy which can have Bayes factors on the order of 20, see Figure 5.
Thus the expected Bayes factor with 100 bins is 8.18, as shown in Table 1.
4. C ONFIDENCE BINS AND B AYES F ACTOR
In this section we present conﬁdence bins for ImageNet-1K.
These bins are concise summaries of the information presented in the larger bins.
Table 2 presents short bins for ImageNet.
Using these bins, we can simply read of from the Uncertainty values, the probability that the model is correct.
For example, on the model, P ( top 5 ) = 0 .
9406 , however, using entropy, 55% of the images had entropy low enough to be conﬁdently classiﬁed with probability .99.
Using U 5 , 66% of images could be binned to have probability .99.
Bins for CIFAR-10 and CIFAR-100 are given in Tables 7 and 6, respectively.
In Table 1 we show the expected Bayes factor for various conﬁdence measures, on CIFAR-10, CIFAR100, and ImageNet-1K.
In addition to the conﬁdence measures already discussed, we considered
Bayesian dropout, and the norm of the gradient of the model.
Larger expected Bayes factors means the information is more valuable.
In Figure 5 we plot the regularized Bayes factor for our two main measure of conﬁdence, U 1 and Ualong with the loss and the model Entropy.
The entropy and U 1 , U 5 have very large Bayes factor in the ﬁrst 10 and last 3 bins, meaning that for these bins, the prediction is 10X (or more) likely to be correct (for the ﬁrst 10) or wrong (for the last 3 bins) than average. [Table] Model Entropy 0. [Figure]
Model Entropy 0.
[Figure]  E XTENSIONS In this section we discuss some extensions of the conﬁdence results.
We show that we can detect mislabeled images in the test set.
We also show that we can obtain some conﬁdence results for off manifold images, as well as adversarial images.
5. D ETECTION OF MISLABELED IMAGES
We are able to detect test images which are mis-labeled: images which the network correctly classiﬁed, but who’s label is incorrect, or for which multiple labels could apply.
These are images with high loss but low model entropy.
For example in Figure 1(c) we show six images from the ImageNet-1k test set who’s predictions where not in the top5, but had low model entropy.
All six of these images either have an incorrect dataset label, or could be described by multiple labels.
For example, in the second image, the animal is a wallaby, not a wombat.
In the fourth image, a paintbrush is a kind of plant, but there is also a pot in the image.
5. C ONFIDENCE ON OUT - OF - DISTRIBUTION AND ADVERSARIAL IMAGES
Next we studied whether we could detect out-of-distribution images generated by COCO.
In Figurewe show how the histogram of the model entropy is shifted to the right compared to the on-distribution images.
Table 3 give the results of our test: choosing a conﬁdence measure which rejects 10% of the on-distribution images, our conﬁdence measures rejected as much as 38% of COCO images (for
Entropy) with similar values for U 1 , U 5 .
On the other hand Dropout was completely ineffective.
[Table]
[Table]
[-[Figure] f r e q u e n c y [Figure] COCO images 
-]C ONCLUSIONS
With the goal of using measures such as model entropy as a surrogate for the (unknown) model loss
[Table]
we deﬁned conﬁdence measures as random variables which are large when the loss is large. Using this deﬁnition we proved that conﬁdence variables can be used to estimate the probability that a model low expected loss makes a correct prediction. In practical terms, this amounts to deﬁning a conﬁdence measure (such as model entropy or log pmax) and binning the values.
We presented the expected Bayes factor as an effective measure of conﬁdence. Since models are already very accurate, it is important to measure the relative conﬁdence.
The Bayes factor is a multiplicative factor to the probability (or odds) that a model is correct. For example, showing the that model entropy on ImageNet with 100 bins is 8 means that knowledge of the model entropy (and the bayes factors for the bins) allows us to predict the probability that the model is correct 8 times more effectively. The Bayes factors was used to compare existing conﬁdence measures on different tasks. The main task was estimating the probability of a correct prediction on the images from the data set.
Additional tasks included: detection of off manifold data, detection of adversarial examples, and detection of mislabelled images.
The latter were found by searching for highly conﬁdent predictions which were labelled incorrect.
A A DVERSARIAL ATTACK DETECTION
In this section we empirically demonstrate that image vulnerability may also be used to detect adversarial examples.
We hypothesize that unless otherwise penalized, gradient based attacks will tend to move images to regions where the gradient of the loss is large.
Based on this heuristic, we propose the norm of the loss gradient norm as criterion for detecting adversarial perturbations.
Because the loss is not available during inference, we propose using the norm of the model gradient as a rejection criteria: an image has been adversarially perturbed if
[Equation]
(12) for some threshold value, c .
The threshold is determined by setting the signiﬁcance level (the rate of false positives) to 5%.
For example on CIFAR-10 we obtained c = 2 . 45 for our model.
The results are reported in Table 4 and in Figure 3.
Only 6% of clean test images were rejected.
However, 100% of
Boundary attacks and Carlini-Wagner attacks were detected, as well as 96% of PGD attacked images.
This leads to the question, is it possible to successfully perturb all images in the test set, and avoid detection?
We built a targeted attack, designed to avoid detection.
We use a Carlini-Wagner style attack, modiﬁed with a penalty to avoid detection.
We augmented the attack loss function with a penalty for (cid:107)∇ (cid:96) ( x ) (cid:107) 2 ∗ , which penalizes attacks for being detectable.
We call this attack an evasive
Carlini-Wagner attack.
The evasive CW attack was successful at avoiding detection 78% of the time, but in order to do so, it increased the median adversarial distance signiﬁcantly, from 0.31 to 0.81, see
Table 4.
B M ARKOV ’ S INEQUALITY
Lemma B.1 (Markov’s Inequality) .
For a random variable Z with ﬁnite expectation, let S ⊂ { Z ≥ a } then
[Equation]
[Figure]
[Figure]
B a y e s r a t i o
Loss log( f (1) ) log( f (1:5) )
Model Entropy
[Table]
[Table]
[Table]
[Table]
C W ORKED EXAMPLE OF B AYES F ACTORS
Consider the situation where you have exchanged phone numbers with someone, and you wish to contact them.
The question is whether to send a text message or phone their number.
Approximately 95% of people prefer to message.
Let X be the probability that a person prefers to message.
The expected value and odds for X is given by
[Equation]
Now suppose we have additional information, which gives these statistics based on age.
Suppose we wish to predict X .
Knowing the age U has a value.
Let U ( x ) be the age, and consider three bins for
U given by the values 20 , 65 and let Y 1 , Y 2 , Y 3 be the corresponding histogram random variables.
[Equation]
Since older people are more likely to prefer to use a phone, the conditional probabilities and corresponding odds are given by
[Equation]
In particular, knowing if they are younger or older is more valuable than the middle range.
The Bayes factor (relative odds) expresses the value of knowing the age if someone is willing to bet with the odds O ( p X ) .
So this information allows an expected proﬁt on the bet given by the factor.
[Equation]
So the value of the information depends on the cases.
Finally, if we wish to ﬁnd the expected value of the information, we take an expectation with respect to the probabilities of the events.
[Equation]
i )]
= 53 × .
4 + 1 . 2 × . 5 + 2 . 1 × . 1 =(17) Some other information about the person may be much less useful in prediction their preference.
For example, suppose you know the region where they live and let Y 1 , Y 2 , Y 3 be the histogram random variables.
Suppose
[Equation]
Since E [ X ] = .
[Equation]
So with an expected value of 1.5, compared to age, with an expected value of 22, the location information is much less valuable.