C ALIBRATION OF N EURAL N ETWORKS USING S PLINES
Anonymous authors
Paper under double-blind review
A BSTRACT
Calibrating neural networks is of utmost importance when employing them in safety-critical applications where the downstream decision making depends on the predicted probabilities.
Measuring calibration error amounts to comparing two empirical distributions.
In this work, we introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test in which the main idea is to compare the respective cumulative probability distributions.
From this, by approximating the empirical cumulative distribution using a differentiable function via splines, we obtain a recalibration function , which maps the network outputs to actual (calibrated) class assignment probabilities.
The spline-ﬁtting is performed using a held-out calibration set and the obtained recalibration function is evaluated on an unseen test set.
We tested our method against existing calibration approaches on various image classiﬁcation datasets and our spline-based recalibration approach consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
I NTRODUCTION
Despite the success of modern neural networks they are shown to be poorly calibrated (Guo et al.
(2017)), which has led to a growing interest in the calibration of neural networks over the past few years (Kull et al.
; Kumar et al.
; Müller et al.
Considering classiﬁcation problems, a classiﬁer is said to be calibrated if the probability values it associates with the class labels match the true probabilities of correct class assignments.
For instance, if an image classiﬁer outputs 0.2 probability for the “horse” label for 100 test images, then out of those 100 images approximately 20 images should be classiﬁed as horse.
It is important to ensure calibration when using classiﬁers for safety-critical applications such as medical image analysis and autonomous driving where the downstream decision making depends on the predicted probabilities.
One of the important aspects of machine learning research is the measure used to evaluate the performance of a model and in the context of calibration, this amounts to measuring the difference between two empirical probability distributions.
To this end, the popular metric, Expected Calibration Error (ECE) (Naeini et al. (2015)), approximates the classwise probability distributions using histograms and takes an expected difference.
This histogram approximation has a weakness that the resulting calibration error depends on the binning scheme (number of bins and bin divisions).
Even though the drawbacks of ECE have been pointed out and some improvements have been proposed (Kumar et al.
; Nixon et al.
, the histogram approximation has not been eliminated.
In this paper, we ﬁrst introduce a simple, binning-free calibration measure inspired by the classical
Kolmogorov-Smirnov (KS) statistical test (Kolmogorov (1933); Smirnov (1939)), which also provides an effective visualization of the degree of miscalibration similar to the reliability diagram (NiculescuMizil & Caruana (2005)).
To this end, the main idea of the KS-test is to compare the respective classwise cumulative (empirical) distributions.
Furthermore, by approximating the empirical cumulative distribution using a differentiable function via splines (McKinley & Levine (1998)), we obtain an analytical recalibration function which maps the given network outputs to the actual class assignment probabilities.
Such a direct mapping was previously unavailable and the problem has been approached indirectly via learning, for example, by optimizing the (modiﬁed) cross-entropy loss (Guo et al.
; Mukhoti et al.
; Müller et al.
Similar to the existing methods (Guo et al.
(2017); Kull et al.
the spline-ﬁtting is performed using a held-out calibration set and the obtained recalibration function is evaluated on an unseen test set.
We evaluated our method against existing calibration approaches on various image classiﬁcation datasets and our spline-based recalibration approach consistently outperforms existing methods on
KS error, ECE as well as other commonly used calibration measures.
Our approach to calibration does not update the model parameters, which allows it to be applied on any trained network and it retains the original classiﬁcation accuracy in all the tested cases.
N OTATION AND P RELIMINARIES
We abstract the network as a function f θ : D → [0 , 1] K , where D ⊂ IR d , and write f θ ( x ) =
z .
Here, x may be an image, or other input datum, and z is a vector, sometimes known as the vector of logits .
In this paper, the parameters θ will not be relevant, and we write simply f to represent the network function.
Moreover, a function of this type will be referred to as a classiﬁer , which may be of some other kind than a neural network.
In a classiﬁcation problem, K is the number of classes to be distinguished, and we call the value
z k (the k -th component of vector z ) the score for the class k .
If the ﬁnal layer of a network is a softmax layer, then the values z k satisfy (cid:80)
Kk =
z k = 1 , and z k ≥ 0 .
Hence, the z k are pseudoprobabilities, though they do not necessarily have anything to do with real probabilities of correct class assignments.
Typically, the value y ∗ =
arg max k z k is taken as the ( top- 1 ) prediction of the network, and the corresponding score, max k z k is called the conﬁdence of the prediction.
However, the term conﬁdence does not have any mathematical meaning in this context and we deprecate its use.
We assume we are given a set of training data ( x i , y i ) ni
=1 , where x i ∈
D is an input data element, which for simplicity we call an image, and y i ∈ K
= { 1 , . . .
, K } is the so-called ground-truth label.
Our method also uses two other sets of data, called calibration data and test data .
It would be desirable if the numbers z k output by a network represented true probabilities.
For this to make sense, we posit the existence of joint random variables ( X, Y ) , where X takes values in a domain D ⊂ IR d , and Y takes values in K .
Further, let Z = f
( X ) , another random variable, and Z k = f k ( X ) be its k -th component.
Note that in this formulation X and Y are joint random variables, and the probability P ( Y | X ) is not assumed to be 1 for single class, and 0 for the others.
A network is said to be calibrated if for every class k ,
[Equation]
This can be written brieﬂy as P ( k | f ( x )) =
f k ( x ) =
z k .
Thus, if the network takes input x and outputs z = f ( x ) , then z k represents the probability (given f ( x ) ) that image x belongs to class k .
The probability P ( k | z ) is difﬁcult to evaluate, even empirically, and most metrics (such as ECE) use or measure a different notion called classwise calibration (Kull et al.
(2019); Zadrozny & Elkan (2002)), deﬁned as,
[Equation]
This paper uses this deﬁnition (2) of calibration in the proposed KS metric.
Calibration and accuracy of a network are different concepts.
For instance, one may consider a classiﬁer that simply outputs the class probabilities for the data, ignoring the input x .
Thus, if f k ( x ) =
z k = P ( Y = k ) , this classiﬁer f is calibrated but the accuracy is no better than the random predictor.
Therefore, in calibration of a classiﬁer, it is important that this is not done while sacriﬁcing classiﬁcation (for instance top- 1 ) accuracy.
The top- r prediction.
The classiﬁer f being calibrated means that f k ( x ) is calibrated for each class k , not only for the top class.
This means that scores z k for all classes k give a meaningful estimate of the probability of the sample belonging to class k .
This is particularly important in medical diagnosis where one may wish to have a reliable estimate of the probability of certain unlikely diagnoses.
Frequently, however, one is most interested in the probability of the top scoring class, the topprediction, or in general the top- r prediction.
Suppose a classiﬁer f is given with values in [0 , 1] K and let y be the ground truth label.
Let us use f ( − r ) to denote the r -th top score (so f ( − 1) would denote the top score; the notation follows python semantics in which A [ − 1] represents the last element in array A ).
Similarly we deﬁne max ( − r ) for the r -th largest value.
Let f ( − r ) :
D → [0 , 1] be deﬁned as f ( − r )
( x ) = max ( − r ) k f k ( x ) , and y ( − r )
=
(cid:40)if y = arg
[Equation]
( −
[Equation]
k
[Equation]
otherwise .
In words, y ( − r ) is 1 if the r -th top predicted class is the correct (ground-truth) choice.
The network is calibrated for the top- r predictor if for all scores σ ,
[Equation]
In words, the conditional probability that the top- r -th choice of the network is the correct choice, is equal to the r -th top score.
Similarly, one may consider probabilities that a datum belongs to one of the top- r scoring classes.
The classiﬁer is calibrated for being within-the-top- r classes if
[Equation]
Here, the sum on the left is 1 if the ground-truth label is among the top r choices, 0 otherwise, and the sum on the right is the sum of the top r scores.
K OLMOGOROV -S MIRNOV C ALIBRATION E RROR
We now consider a way to measure if a classiﬁer is classwise calibrated, including top- r and withintop- r calibration.
This test is closely related to the Kolmogorov-Smirnov test (Kolmogorov (1933);
Smirnov (1939)) for the equality of two probability distributions.
This may be applied when the probability distributions are represented by samples.
We start with the deﬁnition of classwise calibration:
[Equation]
Bayes’ rule
[Equation]
This may be written more simply but with a less precise notation as
[Equation]
Motivation of the KS test.
One is motivated to test the equality (or difference between) two distributions, deﬁned on the interval [0 , 1] .
However, instead of having a functional form of these distributions, one has only samples from them.
Given samples ( x i , y i ) , it is not straight-forward to estimate P ( z k ) or P ( z k | k ) , since a given value
z k is likely to occur only once, or not at all, since the sample set is ﬁnite.
One possibility is to use histograms of these distributions.
However, this requires selection of the bin size, and the division between bins, and the result depends on these parameters.
For this reason, we abjure this solution.
The approach suggested by the Kolmogorov-Smirnov test is to compare the cumulative distributions.
Thus, with k given, one tests the equality
[Equation]
Writing φ 1 ( σ ) and φ 2 ( σ ) to be the two sides of this equation, the KS-distance between these two distributions is deﬁned as KS = max σ | φ 1 ( σ ) − φ 2 ( σ ) | .
The fact that simply the maximum is used here may suggest a lack of robustness, but this is a maximum difference between two integrals, so it reﬂects an accumulated difference between the two distributions.
To provide more insights into the KS-distance, let us a consider a case where z k consistently over or under-estimates P ( k | z k ) (which is usually the case, at least for top- 1 classiﬁcation (Guo et al. (2017))), then P ( k | z k ) − z k has constant sign for all values of z k .
It follows that P ( z k , k ) − z k P ( z k ) has constant sign and so the maximum value in the KS-distance is achieved when σ = 1 .
In this case,
[Equation]
which is the expected difference between z k and P ( k | z k )
This can be equivalently referred to as the expected calibration error for the class k .
[Figure]
Percentile 0.
C u m u l a ti v e S c
o r e / P r ob a b ilit y (a)
[Figure]
1.0 Cumulative Score 0.
[Figure] [Figure]
Percentile 0.
[Figure]
[Figure]
c
[Figure]
e /
P
[Figure]
ob
[Figure]
ilit y (
[Figure]
ScoreProbability
[Figure]
ScoreProbability
[Figure]
Sampled distributions.
Given samples ( x i , y i ) Ni =1 , and a ﬁxed k , one can estimate these cumulative distributions by
[Equation]
where 1 : B → { 0 , 1 } is the function that returns 1 if the Boolean expression is true and otherwise 0 .
Thus, the sum is simply a count of the number of samples for which y i
= k and f k ( x i ) ≤ σ , and so the integral represents the proportion of the data satisfying this condition.
Similarly,
[Equation]
These sums can be computed quickly by sorting the data according to the values f k ( x i ) , then deﬁning two sequences as follows.
[Equation]
The two sequences should be the same, and the metric
[Equation]
gives a numerical estimate of the similarity, and hence a measure of the degree of calibration of f k .
This is essentially a version of the Kolmogorov-Smirnov test for equality of two distributions.
Remark.
All this discussion holds also when k < 0 , for top- r and within-top- r predictions as discussed in section 2.
In (11), for instance, f − 1 ( x i ) means the top score, f − 1 ( x i ) =
max k ( f k ( x i )) , or more generally, f − r ( x i ) means the r -th top score.
Similarly, the expression y i = − r means that y i is the class that has the r -th top score.
Note when calibrating the top- 1 score, our method is applied after identifying the top- 1 score, hence, it does not alter the classiﬁcation accuracy.
R ECALIBRATION USING S PLINES
The function h i deﬁned in (11) computes an empirical approximation
[Equation]
For convenience, the value of f k will be referred to as the score .
We now deﬁne a continuous function h ( t ) for t ∈ [0 , 1] by
[Equation]
where s ( t ) is the t -th fractile score, namely the value that a proportion t of the scores f k ( X ) lie below.
For instance s (0 . 5) is the median score.
So, h i is an empirical approximation to h ( t )
where t = i/N .
We now provide the basic observation that allows us to compute probabilities given the scores.
Proposition 4.1.
If h ( t ) = P ( Y = k, f k ( X ) ≤ s ( t )) as in (14) where s ( t ) is the t -th fractile score, then h (cid:48) ( t ) =
P ( Y = k |
f k ( X ) =
s ( t )) , where h (cid:48) ( t ) = dh/dt .
Proof.
The proof relies on the equality P ( f k ( X ) ≤ s ( t ))
=
t .
In words, since s ( t ) is the value that a fraction t of the scores are less than or equal, the probability that a score is less than or equal to s ( t ) , is (obviously) equal to t .
See the supplementary material for a detailed proof.
Notice h (cid:48) ( t ) allows a direct conversion from score to probability.
Therefore, our idea is to approximate h i using a differentiable function and take the derivative which would be our recalibration function.
4. S PLINE FITTING
The function h
i (shown in ﬁg 1a) is obtained through sampling only.
Nevertheless, the sampled graph is smooth and increasing.
There are various ways to ﬁt a smooth curve to it, so as to take derivatives.
We choose to ﬁt the sampled points h i to a cubic spline and take its derivative.
Given sample points ( u i , v i ) Ni =1 in IR × IR , easily available references show how to ﬁt a smooth spline curve that passes directly through the points ( u i , v i ) .
A very clear description is given in McKinley &
Levine (1998), for the case where the points u i are equally spaced.
We wish, however, to ﬁt a spline curve with a small number of knot points to do a least-squares ﬁt to the points.
For convenience, this is brieﬂy described here.
A cubic spline v ( u ) is deﬁned by its values at certain knot points (ˆ u k , ˆ v k )
Kk
=1 .
In fact, the value of the curve at any point u can be written as a linear function v ( u ) =
(cid:80)
Kk =1 a k ( u )ˆ v k
= a (cid:62) ( u ) ˆ v , where the coefﬁcients a k depend on u .
Therefore, given a set of further points ( u i , v i ) Ni =1 , which may be different from the knot points, and typically more in number, least-squares spline ﬁtting of the points ( u i , v i ) can be written as a least-squares problem min ˆ v (cid:107) A ( u )ˆ v − v (cid:107) 2 , which is solved by standard linear least-squares techniques.
Here, the matrix A has dimension N × K with N > K .
Once ˆ v is found, the value of the spline at any further points u is equal to v ( u ) = a ( u ) (cid:62) ˆ v , a linear combination of the knot-point values ˆ v k .
Since the function is piecewise cubic, with continuous second derivatives, the ﬁrst derivative of the spline is computed analytically.
Furthermore, the derivative v (cid:48) ( u ) can also be written as a linear combination v (cid:48) ( u ) = a (cid:48) ( u ) (cid:62) ˆ v , where the coefﬁcients a (cid:48) ( u ) can be written explicitly.
Our goal is to ﬁt a spline to a set of data points ( u i , v i )
=
( i/N, h i ) deﬁned in (11), in other words, the values h
i plotted against fractile score.
Then according to Proposition 4.1, the derivative of the spline is equal to P ( k | f k ( X ) =
s ( t )) .
This allows a direct computation of the conditional probability that the sample belongs to class k .
Since the derivative of h i is a probability one might constrain the derivative to be in the range [0 , 1] while ﬁtting splines.
This can be easily incorporated because the derivative of the spline is a linear expression in ˆ v i .
The spline ﬁtting problem thereby becomes a linearly-constrained quadratic program (QP).
However, although we tested this, in all the reported experiments, a simple least-squares solver is used without the constraints.
4. R ECALIBRATION
We suppose that the classiﬁer f = f θ is ﬁxed, through training on the training set.
Typically, if the classiﬁer is tested on the training set, it is very close to being calibrated.
However, if a classiﬁer f is then tested on a different set of data, it may be substantially mis-calibrated.
See ﬁg 1.
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
a
[Figure]
c
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
Our method of calibration is to ﬁnd a further mapping γ : [0 , 1] → [0 , 1] , such that γ ◦ f k is calibrated.
This is easily obtained from the direct mapping from score f k ( x ) to P ( k | f k ( x )) (refer to ﬁg 1d).
In equations, γ ( σ ) = h (cid:48) ( s − 1 ( σ )) .
The function h (cid:48) is known analytically, from ﬁtting a spline to h ( t ) and taking its derivative.
The function s − 1 is a mapping from the given score σ to its fractile s − 1 ( σ ) .
Note that, a held out calibration set is used to ﬁt the splines and the obtained recalibration function γ is evaluated on an unseen test set.
To this end, given a sample x from the test set with f k ( x ) = σ , one can compute h (cid:48)
( s − 1 ( σ )) directly in one step by interpolating its value between the values of h (cid:48) ( f k ( x i )) and h (cid:48) ( f k ( x i +1 ))
where x i and x i +1 are two samples from the calibration set, with closest scores on either side of σ .
Assuming the samples in the calibration set are ordered, the samples x i and x i +1 can be quickly located using binary search.
Given a reasonable number of samples in the calibration set, (usually in the order of thousands), this can be very accurate.
In our experiments, improvement in calibration is observed in the test set with no difference to the accuracy of the network (refer to ﬁg 2d).
In practice, spline ﬁtting is much faster than one forward pass through the network and it is highly scalable compared to learning based calibrtaion methods.
Our code will be published to ensure reproducibility.
R ELATED W ORK
Modern calibration methods.
In recent years, neural networks are shown to overﬁt to the Negative
Log-Likelihood (NLL) loss and in turn produce overconﬁdent predictions which is cited as the main reason for miscalibration (Guo et al.
To this end, modern calibration methods can be broadly categorized into 1) methods that adapt the training procedure of the classiﬁer, and 2) methods that learn a recalibration function post training.
Among the former, the main idea is to increase the entropy of the classiﬁer to avoid overconﬁdent predictions, which is accomplished via modifying the training loss (Kumar et al.
; Mukhoti et al.
Seo et al.
, label smoothing (Müller et al.
; Pereyra et al.
and data augmentation techniques (Thulasidasan et al.
; Yun et al.
; Zhang et al.
[Table]
bold and second lowest underlined) on various image classiﬁcation datasets and models with different calibration methods.
Note, our method consistently reduces calibration error to < 1 % in almost all experiments, outperforming state-of-the-art methods.
On the other hand, we are interested in calibrating an already trained classiﬁer that eliminates the need for training from scratch.
In this regard, a popular approach is Platt scaling (Platt et al.
(1999)) which transforms the outputs of a binary classiﬁer into probabilities by ﬁtting a scaled logistic function on a held out calibration set.
Similar approaches on binary classiﬁers include Isotonic
Regression (Zadrozny & Elkan (2001)), histogram and Bayesian binning (Naeini et al.
Zadrozny & Elkan (2001)), and Beta calibration (Kull et al.
(2017)), which are later extended to the multiclass setting (Guo et al.
Kull et al.
; Zadrozny & Elkan (2002)).
Among these, the most popular method is temperature scaling (Guo et al.
(2017)), which learns a single scalar on a held out set to calibrate the network predictions.
Despite being simple and one of the early works, temperature scaling is the method to beat in calibrating modern networks.
Our approach falls into this category, however, as opposed to minimizing a loss function, we obtain a recalibration function via spline-ﬁtting, which directly maps the classiﬁer outputs to the calibrated probabilities.
Calibration measures.
Expected Calibration Error (ECE) (Naeini et al.
(2015)) is the most popular measure in the literature, however, it has a weakness that the resulting calibration error depends on the histogram binning scheme such as the bin endpoints and the number of bins.
Even though, some improvements have been proposed (Nixon et al.
; Vaicenavicius et al.
, the binning scheme has not been eliminated and it is recently shown that any binning scheme leads to underestimated calibration errors (Kumar et al.
; Widmann et al.
Note that, there are binning-free metrics exist such as Brier score (Brier (1950)), NLL, and kernel based metrics for the multiclass setting (Kumar et al.
; Widmann et al.
Nevertheless, the Brier score and
NLL measure a combination of calibration error and classiﬁcation error (not just the calibration which is the focus).
Whereas kernel based metrics, besides being computationally expensive, measure the calibration of the predicted probability vector rather than the classwise calibration error (Kull et al.
(or top- r prediction) which is typically the quantity of interest.
To this end, we introduce a binning-free calibration measure based on the classical KS-test, which has the same beneﬁts of ECE and provides effective visualizations similar to reliability diagrams.
Furthermore, KS error can be shown to be a special case of kernel based measures (Gretton et al.
E XPERIMENTS
Experimental setup.
We evaluate our proposed calibration method on four different imageclassiﬁcation datasets namely CIFAR-10/100 (Krizhevsky et al.
, SVHN (Netzer et al.
and ImageNet (Deng et al.
using LeNet (LeCun et al.
, ResNet (He et al.
ResNet with stochastic depth (Huang et al.
, Wide ResNet (Zagoruyko & Komodakis (2016)) and DenseNet (Huang et al.
network architectures against state-of-the-art methods that calibrate post training.
We use the pretrained network logits 3 for spline ﬁtting where we choose calibration
[Table]
and second lowest underlined) on various image classiﬁcation datasets and models with different calibration methods.
Again, our method consistently reduces calibration error to < 1 % (less then 0 .
7 %, except for one case), in all experiments, the only one of the methods to achieve this.
validation set as the calibration set, similar to the standard practice.
Our ﬁnal results for calibration are then reported on the test set of all datasets.
Since ImageNet does not comprise of the validation set, test set is divided into two halves: calibration set and test set.
We use the natural cubic spline ﬁtting method (that is, cubic splines with linear run-out) with 6 knots for all our experiments.
Further experimental details are provided in the supplementary.
For baseline methods namely: Temperature scaling, Vector scaling, Matrix scaling with ODIR (Off-diagonal and Intercept Regularisation), and
Dirichlet calibration, we use the implementation of Kull et al .
(Kull et al.
Results.
We provide comparisons of our method using proposed KS error for the top most prediction against state-of-the-art calibration methods namely temperature scaling (Guo et al.
(2017)), vector scaling, MS-ODIR and Dirichlet Calibration (Dir-ODIR)
(Kull et al.
in Table 1.
Our method reduces calibration error to 1% in almost all experiments performed on different datasets without any loss in accuracy.
It clearly reﬂects the efﬁcacy of our method irrespective of the scale of the dataset as well as the depth of the network architecture.
It consistently performs better than recently introduced
Dirichlet calibration and Matrix scaling with ODIR (Kull et al.
in all the experiments.
Note this is consistent with the top-1 calibration results reported in Table 15 of (Kull et al.
The closest competitor to our method is temperature scaling, against which our method performs better in 9 out of 13 experiments.
Note, in the cases where temperature scaling outperforms our method, the gap in KS error between the two methods is marginal ( < 0.3 % ) and our method is the second best.
We provide comparisons using other calibration metrics in the supplementary.
From the practical point of view, it is also important for a network to be calibrated for top second/third predictions and so on.
We thus show comparisons for top-2 prediction KS error in Table 2.
An observation similar to the one noted in Table 1 can be made for the top-2 predictions as well.
Our method achieves < 1 % calibration error in all the experiments.
It consistently performs well especially for experiments performed on large scale ImageNet dataset where it sets new state-of-the-art for calibration .
We would like to emphasize here, though for some cases Kull et al .
(Kull et al.
and Vector Scaling perform better than our method in terms of top-2 KS calibration error, overall (considering both top-1 and top-2 predictions) our method performs better.
C ONCLUSION
In this work, we have introduced a binning-free calibration metric based on the Kolmogorov-Smirnov test to measure classwise or (within)-top- r calibration errors.
Our KS error eliminates the shortcomings of the popular ECE measure and its variants while accurately measuring the expected calibration error and provides effective visualizations similar to reliability diagrams.
Furthermore, we introduced a simple and effective calibration method based on spline-ﬁtting which does not involve any learning and yet consistently yields the lowest calibration error in the majority of our experiments.
We believe, the KS metric would be of wide-spread use to measure classwise calibration and our spline method would inspire learning-free approaches to neural network calibration. We intend to focus on calibration beyond classiﬁcation problems as a future work.
Appendices
Here we ﬁrst provide the proof of our main result, discuss more about top- r calibration and splineﬁtting and then turn to additional experiments
A P ROOF OF P ROPOSITION 4.
We ﬁrst restate our proposition below.
Proposition A.2.
If h ( t ) = P ( Y = k, f k ( X ) ≤ s ( t )) as in (14) of the main paper where s ( t ) is the t -th fractile score.
Then h (cid:48) ( t ) =
P ( Y = k |
f k ( X ) =
s ( t )) , where h (cid:48) ( t ) = dh/dt .
Proof.
The proof is using the fundamental relationship between the Probability Distribution Function (PDF) and the Cumulative Distribution Function (CDF) and it is provided here for completeness.
Taking derivatives, we see (writing P ( k ) instead of P ( Y = k ) ):
[Equation]
The proof relies on the equality P ( f k ( X ) ≤ s ( t ))
=
t .
In words: s ( t ) is the value that a fraction t of the scores are less than or equal.
This equality then says: the probability that a score is less than or equal to the value that a fraction t of the scores lie below, is (obviously) equal to t .
B M ORE ON TOP - r AND WITHIN - TOP - r C ALIBRATION
In the main paper, deﬁnitions of top- r and within-top- r calibration are given in equations (4) and (5).
Here, a few more details are given of how to calibrate the classiﬁer f for top- r and within-top- r calibration.
The method of calibration using splines described in this paper consists of ﬁtting a spline to the cumulative accuracy, deﬁned as h i in equation (11) in the main paper.
For top- r classiﬁcation, the method is much the same as for the classiﬁcation for class k .
Equation (11) is replaced by sorting the data according to the r -th top score, then deﬁning
[Equation]
where y ( − r ) and f ( − r ) ( x i ) are deﬁned in the main paper, equation (3).
These sequences may then be used both as a metric for the correct top- r calibration and for calibration using spline-ﬁtting as described.
For within-top- r calibration, one sorts the data according to the sum of the top r scores, namely (cid:80) rs
=1 f ( − s ) ( x i ) , then computes
[Equation]
As before, this can be used as a metric, or as the starting point for within-top- r calibration by our method.
Examples of this type of calibration (graphs for uncalibrated networks in ﬁg 7 and ﬁg 9) is given in the graphs provided in ﬁg 8 and ﬁg 10 for within-top-2 predictions and within-toppredictions respectively.
It is notable that if a classiﬁer is calibrated in the sense of equation (1) in the main paper (also called multi-class-calibrated), then it
C L EAST S QUARE S PLINE F ITTING
Least-square ﬁtting using cubic splines is a known technique.
However, details are given here for the convenience of the reader.
Our primary reference is (McKinley & Levine (1998)), which we adapt to least-squares ﬁtting.
We consider the case where the knot-points are evenly spaced.
We change notation from that used in the main paper by denoting points by ( x, y ) instead of ( u, v ) .
Thus, given knot points (ˆ x i , ˆ y i )
Kk =1 one is required to ﬁt some points ( x i , y i ) Ni =1 .
Given a point x , the corresponding spline value is given by y = a ( x ) (cid:62) M ˆ y , where ˆ y is the vector of values ˆ y i .
The form of the vector a ( x ) and the matrix M are given in the following.
The form of the matrix M is derived from equation (25) in McKinley & Levine (1998).
Deﬁne the matrices
[Equation]
where h is the distance between the knot points.
These matrices are of dimensions K −
2 × K −and K − 2 × K respectively.
Finally, let M be the matrix
[Equation]
Here, 0 K is a vector of zeros of length K , and I K × K is the identity matrix.
The matrix M has dimension 2 K × K .
Next, let the point x lie between the knots j and j + 1 and let u = x − ˆ x j .
Then deﬁne the vector v = a ( x ) by values
[Equation]
with other entries equal to 0 .
Then the value of the spline is given by
[Equation]
as required.
This allows us to ﬁt the spline (varying the values of ˆ y ) to points ( x i , y i ) by least-squares ﬁt, as described in the main paper.
The above description is for so-called natural (linear-runout) splines.
For quadratic-runout or cubicrunout splines the only difference is that the ﬁrst and last rows of matrix A are changed – see McKinley & Levine (1998) for details.
As described in the main paper, it is also possible to add linear constraints to this least-squares problem, such as constraints on derivatives of the spline.
This results in a linearly-constrained quadratic programming problem.
D A DDITIONAL E XPERIMENTS
We ﬁrst provide the experimental setup for different datasets in Table 3.
Note, the calibration set is used for spline ﬁtting in our method and then ﬁnal evaluation is based on an unseen test set.
[Figure]
c
[Figure]
ilit
[Figure]
described in Section 4 of main paper
[Figure]
[Figure] [Figure]
ob a y
[Figure]
[Figure] [Figure]
v e
[Figure]
c o r e / P
r ob a b ilit y
[Figure]
[Figure] [Figure]
[Figure]
C u m u l a ti v e S c
o r e / P r ob a b ilit y
[Figure]
[Figure] [Figure]
C u m u l a ti v e S c
o r e / P r ob a b ilit y
[Figure]
[Table] [Table]
We also provide comparisons of our method against baseline methods for within-top-2 predictions (equation 5 of the main paper) in Table 4 using KS error.
Our method achieves comparable or better results for within-top-2 predictions. It should be noted that the scores for top-3 ( f ( − 3) ( x ) ) or even top-4, top-5, etc., are very close to zero for majority of the samples (due to overconﬁdence of top- 1 predictions). Therefore the calibration error for top- r with r > 2 predictions is very close to zero and comparing different methods with respect to it is of little value. Furthermore, for visual illustration, we provide calibration graphs of top-2 predictions in ﬁg 3 and ﬁg 4 for uncalibrated and calibrated network respectively.
Similar graphs for top-3, within-top-2, and within-top-3 predictions are presented in ﬁgures 5 – 10.
We also provide classiﬁcation accuracy comparisons for different post-hoc calibration methods against our method if we apply calibration for all top- 1 , 2 , 3 , . .
. , K predictions for K -class classiﬁcation problem in Table 5.
We would like to point out that there is negligible change in accuracy between the calibrated networks (using our method) and the uncalibrated ones.
For the sake of completeness, we present calibration results using the existing calibration metric,
Expected Calibration Error (ECE) (Naeini et al.
in Table 6.
We would like to reiterate the fact that ECE metric is highly dependent on the chosen number of bins and thus does not really reﬂect true calibration performance.
To reﬂect the efﬁcacy of our proposed calibration method, we also present calibration results using other calibration metrics such as recently proposed binning free measure
KDE-ECE (Zhang et al.
MCE (Maximum Calibration Error) (Guo et al. (2017)) and Brier Scores for top-1 predictions on ImageNet dataset in Table 7.
Since, the original formulation of Brier
Score for multi-class predictions is highly biased on the accuracy and is approximately similar for all calibration methods, we hereby use top-1 Brier Score which is the mean squared error between topscores and ground truths for the top-1 predictions (1 if the prediction is correct and 0 otherwise).
It can be clearly observed that our approach consistently outperforms all the baselines on different calibration measures.
[Table]
[Table]
datasets and with
[Table]