S HOULD E NSEMBLE M EMBERS B E C ALIBRATED ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Underlying the use of statistical approaches for a wide range of applications is the assumption that the probabilities obtained from a statistical model are representative of the “true” probability that event, or outcome, will occur.
Unfortunately, for modern deep neural networks this is not the case, they are often observed to be poorly calibrated.
Additionally, these deep learning approaches make use of large numbers of model parameters, motivating the use of Bayesian, or ensemble approximation, approaches to handle issues with parameter estimation.
This paper explores the application of calibration schemes to deep ensembles from both a theoretical perspective and empirically on a standard image classiﬁcation task,
CIFAR-100.
The underlying theoretical requirements for calibration, and associated calibration criteria, are ﬁrst described.
It is shown that well calibrated ensemble members will not necessarily yield a well calibrated ensemble prediction, and if the ensemble prediction is well calibrated its performance cannot exceed that of the average performance of the calibrated ensemble members.
On CIFAR-100 the impact of calibration for ensemble prediction, and associated calibration is evaluated.
Additionally the situation where multiple different topologies are combined together is discussed.
I NTRODUCTION
Deep learning approaches achieve state-of-the-art performance in a wide range of applications, including image classiﬁcation.
However, these networks tend to be overconﬁdent in their predictions, they often exhibit poor calibration.
A system is well calibrated, if when the system makes a prediction with probability of 0.6 then 60% of the time that prediction is correct.
Calibration is very important in deploying system, especially in risk-sensitive tasks, such as medicine (Jiang et al., 2012), auto-driving (Bojarski et al., 2016), and economics (Gneiting et al., 2007).
It was shown by Niculescu-Mizil & Caruana (2005) that shallow neural networks are well calibrated.
However, Guo et al.
(2017) found that more complex neural network model with deep structures do not exhibit the same behaviour. This work motivated recent research into calibration for general deep learning systems.
Previous research has mainly examined calibration based on samples from the true data distribution { x ( i ) , y ( i ) }
Ni =1 ∼ p ( x , ω ) , y ( i ) ∈ { ω 1 , ..., ω K } (Zadrozny & Elkan, 2002; Vaicenavicius et al., 2019). This analysis relies on the limiting behaviour as N → + ∞ to deﬁne a well calibrated system
[Equation]
where S pj =
{ i | P (ˆ y
i ) =
j | x ( i ) ; θ ) = p, i = 1 , ..., N } and ˆ y ( i ) the model prediction for x ( i ) .
However, Eq.
(1) doesn’t explicitly reﬂect the relation between P ( y = ˆ y | P (ˆ y | x ; θ ) = p ) and the underlying data distribution p ( x , y ) .
In this work we examine this explicit relationship and use it to deﬁne a range of calibration evaluation criteria, including the standard sample-based criteria.
One issue with deep-learning approaches is the large number of model parameters associated with the networks. Deep ensembles (Lakshminarayanan et al., 2017) is a simple, effective, approach for handling this problem.
It has been found to improve performance, as well as allowing measures of uncertainty.
In recent literature there has been “contradictory” empirical observations about the relationship between the calibration of the members of the ensemble and the calibration of the ﬁnal ensemble prediction (Rahaman & Thiery, 2020; Wen et al., 2020).
In this paper, we examine the underlying theory and empirical results relating to calibration with ensemble methods.
We found, both theoretically and empirically, that ensembling multiple calibrated models decreases the conﬁdence of ﬁnal prediction, resulting in an ill-calibrated ensemble prediction.
To address this, strategies to calibrate the ﬁnal ensemble prediction, rather than individual members, are required. Additionally we empiricaly examine the situation where the ensemble is comprised of models with different topologies, and resulting complexity/performance, requiring non-uniform ensemble averaging.
In this study, we focus on post-hoc calibration of ensemble, based on temperature annealing.
Guo et al.
(2017) conducted a thorough comparison of various existing post-hoc calibration methods and found that temperature scaling was a simple, fast, and often highly effective approach to calibration.
However, standard temperature scaling acts globally for all regions of the input samples, i.e. all logits are scaled towards one single direction, either increasing or decreasing the distribution entropy.
To address this constraint, that may hurt some legitimately conﬁdent predictions, we investigate the effect of region-speciﬁc temperatures.
Empirical results demonstrate the effectiveness of this approach, with minimal increase in the number of calibration parameters.
R ELATED W ORK
Calibration is inherently related to uncertainty modeling.
Two of the most important scopes of calibration are calibration evaluation and calibration system construction.
One method to assessing calibration is the reliability diagram (Vaicenavicius et al., 2019;
Br ¨ ocker, 2012).
Though informative, It is still desirable to have an overall metric.
Widmann et al.
(2019) investigate different distances in the probability simplex for estimating calibration error.
Nixon et al.
(2019) point out the problem of ﬁxed spaced binning scheme, bins with few predictions may have low-bias but high-variance measurement.
Calibration error measure adaptive to dense populated regions have also been proposed (Nixon et al., 2019).
Vaicenavicius et al.
(2019) treated the calibration evaluation as hypotheses tests.
All these approaches examine calibration criteria from a sample-based perspective, rather than as a function of the underlying data distribution which is used in the thoretical analysis in this work.
There are two main approaches to calibrating systems.
The ﬁrst is to recalibrate the uncalibrated systems with post-hoc calibration mapping, e.g. Platt scaling (Platt et al., 1999), isotonic regression (Zadrozny & Elkan, 2002), Dirichlet calibration (Kull et al., 2017; 2019).
The second is to directly build calibrated systems, via: (i) improving model structures, e.g. deep convolutional Gaussian processes (Tran et al., 2019); (ii) data augmentation, e.g. adversarial samples (Stutz et al., 2020) or Mixup (Zhang et al., 2018); (iii) minimize calibration error during training (Kumar et al., 2018).
Calibration based on histogram binning (Zadrozny & Elkan, 2001), Bayesian binning (Naeini et al., 2015) and scaling binning (Kumar et al., 2019) are related to our proposed dynamic temperature scaling, in the sense that the samples are divided into regions and separate calibration mapping are applied.
However, our method can preserve the property that all predictions belonging to one sample sum to 1.
The region-based classiﬁer by Kuleshov & Liang (2015) is also related to our approach.
Ensemble diversity has been proposed for improved calibration (Raftery et al., 2005; Stickland & Murray, 2020).
In Zhong & Kwok (2013), ensembles of SVM, logistic regressor, boosted decision trees are investigated, where the combination weights of calibrated probabilities is based on AUC of
ROC.
In this work we investigate the combination of different deep neural network structures.
The weights assigned to the probabilities is either optimised using AUC as in (Ashukha et al., 2020) or a likelihood-based metric.
C ALIBRATION F RAMEWORK
Let X ⊆
R
d be the d -dimensional input space and Y = { ω 1 , ..., ω K } be the discrete output space consisting of K classes.
The true underlying joint distribution for the data is p ( x , ω ) =
P ( ω | x ) p ( x ) , x ∈ X , ω ∈ Y .
Given some training data D ∼ p ( x , ω ) , a model θ is trained to predict the distribution P ( ω | x ; θ ) given observation features. For a calibrated system the average predicted posterior probability should equate to the average posterior of the underlying distribution for a speciﬁc probability region.
Two extreme cases will always yield perfect calibration.
First when the predictions that are the same, and equal to the class prior for all inputs, P ( ω j | x ; θ ) =
P ( ω j ) .
Second the minimum Bayes’ risk classiﬁer is obtained, P ( ω j | x ; θ ) = p [Equation] x
[Equation] ω [Equation] x
[Equation]
k ) .
Note that perfect calibration doesn’t imply high accuracy, as shown by the system predicting the prior distribution.
3. D ISTRIBUTION C ALIBRATION
A system is calibrated if the predictive probability values can accurately indicate the portion of correct predictions.
Perfect calibration for a system that yields P ( ω | x ; θ ) when the training and test data are obtained form the joint distribution p ( x , ω ) can be deﬁned as:
[Equation]
R pj ( θ , (cid:15) ) denotes the region of input space where the system predictive probability for class ω
j is sufﬁciently close, within error of (cid:15) , to the probability p .
A perfectly calibrated system will satisfy this expression for all regions, the expected predictive probability (left side of Eq.
(2)) is identical to the expected correctness, i.e., expected true probability (right side of Eq.
R pj ( θ , (cid:15) ) deﬁnes the region in which calibration is deﬁned.
For top-label calibration , only the most probable class is considered and the region deﬁned in Eq.
(3) is modiﬁed to reﬂect this:
[Equation]
Eq.
(4) is a strict subset of Eq.
As the two calibration regions are different between calibration and top-label calibration, perfect calibration doesn’t imply top-label calibration, and vise versa.
A simple illustrative example of this property is given in A.3.
Binary classiﬁcation, K = 2 , is an exception to this general rule, as the regions for top-label calibration are equivalent to those for perfect calibration, i.e. ˜ R pj ( θ , (cid:15) )
=
R pj
( θ , (cid:15) ) .
Hence, perfect calibration is equivalent to top-label calibration for binary classiﬁcation (Nguyen & O’Connor, 2015).
Eq.
(2) deﬁnes the requirements for a perfectly calibrated system.
It is useful to deﬁne metrics that allow how close a system is to perfect calibration to be assessed. Let the region calibration error be:
[Equation]
This then allows two forms of expected calibration losses to be deﬁned
[Equation]
All Calibration Error (ACE) only considers the expected calibration error for a particular probability, irrespective of the class associated with the data 1 (Hendrycks et al., 2019).
Hence, All Class Calibration Error (ACCE) that requires that all classes minimises the calibration error for all probabilities is advocated by Kull et al.
; Kumar et al.
Nixon et al.
(2019) propose the Thresholded
Adaptive Calibration Error (TACE) to consider only the prediction larger than a threshold, and it can be described as a special case of ACCE by replacing the integral range.
Naeini et al.
(2015) also propose to only consider the region with maximum error. Though measures such as ACE and ACCE require consistency of the expected posteriors with the true distribution, for tasks with multiple classes, particularly large numbers of classes, the same weight is given to the ability of the model to assign low probabilities to highly unlikely classes, and high probabilities to the “correct” class.
For systems with large numbers of classes this can yield artiﬁcially low scores.
To address this problem it is more common to replace the regions in Eq. (5) with the top-label regions in Eq. (4), to give a top-label calibration error
˜ C pj ( θ , (cid:15) ) .
This then yields the expected top-label equivalents of ACCE and ACE, Expected Class Calibration Error (ECCE) and Expected Calibration Error (ECE).
Here for example ECE by Guo et al. (2017) is expressed as
[Equation]
where O ( θ , p ) =
(cid:80) Kj =1 (cid:82) x ∈ ˜ R pj ( θ ,(cid:15) ) p ( x )d x is the fraction observations that are assigned to that particular probability and Conf ( θ , p ) and
Acc ( θ , p ) are the ideal distribution accuracy and conﬁdences from the model for that probability.
For more details see the appendix.
3. S AMPLE - BASED C ALIBRATION
Usually only samples from the true joint distribution are available.
Any particular training set is drawn from the distribution to yield
[Equation]
The region deﬁned in Eq.
(3) is now changed to be indices of the samples:
[Equation]
The sample-based version of “perfect” calibration in Eq.
(2) can then be expressed as:
[Equation]
as N → ∞ . When considering ﬁnite data, in this case N samples, it is important to set (cid:15) appropriately.
Setting different (cid:15) yields different regions and leads to different calibration results (Kumar et al., 2019).
Thus it is important to specify (cid:15) when deﬁning calibration for a system.
Similarly, the distribution form of top-label calibration can be written in terms of samples as Eq.
(4), with different regions considered:
[Equation]
The sample-based calibration losses in region S pj ( θ , (cid:15) ) can be deﬁned based on Eq.
For example ACE in Eq.
(6) can be expressed in its sample-based form (Hendrycks et al., 2019)
[Equation]
where P ( (cid:15) ) = { p | p = min { 1 , (2 z − 1) (cid:15) } , z ∈ Z + } , and Z + is the set of positive integers.
The measure of ECE relating to Eq.
(7), which only considers the top regions in Eq.
(11) can be deﬁned as Guo et al.
[Equation]
It should be noted that for ﬁnite number of sample, the regions S pj ( θ , (cid:15) ) and
˜ S pj ( θ , (cid:15) ) derived from the samples can be different from the theoretical regions, leading to difference between theoretical calibration error measures and the values estimated from the ﬁnite samples.
This is also referred to as “estimator randomness” by Vaicenavicius et al.
An example is given in A.3 to illustrate this mismatch.
The simplest region speciﬁcation for calibration is to set (cid:15) = 1 .
In this case, |S pj ( θ , 1) | = N , and the “minimum” perfect calibration requirement for a system with parameters θ becomes
[Equation]
This is also referred to as global calibration in this paper.
Similarly, global top-label calibration can be deﬁned as
[Equation]
E NSEMBLE C ALIBRATION
An interesting question when using ensembles is whether calibrating the ensemble members is sufﬁcient to ensure calibrated predictions.
Initially the ensemble model will be viewed as an approximation to Bayesian parameter estimation. Given training data D , the prediction of class ω j is:
[Equation]
where Eq.
(17) is an ensemble, Monte-Carlo, approximation to the full Bayesian integration, with θ ( m ) the m -th ensemble member parameters in the ensemble Θ .
The predictions of ensemble and members are
[Equation]
4. T HEORETICAL A NALYSIS
For ensemble methods it is only important that the ﬁnal ensemble prediction, ˆ y E , is well calibrated, rather than the individual ensemble members.
It is useful to examine the relationship between this ensemble prediction and the predictions from the individual models when the ensemble members are top-label calibrated.
Consider a particular top-label calibration region for the ensemble prediction, ˜
R p ( Θ , (cid:15) ) , related to Eq. (4), the following expression is true
[Equation] where the ensemble region is deﬁned as ˜ R p ( Θ , (cid:15) ) = (cid:110) x
(cid:12)(cid:12)(cid:12) | P (ˆ y E | x ; Θ ) − p | ≤ (cid:15), x ∈ X (cid:111) . For all regions ˜ R p ( Θ , (cid:15) ) the ensemble is no more conﬁdent than the average conﬁdence of individual member predictions.
This puts bounds on performance of the ensemble prediction if the resulting ensemble prediction is top-label calibrated, and the ensemble member regions yield ˜ R p ( Θ , (cid:15) ) .
Now [Equation]
From Eq. (18) the left hand-side of this expression, the ensemble region conﬁdence, cannot be greater that than the average ensemble member conﬁdence. If the regions associated with the ensemble prediction and members are the same, then for top-label calibrated members this average conﬁdence is the same as the average ensemble member accuracy.
Furthermore, if the ensemble prediction is top-label calibrated, then this average ensemble member accuracy bounds the ensemble prediction accuracy, meaning there is no ensemble performance gain. However in general the regions are not the same, and the ensemble prediction is not necessarily calibrated.
It is possible to consider global regions.
If the members of the ensemble are globally calibrated, then the ensemble prediction will be globally calibrated.
However, this is not the case for global top-label calibration so Eq. (19) doesn’t necessarily follow even for global regions, see A.1 for proof. Now consider sample-calibration applying Eq.
(18) at the global region level with ensemble members that are global top-label calibrated. This yields
[Equation]
where ˆ y ( i ) E and ˆ y
( i ) m are the ensemble and member sample x ( i ) predictions and from Eq.
[Equation]
Combining Eq. (20) and Eq. (21) yields the following inequality
[Equation]
Given this expression it is worth examining two scenarios, illustrated by the two inequalities in
Eq.
(23), that relate the inequality in Eq. (22) to the accuracy of the ensemble prediction.
[Equation]
If the ensemble prediction is global top-label calibrated, equality on the left expression, then the ensemble performance cannot exceed that of the ensemble average, the right-hand inequality. In this scenario there is no beneﬁt in using an ensemble. If the ensemble is not top-label calibrated and under-conﬁdent, the left inequality above, then the ensemble performance is not bounded by the right-hand inequality. In practice, there is no constraint that the ensemble prediction should be calibrated, thus ensemble prediction calibration is required even for top-label calibrated members.
In the above discussion, the ensemble members are combined with uniform weights, motivated from a Bayesian approximation perspective. When, for example, multiple different topologies are used as members of the ensemble, a non-uniform averaging of the members of the ensemble, reﬂecting the model complexities and performance may be useful. Using non-uniform weights, Eq. (20) is still true and the discussion of Eq.
(21)-(23) still apply.
4. T EMPERATURE A NNEALING FOR E NSEMBLE C ALIBRATION
Calibrating the ensemble in Eq.
(23) can be performing using a function with some parameters, t , f : [0 , 1] → [0 , 1] for scaling probabilities.
There are two modes for calibrating an ensemble:
Pre-combination Mode.
the function is applied to the probabilities predicted by members, prior to combining the members to obtain ensemble prediction using a set of calibration parameters T .
[Equation]
Post-combination Mode. the function is applied to the ensemble predicted probability after combining members’ predictions.
[Equation]
There are many functions for transforming predicted probability in the calibration literature, e.g. histogram binning, Platt scaling and temperature annealing.
However, histogram binning shouldn’t be adopted in the pre-combination mode as scaling function f for calibrating multi-class ensemble, as the transformed values may not yield a valid PMF.
As shown in Guo et al.
(2017), temperature scaling is a simple, effective, option for the mapping function f , which scales the logit values associated with the posterior by a temperature t , f
( z ; t ) =
exp { z /t } / (cid:80) j exp { z j /t } . Here a single temperature is used for scaling logits for all samples.
This leads to the problem that the entropy of the predictions for all regions are either increased or decreased.
From Eq.
(2) the temperature can be made region speciﬁc. [Equation]
To determine the optimal set of temperatures, the samples in the validation set are divided into R regions based on the ensemble predictions (e.g. R 1 = [0 , 0 . 3) , R 2 =
[0 . 3 , 0 . 6) , and R 3 =
Each region has an individual temperature for scaling {R r , t r } Rr =1 .
4. E MPIRICAL R ESULTS
Experiments were conducted on CIFAR-100 (and CIFAR-10 in the A.4).
The data partition was 45,000/5,000/10,000 images for train/validation/test. We train LeNet (LeCun et al., 1998), DenseNet (Huang et al., 2017) and Wide ResNet (Zagoruyko & Komodakis, 2016) following the original training recipes in each paper (more details in A.4). The results presented are slightly lower than that in the original papers, as 5,000 images were held-out to enable calibration parameter optimisation.
[Figure]
Figure 1 examines the empirical performance of ensemble calibration on CIFAR-100 test set using the three trained networks. The middle row shows the ECE of ensemble members and ensemble prediction at different temperatures.
The optimal calibration temperature for the ensemble prediction are consistently smaller than those associated with the ensemble members. This indicates that the ensemble predictions are less conﬁdent than those of the members, as stated in Eq.
The bottom row of ﬁgures show the reliability curves when the ensemble members are calibrated with optimal temperature values, and the resulting combination.
It is clear that calibrating the ensemble members, using temperature, does not yield a calibrated ensemble prediction. Furthermore for all models the ensemble prediction is less conﬁdent than it should be, the line is above the diagonal.
As discussed in Eq.
(23), this is necessary, or the ensemble prediction is no better, which is clearly not the case for the performance plots in the top row. This ensemble performance is relatively robust to poorly calibrated ensemble members, with consistent performance over a wide range of temperatures.
[Figure] Table 1 shows the calibration performance using three temperature scaling methods, pre-, post- and dynamic post-combination. The temperatures are optimized to minimize ECE (Liang et al., 2020) on the validation data. All three methods effectively improve the ensemble prediction calibration, with the dynamic approach yielding the best performance
[Table] Finally [Table] for the topology ensemble
[Table] weights were optimised using either maximum likelihood (Max LL) or area under curve (AUC) Zhong & Kwok (2013) (results in A.4). In Figure 2, the ensemble of calibrated structures is shown to be uncalibrated
[Table]
with reliability curves typically slightly above the diagonal line.
When the ensemble prediction is calibrated it can be seen that the calibration for the ensemble prediction is lower than the individual calibration errors in Table 1 (“post” lines) [Table] using post-combination mode (“post” in Table 1).
C ONCLUSIONS
State-of-the-art deep learning models often exhibit poor calibration performance.
In this paper two aspects of calibration for these models are investigated: the theoretical deﬁnition of calibration and associated attributes for both general and top-label calibration; and the application of calibration to ensemble methods that are often used in deep-learning approaches for improved performance and uncertainty estimation. It is shown that calibrating members of the ensemble is not sufﬁcient to ensure that the ensemble prediction is itself calibrated. The resulting ensemble predictions will be under-conﬁdent, requiring calibration functions to be optimised for the ensemble prediction, rather than ensemble members. These theoretical results are backed-up by empirical analysis on CIFAR100 deep-learning models, with ensemble performance being robust to poorly calibrated ensemble members but requiring calibration even with well calibrated members.
A A PPENDIX A. T HEORETICAL P ROOF
First we look at general calibration and global general calibration in Proposition 1 and 2. Then with
Proposition 3 and 4 we discuss global top-label calibration. Proposition 1. If all members are calibrated and the regions are the same, i
[Equation]
e., for different members θ ( m ) and θ ( m (cid:48) ) [Equation] then the ensemble is also calibrated on the same regions [Equation]
Proof.
If
[Equation]
The ensemble is also calibrated and the regions are the same:
[Equation]
Proposition 2.
If all members are global calibrated, i
[Equation]
e. [Equation] then the overall ensemble is global calibrated [Equation]
Proof.
If all members are global calibrated, then
[Equation]
Hence,
[Equation] [Equation]
m
[Equation]
(29) =
M
[Equation]
N
[Equation] N [Equation] Proposition 3.
If all members and the corresponding ensemble are globally top-label calibrated, the ensemble performance is no better than the average performance of the members:
[Equation]
Proof.
If all members and the ensemble are globally top-label calibrated,
[Equation]
By deﬁnition
[Equation]
Hence,
[Equation]
Combining Eq.
(32), (33) and (35): [Equation] Proposition 4.
When class number K > 2 , if all members are globally top-label calibrated, then the ensemble is not necessarily global top-label calibrated.
Proof.
Assume globally top-label calibrated members imply globally top-label calibrated ensemble, that is, given
[Equation]
the following is true
[Equation]
If ∃ n, ˜ m, τ > 0 , such that ˆ y ( n ) E (cid:54) =
ˆ y ( n ) ˜ m , then it is possible to write
[Equation]
For top-label calibration there are no constraints on the second term in Eq.
(39) as it is not the toplabel for model θ ( ˜ m ) .
Thus there are a set of models that satisfy the top-label calibration constraints for member ˜ m that only need to satisfy the following constraints
[Equation] [Equation]
and the standard sum-to-one constraint over all classes.
Consider replacing member ˜ m of the ensemble with a member having parameters ˜
θ
( ˜ m ) , to yield ˜ Θ , that satisﬁes
[Equation]
where τ > 0 , and the standard sum-to-one constraint is satisﬁed
[Equation]
and all other predictions are unaltered.
This results in the following constraints
[Equation]
The accuracy of the two ensembles Θ and ˜ Θ are the same from Eq.
(44), but the probabilities associated with those predictions cannot be the same from Eq.
(45), so both ensemble predictions cannot be calibrated, as assuming that the ensemble prediction for Θ is calibrated
[Equation]
( y ( i ) , ˆ y ( i ) E ) (46)
Hence there are multiple values of P (ˆ y ( n ) E | x ( n ) ; Θ ) for which all the models satisfy the topcalibration constraints, but these cannot all be consistent with Eq.
For the situation where there is no sample or model where ˆ y ( n ) E (cid:54)
=
ˆ y ( n ) ˜ m then the predictions for all models for all samples are the same as the ensemble prediction, so by deﬁnition there can be no performance gain.
A. G LOBAL G ENERAL C ALIBRATION AND T OP - LABEL C ALIBRATION
To demonstrate the differences between global top-label calibration and global calibration, a set of ensemble member predictions were generated using Algorithm 1, this ensures that the predictions are perfectly calibrated.
Since the member predictions are perfectly calibrated, the ensemble members will be globally calibrated.
Figure 3 (a) shows the performance in terms of ACE of the ensemble prediction as the value of (cid:15) increases, note when (cid:15)
=
1 this is a global calibration version of ACE.
It can be seen that as (cid:15) increases ACE decreases, and for the global case reduces to zero for the ensemble predictions as the theory states.
In terms of top-label calibration, as the ensemble members are perfectly calibrated, they will again be global top-label calibrated.
This is illustrated in Figure 3 (b) where ECE is zero for all ensemble members.
For top-label calibration the value of ECE does not decrease to zero as the (cid:15) → 1 , again as the theory states.
This is because the underlying probability regions associated with each of the members of the ensemble are different.
Hence, even for perfectly calibrated ensemble members, the ensemble prediction is not global top-label calibrated.
A. T OY D ATASETS
Example 1.
In this example, we show the mismatch between general calibration and top-label calibration which consider the different regions in Eq.
(3) and Eq.
Assuming p ( x ) ∝ 1 , the whole input space X is consisted of three regions R 1 , R 2 and R 3 , and (cid:90) x
[Equation] [Figure]
The corresponding system prediction ˆ P and the true distribution P is:
[Equation]
It can be veriﬁed that
[Equation]
However, when p = 0 . 4 , j = [Equation] Example 2.
This example shows that combination of calibrated members yields uncalibrated ensemble. Algorithm 1 generates the true data distribution p by sampling from a Dirichlet distribution with equal concentration parameters of 1.
To generate the member predictions, the N samples are randomly assigned to N/b bins with size of b .
In each bin, the member predictions ˆ p are equally the average of true data distribution of the associated samples. This ensures that for each region, Eq. (2) holds for each member. However, regions for different member are different due to the random assignment.
Therefore, the corresponding ensemble is not automatically calibrated.
Figure 3 shows that the ensemble is uncalibrated with ACE of 0.0697.
Example 3.
In this example, we show that for ﬁnite number of samples, the regions S pj ( θ , (cid:15) ) and
˜ S pj ( θ , (cid:15) ) derived from the samples is different from the theoretical regions, leading to difference between theoretical calibration error measures and the values estimated from the ﬁnite samples.
Algorithm 2 generates data with difference between ﬁnite sample-based calibration error and theoretical error.
The theoretical ACE of the predicted probabilities in Algorithm 2 is 932 .
However, the ﬁnite sample-based ACE is 0.
The true data distribution is P ( ω j | x ) = 14 , p ( x ) ∝ 1 .
The samples in D are assigned to bins of size 2.
The type of bin that a sample is assigned to determines the predicted probability of sample.
Considering each class ω j , there are three types of bins:
B p =1 : both samples belong to class ω j .
B p =0 .
5 : only one sample is class ω
j .
Result: { p ( i ) } Ni =1 , { ˆ p ( i ) , ˆ y ( i ) } Ni = b = 2 ; //bin size, number of samples in one bin;
K = 4 ; //number of classes;
M = 10 ; //number of members; N = 1000000 ; //number of samples; p ( i ) ∼ Dirichlet ( α = 1 ) , i = 1 , ..., N ; //true data distribution, sampled from Dirichlet distribution with equal concentration parameters of 1; I = [1 , 2 , ..., N ] ; // index vector; for m in [1 , ..., M ] do ˜
I ← shufﬂe( I ); for j in [1 , ..., (cid:100) N/b (cid:101) ] do B j = ( b ∗ ( j − 1) , min { b ∗ j, N } ] ; ˆ p B j = 1 b (cid:80) l ∈ B j p (˜ I l ) ; endfor i in [1 , ..., N ] do j = (cid:100) ib (cid:101) ; ˆ p (˜ I i ,m ) = ˆ p B j ; end end
B p =0 : both samples are not class ω j .
[Equation]
(52) then (cid:90)
[Equation]
[Equation]
(54) therefore, ACE
[Equation]
= [Equation] A. A DDITIONAL E XPERIMENTAL R ESULTS In this section, we show some comparison experiments to the empirical results in Section 4.3.
We conducted experiments on CIFAR-100 and CIFAR-10 dataset.
Table 3 and 4 display the performance of inidividual models, LeNet, DenseNet 100, DenseNet 121, and wide ResNet 28-10.
All systems are trained with data augmentation of random cropping and horizontal ﬂipping, and simple mean/std normalization.
The original training/test images in CIFAR datasets is 50,000/10,000.
We hold out 5,000 images from training set as validation set (10%) for temperature and combination weights optimization, this leads to a slight accuracy degradation compared to training with all 50,000 images.
In Section 4.3, we presented ensemble calibration on CIFAR-100.
The counterpart on CIFAR-10 is given in Table 5.
Other than separately speciﬁed, all sample based evaluation criteria in this paper use 15 bins following previous literature (Guo et al., 2017).
The temperatures in pre-, post- and dynamic post-combination modes are optimized on the validation set by minimizing ECE (Liang et al.,
Result: { ˆ p ( i ) , ˆ y
( i ) }
Ni =
b
= 2 ; //bin size, number of samples in one bin;
K = 4 ; //number of classes; g =
[ g 1 , ..., g N ] ; //vector of ground-truth labels , where g ∈ { 1 , ..., K } .
Numbers of labels for different classes are equal;
I =
[1 , 2 , ..., N ] ; // index vector; ˜
I ← shufﬂe( I ); for j in [1 , ..., (cid:100) N/b (cid:101) ] do
B j = ( b ∗ ( j − 1) , min { b ∗ j, N } ] ; ˆ p B j = 1 b (cid:80) l ∈ B
j
[ δ (1 , g ˜ I l ) , δ (2 , g ˜ I l ) , ..., δ ( K, g ˜ I l )] ; endfor i in [1 , ..., N ] do j = (cid:100) ib (cid:101) ; ˆ p (˜ I i ) = ˆ p B j ; end 2020), using SGD with learning rate of 0.1 for 400 iterations. It can be observed that combination of
DenseNet and ResNet improves the calibration performance, while combination of LeNet doesn’t help.
This is because LeNet is not as over-conﬁdent as DenseNet and ResNet (as shown in Figure 1). Hence the simple ensemble combination doesn’t help, but aggravates the calibration. The three temperature-based calibration methods effectively improve the system calibration performance on CIFAR-10 as well.
Table 6 gives the ensemble combination based on AUC weights (Zhong & Kwok, 2013).
The AUC weights are much even than the Max LL weights in Table 2. The structures combined are ﬁrst calibrated, nevertheless, applying post-combination calibration to the ensemble obtains gains. We evaluated post-combination for topology combination in Table 7. The post- and dynamic postcombination methods are applied to calibrate the topologies and the topology ensemble.
The dynamic temperature method shows clear advantage in obtaining calibrated ensemble of mutiple topologies.
[Table]
[Table]
[Table]
[Table]
[Table]
dyn.
[Table]
[Table]
dyn.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
- post 83.
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
post -
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
post post
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
- post
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
post -
[Table]