D ISTANCE -B ASED L EARNING FROM E RRORS FOR C ONFIDENCE C ALIBRATION
A BSTRACT
Deep neural networks (DNNs) are poorly calibrated when trained in conventional ways.
To improve conﬁdence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE).
DBLE bases its conﬁdence estimation on distances in the representation space.
In DBLE, we ﬁrst adapt prototypical learning to train classiﬁcation models.
It yields a representation space where the distance between a test sample and its ground truth class center can calibrate the model’s classiﬁcation performance.
At inference, however, these distances are not available due to the lack of ground truth labels.
To circumvent this by inferring the distance for every test sample, we propose to train a conﬁdence model jointly with the classiﬁcation model. We integrate this into training by merely learning from mis-classiﬁed training samples, which we show to be highly beneﬁcial for effective learning.
On multiple datasets and DNN architectures, we demonstrate that DBLE outperforms alternative single-model conﬁdence calibration approaches.
DBLE also achieves comparable performance with computationally-expensive ensemble approaches with lower computational cost and lower number of parameters.
I NTRODUCTION
Deep neural networks (DNNs) are being deployed in many important decision-making scenarios (Goodfellow et al., 2016).
Making wrong decisions could be very costly in most of them (Brundage et al., 2018) – it could cost human lives in medical diagnosis and autonomous transportation and it could cost signiﬁcant business losses in loan categorization and sales forecasting.
To prevent these from happening, it is strongly desired for a DNN to output conﬁdence estimations on its decisions.
In almost all of the aforementioned scenarios, detrimental consequences could be avoided by refraining from making decisions or consulting human experts, in the cases of decisions with insufﬁcient conﬁdence.
In addition, by tracking the conﬁdence in decisions, dataset shifts can be detected and developers can build insights towards improving the model performance.
However, conﬁdence estimation (also referred as ‘conﬁdence calibration’) is still a challenging problem for DNNs.
For a ‘well-calibrated’ model, predictions with higher conﬁdence should be more likely accurate.
However, as studied in (Nguyen et al., 2014; Guo et al., 2017), for DNNs with conventional (also referred as ‘vanilla’) training to minimize the softmax cross-entropy loss, the outputs do not contain sufﬁcient information for well-calibrated conﬁdence estimation.
Posterior probability estimates (e.g. the softmax outputs) can be interpreted as conﬁdence estimation, but it calibrates the decision quality poorly (Gal & Ghahramani, 2016)
– the conﬁdence tend to be large even when the classiﬁcation is inaccurate.
Therefore, it would be desirable if the training approach is redesigned to make its decision making more transparent, thus providing more information for accurate conﬁdence calibration.
In this paper, we propose a novel training method, Distance-based Learning from Errors (DBLE), towards better-calibrated DNNs.
DBLE learns a distance-based representation space through classiﬁcation and exploits distances in the space to yield well-calibrated classiﬁcation.
Our motivation is that a test sample’s location in the representation space and its distance to training samples should contain important information about the model’s decision-making process, which is useful for guiding conﬁdence estimation.
However, in vanilla training, since both training and inference are not based on distances in the representation space, they are not optimized to fulﬁll this goal.
Therefore, in DBLE, we propose to adapt prototypical learning for training and inference, to learn a distance-based representation space through classiﬁcation.
In this space, a test sample’s distance to its ground-truth class center can calibrate the model’s performance.
However, this distance cannot be calculated at inference directly, since the ground truth label is unknown.
To this end, we propose to train a separate conﬁdence model in DBLE, jointly with the classiﬁcation model, to estimate this distance at inference.
To train the conﬁdence model, we utilize the mis-classiﬁed training samples during the training of the classiﬁcation model.
We demonstrate that on multiple DNN models and datasets, DBLE achieves superior conﬁdence calibration without increasing the computational cost as ensemble methods.
R ELATED W ORK
Many studies (Nguyen et al., 2014; Guo et al., 2017) have shown that the classiﬁcation performance of DNNs are poorly-calibrated under vanilla training.
One direction of work to improve calibration is adapting regularization methods for vanilla training.
Such regularization methods are often originally proposed for other purposes.
For example, Label Smoothing (Szegedy et al., 2016), which is proposed for improving generalization, has empirically shown to improve conﬁdence calibration (M¨uller et al., 2019).
Mixup (Zhang et al., 2017), which is designed mostly for adversarial robustness, also improves conﬁdence calibration (Thulasidasan et al., 2019).
The beneﬁts from such approaches are typically limited, as the modiﬁed objective functions do not represent the goal of conﬁdence estimation.
To directly minimize a conﬁdence scoring objective, Temperature Scaling (Guo et al., 2017), modiﬁes learning by minimizing a Negative Log-Likelihood (Friedman et al., 2001) objective on a small training subset.
It yields better-calibrated conﬁdence by directly minimizing an evaluation metric of the calibration task.
However, since it requires training set to be split for the two tasks, classiﬁcation and calibration, it results in a trade-off – if more training data is used for calibration, then the model’s classiﬁcation performance would drop because less training data are left for classiﬁcation.
Bayesian DNNs constitute another direction of related work.
Bayesian DNNs aim to directly model the posterior distribution for a test sample given the training set (Rohekar et al., 2019).
In Bayesian
DNNs, since getting the posterior of large amount of parameters is usually intractable, different approximations are applied, such as Markov chain Monte Carlo (MCMC) methods (Neal, 2012) and variational Bayesian methods (Blundell et al., 2015; Graves, 2011).
Therefore, the calibration of Bayesian DNNs heavily depends on the degree of approximation and the pre-deﬁned prior in variational methods.
Moreover, in practice, Bayesian DNNs often yield prohibitively slow training due to the slow convergence of the posterior predictive estimation.
(Lakshminarayanan et al., 2017).
Recent work on Bayesian DNNs (Heek & Kalchbrenner, 2019) reports that the training time of their model is an order of magnitude higher than vanilla training.
As Bayesian-inspired approaches, Monte
Carlo Dropout (Gal & Ghahramani, 2016) and Deep Ensembles (Lakshminarayanan et al., 2017) are two widely-used methods since they only require minor changes to the classic training method of
DNNs and are relatively faster to train.
L EARNING A DISTANCE - BASED SPACE FOR CONFIDENCE CALIBRATION
We ﬁrst introduce how DBLE applies prototypical learning to train DNNs for classiﬁcation.
Then, we explain that under this new training method, a distance-based representation space can be achieved to beneﬁt conﬁdence calibration.
3. P ROTOTYPICAL L EARNING FOR C LASSIFICATION
DBLE bases the training of the classiﬁcation model on prototypical learning (Snell et al., 2017).
Prototypical learning is originally proposed to learn a distance-based representation space for few-shot learning (Ravi & Larochelle, 2016; Oreshkin et al., 2018;
Xing et al., 2019).
In prototypical learning, both training and prediction solely depend on the distance of the samples to their corresponding class ‘prototypes’ (referred as ‘class centers’ in the paper) in the representation space.
It trains the model with the goal of minimizing the intra-class distances, whereas maximizing inter-class distances such that related samples are clustered together.
We adapt this training method for classiﬁcation training in
DBLE.
Subsequent subsections describe the two main components of prototypical training: episodic training and prototypical loss for classiﬁcation.
Later, we explain how to run inference at test time and why it leads to a representation space that beneﬁts conﬁdence calibration.
3.1. E PISODIC TRAINING
Vanilla DNN training for classiﬁcation is based on variants of mini-batch gradient descent (Bottou, 2010).
Episodic training, on the other hand, samples K -shot, N -way episodes at every update (as opposed to sampling a batch of training data).
An episode e is created by ﬁrst sampling N random classes out of M total classes 1 , and then sampling two sets of training samples for these classes: (i) the support set S e
= { ( s j , y j ) } N × K j =1 containing K examples for each of the N classes and (ii) the query set Q e = { ( x i , y i ) }
Qi =1 containing different examples from the same N classes.
[Equation]
[Figure]
[Equation]
g
[Equation] [Figure]
At episode e , the model is updated to map the query x i to its correct label y i , with the help of support set S e .
The model is a function parameterized by θ and the loss is negative log-likelihood of the ground-truth class label of each query sample given the support set according to Eq.
3.1. P ROTOTYPICAL LOSS
How does the support set S e help the classiﬁcation of query samples at episode e ?
We ﬁrstly employ the classiﬁcation model, f : R n v → R n c to encode inputs in a representation space, where θ are trainable parameters.
Prototypical training then uses the support set
S e to compute the center for each class (in the sampled episode).
The query samples are classiﬁed based on their distance to each class center in the representation space.
For every episode e , each center c k is computed by averaging the representations of the support samples from class k :
[Equation]
where S ke ⊂ S e is the subset of support samples belonging to class k .
The prototypical loss calculates the predictive label distribution of query x i based on its distances to the N centers:
[Equation]
where h i is the representation of x i in the space:
[Equation]
The model is trained by minimizing Eq.
1 with p ( y i | x i , S e ; θ ) in Eq.
Through this training, in the representation space that we calculate h i and class centers, the inter-class distances are maximized and the intra-class distances are minimized.
Therefore, training samples belonging to the same class are clustered together and clusters representing different classes are pushed apart.
3.1. I NFERENCE AND THE TEST SAMPLE ’ S DISTANCE TO ITS GROUND - TRUTH CLASS CENTER
At inference, we calculate the center of every class c using the training set, by averaging the representations of all corresponding training samples:
[Equation]
where T k is the set of all training samples belonging to class k .
Then, given a test sample x t in the test set D test = { ( x t , y t ) } N test t =1 (where y t is the ground-truth label), the distances of its representation h t (Eq. 4) to all class centers are calculated.
The prediction of the label of x t is based on these distances:
[Equation]
where M consists of all classes in the training set.
In other words, x t is assigned to the class with the closest center in the representation space.
Consequently, for a test sample ( x t , y t ) , if at inference h t is too far away from its ground-truth class center c testy t , it is likely to be mis-classiﬁed.
Therefore, in this distance-based representation space, a test sample’s distance to its ground-truth class center, is a strong signal indicating how well the model would perform on this sample.
In the next section, we empirically show it under DBLE and compare with other methods.
3. D ISTANCE TO THE GROUND -
[Figure] CLASS CENTER CALIBRATES CLASSIFICATION PERFORMANCE IN DBLE
[Figure]
We design an experiment to show that in the space learned by prototypical learning of DBLE, the model’s performance given a test sample can be estimated by the test sample’s L2-distance to its ground-truth class center.
On the other hand, other intuitive measures such as L2-distance to the predicted class center in prototypical learning, L2-distance to the predicted class center in vanilla training or L2-distance to the ground-truth center in vanilla training, can’t calibrate performance very well.
Here, we describe the empirical observations on DBLE’s classiﬁcation model with prototypical learning, to motivate the conﬁdence modeling of DBLE before we explain the method.
In the learned representation space (either the output space of a model from vanilla training or prototypical learning), for every sample ( x t , y t ) in the test set D test =
{ ( x t , y t ) } N test t =1 , we calculate its L2-distance to its ground-truth class center c testy t as:
[Equation]
where c testy t is given by Eq.
f θ is the trained classiﬁcation model and d ( , ) is L2-distance.
We can also calculate x t ’s distance to its predicted class center c testy (cid:48) t where y (cid:48) t is the predicted label, in both
DBLE and vanilla training as comparison.
We denote x t ’s distance to its predicted class center as d (cid:48) t
= d ( f θ ( x t ) , c testy (cid:48) t ) .
We then sort { d t } N test t =1 or { d (cid:48) t } N test t =1 in ascending order and partition them in
I equally-spaced bins.
For every bin, we calculate the model’s classiﬁcation accuracy of the test samples lying in the bin.
Then, we plot the test accuracy curve as the distance increases.
The curve shows the the relationship between the distance and the model’s performance on x t , illustrating how well the calculated distance can act as conﬁdence score, calibrating the model’s prediction.
As shown in Fig.
2, d t of DBLE, which is the distance of a test sample x t to its ground-truth class center, calibrates the model’s performance the best.
The curve of d t under DBLE’s prototypical learning, despite its slight oscillation near 0 in the end, decreases monotonically to almost zero.
It indicates that farther away the sample is from its ground-truth class center, more poorly the model performs on it.
These results suggest that d t can be used as a gold conﬁdence measure, calibrating the quality of the model’s decision of x t for a classiﬁcation model under prototypical learning.
This beneﬁt of DBLE’s prototypical learning is due to the distance-based training and inference – if a test sample is farther away from its ground-truth class center, it is more likely to be mis-classiﬁed as other classes.
Although we have observed this gold calibration in DBLE enabled by prototypical learning, it requires access to the ground-truth labels of test samples, d t .
At inference however, we do not have access to the ground-truth label of x t at inference and d t cannot be directly calculated.
Therefore, we propose to use a separate conﬁdence model to estimate d t , trained jointly with the classiﬁcation training of
DBLE.
It regresses d t by learning from the mis-classiﬁed training samples in DBLE’s classiﬁcation training, described next.
C ONFIDENCE M ODELING BY L EARNING FROM E RRORS
In the classiﬁcation model learned by DBLE, a test sample x t ’s L2-distance d t to its ground-truth class center (Eq. 7), is highly-calibrated with the model’s performance on it, as described in Sec. 3.
However, d t cannot be directly computed without ground-truth labels, which is the case at inference.
Therefore, we introduce a conﬁdence model parameterized by φ , to learn to estimate d t jointly with the training of classiﬁcation in DBLE.
To train φ , a straightforward option would be using the distances for all training samples, { ( x i , d i ) } |T
| i =1 .
Through this way, φ can be trained to give an estimate of d t for the test sample x t at inference.
However, correctly-classiﬁed samples constitute the vast majority during training, especially considering that state-of-the-art DNNs yield much lower training errors compared to test errors (Neyshabur et al., 2014).
Therefore, if all data is used, training of g φ would be dominated by the small distances of the correctly-classiﬁed samples, which would make it harder for g φ to capture the larger d t for the minor mis-classiﬁed samples.
Moreover, given that we choose g φ as a simple MLP with limited capacity for fast training, it becomes more challenging to capture larger d t from the incorrectly-classiﬁed samples if all training samples are used (i.e. the conﬁdence model would underﬁt mis-classiﬁed training data).
Therefore, we propose to track all training samples that are mis-classiﬁed in episode e during the training of the classiﬁcation model.
We save them in M e = { ( x s , y s ) , where y (cid:48) s (cid:54) =
y s } for each episode e to train g φ .
In our ablation studies in Sec.
5.4, we demonstrate the importance of learning from mis-classiﬁed training samples vs. learning from all.
Next, we introduce the training procedure of g φ with mis-classiﬁed samples for estimation of d t at inference for test sample t .
We train g φ by sampling from the isotropic Gaussian distribution
N ( h s , diag ( σ s (cid:12) σ s )) , where the standard deviation σ s is g φ
’s output given mis-classiﬁed training sample x s .
g φ is trained to output a larger σ s for x s with a larger d s .
g φ takes the representation h s (calculated with Eq. 4) of a mis-classiﬁed training sample x s in M e = { ( x s , y s ) } as input, and outputs σ s as, σ s =
g φ ( h s ) .
To train g φ , we ﬁrstly sample another representation z s for x s from the isotropic Gaussian distribution parameterized by h s and σ s , z s ∼ N ( h s , diag ( σ s (cid:12) σ s )) .
Then, we optimize g φ based on the prototypical loss with z s , using z s ’s predictive label distribution:
[Equation]
When updating φ with the prototypical loss given z s , it encourages a larger σ s when the d s of x s is larger.
This is because when h s is ﬁxed for x s in the space, maximizing Eq.
9 forces z s to be as close to its ground-truth class center as possible.
Given z s is sampled from N ( h s , diag ( σ s (cid:12) σ s )) , if h s is farther away from its ground-truth center (which is the case of mis-classiﬁed training samples), then it requires a larger σ s for z s to go close to it.
Fig.
3 visualizes how σ changes before vs. after updating φ .
The training of DBLE is described in Fig.
1 and in Algorithm 1 in the Appendix A.1.
Note that in order to make sampling of z s differentiable, we use the reparameterization trick (Kingma & Welling, 2013).
At inference, for every test sample x t , we make prediction y (cid:48) t with h t (see Eq. 6).
While for conﬁdence estimation of the predictions, we take advantage of σ t .
After getting h t and σ t , we sample multiple z ut from N ( h t , diag ( σ t (cid:12) σ t )) and average their predictive label distributions as conﬁdence [Figure] estimation:
[Equation]
U is the total number of representation samples z t .
ˆ p ( y (cid:48) t | x t ; φ ) is used as the conﬁdence score calibrating the prediction y (cid:48) t .
Through this way, for a test sample farther away from its ground-truth class center (which means they are more likely to be mis-classiﬁed), the model will add more randomness to the representation sampling since its estimated variance from g φ is large.
More randomness in the softmax input space leads to a lower expected softmax output (Gal & Ghahramani, 2016), which is the conﬁdence score ˆ p ( y (cid:48)
t | x t ; φ ) .
E XPERIMENTS
In this section, we ﬁrst compare DBLE to recent baselines on conﬁdence calibration of DNNs.
We conduct experiments on a variety of data sets and network architectures and evaluate DBLE’s performance on two most commonly used metrics for conﬁdence calibration: Expected Calibration
Error (ECE) (Naeini et al., 2015) and Negative Log Likelihood (NLL) (Friedman et al., 2001).
Results show that DBLE outperforms single-modal baselines on conﬁdence calibration in every scenario tested.
We then conduct ablation study to verify the effectiveness of the two main components of
DBLE.
Implementation, training details, and the details of the baselines are described in Appendix.
5. E XPERIMENTAL S ETUP
Baselines.
We compare our method with 5 baselines that use a single DNN: vanilla training, MCDropout (Gal & Ghahramani, 2016), Temperature Scaling (Guo et al., 2017), Mixup (Thulasidasan et al., 2019), Label Smoothing (Szegedy et al., 2016) and TrustScore (Jiang et al., 2018).
We also compare DBLE with Deep Ensemble (Lakshminarayanan et al., 2017) with 4 types of DNNs.
Datasets and Network Architectures.
We conduct experiments on various combinations of datasets and architectures: MLP on MNIST (LeCun et al., 1998), VGG-11 (Simonyan & Zisserman, 2014) on
CIFAR-10 (Krizhevsky et al., 2009), ResNet-50 (He et al., 2016) on CIFAR-100 (Krizhevsky et al., 2009) and ResNet-50 on Tiny-ImageNet (Deng et al., 2009).
Evaluation Metrics.
We evaluate DBLE on model calibration with Expected Calibration Error (ECE) and Negative Log Likelihood (NLL).
ECE approximates the expectation of the difference between accuracy and conﬁdence.
It partitions the conﬁdence estimations (the likelihood of the predicted label p
( y (cid:48) t | x t ) ) of all test samples into L equally-spaced bins and calculates the average conﬁdence and accuracy of test samples lying in each bin
I l :
[Equation]
where y (cid:48) t is the predicted label of x t .
NLL averages the negative log-likelihood of all test samples:
[Equation]
[Table]
5. R ESULTS
Table 1 shows the results.
In every scenario tested, DBLE is comparable with vanilla training in test accuracy – applying prototypical learning for classiﬁcation problems does not hurt generalization of models on classiﬁcation.
On conﬁdence calibration, DBLE performs all single-modal baselines on every scenario tested on both ECE and NLL.
Moreover, our method reaches comparable results with
Deep Ensemble of 4 networks with a smaller time complexity and parameter size.
For MNIST-MLP,
CIFAR10-VGG11 and CIFAR100-ResNet50, our method outperforms Deep ensemble (4 networks) on ECE.
Among other baselines, Temperature Scaling performs the best in every scenario, which is because Temperature Scaling directly optimizes NLL on the small sub-training set.
MC-dropout gives the least improvement on conﬁdence calibration, especially on NLL.
The potential reason for
MC-dropout’s limited improvement on NLL can be that when applying dropout at inference, it adds similar level of randomness on mis-classiﬁed samples and correctly classiﬁed samples.
Therefore, the predictive likelihood of the correct labels becomes smaller in general, which leads to worse NLL.
Compared to Temperature Scaling, DBLE decouples classiﬁcation and calibration, therefore achieves better calibration without sacriﬁcing classiﬁcation performance.
Compared to MC-dropout, our model is trained to add randomness on predictive distributions for mis-classiﬁed samples speciﬁcally, thus achieving signiﬁcantly better calibration.
5. C OMPUTATIONAL C OST
DBLE adds extra training time complexity and trainable parameters compared with vanilla training.
However, compared with Deep Ensembles and other Bayesian methods, DBLE’s cost is signiﬁcantly less.
For training time complexity, although DBLE requires two forwards passes for f θ at each update step (see Algorithm 1), its actual training time is less than twice of vanilla training since the number of iterations required until convergence is smaller.
Empirically we observe that DBLE’s total training time on CIFAR10-VGG11 is 1.4x of vanilla training and on CIFAR100-ResNet50 is 1.7x of vanilla training.
While Deep Ensemble of 4 networks takes 4x vanilla training time.
DBLE adds extra trainable parameters φ compared with vanilla training.
φ is the parameters of a MLP in practice.
It’s size is usually 1% − 3% of the classiﬁcation model, while ensembling 4 DNNs in Deep Ensemble increases the size of trainable parameters to 4x of vanilla training. [Figure]
[Table]
5. A LGORITHM A NALYSIS AND A BLATION S TUDY
DBLE alleviates the NLL overﬁtting problem.
NLL overﬁtting problem of vanilla training has been observed in (Guo et al., 2017) – after annealing the learning rate, as the test accuracy goes up, the model overﬁts to NLL score (test NLL starts to increase instead of decreasing or maintaining ﬂat).
Fig.
4 shows that this phenomenon is alleviated by DBLE.
Under vanilla training, the model starts overﬁtting to NLL after the ﬁrst learning rate annealing.
DBLE on the other hand, decreases and maintains its test NLL every time after learning rate annealing.
We see the same trend for test ECE as well – with vanilla training the model overﬁts to ECE after learning rate annealing while DBLE decreases and maintains its test ECE.
Distance-based representation space and learning from errors are both essential for DBLE.
We conduct ablation studies to verify the effectiveness of the two main design choices of our model, the distance-based space and learning calibration from training errors.
Table 2 shows the results.
In “Learning with errors in Vanilla training”, we conduct calibration learning with errors in vanilla training.
In other words, instead of updating φ by maximizing Eq.
9, we update φ by maximizing the softmax likelihood with z as logits and f θ is updated with vanilla training.
In “DBLE with calibration learning using all samples”, we update φ with all training samples by maximizing Eq. 9.
We can see from the results that ﬁrstly, learning to calibrate with errors also helps vanilla training.
It improves calibration in vanilla training slightly since it to some extent, introduces more randomness in the decision making process for misclassiﬁed samples.
However, the improvement is very small without the distance-based space.
Secondly, we notice that in DBLE if we learn calibration with all training samples, it signiﬁcantly decreases DBLE’s performance on calibration.
The potential reason is the model’s underﬁtting to mis-classiﬁed training samples in learning to estimate the distance.
Auxiliary analysis on these two can be found in Appendix A.3.
C ONCLUSION
Conﬁdence calibration of DNNs, which has signiﬁcant practical impacts, still remains an open problem.
In this paper, we have proposed Distance-Based Learning from Errors (DBLE) to try to solve this problem.
DBLE starts with applying prototypical learning to train the DNN classiﬁcation model.
It results in a distance-based representation space in which the model’s performance on a test sample is calibrated by the sample’s distance to its ground-truth class center.
DBLE then trains a conﬁdence model with mis-classiﬁed training samples for conﬁdence estimation.
We have empirically shown the effectiveness of DBLE on various classiﬁcation datasets and DNN architectures.
A CKNOWLEDGMENTS
Discussions with Kihyuk Sohn are gratefully acknowledged
A. I MPLEMENTATION AND T RAINING D ETAILS OF DBLE AND B ASELINES
For a fair comparison with baseline methods, in every scenario tested, the network architecture is identical for DBLE and all other baselines.
As regularization techniques such as BatchNorm, weight decay are observed to affect conﬁdence calibration (Guo et al., 2017), we also keep them constant while comparing to baselines in every scenario.
All models are trained with stochastic gradient descent with momentum (Sutskever et al., 2013).
We use an initial learning rate of 0.1 and a ﬁxed momentum coefﬁcient of 0.9 for all methods tested.
The learning rate scheduling is tuned according to classiﬁcation performance on validate set.
All other hyperparameters of classiﬁcation models for baselines are also chosen based on accuracy on validation set.
There are several unique hyper-parameters in DBLE.
We describe how we choose them in the following paragraph.
In DBLE, we stop the training when the classiﬁcation model converges.
The conﬁdence model g φ is a two-layer MLP with Dropout (Srivastava et al., 2014) added in between.
we use ReLU non-linearity (Glorot et al., 2011) for the MLP.
We ﬁx the dropout rate as 0.5.
The
N (number of classes), K (number of shots) and the batch-size of the query set for every episode in DBLE’s are ﬁrstly tuned according to the classiﬁcation performance on validation set to reach comparable performance with vanilla training.
Then the N , K and the batch-size of the query set for every episode are ﬁne-tuned according to the DBLE’s calibration performance on validation set.
At inference, we ﬁx the number of representation sampling U in Eq.
10 as 20.
This is because we empirically observed that a U larger than 20 doesn’t give performance improvements.
We set the number of bins L in Eq.
11 as 15 following (Guo et al., 2017).
A. T RAINING THE CONFIDENCE WITH ALL TRAINING SAMPLES VS . WITH MIS - CLASSIFIED TRAINING SAMPLES
In addition to the ﬁnal results we report in the ablation study (the last two rows of Table 2) comparing training the conﬁdence with all training samples vs. with mis-classiﬁed training samples, we also checked statistics of σ , which is the standard deviation predicted by the conﬁdence model.
For
CIFAR-10, the mean of σ of the model trained with all samples is 0.203, while that of the model trained with mis-classiﬁed training samples is 0.740.
This suggests that if we train the conﬁdence model with all training samples, the model outputs for test samples are indeed smaller in general, which is aligned with our intuition.
The histogram of σ under these two methods are shown in
Figure 5.