ESD: E XPECTED S QUARED D IFFERENCE AS A T UNINGF REE T RAINABLE C ALIBRATION M EASURE
A BSTRACT
Recent studies have shown that modern neural networks tend to be poorly calibrated due to over-confident predictions.
Traditionally, post-processing methods have been used to calibrate the model after training.
In recent years, various trainable calibration measures have been proposed to incorporate them directly into the training process.
However, these methods all incorporate internal hyperparameters, and the performance of these calibration objectives relies on tuning these hyperparameters, incurring more computational cost as the size of neural networks and datasets become larger.
As such, we present Expected Squared
Difference (ESD), a tuning-free (i.e., hyperparameter-free) trainable calibration objective loss, where we view the calibration error from the perspective of the squared difference between two expectations.
With extensive experiments on several architectures (CNNs, Transformers) and datasets, we demonstrate that (1) incorporating ESD into the training improves model calibration in various batch size settings without the need for internal hyperparmeter tuning, (2) ESD yields the best calibrated results compared with previous approaches, and (3) show that ESD drastically improves the computational cost required for calibration during training due to the absence of internal hyperparamater.
The code is publicly accessible at https: github. hee-suk-yoon/
ESD.
I NTRODUCTION
The calibration of a neural network measures the extent to which the predictions it makes align with the true probability distribution.
Possessing this property becomes especially important in real-world applications, such as identification (Kim & Yoo, 2017;
Yoon et al., 2022), autonomous driving (Bojarski et al., 2016;
Ko et al., 2017), and medical diagnosis (Kocbek et al., 2020; Pham et al., 2022), where uncertainty-based decisions of the neural network are crucial to guarantee the safety of the users.
However, despite the success of modern neural networks in accurate classification, they are shown to be poorly calibrated due to the tendency of the network to make predictions with high confidence regardless of the input.
(i.e., over-confident predictions)
(Guo et al., 2017).
Traditionally, post-processing methods have been used, such as temperature scaling and vector scaling (Guo et al., 2017), to calibrate the model using the validation set after the training by adjusting the logits prior to the final softmax layer.
Recently, various trainable calibration objectives have been proposed, such as MMCE (Kumar et al., 2018) and SB-ECE (Karandikar et al., 2021), which are added to the loss function as a regularizer to jointly optimize accuracy and calibration during training.
A key advantage of calibration during training is that it is possible to cascade post-processing calibration methods after training to achieve even better-calibrated models.
Unfortunately, these existing approaches introduce additional hyperparameters in their proposed calibration objectives, and the performance of the calibration objectives is highly sensitive to these design choices.
Therefore these hyperparameters need to be tuned carefully on a per model per dataset basis, which greatly reduces their viability for training on large models and datasets.
∗ Equal contribution † Corresponding Author
To this end, we propose Expected Squared Difference (ESD), a trainable calibration objective loss that is hyperparameter-free.
ESD is inspired by the KS-Error (Gupta et al., 2020), and it views the calibration error from the perspective of the difference between two expectations.
In detail, our contributions can be summarized as follows:
• We propose ESD as a trainable calibration objective loss that can be jointly optimized with the negative log-likelihood loss (NLL) during training.
ESD is a binning-free calibration objective loss, and no additional hyperparamters are required.
We also provide an unbiased and consistent estimator of the Expected Squared Difference, and show that it can be utilized in small batch train settings.
• With extensive experiments we demonstrate that across various architectures (CNNs & Transformers) and datasets (in vision & NLP domains), ESD provides the best calibration results when compared to previous approaches.
Calibrations of these models are further improved by post-processing methods.
• We show that due to the absence of an internal hyperparameter in ESD that needs to be tuned, it shows a drastic improvement compared to previous calibration objective losses with regards to the total computational cost for training.
The discrepancy in computational cost between ESD and tuning-required calibration objective losses becomes larger as the model complexity and dataset size increases.
R ELATED W ORK
Calibration of neural networks has gained much attention following the observation from Guo et al.
(2017) that modern neural networks are poorly calibrated.
One way to achieve better calibration is to design better neural network architectures tailored for uncertainty estimation, e.g., Bayesian Neural
Networks (Blundell et al., 2015; Gal & Ghahramani, 2016) and Deep Ensembles (Lakshminarayanan et al., 2017).
Besides model design, post-processing calibration strategies have been widely used to calibrate a trained machine learning model using a hold-out validation dataset.
Examples include temperature scaling (Guo et al., 2017) which scales the logit output of a classifier with a temperature parameter; Platt scaling (Platt, 1999) which fits a logistic regression model on top of the logits; and
Conformal prediction (Vovk et al., 2005; Lei et al., 2018) which uses validation set to estimate the quantiles of a given scoring function.
Other post-processing techniques include histogram binning (Zadrozny & Elkan, 2001), isotonic regression (Zadrozny & Elkan, 2002), and Bayesian binning into quantiles (Pakdaman Naeini et al., 2015).
Our work focuses on trainable calibration methods which train neural networks using a hybrid objective, combining a primary training loss with an auxiliary calibration objective loss.
In this regard, one popular objective is Maximum Mean Calibration Error (MMCE) (Kumar et al., 2018), which is a kernel embedding based measure of calibration that is differentiable and therefore suitable as a calibration objective loss.
Moreover, Karandikar et al.
(2021) proposes a trainable calibration objective loss, SB-ECE and S-AvUC which softens previously defined calibration measures.
P ROBLEM S ETUP 3. C ALIBRATION E RROR AND M ETRIC
Let us first consider an arbitrary neural network as f θ : D → [0 , 1] C with network parameters θ , where
D is the input domain and C is the number of classes in the multiclass classification task.
Furthermore, we assume that the training data, ( x i , y i ) ni
=1 are sampled i.i.d.
from the joint distribution P ( X , Y ) (here we use one-hot vector for y ).
Here, Y =
( Y 1 , ..., Y C ) , and y =
( y 1 , ..., y C ) are samples from this distribution.
We can further define a multivariate random variable Z = f θ ( X ) as the distribution of the outputs of the neural network.
Similarly, Z = ( Z 1 , ..., Z C ) , and z = ( z 1 , ..., z C ) are samples from this distribution.
We use ( z K,i , y K,i ) to denote the output confidence and the one-hot vector element associated with the K -th class of the i -th training sample.
Using this formulation, a neural network is said to be perfectly calibrated for class K, if and only if
[Equation]
Intuitively, Eq.
(1) requires the model accuracy for class K to be z K on average for the inputs where the neural network produces prediction of class K with confidence z K .
In many cases, the research of calibration mainly focuses on the calibration of the max output class that the model predicts.
Thus, calibration error is normally reported with respect to the predicted class (i.e., max output class) only.
As such, k will denote the class with maximum output probability.
Furthermore, we also write I ( · ) as the indicator function which returns one if the Boolean expression is true and zero otherwise.
With these notations, one common measurement of calibration error is the difference between confidence and accuracy which is mathematically represented as (Guo et al., 2017)
[Equation]
To estimate this, the Expected Calibration Error (ECE) (Naeini et al., 2015) uses B number of bins with disjoint intervals B j = ( jB , j +1 B ] , j = 0 , 1 , ..., B − 1 to compute the calibration error as follows:
[Equation]
3. C ALIBRATION D URING T RAINING
Post-processing method and calibration during training are the two primary approaches for calibrating a neural network.
Our focus in this paper is on the latter, where our goal is to train a calibrated yet accurate classifier directly.
Note that calibration and predictive accuracy are independent properties of a classifier.
In other words, being calibrated does not imply that the classifier has good accuracy and vice versa.
Thus, training a classifier to have both high accuracy and good calibration requires jointly optimizing a calibration objective loss alongside the negative log-likelihood (NLL) with a scaling parameter λ for the secondary objective:
[Equation]
3.2. E XISTING T RAINABLE C ALIBRATION O BJECTIVES NEED T UNING
Kumar et al.
and
Karandikar et al.
(2021) suggested that the disjoint bins in ECE can introduce discontinuities which are problematic when using it as a calibration loss in training.
Therefore, additional parameters were introduced for the purpose of calibration during training.
For example,
Kumar et al.
(2018) proposed Maximum Mean Calibration Error (MMCE), where it utilizes a
Laplacian Kernel instead of the disjoint bins in ECE.
Furthermore, Karandikar et al.
(2021) proposed soft-binned ECE (SB-ECE) and soft AvUC, which are softened versions of the ECE metric and the AvUC loss (Krishnan & Tickoo, 2020), respectively.
All these approaches address the issue of discontinuity which makes the training objective differentiable.
However they all introduce additional design choices - MMCE requires a careful selection of the kernel width ϕ , SB-ECE needs to choose the number of bins M and a softening parameter T , and S-AvUC requires a user-specified entropy threshold κ in addition to the softening parameter T .
Searching for the optimal hyper-parameters can be computationally expensive especially as the size of models and dataset become larger.
3.2. C ALIBRATION D URING T RAINING S UFFERS FROM O VER - FITTING P ROBLEM
NLL loss is known to implicitly train for calibration since it is a proper scoring rule, so models trained with NLL can overfit in terms of calibration error (Mukhoti et al., 2020; Karandikar et al., 2021).
Figure 1 provides an example of an accuracy curve and its corresponding ECE curve for a model trained with NLL loss.
We see that the model is overfitting in terms of both accuracy and ECE, which causes the gap between train and test ECE to become larger during training.
As such, adding a calibration objective to the total loss will not be able to improve model calibration as it mainly helps improve calibration of the model with respect to the training data only.
Kumar et al.
(2018) tackle this issue by introducing a weighted version of their calibration objective loss.
They consider using larger weights to incorrect prediction samples after observing that the fraction of incorrect to the correct samples on the training data is smaller than that of the validation and test data.
Instead of changing the calibration objective function itself, Karandikar et al.
(2021) introduced a new training scheme called “interleaved training”, where they split the training data further and dedicate a small portion of it to optimize the calibration objective loss.
We will follow this strategy and introduce the technical details in the later sections.
ESD: E XPECTED S QUARED D IFFERENCE
We propose Expected Squared Difference (ESD) as a tuning-free (i.e., hyperparmeter-free) calibration objective.
Our approach is inspired by the viewpoint of calibration error as a measure of distance between two distributions to obtain a binning-free calibration metric (Gupta et al., 2020).
By using this approach, there is no need to employ kernels or softening operations to handle the bins within calibration metrics, such as ECE, in order to make them suitable for training.
In particular, we consider calibration error as the difference between the two expectations.
We first start with the definition of perfect calibration in Eq.
[Equation]
by Bayes rule)
[Equation] [Equation]
Now considering the accumulation of the terms on both sides for arbitrary confidence level α ∈ [0 , 1] , the perfect calibration for an arbitrary class k can now be written as:
[Equation]
This allows us to write the difference between the two expectations as
[Equation]
and d k ( α ) = 0 , ∀ α ∈ [0 , 1]
if and only if the model is perfectly calibrated for class k .
Since this has to hold ∀ α ∈ [0 , 1] , we propose ESD as the expected squared difference between the two expectations:
[Equation]
Due to the close relationship between d k ( α ) and calibration of a neural network, the difference between an uncalibrated and a calibrated neural network can be clearly observed using E Z ′ k
[ d k ( Z ′ k ) 2 ] as visually shown in Appendix A.
This new metric is a good measure of calibration since ESD =
iff.
the model is perfectly calibrated as shown in the following theorem.
Theorem 1.
E Z ′ k
[ E 2 Z k ,Y k
[ I ( Z k ≤ Z ′ k )
( I ( Y k = 1) − Z k )]]
=
iff.
the model is perfectly calibrated.
Proof.
Since d k ( Z ′ k )
=
E 2 Z k ,
Y k
[ I ( Z k ≤ Z ′ k )
( I ( Y k = 1) − Z k )] is a non-negative random variable induced by Z ′ k , E Z ′ k
[ d k ( Z ′ k ) 2 ] =
iff.
P ( d k ( Z ′ k ) = 0)
= 1 .
Furthermore,
P ( d k ( Z ′ k ) = 0) =
⇐⇒ d k ( α ) =
∀ α ∈
I where I is the support set of Z ′ k .
Thus,
[Equation]
From lemma 1.1 below, we have d k ( α ) =
∀ α ∈ [0 , 1] iff.
d k ( α ) = 0 ∀ α s.t.
P ( Z ′ k
= α ) ̸
= 0 .
Consequently, E Z ′ k
[ E 2 Z k ,Y k
[ I ( Z k ≤ Z ′ k )
( I ( Y k = 1) − Z k )]]
=
iff.
the model is perfectly calibrated.
Lemma 1.1 Let I be the support set of random variable Z k , then d k ( α ) =
∀ α ∈
I iff.
d k ( α ) =∀ α ∈ [0 , 1] .
Proof.
The backward direction result is straight-forward, so we only prove for the forward direction:
If d k ( α ) = 0 ∀ α ∈
I , then d k ( α ) =
∀ α ∈ [0 , 1] .
For arbitrary α ′ ∈ I c , let α = arg min a ∈I | a − α ′
|
and α < α ′ .
We then have,
[Equation]
4. A N ESTIMATOR FOR ESD
In this section, we use ( z k,i , y k,i ) to denote the output confidence and the one-hot vector element associated with the k -th class of the i -th training sample respectively.
As the expectations in the true
Expected Squared Difference (Eq.
(8)) are intractable, we propose a Monte Carlo estimator for it which is unbiased.
A common approach is to use a naive Monte Carlo sampling with respect to both the inner and outer expectations to give the following:
[Equation]
where ¯ g i =
N
[Equation]
N (cid:88) j =1 j ̸ = i g ij and g ij =
I ( z k,j ≤ z k,i )
[ I ( y j = k ) − z k,j ] .
However, Eq.
(9) results in a biased estimator that is an upper bound of the true Expected Squared
Difference.
To account for the bias, we propose the unbiased and consistent estimator (proof in
Appendix B) of the true Expected Squared Difference 1 ,
[Equation]
where S 2 g i =
[Equation]
N
[Equation]
j =
[Equation]
g
[Equation]
4. I NTERLEAVED T RAINING
Negative log-likelihood (NLL) has be shown to greatly overfit to ECE of the data it is trained on.
Thus, training for calibration using the same data for the NLL has limited effect on reducing the calibration error of the model.
Karandikar et al.
(2021) proposed ”Interleaved Training” where they split the train set into two subsets - one is used to optimize the NLL and the other is used to optimize the calibration objective.
Following this framework, let D train denote the entire train set.
We separate the train set into two subsets - D ′ train and D ′ cal .
The joint training of ESD with NLL becomes,
[Equation]
[Equation]
With this training scheme, NLL is optimized on D ′ train and ESD is optimized on D ′ cal .
This way,
Karandikar et al.
(2021) showed we can avoid minimizing ECE that is already overfit to the train set.
E XPERIMENTAL S ETTING 5. D ATASETS AND M ODELS
Image Classification For image classification tasks we use the following datasets:
• MNIST (Deng, 2012): 54,000/6,000/10,000 images for train/validation/test split was used.
We resized the images to (32x32) before inputting to
the
network.
• CIFAR10 & CIFAR100 (Krizhevsky et al.
, a;b): 45,000/5,000/10,000 images for train/validation/test split was used.
Used random cropping of 32 with padding of 4.
Normalized each RGB channel with mean of 0.5 and standard deviation of 0.5.
• ImageNet100 (Deng et al., 2009):
A subset dataset from ImageNet Large Scale Visual Recognition Challenge 2012 with 100 classes.
Since the labels of test sets are unavailable, we use the official validation set as the test set, and we dedicate 10% of the training data as the validation set for the experiments (
i.e., 117,000/
000 split for train/val/test set).
Natural Language Inference (NLI) NLI is a task in Natural Language Processing where it involves classifying the inference relation (entailment, contradiction, or neutral) between two texts (MacCartney & Manning, 2008).
For NLI tasks we use the following datasets:
• SNLI (Bowman et al., 2015): SNLI corpus is a collection of human-written English sentence pairs manually labeled for balanced classification with the labels entailment, contradiction, and neutral.
The data consists of 550,152/10,000/10,000 sentence pairs for train/val/test set respectively.
The max length of the input is set to 158.
• ANLI (Nie et al., 2019): ANLI dataset is a large-scale NLI dataset, collected via an, adversarial human-and-model-in-the-loop procedure.
The data consists of 162,865/3,200/3, sentence pairs for train/val/test set respectively.
The max length of the input is set to 128.
For the Image Classification datasets, we used Convolutional Neural Networks (CNNs).
Specifically, we used LeNet5 (Lecun et al., 1998), ResNet50, ResNet34, and ResNet18 (He et al., 2015), for
MNIST, CIFAR10, CIFAR100, and ImageNet100, respectively.
For the NLI datasets, we finetuned transformer based Pre-trained Language Models (PLMs).
Specifically, we used Bert-base (Devlin et al., 2018) and Roberta-base (Liu et al., 2019), for SNLI and ANLI, respectively.
5. E XPERIMENTAL SETUP
We compare our Expected Squared Difference (ESD) to previously proposed trainable calibration objectives, MMCE and SB-ECE, on the datasets and models previously mentioned.
For fair comparison, interleaved training has been used for all three calibration objectives.
For MMCE, in which Kumar et al.
(2018) proposed an unweighted and weighted version of the objective, we use the former to set the method to account for the overfitting problem mentioned in section 3.2.2 consistent to interleaved training.
For the interleaved training settings, we held out 10% of the train set to the calibration set.
The regularizer hyperparameter λ for weighting the calibration measure with respect to NLL is chosen via fixed grid search 2 .
For measuring calibration error, we use ECE with 20 equally sized 2 For λ , we search for [0.2, 0.4, 0.6, 0.8, 1.0, 2.0, 3.0, ... 10.0] (Appendix E).
[Table]
bins.
For the image classification tasks we use AdamW (Loshchilov & Hutter, 2017) optimizer with 10 − 3 learning rate and 10 − 2 weight decay for 250 epochs, except for ImageNet100, in which case we used 10 − 4 weight decay for 90 epochs.
For the NLI tasks, we use AdamW optimizer with 10 −learning rate and 10 − 2 weight decay for 15 epochs.
For both tasks, we use a batch size of 512.
The internal hyperparameters within MMCE ( ϕ ) and SB-ECE ( M , T ) were sequentially optimizedfollowing the search for optimal λ .
Following the original experimental setting of SB-ECE by
Karandikar et al.
(2021), we fix the hyperparameter M to 15.
Similar to the model selection criterion utilized by Karandikar et al.
(2021), we look at the accuracy and the ECE of all possible hyperparameter configurations in the grid.
We choose the lowest ECE while giving up less than 1 .
5% accuracy relative to baseline accuracy on the validation set.
All experiments were done using
NVIDIA Quadro RTX 8000 and NVIDIA RTX A6000.
E XPERIMENTAL R ESULT
In Table 1, we report the accuracy and ECE for the models before and after post-processing (i.e., temperature scaling and vector scaling) of various datasets and models.
Compared to the baseline or other trainable calibration objective loss (MMCE and SB-ECE), jointly training with ESD as the secondary loss consistently results in a better-calibrated network for all the datasets and models with around 1% degradation of accuracy.
Moreover, we observe that applying post-processing (i.e., temperature scaling (TS) and vector scaling (VS)) after training with a calibration objective loss generally results in better-calibrated models over baseline post-processing, in which case ESD still outperforms other methods.
Comparing the calibration outcomes in temperature scaling with vector scaling for models trained with ESD, vector scaling worked comparable if not better as a post-processing method, except for ANLI, with minimal impact on the accuracy for all datasets.
Additionally, Ovadia et al.
(2019) demonstrated that postprocessing methods, such as temperature scaling, are not effective in achieving good calibration under distribution shifts.
Building on this research, Karandikar et al.
(2021) have provided evidence that incorporating calibration during model training can enhance a model’s ability to maintain calibration under distribution shifts.
This finding has been replicated in Appendix F.
We show in Figure 2 that across different models and datasets the calibration performance of MMCE and SB-ECE is sensitive to the internal hyperparmater.
This shows the importance of hyperparameter tuning in these methods.
On the other hand, ESD does not only outperforms MMCE and SB-ECE in terms of ECE but does not have an internal hyperparameter to tune for.
A BLATION S TUDY 7. C OMPUTATIONAL C OST OF H YPERPARAMETER S EACH
We investigate the computational cost required to train a model with ESD compared to MMCE and
SB-ECE.
As depicted in Figure 3, the computational cost required to train a model for a single run remains nearly identical across different models and datasets.
However, considering the need for tuning additional hyperparameters within MMCE and SB-ECE, the discrepancy in total computational cost between ESD and tuning-required calibration objective losses becomes more prominent as the model complexity and dataset size increases.
7. B ATCH S IZE E XPERIMENTS
Table 2 shows that ESD is robust to varying batch sizes which could be due to the fact that it is an unbiased and consistent estimator.
Regardless of the batch size, being unbiased guarantees that the average of the gradient vectors of ESD equals the gradient of the true estimate (Appendix D).
[Table]
7. A RE I NDICATOR F UNCTIONS T RAINABLE ?
Recent papers, Kumar et al.
and
Karandikar et al.
(2021), have suggested that ECE is not suitable for training as a result of its high discontinuity due to binning, which can be seen as a form of an indicator function.
However, the results from our method suggest that our measure was still able to train well despite the existence of indicator functions.
In addition, previous measures also contain indicator functions in the form of argmax function that introduces discontinuities but remains to be trainable.
As such, this brings to rise the question of whether calibration measures with indicator functions can be used for training.
To investigate this, we ran ECE as an auxiliary calibration loss on different batch sizes and observed its performance on CIFAR 100 (Table 3).
We found that ECE, contrary to previous belief, is trainable under large batch sizes and not trainable under small batch sizes while ours maintains good performance regardless of batch size.
The poor performance of ECE under small batch size setting could potentially be attributed to the high bias present in such cases.
From this, it seems to suggest that indicator functions do not seem to inhibit the training for calibration.
[Table]
C ONCLUSIONS
Motivated by the need for a tuning-free trainable calibration objective, we proposed Expected Squared
Difference (ESD), which does not contain any internal hyperparameters.
With extensive comparison with existing methods for calibration during training across various architectures (CNNs & Transformers) and datasets (in vision & NLP domains), we demonstrate that training with ESD provides the best-calibrated models compared to other methods with minor degradation in accuracy over baseline.
Furthermore, the calibration of these models is further improved after post-processing.
In addition, we demonstrate that ESD can be utilized in small batch settings while maintaining performance.
More importantly, in contrast to previously proposed trainable calibration objectives, ESD does not contain any internal hyperparameters, which significantly reduces the total computational cost for training.
This reduction in cost is more prominent as the complexity of the model and dataset increases, making ESD a more viable calibration objective option in modern deep learning settings.
A CKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea government(MSIT)
(No.2022-0-00184, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy on Ethics), and Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT)
(No. 2021-0-01381, Development of Causal AI through Video Understanding and
Reinforcement Learning, and Its Applications to Real Environments)
A V ISUAL I NTUITION OF E XPECTED S QUARED D IFFERENCE (ESD)
From Eq.
[Equation]
This can be viewed as the difference between two quantities:
[Equation]
Due to the close relationship between d k ( α ) and the calibration of a neural network, the average squared difference ( E Z ′ k
[ d k ( Z ′ k ) 2 ] ) between the cumulative accuracy and confidence closely corresponds with the calibration of a network (Figure 4).
That is, the average squared difference between the cumulative accuracy and confidence is larger for an uncalibrated network compared to a calibrated network.
Jointly training with our proposed Expected Squared Difference (ESD) as an auxiliary loss tries to minimize this squared difference between the two curves during training on average, thus achieving a better calibrated model.
B P ROOF THAT ESD IS AN UNBIASED AND CONSISTENT ESTIMATOR
In Theorem 2, we prove that ESD is an unbiased estimator, that is taking the expectation of ESD is equal to the true Expected Squared Difference.
Theorem 2 ESD is an unbiased estimator, i.e. E Z k , Y k
[ ESD ] =
E Z ′ k
[ d 2 k ( Z ′ k )]] where d k ( Z ′ k ) =
| E Z k ,Y k
[ I ( Z k ≤ Z ′ k )
( I ( Y k = 1) − Z k )] | .
Proof.
Let Z k =( Z k, 1 ,..., Z k,n ), Y k
= ( Y k, 1 ,..., Y k,n ) and G i = ¯ g
i 2 − S 2 gi N − 1 .
We have that
[Equation]
(by linearity of expectation)
[Equation]
by lemma
[Equation] [Equation]
since E
[Equation]
Lemma 2.1 E Z k, − i , Y k, − i
[ G i ] = µ 2 i where µ i =
E Z k ,Y k
[ I ( Z k ≤ Z k,i )
( I ( Y k = 1) − z k )] .
Proof.
Since samples are i.i.d., for a fixed i , it holds that:
[Equation]
N −(where Var [ g i ]
= σ 2 i )
Var
[Equation]
¯ g i ] =
E Z k, − i , Y k,
[Equation]
i [ ¯ g
[Equation]
µ
[Equation]
i .
Shifting variables around, we get
[Equation]
(cid:35) (since Var [ ¯ g i ]
= σ 2 i
N −
[Equation]
In Theorem 3, we prove that ESD is a consistent estimator, which means that ESD converges in probability to the true Expected Squared Difference.
Theorem 3 ESD is a consistent estimator, i.e. ESD P −→ E Z ′
k
[ d 2 k ( Z ′ k )] .
Proof.
Since ESD is an unbiased estimator, it is sufficient to prove lim n →∞ Var [ ESD ] = 0 .
For simplicity, we use where (cid:80) Ni,j =1 = (cid:80) Ni =1 (cid:80) Nj =1 and E Z k , Y k
[ G i ] =
E Z ′ k
[ d 2 k ( Z ′ k )] ∀ i,
[Equation]
Since E Z k , Y k [ G i ] =
E Z ′ k
[ d 2 k ( Z ′ k )] ∀ i,
[Equation]
N
[Equation]
N
[Equation]
i
[Equation]
(by lemma 3.1 and 3.2)
[Equation]
Thus, by using lemma 3.4 and the non-negativeness of variance,
[Equation] [Equation]
d
[Equation]
Consequently, lim n →∞ Var [ ESD ] = 0 .
Lemma 3.1 Var [ G i ] ≤ 2 < ∞ .
Proof.
[Equation]
Y k [ G 2 i ] =
E Z k , Y k  
¯ g i 2 − 2 S 2 g i N − 1 + (cid:32) S 2 g
i
N −(cid:33) 2   ≤ E Z k , Y k  
¯ g i 2 + (cid:32) S 2 g
i
N −(cid:33) 2   .
By lemma 3.2, E Z k , Y k
[ g 2 i ] ≤ 1 and E Z k , Y k
(cid:34)(cid:18) S 2 gi
N −(cid:19) 2 (cid:35) ≤ (cid:16) 2 N − 2 (cid:17) 2 ≤ 1 .
Thus,
E Z k , Y k (cid:34) ¯ g
i 2 + (cid:18) S 2 gi N − 1 (cid:19)(cid:35) ≤ 2 .
Lemma 3.2 | S 2 gi N − 1 | ≤ 2 N − 2 and | ¯ g i | ≤ 1 .
Proof.
By the triangle inequality, | ¯ g i | = | 1 N − 1 N (cid:80) m
[Equation]
m ̸
[Equation]
i g im | ≤ 1 N
[Equation]
N (cid:80) m =1 m ̸
= i | g
im | ≤ 1 , since ∀ m | g
im | ≤ 1 .
Furthermore, we have
[Equation]
The last inequality is by ∀ m , | g im | ≤ 1 , and the direct calculation implies that
[Equation]
Lemma 3.3 E Z k , Y k [ ¯ g i 2 ¯ g j 2 ] =
E Z k , Y k
[ ¯ g 12 ¯ g 22 ] ∀ i, j where i ̸ = j.
Proof.
N (cid:88) m,n =1 { m,n }̸ = { i,j } g
im g jn =
N (cid:88) m,n =1 { m,n }̸ = { i,j } , m ̸
= n g im g jn
+
N (cid:88) m,n =1 n ̸ = { i,j } g ij g jn +
N (cid:88) m,n =1 m ̸
= { i,j } g
im g ji +
N (cid:88) m,n =1 m ̸
= { i,j } g
im g jm + g ij g ji (since these terms are disjoint terms)
d = N (cid:88) m ′ ,n ′ =1 { m ′ ,
n ′ }̸
= { 1 , 2 } , m ′ ̸
= n ′ g 1 m ′ g 2 n ′ +
N (cid:88) m ′ ,n ′ =1 n ′ ̸ = { 1 , 2 } g 12 g 2 n ′ +
N (cid:88) m ′ ,n ′ =1 m ′ ̸
= { 1 , 2 } g 1 m ′ g+
N (cid:88) m ′ ,n ′ =1 m ′ ̸
= { 1 , 2 } g 1 m ′ g 2 m ′
+ g 12 g=
N (cid:88) m ′ ,n ′ =1 m ′ ̸ =1 ,n ′ ̸ = g 1 m ′ g 2 n ′ .
Proof of claim above:
As the summation can be divided into five distinct cases,
Let i and j be arbitrary such that i ̸
= j ,
Case 1: { m, n } ̸ = { i, j } and m ̸
= n ,
[Equation]
where { m ′ , n ′ } ̸ = { 1 , 2 } and m ′ ̸
= n ′
Case 2: m = j and n ̸
= { i, j } ,
[Equation]
Z
[Equation]
Y
[Equation]
, Y
[Equation]
where m ′ = 2 and n ′ ̸
= { 1 , 2 }
Case 3: n = i and m ̸
= { i, j } ,
[Equation]
where n ′ = 1 and m ′ ̸
= { 1 , 2 }
Case 4: m = n
[Equation]
= { i, j } ,
[Equation]
where m ′ = n ′ ̸
= { 1 , 2 }
Case
[Equation] [Equation]
and
[Equation]
where m ′ = 2 and n ′ =
This follows from the fact that Z k,i ’s abd Y k,i ’s are i.i.d.
random variables.
Therefore, (
[Equation]
Lemma 3.
[Equation]
Proof.
For arbitrary i, ¯ g i P −→ µ i by the strong law of large number.
Thus, ¯ g 1 P −→ µ 1 and ¯ g 2 P −→ µ 2 .
Therefore, since marginal convergence in probability implies joint convergence in probability (Vaart, 1998),
[Equation]
By continuous mapping theorem, since f ( x, y ) = x 2 y 2 is a continuous function,
[Equation]
Additionally, since | ¯ g 12 ¯ g 22 | ≤ 1 ∀
N
, it is uniformly bounded and thus uniformly integrable.
Combining this with the fact that it converges in probability,
[Equation]
Thus, lim N →∞
[Equation]
C E XPECTED S QUARED D IFFERENCE (ESD) P SEUDOCODE
In this section, we provide a pseudocode for calculating the Expected Squared Difference (ESD) for a given batch output.
# n:number of samples in
# confidence:1D tensor of n elements containing max softmax outputs of each sample from a neural network.
# correct:1D tensor of n elements containing 1/0 corresponding to the correctness of each sample.
def ESD loss(n, confidence, correct): # compute the difference between confidence and correctness.
diff = correct.float() - confidence # Prepare the split between inner and outer expectation estimation.
split=torch.ones(n,n) - torch.eye(n) # compute the inner expectation estimation.
confidence mat= confidence.expand(n,n) ineq =
torch.le(confidence mat,confidence mat.
T).float() diff mat =
diff.view(1,n).expand(n,n) x mat =
torch.mul(diff mat,ineq)*split mean row =
torch.sum(x mat, dim = 1)/(n-1) x mat squared =
torch.mul(x mat, x mat) var =
1/(n-2) *
torch.sum(x mat squared,dim=1) - (n-1)/(n-2)
* torch.mul(mean row,mean row) # compute the outer expectation estimation.
d k sq vector = torch.mul(mean row, mean row) - var/(n-1)
ESD =
torch.sum(d
k sq vector)/n return ESD
D
U NBIASED ESTIMATORS AND ITS
G RADIENT
Since ESD is an unbiased estimator,
[Equation]
law of unconscious statistician)
[Equation]
(since ( X , Y ) are independent of θ )
Thus, the gradient of the unbiased estimator is on average the gradient of the desired metric.
E A DDITIONAL I NFORMATION ON THE C HOICE OF λ R ANGE
To validate the choice of λ grid range in our experimental settings, we plot the variations in accuracy with respect to increasing λ for all methods (Figure 5).
We see that our choice of λ contains points within the acceptable range of accuracy (i.e., within 1.5% degradation in accuracy compared to baseline).
However, after a particular λ value, which is different on a per method per dataset basis, the accuracy decreases consistently with increasing λ .
As such, the grid chosen is suitable for the experiments conducted.
F T RAINABLE C ALIBRATION M EASURES U NDER D ISTRIBUTION S HIFT
In Figure we evaluate the performance of calibration methods under distribution shift benchmark datasets, CIFAR 10-C and CIFAR 100-C, introduced in Hendrycks & Dietterich (2019).
For models jointly trained with an auxiliary calibration objective (NLL+MMCE, NLL+SB-ECE, NLL+ESD), we observe that they are more robust to distribution shifts when compared to those solely trained with
NLL.
In addition, prior work (Ovadia et al., 2019) has shown that the calibration performance of a neural network after temperature scaling may be significantly reduced under distribution shifts.
Our results on CIFAR 10-C and CIFAR 100-C imply a similar trend.
Stacking calibration during training methods with temperature scaling (NLL+MMCE+T, NLL+SB-ECE+T, NLL+ESD+T), we observe that they perform comparable to temperature scaled models trained with NLL (NLL+T) with the exception of ESD, where it performs marginally better.
As such, training with ESD could potentially improve a model’s robustness after temperature scaling regarding distribution shifts and mitigate this issue.