AdaFocal: Calibration-aware Adaptive Focal Loss
Abstract
Much recent work has been devoted to the problem of ensuring that a neural network’s confidence scores match the true probability of being correct, i.e. the calibration problem.
Of note, it was found that training with focal loss leads to better calibration than cross-entropy while achieving similar level of accuracy [19].
This success stems from focal loss regularizing the entropy of the model’s prediction (controlled by the parameter γ ), thereby reining in the model’s overconfidence.
Further improvement is expected if γ is selected independently for each training sample (Sample-Dependent Focal Loss (FLSD-53) [19]). However, FLSD-53 is based on heuristics and does not generalize well.
In this paper, we propose a calibration-aware adaptive focal loss called AdaFocal that utilizes the calibration properties of focal (and inverse-focal) loss and adaptively modifies γ t for different groups of samples based on γ t − 1 from the previous step and the knowledge of model’s under/over-confidence on the validation set.
We evaluate AdaFocal on various image recognition and one NLP task, covering a wide variety of network architectures, to confirm the improvement in calibration while achieving similar levels of accuracy.
Additionally, we show that models trained with AdaFocal achieve a significant boost in out-of-distribution detection.
Introduction
Neural networks have found tremendous success in almost every field including computer vision, natural language processing, and speech recognition.
Over time, these networks have grown complex and larger in size to achieve state-of-the-art performance and they continue to evolve in that direction.
Along with these advances it has also been well established that these networks suffer from poor calibration [4], i e. the confidence scores of the predictions do not reflect the real world probabilities of those predictions being true.
For example, if the network assigns 0 .
8 confidence to a set of predictions, we should expect 80% of those predictions to be correct.
However, this is far from reality since modern networks tend to be grossly over-confident.
This is of great concern, particularly for mission-critical applications such as autonomous driving or medical diagnosis, wherein the downstream decision making relies not only on the predictions but also on their confidences.
In recent years, there has been a growing interest in developing methods for neural network calibration.
These roughly fall into two categories (1) post hoc approaches that perform calibration after training (2) methods that calibrate the model during training.
The first category includes methods such as temperature scaling [4], histogram binning [31], isotonic regression [32], Bayesian binning and averaging [22, 21], Dirichlet scaling [10], mix-n-match methods [34], and spline fitting [5].
Methods in the second category focus on designing objective functions that account for calibration during training, such as Maximum Mean Calibration Error (MMCE) [13], Label smoothing [20], and recently focal loss [19].
These aim to inherently calibrate the model during training, yet when combined with post hoc calibration further improvement is often obtained.
Contribution Our work belongs to the second category.
We first show that while regular focal loss (with fixed γ ) improves the overall calibration by preventing samples from being over-confident, it also leaves other samples under-confident.
To address this issue, we propose a modification to the focal loss, while utilizing inverse-focal loss [30, 17], named AdaFocal that adjusts the γ for each training sample (or rather a group of samples) separately by taking into account the model’s under/over-confidence about a corresponding sample (or group) in the validation set.
AdaFocal also adaptively switches from focal to inverse focal loss when focal loss fails to overcome the level of under-confidence.
We evaluate our method on four image classification tasks (CIFAR-10,
CIFAR-100, Tiny-ImageNet and ImageNet) and one text classification task (20 Newsgroup) using various model architectures.
We find that AdaFocal substantially outperforms regular focal loss and other state-of-the-art calibration-during-training techniques in the literature.
Models calibrated by AdaFocal benefit more from post hoc calibration techniques to further reduce the calibration error. Finally, we study the performance of AdaFocal on an out-of-distribution detection task and find a substantial improvement in performance.
Problem Setup and Definitions
For a classification problem with training data { ( x n , y true ,n ) } , where x n is the input and y true ,n ∈
Y = { 1 , 2 , . . .
, K } is the ground-truth, we train a model f that outputs a probability vector ˆ p over the K classes.
We further assume access to a validation set for hyper-parameter tuning and a test set for evaluation.
For example, f θ ( · ) can be a neural network with learnable parameters θ , x is an image, and ˆ p is the output of a softmax layer whose k th element ˆ p k is the probability score for class k .
We refer to ˆ y = arg max k ∈Y ˆ p k as the network’s prediction and the probability score ˆ p ˆ y as the confidence.
Then, a model is said to be perfectly calibrated if the confidence score
ˆ p ˆ y matches the probability of the model classifying x correctly i.e. P (ˆ y =
y true
| ˆ p ˆ y = p ) =
p, ∀ p ∈ [0 , 1] [4].
Continuing our example, if the network assigns an average confidence score of 0 .
8 to a set of predictions then we should expect 80% of those to be correct.
To quantify calibration, we use Calibration Error as E = ˆ p ˆ y − P (ˆ y =
y true
| ˆ p ˆ y ) and the Expected
Calibration Error as E ˆ p
ˆ
y
[ E ] =
E
ˆ p ˆ y [ | ˆ p ˆ y − P
(ˆ y =
y true
| ˆ p ˆ y ) | ] [4].
Since the true calibration error cannot be computed empirically with a finite sized dataset, the following approximations are generally used in the literature.
For a dataset { ( x n , y true ,n ) } Nn =1 , (1) ECE EW =
(cid:80) Mi
=1 | B i | N | C
i − A
i | [4], where B i is equal-width (EW) bin that contains all examples j with ˆ p ˆ y,j in the range [ iM , i +1 M ) ,
C i =
| B i | (cid:80) j ∈ B i ˆ p ˆ y,j is the average confidence and A i =
| B i | (cid:80) j ∈ B i 1 (ˆ y j = y true ,j ) is the bin accuracy.
Note that E i
=
C
i − A
i is the empirical approximation of the calibration error E , (2) ECE EM =
(cid:80) Mi
=1 | B i | N | C
i − A
i | [24], where ∀ i, j | B i | =
| B j | are equal-mass (EM) bins.
Furthermore, as ECE has been shown to be a biased estimate of the true calibration error [29], we additionally use ECE DEBIAS [11] and ECE SWEEP [26] to corroborate the results in the paper.
Calibration Properties of Focal Loss
Focal loss [16] L FL ( p ) =
− (1 − p )
γ log p was originally proposed to improve the accuracy of classifiers by focusing on hard examples and down-weighting well classified examples.
Recently, it was shown that focal loss can also be used to improve calibration [19].
This is because, based on the relation L FL ≥ KL ( q || ˆ p ) − γ H (ˆ p ) ( q is the one-hot target vector), focal loss while minimising the KL divergence objective also increases the entropy of the prediction ˆ p .
This prevents the network from being overly confident on wrong predictions thereby improving calibration.
However, as we show next, focal loss with fixed γ falls short of achieving the best calibration.
In Figure 1, we plot the calibration behaviour of ResNet50 in different probability regions (bins) when trained on CIFAR-10 with different focal losses.
The i th bin’s calibration error
E val,i =
C val,i − A val,i is computed on the validation set using 15 equal-mass binning.
The figure plots a lower (bin-0), middle (bin-7) and higher bin (bin-14) (rest of the bins and their bin boundaries are shown in Appendix B).
We see that, although focal loss γ = 4 achieves the overall lowest ECE EM , there’s no single γ that performs the best across all the bins.
For example, in bin- 0 , γ = 4 , 5 achieves better calibration whereas γ = 0 , 3 are over-confident.
On the other hand, in bin- 7 γ = 3 seems to be better calibrated whereas γ = 4 , 5 are under-confident and γ = 0 is over-confident.
[Figure]
This shows that using different γ s for different bins can bring further improvement.
Such an attempt called the Sample-Dependent Focal Loss (FLSD-53) is presented in [19] that assigns γ = 5 if the training sample’s true class posterior
ˆ p y true ∈ [0 , 0 . 2) and γ = 3 if ˆ p y true ∈ [0 . 2 , 1] .
However, from
Figure 1(b), FLSD-53 is also not the best calibrated method across all bins.
It is a strategy based on fixed heuristics of choosing higher γ for smaller values of ˆ p y true and relatively lower γ for higher values of ˆ p y true .
This clearly motivates the design of a strategy that can assign appropriate γ i s for each bin-
i based on the magnitude and sign of E val,i .
To design such a strategy one, however, faces two challenges:
How do we find some correspondence between the confidence of training samples (which we can manipulate during training using the parameter γ ) and the confidence of the validation/test samples (which are our actual target but we do not have direct control over them)?
In other words, to indirectly control the confidence of a group of validation samples, how do we know which particular group of training samples’ confidence to be manipulated?
Given that some correspondence is established, how do we arrive at the appropriate values of γ that will lead to the best calibration?
We try to answer the first question in the next section and answer to the second question leads to the main contribution of the paper: AdaFocal.
Additionally, alongside focal loss, we make use of the inverse-focal loss [30, 17] for cases where regular focal loss fails to overcome under-confidence.
See, for example, ResNet-50 trained on
ImageNet in section 6 and Fig.
5 where even cross entropy ( γ = 0 ) can not reach the desired level of confidence.
Inverse-focal loss, plotted in Fig. 3(a), and given by
[Equation]
serves the opposite purpose of focal loss.
While focal loss reduces the over-confidence of the network, inverse-focal loss helps recover from under-confidence by providing larger gradients to the samples with higher confidences (easy samples), thereby pushing their scores even further.
Correspondence between Confidence of Training and Validation Samples
One way to check for any correspondences is to simply group the validation samples into M equalmass bins (henceforth called validation-bins) and compare the confidence with the training samples that fall into the respective validation-bin boundaries.
Before proceeding further, we first clarify a few notations of interest.
Quantities of interest For binning the validation samples, we look at the confidence of the top predicted class ˆ y denoted by ˆ p val,top (bin average: C val,top ).
For training samples, on the other hand, instead of the confidence of the top predicted class ˆ y denoted by ˆ p train,top (bin average: C train,top ), we will focus on the confidence of the true class y true denoted by ˆ p train,true (average: C train,true ) because during training we only care about ˆ p train,true that gets manipulated by some loss function (Figure 8 in Appendix C compares C train and to show that as the training accuracy approaches 100% , the top predicted class and the true class become the same). For brevity, we will henceforth refer to C train ≡ C train,true and C val ≡ C val
In Figure 2, we compare C train,i in validation-bin- i 1 with C val,i and find that there is indeed a good correspondence between the two quantities.
For example in Figure 2, as γ increases from 0 (CE), toto 5, the solid-line C train,i gets lower, and the same behaviour is observed for the starred-line C val,i .
For more evidence refer to Fig.
11, 12, and 13 in Appendix C where similar behaviour is observed for ResNet-50 and WideResNet on CIFAR-100, and ResNet-50 on TinyImageNet, respectively.
We also look at the case when training samples and validation samples are grouped independently into their respective training-bins and validation-bins.
Figure 9 in Appendix C compares C train,i in training-bin- i with C val,i in validation-bin- i .
Again, we observe a similar correspondence.
Note here that, since the binning is independent, the boundaries of training-bin- i will not be exactly the same as that of validation-bin- i , but, as shown in Figure 9, they are very close to each other.
These observations, therefore, are very encouraging as now we have a way to indirectly control C val,i by manipulating C train,i , i.e. we can expect (even if loosely) that if we increase/decrease the confidence of a group of training samples in some lower (or middle, or higher) probability region then the same will get reflected on the validation samples in a similar lower (or middle, or higher) probability region.
From a calibration point of view, our strategy going forward would be to exploit this correspondence to push C train,i (which we have control over during training) closer to A val,i (the validation set accuracy in validation-bin- i ) so that C val,i also gets closer to A val,i , and, therefore, achieve a very low calibration error C val,i − A val,i .
For simplifying the design of AdaFocal, we will employ the first method of common binning i.e. using validation-bins to bin the training samples.
Proposed Method
Let’s denote the n th training sample’s true class posterior ˆ p y true by p n and p n falls into validation-bin b .
Our goal then is to keep p n (or its averaged equivalent C train,b ) closer to A val,b so that the same is reflected on C val,b .
For manipulating p n , we will utilize the regularization effect of focal loss’s parameter γ .
At this point, one can choose to update the γ of validation-bin-
b denoted by γ b either based on (1) how far p n is from A val,b i.e. γ =
g
( p n − A val,b ) or (2) how far C val,b is from
A val,b i.e. γ =
g
( C val,b − A val,b ) .
Such a γ -update-rule should ensure that whenever the model is over-confident, i.e. p n > A val,b (or C val,b > A val,b ), γ is increased so that the gradients get smaller to prevent p n from increasing further.
On the other hand, when p n < A val,b (or C val,b <
A val,b ), i.e. the model is under-confident, we decrease γ so as to get larger gradients that in turn will increase p n 2 . gradients.
This mostly holds true in the region p n approximately > 0 .
2 (see Figure 3(a) in [19]).
However, in practice and as shown by the training-bin boundaries of bin-0 and bin-1 in Figure in Figure 9 Appendix C, we find majority of the training samples to lie above 0 .
2 during the majority of the training, and therefore, for the experiments in this paper, we simply stick to the rule of increasing γ to decrease gradients and stop p n from increasing and vice versa.
[Figure]
To test this strategy, we first design a calibration-aware γ -update method (called CalFocal), which with some additional modifications will lead to the final AdaFocal algorithm.
5.1 Calibration-aware Focal Loss (CalFocal)
Case 1: γ = g ( p n − A val,b ) Treating A val,b as the point that we want p n to not deviate from, we make the focal loss parameter γ a function of p n − A val,b to get
[Equation]
where γ n = exp( λ ( p n − A val,b )) , (2) and b is the validation-bin in which p n falls.
The hyper-parameter λ is the scaling factor which combined with the exponential function helps to quickly ramp up/down γ .
The exponential function adheres to the γ -update rule mentioned earlier.
Figure 3(a) plots L CalFocal vs. p n for A val,b = 0 .
We see that based on the strength of λ , the loss drastically drops near p n = 0 .
8 and thereafter remains close to zero.
This shows that L CalFocal aims is to first push p towards 0 .
and then slow its growth towards overconfidence.
Next, in Figure 3(c), we find that CalFocal with λ = 10 , 100 is able to reduce the calibration error ( ECE EM ) but it is still far from FLSD-53’s performance.
Also note in
Figure 3(b) that too high λ (=100) affects the accuracy of model.
Most interesting is Fig. 3(d) which compares C train,i with C val,i (and also A val,i ) for bin-0, where we find evidence that the strategy of bringing p n or C train,i (solid lines) closer to A val,i (dashed lines) results in C val,i (starred lines) getting closer to A val,i as well, thus reducing the calibration error E val,i =
C val,i − A val,i .
Case 2: γ = g ( C val,b − A val,b ) Note that Eq.
2 assigns a different γ n for each training sample p n .
To reduce computation and avoid keeping track of γ n for each training sample, we can assign one γ b to each validation-bin- b by making it a function of C val,b − A val,b (instead of p n − A val,b ).
Then all p n that fall into the validation-bin- b are assigned γ b and the loss function is modified to
[Equation]
where γ b = exp( λ ( C val,b − A val,b )) (3) b is the validation-bin in which p n falls.
The performance of this strategy, as shown in Appendix P, is very similar (or slightly better than) Eq. 2.
Further, it makes more sense to update γ based on how far
C val,b is from A val,b instead of how far p n is from A val,b because, as shown in Figure 3(d) bin-0, one may find C val,b (starred lines) quite closer to A val,b (dashed lines) even when p n or C train (solid lines) is still far from A val,b .
At this point where C val,b = A val,b , we should stop updating γ further, even if p n − A val,b ̸ = 0 , as we have reached our goal of making E val,b = C val,b − A val,b =
Limitations of CalFocal (1) Let’s say at some step of training, a high γ b over some epochs reduces the error C val,
b − A val,b .
Then, it is desirable to continue training with the same high γ b .
However, note CalFocal’s update rule in Eq.
3 which will reduce γ → 1 as the C val,b − A val,b → 0 .
(2) Let’s say, at some point C val,
b − A val,b is quite high.
This will set γ b to some high value depending on the hyper-parameter λ .
Assuming this γ b is still not high enough to bring down the confidence, we would want a way to further increase γ b .
However, CalFocal is incapable of doing so as it will continue to hold at γ b = exp( λ ( C val,b − A val,b )) .
5.2 Calibration-aware Adaptive Focal Loss (AdaFocal)
We propose to address the above limitations by making γ t,b depend on γ t − 1 ,b from previous time step
[Equation]
This update rule address the limitations of CalFocal in the following way: let’s say at some point we observe over-confidence i.e. E val,b = C val,b − A val,b > 0 .
Then, in the next step γ b will be increased.
In the subsequent steps, it will continue to increase unless the calibration error E val,b starts decreasing (this additional increase in γ was not possible with CalFocal).
At this point, if we find E val,b to start decreasing, that would reduce the increase in γ b over the next epochs and γ b will ultimately settle down to a value when E val,b =
(CalFocal at E val,b = 0 will cause γ to go down to 1 ).
Next, if this current value of γ b starts causing under-confidence i.e. C val,b − A val,b < 0 , then the update rule will kick in to reduce γ thus allowing C val,b to be increased back to A val,b .
This oscillating behaviour of AdaFocal around the desired point of C val,b =
A val,b is its main strength in reducing the calibration error in every bin.
Next, to deal with cases where even cross entropy suffers from under-confidence, we switch to inversefocal loss which can further increase the confidence of the predictions.
For the switch between focal and inverse-focal loss, we simply set a threshold S th below which if gamma falls, we switch to the other loss function.
Note here that, for notational purpose, we will denote the inverse-focal loss by a negative γ i.e. γ t,b > 0 means focal loss with parameter γ t,b whereas γ t,b < 0 implies inverse-focal loss with parameter |
γ t,b | .
The complete gamma-update rule and the switching criteria is given in
Algorithm 1.
If not stated explicitly, we use S th = 0 .
2 for all AdaFocal experiments.
For reference, we also plot results with S th = 0 .
5 for ImageNet, ResNet-50 in Fig.
4 (d) and Fig. 5(b).
Finally, note the unbounded exponential update in Eq. 4 which is an undesirable property.
This may easily cause γ t to explode as when expanded γ t = γ t − 1 exp( E val,t ) =
γ 0 exp( E val, 0 + E val, 1 + ... + E val,t − 1 + E val,t ) , and if E val,t > 0 for quite a few number of epochs, γ t will become so large that even if E val,t < 0 in the subsequent epochs, it may not reduce to a desired level.
We remedy this by constraining γ t to an upper bound γ max when γ > 0 (focal loss) and lower bound of γ min when γ <
(inverse-focal loss).
Therefore, the final AdaFocal loss is given by
[Equation]
and the complete algorithm along with the gamma-update rules is given in Algorithm 1.
A discussion on the selection of hyper-parameters is presented in the next section.
Experiments
We evaluate the performance of our proposed method on image and text classification tasks.
For image classification, we use CIFAR-10, CIFAR-100 [9], Tiny-ImageNet [2], and ImageNet [27] to analyze the calibration of ResNet50, ResNet-100 [6], Wide-ResNet-26-10 [33], and DenseNet- [8] models.
For text classification, we use the 20 Newsgroup dataset [14] and train a global-pooling
CNN [15] and fine-tune a pre-trained BERT model [3].
More details about the datasets, models and experimental configurations are given in Appendix D. In the main paper, we report results using only
ECE EM , whereas other ECE metrics are reported in Appendix.
Baseline Among calibration-during-training methods we use MMCE [13], Brier loss [1], Label smoothing (LS-0.05) [20] and sample-dependent focal loss FLSD-53 as baselines.
For post hoc calibration, we report the effect of temperature scaling, ensemble temperature scaling (ETS) [34] and spline fitting [5] on top of these methods.
For temperature scaling, we select the optimal temperature ∈ (0 , 10] (step size 0 . 1 ) that gives the lowest ECE EM on the validation set.
Results In Figure 4, we compare AdaFocal against cross entropy (CE) and FLSD-53, for ResNet50 trained on various small to large-scale image datasets.
Among various focal losses, we chose
FLSD-53 as our baseline because it was shown to be consistently better than MMCE, Brier Loss and Label smoothing [19] across many datasets-model pairs.
The figure plots the test set error and
ECE EM .
In Figure 5, for ResNet-50 on CIFAR-10 and ImageNet, we plot (1) the calibration statistics
E val =
C val − A val used by AdaFocal during training and (2) the dynamics of γ t for a few bins in lower, middle, and higher probability regions.
[Table]
[Table] [Table]
[Table]
// Inverse-focal γ -update; if | γ t +1 ,
i | <
S th then γ t +1 ,i =
S th // Switch to focal loss;
First, we observe that for CIFAR-10, CIFAR-100 and Tiny-ImageNet, FLDS-53 is much better calibrated than CE.
This is because, as shown in Fig.
5(a) for ResNet-50, CIFAR-10, CE is overconfident compared to FLSD-53 in almost every bin.
For ImageNet, however, the behaviour is reversed: FLSD-53 is poorly calibrated than CE.
This, as shown in Figure 5(b), is due to the use of high values of γ ( = 5 , 3 ) by FLSD-53 which makes the model largely under-confident in each bin, leading to a overall high calibration error.
AdaFocal, on the other hand, maintains a well calibrated model throughout the training for all cases.
Next, for ResNet-50 trained on CIFAR-10, we find γ t to be closer to 1 for higher bins and closer to 20 for lower bins.
These γ t s result in better calibration than γ t = 5 , 3 of FLSD-53.
For ImageNet, on the other hand, we find, except bin-14, AdaFocal switches to inverse-focal loss at some point in the training.
This makes sense because for ImageNet even cross entropy ( γ t = 0 ) suffers from under-confidence, therefore, AdaFocal, starting from γ t
= 1 , first approaches cross entropy ( γ = 0 ) to ultimately switch to inverse-focal loss ( γ t < 0 ) at S th = 0 .
Since, in retrospect, we already know that switching to inverse focal loss is beneficial for ImageNet, switching early at S th = 0 .
5 helps to reach the same level of calibration early.
Overall, these experiments confirm that during training
AdaFocal being aware of the network’s current under/over-confidence is able to guide the γ t s to values that maintain a well calibrated model at all times.
Also note that for an unseen dataset-model pair there’s no way to know apriori which γ will perform better but AdaFocal will automatically find these appropriate values thus avoiding an expensive and extensive hyper-parameter search.
Comparison of ECE EM and test set error on for rest of the experiments are shown in Table 1 and 2 respectively.
From Table 1, we observe that prior to temperature scaling, AdaFocal outperforms the other methods in 14 out of 15 cases.
Post-temperature scaling, AdaFocal achieves the lowest calibration error in 12 out of the 15 cases.
Further, observe that the optimal temperatures are mostly close to 1 indicating that AdaFocal produces inherently calibrated models from training itself.
Effects of other post hoc calibration methods (ETS and spline fitting) are shown in
Appendix E. Again, we observe that pre-calibrated models from AdaFocal benefit from post processing to further lower the overall calibration error.
Besides ECE EM , the consistency of the results across other calibration
[Figure] [Figure]
[Table]
Out-of-Distribution (OOD) detection Following [19], we report the performance of AdaFocal on an OOD detection task.
We train ResNet-110 and Wide-ResNet26-10 on CIFAR-10 as the indistribution data and test on SVHN [23] and CIFAR-10-C [7] (with level 5 Gaussian noise corruption) as OOD data.
Using entropy of the softmax as the measure of uncertainty, the corresponding ROC plots are shown in Figure 6 and AUROC scores are reported in Table 8 in Appendix K.
We see that models trained with AdaFocal outperform focal loss γ = 3 (FL-3) and FLSD-53.
For the exact
AUROC scores, please refer to Appendix K. These results further highlight the benefits of inherently calibrated model produced by AdaFocal as post-hoc calibration techniques such as temperature scaling, as shown in the figure, is ineffective under distributional shift [28].
Hyper-parameters
The hyper-parameters introduced by AdaFocal are λ , γ min / max , and S th .
However, these do not require an extensive hyper-parameter search and are much easier to select compared to γ (which, otherwise, needs to be searched for every bin at every time step).
Based on our experiments,
• λ is redundant and one may choose to ignore it as for all our experiments λ
= 1 worked very well.
However, for an unknown dataset-model pair, if increasing/decreasing the rate of change of γ improves calibration (or helps to reach the desired level faster), then one can use λ to achieve so.
[Figure]
Dataset
Model
CE
Brier
MMCE
LS
FLSDAdaFocal
CIFARResNet 4.
ResNet 4.
WideResNet 3.
DenseNet 5.
CIFARResNet 23.
ResNet 22.
WideResNet 20.
DenseNet 24.
Tiny-ImageNet
ResNet 42. ResNet 42.
[Table]
• γ min / max do not require any special fine-tuning as their sole purpose is to stop γ from exploding in either directions.
This is similar to the common practice of gradient clipping for stable training.
For
all
our experiments
[Table]
we use γ max =
but any reasonable value around that range should also work well in practice.
For comparison of results with γ max = 20 , γ max = 50 and γ max = ∞ , please refer to Appendix N. • γ min = − 2 is selected based on the observation that values beyond − 2 led to unstable training.
However, if, for a new untested dataset-model pair
[Table]
γ min = − 2 turns out to be unsuitable, it should
[Figure]
still be fairly easy to select a new threshold by simply looking at the “dynamics of γ ” plots (similar to Fig. 5) at the time step the training becomes unstable.
• We use S th = 0 .
2 for all our experiments.
This also does not require extensive tuning and can be easily selected based on the evolution of γ in various bins for a trial run of AdaFocal ( without switching to inverse focal loss).
For example, for ImageNet (Fig. 5 ), where some of the bins are always under-confident, AdaFocal decreases γ (starting at γ = 1 ) towards negative values.
In this case, it makes more sense to have a higher S th ( = 0 . 5 ) so that AdaFocal can switch early to inverse-focal loss and overcome the under-confidence (see ImageNet results in Fig 4 and 5).
• Number of bins.
We experimented with AdaFocal using 5, 10, 15, 20, 30, and 50 equal-mass bins during training to draw calibration statistics form the validation set.
As reported in Appendix H, the best results are for number of bins in the range of 10 to 20.
Performance degrades when the number of bins are too small ( < 10 ) or too large ( > 20 ).
Therefore, we use 15 bins for all AdaFocal trainings.
Note that for computing ECE metrics as well, we use 15 bins so as to be consistent with previous works in literature [19, 4].
Best choice of AdaFocal + post-hoc calibration From the results in Table 1, 3, and 4 for temperature scaling, ETS and Spline fitting, respectively, there is not a clear choice of post-hoc calibration method that gives the best results across all dataset-model pairs when combined with AdaFocal.
However, we do observe that in almost all cases it is the pre-calibrated model by AdaFocal that gives the lowest ECE when combined with one of these post-hoc calibration techniques.
Therefore, for an unknown dataset-model pair, the choice of the best post-hoc calibration method (to be used on top of AdaFocal) might require more investigation and is a different study in itself.
Overall, the evidence in our paper show that pre-calibration from training leads to even better calibrated models post hoc, and AdaFocal is much better in producing such pre-calibrated models.
Conclusion
In this work, we first revisit the calibration properties of regular focal loss to highlight the downside of using a fixed γ for all samples.
Particularly, by studying the calibration behaviour of different samples in different probability region, we find that there’s no single γ that achieves the best calibration over the entire region.
We use this observation to motivate the selection of γ independently for each sample (or group of samples) based on the knowledge of models’s under/over-confidence from the validation set.
We propose a calibration-aware adaptive strategy called AdaFocal that accounts for such information and updates the γ t at every step based on γ t − 1 from the previous step and the magnitude of the model’s under/over-confidence.
We find AdaFocal to perform consistently better across different dataset-model pairs producing inherently calibrated models that benefit further from post-hoc calibration techniques.
Additionally, we find that models trained with AdaFocal are much better at out-of-distribution detection.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] in Section 6. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] in Appendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[N/A] (b) Did you include complete proofs of all theoretical results?
[N/A] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] As part of the supplementary material and details are mentioned in Appendix D. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] see Appendix D. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] see Appendix G. (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] in Appendix D. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] in Appendix D. (b) Did you mention the license of the assets?
[Yes] (c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix A Broader Impact
Overconfidence in deep neural networks could easily lead to deployments where predictions are made that should have been withheld.
For example, in medical diagnosis applications the Bayes optimal decision depends heavily on the model accurately modeling the distribution over its output classes.
We hope that our work is a small step towards avoiding high loss errors in decision making applications.
The metrics used in this paper assume that improving average calibration is the goal; but other metrics should be considered if we want to, for example, ensure good average calibration across different strata (e.g. if instances correspond to users of different social strata).
B Calibration Behaviour of Focal Loss in Different Bins
In the main paper, we have showed the calibration behavior of different focal losses for ResNet trained on CIFAR-10 for only a few bins.
For completeness, the rest of the bins and their calibration error E i =
C val,i − A val,i are shown in Figure 7 for focal losses with γ = 0 , 3 , 4 , 5 .
We observe that there’s no single γ that performs the best across all the bins.
Rather, every bin has a particular γ that achieves the best calibration.
C Correspondence between Confidence of Training and Validation Samples C.
1 Closeness of C train,true and C train,top as Training Progresses
For a training sample, the confidence of the true class y true is denoted by ˆ p train,true and the average equivalent in a bin by C train,true .
Similarly, the confidence of the top predicted class ˆ y (for the training sample) is denoted by ˆ p train,top and the average equivalent in a bin by C train,top .
For the training set, we care only about the confidence of the "true class" ˆ p train,true as that is the quantity which gets manipulated by some loss function.
For validation set, on the other hand, we care about the confidence of the "top predicted class".
Therefore, it would be more natural to look for correspondence between similar quantities, particularly C train,top,i , across the two datasets.
However, as we shown in Fig. 8, C train,true,i and C train,top,i are almost the same during major part of the training.
This is because as the model approaches towards 100% accuracy on the training set, the top predicted class and the true class for a training sample become the same.
Therefore we can directly compare the two different quantities C train,true,i and C train,top,i across the training and validation set.
C
CIFAR-10, ResNet
: Correspondence
between C train and C val
Fig.
9 and 10 show the correspondence between the confidence of training samples C train ≡
C train,true and the confidence of the validation samples C val ≡ C val,top for the dataset-model pair
CIFAR-10, ResNet-50, under following two cases:
• Independent binning : when training samples and validation samples are grouped independently into their respective training-bins and validation-bins (Fig. 9).
• Common binning : when training samples are grouped using the common bin boundaries of the validation-bins that were formed by binning the validation bins (Fig. 10).
C.2.1 Independent binning
[Figure]
[Figure] [Figure]
[Figure] [Figure]
CIFAR
[Figure]
WideResNet
[Figure]
[Figure] [Figure]
[Figure]
D Datasets and Experiments D.1 Dataset Description
CIFAR-10 [9]: This dataset contains 60 , 000 coloured images of size 32 × 32 , which are equally divided into 10 classes.
A split of 45 , 000 / 5 , 000 / 10 , 000 images is used as train/validation/test sets respectively.
CIFAR-100 [9]: This dataset contains 60 , 000 coloured images of size 32 × 32 , which are equally divided into 100 classes.
A split of 45 , 000 / 5 , 000 / 10 , 000 images is used as train/validation/test sets respectively.
ImageNet [27]: ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012-2017 is an image classification and localization dataset.
This dataset spans 1000 object classes and contains 1,281,167 training images and 50,000 validation images.
Tiny-ImageNet [2]: It is a subset of the ImageNet dataset with 64 × 64 dimensional images andclasses.
It has 500 images per class in the training set and 50 images per class in the validation set.
20 Newsgroups [14]: This dataset contains 20 , 000 news articles, categorised evenly intodifferent newsgroups.
Some of the newsgroups are very closely related to each other (e.g. comp.sys.ibm.pc.hardware / comp.sys.mac.hardware), while others are highly unrelated (e.g misc.forsale / soc.religion.christian).
We use a train/validation/test split of 15 , 098 / 900 / 3 ,documents.
D.2 Experiment Configurations
For our experiments, we have used Nvidia Titan X Pascal GPU with 12 GB of memory.
Training configuration for each dataset is given below.
CIFAR-10 and CIFAR-100 : We use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0 .
1 for the first 150 epochs, 0 .
01 for the nextepochs, and 0 .
001 for the last 100 epochs.
We use a training batch size of 128 .
The training data is augmented by applying random crops and random horizontal flips.
Tiny-ImageNet : We use SGD with a momentum of 0.9 as our optimiser, and train the models for 100 epochs with a learning rate of 0 .
1 for the first 40 epochs, 0 .
01 for the next 20 epochs and 0 .for
the last 40 epochs.
We use a training batch size of 64 .
Note that we use 50 samples per class (i.e. a total of 10000 samples) from the training set as the validation set.
Hence, the training is only on 90000 images.
We use the Tiny-ImageNet validation set as our test set.
ImageNet : We use SGD as our optimiser with momentum of 0.9 and weight decay 10 − 4 , and train the models for 90 epochs with a learning rate of 0 .
01 for the first 30 epochs, 0 .
001 for the nextepochs and 0 .
0001 for the last 30 epochs.
We use a training batch size of 128 .
We divide the 50, validation images into validation and test set of 25,000 images each.
20 Newsgroups, CNN : We train the Global Pooling CNN Network [15] using the Adam optimiser, with learning rate 0 .
001 , and default betas 0 .
9 and 0 .
We used Glove word embeddings [25] to train the network.
We train the model for 50 epochs and use the model at the end to evaluate the performance.
20 Newsgroups, BERT : We fine-tune a BERT model by adding a single linear classification layer on top of pre-trained "bert-base-uncased" model (12-layer, 768-hidden, 12-heads, 110M parameters) [3], using the AdamW optimiser (Adam with weight decay), with batch size of 32, learning rate 2 e − 5 , weight decay of 0 .
01 and warm up steps of 0 .
2 the number of batches in the training set.
We limit the length of the input sequence to 128 for training and 512 for testing.
We train the model for 10 epochs and select the model that has the lowest error on validation set.
The experiments are implemented using PyTorch library.
The hyperparameters that are not explicitly mentioned above are set to their default values in PyTorch.
For CIFAR-10/100 and Tiny-ImageNet,
AdaFocal is implemented on top of the base code available at [18].
The code for 20 Newsgroups is implemented in PyTorch by adapting the TensorFlow code available at [12].
D.3 Model Selection
For all experiments, except Tiny-ImageNet, we select the model at the end of the training mainly to be consistent with [19] i.e. the work we are trying to improve upon in this paper.
As confirmed by the authors of [19]
[Table]
they use the model at the end of the training to report results in the paper.
Therefore, for the following datasets
[Table]
the error and ECE results are reported for the model at
• CIFAR-10: 350 epochs • CIFAR-100: 350 epochs • ImageNet: 90 epochs • 20 NewsGroups, CNN: 50 epochs
For Tiny-ImageNet and BERT
[Table]
we have reported the model that has the lowest error on the validation set
[Table]
E Other Post Hoc Calibration Techniques E.1 Ensemble Temperature Scaling (ETS)
[Table]
E.2 Spline Fitting
[Table]
F Debiased Estimates of ECE: ECE DEBIAS and ECE SWEEP
As shown in [26]
[Table]
binning-based estimators ECE EW and ECE EM may suffer from statistical bias ( ECE EM has lower bias than ECE EW ) and if the bias is strong enough it may lead to mis-estimation of calibration error and a wrong model selection.
Therefore, to confirm that the results presented in the paper using ECE EM and ECE EW are consistent and reliable
[Table]
we additionally present here
ECE DEBIAS [11] and ECE SWEEP [26] (equal-mass) as debiased estimates of ECE
[Table]
[Table]
G ECE EW error bars
cross-validated based on ECE EW .
H Number of bins used for AdaFocal training
Experiment details:
ResNet-50 trained on CIFAR-10 for 350 epochs.
The reported results below are without temperature scaling.
We compare AdaFocal with 5, 10, 15, 20, 30, and 50 equal mass bins vs FLSD-53.
Note that there are two types of binning involved:
• For training : the binning that is performed on the validation set from where AdaFocal draws calibration related information to adjust γ .
These correspond to the columns in the table 7.
• For evaluation : once we have a trained model, the binning that is used to compute the ECE metric.
These correspond to the rows in the table below.
[Table]
I Frequency of γ -update
In this section, we study how the frequency of the γ -update affect the performance of AdaFocal on the 20Newsgroup dataset (with CNN and BERT models)
i.e if the validation-bin boundaries and γ are updated every mini-batch or a few times per training epoch.
Intuitively, one would expect that if the validation-bin boundaries are updated more frequently then AdaFocal would be able to more closely track the changes in the calibration behaviour of the validation set and accordingly adjust it’s γ s to better respond to the changes.
This is supported by the experiments on 20 Newsgroup dataset using CNN and BERT model as shown in Fig.
16 and 17.
In these figures, AdaFocal on its own means that γ is updated at the end of every epoch.
AdaFocal- n , where n = 100 , 50 , 10 , 1 , means that γ is updated every n mini-batches.
Since the number of training batches for 20Newsgroup with CNN is 118 , AdaFocal- 50 (respectively 10 , 1 ) means γ is updated(respectively 11 , 118 ) times per epoch.
Similarly, for 20Newsgroup and BERT, as the the number of training batches is 472 , AdaFocal- 100 (respectively 10 , 1 ) means γ is updated 4 (respectively 47 , 472 ) times per epoch.
From these experiments, we observe that
Frequent updates keep the model better calibrated at all time steps and prevent it from getting mis-calibrated .
For example in Fig. 16(b), AdaFocal and AdaFocal- 50 are miscalibrated at the start of the training and at around 15-20 epoch, whereas AdaFocaland AdaFocal- 1 remains very well calibrated at all epochs.
We observe the same in Fig.
17(b), where AdaFocal and AdaFocal- 100 are miscalibrated at the start of the training and around epoch 6 − 9 , whereas AdaFocal- 10 and AdaFocal- 1 remains well calibrated at all times.
For cases where updating γ once per epoch leads to only a few γ updates in total, frequent updates may lead to improved ECE performance .
We observe this in Fig.
17(b) where AdaFocal (with only 10 updates as BERT is trained for only 10 epochs) is unable to reach the calibration level of AdaFocal- 10 and AdaFocal
For 20Newsgroup and CNN, as the model is trained for 50 epochs, AdaFocal, first unable to keep up with AdaFocal- 10 and
AdaFocal- 1 , is able to ultimately reach their level of calibration with enough updates.
Drawback: Increased training time .
Updating γ more frequently comes at the cost of increased training time.
For example, as discussed in J, standard training of ResNet-50 on CIFAR-10 takes
1 s per epoch and γ -update step takes 2 .
8 s. When trained for 350 epochs, updating γ every epoch adds 16 minutes to the the total training time of 7 .
7 h. Therefore, increasing the update frequency will linearly increase the additional overhead as well.
[Figure]
AdaFocal on its own means that γ is updated at the end of every epoch.
AdaFocal- n , where n = 50 , 10 , 1 , means that γ is updated after every n mini-batches.
Since the number of training batches for 20Newsgroup with CNN is 118 , AdaFocal- 50 (respectively 10 , 1 ) means γ is updated(respectively 11 , 118 ) times per epoch
[Figure]
J Computation overhead for AdaFocal
To update γ of AdaFocal the extra operations that are required are
Forward pass on the validation set to compute the logits/softmaxes.
Compute bin statistics and update γ .
In general, if we update γ at the end of every epoch, then compared to the time it takes to train the model for one whole epoch, these two overheads are quite negligible.
For example, for ResNettrained on CIFAR-10 (train set contains 45000 examples, val set contains 5000 examples) using
Nvidia Titan X Pascal GPU with 12GB memory,
• Training for one epoch = 79 , 123 ms = 79 .
1 s • Forward pass on validation set = 2 , 886 ms = 2 .
8 s • Compute bin statistics and update γ =
ms
So if the standard training with cross entropy, without any involvement of a validation set, forepochs requires in total 79 . 1 × 3503600 = 7 .
7 hours, then AdaFocal will add 2 .
× 35060 = 16 minutes on top of the entire training.
Naturally, if we update γ more often during the epoch then this overhead will increase and may become significant.
However, for all our experiments we update γ at the end of an epoch and that works quite well.
Nonetheless, for a comparison of performance of AdaFocal when the update frequency of γ is varied, please refer to Appendix I.
K AUROC for Out-of-Distribution Detection
For ResNet110 trained on in-distribution CIFAR-10 and tested on out-of-distribution SVHN, we were not able to reproduce the reported results of 96.74, 96.92 for focal loss γ = 3 (FL-3) as given in [19].
Instead we found those values to be 90.27, 90.39 and report the same in Table 8 below.
[Table]
L Reliability Diagrams
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
M "Calibrate-able" Property
In Fig. 34, following [30], for ResNet-50 on CIFAR-10 and ResNet-50 on CIFAR-100, we plot the distribution of max-logits of training examples at the end of training (i.e. at epoch 350) grouped as per different "learned epochs".
Observations :
• Unlike for ResNet-32 in [30], we do not find cross entropy + Temperature Scaling to be better than focal loss + temperature scaling for ResNet-50 (the same is observed in [19]).
• Although the distribution of FLSD-53 is compressed, similar to what observed for focal loss in [30], the separation of samples is not seen here for cross entropy (CE).
• For AdaFocal, we see better separation of easy and hard examples grouped as per their "learned epoch".
This makes AdaFocal more "calibrate-able" as per [30].
[Figure]
N Multiple runs of AdaFocal with different γ max
Due to the stochastic nature of the experiments, AdaFocal γ s may end up following different trajectories across different runs (initialization), which in turn might lead to variations in the final results.
In this section, we look at the extent of such variations for ResNet-50 trained on CIFAR-10 for γ max = 20 , γ max = 50 and unconstrained γ ( γ max = ∞ ).
For all these experiments, the minimum γ for inverse-focal loss is set to γ min = − 2 and the switching threshold is set to S th = 0 .
N.1 AdaFocal γ max =
In Fig.
35, we observe that AdaFocal with γ max
= 20 is consistently better than FLSD-53.
Fig.shows the variation in dynamics of γ during training across different runs.
[Figure]
N.2 AdaFocal γ max =
In Fig. 37, we observe that AdaFocal with γ max = 50 has more variability than AdaFocal γ max =but is mostly better than FLSD-53.
Fig.
37 shows the variation in dynamics of γ during training across different runs.
[Figure]
N.3 AdaFocal, unconstrained γ ( γ max = ∞ )
In Fig. 39, we observe that AdaFocal with unconstrained γ exhibit grater variability across different runs: 7 out of 9 times it performs better than FLSD-53 whereas the other two times it is similar or slightly worse.
[Figure]
The above behaviour is mostly due to large variations in the trajectory of γ s especially for lower bins as shown in Fig.
For higher bins, γ s do not explode and settle to similar nearby values, whereas, for lower bins, as the γ s are unconstrained they blow up to udesirably high values.
Error, ECE, dynamics of γ , and bin statistics during training
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
P Comparison of CalFocal Loss Case 1 (Eq. 2) and Case 2 (Eq. 3)
For Fig.
56 below, please note the following legend:
• CE = Cross Entropy.
• L Exp,λ =
CalFocal loss case 1 (Eq. 2 in the main paper) which assigns γ s to each training sample.
• λ =
CalFocal loss case 2 (Eq. 3 in the main paper) which assigns a common γ b to all training samples that fall in validation-bin b .
[Figure]
[Figure] [Figure]