AdaFocal: Calibration-aware Adaptive Focal Loss
Anonymous Author s)
AffiliationAddress email
Abstract
Much recent work has been devoted to the problem of ensuring that a neuralnetwork’s confidence scores match the true probability of being correct, i.e. thecalibration problem.
Of note, it was found that training with focal loss leads tobetter calibrated neural networks than cross-entropy, while achieving the same levelof accuracy [18].
This success stems from focal loss regularizing the entropy of themodel’s prediction (controlled by the parameter γ ), thereby reining in the model’soverconfidence.
Further improvement is expected if γ is selected independentlyfor each training sample; Sample-Dependent Focal Loss (FLSD-53) in [18] doesjust that, but is based on heuristics that do not generalize well to all dataset-modelpairs.
In this paper, we propose a calibration-aware adaptive focal loss called
AdaFocal that utilizes the calibration properties of focal (and inverse-focal loss)and adaptively modifies γ t for different groups of samples based on (1) γ t − 1 fromthe previous step (2) the magnitude of the model’s under/over-confidence.
Weevaluate AdaFocal on various image recognition tasks and one NLP task, coveringa variety of network architectures, to confirm the improvement in calibration whileacieving similar levels of accuracy.
Additionally, models trained with AdaFocalare shown to exhibit a significant boost in out-of-distribution detection capability.
Introduction
Neural networks have found tremendous success in almost every field including computer vision,natural language processing and speech recognition.
Over time, these networks have grown complexand larger in size to achieve state-of-the-art performance and they continue to evolve in that direction.
However, it has been well established that such high capacity networks suffer from poor calibration[3], i e. the confidence scores of the predictions do not reflect the real world probabilities of thosepredictions being true.
For example, if the network assigns 0 .
8 confidence to a set of predictions, weshould expect 80% of those predictions to be correct.
However, this is far from reality since modernnetworks tend to be grossly over-confident.
This is of great concern, particularly for mission-criticalapplications such as autonomous driving or medical diagnosis, wherein the downstream decisionmaking relies not only on the predictions but also on their confidences.
In recent years, there has been a growing interest in developing methods for calibrating neuralnetworks.
These roughly divide into two categories (1) post-hoc approaches that perform calibrationafter training (2) methods that calibrate the model during training itself.
The first category includesmethods such as temperature scaling [3], histogram binning [30], isotonic regression [31], Bayesianbinning and averaging [21, 20], Dirichlet scaling [9], mix-n-match methods [33], and spline fitting [4].
Methods in the second category focus on designing objective functions that account for calibrationduring training itself, such as Maximum Mean Calibration Error (MMCE) [12], Label smoothing[19], and recently focal loss [18].
These aim to inherently calibrate the model during training, yetwhen combined with post hoc calibration further improvement is often obtained.
Contribution.
Our work belongs to the second category.
We first show that while regular focal loss(with fixed γ ) improves the overall calibration by preventing samples from being over-confident, italso leaves other samples under-confident.
To address this drawback, we propose a modification tofocal (and inverse-focal) loss [29, 16] called AdaFocal that adjusts the γ for each training sample(or rather a group of samples) separately by taking into account the model’s under/over-confidenceabout a corresponding sample (or group) in the validation set.
AdaFocal also adaptively switchesbetween focal and inverse focal loss when focal loss fails to reach the desired level of confidence.
Weevaluate our method on four image classification tasks (CIFAR-10, CIFAR-100, Tiny-ImageNet and
ImageNet) and one text classification task (20 Newsgroup) using various model architectures.
Wefind that AdaFocal substantially outperforms regular focal loss and other state-of-the-art calibrationduring-training techniques in the literature.
We further study the performance of AdaFocal on anout-of-distribution detection task and find a substantial improvement in performance.
Finally, we seethat models calibrated by AdaFocal may further benefit from post hoc calibration techniques.
Problem Setup and Definitions
For a classification problem with training data { ( x n , y true ,n ) } , where x n is the input and y true ,n ∈
Y = { 1 , 2 , . . .
, K } is the ground-truth, we wish to train a model f that outputs a probability vectorˆ p over the K classes.
We further assume access to a validation set for hyper-parameter tuning and atest set for evaluation.
For example, f θ ( · ) can be a neural network with learnable parameters θ , x isan image, and ˆ p is the output of a softmax layer whose k th element ˆ p k is the probability
score forclass k .
We refer to ˆ y = arg max k ∈Y ˆ p k as the network’s prediction and the probability score ˆ p ˆ y asthe confidence.
Then, a model is said to be perfectly calibrated if the confidence score
ˆ p ˆ y matchesthe true probability of the model classifying x correctly i.e. P (ˆ y =
y true
| ˆ p ˆ y = p ) =
p, ∀ p ∈ [0 , 1][3].
Continuing our example, if the network assigns an average confidence score of 0 .
8 to a set ofpredictions then we should expect 80% of those to be correct.
To quantify calibration, we use Calibration Error as E = ˆ p ˆ y − P (ˆ y =
y true
| ˆ p ˆ y ) and the Expected
Calibration Error as E ˆ p
ˆ
y
[ E ] =
E
ˆ p ˆ y [ | ˆ p ˆ y − P
(ˆ y =
y true
| ˆ p ˆ y ) | ] [3].
Since the true calibration errorcannot be computed empirically with a finite sized dataset, the following approximations are generallyused in the literature.
For a dataset { ( x n , y true ,n
[Equation]
[Equation]
where B i is equal-width (EW) bin that contains all examples j with ˆ p ˆ y,j in the range [ iM , i +1 M ) ,
C i =
| B i | (cid:80) j ∈ B i ˆ p ˆ y,j is the average confidence and A i =
| B i | (cid:80) j ∈ B i 1 (ˆ y j = y true ,j ) is the binaccuracy.
Note that E i
=
C
i − A
i is the empirical approximation of the calibration error E , (2)
ECE EM =
(cid:80) Mi
=1 | B i | N | C
i − A
i | [23], where ∀ i, j | B i | =
| B j | are equal-mass (EM) bins.
However,as ECE has been shown to be a biased estimate of the true calibration error [28], we additionally use
ECE DEBIAS [10] and ECE SWEEP [25] to further corroborate the results in the paper.
Calibration Properties of Focal Loss
Focal loss [15] L FL ( p ) =
− (1 − p )
γ log p was originally proposed to improve the accuracy ofclassifiers by focusing on hard examples and down-weighting well classified examples.
Recently, itwas shown that focal loss can also be used to significantly improve calibration [18].
This is because,based on the relation L FL ≥ KL ( q || ˆ p ) − γ H (ˆ p ) ( q is the one-hot target vector), focal loss whileminimising the KL divergence objective also increases the entropy of the prediction ˆ p .
This preventsthe network from being overly confident on wrong predictions thereby improving calibration.
However, focal loss with fixed γ , as we show next, falls short of achieving the best calibration.
In Figure 1, we plot the calibration behaviour of ResNet50 in different probability regions (bins)when trained on CIFAR-10 with different focal losses.
The i th bin’s calibration error
E val,i =
C val,i − A val,i is computed on the validation set using 15 equal-mass binning.
The figure plots alower (bin-0), middle (bin-7) and higher bin (bin-14) (rest of the bins and their bin boundaries areshown in Appendix B).
From Figure 1 (a), although focal loss γ = 4 achieves the overall lowest
ECE EM , there’s no single γ that performs the best across all the bins.
For example, in bin- 0 , γ = 4 , 5achieves better calibration whereas γ = 0 , 3 are over-confident.
On the other hand, in bin- 7 γ = 3seems to be better calibrated whereas γ = 4 , 5 are under-confident and γ = 0 is over-confident.
[Figure]
This shows that using different γ s for different bins can further improve calibration.
Such an attemptcalled the Sample-Dependent Focal Loss (FLSD-53) is presented in [18] that assigns γ = 5 if thetraining sample’s true class posterior
ˆ p y true ∈ [0 , 0 . 2) and γ = 3 if ˆ p y true ∈ [0 . 2 , 1] .
However, from
Figure 1(b), FLSD-53 is also not the best calibrated across all bins.
Rather, this is a fixed strategybased on heuristics of choosing higher γ for smaller values of ˆ p y true and relatively lower γ for highervalues of ˆ p y true .
This, therefore, motivates the design of a strategy that can assign appropriate γ i s foreach bin-
i based on the magnitude and sign of E val,i .
To design such a strategy one, however, faces two challenges:
How do we find some correspondence between the confidence of training samples (whichwe can manipulate during training using the parameter γ ) and the confidence of the validation/test samples (which is our actual target but we do not have direct control over)?
Inother words, to indirectly control the confidence of a group of validation samples, how dowe know which particular group of training samples’ confidence to be manipulated?
Given that a correspondence is established, how do we arrive at the appropriate values of γthat will lead to the best calibration?
We try to answer the first question in the next section and answer to the second question leads to themain contribution of the paper: AdaFocal.
Additionally, alongside focal loss, we will make use of the inverse-focal loss [29, 16] for cases whereno γ of regular focal loss is able to avoid under-confidence.
See, for example, ResNet-50 trained on
ImageNet in section 6 and Fig.
5 where even cross entropy ( γ = 0 ) doesn’t achieve the desired levelof confidence.
Inverse-focal loss, plotted in Fig. 3(a), and given by
[Equation]
serves the opposite purpose of focal loss.
While focal loss reduces the over-confidence of the network,inverse-focal loss helps recover from under-confidence by providing larger gradients to the sampleswith higher confidences (easy samples) by pushing their score even further.
Correspondence between Confidence of Training and Validation Samples
One way to check for any correspondences is to simply group the validation samples into M equalmass bins (henceforth called validation-bins) and compare the confidence with the training samplesthat fall into the same validation-bins.
But first we clarify a few notations that are of interest.
Quantities of interest For binning validation samples, we look at the confidence of the top predictedclass ˆ y denoted by ˆ p val,top (bin average: C val,top ).
For training samples, on the other hand, insteadof the confidence of the top predicted class
ˆ y denoted by ˆ p train,top (bin average: C train,top ), we willfocus on the confidence of the true class y true denoted by ˆ p train,true (average: C train,true ) becauseduring training we only care about ˆ p train,true which gets manipulated by some loss function (forreference, Figure 8 in Appendix C compares C train,true and C train,top to show that as the training accuracy approaches 100% , the top predicted class and the true class become the same).
For brevity,we will refer to C train ≡ C train,true and C val ≡ C val,top .
In Figure 2, we compare C train,i in validation-bin- i 1 with C val,i and find that there is indeed a goodcorrespondence between the two quantities.
For example in Figure 2, as γ increases from 0 (CE), to 3to 5, the solid-line C train,i gets lower, and the same behaviour is observed for the starred-line C val,i .
For more evidence refer to Fig.
11, 12, and 13 in Appendix C where similar behaviour is observedfor ResNet-50 and WideResNet on CIFAR-100, and ResNet-50 on TinyImageNet, respectively.
Additionally, we look at the case when training samples and validation samples are grouped independently into their respective training-bins and validation-bins.
Figure 9 in Appendix C compares
C train,i in training-bin- i with C val,i in validation-bin- i .
Here again we observe a similar correspondence.
Further, note that since the binning is independent, the boundaries of training-bin- i will not beexactly the same as that of validation-bin- i , however as shown in Figure 9 they are very close.
This observation is very encouraging as now we have a way to indirectly control C val,i by manipulating C train,i i.e. we can expect (even though loosely) that if we increase/decrease the confidence of agroup of training samples in some lower (or middle, or higher) probability region then the same willget reflected on a group of validation samples in a similar lower (or middle, or higher) probabilityregion.
Therefore, from a calibration point of view, our strategy going forward would be to exploitthis correspondence to keep C train,i (which we have control over during training) closer to A val,i(the validation set accuracy in validation-bin- i ) so that C val,i also stays closer to A val,i and thereforeachieve a very low calibration gap C val,i − A val,i .
For simplifying the design of AdaFocal, we willemploy the first method of common binning i.e. using validation-bins to bin the training samples.
Proposed Method
Let’s denote the n th training sample’s true class posterior ˆ p y true by p n .
Given that p n falls intovalidation-bin- b , our goal is to keep p n (or its averaged equivalent C train,b ) closer to A val,b so thatthe same is reflected on C val,b .
For manipulating p n , we will utilize the regularization effect of focalloss’s parameter γ .
At this point, one can choose to update the γ of validation-bin-
b denoted by γ beither based on (1) how far p n is from A val,b i.e. γ =
g
( p n − A val,b ) or (2) how far C val,b is from
A val,b i.e. γ =
g
( C val,b − A val,b ) .
Such a γ -update-rule should ensure that whenever the model isover-confident, i.e. p n > A val,b (or C val,b > A val,b ), γ is increased so that the gradients get smallerto prevent p n from increasing further.
On the other hand, when p n < A val,b (or C val,b <
A val,b ), i.e.the model is under-confident, we decrease γ so as to get larger gradients that in turn will increase p n 2 .
To test this strategy, we first design a calibration-aware γ -update method (called CalFocal), whichwith some additional modifications will lead to the final AdaFocal
[Figure]
5.1 Calibration-aware Focal Loss (CalFocal)
Case 1: γ = g ( p n − A val,b ) Treating A val,b as the point that we want p n to not deviate from, wemake the focal loss parameter γ a function of p n − A val,b to get
L CalFocal ( p n ) = − (1 − p n ) γ n log p n , where γ n = exp( λ ( p n − A val,b )) , (2) and b is the validation-bin in which p n falls.
The hyper-parameter λ is the scaling factor whichcombined with the exponential function helps to quickly ramp up/down γ .
The exponential functionadheres to the γ -update rule mentioned earlier.
Figure 3(a) plots L CalFocal vs. p n for A val,b = 0 .
We see that based on the strength of λ , the loss drastically drops near p n = 0 .
8 and thereafter remainsclose to zero.
This shows that L CalFocal aims is to first push p towards 0 .
and then slow its growthtowards overconfidence.
Next, in Figure 3(c), we find that CalFocal with λ = 10 , 100 is able toreduce the calibration error ( ECE EM ) but it is still far from FLSD-53’s performance.
Also note in
Figure 3(b) that too high λ (=100) affects the accuracy of model.
Most interesting is Fig. 3(d)
whichcompares C train,i with C val,i (and also A val,i ) for bin-0, where we find evidence that the strategyof bringing p n or C train,i (solid lines) closer to A val,i (dashed lines) results in C val,i (starred lines)getting closer to A val,i as well, thus reducing the calibration error E val,
i =
C val,i − A val,i .
Case 2: γ = g (
C val
,b −
A
val,b ) Note that Eq.
2 assigns a different γ n for each training sample p n .
To reduce computation and avoid keeping track of γ n for each training sample, we can assign one γ bto each validation-bin- b by making it a function of C val,b − A val,b (instead of p n − A val,b ).
Then allp n that fall into the validation-bin- b are assigned γ b and the loss function is modified to
L CalFocal ( p
[Equation]
p n , where γ b = exp( λ ( C val,b − A val,b )) (3) b is the validation-bin in which p n falls.
The performance of this strategy, as shown in Appendix D,is very similar (or slightly better than) Eq.
Further, it makes more sense to update γ based on howfar C val,b is from A val,b instead of how far p n is from A val,b because, as shown in Figure 3(d) bin-0,one may find C val,b (starred lines) quite closer to A val,b (dashed lines) even when p n or C train (solidlines) is still far from A val,b .
At this point where C val,b = A val,b , we should stop updating γ further,even if p n − A val,b ̸ = 0 , as we have reached our goal of making E val,b = C val,b − A val,b = 0 .
Therefore, for AdaFocal we will use Eq.
3 as the base for AdaFocal loss function.
Limitations of CalFocal: (1) Let’s say at some step of training, a high γ b over some epochs reducesthe gap C val,
b − A val,b .
Then, it is desirable to continue training with the same high γ b .
However,note CalFocal’s update rule in Eq.
3 which will reduce γ → 1 as the C val,b − A val,b → 0 .
(2) Let’ssay, at some point C val,b − A val,b is quite high.
This will set γ b to some high value depending on thehyper-parameter λ .
Assuming this γ b is still not high enough to bring down the confidence, we wouldwant a way to further increase γ b .
However, CalFocal is incapable of doing so as it will continue tohold at γ b = exp( λ ( C val,b − A val,b )) .
5.2 Calibration-aware Adaptive Focal Loss (AdaFocal)
We propose to address the above limitations by making γ t,b depend on γ t − 1 ,b from previous time stepγ t,b = γ t − 1 ,b ∗ exp( λ ( C val,b − A val,b )) .
This update rule address the limitations of CalFocal in the following way: let’s say at some pointwe observe over-confidence i.e. E val,
b = C val,b − A val,b > 0 .
Then, in the next step γ b will beincreased.
In the subsequent steps, it will continue to increase unless the calibration error E val,bstarts decreasing (this additional increase in γ was not possible with CalFocal).
At this point, if wefind E val,b to start decreasing, that would reduce the increase in γ b over the next epochs and γ b willultimately settle down to a value when E val,b =
(CalFocal at E val,b = 0 will cause γ to go downto 1 ).
Next, if this current value of γ b starts causing under-confidence i.e. C val,b − A val,b < 0 , thenthe update rule will kick in to reduce γ thus allowing C val,b to be increased back to A val,b .
Thisoscillating behaviour of AdaFocal around the desired point of C val,b
= A val,b is its main strength inreducing the calibration error in every bin.
Next, to deal with cases where even cross entropy suffers from under-confidence, we switch to inversefocal loss which can further increase the confidence of the predictions.
For the switch between focaland inverse-focal loss, we simply set a threshold S th below which if gamma falls, we switch to theother loss function.
Note here that, for notational purpose, we will denote the inverse-focal loss by anegative γ i.e. γ t,b > 0 means focal loss with parameter γ t,b whereas γ t,b < 0 implies inverse-focalloss with parameter
| γ t,b | .
The complete gamma-update rule and the switching criteria is given in
Algorithm 1.
If not stated explicitly, we use S th = 0 .
2 for all AdaFocal experiments.
For reference,we also plot results with S th = 0 .
5 for ImageNet, ResNet-50 in Fig.
4 (d) and Fig. 5(b).
Finally, note the unbounded exponential update in Eq. 4 which is an undesirable property.
This mayeasily cause γ t to explode as when expanded γ t = γ t − 1 exp( E val,t ) = γ 0 exp( E val, 0 + E val, 1 + 208 ... + E val,t − 1 + E val,t ) , and if E val,t > 0 for quite a few number of epochs, γ t will become so largethat even if E val,t < 0 in the subsequent epochs, it may not reduce to a desired level.
We remedythis by constraining γ t to an upper bound γ max when γ > 0 (focal loss) and lower bound of γ minwhen γ < 0 (inverse-focal loss).
One may argue that γ max and γ min are again hyper-parametera;however, note that they do not require any special fine-tuning and their sole purpose is to stop γ fromexploding.
For all our experiments, we use γ min = − 2 and γ max = 20 .
For comparison of resultswith γ max = 20 , γ max = 50 and γ max = ∞ , please refer to Appendix M.
The choice of γ min = − 2is based on the observation that on ImageNet values beyond − 2 led to unstable training for ResNet-50.
Additionally, note that although λ is included in the exponent of Eq. 4, we find λ = 1 to work wellfor all our experiments, therefore, one can treat AdaFocal to be free of this hyper-parameter.
Therefore, the final loss function for AdaFocal is given by
[Equation]
where the gamma-update rules, along with the complete algorithm, is presented in Algorithm 1.
Experiments
Experimental setup We evaluate the performance of our proposed method on image and textclassification tasks.
For image classification, we use CIFAR-10, CIFAR-100 [8], Tiny-ImageNet [2],and ImageNet [26] to analyze the calibration of ResNet50, ResNet-100 {+[5], Wide-ResNet-26-
 [32],and+} DenseNet-121 [7] models.
For text classification, we use the 20 Newsgroup dataset [13] andtrain the Global Pooling CNN model [14].
More details about datasets, models and experimentalconfigurations are given in Appendix E.
In the paper, we report results using only ECE EM , whereasother ECE metrics are reported in Appendix.
Baseline Among calibration-during-training methods we use MMCE [12], Brier loss [1], Labelsmoothing (LS-0.05) [19] and sample-dependent focal loss FLSD-53 as baselines.
For post hoccalibration, we report the effect of temperature scaling, ensemble temperature scaling (ETS) [33] andspline fitting [4] on top of these methods.
For temperature scaling, we select the optimal temperature∈ (0 , 10] (step size 0 . 1 ) that gives the lowest ECE EM on the validation set.
Results.
In Figure 4, we compare AdaFocal against cross entropy (CE) and FLSD-53 for ResNet50 trained on various small to large-scale image datasets.
Among various focal losses, we chose
FLSD-53 as our baseline because it was shown to be consistently better than MMCE, Brier Lossand Label smoothing [18] across many datasets-model pairs.
The figure plots the test set error and
ECE EW .
In Figure 5, for ResNet-50 on CIFAR-10 and ImageNet, we plot (1) the calibration statistics
[Figure]
if | γ t +1 ,i | <
S th then
[Figure]
focal
[Figure]
E val =
C val − A val of the
First, we observe that for CIFAR-10, CIFAR-100 and Tiny-ImageNet, FLDS-53 is much bettercalibrated than CE.
This is because, as shown in Fig.
5(a) for ResNet-50, CIFAR-10, CE is overconfident compared to FLSD-53 in almost every bin.
For ImageNet, however, the behaviour isreversed: FLSD-53 is poorly calibrated than CE.
The reason, as shown in Figure 5(b), is that due tothe use of high values of γ ( = 5 , 3 ), FLSD-53 makes the model largely under-confident in each bin,leading to a overall high calibration error.
AdaFocal, on the other hand, is always well calibrated for dataset-model pairs while achieving similar accuracy
[Figure]
Next, for ResNet-50 CIFAR-10 we find γ t to be closer to 1 for higher bins and closer to 20 forlower bins.
These gammas result in better calibration than γ t = 5 , 3 of FLSD-53.
For ImageNet,on the other hand, we find, except bin-14, AdaFocal switches to inverse-focal loss at some pointin the training.
This makes sense because for ImageNet even cross entropy ( γ t = 0 ) suffers fromunder-confidence, therefore, AdaFocal, starting from γ t = 1 , approaches cross entropy ( γ = 0 ) toultimately switch to inverse-focal loss ( γ t < 0 ) at S th = 0 .
Since, in retrospect, we already knowthat switching to inverse focal loss is beneficial for ImageNet, switching early at S th = 0 .
5 helpsreach a better level of calibration early.
However, ultimately S th = 0 .
and S th = 0 .
5 lead to thesame calibration error.
Overall, these experiments confirm that during training AdaFocal being awareof the network’s current under/over-confidence is able to guide the γ t s to values that maintain a wellcalibrated model at all times.
Also note that for an unseen dataset-model pair there’s no way to knowapriori which γ will perform better but AdaFocal will automatically find these appropriate valuesthus avoiding an expensive and extensive hyper-parameter search.
Comparison of test set error and ECE EM for rest of the experiments are shown in Table 2 and Table1 respectively.
From Table 1, we observe that prior to temperature scaling AdaFocal outperforms thebaseline methods by a substantial margin in all 11 cases.
Post-temperature scaling, AdaFocal achievesthe lowest calibration error in 9 out of the 11 experiments.
Also observe that the optimal temperaturesare mostly close to 1 indicating that AdaFocal produces innately calibrated models from trainingitself.
Effects of other post hoc calibration methods (ETS and spline fitting) are shown in Appendix
F. Again, we observe that pre-calibrated models from AdaFocal benefit from post processing tofurther lower the overall calibration error.
Besides ECE EM , the consistency of the results across othercalibration metrics is shown through ECE EW in Appendix G, and ECE DEBIAS (15 and 30 bins), ECE SWEEP (equal-width and equal-mass) in Appendix H. Statistical significance of the results isconfirmed through ECE EW error bars in Appendix I where mean and standard deviations are plottedover 5 runs.
Number of bins: For AdaFocal, we experiment with 5, 10, 15, 20, 30, and 50 equal-mass binsused during training to draw calibration statistics form the validation set.
As reported in Appendix J,the best results are in the range of 10 to 20.
Performance degrades when the number of bins are toosmall ( < 10 ) or too large ( > 20 ).
Therefore, for AdaFocal we use 15 bins as well during training.
For computing ECE metrics, if not stated explicitly, we always use 15 bins.
[Table] [Table]
Out-of-Distribution (OOD) detection: Following [18], we report the performance of AdaFocalon an OOD detection task.
We train ResNet-110 and Wide-ResNet26-10 on CIFAR-10 as the indistribution data and test on SVHN [22] and CIFAR-10-C [6] (with level 5 Gaussian noise corruption)as
OOD data.
Using entropy of the softmax as the measure of uncertainty, the corresponding ROCplots are shown in Figure 6 and AUROC scores are reported in Table 11 in Appendix K. We seethat models trained with AdaFocal outperform focal loss γ = 3 (FL-3) and FLSD-53.
For the exact
AUROC scores, please refer to Appendix K. These results further highlight the benefits of inherentlycalibrated model produced by AdaFocal as post-hoc calibration techniques such as temperaturescaling, as shown in the figure, is ineffective under distributional shift [27].
[Figure]
Conclusion
In this work, we first revisit the calibration properties of regular focal loss to highlight the downside ofusing a fixed γ for all samples.
Particularly, by studying the calibration behaviour of different samplesin different probability region, we find that there’s no single γ that achieves the best calibrationover the entire region.
We use this observation to motivate the selection of γ independently foreach sample (or group of samples) based on the knowledge of models’s under/over-confidence.
Wepropose a calibration-aware adaptive strategy called AdaFocal that accounts for such information andupdates the γ t at every step based on γ t − 1 from the previous step and the magnitude of the model’sunder/over-confidence.
We find AdaFocal to perform consistently better across different dataset-modelpairs producing innately calibrated models that benefit further from post-hoc calibration techniques.
Additionally, we find models trained with AdaFocal to be much better at out-of-distribution detection.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] in Appendix A. (c) Did you discuss any potential negative societal impacts of your work?
[Yes] in Ap-
pendix A. (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
If you are including theoretical results...
(a) Did you state the
[Table]
[Table]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes] As part of the supplementary material and details are mentioned in Appendix E.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] see Appendix E.
(c) Did you report error bars (e g., with respect to the random seed after running experiments multiple times)?
[Yes] see Appendix I.
(d) Did you include the total amount of compute and the type of resources used (e g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] in Appendix E.4.
If you are using existing assets (e g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] in Appendix E. (b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you’reusing/curating?
[Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]