Original: T HE I NTRIGUING E FFECTS OF F OCAL L OSS ON THE C ALIBRATION OF D EEP N EURAL N ETWORKS
Revised: T HE I NTRIGUING E FFECTS OF F OCAL L OSS ON THE C ALIBRATION OF D EEP N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A
Revised: A

Original: BSTRACT
Revised: BSTRACT

Original: Miscalibration – a mismatch between a model’s conﬁdence and its correctness – of
Revised: Miscalibration – a mismatch between a model’s conﬁdence and its correctness – of

Original: Deep Neural Networks (DNNs) makes their predictions hard for downstream components to trust.
Revised: Deep Neural Networks (DNNs) makes their predictions hard for downstream components to trust.

Original: Ideally, we want networks to be accurate, calibrated and conﬁdent.
Revised: Ideally, we want networks to be accurate, calibrated and conﬁdent.

Original: Temperature scaling, the most popular calibration approach, will calibrate a DNN without affecting its accuracy but it will also make its correct predictions underconﬁdent.
Revised: Temperature scaling, the most popular calibration approach, will calibrate a DNN without affecting its accuracy but it will also make its correct predictions underconﬁdent.

Original: In this paper, we show that replacing the widely used cross-entropy loss with focal loss allows us to learn models that are already very well calibrated.
Revised: In this paper, we show that replacing the widely used cross-entropy loss with focal loss allows us to learn models that are already very well calibrated.

Original: When combined with temperature scaling, focal loss, whilst preserving accuracy and yielding state-of-the-art calibrated models, also preserves the conﬁdence of the model’s correct predictions, which is extremely desirable for downstream tasks.
Revised: When combined with temperature scaling, focal loss, whilst preserving accuracy and yielding state-of-the-art calibrated models, also preserves the conﬁdence of the model’s correct predictions, which is extremely desirable for downstream tasks.

Original: We provide a thorough analysis of the factors causing miscalibration and use the insights we glean from this to theoretically justify the empirically excellent perfor- mance of focal loss.
Revised: We provide a thorough analysis of the factors causing miscalibration and use the insights we glean from this to theoretically justify the empirically excellent perfor- mance of focal loss.

Original: We perform extensive experiments on a variety of computer vision (CIFAR- and
Revised: We perform extensive experiments on a variety of computer vision (CIFAR- and

Original: NLP (SST
Revised: NLP (SST

Original: 20 Newsgroup) datasets, and with a wide variety of different network architectures, and show that our approach achieves state-of the-art accuracy and calibration in almost all cases.
Revised: 20 Newsgroup) datasets, and with a wide variety of different network architectures, and show that our approach achieves state-of the-art accuracy and calibration in almost all cases.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks have dominated computer vision and machine learning in recent years, and this has led to their widespread deployment in real-world systems (Cao et al., 2018;
Revised: Deep neural networks have dominated computer vision and machine learning in recent years, and this has led to their widespread deployment in real-world systems (Cao et al., 2018;

Original: Chen et al., 2018;
Revised: Chen et al., 2018;

Original: Kamilaris and Prenafeta-Boldú, 2018; Ker et al., 2018; Wang et al., 2018).
Revised: Kamilaris and Prenafeta-Boldú, 2018; Ker et al., 2018; Wang et al., 2018).

Original: State-of-the-art networks achieve high levels of accuracy for many tasks.
Revised: State-of-the-art networks achieve high levels of accuracy for many tasks.

Original: However, many current multi-class classiﬁcation networks in particular are poorly calibrated, in the sense that the probability values that they associate with the class labels they predict for different test samples overestimate the likelihoods of those class labels being correct in the real world.
Revised: However, many current multi-class classiﬁcation networks in particular are poorly calibrated, in the sense that the probability values that they associate with the class labels they predict for different test samples overestimate the likelihoods of those class labels being correct in the real world.

Original: This is a major problem, since if networks are routinely overconﬁdent, then downstream components cannot trust their predictions.
Revised: This is a major problem, since if networks are routinely overconﬁdent, then downstream components cannot trust their predictions.

Original: The underlying cause is hypothesised to be that these networks’ high capacity leaves them vulnerable to overﬁtting on the negative log-likelihood (NLL) loss they conventionally use during training (Guo* et al., 2017).
Revised: The underlying cause is hypothesised to be that these networks’ high capacity leaves them vulnerable to overﬁtting on the negative log-likelihood (NLL) loss they conventionally use during training (Guo* et al., 2017).

Original: Given the importance of this problem, numerous suggestions for how to address it have been proposed.
Revised: Given the importance of this problem, numerous suggestions for how to address it have been proposed.

Original: Much work has been inspired by early approaches from the pre-deep learning era such as Platt scaling (Platt, 1999), histogram binning (Zadrozny and Elkan, 2001), isotonic regression (Zadrozny and Elkan, 2002), and Bayesian binning and averaging (Naeini et al., 2015; Naeini and Cooper, 2016).
Revised: Much work has been inspired by early approaches from the pre-deep learning era such as Platt scaling (Platt, 1999), histogram binning (Zadrozny and Elkan, 2001), isotonic regression (Zadrozny and Elkan, 2002), and Bayesian binning and averaging (Naeini et al., 2015; Naeini and Cooper, 2016).

Original: As deep learning has become more dominant, works have begun to directly target the calibration of deep networks.
Revised: As deep learning has become more dominant, works have begun to directly target the calibration of deep networks.

Original: For example, Guo et al.
Revised: For example, Guo et al.

Original: (Guo* et al., 2017) have popularised a modern variant of
Revised: (Guo* et al., 2017) have popularised a modern variant of

Original: Platt scaling known as temperature scaling , which works by dividing a network’s logits by a scalar
Revised: Platt scaling known as temperature scaling , which works by dividing a network’s logits by a scalar

Original: T > 0 (learnt on a validation subset) prior to performing softmax.
Revised: T > 0 (learnt on a validation subset) prior to performing softmax.

Original: Temperature scaling has the desirable property that it can improve the calibration of a network without in any way affecting its accuracy.
Revised: Temperature scaling has the desirable property that it can improve the calibration of a network without in any way affecting its accuracy.

Original: Mozafari et al.
Revised: Mozafari et al.

Original: (Mozafari et al., 2018) noted the downsides of using cross-entropy loss with temperature scaling, and proposed an alternative loss called Attended-NLL that helps temperature scaling achieve better calibration.
Revised: (Mozafari et al., 2018) noted the downsides of using cross-entropy loss with temperature scaling, and proposed an alternative loss called Attended-NLL that helps temperature scaling achieve better calibration.

Original: More recently, Shrikumar and Kundaje (Shrikumar and Kundaje, 2019) have proposed an extension to temperature scaling that adds class-speciﬁc bias parameters to help eliminate systematic bias when performing domain adaptation.
Revised: More recently, Shrikumar and Kundaje (Shrikumar and Kundaje, 2019) have proposed an extension to temperature scaling that adds class-speciﬁc bias parameters to help eliminate systematic bias when performing domain adaptation.

Original: Separately, Hendrycks et al.
Revised: Separately, Hendrycks et al.

Original: (Hendrycks et al., 2019) have studied the effects of pre-training (vs. training from scratch) on model robustness and uncertainty: they make the interesting observation that because long periods of training can cause a network to become miscalibrated, tuning a pre-trained network, which facilitates faster convergence, can seemingly lead to a more calibrated model.
Revised: (Hendrycks et al., 2019) have studied the effects of pre-training (vs. training from scratch) on model robustness and uncertainty: they make the interesting observation that because long periods of training can cause a network to become miscalibrated, tuning a pre-trained network, which facilitates faster convergence, can seemingly lead to a more calibrated model.

Original: Notably, since their approach complements temperature scaling, the two techniques can also be used together to achieve even better calibration overall.
Revised: Notably, since their approach complements temperature scaling, the two techniques can also be used together to achieve even better calibration overall.

Original: Whilst temperature scaling’s simplicity and effectiveness have made it a popular and state-of-the-art network calibration technique, it does have downsides.
Revised: Whilst temperature scaling’s simplicity and effectiveness have made it a popular and state-of-the-art network calibration technique, it does have downsides.

Original: For example, whilst it scales the logits to reduce the network’s conﬁdence in incorrect predictions, this also slightly reduces the network’s conﬁdence in predictions that were actually correct.
Revised: For example, whilst it scales the logits to reduce the network’s conﬁdence in incorrect predictions, this also slightly reduces the network’s conﬁdence in predictions that were actually correct.

Original: By contrast, Kumar et al.
Revised: By contrast, Kumar et al.

Original: (Kumar et al., 2018) eschew temperature scaling altogether in favour of minimising a differentiable proxy for calibration error at training time, called Maximum Mean Calibration Error (MMCE).
Revised: (Kumar et al., 2018) eschew temperature scaling altogether in favour of minimising a differentiable proxy for calibration error at training time, called Maximum Mean Calibration Error (MMCE).

Original: However, they also use temperature scaling as a post-processing step to obtain better results than cross-entropy followed by temperature scaling (Guo* et al., 2017).
Revised: However, they also use temperature scaling as a post-processing step to obtain better results than cross-entropy followed by temperature scaling (Guo* et al., 2017).

Original: In this paper, we propose a technique for improving network calibration that works by replacing the cross-entropy loss conventionally used when training multi-class classiﬁcation networks with the focal loss proposed by Lin et al.
Revised: In this paper, we propose a technique for improving network calibration that works by replacing the cross-entropy loss conventionally used when training multi-class classiﬁcation networks with the focal loss proposed by Lin et al.

Original: (Lin et al., 2017) for dense object detection.
Revised: (Lin et al., 2017) for dense object detection.

Original: Since focal loss, as shown in §4, is dependent on a hyperparameter, γ , which needs to be cross-validated, we also provide a theoretically justiﬁed way to choose γ automatically for each sample and show it to outperform all the baseline models.
Revised: Since focal loss, as shown in §4, is dependent on a hyperparameter, γ , which needs to be cross-validated, we also provide a theoretically justiﬁed way to choose γ automatically for each sample and show it to outperform all the baseline models.

Original: Informally, the intuition behind using focal loss is to direct the network’s attention during training towards samples for which it is currently predicting a low probability value for the correct class, since trying to reduce the NLL on samples for which it is currently predicting a high probability value for the correct class is liable to lead to NLL overﬁtting, and thereby miscalibration (Guo* et al., 2017).
Revised: Informally, the intuition behind using focal loss is to direct the network’s attention during training towards samples for which it is currently predicting a low probability value for the correct class, since trying to reduce the NLL on samples for which it is currently predicting a high probability value for the correct class is liable to lead to NLL overﬁtting, and thereby miscalibration (Guo* et al., 2017).

Original: More formally, we show in §4 that focal loss can be seen as implicitly regularising the weights of the network by causing the gradient norm to be lower than it would have been with cross-entropy loss as training proceeds, which we would theoretically expect to reduce overﬁtting and improve the calibration of the network.
Revised: More formally, we show in §4 that focal loss can be seen as implicitly regularising the weights of the network by causing the gradient norm to be lower than it would have been with cross-entropy loss as training proceeds, which we would theoretically expect to reduce overﬁtting and improve the calibration of the network.

Original: In §5, we perform extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (20 Newsgroups/SST) datasets, and with a wide variety of different network architectures (e.g. ResNet-110/50, Wide-ResNet, DenseNet), to show that this is indeed the case.
Revised: In §5, we perform extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (20 Newsgroups/SST) datasets, and with a wide variety of different network architectures (e.g. ResNet-110/50, Wide-ResNet, DenseNet), to show that this is indeed the case.

Original: Our experiments show that in almost all cases, DNNs trained with focal loss are more calibrated than those trained with cross-entropy loss, MMCE, and Brier loss Brier (1950).
Revised: Our experiments show that in almost all cases, DNNs trained with focal loss are more calibrated than those trained with cross-entropy loss, MMCE, and Brier loss Brier (1950).

Original: Moreover, since our approach, like that of (Hendrycks et al., 2019), is complementary to the temperature scaling, signiﬁcant improvements in calibration over temperature scaling alone, and state-of-the-art results, can be achieved by training with focal loss and then performing temperature scaling.
Revised: Moreover, since our approach, like that of (Hendrycks et al., 2019), is complementary to the temperature scaling, signiﬁcant improvements in calibration over temperature scaling alone, and state-of-the-art results, can be achieved by training with focal loss and then performing temperature scaling.

Original:  P ROBLEM S ETUP
Revised:  P ROBLEM S ETUP

Original: Let D = (cid:104) ( x i , y i ) (cid:105) Ni =1 denote a dataset consisting of samples from a joint distribution D ( X , Y ) , where for each sample i , x i ∈
Revised: Let D = (cid:104) ( x i , y i ) (cid:105) Ni =1 denote a dataset consisting of samples from a joint distribution D ( X , Y ) , where for each sample i , x i ∈

Original: X is the input and y i ∈ Y = { 1 , 2 , ..., K } is the ground-truth class label.
Revised: X is the input and y i ∈ Y = { 1 , 2 , ..., K } is the ground-truth class label.

Original: Let ˆ p
Revised: Let ˆ p

Original: i,y = f θ ( y | x i ) be the probability that a neural network f with model parameters θ predicts for a class y on a given input x i .
Revised: i,y = f θ ( y | x i ) be the probability that a neural network f with model parameters θ predicts for a class y on a given input x i .

Original: The class that f predicts for x i is computed as ˆ y i = argmax y ∈Y ˆ p
Revised: The class that f predicts for x i is computed as ˆ y i = argmax y ∈Y ˆ p

Original: i,y , and the predicted conﬁdence as ˆ p i = max y ∈Y ˆ p
Revised: i,y , and the predicted conﬁdence as ˆ p i = max y ∈Y ˆ p

Original: i,y .
Revised: i,y .

Original: Note that the conﬁdence, by deﬁnition, does not depend on the ground-truth label.
Revised: Note that the conﬁdence, by deﬁnition, does not depend on the ground-truth label.

Original: The network is said to be perfectly calibrated when, for each sample ( x i , y i ) ∈ D , the conﬁdence
Revised: The network is said to be perfectly calibrated when, for each sample ( x i , y i ) ∈ D , the conﬁdence

Original: ˆ p
Revised: ˆ p

Original: i is equal to the model accuracy P (ˆ y i
Revised: i is equal to the model accuracy P (ˆ y i

Original: = y i ) , i.e. the probability that the predicted class is correct.
Revised: = y i ) , i.e. the probability that the predicted class is correct.

Original: For instance, of all the samples that a perfectly calibrated neural network classiﬁes with a conﬁdence of 0 . 8 , 80% should be correctly predicted.
Revised: For instance, of all the samples that a perfectly calibrated neural network classiﬁes with a conﬁdence of 0 . 8 , 80% should be correctly predicted.

Original: One of the most popular metrics used to measure model calibration is the expected calibration error (ECE) (Naeini et al., 2015), deﬁned as the expected absolute difference between the model’s conﬁdence and its accuracy, i.e. E ( x i ,y i ) ∼D (cid:2)
Revised: One of the most popular metrics used to measure model calibration is the expected calibration error (ECE) (Naeini et al., 2015), deﬁned as the expected absolute difference between the model’s conﬁdence and its accuracy, i.e. E ( x i ,y i ) ∼D (cid:2)

Original: | P (ˆ y i =
Revised: | P (ˆ y i =

Original: y i )
Revised: y i )

Original: − ˆ p i | (cid:3) .
Revised: − ˆ p i | (cid:3) .

Original: Since calculating P (ˆ y i = y i ) is infeasible, the ECE cannot be directly computed.
Revised: Since calculating P (ˆ y i = y i ) is infeasible, the ECE cannot be directly computed.

Original: A workaround is to divide the interval [0 , 1] into
Revised: A workaround is to divide the interval [0 , 1] into

Original: M equispaced bins, where the i th bin is the interval (cid:0) i − 1 M , iM (cid:3) .
Revised: M equispaced bins, where the i th bin is the interval (cid:0) i − 1 M , iM (cid:3) .

Original: Let B i denote the set of samples with conﬁdence scores belonging to the i th bin.
Revised: Let B i denote the set of samples with conﬁdence scores belonging to the i th bin.

Original: Then the accuracy A i of this bin is computed as
Revised: Then the accuracy A i of this bin is computed as

Original: A i = 1 | B i | (cid:80) j ∈ B i 1 (ˆ y j = y j ) , where 1 is the indicator function, and ˆ y j
Revised: A i = 1 | B i | (cid:80) j ∈ B i 1 (ˆ y j = y j ) , where 1 is the indicator function, and ˆ y j

Original: and y j are respectively the predicted and ground-truth labels for the j th sample.
Revised: and y j are respectively the predicted and ground-truth labels for the j th sample.

Original: Similarly, the conﬁdence C i of the i th bin is computed as C i = 1 | B i | (cid:80) j ∈ B
Revised: Similarly, the conﬁdence C i of the i th bin is computed as C i = 1 | B i | (cid:80) j ∈ B

Original: i ˆ p j , i.e. C
Revised: i ˆ p j , i.e. C

Original: i is the average conﬁdence of all samples in the bin.
Revised: i is the average conﬁdence of all samples in the bin.

Original: The
Revised: The

Original: ECE can be approximated as a weighted average of the absolute difference between the accuracy and conﬁdence of each bin:
Revised: ECE can be approximated as a weighted average of the absolute difference between the accuracy and conﬁdence of each bin:

Original: [Equation]
Revised: [Equation]

Original: A similar metric, the maximum calibration error (MCE) (Naeini et al., 2015), is deﬁned as the maximum absolute difference between the conﬁdence and accuracy of each bin:
Revised: A similar metric, the maximum calibration error (MCE) (Naeini et al., 2015), is deﬁned as the maximum absolute difference between the conﬁdence and accuracy of each bin:

Original: [Equation]
Revised: [Equation]

Original: A common way of visually exploring the calibration of a model is to use a reliability plot (NiculescuMizil and Caruana, 2005), which plots the accuracies of the various conﬁdence bins as a bar chart (see Figure 2).
Revised: A common way of visually exploring the calibration of a model is to use a reliability plot (NiculescuMizil and Caruana, 2005), which plots the accuracies of the various conﬁdence bins as a bar chart (see Figure 2).

Original: Reliability plots not only give a quick visual indication of a model’s MCE (but not
Revised: Reliability plots not only give a quick visual indication of a model’s MCE (but not

Original: ECE, since the numbers of samples in the different bins are not shown), but also capture whether or not a model is under-conﬁdent or over-conﬁdent in general.
Revised: ECE, since the numbers of samples in the different bins are not shown), but also capture whether or not a model is under-conﬁdent or over-conﬁdent in general.

Original: For a perfectly calibrated model, the accuracy for each bin will match the conﬁdence, and hence all of the bars will lie on a diagonal.
Revised: For a perfectly calibrated model, the accuracy for each bin will match the conﬁdence, and hence all of the bars will lie on a diagonal.

Original: By contrast, if most of the bars lie above the diagonal, meaning that the model is more accurate than it expects, then it is said to be under-conﬁdent, and if most of the bars lie below the diagonal, then it is said to be over-conﬁdent.
Revised: By contrast, if most of the bars lie above the diagonal, meaning that the model is more accurate than it expects, then it is said to be under-conﬁdent, and if most of the bars lie below the diagonal, then it is said to be over-conﬁdent.

Original: We also consider another metric which we call AdaECE (Adaptive ECE) where the bin sizes are decided such that each bin contains the same number of samples, unlike ECE (1) where every bin size is ﬁxed and might contain very different number of samples (for example, Figure 2).
Revised: We also consider another metric which we call AdaECE (Adaptive ECE) where the bin sizes are decided such that each bin contains the same number of samples, unlike ECE (1) where every bin size is ﬁxed and might contain very different number of samples (for example, Figure 2).

Original: However,
Revised: However,

Original: AdaECE allows better approximation of ECE as it ensures that every approximate bin has the same sample density.
Revised: AdaECE allows better approximation of ECE as it ensures that every approximate bin has the same sample density.

Original: [Equation]
Revised: [Equation]

Original:  W HAT C AUSES M ISCALIBRATION ?
Revised:  W HAT C AUSES M ISCALIBRATION ?

Original: We now discuss why high-capacity neural networks, despite achieving low classiﬁcation errors on well-known datasets, tend to be miscalibrated.
Revised: We now discuss why high-capacity neural networks, despite achieving low classiﬁcation errors on well-known datasets, tend to be miscalibrated.

Original: A key empirical observation made by (Guo* et al., 2017) was that poor calibration of such networks appears to be linked to overﬁtting on the negative log-likelihood (NLL) during training.
Revised: A key empirical observation made by (Guo* et al., 2017) was that poor calibration of such networks appears to be linked to overﬁtting on the negative log-likelihood (NLL) during training.

Original: In this section, we further inspect this observation to provide new insights, and discuss the main factors that play a role in causing the above-mentioned NLL overﬁtting.
Revised: In this section, we further inspect this observation to provide new insights, and discuss the main factors that play a role in causing the above-mentioned NLL overﬁtting.

Original: For the analysis in this section, we train a ResNet-50 network on CIFAR-10 using a training infrastructure PyTorch-CIFAR which is known to produce state-of-the-art accuracy.
Revised: For the analysis in this section, we train a ResNet-50 network on CIFAR-10 using a training infrastructure PyTorch-CIFAR which is known to produce state-of-the-art accuracy.

Original: We train it forepochs using Stochastic Gradient Descent (SGD), with a momentum of 0.9, and a learning rate of 0.
Revised: We train it forepochs using Stochastic Gradient Descent (SGD), with a momentum of 0.9, and a learning rate of 0.

Original: for the ﬁrst 150 epochs, 0.01 for the next 100 epochs and 0.001 for the last 100 epochs.
Revised: for the ﬁrst 150 epochs, 0.01 for the next 100 epochs and 0.001 for the last 100 epochs.

Original: We use a mini-batch size of 128.
Revised: We use a mini-batch size of 128.

Original: Following standard practice, we minimise cross-entropy loss, which, for the i th training sample, is computed as L c = − log ˆ p i,y i where ˆ p
Revised: Following standard practice, we minimise cross-entropy loss, which, for the i th training sample, is computed as L c = − log ˆ p i,y i where ˆ p

Original: i,y i is the probability assigned by the network to the correct class for the i th sample.
Revised: i,y i is the probability assigned by the network to the correct class for the i th sample.

Original: It is interesting to note that NLL (a.k.a.
Revised: It is interesting to note that NLL (a.k.a.

Original: cross-entropy) loss is a differentiable proxy for the actual metric that we aim to minimise by training a model, namely the classiﬁcation error.
Revised: cross-entropy) loss is a differentiable proxy for the actual metric that we aim to minimise by training a model, namely the classiﬁcation error.

Original: However, there are certain differences between minimising the NLL and minimising the classiﬁcation error.
Revised: However, there are certain differences between minimising the NLL and minimising the classiﬁcation error.

Original: For instance, the NLL is minimised when for every training sample i , ˆ p
Revised: For instance, the NLL is minimised when for every training sample i , ˆ p

Original: i,y i = 1 , whereas the classiﬁcation error is minimised if for every sample i , ˆ p
Revised: i,y i = 1 , whereas the classiﬁcation error is minimised if for every sample i , ˆ p

Original: i,y i >
Revised: i,y i >

Original: ˆ p
Revised: ˆ p

Original: i,y for all y (cid:54) =
Revised: i,y for all y (cid:54) =

Original: y i .
Revised: y i .

Original: This indicates that even when the classiﬁcation error is zero, the NLL can be positive, and the optimisation algorithm could in that case try to reduce it to zero by further increasing the value of ˆ p
Revised: This indicates that even when the classiﬁcation error is zero, the NLL can be positive, and the optimisation algorithm could in that case try to reduce it to zero by further increasing the value of ˆ p

Original: i,y i for each sample.
Revised: i,y i for each sample.

Original: To study how miscalibration occurs during training, we plot the average NLL for the train and test sets at each training epoch in Figures 1(a) and 1(b).
Revised: To study how miscalibration occurs during training, we plot the average NLL for the train and test sets at each training epoch in Figures 1(a) and 1(b).

Original: We also plot the average NLL and the entropy of the softmax distribution produced by the network for the correctly and incorrectly classiﬁed samples.
Revised: We also plot the average NLL and the entropy of the softmax distribution produced by the network for the correctly and incorrectly classiﬁed samples.

Original: In Figure 1(c), we plot the classiﬁcation errors on the train and test sets, along with the test set ECE.
Revised: In Figure 1(c), we plot the classiﬁcation errors on the train and test sets, along with the test set ECE.

Original: Curse of misclassiﬁed samples: Figures 1(a) and 1(b) show that although the average train NLL (for both correctly and incorrectly classiﬁed training samples) broadly decreases throughout training, after the 150 th epoch (where the learning rate drops by a factor of 10 ), there is a marked rise in the average test NLL, indicating that the network starts to overﬁt on average NLL.
Revised: Curse of misclassiﬁed samples: Figures 1(a) and 1(b) show that although the average train NLL (for both correctly and incorrectly classiﬁed training samples) broadly decreases throughout training, after the 150 th epoch (where the learning rate drops by a factor of 10 ), there is a marked rise in the average test NLL, indicating that the network starts to overﬁt on average NLL.

Original: However, the increase in average test NLL is caused only by the incorrectly classiﬁed samples , as the average NLL for the correctly classiﬁed samples continues to decrease even after the 150 th epoch.
Revised: However, the increase in average test NLL is caused only by the incorrectly classiﬁed samples , as the average NLL for the correctly classiﬁed samples continues to decrease even after the 150 th epoch.

Original: We also observe that after epoch 150 , the test set ECE rises, indicating that the network is becoming miscalibrated.
Revised: We also observe that after epoch 150 , the test set ECE rises, indicating that the network is becoming miscalibrated.

Original: This corroborates the observation in (Guo* et al., 2017) that miscalibration and NLL overﬁtting are linked.
Revised: This corroborates the observation in (Guo* et al., 2017) that miscalibration and NLL overﬁtting are linked.

Original: [Figure]
Revised: [Figure]

Original: Peak at the wrong place: We further observe that the entropies of the softmax distributions for both the correctly and incorrectly classiﬁed test samples decrease throughout training (in other words, the distributions get peakier).
Revised: Peak at the wrong place: We further observe that the entropies of the softmax distributions for both the correctly and incorrectly classiﬁed test samples decrease throughout training (in other words, the distributions get peakier).

Original: This observation, coupled with the one we made above, indicates that for the wrongly classiﬁed test samples, the network gradually becomes more and more conﬁdent about its incorrect predictions.
Revised: This observation, coupled with the one we made above, indicates that for the wrongly classiﬁed test samples, the network gradually becomes more and more conﬁdent about its incorrect predictions.

Original: Finally, we notice that the training set classiﬁcation error drops from around 20% to nearly 0% at epoch 150 , and further drops to exactly 0% from epoch 250 onwards.
Revised: Finally, we notice that the training set classiﬁcation error drops from around 20% to nearly 0% at epoch 150 , and further drops to exactly 0% from epoch 250 onwards.

Original: The network thus has a very low training set classiﬁcation error throughout the period of miscalibration.
Revised: The network thus has a very low training set classiﬁcation error throughout the period of miscalibration.

Original: Favouring correctly classiﬁed samples: One potential explanation for the above observations is that after the 150 th training epoch, the network classiﬁes almost all of the training samples correctly, and hence, in order to further minimise the training NLL, it focuses on increasing the conﬁdence of its correct predictions, rather than focusing on the incorrectly classiﬁed samples.
Revised: Favouring correctly classiﬁed samples: One potential explanation for the above observations is that after the 150 th training epoch, the network classiﬁes almost all of the training samples correctly, and hence, in order to further minimise the training NLL, it focuses on increasing the conﬁdence of its correct predictions, rather than focusing on the incorrectly classiﬁed samples.

Original: In order to experimentally validate this, we use two observations.
Revised: In order to experimentally validate this, we use two observations.

Original: Firstly, we divide the conﬁdence range [0 , 1] into 25 bins, and present reliability plots computed on the training set at epochs 100 , 200 , 300 and 350 (see the top row of Figure 2).
Revised: Firstly, we divide the conﬁdence range [0 , 1] into 25 bins, and present reliability plots computed on the training set at epochs 100 , 200 , 300 and 350 (see the top row of Figure 2).

Original: In Figure 2, we also show the percentage of samples in each conﬁdence bin.
Revised: In Figure 2, we also show the percentage of samples in each conﬁdence bin.

Original: It is quite clear from these plots that over time, the network gradually pushes all of the training samples towards the highest conﬁdence bin.
Revised: It is quite clear from these plots that over time, the network gradually pushes all of the training samples towards the highest conﬁdence bin.

Original: Furthermore, even though the network has achieved 100% accuracy on the training set by epoch 300 , it still pushes some of the samples lying in lower conﬁdence bins to the highest conﬁdence bin by epoch 350 .
Revised: Furthermore, even though the network has achieved 100% accuracy on the training set by epoch 300 , it still pushes some of the samples lying in lower conﬁdence bins to the highest conﬁdence bin by epoch 350 .

Original: It thus keeps on increasing the conﬁdence of correct predictions even after having reached perfect accuracy on the training set.
Revised: It thus keeps on increasing the conﬁdence of correct predictions even after having reached perfect accuracy on the training set.

Original: Secondly, we observe from Figure 1(c) that between training epochs 150 and 250 , i.e. between the two points where there are drops in the learning rate, the training set classiﬁcation error rises slightly, before dropping to exactly 0 after epoch 250 .
Revised: Secondly, we observe from Figure 1(c) that between training epochs 150 and 250 , i.e. between the two points where there are drops in the learning rate, the training set classiﬁcation error rises slightly, before dropping to exactly 0 after epoch 250 .

Original: This further conﬁrms our hypothesis that in this period, the network focuses more on increasing the conﬁdence of its correct predictions rather than increasing the classiﬁcation accuracy on the training set.
Revised: This further conﬁrms our hypothesis that in this period, the network focuses more on increasing the conﬁdence of its correct predictions rather than increasing the classiﬁcation accuracy on the training set.

Original: [Figure]
Revised: [Figure]

Original: Weight magniﬁcation:
Revised: Weight magniﬁcation:

Original: The above-mentioned increase in conﬁdence of the network’s predictions can happen if the network increases its weights to increase the magnitudes of the logits.
Revised: The above-mentioned increase in conﬁdence of the network’s predictions can happen if the network increases its weights to increase the magnitudes of the logits.

Original: We explore this in §4.
Revised: We explore this in §4.

Original: This increase in the network’s conﬁdence during training is a key cause of miscalibration.
Revised: This increase in the network’s conﬁdence during training is a key cause of miscalibration.

Original:  I MPROVING C ALIBRATION USING F OCAL L OSS
Revised:  I MPROVING C ALIBRATION USING F OCAL L OSS

Original: As discussed above, overﬁtting on NLL, which is observed as the network grows more conﬁdent on all of its predictions irrespective of their correctness, is strongly related to poor calibration in neural networks.
Revised: As discussed above, overﬁtting on NLL, which is observed as the network grows more conﬁdent on all of its predictions irrespective of their correctness, is strongly related to poor calibration in neural networks.

Original: One cause of this phenomenon is that the cross-entropy objective minimises the difference between the softmax distribution and the ground-truth one-hot encoding over an entire mini-batch, irrespective of how well a network classiﬁes individual samples in the mini-batch.
Revised: One cause of this phenomenon is that the cross-entropy objective minimises the difference between the softmax distribution and the ground-truth one-hot encoding over an entire mini-batch, irrespective of how well a network classiﬁes individual samples in the mini-batch.

Original: In this work, we study an alternative loss function called focal loss (Lin et al., 2017), which tackles this problem by weighting loss components generated from individual samples in a mini-batch by how well the model classiﬁes them.
Revised: In this work, we study an alternative loss function called focal loss (Lin et al., 2017), which tackles this problem by weighting loss components generated from individual samples in a mini-batch by how well the model classiﬁes them.

Original: Focal loss for the i th sample in the training set is computed as
Revised: Focal loss for the i th sample in the training set is computed as

Original: L f = − (1 − ˆ p
Revised: L f = − (1 − ˆ p

Original: i,y i ) γ log ˆ p i,y i , where γ ≥ 0 is a user-deﬁned hyperparameter.
Revised: i,y i ) γ log ˆ p i,y i , where γ ≥ 0 is a user-deﬁned hyperparameter.

Original: Focal loss attributes more importance to samples that the network misclassiﬁes than to ones that are correctly classiﬁed.
Revised: Focal loss attributes more importance to samples that the network misclassiﬁes than to ones that are correctly classiﬁed.

Original: In this paper, we study properties of focal loss that help in calibration.
Revised: In this paper, we study properties of focal loss that help in calibration.

Original: Empirical Observations:
Revised: Empirical Observations:

Original: In order to analyse the behaviour of neural networks trained on focal loss, we use the same framework as mentioned above, and train four ResNet-50 networks on CIFAR-10, one using cross-entropy loss, and three using focal loss with γ = 1 , 2 or 3 .
Revised: In order to analyse the behaviour of neural networks trained on focal loss, we use the same framework as mentioned above, and train four ResNet-50 networks on CIFAR-10, one using cross-entropy loss, and three using focal loss with γ = 1 , 2 or 3 .

Original: We then observe various metrics throughout the training period of these networks, for comparison.
Revised: We then observe various metrics throughout the training period of these networks, for comparison.

Original: In Figures 3(a), (b) and (c), we plot the average NLL over the test set, and for the correctly and incorrectly classiﬁed test samples, respectively, at each training epoch.
Revised: In Figures 3(a), (b) and (c), we plot the average NLL over the test set, and for the correctly and incorrectly classiﬁed test samples, respectively, at each training epoch.

Original: In Figure 3(d), we plot the softmax entropy for the misclassiﬁed test samples.
Revised: In Figure 3(d), we plot the softmax entropy for the misclassiﬁed test samples.

Original: In Figures 3(e) and (f), we plot the average probability value ˆ p i,y i predicted for the ground-truth class, for correctly and incorrectly classiﬁed test samples, respectively.
Revised: In Figures 3(e) and (f), we plot the average probability value ˆ p i,y i predicted for the ground-truth class, for correctly and incorrectly classiﬁed test samples, respectively.

Original: In Figure 3(g), we plot the test set ECE for all four models against the training epoch.
Revised: In Figure 3(g), we plot the test set ECE for all four models against the training epoch.

Original: Figure 3(c) shows that in contrast to the network trained using cross-entropy, for which the NLL for misclassiﬁed test samples increases signiﬁcantly after epoch 150 , the rise in NLL for the networks trained on focal loss is much less severe.
Revised: Figure 3(c) shows that in contrast to the network trained using cross-entropy, for which the NLL for misclassiﬁed test samples increases signiﬁcantly after epoch 150 , the rise in NLL for the networks trained on focal loss is much less severe.

Original: Moreover, from Figure 3(d), we notice that the softmax entropy for misclassiﬁed test samples is consistently (if marginally) higher for focal loss than for cross-entropy.
Revised: Moreover, from Figure 3(d), we notice that the softmax entropy for misclassiﬁed test samples is consistently (if marginally) higher for focal loss than for cross-entropy.

Original: These observations indicate that the focal loss models are less conﬁdent about their incorrect predictions than the cross-entropy model.
Revised: These observations indicate that the focal loss models are less conﬁdent about their incorrect predictions than the cross-entropy model.

Original: From Figure 3(f), we notice that the probabilities predicted for the correct classes for misclassiﬁed test samples are slightly higher for focal loss than for cross-entropy.
Revised: From Figure 3(f), we notice that the probabilities predicted for the correct classes for misclassiﬁed test samples are slightly higher for focal loss than for cross-entropy.

Original: Conversely, Figure 3(e) shows that the probabilities predicted for the correct classes for correctly classiﬁed test samples are somewhat lower than for cross-entropy.
Revised: Conversely, Figure 3(e) shows that the probabilities predicted for the correct classes for correctly classiﬁed test samples are somewhat lower than for cross-entropy.

Original: (Note that in Figure 3(b), the average test NLL values for the focal loss networks on the correctly classiﬁed samples are also somewhat higher than for cross-entropy, which similarly indicates a slight decrease in conﬁdence on those samples.)
Revised: (Note that in Figure 3(b), the average test NLL values for the focal loss networks on the correctly classiﬁed samples are also somewhat higher than for cross-entropy, which similarly indicates a slight decrease in conﬁdence on those samples.)

Original: Nevertheless, for all four models, average conﬁdence on correctly classiﬁed samples remains over 0 .
Revised: Nevertheless, for all four models, average conﬁdence on correctly classiﬁed samples remains over 0 .

Original: 95 by the end of training, which is more than enough to make these
Revised: 95 by the end of training, which is more than enough to make these

Original: [Figure]
Revised: [Figure]

Original: predictions usable by downstream tasks.
Revised: predictions usable by downstream tasks.

Original: Furthermore, the classiﬁcation errors on the test set for all four models are also almost the same (refer ﬁfth row of Table 3 and Table 6).
Revised: Furthermore, the classiﬁcation errors on the test set for all four models are also almost the same (refer ﬁfth row of Table 3).

Original: Given both of these points, along with the ECE plot in Figue 3(g), it is thus reasonable to conclude that on a practical level, the models trained on focal loss are more calibrated, whilst retaining both their classiﬁcation accuracy and the majority of their conﬁdence on correct predictions.
Revised: Given both of these points, along with the ECE plot in Figue 3(g), it is thus reasonable to conclude that on a practical level, the models trained on focal loss are more calibrated, whilst retaining both their classiﬁcation accuracy and the majority of their conﬁdence on correct predictions.

Original: Theoretical Justiﬁcation: As mentioned previously, once a model trained using cross-entropy reaches high accuracy on the training set, the optimisation algorithm may try to further reduce the training NLL by increasing the conﬁdence values for the correctly classiﬁed samples, rather than focusing on the misclassiﬁed samples.
Revised: Theoretical Justiﬁcation: As mentioned previously, once a model trained using cross-entropy reaches high accuracy on the training set, the optimisation algorithm may try to further reduce the training NLL by increasing the conﬁdence values for the correctly classiﬁed samples, rather than focusing on the misclassiﬁed samples.

Original: One way it could achieve this would be to increase the weights of the network to increase the magnitudes of the logits.
Revised: One way it could achieve this would be to increase the weights of the network to increase the magnitudes of the logits.

Original: In fact, this hypothesis would help to explain the observation made in (Guo* et al., 2017) that models trained using some form of weight decay are better calibrated.
Revised: In fact, this hypothesis would help to explain the observation made in (Guo* et al., 2017) that models trained using some form of weight decay are better calibrated.

Original: To verify this hypothesis, we plot the L 2 norm of the weights of the last linear layer for all four networks as a function of the training epoch (see Figure 3(h)).
Revised: To verify this hypothesis, we plot the L 2 norm of the weights of the last linear layer for all four networks as a function of the training epoch (see Figure 3(h)).

Original: It is interesting to note that although the norms of the weights for the models trained on focal loss are initially higher than that for the cross-entropy model, a complete reversal in the ordering of the weight norms occurs between epochs 150 and 250 .
Revised: It is interesting to note that although the norms of the weights for the models trained on focal loss are initially higher than that for the cross-entropy model, a complete reversal in the ordering of the weight norms occurs between epochs 150 and 250 .

Original: In other words, as the network starts to become miscalibrated, the weight norm for the cross-entropy model also starts to become greater than those for the focal loss models.
Revised: In other words, as the network starts to become miscalibrated, the weight norm for the cross-entropy model also starts to become greater than those for the focal loss models.

Original: In practice, the reason for this is that focal loss, by design, starts to act as a regulariser on the network’s weights once the model has gained a certain amount of conﬁdence in its predictions.
Revised: In practice, the reason for this is that focal loss, by design, starts to act as a regulariser on the network’s weights once the model has gained a certain amount of conﬁdence in its predictions.

Original: To better understand this, we start by considering the following proposition (see the Appendix B for a proof).
Revised: To better understand this, we start by considering the following proposition (see the Appendix B for a proof).

Original: Proposition 1.
Revised: Proposition 1.

Original: For focal loss L f and cross-entropy loss L c , the gradients ∂ L f ∂ w = ∂
Revised: For focal loss L f and cross-entropy loss L c , the gradients ∂ L f ∂ w = ∂

Original: L c ∂ w g (ˆ p i,y i , γ ) , where g (ˆ p
Revised: L c ∂ w g (ˆ p i,y i , γ ) , where g (ˆ p

Original: i,y i , γ ) =
Revised: i,y i , γ ) =

Original: (1 − ˆ p
Revised: (1 − ˆ p

Original: i,y i ) γ − γ ˆ p
Revised: i,y i ) γ − γ ˆ p

Original: i,y i (1 − ˆ p
Revised: i,y i (1 − ˆ p

Original: i,y i ) γ − 1 log(ˆ p i,y i ) , γ ≥ 0 is the focal loss hyperparameter, and w are the last layer network parameters.
Revised: i,y i ) γ − 1 log(ˆ p i,y i ) , γ ≥ 0 is the focal loss hyperparameter, and w are the last layer network parameters.

Original: Thus, (cid:13)(cid:13)(cid:13) ∂
Revised: Thus, (cid:13)(cid:13)(cid:13) ∂

Original: L f ∂ w (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)
Revised: L f ∂ w (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)

Original: ∂ L c ∂ w (cid:13)(cid:13)
Revised: ∂ L c ∂ w (cid:13)(cid:13)

Original: iff g
Revised: iff g

Original: (ˆ p i,y i , γ ) ∈ [0 , 1] .
Revised: (ˆ p i,y i , γ ) ∈ [0 , 1] .

Original: To visualise the behaviour of g ( p, γ ) , we plot it against p for various values of γ in Figure 4(a).
Revised: To visualise the behaviour of g ( p, γ ) , we plot it against p for various values of γ in Figure 4(a).

Original: It is quite clear that g ( p, γ ) lies in [0 , 1) for values of p that lie above a certain threshold p 0 .
Revised: It is quite clear that g ( p, γ ) lies in [0 , 1) for values of p that lie above a certain threshold p 0 .

Original: In the interval [0 , p 0 ) , however, g ( p, γ ) > 1 .
Revised: In the interval [0 , p 0 ) , however, g ( p, γ ) > 1 .

Original: In fact, for a given threshold p 0 , we can efﬁciently compute (using the following proposition) a value γ ∗ such that g ( p, γ ∗ ) < 1 for p > p 

Revised: In fact, for a given threshold p 0 , we can efﬁciently compute (using the following proposition) a value γ ∗ such that g ( p, γ ∗ ) < 1 for p > p

Original: (see the Appendix B for a proof).
Revised: (see the Appendix B for a proof).

Original: Proposition 2.
Revised: Proposition 2.

Original: For p ≥ p 0 , g ( p, γ∗ ) ≤ 1 for all γ ≥ γ ∗ =
Revised: For p ≥ p 0 , g ( p, γ ) ≤ 1 for all γ ≥ γ ∗ =

Original: ab + 1 log a W − 1 (cid:0) − a (1 − a/b ) b log a (cid:1) , where a = 1 − p 0 , b = p 0 log p 0 and W − 1 is the Lambert-W function (Corless et al., 1996).
Revised: ab + 1 log a W − 1 (cid:0) − a (1 − a/b ) b log a (cid:1) , where a = 1 − p 0 , b = p 0 log p 0 and W − 1 is the Lambert-W function (Corless et al., 1996).

Original: Moreover, for p ≥ p 0 and g ( p, γ ∗) ≤ 1 , the equality g ( p, γ∗ ) = 1 holds only for p = p 0 and γ =
Revised: Moreover, for p ≥ p 0 and γ ≥ γ ∗ , the equality g ( p, γ ) = 1 holds only for p = p 0 and γ =

Original: γ ∗ .
Revised: γ ∗ .

Original: Thus, given a p 0 , we can always use Proposition 2 to compute γ ∗ such that(i) g ( p 0 , γ ∗ )
Revised: Thus, given a p 0 , we can always use Proposition 2 to compute γ ∗ such that

Original: = 1 , (ii) for p < p 0 , g ( p, γ ∗ ) > 1 , and (iii) for p > p 0 , g ( p, γ ∗ ) < 1 .

Original: Implicit Regularisation: For a network trained using focal loss with a ﬁxed γ  , and in the initial stages of training, when ˆ p i,y i < p 0 in general, g (ˆ p i,y i , γ ) > 1 and the model gains conﬁdence on its predictions faster than it would for cross-entropy.
Revised: Implicit Regularisation: For a network trained using focal loss with a ﬁxed γ obtained using psuch that g ( p 0 , γ ) = 1 then in the initial stages of training, when ˆ p i,y i < p 0 in general, g (ˆ p i,y i , γ ) >and the model gains conﬁdence on its predictions faster than it would for cross-entropy.

Original: However, as soon as ˆ p
Revised: However, as soon as ˆ p

Original: i,y i crosses the threshold p 0 , g (ˆ p i,y i , γ ) falls below 1 and reduces the size of the gradient updates made to the network weights, thereby having a regularising effect on the weights.
Revised: i,y i crosses the threshold p 0 , g (ˆ p i,y i , γ ) falls below 1 and reduces the size of the gradient updates made to the network weights, thereby having a regularising effect on the weights.

Original: This is why, in Figure 3(h), we ﬁnd that the weight norms of the models trained with focal loss are initially higher than that for the model trained using cross-entropy.
Revised: This is why, in Figure 3(h), we ﬁnd that the weight norms of the models trained with focal loss are initially higher than that for the model trained using cross-entropy.

Original: However, as training progresses, we ﬁnd that the ordering of the weight norms reverses, as focal loss starts regularising the network weights.
Revised: However, as training progresses, we ﬁnd that the ordering of the weight norms reverses, as focal loss starts regularising the network weights.

Original: Figure 4(b-d) give further insights by plotting histograms of the gradient norms of the last linear layer (over all samples in the training set) at epochs 10 , 100 and 200 , respectively.
Revised: Figure 4(b-d) give further insights by plotting histograms of the gradient norms of the last linear layer (over all samples in the training set) at epochs 10 , 100 and 200 , respectively.

Original: At epoch 10 , the gradient norms for cross-entropy and focal loss are similar, but as training progresses, those for cross-entropy decrease less rapidly than those for focal loss.
Revised: At epoch 10 , the gradient norms for cross-entropy and focal loss are similar, but as training progresses, those for cross-entropy decrease less rapidly than those for focal loss.

Original: Finally, note from Figure 4(a) that with higher values of γ , the fall in g (ˆ p i,y i , γ ) gets steeper.
Revised: Finally, note from Figure 4(a) that with higher values of γ , the fall in g (ˆ p i,y i , γ ) gets steeper.

Original: We would thus expect the weight regularisation effect to be higher for models that use higher values of γ .
Revised: We would thus expect the weight regularisation effect to be higher for models that use higher values of γ .

Original: This explains why, of the three models we trained using focal loss, the one with γ = 3 outperforms (in terms of calibration) the one with γ = 2 , which in turn outperforms the model with γ = 1 .
Revised: This explains why, of the three models we trained using focal loss, the one with γ = 3 outperforms (in terms of calibration) the one with γ = 2 , which in turn outperforms the model with γ = 1 .

Original: Based on this observation, one might think that, in general, a higher value of gamma would lead to a more calibrated model.
Revised: Based on this observation, one might think that, in general, a higher value of gamma would lead to a more calibrated model.

Original: However, this is not the case, as we notice from Figure 4(a) that for γ ≥ 7 , g
Revised: However, this is not the case, as we notice from Figure 4(a) that for γ ≥ 7 , g

Original: (ˆ p i,y i , γ ) reduces to nearly 0 for a relatively low value of ˆ p
Revised: (ˆ p i,y i , γ ) reduces to nearly 0 for a relatively low value of ˆ p

Original: i,y i (around 0 .
Revised: i,y i (around 0 .

Original: As a result, using values of γ that are too high will cause the gradients to die (i.e. reduce to nearly 0 ) early, at a point at which the network’s predictions remain ambiguous, thereby causing the training process to fail.
Revised: As a result, using values of γ that are too high will cause the gradients to die (i.e. reduce to nearly 0 ) early, at a point at which the network’s predictions remain ambiguous, thereby causing the training process to fail.

Original: To this end, we provide Proposition 2 that allows us to efﬁciently get a sampled-dependent gamma in a principled manner.
Revised: To this end, we provide Proposition 2 that allows us to efﬁciently get a sampled-dependent gamma in a principled manner.

Original: Experimentally we show that the sample-dependant γ does provide the best result among all the approaches we compare with.
Revised: Experimentally we show that the sample-dependant γ does provide the best result among all the approaches we compare with.

Revised: We also provide an empirical analysis of focal loss and cross entropy on a linear model in Appendix G.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We perform experiments on multiple image and document classiﬁcation datasets to verify the effectiveness of focal loss for training calibrated models.
Revised: We perform experiments on multiple image and document classiﬁcation datasets to verify the effectiveness of focal loss for training calibrated models.

Original: For image classiﬁcation experiments, we use
Revised: For image classiﬁcation experiments, we use

Original: CIFAR-10 (Krizhevsky, 2009) and CIFAR-100 (Krizhevsky, 2009) and for document classiﬁcation we use 20 Newsgroups and the Stanford Sentiment Treebank (SST)
Revised: CIFAR-10 (Krizhevsky, 2009) and CIFAR-100 (Krizhevsky, 2009) and for document classiﬁcation we use 20 Newsgroups and the Stanford Sentiment Treebank (SST)

Original: (Socher et al., 2013).
Revised: (Socher et al., 2013).

Original: We provide details on the datasets and the train/validation/test splits for each dataset in the Appendix
Revised: We provide details on the datasets and the train/validation/test splits for each dataset in the Appendix

Original: C. On CIFAR-10 and CIFAR-100, we train the networks: ResNet-50, ResNet-110 (He et al., 2016),
Revised: C. On CIFAR 10 and CIFAR 100, we train the networks

Revised: ResNet-

Revised: ResNet-

Revised: He et al.

Revised: Wide

Revised: ResNet

Original: Wide-ResNet-26-10 (Zagoruyko and Komodakis, 2016), and DenseNet-121 (Huang et al., 2017).We train the Global Pooling Convolutional Network (Lin et al., 2013) on 20 Newsgroups and the TreeLSTM (Tai et al., 2015) on the SST Binary dataset.
Revised: Zagoruyko and Komodakis and DenseNet- Huang et al., .We train the Global Pooling Convolutional Network (Lin et al., 2014) on 20 Newsgroups and the TreeLSTM (Tai et al., 2015) on the SST Binary dataset.

Original: We provide implementation details for each of the state of the art networks in Appendix D. For each dataset-network pair, we train the network using each of the following loss functions:
Revised: We provide implementation details for each of the state of the art networks in Appendix D. For each dataset-network pair, we train the network using each of the following loss functions:

Original: Baselines (Cross-Entropy, MMCE (Kumar et al., 2018)) and Brier Score (Brier, 1950) :
Revised: Baselines (Cross-Entropy, MMCE (Kumar et al., 2018)) and Brier Score (Brier, 1950) :

Original: Models trained on cross-entropy loss and MMCE loss (i.e. cross-entropy with an MMCE regulariser) serve as two of the baselines for our comparison. The third baseline consists of models trained using
Revised: Models trained on cross-entropy loss, MMCE loss (i.e. cross-entropy with an MMCE regulariser) and

Original: Brier Score as the loss function.
Revised: Brier score serve as our baselines for comparison.

Revised: MMCE (Maximum Mean Calibration Error) is a continuous and differentiable proxy for calibration error and hence, can be directly optimised using standard optimisers like Stochastic Gradient Descent.

Revised: Normally, MMCE is optimised as a regulariser alongside cross-entropy.

Original: Brier score, for a single sample, is computed as the squared error between the predicted softmax vector and the one-hot ground truth encoding.
Revised: Brier score, for a single sample, is computed as the squared error between the predicted softmax vector and the one-hot ground truth encoding.

Original: We ﬁnd Brier score to be a particularly relevant baseline for calibration as there is a distinct penalty for increasing the conﬁdence on incorrect classes.
Revised: We ﬁnd Brier score to be a particularly relevant baseline for calibration as there is a distinct penalty for increasing the conﬁdence on incorrect classes.

Original: In addition, it is relevant as it can be decomposed into calibration and reﬁnement (DeGroot and Fienberg, 1983).
Revised: In addition, it is relevant as it can be decomposed into calibration and reﬁnement (DeGroot and Fienberg, 1983).

Original: Focal Loss (Fixed γ ) :
Revised: Focal Loss (Fixed γ ) :

Original: We train various models each using three values of γ set to 1 , 2 and 3 .
Revised: We train various models each using three values of γ set to 1 , 2 and 3 .

Original: Focal Loss (Sample-Dependent γ ) :
Revised: Focal Loss (Sample-Dependent γ ) :

Original: From Proposition 2, we know that for a given probability p , we can ﬁnd a value of γ such that either g ( p, γ ) ∈ [0 , 1) , and focal loss regularises the weights of the network, or g ( p, γ ) > 1 , and focal loss speeds up the process of making the network more conﬁdent on the correct class.
Revised: From Proposition 2, we know that for a given probability p , we can ﬁnd a value of γ such that either g ( p, γ ) ∈ [0 , 1) , and focal loss regularises the weights of the network, or g ( p, γ ) > 1 , and focal loss speeds up the process of making the network more conﬁdent on the correct class.

Original: In this approach, we follow the idea of accelerating the rise in the value of ˆ p
Revised: In this approach, we follow the idea of accelerating the rise in the value of ˆ p

Original: i,y i (the conﬁdence on the correct class
Revised: i,y i (the conﬁdence on the correct class

Original: y i for the i th training sample) as long as ˆ p
Revised: y i for the i th training sample) as long as ˆ p

Original: i,y i < 0 .
Revised: i,y i < 0 .

Original: As soon as ˆ p
Revised: As soon as ˆ p

Original: i,y i ≥ 0 .
Revised: i,y i ≥ 0 .

Original: 5 , we can conﬁrm that the i th training sample has been predicted correctly.
Revised: 5 , we can conﬁrm that the i th training sample has been predicted correctly.

Original: Hence, when ˆ p i,y i ≥ 0 . 5 , we want to regularise the weights to prevent the network from becoming overconﬁdent and miscalibrated.
Revised: Hence, when ˆ p i,y i ≥ 0 . 5 , we want to regularise the weights to prevent the network from becoming overconﬁdent and miscalibrated.

Original: There can be multiple ways of choosing sample-dependent γ that satisfy the above condition.
Revised: There can be multiple ways of choosing sample-dependent γ that satisfy the above condition.

Original: However, with just few initial tries we found these two policies to produce quite competitive results on the validation sets: (a)  γ = 5 for ˆ p
Revised: However, with just few initial tries we found these two policies to produce quite competitive results on the validation sets: (a) Focal Loss (sample-wise γ 5,3,2): γ = 5 for ˆ p

Original: i,y i ∈ [0 , 0 .
Revised: i,y i ∈ [0 , 0 .

Original: 19] (as g (0 .
Revised: 19] (as g (0 .

Original: ≈ 1 ), γ =for ˆ p
Revised: ≈ 1 ), γ = 3 for ˆ p

Original: i,y i ∈ (0 . 19 , 0 . 5] and γ = 2 for ˆ p
Revised: i,y i ∈ (0 . 19 , 0 . 5] and γ = 2 for ˆ p

Original: i,y i ∈ (0 . 5 , 1] , and (b)  γ = 5 for ˆ p
Revised: i,y i ∈ (0 . 5 , 1] , and (b) Focal Loss (sample-wise γ 5,3): γ = 5 for ˆ p

Original: i,y i ∈ [0 , 0 .
Revised: i,y i ∈ [0 , 0 .

Original: 19] , and γ =for ˆ p
Revised: 19] , and γ = 3 for ˆ p

Original: i,y i ∈ (0 . 19 , 1] .
Revised: i,y i ∈ (0 . 19 , 1] .

Original: Focal Loss (Scheduled γ ) :
Revised: Focal Loss (Scheduled γ ) :

Original: As a simpliﬁcation to the above approach, we also investigated the use of a schedule for γ during training, as we expect the value of ˆ p
Revised: As a simpliﬁcation to the above approach, we also investigated the use of a schedule for γ during training, as we expect the value of ˆ p

Original: i,y i to increase in general for all samples over time.
Revised: i,y i to increase in general for all samples over time.

Original: In particular, we report results for two different schedules: (a)  γ = 5 for the ﬁrstepochs, γ = 3 for the next 150 epochs, and γ = 2 for the last 100 epochs, and (b)  γ = 5 for the ﬁrst 100 epochs, γ = 3 for the next  150 epochs, and γ = 1 for the last 100 epochs.
Revised: In particular, we report results for two different schedules: (a) Focal Loss (scheduled γ 5,3,2): γ = 5 for the ﬁrst 100 epochs, γ = 3 for the next 150 epochs, and γ = 2 for the lastepochs, and (b) Focal Loss (scheduled γ 5,3,1): γ = 5 for the ﬁrst 100 epochs, γ = 3 for the next [Table] [Table] 150 epochs, and γ = 1 for the last 100 epochs.

Original: We also tried various other schedules, but found these two to produce the best results on the validation sets.
Revised: We also tried various other schedules, but found these two to produce the best results on the validation sets.

Original: We ﬁnd the best policy for each of the above three focal loss variants by cross-validating on the validation set, and report its results in Table 4 and 8.
Revised: We ﬁnd the best policy for each of the above three focal loss variants by cross-validating on the validation set, and report its results in Table 4 and 7.

Original: We report results for all other policies in
Revised: We report results for all other policies in

Original: Appendix E.
Revised: Appendix E.

Original: [Table]

Original: [Table]

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Temperature Scaling: We report the ECE % (using 15 bins), both before and after temperature scaling, for focal loss (sample-dependant which performed the best) vs baselines in Table 1.
Revised: Temperature Scaling: We report the ECE % (using 15 bins), both before and after temperature scaling, for focal loss (sample-dependant which performed the best) vs baselines in Table 1.

Original: Further, we report the same metric for different variants of focal loss in Table 4.
Revised: Further, we report the same metric for different variants of focal loss in Table 4.

Original: Similarly we report AdaECE in Table 2 and Table 5.
Revised: Similarly we report AdaECE in Table 2 and Table 5.

Original: In fact our results show that in situations where ECE might imply that the model is well calibrated, AdaECE brings out the miscalibration present in the model.
Revised: In fact our results show that in situations where ECE might imply that the model is well calibrated, AdaECE brings out the miscalibration present in the model.

Original: For example, in the case of WideResNet on CIFAR10 for Cross Entropy, the best ECE obtained is 0 . 92 , implying almost no miscalibration [Table] whereas AdaECE is 1 .
Revised: For example, in the case of WideResNet on CIFAR10 for Cross Entropy, the best ECE obtained is 0 . 92 , implying almost no miscalibration, whereas AdaECE is 1 .

Original: 69 showing further scope for improvements.
Revised: 69 showing further scope for improvements.

Original: We compute the optimal temperature using two different approaches: (a) learning the optimal temperature by optimising NLL over a validation set, and (b) performing grid search over temperature values between 0 and 10, with a step of 0.1, and choosing the temperature that minimises the
Revised: We compute the optimal temperature using two different approaches: (a) learning the optimal temperature by optimising NLL over a validation set, and (b) performing grid search over temperature values between 0 and 10, with a step of 0.1, and choosing the temperature that minimises the

Original: ECE/AdaECE over a validation set.
Revised: ECE/AdaECE over a validation set.

Original: We ﬁnd the second approach to produce better results in general.
Revised: We ﬁnd the second approach to produce better results in general.

Original: Since we report ECE/AdaECE as the primary performance metric and grid search does not require a differentiable objective function, we directly minimise ECE/AdaECE over the validation set during grid search.
Revised: Since we report ECE/AdaECE as the primary performance metric and grid search does not require a differentiable objective function, we directly minimise ECE/AdaECE over the validation set during grid search.

Original: Furthermore, since NLL is not a convex function of temperature, an optimisation algorithm may get stuck at a locally optimal temperature.
Revised: Furthermore, since NLL is not a convex function of temperature, an optimisation algorithm may get stuck at a locally optimal temperature.

Original: We thus report the optimal temperatures and their corresponding ECEs/AdaECEs obtained using grid search (i.e. the second approach).
Revised: We thus report the optimal temperatures and their corresponding ECEs/AdaECEs obtained using grid search (i.e. the second approach).

Original: Performance Gains: It should be noted that for all the dataset-network pairs, we obtain state-ofthe-art classiﬁcation accuracies, as shown inthe Table 3 and Table 6.
Revised: Performance Gains: It should be noted that for all the dataset-network pairs, we obtain state-ofthe-art classiﬁcation accuracies, as shown in Table 3.

Original: For focal loss with a ﬁxed γ , we found that γ = 3 produced the best ECE results.
Revised: For focal loss with a ﬁxed γ , we found that γ = 3 produced the best ECE results.

Original: This corroborates the observation we made in §4 that γ = 3 should produce better results than γ = 1 or γ = 2 , as the regularising effect for γ = 3 is higher.
Revised: This corroborates the observation we made in §4 that γ =should produce better results than γ = 1 or γ = 2 , as the regularising effect for γ = 3 is higher.

Original: For the sample-dependent γ approach, we found the second policy (i.e.  γ = 5 for ˆ p
Revised: For the sample-dependent γ approach, we found the second policy (i.e. Focal Loss (sample-wise γ 5,3) with γ = 5 for ˆ p

Original: i,y i ∈ [0 , 0 .
Revised: i,y i ∈ [0 , 0 .

Original: 19] , and γ = 3 for ˆ p
Revised: 19] , and γ = 3 for ˆ p

Original: i,y i ∈ (0 . 19 , 1] ) to produce better results.
Revised: i,y i ∈ (0 . 19 , 1] ) to produce better results.

Revised: Of the two approaches for scheduled γ , we found the ﬁrst schedule (i.e. Focal Loss (scheduled γ 5,3,2) with

Revised: [Table]

Original: Of the two approaches for scheduled γ , we found the ﬁrst schedule (i.e. γ = 5 for the ﬁrst 100 epochs, γ = 3 for the nextepochs, and γ = 2 for the last 100 epochs) to produce better ECEs/AdaECEs for every model and dataset.
Revised: [Table] [Table] [Table] γ = 5 for the ﬁrst 100 epochs, γ = 3 for the next 150 epochs, and γ = 2 for the last 100 epochs) to produce better ECEs/AdaECEs for every model and dataset.

Original: We report the ECE values for the best performing approaches in Table 1 and 4.
Revised: We report the ECE values for the best performing approaches in Table 1 and 4.

Original: Similarly we show the AdaECE values of these models in 2 and 5.
Revised: Similarly we show the AdaECE values of these models inand 5.

Original: Full results (ECE, AdaECE, MCE, NLL and test error) for all approaches are reported in theAppendix E.
Revised: Full results (ECE, AdaECE, MCE, NLL and test error) for all approaches are reported in the

Original: [Table] [Table] [Table] [Table]
Revised: Appendix E.

Original: It is clear from Table 1 and 4 that focal loss with sample-dependent γ outperforms all the baselines (cross-entropy, brier score and MMCE).
Revised: It is clear from Table 1 and 4 that focal loss with sample-dependent γ outperforms all the baselines (cross-entropy, brier score and MMCE).

Original: It broadly produces the lowest ECE and AdaECE values both before and after temperature scaling.
Revised: It broadly produces the lowest ECE and AdaECE values both before and after temperature scaling.

Original: This observation is particularly encouraging, as it indicates that a principled method of obtaining values of γ for focal loss can work well, thereby alleviating the[Table] need to naively cross-validate the γ hyperparameter .
Revised: This observation is particularly encouraging, as it indicates that a principled method of obtaining values of γ for focal loss can work well, thereby alleviating the need to naively cross-validate the γ hyperparameter .

Original: Also, Table 4 and Table 5 show that other focal loss based approaches are also very competitive. 
Revised: Also, Table 4 and Table 5 show that other focal loss based approaches are also very competitive. [Table]

Original: Conﬁdent and Calibrated Models: It is interesting to note that for focal loss with sample-based γ (refer Tab. 1 and Tab. 2), beside most other focal loss models, the optimal temperatures are very close to 1, mostly lying between 0.9 and 1.1.
Revised: Conﬁdent and Calibrated Models: It is interesting to note that for focal loss with sample-based γ (refer Tab. 1 and Tab. 2), beside most other focal loss models, the optimal temperatures are very close to 1, mostly lying between 0.9 and 1.1.

Original: By contrast, the optimal temperatures for the baselines (cross-entropy and MMCE) are signiﬁcantly higher, with values lying between 2.0 to 2.8.
Revised: By contrast, the optimal temperatures for the baselines (cross-entropy and MMCE) are signiﬁcantly higher, with values lying between 2.0 to 2.8.

Original: An optimal temperature close to 1 indicates that the model is innately calibrated and cannot be made signiﬁcantly more calibrated by temperature scaling.
Revised: An optimal temperature close to 1 indicates that the model is innately calibrated and cannot be made signiﬁcantly more calibrated by temperature scaling.

Original: Furthermore, an optimal temperature that is much greater than 1 can make the network underconﬁdent in general, as its outputs are temperature-scaled irrespective of their correctness.
Revised: Furthermore, an optimal temperature that is much greater than 1 can make the network underconﬁdent in general, as its outputs are temperature-scaled irrespective of their correctness.

Original: To see this, we follow the approach adopted in
Revised: To see this, we follow the approach adopted in

Original: Kumar et al.
Revised: Kumar et al.

Original: (2018), and measure the percentage of test samples that are predicted with a conﬁdence of 0.99 or more (we call this set of test samples S 99 ).
Revised: (2018), and measure the percentage of test samples that are predicted with a conﬁdence of 0.99 or more (we call this set of test samples S 99 ).

Original: In Table 7, we report | S 99 | as a percentage of the total number of test samples, along with the accuracy of the samples in S 99 for ResNet-50 and
Revised: In Table 6, we report | S 99 | as a percentage of the total number of test samples, along with the accuracy of the samples in S 99 for ResNet-50 and

Original: ResNet-110 trained on CIFAR-10, using cross-entropy loss, MMCE loss, and focal loss.
Revised: ResNet-110 trained on CIFAR-10, using cross-entropy loss, MMCE loss, and focal loss.

Original: We observe that | S 99 | for the focal loss model is much lower than for the cross-entropy or MMCE models before temperature scaling.
Revised: We observe that | S 99 | for the focal loss model is much lower than for the cross-entropy or MMCE models before temperature scaling.

Original: However, after temperature scaling, | S 99 | for focal loss is signiﬁcantly higher than for both MMCE and cross-entropy.
Revised: However, after temperature scaling, | S 99 | for focal loss is signiﬁcantly higher than for both MMCE and cross-entropy.

Original: The reason is that with an optimal temperature of 1.1, the conﬁdence of the temperature-scaled model for focal loss does not reduce as much as those of the models for cross-entropy and MMCE, for which the optimal temperatures lie between 2.5 to 2.8.
Revised: The reason is that with an optimal temperature of 1.1, the conﬁdence of the temperature-scaled model for focal loss does not reduce as much as those of the models for cross-entropy and MMCE, for which the optimal temperatures lie between 2.5 to 2.8.

Original: We thus conclude that models trained on focal loss are not only more calibrated, but also better preserve their conﬁdence on predictions, even after being post-processed with temperature scaling.
Revised: We thus conclude that models trained on focal loss are not only more calibrated, but also better preserve their conﬁdence on predictions, even after being post-processed with temperature scaling.

Revised: We present some qualitative results to support this claim in Appendix F.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have shown that training using focal loss can yield multi-class classiﬁcation networks that are more naturally calibrated than those trained using more conventional cross-entropy loss.
Revised: In this paper, we have shown that training using focal loss can yield multi-class classiﬁcation networks that are more naturally calibrated than those trained using more conventional cross-entropy loss.

Original: There are sound theoretical reasons to expect this: in particular, as we show in §4, focal loss implicitly regularises the weights of a network during training, reducing NLL overﬁtting and thereby improving calibration.
Revised: There are sound theoretical reasons to expect this: in particular, as we show in §4, focal loss implicitly regularises the weights of a network during training, reducing NLL overﬁtting and thereby improving calibration.

Original: Extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (
Revised: Extensive experiments on a variety of computer vision (CIFAR-10/100) and NLP (

Original: Newsgroups/SST) datasets, and with a wide variety of different network architectures, show that this expectation is also borne out in practice.
Revised: Newsgroups/SST) datasets, and with a wide variety of different network architectures, show that this expectation is also borne out in practice.

Original: Our results show that in almost all cases, networks trained with focal loss are more calibrated than those trained with cross-entropy loss, whilst having similar levels of accuracy, making their predictions much easier for downstream components to trust.
Revised: Our results show that in almost all cases, networks trained with focal loss are more calibrated than those trained with cross-entropy loss, whilst having similar levels of accuracy, making their predictions much easier for downstream components to trust.

Original: We also introduce principled way of getting sample-dependent γ for focal loss and we show that it produces good results across datasets and models.
Revised: We also introduce principled way of getting sample-dependent γ for focal loss and we show that it produces good results across datasets and models.

Original: A A PPENDIX
Revised: A A PPENDIX

Original: In § B, we show the proofs of the propositions formulated in the main text.
Revised: In § B, we show the proofs of the propositions formulated in the main text.

Original: We then describe all the datasets we used for our experiments in § C which is followed by implementation details of the state of the art networks in § D.
Revised: We then describe all the datasets we used for our experiments in § C which is followed by implementation details of the state of the art networks in § D.

Original: B P ROOFS
Revised: B P ROOFS

Original: Here we provide the proofs of both the propositions presented in the main text.
Revised: Here we provide the proofs of both the propositions presented in the main text.

Original: While Propositionhelps us understand the regularization effect of using the focal loss, the Proposition 4 efﬁciently provides us the γ values in a principle way such that it is sample-dependent.
Revised: While Propositionhelps us understand the regularization effect of using the focal loss, the Proposition 4 efﬁciently provides us the γ values in a principle way such that it is sample-dependent.

Original: Implementing the sample-dependent γ is very easy as implementation of the Lambert-W function (Corless et al., 1996) is available in standard libraries (e.g. python scipy).
Revised: Implementing the sample-dependent γ is very easy as implementation of the Lambert-W function (Corless et al., 1996) is available in standard libraries (e.g. python scipy).

Original: Using it we could get quite competitive results with just few initial sample-dependent γ policies that we tried.
Revised: Using it we could get quite competitive results with just few initial sample-dependent γ policies that we tried.

Original: Proposition 3.
Revised: Proposition 3.

Original: For focal loss L f and cross-entropy loss L c , the gradients ∂ L f ∂ w = ∂
Revised: For focal loss L f and cross-entropy loss L c , the gradients ∂ L f ∂ w = ∂

Original: L c ∂ w g (ˆ p i,y i , γ ) , where g (ˆ p
Revised: L c ∂ w g (ˆ p i,y i , γ ) , where g (ˆ p

Original: i,y i , γ ) =
Revised: i,y i , γ ) =

Original: (1 − ˆ p
Revised: (1 − ˆ p

Original: i,y i ) γ − γ ˆ p
Revised: i,y i ) γ − γ ˆ p

Original: i,y i (1 − ˆ p
Revised: i,y i (1 − ˆ p

Original: i,y i ) γ − 1 log(ˆ p i,y i ) , γ ≥ 0 is the focal loss hyperparameter, and w are the last layer network parameters.
Revised: i,y i ) γ − 1 log(ˆ p i,y i ) , γ ≥ 0 is the focal loss hyperparameter, and w are the last layer network parameters.

Original: Thus, (cid:13)(cid:13)(cid:13) ∂
Revised: Thus, (cid:13) (cid:13) (cid:13) ∂

Original: L f ∂ w (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)
Revised: L f ∂ w (cid:13) (cid:13) (cid:13) ≤ (cid:13)(cid:13)

Original: ∂ L c ∂ w (cid:13)(cid:13)
Revised: ∂ L c ∂ w (cid:13)(cid:13)

Original: iff g
Revised: iff g

Original: (ˆ p i,y i , γ ) ∈ [0 , 1] .
Revised: (ˆ p i,y i , γ ) ∈ [0 , 1] .

Original: Proof.
Revised: Proof.

Original: Let w be the linear layer parameters connecting the feature map to the logit s .
Revised: Let w be the linear layer parameters connecting the feature map to the logit s .

Original: Then, using the chain rule, ∂ L f ∂ w = (cid:16) ∂s∂ w (cid:17)(cid:16) ∂
Revised: Then, using the chain rule, ∂ L f ∂ w = (cid:16) ∂s∂ w (cid:17)(cid:16) ∂

Original: ˆ p
Revised: ˆ p

Original: i,yi ∂s (cid:17)(cid:16) ∂
Revised: i,yi ∂s (cid:17)(cid:16) ∂

Original: L f ∂ ˆ p
Revised: L f ∂ ˆ p

Original: i,yi (cid:17) .
Revised: i,yi (cid:17) .

Original: Similarly, ∂ L c ∂ w = (cid:16) ∂s∂ w (cid:17)(cid:16) ∂
Revised: Similarly, ∂ L c ∂ w = (cid:16) ∂s∂ w [Equation]

Original:  ˆ p
Revised: ∂ ˆ p

Original: i,yi ∂s (cid:17)(cid:16) ∂
Revised: i [Equation] yi ∂s [Equation]

Original:  L c ∂ ˆ p
Revised: ∂ L c ∂ ˆ p

Original: i,yi (cid:17) .
Revised: i,yi (cid:17) .

Original: The derivative of the focal loss with respect to ˆ p
Revised: The derivative of the focal loss with respect to ˆ p

Original: i,y i , the softmax output of the network for the true class
Revised: i,y i , the softmax output of the network for the true class

Original: y i , takes the form
Revised: y i , takes the form

Original: [Equation]
Revised: [Equation]

Original: in which g (ˆ p i,y i , γ ) = (1 − ˆ p
Revised: in which g (ˆ p i,y i , γ ) = (1 − ˆ p

Original: i,y i ) γ − γ ˆ p
Revised: i,y i ) γ − γ ˆ p

Original: i,y i (1 − ˆ p
Revised: i,y i (1 − ˆ p

Original: i,y i ) γ − 1 log(ˆ p i,y i ) and
Revised: i,y i ) γ − 1 log(ˆ p i,y i ) and

Original: ∂ L c ∂ ˆ p
Revised: ∂ L c ∂ ˆ p

Original: i,y i = − 1 ˆ p
Revised: i,yi = − 1 ˆ p

Original: i,y i .
Revised: i,yi .

Original: It is thus straightforward to verify that if g (ˆ p
Revised: It is thus straightforward to verify that if g (ˆ p

Original: i,y i , γ ) ∈ [0 , 1] , then (cid:13)(cid:13)(cid:13) ∂
Revised: i,y i , γ ) ∈ [0 , 1] , then (cid:13)(cid:13)(cid:13) ∂

Original: L f ∂ ˆ p
Revised: L f ∂ ˆ p

Original: i,yi (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)
Revised: i,yi (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13)

Original: ∂
Revised: ∂

Original: L c ∂ ˆ p
Revised: L c ∂ ˆ p

Original: i,yi (cid:13)(cid:13)(cid:13) , which itself implies that (cid:13)(cid:13)(cid:13) ∂
Revised: i,yi (cid:13)(cid:13)(cid:13) , which itself implies that (cid:13)(cid:13)(cid:13) ∂

Original: L f ∂ w (cid:13)(cid:13)(cid:13) ≤ [Equation]
Revised: L f ∂ w (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13) ∂ L c ∂ w (cid:13)(cid:13) .

Original: Proposition 4.
Revised: Proposition 4.

Original: For p ≥ p 0 , g ( p, γ∗ ) ≤ 1 for all γ ≥ γ ∗ =
Revised: For p ≥ p 0 , g ( p, γ ) ≤ 1 for all γ ≥ γ ∗ =

Original: ab + 1 log a W − 1 (cid:0) − a (1 − a/b ) b log a (cid:1) , where a = 1 − p 0 , b = p 0 log p 0 and W − 1 is the Lambert-W function (Corless et al., 1996).
Revised: ab + 1 log a W − 1 (cid:0) − a (1 − a/b ) b log a (cid:1) , where a = 1 − p 0 , b = p 0 log p 0 and W − 1 is the Lambert-W function (Corless et al., 1996).

Original: Moreover, for p ≥ p 0 and g ( p, γ ∗) ≤ 1 , the equality g ( p, γ∗ ) = 1 holds only for p = p 0 and γ =
Revised: Moreover, for p ≥ p 0 and γ ≥ γ ∗ , the equality g ( p, γ ) = 1 holds only for p = p 0 and γ =

Original: γ ∗ .
Revised: γ ∗ .

Original: Proof.
Revised: Proof.

Original: We derive the value of γ > 0 for which g ( p 0 , γ ) = 1 for a given p 0 ∈ [0 , 1] .
Revised: We derive the value of γ > 0 for which g ( p 0 , γ ) = 1 for a given p 0 ∈ [0 , 1] .

Original: From Proposition 4.1, we already know that
Revised: From Proposition 4.1, we already know that

Original: [Equation]
Revised: [Equation]

Original: where L f is focal loss, L c is cross entropy loss, ˆ p
Revised: where L f is focal loss, L c is cross entropy loss, ˆ p

Original: i,y i is the probability assigned by the model to the ground-truth correct class for the i th sample, and
Revised: i,y i is the probability assigned by the model to the ground-truth correct class for the i th sample, and

Original: [Equation]
Revised: [Equation]

Original: For p ∈ [0 , 1] , if we look at the function g ( p, γ ) , then we can clearly see that g ( p, γ ) → 1 as p → 0 , and that g ( p, γ ) = 0 when p = 1 .
Revised: For p ∈ [0 , 1] , if we look at the function g ( p, γ ) , then we can clearly see that g ( p, γ ) → 1 as p → 0 , and that g ( p, γ ) = 0 when p = 1 .

Original: To observe the behaviour of g ( p, γ ) for intermediate values of p , we ﬁrst take its derivative with respect to p :
Revised: To observe the behaviour of g ( p, γ ) for intermediate values of p , we ﬁrst take its derivative with respect to p :

Original: [Equation]
Revised: [Equation]

Original: In Equation 7, γ (1 − p ) γ − 2 > 0 except when p = 1 (in which case γ (1 − p ) γ − 2 = 0 ).
Revised: In Equation 7, γ (1 − p ) γ − 2 > 0 except when p = 1 (in which case γ (1 − p ) γ − 2 = 0 ).

Original: Thus, to observe the sign of the gradient ∂g ( p,γ ) ∂p , we focus on the term
Revised: Thus, to observe the sign of the gradient ∂g ( p,γ ) ∂p , we focus on the term

Original: [Equation]
Revised: [Equation]

Original: Dividing Equation 8 by ( − log p ) , the sign remains unchanged and we get
Revised: Dividing Equation 8 by ( − log p ) , the sign remains unchanged and we get

Original: [Equation]
Revised: [Equation]

Original: We can see that k ( p, γ ) → 1 as p → 0 and k ( p, γ ) → − (1 + γ ) as p → 1 (using l’Hôpital’s rule).
Revised: We can see that k ( p, γ ) → 1 as p → 0 and k ( p, γ ) → − (1 + γ ) as p → 1 (using l’Hôpital’s rule).

Original: Furthermore, k ( p, γ ) is monotonically decreasing for p ∈ [0 , 1] .
Revised: Furthermore, k ( p, γ ) is monotonically decreasing for p ∈ [0 , 1] .

Original: Thus, as the gradient ∂g ( p,γ ) ∂p monotonically decreases from a positive value at p = 0 to a negative value at p = 1 , we can say that g ( p, γ ) ﬁrst monotonically increases starting from 1 (as p → 0 ) and then monotonically decreases down to 0 (at p = 1 ).
Revised: Thus, as the gradient ∂g ( p,γ ) ∂p monotonically decreases from a positive value at p = 0 to a negative value at p = 1 , we can say that g ( p, γ ) ﬁrst monotonically increases starting from 1 (as p → 0 ) and then monotonically decreases down to 0 (at p = 1 ).

Original: Thus, if for some threshold p 0 > 0 and for some γ > 0 , g ( p, γ ) = 1 , then ∀ p > p 0 , g ( p, γ ) < 1 .
Revised: Thus, if for some threshold p 0 > 0 and for some γ > 0 , g ( p, γ ) = 1 , then ∀ p > p 0 , g ( p, γ ) < 1 .

Original: We now want to ﬁnd a γ such that ∀ p ≥ p 0 , g ( p, γ ) ≤ 1 .
Revised: We now want to ﬁnd a γ such that ∀ p ≥ p 0 , g ( p, γ ) ≤ 1 .

Original: First, let a = (1 − p 0 ) and b = p 0 log p 0 .
Revised: First, let a = (1 − p 0 ) and b = p 0 log p 0 .

Original: Then:
Revised: Then:

Original: [Equation]

Original: a

Original: [Equation]
Revised: [Equation]

Original: where a = (1 − p 0 ) and b = p 0 log p 0 .
Revised: where a = (1 − p 0 ) and b = p 0 log p 0 .

Original: We know that the inverse of y =
Revised: We know that the inverse of y =

Original: xe x is deﬁned as x = W ( y ) , where W is the Lambert-W function (Corless et al., 1996).
Revised: xe x is deﬁned as x = W ( y ) , where W is the Lambert-W function (Corless et al., 1996).

Original: Furthermore, the r.h.s.
Revised: Furthermore, the r.h.s.

Original: of the inequality in Equation 10 is always negative, with a minimum possible value of − 1 /e that occurs at p 0 = 0 .
Revised: of the inequality in Equation 10 is always negative, with a minimum possible value of − 1 /e that occurs at p 0 = 0 .

Original: Therefore, applying the Lambert-W function to the r.h.s.
Revised: Therefore, applying the Lambert-W function to the r.h.s.

Original: will yield two real solutions (corresponding to a principal branch denoted by W 0 and a negative branch denoted by W − 1 ).
Revised: will yield two real solutions (corresponding to a principal branch denoted by W 0 and a negative branch denoted by W − 1 ).

Original: We ﬁrst consider the solution corresponding to the negative branch (which is the smaller of the two solutions):
Revised: We ﬁrst consider the solution corresponding to the negative branch (which is the smaller of the two solutions):

Original: [Equation]
Revised: [Equation]

Original: If we consider the principal branch, the solution is
Revised: If we consider the principal branch, the solution is

Original: [Equation]
Revised: [Equation]

Original: which yields a negative value for γ that we discard
Revised: which yields a negative value for γ that we discard

Original: Thus Equation 11 gives the values of γ for which if p > p 0 , then g ( p, γ ) < 1 .
Revised: Thus Equation 11 gives the values of γ for which if p > p 0 , then g ( p, γ ) < 1 .

Original: In other words, g ( p 0 , γ ) = 1 , and for any p < p 0 , g ( p, γ ) > 1 .
Revised: In other words, g ( p 0 , γ ) = 1 , and for any p < p 0 , g ( p, γ ) > 1 .

Original: C D ATASET D ESCRIPTION
Revised: C D ATASET D ESCRIPTION

Original: We use the following image- and document-classiﬁcation datasets in our experiments:
Revised: We use the following image- and document-classiﬁcation datasets in our experiments:

Original: CIFAR-10 (Krizhevsky, 2009):
Revised: CIFAR-10 (Krizhevsky, 2009):

Original: This dataset has 60,000 colour images of size 32 × 32 , divided equally into 10 classes.
Revised: This dataset has 60,000 colour images of size 32 × 32 , divided equally into 10 classes.

Original: We use a train/validation/test split of 45,000/5,000/10,000 images.
Revised: We use a train/validation/test split of 45,000/5,000/10,000 images.

Original: CIFAR-100 (Krizhevsky, 2009):
Revised: CIFAR-100 (Krizhevsky, 2009):

Original: This dataset has 60,000 colour images of size 32 × 32 , divided equally into 100 classes.
Revised: This dataset has 60,000 colour images of size 32 × 32 , divided equally into 100 classes.

Original: (Note that the images in this dataset are not the same images as in CIFAR-10.)
Revised: (Note that the images in this dataset are not the same images as in CIFAR-10.)

Original: We again use a train/validation/test split of 45,000/5,000/10,000 images.
Revised: We again use a train/validation/test split of 45,000/5,000/10,000 images.

Original: 20 Newsgroups : This dataset contains 20,000 news articles, categorised evenly into 20 different newsgroups based on their content.
Revised: 20 Newsgroups : This dataset contains 20,000 news articles, categorised evenly into 20 different newsgroups based on their content.

Original: It is a popular dataset for text classiﬁcation.
Revised: It is a popular dataset for text classiﬁcation.

Original: Whilst some of the newsgroups are very related (e g. rec.motorcycles and rec.autos), others are quite unrelated (e g. sci.space and misc.forsale).
Revised: Whilst some of the newsgroups are very related (e.g. rec.motorcycles and rec.autos), others are quite unrelated (e.g. sci.space and misc.forsale).

Original: We use a train/validation/test split of 15,098/900/3,999 documents.
Revised: We use a train/validation/test split of 15,098/900/3,999 documents.

Original: Stanford Sentiment Treebank (SST)
Revised: Stanford Sentiment Treebank (SST

Original: (Socher et al., 2013):
Revised: (Socher et al., ): This

Original: This dataset contains movie reviews in the form of sentence parse trees, where each node is annotated by sentiment.
Revised: dataset contains movie reviews in the form of sentence parse trees, where each node is annotated by sentiment.

Original: We use the dataset version with binary labels, for which 6,920/872/1,821 documents are used as the training/validation/test split.
Revised: We use the dataset version with binary labels, for which 6,920/872/1,821 documents are used as the training/validation/test split.

Original: In the training set, each node of a parse tree is annotated as positive, neutral or negative.
Revised: In the training set, each node of a parse tree is annotated as positive, neutral or negative.

Original: At test time, the evaluation is done based on the model classiﬁcation at the root node, i.e. considering the whole sentence, which contains only positive or negative sentiment.
Revised: At test time, the evaluation is done based on the model classiﬁcation at the root node, i.e. considering the whole sentence, which contains only positive or negative sentiment.

Original: D I MPLEMENTATION D ETAILS
Revised: D I MPLEMENTATION D ETAILS

Original: For training networks on the image classiﬁcation datasets (CIFAR-10 and CIFAR-100), we use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the ﬁrst 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs.
Revised: For training networks on the image classiﬁcation datasets (CIFAR-10 and CIFAR-100), we use SGD with a momentum of 0.9 as our optimiser, and train the networks for 350 epochs, with a learning rate of 0.1 for the ﬁrst 150 epochs, 0.01 for the next 100 epochs, and 0.001 for the last 100 epochs.

Original: We use a training batch size of 128.
Revised: We use a training batch size of 128.

Revised: Furthermore, we augment the training images by applying random crops and random horizontal ﬂips.

Original: For 20 Newsgroups, we train the Global Pooling Convolutional Network (Lin et al., 2013) using the
Revised: For 20 Newsgroups, we train the Global Pooling Convolutional Network (Lin et al., 2014) using the

Original: Adam optimiser, with learning rate 0 .
Revised: Adam optimiser, with learning rate 0 .

Original: 001 , and betas 0 .
Revised: 001 , and betas 0 .

Original: 9 and 0 . 999 .
Revised: 9 and 0 . 999 .

Original: The code is a PyTorch adaptation of (Ng).
Revised: The code is a PyTorch adaptation of (Ng).

Original: We used Glove word embeddings (Pennington et al., 2014) to train the network.
Revised: We used Glove word embeddings (Pennington et al., 2014) to train the network.

Original: We trained all the models for 50 epochs and used the models with the best validation accuracy.
Revised: We trained all the models for 50 epochs and used the models with the best validation accuracy.

Original: For the SST Binary dataset, we train the Tree-LSTM (Tai et al., 2015) using the AdaGrad optimiser with a learning rate of 0 .
Revised: For the SST Binary dataset, we train the Tree-LSTM (Tai et al., 2015) using the AdaGrad optimiser with a learning rate of 0 .

Original: 05 and a weight decay of 10 − 4 , as suggested by the authors.
Revised: 05 and a weight decay of 10 − 4 , as suggested by the authors.

Original: We used the constituency model, which considers binary parse trees of the data and trains a binary tree LSTM on them.
Revised: We used the constituency model, which considers binary parse trees of the data and trains a binary tree LSTM on them.

Original: The Glove word embeddings (Pennington et al., 2014) we used were also tuned for best results.
Revised: The Glove word embeddings (Pennington et al., 2014) we used were also tuned for best results.

Original: The code framework we used is inspired by (TreeLSTM).
Revised: The code framework we used is inspired by (TreeLSTM).

Original: We trained all of our models for 25 epochs and used the models with the best validation accuracy.
Revised: We trained all of our models for 25 epochs and used the models with the best validation accuracy.

Original: For all our models, we use the PyTorch framework, setting any hyperparameters not explicitly mentioned to the default values used in the standard models.
Revised: For all our models, we use the PyTorch framework, setting any hyperparameters not explicitly mentioned to the default values used in the standard models.

Original: For MMCE, we used λ = 2 for all the image-classiﬁcation tasks, whilst we found λ = 8 to perform better for document classiﬁcation.
Revised: For MMCE, we used λ = 2 for all the image-classiﬁcation tasks, whilst we found λ = 8 to perform better for document classiﬁcation.

Revised: A calibrated model which does not generalise well to an unseen test set is not very useful.

Revised: Hence, for all the experiments, we set the training parameters in a way such that we get state-of-the-art test set accuracies on all datasets for each model.

Original: E A DDITIONAL R ESULTS
Revised: E A DDITIONAL R ESULTS

Original: We use various metrics to compare the proposed methods based on focal loss with the baselines (i.e. cross-entropy, Brier score and MMCE).
Revised: We use various metrics to compare the proposed methods based on focal loss with the baselines (i.e. cross-entropy, Brier score and MMCE).

Original: We show all test ECE of all the focal loss models before and after temperature scaling in Table 8.
Revised: We show all test ECE of all the focal loss models before and after temperature scaling in Table 7.

Original: Similarly, we show all test AdaECE of all the focal loss models before and after temperature scaling in Table 9.
Revised: Similarly, we show all test AdaECE of all the focal loss models before and after temperature scaling in Table 8.

Original: We present the test set error for all the focal loss models in Table 10.
Revised: We present the test set error for all the focal loss models in Table 9.

Original: We present the test
Revised: We present the test

Original: NLL % before and after temperature scaling in Tables 11 and 12, respectively.
Revised: NLL % before and after temperature scaling in Tables 10 and 11, respectively.

Original: We report the test set
Revised: We report the test set

Original: MCE % before and after temperature scaling in Tables 13 and 14, respectively.
Revised: MCE % before and after temperature scaling in Tables 12 and 13, respectively.

Original: In this section we use the following abbreviation to report results on different varieties of Focal Loss.
Revised: In this section we use the following abbreviation to report results on different varieties of Focal Loss.

Original: FL-1 refers to Focal Loss (ﬁxed γ 1), FL-2 refers to Focal Loss (ﬁxed γ 2), FL-3 refers to Focal
Revised: FL-1 refers to Focal Loss (ﬁxed γ 1), FL-2 refers to Focal Loss (ﬁxed γ 2), FL-3 refers to Focal

Original: Loss (ﬁxed γ 3), FLS-531 refers to Focal Loss (scheduled γ 5,3,1), FLS-532 refers to Focal Loss (scheduled γ 5,3,2), FLA-532 refers to Focal Loss (sample-wise γ 5,3,2) and FLA-53 refers to Focal
Revised: Loss (ﬁxed γ 3), FLS-531 refers to Focal Loss (scheduled γ 5,3,1), FLS-532 refers to Focal Loss (scheduled γ 5,3,2), FLA-532 refers to Focal Loss (sample-wise γ 5,3,2) and FLA-53 refers to Focal

Original: Loss (sample- wise γ 5,
Revised: Loss (sample-wise γ 5,3)

Original: [Figure] [Figure]
Revised: In Figure 8, we present some qualitative results to show the improvement in the conﬁdence estimates of focal loss in comparison to other baselines (i.e., cross entropy, MMCE and Brier score).

Original: FL
Revised: For this, we take ResNet-50 networks trained on CIFAR-10 using all the four loss functions (cross entropy, MMCE, Brier score and Focal loss with sample-wise γ 5,3) and measure the conﬁdence of their predictions for four correctly and four incorrectly classiﬁed test samples. We report these conﬁdences both before and after temperature scaling. It is clear from Figure 8 that for all the

Original: [Figure]
Revised: [Figure]

Original: FL

Original: [Figure]
Revised: [Figure]

Original: FLSFLS- FLA-532)
Revised: [Table]

Original: FLS
Revised: [Table] [Table] [Table]

Original: [Figure]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Wide ResNet
Revised: SST

Original: [Table]
Revised: [Table]

Original: DenseNet23. CIFAR
Revised: Tree LSTM

Original: [Table]
Revised: [Table]

Original: ResNet4. ResNet4. Wide ResNet 264. DenseNet5.
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Newsgroups Global Pooling CNN 26. SST Binary Tree LSTM 12.
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Revised: correctly classiﬁed samples, the model trained using focal loss has very conﬁdent predictions both pre and post temperature scaling.

Revised: However, on misclassiﬁed samples, we observe a very low conﬁdence for the focal loss model.

Revised: The ResNet-50 network trained using cross entropy is very conﬁdent even

Original: [Table]
Revised: [Table]

Original: for different approaches post temperature scaling (cross validated on
Revised: for different approaches post temperature scaling (cross validated on

Original: ECE).
Revised: ECE).

Original: [Table]
Revised: [Table]

Original: pre temperature scaling
Revised: pre temperature scaling.

Original: [Table]
Revised: [Table]

Revised: on the misclassiﬁed samples, particularly before temperature scaling.

Revised: Apart from focal loss, the only model which has relatively low conﬁdences on misclassiﬁed test samples is the one trained using

Revised: Brier score.

Revised: These observations support our claim that focal loss produces not only a calibrated model but also one which is conﬁdent on its correct predictions.

Revised: G F OCAL L OSS AND C ROSS E NTROPY ON A L INEAR M ODEL

Revised: To understand the effect of focal loss on simpler to analyse models, we conducted experiments on a generalised linear model on a simple data distribution.

Revised: Setup We consider a binary classiﬁcation problem.

Revised: The data matrix X ∈ R 2 × N is created by assigning each class two normally distributed clusters such that the mean of the clusters are linearly separable.

Revised: The mean of the clusters are situated on the vertices of a two-dimensional hypercube.

Revised: Further to that, for 10% of the data points, the labels were ﬂipped.

Revised: The model consists of a simple 2-parameter logistic regression model.

Revised: f w ( x ) = σ ( w 1 x 1 + w 2 x 2 ) .

Revised: We train this model using both cross entropy and focal loss with γ = 1 .

Revised: [Figure]

Revised: N o r m o f

Revised: Log i t s

Revised: [Figure]

Revised: Weight Magniﬁcation We have argued that focal loss implicitly regularizes the weights of the model by providing smaller gradients as compared to cross entropy.

Revised: This helps in calibration as - if all the weights are large, the logits are large and thus the conﬁdence of the network is large on all test points.

Revised: When the model misclassiﬁes, it misclassiﬁes with a high conﬁdence.

Revised: Figure 5 shows, for a generalised linear model

Revised: [Figure]

Revised: that the norm of the logits and the weights of a network blows for Cross

Revised: Entropy as compared to Focal Loss.

Revised: High Conﬁdence for mistakes Figure 6 (b) and (c) shows that running gradient descent with cross entropy (CE) and focal loss (FL) both gives the same decision regions

Revised: i

Revised: [Figure]

Revised: e. the weight vector points in the same region for both FL and CE.

Revised: However, as we have seen that the norm of the weights is much larger for FL as compared to CE, we would expect the conﬁdence of misclassiﬁed test points to be large for CE as compared to FL.

Revised: Figure 6 (a) plots a histogram of the conﬁdence of the misclassiﬁed points and it shows that CE misclassiﬁes almost always with greater than 90% conﬁdence

Revised: whereas

Revised: FL misclassiﬁes with much lower conﬁdence

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: H F OCAL L OSS M INIMISES R EGULARISED B REGMAN D IVERGENCE

Revised: In this section, we show that just like Cross Entropy minimizes an upper bound on the KL-Divergence between the true distribution p and the predicted distribution q , focal loss minimizes a regularized KL divergence between p and q .

Revised: We use FL γ ( p, q ) and CE ( p, q ) to denote the focal loss with parameter

Revised: [Equation]

Revised: By Bernoulli’s inequality ∀ γ ≥ 1 and 0 ≤ q i ≤

Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: | q

Revised: i log q i | By Holder’s inequality with p = ∞ , q =

Revised: [Equation]

Revised: We know that

Revised: [Equation]

Revised: Experimentally, we found the solution of the Cross Entropy and Focal loss equation, i.e. the value of the predicted probability ˆ q which minimizes the loss, for various values of p in a binary classiﬁcation problem (i.e. K = 2 ) and plotted it in Figure 7.

Revised: Note how Focal loss favours a more entropic solution ˆ q that is closer to 0 .

Revised: In other words, as Figure 7 shows, solutions to focal loss (Eqn equation 13) will always have higher entropy than that of Cross Entropy

Revised: [Figure]

Revised: [Equation]

Revised: [Figure]