Original: Can Calibration Improve Sample Prioritization?
Revised: Can Calibration Improve Sample Prioritization?

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: Calibration can reduce overconfident predictions of deep neural networks, but cancalibration also accelerate training by selecting the right samples ?
Revised: Calibration can reduce overconfident predictions of deep neural networks, but can calibration also accelerate training?

Original: In this paper, weshow that it can.
Revised: In this paper, we show that it can when used to prioritize some examples for performing subset selection.

Original: We study the effect of popular calibration techniques in selectingbetter subsets of samples during training (also called sample prioritization) andobserve that calibration can improve the quality of subsets, reduce the number ofexamples per epoch (by at least 70%), and can thereby speed up the overall trainingprocess.
Revised: We study the effect of popular calibration techniques in selecting better subsets of samples during training (also called sample prioritization) and observe that calibration can improve the quality of subsets, reduce the number of examples per epoch (by at least 70%), and can thereby speed up the overall training process.

Original: We further study the effect of using calibrated pre-trained models coupledwith calibration during training to guide sample prioritization, which again seemsto improve the quality of samples selected.
Revised: We further study the effect of using calibrated pre-trained models coupled with calibration during training to guide sample prioritization, which again seems to improve the quality of samples selected.

Original:  Introduction
Revised:  Introduction

Original: Calibration is a widely used technique in machine learning to reduce overconfidence in predictions.
Revised: Calibration is a widely used technique in machine learning to reduce overconfidence in predictions.

Original: Modern deep neural networks are known to be overconfident classifiers or predictors, and calibratednetworks provide trustworthy and reliable confidence estimates [1].
Revised: Modern deep neural networks are known to be overconfident classifiers or predictors, and calibrated networks provide trustworthy and reliable confidence estimates [1].

Original: Hence, finding new calibrationtechniques and improving them has been an active area of research [1, 6, 10, 11].
Revised: Hence, finding new calibration techniques and improving them has been an active area of research [1, 6, 10, 11].

Original: In this paper, we ask if calibration aids in accelerating training by using sample prioritization , i.e.,we select training samples based on calibrated predictions to better steer the training performance.
Revised: In this paper, we ask if calibration aids in accelerating training by using sample prioritization , i.e., we select training samples based on calibrated predictions to better steer the training performance.

Original: We explore different calibration techniques and focus on selecting a subset with the most informativesamples during each epoch.
Revised: We explore different calibration techniques and focus on selecting a subset with the most informative samples during each epoch.

Original: We observe that calibration performed during training plays a crucialrole in choosing the most informative subsets , which in turn accelerates neural network training.
Revised: We observe that calibration performed during training plays a crucial role in choosing the most informative subsets , which in turn accelerates neural network training.

Original: We then investigate the effect of an external pre-trained model which is well-calibrated (with largercapacity), on the sample selection process during training.
Revised: We then investigate the effect of an external pre-trained model which is well-calibrated (with larger capacity) on the sample selection process during training.

Original: Our contributions are as follows,
Revised: Our contributions are as follows,

Original: We provide an in-depth study analyzing the effect of various calibration techniques on sampleprioritization during training.
Revised: We provide an in-depth study analyzing the effect of various calibration techniques on sample prioritization during training.

Original: We also consider pre-trained calibrated target models and observe theireffect on sample prioritization along with calibration during training.
Revised: We also consider pre-trained calibrated target models and observe their effect on sample prioritization along with calibration during training.

Original: We benchmark our findingson widely used CIFAR-10 and CIFAR-100 datasets and observe the improved quality of the chosensubsets across different subset sizes, which ensures faster deep neural network training.
Revised: We benchmark our findings on widely used CIFAR-10 and CIFAR-100 datasets and observe the improved quality of the chosen subsets across different subset sizes, which ensures faster deep neural network training.

Original:  Background2.1 Problem Statement
Revised:  Background 2.1 Problem Statement

Original: We formulate the problem in the paper as follows.
Revised: We formulate the problem in the paper as follows.

Original: A calibration technique C is performed duringtraining at each epoch, and a sample prioritization function a is then used to select the most informativesamples for training each subsequent epoch.
Revised: A calibration technique C is performed during training at each epoch, and a sample prioritization function a is then used to select the most informative samples for training each subsequent epoch.

Original: We use Expected Calibration Error (ECE) for modelcalibration [10], which measures the absolute difference between the model’s accuracy and itsconfidence.
Revised: We use Expected Calibration Error (ECE) for model calibration [10], which measures the absolute difference between the model’s accuracy and its confidence.

Original: The paper discusses how a calibration technique C , when coupled with a sample prioritizationfunction a , affects the performance (accuracy and calibration error (ECE)) of the model.
Revised: The paper discusses how a calibration technique C , when coupled with a sample prioritization function a , affects the performance (accuracy and calibration error (ECE)) of the model.

Original: In addition,we also observe if this phenomenon can aid in faster and more efficient training.
Revised: In addition, we also observe if this phenomenon can aid in faster and more efficient training.

Original: We hypothesizea closer relationship between calibration and sample prioritization during training wherein, thecalibrated model probabilities at each epoch are used by a sample prioritization criterion to select themost informative samples for training each subsequent epoch.
Revised: We hypothesize a closer relationship between calibration and sample prioritization during training, wherein the calibrated model probabilities at each epoch are used by a sample prioritization criterion to select the most informative samples for training each subsequent epoch.

Original: 2.2 Calibration
Revised: 2.2 Calibration

Original: Calibration is a technique that curbs overconfident predictions in deep neural networks, wherein thepredicted (softmax) probabilities reflect true probabilities of correctness (better confidence estimates)[1].
Revised: Calibration is a technique that curbs overconfident predictions in deep neural networks, wherein the predicted (softmax) probabilities reflect true probabilities of correctness (better confidence estimates) [1].

Original: In this paper, we consider various prominently used calibration techniques which are performedduring training.
Revised: In this paper, we consider various prominently used calibration techniques which are performed during training.

Original: Label Smoothing[9] implicitly calibrates a model by discouraging overconfident prediction probabilities during training.
Revised: Label Smoothing implicitly calibrates a model by discouraging overconfident prediction probabilities during training [9].

Original: The one-hot encoded ground truth labels ( y k ) are smoothened using a parameterα , that is y LSk =
Revised: The one-hot encoded ground truth labels ( y k ) are smoothened using a parameter α , that is y LSk =

Original: y k (1 − α ) + α/K , where K is the number of classes.
Revised: y k (1 − α ) + α/K , where K is the number of classes.

Original: These smoothened targetsy LSk and predicted outputs p k are then used to minimize the cross-entropy loss.
Revised: These smoothened targets y LSk and predicted outputs p k are then used to minimize the cross-entropy loss.

Original: Mixup is a data augmentation method [14] which is shown to output well-calibrated predictive scores[13], and is again performed during training.
Revised: Mixup is a data augmentation method [14] which is shown to output well-calibrated predictive scores [13], and is again performed during training.

Original: [Equation]
Revised: [Equation]

Original: where x i and x j are two input data points that are randomly sampled, and y i and y j are their respectiveone-hot encoded labels.
Revised: where x i and x j are two input data points that are randomly sampled, and y i and y j are their respective one-hot encoded labels.

Original: Here, λ ∼ Beta ( α, α ) with λ ∈ [0 , 1] .
Revised: Here, λ ∼ Beta ( α, α ) with λ ∈ [0 , 1] .

Original: Focal Loss is an alternative loss function to cross-entropy which yields calibrated probabilities by minimizing a regularized KL divergence between the predicted and target distributions [8].
Revised: Focal Loss is an alternative loss function to cross-entropy which yields calibrated probabilities by minimizing a regularized KL divergence between the predicted and target distributions [8].

Original: [Equation]
Revised: [Equation]

Original: where p is the probability assigned by the model to the ground-truth correct class, and γ is ahyperparameter.
Revised: where p is the probability assigned by the model to the ground-truth correct class, and γ is a hyperparameter.

Original: When compared with cross-entropy, focal loss has an added factor that encouragesthe samples predicted with correct classes to have lower probabilities.
Revised: When compared with cross-entropy, Focal Loss has an added factor that encourages the samples predicted with correct classes to have lower probabilities.

Original: This enables the predicteddistribution to have higher entropy, thereby helping avoid overconfident predictions.
Revised: This enables the predicted distribution to have higher entropy, thereby helping avoid overconfident predictions.

Original: 2.3 Sample Prioritization
Revised: 2.3 Sample Prioritization

Original: Sample prioritization/subset selection is the process of selecting important samples during differentstages of training to accelerate the training process of a deep neural network without compromisingon performance.
Revised: Sample prioritization is the process of selecting important samples during different stages of training to accelerate the training process of a deep neural network without compromising on performance.

Original: In this paper, we perform sample prioritization during training using Max Entropy ,which is a de facto uncertainty sampling technique to select the most efficient samples at each epoch.
Revised: In this paper, we perform sample prioritization during training using Max Entropy , which is a de facto uncertainty sampling technique to select the most efficient samples at each epoch.

Original: Max Entropy selects the most informative samples (top- k ) that maximize the predictive entropy [12].
Revised: Max Entropy selects the most informative samples (top- k ) that maximize the predictive entropy [12].

Original: [Equation]
Revised: [Equation]

Original: 2.4 Pre-trained Calibrated Target models
Revised: 2.4 Pre-trained Calibrated Target models

Original: Pre-trained models have been widely used in literature to obtain comprehensive sample representationsbefore training a downstream task [5].
Revised: Pre-trained models have been widely used in literature to obtain comprehensive sample representations before training a downstream task [5].

Original: We use a pre-trained calibratedtarget model with largercapacity [3] and use the Max Entropy estimates obtained from this target model at each epoch toselect the samples, thereby guiding the corresponding epochs of the model during training.
Revised: We use a pre-trained calibrated model with larger capacity which we call the target model [3] and use the Max Entropy estimates obtained from this target model at each epoch to select the samples, thereby guiding the corresponding epochs of the model during training.

Original: This isperformed in addition to calibrating a model during training.
Revised: Further, we call the model which is being trained as the current model. In this paper, we perform sample prioritization with the target model in addition to calibrating the current model. [Table]

Original:  Experiments and Results
Revised:  Experiments and Results

Original: We perform our experiments on CIFAR-10 and CIFAR-100 [4] datasets with the setting mentionedin Section 2.1, and we usethe Resnet-34architecture [2] for training.
Revised: We perform our experiments on CIFAR-10 and CIFAR-100 [4] datasets with the setting mentioned in Section 2.1, and we use Resnet-34 [2] as the current model.

Original: For both datasets, the initial[Table] [Figure] training set  of 50000 samples is split into 90% training data and 10% validation data, while the testset contains 10000 samples.
Revised: For both datasets, the initial training set consisting of 50,000 samples is split into 90% training data and 10% validation data, while the test set contains 10,000 samples.

Original: In the sample prioritization setting, we start with 10 warm-up epochs inwhich all samples are selected during training (no subset selection), following which we select n % oftotal training samples in each epoch using the Max Entropy criterion.
Revised: In the sample prioritization setting, we start with 10 warm-up epochs in which all samples are selected during training (no subset selection), following which we select n % of total training samples in each epoch using the Max Entropy criterion.

Original: We choose different settings ofsubset sizes for each epoch with n – {10, 20, 30}.
Revised: We choose different settings of subset sizes for each epoch with n – {10, 20, 30}.

Original: For both datasets, we use the Stochastic Gradient Descent optimizer with initial learning rates of 0.01and 0.1  respectively,  with a cosine annealing [7] scheduler and 200 training epochs, weight decay of5 e − 4 and momentum of 0 . 9 .
Revised: We use the Stochastic Gradient Descent optimizer with initial learning rates of 0.01 and 0.1 for CIFAR-10 and CIFAR-100 respectively, trained for 200 epochs with a cosine annealing [7] scheduler, weight decay of 5 e − 4 and momentum of 0 . 9 for both datasets.

Original: The models are trained using a V100 GPU.
Revised: The models are trained using a V GPU.

Original: We consider classificationaccuracies and ECEs across different calibration techniques with their respective parameter sweepsas follows: 
Revised: We consider classification accuracies and ECEs across different calibration techniques with their respective parameter sweeps as follows: Label Smoothing ( α ) – {0.01, 0.03, 0.05, 0.07, 0.09},

Original: Label Smoothing ( α ) – {0.01, 0.03, 0.05, 0.07, 0.09}, Mixup ( α ) – {0.1, 0.15, 0.2, 0.25,0.3, 0.35} and Focal Loss ( γ ) – {1, 2, 3, 4, 5}.
Revised: Mixup ( α ) – {0.1, 0.15, 0.2, 0.25, 0.3, 0.35} and Focal Loss ( γ ) – {1, 2, 3, 4, 5}.

Original: As baselines, we consider uncalibrated models withstandard cross-entropy loss.
Revised: As baselines, we consider uncalibrated models with standard cross-entropy loss.

Original: For the target experiments, we choose Resnet-50 models [2] trained with Mixup (with α = 0 .
Revised: For the target experiments, we choose Resnet-50 models [2] trained with Mixup (with α = 0 .

Original: 3 for CIFAR-10, and α = 0 .
Revised: 3 for CIFAR-10, and α = 0 .

Original: 25 for CIFAR-100) as our calibrated pre-trainedtarget models after performing parameter sweeps across all calibration techniques.
Revised: 25 for CIFAR-100) as our target models after performing parameter sweeps across all calibration techniques.

Original: 3.1 Discussion on Results
Revised: 3.1 Discussion on Results

Original: Table 1 shows the test ECEs and accuracies  across different calibration techniques and subset sizeswith Max Entropy criterion.
Revised: Table 1 shows the test ECEs and accuracies for the current model across different calibration techniques and subset sizes with Max Entropy criterion.

Original: We can observe that all calibration techniques have lower test ECEsthan their respective uncalibrated models for all subset sizes.
Revised: We can observe that all calibration techniques have lower test ECEs than their respective uncalibrated models across all subset sizes for both datasets.

Revised: This demonstrates that performing calibration during training improves sample prioritization .

Revised: Moreover, these results indicate that there are no significant trade-offs between model accuracy and model confidence (ECE) when calibration is performed with sample prioritization.

Original: Figures 1a and 1b also illustrate lowervalidation ECEs across training epochs for calibrated  models than their respective uncalibratedmodels, for 30% subset size.
Revised: Figures 1a and 1b also illustrate lower validation ECEs across training epochs for calibrated current models when compared to their uncalibrated counterparts, with 30% subset size as an instance.

Original: In addition, we observe that test accuracies for Mixup  are consistentlycomparable and often higher than their respective uncalibrated models for both datasets.
Revised: In particular, we observe that test accuracies for Mixup across all subset sizes are consistently comparable and often higher than their respective uncalibrated models. [Figure] [Table]

Original: These results
Revised: As expected

Original: [Table]
Revised: [Table]

Original: indicate that there are no significant trade-offs between model accuracy and model confidence (ECE)when calibration is performed with sample prioritization, thereby demonstrating that performingcalibration during training improves sample prioritization .

Original: As expected, we observe  that the test accuracies reduce when the subset size becomes smaller.
Revised: we observe from Table 1 that the test accuracies reduce when the subset size becomes smaller.

Original: Moreover, Mixup consistently has higher test accuracies and low test ECEs across different subsetsizes compared to other calibration techniques for both datasets.
Revised: Moreover, Mixup consistently has higher test accuracies and low test ECEs across different subset sizes compared to other calibration techniques for both datasets.

Original: This could be attributed to Mixupbeing a data transformation/augmentation technique, thereby performing well even in the low-dataregime.
Revised: This could be attributed to Mixup being a data transformation/augmentation technique, thereby performing well even in the low-data regime.

Original: Further,  training with Mixup leads to a  balanced representation of classes in the chosensubsets (Refer Appendix A Figure 2a).
Revised: Further, Figure 2 shows that training with Mixup leads to a relatively balanced representation of classes in the chosen subsets.

Original: In contrast, Label Smoothing and Focal Loss are loss-basedcalibration techniques with no explicit transformation performed on the underlying training data.
Revised: In contrast, Label Smoothing and Focal Loss are loss-based calibration techniques with no explicit transformation performed on the underlying training data.

Original: We also observe that Mixup has lower test ECEs and also consistently high test accuracies across theboard on both datasets, contrary to Focal Loss which results in lower test ECEs only for CIFAR-10with comparable test accuracies.

Original: Here, it is important to note that Focal Loss is more effective whencoupled with a post hoc calibration technique like temperature scaling [8].
Revised: Here, we note that Focal Loss can be more effective when coupled with a post hoc calibration technique like temperature scaling [8].

Original: However, this setting isnot applicable in our paper since we explicitly focus on calibration techniques during training.
Revised: However, this setting is not applicable in our paper since we explicitly focus on calibration techniques during training.

Revised: In addition, Mixup and Focal Loss are known to outperform Label Smoothing [13, 8].

Original: Effectiveness of Target:
Revised: Effectiveness of Target:

Original: Table 2 exhibits the results when acalibrated pre-trained  target (Resnet-50with Mixup) is used for guiding  sample prioritization with calibration during training.
Revised: Table 2 exhibits the results when a pre-trained calibrated target model (Resnet-50 with Mixup) is used for guiding the current model while performing sample prioritization with calibration during training.

Original: Interestingly, awell-calibrated target model can boost the performance of under-performing calibration techniques(like Label Smoothing for CIFAR-10 and CIFAR-100) performed  without a target.
Revised: Interestingly, a well-calibrated target model can boost the performance on both datasets for under-performing calibration techniques (like Label Smoothing) performed only on the current model without the target’s influence.

Original: However, forcalibration techniques that are already performing well (like Mixup and Focal Loss), there is nosignificant loss in performance on CIFAR10, while there is a significant improvement in performanceon CIFAR100.
Revised: However, for calibration techniques that are already performing well (like Mixup and Focal Loss), there is no significant loss in performance on CIFAR-10, while there is a considerable improvement in performance in general on CIFAR-100.

Original: This can be clearly observed when comparing Table 2 with Table 1.
Revised: This can be clearly observed when comparing Table 2 with Table 1.

Original: We assume thatthe performance of an uncalibrated model(no calibration) trained with full (100%) data is similar forexperiments performed with and without a target model.
Revised: We assume that the performance of a current model trained with 100% data is similar for all experiments performed with and without a target model for each respective dataset. [Figure]

Revised: Class Distribution: Figures 2a - 2d show the class distribution across training epochs for different calibration techniques with sample prioritization performed using Max Entropy.

Revised: For comparison, we also consider Random sampling as a baseline (Figure 2e).

Revised: As discussed above, performing

Original:   Conclusion
Revised: Mixup (Figure 2a) during training leads to a relatively balanced representation of classes in each epoch. In contrast, Label Smoothing and Focal Loss (Figures 2b and 2c) do not exhibit a balanced representation of classes in the chosen subsets. The class representations are imbalanced for the uncalibrated setting as well (Figure 2d).  Conclusion

Original: In this paper, we investigate whether existing calibration techniquescan improve sample prioritizationduring training.
Revised: In this paper, we investigate whether existing calibration techniques improve sample prioritization during training.

Original: We empirically showcase that a deep neural networkthat is calibrated during training selects better subsets of samples than an uncalibrated model.
Revised: We empirically show that a deep neural network calibrated during training selects better subsets of samples than an uncalibrated model.

Original: We also demonstrate the effectiveness of pre-trained calibrated target models in boosting calibration efficiencies when performing sample prioritization during training.
Revised: We also demonstrate the effectiveness of pre-trained calibrated target models in guiding sample prioritization during training, thereby boosting calibration performance.

Original: Finally,we show that since calibration improves sample prioritization, it also accelerates neural network training multifold
Revised: Finally, since calibration aids in sample prioritization by improving the quality of subsets, it ensures faster neural network training.

Original: B Common Samples between Epochs
Revised: A Common Samples between Epochs

Original: [Figure] Figures 3a and 3b show the percentage of common samples between consecutive epochs for CIFAR-10and CIFAR-100 respectively, across various calibration techniques, with Max Entropy as the sampleprioritization mechanism with 30% subset size.
Revised: Figures 3a and 3b show the percentage of common samples between consecutive epochs for CIFAR10 and CIFAR-100 respectively across various calibration techniques, with Max Entropy and Random

Original: For comparison, we consider random sampling as abaseline sample prioritization technique.
Revised: Sampling as the sample prioritization criteria with 30% subset size.

Original: In all settings, sample selection based on Max Entropy leadsto a significantly higher percentage of common samples between consecutive epochs. In contrast,random selection results in very few common samples between consecutive epochs throughouttraining.
Revised: In all settings, sample selection with Max Entropy leads to a significantly higher percentage of common samples between consecutive epochs throughout training, as opposed to random sampling which results in very few common samples.

Original: Despite changing only a small percentage of samples  across epochs, sample prioritizationwith Max Entropy coupled with any calibration technique performs well in terms of classificationaccuracy and ECE. C Validation Accuracies
Revised: Although only a small percentage of samples change across epochs, sample prioritization with Max Entropy coupled with any calibration technique yields good classification performance.

Original: [Figure]
Revised: [Figure]

Revised: B Validation Accuracies

Original: Wealso report the validation accuracies of CIFAR-10 and CIFAR-100 with 30% subset size  across allcalibration techniques with Max Entropy as the sample prioritization technique in Figures 4a and 4b. 
Revised: We report the validation accuracies of CIFAR-10 and CIFAR-100 with 30% subset size for the current model across all calibration techniques with Max Entropy as the sample prioritization criterion in Figures 4a and 4b. [Figure]

Original: D Reliability Diagrams
Revised: C Reliability Diagrams

Original: We report the reliability diagrams of CIFAR-10 and CIFAR-100 with 30% subset size  across allcalibration techniques with Max Entropy as the sample prioritization criterion in Figures 5 and 6.
Revised: We report the reliability diagrams of CIFAR-10 and CIFAR-100 with 30% subset size for the current model across all calibration techniques with Max Entropy as the sample prioritization criterion in Figure 5 and 6.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]