Original: M ULTI -C LASS U NCERTAINTY C ALIBRATION VIA M UTUAL I NFORMATION M AXIMIZATION - BASED B INNING
Revised: M ULTI -C LASS U NCERTAINTY C ALIBRATION VIA M UTUAL I NFORMATION M AXIMIZATION - BASED B INNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Post-hoc  calibration is a common approach for providing high-quality conﬁdence estimates of deep neural network predictions.
Revised: Post-hoc multi-class calibration is a common approach for providing high-quality conﬁdence estimates of deep neural network predictions.

Original: Recent work has shown that widely used scaling methods underestimate their calibration error, while alternative Histogram Binning (HB) methods often fail to preserve classiﬁcation accuracy.
Revised: Recent work has shown that widely used scaling methods underestimate their calibration error, while alternative Histogram Binning (HB) methods often fail to preserve classiﬁcation accuracy.

Original: In the case of multi-class calibration with a large number of classes K , HB also faces the issue of severe sample-inefﬁciency due to a large class imbalance resulting from the conversion into K one-vs-rest class-wise calibration problems.
Revised: When classes have small prior probabilities, HB also faces the issue of severe sample-inefﬁciency after the conversion into K one-vs-rest class-wise calibration problems.

Original: The goal of this paper is to resolve the identiﬁed issues of HB in order to provide calibrated conﬁdence estimates using only a small holdout calibration dataset for bin optimization while preserving multi-class ranking accuracy.
Revised: The goal of this paper is to resolve the identiﬁed issues of HB in order to provide calibrated conﬁdence estimates using only a small holdout calibration dataset for bin optimization while preserving multi-class ranking accuracy.

Original: From an information-theoretic perspective, we derive the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits.
Revised: From an information-theoretic perspective, we derive the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits.

Original: This concept mitigates potential loss in ranking performance due to lossy quantization, and by disentangling the optimization of bin edges and representatives allows simultaneous improvement of ranking and calibration performance.
Revised: This concept mitigates potential loss in ranking performance due to lossy quantization, and by disentangling the optimization of bin edges and representatives allows simultaneous improvement of ranking and calibration performance.

Original: To improve the sample efﬁciency and estimates from a small calibration set, we propose a shared class-wise (sCW) strategy that ﬁts a single calibrator on themerged training sets of all K class-wise problems.
Revised: To improve the sample efﬁciency and estimates from a small calibration set, we propose a shared class-wise (sCW) calibration strategy, sharing one calibrator among similar classes (e.g., with similar class priors) so that the training sets of their class-wise calibration problems can be merged to train the single calibrator.

Original: The combination of sCW and I-Max binning outperforms the state of the art calibration methods on various evaluation metrics across different benchmark datasets and models, using a small calibration set (e.g. 1k samples for ImageNet).
Revised: The combination of sCW and I-Max binning outperforms the state of the art calibration methods on various evaluation metrics across different benchmark datasets and models, using a small calibration set (e.g., 1k samples for ImageNet).

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite great ability in learning discriminative features, deep neural network (DNN) classiﬁers often make over-conﬁdent predictions.
Revised: Despite great ability in learning discriminative features, deep neural network (DNN) classiﬁers often make over-conﬁdent predictions.

Original: This can lead to potentially catastrophic consequences in safety critical applications, e g., medical diagnosis andperception tasks in autonomous driving.
Revised: This can lead to potentially catastrophic consequences in safety critical applications, e.g., medical diagnosis and autonomous driving perception tasks.

Original: A multi-class classiﬁer is perfectly calibrated if among the cases receiving the prediction distribution q , the ground truth class distribution is also q .
Revised: A multi-class classiﬁer is perfectly calibrated if among the cases receiving the prediction distribution q , the ground truth class distribution is also q .

Original: The mismatch between the prediction and ground truth distribution can be measured through the Expected Calibration Error (ECE) (Guo et al., 2017).
Revised: The mismatch between the prediction and ground truth distribution can be measured using the Expected Calibration Error (ECE) (Guo et al., 2017; Kull et al., 2019).

Original: Since the pioneering work of (Guo et al., 2017), scaling methods have been widely acknowledged as an efﬁcient post-hoc  calibration solution for modern DNNs.
Revised: Since the pioneering work of (Guo et al., 2017), scaling methods have been widely acknowledged as an efﬁcient post-hoc multi-class calibration solution for modern DNNs.

Original: The common practice of evaluating their ECE resorts to histogram density estimation (HDE) for modeling the distribution of the predictions.
Revised: The common practice of evaluating their ECE resorts to histogram density estimation (HDE) for modeling the distribution of the predictions.

Original: However, Vaicenavicius et al.
Revised: However, Vaicenavicius et al.

Original: (2019) proved that with a ﬁxed number of evaluation bins the ECE of scaling methods is underestimated even with an inﬁnite number of samples.
Revised: (2019) proved that with a ﬁxed number of evaluation bins the ECE of scaling methods is underestimated even with an inﬁnite number of samples.

Original: Widmann et al.
Revised: Widmann et al.

Original:  Kumar et al.
Revised: ; Kumar et al.

Original: ; Wenger et al.
Revised: ; Wenger et al.

Original: (2020) also empirically showed this underestimation phenomena.
Revised: (2020) also empirically showed this underestimation phenomena.

Original: This deems scaling methods as unreliable calibration solutions, as their true ECEs can be larger than evaluated, putting many applications at risk.
Revised: This deems scaling methods as unreliable calibration solutions, as their true ECEs can be larger than evaluated, putting many applications at risk.

Original: Additionally, setting HDE also faces the bias/variance trade-off.
Revised: Additionally, setting HDE also faces the bias/variance trade-off.

Original: Increasing its number of evaluation bins reduces the bias, as the evaluation quantization error is smaller, however, the estimation of the ground truth correctness begins to suffer from high variance.
Revised: Increasing its number of evaluation bins reduces the bias, as the evaluation quantization error is smaller, however, the estimation of the ground truth correctness begins to suffer from high variance.

Original: Fig.
Revised: Fig.

Original: 1-a) shows that the empirical ECE estimates of both the raw network outputs and the temperature scaling method (TS) (Guo et al., 2017) are sensitive to the number of evaluation bins.
Revised: 1-a) shows that the empirical ECE estimates of both the raw network outputs and the temperature scaling method (TS) (Guo et al., 2017) are sensitive to the number of evaluation bins.

Original: It remains unclear how to optimally choose the number of evaluation bins so as to minimize the estimation error.
Revised: It remains unclear how to optimally choose the number of evaluation bins so as to minimize the estimation error.

Original: Recent work (Zhang et al., 2020; Widmann et al., 2019) suggested kernel density estimation (KDE) instead of HDE.
Revised: Recent work (Zhang et al., 2020; Widmann et al., 2019) suggested kernel density estimation (KDE) instead of HDE.

Original: However, the choice of the kernel and bandwidth also remains unclear, and the smoothness of the ground truth distribution is hard to verify in practice.
Revised: However, the choice of the kernel and bandwidth also remains unclear, and the smoothness of the ground truth distribution is hard to verify in practice.

Original: [Figure]
Revised: [Figure]

Original:  An alternative technique for post-hoc calibration is Histogram Binning (HB) 
Revised: reliable An alternative technique for post-hoc calibration is Histogram Binning (HB) (Zadrozny & Elkan, 2001;

Original: Guo et al.
Revised: Guo et al., 2017; Kumar et al., 2019).

Original: Note, here HB serves as a calibration method and is different to the HDE used for evaluating ECEs of scaling methods.
Revised: Note, here HB is a calibration method and is different to the HDE used for evaluating ECEs of scaling methods.

Original: HB produces discrete predictions, whose probability mass functions can be empirically estimated without using HDE/KDE.
Revised: HB produces discrete predictions, whose probability mass functions can be empirically estimated without using HDE/KDE.

Original: Therefore, its ECE estimate is constant and unaffected by the number of evaluation bins in Fig.
Revised: Therefore, its ECE estimate is constant and unaffected by the number of evaluation bins in Fig.

Original: 1-a)
Revised: 1-a)

Original: and it can converge to the true value with increasing evaluation samples (Vaicenavicius et al., 2019), see Fig.
Revised: and it can converge to the true value with increasing evaluation samples (Vaicenavicius et al., 2019), see Fig.

Original: 1-b).
Revised: 1-b).

Original: The most common variants of HB are Equal (Eq.)
Revised: The most common variants of HB are Equal (Eq.)

Original: size (uniformly partitioning the probability interval [0 , 1] ), and Eq.
Revised: size (uniformly partitioning the probability interval [0 , 1] ), and Eq.

Original: mass (uniformly distributing samples over bins) binning.
Revised: mass (uniformly distributing samples over bins) binning.

Original: These simple methods for multi-class calibration are known to degrade accuracy, since quantization through binning may remove a considerable amount of label information contained by the classiﬁer’s outputs.
Revised: These simple methods for multi-class calibration are known to degrade accuracy, since quantization through binning may remove a considerable amount of label information contained by the classiﬁer’s outputs.

Original: In this work we show that the key for HB to retain the accuracy of trained classiﬁers is choosing bin edges that minimize the amount of label information loss.
Revised: In this work we show that the key for HB to retain the accuracy of trained classiﬁers is choosing bin edges that minimize the amount of label information loss.

Original: Both Eq.
Revised: Both Eq.

Original: size and mass binning are suboptimal.
Revised: size and mass binning are suboptimal.

Original: We present I-Max , a novel iterative method for optimizing bin edges with proved convergence.
Revised: We present I-Max , a novel iterative method for optimizing bin edges with proved convergence.

Original: As the location of its bin edges inherently ensures sufﬁcient calibration samples per bin, the bin representatives of I-Max can then be effectively optimized for calibration.
Revised: As the location of its bin edges inherently ensures sufﬁcient calibration samples per bin, the bin representatives of I-Max can then be effectively optimized for calibration.

Original: Two design objectives, calibration and accuracy, are thus nicely disentangled under I-Max.
Revised: Two design objectives, calibration and accuracy, are thus nicely disentangled under I-Max.

Original: For multi-class calibration, I-Max adopts the one-vs-rest (OvR) strategy to individually calibrate the prediction probability of each class.
Revised: For multi-class calibration, I-Max adopts the one-vs-rest (OvR) strategy to individually calibrate the prediction probability of each class.

Original: To cope with a limited number of calibration samples, we propose to ﬁt one binning scheme for  all per-class calibrations, which we call shared class-wise (sCW) I-Max binning.
Revised: To cope with a limited number of calibration samples, we propose to share one binning scheme for calibrating the prediction probabilities of similar classes, e.g., with similar class priors or belonging to the same class category. At small data regime, we can even choose to ﬁt one binning scheme on the merged training sets of all per-class calibrations.

Original: For large number of classes, sCW greatly improves the sample efﬁciency.
Revised: Such a shared class-wise (sCW) calibration strategy greatly improves the sample efﬁciency of I-Max binning.

Original: I-Max is evaluated according to multiple performance metrics, including accuracy, ECE, Brier and
Revised: I-Max is evaluated according to multiple performance metrics, including accuracy, ECE, Brier and

Original: NLL, and compared against benchmark calibration methods across multiple datasets and trained classiﬁers.
Revised: NLL, and compared against benchmark calibration methods across multiple datasets and trained classiﬁers.

Original: For ImageNet, I-Max obtains up to 66 .
Revised: For ImageNet, I-Max obtains up to 66 .

Original: 11 % reduction in ECE compared to the baseline and up to 38 .
Revised: 11 % reduction in ECE compared to the baseline and up to 38 .

Original: 14 % reduction compared to the state-of-the-art GP-scaling method (Wenger et al., 2020).
Revised: 14 % reduction compared to the state-of-the-art GP-scaling method (Wenger et al., 2020).

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: For conﬁdence calibration, Bayesian DNNs, e.g. (Blundell et al., 2015)and their approximations (Gal & Ghahramani, 2016) are resource-demanding methods to consider predictive model uncertainty.
Revised: For conﬁdence calibration, Bayesian DNNs and their approximations, e.g. (Blundell et al., 2015) (Gal & Ghahramani, 2016) are resource-demanding methods to consider predictive model uncertainty.

Original: However, applications with limited complexity overhead and latency require sampling-free and singlemodel based calibration methods.
Revised: However, applications with limited complexity overhead and latency require sampling-free and singlemodel based calibration methods.

Original: Examples include modifying the training loss (Kumar et al., 2018), scalable Gaussian processes (Milios et al., 2018), sampling-free uncertainty estimation (Postels et al., 2019), data augmentation (Patel et al., 2019; Thulasidasan et al., 2019;
Revised: Examples include modifying the training loss (Kumar et al., 2018), scalable Gaussian processes (Milios et al., 2018), sampling-free uncertainty estimation (Postels et al., 2019), data augmentation (Patel et al., 2019; Thulasidasan et al., 2019;

Original: Yun et al., 2019; Hendrycks et al., 2020) and ensemble distribution distillation (Malinin et al., 2020).
Revised: Yun et al., 2019; Hendrycks et al., 2020) and ensemble distribution distillation (Malinin et al., 2020).

Original: In comparison, a simple approach that requires no retraining of the models is post-hoc calibration (Guo et al., 2017).
Revised: In comparison, a simple approach that requires no retraining of the models is post-hoc calibration (Guo et al., 2017).

Original: Prediction probabilities (logits) scaling and binning are the two main solutions for post-hoc calibration.
Revised: Prediction probabilities (logits) scaling and binning are the two main solutions for post-hoc calibration.

Original: Scaling methods use parametric or non-parametric models to adjust the raw logits.
Revised: Scaling methods use parametric or non-parametric models to adjust the raw logits.

Original: Guo et al.
Revised: Guo et al.

Original: (2017) investigated linear models, ranging from the single-parameter based TS to more complicated vector/matrix scaling.
Revised: (2017) investigated linear models, ranging from the single-parameter based TS to more complicated vector/matrix scaling.

Original: To avoid overﬁtting, Kull et al.
Revised: To avoid overﬁtting, Kull et al.

Original: (2019) suggested to regularize matrix scaling with a L 2 loss on the model weights.
Revised: (2019) suggested to regularize matrix scaling with a L 2 loss on the model weights.

Original: Recently, Wenger et al.
Revised: Recently, Wenger et al.

Original: (2020) adopted a latent Gaussian process for multi-class calibration.
Revised: (2020) adopted a latent Gaussian process for multi-class calibration.

Original: Ji et al.
Revised: Ji et al.

Original: (2019) extended TS to a bin-wise setting, by learning separate temperatures for various conﬁdence subsets.
Revised: (2019) extended TS to a bin-wise setting, by learning separate temperatures for various conﬁdence subsets.

Original: To improve the expressive capacity of TS, an ensemble of temperatures were adopted by Zhang et al.
Revised: To improve the expressive capacity of TS, an ensemble of temperatures were adopted by Zhang et al.

Original: Owing to continuous outputs of scaling methods, one critical issue discovered in the recent work is: Their empirical ECE estimate is not only non-veriﬁable (Kumar et al., 2019), but also asymptotically smaller than the ground truth (Vaicenavicius et al., 2019).
Revised: Owing to continuous outputs of scaling methods, one critical issue discovered in the recent work is: Their empirical ECE estimate is not only non-veriﬁable (Kumar et al., 2019), but also asymptotically smaller than the ground truth (Vaicenavicius et al., 2019).

Original: Recent work (Zhang et al., 2020; Widmann et al., 2019) exploited
Revised: Recent work (Zhang et al., 2020; Widmann et al., 2019) exploited

Original: KDEs for an improved ECE evaluation, however, the parameter setting requires further investigation.
Revised: KDEs for an improved ECE evaluation, however, the parameter setting requires further investigation.

Original: Nixon et al.
Revised: Nixon et al.

Original: (2019) discussed potential issues of the ECE metric, and  suggested to 1) use equal mass binning for ECE evaluation; 2) measure both top-1 and class-wise ECE to evaluate multi-class calibrators, 3) only include predictions with a conﬁdence above some epsilon in the class-wise ECE score.
Revised: and (Ashukha et al., 2020) discussed potential issues of the ECE metric, and the former suggested to 1) use equal mass binning for ECE evaluation; 2) measure both top-1 and class-wise ECE to evaluate multi-class calibrators, 3) only include predictions with a conﬁdence above some epsilon in the class-wise ECE score.

Original: As an alternative to scaling, HB quantizes the raw conﬁdences with either Eq.
Revised: As an alternative to scaling, HB quantizes the raw conﬁdences with either Eq.

Original: size or Eq.
Revised: size or Eq.

Original: mass bins (Zadrozny & Elkan, 2001).
Revised: mass bins (Zadrozny & Elkan, 2001).

Original: It offers asymptotically convergent ECE estimation (Vaicenavicius et al., 2019), but is less sample efﬁcient than scaling methods and also suffers from accuracy loss (Guo et al., 2017).
Revised: It offers asymptotically convergent ECE estimation (Vaicenavicius et al., 2019), but is less sample efﬁcient than scaling methods and also suffers from accuracy loss (Guo et al., 2017).

Original: Kumar et al.
Revised: Kumar et al.

Original: (2019) proposed to perform scaling before binning for an improved sample efﬁciency.
Revised: (2019) proposed to perform scaling before binning for an improved sample efﬁciency.

Original: Isotonic regression (Zadrozny & Elkan, 2002) and Bayesian binning into quantiles (BBQ)
Revised: Isotonic regression (Zadrozny & Elkan, 2002) and Bayesian binning into quantiles (BBQ)

Original: (Naeini et al., 2015) are often viewed as binning methods.
Revised: (Naeini et al., 2015) are often viewed as binning methods.

Original: However, their ECE estimates face the same issue as scaling methods: though isotonic regression ﬁts a piecewise linear function, its predictions are continuous as they are interpolated for unseen data.
Revised: However, their ECE estimates face the same issue as scaling methods: though isotonic regression ﬁts a piecewise linear function, its predictions are continuous as they are interpolated for unseen data.

Original: BBQ considers multiple binning schemes with different numbers of bins, and combines them using a continuous Bayesian score, resulting in continuous predictions.
Revised: BBQ considers multiple binning schemes with different numbers of bins, and combines them using a continuous Bayesian score, resulting in continuous predictions.

Original: In this work, we improve the current HB design by casting bin optimization into a MI maximization problem.
Revised: In this work, we improve the current HB design by casting bin optimization into a MI maximization problem.

Original: Furthermore, our ﬁndings can also be used to improve scaling methods.
Revised: Furthermore, our ﬁndings can also be used to improve scaling methods.

Original:  M ETHOD
Revised:  M ETHOD

Original: Here we introduce the I-Max binning scheme, which addresses the issues of HB in terms of preserving label-information in multi-class calibration.
Revised: Here we introduce the I-Max binning scheme, which addresses the issues of HB in terms of preserving label-information in multi-class calibration.

Original: After the problem setup in Sec. 3.1, Sec. 3.2) presents a sample-efﬁcient technique for one-vs-rest calibration.
Revised: After the problem setup in Sec. 3.1, Sec. 3.2) presents a sample-efﬁcient technique for one-vs-rest calibration.

Original: In Sec.
Revised: In Sec.

Original: 3.3 we formulate the training objective of binning as MI maximization and derive a simple algorithm for I-Max binning.
Revised: 3.3 we formulate the training objective of binning as MI maximization and derive a simple algorithm for I-Max binning.

Original: 3. P ROBLEM S ETUP
Revised: 3. P ROBLEM S ETUP

Original: We address supervised multi-class classiﬁcation tasks, where each input x ∈ X belongs to one of
Revised: We address supervised multi-class classiﬁcation tasks, where each input x ∈ X belongs to one of

Original: K classes, and the ground truth labels are one-hot encoded
Revised: K classes, and the ground truth labels are one-hot encoded

Original: , i.e., y =
Revised: , i.e., y =

Original: [ y 1 , y 2 , . . .
Revised: [ y 1 , y 2 , . . .

Original: , y K ] ∈ { 0 , 1 } K .
Revised: , y K ] ∈ { 0 , 1 } K .

Original: Let f : X
Revised: Let f : X

Original: (cid:55)→ [0 , 1] K be a DNN trained using cross-entropy loss.
Revised: (cid:55)→ [0 , 1] K be a DNN trained using cross-entropy loss.

Original: It maps each x onto a probability vector q = [ q 1 , . . .
Revised: It maps each x onto a probability vector q = [ q 1 , . . .

Original: , q K ] ∈ [0 , 1] K , which is used to rank the K possible classes of the current instance, e.g., arg max k q k being the top- 1 ranked class.
Revised: , q K ] ∈ [0 , 1] K , which is used to rank the K possible classes of the current instance, e.g., arg max k q k being the top- 1 ranked class.

Original: As the trained classiﬁer tends to overﬁt to the cross-entropy loss rather than the accuracy (i.e., 0 / 1 loss), q as the prediction distribution is typically poorly calibrated.
Revised: As the trained classiﬁer tends to overﬁt to the cross-entropy loss rather than the accuracy (i.e., 0 / 1 loss), q as the prediction distribution is typically poorly calibrated.

Original: A post-hoc calibrator h to revise q can deliver an improved performance.
Revised: A post-hoc calibrator h to revise q can deliver an improved performance.

Original: To evaluate the calibration performance of h ◦ f , class-wise ECE averaged over the K classes is a common metric, measuring the expected deviation of the predicted per-class conﬁdence after calibration, i.e., h k ( q ) , from the ground truth probability p ( y k = 1 | h ( q )) :
Revised: To evaluate the calibration performance of h ◦ f , class-wise ECE averaged over the K classes is a common metric, measuring the expected deviation of the predicted per-class conﬁdence after calibration, i.e., h k ( q ) , from the ground truth probability p ( y k = 1 | h ( q )) :

Original: [Equation]
Revised: [Equation]

Original: When h is a binning scheme, h k ( q ) is discrete and thus repetitive.
Revised: When h is a binning scheme, h k ( q ) is discrete and thus repetitive.

Original: We can then empirically set p ( y k = 1 | h ( q )) as the frequency of label- 1 samples among those receiving the same h k ( q ) .
Revised: We can then empirically set p ( y k = 1 | h ( q )) as the frequency of label- 1 samples among those receiving the same h k ( q ) .

Original: On the contrary, scaling methods are continuous.
Revised: On the contrary, scaling methods are continuous.

Original: It is unlikely that two samples attain the same h k ( q ) , thus requiring additional quantization, i.e., applying HDE for modeling the distribution of h k ( q ) , or alternatively using KDE.
Revised: It is unlikely that two samples attain the same h k ( q ) , thus requiring additional quantization, i.e., applying HDE for modeling the distribution of h k ( q ) , or alternatively using KDE.

Original: It is noted that ideally we should compare the whole distribution h ( q ) with the ground truth p ( y | h ( q )) .
Revised: It is noted that ideally we should compare the whole distribution h ( q ) with the ground truth p ( y | h ( q )) .

Original: However, neither HDE nor KDE scales well with the number of classes.
Revised: However, neither HDE nor KDE scales well with the number of classes.

Original: Therefore, the multi-class ECE evaluation often boils down to the one-dimensional class-wise ECE as in (1) or the top- 1 ECE, i.e., E (cid:2) | p ( y [Equation]
Revised: Therefore, the multi-class ECE evaluation often boils down to the one-dimensional class-wise ECE as in (1) or the top- 1 ECE, i.e., E (cid:2) | p ( y k

Original: =
Revised: =

Original: [Equation]
Revised: argmax

Original:  = 1{+
+} | h ( q ))
Revised: k h k ( q ) = {+
+} | h ( q ))

Original: − max k h k ( q ) | (cid:3) .
Revised: − max k h k ( q ) | (cid:3) .

Original: 3. O NE - VS -R EST (O V R) S TRATEGY FOR M ULTI - CLASS C ALIBRATION
Revised: 3. O NE - VS -R EST (O V R) S TRATEGY FOR M ULTI - CLASS C ALIBRATION

Original: Histogram binning (HB) was initially developed for two-class calibration.
Revised: HB was initially developed for two-class calibration.

Original: When dealing with multiclass calibration, it separately calibrates  each class in a one-vs-rest (OvR) fashion: For any class- k we use y k as the binary label for a two-class calibration task, in which the class- 1 means y k
Revised: When dealing with multi-class calibration, it separately calibrates the prediction probability q k of each class in a one-vs-rest (OvR) fashion: For any class- k , HB takes y k as the binary label for a two-class calibration task in which the class- 1 means y k

Original: = 1 and class0 collects all other K − 1 classes.
Revised: = 1 and class- 0 collects all other K − 1 classes.

Original: HB then revises  q k  by mapping its logit λ k ∆ = log q k − log(1 − q k ) onto a given number of bins, and reproducing them with the empirical frequency of class- 1 samples in each bin. The empirical frequencies are estimated from the class- k training set S k = { ( y k , λ k ) } .
Revised: It then revises the prediction probability q k of y k = 1 by mapping its logit λ k ∆ = log q k − log(1 − q k ) onto a given number of bins, and reproducing it with the calibrated prediction probability.

Original: We choose to bin the logit λ
Revised: Here, we choose to bin the logit λ

Original: k instead of q k , as the former is unbounded, i.e., λ k ∈ R , which eases the bin edge optimization process.
Revised: k instead of q k , as the former is unbounded, i.e., λ k ∈ R , which eases the bin edge optimization process.

Original: Nevertheless, as q k and λ k have  monotonic bijective relation, binning q k and λ k are equivalent.
Revised: Nevertheless, as q k and λ k have a monotonic bijective relation, binning q k and λ k are equivalent.

Original: It is further noted that after K class-wise calibrations we do not perform the extra normalization step as in (Guo et al., 2017).
Revised: It is further noted that after K class-wise calibrations we do not perform the extra normalization step as in (Guo et al., 2017).

Original: After OvR marginalizes the multi-class predictive distribution, each class shall then be treated independently (see Sec. A1).
Revised: After OvR marginalizes the multi-class predictive distribution, each class shall then be treated independently (see Sec. A1).

Original: Training HBs over the sets S 1 , . .
Revised: The calibration performance of HB depends on the setting of its bin edges and representatives.

Original:  , S K  ,  the two-class ratio in them is 1 : ( K − 1) , so when K is large we will need a large calibration set C
Revised: From a calibration set C = { ( y , z ) } , we can construct K training sets, i.e., S k = { ( y k , λ k ) } ∀ k , under the one-vs-rest strategy, and then optimize the class-wise (CW) HB over each training set. As two common solutions in the literature, Eq. size and Eq.

Original:  = { ( y  , x ) } to collect enough class- 1 samples in S k .
Revised: mass binning focus on bin representative optimization. Their bin edge locations, on the other hand, are either ﬁxed (independent of the calibration set) or only ensures a balanced training sample distribution over the bins. After binning the logits in the calibration set S k = { ( y k , λ k ) } , the bin representatives are set as the empirical frequencies of samples with y k = 1 in each bin.

Original: For ImageNet with 1 k classes, S k constructed from the  calibration  set  of size 10 k contains only 10k /K
Revised: To improve the sample efﬁciency of bin representative optimization, Kumar et al. (2019) proposed to perform scaling-based calibration before HB. Namely, after properly scaling the logits { λ k } , the bin representative per bin is then set as the averaged sigmoid -response of the scaled logits in S k belonging to each bin.

Original: = 10 class- 1 samples.
Revised: However, pre-scaling does not resolve the sample inefﬁciency issue arising from a small class prior p k .The

Original: To address this, we propose to merge { S k } across K classes, i.e., S = ∪ k S k 
Revised: two-class ratio in S k is p k : 1 − p k . When p k is small we will need a large calibration set C

Original: ={ ( y, λ ) } ; and then use  S for HB training, yielding one binning scheme shareable to all two-class calibration tasks, i.e., shared class-wise (sCW) binning instead of CW binning respectively trained on S k . 
Revised: = ( y , x ) } to collect enough class- 1 samples in S k for setting the bin representatives. To address this, we propose to merge { S k } across similar classes and then use the merged set S for HB training, yielding one binning scheme shareable to multiple per-class calibration tasks, i.e., shared class-wise (sCW) binning instead of CW binning respectively trained on S k . In Sec. 4, we respectively experiment using a single binning schemes for all classes in the balanced multi-class setting, and sharing one binning among the classes with similar class priors in the imbalanced multi-class setting.

Original: Both S k and S serve as empirical approximations to the inaccessible ground truth distribution p ( y k , λ k ) for bin optimization.
Revised: Note, both S k and S serve as empirical approximations to the inaccessible ground truth distribution p ( y k , λ k ) for bin optimization.

Original: The former suffers from high variances, arising from insufﬁcient samples (Fig.
Revised: The former suffers from high variances, arising from insufﬁcient samples (Fig.

Original: A1-a), while the latter is biased due to having samples drawn from the other classes (Fig.
Revised: A1-a), while the latter is biased due to having samples drawn from the other classes (Fig.

Original: A1-b).
Revised: A1-b).

Original: As the calibration set size is usually small, the variance is expected to outweigh the approximation error over the bias (see an empirical analysis in Sec. A2).
Revised: As the calibration set size is usually small, the variance is expected to outweigh the approximation error over the bias (see an empirical analysis in Sec. A2).

Original: 3. B IN O PTIMIZATION VIA M UTUAL I NFORMATION (MI) M AXIMIZATION
Revised: 3. B IN O PTIMIZATION VIA M UTUAL I NFORMATION (MI) M AXIMIZATION

Original: Binning can be viewed as a quantizer Q that maps the real-valued logit λ ∈ R to the bin interval m ∈ { 1 , . . .
Revised: Binning can be viewed as a quantizer Q that maps the real-valued logit λ ∈ R to the bin interval m ∈ { 1 , . . .

Original: , M } if λ ∈ I m =
Revised: , M } if λ ∈ I m =

Original: [ g m − 1 , g m ) , where M is the total number of bin intervals, and the bin edges g m are sorted ( g m − 1 < g m , and g 

Revised: [ g m − 1 , g m ) , where M is the total number of bin intervals, and the bin edges g m are sorted ( g m − 1 < g m , and g

Original: =
Revised: =

Original: −∞ , g M = ∞ ).
Revised: −∞ , g M = ∞ ).

Original: Any logit binned to I m will be reproduced to the same bin representative r m .
Revised: Any logit binned to I m will be reproduced to the same bin representative r m .

Original: In the context of calibration, the bin representative r m assigned to the logit λ
Revised: In the context of calibration, the bin representative r m assigned to the logit λ

Original: k is used as the calibrated prediction probability of the class- k .
Revised: k is used as the calibrated prediction probability of the class- k .

Original: As multiple classes can be assigned with the same bin representative, we will then encounter ties when making top- k predictions based on calibrated probabilities.
Revised: As multiple classes can be assigned with the same bin representative, we will then encounter ties when making top- k predictions based on calibrated probabilities.

Original: Therefore, binning as lossy quantization generally does not preserve the raw logit-based ranking performance, being subject to potential accuracy loss.
Revised: Therefore, binning as lossy quantization generally does not preserve the raw logit-based ranking performance, being subject to potential accuracy loss.

Original: Unfortunately, increasing M to reduce the quantization error is not a good solution here.
Revised: Unfortunately, increasing M to reduce the quantization error is not a good solution here.

Original: For a given calibration set, the number of samples per bin generally reduces as M increases, and a reliable frequency estimation for setting the bin representatives { r m } demands sufﬁcient samples per bin.
Revised: For a given calibration set, the number of samples per bin generally reduces as M increases, and a reliable frequency estimation for setting the bin representatives { r m } demands sufﬁcient samples per bin.

Original: Considering that the top- k accuracy reﬂects how well the ground truth label can be recovered from the logits, we propose bin optimization via maximizing the MI between the quantized logits Q ( λ ) and the label y
Revised: Considering that the top- k accuracy reﬂects how well the ground truth label can be recovered from the logits, we propose bin optimization via maximizing the MI between the quantized logits Q ( λ ) and the label y

Original: [Equation]
Revised: [Equation]

Original:  where the index m is viewed as a discrete random variable with P ( m | y )
Revised: [Figure] where the index m is viewed as a discrete random variable with P ( m | y )

Original: = (cid:82) g [Equation]
Revised: = (cid:82) g m g m − 1 p ( λ | y )

Original: ( λ | y )d λ and
Revised: d λ and

Original: P ( m ) =
Revised: P ( m ) =

Original: (cid:82) g m g m − 1 p ( λ )d λ , and the equality ( a ) is based the relation of MI to the entropy H ( m ) and conditional entropy H ( m | y ) of m .
Revised: (cid:82) g m g m − 1 p ( λ )d λ , and the equality ( a ) is based the relation of MI to the entropy H ( m ) and conditional entropy H ( m | y ) of m .

Original: Such a formulation offers a quantizer Q ∗ optimal at preserving the label information for a given budget on the number of bins.
Revised: Such a formulation offers a quantizer Q ∗ optimal at preserving the label information for a given budget on the number of bins.

Original: Unlike designing distortion-based quantizers, the reproducer values of raw logits, i.e., the bin representatives { r m } , are not a part of the optimization space, as it is sufﬁcient to know the mapped bin index m of each logit.
Revised: Unlike designing distortion-based quantizers, the reproducer values of raw logits, i.e., the bin representatives { r m } , are not a part of the optimization space, as it is sufﬁcient to know the mapped bin index m of each logit.

Original: Once the bin edges { g ∗ m } are obtained, the bin representative r m to achieve zero calibration error shall equal
Revised: Once the bin edges { g ∗ m } are obtained, the bin representative r m to achieve zero calibration error shall equal

Original: P ( y = 1 | m ) , which can be empirically estimated from the samples within the bin interval I m .
Revised: P ( y = 1 | m ) , which can be empirically estimated from the samples within the bin interval I m .

Original: It is interesting to analyze the objective function after the equality ( a ) in (2).
Revised: It is interesting to analyze the objective function after the equality ( a ) in (2).

Original: The ﬁrst term H ( m ) is maximized if P ( m ) is uniform, which is attained by Eq.
Revised: The ﬁrst term H ( m ) is maximized if P ( m ) is uniform, which is attained by Eq.

Original: mass binning.
Revised: mass binning.

Original: A uniform sample distribution over the bins is a sample-efﬁcient strategy to optimize the bin representatives for the sake of calibration.
Revised: A uniform sample distribution over the bins is a sample-efﬁcient strategy to optimize the bin representatives for the sake of calibration.

Original: However, it does not consider any label information, and thus can suffer from severe accuracy loss.
Revised: However, it does not consider any label information, and thus can suffer from severe accuracy loss.

Original: Through MI maximization, we can view I-Max as revising Eq.
Revised: Through MI maximization, we can view I-Max as revising Eq.

Original: mass by incorporating the label information into the optimization objective, i.e., having the second term H ( m | y ) .
Revised: mass by incorporating the label information into the optimization objective, i.e., having the second term H ( m | y ) .

Original: As a result, I-Max not only enjoys a well balanced sample distribution for calibration, but also maximally preserved label information for accuracy.[Figure] Note, Eq.
Revised: As a result, I-Max not only enjoys a well balanced sample distribution for calibration, but also maximally preserved label information for accuracy.

Original: size binning uniformly partitions the interval [0 , 1] in the probability domain.

Original: The observed dense and symmetric bin location around zero is the outcome of probability-to-logit translation.

Original: In the example of Fig. 2, the bin edges of I-Max binning are densely located in an area where the uncertainty of y given the logit is high.
Revised: In the example of Fig. 2, the bin edges of I-Max binning are densely located in an area where the uncertainty of y given the logit is high.

Original: This uncertainty results from small gaps between the top class predictions.
Revised: This uncertainty results from small gaps between the top class predictions.

Original: With small bin widths, such nearby prediction logits are more likely located to different bins, and thus distinguishable after binning.
Revised: With small bin widths, such nearby prediction logits are more likely located to different bins, and thus distinguishable after binning.

Original: On the other hand, Eq.
Revised: On the other hand, Eq.

Original: mass binning has a single bin stretching across this high-uncertainty area due to  the class imbalance.
Revised: mass binning has a single bin stretching across this high-uncertainty area due to an imbalanced ratio between the p ( λ | y = 1) and p ( λ | y = 0) samples.

Original: Eq.
Revised: Eq.

Original: size binning follows a pattern closer to I-Max binning.
Revised: size binning follows a pattern closer to I-Max binning.

Original: However, its very narrow bin widths around zero may introduce large empirical frequency estimation errors when setting the bin representatives.
Revised: However, its very narrow bin widths around zero may introduce large empirical frequency estimation errors when setting the bin representatives.

Original: For solving the problem (2), we formulate an equivalent problem.
Revised: For solving the problem (2), we formulate an equivalent problem.

Original: Theorem 1.
Revised: Theorem 1.

Original: The MI maximization problem given in (2) is equivalent to
Revised: The MI maximization problem given in (2) is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where the loss L ( { g m , φ m } ) is deﬁned as
Revised: where the loss L ( { g m , φ m } ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: and { φ m } as a set of real-valued auxiliary variables are introduced here to ease the optimization.
Revised: and { φ m } as a set of real-valued auxiliary variables are introduced here to ease the optimization.

Original: Proof .
Revised: Proof .

Original: See A3 for the proof.
Revised: See A3 for the proof.

Original: Next [Equation] we compute the derivatives of the loss L with respect to { g m , φ m } .
Revised: Next, we compute the derivatives of the loss L with respect to { g m , φ m } .

Original: When the conditional distribution P ( y | λ ) takes the sigmoid model, 
Revised: When the conditional distribution P ( y | λ ) takes the sigmoid model, i.e., P ( y | λ )

Original: i [Equation] e., P ( y | λ ) ≈
Revised: ≈

Original: σ [(2 y − 1) λ ] , the stationary points of
Revised: σ [(2 y − 1) λ ] , the stationary points of

Original: L , zeroing the gradients over { g m , φ m } , have a closed-form expression
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: where the approximation for φ m arises from using the logits in the training set S as an empirical approximation to p ( λ ) and S m ∆ =
Revised: where the approximation for φ m arises from using the logits in the training set S as an empirical approximation to p ( λ ) and S m ∆ =

Original: S ∩ [ g m , g m +1 ) .
Revised: S ∩ [ g m , g m +1 ) .

Original: So, we can solve the problem by iteratively and alternately updating { g m } and { φ m } based on (A12).
Revised: So, we can solve the problem by iteratively and alternately updating { g m } and { φ m } based on (5) (see Algo.

Original:  The convergence and initialization of such an iterative method as well as the sigmoid -model assumption are discussed along with the proof of 
Revised: 1 in the appendix for pseudocode). The convergence and initialization of such an iterative method as well as the sigmoid -model assumption are discussed along with the proof of Theorem 1 in Sec.

Original: Theorem 1 in A3.
Revised: A3.

Original: As the iterative method operates under an approximation of the inaccessible ground truth distribution p ( y, λ ) ,here we synthesize an example, see Fig.
Revised: As the iterative method operates under an approximation of the inaccessible ground truth distribution p ( y, λ ) , we synthesize an example, see Fig.

Original: 3, to assess its effectiveness.
Revised: 3, to assess its effectiveness.

Original: In Fig. 
Revised: As quantization can only reduce the MI, we evaluate I ( y ; λ ) , serving as the upper bound in Fig. 3-a) for I ( y ; Q ( λ )) . Among the three realizations of Q , I-Max achieves higher MI than Eq. size and Eq.

Original: 3-a, we show the monotonic increase in MI over the iterative steps.
Revised: mass, and more importantly, it approaches the upper bound over the iterations. Next, we assess the performance within the framework of information bottleneck (IB) (Tishby et al., 1999), see Fig. 3-b).

Original: Furthermore, Fig.
Revised: In the context of our problem, IB tackles min 1 /β × I ( λ ; Q ( λ )) − I ( y ; Q ( λ )) with the weight factor β > 0 to balance between 1) maximizing the information rate I ( y ; Q ( λ )) , and 2) minimizing the compression rate I ( λ ; Q ( λ )) . By varying β , IB gives the maximal achievable information rate for the given compression rate. Fig.

Original: 3-b shows that I-Max approaches the theoretical limits and provides an information-theoretic perspective on the sub-optimal performance of the alternative binning schemes.
Revised: 3-b) shows that I-Max approaches the theoretical limits and provides an information-theoretic perspective on the sub-optimal performance of the alternative binning schemes.

Original: [Figure]
Revised: Sec. A3.2 has a more detailed discussion on the connection of IB and our problem formulation.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Datasets and Models We evaluate post-hoc calibration methods on three benchmark datasets, i.e.,
Revised: Datasets and Models We evaluate post-hoc calibration methods on four benchmark datasets, i.e.,

Original: ImageNet (Deng et al.,  2009) and both CIFARs (Krizhevsky, 2009), and across three modern DNNs per dataset.
Revised: ImageNet (Deng et al., 2009), CIFAR 10/100 (Krizhevsky, 2009) and SVHN (Netzer et al., 2011), and across various modern DNNs architectures.

Original: All trained models are publicly available and details are reported in Sec.
Revised: More details are reported in Sec.

Original: A7.1.
Revised: A8.1.

Original: Training and Evaluation Details We perform class-balanced random splits of the data test set: the calibration and evaluation set sizes are both 25 k for ImageNet, and 5 k for  the CIFAR datasets.
Revised: Training and Evaluation Details We perform class-balanced random splits of the data test set, unless stated otherwise: the calibration and evaluation set sizes are both 25 k for ImageNet, and 5 k for CIFAR10/100. Different to ImageNet and CIFAR10/100, the test set of SVHN is class imbalanced.

Original:  All reported numbers are the means across 5 random splits; stds can be found in the appendix.
Revised: We evenly split it into the calibration and evaluation set of size 13 k. All reported numbers are the means across 5 random splits; stds can be found in the appendix.

Original: Note that some calibration methodsonly use a subset of the available calibration samples for training, showing their sample efﬁciency. Further calibrator training details are provided in Sec. A7.1.
Revised: Note that some calibration methods

Original: We empirically evaluate MI, Accuracy (top- 1 and 5 ACCs), ECE (class-wise and top- 1 ), Brier and
Revised: We empirically evaluate MI, Accuracy (top- 1 and 5 ACCs), ECE (class-wise and top- 1 ), Brier and

Original: NLL; the latter are shown in the appendix.
Revised: NLL; the latter are shown in the appendix.

Original: The ECEs of the scaling methods are evaluated by using
Revised: Analogous to (Nixon et al., 2019), we use thresholding when evaluating the class-wise ECE ( CW ECE thr ). Without thresholding, the empirical class-wise ECE score may be misleading. When a class- k has a small class prior (e.g. 0 .

Revised: 01 or 0 . 001 ), the empirical class-wise ECE score will be dominated by prediction samples where the class- k is not the ground truth.

Revised: For these cases, a properly trained classiﬁer will often not rank this class-

Revised: k among the top classes and instead yield only small calibration errors.

Revised: While it is good to have many cases with small calibration errors, they should not wash out the calibration errors of the rest of the cases (prone to poor calibration) through performance averaging.

Revised: These include (1) class-

Revised: k is the ground truth class and not correctly ranked and (2) the classiﬁer mis-classiﬁes some class- j as class- k .

Revised: The thresholding remedies the washing out by focusing more on crucial cases (i.e. only averaging across cases where the prediction of the class- k is above a threshold).

Revised: In the following experiments, our primary choice of threshold is to set it according to the class prior for the reason that the class- k is unlikely to be the ground truth if its a-posteriori probability becomes lower than its prior after observing the sample.

Original:  HDE with 100 equal size evaluation bins, as in (Wenger et al., 2020).
Revised: While empirical ECE estimation of binning schemes is simple, we resort to HDE with 100 equal size evaluation bins (Wenger et al., 2020) for scaling methods.

Original: Sec.
Revised: Sec.

Original: A6 also reports the results attained by HDE with additional binning schemes and KDE.
Revised: A6 also reports the results attained by HDE with additional binning schemes and KDE.

Original: For HDE-based ones, we notice that with 100 evaluation bins, the ECE estimate is insensitive to the choice of binning scheme.
Revised: For HDE-based ones, we notice that withevaluation bins, the ECE estimate is insensitive to the choice of binning scheme.

Original: 4. EQ . S IZE , E Q . M ASS VS . I-M AX B INNING
Revised: 4. E

Original: When comparing different calibration methods, it is important to jointly consider ECE and ACC. A naive classiﬁer that always predicts all classes with conﬁdence 1 /K minimizes the ECE to zero, but its ACC is only at chance level. Additionally, only having a small CW ECE may be an artifact.

Original: For instance, the CW ECE of ImageNet will be dominated by cases where a properly trained classiﬁer ranks the class- k as very unlikely, as the prior ensures the correctness in 99 .

Original: 9% cases.

Original: However, in practice, the relevance of the prediction probability being calibrated or not only matters when the event is likely to happen.

Original: Given that, we introduce a threshold into the CW ECE evaluation as in (1) to only consider cases when the predicted probability of class- k exceeds its prior 1 /K .

Original: In Tab. 1, we compare three binning schemes:
Revised: In Tab. 1, we compare three binning schemes:

Original: Eq. size, Eq.
Revised: Eq. size, Eq.

Original: mass and I-Max binning.
Revised: mass and I-Max binning.

Original: The accuracy performances of thethree binning schemes are proportional to their MI (in nats); Eq.
Revised: The accuracy performances of the binning schemes are proportional to their MI; Eq.

Original: mass binning is highly sub-optimal at label information preservation, and thus shows a severe accuracy drop.
Revised: mass binning is highly suboptimal at label information preservation, and thus shows a severe accuracy drop.

Original: Eq.
Revised: Eq.

Original: size binning accuracy is more similar to that of I-Max binning, but still lower, in particular at Acc top5 .
Revised: size binning accuracy is more similar to that of I-Max binning, but still lower, in particular at Acc top5 .

Original: Also note that I-Max can approach the MI theoretical limit of I ( y ; λ ) = 0 . 0068 .
Revised: Also note that I-Max approaches the MI theoretical limit of I ( y ; λ ) = 0 . 0068 .

Original: Advantages of I-Max become even more prominent when comparing the NLLs of the binning schemes.
Revised: Advantages of I-Max become even more prominent when comparing the NLLs of the binning schemes.

Original: For allthree ECE evalution metrics, I-Max binning improves on the baseline calibration performance, and outperforms 
Revised: For all ECE evalution metrics, I-Max binning improves on the baseline calibration performance, and outperforms Eq.

Original: Eq. size binning.
Revised: size binning.

Original: Eq.
Revised: Eq.

Original: mass binning is out of this comparison scope due to its poor accuracy deeming[Table] the method impractical.
Revised: mass binning is out of this comparison scope due to its poor accuracy deeming the method impractical.

Original: Overall [Table] I-Max successfully mitigates the negative impact of quantization on ACCs while still providing an improved and veriﬁable ECE performance. 
Revised: Overall, I-Max successfully mitigates the negative impact of quantization on ACCs while still providing an improved and veriﬁable ECE performance. Additionally, one-for-all sCW

Original: Additionally, I-Max achieves an even better calibrationwhen using sCW with  1 k calibration samples, instead of the standard CW binning with 25 k calibration samples, highlighting the effectiveness of  sCW binning.
Revised: I-Max achieves an even better calibration with only 1 k calibration samples, instead of the standard CW binning with 25 k calibration samples, highlighting the effectiveness of the sCW strategy. Furthermore, it is interesting to note that CW ECE of the Baseline classiﬁer is very small, i.e., 0 . 000442 , thus it may appear as the Baseline classiﬁer is well calibrated. However, top1 ECE is much larger, i.e., 0 . 0357 .

Revised: Such inconsistent observations disappear after thresholding the class-wise ECE with the class prior.

Revised: This example conﬁrms the necessity of thresholding the class-wise ECE.

Original: In Sec.
Revised: In Sec.

Original: A5 we perform additional ablations on the number of bins and calibration samples.
Revised: A5 we perform additional ablations on the number of bins and calibration samples.

Original: Accordingly, a post-hoc analysis investigates how the quantization error of the binning schemes change the ranking order.
Revised: Accordingly, a post-hoc analysis investigates how the quantization error of the binning schemes change the ranking order.

Original: Observations are consistent with the intuition behind the problem formulation (see
Revised: Observations are consistent with the intuition behind the problem formulation (see

Original: Sec. 3.3) and empirical results from Tab. 1 that MI maximization is a proper criterion for multi-class calibration and it maximally mitigates the potential accuracy loss.
Revised: Sec. 3.3) and empirical results from Tab. 1 that MI maximization is a proper criterion for multi-class calibration and it maximally mitigates the potential accuracy loss.

Original: 4. S CALING VS . I-M AX B INNING
Revised: 4. S CALING VS . I-M AX B INNING

Original: In Tab. 2, we compare I-Max binning to benchmark scaling methods.
Revised: In Tab. 2, we compare I-Max binning to benchmark scaling methods.

Original: Namely, matrix scaling w. Lregularization (Kull et al., 2019) has a large model capacity compared to other parametric scaling methods, while TS (Guo et al., 2017) only uses a single parameter and MnM (Zhang et al., 2020) uses three temperatures as an ensemble of TS (ETS).
Revised: Namely, matrix scaling with L 2 regularization (Kull et al., 2019) has a large model capacity compared to other parametric scaling methods, while TS (Guo et al., 2017) only uses a single parameter and MnM (Zhang et al., 2020) uses three temperatures as an ensemble of TS (ETS).

Original: As a non-parametric method, GP (Wenger et al., 2020) yields state of the art calibration performance.
Revised: As a non-parametric method, GP (Wenger et al., 2020) yields state of the art calibration performance.

Original: Additional 8 scaling methods can be found in Sec.
Revised: Additional 8 scaling methods can be found in Sec.

Original: A9.
Revised: A10.

Original: Beneﬁting from its model capacity, matrix scaling achieves the best accuracy.
Revised: Beneﬁting from its model capacity, matrix scaling achieves the best accuracy.

Original: I-Max binning achieves the best calibration on CIFAR-100; on ImageNet, it has the best CW ECE, and is similar to GP on top1 ECE.
Revised: I-Max binning achieves the best calibration on CIFAR-100; on ImageNet, it has the best CW ECE, and is similar to GP on top1 ECE.

Original: For a broader scope of comparison, we refer to Sec. 
Revised: For a broader scope of comparison, we refer to Sec. A9. [Table] [Table]

Revised: except for Mtx.

Revised: Scal.

Revised: and ETS-MnM, which requires the complete calibration set,

Original: A8.
Revised: i [Table] e., 25 k/ 5 k for ImageNet/CIFAR100. Additional 6 scaling methods can be found in A10.

Original:  To showcase the complementary nature of scaling and binning, we investigate combining binning with
Revised: [Table] To showcase the complementary nature of scaling and binning [Table] we investigate combining binning with

Original: GP (a top performing non-parametric scaling method, though with the drawback of high complexity) and TS (a commonly used scaling method).
Revised: GP (a top performing non-parametric scaling method, though with the drawback of high complexity) and TS (a commonly used scaling method).

Original: Here, we propose to bin the raw logits and use the
Revised: Here [Table] we propose to bin the raw logits and use the

Original: GP/TS scaled logits of the samples per bin for setting the bin representatives, replacing the empirical frequency estimates.
Revised: GP/TS scaled logits of the samples per bin for setting the bin representatives, replacing the empirical frequency estimates.

Original: As GP is then only needed at the calibration learning phase, complexity is no longer an issue.
Revised: As GP is then only needed at the calibration learning phase, complexity is no longer an issue.

Original: Being mutually beneﬁcial, GP helps improving ACCs and ECEs of binning, i.e., marginal ACC drop 0 .
Revised: Being mutually beneﬁcial, GP helps improving ACCs and ECEs of binning, i.e., marginal ACC drop 0 .

Original: on Acc top1 for ImageNet (CIFAR100) and 0 .
Revised: on Acc top1 for ImageNet (CIFAR100) and 0 .

Original: 24 % on Acc top for ImageNet; and large ECE reduction 38 .
Revised: 24 % on Acc top5 for ImageNet; and large ECE reduction 38 .

Original: 78 %) in CW ECE 1 K and 66 .
Revised: 78 %) in CW ECE cls − prior and 66 .

Original: 11 % ( 76 . 07 %) in top1 ECE of the baseline for ImageNet (CIFAR100).
Revised: 11 % ( 76 . 07 %) in top1 ECE of the baseline for ImageNet (CIFAR100).

Original: 4. S HARED C LASS W ISE H ELPS S CALING M ETHODS
Revised: 4. S HARED C LASS W ISE H ELPS S CALING M ETHODS

Original: Though without quantization loss, some scaling methods, i.e., Beta (Kull et al., 2017), Isotonic regression (Zadrozny & Elkan, 2002), and Platt scaling (Platt, 1999), even suffer from more severe accuracy degradation than I-Max binning.
Revised: Though without quantization loss, some scaling methods, i.e., Beta (Kull et al., 2017), Isotonic regression (Zadrozny & Elkan, 2002), and Platt scaling (Platt, 1999), even suffer from more severe accuracy degradation than I-Max binning.

Original: As they also use the one-vs-rest strategy for multi-class calibration, we ﬁnd that the proposed shared CW binning strategy is beneﬁcial for reducing their accuracy loss and improving their ECE performance, with only 1 k calibration samples, see Tab. 3. 
Revised: As they also use the one-vs-rest strategy for multi-class calibration, we ﬁnd that the proposed shared CW binning strategy is beneﬁcial for reducing their accuracy loss and improving their ECE performance, with only 1 k calibration samples, see Tab. 3. 4. I MBALANCED M ULTI - CLASS S ETTING Lastly, we turn our experiments to an imbalanced multi-class setting.

Original: [Table]
Revised: The adopted SVHN dataset has non-uniform class priors, ranging from 6% (e.g. digit 8 ) to 19% (e.g. digit 0 ). We reproduce Tab.for SVHN, yielding Tab. 4. In order to better control the bias caused by the calibration set merging in the imbalanced multi-class setting, the former one-for-all sCW strategy in the balanced multi-class setting changes to sharing I-Max among classes with similar class priors. Despite the class imbalance,

Original: For each scheme and metric, we display the best variant of the scaling methods in bold.
Revised: I-Max and its variants perform best compared to the other calibrators, being similar to Tab. 2. This shows that I-Max and the sCW strategy both can generalize to imbalanced multi-class setting. [Table] [Table]

Revised: [Table]

Revised: In Tab.

Original: [Table]
Revised: [Table]

Revised: we additionally evaluate the class-wise ECE at multiple threholds.

Original: {+We ablate various thresholds settings, namely, 1) 
+}
Revised: {+We ablate various thresholds settings, namely, 1) 
+}

Revised: (no thresholding); 2) the class prior; 3) 1 /K

Revised: (any class with prediction probability below 1 /K will not be the top-1); and 4) a relatively large number 0 .

Revised: 5 (the case when the conﬁdence on class- k outweighs NOT class- k ).

Revised: We observe that I-Max and its variants are consistently top performing across the different thresholds.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed I-Max binning for multi-class calibration, which maximally preserves the labelinformation under quantization, reducing potential accuracy losses.
Revised: We proposed I-Max binning for multi-class calibration, which maximally preserves the labelinformation under quantization, reducing potential accuracy losses.

Original: Using the shared class-wise (sCW) strategy, we also addressed the sample-inefﬁciency issue of binning and scaling methods that rely on one-vs-rest (OvR) for multi-class calibration.
Revised: Using the shared class-wise (sCW) strategy, we also addressed the sample-inefﬁciency issue of binning and scaling methods that rely on one-vs-rest (OvR) for multi-class calibration.

Original: Our experiments showed that I-Max yields consistent class-wise and top- 1 calibration improvements over multiple datasets and model architectures, outperforming HB and state-of-the-art scaling methods.
Revised: Our experiments showed that I-Max yields consistent class-wise and top- 1 calibration improvements over multiple datasets and model architectures, outperforming HB and state-of-the-art scaling methods.

Original: Combining I-Max with scaling methods offers further calibration performance gains, and more importantly, ECE estimates that can converge to the ground truth in the large sample limit.
Revised: Combining I-Max with scaling methods offers further calibration performance gains, and more importantly, ECE estimates that can converge to the ground truth in the large sample limit.

Original: Future work will investigate extensions of I-Max that jointly calibrate multiple classes, and thereby directly model class correlations.
Revised: Future work will investigate extensions of I-Max that jointly calibrate multiple classes, and thereby directly model class correlations.

Original: Interestingly, even on datasets such as ImageNet which contain several closely related classes, there is no clear evidence that methods that do model class correlations, e.g. Mtrx.
Revised: Interestingly, even on datasets such as ImageNet which contain several closely related classes, there is no clear evidence that methods that do model class correlations, e.g. Mtrx.

Original: Scal.
Revised: Scal.

Original: capture uncertainties better.
Revised: capture uncertainties better.

Original: In fact, I-Max empirically outperforms such methods, although all classes are calibrated independently under the OvR assumption.
Revised: In fact, I-Max empirically outperforms such methods, although all classes are calibrated independently under the OvR assumption.

Original: Non-OvR based methods may fail due to various reasons such as under-parameterized models (e.g. TS), limited data (e.g. Mtrx.
Revised: Non-OvR based methods may fail due to various reasons such as under-parameterized models (e.g. TS), limited data (e.g. Mtrx.

Original: Scal.) or complexity constraints (e.g. GP).
Revised: Scal.) or complexity constraints (e.g. GP).

Original: Joint class calibration therefore strongly relies on new sample efﬁcient evaluation measures that estimate how accurately class correlations are modeled, and which can be included as additional optimization criteria.
Revised: Joint class calibration therefore strongly relies on new sample efﬁcient evaluation measures that estimate how accurately class correlations are modeled, and which can be included as additional optimization criteria.

Original: A N O E XTRA N ORMALIZATION AFTER K C LASS - WISE C ALIBRATIONS
Revised: A N O E XTRA N ORMALIZATION AFTER K C LASS - WISE C ALIBRATIONS

Original: There is a group of calibration schemes that rely on one-vs-rest conversion to turn multi-class calibration into K class-wise calibrations, e.g., histogram binning (HB), Platt scaling and Isotonic regression.
Revised: There is a group of calibration schemes that rely on one-vs-rest conversion to turn multi-class calibration into K class-wise calibrations, e.g., histogram binning (HB), Platt scaling and Isotonic regression.

Original: After per-class calibration, the calibrated prediction probabilities of all classes no longer fulﬁll the constraint, i.e., (cid:80)
Revised: After per-class calibration, the calibrated prediction probabilities of all classes no longer fulﬁll the constraint, i.e., (cid:80)

Original: Kk =
Revised: Kk =

Original: 1 q k (cid:54) = 1 .
Revised: 1 q k (cid:54) = 1 .

Original: An extra normalization step was taken in Guo et al.
Revised: An extra normalization step was taken in Guo et al.

Original: (2017) to regain the normalization constraint.
Revised: (2017) to regain the normalization constraint.

Original: Here, we note that this extra normalization is unnecessary and partially undoes the per-class calibration effect.
Revised: Here, we note that this extra normalization is unnecessary and partially undoes the per-class calibration effect.

Original: For HB, normalization will make its outputs continuous like any other scaling methods, thereby suffering from the same issue at ECE evaluation.
Revised: For HB, normalization will make its outputs continuous like any other scaling methods, thereby suffering from the same issue at ECE evaluation.

Original: One-vs-rest strategy essentially marginalizes the multi-class predictive distribution over each class.
Revised: One-vs-rest strategy essentially marginalizes the multi-class predictive distribution over each class.

Original: After such marginalization, each class and its prediction probability shall be treated independently, thus no longer being constrained by the multi-class normalization constraint.
Revised: After such marginalization, each class and its prediction probability shall be treated independently, thus no longer being constrained by the multi-class normalization constraint.

Original: This is analogous to train a CIFAR or ImageNet classiﬁer with sigmoid rather than softmax cross entropy loss, e.g., Ryou et al.
Revised: This is analogous to train a CIFAR or ImageNet classiﬁer with sigmoid rather than softmax cross entropy loss, e.g., Ryou et al.

Original: At training and test time, each class prediction probability is individually taken from the respective sigmoid -response without normalization.
Revised: At training and test time, each class prediction probability is individually taken from the respective sigmoid -response without normalization.

Original: The class with the largest response is then top-ranked, and normalization itself has no inﬂuence on the ranking performance.
Revised: The class with the largest response is then top-ranked, and normalization itself has no inﬂuence on the ranking performance.

Original: A S VS . S k AS E MPIRICAL A PPROXIMATIONS TO p ( λ k , y k ) FOR B IN O PTIMIZATION
Revised: A S VS . S k AS E MPIRICAL A PPROXIMATIONS TO p ( λ k , y k ) FOR B IN O PTIMIZATION

Original: In Sec.
Revised: In Sec.

Original: 3.2 of the main paper, we discussed the  class imbalance issue, arising from one-vs-rest conversion.
Revised: 3.2 of the main paper, we discussed the sample inefﬁciency issue when there are classes with small class priors.

Original: Fig.
Revised: Fig.

Original: A1 shows  the effect of the shared CW (sCW) strategy discussed in Sec. 3.2.
Revised: A1-a) shows an example for ImageNet with 1 k classes. The class prior for the class- 394 is about 0 .

Original: Speciﬁcally, it can be seen that the sCW strategy greatly improves the sample efﬁciency compared to using the class-wise strategy.
Revised: Among the 10 k calibration samples, we can only collect 10 samples with ground truth is the class- 394 .

Original: [Figure]
Revised: Estimating the bin representatives from these 10 samples is highly unreliable, resulting into poor calibration performance.

Original: To tackle it, we proposed to merge theclass-wise (CW) bin training sets { S k }  and use the mergedone S to train a single binning scheme for all classes, i.e., shared CW (sCW) instead of CW binning.
Revised: To tackle this, we proposed to merge the training sets { S k } across a selected set of classes (e.g., with similar class priors, belonging to the same class category or all classes) and use the merged S to train a single binning scheme for calibrating these classes, i.e., shared class-wise (sCW) instead of CW binning.

Original: Tab.
Revised: Fig.

Revised: A1-b) shows that after merging over the 1 k ImageNet classes, the set S has sufﬁcient numbers from both the positive y = 1 and negative y = 0 class under the one-vs-rest conversion.

Original: A showed that our proposal  is also beneﬁcial to scaling methods that use one-vs-rest for multi-class calibration.
Revised: Tab. 1 showed the beneﬁts of sCW over CW binnings. Tab. 3 showed that our proposal sCW is also beneﬁcial to scaling methods which use one-vs-rest for multi-class calibration.

Original: As pointed out in Sec. 3.2, both S and S k are empirical approximations to the inaccessible ground truth p ( λ k , y k ) for bin optimization.
Revised: As pointed out in Sec. 3.2, both S and S k are empirical approximations to the inaccessible ground truth p ( λ k , y k ) for bin optimization.

Original: In Fig.
Revised: In Fig.

Original: A2, we empirically analyze their approximation errors.
Revised: A2, we empirically analyze their approximation errors.

Original: From the CIFAR10 test set, we take 5 k samples to approximate per-class logit distribution p ( λ k | y k = 1) by means of histogram density estimation 2 , and then use it as the baseline for comparison, i.e., BS k in Fig.
Revised: From the CIFAR10 test set, we take 5 k samples to approximate per-class logit distribution p ( λ k | y k = 1) by means of histogram density estimation 1 , and then use it as the baseline for comparison, i.e., BS k A [Figure] [Figure]

Revised: in Fig.

Original: A2.
Revised: A2.

Original: The rest of the 5 k samples in the CIFAR10 test set are reserved for constructing S k and S .
Revised: The rest of the 5 k samples in the CIFAR10 test set are reserved for constructing S k and S .

Original: For each class, we respectively evaluate the square root of the Jensen-Shannon divergence (JSD) from the baseline BS k to the empirical distribution of S or S k attained at different numbers of samples.
Revised: For each class, we respectively evaluate the square root of the Jensen-Shannon divergence (JSD) from the baseline BS k to the empirical distribution of S or S k attained at different numbers of samples.

Original: Despite suffering from the bias induced by set merging, S achieves noticeably smaller JSDs than S k in nearly all cases, and shows to be more sample efﬁcient.
Revised: In general, Fig. A2 conﬁrms that variance (due to not enough samples) outweighs bias (due to training set merging). Nevertheless, sCW does not always have smaller JSDs than CW, for instance, the class 7 with the samples larger than 2k (the blue bar "sCW" is larger than the orange bar "CW").

Original: Only after using all 5 k samples,
Revised: So, for the class- 7 , the bias of merging logits starts outweighing the variance when the number of samples is more than 2k. Unfortunately, we don’t have more samples to further evaluate JSDs, i.e., making the variance sufﬁciently small to reveal the bias impact. Another reason that we don’t observe large

Original: S k has slightly smaller JSDs than S in the class- 1 , and 6 and 7 .
Revised: JSDs of sCW for CIFAR10 is that the logit distributions of the 10 classes are similar.

Revised: Therefore, the bias of sCW is small, making CIFAR10 a good use case of sCW.

Original: From CIFAR10 to CIFAR100 and ImageNet, the class imbalance issue becomes more severe.
Revised: From CIFAR10 to CIFAR and ImageNet, there are more classes with even smaller class priors.

Original: We expect  the sample inefﬁciency issue of S k becomes more critical, and therefore it is advisable to use S for bin optimization.
Revised: Therefore, we expect that the sample inefﬁciency issue of S k becomes more critical. It will be beneﬁcial to exploit sCW for bin optimization as well as for other methods based on the one-vs-rest conversion for multi-class calibration. A

Revised: A P ROOF OF T HEOREM 1 AND A LGORITHM D ETAILS OF I-M AX

Original: [Figure]
Revised: In this section, we proves Theorem 1 in Sec. 3.3, discuss the connection to the information bottleneck (IB) (Tishby et al., 1999), analyze the convergence behavior of the iterative method derived in Sec. 3.3 and modify the k-means++ algorithm (Arthur & Vassilvitskii, 2007) for initialization.

Revised: To assist the implementation of the iterative method, we further provide the pseudo code and perform complexity/memory cost analysis.

Original: A P ROOF OF T HEOREM
Revised: A3. P ROOF OF T HEOREM

Original: Theorem 1.
Revised: Theorem 1.

Original: The mutual information (MI) maximization problem given as follows:
Revised: The mutual information (MI) maximization problem given as follows:

Original: [Equation]
Revised: [Equation]

Original: is equivalent to
Revised: is equivalent to

Original: [Equation]
Revised: [Equation]

Original: where the loss L ( { g m , φ m } ) is deﬁned as
Revised: where the loss L ( { g m , φ m } ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with P σ ( y ; φ m ) ∆ =
Revised: with P σ ( y ; φ m ) ∆ =

Original: σ [(2 y − 1) φ m ] .
Revised: σ [(2 y − 1) φ m ] .

Original: [Equation]
Revised: [Equation]

Original: A4)

Original: As a set of real-valued auxiliary variables, { φ m } are introduced here to ease the optimization.A
Revised: As a set of real-valued auxiliary variables, { φ m } are introduced here to ease the optimization.

Original: Proof .
Revised: Proof .

Original: Before staring our proof, we note that the upper-case P indicates probability mass functions of discrete random variables, e.g., the label y ∈ { 0 , 1 } and the bin interval index m ∈ { 1 , . . .
Revised: Before staring our proof, we note that the upper-case P indicates probability mass functions of discrete random variables, e.g., the label y ∈ { 0 , 1 } and the bin interval index m ∈ { 1 , . . .

Original: , M } ; whereas the lower-case p is reserved for probability density functions of continuous random variables, e.g., the raw logit λ ∈ R .
Revised: , M } ; whereas the lower-case p is reserved for probability density functions of continuous random variables, e.g., the raw logit λ ∈ R .

Original: The key to prove the equivalence is to show the inequality
Revised: The key to prove the equivalence is to show the inequality

Original: I

Original: [Equation]
Revised: [Equation]

Original: and the equality is attainable by minimizing L over { φ m } .
Revised: and the equality is attainable by minimizing L over { φ m } .

Original: By the deﬁnition of MI, we ﬁrstly expand I ( y ; m = Q ( λ )) as 
Revised: By the deﬁnition of MI, we ﬁrstly expand I ( y ; m = Q ( λ )) as [Equation] y

Original: [Equation]
Revised: [Equation]

Original: where the conditional distribution P ( y | m ) is given as
Revised: where the conditional distribution P ( y | m ) is given as

Original: [Equation]
Revised: [Equation]

Original: From the above expression, we note that MI maximization effectively only accounts to the bin edges { g m } .
Revised: From the above expression, we note that MI maximization effectively only accounts to the bin edges { g m } .

Original: The bin representatives can be arbitrary as long as they can indicate the condition λ ∈
Revised: The bin representatives can be arbitrary as long as they can indicate the condition λ ∈

Original: [ g m , g m +1 ) .
Revised: [ g m , g m +1 ) .

Original: So, the bin interval index m is sufﬁcient to serve the role in conditioning the probability mass function of y , i.e., P ( y | m ) .
Revised: So, the bin interval index m is sufﬁcient to serve the role in conditioning the probability mass function of y , i.e., P ( y | m ) .

Original: After optimizing the bin edges, we have the freedom to set the bin representatives for the sake of post-hoc calibration.
Revised: After optimizing the bin edges, we have the freedom to set the bin representatives for the sake of post-hoc calibration.

Original:  Next, based on the MI expression, we compute its sum with L 
Revised: A Next, based on the MI expression, we compute its sum with L I ( y

Original: [Equation]
Revised: [Equation]

Original: The equality ( a ) is based on
Revised: The equality ( a ) is based on

Original: [Equation]
Revised: [Equation]

Original: g m + [Equation] From the equality ( a ) to ( b ) , it is simply because of identifying the term in [ · ] of the equality ( a ) as the Kullback-Leibler divergence (KLD) between two probability mass functions of y .
Revised: From the equality ( a ) to ( b ) , it is simply because of identifying the term in [ · ] of the equality ( a ) as the Kullback-Leibler divergence (KLD) between two probability mass functions of y .

Original: As the probability mass function P ( m ) and the KLD both are non-negative, we reach to the inequality at ( c ) , where the equality holds if P σ ( y ; φ m ) =
Revised: As the probability mass function P ( m ) and the KLD both are non-negative, we reach to the inequality at ( c ) , where the equality holds if P σ ( y ; φ m ) =

Original: P ( y | m ) .
Revised: P ( y | m ) .

Original: By further noting that L is convex over { φ m } and P σ ( y ; φ m ) =
Revised: By further noting that L is convex over { φ m } and P σ ( y ; φ m ) =

Original: P ( y | m ) nulls out its gradient over { φ m } , we then reach to
Revised: P ( y | m ) nulls out its gradient over { φ m } , we then reach to

Original: [Equation]
Revised: [Equation]

Original: The obtained equality then concludes our proof
Revised: The obtained equality then concludes our proof

Original: [Equation]
Revised: [Equation]

Original: A Lastly, we note that L ( { g m , φ m } ) can reduce to a NLL loss (as P ( y ) in the log probability ratio is omittable), which is a common loss for calibrators.
Revised: Lastly, we note that L ( { g m , φ m } ) can reduce to a NLL loss (as P ( y ) in the log probability ratio is omittable), which is a common loss for calibrators.

Original: However, only through this equivalence proof and the MI maximization formulation, can we clearly identify the great importance of bin edges in preserving label information.
Revised: However, only through this equivalence proof and the MI maximization formulation, can we clearly identify the great importance of bin edges in preserving label information.

Original: So even though { g m , φ m } are jointly optimized in the equivalent problem, only { g m } play the determinant role in maximizing the MI.
Revised: So even though { g m , φ m } are jointly optimized in the equivalent problem, only { g m } play the determinant role in maximizing the MI.

Original: A3. C ONVERGENCE OF THE I TERATIVE M ETHOD
Revised: A3. C ONNECTION TO I NFORMATION B OTTLENECK (IB)

Revised: IB (Tishby et al., 1999) is a generic information-theoretic framework for stochastic quantization design.

Revised: Viewing binning as quantization, IB aims to ﬁnd a balance between two conﬂicting goals: 1) maximizing the information rate, i.e., the mutual information between the label and the quantized logits

Revised: I ( y ; Q ( λ )) ; and 2) minimizing the compression rate, i.e., mutual information between the logits and the quantized logits I ( λ ; Q ( λ )) .

Revised: It uniﬁes them by minimizing

Revised: [Equation]

Revised: where m is the bin index assigned to λ and β is the weighting factor (with larger value focusing more on the information rate and smaller value on the compression rate).

Revised: The compression rate is the bottleneck for maximizing the information rate.

Revised: Note that IB optimizes the distribution p ( m | λ ) , which describes the probability of λ being assigned to the bin with the index m .

Revised: Since it is not a deterministic assignment, IB offers a stochastic rather than deterministic quantizer.

Revised: Our information maximization formulation is a special case of IB, i.e., β being inﬁnitely large, as we care predominantly about how

Revised: A well the label can be predicted from a compressed representation (quantized logits), in other words, making the compression rate as small as possible is not a request from the problem.

Revised: For us, the only bottleneck is the number of bins usable for quantization.

Revised: Furthermore, with β → ∞ , stochastic quantization degenerating to a deterministic one.

Revised: If using stochastic binning for calibration, it outputs a weighted sum of all bin representatives, thereby being continuous and not ECE veriﬁable.

Revised: Given that, we do not use it for calibration.

Revised: As the IB deﬁnes the best trade-off between the information rate and compression rate, we use it as the upper limit for assessing the optimality of I-Max in Fig.

Revised: 3-b).

Revised: By varying β , IB depicts the maximal achievable information rate for the given compression rate.

Revised: For binning schemes (Eq. size, Eq.

Revised: mass and I-Max), we vary the number of bins, and evaluate their achieved information and compression rates.

Revised: As we can clearly observe from Fig.

Revised: 3-b), I-Max can approach the upper limit deﬁned by IB.

Revised: Note that, the compression rate, though being measured in bits, is different to the number of bins used for the quantizer.

Revised: As quantization is lossy, the compression rate deﬁnes the common information between the logits and quantized logits.

Revised: The number of bins used for quantization imposes an upper limit on the information that can be preserved after quantization.

Revised: A3. C ONVERGENCE OF THE I TERATIVE M ETHOD

Original: For convenience, we recall the update equations for { g m , φ m } in Sec.
Revised: For convenience, we recall the update equations for { g m , φ m } in Sec.

Original: 3.3 of the main paper here
Revised: 3.3 of the main paper here

Original: [Equation]
Revised: [Equation]

Original: In the following, we show that the updates on { g m } and { φ m } according to (A12) continuously decrease the loss L , i.e.,
Revised: In the following, we show that the updates on { g m } and { φ m } according to (A13) continuously decrease the loss L , i.e.,

Original: [Equation]
Revised: [Equation]

Original: The second inequality is based on the explained property of L .
Revised: The second inequality is based on the explained property of L .

Original: Namely, it is convex over { φ m } and the minimum for any given { g m } is attained by P σ ( y ; φ m )
Revised: Namely, it is convex over { φ m } and the minimum for any given { g m } is attained by P σ ( y ; φ m )

Original: =
Revised: =

Original: P ( y | m ) .
Revised: P ( y | m ) .

Original: As φ m is the log-probability ratio of P σ ( y ; φ m ) , we shall have
Revised: As φ m is the log-probability ratio of P σ ( y ; φ m ) , we shall have

Original: [Equation]
Revised: [Equation]

Original: where P ( y = 1 | m ) in this case is induced by { g l +1 m } and P ( y | λ )
Revised: where P ( y = 1 | m ) in this case is induced by { g l +1 m } and P ( y | λ )

Original: =
Revised: =

Original: σ [(2 y − 1) λ ] .
Revised: σ [(2 y − 1) λ ] .

Original: Plugging { g l +1 m } and P ( y | λ )
Revised: Plugging { g l +1 m } and P ( y | λ )

Original: =
Revised: =

Original: σ [(2 y − 1) λ ] into (A7), the resulting P ( y = y (cid:48) | m ) at the iteration l + 1 yields the update equation of φ m as given in (A12).
Revised: σ [(2 y − 1) λ ] into (A7), the resulting P ( y = y (cid:48) | m ) at the iteration l + 1 yields the update equation of φ m as given in (A13).

Original: To prove the ﬁrst inequality, we start from showing that { g l +1 m } is a local minimum of L ( { g m , φ lm } ) .
Revised: To prove the ﬁrst inequality, we start from showing that { g l +1 m } is a local minimum of L ( { g m , φ l m } ) .

Original: The update equation on { g m } is an outcome of solving the stationary point equation of L ( { g m , φ lm } ) over { g m } under the condition p ( λ = g m ) > 0 for any m
Revised: The update equation on { g m } is an outcome of solving the stationary point equation of L ( { g m , φ lm } ) over { g m } under the condition p ( λ = g m ) > 0 for any m

Original: ∂ L ( { g m, φ lm } )
Revised: ∂ L ( { g m

Original: [Equation]
Revised: [Equation]

Original: Being a stationary point is the necessary condition of local extremum when the function’s ﬁrst-order derivative exists at that point, i.e., ﬁrst-derivative test.
Revised: Being a stationary point is the necessary condition of local extremum when the function’s ﬁrst-order derivative exists at that point, i.e., ﬁrst-derivative test.

Original: To further show that the local extremum is actually a local minimum, we resort to the second-derivative test, i.e., if the Hessian matrix of L ( { g m , φ lm } ) is positive deﬁnite at the stationary point { g l +1 m } .
Revised: To further show that the local extremum is actually a local minimum, we resort to the second-derivative test, i.e., if the Hessian matrix of L ( { g m , φ lm } ) is positive deﬁnite at the stationary point { g l +1 m } .

Original: Due to φ m > φ m − 1 with the monotonically increasing function sigmoid in its update equation, we have
Revised: Due to φ m > φ m − 1 with the monotonically increasing function sigmoid in its update equation, we have

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: implying that all eigenvalues of the Hessian matrix are positive (equivalently, is positive deﬁnite).
Revised: implying that all eigenvalues of the Hessian matrix are positive (equivalently, is positive deﬁnite).

Original: Therefore, { g l +1 m } as the stationary point of L ( { g m , φ lm } ) is a local minimum.
Revised: Therefore, { g l +1 m } as the stationary point of L ( { g m , φ lm } ) is a local minimum.

Revised: A

Original: It is important to note that from the stationary point equation (A15), { g l +1 m } as a local minimum is unique among { g m } with p ( λ = g m ) > 0 for any m .
Revised: It is important to note that from the stationary point equation (A16), { g l +1 m } as a local minimum is unique among { g m } with p ( λ = g m ) > 0 for any m .

Original: In other words, the ﬁrst inequality holds under the condition p ( λ = g lm ) > 0 for any m .
Revised: In other words, the ﬁrst inequality holds under the condition p ( λ = g lm ) > 0 for any m .

Original: Binning is a lossy data processing.
Revised: Binning is a lossy data processing.

Original: In order to maximally preserve the label information, it is natural to exploit all bins in the optimization, not wasting any single bin in the area without mass, i.e., p ( λ = g m ) = 0 .
Revised: In order to maximally preserve the label information, it is natural to exploit all bins in the optimization, not wasting any single bin in the area without mass, i.e., p ( λ = g m ) = 0 .

Original: Having said that, it is reasonable to constrain { g m } with p ( λ = g m ) >
Revised: Having said that, it is reasonable to constrain { g m } with p ( λ = g m ) >

Original: 0 ∀ m over iterations, thereby concluding that the iterative method will converge to a local minimum based on the two inequalities (A13).
Revised: 0 ∀ m over iterations, thereby concluding that the iterative method will converge to a local minimum based on the two inequalities (A14).

Original: A A3. I NITIALIZATION OF THE I TERATIVE M ETHOD
Revised: A3. I NITIALIZATION OF THE I TERATIVE M ETHOD

Original: We propose to initialize the iterative method by modifying the k-means++ algorithm Arthur & Vassilvitskii (2007) that was developed to initialize the cluster centers for k-means clustering algorithms.
Revised: We propose to initialize the iterative method by modifying the k-means++ algorithm (Arthur & Vassilvitskii, 2007) that was developed to initialize the cluster centers for k-means clustering algorithms.

Original: It is based on the following identiﬁcation
Revised: It is based on the following identiﬁcation

Original: [Equation]
Revised: [Equation]

Original: As I ( y ; λ ) is a constant with respect to ( { g m , φ m } ) , minimizing L is equivalent to minimizing the term on the RHS of (A17).
Revised: As I ( y ; λ ) is a constant with respect to ( { g m , φ m } ) , minimizing L is equivalent to minimizing the term on the RHS of (A18).

Original: The last approximation is reached by turning the binning problem into a clustering problem, i.e., grouping the logit samples in the training set S according to the KLD measure, where { φ m } are effectively the centers of each cluster.
Revised: The last approximation is reached by turning the binning problem into a clustering problem, i.e., grouping the logit samples in the training set S according to the KLD measure, where { φ m } are effectively the centers of each cluster.

Original: k-means++ algorithm Arthur &
Revised: k-means++ algorithm Arthur &

Original: Vassilvitskii (2007) initializes the cluster centers based on the Euclidean distance.
Revised: Vassilvitskii (2007) initializes the cluster centers based on the Euclidean distance.

Original: In our case, we alternatively use the JSD as the distance measure to initialize { φ m } .
Revised: In our case, we alternatively use the JSD as the distance measure to initialize { φ m } .

Original: Comparing with KLD, JSD is symmetric and bounded.
Revised: Comparing with KLD, JSD is symmetric and bounded.

Original: A3. A R EMARK ON THE I TERATIVE M ETHOD D ERIVATION
Revised: A3. A R EMARK ON THE I TERATIVE M ETHOD D ERIVATION

Original: The closed-form update on { g m } in (A12) is based on the sigmoid -model approximation, which has been validated through our empirical experiments.
Revised: The closed-form update on { g m } in (A13) is based on the sigmoid -model approximation, which has been validated through our empirical experiments.

Original: It is expected to work with properly trained classiﬁers that are not overly overﬁtting to the cross-entropy loss, e.g., using data augmentation and other regularization techniques at training.
Revised: It is expected to work with properly trained classiﬁers that are not overly overﬁtting to the cross-entropy loss, e.g., using data augmentation and other regularization techniques at training.

Original: Nevertheless, even in corner cases that classiﬁers are poorly trained, the iterative method can still be operated without the sigmoid -model approximation.
Revised: Nevertheless, even in corner cases that classiﬁers are poorly trained, the iterative method can still be operated without the sigmoid -model approximation.

Original: Namely, as shown in Fig.
Revised: Namely, as shown in Fig.

Original: 2 of the main paper, we can resort to KDE for an empirical estimation of the ground truth distribution p ( λ | y ) .
Revised: 2 of the main paper, we can resort to KDE for an empirical estimation of the ground truth distribution p ( λ | y ) .

Original: Using the KDEs, we can compute the gradient of L over { g m } and perform iterative gradient based update on { g m } , replacing the closed-form based update.
Revised: Using the KDEs, we can compute the gradient of L over { g m } and perform iterative gradient based update on { g m } , replacing the closed-form based update.

Original: Essentially, the sigmoid -model approximation is only necessary to ﬁnd the stationary points of the gradient equations, speeding up the convergence of the method.
Revised: Essentially, the sigmoid -model approximation is only necessary to ﬁnd the stationary points of the gradient equations, speeding up the convergence of the method.

Original: If attempting to keep the closed-form update on { g m } , an alternative solution could be to use the KDEs for adjusting the sigmoid -model, e.g., p ( y | λ )
Revised: If attempting to keep the closed-form update on { g m } , an alternative solution could be to use the KDEs for adjusting the sigmoid -model, e.g., p ( y | λ )

Original: ≈
Revised: ≈

Original: σ [(2 y − 1)( aλ + ab )] , where a and b are chosen to match the KDE based approximation to p ( y | λ ) .
Revised: σ [(2 y − 1)( aλ + ab )] , where a and b are chosen to match the KDE based approximation to p ( y | λ ) .

Original: After setting a and b , they will be used as a scaling and bias term in the original closed-form update equations
Revised: After setting a and b , they will be used as a scaling and bias term in the original closed-form update equations

Original: [Equation]
Revised: [Equation]

Revised: A3.

Revised: C OMPLEXITY AND M EMORY A NALYSIS

Revised: To ease the reproducibility of I-Max, we provide the pseudocode in Algorithm.

Revised: Based on it, we further analyze the complexity and memory cost of I-Max at training and test time.

Revised: We simplify this complexity analysis as our algorithm runs completely ofﬂine and is purely numpybased.

Revised: We note that despite the underlying (numpy) operations performed at each step of the

Revised: A

Revised: Result

Revised: Initialization: { φ m } ← Kmeans++( { λ n } N

Revised: M

Revised: ; for iteration

Revised: , 200 do for

Revised: M do g m ←

Revised:   log (cid:20)

Revised: [Equation]

Original: A19)
Revised: log (cid:20) [Equation] (cid:21)  ; endfor do

Revised: S

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ; end end algorithm differs, we treat multiplication, division, logarithm and exponential functions each counting as the same unit cost and ignore the costs of the logic operations and add/subtract operators.

Revised: The initialization has complexity of O ( NM ) , for the one-dimensional logits.

Revised: We exploit the sklearn implementation of Kmeans++ initialization initially used for Kmeans clustering, but replace the MSE with JSD in the distance measure.

Revised: Following Algorithm 1, we arrive at the following complexity of O ( N ∗ M

Revised: + I ∗ (10 ∗ M + 2 ∗ M )) .

Revised: Our python codes runs Algorithm.

Revised: 1 within seconds for classiﬁers as large as ImageNet and performed purely in Numpy.

Revised: The largest storage and memory consumption is for keeping the N logits used during the I-Max learning phase.

Revised: At test time, there is negligible memory and storage constraints, as only (2 M − 1) ﬂoats need to be saved for the M bin representatives { φ m } M − 1and M − 1 bin edges { g m } M − 1.

Revised: The complexity at test time is merely logic operations to compute the bin assignments of each logit and can be done using numpy’s efﬁcient ‘quantize‘ function.

Revised: I-Max offers a real-time post-hoc calibrator which adds an almost-zero complexity and memory cost relative to the computations of the original classiﬁer.

Revised: We will release our code soon.

Original: A P OST - HOC A NALYSIS ON THE E XPERIMENT R ESULTS IN S EC . 4. 
Revised: A P OST - HOC A NALYSIS ON THE E XPERIMENT R ESULTS IN S EC . 4.

Original: In Tab. 1 of Sec. 4.1, we compared three different binning schemes by measuring their ACCs and
Revised: In Tab. 1 of Sec. 4.1, we compared three different binning schemes by measuring their ACCs and

Original: ECEs.
Revised: ECEs.

Original: The observation on their accuracy performance is aligned with our mutual information maximization viewpoint introduced in Sec.
Revised: The observation on their accuracy performance is aligned with our mutual information maximization viewpoint introduced in Sec.

Original: 3.3 and Fig.
Revised: 3.3 and Fig.

Original: Here, we re-present Fig.
Revised: Here, we re-present Fig.

Original: 2 and provide an alternative explanation to strengthen our understanding on how the location of bin edges affects the accuracy, e.g., why Eq.
Revised: 2 and provide an alternative explanation to strengthen our understanding on how the location of bin edges affects the accuracy, e.g., why Eq.

Original: Size binning performed acceptable at the top- 1 ACC, but failed at the top- 5 ACC.
Revised: Size binning performed acceptable at the top- 1 ACC, but failed at the top- 5 ACC.

Original: Speciﬁcally, Fig.
Revised: Speciﬁcally, Fig.

Original: A3 shows the histograms of raw logits that are grouped based on their ranks instead of their labels as in Fig.
Revised: A3 shows the histograms of raw logits that are grouped based on their ranks instead of their labels as in Fig.

Original: As expected, the logits with low ranks (i.e., rest below top- 5 in Fig.
Revised: As expected, the logits with low ranks (i.e., rest below top- 5 in Fig.

Original: A3) are small and thus take the left hand side of the plot, whereas the top- 1 logits are mostly located on the right hand side.
Revised: A3) are small and thus take the left hand side of the plot, whereas the top- 1 logits are mostly located on the right hand side.

Original: Besides sorting logits according to their ranks, we additionally estimate the density of the ground truth (GT) classes associated logits, i.e., GT in Fig.
Revised: Besides sorting logits according to their ranks, we additionally estimate the density of the ground truth (GT) classes associated logits, i.e., GT in Fig.

Original: A3).
Revised: A3).

Original: With a properly trained classiﬁer, the histogram of top- 1 logits shall largerly overlap with the density curve
Revised: With a properly trained classiﬁer, the histogram of top- 1 logits shall largerly overlap with the density curve

Original: GT , i.e., top- 1 prediction being correct in most cases.A [Figure] [Table]
Revised: GT , i.e., top- 1 prediction being correct in most cases.

Original: [Table]

Original: From the bin edge location of Eq.
Revised: From the bin edge location of Eq.

Original: Mass binning [Table] it attempts to attain small quantization errors for logits of low ranks rather than top- 5 .
Revised: Mass binning, it attempts to attain small quantization errors for logits of low ranks rather than top- 5 .

Original: This will certainly degrade the accuracy performance after binning.
Revised: This will certainly degrade the accuracy performance after binning.

Original: On contrary, Eq.
Revised: On contrary, Eq.

Original: Size binning aims at small quantization error for the top- 1 logits, but ignores top- 5 ones.
Revised: Size binning aims at small quantization error for the top- 1 logits, but ignores top- 5 ones.

Original: As a result, we observed its poor top- 5 ACCs.
Revised: As a result, we observed its poor top- 5 ACCs.

Original: I-Max binning nicely distributes its bin edges in the area where the GT logits are likely to locate, and the bin width becomes smaller in the area where the top- 5 logits are close by (i.e., the overlap region between the red and blue histograms).
Revised: I-Max binning nicely distributes its bin edges in the area where the GT logits are likely to locate, and the bin width becomes smaller in the area where the top- 5 logits are close by (i.e., the overlap region between the red and blue histograms).

Original: Note that, any logit larger than zero must be top- 1 ranked, as there can exist at most one class with prediction probability larger than 0 .
Revised: Note that, any logit larger than zero must be top- 1 ranked, as there can exist at most one A [Figure] [Table]

Original:  Given that, the bins located above zero are no longer to maintain the ranking order, rather to reduce the precision loss of top- 1 prediction probability after binning.
Revised: [Table] class with prediction probability larger than 0 . Given that, the bins located above zero are no longer to maintain the ranking order, rather to reduce the precision loss of top- 1 prediction probability after binning.

Original: The second part of our post-hoc analysis is on the sCW binning strategy.
Revised: The second part of our post-hoc analysis is on the sCW binning strategy.

Original: When using the same binning scheme for all per-class calibration, the chance of creating ties in top- k predictions is much higher than CW binning, e.g., more than one class are top- 1 ranked according to the calibrated prediction probabilities.
Revised: When using the same binning scheme for all per-class calibration, the chance of creating ties in top- k predictions is much higher than CW binning, e.g., more than one class are top- 1 ranked according to the calibrated prediction probabilities.

Original: Our reported ACCs in the main paper are attained by simply returning the ﬁrst found class, i.e., using the class index as the secondary sorting criterion.
Revised: Our reported ACCs in the main paper are attained by simply returning the ﬁrst found class, i.e., using the class index as the secondary sorting criterion.

Original: This is certainly a suboptimal solution.
Revised: This is certainly a suboptimal solution.

Original: Here, we investigate on how the ties affect ACCs of sCW binning.
Revised: Here, we investigate on how the ties affect ACCs of sCW binning.

Original: To this end, we use raw logits (before binning) as the secondary sorting criterion.
Revised: To this end, we use raw logits (before binning) as the secondary sorting criterion.

Original: The resulting ACC ∗ top and ACC ∗ top5 are shown in Tab.A1.
Revised: The resulting ACC ∗ top and ACC ∗ top5 are shown in Tab.A1.

Original: Interestingly, such a simple change reduces the accuracy loss of Eq.
Revised: Interestingly, such a simple change reduces the accuracy loss of Eq.

Original: Mass and I-Max binning to zero, indicating that they can preserve the top- 5 ranking order of the raw logits but not in a strict monotonic sense, i.e., some > are replaced by = .
Revised: Mass and I-Max binning to zero, indicating that they can preserve the top- 5 ranking order of the raw logits but not in a strict monotonic sense, i.e., some > are replaced by = .

Original: As opposed to
Revised: As opposed to

Original: I-Max binning, Eq.
Revised: I-Max binning, Eq.

Original: Mass binning has a poor performance at calibration, i.e., the really high NLL and
Revised: Mass binning has a poor performance at calibration, i.e., the really high NLL and

Original: ECE.
Revised: ECE.

Original: This is because it trivially ranks many classes as top- 1 , but each of them has a very and same small conﬁdence score.
Revised: This is because it trivially ranks many classes as top- 1 , but each of them has a very and same small conﬁdence score.

Original: Given that, even though the accuracy loss is no longer an issue, it is still not a good solution for multi-class calibration.
Revised: Given that, even though the accuracy loss is no longer an issue, it is still not a good solution for multi-class calibration.

Original: For Eq.
Revised: For Eq.

Original: Size binning, resolving ties only helps restore the baseline top-
Revised: Size binning, resolving ties only helps restore the baseline top-

Original: 5 but not top- 1 ACC.
Revised: 5 but not top- 1 ACC.

Original: Its poor bin representative setting due to unreliable empirical frequency estimation over too narrow bins can result in a permutation among the top- 5 predictions.
Revised: Its poor bin representative setting due to unreliable empirical frequency estimation over too narrow bins can result in a permutation among the top- 5 predictions.

Original: Concluding from the above, our post-hoc analysis conﬁrms that I-Max binning outperforms the other two binning schemes at mitigating the accuracy loss and multi-class calibration. In particular, there 
Revised: Concluding from the above, our post-hoc analysis conﬁrms that I-Max binning outperforms the other two binning schemes at mitigating the accuracy loss and multi-class calibration. In particular, there exists a simple solution to close the accuracy gap to the baseline, at the same time still retaining the desirable calibration gains.

Original: A
Revised: A

Original: [Table] [Table]
Revised: [Table] [Table]

Original: exists a simple solution to close the accuracy gap to the baseline, at the same time still retaining the desirable calibration gains.

Original: A A BLATION ON THE N UMBER OF B INS AND C ALIBRATION S ET S IZE
Revised: A A BLATION ON THE N UMBER OF B INS AND C ALIBRATION S ET S IZE

Original: In Tab. 1 of Sec. 4.1, sCW I-Max binning is the top performing one at the ACCs, ECEs and NLL measures.
Revised: In Tab. 1 of Sec. 4.1, sCW I-Max binning is the top performing one at the ACCs, ECEs and NLL measures.

Original: In this part, we further investigate on how the number of bins and calibration set size inﬂuences its performance.
Revised: In this part, we further investigate on how the number of bins and calibration set size inﬂuences its performance.

Original: Tab.
Revised: Tab.

Original: A2 shows that in order to beneﬁt from more bins we shall accordingly increase the number of calibration samples.
Revised: A2 shows that in order to beneﬁt from more bins we shall accordingly increase the number of calibration samples.

Original: More bins help reduce the quantization loss, but increase the empirical frequency estimation error for setting the bin representatives.
Revised: More bins help reduce the quantization loss, but increase the empirical frequency estimation error for setting the bin representatives.

Original: Given that, we observe a reduced ACCs and increased ECEs for having 50 bins with only 1 k calibration samples.
Revised: Given that, we observe a reduced ACCs and increased ECEs for having 50 bins with only 1 k calibration samples.

Original: By increasing the calibration set size to 5 k, then we start seeing the beneﬁts of having more bins to reduce quantization error for better ACCs.
Revised: By increasing the calibration set size to 5 k, then we start seeing the beneﬁts of having more bins to reduce quantization error for better ACCs.

Original: Next, we further exploit scaling method, i.e., GP Wenger et al.
Revised: Next, we further exploit scaling method, i.e., GP Wenger et al.

Original: (2020), for improving the sample efﬁciency of binning at setting the bin representatives.
Revised: (2020), for improving the sample efﬁciency of binning at setting the bin representatives.

Original: As a result, the combination is particularly beneﬁcial to improve the ACCs and top- 1 ECE.
Revised: As a result, the combination is particularly beneﬁcial to improve the ACCs and top- 1 ECE.

Original: Overall, more bins are beneﬁcial to ACCs, while ECEs favor less number of bins.
Revised: Overall, more bins are beneﬁcial to ACCs, while ECEs favor less number of bins

Original:  A E MPIRICAL ECE E STIMATION OF S CALING M ETHODS UNDER M ULTIPLE E VALUATION S CHEMES
Revised: . A E MPIRICAL ECE E STIMATION OF S CALING M ETHODS UNDER M ULTIPLE E VALUATION S CHEMES

Original: As mentioned in the main paper, scaling methods suffer from not being able to provide veriﬁable
Revised: As mentioned in the main paper, scaling methods suffer from not being able to provide veriﬁable

Original: ECEs, see Fig.
Revised: ECEs, see Fig.

Original: Here, we discuss alternatives to estimate their ECEs.
Revised: Here, we discuss alternatives to estimate their ECEs.

Original: The current literature can be split into two types of ECE evaluation: histogram density estimation (HDE) and kernel density estimation (KDE).
Revised: The current literature can be split into two types of ECE evaluation: histogram density estimation (HDE) and kernel density estimation (KDE).

Original: A6. HDE- BASED ECE EVALUATION
Revised: A6. HDE- BASED ECE EVALUATION

Original: HDE bins the prediction probabilities (logits) for density modeling.
Revised: HDE bins the prediction probabilities (logits) for density modeling.

Original: The binning scheme has different variants, where changing the bin edges can give varying measures of the ECE.
Revised: The binning scheme has different variants, where changing the bin edges can give varying measures of the ECE.

Original: Two bin edges schemes have been discussed in the literature (Eq. size and Eq. mass) as well as a new scheme was introduced (I-Max).
Revised: Two bin edges schemes have been discussed in the literature (Eq. size and Eq. mass) as well as a new scheme was introduced (I-Max).

Original: Alternatively, we also evaluate a binning scheme which is based on KMeans clustering to determine the bin edges.A [Figure]
Revised: Alternatively, we also evaluate a binning scheme which is based on KMeans clustering to determine the bin edges.

Original: A6. KDE- BASED ECE EVALUATION
Revised: A6. KDE- BASED ECE EVALUATION

Original: Recent work (Zhang et al., 2020) presented an alternative ECE evaluation scheme which exploits
Revised: Recent work (Zhang et al., 2020) presented an alternative ECE evaluation scheme which exploits

Original: KDEs to estimate the distribution of prediction probabilities { q k } from the test set samples.
Revised: KDEs to estimate the distribution of prediction probabilities { q k } from the test set samples.

Original: Using the code provided by Zhang et al.
Revised: Using the code provided by Zhang et al.

Original: (2020), we observe that the KDE with the setting in their paper can  have a sub-optimal ﬁtting in the probability space.
Revised: (2020), we observe that the KDE with the setting in their paper can A [Figure] have a sub-optimal ﬁtting in the probability space.

Original: This can be observed from Fig.
Revised: This can be observed from Fig.

Original: A4a and Fig.
Revised: A4a and Fig.

Original: A4c, where the ﬁtting is good for ImageNet/Inceptionresnetv2 though when the distribution is signiﬁcantly skewed to the right (as in the case of CIFAR100/WRN) the mismatch becomes large.
Revised: A4c, where the ﬁtting is good for ImageNet/Inceptionresnetv2 though when the distribution is signiﬁcantly skewed to the right (as in the case of CIFAR100/WRN) the mismatch becomes large.

Original: We expect that the case of CIFAR100/WRN is much more common in modern DNNs, due to their high capacities and prone to overﬁtting.
Revised: We expect that the case of CIFAR100/WRN is much more common in modern DNNs, due to their high capacities and prone to overﬁtting [Figure]

Original: Equivalently, we can learn the distribution in its log space by the bijective transformation, i.e., λ = log q − log(1 − q ) and q = σ ( λ ) .
Revised: Equivalently, we can learn the distribution in its log space by the bijective transformation, i.e., λ = log q − log(1 − q ) and q = σ ( λ ) .

Original: As we can observe from Fig.
Revised: As we can observe from Fig.

Original: A4b and Fig.
Revised: A4b and Fig.

Original: A4d, the KDE ﬁtting for both models is consistently good.
Revised: A4d, the KDE ﬁtting for both models is consistently good.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020) empirically validated their KDE in a toy example, where the ground truth ECE can be analytically computed.
Revised: (2020) empirically validated their KDE in a toy example, where the ground truth ECE can be analytically computed.

Original: By analogy, we reproduce the experiment and further compare it with the log-space KDE evaluation.
Revised: By analogy, we reproduce the experiment and further compare it with the log-space KDE evaluation.

Original: Using the same settings as in (Zhang et al., 2020), we assess the ECE evaluation error by KDE, i.e., | ECE gt − ECE kde | , in both the log and probability space, achieving prob 0 .
Revised: Using the same settings as in (Zhang et al., 2020), we assess the ECE evaluation error by KDE, i.e., | ECE gt − ECE kde | , in both the log and probability space, achieving prob 0 .

Original: 0020 vs. log 0 .
Revised: 0020 vs. log 0 .

Original: 0017 for the toy example setting β 0 = 0 .
Revised: 0017 for the toy example setting β 0 = 0 .

Original: 5; β 1 = − 1 .
Revised: 5; β 1 = − 1 .

Original: For an even less calibrated setting, β 0 = 0 . 2; β 1 = − 1 .
Revised: For an even less calibrated setting, β 0 = 0 . 2; β 1 = − 1 .

Original: 9 , we obtain prob 0 . 0029 vs. log 0 .
Revised: 9 , we obtain prob 0 . 0029 vs. log 0 .

Original: So the log-space
Revised: So the log-space

Original: KDE-based ECE evaluation ( kdeECE log ) has lower estimation error than in the probability space.
Revised: KDE-based ECE evaluation ( kdeECE log ) has lower estimation error than in the probability space.

Original: A6. A LTERNATIVE ECE EVALUATION SCHEMES
Revised: A6. A LTERNATIVE ECE EVALUATION SCHEMES

Original: Concluding from the above, Tab. A3 shows the ECE estimates attained by HDEs (from four different bin setting schemes) and KDE (from (Zhang et al., 2020), but in the log space).
Revised: Concluding from the above, Tab. A3 shows the ECE estimates attained by HDEs (from four different bin setting schemes) and KDE (from (Zhang et al., 2020), but in the log space).

Original: As we can see, the obtained results are evaluation scheme dependent.
Revised: As we can see, the obtained results are evaluation scheme dependent.

Original: On contrary, I-Max binning with and without GP 
Revised: On contrary, I-Max binning with and without GP are not affected, and more importantly, their ECEs are better than that of scaling methods, regardless of the evaluation scheme.

Original: A
Revised: A

Original: NetV2.
Revised: NetV2.

Original: Overall, we consider ﬁve evaluation schemes, namely (1) dECE : equal size binning; 2) mECE : equal mass binning, 3) kECE : MSE-based KMeans clustering; 4) iECE : I
Revised: Overall, we consider ﬁve evaluation schemes, namely (1) dECE : equal size binning; (2) mECE : equal mass binning, (3) kECE : MSE-based KMeans clustering; (4) iECE : I-Max binning; 5) kdeECE : KDE.

Original: Max binning

Original: : KDE. The HDEs based schemes, i.e., (1) (4), use 10 2 bins. 
Revised: The HDEs based schemes, i.e., (1)-(4), use 10 2 bins. Note that, the ECEs of

Original: Note that, the binning  calibrator rather than evaluation scheme) are agnostic to the evaluation scheme
Revised: I-Max binning (as a calibrator rather than evaluation scheme) are agnostic to the evaluation scheme.

Original: Furthermore, BBQ suffers from severe accuracy degradation
Revised: Furthermore, BBQ suffers from severe accuracy degradation. [Table] A

Original: [Table]
Revised: [Table]

Revised: A P OST - HOC VS . D URING -T RAINING C ALIBRATION

Revised: To calibrate a DNN-based classiﬁer, there exists two groups of methods.

Revised: One is to improve the calibration during training, whereas the other is post-hoc calibration.

Revised: In this paper, we focus on post-hoc calibration because it is simple and does not require re-training of deployed models.

Revised: In the following, we brieﬂy discuss the advantages and disadvantages of post-hoc and during-training calibration.

Revised: In general, post-hoc and during-training calibration can be viewed as two orthogonal ways to improve the calibration, as they can be easily combined.

Revised: Exemplarily, we compare/combine posthoc calibration methods against/with during-training regularization which directly modiﬁes the training objective to encourage less conﬁdent predictions through an entropy regularization term (Entr. Reg.)

Revised: (Pereyra et al., 2017).

Revised: Additionally, we adopt Mixup (Zhang et al., 2018) which is a data augmentation shown to improve calibration (Thulasidasan et al., 2019).

Revised: We re-train the CIFAR

Revised: WRN classiﬁer respectively using Entr.

Revised: Reg.

Original: are not affected, and more importantly, their ECEs are better than that of scaling methods, regardless of the evaluation scheme
Revised: and Mixup. It can be seen in Tab. A4 that compared to the Baseline model (without training calibration Entr. Reg. or Mixup), EntrReg improves the top1 ECE from 0 .

Revised: 06880 to 0 .

Revised: Further applying post-hoc calibration, I-Max and I-Max w. GP can reduce the 0 . 04806 to 0 .

Revised: 02202 and 0 . 01562 , respectively.

Revised: This indicates that their combination is beneﬁcial.

Revised: In this particular case, we also observed that without Entr.

Revised: Reg., directly post-hoc calibrating the Baseline model appears to be more effective, e.g., top 1 ECE of 0 . 01428 and class-wise

Revised: ECE 0 . 04574 .

Revised: Switching to Mixup, the best top 1 ECE 0 .

Original: [Table]
Revised: 01364 is attained by combining Mixup with post-hoc I-Max w. GP, while I-Max alone without during-training calibration is still the best at class-wise ECE. While post-hoc calibrator is simple and effective at calibration, during-training techniques may deliver more than improving calibration, e.g., improving the generalization performance and providing robustness against adversarials. Therefore, instead of choosing either post-hoc or during training technique, we recommend the combination. While during-training techniques improve the generalization and robustness of the Baseline classiﬁer, post-hoc calibration can further boost its calibration at a low computational cost.

Original: A T RAINING D ETAILS A7. P RE - TRAINED C LASSIFICATION N ETWORKS
Revised: A T RAINING D ETAILS A8. P RE - TRAINED C LASSIFICATION N ETWORKS

Original: We evaluate post-hoc calibration methods on three benchmark datasets, i
Revised: We evaluate post-hoc calibration methods on four benchmark datasets, i.e., ImageNet Deng et al.

Original: [Table]
Revised: (2009), CIFAR-100 Krizhevsky (2009), CIFAR-10 Krizhevsky (2009) and SVHN (Netzer et al., 2011), and across three modern DNNs for each dataset, i.e., InceptionResNetV2 Szegedy et al. DenseNet161 Huang et al.

Original: e., ImageNet Deng et al.
Revised: (2017) and ResNet152 He et al.

Original: (2009), CIFAR-100 Krizhevsky (2009) [Table] and CIFAR-10 Krizhevsky (2009) [Table]
Revised: (2016) for ImageNet, and Wide ResNet (WRN) Zagoruyko & Komodakis (2016) for the two CIFAR datasets and SVHN.

Original: and across three
Revised: Additionally, we A the

Original: All these models are publicly available pre-trained networks and details are reported at the respective websites, i.e., ImageNet classiﬁers: https://github.com/Cadene/ pretrained-models.pytorch and CIFAR classiﬁers: https://github.com/ 
Revised: CIFAR datasets. The ImageNet and CIFAR models are publicly available pre-trained networks and details are reported at the respective websites, i.e., ImageNet classiﬁers: https://github.com/ Cadene/pretrained-models.pytorch and CIFAR classiﬁers: https://github.com/ bearpaw/pytorch-classification .

Original: bearpaw/pytorch-classification A7. T RAINING S CALING M ETHODS
Revised: A8. T RAINING S CALING M ETHODS

Original: The hyper-parameters were decided based on the original respective scaling methods publications with some exceptions.
Revised: The hyper-parameters were decided based on the original respective scaling methods publications with some exceptions.

Original: We found that the following parameters were the best for all the scaling methods.
Revised: We found that the following parameters were the best for all the scaling methods.

Original: All scaling methods use the Adam optimizer with batch size 256 for CIFAR and 4096 for
Revised: All scaling methods use the Adam optimizer with batch size 256 for CIFAR and 4096 for

Original: ImageNet.
Revised: ImageNet.

Original: The learning rate was set to 10 − 3 for temperature scaling Guo et al.
Revised: The learning rate was set to 10 − 3 for temperature scaling Guo et al.

Original: (2017) and Platt

Original: A [Table] [Table] scaling Platt (1999), 0 . 0001 for vector scaling Guo et al.
Revised: and Platt scaling Platt (1999), 0 . 0001 for vector scaling Guo et al.

Original: (2017) and
Revised: (2017) and

Original: 10 − 5 for matrix scaling Guo et al.
Revised: 10 − 5 for matrix scaling Guo et al.

Original: Matrix scaling was further regularized as suggested by Kull et al.
Revised: Matrix scaling was further regularized as suggested by Kull et al.

Original: with a L 2 loss on the bias vector and the off-diagonal elements of the weighting matrix.
Revised: with a L 2 loss on the bias vector and the off-diagonal elements of the weighting matrix.

Original: BBQ Naeini et al.
Revised: BBQ Naeini et al.

Original: , isotonic regression Zadrozny & Elkan (2002) and Beta Kull et al.
Revised: , isotonic regression Zadrozny & Elkan (2002) and Beta Kull et al.

Original: (2017) hyper-parameters were taken directly from Wenger et al.
Revised: (2017) hyper-parameters were taken directly from Wenger et al.

Original: A7. T RAINING I-M AX B INNING
Revised: A8. T RAINING I-M AX B INNING

Original: The I-Max bin optimization started from k-means++ initialization, which uses JSD instead of
Revised: The I-Max bin optimization started from k-means++ initialization, which uses JSD instead of

Original: Euclidean metric as the distance measure, see Sec. A3.2.
Revised: Euclidean metric as the distance measure, see Sec. A3.4.

Original: Then, we iteratively and alternatively updated { g m } and { φ m } according to (A12) until 200 iterations.
Revised: Then, we iteratively and alternatively updated { g m } and { φ m } according to (5) until 200 iterations.

Original: With the attained bin edges { g m } , we set the bin representatives { r m } based on the empirical frequency of class- 1 .
Revised: With the attained bin edges { g m } , we set the bin representatives { r m } based on the empirical frequency of class- 1 .

Original: If a scaling method is combined with binning, an alternative setting for { r m } is to take the averaged prediction probabilities based on the scaled logits of the samples per bin, e.g., in Tab. 2 in Sec.
Revised: If a scaling method is combined with binning, an alternative setting for { r m } is to take the averaged prediction probabilities based on the scaled logits of the samples per bin, e.g., in Tab. 2 in Sec.

Original: Note that, for CW binning in 1, the number of samples from the minority class is too few, i.e., 25k / 1k = 25 .
Revised: Note that, for CW binning in 1, the number of samples from the minority class is too few, i.e., 25k / 1k = 25 .

Original: We only have about 25 / 15 ≈ 2 samples per bin, which are too few to use empirical frequency estimates.
Revised: We only have about 25 / 15 ≈ 2 samples per bin, which are too few to use empirical frequency estimates.

Original: Alternatively, we set { r m } based on the raw prediction probabilities.
Revised: Alternatively, we set { r m } based on the raw prediction probabilities.

Revised: For ImageNet and CIFAR 10/100, which have test sets with uniform class priors, the used sCW setting is to share one binning scheme among all classes.

Revised: Alternatively, for the imbalanced multi-class SVHN setting, we share binning among classes with similar class priors, and thus use the following class (i.e. digit) groupings: { 0 − 1 } , { 2 − 4 } , { 5 − 9 } .

Original: A E XTEND T AB . 1 FOR M ORE D ATASETS AND M ODELS .
Revised: A E XTEND T AB . 1 FOR M ORE D ATASETS AND M ODELS .

Original: Tab. 1 in Sec.
Revised: Tab. 1 in Sec.

Original: 4.1 of the main paper is replicated across datasets and models, where the basic setting remains the same.
Revised: 4.1 of the main paper is replicated across datasets and models, where the basic setting remains the same.

Original: Speciﬁcally, three different ImageNet models can be found in Tab. A4, Tab.
Revised: Speciﬁcally, three different ImageNet models can be found in Tab. A5, Tab.

Original: A and Tab.
Revised: A and Tab.

Original: A6.
Revised: A7.

Original: Three models for CIFAR100 can be found in Tab.
Revised: Three models for CIFAR100 can be found in Tab.

Original: A7, Tab. A8 and Tab. A9.
Revised: A8, Tab. A9 and Tab. A10.

Original: Similarly,
Revised: Similarly,

Original: CIFAR10 models can be found in Tab.
Revised: CIFAR10 models can be found in Tab.

Original: A10, Tab. A11 and Tab.
Revised: A11, Tab. A12 and Tab.

Original: A12.
Revised: A13.

Original: The accuracy degradation of Eq.
Revised: The accuracy degradation of Eq.

Original: Mass reduces as the dataset has less number of classes, e.g., CIFAR10.
Revised: Mass reduces as the dataset has less number of classes, e.g., CIFAR10.

Original: This is because the class imbalance under one-vs-rest conversion becomes less critical in CIFAR10 than ImageNet.
Revised: This is a result of a higher class prior, where the one-vs-rest conversion becomes less critical for CIFAR10 than ImageNet.

Original: Nevertheless, its accuracy losses are still much larger than the other binning schemes, i.e., Eq.
Revised: Nevertheless, its accuracy losses are still much larger than the other binning schemes, i.e., Eq.

Original: Size and I-Max binning.
Revised: Size and I-Max binning.

Original: Therefore, its calibration performance is not considered for comparison.
Revised: Therefore, its calibration performance is not considered for comparison.

Original: Overall, the observations of Tab.
Revised: Overall, the observations of Tab.

Original: A4- A12 are similar to Tab.  stable performance gains A [Table]
Revised: A5- A13 are similar to Tab. 1, showing the stable performance gains of

Original: [Table] [Table] [Table] [Table]
Revised: I-Max binning across datasets and models.

Original: A 
Revised: A E

Original: [Table]
Revised: [Equation]

Revised: FOR M

Original: [Table]
Revised: [Equation]

Original: A E XTEND T AB . 2 FOR M ORE S CALING M ETHODS , D ATASETS AND M ODELS
Revised: S CALING M ETHODS , D ATASETS AND M ODELS

Original: Tab. 2 in Sec.
Revised: Tab. 2 in Sec.

Original: 4.2 of the main paper is replicated across datasets and models, and include more scaling methods for comparison.
Revised: 4.2 of the main paper is replicated across datasets and models, and include more scaling methods for comparison.

Original: The three binning methods all use the shared CW strategy, therefore 1 k calibration samples are sufﬁcient.
Revised: The three binning methods all use the shared CW strategy, therefore 1 k calibration samples are sufﬁcient.

Original: The basic setting remains the same as Tab. 2.
Revised: The basic setting remains the same as Tab. 2.

Original: Three different
Revised: Three different

Original: ImageNet models can be found in Tab.
Revised: ImageNet models can be found in Tab.

Original: A13, Tab. A14 and Tab.
Revised: A14, Tab. A15 and Tab.

Original: A15.
Revised: A16.

Original: Three models for CIFAR can be found in Tab.
Revised: Three models for CIFAR A [Table] [Table]

Original: A16, Tab. A17 and Tab.
Revised: [Table] [Table] [Table] A

Revised: [Table]

Revised: [Table]

Original: A18.
Revised: [Table] [Table] A [Table]

Revised: can be found in Tab.

Revised: A17, Tab.

Revised: A18 and Tab.

Original:  Similarly, CIFAR10 models can be found in
Revised: A19. Similarly, CIFAR10 models can be found in

Original: Tab.
Revised: Tab.

Original: A19, Tab. A20 and Tab.
Revised: A20, Tab. A21 and Tab.

Original: A21.
Revised: A22.

Original: Being analogous to Tab. [Table] we observe that in most cases matrix scaling performs the best at the accuracy, but fail to provide satisfactory calibration performance measured by ECEs, Brier scores and NLLs.
Revised: Being analogous to Tab. 2, we observe that in most cases matrix scaling performs the best at the accuracy, but fail to provide satisfactory calibration performance measured by ECEs, Brier scores and NLLs.

Original: Among the scaling methods, GP Wenger et al. (2020) is the top performing one.
Revised: Among the scaling methods, GP (Wenger et al., 2020) is the top performing one.

Original: Among the binning schemes, our proposal of I-Max binning outperforms Eq.
Revised: Among the binning schemes, our proposal of I-Max binning outperforms Eq.

Original: Mass and Eq.
Revised: Mass and Eq.

Original: Size at both calibration and accuracy.
Revised: Size at accuracies,

Original:  The combination of I-Max binning with GP excels at the ECE performance.It achieves slightly worse results than GP alone at Brier scores and NLLs.
Revised: ECEs, NLLs and Brier scores. The combination of I-Max binning with GP excels at the ECE performance.

Original: This relates to the reduced precision of prediction probabilities after binning.

Original: Note that, among all methods, Eq.
Revised: Note that, among all methods, Eq.

Original: Mass binning and VUC Kumar et al. (2019) suffer from severe accuracy degradation after multi-class calibration.
Revised: Mass binning and VUC (Kumar et al., 2019) suffer from severe accuracy degradation after multi-class calibration.

Original: The reason behind Eq.
Revised: The reason behind Eq.

Original: Mass binning was discussed in Sec.
Revised: Mass binning was discussed in Sec.

Original: 3.3 of the main paper.
Revised: 3.3 of the main paper.

Original: As VUC Kumar et al. combined Eq.
Revised: As VUC (Kumar et al., 2019) combined Eq.

Original: Mass binning with Platt scaling, its accuracy loss is due to the same reason.
Revised: Mass binning with Platt scaling, its accuracy loss is due to the same reason.

Original: Given their poor accuracy, they are not in the scope of calibration performance comparison.
Revised: Given their poor accuracy, they are not in the scope of calibration performance comparison.

Original: We also observe that GP performs better at NLL/
Revised: We also observe that GP performs better at NLL/

Original: Brier than the I-Max variants.
Revised: Brier than the I-Max variants.

Original: The quantized nature of binning causes worse performance than GP at these metrics though improve with increasing the number of bins, e.g., reducing 0 .
Revised: GP is trained by directly optimizing the NLL as its loss. As a non-parametric Bayesian method, GP has larger model expressive capacity than binning. While achieving better NLL/Brier, it costs signiﬁcantly more computational complexity and memory.

Original: 7787 of I-Max w. GP in Tab.
Revised: In contrast, I-Max only relies on logic comparisons at test time. Among the binning schemes, I-Max w. GP achieves the best NLL/Brier across the datasets and models. It is noted that I-Max w. GP remains to be a binning scheme. So, the combination does not change the model capacity of I-Max.

Original: A13 to 0 .
Revised: GP is only exploited during training to improve the optimization on I-Max’s bin representatives. Besides the low complexity beneﬁt, I-Max w. GP as a binning scheme does not suffer from the ECE underestimation issue of scaling methods such as GP. We further note that as a cross entropy measure between two distributions, the NLL would be an ideal metric for calibration evaluation. However, empirical NLL and Brier favor high accuracy and high conﬁdent classiﬁers, as each sample only having one hard label essentially implies the maximum conﬁdence on a single class.

Original: 7681 after using 100 bins.
Revised: For the this reason, during training, the empirical NLL loss will keep pushing the prediction probability to one even after reaching 100% training set accuracy. As a result, the trained classiﬁer showed poor calibration performance at test time (Guo et al., 2017). In contrast to NLL/Brier, empirical ECEs use hard labels differently. The ground truth correctness associated to the prediction conﬁdence p is estimated by averaging over the hard labels of the samples receiving the prediction probability p or close to p .

Original: Note, that unlike ECE, NLL/Brier are not direct measures of calibration and care more about accuracy. For example, for N predictions assigned  conﬁdence  1  containing M mistakes, the calibrated conﬁdence is M/N < 1 .
Revised: Due to averaging, the empirical ground truth correctness is usually not a hard label. Lastly, we use a small example to show the difference between the NLL/Brier and ECE: for N predictions, all assigned a conﬁdence of 1 . 0 and containing M mistakes, the calibrated conﬁdence is M/N < 1 .

Original:  NLL/Brier  is  non-zero only for the M wrong predictions, despite all N predictions are miscalibrated.
Revised: Unlike ECE, the NLL/Brier loss is only non-zero only for the M wrong predictions, despite all N predictions being miscalibrated.

Original: A [Table]
Revised: This example shows that

Original: [Table]
Revised: NLL/Brier penalize miscalibration far less than ECE.

Original: A
Revised: A

Original: [Table][Table]
Revised: [Table]

Original: Calibration

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A
Revised: A

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A
Revised: A

Original: [Table] 
Revised: [Table] [Table]

Original: A
Revised: A