Original: A Consistent and Differentiable L p Canonical Calibration Error Estimator
Revised: A Consistent and Differentiable L p Canonical Calibration Error Estimator

Original: Abstract
Revised: Abstract

Original: Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates.
Revised: Calibrated probabilistic classifiers are models whose predicted probabilities can directly be interpreted as uncertainty estimates.

Original: It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions.
Revised: It has been shown recently that deep neural networks are poorly calibrated and tend to output overconfident predictions.

Original: As a remedy, we propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true L p calibration error.
Revised: As a remedy, we propose a low-bias, trainable calibration error estimator based on Dirichlet kernel density estimates, which asymptotically converges to the true L p calibration error.

Original: This novel estimator enables us to tackle the strongest notion of multiclass calibration, called canonical (or distribution) calibration, while other common calibration methods are tractable only for top-label and marginal calibration.
Revised: This novel estimator enables us to tackle the strongest notion of multiclass calibration, called canonical (or distribution) calibration, while other common calibration methods are tractable only for top-label and marginal calibration.

Original: The computational complexity of our estimator is O ( n 2 ) , the convergence rate is O ( n − 1 / 2 ) , and it is unbiased up to O ( n − 2 ) , achieved by a geometric series debiasing scheme.
Revised: The computational complexity of our estimator is O ( n 2 ) , the convergence rate is O ( n − 1 / 2 ) , and it is unbiased up to O ( n − 2 ) , achieved by a geometric series debiasing scheme.

Original: In practice, this means that the estimator can be applied to small subsets of data, enabling efficient estimation and mini-batch updates.
Revised: In practice, this means that the estimator can be applied to small subsets of data, enabling efficient estimation and mini-batch updates.

Original: The proposed method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities based on conditional expectation, such as the sharpness of a probabilistic classifier.
Revised: The proposed method has a natural choice of kernel, and can be used to generate consistent estimates of other quantities based on conditional expectation, such as the sharpness of a probabilistic classifier.

Original: Empirical results validate the correctness of our estimator, and demonstrate its utility in canonical calibration error estimation and calibration error regularized risk minimization.
Revised: Empirical results validate the correctness of our estimator, and demonstrate its utility in canonical calibration error estimation and calibration error regularized risk minimization.

Original:  Introduction
Revised:  Introduction

Original: Deep neural networks have shown tremendous success in classification tasks, being regularly the best performing models in terms of accuracy.
Revised: Deep neural networks have shown tremendous success in classification tasks, being regularly the best performing models in terms of accuracy.

Original: However, they are also known to make overconfident predictions [Guo et al., 2017], which is particularly problematic in safety-critical applications, such as medical diagnosis [Esteva et al., 2017, 2019] or autonomous driving [Caesar et al., 2020, Sun et al., 2020].
Revised: However, they are also known to make overconfident predictions [Guo et al., 2017], which is particularly problematic in safety-critical applications, such as medical diagnosis [Esteva et al., 2017, 2019] or autonomous driving [Caesar et al., 2020, Sun et al., 2020].

Original: In many real world applications it is not only the predictive performance that is important, but also the trustworthiness of the prediction, i.e., we are interested in accurate predictions with robust uncertainty estimates.
Revised: In many real world applications it is not only the predictive performance that is important, but also the trustworthiness of the prediction, i.e., we are interested in accurate predictions with robust uncertainty estimates.

Original: To this end, it is necessary that the models are uncertainty calibrated, which means that, for instance, among all cells that have been predicted with a probability of 0.8 to be cancerous, 80% should indeed belong to a malignant tumor.
Revised: To this end, it is necessary that the models are uncertainty calibrated, which means that, for instance, among all cells that have been predicted with a probability of 0.8 to be cancerous, 80% should indeed belong to a malignant tumor.

Original: The field of uncertainty calibration has been mostly focused on binary problems, often considering only the confidence score of the predicted class.
Revised: The field of uncertainty calibration has been mostly focused on binary problems, often considering only the confidence score of the predicted class.

Original: However, this so called top-label (or confidence) calibration [Guo et al., 2017]) is often not sufficient in multiclass settings.
Revised: However, this so called top-label (or confidence) calibration [Guo et al., 2017]) is often not sufficient in multiclass settings.

Original: A stronger notion of calibration is marginal (or class-wise)
Revised: A stronger notion of calibration is marginal (or class-wise)

Original: [Kull et al., 2019], that splits up the multiclass problem
Revised: [Kull et al., 2019], that splits up the multiclass problem

Original: [Table]
Revised: [Table]

Original: into K one-vs-all binary ones, and requires each to be calibrated according to the definition of binary calibration.
Revised: into K one-vs-all binary ones, and requires each to be calibrated according to the definition of binary calibration.

Original: The most strict notion of calibration, called canonical (or distribution) calibration [Bröcker, 2009, Kull and Flach, 2015, Vaicenavicius et al., 2019], requires the whole probability vector to be calibrated.
Revised: The most strict notion of calibration, called canonical (or distribution) calibration [Bröcker, 2009, Kull and Flach, 2015, Vaicenavicius et al., 2019], requires the whole probability vector to be calibrated.

Original: The curse of dimensionality makes estimation of this form of calibration difficult, and current estimators, such as the binned estimator ECE bin [Naeini et al., 2015], MMCE [Kumar et al., 2018] and Mix-n-Match [Zhang et al., 2020], have computational or statistical limitations that prevent them from being successfully applied in this important setting.
Revised: The curse of dimensionality makes estimation of this form of calibration difficult, and current estimators, such as the binned estimator ECE bin [Naeini et al., 2015], MMCE [Kumar et al., 2018] and Mix-n-Match [Zhang et al., 2020], have computational or statistical limitations that prevent them from being successfully applied in this important setting.

Original: Specifically, the binned estimator is sensitive to the binning scheme and is asymptotically inconsistent in many situations Vaicenavicius et al.
Revised: Specifically, the binned estimator is sensitive to the binning scheme and is asymptotically inconsistent in many situations Vaicenavicius et al.

Original: [2019], MMCE is not a consistent estimator of L p calibration error and Mix-n-Match, although consistent, is intractable in high dimensions and the authors did not implement it in more than one dimension.
Revised: [2019], MMCE is not a consistent estimator of L p calibration error and Mix-n-Match, although consistent, is intractable in high dimensions and the authors did not implement it in more than one dimension.

Original: We propose a tractable, differentiable, and consistent estimator of the expected L p canonical calibration error .
Revised: We propose a tractable, differentiable, and consistent estimator of the expected L p canonical calibration error .

Original: In particular, we use kernel density estimates (KDEs) with a Beta kernel in binary classification tasks and a Dirichlet kernel in the multiclass setting, as these kernels are the natural choices to model densities over a probability simplex.
Revised: In particular, we use kernel density estimates (KDEs) with a Beta kernel in binary classification tasks and a Dirichlet kernel in the multiclass setting, as these kernels are the natural choices to model densities over a probability simplex.

Original: In Table 1, we summarize and compare the properties of our ECE KDE estimator and other commonly used estimators.
Revised: In Table 1, we summarize and compare the properties of our ECE KDE estimator and other commonly used estimators.

Original: ECE KDE scales well to higher dimensions and it is able to capture canonical calibration with O ( n 2 ) complexity.
Revised: ECE KDE scales well to higher dimensions and it is able to capture canonical calibration with O ( n 2 ) complexity.

Original: Our contributions can be summarized as follows:
Revised: Our contributions can be summarized as follows:

Original: We develop a tractable estimator of canonical L p calibration error that is consistent and differentiable.
Revised: We develop a tractable estimator of canonical L p calibration error that is consistent and differentiable.

Original: We demonstrate a natural choice of kernel.
Revised: We demonstrate a natural choice of kernel.

Original: Due to the scaling properties of Dirichlet kernel density estimation, evaluating canonical calibration becomes feasible in cases that cannot be estimated using other methods.
Revised: Due to the scaling properties of Dirichlet kernel density estimation, evaluating canonical calibration becomes feasible in cases that cannot be estimated using other methods.

Original: We provide a second order debiasing scheme to further improve the convergence of the estimator. 4.
Revised: We provide a second order debiasing scheme to further improve the convergence of the estimator. 4.

Original: We empirically evaluate the correctness of our estimator and demonstrate its utility in the task of calibration regularized risk minimization on variety of network architectures and several datasets.
Revised: We empirically evaluate the correctness of our estimator and demonstrate its utility in the task of calibration regularized risk minimization on variety of network architectures and several datasets.

Original:  Related Work
Revised:  Related Work

Original: Calibration of probabilistic predictors has long been studied in many fields.
Revised: Calibration of probabilistic predictors has long been studied in many fields.

Original: This topic gained attention in the deep learning community since Guo et al.
Revised: This topic gained attention in the deep learning community since Guo et al.

Original: [2017] observed that modern neural networks are poorly calibrated and tend to give overconfident predictions due to overfitting on the NLL loss.
Revised: [2017] observed that modern neural networks are poorly calibrated and tend to give overconfident predictions due to overfitting on the NLL loss.

Original: The surge of interest resulted in many calibration strategies that can be split in two general categories, which we discuss subsequently.
Revised: The surge of interest resulted in many calibration strategies that can be split in two general categories, which we discuss subsequently.

Original: Post-hoc calibration strategies learn a calibration map of the predictions from a trained predictor in a post-hoc manner, using a held-out calibration set.
Revised: Post-hoc calibration strategies learn a calibration map of the predictions from a trained predictor in a post-hoc manner, using a held-out calibration set.

Original: For instance, Platt scaling [Platt, 1999] fits a logistic regression model on top of the logit outputs of the model.
Revised: For instance, Platt scaling [Platt, 1999] fits a logistic regression model on top of the logit outputs of the model.

Original: A special case of Platt scaling that fits a single scalar, called temperature, has been popularized by Guo et al.
Revised: A special case of Platt scaling that fits a single scalar, called temperature, has been popularized by Guo et al.

Original: [2017] as an accuracypreserving, easy to implement and effective method to improve calibration.
Revised: [2017] as an accuracypreserving, easy to implement and effective method to improve calibration.

Original: However, it has the undesired consequence that it clamps the high confidence scores of accurate predictions [Kumar et al., 2018].
Revised: However, it has the undesired consequence that it clamps the high confidence scores of accurate predictions [Kumar et al., 2018].

Original: Similar approaches for post-hoc calibration include histogram binning [Zadrozny and Elkan, 2001], isotonic regression [Zadrozny and Elkan, 2002],
Revised: Similar approaches for post-hoc calibration include histogram binning [Zadrozny and Elkan, 2001], isotonic regression [Zadrozny and Elkan, 2002],

Original: Bayesian binning into quantiles [Naeini and
Revised: Bayesian binning into quantiles [Naeini and

Original: Cooper, 2016], Beta [Kull et al., 2017] and Dirichlet calibration [Kull et al., 2019].
Revised: Cooper, 2016], Beta [Kull et al., 2017] and Dirichlet calibration [Kull et al., 2019].

Original: Recently, Gupta et al.
Revised: Recently, Gupta et al.

Original: [2021] proposed a binning-free calibration measure based on the Kolmogorov-Smirnov test.
Revised: [2021] proposed a binning-free calibration measure based on the Kolmogorov-Smirnov test.

Original: In this approach, the recalibration function is obtained via spline-fitting, rather than minimizing a loss function on a calibration set.
Revised: In this approach, the recalibration function is obtained via spline-fitting, rather than minimizing a loss function on a calibration set.

Original: Ma et al.
Revised: Ma et al.

Original: [2021] integrate ensamble-based and post-hoc calibration methods in an accuracy-perserving truth discovery framework.
Revised: [2021] integrate ensamble-based and post-hoc calibration methods in an accuracy-perserving truth discovery framework.

Original: Zhao et al.
Revised: Zhao et al.

Original: [2021] introduce a new notion of calibration, called decision calibration, however, they do not propose an estimator of calibration error with statistical guarnatees.
Revised: [2021] introduce a new notion of calibration, called decision calibration, however, they do not propose an estimator of calibration error with statistical guarnatees.

Original: Trainable calibration strategies integrate a differentiable calibration measure into the training objective.
Revised: Trainable calibration strategies integrate a differentiable calibration measure into the training objective.

Original: One of the earliest approaches is regularization by penalizing low entropy predictions [Pereyra et al., 2017].
Revised: One of the earliest approaches is regularization by penalizing low entropy predictions [Pereyra et al., 2017].

Original: Similarly to temperature scaling, it has been shown that entropy regularization needlessly suppresses high confidence scores of correct predictions [Kumar et al., 2018].
Revised: Similarly to temperature scaling, it has been shown that entropy regularization needlessly suppresses high confidence scores of correct predictions [Kumar et al., 2018].

Original: Another popular strategy is MMCE (Maximum Mean Calibration Error)
Revised: Another popular strategy is MMCE (Maximum Mean Calibration Error)

Original: [Kumar et al., 2018], where the entropy regularizer is replaced by a kernel-based surrogate for the calibration error that can be optimized alongside NLL.
Revised: [Kumar et al., 2018], where the entropy regularizer is replaced by a kernel-based surrogate for the calibration error that can be optimized alongside NLL.

Original: It has been shown that label smoothing [Szegedy et al., 2016, Müller et al., 2019], i.e. training models with a weighted mixture of the labels instead of one-hot vectors, also improves model calibration.
Revised: It has been shown that label smoothing [Szegedy et al., 2016, Müller et al., 2019], i.e. training models with a weighted mixture of the labels instead of one-hot vectors, also improves model calibration.

Original: Liang et al.
Revised: Liang et al.

Original: [2020] propose to add the difference between predicted confidence and accuracy as auxiliary term to the cross-entropy loss.
Revised: [2020] propose to add the difference between predicted confidence and accuracy as auxiliary term to the cross-entropy loss.

Original: Focal loss [Mukhoti et al., 2020,
Revised: Focal loss [Mukhoti et al., 2020,

Original: Lin et al., 2017] has recently been empirically shown to produce better calibrated models than many of the alternatives, but does not estimate a clear quantity related to calibration error.
Revised: Lin et al., 2017] has recently been empirically shown to produce better calibrated models than many of the alternatives, but does not estimate a clear quantity related to calibration error.

Original: Bohdal et al.
Revised: Bohdal et al.

Original: [2021] derive a differentiable approximation to the commonly-used binned estimator of calibration error by computing differentiable approximations to the 0/1 loss and the binning operator.
Revised: [2021] derive a differentiable approximation to the commonly-used binned estimator of calibration error by computing differentiable approximations to the 0/1 loss and the binning operator.

Original: However, this approach does not eliminate the dependence on the binning scheme and it is not clear how it can be extended to calibration of the whole probability vector.
Revised: However, this approach does not eliminate the dependence on the binning scheme and it is not clear how it can be extended to calibration of the whole probability vector.

Original: Kernel density estimation [Parzen, 1962, Rosenblatt, 1956, Silverman, 1986] is a non-parametric method to estimate a probability density function from a finite sample.
Revised: Kernel density estimation [Parzen, 1962, Rosenblatt, 1956, Silverman, 1986] is a non-parametric method to estimate a probability density function from a finite sample.

Original: Zhang et al.
Revised: Zhang et al.

Original: [2020] propose a
Revised: [2020] propose a

Original: KDE-based estimator of the calibration error (Mix-n-Match) for measuring calibration performance.
Revised: KDE-based estimator of the calibration error (Mix-n-Match) for measuring calibration performance.

Original: Although they demonstrate consistency of the method, it requires a numerical integration step that is infeasible in high dimensions.
Revised: Although they demonstrate consistency of the method, it requires a numerical integration step that is infeasible in high dimensions.

Original: In practice, they only implemented binary calibration, and not canonical calibration.
Revised: In practice, they only implemented binary calibration, and not canonical calibration.

Original: Although many calibration strategies have been empirically shown to decrease the calibration error, very few of them are based on an estimator of miscalibration.
Revised: Although many calibration strategies have been empirically shown to decrease the calibration error, very few of them are based on an estimator of miscalibration.

Original: Our estimator is the first consistent, differentiable estimator with favourable scaling properties that has been successfully applied to the estimation of L p canonical calibration error in the multi-class setting.
Revised: Our estimator is the first consistent, differentiable estimator with favourable scaling properties that has been successfully applied to the estimation of L p canonical calibration error in the multi-class setting.

Original:  Methods
Revised:  Methods

Original: We study a classical supervised classification problem.
Revised: We study a classical supervised classification problem.

Original: Let (Ω , A , P ) be a probability space, where Ω is the set of possible outcomes, A
Revised: Let (Ω , A , P ) be a probability space, where Ω is the set of possible outcomes, A

Original: =
Revised: =

Original: A (Ω) is the sigma field of events and P :
Revised: A (Ω) is the sigma field of events and P :

Original: A → [0 , 1] is a probability measure, let X
Revised: A → [0 , 1] is a probability measure, let X

Original: = R d and Y = { 1 , ..., K } .
Revised: = R d and Y = { 1 , ..., K } .

Original: Let x : Ω → X and y :
Revised: Let x : Ω → X and y :

Original: Ω → Y be random variables, while realizations are denoted with subscripts.
Revised: Ω → Y be random variables, while realizations are denoted with subscripts.

Original: Suppose we have a model f : X → △ K , where △ K denotes the K − 1 dimensional simplex as obtained, e.g., from the output of a final softmax layer in a neural network.
Revised: Suppose we have a model f : X → △ K , where △ K denotes the K − 1 dimensional simplex as obtained, e.g., from the output of a final softmax layer in a neural network.

Original: We measure the (mis-)calibration in terms of the L p calibration error, defined below.
Revised: We measure the (mis-)calibration in terms of the L p calibration error, defined below.

Original: Definition 3.1 (Calibration error, [Naeini et al., 2015, Kumar et al., 2019, Wenger et al., 2020]) .
Revised: Definition 3.1 (Calibration error, [Naeini et al., 2015, Kumar et al., 2019, Wenger et al., 2020]) .

Original: The
Revised: The

Original: L p calibration error of f is:
Revised: L p calibration error of f is:

Original: [Equation]
Revised: [Equation]

Original: We note that we consider multiclass calibration, and that f ( x ) and the conditional expectation in
Revised: We note that we consider multiclass calibration, and that f ( x ) and the conditional expectation in

Original: Equation (1) therefore map to points on a probability simplex.
Revised: Equation (1) therefore map to points on a probability simplex.

Original: We say that a classifier f is perfectly calibrated if CE p ( f ) = 0 .
Revised: We say that a classifier f is perfectly calibrated if CE p ( f ) = 0 .

Original: In order to empirically compute the conditional expectation in Equation (1), we need to perform density estimation over the probability simplex.
Revised: In order to empirically compute the conditional expectation in Equation (1), we need to perform density estimation over the probability simplex.

Original: In a binary setting, this has traditionally been done with binned estimates [Naeini et al., 2015, Guo et al., 2017, Kumar et al., 2019].
Revised: In a binary setting, this has traditionally been done with binned estimates [Naeini et al., 2015, Guo et al., 2017, Kumar et al., 2019].

Original: However, this is not differentiable w.r.t.
Revised: However, this is not differentiable w.r.t.

Original: the function f , and cannot be incorporated into a gradient based training procedure.
Revised: the function f , and cannot be incorporated into a gradient based training procedure.

Original: Furthermore, binned estimates suffer from the curse of dimensionality and do not have a practical extension to multiclass settings.
Revised: Furthermore, binned estimates suffer from the curse of dimensionality and do not have a practical extension to multiclass settings.

Original: We consider an estimator for the CE p based on Beta and
Revised: We consider an estimator for the CE p based on Beta and

Original: Dirichlet kernel density estimates in the binary and multiclass setting, respectively.
Revised: Dirichlet kernel density estimates in the binary and multiclass setting, respectively.

Original: We require that this estimator is consistent and differentiable such that we can train it in a calibration error regularized risk minimization framework.
Revised: We require that this estimator is consistent and differentiable such that we can train it in a calibration error regularized risk minimization framework.

Original: This estimator is given by:
Revised: This estimator is given by:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: denotes
Revised: denotes

Original: [Equation]
Revised: [Equation]

Original: E [ y | f ( x )] evaluated at f ( x ) =
Revised: E [ y | f ( x )] evaluated at f ( x ) =

Original: f
Revised: f

Original: ( x j ) .
Revised: ( x j ) .

Original: If probability density p x,y is measurable with respect to the product of the Lebesgue and counting measure, we can define: p x,y ( x i , y i ) =
Revised: If probability density p x,y is measurable with respect to the product of the Lebesgue and counting measure, we can define: p x,y ( x i , y i ) =

Original: p y | x
Revised: p y | x

Original: = x i ( y i ) p x ( x i ) .
Revised: = x i ( y i ) p x ( x i ) .

Original: Then we define the estimator of the conditional expectation as follows:
Revised: Then we define the estimator of the conditional expectation as follows:

Original: [Equation]
Revised: [Equation]

Original: where k is the kernel of a kernel density estimate evaluated at point x and p f ( x ) is uniquely determined by p x and f .
Revised: where k is the kernel of a kernel density estimate evaluated at point x and p f ( x ) is uniquely determined by p x and f .

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Assuming that p f ( x ) ( f ( x )) is Lipschitz continuous over the interior of the simplex, there exists a kernel k such that
Revised: Assuming that p f ( x ) ( f ( x )) is Lipschitz continuous over the interior of the simplex, there exists a kernel k such that

Original: [Equation]
Revised: [Equation]

Original: E [ y | f ( x )] is a pointwise consistent estimator of E [ y | f ( x )] , that is:
Revised: E [ y | f ( x )] is a pointwise consistent estimator of E [ y | f ( x )] , that is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Let k be a Dirichlet kernel
Revised: Let k be a Dirichlet kernel

Original: [Ouimet and Tolosana-Delgado, 2022].
Revised: [Ouimet and Tolosana-Delgado, 2022].

Original: By the consistency of the Dirichlet kernel density estimators [Ouimet and Tolosana-Delgado, 2022, Theorem 4] Lipschitz continuity of the density over the simplex is a sufficient condition for uniform convergence of the kernel density estimate.
Revised: By the consistency of the Dirichlet kernel density estimators [Ouimet and Tolosana-Delgado, 2022, Theorem 4] Lipschitz continuity of the density over the simplex is a sufficient condition for uniform convergence of the kernel density estimate.

Original: This in turn implies that for a given f , for all f ( x ) ∈ (0 , 1) , 1 n (cid:80) ni
Revised: This in turn implies that for a given f , for all f ( x ) ∈ (0 , 1) , 1 n (cid:80) ni

Original: =1 k ( f ( x ); f ( x i ))
Revised: =1 k ( f ( x ); f ( x i ))

Original: y i p −→ (cid:80)
Revised: y i p −→ (cid:80)

Original: y k ∈Y y k p
Revised: y k ∈Y y k p

Original: f
Revised: f

Original: ( x ) ,y ( f ( x ) , y k ) and 1 n (cid:80) ni
Revised: ( x ) ,y ( f ( x ) , y k ) and 1 n (cid:80) ni

Original: =1 k ( f ( x ); f ( x i ))
Revised: =1 k ( f ( x ); f ( x i ))

Original: p −→ p
Revised: p −→ p

Original: f
Revised: f

Original: ( x ) ( f ( x )) .
Revised: ( x ) ( f ( x )) .

Original: Let g ( x ) = 1 /x , then the set of discontinuities of g applied to the denominator of the l.h.s.
Revised: Let g ( x ) = 1 /x , then the set of discontinuities of g applied to the denominator of the l.h.s.

Original: of (4) has measure zero since 1 n (cid:80) ni
Revised: of (4) has measure zero since 1 n (cid:80) ni

Original: =1 k ( f ( x ); f ( x i ))
Revised: =1 k ( f ( x ); f ( x i ))

Original: = 

Revised: =

Original: with probability zero.
Revised: with probability zero.

Original: From the continuous mapping theorem [Mann and Wald, 1943] it follows, that n/ ( (cid:80) ni
Revised: From the continuous mapping theorem [Mann and Wald, 1943] it follows, that n/ ( (cid:80) ni

Original: =1 k ( f ( x ); f ( x i )))
Revised: =1 k ( f ( x ); f ( x i )))

Original: p −→ 1 /p f ( x ) ( f ( x )) .
Revised: p −→ 1 /p f ( x ) ( f ( x )) .

Original: Since products of convergent (in probability) sequences of random variables converge in probability to the product of their limits [Resnick, 2019], we have that (cid:80) ni
Revised: Since products of convergent (in probability) sequences of random variables converge in probability to the product of their limits [Resnick, 2019], we have that (cid:80) ni

Original: =1 k ( f ( x ); f ( x i )
Revised: =1 k ( f ( x ); f ( x i )

Original: [Equation]
Revised: [Equation]

Original: y
Revised: y

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]
Revised: [Equation]

Original: ( f ( x ); f ( x i )))
Revised: ( f ( x ); f ( x i )))

Original: [Equation]
Revised: [Equation]

Original: , which is equal to the r.h.s.
Revised: , which is equal to the r.h.s.

Original: of (4).
Revised: of (4).

Original: The most commonly used loss functions are designed to achieve consistency in the sense of Bayes optimality under risk minimization, however, they do not guarantee calibration - neither for finite samples nor in the asymptotic limit.
Revised: The most commonly used loss functions are designed to achieve consistency in the sense of Bayes optimality under risk minimization, however, they do not guarantee calibration - neither for finite samples nor in the asymptotic limit.

Original: Since we are interested in models f that are both accurate and calibrated, we consider the following optimization problem bounding the calibration error CE( f ) : f
Revised: Since we are interested in models f that are both accurate and calibrated, we consider the following optimization problem bounding the calibration error CE( f ) : f

Original: [Equation]
Revised: [Equation]

Original: arg
Revised: arg

Original: [Equation]
Revised: [Equation]

Original: Risk
Revised: Risk

Original: [Equation]
Revised: [Equation]

Original: f )
Revised: f )

Original: [Equation]
Revised: [Equation]

Original: s.t.
Revised: s.t.

Original: CE
Revised: CE

Original: [Equation]
Revised: [Equation]

Original: B for some B > 0 , and its associated Lagrangian
Revised: B for some B > 0 , and its associated Lagrangian

Original: [Equation]
Revised: [Equation]

Original: Mean squared error in binary classification As a first instantiation of our framework we consider a binary classification setting, with mean squared error MSE( f ) =
Revised: Mean squared error in binary classification As a first instantiation of our framework we consider a binary classification setting, with mean squared error MSE( f ) =

Original: E [( f ( x ) − y ) 2 ] as the risk function, jointly optimized with the L 2 calibration error CE 2 :
Revised: E [( f ( x ) − y ) 2 ] as the risk function, jointly optimized with the L 2 calibration error CE 2 :

Original: [Equation]
Revised: [Equation]

Original: where γ = λλ +1 ∈ [0 , 1) .
Revised: where γ = λλ +1 ∈ [0 , 1) .

Original: The full derivation using the MSE decomposition [Murphy, 1973, Degroot and Fienberg, 1983, Kuleshov and Liang, 2015, Nguyen and O’Connor, 2015] is given in Appendix
Revised: The full derivation using the MSE decomposition [Murphy, 1973, Degroot and Fienberg, 1983, Kuleshov and Liang, 2015, Nguyen and O’Connor, 2015] is given in Appendix

Original: A. For optimization we wish to find an estimator for E
Revised: A. For optimization we wish to find an estimator for E

Original: [ E [ y | f
Revised: [ E [ y | f

Original: ( x )] 2 ] .
Revised: ( x )] 2 ] .

Original: Building upon Equation (3), a partially debiased estimator can be written as:
Revised: Building upon Equation (3), a partially debiased estimator can be written as:

Original: [Equation]
Revised: [Equation]

Original: Thus, the conditional expectation is estimated using a ratio of unbiased estimators of the square of a mean.
Revised: Thus, the conditional expectation is estimated using a ratio of unbiased estimators of the square of a mean.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: Equation (7) is a ratio of two U-statistics and has a bias converging as O (cid:0) 1 n (cid:1) .
Revised: Equation (7) is a ratio of two U-statistics and has a bias converging as O (cid:0) 1 n (cid:1) .

Original: The proof is given in Appendix B.
Revised: The proof is given in Appendix B.

Original: Proposition 3.4.
Revised: Proposition 3.4.

Original: There exist de-biasing schemes for the ratios in Equation (7) and Equation (3) that achieve an improved O (cid:0) 1 n 2 (cid:1) convergence of the bias.
Revised: There exist de-biasing schemes for the ratios in Equation (7) and Equation (3) that achieve an improved O (cid:0) 1 n 2 (cid:1) convergence of the bias.

Original: Proofs are given in Appendix C and D.
Revised: Proofs are given in Appendix C and D.

Original: In a binary setting, the kernels k ( · , · ) are Beta distributions defined as:
Revised: In a binary setting, the kernels k ( · , · ) are Beta distributions defined as:

Original: [Equation]
Revised: [Equation]

Original: with α i = f
Revised: with α i = f

Original: ( x i ) h
Revised: ( x i ) h

Original: + 

Revised: +

Original: and β i = 1 − f ( x i ) h
Revised: and β i = 1 − f ( x i ) h

Original: + 

Revised: +

Original: [Chen, 1999, Bouezmarni and Rolin, 2003, Zhang and
Revised: [Chen, 1999, Bouezmarni and Rolin, 2003, Zhang and

Original: Karunamuni, 2010], where h is a bandwidth parameter in the kernel density estimate that goes to 0 as n → ∞ .
Revised: Karunamuni, 2010], where h is a bandwidth parameter in the kernel density estimate that goes to 0 as n → ∞ .

Original: We note that the computational complexity of this estimator is O ( n 2 ) .
Revised: We note that the computational complexity of this estimator is O ( n 2 ) .

Original: If we would use this within a gradient descent training procedure, the density can be estimated using a mini-batch and therefore the O ( n 2 ) complexity is w.r.t.
Revised: If we would use this within a gradient descent training procedure, the density can be estimated using a mini-batch and therefore the O ( n 2 ) complexity is w.r.t.

Original: the size of a mini-batch, not the entire dataset.
Revised: the size of a mini-batch, not the entire dataset.

Original: The estimator in Equation (7) is a ratio of two second order U-statistics that converge as n − 1 /[Ferguson, 2005].
Revised: The estimator in Equation (7) is a ratio of two second order U-statistics that converge as n − 1 /[Ferguson, 2005].

Original: Therefore, the overall convergence will be n − 1 / 2 .
Revised: Therefore, the overall convergence will be n − 1 / 2 .

Original: Empirical convergence rates are calculated in Appendix G and shown to be close to the theoretically expected value.
Revised: Empirical convergence rates are calculated in Appendix G and shown to be close to the theoretically expected value.

Original: Multiclass calibration with Dirichlet kernel density estimates There are multiple definitions regarding multiclass calibration that differ in the strictness regarding the calibration of the probability
Revised: Multiclass calibration with Dirichlet kernel density estimates There are multiple definitions regarding multiclass calibration that differ in the strictness regarding the calibration of the probability

Original: vector f ( x ) .
Revised: vector f ( x ) .

Original: The strongest notion of multiclass calibration, and the one that we consider in this paper, is canonical (also called multiclass or distribution) calibration [Bröcker, 2009, Kull and Flach, 2015, Vaicenavicius et al., 2019], which requires that the whole probability vector f ( x ) is calibrated (Definition 3.1).
Revised: The strongest notion of multiclass calibration, and the one that we consider in this paper, is canonical (also called multiclass or distribution) calibration [Bröcker, 2009, Kull and Flach, 2015, Vaicenavicius et al., 2019], which requires that the whole probability vector f ( x ) is calibrated (Definition 3.1).

Original: Its estimator is:
Revised: Its estimator is:

Original: [Equation]
Revised: [Equation]

Original: where k Dir is a Dirichlet kernel defined as:
Revised: where k Dir is a Dirichlet kernel defined as:

Original: [Equation]
Revised: [Equation]

Original: with α i = f
Revised: with α i = f

Original: ( x i ) h
Revised: ( x i ) h

Original: + 1 [Ouimet and Tolosana-Delgado, 2022].
Revised: + 1 [Ouimet and Tolosana-Delgado, 2022].

Original: As before, the computational complexity is O ( n 2 ) irrespective of p .
Revised: As before, the computational complexity is O ( n 2 ) irrespective of p .

Original: This estimator is differentiable and furthermore, the following proposition holds:
Revised: This estimator is differentiable and furthermore, the following proposition holds:

Original: Proposition 3.5.
Revised: Proposition 3.5.

Original: The Dirichlet kernel based CE estimator is consistent when p f ( x ) ( f ( x )) is Lipschitz:
Revised: The Dirichlet kernel based CE estimator is consistent when p f ( x ) ( f ( x )) is Lipschitz:

Original: [Equation]
Revised: [Equation]

Original: p .
Revised: p .

Original: Proof.
Revised: Proof.

Original: Dirichlet kernel estimators are consistent when the density is Lipschitz continuous over the simplex [Ouimet and Tolosana-Delgado, 2022, Theorem 4], consequently, by Proposition 3.2 the term inside the norm is consistent for any fixed f ( x j ) (note, that summing over i ̸
Revised: Dirichlet kernel estimators are consistent when the density is Lipschitz continuous over the simplex [Ouimet and Tolosana-Delgado, 2022, Theorem 4], consequently, by Proposition 3.2 the term inside the norm is consistent for any fixed f ( x j ) (note, that summing over i ̸

Original: =
Revised: =

Original: j ensures that the ratio of the KDE’s does not depend on the outer summation).
Revised: j ensures that the ratio of the KDE’s does not depend on the outer summation).

Original: Moreover, for any convergent sequence also the norm of that sequence converges to the norm of its limit.
Revised: Moreover, for any convergent sequence also the norm of that sequence converges to the norm of its limit.

Original: Ultimately, the outer sum is merely the sample mean of consistent summands, which again is consistent.
Revised: Ultimately, the outer sum is merely the sample mean of consistent summands, which again is consistent.

Original: With this development, we have for the first time a consistent, differentiable, tractable estimator of
Revised: With this development, we have for the first time a consistent, differentiable, tractable estimator of

Original: L p canonical calibration error with O ( n 2 ) computational cost and O ( n − 1 / 2 ) convergence rate, with a debiasing scheme that achieves O ( n − 2 ) bias for p ∈ { 1 , 2 } .
Revised: L p canonical calibration error with O ( n 2 ) computational cost and O ( n − 1 / 2 ) convergence rate, with a debiasing scheme that achieves O ( n − 2 ) bias for p ∈ { 1 , 2 } .

Original:  Empirical validation of ECE KDE
Revised:  Empirical validation of ECE KDE

Original: Accurately evaluating the calibration error is a crucial step towards designing trustworthy models that can be used in societally important settings.
Revised: Accurately evaluating the calibration error is a crucial step towards designing trustworthy models that can be used in societally important settings.

Original: The most widely used metric for evaluating miscalibration, and the only other estimator that can be straightforwardly extended to measure canonical calibration, is the histogram-based estimator ECE bin .
Revised: The most widely used metric for evaluating miscalibration, and the only other estimator that can be straightforwardly extended to measure canonical calibration, is the histogram-based estimator ECE bin .

Original: However, as discussed in Vaicenavicius et al.
Revised: However, as discussed in Vaicenavicius et al.

Original: Widmann et al.
Revised: Widmann et al.

Original: [2019], Ding et al.
Revised: [2019], Ding et al.

Original: [2020], Ashukha et al.
Revised: [2020], Ashukha et al.

Original: [2020], it has numerous flaws, such as: (i) it is sensitive to the binning scheme (ii) it is severely affected by the curse of dimensionality, as the number of bins grows exponentially with the number of classes (iii) it is asymptotically inconsistent in many cases.
Revised: [2020], it has numerous flaws, such as: (i) it is sensitive to the binning scheme (ii) it is severely affected by the curse of dimensionality, as the number of bins grows exponentially with the number of classes (iii) it is asymptotically inconsistent in many cases.

Original: To investigate its relationship with our estimator ECE KDE , we first introduce an extension of the top-label binned estimator to the probability simplex in the three class setting.
Revised: To investigate its relationship with our estimator ECE KDE , we first introduce an extension of the top-label binned estimator to the probability simplex in the three class setting.

Original: We start by partitioning the probability simplex into equally-sized, triangle-shaped bins and assign the probability scores to the corresponding bin, as shown in Figure 1a.
Revised: We start by partitioning the probability simplex into equally-sized, triangle-shaped bins and assign the probability scores to the corresponding bin, as shown in Figure 1a.

Original: Then, we define the binned estimate of canonical calibration error as follows: CE p ( f ) p ≈ E (cid:104)
Revised: Then, we define the binned estimate of canonical calibration error as follows: CE p ( f ) p ≈ E (cid:104)

Original: ∥
Revised: ∥

Original: H ( f ( x )) − f ( x ) ∥
Revised: H ( f ( x )) − f ( x ) ∥

Original: pp (cid:105) ≈ 1 n (cid:80) ni
Revised: pp (cid:105) ≈ 1 n (cid:80) ni

Original: =1 ∥ H ( f ( x i ))
Revised: =1 ∥ H ( f ( x i ))

Original: − f ( x i ) ∥ pp , where
Revised: − f ( x i ) ∥ pp , where

Original: H ( f ( x i )) is the histogram estimate, shown in Figure 1b.
Revised: H ( f ( x i )) is the histogram estimate, shown in Figure 1b.

Original: The surface of the corresponding Dirichlet
Revised: The surface of the corresponding Dirichlet

Original: KDE is presented in Figure 1c.
Revised: KDE is presented in Figure 1c.

Original: See Appendix F for (i) an experiment investigating their relationship for the three types of calibration (top-label, marginal, canonical) and with varying number of points used for the estimation, and (ii) another example of the binned estimator and Dirichlet KDE on
Revised: See Appendix F for (i) an experiment investigating their relationship for the three types of calibration (top-label, marginal, canonical) and with varying number of points used for the estimation, and (ii) another example of the binned estimator and Dirichlet KDE on

Original: CIFAR-10.
Revised: CIFAR-10.

Original:                 
Revised:                 

Original:  (
Revised:  (

Original: [Figure]
Revised: [Figure]

Original: Synthetic experiments We consider an extension of ECE bin to arbitrary number of classes and investigate its performance compared to ECE KDE .
Revised: Synthetic experiments We consider an extension of ECE bin to arbitrary number of classes and investigate its performance compared to ECE KDE .

Original: Since on real data the ground truth calibration error is unknown, we generate synthetic data with known transformations with the following procedure.
Revised: Since on real data the ground truth calibration error is unknown, we generate synthetic data with known transformations with the following procedure.

Original: First, we sample uniformly from the simplex using the Kraemer algorithm
Revised: First, we sample uniformly from the simplex using the Kraemer algorithm

Original: [Smith and
Revised: [Smith and

Original: Tromble, 2004].
Revised: Tromble, 2004].

Original: Then, we apply temperature scaling with t 1 = 0 .
Revised: Then, we apply temperature scaling with t 1 = 0 .

Original: 6 to simulate realistic scenarios where the probability scores are concentrated along lower dimensional faces of the simplex.
Revised: 6 to simulate realistic scenarios where the probability scores are concentrated along lower dimensional faces of the simplex.

Original: We generate ground truth labels according to the sampled probabilities and therefore obtain a perfectly calibrated classifier.
Revised: We generate ground truth labels according to the sampled probabilities and therefore obtain a perfectly calibrated classifier.

Original: Subsequently, the classifier is miscalibrated by additional temperature scaling with t 2 = 0 .
Revised: Subsequently, the classifier is miscalibrated by additional temperature scaling with t 2 = 0 .

Original: Figure 2a depicts the performance of the two estimators as a function of the sample size on generated data for 4 and 8 classes.
Revised: Figure 2a depicts the performance of the two estimators as a function of the sample size on generated data for 4 and 8 classes.

Original: ECE KDE converges to the ground truth value obtained by integration in both cases, whereas ECE bin provides poor estimates even with 20000 points.
Revised: ECE KDE converges to the ground truth value obtained by integration in both cases, whereas ECE bin provides poor estimates even with 20000 points.

Original: In another experiment with synthetic data we look at the bias of the sharpness 3 term in a binary setting.
Revised: In another experiment with synthetic data we look at the bias of the sharpness 3 term in a binary setting.

Original: In Figure 2b we plot the estimated value of the sharpness term for varying number of samples, both using the partially debiased ratio from Equation (7) and the ratio debiased with the scheme introduced in Appendix D. A sigmoidal function is applied to the calibrated data to obtain an uncalibrated sample that is used to compute the partially debiased and the fully debiased ratio of the sharpness term.
Revised: In Figure 2b we plot the estimated value of the sharpness term for varying number of samples, both using the partially debiased ratio from Equation (7) and the ratio debiased with the scheme introduced in Appendix D. A sigmoidal function is applied to the calibrated data to obtain an uncalibrated sample that is used to compute the partially debiased and the fully debiased ratio of the sharpness term.

Original: The ground truth value is obtained by using 100 million samples to compute the ratio with the partially debiased version, as it converges asymptotically to the true value due to its consistency.
Revised: The ground truth value is obtained by using 100 million samples to compute the ratio with the partially debiased version, as it converges asymptotically to the true value due to its consistency.

Original: We use a bandwidth of 0.5 and average over 10000 repetitions for each number of samples that range fromto 16384.
Revised: We use a bandwidth of 0.5 and average over 10000 repetitions for each number of samples that range fromto 16384.

Original: We fix the location of the KDE at f ( x j ) = 0 .
Revised: We fix the location of the KDE at f ( x j ) = 0 .

Original: [Figure]
Revised: [Figure]

Original:     
Revised:     

Original: [Figure]
Revised: [Figure]

Original:  Calibration regularized training
Revised:  Calibration regularized training

Original: Empirical setup To showcase our estimator in applications where canonical calibration is crucial, we consider two medical datasets, namely Kather and DermaMNIST.
Revised: Empirical setup To showcase our estimator in applications where canonical calibration is crucial, we consider two medical datasets, namely Kather and DermaMNIST.

Original: The Kather dataset [Kather et al., 2016] consists of 5000 histological images of human colorectal cancer and it has eight different classes of tissue.
Revised: The Kather dataset [Kather et al., 2016] consists of 5000 histological images of human colorectal cancer and it has eight different classes of tissue.

Original: DermaMNIST [Yang et al., 2021] is a pre-processed version of the HAM10000 dataset [Tschandl et al., 2018], containing 10015 dermatoscopic images of skin lesions, categorized in seven classes.
Revised: DermaMNIST [Yang et al., 2021] is a pre-processed version of the HAM10000 dataset [Tschandl et al., 2018], containing 10015 dermatoscopic images of skin lesions, categorized in seven classes.

Original: Both datasets have been collected in accordance with the Declaration of Helsinki.
Revised: Both datasets have been collected in accordance with the Declaration of Helsinki.

Original: According to standard practice in related works, we trained ResNet [He et al., 2016], ResNet with stochastic depth (SD)
Revised: According to standard practice in related works, we trained ResNet [He et al., 2016], ResNet with stochastic depth (SD)

Original: [Huang et al., 2016], DenseNet [Huang et al., 2017] and WideResNet [Zagoruyko and
Revised: [Huang et al., 2016], DenseNet [Huang et al., 2017] and WideResNet [Zagoruyko and

Original: Komodakis, 2016] networks also on CIFAR-10/

Revised: Komodakis, 2016] networks also on CIFAR-10/

Original: [Krizhevsky, 2009].
Revised: [Krizhevsky, 2009].

Original: We use 45000 images for training on the CIFAR datasets, 4000 for Kather and 7007 for DermaMNIST.
Revised: We use 45000 images for training on the CIFAR datasets, 4000 for Kather and 7007 for DermaMNIST.

Original: The code is available at https://github.com/tpopordanoska/ece-kde .
Revised: The code is available at https://github.com/tpopordanoska/ece-kde .

Original: Baselines Cross-entropy : The first baseline model is trained using cross-entropy ( XE ), with the data preprocessing, training procedure and hyperparameters described in the corresponding paper for the architecture.
Revised: Baselines Cross-entropy : The first baseline model is trained using cross-entropy ( XE ), with the data preprocessing, training procedure and hyperparameters described in the corresponding paper for the architecture.

Original: Trainable calibration strategies KDE-XE denotes our proposed estimator ECE KDE , as defined in Equation (9), jointly trained with cross entropy.
Revised: Trainable calibration strategies KDE-XE denotes our proposed estimator ECE KDE , as defined in Equation (9), jointly trained with cross entropy.

Original: MMCE [Kumar et al., 2018] is a differentiable measure of calibration with a property that it is minimized at perfect calibration, i.e., MMCE is 

Revised: MMCE [Kumar et al., 2018] is a differentiable measure of calibration with a property that it is minimized at perfect calibration, i.e., MMCE is

Original: if and only if CE p = 0 .
Revised: if and only if CE p = 0 .

Original: It is used as a regulariser alongside NLL, with the strength of regularization parameterized by λ .
Revised: It is used as a regulariser alongside NLL, with the strength of regularization parameterized by λ .

Original: Focal loss (FL) [Mukhoti et al., 2020] is an alternative to the cross-entropy loss, defined as L f = − (1 − f
Revised: Focal loss (FL) [Mukhoti et al., 2020] is an alternative to the cross-entropy loss, defined as L f = − (1 − f

Original: ( y | x ))
Revised: ( y | x ))

Original: γ log( f ( y | x )) , where γ is a hyperparameter and f ( y | x ) is the probability score that a neural network f outputs for a class y on an input x .
Revised: γ log( f ( y | x )) , where γ is a hyperparameter and f ( y | x ) is the probability score that a neural network f outputs for a class y on an input x .

Original: Their best-performing approach is the sample-dependent FL-53 where γ = 5 for f ( y | x ) ∈ [0 , 0 . 2) and γ = 3 otherwise, followed by the method with fixed γ = 3 .
Revised: Their best-performing approach is the sample-dependent FL-53 where γ = 5 for f ( y | x ) ∈ [0 , 0 . 2) and γ = 3 otherwise, followed by the method with fixed γ = 3 .

Original: Post-hoc calibration strategies Guo et al.
Revised: Post-hoc calibration strategies Guo et al.

Original: [2017] investigated the performance of several post-hoc calibration methods and found temperature scaling to be a strong baseline, which we use as a representative of this group.
Revised: [2017] investigated the performance of several post-hoc calibration methods and found temperature scaling to be a strong baseline, which we use as a representative of this group.

Original: It works by scaling the logits with a scalar T > 0 , typically learned on a validation set by minimizing NLL.
Revised: It works by scaling the logits with a scalar T > 0 , typically learned on a validation set by minimizing NLL.

Original: Following Kumar et al.
Revised: Following Kumar et al.

Original: [2018], Mukhoti et al.
Revised: [2018], Mukhoti et al.

Original: [2020], we also use temperature scaling as a post-processing step for our method.
Revised: [2020], we also use temperature scaling as a post-processing step for our method.

Original: Metrics We report L 1 canonical calibration using our ECE KDE estimator, calculated according to Equation (9).
Revised: Metrics We report L 1 canonical calibration using our ECE KDE estimator, calculated according to Equation (9).

Original: Additional experiments with L 1 and L 2 top-label calibration on CIFAR-10/100 can be found in Appendix E.
Revised: Additional experiments with L 1 and L 2 top-label calibration on CIFAR-10/100 can be found in Appendix E.

Original:     
Revised:     

Original:          ' HQ V L W \ .'( b =0.001 .'( b =0.01 .'( b =0.
Revised:          ' HQ V L W \ .'( b =0.001 .'( b =0.01 .'( b =0.

Original: +LVWRJUDPIURPVDPSOHV
Revised: +LVWRJUDPIURPVDPSOHV

Original: Hyperparameters A crucial parameter for KDE is the bandwidth b , a positive number that defines the smoothness of the density plot.
Revised: Hyperparameters A crucial parameter for KDE is the bandwidth b , a positive number that defines the smoothness of the density plot.

Original: Poorly chosen bandwidth may lead to undersmoothing (small bandwidth) or oversmoothing (large bandwidth), as shown in Figure 3.
Revised: Poorly chosen bandwidth may lead to undersmoothing (small bandwidth) or oversmoothing (large bandwidth), as shown in Figure 3.

Original: A commonly used non-parametric bandwidth selector is maximum likelihood cross validation [Duin, 1976].
Revised: A commonly used non-parametric bandwidth selector is maximum likelihood cross validation [Duin, 1976].

Original: For our experiments we choose the bandwidth from a list of possible values by maximizing the leave-one-out likelihood (LOO MLE).
Revised: For our experiments we choose the bandwidth from a list of possible values by maximizing the leave-one-out likelihood (LOO MLE).

Original: The λ parameter for weighting the calibration error w.r.t the loss is typically chosen via cross-validation or using a holdout validation set.
Revised: The λ parameter for weighting the calibration error w.r.t the loss is typically chosen via cross-validation or using a holdout validation set.

Original: We found that for KDE-XE, values of λ ∈ [0 .
Revised: We found that for KDE-XE, values of λ ∈ [0 .

Original: 001 , 0 . 2] provide a good trade-off in terms of accuracy and calibration error.
Revised: 001 , 0 . 2] provide a good trade-off in terms of accuracy and calibration error.

Original: The p parameter is selected depending on the desired L p calibration error and the corresponding theoretical guarantees.
Revised: The p parameter is selected depending on the desired L p calibration error and the corresponding theoretical guarantees.

Original: The rest of the hyperparameters for training are set as proposed in the corresponding papers for the architectures we benchmark.
Revised: The rest of the hyperparameters for training are set as proposed in the corresponding papers for the architectures we benchmark.

Original: In particular, for the CIFAR-10/100 datasets we used a batch size of 64 for DenseNet and 128 for the other architectures.
Revised: In particular, for the CIFAR-10/100 datasets we used a batch size of 64 for DenseNet and 128 for the other architectures.

Original: For the medical datasets, we used a batch size of 64, due to their smaller size.
Revised: For the medical datasets, we used a batch size of 64, due to their smaller size.

Original: 5.1 Experiments
Revised: 5.1 Experiments

Original: An important property of our ECE KDE estimator is differentiability, allowing it be used in a calibration regularized training framework.
Revised: An important property of our ECE KDE estimator is differentiability, allowing it be used in a calibration regularized training framework.

Original: In this section, we benchmark KDE-XE with several baselines on medical diagnosis applications, where the calibration of the whole probability vector is of particular interest.
Revised: In this section, we benchmark KDE-XE with several baselines on medical diagnosis applications, where the calibration of the whole probability vector is of particular interest.

Original: For completeness, we also include an experiment on CIFAR-10.
Revised: For completeness, we also include an experiment on CIFAR-10.

Original: Table 2 summarizes the canonical L 1 ECE KDE and Table 3 the accuracy, measured across multiple architectures.
Revised: Table 2 summarizes the canonical L 1 ECE KDE and Table 3 the accuracy, measured across multiple architectures.

Original: The bandwidth is chosen by LOO MLE.
Revised: The bandwidth is chosen by LOO MLE.

Original: For MMCE and KDE-XE the best performing regularization weight is reported.
Revised: For MMCE and KDE-XE the best performing regularization weight is reported.

Original: In Table 2 we notice that KDE-XE consistently achieves very competitive ECE values, while also boosting the accuracy, as shown in Table 3.
Revised: In Table 2 we notice that KDE-XE consistently achieves very competitive ECE values, while also boosting the accuracy, as shown in Table 3.

Original: Interestingly, we observe that temperature scaling does not improve canonical calibration error, contrary to its reported improvements on top-label calibration.
Revised: Interestingly, we observe that temperature scaling does not improve canonical calibration error, contrary to its reported improvements on top-label calibration.

Original: This observation that temperature scaling is less effective for stronger notions of calibration is consistent with a similar finding in Kull et al.
Revised: This observation that temperature scaling is less effective for stronger notions of calibration is consistent with a similar finding in Kull et al.

Original: [Table]
Revised: [Table]

Original: where the authors show that although the temperature-scaled model has well calibrated top-label confidence scores, the calibration error is much larger for class-wise calibration
Revised: where the authors show that although the temperature-scaled model has well calibrated top-label confidence scores, the calibration error is much larger for class-wise calibration

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Figure 4 shows the performance of several architectures and datasets in terms of accuracy and L
Revised: Figure 4 shows the performance of several architectures and datasets in terms of accuracy and L

Original: ECE KDE for various choices of the regularization parameter for MMCE and KDE-XE.
Revised: ECE KDE for various choices of the regularization parameter for MMCE and KDE-XE.

Original: The 95 % confidence intervals for ECE KDE are calculated using 100 and 10 bootstrap samples on the medical datasets and CIFAR-10, respectively.
Revised: The 95 % confidence intervals for ECE KDE are calculated using 100 and 10 bootstrap samples on the medical datasets and CIFAR-10, respectively.

Original: In all settings, KDE-XE Pareto dominates the competitors, for several choices of λ .
Revised: In all settings, KDE-XE Pareto dominates the competitors, for several choices of λ .

Original: For example, on DermaMNIST trained with DenseNet, KDE-XE with λ = 0 .reduces
Revised: For example, on DermaMNIST trained with DenseNet, KDE-XE with λ = 0 .reduces

Original: ECE KDE from 66% to 45%.
Revised: ECE KDE from 66% to 45%.

Original:      
Revised:      

Original:  
Revised: 

Original:  $&&      
Revised:  $&&      

Original: 
Revised: 

Original: L 1  E C E
Revised: L 1  E C E

Original: K D E   
Revised: K D E   

Original:        ;()/00&(
Revised:        ;()/00&(

Original: L 1 .'(;( (a) ResNet-110 (SD) on Kather 
Revised: L 1 .'(;( (a) ResNet-110 (SD) on Kather 

Original:     $&&       
Revised:     $&&       

Original: L 1  E C E
Revised: L 1  E C E

Original: K D E  
Revised: K D E  

Original:         ;()/00&(
Revised:         ;()/00&(

Original: L 1 .'(;( (b) DenseNet on DermaMNIST 
Revised: L 1 .'(;( (b) DenseNet on DermaMNIST 

Original:   
Revised:   

Original:  
Revised:  

Original:  $&&       
Revised:  $&&       

Original: L 1  E C E
Revised: L 1  E C E

Original: K D E         ;()/00&(
Revised: K D E         ;()/00&(

Original: L 1 .'(;( (c) ResNet-110 on CIFAR-
Revised: L 1 .'(;( (c) ResNet-110 on CIFAR-

Original: Training time measurements In Table 4 we summarize the running time per epoch of the four architectures, with regularization (KDE-XE) and without regularization (XE).
Revised: Training time measurements In Table 4 we summarize the running time per epoch of the four architectures, with regularization (KDE-XE) and without regularization (XE).

Original: We observe only an insignificant impact on the training speed when using KDE-XE, dispelling any concerns w.
Revised: We observe only an insignificant impact on the training speed when using KDE-XE, dispelling any concerns w.

Original: .t.
Revised: .t.

Original: the computational overhead
Revised: the computational overhead

Original: [Table]
Revised: [Table]

Original: To summarize, the experiments show that our estimator is consistently producing competitive calibration errors with other state-of-theart approaches, while maintaining accuracy and keeping the computational complexity at O ( n 2 ) .
Revised: To summarize, the experiments show that our estimator is consistently producing competitive calibration errors with other state-of-theart approaches, while maintaining accuracy and keeping the computational complexity at O ( n 2 ) .

Original: We note that within the proposed calibrationregularized training framework, this complexity is w.r.t.
Revised: We note that within the proposed calibrationregularized training framework, this complexity is w.r.t.

Original: to a mini-batch, and the added cost is less than a couple percent.
Revised: to a mini-batch, and the added cost is less than a couple percent.

Original: Furthermore, the
Revised: Furthermore, the

Original: O ( n 2 ) complexity shows up in other related works [Kumar et al., 2018, Zhang et al., 2020], and is intrinsic to the problem of density estimators of calibration error.
Revised: O ( n 2 ) complexity shows up in other related works [Kumar et al., 2018, Zhang et al., 2020], and is intrinsic to the problem of density estimators of calibration error.

Original: As a future work, a larger scale benchmarking will be beneficial for exploring the limits of canonical calibration using Dirichlet kernels.
Revised: As a future work, a larger scale benchmarking will be beneficial for exploring the limits of canonical calibration using Dirichlet kernels.

Original:  Conclusion
Revised:  Conclusion

Original: In this paper, we proposed a consistent and differentiable estimator of canonical L p calibration error using Dirichlet kernels.
Revised: In this paper, we proposed a consistent and differentiable estimator of canonical L p calibration error using Dirichlet kernels.

Original: It has favorable computational and statistical properties, with a complexity of
Revised: It has favorable computational and statistical properties, with a complexity of

Original: O ( n 2 ) , convergence of O ( n − 1 / 2 ) and a bias that converges as O ( n − 1 ) , which can be further reduced to O ( n − 2 ) using our debiasing strategy.
Revised: O ( n 2 ) , convergence of O ( n − 1 / 2 ) and a bias that converges as O ( n − 1 ) , which can be further reduced to O ( n − 2 ) using our debiasing strategy.

Original: The ECE KDE can be directly optimized alongside any loss function in the existing batch stochastic gradient descent framework.
Revised: The ECE KDE can be directly optimized alongside any loss function in the existing batch stochastic gradient descent framework.

Original: Furthermore, we propose using it as a measure of the highest form of calibration, which requires the entire probability vector to be calibrated.
Revised: Furthermore, we propose using it as a measure of the highest form of calibration, which requires the entire probability vector to be calibrated.

Original: To the best of our knowledge, this is the only metric that can tractably capture this type of calibration, which is crucial in safety-critical applications where downstream decisions are made based on the predicted probabilities.
Revised: To the best of our knowledge, this is the only metric that can tractably capture this type of calibration, which is crucial in safety-critical applications where downstream decisions are made based on the predicted probabilities.

Original: We showed empirically on a range of neural architectures and datasets that the performance of our estimator in terms of accuracy and calibration error is competitive against the current state-of-the-art, while having superior properties as a consistent estimator of canonical calibration error.
Revised: We showed empirically on a range of neural architectures and datasets that the performance of our estimator in terms of accuracy and calibration error is competitive against the current state-of-the-art, while having superior properties as a consistent estimator of canonical calibration error.

Original: Acknowledgments
Revised: Acknowledgments

Original: This research received funding from the Research Foundation - Flanders (FWO) through project number S001421N, and the Flemish Government under the “Onderzoeksprogramma Artificiële
Revised: This research received funding from the Research Foundation - Flanders (FWO) through project number S001421N, and the Flemish Government under the “Onderzoeksprogramma Artificiële

Original: Intelligentie (AI) Vlaanderen” programme.
Revised: Intelligentie (AI) Vlaanderen” programme.

Original: R.S. was supported in part by the Tübingen AI centre.
Revised: R.S. was supported in part by the Tübingen AI centre.

Original: Ethical statement
Revised: Ethical statement

Original: The paper is concerned with estimation of calibration error, a topic for which existing methods are deployed, albeit not typically for canonical calibration error in a multi-class setting.
Revised: The paper is concerned with estimation of calibration error, a topic for which existing methods are deployed, albeit not typically for canonical calibration error in a multi-class setting.

Original: We therefore consider the ethical risks to be effectively the same as for any probabilistic classifier.
Revised: We therefore consider the ethical risks to be effectively the same as for any probabilistic classifier.

Original: Experiments apply the method to medical image classification, for which misinterpretation of benchmark results with respect to their clinical applicability has been highlighted as a risk, see e.g. Varoquaux and
Revised: Experiments apply the method to medical image classification, for which misinterpretation of benchmark results with respect to their clinical applicability has been highlighted as a risk, see e.g. Varoquaux and

Original: Cheplygina [2022].A Additional derivations A.1 Derivation of the MSE decomposition Definition A.1 (Mean Squared Error (MSE)) . The mean squared error of an estimator is
Revised: Cheplygina [2022].

Original: [Equation]

Original: Proposition A.2.

Original: MSE( f ) ≥ CE 2 ( f )

Original: Proof

Original: [Equation]

Original: which implies

Original: [Equation]

Original: By the law of total expectation

Original: [Equation]

Original: we will write the above as

Original: [Equation]

Original: Focusing on the inner conditional expectation

Original: [Equation]

Original: we have that

Original: [Equation]

Original: and therefore

Original: [Equation]

Original: The expectation in Equation (23) is over variances of Bernoulli random variables with probabilities E

Original: [ y | f ( x )] .

Original: A.2 Derivation of Equation (3)

Original: By considering y ∈ { 0 , 1 }

Original: [Equation]

Original: we have the following:

Original: [Equation]

Original: A.3 Derivation of Equation (6)

Original: We consider the optimization problem for some λ > 0 :

Original: [Equation]

Original: Using Equation (23) we rewrite:

Original: [Equation]

Original: Rescaling Equation (29) by a factor of (1 + λ ) − 1 and a variable substitution γ = λ 1+ λ ∈ [0 , 1) , we have that:

Original: [Equation]

Original: B Bias of ratio of U-statistics

Original: The unbiased estimator for the square of a mean µ 2 X is given by:

Original: [Equation]

Original: This is a second order U-statistics with kernel h ( x 1 , x 2 ) =

Original: x 1 x 2 .

Original: The bias of the ratio of two of these estimators converges as O (cid:0) 1 n (cid:1) , as the following lemma proves.

Original: Lemma B.1.

Original: [-Let θ 
-]
Revised: [-Let θ 
-]

Original: [-and θ 2 be two estimable parameters and let U 
-]
Revised: [-and θ 2 be two estimable parameters and let U 
-]

Original: and U 2 be the two corresponding

Original: U-statistics of order m 1 and m 2 , respectively, based on a sample of n i.i.d.

Original: RVs.

Original: The bias of the ratio

Original: U 1 /U 2 of these two U-statistics will converge as O (cid:0) 1 n (cid:1) .

Original: Proof.

Original: Let R

Original: = θ

Original: 1 /θ 2 be the ratio of two estimable parameters and r

Original: =

Original: U 1 /U 2 the ratio of the corresponding U-statistics.

Original: Note, that U

Original: i is an unbiased estimator of θ i , E

Original: [ U i ]

Original: = θ

Original: i , i = 1 , 2 , however, the ratio is usually biased.

Original: To investigate the bias of that ratio we rewrite

Original: [Equation]

Original: If (cid:12)(cid:12)(cid:12) U 2 − θ 2 θ 2 (cid:12)(cid:12)(cid:12) < 1 , we can expand (cid:18) 1 + U

Original: [Equation]

Original: in a geometric series:

Original: [Equation]

Original: If ζ 1 >

Original: [Equation]

Original: a U-statistic U of order m obtained from a sample of n observations converges in distribution

Original: [Shao, 2003]:

Original: [Equation]

Original: [Equation]

Original: Keeping the terms up to Θ (cid:0) 1 n (cid:1) :

Original: [Equation]

Original: [Equation]

Original: To examine the bias, we take the expectation value of this expression:

Original: [Equation]

Original: We now make use of the following expressions:

Original: [Equation]

Original: Using these expressions the expectation of r becomes:

Original: [Equation]

Original: Using Equation (35)

Original: [Equation]

Original: the linearity of covariance and with Var( aX ) =

Original: a 2 Var( X ) we obtain:

Original: [Equation]

Original: C De-biasing of ratios of straight averages

Original: Let X and Y be random variables and let µ X and µ Y be the means of their distributions, respectively.

Original: Consider the problem of finding an unbiased estimator for the ratio of means:

Original: [Equation]

Original: A first approach to estimate this ratio R is to compute the ratio of the sample means: Let ( X 1 , Y 1 ) , ..., ( X n , Y n ) be pairs of i.i.d.

Original: random variables that are jointly distributed:

Original: [Equation]

Original: This, however, is a biased estimator, which can be seen as follows (we follow [Tin, 1965, Ogliore et al., 2011] here):

Original: [Equation]

Original: This has now the form of a converging geometric series.

Original: Thus, if

Original: [Equation]

Original: we can expand

Original: [Equation]

Original: in a geometric series, which is defined as:

Original: [Equation]

Original: In our case we can identify a = R

Original: [Equation]

Original: and b = − ¯ X − µ X µ X

Original: [Equation]

Original: Thus, using the geometric series expansion, we can write:

Original: [Equation]

Original: = R

Original: [Equation]

Original: Neglecting higher order terms Since ¯ X and ¯ Y are U-statistics

Original: [Equation]

Original: we make use of the asymptotic behaviour of U-statistics.

Original: If ζ 1 > 0 , a U-statistics U n of order m obtained from a sample of n observations behaves as n → ∞ like ([Shao, 2003]):

Original: [Equation]

Original: As we seek an estimator that is unbiased up until order n − 2 and since E [ ¯ X ] = µ X

Original: [Equation]

Original: we can neglect all terms of order 5 or higher since for n → ∞ :

Original: [Equation]

Original: [Equation]

Original: Therefore

Original: [Equation]

Original: we obtain:

Original: [Equation]

Original: Identities to compute the terms of the series expansion of r

Original: [Equation]

Original: [Equation]

Original: Bias Using these expressions we can compute the expectation value of r =

Original: ˆ R :

Original: [Equation]

Original: [Equation]

Original: The bias or r = (cid:98) R is defined as:

Original: [Equation]

Original: [Equation]

Original: Therefore an unbiased version of r is:

Original: [Equation]

Original: A corrected version of the estimator r = ˆ R is consequently given by:

Original: [Equation]

Original: In the above equation we again encounter rations of estimators which again might be biased.

Original: Since we want to achieve a second order de-biasing we have to again recurse on the terms that have a O (cid:0) 1 n (cid:1) dependency.

Original: However, we do not have to recurse on the terms that have a O (cid:0) 1 n 2 (cid:1) dependency, since any recursion would increase the power of the n -dependency.

Original: Therefore a debiased estimator up to order O ( n 2 ) is:

Original: [Equation]

Original: 73) where

Original: [Equation]

Original: (cid:92)

Original: [Equation]

Original: [Equation]

Original: D De-biasing of ratios of squared means

Original: Now consider the problem of finding an unbiased estimator for the ratio of the squared means of x and Y :

Original: [Equation]

Original: Both the numerator and denominator of R can separately be estimated by a second order U-statistics, respectively:

Original: [Equation]

Original: The subscript 2 in ¯ X 2 should emphasize that we are dealing with a second order U-statistics here.

Original: Again, the ratio ¯ Y 2 ¯ X 2 ,is a biased estimator.

Original: Using the approach with the converging geometric series and neglecting the higher order terms, we obtain:

Original: [Equation]

Original: [Equation]

Original: Bias Computing E [ r ] using the above identities:

Original: [Equation]

Original: = R

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Var

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: n

Original: [Equation]

Original: nR

Original: [Equation]

Original: n

Original: [Equation]

Original: [Equation]

Original: ( X

Original: [Equation]

Original: Y )

Original: [Equation]

Original: µ 2 X µ

Original: [Equation]

Original: Y (cid:19)(cid:33)

Original: [Equation]

Original: ) n

Original: [Equation]

Original: ) µ

Original: [Equation]

Original: ) µ

Original: [Equation]

Original: (cid:19)

Original: [Equation]

Original: ) n

Original: [Equation]

Original: nR

Original: [Equation]

Original: n

Original: [Equation]

Original: Var( X

Original: [Equation]

Original: µ

Original: [Equation]

Original: X (cid:19)(cid:33)

Original: [Equation]