Original: Better Uncertainty Calibration via Proper Scores for Classiﬁcation and Beyond
Revised: Better Uncertainty Calibration via Proper Scores for Classiﬁcation and Beyond

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: With model trustworthiness being crucial for sensitive real-world applications,practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks.
Revised: With model trustworthiness being crucial for sensitive real-world applications,practitioners are putting more and more focus on improving the uncertainty calibration of deep neural networks.

Original: Calibration errors are designed to quantify thereliability of probabilistic predictions but their estimators are usually biased andinconsistent.
Revised: Calibration errors are designed to quantify thereliability of probabilistic predictions but their estimators are usually biased andinconsistent.

Original: In this work, we introduce the framework of proper calibration errors ,which relates every calibration error to a proper score and provides a respectiveupper bound with optimal estimation properties.
Revised: In this work, we introduce the framework of proper calibration errors ,which relates every calibration error to a proper score and provides a respectiveupper bound with optimal estimation properties.

Original: This relationship can be usedto reliably quantify the model calibration improvement.
Revised: This relationship can be usedto reliably quantify the model calibration improvement.

Original: We theoretically andempirically demonstrate the shortcomings of commonly used estimators comparedto our approach.
Revised: We theoretically andempirically demonstrate the shortcomings of commonly used estimators comparedto our approach.

Original: Due to the wide applicability of proper scores, this gives a naturalextension of recalibration beyond classiﬁcation
Revised: Due to the wide applicability of proper scores, this gives a naturalextension of recalibration beyond classiﬁcation

Original:  Introduction
Revised:  Introduction

Original: Deep learning became a dominant cornerstone of machine learning research in the last decade anddeep neural networks can surpass human-level predictive performance on a wide range of tasks [19,46, 17].
Revised: Deep learning became a dominant cornerstone of machine learning research in the last decade anddeep neural networks can surpass human-level predictive performance on a wide range of tasks [17,44, 15].

Original: However, Guo et al.
Revised: However, Guo et al.

Original: [14] have shown that for modern neural networks, better classiﬁcationaccuracy can come at the cost of systematic overconﬁdence in their predictions.
Revised: [12] have shown that for modern neural networks, better classiﬁcationaccuracy can come at the cost of systematic overconﬁdence in their predictions.

Original: Practitioners insensitive forecasting domains, such as cancer diagnostics [17], genotype-based disease prediction[25] or climate prediction [57], require for models to not only have high predictive power but alsoto reliably communicate uncertainty.
Revised: Practitioners insensitive forecasting domains, such as cancer diagnostics [15], genotype-based disease prediction[23] or climate prediction [54], require for models to not only have high predictive power but alsoto reliably communicate uncertainty.

Original: This raises the need to quantify and improve the qualityof predictive uncertainty, ideally via a dedicated metric.
Revised: This raises the need to quantify and improve the qualityof predictive uncertainty, ideally via a dedicated metric.

Original: An uncertainty-aware model should giveprobabilistic predictions which represent the true likelihood of events depending on the very prediction.
Revised: An uncertainty-aware model should giveprobabilistic predictions which represent the true likelihood of events depending on the very prediction.

Original: To quantify the extend to which this condition is violated, calibration errors have been introduced.
Revised: To quantify the extend to which this condition is violated, calibration errors have been introduced.

Original: In general, their estimators are usually biased [45] and inconsistent [52].
Revised: In general, their estimators are usually biased [43] and inconsistent [49].

Original: This, in turn, is highlyproblematic since we cannot quantify how reliable a model is if we do not know how reliable themetric is.
Revised: This, in turn, is highlyproblematic since we cannot quantify how reliable a model is if we do not know how reliable themetric is.

Original: Especially the medical ﬁeld is a domain that requires high model trustworthiness, but withlow expert availability and/or disease frequency we often encounter a small data regime.
Revised: Especially the medical ﬁeld is a domain that requires high model trustworthiness, but withlow expert availability and/or disease frequency we often encounter a small data regime.

Original: Resamplingstrategies can be viable options for optimization on small datasets but also reduce the evaluation setsize even more.
Revised: Resamplingstrategies can be viable options for optimization on small datasets but also reduce the evaluation setsize even more.

Original: We will discover that little data exacerbates the estimation bias and propose reliablealternatives for improving uncertainty calibration.
Revised: We will discover that little data exacerbates the estimation bias and propose reliablealternatives for improving uncertainty calibration.

Original: Since deep neural networks often yield uncalibrated conﬁdence scores [36], a variety of differentpost-hoc recalibration approaches have been proposed [7, 31].
Revised: Since deep neural networks often yield uncalibrated conﬁdence scores [34], a variety of differentpost-hoc recalibration approaches have been proposed [6, 29].

Original: These methods use the validation setto transform predictions returned by a trained neural network such that they become better calibrated.
Revised: These methods use the validation setto transform predictions returned by a trained neural network such that they become better calibrated.

Original: A key desired property of recalibration methods is to not reduce the accuracy after the transformation.
Revised: A key desired property of recalibration methods is to not reduce the accuracy after the transformation.

Original: Therefore, most modern approaches are restricted to accuracy preserving transformations of themodel outputs [42, 14, 61, 44].
Revised: Therefore, most modern approaches are restricted to accuracy preserving transformations of themodel outputs [40, 12, 58, 42].

Original: When recalibrating a model, it is crucial to have a reliable estimateof how much the chosen method improves the underlying model.
Revised: When recalibrating a model, it is crucial to have a reliable estimateof how much the chosen method improves the underlying model.

Original: However, when using current
Revised: However, when using current

Original: [Figure]
Revised: [Figure]

Original: estimators the are to the calibration error before after
Revised: estimators the are to the calibration error before after

Original: 1; c.f section 5)
Revised: 1; c.f section 5)

Original: Our contributions for better uncertainty calibration are summarized in the following.
Revised: Our contributions for better uncertainty calibration are summarized in the following.

Original: We...
Revised: We...

Original: • ... give an overview of current calibration error literature, place the errors into a taxonomy,
Revised: • ... give an overview of current calibration error literature, place the errors into a taxonomy,

Original: and show which are insufﬁcient for calibration quantiﬁcation.
Revised: and show which are insufﬁcient for calibration quantiﬁcation.

Original: This also includes several
Revised: This also includes several

Original: theoretical results, which highlight the shortcomings of current approaches.
Revised: theoretical results, which highlight the shortcomings of current approaches.

Original: • ... introduce the framework of proper calibration errors , which gives important guarantees
Revised: • ... introduce the framework of proper calibration errors , which gives important guarantees

Original: and relates every element to a proper score.
Revised: and relates every element to a proper score.

Original: We can reliably estimate the improvement of an
Revised: We can reliably estimate the improvement of an

Original: injective recalibration method w.r.t.
Revised: injective recalibration method w.r.t.

Original: a proper calibration error via its related proper score -
Revised: a proper calibration error via its related proper score -

Original: even in non-classiﬁcation settings.
Revised: even in non-classiﬁcation settings.

Original: • ... show that common calibration estimators are highly sensitive w.r.t.
Revised: • ... show that common calibration estimators are highly sensitive w.r.t.

Original: the test set size.
Revised: the test set size.

Original: We
Revised: We

Original: demonstrate that for commonly used estimators, the estimated improvement of recalibration
Revised: demonstrate that for commonly used estimators, the estimated improvement of recalibration

Original: methods is heavily biased and becomes monotonically worse with fewer test data.
Revised: methods is heavily biased and becomes monotonically worse with fewer test data.

Original:  Related Work
Revised:  Related Work

Original: In this section, we give an extensive overview of published work regarding quantifying modelcalibration and model recalibration.
Revised: In this section, we give an extensive overview of published work regarding quantifying modelcalibration and model recalibration.

Original: Important deﬁnitions will be directly given, while others areplaced in Appendix C. These will be the basis for our theoretical ﬁndings.
Revised: Important deﬁnitions will be directly given, while others areplaced in Appendix C. These will be the basis for our theoretical ﬁndings.

Original: Further, we will motivatethe deﬁnition of proper calibration errors, which are directly related to proper scores.
Revised: Further, we will motivatethe deﬁnition of proper calibration errors, which are directly related to proper scores.

Original: Consequently,we will also present important aspects of the framework around proper scores in later parts of thissection.
Revised: Consequently,we will also present important aspects of the framework around proper scores in later parts of thissection.

Original: 2.1 Calibration errors
Revised: 2.1 Calibration errors

Original: For clarity, we introduce shortly the notation used throughout this work.
Revised: For clarity, we introduce shortly the notation used throughout this work.

Original: Assume we have randomvariables X and Y corresponding to feature and target variable, and feature and target space X and
Revised: Assume we have randomvariables X and Y corresponding to feature and target variable, and feature and target space X and

Original: Y .
Revised: Y .

Original: We have P Y , P Y | X 2 P , where P Y refers to the distribution of Y , P Y | X to the conditional 61 distribution given X , and P a set of distributions on Y .
Revised: We have P Y , P Y | X 2 P , where P Y refers to the distribution of Y , P Y | X to the conditional 61 distribution given X , and P a set of distributions on Y .

Original: Even though some approaches explorecalibration for regression tasks [47, 56, 6], it is most dominantly considered for classiﬁcation.
Revised: Even though some approaches explorecalibration for regression tasks [45, 53, 5], it is most dominantly considered for classiﬁcation.

Original: Todistinguish between the general case and n -class classiﬁcation, we refer to P n as the n -dimensionalsimplex of corresponding categorical distributions.
Revised: Todistinguish between the general case and n -class classiﬁcation, we refer to P n as the n -dimensionalsimplex of corresponding categorical distributions.

Original: A popular task is the calibration of the predicted top-label C = arg max k f k ( X ) of a model f : X !
Revised: A popular task is the calibration of the predicted top-label C = arg max k f k ( X ) of a model f : X !

Original: P n [14, 30, 34, 41, 52, 50, 44].
Revised: P n [12, 28, 32, 39, 49, 48, 42].

Original: Here, the top-label conﬁdence should represent the accuracy of thisvery prediction.
Revised: Here, the top-label conﬁdence should represent the accuracy of thisvery prediction.

Original: Formally, we try to reach the condition f C ( X ) =
Revised: Formally, we try to reach the condition f C ( X ) =

Original: P ( Y = C
Revised: P ( Y = C

Original: | f C ( X )) .
Revised: | f C ( X )) .

Original: However,the condition is weaker as one might expect, and referring to a model fulﬁlling this condition as
Revised: However,the condition is weaker as one might expect, and referring to a model fulﬁlling this condition as

Original: (perfectly) calibrated can give a false sense of security [52, 55].
Revised: (perfectly) calibrated can give a false sense of security [49, 52].

Original: This holds especially in forecastingdomains, where low probability estimates can still be highly relevant.
Revised: This holds especially in forecastingdomains, where low probability estimates can still be highly relevant.

Original: For example, assigningprobability mass to an aggressive type of cancer can still trigger action even if it is not predictedas the most likely outcome.
Revised: For example, assigningprobability mass to an aggressive type of cancer can still trigger action even if it is not predictedas the most likely outcome.

Original: Further, we might also be interested in predictive uncertainty for nonclassiﬁcation tasks.
Revised: Further, we might also be interested in predictive uncertainty fornon-classiﬁcation tasks.

Original: Consequently, we use the  stricter and more general condition that the completeprediction f ( X ) should be equal to the complete conditional distribution of the target given this veryprediction P Y | f ( X ) as introduced by Widmann et al.More formally, we state:
Revised: Consequently, we use the following, stricter and more general conditionintroduced by Widmann et al.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: A model f : X !
Revised: A model f : X !

Original: P is calibrated if and only if f ( X ) =
Revised: P is calibrated if and only if f ( X ) =

Original: P Y | f
Revised: P Y | f

Original: ( X ) .
Revised: ( X ) .

Revised: In other words, the condition holds if each prediction equals the full target distribution given this veryprediction.

Original: Note that P can be a set of arbitrary distributions, which incorporates  P n (classiﬁcation) as a specialcase.
Revised: Note that P can be a set of arbitrary distributions, which incorporates the special case P n .

Original: One of the ﬁrst metrics for assessing model calibration that is still widely used in recent literature isthe Brier score (BS) [38, 42, 16, 44].
Revised: One of the ﬁrst metrics for assessing model calibration that is still widely used in recent literature isthe Brier score (BS) [36, 40, 14, 42].

Original: For a model f : X !
Revised: For a model f : X !

Original: P n the Brier score [3] is deﬁned as
Revised: P n the Brier score [2] is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: where Y 0 is one-hot-encoded Y .
Revised: where Y 0 is one-hot-encoded Y .

Original: The estimator of the BS is equivalent to the mean squared error,illustrating that it does not purely capture model calibration.
Revised: The estimator of the BS is equivalent to the mean squared error,illustrating that it does not purely capture model calibration.

Original: Rather, the BS can be interpreted as acomprehensive measure of model performance, simultaneously capturing model ﬁt and calibration.
Revised: Rather, the BS can be interpreted as acomprehensive measure of model performance, simultaneously capturing model ﬁt and calibration.

Original: This becomes more obvious via the canonical decomposition of the BS into a calibration and sharpnessterm [38].
Revised: This becomes more obvious via the canonical decomposition of the BS into a calibration and sharpnessterm [36].

Original: Based on this decomposition, we can derive the following error.
Revised: Based on this decomposition, we can derive the following error.

Original: For 1  p 2 R , the ` pcalibration error (CE p ) of model f : X !
Revised: For 1  p 2 R , the ` pcalibration error (CE p ) of model f : X !

Original: P n is deﬁned as
Revised: P n is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: The BS decomposition only supports the case p = 2 , but a general ` p formulation became morecommon in recent years [52, 55, 61].
Revised: The BS decomposition only supports the case p = 2 , but a general ` p formulation became morecommon in recent years [49, 52, 58].

Original: The CE p is intractable to reliably estimate in the general case.
Revised: The CE p is intractable to reliably estimate in the general case.

Original: This is mostly due to the term P Y | f ( X ) since we never have samples of every possible predictionfor continuous models.
Revised: This is mostly due to the term P Y | f ( X ) since we never have samples of every possible predictionfor continuous models.

Original: This is in contrast to the original work of Murphy [38], where only modelswith a ﬁnite prediction space are considered so that computing the CE 2 does not require a specialprocedure.
Revised: This is in contrast to the original work of Murphy [36], where only modelswith a ﬁnite prediction space are considered so that computing the CE 2 does not require a specialprocedure.

Original: To assess the calibration of a continuous binary model Platt [42] used histogram estimation,transforming the inﬁnite prediction space to a ﬁnite one.
Revised: To assess the calibration of a continuous binary model Platt [40] used histogram estimation,transforming the inﬁnite prediction space to a ﬁnite one.

Original: This is also referred to as equal width binning.
Revised: This is also referred to as equal width binning.

Original: Similarly, Nguyen & O’Connor [40] introduced an equal mass binning scheme for continuous binarymodels.
Revised: Similarly, Nguyen & O’Connor [38] introduced an equal mass binning scheme for continuous binarymodels.

Original: Both, equal width and equal mass binning schemes, suffer from the requirement of settinga hyperparameter.
Revised: Both, equal width and equal mass binning schemes, suffer from the requirement of settinga hyperparameter.

Original: This can signiﬁcantly inﬂuence the estimated value [31] and there is no optimaldefault since every setting has a different bias-variance tradeoff [41].
Revised: This can signiﬁcantly inﬂuence the estimated value [29] and there is no optimaldefault since every setting has a different bias-variance tradeoff [39].

Original: The ﬁrst calibration estimatorfor a continuous one-vs-all multi-class model was given by Naeini et al.
Revised: The ﬁrst calibration estimatorfor a continuous one-vs-all multi-class model was given by Naeini et al.

Original: [39] and is still the mostcommonly used measure to quantify calibration.
Revised: [37] and is still the mostcommonly used measure to quantify calibration.

Original: It is referred to as the expected calibration error(ECE) of model f : X !
Revised: It is referred to as the expected calibration error(ECE) of model f : X !

Original: P n and deﬁned as
Revised: P n and deﬁned as

Original: [Equation]
Revised: [Equation]

Original: with the bin frequency p i
Revised: with the bin frequency p i

Original: =
Revised: =

Original: P ( f C ( X ) 2 B i ) , the bin-wise mean conﬁdence conf i =
Revised: P ( f C ( X ) 2 B i ) , the bin-wise mean conﬁdence conf i =

Original: E [ f C ( X ) | f C ( X ) 2 B i ] , and the bin-wise accuracy acc
Revised: E [ f C ( X ) | f C ( X ) 2 B i ] , and the bin-wise accuracy acc

Original: i
Revised: i

Original: =
Revised: =

Original: P ( Y = C
Revised: P ( Y = C

Original: | f C ( X ) 2 B i ) , for m 2 Nbins B i :=
Revised: | f C ( X ) 2 B i ) , for m 2 Nbins B i :=

Original:  i  1 m , im ⇤ .
Revised:  i  1 m , im ⇤ .

Original: We can reliably estimate p i , conf i , and acc i via the respective means.
Revised: We can reliably estimate p i , conf i , and acc i via the respective means.

Original: Theseare then ’plugged-in’ to estimate the ECE.
Revised: Theseare then ’plugged-in’ to estimate the ECE.

Original: Kull et al.
Revised: Kull et al.

Original: [29] and Nixon et al.
Revised: [27] and Nixon et al.

Original: [41] independently introduced another calibration estimator, whichalso captures the extend to which the condition P ( Y = k | f k ( X ))
Revised: [39] independently introduced another calibration estimator, whichalso captures the extend to which the condition P ( Y = k | f k ( X ))

Original: =
Revised: =

Original: f k ( X ) is violated for eachclass k 2 Y .
Revised: f k ( X ) is violated for eachclass k 2 Y .

Original: They respectively use equal width and equal mass binning.
Revised: They respectively use equal width and equal mass binning.

Original: Similar to these estimators,
Revised: Similar to these estimators,

Original: Kumar et al.
Revised: Kumar et al.

Original: [31] introduced the class-wise calibration error (CWCE p ) and, similar to the ECE, thetop-label calibration error (TCE p ).
Revised: [29] introduced the class-wise calibration error (CWCE p ) and, similar to the ECE, thetop-label calibration error (TCE p ).

Original: They only formulated the squared case p = 2 but suggested theextension of their deﬁnitions to general p -norms, which we provide in Appendix C.
Revised: They only formulated the squared case p = 2 but suggested theextension of their deﬁnitions to general p -norms, which we provide in Appendix C.

Original: Furthermore, Kumar et al.
Revised: Furthermore, Kumar et al.

Original: [31] and Vaicenavicius et al.
Revised: [29] and Vaicenavicius et al.

Original: [52] proved independently that using aﬁxed binning scheme for estimation leads to a lower bound of the expected error.
Revised: [49] proved independently that using aﬁxed binning scheme for estimation leads to a lower bound of the expected error.

Original: Zhang et al.
Revised: Zhang et al.

Original: [61]circumvent binning schemes by using kernel density estimation to estimate the TCE p .
Revised: [58]circumvent binning schemes by using kernel density estimation to estimate the TCE p .

Original: Orthogonal ways to quantify model miscalibration have been proposed to not depend on binningor kernel density estimation schemes.
Revised: Orthogonal ways to quantify model miscalibration have been proposed to not depend on binningor kernel density estimation schemes.

Original: Gupta et al.
Revised: Gupta et al.

Original: [16] introduced the Kolmogorov-Smirnov calibration error (KS)
Revised: [14] introduced the Kolmogorov-Smirnov calibration error (KS)

Original: (c.f. Appendix C), which is based on the Kolmogorov-Smirnov test betweenempirical cumulative distribution functions.
Revised: (c.f. Appendix C), which is based on the Kolmogorov-Smirnov test betweenempirical cumulative distribution functions.

Original: Its estimator does not require setting a hyperparameterbut the authors did not provide further theoretical aspects.
Revised: Its estimator does not require setting a hyperparameterbut the authors did not provide further theoretical aspects.

Original: Estimators of the TCE p and CWCE p are in general not differentiable.
Revised: Estimators of the TCE p and CWCE p are in general not differentiable.

Original: Consequently, Kumar et al.[30] proposed the Maximum mean calibration error (MMCE) (c.f. Appendix C), which hasa differentiable estimator.
Revised: Consequently, Kumar et al.[28] proposed the Maximum mean calibration error (MMCE) (c.f. Appendix C), which hasa differentiable estimator.

Original: It is a kernel-based error, comparing the top-label conﬁdence and theconditional accuracy, similar to the ECE.
Revised: It is a kernel-based error, comparing the top-label conﬁdence and theconditional accuracy, similar to the ECE.

Original: Widmann et al.
Revised: Widmann et al.

Original: [55] argued that the MMCE is insufﬁcient for quantifying calibration of a model,similar as the ECE.
Revised: [52] argued that the MMCE is insufﬁcient for quantifying calibration of a model,similar as the ECE.

Original: They further proposed the Kernel calibration error (KCE)
Revised: They further proposed the Kernel calibration error (KCE)

Original: (c.f. Appendix C).
Revised: (c.f. Appendix C).

Original: Itis based on matrix-valued kernels and unlike the MMCE, which only uses the top-label prediction,includes the whole model prediction.
Revised: Itis based on matrix-valued kernels and unlike the MMCE, which only uses the top-label prediction,includes the whole model prediction.

Original: The squared KCE has an unbiased estimator based on a
Revised: The squared KCE has an unbiased estimator based on a

Original: U-statistic with quadratic runtime complexity with respect to the data size.
Revised: U-statistic with quadratic runtime complexity with respect to the data size.

Original: However, even though the
Revised: However, even though the

Original: KCE is positive, the U-statistic estimator can give negative values.
Revised: KCE is positive, the U-statistic estimator can give negative values.

Original: To this end, the authors use theestimated value as a test statistic w.r.t.
Revised: To this end, the authors use theestimated value as a test statistic w.r.t.

Original: the null hypothesis that the model is calibrated.
Revised: the null hypothesis that the model is calibrated.

Original: Furthermore, Widmann et al.
Revised: Furthermore, Widmann et al.

Original: [55] and Widmann et al.
Revised: [52] and Widmann et al.

Original: [56] proposed to unify different deﬁnitionsof calibration errors in a theoretical framework, which also includes variance regression calibration.
Revised: [53] proposed to unify different deﬁnitionsof calibration errors in a theoretical framework, which also includes variance regression calibration.

Original: However, the framework allows calibration errors, which are zero even if the model is not calibratedat all.
Revised: However, the framework allows calibration errors, which are zero even if the model is not calibratedat all.

Original: 2.2 Recalibration
Revised: 2.2 Recalibration

Original: A plethora of recalibration methods have been proposed to improve model calibration after trainingby transforming the model output probabilities [18, 42, 58, 59, 39, 40, 28, 14, 29, 31, 61, 16, 44].
Revised: A plethora of recalibration methods have been proposed to improve model calibration after trainingby transforming the model output probabilities [16, 40, 55, 56, 37, 38, 26, 12, 27, 29, 58, 14, 42].

Original: These methods are optimized on a speciﬁc calibration set, which is usually the validation set.
Revised: These methods are optimized on a speciﬁc calibration set, which is usually the validation set.

Original: Keydesiderata of these methods include for the algorithms to be accuracy-preserving and data-efﬁcient[61], reﬂecting that typical use-cases include settings in sensitive domains where accuracy shouldremain unchanged and often little data is available to train and evaluate the models.
Revised: Keydesiderata of these methods include for the algorithms to be accuracy-preserving and data-efﬁcient[58], reﬂecting that typical use-cases include settings in sensitive domains where accuracy shouldremain unchanged and often little data is available to train and evaluate the models.

Original: Such accuracypreserving methods only adjust the probability estimate in such a way that the predicted top-labelremains the same.
Revised: Such accuracypreserving methods only adjust the probability estimate in such a way that the predicted top-labelremains the same.

Original: The most commonly used accuracy-preserving recalibration method is temperaturescaling (TS) [14], where the model logits are divided by a single parameter T 2 R > 0 beforecomputing the predictions via softmax.
Revised: The most commonly used accuracy-preserving recalibration method is temperaturescaling (TS) [12], where the model logits are divided by a single parameter T 2 R > 0 beforecomputing the predictions via softmax.

Original: A more expressive extension of TS is ensemble temperaturescaling (ETS) [61], where a weighted ensemble of TS output, model output, and label smoothingis computed.
Revised: A more expressive extension of TS is ensemble temperaturescaling (ETS) [58], where a weighted ensemble of TS output, model output, and label smoothingis computed.

Original: Rahimi et al.
Revised: Rahimi et al.

Original: [44] proposed different classes of order-preserving transformations.
Revised: [42] proposed different classes of order-preserving transformations.

Original: Aspeciﬁcally interesting one is the class of diagonal intra order-preserving functions (DIAG).
Revised: Aspeciﬁcally interesting one is the class of diagonal intra order-preserving functions (DIAG).

Original: Here, themodel logits are transformed elementwise with a scalar, monotonic, and continuous function, whichis represented by neural networks of unconstrained monotonic functions [54].
Revised: Here, themodel logits are transformed elementwise with a scalar, monotonic, and continuous function, whichis represented by neural networks of unconstrained monotonic functions [51].

Original: 2.3 Proper scores
Revised: 2.3 Proper scores

Original: Gneiting & Raftery [13] give an extensive overview of proper scores.
Revised: Gneiting & Raftery [11] give an extensive overview of proper scores.

Original: Unfortunately, their presenteddeﬁnitions assume maximization as the model training objective.
Revised: Unfortunately, their presenteddeﬁnitions assume maximization as the model training objective.

Original: To stay in line with recent machinelearning literature, we ﬂip the sign when it is required in the following deﬁnitions, similar as in [4].
Revised: To stay in line with recent machinelearning literature, we ﬂip the sign when it is required in the following deﬁnitions, similar as in [3].

Original: We speciﬁcally do not constraint ourselves to classiﬁcation, which is a special case.
Revised: We speciﬁcally do not constraint ourselves to classiﬁcation, which is a special case.

Original: Assume we givea prediction in P for an event in Y and we want to score how good the prediction was.
Revised: Assume we givea prediction in P for an event in Y and we want to score how good the prediction was.

Original: A function
Revised: A function

Original: S : P ⇥ Y !
Revised: S : P ⇥ Y !

Original: R with R := R [ { 1 , 1 } is called scoring rule or just score .
Revised: R with R := R [ { 1 , 1 } is called scoring rule or just score .

Original: Examples are the 157 Brier score and the log score for classiﬁcation, or the Dawid-Sebastiani score (DSS), which extendsthe MSE for variance regression [8, 12].
Revised: Examples are the 157 Brier score and the log score for classiﬁcation, or the Dawid-Sebastiani score (DSS), which extendsthe MSE for variance regression [7, 10].

Original: It is deﬁned as S ( P, y ) =
Revised: It is deﬁned as S ( P, y ) =

Original: ( µ P  y ) 2  2 P + log  2 P .
Revised: ( µ P  y ) 2  2 P + log  2 P .

Original: To comparedistributions, we use the expected score s S : P ⇥ P !
Revised: To comparedistributions, we use the expected score s S : P ⇥ P !

Original: R deﬁned as s S ( P, Q ) =
Revised: R deﬁned as s S ( P, Q ) =

Original: E Y ⇠ Q [ S ( P, Y )] .
Revised: E Y ⇠ Q [ S ( P, Y )] .

Original: A scoring rule S is deﬁned to be proper if and only if s S ( P, Q )
Revised: A scoring rule S is deﬁned to be proper if and only if s S ( P, Q )

Original:  s S ( Q, Q ) holds for all P, Q 2 P ,and strictly proper if and only if P 6 = Q = )
Revised:  s S ( Q, Q ) holds for all P, Q 2 P ,and strictly proper if and only if P 6 = Q = )

Original: s S ( P, Q ) >
Revised: s S ( P, Q ) >

Original: s S ( Q, Q ) .
Revised: s S ( Q, Q ) .

Original: In other words, a scoreis proper if predicting the target distribution gives the best expected value and strictly proper ifno other prediction can achieve this value.
Revised: In other words, a scoreis proper if predicting the target distribution gives the best expected value and strictly proper ifno other prediction can achieve this value.

Original: Given a proper score S and P, Q 2 P , the associateddivergence d S : P ⇥ P !
Revised: Given a proper score S and P, Q 2 P , the associateddivergence d S : P ⇥ P !

Original: R  0 is deﬁned as d S ( P, Q ) =
Revised: R  0 is deﬁned as d S ( P, Q ) =

Original: s S
Revised: s S

Original: ( P, Q )  s S ( Q, Q ) and the associatedgeneralized entropy g S : P !
Revised: ( P, Q )  s S ( Q, Q ) and the associatedgeneralized entropy g S : P !

Original: R as g S ( Q ) =
Revised: R as g S ( Q ) =

Original: s S ( Q, Q ) .
Revised: s S ( Q, Q ) .

Original: For strictly proper S , d S is only zeroif P = Q ; for (strictly) proper S , g S is (strictly) concave.
Revised: For strictly proper S , d S is only zeroif P = Q ; for (strictly) proper S , g S is (strictly) concave.

Original: An example of a divergence-entropycombination is the Kullback-Leibler divergence and the Shannon entropy associated to the log score.
Revised: An example of a divergence-entropycombination is the Kullback-Leibler divergence and the Shannon entropy associated to the log score.

Original: Associated entropies and divergences are used in the calibration-sharpness decomposition introducedby Bröcker [4] for proper scores of categorical distributions.
Revised: Associated entropies and divergences are used in the calibration-sharpness decomposition introducedby Bröcker [3] for proper scores of categorical distributions.

Original: In Lemma 4.1 we will prove that thisresult holds for proper scores of arbitrary distributions, as long as additional conditions are met.
Revised: In Lemma 4.1 we will prove that thisresult holds for proper scores of arbitrary distributions, as long as additional conditions are met.

Original: Further, associated divergences will be the backbone for our deﬁnition of proper calibration errors in
Revised: Further, associated divergences will be the backbone for our deﬁnition of proper calibration errors in

Original: Section 4.
Revised: Section 4.

Original:  Theoretical analysis of calibration errors
Revised:  Theoretical analysis of calibration errors

Original: In this section, we present theoretical results regarding calibration errors used in current literature.
Revised: In this section, we present theoretical results regarding calibration errors used in current literature.

Original: First, we place these calibration errors into a taxonomy, which we introduce in Theorem 3.1.
Revised: First, we place these calibration errors into a taxonomy, which we introduce in Theorem 3.1.

Original: Next,we give an example of how much errors lower in the hierarchy can differ from CE 2 in Proposition3.2.
Revised: Next,we give an example of how much errors lower in the hierarchy can differ from CE 2 in Proposition3.2.

Original: We illustrate the practical relevance by introducing a perfect calibration method according tomost errors in Proposition D.2, which fails to be calibrated.
Revised: We illustrate the practical relevance by introducing a perfect calibration method according tomost errors in Proposition D.2, which fails to be calibrated.

Original: Last, we analyse the ECE estimate withrespect to the data size in Proposition 3.3.
Revised: Last, we analyse the ECE estimate withrespect to the data size in Proposition 3.3.

Original: This proposition is the basis for explaining the empirical(miss-)behaviour of the ECE observed in Section 5.
Revised: This proposition is the basis for explaining the empirical(miss-)behaviour of the ECE observed in Section 5.

Original: All proofs are presented in Appendix D.
Revised: All proofs are presented in Appendix D.

Original: To the best of the authors’ knowledge, other publications provided relations between at most twodistinct calibration errors or none at all while introducing a new one.
Revised: To the best of the authors’ knowledge, other publications provided relations between at most twodistinct calibration errors or none at all while introducing a new one.

Original: The taxonomy in the followingtheorem is a novel contribution, improving overview of a whole body of work regarding quantifyingcalibration.
Revised: The taxonomy in the followingtheorem is a novel contribution, improving overview of a whole body of work regarding quantifyingcalibration.

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: Given a model f : X !
Revised: Given a model f : X !

Original: P n and 1  p 2 R , we have
Revised: P n and 1  p 2 R , we have

Original: [Equation]
Revised: [Equation]

Original: where statements inside curly brackets { . . . } are equivalent.
Revised: where statements inside curly brackets { . . . } are equivalent.

Original: Further, we have
Revised: Further, we have

Original: [Equation]
Revised: [Equation]

Original: where * BS is only included for p = 2 .
Revised: where * BS is only included for p = 2 .

Original: The kernel dependent constant c 2 R is given in Appendix
Revised: The kernel dependent constant c 2 R is given in Appendix

Original: D.2 according to Kumar et al.
Revised: D.2 according to Kumar et al.

Original: From this theorem follows that it is ambiguous to refer to perfect calibration just because there existsa calibration error which is zero for a model.
Revised: From this theorem follows that it is ambiguous to refer to perfect calibration just because there existsa calibration error which is zero for a model.

Original: Now, we further demonstrate how large the gap betweencalibration errors can be.
Revised: Now, we further demonstrate how large the gap betweencalibration errors can be.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Assume the model f : X !
Revised: Assume the model f : X !

Original: P n is surjective.
Revised: P n is surjective.

Original: There exists a joint distribution
Revised: There exists a joint distribution

Original: P X,Y such that for all E 2 { MMCE , KS , ECE , TCE p , CWCE p | 1  p 2 R } :
Revised: P X,Y such that for all E 2 { MMCE , KS , ECE , TCE p , CWCE p | 1  p 2 R } :

Original: [Equation]
Revised: [Equation]

Original: Recall that CE 2 ( f ) =
Revised: Recall that CE 2 ( f ) =

Original: if and only if f is calibrated.
Revised: if and only if f is calibrated.

Original: In other words, most used calibration errorscan be zero, but the model is still far from calibrated.
Revised: In other words, most used calibration errorscan be zero, but the model is still far from calibrated.

Original: An example of a model transformation withperfect ECE but uncalibrated outputs is given in Proposition D.2.
Revised: An example of a model transformation withperfect ECE but uncalibrated outputs is given in Proposition D.2.

Original: Among all calibration errors, the ECE is still the most commonly used [24, 26, 44, 50, 36, 49, 23, 35,37, 15, 53, 10], even though its estimator is knowingly biased [31, 45].
Revised: Among all calibration errors, the ECE is still the most commonly used [22, 24, 42, 48, 34, 47, 21, 33,35, 13, 50, 9], even though its estimator is knowingly biased [29, 43].

Original: Let ˆ ECE ( n ) denote the ECEestimator for n data instances as originally deﬁned in Guo et al.
Revised: Let ˆ ECE ( n ) denote the ECEestimator for n data instances as originally deﬁned in Guo et al.

Original: The following gives an analysishow this estimate behaves approximately with respect to n and how this is further impacted by theground truth ECE value.
Revised: The following gives an analysishow this estimate behaves approximately with respect to n and how this is further impacted by theground truth ECE value.

Original: For simplicity, we omit the ﬁxed model from the notation.
Revised: For simplicity, we omit the ﬁxed model from the notation.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: For µ ( n ) deﬁned in Appendix D.5, we have
Revised: For µ ( n ) deﬁned in Appendix D.5, we have

Original: [Equation]
Revised: [Equation]

Original: In words, the ECE estimator can be approximated by a function,which is strictly convex and monotonically decreasing in thedata size.
Revised: In words, the ECE estimator can be approximated by a function,which is strictly convex and monotonically decreasing in thedata size.

Original: The smaller the data size, the larger the change of thefunction.
Revised: The smaller the data size, the larger the change of thefunction.

Original: Further, this change is also inﬂuenced by the true ECEvalue, such that, for low ground truth ECE, the function changeseven more rapidly with the data size.
Revised: Further, this change is also inﬂuenced by the true ECEvalue, such that, for low ground truth ECE, the function changeseven more rapidly with the data size.

Original: Analogous results can befound for CWCE 1 , CWCE 2 , and TCE 2 with binning estimatorsas used in Kull et al.
Revised: Analogous results can befound for CWCE 1 , CWCE 2 , and TCE 2 with binning estimatorsas used in Kull et al.

Original: [29], Nixon et al.
Revised: [27], Nixon et al.

Original: [41] and Kumar et al.[31].
Revised: [39] and Kumar et al.[29].

Original: To conﬁrm the goodness of the approximation, we evaluated the
Revised: To conﬁrm the goodness of the approximation, we evaluated the

Original: ECE estimator on simulated models in Figure 2.
Revised: ECE estimator on simulated models in Figure 2.

Original: The modelsare designed such that their true level of calibration is known.
Revised: The modelsare designed such that their true level of calibration is known.

Original: Including the results of Figure 1, the empirical curves are consistent with Proposition 3.3.
Revised: Including the results of Figure 1, the empirical curves are consistent with Proposition 3.3.

Original: Further details on the simulationare given in Appendix G.
Revised: Further details on the simulationare given in Appendix G.

Original: The results in this section motivate a formal framework ofproper calibration errors which are zero if and only if the modelis calibrated and with robust estimators.
Revised: The results in this section motivate a formal framework ofproper calibration errors which are zero if and only if the modelis calibrated and with robust estimators.

Original:  Proper calibration errors
Revised:  Proper calibration errors

Original: In this section, we introduce the deﬁnition of proper calibrationerrors .
Revised: In this section, we introduce the deﬁnition of proper calibrationerrors .

Original: We provide an easy-to-estimate upper bound and investigate some properties.
Revised: We provide an easy-to-estimate upper bound and investigate some properties.

Original: As a preliminary step, we generalize aproper score decomposition.
Revised: As a preliminary step, we generalize aproper score decomposition.

Original: Again, all proofs are presented in Appendix D.
Revised: Again, all proofs are presented in Appendix D.

Original: Bröcker [4] introduced a calibration-sharpness decomposition of proper scores w.r.t.
Revised: Bröcker [3] introduced a calibration-sharpness decomposition of proper scores w.r.t.

Original: categoricaldistributions.
Revised: categoricaldistributions.

Original: We extend this decomposition to proper scores of arbitrary distributions.
Revised: We extend this decomposition to proper scores of arbitrary distributions.

Original: Lemma 4.1.
Revised: Lemma 4.1.

Original: Let P be a set of arbitrary distributions for which exists a proper score S with somemild conditions.
Revised: Let P be a set of arbitrary distributions for which exists a proper score S with somemild conditions.

Original: For random variables Q and Y with Q, P Y , P Y | Q 2 P , we have the decomposition
Revised: For random variables Q and Y with Q, P Y , P Y | Q 2 P , we have the decomposition

Original: E [ S ( Q, Y )]
Revised: E [ S ( Q, Y )]

Original: = g S
Revised: = g S

Original: [Equation]
Revised: [Equation]

Original: P Y
Revised: P Y

Original: [Equation]
Revised: [Equation]

Original: | {z } generalized entropy 
Revised: | {z } generalized entropy

Original: E ⇥
Revised: E ⇥

Original: [Equation]
Revised: [Equation]

Original: 
Revised:

Original: [Equation]
Revised: [Equation]

Original: ⇤ | {
Revised: ⇤ | {

Original: [Equation]
Revised: [Equation]

Original: } sharpness
Revised: } sharpness

Original: [Equation]
Revised: [Equation]

Original: ⇤ | {z } calibration .
Revised: ⇤ | {z } calibration .

Original: Substituting Q with f ( X ) and S with the Brier score, the calibration term equals the previouslydeﬁned CE 2 of a model f .
Revised: Substituting Q with f ( X ) and S with the Brier score, the calibration term equals the previouslydeﬁned CE 2 of a model f .

Original: Lemma 4.1 motivates the following deﬁnition, which we introduce:
Revised: Lemma 4.1 motivates the following deﬁnition, which we introduce:

Original: Deﬁnition 4.2.
Revised: Deﬁnition 4.2.

Original: Given a model f : X !
Revised: Given a model f : X !

Original: P , we say
Revised: P , we say

Original: [Equation]
Revised: [Equation]

Original: ⇤ is a (strictly) proper calibration error
Revised: ⇤ is a (strictly) proper calibration error

Original: if and only if d S is a divergence associated with a (strictly)proper score S .
Revised: if and only if d S is a divergence associated with a (strictly)proper score S .

Original: This gives CE BS ⌘ CE 2 as an example of a strictly proper calibration error for classiﬁcation since the
Revised: This gives CE BS ⌘ CE 2 as an example of a strictly proper calibration error for classiﬁcation since the

Original: Brier score is a strictly proper score on P n .
Revised: Brier score is a strictly proper score on P n .

Original: Strictly proper calibration errors have the highly desiredproperty: CE S ( f ) = 

Revised: Strictly proper calibration errors have the highly desiredproperty: CE S ( f ) =

Original: a.s. () f is calibrated.
Revised: a.s. () f is calibrated.

Original: Since proper scores are not restricted to classiﬁcation,the above deﬁnition gives a natural extension of calibration errors beyond classiﬁcation.
Revised: Since proper scores are not restricted to classiﬁcation,the above deﬁnition gives a natural extension of calibration errors beyond classiﬁcation.

Original: Additionally, by generalizing the deﬁnition of proper scores, we can show that the squared KCE isa strictly proper calibration error (Appendix F).
Revised: Additionally, by generalizing the deﬁnition of proper scores, we can show that the squared KCE isa strictly proper calibration error (Appendix F).

Original: But, in general, there does not exist an unbiasedestimator of a proper calibration error, since we cannot estimate E ⇥ g S 
Revised: But, in general, there does not exist an unbiasedestimator of a proper calibration error, since we cannot estimate E ⇥ g S

Original: P Y | f
Revised: P Y | f

Original: ( X ) ⇤ in an unbiasedmanner.
Revised: ( X ) ⇤ in an unbiasedmanner.

Original: Because we do not want lower bounds for errors used in sensitive applications, we introducethe following theorem about how to construct an upper bound.
Revised: Because we do not want lower bounds for errors used in sensitive applications, we introducethe following theorem about how to construct an upper bound.

Original: Theorem 4.3.
Revised: Theorem 4.3.

Original: For all proper calibration errors with inf P 2 P g S ( P ) 2 R , there exists an associated calibration upper bound
Revised: For all proper calibration errors with inf P 2 P g S ( P ) 2 R , there exists an associated calibration upper bound

Original: [Equation]
Revised: [Equation]

Original: deﬁned as U S ( f ) =
Revised: deﬁned as U S ( f ) =

Original: E [ S ( f ( X ) , Y )]
Revised: E [ S ( f ( X ) , Y )]

Original:  inf P 2 P g S ( P ) .
Revised:  inf P 2 P g S ( P ) .

Original: Under a classiﬁcation setting and furthermild conditions, we have lim ACC ( f ) !
Revised: Under a classiﬁcation setting and furthermild conditions, we have lim ACC ( f ) !

Original: 1 U S ( f )  CE S ( f ) = 0 .
Revised: 1 U S ( f )  CE S ( f ) = 0 .

Original: In other words, we can always construct a non-trivial upper bound of a proper calibration erroras long as the generalized entropy function has a ﬁnite inﬁmum.
Revised: In other words, we can always construct a non-trivial upper bound of a proper calibration erroras long as the generalized entropy function has a ﬁnite inﬁmum.

Original: The calibration upper boundapproaches the true calibration error for models with high accuracy.
Revised: The calibration upper boundapproaches the true calibration error for models with high accuracy.

Original: For the Brier score we haveinf P 2 P g BS ( P )= 0 , which gives U BS ( f ) = E
Revised: For the Brier score we haveinf P 2 P g BS ( P )

Original: [Equation] i . This makes the Brier score itselfan upper bound of its induced calibration error.
Revised: = 0 , making the Brier score itself an upper bound of its induced calibration error.

Original: Our proposed calibration upper bounds are provably reliable to use since they all have a minimumvariance unbiased estimator.
Revised: Our proposed calibration upper bounds are provably reliable to use since they all have a minimumvariance unbiased estimator.

Original: Combining Theorem 3.1 and 4.3 gives
Revised: Theorem 3.1 states

Original: [Equation]
Revised: [Equation]

Original:  254 ECE ( f ) .
Revised: f ) .

Original: Consequently, we might also be interested in transforming the estimated upper boundin a non-linear way for interpretability purposes.
Revised: Consequently,we might also be interested in transforming the estimated upper bound in a non-linear way forinterpretability purposes.

Original: In general, any unbiased estimator ˆ ✓ becomesbiased after a non-linear transformation t , since Eh t ⇣ ˆ ✓ ⌘i= t ⇣
Revised: In general, any unbiased estimator ˆ ✓ becomes biased after a non-lineartransformation t , since E

Original: E h ˆ ✓ i⌘ .
Revised: [Equation]

Original: But, if t is continuous,our estimator is still asymptotically unbiased and consistent [48].
Revised: But, if t is continuous, our estimator is still asymptotically unbiased and consistent [46].

Original: 1 We will further investigate theempirical robustness w.r.t.
Revised: 1 We will further investigate the empirical robustness w.r.t.

Original: data size in Section 5 with t as the square root.
Revised: datasize in Section 5 with t as the square root.

Original: Furthermore, U S has the following properties, which are helpful for the application of recalibrationmethod optimization and selection.
Revised: Furthermore, U S has the following properties, which are helpful for the application of recalibrationmethod optimization and selection.

Original: Proposition 4.4.
Revised: Proposition 4.4.

Original: Given injective functions h, h 0 : P !
Revised: Given injective functions h, h 0 : P !

Original: P we have
Revised: P we have

Original: [Equation]
Revised: [Equation]

Original: and (assuming S is differentiable)
Revised: and (assuming S is differentiable)

Original: [Equation]
Revised: [Equation]

Original: This is a generalization of Proposition 4.2 presented in [61].
Revised: This is a generalization of Proposition 4.2 presented in [58].

Original: It tells us that we can reliably estimatethe improvement of any injective recalibration method via the upper bound.
Revised: It tells us that we can reliably estimatethe improvement of any injective recalibration method via the upper bound.

Original: Furthermore, we getaccess to the calibration gradient and can compare different transformations.
Revised: Furthermore, we getaccess to the calibration gradient and can compare different transformations.

Original: At ﬁrst, injectivityseems like a signiﬁcant restriction.
Revised: At ﬁrst, injectivityseems like a signiﬁcant restriction.

Original: But, we argue in the following that injectivity - rather than beingaccuracy-preserving - is a desired property of general recalibration methods.
Revised: But, we argue in the following that injectivity - rather than beingaccuracy-preserving - is a desired property of general recalibration methods.

Original: For example, we canconstruct a recalibration method, which is calibrated and accuracy-preserving, but only predictsa ﬁnite set of distinct values (see Appendix E).
Revised: For example, we canconstruct a recalibration method, which is calibrated and accuracy-preserving, but only predictsa ﬁnite set of distinct values (see Appendix E).

Original: Speciﬁcally, we would only predict two distinctvalues for any input in binary classiﬁcation.
Revised: Speciﬁcally, we would only predict two distinctvalues for any input in binary classiﬁcation.

Original: To exclude such naive solutions which substantiallyreduce model sharpness, we restrict ourselves to injective transformations of P n !
Revised: To exclude such naive solutions which substantiallyreduce model sharpness, we restrict ourselves to injective transformations of P n !

Original: P n .
Revised: P n .

Original: These doprovably not impact the model sharpness and preserve, at least partly, the continuity of the outputspace.
Revised: These doprovably not impact the model sharpness and preserve, at least partly, the continuity of the outputspace.

Original: Examples of injective transformations are TS, ETS, and DIAG.
Revised: Examples of injective transformations are TS, ETS, and DIAG.

Original: These state-of-the-art methodsshow very competitive performances even when compared to non-injective recalibration methods[61, 44].
Revised: These state-of-the-art methodsshow very competitive performances even when compared to non-injective recalibration methods[58, 42].

Original: Further, Proposition 4.4 also holds when replacing U S with the expected score s S withoutfurther conditions.
Revised: Further, Proposition 4.4 also holds when replacing U S with the expected score s S withoutfurther conditions.

Original: This is useful when U S does not exist, but we still want to perform recalibrationas in Section 5 in the case of the DSS.
Revised: This is useful when U S does not exist, but we still want to perform recalibrationas in Section 5 in the case of the DSS.

Original:  Experiments
Revised:  Experiments

Original: In the following, we investigate the behavior of calibration error estimators in three settings.
Revised: In the following, we investigate the behavior of calibration error estimators in three settings.

Original: First, we use varying test set sizes for the estimators and compare their values.
Revised: First, we use varying test set sizes for the estimators and compare their values.

Original: This will showhow well the inequalities in Theorem 3.1 hold in practical settings and how robust the estimatorsare.
Revised: This will showhow well the inequalities in Theorem 3.1 hold in practical settings and how robust the estimatorsare.

Original: Second, we explore what the estimated improvements of several recalibration methods are.
Revised: Second, we explore what the estimated improvements of several recalibration methods are.

Original: This is done after the recalibration methods are already optimized on a given validation set; weonly vary the size of the test set and compute calibration errors on these test sets before and afterrecalibration.
Revised: This is done after the recalibration methods are already optimized on a given validation set; weonly vary the size of the test set and compute calibration errors on these test sets before and afterrecalibration.

Original: In both settings, the straighter a line is, the more robust and, consequently, trustworthyis the estimator for practical applications.
Revised: In both settings, the straighter a line is, the more robust and, consequently, trustworthyis the estimator for practical applications.

Original: Third, we investigate how our framework can be usedto improve calibration for tasks beyond classiﬁcation by performing probabilistic regression withsubsequent recalibration.
Revised: Third, we investigate how our framework can be usedto improve calibration for tasks beyond classiﬁcation by performing probabilistic regression withsubsequent recalibration.

Original: In all experiments we evaluate the following estimators: CWCE 2 with 15 equal width bins (’15b
Revised: In all experiments we evaluate the following estimators: CWCE 2 with 15 equal width bins (’15b

Original: CWCE 2 ’), CWCE 2 with 100 equal width bins (’100b CWCE 2 ’), ECE with 15 equal width bins (’ECE’), TCE 2 with 100 equal width bins (’100b TCE 2 ’), TCE 2 with 15 equal mass bins and debiasterm (’15b d TCE 2 ’), TCE 2 with kernel density estimation (’KDE TCE 2 ’), KS (’KS’) and the rootcalibration upper bound p U BS (’RBS (ours)’).The bin amounts are chosen based on past literature[31, 36].
Revised: CWCE 2 ’), CWCE 2 with 100 equal width bins (’100b CWCE 2 ’), ECE with 15 equal width bins(’ECE’), TCE 2 with 100 equal width bins (’100b TCE 2 ’), TCE 2 with 15 equal mass bins and debias term (’15b d TCE 2 ’), TCE 2 with kernel density estimation (’KDE TCE 2 ’), KS (’KS’) and the rootcalibration upper bound p U BS (’RBS (ours)’).The bin amounts are chosen based on past literature[29, 34].

Original: The experiments are conducted across several model-dataset combinations, for which logitsets are openly accessible 2 [29, 44].
Revised: The experiments are conducted across several model-dataset combinations, for which logitsets are openly accessible 2 [27, 42].

Original: This includes the models Wide ResNet 32 [60], DenseNet 40,and DenseNet 161 [22] and the datasets CIFAR10, CIFAR100 [27], and ImageNet [9].
Revised: This includes the models Wide ResNet 32 [57], DenseNet 40,and DenseNet 161 [20] and the datasets CIFAR10, CIFAR100 [25], and ImageNet [8].

Original: We did notconduct model training ourselves and refer to [29] and [44] for further details.
Revised: We did notconduct model training ourselves and refer to [27] and [42] for further details.

Original: We include TS, ETS,and DIAG as injective recalibration methods.
Revised: We include TS, ETS,and DIAG as injective recalibration methods.

Original: Further details and results on additional models anddatasets are reported in the Appendix G.
Revised: Further details and results on additional models anddatasets are reported in the Appendix G.

Original: Robustness of calibration errors to test set size We illustrate the estimated values of our introduced upper bound and the other errors, which are lower bounds of the unknown CE 2 on theleft of Figure 3.
Revised: Robustness of calibration errors to test set size We illustrate the estimated values of our introduced upper bound and the other errors, which are lower bounds of the unknown CE 2 on theleft of Figure 3.

Original: On the right, we aggregate across several models to show the systematic drop-offaccording to Proposition 3.3.
Revised: On the right, we aggregate across several models to show the systematic drop-offaccording to Proposition 3.3.

Original: The relative bias is computed by Error ( n ) /Error
Revised: The relative bias is computed by Error ( n ) /Error

Original: and allowsan aggregation of models with different calibration levels.
Revised: and allowsan aggregation of models with different calibration levels.

Original: Included models are DenseNet 40, Wide
Revised: Included models are DenseNet 40, Wide

Original: ResNet 32, ResNet 110 SD, ResNet 110, and LeNet 5, all trained on CIFAR10.
Revised: ResNet 32, ResNet 110 SD, ResNet 110, and LeNet 5, all trained on CIFAR10.

Original: All values representthe calibration of the given model without recalibration transformation.
Revised: All values representthe calibration of the given model without recalibration transformation.

Original: Only our proposed upperbound and KS are stable, and Appendix G shows this holds across a wide range of different settings.
Revised: Only our proposed upperbound and KS are stable, and Appendix G shows this holds across a wide range of different settings.

Original: The theoretically highest lower bound (CWCE 2 with 100 bins) is also constantly the highest estimatedlower bound, but it is sensitive to the test set size.
Revised: The theoretically highest lower bound (CWCE 2 with 100 bins) is also constantly the highest estimatedlower bound, but it is sensitive to the test set size.

Original: Results for further settings presented in Appendix
Revised: Results for further settings presented in Appendix

Original: G show similar results.
Revised: G show similar results.

Original: Quantifying recalibration improvement Next, we assessed how well all estimators were able toquantify the improvement in calibration error after applying different injective recalibration methods(Fig. 1).
Revised: Quantifying recalibration improvement Next, we assessed how well all estimators were able toquantify the improvement in calibration error after applying different injective recalibration methods(Fig. 1).

Original: Only our proposed upper bound estimator RBS is again robust throughout all settings.
Revised: Only our proposed upper bound estimator RBS is again robust throughout all settings.

Original: According to Proposition 4.4 and since RBS is asymptotically unbiased and consistent, it can beregarded as a reliable approximation of the real improvement of the presented recalibration methods.
Revised: According to Proposition 4.4 and since RBS is asymptotically unbiased and consistent, it can beregarded as a reliable approximation of the real improvement of the presented recalibration methods.

Original: For all other estimators, there is a general trend to estimate recalibration improvement higher forlarge test set sizes.
Revised: For all other estimators, there is a general trend to estimate recalibration improvement higher forlarge test set sizes.

Original: In other settings, especially for small test set sizes, calibration improvementis underestimated to the extent that negative improvements (poorer calibration than before) aresuggested.
Revised: In other settings, especially for small test set sizes, calibration improvementis underestimated to the extent that negative improvements (poorer calibration than before) aresuggested.

Original: Results on other settings presented in Appendix G show similar results.
Revised: Results on other settings presented in Appendix G show similar results.

Original: Taken together,these experiments demonstrate the unreliability of existing calibration estimators, in particular, whenused to evaluate recalibration methods.
Revised: Taken together,these experiments demonstrate the unreliability of existing calibration estimators, in particular, whenused to evaluate recalibration methods.

Original: In contrast, our proposed upper bound estimator is stableacross different settings.
Revised: In contrast, our proposed upper bound estimator is stableacross different settings.

Original: Initially 
Revised: Initially due bad recalibration adjusts

Original: Variance regression calibration We consider variance regression to demonstrate the usefulness ofproper calibration errors outside the classiﬁcation setting.
Revised: Variance regression calibration We consider variance regression to demonstrate the usefulness ofproper calibration errors outside the classiﬁcation setting.

Original: To this end, we predict sales prices with anuncertainty estimate in the UCI dataset Residential Building , which consists of a high feature (107)to data instances (372) ratio [43].
Revised: To this end, we predict sales prices with anuncertainty estimate in the UCI dataset Residential Building , which consists of a high feature (107)to data instances (372) ratio [41].

Original: Our model of choice is a fully-connected mixture density networkpredicting mean and variance [2].
Revised: Our model of choice is a fully-connected mixture density networkpredicting mean and variance [1].

Original: Similar to classiﬁcation, we are interested in recalibration of thepredicted variance to adjust possible under- or overconﬁdence.
Revised: Similar to classiﬁcation, we are interested in recalibration of thepredicted variance to adjust possible under- or overconﬁdence.

Original: We use our proposed frameworkto derive a proper calibration error induced by the proper score DSS for recalibration.
Revised: We use our proposed frameworkto derive a proper calibration error induced by the proper score DSS for recalibration.

Original: Further,we compare DSS [12] to squared KCE (SKCE) [56] and analyze the average predicted variancethroughout model training.
Revised: Further,we compare DSS [10] to squared KCE (SKCE) [53] and analyze the average predicted variancethroughout model training.

Original: We use Platt scaling ( x 7!
Revised: We use Platt scaling ( x 7!

Original: wx + b with parameters w, b 2 R [42])on the predicted variance in each training iteration to show how recalibration beneﬁts uncertaintyawareness.
Revised: wx + b with parameters w, b 2 R [40])on the predicted variance in each training iteration to show how recalibration beneﬁts uncertaintyawareness.

Original: We expect high uncertainty awareness at the start of the model training with a dropoff at later iterations.
Revised: We expect high uncertainty awareness at the start of the model training with a dropoff at later iterations.

Original: As can be seen in Figure 4, recalibration is able to adjust the uncertaintyestimate of a model as desired.
Revised: As can be seen in Figure 4, recalibration is able to adjust the uncertaintyestimate of a model as desired.

Original: Further, the DSS estimate, which captures predicted mean andvariance correctness, directly communicates the improved variance ﬁt.
Revised: Further, the DSS estimate, which captures predicted mean andvariance correctness, directly communicates the improved variance ﬁt.

Original: Contrary, the SKCE estimateappears more erratic between iteration steps and seemingly ignores the changes in variance and therecalibration improvement.
Revised: Contrary, the SKCE estimateappears more erratic between iteration steps and seemingly ignores the changes in variance and therecalibration improvement.

Original: We perform further variance regression experiments in Appendix G.
Revised: We perform further variance regression experiments in Appendix G.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we address the problem of reliably quantifying the effect of recalibration on predictiveuncertainty for classiﬁcation and other probabilistic tasks.
Revised: In this work, we address the problem of reliably quantifying the effect of recalibration on predictiveuncertainty for classiﬁcation and other probabilistic tasks.

Original: This is critical for adjusting under- oroverconﬁdence via recalibration.
Revised: This is critical for adjusting under- oroverconﬁdence via recalibration.

Original: To this end, we ﬁrst provide a taxonomy of existing calibration errors.
Revised: To this end, we ﬁrst provide a taxonomy of existing calibration errors.

Original: We discover that most errors are lower bounds of a proper calibration error and fail to assess if a modelis calibrated.
Revised: We discover that most errors are lower bounds of a proper calibration error and fail to assess if a modelis calibrated.

Original: This motivates our deﬁnition of proper calibration errors , which provides a general classof errors for arbitrary probabilistic predictions.
Revised: This motivates our deﬁnition of proper calibration errors , which provides a general classof errors for arbitrary probabilistic predictions.

Original: Since proper calibration errors cannot be estimatedin the general case, we introduce upper bounds, which directly measure the calibration change forinjective transformations.
Revised: Since proper calibration errors cannot be estimatedin the general case, we introduce upper bounds, which directly measure the calibration change forinjective transformations.

Original: This allows us to reliably adjust model uncertainty via recalibration.
Revised: This allows us to reliably adjust model uncertainty via recalibration.

Original: Wedemonstrate theoretically and empirically that the estimated calibration improvement can be highlymisleading for commonly used estimators, including the ECE.
Revised: Wedemonstrate theoretically and empirically that the estimated calibration improvement can be highlymisleading for commonly used estimators, including the ECE.

Original: In stark contrast, our upper boundis robust to changes in data size and estimates robustly the improvement via injective recalibration.
Revised: In stark contrast, our upper boundis robust to changes in data size and estimates robustly the improvement via injective recalibration.

Original: We further show in additional experiments that our approach can be applied successfully to varianceregression.Checklist
Revised: We further show in additional experiments that our approach can be applied successfully to varianceregression.

Original: For all authors...

Original: (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s

Original: contributions and scope?

Original: [Yes]

Original: (b) Did you describe the limitations of your work?

Original: [Yes] We mention required conditions

Original: for the proofs either directly or refer to Appendix D.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [No] We see

Original: positive societal impacts from improved uncertainty awareness via recalibration.

Original: (d) Have you read the ethics review guidelines and ensured that your paper conforms to

Original: them?

Original: [Yes]

Original: If you are including theoretical results...

Original: (a) Did you state the full set of assumptions of all theoretical results?

Original: [Yes]

Original: Major condi-

Original: tions are directly stated; minor conditions in Appendix D as referred.

Original: (b) Did you include complete proofs of all theoretical results?

Original: [Yes] In Appendix D.

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experi-

Original: mental results (either in the supplemental material or as a URL)?

Original: [Yes] Full code is

Original: located in the supplementary material and further experimental details are in Appendix

Original: G.

Original: (b) Did you specify all the training details (e.g., data splits, hyperparameters,

Original: how they

Original: were chosen)?

Original: [Yes] Relevant training details are located in Appendix G.

Original: A lot of

Original: models are not trained by ourselves.

Original: Instead, we loaded their results from publicly

Original: accessible URLs.

Original: We refer to each URL and the original work, where the training was

Original: conducted.

Original: (c) Did you report error bars (e

Original: g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] We repeated experiments until the error bars are barely

Original: visible or not visible anymore.

Original: Only exception is Figure 4, where aggregations hinder

Original: visibility due to high variances in training different seeds.

Original: (d) Did you include the total amount of compute and the type of resources used (e

Original: g., type

Original: of GPUs, internal cluster, or cloud provider)?

Original: [Yes] See Appendix G.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We used assets from

Original: [29] and Rahimi et al.

Original: [44] located in https://github.com/markus93/NN_calibration/and https://github.com/AmirooR/IntraOrderPreservingCalibration .

Original: (b) Did you mention the license of the assets?

Original: [No] Each lincense can be viewed in the

Original: respective github repository.

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]

Original: We include the code for reproducing the experiments and ﬁgures in the supplementary

Original: material.

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [No] The data is publicly accessible and we cited each respectivesource.

Original: (e) Did you discuss whether the data you are using/curating contains personally identiﬁable

Original: [Table]

Original: information or offensive content?

Original: [No] The data is often abstract in nature (we used

Original: [Table]

Original: pretrained logits provided by a publicly accessible source) or downloaded the dataset

Original: [Table]

Original: from the UCI dataset database.

Original: [Table]

Original: If you used crowdsourcing or conducted research with human subjects...

Original: [Table]

Original: (a) Did you include the full text of instructions given to participants and screenshots, if

Original: [Table]

Original: applicable?

Original: [N/A] No crowdsourcing or human subjects

Original: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: [Table]

Original: Board (IRB) approvals, if applicable?

Original: [N/A]

Original: No crowdsourcing or human subjects

Original: [Table]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount

Original: [Table]

Original: spent on participant compensation?

Original: [N/A] No crowdsourcing or human subjects

Original: [Table]