C
ALIBRATION TESTS BEYOND CLASSIFICATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Most supervised machine learning tasks are subject to irreducible prediction errors.
Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets, rather than point estimates.
Such models can be a valuable tool in decision-making under uncertainty, provided that the model output is meaningful and interpretable.
Calibrated models guarantee that the probabilistic predictions are neither over- nor under-conﬁdent.
In the machine learning literature, different measures and statistical tests have been proposed and studied for evaluating the calibration of classiﬁcation models.
For regression problems, however, research has been focused on a weaker condition of calibration based on predicted quantiles for real-valued targets.
In this paper, we propose the ﬁrst framework that uniﬁes calibration evaluation and tests for general probabilistic predictive models.
It applies to any such model, including classiﬁcation and regression models of arbitrary dimension.
Furthermore, the framework generalizes existing measures and provides a more intuitive reformulation of a recently proposed framework for calibration in multi-class classiﬁcation.
I NTRODUCTION
We consider the general problem of modelling the relationship between a feature X and a target Y in a probabilistic setting, i.e., we focus on models that approximate the conditional probability distribution P ( Y | X ) of target Y for given feature X .
The use of probabilistic models that output a probability distribution instead of a point estimate demands guarantees on the predictions beyond accuracy, enabling meaningful and interpretable predicted uncertainties.
One such statistical guarantee is calibration, which has been studied extensively in metereological and statistical literature (DeGroot & Fienberg, 1983; Murphy & Winkler, 1977).
A calibrated model ensures that almost every prediction matches the conditional distribution of targets given this prediction.
Loosely speaking, in a classiﬁcation setting a predicted distribution of the model is called calibrated (or reliable), if the empirically observed frequencies of the different classes match the predictions in the long run, if the same class probabilities would be predicted repeatedly.
A classical example is a weather forecaster who predicts each day if it is going to rain on the next day.
If she predicts rain with probability 60% for a long series of days, her forecasting model is calibrated for predictions of 60% if it actually rains on 60% of these days.
If this property holds for almost every probability distribution that the model outputs, then the model is considered to be calibrated.
Calibration is an appealing property of a probabilistic model since it provides safety guarantees on the predicted distributions even in the common case when the model does not predict the true distributions P ( Y | X ) .
Calibration, however, does not guarantee accuracy (or reﬁnement)—a model that always predicts the marginal probabilities of each class is calibrated but probably inaccurate and of limited use.
On the other hand, accuracy does not imply calibration either since the predictions of an accurate model can be too over-conﬁdent and hence miscalibrated, as observed, e.g., for deep neural networks (Guo et al., 2017).
In the ﬁeld of machine learning, calibration has been studied mainly for classiﬁcation problems (Bröcker, 2009;
Guo et al., 2017; Kull et al., 2017; 2019; Kumar et al., 2018; Platt, 2000;
Vaicenavicius et al., 2019; Widmann et al., 2019; Zadrozny, 2002) and for quantiles and conﬁdence intervals of models for regression problems with real-valued targets (Fasiolo et al., 2020; Ho & Lee, 2005; Kuleshov et al., 2018; Rueda et al., 2006; Taillardat et al., 2016).
In our work, however, we do not restrict ourselves to these problem settings but instead consider calibration for arbitrary predictive models.
Thus, we generalize the common notion of calibration as:
Deﬁnition 1.
Consider a model P X :=
P ( Y | X ) of a conditional probability distribution P ( Y | X ) .
Then model P is said to be calibrated if and only if
[Equation]
almost surely .
[Equation]
If P is a classiﬁcation model, Deﬁnition 1 coincides with the notion of (multi-class) calibration by
Bröcker (2009); Kull et al.
; Vaicenavicius et al.
Alternatively, in classiﬁcation some authors (Guo et al., 2017; Kumar et al., 2018; Naeini et al., 2015) study the strictly weaker property of conﬁdence calibration (Kull et al., 2019), which only requires
[Equation]
almost surely
[Equation]
to hold for every target y .
For real-valued targets, Deﬁnition 1 coincides with the so-called distribution-level calibration by Song et al.
Distribution-level calibration implies that the predicted quantiles are calibrated, i.e., the outcomes for all real-valued predictions of the, e.g., 75% quantile are actually below the predicted quantile with 75% probability (Song et al., 2019, Theorem 1).
Conversely, although quantile-based calibration is a common approach for real-valued regression problems (Fasiolo et al., 2020; Ho & Lee, 2005; Kuleshov et al., 2018; Rueda et al., 2006; Taillardat et al., 2016), it provides weaker guarantees on the predictions.
For instance, the linear regression model in Fig.
1 empirically shows quantiles that appear close to being calibrated albeit being uncalibrated according to Deﬁnition 1.
[Figure]
i r i c
[Figure]
l f r
[Figure]
idealdata
Figure 1 also raises the question of how to assess calibration for general target spaces in the sense of
Deﬁnition 1, without having to rely on visual inspection.
In classiﬁcation, measures of calibration such as the commonly used expected calibration error (ECE) (Guo et al., 2017; Kull et al., 2019;
Naeini et al., 2015; Vaicenavicius et al., 2019) and the maximum calibration error (MCE) (Naeini et al., 2015) try to capture the average and maximal discrepancy between the distributions on the left hand side and the right hand side of Eq.
(1) or Eq.
(2), respectively. These measures can be generalized to other target spaces (see Deﬁnition B.1), but unfortunately estimating these calibration errors from observations of features and corresponding targets is problematic.
Typically, the predictions are different for (almost) all observations, and hence estimation of the conditional probability P ( Y | P X ) , which is needed in the estimation of ECE and MCE, is challenging even for low-dimensional target spaces and usually leads to biased and inconsistent estimators (Vaicenavicius et al., 2019).
Kernel-based calibration errors such as the maximum mean calibration error (MMCE) (Kumar et al., 2018) and the kernel calibration error (KCE)
(Widmann et al., 2019) for conﬁdence and multi-class calibration, respectively, can be estimated without ﬁrst estimating the conditional probability and hence avoid this issue.
They are deﬁned as the expected value of a weighted sum of the differences of the left and right hand side of Eq.
(1) for each class, where the weights are given as a function of the predictions (of all classes) and chosen such that the calibration error is maximized.
A reformulation with matrix-valued kernels (Widmann et al., 2019) yields unbiased and differentiable estimators without explicit dependence on P ( Y | P X ) , which simpliﬁes the estimation and allows to explicitly account for calibration in the training objective (Kumar et al., 2018).
Additionally, the kernel-based framework allows the derivation of reliable statistical hypothesis tests for calibration in multi-class classiﬁcation (Widmann et al., 2019).
However, both the construction as a weighted difference of the class-wise distributions in Eq.
(1) and the reformulation with matrix-valued kernels require ﬁnite target spaces and hence cannot be applied to regression problems.
To be able to deal with general target spaces, we present a new and more general framework of calibration errors without these limitations.
Our framework can be used to reason about and test for calibration of any probabilistic predictive model .
As explained above, this is in stark contrast with existing methods that are restricted to simple output distributions, such as classiﬁcation and scalar-valued regression problems.
A key contribution of this paper is a new framework that is applicable to multivariate regression, as well as situations when the output is of a different (e.g. discrete ordinal) or more complex (e.g, graph-structured) type, with clear practical implications.
Within this framework a KCE for general target spaces is obtained.
We want to highlight that for multi-class classiﬁcation problems its formulation is more intuitive and simpler to use than the measure proposed by Widmann et al.
based on matrix-valued kernels.
To ease the application of the KCE we derive several estimators of the KCE with subquadratic sample complexity and their asymptotic properties in tests for calibrated models, which improve on existing estimators and tests in the two-sample test literature by exploiting the special structure of the calibration framework.
Using the proposed framework, we numerically evaluate the calibration of neural network models and ensembles of such models.
C ALIBRATION ERROR : A GENERAL FRAMEWORK
In classiﬁcation, the distributions on the left and right hand side of Eq.
(1) can be interpreted as vectors in the probability simplex.
Hence ultimately the distance measure for ECE and MCE (see Deﬁnition B.1) can be chosen as a distance measure of real-valued vectors.
The total variation,
Euclidean, and squared Euclidean distances are common choices (Guo et al., 2017; Kull et al., 2019; Vaicenavicius et al., 2019).
However, in a general setting measuring the discrepancy between P ( Y | P X ) and P X cannot necessarily be reduced to measuring distances between vectors.
The conditional distribution P ( Y | P X ) can be arbitrarily complex, even if the predicted distributions are restricted to a simple class of distributions that can be represented as real-valued vectors.
Hence in general we have to resort to dedicated distance measures of probability distributions.
Additionally, the estimation of conditional distributions P ( Y | P X ) is challenging, even more so than in the restricted case of classiﬁcation, since in general these distributions can be arbitrarily complex.
To circumvent this problem, we propose to use the following construction: We deﬁne a random variable Z X ∼ P X obtained from the predictive model and study the discrepancy between the joint distributions of the two pairs of random variables ( P X , Y ) and ( P X , Z X ) , respectively, instead of the discrepancy between the conditional distributions P ( Y | P X ) and P X .
Since
[Equation]
if and only if
[Equation]
almost surely , model P is calibrated if and only if the distributions of ( P X , Y ) and ( P X , Z X ) are equal.
The random variable pairs ( P X , Y ) and ( P X , Z X ) take values in the product space P ×Y , where P is the space of predicted distributions P X and Y is the space of targets Y .
For instance, in classiﬁcation,
P could be the probability simplex and Y the set of all class labels, whereas in the case of Gaussian predictive models for scalar targets P could be the space of normal distributions and Y be R .
The study of the joint distributions of ( P X , Y ) and ( P X , Z X ) motivates the deﬁnition of a generally applicable calibration error as an integral probability metric (Müller, 1997; Sriperumbudur et al., 2009; 2012) between these distributions.
In contrast to common f -divergences such as the Kullback-Leibler divergence, integral probability metrics do not require that one distribution is absolutely continuous with respect to the other, which cannot be guaranteed in general.
Deﬁnition 2.
Let Y denote the space of targets Y , and P the space of predicted distributions P X .
We deﬁne the calibration error with respect to a space of functions F of the form f : P × Y → R as
[Equation]
By construction, if model P is calibrated, then CE F = 0 regardless of the choice of F .
However, the converse statement is not true for arbitrary function spaces F .
From the theory of integral probability metrics (see, e.g., Müller, 1997; Sriperumbudur et al., 2009; 2012), we know that for certain choices of F the calibration error in Eq.
(3) is a well-known metric on the product space P ×Y , which implies that CE F =
if and only if model P is calibrated.
Prominent examples include the maximum mean discrepancy 2 (MMD) (Gretton et al., 2007), the total variation distance, the Kantorovich distance, and the Dudley metric (Dudley, 1989, p. 310).
As pointed out above, Deﬁnition 2 is a generalization of the deﬁnition for multi-class classiﬁcation proposed by Widmann et al.
(2019)—which is based on vector-valued functions and only applicable to ﬁnite target spaces—to any probabilistic predictive model .
In Appendix E we show this explicitly and discuss the special case of classiﬁcation problems in more detail.
Previous results (Widmann et al., 2019) imply that in classiﬁcation MMCE and, for common distance measures d ( · , · ) such as the total variation and squared Euclidean distance, ECE d and MCE d are special cases of CE F .
In
Appendix G we show that our framework also covers natural extensions of ECE d and MCE d to countably inﬁnite discrete target spaces, which to our knowledge have not been studied before and occur, e.g., in Poisson regression.
The literature of integral probability metrics suggests that we can resort to estimating CE F from i.i.d.
samples from the distributions of ( P X , Y ) and ( P X , Z X ) .
For the MMD, the Kantorovich distance, and the Dudley metric tractable strongly consistent empirical estimators exist (Sriperumbudur et al., 2012).
Here the empirical estimator for the MMD is particularly appealing since compared with the other estimators “it is computationally cheaper, the empirical estimate converges at a faster rate to the population value, and the rate of convergence is independent of the dimension d of the space (for
S = R d )” (Sriperumbudur et al.
Our speciﬁc design of ( P X , Z X ) can be exploited to improve on these estimators.
If E Z x ∼ P x f ( P x , Z x ) can be evaluated analytically for a ﬁxed prediction P x , then CE F can be estimated empirically with reduced variance by marginalizing out Z X .
Otherwise E Z x ∼ P x f ( P x , Z x ) has to be estimated, but in contrast to the common estimators of the integral probability metrics discussed above the artiﬁcial construction of Z X allows us to approximate it by numerical integration methods such as (quasi) Monte Carlo integration or quadrature rules with arbitrarily small error and variance.
Monte Carlo integration preserves statistical properties of the estimators such as unbiasedness and consistency.
K ERNEL CALIBRATION ERROR
For the remaining parts of the paper we focus on the MMD formulation of CE F due to the appealing properties of the common empirical estimator mentioned above.
We derive calibration-speciﬁc analogues of results for the MMD that exploit the special structure of the distribution of ( P X , Z X ) to improve on existing estimators and tests in the MMD literature.
To the best of our knowledge these variance-reduced estimators and tests have not been discussed in the MMD literature.
Let k :
( P × Y ) × ( P × Y )
→ R be a measurable kernel with corresponding reproducing kernel
Hilbert space (RKHS) H , and assume that
E
[Equation]
Y
[Equation] [Equation]
P X
[Equation]
Y ) ,
[Equation]
P X
[Equation]
Y )
[Equation]
< ∞ and
[Equation]
We discuss how such kernels can be constructed in a generic way in Section 3.1 below.
Deﬁnition 3.
Let F k denote the unit ball in H , i.e., F := { f ∈ H|(cid:107) f (cid:107) H ≤ 1 } .
Then the kernel calibration error (KCE) with respect to kernel k is deﬁned as
[Equation]
As known from the MMD literature, a more explicit formulation can be given for the squared kernel calibration error SKCE k :=
KCE 2 k (see Lemma B.2).
A similar explicit expression for SKCE k was obtained by Widmann et al.
for the special case of classiﬁcation problems.
However, their expression relies on Y being ﬁnite and is based on matrix-valued kernels over the ﬁnite-dimensional probability simplex P .
A key difference to the expression in Lemma B.2 is that we instead propose to use real-valued kernels deﬁned on the product space of predictions and targets.
This construction is applicable to arbitrary target spaces and does not require Y to be ﬁnite.
3. C HOICE OF KERNEL
The construction of the product space P
× Y suggests the use of tensor product kernels k = k P ⊗ k Y , where k P : P × P → R and k Y :
Y × Y → R are kernels on the spaces of predicted distributions and targets, respectively.
By deﬁnition, so-called characteristic kernels guarantee that KCE
=
if and only if the distributions of ( P X , Y ) and ( P X , Z X ) are equal (Fukumizu et al., 2004; 2008).
Many common kernels such as the Gaussian and Laplacian kernel on R d are characteristic (Fukumizu et al., 2008).
4 Szabó &
Sriperumbudur (2018, Theorem 4) showed that a tensor product kernel k P ⊗ k Y is characteristic if k P and k Y are characteristic, continuous, bounded, and translation-invariant kernels on R d , but the implication does not hold for general characteristic kernels (Szabó & Sriperumbudur, 2018,
Example 1).
For calibration evaluation, however, it is sufﬁcient to be able to distinguish between the conditional distributions P ( Y | P X ) and P ( Z X | P X ) =
P X .
Therefore, in contrast to the regular
MMD setting, it is sufﬁcient that kernel k Y is characteristic and kernel k P is non-zero almost surely , to guarantee that KCE
=
if and only if model P is calibrated.
Thus it is suggestive to construct kernels on general spaces of predicted distributions as
[Equation]
where d P ( · , · ) is a metric on P and ν, λ > 0 are kernel hyperparameters.
The Wasserstein distance is a widely used metric for distributions from optimal transport theory that allows to lift a ground metric on the target space and possesses many important properties (see, e.g., Peyré & Cuturi, 2018,
Chapter 2.4).
In general, however, it does not lead to valid kernels k P , apart from the notable exception of elliptically contoured distributions such as normal and Laplace distributions (Peyré &
Cuturi, 2018, Chapter 8.3).
In machine learning, common probabilistic predictive models output parameters of distributions such as mean and variance of normal distributions.
Naturally these parameterizations give rise to injective mappings φ : P → R d that can be used to deﬁne a Hilbertian metric
[Equation]
For such metrics, k P in Eq.
(4) is a valid kernel for all λ > 0 and ν ∈ (0 , 2] (Berg et al., 1984,
Corollary 3.3.3, Proposition 3.2.7).
In Appendix D.3 we show that for many mixture models, and hence model ensembles, Hilbertian metrics between model components can be lifted to Hilbertian metrics between mixture models.
This construction is a generalization of the Wasserstein-like distance for Gaussian mixture models proposed by Chen et al.
; Delon & Desolneux (2019).
3. E STIMATION
Let ( X 1 , Y 1 ) , . . .
, ( X n , Y n ) be a data set of features and targets which are i.i.d.
according to the law of ( X, Y ) .
Moreover, for notational brevity, for ( p, y ) , ( p (cid:48) , y (cid:48) )
∈ P × Y we let h (cid:0) ( p, y ) , ( p (cid:48) , y (cid:48) ) (cid:1)
[Equation]
=
k
[Equation]
p, y
[Equation] [Equation]
p
[Equation]
, y
[Equation] [Equation]
E
[Equation]
k
[Equation]
p,
[Equation] [Equation]
p
[Equation]
Note that in contrast to the regular MMD we marginalize out Z and Z (cid:48) .
Similar to the MMD, there exist consistent estimators of the SKCE, both biased and unbiased.
Lemma 1.
The plug-in estimator of SKCE k is non-negatively biased.
It is given by
[Equation]
Inspired by the block tests for the regular MMD (Zaremba et al., 2013), we deﬁne the following class of unbiased estimators.
Note that in contrast to (cid:92) SKCE k they do not include terms of the form h (cid:0) ( P X i , Y i ) , ( P X i , Y i ) (cid:1) .
Lemma 2.
The block estimator of SKCE k with block size B ∈ { 2 , . . . , n } , given by
[Equation]
is an unbiased estimator of SKCE k .
The extremal estimator with B = n is a so-called U-statistic of SKCE k (Hoeffding, 1948; van der
Vaart, 1998), and hence it is the minimum variance unbiased estimator.
All presented estimators are consistent, i.e., they converge to SKCE k almost surely as the number n of data points goes to inﬁnity.
The sample complexity of (cid:92) SKCE k and (cid:92) SKCE k,B is O ( n 2 ) and O ( Bn ) , respectively.
3. C ALIBRATION TESTS
A fundamental issue with calibration errors in general, including ECE, is that their empirical estimates do not provide an answer to the question if a model is actually calibrated.
Even if the measure is guaranteed to be zero if and only if the model is calibrated, usually the estimates of calibrated models are non-zero due to randomness in the data and (possibly) the estimation procedure.
In classiﬁcation, statistical hypothesis tests of the null hypothesis
[Equation]
so-called calibration tests, have been proposed as a tool for checking rigorously if P is calibrated (Bröcker & Smith, 2007; Vaicenavicius et al., 2019; Widmann et al., 2019).
For multi-class classiﬁcation, Widmann et al.
(2019) suggested calibration tests based on the asymptotic distributions of estimators of the previously formulated KCE.
Although for ﬁnite data sets the asymptotic distributions are only approximations of the actual distributions of these estimators, in their experiments with 10 classes the resulting p -value approximations seemed reliable whereas p -values obtained by so-called consistency resampling (Bröcker & Smith, 2007; Vaicenavicius et al., 2019) underestimated the p -value and hence rejected the null hypothesis too often (Widmann et al., 2019).
For ﬁxed block sizes (cid:112) (cid:98) n/B (cid:99) (cid:0) (cid:92) SKCE k,B − SKCE k (cid:1) d −→ N (cid:0) 0 , σ 2 B (cid:1) as n → ∞ , and, under H 0 , n (cid:92) SKCE k,n d −→ (cid:80) ∞ i =
λ i
( Z i − 1) as n → ∞ , where Z i are independent χ 2 1 distributed random variables.
See Appendix B for details and deﬁnitions of the involved constants.
From these results one can derive calibration tests that extend and generalize the existing tests for classiﬁcation problems, as explained in Remarks B.1 and B.2.
Our formulation illustrates also the close connection of these tests to different two-sample tests (Gretton et al., 2007; Zaremba et al., 2013).
A LTERNATIVE APPROACHES
For two-sample tests, Chwialkowski et al. (2015) suggested the use of the so-called unnormalized mean embedding (UME) to overcome the quadratic sample complexity of the minimum variance unbiased estimator and its intractable asymptotic distribution.
As we show in Appendix C, there exists an analogous measure of calibration, termed unnormalized calibration mean embedding (UCME), with a corresponding calibration mean embedding (CME) test.
As an alternative to our construction based on the joint distributions of ( P X , Y ) and ( P X , Z X ) , one could try to directly compare the conditional distributions P ( Y | P X ) and P ( Z X | P X ) =
P X .
For instance, Ren et al.
(2016) proposed the conditional MMD based on the so-called conditional kernel mean embedding (Song et al., 2009; 2013).
However, as noted by Park & Muandet (2020), its common deﬁnition as operator between two RKHS is based on very restrictive assumptions, which are violated in many situations (see, e.g., Fukumizu et al., 2013, Footnote 4) and typically require regularized estimates.
Hence, even theoretically, often the conditional MMD is “not an exact measure of discrepancy between conditional distributions” (Park & Muandet (2020)).
In contrast, the maximum conditional mean discrepancy (MCMD) proposed in a concurrent work by Park &
Muandet (2020) is a random variable derived from much weaker measure-theoretical assumptions.
The MCMD provides a local discrepancy conditional on random predictions whereas KCE is a global real-valued summary of these local discrepancies.
E XPERIMENTS
In our experiments we evaluate the computational efﬁciency and empirical properties of the proposed calibration error estimators and calibration tests on both calibrated and uncalibrated models.
By means of a classic regression problem from statistics literature, we demonstrate that the estimators and tests can be used for the evaluation of calibration of neural network models and ensembles of such models.
This section contains only an high-level overview of these experiments to conserve space but all experimental details are provided in Appendix A.
5. E MPIRICAL PROPERTIES AND COMPUTATIONAL EFFICIENCY
We evaluate error, variance, and computation time of calibration error estimators for calibrated and uncalibrated Gaussian predictive models in synthetic regression problems.
The results empirically conﬁrm the consistency of the estimators and the computational efﬁciency of the estimator with block size B = 2 which, however, comes at the cost of increased error and variance.
Additionally, we evaluate empirical test errors of calibration tests at a ﬁxed signiﬁcance level α = 0 .
The evaluations, visualized in Fig. 2 for models with ten-dimensional targets, demonstrate empirically that the percentage of incorrect rejections of H 0 converges to the set signiﬁcance level as the number of samples increases.
Moreover, the results highlight the computational burden of the calibration test that estimates quantiles of the intractable asymptotic distribution of (cid:92) SKCE k,n by bootstrapping.
As expected, due to the larger variance of (cid:92) SKCE k, 2 the test with ﬁxed block size B = 2 shows a decreased test power although being computationally much more efﬁcient.
[Figure]
5. F RIEDMAN 1 REGRESSION PROBLEM
The Friedman 1 regression problem (Friedman, 1979; 1991; Friedman et al., 1983) is a classic non-linear regression problem with ten-dimensional features and real-valued targets with Gaussian noise.
We train a Gaussian predictive model whose mean is modelled by a shallow neural network and a single scalar variance parameter (consistent with the data-generating model) ten times with different initial parameters.
Figure 3 shows estimates of the mean-squared error (MSE), the average negative log-likelihood (NLL), SKCE k , and a p -value approximation for these models and their ensemble on the training and a separate test data set.
All estimates indicate consistently that the models are overﬁt after 1500 training iterations.
The estimations of SKCE k and the p -values allow to focus on calibration speciﬁcally, whereas MSE indicates accuracy only and NLL, as any proper scoring rule (Bröcker, 2009), provides a summary of calibration and accuracy.
The estimation of SKCE k in addition to NLL could serve as another source of information for early stopping and model selection.
C ONCLUSION
We presented a framework of calibration estimators and tests for any probabilistic model that captures both classiﬁcation and regression problems of arbitrary dimension as well as other predictive models.
We successfully applied it for measuring calibration of (ensembles of) neural network models.
Our framework highlights connections of calibration to two-sample tests and optimal transport theory which we expect to be fruitful for future research.
For instance, the power of calibration tests could be improved by heuristics and theoretical results about suitable kernel choices or hyperparameters (cf.
Jitkrittum et al., 2016).
It would also be interesting to investigate alternatives to KCE captured by our framework, e.g., by exploiting recent advances in optimal transport theory (cf.
Genevay et al., 2016).
Since the presented estimators of SKCE k are differentiable, we imagine that our framework could be helpful for improving calibration of predictive models, during training (cf.
Kumar et al., 2018) or post-hoc.
Currently, many calibration methods (see, e.g., Guo et al., 2017; Kull et al., 2019; Song et al., 2019) are based on optimizing the log-likelihood since it is a strictly proper scoring rule and thus encourages both accurate and reliable predictions.
However, as for any proper scoring rule, “Per se, it is impossible to say how the score will rank unreliable forecast schemes [...].
The lack of reliability of one forecast scheme might be outbalanced by the lack of resolution of the other” (Bröcker (2009)).
In other words, if one does not use a calibration method such as temperature scaling (Guo et al., 2017) that keeps accuracy invariant 6 , it is unclear if the resulting model is trading off calibration for accuracy when using log-likelihood for re-calibration.
Thus hypothetically ﬂexible calibration methods might beneﬁt from using the presented calibration error estimators.