Q UANTILE R EGULARIZATION : T OWARDS I MPLICIT C ALIBRATION OF R EGRESSION M ODELS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning models are often poorly calibrated, i.e., they may produce overconﬁdent predictions that are wrong, implying that their uncertainty estimates are unreliable.
While a number of approaches have been proposed recently to calibrate classiﬁcation models, relatively little work exists on calibrating regression models.
Isotonic Regression has recently been advocated for regression calibration.
We provide a detailed formal analysis of the side-effects of Isotonic Regression when used for regression calibration.
To address these, we investigate the idea of quantile calibration (Kuleshov et al., 2018), recast it as entropy estimation, and leverage the new formulation to construct a novel quantile regularizer, which can be used as a blackbox to calibrate any probabilistic regression model.
Unlike most of the existing approaches for calibrating regression models, which are based on post-hoc processing of the model’s output, and require an additional dataset, our method is trainable in an end-to-end fashion, without requiring an additional dataset.
We provide empirical results demonstrating that our approach improves calibration for regression models trained on diverse architectures that provide uncertainty estimates, such as Dropout VI, Deep Ensembles.
I NTRODUCTION
For supervised machine learning, the notion of calibration of a learned predictive model is a measure of evaluating how well a model’s conﬁdence in its prediction matches with the correctness of these predictions.
For example, a binary classiﬁer will be considered perfectly calibrated if, among all predictions with probability score 0.9, 90% of the predictions are correct Guo et al.
Likewise, consider a probabilistic regression model that produces credible interval for the predicted outputs.
In this setting, the model will be considered perfectly calibrated if the 90% conﬁdence interval contains 90% of the true test outputs (Kuleshov et al., 2018).
Unfortunately, modern deep neural networks are known to be poorly calibrated (Guo et al., 2017), raising questions on their reliability.
The notion of calibration for classiﬁcation problems was originally ﬁrst considered in meteorology literature (Brier, 1950; Murphy, 1972; Gneiting & Raftery, 2007) and saw one of its ﬁrst prominent usage used in the machine learning literature by (Platt et al., 1999) in context of Support Vector Machines (SVM), in order to obtain probabilistic predictions from SVMs which are inherently non-probabilistic models.
Recently, there has been renewed interested in calibration, especially for classiﬁcation models, after it has shown (Guo et al., 2017) that modern deep neural networks for classiﬁcation are often poorly calibrated.
Currently there are three main notions of calibration in case of classiﬁcation (Kumar et al., 2019;
Vaicenavicius et al., 2019; Kull et al., 2019). Most calibration methods (Platt et al., 1999; Zadrozny & Elkan, 2001; 2002; Guo et al., 2017; Kull et al., 2017; 2019) for classiﬁcation models are post hoc , where they learn a calibration mapping R : [0 , 1] → [0 , 1] using an additional dataset to recalibrate an already trained model.
There has been recent work showing some of these popular post hoc methods are either themselves miscalibrated or sample-inefﬁcient (Kumar et al., 2019) and they do not actually help the model output well-calibrated probabilities.
An alternative to post hoc processing is to ensure that model outputs well-calibrated probabilities after model training ﬁnishes.
We refer to these as implicit calibration methods.
Notably, such an approach does not require an additional dataset to learn the calibration mapping.
While almost all post hoc calibration methods for classiﬁcation models can be seen in a uniﬁed manner as density estimation methods( see section 2.1 ), existing implicit calibration methods for classiﬁcation models have been designed with various, often distinct, considerations/approaches.
Several heuristics like
Mixup (Zhang et al., 2017; Thulasidasan et al., 2019) and Label Smoothing (Szegedy et al., 2016;
M¨uller et al., 2019) that were part of high performance deep networks for classiﬁcation were later shown empirically to achieve calibration.
(Maddox et al., 2019) show that their optimization method instrinsically improves calibration.
(Pereyra et al., 2017) found that penalizing high-conﬁdence predictions acts as a regularizer.
A more principled way of achieving implicit calibration is by minimizing a loss function that is tailored for calibration (Kumar et al., 2018).
This is somewhat similar in spirit to our proposed approach that aims to do it for regression models.
Among the early approaches for calibrating regression models, (Gneiting et al., 2007) were the ﬁrst to propose a framework for calibrating regression models.
However, they do not provide any procedure to correct a miscalibrated model.
Recently, (Kuleshov et al., 2018) introduced the notion of Quantile Calibration which intuitively says that the p conﬁdence interval predicted by model should have target variable with probability p .
They use a post hoc calibration method based on Isotonic Regression (Fawcett & Niculescu-Mizil, 2007), which is a well-known calibration technique for classiﬁcation models.
The difference between Isotonic Calibration in classiﬁcation and Isotonic Calibration in regression is that ( i ) dataset on which calibration mapping is learnt; and ( ii ) the function with which learnt calibration mapping is pre-composed .
In the former case, it is pre-composed with a probability mass function (PMF) and whereas in the latter, it is pre-composed with a conditional density function (CDF).
Both these differences produces side effects i) nature of recalibration dataset already satisﬁes monotonicity constraint , so there is overﬁtting in case of smaller calibration datsets ii) since we are composing CDF with piecewise linear function, we will make the resultant CDF discontinuous and corresponding PDF non differentiable (see Sec.
3 for detailed discussion for side effects of the Isotonic Calibration approach).
In another recent work, (Song et al., 2019) proposed a much stronger notion of calibration called Distributional Calibration which guarantees that among all instances whose predicted probability density function (PDF) of the response variable has mean µ and standard deviation σ , the marginal distribution of the target variable should have mean µ and standard deviation σ .
They too propose a post hoc recalibration method based on expensive Gaussian processes.
Among other work, (Keren et al., 2018), consider a different setting where neural networks for classiﬁcation are used for regression problems and showed that temperature scaling (Hinton et al., 2015; Guo et al., 2017) and their proposed method based on empirical prediction intervals improves calibration for regression problems as well.
Again, these are post hoc methods.
Our contributions are summarized below:
We analyze in detail the side effects of Isotonic Calibration for regression models.
We show how using Isotonic Calibration results in truncation of the support, which will result in assigning zero likelihood at test time.
We also discuss about Isotonic Calibration resulting in nonsmooth PDFs, and its tendency to produce miscalibration when using small calibration datasets.
At test time, after composing predicted CDF with the learned isotonic mapping, the mean prediction (point estimate) also changes.
Kuleshov et al.
(2018) do not acknowledge changes in the mean estimate.
While Song et al.
(2019) acknowledge this issue, they use a trapezoidal approximation to remedy this.
In contrast, we derive an analytical expression for the updated point estimate after isotonic calibration.
We also provide a different expression for updated point estimate, which reduces the time-complexity from O ( m ) to O (1) , where m is calibration dataset size.
In order to mitigate these shortcomings of Isotonic Calibration, we propose a simple, yet novel and general purpose, trainable loss function for quantile calibration where the smoothness of PDF/CDF is not sacriﬁced for well-calibrated probabilties.
Our approach also eliminates the need for additional calibration dataset.
We conduct extensive experiments on a wide range of architectures using the proposed loss function (Quantile Regularization) and show empirically that it improves calibration on wide range of architectures that produce uncertainty estimates.
B ACKGROUND AND D EFINITIONS
Probabilistic regression model can be seen as conditional PDF/conditional CDF.
In the rest of the paper, we express it as conditional CDF M : X → ( Y → [0 , 1]) .
In practice, model outputs µ and σ or parameters of likelihood distribution under consideration.
In the rest of paper we consider
Gaussian likelihood unless otherwise stated because it is the most prevalent case.
Kuleshov et al.
(2018) proposed following notion of calibration of regression models called quantile calibration .
Importance of following deﬁnition is we get reliable conﬁdence intervals.
Deﬁnition 1 (Quantile Calibration)
Given a regression model M : X → ( Y → [0 , 1]) and X, Y jointly distributed as P , the model M is said to be Quantile Calibrated iff
[Equation]
Note that [ M ( X ) ]
( Y ) is cumulative density that the model predicts for random input-response pair drawn from the joint distribution of X and Y. Pick a value between [0 , 1] , say 0 .
68 , if the model is
Quantile Calibrated, 68% conﬁdence interval contains target points 68% on average.
2. F UNDAMENTAL T HEOREM OF P OST H OC C ALIBRATION
The objective of post hoc calibration is to recalibrate a miscalibrated model M by learning a mapping R : [0 , 1] → [0 , 1] s.t.
R ◦ M is calibrated model.
One such mapping can be obtained from the deﬁnition of calibration itself.
Setting R ( p ) =
P (cid:2)
[ M ( X )]( Y ) ≤ p (cid:3) makes R ◦ M a quantile calibrated model.
(Vaicenavicius et al., 2019) call an analogous mapping in the context of classiﬁcation as canonical calibration mapping.
Theorem 1 For any Model M : X → ( Y → [0 , 1]) and given canonical calibration mapping R ( p ) = P (cid:2) [ M
[Equation]
X )]
[Equation]
Y [Equation] ≤ p (cid:3) , R ◦ M is quantile calibrated Note that learning the mapping R reduces to density estimation of P (cid:2)
[ M ( X )]( Y ) ≤ p (cid:3) , which is a hard problem in itself.
With this insight, and using the fact that the mapping is monotonically increasing, (Kuleshov et al., 2018) use Isotonic Regression to learn the mapping on a separate calibration dataset.
Similar theorems hold for other notions of calibration, too.
2. I SOTONIC C ALIBRATION
We next describe Isotonic Calibration which is based on Isotonic Regression.
Given ordered inputs x 1 , x 2 , . . .
x m i.e., x 1 ≤ x 2 · · · ≤ x m and corresponding target variables y 1 , y 2 , . . .
y m , Isotonic Regression ﬁnds z 1 , z 2 , . . .
z m by solving the following optimization problem
[Equation]
(Mair et al., 2009) provides a nice survey of algorithms for Isotonic Regression.
In particular
Scikit-learn (Pedregosa et al., 2011) uses Pool Adjacent Violaters Algorithm (PAVA) of complexity
O ( m ) , .
Given a calibration dataset { x i , y i } mi =1 Isotonic Calibration ﬁrst builds a recalibration set as D = (cid:40) (cid:16)
[Equation]
, and then applies Isotonic
Regression on the dataset D to get isotonic mapping with linear interpolation.
A NALYSIS OF I SOTONIC C ALIBRATION
Let { ( x k , y k ) }
Nk =1 be a calibration dataset.
Let { ( µ k , σ 2 k ) }
Nk =1 be means and variances predicted by model trained.
Suppose we have obtained CDF values at actual outputs ( y 1 , y 2 , · · · , y n ) .
Let these be ( c 1 , c 2 , · · · , c n ) .
Then the calibration dataset is constructed as
D = { ( c (1) , 1 n ) , ( c (2) , 2 n ) , · · · , ( c ( n ) , 1) } where
C = ( c (1) , c (2) , · · · c ( n ) ) is obtained by sorting ( c 1 , c 2 , · · · c n ) in ascending order.
Now we have to ﬁt isotonic regression on this dataset.
Since ( 1 n , 2 n , · · · , 1) are already in increasing order, isotonic regression does not modify values.
So we don’t even need to ﬁt per say except for basic interpolation.
Complete Mapping is obtained by adding (0 , 0) to D and connecting these points.
We ﬁrst give conditions under which smoothness is lost and later derive the correction one has to use after isotonic calibration.
Note that all proofs are present in supplementary material.
Claim 1 Let Y be a random variable with CDF F , and let G
= R ◦ F be its CDF after composing with mapping R obtained from isotonic regression characterized by C =
{ c (1) , c (2) , · · · , c ( n ) } .
If there exist i − 1 , i, i +1 ∈ { 0 , n }∧ c ( i ) − c
( i − 1) (cid:54) =
c ( i +1) − c
( i ) then the CDF G is not differentiable and its corresponding probability density function g is not continuous at F − 1 ( c ( i ) )
Because of this, PDF of transformed r.v becomes discontinuous and spiky (see Fig. 1d ).
Update likelihood inversely depends on n ( c ( i ) − c
( i +1) ) .
In many of cases it becomes so small that somtimes updated likelihood increases by factor of 10 2 to 10 5 for single point there by completely destroying what average log likelihood represents .
To show this, we report maximum likelihood attained for single test time point after isotonic calibration in UCI experiments (see Table 1, Table 2 )
Now we derive analytical expression for updated mean after Isotonic Calibration.
Claim 2 Let Y iso be transformed random variable after applying isotonic mapping R on random variable Y .
Then the expectation of Y iso is as follows.
[Equation]
Summation involves both recalibration dataset and test time prediction.
Now we will use properties of quantile functions to decouple the dependency , by which summation just depends on recalibration dataset.
Claim 3 let c ( i ) =
F µ ( i ) ,σ ( i ) ( y ( i ) ) and p ( i ) = f µ ( i ) ,σ ( i
[Equation] ( y (
[Equation]
Now δ can be computed once , so at test time given µ, σ as model prediction, updated mean after isotonic calibration is µ − δ.σ .
This way the time required is reduced from O ( m ) to O (1) .
Also, the construction of recalibration dataset as suggested in (Kuleshov et al., 2018) can result in truncation of support of updated r.v (see ﬁg. 1d).
One way to remedy is to use ( x , ∞ ) to calibration dataset for any x ∈ R n .
So recalibration dataset becomes { ( c (1) , 1 n +1 ) , . . .
, ( c ( n ) , nn +1 ) , (1 , 1) } .
To illustrate all these , we use Bayesian linear regression with 256 training examples generated from y = 3 x + (cid:15) where (cid:15) ∼ N (0 , 1) .
and x ∈ [ − 4 , 4] randomly sampled and calibration dataset of size 32 .
Q UANTILE R EGULARIZATION
In quantile calibration, we want P (cid:2)
[ M ( X )]( Y ) ≤ p (cid:3) =
p ∀ p ∈ [0 , 1] .
Our method is based on the idea that both right and left hand sides can be viewed as CDF.
Let R ( p ) =
P (cid:2)
[ M ( X )]( Y ) ≤ p (cid:3) and S ( p ) = p .
Here R can be seen as the CDF of [ M ( X )]
( Y ) while S can be seen as the CDF of Uniform[0,1].
Quantile calibration mandates these two CDFs to be equal.
So, for a perfectly calibrated quantile model M , we have that [ M ( X )]
( Y ) is a uniform distribution.
We seek to penalize the model this r.v deviates from a uniform distribution which gives us a calibration metric that can be used as a regularizer with any regression loss function, achieving highly desirable calibration during
[Figure]
Claim 4 Let M be any regression model.
Then M is perfectly quantile calibrated iff
[Equation]
4. D IFFERENTIAL E NTROPY E STIMATION
We need the differential entropy estimator for deriving our calibration loss function.
There is a rich literature for entropy estimation.
A brief overview about non-parametric entropy estimation can be found at (Beirlant & Dudewicz, 1997).
We use sample-spacing entropy estimation originally proposed in (Vasicek, 1976).
An m spacing of random variable is deﬁned as amount of probability mass between ordered samples that are m − 1 samples apart.
Let S be a one-dimension r.v. with
CDF F and assume we are given n samples { s i } ni =1 ∼ X and m s.t. 1 ≤ m ≤ n .
Sample spacing entropy estimation is based idea on following observation of m-spacings of random variable
[Equation]
There are many formulations of sample spacing entropy estimators but we use the one in (LearnedMiller et al., 2003).
Let S be one dimension r.v and assume we are given n samples { s i } ni =1 ∼ X and m s.t. 1 ≤ m ≤ n , then the sample spacing entropy estimator is given as follows
[Equation]
4. C ALIBRATION L OSS F UNCTION
In our case, the random variable is [ M ( X )]
( Y ) .
Given training data ( x k , y k ) we need to get samples [ M ( x i )]
( y i ) , and compute above expression ˆ H ( S ) and maximize it.
Note that,we want to make this part of training loop to achieve implicit calibration.
To do so, we need ordered samples to compute the Entropy in Eq. 6.
However, inherently, sorting is not a differentiable operation.
We use NeuralSort (Grover et al., 2019) as a differentiable relaxation to sorting.
Justiﬁcation for the sample spacing entropy estimator despite its need for differentiable sorting is that the r.v.
under consideration has support [0 , 1] and one has to use beta kernel (Chen, 1999) whose entropy doesn’t have a simple form while the spacing estimator does.
We summarize our Quantile Regularization algorithm below
Algorithm
Precondition: ( x k , y k ) are n i.i.d training instances and µ k , σ k
= MODEL w ( x k ) and DIFFSORT is any differentiable relaxation to sorting operation.
1: function C ALIBRATION L OSS FUNCTION ( y , µ , σ ) 2: 3: for k ← 1 to n do 4: Φ k ← Φ( µ k , σ k ) 5: c k ← Φ k
[ y k ] 6: end for 7: s ← DIFFSORT ( c ) 8: m ← √ n 9: e ← n − m n − m (cid:88)
i = log (cid:104) n +m ( s i + m − s i ) (cid:105) 10: return e 11: 12: end function
Assume that X , y is the training data.
Let ( µ w , σ w ) =
MODEL w ( X ) , where w denotes the parameters of the model, NLL ( y , µ w , σ w ) be the negative log likelihood and CL ( y , µ w , σ w ) be the quantile regularization based calibrated loss computed by Algorithm 1.
The overall loss function can be written as , where the hyperparameter λ controls calibration.
[Equation]
4. D EGENERATE B UT P ERFECTLY Q UANTILE C ALIBRATED M ODEL
Every notion of calibration has examples of models such that it is perfectly calibrated according to that speciﬁc notion but far from the ideal model.
We prove below that Quantile Calibration is no different.
Speciﬁcally, we give conditions under which is model is quantile calibrated and then use it to build a model that is degenerate in the sense that the model does not depend on the inputs while predicting output, but is a perfectly quantile calibrated model.
Claim 5 Let f be the marginal distribution of Y and F x = M [ x ] be the model’s predicted cumulative distribution for x ∈ X and f x be corresponding predicted probability density then if following holds, M is Quantile Calibrated
[Equation]
Now if we set f = f x ∀ x above condition easily holds.
So, a model that outputs marginal distribution of Y for every input x ∈ X , is perfectly quantile calibrated.
Note that this observation is general
[Table]
does not require
, for
As a simple example, consider f ( x, y ) =
N ( y | 5 x, 1) N ( x | 0 , 4) .
then the marginal distribution of y is f ( y ) =
(cid:82) ∞
N ( y | 5 x, 1) N ( x | 0 , 4) dx =
N ( y | 5 . 0 , 5 . 4 2 .
5 + 1 2 ) =
N ( y | 0 , 401) so the model which predicts N ( y | 0 , 401) regardless of input is perfectly quantile calibrated, but the true model is N ( y | 5 x, 1) .
So, a model can predict good conﬁdence intervals despite being far from ideal. Hence we need model to be both well-calibrated and sharp.  E XPERIMENTS We evaluate our approach on various regression datasets in terms of the calibration error as well as other standard metrics, such as root-mean-squared-error (RMSE) and negative log-likelihood (NLL).
5. l 2 Q UANTILE C ALIBRATION E RROR
Given any model M : X → ( Y → [0 , 1]) , we deﬁne the l 2 calibration error as follows.
Let us choose m equidistant points { p m } Mm =1 in (0 , 1] with p M = 1 .
Given a test set { x n , y n } Nn =1 , whose predictions are F n = F ( x n ) , the M-bin estimator of above integral will give us metric used in (Kuleshov et al., 2018).
[Equation]
5. UCI D ATA E XPERIMENTS
we consider two different architectures Dropout VI (Gal, 2016) and Deep Ensembles(Lakshminarayanan et al., 2017).
The dataset sizes ranges from 308 to 515345 and input feature dimensions ranges from 6 to 91 .
Every dataset, except Year Prediction MSD, is divided into 5 splits
[Table]
//github.com/occam-ra-zor/QR
5. D ISCUSSION
Both, in case of Dropout VI and Deep Ensembles, calibration error improves when trained with quantile regularizer.
Also in some cases, Isotonic Calibration increases the calibration error (Ex. for dropout VI see Concrete Strength, Yacht Hydrodynamics and for Deep Ensembles see Bouston
Housing and Yacht Hydrodynamics).
The reason is that unlike classiﬁcation calibration , recalibration dataset in case of regression calibration already satisﬁes monotonicity constraint, So isotonic regression completely overﬁts in case of smaller calibration datasets.
For pretty large calibration datasets like protein structure (calib dataset size m = 36 , 584 ) and Year Prediction MSD ( calib dataset size m = 463715 ), Isotonic Regression has better improvement .
Although, it can work well in such setting, PDF after calibration becomes unusable.
QR not only works well on small datasets but also works well in very large datasets like Year prediction MSD( nearly 60 % improvement in both architectures) .
Also, there is small reduction in RMSE,NLL in models trained with QR, but again it is negligible.
The reason for this is that if we plainly optimize for calibration there exists a non-degenerate case as solution apart from true distribution as we discussed in Sec. 4.
C ONCLUSION
We have proposed a black-box calibration loss function that can be used in any probabilistic regression models.
Unlike current methods for quantile calibration, our method is implicit in nature and does not require an additional calibration dataset and, more importantly, the smoothness of the PDF is not lost.
We conduct experiments to show effectiveness of our proposed method.