Temperature Scaling for Quantile Calibration
Abstract
Deep learning models are often poorly calibrated, i e., they may produce overconﬁdent predictions that are wrong, implying that their uncertainty estimates are unreliable.
While a number of approaches have been proposed recently to calibrate classiﬁcation models, relatively little work exists on calibrating regression models.
Temperature Scaling is one of the most popular methods for classiﬁcation calibration , often performing better than or comparably to more sophisticated methods.
We investigate the use of Temperature Scaling for regression calibration under notion of quantile calibration.
Introduction
One of main reasons probabilistic machine learning models are important is that they provide uncertainty estimates.
Uncertainty quantiﬁcation enables informed decision making.
The caveat however is that these decisions are reliable only if uncertainty is "reliable".
Calibration offers a precise mathematical deﬁnition of what reliability means.
Calibration is important for critical applications like healthcare, self-driving cars, etc.
Calibration in context of classiﬁcation models has been studied extensively.
Recently, [9] proposed a new notion of calibration for regression called Quantile Calibration .
We investigate the use of temperature scaling, which is one of the most popular classiﬁcation calibration methods, for regression calibration under notion of quantile calibration.
Background and Deﬁnitions 2.1 Classiﬁcation Calibration
Deﬁnition 1 (Binary Classiﬁcation Calibration) .
Given M : X → [0 , 1] , we say that M is calibrated if the following holds
[Equation]
If we pick a value between [0 , 1] , say 0 .
8 , then among all the examples whose predicted probability of belonging to class 1 is 0 .
8 , the proportion of examples that actually belong to class 1 should be 0 . 8 .
The objective of post-hoc calibration is to learn a mapping R s.t the new model R ◦ M is calibrated [17].
Usually, the calibration mapping is learned on the training set or the validation set.
However, ideally, it should be done on a separate calibration dataset.
Given such a calibration dataset { x k , y k } , a mapping R is learned on a re-calibration dataset (cid:110) M [ x k ] ,
[Equation]
which is essentially empirical approximation to Eq.1.
Based on the mappings considered, we can get different calibration methods.
With Logistic mapping we get Platt Scaling [16]; with Isotonic mapping, we get Isotonic Calibration [20].
etc.
The notion of calibration in classiﬁcation models has been extended to multi-class settings as well [10, 12, 17])
2.2 Temperature Scaling
Temperature Scaling is one of the state-of-art methods for classiﬁcation calibration.
Temperature
Scaling was originally conceived in context of knowledge distillation [8].
Despite its simplicity, it performs better than or comparaby to more sophisticated methods, like Bayesian Binning into
Quantiles, Isotonic Regression, Dirichlet Calibration, etc.
Essentially, temperature scaling learns a mapping of form R ( p ) =
p T
2.3 Quantile Calibration
Unlike classiﬁcation calibration, notion of calibration for regression is relatively new.
In one of the earliest attempts in this direction, [5] proposed various notions of calibration for regression but didn’t propose algorithms to recalibrate a miscalibrated model.
Recently, [9] proposed a new notion of calibration called Quantile Calibration based on Probabilistic Calibration in [5] and applied it to calibrate regression models. A probabilistic regression model can be seen as conditional PDF/conditional CDF.
In the rest of the paper, we express it as conditional CDF M : X → ( Y → [0 , 1]) .
So, M ( x ) denotes model’s predicted CDF for x ∈ X denoted as F x
Deﬁnition 2 ( Quantile Calibration ) .
Given a regression model M : X → ( Y → [0 , 1]) and X, Y jointly distributed as P , the model M is said to be Quantile Calibrated iff
[Equation]
An appealing aspect of quantile calibration is that we get calibrated conﬁdence intervals.
Just like post-hoc classiﬁcation calibration, the objective of post-hoc regression calibration is to learn a mapping R s.t.
R ◦ M is quantile calibrated.
The mapping R to be learned is given by following observation.
Theorem 1.
For any Model M : X → ( Y → [0 , 1]) , and given canonical calibration mapping
R ( p ) =
P (cid:2)
[ M ( X )]( Y ) ≤ p (cid:3) , R ◦
M is quantile calibrated
In addition to proposing above deﬁnition of calibration, [9] suggested use of Isotonic Calibration, well known technique for classiﬁcation calibration.
Given calibration dataset { x i , y i } mi =1 Isotonic
Calibration for quantile calibration is obtained by using isotonic regression on a re-calibration dataset
[Equation]
m i =1 .
Note that the only difference between isotonic calibration in classiﬁcation calibration and regression calibration is how recalibration dataset is constructed.
Quantitatively, calibration is measured by (cid:96) 2 quantile calibration error.
Given a test set { x n , y n } Nn =1 , with predictions are F n = M ( x n ) and m equidistant points { p m } Mm =1 in (0 , 1]
[Equation]
Temperature Scaling for Quantile Calibration
We propose to learn canonical calibration mapping of quantile calibration by temperature scaling
R ( x ) = x p .
Our proposal is justiﬁed by following simple lemma.
Claim 1.
let F be CDF of any r.v then for any α >
we have that F α is valid CDF again.
Proof.
T = 0.1 T = 0.
T = 0.7 T = 1.
T = 1.3 T = 2.
T = 9.
[Figure]
Such family of distributions are called exponentiated distributions which have been well-studied in the Statistics literature [7, 1, 2, 14]
An ideal desirable for the family of mappings is that it should be ﬂexible enough to correct wide ranges of mis-calibrations.
Importantly, it should contain y = x , because it shouldn’t harm already well-calibrated model.
Fig.
1 shows that the temperature scaling family is an ideal candidate.
Now for ﬁtting the parameter T , we use Eq.
[Equation]
A couple of key advantages of temperature scaling over isotonic regression are as follows
One of important drawbacks of isotonic regression is that, after isotonic calibration, the CDF losses its smoothness and the PDF becomes discontinuous, which is undesirable.
With temperature scaling, the smoothness is preserved 2.
It is much simpler and easier to use than isotonic regression as we are just ﬁtting a single parameter T.
In particular, the updated PDF and CDF values can be obtained in O (1) time, unlike isotonic regression
Experiments
We consider two different architectures - Dropout VI [3, 4] and Deep Ensembles [13].
The dataset sizes ranges from 308 to 515345 and input feature dimensions ranges from 6 to 91 .
Every dataset, except Year Prediction MSD, is divided into 5 splits whereas for Year Prediction MSD there is a single split where we train on 463715 points and test on 51630 points.
This experiment is repeatedtimes and averages are reported except for year prediction MSD.
We use 2 hidden layer network with 128 units with ReLU activation, and trained with Adam Optimizer with a learning rate of 10 − 2 for 64 epochs.
For Temperature Scaling we use the LBFGS optimizer and run it for 50 epochs, and for calibration dataset we use the training dataset.
The results are presented in Tab. 1 and Tab. 2.
The calibration plots are shown in Fig.
2 and Fig.
The average temperature is shown in Tab.
and the plot of calibration loss vs temperature is shown in Fig.
[Table]
and TS denotes the model after Temperature scaling
[Figure]
p r e d i c t e d
[Figure]
Deep Ensembles with Adversarial Training
[Table]
isotonic calibration and TS denotes the model after Temperature scaling
[Figure]
[Table]
[Figure]
[Table]
li b r a t i o n l o ss
D ee p E n s e m b l e s
[Figure]
Discussion
Tab. 1 and Tab. 2 show that Temperature Scaling doesn’t perform as well as expected in case of quantile calibration.
One would think one reason for this is because we are using training data as calibration dataset.
But this is not the case because in case of Isotonic regression there is two orders of magnitude improvement for large datasets like Protein,Year prediction MSD.
We conjecture two hypotheses as to why Temperature Scaling is not performing as expected:
Temperature Scaling is ﬂexible but may not be ﬂexible enough because we are just using single parameter ( T ), while isotonic regression is non-parametric method.
Using Calibration error as objective for ﬁtting T may be another
reason.
If this is the case, using better suited and properly regularized objective could alleviate the problem.
Conclusion
We investigated the performance of Temperature Scaling for regression calibration in context of quantile calibration and found that it doesn’t perform as well as it does for classiﬁcation calibration.
We have identiﬁed some potential reasons for this and it would be interesting to investigate them further to see whether this simple method can be useful for regression calibration.