C ONDITIONAL C OVERAGE E STIMATION FOR H IGHQUALITY P REDICTION I NTERVALS
Anonymous authors
Paper under double-blind review
A BSTRACT
Deep learning has achieved state-of-the-art performance to generate high-quality prediction intervals (PIs) for uncertainty quantiﬁcation in regression tasks.
The high-quality criterion requires PIs to be as narrow as possible, whilst maintaining a pre-speciﬁed level of data (marginal) coverage.
However, most existing works for high-quality PIs lack accurate information on conditional coverage, which may cause unreliable predictions if it is signiﬁcantly smaller than the marginal coverage.
To address this problem, we propose a novel end-to-end framework which could output high-quality PIs and simultaneously provide their conditional coverage estimation.
In doing so, we design a new loss function that is both easyto-implement and theoretically justiﬁed via an exponential concentration bound.
Our evaluation on real-world benchmark datasets and synthetic examples shows that our approach not only outperforms the state-of-the-arts on high-quality PIs in terms of average PI width, but also accurately estimates conditional coverage information that is useful in assessing model uncertainty.
I NTRODUCTION
Prediction interval (PI) is poised to play an increasingly prominent role in uncertainty quantiﬁcation for regression tasks (Khosravi et al., 2010; 2011; Galv´an et al., 2017; Rosenfeld et al., 2018;
Tagasovska & Lopez-Paz, 2018; 2019; Romano et al., 2019; Wang et al., 2019; Kivaranovic et al., 2020).
A high-quality PI should be as narrow as possible, whilst maintaining a pre-speciﬁed level of data coverage or marginal coverage (Pearce et al., 2018).
Compared with PIs obtained based on coverage-only consideration, the “high-quality” criterion is beneﬁcial in balancing between marginal coverage probability and interval width.
However, the conditional coverage given a feature, which is critical for making reliable context-based decisions, is unassessed and missing in most existing works on high-quality PIs.
In the presence of heteroskedasticity and model misspeciﬁcation, the marginal coverage can be very different from the conditional coverage at a given point, which affects the downstream decision-making task that relies on the uncertainty information provided by the PI. Our main goal is to meaningfully incorporate and assess conditional coverages in high-quality PIs.
Conditional coverage estimation is challenging for two reasons.
First is that the natural evaluation metric of conditional coverage error , an L p distance between the estimated and ground-truth conditional coverages, is difﬁcult to compute as it requires obtaining the conditional probability given feature x , which is arguably as challenging as the regression problem itself.
Our ﬁrst goal in this paper is to address this issue by developing a new metric called calibration-based conditional coverage error for conditional coverage estimation measurement.
Our approach is inspired from the calibration notion in classiﬁcation (Guo et al., 2017).
The basic idea is to relax conditional coverage at any given point to being averaged over all points that bear the same estimated value.
An estimator satisfying the relaxed property is regarded as well-calibrated.
In regression, calibration-based conditional coverage error provides a middle ground between the enforcement of marginal coverage (lacking any conditional information) and conditional coverage (computationally intractable).
Compared with conditional coverage, this middle-ground metric can be viewed as a “dimension reduction” of the conditioning variable from the original sample space to the space [0 , 1] , so that we can easily discretize to compute the empirical metric values.
The second challenge is the discontinuity in the above metrics that hinders efﬁcient training of PIs that are both high-quality and possess reliable conditional coverage information.
To address this, we design a new loss function based on a combination of the high-quality criterion and a coverage assessment loss .
The latter can be ﬂexibly added as a separate module to any neural network (NN) used to train PIs.
It is based on an empirical version of a tight upper bound on the coverage error in terms of a Kullback–Leibler (KL) divergence, which can be readily employed for running gradient descent.
We theoretically show how training with our proposed loss function attains this upperbounding value via a concentration bound.
We also demonstrate the empirical performance of our approach in terms of PI quality and conditional coverage assessment compared with benchmark methods.
Summary of Contributions : (1) We identify the conditional coverage estimation problem as a new challenge for high-quality PIs and introduce a new evaluation metric for coverage estimation.
We propose an end-to-end algorithm that can simultaneously construct high-quality PIs and generate conditional coverage estimates.
In addition, we provide theoretical justiﬁcations on the effectiveness of our algorithm by developing concentration bounds relating the coverage assessment loss and conditional coverage error.
(3) By evaluating on benchmark datasets and synthetic examples, we empirically demonstrate that our approach not only achieves high performance on conditional coverage estimation, but also outperforms the state-of-the-art algorithms on high-quality PI generation.
E VALUATING C ONDITIONAL C OVERAGE FOR H IGH -Q UALITY PI S
Let X ∈ X and Y ∈ Y ⊂ R be random variables denoting the input feature and label, where the pair ( X, Y ) follows an (unknown) ground-truth joint distribution
π
( X, Y ) .
Let π ( Y | X ) be the conditional distribution of Y given X .
We are given the training data D := { ( x i , y i ) , i = 1 , 2 , · · · , n } where ( x i , y i ) are i.i.d.
realizations of random variables ( X, Y ) .
A PI refers to an interval [ L ( x ) , U ( x )] where L , U are two functions mapping from X to Y trained on the data D .
[ L ( x ) , U ( x )] is called a PI at prediction level 1 − α ( 0 ≤ α ≤ 1 ) if its marginal coverage is not less than 1 − α , i.e., P [ Y ∈
[ L ( X ) , U ( X )] | L, U ] ≥ 1 − α where P is with respect to a new test point ( X, Y ) ∼ π .
We say that [ L ( x ) , U ( x )] is of high-quality if its marginal coverage attains a pre-speciﬁed target prediction level and has a short width on average.
In particular, a best-quality PI at prediction level 1 − α is an optimal solution to the following constrained optimization problem:
[Equation]
The high-quality criterion has been widely adopted in previous work (see Section 6).
However, this criterion alone may fail to carry important model uncertainty information at speciﬁc test points.
Consider a simple example where x ∼ Uniform [0 , 1] , y = 0 for x ∈ [0 , 0 .
95] and y
| x ∼ Uniform [0 , 1] for x ∈ (0 .
Then according to equation 2.1, a best-quality 95% PI is precisely L ( x ) =
U ( x ) = 0 for all x ∈ [0 , 1] .
This PI has nonconstant coverage if we condition at different points (1 for x ∈ [0 , 0 . 95] and 0 for x ∈ (0 .
95 , 1] ), and can deviate signiﬁcantly from the overall coverage 95%.
More examples to highlight the need of obtaining conditional coverage information can be found in our numerical experiments in Section 5.1.
To mitigate the drawback of the high-quality criterion, we deﬁne:
Deﬁnition 2.1 (Conditional Coverage and Its Estimator) .
The conditional coverage associated with a PI [ L ( x ) , U ( x )] is A ( x ) :
= P [ Y ∈
[ L ( X ) , U ( X )] | L, U, X = x ] for a.e. x ∈ X , where P is taken with respect to π ( Y | X ) .
For a (conditional) coverage estimator ˆ P , which is a measurable function from X to [0 , 1] , we deﬁne its L p conditional coverage error ( (cid:103) CE p ) as
[Equation]
where the L p -norm is taken with respect to the randomness of X ( 1 ≤ p ≤ + ∞ ).
Note that evaluating (cid:103) CE p relies on approximating the conditional coverage A ( x ) , which can be as challenging as the original prediction problem.
To address this, we leverage the similarity of estimating A ( x ) to generating prediction probabilities in binary classiﬁcation, which motivates us to borrow the notion of calibration in classiﬁcation.
This idea is based on a relaxed error criterion by looking at the conditional coverage among all points that bear the same coverage estimator value, instead of conditioning at any given point.
The resulting error metric then only relies on probabilities conditioned on variables in a much lower-dimensional space [0 , 1] than X .
To explain concretely, we introduce a “perfect-calibrated coverage estimator” as:
Deﬁnition 2.2 (Perfect Calibration) .
A coverage estimator ˆ P is called a perfect-calibrated coverage estimator associated with [ L ( x ) , U ( x )] if it satisﬁes
[Equation]
where a.e. is with respect to the probability measure on [0 , 1] induced by the random variable ˆ P ( X ) .
Equation 2.2 means that a point x with conditional coverage estimate ˆ P ( x ) = p has an average coverage of precisely p , among all points in X that possess the same conditional coverage estimated value.
That is, the average coverage of the PI restricted on the subset { x ∈ X :
ˆ P ( x ) = p } should be precisely p .
Corresponding to Deﬁnition 2.2, we deﬁne:
Deﬁnition 2.3 (Calibration-based Error) .
An L p ( 1 ≤ p ≤ + ∞ ) calibration-based conditional coverage error, or coverage error for short ( CE p ), of a coverage estimator ˆ P is:
[Equation]
where L p -norm is taken with respect to the randomness of ˆ P ( X ) .
In the above deﬁnition the conditional probability P [ Y ∈
[ L ( X ) , U ( X )] | L, U, ˆ P ( X )] is a measurable function of random variable ˆ P ( X ) , say γ ( ˆ P ( X )) .
By a change of variable,
[Equation]
where F ˆ P ( X ) is a probability distribution of ˆ P ( X ) on [0 , 1] .
Here, CE p only requires estimating γ ( t ) for t ∈ [0 , 1] , which can be done easily by discretizing [0 , 1] for empirical calculation.
We call the empirical L p calibration-based conditional coverage error ECE p .
More details of this empirical error can be found in Appendix A.4.
A calibration-based error CE p provides a middle ground between the enforcement of marginal coverage and conditional coverage.
The ground-truth conditional coverage is perfectly calibrated, but not vice versa.
However, if we enforce the perfect calibration criterion for a coverage estimator to hold when restricted to any measurable subset in X , then the choice of estimator will reduce uniquely to the conditional coverage (Deﬁnition A.1 and Lemma A.2).
In this sense, CE p is an error metric that is a natural relaxation of (cid:103) CE p , and although less precise, CE p is computationally much more tractable than (cid:103) CE p .
Evaluation Metric for Coverage Estimator.
We use ECE 1 as the primary evaluation metric to measure the quality of a coverage estimator.
A high ECE 1 value of a coverage estimator indicates an unreliable coverage estimation while a small ECE 1 value indicates that the coverage estimator is close to the perfect-calibrated property.
Ideally, an effective algorithm should output a coverage estimator with a small ECE 1 value.
M ETHODOLOGY
We propose a novel end-to-end algorithm, named coverage assessment network (CaNet), to simultaneously generate a coverage estimator along with the high-quality PI.
As illustrated in Figure 1, our
CaNet consists of two major modules: (1) predictor module and (2) coverage assessment module (Ca-Module).
The predictor module provides the upper and lower bound of the estimated PIs.
Meanwhile, the Ca-Module is added to the output layer to access the conditional coverage information of PIs from the predictor module.
Our model is jointly optimized by three loss functions: coverage assessment loss L CA , intervals width loss L IW , and coverage probability loss L CP .
Beneﬁted from these powerful modules, the CaNet can generate and validate the coverage estimator at the same time without any requirement for further post-processing steps.
Coverage Assessment Module .
Our Ca-Module consists of two neurons fully connected to the last hidden layer to estimate the conditional coverage.
After passing through the softmax activation function, it outputs a two-point probability distribution ˆ P ( x ) , 1 − ˆ P ( x ) where ˆ P ( x ) is the coverage estimator of the PI from the predictor module.
Our Ca-Module can be easily integrated into the output layer of deep neural networks to estimate their conditional coverage.
Loss Function Design and Tuning Procedure.
Our loss function is a sum of the predictor loss and the coverage loss .
The predictor loss aims to narrow the prediction intervals as much as possible, while maintaining a speciﬁed marginal coverage of data.
Inspired by (Khosravi et al., 2010; 2011;
Pearce et al., 2018; Rosenfeld et al., 2018), our predictor loss is formed by the sum of intervals width (IW) loss L IW and coverage probability (CP) loss L CP : [Equation]
[Equation]
where k i indicates whether each data point has been captured by the PIs: k i = 1 if L ( x i ) ≤ y i ≤
U ( x i ) and k i = 0 otherwise.
˜
k
i is a soft version of k i , which is deﬁned as: ˜ k
i :
=
σ ( λ 3 ( U ( x i ) − y i ))
· σ ( λ 3 ( y i − L ( x i ))) , where λ 3 ≥ 0 is a tunable parameter and σ ( t ) : = 1 1+ e − t is the sigmoid function.
Therefore, L CP is a soft version of CP that can be used for gradient descent.
Associated with the Ca-Module, we introduce a coverage assessment loss L CA to estimate the conditional coverage:
[Equation]
We will show in Section 4 that the expectation of coverage assessment loss L CA provides an upper bound for both the conditional coverage error (Deﬁnition 2.1) and calibration-based conditional coverage error (Deﬁnition 2.3). Hence, minimizing L CA contributes to the recovery of the conditional coverage. In order to run gradient-based methods, we replace the discrete indicator ( k i , 1 − k i ) in L CA with its soft version (˜ k
i , 1 − ˜ k i [Equation] [Equation]
Our total loss function for the CaNet is deﬁned as:
[Equation]
where λ 1 ≥ 0 , λ 2 ≥ 0 are tunable parameters.
We propose an easy-to-implement yet effective tuning procedure to pick up these parameters.
Please refer to Appendix D for more algorithm details.
Deep Ensembles.
Following previous research (Lee et al., 2015; Lakshminarayanan et al., 2017;
Pearce et al., 2018; Fort et al., 2019; Ovadia et al., 2019; Gustafsson et al., 2020; Pearce et al., 2020), we apply the deep ensemble technique to provide more robust and better results.
During the training period, with the same hyperparameters λ
i , i = 1 , 2 , 3 , m networks are trained with different [Equation] initializations.
The prediction results from i -th network are denoted as: ([ L i ( x ) , U i ( x )] , ˆ P
i ( x ) , 1 − ˆ P
i ( x )) .
Finally, the output from CaNet is:
Lower
[Equation]
estimator
[Equation]
T HEORETICAL A NALYSIS
In this section, we provide theoretical insights that minimizing L CA is equivalent to minimizing a tight upper bound of the conditional coverage error and thus can recover the true conditional coverage.
To achieve this, we ﬁrst prove that both CE p and (cid:103) CE p are bounded above by the expectation of a Kullback–Leibler divergence-type random variable K 1 ( X ) .
Then we show that L CA is an empirical counterpart of K 1 ( X ) and establish a concentration bound between L CA and E
[ K 1 ( X )] .
Theorem 4.1.
Let A ( x ) :
= P [ Y ∈
[ L ( X ) , U ( X )] | L, U, X
= x ] be the conditional coverage in
Deﬁnition 2.1.
Let K ( x ) =
A ( x ) log (cid:16) A
( x ) ˆ P ( x ) (cid:17) + (1 − A ( x )) log (cid:16) 1 − A ( x )
1 − ˆ P ( x ) (cid:17) .
Then
[Equation]
where α p = 1 , ∀ 1 ≤ p ≤ 2 and α p = 2 p , ∀ 2 ≤ p ≤ + ∞ .
Moreover, the inequality is attainable if, e.g., ˆ P ( x ) equals the conditional coverage A ( x ) .
From Theorem 4.1, we see that minimizing E
[ K ( x )] is equivalent to minimizing a tight upper bound for the coverage error.
For every x , K ( x ) is the Kullback–Leibler divergence between the distributions represented by ( ˆ P ( x ) ,
1 − ˆ P ( x )) and ( A ( x ) , 1 − A ( x )) .
K ( x ) = K 0 ( x ) + K 1 ( x ) , where
K 0 ( x ) =
A ( x ) log( A ( x ))
+ (1 − A ( x ))
log(1 − A ( x )) and K 1 ( x ) = − A ( x ) log( ˆ P ( x )) − (1 −
A ( x )) log(1 − ˆ P ( x )) .
Minimizing E [ K ( X )] over ˆ P is equivalent to minimizing E
[ K 1 ( X )] .
The type of results in Theorem 4.1 that bounds an L p conditional coverage error via a Kullback-Leiblertype error is new as far as we know.
Next, to show L CA approximates E [ K 1 ( X )] , we need the following assumptions:
Assumption 4.2.
The four classes of functions ([ L ( x ) , U ( x )] , ˆ P ( x ) , 1 − ˆ P ( x )) output by the neural network (NN) in Figure 1 have ﬁnite VC dimensions, say they are bounded above by V 0 .
Assumption 4.2 holds for a wide range of NNs (e.g., Theorem 8.14 in Anthony & Bartlett (2009),
Theorem 7 in Bartlett et al.
In particular, it holds for the one we adopt in the experiments (where we use the ReLU-activated NN to construct ψ i , i = 1 , 2 , 3 , 4 ; see Section 5):
Theorem 4.3.
Suppose ψ i , i = 1 , 2 , 3 , 4 are the pre-activated output neurons of the NN in Figureusing the ReLU activation function.
Then Assumption 4.2 holds.
Moreover, suppose the NN has W parameters and U computation units (nodes).
Then V
= O ( WU ) .
Assumption 4.4.
| log( ˆ P ( x ))
| ≤ M , | log(1 − ˆ P ( x ))
| ≤ M for all x and ˆ P .
This is a natural assumption in practice because log( ˆ P ( x )) and log(1 − ˆ P ( x )) are replaced by log( ˆ P ( x ) + (cid:15) ) and log(1 − ˆ P ( x ) + (cid:15) ) respectively to avoid explosion when implementing the algorithm.
In particular, in our experiments in Section 5, (cid:15) = 0 .
1 6 and thus M = 14 .
Let
F = { f ( x, y ) = I y ∈
[ L ( x ) ,U ( x )] : L, U are output by the NN } ,
G = { ˆ P ( x ) : ˆ P is output by the NN } .
Theorem 4.5.
Suppose Assumptions 4.2 and 4.4 hold.
The training data D = { ( x i , y i ) , i = 1 , 2 , · · · , n } where ( x i , y i ) are i.i.d.
samples ∼ π .
Recall that the (hard) coverage estimator assessment loss is L CA = − 1 n (cid:80) ni
=1 (cid:16) f
( x i , y i ) log( ˆ P ( x i ))
+ (1 − f ( x i , y i ))
log(1 − ˆ P ( x i )) (cid:17) .
Then for any t > 0 , we have P
[Equation]
where C ∗ only depends on V 0 in Assumption 4.2.
Theorem 4.5 shows that the coverage assessment loss approximates E
[ K 1 ( x )] well with an exponential tail bound.
The difﬁculty in analyzing Theorem 4.5 lies in the fact that the hypothesis classes in Assumption 4.2 (which are constructed by the NN) are different from the hypothesis class used in L CA .
To overcome this difﬁculty, we use the theory of VC-subgraph classes to connect the VC dimension among multiple hypothesis classes, including the class of ψ i , the four classes of output functions, and F , log G .
Then we establish the covering number bound for the class F and log G , and ﬁnally prove Theorem 4.5.
To conclude, minimizing E [ K 1 ( X )] over ˆ P is equivalent to minimizing E [ K ( X )] , which in turn is minimizing a tight upper bound for the coverage assessment loss.
Our coverage assessment loss empirically approximates E [ K 1 ( X )]
well, so that its minimization can ultimately reduce the conditional coverage error.
E XPERIMENTS
Experimental Setup.
We empirically verify the effectiveness of our proposed CaNet on both synthetic examples and benchmark regression datasets.
These datasets have been widely used for the evaluation of methods in regression tasks (Hern´andez-Lobato & Adams, 2015; Gal & Ghahramani, 2016;
Lakshminarayanan et al., 2017; Rosenfeld et al., 2018; Pearce et al., 2018; Zhu et al., 2019).
In addition, we adopt the same experimental procedure in Pearce et al.
(2018) for data normalization and dataset splitting.
To avoid overﬁtting, we apply a simple network architecture with only 2 hidden layers and each hidden layer has 64 neurons.
For each hidden layer, the ReLU activation function is applied to capture the non-linear features.
We empirically set the ensemble number m to 5, as the smallest number leading to a stable prediction results.
Please refer to Appendix E for implementation details, including those for baseline algorithms.
Evaluation Metrics.
To evaluate the conditional coverage estimation of our CaNet, we examine the quality of our Ca-Module measured by the empirical coverage error ECE 1 over a partition ∆ .
∆ is constructed by equally dividing the width of [0 , 1] into M sub-intervals.
The value of M depends on the size of the dataset, which is determined by the following strategy:
M = min {(cid:98) the number of data in validation / 50 (cid:99) + 1 , 20 } .
In addition, we also report the IW and
CP , where IW = 1 n (cid:80) ni
=
( U ( x i ) − L ( x i )) (equation 3.1) and CP = 1 n (cid:80) ni
=
k
i , (equation 3.2) to show the effectiveness of our predictor module under the high-quality criteria.
5. C ONDITIONAL C OVERAGE ON S YNTHETIC E XAMPLES
In this section, we conduct a series of experiments on synthetic examples to directly compare our prediction results with the ground-truth conditional coverage.
In these examples, the conditional coverage can be analytically calculated under the known data distribution.
Figure 2 compares the conditional coverage with our predicted coverage under the following settings: x ∼ Uniform [ − 2 , 2] and y | x is drawn from f i ( x ) = 13 sin( x ) +
ε i ( x ) , x ∈ [ − 2 , 2] where ε 1 ( x ) = 0 .
1 × N (0 , 1) , ε 2 ( x ) = 0 . 1 | x | × N (0 , 1) , ε 3 ( x ) = 0 . 1 | x | × t 4 .
N (0 , 1) is the standard Gaussian variable and t 4 is the standard t random variable with 4 degrees of freedom.
Then, the conditional coverage in Deﬁnition 2.1 can be analytically calculated as:
[Equation]
NNKCDE
[Table]
CP
[Table]
CP [Table] [Table] where F i is the cumulative distribution function of N (0 , 0 .
1 2 ) for i
= 1 , N (0 , (0 . 1 x ) 2 ) for i = 2 , and 0 .
1 | x | × t 4 for i = 3 .
As shown in Figure 2, the conditional coverages of high-quality PIs at different points diverge among each other and they deviate from the marginal coverage.
Thus, having access to the marginal coverage for the whole dataset is not sufﬁcient for decision making, which highlights the need of conditional coverage.
In addition, the predicted coverage estimator of our model is highly consistent with the conditional coverage on all of the synthetic examples.
These results conﬁrm that our CaNet can accurately estimate the conditional coverage on noisy datasets.
5. P ERFORMANCE OF PI S ON B ENCHMARK D ATASETS
In this section, we compare the predictor module of our CaNet on real-world benchmark datasets with following baseline algorithms: (1) nearest-neighbors kernel conditional density estimation (NNKCDE)
(Dalmasso et al., 2020), (2) quantile regression forest (QRF) (Meinshausen, 2006), (3) split conformal learning (SCL)
(Lei et al., 2018) and (4) the quality-driven PI method (QD-Ens) (Pearce et al., 2018).
We quote the results from (Pearce et al., 2018) as a comparison since we share the same experiment setup.
Table 1 reports the results of CP , IW and ECE 1 for generating PIs at 95% prediction level on benchmark datasets.
We employed the criteria in Pearce et al.
(2018) to evaluate the performance of PIs: the best IW is achieved by the model with the smallest IW value among those with CP ≥ 95% .
As can be seen, our model achieves the best performance on
PI generation under the high-quality criteria.
With special consideration on interval width quality, it obtains the smallest average interval width ( IW ) while maintaining high coverage probability ( CP ≥ 95% ) on all datasets.
5. P ERFORMANCE OF C OVERAGE E STIMATOR ON B ENCHMARK D ATASETS
Coverage for 95% PIs.
We use ECE 1 to evaluate the coverage estimation performance on realworld datasets as the conditional coverage is unknown.
As shown in Table 1, the ECE 1 on all experiments are generally around or less than 1%, with better performance on larger datasets.
The coverage estimators produced by CaNet have small ECE 1 values, which are very close to the perfect-calibrated coverage estimators (Deﬁnition 2.2).
Compared with ECE 1 values obtained from the state-of-the-art algorithms in classiﬁcation tasks (Guo et al., 2017; Kull et al., 2019), the ECEvalues from CaNet are similar and sometimes less than their post-calibrated ECE 1 results (usually around 1% to 3% ), even though the size of most regression datasets are smaller than the classiﬁcation datasets.
These results further demonstrate that our CaNet can accurately estimate the coverage information of 95% high-quality PIs on real-world regression tasks.
Coverage for PIs at Different Prediction Levels.
We conduct multiple experiments on different
PI prediction levels to show the robustness of our CaNet.
By only modifying the parameter λ 1 in
Equation 3.5, our Ca-Module could get access to different levels of coverage probability.
Tablereports the CP, IW and ECE 1 values from the CaNet at different PI prediction levels on three benchmark datasets.
Results for more datasets can be found in Appendix E. As can be seen, all
ECE 1 values in Table 2 are fairly small ( ∼ 1% ), demonstrating the stability of our proposed model.
Thus, our CaNet can provide accurate coverage estimation on PIs at different prediction levels.
These results demonstrate the robustness of our CaNet on real-world datasets, further suggesting its broad applicability.
R ELATED W ORK
Prediction Interval Estimation.
High-quality PIs, which can be viewed via a constrained optimization problem where the constraint concerns marginal coverage and the objective is the PI width, has been extensively studied in Khosravi et al.
Galv ´ an et al.
; Pearce et al.
Rosenfeld et al.
; Zhang et al.
Zhu et al.
Such intervals are in the same spirit as the highest density intervals in statistics (Box & Tiao, 2011).
While powerful, these approaches could not directly provide the conditional coverage information investigated in this work.
Coverageonly criteria, on the other hand, focus solely on coverage satisfaction as the guarantee.
These approaches include conformal learning (CL) and its conditional variants (Vovk et al., 2005; 2009; Lei & Wasserman, 2014; Lei et al., 2015; 2018; Kuchibhotla & Ramdas, 2019; Romano et al., 2019;
Barber et al., 2019a;b).
CL is desirably distribution- or model-free, and in many cases enjoys ﬁnitesample guarantees.
However, unlike high-quality PIs, they do not explicitly account for the interval width as a quality metric.
We also mention conditional density estimation (Holmes et al., 2007;
Dutordoir et al., 2018; Izbicki & Lee, 2016; Dalmasso et al., 2020; Freeman et al., 2017; Izbicki et al., 2017) and closely relatedly quantile regression (Koenker & Hallock, 2001; Meinshausen, 2006) as PI construction approaches by converting from the estimated conditional quantile function.
These approaches focus on the quality of conditional distribution/quantile, instead of the high-quality criterion only.
In this work, we use deep learning to construct high-quality PIs (Khosravi et al., 2010;
Pearce et al., 2018; Kivaranovic et al., 2020).
Uncertainty Measurement in Deep Learning.
The Bayesian framework offers principled approaches for model uncertainty measurement by computing the posterior distribution over the NN parameters (MacKay, 1992; Neal, 2012).
These approaches can also be used to construct PIs.
However, they provide a different perspective from the frequentist view taken in this paper, and they focus on the parameter uncertainty of the NN instead of the coverage over a test point.
In addition, exact Bayesian inference is computationally intractable for deep neural networks, making it less practical to implement.
Gal & Ghahramani (2016) applied a Monte Carlo dropout method to proxy the inference.
Directly generated from the networks, softmax response is also commonly used for uncertainty measurement on deep learning models (Bridle, 1990;
Lakshminarayanan et al., 2017; Geifman et al., 2018; Sensoy et al., 2018;
Ozbulak et al., 2018).
Moreover, Niculescu-Mizil & Caruana (2005) showed that NNs typically produce well-calibrated probabilities on binary classiﬁcation tasks without the need of any post-hoc techniques.
In this paper, we also follow this line of work and use the softmax output to access the model uncertainty information.
C ONCLUSION
In this paper we identify and investigate the conditional coverage estimation problem for highquality PIs, which is critical for risk-based decision making in regression ﬁelds.
To address the challenge we propose an end-to-end algorithm with two powerful modules: coverage assessment module and predictor module.
Beneﬁted from these powerful modules, our model can generate and validate the coverage estimator without any requirement for further post-processing steps.
In addition we conduct theoretical analysis to show the effectiveness of our proposed model.
Experimental results on synthetic examples and benchmark datasets further demonstrate that our model can robustly provide accurate coverage estimation while simultaneously producing a high-quality
PI.
Moreover, our Ca-Module can be easily integrated into other deep-learning-based algorithms to get access to their coverage information, opening up more opportunities for broad applications.
In the future, we will extend our work by conducting comparison studies with Bayesian methods
A F URTHER D ETAILS ON C OVERAGE E STIMATOR AND C OVERAGE E RROR A. C OVERAGE P ROBABILITY T YPES OF PI S
Zhang et al.
(2019) introduce the following four coverage probability types of PIs.
In general, most of the coverage in PIs considered in the literature falls into one of these types.
Type I: P [ Y ∈
[ L ( X ) , U ( X )]] (marginal coverage); Type II: P [ Y ∈
[ L ( X ) , U ( X )] | L, U ] (conditional coverage given the PI); Type III: P [ Y ∈
[ L ( X ) , U ( X )]
| X = x ] (conditional coverage given X = x ); Type IV: P [ Y ∈
[ L ( X ) , U ( X )] | L, U, X
= x ] (conditional coverage given the PI and X = x ).
Note that in high-quality criteria, only Type II coverage is considered in the constraint but Type IV coverage is lacking.
Since Type I and III are not considered in our paper, for simplicity Type II coverage is called the marginal coverage, and Type IV coverage is called the conditional coverage in Deﬁnition 2.1.
Throughout Appendix, we let A ( x ) :=
P [ Y ∈
[ L ( X ) , U ( X )] | L, U, X
= x ] denote the conditional coverage in Deﬁnition 2.1.
Moreover, since we only concern Type II and IV coverage in this work, we make the following convention.
Throughout the Appendix A-B, P and E should be understood as probability and expectation conditional on L, U .
So for A ( x ) , we could simply write A ( x ) :
= P [ Y ∈
[ L ( X ) , U ( X )]
| X = x ] and omit “conditional on L, U ”.
A. T HE T ERMINOLOGY “P ERFECT -C ALIBRATED ”
The terminology “perfect-calibrated” is borrowed from the conﬁdence calibration in classiﬁcation tasks.
We ﬁrst review the name of “conﬁdence” in classiﬁcation.
Conﬁdence calibration is the problem of predicting probability estimates representative of the true correctness likelihood (Guo et al., 2017).
Intuitively, an reliable conﬁdence should reﬂect the true correctness likelihood of the prediction (Kumar et al., 2019).
For example, given 100 predictions, each with conﬁdence of 0.8, we expect that 80 should be correctly classiﬁed (Guo et al., 2017).
Now let h be the prediction of any models, which is a map from X to Y , trained on the data D .
According to their deﬁnition, the “best” conﬁdence map ˆ P should be the true probability of correctness:
[Equation]
which is a measureable function from X to [0 , 1] .
In particular, we consider h as a PI in regression tasks and “correctness” of a PI is naturally deﬁned as the success of coverage on the outcome Y .
Then the right hand side of Equation (A.1) becomes
[Equation]
which is the conditional coverage in our Deﬁnition 2.1.
In addition, Guo et al.
(2017) introduce the perfect-calibrated conﬁdence as follows:
[Equation]
where ˆ Y is the class prediction and
ˆ Y =
Y means that the predicted and true class label coincide.
Obviously, if ˆ P ( X ) =
P ( ˆ Y ( X ) =
Y | X ) , i.e., the “best” conﬁdence, then the above equality holds.
Transferring this idea into PIs, we can naturally deﬁne the perfect-calibrated coverage estimator as
[Equation]
which is the conditional coverage in our Deﬁnition 2.2.
A. D ETAILS ON C OVERAGE E STIMATOR
A perfect-calibrated coverage estimator inherits some properties of a conditional coverage.
For example, both of them have the following interpretation: If we have 1000 testing points for a PI, each with the same conditional/perfect-calibrated coverage 0.9, then approximately 900 of them are correctly covered by the PI.
Note that the conditional coverage is uniquely deﬁned, but a perfectcalibrated coverage estimator is not necessarily so.
Moreover, we have the following facts: (a)
The conditional coverage is always perfect-calibrated, but not vice versa.
(b) A perfect-calibrated coverage estimator can be viewed as an averaged conditional coverage.
(c) A perfect-calibrated coverage estimator is less “informative” than the conditional coverage.
A perfect-calibrated coverage estimator is an averaged conditional coverage.
Let ˆ P be a (general) coverage estimator.
We have
[Equation]
Suppose ˆ P is a perfect-calibrated coverage estimator.
Then we have
[Equation]
which implies that ˆ P ( x ) is a weighted average of A ( t ) over the set
ˆ P − 1 ( ˆ P ( x )) with weights based on the marginal distribution of X .
A conditional coverage is perfect-calibrated.
If ˆ P ( x ) =
A ( x ) , then A ( t ) =
A ( x ) for any t ∈
A − 1 ( A ( x ))
so
[Equation]
This shows that A ( x ) must be a perfect-calibrated coverage estimator.
Another way to see this is taking conditional expectation given A ( X ) =
p in the Deﬁnition 2.1.
Then we get
[Equation]
by the tower property.
A perfect-calibrated coverage estimator may be less informative and may not be the conditional coverage.
Suppose we have a PI [ L ( X ) , U ( X )] at the exact prediction level of 1 − α
, i.e., P [ Y ∈
[ L ( X ) , U ( X )]]
= 1 − α .
Then the constant coverage estimator
[Equation]
can be viewed as an average coverage estimator over the entire space X .
It is a perfect-calibrated coverage estimator since by deﬁnition,
[Equation]
But it is not a conditional coverage in general (e.g., the second synthetic example in Section 5).
Now we give an extension of the deﬁnition of “perfect-calibrated” coverage estimator, allowing it to be deﬁned on any measurable subsets.
Deﬁnition A.1.
A coverage estimator ˆ P is called a perfect-calibrated coverage estimator on a measurable subset S ⊂ X with P ( S ) > 0 associated with [ L ( x ) , U ( x )] if it satisﬁes
[Equation]
where a.e. is with respect to the probability measure on [0 , 1] induced by the random variable ˆ P ( X )
| S .
Note that the conditional probability space is standard: ( S , F S :=
{ A ∩ S : A ∈
F} , P S ( A ∩ S ) :
= P ( A |S )) .
(As our convention in Section A.1, we will omit “conditional on L, U ” for simplicity.)
Lemma A.2.
(a) A coverage estimator is the conditional coverage if and only if it is a perfectcalibrated coverage estimator on any positive-probability measurable subset S of X .
(b) Suppose ˆ P is a perfect-calibrated coverage estimator on two disjoint positive-probability measurable subsets S 1 , S 2 .
Then ˆ P is a perfect-calibrated coverage estimator on S 1 ∪ S 2 .
Proof.
(a) The proof can be found in Lemma A.4.
(b) We note that by law of total probability,
[Equation]
Hence ˆ P is a perfect-calibrated coverage estimator on S 1 ∪ S 2 .
Lemma A.2(a) is motivated from a theoretical point of view.
It provides a guidance that in order to well resemble the conditional coverage, an estimator should be perfect-calibrated on as many subsets on the feature space as possible.
A. D ETAILS ON C OVERAGE E RROR
In Section 2, we have introduced CE p to quantify the discrepancy between a coverage estimator and a perfect-calibrated coverage estimator, and (cid:103) CE p to quantify the discrepancy between a coverage estimator and the conditional coverage.
We note that by H ¨ older’s inequality
[Equation]
A larger value of p corresponds to a larger CE value.
Continuing Deﬁnition A.1, we can further introduce the calibration-based conditional coverage error on a measurable subset as follows:
Deﬁnition A.3.
An L p ( 1 ≤ p ≤ + ∞ ) calibration-based conditional coverage error, or coverage error for short, of a coverage estimator ˆ P on a measurable subset S ⊂ X with P ( S ) > 0 is deﬁned as:
[Equation]
where L p -norm is taken with respect to the randomness of ˆ P ( X ) on the conditional probability space ( S , F S :
= { A ∩ S : A ∈ F} , P S [Equation] A [Equation]
S ) : = P [Equation] A
[Equation]
S )) .
In particular, we have CE p :=
CE p ( X ) .
Lemma A.4.
A coverage estimator ˆ P is the conditional coverage if and only if its coverage error
CE p ( S ) =
for any measurable subset S ⊂ X with P ( S ) > 0 .
In particular, a coverage estimator is the conditional coverage if and only if it is a perfect-calibrated coverage estimator on any measurable subset S of X with P ( S ) > 0 .
Proof.
We ﬁrst show that the conditional coverage is perfect-calibrated.
Taking conditional expectation given { A ( X ) = p, X ∈ S} in the Deﬁnition 2.1.
Then we get
[Equation]
by the tower property.
So A ( x ) is a perfect-calibrated coverage estimator on any measurable subset S with P ( S ) > 0 .
Hence CE p ( S ) = 0 for any measurable subsets S ⊂ X with P ( S ) > 0 .
On the other hand, similarly to Section A.3, we can express
[Equation]
Suppose ˆ P ( x ) is not the conditional coverage, then P [ ˆ P ( X ) (cid:54) =
A ( X )] > 0 .
Without loss of generality, we assume P [ ˆ P ( X ) >
A ( X )] > 0 .
Let S 0 :
= { x ∈ X :
ˆ P ( x ) >
A ( x ) } .
Note that S 0 = ∪ + ∞ n =1 { x ∈ X : ˆ P ( x ) >
A ( x ) + 1 n } .
Since P ( S 0 ) > 0 , there exists a n 0 such that
S := { x ∈ X : ˆ P ( x ) >
A ( x ) + 1 n 0 } and P ( S ) > 0 .
Then for x ∈ S , we have
[Equation]
Then we have
[Equation]
so CE p ( S ) > 0 , which is a contradiction.
Hence ˆ P ( x ) is the conditional coverage.
Next, we describe in detail how to discretize the right-hand-side of equation 2.4 for empirical calculation.
We construct a discrete version of (2.2) and then introduce an empirical counterpart of CE p (2.3), which we refer to as L p empirical calibration-based conditional coverage error ECE p .
The ideas behind these are natural extensions of the classiﬁcation case (Guo et al., 2017; Kull et al., 2019; Kumar et al., 2019; Nixon et al., 2019) into PIs.
We consider the following partition ∆ of [0 , 1] .
Let [0 , 1] be divided into M intervals I m =
( a m − 1 , a m ] ( m = 1 , · · · , M ) where
= a 0 ≤ a 1 ≤ · · · ≤ a M = 1 .
Let B m =
{ i = 1 , · · · , n : ˆ P ( x i ) ∈
I m } , i.e., the set (bin) of indices i of samples whose prediction coverage estimator ˆ P ( x i ) falls into the interval I m .
Note that coverage estimators that are close to each other will fall into the same interval.
The coverage probability (i.e., the proportion of successful coverage) in B m is deﬁned as:
[Equation]
The average estimated coverage in B m is deﬁned as:
[Equation]
where ˆ P ( x i ) is the coverage estimator for sample i .
cp ( B m ) and cove ( B m ) approximate the left and right hand sides of (2.2) respectively in the interval I m .
A perfect-calibrated coverage estimator should satisfy cp ( B m )
= cove ( B m ) for all m ∈ { 1 , · · · , M } .
The diagram of cp ( B m ) versus cove ( B m ) for all m ∈ { 1 , · · · , M } is called the reliability diagram in some literature (Guo et al., 2017).
Based on the partition ∆ , we can introduce an empirical version of CE p (which we refer to as
ECE p ) as:
Deﬁnition A.5.
The L p empirical calibration-based conditional coverage error ( ECE p ) of a coverage estimator ˆ P is deﬁned as
[Equation]
where the l p is the standard p -norm in R n and B m ( i ) is the bin containing sample i .
Equivalently,
[Equation]
Note that ECE p is discontinuous and cannot be used easily for training with gradient-based methods.
Therefore, we use the coverage assessment loss L CA introduced in Section 3 for conditional coverage estimation.
Finally, we give some explanations about why the conditional coverage error (cid:103) CE p cannot be used easily for training.
Unlike L CA which is unbiased and provides guaranteed estimation accuracy for
E [ K 1 ( X )]
(Theorem 4.5), it is in general not easy to establish an empirical calculation for (cid:103) CE p .
Take (cid:103) CE 2 as an instance.
A heuristic argument is to use
[Equation]
to approximate
[Equation]
where k i = 0 or 1 indicates whether each data point has been captured by the PIs; see Section 3.
Unfortunately, ˆ θ is in general not an unbiased estimator of (cid:103) CE 2 2 due to the following observations:
[Equation]
Hence E [ˆ θ ] (cid:54) = (cid:103) CE 2 2 unless A ( X ) = 0 or 1 almost surely.
Note that the gap E [ A ( X ) − A ( X ) 2 ] does not depend on n and thus not vanish as n grows.
To obtain a reasonable estimator for (cid:103) CE p , one need more information about A ( x ) besides the marginal coverage
E
[ A ( X )] .
However, this information is usually hard to obtain locally because of the nature of (possibly high-dimensional) feature space.
In fact, this has led us to the idea of “dimension reduction” of the sample space to the space [0 , 1] and the deﬁnition of CE p .
B M ATHEMATICAL D EVELOPMENTS FOR T HEOREM 4.
This section proves Theorem 4.1: we show that both coverage error and conditional coverage error are tightly bounded above by the expectation of a Kullback–Leibler divergence-type random variable K 1 ( x ) .
This means that minimizing E [ K 1 ( X )] can recover the true conditional coverage and effectively reduce the coverage error.
The proof consists of several inequalities regarding coverage errors and their relations.
We ﬁrst begin with the following connection between CE p and (cid:103) CE p .
Theorem B.1.
For any PI and its associated ˆ P , the L p coverage error is always less than or equal to the L p conditional coverage error, i.e.,
[Equation]
Proof.
We note that the function t (cid:55)→ | t | p is a convex function.
We also note that σ ( ˆ P ( X ))
⊂
σ ( X ) where σ ( Y ) represents the σ -ﬁeld generated by a random variable Y .
[Equation]
by Jensen’s inequality
[Equation]
by the tower property
[Equation]
Therefore we have
[Equation]
Next we have the following bounds on L p conditional coverage error:
Theorem B.2.
The L p conditional coverage error is bounded above by a power function of the Lconditional coverage error.
Formally,
[Equation]
where α p = 1 , ∀ 1 ≤ p ≤ 2 and α p = 2 p , ∀ 2 ≤ p ≤ + ∞ .
Proof.
By H ¨ older’s inequality,
[Equation]
Since
[Equation]
then,
[Equation]
A
[Equation]
and thus
[Equation]
Next, recall that
[Equation]
Theorem B.3.
The L 2 conditional coverage error is bounded above by the expectation of K ( x ) .
Formally,
[Equation]
where α p is deﬁned in Theorem B.
[Equation]
Proof.
For any ﬁxed x , consider two random variables with Bernoulli distributions:
[Equation]
Let P i be the distribution of W i .
It follows from Pinsker’s inequality, e.g., Theorem 2.16 in (Massart, 2007), that
[Equation]
where TV denotes the total variation distance and K denotes the KL divergence.
Since P i is the
Bernoulli distribution, we can express it as
[Equation]
Taking expectation, we obtain
[Equation]
Hence,
[Equation]
Combining Theorem B.1, B.2 and B.3, we immediately conclude that:
Theorem B.4 (Restated Theorem 4.1) .
[Equation]
where α p is deﬁned in Theorem B.2.
Moreover, all inequalities are attainable, e.g., if ˆ P ( x ) is the conditional coverage A ( x ) .
C J USTIFICATION OF A SSUMPTION 4.2 AND M ATHEMATICAL D EVELOPMENTS FOR T HEOREM 4.
In this section, we analyze the rationality of Assumption 4.2 and build essential ingredients for proving Theorem 4.3 and 4.5.
The difﬁculty in analyzing Theorem 4.5 lies in the fact that the hypothesis classes in Assumption 4.2 (which are constructed by the NN) are different from the hypothesis class used in L CA .
To overcome this difﬁculty, we use the theory of VC-subgraph classes to analyze the connection between the VC dimension of the two hypothesis classes.
C. R EVIEW OF THE VC D IMENSION
For self-contained purpose, we ﬁrst review the deﬁnition of the VC-subgraph class and VC dimension.
Deﬁnition C.1.
Consider an arbitrary collection { x 1 , · · · , x n } of points in a set X and a collection
C of subsets of X .
We say that C shatters { x 1 , · · · , x n } if all of 2 n possible subsets of { x 1 , · · · , x n } can be written as A = C ∩ { x 1 , · · · , x n } for some C ∈ C .
The VC dimension V ( C ) of the class
C is the smallest n for which no set of size n { x 1 , · · · , x n } is shattered by C .
If C shatters sets of arbitrarily large size, we set V ( C ) = ∞ .
We say that C is a VC-class if V ( C ) < ∞ .
In some literature, the VC dimension V ( C ) of the class C is alternatively deﬁned as the largest n for which there exists a set of size n { x 1 , · · · , x n } shattered by C , i.e., it is the value in deﬁnition C. minus 1.
We can more formally deﬁne the VC dimension by the growth function as follows:
Deﬁnition C.2.
Deﬁne the n th shatter coefﬁcient (or growth function) of C as
[Equation]
for some
[Equation]
∈
[Equation]
|
Then
[Equation]
Deﬁnition C.3.
For a function f : X → R , the subset of X × R given by { ( x, t ) : t < f ( x ) } is the (open) subgraph of f .
A collection F of measurable real functions on the sample space X is a
VC-subgraph class or VC-class, if the collection of all subgraphs of functions in F forms a VC-class of sets (as sets in X × R ).
Let V ( F ) denote the VC dimension of the set of subgraphs of F .
Lemma C.4.
In Deﬁnition C.3, the open subgraph of f , { ( x, t ) : t < f ( x ) } , can be replaced by the close subgraph { ( x, t ) : t ≤ f ( x ) } , the close supergraph { ( x, t ) : t ≥ f ( x ) } or the open supergraph { ( x, t ) : t > f ( x ) } .
All of them lead to the equivalent deﬁnition of the VC-class and the equal VC dimension.
Proof.
This result follows from Lemma 9.33 and Lemma 9.9(iv) in Kosorok (2007).
For indicator functions of sets, we have the following equivalence.
Lemma C.5.
For any class C of sets in a set X , the class F C of indicator functions of sets in C is a
VC-class
if and only if C is a VC-class.
Moreover, whenever at least one of C or F C is VC-class, the respective VC dimensions are equal.
Proof.
This is Lemma 9.8 in Kosorok (2007).
Note that the sets of C are in X while the subgraphs of functions of F C are in X × R .
C. J USTIFYING A SSUMPTION 4.
We ﬁrst restate the assumption:
Assumption C.6 (Restated Assumption 4.2) .
The four classes of functions ([ L ( x ) , U ( x )] , ˆ P ( x ) , 1 − ˆ P ( x )) output by the neural network (NN) in Figure 1 have ﬁnite VC dimensions, say they are bounded above by V 0 .
In Figure 1, the output four neurons of the NN are denoted as ( L ( x ) , U ( x ) , ˆ P ( x ) , 1 − ˆ P ( x )) .
We further let ( ψ 1 ( x ) , ψ 2 ( x ) , ψ 3 ( x ) , ψ 4 ( x )) denote the pre-activated values of ( L ( x ) , U ( x ) , ˆ P ( x ) , 1 − ˆ P ( x )) .
In other words,
[Equation]
where σ is the sigmoid function.
Let the function classes
H 1 =
{ L ( x ) :
L is output by the the NN } ,
[Equation]
{ U
[Equation]
x )
[Equation]
U is output by the the NN } ,
[Equation]
output by the NN } ,
[Equation]
output by the NN } .
Assumption 4.2 holds for a wide range of NNs, in particular the one we adopt in the experiments (where we use the ReLU-activated NN to construct ψ i , i = 1 , 2 , 3 , 4 ; see Section 5).
Our ﬁrst result is to concretely show that the four NN outputs above, H 1 , H 2 , G and 1 −G , under the ReLU setting, all have ﬁnite VC dimensions and thus satisfy Assumption 4.2.
Theorem C.7 (Restated Theorem 4.3) .
Suppose ψ i , i = 1 , 2 , 3 , 4 are the pre-activated output neurons of the NN in Figure 1 using the ReLU activation function.
Then Assumption 4.2 holds.
Moreover, suppose the NN has W parameters and U computation units (nodes).
Then V
= O ( WU ) .
Proof.
First, we look at L ( x ) and U ( x ) .
Note that the class of ψ i ( i = 1 , 2 ) is constructed by a NN with the ReLU activation function.
Therefore by Theorem 8 in Bartlett et al. (2019) (see also C. below),
[Equation]
By Lemma 9.9 (i) in Kosorok (2007), we have
[Equation]
[Equation]
By Lemma 9.9 (ii) in Kosorok (2007), we have
[Equation]
Next, we look at ˆ P ( x ) and 1 − ˆ P ( x ) .
We add an additional neuron after the layer where ψ 3 , ψstand.
This neuron is deﬁned as ψ 5 = ψ 3 − ψ 4 which is a linear combination of ψ 3 and ψ 4 .
Note that the class of ψ 5 is constructed by a NN with the ReLU activation function and linear activation function (by adding one unit and two parameters in the originial NN).
Therefore by Theorem 8 in Bartlett et al.
[Equation]
By Lemma 9.9 (viii) in Kosorok (2007), we have
[Equation]
since σ is a monotone function.
Again, by Lemma 9.9 (viii) in Kosorok (2007), we have
[Equation]
since t (cid:55)→ 1 − t is a monotone function.
We also list some results for other activations here.
From these results, and using the same argument as above, we see that Assumption 4.2 holds similarly for all these activations.
Lemma C.8.
Suppose the class of functions is constructed by a NN with W parameters and U units with activation functions that are piecewise polynomials with at most p pieces and of degree at most d .
Then it has VC dimension O ( WU log(( d + 1) p )) .
Proof.
This is Theorem 8 in Bartlett et al.
Note that the activation functions in Lemma C.8 include in particular the ReLU activation and linear activation.
Lemma C.9.
Suppose the class of functions is constructed by a NN with W parameters with binary as well as linear activation function.
Then it has VC dimension
O
( W 2 ) .
Proof.
This is Theorem 5 in Sontag (1998).
Lemma C.10.
Suppose the class of functions is constructed by a NN with W parameters and U units with activation function that is the standard sigmoid function (except that the output unit being a linear threshold unit).
Then it has VC dimension
O
( W 2 U 2 ) .
Proof.
This is Theorem 8.13 in Anthony & Bartlett (2009).
C. C ONNECTIONS AMONG D IFFERENT H YPOTHESIS C LASSES
To prove Theorem 4.5, we need to study several building blocks on the relations between different hypothesis classes.
Our ﬁrst observation is:
Theorem C.11.
Suppose V ( G ) <
+ ∞ .
Then all of the following classes have VC dimension ≤ V ( G ) : 1 − G :=
{ 1 − ˆ P ( x ) : ˆ P is output by the NN } .
G (cid:48) :
= log( G ) :
= { log( ˆ P ( x )) :
ˆ P is output by the NN } .
log(1 − G ) :
= { log(1 − ˆ P ( x )) : ˆ P is output by the NN } .
Proof.
The result follows from Lemma 9.9 (viii) in Kosorok (2007) since all of the transformations are monotone functions.
Our second observation is about
[Equation]
output by the
[Equation]
Note that the domain of functions in F is different from the domain of functions in H
i ( i = 1 , 2 ) as it includes the outcome space.
Below we derive a result that connects the VC dimension of H i with that of F .
Theorem C.12.
Suppose V ( H i ) ≤ V
( i = 1 , 2 ).
We have that
[Equation]
where
[Equation]
Proof.
The ﬁrst inequality follows from Lemma 9.9 (viii) in Kosorok (2007).
We consider the following two classes:
[Equation]
Since the functions in F 1 are all indicator functions, by Lemma C.5,
[Equation]
Note that the latter is the VC dimension of the close supergraphs of all functions in H 1 .
Then by
Deﬁnition C.3 and Lemma C.4, we have
[Equation]
Therefore we have
[Equation]
Similarly,
[Equation]
Note that we can write
[Equation]
By the deﬁnition of growth functions,
[Equation]
for all m ≥ V 0 where the last inequality is due to the Sauer–Shelah lemma.
Taking m = 10( V 0 − 1) , we obtain
[Equation]
Combining the above inequality, we have Π F ( m ) < 2 m .
This shows that V ( F ) ≤ m =
10( V 0 − 1) .
C. P ROOF OF T HEOREM 4.
This subsection proves Theorem 4.5.
Recall that
[Equation]
by the NN }
[Equation]
Let N ( (cid:15), F , L 2 ( Q )) denote the covering number, i.e., the minimal number of balls { g : (cid:107) g − h (cid:107) L 2 ( Q ) < (cid:15) } of radius (cid:15) needed to cover the set F .
We need the following bounds:
Lemma C.13.
Suppose F is a class of functions f : X × Y → [0 , 1] with a ﬁnite VC dimension
V ( F ) .
For every 0 < (cid:15) < 1 ,
[Equation]
where the constant K 2 depends on V ( F ) only.
Proof.
It follows from Theorem 2.6.7 in Van der Vaart & Wellner (1996) that there exists a universal constant K such that
[Equation]
for any 0 < (cid:15) < 1 .
Since (cid:18)
[Equation]
max
[Equation]
<
[Equation]
< 1 , we have
[Equation]
where K 3 := log( KV ( F )(16 e ) V ( F ) ) and K 2 = K 3 + V ( F ) − 1 only depending on V ( F ) .
We remark that a similar result can also be obtained for the class 1 − F by Theorem C.12.
Lemma C.14.
Suppose G is a class of functions ˆ P : X → [0 , 1] with a ﬁnite VC dimension V ( G ) and | log( ˆ P ( x )) | ≤ M .
Let G (cid:48) :
= { log( ˆ P ) : ˆ P ∈ G} .
Then, for every 0 < (cid:15) < 1 ,
[Equation]
where the constant K 2 depends on V ( G ) only.
Proof.
First note that φ ( t ) :
=
log( t ) is a monotone function.
Hence G (cid:48) :
= { log( ˆ P ) : ˆ P ∈ G} is a VC-class with VC dimension ≤ V ( G ) by Lemma 9.9 (viii) in Kosorok (2007).
The rest of the proof is similar to C.13.
We remark that a similar result can also be obtained for the class log(1 − G ) by Theorem C.11.
Next we restate Assumption 4.4:
Assumption C.15 (Restated Assumption 4.4) .
| log( ˆ P ( x )) | ≤ M , | log(1 − ˆ P ( x )) | ≤ M for all x and ˆ P .
As discussed in Section 4, this is a natural assumption in practice because log( ˆ P ( x )) and log(1 − ˆ P ( x )) are replaced by log( ˆ P ( x ) + (cid:15) ) and log(1 − ˆ P ( x ) + (cid:15) ) respectively to avoid explosion when implementing the algorithm.
In particular, in our experiments in Section 5, (cid:15) = 0 .
1 6 and thus
M = 14 .
We are now ready to prove Theorem 4.5:
Theorem C.16 (Restated Theorem 4.5) .
Suppose Assumptions 4.2 and 4.4 hold.
The training data
D = { ( x i , y i ) , i = 1 , 2 , · · · , n } where ( x i , y i ) are i.i.d.
samples ∼ π .
Recall that the (hard) coverage estimator assessment loss is
[Equation]
Then for any t > 0 , we have
[Equation]
where C ∗ only depends on V 0 in Assumption 4.2.
Proof.
Note that E [ f ( x i , y i ) | x i ]
=
A ( x i ) for any ﬁxed L and U .
Taking expectation on L CA , we have
[Equation]
We consider the ﬁrst part A ( x ) log( ˆ P ( x )) .
The second part can be done using the same argument.
Note that by Theorem 9.15 in Kosorok (2007), we have
[Equation]
L
[Equation]
Q
[Equation]
Consider the class
+ 12 M F · G (cid:48) :
= { 12 + 12 M ( f ( x, y ) log( ˆ P ( x ))) : f ∈ F , log( ˆ P ) ∈ G (cid:48) } which consists of functions taking values in [0 , 1] .
We have
[Equation]
where the last inequality follows from Lemma C.13 and Lemma C.14, and K 2 only depends on
V ( F ) and V ( G ) .
(Recall that we have shown V ( G (cid:48) )
≤ V ( G ) in Lemma C.14.)
Moreover, by
Theorems C.11 and C.12, we can claim that K 2 only depends on V 0 .
This inequality shows that 12 + 12 M F · G (cid:48) satisﬁes the conditions in Theorem 2.14.10 in Van der Vaart & Wellner (1996) and thus for every δ > 0 and t > 0 , [Equation] nt
[Equation]
where U = 1 e (6 − 1 e ) 2+ 1 e < 1 and the constants C and D depend on K 2 and δ only.
Let δ = 1 − U .
Note that
[Equation]
Hence we have
[Equation]
where C ∗ only depends on K 2 , or, only depends on V 0 .
This shows that
[Equation]
A similar result can be established for the second part since the hypothesis classes there have been studied in Theorem C.11 and C.12:
[Equation]
Combining the two parts and noting the following fact:
[Equation]
we conclude that
[Equation]
where C ∗ only depends on V 0 .
Lastly, the following corollary explicitly connects our theoretical developments to the experimental setup:
Corollary C.17.
Suppose the NN is designed as the one speciﬁed in the experiments (Section 5).
The training data D = { ( x i , y i ) , i = 1 , 2 , · · · , n } where ( x i , y i ) are i.i.d.
samples ∼ π .
Then for any t > 0 , we have
[Equation]
where C ∗ only depends on V 0 in Assumption 4.2.
Proof.
We note that Assumptions 4.2 and 4.4 hold in this case by Theorem 4.3 and the observation after Assumption 4.4.
So Theorem 4.5 implies Corollary C.17.
D A LGORITHM D ETAILS
We provide additional algorithm details for our framework in Section 3.
Algorithm 1 is the description of our tuning procedure for hyper-parameters λ 1 , λ 2 , λ 3 .
Let the marginal coverage probability CP D (cid:48) and the average coverage estimation AC D (cid:48) on the validation set D (cid:48) be deﬁned as
[Equation]
where ([ L ( x ) , U ( x )] , ˆ P ( x )) are prediction results from the deep ensemble.
Then, λ i , i = 1 , 2 , 3 are adjusted to ensure that CP D (cid:48) coincides roughly with AC D (cid:48) , and CP D (cid:48) attains the target prediction level.
E E XPERIMENTAL D ETAILS AND M ORE R ESULTS
This section illustrates experimental details and more experimental results from our proposed model.
E. E XPERIMENTAL D ETAILS
Table 3 gives a detailed description about the datasets we use.
These open-access real-world benchmark regression datasets are widely used for the evaluation of methods in regression tasks (Hern´andez-Lobato & Adams, 2015; Gal & Ghahramani, 2016; Lakshminarayanan et al., 2017;
Rosenfeld et al., 2018; Pearce et al., 2018; Zhu et al., 2019).
For synthetic datasets, 2000 i.i.d data are generated for each synthetic setting and randomly split into 1000 training data and 1000 testing data.
For benchmark datasets, we ﬁrst do the data normalization and then randomly split 80% data for training and 20% for testing.
The choice of 80%/20% split, compared to the 90%/10% split in Pearce et al.
(2018), is motivated from the need to increase the test size in order to get a meaningful ECE evaluation. The latter is due to that evaluating ECE requires binning, where using a larger number of bins approximates more closely CE , but also requires a larger test size to sustain enough statistical quality for the resulting ECE estimate. This delicate tradeoff motivates us to increase the share of the test set in our split. Following Pearce et al.
, our hyper-parameters are selected using the validation set from a random split. Then, they are ﬁxed during the evaluation on other random splits. As speciﬁed in Section 2 (Equation (2.4)) and Appendix A.4 (Equation (A.6))
, ECE is evaluated based on dividing [0,1] into M sub-intervals.
The larger M is, the more precise is in using ECE to approximate CE, the latter being the ideal conditional coverage error estimator.
On the other hand, a larger test set size is needed to support the use of a larger M without deteriorating the statistical quality of the ECE.
This delicate tradeoff motivates us to increase the share of the test set in our split.
For synthetic datasets, the hyperparameters and corresponding results in Figure 2 are:
(a) λ 1 = 1 . 7 , λ 2 = 10 − 5 , λ 3 = 1500 , CP = 0 .
95 , IW = 0 .
40 , ECE 1 = 0 .
(b) λ 1 = 1 .
9 , λ 2 = 10 − 5 , λ 3 = 1000 , CP = 0 .
96 , IW = 0 .
40 , ECE 1 = 0 .
(c) λ 1 = 3 . 4 , λ 2 = 10 − 5 , λ 3 = 1000 , CP = 0 .
95 , IW = 0 .
50 , ECE 1 = 0 .
For benchmark datasets, the implementation details for baseline algorithms in Table 1 are:
(1) Nearest- neighbors kernel conditional density estimation ( NNKCDE).
The algorithm is based on Section 2.1 in Dalmasso et al.
We use the same Python code provided by Dalmasso et al.
(2020) with the default Gaussian kernel.
Two tuning parameters, i.e., the number of nearest neighbors k and the bandwidth h of the smoothing kernel, are chosen in a principled way by minimizing the CDE loss on validation data, the same way as in Dalmasso et al.
(2) Quantile regression forest (QRF).
The algorithm is based on Meinshausen (2006).
We use the RandomForestQuantileRegressor from the package scikit-garden in Python.
(3) Split conformal learning (SCL).
The algorithm based on Algorithm 2 in Lei et al.
The regression algorithm inside SCL that we use is a neural network with mean square loss.
The neural network has the same structure of hidden layers as in Section 5.
E. A DDITIONAL E XPERIMENTAL R ESULTS
Table 4 gives additional experimental results. [Table]
[Table]
E. C OMPARISONS WITH A TWO - STAGE APPROACH
We compare the performance of CaNet with a two-stage approach to further demonstrate the effectiveness of our Ca-Module.
The two-stage approach is implemented with two separate steps: (1) given a regression dataset, we train a neural network to generate the prediction interval, (2) after getting the predictor, we train another network to estimate the conditional coverage of the PI from the previous stage using L CA loss.
Figure 3 compares the reliability diagrams (introduced in A.4) and the coverage histograms of CaNet and the two-stage approach on the dataset “Protein”.
The coverage histograms demonstrate the percentage of samples in each bin B m (equation A.5) for m ∈ { 1 , · · · , M } .
The average estimated coverage of our model closely matches its coverage probability, while the average estimated coverage of the two-stage algorithm is substantially lower than its coverage probability.
In addition, the
ECE 1 from CaNet ( 0 .
77% ) is much lower than the ECE 1 from the two-stage approach ( 3 . 9% ).