PAC
C ONFIDENCE
S
ETS FOR
D
EEP
N EURAL
N ETWORKS VIA C ALIBRATED P REDICTION
Anonymous authors
Paper under double-blind review
A BSTRACT
We propose an algorithm combining calibrated prediction and generalization bounds from learning theory to construct conﬁdence sets for deep neural networks with PAC guarantees—i e., the conﬁdence set for a given input contains the true label with high probability.
We demonstrate how our approach can be used to construct PAC conﬁdence sets on ResNet for ImageNet, and on a dynamics model the half-cheetah reinforcement learning problem.
I NTRODUCTION
A key challenge facing deep neural networks is that they do not produce reliable conﬁdence estimates, which are important for applications such as safe reinforcement learning (Berkenkamp et al., 2017), guided exploration (Malik et al., 2019), and active learning (Gal et al., 2017).
We consider the setting where the test data follows the same distribution as the training data (i.e., we do not consider adversarial examples designed to fool the network (Szegedy et al., 2014)); even in this setting, conﬁdence estimates produced by deep neural networks are notoriously unreliable (Guo et al., 2017).
One intuition for this shortcoming is that unlike traditional supervised learning algorithms, deep learning models typically overﬁt the training data (Zhang et al., 2017).
As a consequence, the conﬁdence estimates of deep neural networks are ﬂawed even for test data from the training distribution since, by construction, they overestimate the likelihood of the training data.
A promising approach to addressing this challenge is temperature scaling (Platt, 1999).
This approach takes as input a trained neural network f
ˆ φ ( y | x ) —i.e., whose parameters ˆ φ have already been ﬁt to a training dataset Z train —which produces unreliable probabilities f
ˆ φ ( y | x ) .
Then, this approach rescales these conﬁdence estimates based on a validation dataset to improve their “calibration”.
More precisely, this approach ﬁts conﬁdence estimates of the form
[Equation]
where τ ∈ R > 0 is a temperature scaling parameter that is ﬁt based on the validation dataset.
The goal is to choose τ to minimize calibration error , which roughly speaking measures the degree to which the reported error rate differs from the actual error rate.
The key insight is that in the temperature scaling approach, only a single parameter T is ﬁt to the validation data—thus, unlike ﬁtting the original neural network, the temperature scaling algorithm comes with generalization guarantees based on traditional statistical learning theory.
Despite the improved generalization guarantees, these conﬁdence estimates still do not come with theoretical guarantees.
We are interested in producing conﬁdence sets that satisfy statistical guarantees while being as small as possible.
Given a test input x ∈ X , a conﬁdence set C T ( x ) ⊆
Y (parameterized by T ∈ R > 0 ) should contain the true label y for at least a 1 − (cid:15) fraction of cases:
[Equation]
Since we are ﬁtting a parameter T to based on Z val , we additionally incur a probability of failure due to the randomness in Z val .
In other words, given (cid:15), δ ∈ R > 0 , we aim to obtain probably approximately correct (PAC) conﬁdence sets C T ( x ) ⊆
Y satisfying the guarantee
[Equation]
| C ( x ) | =5 ≤ | C
( x ) | ≤50 ≤ | C ( x ) | ≤| C ( x )
| ≥
[Figure]
Indeed, techniques from statistical learning theory (Vapnik, 1999) can be used to do so (Vovk, 2013).
There are a number of reasons why conﬁdence sets can be useful.
First, they can be used to inform safety critical decision making.
For example, consider a doctor who uses prediction tools to help perform diagnosis.
Having a conﬁdence set would both help the doctor estimate the conﬁdence of the prediction (i.e., smaller conﬁdence sets imply higher conﬁdence), but also give a sense of the set of possible diagnoses.
Second, having a conﬁdence set can be useful for reasoning about safety since they contain the true outcome with high probability.
For instance, robots may use a conﬁdence set over predicted trajectories to determine whether it is safe to act with high probability.
As a concrete example, consider a self-driving car that uses a deep neural network to predict the path that a pedestrian might take.
We require that the self-driving car avoid the pedestrian with high probability, which it can do by avoiding all possible paths in the predicted conﬁdence set.
Contributions.
We propose an algorithm combining calibrated prediction and statistical learning theory to construct PAC conﬁdence sets for deep neural networks (Section 3).
We propose instantiations of this framework in the settings of classiﬁcation, regression, and learning models for reinforcement learning (Section 3.6).
Finally, we evaluate our approach on two benchmarks: ResNet (He et al., 2016) for ImageNet (Russakovsky et al., 2015), and on a probabilistic dynamics model (Chua et al., 2018) learned for the half-cheetah environment (Brockman et al., 2016)
(Section 4).
Examples of ImageNet images with different sized ResNet conﬁdence sets are shown in Figure 1.
As can be seen, our conﬁdence sets become larger and the images become more challenging to classify.
Related work.
There has been work on constructing conﬁdence sets with theoretical guarantees.
Oftentimes, these guarantees are asymptotic rather than ﬁnite sample (Steinberger & Leeb, 2016; 2018).
Alternatively, there has been work focused on predicting conﬁdence sets with a given expected size (Denis & Hebiri, 2017).
More relatedly, there has been recent work on obtaining PAC guarantees.
For example, there has been some work speciﬁc prediction tasks such as binary classiﬁcation (Lei, 2014; Wang & Qiao, 2018).
There has also been work in the setting of regression (Lei et al., 2018; Barber et al., 2019).
However, in this case, the conﬁdence sets are ﬁxed in size—i.e., they do not depend on the input x
(Barber et al., 2019).
Furthermore, they make stability assumptions about the learning algorithm (though they achieved improved rates by doing so)
(Lei et al., 2018; Barber et al., 2019).
The most closely related work is on conformal prediction (Papadopoulos, 2008; Vovk, 2013).
Like our approach, this line of work provides a way to construct conﬁdence sets from a given conﬁdence predictor, and provided PAC guarantees for the validity of these conﬁdence sets.
Indeed, with some work, our generalization bound Theorem 1 can be shown to be equivalent to Theorem 1 in Vovk (2013).
In contrast to their approach, we proposed to use calibrated prediction to construct conﬁdence predictors that can suitably be used with deep neural networks.
Furthermore, our approach makes explicit the connections to temperature scaling and as well as to generalization bounds from statistical learning theory (Vapnik, 1999).
In addition, unlike our paper, they do not explicitly provide an efﬁcient algorithm for constructing conﬁdence sets.
Finally, we also propose an extension to the case of learning models for reinforcement learning.
Finally, we build on a long line of work on calibrated prediction , which aims to construct “calibrated” probabilities (Murphy, 1972; DeGroot & Fienberg, 1983; Platt, 1999; Zadrozny & Elkan, 2001; 2002; Naeini et al., 2015; Kuleshov & Liang, 2015).
Roughly speaking, probabilities are calibrated if events happen at rates equal to the predicted probabilities.
This work has recently been applied to obtaining conﬁdence estimates for deep neural networks (Guo et al., 2017; Kuleshov et al., 2018; Pearce et al., 2018), including for learned models for reinforcement learning (Malik et al., 2019).
However, these approaches do not come with PAC guarantees.
PAC C ONFIDENCE S ETS
Our goal is to estimate conﬁdence sets that are as small as possible, while simultaneously ensuring that they are probably approximately correct (PAC)
(Valiant, 1984).
Essentially, a conﬁdence set is correct if it contains the true label.
More precisely, let X be the inputs and Y be the labels, and let
D be a distribution over Z = X
× Y .
A conﬁdence set predictor is a function C : X → 2 Y such that C ( x ) ⊆
Y is a set of labels; we denote the set of all conﬁdence set predictors by C .
For a given example ( x, y ) ∼ D , we say C is correct if y ∈ C ( x ) .
Then, the error of C is
[Equation] [Equation]
Finally, consider an algorithm A that takes as input a validation set Z val ⊆
Z consisting of n i.i.d.
samples ( x, y ) ∼ D , and outputs a conﬁdence set predictor ˆ C .
Given (cid:15), δ ∈ R > 0 , we say that A is probably approximately correct (PAC) if
[Equation]
Our goal is to design an algorithm A that satisﬁes (2) while constructing conﬁdence sets C ( x ) that are as “small in size” as possible on average.
The size of C ( x ) depends on the domain.
For classiﬁcation, we consider conﬁdence sets that are arbitrary subsets of
labels C
( x ) ⊆
Y = { 1 , ..., Y } , and we measure the size by | C ( x ) | ∈ N —i.e., the number of labels in C ( x ) .
For regression, we consider conﬁdence sets that are intervals C ( x ) =
[ a, b ] ⊆ Y = R , and we measure size by b − a —i.e., the length of the predicted interval.
Note that there is an intrinsic tradeoff between satisfying (2) and average size of C ( x ) —larger conﬁdence sets are more likely to satisfy (2).
PAC A LGORITHM FOR C ONFIDENCE S ET C ONSTRUCTION
Our algorithm is formulated in the empirical risk framework .
Typically, this framework refers to empirical risk minimization .
In our setting, such an algorithm would take as input (i) a parametric family of conﬁdence set predictors C
= { C θ | θ ∈ Θ } , where Θ is the parameter space, and (ii) a training set Z val ⊆
Z of n i.i.d.
samples ( x, y ) ∼ D , and output the conﬁdence set predictor C ˆ θ , where ˆ θ minimizes the empirical risk:
[Equation]
where
[Equation]
Here, I [ φ ] ∈ { 0 , 1 } is the indicator function, and the empirical risk ˆ L in an estimate of the conﬁdence set error (1) based on the validation set Z val .
However, our algorithm does not minimize the empirical risk.
Rather, recall that our goal is to minimize the size of the predicted conﬁdence sets given a PAC constraint on the true risk L ( ˆ θ ) based on the given PAC parameters (cid:15), δ ∈ R > 0 and the number of available validation samples n = | Z val | .
Thus, the risk shows up as a constraint in the optimization problem, and the objective is instead to minimize the size of the predicted conﬁdence sets:
[Equation]
At a high level, the value α = α ( n, (cid:15), δ ) ∈ R ≥ 0 is chosen to enforce the PAC constraint, and is based on generalization bounds from statistical learning theory (Valiant, 1984).
Furthermore, following the temperature scaling approach (Platt, 1999), the parameter space Θ is chosen to be as small as possible (in particular, one dimensional) to enable good generalization.
Finally, our choice of size metric S follows straightforwardly based on our choice of parameter space.
In the remainder of this section, we describe the choices of (i) parameter space Θ , (ii) size metric S ( θ ) , and (iii) conﬁdence level α ( n, (cid:15), δ ) in more detail, as well as how to solve (3) given these choices.
3. C HOICE OF P ARAMETER S PACE Θ
Probability forecasters.
Our construction of the parameteric family of conﬁdence set predictors C θ assumes given a probability forecaster f : X → P Y , where P Y is a space of probability distributions over Y .
Given such an f , we use f ( y | x ) to denote the probability of label y under distribution f ( x ) .
Intuitively, f ( y | x ) should be the probability (or probability density) that y is the true label for a given input x —i.e., f ( y | x ) ≈ P ( X,Y ) ∼ D
[ Y = y
| X = x ] .
For example, in classiﬁcation, we can choose P Y to be the space of categorical distributions over Y , and f may be a neural network whose last layer is a softmax layer with |Y| outputs.
Then, f ( y | x ) =
f
( x ) y .
Alternatively, in regression, we can choose P Y to be the space of Gaussian distributions, and f may be a neural network whose last layer outputs the values ( µ, σ ) ∈ R × R > 0 of a Gaussian distribution.
Then, f ( y | x ) =
N ( x ; µ ( x ) , σ ( x ) 2 ) , where ( µ ( x ) , σ ( x ))
=
f
( x ) , and N ( · ; µ, σ 2 ) is the Gaussian density function with mean µ and variance σ 2 .
Training a probability forecaster.
To train a probability forecaster, we use a standard approach to calibrated prediction that combines maximum likelihood estimation with temperature scaling .
First, we consider a parametric model family F = { f φ | φ ∈ Φ } , where Φ is the parameter space.
Note that Φ can be high-dimensional—e.g., the weights of a neural network model.
Given a training set Z train ⊆
Z of m i.i.d.
samples ( x, y ) ∼ D , the maximum likelihood estimate (MLE) of φ is
[Equation]
where
[Equation]
We could now use f ˆ φ as the probability forecaster.
However, the problem with directly using ˆ φ is that because ˆ φ may be high-dimensional, it often overﬁts the training data Z train .
Thus, the probabilities are typically overconﬁdent compared to what they should be.
To reduce their conﬁdence, we use the temperature scaling approach to calibrate the predicted probabilities (Platt, 1999; Guo et al., 2017).
Intuitively, this approach is to train an MLE estimate using exactly the same approach used to train ˆ φ , but using a single new parameter τ ∈ R > 0 .
The key idea is that this time, the model family is based on the parameters ˆ φ from (4).
In other words, the “shape” of the probabilities forecast by f ˆ φ are preserved, but their exact values are shifted.
More precisely, consider the model family F (cid:48)
=
{ f ˆ φ,τ | τ ∈ R > 0 } , where
[Equation]
Then, we have the following MLE for τ :
[Equation]
where
[Equation]
Note that ˆ τ is estimated based on a second training set Z (cid:48) train .
Because we are only ﬁtting a single parameter, this training set can be much smaller than the training set Z train used to ﬁt ˆ φ .
Parametric family of conﬁdence set predictors.
Finally, given a probability forecaster f , we consider one dimensional parameter space Θ
= R > 0 ; in an analogy to the temperature scaling technique for calibrated prediction, we denote this parameter by T ∈ Θ .
In particular, we assume a conﬁdence probability predictor f is given, and consider
[Equation]
In other words, C T ( x ) is the set of y with high probability given x according to f .
Considering this scalar parameter space, we denote the minimum of (3) by ˆ T .
3. C HOICE OF S IZE M ETRIC S ( T )
To choose the size metric S ( T ) , we note that for our chosen parametric family of conﬁdence set predictors, smaller values correspond to uniformly smaller conﬁdence sets—i.e.,
[Equation]
Thus, we can simply choose the size metric to be
[Equation] [Equation]
This choice minimizes the size of the conﬁdence sets produced by our algorithm.
3. C HOICE OF C ONFIDENCE L EVEL α ( n, (cid:15), δ )
Naive approach based on VC generalization bound.
A naive approach to choosing α ( n, (cid:15), δ ) is to do so based on the VC dimension generalization bound (Vapnik, 1999).
It is not hard to show that the problem of estimating ˆ T is equivalent to a binary classiﬁcation problem, and that the VC dimension of Θ for this problem is 1 .
Thus, the VC dimension bound implies that for all T ∈ Θ ,
P Z val ∼ D n
[Equation]
n
[Equation] [Equation]
The details of this equivalence are given in Appendix B.2.
Then, suppose we choose
[Equation]
With this choice, for the solution ˆ T of (3) with α = α ( n, (cid:15), δ ) , the constraint in (3) ensures that ˆ L ( C ˆ T ; Z val ) ≤ α ( n, (cid:15), δ ) .
Together with the VC generalization bound (7), we have
[Equation]
which is exactly desired the PAC constraint on our predicted conﬁdence sets.
Direct generalization bound.
In fact, we can get better choices of α by directly bounding generalization error.
For instance, in the realizable setting (i.e., we always have ˆ L ( C ˆ T ; Z val ) = 0 ), we can get rates of n = ˜ O
(1 /(cid:15) ) instead of n = ˜ O (1 /(cid:15) 2 ) (Kearns & Vazirani, 1994); see Appendix A.2 for details.
We can achieve these rates by choosing α = 0 , but then, the PAC guarantees we obtain may actually be stronger than desired (i.e., for (cid:15) (cid:48) < (cid:15) ).
Intuitively, we can directly prove a bound that interpolates between the realizable setting and the VC generalization bound—in particular:
Theorem 1.
For any (cid:15) ∈ [0 , 1] , n ∈ N > 0 , and k ∈ { 0 , 1 , ..., n } , we have
[Equation]
where ˆ T is the solution to (3) with α = k/n .
We give a proof in Appendix B.2.
Based on Theorem 1, we can choose α
[Equation]
n,
[Equation] [Equation]
∈ N
[Equation]
/n subj.
to
[Equation]
3. T HEORETICAL G UARANTEES
We have the following guarantee, which follows straightforwardly from Theorem 1:
Corollary 1.
Let ˆ T be the solution to (3) for α
= α ( n, (cid:15), δ ) chosen according to (8).
Then, we have
[Equation]
In other words, our algorithm is probably approximately correct.
Algorithm 1 Algorithm for solving (3).
procedure E STIMATE C ONFIDENCE S ET P REDICTOR ( Z train , Z (cid:48) train , Z val )
Estimate ˆ φ, ˆ τ using (4) and (5), respectively
Compute α ( n, (cid:15), δ ) according to (8) by enumerating k ∈ { 0 , 1 , ..., n }
Let k ∗ =
n · α ( n, (cid:15), δ ) (note that k ∈ { 0 , 1 , ..., n } )
Sort ( x, y ) ∈
Z val in ascending order of f ˆ φ, ˆ τ ( y | x )
Let ( x k ∗ +1 , y k ∗ +1 )
be the ( k ∗ + 1) st element in the sorted Z val
Solve (3) by choosing ˆ T = − log f
ˆ φ, ˆ τ ( y k ∗ +
| x k ∗ +1 )
Return C
ˆ T : x (cid:55)→ { y ∈ Y
| f ˆ φ, ˆ τ ( y | x ) ≥ e − ˆ T } end procedure
3. P RACTICAL I MPLEMENTATION
Our algorithm for estimating a conﬁdence set predictor C ˆ T is summarized in Algorithm 1.
The algorithm solves the optimization problem (3) using the choices of Θ , S ( T ) , and α ( n, (cid:15), δ ) described in the preceding sections.
There are two key implementation details that we describe here.
Computing α ( n, (cid:15), δ ) .
To compute α ( n, (cid:15), δ ) , we need to solve (8).
A straightforward approach is to enumerate all possible choices of k ∈ { 0 , 1 , ..., n } .
There are two optimizations.
First, the objective is monotone increasing in k , so we can enumerate k in ascending order until the constraint no longer holds.
Second, rather than re-compute the left-hand side of the constraint (cid:80) ki
=0 (cid:0) ni (cid:1) (cid:15) i (1 − (cid:15) )
n − i , we can accumulate the sum as we iterate over k .
We can also incrementally compute (cid:0) ni (cid:1) , (cid:15) i , and (1 − (cid:15) )
n − i .
For numerical stability, we perform these computations in log space.
Solving (3).
To solve (3), note that the constraint in (3) is equivalent to (cid:88)
[Equation]
where E
[Equation]
Also, note that k ∗ = n · α ( n, (cid:15), δ ) is an integer due to the deﬁnition of α ( n, (cid:15), δ ) in (8).
Thus, we can interpret (9) as saying that E ( x, y ; T ) = 1 for at most k ∗ of the points ( x
, y ) ∈
Z val .
In addition, note that E ( x, y ; T ) decreases monotonically as f ˆ φ, ˆ τ ( y | x ) becomes larger.
Thus, we can sort the points ( x
, y ) ∈
Z val in ascending order of f ˆ φ, ˆ τ ( y | x ) , and require that only the ﬁrst k ∗ points ( x, y ) in this list satisfy E ( x, y ; T )
= 1 .
In particular, letting ( x k ∗ +1 , y k ∗ +1 ) be the ( k ∗ +1)
st point, (9) is equivalent to
[Equation]
In other words, this constraint says that T must satisfy y k ∗ +1 ∈ C T ( x k ∗ +1 ) .
Finally, the solution ˆ T to (3) is the smallest T that satisﬁes (10), which is the T that makes (10) hold with equality—i.e.,
[Equation]
We have assumed f ˆ φ, ˆ τ ( y k ∗ +1 | x k ∗ +1 ) >
f ˆ φ, ˆ τ ( y k ∗ | x k ∗ ) ; if not, we increment k ∗ until this holds.
3. P ROBABILITY F ORECASTERS FOR S PECIFIC T ASKS
We brieﬂy discuss the architectures we use for probability forecasters for various tasks.
We give details, including how we measure the sizes of predicted conﬁdence sets C T ( x ) , in Appendix C.
We consider three tasks: classiﬁcation, regression, and model-based reinforcement learning.
For classiﬁcation, we use the standard approach of using a soft-max layer to predict label probabilities f ( y | x ) .
For regression, we also use a standard approach where the neural network predicts both the mean µ ( x ) and covariance Σ( x ) of a Gaussian distribution N ( µ ( x ) , Σ( x ))
; then, f ( y | x ) =
N ( y ; µ ( x ) , Σ( x )) is the probability density of y according to this Gaussian distribution.
Finally, for model-based reinforcement learning, our goal is to construct conﬁdence sets over trajectories predicted using a learned model of the dynamics.
We consider unknown dynamics
[Figure]
g ∗ ( x (cid:48) | x, u ) mapping a state-action pair ( x, u ) to a distribution over states x (cid:48) , and consider a known (and ﬁxed) policy π ( u | x ) mapping a given state x to a distribution over actions u ∈ U ⊆ R d U .
Then, we let f ∗ ( x (cid:48) | x ) =
E π ( u | x ) [ g ∗ ( x (cid:48)
| u )] denote the (unknown) closed-loop dynamics.
Next, we consider a forecaster f ( x (cid:48) | x ) ≈ f ∗ ( x (cid:48) | x ) of the form f ( x (cid:48) | x ) =
N ( x (cid:48) ; µ ( x ) , Σ( x )) , and our goal is to construct conﬁdence sets for the predictions of f .
However, we want to do so for not just for one-step predictions, but for predictions over a time horizon H ∈ N .
In particular, given initial state x 0 ∈ X , we can sample x ∗ 1: H = ( x 1 , ..., x H ) ∼ f ∗ by letting x ∗
= x 0 and sequentially sampling x ∗ t +1 ∼ f ( · | x ∗ t ) for each t ∈ { 0 , 1 , ..., H − 1 } .
Then, our goal is to construct a conﬁdence set that contains x ∗ 1: H ∈ X H with high probability (over both the randomness in an initial state distribution x 0 ∼ d 0 and the randomness in f ∗ ).
To do so, we construct and use a forecaster ˜ f
( x 1: H | x 0 ) based on f .
In principle, this task is a special case of multivariate regression, where the inputs are X (i.e., the initial state x 0 ) and the outputs are Y =
X H (i.e., a predicted trajectory x 1: H ).
However, the variance Σ( x ) predicted by our probability forecaster is only for a single step, and does not take into account the fact that x is itself uncertain.
Thus, we use a simple heuristic where we accumulate variances over time.
More precisely, we construct (i) the predicted mean ¯ x 1: H =
(¯ x 1 , ..., ¯ x H ) by ¯ x 0 = x 0 and
¯ x t +1 = µ (¯ x t ) for t ∈ { 0 , 1 , ..., H − 1 } , and (ii) the predicted variances ˜Σ 1: H = (˜Σ 1 , ..., ˜Σ H ) by ˜Σ t = Σ(¯ x 0 )
+ Σ(¯ x 1 ) + ...
+ Σ(¯ x t − 1 ) .
Then, we use a probability forecaster ˜ f
( x 1: H | x 0 )
=
N (¯ x 1: H , ˜Σ 1: H ) to construct conﬁdence sets.
E XPERIMENTS
We describe our experiments on ImageNet (a classiﬁcation task) and the half-cheetah environment (a model-based reinforcement learning task).
We give additional results in Appendix D.
ResNet for ImageNet.
We use our algorithm to compute conﬁdence sets for ResNet (He et al., 2016) on ImageNet (Russakovsky et al., 2015), for (cid:15)
= 0 .
01 , δ = 10 − 5 , and n = 20000 validation
[Figure]
We show the results in Figure 1.
In (a), we compare to two ablations.
In particular, C refers to performing an initial temperature scaling step to calibrate the neural network predictor (i.e., using f ˆ φ instead of f ˆ φ, ˆ τ , and (ii) D refers to using Theorem 1 instead of the VC generalization bound.
Thus, C + D refers to our approach.
As can be seen, using Theorem 1 is performs substantially better than using the VC generalization bound.
Using calibrated predictor produces a smaller gain; nevertheless, there is a noticeable reduction in the maximum conﬁdence set size.
In (b), we show the conﬁdence set sizes for images correctly vs. incorrectly labeled by ResNet.
As expected, the sizes are substantially larger for incorrectly labeled images.
Finally, in (c) and (d), we show how the sizes vary with (cid:15) and δ , respectively.
As expected, the dependence on (cid:15) is much more pronounced (note that δ is log-scale).
Half-cheetah.
We use our algorithm to compute conﬁdence sets for a probabilistic neural network dynamics model (Chua et al., 2018) for the half-cheetah environment (Brockman et al., 2016), for (cid:15) = 0 .
01 , δ = 10 − 5 , H = 20 time steps, and n = 5000 validation rollouts.
When using temperature scaling to calibrate f ˆ φ to obtain f ˆ φ, ˆ τ , we calibrate each dimension of time steps independently (i.e., we ﬁt H parameters, where H is time horizon).
We show the results in Figure 2.
In (a), we compare to three ablations.
In addition to C and D (which are as for ResNet), A refers to using the accumulated variance ˜Σ t instead of the one-step predicted variances Σ t = Σ(¯ x t − 1 ) .
Thus, A + C + D refers to our approach.
In (b), we show the same ablations over the entire trajectory until t = 20 .
As can be seen, using the calibrated predictor produces a large gain and using the direct bound produce a large gain on average; these gains are most noticeable in the tails.
Using the accumulated conﬁdence produces a smaller, but still signiﬁcant, gain.
In (c) and (d), we show how the sizes vary with (cid:15) and δ , respectively.
The trends are similar those for ResNet.
Conclusion.
We have proposed an algorithm for constructing PAC conﬁdence sets for deep neural networks.
Future work includes extending these results to more complex tasks (e.g., structured prediction), and handling covariate shift (e.g., to handle policy updates in reinforcement learning).
A D ISCUSSION OF A LGORITHM D ESIGN C HOICES A. U SEFULNESS OF T EMPERATURE S CALING
In this section, we discuss why temperature scaling can help improve the predicted conﬁdence sets.
A concern is that temperature scaling does not change the ordering of label probabilities.
Thus, we may expect that temperature scaling does not affect the predicted conﬁdence sets.
However, this fact only holds when considering a single input x —i.e., the ordering of the probabilities p ( y | x ) for y ∈ Y is not changed by temperature scaling.
Indeed, the order of conﬁdences for labels for different inputs can change.
For a concrete example, consider two inputs x and x (cid:48) , and the case
Y = { 0 , 1 , 2 } .
Assume that the label probabilities are
[Equation]
Now, if we take temperature τ very large, then the labels become roughly
[Equation]
As a consequence, there are conﬁdence sets that are achievable when using f τ that are not achievable when using f .
In particular, the conﬁdence sets
[Equation]
can be achieved using f τ (e.g., with e − T = 2 / 5 ).
However, it is impossible to achieve these conﬁdence sets using f for any choice of T , since if 1 ∈ C T ( x (cid:48) ) , then it must be the case that
C T ( x ) = { 0 , 1 , 2 } .
Intuitively, we expect calibrated prediction to improve the ordering of probabilities across different inputs.
Our experiments support this intuition, since they show that empirically, using calibrated predictors f τ produces conﬁdence sets of smaller size.
A. U SEFULNESS OF D IRECT B OUND
One key design choice is to use a specialized generalization bound that directly provides PAC guarantees on our conﬁdence sets rather than simply applying the VC dimension bound.
The easiest way to determine which bound is better is to examine which one produces a smaller conﬁdence set.
In our approach, the size of the conﬁdence set decreases monotonically with the choice of α = α ( n, (cid:15), δ ) in (3).
Thus, the bound that produces larger α is better.
Recall that the VC dimension bound produces
[Equation]
whereas our direct bound produces (for k = 0 )
[Equation]
Directly comparing these two choices of α is difﬁcult, but our experiments show empirically that using the direct bound outperforms using the indirect bound.
A more direct way to compare the two approaches is to instead ask how large n needs to be to achieve α ( n, (cid:15), δ )
= 0 .
For α VC , it is easy to check that we need
[Equation]
Thus, we need n to be at least O (log(1 /δ ) /(cid:15) 2 ) (and possibly greater, to account for the log(2 n ) term).
In contrast, for our direct bound, α = 0 corresponds to the case k = 0 .
To achieve k = 0 , it sufﬁces to have n satisfying (1 − (cid:15) ) n < δ .
Using (1 − (cid:15) ) n ≤ e − n(cid:15) , it sufﬁces to have n satisfying n ≥ log(1 /δ )
(cid:15) .
[Figure]
In other words, n only needs to be O (log(1 /δ ) /(cid:15) ) .
For small (cid:15) (e.g., (cid:15) = 0 . 01 ), we need 100 × fewer samples to achieve the same size conﬁdence set (i.e., with choice α ( n, (cid:15), δ ) = 0 ).
In Figure(right), we compute the exact values of n needed to get α ( n, (cid:15), δ ) =
as a function of (cid:15) for each bound (ﬁxing δ = 10 − 5 ).
As expected, our bound requires substantially smaller n .
Finally, in Figure 3 (right), we compare the magnitude of n needed to achieve larger values of α using our direct bound; for simplicity, we actually consider larger values of k (where α = k/n ), but the qualitative insights are the same.
As can be seen, even for large k , (e.g., k = 50 ), the number of samples increases, but not substantially.
B T HEORETICAL G UARANTEES B. A SSUMPTIONS
We make two additional technical assumptions in Theorem 1, both of which are standard.
First, we assume that f is measurable; this assumption holds for all models used in practice, including neural networks (e.g., it holds as long as f is continuous).
Second, letting φ : Z → R > 0 , where Z = X × Y , be deﬁned by φ (( x, y ))
= − log f
( y | x ) , we assume that the distribution ¯ D induced by φ on R has continuous cumulative distribution function (CDF).
More precisely, letting µ D be the measure deﬁning D , then ¯ D is deﬁned by the measure
[Equation]
where φ − 1 : R → 2 Z is the inverse of φ in the sense that z ∈
φ − 1 ( φ ( z )) for all z ∈ Z .
Then, we assume that the CDF corresponding to ¯ D is continuous.
This second assumption is standard in statistical learning theory (Kearns & Vazirani, 1994).
Essentially, it says that for any t ∈ R , the probability that t = − log f ( y | x ) must equal zero.
This assumption should hold unless p ( x, y ) or f ( y | x ) are degenerate in some way.
Furthermore, we can detect this case.
In particular, the failure mode corresponds to the case that we see multiple points with the same value − log f
( y | x ) .
Thus, choosing ˆ T = − log f
( y | x ) would include all these points, so the realized error rate α is larger than desired for ˆ T .
In this case, we can simply choose a slightly larger ˆ T to avoid this problem.
B. P ROOF OF T HEOREM
At a high level, our proof proceeds in three steps.
First, we show that a conﬁdence set predictor C T can be encoded as a binary classiﬁer M T .
Second, we show that a PAC bound for M T implies a
PAC bound for C T (where in both cases, the unknown parameter is T ∈ R > 0 ).
Third, we prove PAC bounds on the error of M ˆ T ; by the second step, these bounds complete our proof.
Encoding C T as a binary classiﬁer M T .
We begin by showing how the problem of learning a PAC conﬁdence set predictor C T reduces to the problem of learning a PAC binary classiﬁer M T .First, we show that for any T ∈ R > 0 , the conﬁdence set predictor C T can be encoded as a binary classiﬁer
M T .
Consider any parameter T ∈
Θ = R > 0 .
Recall that we use the model f
( y | x ) to construct the conﬁdence set predictor
[Equation]
Now, deﬁne the map φ : Z → R > 0 by φ ( x, y )
= − log f
( y | x ) , where Z = X
× Y , and deﬁne the binary classiﬁer M T : R > 0 → { 0 , 1 } by
M T ( t ) =
I [ t ≤ T ] .
Here, I [ s ] is the indicator function, which returns one if a statement s is true and zero otherwise.
We claim that
[Equation]
To see this claim, note that
[Equation]
as claimed.
PAC bound for M T implies PAC bound for C T .
Next, we show that a PAC bound for M T implies a PAC bound for C T .
More precisely, we design a data distribution ˜ D and loss ˜ (cid:96) , and show that (i) the distribution of ˜ T (trained to optimize M T ) is the same as the distribution of ˆ T (constructed using our algorithm), and (ii) a PAC bound for M ˜ T (where ˜ T is trained on data from ˜ D ) implies a PAC bound for C ˜ T .
We show that as a consequence, a PAC bound on M ˜ T implies a PAC bound on C ˆ T .
We begin by constructing ˜ D and ˜ (cid:96) .
To this end, recall that D is a given distribution over X × Y .
We deﬁne a data distribution ˜ D over ˜ X × ˜ Y , where ˜ X
= R > 0 and ˜ Y
= { 0 , 1 } , as follows.
The ﬁrst component of ˜ D is the distribution over ˜ X induced by φ from D , and the second component is the distribution over ˜ Y that places all probability mass on 1 .
Formally, ˜ D exists assuming φ is measurable, so the induced distribution exists; for all our choices of f (i.e., categorical or Gaussian), this property is satisﬁed.
Then,
[Equation]
where µ ˜ D is the measure encoding ˜ D and µ D is the measure encoding D .
Furthermore, we deﬁne (cid:96) :
˜ Y ×
˜ Y → { 0 , 1 } to be the 0-1 loss (cid:96) ( a, a (cid:48) )
=
I [ a (cid:54) = a (cid:48) ] .
Finally, let ˆ T be chosen using our algorithm—i.e.,
[Equation] [Equation]
L
[Equation]
for any α ∈ R ≥ 0 , and let ˜ T be chosen similarly for M T —i.e.,
[Equation]
Now, we show (i) above.
In particular, we claim that ˆ T ( Z ) has the same distribution as ˜ T ( ˜ Z ) , where
Z ∼ D n and ˜ Z ∼ ˜ D n are random datasets.
To this end, deﬁne Φ : Z n (cid:55)→ ˜
Z n by Φ(( z 1 , ..., z n ))
=
(( φ ( z 1 ) , 1) , ..., ( φ ( z n ) , 1)) .
Note that
[Equation]
Z )
|
[Equation] [Equation]
[Equation] [Equation]
from which it follows that
[Equation]
By construction of Φ , the random variables ˜ Z and Φ( Z ) have the same distribution; thus, it follows that the random variables ˜ T ( ˜ Z ) and ˜ T (Φ( Z )) have the same distribution as well.
Since ˆ T ( Z ) =
˜ T (Φ( Z )) , it follows that ˆ T ( Z ) has the same distribution as ˜ T ( ˜ Z ) , as claimed.
Next, we show (ii) above.
In particular, we claim that a PAC bound for M ˜ T ( ˜ Z ) —i.e.,
[Equation]
implies a PAC bound for C ˜ T ( ˜ Z ) —i.e.,
[Equation]
where the true losses are
[Equation]
Note that it sufﬁces to show that the true loss for C T equals the true loss for M T —i.e.,
[Equation]
since this equation (together with the PAC bound for M ˜ T ( ˜ Z ) ) implies
[Equation]
M
[Equation]
Z ) )
[Equation]
, as desired.
To see the claim, note that
[Equation]
Now, using the change of variables t (cid:55)→ φ ( z ) , we have
[Equation]
Then, using (12), we have
[Equation]
as claimed.
Finally, combining (i) and (ii), we have
[Equation]
where the ﬁrst equality follows since (i) says that ˆ T ( Z ) (where Z ∼ D n ) has the same distribution as ˜ T ( ˜ Z ) (where ˜ Z ∼ ˜ D n ), and the second inequality follows by (ii).
Generalization bound.
Finally, we prove the PAC bound
[Equation]
for M ˜ T , where δ 0 =
(cid:80) ki =0 (cid:0) ni (cid:1) (cid:15) i (1 − (cid:15) )
n − i ; for conciseness, we have dropped the dependence of ˜ T on ˜ Z .
By the previous step, this bound implies the theorem statement.
To this end, we ﬁrst simplify the left-hand side of the inequality (13).
In particular, let T ∗ be the smallest T for which ˜
L ( M T ∗ ) = (cid:15) ; such a T ∗ exists by our assumption that ˜ D has continuous density function.
First, we claim that T < T ∗ implies ˜ L ( M T ) >
˜ L ( M T ∗ ) .
Assuming T < T ∗ , then
[Equation]
Assuming T ≥ T ∗ , we can similarly show that ˜ L ( M ˆ T ) ≤ ˜ L ( M T ∗ ) .
It follows that
[Equation]
As a consequence, (13) is equivalent to
[Equation]
Next, recall that ˜ T must satisfy ˜ L ( M ˜ T ; ˜ Z ) ≤ α , where
[Equation]
Assuming ˆ T < T ∗ , and using k = n · α , it follows that
[Equation]
t
[Equation] [Equation]
As a consequence, we have
[Equation]
By our deﬁnition of T ∗ , the event in the ﬁnal expression says that the sum of n i.i.d.
Bernoulli random variables I [ t > T ∗ ] ∼ Bernoulli ( (cid:15) ) is at most k .
Thus, this event follows a distribution
Binomial ( n, (cid:15) ) , so
[Equation]
Binomial
[Equation]
as claimed.
The theorem statement follows.
(cid:4)
C D ETAILS ON P ROBABILITY F ORECASTERS FOR S PECIFIC T ASKS
In this section, we describe architectures for probability forecasters for classiﬁcation, regression, and model-based reinforcement learning.
Classiﬁcation.
For the case Y = { 1 , ..., Y } , we choose the probability forecaster f to be a neural network with a softmax output.
Then, we can compute a given conﬁdence set
[Equation]
by explicitly enumerating y ∈ Y .
We measure the size of C T ( x ) as | C T ( x ) | .
Regression.
For the case Y
=
R , we choose the probability forecaster f to be a neural network that outputs the parameters ( µ, σ
) ∈ Y × R > 0 of a Gaussian distribution.
Then, we have
[Equation]
This choice generalizes to Y = R d by having f output the parameters ( µ, Σ)
∈
Y × S d (cid:31)
(where
S d (cid:31) 0 is the set of d dimensional symmetric positive deﬁnite matrices) of a d dimensional Gaussian distribution.
Note that C T ( x ) is an ellipsoid C T ( x ) = µ + Λ S
d
− 1 , where Λ ∈ R d × d and S d − 1 is the unit sphere in R d ; in particular, Λ = D − 12 Q , where QDQ (cid:62) is the eigendecomposition of (2 T − d ln 2 π − ln det Σ) − 1 · Σ − 1 .
We measure the size of C T ( x ) as (cid:107) Λ (cid:107) F , where (cid:107) · (cid:107) F is the Frobenius norm.
Model-based reinforcement learning.
In model-based reinforcement learning, the goal is to predict trajectories based on a model of the dynamics.
We consider an MDP with states X
⊆
R
d X , actions
U ⊆ R d U , an unknown distribution over initial states x 0 ∼ d 0 , and unknown dynamics g ∗ ( x (cid:48) | x, u ) mapping a state-action pair ( x, u ) ∈
X × U to a distribution over states x (cid:48) ∈ X .
We assume a ﬁxed, known policy π ( u | x ) , mapping a state x ∈ X to a distribution over actions u ∈ U .
The (unknown) closed-loop dynamics are f ∗ ( x (cid:48) | x ) =
E π ( u | x ) [ g ∗ ( x (cid:48)
| x, u )] .
Given initial state x 0 ∈ X and time horizon H ∈ N , we can sample a trajectory x ∗ 1: H =
( x ∗ 1 , ..., x ∗ H ) ∼ f ∗ by setting x ∗ 0 = x 0 and sequentially sampling x ∗ t +1 ∼ f ∗ ( · | x ∗ t ) for t ∈ { 0 , 1 , ..., H − 1 } .
Our goal is to predict a conﬁdence set C T ( x 0 ) ⊆
X H that contains x ∗ 1: H ∈ X H with high probability (according to both the randomness in initial states x
0 ∼ d 0 and in f ).
This problem is a multivariate regression problem with inputs X and outputs Y = X H .
We assume given a probability forecaster f
( x (cid:48)
| x ) =
N ( x (cid:48) ; µ ( x ) , Σ( x )) trained to predict the distribution over next states—i.e., f ( x (cid:48)
| x ) ≈ f ∗ ( x (cid:48)
| x ) .
Given initial state x 0 ∈ X and time horizon H ∈ N , we construct the mean trajectory ¯ x 1: H by setting ¯ x 0 = x 0 and letting ¯ x t +1 = µ (¯ x t ) .
To account for the fact that the variances accumulate over time, we sum them together to obtain the predicted variances ˜Σ 1:
[Equation]
—i.e
[Equation] [Equation]
Then, we use the probability forecast ˜ f
(¯ x 1: H , ˜Σ 1: H ) =
N (¯ x 1: H , ˜Σ 1: H ) (where we think of ¯ x 1: H as a vector in R H · d X and ˜Σ 1: H as a block diagonal matrix in R ( H · d X ) × ( H · d X ) ) to construct conﬁdence sets.
Finally, we describe how we measure the size of a predicted conﬁdence set C T ( x 0 ) ⊆
X H .
In particular, note that C T ( x 0 ) has the form
[Equation]
[Figure]
i.e., C T,t ( x 0 ) is the conﬁdence set for the state x t reached after t time steps.
Then, we measure the size of the conﬁdence set for each component C T,t ( x 0 ) (for t ∈ { 1 , ..., H } ) individually, and take the average.
As in the case of regression, C T,t ( x 0 ) is an ellipsoid C T,t ( x 0 ) =
¯ x t + Λ t S d X −
; then, the size of C T ( x 0 ) is H − 1 (cid:80)
[Equation]
An additional detail is that when we calibrate this forecaster, we calibrate each component C T,t ( x 0 ) individually—i.e., we use H calibration parameters τ 1 , ..., τ H .
D A DDITIONAL R ESULTS D. C OMPARISON TO A DDITIONAL B ASELINES
We compare to two baselines that do not have theoretical guarantees.
We assume given a probability forecaster f
( y | x ) .
Then, given an input x ∈ X , we construct the conﬁdence set to satisfy (cid:88)
[Equation]
More precisely, we ﬁrst rank the labels in decreasing order of f ( y | x ) , to obtain a list ( y 1 , y 2 , ..., y |Y| ) .
Then, we choose the smallest k such that (14) holds for C ( x )
=
{ y 1 , ..., y k } .
Intuitively, if the probabilities f ( y | x ) are correct (i.e., f ( y | x ) is the true probability of y given x ), then this conﬁdence set should contain the true label y with high probability.
For regression, we cannot explicitly rank labels
y ∈ Y ⊆ R d , but they are monotonically decreasing away from the mean.
Then, assuming f ( y | x ) =
N ( y ; µ ( x ) , Σ( x )) is Gaussian, we take an ellipsoid of shape Σ( x ) around µ ( x ) with minimum radius that captures 1 − (cid:15) of the probability mass of f ( y | x ) .
More precisely, we choose
[Equation]
D
C+D
C o n f .
s e t
s i
z
e
D
C+D
C o n f .
s e t
s i
z e where C T ( x ) =
{ y ∈ Y | f
( y | x ) ≥ e − T } as before.
Note that unlike our algorithm, the threshold ˆ T ( x ) is not a learned parameter, but is computed independently for each new input x .
We can solve for ˆ T ( x ) efﬁciently by changing basis to convert f ( y | x ) to a standard Gaussian distribution, and then using the error function to compute the cutoff that includes the desired probability mass.
In Figure 4, we compare the conﬁdence sets constructed using this approach with (i) the forecaster f ˆ φ ( y | x ) without any calibration, and (ii) the calibrated forecaster f ˆ φ, ˆ τ ( y | x ) .
We plot both the conﬁdence set sizes and the empirical error rates.
For the latter, recall that a conﬁdence set predictor
C is correct if L ( C ) < (cid:15) , where L ( C ) the true error rate.
However, we cannot measure L ( C ) ; instead, we approximate it on a held-out test set Z test ⊆
X × Y —i.e., L ( C )
≈ ˆ L ( C ; Z test ) , where ˆ L ( C ; Z test )
= | Z test | (cid:88) ( x
[Equation]
y
[Equation]
∈ Z test
I [ y (cid:54)∈ C ( x )] .
Intuitively, ˆ L ( C ; Z test ) is the fraction of inputs (
x, y ) ∈
Z test such that the predicted conﬁdence set for x does not contain y .
We say a conﬁdence set C is empirically correct when ˆ L ( C ; Z test ) < (cid:15) .
Recall that our algorithm guarantees correctness with probability at least 1 − δ , where δ = 10 − 5 .
As can be seen, the baseline approaches are not empirically correct in all cases.
In one case—namely, the baseline with the calibrated forecaster on ImageNet—the conﬁdence sets are almost empirically correct.
However, in this case, the conﬁdence sets are much larger than ours, despite the fact that the error rate of our conﬁdence sets are empirically correct.
Thus, our algorithms outperform the baselines in all cases.
D. R ESULTS ON R EGRESSION D ATASETS
We ran our algorithm on two small regression baselines—the Auto MPG dataset (Quinlan, 1993) and the student grade dataset (Cortez & Silva, 2008).
We show results in Figure 5.
The parameters we use are (cid:15) = 0 .
1 and δ = 0 .
05 ; we use larger parameters since the datasets are very small, so it is hard to get reasonable conﬁdence sets for larger parameter values.
For the Auto MPG dataset, the empirical conﬁdence set error of our ﬁnal model (C+D) is ˆ L ( C ; Z test )
= 0 .
0597 , so these are empirically correct.
For the student grade dataset, the error is ˆ L ( C ; Z test )
= 0 .
1250 , which is slightly larger than desired; this failure is likely due to the fact that the failure probability δ = 0 .is
somewhat large.
D. A DDITIONAL R ESULTS ON I MAGE N ET AND H ALF -C HEETAH
Table 2 & 3 show examples of ResNet conﬁdence set sizes for ImageNet images.
Table 4 shows results for varying (cid:15), δ on ResNet.
Tables 5 & 6 show
[Table]
a i r s h
i p ze
b r a b a n a n a ca r ou s e l s t a r ﬁ s h s t r ee t s i gn
1 ≤ | C
( x ) | <5 ≤ | C ( x ) | <10 ≤ | C ( x ) | <
[Table]
 
[Table]

[Table]
   
[Table]
 
[Table]
 
[Table]
   
[Table]
 
[Table]
 
[Table]
  
[Table]
 
[Table]
   
[Table]
    
[Table]
     
[Table]
  
[Figure] [Figure]
[Table]
[Figure] [Figure]