PAC P REDICTION S ETS U NDER C OVARIATE S HIFT
A BSTRACT
An important challenge facing modern machine learning is how to rigorously quantify the uncertainty of model predictions.
Conveying uncertainty is especially important when there are changes to the underlying data distribution that might invalidate the predictive model.
Yet, most existing uncertainty quantiﬁcation algorithms break down in the presence of such shifts.
We propose a novel approach that addresses this challenge by constructing probably approximately correct (PAC) prediction sets in the presence of covariate shift.
Our approach focuses on the setting where there is a covariate shift from the source distribution (where we have labeled training examples) to the target distribution (for which we want to quantify uncertainty).
Our algorithm assumes given importance weights that encode how the probabilities of the training examples change under the covariate shift.
In practice, importance weights typically need to be estimated; thus, we extend our algorithm to the setting where we are given conﬁdence intervals for the importance weights.
We demonstrate the effectiveness of our approach on covariate shifts based on DomainNet and ImageNet.
Our algorithm satisﬁes
the
PAC constraint, and gives prediction sets with the smallest average normalized size among approaches that always satisfy the PAC constraint.
I NTRODUCTION
A key challenge in machine learning is quantifying prediction uncertainty.
A promising approach is via prediction sets , where the model predicts a set of labels instead of a single label.
For example, prediction sets can be used by a robot to navigate to avoid regions where the prediction set includes an obstacle, or in healthcare to notify a doctor if the prediction set includes a problematic diagnosis.
Various methods based on these approaches can provide probabilistic correctness guarantees (i.e., that the predicted set contains the true label with high probability) when the training and test distributions are equal—formally, assuming the observations are exchangeable (Vovk et al., 2005;
Papadopoulos et al., 2002; Lei et al., 2015) or i.i.d.
(Vovk, 2013; Park et al., 2020a; Bates et al., 2021).
However, this assumption often fails to hold in practice due to covariate shift —i.e., where the input distribution changes but the conditional label distribution remains the same (Sugiyama et al., 2007; Qui ˜ nonero-Candela et al., 2009).
These shifts can be both natural (e.g., changes in color and lighting)
(Hendrycks & Dietterich, 2019) or adversarial (e.g., (cid:96) ∞ attacks) (Szegedy et al., 2014).
We consider unsupervised domain adaptation (Ben-David et al., 2007), where we are given labeled examples from the source domain, but only unlabeled examples from the target (covariate shifted) domain.
We propose an algorithm that constructs probably approximately correct (PAC) prediction sets under bounded covariate shifts (Wilks, 1941; Valiant, 1984)—i.e., with high probability over [Figure]
Our algorithm uses importance weights to capture the likelihood of a source example under the target domain.
When the importance weights are known, it uses rejection sampling (von Neumann, 1951) to construct the prediction sets.
Often, the importance weights must be estimated, in which case we need to account for estimation error.
When only given conﬁdence intervals around the importance weights, our algorithm constructs prediction sets that are robust to this uncertainty.
We evaluate our approach in two settings.
First, we consider rate shift , where the model is trained on a broad domain, but deployed on a narrow domain—e.g., an autonomous car may be trained on both daytime and nighttime images but operate at night.
Even if the model continues to perform well, this covariate shift may invalidate the prediction sets.
We show that our approach constructs valid prediction sets under rate shifts constructed from DomainNet (Peng et al., 2019), whereas several existing approaches do not.
{+Second, we consider support shift , where the model is trained on one domain, but evaluated on another domain with shifted support. We train ResNet
+}
(He et al., 2016) using unsupervised domain adaptation (Ganin et al., 2016) on both ImageNet-C synthetic perturbations (Hendrycks & Dietterich, 2019) and PGD adversarial perturbations (Madry et al., 2017)
to
ImageNet (Russakovsky et al., 2015).
Our PS-W algorithm satisﬁes the PAC constraint, and gives prediction sets with the smallest average normalized size among approaches that always satisfy the
PAC constraint.
See Figure 1 for a summary of our approach and results.
Related work.
Our work is related to conformal prediction (Vovk et al., 2005; Balasubramanian et al., 2014) without a shift, where the goal is to construct models that predict sets of labels designed to include the true label with high probability (instead of predicting individual labels) under the assumption that the source and target distributions are the same.
In particular, our setting is related to inductive (or split ) conformal prediction (Papadopoulos et al., 2002; Vovk, 2013; Lei et al., 2015), where the training set is split into a proper training set used to train a traditional predictive model, and a calibration set used to construct the prediction sets.
The general approach is to train a model f : X × Y → R that outputs conformity scores , and then to choose a threshold τ ∈ R that satisﬁes a correctness guarantee, where the corresponding prediction sets are C ( x ) = { y ∈ Y | f
( x, y ) ≥ τ } (Park et al., 2020a; Gupta et al., 2021); this notation is deﬁned in Section 2.
Several kinds of correctness guarantees under no shift have been considered.
One possibility is input conditional validity (Vovk, 2013; Barber et al., 2020), which ensures correctness for all future covariates x , with high probability only over the conditional label distribution p ( y | x ) .
This guarantee is very strong, and hard to ensure in practice.
A weaker notion is approximate (Lei & Wasserman, 2014; Barber et al., 2020) or group (Romano et al., 2019) conditional validity, which ensures correctness with high probability over p ( y | x ) as well as some distribution p ( x ) over a subgroup.
Finally, unconditional validity ensures only correctness over the joint distribution p ( x, y ) .
We focus on unconditional validity, though our approach extends readily to group conditional validity.
A separate issue, arising in the no shift setting, is how to condition on the calibration set Z .
A conventional goal is unconditional validity, over the distribution p ( x, y, Z ) .
We refer to this strategy as fully unconditional validity .
However, the guarantee we consider uses a separate conﬁdence level for the training data, which is called a training conditional guarantee (Vovk, 2013); this correctness notion is equivalent to a PAC correctness guarantee (Park et al., 2020a), and is also equivalent to the standard “content” guarantee with a certain conﬁdence level for tolerance regions (Wilks, 1941; Fraser, 1956).
We build on Park et al.
(2020a), which formulates the problem of choosing τ as learning a binary classiﬁer where the input and parameter spaces are both one-dimensional; thus, the correctness guarantee corresponds to a PAC generalization bound.
This approach is widely applicable since it can work with a variety of objectives (Bates et al., 2021).
Recent work has extended inductive conformal prediction to a setting with covariate shift (Tibshirani et al., 2019; Lei & Cand`es, 2020); similarly, Podkopaev & Ramdas (2021) considers conformal prediction under label shift (Lipton et al., 2018), i.e., assuming the conditional probabilities p ( x | y ) do not change.
These approaches start from the assumption that the true importance weights (IWs) are known (Tibshirani et al., 2019; Podkopaev & Ramdas, 2021), or assume some properties of the estimated IWs (Lei & Cand`es, 2020), whereas our approach considers a special form of “ambiguity” in the estimated IWs.
Furthermore, they are focused on fully unconditional validity, whereas we obtain PAC prediction sets.
In addition, Cauchois et al.
(2020) designs conﬁdence sets that are robust to all distribution shifts with bounded f -divergence; in contrast, we consider the unsupervised learning setting where we have unlabeled examples from the target distribution.
We provide additional related work in Appendix A.
B ACKGROUND ON PAC P REDICTION S ETS
We give background on PAC prediction sets (Park et al., 2020a); we also re-interpret this approach using Clopper-Pearson conﬁdence intervals (Clopper & Pearson, 1934), which aids our analysis.
2. PAC P REDICTION S ETS A LGORITHM
Let x ∈
X
be covariates and y ∈ Y be labels.
We consider a source distribution P over X × Y with probability density function (PDF) p ( x, y ) .
1 A prediction set 2 is a set-valued function C : X → 2 Y .
Inputs.
We are given a held-out calibration set S m of i.i.d.
samples ( x i , y i ) ∼ P for i ∈ [ m ] := { 1 , . . .
, m } , written as S m ∼ P m , and a score function f : X × Y
→ R ≥ 0 .
For example, f ( x, y ) can be a prediction for the probability that y is the label for x .
The score function can be arbitrary, but score functions assigning higher scores to likely labels yield smaller prediction sets.
PAC prediction set.
A PAC prediction set is a set-valued function C : X → 2 Y satisfying the following two conditions.
First, C is approximately correct in the sense that its expected error (failing to contain the true label) is bounded by a given ε ∈ (0 , 1) , i.e., L P ( C [Equation]
[Equation]
Second, noting that C S m is constructed based on a calibration set S m ∼ P m , the condition that C S m is approximately correct must be satisﬁed with high probability—i.e., for given δ ∈ (0 , 1) , we have
[Equation]
Our goal is to devise an algorithm for constructing a PAC prediction set C .
Letting C ( x )
=
Y for all x ∈ X always satisﬁes both conditions above, but this extreme case would be uninformative if used as a prediction set.
Therefore, we additionally want to minimize the expected size
E
[ S ( C ( x ))] of the prediction sets C ( x ) , where S : 2 Y → R ≥ 0 is a size measure, which is application speciﬁc (e.g., the cardinality of a set in classiﬁcation); however, we only rely on the monotonicity of the size measure with respect to the prediction set parameterization in construction.
Algorithm.
To construct C , we ﬁrst deﬁne the search space of possible prediction sets along with the size measure S .
We parameterize C by a scalar τ ∈ T :
= R ≥ 0 as
[Equation]
i.e., τ is the threshold on f ( x, y ) above which we include y in C ( x ) .
Importantly, τ controls the tradeoff between size and expected error.
The reason is that if τ 1 ≤ τ 2 , then C τ 2 ( x ) ⊆
C τ 1 ( x ) for all x ∈ X .
Thus, size is monotonically decreasing in τ —i.e., S ( C τ 2 ( x )) ≤ S ( C τ 1 ( x )) for all x ∈ X , and error is monotonically increasing in τ —i.e., L P ( C τ 1 ) ≤ L P ( C τ 2 ) .
See Figure 1b for an illustration, and Park et al.
(2020a) and Gupta et al.
(2021) for details.
As a consequence, a typical goal is to construct C τ that provably contains the true label with high probability, while empirically minimizing size (Vovk et al., 2005; Gupta et al., 2021).
In our setting, we want to maximize τ (equivalently, minimize expected size) subject to the constraint that
C τ is PAC.
Let ¯ L S m ( C τ ) :
= (cid:80) ( x
,y ) ∈ S m 1 ( y / ∈ C τ ( x )) be the empirical error count on S m , and
F ( k ; m, ε ) =
(cid:80) ki =0 (cid:0) mk (cid:1)
ε i (1 − ε ) m −
i be the cumulative distribution function (CDF) of the binomial distribution Binom ( m, ε ) with m trials and success probability ε .
In prior work, Park et al.
(2020a) constructs C ˆ τ by solving the following problem:
[Equation]
where k ( m, ε, δ ) = max k ∈ N ∪{ 0 } k subj.
to F ( k ; m, ε ) ≤ δ.
Intuitively, the PAC guarantee via this construction is related to the Binomial distribution; ¯ L S m ( C ) has distribution Binom ( m, L P ( C ))
(given a ﬁxed C ), since 1 ( y (cid:54)∈ C ( x )) has a Bernoulli ( L P ( C )) distribution when ( x, y ) ∼ P .
Thus, k ( m, ε, δ ) deﬁnes a “conﬁdence interval” such that if ¯ L S m ( C ) ≤ k ( m, ε, δ ) , then L P ( C ) ≤ ε with probability at least 1 − δ .
Below, we formalize this intuition by drawing a connection to the Clopper-Pearson conﬁdence interval.
2. C LOPPER -P EARSON I NTERPRETATION
We interpret (1) using the Clopper-Pearson (CP) upper bound θ ( k ; m, δ ) ∈ [0 , 1] (Clopper &
Pearson, 1934; Park et al., 2021).
This is an upper bound on the true success probability µ , constructed from a sample k ∼ Binom ( m, µ ) , which holds with probability at least 1 − δ , i.e.,
P k ∼ Binom ( m,µ ) [ µ ≤ θ ( k ; m, δ )] ≥ 1 − δ , where
[Equation]
Intuitively, for a ﬁxed C , ¯ L S m ( C ) ∼ Binom ( m, L P ( C )) , so the true error L P ( C ) is contained in the CP upper bound θ ( L S m ( C ); m ; δ ) with probability at least 1 − δ .
Intuitively, we can therefore choose τ so that this upper bound is ≤ ε .
However, we need to account for generalization error of our estimator.
To this end, we have the following (see Appendix D.1 for a proof and Algorithm 2 in
Appendix E for implementation details on the corresponding algorithm):
Theorem 1 Let U CP ( C, S m , δ ) :
= θ (¯ L S m ( C ); m ; δ ) , where θ is deﬁned in (2).
Let ˆ τ be the solution of the following problem 3 :
[Equation]
Then, we have P S m ∼ P m
[ L P ( C τ ) ≤ ε ] ≥ 1 − δ for any τ ≤ ˆ τ .
PAC P REDICTION S ETS U NDER C OVARIATE S HIFT
We extend the PAC prediction set algorithm described in Section 2 to the covariate shift setting.
Our novel approach combines rejection sampling (von Neumann, 1951) with Theorem 1.
3. P ROBLEM F ORMULATION
We assume labeled training examples from the source distribution P are given, but want to construct prediction sets that satisfy the PAC property with respect to a (possibly) shifted target distribution Q .
Both of these are distributions over X × Y ; denote their PDFs by p ( x, y ) and q ( x, y ) , respectively.
The challenge is that we are only given unlabeled examples from Q .
Preliminaries and assumptions.
We denote the likelihood ratio of covariate distributions by w ∗ ( x ) :
= q ( x )
/p
( x ) , also called the importance weight (IW) of x .
Our main assumption is the covariate shift condition, which says the label distributions are equal (i.e., p ( y | x ) =
q ( y | x ) ), while the covariate distributions may differ (i.e., we can have p ( x ) (cid:54) =
q ( x ) ).
Inputs.
We assume a labeled calibration set S m consisting of i.i.d.
samples ( x i , y i ) ∼ P (for i ∈ [ m ] ) is given, and a score function f : X × Y
→ R ≥ 0 .
For now, we also assume we have the true importance weights w ∗ i :=
w ∗ ( x i ) for each ( x i , y i ) ∼ P , as well as an upper bound b ≥ max x ∈X w ∗ ( x ) on the IWs.
In Sections 3.3 and Appendix B, we describe how to estimate these quantities in a way that provides guarantees under smoothness assumptions on the distributions.
Problem.
Our goal is to construct C S m that is a PAC prediction set under Q —i.e.,
[Equation]
where L Q ( C ) :
= P ( x,y ) ∼ Q [ y (cid:54)∈ C ( x )] is the error of C on Q , while empirically controlling its size.
3. R EJECTION S AMPLING S TRATEGY
Our strategy is to use rejection sampling to convert S m consisting of i.i.d.
samples from P into a labeled calibration set consisting of i.i.d.
samples from Q .
Rejection sampling.
Rejection sampling (von Neumann, 1951; Owen, 2013; Rubinstein & Kroese, 2016) is a technique for generating samples from a target distribution q ( x ) based on samples from a proposal distribution p ( x ) .
Given importance weights (IWs)
w ∗ ( x ) and an upper bound b ≥ max x ∈X w ∗ ( x ) , it constructs a set of i.i.d.
samples from q ( x ) .
Typically, rejection sampling is used when the proposal distribution is easy to sample compared to the target distribution.
In contrast, we use it to convert samples from the source distribution into samples from the target distribution.
In particular, our algorithm takes the proposal distribution to be the source covariate distribution P X , and the target distribution to be our target covariate distribution Q X .
Let V i ∼ U :=
Uniform ([0 , 1]) be i.i.d.
, V :=
( V 1 , . .
, V m ) , and (cid:126)w ∗ :
= ( w ∗ 1 , . .
. , w ∗ m ) with w ∗ i =
w ∗ ( x i ) as deﬁned before.
Then, given S m , it uses rejection sampling to construct a randomly chosen set of N samples
[Equation]
from Q X .
The expected number of samples is E [ N ] = m/b ; thus, rejection sampling is more effective when the proposal distribution is similar to the target.
Rejection sampling Clopper-Pearson (RSCP) bound.
Given T N , our algorithm uses the CP bound to construct a PAC prediction set C for Q .
Let σ i :=
1 ( V i ≤ w ∗ i /b ) indicate whether example ( x i , y i )
∈
S m is accepted—i.e., T N ( S m , V, (cid:126)w ∗ , b ) = { ( x i , y i )
∈ S m | σ i = 1 } ,where | T N ( S m , V, (cid:126)w ∗ , b ) | = (cid:80) mi
=1 σ i .
Then, it follows that U RSCP bounds the error on Q , where
[Equation]
Thus, we have the following (see Appendix D.3 for the proof, and Algorithm 4 in Appendix E for a detailed description of the PS-R algorithm that leverages this bound):
Theorem 2 Deﬁne U RSCP as in (5).
Let ˆ τ be the solution of the following problem:
[Equation]
Then, we have P S m ∼ P m ,V ∼
U m [ L Q ( C τ ) ≤ ε ] ≥ 1 − δ for any τ ≤ ˆ τ .
Note that V is sampled only once for Algorithm 4, so the randomness in V can contribute to the failure of the correctness guarantee; however, the failure rate is controlled by δ .
3. A PPROXIMATE I MPORTANCE W EIGHTS
So far, we have assumed that the true importance weight w ∗ ( x ) is known.
Since in practice, it needs to be estimated, we relax this assumption to only needing an uncertainty set of possible importance weights.
This allows us to handle estimation error in the weights.
Problem.
Let S Xm be unlabeled calibration examples from the source distribution (i.e., the covariates in S m ), T Xn be n unlabeled calibration examples from the target distribution (denoted by T Xn ∼
Q nX ), and (cid:126)w ∗ :=
( w ∗ 1 , ..., w ∗ m ) ∈ R m be the vector of true importance weights w ∗ i :=
w ∗ ( x i ) , for ( x i , y i ) ∈
S m .
Then, we assume an uncertainty set W ⊆
R m that has (cid:126)w ∗ with high probability, i.e.,
[Equation]
where δ w ∈ (0 , 1) .
We assume W has the form
[Equation]
for some w i and w i .
See the discussion on our choice of W in Appendix C.1.
Robust Clopper-Pearson bound.
To construct a PAC prediction set C for Q , it sufﬁces to bound the worst-case error over w ∈ W , i.e., we have the following (see Appendix D.4 for the proof):
Theorem 3 Suppose W satisﬁes (7).
Deﬁne U RSCP as in (5).
Let ˆ τ be the solution of the following:
[Equation]
Then, we have P S m ∼ P m ,V [Equation] [Equation] ) ≤ ε ] ≥ 1 − δ C − δ w for any τ ≤ ˆ τ .
A key challenge applying Theorem 3 is solving the maximum over w ∈ W .
We propose a simple greedy algorithm that achieves the maximum.
Greedy algorithm for robust U RSCP .
The RSCP bound U RSCP satisﬁes certain monotonicity properties that enable us to efﬁciently compute an upper bound to the maximum in (8).
In particular, if C makes an error on ( x i , y i ) (i.e., y i (cid:54)∈ C ( x i ) ), then U RSCP is monotonically non-decreasing in w ∗ i =
w ∗ ( x i ) ; intuitively, this holds since a larger w ∗ i increases the probability that ( x i , y i ) is included in T N ( S m , V, w ∗ , b ) , which in turn increases the empirical error ¯ L T N ( S m ,V,w ∗ ,b ) ( C ) .
Conversely, if C does not make an error on ( x i , y i )
(i.e., y i ∈ C ( x i ) ), U RSCP is non-increasing in w ∗ i .
More formally, we have the following result (see Appendix D.5 for a proof):
Lemma 1 For any i ∈ [ m ] , U RSCP ( C, S m , V, w ∗ , b, δ ) is monotonically non-decreasing in w ∗
i
if y i / ∈ C ( x i ) , and monotonically non-increasing in w ∗ i if y i ∈ C ( x i ) .
Our greedy algorithm leverages the monotonicity of U RSCP .
In particular, given W and C , the choice ˆ w :=
( ˆ w 1 , . . .
, ˆ w m ) , where
[Equation]
(9) is the maximum value over w ∈ W of the constraint U RSCP ( C τ , S m , V, w, b, δ C ) in (8), i.e.,
[Equation]
Thus, we have the following, which follows by (10) and the same argument as the Theorem 3:
Theorem 4 Suppose W satisﬁes (7).
Deﬁne ˆ w τ,S Xm ,T Xn as in (9), making the dependency on τ , S Xm , and T Xn explicit, and U RSCP as in (5).
Let ˆ τ be the solution of the following problem:
[Equation]
Then, we have P S m ∼ P m , [Equation] for any τ ≤ ˆ τ .
Importance weight estimation.
In general, to estimate the importance weights (IWs), some assumptions on their structure are required (Kanamori et al., 2009; Cortes et al., 2008; Nguyen et al., W ←
Xm 2010; Lipton et al., 2018).
We use a cluster-based approach (Cortes et al., 2008).
In particular, we heuristically partition the feature space of the score function f into K bins by using a probabilistic classiﬁer g that separates source and target examples, and then estimate the source and target covariate distributions p ( x ) and q ( x ) based on the smoothness assumption over the distributions, where the degree of the smoothness is parameterized by E .
Then, we can construct the uncertainty set W that satisﬁes the speciﬁed guarantee in (7).
See Appendix B for details.
Algorithm.
Our algorithm, called PS-W, is detailed in Algorithm 1; it solves (20) and also performs importance weight estimation according to (17) in Appendix B.
In particular, Algorithm 1 constructs a prediction set that satisﬁes the PAC guarantee in Theorem 4.
See Section 4.1 for our choice of parameters (e.g., K , E , and grid search parameters).
E XPERIMENTS
We show the efﬁcacy of our approach on rate and support shifts on DomainNet and ImageNet.
4. E XPERIMENTAL S ETUP
Models.
For each source-target distribution pair, we split each the labeled source data and unlabeled target data into train and calibration sets.
We use a separate labeled test set from the target for evaluation.
For each shift from source to target, we use a deep neural network score function f based on ResNet101 (He et al., 2016), trained using unsupervised domain adaptation based on the source and target training sets.
See Appendix F for details, including data split.
Prediction set construction.
To construct our prediction sets, we ﬁrst estimate IWs by training a probabilistic classiﬁer g using the source and target training sets.
Next, we use g to construct heuristic IWs w ( x ) =
/g ( s = 1 | x ) − 11 , where s = 1 if x is from source.
Then, we estimate the lower and upper bound of the true IWs using Theorem 5 with E = 0 .
001 and K = 10 bins (chosen to contain equal numbers of heuristic IWs), where we compute the lower and upper Clopper-Pearson interval using the source and target calibration sets.
Furthermore, given a conﬁdence level δ , we use δ C = δ w = δ/ 2 .
For the grid search in line 12 of Algorithm 1, we increase τ by 10 − 7 until the bound U RSCP exceeds 1 .
5 ε .
Finally, we evaluate the prediction set error on the labeled target test set.
Baselines.
We compare the proposed method in Algorithm 1 ( PS-W ) with the following:
• PS: The prediction set using Algorithm 2 that satisﬁes the PAC guarantee from Theorem 1, based on Park et al.
(2020a), which makes the i.i.d.
assumption.
• PS-C:
A Clopper-Pearson method addressing covariate shift by a conservative upper bound on the empirical loss (see Appendix E.2 for details), resulting in Algorithm 3 that solves the following:
[Equation]
• WSCI:
Weighted split conformal inference, proposed in (Tibshirani et al., 2019).
Under the exchangeability assumption (which is somewhat weaker than our i.i.d. assumption), this approach [Figure] provides correctness guarantees only for a single future test instance, i
e., it includes an ε conﬁdence level (denoted α in their paper) but not δ .
Also, it assumes the true IWs are known; following their experiments, we use the heuristic IWs constructed using g . • PS-R:
The prediction set using Algorithm 4 that satisﬁes the PAC guarantee from Theorem 2 with the heuristic IWs constructed using a probabilistic classiﬁer g . • PS-M: The prediction set using Algorithm 5, which is identical to PS-R except that PS-M estimates IWs heuristically using histogram density estimation and a calibration set.
In particular, we partition the IW values into bins, project source and target calibration examples into bins based on the value of the probabilistic classiﬁer g , and estimate the source density ˆ p B and target density ˆ q B for each bin The
estimated importance weight is ˆ w ( x )
= ˆ q B ( x ) / ˆ p B ( x ) , where ˆ p B and ˆ q B
are deﬁned in (14) and (15), respectively
Metrics.
We measure performance via the prediction set error and size on a held-out test set—i.e., error is the fraction of ( x, y ) such that y (cid:54)∈ C ( x ) and size is the average of | C ( x ) | over x .
Rate shifts on DomainNet.
We consider settings where the model is trained on data from a variety of domains, but is then deployed on a speciﬁc domain; we call such a shift rate shift .
For instance, a self-driving car may be trained on both day and night images, but tested during the night time.
While the model should still perform well since the target is a subset of the source, the covariate shift can nevertheless invalidate prediction set guarantees.
We consider rate shifts on DomainNet (Peng et al., 2019), which consists of images of 345 classes from six domains (sketch, clipart, painting, quickdraw, real, and infograph); we use all domains as the source and each domain as a target.
Support shifts on ImageNet.
Next, we consider support shifts, where the support of the target is different from the support of the source, but unsupervised domain adaptation is used to learn feature representations that align the two distributions (Ganin et al., 2016); then, the score function f is trained on this representation.
First, we consider ImageNet-C (Hendrycks & Dietterich, 2019), which modiﬁes the original ImageNet dataset (Russakovsky et al., 2015) using 15 synthetic perturbations with 5 severity levels.
We use 13 perturbations, omitting “snow” and “glass blur”, which are computationally expensive to run.
We consider the original ImageNet dataset as the source, and the all synthetic perturbations on all of ImageNet (denoted ImageNet-C13) as the target.
See
Appendix F.3 for data split.
Second, we consider adversarial shifts, where we generate adversarial examples for ImageNet using the PGD attack (Madry et al., 2017) with 0 .
01 (cid:96) ∞ -norm perturbations with respect to a pretrained ResNet101.
We consider the original ImageNet as the source and the adversarially perturbed ImageNet as the target.
4. E XPERIMENTAL R ESULTS
We summarize our results in Table 1, and provide more details in Figure 4 in Appendix G.2.
Our
PS-W algorithm satisﬁes the PAC constraint, and gives prediction sets with the smallest average normalized size among approaches that always satisfy the PAC constraint.
Rate shifts on DomainNet.
We use ε = 0 .
1 and δ = 10 − 5 .
As can be seen in Table 1, the prediction set error of our approach (PS-W) does not violate the error constraint ε = 0 .
1 , while all other comparing approaches (except for PS-C) violate it for at least one of the shifts.
While
PS-C satisﬁes the desired bound, it is overly conservative; its prediction sets are signiﬁcantly larger than necessary, making it less useful for uncertainty quantiﬁcation.
For the shift to Infograph in
Table 1, PS-W achieves relatively larger average prediction set size compared to other shifts; this is because the classiﬁcation error of the score function f over Infograph is large— 71 .
28% , whereas [Table] [Table] that over Sketch is 37 .
Even with the poor score function, our proposed approach still satisﬁes the ε = 0 .
1 error constraint.
Finally, while our approach PS-W generally performs best subject to satisfying the error constraint, we note that our ablations PS-R and PS-M also perform well, providing different tradeoffs.
First, the performance of PS-W is signiﬁcantly more reliable than
PS-R, but in some cases PS-R performs better (e.g., it produces slightly smaller prediction sets on rate shifts but signiﬁcantly larger sets on support shifts).
Alternatively, PS-M consistently produces smaller prediction sets, though it sometimes violates the ε error constraints.
Support shifts on ImageNet.
We show results for synthetic and adversarial shifts in Table 1 (and
Figure 4 in Appendix G.2).
As can be seen, the error of our approach (PS-W) is below the desired level.
PS-R performs poorly, likely due to the uncalibrated point IW estimation—we ﬁnd that calibrated importance weights mitigate these issues, though accounting for uncertainty in the IWs is necessary for achieving the desired error rate; see Appendix G.3 for details.
For adversarial shifts, the target classiﬁcation error of the source-trained ResNet101 and the domainadapted ResNet101 is 99 .
97% and 28 .
05% , respectively.
Domain adaptation can signiﬁcantly decrease average error rate, but label predictions still do not have guarantees.
However, our prediction set controls the prediction set error rate as speciﬁed by ε .
As shown in Figure 4, our prediction set function outputs a prediction set for a given example that includes the true label at least 90% of the time.
Thus, downstream modules can rely on this guarantee for further decision-making.
Ablation and sensitivity study.
We conduct an ablation study on the effect of IW calibration and the smoothness parameter E for
PS-W.
We observe that the IW calibration considering uncertainty intervals around calibrated IWs is required for the PAC guarantee (e.g., Figure 2b).
Also, we ﬁnd that a broad range of E -s can be used to satisfy the PAC guarantee; we believe this result is due to the use of a domain adapted score function.
See Appendices G.3 & G.5 for details.
C ONCLUSION
We propose a novel algorithm for building PAC prediction sets under covariate shift; it leverages rejection sampling and the Clopper-Pearson interval, and accounts for uncertain IWs.
We demonstrate the efﬁcacy of our approach on natural, synthetic, and adversarial covariate shifts on DomainNet and
ImageNet.
Future work includes providing optimality guarantees on the prediction set size, rigorous estimation of the hyperparameter E , and incorporating probabilistic IW uncertainty estimates.
Ethics statement.
We do not foresee any signiﬁcant ethical issues with our work.
One possible issue is that end-users may overly trust the prediction sets if they do not understand the limitations of our approach—e.g., it is only a high probability guarantee.
Reproducibility statement.
Algorithms used for evaluation, including ours, are stated in Algorithm 1, Algorithm 2, Algorithm 3, Algorithm 4, and Algorithm 5, along with hyperparameters for algorithms in Section 4.1 and Appendix F. Related code is released 6 .
The proof of our theorems are stated in Appendix D; the required assumption is stated in Assumption 1.
A CKNOWLEDGMENTS
This work was supported in part by ARO W911NF-20-1-0080, AFRL and DARPA FA8750-18-C-0090, NSF award CCF 1910769, and NSF award 2031895 on the Mathematical and Scientiﬁc Foundations of Deep Learning (MoDL).
Any opinions, ﬁndings and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reﬂect the views of the Air Force Research Laboratory (AFRL), the Army
Research Ofﬁce (ARO), the Defense Advanced Research Projects Agency (DARPA), or the Department of
Defense, or the United States Government. The authors are grateful to Art Owen for helpful comments. Sample selection A A DDITIONAL R ELATED W ORK
Prediction sets under i.i.d.
assumption.
We built our proposed approach on the known PAC prediction set approach (Wilks, 1941; Park et al., 2020a) due to its simplicity and sample efﬁciency.
As other candidates, nested conformal prediction (Vovk et al., 2005; Gupta et al., 2021) reinterprets multiple known conformal prediction approaches using a scalar parameteriztion of prediction sets; but different from Wilks (1941) and Park et al. (2020a), it is not known to provide PAC guarantees.
Kivaranovic et al.
(2020) provides a PAC style guarantee, but their approach is limited to regression and is sample-inefﬁcient, i.e., it requires a sample of size O (1 /(cid:15) 2 ) , while Park et al.
(2020a) has a better sample complexity, as demonstrated in their paper.
Prediction sets under various settings.
Prior prediction set algorithms have been considered in several settings.
First, traditional conformal prediction (Vovk et al., 2005) often considers the setting where labeled examples arrive sequentially from the same distribution; there has also been work extending conformal prediction to the setting where the distribution is time-varying (Politis, 2015; Chernozhukov et al., 2018; Xu & Xie, 2021; Gibbs & Cand`es, 2021).
Alternatively, there has been work on constructing risk-controlling prediction sets for supervised learning setting (Vovk, 2013; Park et al., 2020a; Bates et al., 2021; Angelopoulos et al., 2021).
Finally, there has been recent work on conformal prediction in the meta-learning setting (Fisch et al., 2021); in particular, given a few labeled examples drawn from the new task, their approach leverages labeled examples from previous tasks to construct a conformal predictor for the new task, assuming the tasks are exchangeable. Developments after our work.
After our work was made publicly available, Jin et al.
(2021) has developed a different, robust conformal inference approach to constructing prediction sets with estimated weights under covariate shift. Their algorithm assumes given upper and lower bounds on the importance weights, and uses the worst-case quantile over all weights that satisfy the constraint to set the critical values. Further, Yang et al. (2022) have developed a doubly robust approach to construct prediction sets satisfying approximate marginal coverage under covariate shift (which can be robust to estimating the weights and per-covariate prediction error), leveraging semiparametric efﬁciency theory.
Qiu et al.
(2022) have developed a parallel approach for the PAC case.
Calibration.
An alternative way to quantify uncertainty is calibrated prediction (e.g., Brier, 1950;
Cox, 1958; Miller, 1962; Murphy, 1972; Lichtenstein et al., 1977; DeGroot & Fienberg, 1983; Guo et al., 2017, etc), which aims to ensure that among instances with a predicted conﬁdence p , the model is correct a fraction p of the time.
Techniques have been proposed to re-scale predicted conﬁdences to improve calibration (Platt, 1999; Guo et al., 2017; Zadrozny & Elkan, 2001; 2002;
Kuleshov & Liang, 2015; Kuleshov et al., 2018; Malik et al., 2019); including ones with theoretical guarantees (Kumar et al., 2019; Park et al., 2021) and ones that handle covariate shift (Park et al., 2020b; Wang et al., 2020). There are also methods to rigorously test calibration, dating back to Cox (1958); Miller (1962), see e.g., Lee et al. for a recent approach.
These approaches provide a qualitatively different form of uncertainty quantiﬁcation compared to the one we consider.
Rejection sampling.
Rejection sampling (sometimes accept-reject sampling) is a well-known technique (Owen, 2013; Rubinstein & Kroese, 2016) dating back at least to von Neumann (1951).
We highlight that most work on covariate shift relies on importance weighting; our rejection sampling approach is relatively less common and more novel.
In fact, to the best of our knowledge, only
Pagnoni et al.
(2018) has used rejection sampling for a different problem in this area.
IW estimation.
There has been a long line of work studying the problem of estimating importance weights (IWs), also called likelihood ratios, in a way that provides theoretical guarantees.
For instance, (Nguyen et al., 2010) provides a convergence rate analysis of IW estimators—under smoothness assumptions, they provide a ﬁnite sample bound on the Hellinger distance between the true IW and estimated IW.
Next, (Kanamori et al., 2009) shows a similar ﬁnite sample guarantee, assuming the true IW can be represented as a linear combination of kernels.
Finally, (Cortes et al., 2008) proposes non-parametric IW estimators, modeling the source and target distribution by histograms over clusters in sample space.
The IW estimation approaches can be used in conjuction with prediction set construction.
Compared to Cand`es et al.
(2021), which only guarantees asymptotic validity under certain assumptions on the estimated IWs, Theorem 4 provides a ﬁnite-sample correctness guarantee.
Furthermore, we explicitly describe an algorithm for approximate IWs, required by Theorem 4, in Appendix B.
B I MPORTANCE W EIGHT E STIMATION
In general, to estimate the importance weights (IWs), some assumptions on their structure are required.
A number of approaches have been proposed, with varying guarantees under different assumptions (Kanamori et al., 2009; Cortes et al., 2008; Nguyen et al., 2010; Lipton et al., 2018).
We use a cluster-based approach (Cortes et al., 2008); our approach is compatible with any of these strategies if they can be modiﬁed to provide uncertainty estimates of the IWs.
In our approach, given a partition X =
(cid:83) Kj =1 B j into bins, we can estimate the IWs based on the fractions of source and target samples in each bin.
If the partition is sufﬁciently ﬁne, then we can obtain conﬁdence intervals around the estimated IWs with ﬁnite-sample guarantees.
However, this strategy requires the number of bins in the partition to be exponential in the dimension of X .
Thus, in practice, we use a heuristic to construct the partition.
We describe the cluster-based approach and our partition construction heuristic below.
B. C LUSTER - BASED APPROACH
We assume given unlabeled calibration sets S Xm and T Xn , where S Xm consists of i.i.d.
samples x i ∼ p ( x ) for i ∈ [ m ] , and T X n consists of i.i.d.
samples x i ∼ q ( x ) for i ∈ [ n ] , respectively 7 .
Roughly speaking, the cluster-based strategy estimates the average IW in each bin B j ; assuming p and q are roughly constant in each bin, these accurately estimate the true IWs.
Let j ( x ) be the bin containing x
(i.e., x ∈ B j ( x ) ), and let p B ( x ) := p j ( x ) s.t. p
j = (cid:90)
B j p ( x (cid:48) )
d x (cid:48) and q B ( x ) := q j ( x ) s.t. q j = (cid:90)
[Equation]
be the (unnormalized) approximations of the densities p and q , respectively, that are constant on each bin.
We assume that p B and q B are accurate approximations:
Assumption 1 Given E ∈ R ≥ 0 , the partition satisﬁes
[Equation]
and
[Equation]
Thus, p and q are roughly constant on the partitions.
In general, (12) can hold for any E ∈ R > 0 if p and q are Lipschitz continuous and each B j is sufﬁciently small (see Appendix B.3 for discussion).
Then, under Assumption 1, it can be veriﬁed that
[Equation]
and
[Equation]
where v ( x ) = v j ( x ) , and v j = (cid:82) B j d x (cid:48) is the volume of bin B j (see the proof of Theorem 5 for the validity of (13)).
Next, we have the following empirical estimates of p B and q B , respectively:
[Equation]
Now, 1 ( x (cid:48) ∈ B j ) has distribution Bernoulli ( p j ) when x ∼ P , thus m · ˆ p j has distribution
Binom ( m, p j ) , and p j is contained in a Clopper-Pearson interval around ˆ p j with high probability; in particular, let θ be the Clopper-Pearson lower bound corresponding to the Clopper-Pearson upper bound deﬁned in Section 2.2, i.e., P k ∼ Binom ( m,µ ) [ µ ≥ θ ( k ; m, δ )] ≥ 1 − δ .
Then, we have
[Equation]
[Equation]
with probability at least 1 − δ (cid:48) with respect to the samples S Xm .
Combining (13) and (16), we have the following result (and see Appendix D.6 for a proof):
Theorem 5 Letting δ (cid:48)
=
δ w / (2 K ) and [ v ] + := max { 0 , v } for all v ∈ R , we have
[Equation]
with probability at least 1 − δ w over S Xm and T Xn .
We use these upper and lower bounds on the IWs as the inputs w and w to our algorithm—i.e.,
[Equation] and use b = max x ∈X w ( x ) = max j ∈ [ K ] w j as the maximum IW.
If b is known, we need importance weights associated with source calibration samples S Xm ; thus we use a simpler form of W as follows:
[Equation]
where w i := w ( x i ) and w i := w ( x i ) for x i ∈ S Xm .
Theorem 5 under Assumption 1 is one way to construct uncertainty set W that contains the true IWs.
Corollary 1 Given K ∈ N , E ∈ R ≥ 0 , and δ w ∈ (0 , 1) , suppose Assumption 1 is satisﬁed and W is constructed using Theorem 5.
Then, we have
[Equation]
B. P ARTITION CONSTRUCTION HEURISTIC
In general, exponentially many bins are needed to guarantee Assumption 1.
Instead, we consider an intuitive heuristic for constructing these bins, so that the importance weights w ( x ) —rather than the density functions p ( x ) and q ( x ) individually—are roughly constant on each bin, inspired by (Park et al., 2021; 2020b); a standard heuristic for estimating IWs is to train a probabilistic classiﬁer g ( s | x ) to distinguish source and target training examples, and then use these probabilities to construct the IWs.
In particular, deﬁne the distribution
[Equation]
Then, letting g ∗ ( y | x ) be the conditional distribution, we have w ∗ ( x ) = 1 /g ∗ ( s = 1 | x ) − 1 (Bickel et al., 2007).
Thus, we train g ( s | x ) ≈ g ∗ ( s | x ) and construct bins according to w ( x )
= 1 /g ( s = 1 | x ) − 1 , i.e.,
[Equation]
= [Equation] [Equation] where 0 = w 1 ≤ w 2 ≤ ... ≤ w K
+1 = ∞ .
Finally, we describe how to train g .
Let S X m (cid:48) and T X n (cid:48) be unlabeled training examples from a source and target, respectively; then, the set
[Equation]
consists of i.i.d.
samples ( x, y ) ∼ s ∗ ( x, y ) .
Thus, we can train s on R Xm (cid:48) ,n (cid:48) using supervised learning.
In practice, the corresponding IW estimates w ( x ) can be inaccurate partly since w is likely overﬁt to R Xm (cid:48) ,n (cid:48) , which is why re-estimating the IWs in each bin according to Theorem 5 remains necessary.
B. D ENSITY E STIMATION S ATISFYING A SSUMPTION
We consider the following binning strategy that satisﬁes Assumption 1 for Lipschitz continuous
PDFs.
In particular, assume the PDFs p ( x ) and q ( x ) are L -Lipschitz continuous for some norm (cid:107)·(cid:107) .
Then, for any given E , construct each bin B j such that
[Equation]
where v j is the volume of bin B j .
Then, we know that
[Equation]
so
[Equation]
Similarly, we have
[Equation]
Thus, the bins satisfy Assumption 1.
Finally, assuming (cid:107) · (cid:107) is the L ∞ norm, we describe one way to construct bins.
In particular, construct bins by taking an ε -net with ε = ( E/L ) 1 / ( d +1) , where d is the dimension of X .
Then, we have
[Equation]
as desired.
C A DDITIONAL D ISCUSSION ON I MPORTANCE W EIGHTS C. A PPROXIMATE I MPORTANCE W EIGHTS
In Section 3.3, we assume W has the following form:
[Equation]
However, considering the fact that the expected importance weight is one, i.e., E x ∼ P [ w ∗ ( x )] = 1 , the uncertainty set W that contains the true importance weights with high probability can be further constrained as follows:
[Equation]
for some c and c .
In particular, using the Hoeffding’s inequality for example, we can estimate c and c such that w 1 , . . .
, w m can be the part of the true importance weight w ∗ that satisﬁes the mass constraint E x ∼ P [ w ∗ ( x )] = 1 .
Recall that we have to ﬁnd a maximizer in the uncertainty set W (cid:48) as in (8); however, due to the additional constraint on (cid:80) mi =1 w i in W (cid:48) , it is challenging to solve the maximization problem exactly.
C. M AXIMUM I MPORTANCE W EIGHT E STIMATION
To generalize our approach to estimate the maximum importance weight b , we redeﬁne the uncertainty set W over importance weights as follows:
[Equation]
for some w : X → R and w : X → R such that it contains the true importance weight w ∗ with high probability—i.e.,
[Equation]
Given this, the maximum importance weight is obtained as follows:
[Equation]
Considering that W can be estimated using binning as in Appendix B, the maximum importance weight is rewritten as follows:
[Equation]
where
[Equation]
Here, θ , θ , m, n, δ (cid:48) , E, ˆ p j , and ˆ q j are deﬁned in Theorem 5 and
Appendix B.
The guarantee (18) implies the guarantee (7).
Thus, for the maximization over the uncertainty set in (8), we use the original deﬁnition W and the greedy algorithm for ˆ w in (9).
Then, Theorem 4 with the estimated b using (19) still holds as follows:
Theorem 6 Suppose Assumption 1 is satisﬁed and W is estimated using Theorem 5.
Deﬁne ˆ w τ,S Xm ,T Xn as in (9) and ˆ b S Xm ,T Xn as in (19), making the dependency on τ , S Xm , and T Xn explicit, and U RSCP as in (5).
Let ˆ τ be the solution of the following problem:
[Equation]
Then, we have P S m ∼ P m ,V ∼ U m ,T Xn ∼ Q nX [ L Q ( C τ [Equation] ≤ [Equation]
] ≥
[Equation] − [Equation] for any τ ≤ ˆ τ
[Equation]
C. C HOOSING H YPERPARAMETERS
In general, there is no systematic way to choose the smoothness parameter E and the number of bins
K ; we brieﬂy discuss strategies for doing so.
Number of bins K .
The bins are deﬁned in one dimensional space as described in Appendix B.2, so we follow the standard practice in the calibration literature for binning (Guo et al., 2017; Park et al., 2021), where K is between 10 and 20 .
As we are using equal-mass binning, we choose the number of bins so that each bin contains sufﬁciently many source examples (in our case, 5000 examples) for the length of the Clopper-Pearson interval over IWs of each bin to be below some threshold (in our case, 10 − 3 ), which leads us to K = 10 .
We provide a sensitivity analysis in Appendix G.6.
Smoothness parameter E .
Estimating E (i.e., computing the integral of the difference of source and target probabilities) is intractable in general as it requires that we perform density estimation in a high-dimensional space (i.e., 2048 in our case), and then integrate this density over each bin.
To avoid this hyperparameter selection, we choose E equal to zero or close to zero (in our case, 0 . 001 ).
Intuitively, we ﬁnd that binning based on the source-discriminator scores is an effective way to group examples with similar IWs; thus, the main contribution to the uncertainty is the error of the point estimate.
We provide a sensitivity analysis on E in Appendix G.5.
Importantly, note that
PS-W even with E = 0 satisﬁes PAC criterion.
One direction for future work is devising better strategies for choosing E .
C. C OMPARISON WITH WSCI
Guarantee.
The main difference between WSCI and PS-W lies in the guarantee provided (rather than prediction set size); PS-W provides a stronger guarantee than WSCI.
In particular, WSCI does not provide a PAC guarantee.
Instead, to satisfy the coverage probability guarantee, it requires a new calibration set for every new test example.
However, in practice, we usually have a single held-out calibration set.
Thus, our approach PS-W provides a guarantee that holds conditioned on this set.
This difference is illustrated in Figure 3a, which compares WSCI and PS-W given the true importance weight.
PS-W produces a larger set size than WSCI, but strictly satisﬁes the error constraint.
The shortcoming of WSCI can also be observed in Figure 2 in Tibshirani et al.
(2019), which empirically shows that the guarantee only holds on average over examples.
Usage of IWs.
WSCI requires the true IWs, whereas our method can use approximate IWs.
Also,
IWs are used differently.
Given the true importance weights, WSCI uses the importance weights to reweight examples in the calibration set, and PS-W uses the importance weights to generate a target labeled calibration set using rejection sampling.
D P ROOFS D. P ROOF OF T HEOREM
First, note that the constraint in (1) implies F (¯ L S m ( C τ ); m, ε ) ≤ δ ; conversely, any value of τ satisfying F (¯ L S
m
( C τ ); m, ε ) ≤ δ also satisﬁes ¯ L S m ( C τ )
≤ k ( m, ε, δ ) .
Thus, we can rewrite (1) as [Equation]
to F [Equation]
On the other hand, if τ satisﬁes (3), then by deﬁnition of U CP ( C τ , S m , δ ) = θ ( k ; m, δ ) , we have [Equation] which implies that F (¯ L S m ( C τ ); m, ε ) ≤ δ (since the inﬁmum is obtained within the set since the binomial CDF F is continuous in ε ).
Conversely, any value of τ satisfying F (¯ L S
m
( C τ ); m, ε ) ≤ δ also satisﬁes U CP ( C τ , S m , δ ) ≤ ε .
Thus, we can rewrite (3) as
[Equation]
implying (1) and (3) are equal.
Thus, the claim follows from Theorem 1 of (Park et al., 2020a) and the fact that the error L P ( C τ ) is monotonically increasing in τ .
(cid:3)
D. M ONOTONICITY OF THE C LOPPER -P EARSON B OUND
The CP bound U CP enjoys certain monotonicity properties that we will need.
Intuitively, the CDF decreases as the number of observations m increases while holding the number of successes k ﬁxed, but increases if both m and k are increased by the same amount (i.e., holding the number of failures m − k ﬁxed).
In particular, we have the following:
Lemma
We have θ ( k ; m − 1 , δ ) ≥ θ ( k ; m, δ ) and θ ( k − 1; m − 1 , δ ) ≤ θ ( k ; m, δ ) .
Proof.
Recall that F ( k ; m, θ ) is the cumulative distribution function of a binomial distribution
Binom ( m, θ ) , or equivalently of the random variable (cid:80) mi =1 X i , where X i ∼ Bernoulli ( θ ) are i.i.d.
Decreasing case.
If k ≤ m − 1 , then we have m (cid:88) i =
X i ≤ k ⇒ m − 1 (cid:88) i =
X i ≤ k, hence
P (cid:34) m (cid:88) i = X i ≤ k (cid:35) ⊆ P (cid:34) m − 1 (cid:88) i =
X i ≤ k (cid:35) , so F ( k ; m, θ ) ≤ F ( k ; m − 1 , θ ) .
Then, we have θ ( k ; m, δ ) :
= inf { θ ∈ [0 , 1] | F ( k ; m, θ ) ≤ δ } ∪ { 1 } ≤ inf { θ ∈ [0 , 1] | F ( k ; m − 1 , θ ) ≤ δ } ∪ { 1 } =: θ ( k ; m − 1 , δ ) , thus θ is monotonically non-increasing in m .
Increasing case.
We have m − 1 (cid:88) i =
X i ≤ k − 1 ⇒ m (cid:88) i = X i ≤ k, hence P (cid:34) m − 1 (cid:88) i = X i ≤ k −(cid:35) ⊆
P (cid:34) m (cid:88) i =
X i ≤ k (cid:35) , so F ( k − 1; m − 1 , θ ) ≤ F ( k ; m, θ ) .
Then, we have
[Equation]
thus θ is monotonically jointly non-decreasing in ( m, k ) .
[Equation]
D. P ROOF OF T HEOREM
The rejection sampling prediction set consists of two steps: (i) generate target samples, using source samples S m , importance weights w , and an upper bound on their maximum value b , and (ii) construct the Clopper-Pearson prediction set using the generated target samples.
From rejection sampling, we choose N := (cid:80) mi
=1 σ i samples from S m , denoting them by T N ; here,
N ∼ Binom ( m, 1 /b ) , and 1 /b is the acceptance probability (von Neumann, 1951)—i.e.,
[Equation]
where V (cid:48) ∼ Uniform ([0 , 1]) .
The samples in T N are independent and identically distributed, conditionally on the random number N of samples being equal to any ﬁxed value n .
The reason is that one can view the rejection sampling algorithm proceeding in stages, iterating through the samples one by one.
The ﬁrst stage starts at the very beginning, and then each stage ends when a datapoint is accepted, followed by starting a new stage at the next datapoint.
The last stage ends at the last datapoint.
Based only on the source samples observed in one stage, rejection sampling produces a sample from the target distribution.
Thus, within each stage, we produce one sample from the target distribution, and because each stage is independent of all the other ones, conditionally on any number of stages reached, our produced target samples are iid.
Thus, we can use the Clopper-Pearson bound conditionally on each N = n .
To this end, let ˆ τ ( S m , V )
=
ˆ τ to explicitly denote the dependence on S m and V , and let
[Equation]
Note that conditioned on obtaining n samples using rejection sampling (i.e., | T n ( S m , V, w, b ) | = n ), we have ˆ τ ( S m , V ) D =
˜
τ
( T n ) , where D = denotes equality in distribution.
Then, we have
[Equation]
where the inequality follows by Theorem 1.
The claim follows.
(cid:3)
D. P ROOF OF T HEOREM
First, let
[Equation]
which satisﬁes P S m ∼ P m ,V ∼ U m [ L Q ( C ˜ τ ) ≤ ε ] ≥ 1 − δ C by Theorem 2.
Now, with probability at least 1 − δ w , we have (cid:126)w ∗ ∈ W .
Under this event, we have
[Equation]
so ˆ τ satisﬁes the constraint in (21).
Thus, we must have ˆ τ ≤ ˜ τ .
By monotonicity of L Q ( C τ ) in τ , we have L Q ( C ˆ τ ) ≤ L Q ( C ˜ τ ) , which implies that
[Equation]
where the last step follows by Theorem 2.
The claim follows by a union bound, since (cid:126)w ∗ ∈ W with probability at least 1 − δ w .
(cid:3)
D. P ROOF OF L EMMA
Let w and v be IWs where w ( x i ) ≥ v ( x i ) and w ( x j )
= v ( x j ) for j (cid:54) =
i .
Additionally, we use the following shorthands:
[Equation]
(cid:18) V i ≤ v ( x i ) b (cid:19) ,
T n v := (cid:26) ( x i , y i )
∈
S m (cid:12)(cid:12)(cid:12)(cid:12) V i ≤ v ( x i ) b (cid:27) , and k v := (cid:88) ( x,y ) ∈
T nv 1 ( y / ∈ C ( x )) .
Here, n w ≥ n v since w ( x i ) ≥ v ( x i ) .
Finally, recall that F ( k ; m, θ ) be the cumulative distribution function of a binomial random variable (cid:80) mi =1 X i , where X i ∼ Bern
( θ ) .
Non-decreasing case.
If y i / ∈ C ( x i ) , there are two cases to consider:
If v ( x i ) b < V i ≤ w ( x i ) b , then we can verify that n w = n v + 1 and k w = k v + 1 . 2.
Otherwise, we can verify that n w = n v and k w = k v .
In both cases, k w ≥ n v and n w ≥ n v .
Since θ is monotonically jointly non-decreasing in ( m, k ) as in Lemma 2, we have
[Equation]
thus U RSCP is monotonically non-decreasing in w ( x i ) .
Non-increasing case.
If y i ∈ C ( x i ) , then k w = k v .
Since θ is monotonically non-increasing in m as in Lemma 2, we have
[Equation]
thus U RSCP is monotonically non-increasing in w ( x i ) .
D. P ROOF OF T HEOREM
Recall that
[Equation]
Due to the assumption of (12)
[Equation]
| v ( x ) · p ( x ) − p K ( x ) | is bounded for any x ∈ B j as follows:
[Equation]
[Equation]
Similarly,
[Equation]
Observe that m ˆ p ( x ) ∼ Binom (cid:16) m, (cid:82) B j p ( x (cid:48) )
d x (cid:48) (cid:17) for any x ∈ B j ; thus p K is bounded with probability at least 1 − δ (cid:48) as follows due to the Clopper-Pearson interval ( θ, θ ) :
[Equation]
Similarly,
[Equation]
From (22), (23), (24), and (25), the following holds:
[Equation]
Therefore, for any x ∈ B j , w ∗ ( x ) is bounded as follows:
[Equation]
Since we apply the Clopper-Pearson interval for K partitions for both source and target, the claim holds due to the union bound.
Algorithm 2 PS: an algorithm using the CP bound in (3) procedure PS( S m , f, T , ε, δ ) ˆ τ ←for τ ∈ T do ( (cid:46) )
Grid search in ascending order if U CP ( C τ , S m , δ ) ≤ ε then ˆ τ ← max(ˆ τ, τ ) elsebreak return ˆ τ
procedure PS-C( S m , f, T , b, ε, δ ) return PS( S m , f, T , ε/b, δ )
We describe the PS-C algorithm, which uses a conservative upper bound on the CP interval.
Let
[Equation]
Then, we have
[Equation]
Thus, L Q ( C ) ≤ ε if b · L P ( C ) ≤ ε .
Equivalently, L Q ( C ) ≤ ε if L P ( C ) ≤ ε/b .
As a consequence, we can choose C based on the CP bound for the i.i.d.
case (i.e., Algorithm 2), except using the desired error of ε/b (instead of ε ).
The algorithm is described in Algorithm 3.
E
PS-R A LGORITHM
procedure PS-R( S m , f, T , w, b, ε, δ )
V ∼ Uniform ([0 , 1]) m ˆ τ ←for τ ∈ T do ( (cid:46) )
Grid search in ascending order if U RSCP ( C τ , S m , V, w, b, δ ) ≤ ε then ˆ τ ← max(ˆ τ, τ ) elsebreak return ˆ τ
E. PS-M A LGORITHM procedure PS-M( S m , T Xn , f, T , w, b, ε, δ ) ˆ w ( x ) ← ˆ q B ( x ) ˆ p B ( x ) for x ∈ X , where ˆ p B and ˆ q B are deﬁned in (14) and (15), respectively return PS-R( S m , f, T , ˆ w, b, ε, δ ) F E XPERIMENT D ETAILS F. D OMAIN A DAPTATION
We use a fully-connected network (with two hidden layers, where each layers has 500 neurons followed by ReLU activations and a 0 . 5 -dropout layer) as the domain classiﬁer (recall that the input of this domain classiﬁer is the last hidden layer of ResNet101).
We use the last hidden layer of the model as example space X , where its dimension is 2048.
For neural network training, we run stochastic gradient descent (SGD) for 100 epochs with an initial learning rate of 0.1, decaying it by half once every 20 epochs.
The domain adaptation regularizer is gradually increased as in (Ganin et al., 2016).
We use the same hyperparameters for all experiments.
F. D OMAIN N ET
We split the dataset into 409,832 training, 88,371 calibration, and 88,372 test images.
F. I
MAGE N ET C
[Equation]
We split ImageNet into 1.2M training, 25K calibration, and 25K test images, and ImageNet-C into 83M training, 1.6M calibration, and 1.6M test images.
To train a model using domain adaptation, due to the large size of the target training set, we subsample the target training set to be the same size as the source training set on for each random trials.
G A DDITIONAL R ESULTS G. S YNTHETIC R ATE S HIFT BY T WO G AUSSIANS
We demonstrate the efﬁcacy of the proposed approaches (i.e., PS-R with the known IWs and PS-W with the estimated IWs) using a synthetic dataset consisting of samples from two Gaussian distributions.
Dataset.
We consider two Gaussian distributions N ( µ, Σ) and N ( µ, Σ (cid:48) ) over 2048 -dimensional covariate space X .
Here, µ = 0 ; Σ and Σ (cid:48) are diagonal where Σ 1 , 1 = 5 2 ,
Σ i,i = 10 − 1 , Σ (cid:48) 1 , 1 = 1 , [Figure] and Σ (cid:48)
i,i = 10 − 1 for i ∈ { 2 , . .
We consider the “ﬂat” Gaussian N ( µ, Σ) as the source and the “tall” Gaussian N ( µ, Σ (cid:48) ) as the target.
Intuitively, there is a rate shift from the source to the target—i.e., the target examples are a subset of the source, but occur with higher frequencies.
We use the following labeling function: p ( y | x ) =
σ (5 x 1 ) , where σ is the sigmoid function.
Finally, we generate 50,000 labeled examples for each training, calibration, and test.
Results.
We consider two different setups: 1) the true IW is known, and 2) the true IW is unknown.
In Figure 3a, we demonstrate the prediction set errors given the true IW.
As expected PS-R satisﬁes the PAC guarantee—i.e., the error is below ε .
However, as shown in Figure 3b, when we need to estimate IWs, using just the point-estimate of the IW results in PS-R performing poorly in terms of prediction set error; it still satisﬁes the ε constraint, but the error is close to zero, indicating that the prediction set size is too large to be useful for an uncertainty quantiﬁer.
In contrast, PS-W (i.e., rejection sampling based on interval estimates of IWs) produces a larger, more reasonable error rate while still satisfying the PAC condition.
These experiments demonstrate that PS-R works well when given the true IW, but accounting for IW uncertainty is important when using estimated IWs.
[Figure]
[Figure]
[Figure]
n = ∞ ) and with E = 0 .
As can be seen in the results on the shift to various domains, our approach satisﬁes the
PAC guarantee but this version does not—in fact, its error is even worse than PS-R, which uses the non-rescaled importance weights from the probabilistic classiﬁer s .
G. C OMPARISON WITH V ARIOUS T OP - K P REDICTION S ETS
[Figure]
[Figure]
z
[Figure] G. V ARYING A S MOOTHNESS P ARAMETER
[Figure]
a hyperparameter.
In this ﬁgure, we show the error of PS-W as a function of E .
As E becomes smaller, the prediction sets become more optimistic while still satisfying the PAC guarantee.
Note that the optimal case is E = 0 , since PS-W still satisﬁes the PAC guarantee; this result suggests that our IW estimates are reasonably accurate.
G.
ARYING A N UMBER B INS
[Figure]
[Figure]
[Figure]
[Figure]
G. P REDICTION S ET V ISUALIZATION
[Table]
 
[Table] [Table]
 
[Table]

[Table]

[Table]
[Table]
[Table]
 
[Table]
 
[Table]
, ocean , star , tent  
[Table]
[Table]
 
[Table]
[Table]
[Table]
, teapot   (cid:8)
[Table]
ocean (cid:9) 
 hurricane , (cid:91) ocean , square , tornado
 
[Table]
brain [Table]  
[Table]
brain , ﬁsh , lion , lollipop , sea turtle   (cid:110)
[Table]
penguin (cid:111)   ﬁrehydrant , foot , (cid:92) penguin , telephone   (cid:26) hat , [Table] hot tub (cid:27) 
[Table]

[Table]
[Figure]
 
[Figure]

[Figure]
 
[Figure]
 
[Figure]
 
[Figure]

[Table]
, (cid:92) wallclock   (cid:26) chameleon , (cid:92) green lizard (cid:27)   chameleon , (cid:92) greenlizard , greensnake , walingstick , mantis     (cid:92) barbell , dumbbell , lens cap , puck     (cid:92) barbell , barometer , carwheel , dumbbell , lenscap , powerdrill , puck , stethoscope   (cid:26) (cid:92) Pembroke ,
Cardigan (cid:27) (cid:26) (cid:92) Pembroke ,
Cardigan (cid:27)  
[Figure]
, (cid:92) lipstick , paintbrush  
 
[Figure]
(cid:92)
[Figure]
[Figure]
[Table]
[Figure]
footballhelmet  