JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Abstract
We propose JAWS , a series of wrapper methods for distribution-free uncertainty quantiﬁcation tasks under covariate shift, centered on the core method JAW , the JA ckknife+ W eighted with data-dependent likelihood-ratio weights.
JAWS also includes computationally efﬁcient A pproximations of JAW using higherorder inﬂuence functions: JAWA .
Theoretically, we show that JAW relaxes the jackknife+’s assumption of data exchangeability to achieve the same ﬁnite-sample coverage guarantee even under covariate shift.
JAWA further approaches the JAW guarantee in the limit of the sample size or the inﬂuence function order under common regularity assumptions.
Moreover, we propose a general approach to repurposing predictive interval-generating methods and their guarantees to the reverse task: estimating the probability that a prediction is erroneous, based on user-speciﬁed error criteria such as a safe or acceptable tolerance threshold around the true label.
We then propose JAW-E and JAWA-E as the repurposed proposed methods for this E rror assessment task.
Practically, JAWS outperform state-ofthe-art predictive inference baselines in a variety of biased real world data sets for interval-generation and error-assessment predictive uncertainty auditing tasks.
Introduction
Auditing the uncertainty under data shift Principled quantiﬁcation of predictive uncertainty is crucial for enabling users to calibrate how much they should or should not trust a given prediction [Thiebes et al., 2021, Ghosh et al., 2021, Tomsett et al., 2020, Bhatt et al., 2021].
Uncertainty-based predictor auditing can be considered a type of uncertainty quantiﬁcation performed post-hoc , for example by a regulator without detailed knowledge of a predictor’s architecture and with limited resources
[Schulam and Saria, 2019].
Data shift poses a major challenge to uncertainty quantiﬁcation due to violation of the common assumption that the training and test data are exchangeable, or more speciﬁcally independent and identically distributed (i.i.d.)
[Ovadia et al., 2019, Ulmer et al., 2020,
Zhou and Levine, 2021, Chan et al., 2020].
Therefore, it is essential to develop convenient tools for users or regulators to audit the uncertainty of a given prediction even when training data is biased.
Predictor auditing: Interval generation
In this work we distinguish between two types of predictive uncertainty auditing.
We describe the ﬁrst type as interval generation , which refers to a common goal in the distribution-free uncertainty quantiﬁcation literature: to generate a predictive conﬁdence interval (or set) that covers the true label with at least a user-speciﬁed probability.
For instance, an auditor might ask for predictive intervals that contain the true label with at least, say 90% frequency.
Predictor auditing: Error assessment While predictive interval generation has been a central focus of the distribution-free uncertainty quantiﬁcation literature [Angelopoulos and Bates, 2021], in some applications the reverse computation may be more actionable: estimating the probability that a prediction is erroneous or not, based on user-speciﬁed error critieria such as a safe or acceptable tolerance region around the true label.
We thus refer to this task as error assessment .
For instance, take the setting of chemical or radiation therapy dose prediction for cancer treatment, where administering a dose within approximately ± 10% of the optimal dose is considered safety-critical (see Appendix
A.1 for details).
Whereas predictive interval generation could fail to provide safety assurance (e.g., if the predictive conﬁdence interval is larger than the safe tolerance region), error assessment would give a worst-case probability of the prediction being safe.
Similar examples could be formulated in other applications, such as incision planning in surgical robotics and autonomous vehicle navigation.
Coverage We assume a standard regression setup with a multiset of training data { ( X 1 , Y 1 ) , ..., ( X n , Y n ) } and a test point ( X n +1 , Y n +1 ) with unknown label Y n +1 , where ( X i , Y i )
R d ⇥ R for all
i 2 { 1 , ..., n + 1 } .
Also, we denote a predictor as
[Equation]
where A is a model-ﬁtting algorithm.
For a predictive interval (or set)
[Equation]
R
[Equation]
{ subsets of R } , a coverage guarantee gives a lower bound to the probability that the interval covers the true test label:
[Equation]
The coverage guarantee provides the basis for both interval-generation and error-assessment auditing, though it is important to note that in this work we focus on marginal rather than conditional coverage (see [Foygel Barber et al., 2021] for more details on this distinction).
Standard conformal prediction methods [Vovk et al., 2005, Shafer and Vovk, 2008, Vovk, 2013] along with the jackknife+ and related methods [Barber et al., 2021], which we refer to together as “predictive inference” methods, provide a framework for generating predictive intervals with ﬁnite-sample guaranteed coverage.
Exchangeability Standard conformal prediction and the jackknife+ rely on two crucial notions of exchangeability : data exchangeability, that is that the training and test data are all exchangeable (e.g., i.i.d.); and secondly that the model-ﬁtting algorithm A treats the data symmetrically [Barber et al., 2022].
In common situations of dataset shift, however, the data exchangeability assumption is violated.
Empirically, the coverage performance of standard conformal prediction methods can suffer under data shift [Tibshirani et al., 2019, Podkopaev and Ramdas, 2021].
In this work, we build on the jackknife+ method due to its beneﬁcial compromise between the statistical and computational limitations of other conformal prediction methods
However, jackknife+ coverage performance can still degrade under data shift, such as shown in Figure 1, and in some applications its computational requirements can still be limiting.
To address these concerns and make extensions to error assessment, we develop JAWS, a series of wrapper methods for distribution-free uncertainty quantiﬁcation under covariate shift (see Table 1 for key properties).
[Table]
Our contributions can be summarized as follows :
We develop JAW: a jackknife+ method with data-dependent likelihood-ratio weights for predictive interval generation under covariate shift.
We show that JAW achieves the same rigorous, ﬁnite-sample coverage guarantee as jackknife+ [Barber et al., 2021] while relaxing the data exchangeability assumption to allow for covariate shift.
We develop JAWA: a sequence of computationally efﬁcient approximations to JAW that uses higher-order inﬂuence functions to avoid retraining.
Under assumptions outlined in Giordano et al.
[2019a] regarding the regularity of the data, Hessian of the objective (local strong convexity), and the existence and boundedness of higher order derivatives, we provide an asymptotic guarantee for the JAWA coverage in the limit of the sample size or inﬂuence function order.
We propose a general approach to repurposing any distribution-free predictive inference method to the error assessment task, with rigorous guarantees for the coverage probability estimation.
Our approach applies to methods that assume exchangeable data and to methods like JAW and JAWA that allow for covariate shift—JAW-E and JAWA-E refer to the error assessment versions.
We demonstrate superior empirical performance of JAWS over other distribution-free predictive inference baselines on a variety of benchmark datasets under covariate shift.
Background and related work 2.1 Standard conformal prediction
Conformal prediction has grown into a broad research ﬁeld since arising in the 1990s [Vovk et al., 2005, Shafer and Vovk, 2008, Balasubramanian et al., 2014, Angelopoulos and Bates, 2021].
Standard conformal prediction methods generate a prediction interval (or set) with a ﬁnite-sample coverage guarantee as in (1), which is distribution-free in the sense that the guarantee applies to any exchangeable data distribution [Lei and Wasserman, 2014, Lei et al., 2018].
With the exchangeability assumptions in Section 1, standard conformal prediction methods rely on a pre-ﬁt score function b
S : R d ⇥ R !
R (in regression, the absolute-value residual score b
S
[Equation]
x
[Equation]
µ
[Equation]
x ) | is commonly used).
A conformal prediction interval at conﬁdence level 1  ↵ is then determined by a corresponding quantile on a multiset of (exchangeable) score values.
Split conformal and full conformal are two main types of standard conformal prediction, and each bears its own limitation [Vovk et al., 2005, Shafer and Vovk, 2008].
Split conformal generates scores on labeled holdout data and is computationally efﬁcient due to not requiring retraining, but sample splitting to obtain the holdout set can reduce model accuracy [Papadopoulos, 2008, Lei et al., 2018,
Vovk, 2012].
On the other hand, full conformal prediction avoids the holdout set requirement, but at the heavy computational cost of retraining the model on every possible target value (or, in practice, on a ﬁne grid of target values)
[Ndiaye and Takeuchi, 2019, Zeni et al., 2020].
2.2 Covariate shift
Under the covariate shift assumption, the Y | X distribution is assumed to be the same between training and test data but the marginal X distributions may change [Sugiyama et al., 2007, Shimodaira, 2000]:
[Equation]
Rich literature exist in this domain—see Appendix A.2 for more details.
Uncertainty quantiﬁcation is relatively less explored under covariate shift, though recent work [Ovadia et al., 2019, Zhou and
Levine, 2021, Chan et al., 2020] emphasizes its importance, especially in deep learning.
2.3 Conformal prediction under covariate shift and beyond exchangeability
Tibshirani et al.
[2019] develop the idea of weighted exchangeability for adapting conformal prediction to the covariate shift setting.
Random variables V 1 , ..., V n are weighted exchangeable with weight functions w 1 , ..., w n if their joint density f can be factorized as f ( v 1 , ..., v n ) =
Q ni =
w i ( v i ) · g ( v 1 , ..., v n ) , where g is independent of ordering on its inputs.
For covariate shift as in (2), if e
P X is absolutely continuous with respect to P X , then the data { ( X i , Y i ) } are weighted exchangeable with weight functions w 1 = ...
= w n = 1 and w n +1 = w = d e
P X / d P X
[Tibshirani et al., 2019].
If { v i } represents a set of scores for standard conformal prediction, then we can represent the empirical distribution of { v i } as
[Equation]
1 , where  v i denotes a point mass at v
i [Barber et al., 2022].
By extension, weighted conformal prediction uses the weighted empirical distribution deﬁned as
[Equation]
, with weights given by
[Equation]
and
[Equation]
where w = d e
P X / d P X , so p wi ( X n +1 ) can be thought of as a normalized likelihood ratio weight for each i 2 { 1 , ..., n + 1 } .
Corollary 1 in [Tibshirani et al., 2019] provides the coverage guarantee of weighted conformal prediction that takes the form of (1) but relaxes the exchangeable data assumption to allow for covariate shift.
However, weighted split and weighted full conformal inherit the same statistical and computational limitations, respectively, from their standard (exchangeable) variants.
The recent work of [Barber et al., 2022] provides a novel extension of conformal prediction and the jackknife+ to unknown violations of the exchangeability assumption, including a “nonexchangeable jackknife+” deﬁned with ﬁxed weights.
The key difference between the nonexchangeable jackknife+ in Barber et al.
[2022] and our proposed JAW method is that Barber et al.
[2022] use ﬁxed weights to compensate for unknown exchangeability violations (not limited to covariate shift) but at the expense of a bounded but generally nonzero “coverage gap” (drop in guaranteed coverage relative to if the data were exchangeable), whereas our JAW method with data-dependent weights assumes covariate shift but does not suffer from any similar coverage gap.
See Appendix A.3 for more details.
2.4 Jackknife+
The jackknife+ [Barber et al., 2021], which is closely related to cross conformal prediction [Vovk et al., 2018], offers a compromise between the statistical limitation of split conformal and the computational limitation of full conformal, at the cost of a slightly weaker coverage guarantee.
The jackknife+ predictive interval can most easily be understood as a modiﬁcation to a predictive interval from the classic jackknife resampling method [Miller, 1974, Steinberger and Leeb, 2018, 2016].
For a set of point masses {  v i } at values v 1 , ..., v n , let Q   { 1 n +1  v i } denote the level  quantile on the empirical distribution
P ni =1 1 n +1  v i + 1 n +1  1 and let Q +  { 1 n +1  v i } denote the level  quantile on the empirical distribution
P ni =1 1 n +1  v i + 1 n +1  1 .
Then, denoting the model trained without the i th point as
[Equation]
and the leave-one-out residual R LOOi =
| Y i  b µ  i
[Equation]
X i ) | , the jackknife prediction interval can be written as
[Equation]
In contrast, we obtain the jackknife+ predictive interval in Barber et al.
[2021] by replacing the full model prediction b
[Equation]
X n +1 ) in (4) with
[Equation]
[Barber et al., 2021] prove that, with the same exchangeability assumptions as in standard conformal prediction, the jackknife+ prediction interval satisﬁes
[Equation]
2.5 Approximating leave-one-out models with higher-order inﬂuence functions
Inﬂuence functions (IFs)
[Cook, 1977] have a long history in robust statistics for estimating the dependence of parameters on sample data.
Recently, IFs have become more widespread in machine learning for uses including model interpretability Koh and Liang [2017] and approximating classic resampling-based uncertainty quantiﬁcation methods including bootstrap [Schulam and Saria, 2019], jackknife, and leave- k -out cross validation [Giordano et al., 2019b,a].
In each of these cases, IFs enable approximation of the parameters that would be obtained if the model were retrained on resampled data by instead estimating the effect of a corresponding reweighting.
In prior work,
Alaa and Van Der Schaar [2020] proposed approximating the leave-one-out models required by the jackknife+ with higher-order IFs, but their work assumes exchangeable or i.i.d.
train and test data.
Let ˆ ✓ denote the ﬁtted parameters for predictor b µ trained on the full training data.
Given Assumptions 1-4 in Giordano et al.
[2019a]—which require that ˆ ✓ is a local minimum of the objective function, that the objective is k + 1 times continuously differentiable with bounded norms, and that the objective is strongly convex in the neighborhood of ˆ ✓ —then the k -th order leave-one-out IF refers to the k -th order directional derivative of the model parameters ˆ ✓ with respect to the data weights, in the direction of the leave-one-out change in weights (See A.4 for more details).
With each of these k th order leave-one-out IFs for k 2 { 1 , ..., K } , denoted with condensed notation  k  i ˆ ✓ , we can construct a
K -th order Taylor series approximation to estimated the leave-one-out model parameters ˆ ✓  i :
[Equation]
In this work we implement the algorithm proposed by Giordano et al.
[2019a] to compute higher-order
IFs, a recursive procedure based on foreward-mode automatic differentiation [Maclaurin et al., 2015] for memory efﬁciency in computing higher-order directional derivatives.
Our introduction of IFs is highly simpliﬁed—we refer to Appendix A.4 and to Giordano et al.
[2019a] for more details.
2.6 Error assessment
Whereas conformal prediction and related methods generate prediction intervals that control the error probability (miscoverage level ↵ ) at a user-speciﬁed level, we refer to the reverse task as error assessment : estimating the probability that a prediction is erroneous or not, based on user-speciﬁed error criteria.
For instance, a user might deﬁne an error as any deviation between the prediction b µ ( X n +1 ) and the true label Y n +1 greater than some acceptable tolerance threshold ⌧ : that is, when | Y n +1  b µ ( X n +1 ) | > ⌧ .
In Section 3.3, we present a general approach to repurposing predictive inference methods with validity under covariate shift to error assessment.
We note that for score functions that are monotonic in y , such as b
S ( x, y ) =
y  b µ ( x ) , guarantees for this error assessment task can be obtained using conformal predictive distributions as described by
Vovk et al.
[2017] (also see Vovk et al.
[2020], Vovk and Bendtsen [2018], Xie and Zheng [2022]).
In regression tasks assuming exchangeable data, CPDs generate a probability distribution for the label over R .
However, CPDs require that score functions be monotonic in y , whereas we allow for certain non-monotone conformity scores such as the commonly used absolute-value residual | y  b µ ( x ) | ; Moreover, CPDs assume exchangeable data, whereas our approach extends to covariate shift.
Proposed approach and theoretical results 3.1 JAW: Jackknife+ weighted with data-dependent weights
We present JAW , the JA ckknife+ W eighted with data-dependent likelihood-ratio weights, deﬁned by the following predictive interval:
[Equation]
where R LOOi = b µ  i ( X i )  Y i  , with p wi ( x ) for i 2 { 1 , ..., n + 1 } as in (3), where Q  ↵  p wi
[Equation]
X n +
[Equation]
·  b µ  i ( X n +1 )  R LOOi denotes the level ↵ quantile of the empirical distribution
P
[Equation]
[Equation]
1 , and where Q +1  ↵
[Equation]
X n +
[Equation]
X
[Equation]
+ R LOOi is the level 1  ↵ quantile for
[Equation]
We choose to deﬁne JAW using likelihood-ratio weights w ( X i ) =
d e
P X ( X i ) /
d P X
( X i ) in the p wi ( x ) to address covariate shift, but a similar result holds for other instances of weighted exchangeability and corresponding data-dependent weight functions (see Appendix B.1).
We show that b
[Equation]
n, ↵ ( X n +1 ) satisﬁes the same coverage guarantee as the jackknife+ except relaxing the data exchangeability assumption to allow for covariate shift, which we state formally in the following theorem.
Theorem 1.
Assume data under covariate shift from (2) .
If e
P X is absolutely continuous with respect to P X , then the JAW interval in (8) satisﬁes
[Equation] [Equation]
Remark 1.
The results from Tibshirani et al.
[2019] do not directly imply Theorem 1.
The approach in Tibshirani et al.
[2019] relies on leveraging the weighted exchangeability of the data to reweight the nonconformity scores { V 1 , ..., V n +1 } so they can be treated as exchangeable, and for the jackknife+ this approach would entail treating
[Equation] [Equation]
±
[Equation]
LOOi as implicit nonconformity scores.
But, observe that for i
[Equation]
..., n } ,
[Equation]
µ  i is trained on n  1 datapoints, whereas b µ
[Equation]
n +
[Equation]
=
[Equation]
µ is trained on n datapoints.
Thus, no reweighting can make b µ  i equivalent in distribution to b µ and thereby allow us to treat the reweighted
[Equation]
Proof sketch:
Our proof technique for Theorem 1 extends the jackknife+ coverage guarantee proof in Barber et al.
[2021] to the covariate shift setting for JAW using likelihood ratio weights as in
Tibshirani et al.
The full proof is given in Appendix C.1, but the outline is as follows:
Setup : Following
Barber et al.
[2021], we deﬁne
a set of leave- two -out models
˜ µ
( i,j )
We
then
generalize the notion of “strange” points described in
Barber
et al.
to
covariate
shift.
Bounding the total normalized weight of strange points : We establish deterministically that the total normalized weight of strange points cannot exceed 2 ↵ .
Weighted exchangeability using the leave-two-out models : Using the leave-two-out model construction, we leverage weighted exchangeability to show that the probability that a test point n +is strange is thus bounded by 2 ↵ .
Connection to JAW :
Lastly, we show that the JAW interval can only fail to cover the test label value Y n +1 if n + 1 is a
strange point.
While JAW assumes access to oracle likelihood ratio weights, in practice this information often has to be estimated.
See Appendix D.5 for a discussion and experiments of JAW with estimated weights.
3.2 JAWA: Using higher-order inﬂuence functions to approximate JAW without retraining For computationally efﬁcient JAW A pproximations that avoid retraining n leave-one-out models,
we propose the JAWA sequence, which approximates the leave-one-out models required by JAW using higher-order inﬂuence functions.
For each training point i 2 { 1 , ..., n } , deﬁne the K -th order inﬂuence function approximation to the leave-one-out reﬁt parameters ˆ ✓  i , obtained from Algorithm 4 in Giordano et al.
[2019a], as given by equation (7), and let b µ IF- K  i be the model with with these approximated parameters ˆ ✓ IF- K  i for each
i 2 { 1 , ..., n } .
Then, the prediction interval for the K -th order JAWA (i.e., for JAWA- K ) is given by
[Equation]
with R IF- K,LOO i =  b µ IF- K  i ( X i )  Y i  , p wi ( x ) as in (3), and quantiles deﬁned analogously to JAW.
We now provide an asymptotic coverage guarantee for b
C JAWA- K n, ↵ ( X n +1 ) that holds either in the limit of the sample size or in the limit of the inﬂuence function order, under regularity conditions formally described in Giordano et al.
[2019a].
These assumptions concern the regularity and continuity of the training data, local convexity of the objective (or that the Hessian is strongly positive deﬁnite), and the existence and boundedness of the objective’s 1st through K + 1 th order directional derivatives.
Theorem 2.
Assume data under covariate shift from (2) and that e
P X is absolutely continuous with respect to P X .
Let Assumptions 1 - 4 and either Condition 2 or Condition 4 from Giordano et al.
[2019a] hold uniformly for all n .
Then, in the limit of the training sample size n ! 1 or in the limit of the inﬂuence function order K ! 1 , the JAWA- K interval in (10) satisﬁes
[Equation]
We leave the proof to Appendix C.2, but we note that the result follows by combining Propositionsand 3 in Giordano et al.
[2019a] with the JAW coverage guarantee that we present in Theorem 1.
3.3 Error assessment under covariate shift
We now propose a general approach to repurposing predictive inference methods with validity under covariate shift from predictive interval generation to the reverse task: estimating the probability that a prediction is erroneous or not, based on user-speciﬁed error criteria.
For example, consider a user that deﬁnes a prediction b µ ( X n +1 ) as erroneous, relative to the true label Y n +1 , if it is farther than some acceptable tolerance threshold ⌧ from Y n +1 : i.e., if | Y n +1  b µ ( X n +1 ) | > ⌧ .
For this common regression error criterion, our approach to adapting a method such as JAW (8) or weighted split conformal prediction [Tibshirani et al., 2019] to error assessment reduces to ﬁrst deﬁning the set of labels that would not be considered erroneous, E =
[ b µ ( X n +1 )
⌧ , b µ ( X n +1 ) + ⌧ ] , and then ﬁnding the method’s largest predictive interval contained within E , call it b
C w-audit n, ↵ E ( X n +1 ) .
The coverage guarantee for b
C w-audit n, ↵ E ( X n +1 ) then yields a lower bound on P { Y n +1 2 E } , the probability of no error (or an upper bound on the error probability).
See Figure 2 for an illustration of this example.
Generally, a user must specify error criteria by a test point score function b
S : R d ⇥ R !
R (for conformal prediction, a nonconformity score), as well as minimum and maximum acceptable score values ⌧  and ⌧ + , where ⌧  < ⌧ + ; i.e., b µ ( X n +1 ) is considered erroneous if b
S ( X n +1 , Y n +1 ) <
⌧  or if ⌧ + < b
S ( X n +1 , Y n +1 ) .
For nonnegative b
S , we might let ⌧  = 0 .
Then, the values of y for which observing Y n [-+
-]+1 =
y would not imply [Equation] n
[Equation]
erroneous are:
[Equation]
Now, assume a predictive inference method with predictive sets that can be written in the form b
C w-audit n, ↵
[Equation] [Equation]
with valid coverage guaranteed under covariate shift.
(Note, (13) gives the JAW interval (8) by setting b
S ( x, y ) =
y  b µ ( x ) , V Li =
b µ  i ( X n + [Equation] )  b [Equation]
+ [Equation] [Equation] , and V Ui = b µ
[Equation] X n +1 )  b µ ( X n +1 )
+
R LOOi ; see Appendix B.3.
Similarly, (13) gives the prediction interval for weighted split conformal prediction [Tibshirani et al., 2019] for absolute value residual scores when b
S ( x, y ) =
| y  b µ ( x ) | , and for all calibration data i we let V Ui =
| Y
i  b µ ( X i ) | and V Li
= 0 .)
Then, deﬁning ↵ w-audit E = min ⇣n
[Equation]
(14) we can estimate the probability of b µ ( X n +1 ) not resulting in an error as in (12) as:
[Equation]
↵
[Equation]
exists
[Equation]
otherwise .
While the target coverage for b
C w-audit n, ↵ E ( X n +1 ) is used in (15), the following theorem gives the worstcase error assessment guarantee for covariate shift (proof in Appendix C.3).
Corollary 1 in Appendix
B.3 and Corollary 2 in Appendix B.4 give the error assessment guarantees for JAW-E and JAWA-E respectively.
Appendix B.2 gives the analogous guarantee for exchangeable data.
Theorem 3.
Assume a predictive inference method of the form (13) has coverage guarantee
P {
[Equation]
+
[Equation]
[Equation]
[Equation]
c
[Equation]
↵  c 2 , with c 1 , c 2 2 R , under covariate shift (2) where e
P X is absolutely continuous with respect to P X .
Deﬁne E as in (12) and ↵ w-audit E as in (14) .
Then,
[Equation]
if ↵ w
[Equation]
E exists and ↵ w-audit E < 1  c 2 c otherwise .
[Equation]
4.1 Datasets and creation of covariate shift
We conduct experiments on ﬁve UCI datasets Dua and Graff [2017] with various dimensionality (Table 2): airfoil self-noise, red wine quality prediction [Cortez et al., 2009], wave energy converters, superconductivity [Hamidieh, 2018], and communities and crime [Redmond and Baveja, 2002].
[Table]
We use exponential tilting to induce covariate shift on the test data, based on the approach used in
Tibshirani et al.
We ﬁrst randomly sample 200 points for the training data, and then sample the biased test data from the remaining datapoints that are not used for training with probabilities proportional to exponential tilting weights.
See Appendix D.1 for additional details
[Table]
4.2 Baselines
Baselines for comparison to JAW We compared JAW to the following baselines:
Naive estimates are based on training data residuals | Y i  b µ ( X i ) | , which suffers from overﬁtting.
Jackknife uses the classic Jackknife resampling as in (4).
Jackknife+ follows (5), which replaces the prediction b µ ( X n +1 ) in jackknife with b µ  i ( X n +1 ) .
Jackknife-mm is a more conservative alternative to the jackknife+ method that guarantees coverage at the ↵ level with exchangeable data, but usually with overly-wide intervals.
[Table]
i 5.
Cross validation+ (CV+) is similar to jackknife+ except splits data into k portions and replaces the b µ  i ( X n +1 ) with b µ  k ( X n +1 ) , the model trained with the k th subset removed.
Split method follows split conformal prediction, which uses half the data for training and the other half for generating the nonconformity scores.
Weighted split is a version of split conformal with likelihood ratio weights to maintain coverage under covariate shift, as in Tibshirani et al.
Baselines for comparison to JAWA For inﬂuence function orders K 2 { 1 , 2 , 3 } , we compared the proposed JAWA-
K method with K -th order inﬂuence function approximations of the jackknife-based baselines that we used as comparisons to JAW—we thus refer to these approximations as IF- K jackknife, IF- K jackknife+, and IF- K jackknife-mm.
Each baseline compared to JAWA-
K is thus also approximated with the same K -th order leave-one-out inﬂuence function models.
4.3 Experimental results
We report experimental results on the predictive interval-generation task for both JAW and JAWA and on the error assessment task for JAW, compared to baselines.
Additional experimental details and supplementary experiments can be found in Appendix D, including for estimated likelihood ratio weights in D.5, ablation study with shift magnitudes in D.6, and coverage histograms in D.9.
4.3.1 Interval generation results for JAW: Coverage and interval width
Figure 3 compares JAW and its baselines, ﬁrstly regarding mean coverage and secondarily regarding median interval width, on all ﬁve UCI datasets for both neural network and random forest predictors, averaged over 200 experimental replicates.
See Appendix D.2 for predictor function details.
Meeting the target coverage level of 1  ↵ is the primary goal of the interval-generation audit task, but for methods that meet or nearly meet the target coverage level, smaller interval widths are more informative.
Additionally, smaller variance in coverage indicates a more reliable or consistent method.
As seen in Figure 3, the JAW predictive interval coverage is above the target level of 0.9 across all datasets, for both random forest and neural network b µ functions, along with the jackknife-mm and weighted split methods.
However, JAW’s interval widths are generally smaller and thus more informative than those of jackknife-mm (which are often overly large, as noted in Barber et al.
Weighted split and JAW perform similarly on mean coverage and median interval width (both methods have coverage guarantees under covariate shift), but JAW avoids sample splitting and as a result has lower coverage variance than weighted split for all dataset and predictor conditions (see Appendix D.3), which suggests that JAW’s predictive intervals are more reliable.
[Figure]
4.3.2 Interval generation results for JAWA: Coverage and interval width
Figure 4 evaluates JAWA coverage and interval width compared to baselines for IF orders K{ 1 , 2 , 3 } with a neural network predictor (see Appendix D.2 for predictor details).
As with the JAW experiments, coverage at the target level of 1  ↵ = 0 .
9 is the primary goal, while secondarily, smaller intervals are more informative for methods that meet or nearly meet target coverage.
For three of the ﬁve datasets (airfoil, wine, and communities), JAWA is the only method that consistently reaches or nearly reaches the target coverage level.
JAWA and all the baselines perform well on the wave datasets, and in the superconduct dataset JAWA still outperforms approximations of jackknife and jackknife+ for all IF orders.
Appendix D.7 provides an example empirical comparison of JAWA and JAW runtimes, which demonstrates that JAWA can be orders of magnitude faster to compute.
4.3.3 Error assessment results for JAW-E: AUC
We now turn to an error-assessment audit task where the goal is to evaluate a method’s ability to estimate the probability that a given prediction is erroneous or not, based on the error criterion | Y n +1  b µ ( X n +1 ) | > ⌧ .
Let E =
[ b µ ( X n +1 )
⌧ , b µ ( X n +1 ) + ⌧ ] .
Then, the goal is to estimate the probability that b µ ( X n +1 ) is correct, i.e., Y n +1 2 E ; or an error, i.e., Y n +1 62 E .
For ﬁve predictive interval-generation methods repurposed to the error assessment task (JAW-E, jackknife+E, cross validation+E, split conformal-E, and weighted split conformal-E), Figure 5 reports the area under the receiver operating characteristic curve (AUROC) for 50 repeated experiments with a neural network predictor, with dataset-speciﬁc values of ⌧ (see Appendix D.4 for details and additional experiments with random forest predictor).
Better performing methods have higher AUROC values for all values ⌧ .
For most tolerance levels and datasets, JAW achieves AUROC values comparable to jackknife+ and CV+ as well as higher AUROC values than split and weighted split conformal prediction.
The
[Figure] [Figure]
comparable performance of JAW and jackknife+ is likely due to a tradeoff between the beneﬁt of
JAW’s validity under covariate shift and its reduced effective sample size inherent to likelihood-ratio weighting, as jackknife+’s and CV+’s AUROC degrades with reduced sample size (Appendix D.6).
Conclusion
In this paper, we develop JAWS, a series of wrapper methods for distribution-free predictive uncertainty auditing tasks when the data exchangeability assumption is violated due to covariate shift.
We also propose a general approach to repurposing any distribution-free predictive inference method to the error assessment task.
We provide rigorous ﬁnite-sample guarantees for JAW and JAW-E on the interval generation and error assessment tasks respectively, and analogous asymptotic guarantees for the computationally efﬁcient JAWA and
JAWA-E. We moreover demonstrate superior performance of the JAWS series on a variety of datasets.
In supplementary experiments we investigate a number of
JAWS’ limitations: weight estimation can address the assumed access to oracle weights with similar empirical performance (Appendix D.5), and JAW’s increased coverage variance with covariate shift can be explained by reduced effective sample size due to importance weighting (Appendix D.6).
Additionally, we note that JAW and JAWA share a limitation with weighted conformal prediction [Tibshirani et al., 2019] of potentially producing overly large intervals in extreme covariate shift cases where a test point’s normalized likelihood ratio approaches or exceeds ↵ .
In the future, we aim to address the problems of reducing coverage variance and improving predictive interval sharpness.
Acknowledgments
This work was supported by the National Science Foundation grant IIS-1840088.
We thank Yoav
Wald for helpful discussions and advice, as well as Peter Schulam for sharing code that facilitated our inﬂuence function approximation implementation and AUC experiments.