JAWS: Auditing Predictive Uncertainty Under Covariate Shift
Anonymous Author(s)
AfﬁliationAddress email
Abstract
We propose JAWS , a series of wrapper methods for distribution-free uncertaintyquantiﬁcation tasks under covariate shift, centered on the core method JAW ,the JA ckknife+ W eighted with data-dependent likelihood-ratio weights.
JAWSalso includes computationally efﬁcient A pproximations of JAW using higherorder inﬂuence functions: JAWA .
Theoretically, we show that JAW relaxes thejackknife+’s assumption of data exchangeability to achieve the same ﬁnite-samplecoverage guarantee even under covariate shift.
JAWA further approaches the JAWguarantee in the limit of either the sample size or the inﬂuence function orderunder common regularity assumptions.
Moreover, we propose a general approachto repurposing any distribution-free uncertainty quantiﬁcation method and itsguarantees to the task of risk assessment: a task that estimates the probability thatthe true label lies within a user-speciﬁed interval.
We then propose JAW-R and JAWA-R as the repurposed versions of proposed methods for R isk assessment.
Practically, JAWS outperform the state-of-the-art predictive inference baselinesin a variety of biased real world data sets for both interval-generation and riskassessment predictive uncertainty auditing tasks.
Introduction
Auditing the uncertainty under data shift Principled quantiﬁcation of predictive uncertainty iscrucial for enabling users to calibrate how much they should or should not trust a given prediction[Thiebes et al., 2021, Ghosh et al., 2021, Tomsett et al., 2020, Bhatt et al., 2021].
Uncertainty-basedpredictor auditing can be considered a type of uncertainty quantiﬁcation performed post-hoc , forexample by a regulator without detailed knowledge of a predictor’s architecture and with limitedresources
[Schulam and Saria, 2019].
Data shift poses a major challenge to uncertainty quantiﬁcationdue to violation of the common assumption that the training and test data are exchangeable, or morespeciﬁcally independent and identically distributed (i.i.d.)
[Ovadia et al., 2019, Ulmer et al., 2020, Zhou and Levine, 2021, Chan et al., 2020].
Therefore, it is essential to develop convenient tools forusers or regulators to audit predictive uncertainty even when training data is biased.
Predictor auditing: interval generation In this work we distinguish between two types predictiveuncertainty auditing.
We describe a ﬁrst type by the task of interval generation , which refers to acommon goal in the distribution-free uncertainty quantiﬁcation literature: to generate a predictiveinterval (or conﬁdence interval) that covers the true label with at least some user-speciﬁed probability.
For instance, an auditor may wish to generate conﬁdence intervals around a prediction that containthe true label with at least, say 90% probability.
Predictor auditing: risk assessment While predictive interval generation has been a central focusof the distribution-free uncertainty quantiﬁcation literature, in some applications the converse computation may be more actionable: estimating the probability that a user-speciﬁed interval around a prediction contains the true label.
We describe this type of predictor auditing task as risk assessmentbecause a natural type of interval for a user to specify is one deﬁned by a safe or acceptable errorthreshold around a prediction, where the goal is then to estimate the probability that the true label iswithin the error threshold (or the “risk" that it is outside the threshold).
A concrete example of whensuch a risk-assessment audit may be more useful to a stakeholder than an interval-generation audit isthe setting of chemical or radiation therapy dose prediction for cancer treatment, where administeringthe correct dosage within 5%  10% is safety-critical (see Appendix A.1 for more details).
As wewill demonstrate, any interval generation method with a distribution-free coverage guarantee can berepurposed for the task of risk assessment with minor modiﬁcations (as we will see in Section 3.3).
Coverage We assume a standard regression setup with training data { ( X 1 , Y 1 ) , ..., ( X n , Y n ) } anda test point ( X n +1 , Y n +1 )
where ( X i , Y i ) 2 R d ⇥ R for all i 2 { 1 , ..., n + 1 } .
Also, we denote apredictor as
[Equation]
[Equation]
X 1 , Y
[Equation]
) , ..., ( X n , Y n ) } ) , where A is the algorithm for ﬁtting the model.
Forpredictive or conﬁdence interval (or region)
[Equation]
{ subsets of R } , a coverage guarantee 49 aims to give a lower bound to the probability that the interval covers a test point’s true label:
[Equation]
The coverage guarantee provides the basis for both interval generation and risk assessment auditing,though it is important to note that in this work we focus on marginal rather than conditional coverage(see [Foygel Barber et al., 2021] for more details on this distinction).
Standard conformal predictionmethods [Vovk et al., 2005, Shafer and Vovk, 2008, Vovk, 2013] along with the jackknife+ andrelated methods [Barber et al., 2021], which we refer to together as “predictive inference" methods,provide a framework for generating predictive intervals with ﬁnite-sample guaranteed coverage.
Exchangeability Standard conformal prediction and thejackknife+ rely on two crucial notions of exchangeability :data exchangeability, that is that the training and test dataare all exchangeable (e.g., i.i.d.); and secondly that themodel-ﬁtting algorithm
A treats the data symmetrically[Barber et al., 2022].
In common situations of datasetshift, however, the data exchangeability assumption is violated.
Empirically, the coverage performance of standardconformal prediction methods can suffer under data shift[Tibshirani et al., 2019, Podkopaev and Ramdas, 2021].
Inthis work, we build on the jackknife+ method due to itsbeneﬁcial compromise between the statistical and computational limitations of other conformal prediction methods[Barber et al., 2021].
However, jackknife+ coverage performance can still degrade in the presence of dataset shift, such as shown in Figure 1.
In this work, we develop JAWS, a series of wrapper methods for distribution-free uncertaintyquantiﬁcation under covariate shift.
Table 1 gives a summary of key properties (details in Section 3).
Our contributions can summarized as follows :
We develop JAW: a jackknife+ method with data-dependent likelihood-ratio weights for uncertainty quantiﬁcation under covariate shift.
We show that JAW achieves the same rigorous,ﬁnite-sample coverage guarantee as jackknife+ [Barber et al., 2021] while relaxing the dataexchangeability assumption to allow for covariate shift.
We develop JAWA: a sequence of computationally efﬁcient approximations to JAW that useshigher-order inﬂuence functions to avoid retraining.
Under assumptions outlined in Giordano et al.[2019a] regarding the regularity of the data, Hessian of the objective (local strong convexity), andthe existence and boundedness of higher order derivatives, we provide an asymptotic guaranteefor the JAWA coverage in the limit of either the sample size or
inﬂuence function order.
We propose a general approach to repurposing any distribution-free predictive inference method
to the risk assessment task, with rigorous guarantees for the coverage probability estimation.
Our approach applies to methods that assume exchangeable data and to methods like JAW and JAWA that allow for covariate shift—JAW-R and JAWA-R refer to the risk assessment versions.
We demonstrate superior empirical performance of JAWS over other distribution-free predictive
inference baselines on a variety of benchmark datasets under covariate shift.
[Table]
Background and related Work2.1 Standard conformal prediction
Conformal prediction has grown into a broad research ﬁeld since arising in the 1990s [Vovk et al. ,2005, Shafer and Vovk, 2008, Vovk et al., 2009, Vovk, 2013, Burnaev and Vovk, 2014].
Withthe exchangeability assumptions in Section 1, standard conformal prediction methods generatea score distribution for the data using a function S : R d ⇥ R !
R (such as the residual score S ( X i , Y i ) =
| Y i  b µ ( X i ) | ) to determine the prediction interval.
These estimates are distribution-free[Lei and Wasserman, 2014, Lei et al., 2018] in that they apply to any exchangeable data distribution.
Split conformal and full conformal are two main types of standard conformal prediction, each bearsits own limitation [Vovk et al., 2005, Shafer and Vovk, 2008].
Split conformal generates scores onlabeled holdout data and is computationally efﬁcient due to not requiring retraining, but samplesplitting to obtain the holdout set can reduce model accuracy [Papadopoulos, 2008, Lei et al., 2018,
Vovk, 2012].
On the other hand, full conformal prediction avoids the holdout set requirement, but atthe expense of the heavy computational cost of retraining the model on every possible target value(or, in practice, on a ﬁne grid of target values) [Ndiaye and Takeuchi, 2019, Zeni et al., 2020].
2.2 Covariate shift
Under the covariate shift assumption, the Y | X distribution is assumed to be the same between trainingand test data but the marginal X distributions may change [Sugiyama et al., 2007, Shimodaira, 2000]:
[Equation]
Rich literature exist in this domain—see Appendix A.2 for more details.
Uncertainty quantiﬁcationis relatively less explored under covariate shift, though recent work [Ovadia et al., 2019, Zhou and
Levine, 2021, Chan et al., 2020] emphasizes its importance, especially in deep learning.
2.3 Conformal prediction under covariate shift and beyond exchangeability
Tibshirani et al.
[2019] develop the idea of weighted exchangeability for adapting conformal predictionto the covariate shift setting.
Random variables V 1 , ..., V n are weighted exchangeable with weightfunctions w 1 , ..., w n if their joint density f can be factorized as f ( v 1 , ..., v n ) =
Q [-ni =
-]n i =1 w i ( v i ) · 114 g ( v 1 , ..., v n ) , where g is independent of ordering on its inputs.
For covariate shift as in (2), thedata { ( X i , Y i ) } are weighted exchangeable with weights given by the likelihood ratio w ( X i )
=
d ˜ P X ( X i ) /
d P X
( X i )
[Tibshirani et al., 2019].
If { V i } represents a set of scores for standard conformal prediction, then we can represent theempirical distribution of { V i } as 1 n +
P ni =
V i + 1 n +1  1 , where  V i denotes a point mass at V i[Barber et al., 2022].
By extension, weighted conformal prediction uses the weighted empiricaldistribution deﬁned as
[-+
-]
P
[Equation]
[Equation]
x )  1 , with weights given by
[Equation]
and
[Equation]
where p wi ( x ) can be thought of as a normalized likelihood ratio weight for each i 2 { 1 , ..., n + 1 } .
Corollary 1 in [Tibshirani et al., 2019] provides the coverage guarantee of weighted conformalprediction that takes the form of (1) but relaxes the exchangeable data assumption to allow forcovariate shift.
However, weighted split and weighted full conformal inherit the same statistical andcomputational limitations, respectively, from their standard (exchangeable) variants.
The recent work of [Barber et al., 2022] provides a novel extension of conformal prediction and thejackknife+ to unknown violations of the exchangeability assumption, including “nonexchangeablejackknife+" deﬁned with ﬁxed weights.
The key difference between the nonexchangeable jackknifein Barber et al.
[2022] and our proposed JAW method is that Barber et al.
[2022] use ﬁxed weights tocompensate for unknown exchangeability violations (not limited to covariate shift) but at the expenseof a bounded but generally nonzero “coverage gap” (drop in guaranteed coverage relative to if thedata were exchangeable), whereas our JAW method with data-dependent weights assumes covariateshift but does not suffer from any similar coverage gap.
See Appendix A.3 for more details.
2.4 Jackknife+
The jackknife+ [Barber et al., 2021], which is closely related to cross conformal prediction [Vovk et al. ,2018], offers a compromise between the statistical limitation of split conformal and the computationallimitation of full conformal, at the cost of a slightly weaker coverage guarantee.
The jackknife+predictive interval can most easily be understood as a modiﬁcation to a predictive interval from theclassic jackknife resampling method [Miller, 1974, Steinberger and Leeb, 2018, 2016] that it buildson.
For some values v 1 , ..., v n , let b q  n, ↵ { v i } and b q + n, ↵ { v i } respectively denote the level ↵ and 1  ↵quantiles of the empirical distribution 1 n +
P ni =1  v i + 1 n +1  1 . Then, denoting the model trainedwithout the i th point as b µ  i = A  ( X 1 , Y 1 ) , ..., ( X i  1 , Y i  1 ) , ( X i +1 , Y i +1 ) , ..., ( X n , Y n )  and theleave-one-out residual R LOOi =
| Y i  b µ  i ( X i ) | , the classic jackknife prediction interval is given by
[Equation]
In contrast, we obtain the jackknife+ predictive interval in Barber et al.
[2021] by replacing the fullmodel prediction b µ ( X n +1 ) in (4) with b [Equation]
i
[Equation]
X n +1 ) :
[Equation]
[Barber et al., 2021] prove that, with the same exchangeability assumptions as in standard conformalprediction, the jackknife+ prediction interval satisﬁes [Equation]
2.5 Approximating leave-one-out models with higher-order inﬂuence functions
Inﬂuence functions (IFs)
[Cook, 1977] have a long history in robust statistics for estimating thedependence of parameters on sample data, and recently IFs have become more widespread inmachine learning for uses including model interpretability Koh and Liang [2017] and approximatingclassic resampling uncertainty quantiﬁcation methods including bootstrap [Schulam and Saria, 2019],jackknife, and leave- k -out cross validation [Giordano et al., 2019b]
[Giordano et al., 2019a].
In eachof these cases IFs enable approximation of the parameters that would be obtained by retraining themodel on resampled data by instead estimating the effect of a corresponding reweighting.
Alaa and Van Der Schaar [2020] proposed approximating the leave-one-out models required by the jackknife+with higher-order IFs, but their work assumes exchangeable or i.i.d.
train and test data.
Let ˆ ✓ denote the ﬁtted parameters for predictor b µ trained on the full training data.
Given Assumptions1-4 in Giordano et al.
[2019a]—which require that ˆ ✓ is a local minimum of the objective function, thatthe objective is k + 1 times continuously differentiable with bounded norms, and that the objective isstrongly convex in the neighborhood of ˆ ✓ —then the k -th order leave-one-out IF refers to the k -thorder directional derivative of the model parameters ˆ ✓ with respect to the data weights, in the directionof the leave-one-out change in weights (See A.4 for more details).
With each of these k th orderleave-one-out IFs for k 2 { 1 , ..., K } , denoted with condensed notation  k  i ˆ ✓ , we can construct a
K -th order Taylor series approximation to estimated the leave-one-out model parameters ˆ ✓  i :
[Equation]
In this work we implement the algorithm proposed by Giordano et al.
[2019a] to compute higher-order
IFs, a recursive procedure based on foreward-mode automatic differentiation [Maclaurin et al., 2015]for memory efﬁciency in computing higher-order directional derivatives.
Our introduction of IFs ishighly simpliﬁed—we refer to Appendix A.4 and to Giordano et al.
[2019a] for more details.
Proposed approach and theoretical results3.1 JAW: Jackknife+ weighted with data-dependent weights
We present JAW , the JA ckknife+ W eighted with data-dependent likelihood-ratio weights, deﬁned bythe following predictive interval:
[Equation]
where R LOOi =
[Equation]
Y
[Equation]
, where
[Equation]
LOOi denotes the level ↵quantile of the empirical distribution
[Equation]
where b
[Equation]
LOOi is the level 1  ↵ quantile for
P
[Equation] [Equation]
1 , and the p wi ( X ) are as deﬁned in (3)
[Equation]
We show that b
C JAW n, ↵ ( X n +1 ) satisﬁes the same coverage guarantee as the jackknife+ except relaxingthe data exchangeability assumption to allow for covariate shift, which we state formally as follows:
Theorem 1.
With covariate shift between the training and test data, the JAW interval in (8) satisﬁes
[Equation]
Remark.
The results from Tibshirani et al.
[2019] do not directly imply Theorem 1.
Applying theapproach from Tibshirani et al.
[2019] to the jackknife+ entails treating b µ  i ( X n +1 ) ± R LOOi as 183 implicit nonconformity scores.
Tibshirani et al.
[2019] assume that the nonconformity scores for thetraining data { V 1 , ..., V n } are weighted exchangeable with the nonconformity score for the test point 185 V n +1 .
But, observe that for i 2 { 1 , ..., n } , b µ  i is trained on n  1 datapoints, whereas b µ  ( n +1) =
b µis trained on n datapoints.
Thus, no reweighting can make b µ  i equivalent to b µ in distribution, andtherefore
[Equation]
Proof sketch:
Our proof technique for Theorem 1 extends the jackknife+ coverage guarantee proofin Barber et al.
[2021] to the covariate shift setting for JAW using likelihood ratio weights as in
Tibshirani et al.
The outline is as follows:
Setup :
Following Barber et al.
[2021], we deﬁne a set of leave- two -out models { ˜ µ  ( i,j ) } .
We thengeneralize the notion of “strange" points described in Barber et al.
[2021] to covariate shift.
Bounding the total normalized weight of strange points : We establish deterministically that thetotal normalized weight of strange points cannot exceed 2 ↵ .
Weighted exchangeability using the leave-two-out models : Using the leave-two-out model construction, we leverage weighted exchangeability to show that the probability that a test point n + is strange is thus bounded by 2 ↵ .4.
Connection to JAW :
Lastly , we show that the JAW interval can only fail to cover the test labelvalue Y n +1 if n + 1 is a strange point.
While JAW assumes access to oracle likelihood ratio weights, in practice this information often hasto be estimated.
See Appendix D.5 for a discussion and experiments of JAW with estimated weights.
3.2 JAWA: Using higher-order inﬂuence functions to approximate JAW without retraining
For computationally efﬁcient JAW A pproximations that avoid retraining n leave-one-out models,we propose the JAWA sequence, which approximates the leave-one-out models required by JAWusing higher-order inﬂuence functions.
For each training point i 2 { 1 , ..., n } , deﬁne the K -th orderinﬂuence function approximation to the leave-one-out reﬁt parameters ˆ ✓  i , obtained from Algorithm4 in Giordano et al.
[2019a], as given by equation (7), and let b µ IF- K  i be the model with with theseapproximated parameters ˆ ✓ IF- K  i for each
i 2 { 1 , ..., n } .
Then, the prediction interval for the K -thorder JAWA (i.e., for JAWA- K ) is given by
[Equation]
i (10) with R IF- K,LOO i = b µ IF- K  i ( X i )  Y i  , quantiles deﬁned analogously to JAW, and p wi ( X ) as in (3).
We now provide an asymptotic coverage guarantee for b
C JAWA- K n, ↵ ( X n +1 ) that holds either in the limitof the sample size or in the limit of the inﬂuence function order, under regularity conditions formallydescribed in Giordano et al.
[2019a].
These assumptions concern the regularity and continuity of thetraining data, local convexity of the objective (or that the Hessian is strongly positive deﬁnite), andthe existence and boundedness of the objective’s 1st through K + 1 th order directional derivatives.
Theorem 2.
Let Assumptions 1 - 4 and either Condition 2 or Condition 4 from Giordano et
al.[2019a] hold uniformly for all n .
Then, with covariate shift between the training and test data, ineither the limit of the training sample size n ! 1 or in the limit of the inﬂuence function order
K ! 1 , the JAWA- K interval in (10) satisﬁes
[Equation]
We leave the proof to Appendix C.2, but we note that the result follows by combining Propositions 2and 3 in Giordano et al.
[2019a] with the JAW coverage guarantee that we present in Theorem 1.
3.3 Risk assessment under covariate shift
We now propose a general approach to adapting any weighted predictive inference method andassociated covariate-shift coverage guarantee to the task of risk assessment, that is, estimating theprobability that a true test point’s label Y n +1 lies within a given arbitrary, ﬁnite interval I =
[ a L , a U ] .
We also provide a similar result for uniform-weighted methods that assume exchangeable data, butfor brevity we defer the exchangeable data approach and result to Appendix B.1.
The key idea is as follows: given I (violet interval in Figure 2), leverage a predictive inferencemethod’s weighted predictive distribution (blue and red lines in Figure 2; quantiles on this distributiondetermine predictive intervals) to ﬁnd the method’s largest predictive interval contained in I (greeninterval in Figure 2), and then to use this predictive interval’s target coverage to estimate the coverageof I .
That is, for a predictive inference method with data-dependent weights for generating intervals [Equation] n,
[Equation]
⇤ under covariate shift, deﬁne ↵ w-audit I as
[Equation]
[Equation]
Then, we estimate the coverage probability for interval I as the predictive inference method’s targetcoverage for the interval corresponding to ↵ w-audit I :
[Equation]
otherwise
[Equation]
While the target coverage is used in (13), the following theorem gives the worst-case risk assessmentguarantee for covariate shift (proof in Appendix C.3; see Appendix B.1 for the analogous exchangeable data result).
Corollary 1 in Appendix B.2 and corollary 2 in Appendix B.3 follow from Theorem3 and give the risk assessment guarantees for JAW-R and JAWA-R respectively.
Theorem 3.
Given a predictive inference method for generating intervals b
C audit n, ↵ ( X n +1 ) =⇥
q  n, ↵ { p wi ( x )  V Li } , q + n, ↵ { p wi ( x )  V Ui } ⇤ with coverage guarantee P { Y n +1b
C audit n, ↵ ( X n +1 ) }
1  c 1 ↵  c 2 under covariate shift where c 1 , c 2  0 are constants, suppose we are given some 243 arbitrary interval I =
[ a L , a U ] ✓ R (assuming a L  a U WLOG).
Deﬁne ↵ w-audit
I as follows:
↵ w
[Equation]
audit I = min
[Equation]
o◆ .
Then, I has coverage given by
[Equation]
if ↵ w [Equation] I exists and ↵ w-audit I < 1  c 2 c otherwise .
Experiments 14.1 Datasets and creation of covariate shift
We conduct experiments on ﬁve UCI datasets Dua and Graff [2017] with various dimensionality(Table 2): airfoil self-noise, red wine quality prediction [Cortez et al., 2009], wave energy converters,superconductivity [Hamidieh, 2018], and communities and crime [Redmond and Baveja, 2002].
[Table] [Table]
We use exponential tilting to induce covariate shift on the test data, based on the approach used in Tibshirani et al.
We ﬁrst randomly sample 200 points for the training data, and then samplethe biased test data from the remaining datapoints that are not used for training with probabilitiesproportional to exponential tilting weights.
See Appendix D.1 for additional details.
4.2 Baselines
Baselines for comparison to JAW We compared JAW to the following baselines:
Naive estimates are based on training data residuals | Y i  b µ ( X i ) | , which suffers from overﬁtting.2.
Jackknife uses the classic Jackknife resampling as in (4).3.
Jackknife+ follows (5), which replaces the prediction [Equation] µ [Equation]
X n +1 ) in jackknife with [Equation]  i ( X n +1 ) .4. Jackknife-mm is a more conservative alternative to the jackknife+ method that guaranteescoverage at the 1  ↵ level with exchangeable data, but usually with overly-wide intervals.
[Equation]
CV+ (cross validation) similar to jackknife+ except splits data into k portions and replaces theb µ
i
X n +1 ) with b µ  k ( X n +1 ) , the model trained with the k th subset removed.6.
Split method follows the split conformal prediction and use half of the data for training and theother half of the data for generating the nonconformal scores.7.
Weighted split version of split conformal with likelihood ratio weights for covariate shift, as inTibshirani et al.
Baselines for comparison to JAWA For inﬂuence function orders K 2 { 1 , 2 , 3 } , we compared the 269 proposed JAWA-
K method with K -th order inﬂuence function approximations of the jackknife-basedbaselines that we used as comparisons to JAW—we thus refer to these approximations as IF- Kjackknife, IF- K jackknife+, and IF- K jackknife-mm.
Each baseline compared to JAWA-
K is thusalso approximated with the same K -th order leave-one-out inﬂuence function models.
4.3 Experimental results
We report experimental results on the interval generation task for both JAW and JAWA and on the riskassessment task for JAW, compared to baselines.
Additional experimental details and supplementaryexperiments can be found in Appendix D, including for estimated likelihood ratio weights in D.5,ablation study with shift magnitudes in D.6, and coverage histograms in D.9.
4.3.1 Interval generation results for JAW: coverage and interval width
Figure 3 compares JAW and its baselines, ﬁrstly regarding mean coverage and secondarily regardingmedian interval width, on all ﬁve UCI datasets for both neural network and random forest predictors,averaged over 200 experimental replicates.
See Appendix D.2 for predictor function details.
Meetingthe target coverage level of 1  ↵ is the primary goal of the interval-generation audit task, butfor methods that meet or nearly meet the target coverage level, smaller interval widths are moreinformative.
Additionally, smaller variance in coverage indicates a more reliable or consistent method.
As seen in Figure 3, the JAW predictive interval coverage is above the target level of 0.9 acrossall datasets, for both random forest and neural network b µ functions, along with the jackknife-mmand weighted split methods.
However, JAW’s interval widths are generally smaller and thus moreinformative than those of jackknife-mm (which are often overly large, as noted in Barber et al.[2021]).
Weighted split and JAW perform similarly on mean coverage and median interval width(both methods have coverage guarantees under covariate shift), but JAW avoids sample splitting andas a result has lower coverage variance than weighted split for all dataset and predictor conditions(see Appendix D.3), which suggests that JAW’s predictive intervals are more reliable.
[Figure]
repeated experiments.
4.3.2 Interval generation results for JAWA: coverage and interval width
Figure 4 evaluates JAWA coverage and interval width compared to baselines for IF orders K 2{ 1 , 2 , 3 } with a neural network predictor (see Appendix D.2 for predictor details).
As with the JAWexperiments, coverage at the target level of 1  ↵ = 0 .
9 is the primary goal, while secondarily, smallerintervals are more informative for methods that meet or nearly meet target coverage.
For three ofthe ﬁve datasets (airfoil, wine, and communities) JAWA is the only method that consistently reachesor nearly reaches the target coverage level.
JAWA and all the baselines perform well on the wavedatasets, and in the superconduct dataset JAWA still outperforms approximations of jackknife andjackknife+ for all IF orders.
Appendix D.7 provides an an example empirical comparison of JAWAand JAW runtimes, which demonstrates that JAWA can be orders of magnitude faster to compute.
4.3.3 Risk assessment results for JAW: AUC
We now turn to the risk-assessment audit task where the goal is to evaluate a method’s ability toestimate the coverage probability of a given interval, or to estimate the risk that the true label liesoutside some error threshold ⌧ around the original prediction b µ ( X n +1 ) .
Given prediction b µ ( X n +1 )and error threshold ⌧ , deﬁne the interval of interest as I ( b µ ( X n +1 ) , ⌧ ) =
[ b µ ( X n +1 )
⌧ , b µ ( X n +1 )+ ⌧ ] .
[Figure]
Error bar shows the standard error of 200 repeated experiments. Then, the goal is to estimate the probability that b µ ( X n +1 ) is correct, i.e., Y n {++
+}
I ( b µ ( X n +1 ) , ⌧ ) ; orincorrect, i.e., Y n +1 62 I ( b µ ( X n +1 ) , ⌧ ) .
For four predictive interval-generation methods repurposedto the risk assessment task (JAW-R, jackknife+R, cross validation+R, and split conformal-R), Figure5 reports the area under the receiver operating characteristic curve (AUROC) for values of ⌧ selectedwith uniform spacing in the interval [ q  n, 0 .
05 { R LOOi } , q + n, 0 .
05 { R LOOi } ] (between the 0.05 and 0.95 313 quantiles of the leave-one-out residuals for a dataset) for ﬁve repeated experiments with the randomforest predictor.
Better performing methods have higher overall AUROC values for all values ⌧ .
[Figure]
Airfoil
[Figure]
JAW achieves AUROC values comparable to jackknife+ and CV+ for most tolerance levels anddatasets.
JAW moreover achieves higher AUROC values than its baselines in the wine dataset withthe random forest predictor.
These results suggest that, for risk assessment, JAW likely performs atleast as well as its baselines and may offer a marginal beneﬁt depending on the dataset and predictor.
Conclusion and Future Work
In this paper, we develop JAWS, a series of wrapper methods for distribution-free predictive uncertainty auditing tasks when the data exchangeability assumption is violated due to covariate shift.
Wealso propose a general approach to repurposing any distribution-free predictive inference method tothe risk assessment task.
We provide rigorous ﬁnite-sample guarantees for JAW and JAW-R on theinterval generation and risk assessment tasks respectively, and analogous asymptotic guarantees forthe computationally efﬁcient JAWA and JAWA-R.
We moreover demonstrate superior performance ofthe JAWS series on a variety of datasets.
In supplementary experiments we investigate a number of JAWS’ limitations: weight estimation can address the assumed access to oracle weights with similarempirical performance (Appendix D.5), and JAW’s increased coverage variance with covariate shiftcan be explained by reduced effective sample size due to importance weighting (Appendix D.6).
Additionally, we note that JAW and JAWA share a limitation with weighted conformal prediction[Tibshirani et al., 2019] of potentially producing overly large intervals in extreme covariate shiftcases where a test point’s normalized likelihood ratio approaches or exceeds ↵ .
In the future, we aimto address the problems of reducing coverage variance and improving predictive interval sharpness.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Appendix E.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A] Please do not modify the questions and only use the provided macros for your answers.
Note that the Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitations of
our method in the experiments (Section 4) and the conclusion (Section 5).
We aim to
resolve them in our future work.
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
Our work
are motivated from the predictor auditing perspective, which aims to facilitate more
convenient quality estimation of the predictive models for the users and regulators.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] We include a proof sketch of Theorem 1 in Section 3.1, and have the full proof of all our theoretical results in Appendix C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We include an
anonymous link to our code repository in the supplemental material, Appendix E.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Section 4 and Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes] See Figure 3 and Figure 4.
We repeat our experiments1000 times for Figure 3 and 200 times for Figure 4 and report the standard error aserror bars.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We include the computational
[Table]
details in the supplemental materials, Appenidx E
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes]
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
[Table]
We include an anonymous link to our code repository in the supplemental material.
[Table] (d) Did you discuss whether and how consent was obtained from people whose data you’re [Table] using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table] (a) Did you include the full text of instructions given to participants and screenshots, if [Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table] (c) Did you include the estimated hourly wage paid to participants and the total amount [Table] spent on participant compensation?
[N/A]
[Table]