PAC Prediction Sets for Meta-Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Uncertainty quantiﬁcation is a key component of machine learning models targetedat safety-critical systems such as in healthcare or autonomous vehicles.
We studythis problem in the context of meta learning, where the goal is to quickly adapt apredictor to new tasks.
In particular, we propose a novel algorithm to construct PACprediction sets , which capture uncertainty via sets of labels, that can be adaptedto new tasks with only a few training examples.
These prediction sets satisfy anextension of the typical PAC guarantee to the meta learning setting; in particular,the PAC guarantee holds with high probability over future tasks.
We demonstratethe efﬁcacy of our approach on four datasets across three application domains:mini-ImageNet and CIFAR10-C in the visual domain, FewRel in the languagedomain, and the CDC Heart Dataset in the medical domain.
In particular, ourprediction sets satisfy the PAC guarantee while having smaller size compared toother baselines that also satisfy this guarantee.
Introduction
Uncertainty quantiﬁcation is a key component for safety-critical systems such as healthcare androbotics, since it enables agents to account for risk when making decisions.
Prediction sets are apromising approach, since they provide theoretical guarantees when the training and test data arei.i.d.
[1, 2, 3, 4]; there have been extensions to handle covariate shift [5, 6, 7] and online learning [8].
We consider the meta learning setting, where the goal is to adapt an existing predictor to new tasksusing just a few training examples [9, 10, 11, 12, 13].
Most approaches follow two steps: metalearning ( i.e., learn a predictor for fast adaptation) and adaptation ( i.e., adapt this predictor to the newtask).
While there has been work on prediction sets in this setting [14], a key shortcoming is thattheir guarantees do not address some important needs in meta learning; in particular, they do not holdwith high probability over future tasks.
For instance, in autonomous driving, the future tasks mightbe new regions, and correctness guarantees ought to hold for each region; in healthcare, future tasksmight be new patient cohort, and correctness guarantees ought to hold for each cohort.
We propose an algorithm for constructing probably approximately correct (PAC) prediction sets forthe meta learning setting.
As an example, consider learning an image classiﬁer for self-driving carsas our use case (Figure 1).
Labeled images for image classiﬁer learning are collected, with differentregions representing different but related tasks ( e.g., New York, Pennsylvania, and California).
Thegoal is to deploy this learned image classiﬁer to a new target region ( e.g., Alaska) in a cost-efﬁcientway, e.g., minimizing label gathering cost on labeled images from Alaska for the classiﬁer adaptation.
Due to the safety-critical nature of the task, it is desirable to quantify uncertainty in a way thatprovides rigorous guarantees, where the guarantees are adapted to a new task ( e.g., Alaska images)—i.e.
, they hold with high probability speciﬁcally for future observations from this new task.
In contrast,existing approaches only provide guarantees on average across future tasks [14].
car person adaptation set Acar person calibration set task dataset p ✓f A
F A 1 ⌧car person adaptation set Acar person calibration set task dataset p ✓f A F A 2 ⌧⌧ meta learning and calibration car person adaptation set A task dataset p ✓ f A adaptation car person car person evaluation set task dataset p ✓ [ Y / 2 F A ⌧ ( X )]  " inference
Our proposed PAC prediction set starts with a meta-learned and potentially adaptable score functionas our base predictor (using a standard meta learning or few shot classiﬁcation approach).
Then, welearn the parameter of a prediction set for meta “calibration” by using calibration task distributions( e.g., labeled images from Texas and Massachusetts).
Given a test task distribution ( e.g., labeledimages from Alaska), we only have a few labeled examples for adapting the score function to thistarget task without further updating the prediction set parameter, which potentially saves additionallabeled examples.
We prove that the proposed algorithm satisﬁes the PAC guarantee tailored to themeta learning setup.
Additionally, we demonstrate the validity of the proposed algorithm over fourdatasets across three application domains: mini-ImageNet [10] and CIFAR10-C [15] in the visualdomain, FewRel [16] in the language domain, and CDC Heart Dataset [17] in the medical domain.
Related Work
Meta learning for few-shot learning.
Meta-learning follows the paradigm of learning-to-learn [9],usually formulated as few-shot learning: learning or adapting a model to a new task or distributionby leveraging a few examples from the distribution.
Existing approaches propose neural networkpredictors or adaptation algorithms [10, 11, 12, 13] for few-shot learning.
However, there is noguarantee that these approaches can quantify uncertainty properly, and thus they do not provideguarantees for probabilistic decision-making.
Conformal prediction.
The classical goal in conformal prediction (and inductive conformal prediction) [18, 2, 19] is to ﬁnd a prediction set that contains the label with a given probability, withrespect to randomness in both calibration examples and a test example, where a score function (ornon-conformity score) is given for the inductive conformal prediction.
This property is known as marginal coverage.
If the labeled examples are independent and identically distributed (or moreprecisely exchangeable), prediction sets constructed via inductive conformal prediction have marginalcoverage [19, 2].
However, the identical distribution assumption can fail if the covariates of a testdatapoint are differently distributed from the training covariates, i.e., covariate shift occurs.
Assumingthat the likelihood ratio is known, the weighted split conformal prediction set [5] satisﬁes marginalcoverage under covariate shift.
So far, we have considered only two distributions, a calibrationdistribution and test distribution.
In meta learning, a few examples from multiple distributionsare given, providing a different setup.
Assuming exchangeability of these distributions, conformalprediction sets for meta learning have been constructed, covering over a random sample from a newrandomly drawn distribution, exchangeable with the training distributions [14].
PAC prediction sets.
Marginal coverage is unconditional over calibration data, and does not holdconditionally over calibration data ( i.e., with speciﬁed probability for a ﬁxed calibration set).
Trainingconditional conformal prediction was introduced as a way to construct prediction sets with PACguarantees [20]; Speciﬁcally, for independent and identically distributed calibration examples, theyconstruct a prediction set that contains a true label with high probability with respect to calibrationexamples; thus, these prediction sets cover the labels for most future test examples. This approachapplies classical techniques for constructing tolerance regions [1, 21] to the non-conformity score toobtain these guarantees.
Recent work casts this technique in the language of learning theory [3], bothenabling a more efﬁcient algorithm as well as informing our extension to the meta-learning setting;thus, we adopt this formalism in our exposition.
PAC prediction sets can be constructed in otherlearning settings— e.g., under covariate shift (assuming the distributions are sufﬁciently smooth [6] orwell estimated [7]).
In the meta learning setting, the meta conformal prediction set approach also hasa PAC property [14], but does not control error over the randomness in the adaptation test examples.
In contrast, we propose a prediction set satisfying a meta-PAC property— i.e., separately conditionalwith respect to the calibration data and the test adaptation examples; see Section 4.1 for details.
Background3.1 PAC Prediction Sets
We describe a training-conditional inductive conformal prediction strategy for constructing probablyapproximately correct (PAC) prediction sets [20], using the reformulation as a learning theory problemin [3].
Let X be a set of examples or features and Y be a set of labels.
We consider a given scorefunction f : X ⇥ Y !
R  0 , which is higher if y 2 Y is the true label for x 2 X .
We consider aparametrized prediction set; letting ⌧ 2 R  0 , we deﬁne a prediction set F ⌧ : X ! 2 Y as follows:
[Equation]
Consider a family of distributions ( p ✓ ) ✓ 2 ⇥ over Z :
=
X ⇥ Y , parametrized by ✓ 2 ⇥ .
For an 90 unknown ✓ , we observe independent and identically distributed (i.i.d.)
calibration datapoints S ⇠ p n ✓ .
91 We deﬁne a prediction set, or equivalently the corresponding parameter ⌧ , to be " -correct, as follows:
Deﬁnition 1.
A parameter ⌧ 2 R  0 is " -correct for p ✓ if
[Equation]
Letting T " ( ✓ ) be the set of ⌧ satisfying (1) for p ✓ , we can write ⌧ 2 T " ( ✓ ) ✓ R  0 if ⌧ is " -correct for 94 p ✓ .
If an estimator of ⌧ is " -correct on most random calibration sets S ⇠ p n ✓ , we say it is PAC.
Deﬁnition 2.
An estimator ˆ ⌧ " ,  : Z n !
R  0 is ( " ,  ) -probably approximately correct (PAC) for p ✓ if
[Equation]
Along with the PAC guarantee, the prediction set should be small in size.
Here, by increasingthe scalar parameterization, we see that the expected prediction set size is decreasing with anincreasing prediction set error ( i.e.,
X,Y [ Y / 2 F ⌧ ( X )] ).
Thus, a practical ( " ,  ) -PAC algorithmthat implements the estimator ˆ ⌧ " ,  maximizes ⌧ while satisfying a PAC constraint to minimize theexpected prediction set size [3].
Notably, the time complexity of evaluating a prediction set is O (1)due to the scalar parameterization, while being O ( n ) using the naive implementation of [20], withrespect to the number of calibration sample size n .
3.2 Meta Learning
Meta learning aims to learn a score function that can be conveniently adapted to a new distribution, also called a task .
Consider a task distribution p ⇡ over ⇥ , and M training task distributionsp ✓ 1 , . .
. , p ✓ M , where ✓ 1 , . .
. , ✓ M is an i.i.d.
sample from p ⇡ .
Here, we mainly consider meta learningwith adaptation ( e.g., few-shot learning) to learn the score function, while having learning withoutadaptation ( e.g., zero-shot learning) as a special case of our main setup.
For meta learning without adaptation, labeled examples are drawn from each training task distributionto learn the score function f ; : X ⇥ Y !
R  0 , and the same score function is used for inference( i.e., at test time).
For meta learning with adaptation, t adaptation examples are also drawn fromeach training task distribution to learn the score function f : Z t ⇥ X ⇥ Y !
R  0 , and the same 113 number of adaptation examples are drawn from a new test task distribution for adaptation.
Givenan adaptation set A 2 Z t , the adapted score function is denoted by f A : X ⇥ Y !
R  0 , wheref A ( x, y ) :
= f
( A, x, y ) .
We consider Y large enough to include all label classes, some of whichpossibly seen during testing but not during training.
This formulation includes learning setups( e.g., few-shot learning) where unseen label classes are observed during inference, by consideringdifferent distributions over labeled examples during training and inference.
Meta PAC Prediction Sets
We propose a PAC prediction set for meta learning.
In particular, we consider a meta-learned scorefunction, which is typically trained to be a good predictor after adaptation to the test distribution.
Ourprediction set has a correctness guarantee for future test distributions.
There are three sources of randomness: (1) an adaptation set and a calibration set for each calibrationtask, (2) an adaptation set for a new task, and (3) an evaluation set for the same new task. Wecontrol the three sources of randomness and error by  , ↵ , and " , respectively, aiming to constructa prediction set that contains labels from a new task. Given a meta-learned model, the proposedalgorithm consists of two steps: constructing a prediction set—captured by a speciﬁc parameter to bespeciﬁed below—for each calibration task (related to " and ↵ ), and then constructing a predictioninterval over parameters (related to ↵ and  ). Suppose we have prediction sets for each task, parametrized by some threshold parameters.
Supposethat, for each task, the parameters only depend on an adaptation set for the respective task.
As tasksare i.i.d., the parameters also turn out to be i.i.d..
The distribution of parameters represents a possiblerange of the parameter for a new task.
In particular, if for each task, the prediction set containsthe true label a fraction 1  " of the time, the distribution of the parameters suggests how one mayachieve " -correct prediction sets for a new task.
Based on this observation, we construct a predictioninterval that contains a fraction 1  ↵ of " -correct prediction set parameters.
However, the parametersof prediction sets need to be estimated from calibration tasks, and the estimated parameters dependon the randomness of calibration tasks.
Thus, we construct a prediction interval over parameterswhich accounts for this randomness.
The interval is chosen to be correct ( i.e., to contains " -correctprediction set parameters for a fraction 1  ↵ of new tasks) with probability at least 1   over therandomness of calibration tasks.
4.1 Setup and Problem
We describe the problem on PAC prediction set construction for meta learning with adaptation; see
Appendix A for a simpler problem setup without adaptation, a special case of our main problem.
Setup.
In the setting of Section 3.2, the training samples from some training distributions p ✓ 0 i ,i = 1 , . . .
, M are used to learn a score function via an arbitrary meta learning method, includingmetric-based meta-learning [11] and optimization-based meta-learning [12].
We denote the givenmeta-learned score function by f .
Additionally, we have N calibration distributions p ✓ 1 , . .
p ✓ N ,where ✓ 1 , . .
. , ✓ N is an i.i.d.
sample from p ⇡ .
From each calibration distribution p ✓ i , we draw n i.i.d.labeled examples for a calibration set, i.e., ( S 1 , . . .
, S N ) ⇠ p n ✓ 1 · · · p n ✓ N .
Additionally, we draw t i.i.d.labeled examples for an adaptation set, i.e., ( A 1 , . . .
, A N ) ⇠ p t ✓ 1 · · · p t ✓ N ; we use them to constructthe meta PAC prediction set.
Finally, we consider a test distribution p ✓ , where ✓ ⇠ p ⇡ .
From it, wedraw i.i.d.
labeled adaptation examples A and i.i.d.
evaluation examples.
✓ x S y ( y / 2 F 0 ( x ; S )) [Figure] ] ✓ x S y ⌧ ( y / 2 F ⌧ ( x )) (b) PAC guarantee [1, 20, 3]✓ x A y
( y / 2 F 0 ( A,x ; A 1: N ,S 1: N )
S i
A
i ✓ i ⇡
N (c) Meta task-marginal guarantee [14] ✓ x A y
( y / 2 F ⌧ ( A,x ))
S i
A
i ✓ i ⌧ ⇡
N (d)
[Figure]
PAC guarantee (ours)
Problem.
Our goal is to ﬁnd a prediction set F ⌧ : Z t ⇥ X !
Y parametrized by ⌧ 2 R  0 155 which satisﬁes a meta-learning variant of the PAC deﬁnition.
In particular, consider an estimatorˆ ⌧ " , ↵ ,  : Z n ⇥ N ⇥ Z t ⇥ N !
R  0 that estimates ⌧ for a prediction set, given an augmented calibration [Figure] set ( S 1 , . .
S N , A 1 , . . . , A N ) .
Let T " ( ✓ , A ) be a set of all " -correct parameters ⌧ with respect to fadapted using a test adaptation set A .
We also consider F A ⌧ ( x, y ) :
= F ⌧ ( A, x, y ) .
We say that a prediction set F ⌧ parametrized by ⌧ is " -correct for p ✓ for a ﬁxed adaptation set A ,if ⌧ 2 T " ( ✓ , A ) .
Moreover, we want the prediction set F ⌧ where ⌧ is estimated via ˆ ⌧ " , ↵ ,  to becorrect on most test sets A for adaptation and for most test tasks ✓ .
Letting S 1: k :
=
( S 1 , . . .
, S k ) and
A 1: k :=
( A 1 , . . . ,
A k ) for any k  1 , we want that, given S 1:
N 2 Z n ⇥ N and A 1: N 2 Z t ⇥ N , [Equation] where ✓ ⇠ p ⇡ and A ⇠ p t ✓ .
Intuitively, we want a prediction set, adapted via A , to be correct onfuture labeled examples from p ✓ .
Finally, we want the above to hold for most augmented calibrationsets ( S 1: N , A 1: N ) from most calibration tasks ✓ 1 , . . .
, ✓ N ⇠ p ⇡ , leading to the following deﬁnition.
Deﬁnition 3.
An estimator ˆ ⌧ " , ↵ ,  : Z n ⇥ N ⇥ Z t ⇥ N !
R  0 is ( " , ↵ ,  ) -meta probably approximately 167 correct (mPAC) for p ⇡ if
[Equation]
where ✓ i , ✓ ⇠ p ⇡ , S i ⇠ p n ✓ i for i = 1 , . .
[-(a) Marginal guarantee [2] ✓ x S y ⌧ ( y / 
-]
[-( y / 2 f 
-]
, N , S ⇠ p n ✓ , A i ⇠ p t ✓ i for i = 1 , . . . , N , and A ⇠ p t ✓ .
Notably, Deﬁnition 3 is tightly related to the PAC deﬁnition of meta conformal prediction [14].
Weillustrate the difference in Figure 2; the ﬁgure represents the different guarantees of prediction setsvia graphical models.
Each stripe pattern indicates the random variables that are conditioned on inthe respective guarantee.
Figure 2d represents the proposed guarantee, having Figure 2b and 6b asspecial cases.
Figure 2c shows the guarantee of meta conformal prediction proposed in [14], also having Figure 2a and 6a as special cases.
Figure 2d shows that the proposed correctness guarantee isconditioned over calibration tasks , an adaptation set A from a test task , and test data ( x, y ) .
Thisimplies that the correctness guarantee of an adapted prediction set (thus conditioned on the adaptationset) holds on most future test datasets from the same test task.
In contrast, the guarantee in Figure 2cis conditioned on the entire test task.
Thus, the correctness guarantee does not hold conditionallyover adaptation sets; this means that the adapted prediction set (thus conditioned on an adaptationset) does not need to be correct over most future datasets in the same task.
4.2 Algorithm
Next, we propose our meta PAC prediction set algorithm for an estimator ˆ ⌧ " , ↵ ,  that satisﬁes Deﬁnition 3 while aiming to minimize the prediction set size, by leveraging the scalar parameterization ofa prediction set.
The algorithm consists of two steps: ﬁnd a prediction set parameter over labeledexamples S i and A i from the i -th calibration task distribution, and ﬁnd a prediction set over theparameter of prediction sets in the previous step.
At inference, the constructed prediction set isadapted to a test task distribution via an adaptation set A .
Importantly, in ﬁnding prediction sets( i.e., implementating an ( " ,  ) -PAC estimator ˆ ⌧ " ,  ), we can use any PAC prediction set algorithms( e.g., Algorithm 2 in Appendix B).
Meta calibration.
For the ﬁrst step, for each i 2 { 1 , . . . , N } , we estimate a prediction set parameterˆ ⌧ " , ↵ / 2 ( S i ) , satisfying a ( " , ↵ / 2 )-PAC property using a calibration set S i and a score functionf adapted via an adaptation set A i , leading to f A i , by using any meta learning and adaptationalgorithm.
We consider the N learned prediction set parameters—with a dummy label 1 to ﬁtthe convention of the PAC prediction set algorithm—to form a new labeled sample, i.e., S 0 :=((ˆ ⌧ " , ↵ / 2 ( S 1 ) , 1) , . . .
, (ˆ ⌧ " , ↵ / 2 ( S N ) , 1)) .
In the second part, we construct an ( ↵ / 2 ,  )-PAC prediction set for the threshold ⌧ using S 0 anda score function g deﬁned as g ( ⌧ , y ) :
= ⌧ ( y = 1) , where ( · ) is the indicator function 2 .
Thus,the constructed prediction set can be equivalently viewed as an interval [ˆ ⌧ ↵ / 2 ,  ( S 0 ) , 1 ) where thethresholds ⌧ of most calibration distributions are included; see Section 4.3 for a formal statementalong with intuition on our design choice, and see Algorithm 1 for an implementation.
Adaptation and inference.
We then use labeled examples A from a test distribution to adapt themeta-learned score function f as in the meta calibration part.
Then, we use the same parameterˆ ⌧ " , ↵ ,  ( S 1: N , A 1: N ) of the meta prediction set along with the adapted score function as our ﬁnal metaprediction set f ˆ ⌧ " , ↵ ,  ( S 1: N ,A 1: N ) for inference in the future.
Reusing the same meta prediction set parameter after adaptation is valid, as the meta prediction setparameter is also chosen after adaptation via a hold-out adaptation set from the same distributionas each respective calibration set.
Moreover, by reusing the same parameter, we only need a fewlabeled examples from the test distribution for an adaptation set, without requiring additional labeledexamples for calibration on the test distribution.
Finally, we rely on a PAC prediction set algorithmthat aims to minimize the expected prediction set size [20, 3]; Algorithm 1 relies on this property,thus making the meta prediction set size small.
4.3 Theory
The proposed Algorithm 1 outputs a threshold for a prediction set that satisﬁes the mPAC guarantee,as stated in the following result (see Appendix C for a proof): [Equation] S ) ⇤  1  ↵ / 2 with
at
to
Theorem 1.
The estimator ˆ ⌧ " , ↵ ,  implemented in Algorithm 1 is ( " , ↵ ,  ) -mPAC for p ⇡ .
Intuitively, the algorithm implicitly constructs an interval [ˆ ⌧ ↵ / 2 ,  ( S 0 ) , 1 ) over scalar parameters ofprediction sets.
In particular, for the i -th distribution, the Algorithm in Line 3 ﬁnds a scalar parameterof a prediction set for the i -th distribution, which forms an empirical distribution over the scalarparameters.
Due to the i.i.d.
nature of ✓ i , i = 1 , . .
, N , the parameter of a prediction set for a testdistribution follows the same distribution over scalar parameters.
Thus, choosing a conservativescalar parameter as in Line 5 of the Algorithm sufﬁces to satisfy the mPAC guarantee.
Experiments
We demonstrate the efﬁcacy of the proposed meta PAC prediction set based on four few-shot learningdatasets across three application domains: mini-ImageNet [10] and CIFAR10-C [15] for the visualdomain, FewRel [16] for the language domain, and CDC Heart Dataset [17] for the medical domain.
See Appendix E for additional experiments, including the results for CIFAR10-C.
5.1 Experimental Setup
Few-shot learning setup.
We consider k -shot c -way learning except for the CDC Heart Dataset; Inparticular, there are c classes for each task dataset, and k adaptation examples for each class.
Thus,we have t :=
kc labeled examples to adapt a model to a new task.
As the CDC Heart dataset is alabel unbalanced dataset, we do not assume equal shots per class.
Score functions.
We use a prototypical network [11] as a score function.
It uses t adaptationexamples to adapt the network to a given task dataset, and predicts labels on query examples.
Baselines.
We compare four approaches, including our proposed approach.
• PS : We consider a naive application of PAC prediction sets [3].
We pool all nN labeled examples
from calibration datasets into one calibration set to run the PAC prediction set algorithm.
• PS-Test : We construct PAC prediction sets by using 20 shots for each class directly drawn fromthe test distribution.
• Meta-CP : We use the PAC variant of meta conformal prediction [14], which trains a quantilepredictor along with a score function, while our approach only needs a score function.
We run theauthors’ code with the same evaluation parameters for comparison.•
Meta-PS : This is the proposed meta PAC prediction set from Algorithm 1.
Metric.
We evaluate prediction sets mainly via their empirical prediction set error over a test sample,i.e., having learned a parameter ⌧ , we ﬁnd 1 / | E |
P ( x,y ) 2 E ( y / 2 F A ⌧ ( x )) , where A is an adaptationset, and E is an evaluation set drawn from each test task distribution.
We desire this to be at most "given ﬁxed calibration and test datasets.
This should hold with probability at least 1   over therandomness due to the calibration and adaptation data; and with probability at least 1  ↵ over therandomness due to the test adaptation set.
To evaluate this, we conduct 100 random trials, drawingindependent calibration datasets, and for each ﬁxed calibration dataset, we conduct 50 random trialsdrawing independent test datasets.
The prediction set size evaluation is done similarly to the error;we compute the empirical prediction set size 1 / | E |
P ( x,y ) 2 E S ( F A ⌧ ( x )) , where S is a size measure,e.g., set cardinality for classiﬁcation.
5.2 Mini-ImageNet
Mini-ImageNet [10] is a smaller version of the ImageNet dataset for few-shot learning.
Setup.
Mini-ImageNet consists of 100 classes with 64 classes for training, 16 classes for calibration,and 20 classes for testing; each class has 600 images.
Following a standard setup, we consider ﬁverandomly chosen classes as one task and 5 -way classiﬁcation.
In training, we follows 5 -shot 5 -waylearning, considering M = 800 training task distribution randomly drawn from the   possibletasks.
In calibration, we have N = 500 calibration task datasets randomly drawn from the   possible tasks, and use 5 shots for adaptation and 500 shots for calibration ( i.e., t = 25 and n = 2500 ).
In evaluation, we have 50 test task datasets randomly drawn from the   possible tasks and use 5shots from each of 5 classes for adaptation and 100 shots from each 5 classes for evaluation. [Figure] are N = 500 , n = 2500 , t = " =
↵ =
= = 5 for methods. Results.
Figure 3 shows the prediction set error and size of the various approaches in box plots.
Whiskers in each box plot for the error cover the range between the 10 -th and 90 -th percentilesof the empirical error distribution, while whiskers in each box plot for the size represents theminimum and maximum of the empirical size distribution.
The randomness of error and size is dueto the randomness of the augmented calibration ( S 1: N , A 1: N ) and a test adaptation set A ; but since
=
10  5 , the randomness is mostly due to the adaptation set.
The prediction set error of the proposed approach is below the desired level " = 0 .
1 at least a fraction1  ↵ = 0 .
9 of the time.
This supports that the proposed prediction set satisﬁes the meta PACcriterion.
However, PS and Meta-CP do not empirically satisfy the meta PAC criterion, and PS-Testis too conservative.
In particular, Meta-CP does not empirically control the randomness due to thetest distribution.
To make this point precise, Figure 3c shows a prediction set error distribution overdifferent test distributions given a ﬁxed calibration dataset.
As can be seen, the proposed Meta-PSapproach empirically satisﬁes the " error criterion at least a fraction 1  ↵ = 0 .
9 of the time (asshown in the whiskers), but Meta-CP does not empirically control it.
Figure 3b demonstrates thatthe proposed Meta-PS produces smaller prediction sets compared to the PS-Test approach that alsosatisﬁes the " error constraint.
5.3 FewRel
FewRel [16] is an entity relation dataset, where each example is the tuple of the ﬁrst and secondentities in a sentence along with their relation as a label.
Setup.
FewRel consists of 100 classes, where 64 classes are used for training and 16 classes areoriginally used validation; each class has 700 examples.
The detailed setup for training, calibration,and test are the same as those of mini-ImageNet, except that the test task distributions are randomlydrawn from the original validation split.
Results.
Figure 4 presents the prediction set error and size of the approaches.
The trend of the resultsis similar to the mini-ImageNet result.
Ours empirically satisﬁes the desired prediction set errorspeciﬁed by " with empirical probabilities of at least 1  ↵ over the test adaptation set and 1   overthe (augmented) calibration data.
In contrast, the other approaches either do not empirically satisfythe meta PAC criterion or are highly conservative.
Figure 8a also conﬁrms that the proposed approachempirically controls the error with probability at least 1  ↵ over the randomness in the test dataset. [Figure] [Figure]
5.4 CDC Heart Dataset
The CDC Behavioral Risk Factor Surveillance System [17] consists of health-related telephonesurveys started in 1984.
The survey data is collected from US residents.
We use the data collectedfrom 2011 to 2020 and consider predicting the risk of heart attack given other features ( e.g., level ofexercise or weight).
We call this curated dataset the CDC Heart Dataset.
See Appendix D.4 for thedetails of the experiment setup.
Setup.
The 10 years of the CDC Heart Dataset consists of 530 task distributions from various USstates.
We consider data from 2011-2014 as the training task distributions, consisting of 212 differenttasks and 1,805,073 labeled examples, and data from 2015-2019 as calibration task distributions,consisting of 265 different tasks and 2,045,981 labeled examples. We use the data from 2020 as a testtask distribution, consisting of 53 different tasks and 367,511 labeled examples.
We use M = 200 ,
N = 250 , n = 3000 , and t = 10 ; see Appendix D.4 for the details.
Results.
Figure 5 includes the prediction set error and size of each approach.
The trend is as before.
The proposed Meta-PS empirically satisﬁes
the " = 0 .
1 constraint ( i.e., the top of the whisker isbelow of the " = 0 . 1 line), while PS and Meta-CP violate this constraint.
Meanwhile, PS-Testconservatively satisﬁes it.
As before, Figure 8b empirically justiﬁes that the proposed approachcontrols the prediction set error due to the randomness over the test adaptation data.
Conclusion
We propose a PAC prediction set algorithm for meta learning, which satisﬁes a PAC property tailoredto a meta learning setup.
The efﬁcacy of the proposed algorithm is demonstrated in four datasetsacross three application domains.
Speciﬁcally, we observe that the proposed algorithm ﬁnds aprediction set that satisﬁes a speciﬁed PAC guarantee, while producing a small set size.
A limitationof the current approach is that it requires enough calibration datapoints to satisfy the PAC guarantee,see [3] for an analysis.
A potential societal impact is that a user of the proposed algorithm mightmisuse the guarantee without a proper understanding of assumptions; see Appendix F for details.
Checklist
For all authors...(
a)
Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Section 6.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] SeeSection
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] See Section4.1.
(b) Did you include complete proofs of all theoretical results?
[Yes] See Section C.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[No] Code will be
released once accepted along with data and precise instructions to run it.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they
were chosen)?
[Yes] See Section 5 and D.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)?
[Yes] We use box plots to represent the randomness on
error.
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See Section D.1.
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[Yes] See Section D.2.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[Yes] See Section D.2.
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[Yes] See Section D.2.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]