Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction
Abstract
Conventional frequentist learning is known to yield poorly calibrated models that fail to reliably quantify the uncertainty of their decisions.
Bayesian learning can improve calibration, but formal guarantees apply only under restrictive assumptions about correct model speciﬁcation.
Conformal prediction (CP) offers a general framework for the design of set predictors with calibration guarantees that hold regardless of the underlying data generation mechanism.
However, when training data are limited, CP tends to produce large, and hence uninformative, predicted sets.
This paper introduces a novel meta-learning solution that aims at reducing the set prediction size.
Unlike prior work, the proposed meta-learning scheme, referred to as meta-XB, (i) builds on cross-validation-based CP, rather than the less efﬁcient validation-based CP; and (ii) preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees.
Introduction 1.1 Context and Motivation
Recent work on calibration for AI has focused on Bayesian learning, or related ensembling methods, as means to quantify epistemic uncertainty
[Finn et al., 2018, Yoon et al., 2018, Ravi and Beatson, 2018, Jose et al., 2022].
However, recent studies have shown the limitations of Bayesian learning when the assumed model likelihood or prior distribution are misspeciﬁed [Masegosa, 2020].
Furthermore, exact Bayesian learning is computationally infeasible, calling for approximations such as Monte Carlo (MC) sampling
[Robert et al., 1999] and variational inference (VI) [Blundell et al., 2015].
Overall, under practical conditions, Bayesian learning does not provide formal guarantees of calibration.
Conformal prediction (CP) [Vovk et al., 2005] provides a general framework for the calibration of (frequentist or Bayesian) probabilistic models.
The formal calibration guarantees provided by CP hold irrespective of the (unknown) data distribution, as long as the available data samples and the test samples are exchangeable – a weaker requirement than the standard i.i.d.
assumption.
As illustrated in Fig. 1, CP produces set predictors that output a subset of the output space Y for each input x , with the property that the set contains the true output value with probability no smaller than a desired value 1  ↵ for ↵ 2 [0 , 1] .
Mathematically, for a given learning task ⌧ , assume that we are given a data set D ⌧ with N ⌧ samples, i.e., D ⌧ = { z ⌧ [ i ] } N ⌧ i =1 , where the i th sample z ⌧ [ i ] = ( x ⌧ [ i ] , y ⌧ [ i ]) contains input x ⌧ [ i ] 2 X ⌧ and target y ⌧ [ i ] 2 Y ⌧ .
CP provides a set predictor  ( ·|D ⌧ , ⇠ ) :
X ⌧ !
2 Y ⌧ , speciﬁed by a hyperparameter vector ⇠ , that maps an input x ⌧ 2 X ⌧ to a subset of the output domain Y ⌧ based on a data set D ⌧ .
FM 16PSK 16QAM with abbreviation of p ( y | x, ˜ D , ⇠ ) :=
E   p (  | ˜ D , ⇠ ) p ( y | x,  ) and for any monotonically strictly increasing function f : R !
R .
We now propose a differentiable adaptive NC score ˆ NC ada ( z | ˜ D , ⇠ ) byreplacing the indicator function 1 ( · ) with the sigmoid function  c ( · )ˆ NC ada ( z | ˜ D , ⇠ ) =
X y 0 2 Y  c
f ( p ( y 0 | x, ˜ D , ⇠ ))  f ( p ( y | x, ˜ D , ⇠ ))
p ( y
| x, ˜ D , ⇠ ) .
For the function f , we choose soft ranking operator R : R N !
R N deﬁned in [Cuturi et al., 2019] aswe found that adopting soft ranking empirically shows better input conditional coverage compared toidentity mapping f ( p )
= p .
Meta-XB optimize hyperparameter vector ⇠ by addressing the problem⇠ ⇤ = argmin
⇠
T T
X t =
M t M t
X j =
ˆ  K-XB ↵ ( x jt |D jt , ⇠ )  , (26) with   ˆ  K-XB ↵ ( x jt |D jt , ⇠ )   in (22).
Theorem 3 For any ﬁxed meta-testing task ⌧ , ﬁxed T meta-training tasks ⌧ 1 , . .
. , ⌧ T , and ﬁxedmeta training data set
D ⌧ 1 : ⌧ T , meta-XB satisﬁes the per-task marginal validity condition in (1) .[Proof
sketch]
Since the optimization of ⇠ in (26) does not depend on meta-test task ⌧ , it does notaffect exchangeability condition of the joint distribution p (
D ⌧ , z ⌧ ) .
Full proof can be found insupplementary material.
As mentioned in Sec. 1, both [Fisch et al., 2021] and [Park et al., 2022] cannot achieve the per-taskvalidity condition in (1) to require additional marginalization [Fisch et al., 2021] or achieving looser
PAC-bounds [Park et al., 2022].
The optimization of (26) is handled via iterative gradient-based updates as⇠  ⇠    ⇠
T
T
X t =
M t M t
X j =
ˆ  K-XB ↵ ( x jt |D jt , ⇠ )  , (27) assuming gradient descent (GD).
In practice, batch of tasks and corresponding examples are sampledto speed up the training.
Moreover, in practice, as the meta-training data set for each task t isgenerally given as a single large data set
D t :=
{ z [ i ] } N mtr i =1 with N mtr  N i.i.d.
examples, werandomly sample N + 1 examples from the set
D t to form a single realization of the pair ( D jt , z jt ) .
Lastly, to further speed up the training process, we replace single realization of the empiricalinefﬁciency | ˆ  K-XB ↵ ( x jt |D jt , ⇠ ) | with averaged empirical inefﬁciency 1 L
Ll =1 | ˆ  K-XB ↵ ( x j,lt |D jt , ⇠ ) | over
L test inputs x j, 1: L t := { x j, 1 t , . . .
, x j,Lt } sampled in i.i.d.
fashion from the total data set
D t .
Detailscan be found in Algorithm 1.
Experiments
4. Toy (GAM)
1.3 Related Work
[Figure]
[Table]
[Figure]
[Table]
This degradation becomes more severe when the prior is also misspeciﬁed [Wenzel et al., 2020] which brings the concept of cold posterior
[Wenzel et al., 2020, Noci et al., 2021] that downweights the effect of distribution.
In terms of mitigating the model likelihood misspeciﬁcation, Masegosa [2020] considered second-order PAC-Bayes bound; while Morningstar et al.
[2022] proposed multi-sample ensembling loss for training the model.
In terms of misspeciﬁcation in prior distribution, Bayesian meta-learning, which learns the prior from the available data, has been considered [Simchowitz et al.,
[Table] [Table]
Bayesian
[Table]
While frequentist meta-learning has shown remarkable success in few-shot
[Finn et al., 2017, Snell et al., 2017], attempts to quantify uncertainty from few training
[Figure]
samples have been raised extensively in terms of Bayesian meta-learning [Amit and Meir, 2018, Finn
[Figure]
et al., 2018, Yoon et al., 2018, Ravi and Beatson, 2018, Nguyen et al., 2020, Jose et al., 2022].
While
Bayesian meta-learning learns a prior distribution by ﬁnding hyper-posterior from multiple tasks
Amit and Meir [2018], Rothfuss et al.
[2020], Jose et al.
[2022], misspeciﬁcatin in hyper-prior is inevitable, though Rothfuss et al.
[2021] mitigated this problem via treating hyper-prior as stochastic
[Figure]
processes
[Figure]
CP-aware loss.
As introduced in Sec. 1, CP provides a formal calibration guarantee regardless of any misspeciﬁcation reside in the assumed model.
Since validity (1) holds regardless of any
[Figure]
choice of ﬁtting algorithm, Stutz et al.
[2021], Einbinder et al.
[2022] proposed a particularly design
[Figure]
loss function that aids VB-CP.
While CP-aware losses improve the performance either with better
[Figure]
efﬁciency [Stutz et al., 2021] or enhanced per-input validity (3)[Einbinder et al., 2022], they assumed
[Figure]
sufﬁcient number
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table] [Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table]
[Table]
[Table] [Table]
QPSK vs. p  a [1]
]  = p  a [ a
(4)
Assumption 1 We assume that the data set D ⌧ and a test data point z ⌧ are exchangeable random 77 variables.
Accordingly, the joint distribution p (
D ⌧ ,z ⌧ ) can be written as [De Finetti, 1929, Hewitt 78 and Savage, 1955]
[Equation]
where p ( y ⌧ | x ⌧ ,  ) is the likelihood of random variable y ⌧ given model parameter vector  and input 80 x ⌧ ; p ( x ⌧ |  ) being the likelihood of random variable x ⌧ given parameter vector  ; and p (  ) is a 81 prior distribution over the random vector  .
CP measures the conformity via NC score , generally deﬁned as follows.
Deﬁnition
Given data set ˜ D ⌧ = { ˜ z
⌧ [ i ] =
(˜ x ⌧ [ i ] , ˜ y ⌧ [ i ]) } ˜ Ni =1 ✓ D ⌧ with ˜ N  N samples ˜
z [ i ]
D ⌧ , a nonconformity (NC) score is a function NC ( z | ˜ D ⌧ , ⇠ ) that maps the data set ˜ D ⌧ and any 85 input-output pair z
= ( x ⌧ ,y ) to a real number.
The NC score generally depends on a vector ⇠ ofhyperparameters, and it must satisfy the permutation-invariance property
[Equation]
) for any permutation operator ⇡ ( · ) .
A good NC score should express how poorly the point ( x ⌧ ,y ) “conforms” to the data set ˜ D ⌧ .
The 89 most common way to obtain an NC score is via a parametric two-step approach.
Deﬁnition 2 Let  ( z |  ) represent the loss of a machine learning model parametrized by vector  91 and the input-output pair z
=
( x ⌧ ,y ) .
We will speciﬁcally consider the standard log-loss  ( z |  ) =
log p ( y | x ⌧ ,  ) .
Let p (  | ˜ D
⌧ , ⇠ ) be the probability distribution of the model parameter vector  93 obtained using data set ˜
D ⌧ via some training algorithm determined by hyperparameter vector ⇠ .
A 94 conventional NC score function is deﬁned as
NC ( z | ˜ D ⌧ , ⇠ ) :=
E   p (  | ˜ D ⌧ , ⇠ )   ( z |  )  .
(7) • We incorporate adaptive NC scores [Romano et al., 2020] in the design of meta-XB, demonstrating via experiments that adaptive NC scores can enhance conditional validity as deﬁnedby condition (3).
64 task ⌧ 1 65 task ⌧ 2 66 task ⌧ 3 67 task ⌧ 4 68 2 Preliminaries and Deﬁnitions
[Table]
Roughly speaking, CP provides a set prediction for input x ⌧ by selecting outputs y 2 Y ⌧ only 71 if the corresponding candidate pair ( x ⌧ ,y ) conforms well with the examples in the available dataset that are exchangeable with the (unknown) ground-truth test point ( x ⌧ ,y ⌧ ) .
More precisely, 73 a set ( a [1] ,...,a [ M ]) of random variables is said to be exchangeable if the joint distributionp ( a [1] ,...,a [ M ]) is invariant to the ordering of the M samples, i.e.,p  a [1] ,...,a [ M ]  = p  a [ ⇡ (1)] ,...,a [ ⇡ ( M )]  (4) for any permutation operator ⇡ ( · ) .
Assumption 1 We assume that the data set D ⌧ and a test data point z ⌧ are exchangeable random 77 variables.
Accordingly, the joint distribution p (
D ⌧ ,z ⌧ ) can be written as [De Finetti, 1929, Hewitt 78 and Savage, 1955]p ( D ⌧ ,z ⌧ ) =
N
i
[Equation]
where p ( y ⌧ | x ⌧ ,  ) is the likelihood of random variable y ⌧ given model parameter vector  and input 80 x ⌧ ; p ( x ⌧ |  ) being the likelihood of random variable x ⌧ given parameter vector  ; and p (  ) is a 81 prior distribution over the random vector  .
CP measures the conformity via NC score , generally deﬁned as follows.
Deﬁnition
Given data set
[Equation]
with
[Equation]
N samples
[Equation]
D ⌧ , a nonconformity (NC) score is a function NC ( z | ˜ D ⌧ , ⇠ ) that maps the data set ˜ D ⌧ and any 85 input-output pair z
= ( x ⌧ ,y ) to a real number.
The NC score generally depends on a vector ⇠ ofhyperparameters, and it must satisfy the permutation-invariance property
NC
[Equation]
for any permutation operator ⇡
[Equation] [Equation]
A good NC score should express how poorly the point ( x ⌧ ,y ) “conforms” to the data set ˜ D ⌧ .
The 89 most common way to obtain an NC score is via a parametric two-step approach.
Deﬁnition 2 Let  ( z |  ) represent the loss of a machine learning model parametrized by vector  91 and the input-output pair z
=
( x ⌧ ,y ) .
We will speciﬁcally consider the standard log-loss  ( z |  ) =
log p ( y | x ⌧ ,  ) .
Let p (  | ˜ D
⌧ , ⇠ ) be the probability distribution of the
[Table]
NC ( z | ˜ D ⌧ , ⇠ ) :=
E   p (  | ˜ D ⌧ , ⇠
[Table]
( z |  )  .
(7) task task
XB-CP … … … … … … … …
OQPSK 8PSK 1.
Main Contributions
In this paper, we introduce a novel meta-learning approach that builds on XB-CP, targeting a lowerinefﬁciency (2) with respect to meta-VB [Fisch et al., 2021]; while preserving the strict per-taskvalidity condition (1) for every task ⌧ .
Furthermore, we incorporate in the design of the proposed meta-XB algorithm the adaptive nonconformity (NC) scores introduced in [Romano et al., 2020].
As argued in [Romano et al., 2020]for conventional CP, adaptive NC scores are empirically known to improve the per-task conditionalvalidity condition
P ( y ⌧ 2  ( x ⌧ | D ⌧ , ⇠ ) | x ⌧ = x ⌧ )  1  ↵ .
This condition is signiﬁcantly stronger than (1) as it holds for any test input x ⌧ .
D ⌧  ( x 1 ⌧ |D ⌧ , ⇠ )  ( x 2 ⌧ |D ⌧ , ⇠ )  ( x 3 ⌧ |D ⌧ , ⇠ )
The contribution of this work can be summarized as follows:
• We introduce a meta-learning algorithm for XB-CP, referred to as meta-XB, that can reduce
the average prediction set size (2) as compared to existing solutions, while satisfying the
per-task validity condition (1), unlike existing meta-learning algorithms for CP;
• We incorporate adaptive NC scores [Romano et al., 2020] in the design of meta-XB, de.. viaexperiments that adaptive NC scores can enhance conditional validity as per condition(3).
1. Related Work
Model misspeciﬁcation.
When the model is misspeciﬁced, i.e., assumed model likelihood cannotexpress the ground-truth data generating distribution [Masegosa, 2020], Bayesian learning may yieldpoor generalization performance [Masegosa, 2020, Morningstar et al., 2022].
This degradationbecomes more severe when the prior is also misspeciﬁed [Wenzel et al., 2020] which brings theconcept of cold posterior
[Wenzel et al., 2020, Noci et al., 2021] that downweights the effect ofprior distribution.
In terms of mitigating the model likelihood misspeciﬁcation, Masegosa [2020]considered second-order PAC-Bayes bound; while Morningstar et al.
[2022] proposed multi-sampleensembling loss for training the model.
In terms of misspeciﬁcation in prior distribution, Bayesianmeta-learning, which learns the prior from the available data, has been considered [Simchowitz et al.,2021].
Bayesian meta-learning While frequentist meta-learning has shown remarkable success in few-shotlearning tasks
[Finn et al., 2017, Snell et al., 2017], attempts to quantify uncertainty from few trainingsamples have been raised extensively in terms of Bayesian meta-learning [Amit and Meir, 2018, Finnet al., 2018, Yoon et al., 2018, Ravi and Beatson, 2018, Nguyen et al., 2020, Jose et al., 2022].
While
Bayesian meta-learning learns a prior distribution by ﬁnding hyper-posterior from multiple tasks
Amit and Meir [2018], Rothfuss et al.
[2020], Jose et al.
[2022], misspeciﬁcatin in hyper-prior isinevitable,
though Rothfuss et al.
[2021] mitigated this problem via treating hyper-prior as stochasticprocesses.
CP-aware loss.
As introduced in Sec. 1, CP provides a formal calibration guarantee regardlessof any misspeciﬁcation reside in the assumed model.
Since validity (1) holds regardless of anychoice of ﬁtting algorithm, Stutz et al.
[2021], Einbinder et al.
[2022] proposed a particularly designloss function that aids VB-CP.
While CP-aware losses improve the performance either with betterefﬁciency [Stutz et al., 2021] or enhanced per-input validity (3)
[Einbinder et al., 2022], they assumedsufﬁcient number of data samples, i.e., N  1 .
Meta-learning on CP.
Unlike the works in CP-awareloss that reduces the average prediction set size by addressing a new loss function [Stutz et al., 2021],
Fisch et al.
[2021], Park et al.
[2022] modiﬁed the structure of VB-CP to lose strict validity condition(1) while to improve the efﬁciency of the set predictors in the presence of limited data samples.
Instead of condition (1), Fisch et al.
[2021] achieved looser validity condition that requires averagingover the task distribution [Fisch et al., 2021]; while [Park et al., 2022] showed that validity condition(1) holds with high probability over random tasks and random data sets.
task vs. … … … …
Calibration amounts to the per-task validity condition
[Equation]
which indicates that the set predictor  ( x ⌧ | D ⌧ , ⇠ ) contains the true target y ⌧ with probability at least ↵ .
In (1), the probability P ( · ) is taken over the ground-truth, exchangeable, joint distribution p ( D ⌧ , z ⌧ ) , and bold letters represent random variables.
The most common form of CP, referred to as validation-based CP (VB-CP), splits the data set into training and validation subsets [Vovk et al., 2005].
The validation subset is used to calibrate the set prediction  VB ↵ ( x ⌧ |D ⌧ , ⇠ ) on a test example x ⌧ for a given desired miscoverage level ↵ in (1).
The drawback of this approach is that validation data is not used for training, resulting in inefﬁcient set predictors  VB ↵ ( x ⌧ |D ⌧ , ⇠ ) in the presence of a limited number N ⌧ of data samples.
The average size of a set predictor  ( x ⌧ |D ⌧ , ⇠ ) , referred to as inefﬁciency , is deﬁned as
[Equation]
where the average is taken with respect to the ground-truth joint distribution p ( D ⌧ , z ⌧ ) .
A more efﬁcient CP set predictor was introduced by Barber et al.
[2021] based on cross-validation.
The cross-validation-based CP (XB-CP) set predictor  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) splits the data set D ⌧ into K folds to effectively use the available data for both training and calibration.
XB-CP can also satisfy the per-task validity condition (1) 1 .
Further improvements in efﬁciency can be obtained via meta-learning [Thrun, 1998].
Meta-learning jointly processes data from multiple learning tasks, say ⌧ 1 , . .
. , ⌧ T , which are assumed to be drawn i.i.d.
from a task distribution p ( ⌧ ) .
These data are used to optimize the hyperparameter ⇠ of the set predictor  ( x ⌧ | D ⌧ , ⇠ ) to be used on a new task ⌧ ⇠ p ( ⌧ ) .
Speciﬁcally, reference [Fisch et al., 2021] introduced a meta-learning-based method that modiﬁes VB-CP.
The resulting meta-VB algorithm satisﬁes a looser validity condition with respect to the per-task inequality (1), in which the probability in (1) is no smaller than 1  ↵ only on average with respect to the task distribution p ( ⌧ ) .
In this paper, we introduce a novel meta-learning approach, termed meta-XB , with the aim of reducing the inefﬁciency (2) of XB-CP, while preserving, unlike [Fisch et al., 2021], the per-task validity condition (1) for every task ⌧ .  Meta-Learning Algorithm for XB-CP (Meta-XB)
In this section, we brieﬂy introduce the proposed meta-XB algorithm.
We start by describing the meta-learning framework.
2.1 Meta-Learning
Meta-learning utilizes data from multiple tasks to enhance the efﬁciency of the learning procedure for new tasks.
Following the standard meta-learning formulation [Baxter, 2000, Amit and Meir, 2018], as anticipated in Section 1, the learning environment is characterized by a task distribution p ( ⌧ ) over the task identiﬁer ⌧⌧⌧ .
Given T meta-training tasks realizations ⌧⌧⌧ 1 = ⌧ 1 , . . .
, ⌧⌧⌧ T = ⌧ T drawn i.i.d.
from the task distribution p ( ⌧ ) , the meta-training data set D ⌧ 1: T :=
{{D jt , z jt } M t j =1 } Tt = consists of M t realizations {D jt , z jt } M t j =1 of data sets D j ⌧ t = D jt with N ⌧ t = N t examples and test sample z j ⌧ t =
z jt for each task ⌧ t .
Pairs {D jt , z jt } M t j =1 are generated i.i.d.
from the joint distribution p ( D ⌧ t , z ⌧ t ) , satisfying exchangeability assumption for all tasks t .
The goal of meta-learning for CP is to optimize the vector of hyperparameter ⇠ based on the metatraining data D ⌧ 1: T , so as to obtain a more efﬁcient set predictor  ( x ⌧ |D ⌧ , ⇠ ) .
While reference [Fisch et al., 2021] proposed a meta-learning solution for VB-CP [Vovk et al., 2005], here we introduce a meta-learning method for XB-CP.
Cross-Validation-Based Conformal Prediction (XB-CP)
XB-CP leverages K -fold cross-validation [Barber et al., 2021].
K -fold cross-validation partitions the per-task data set D ⌧ = { z ⌧ [ i ] } N ⌧ i =1 into K  2 disjoint subsets D ⌧ , 1 , . . .
, D ⌧ ,K such that the condition
S
Kk =1 D ⌧ ,k = D ⌧ is satisﬁed.
We deﬁne the leave-one-out data set D ⌧ , ¬ k =
S
Kk
=1 ,k 0 6 = k D ⌧ ,k 0 that excludes the subset
D ⌧ ,k .
We also introduce a mapping function k : { 1 , . .
. , N ⌧ } !
, K } to identify the subset D ⌧ ,k ( i ) that includes the sample z ⌧ [ i ] , i.e., z ⌧ [ i ] 2 D ⌧ ,k ( i ) .
A nonconformity (NC) score is a function NC ( z | ˜ D ⌧ , ⇠ ) that maps a data set ˜ D ⌧ and any input-output pair z = ( x, y ) with x 2 X ⌧ and y 2 Y ⌧ to a real number while satisfying the permutation-invariance property NC ( z |{ ˜ z
⌧ [1] , . . . ,
˜ z ⌧ [ ˜ N ] } , ⇠ ) =
NC ( z |{ ˜ z
⌧ [ ⇡ (1)] , . . . , ˜ z ⌧ [ ⇡ ( ˜ N )] } , ⇠ ) for any permutation operator ⇡ ( · ) .
A good NC score should express how poorly the point ( x ⌧ , y ) “conforms” to the data set ˜ D ⌧ .
The most common way to obtain an NC score is via a parametric two-step approach.
This involves a training algorithm deﬁned by a conditional distribution p (  | ˜ D ⌧ , ⇠ ) , which describes the output  of the algorithm as a function of training data set ˜ D
⌧ ✓ D ⌧ and hyperparameter vector ⇠ .
Given a test input x ⌧ , XB-CP computes the NC score for a candidate pair z
= ( x ⌧ , y ) with y 2 Y ⌧ by taking the minimum NC score NC ( z |D ⌧ , ¬ k , ⇠ ) over all possible subsets k 2 { 1 , . .
, K } , i.e., as min k 2 { 1 ,...,K } NC ( z |D ⌧ , ¬ k , ⇠ ) .
Furthermore, for each data point z ⌧ [ i ] 2 D ⌧ , the NC score is evaluated by excluding the subset D ⌧ ,k ( i ) as NC ( z ⌧ [ i ] |D ⌧ , ¬ k ( i ) , ⇠ ) .
Note that evaluating the resulting
N ⌧ + 1 NC scores requires running the training algorithm p
(  |D ⌧ , ¬ k , ⇠ ) K times, once for each subset D ⌧ , ¬ k .
Finally, a candidate y 2 Y ⌧ is included in the prediction set  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) if the NC score for z = ( x ⌧ , y ) is smaller (or equal) than for a fraction (at least)
b ↵ 0 ( N ⌧ + 1) c /N ⌧ of the validation data points with ↵ 0 = ↵  1  K/N ⌧ K
+1 .
We refer to supplementary material for details.
2.3 Meta-XB
Meta-XB aims at ﬁnding a hyperparameter vector ⇠ that minimizes the average size of the prediction set  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) for tasks ⌧ that follow the distribution p ( ⌧ ) .
To this end, it addresses the problem of minimizing the empirical average of the sizes of the prediction sets  K -XB ↵ ( x ⌧ t |D ⌧ t , ⇠ ) across the meta-training tasks ⌧ 1 , . . .
, ⌧ T over the hyperparameter vector ⇠ .
This amounts to the optimization
[Equation]
where the ﬁrst sum is over the meta-training tasks and the second is over the available data for each task.
Since it involves hard comparisons among NC scores, the size |  K -XB ↵ ( x |D , ⇠ ) | is not a differentiable function of the hyperparameter vector ⇠ .
Therefore, in order to address (3) via gradient descent, we introduce a differentiable soft inefﬁciency criterion by replacing the indicator function with the sigmoid  ( u ) =
(1 + exp
(  u/c  ))
1 for some c  > 0 ; the quantile Q 1  ↵ ( · ) with a differentiable soft empirical quantile ˆ Q 1  ↵ ( · ) constructed via the pinball loss [Koenker and
Bassett Jr, 1978]; and the minimum operator with the softmin function [Goodfellow et al., 2016,
Cuturi et al., 2019].
Details can be found in the supplementary material.
Experiments
In this section, we provide experimental results to validate the performance of meta-XB in terms of (i) per-task coverage P ( y ⌧ 2  ( x ⌧ | D ⌧ , ⇠ )) ; and (ii) per-task inefﬁciency (2).
As benchmark schemes, we consider (i) VB-CP, (ii) XB-CP, and (iii) meta-VB [Fisch et al., 2021].
We refer to the supplementary material for other details and additional experiments including mini Imagenet classiﬁcation [Vinyals et al., 2016] and modulation classiﬁcation [O’Shea et al., 2018].
3.1 Multinomial Model and Inhomogeneous Features
We consider here the synthetic-data experiment introduced in [Romano et al., 2020].
In Fig.
2, we demonstrate the performance of the considered set predictors as a function of number of tasks T .
Both meta-VB and meta-XB achieve lower inefﬁciency (2) as compared to the conventional set predictors VB-CP and XB-CP, as soon as the number of meta-training tasks is sufﬁciently large to ensure successful generalization across tasks [Yin et al., 2019, Jose and Simeone, 2020].
For example, meta-XB with T = 100 tasks obtain an average prediction set size of 3 , while XB-CP has an inefﬁciency larger than 4 .
Furthermore, all schemes satisfy the validity condition (1), except for meta-VB for T & 10 4 , conﬁrming the analytical results.
Conclusion
This paper has introduced meta-XB, a meta-learning solution for cross-validation-based conformal prediction that aims at reducing the average prediction set size, while formally guaranteeing per-task calibration.
The approach is based on the use of soft quantiles.
Through experimental results, meta-XB was shown to outperform both conventional conformal prediction-based solutions and metalearning conformal prediction schemes.
Future work may integrate meta-learning with CP-aware training criteria [Stutz et al., 2021, Einbinder et al., 2022], or with stochastic set predictors.
Acknowledgments
The work of S. Park, K. M. Cohen, and O. Simeone has been supported by the European Research
Council (ERC) under the European Union’s Horizon 2020 research and innovation programme, grant agreement No. 725731.
The work of O. Simeone has also been supported by an Open Fellowship of the EPSRC.
Main Contributions
As summarized in the main document, this work introduces a novel meta-learning approach, termedmeta-XB , with the aim of reducing the inefﬁciency (2) of XB-CP, while preserving the per-taskvalidity condition (1) for every task ⌧ unlike [Fisch et al., 2021].
In this supplementary material, weprovide full details, and we also incorporate in the design of meta-XB the adaptive nonconformity(NC) scores introduced in [Romano et al., 2020].
As argued in [Romano et al., 2020] for conventional
CP, adaptive NC scores are empirically known to improve the per-task conditional validity condition
[Equation]
This condition is signiﬁcantly stronger than (1) as it holds for any test input x ⌧ .
A summary of theconsidered CP schemes can be found in Fig.
Overall, the contribution of this work, including also the supplementary material can be summarizedas follows:
• We introduce meta-XB, a meta-learning algorithm for XB-CP, that can reduce the averageprediction set size (2) as compared to XB-CP, while satisfying the per-task validity condition(1), unlike existing meta-learning algorithms for CP;• We incorporate adaptive NC scores [Romano et al., 2020] in the design of meta-XB, demonstrating via experiments that adaptive NC scores can enhance conditional validity as deﬁnedby condition (4).
Deﬁnitions and Preliminaries
In this section, we describe necessary background material on CP [Vovk et al., 2005, Balasubramanianet al., 2014], VB-CP [Vovk et al., 2005], XB-CP
[Barber et al., 2021], and adaptive NC scores[Romano et al., 2020].
2.1 Nonconformity (NC) Scores
At a high level, given an input x ⌧ for some learning task ⌧ , CP outputs a prediction set  ( x ⌧ |D ⌧ , ⇠ )that includes all outputs y 2 Y ⌧ such that the pair ( x ⌧ , y ) conforms well with the examples in theavailable data set D ⌧ = { z ⌧ [ i ] = ( x ⌧ [ i ] , y ⌧ [ i ]) } N ⌧ i =1 .
We recall from Section 1 that ⇠ represents avector of hyperparameter.
The key underlying assumption is that data set D ⌧ and test pair
z ⌧ =( x ⌧ , y ⌧ ) are realizations of exchangeable random variables D ⌧ and z ⌧ .
Assumption 1 For any learning task ⌧ , data set D ⌧ and a test data point z ⌧ are exchangeablerandom variables, i.e., the joint distribution p ( D ⌧ , z ⌧ ) =
p ( z ⌧ [1] , . . .
, z ⌧ [ N ⌧ ] , z ⌧ ) is invariant toany permutation of the variables { z ⌧ [1] , . .
, z ⌧ [ N ⌧ ] , z ⌧ } .
Mathematically, we have the equalityp ( z ⌧ [1] , . . .
, z ⌧ [ N ⌧ + 1]) = p ( z ⌧ [ ⇡ (1)] , . . .
, z ⌧ [ ⇡ ( N ⌧ + 1)]) with z ⌧ = z ⌧ [ N ⌧ + 1] , for any permutation operator ⇡ ( · ) .
Note that the standard assumption of i.i.d.
random variables satisﬁesexchangeability.
CP measures conformity via NC scores , which are generally functions of the hyperparameter vector⇠ , and are deﬁned as follows.
Deﬁnition 1 (NC score) For a given learning task ⌧ , given a data set ˜ D ⌧ = { ˜ z
⌧
[ i ] =
(˜ x ⌧ [ i ] , ˜ y ⌧ [ i ]) } ˜ N ⌧ i =1 ✓ D ⌧ with ˜ N ⌧  N ⌧ samples, a nonconformity (NC) score is a function
NC ( z | ˜ D ⌧ , ⇠ ) that maps the data set ˜ D ⌧ and any input-output pair z
=
( x, y ) with x 2 X ⌧ and y 2 Y ⌧to a real number while satisfying the permutation-invariance property NC ( z |{ ˜ z
⌧ [1] , . . . ,
˜ z ⌧ [ ˜ N ] } , ⇠ ) =
NC ( z |{ ˜ z
⌧ [ ⇡ (1)] , . . . , ˜ z ⌧ [ ⇡ ( ˜ N )] } , ⇠ ) for any permutation operator ⇡ ( · ) .
A good NC score should express how poorly the point ( x ⌧ , y ) “conforms” to the data set ˜ D ⌧ .
Themost common way to obtain an NC score is via a parametric two-step approach.
This involves atraining algorithm deﬁned by a conditional distribution p (  | ˜ D ⌧ , ⇠ ) , which describes the output of the algorithm as a function of training data set
˜ D
⌧ ✓ D ⌧ and hyperparameter vector ⇠ .
Thisdistribution may describe the output of a stochastic optimization algorithm, such as stochastic gradientdescent (SGD), for frequentist learning, or of a Monte Carlo method for Bayesian learning [Guedj,2019, Angelino et al., 2016, Simeone, 2022].
The hyperparameter vector ⇠ may determine, e.g.,learning rate schedule or initialization.
Deﬁnition 2 (Conventional two-step NC score) For a learning task ⌧ , let ` ⌧ ( z |  ) represent the lossof a machine learning model parametrized by vector  on an input-output pair z
=
( x, y ) withx 2 X ⌧ and y 2 Y ⌧ .
Given a training algorithm p
(  | ˜ D ⌧ , ⇠ ) that is invariant to permutation of thetraining set ˜ D ⌧ , a conventional two-step NC score for input-output pair z given data set ˜ D ⌧ is deﬁned as
[Equation]
Due to the permutation-invariance of the training algorithm, it can be readily checked that (5) is avalid NC score as per Deﬁnition 1.
2.2 Validation-Based Conformal Prediction (VB-CP)
VB-CP [Vovk et al., 2005] divides the data set D ⌧ into a training data set D tr ⌧ of N tr ⌧ samples and avalidation data set D val ⌧ of N val ⌧ samples with N tr ⌧ + N val ⌧ = N ⌧ .
It uses the training data set D tr ⌧ toevaluate the NC scores NC ( z |D tr ⌧ , ⇠ ) , while the validation data set D val ⌧ is leveraged to construct theset predictor  VB ↵ ( x ⌧ |D ⌧ , ⇠ ) as detailed next.
Given an input x ⌧ , the prediction set  VB ↵ ( x ⌧ |D ⌧ , ⇠ ) of VB-CP includes all output values y 2 Y ⌧whose NC score NC ( z = ( x ⌧ , y )
|D tr ⌧ , ⇠ ) is smaller than (or equal to) a fraction (at least) b ↵ ( N val ⌧ +1) c /N val ⌧ of the NC scores { NC ( z ⌧ [ i ] |D tr ⌧ , ⇠ ) } N val ⌧ i =1 for validation data points z ⌧
[ i ] 2 D val ⌧ .
Deﬁnition
The (1  ↵ ) -empirical quantile Q 1  ↵ ( { a [ i ] } Mi =1 ) of M real numbers a [1] , . . .
, a [ M ] ,with a [ i ] 2 R , is deﬁned as the ⌃ (1  ↵ )
( M + 1)
⌥ th smallest value in the set { a [1] , . . .
, a [ M ] , 1 } .
With this deﬁnition, the set predictor for VB-CP can be thus expressed as
[Equation]
=
[Equation]
Intuitively, by the exchangeability condition, the empirical ordering condition among the NC scoresused to deﬁne set (6) ensures the validity condition (1)
[Vovk et al., 2005].
Theorem 1 [Vovk et al., 2005]
Under Assumption 1, for any miscoverage level ↵ 2 [1 / ( N val ⌧ +1) , 1) ,given any NC score as per Deﬁnition 1, the VB-CP set predictor (6) satisﬁes the validity condition(1).
2.3 Cross-Validation-Based Conformal Prediction (XB-CP)
In VB-CP, the validation data set is only used to compute the empirical quantile in (6), and ishence not leveraged by the training algorithm p (  |D tr ⌧ , ⇠ ) .
This generally causes the inefﬁciency(2) of VB-CP to be large if number of data points, N ⌧ , is small.
XB-CP addresses this problemvia K -fold cross-validation [Barber et al., 2021].
K -fold cross-validation partitions the per-taskdata set D ⌧ = { z ⌧ [ i ] } N ⌧ i =1 into K  2 disjoint subsets D ⌧ , 1 , . . .
, D ⌧ ,K such that the condition
S
Kk =1 D ⌧ ,k = D ⌧ is satisﬁed.
We deﬁne the leave-one-out data set D ⌧ , ¬ k =
S
Kk
=1 ,k 0 6 = k D ⌧ ,k 0 that 78 excludes the subset
D ⌧ ,k .
We also introduce a mapping function k : { 1 , . .
. , N ⌧ } !
, K } toidentify the subset D ⌧ ,k ( i ) that includes the sample z ⌧
[ i ] , i.e., z ⌧ [ i ] 2 D ⌧ ,k ( i ) .
We focus here on a variant of XB-CP that is referred to as min-max jacknife+ in [Barber et al.,2021].
This variant has stronger validity guarantees than the jacknife+ scheme also studied in [Barberet al., 2021].
Accordingly, given a test input x ⌧ , XB-CP computes the NC score for a candidatepair z = ( x ⌧ , y ) with y 2 Y ⌧ by taking the minimum NC score NC ( z |D ⌧ , ¬ k , ⇠ ) over all possiblesubsets k 2 { 1 , . . .
, K } , i.e., as min k 2 { 1 ,...,K } NC ( z |D ⌧ , ¬ k , ⇠ ) .
Furthermore, for each data pointz ⌧ [ i ] 2 D ⌧ , the NC score is evaluated by excluding the subset D ⌧ ,k ( i ) as NC ( z ⌧ [ i ] |D ⌧ , ¬ k ( i ) , ⇠ ) .
Notethat evaluating the resulting N ⌧ + 1 NC scores requires running the training algorithm p
(  |D ⌧ , ¬ k , ⇠ )
K times, once for each subset D ⌧ , ¬ k .
Finally, a candidate y 2 Y ⌧ is included in the prediction set ifthe NC score for z = ( x ⌧ , y ) is smaller (or equal) than for a fraction (at least)
b ↵ 0 ( N ⌧ + 1) c /N
⌧ ofthe validation data points with ↵ 0 =
↵  1  K/N ⌧ K +1 .
Overall, given data set D ⌧ = { z ⌧ [ i ] = ( x ⌧ [ i ] , y ⌧ [ i ]) } N ⌧ i =1 and test input x ⌧ 2 X ⌧ , K -fold XB-CPproduces the set predictor K -XB ↵
[Equation]
where 1 ( · ) is the indicator function ( 1 ( true )
= 1 and 1 ( false ) = 0 ).
Theorem 2 [Barber et al., 2021]
Under Assumption 1, for any miscoverage level ↵⇥ 1 N ⌧ +1 +1  K/N ⌧ K +1 , , given any NC score as per Deﬁnition 1, the XB-CP set predictor (7) satisﬁes the validitycondition (1).
While a proof of Theorem 2 for K = N ⌧ can be found in [Barber et al., 2021], the general casefor K < N ⌧ follows from the same proof techniques in [Barber et al., 2021] and is included forcompleteness in Appendix A.1.
2.4 Adaptive Parametric NC Score
The CP methods reviewed so far achieve the per-task validity condition (1).
In contrast, the per-inputconditional validity (4) is only attainable with strong additional assumptions on the joint distributionp ( D ⌧ , x ⌧ )
[Vovk, 2012, Lei and Wasserman, 2014].
However, the adaptive NC score introduced by
Romano et al.
[2020] is known to empirically improve the per-input conditional validity of VB-CP(6) and XB-CP (7).
In this subsection, we assume that a model class of probabilistic predictors p ( y | x,  ) is available, e.g.,a neural network with a softmax activation in the last layer.
To gain insight on the deﬁnition of adaptive
NC scores, let us assume for the sake of argument that the ground-truth conditional distributionp ( y ⌧ | x ⌧ ) is known.
The most efﬁcient (deterministic) set predictor satisfying the conditional coveragecondition (4) would then be obtained as the smallest-cardinality subset of target values in Y ⌧ thatsatisﬁes the conditional coverage condition (4), i.e.,
[Equation]
Note that set (8) can be obtained by adding values y 2 Y ⌧ to set predictor  ⇤ ↵ ( x ⌧ ) in order fromlargest to smallest value of p ( y | x ⌧ ) until the constraint in (8) is satisﬁed.
In practice, the conditional distribution p ( y ⌧ | x ⌧ ) is estimated via the model p ( y ⌧ | x ⌧ ,  ) where theparameter vector  is produced by a training algorithm p
(  | ˜ D ⌧ , ⇠ ) applied to some training data set˜ D ⌧ .
This yields the naïve set predictor
[Equation]
[Equation]
where we have used for generality the ensemble predictor obtained by averaging over the output ⇠ p (  | ˜ D ⌧ , ⇠ ) of the training algorithm.
Unless the likelihood model is perfectly calibrated, i.e.,unless the equality p ( y ⌧ | x ⌧ ) =
E  ⇠ p (  | ˜ D ⌧ , ⇠ )
[ p ( y ⌧ | x ⌧ ,  )] holds, there is no guarantee that the setpredictor in (9) satisﬁes the conditional coverage condition (4) or the marginal coverage condition (1)with ↵ = ↵ naïve .
To tackle this problem, Romano et al.
[2020] proposed to apply VB-CP or XB-CP with a modiﬁed
NC score inspired by the naïve prediction (9).
Deﬁnition 4 (Adaptive NC score) For a learning task ⌧ , given a training algorithm p
(  | ˜ D ⌧ , ⇠ )that is invariant to permutation of the training set ˜ D ⌧ , the adaptive NC score for input-output pairz =
( x, y ) with x 2 X ⌧ and y 2 Y ⌧ given data set ˜ D ⌧ , is deﬁned as
[Equation]
Intuitively, if the adaptive NC score is large, the pair z does not conform well with the probabilisticmodel E  ⇠ p (  | ˜ D ⌧ , ⇠ ) p ( y | x,  ) obtained by training on set ˜ D ⌧ .
The adaptive NC score satisﬁes thecondition in Deﬁnition 1, and hence by Theorems 1 and 2, the set predictors (6) and (7) for VB-CPand XB-CP, respectively, are both valid when the adaptive NC score is used.
Furthermore, Romanoet al.
[2020] demonstrated improved conditional empirical coverage performance as compared to theconventional two-step NC score in Deﬁnition 2.
This may be seen as a consequence of the conditionalvalidity of the naïve predictor (9) under the assumption of a well-calibrated model.
The adaptive NC score (10) can be equivalently expressed as
[Equation]
where we have used the notation p ( y | x, ˜ D ⌧ , ⇠ ) :
= E  ⇠ p (  | ˜ D ⌧ , ⇠ ) p ( y | x,  ) .
Meta-Learning Algorithm for XB-CP (Meta-XB)
In this section, we introduce the proposed meta-XB algorithm.
We start by describing the metalearning framework.
3.1 Meta-Learning
Up to now, we have focused on a single task ⌧ .
Meta-learning utilizes data from multiple tasksto enhance the efﬁciency of the learning procedure for new tasks.
Following the standard metalearning formulation [Baxter, 2000, Amit and Meir, 2018], as anticipated in Section 1, the learningenvironment is characterized by a task distribution p ( ⌧ ) over the task identiﬁer ⌧⌧⌧ .
Given T metatraining tasks realizations ⌧⌧⌧ 1 = ⌧ 1 , . . .
, ⌧⌧⌧ T = ⌧ T drawn i.i.d.
from the task distribution p ( ⌧ ) , themeta-training data set D ⌧ 1: T :=
{{D jt , z jt } M t j =1 } Tt =1 consists of M t realizations {D jt , z jt } M t j =1 of datasets D j ⌧ t = D jt with N ⌧ t =
N t examples and test sample z j ⌧ t =
z jt for each task ⌧ t .
Pairs {D jt , z jt } M t j =1are generated i.i.d.
from the joint distribution p ( D ⌧ t , z ⌧ t ) , satisfying Assumption 1 for all tasks t .
The goal of meta-learning for CP is to optimize the vector of hyperparameter ⇠ based on the metatraining data D ⌧ 1: T , so as to obtain a more efﬁcient set predictor  ( x ⌧ |D ⌧ , ⇠ ) .
While reference [Fischet al., 2021] proposed a meta-learning solution for VB-CP [Vovk et al., 2005], here we introduce ameta-learning method for XB-CP.
3.2 Meta-XB
Meta-XB aims at ﬁnding a hyperparameter vector ⇠ that minimizes the average size of the predictionset  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) in (7) for tasks ⌧ that follow the distribution p ( ⌧ ) .
To this end, it addresses theproblem of minimizing the empirical average of the sizes of the prediction sets  K -XB ↵ ( x ⌧ t |D ⌧ t [Equation] ⇠ ) 155 across the meta-training tasks ⌧ 1 , . .
⌧ T over the hyperparameter vector ⇠ .
This amounts to theoptimization
[Equation]
where the ﬁrst sum is over the meta-training tasks and the second is over the available data for eachtask.
By (7), the size of the prediction set |  K -XB ↵
[Equation]
, ⇠ ) | is not a differentiable function of thehyperparameter vector ⇠ .
Therefore, in order to address (12) via gradient descent, we introduce adifferentiable soft inefﬁciency criterion by replacing the indicator function with the sigmoid  ( u ) =(1 + exp(  u/c  ))
1 for some c  > 0 ; the quantile Q 1  ↵ ( · ) with a differentiable soft empiricalquantile ˆ Q 1  ↵ ( · ) ; and the minimum operator with the softmin function [Goodfellow et al., 2016,
Cuturi et al., 2019].
For an input set { a [ j ] } Mj =1 , the softmin function is deﬁned as [Goodfellow et al., 2016, Section 6.2.2.3]
[Equation]
for some c S > 0 .
Finally, given an input set { a [1] , . . .
, a [ M ] } , the soft empirical quantile ˆ Q 1  ↵ ( · ) isdeﬁned as
[Equation]
[Equation] some > 0 , where we have used the pinball loss⇢ [Equation]
↵ [Equation] a |{ a [1] , . . . , a [ M ] } )
[Koenker and Bassett Jr, 1978]
[Equation]
with ReLU ( a ) = max(0 , a ) .
With these deﬁnitions, the softinefﬁciency metric is derived from (7) as follows (see details in
Appendix A.2).
Deﬁnition
Given a data set D ⌧ and a test input x ⌧ , the soft inefﬁciency for the K -fold XB-CPpredictor (7) is deﬁned as
[Equation]
where ↵ 0 = ↵  1  K/N ⌧ K
+ and c  , c S , c Q > 0 .
The parameters c  , c S , and c Q dictate the trade-off between smoothness and accuracy of theapproximation | ˆ  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) | with respect to the true inefﬁciency |  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) | :
As 178 c  , c S , c Q ! 0 , the approximation becomes increasingly accurate for any  > 0 , as long as we [-have↵⇥ 1 N ⌧ +
-]have
[Equation]  , but the function | ˆ  K -XB ↵ ( x ⌧ |D ⌧ , ⇠ ) | is increasingly less smooth (see Fig.
4for an illustration of the accuracy of the soft quantile).
Replacing the soft inefﬁciency (16) into problem (12) yields a differentiable program when conventional two-step NC scores (Deﬁnition 2) are used.
We address the corresponding problem viastochastic gradient descent (SGD), whereby at each iteration a batch of tasks and examples per taskare sampled.
The overall meta-learning procedure is summarized in Algorithm 1.
3.3 Meta-XB with Adaptive NC Scores
Adaptive NC scores are not differentiable.
Therefore, in order to enable the optimization of problem(12) with the soft inefﬁciency (16), we propose to replace the indicator function 1 ( · ) in (11) withthe sigmoid function  ( · ) .
We also have found that approximating the number of outputs y 0 2 Y ⌧that satisfy (11) rather than direct application of sigmoid function empirically improves per-inputcoverage performance.
This yields the soft adaptive NC score ˆ NC ada ( z | ˜ D ⌧ , ⇠ ) , which is detailed in
Appendix B. With the soft adaptive NC score, meta-XB is then applied as in Algorithm 1.
3.4 Per-Task Validity of Meta-XB
As mentioned in Section 1, existing meta-learning schemes for CP cannot achieve the per-task validitycondition in (1), requiring an additional marginalization over distribution p ( ⌧ ) [Fisch et al., 2021] orachieving looser validity guarantees formulated as probably approximately correct (PAC)-bounds[Park et al., 2022].
In contrast, meta-XB has the following property.
Theorem 3 Under Assumption 1, for any miscoverage level ↵⇥ 1 N ⌧ +1 + 1  K/N ⌧ K
+1 , , given any
NC score (Deﬁnition 1), the XB-CP set predictor (7) with ⇠ = ⇠ ⇤ in (12) satisﬁes the validity condition(1).
Theorem 3 is a direct consequence of Theorem 2, since meta-XB maintains the permutation-invarianceof the training algorithm p
(  | ˜ D ⌧ , ⇠ ⇤ ) as required by Deﬁnition 2.
initialize hyperparameter vector ⇠ while convergence criterion not met do choose ˜ T tasks randomly from set { 1 , . .
, T } and denote the corresponding task set as ˜ T for each sampled task
t 2 ˜ T do randomly sample ˜ M t pairs from the data set D ⌧ t , i.e., {D jt , z jt }
j 2 ˜
J t denoting the corresponding index set as ˜ J t , and compute the soft inefﬁciency
[Equation]
(17) end update hyperparameter
Related Work
Bayesian learning and model misspeciﬁcation.
When the model is misspeciﬁed, i.e., when theassumed model likelihood or prior distribution cannot express the ground-truth data generating distribution [Masegosa, 2020], Bayesian learning may yield poor generalization performance [Masegosa,2020, Morningstar et al., 2022, Wenzel et al., 2020].
Downweighting the prior distribution and/or thelikelihood, as done in generalized Bayesian learning [Knoblauch et al., 2019, Simeone, 2022] or in“cold” posteriors [Wenzel et al., 2020], improve the generalization performance.
In order to mitigatethe model likelihood misspeciﬁcation, alternative variational free energy metrics were introducedby Masegosa [2020] via second-order PAC-Bayes bounds, and by Morningstar et al.
[2022] viamulti-sample PAC-Bayes bounds.
Misspeciﬁcation of the prior distribution can be also addressed via
Bayesian meta-learning, which optimizes the prior from data in a manner similar to empirical Bayes[MacKay, 2003].
Bayesian meta-learning While frequentist meta-learning has shown remarkable success in few-shotlearning tasks in terms of accuracy
[Finn et al., 2017, Snell et al., 2017], improvements in termsof calibration can be obtained by Bayesian meta-learning that optimizes over a hyper-posteriordistribution from multiple tasks [Amit and Meir, 2018, Finn et al., 2018, Yoon et al., 2018, Ravi and
Beatson, 2018, Nguyen et al., 2020, Jose et al., 2022].
The hyper-prior can also be modelled as astochastic process to avoid the bias caused by parametric models [Rothfuss et al., 2021].
CP-aware loss.
Stutz et al.
[2021] and Einbinder et al.
[2022] proposed CP-aware loss functions toenhance the efﬁciency or per-input validity (4) of VB-CP.
The drawback of these solutions is that theyrequire a large amount of data samples, i.e., N ⌧  1 , unlike the meta-learning methods studied here.
Per-input validity and local validity.
As discussed in Section 2.4, the per-input validity condition(4) cannot be satisﬁed without strong assumptions on the joint distribution p ( D ⌧ , z ⌧ )
[Vovk, 2012,
Lei and Wasserman, 2014].
Given the importance of adapting the prediction set size to the input tocapture heteroscedasticity [Romano et al., 2019, Izbicki et al., 2020], a looser local validity condition,which conditions on a subset of the input data space A x ⌧ ⇢ X ⌧ containing the input x ⌧ of interest, i.e.,x ⌧ 2 A x ⌧ , has been considered in [Lei and Wasserman, 2014, Foygel Barber et al., 2021].
Choosinga proper subset A x ⌧ becomes problematic especially in high-dimensional input space [Izbicki et al.
,2020, LeRoy and Zhao, 2021], and Tibshirani et al.
[2019], Lin et al.
[2021] proposed to reweight thesamples outside the subset A x ⌧ by treating the problem as distribution-shift between the data set D ⌧and the test input x ⌧ .
Experiments
In this section, we provide experimental results to validate the performance of meta-XB in termsof (i) per-task coverage P ( y ⌧ 2  ( x ⌧ | D ⌧ , ⇠ )) ; (ii) per-task inefﬁciency (2); (iii) per-task conditional coverage P ( y ⌧ 2  ( x ⌧ | D ⌧ , ⇠ ) | x ⌧ = x ⌧ ) ; and (iv) per-task conditional inefﬁciency
E [ |  ( x ⌧ | D ⌧ , ⇠ ) || x ⌧ = x ⌧ ] .
To evaluate input-conditional quantities, we follow the approach in[Romano et al., 2020, Section S1.2].
As benchmark schemes, we consider (i) VB-CP, (ii) XB-CP, and(iii) meta-VB
[Fisch et al., 2021], with either the conventional NC score (Deﬁnition 2 with log-loss` ⌧ ( z |  ) ) or adaptive NC score with Deﬁnition 4).
Note that meta-VB was described in [Fisch et al.
,2021] only for the conventional NC score, but the application of the adaptive NC score is direct.
Forall the experiments, unless speciﬁed otherwise, we consider a number of examples N ⌧ = 9 for the data set D ⌧ and the desired miscoverage level ↵ = 0 .
For the cross-validation-based set predictors
XB-CP and meta-XB, we set number of folds to K = N ⌧ .
The aforementioned performance measuresare estimated by averaging over 1000 realizations of data set D ⌧ and over 500 realizations for thetest sample z ⌧ of each task ⌧ .
We report in this section the 100 different per-task quantities whichare computed from 100 different tasks.
During meta-training, for T different tasks, we assumeavailability of M t ( N ⌧ + 1) i.i.d.
examples, from which we sample ˜ M t pairs {D jt , z jt }
j 2 ˜
J t whencomputing inefﬁciency (17), with which we use Adam optimizer [Kingma and Ba, 2014] to updatethe hyperparameter vector ⇠ via SGD.
Lastly, we set the value of the approximation parametersc  , c S , and c Q to be one.
Following [Romano et al., 2020], for VB-CP and XB-CP, we adopt a support vector classiﬁer astraining algorithm p (  | ˜ D ⌧ , ⇠ ) as it does not require any tuning of the hyperparameter vector ⇠ .
Incontrast, for meta-VB and meta-XB, we adopt a neural network classiﬁer [Romano et al., 2019], andset the training algorithm p
(  | ˜ D ⌧ , ⇠ ) to output the last iterate of a pre-deﬁned number of steps of GD( 1 , unless speciﬁed otherwise) with initialization given by the hyperparameter vector ⇠
[Finn et al.
Note that using full-batch GD ensures the permutation-invariance of the training algorithm asrequired by Deﬁnition 2.
All the experiments are implemented by PyTorch [Paszke et al., 2019] and ran over a GPU serverwith single NVIDIA A100 card.
5.1 Multinomial Model and Inhomogeneous Features
We start with the synthetic-data experiment introduced in [Romano et al., 2020] in which the inputx 2 R 10 is such that the ﬁrst element equals x 1 = 1 with probability 1 / 5 and x 1 =  8 otherwise,while the other elements x 2 , . . .
, x 10 are i.i.d.
standard Gaussian variables.
For each task ⌧ , matrix⌧ 2 R 10 ⇥ |Y ⌧ | is sampled with i.i.d.
standard Gaussian entries and the ground-truth conditional distribution p ( y ⌧ | x ⌧ ) is deﬁned as the categorical distribution
[Equation]
for y 2 { 1 , . . . , |Y ⌧ |} , where ⌧ y 2 R |Y ⌧ | is the y th column of the task information matrix ⌧ .
Thenumber of classes is |Y ⌧ | = 5 and neural network classiﬁer consists of two hidden layers with
Exponential Linear Unit (ELU) activation [Clevert et al., 2015] in the hidden layers and a softmaxactivation in the last layer.
In Fig.
5, we demonstrate the performance of the considered set predictors as a function of number oftasks T .
Both meta-VB and meta-XB achieve lower inefﬁciency (2) as compared to the conventionalset predictors VB-CP and XB-CP, as soon as the number of meta-training tasks is sufﬁciently largeto ensure successful generalization across tasks [Yin et al., 2019, Jose and Simeone, 2020].
Forexample, meta-XB with T = 100 tasks obtain an average prediction set size of 3 , while XB-CPhas an inefﬁciency larger than 4 .
Furthermore, all schemes satisfy the validity condition (1), exceptfor meta-VB for T & 10 4 , conﬁrming the analytical results.
Adaptive NC scores are seen to beinstrumental in improving the conditional validity (4) when used with meta-XB, although this comesat the cost of a larger inefﬁciency.
Next, we investigate the impact of number of per-task examples N ⌧ in data set D ⌧ using adaptive NCscores.
As shown in Fig.
6, the average size of the set predictors decreases as N ⌧ grows larger.
In thefew-examples regime, i.e., with N ⌧  40 , the meta-learned set predictors meta-VB and meta-XBoutperform the conventional set predictors VB-CP and XB-CP in terms of inefﬁciency.
However,when N ⌧ is large enough, i e., when N ⌧  80 , conventional set predictors are preferable, as transferof knowledge across tasks becomes unnecessary, and possibly deleterious [Amit and Meir, 2018]
(seealso [Park et al., 2020] for related discussions).
In terms of conditional coverage, Fig. 6 shows thatcross-validation-based CP methods are preferable as compared to validation-based CP approaches.
5.2 Modulation Classiﬁcation
per-task coverage
Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction
Anonymous Author(s)
AfﬁliationAddress email
Abstract ...
1 Introduction check Q
1   ,N  !! 1.1 Context and Motivation
In modern application of artiﬁcial intelligence (AI), calibration is often deemed as important as the 3 standard criteria of (average) accuracy.
A well-calibrated model is one that can reliably quantify theuncertainty of its decisions [Guo et al., 2017, Hermans et al., 2021].
Information about uncertainty 5 is critical when access to data is limited and AI decisions are to be acted on by human operators,machines, or other algorithms.
The majority of existing papers on calibration for AI focuses on 7 Bayesian learning as a means to quantify epistemic uncertainty [Finn et al., 2018, Yoon et al., 2018,
Ravi and Beatson, 2018, Jose et al., 2022].
However, recent studies have shown the limitations of 9 Bayesian learning when the assumed model likelihood and, or prior distribution, is misspeciﬁed[Masegosa, 2020].
Furthermore, exact Bayesian learning is computationally infeasible, calling for 11 approximations such as Monte Carlo (MC) sampling
[Robert et al., 1999] and variational inference(VI)
[Blundell et al., 2015].
Overall, under practical conditions, Bayesian learning does not provide 13 formal guarantees of calibration.
Conformal prediction (CP) [Vovk et al., 2005] provides a general framework for the calibration of 15 (frequentist or Bayesian) probabilistic models.
The formal calibration guarantees provided by CPhold irrespective of the (unknown) data distribution, as long as the available data samples and the test 17 samples are exchangeable – a weaker requirement than the standard i.i.d.
assumption.
As illustratedin Fig. 1, CP produces set predictors that output a subset of the output space Y for each input x , with 19 the property that the set contains the true label with probability no smaller than a desired value 1  ↵ .
Mathematically, for a given learning task ⌧ , assume that we are given a data set D ⌧ with N ⌧ samples, 21 i.e.,
D ⌧ = { z ⌧ [ i ] } N ⌧ i =1 , where the i th sample z ⌧ [ i ] = ( x ⌧ [ i ] ,y ⌧ [ i ]) contains input x ⌧ [ i ] 2 X ⌧ and 22 target y ⌧
[ i ] 2 Y ⌧ .
The input data space X ⌧ and the target data space Y ⌧ may depend on the task ⌧ .
23 CP provides a set predictor  ( ·|D ⌧ , ⇠ ) :
X ⌧ !
2 Y ⌧ , speciﬁed by a hyperparameter vector ⇠ , that 24 maps an input x ⌧ 2 X ⌧ to a subset of the output domain Y ⌧ based on a data set D ⌧ .
Calibration 25 amounts to the per-task validity condition
P
[Equation] , (1) which indicates that the set predictor  ( x ⌧ | D ⌧ , ⇠ ) contains the true target y ⌧ with probability at least 27 meta-VB
VB-CP
XB-CP meta-XB inefficiency meta-VB
VB-CP
XB-CP meta-XB classiﬁcation [O’Shea et al., 2018].
We consider N ⌧ = 9 examples for each data set D ⌧ .
The boxes represent the 25% (lower edge), 50% (line within the box), and 75% (upper edge) percentiles of the per-task performance metrics evaluated over 100 different meta-test tasks.
We now consider the real-world modulation classiﬁcation example illustrated in Fig.
1, in which the goal is classifying receivedradio signals depending on the modulation scheme used to generate it
[O’Shea et al., 2016, 2018].
The RadioML 2018.01Adata set consists 98 , 304 inputs with dimension 2 ⇥ 1024 , accounting for complex baseband signals sampled over 1024 timeinstants, generated from 24 different modulation types [O’Sheaet al., 2018].
Each task ⌧ amounts to the binary classiﬁcation ofsignals from two randomly selected modulation types.
Specifically, we divide the 24 modulations types into 16 classes usedto generate meta-training tasks, and 8 classes used to producemeta-testing tasks, following the standard data generation approach in few-shot classiﬁcations [Lake et al., 2011, Ravi and
Larochelle, 2016].
We adopt VGG
[Simonyan and Zisserman, 2014] as the neural network classiﬁer as in [O’Shea et al.,2018].
Furthermore, for meta-VB and meta-XB, we apply asingle GD step during meta-training and ﬁve GD steps duringmeta-testing [Finn et al., 2017, Ravi and Beatson, 2018].
Fig.
7 shows per-task coverage and inefﬁciency for all schemesassuming conventional NC scores.
While the conventional setpredictors VB-CP and XB-CP produce large, uninformative setpredictors that encompass the entire target data space Y ⌧ of dimension |Y ⌧ | = 2 , the meta-learnedset predictors meta-VB and meta-XB can signiﬁcantly improve the prediction efﬁciency.
However,meta-VB fails to achieve per-task validity condition (1), while the proposed meta-XB is valid asproved by Theorem 3.
5.3 Image Classiﬁcation
Lastly, we consider image classiﬁcation problem with the mini Imagenet dataset [Vinyals et al., 2016]considering N ⌧ = 4 data points per task with desired miscoverage level ↵ = 0 .
We considerbinary classiﬁcation with tasks being deﬁned by randomly selecting two classes of images, anddrawing training data sets by choosing among all examples belonging to the two chosen classes.
[Vinyals et al., 2016].
We consider
N ⌧ = 4 examples for each data set
D ⌧ .
The boxes represent the 25% (lower edge), 50% (line within the box), and 75% (upper edge) percentiles of the per-task performance metrics evaluated over 100 different meta-test tasks.
Conventional NC scores are used, and the neural network classiﬁer consists of the convolutional neural network (CNN)
usedin
[Finn et al., 2017].
For meta-VB and meta-XB, a single step
GD update is used during meta-training, while ﬁve GD updatesteps are applied during meta-testing.
Fig.
8 shows that metalearning-based set predictors outperform conventional schemes.
Furthermore, meta-VB fails to meet per-task coverage in contrast to the proposed meta-XB.
Conclusion
This paper has introduced meta-XB, a meta-learning solutionfor cross-validation-based conformal prediction that aims atreducing the average prediction set size, while formally guaranteeing per-task calibration.
The approach is based on theuse of soft quantiles, and it integrates adaptive nonconformityscores for improved input-conditional calibration.
Throughexperimental results, including for modulation classiﬁcation[O’Shea et al., 2016, 2018], meta-XB was shown to outperform both conventional conformal prediction-based solutionsand meta-learning conformal prediction schemes.
Future workmay integrate meta-learning with CP-aware training criteria[Stutz et al., 2021, Einbinder et al., 2022], or with stochasticset predictors.
A ProofsA.1 Proof of Theorem 2
The proof mainly follows [Barber et al., 2021, Section B.1] and [Barber et al., 2021, Section B.2.2]with the following changes.
We ﬁrst extend the result for regression problems in [Barber et al., 2021] to classiﬁcation, startingwith the case K = N ⌧ , for which the mapping function k ( · ) is the identity k ( i )
=
i .
Unlike[Barber et al., 2021], which deﬁned “comparison matrix” of residuals for the regression problem, weconsider a more general comparison matrix deﬁned in terms of NC scores that can be applied for bothclassiﬁcation and regression problems in a manner similar to [Romano et al., 2020].
Accordingly, wedeﬁne the comparison matrix
A 2 { 0 , 1 } ( N ⌧ +
[Equation]
⇥ [Equation] N [Equation]
[Equation]
⌘
[Equation]
k
[Equation]
for a ﬁxed vector of hyperparameter ⇠ .
The cardinality of the set S ( A ) of “strange” points
[Equation]
can be bounded as | S ( A ) |  N ⌧ + 1  (1  ↵ 0 )( N ⌧ + 1)
[Barber et al., 2021, Romano et al., 2020].
Therefore, theorem 2 holds for K = N ⌧ , since any N ⌧ + 1 points can be “strange points” with equalprobability thanks to Assumption 1.
To address the case K < N ⌧ , we follow [Barber et al., 2021, Section B.2.2] by drawing N ⌧ /K  1additional test examples that are all assigned to the ( K + 1)
th fold.
This way, the actual ( N ⌧ + 1) thtest point is equally likely to be in any of the K + 1 folds.
Now, taking the augmented data set ¯ D thatcontains all the N ⌧ + N ⌧ /K examples in lieu of D in (19), we can bound the number of “strangepoints” in set (20) as
[Equation]
Finally, by using the same proof technique in [Barber et al., 2021, Section B.2.2], we have theinequality
[Equation]
In Theorem 2, we choose ↵ 0 = 1  K/N ⌧ K +1 , which satisﬁes per-task validity condition (1) from (22).
A.2 Proof for Deﬁnition
From the deﬁnition of the XB-CP set predictor (7), the inefﬁciency can be obtained as
[Equation]
with Q  1  ↵ ( { a [ i ] } Mi =1 ) :
=  Q 1  ↵ ( {  a [ i ] } Mi =1 ) being the b ↵ ( M + 1)
c th smallest value inthe set { a [1] , . .
, a [ M ] , 1 } .
The equality in (23) is proved as follows.
Deﬁning g ( z ⌧ ) :=
min k 2 { 1 ,...,K } NC (( x ⌧ , y ) |D ⌧ , ¬ k , ⇠ ) with z ⌧ = ( x ⌧ , y ) and f ( z ⌧ [ i ]) :
= NC ( z ⌧ [ i ] |D ⌧ , ¬ k ( i ) , ⇠ ) ,we show that the inequality
P
[Equation]
[Equation]
↵ 0 ( N ⌧ + 1) c is equivalent to
[Equation]
{ g (
[Equation]
)  f ( z ⌧ [ i ]) } N
[Equation]
 0 .
This is a consequence of the following equivalence relations:
[Equation]
, at least b ↵ 0 ( N ⌧ + 1) c values of g ( z ⌧ )  f ( z ⌧ [ i ]) are smaller than or equal to,b ↵ 0 ( N ⌧ + 1)
c th smallest value of g ( z ⌧ )  f ( z ⌧ [ i ]) is smaller than or equal to, [Equation]
[Equation]
By replacing 1 ( · ) with the sigmoid  ( · ) , min( · ) with softmin ( · ) (13), and the quantile Q 1  ↵ 0 ( · ) withˆ Q 1  ↵ 0 ( · ) (14), we ﬁnally obtain the soft inefﬁciency of K -fold XB-CP predictor in (16) from (24).
B Details on Soft Adaptive NC Scores
Recalling (11), while denoting p y 0 :
= p ( y 0 | x, ˜ D ⌧ , ⇠ ) and p y :
= p ( y | x, ˜ D ⌧ , ⇠ ) , the adaptive NC scorefor input-output pair z
=
( x, y ) with x 2 X ⌧ and y 2 Y ⌧ can be computed as
[Equation]
We deﬁne the soft adaptive NC score by approximating the indicator function 1 ( · ) with the sigmoid ( · ) as
[Equation]
Note that, we have found that the preprocessing (26) yields better empirical per-input coverage ascompared to the direct approximation of (11) that replaces the indicator function 1 ( · ) with sigmoid ( · ) , i.e.,
P
[Equation]
C Additional ExperimentsC.1 Demodulation
To elaborate further on the last column in Fig.
3, here we present a toy example that allows us tovisualize the set predictors obtained by XB-CP and meta-XB, both with conventional and adaptive
NC.
To this end, we implement XB-CP with the same neural network used for meta-XB but with thehyperparameter vector ⇠ deﬁning the initialization of GD set to a random vector [He et al., 2015].
Given a learning task ⌧ , the input and output space X ⌧ and Y ⌧ are given by the set of complex points,i.e., by the two-dimensional real vectors [Larsson, 2017]
[Equation]
some task-speciﬁc phase shift  ⌧ 2 [0 , 2 ⇡ ] .
Denoting as N ⌧ ( x ) =
{ y 2 Y ⌧ : | x  y |  r and y
=x } the set of neighboring points within some radius r , the ground-truth distribution p ( y ⌧ | x ⌧ ) is suchthat y ⌧ equals x ⌧ with probability 1  p , and it equals any neighboring point y 2 N ⌧ ( x ) withprobability p/ |N
⌧ ( x ) |
M = r = and p = design neural network classiﬁer to consist of two hidden layers with Exponential Linear Unit (ELU) activation [Clevert et al., 2015] in the hidden layers and a softmax activation in the last
Fig.
9 visualizes the set predictors for XB-CP, i.e., with a random hyperparameter vector ⇠ , and formeta-XB, after meta-training with 1000 tasks, by focusing on a speciﬁc realizations of phase shift thatfollows the distribution  ⌧ ⇠ Unif [0 , 2 ⇡ ) .
By transferring knowledge from multiple tasks, meta-XBis seen to yield more efﬁcient set predictors.
Furthermore, by using adaptive NC scores, meta-XBcan adjust the prediction set size depending on the “difﬁculty” of classifying the given input, whilea conventional NC score tends to produce set predictors of similar sizes across all inputs.
Note, infact, that inputs close to the center of the set, as the green example in Fig.
9, have more neighbors ascompared to points at the edge, as the red points in Fig.
9, making it harder to identify the true valueof y given x .