L OCALIZED M ETA -L EARNING : A PAC-B AYES A NALYSIS FOR M ETA -L EARNING B EYOND G LOBAL P RIOR
Anonymous authors
Paper under double-blind review
A BSTRACT
Meta-learning methods learn the meta-knowledge among various training tasksand aim to promote the learning of new tasks under the task similarity assumption.
However, such meta-knowledge is often represented as a ﬁxed distribution, whichis too restrictive to capture various speciﬁc task information.
In this work, wepresent a localized meta-learning framework based on the PAC-Bayes theory.
Inparticular, we propose an LCC-based prior predictor that allows the meta learner toadaptively generate local meta-knowledge for speciﬁc tasks.
We further developa practical algorithm with deep neural network based on the bound.
Empiricalresults on real-world datasets demonstrate the efﬁcacy of the proposed method.
I NTRODUCTION
Recent years have seen a resurgence of interest in the ﬁeld of meta-learning, or learning-to-learn(Thrun & Pratt, 2012), especially for empowering deep neural networks the capability of fast adapting to unseen tasks just as humans (Finn et al., 2017; Ravi & Larochelle, 2017).
More concretely,the neural networks are trained from a sequence of datasets, associated with different learning taskssampled from a meta-distribution (also called task environment (Baxter, 2000; Maurer, 2005)).
Theprincipal aim of meta learner is to extract transferable meta-knowledge from observed tasks andfacilitate the learning of new tasks sampled from the same meta-distribution.
The performance ismeasured by the generalization ability from a ﬁnite set of observed tasks, which is evaluated bylearning related unseen tasks.
For this reason, there has been considerable interest in theoreticalbounds on the generalization in terms of the meta-learning algorithm (Denevi et al., 2018b;a).
One typical line of work (Pentina & Lampert, 2014; Amit & Meir, 2018) use PAC-Bayes bound toanalyze the generalization behavior of the meta learner and quantify the relation between the expected loss on new tasks and the average loss on the observed tasks.
In this setup, we formulatemeta-learning as hierarchical Bayes.
Accordingly, meta-knowledge is instantiated as a global distribution over all possible priors, which we call hyperprior and is chosen before observing trainingtasks.
Each prior is a distribution over a family of classiﬁers w.r.t.
a particular task.
To learn versatile meta-knowledge across tasks, the meta learner observes a sequence of training tasks and adjustsits hyperprior into a hyperposterior distribution over the set of priors.
To solve a new task, the baselearner produces a posterior distribution over a family of classiﬁers based on the associated sampleset and the prior generated by the hyperposterior.
However, such meta-knowledge is shared across tasks.
The global hyperposterior is rather generic,typically not well-tailored to various speciﬁc tasks.
Consequently, it leads to sub-optimal performance for any individual prediction task.
As a motivational example, suppose we have two different tasks: distinguishing motorcycle versus bicycle and distinguishing motorcycle versus car.
Intuitively, each task uses distinct discriminative patterns and thus the desired meta-knowledge isrequired to extract these patterns simultaneously.
This could be a challenging problem to representit with a global hyperposterior since the most signiﬁcant patterns in the ﬁrst task could be irrelevantor even detrimental to the second task.
Hence, we are motivated to pursue a meta-learning framework to effectively deﬁne the hyperposterior.
The inspiration comes from the PAC-Bayes literature on data distribution dependent priors(Catoni, 2007; Parrado-Hern´andez et al., 2012; Dziugaite & Roy, 2018).
The choice of posteriorin each task is constrained by the need to minimize the relative entropy between prior and posterior since this divergence forms part of the bound and is typically large in standard PAC-Bayes approaches (Lever et al., 2013).
Thus, choosing an appropriate prior for each task which is close tothe related posterior could yield improved generalization bounds.
Inspired by this, we propose a Localized Meta-Learning (LML) framework.
Instead of formulatingmeta-knowledge as a global hyperposterior, we learn a conditional hyperposterior given task datadistribution that allows a meta learner to adaptively generate an appropriate prior for a new task.
However, the task data distribution is unknown, and our only perception for it is via the associatedsample set.
Nevertheless, if the conditional hyperposterior is relatively stable to perturbations of thesample set, then the generated prior could still reﬂect the underlying task data distribution, resultingin a generalization bound that still holds with smaller probability.
Following this intuition, thedependence of a conditional hyperposterior on the task data distribution is parameterized by a priorpredictor using Local Coordinate Coding (LCC)(Yu et al., 2009).
In particular, if the classiﬁer ineach task is specialized to a parametric model, including deep neural network, the proposed LCCbased prior predictor predicts the model parameters using the sample set by exploiting the localinformation on the latent manifold.
LCC-based prior predictor is invariant under permutations of itsinputs and could be further used for unseen tasks.
The main contributions of this work include: (i)
We present a localized meta-learning frameworkwhich provides a means to tighten the original PAC-Bayes meta-learning bound (Pentina & Lampert, 2014; Amit & Meir, 2018) by minimizing the task-complexity term by choosing data-dependentprior; (ii) We propose an LCC-based prior predictor, an implementation of conditional hyperposterior, to generate local meta-knowledge for speciﬁc task; (iii) We derive a practical localized metalearning algorithm for deep neural networks by minimizing the bound; (iv) Experimental resultsdemonstrate improved performance over meta-learning method in this ﬁeld.
P RELIMINARIES2. L OCAL C OORDINATE C ODING
We ﬁrst review some deﬁnitions of Local Coordinate Coding (LCC) (Yu et al., 2009) based on whichwe develop the proposed LCC-based prior predictor.
Deﬁnition 1.
(Lipschitz Smoothness (Yu et al., 2009).)
A function f ( x ) on R d is a ( α, β ) -Lipschitzsmooth w.r.t.
a norm (cid:107)·(cid:107) if (cid:107) f
( x ) − f ( x (cid:48) )
(cid:107) ≤ α (cid:107) x − x (cid:48)
(cid:107) and (cid:107) f
( x (cid:48) )
− f ( x ) −∇ f ( x ) (cid:62) ( x (cid:48) − x ) (cid:107) ≤β (cid:107) x − x (cid:48) (cid:107) 2 .
Deﬁnition 2.
(Coordinate Coding (Yu et al., 2009).)
A coordinate coding is a pair ( γ , C ) , where
C ⊂ R d is a set of anchor points, and γ is a map of x ∈ R d to [ γ u ( x )] u ∈ C ∈ R | C | suchthat (cid:80) u γ u ( x ) = 1 .
It induces the following physical approximation of x in R d : γ ( x ) = 75 (cid:80) u ∈ C γ u ( x ) u .
Deﬁnition 3.
(Latent Manifold (Yu et al., 2009).)
A subset M ⊂ R d is called a smooth manifold with an intrinsic dimension d := d M if there exists a constant c M such that given any x ∈ M , there exists d bases u 1 ( x ) , . .
, u d ( x ) ∈ R d so that ∀ x (cid:48) ∈ M : inf
[Equation]
where γ =
[ γ 1 , . . . , γ d ] (cid:62) are the local codings w.r.t.
the bases.
Deﬁnition 4.
(Covering Number (Yu et al., 2009).)
The covering number N ( (cid:15), M ) is the smallest cardinality of an (cid:15) -ccover C ⊂ M .
That is, [Equation] Deﬁnition 2 and 3 imply that any point in R d can be expressed as a linear combination of a set ofanchor points.
Later, we will use them to develop the prior predictor.
2. PAC-B AYES M ETA -L EARNING
In order to present the advances proposed in this paper, we next recall some deﬁnitions in PAC-Bayestheory for single-task learning and meta-learning (Catoni, 2007; Baxter, 2000; Pentina & Lampert, ; Amit & Meir, 2018).
In the context of classiﬁcation, we assume all tasks share the same inputspace X , output space Y , space of classiﬁers (hypotheses)
H ⊂ { h : X → Y} and loss function(cid:96) :
Y × Y → [0 , 1] .
The meta learner observes n tasks in the form of sample sets S 1 , . .
. , S n .
The number of samples in task i is denoted by m i .
Each observed task i consists of a set of i.i.d.samples S i
= { ( x j , y j ) } m i j =1 , which is drawn from a data distribution S
i ∼ D m i i .
Following themeta-learning setup in (Baxter, 2000), we assume that each data distribution D i is generated i.i.d.from the same meta distribution τ .
Let h ( x ) be the prediction of x , the goal of each task is to ﬁnda classiﬁer h that minimizes the expected loss E x ∼ D (cid:96) ( h ( x ) , y ) .
Since the underlying ‘true’ datadistribution D i is unknown, the base learner receives a ﬁnite set of samples S i and produces an“optimal” classiﬁer h
=
A b ( S i ) with a deterministic learning algorithm
A b ( · ) that will be used topredict the labels of unseen inputs.
PAC-Bayes theory studies the properties of randomized classiﬁer, called Gibbs classiﬁer.
Let Q bea posterior distribution over H , to make a prediction, the Gibbs classiﬁer samples a classiﬁer h ∈ Haccording to Q and then predicts a label with the chosen h .
The expected error under data distribution D and empirical error on the sample set S are then given by averaging over distribution Q , namelyer ( Q ) =
E h ∼ Q E ( x,y ) ∼ D (cid:96) ( h ( x ) , y ) and (cid:98) er
( Q ) =
E h ∼ Q 1 m (cid:80) mj
=1 (cid:96) ( h ( x j ) , y j ) , respectively.
Then,we can get the following PAC-Bayes generalization bound of Catoni (2007) in a simpliﬁed formsuggested by Germain et al. Theorem 1. ( Catoni’s bound )
Let P be some prior distribution over H .
Then for any δ ∈ (0 , 1] , and any real number c > 0 , the following inequality holds uniformly for all posteriors distribution
Q with probability at least 1 − δ ,
[Equation]
The PAC-Bayes bound holds uniformly for all Q , it also holds for the data dependent Q .
By choosingthe posterior Q that minimizes the PAC-Bayes bound, we obtain an learning algorithm with generalization guarantees.
Note that the value c allows to control the trade-off between the empirical errorand the complexity term.
The goal of the meta learner is to extract meta-knowledge contained in the observed tasks that will be used as prior knowledge for learning new tasks.
The prior knowledge P is in the form of a distribution over classiﬁers H .
In each task, the base learner produces a posterior Q =
A b ( S, P ) over H based on a sample set S and a prior P .
All tasks are learned through the same learning procedure.
The meta learner treats the prior P itself as a random variable and assumes the metaknowledge is in the form of a distribution over all possible priors.
Let hyperprior P be an initial distribution over priors, meta learner uses the observed tasks to adjust its original hyperprior P into hyperposterior Q from the learning process.
The quality of the hyperposterior Q is measured by the expected task error of learning new tasks using priors generated from it, which is formulated as:
[Equation]
Accordingly, the empirical counterpart of the above quantity is given by:
[Equation]
PAC-B AYES M ETA -L EARNING B OUND WITH G AUSSIAN R ANDOMIZATION
Based on the above deﬁnition, Pentina & Lampert (2014) and Amit & Meir (2018) present metalearning PAC-Bayes generalization bounds w.r.t.
hyperposterior Q by using McAllester’s single-taskbound (McAllester, 1999).
Here we present a new meta-learning generalization bound with Gaussian randomization by using Catoni’s bound in Eq.
In particular, the classiﬁer h is parameterizedas h w with w ∈ R d w .
The prior and posterior is a distribution over the set of all possible parametersw .
We choose both the prior P and posterior Q to be spherical Gaussians, i.e. P = N ( w P , σ 2 w I d w )and Q = N ( w Q , σ 2 w I d w ) .
The mean w P is a random variable distributed ﬁrst according to thehyperprior P , which we formulate as N (0 , σ 2 w I d w ) , and later according to hyperposterior Q , whichwe model as N ( w Q , σ 2 w I d w ) .
When encountering a new task i , we ﬁrst sample the mean of prior w Pi from the hyperposterior N ( w Q , σ 2 w I d w ) , and then use it as a basis to learn the mean of posterior w Qi =
A b ( S i , P ) , as shown in Figure 1(left).
Then, we could derive the following PAC-Bayesmeta-learning bound.
Theorem 2.
Consider the Meta-Learning (ML) framework, given the hyperprior P = N (0 , σ 2 v I d v ) , then for any hyperposterior Q , any c 1 , c 2 > 0 and any δ ∈ (0 , 1] with probability ≥ 1 − δ we have, er
[Equation]
where c (cid:48) 1 =
[Equation]
and c
[Equation]
=
[Equation]
Proof.
See Appendix B.3 for the proof.
Notice that the expected task generalization error is bounded by the empirical multi-task error plustwo complexity terms.
The ﬁrst term demonstrates the environment-complexity which convergesto zero if an inﬁnite number of tasks are observed from the task environment ( n → ∞ ), while thesecond is the task-complexity of the observed tasks which converges to zero when the sufﬁcientsamples in each task is observed ( m i → ∞ ).
Besides, the derived bound converges at the rateof O ( 1 m ) instead of O ( 1 √ m ) in (Pentina & Lampert, 2014; Amit & Meir, 2018), due to the use of
Catino’s bound.
PAC-B AYES L OCALIZED M ETA - LEARNING4. O VERALL F RAMEWORK
Our motivation stems from a core challenge in PAC-Bayes meta-learning bound in 41, wherein thecomplexity term (cid:80) ni =
[Equation]
Qi − w Q (cid:107) 2 is typically vital to the bound and so ﬁnding thetightest possible bound generally depends on minimizing this term.
It is obvious that the optimalw Q is
[Equation]
However, if the learned posteriors for each task are mutually exclusive, i.e.,one learned posterior has a negative effect on another task, this term could be inevitably large.w Q is the mean of hyperposterior Q and this term naturally indicates the divergence between themean of prior w Pi sampled from the hyperposterior Q and the mean of posterior w Qi in each task.
Therefore, we propose to adaptively choose the mean of prior w Pi according to task i .
It is obviousthat the complexity term vanishes if we set w Pi = w Qi , but the prior P i in each task has to bechosen independently of the sample set S i .
Fortunately, the PAC-Bayes theorem allows us to chooseprior upon the data distribution D i .
Therefore, we propose a prior predictor Φ : D m → w P whichreceives task data distribution D m and outputs the mean of prior w P .
In this way, the generatedpriors could focus locally on those regions of model parameters that are of particular interest insolving speciﬁc tasks.
Particularly, the prior predictor is parameterized as Φ v with v ∈ R d v .
We abuse notation P and
Q and assume v as a random variable distributed ﬁrst according to the hyperprior P , which wereformulate as N (0 , σ 2 v I d v ) , and later according to hyperposterior Q , which we reformulate as
N ( v Q , σ 2 v I d v ) .
Given a new task i , we ﬁrst sample v from hyperposterior N ( v Q , σ 2 v I d v ) andestimate the mean of prior w Pi by leveraging prior predictor w Pi =
Φ v ( D mi ) .
Then, the baselearner utilizes the sample set S i and the prior P i = N ( w Pi , σ 2 w I d w ) to produce a mean posteriorw Q i =
A b ( S i , P i ) , as illustrated in Figure 1(right).
4. LCC-B ASED P RIOR P REDICTOR
To make w P close to w Q in each task, the prior predictor is required to (i) uncover the tight relationship between the sample set and model parameter.
Intuitively, features and parameters yield similarlocal and global structures in their respective spaces in the classiﬁcation problem.
Features in the [Figure] same category tend to be spatially clustered together while maintaining the separation between different classes.
Take linear classiﬁers as an example, let w k be the parameters w.r.t.
category k , theseparability between classes is implemented as x · w k , which also explicitly encourages intra-classcompactness.
A reasonable choice of w k is to maximize the inner product distance with the inputfeatures in the same category and minimize the distance with the input features of the non-belongingcategories.
Besides, the prior predictor should be (ii) category-agnostic since it will be used continuously as new tasks and hence new categories become available.
Lastly, it should be (iii) invariantunder permutations of its inputs.
To satisfy the above conditions, we follow the idea of nearest class mean classiﬁer (Mensink et al., 2013), which represents class parameter by averaging its feature embeddings.
This idea has been explored in transductive few-shot learning problem (Bertinetto et al., 2016; Yang et al., 2018).
Snell et al.
(2017) learns a metric space across tasks such that when represented in this embedding, prototype (centroid) of each class can be used for label prediction in the new task.
Qiao et al.
(2018) directly predicts the classiﬁer weights using the activations by exploiting the close relationship between the parameters and the activations in a neural network associated with the same category.
In summary, the classiﬁcation problem of each task is transformed as a generic metric learning problem which is shared across tasks.
Once this mapping has been learned on observed tasks, due to the structure-preserving property, it could be easily generalized to new tasks.
Formally, let each task be a K -class classiﬁcation problem.
Then the parameter of the classiﬁer in task i is represented as w i =
[ w i [1] , . .
, w i [ k ] , . .
, w i [ K ]] .
The prior predictor for class k could be deﬁned as:
[Equation]
where φ v ( · ) : R d → R d w is the feature embedding function, m ik is the number of samples belonging to category k , S ik and D ik are the sample set and data distribution for category k in task i .
We call this function the expected prior predictor .
Since data distribution D ik is considered unknown and our only insight as to D ik is through the sample set S ik , we approximate the expected prior predictor by its empirical counterpart, based on m ik observed samples in the category k :
[Equation]
which we call the empirical prior predictor .
Although we can implement the embedding function φ v ( · ) with a multilayer perceptron (MLP), both input x and model parameter w are highdimensional, making the empirical prior predictor ˆΦ v ( · ) difﬁcult to learn.
According to Deﬁnition(3), any points on the latent manifold can be approximated by a linear combination of a set of anchorpoints.
Inspired by this, if the anchor points are sufﬁciently localized, the empirical prior predictor ˆΦ v ( S ) can also be approximated by a linear function w.r.t.
a set of codings.
Accordingly, wepropose an LCC-based prior predictor, which is deﬁned as:
[Equation]
where Φ v ( u ) ∈ R
d w is the feature embedding of base u ∈ R d .
As such, the parameters of LCC-based prior predictor w.r.t.
category k can be represented as v k =[Φ v k ( u 1 ) , Φ v k ( u 2 ) , . .
, Φ v k ( u | C | )] .
Lemma 1 illustrates the approximation error.
[Figure]
Lemma 1.
(Empirical Pior Predictor Approximation)
Given the deﬁnition of ˆ w Pi [ k ] and ¯ w Pi [ k ] in
Eq.
(6) and Eq.
(7), let ( γ, C ) be an arbitrary coordinate coding on R d and φ be an ( α, β ) -Lipschitz smooth function.
We have for all x ∈ R d
[Equation]
where ¯ x j = (cid:80) u ∈ C γ u ( x j ) u .
Then given any (cid:15) > 0 , there exists a coding ( γ, C ) such that
[Equation]
Proof.
See appendix B.1 for the proof.
The ﬁrst inequality of Lemma 1 demonstrates that a good LCC-based prior predictor should make xclose to its physical approximation ¯ x and should be localized.
The second and third inequality showthat if a set of anchor points C has cardinality O ( d M N ( (cid:15), M )) , emprical prior predictor can belinearly approximated using LCC up to accuracy O ( √ d M (cid:15) 2 ) .
The complexity of the LCC codingscheme depends only on the number of anchor points | C | instead of the input dimension.
In fact, asmall | C | is usually sufﬁcient to achieve good approximation.
Optimization of LCC.
We minimize the ﬁrst inequality in (8) to obtain a set of anchor points.
As with (Yu et al., 2009), we simplify the localization error term by assuming ¯ x = x , and then we optimize the following objective function:
[Equation]
where ¯ x = (cid:80) u ∈ C γ u ( x ) u .
In practice, we update C and γ by alternately optimizing a LASSOproblem and a least-square regression problem, respectively.
4. PAC-B AYES L OCALIZED M ETA -L EARNING B OUND WITH G AUSSIAN R ANDOMIZATION
In order to derive a PAC-Bayes generalization bound for localized meta-learning, we ﬁrst bound theapproximation error between expected prior predictor and LCC-based prior predictor.
Lemma 2.
Given the deﬁnition of w P and ¯ w P in Eq.
(5) and (7), let X be a compact set with radius R , i.e., ∀ x , x (cid:48)
∈ X , (cid:107) x − x (cid:48) (cid:107) ≤ R .
For any δ ∈ (0 , 1] with probability ≥ 1 − δ , we have
[Equation]
Proof.
See appendix B.2 for the proof.
Lemma 2 shows that the approximation error between expected prior predictor and LCC-basedprior predictor depends on (i) the concentration of prior predictor and (ii) the quality of LCC codingscheme.
The ﬁrst term implies the number of samples for each category should be larger for betterapproximation.
This is consistent with the results of estimating the center of mass (Cristianini &
Shawe-Taylor, 2004).
Based on Lemma 2, we have the following PAC-Bayes LML bound.
Theorem 3.
Consider the Localized Meta-Learning (LML) framework, give the hyperprior P =
N (0 , σ 2 v I d v ) , then for any hyperposterior Q , any c 1 , c 2 > 0 and any δ ∈ (0 , 1] with probability ≥ 1 − δ we have,
[Equation]
where c (cid:48)
= c 1 1 − e − c 1 and c (cid:48)
= c 2 1 − e − c 2 .
To get a better understanding, we further simplify the notation and obtain that
[Equation]
Proof.
See appendix B.3 for the proof.
Similarly with the PAC-Bayes meta-learning bound in Theorem 2 and the bounds in (Pentina & Lampert, 2014; Amit & Meir, 2018), the expected task error er ( Q ) is bounded by the empiricaltask error ˆ er ( Q ) plus the task-complexity and environment-complexity terms.
The main innovationhere is to exploit the potential to choose the mean of prior w P based on task data S .
Intuitively, ifthe selection of the LCC-based prior predictor is appropriate, it will narrow the divergence betweenthe mean of prior w Pi sampled from the hyperposterior Q and the mean of posterior w Qi in eachtask.
Therefore, the bound can be tighter than the ones in the meta-learning framework.
Our empirical study in Section 5 illustrates that the algorithms derived from this bound can achieve betterperformance than the methods derived from standard PAC-Bayes meta-learning bounds.
When one is choosing the LCC-based prior predictor ¯Φ v ( · ) , the number of anchor points | C | , thereis a balance between accuracy and simplicity.
As we increase | C | , it will essentially increase theexpressive power of ¯Φ v ( · ) and reduce the complexity term (cid:107) E v w Q − ¯Φ v Q ( S ) (cid:107) 2 .
However, at thesame time, it will increase the complexity term (cid:107) v Q (cid:107) 2 and make the bound loose.
If we set | C | to1 , it is degraded to the regular meta-learning framework.
4. L OCALIZED M ETA -L EARNING A LGORITHM
Since the bound in (27) holds uniformly w.r.t.
Q , the guarantees of Theorem 3 also hold for the resulting learned hyperposterior Q = N ( v Q , σ 2 v I d v ) , so the mean of prior w P sampled from the learned hyperposterior work well for future tasks.
The PAC-Bayes localized meta-learning bound in (27) can be compactly written as
[Equation]
where α 1 , α 2 > 0 are hyperparameters.
For task i , the learning algorithm
A b ( · ) can be formulated as w (cid:63)i = arg min w Qi E v
ˆ er i ( Q i = N ( w Qi , σ 2 w I d w )) .
Following Amit & Meir (2018), we jointly opti- mize the parameters of LCC-based prior predictor v and the parameters of classiﬁers in each task w 1 , w 2 , . . . , w n , which is formulated as
[Equation]
We can optimize v and w via mini-batch SGD.
The details of algorithms for meta-training aregiven in Algorithms 1. The expectation over Gaussian distribution and its gradient can be efﬁciently Algorithm Input: Data sets of observed tasks: S
, S n .
Output: Learned prior predictor ¯Φ parameterized by v .
Initialize v ∈ R d v and w i ∈ R d w for i = 1 . . . , n . Construct LCC scheme ( γ, C ) from the whole training data by optimizing Eq.
while not converged do for each task i ∈ { 1 , . .
. , n } do
Sample a random mini-batch from the data S (cid:48) i ⊂ S i .
Approximate E
v ˆ er i ( w i ) using S (cid:48) i .
end for
Compute the objective in (15), i.e. J ← (cid:80) ni
=1 E v
ˆ er i ( w {+i ) + α 1 (cid:107) v Q (cid:107) 
 + (cid:80) ni =
+}
α 2 m i (cid:107) E v w Qi − ¯Φ v Q ( S i ) (cid:107) 2 . Evaluate the gradient of J w.r.t. { v , w 1 , . . . , w n } using backpropagation.
Take an optimization step.
end while 207 estimated by using the re-parameterization trick (Kingma & Welling, 2014; Rezende et al., 2014).
For example, to sample w from the posterior Q = N ( w Q , σ 2 w I d w )
[Figure]
we ﬁrst draw ξ ∼ N (0 , I d w )and then apply the deterministic function w Q + ξ (cid:12) σ , where (cid:12) is an element-wise multiplication
[Figure]
E XPERIMENTS
[Figure] [Figure]
5. D ATASETS AND S ETUP
We use CIFAR-100 and Caltech-256 in our experiments.
CIFAR-100 (Krizhevsky, 2009) contains60,000 images from 100 ﬁne-grained categories and 20 coarse-level categories.
As in (Zhou et al.,
2018), we use 64, 16, and 20 classes for meta-training, meta-validation, and meta-testing, respectively.
Caltech-256 has 30,607 color images from 256 classes (Grifﬁn et al., 2007).
Similarly, wesplit the dataset into 150, 56 and 50 classes for meta-training, meta-validation, and meta-testing.
Weconsider 5-way classiﬁcation problem.
Each task is generated by randomly sampling 5 categoriesand each category contains 50 samples.
The base model uses the convolutional architecture in (Finnet al., 2017), which consists of 4 convolutional layers, each with 32 ﬁlters and a fully-connectedlayer mapping to the number of classes on top.
High dimensional data often lies on some low dimensional manifolds.
We utilize an auto-encoder to extract the semantic information of image dataand then construct the LCC scheme based on the embeddings.
The parameters of prior predictor andbase model are random perturbations in the form of Gaussian distribution.
We design two differentmeta-learning environment setting to validate the efﬁcacy of the proposed method. The ﬁrst one usesthe pre-trained base model as an initialization, which utilizes all the meta-training classes (64-classclassiﬁcation in CIFAR-100 case) to train the feature extractor. The second one uses the randominitialization.
We compare the proposed LML method with ML-PL method (Pentina & Lampert,2014), ML-AM method (Amit & Meir, 2018) and ML-A which is derived from Theorem 2.
In thesemethods, we use their main theorems about the PAC-Bayes generalization bound to derive the objective for the algorithm.
We also compare with two typical meta-learning few-shot learning methods:
MAML (Finn et al., 2017) and MatchingNet (Vinyals et al., 2016).
To ensure a fair comparison, allapproaches adopt the same network architecture and pre-trained feature extractor.
5. R ESULTS
[Table] In Figure 3, we demonstrate the average test error of learning a new task based on the number oftraining tasks in different settings (with or without a pre-trained feature extractor).
It is obviousthat the performance continually increases as we increase the number of training tasks for all themethods.
This is consistent with the generalization bounds that the complexity term converges tozero if large numbers of tasks are observed.
ML-A consistently outperforms ML-PL and ML-AMsince the bound w.r.t.
ML-A in Theorem 2 converges at the rate of O ( 1 m ) while the bounds w.r.t.
ML-PL and ML-AM converge at the rate of O ( 1 √ m ) .
This demonstrates the importance of using tightgeneralization bound.
Our proposed LML signiﬁcantly outperforms the baselines, which validatesthe effectiveness of the proposed LCC-based prior predictor.
It is a more suitable representation formeta-knowledge than the traditional global hyperposterior in ML-A, ML-AM, and ML-PL.
Moreover, we can ﬁnd that all PAC-Bayes baselines outperform MAML and MatchingNet.
Notethat MAML and MatchingNet adopt the episodic training paradigm to solve the few-shot learning problem.
The meta-training process requires millions of tasks and each task contains limitedsamples, which is not the case in our experiment.
Scarce tasks in meta-training leads to severelymeta-overﬁtting.
In our method, the learned prior serves both as an initialization of base model andas a regularizer which restricts the solution space while allowing variation based on speciﬁc taskdata.
It yields a model with smaller error than its unbiased counterpart when applied to a similartask. Finally, we observe that if the pre-trained feature extractor is provided, all of these methods do betterthan meta-training with random initialization. This is because pre-trained feature extractor can beregarded as a data-dependent hyperpior. It is closer to the hyperposteior than the randomly initializedhyperprior.
Therefore, it reduces the environment complexity term and improves the generalizationperformance.
In Figure 4(b), we show the divergence between the mean of generated prior w P from meta modeland the mean of learned posterior w Q for LML and
ML-A.
This further validates the effectivenessof the LCC-based prior predictor which could narrow the divergence term and thus tight the bound.
In Figure 4(a), we vary the number of bases | C | in LCC scheme from 4 to 256 , the optimal value isaround 64 in both datasets.
This indicates that LML is sensitive to the number of bases | C | , whichfurther affects the quality of LCC-based prior predictor and the performance of LML.
R ELATED W ORK
Meta-Learning.
Meta-learning literature commonly considers the empirical task error by directlyoptimizing a loss of meta learner across tasks in the training data.
Recently, this has been success- fully applied in a variety of models for few-shot learning (Ravi & Larochelle, 2017; Snell et al.
,2017; Finn et al., 2017; Vinyals et al., 2016).
Although Vuorio et al. ; Rusu et al.
Zintgraf et al.
; Wang et al.
(2019) consider task adaptation when using meta-knowledge forspeciﬁc tasks, all of them are not based on generalization error bounds, which is the focus of ourwork.
Meta-learning in the online setting has regained attention recently (Denevi et al., 2018b;a;2019; Balcan et al., 2019), in which online-to-batch conversion results could imply generalizationbounds. Galanti et al.
analyzes transfer learning in neural networks with PAC-Bayes tools.
Most related to our work are (Pentina & Lampert, 2014; Amit & Meir, 2018) which provide a PAC-Bayes generalization bound for meta-learning framework.
In contrast, neither work considerslocalized meta-knowledge for speciﬁc tasks.
Localized PAC-Bayes Learning.
There has been a prosperous line of research for learning priorsto improve the PAC-Bayes bounds Catoni (2007); Guedj (2019).
(Parrado-Hern´andez et al., 2012)showed that priors can be learned by splitting the available training data into two parts, one forlearning the prior, one for learning the posterior.
(Lever et al., 2013) derived an expression for theoverall best prior, i.e. the distribution resulting in the smallest possible bound value and boundedthe KL divergence by a term independent of data distribution.
Recently, (Rivasplata et al., 2018)bounded the KL divergence by investigating the stability of the hypothesis.
(Dziugaite & Roy, 2018)optimized the prior term in a differentially private way.
In summary, theses methods construct somequantities that reﬂect the underlying data distribution, rather than the sample set, and then choosethe prior P based on these quantities.
These works, however, are only applicable for single-taskproblem and could not transfer knowledge across tasks in meta-learning setting.
C ONCLUSION
This work contributes a novel localized meta-learning framework from a theoretical perspective.
Wepropose a generalization bound based on PAC-Bayes theory with Gaussian randomization.
Insteadof formulating meta-knowledge as a global distribution, we propose an LCC-based prior predictorto output local meta-knowledge by using task information.
We further develop a practical algorithmwith deep neural network based on the bound.
An interesting topic for future work would be toexplore other principle to construct the prior predictor and apply the localized meta-learning framework to a more realistic scenario that tasks are sampled non-i i d. from an environment. Anotherchallenging problem is to extend our techniques to derive localized meta-learning algorithms forregression and reinforcement learning problem
A N OTATIONS
Let φ v ( · ) :
R d → R d w be the feature embedding function.
m ik denotes the number of samples belonging to category k .
S ik and D ik are the sample set and data distribution for category k in task i , respectively.
Then, the expected prior predictor w.r.t.
class k in task i is deﬁned as:
[Equation]
The empirical prior predictor w.r.t.
class k in task i is deﬁned as:
[Equation]
The LCC-based prior predictor w.r.t.
class k in task i is deﬁned as:
[Equation]
B T HEORETICAL R ESULTSB. P ROOF OF L EMMA 1
This lemma bounds the error between the empirical prior predictor ˆ w Pi [ k ] and the LCC-based priorpredictor ¯ w Pi [ k ] .
Lemma
Given the deﬁnition of ˆ w Pi [ k ] and ¯ w Pi [ k ] in Eq.
(6) and Eq.
(7), let ( γ, C ) be an arbitrary coordinate coding on R d x and φ be an ( α, β ) -Lipschitz smooth function and .
We have for all x ∈ R d x
[Equation]
where ¯ x j = (cid:80) u ∈ C γ u ( x j ) u .
Then given any (cid:15) > 0 , there exists a coding ( γ, C ) such that
[Equation]
Proof.
Let ¯ x j = (cid:80) u ∈ C γ u ( x j ) u .
We have (cid:107) ˆΦ v ( S ik ) − ¯Φ v ( S ik ) (cid:107)=m ik (cid:88) x
j ∈ S ik (cid:107) Φ v ( x j ) − (cid:88)
u ∈ C γ u ( x j )Φ v ( u ) (cid:107)≤m ik (cid:88) x j ∈ S
ik (cid:16) (cid:107) Φ v ( x j ) − Φ v (¯ x j ) (cid:107) 2 + (cid:107) (cid:88) [Table] =m ik (cid:88) x
[Table] S
[Table]
≤m ik (cid:88) x j [Table] S [Table]
(cid:16) (cid:107) Φ v ( x j ) − Φ v (¯ x j )
[Table]
+ (cid:88)
[Table] | [Table]
u (
[Table]
) |
[Table]
Φ v
[Table]
[Table]
≤m ik (cid:88) x j
[Table]
S ik (cid:16) α (cid:107) x
j − ¯ x j
[Table]
[Equation]
In the above derivation, the ﬁrst inequality holds by the triangle inequality.
The second equalityholds since (cid:80) u ∈ C γ u ( x j ) = 1 for all x j .
The last inequality uses the assumption of ( α, β ) -Lipschitzsmoothness of Φ v ( · ) .
According to the Manifold Coding Theorem in (Yu et al., 2009), if the data points x lie on a compact smooth manifold M .
Then given any (cid:15) > 0 , there exists anchor points C ⊂ M and coding γ such that
[Equation]
This implies the desired bound.
The ﬁrst inequality of this lemma demonstrates that the quality of LCC approximation is boundedby two terms: the ﬁrst term (cid:107) x j − ¯ x j (cid:107) 2 indicates x should be close to its physical approximation¯ x , the second term (cid:107) ¯ x j − u (cid:107) implies that the coding should be localized.
The second and thirdinequality show that the approximation error of local coordinate coding depends on the intrinsic dimension of the manifold instead of the dimension of input.
If a set of anchor points C has cardinality
O ( d M N ( (cid:15), M )) , emprical prior predictor can be linearly approximated using LCC up to accuracy
O ( √ d M (cid:15) 2 ) .
B. P ROOF OF L EMMA 2
In order to proof Lemma 2, we ﬁrst introduce a relevant theorem.
Theorem 4.
( Vector-valued extension of McDiarmid’s inequality (Rivasplata et al., 2018) ) Let
X 1 , . . .
X m ∈
X be independent random variables, and f : X m → R d w be a vector-valued mapping function.
If, for all i ∈ { 1 , . . .
, m } , and for all x 1 , . .
. , x m , x (cid:48)
i ∈ X , the function f satisﬁes sup
[Equation]
i
[Equation]
Then E (cid:107) f
( X 1: m ) − E
[ f ( X 1: m )] (cid:107) ≤ (cid:112)(cid:80) mi
{+=
+}
c 2 i .
For any δ ∈ (0 , 1) with probability ≥ 1 − δ we have
[Equation]
The above theorem indicates that bounded differences in norm implies the concentration of f ( X 1: m )around its mean in norm, i.e., (cid:107) f
( X 1: m ) − E
[ f ( X 1: m )] (cid:107) is small with high probability.
Then, we bound the error between expected prior predictor w Pi and the empirical prior predictorˆ w Pi .
Lemma 3.
Given the deﬁnition of w Pi [ k ] and ˆ w Pi [ k ] in (5) and (6), let X be a compact set with radius R , i.e., ∀ x , x (cid:48)
∈ X , (cid:107) x − x (cid:48) (cid:107) ≤ R .
For any δ ∈ (0 , 1] with probability ≥ 1 − δ , we have
[Equation]
Proof.
According to the deﬁnition of ˆΦ v ( · ) in (6), for all points x 1 , . . .
, x j − 1 , x j +1 , . . .
x m k , x (cid:48)
j in the sample set S ik , we have
[Equation]
where R denotes the domain of x , say R = sup x (cid:107) x (cid:107) .
The ﬁrst inequality follows from the Lipschitz smoothness condition of Φ v ( · ) and the second inequality follows by the deﬁnition of domain X .
Utilizing Theorem 4, for any δ ∈ (0 , 1] with probability ≥ 1 − δ we have
[Equation]
This implies the bound.
Lemma 3 shows that the bounded difference of function Φ v ( · ) implies its concentration, which canbe further used to bound the differences between empirical prior predictor ¯ w Pi [ k ] and expected priorpredictor w Pi [ k ] . Now, we bound the error between expected prior predictor w Pi and the LCC-basedprior predictor ¯ w Pi .
Lemma
Given the deﬁnition of w Pi and ¯ w Pi in (5) and (7), let X be a compact set with radius R , i.e., ∀ x , x (cid:48)
∈ X , (cid:107) x − x (cid:48) (cid:107) ≤ R .
For any δ ∈ (0 , 1] with probability ≥ 1 − δ , we have
[Equation]
Proof According to the deﬁnition of w P , ¯ w P and ˆ w P , we have [Equation] = K
[Equation]
k
[Equation]
=
[Equation]
k
[Equation]
+
[Equation]
=
K
[Equation]
k
[Equation]
(cid:16)
[Equation]
E
[Equation]
ˆΦ v (
[Equation]
Substitute Lemma 3 and Lemma 1 into the above inequality
[Equation]
we can derive
[Equation]
This gives the assertion.
Lemma 2 shows that the approximation error between expected prior predictor and LCC-based priorpredictor depends on the number of samples in each category and the quality of the LCC codingscheme.
B. P ROOF OF T HEOREM 3
Theorem 3 Let Q be the posterior of base learner Q = N ( w Q , σ 2 w I d w ) and P be the prior
N (¯Φ v ( S ) , σ 2 w I d w ) .
The mean of prior is produced by the LCC-based prior predictor ¯Φ v ( S ) in
Eq.
(7) and its parameter v is sampled from the hyperposterior of meta learner Q = N ( v Q , σ 2 v I d v ) .
Give the hyperprior P = N (0 , σ 2 v I d v ) , then for any hyperposterior Q , any c 1 , c 2 > 0 and any δ ∈ (0 , 1] with probability ≥ 1 − δ we have, [Equation]
[Equation]
where c (cid:48) {+
+}
= c 1 1 − e − c 1 and c (cid:48) {+
+} = c 2 1 − e − c 2 .
We can simplify the notation and obtain that
[Equation]
Proof Our proof contains two steps.
First, we bound the error within observed tasks due to observing a limited number of samples.
Then we bound the error on the task environment level dueto observing a ﬁnite number of tasks.
Both of the two steps utilize Catoni’s classical PAC-Bayesbound (Catoni, 2007) to measure the error.
We give here a general statement of the Catoni’s classical PAC-Bayes bound.
Theorem 5.
( Classical PAC-Bayes bound, general notations )
Let X be a sample space and X be some distribution over X , and let F be a hypotheses space of functions over X .
Deﬁne a loss function g ( f, X ) : F × X → [0 , 1] , and let X G 1 (cid:44) { X 1 , . . .
, X G } be a sequence of K independent random variables distributed according to X .
Let π be some prior distribution over F (which must not depend on the samples X 1 , . . .
, X k ).
For any δ ∈ (0 , 1] , the following bounds holds uniformly for all posterior distribution ρ over F (even sample dependent),
[Equation]
First step We utilize Theorem 5 to bound the generalization error in each of the observed tasks.
Let i ∈ 1 , . . .
, n be the index of task.
For task i, we substitute the following deﬁnition intothe Catoni’s PAC-Bayes Bound.
Speciﬁcally, X g (cid:44) ( x ij , y ij ) , K (cid:44) m i denote the samples and
X (cid:44) D i denotes the data distribution.
We instantiate the hypotheses with a hierarchical modelf (cid:44) ( v , w ) , where v ∈ R d v and w ∈ R d w are the parameters of meta learner (prior predictor)
Φ v ( · ) and base learner h ( · ) respectively.
The loss function only considers the base learner,which is deﬁned as g ( f, X ) (cid:44) (cid:96) ( h w ( x ) , y ) .
The prior over model parameter is representedas π (cid:44) ( P , P ) (cid:44) ( N (0 , σ 2 v I d v ) , N ( w P , σ 2 w I d w )) , a Gaussian distribution (hyperprior of meta learner) centered at 0 and a Gaussian distribution (prior of base learner) centered at w P , respectively.
We set the posterior to ρ (cid:44) ( Q , Q ) (cid:44) ( N ( v Q , σ 2 v I d v ) , N ( w Q , σ 2 w I d w )) , a Gaussian distribution (hyperposterior of meta learner) centered at v Q and a Gaussian distribution (posterior ofbase learner) centered at w Q .
According to Theorem 5, the generalization bound holds for anyposterior distribution including the one generated in our localized meta-learning framework.
Specifically, we ﬁrst sample v from hyperposterior N ( v Q , σ 2 v I d v ) and estimate w P by leveraging expected prior predictor w P =
Φ v ( D ) .
The base learner algorithm
A b ( S, P ) utilizes the sample set
S and the prior P = N ( w P , σ 2 w I d w ) to produce a posterior Q =
A b ( S, P ) =
N ( w Q , σ 2 w I d w ) .
Then we sample base learner parameter w from posterior N ( w Q , σ 2 w I d w ) and compute the incurred loss (cid:96) ( h w ( x ) , y ) .
On the whole, meta-learning algorithm A m ( S 1 , . . .
, S n , P ) observesa series of tasks S 1 , . .
S n and adjusts its hyperprior P = N ( v P , σ 2 v I d v ) into hyperposterior
Q =
A m ( S 1 , . . .
, S n , P ) =
N ( v Q , σ 2 v I d v ) .
The KL divergence term between prior π and posterior ρ is computed as follows:
[Equation]
In our localized meta-learning framework
[Equation]
in order to make KL ( Q || P ) small [Equation] the center of prior distribution w P is generated by the expected prior predictor w P =
Φ v ( D ) . However [Equation] the data distribution D is considered unknown and our only insight as to D ik is through the sample set
S ik . In this work
[Equation]
we approximate the expected prior predictor Φ v ( D ) with the LCC-based prior predictor ¯ w P = ¯Φ v ( S ) . Denote the term
[Equation] for convenience [Equation]
we have
[Equation]
Since ¯ w Pi = ¯Φ v ( S i ) =
[¯Φ v ( S i 1 ) , . .
, ¯Φ v ( S ik ) , . . . , ¯Φ v ( S iK )]
[Equation]
we have
[Equation]
[Equation]
where V v [ (cid:107) ¯Φ v ( S ik ) (cid:107) ] denotes the variance of (cid:107) ¯Φ v ( S ik ) (cid:107) .
The last equality uses the fact that d v = | C | d w .
Combining Lemma 2, for any δ (cid:48) ∈ (0 , 1] with probability ≥ 1 − δ (cid:48)
we have
[Equation]
Then, according to Theorem [Equation] we obtain that for any δ i 2 >
[Equation]
for all observed tasks i = 1 , . .
. , n .
Deﬁne δ (cid:48)
=
δ i 2 and combine inequality (33), we obtain
[Equation]
Using the notations in Section
[Equation]
the above bound can be simpliﬁed as
[Equation]
Second step Next we bound the error due to observing a limited number of tasks from the environment.
We reuse Theorem 5 with the following substitutions.
The samples are ( D i , m i , S i ) , i = 1 , . . . , n , where ( D i , m i ) are sampled from the same meta distribution τ and S i ∼ D
m i i .
The hyposthesis is parameterized as Φ v ( D ) with meta learner parameter v .
The loss function is g ( f, X ) (cid:44)
[Equation]
h w
[Equation]
x ) [Equation] y ) , where w Q =
A b ( S i , P i ) .
Let π (cid:44) N
[Equation]
0 , σ 2 v I d v ) be
[Equation]
Using the term in Section
[Equation]
the above bound can be simpliﬁed as
[Equation]
Finally, by employing the union bound
[Equation]
we could bound the probability of the intersection of the events in (36) and (38) For any δ > 0 , set δ 0 (cid:44) δ 2 and δ i (cid:44) δ 2 n for i = 1 , . . .
, n , we have
[Equation]
We can further simplify the notation and obtain that
[Equation]
const (
[Equation]
(40) where c (cid:48) {+
+}
= c 1 1 − e − c 1 and c (cid:48)
= c 2 1 − e − c 2 .
This completes the proof.
B. P ROOF OF T HEOREM
Theorem 2 Let Q be the posterior of base learner Q = N ( w Q , σ 2 w I d w ) and P be the prior
N ( w P , σ 2 w I d w ) .
The mean of prior is sampled from the hyperposterior of meta learner Q =
N ( w Q , σ 2 w I d w ) .
Give the hyperprior P = N (0 , σ 2 w I d w ) , then for any hyperposterior Q , any c 1 , c 2 > 0 and any δ ∈ (0 , 1] with probability ≥ 1 − δ we have,
[Equation]
where c (cid:48) 1 =
c
[Equation]
1 and c (cid:48) 2 = c
[Equation]
c 2 .
Proof Instead of generating the mean of prior with a prior predictor, the vanilla meta-learning framework directly produces the mean of prior w P by sampling from hyperposterior Q =
N ( w Q , σ 2 w I d w ) .
Then the base learner algorithm A b ( S, P ) utilizes the sample set S and the prior
P = N ( w P , σ 2 w I d w ) to produce a posterior Q =
A b ( S, P ) =
N ( w Q , σ 2 w I d w ) .
Similarly with the two-steps proof in Theorem 3, we ﬁrst get an intra-task bound by using Theorem 5.
For any δ i > 0 , we have
[Equation]
The term
E
[Equation]
Pi
[Equation]
2 can be simpliﬁed as
[Equation]
where V w Pi [ (cid:107) w Pi (cid:107) ] denotes the variance of (cid:107) w Pi (cid:107) .
Then we get an inter-task bound.
For any δ 0 > 0 , we have
P
[Equation]
∼ D mii ,i =1 ,...,n (cid:40) E
[Equation]
For any δ > 0 , set δ 0 (cid:44) δ 2 and δ i (cid:44) δ 2 n for i = 1 , . . . , n .
Using the union bound, we ﬁnally get
P
[Equation]
Similarly
[Equation]
we can further simplify the notation and obtain that
[Equation] i = c (cid:48) 1 c (cid:48)2 c 2 nm i σ 2 w (cid:107) E w P w Qi − w Q (cid:107)+ const ( δ, n, m i ) , ∀Q (cid:41) ≥ 1 − δ, (46) where c (cid:48)
1 =
c 1 1 − e − c 1 and c (cid:48)
= c 2 1 − e − c 2 .
This completes the proof.
C D ETAILS OF E XPERIMENTSC. D ATA P REPARATION
We used the 5-way 50-shot classiﬁcation setups, where each task instance involves classifying images from 5 different categories sampled randomly from one of the meta-sets.
We did not employany data augmentation or feature averaging during meta-training, or any other data apart from thecorresponding training and validation meta-sets.
C. N ETWORK A RCHITECHTURE
Auto-Encoder for LCC For CIFAR100, the encoder is 7 layers with 16-32-64-64-128-128-256channels.
Each convolutional layer is followed by a LeakyReLU activation and a batch normalization layer.
The 1st, 3rd and 5th layer have stride 1 and kernel size (3 , 3) .
The 2nd, 4th and 6th layerhave stride 2 and kernel size (4 , 4) .
The 7th layer has stride 1 and kernel size (4 , 4) .
The decoder isthe same as encoder except that the layers are in reverse order.
The input is resized to 32 × 32 .
For
Caltech-256, the encoder is 5 layers with 32-64-128-256-256 channels.
Each convolutional layer isfollowed by a LeakyReLU activation and a batch normalization layer.
The ﬁrst 4 layers have stride2 and kernel size (4 , 4) .
The last layer has stride 1 and kernel size (6 , 6) .
The decoder is the sameas encoder except that the layers are in reverse order.
The input is resized to 96 × 96 .
Base Model
The network architecture used for the classiﬁcation task is a small CNN with 4 convolutional layers, each with 32 ﬁlters, and a linear output layer, similar to (Finn et al., 2017).
Eachconvolutional layer is followed by a Batch Normalization layer, a Leaky ReLU layer, and a maxpooling layer.
For CIFAR100, the input is resized to 32 × 32 .
For Caltech-256, the input is resizedto 96 × 96 .
C. O PTIMIZATION
Auto-Encoder for LCC
As optimizer we used Adam(Kingma & Ba, 2015) with β 1 = 0 .
9 andβ 2 = 0 .
The initial learning rate is 1 × 10 − 4 .
The number of epochs is 100.
The batch size is512.
LCC Training We alternatively train the coefﬁcients and bases of LCC with Adam with β 1 = 0 .
9and β 2 = 0 .
In speciﬁcs, for both datasets, we alternatively update the coefﬁcients for 60 timesand then update the bases for 60 times.
The number of training epochs is 3.The number of bases is64.
The batch size is 256.
Pre-Training of Feature Extractor
We use a 64-way classiﬁcation in CIFAR-100 and 150-wayclassiﬁcation in Caltech-256 to pre-train the feature embedding only on the meta-training dataset.
For both CIFAR100 and Caltech-256, an L2 regularization term of 5 e − 4 was used.
We used the Adam optimizer.
The initial learning rate is 1 × 10 − 3 , β 1 is 0.9 and β 2 is 0.999.
The number ofepochs is 50.
The batch size is 512.
Meta-Training We use the cross-entropy loss as in (Amit & Meir, 2018).
Although this is inconsistent with the bounded loss setting in our theoretical framework, we can still have a guarantee on avariation of the loss which is clipped to [0 , 1] .
In practice, the loss is almost always smaller than one.
For CIFAR100 and Caltech-256, the number of epochs of meta-training phase is 12; the number ofepochs of meta-testing phase is 40.
The batch size is 32 for both datasets.
As optimizer we used
Adam with β 1 = 0 .
9 and β 2 = 0 .
In the setting with a pre-trained base model, the learning rateis 1 × 10 − 5 for convolutional layers and 5 × 10 − 4 for the linear output layer.
In the setting withouta pre-trained base model, the learning rate is 1 × 10 − 3 for convolutional layers and 5 × 10 − 3 for thelinear output layer.
The conﬁdence parameter is chosen to be δ = 0 .
The variance hyper-parameterfor prior predictor and base model are σ w = σ v = 0 .
The hyperparameter α 1 , α 2 in LML and
ML-A are set to 0 .
[-=1 E v ˆ er i ( w i ) + α 1 (cid:107) v Q (cid:107) 
 + (cid:80) ni-]For MAML (Finn et al., 2017) and MatchingNet (Vinyals et al., 2016).
[-=
-]
Bothtwo methods use the Adam optimizer with initial learning rate 0 . 0001 .
In the meta-training phase,we randomly split the samples of each class into support set (5 samples) and query set (45 samples).
The number of epochs is 100.
For MAML, the learning rate of inner update is 0 .