H OW I MPORTANT IS THE T RAIN -V ALIDATION S PLIT IN M ETA -L EARNING ?
Anonymous authors
Paper under double-blind review
A BSTRACT
Meta-learning aims to perform fast adaptation on a new task through learning a “prior” from multiple existing tasks.
A common practice in meta-learning is to perform a train-validation split where the prior adapts to the task on one split of the data, and the resulting predictor is evaluated on another split.
Despite its prevalence, the importance of the train-validation split is not well understood either in theory or in practice, particularly in comparison to the more direct non-splitting method, which uses all the per-task data for both training and evaluation.
We provide a detailed theoretical study on whether and when the train-validation split is helpful on the linear centroid meta-learning problem, in the asymptotic setting where the number of tasks goes to inﬁnity.
We show that the splitting method converges to the optimal prior as expected, whereas the non-splitting method does not in general without structural assumptions on the data.
In contrast, if the data are generated from linear models (the realizable regime), we show that both the splitting and non-splitting methods converge to the optimal prior.
Further, perhaps surprisingly, our main result shows that the non-splitting method achieves a strictly better asymptotic excess risk under this data distribution, even when the regularization parameter and split ratio are optimally tuned for both methods.
Our results highlight that data splitting may not always be preferable, especially when the data is realizable by the model.
We validate our theories by experimentally showing that the non-splitting method can indeed outperform the splitting method, on both simulations and real meta-learning tasks
I NTRODUCTION
Meta-learning, also known as “learning to learn”, has recently emerged as a powerful paradigm for learning to adapt to unseen tasks (Schmidhuber, 1987).
The high-level methodology in metalearning is akin to how human beings learn new skills, which is typically done by relating to certain prior experience that makes the learning process easier.
More concretely, meta-learning does not train one model for each individual task, but rather learns a “prior” model from multiple existing tasks so that it is able to quickly adapt to unseen new tasks.
Meta-learning has been successfully applied to many real problems, including few-shot image classiﬁcation (Finn et al., 2017; Snell et al., 2017), hyper-parameter optimization (Franceschi et al., 2018), low-resource machine translation (Gu et al., 2018) and short event sequence modeling (Xie et al., 2019).
A common practice in meta-learning algorithms is to perform a sample splitting , where the data within each task is divided into a training split which the prior uses to adapt to a task-speciﬁc predictor, and a validation split on which we evaluate the performance of the task-speciﬁc predictor (Nichol et al., 2018; Rajeswaran et al., 2019; Fallah et al., 2020; Wang et al., 2020a).
For example, in a 5-way k -shot image classiﬁcation task, standard meta-learning algorithms such as
MAML (Finn et al., 2017) use 5 k examples within each task as training data, and use additional examples (e.g. k images, one for each class) as validation data.
This sample splitting is believed to be crucial as it matches the evaluation criterion at meta-test time, where we perform adaptation on training data from a new task but evaluate its performance on unseen data from the same task.
Despite the aformentioned importance, performing the train-validation split has a potential drawback from the data efﬁciency perspective — Because of the split, neither the training nor the evaluation stage is able to use all the available per-task data.
In the few-shot image classiﬁcation example, each task has a total of 6 k examples available, but the train-validation split forces us to use these data separately in the two stages.
Meanwhile, performing the train-validation split is also not the only option in practice: there exist algorithms such as Reptile (Nichol & Schulman, 2018) and
Meta-MinibatchProx
(Zhou et al., 2019) that can instead use all the per-task data for training the task-speciﬁc predictor and also perform well empirically on benchmark tasks.
These algorithms modify the loss function in the outer loop so that the training loss no longer matches the meta-test loss, but may have the advantage in terms of data efﬁciency for the overall problem of learning the best prior.
So far it is theoretically unclear how these two approaches (with/without train-validation split) compare with each other, which motivates us to ask the following
Question :
Is the train-validation split necessary and optimal in meta-learning?
In this paper, we perform a detailed theoretical study on the importance of the train-validation split.
We consider the linear centroid meta-learning problem (Denevi et al., 2018b), where for each task we learn a linear predictor that is close to a common centroid in the inner loop, and ﬁnd the best centroid in the outer loop (see Section 2 for the detailed problem setup).
This problem captures the essence of meta-learning with non-linear models (such as neural networks) in practice, yet is sufﬁciently simple that allows a precise theoretical characterization.
We use a biased ridge solver as the inner loop with a (tunable) regularization parameter, and compare two outer-loop algorithms of either performing the train-validation split (the train-val method ) or using all the per-task data for both training and evaluation (the train-train method ).
Speciﬁcally, we compare the two methods when the number of tasks T is large, and examine if and how fast they converge to the (properly deﬁned) best centroid at meta-test time.
We summarize our contributions as follows:
• On the linear centroid meta-learning problem, we show that the train-validation split is necessary in the general agnostic setting: As T → ∞ , the train-val method converges to the optimal centroid for test-time adaptation, whereas the train-train method does not without further assumptions on the tasks (Section 3).
The convergence of the train-val method is expected since its (population) training loss is equivalent to the meta-test time loss, whereas the non-convergence of the train-train method is because these two losses are not equivalent in general.
• Our main theoretical contribution is to show that the train-validation split is not necessary and even non-optimal, in the perhaps more interesting regime when there are structural assumptions on the tasks: When the data are generated from noiseless linear models, both the train-val and traintrain methods converge to the common best centroid, and the train-train method achieves a strictly better (asymptotic) estimation error and test loss than the train-val method (Section 4).
This is in stark contrast with the agnostic case, and suggests that data efﬁciency may indeed be more important when the tasks have a nice structure.
Our results build on tools from random matrix theory in the proportional regime, which may be of broader technical interest.
• We perform meta-learning experiments on simulations and benchmark few-shot image classiﬁcation tasks, showing that the train-train method consistently outperforms the train-val method (Section 5 & Appendix D).
This validates our theories and presents empirical evidence that samplesplitting may not be crucial; methods that utilize the per-task data more efﬁciently may be preferred.
1. R ELATED WORK
Meta-learning and representation learning theory Baxter (2000) provided the ﬁrst theoretical analysis of meta-learning via covering numbers, and Maurer et al.
(2016) improved the analysis via
Gaussian complexity techniques.
Another recent line of theoretical work analyzed gradient-based meta-learning methods (Denevi et al., 2018a;
Finn et al., 2019; Khodak et al., 2019; Ji et al., 2020) and showed guarantees for convex losses by using tools from online convex optimization.
Saunshi et al.
(2020) proved the success of Reptile in a one-dimensional subspace setting.
Wang et al.
(2020b)
compared the performance of train-train and train-val methods for learning the learning rate.
Denevi et al.
(2018b) proposed the linear centroid model studied in this paper, and provided generalization error bounds for train-val method; the bounds proved also hold for train-train method, so are not sharp enough to compare the two algorithms.
Wang et al.
(2020a) studied the convergence of gradient-based meta-learning by relating to the kernelized approximation.
On the representation learning end, Du et al.
Tripuraneni et al.
(2020a;b) showed that ERM can successfully pool data across tasks to learn the representation.
Yet the focus is on the accurate estimation of the common representation, not on the fast adaptation of the learned prior.
Lastly, we remark that there are analyses for other representation learning schemes (McNamara & Balcan, 2017; Galanti et al., 2016;
Alquier et al., 2016).
Empirical understandings of meta-learning Raghu et al.
(2020) investigated the representation learning perspective of meta-learning and showed that MAML with a full ﬁnetuning inner loop mostly learns the top-layer linear classiﬁer and does not change the representation layers much.
This result partly justiﬁes the validity our linear centroid meta-learning problem in which the features (representations) are ﬁxed and only a linear classiﬁer is learned.
Goldblum et al.
(2020) investigated the difference of the neural representations learned by classical training (supervised learning) and meta-learning, and showed that the meta-learned representation is both better for downstream adaptation and makes classes more separated than the classically trained one.
Although the classical training method in (Goldblum et al., 2020) does not perform a train-validation split, it is not exactly the same as the train-train method considered in this work as it effectively performs a supervised learning on all tasks combined and does not do a per-task adaptation.
Multi-task learning Multi-task learning also exploits structures and similarities across multiple tasks.
The earliest idea dates back to Caruana (1997); Thrun & Pratt (1998); Baxter (2000), initially in connections to neural network models.
They further motivated other approaches using kernel methods (Evgeniou et al., 2005; Argyriou et al., 2007) and multivariate linear regression models with structured sparsity (Liu et al., 2009; 2015).
More recent advances on deep multi-task learning focus on learning shared intermediate representations across tasks Ruder (2017).
These multi-task learning approaches usually minimize the joint empirical risk over all tasks, and the models for different tasks are enforced to share a large amount of parameters.
In contrast, meta-learning only requires the models to share the same “prior”, which is more ﬂexible than multi-task learning.
P RELIMINARIES
In this paper, we consider the standard meta-learning setting, in which we observe data from T ≥supervised learning tasks, and the goal is to ﬁnd a prior (or “initialization”) using the combined data, such that the ( T + 1) -th new task may be solved sample-efﬁciently using the prior.
Linear centroid meta-learning We instantiate our study on the linear centroid meta-learning problem (also known as learning to learn around a common mean, Denevi et al.
(2018b)), where we wish to learn a task-speciﬁc linear predictor
w t ∈ R d in the inner loop for each task t , and learn a “centroid” w 0 in the outer loop that enables fast adaptation to w t within each task:
Find the best centroid w 0 ∈ R d for adapting to a linear predictor w t on each task t .
Formally, we assume that we observe training data from T ≥ 1 tasks, where for each task index t we sample a task p t (a distribution over R d × R ) from some distribution of tasks Π , and observe n examples ( X t , y t )
∈ R n × d × R n that are drawn i.i.d.
from p t :
[Equation]
t
[Equation]
We do not make further assumptions on ( n, d ) ; in particular, we allow the underdetermined setting n ≤ d , in which there exists (one or many) interpolators (cid:101) w t that perfectly ﬁt the data: X t
[Equation]
Inner loop: Ridge solver with biased regularization towards the centroid Our goal in the inner loop is to ﬁnd a linear predictor w t that ﬁts the data in task t while being close to the given “centroid” w 0 ∈ R d .
We instantiate this through ridge regression (i.e. linear regression with L 2 regularization) where the regularization biases w t towards the centroid.
Formally, for any w 0 ∈ R d and any dataset ( X , y ) , we consider the algorithm
[Equation]
where λ > 0 is the regularization strength (typically a tunable hyper-parameter).
As we regularize by (cid:107) w − w 0 (cid:107) 2 , this inner solver encourages the solution to be close to w 0 , as we desire.
Such a regularizer is widely used in practical meta-learning algorithms such as MetaOptNet (Lee et al.,
and Meta-MinibatchProx
(Zhou et al., 2019).
In addition, as λ → 0 , this solver recovers gradient descent ﬁne-tuning: we have
[Equation]
(where X † ∈ R d × n denotes the pseudo-inverse of X ).
This is the minimum-distance interpolator of ( X , y ) and also the solution found by gradient descent 1 on (cid:107)
Xw − y (cid:107) 2 initialized at w 0 .
Therefore our ridge solver with λ > 0 can be seen as a generalized version of the gradient descent solver used in MAML (Finn et al., 2017).
Outer loop: Finding the best centroid In the outer loop, our goal is to ﬁnd the best centroid w 0 .
The standard approach in meta-learning is to perform a train-validation split , that is, (1) execute the inner solver on a ﬁrst split of the task-speciﬁc data, and (2) evaluate the loss on a second split, yielding a function of w 0 that we can optimize.
This two-stage procedure can be written as
Compute w t ( w
[Equation]
) =
[Equation]
; X
[Equation]
, y
[Equation]
and evaluate (cid:13)(cid:13)
[Equation]
where ( X train t , y train t )
= { ( x t,i , y t,i ) } n 1 i =1 and ( X val t , y val t ) =
{ ( x t,i , y t,i ) } ni = n 1 +1 are two disjoint splits of the per-task data ( X t , y t ) of size ( n 1 , n 2 ) , with n 1 + n 2 = n .
Written concisely, this is to consider the “split loss”
[Equation]
In this paper, we will also consider an alternative version, where we do not perform the trainvalidation split, but instead use all the per-task data for both training and evaluation .
Mathematically, this is to look at the “non-split loss”
[Equation]
Our overall algorithm is to solve the empirical risk minimization (ERM) problem on the T observed tasks, using either one of the two losses above:
[Equation]
Let L { tr - val , tr - tr } ( w 0 ) :=
E p t ∼ Π , ( X t , y t ) ∼ p t
[Equation]
{ tr - val , tr -
[Equation]
( w
[Equation]
) (cid:105) be the population risks.
(Meta-)Test time The meta-test time performance of any meta-learning algorithm is a joint function of the (learned) centroid w 0 and the inner algorithm Alg .
Upon receiving a new task p T +1 ∼ Π and training data ( X T +1 , y T +1 ) ∈ R n × d × R n , we run the inner loop Alg with prior w 0 on the training data, and evaluate it on an (unseen) test example ( x (cid:48) , y (cid:48) )
∼ p T +1 :
L test ( w 0 ; Alg ) :=
E p T +1 ∼ Π E
( X T +1 , y T +1 ) , ( x (cid:48) ,y (cid:48)
[Equation]
iid ∼
[Equation]
T
[Equation] [Equation]
Additionally, for both train-val and train-train methods, we need to ensure that the inner loop used for meta-test is exactly the same as that used in meta-training.
Therefore, the meta-test performance for the train-val and train-train methods above should be evaluated as
[Equation]
where A λ,m denotes the ridge solver with regularization strength λ > 0 on m ≤ n data points.
Finally, we let
[Equation] [Equation]
denote the best centroid if the inner loop uses A λ,n .
The performance of the train-val algorithm (cid:98) w tr - val 0 ,T should be compared against w 0 ,(cid:63) ( λ, n 1 ) , whereas the train-train algorithm
[Equation]
w tr - tr 0 ,T should be compared against w 0 ,(cid:63) ( λ, n ) .
2. T ASK - ABUNDANT SETTING THROUGH ASYMPTOTIC ANALYSIS
In this paper we are interested in the task-abundant setting where we ﬁx some ﬁnite ( d, n ) and let T be very large.
We analyze such a task-abundant setting through the asymptotic analysis framework, that is, examine the limiting properties of the estimator (e.g.
(cid:98) w { tr - val , tr - tr } 0 ,T ) as T → ∞ .
Here we set up the basic notation of asymptotic analysis required in this paper.
We emphasize that our large
T setting captures practical meta-learning scenarios; for example, 5 -way image classiﬁcation on miniImageNet (Ravi & Larochelle, 2017) contains (cid:0) (cid:1) diverse tasks (at train time).
Asymptotic rate of estimation & excess risk Let L be any population risk with minimizer w 0 ,(cid:63) (which we assume is unique), (cid:98)
L T be the empirical risk on the observed data from T tasks, and (cid:98) w 0 ,T be the minimizer of (cid:98)
L T (i.e. the ERM).
We say that (cid:98) w 0 ,T is consistent if (cid:98) w 0 ,T → w 0 ,(cid:63) in probability as T → ∞ .
For consistent ERMs, we deﬁne its asymptotic parameter estimation error (in MSE loss) and asymptotic excess risk as follows 2 :
[Equation]
We emphasize that asymptotic statements are more reﬁned than non-asymptotic O ( · ) style upper bounds in the T → ∞ limit: they already imply the { MSE, excess risk } has order O (1 /T ) and speciﬁes the leading constant .
T HE IMPORTANCE OF SAMPLE SPLITTING
We begin by analyzing whether the algorithms
[Equation] [Equation]
[Equation]
deﬁned in (4) converge to the best test-time centroid w 0 ,(cid:63) ( λ ; n 1 ) or w 0 ,(cid:63) ( λ ; n ) (deﬁned (5)) respectively as T → ∞ , in the general situation where we do not make structural assumptions on the data distribution p t .
Proposition 1 (Consistency and asymptotics of train-val method) .
Suppose E x ∼ p t
[ xx (cid:62) ] (cid:31) 0 ,
E x ∼ p t
[ (cid:107) x (cid:107) 4 ] < ∞ and E ( x ,y ) ∼ p t [ (cid:107) x y (cid:107) ] < ∞ for almost surely all p t ∼ Π .
Then for any λ >and any ( n 1 , n 2 ) such that n 1 + n 2 = n , the train-val method (cid:98) w tr - val 0 ,T converges to the best test-time centroid:
[Equation] [Equation]
( λ, n 1 ) almost surely as T → ∞ .
Further, we have
[Equation]
AsymExcessRisk L test λ,n 1 ( (cid:98) w tr
[Equation]
val 0 ,T ) = tr (cid:0) ∇
[Equation]
2 L test λ,n 1 ( w 0 ,(cid:63) ( λ, n 1 )) ·
Cov (cid:0) ∇ (cid:96) tr - val t ( w 0 ,(cid:63) ( λ, n 1 )) (cid:1)(cid:1) .
Proposition 2 (Inconsistency of train-train method) .
There exists a distribution of tasks Π on d =satisfying the conditions in Proposition 1 on which the train-train method does not converge to the best test-time centroid: for any n ≥ 1 and any λ > 0 , the estimation error (cid:107) (cid:98) w tr
[Equation]
tr 0 ,T − w 0 ,(cid:63) ( λ, n ) (cid:107) and the excess risk L test λ,n ( (cid:98) w tr - tr 0 ,T ) − L test λ,n ( w 0 ,(cid:63) ( λ, n )) are both bounded away from 0 almost surely as
T → ∞ .
Propositions 1 and 2 justify the importance of sample splitting: the train-val method converges to the best test-time centroid, whereas the train-train method does not converge to the best centroid in general.
The reason behind Proposition 1 is simple: the population loss L tr - val for the trainval method is indeed equal to the test-time loss L test λ,n 1 , making the train-val method a proper ERM for the meta-test time we care about, and thus the consistency and asymptotic normality follow from classical results for the ERM (e.g. (Van der Vaart, 2000; Liang, 2016)).
In contrast, for the train-train method, its expected loss L tr - tr is in general not equivalent to L test λ,n :
L tr - tr measures the in-sample prediction error of the per-task predictor, whereas L test λ,n measures the out-of-sample prediction error.
Consequently, the population minimizers of L test λ,n and L tr - tr are not equal in general, which leads to
[Equation]
converging to the minimizer of L tr - tr , not of L test λ,n .
The proof of Proposition 2 constructs a simple counter-example in d = 1 , but we expect such a mismatch to generally hold in any dimension.
Appendix A gives the proofs of Proposition 1 and 2.
I S SAMPLE SPLITTING ALWAYS OPTIMAL ?
Proposition 2 states a negative result for the train-train method, showing that it does not converge to the best test-time centroid without further assumptions on the data distribution.
However, such a negative result is inherently worst-case , and does not preclude the possibility that there exists a data distribution on which the train-train method can also work well.
In this section, we construct a simple data distribution in which we can analyze the performance of both the train-val and the train-train methods more explicitly, showing that sample splitting is indeed not optimal, and the train-train method can work better.
Realizable linear model We consider the following instantiation of the (generic) data distribution assumption in (1): We assume that each task p t is speciﬁed by a w t ∈ R d sampled from some distribution Π (overloading notation), and the observed data follows the noiseless linear model with ground truth parameter w t :
[Equation]
where the inputs x t,i iid ∼ N ( 0 , I d ) and are independent of w t .
We assume that Π has a ﬁnite second moment (i.e. E w t ∼ Π
[ (cid:107) w t (cid:107) 2 ] < ∞ ).
Note that when n ≥ d , we are able to perfectly recover w t for all t (by solving linear equations), therefore the problem in the inner loop is in a sense “easy”; when n < d , we cannot hope for such perfect recoveries.
Our goal in the outer loop is to ﬁnd the best w 0 , measured by the test loss L test λ,n for the train-train method and L test λ,n 1 for the train-val method.
4. C OMPARISON OF TRAIN - TRAIN AND TRAIN - VAL ON THE REALIZABLE MODEL
We begin by showing that on this task and data distribution, the population best centroids w 0 ,(cid:63) ( λ, n ) =
arg min w
L test λ,n ( w 0 ) is the same for any ( λ, n ) , and both the train-val and traintrain methods are asymptotically consistent and converge to same best centroid.
Theorem 3 (Consistency of both train-val and train-train methods) .
On the realizable linear model (6) , the test-time meta loss for all λ > 0 and all n is minimized at the same point, that is, the mean of the ground truth parameters:
[Equation]
Further, both the train-val method and the train-train method are asymptotically consistent: for any λ > 0 , n , and ( n 1 , n 2 ) , we have
[Equation]
See its proof in Appendix B.1.
Theorem 3 shows that both train-val and train-train methods are consistent, and they converge to the same optimal parameter w 0 ,(cid:63) which is the mean of w t .
This is a consequence of the good structure in our realizable linear model (6): at a high level, w 0 ,(cid:63) is indeed the best centroid since it has (on average) the closest distance to a randomly sampled w t .
Theoerem 3 suggests that we are now able to compare performance of the two methods based on their asymptotic parameter estimation error (for estimating w 0 ,(cid:63) ).
Throughout the rest of this section, let
[Equation]
denote the variance of w t .
We are now ready to state our main result.
Theorem 4 (Comparison of asymptotic MSE of the train-val and train-train methods) .
In the high-dimensional limiting regime d, n → ∞ , d/n → γ ∈ (0 , ∞ ) , the optimal rate of the train- train method obtained by tuning the regularization λ ∈ (0 , ∞ ) satisﬁes
[Equation]
where ρ λ,γ =4 γ
[Equation]
(cid:1)
[Equation]
[Equation]
, and the inequality becomes equality at γ = 1 .
In contrast, the optimal rate of the train-val method by tuning the regularization λ ∈ (0 , ∞ ) and split ratio s ∈ (0 , 1) is
[Equation]
As max { 1 + 5 γ/ 27 , 5 / 27 + γ } < 1+ γ ( ∀ γ > 0 ), the train-train method achieves a strictly better asymptotic rate than the train-val method when λ and s are optimally tuned in both methods.
Implications Comparison between the analytical upper bound max { 1 + 5 γ/ 27 , 5 / 27 + γ } R 2 for train-train (1 + γ ) R 2 for train-val in Theorem 4 shows that the train-train method achieves a strictly better asymptotic MSE than the train-val method, for any γ > 0 3 .
(See Figure 1(a) for a visualization of the exact optimal rates and the upper bound ( (cid:63) ) .)
Perhaps surprisingly, this suggests that the traintrain method is not only “correct” (converging to the best centroid), but can be even better than the train-val method, when the data is structured.
While the “correctness” of the train-train method is a consequence of the realizable linear model, we believe its superior asymptotic MSE is due to the fact that the train-train method is able to use the data more efﬁciently than the train-val method.
Also, while we reached such a conclusion on this particular problem of linear centroid meta-learning, we suspect that this phenomenon to be not restricted to this problem, and may hold in more generality when data is structured or when the signal-to-noise ratio is high.
As we are going to see, our real data experiments in Section 5.2 indeed suggests that the superiority of the train-train method may also hold on real meta-learning tasks with neural networks.
4. P ROOF HIGHLIGHTS OF T HEOREM
Here we sketch the technical highlights in proving Theorem 4.
We defer the full proof to Appendix B.5.
Exact asymptotic MSE for both methods The proof begins by calculating the exact asymptotic
MSE for both methods, which we provide in the following theorem.
Lemma 5 (Exact asymptotic rates of the train-val and train-train methods) .
Suppose that ρ tr - tr =
[Equation]
( n ) i + λ ) 2 (cid:105)(cid:17) 2 and ρ tr - val =
E (cid:20)(cid:16)(cid:80) di =1 λ
[Equation]
/ ( σ ( n 1) i + λ )
[Equation]
(cid:17) 2 +( n 2 +1) (cid:80) di
=1 λ 4 / ( σ ( n 1)
[Equation]
+ λ ) 4 (cid:21) (cid:16) E (cid:104)(cid:80) di
=1 λ 2 / ( σ ( n 1)
i + λ ) 2 (cid:105)(cid:17), where for any n , σ ( n )≥ · · · ≥ σ ( n ) d denotes the eigenvalues of the matrix 1 n X (cid:62)
t X t ∈ R d × d , where we recall X t ∈ R n × d is a random matrix with i.i.d.
standard Gaussian entries.
For any ( n, d ) , we have on the realizable linear model (6) that
[Equation]
See its proof in Appendix B.2.
Lemma 5 follows straightforwardly from the classical asymptotic result for empirical risk minimization (Van der Vaart, 2000) and simpliﬁcations of certain matrix traces in terms of the spectrum of the empirical covariance matrix 1 n
[Equation]
Simplifying and optimizing the asymptotic MSEs The asymptotic MSEs of the train-train and train-val method in Lemma 5 are not yet directly comparable, as the quantities ρ tr - tr and ρ tr - val depend on the spectrum of the empirical covariance matrix as well as additional tunable parameters such as λ and
( n
, n
) (for the
train val method).
Towards proving Theorem 4, we further simplify the rates and analyze the optimal tunable parameters, using separate strategies for the two methods:
• For the train-val method, we show that the optimal tunable parameters for any ( n, d ) is taken at a special case λ = ∞ and ( n 1 , n 2 ) =
(0 , n ) , at which the rates only depends on 1 n 1 X train (cid:62) t
X train t through its rank (and thus has a simple closed-form).
We state this result in Corollary 8.
The proof builds on algebraic manipulations of the quantity ρ tr - val , and can be found in Appendix B.3.
• For the train-train method, we apply random matrix theory to simplify the spectrum of 1 n X (cid:62) t X t in the proportional limit where d, n → ∞ and d/n stays a constant (Bai & Silverstein, 2010; Anderson et al., 2010), and obtain a closed-form expression of the asymptotic MSE for any λ > 0 , which we can analytically optimize over λ .
We state this result in Theorem 9.
The proof builds on the Steiltjes transform and its “derivative trick” (Dobriban et al., 2018), and is deferred to Appendix B.4.
E XPERIMENTS 5. S IMULATIONS
We experiment on the realizable linear model studied in Section 4.
Recall that the observed data of the t -th task are generated as
[Equation]
We independently generate w t iid ∼ N ( w 0 ,(cid:63) , I d / √ d ) , where w 0 ,(cid:63) is the linear centroid and the corresponding R 2 = 1 here.
The goal is to learn the linear centroid w 0 ,(cid:63) using the train-train method and train-val method, i.e., minimizing (cid:98)
L tr - tr T and (cid:98)
L tr - val T , respectively.
Note that both (cid:98)
L tr - tr T and (cid:98)
L tr - val T are quadratic in w 0 , therefore, we can ﬁnd the close-form solutions (cid:98) w { tr - tr , tr - val } 0 ,T .
We measure the performance of train-train and train-val methods using the (cid:96) 2 -error (cid:107) w 0 ,(cid:63) −
[Equation]
We present the comparison among train-train and train-val methods in Figure 1 with scatter plots representing the simulation outputs under different settings.
Across all the simulations, we well-tune the regularization coefﬁcient λ in the train-train method, and use a sufﬁciently large λ = 10000 in the train-val method according to Corollary 8.
The simulated results concentrate around the reference curves corresponding to our theoretical ﬁndings.
This corroborates our analyses and demonstrates the better performance of train-train method on the realizable linear model.
We additionally investigate the effect of averaging the loss over multiple splits in the trainval method (a “cross-validation” type loss) rather than the vanilla single split. We show that such cross-validation can indeed improve over the vanilla single-split train-val method, yet still performs worse than the train-train method (Appendix E)
[Table]
= =</latexit>  w 0 , ?  b w { tr - tr , tr - v a l } 0 , T
5. F EW - SHOT IMAGE CLASSIFICATION
We further investigate the comparison between the train-train and train-val type methods in fewshot image classiﬁcation on miniImageNet (Ravi & Larochelle, 2017) and tieredImageNet (Ren et al., 2018).
Methods We instantiate the train-train and train-val method in the centroid meta-learning setting with a ridge solver.
The methods are almost exactly the same as in our theoretical setting in (2) and (3), with the only differences being that the parameters w t (and hence w 0 ) parametrize a deep neural network instead of a linear classiﬁer, and the loss function is the cross-entropy instead of squared loss.
Mathematically, we minimize the following two loss functions:
[Equation]
where ( X t , y t ) is the data for task t of size n , and ( X train t , y train t ) and ( X val t , y val t ) is a split of the data of size ( n 1 , n 2 ) .
We note that both loss functions above have been considered in prior work ( L tr - val in iMAML (Rajeswaran et al., 2019), and L tr - tr in Meta-MinibatchProx
(Zhou et al., 2019)), though we use slightly different implementation details from these prior work to make sure that the two methods here are exactly the same except for whether the split is used.
Additional details about the implementation can be found in Appendix D.
Experimental settings We adopt the episodic training procedure (Finn et al., 2017; Zhou et al., 2019; Rajeswaran et al., 2019).
In meta-test, we sample a set of N -way
( K +1)
-shot test tasks.
The ﬁrst K instances are for training and the remaining one is for testing.
In meta-training, we use the “higher way” training strategy.
We set the default choice of the train-validation split ratio to be an even split n 1 = n 2 = n/ 2 following Zhou et al.
; Rajeswaran et al.
For example, for a 5 -way 5 -shot classiﬁcation setting, each task contains 5 × (5 + 1) =
30 total images, and we set n 1 = n 2 = 15 .
(We additionally investigate the optimality of this split ratio in Appendix D.1.)
We evaluate both methods under the transduction setting where the information is shared between the test data via batch normalization.
We report the average accuracy over 2 , 000 random test episodes with 95% conﬁdence interval.
Results Table 1 presents the percent classiﬁcation accuracy on miniImagenet and tieredImageNet.
We ﬁnd that the train-train method consistently outperforms the train-val method.
Speciﬁcally, on miniImageNet, train-train method outperforms train-val by 2 .
01% and 3 .
87% on the 1 -shot 5 -way and 5 -shot 5 -way tasks respectively; On tieredImageNet, train-train on average improves by about 6 .
40% on the four testing cases.
These results show the advantages of train-train method over trainval and support our theoretical ﬁndings in Theorem 4.
[Table]
C ONCLUSION
We study the importance of train-validation split on the linear-centroid meta-learning problem, and show that the necessity and optimality of train-validation split depends greatly on whether the tasks are structured: the sample splitting is necessary in general situations, and not necessary and nonoptimal when the tasks are nicely structured.
It would be of interest to study whether a similar conclusion holds on other meta-learning problems such as learning a representation, or whether our insights can be used towards designing meta-learning algorithms with better empirical performance.
A P ROOFS FOR S ECTIONA. D ETAILED INTRODUCTION ON ASYMPTOTIC ANALYSIS
Here we provide more details on the asymptotic analysis framework sketched in Section 2.
In typical scenarios for consistent ERMs, the limiting distribution of (cid:98) w 0 ,T is asymptotically normal with a known covariance matrix, as is characterized in the following classical result (see, e g. Van der
Vaart (2000, Theorem 5.21) and also Liang (2016)).
Proposition 6 (Asymptotic normality and excess risk of ERMs) .
Assume the population minimizer w 0 ,(cid:63) is unique and the ERM (cid:98) w 0 ,T is consistent (i e. it converges to w 0 ,(cid:63) in probability as T → ∞ ).
Further assume the following regularity conditions:
(a) There exists some random variable A t =
A ( p t , X t , y t ) such that E
[ A 2 t ] < ∞ and [Equation] (cid:96) [Equation]
( w [Equation] ∇ (cid:96) [Equation]
( w [Equation] ≤ [Equation]
for all w 1 , w 2 ∈ R d ; (b) E [ (cid:107)∇ [Equation] (c) L is twice-differentiable with ∇ 2 L ( w 0 ,(cid:63) ) (cid:31) [Equation]
then the ERM (cid:98) w ,T
is
asymptotically
normally distributed
, with
[Equation]
where ∆ ∼ P
[Equation]
where d → denotes convergence in distribution and (cid:96) t : R d → R is the loss function on a single task.
When this happens, we deﬁne the asymptotic rate of estimation (in MSE loss) and asymptotic excess risk (cid:98) w 0 ,T as those of its limiting distribution:
[Equation]
AsymExcessRisk( (cid:98)
[Equation]
A. P ROOF OF P ROPOSITION
Equivalence of test-time risk and training loss for train-val method We ﬁrst show that
[Equation]
for all w 0 , that is, the population meta-test loss is exactly the same as the population risk of the trainval method.
This is straightforward: as the tasks are i.i.d.
and A λ ( w 0 ; X train t , y train t ) is independent of the test points ( X val t , y val t ) , we have for any w 0 that
[Equation]
[Equation]
Therefore the train-val method is acutally a valid ERM for the test loss L test λ,n 1 , and it remains to show that the train-val method is (itself) consistent
[Equation]
Consistency We expand the empirical risk of the train-val method as [Equation] tr [Equation]
val T
[Equation]
where
[Equation]
Noticing that ( X train (cid:62) t
X train t /n {+
+}
+ λ I d ) − 1 (cid:22) λ − 1 I d and by the assumption that E ( x ,y ) ∼ p t
[ xx (cid:62) ] ≺ ∞ , E ( x ,y ) ∼ p t [ xy ] < ∞ , we have E [ (cid:107) M T (cid:107) ] < ∞ and E [ (cid:107) b T (cid:107) ] < ∞ .
Since the task p t ’s are i.i.d., by the law of large numbers, we have with probability one that
[Equation]
(8) (where Σ t =
E x ∼ p t [ xx (cid:62) ] (cid:31) 0 ) and b T → E [ b T ] =
E p t , ( X t , y t )
[Equation]
< ∞ (
[Equation]
as T → ∞ .
Therefore, by Slutsky’s Theorem, we have
[Equation]
as T → ∞ .
This proves the consistency of the train-val method.
Asymptotic normality Similar as above, we can write the per-task loss as
[Equation]
where
[Equation]
In order to show the desired asymptotic normality result, it sufﬁces to check the conditions in Proposition 6.
First, we have
[Equation]
This is Lipschitz in w 0 with Lipschitz constant
[Equation]
As E x ∼ p t [ (cid:107) x (cid:107) 4 ] < ∞ , the above quantity is clearly square integrable, therefore verifying (a).
As w 0 ,(cid:63) =
w 0 ,(cid:63) ( λ, n 1 ) is ﬁnite, we can use similar arguments as above to show (b) holds.
Finally, we have already seen L is twice-differentiable (since it is quadratic in w 0 ) and ∇ 2 L ( w 0 ,(cid:63) )
(cid:31) 0 , which veriﬁes (c).
Therefore the conditions of Proposition 6 hold, which yields the desired asymptotic normality result.
A. P ROOF OF P ROPOSITION
High-level idea At a high level, this proof proceeds by showing that the train-train method is also consistent to the (population) minimizer of L tr - tr , and constructing a simple counter-example on which the minimizers of L tr - tr is not equal to that of L test λ,n .
Population minimizers of L tr - tr and L tr - val
We begin by simplifying the non-splitting risk.
We have
[Equation]
where
[Equation]
Using similar arguments as in the proof of Proposition 1 (Appendix A.2)
[Equation]
we see that the traintrain method (cid:98) w tr - tr 0 ,T converges with probability one to the minimizer of the pouplation risk L tr - tr , which is
[Equation]
On the other hand
[Equation]
recall from Proposition 1 ((8) and (9)) that the population minimizer of L test λ,n is
[Equation]
Construction of the counter-example
We now construct a distribution for which (10) is not equal to (11).
Let d
= 1 and let all p t be the following distribution:
[Equation]
Clearly, we have Σ t = 5 , s t :=
X (cid:62) t X t /n ∈ [1 , 9] , and E x (cid:48) ,y (cid:48) ∼ p t [ x (cid:48) y (cid:48) ]
= 0 .
Therefore we have
[Equation]
n
[Equation]
and w
[Equation]
n
[Equation]
=
[Equation]
(cid:88)
[Equation]
We now show that w tr - tr 0 ,(cid:63) (cid:54) =
w 0 ,(cid:63) ( λ, n ) by showing that
[Equation]
for any λ > 0 .
Indeed, conditioning on ( x t, 1 , y t, 1 ) =
(1 , 3) , we know that the sum-of-squares in s t has one term that equals 1 , and all others i.i.d.
being 1 or 9 with probability one half.
On the other hand, if we condition on ( x t, 1 , y t, 1 ) = (3 , − 1) , then we know the sum in s t has one term that equals 9 and all others i.i.d..
This means that the negative contribution in the expectation is smaller than the positive contribution, in other words
[Equation]
This shows w tr - tr 0 ,(cid:63) (cid:54) =
w 0 ,(cid:63) ( λ, n ) and consequently the (cid:98) w tr - tr 0 ,T does not converge to w 0 ,(cid:63) ( λ, n ) and the difference is bounded away from zero as T → ∞ .
Finally, for this distribution, the risk L test λ,n ( w 0 ) is strongly convex (since it has a positive second derivative), this further implies that L test λ,n (
[Equation]
n ( w
[Equation]
(cid:63) ( λ
[Equation]
n )) is bounded away from zero almost surely as T → ∞ .
B P ROOFS FOR S ECTIONB. P ROOF OF T HEOREM
We ﬁrst show that w 0 ,(cid:63) =
E w t ∼ Π [ w t ] is a global optimizer for L tr - tr and L tr - val with any regularization coefﬁcient λ > 0 , any n , and any split ( n 1 , n 2 ) .
To do this, it sufﬁces to check that the gradient at w 0 ,(cid:63) is zero and the Hessian is positive deﬁnite (PD).
Optimality of w 0 ,(cid:63) in both L tr - tr and L tr - val .
We ﬁrst look at L tr - tr : for any w 0 ∈ R d we have
[Equation]
tr
[Equation] [Equation]
Similarly, L tr - val can be written as
L tr - val ( w 0 ) (13)
[Equation]
We denote
[Equation]
and
[Equation]
to simplify the notations in (12) and (14).
We take gradient of L tr - tr and L tr - val with respect to w
[Equation]
Substituting w 0 ,(cid:63) into (17) and taking expectation, we deduce
[Equation]
To see this, observe that by deﬁnition E [ w t − w 0 ,(cid:63) ]
= 0 .
Combining with w t being generated independently of X t , we have the ﬁrst term in RHS of (19) vanish.
In addition, z t is independent white noise, therefore, the second term in RHS of (19) also vanishes.
Following the same argument, we can show
[Equation]
since X (cid:48) t is also independent of w t .
The above reasonings indicates that w 0 ,(cid:63) is a stationary point of both L tr - tr and L tr - val .
The remaining step is to check ∇ w 0 L tr - tr ( w 0 ,(cid:63) ) and ∇ w 0 L tr - val ( w 0 ,(cid:63) ) are
PD.
From (17) and (18), we derive respectively the hessian of L tr - tr and L tr - val as
[Equation]
[Equation]
Let v ∈ R
d be any nonzero vector, we check v (cid:62) ∇ 2 w 0 L tr - tr ( w 0 ,(cid:63) ) v > 0 .
A key observation is that
[Equation]
is positive deﬁnite for any λ (cid:54) = 0 .
To see this, let σ 1 ≥ · · · ≥ σ d be eigenvalues of 1 n X (cid:62) t X t , some algebra yields the eigenvalues of (cid:16)
I d −
[Equation]
are λ λ + σ i > 0 for λ (cid:54)
=
and i = 1 , . . .
, d .
Hence, we deduce
[Equation]
since X t is isotropic (an explicit computation of the hessian matrix can be found in Appendix B.2).
As a consequence, we have shown that w 0 ,(cid:63) is a global optimum of L tr - tr .
The same argument applies to L tr - val , and the proof is complete.
Consistency of
(cid:98)
[Equation]
To check the consistency, we need to verify the conditions (a) – (c) in
Proposition 6.
For condition (a), we derive from (17) and (18) that
[Equation]
where n should be replaced by n 2 for the split method (we slightly abuse the notation for simplicity).
It sufﬁces to show E
[Equation]
op
[Equation]
< ∞ , which follows from the same argument in the proof of Proposition 1.
In particular, we know
(cid:22) I d −
[Equation]
and E [ (cid:13)(cid:13)
X (cid:62) t X t (cid:13)(cid:13) op ] < ∞ since X t is Gaussian.
As a consequence, for the no-split method, we have E (cid:104)(cid:13)(cid:13) ( M tr - tr t ) (cid:62) M tr - tr t (cid:13)(cid:13) 2 op (cid:105) < ∞ .
For the split method, we also have 0 (cid:22) I
d − (cid:0) ( X train t ) (cid:62) X train t + nλ I d (cid:1) − 1 ( X train t ) (cid:62) X train t (cid:22) I
d and E [ (cid:13)(cid:13)
( X val t ) (cid:62) X val t (cid:13)(cid:13) op ] < ∞ , which implies
E (cid:104)(cid:13)(cid:13) ( M tr - val t ) (cid:62) M tr - val t (cid:13)(cid:13) 2 op (cid:105) < ∞ .
For condition (b), using a similar argument in condition (a) and combining with R 2 = E [ (cid:107) w 0 ,(cid:63) − w t (cid:107) 2 ] , we have E [ (cid:107)∇ (cid:96) { tr - tr , tr - val } t ( w 0 ,(cid:63) ) (cid:107) 2 ] < ∞ .
For condition (c), using (20), we directly verify that L { tr - tr , tr - val } is twice-differentiable and ∇ 2 L { tr - tr , tr - val } (cid:31) 0 .
B. P ROOF OF L EMMA
Proof.
In this section we prove Lemma 5.
Using the the asymptotic normality in Proposition 6, the asymptotic covariance is ∇ − 2 L { tr - tr , tr - val } Cov[ ∇ (cid:96) { tr - tr , tr - val } t ] ∇ − 2 L { tr - tr , tr - val } .
Therefore, in the following, we only need to ﬁnd ∇ − 2 L { tr - tr , tr - val } and Cov[ ∇ (cid:96) { tr - tr , tr - val } t ] .
• Asymptotic variance of
(cid:98)
w
tr - tr 0 ,T .
We begin with the computation of the expected Hessian 1 n E [( M tr - tr t )
[Equation]
tr -
[Equation] [Equation]
where the equality ( i ) is obtained by plugging in the SVD of X t
= U t D t V (cid:62) t with U t ∈ R n × n ,
D t ∈ R n × d , and V t ∈ R d × d .
A key observation is that U t and V t are independent, since X t is isotropic, i.e., homogeneous in each orthogonal direction.
To see this, for any orthogonal matrices
Q ∈ R n × n and P ∈ R d × d , we know X t and QX t P (cid:62) share the same distribution.
Moreover, we have QX t P (cid:62) =
( QU t ) D t ( PV t ) (cid:62) as the SVD.
This shows that the left and right singular matrices are independent and both uniformly distributed on all the orthogonal matrices of the corresponding dimensions ( R n × n and R d × d , respectively).
Recall that we denote σ ( n )≥
[Equation]
σ ( n ) d as the eigenvalues of 1 n X (cid:62) t X t .
Thus, we have D (cid:62) t D t =
Diag( nσ ( n ) 1 , . .
, nσ ( n ) d ) .
We can further simplify (21) as
[Equation]
We will utilize the isotropicity of X t to ﬁnd (22).
Recall that we have shown that V t is uniform on all the orthogonal matrices.
Let P ∈ R
d
×
d be any permutation matrix, then V t P has the same distribution as V t .
For this permuted data matrix V t P , (22) becomes
E
[Equation] [Equation]
with τ p ( i ) denotes the permutation of the i -th element in P .
Summing over all the permutations P (and there are totally d ! instances), we deduce
[Equation] [Equation]
Dividing ( d − 1)!
on both sides of (23) yields
[Equation]
Next, we ﬁnd the expected covariance matrix 1 n
[Equation]
[ ∇
[Equation]
Here step ( i ) uses the SVD of X t and the computation in (22).
Combining (24) and (25), we derive the asymptotic covariance matrix of using L tr - tr as
[Equation]
Taking trace in (26), we deduce
AsymMSE
[Equation]
d
[Equation] [Equation]
[Equation]
(cid:35)(cid:33) ( i ) =
dn
[Equation]
(cid:32)
[Equation]
(cid:34) d (cid:88) i =
λ 2 σ ( n ) i ( λ
[Equation]
σ ( n ) i )
(cid:35)(cid:33) −
[Equation]
where step ( i ) utilizes the independence between w t and X t and applies the permutation trick in (23) to ﬁnd
[Equation]
• Asymptotic variance of
[Equation]
Similar to the no-split case, we compute the Hessian
[Equation]
where ( i ) uses the data generating assumption E
[( X val t ) (cid:62) X val t ] = n 2 I d and the independence between X train t and X val t , and ( ii ) follows from the SVD of X train t =
U
[Equation]
Here we denote σ ( n 1 )≥ · · · ≥ σ ( n 1 ) d as the eigenvalues of 1 n 1 ( X train t ) (cid:62) X train t .
Thus, we have ( D train t )
[Equation]
D
[Equation]
t
[Equation]
We can now further simplify (28) as
[Equation]
Step ( i ) follows from the same computation in (22), and step ( ii ) uses the permutation trick in (23).
Next, we ﬁnd the expected covariance matrix 1 n 22 E [ ∇ (cid:96) tr - tr t
[Equation]
[Equation]
Combining (29) and (30)
[Equation]
we derive the asymptotic covariance matrix of using L tr - val as
[Equation]
[Equation]
Taking trace in (31), we deduce
[Equation]
where ( i ) follows from the cyclic property of the matrix trace operation.
Due to the isotropicity of
X train t and X val t , we claim that
[Equation]
is a diagonal matrix
c
I d with all the diagonal elements identical.
We can show the claim bying taking expectation with respect to X val t ﬁrst.
Since V train t is an orthogonal matrix, X val t V train t has the same distribution as X val t and independent of X t .
We verify that any off-diagonal element of the matrix expectation
[Equation]
is zero.
We denote X val t V train t
[Equation]
R n 2 × d with x i iid ∼ N ( 0 , I d ) .
For k (cid:54) =
(cid:96) , the ( k, (cid:96) ) -th
entry A k,(cid:96) of A is
[Equation]
where x i,j denotes the j -th element of x i .
Equality ( i ) holds, since either x k,m or x (cid:96),n only appears once in each summand.
Therefore, we can write A =
Diag ( A 1 , 1 , . . .
, A d,d ) with A k,k being
[Equation]
Observe that A k,k only depends on σ ( n 1 ) k .
Plugging back into (33), we have
[Equation]
[Equation]
where equality ( i ) utilizes the permutation trick in (24).
To this end
[Equation]
it is sufﬁcient to ﬁnd c as
[Equation]
Observe again that X val t V train t ∈ R n 2 × d is a Gaussian random matrix.
We rewrite (34) as
[Equation]
where v i iid ∼ N ( 0 , I d ) is i.i.d.
Gaussian random vectors for i = 1 , . .
. , n 2 .
To compute (35), we need the following result.
Claim 7.
Given any symmetric matrix A ∈ R d × d and i.i.d.
standard Gaussian random vectors v , u iid ∼ N ( 0 , I d ) , we have
[Equation]
Proof of Claim 7.
We show (36) ﬁrst.
We denote A i,j as the ( i, j ) -th element of A and v i as the i -th element of v .
Expanding the quadratic form, we have
[Equation]
Next, we show (37) by the cyclic property of race
[Equation]
[Equation] [Equation]
λ
[Equation]
n
[Equation]
λ )
[Equation]
(cid:33)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Fr  
[Equation]
Combining (38) and (33), by the independence between w t and X train t , X val t , we compute (32) as
[Equation]
di
=1 λ 2 / ( σ ( n 1 )
i + λ )(cid:105)
+ ( n 2 + 1) E (cid:104)(cid:80) di
=1 λ 4 / ( σ ( n 1 ) i + λ )(cid:105) (cid:16)
E (cid:104)(cid:80) di
=1 λ 2 / ( λ + σ ( n 1 ) i )(cid:105)(cid:17) .
The proof is complete.
B. O PTIMAL RATE OF THE TRAIN - VAL METHOD AT FINITE ( n, d )
Corollary 8 (Optimal rate of the train-val method at ﬁnite ( n, d ) ) .
For any ( n, d ) and any split ratio ( n 1 , n 2 ) =
( n 1 , n − n 1 ) , the optimal rate (by tuning the regularization λ > 0 ) of the train-val method is achieved at
[Equation]
Further optimizing the rate over n 2 , the best rate is taken at ( n 1 , n 2 ) =
(0 , n ) , in which the rate is inf λ> 0 , n 2 ∈ [ n ] AsymMSE
[Equation]
n
[Equation]
Discussion: Using all data as validation Corollary 8 suggests that the optimal asymptotic rate of the train-val method is obtained at λ = ∞ and ( n 1 , n 2 ) =
(0 , n ) .
In other words, the optimal choice for the train-val method is to use all the data as validation .
In this case, since there is no training data, the inner solver reduces to the identity map: A ∞ , 0 ( w 0 ; X t , y t ) =
w 0 , and the outer loop reduces to learning a single linear model w 0 on all the tasks combined.
We remark that while the optimality of such a split ratio is likely an artifact of the data distribution we assumed (noiseless realizable linear model) and may not generalize to other meta-learning problems, we do ﬁnd experimentally that using more data as validation (than training) can also improve the performance on real metalearning tasks (see Table 2).
Proof of Corollary
Fix n 1 ∈ [ n ] and n 2 = n − n 1 .
Recall from Lemma 5 that
AsymMSE
[Equation]
Clearly, as λ → ∞ , we have
[Equation]
It remains to show that the above quantity is a lower bound for AsymMSE
[Equation]
for any λ > 0 , which is equivalent to
[Equation] [Equation]
We now prove (39).
For i ∈ [ n 1 ] , deﬁne random variables
[Equation]
i
[Equation]
and Y i :
= 1 − X i ∈ [0 , 1] .
Then the left-hand side of (39) can be rewritten as
E
[Equation]
Y i ])
By algebraic manipulation, inequality (39) is equivalent to showing that
[Equation] [Equation]
Clearly, E [( (cid:80)
Y i ) 2 ] ≥ ( E [ (cid:80)
Y i ]) 2 .
By Cauchy-Schwarz we also have
E (cid:104)(cid:88)
[Equation]
E (cid:104)(cid:88)
[Equation]
(cid:105)(cid:17) 2 .
Therefore we have
E
[Equation]
where we have used that n 1 ≤
B. R ATE OF THE TRAIN - TRAIN METHOD IN THE PROPORTIONAL LIMIT
Theorem 9 (Exact rates of the train-train method in the proportional limit) .
In the high-dimensional limiting regime d, n → ∞ , d/n → γ where γ ∈ (0 , ∞ ) is a ﬁxed shape parameter, for any λ >
[Equation]
where ρ λ,γ =4 γ
[Equation]
Proof of Theorem
Let (cid:98) Σ n := 1 n X t X (cid:62) t denote the sample covariance matrix of the inputs in a single task ( t ).
By Lemma 5, we have
[Equation]
We now evaluate quantities I n,d and II n,d in the high-dimensional limit of d, n → ∞ , d/n → γ ∈ (0 , ∞ ) .
Consider the (slightly generalized) Stieltjes transform of (cid:98) Σ n deﬁned for all λ 1 , λ 2 > 0 :
[Equation]
As the entries of X t are i.i.d.
N (0 , 1) , the above limiting Stieltjes transform is the Stieltjes form of the Marchenko-Pastur law, which has a closed form (see, e.g. (Dobriban et al., 2018, Equation (7)))
[Equation]
[Equation]
Now observe that differentiating s ( λ 1 , λ 2 ) yields quantity II (known as the derivative trick of Stieltjes transforms).
Indeed, we have
[Equation]
(Above, the exchange of differentiation and limit is due to the uniform convergence of the derivatives
[Equation]
which holds at any λ 1 , λ 2 > 0 .
See Appendix B
[Equation]
4.1 for a detailed justiﬁcation
[Equation]
) Taking λ 1 = λ and λ 2 = 1 , we get
[Equation]
Similarly we have
[Equation]
Evaluating the right-hand sides from differentiating the closed-form expression (42), we get
[Equation]
[Equation]
Substituting back to (41) yields that
[Equation]
This proves the desired result.
B.4. I NTERCHANGING DERIVATIVE AND EXPECTATION / LIMIT
Here we rigorously establish the interchange of the derivative and the expectation / limit used in (43)
[Equation]
For convenience of notation let Σ = (cid:98) Σ n = X (cid:62) t X t /n denote the empirical covariance matrix of X t .
We wish to show that
[Equation]
1 I d +
[Equation]
2 Σ
[Equation]
−
[Equation]
This involves the interchange of derivative and limit, and then the interchange of derivative and expectation.
Interchange of derivative and expectation First, we show that for any ﬁxed ( d, n ) ,
[Equation]
By deﬁnition of the derivative, we have
[Equation]
For any A (cid:31) 0 , the function t (cid:55)→ tr(( A + t B ) − 1 ) is continuously differentiable at t = 0 with derivative − tr( A − 2 B ) , and thus locally Lipschitz around t = 0 with Lipschitz constant | tr( A − 2 B ) | + 1 .
Applying this in the above expectation with A = λ 1 I d + λ 2 Σ (cid:23) λ 1 I d and B = Σ , we get that for sufﬁciently small | t | , the fraction inside the expectation is upper bounded by | tr( λ − 2 1 Σ ) | + 1 < ∞ uniformly over t .
Thus by the Dominated Convergence Theorem, the limit can be passed into the expectation, which yields the expectation of the derivative.
Interchange of derivative and limit Deﬁne f n,d ( λ
[Equation]
:=
[Equation]
λ
[Equation]
It sufﬁces to show that
[Equation]
where
[Equation]
by the result of the preceding part.
As f n,d ( λ 2 ) → s ( λ 1 , λ 2 ) pointwise over λ 2 by properties of the Wishart matrix (Bai & Silverstein, 2010) and each individual f n,d is differentiable, it sufﬁces to show that the derivatives f
[Equation]
n,d (
[Equation]
) converges uniformly for (cid:101)
λ 2 in a neighborhood of λ 2 .
Observe that can rewrite f (cid:48) n,d as
[Equation]
where
[Equation]
µ n,d is the empirical distribution of the eigenvalues of Σ , and
[Equation]
for all λ ≥
[Equation]
Therefore, as (cid:98) µ n,d converges weakly to the Marchenko-Pastur distribution with probability one and g (cid:101) λ 2 is uniformly bounded for (cid:101) λ 2 in a small neighborhood of λ 2 , we get that f (cid:48) n,d ( (cid:101) λ 2 ) does converge uniformly to the expectation of g (cid:101)
λ
2 ( λ ) under the Marchenko-Pastur distribution.
This shows the desired interchange of derivative and limit.
B. P ROOF OF T HEOREM
Throughout this proof we assume that R 2 = 1 without loss of generality (as all the rates are constant multiples of R 2 ).
Part I: Optimal rate for L tr - tr
By Theorem 9, we have
[Equation]
In order to bound inf λ> 0 f ( λ, γ ) , picking any λ = λ ( γ ) gives f ( λ ( γ ) , γ ) as a valid upper bound, and our goal is to choose λ that yields a bound as tight as possible.
Here we consider the choice
[Equation]
} which we now show yields the claimed upper bound.
Case 1: γ ≤ 1 Substituting λ = 1 − γ/ 2 into f ( λ, γ ) and simplifying, we get
[Equation]
Clearly, g 1 (0) =
and g 1 (1) = 32 / 27 .
Further differentiating g 1 twice gives
[Equation]
for all γ ∈ [0 , 1] .
Thus g 1 is convex on [0 , 1] , from which we conclude that g
[Equation]
27 γ.
Case 2: γ > 1 Substituting λ = γ − 1 / 2 into f ( λ, γ ) and simplifying, we get
[Equation]
γ − 1 /
[Equation] [Equation]
=: g 2 ( γ ) .
We have g 2 (1) =
g 1 (1) = 32 / 27 .
Further differentiating g 2 gives
[Equation]
Therefore we have for all γ > 1 that
[Equation]
Combining Case 1 and 2, we get
[Equation]
This is the desired upper bound for L tr - tr .
Equality at γ = 1 We ﬁnally show that the above upper bound becomes an equality when γ = 1 .
At γ = 1 , we have
[Equation]
Make the change of variable t = (cid:112) 1 + 4 /λ so that λ − 1 =
( t 2 − 1) / 4 , minimizing the above expression is equivalent to minimizing
[Equation]
over t > 1 .
It is straightforward to check (by computing the ﬁrst and second derivatives) that the above quantity is minimized at t = 3 with value 32 / 27 .
In other words, we have shown
[Equation]
, that is, the equality holds at γ = 1 .
Part II: Optimal rate for L tr - val
We now prove the result on L tr - val , that is,
[Equation]
First, equality (ii) follows from Corollary 8 and the fact that ( d + n + 1) /n
→ 1 + γ .
Second, the “ ≥ ” direction of equality (i) is trivial (since we always have “ inf lim ≥ lim inf ”).
Therefore we get the “ ≥ ” direction of the overall equality, and it remains to prove the “ ≤ ” direction.
For the “ ≤ ” direction, we ﬁx any λ > 0 , and bound AsymMSE( (cid:98) w tr - val 0 ,T ( n 1 , n 2 ; λ ))
(and consequently its limit as d, n → ∞ .)
We have from Lemma 5 that
[Equation]
≤ d
[Equation] [Equation]
Observe that
[Equation]
where (i) follows from the convexity of t (cid:55)→ λ 2 / ( t + λ ) 2 on t ≥ 0 ; (ii) follows from the same convexity and Jensen’s inequality, and (iii) is since E
[Equation]
X t
[Equation]
(cid:105) =
d .
Applying this in the preceeding bound yields
[Equation]
Further plugging in n 1 = ns and n 2 = n (1 − s ) for any s ∈ (0 , 1) yields
[Equation]
Finally, the right-hand side is minimized at λ → ∞ and s = 0 , from which we conclude that
[Equation]
which is the desired “ ≤ ” direction.
C C ONNECTIONS TO B AYESIAN ESTIMATOR
Here we discuss the relationship between our train-train meta-learining estimator using ridge regression solvers and a Bayesian estimator under a somewhat natural hierarchical generative model for the realizable setting in Section 4.
We show that these two estimators are not equal in general, albeit they have some similarities in their expressions.
We consider the following hierarchical probabilitistic model:
[Equation]
This model is similar to our realizable linear model (6), except that w 0 has a prior and that there is observation noise in the data (such that data likelihoods and posteriors are well-deﬁned).
We also note that the R 2 /d variance for w t guarantees that E [ (cid:107) w t − w 0 ,(cid:63) (cid:107) 2 ] =
R 2 , consistent with our deﬁnition (7).
Bayesian estimator We now derive the Bayesian posterior mean estimator of w 0 ,(cid:63) , which requires us to compute the posterior distribution of w 0 ,(cid:63) given the data { ( X t , y
[Equation]
Tt
[Equation]
We begin by computing the likelihood of one task by marginalizing over w t :
[Equation] [Equation]
where (i) is obtained by integrating a multivariate Gaussian density over w t
[Equation]
and “ ∝ ” drops all the terms that do not depend on w 0 ,(cid:63) .
Therefore, by the Bayes rule
[Equation]
the overall posterior distribution of w 0 ,(cid:63) is given by
[Equation]
This means that the posterior distribution of w 0 ,(cid:63) is Gaussian, with mean , i.e. the Bayesian estimator, equal to(cid:98)
[Equation]
where
[Equation]
We note that (cid:98) w
[Equation]
0 ,T has a similar form as our train-train estimator, but is not exactly the same.
Indeed, recall the closed form of our train-train estimator is (cf.
[Equation]
where
[Equation]
As (cid:98) w Bayes 0 ,T uses the inverse and (cid:98) w tr - tr 0 ,T uses the squared inverse , these two sets of estimators are not the same in general, no matter how we tune the λ in the train-train estimator.
This is true even if we set σ w = ∞ so that the prior of w 0 ,(cid:63) becomes degenerate (and the Bayesian estimator reduces to the MLE).
D D ETAILS ON THE FEW - SHOT IMAGE CLASSIFICATION EXPERIMENT
Here we provide additional details of the few-shot image classiﬁcation experiment in Section 5.2.
Optimization and architecture For both methods, we run a few gradient steps on the inner argmin problem to obtain (an approximation of) w t , and plug w t into the ∇ w 0 (cid:96) { tr - val , tr - tr } t ( w 0 ) (which involves w t through implicit function differentiation) for optimizing w 0 in the outer loop.
For both train-train and train-val methods, we use the standard 4 -layer convolutional network in (Finn et al., 2017; Zhou et al., 2019) as the backbone (i.e. the architecture for w t ).
We further tune their hyper-parameters, such as the regularization constant λ , the learning rate (initial learning rate and its decay strategy), and the gradient clipping threshold.
Datasets We experiment on miniImageNet (Ravi & Larochelle, 2017) and tieredImageNet (Ren et al., 2018).
MiniImageNet consists of 100 classes of images from ImageNet (Krizhevsky et al., 2012) and each class has 600 images of resolution 84 ×
×
We use 64 classes for training,classes for validation, and the remaining 20 classes for testing (Ravi & Larochelle, 2017).
TieredImageNet consists of 608 classes from the ILSVRC- 12 data set (Russakovsky et al., 2015) and each image is also of resolution 84 × 84 × 3 .
TieredImageNet groups classes into broader hierarchy categories corresponding to higher-level nodes in the ImageNet.
Speciﬁcally, its top hierarchy has 20 training categories ( 351 classes), 6 validation categories ( 97 classes) and 8 test categories ( classes).
This structure ensures that all training classes are distinct from the testing classes, providing a more realistic few-shot learning scenario.
D. E FFECT OF THE SPLIT RATIO FOR THE TRAIN - VAL METHOD
We further tune the split ( n 1 , n 2 ) in the train-val method and report the results in Table 2.
As can be seen, as the number of test samples n 2 increases, the percent classiﬁcation accuracy on both the miniImageNet and tieredImageNet datasets becomes higher.
This testiﬁes our theoretical afﬁrmation in Corollary 8.
However, note that even if we take the best split ( n 1 , n 2 ) =
(5 , 25) (and compare again with Table 1), the train-val method still performs worse than the train-train method.
We remark that our theoretical results on train-train performing better than train-val (in Section 4) rely on the assumptions that the data can be exactly realized by the representation and contains no label noise.
Our experimental results here may suggest that the miniImageNet and tieredImageNet few-shot tasks may have a similar structure (there exists a NN representation that almost perfectly realizes the label with no noise) that allows the train-train method to perform better than the trainval method.
[Table]
E C OMPARISON WITH C ROSS -V ALIDATION ON S YNTHETIC D ATA
We test the effect of using cross-validation for the train-val method on the same synthetic data (realizable linear centroid meta-learning) as in Section 5.1.
Method We ﬁx the number of per-task data n = 20 , and use 4 -fold cross validation in the following two settings: ( n 1 , n 2 ) =
(5 , 15) , and ( n 1 , n 2 )
= (15 , 5) .
In both cases, we partition the data into 4 parts each with 5 data points, and we roulette over 4 possible partitions of which one as train and which one as validation.
The estimated optimal (cid:98) w cv 0 is obtained by minimize the averaged train-val loss over the 4 partitions:
[Equation]
where superscript j denotes the index of the cross-validation.
The performance is depicted in Figure 2.
tr-val, n1 = 15, ref.
curve 1+ 4d/n tr-val, n1 = 5, ref.
curve 1+4d/3n tr-tr, ref.
curve tr-val + cross-validation, n1= tr-val + cross-validation, n1= <latexit sha1_base64="/j/B0g/wUZpxCYZLQ2Xhg+21US0=">AAACXnicbVFBb9MwFHazMUZhrIMLEhdrDRKHEpJKCI4TXDgOad0mLSVynJfVmmNH9ku3ysrv22/gttNuXOGK0/XAur2LP33fe/78Pue1FBbj
[Equation]
Fcv2Nh8svV
[Equation]
1n
[Equation]
Yudl7uDvVfHVjeGw4Rrqc1pzixIoWCCAiWc1gZYlUs4yS
[Equation]
dfrJHIwVWh3hooZpxc6VKAVn6KlswMIUpMzGIQVjtKG6pGF6KQqYMXRpxXCWl
[Equation]
xgPD76uItomb8k
[Equation]
kkEwIJ9fkN
[Equation]
/4BEyK3eg
[Equation]
</latexit> ` 2 error of b w { tr
[Equation]
tr
[Equation]
tr
[Equation] [Equation]
v.s. d/n ratio <latexit
[Figure]
T
[Figure]
v
[Figure]
ﬁxed).
For the cross-validation method, the regularization coefﬁcient λ = 0 .
5 is tuned.
Result As showin in Figure 2, for both ( n 1 , n 2 ) =
(15 , 5) and ( n 1 , n 2 ) =
(5 , 15) , using cross-validation consistently beats the performance of the train-val method.
This demonstrates the variance-reduction effect of cross-validation.
Note that the best performance (among the crossvalidation methods) is still achieved at n 1 = 5 , similar as for the vanilla train-val method.
However, numerically, the best cross-validation performance is still not as good as the train-train method.