Provable Generalization of Overparameterized Meta-learning Trained with SGD
Anonymous Author(s)
AffiliationAddress email
Abstract
Despite the superior empirical success of deep meta-learning, theoretical understanding of overparameterized meta-learning is still limited.
This paper studies thegeneralization of a widely used meta-learning approach, Model-Agnostic MetaLearning (MAML), which aims to find a good initialization for fast adaptationto new tasks.
Under a mixed linear regression model, we analyze the generalization properties of MAML trained with SGD in the overparameterized regime.
We provide both upper and lower bounds for the excess risk of MAML, whichcaptures how SGD dynamics affect these generalization bounds.
With such sharpcharacterizations, we further explore how various learning parameters impact thegeneralization capability of overparameterized MAML, including explicitly identifying typical data and task distributions that can achieve diminishing generalizationerror with overparameterization, and characterizing the impact of adaptation learning rate on both excess risk and the early stopping time.
Our theoretical findingsare further validated by experiments.
Introduction
Meta-learning [22] is a learning paradigm which aims to design algorithms that are capable of gainingknowledge from many previous tasks and then using it to improve the performance on future tasksefficiently.
It has exhibited great power in various machine learning applications spanning overfew-shot image classification [31, 32], reinforcement learning [21] and intelligent medicine [20].
One prominent type of meta-learning approaches is an optimization-based method, Model-Agnostic
Meta-Learning (MAML) [16], which achieves impressive results in different tasks [30, 4, 2].
Theidea of MAML is to learn a good initialization ω ∗ , such that for a new task we can adapt quicklyto a good task parameter starting from ω ∗ .
MAML takes a bi-level implementation: the inner-levelinitializes at the meta parameter and takes task-specific updates using a few steps of gradient descent(GD), and the outer-level optimizes the meta parameter across all tasks.
With the superior empirical success, theoretical justifications have been provided for MAML andits variants over the past few years from both optimization [18, 36, 14, 25] and generalizationperspectives [1, 11, 15, 9].
However, most existing analyses did not take overparameterization intoconsideration, which we deem as crucial to demystify the remarkable generalization ability of deepmeta-learning [37, 22].
More recently, [35] studied the MAML with overparameterized deep neuralnets and derived a complexity-based bound to quantify the difference between the empirical andpopulation loss functions at their optimal solutions.
However, complexity-based generalizationbounds tend to be weak in the high dimensional, especially in the overparameterized regime.
Recentworks [6, 39] developed more precise bounds for overparameterized setting under a mixed linearregression model, and identified the effect of adaptation learning rate on the generalization.
Yet, theyconsidered only the simple isotropic covariance for data and tasks, and did not explicitly capture how the generalization performance of MAML depends on the data and task distributions.
Therefore, thefollowing important problem still remains largely open:
Can overparameterized MAML generalize well to a new task, under general data and task
distributions
In this work, we utilize the mixed linear regression, which is widely adopted in theoretical studies formeta-learning [27, 6, 12, 3], as a proxy to address the above question.
In particular, we assume thateach task τ is a noisy linear regression and the associated weight vector is sampled from a commondistribution.
Under this model, we consider one-step MAML meta-trained with stochastic gradientdescent (SGD), where we minimize the loss evaluated at single GD step further ahead for each task.
Such settings correspond to real-world implementations of MAML [17, 28, 22] and are extensivelyconsidered in theoretical analysis [14, 8, 15].
The focus of this work is the overparameterized regime,i.e., the data dimension d is far larger than the meta-training iterations T ( d ≫ T ).
1.1 Our Contributions
Our goal is to characterize the generalization behaviours of the MAML output in the overparameterized regime, and to explore how different problem parameters, such as data and task distributions, theadaptation learning rate β tr , affect the test error.
The main contributions are highlighted below.
• Our first contribution is a sharp characterization (both upper and lower bounds) of the excess
risk of MAML trained by SGD.
The results are presented in a general manner, which depend
on a new notion of effective meta weight, data spectrum, task covariance matrix
and other
hyperparameters such as training and test learning rates.
In particular, the effective meta
weight captures an essential property of MAML, where the inner-loop gradient updates
have distinctive effects on different dimensions of data eigenspace,
i
e., the importance of
"leading" space will be magnified whereas the "tail" space will be suppressed.
• We investigate the influence of data and task distributions on the excess risk of MAML.
For
log -decay data spectrum, our upper and lower bounds establish a sharp phase transition of
the generalization.
Namely, the excess risk vanishes for large T (where benign fitting occurs)
if the data spectrum decay rate is faster than the task diversity rate, and non-vanishing riskoccurs otherwise.
In contrast, for polynomial or exponential data spectrum decays, excess
risk always vanishes for large T irrespective of the task diversity spectrum.•
We showcase the important role the adaptation learning rate β tr plays in the excess riskand the early stopping time of MAML.
We provably identify a novel tradeoff betweenthe different impacts of β tr on the "leading" and "tail" data spectrum spaces as the mainreason behind the phenomena that the excess risk will first increase then decrease as β tr
changes from negative to positive values under general data settings.
This complements the
explanation based only on the "leading" data spectrum space given in [6] for the isotropic
case.
We further theoretically illustrate that β tr plays a similar role in determining the earlystopping time,
i
e., the iteration at which MAML achieves steady generalization error.
Notations.
We will use bold lowercase and capital letters for vectors and matrices respectively.
N (cid:0) 0 , σ 2 (cid:1) denotes the Gaussian distribution with mean 0 and variance σ 2 .
We use f ( x ) ≲ g ( x ) todenote the case f ( x ) ≤ cg ( x ) for some constant c > 0 .
We use the standard big-O notation and itsvariants: O ( · ) , Ω( · ) , where T is the problem parameter that becomes large.
Occasionally, we use thesymbol (cid:101) O ( · ) to hide polylog( T ) factors.
1 ( · ) denotes the indicator function.
Let x + = max { x, 0 } .
Related Work
Statistical theory for MAML-type approaches.
One line of theoretical analyses lie in the statisticalaspect.
[15] studied the generalization of MAML on recurring and unseen tasks.
Informationtheory-type generalization bounds for MAML were developed in [26, 9].
[8] characterized thegap of generalization error between MAML and Bayes MAML.
[35] provided the statistical errorbound for MAML with overparameterized DNN.
Our work falls into this category, where theoverparameterization has been rarely considered in previous works.
Note that [35] only derived thegeneralization bound from the complexity-based perspective to study the difference between theempirical and population losses for the obtained optimization solutions.
Such complexity bound is typically related to the data dimension [29] and may yield vacuous bound in the high dimensionalregime.
However, our work show that the generalization error of MAML can be small even the datadimension is sufficiently large.
Overparamterized meta-learning.
[13, 34] studied overparameterized meta-learning from a representation learning perspective.
The most relevant papers to our work are [39, 6], where they derivedthe population risk in overparameterized settings to show the effect of the adaptation learning ratefor MAML.
Our analysis differs from these works from two essential perspectives: i).
we analyzethe excess risk of MAML based on the optimization trajectory of SGD in non-asymptotic regime,highlighting the dependence of iterations T , while they directly solved the MAML objective asymptotically; ii).
[39, 6] mainly focused on the simple isotropic case for data and task covariance, whilewe explicitly explore the role of data and task distributions under general settings.
More detailed discussions for related work can be found in Appendix.
Preliminary
3.1 Meta Learning Formulation
In this work, we consider a standard meta-learning setting [15], where a number of tasks share somesimilarities, and the learner aims to find a good model prior by leveraging task similarities, so that thelearner can quickly find a desirable model for a new task by adapting from such an initial prior.
Learning a proper initialization.
Suppose we are given a collection of tasks { τ t } Tt =1 sampled from some distribution T .
For each task τ t , we observe N samples D t ≜ ( X t , y t ) =(cid:8) ( x t,j , y t,j
) ∈ R d × R (cid:9) j ∈ [ N ] i.i.d.
∼ P ϕ t ( y | x ) P ( x ) , where ϕ t is the model parameter for the t -thtask.
The collection of {D t } Tt =1 is denoted as D .
Suppose that D t is randomly split into training andvalidation sets, denoted respectively as D in t ≜ ( X in t , y in t ) and D out t ≜ ( X out t , y out t ) , correspondinglycontaining n 1 and n 2 samples (i.e., N = n 1 + n 2 ).
We let ω ∈ R d denote the initialization variable.
Each task τ t applies an inner algorithm A with such an initial and obtains an output A ( ω ; D in t ) .
Thus,the adaptation performance of ω for task τ t can be measured by the mean squared loss over thevalidation set given by ℓ ( A ( ω ; D in t ); D out t ) :
= 12 n 2 (cid:80) n 2 j =1 (cid:0)(cid:10) x out t,j , A ( ω ; D in t ) (cid:11) − y out t,j (cid:1) 2 .
The goal ofmeta-learning is to find an optimal initialization (cid:98) ω ∗ ∈ R
d
by minimizing the following empiricalmeta-training loss:
[Equation]
where (cid:98) L ( A , ω ; D )
= 1 T
T (cid:88) t = ℓ ( A ( ω ; D in t ); D out t ) .
In the testing process, suppose a new task τ sampled from T is given, which is associated with thedataset Z consisting of m points with the task.
We apply the learned initial (cid:98) ω ∗ as well as the inneralgorithm
A on Z to produce a task predictor.
Then the test performance can be evaluated via thefollowing population loss:
[Equation]
Inner Loop with one-step GD.
Our focus of this paper is the popular meta-learning algorithm
MAML [16], where inner stage takes a few steps of GD update initialized from ω .
We consider onestep for simplicity, which is commonly adopted in the previous studies [6, 10, 19].
Formally, for anyω ∈ R d , and any dataset ( X , y ) with n samples, the inner loop algorithm for MAML with a learningrate β is given by
[Equation]
We allow the learning rate to differ at the meta-training and testing stages, denoted as β tr and β terespectively.
Moreover, in subsequent analysis, we will include the dependence on the learning rateto the inner loop algorithm and loss functions as A ( ω , β ; ( X , y )) , (cid:98) L ( A , ω , β ; D ) and L ( A , ω , β ) .
Outer Loop with SGD.
We adopt SGD to iteratively update the meta initialization variable ω basedon the empirical meta-training loss eq.
(1), which is how MAML is implemented in practice [17].
Specifically, we use the constant stepsize SGD with iterative averaging [15, 12, 11], and the algorithmis summarized in Algorithm 1.
Note that at each iteration, we use one task for updating the metaparameter, which can be easily generalized to the case with a mini-batch tasks for each iteration.
Stepsize α > 0 , meta learning rate β tr >
Initialization
Randomly divided into t
1 = ω
[Equation]
− α ∇ ℓ ( A ( ω , β tr ; D t )
Meta Excess Risk of SGD.
Let ω ∗ denote the optimal solution to the population meta-test erroreq.
We define the following excess risk for the output ω T of SGD:
[Equation] [Equation]
which identifies the difference between adapting from the SGD output
ω T and from the optimalinitialization ω ∗ .
Assuming that each task contains a fixed constant number of samples, the totalnumber of samples over all tasks is O ( T ) .
Hence, the overparameterized regime can be identified asd ≫ T , which is the focus of this paper, and is in contrast to the well studied underparameterizedsetting with finite dimension d ( d ≪ T ) .
The goal of this work is to characterize the impact of SGDdynamics, demonstrating how the iteration T affects the excess risk, which has not been consideredin the previous overparameterized MAML analysis [6, 39].
3.2 Task and Data Distributions
To gain more explicit knowledge of MAML, we specify the task and data distributions in this section.
Mixed Linear Regression.
We consider a canonical case in which the tasks are linear regressions.
This setting has been commonly adopted recently in [6, 3, 27].
Given a task τ , its model parameter ϕis determined by θ ∈ R d , and the output response is generated as follows:
[Equation]
where x is the input feature, which follows the same distribution P x across different tasks, and z is thei.i.d.
Gaussian noise sampled from N (0 , σ 2 ) .
The task signal θ has the mean θ ∗ and the covarianceΣ θ ≜ E [ θθ ⊤ ] .
Denote the distribution of θ as P θ .
We do not make any additional assumptions on
P θ , whereas recent studies on MAML [6, 39] assume it to be Gaussian and isotropic.
Data distribution.
For the data distribution P x , we first introduce some mild regularity conditions:
1. x ∈ R
d is mean zero with covariance operator Σ
= E [ xx ⊤ ] ;
The spectral decomposition of Σ is V Λ V ⊤ =
(cid:80) i>
λ i v i v ⊤ i , with decreasing eigenvaluesλ 1 ≥ · · · ≥ λ d > 0 , and suppose (cid:80) i>
λ i < ∞ .3.
Σ − 12 x is σ x -subGaussian.
To analyze the stochastic approximation method SGD, we take the following standard fourth momentcondition [38, 24, 7].
Assumption 1 (Fourth moment condition) .
There exist positive constants c 1 , b 1 > 0 , such that forany positive semidefinite (PSD) matrix A , it holds that
[Equation]
For the Gaussian distribution, it suffices to take c 1 = 3 , b 1 = 2 .
3.3 Connection to a Meta Least Square Problem
After instantiating our study on the task and data distributions in the last section, note that∇ ℓ ( A ( ω , β tr ; D in t ); D out t ) is linear with respect to ω .
Hence, we can reformulate the problem eq.
(1)as a least square (LS) problem with transformed meta inputs and output responses.
Proposition 1 (Meta LS Problem) .
Under the mixed linear regression model, the expectation of themeta-training loss eq.
(1) taken over task and data distributions can be rewritten as:
[Equation]
The meta data are given by
[Equation]
where X in ∈ R n 1 × d , z in ∈ R n 1 , X out ∈ R n 2 × d and z out ∈ R n 2 denote the inputs and noise fortraining and validation.
Furthermore,we have
[Equation]
Therefore, the meta-training objective is equivalent to searching for a ω , which is close to the taskmean θ ∗ .
Moreover, with the specified data and task model, the optimal solution for meta-testloss eq.
(2) can be directly calculated [19], and we obtain ω ∗ =
E [ θ ] = θ ∗ .
Hence, the metaexcess risk eq.
(4) is identical to the standard excess risk [5] for the linear model eq.
(8), i.e.,
R ( ω T , β te ) =
E B , γ 1 2 (cid:104)
∥ B ω T − γ ∥ 2 −
∥
B θ ∗ − γ ∥ 2 (cid:105) , but with more complicated input and outputdata expressions.
The following analysis will focus on this transformed linear model.
Furthermore, we can calculate the statistical properties of the reformed input B , and obtain the meta-covariance:
[Equation]
where F = E [ xx ⊤ Σ xx ⊤ ] .
Let X ∈ R n × d denote the collection of n i.i.d.
samples from P x , and denote
H n,
[Equation]
We can then write E [ B ⊤ B ] =
H n 1 ,β tr .
Regarding the form of B and H n 1 ,β tr , we need some furtherconditions on the higher order moments of the data distribution.
Assumption 2 (Commutity) .
F = E [ xx ⊤ Σxx ⊤ ] commutes with the data covariance Σ .
Assumption 2 holds for Gaussian data.
Such commutity of Σ has also been considered in [38].
Assumption 3 (Higher order moment condition) .
Given | β | < 1 λ 1 and Σ , there exists a constant
C ( β, Σ ) > 0 , for large n > 0 , s.t. for any unit vector v ∈ R d , we have:
[Equation]
In Assumption 3, the analytical form of C ( β, Σ) can be derived if Σ − 12 x is Gaussian.
Moreover, ifβ = 0 , then we obtain C ( β, Σ ) = 1 .
Further technical discussions are presented in Appendix.
Main Results
In this section, we present our analyses on generalization properties of MAML optimized by average
SGD and derive insights on the effect of various parameters.
Specifically, our results consist of threeparts.
First, we characterize the meta excess risk of MAML trained with SGD.
Then, we establish thegeneralization error bound for various types of data and task distributions, to reveal which kind ofoverparameterization regarding data and task is essential for diminishing meta excess risk.
Finally,we explore how the adaptation learning rate β tr affects the excess risk and the training dynamics.
4.1 Performance Bounds
Before starting our results, we first introduce relevant notations and concepts.
We define the followingrates of interest (See Remark 3 for further discussions)
[Equation]
Moreover, for a positive semi-definite matrix H , s.t. H and Σ can be diagonalized simultaneously,let µ i ( H ) denote its corresponding eigenvalues for v i , i.e. H =
(cid:80) i µ i ( H ) v i v
⊤ i
(Recall v
i is thei -th eigenvector of Σ ).
We next introduce the following new notion of the effective meta weight , which will serve as animportant quantity for capturing the generalization of MAML.
Definition 1 (Effective Meta Weights) .
For | β tr | , | β te | < 1 /λ 1 , given step size α and iteration T ,define
[Equation]
We call µ i ( H m,β te ) /µ
i ( H n 1 ,β tr ) and µ i ( H m,β te ) µ i ( H n 1 ,β tr ) the meta ratio (See Remark 2).
We omit the arguments of the effective meta weight Ξ i for simplicity in the following analysis.
Our first results characterize matching upper and lower bounds on the meta excess risk of MAML interms of the effective meta weight.
Theorem 1 (Upper Bound) .
Let ω i = ⟨ ω 0 − θ ∗ , v i ⟩ .
If | β tr | , | β te | < 1 /λ 1 , n 1 is large ensuring thatµ
i ( H n 1 ,β tr ) > 0 , ∀ i and α < 1 / ( c ( β tr , Σ ) tr( Σ )) , then the meta excess risk R ( ω T , β te ) is boundedabove as follows
[Equation]
where
Bias =
[Equation]
Remark 1 .
The primary error source of the upper bound are two folds.
The bias term corresponds tothe error if we directly implement GD updates towards the meta objective eq.
The variance erroris composed of the disturbance of meta noise ξ (the V 1 term), and the randomness of SGD itself (the
V 2 term).
Regardless of data or task distributions, for proper stepsize α , we can easily derive that thebias term is O ( 1 T ) , and the V 2 term is also O ( 1 T ) , which is dominated by V 1 term ( Ω(1) ).
Hence, toachieve the vanishing risk, we need to understand the roles of Ξ i and f ( · )
Remark 2 (Effective Meta Weights) .
By Definition 1, we separate the data eigenspace into “ leading ”( ≥ 1 αT ) and “ tail ” ( < 1 αT ) spectrum spaces with different meta weights.
The meta ratios indicate theimpact of one-step gradient update.
For large n , µ i ( H n,β ) ≈ (1 − βλ i )
2 λ i , and hence a larger β tr intraining will increase the weight for “leading” space and decrease the weight for “tail” space, while alarger β te always decreases the weight.
Remark 3 (Role of f ( · ) ) .
f ( · ) in variance term consists of various sources of meta noise ξ , includinginner gradient updates ( β ), task diversity ( Σ θ ) and noise from regression tasks ( σ ).
As mentionedin Remark 1, understanding f ( · ) is critical in our analysis.
Yet, due to the multiple randomnessorigins, techniques for classic linear regression [38, 24] cannot be directly applied here.
Our analysisovercomes such non-trivial challenges.
g ( · ) in Theorem 2 plays a similar role to f ( · ) .
Therefore, Theorem 1 implies that overparameterization is crucial for diminishing risk under thefollowing conditions:
• For f ( · ) : tr( ΣΣ θ ) and tr( Σ 2 ) is small compared to T ;
• For Ξ i : the dimension of "leading" space is o ( T ) , and the summation of meta ratio over
"tail" space is o ( 1 T ) .
We next provide a lower bound on the meta excess risk, which matches the upper bound in order.
Theorem 2 (Lower Bound) .
Following the similar notations in Theorem 1, Then
R ( ω T ,
[Equation]
Our lower bound can also be decomposed into bias and variance terms as the upper bound.
The biasterm well matches the upper bound up to absolute constants.
The variance term differs from the upperbound only by 1 n 2 , where n 2 is the batch size of each task, and is treated as a constant (i.e., does notscale with T ) [23, 33] in practice.
Hence, in the overparameterized regime where d ≫ T and T tendsto be sufficiently large, the variance term also matches that in the upper bound w.r.t.
T .
4.2 The Effects of Task Diversity
From Theorem 1 and Theorem 2, we observe that the task diversity Σ θ in f ( · ) and g ( · ) plays acrucial role in the performance guarantees for MAML.
In this section, we explore several types ofdata distributions to further characterize the effects of the task diversity.
We take the single task setting as a comparison with meta-learning, where the task diversity diminishes(tentatively say Σ θ → 0 ), i.e., each task parameter θ
= θ ∗ .
In such a case, it is unnecessary todo one-step gradient in the inner loop and we set β tr = 0 , which is equivalent to directly running
SGD.
Formally, the single task setting can be described as outputting ω sin T with iterative SGD thatminimizes (cid:98) L ( A , ω , 0; D ) with meta linear model as γ = 1 √ n 2 ( X out θ ∗ + z out ) .
Theorem 1 implies that the data spectrum should decay fast, which leads to a small dimension of"leading" space and small meta ratio summation over "tail" space.
Let us first consider a relativelyslow decaying case: λ k = k − 1 log − p ( k + 1) for some p > 1 .
Applying Theorem 1, we immediatelyderive the theoretical guarantees for single task:
Lemma 1 (Single Task) .
If | β te | < 1 λ 1 and if the spectrum of Σ satisfies λ k = k − 1 log − p
( k + 1) ,then R ( ω sin T , β te )
=
O ( 1 log p ( T ) )
At the test stage, if we set β te = 0 , then the meta excess risk for the single task setting, i.e., R ( ω sin T , 0) ,is exactly the excess risk in classical linear regression [38].
Lemma 1 can be regarded as a generalizedversion of Corollary 2.3 in [38], where they provide the upper bound for R ( ω sin T , 0) , while we allow aone-step fine-tuning for testing.
Lemma 1 suggests that the log -decay is sufficient to assure that R ( ω sin T , 0) is diminishing whend ≫ T .
However, in meta-learning with multi-tasks, the task diversity captured by the task spectraldistribution can highly affect the meta excess risk.
In the following, our Theorem 1 and Theorem 2(i.e., upper and lower bounds) establish a sharp phase transition of the generalization for MAML forthe same data spectrum considered in Lemma 1, which is in contrast to the single task setting (see
Lemma 1), where log -decay data spectrum always yields vanishing excess risk.
Proposition 2 (MAML, log -Decay Data Spectrum) .
Given | β tr | , | β te | < 1 λ 1 , under the same data distribution as in Lemma 1, and the spectrum of Σ θ , denoted as ν i , satisfies ν k = log r
( k + 1) for some r > 0 , then
[Equation]
Proposition 2 implies that under log -decay data spectrum parameterized by p , the meta excess riskof MAML experiences a phase transition determined by the spectrum parameter r .
Since large rimplies large eigenvalues and high variations for task vectors, we adopt r to measure the diversityof task distributions, and call r as the task diversity in the sequel.
While slower task diversity rater < 2 p − 1 guarantees vanishing excess risk, faster task diversity rate r ≥ 2 p − 1 necessarily resultsin non-vanishing excess risk.
Proposition 2 and Lemma 1 together indicate that while log -decaydata spectrum always yields benign fitting (vanishing risk) in the single task setting, it can yieldnon-vanishing risk in meta learning due to fast task diversity rate.
We further validate our theoretical results in Proposition 2 by experiments.
We consider the casep = 2 .
As shown in Figure 1a, when r < 2 p − 1 , the test error quickly converges to the Bayes error.
When r > 2 p − 1 , Figure 1b illustrates that MAML already converges on the training samples, butthe test error (which is further zoomed in Figure 1c) levels off and does not vanish, showing MAMLgeneralizes poorly when r > 2 p − 1 .
[Figure]
[Figure]
Furthermore, we show that the above phase transition that occurs for log -decay data distributions nolonger exists for data distributions with faster decaying spectrum.
Proposition 3 (MAML, Fast-Decay Data Spectrum) .
Under the same task distribution as in Proposition 2, i.e., the spectrum of Σ θ , denoted as ν i , satisfies ν k = log r
( k + 1) =
(cid:101) O
(1) for some r > 0 ,and the data distribution satisfies:
λ k = k − q for some q > 1 , R ( ω sin T , β te )
[Figure] λ k = e − k , R ( ω sin T , β te ) = (cid:101) O
( 1 T ) and R ( ω T , β te ) = (cid:101) O
( 1 T )
4.3 On the Role of Adaptation Learning Rate
The analysis in [6] suggests a surprising observation that a negative learning rate (i.e., when β tr takesa negative value) optimizes the generalization for MAML under mixed linear regression models.
Their results indicate that the testing risk initially increases and then decreases as β tr varies fromnegative to positive values around zero for Gaussian isotropic input data and tasks.
Our followingproposition supports such a trend, but with a novel tradeoff in SGD dynamics as a new reason forthe trend, under more general data distributions.
Denote ω βT as the average SGD solution of MAMLafter T iterations that uses β as the inner loop learning rate.
Proposition 4.
Let s =
T log − p ( T ) and
d = T log q ( T ) , where p, q > 0 .
Suppose P x is Gaussian and the spectrum of Σ satisfies
[Equation]
Suppose the spectral parameter ν i of Σ θ is O (1) , and let the step size α = 1 2 c ( β tr , Σ )tr( Σ ) .
Then forlarge n 1 , | β tr | , | β te | < 1 λ 1 , we have
[Equation]
The first two terms in the bound of eq.
(11) correspond to the impact of effective meta weights Ξ ion the "leading" and "tail" spaces, respectively, as we discuss in Remark 2.
Clearly, the learningrate β tr plays a tradeoff role in these two terms, particularly when p is close to q .
This explains thefact that the test error first increases and then decreases as β tr varies from negative to positive valuesaround zero.
Such a tradeoff also serves as the reason for the first-increase-then-decrease trend of thetest error under more general data distributions as we demonstrate in Figure 2.
This complementsthe reason suggested in [6], which captures only the quadratic form
[Equation]
) 2 of β tr for isotropic Σ ,where there exists only the "leading" space without "tail" space.
Based on the above results, incorporating with our dynamics analysis, we surprisingly find that β trnot only affects the final risk, but also plays a pivot role towards the early iteration that the testingerror tends to be steady.
To formally study such a property, we define the stopping time as follows.
R i s k 𝛽 !"
(a) λ i
=
i log( i +1)
R i s k 𝛽 !"
(b) λ i
=
i log( i +1)
R i s k 𝛽 !" (c) λ i
= 1 i
Definition 2 (Stopping time) .
Given β tr , β te , for any ϵ > 0 , the corresponding stopping timet ϵ ( β tr , β te ) is defined as:
[Equation]
In the sequel, we may omit the arguments in t ϵ for simplicity.
We consider the similar data distributionin Proposition 4 but parameterized by K , i.e., s =
K log − p ( K ) and d = K log q ( K ) , where p, q > 0 .
Then we can derive the following characterization for t ϵ .
Corollary 1.
If the assumptions in Proposition 4 hold and p = q .
Further, let Σ θ =
η 2 I , and| β tr | < 1 λ 1 .
Then for t ϵ ( β tr , β te ) ∈ ( s, K ] , we have:
[Equation]
(12) where L l , L t , U l , U t > 0 are factors for "leading" and "tail" spaces that are independent of K 1 .
Equation (12) suggests that the early stopping time t ϵ is also controlled by the tradeoff role that β trplays in the "leading" ( U l , L l ) and "tail" spaces ( U t , L t ), which takes a similar form as the bound in
Proposition 4.
Therefore, the trend for t ϵ in terms of β tr will exhibit similar behaviours as the finalexcess risk, and hence the optimal β tr for the final excess risk will lead to an earliest stopping time.
We plot the training and test errors for different β tr in Figure 3, under the same data distributionsas Figure 2a to validate our theoretical findings.
As shown in Figure 3a, β tr does not make muchdifference in the training stage (the process converges for all β tr when T is larger than 100 ).
However,in Figure 3b at test stage, β tr significantly affects the iteration when the test error starts to becomerelatively flat.
Such an early stopping time first increases then decreases as β tr varies from − 0 .
5 to0 .
7 , which resembles the change of final excess risk in Figure 2a.
[Figure]
[Equation]
Conclusions
In this work, we give the theoretical treatment towards the generalization property of MAML based ontheir optimization trajectory in non-asymptotic and overparameterized regime.
We provide both upperand lower bounds on the excess risk of MAML trained with average SGD.
Furthermore, we explorewhich type of data and task distributions are crucial for diminishing error with overparameterization,and discover the influence of adaption learning rate both on the generalization error and the dynamics,which brings novel insights towards the distinct effects of MAML’s one-step gradient updates on"leading" and "tail" parts of data eigenspace.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]