U NRAVELING M ODEL -A GNOSTIC M ETA -L EARNING VIA T HE A DAPTATION L EARNING R ATE
Anonymous authors
Paper under double-blind review
A BSTRACT
Model-Agnostic Meta-Learning (MAML) aims to find initial weights that allow fast adaptation to new tasks.
The adaptation (inner loop) learning rate in MAML plays a central role in enabling such fast adaptation.
However, how to choose this value in practice and how this choice affects the adaptation error remains less explored.
In this paper, we study the effect of the adaptation learning rate in meta-learning with mixed linear regression.
First, we present a principled way to estimate optimal adaptation learning rates that minimize the population risk of MAML.
Second, we interpret the underlying dependence between the optimal adaptation learning rate and the input data.
Finally, we prove that compared with empirical risk minimization (ERM), MAML produces an initialization with a smaller average distance to the task optima, consistent with previous practical findings.
These results are corroborated with numerical experiments
I NTRODUCTION
Meta-learning or learning to learn provides a paradigm where a machine learning model aims to find a general solution that can be quickly adapted to new tasks.
Due to its fast adaptability, metalearning has been widely applied to challenging tasks such as few-shot learning (Vinyals et al., 2016;
Snell et al., 2017; Rusu et al., 2018), continual learning (Finn et al., 2019; Javed & White, 2019), and neural architecture search (Zhang et al., 2019; Lian et al., 2019).
One promising approach in meta-learning is Model-Agnostic Meta-Learning (MAML)
(Finn et al., 2017), which consists of two loops of optimization.
In the outer loop, MAML aims to learn a good meta-initialization that can be quickly adapted to new task in the inner loop with limited adaptation (parameter optimization) steps.
The double loops optimization serve as “learning-to-adapt” process, thus enabling the trained model to adapt to new tasks faster than direct Empirical Risk Minimization (ERM) algorithms (Finn et al., 2017; Raghu et al., 2020).
Recent works (Nichol et al., 2018; Fallah et al., 2020; Collins et al., 2020; Raghu et al., 2020) attribute the fast adaptability to the phenomenon that the learned meta-initialization lies in the vicinity of all task solutions.
However, the theoretical justification of this empirical statement, and more generally how fast adaptability of MAML depends on the inner loop optimization remains unclear.
As a key component of MAML, the adaptation (inner loop) learning rate (hereafter called α ) is shown empirically to plays a crucial role in determining the performance of the learned meta-initialization (Rajeswaran et al., 2019).
In particular, the value of α bridges
ERM and MAML, in the sense that the latter reduces to the former when α = 0 .
However, from a theoretical viewpoint, the dependence of MAML performance on the choice of α remains unclear, and furthermore, there lacks a precise practical guideline on how to pick a near-optimal value.
In this paper, we address these issues by answering the following two questions: (1) How to choose the optimal α that minimizes population risk of MAML?
(2) What is the effect of α on fast adaptability of MAML?
To this end, we consider a regression problem with random feature models.
For the first question, we derive the optimal α which minimizes the population risk of MAML in the limit of an infinite number of tasks.
This can then be used to estimate an effective α prior to training.
Moreover, we analyze the underlying statistical dependency between the optimal α and the input data, e.g. relation to the moments of data distribution.
This in turn allows the heuristic application of our results beyond linear models, and we demonstrate this with experiments.
To answer the second question, we compare MAML with an ERM algorithm (without inner loop optimization) in order to reflect the effect of α in optimization.
As stated in many works, like Nichol et al.
(2018), that meta-initialization learned by MAML in parameter space is close to all training tasks thus contributes to fast adaptability.
We conduct an experiment and observe that MAML with a not too large α yields a shorter mean distance to task optima than ERM.
To justify this empirical finding, we define a metric measuring the expected geometric distance between the learned meta-initialization and task optima.
We prove that in our setting, the MAML solution indeed possess a smaller value of this metric compared with that of ERM for small α , providing theoretical evidence for the observed phenomena.
Our contributions can be summarized as follows:
• We provide a principled way to select the optimal adaptation learning rate α ∗ for MAML which minimizes population risk (Theorem 1 & Proposition 1 ).
We also interpret the underlying statistical dependence of α ∗ to input data (Corollary 1 ) with two examples.
• We validate the observation that MAML learns a good meta-initialization in the vicinity of the task optima, which reveals the connection between the adaptation learning rate α and the fast adaptability in optimization.
(Theorem 2 ) • We also extend our result about the choice of α ∗ to more practical regime, including deep regression.
All of our theoretical results are well corroborated with experimental results.
P ROBLEM FORMULATION
We study the MAML algorithm under the mixed linear regression setting.
Suppose we have a task T that is sampled from the distribution D ( T ) .
Each task T corresponds to a linear relationship
[Equation]
where X T ∈ R K × d x is the input data of task T which has K vector samples { x T, 1 , ..., x T,K } , x T,j ∈
R d x i.i.d sampled from D ( x ) 1 .
For each input data, we have a mapping ϕ : R d x → R d transform each point of X T from input data space R d x to a d -dimensional feature space R d where we denote the transformation of all data in task T by Φ( X T ) =
[ ϕ ( X T, 1 ) , ..., ϕ ( X T,K )] ⊤ as the feature of that task.
Then, we assume optimal solution a T ∈ R d for task T is i.i.d sampled from D ( a ) .
The corresponding label y T ∈ R K can be obtained by (1).
Our target is to learn a model to minimize the risk of different tasks across D ( T ) .
Note that each task
T is determined by a feature-solution pair (Φ( X T ) , a T ) .
Therefore, we can formulate this multi-task problem with parameter space R d and loss function ℓ as
[Equation]
To solve this problem, ERM and MAML algorithms yield different iterations.
Specifically, ERM uses all data from all tasks to directly minimize the square error loss ℓ , such that population risk of ERM is
[Equation]
As a counterpart, MAML first adapts with an adaptation learning rate α on each task using its training set – a subset of task data in the inner loop.
Then, in the outer loop, MAML minimizes the evaluation loss for each adapted task-specific solution using a validation set.
For simplicity, since data is i.i.d sampled from the same distribution, we first consider the setting where all data in each task is used as training set and validation set in our main results.
We present later the extension of these results to the case with a different train-validation split.
(Please refer to Appendix H.1)
Thus, the general population risk of one-step MAML is defined by
[Equation]
In practice, we use the empirical objective function as a surrogate objective function.
We first sample
N tasks with task optima { a 1 , ..., a N } from D ( a ) and then sample K data for each task.
Then, the empirical risk of ERM can be specified as ˆ L r , given by
[Equation]
Similarly, we have the empirical risk of MAML, denoted as ˆ L m
[Equation]
where w ′ i = (cid:2) w − 2 α Φ( X i ) ⊤ (Φ( X i ) w − Φ( X i ) a i ) /K
(cid:3) is adapted parameters after inner loop.
In addition, we follow the original MAML (Finn et al., 2017) to use the same α for training and testing.
Notation We denote an optimal adaptation learning rate as α ∗ .
Global minima of empirical risk of
MAML and ERM (when they are unique) are denoted by w m , w r .
We write { 1 , ..., N } as [ N ] and use ∥ · ∥ to denote the Euclidean norm.
We use subscripts to index the matrices/vectors corresponding to task instances, and bracketed subscripts to index the entries of matrices.
Other notations are summarized in Appendix Table 1.
Assumption 1 (Normalization) .
For simplicity, we consider a centered parameter space such that E a ∼D ( a ) [ a ] =
and Var [ a ] = σ 2 a .
Assumption 2 (Bounded features) .
With probability 1, the covariance matrix of input features Φ( X ) ⊤ Φ( X ) is positive definite and has uniformly bounded eigenvalues from above by λ S > 0 and below by λ I > 0 .
M AIN RESULTS
In this section, we analyze MAML through the adaptation learning rate α .
Our derived insights are summarized into three theoretical results: (1) The estimation of an optimal adaptation learning rate α ∗ which minimizes MAML population risk; (2) The statistical meaning of α ∗ in terms of the data distribution, and (3)
The geometric interpretation of the effect of α on fast adaptability of MAML compared to ERM.
3. O N THE OPTIMAL ADAPTATION LEARNING RATE α ∗
We focus on the underparameterized case ( K ≥ d ) .
Given the empirical objective functions ˆ L r , ˆ L m defined in (5) and (6), we can derive the global minima by the first-order optimality condition.
We obtain the global minimum of ERM w r and minimum of MAML w m in the following closed-forms,
[Equation]
where C i ( α ) =
Φ( X i ) (cid:2) I − (2 α/K )Φ( X i ) ⊤ Φ( X i ) (cid:3) , C i ( α ) ∈
R K × d can be viewed as the adapted feature of task i .
Observe that w m ( α ) (and thus the MAML algorithm) depends on α .
If α = 0 ,
MAML reduces to ERM.
For large α , instabilities may occur, thus there may exist an optimum, α ∗ that minimizes the MAML population risk.
The later intuition is worthwhile to be proved, from which we do not have a principled way to guide the choice of optimal hyperparameter α ∗ for MAML so far.
To this end, we focus on the generalization error by taking the population risk on the global minimum of empirical risk.
In particular, we consider the population risk of the MAML optimizer in the average sense, where the average population risk is
[Equation]
(8) whose minimizer we denote as α ∗ ( N, K ) .
In this way, we eliminate randomness of the global minimum learned from sampled tasks.
The following theorem gives a precise value of α ∗ ( N, K ) in the limit N → ∞ .
Theorem 1.
Under assumptions 1 & 2, we have as N → ∞ , α ∗ ( N, K ) → α ∗ lim ( K ) , where
[Equation]
Φ( X ) ∈ R K × d , K is the sample size per task and N is the number of tasks.
The proof is found in Appendix B.
In this theorem, we give the nearly optimum α ∗ lim which is an alternative form for true optimal α , namely α ∗ , to minimize the MAML generalization error.
As dictated in (9), the desired α ∗ is determined by the feature covariance matrix in expectation.
Remark.
The precise derivation of the case where N is finite is complicated, thus we derive the limiting case here as an estimator of true α ∗ .
Our estimation α ∗ lim is the unique minimum.
We will show later that this allows us to compute near optimal values efficiently in practice, each of which is close to the optimal α ∗ ( N, K ) in corresponding problem.
Remark.
The estimator (9) can be generalized to different scenarios.
For overparameterized models, we obtain a similar result for the minimum norm solution if the number of tasks N is limited ( NK ≪ d ).
Further, we show a computationally efficient estimator (145) in Appendix H.2.
For deep learning, we can compute a range of effective α values based on α ∗ lim .
We also give the numerical form when the training data is different from the test data in each task.
These are presented in
Appendix H.4 and H.1 respectively.
In the above we considered the average population risk (8).
This simplifies the calculations of finding the α ∗ .
Below, we justify this simplification by showing that in the limit of large number of tasks, the average population risk is a good estimate of the true population risk.
Proposition 1 (Informal) .
Assume u
= C ( α ) ⊤ C ( α ) a is sub-gaussian random variable with subgaussian norm ∥ u ( i ) ∥ Ψ 2 ≤ L , assumption 1 & 2 hold, then with probability at least 1 − δ
that
[Equation]
where ε ( α, K ) =
O (1 / ( c 0 + α ) 2 ) .
Here c 0 > 0 is a constant and d is the feature size.
The proof is found in Appendix C. Proposition 1 complements Theorem 1 by guaranteeing that the gap between the average population risk and population risk with same argument α will disappear along with N goes to infinity.
Large α makes the bound tighter while small α makes ε ( α, K ) converge to a positive constant; thus (10) provides a non-vacuous bound with regard to α .
Hence, it is justified to make an estimation of α ∗ using the average case.
By Theorem 1 and Proposition 1 , we give an explicit form to estimate α ∗ for MAML where this estimation α ∗ lim is not too far from the true α ∗ of a specific case.
Later experiments show our estimation α ∗ lim is close to true α ∗ in both underparameterized and overparameterized models (see Section 5.1).
This is meaningful for selecting an α ∗ minimizing
MAML risk, instead of randomly choosing it.
Compared to (Bernacchia, 2021), we have a certain value result for α ∗ instead of range for linear models.
(Details refer to Appendix H.5)
Relation to data distribution.
After estimating the value of α ∗ through Theorem 1 , we are now interested in the statistical interpretation of α ∗ .
In particular, we aim to summarize the dependence of an estimation of a ∗ on the distribution of the inputs and tasks.
This in turn allows us to devise strategies for choosing near optimal α for MAML beyond the simple settings considered here.
Corollary 1.
With a feature mapping ϕ : R d x → R d for each data x ∈ R d x , the α ∗ lim in Theoremwill satisfy the following inequality
[Equation]
where σ 2 ( ϕ ( x 1 ) , . . .
, ϕ ( x K )) is variance of the feature.
See proof in Appendix D. According to Corollary 1 , we can see that α ∗ lim is bounded by the statistics of the input data.
These bounds are governed by the standard derivation terms.
More specifically, it (9) holds an inverse relationship to higher order moment of data distribution while its bounds (11) have an inverse relationship to data variance .
As a consequence, the α ∗ lim for different [Figure] problems mainly depend on the standard derivations.
For example, α ∗ lim and thus α ∗ will shrink to zero as the variance of data increases and vice versa.
In other words, small α is tailored to those tasks with large data variance when the model size is fixed.
To illustrate the insight more clearly, we present two examples – regression with polynomial basis functions (Example 1) and the case where Φ( X ) is a random matrix with a prescribed distribution (see Appendix E).
In this following example, we narrow the range and get the exact relationship where the expression of α ∗ lim rather than its bounds depend on data variance and model size d .
In later experiments, we also validate this relationship on various models with different basis functions.
Example 1 (Polynomial basis function) .
Assume we have K i.i.d samples x 1 , ..., x K ∼ N (0 , σ 2 ) for each task.
Consider polynomial basis function ϕ : R → R d , where ϕ ( x ) =
(1 , ..., x d − 1 ) .
Then value of α ∗ lim has an inverse relationship to σ 2 and dimension d
(Proof is in Appendix E).
3. G EOMETRIC I NTERPRETATION OF MAML A DAPTATION
In another direction, we aim to investigate geometric properties of the meta-initialization learned by
MAML as α varies.
In previous experimental investigations, it is suggested that MAML learns the near meta-initialization to all tasks Nichol et al.
or trade-offs on easy and hard tasks Fallah et al.
We can also observe the new phenomena in toy experiments.
As shown in the Figure 1 (a), we sampled 500 tasks in R 2 parameter space.
Specifically, we i.i.d sample and stack data as
X i ∈
R K × 2 , ∼ D ( x ) and task optima a i ∼ D ( a ) , a i ∈ R 2 (scattered orange dots) for each task i .
Green cross shows the location of MAML solution w m ( α ) with α = 0 , namely ERM solution w r .
Since D ( x ) , D ( a ) are some symmetric zero-mean distributions, the optimal solution is expected at the origin.
When several new training tasks (with higher penalties) have been added as shown in the red circle area, then new w r will be closer to new tasks.
Along α increasing, w m ( α ) generates a trajectory shown as the purple curve.
The dynamics of global minimum w m ( α ) will start from green cross and move away from the red circle until reach an optimum location of point (red star w m ( α ∗ lim ) ) which minimizes the total distances to red and orange dots (Other cases shown in Appendix H.4).
It indicates that the effect of α (inner loop optimization) is to help MAML minimize total distances to all training task optima.
Unlike ERM learning a biased solution to dense tasks area, MAML converges to a distance-aware solution that tries to minimize the distances within one-step adaptation at stepsize α .
The α ∗ lim is the optimum adaptation stepsize to learn the optimum location of point, or nearest point, to all tasks.
Figure 1 (b) displays the mean distance for each point in purple trajectory to all tasks.
As we can see, the distance decreases at beginning as α increases until reach the minimum.
To theoretically prove the insight in Figure 1, we characterize it by measuring the average postadaptation distance between the meta-initialization (global minimum) learned by a specific algorithm and task optima in a task distribution.
Definition 1 (Average Distance under Fast Adaptation) .
Given task distribution D ( T ) , metainitialization w
A learned by algorithm A , optimum a T of task T , the average distance under t -step ( t ≥ 0 ) fast adaptation is defined by
[Equation]
where w t A ,T is the adapted parameter of task T with t steps, η is the step size, ℓ is the loss function.
F t evaluates the distance between adapted parameters and true task optimum for a given metainitialization at any adaptation step t .
If t is small, F t describes the fast adaptation error in solution distance of the meta-initialization learned by an algorithm.
Hence, we can measure the fast adaptability of MAML with F t ( w m ) .
Observe that for small α , w m can be linearized as
[Equation]
In this regime, the effect of MAML is dictated by the α gradient ∇ α w m (0) , which can be visualized as the tangent of the purple curve at the green cross in Figure 1(b).
By comparing F t of the metainitializations w r , w m learned by ERM and MAML, we are able to find the connection between α and the fast adaptability in meta-learning, at least in the small alpha regime.
For simplicity, we assume that the input data features are uncorrelated, thus the covariance matrix is diagonal.
Theorem 2.
Let w m ( α ) , w r be the meta-initializations learned from T 1 , ..., T N by MAML and ERM.
With F t ( · ) , under Assumption 1 & 2, for any α ∈
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
[Figure]
[Equation]
number of step t , we have
[Equation]
where η is the step size in Definition 1 , (cid:101) c > 0 is a constant.
See proof in Appendix G.
This theorem prove our insight at small α that MAML has smaller average solution distance than ERM.
As it illustrated in Theorem 2 , at any step t ≥ 0 , F t ( w m ( α )) ≤ F t ( w r ) holds if α is smaller than some constant.
This means adapting to different tasks with MAML metainitialization leads to shorter average solution distance than ERM’s at any number of adaptation steps.
But the gap will disappear along number of steps t increasing to infinity, which is sensible.
Note that even t = 0 , this inequality still holds true.
Therefore the meta-initialization of MAML w m has shorter expected distance to new task than ERM w r before adaptation.
Theorem 2 has revealed the connection between α and fast adaptability.
Even with small α , MAML learns a more adaptive solution than ERM which is closer to the new tasks in expectation enabling quick approximation.
It benefits from learning a closer meta-initialization for all tasks on average.
Thus α plays a role in learning a distance-aware solution.
This result is consistent with our observation in the Figure 1.
Compared to ERM algorithms, the fast adaptability of MAML stems from the learned metainitialization determined by the adaptation learning rate α .
When facing a multi-task problem, traditional ERM algorithms bias its learned initialization to minimize the averaged risk.
However, this strategy fails to take the further adaptation into account, and thus learns a solution far from unknown task optima.
On the contrary, MAML learns a distant-aware meta-initialization and converges to the vicinity of all task optima with a limited adaptation budget (Nichol et al., 2018; Rajeswaran et al., 2019), or tends to favor “hard tasks” (Fallah et al., 2020; Collins et al., 2020).
Hence, before adaptation, ERM may have lower population risk than MAML.
However, after adaptation, the situation will reverse since MAML can adapt to most unknown task optima closer (see Figure 5(a)).
This benefit is also illustrated by (Zhou et al., 2020) that the shorter solution distance leads to a better meta-initialization for fast adaptation.
We note that “task hardness” may not always be easy to define, especially for non-linear cases (Collins et al., 2020).
Here, we instead focus on directly analyzing the geometric distance (Theorem 2 ), which has substantiated the aforementioned findings in optimization behavior from different angles.
R ELATED W ORK
Meta learning learns a general solution based on previous experience which can be quickly adapted to unknown tasks (Finn et al., 2017;
Li et al., 2017;
Snell et al., 2017; Vinyals et al., 2016; Nichol [Figure] et al., 2018; Rusu et al., 2018; Rajeswaran et al., 2019; Finn & Levine, 2018; Rajeswaran et al., 2019;
Finn et al., 2018; Yin et al., 2020) (Grant et al., 2018; Harrison et al., 2018).
One promising approach to meta-learning is Model-Agnostic Meta-Learning (MAML)
(Finn et al., 2017) which learns a meta-initialization such that the model can adapt to a new task via only few steps gradient descent.
Understanding the fast adaptability of meta-learning algorithm, especially on MAML, is now an important question.
As a variant of MAML, (Nichol et al., 2018) attribute fast adaptation to the shorter solution distance, and devises a first-order approximation algorithm based on this intuition.
Other first-order methods (Denevi et al., 2019; Zhou et al., 2019) try to achieve adaptation by adding a regularized term to get a distance-aware solution.
(Raghu et al., 2020) shows that even performing inner loop optimization on part of the parameters still leads to fast adaptation.
A shared empirical finding of these results is that MAML produces initial weights that are closer to the population optimum of individual tasks on average, and it is argued that this partly contributes to its fast adaptibility.
Here, we present a rigorous result that confirms the distance reduction property of MAML, at least in the considered setting, lending theoretical backing to these empirical observations.
On the theoretical front, analyses of meta-learning mainly focus on generalization error bounds and convergence rates (Amit & Meir, 2018; Denevi et al., 2019; Finn et al., 2019; Balcan et al., 2019;
Khodak et al., 2019;
Zhou et al., 2019; Fallah et al., 2020; Ji et al., 2020b; Zhou et al., 2020; Ji et al., 2020a).
For example, Fallah et al.
studies MAML by recasting it as SGD on a modified loss function and bound the convergence rate using the batch size and smoothness of the loss function.
Ji et al.
(2020b) extend this result to the multi-step version of MAML.
Other works (Charles &
Kone ˇ cn ` y, 2020; Wang et al., 2021; Gao & Sener, 2020; Collins et al., 2020) investigate the MAML optimization landscape and the trade-off phenomena in terms of task difficulty: e.g. MAML tend to find meta-initializations that are closer to difficult tasks.
However, the effect of inner loop learning rate α on the MAML dynamics and learned solution are not explored in these works.
Of particular relevance is the work of Bernacchia (2021), which derives, under an ideal setting of
Gaussian inputs and regression coefficients, a range of α values that can help guide its choice.
In this paper, we adopt a more general setting, where we do not assume specific input distributions.
We derive a precise optimal value of α (instead of a range), which can be estimated from input data.
Furthermore, we show using experiments that the optimal values may not be negative (c.f. Bernacchia (2021)) in the standard meta-learning setting, where the same α is used for training and testing.
E XPERIMENTS 5. E STIMATION OF α ∗
We verify our theorem through Neural Tangent Kernel (NTK) (Jacot et al., 2018) and deep learning on the Omniglot dataset (Lake et al., 2011).
In the former setting, we followed the problem setup in (Bernacchia, 2021) to perform quadratic regression.
Different from their model size of 60 , we used a two-layer Neural Tangent Kernel (NTK)
(Jacot et al., 2018) with sufficiently wide hidden layers (size 10 , 000 ).
Then, we can estimate α ∗ by the neural tangent feature to obtain α ∗ est = 1 / (2 NK ˜ σ 2 )
[Figure] [Figure] ( ˜ σ is the variance of NTK feature, whose derivation is found in Appendix H.2).
Shown as a vertical dotted line ending with the red star in Figure 2, we can see our estimation is nearly optimal.
To reduce fortuity, we choose arbitrary values of N, K to compute the estimation α ∗ lim .
Furthermore, we also test our estimation on uniform initialization with other groups of hyper-parameters and obtained similar results.
Then, for deep learning classification, we use online estimation to compute α ∗ for
ANIL Raghu et al.
(2020) on the Omniglot dataset Lake et al. To keep training stable, we normalize the features before the last layer and compute the corresponding α ∗ est . Then, we compare our estimation scheme with the default selection method where the model and training learning rates are the same. Test loss and accuracy are reported with mean and variance in Figure 3.
Both training schemes achieve similar results after 4 × 10 4 iterations.
We only plot the first 1 . 5 × 10 4 iters ( iters per scale) to see the differences clearly.
As shown, our estimation of α ∗ converges faster than that in the default configuration.
Other experimental parameters and additional results, including non-central distributions and deep regression experiments, are found in Appendix H.4.
Overall, these experiments suggest that our estimation derived in the idealized linear setting can guide practical hyper-parameter selection.
5. R ELATION OF DATA VARIANCE AND OPTIMAL α
In this section, we verified our theoretical results of α ∗ lim and its relation to data variance.
As drawn the Figure 4, value of α ∗ lim and data variance have an inverse relationship.
We first verified [Figure] this with a Gaussian basis function Φ( X ) ( ij ) = exp( − (cid:0)
X ( ij ) − µ j (cid:1) 2 / 2 σ 2 i ) . Then, we conducted experiments on three different data distributions: normal distribution N (0 , σ ) , uniform distribution U ( −√ 12 σ/ 2 , √ 12 σ/ 2) and exponential distribution Exp (1 /σ ) .
From in (a), we can see the smooth curves perfectly fitted with some inverse proportional function e.g. y = 0 .
35 /σ 2 .
Next, we used NTK as the basis function to verify our result in overparameterized regime.
We used two layers MLP with width = 10 , 240 and uniform initialization to compute the neural tangent feature.
As we can see from Figure 4(b), the diagram also shows the tendency that α ∗ lim decreases as σ increasing.
As a consequence, variance, as a part of the statistical property of data, will influence the optimal learning rate α ∗ .
5. F AST ADAPTATION
To understand the effect of α on F t , we set α = 10 − 4 to train MAML such that its global minimum w m is inched from ERM w r .
Then, we tracked their adaptation losses and adaptation errors with growing adaptation steps, shown in the Figure 5.
Adaptation loss for task i is defined by ℓ it ( w ) =
∥
Φ( X i ) Adapt ( w , i, t, η ) − y i ∥
where Adapt ( w , i, t, η ) is t -step adaptation parameter with learning rate η = 1 e − 5 .
The adaptation loss difference between MAML and ERM is described as (cid:80) 5000 i =1 ℓ it ( w m ) − ℓ it ( w r ) .
From Figure 5 (a) we can see, the loss of MAML is marginally higher than ERM before adapting.
But the difference dramatically decreases to negative values, which illustrates that MAML has better performance than ERM with only few steps adaptation.
Similar results appear on various data distributions: uniform distribution U ( − 5 , 5) , normal distribution
N (0 , 2) and exponential distribution Exp (1) .
It makes sense because w r , w m are the minimizers of non-adaptation loss and one-step adaptation loss, respectively.
Then we plot the difference of adaptation errors in distance F t ( w m ) − F t ( w r ) along adaptation step t .
In Figure 5(b) we can see,
F t of MAML is always smaller than ERM’s, including t = 0 .
Since F t measures distances of adapted solution and task optimum solution, this result has substantiated our Theorem 2 .
Furthermore, it also demonstrate that the effect of α , even it is small, is acting as the guide to find a distance-aware meta-initialization for target tasks which possesses faster adaptability compared to ERM.
C ONCLUSION
In this paper, we investigated MAML through the lens of adaptation learning rate α .
We gave a principled way to estimate an optimal adaptation learning rate α ∗ minimizing MAML population risk.
Further investigation has revealed the underlying data statistics that α ∗ depends on.
This statistical dependency also motivates us to explore other effect of α , such as the optimization behavior in a geometric context.
By studying the role of α on optimization, we confirmed theoretically that MAML obtains solutions with shorter average distance to individual task optima than ERM - an empirical observation that was suggested to contributes to MAML’s fast adaptability.
We believe these results are instructive in contributing to the theoretical understanding of meta-learning and its algorithm design.
R EPRODUCIBILITY S TATEMENT
All experimental setup are clearly described in Experiments section and Appendix H.3.
Our theoretical results are strictly computed using the equations while all experimental results are reported with mean and variances.
For every synthetic experiment, we have specified the distributions of data and task optima we use.
All hyperparameters are included as well.
Deep learning experiments are ran with original algorithm and our codes will be open to the public
A D EFINITIONS , NOTATIONS AND LEMMAS
Notation We denote an optimal adaptation learning rate as α ∗ .
Global minima of empirical risk of
MAML and ERM (when they are unique) are denoted by w m , w r .
We write { 1 , ..., N } as [ N ] and use ∥ · ∥ to denote the Euclidean norm.
We use subscripts to index the matrices/vectors corresponding to task instances, and bracketed subscripts to index the entries of matrices.
For function f depends on a, b, x , we omit other variables by f ( ..., x ) when we discuss with x .
[Table]
Definition 2 (Gamma Convergence) .
Let F n : X → R for each n ∈ N .
We say that ( F n ) n ∈ N Γ -converges to F : X → R
[Table]
and write Γ − lim n →∞ F n = F or F n Γ → F, if
• For every x ∈ X and every ( x n ) n ∈ N such that x n → x in X
[Equation]
• for every x ∈ X , there exists some ( x n ) n ∈ N such that x n → x in X and
[Equation]
Definition 3 (Lower Semicontinuous Envelope) .
Given F : X → R , the lower semicontinuous envelope (or relaxation) of F is the ”greatest lsc function bounded above by F ”
[Equation]
Lemma 1 (Remark 2.2, (Braides, 2006)) .
If F n uniform converge to F , then F n Γ → F lsc where F lsc is Lower Semicontinuous Envelope of F .
Lemma 2 ( Γ Convergence, (Braides, 2006)) .
Let X be a topological space.
Let { F n } be a equicoercive family of functions and let F n Γ -converges to F in X , then
• lim n →∞ d n = d where
d n = inf x ∈ X and d = inf x ∈ X F ( x ) .
That is
, the minima converges F n ( x ) • The minimizers of F n converge to a minimizer of F .
Proposition 2.
If both A and B are positive semidefinite, the inequality is true:
[Equation]
and if A is n -by- n symmetric PSD, we have
[Equation]
Proof.
Let a = tr ( A ) .
For PSD matrix A , we have A ⪯ aI .
Then
[Equation]
For second inequality, we can apply spectral decomposition on A as A = QDQ − 1 .
So we have
[Equation]
where { λ i } , i ∈ [1 , n ] is the eigenvalues of matrix A .
Then by Cauchy-Schwarz inequality we can get
[Equation]
Lemma 3 (Hanson-Wright inequality, (Rudelson & Vershynin, 2013)) .
Let X =
( X 1 , . . .
X n ) ∈ R n be a random vector with independent components
X i which satisfy E X i = [-
-]0 and ∥ X i ∥ ψ 2 ≤ L .
Let
A be an n × n matrix.
Then, for every t ≥ 0 ,
[Equation]
where ∥ ξ ∥ ψ 2 = sup p ≥ 1 p − 1 / 2 ( E | X | p ) 1 /p is sub-gaussian norm, ∥
A ∥ = max x ̸ =0 ∥ Ax ∥ 2 / ∥ x ∥ 2 is operator norm and ∥ A ∥ HS =
( (cid:80) i,j | a i,j | 2 ) 1 / 2 is Hilbert-Schmidt (or Frobenius) norm.
Definition 4 (Fast Adaptation Error) .
Given the task distribution D ( T ) , meta-initialization w 0 , the optimal solution of task T is w ∗ T , then t -step fast adaptation error is defined by
[Equation]
where w tT = w 0 − η (cid:80) tj ∇ w ji ℓ i ( w jT ) is the adapted parameter of task T with t steps.
Lemma 4 (Ruhe’s trace inequality, (Marshall et al., 1979)) .
If A, B are n × n positive semidefinite
Hermitian matrices with eigenvalues,
[Equation]
respectively, then
[Equation]
Proposition 3.
For any positive random variable x ∼ D ( x ) , we have following inequality holds true
[Equation]
Proof.
With the fact that
[Equation]
where x > 0 and p ( x ) > 0 .
Let f = ( xp ( x )) 12 > 0 and g = (cid:0) x 3 p ( x ) (cid:1) 12 > 0 , with Cauchy-Schwarz Inequality,
[-(21) we have that (cid:0) E x ∼D ( x ) ( x 2 ) (cid:1) 
-]
[Equation]
we have that
[Equation]
B P ROOF OF T HEOREM
Proof Sketch We list our proof steps as follows
Get global minima of ERM and MAML by first order optimality condition.
Let average MAML population risk ¯ L m as the target function in order to eliminate the randomness (Proposition 1 guarantees the upper bound between ¯ L m and population risk L m ).
Approximate this target function ¯ L m by another function L apxm which is the limit of ¯ L m as number of tasks N → ∞ .
According to positive definiteness, we can get the range of α . 5.
With notion of gamma convergence, the minimizer of ¯ L m will also converge to the minimizer of L apxm .
Our estimation of α ∗ is in the range of α in Step 4.
Notation for this proof: For simplicity, we omit the arguments of the function if its symbol has a index e.g. Φ i =
Φ( X i ) , C i
= C i ( α ) .
Then we give the full proof on estimation of α ∗ .
Proof.
Recall that the global minimum closed form for ERM and MAML are
[Equation]
[Equation]
where C i =
Φ i − 2 αK Φ i Φ ⊤ i Φ i ,
C i ∈
R K × d .
Since w m depends on random variables a 1 , ..., a N .
The average population risk of MAML, defined in (8), where
[Equation]
[Equation]
of global minimum w m in (23) can be written as
[Equation]
(cid:33) [Equation] C
[Equation]
  − a   (cid:13)(cid:13)(cid:13)(cid:13) (cid:13)(cid:13) (25)
Let Λ j = (cid:18) N (cid:80) i =1 C ⊤ i C
i (cid:19) −
C ⊤ j C j , j ∈ [1 , N ] , Λ j ∈ R d × d .
The (25) can be rewritten as,
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) =
K
E a , { a i } Ni =
∼D
( a )
E X
∼D ( x )   (cid:32) N (cid:88) i
=
Λ i
a i (cid:33) ⊤
C ( α ) ⊤ C ( α )  
N (cid:88)
j
= Λ j a j   − a ⊤ C ( α ) ⊤ C ( α )  
N (cid:88)
j
= Λ j a j   − (cid:32) N (cid:88) i
= Λ i
a i (cid:33) ⊤
C ( α ) ⊤ C ( α ) a + a ⊤ C ( α ) ⊤ C ( α ) a (cid:3) (26)
Under Assumption 1 and a is independent to X , then we have
[Equation]
Let L apxm ( α ) be an approximation function of ¯ L m ( α, N, K )
[Equation]
Then the approximation error will be
[Equation]
where
[Equation]
With Assumption
[Equation]
there exists constants 0 < c 1 < c 2 where
[Equation]
With Proposition 2 hold [Equation] ∀ i ∈
[ N ] we have
[Equation]
Then by applying multiple times of Proposition 2 we have
[Equation]
Next, we need upper bound the last inverse term.
With Assumption 2 we know that all eigenvalues of Φ( X ) ⊤ Φ( X ) , X ∼ D ( x ) are bounded by [ λ I , λ S ] .
Let C all ( α )
=
(cid:80) Ni =1 C ⊤ i C i , then with probability 1 the max/min eigenvalues of C all ( α ) will have following constraints,
[Equation]
Since the C all ( α ) is a positive matrix, we need have the constrain on λ min ( C all ( α )) > [Equation] which means
[Equation]
There exists a positive definite matrix λ s ( α, N ) I where
[Equation]
I
[Equation]
[Equation]
and the following inequality is easy to get
[Equation]
So the last inverse term will be
[Equation]
Apply these inequalities to (29), we can get the upper bound
[Equation]
When α ∈ (cid:20)
[Equation]
, the limit will go to zero, lim N →∞ sup
[Equation]
which means ¯ L m ( α, N, K ) will uniformly converge to L apxm ( α ) for α belongs to the interval above.
Note that ¯ L m ( α, N, K ) is a continuous function of α .
So with Lemma 1 , we have
[Equation]
[Equation]
So we have estimation of true α ∗ , denoted as α ∗ lim
[Equation]
According to Lemma 2 where α ∗ ( N, K ) is the minimizer of ¯ L m ( α, N, K )
[Equation]
C P ROOF OF P ROPOSITION
Proposition 4 (Formal state of Proposition 1) .
Assume u
= C ( α ) ⊤ C ( α ) a is sub-gaussian random variable with sub-gaussian norm ∥ u ( i ) ∥
Ψ 2 = sup p ≥ 1 p − 1 / 2 (cid:0) E | u ( i ) | p (cid:1) 1 /p ≤ L .
Then with probability at least 1 − δ that
[Equation]
Nlogδ , ε ( α, K )
N logδ (cid:41) (44) if α ∈ (cid:20) 0 , K ( λ 2 S − √ λ 4 S − λ 4 I ) 2 λ
I (cid:19) ∪ (cid:18) K ( λ 2 S + √ λ 4 S − λ 4 I ) 2 λ 3 I , ∞ (cid:19) and ε ( α, K {+) = ( λ S − 4 αλ 
+}
I /K + 4 α 2 λ 3 S /K 2 ) / ( λ I − 4 αλ 2 S /K
{++ 
+}
α 2 λ I /K 2 )when assumption 1 & 2 holds.
d is the feature size, N is the number of tasks and K is the sample size per task.
Here, we follow the same proof notation as Theorem 1.
Proof.
By definition, we have
[Equation]
[Equation]
Similar to (25), let Λ j = (cid:16)(cid:80)
Ni =1 C ⊤ i C i (cid:17) − 1 C ⊤ j C j , we have,
[Equation]
(cid:33) ⊤ C ( α ) ⊤ C ( α )  
N (cid:88)
j
= Λ j a j   − a ⊤ C ( α ) ⊤ C ( α )  
N (cid:88)
j
= Λ j a j   − (cid:32) N (cid:88) i
= Λ i
a i (cid:33) ⊤
C ( α ) ⊤ C ( α ) a + a ⊤ C ( α ) ⊤ C ( α ) a (cid:3) (46)
With Assumption 1, second term and third term of (46) will be cancelled.
So the L m is
[Equation]
As the comparison, ¯ L m ( α, N, K ) is the averaged function of L m , which is given by
[Equation]
Let A be the common matrix of cross-term of a i and a j Λ ⊤ i C
[Equation]
i [Equation] So cancel the their second terms, the difference of L m ( w m , α ) and ¯ L m ( α, N, K ) will be
[Equation]
m
[Equation]
w m , α, K
[Equation]
N
[Equation]
where u i =
C ⊤ i C i a i = (cid:101) C
i a i is sub-gaussian random variable and with Assumption 1,
[Equation]
Let U [ N ] = ( u 1 ; ... ; u N ) , ∈ R Nd , we write the quadratic form into a bilinear form for each product term u ⊤ i A u j
[Equation]
) where
[Equation]
53) is a N × N block matrix and ⊗ is Kronecker product.
And the relations of A and ˜ A are
[Equation]
By applying Hanson-Wright inequality we have
[Equation]
Further with Cauchy Inequality
[Equation]
we can get the following equation
[Equation]
According to Theorem 1, all eigenvalues of second term falls in [ λ I − 4 αλ 2 S /K
+
α 2 λ
I /K 2 , λ S − 4 αλ 2 I /K
+ 4 α 2 λ 3 S /K 2 ] and of order 1 /N for first term.
[Equation] [Equation]
Let ε ( α, K
[Equation]
[Equation]
Next, we can bound ∥
A ∥ HS by ∥ A ∥ . It’s obvious that ∥ A ∥ HS ≤ (cid:112) rank ( A ) ∥ A ∥ .
So ∥ A ∥ 2 HS can be upper bounded by
[Equation]
Thus, the ∥ (cid:101)
A ∥ 2 HS will no more than
[Equation]
In summary, we can get the bound
Pr
[Equation] [Equation]
Finally, we rewrite the inequality by eliminating t , we have at least 1 − δ ,
[Equation]
D P ROOF OF C OROLLARY
Proof.
Recall our estimation of α ∗ is given by,
[Equation]
[Equation]
For each task, we have K samples with d dimensional features Φ( X ) ∈ R K × d .
Since Φ( X ) ⊤ Φ( X ) is positive definite matrix, by applying spectral decomposition, we have
[Equation]
) where U is an orthogonal matrix and Σ X is the a diagonal matrix filled by eigenvalues λ 1 , ..., λ d of the covariance matrix of the feature.
It’s easy to prove in Principle Component Analysis (PCA) that tr E (Σ X ) is the variance of features where
[Equation]
where ϕ ( x i ) ∈ R
d is each row of Φ( X ) and µ is zero.
With Jensen’s inequality [Equation] we have
[Equation]
Thus [Equation] we can write the inequalities
[Equation]
thereby
[Equation]
E E XAMPLES
Example 1 (Normal, Polynomial feature) Assume we have K i.i.d samples x 1 , ..., x K ∼ N (0 , σ 2 ) and a is a random vector from zero-mean distribution.
Consider polynomial basis function ϕ : R → R d , where ϕ ( y ) =
(1 , ..., y d − 1 ) .
[Equation]
 ϕ ( x [Equation] [Equation] 
[Equation]
[Equation]
− 1...
[Equation]
[Equation]
d [Equation] K   (69) Since we have α
[Equation]
So that tr [ E X [(Φ( X ) ⊤ Φ( X )) 2 ]]
= E x d (cid:88) j = (cid:32) K (cid:88) i =
x ( j − 1)+0 i (cid:33)+ . . .
+ (cid:32) K (cid:88)
i =
x ( j − 1)+ d i (cid:33)= E x d (cid:88)
j = d (cid:88) m = (cid:32) K (cid:88) i =
x j + m −
i
(cid:33)= K d (cid:88) j =
[Equation] [Equation] [Equation] [Equation]
[Equation] [Equation] Similarly, the denominator is tr [Equation]
[Equation]
3 ]] =
[Equation]
(cid:88)
j = d (cid:88) l = (cid:34) d (cid:88) m
= (cid:32) K (cid:88) i =
x j + m −
i (cid:33) (cid:32) K (cid:88) i ′ =
x j + l − 2 i ′ (cid:33)(cid:35) (cid:34)(cid:32) K (cid:88) t = x j + l − 2 t (cid:33)(cid:35) =
E x d (cid:88)
j = d (cid:88) l =   x 2 j + l − 3+ . . .
+ x j + d − 2 i x j + l − 2 i ′ (cid:124) (cid:123)(cid:122) (cid:125)
dK
 (cid:34)(cid:32) K (cid:88) t
= x j + l − 2 t (cid:33)(cid:35) =
E x d (cid:88)
j = d (cid:88) l =   x 2 j + m + l − 4 i + . . .
+ (cid:124) (cid:123)(cid:122) (cid:125)
dK x j + d − 2 i x j + l − 2 i ′
+ . . .
(cid:124) (cid:123)(cid:122) (cid:125) dK
( K − 1) 
 (cid:34)(cid:32) K (cid:88) t
= x j + l − 2 t (cid:33)(cid:35) =
d (cid:88)
j = d (cid:88) l = d (cid:88)
[Equation]
If K = 1 , σ → 0 the optimal α ∗ lim will be
[Equation]
where C ( n, k ) = (cid:18) nk (cid:19) is the binomial coefficient and g 1 ( d, i ) =
C ( i + 1 , 1) − 2 C ( i − d + 1 , 1) .
If K → ∞ , σ →
[Equation]
We show the coefficients of each moment in the Figure 6.
As we can see, denominator becomes dominant since the coefficient of every moment, number of terms and order of moment are all larger (higher) than numerator.
So in this case, the α ∗ lim has an inverse relationship with σ 2 .
Example 2 (Random Matrices) Assume all elements Y ij in feature matrix are independent.
Then let Y be a random matrix we have
[Figure]
index i, (d=50)
[Figure]
index i, (d=50)
[Figure]
gg
[Equation]
For the numerator in (63),
[Equation] [Equation]
[Equation]
Here, the row m column n entry of (Φ( X ) ⊤ Φ( X )) 2 is
[Equation]
s = (cid:32)
[Equation]
(cid:88) i =
Y im Y is (cid:33) (cid:32) K (cid:88)
i ′ =
[Equation]
The diagonal entry of (Φ( X ) ⊤ Φ( X )) 3 at row m will given by
[Equation]
Y
[Equation]
  K (cid:88) j =
[Equation]
Y jm Y
[Equation]

 
 (78)
Similarly, the denominator is tr [Equation]
Y [Equation]
is (cid:33) [Equation]
i ′ =
Y i ′
n Y i ′ [Equation] (cid:33)  
[Equation]
Y jm Y js 
 
 =
Kd E
[ Y 6 ] + Kd ( d − 1)
( E [ Y 5 ] E [ Y ] + E
[ Y 4 ] E [ Y 2 ] + E
[ Y 3 ]) (cid:124) (cid:123)(cid:122) (cid:125)
i = i ′ =
j + b 1 + K ( K − 1) d
( d − 1)( E [ Y 3 ] E 3 [ Y ] + E
[ Y 3 ] E [ Y 2 ] E [ Y ] + E 2 [ Y 2 ] E 2 [ Y ]) (cid:124) (cid:123)(cid:122) (cid:125)
i = i ′
̸
= j + b 1 + K ( K − 1) d
( d − 1)( E [ Y 3 ] E 3 [ Y ] + E
[ Y 3 ] E [ Y 2 ] E [ Y ] + E 2 [ Y 2 ] E 2 [ Y ]) (cid:124) (cid:123)(cid:122) (cid:125) i ̸
= i ′ =
j + b 1 + K ( K − 1) d
( d − 1)( E [ Y 4 ] E 2 [ Y ] + E
[ Y 2 ] + E 2 [ Y 2 ] E
[ Y ]) (cid:124) (cid:123)(cid:122) (cid:125) i =
j ̸ = i {+′ + b 
+}
+ ( K 3 − 3 K 2 + 2 K ) d ( d − 1)( E 6 [ Y ] + E
[ Y 2 ] E 2 [ Y ] + E
[ Y 2 ] E 4 [ Y ]) (cid:124) (cid:123)(cid:122) (cid:125)
i [Equation] = i ′ ̸
=
j (79) where b 1 = K ( K − 1) d E
[ Y 4 ] E [ Y 2 ] , b 2 =
( K 3 − 3 K 2 + 2 K ) d E 3 [ Y 2 ] .
If K = 1 , the optimal α ∗ lim will be
[Equation]
If K → ∞ and d → ∞ , α ∗ lim =
[Equation]
Both two examples are related to the high order moments of data distributions.
In polynomial feature example, we focus on the gaussian distributed data and its inverse relationship to data variance.
As for any random matrix, it depends on the fourth moment over sixth moment.
F P ROPOSITION FOR T HEOREM
Proposition 5.
∃ ε , α ∈ [ − ε, ε ] global minimum of MAML w m ( α ) is given by following equation
[Equation]
where w r is ERM global minimum.
Proof.
As for the MAML, the global minimum is,
[Equation]
with C i ( α ) =
Φ i − 2 αK Φ i Φ ⊤ i Φ i ,
C i ( α ) ∈ R K × d .
Let W α
= (cid:80) Ni =1 C ⊤ i ( α ) C i and ν α denote (cid:80) Nj =1 C ⊤ j C j a j , then
[Equation]
[Equation]
Similarly, we have
[Equation]
For first-order derivative, we have the following form,
[Equation]
Recall that
[Equation]
So for small α
[Equation]
we have the Taylor expansion at zero that
[Equation]
G P ROOF OF T HEOREM
Proof Sketch We list our proof steps as follows
Based on Definition 1 , our target is to illustrate that fast adaptation distance gap between w m and w r is always negative which means MAML has smaller distance to all the tasks at any adaptation steps in expectation.
We first get the linearized expression of w m by Proposition 5 . 3.
Compute fast adaptation distance gap ∆ =
E T 1 ,...,T N ∼D ( T ) F t ( w m ) − F t ( w r ) across same task distribution D ( T ) and take expectations with respect to all random variables.
With lemma of trace inequalities and assumptions, we can get the upper bound for the dominant term of ∆ , refer to (118).
Bound the reminder terms, we can get the range of α .
Notation for this proof Follow the Theorem 1 , we omit the arguments of the function if its symbol has a index e.g. Φ T =
Φ( X T ) .
Covariance matrix Φ ⊤ T Φ T = G T and inverse of sum covariance matrix V =
( (cid:80) i ∈
[ N ] Φ ⊤ i Φ i ) − 1 for short.
Proof.
For each task T sampled from distribution D ( T ) , gradient descent iteration yields
[Equation]
where w 0 is the initialization.
Let G T =
Φ ⊤ T Φ T , the adapted error will be
[Equation] [Equation]
For simplicity, let
[Equation]
T [Equation] then with Definition 1 , we can get t -step fast adaptation error for MAML, [Equation]
[Equation]
and the ERM fast adaptation error is
[Equation] [Equation]
Note that the sum of geometric series in S T is
[Equation]
Now, let’s focus on the error gap of MAML and ERM, denoted as ∆ , then we have
[Equation]
For small α [Equation] we get its linear expansion in Proposition
[Equation]
where R 1 = (cid:82) α 0 ∇ 2 α w 0 m ( ξ ) 2!
( α − ξ ) 2 dξ is the reminder term.
So it would be
[Equation]
Let V =
(cid:16)(cid:80)
Nj G j (cid:17) − 1 and the first derivative ∇ α w 0 m (0) is split into
[Equation]
thus inner product will be four product terms and a reminder term which is ∆ =
[Equation]
where the remainder terms are
[Equation]
Now, let’s look at the expectation.
Recall that task T is defined by random variables (Φ( X T ) , a T ) .
With simultaneously diagonalizable assumption, all feature covariance matrix in tasks can be factorized to
[Equation]
where U is the basis of features, Σ T is the only random variable of G T .
So the data of each task depends on eigenvalue diagonal matrix Σ T .
So taking expectation over T means the two independent expectation E Σ T ∼D (Σ) , E a T ∼D ( a ) .
Similarly, w 0 r in previous section is expressed by
[Equation]
⇒ E
[Equation]
For each product term in (98), we list four main terms as following.
First term is
[Equation]
Similarly
[Equation]
we can get the second term
[Equation]
For third and fourth terms
[Equation]
E a T ∼D ( a ) is a marginal expectation of E T ∼D ( T ) , thus
[Equation]
Similarly, the fourth term is
[Equation]
[Equation]
So overall, the we care about above four terms as a function of α, N, t, ... denoted as δ t ( α, N ) δ t ( α, N ) =
∆ − R 2 − R {+′=
+} α
K
E { G i } [ N ] tr  
N (cid:88) i =
G i V Q tT Q tT V  
N (cid:88)
j
G 2 j  
V G i 
 − 8 α
K
E { G i } [ N ] tr (cid:34) N (cid:88) i =
G i V Q tT Q tT V G 2 i (cid:35) =8 α
K
E { G i } [ N ] tr  
N (cid:88) i =
G i V Q tT Q tT V  
N (cid:88)
j
G 2 j  
V G i −
N (cid:88) i =
G i V Q tT Q tT V G 2 i   =8 α
K
E { G i } [ N ] tr  
N (cid:88) i =
G
i V Q tT Q tT V    
N (cid:88)
j
G 2 j  
V G i − G 2 i 
 
 =8 α
K
E { G i } [ N ] tr  
N (cid:88) i =
V Q tT Q tT V    
N (cid:88) j
[Equation]
By simultaneously diagonalizable assumption, we have
[Equation]
where (cid:98) Σ N = (cid:16)(cid:80) k ∈
[ N ] Σ k (cid:17) − 1 is a PD matrix and Q tT =
( I − (2 η/K ) G T ) t is a exponential decay term w.r.t η .
With probability 1, λ s I ⪯ Σ
i ⪯ λ x I
[Equation]
[Equation]
Note that V Q tT Q tT V =
( Q tT V ) ⊤ Q tT V is a symmetric positive definite matrix where
[Equation]
Note that m -th diagonal entry of E { Σ i } [ N ] (cid:16)(cid:80)
Ni (cid:80) Nj
[Equation]
i
[Equation]
N Σ 2 j
[Equation]
⊤
[Equation]
[Equation]
⊤
[Equation] where (C-S) is according to Cauchy-Schwarz inequality for integrals in Proposition 3 .
So the above matrix is a negative definite matrix.
Now, let us can derive following trace inequality for NSD and PSD.
Proposition 6.
If A is a n -by- n negative definite matrix and B is a n -by- n PSD matrix, we have
[Equation]
Proof.
By Ruhe’s trace inequality (Lemma 4 ), we have tr ( AB ) ≥ (cid:80) i λ i
( A ) λ n − i +1 ( B ) .
Eigenvalues of A are negative where λ i ( A ) < 0 , ∀ i ∈ [ n ] .
So we have tr ( AB ) ≤ λ min ( B ) (cid:80)
i λ i
( A ) =
λ min ( B ) tr
( A )
By Proposition 6 , we have
[Equation]
with (108) and (109)
[Equation]
For the second part, we have
[Equation]
Overall, with probability 1, we have
[Equation]
Specifically,
[Equation]
With Proposition 3 we know that any eigenvalue λ = Σ ( i ) > 0 obeys the statistical condition
E 2 [ λ 2 ] − E [ λ 3 ] E [ λ ] .
Thus there exists a constant (cid:101) c > 0 such that
[Equation]
Finally
[Equation]
we have
[Equation]
Now, let’s bound the remainder terms R ′ 1 , R 2 where
[Equation]
R 1 is the remainder term in Taylor expansion (95).
With Integral Mean Value Theorem
[Equation]
We have the locally Lipschitz property for C ∞ function w m ( α ) , in a small region with small α such that
[Equation]
Then we have
[Equation]
Recall (87) the w 0 r = ( (cid:80) Ni Φ ⊤ i Φ i ) − 1 ( (cid:80) Ni ′
Φ ⊤ i ′
Φ i ′ a i ′ ) while E a [ N ] ∼D ( a ) w 0 s − a i = 0 .
The R ′ 1 in above equation is
[Equation]
And the eigenvalues of Σ i are bounded in [ λ s [Equation] λ x ] where
E
[Equation]
i
[Equation]
[Equation]
In summary,
[Equation]
Similar to R ′ 1 let’s bound the R 2 in (98),
[Equation]
Thus the final constraint of α will be
[Equation]
where ratio factor ˆ r =
[(1 − 2 η/Kλ x ) / (1 − 2 η/Kλ s )] 2 t .
We have the extreme points of α
[Equation]
For K ∈ Z + ≥ 1 , small α , t = 0 and large t we have
[Equation]
[Equation]
So finally
[Equation]
the α needs to satisfy
[Equation]
H E XPERIMENTS H. P RACTICAL FORM OF T HEOREM
For practical use, we show a numerical form to estimate α ∗ for the case where number of tasks is finite and training data u ∈ R k 1 is different from validation data t ∈ R k 2 .
The corresponding feature matrices are Φ( u ) and Φ( t ) .
Let’s derive corollary of Theorem 1 for realistic meta-learning setting.
Corollary 2.
If training feature Φ( u ) ∈ R k 1 × d is different from validation feature Φ( t )
∈
R k 2 ×
d for every task, then
[Equation] [Equation]
Proof.
Similar to proof of Theorem 1 , we have
[Equation]
where (cid:98) C i (
[Equation]
=
[Equation]
The corresponding average population risk will be
[Equation]
Then we can define similar approximation function L apxm ( α ) as (28) where
[Equation]
And with same bound for ∥ Φ( u i ) ∥ , ∥ Φ( t i ) ∥ ∈ [ c 1 , c 2 ] , we have
[Equation]
With Gamma convergence lemma, we can get the final estimation α ∗ lim in (131).
In experiments, we use the following numerical form
[Equation]
to evaluate our estimation
[Equation]
H. O VERPARAMETERIZED SETTING
Let’s consider overparameterized setting.
Thus [Equation] we have feature for each task ( K < d ), Ψ =
[ ψ ( x 1 ) , ..., ψ ( x K )] ⊤ ∈ R K × d .
Correspondingly, empirical objective of ERM is given by
[Equation]
Assume meta-initialization is the mean of all task optima that ¯ a = mean ( a 1 , ..., a N ) .
Then concatenate all task features we have
[Equation]
So MAML objective is given by
[Equation]
where w ′ = (cid:2) w − 2 α/ ( NK )Ψ all (cid:0) Ψ ⊤ all w − Ψ ⊤ all ¯ a (cid:1)(cid:3) is adapted parameters and C
all ( α ) =
Ψ all ( I − (2 α/NK )Ψ ⊤ all Ψ all ) .
The minimum norm solution is
[Equation]
[Equation] Note that, in overparameterized setting, Theorem 1 will not perfectly give the precise form for α ∗ .
But the technique can be easily extend to large d setting where
[Equation]
where C gramall =
( C all ( α ) C
all ( α ) ⊤ ) − 1 .
By Proposition
[Equation]
the ¯ L m ( α, N, K ) will be upper bounded where
[Equation]
Hence
[Equation]
minimizing the upper bound also tells us how to select α ∗ .
In another word [Equation] we are seeking an estimation α ∗ est nearly minimize the upper bound. [Equation]
[Equation]
and the C ( α ) ⊤ C ( α ) is a covariance matrix where
[Equation]
We can derive that
[Equation]
Since d is large, its computational cost is high.
Here we assume second moment of all elements of Ψ ⊤ Ψ are ˜ σ 2 , which means ˜ σ is the variance of all elements of features .
Finally we have
[Equation]
Let’s take the Neural Tangent Kernel (NTK)
(Jacot et al., 2018) for example,
[Equation]
Then we have neural tangent feature
[Equation]
Next, we stack all the features to Ψ all to compute the variance e.g. recall σ = Ψ all .
std().
After that, we can compute the estimation α ∗ est using (145).
H. E XPERIMENTAL SETUP
Estimation of α ∗ , underparameterized model We set hyperparameters dimension d = 20 , number of training/validation samples per task K = 50 , number of tasks N = 5000 .
Each x is i.i.d sampled from a distribution U ( − 5 , 5) while each a is i.i.d sampled from high dimension Gaussian distribution
N ( 0 , 3 I ) .
Then computing the Ordinary Least Square (OLS) solution with different α , we can show the training loss landscape in terms of α .
The true α ∗ ( N ) =
arg min α min w L ( α, 5000 , 50 , w ) is the minimizer of the training loss.
Our estimation α ∗ lim described in Theorem 1 is evaluated by comparing the error to true α ∗ .
Estimation of α ∗ , overparameterized model We perform the nonlinear regression on two different models.
The first is quadratic regression using neural tangent feature (see H.2).
All hyperparameters are set to be same as (Bernacchia, 2021)
where
[Equation]
But to guarantee the overparameterization, we set hidden size with 10 , 000 , which means the total dimension will be 30 , 001 .
Then we perform quadratic function regression with ranging α value to see the test loss.
After that, we can evaluate the how accurate our estimation using (145) is by comparing to optimum of test loss.
Second experiment is sine function regression using 3-layer MLPs activated with ReLU.
The data and labels are generated from a stochastic function
[Equation]
To get a good representation, we pre-train the model with ERM loss and then freeze the first two layers as the feature extractor.
Then we use the output from second layer as the random feature to train last layer on 1 , 000 training tasks.
At the same time, α ∗ lim can be computed from the features of 1 , 000 training tasks.
Then last layer trained with differnet α will be evaluated on 10 , 000 test tasks.
Fast adaptation distance We run experiments with random matrices.
For each task, the data are i.i.d drawn for the prescribed distributions which represent three common types, Uniform
U ( − 5 , 5) , Gaussian N (0 , 2) and Exponential Exp (1) .
Specifically, each entry in random matrix
X ∈ R K × d , ( K = 50 , d = 20) is sampled variable from a distribution X ( ij ) ∼ D ( x ) .
The feature map is an identity map Φ( X ) =
X .
First we sample 5000 training tasks to compute the closed-form meta-initializations for ERM and MAML with a small α ( 10 − 4 ).
Then we perform t -step adaptation on 5000 test tasks and compare the fitting losses and the Average Distance under Fast Adaptation
F t ( w m ) , F t ( w r ) .
The learning rate η in fast adaptation evaluation is 10 − 5 .
Estimation of α ∗ , deep learning In our experiments, we valid our estimation of α ∗ on sine regression and few-shot classification. • For deep regression, we follow the (Finn et al., 2017) to perform sine regression with 3-layer MLP whose hidden size is 40 . Then each task is an instance in stochastic function y =
a sin( x + b ) , a, b ∼ U (0 , π ) while the training set is 10 i.i.d sampled data pair from the corresponding sine function and test set is consists of another sampled 256 points.
During test, we sample 10 , 000 tasks to evaluate the learned model.
• For deep classification, we follow the Omniglot experiments in (Raghu et al., 2020).
Here, we adopt the online estimation scheme to compute the α ∗ for the adaptation learning rate of last layer.
To this end, we apply α ∗ = 1 / (2 × N way × N shot × ˆ σ 2 ) where ˆ σ 2 is mean of covariance of the normalized feature F ⊤ F , F = ( F 1 / ∥ F 1 ∥ , ...,
F N w ay / ∥
F N w ay ∥ ) . Then α ∗ buffer in the buffer is online updated using α ∗ buffer ← 0 . α ∗ buffer + 0 . 1 α ∗ .
H. A DDITIONAL R ESULT H.4. O PTIMIZATION BEHAVIOR We add more illustrative experiments on visualizing the trajectory of global minima of MAML.
We consider normally distributed task optima with centralized data and uncentralized data.
As shown in 7, the MAML minimum still try to balance the distances to different task optima.
But the situation is more complex in uncentralized data (second row).
However, α always minimize the geometric distance at beginning where the shape of mean distance function appears to be convex.
This has confirmed our Theorem 2 where small α always lead to a shorter mean distance to different task optima than ERM algorithm. [Table]
E STIMATION OF α ∗ ON B ASIS FUNCTION FEATURE
Firstly, we used the random matrix for task i as the feature matrix, Φ i ∈ R K × d . All elements of Φ i are i.i.d sampled from U ( − 5 , 5) . Secondly, we created the random features using Gaussian basis function.
Gaussian basis function Φ( X ) ( ij ) =
exp( − (cid:0) X ( ij ) − µ j (cid:1) 2 / 2 σ 2 i ) is a function whose value depends only on the distance between the input and some fixed point.
Thirdly, we used polynomial feature Φ( X ) ( i, :) =
( c 0 , · · · , c n x d − 1 ( i ) ) which is based on Taylor series.
With N tasks, we compute the one-step adaptation loss.
Optimal learning rate minimizing the loss is denoted by α ∗ ( N ) .
The error gap between true optimum and estimation | α ∗ ( N ) − α ∗ lim | with three random features are shown in Figure 8 (a), (b) & (c) respectively.
To reduce random errors, there was an average of 10 sampling trials, shown as the solid lines.
The shadow area represents standard deviation.
As the number of tasks N increasing, the estimation error will shrink down to zero and its uncertainty reduces as well.
So our estimation is reliable and accurate when number of tasks becomes large.
[Figure]
We use Gaussian basis function as the random feature and conduct the experiments on different types of distribution to evaluate our estimation quality.
Then, we use uniform/normal distribution with zero mean U , N as the stereotype of central symmetric distribution.
In experiments, we set d = 10 , K = 15 , N = 3000 and the parameters in Gaussian function depends on the range of data.
As shown in the Figure 9, the estimation α ∗ lim is close to true optimum, α ∗ in four different cases: (a) data is sampled from a central uniform distribution U ( − 5 , 5) , task optima are sampled from a central normal distribution N ( 0 , 3 2 I ) ; (b) data is sampled from a non-central normal distribution N (0 , 2 2 ) , task optima are sampled from a central normal distribution N ( 0 , 3 2 I ) ; (c) data is sampled from a central uniform distribution U ( 5 , 5) , task optima are sampled from a non-central normal distribution N ( 5 , I data is sampled from a non-central Chi-Sqaure distribution χ , task are sampled from a imbalanced Zipf x = k ) =
1 ζ ( s ) k − s where ζ ( s ) is the
Riemann Zeta function. Note that results in (c) and (d) are beyond our Assumption 1. So our theorem can be extended to more general scenarios. H.4. E STIMATION OF α ∗ ON N EURAL N ETWORK FEATURE
We used neural network based feature to verify our theorem in underparameterized (original model size in Finn et al.
and overparameterized setting ( NK < d ).
In former setting, we used 3-layer Multilayer Perceptron (MLP) activated with ReLU for sine functions family regression where each task is to regress an instance in stochastic function y = a sin( x + b ) , a, b ∼ U (0 , π ) .
We used
ERM to train and freeze the first 2 layers (as feature extractor) and then only fine-tune last layer with MAML.
Compute α ∗ lim through features of sampled training tasks we got α ∗ lim = 0 .
As shown in Table.
2, the optimal α of lowest MSE after adapting is 0 .
1 which is nearest discrete value in table to α ∗ lim .
H.4.
H EURISTIC ESTIMATION RANGE FOR DEEP L EARNING
To make it practical for deep learning, we give the heuristic estimation range where α ∗ it might be based on our theorem.
Previously, we show (9) for underparameterized model ( K > d ) and (145) for overparameterized model ( NK ≪ d ).
Besides, the trace term for covariance matrix in underparameterized setting can be simplied by Kd ˜ σ 2 where ˜ σ 2 is the covariance of the feature (second moment).
So here, the heuristic estimation by merging these two settings, where it derived as
[Equation]
where N, K are number of training task and its training sample size, d is model size.
Next, we show a simple way to estimate the range of ˜ σ 2 .
In general, each element of feature with probabilitywill fall into the [0 , 1] .
Then, given ˜ n observations, we have (Popoviciu’s inequality on covariance (Sharma et al., 2010))
[Equation]
Proof.
Assume with probability 1, each element x of Φ( X ) has x ∈ [ m, M ] .
Define a function f in terms of random variable x by
[Equation]
Computing the derivative f ′ , and solving the minimum
[Equation]
So we have the covariance has following upper bound
[Equation]
where
[Equation]
Thus for m = 0 , M = 1 , we have
[Equation]
for an independent sample of ˜ n observations from a bounded probability distribution, the von
Szokefalvi Nagy inequality shows that
[Equation]
Here, we conduct following deep regression experiments to evaluate our heuristic estimation.
We plot the estimated range of α ∗ given by our bounds – the red area between the two star-lines in Figure 10.
Then we perform quadratic regression with 2-layer neural network follow the hyperparameter in (Bernacchia, 2021).
From Figure 10(a), we can see, the optimal α for this task is positive and our estimated range includes suboptimal points.
Follow the setting of (Finn et al., 2017)
(All hyperparameters are same), we use 3-layer NN with hidden size 40 to test sine regression tasks.
As shown in the Figure 10(b), our estimated range includes the optimal α and other good α .
[Figure]
H. R ELATION TO NEGATIVE LEARNING RATE
As we mentioned before, (Bernacchia, 2021) show negative learning rate minimizing the test loss of
MAML.
In this section, we compare their results with ours.
Specifically, we follow the setting of underparameterized experiment (Bernacchia, 2021) where the defined hyperparameters are set to be same, n t = 5 , n v = 25 , n r = 10 , m = 40 , p = 30 , σ = 0 .
2 , ν = 0 .
Parameters are sampled from following distributions
[Equation]
We conduct experiments on numerical fitting loss on meta-learning instead of the closed-forms ¯ L test in theorems (Bernacchia, 2021) 2 .
As we can see from the Figure 11, (Bernacchia, 2021) only give the result on special case where α r = 0 . 2 is fixed.
However, this strategy highly depends on the selection of α r that may not achieve the minimum of meta-learning loss.
In addition, we run deep learning experiments to demonstrate that optimal learning rate α is positive.
All hyperparameters and generating process are set to be same as the non-linear regression experiments in (Bernacchia, 2021).
Furthermore, we train the 2-layer neural network to regress quadratic functions with 5 adaptation steps and evaluate models on same 10 folds with each fold consists of 1000 test tasks.
The results (with error bar) are shown in the Figure 12.
As we can see, the optimal learning rate for both strategies are positive.