Understanding Benign Overﬁtting in Nested Meta Learning
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Meta learning has demonstrated tremendous success in few-shot learning with limited supervised data.
In those settings, the meta model is usually overparameterized.
While the conventional statistical learning theory suggests that overparameterizedmodels tend to overﬁt, empirical evidence reveals that overparameterized metalearning methods still work well – a phenomenon often called “benign overﬁtting.”
In an attempt to understand this phenomenon, we focus on the meta learning settings with a challenging nested structure that we term the nested meta learning andanalyze its generalization performance under a overparameterized meta learningmodel.
While our analysis uses the relatively tractable linear models, our theorycontributes to understanding the delicate interplay among data heterogeneity, modeladaptation and benign overﬁtting in nested meta learning tasks.
We corroborateour theoretical claims through numerical simulations.
Introduction
Meta learning, also referred to as “learning to learn”, usually learns a prior model from multipletasks so that the learned model is able to quickly adapt to unseen tasks.
Meta learning has beensuccessfully applied to few-shot learning and multi-task learning [2, 11, 19, 24, 35].
While thereare many exciting meta learning methods today, in this paper, we will study a representative metalearning setting where the goal is to learn a shared initial model that can quickly adapt to task-speciﬁcmodels.
This adaptation may take an explicit form such as the output of one gradient descent step,which is referred to as the model agnostic meta learning (MAML) method [19].
Alternatively, theadaptation step may take an implicit form such as the solution of another optimization problem,which is referred to as the implicit MAML (iMAML) method [31].
Since both MAML and iMAMLwill solve a nested optimization problem, we term them the nested meta learning thereafter.
In manycases, overparameterized models are used as the initial models in nested meta learning for quickadaptation.
However, training such initial models is often difﬁcult in meta learning because thenumber of training data is much smaller than the dimension of the model parameter.
Previous works on nested meta learning mainly focus on addressing the optimization challengesor analyzing the generalization performance with sufﬁcient data [12,16,17].
Different from theseworks, we are particularly interested in the generalization performance of the sought initial model inpractical scenarios where the total number of data from all tasks is smaller than the dimension of theinitial model, which we term overparameterized meta learning .
In those overparameterized regimes,the generalization error of meta learning models is not fully understood.
Motivated by this, we ask:
Whether overparameterized nested meta learning models would lead to overﬁtting or not?
In this paper, we take an initial step by answering this question in the meta linear regression setting
[Table]
1.1 Prior art
[Table]
We review prior art that we group in the following two categories.
Benign overﬁtting analysis.
The empirical success of overparameterized deep neural networks hasinspired theoretical studies of overparameterized learning.
The most closest line of work is benignoverﬁtting in linear regression [5], which provides excess risk that measures the difference betweenexpected population risk of the empirical solution and the optimal population risk.
Analysis ofoverparameterized linear regression model with the minimum norm solution.
It concludes that certaindata covariance matrices lead to benign overﬁtting, explaining why overparameterized models thatperfectly ﬁt the noisy training data can work well during testing.
The analysis has been extended toridge regression [38], multi-class classiﬁcation [42], and adversarial learning with linear models [8].
While previous theoretical efforts on benign overﬁtting largely focused on linear models, mostrecently, the analysis of benign overﬁtting has been extended to two-layer neural networks [7,20,26].
However, existing works mainly study benign overﬁtting for single level empirical risk minimizationproblems, rather than bilevel problems such as nested meta learning, which is the focus of this work.
Meta learning.
Early works of meta learning build black-box recurrent models that can make predictions based on a few examples from new tasks [2,11,24,35], or learn shared feature representationamong multiple tasks [36,40].
More recently, meta learning approaches aim to ﬁnd the initializationof model parameters that can quickly adapt to new tasks with a few number of optimization steps suchas MAML [19,30,33].
The empirical success of meta learning has also stimulated recent interestson building the theoretical foundation of meta learning methods.
Generalization of meta learning.
The excess risk , as a metric of generalization ability of nestedmeta learning has been analyzed recently [3,4,9,13,17,41].
Generalization performance has also beenstudied in a relevant but different setting - representation based meta learning [12,15].
Informationtheoretical generalization bounds have been proposed in [10,25], which bound the generalization errorin terms of mutual information between the input training data and the output of the meta-learningalgorithms.
The PAC-Bayes framework has been extended to meta learning to provide a PAC-Bayesmeta-population risk bound [1,14,18,32].
These works mostly focus on the case where the metalearning model is underparameterized; that is, the total number of meta training data from all tasksis larger than the dimension of the model parameter.
Recently, overparameterized meta learninghas attracted much attention.
Bernacchia [6] suggests that in overparameterized MAML, negativelearning rate in the inner loop is optimal during meta training for linear models with Gaussian data.
Sun et al.
[37] shows that the optimal representation in representation-based meta linear regression isoverparameterized and provides sample complexity bounds for the method of moment estimator.
Compared to the most relevant works, our work is different in the following aspects.
Compared to theworks that also analyze generalization error or sample complexity in linear meta learning models suchas [3,9,13], we focus on the overparameterized case when the total number of training data is smallerthan the dimension of the model parameter.
Compared to the work that focus on representation-basedmeta learning with a bilinear structure [37], we consider initialization-based meta learning methodswith a nested structure such as MAML and iMAML.
Furthermore, we provide tight analysis of theexcess risk with explicit consideration of the benign overﬁtting condition.
A summary of key differences compared to prior art is provided in Table 1.
We distinguish two different overparameterization settings: i) the per-task level overparameterization where the dimension ofmodel parameter is larger than the number of training data per task, but smaller than the total numberof data across all tasks; and, ii) the meta level overparameterization where the dimension of modelparameter is larger than the total number of training data from all tasks.
1.2 This work
This paper provides a unifying analysis of the generalization performance for nested meta learningproblems with overparameterized meta models.
To our best knowledge, this is the ﬁrst work thatprovides the condition for benign overﬁtting in nested meta learning including MAML and iMAML.
Technical challenges.
Before we introduce the key result of our paper, we ﬁrst highlight thechallenges of analyzing the generalization of nested meta learning and characterizing its benignoverﬁtting condition, compared to the non-nested setting such as in [5,37,38].
T1)
Due to the bilevel structure of nested meta learning, the solution to the meta training objective involves high order terms of data covariance. As a
result, the dominating term in the excess riskpropagated from the label noise contains higher order terms, which is harder to quantify and
can potentially lead to
orders of magnitude higher excess risk
than the
linear regression case [5,37, T2) The existing analysis of benign overﬁtting [5, 38] critically relies on the monotonicity of theeigenvalues of the solution matrix on the eigenvalues of the data matrix.
However, due to the structureof nested meta learning and thus the solution matrix, this monotonicity is no longer true in general.
T3)
Due to the multi-task learning nature of meta learning, the excess risk of MAML depends on theheterogeneity across different tasks in terms of both the task data covariance and the ground truth taskparameter.
As a result, the data covariance matrices from different tasks have different eigenvectors.
This is in contrast to the linear regression case where all the data follow the same distribution.
In view of these challenges, our contributions can be summarized as follows.
C1)
We derive the upper bound of the excess risk for overparameterized nested meta learningincluding MAML and iMAML, with their minimum norm solution.
Speciﬁcally, the excessrisk upper bound adopts the following formCross-task variance + Per-task variance
+ Bias where the cross-task variance quantiﬁes the error caused by ﬁnite task number and thevariation of the ground truth task speciﬁc parameter, which is a unique term compared to
single task learning.
The bias quantiﬁes the bias resulting from the minimum norm solution.
And the per-task variance quantiﬁes the error caused by noise in the training data.
C2)
We compare the benign overﬁtting condition for the overparameterized nested meta learningmodels and that for the empirical risk minimization (ERM) which learns a single sharedparameter for all tasks, and show that overﬁtting is more likely to happen in MAML and itsvariants such as implicit MAML than in ERM.
In addition, larger data heterogeneity acrosstasks will make overﬁtting more likely to happen.
C3)
We discuss the choice of hyperparameter, e
g., the step size in MAML and the weight of
the regularizer in iMAML, such that if the data leads to benign overﬁtting in ERM, it also
leads to benign overﬁtting in MAML and iMAML.
We show that a negative step size can
preserve benign overﬁtting in MAML.
This is complementary to the recent discovery that
the optimal step size of overparameterized MAML during training is negative [6].
Problem Formulation and Methods
In this section, we will introduce the problem setup and the considered nested meta learning methods.
Problem setup.
In the meta-learning setting, assume task m is drawn from a task distribution, i.e.m ⇠ M .
For each task m , we observe N samples with input feature x m 2 X m ⇢ R d and target 119 label y m 2 Y m ⇢ R drawn i.i.d.
from a task-speciﬁc data distribution P m .
These samples are 120 collected in the dataset D m = { ( x m,n , y m,n ) } Nn =1 , which is divided into the train and validation 121 datasets, denoted as D tr m and D va m .
And |D tr m | = N tr and |D va m | = N va with N = N tr + N va .
We use 122 the empirical loss ` m ( ✓ m , D m ) of per-task parameter ✓ m 2 ⇥ m as a measure of the performance.
Inthis paper, we consider regression problems, where ` m is deﬁned as the mean squared error.
The goal for nested meta learning methods, such as MAML [19] and iMAML [31], is to learn aninitial parameter ✓ 0 2 ⇥ 0 , which, with an adaptation method A : ⇥ 0 ⇥ ( X m ⇥ Y m ) N tr ! ⇥ m , can 126 generate a per-task parameter ✓ m that performs well on the validation data for task m .
Given M tasks, our meta-learning objective is computed as the average of the per-task objective, given by
Meta training objective
[Equation]
Obtaining the empirical solution ˆ ✓ A 0 by minimizing (1) under a meta learning method A , in the metatesting stage, we evaluate ˆ ✓ A 0 on the population risk, given by Meta testing objective
[Equation]
Methods.
We focus on understanding the generalization performance of two representative nested meta learning methods MAML [19] and iMAML [31] in the overparameterized regime.
MAML obtains the task-speciﬁc parameter ˆ ✓ m ( ✓ 0 ) by takingone step gradient descent with step size ↵ of the per-task lossfunction ` m from the initial parameter ✓ 0 , that is
[Equation]
On the other hand, iMAML obtains the task-speciﬁc parameterˆ ✓ m from the initial parameter ✓ 0 by optimizing the task-speciﬁcloss regularized by the distance between ˆ ✓ m and ✓ 0 , that is
[Equation]
where  > 0 is the weight of the regularizer.
As summarizedin Figure 1, MAML has smaller computation complexity thaniMAML since iMAML requires solving an inner optimizationproblem during adaptation, while iMAML can achieve smaller test error on each task since it explicitlyminimize the regularized per-task loss.
Main Results: Benign Overﬁtting for Nested Meta Learning
In this section, we introduce the data model and some necessary assumptions for the analysis.
Wepresent the main results, highlight the key steps of the proof and conduct simulations to verify ourresults.
Due to space limitations, we will defer the complete proofs to the supplementary document.
3.1 Model and assumptions
To make a precise analysis, we will assume the following linear data model.
Denoting the groundtruth parameter on task m as ✓ ? m 2 R d , and the noise as ✏ m , we assume the data model for task m is
[Equation]
Given the linear model (5), the meta training problem (1) with adaptation method (3) or (4) generallyhave unique solutions when d  NM .
However, when the meta model ✓ 0 and thus the per-taskmodel ✓ m are overparameterized, i.e. d > NM , the training problem (1) may have multiple solutions.
In the subsequent analysis, we will analyze the performance of the minimum norm solution becauserecent advances in training overparameterized models reveal that gradient descent-based methodsconverge to the minimum norm solution [22,27].
We provide a formal deﬁnition below.
Deﬁnition 1 (Minimum norm solution) .
Denote X va m :=
[ x m, 1 , . . .
, x m,N va ] > 2 R N va ⇥ d , y va m :=[ y m, 1 , . . .
, y m,N va ] > 2 R N va .
With A ( ✓ , D tr m ) being either (3) or (4) , the minimum norm solution tothe meta training problem (1) under the linear regression loss is expressed by
[Equation]
In our analysis, we make the following basic assumptions.
Assumption 1 (Overparameterized model) .
The total number of meta training data is smaller thanthe dimension of the model parameter; i.e. NM < d .
Assumption 2 (SubGaussian data) .
The noise ✏ m is subGaussian with E [ ✏ m ] = 0 and E
[ ✏ 2 m ] =  2 .
For the m -th task, data x m = V m ⇤ 12 m z m , where z m has independent,  x -subGaussian entries;
E [ z m ] = 0 , E [ z m z > m ] = I d , with I d being a d ⇥ d identity matrix.
Assumption 3 (Data matrix) .
Deﬁne Q m := E [ x m x > m ] , Q :=
E m [ Q m ] =
V ⇤ V > with orthonormalmatrices V and V m .
Denote the i -th eigenvalue of Q and Q m in descending order as  i and  m,i ,respectively.
Assume
<
  m,i  ¯  with  and ¯  being the smallest and the largest eigenvalues.
Cross-task data heterogeneity V ( { Q m } Mm =1 ) :
= max i,m |  i   m,i | is bounded.
Assumption 4 (Ground truth task parameter) .
The ground truth parameter ✓ ? m is independent of X mand satisﬁes Cov[ ✓ ? m ]
= ( R 2 /d )
I d , where R is a constant, and the individual entries of ✓ are i.i.d.and O
( R/ p d ) -subGaussian.
Assumption 1 deﬁnes the setting that the meta level is overparameterized, which has also been usedin [37].
Note that Assumptions 2-4 are common in the analysis of meta learning, and also usedin [3,9,13,21].
With the linear data model (5), the (minimum norm) solutions to the meta training objective (1) andthe meta testing objective (2) can be computed analytically which we will summarize next.
Proposition 1.
(Empirical and population level solutions)
Under the data model (5) , the metatesting objective of method A in (2) can be equivalently written as
[Equation]
where the matrix W A m and its empirical version ˆ W A m are given in Table 2 with ˆ Q al m :=
1 N X al > m X al m .
The optimal solutions to the meta-test risk and the minimum norm solutions to the empirical metatraining loss are given below respectively
[Equation]
where † denotes the Moore–Penrose pseudo inverse;  A M is a constant that depends on X m , ✏ mspeciﬁed in the supplementary document.
[Table]
[Equation]
To study overﬁtting in the meta learningmodel, we quantify its generalization ability via the widely used metric - excess risk .
The excess risk of method A (which can be“ ma ” for MAML and “ im ” for iMAML),with an empirical solution ˆ ✓ A 0 and population solution ✓ A 0 , is deﬁned as
[Equation]
In (9), the excess risk measures the difference between the population risk of the empirical solution, ˆ ✓ 0 and the optimal population risk.
Giventotal number of training samples MN , if d ! 1 , the classic learning theory implies that the excessrisk E A (ˆ ✓ A 0 ) also grows, which leads to overﬁtting [23].
The larger the excess risk, the further theempirical solution ˆ ✓ A 0 is from the optimal population solution ✓ A 0 , indicating more severe overﬁtting .
3.2 Main results
With the closed-form solutions given in Proposition 1, we are ready to bound the excess risk of MAML and iMAML in the overparameterized regime.
For notation brevity, we ﬁrst introduce someuniversal constants such as c 0 , c 1 , c 2 , . . .
, and only present the dominating terms in the subsequentresults.
The precise presentation of non-dominant terms are deferred to the supplementary document.
We ﬁrst decompose the excess risk into three terms in Proposition 2.
Proposition 2.
Denote W A :=
E m [ W A m ] .
There exists a universal constant c 0 > 0 that the excessrisk of a meta learning method A can be bounded by
[Equation]
where the ﬁrst term E ✓ ?
m is a function of ✓ ? m , ✓ A 0 , W A , ˆ W A m , which quantiﬁes the weighted varianceof the ground truth task speciﬁc parameters ✓ ? m ; the second term E ✏ m , as a function of ✏ m , is theweighted noise variance; and the third term E b , as a function of ✓ A 0 , W A , ˆ W A m , is the bias of theminimum norm solution in overparameterized MAML or iMAML.
Based on this decomposition, as we will show in Section 4, the bound of the excess risk can bederived from the bound of these three terms E ✓ ? m , E ✏ ? m , E b , respectively, which gives Theorem 1.
Theorem 1 (Excess risk bound) .
Suppose Assumptions 1-4 hold.
Let µ 1 ( · )  µ 2 ( · ) . . .
denote theeigenvalues of a matrix in the descending order.
Deﬁne W A M := 1 M P
Mm [Equation] A m .
For the meta linear 213 regression problem with the minimum norm solution (6) , for k  d , deﬁne the effective ranks as
[Equation]
With the cross-task data heterogeneity V deﬁned in Assumption 3, if there exist universal constants c 1 , c 2 , c 3 > 1 such that the effective dimension k ⇤ =
min { k  0 : r k ( W A M )  c 1 NM } , 216 c 2 log(1 /  ) < NM and k ⇤ < NM/c 3 , then with probability at least 1   , the excess risk satisﬁes
[Equation]
Theorem 1 provides the excess risk bound via the effective ranks.
In (11), the effective ranks r k and
R k of a matrix capture the distribution of the eigenvalues of this matrix, and the effective dimensionk ⇤ determines the above upper bound by considering the asymmetry of the eigenvalues of the solutionmatrix.
The idea is to choose k ⇤ that makes R k ⇤ large enough and keeps k ⇤ small enough compared to
MN so that the variance term of the excess risk is controlled.
For example, r 0 is the trace normalizedby the largest eigenvalue, which is bounded above by R 0 .
And both r 0 and R 0 are no larger than therank of the matrix, and they are equal to the rank only when all non-zero eigenvalues are equal.
If theeigenvalues distribute more uniformly, the effective rank will be larger, otherwise smaller.
Remark 1. 1)
The deﬁnition of effective rank has been also given in [5] but only on the data matrix
Q .
And our setting reduces to the single task ERM learning, or the linear regression case in [5], when
M = 1 , ✓ ? m = ✓ 0 , W A m = Q , which implies that the cross-task variance in (10) as well as the dataheterogeneity V ( · ) reduces to zero.
Accordingly, Theorem 1 reduces to Theorem 4 in [5].2)
Given Theorem 1, in order to control the excess risk of solution ˆ ✓ A 0 , we want r {+
+} ( W A m ) to be smallcompared to the total number of training samples MN , but r k ⇤ ( W A m ) and R k ⇤ W A m ) to be largecompared to MN .
In addition, the cross-task heterogeneity V should be small.
Since for a matrix
W , r k ( W )  R k ( W )  d , this suggests the model beneﬁts from overparameterization.
Building upon Theorem 1, we now discuss the conditions for “benign overﬁtting”, which refers to thesituation that overparameterization does not “harm” the excess risk, or the excess risk still vanisheswhen d > MN and N, M, d increase.
Deﬁnition 2 (Condition for benign overﬁtting in nested meta learning) .
The weight matrices W A M for method A satisfy the benign overﬁtting condition in nested meta learning, if and only if
[Equation]
This guarantees the excess risk (12) goes to zero in overparameterized meta learning models withsufﬁcient training data from all tasks.
To provide an intuitive explanation, Figure 2 plots the population
[Table]
[Table]
[Figure] [Figure]
When d/ ( NM ) > 1 , the model is overparameterized, which can overﬁt the training data, leading tolarger excess risk as N decreases.
However, Figure 2 shows the excess risk does not become too largeas N decreases, indicating that overﬁtting does not severely harm the population risk in this case.
3.3 Examples and discussion
In this section, we discuss how the benign overﬁtting condition (13) in nested meta learning reducesto that in single task linear regression; e.g., in [5,38].
We also provide examples to show
i) how certain properties of meta training data affect the excess risk; and,ii) how to choose the hyperparameters that preserve benign overﬁtting.
Data covariance and cross-task heterogeneity.
Theorem 1 reveals that the excess risk dependson both the eigenvalues of the data covariance matrix Q m , and the cross-task data heterogeneity,measured by V ( { Q m } Mm =1 ) .
We give an example below to better demonstrate how these twoproperties of nested meta training data affect the excess risk.
Example 1 (Data covariance) .
Suppose Q m =
diag
( I d 1 ,  I d  d 1 ) , 8 m .
Set M = 10 , d = 200 , d 1 = 255 20 , ↵ = 0 .
1 for MAML and  = 10 3 for iMAML.
Then the benign overﬁtting condition (13) is satisﬁedby MAML and iMAML.
We plot the excess risk under different  in Figure 3.
From Figure 3 we can observe that given a ﬁxed number of training data N , the population riskincreases with  for both MAML and iMAML.
This observation veriﬁes our theory since larger results in a smaller R A k ( W A M ) , leading to a larger upper bound on the variance term in (12).
Example 1 demonstrates how the per-task data matrix Q m affects the excess risk.
We consideranother example that demonstrates how the data heterogeneity across tasks affects the excess risk.
Example 2 (Data heterogeneity) .
Suppose Q m = | ! m
+ 1 | diag( I d 1 ,  I d  d 1 ) with ! m ⇠ N (0 ,  2 ! ) 263 for all m .
Set M = 10 , d = 200 , d 1 = 20 ,  = 0 . 3 , ↵ = 0 .
1 for MAML and  = 0 .
1 for iMAML.
Then it satisﬁes the benign overﬁtting condition (13) for MAML and iMAML.
Figure 4 plots the excessrisk with different choices of  ! .
Observing from Figure 4 that the larger  2 ! , the higher the excess risk, and the more difﬁcult for thebenign overﬁtting condition to be satisﬁed for both MAML and iMAML.
Therefore, compared to
[Table]
[Figure]
ERM with a single task, the benign overﬁtting condition for MAML is more restrictive as it imposesconstraints for both the expected data covariance Q m , and the data heterogeneity V ( { W A m } Mm =1 ) .
Connection to multi-task ERM.
To compare benign overﬁtting in the nested meta learning withthat in the conventional ERM, where ✓ m = ✓ 0 , we can set the step size ↵ = 0 in MAML, or  ! 1in iMAML, and N va = N , which reduces to conventional ERM without adaptation.
Compared to that of MAML and iMAML in (13), the benign overﬁtting condition is less restrictivefor ERM since it does not impose constraints on ↵ or  .
Intuitively, benign overﬁtting is morelikely to happen in MAML or iMAML than in ERM.
The hyperparameters ↵ and  will affect theeigenvalues of W ma m , W im m , respectively, thus affecting their corresponding excess risk.
Here weprovide a sufﬁcient condition where the benign overﬁtting condition in ERM is preserved in MAMLor iMAML.
We summarize the results in the corollary below.
Corollary 1 (Hyperparameters that preserve benign overﬁtting) .
Recall  1 is the largest eigenvalueof Q .
For MAML, when 0 < ↵  13  1 , and for iMAML, when    1 , then the effective ranks of
W ma and W im are bounded above and below by a positive constant times the effective rank of Q ,and therefore the benign overﬁtting condition holds for MAML and iMAML if it holds for ERM.
Tosummarize, there are constants c 1 , c 2 , c 3 , c such that for k ⇤ =
min { k  0 : r k ( Q M )  c 1 NM } .
For  < 1 , c 2 log(1 /  ) < NM and k ⇤ < NM/c 3 , with probability at least 1  7 e  2 NM/c , it follows
[Equation]
Remark 2.
For MAML, let the unordered eigenvalues ˜ µ i ( W ma ) =
i (1  ↵ i ) 2 .
One challengeto control ˜ µ i ( W ma ) is that ˜ µ i ( W ma ) are not necessarily monotonic w.r.t.
i ; that is, it does notnecessarily hold that ˜ µ 1  ˜ µ 2  · · ·  ˜ µ d .
For any  i   j , if ˜ µ i ( W A )  ˜ µ j ( W A ) , then we say 288 the order of the eigenvalues is preserved.
For this to hold, it requires ˜ µ i (  i ) to be a monotonicallynon-decreasing function of  i , which yields ↵  13  1 .
Similar results can be obtained for iMAML bycontrolling the value of  .
And the bound on ↵ or  further ensures that ˜ µ i ( W A ) is bounded aboveand below by a positive constant times the effective rank of Q .
Proof Outline
In this section, we highlight the key steps of the proof for Theorem 1.
We achieve so by analyzingthe three terms in Proposition 2 respectively.
The ﬁrst two terms in (10) can be bounded based on the concentration inequalities on subGaussianvariables, given in Lemmas 1 and 2.
Lemma 1 (Bound on cross-task variance) .
With probability at least 1   , it follows
[Equation]
(15) where e O ( · ) hides the log polynomial dependence on N, M, d . [Table]
[Figure]
The cross-task variance term analzyed in Lemma 1 is unique in meta learning, which captures the data heterogeneity across different tasks.
To elaborate Lemma 1, we plot the cross-task variance versus the task number in Figure 5a with the dimension of model parameter d = 200 , training validationsplit parameter s = N tr /N = 0 .
5 , per-task data number N = 10 .
This ﬁgure demonstrates that thecross-task variance decreases with number of tasks M with rate O ( 1 M ) , which is consistent with thetheoretical results in Lemma 1.
Interestingly, we can observe a double descent phenomenon when Mincreases so that the model goes from overparameterized setting to underparameterized setting.
Notethat the “double descent” phenomenon in Figure 5a is different from that in the conventional linearregression case in that increasing the number of tasks M will increase data from different per-taskdistributions, but they still help reduce the excess risk.
Lemma 2 (Bound on bias) .
There is a constant c 3 that depends only on  x , such that for any [Figure] 1 < log(1 /  ) <
MN va , with probability at least 1   , we have
[Equation]
This term is similar to the bias term in the linear regression case, but directly depending on thesolution matrix W instead of the data matrix Q .
To elaborate Lemma 2, Figure 5b demonstrates thatthe bias term decays with M until it reaches zero when the model is underparameterized.
These twoterms in Lemma 1 and Lemma 2 do not go to inﬁnity as N, M, d increase.
Note that, the key step is the bound on E ✏ m , which is the dominating term in the decomposition ofexcess risk (10) in the overparameterized regime.
We will bound it below.
Lemma 3 (Bound on per-task variance) .
There exist constants c 1 , c 2 , c 3 , c such that for 0  k 2 NM/c 3 , r k ( W A M )  c 1 NM , and k 0  k , with probability at least 1  7 e  2 NM/c , it follows
[Equation]
[Equation]
Note that, in the single task linear regression case, the cross-task data heterogeneity becomes none,i.e., V = 0 .
This term is unique in the meta learning setting with multiple tasks.
Plugging the resultsof Lemmas 1, 2 and 3 into (10), we will reach Theorem 1.
Conclusions and Limitations
This paper studies the generalization performance of the nested meta learning with an overparameterized model.
For a precise analysis, we focus on linear models where the total number of data from alltasks is smaller than the dimension of the model parameter.
We show that when the data heterogeneityacross tasks is relatively small, the per-task data covariance matrices with certain properties leadto benign overﬁtting for nested meta learning with the minimum norm solution.
This explains whyoverparameterized meta learning models can generalize well in new data and new tasks.
Furthermore,our theory shows that overﬁtting is more likely to happen in nested meta learning than in ERM,especially when the data heterogeneity across tasks is relatively high in meta learning.
Though ourcurrent analysis is limited to the linear models and non-Bayesian estimate, it is possible to extendit for analyzing Bayesian meta learning with more general fully connected or convolutional neuralnetworks in the neural tangent kernel regime. We will pursue those in our future work.
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] or[N/A] .
You are strongly encouraged to include a justiﬁcation to your answer either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Supplementary material forthe code.
Datasets are public with a link.
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A] Work oftheoretical nature, no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes] See Supplementarymaterial for the complete proofs.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] As a URL in
the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters,
how they
were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi-
ments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A] Work of theoretical
nature.
We cite the authors who propose the baseline models.
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]