Understanding Benign Overfitting in Gradient-based Meta Learning
Abstract
Meta learning has demonstrated tremendous success in few-shot learning with limited supervised data.
In those settings, the meta model is usually overparameterized.
While the conventional statistical learning theory suggests that overparameterized models tend to overfit, empirical evidence reveals that overparameterized meta learning methods still work well – a phenomenon often called “benign overfitting.”
In an attempt to understand this phenomenon, we focus on the meta learning settings with a challenging bilevel structure that we term the gradient-based meta learning, and analyze its generalization performance under an overparameterized meta linear regression model.
While our analysis uses the relatively tractable linear models, our theory contributes to understanding the delicate interplay among data heterogeneity, model adaptation and benign overfitting in gradient-based meta learning tasks.
We corroborate our theoretical claims through numerical simulations.
Introduction
Meta learning, also referred to as “learning to learn”, usually learns a prior model from multiple tasks so that the learned model is able to quickly adapt to unseen tasks [42, 26].
Meta learning has been successfully applied to few-shot learning learning [2, 13], image recognition [50], federated learning [29], reinforcement learning [21] and communication systems [10].
While there are many exciting meta learning methods today in this paper we will study a representative meta learning setting where the goal is to learn a shared initial model that can quickly adapt to task-specific models.
This adaptation may take an explicit form such as the output of one gradient descent step, which is referred to as the model agnostic meta learning (MAML) method [21].
Alternatively, the adaptation step may take an implicit form such as the solution of another optimization problem, which is referred to as the implicit MAML (iMAML) method [38].
Since both MAML and iMAML will solve a bilevel optimization problem, we term them the gradient-based meta learning thereafter.
In many cases, overparameterized models are used as the initial models in meta learning for quick adaptation.
For example, ResNets-based MAML models typically have around 6 million parameters, but are trained on 1-3 million meta-training data [12]. Training such initial models is often difficult in meta learning because the number of training data is much smaller than the dimension of the model parameter.
Previous works on meta learning mainly focus on addressing the optimization challenges or analyzing the generalization performance with sufficient data [18, 19, 14].
Different from these works, we are particularly interested in the generalization performance of the sought initial model in practical scenarios where the total number of data from all tasks is smaller than the dimension of the initial model, which we term overparameterized meta learning .
In those overparameterized regimes, the generalization error of meta learning models is not fully understood.
Motivated by this, we ask:
Whether overparameterized meta learning models would lead to overfitting, provably? [Table]
Empirical studies have demonstrated that the testing performance of MAML improves as the number of parameters increases [12] – a phenomenon often called “benign overfitting.”
Complementing this, we take an initial step by answering this theoretical question in the meta linear regression setting
[Table]
1.1 Prior art
We review prior art that we group in the following two categories.
Benign overfitting analysis.
The empirical success of overparameterized deep neural networks has inspired theoretical studies of overparameterized learning.
The most closest line of work is benign overfitting in linear regression [5], which provides excess risk that measures the difference between expected population risk of the empirical solution and the optimal population risk.
Analysis of overparameterized linear regression model with the minimum-norm solution.
It concludes that certain data covariance matrices lead to benign overfitting, explaining why overparameterized models that perfectly fit the noisy training data can work well during testing.
The analysis has been extended to ridge regression [45], multi-class classification [49] [Table] and adversarial learning with linear models [8].
While previous theoretical efforts on benign overfitting largely focused on linear models, most recently, the analysis of benign overfitting has been extended to two-layer neural networks [7, 32, 22].
However, existing works mainly study benign overfitting for empirical risk minimization problems, rather than bilevel problems such as gradient-based meta learning, which is the focus of this work.
Meta learning.
Early works of meta learning build black-box recurrent models that can make predictions based on a few examples from new tasks [42, 26, 2, 13], or learn shared feature representation among multiple tasks [43, 47].
More recently, meta learning approaches aim to find the initialization of model parameters that can quickly adapt to new tasks with a few number of optimization steps such as MAML [21, 37, 40].
The empirical success of meta learning has also stimulated recent interests on building the theoretical foundation of meta learning methods.
Generalization of meta learning.
The excess risk , as a metric of generalization ability of gradientbased meta learning has been analyzed recently [15, 3, 9, 48, 4, 19].
The generalization of meta learning has been studied in [31] in the context of mixed linear regression, where the focus is on investigating when abundant tasks with small data can compensate for lack of tasks with big data.
Generalization performance has also been studied in a relevant but different setting - representation based meta learning [14, 17].
Information theoretical bounds have been proposed in [30, 11], which bound the generalization error in terms of mutual information between the input training data and the output of the meta-learning algorithms.
The PAC-Bayes framework has been extended to meta learning to provide a PAC-Bayes meta-population risk bound [1, 39, 16, 20].
These works mostly focus on the case where the meta learning model is underparameterized; that is, the total number of meta training data from all tasks is larger than the dimension of the model parameter.
Recently, overparameterized meta learning has attracted much attention.
Bernacchia [6] suggests that in overparameterized MAML, negative learning rate in the inner loop is optimal during meta training for linear models with Gaussian data.
Sun et al.
[44] shows that the optimal representation in representation-based meta learning is overparameterized and provides sample complexity for the method of moment estimator. After our conference submission in May 2022, a concurrent work [28] was posted on arXiv. Both our work and [28] study a common setting where the meta learning models incur overparameterization in the meta level, and we both cover the nested MAML method. However, the two studies differ in terms of how the empirical solution of the meta parameter is obtained.
In our case, we consider the minimum ℓ -2 norm solution, while [28] consider the solution trained with T -step stochastic gradient descent (SGD).
Furthermore, our analysis covers both MAML and iMAML, while [28] only considers MAML.
Compared to the most relevant works, our work is different in the following aspects.
Compared to the works that also analyze generalization error or sample complexity in linear meta learning models such as [15, 3, 9], we focus on the overparameterized case when the total number of training data is smaller than the dimension of the model parameter.
Compared to the work that focus on representation-based meta learning with a bilinear structure [44], we consider initialization-based meta learning methods with a bilevel structure such as MAML and iMAML.
Furthermore, we provide tight analysis of the excess risk with explicit consideration of the benign overfitting condition.
A summary of key differences compared to prior art is provided in Table 1.
We distinguish two different overparameterization settings: i) the per-task level overparameterization where the dimension of model parameter is larger than the number of training data per task, but smaller than the total number of data across all tasks; and, ii) the meta level overparameterization where the dimension of model parameter is larger than the total number of training data from all tasks.
1.2 This work
This paper provides a unifying analysis of the generalization performance for meta learning problems with overparameterized meta linear models.
To our best knowledge, this is the first work that provides the condition for benign overfitting in gradient-based meta learning including MAML and iMAML.
Technical challenges.
Before we introduce the key result of our paper, we first highlight the challenges of analyzing the generalization of gradient-based meta learning and characterizing its benign overfitting condition, compared to the non-bilevel setting such as in [5, 45, 44].
T1)
Due to the bilevel structure of gradient-based meta learning, the solution to the meta training objective involves high order terms of data covariance
As a result, the dominating term in the excess risk propagated from the label noise contains higher order terms, which is harder to quantify and can potentially lead to orders of magnitude higher excess risk than the linear regression case [5, 45, 44].
T2)
The existing analysis of benign overfitting [5, 45] has a solution that is directly related to the data covariance matrix.
However, due to the nested structure of gradient-based meta learning and thus the solution matrix, the solution matrix is a function of both the data covariance matrix and the hyperparameters such as the step size. Therefore, what kind of data matrices can satisfy the benign overfitting condition cannot be directly implied.
T3)
Due to the multi-task learning nature of meta learning, the excess risk of MAML depends on the heterogeneity across different tasks in terms of both the task data covariance and the ground truth task parameter.
As a result, the data covariance matrices from different tasks have different eigenvectors.
This is in contrast to the linear regression case where all the data follow the same distribution.
Contributions. In view of challenges, our contributions can be summarized as follows
C1)
Focusing on the relatively tractable linear models, we derive the excess risk for the minimumnorm solution to overparameterized gradient-based meta learning including MAML and iMAML.
Specifically, the excess risk upper bound adopts the following form
[Equation] where the cross-task variance quantifies the error caused by finite task number and the variation of the ground truth task specific parameter, which is a unique term compared to single task learning.
The bias quantifies the bias resulting from the minimum-norm solution.
And the per-task variance quantifies the error caused by noise in the training data.
C2)
We compare the benign overfitting condition for the overparameterized gradient-based meta learning models and that for the empirical risk minimization (ERM) which learns a single shared parameter for all tasks. We show that overfitting is more likely to happen in MAML and its variants such as implicit MAML than in ERM.
In addition, larger data heterogeneity across tasks will make overfitting more likely to happen.
C3)
We discuss the choice of hyperparameter, e.g., the step size in MAML and the weight of the regularizer in iMAML, such that if the data leads to benign overfitting in ERM, it also leads to benign overfitting in MAML and iMAML.
We show that a negative step size can preserve benign overfitting in MAML.
This is complementary to the recent discovery that the optimal step size of overparameterized MAML during training is negative [6].
Problem Formulation and Methods
In this section, we will introduce the problem setup and the considered meta learning methods.
Problem setup.
In the meta-learning setting, assume task m is drawn from a task distribution, i.e. m ∼ M .
For each task m , we observe N samples with input feature x m ∈ X m ⊂ R d and target label y m ∈ Y m ⊂ R drawn i.i.d.
from a task-specific data distribution P m .
These samples are collected in the dataset D m = { ( x m,n , y m,n ) } Nn =1 , which is divided into the train and validation datasets, denoted as D tr m and D va m .
And |D tr m | = N tr and |D va m | = N va with N = N tr + N va .
We use the empirical loss ℓ m ( θ m , D m ) of per-task parameter θ m ∈ Θ m as a measure of the performance.
In this paper, we consider regression problems, where ℓ m is defined as the mean squared error.
The goal for gradient-based meta learning methods, such as MAML [21] and iMAML [38], is to learn an initial parameter θ 0 ∈ Θ 0 , which, with an adaptation method A : Θ 0 × ( X m × Y m ) N tr → Θ m , can generate a per-task parameter θ m that performs well on the validation data for task m .
Given M tasks, our meta-learning objective is computed as the average of the per-task objective, given by
Meta training objective
[Equation]
Obtaining the empirical solution ˆ θ A 0 by minimizing (1) under a meta learning method A , in the meta testing stage, we evaluate ˆ θ A 0 on the population risk, given by
Meta testing objective
[Equation]
Methods.
We focus on understanding the generalization of two representative gradient-based meta learning methods MAML [21] and iMAML [38] in the overparameterized regime.
MAML obtains the task-specific parameter ˆ θ m ( θ 0 ) by taking one step gradient descent with step size α of the per-task loss function ℓ m from the initial parameter θ 0 , that is
[Equation]
On the other hand, iMAML obtains the task-specific parameter ˆ θ m from the initial parameter θ 0 by optimizing the taskspecific loss regularized by the distance between ˆ θ m and θ 0 , that is
[Equation]
where γ > 0 is the weight of the regularizer.
As summarized in Figure 1, MAML has smaller computation complexity than iMAML since iMAML requires solving an inner problem during adaptation, while iMAML may achieve smaller test error since it explicitly minimize the loss.
Main Results: Benign Overfitting for Gradient-based Meta Learning
In this section, we introduce the meta linear regression model and some necessary assumptions for the analysis.
We present the main results, highlight the key steps of the proof and conduct simulations to verify our results.
Due to space limitations, we will defer the proofs to the supplementary document.
3.1 Meta linear regression setting
To make a precise analysis, we will assume the following linear data model.
Denoting the ground truth parameter on task m as θ ⋆m ∈ R d , and the noise as ϵ m , we assume the data model for task m is
[Equation]
Given the linear model (5), the meta training problem (1) with adaptation method (3) or (4) generally have unique solutions when d ≤ NM .
However, when the meta model θ 0 and thus the per-task model θ m are overparameterized, i.e. d > NM , the training problem (1) may have multiple solutions.
In the subsequent analysis, we will analyze the performance of the minimum norm solution because recent advances in training overparameterized models reveal that gradient descent-based methods converge to the minimum norm solution [24, 34].
We provide a formal definition below.
Definition 1 (Minimum norm solution) .
Denote X va m :=
[ x m, 1 , . . .
, x m,N va ] ⊤ ∈ R N va × d , y va m := [ y m, 1 , . . .
, y m,N va ] ⊤ ∈ R N va .
With A ( θ, D tr m ) being either (3) or (4) , the minimum norm solution to the meta training problem (1) under the linear regression loss is expressed by
[Equation]
In our analysis, we make the following basic assumptions.
Assumption 1 (Overparameterized model) .
The total number of meta training data is smaller than the dimension of the model parameter; i.e. NM < d .
Assumption 2 (Subgaussian data) .
The noise ϵ m is subGaussian with E [ ϵ m ] = 0 and E [ ϵ 2 m ] =
σ 2 .
For the m -th task, data x m = V m Λ 12 m z m , where z m has centered, independent, σ x -subGaussian entries; E [ z m ] = 0 , E
[ z m z ⊤ m ] = I d , with I d being a d × d identity matrix.
Assumption 3 (Data covariance matrix) .
1) Assume for all m ∈ [ M ] , i ∈ [ d ] , λ m,i > 0 ,
Tr( Λ m ) , Tr( Λ ) are bounded, i.e. for all m [-in descending order as  i and  m,i ,respectively. Assume 
-]∈
[ M ] , Tr( Λ m ) ≤ c λ .
2) Cross-task data heterogeneity V ( { Q m } Mm =1 ) :
= max i,m | ( λ i − λ m,i ) /λ i | is bounded above and below.
Assumption 4 (Task parameter) .
The ground truth parameter θ ⋆m is independent of X m and satisfies Cov [ θ ⋆m ]
= ( R 2 /d )
I d , where R is a constant, and the entries of θ ⋆m are i.i.d.
O ( R/ √ d ) -subGaussian.
Assumption 1 defines the setting that the meta level is overparameterized, which has also been used in [44].
Note that Assumptions 2-4 are common in the analysis of meta learning in [15, 3, 9, 23].
With the linear data model (5), the (minimum norm) solutions to the meta training objective (1) and the meta testing objective (2) can be computed analytically which we will summarize next.
Proposition 1.
(Empirical and population level solutions)
Under the meta linear regression model (5) , the meta testing objective of method A in (2) can be equivalently written as
[Equation]
) where the matrix W A m and its empirical version ˆ W A m are given in Table 2 with ˆ Q al m := 1 N X al ⊤ m X al m .
The optimal solutions to the meta-test risk and the minimum-norm solutions to the empirical meta training loss are given below respectively
[Equation]
where † denotes the Moore–Penrose pseudo inverse; ∆ A M is a constant that depends on X m , ϵ m specified in the supplementary document. [Table] [Equation] [Table]
[Equation] [Table] [Equation]
To study overfitting in the meta learning model, we quantify its generalization ability via the widely used metric - excess risk .
The excess risk of method A (which can be “ ma ” for MAML and “ im ” for iMAML), with an empirical solution ˆ θ A 0 and population solution θ A 0 , is defined as
[Equation]
In (9), the excess risk measures the difference between the population risk of the empirical solution, ˆ θ 0 and the optimal population risk.
Given total number of training samples MN , if d → ∞ , the classic learning theory implies that the excess risk E A (ˆ θ A 0 ) also grows, which leads to overfitting [25]. The larger the excess risk, the further the empirical solution ˆ θ A 0 is from the optimal population solution θ A 0 , indicating more severe overfitting . 3.2 Main results
With the closed-form solutions given in Proposition 1, we are ready to bound the excess risk of MAML and iMAML in the overparameterized linear regime.
For notation brevity, we first introduce some universal constants such as c 0 , c 1 , c 2 , . . .
, and only present the dominating terms in the subsequent results.
The precise presentation of non-dominant terms are deferred to the supplementary document.
We first decompose the excess risk into three terms in Proposition 2.
Proposition 2.
Denote W A :=
E m [ W A m ] .
There exists a universal constant c 0 > 0 that the excess risk of a meta learning method A can be bounded by
[Equation]
[Equation]
where the first term E θ ⋆m is a function of θ ⋆m , θ A 0 , W A , ˆ W A m , which quantifies the weighted variance of the ground truth task specific parameters θ ⋆m ; the second term E ϵ m , as a function of ϵ m , is the weighted noise variance; and the third term E b , as a function of θ A 0 , W A , ˆ W A m , is the bias of the minimum-norm solution in overparameterized MAML or iMAML.
Based on this decomposition, as we will show in Section 4, the bound of the excess risk can be derived from the bound of these three terms E θ ⋆m , E ϵ ⋆m , E b , respectively, which gives Theorem 1.
Theorem 1 (Excess risk bound) .
Suppose Assumptions 1-4 hold.
Let µ 1 ( · ) ≥ µ 2 ( · ) . . .
denote the eigenvalues of a matrix in the descending order.
Define W A M :=
1 M (cid:80) Mm =1 W A m .
For the meta linear regression problem with the minimum-norm solution (6) , for 0 ≤ k ≤ d , define the effective ranks as
[Equation]
[Equation]
With the cross-task data heterogeneity V defined in Assumption 3, if there exist universal constants c 1 , c 2 , c 3 > 1 such that the effective dimension k ∗ = min { k ≥ 0 : r k ( W A M ) ≥ c 1 NM } , c 2 log(1 /δ )
<
NM and k ∗ < NM/c 3 , then with probability at least 1 − δ , the excess risk satisfies
E A
[Equation]
≤
[Equation]
Theorem 1 provides the excess risk bound via the effective ranks.
In (11), the effective ranks r k and
R k of a matrix capture the distribution of the eigenvalues of this matrix, and the effective dimension k ∗ determines the above upper bound by considering the asymmetry of the eigenvalues of the solution matrix.
The idea is to choose k ∗ that makes R k ∗ large enough and keeps k ∗ small enough compared to
MN so that the variance term of the excess risk is controlled.
For example, r 0 is the trace normalized
80 N 0.
E ( ^ 3 0 )
ERM , = 0 : 01 , = 0 :, = 0 : 2 , = 0 :(a) MAML with different α .
80 N
E ( ^ 3 0 )
ERM .
= 0 : 01 .
= 0 :.
[Figure]
by the largest eigenvalue, which is bounded above by R 0 .
And both r 0 and R 0 are no larger than the rank of the matrix, and they are equal to the rank only when all non-zero eigenvalues are equal.
If the eigenvalues distribute more uniformly, the effective rank will be larger, otherwise smaller.
Remark 1. 1)
The definition of effective rank has been also given in [5] but only on the data matrix
Q .
And our setting reduces to the single task ERM learning, or the linear regression case in [5], when
M = 1 , θ ⋆m
= θ 0 , W A m = Q , which implies that the cross-task variance in (10) as well as the data heterogeneity V ( · ) reduces to zero.
Accordingly, Theorem 1 reduces to Theorem 4 in [5].
Given Theorem 1, in order to control the excess risk of solution ˆ θ A 0 , we want r 0 ( W A M ) to be small compared to the total number of training samples MN , but r k ∗ ( W A M ) and R k ∗ ( W A M ) to be large compared to MN .
In addition, the cross-task heterogeneity V should be small.
Since for a matrix
W , r k ( W ) ≤ R k
( W ) ≤ d , this suggests the model benefits from overparameterization.
Building upon Theorem 1, we now discuss the conditions for “benign overfitting”, which refers to the situation that overparameterization does not “harm” the excess risk, or the excess risk still vanishes when d > MN and N, M, d increase.
Definition 2 (Condition for benign overfitting in meta learning) .
The weight matrices W A M for method A satisfy the benign overfitting condition in gradient-based meta learning, if and only if lim NM [Equation] d [Equation]
This guarantees the excess risk (12) goes to zero in overparameterized meta learning models with sufficient training data from all tasks. To provide an intuitive explanation, Figure 2 plots the population risk versus the number of the training data, which demonstrates the “double descent” curve. Namely, as N increases, E (ˆ θ 0 ) first decreases, then increases and then decreases again, as is discovered in overparameterized neural networks [35]. The trend in Figure 2 is similar to the trend observed in [36].
When d/ ( NM ) > 1 , the model is overparameterized, which can overfit the training data, leading to larger excess risk as N decreases. However, Figure 2 shows the excess risk does not become too large as N decreases, indicating that overfitting does not severely harm the population risk in this case.
In this section, we discuss how the benign overfitting condition (13) in gradient-based meta learning reduces to that in single task linear regression; e.g., in [5, 45]. We also provide examples to show Q1)
how certain properties of meta training data affect the excess risk; and, Q2) how to choose the hyperparameters that preserve benign overfitting. Data covariance and cross-task heterogeneity.
Theorem 1 reveals that the excess risk depends on both the eigenvalues of the data covariance matrix Q m , and the cross-task data heterogeneity, measured by V ( { Q m } Mm =1 ) . We give an example below to better demonstrate how these two properties of gradient-based meta training data affect the excess risk.
[Figure] [Figure] Example 1 (Data covariance) .
Suppose Q m = diag
( I d 1 , β I d − d 1 ) , ∀ m .
Set M = 10 , d = 200 , d 1 = 20 , α = 0 . 1 for MAML and γ = 10 3 for iMAML. Then the benign overfitting condition (13) is satisfied by MAML and iMAML.
We plot the excess risk under different β in Figure 3.
From Figure 3 we can observe that given a fixed number of training data N , the population risk increases with β for both MAML and iMAML. This observation verifies our theory since larger β results in a smaller R A k ( W A M ) , leading to a larger upper bound on the variance term in (12). Example 1 demonstrates how the per-task data matrix Q m affects the excess risk.
We consider another example that demonstrates how the data heterogeneity across tasks affects the excess risk. Example 2 (Data heterogeneity) .
Suppose Q m =
| ω m + 1 | diag( I d 1 , β I d − d 1 ) with ω m ∼ N (0 , σ 2 ω ) for all m .
Set M = 10 , d = 200 , d 1 = 20 , β = 0 . 3 , α = 0 . 1 for MAML and γ = 0 . 1 for iMAML.
Then it satisfies the benign overfitting condition (13) for MAML and iMAML. Figure 4 plots the excess risk with different choices of σ ω . Observing from Figure 4 that the larger σ 2 ω , the higher the excess risk, and the more difficult for the benign overfitting condition to be satisfied for both MAML and iMAML. Therefore, compared to
ERM with a single task, the benign overfitting condition for MAML is more restrictive as it imposes constraints for both the expected data covariance Q m , and the data heterogeneity V ( { W A m } Mm =1 ) . Connection to multi-task ERM. To compare benign overfitting in the gradient-based meta learning with that in the conventional ERM, where θ m = θ 0 , we can set the step size α = 0 in MAML, or γ → ∞ in iMAML, and N va = N , which reduces to conventional ERM without adaptation. Compared to that of MAML and iMAML in (13), the benign overfitting condition is less restrictive for ERM since it does not impose constraints on α or γ .
Intuitively, benign overfitting is more likely to happen in MAML or iMAML than in ERM. The hyperparameters α and γ will affect the eigenvalues of W ma m , W im m , respectively, thus affecting their corresponding excess risk. Here we provide a sufficient condition where the benign overfitting condition in ERM is preserved in MAML or iMAML. We summarize the results in the corollary below.
Corollary 1 (Hyperparameters that preserve benign overfitting) . Recall λ 1 is the largest eigenvalue of Q . For MAML, when 0 < α ≤ 13 λ 1 , and for iMAML, when γ ≥ λ 1 , then the effective ranks of W ma and W im are bounded above and below by a positive constant times the effective rank of Q ,
2 log 10 M ( N = 10 ;d = 200 ;s = 0 : 5) -0. l og t a s k p a r a m e t e
r
d
i s t a n ce
ERMMAMLiMAML (a) Cross-task variance vs task number      60 M ( N = 10 ;d = 200 ;s = 0 : 5)      b i a s ERMMAMLiMAML
[Figure]
and therefore the benign overfitting condition holds for MAML and iMAML if it holds for ERM. To summarize, there are constants c 1 , c 2 , c 3 , c such that for k ∗ =
min { k ≥ 0 : r k ( Q M ) ≥ c 1 NM } . For δ < 1 , c 2 log(1 /δ ) < NM and k ∗ < NM/c 3 , with probability at least 1 − 7 e − 2 NM/c , it follows
[Equation]
Remark 2. For MAML, let the unordered eigenvalues ˜ µ i ( W ma ) =
λ i (1 − αλ i ) 2 . One challenge to control ˜ µ i ( W ma ) is that ˜ µ i ( W ma ) are not necessarily monotonic w.r.t. λ
i ; that is, it does not necessarily hold that ˜ µ 1 ≥ ˜ µ 2 ≥ · · · ≥ ˜ µ d . For any λ i ≥ λ j , if ˜ µ i ( W A ) ≥ ˜ µ j ( W A ) , then we say the order of the eigenvalues is preserved. For this to hold, it requires ˜ µ i ( λ i ) to be a monotonically non-decreasing function of λ i , which yields α ≤ 13 λ 1 .
Similar results can be obtained for iMAML by controlling the value of γ . And the bound on α or γ further ensures that ˜ µ i ( W A ) is bounded above and below by a positive constant times the effective rank of Q .  Proof Outline
In this section, we highlight the key steps of the proof for Theorem 1. We achieve so by analyzing the three terms in Proposition 2 respectively. The first two terms in (10) can be bounded based on the concentration inequalities on subGaussian variables, given in Lemmas 1 and 2. Lemma 1 (Bound on cross-task variance) .
With probability at least 1 − δ , it follows [Equation]
where (cid:101) O ( · ) hides the log polynomial dependence on N, M, d . The cross-task variance term analzyed in Lemma 1 is unique in meta learning, which captures the data heterogeneity across different tasks. To elaborate Lemma 1, we plot the cross-task variance versus the task number in Figure 5a with the dimension of model parameter d = 200 , training validation split parameter s = N tr /N = 0 . 5 , per-task data number N = 10 .
This figure demonstrates that the cross-task variance decreases with number of tasks M with rate O ( 1 M ) , which is consistent with the theoretical results in Lemma 1.
Interestingly, we can observe a double descent phenomenon when M increases so that the model goes from overparameterized setting to underparameterized setting. Note that the “double descent” phenomenon in Figure 5a is different from that in the conventional linear regression case in that increasing the number of tasks M will increase data from different per-task distributions, but they still help reduce the excess risk.
Lemma 2 (Bound on bias) .
There is a constant c 3 that depends only on σ x , such that for any 1 < log(1 /δ ) < MN va , with probability at least 1 − δ , we have
[Equation] This term is similar to the bias term in the linear regression case, but directly depending on the solution matrix W instead of the data matrix Q . To elaborate Lemma 2, Figure 5b demonstrates that the bias term decays with M until it reaches zero when the model is underparameterized. These two terms in Lemma 1 and Lemma 2 do not go to infinity as N, M, d increase.
Note that, the key step is the bound on E ϵ m , which is the dominating term in the decomposition of excess risk (10) in the overparameterized regime. We will bound it below. Lemma 3 (Bound on per-task variance) .
There exist constants c 1 , c 2 , c 3 , c such that for 0 ≤ k ≤ 2 NM/c 3 , r k ( W A M ) ≥ c 1 NM , and k 0 ≤ k , with probability at least 1 − 7 e − 2 NM/c , it follows [Equation]
Note that, in the single task linear regression case, the cross-task data heterogeneity becomes none, i.e., V = 0 . This term is unique in the meta learning setting with multiple tasks. Plugging the results of Lemmas 1, 2 and 3 into (10), we will reach Theorem 1.  Conclusions and Limitations
This paper studies the generalization performance of the gradient-based meta learning with an overparameterized model. For a precise analysis, we focus on linear models where the total number of data from all tasks is smaller than the dimension of the model parameter. We show that when the data heterogeneity across tasks is relatively small, the per-task data covariance matrices with certain properties lead to benign overfitting for gradient-based meta learning with the minimum-norm solution. This explains why overparameterized meta learning models can generalize well in new data and new tasks.
Furthermore, our theory shows that overfitting is more likely to happen in meta learning than in ERM, especially when the data heterogeneity across tasks is relatively high. One limitation of this work is that the analysis focuses on the meta linear regression case. While this analysis can capture practical cases where we [-want r 
-]reuse the feature extractor from pre-trained models and only meta-train the parameters in the last linear layer, it is also promising to extend our analysis to nonlinear cases via means of random features and neural tangent kernels in the future work Acknowledgments
This work was partially supported by National Science Foundation MoDL-SCALE Grantand the Rensselaer-IBM AI Research Collaboration ( http://airc.rpi.edu ), part of the IBM AI
Horizons Network ( http://ibm.biz/AIHorizons ).
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] Work of theoretical nature, no potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Supplementary material for the complete proofs.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] As a URL in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] (c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] Work of theoretical nature.
We cite the authors who propose the baseline models.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Supplementary Material
In this supplementary document, we present the missing derivations of some claims, as well as the proofs of all the lemmas and theorems in the paper.
Table of
Contents
A Notations B Proof of Proposition B.1 Model agnostic meta learning method . . B.2 Implicit model agnostic meta learning method . . . . . . . . . . . . . . . . . .
C Proof of Theorem
C.1 Proof of Proposition 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.3 Proof of Lemma 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.4 Proof of Lemma 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D Auxiliary Lemmas
D.1 Basic algebraic properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Concentration inequalities . .
D.3 Other intermediate results . .
A Notations
We use [ X m ] m to represent row stack of matrices X m with indices m , i.e.
[ X m ] m = (cid:2) X ⊤ 1 , X ⊤ 2 , . .
, X ⊤ M (cid:3) ⊤ .
We use µ i ( · ) to denote the i -th eigenvalue of a matrix with descending order.
For any matrix M ∈ R n × d denote M 0: k to be the matrix which is comprised of the first k columns of
M , and M k : ∞ to be the matrix comprised of the rest of the columns of M .
E [ · ] represents expectation and Cov[ · ] represents covariance.
For any vector η ∈ R d denote η 0: k to be the vector comprised of the first k components of η , and η k : ∞ to be the vector comprised of the rest of the coordinates of η .
Denote Σ 0: k = diag ( λ 1 , . . . , λ k ) , and Σ k : ∞ =
diag ( λ k +1 , λ k +2 , . . . ) .
B Proof of Proposition
Proposition 3 (Empirical and population level solutions) .
Under the data model (5) , the meta-test risk of method A defined in (2) can be computed by
[Equation] The optimal solutions to the meta-test risk and the minimum-norm solution are given below respectively [Equation] where † denotes the Moore–Penrose pseudo inverse, the error term ∆ A M is a polynomial function of M, N, d , which will be specified in the following sections for MAML and iMAML.
And ˆ Q al m := 1 N X al ⊤ m X al m . The weight matrices of different methods, W A m and ˆ W A m , are given in Table 2. B.1 Model agnostic meta learning method
Without loss of generality, assume σ = 1 to simplify notation. We use meta-test risk R A N to represent expected test risk with finite number of adaptation data N during testing, which is slightly different compared to population risk R A =
lim N →∞ R A N .
The MAML meta-test risk is defined as [23] [Equation] [Equation] where the matrix is defined as
[Equation]
Assume during meta testing [Equation] we have infinite adaptation data, i.e., N → ∞ , then the optimal population risk of MAML is
[Equation] [Equation]
In MAML
[Equation]
define θ ma0 as the minimizer of the optimal population risk of MAML, given by
[Equation]
Using the optimality condition of L ma ( θ 0 , D ) given in (1) [Equation] we have ˆ [Equation]
Therefore
[Equation]
we can arrive at (8b) by defining [Equation] It is worth noting that, from (20) and the above discussion [Equation]
we have the property [Equation] and [Equation]
which will be used in later sections to derive the excess risk [Equation] B.2 Implicit model agnostic meta learning method For the iMAML method, the task-specific parameter ˆ θ im m is computed from the initial parameter θby optimizing the regularized task-specific empirical loss, given by
[Equation] (26) where γ is the weight of the regularizer [Equation] and D m is the adaptation data during meta-testing or training data during meta-training.
The estimated task-specific parameter can be computed by [Equation] The empirical loss of iMAML is defined as the average per-task loss, given by [Equation]
[Equation] whose minimizer is
[Equation] Using the optimality condition of the above problem [Equation]
we obtain
[Equation] with ∆ im M = (cid:16) M (cid:88)
[Equation] = [Equation] 30b) where we define
[Equation] The meta-test risk of iMAML is defined as [Equation]
where the weight matrix is defined as [Equation] where W im m = ( γ − 1 Q m + I ) − 1 Q m ( γ − 1 Q m + I ) − 1 .
Simplify the notation of X m,N a , y m,N a , ˆ Q m,N a as X m [Equation] y m [Equation]
ˆ Q m . The derivation of (31) is given by [Equation] where ( a ) follows from the definition of Σ θ m
= ( ˆ Q al m + γ I ) − 1 , and W im m,N a = γ 2 Σ θ m Q m Σ θ m . Applying the fact that y m = X m θ ⋆ m + e m
and
E e m [ e m ] = 0 , one can further derive from (32) that [Equation]
Based on the linearity of trace and expectation [Equation] and the cyclic property of trace, the last term inside the expectation in the above equation can be computed as
[Equation] To derive all the terms related to θ ⋆m , based on the Woodbury matrix identity
[Equation] I − ˆ Q m Σ θ m =
I − Σ θ m ˆ Q
m = γ Σ θ m [Equation] we have
[Equation]
and then the terms related to θ ⋆m in (33) can be computed by
[Equation]
where ( a ) follows from (34)
[Equation]
and ( b ) follows from the definition of W
im m,N a .
Combining (33) and (35) and rearranging the equations, we obtain
[Equation]
where ( c ) follows from rearranging the equations; ( d ) follows from the fact that
[Equation]
Since lim N a →∞ 1 N a E [ γ − 2 Tr [Equation] W im m,N a ˆ Q m,N a )] = 0 , from the definition of the population risk in (2)
[Equation]
the population risk of iMAML is given by [Equation] 38b
[Equation]
whose minimizer is given by [Equation] [Equation] It is worth noting that, we have the property E x m [ ˆ W im m ]
= W
im m,N , lim N →∞ W im m,N = W im m , which will be used in later sections to derive the specific optimal population risk and statistical error.
The above discussion provides proof for Proposition 1. C Proof of Theorem
Section B gives solutions to the empirical and population risks. In this section, we provide proof to the main theorem, starting with the decomposition of the excess risk in Proposition 2. Note that our proof of the bound on the variance follows the idea of [5] by separately bounding the terms related to the first k largest eigenvalues and the rest eigenvalues of the per-task weight matrices.
C.1 Proof of Proposition
Next we analyze the excess risk defined in (9) based on the solutions of MAML and iMAML. First we restate the complete version of Proposition 2 in Lemma 4.
Lemma 4 (Restatement of Proposition 2) . With probability at least 1 − δ , the excess risk of the MAML with the minimum-norm solution is bounded by [Equation]
where the weight matrix and the constants are defined as [Equation] Note that C A 2 can be either C ma2 for MAML or C im2 for iMAML.
Proof.
The excess risk E A can be derived as
E A (ˆ θ 0 ) :
= R
[Equation]
In (41), I 1 can be bounded by
{+I 1 = (cid:13)(cid:13)(cid:13)(cid:13)(cid:16)(cid:88) m ˆ W m (cid:17) † (cid:16)(cid:88) m ˆ W m θ ⋆m (cid:17) − θ 
+}
(cid:13) (cid:13)(cid:13)(cid:13)
W A = (cid:13)(cid:13)(cid:13)(cid:13)(cid:16)(cid:88) m
ˆ W m (cid:17) † (cid:16)(cid:88) m
ˆ W m ( θ ⋆m − θ 0 ) (cid:17)
+ (cid:16)(cid:16)(cid:88) m
ˆ W m (cid:17) † (cid:16)(cid:88) m ˆ W m ) − I (cid:17) θ 0 (cid:13)(cid:13)(cid:13)(cid:13)
W A ≤ 2 (cid:13)(cid:13)(cid:13)(cid:16)(cid:88) m
ˆ W m (cid:17) † (cid:16)(cid:88) m
ˆ W m ( θ ⋆m − θ 0 )
(cid:17)(cid:13)(cid:13)(cid:13) 2 W A + 2 (cid:13)(cid:13)(cid:13)(cid:16)(cid:16)(cid:88) m
ˆ W m (cid:17) † (cid:16)(cid:88) m ˆ W m ) − I (cid:17) θ 0 (cid:13)(cid:13)(cid:13) 2 W A =2 (cid:13)(cid:13)(cid:13)(cid:16)(cid:88) m ˆ W m (cid:17) † (cid:16)(cid:88) m
ˆ W m ( θ ⋆m − θ 0 )
(cid:17)(cid:13)(cid:13)(cid:13) 2 W A + 2 θ ⊤ 0 B θ(42) with the matrix B defined as
B = (cid:16)(cid:16)(cid:88)
[Equation]
(cid:16)(cid:16)(cid:88)
[Equation]
(cid:17)
[Equation]
(cid:16)(cid:88) m
[Equation]
And (a) is from the relationship of ˆ W and ˜ X , recall we use [ · ] m to represent row concatenation of matrices or vectors.
In (41), I 2 can be bounded by
[Equation]
where ( b ) follows from expanding the quadratic terms
[Equation]
and
[Equation]
By taking the expectation w.r
[Equation]
t. e m
[Equation]
we need to bound Tr( C 1 ) ,
Tr( C 2 ) .
Based on the cyclic property of trace, Tr( C ma2 ) can be further derived as
[Equation]
For a given square matrix P m
[Equation]
define
[Equation]
Then Tr( C ma2 ) can be further written as
[Equation]
Since we have
[Equation]
by the subGaussian concentration inequality [46], it holds with probability at least 1 − δ over ϵ that
[Equation]
{+Combining the bounds for I 
+}
and I 2 in (42)
and (48) completes the proof. C.2 Proof of Lemma Define [Equation]
Then we can derive that
[Equation]
By the Hanson-Wright inequality
[Equation]
with probability at least 1 − δ over θ ⋆ m [Equation] we have [Equation]
To compute E θ ⋆m | ˆ W A m (cid:2)(cid:13)(cid:13) U ⊤A z A (cid:13)(cid:13) 2 W A (cid:3) , first recall Cov
[ θ ⋆m ]
= R 2 d
I , then we have
E θ ⋆m | ˆ W A m
[ z ⊤A U
A W A U ⊤A z A ]
= R
[Equation]
[Equation]
Combining (49) and (50) leads to the conclusion.
C.3 Proof of Lemma
Proof.
Recall B := (cid:16) ˜ X ⊤ ( ˜ X ˜ X ⊤ ) − 1 ˜
X
− I (cid:17) W (cid:16) ˜ X ⊤ ( ˜ X ˜ X ⊤ ) − 1 ˜ X − I (cid:17) .
First note that (cid:16) ˜ X ⊤ ( ˜ X ˜ X ⊤ ) −
[Equation] [Equation]
Thus, for any u in the column space of ˜ X ⊤ , u can be represented as u = ˜
X
⊤ ¯ u , ¯ u ̸ = 0 , then we have
[Equation]
And for any u orthogonal to the colomn space of ˜ X ⊤ , ˜ Xu = 0 , therefore
[Equation]
[Equation]
Since any u ∈ R d can be represented as a combination of a vector in the colomn space of ˜ X ⊤ and a vector orthogonal to the colomn space of ˜ X ⊤ , (cid:0) ˜ X ⊤ ( ˜ X ˜ X ⊤ ) − 1 ˜
X − I (cid:1) has eigenvalues whose absolute values are smaller than 1 , i.e.
[Equation]
{+Then expanding θ ⊤ 
+}
Bθ 0 , we have [Equation] where ( a ) follows from (51) [Equation]
and ( b ) follows from (54) [Equation] Thus, due to Theorem 9 in [27] [Equation]
there is an absolute constant c such that for any 1 < t < MN va with probability at least 1 − e − t
[Equation]
it follows that
[Equation]
where r ( W ) is defined as
[Equation]
This completes the proof.
C.4 Proof of Lemma
To prove Lemma
[Equation]
we need to bound Tr( C ) =
Tr( C 1 ) +
Tr( C 2 ) . We first show in Lemma 5 that Tr( C 2 ) can be bounded as Θ (Tr( C 1 )) . Then the key step is to bound Tr( C 1 ) .
To bound Tr( C 1 ) , first we show in Lemma 6 that Tr( C 1 ) can be decomposed into terms that are related to the first k largest eigenvalues of W and the term that is only related to the rest eigenvalues of W .
Next we bound the term related to the d − k smallest eigenvalues of W , as a function of µ n ( A ) , given in
Lemma 7.
And then we bound the term related to the k largest eigenvalues of W , given in Lemma 8. Finally, we bound the eigenvalues of µ n ( A ) in Lemma 9. Lemma 5 (Bound on Tr( C A 2 ) in terms of Tr( C A 1 ) ) .
Recall α is the step size.
Let c be a constant that depends only on σ x . It holds with probability at least 1 − δ that [Equation] Proof.
Recall
[Equation]
59) and X va m ˆ Q tr m X va ⊤ m ∈ R N va × N va , where X va m = Z va m Λ 12 m V ⊤ m . Then it satisfies that [Equation]
By Lemma 13, we have [Equation] [Equation] Then by concentration inequality on Z , with probability at least 1 − δ over Z , it holds that
[Equation]
This completes the proof.
Lemma 5 shows that Tr( C 2 ) can be bounded as Θ (Tr( C 1 )) .
Then we proceed to bound Tr( C 1 ) .
In
Lemma 6, we decompose Tr( C 1 ) can be decomposed into terms that are related to the first k largest eigenvalues of W and the term that is only related to the rest eigenvalues of W .
Lemma 6 (Decomposition of Tr( C 1 ) ) .
For both MAML and iMAML, Tr( C A 1 ) can be decomposed into
[Equation]
where the matrix is defined as A = ˜ X ˜ X ⊤ .
For notation simplicity
[Equation]
we omit the superscript A
[Equation] Proof.
Denote
[Equation]
Recall
[Equation]
Denote
[Equation]
where ¯ Λ ma m = Λ 12 m ( I − α Λ m ) , ¯ X
=
[ ¯ X m ] m , and ˜ Λ A m = ¯ Λ A
m + ˜ Λ A m − ¯ Λ
A m = ¯ Λ A m + ∆ A Λ m
[Equation]
Let A = ˜ X ˜ X ⊤ , and the singular value decomposition of W as W = V W Λ W V ⊤ W , then for any 0 ≤ k ≤ d , W can be computed by
[Equation]
Similarly, ¯ X m and ˜ X m can be computed by
[Equation]
Plugging (62) and (63) into (44), we have Tr( C 1 ) can be computed by
[Equation]
from which the proof is complete.
Then we bound the term related to the d − k smallest eigenvalues of W as a function of µ n ( A ) , given in Lemma
[Equation]
Lemma 7 (Bound on Tr( ˜ X k : ∞ W k : ∞
˜ X ⊤ k : ∞
A − 2 ) in Tr( C 1 ) ) .
With probability at least 1 − 3 e − t over Z , and for c ≥ t , it holds that
[Equation]
where µ n is the smallest eigenvalue of a matrix.
Proof.
By Von Neumann’s trace inequality in Lemma 13, Tr( ˜ X
k : ∞ W k : ∞
˜ X ⊤ k : ∞
A − 2 ) is bounded by
Tr
[Equation]
To bound Tr( W k : ∞
˜ X ⊤ k : ∞ ˜ X k : ∞ ) , we first rewrite ˜ X ⊤ k : ∞ ˜ X k : ∞ as ˜ X ⊤
[Equation]
: ∞
[Equation]
=
[Equation]
With ∆ Λ m = α Λ m ( I − ˆ D tr m ) Λ 12 m , we have
[Equation]
Therefore,
Tr( W k : ∞ ˜ X ⊤ k : ∞ ˜
X k : ∞ ) can be bounded by
[Equation]
(66) where ( a ) is from (65), ( b ) follows from AM–GM inequality, and ( c ) is from rearranging terms.
From Lemma 13, the first term in (66), (cid:80)
{+Mm =
+}
Tr( W k : ∞ W m,k : ∞ ) is bounded by
[Equation]
Recall Assumption 2 that z va mj has σ 2 x -subGaussian entries and with zero mean and unit variance. By Lemma 13 and the subGaussian concentration inequality, with probability at least 1 − δ over Z va m , the second term in (66) is bounded by [Equation]
Similarly, based on Lemma [Equation] and the subGaussian concentration inequality, with probability at least 1 − δ over Z va m and Z tr m , the last term in (66) is bounded by M (cid:88)
[Equation] Combining (66)-(69), with probability at least 1 − 3 e − t , it holds that
[Equation] This completes the proof. Next we bound the term related to the k largest eigenvalues of W , given in Lemma 8.
Lemma 8 (Bound on terms in Tr( C 1 ) related to the first k eigenvalues) .
There exists c with 0 ≤ k ≤ c such that with probability at least 1 − 2 e MN/c , the following holds [Equation] Proof. Recall that
[Equation]
Based on Lemma
[Equation]
disregarding the terms that has corresponding zero eigenvalues
[Equation] we have [Equation] Therefore
[Equation]
we have
[Equation]
where ˜ x i ∈ R NM × 1 is computed by
[Equation]
And based on Lemma 10, we have [Equation] where L j is the span of the n − k eigenvectors with the smallest eigenvalues of A − j , and Π L j represents the projection to L j . Since ˜ x j = ¯ x j + [ Z va m ∆ Λ m ( V ⊤ m ) j ] m and ¯
[Equation] It follows that [Equation] [Equation]
Based on Lemma 15, we have ∥
[Equation]
Each term in (72) can be further derived as
[Equation] Its expectation can be computed by [Equation] Based on the subGaussian concentration inequality, it holds with probability at least 1 − e − t that,
[Equation] Let M = Π ⊤ L ⊥ j Π L ⊥ j , then ∥ M ∥ = 1 , Tr( M ) = Tr( M 2 ) =
k . With probability at least 1 − e − t , it holds that [Equation] Let S =
N (cid:80) Mm =1 ( (cid:80) di =1 ¯ λ mi ) 2 .
With probability at least 1 − e − t , it follows that [Equation] ≥ ∥ ˜ x j ∥
[Equation]
− [Equation] /c. According to Lemma 12, A − j =
{+A − ˜ x j ˜ x ⊤ j ⪯ A , which, combined with Lemma 11, leads to µ k +
 ( A − j ) < µ k +1 ( A ) = µ 1 ( A k ) . Since µ n ( A − j ) ≥ (cid:80) di =1 ζ i − max i ζ i ≥ µ n ( A k ) , we have [Equation]+}
where ( a ) is because µ n ( A − j ) ≥ µ n ( A k ) and µ k +1 ( A − j ) < µ 1 ( A k ) . Finally in Lemma 9, we bound the eigenvalues of A to complete the bound on the term related to the d − k smallest eigenvalues of W . Lemma 9 (Bound on eigenvalues of A ) . Denote µ n ( A ) as the smallest eigenvalue of A , then
[Equation]
[Equation] [Equation]
Proof. Recall that
[Equation]
For any fixed unit vector u , we have
[Equation]
{+77) where u ⊤ ˜ X = (cid:80) Mm =
 u ⊤ m ˜ X m =+}
{+(cid:80) Mm =
+}
u ⊤ m Z va m ( ¯ Λ m + ∆ Λ m ) V ⊤ m . Therefore, it follows that u ⊤ Au = [Equation] M (cid:88)
[Equation] = [Equation] The first term I 1 can be further derived as
[Equation]
where u ⊤ m z va mi is c ∥ u m ∥ σ x -subGaussian. First, we compute its expectation, which gives [Equation] Recall ∆ Λ m = α Λ m
( I − ˆ D tr m ) Λ 12 m . Then the second term I 2 can be derived as [Equation] And the the expectation of I 2 is
[Equation]
We first compute E
[ Λ m ( I − ˆ D tr m ) Λ m I − ˆ D tr m ) Λ m ] .
By Lemma 16 we have [Equation] The expectation of I 2 over both Z tr m and Z va m is
[Equation] Note the expectation of I 3 is 0 .
This is because we can first take expectation w.r.t. z tr m , which yields that E z tr m [∆ Λ m ] = 0 , therefore E [ I 3 ] = 0 . Combining the {+expectations of I 1 , I 2 , I 
+}
we have [Equation] Au ]
[Equation]
where ζ
[Equation]
= [Equation] Denote the upper and lower bound of ζ i independent of u as ζ ui and ζ li , respectively, i.e., ζ li ≤ ζ i ≤ ζ ui . Then there exists c such that
[Equation]
From Proposition 2.6.1 in [46], we know that for a fixed vector u ∈ R n , for some constant c and any i , given the σ 2 x -subGaussian variable z , the random variable u ⊤ z i is c ∥ u
∥ 2 σ 2 x -subGaussian. Thus, ( u ⊤ z i ) 2 is subexponential. Based on Lemma 14, for a fixed unit vector u , we have
[Equation]
Let N be a 14 -net on S N − 1 with |N| ≤ 9 N . Based on the ϵ -net argument and union bound over every u ∈ N , we have with probability at least 1 − 2 e − t that
[Equation] ui where c 5 N ≥ t + N ln 9 . Therefore, there is a universal constant c such that with probability at least 1 − 2 e − N/c , d (cid:88) i =
[Equation]
Since 1 M (cid:80) Mm =1 ¯ λ 2 mi = µ i ( W M ) , substituting A with A or A k , then if r k ( W M ) ≥ bN , we have [Equation] This completes the proof.
Lemma 10. (Lemma 20 in [5]) Suppose k < n, A ∈ R n × n is an invertible matrix, and Z ∈ R n × k is such that ZZ ⊤ + A is invertible. Then
[Equation] Lemma 11 (Monotonicity of eigenvalues) . (Lemma 28 in [5]) If symmetric matrices A, B ∈ R n × n satisfy A ⪯ B , then, for any i ∈ [ n ] , we have µ i ( A ) ≤ µ i ( B ) .
Lemma 12.
Let z i ∈ R n , λ i ≥ 0 , ∀ i , if B ∈ R n × n and B = A + (cid:80) di =1 λ i z
i z ⊤ i , then B ⪰ A . Proof. B − A = zz ⊤ , therefore for any v ̸ = 0 , v ⊤ ( B − A ) v = (cid:80) di
{+=
 λ i ( v ⊤ z i ) 2 ≥ 0 , therefore B − A ⪰ 0 , B ⪰ A . Lemma 13 (Von Neumann’s trace inequality [33]) .+}
If A, B are n × n matrices with singular values, a 1 ≥ · · · ≥ a n , b 1 ≥ · · · ≥ b n respectively, then
[Equation]
Lemma 14. (Corollary 23 in [5]) There is a universal constant c such that for any non-increasing sequence { λ i } ∞ i =1 of non-negative numbers such that (cid:80) ∞ i =
λ i < ∞ , and any independent, centered, σ -subexponential random variables { ξ i } ∞ i =1 , and any t > 0 , with probability at least 1 − 2 e − t
[Equation]
Lemma 15. (Corollary 24 in [5]) Suppose z ∈ R n is a centered random vector with independent σ 2 -subGaussian entries with unit variances, L is a random subspace of R n of codimension k , and L is independent of z .
Then for some constant a and any t > 0 , with probability at least 1 − 3 e − t ,
[Equation]
where Π L is the orthogonal projection on L . D.3 Other intermediate results Lemma 16 (Expectation of ∆ Λ m ∆ ⊤ Λ m and ∆ ⊤ Λ m ∆ Λ m ) . The expectation of ∆ Λ m ∆ ⊤ Λ m and ∆ ⊤ Λ m ∆ Λ m over z tr m can be computed by
[Equation] Proof. First recall that ∆ Λ m ∈ R d × d , the element in the i -th row and j -th column in the matrix can be represented by
[Equation] When i = j
[Equation]
taking expectation of ∆ Λ m ∆ ⊤ Λ m leads to
[Equation]
when i ̸ = j , it follows that
[Equation]
Therefore, we can arrive at (81a).
Similarly, we can also get (81b).
This completes the proof.