T HEORETICAL C HARACTERIZATION OF THE G ENERALIZATION P ERFORMANCE OF O VERFITTED M ETA -L EARNING
A BSTRACT
Meta-learning has arisen as a successful method for improving training performance by training over many similar tasks, especially with deep neural networks (DNNs).
However, the theoretical understanding of when and why overparameterized models such as DNNs can generalize well in meta-learning is still limited.
As an initial step towards addressing this challenge, this paper studies the generalization performance of overfitted meta-learning under a linear regression model with Gaussian features.
In contrast to a few recent studies along the same line, our framework allows the number of model parameters to be arbitrarily larger than the number of features in the ground truth signal, and hence naturally captures the overparameterized regime in practical deep meta-learning.
We show that the overfitted min ℓ 2 -norm solution of model-agnostic meta-learning (MAML) can be beneficial, which is similar to the recent remarkable findings on “benign overfitting” and “double descent” phenomenon in the classical (single-task) linear regression.
However, due to the uniqueness of meta-learning such as task-specific gradient descent inner training and the diversity/fluctuation of the ground-truth signals among training tasks, we find new and interesting properties that do not exist in single-task linear regression.
We first provide a high-probability upper bound (under reasonable tightness) on the generalization error, where certain terms decrease when the number of features increases.
Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
Under this circumstance, we show that the overfitted min ℓ 2 -norm solution can achieve an even lower generalization error than the underparameterized solution.
I NTRODUCTION
Meta-learning is designed to learn a task by utilizing the training samples of many similar tasks, i.e., learning to learn (Thrun & Pratt, 1998).
With deep neural networks (DNNs), the success of meta-learning has been shown by many works using experiments, e.g., (Antoniou et al., 2018;
Finn et al., 2017).
However, theoretical results on why DNNs have a good generalization performance in meta-learning are still limited.
Although DNNs have so many parameters that can completely fit all training samples from all tasks, it is unclear why such an overfitted solution can still generalize well, which seems to defy the classical knowledge bias-variance-tradeoff (Bishop, 2006; Hastie et al., 2009; Stein, 1956; James & Stein, 1992; LeCun et al., 1991; Tikhonov, 1943).
The recent studies on the “benign overfitting” and “double-descent” phenomena in classical (singletask) linear regression have brought new insights on the generalization performance of overfitted solutions.
Specifically, “benign overfitting” and “double descent” describe the phenomenon that the test error descends again in the overparameterized regime in linear regression setup (Belkin et al., 2018; 2019; Bartlett et al., 2020;
Hastie et al., 2019; Muthukumar et al., 2019;
Ju et al., 2020; Mei & Montanari, 2019).
Depending on different settings, the shape and the properties of the descent curve of the test error can differ dramatically.
For example, Ju et al.
(2020) showed that the min ℓ 1 - norm overfitted solution has a very different descent curve compared with the min ℓ 2 -norm overfitted solution.
A more detailed review of this line of work can be found in Appendix A.
Compared to the classical (single-task) linear regression, model-agnostic meta-learning (MAML) (Finn et al., 2017; Finn, 2018), which is a popular algorithm for meta-learning, differs in many aspects.
First, the training process of MAML involves task-specific gradient descent inner training and outer training for all tasks.
Second, there are some new parameters to consider in meta-training, such as the number of tasks and the diversity/fluctuation of the ground truth of each training task.
These distinct parts imply that we cannot directly apply the existing analysis of “benign overfitting” and “double-descent” on single-task linear regression in meta-learning.
Thus, it is still unclear whether meta-learning also has a similar “double-descent” phenomenon, and if yes, how the shape of the descent curve of overfitted meta-learning is affected by system parameters.
A few recent works have studied the generalization performance of meta-learning.
In Bernacchia (2021), the expected value of the test error for the overfitted min ℓ 2 -norm solution of MAML is provided for the asymptotic regime where the number of features and the number of training samples go to infinity.
In Chen et al.
(2022), a high probability upper bound on the test error of a similar overfitted min ℓ 2 -norm solution is given in the non-asymptotic regime.
However, the eigenvalues of weight matrices that appeared in the bound of Chen et al.
(2022) are coupled with the system parameters such as the number of tasks, samples, and features, so that their bound is not fully expressed in terms of the scaling orders of those system parameters.
This makes it hard to explicitly characterize the shape of the double-descent or analyze the tightness of the bound.
In Huang et al.
(2022), the authors focus on the generalization error during the SGD process when the training error is not zero, which is different from our focus on the overfitted solutions that make the training error equal to zero (i.e., the interpolators).
(A more comprehensive introduction on related works can be found in Appendix A.)
All of these works let the number of model features in meta-learning equal to the number of true features, which cannot be used to analyze the shape of the double-descent curve that requires the number of features used in the learning model to change freely without affecting the ground truth (just like the setup used in many works on single-task linear regression, e.g., Belkin et al.
Ju et al.
To fill the gap, we study the generalization performance of overfitted meta-learning, especially in quantifying how the test error changes with the number of features.
As the initial step towards the
DNNs’ setup, we consider the overfitted min ℓ 2 -norm solution of MAML using a linear model with
Gaussian features.
We first quantify the error caused by the one-step gradient adaption for the test task, with which we provide useful insights on 1) practically choosing the step size of the test task and quantifying the gap with the optimal (but not practical) choice, and 2) how overparameterization can affect the noise error and task diversity error in the test task.
We then provide an explicit highprobability upper bound (under reasonable tightness) on the error caused by the meta-training for training tasks (we call this part “model error”) in the non-asymptotic regime where all parameters are finite.
With this upper bound and simulation results, we confirm the benign overfitting in metalearning by comparing the model error of the overfitted solution with the underfitted solution.
We further characterize some interesting properties of the descent curve.
For example, we show that the descent is easier to observe when the noise and task diversity are large, and sometimes has a descent floor.
Compared with the classical (single-task) linear regression where the double-descent phenomenon critically depends on non-zero noise, we show that meta-learning can still have the double-descent phenomenon even under zero noise as long as the task diversity is non-zero.
S YSTEM M ODEL
In this section, we introduce the system model of meta-learning along with the related symbols.
For the ease of reading, we also summarize our notations in Table 2 in Appendix B.
2. D ATA G ENERATION M ODEL
We adopt a meta-learning setup with linear tasks first studied in Bernacchia (2021) as well as a few recent follow-up works Chen et al.
; Huang et al.
However, in their formulation, the number of features and true model parameters are the same.
Such a setup fails to capture the prominent effect of overparameterized models in practice where the number of model parameters is much larger than the number of actual feature parameters.
Thus, in our setup, we introduce an additional parameter s to denote the number of true features and allow s to be different and much smaller than the number p of model parameters to capture the effect of overparameterization.
In this way, we can fix s and investigate how p affects the generalization performance.
We believe this setup is closer to reality because determining the number of features to learn is controllable, while the number of actual features of the ground truth is fixed and not controllable.
We consider m training tasks.
For the i -th training task (where i = 1 , 2 , · · · , m ), the ground truth is a linear model represented by y = x
Ts w ( i ) s + ϵ , where s denotes the number of features, x s ∈ R s is the underlying features, ϵ ∈ R denotes the noise, and y ∈ R denotes the output.
When we collect the features of data, since we do not know what are the true features, we usually choose more features than s , i.e., choose p features where p ≥ s to make sure that these p features include all s true features.
For analysis, without loss of generality, we let the first s features of all p features be the true features.
Therefore, although the ground truth model only has s features, we can alternatively expressed it with p features as y =
x
T w ( i ) + ϵ where the first s elements of x ∈ R p is x s and w ( i ) :
= (cid:104) w ( i ) s 0 (cid:
) ∈ R p .
The collected data are split into two parts with n t training data and n v validation data.
With those notations, we write the data generation model into matrix equations as
[Equation]
where each column of X t ( i ) ∈ R p × n t corresponds to the input of each training sample, each column of X v ( i ) ∈
R p × n v corresponds to the input of each validation sample, y t ( i
) ∈ R n t denotes the output of all training samples, y v ( i )
∈ R n v denotes the output of all validation samples, ϵ t ( i ) ∈ R n t denotes the noise in training samples, and ϵ v ( i ) denotes the noise invalidation samples.
Similarly to the training tasks, we denote the ground truth of the test task by w rs ∈ R s , and thus y =
x
Ts w rs .
Let w r
=
(cid:2)
w rs 0 (cid:3) ∈ R p .
Let n r denote the number of training samples for the test task, and let each column of X r ∈ R p × n r denote the input of each training sample.
Similar to Eq.
(1), we then have
[Equation]
where ϵ r ∈ R n r denotes the noise and each element of y r ∈ R n r corresponds to the output of each training sample.
In order to simplify the theoretical analysis, we adopt the following two assumptions.
Assumptionis commonly taken in the theoretical study of the generalization performance, e.g., Ju et al.
Bernacchia (2021).
Assumption 2 is less restrictive (no requirement to be any specific distribution) and a similar one is also used in Bernacchia (2021).
Assumption 1 (Gaussian features and noise) .
We adopt i.i.d.
Gaussian features x ∼ N (0 , I p ) and assume i.i.d.
Gaussian noise.
We use σ and σ r to denote the standard deviation of the noise for training tasks and the test task, respectively.
In other words, ϵ t ( i ) ∼ N ( 0 , σ 2 I n t ) , ϵ v ( i ) ∼
N ( 0 , σ 2 I n v ) for all i = 1 , · · · , m , and ϵ r ∼ N ( 0 , σ 2 r I n r ) .
Assumption 2 (Diversity/fluctuation of unbiased ground truth) .
The ground truth w rs and w ( i ) s for all
i = 1 , 2 , · · · , m share the same mean w 0 s , i.e., E [ w ( i ) s ]
= w 0 s =
E [ w rs ] .
For the i -th training task, elements of the true model parameter w ( i ) s ∈ R s are independent, i.e.,
E [( w ( i ) s − w 0 s ) T ( w ( i ) s − w 0 s )]
= Λ ( i ) :
= diag (cid:0) ( ν ( i ) , 1 ) 2 , ( ν ( i ) , 2 ) 2 , · · · , ( ν ( i ) ,s ) 2 (cid:1) .
Let ν ( i ) :=
(cid:112)
Tr( Λ ( i ) ) , ν := (cid:113)(cid:80) mi
=1 ν 2( i ) /m , ν r := (cid:113) E ∥
w rs − w 0 ∥ 22 , and w 0 :
= (cid:104) w 0 s 0 (cid:
) ∈ R p .
2. MAML PROCESS
We consider the MAML algorithm
(Finn et al., 2017; Finn, 2018), the objective of which is to train a good initial model parameter among many tasks, which can adapt quickly to reach a desirable model parameter for a target task.
MAML generally consists of inner-loop training for each individual task and outer-loop training across multiple tasks.
To differentiate the ground truth parameters w ( i ) , we use ˆ ( · ) (e.g., ˆ w ( i ) ) to indicate that the parameter is the training result.
In the inner-loop training, the model parameter for every individual task is updated from a common meta parameter ˆ w .
Specifically, for the i -th training task ( i = 1 , . . .
, m ), its model parameter ˆ w ( i ) is updated via a one-step gradient descent of the loss function based on its training data X t ( i ) :
[Equation]
where α t ≥ 0 denotes the step size.
In the outer-loop training, the meta loss L meta is calculated based on the validation samples of all training tasks as follows:
[Equation]
where L (
[Equation] [Equation]
y
[Equation]
The common (i.e., meta) parameter ˆ w is then trained to minimize the meta loss L meta .
At the test stage, we use the test loss L r inner :
= 12 (cid:13)(cid:13)(cid:13)
y r − X rT ˆ w (cid:13)(cid:13)(cid:13) 2
and adapt the meta parameter ˆ w via a single gradient descent step to obtain a desirable model parameter ˆ w r , i.e., ˆ w r :
= ˆ w −
α r n r ∂ L r inner ∂ ˆ w where α r ≥ 0 denotes the step size.
The squared test error for any input x is given by
[Equation]
S
OLUTIONS OF MINIMIZING META LOSS
The meta loss in Eq.
(4) depends on the meta parameter ˆ w via the inner-loop loss in Eq.
It can be shown (see the details in Appendix C) that L meta can be expressed as:
[Equation]
where γ ∈ R ( mn v ) × 1 and B ∈ R ( mn v ) × p are respectively stacks of m vectors and matrices given by
[Equation]
[Equation]
By observing Eq.
(6) and the structure of B , we know that min ˆ w L meta has a unique solution almost surely when the learning model is underparameterized , i.e., p ≤ mn v .
However, in the real-world application of meta learning, an overparameterized model is of more interest due to the success of the DNNs.
Therefore, in the rest of this paper, we mainly focus on the overparameterized situation, i.e., when p > mn v (so the meta training loss can decrease to zero).
In this case, there exist (almost surely) numerous ˆ w that make the meta loss become zero, i.e., interpolators of training samples.
Among all overfitted solutions, we are particularly interested in the min ℓ 2 -norm solution, since it corresponds to the solution of gradient descent that starts at zero in a linear model.
Specially, the min ℓ 2 -norm overfitted solution ˆ w ℓ 2 is defined as
[Equation]
In this paper, we focus on quantifying the generalization performance of this min ℓ 2 -norm solution with the metric in Eq.
M AIN R ESULTS
To analyze the generalization performance, we first decouple the overall test error into two parts: i) the error caused by the one-step gradient adaption for the test task, and ii) the error caused by the meta training for the training tasks.
The following lemma quantifies such decomposition.
Lemma 1.
With Assumptions 1 and 2, for any learning result ˆ w (i.e., regardless how we train ˆ w ), the expected squared test error is
[Equation]
where f test
[Equation]
Notice that in the meta-learning phase, the ideal situation is that the learned meta parameter ˆ w perfectly matches the mean w 0 of true parameters.
Thus, the term ∥
ˆ w − w
∥ 22 characterizes how well the meta-training goes.
The rest of the terms in the Lemma 1 then characterize the effect of the one-step training for the test task.
The proof of Lemma 1 is in Appendix H. Note that the expression of f test ( ζ ) coincides with Eq.
(65) of Bernacchia (2021).
However, Bernacchia (2021) uses a different setup and does not analyze its implication as what we will do in the rest of this section.
3. UNDERSTANDING THE TEST ERROR
Proposition 1.
We have
[Equation]
Further, by letting α r = n r n r + p +1 (which is optimal when σ r = 0 ), we have
[Equation]
The derivation of Proposition 1 is in Appendix H.1.
Some insights from Proposition 1 are as follows.
1) Optimal α r does not help much when overparameterized.
For meta-learning, the number of training samples n r for the test task is usually small (otherwise there is no need to do meta-learning).
Therefore, in Proposition 1, when overparameterized (i.e., p is relatively large), the coefficient p + n r + p +1 of ( ∥ ˆ w − w 0 ∥
22 + ν 2 r ) is close to 1 .
On the other hand, the upper bound ∥
ˆ w − w 0 ∥
22 + ν 2 r can be achieved by letting α r = 0 , which implies that the effect of optimally choosing α r is limited under this circumstance.
Further, calculating the optimal α r requires the precise values of ∥ ˆ w − w 0 ∥ 22 , ν 2 r , and σ 2 r .
However, those values are usually impossible/hard to get beforehand.
Hence, we next investigate how to choose an easy-to-obtain α r .
2) Choosing α r = n r / ( n r + p + 1) is practical and good enough when overparameterized.
Choosing α r = n r / ( n r + p + 1) is practical since n r and p are known.
By Proposition 1, the gap between choosing α r = n r / ( n r + p + 1) and choosing optimal α r is at most
[Equation]
When p increases, this gap will decrease to zero.
In other words, when heavily overparameterized, choosing α r = n r / ( n r + p + 1) is good enough.
3) Overparameterization can reduce the noise error to zero but cannot diminish the task diversity error to zero.
In the expression of f test (cid:16) ∥ ˆ w − w 0 ∥ 22 (cid:17) in Lemma 1, there are two parts related to the test task: the noise error (the term for σ r ) and the task diversity error (the term for ν r ).
By
Proposition 1, even if we choose the optimal α r , the term of ν 2 r will not diminish to zero when p increases.
In contrast, by letting α r = n r / ( n r + p + 1) , the noise term n r p ( n r + p +1) 2 σ 2 r ≤ n r p σ 2 r will diminish to zero when p increases to infinity.
3. CHARACTERIZATION OF MODEL ERROR
Since we already have Lemma 1, to estimate the generalization error, it only remains to estimate ∥ ˆ w − w 0 ∥ 2 2 , which we refer as model error .
The following Theorem 1 gives a high probability upper bound on the model error.
Theorem 1.
Under Assumptions 1 and 2, when min { p, n t } ≥ 256 , we must have
[Equation]
where b w : =
b w 0 + b ideal w and η : =
[Equation]
The value of b w 0 and b ideal w are completely determined by the finite (i.e., non-asymptotic region) system parameters p, s, m, n t , n v , ∥ w 0 ∥ 22 , ν 2 , σ 2 , and α t .
The precise expression will be given in
Section 4, along with the proof sketch of Theorem 1.
Notice that although b w is only an upper bound, we will show in Section 4 that each component of this upper bound is relatively tight.
Theorem 1 differs from the result in Bernacchia (2021) in two main aspects.
First, Theorem 1 works in the non-asymptotic region where p and n t are both finite, whereas their result holds only in the asymptotic regime where p, n t → ∞ .
In general, a non-asymptotic result is more powerful than an asymptotic result in order to understand and characterize how the generalization performance changes as the model becomes more overparameterized.
Second, the nature of our bound is in high probability with respect to the training and validation data, which is much stronger than their result in expectation.
Due to the above key differences, our derivation of the bound is very different and much harder.
As we will show in Section 4, the detailed expressions of b w 0 and b ideal w are complicated.
In order to derive some useful interpretations, we provide a simpler form by approximation 1 for an overparameterized regime, where α t p ≪ 1 and min { p, n t } ≫ mn v .
In such a regime, we have
[Equation]
where b δ ≈ mn v (cid:16) (1 + C 1 n t ) σ 2 + C 2 (1 + C 3 n t ) ν 2 (cid:17) , and C 1 to C 4 are some constants.
It turns out that a number of interesting insights can be obtained from the simplified bounds above.
1) Heavier overparameterization reduces the negative effects of noise and task diversity.
From Eq.
(9), we know that when p increases, b ideal w decreases
to zero.
Notice that b ideal w is the only term that is related to σ 2 and ν 2 , i.e., b ideal w corresponds to the negative effect of noise and task diversity/fluctuation.
Therefore, we can conclude that using more features/parameters can reduce the negative effect of noise and task diversity/fluctuation.
Interestingly, b ideal w can be interpreted as the model error for the ideal interpolator ˆ w ideal defined as ˆ
[Equation]
[Equation]
arg
[Equation]
subject to
B ˆ w = γ.
Differently from the min ℓ 2 -norm overfitted solution in Eq.
(8) that minimizes the norm of ˆ w , the ideal interpolator minimizes the distance between ˆ w and w 0 , i.e., the model error (this is why we define it as the ideal interpolator).
The following proposition states that b ideal w corresponds to the model error of ˆ w ideal .
Proposition 2.
When min { p, n t } ≥ 256 , we must have
[Equation]
2) Overfitting is beneficial to reduce model error for the ideal interpolator.
Although calculating ˆ w ideal is not practical since it needs to know the value of w 0 , we can still use it as a benchmark that describes the best performance among all overfitted solutions.
From the previous analysis, we have already shown that b ideal w → 0 when p → ∞ , i.e., the model error of the ideal interpolator decreases to 0 when the number of features grows.
Thus, we can conclude that overfitting is beneficial to reduce the model error for the ideal interpolator.
This can be viewed as evidence that overfitting itself should not always be viewed negatively, which is consistent with the success of DNNs in meta-learning.
The descent curve is easier to observe under large noise and task diversity, and the curve sometimes has a descent floor.
In Eq.
(9), when p increases, b w 0 increases but b ideal w decreases.
When σ and ν becomes larger, b ideal w becomes larger while b w 0 does not change, so that the descent of b ideal w contributes more to the trend of b w = b w 0 + b ideal w (i.e., the overall model error).
By further calculating the derivative of b w with respect to p (see details in Appendix D), we observe that, if g ≥ 1 , then b w always decreases for p > C 4 mn v .
If g :=
[Equation]
mn
[Equation]
∥ w 0 ∥ 22 < 1 , then b w decreases
101010 3 p 101010m o d e l e rr o r (a)    38 p 101010(b)
ν=60,σ=
ν=10,σ= ν=0.5,σ=0.5 ν=0.3,σ=0. ν=0.1,σ=0.
null risk when p ∈ ( C 4 mn v , C 4 mn v 1 −√ g ) , and increases when p > C 4 mn v 1 −√ g .
Thus, the minimal value (i.e., the floor value) of b w is equal to ∥ w 0 ∥ 22 (cid:16) 1 − (1 −√ g ) 2 C(cid:17) , which is achieved at p =
C 4 mn v 1 −√ g .
This implies that the descent curve of the model error has a floor only when b δ is small, i.e., when the noise and task diversity are small.
Notice that the threshold C 4 mn v 1 −√ g and the floor value ∥ w 0 ∥ 22 (cid:16) 1 − (1 −√ g ) 2 C(cid:17) increase as g increases.
Therefore, we anticipate that as ν and σ increase, the descent floor value and its location both increase.
In Fig.
1, we draw the curve of the model error with respect to p for the min ℓ 2 -norm solution.
In subfigure (a), the blue curve (with the marker “ + ”) and the yellow curve (with the marker “ × ”) have relatively large ν and/or σ .
These two curves always decrease in the overparameterized region p > mn v and have no descent floor.
In contrast, the rest three curves (purple, red, green) in subfigure (a) have descent floor since they have relatively small ν and σ .
Subfigure (b) shows the location and the value of the descent floor.
As we can see, when ν and σ increase, the descent floor becomes higher and locates at larger p .
These observations are consistent with our theoretical analysis.
In Appendix E.2, we provide a further experiment where we train a two-layer fully connected neural network over the MNIST data set.
We observe that a descent floor still occurs.
Readers can find more details about the experiment in Appendix E.2.
4) Task diversity yields double descent under zero noise.
For a single-task classical linear regression model y =
x
T w 0 + ϵ , the authors of Belkin et al.
(2020) study the overfitted min ℓ 2 -norm solutions w ℓ 2 , single learned by interpolating n training samples with p ≥ n + 2 i.i.d.
Gaussian features.
The result in Belkin et al.
(2020) shows that its expected model error is
[Equation]
We find that both meta-learning and the single-task regression have similar bias terms p − mn v p ∥ w 0 ∥and p − np ∥ w 0 ∥ 22 , respectively.
When p → ∞ , these bias terms increase to ∥ w 0 ∥ 22 , which corresponds to the null risk (the error of a model that always predicts zero).
For the single-task regression, the remaining term
[Equation]
2 when p ≫ n , which contributes to the descent of generalization error when p increases.
On the other hand, if there is no noise ( σ = 0 ), then the benign overfitting/double descent will disappear for the single-task regression.
In contrast, for meta-learning, the term that contributes to benign overfitting is b ideal w .
As we can see in the expression of b ideal w in Eq.
(9), even if the noise is zero, as long as there exists task diversity/fluctuation (i.e., ν > 0 ), the descent of the model error with respect to p should still exist.
This is also confirmed by Fig. 1(a) with the descent blue curve (with marker “+”) of ν = 60 and σ = 0 .
5) Overfitted solution can generalize better than underfitted solution.
Let ˆ w ( p =1) ℓdenote the solution when p = s
= 1 .
We have
[Equation]
The derivation is in Appendix O. Notice that p = s means that all features are true features, which is ideal for an underparameterized solution.
Compared to the model error of overfitted solution, there is no bias term of w 0 , but the terms of ν 2 and σ 2 are only discounted by m and n v .
In contrast, for the overfitted solution when p ≫ mn v , the terms of ν 2 and σ 2 are discounted by p .
Since the value of m and the value of n v are usually fixed or limited, while p can be chosen freely and made arbitrarily large, the overfitted solution can do much better to mitigate the negative effect caused by noise and task divergence.
This provides us a new insight that when σ , ν are large and ∥ w 0 ∥is small, the overfitted ℓ 2 -norm solution can have a much better overall generalization performance than the underparameterized solution.
This is further verified in Fig. 1(a) by the descent blue curve (with marker “+”), where the first point of this curve with p =
= s (in the underparameterized regime) has a larger test error than its last point with p = 1000 (in the overparameterized regime).
T IGHTNESS OF THE BOUND AND ITS PROOF SKETCH
We now present the main ideas of proving Theorem 1 and theoretically explain why the bound in
Theorem 1 is reasonably tight by showing each part of the bound is tight.
(Numerical verification of the tightness is provided in Appendix E.1.)
The expressions of b w 0 and b ideal w (along with other related quantities) in Theorem 1 will be defined progressively as we sketch our proof.
Readers can also refer to the beginning of Appendix I for a full list of the definitions of these quantities.
We first re-write ∥ w 0 − ˆ w ℓ 2 ∥ 22 into terms related to B .
When B is full row-rank (which holds almost surely when p > mn v ), we have
[Equation]
Define δγ as
[Equation]
Doing some algebra transformations (details in Appendix F), we have
[Equation]
To estimate the model error, we take a divide-and-conquer strategy and provide a set of propositions as follows to estimate Term 1 and Term 2 in Eq.
Proposition 3.
Define
[Equation]
When p ≥ mn v and p ≥ 16 , We then have
Pr[ Term 1 of Eq.
(12) ≤ b w 0 ] ≥ 1 − 2 /p,
Pr[ Term 1 of Eq.
[Equation]
≥ ˜
[Equation]
Proposition 3 gives three estimates on Term 1 of Eq.
(12): the upper bound b w 0 , lower bound ˜ b w 0 , and the mean value.
If we omit all logarithm terms, then these three estimates are the same, which implies that our estimation on Term 1 of Eq.
(12) is fairly precise.
(Proof of Proposition 3 is in
Appendix J.)
It then remains to estimate Term 2 in Eq.
Indeed, this term is also the model error of the ideal interpolator.
To see this, since B ˆ w = γ , we have B ( ˆ w − w 0 )
= γ − B w 0 = δγ .
Thus, to get min ∥
ˆ w − w
∥
as the ideal interpolator, we have ˆ w ideal − w
= B T ( BB T ) − 1 δγ .
Therefore, we have
[Equation]
Now we focus on (cid:13)(cid:13) B T ( BB T ) − 1 δγ (cid:13)(cid:13) 2 2 .
By Lemma 6 in Appendix G.2, we have
[Equation]
We have the following results about the eigenvalues of BB T .
Proposition 4.
Define α ′ t :=
α t n t (cid:0) √ p + √ n t + ln √ n t (cid:1) 2 and
[Equation]
eig ,
[Equation]
p.
When p ≥ n t ≥ 256 , we must have
[Equation]
Proposition 5.
Define
[Equation]
When n t ≥ p ≥ 256 , we have
[Equation]
To see how the upper and lower bounds of the eigenvalues of BB T match, consider α t p ≪ 1 , which implies α ′ t ≪
[Equation]
and the fact that √ p and ln p are lower order terms than p , then each of b eig , min , b eig , max , c eig , min , c eig , max can be approximated by p ± ˜
Cmn v for some constant ˜ C .
Further, when p ≫ mn v , all b eig , min , b eig , max , c eig , min , c eig , max can be approximated by p , i.e., the upper and lower bounds of the eigenvalues of BB T match.
Therefore, our estimation on λ max ( BB T ) and λ min ( BB T ) in Proposition 4 and Proposition 5 are fairly tight.
(Proposition 4 is proved in
Appendix L, and Proposition 5 is proved in Appendix M.) From Eq.
(14), it remains to estimate ∥ δγ ∥ 22 .
Proposition 6.
Define
[Equation]
When min { p, n t } ≥ 256 , we must have
[Equation]
(cid:111) ≥ 1 − 5 mn v n t − 2 mn v p 0 .
We also have E ∥ δγ ∥ 22 =
mn v σ 2 (cid:16) 1 + α 2 t p n t (cid:17) + ν 2 mn v (cid:16) (1 − α t )
+ α 2 t ( p +1)
n t (cid:17) , where the expectation is on all random variables.
Proposition 6 provides an upper bound b δ on ∥ δγ ∥ 22 and an explicit form for E ∥ δγ ∥ 22 .
By comparing b δ and E ∥ δγ ∥ 2 2 , the differences are only some coefficients and logarithm terms.
Thus, the estimation on ∥ δγ ∥ 22 in Proposition 6 is fairly tight.
Proposition 6 is proved in Appendix N.
Combining Eq.
(13), Eq.
(14), Proposition 4, Proposition 5, and Proposition 6, we can get the result of Proposition 2 by the union bound, where b ideal w
[Equation]
The detailed proof is in Appendix K.
Then, by Eq.
(13), Proposition 2, Proposition 3, and Eq.
(12), we can get a high probability upper bound on ∥ w 0 − ˆ w ℓ 2 ∥ 22 , i.e., Theorem 1.
The detailed proof of Theorem 1 is in Appendix I. Notice that we can easily plug our estimation of the model error (Proposition 2 and Theorem 1) into Lemma 1 to get an estimation of the overall test error defined in
Eq.
(5), which is omitted in this paper by the limit of space.
C ONCLUSION
We study the generalization performance of overfitted meta-learning under a linear model with Gaussian features.
We characterize the descent curve of the model error for the overfitted min ℓ 2 -norm solution and show the differences compared with the underfitted meta-learning and overfitted classical (single-task) linear regression.
Possible future directions include relaxing the assumptions and extending the result to other models related to DNNs (e.g., neural tangent kernel (NTK) models).
A CKNOWLEDGEMENT
The work of P. Ju and N. Shroff has been partly supported by the NSF grants NSF AI Institute (AIEDGE) CNS-2112471, CNS-2106933, 2007231, CNS- and CNS-1901057, and in part by Army Research Office under Grant W911NF-21-1-0244.
The work of Y. Liang has been partly supported by the NSF grants NSF AI Institute (AI-EDGE) CNS-2112471 and DMS-2134145.
A R ELATED WORK
Our work is related to recent studies on characterizing the double-descent phenomenon for overfitted solutions of single-task linear regression.
Some works study the min ℓ 2 -norm solutions for linear regression with simple features such as Gaussian or Fourier features (Belkin et al., 2018; 2019;
Bartlett et al., 2020; Hastie et al., 2019; Muthukumar et al., 2019), where they show the existence of the double-descent phenomenon.
Some others (Mitra, 2019; Ju et al., 2020) study the min ℓ 1norm overfitted solution and show it also has the double-descent phenomenon, but with a descent curve whose shape is very different from that of the min ℓ 2 -norm solution.
Some recent works study the generalization performance when overparameterization in random feature (RF) models (Mei & Montanari, 2019), two-layer neural tangent kernel (NTK) models (Arora et al., 2019; Satpathi & Srikant, 2021;
Ju et al., 2021), and three-layer NTK models (Ju et al., 2022).
(RF and NTK models are linear approximations of shallow but wide neural networks.)
These works show the benign overfitting for a certain set of learnable ground truth functions that depends on the network structure and which layer to train.
However, all those works are on single-task learning, which cannot be directly used to characterize the generalization performance of meta-learning due to their differences in many aspects mentioned in the introduction.
Our work is related to a few recent works on the generalization performance of meta-learning.
In
Bernacchia (2021), the expected value of the test error for the overfitted min ℓ 2 -norm solution of
MAML is provided.
However, the result only works in the asymptotic regime where the number of features and the number of training samples go to infinity, which is different from ours where all quantities are finite.
In Chen et al.
(2022), a high probability upper bound on the test error of a similar overfitted min ℓ 2 -norm solution is given in the non-asymptotic regime.
A significant difference between ours and Chen et al.
(2022) is that our theoretical bound describes the shape of double descent in a straightforward manner (i.e., our theoretical bound consists of decoupled system parameters).
In contrast, the bound in Chen et al.
(2022) contains coupled parts (e.g., eigenvalues of weight matrices are affected by the number of tasks, samples, and features), which may not be directly used to analyze the shape of double descent.
Besides, we also explain and verify the tightness of the bound, while Chen et al.
(2022) does not 2 .
In Huang et al.
(2022), the authors focus on the generalization error during the SGD process when the training error is not zero, which is different from our focus on the overfitted solutions that make the training error equal to zero (i.e., the interpolators).
Our work also differs from Bernacchia (2021);
Chen et al.
; Huang et al.
(2022) in the data generation model for the purpose of quantifying how the number of features affects the test error, which has been explained in detail at the beginning of Section 2.1.
[Table]
B N OTATION T ABLE
We summarize the important notations in Table 2.
[Table]
Meaning
Type
[Table]
integer
[Table]
integer
[Table]
integer
[Table]
vector R p
[Table]
vector R p
[Table]
scalar
[Table]
vector R s
[Table]
vector R s
[Table]
vector R p
[Table]
vector R s
[Table]
vector R p
[Table]
vector R s
[Table]
vector R p
[Table]
vector R p
[Table]
vector R p
[Table]
vector R p
[Table]
vector R p
[Table]
vector R p
[Table]
vector R p
[Table]
integer
[Table]
integer
[Table]
integer
[Table]
the matrix formed by n
[Table]
training inputs of the i -th task matrix R p × n t
[Table]
the output vector corresponding to X
[Table]
( i )
vector R n t
[Table]
the noise in y
[Table]
( i )
vector R n t
[Table]
the matrix formed by n v validation inputs of the i -th task matrix R p × n v
[Table]
C T HE CALCULATION OF META L OSS
By the definition of L ( i ) inner in Eq.
(3), we have
[Equation]
Plugging it into Eq.
(3), we thus have
[Equation]
Plugging Eq.
(15) into Eq.
(4), we thus have
[Equation]
By the definition of B in Eq.
(7) and the definition of γ in Eq.
(7), we thus have
L meta
[Equation] [Equation]
Eq.
(6) thus follows.
D C ALCULATION OF DESCENT FLOOR
Define h ( p ) :
= p − mn v p
[Equation]
δ p − C 4 mn v where p > C 4 mn v .
We have
[Equation]
(recall that g :=
b δ mn v ∥ w 0 ∥ 22 ) .
Notice that when p > C 4 mn v and w 0 ̸ = 0 , the first and the second factors are positive.
Thus, we only need to consider the sign of A := (cid:16) 1 − C 4 mn v p − √ g (cid:17) .
If g ≥ 1 , then A < 0 , which implies that h ( p ) is monotone decreasing.
If g < 1 , then we have
A (cid:40) < 0 , when p ∈ (cid:16) C 4 mn v , C 4 mn v 1 −√ g (cid:17) , > 0 , when p > C 4 mn v 1 − √ g ,
[Figure]
d
[Figure]
m o
d e l e rr o r (b) ν
= 10, σ =experiment theory null risk
[Figure]
which implies that h ( p ) is monotone decreasing when p ∈ (cid:16) C 4 mn v , C 4 mn v 1 −√ g (cid:17) , and is monotone increasing when p > C 4 mn v 1 −√ g .
This suggests a descent floor at p =
C 4 mn v 1 −√ g , where at this point h ( p
[Equation]
E A DDITIONAL E XPERIMENTS E. S IMULATIONS TO VERIFY THE TIGHTNESS OF T HEORY
In Fig.
2, we plot both the experimental values (denoted by separated markers) and the theoretical values (denoted by the continuous curve) of the model error.
The simulation setup and (consequently)
the experimental values are the same as those in Fig.
1 (note that we only show the points in the overparameterized region, i.e., p ≥ mn v ).
The theoretical value is calculated by Theorem
[Figure]
t e s t
e rr o r and the approximation Eq.
(9) with constants 4 C 1 = C 3 = 0 .
001 and C 2 =
C 4 = 0 .
As we can see in Fig.
2, the experimental value points closely match the theoretical curves, which suggests that Theorem 1 and Eq.
(9) are fairly tight.
E. E XPERIMENTS OF META - LEARNING WITH N EURAL N ETWORK ON R EAL -W ORLD D ATA
In this section, we further verify our theoretical findings by an experiment over a two-layer fullyconnected neural network on the MNIST data set.
Neural network structure: The input dimension of the neural network is 49 (i.e., 7 × 7 gray-scale image shrunk from the original 28 × 28 gray-scale image).
The input is multiplied by the firstlayer weights and fully connected to the hidden layer that consists of ReLUs (rectified linear units max { 0 , ·} ) with bias.
The output of these ReLUs is then multiplied by the second layer weights and then goes to the output layer.
The output layer is a sigmoid activation function with bias.
Experimental setup: There are 4 training tasks and 1 test task.
The objective of each task is to identify whether the input image belongs to a set of 5 different digits.
Specifically, the sets fortraining tasks are { 1 , 2 , 3 , 7 , 9 } , { 0 , 2 , 3 , 6 , 9 } , { 0 , 1 , 4 , 6 , 8 } , { 1 , 2 , 3 , 5 , 7 } , respectively.
The set for the test task is { 0 , 2 , 3 , 6 , 8 } .
For each training task, there are 1000 training samples andvalidation samples.
All samples (except the ones to calculate the test error) are corrupted by i.i.d.
Gaussian noise in each pixel with zero mean and standard deviation 0 .
The number of samples for the one-step gradient is 1000 for these 4 training tasks and is 500 for the test task, i.e., n t =and n r = 500 .
The number of validation samples is n v = 100 for each of these 4 training tasks.
The initial weights are uniformly randomly chosen in the range [0 , 1] .
Training process: We use gradient descent to train the neural network.
The step size in the outerloop training is 0 .
3 and the step size of the one-step gradient adaptation is α t = α r = 0 . 05 .
After training 500 epochs, the meta-training error for each simulation is lower than 0 .
025 (the range of the meta-training error is [0 , 1] ), which means that the trained model almost completely fits all validation samples.
Simulation results and interpretation: We run the simulation 30 times with different random seeds.
In Fig.
3, we draw a box plot showing the test error for the test task, where the blue curve denotes the average value of these 30 runs.
We can see that the overall trend of the test curve is decreasing, which suggests that more parameters help to enhance the overfitted generalization performance.
Another interesting phenomenon we find from Fig.
3 is that the curve is not strictly monotonically decreasing and there exist some decent floors (e
[Equation]
g., when the network width is 80 ).
F D ETAILS OF DERIVING E Q . (12)
Notice that
[Equation]
We thus have
[Equation]
2(cid:124) (cid:123)(cid:122) (cid:125) Term(by Eq.
G S OME U SEFUL L EMMAS G. E STIMATION ON LOGARITHM
Lemma 2.
For any x > 0 , we must have ln x ≥ 1 − 1 x .
Proof.
This can be derived by examining the monotonicity of ln x − (1 − 1 x ) .
The complete proof can be found, e.g., in Lemma 33 of Ju et al.
Lemma 3.
When k ≥ 16 , we must have 2 (cid:113) ln kk < 1 .
Proof.
We only need to prove that the function g ( k ) :
= 4 ln k − k ≤ 0 when k ≥ 16 .
To that end, when k ≥ 16 , we have
[Equation]
Thus, g ( k ) is monotone decreasing when k ≥ 16 .
Also notice that g (16) =
4 ln 16 − 16 ≈ − 4 .
91 < 0 .
The result of this lemma thus follows.
G. E STIMATION ON NORM AND EIGENVALUES
Let λ min ( · ) and λ min ( · ) denote the minimum and maximum singular value of a matrix.
Lemma 4.
Consider an orthonormal basis matrix U ∈ R k × k and any vector a ∈ R k × 1 .
Then we must have
[Equation]
Proof.
We have
[Equation]
where U i is the i -th column of U and a i is the i -th element of a .
Because U is an orthonormal basis matrix, we know that U i are orthogonal to each other.
In other words,
[Equation]
Thus, we have
[Equation]
The result of this lemma thus follows.
Lemma 5.
Consider a diagonal matrix D = diag ( d 1 , d 2 , ·
· · , d k ) ∈
R k × k where min i d i ≥ 0 .
For any vector a , we must have
[Equation]
Proof.
We have
[Equation]
The result of this lemma thus follows.
Lemma 6.
For any A ∈ R q × k and a ∈ R k × 1 , we must have
[Equation]
Proof.
Do singular value decomposition of A as A = UDV T .
Here D ∈ R q × k is a diagonal matrix that consists of all singular values,
U ∈ R q × q and V ∈ R k × k (and their transpose) are orthonormal basis matrices.
We have
[Equation]
The result of this lemma thus follows.
The following lemma is useful to estimate the eigenvalues of a matrix whose off-diagonal elements are relatively small.
Lemma 7 (Gershgorin’s Circle Theorem (Marquis et al., 2016)) .
If A is an n × n matrix with complex entries a i,j then r i ( A )
=
(cid:80)
j ̸
= i | a i,j | is defined as the sum of the magnitudes of the non-diagonal entries of the i -th row.
Then a Gershgorin disc is the disc D ( a i,i , r i ( A )) centered at a i,i on the complex plane with radius r i
( A ) .
Theorem: Every eigenvalue of a matrix lies within at least one
Gershgorin disc.
G. E STIMATION ON RANDOM VARIABLES OF CERTAIN DISTRIBUTIONS
Lemma 8 (Corollary 5.35 of Vershynin (2010)) .
Let A be an N 1 × N 2 matrix ( N 1 × N 2 ) whose entries are independent standard normal random variables.
Then for every t ≥ 0 , with probability at least 1 − 2 exp( − t 2 / 2) one has
[Equation]
Lemma 9 (stated on pp.
1325 of Laurent & Massart (2000)) .
Let U follow χ 2 distribution with D degrees of freedom.
For any positive x, we have
[Equation]
Lemma 10 (Lemma 31 of Ju et al.
Let u 1 , u 2 , · · · , u k and u 1 , u 2 , · · · , u k denote 2 k random variables that follow i.i.d.
standard normal distribution.
For any a > 0 , we must have
[Equation]
where
[Equation]
Lemma 11.
For any k ≥ 16 , let u 1 , u 2 , · · · , u k and u 1 , u 2 , · · · , u k denote 2 k random variables that follows i.i.d c
, we must have
[Equation]
Further, by letting c = 1 and q = k ≥ 16 , we have
[Equation]
By letting q = e , we have
[Equation]
Proof.
Recall the definition of t in Lemma 10, we first want to prove at + ln 2 ta ≥ a 2 2( √ 1+ a 2 +1) .
To that end, we have at + ln
[Equation]
+ √ 1 + a 2 ) a(by the definition of t in Lemma 10) ≥ − 1 + (cid:112)
+ a 2 + 1 − a2( − 1 + √ 1 + a 2 ) (by Lemma 2)
=
a1 + √ 1 + a 2 + 1 − 1 + √ 1 + a (since a 2 = ( − 1 + (cid:112)
[Equation]
Now we let a = 2 c · (cid:113) ln qk .
Thus, we have
[Equation]
We have
[Equation]
Because c ≥ 1 , we have
[Equation]
We thus have
[Equation]
a
[Equation]
ln
[Equation]
k and
[Equation] [Equation]
Thus
[Equation]
we have
[Equation]
Plugging Eq.
(20) and Eq.
(24) into Lemma 10, the result of this lemma thus follows.
Lemma 12 (Isserlis’ theorem (Michalowicz et al., 2009)) .
If ( x 1 , x 2 , · · · , x n ) is a zero-mean multivariate normal random vector, then
[Equation]
where A denotes a partition of 1 , 2 , · · · , n into pairs, and A 2 n denotes all such partitions.
For example,
[Equation]
The following lemma is mentioned in Bernacchia (2021) without a detailed proof.
For the ease of readers, we provide a detailed proof of this lemma here.
Lemma 13.
Consider a random matrix X ∈ R p × n whose each element follows i.i.d.
standard
Gaussian distribution (i.e., i.i.d.
N (0 , 1) ).
We mush have
[Equation]
Proof.
Since each row of X are i.i.d.
, we immediately have E [ XX T ] = n I p and E [ X T X ] =
p
I n .
It remains to prove E [ XX T XX T ] = n ( n + p + 1) I p .
To that end, we have
[Equation]
where X i denotes the i -th row of X , and [ · ] i,j denotes the element in the i -th row, j -th column of the matrix.
Thus, we have
[Equation]
Now we examine the value of E X i,l X k,l X j,l
′ X k,l ′ by Isserlis’ theorem (Lemma 12).
[Equation]
=    0 , when i ̸
= j, 0 , when i = j and k
̸
=
i and l ̸ = l ′ , 1 , when i = j and k ̸
=
i and l = l ′ (there are n ( p − 1) such terms for each i ) , 1 , when i =
j = k and l ̸
= l ′ (there are n ( n − 1) such terms for each i ) , 3 , when i =
j = k and l = l ′ (there are n such terms for each i ) .
By Eq.
(26), we thus have
[Equation]
The result of this lemma thus follows.
Lemma 14 (Lemma 24 of Ju et al.
Considering a standard Gaussian distribution a ∼
N (0 , 1) , when t ≥ 0 , we have
[Equation]
Notice that t + (cid:113) t 2 + 8 π ≥ 2 (cid:113) 2 π when t ≥ 0 .
We thus have
Pr { a ≥
[Equation] [Equation]
e − t 2 / 2 .
By the symmetry of the standard Gaussian distribution, we thus have
Pr {
[Equation]
H P ROOF OF L EMMA
Proof.
Similar to Eq.
(15), we have
[Equation]
By Eq.
(27), we can express the learned result for the test task as
[Equation]
Considering a new input x for the test task, the ground-truth is x T w r .
The distance between the ground-truth and the output of the learned model is
[Equation]
Taking the expectation on x for the square of the distance, we have
[Equation]
Assumption 1)
[Equation]
Notice that
[Equation]
Then, taking the expectation on ϵ r , we have
[Equation]
(since ϵ r is independent of other random variables)
[Equation]
Notice that
[Equation]
The result of this proposition thus follows by plugging Eq.
(30) into Eq.
H. P ROOF OF P ROPOSITION
Proof.
For ease of notation, we let K :=
∥
ˆ w − w
∥
+ ν 2 r .
From Lemma 1, we can see that f test (cid:16) ∥ ˆ w − w 0 ∥ 22 (cid:17) is a quadratic function of α r :
[Equation]
Thus, to minimize the test loss shown by Lemma 1, we can calculate the optimal choice of α r as
[Equation]
(Thus, we have [ α r ] opti = n r n r + p +1 when σ r = 0 .)
Plugging [ α r ] opti into f test (cid:16)
∥
ˆ w − w 0 ∥ 22 (cid:17) , we thus have f test
[Equation]
The right-hand side of Eq.
(31) increases when σ 2 r increases.
Therefore, by letting σ r = 0 and σ r = ∞ , we can get the lower and upper bound of Eq.
(31), i.e.,
[Equation]
The result of this lemma thus follows.
I P ROOF OF T HEOREM
We summarize the definition of the quantities related to the definition of b w as follows:
[Equation]
Proof.
Define the event in Proposition 2 as
[Equation]
Define the event in Proposition 3 as
A target , 3 := { Term 1 of Eq.
(12) ≤ b w 0 } .
By the definition of b w in Eq.
(32) and the definition of A target , 3 and A target , we have A target , 3 ∩
A target = ⇒ ∥
ˆ w ℓ 2 − w 0 ∥ 22 ≤ b w .
Thus, we have
Pr (cid:110)
[Equation] [Equation]
≥ Pr {A target
[Equation]
3 ∩ A target }
[Equation]
≥ 1 − Pr
[ A c target , 3 ] − Pr
[ A c target ] (by the union bound)
[Equation] [Equation]
[Equation]
(by Proposition 3 and Proposition 2) ≥
[Equation]
− 27 m
[Equation] [Equation]
The last inequality is because 2 p ≤ 1 p 0 .
4 since p 0 .
6 ≥ 256 0 .
6 ≈ 27 .
86 ≥ 2 .
The result of this theorem thus follows.
J P ROOF OF P ROPOSITION
Define P :=
B T ( BB T ) − 1 B .
In this section, we focus on estimating ∥ ( I p − P ) w 0 ∥ 22 .
Since
P 2 =
P , we know P is indeed a projection from R p to the subspace spanned by the columns of B T (i.e., the rows of B ).
The following Lemma 15 shows that the subspace spanned by the columns of
B T has rotational symmetry.
Then, we will use Lemma 16 to show how the rotational symmetry helps in estimating the expected value of the squared norm of the projected vector.
We then prove
Proposition 3 by utilizing Lemma 15 and Lemma 16.
Lemma 15.
The subspace spanned by the columns of B T has rotational symmetry (with respect to the randomness of X t (1: m ) and X v (1: m ) ).
Specifically, for any rotation S ∈ SO ( p ) where SO ( p ) ⊆
R p × p denotes the set of all rotations in p -dimensional space, the rotated random matrix SB T shares the same probability distribution with the original B T .
Proof.
Notice that for any i = 1 , 2 , · · · , m ,
[Equation]
(cid:16) SX
[Equation]
( i ) (cid:17)
[Equation]
(cid:16)
[Equation]
( i ) (cid:17) (cid:16) SX t ( i ) (cid:17) T (since S − 1 = S T because S is a rotation)
[Equation]
(cid:16)
[Equation] [Equation]
(cid:17) (cid:16) SX
[Equation]
( i ) (cid:17)
[Equation]
We thus have
[Equation]
Because of the rotational symmetry of Gaussian distribution
[Equation]
we know that the rotated random matrices SX v ( i ) and SX t ( i ) have the same probability distribution with the original random matrices
X v ( i ) and X t ( i ) , respectively.
Therefore, by Eq.
[Equation]
we can conclude that SB T has the same probability distribution as B T .
The result of this lemma thus follows.
The following lemma shows how to use rotational symmetry to calculate the expected value of the squared norm of the projected vector.
Such a result has also been used in literature (e.g., (Belkin et al., 2020)).
Lemma 16.
Considering any random projection P 0 ∈ R p × p to a k -dim subspace where the subspace has rotational symmetry, then for any given v ∈ R p × 1 we must have
[Equation]
Proof.
Since the subspace has rotational symmetry, to calculate the expected value, we can fix a subspace and integral over all rotations.
Specifically, consider any fixed projection A that projects to k -dim subspace that is spanned by a set of orthogonal vectors a 1 , a 2 , · · · , a k ∈ R p .
Therefore, after projecting v with A , the squared norm of the projected vector is equal to
[Equation]
Noticing that applying a rotation in the projected space of A is equivalent to applying the rotation to a 1 , a 2 , · · · , a k , we then have
[Equation]
(since making a rotation S − 1 on both vectors does not change the inner product value)
[Equation]
where S p − 1 denotes the unit sphere in R p and S p − 1 denotes its area.
Since A can be any fixed projection to k -dim subspace, without loss of generality, we can simply let a i be the i -th standard basis in R p (i.e., only the i -th element is nonzero and is equal to 1 ).
(Note that there are p standard bases although A is spanned by only the first k standard bases.)
In this situation, we have
[Equation]
S p
[Equation] [Equation]
i ,
[Equation]
˜
[Equation]
=
[Equation]
S p − 1 ⟨ a j , ˜ v ⟩ 2 d ˜ v , for all i, j ∈ { 1 , 2 , · · · , p } , and p
[Equation]
Therefore, we have
[Equation]
By Eq.
(35), we thus have
[Equation]
The result of this lemma thus follows.
Now we are ready to prove Proposition 3.
Proof of Proposition 3.
Define
[Equation]
Thus, we have
[Equation] [Equation]
By Lemma
[Equation]
we know that the distribution of the hyper-plane spanned by the rows of B has rotational symmetry (which is a mn v -dimensional space).
Therefore, θ follows the same distribution of the angle between a uniformly distributed random vector a ∈ R p and a fixed mn v -dimensional hyper-plane.
To characterize such distribution of θ (or equivalently, sin θ ), without loss of generality, we let a ∼ N (0 , I p ) and the hyper-plane be spanned by the first mn v standard bases of R p .
Thus, we have
[Equation]
Notice that (cid:13)(cid:13) a [ mn v +1: p ] (cid:13)(cid:13) 2 2 and ∥ a ∥ 22 follow χ 2 distribution with p − mn v degrees and p degrees of freedom, respectively.
By Lemma 9 and letting x = ln p in Lemma
[Equation]
we have
[Equation] [Equation]
[Equation]
Because p ≥ 16 , by Lemma 3, we have
[Equation] [Equation]
We define
A target ,
[Equation]
= { Term 1 of Eq
[Equation] [Equation]
b
[Equation] [Equation]
of
[Equation] [Equation]
We thus have
[Equation]
(by Eq.
(36) and the definition of P )
[Equation] [Equation]
)) ≤
[Equation]
p
[Equation]
p
[Equation]
p
[Equation]
+
[Equation]
a [ mn v +1: p ] (cid:13)(cid:13) 2 2 ≥ ( p
[Equation]
) + 2 (cid:112)
[Equation]
p − mn v ) ln p + 2 ln p (cid:111) (by Eq.
(42) and the union bound) ≤p (by Eq.
(38) and Eq.
[Equation]
By Eq.
(36) and Eq.
(43), we thus have
[Equation]
Similarly, using Eq.
(39) and Eq.
(41) (also by the union bound), we can prove
[Equation]
By Lemma 16, we have E [ ∥ P w 0 ∥ 22 ] =
mn v p ∥ w 0 ∥ 22 .
Thus, we have
E [ Term 1 of Eq.
= p − mn v p ∥ w 0 ∥ 22 .
The result of this lemma thus follows.
K P ROOF OF P ROPOSITION
Proof.
Define the event in Proposition 2 as
[Equation]
Define
[Equation]
Combining Proposition 4 and Proposition 5 with the union bound, we have
[Equation]
We adopt the event in Proposition 6 as
[Equation]
By Eq.
(14), we have (cid:84) 2 i =1 A target ,i = ⇒
A target .
Thus, we have
[Equation]
by the union bound)
[Equation]
(by Eq.
(44) and Proposition 6)
[Equation]
The last inequality is because m ≥ 1 , n v ≥ 1 , and
[Equation]
The result of this proposition thus follows.
L P ROOF OF P ROPOSITION
We prove Proposition 4 by estimating every element of BB T .
We split BB T into m × m blocks (each block is of size n v × n v ).
Recalling the definition of B in Eq.
(7), we identify three types of elements in BB T as diagonal elements (type 1), off-diagonal elements of diagonal block (type 2), and other elements (type 3).
Fig.
4 illustrates these three types of elements when m = 2 and n v = 3 .
In the rest of this part, we will first define and estimate each type of elements separately in
Appendices L.1, L.2, and L.3.
Using these results, we will then estimate the eigenvalues of BB T and finish the proof of Proposition 4.
[Figure]
T YPE 1: DIAGONAL ELEMENTS
Type 1 elements are the diagonal elements of BB T , which can be denoted as
[Equation]
where i ∈ { 1 , 2 , · · · , m } corresponds to the i -th training task, j ∈ { 1 , 2 , · · · , n v } corresponds to the input vector of the j -th validation sample (of the i -th training task).
To estimate Eq.
(45), we have the following lemma.
Lemma 17.
For any i ∈ { 1 , 2 , · · · , m } and any j ∈ { 1 , 2 , · · · , n v } , when p ≥ n t ≥ 256 , we must have
[Equation]
where
[Equation]
Proof.
See Appendix L.1.
T YPE 2: OFF - DIAGONAL ELEMENTS OF DIAGONAL BLOCKS
Type 2 elements are the off-diagonal elements of diagonal blocks.
Similar to Eq.
(45), Typeelements can be denoted by
[Equation]
where j ̸ = k .
We have the following lemma.
Lemma 18.
For any i ∈ { 1 , 2 , · · · , m } and any j, k ∈ { 1 , 2 , · · · , n v } that j ̸ = k , when p ≥ n t ≥ 256 , we must have
[Equation]
where
[Equation]
Proof.
See Appendix L.2.
T YPE 3: OTHER ELEMENTS ( ELEMENTS OF OFF - DIAGONAL BLOCKS )
Type 3 elements are other elements that are not Type 1 or Type 2.
In other words, Type 3 elements belong to off-diagonal blocks.
Similar to Eq.
(45) and Eq.
(46), Type 3 elements can be denoted by
[Equation]
where i ̸ = l .
We have the following lemma.
Lemma 19.
For any i, l ∈ { 1 , 2 , · · · , m } and any j, k ∈ { 1 , 2 , · · · , n v } that i ̸ = l , when p ≥ n t ≥ 256 , we must have
[Equation]
where
[Equation]
Proof.
See Appendix L.3.
Now we are ready to prove Proposition 4.
Proof of Proposition 4.
Define a few events as follows:
[Equation]
all type 2 elements of BB T are in [
[Equation]
type 3 elements of BB T are in [ − b 3 , b 3 ] (cid:9) .
Notice that there are mn v type 1 elements, mn v ( n v − 1) type 2 elements, and m ( m − 1) n 2 v typeelements.
By the union bound, Lemmas 17, 18, and 19, we have
[Equation]
[Equation] [Equation]
We first prove that (cid:84) 3 i =
A
i = ⇒
A target , 1 .
To that end, recall the definition of disc D ( a i,i , r i ( A )) and the radius of the disc r
i ( A ) for a matrix A in Lemma 7.
We now apply Lemma 7 on BB T .
Because of A 2 and A 3 , for any i ∈ { 1 , 2 , · · · , mn v } , we have
[Equation]
Because of (cid:84) 3 i =
A i and Lemma 7, we have all eigenvalues of BB T is in
[Equation]
Since p ≥ n t ≥ 256 , we have
[Equation]
Since n t ≥ 256 , by Eq.
(60) of Lemma 20, we have ln n t ≤ 145 n t .
Thus, we have
[Equation]
Recalling the values of b 1 , b 1 , b 2 , and b 3 in Lemmas 17, 18, and 19, we have
[Equation]
) b 2 + ( m − 1) n v b 3 (cid:1) =
max { 0 , 1 − α ′ t } 2 (cid:16) n t − (cid:112) 2 n t ln n t (cid:17) + p − n t − (cid:112) 2( p − n t ) ln n t − ( n v − 1) (cid:16) max { α ′ t , 1 − α ′ t } 2 (cid:112) n t ln n t + √ p ln p (cid:17) − ( m − 1) n v · 6 (cid:112) p ln p ≥ p + (cid:0) max { 0 , 1 − α ′ t } 2 − 1 (cid:1) n t − (cid:16)(cid:16) √ 2 + ( n v − 1) (cid:17) max { α ′ t , 1 − α ′ t } 2 + √ 2 + ( n v − 1) + 6( m − 1) n v (cid:17) √ p ln p (by Eq.
(51) and max { 0 ,
1 − α ′ t } 2 ≤ max { α ′ t , 1 − α ′ t } 2 ) ≥ p + (cid:0) max { 0 , 1 − α ′ t } 2 − 1 (cid:1) n t − (cid:0) ( n v + 1) max { α ′ t , 1 − α ′ t } 2 + 6 mn v (cid:1) √ p ln p =
b eig , min , and b 1 + (cid:0) ( n v − 1) b
2 + ( m − 1) n v b 3 (cid:1) =
max { α ′ t , 1 − α ′ t } 2 (cid:16) n t + (cid:112) 2 n t ln n t + ln n t (cid:17) + p − n t + (cid:112) 2( p − n t ) ln n t + ln n t + ( n v − 1) (cid:16) max { α ′ t , 1 − α ′ t } 2 (cid:112) n t ln n t + √ p ln p (cid:17)
+ ( m − 1) n v · 6 (cid:112) p ln p ≤ p + (cid:0) max { α ′ t , 1 − α ′ t } 2 − 1 (cid:1) n t + (cid:18)(cid:16) √ 2 + ( n v − 1) (cid:17) max { α ′ t , 1 − α ′ t } 2 + 1 √ 45 + √ 2 + ( n v − 1) + 6( m − 1) n v (cid:19) √ p ln p
(by Eq.
(51) and max { 0 ,
1 − α ′ t } 2 ≤ max { α ′ t , 1 − α ′ t } 2 ) ≤ p + (cid:0) max { α ′ t , 1 − α ′ t } 2 − 1 (cid:1) n t
+ (cid:0) ( n v + 1) max { α ′ t , 1 − α ′ t } 2 + 6 mn v (cid:1) √ p ln p =
b eig , max .
Define
A ( p ≥ n t ) target , 1 := (cid:8) b eig , min ≤ λ min ( BB T ) ≤ λ max ( BB T ) ≤ b eig , max (cid:9) ,
Therefore, we have proven that (cid:84) 3 i =
A
i = ⇒
A ( p ≥ n t ) target , 1 .
It remains to estimate the probability of A ( p ≥ n t ) target , 1 .
To that end, we have
[Equation]
(since
[Equation]
i
[Equation]
A i =
[Equation]
A
[Equation]
n t ) target ,
[Equation]
(cid:40)
[Equation]
(cid:91)
i
[Equation]
A
[Equation]
(cid:41)
[Equation]
(cid:88)
i
[Equation]
(by the union bound)
[Equation]
(by Eqs
[Equation] [Equation]
The result of this proposition thus follows.
L. P ROOF OF L EMMA
Since all training inputs are independent with each other, without loss of generality, we let i =and replace [ X v ( i ) ]
j by a random vector a ∼ N ( 0 , I p ) ∈
R p × 1 which is independent of X t ( i ) .
In other words, estimating Eq.
(45) is equivalent to estimate
[Equation]
We further introduce some extra notations as follows.
Since p ≥ n t , we can define the singular values of X t (1) ∈ R p × n t as
[Equation]
Define
[Equation]
Do singular value decomposition of X t (1) , we have
[Equation]
Notice that U t (1) ∈
R p × p is an orthogonal matrix, D t (1) = (cid:2) Λ t (1) 0 (cid:3)
∈ R p × n t , and V t (
) ∈ R n t × n t is an orthogonal matrix.
Using these notations, we thus have
I p
[Equation]
The following two lemmas will be useful in the proof of Lemma 17.
Lemma 20.
If x ≥ 16 , then
[Equation] [Equation]
Further, if x ≥ 256 , then
[Equation] [Equation]
Proof.
We prove each equation sequentially as follows.
Proof of Eq.
When x ≥ 16 , we have
[Equation]
Thus, 316 x − ln x is monotone increasing when x ≥ 16 .
Thus, in order to prove 316 x ≥ ln x for all x ≥ 16 , we only need to prove 316 · 16 ≥ ln 16 .
Notice that ln(16)
≈ 2 .
7726 < 3 .
Therefore,
Eq.
(56) holds.
Proof of Eq.
Using Eq.
(56), we have
[Equation]
Eq.
(57) thus follows.
Proof of Eq.
Doing square root on both sides of 316 x ≥ ln x , we thus have
[Equation]
Eq.
(58) thus follows.
Proof of Eq.
We have
[Equation]
since
[Equation]
Eq.
(59) thus follows.
Proof of Eq.
When x ≥ 256 , we have
[Equation]
Thus, 145 x − ln x is monotone increasing when x ≥ 256 .
Thus, in order to prove 145 x ≥ ln x for all x ≥ 256 , we only need to prove 145 · 256 ≥ ln 256 .
Notice that 256 / 45 − ln 256 ≈ 0 .
1437 ≥ 0 .
Therefore, Eq.
(60) holds.
Now we are ready to prove Lemma 17.
Proof of Lemma 17.
Recalling U t (1) in Eq.
(54), we define
[Equation]
We then have
[Equation]
(by Eq.
(62) and Eq.
[Equation]
Because of rotational symmetry of normal distribution of a
[Equation]
we know that χ 2 numTrain and χ 2 p − n t follows χ 2 distribution of p and p − n t degrees of freedom
[Equation]
respectively 5 .
We define several events as follows:
[Equation]
[Figure]
[Equation]
[Figure]
We have
[Equation]
Define the target event
[Equation]
It remains to prove that (cid:84) 6 i =1 A ci =
⇒
A .
To that end, when (cid:84) 6 i =
A ci , we have
[Equation]
t + χ
[Equation]
p − n t (by Eq.
(63)) ≥ max { 0 , 1 − α ′ t } 2 (cid:16) n t − (cid:112) 2 n t ln n t (cid:17) + p − n t − (cid:112) 2( p − n t ) ln n t (by A c 3 and A c 6 ) =
b 1 , and (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
[Equation]
≤ (cid:18) max
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
≤
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(by A c 1 , A c 2 , A c 4 and A c 5 )
[Equation]
L. P ROOF OF L EMMA
Since all samples are independent with each other, without loss of generality, we let i = 1 and replace [ X v ( i ) ]
j , [ X v ( i ) ]
k by two i.i.d.
random vector a , b ∼ N ( 0 , I p )
∈
R p × 1 which are independent of X t ( i ) .
In other words, estimating Eq.
(46) is equivalent to estimate
[Equation]
Proof of Lemma 18.
Recalling U t (1) in Eq.
(54), we define
[Equation]
We then have
[Equation]
Eq
[Equation] [Equation]
Eq
[Equation] [Equation]
Because of the rotational symmetry of normal distribution
[Equation]
we know that ϕ n t and ϕ p − n t have the same probability distribution if a ′ and b ′ follow i.i
[Equation]
d. N ( 0 , I p ) 6 .
Define several events as follows:
[Equation]
We have
[Equation]
union
[Equation]
Lemma
[Equation] [Equation]
Define the target event
[Equation]
[Table]
[Equation]
By Eq.
(69) (which implies (cid:84) 4 i =
A ci = ⇒ A ) and Eq.
(68), the result of this lemma thus follows.
L. P ROOF OF L EMMA
Because all inputs are i.i.d.
, Eq.
(47) is the inner product of two independent vectors, where each vector follows the same distribution as the vector
[Equation]
where a ∼ N ( 0 , I p ) and is independent of X t (1) .
In other words, it is equivalent to estimate ρ T 1 ρwhere ρ 1 and ρ 2 follow the i.i.d.
shown in (70).
If we can characterize the probability distribution of ρ , then we can estimate Eq.
The following lemma shows the rotational symmetry of Eq.
Lemma 21.
The probability distribution of ρ has rotational symmetry.
In other words, for any rotation S ∈ SO ( p ) where SO ( p ) ⊆
R p × p denotes the set of all rotations in p dimension, the rotated random vector S ρ shares the same probability distribution with the original vector ρ .
Proof.
By Eq.
(70), we have
[Equation]
(because S − 1 = S T , as a rotation is an orthogonal matrix)
[Equation]
Notice that S a and SX t (1) are the rotated vectors of a and X t (1) , respectively.
Since a and X t (1) are independent Gaussian vector/matrix, by rotational symmetry, we know their distribution and independence do not affected by a common rotation S .
Thus, S a − α t n t ( SX t (1) )( SX t (1) ) T ( S a ) has the same probability distribution as ρ .
The result of this lemma thus follows.
The following lemma characterize the distribution of the angle between two independent random vectors where both vectors have rotational symmetry.
Lemma 22.
Consider two i.i.d.
random vector c 1 , c 2 ∈ R p that have rotational symmetry.
We then have
[Equation]
Proof.
Notice that | c
[Equation]
1 c 2 | ∥ c 1 ∥
[Equation]
c 2 ∥ 2 denotes the angle between c 1 and c 2 .
By rotational symmetry, it is equivalent to prove that the angle between two independent random vectors that have rotational symmetry.
To that end, consider two i.i.d.
random vectors x 1 , x 2 ∼ N ( 0 , I p ) .
The distribution of the angle between x 1 and x 2 should be the same as the angle between c 1 and c 2 .
In other words, we have
[Equation]
By Lemma 11, we have
[Equation]
Noticing that ∥ x 1 ∥ 2 and ∥ x 2 ∥ 2 follow chi-square distribution with p degrees of freedom, by
Lemma
[Equation]
we have
[Equation]
By Eqs.
(72)(73)(74) and the union bound, we thus have
[Equation]
By Eq.
(71), the result of this lemma thus follows.
Now we are ready to prove Lemma 19.
Proof of Lemma 19.
Since p ≥ n t ≥ 256 , by Eq.
(60) of Lemma 20, we have
[Equation]
We define some events as follows:
[Equation]
By Lemma 17, we have
[Equation]
By Lemma 22, we have
[Equation]
Since n t ≥ 256 , by letting x = n t in Eq.
(57) of Lemma 20, we have
[Equation] [Equation]
We also have
[Equation]
Therefore, we have
[Equation]
p
[Equation]
First, we want to show (cid:84) 3 i =
A ci = ⇒
A .
To that end, when (cid:84) 3 i =
A ci , we have
[Equation]
Lemma 17)
[Equation]
Eq.
[Equation] [Equation]
and Eq.
[Equation] [Equation]
Thus, we have proven that (cid:84) 3 i =
A ci = ⇒ A , which implies that
Pr {A}
[Equation]
Pr
[Equation]
(cid:92) i
[Equation]
(cid:41)
[Equation]
Pr (cid:40)
[Equation]
(cid:91) i =
A i (cid:41) ≥ 1 − 3 (cid:88) i =
Pr {A i } (by the union bound) ≥ 1 − 10 √ n t − 2 p − 2 p 0 .
4 (by Eq.
(76) and Eq.
(77)) ≥ 1 − 13 n 0 .
4 t (by p 0 .
6 ≥ 2 since p ≥ n t ≥ 256 ) .
The result of this lemma thus follows.
M P ROOF OF P ROPOSITION
Lemma 23.
For any i ∈ { 1 , 2 , · · · , m } and any j ∈ { 1 , 2 , · · · , n v } , when n t ≥ p ≥ 256 , we must have
[Equation]
where
[Equation]
Proof.
See Appendix M.
Lemma 24.
For any i ∈ { 1 , 2 , · · · , m } and any j, k ∈ { 1 , 2 , · · · , n v } that j ̸ = k , when n t ≥ p ≥ 256 and α t ≤ 0 .
2 , we must have
[Equation]
where
[Equation]
Proof.
See Appendix M.2.
Lemma 25.
For any i, l ∈ { 1 , 2 , · · · , m } that i ̸ = l and for any j, k ∈ { 1 , 2 , · · · , n v } , when n t ≥ p ≥ 256 and α t ≤ 0 .
2 , we must have
[Equation]
where
[Equation]
Proof.
See Appendix M.3.
Now we are ready to prove Proposition
Proof of Proposition 5.
Define a few events as follows:
[Equation]
Because n t ≥ p ≥ 256 , we have
[Equation]
Since p ≥ 256 , by Eq.
(60) in Lemma 20, we have
[Equation]
Notice that there are mn v type 1 elements, mn v ( n v − 1) type 2 elements, and m ( m − 1) n 2 v typeelements.
By the union bound, Lemmas 17, 18, 19, and Eq.
(81), we have
[Equation]
By (cid:84) 3 i =
A i and Lemma 7, we have all eigenvalues of BB T in [ c 1 − (( n v − 1)
c
[Equation]
+
[Equation]
m − 1) n
[Equation]
c 3 ) , c 1 + (( n v − 1)
c
2 + ( m − 1) n v c 3 )] .
Recalling the values of c 1 , c 2 , and c 3 in Lemmas 23, 24, 25, we have
[Equation]
(noticing that max { α ′ t , 1 − α ′ t } 2 ≥ max { 0 , 1 − α ′ t } 2 )
=
c eig , min .
and
[Equation] [Equation]
Define the event
[Equation]
Therefore, we have proven (cid:84) 3 i =
A i = ⇒
A ( p ≤ n t ) target , 1 .
Thus, we have
[Equation]
th union bound)
[Equation] [Equation]
The result of this proposition thus follows.
M. P ROOF OF L EMMA
Proof.
When p < n t , we define the singular values of X t (1) ∈ R p × n t as
[Equation]
Define
[Equation]
We can still do the singular value decomposition as Eq.
(54), but here D t (1) = (cid:2) Λ t (1) 0 (cid:3)
∈
R p
[Equation]
n t since p < n t .
Using these notations, we thus have
[Equation]
Similar to Eq.
(63), we have
[Equation]
where χ 2 p :=
(cid:13)(cid:13)(cid:13) U t (1) T a (cid:13)(cid:13)(cid:13) 2 2 = ∥
a ∥ 22 follows χ 2 distribution with p degrees of freedom.
We define several events as follows:
[Equation]
We have
[Equation]
union
[Equation]
Define the target event
[Equation]
It remains to prove that (cid:84) 4 i =
A ci = ⇒
A .
To that end, when (cid:84) 4 i =
A ci , we have
[Equation]
max
[Equation]
max
[Equation]
c 1 and A c 3 ) = max { 0 , 1 − α ′ t } 2 · ( p − 2 (cid:112) p ln p ) .
When (cid:84) 4 i =
A ci , we also have (cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
I p − α t n t X t (1) X t (1) T (cid:19) a (cid:13)(cid:13)(cid:13)(cid:13) ≤ (cid:18) max (cid:26)(cid:12)(cid:12)(cid:12)(cid:12) 1 − α t n t λ t (1) p 2 (cid:12)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:12) 1 − α t n t λ t (1)12 (cid:12)(cid:12)(cid:12)(cid:12)(cid:27)(cid:19) 2 χ 2 p (by Eq.
(84)) ≤ max (cid:26)(cid:12)(cid:12)(cid:12)(cid:12) 1 − α t n t ( √ n t − √ p − ln √ n t ) 2 (cid:12)(cid:12)(cid:12)(cid:12) , (cid:12)(cid:12)(cid:12)(cid:12) 1 − α t n t ( √ n t + √ p + ln √ n t ) 2 (cid:12)(cid:12)(cid:12)(cid:12)(cid:27)· (cid:16) p + 2 (cid:112) p
ln p + 2 ln p (cid:17) (by A c 2 , A c 3 and A c 4 ) ≤ max { α ′ t , 1 − α ′ t } 2 (cid:16) p + 2 (cid:112)
p ln p + 2 ln p (cid:17) .
Proof.
Similar to Eq.
(67), we have
[Equation]
Define several events as follows:
[Equation]
When (cid:84) 3 i =
A ci , we have
[Equation]
Eq
[Equation] [Equation]
and A c 3 )
[Equation]
Thus
[Equation]
we have
[Equation]
the union bound) ≥ 1 − 2 p 0 .
4 − 2 exp (cid:0) − (ln √ n t ) 2 / 2 (cid:1) (by Lemma 11 and Lemma 8) ≥ 1 − 2 p 0 .
4 − 2 √ n t (since ln √ n t ≥ 2 when n t ≥ 256 ) .
M. P ROOF OF L EMMA
Proof.
Define several events as follows:
[Equation]
By Lemma 23, we have
[Equation]
By Lemma 22, we have
[Equation]
When p ≥ 256 , by Eq.
(60) of Lemma 20, we have
[Equation]
First, we want to show (cid:84) 3 i =
A ci = ⇒
A .
To that end, when (cid:84) 3 i =
A ci , we have
[Equation]
(by
[Equation]
(cid:113)
[Equation] [Equation]
(cid:113)(cid:112)
[Equation]
(by Eq.
[Equation] [Equation]
(cid:112)
[Equation]
ln
[Equation]
(because 1 +(cid:113) 145 +1 −
[Equation]
(cid:113)≈ 1 . 91 ≤ 2 ) .
Thus, we have proven (cid:84) 3 i =
A c i = ⇒
A , which implies that
Pr
[Equation]
A
[Equation]
(cid:41) ≥
[Equation]
(cid:88) i =
[Equation]
Pr[ A
[Equation]
] (by the union bound) ≥ 1 −
[Equation]
p
[Equation] [Equation]
(by Eq.
(86) and Eq.
The result of this lemma thus follows.
N P ROOF OF P ROPOSITION
Plugging Eq.
(1) into Eq.
(7), we have
[Equation]

[Equation]
(cid:16) I p − α t n t X t (1) X t (1) T (cid:17) w (1) − α t n t X v (1) T X t (1)
ϵ t (1) + ϵ v (1)
X v (2) T (cid:16) I p − α t n t X t (2)
X t (2) T (cid:17) w (2) − α t n t X v (2) T X t (2)
ϵ t (2) + ϵ v (2) ...
X v ( m ) T (cid:16) I p − α t n t X t ( m )
X t ( m ) T (cid:17) w ( m ) − α t n t X v ( m ) T X t ( m ) ϵ t ( m ) + ϵ v ( m ) 
 .
By Eq.
(11), we thus have δγ =  
[Equation] [Equation]
I
[Equation]
α t n t
[Equation]
[Equation]
X t ( m ) T (cid:17) ( w ( m ) − w
[Equation]
) − α t n t X v ( m ) T X t ( m ) ϵ t ( m ) + ϵ v ( m ) 
 .
In Eq.
(89), since terms ϵ t (1: m ) and ϵ v (1: m ) have zero mean and are independent of each other, we have
[Equation]
Notice that
E
[Equation]
by trace trick
Tr[
[Equation] [Equation]
[ ZW ] )
[Equation]
since E
[Equation] [Equation]
σ
[Equation]
I
[Equation] [Equation]
Plugging Eq.
(91) into Eq.
(90), we thus have
E
[Equation] [Equation]
1: m
[Equation]
The following two lemmas estimate Term A and Term B.
Lemma 26.
When n t ≥ 16 , we have
[Equation]
and
E
[Equation]
Proof.
See Appendix N.1.
Lemma 27.
We have
[Equation]
Proof.
See Appendix N.2.
Now we are ready to prove Proposition 6.
Proof of Proposition 6.
By Eq.
(92), Lemma
[Equation]
and Lemma 27, the result of Proposition 6 thus follows.
Notice that the probability is estimated by the union bound.
N. P ROOF OF L EMMA
Proof.
We have
[Equation]
(by Assumption 2) .
[Equation]
Define
A (
[Equation]
Plugging A ( i ) into Eq.
(95), we thus have
E
[Equation]
[Equation]
Here [ · ] j,k denotes the element at the j -th row, k -th column of the matrix, [ · ] l, : denotes the l -th row (vector) of the matrix, [ · ] : ,k denotes the k -th column (vector) of the matrix.
Notice that only the first s rows of A ( i ) is non-zero.
Define
[Equation]
We thus have
[Equation]
when j = 1 , · · · , s, 0 , when j = s + 1 , · · · , p, =
[Equation]
ν
[Equation]
i ) ,j Q i,j,k ,
when j = 1 , · · · , s, 0 , when j = s + 1 , · · · , p.
Therefore, for any k ∈ { 1 , 2 , · · · , n v } , we have [ A
[Equation]
i
[Equation]
A
[Equation]
i
[Equation] [Equation]
[ A
[Equation]
i ) ]
[Equation]
[Equation] [Equation]
i ) ] :
[Equation]
=
[Equation] [Equation]
(cid:88)
[Equation]
= ν (
[Equation] [Equation]
Q
[Equation]
By Eq.
(96) and Eq.
(97), we thus have
Term A in Eq.
= m (cid:88) i
= n v (cid:88) k
= s (cid:88)
j = ν ( i ) ,j 2 Q 2 i,j,k .
Part 1: calculate the expected value of Q 2 i,j,k
By Assumption 1 and Lemma 13, we have
E
[Equation]
(cid:13)(cid:13)(cid:13) 2 2 = n t , and E (cid:13)(cid:13)(cid:13) X t ( t ) j, : (cid:13)(cid:13)(cid:13) 4 2 = n t ( n t + 2) .
We also have
[Equation]
t ( i )
[Equation]
(cid:32)
[Equation]
(cid:33)
[Equation]
= n
[Equation]
(cid:88)
[Equation]
E ( X
[Equation]
( i )
[Equation]
X t ( i ) l
[Equation]
q )
[Equation]
(by Assumption 1) =
[Equation]
(cid:88)
[Equation]
If we fix X t ( i ) and only consider the randomness in X v ( i ) , since each element of X v ( i ): ,k are i.i.d.
standard Gaussian random variables, then we have
Q i,j,k ∼ N (0 , σ 2 Q i,j,k ) , where σ 2 Q i,j,k = (cid:18) 1 − α t n t (cid:13)(cid:13) (cid:13) X t ( i ) j, : (cid:13)(cid:13) (cid:13) 2 2 (cid:19) + (cid:88) l = { 1 , 2 , ··· ,p }\{ j } (cid:18) α t n t ⟨ X t ( i ) j, : , X t ( i ) l, : ⟩ (cid:19).
Thus, we have
E
X v ( i ) , X t ( i ) Q 2 i,j,k =
E X t ( i ) E
X v ( i ) Q 2 i,j,k =
E X t ( i ) σ 2 Q i,j,k (by Eq.
=1 −
[Equation]
α t n
[Equation]
E
[Equation]
( i ) (cid:13)(cid:13) (cid:13) X t ( i ) j, : (cid:13)(cid:13) (cid:13) 2
+ α 2 t n 2 t
E
X t ( i ) (cid:13)(cid:13) (cid:13) X t
( i ) j, : (cid:13)(cid:13) (cid:13)
[Equation]
n
[Equation]
( i ) j
[Equation] [Equation]
t ( i ) l
[Equation]
: ⟩=1 − 2 α t + α 2 t ( n t + 2) n t + α 2 t ( p − 1) n t (by Eq.
(99) and Eq.
(100)) =(1 − α t )
+ α 2 t ( p + 1) n t .
By Eq.
(102) and Eq.
(97), we thus have
[Equation]
Thus, we have
E [ Term
[Equation]
Notice that we use the definition of ν and ν ( i ) in Assumption 2 that ν 2 = (cid:80) mi
=1 ν
[Equation]
m and ν 2( i )
=
(cid:80) sj =1 ν ( i ) ,j 2 .
Part 2: derive high probability upper bound
By Assumption 1 and Lemma 11 (where c = 2 .
5 ln( spn t ) and q = e ), for any given i ∈ { 1 , 2 , · · · , m } , j ∈ { 1 , 2 , · · · , s } , and l ∈ { 1 , 2 , · · · , p } \ { j } , we must have
[Equation]
By Assumption 1 and Lemma 9, for any given i ∈ { 1 , 2 , · · · , m } and j ∈ { 1 , 2 , · · · , s } , we have
Pr
[Equation]
X
[Equation]
By Eq.
(101) and Lemma 14, for any X t ( i ) , we have
Pr X v ( i )
[Equation]
Thus, we have
[Equation]
We define A 1 , ( i,j,l ) , A 2 , ( i,j ) , and A 3 , ( i,j,k ) as shown in Eq.
(103), Eq.
(104), and Eq.
(105), respec- tively.
Then, we define the event A as
A := (cid:26)
A c 1 , ( i,j,l ) , A c 2 , ( i,j ) , A c 3 , ( i,j,k ) hold for all i ∈ { 1 , 2 , · · · , m } , j ∈ { 1 , 2 , · · · , s } , l ∈ { 1 , 2 , · · · , p } \ { j } , and k ∈ { 1 , 2 , · · · , n v } (cid:27) ,
Applying the union bound, we then have
[Equation]
When A happens, we have
[Equation] [Equation]
(by A c 1 , ( i,j,l ) and A c 2 , ( i,j ) for all i, j, l ) .
[Equation] [Equation]
Thus, we have
[Equation]
=
[Equation]
(by Eq.
(106) and the definition of ν 2 in Assumption 2) .
The result of this lemma thus follows by combining Part 1 and Part 2.
N. P ROOF OF L EMMA
Proof.
In order to show Eq.
(93), it suffices to show that
Pr
[Equation]
and in order to show Eq.
(94), it suffices to show that for any i ∈ { 1 , 2 , · · · , m } ,
[Equation]
[Equation]
We first prove Eq.
To that end, we notice that X v ( i ) T X t ( i ) is a n v × n t matrix.
For any i = 1 , 2 , · · · , m and j = 1 , 2 , · · · , n v , we have (cid:104)
[Equation]
i
[Equation]
( i ) X t ( i ) T
[Equation]
v ( i ) (cid:105) j,j = (cid:13)(cid:13)(cid:13)
[ X v ( i ) T X t ( i ) ]
j, : (cid:13)(cid:13)(cid:13) 2= n t (cid:88) k
= (cid:104) X
v ( i ) T X t
( i ) (cid:105)j,k = n t (cid:88) k
= ⟨ [ X v ( i ) ] :
[Equation]
j
[Equation]
[ X t ( i ) ] : ,k ⟩= n t (cid:88) k
= (cid:32) p (cid:88) l
=
[ X v ( i ) ]
l,j
[ X t ( i ) ] l,k (cid:33).
Thus, we have max i ∈{ 1 , 2 , ·
·· ,m } Tr (cid:104) X v ( i ) T X t ( i ) X t
( i ) T X v ( i ) (cid:105) =
max i ∈{ 1 , 2 , ··· ,m } n v (cid:88)
j = (cid:104)
X v ( i ) T X t ( i ) X t
( i ) T X v ( i ) (cid:105) j,j = max i ∈{ 1 , 2 , ··· ,m } n v (cid:88)
j = n t (cid:88) k
= (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) p (cid:88) l
=
[ X v ( i ) ]
l,j
[ X t ( i ) ] l,k (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (by Eq.
[Equation]
Notice that training input X t ( i ) and validation input X v ( i ) are independence with each other and each element follows i
[Equation]
i
[Equation]
d. standard Gaussian distribution.
Therefore, by applying Lemma 11 (where c = ln n t , k = p , and q = p ), for any given i , j , and k , we have
[Equation]
The last inequality we use the fact that ln n t · 0 .
[Equation]
ln p
[Equation]
+ 0 .
[Equation]
p when min { n t , p } ≥ 256 .
By the union bound, we thus have
[Equation]
By Eq.
(110) and Eq.
(111), we have proven Eq.
The result Eq.
(93) of this lemma thus follows.
It remains to prove Eq.
To that end, by Eq.
(109), we have
[Equation]
Thus, by Assumption 1, we have
[Equation]
=
[Equation]
=
[Equation]
i
[Equation]
e., we have proven Eq.
(108) (and therefore Eq.
(94) holds).
The result of this lemma thus follows
[Equation]
O U NDERPARAMETERIZED SITUATION
In this case, we have p ≤ mn v .
The solution that minimize the meta loss is
[Equation]
When B is full column-rank, we have
[Equation]
Thus, we have
[Equation]
Define
[Equation]
Lemma 28.
Consider the case p =
s = 1 .
If there exist g, g, h ∈ R such that
[Equation]
[Equation] [Equation]
then we must have
[Equation]
Further, if n v (cid:88)
[Equation]
=
[Equation] [Equation]
( i
[Equation] [Equation]
∈ [ r
[Equation]
r ] for all i ∈ {
[Equation] [Equation]
[Equation] [Equation]
m }
[Equation]
(115) we then have
E
[Equation]
Proof.
Since p = s = 1 , B becomes a vector and B T B is a scalar that equals to
[Equation] [Equation]
By Eq.
(112), we have
[Equation]
In this case, B T δγ is also a scalar that equals to
B T δγ = m (cid:88) i
= n v (cid:88)
j = (cid:40)(cid:18)
[Equation]
[Equation]
By the independence and zero mean of w ( i ) − w 0 , ϵ t ( i ) , and ϵ v ( i ) , we have
E w (1: m ) , ϵ t (1: m ) , ϵ v (1: m ) (cid:0) B T δγ (cid:1)= νm (cid:88) i = (cid:18) 1 − α t n t (cid:13)(cid:13)(cid:13) X t
( i ) T (cid:13)(cid:13)(cid:13) (cid:19)·   n v (cid:88)
j
=
[ X v ( i ) ] 2 j   (cid:124) (cid:123)(cid:122) (cid:125) Term+ (cid:18)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
( i
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:19)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
( i )
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:88)
[Equation] [Equation]
( i
[Equation] [Equation]
(cid:125)
[Equation]
By Cauchy-Schwarz inequality, we have
[Equation]
We then have
Term 1 of Eq.
(118) ∈
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
=
[Equation]
( i )
[Equation]
 (by Eq.
(113) and Eq.
(114)) ∈
[Equation]
 (by Eq.
(119)) ∈
[Equation]
m
[Equation]

[Equation]
= n
[Equation]
j =
[ X v ( i ) ] 2 j 
 , ν 2 (cid:18) 1 − α t n t g (cid:19)mr  (by Eq.
(115)) , and
Term 2 of Eq.
(118) ∈ 
[Equation]
g n
[Equation]
(cid:18) 1 − α t n t g (cid:19)
[Equation]
m (cid:88) i =   n v (cid:88)
j
=
[ X v ( i ) ] 2 j   , (cid:18) σα t g n t (cid:19) 2 (cid:18) 1 − α t n t g (cid:19) 2 m (cid:88) i =   n v (cid:88)
j
=
[ X v ( i ) ] 2 j   2 
 (by Eq.
(113) and Eq.
∈  (cid:18) σα t g n t (cid:19)
[Equation]
(cid:18) 1 − α t n t g (cid:19) 2m   m (cid:88) i
= n v (cid:88) j
=
[ X v ( i ) ] 2 j   , (cid:18) σα t g n t (cid:19) 2 (cid:18) 1 − α t n t g (cid:19) 2 m (cid:88) i =   n v (cid:88)
j
=
[ X v ( i ) ] 2 j   2 
 (by Eq.
∈    (cid:18) σα t g n t (cid:19) 2 (cid:18) 1 − α t n t g (cid:19) 2m   m (cid:88) i
= n v (cid:88)
[Equation]
by Eq
[Equation]
[Equation] [Equation]
Similarly
[Equation]
we have
[Equation]
Plugging the above equations into Eq.
(117), the result of this lemma thus follows.
Proposition 7.
Define
[Equation]
When p = s = 1 and α t is relatively small such that 1 − α t n t g ≥
[Equation]
we must have
[Equation]
Proof.
Notice that (cid:13)(cid:13)
X t ( i ) (cid:13)(cid:13) 2 2 follows χ 2 distribution with n t degrees of freedom, and (cid:80) mi =1 (cid:80) n v j =
[ X v ( i ) ] 2 j follows χ 2 distribution with mn v degrees of freedom.
Given any fixed i ∈ { 1 , 2 , · · · , m } , by Lemma 9 (letting x = log n t ), we have
Pr
[Equation]
X
[Equation]
( i )
[Equation] [Equation]
g (cid:27)
[Equation]
n
[Equation]
Pr (cid:26)(cid:13)(cid:13)(cid:13)
X t ( i ) (cid:13)(cid:13)(cid:13)
[Equation]
(cid:27)
[Equation]
n t .
By Lemma 9 (letting x = log( mn v ) ), we have
Pr
[Equation]
By the union bound, we thus have
Pr (cid:26)(cid:13)(cid:13)(cid:13)
X t ( i ) (cid:13)(cid:13)(cid:13)2 ∈ [ g, g ] , for all i ∈ { 1 , 2 , · · · , m } (cid:27) ≤ 2 m n t .
Similarly, we have
Pr (cid:26)(cid:13)(cid:13)(cid:13)
X v ( i ) (cid:13)(cid:13)(cid:13)2 ∈ [ r, r ] , for all i ∈ { 1 , 2 , · · · , m } (cid:27) ≤ 2 m n v .
The result thus follows by Lemma 28.
We interpret the meaning of Proposition 7 as follows.
We first approximate each part by the highest order term.
Then we have g ≈ g ≈ n t , r ≈ r ≈ n v , and h ≈ mn v ≈ mr .
Thus, we have b ( p =1) w ( g, g, 1 , h )
≈
b ( p =1) w ( g, g, ( r/r ) 2 , mr ) .
Therefore, we can conclude that our estimation on the model error ∥
ˆ w ℓ 2 − w
∥ 2 2 in this case is relatively tight.
In other words, we know that (with high probability when n v and n t is relatively large, and α t is relatively small)
[Equation]