S HORT OPTIMIZATION PATHS LEAD TO GOOD GENERALIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
Optimization and generalization are two essential aspects of machine learning.
In this paper, we propose a framework to connect optimization with generalization by analyzing the generalization error based on the length of optimization trajectory under the gradient flow algorithm after convergence.
Through our approach, we show that, with a proper initialization, gradient flow converges following a short path with an explicit length estimate.
Such an estimate induces a length-based generalization bound, showing that short optimization paths after convergence indicate good generalization.
Our framework can be applied to broad settings.
For example, we use it to obtain generalization estimates on three distinct machine learning models: underdetermined ℓ p linear regression, kernel regression, and overparameterized two-layer ReLU neural networks.
I NTRODUCTION
From the perspective of statistical learning theory, the goal of machine learning is to find a predictive function that can give accurate predictions on new data.
For supervised learning problems, empirical risk minimization (ERM) is a common practice to achieve this goal.
The idea of ERM is to minimize a cost function on observed data by an optimization algorithm.
Therefore, a fundamental question is whether an optimization algorithm produces a solution with good generalization .
Recent works have shed light on providing theoretical explanations to this question from different angles.
One line of works considered the case when gradient methods converge to minimal norm solutions (Bartlett et al., 2020; Tsigler & Bartlett, 2020; Liang & Rakhlin, 2020; Liang et al., 2020) on kernel regression, and then analyzed the generalization of those minimal norm solutions.
However, the phenomenon of norm minimization has been known to happen only for the quadratic loss with an appropriate initialization.
Another line of works focused on overparameterized models, e.g., neural networks under the Neural Tangent Kernel (NTK) regime (Allen-Zhu et al., 2019; Arora et al., 2019;
Cao & Gu, 2020; Ji & Telgarsky, 2020; Chen et al., 2021), proving that overparameterized neural networks trained by (stochastic) gradient descent ((S)GD) have good generalization performance on certain target functions (for example, polynomial functions).
Although existing works have made significant progress on the interplay of optimization and generalization, they focused on studying specific models, such as the NTK and models possessing minimal norm solutions.
In this paper, instead, we study general loss function conditions that induce interesting connections between optimization and generalization.
We start with a simple observation, as shown in Figure 1, that under a generic random initialization, the generalization performance for both linear regression model and random feature regression model is closely related to the length of the optimization path 1 after convergence.
In particular, short optimization path are associated with good generalization.
Here, by length we mean the trajectory or path length of the parameter evolution during training.
This is not the “length of time” used for training, as is usually analyzed in early-stopping type of algorithms.
Thus, the generalization error concerns the weights of model trained to completion by empirical risk minimization.
This empirical investigation motivates us to use the trajectory length to connect optimization and generalization.
Intuitively, the length of the optimization path can be viewed as a kind of capacity control, and a
We observe that short optimization paths lead to good generalization.
See experiment settings and more numerical results in Appendix
.1 & A. short path signifies low “complexity”.
In other words, a general condition that guarantees a short optimization path can be used to induce good generalization performance.
Thus, inspired by the theory in (Bolte et al., 2007) that Łojasiewicz gradient inequality (LGI) induces an explicit bound for the gradient flow path, we consider Uniform-LGI (Definition 1).
This is a modified version of
LGI and plays a critical role in obtaining a length estimate for the gradient flow trajectory.
Once the length is estimated, we can use our length-based generalization bounds to show the generalization performance.
Contributions.
We summarize our contributions as follows:
• We focus on the gradient flow algorithm and propose a framework for combining optimization and generalization.
This framework is based on the length of the gradient flow trajectory, and its key component is the Uniform-LGI property.
• We then prove that under appropriate conditions, gradient flow returns a global minimum and gives an explicit length-estimate for the gradient flow trajectory (Theorem 1).
If this length-estimate has an uniform bound for a certain initialization method, then we can give a length-based generalization bound (Theorem 2).
• We further show applications of Theorem 1 and Theorem 2 to obtain generalization bounds on underdetermined ℓ p linear regression (Theorem 3), kernel regression (Theorem 4)
and overparameterized two-layer ReLU neural networks (Theorem 5).
These bounds match or expand the type of scenarios where we can rigorously establish the phenomenon of benign overfitting.
R ELATED W ORKS
Optimization.
Theoretically analyzing the training process of most machine learning models is a challenging problem as the loss landscapes are highly non-convex.
One approach to studying non-convex optimization problems is to use the Polyak-Łojasiewicz (PL) condition (Polyak, 1963), which characterizes the local geometry of loss landscapes and ensures the existence of global minima.
It is shown in (Karimi et al., 2016) that GD admits linear convergence for a class of optimization objective functions under the PL condition.
In this paper, we consider the Łojasiewicz gradient inequality (Lojasiewicz, 1965), an extended version of the PL condition that can be applied to more cases.
In particular, we modify the original LGI to get length estimates of the optimization paths, which will be used to derive length-based generalization bounds.
Generalization.
Traditional VC dimension-based generalization bounds depend on the number of parameters and will be vacuous for huge models such as overparameterized neural networks.
To overcome this limitation, several non-vacuous generalization bounds are proposed.
For example, the norm/margin-based generalization bounds (Neyshabur et al., 2015; Bartlett et al., 2017; Golowich et al., 2018; Neyshabur et al., 2019), and the PAC-Bayes-based bounds (Dziugaite & Roy, 2017;
Neyshabur et al., 2018;
Zhou et al., 2019; Rivasplata et al., 2020).
However, these bounds tend to focus less on optimization, e.g., norm-based generalization bounds may not discuss how small-norm solutions are obtained through practical training.
In this paper, we connect the optimization and generalization by deriving generalization bounds based on the optimization trajectory length.
Interface between optimization and generalization.
Implicit bias builds the bridge between optimization and generalization, which has been widely studied to explain the generalization ability of machine learning models.
Recent works (Soudry et al., 2018a;b; Nacson et al., 2019a;b; Lyu & Li, 2020) showed that linear classifiers or deep neural networks trained by GD/SGD maximizes the margin of the separating hyperplanes and therefore generalizes well.
Other works (Arora et al., 2019; Zou & Gu, 2019; Cao & Gu, 2020; Ji & Telgarsky, 2020; Chen et al., 2021) concentrated on overparameterized neural networks, showing that GD/SGD trajectories fall within the NTK regime, where the minimizer has good generalization due to the small “complexity” of the parameter space.
In this work, we focus on specific conditions on loss functions under which we can connect optimization and generalization based on the lengths of optimization paths.
M AIN RESULTS
In this section, we aim to address optimization and generalization together by proving results based on the Uniform-LGI property.
We begin with describing notations and the problem setting in Section 3.1.
Then we give an explicit length estimate for the gradient flow trajectory in Section 3.2.
Lastly, we show a length-based generalization bound in Section 3.3.
3. S ETUP AND N OTATIONS
Consider a supervised learning problem on a hypothesis space F = (cid:8) f ( w, · ) : R d −→ R | w ∈ W (cid:9) , where W is a parameter set in Euclidean space.
Given a loss function ℓ : R × R −→ R , and a training set S
=
( x i , y i ) } ni =1 ⊆ R d × R with n independent and identically distributed (i.i.d.)
samples from a joint distribution D , the goal of ERM is to optimize the empirical loss function L n ( w ) on S :
[Equation]
We assume that each input vector and label have bounded norms.
Specifically, we assume that, for any ( x, y ) following D , ∥ x ∥ = 1 , | y | ≤ 1 .
This can be achieved by data normalization.
To simplify the analysis, we optimize the empirical loss by gradient flow 2 :
[Equation]
where w ( t ) is the parameter value at time t , w (0) is the initialization parameter set, and ∇L n ( w ) is the gradient of L n ( w ) with respect to w .
Given the gradient flow, we may define the length of the gradient flow curve from w (0) to w ( t ) as len ( w (0)
[Equation]
ds .
Notations.
We use ∥·∥ to denote the ℓ 2 norm of a vector or the spectral norm of a matrix, and use ∥·∥ F to denote the Fronenius norm of a matrix.
For a set S ∈ R n , we use ∂ S to denote its boundary.
We use d ( S 1 , S 2 ) to represent the Euclidean distance between two sets S 1 , S 2 , which is defined as d ( S 1 , S 2 ) = { inf ∥ s 1 − s 2 ∥ : s 1 ∈ S 1 , s 2 ∈ S 2 } .
For two vectors, we use ⟨ , ⟩ to denote their inner product.
Let λ min ( A ) and λ max ( A ) be the smallest and largest eigenvalues of a symmetric matrix A .
For any positive integer n , we denote [ n ] = { 1 , 2 , . . . , n } .
We use O ( · ) to stand for Big-O notation.
The classical LGI gives a lower bound of the gradient of a differentiable function based on its value above its minimum.
Many functions, e.g., real analytic functions and subanalytic functions, satisfy this property, at least locally (Bolte et al., 2007).
Here, we require a global version of this inequality as a condition to control the gradient flow trajectory.
Let us define this notion below.
Definition 1 (Uniform-LGI) .
A function L ( w ) satisfies Uniform-LGI on a set S with two constants c > 0 and θ ∈ [ 12 , 1) , if ∥∇L ( w ) ∥ ≥ c ( L ( w ) − min v ∈W L ( v )) θ , ∀ w ∈ S .
In the special case when θ = 1 / 2 and c = √ 2 µ , the Uniform-LGI corresponds to the µ -PL condition (Karimi et al., 2016).
Here we give examples of functions satisfying the Uniform-LGI but not the PL condition: L ( w ) =
w 2 k , k ∈ Z + .
When k ≥ 2 , L ( w ) satisfies the Uniform-LGI on R with c = 2 k and θ = 1 − 1 / 2 k .
3. G RADIENT FLOW TRAJECTORY LENGTH ESTIMATE
Our first result shows that under the uniform-LGI condition, gradient flow returns a global minimum with an explicit estimate of the gradient flow trajectory length as in the following theorem.
Theorem 1.
For any initialization w (0) , if L n ( w ) satisfies Uniform-LGI on a closed set S n with constants c n , θ n such that S n ⊇ B (cid:0) w (0) , r n ( w (0) ) (cid:1) , where
[Equation]
then w ( t ) converges to a global minimum w ( ∞ ) with the trajectory length upper bounded by len ( w (0) , ∞ ) ≤ r n ( w (0) ) .
The convergence rate of L n ( w ( t ) ) is given by:
[Equation]
where M n = c 2 n (2 θ n − 1) (cid:0) L n ( w
[Equation]
The proof of Theorem 1 is given in Appendix B.
This theorem yields that once the loss function satisfies the Uniform-LGI around the initialization, gradient flow returns a global minimum with an explicit trajectory length estimate.
Moreover, for any fixed n , len ( w (0) , ∞ ) decreases as the initial loss value decreases.
The calculation of c n , θ n for different n should be analyzed case by case based on the loss landscape, as shown in Section 4.
3. H OW DOES THE OPTIMIZATION PATH LENGTH AFFECT GENERALIZATION ?
The Uniform-LGI property allows us to have an upper bound for the gradient flow trajectory length.
In this subsection, we use the length estimate we get in Theorem 1 to study generalization.
Throughout this subsection, we assume that there exists an almost everywhere differentiable function Ψ : R p + q −→ R such that f ( w, · ) can be represented in the following form,
[Equation]
with α 1 , . . .
, α p ∈ R d , β 1 , . . . , β q ∈ R , and w = vec ( { α 1 , . . .
, α p , β 1 , . . . , β q } )
∈ R pd + q .
Here vec is the vectorization operator that concatenates all elements into a column vector.
A wide class of functions can be represented in the form (3).
Examples include linear functions f ( w, x ) =
w ⊤ x and two-layer neural networks f ( w, x ) =
a ⊤ ϕ ( Wx ) with a ∈ R m , W ∈ R m × d .
Additional notations.
For the loss function ℓ , we use L ℓ ( S ) to denote its Lipschitz constant (the maximal gradient norm) on S with respect to its first argument.
For Ψ in (3), we define L Ψ ( S ) :
= (cid:16) L (1)Ψ ( S ) , · · · , L ( p )Ψ ( S ) , L ( p +1)Ψ ( S ) , · · · , L ( p + q ) Ψ ( S ) (cid:17) ⊤ , where L ( i )Ψ ( S ) is the Lipschitz constant of Ψ on S with respect to the i -th variable.
Let w (0) :=
vec (cid:16)(cid:110) α (0)1 , . .
, α (0) p , β (0)1 , . .
, β (0) q (cid:111)(cid:17) and use L D ( w ) to denote the expected loss E ( x,y )
∼D [ ℓ ( f ( w, x ) , y )] .
For a = ( a 1 , . . .
, a p ) ⊤ and b = ( b 1 , . . .
, b q ) ⊤ , we define S a,b = { w : ∀ i ∈
[ p ] , j ∈ [ q ] , ∥ α i ∥ ≤ a i , | β j | ≤ b j } , and M a,b = sup w ∈S a,b , ∥ x ∥≤ 1 , | y |≤ 1 ℓ ( f ( w, x ) , y ) .
In the following theorem, we give a path-based generalization bound if one has a length estimate for optimization trajectory length len ( w (0) , ∞ ) .
Based on the Uniform-LGI property, Theoremserves as a sufficient condition to ensure that the trajectory length can be estimated.
In that sense, we connect optimization with generalization.
The rigorous statement is as follows:
Theorem 2.
Consider an initialization method so that w (0) is independent of the training samples and suppose that for any δ ∈ (0 , 1) , there exist M δ , R n,δ >
such that (cid:13)(cid:13) w (0) (cid:13)(cid:13) ≤ M δ , len ( w (0) , ∞ ) ≤
R n,δ with probability at least 1 − δ over the initialization and the training samples.
Then, we have with probability at least 1 − δ over initialization and the training samples, the generalization error of the global minimum w ( ∞ ) is bounded as:
[Equation]
where R = √ 2( M δ + R n,δ ) .
The proof of Theorem 2 is given in Appendix C. There are five key terms in (4).
To make the bound easier to understand, we discuss them in sequence:
• M δ : This is a high probability upper bound for the ℓ 2 norm of the initialized vector (cid:13)(cid:13) w (0) (cid:13)(cid:13) .
In practice, for commonly used initialization methods in deep learning, such as Xavier initialization (Glorot & Bengio, 2010) and Kaiming initialization (He et al., 2015), the norm (cid:13)(cid:13) w (0) (cid:13)(cid:13) is uniformly bounded with high probability.
• R n,δ : This is an high probability upper bound for the optimization path length over the initialization and training samples.
Theorem 2 assumes that such a bound exists, and show that it immediately implies a generalization estimate based on this bound.
Theorem 1 gives a sufficient condition to obtain such a path estimate, in which case it depends on the Uniform-LGI constants c n , θ n and data distribution D .
The asymptotic analysis of R n,δ is problem-dependent, and we provide several examples in Section 4.
• L ℓ ( S a,b ) and M a,b : They depend on the loss function ℓ .
For example, for any loss function ℓ : R × R −→ [0 , 1] that is 1-Lipschitz in the first argument, L ℓ ( S a,b ) =
M a,b = 1 . • ∥
L Ψ ( S a,b ) ∥ :
This term relies on the structure of f .
For example, when f is linear, Ψ( x, y ) =
x + y , then ∥ L Ψ
( S a,b ) ∥ = √ 2 .
Remark 1.
Theorem 2 shows that if the optimization trajectory length is small, then this implies a good generalization bound.
First, note that our generalization bound concerns the weights of the final model.
Nevertheless, by the convergence analysis in Theorem 1, our framework can be extended to derive generalization estimates that evolves according the length of time of training.
See Appendix
C.1 for additional results.
Second, we emphasize again that the generalization bound in Theorem 2 only relies on a path length estimate, and theorem 1 gives a sufficient condition to ensure that such an estimate exists.
Consequently, as long as one can obtain path length estimates (say from stochastic analysis of SGD), one can still apply Theorem 2 to obtain generalization estimates.
Path length estimates for other types of training algorithms is an interesting future direction.
A PPLICATIONS
In this section, we apply the framework to three models.
To obtain clean expressions of the generalization bound in terms of the sample size n , it is helpful to consider a range of n relating to the dimension d .
In particular, we consider underdetermined systems where the ratio n/d remains finite unless stated otherwise:
[Equation]
Note that this setting is non-asymptotic since we do not require d to be infinite.
For each application, to simplify the analysis, we evaluate the test error on a loss function ˜
ℓ : R × R −→ [0 , 1] is 1 -Lipschitz (on the first argument) with ˜ ℓ ( y, y )
= 0 .
The following steps are taken in our framework:
Step 1.
Establish the Uniform-LGI property and find the Uniform-LGI constants c n and θ n .
Step 2.
Apply Theorem 1 to get optimization results to estimate path length.
Step 3.
Apply Theorem 2 to get generalization results from the path length estimates.
4. U NDERDETERMINED ℓ p LINEAR REGRESSION
We begin with an underdetermined linear regression model f ( w, x ) =
w ⊤ x with an ℓ p loss function ( p ≥ 2 is an even positive integer):
[Equation]
where the input data matrix X
=
( x 1 , . .
, x n ) ⊤ ∈ R n × d has full row rank.
Then the above regression model has at least one global minimum with zero loss.
Target function.
Suppose the training data is generated from an underlying function g : R d −→ R with y
i =
g ( x i ) , ∀ i ∈ [ n ] .
Let Y =
( y 1 , · · · , y n ) ⊤ , and assume that there exits c ∗ > 0 such that
[Equation]
The inequality (6) actually indicates that g is Lipschitz with a dimension independent Lipschitz constant.
Functions satisfying (6) include linear/non-linear functions.
For example, g ( x ) = ϕ
( x ⊤ w ∗ ) , where w ∗ ∈ R d with ∥ w ∗ ∥ 2 ≤ c ∗ for some constant c ∗ , and ϕ ( · ) is Lipschitz with ϕ (0) = 0 .
Assumption 1.
The entries of X are i.i.d.
subgaussian random variables with zero mean, unit variance, and subgaussian moments 3 bounded by 1 .
This assumption allows us to study the spectral properties of the sample matrix by random matrix theory.
Especially, when n ( d ) /d converges to some constant γ ∈ (0 , 1) , the Marchenko–Pastur law (Mar ˇ cenko & Pastur, 1967) shows that λ min ( XX ⊤ ) /n converges to (1 − √ γ ) 2 almost surely.
The non-asymptotic results are provided in (Rudelson & Vershynin, 2010).
Performing our three-step analysis, we get the following results:
Theorem 3.
Consider the undertermined ℓ p linear regression model (5).
Suppose that there exists a universal constant c 0 ≥ 1 such that ∀ d , (cid:13)(cid:13) w (0) (cid:13)(cid:13) 2 ≤ c
[Equation]
Step 1.
L n ( w ) satisfies the Uniform-LGI globally on R d with
[Equation]
Step 2.
L n ( w ( t ) ) converges to zero linearly for p
= 2 and sublinearly for p ≥ 4 , i.e., p = 2 ,
[Equation]
where M = p 1 − 2 p ( p − 2) λ min ( XX ⊤ ) n
[Equation]
Step 3.
Under Assumption 1, for any target function that satisfies (6), we have with probability at least 1 − δ − τ d − n +1 − τ d over the samples,
[Equation]
where τ ∈ (0 , 1) depends only on the subgaussian moment of the entries.
The proof of Theorem 3 is given in Appendix D.1.
This theorem shows that compared to the PL condition that corresponds to p = 2 , the uniform-LGI is more general and can be applied to more cases.
Comparison.
This result is related to (Bartlett et al., 2020) that studied the phenomenon of benign overfitting in high-dimensional ℓ 2 linear regression.
Our result coincides with theirs as both results uncover some scenarios for benign overfitting in linear regression.
In particular, (Bartlett et al.,
2020) focused on the minimum ℓ 2 norm estimator.
They showed that, if the eigenvalue sequence of the covariance operator Σ := E [ xx ⊤ ] have suitable decay rates, then the generalization error will decrease to zero as n increases.
In contrast, our result differs from theirs in the problem settings.
Specifically, we do not assume the minimum norm property and consider the optimization process that is neglected in (Bartlett et al., 2020).
Also, we evaluate the generalization error of the convergence point on a globally Lipschitz loss function.
In our settings, entries of each input x are i.i.d.
random variables, meaning that
Σ
= I d × d without decaying eigenvalues.
The requirement of i.i.d.
entries for each input is a limitation due to the lack of non-asymptotic results of the spectral properties for random matrices with non-i.i.d.
entries.
However, this limitation can be overcome if one considers the asymptotic results.
Moreover, our result works for ℓ p loss functions with any even positive integer p .
This expands the understanding of benign overfitting in the high-dimensional setting.
4. K ERNEL R EGRESSION
Consider a positive definite kernel k : X × X −→ R with its corresponding feature map φ : R d −→ F satisfying ⟨ φ ( x ) , φ ( y ) ⟩ F = k ( x, y ) .
We assume that | k
( x, x ) | ≤ 1 , ∀ x ∈ X .
Let H be the reproducing kernel Hilbert space (RKHS) with respect to k .
If F = R s , then the kernel regression model with ℓ p loss is to solve the following problem
[Equation]
where p ≥ 2 is an even integer.
Similar to the ℓ p linear regression case, we consider the following target function class:
Target function.
Suppose the training data is generated from an underlying function g : R d −→ R with y
i =
g ( x i ) , ∀ i ∈ [ n ] .
We further assume that there exits a constant c ∗ > 0 such that
[Equation]
where k ( X , X ) is the n × n kernel matrix on X with k ( X , X ) ij
= k
( x i , x j ) .
Here, we list an example of class of functions that satisfies (8): g ( x ) =
ϕ
( φ ( x ) ⊤ w ∗ ) where w ∗ ∈ R s with ( ∀ s ) ∥ w ∗ ∥ 2 ≤ c ∗ for some constant c ∗ , and ϕ ( · ) is Lipschitz with ϕ (0) = 0 .
To get the generalization results of kernel regression, we will discuss two types of kernels separately: radial basis function (RBF) kernel and inner product kernel.
RBF kernel.
We study the RBF kernel of the form k
( x, y )
=
ϱ ( ∥ y − x ∥ ) for a certain RBF ϱ .
For the input data, we define the separation distance of X as SD :
= 12 min i ̸
= j ∥ x i − x j ∥ , ∀ i, j ∈ [ n ] .
Inner product kernel.
For the inner product kernel, we consider k ( x, y )
=
ϱ (cid:16) x ⊤ yd (cid:17) .
Following (El Karoui et al., 2010), we make the following assumption on the function ϱ :
Assumption 2.
ϱ is C 3 in a neighborhood of 0 with ϱ (0) = 0 , ϱ (1) > ϱ ′ (0) ≥ 0 , ϱ ′′ (0) ≥ 0 .
Now we are ready to apply our three-step analysis to get optimization and generalization results.
For the RBF kernel, the generalization result depends on the separation distance of the samples.
For the inner product kernel, we study the high-dimensional random kernel matrix.
Theorem 4.
Consider the kernel regression model (7).
Suppose that there exists a universal constant c 0 ≥ 1 such that ∀ s , (cid:13)(cid:13) w (0) (cid:13)(cid:13) 2 ≤ c 0 .
Step 1.
L n ( w ) satisfies the Uniform-LGI globally on R s with
[Equation]
where c n is controlled by the kernel and input samples.
Step 2.
L n ( w ( t ) ) converges to zero linearly for p
= 2 and sublinearly for p ≥ 4 , i.e.,
[Equation]
where M = p 1 − 2 p ( p − 2) λ min ( k ( X , X ))
n
[Equation]
Step 3.
For any target function that satisfies we have
: • For the RBF kernel 4 , suppose that ϱ : R ≥
−→ R ≥ 0 is a decreasing function and ϱ ( ∥ x ∥ )
∈ L 1 ( R d ) .
If there exists two positive constants q min and q max such that SD ∈ [ q min , q max ] for all n , then with probability at least 1 − δ over the samples,
[Equation]
• For the inner product kernel, under Assumption 1 and Assumption 2, if d is large enough and δ >is small enough such that d − 1 / 2 (cid:0) √ 3 δ
[Equation] [Equation]
2 +
[Equation]
(cid:1)
[Equation] [Equation]
ϱ
[Equation]
[Equation]
− ϱ ′ (0)) , then with probability at least 1 − δ − d − 2 over the samples, we have
[Equation]
The proof
of Theorem 4 is given in Appendix
D.
Example 1.
RBF kernels satisfying the conditions and assumptions in Theorem 4 include (1)
Gaussian: ϱ ( r ) =
e − ρr 2 , ρ > 0 ; (2) (inverse)
Multiquadrics: ϱ ( r ) =
( ρ + r 2 ) β/ 2 , ρ > 0 , β ∈ R \ 2 N , β < − d .
Inner product kernels satisfying the conditions and assumptions in Theorem 4 include (1) Polynomial kernel: ϱ ( r ) =
r β , β ∈ Z + , β ≥ 2 ; (2) NTK corresponding to Two-layer ReLU neural networks on
S d − 1 ( √ d ) : ϱ ( r ) =
r ( π − arccos( r )) 2 π .
Comparison.
The p = 2 result of the inner product kernel is related to (Liang & Rakhlin, 2020) who derived generalization bounds for the minimum RKHS norm estimator.
They showed that when the data covariance matrix and the kernel matrix enjoy certain decay of the eigenvalues, the generalization bound vanishes as n goes to infinity.
For example, for exponential kernel and the covariance matrix Σ :=
E [ xx ⊤ ] with the j -th eigenvalue λ j (Σ) =
j − α , the ℓ 2 generalization bound becomes O ( n − α 2 α +1 ) when α ∈ (0 , 1) and n > d .
In comparison, we get an optimal generalization bound O (cid:0) n − 1 / 2 (cid:1) for identity covariance matrix on a globally Lipschitz loss function.
Again, we emphasize that the i.i.d.
assumption can be relaxed if more new results of random matrix theory are available.
Further, we extend the works in (Liang & Rakhlin, 2020) by proving a new result of the
RBF kernel.
Note that the result of the RBF kernel is not under the high-dimensional setting; thus it is not a direct adaptation of (Liang & Rakhlin, 2020), and the proof itself is of independent interest.
4. O VERPARAMETERIZED TWO - LAYER N EURAL N ETWORKS
The first two applications are on traditional machine learning models.
Indeed, our framework can be applied not only to linear/kernel regression but also to neural network models.
In this subsection, we use our framework to study shallow neural networks in an overparameterization regime.
First, define a two-layer ReLU neural network under the NTK parameterization (Lee et al., 2019):
[Equation]
where ϕ ( x ) = max { 0 , x } , x ∈ R d is the input, W 1 ∈ R m × d , w 2 ∈ R m are parameters, w = vec ( { W 1 , w 2 } )
∈
R m ( d +1) , and m is the width (number of hidden units).
We consider minimizing the quadratic loss by gradient flow:
[Equation]
Random initialization.
W (0) is drawn from Gaussian N (0 , 1 d
I m × d ) and w (0)2 ,i are drawn i.i.d.
from uniform distribution U {− 1 , 1 } , ∀ i ∈ [ m ] .
Following the setting in (Du et al., 2019), we only train the hidden layer W 1 and leave the output layer w 2 as random initialization to simplify analyses.
NTK matrix.
The NTK matrix Θ( t ) is defined as: Θ ij (
[Equation]
and denote (cid:98) Θ by the limiting matrix
[Equation]
Similarly, we apply our three-step analysis to derive the following results.
Theorem 5.
Consider the two-layer ReLU neural network model (9).
For any δ ∈ (0 , 1) , if m ≥ poly (cid:16) n, λ − 1min ( (cid:98) Θ) , δ − 1 (cid:17) , then
Step 1.
With probability at least 1 − δ over training samples and random initialization, L n ( w ( t ) ) satisfies the Uniform-LGI on (cid:8) w ( t ) : t ≥ 0 (cid:9) with
[Equation]
Step 2.
L n ( w ( t ) ) converges to zero with a linear convergence rate:
[Equation]
Step 3.
Under Assumption 1, for any target function that satisfies (6), if γ 1 ∈ (0 , 1) , then with probability at least 1 − δ − τ d − n +1 − τ d over the samples and random initialization,
[Equation]
where τ ∈ (0 , 1) depends only on the subgaussian moment of the entries.
The proof of Theorem 5 is deferred to Appendix D.3.
Comparison.
Our result is related to (Arora et al., 2019), which gave an NTK-based generalization bound for overparameterized two-layer ReLU neural networks.
This result matches with theirs in the sense that we discover some underlying functions that are provably learnable.
Examples of learnable target functions in (Arora et al., 2019) include polynomials
y = ( β ⊤ x ) p , non-linear activations y = cos( β ⊤ x ) − 1 , y =
˜ ϕ
( β ⊤ x ) with ˜ ϕ
( z ) =
z · arctan( z/ 2) , ∥ β ∥ ≤ 1 .
Our result, furthermore, expands the target function class that is provably learnable since we only require ˜ ϕ to be Lipshcitz.
In addition, they set the standard deviation of the initialization to be at most O (1 /n ) , whereas we use a different initialization with order O (1 / √ d ) that is more often applied in practice.
Our result is also related to (Liu et al., 2020), which proved that overparameterized deep neural networks satisfy the
PL condition.
Further, we extend this work by analyzing the generalization based on the length of optimization trajectories.
C ONCLUSION
In this work, we address when and why does an optimization algorithm finds a minimum with good generalization performance.
For this problem, we propose a framework to bridge optimization and generalization based on the trajectory length.
The pivotal component is the Uniform-LGI property: a condition on the loss function, by which we show that gradient flow returns a global minimum with an explicit length estimate.
Further, we derive a length-based generalization bound given such a length estimate.
Finally, we apply the framework to three machine learning models with certain target functions.
By estimating the trajectory lengths, we get non-vacuous generalization bounds that do not suffer from the curse of dimensionality.
This framework is not a direct variant of the NTK method, and the results show that our framework is favorable for inducing connections between optimization and generalization.
R EPRODUCIBILITY S TATEMENT
For the experiments, details about the models and the synthetic data are described in Appendix A. and Appendix A.2.
For the theoretical results, we have included clear explanations of all assumptions.
The complete proofs are provided in Appendix B, C & D.
A E XPERIMENTS A. E XPERIMENTS SETTINGS
In this subsection, we describe the details of our experiments in the introduction part (Figure 1).
Linear Regression model.
The model that we use is underdetermined ℓ 2 linear regression without regularization.
The data set consists of 130 data points { ( x i , y i ) } 130 i =1 ⊆ R 200 × R .
The inputs x i ( ∀ i ∈ [130]) are uniformly drawn from the 200 -dimensional unit sphere.
All labels y
i
( ∀ i ∈ [130]) are generated by a linear target function y i =
β ⊤ x i with some β ∈ R 200 that satisfies ∥ β ∥ = 1 .
We train this model by gradient descent w ( k +1) =
w ( k ) − η ∇L n ( w ( k ) )
with step size η = 0 .on
the mean square loss (MSE).
Entries of w (0) are initialized i.i.d.
from Gaussian distribution
N ( µ, σ 2 ) with µ = 2 − 5 , 2 − 2 , 2 1 , 2 4 and σ = 2 − 5 , 2 − 2 , 2 1 , 2 4 respectively.
We stop the training once the difference of loss between two consecutive steps |L n ( w ( K +1) )
− L n ( w ( K ) )
| < 10 − 8 .
Then len( w (0) , ∞ ) is approximated by (cid:80) Kk =1 ∥ w ( k ) − w ( k − 1) ∥ .
For each pair of µ and σ , we record the length len( w (0) , ∞ ) , the training error after convergence and the test error on the convergence point.
Random feature model.
In this experiment, the model that we consider is a two-layer ReLU neural network w ⊤ 2 ϕ
( W 1 x ) .
Here ϕ is the ReLU function, W 1 ∈ R 200 × 200 , w 2 ∈ R 200 , We only train the last layer w 2 , and thus this can be viewed as a random feature model with features extracted by ϕ ( W 1 x ) .
The data set consists of 130 data points { ( x i , y i ) } 130 i =1 ⊆ R 200 × R .
The inputs x i ( ∀ i ∈ [130]) are uniformly drawn from the 200 -dimensional unit sphere.
All labels y
i
( ∀ i ∈ [130]) are generated by a teacher network (cid:98) w ⊤ 2 ϕ ( (cid:99) W 1 x i ) with the same architecture for some (cid:98) w 2 , (cid:99) W 1 .
We train only the top layer by gradient descent with momentum 0 .
9 and step seize 0 .
under the mean square loss (MSE).
Entries of w (0)2 are initialized i.i.d.
from N ( µ, σ 2 ) with µ = − 5 , − 2 , 1 , 4 and σ = 2 − 5 , 2 − 2 , 2 1 , 2 4 respectively.
We stop the training once the difference of loss between two consecutive steps is less than 10 − 8 .
For each pair of µ and σ , we use the same method as on linear regression to approximate the trajectory length.
Then we record the training error after convergence and the test error on the convergence point.
Varied learning rate.
In this experiment, we provide additional results by only changing the learning rate.
Figure 2, 3 and 4 correspond to step size η = 0 .
5 respectively.
We observe that the optimization path lengths are nearly the same for both small learning rate and moderate learning rate, and the numerical results are associate with our theory.
A. MNIST CLASSIFICATION
In this subsection, we calculate the trajectory length on the MNIST classification problem to show the relation between optimization and generalization under different initializations.
Experiment settings.
In this experiment, we train a two-layer ReLU neural network with 100 hidden units.
We use stochastic gradient descent with mini-batch 64 , momentum 0 .
9 and initial learning rate 0 .
1 to train the model.
Parameters of the hidden layer are initialized by standard random initialization method in Pytorch, and the parameters of the top layer are initialized i.i.d.
from Gaussian distribution
N ( µ, σ 2 ) with µ = 2 − 5 , 2 0 , 2 5 and σ = 2 − 5 , 2 − 2 , 2 1 , 2 4 respectively.
During training the model, we reduce the learning rate by a factor 0 .
1 once learning stagnates.
We stop the training once the cross-entropy loss decreases to 0 .
01 or the number of epochs reaches 1000 .
For each pair of µ and σ , we calculate the trajectory length by (cid:80) Kk =1 (cid:13)(cid:13) w ( k ) − w ( k − 1) (cid:13)(cid:13) .
We plot the relation between optimization and generalization in terms of the trajectory length in Figure 5.
From this figure, we can see that for different random initializations, short optimization paths is associated with good generalization gap.
This numerical result exhibits the generality of our theory and suggests that the path length plays an important role to connect optimization and generalization.
B P ROOF OF T HEOREM
In this section we will prove Theorem 1, our proof is based on the next lemma, showing that the gradient flow trajectory is always inside S n .
Lemma B.1.
For any initialization w (0) , if L n ( w ) satisfies Uniform-LGI on a closed set S n ⊇
B (cid:0) w (0) , r n ( w (0) ) (cid:1) with constants c n , θ n , where r n ( w (
[Equation] [Equation]
, then
[Equation]
Proof.
Let
[Equation]
then it is sufficient to prove that T = ∞ .
Otherwise, if T < ∞ , then by the continuity of the curve { w ( t ) } t ≥ 0 , we know that w ( T ) is in the boundary of S n , therefore
[Equation]
Now we consider the following two cases.
Case (i).
L n ( w ( T ) )
=
min w L n ( w ) .
Since L n ( w ( t ) ) is non-increasing, we have L n ( w ( t ) )
=
min w L n ( w ) and w ( t ) =
w ( T ) for t ≥ T .
Notice that S n is a closed set, thus w ( t ) ∈ S n for t ∈ [0 , T ] , meaning that T = ∞ , which contradicts with T < ∞ .
Case
(ii).
L n ( w ( T ) ) > min w L n ( w ) .
By chain rule, we
have
for
all
t ∈
0 , T ] ,
[Equation]
=
[Equation]
dt (cid:29)
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
( t )
[Equation]
(cid:13)(cid:13)(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) dw ( t ) dt (cid:13)(cid:13)(cid:13)(cid:13) ≤
[Equation]
(cid:17) θ n (cid:13)(cid:13)(cid:13)(cid:13) dw ( t ) dt
(cid:13)(cid:13)(cid:13)(cid:13) .
Then we can bound the trajectory length len ( w (0) , T ) as len
[Equation] [Equation]
=
[Equation]
11) which contradicts with (10).
Now we begin to prove Theorem 1.
Proof.
By Lemma B.1, we know that L n ( w ( t ) ) satisfies Uniform-LGI for all t ∈ [0 , ∞ ) .
Then by the proof of Lemma B.1, we have ∀ t ∈ [0 , ∞ ) , len
[Equation]
For the convergence rate, note that
[Equation]
Therefore
[Equation]
Integrating on both sides of the equation, we can get ∀ t ∈ [0 , ∞ ) , when θ n = 12 ,
[Equation] [Equation]
14) when 12 < θ n < 1 ,
[Equation]
(15) where M = c 2 n (2 θ n − 1) (cid:0) L n ( w (0) )
− min w L n ( w ) (cid:1) 2 θ n − 1 .
Taking the limit on both sides of (14) and (15), since w ( ∞ ) is the limit of w ( t ) , we have
L n
[Equation]
Hence w ( ∞ ) is a global minimum.
Taking the limit on both sides of (12), we get len
[Equation]
This completes the proof.
C P ROOF OF T HEOREM
In this section, we will prove Theorem 2.
This proof is based on the Rademacher complexity theory and the covering number of ℓ 2 balls.
Now we introduce some known technical lemmas that are used to build our proof.
The first lemma gives a generalization bound of a function class based on the Rademacher complexity, which is proved in (Mohri et al., 2018).
Lemma C.1.
Consider a family of functions F mapping from Z to [ a, b ] .
Let D denote the distribution according to which samples are drawn.
Then for any δ > 0 , with probability at least 1 − δ over the draw of an i.i.d.
sample S = { z 1 , . . .
, z n } of size n , the following holds for all f ∈ F :
[Equation]
where R S ( F ) is the empirical Rademacher complexity with respect to the sample S , defined as:
[Equation]
Here { σ i } ni =1 are i.i.d.
random variables drawn from U {− 1 , 1 } .
In the next lemma, we prove a shifted version of the Ledoux-Talagrand contraction inequality (Ledoux & Talagrand, 2013), which is useful to bound the length-based Rademacher complexity.
Lemma C.2.
Let g : R −→ R be a convex and increasing function.
Let ϕ
i : R −→ R be L -Lipschitz functions, then for any bounded set T ⊂ R and any t (0)
∈ R , we have
E σ 
[Equation]

[Equation]
and
E σ   sup
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
i
[Equation] [Equation]
(cid:17)(cid:17)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]

[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n
[Table]
= σ i (cid:16) t i − t
[Table]
i
[Table]
The special case for t (0) = 0 corresponds to the original Ledoux-Talagrand contraction inequality.
Here we prove a shifted version.
Proof.
First notice that
E σ   g   sup
[Equation]
(cid:17)(cid:17)
[Equation]
(cid:17)(cid:17) .
Let u n − 1 ( t ) =
(cid:80) n − 1 i =
σ i (cid:16) ϕ i ( t i ) − ϕ i (cid:16) t (0) i (cid:17)(cid:17) , then
E σ n   g   sup ( t − t (0) )
∈ T n (cid:88) i =
σ i (cid:16) ϕ i ( t i ) − ϕ i (cid:16) t (0)
i (cid:17)(cid:17) = 2 g   sup ( t − t (0) )
∈
T u n − 1 ( t ) + (cid:16) ϕ n ( t n ) − ϕ n (cid:16) t (0) n (cid:17)(cid:17)  +
[Equation]
g   sup ( t
[Equation]
t (0) )
∈ T u n −
[Equation]
Suppose that the above two suprema can be reached at t ′ and ˜ t respectively,
i
[Equation]
e.,
[Equation]
Otherwise we add an arbitrary positive number ε in the above equations.
Therefore,
[Equation]
[Table]
Without loss of generality, we assume
[Equation]
For the other cases
[Equation]
the method remains the same.
We set
[Equation]
Now our goal is to prove:
[Equation]
Considering the following four cases:
t ′ n ≥ t (0) n and ˜ t n ≥ t (0) n .
By the Lipschitzness
of ϕ n and equation (16) we know a ≥ b, b ′ ≥ b , and
[Equation]
If t ′ n ≥ ˜ t n , we can get a − b ≤ a ′ − b ′ .
By the fact that g is convex and increasing, we have g ( y + x ) − g
x ) is increasing in y for every x ≥ 0 .
Hence for x = a − b , g
[Equation]
a )
[Equation]
b
[Equation]
g
[Equation]
If t ′ n < ˜ t n , we change ϕ n into − ϕ n and switch t ′ and ˜ t , and the proof is similar.
t ′ n ≤ t (0) n and ˜ t n ≤ t (0) n .
Similarly
by
changing the signs we can
get the same result.
t ′ n ≥ t (0) n and ˜ t n ≤ t (0) n .
For this case we have a ≤ b and b ′ ≤ a ′ , so g ( a ) + g ( b ′ )
≤ g ( b ) + g ( a ′ ) .
t ′ n ≤ t (0) n and ˜ t n ≥ t (0) n .
For this case we can change ϕ n to − ϕ n , then we have a ≥ b and a ′ ≤ b ′ , and finally we get g ( a ) + g ( b ′ )
≤ g ( b ) + g ( a ′ ) .
Thus equation (17) yields that
[Equation]
[Table]
[Equation]
  
[Equation]

[Equation]
Applying the same method to σ n − 1 , . .
, σ 1 successively, we obtain the first inequality
E σ
[Equation]
For the second inequality, since | x | = [ x ] + + [ x ] − with [ x ] + = max(0 , x ) and [ x ] − = max(0 , − x ) ,
E σ 
[Equation]
≤
[Equation]
[Table]
 
[Equation]
+ 
[Equation]
i =
σ i (cid:16)
[Table] [Table]
) −
[Table]
(cid:16)
[Table]
(cid:17)(cid:17)(cid:35) − 
 =2 E σ
[Equation]
where the last equality is by [ − x ] − =
[ x ] + and σ has the same distribution with − σ .
A simple fact is that sup
[Equation]
+ .
Since max(0 , x ) is convex and increasing, then by the first inequality we have
E σ
[Equation]

 ≤ L E σ   sup ( t − t (0) )
∈ T (cid:12)(cid:12) (cid:12)(cid:12)(cid:12) n (cid:88) i =
σ i (cid:16) t i − t (0) i (cid:17)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12) 

This completes the proof.
Now we apply Lemma C.2 to bound the empirical Rademacher complexity of an element-wise distance constrained function class.
In the following lemma, all the notations are consistent with
Theorem 2 unless stated otherwise.
Lemma C.3.
Given a function class F a,b := { x (cid:55)→ f ( w, x ) : w ∈ S a,b } and sample S = { x 1 , . . .
, x n } with ∥ x i ∥ = 1 for all i ∈ [ n ] , then we have
[Equation]
Proof.
By definition,
[Equation]
Now we decompose the term f
( w, x i ) − f (0 , x i ) as: f ( w, x i ) − f (0 , x i ) =
Ψ
[Equation]
x ⊤ i α 1 , . . .
, x ⊤ i
[Equation]
p
[Equation]
β q
[Equation] [Equation]
[Equation]
Then by the above decomposition and Lemma C.2, we have
[Equation]
i =
σ i ( f ( w, x i ) − f (0 , x i )) (cid:35) ≤ L (1)Ψ ( S a,b ) E σ (cid:34) sup w ∈S a,b n (cid:88)
i =
σ i x ⊤ i α(cid:35) + · · · + L ( p + q ) Ψ ( S a,b ) E σ (cid:34) sup w ∈S a,b n (cid:88)
i =
σ i β q (cid:35) .
Notice that
E σ (cid:34) sup
[Equation]
a
[Equation]
n
[Equation]
i =
σ
[Equation]
x ⊤ i α(cid:35)
=
[Equation]
σ (cid:34) sup ∥ α 1 ∥≤ an (cid:88) i
= σ i x ⊤ i α(cid:35) ≤ a 1 E σ (cid:34)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i =
σ i x i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:35) ≤ a 1 (cid:118)(cid:117)(cid:117)(cid:117)(cid:116) E σ   (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) n (cid:88) i
=
σ i x i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   = a 1 √ n.
And
E σ (cid:34) sup w ∈S a,b n (cid:88) i
= σ i β q (cid:35) =
E σ (cid:34) sup |
β q |≤ b q n (cid:88) i =
σ i β q
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35)
[Equation]
Therefore, by the Cauchy-Schwarz inequality, we can get
[Equation]
Finally, we have
[Equation]
Lemma C.3 gives an upper bound of the Rademacher complexity based on the element-wise distance.
Notice that (cid:13)(cid:13) w ( ∞ ) (cid:13)(cid:13) ≤ (cid:13)(cid:13) w (0) (cid:13)(cid:13)
+ len ( w (0) , ∞ ) .
To obtain a length-based generalization bound, we consider to use S a,b to cover the length-constrained space { w :
∥ w ∥ ≤ R } , and then taking a union bound.
For the ℓ 2 ball covering number, we use the following result from (Neyshabur et al., 2019,
Lemma 11).
Lemma C.4.
Given any ϵ, D, β > 0 , consider the set S
Dβ = (cid:8) x ∈ R D : ∥ x ∥ ≤ β (cid:9) .
Then there exist
N sets { T i } Ni =1 of the form T i = (cid:8) x ∈ R D : | x j | ≤ α ij , ∀ j ∈ [ D ] (cid:9) such that S Dβ ⊆
(cid:83) Ni
=1 T i and (cid:13)(cid:13)
α i (cid:13)(cid:13) ≤ β (1 + ϵ ) , ∀ i ∈ [ N ] where N = (cid:0) K + D − 1 D − 1 (cid:1) and
K = (cid:24)
D (1 + ϵ ) 2 −(cid:25) .
Lemma C.5.
For any two positive integers n, k with n ≥ k , we have
[Equation]
Proof.
Note that (cid:18)
[Equation]
The last step is by
[Equation]
Now combining Lemma C.1, C.3, C.4 and C.5, we are ready to prove Theorem 2.
Proof of Theorem 2.
First we apply Lemma C.4 with ϵ = √ 2 − 1 , D = p + q , and β = M δ + R n,δ , then there exist N sets S a k ,b k such that S Dβ ⊆
(cid:83)
Nk =1 S a k ,b k and (cid:113) ∥ a k ∥ 2 + ∥ b k ∥ 2 ≤ √ 2 β , with
N = (cid:0) 2 D − 1 D − 1 (cid:1) .
Therefore, for each parameter space S a k ,b k , by Lemma C.3 we have
[Equation]
Notice that the local Lipschitz constant of ℓ in S a k ,b k is L ℓ ( S a k ,b k ) .
Hence, by Lemma C.1 and the
Ledoux-Talagrand contraction inequality, for any δ > 0 , with probability at least 1 − δ/N over the training sample, the following holds for all w ∈ S a k ,b k :
[Equation]
where M β = sup ∥ a k ∥ 2 + ∥ b k ∥ 2 ≤ 2 β 2 sup w ∈S ak,bk , ∥ x ∥≤ 1 , | y |≤ 1 ℓ ( f ( w, x ) , y ) .
Since w ( ∞ ) ∈ S Dβ ⊆ (cid:83) Nk =1 S a k ,b k , by taking the union bound over all sets S a k ,b k , we can get with probability at least 1 − δ over the initialization I the training sample,
[Equation]
Theorem 1 already showed that L n ( w ( ∞ ) )
=
min w L n ( w ) .
Thus it remains to bound the term log N .
For D = 1 , N = 1 .
For D ≥ 2 , by Lemma C.5,
[Equation]
Finally, let R = √ 2 β , we complete the proof of Theorem 2.
C. A DDITIONAL OF RESULTS FOR THE GENERALIZATION DURING TRAINING THE MODEL
Theorem 2 gives a length-based generalization bound for the final model.
In this subsection, we apply our framework to derive generalization estimates that evolves according to the length of time (number of epochs) of training by combining the length estimate obtained in Theorem 1.
The approach is to give a generalization bound for early stopping when the loss value first reaches ε ≥ 0 .
The idea is that, when there exists T > 0 such that L n ( w T ) =
ε , then by the inequality (11) in the proof of Lemma B.1, we can get an upper bound for the length len ( w (0) , T ) in terms of L n ( w (0) ) , min w L ( w ) , ε, c n , θ n .
Finally, we can get a generalization bound by our new length estimate.
To get a clean expression for the generalization bound, we assume the optimal value of the empirical loss function L n ( w ) is zero.
Then the rigorous statement is stated as follows:
Corollary 1.
Consider a training criterion of early stopping that the training is stopped once the empirical loss value first reaches ε ≥ 0 .
Then for any given ε ≥ 0 , under the notations and conditions in Theorem 1, suppose that for any δ ∈ (0 , 1) , there exists r n,δ such that r n ( w (0) ) ≤ r n,δ with probability at least 1 − δ over the initialization and the training samples.
Then, we have with probability at least 1 − δ over initialization and the training samples, the generalization error for the stopping parameter w is given by:
[Equation]
where r ε,n,δ = √ 2 (cid:16) M δ + r n,δ − ε 1 − θn c n (1 − θ n ) (cid:17) .
Remark 2.
Corollary 1 shows a trade-off between ε and the term r ε,n,δ .
The case for ε =corresponds to the combining results of Theorem 2 and Theorem 1.
Proof.
The proof is straightforward.
Notice that by the inequality (11), we can bound the length len ( w (0) , w ) as
[Equation]
where r n ( w (0) ) is specified in equation (2).
Then by the same argument in the proof of Theorem 2, we may directly replace r n ( w (0) ) with r n ( w (0) )
− ε 1 − θn c n (1 − θ n ) to get the desired bound.
D P ROOFS FOR S ECTION
In this section, our goal is to prove all the theorems in Section 4.
A crucial part of the proofs is the spectral analysis of the random matrix X .
Therefore, we start with introducing the non-asymptotic results of λ max ( XX ⊤ ) and λ min ( XX ⊤ ) from (Rudelson & Vershynin, 2010).
The first result is from (Rudelson & Vershynin, 2010, Proposition 2.4), characterizing the nonasymptotic behavior of the largest singular value of subgaussian matrices.
Lemma D.1.
Let A be an N × n random matrix whose entries are independent mean zero subgaussian random variables whose subgaussian moments are bounded by 1 .
Then for every t ≥ 0 , with probability at least 1 − 2 e − ct 2 over the randomness of the entries,
[Equation]
where c and C are two positive constants that depend only on the subgaussian moment of the entries.
The second result is from (Rudelson & Vershynin, 2009, Theorem 1.1), characterizing the nonasymptotic behavior of the smallest singular value of subgaussian matrices.
Lemma D.2.
Let A be an N × n random matrix whose entries are independent and identically distributed subgaussian random variables with zero mean and unit variance.
If N > n , then for every ε > 0 , with probability at least 1 − ( C 1 ε ) N − n +1 − c N 1 over the randomness of the entries,
[Equation]
where C 1 > 0 and c 1 ∈ (0 , 1) depend only on the subgaussian moment of the entries.
D. P ROOF OF T HEOREM
In this section, we will prove Theorem 3 based on the three steps in our framework.
All the notations are consistent with Theorem 3 unless stated otherwise.
Proof of Theorem 3.
First, we prove the result for Step 1.
For a vector a = ( a 1 , . . .
, a n ) ⊤ ∈ R n , we use a ◦ m to denote the element-wise power, i.e., a ◦ m =
( a m 1 , . . . , a mn ) ⊤ .
For the ℓ p linear regression loss function L n ( w ) , notice that
[Equation]
Then since X has full row rank, we have ∀ w ∈ R d ,
[Equation]
Therefore,
[Equation]
For Step 2, note that min w L n ( w ) = 0 , then the result can be proved by directly plugging c n and θ n into Theorem 1.
Next, we prove the result for Step 3.
By Theorem 1 and the property of the target function, we have for any w (0) that satisfies
[Equation]
Now we apply Lemma D.1 with A = X and t = (cid:113) log(4 /δ )
c , then we have with probability at least 1 − δ/ 2 over the samples,
[Equation]
where c and C are two positive constants that depend only on the subgaussian moment of the entries.
Similarly, let τ = c 1 ∈ (0 , 1) , ε = τ/C 1 > 0 , then Lemma D.2 implies that with probability at least 1 − τ d − n +1 − τ d over the samples,
[Equation] [Equation]
where C 1 > 0 and τ ∈ (0 , 1) depend only on the subgaussian moment of the entries.
Taking the union bound, we have with probability at least 1 − δ/ 2 − τ d − n +1 − τ d over the initialization and the training samples,
[Equation]
Recall for the linear regression model (5)
[Equation]
f ( w, x ) = w ⊤ x .
Thus Ψ( x ) = x is an identity function with p = 1 , q = 0 , and ∥ L Ψ ( S a,b ) ∥ = 1 for any a, b .
Since the loss function ˜ ℓ is bounded byand 1 -Lipschitz, we know that L ℓ ( S a,b ) =
M R = 1 for any a and b .
Finally by Theorem 2 and ˜ ℓ ( y, y ) = 0 , we have with probability at least 1 − δ/ 2 over the samples,
[Equation]
Combining the inequality (21), we have with probability at least 1 − δ − τ d − n +1 − τ d over training samples,
[Equation]
[Table]
≤
[Table]
p
[Table]
+ O (cid:32)(cid:114) log(1 /δ )
n (cid:33) .
This completes the proof of Theorem 3.
D. P ROOF OF T HEOREM
In this section, we will prove Theorem 4.
First, we present some useful lemmas for proving our results, and then we give the proofs of Theorem 4 for the RBF kernel and the inner product kernel separately.
For the RBF kernel k ( x, y ) =
ϱ ( ∥ y − x ∥ ) , the following two lemmas give non-asymptotic bounds for λ max ( k ( X , X )) and λ min ( k ( X , X )) based on the separation distance SD of X .
The first lemma is from (Diederichs & Iske, 2019, Lemma 3.1), providing an upper bound for λ max ( k ( X , X )) .
Lemma D.3.
For the RBF kernel, if ϱ : R ≥ 0 −→ R ≥ 0 is a decreasing function, then
[Equation]
and the sum of the infinite series in equation (22) is finite if and only if ϱ ( ∥ x ∥ )
∈ L 1 ( R d ) .
The next lemma is from (Wendland, 2004, Theorem 12.3), giving a lower bound for λ min ( k ( X , X )) .
Lemma D.4.
Suppose that k is a positive-definite RBF kernel.
If ϱ ( ∥ x ∥ ) ∈ L 1 ( R d ) , one can define the Fourier transform of ϱ as ˆ ϱ ( ω ) :
= (2 π ) − d/ 2 (cid:82) R d ϱ ( ω ) e − ix ⊤ ω dω .
With a decreasing function ϱ 0 ( M ) and two constants M d , C d defined as
[Equation]
where Γ is the gamma function.
Then a lower bound on λ min ( k ( X , X )) is given by
[Equation]
For the inner product kernel k ( x, y ) = ϱ (cid:16) x
⊤ yd (cid:17) , it is shown in (El Karoui et al., 2010) that the kernel matrix k
( X , X ) can be approximated by the linear combination of all-ones matrix 11 ⊤ , sample covariance matrix XX ⊤ and identity matrix.
To obtain non-asymptotic results on the spectra of the kernel matrix k
( X , X ) , we borrow the technique from (Liang & Rakhlin, 2020, Proposition A.2), and show the result for subgaussian entries in the next lemma.
Lemma D.5.
For the inner product kernel, suppose that the entries of X are i.i.d.
subgaussian random variables with zero mean and unit variance, then with probability at least 1 − δ − d − 2 over the entries,
[Equation]
where k lin ( X , X ) is defined as
[Equation]
Proof.
Note that the sample covariance matrix Σ d = I d × d , then by applying (Liang & Rakhlin, 2020,
Proposition A.2) with subgaussian random entries we can prove this lemma.
Lemma D.6.
Suppose that A, B ∈ R n × n are two symmetric matrices, then we have
[Equation]
Proof.
Note that for any x ∈ R n with ∥ x ∥ = 1 ,
[Equation]
By definition, we have
[Equation]
which completes the proof.
Now we are ready to prove Theorem 4.
Proof of Theorem 4.
For Step 1.
Notice that ∀ w ∈ R s ,
[Equation]
Therefore, L n ( w ) satisfies the Uniform-LGI globally on R s with
[Equation]
For Step 2.
Since k is a positive-definite kernel, and θ n = 1 − 1 /p , then min w L n ( w ) = 0 , thus by
Theorem 1 we can directly get the result.
The proof of Step 3 is two-sided.
First, since ∀ x ∈ X , ∥ φ ( x ) ∥ = (cid:112) k
( x, x ) ≤ 1 , then the kernel regression model (7) can be viewed as ℓ p linear regression on inputs φ ( X ) .
Hence, Ψ is an identity function with p = 1 , q = 0 , and ∥ L Ψ ( S a,b ) ∥ =
L ℓ ( S a,b ) =
M R = 1 for any a, b .
This means that we only need to bound the term r n ( w (0) ) / √ n .
By Theorem 1 and the property of the target function, for any w (0) that satisfies (cid:13)(cid:13) w (0) (cid:13)(cid:13) 2 ≤ c 0 , we have
[Equation]
Then for the RBF kernel, Lemma D.3 and Lemma D.4 indicate that there exists a positive constant
C ( ϱ, d, q min , q max ) that only depends on ϱ, d, q min , q max such that
[Equation]
which implies that for all initialization w (0) ,
[Equation]
Therefore, by Theorem 2 and Lemma C.1, we have with probability at least 1 − δ over training samples,
[Equation]
which completes the proof for the RBF kernel.
For the inner product kernel, to obtain an upper bound for λ max ( k ( X , X ))
λ min ( k ( X , X )) , first notice that
[Equation]
By Lemma D.6, we can get
[Equation]
Under Assumption 2, Lemma D.5 implies that
[Equation]
and
[Equation]
Thus, by equation (26) we have
[Equation]
Under Assumption 1, by equation (19)
[Equation]
we have with probability at least 1 − δ/ 3 over the samples, λ max ( XX ⊤
[Equation]
Therefore, by equation (25), with probability at least 1 − δ/ 3 over the samples, λ max ( k ( X , X )
[Equation]
≤ γ 1 ϱ ′′ ( [Equation] + ϱ ′ (
[Equation] [Equation]
By Lemma D.5, for large d and small δ such that d − 1 / 2 (cid:0) √
[Equation]
δ
[Equation]
≤ [Equation] 5( ϱ
[Equation] 1) − ϱ ′ (0)) , we have with probability at least 1 − δ/ 3 − d − 2 over the entries,
[Equation]
Then equation (27) and (28) yields that with probability at least 1 − 2 δ/ 3 − d − 2 over the samples, λ min ( k ( X , X )) ≥ 0 .
5( ϱ (1) − ϱ ′ (0)) , λ max ( k ( X , X )) ≤ γ 1 ϱ ′′ (0)
+ ϱ
[Equation]
[Equation]
ϱ
[Equation]
Hence, by equation (23), with probability at least 1 − 2 δ/ 3 − d − 2 over the samples, we have
[Equation]
Combining Theorem 2, we get with probability at least 1 − δ − d − 2 over the samples,
[Equation]
which completes the proof.
D. P ROOF OF T HEOREM
In this section, we will prove Theorem 5.
We first introduce some important lemmas for proving our final results.
Lemma D.7 shows that the smallest eigenvalue of the NTK matrix Θ( t ) has a lower bounded given the overparameterization, by which we can prove the optimization result.
In Lemma
D.8, we show that the eigenvalues of the NTK matrix are related to the data covariance matrix.
Then by combining Lemma D.11 and Lemma D.9 we can prove the generalization result.
Lemma D.7.
For any δ ∈ (0 , 1) , if m ≥ poly (cid:16) n, λ − 1min ( (cid:98) Θ) , δ − 1 (cid:17) , then with probability at least 1 − δ over the random initialization,
[Equation]
Proof.
The proof is the same as the proof of (Du et al., 2019, Lemma 3.4).
Lemma D.8.
[Equation]
Proof.
Notice that for ReLU activation ϕ , a simple fact is that
ϕ ′ ( ax ) =
ϕ ′
( x ) holds for any x ∈ R given that a > 0 .
Therefore,
[Equation]
Then
[Equation]
where ◦ is the element-wise product, and ⊙ is the Khatri-Rao product 6 .
Since (cid:0) ( X ⊤ ) ⊙ (2 k +
[Equation]
(cid:1) ⊤ ( X ⊤ ) ⊙ (2 k +2) is positive semidefinite, we have
[Equation]
which completes the proof.
In the next lemma, we adopt an inequality from (Montgomery-Smith, 1990).
Lemma D.9.
If { σ i } ni =1 are i.i.d.
drawn from U {− 1 , 1 } , then for any x = ( x 1 , . . .
, x n ) ⊤ ∈ R n , with probability at least 1 − δ over σ ,
[Equation]
The following lemma gives a sharp bound for a Chi-square variable, which is from (Laurent &
Massart, 2000, Lemma 1).
Lemma D.10.
Let ( Y 1 , . . .
, Y D ) be i.i.d.
Gaussian variables, with mean 0 and variance 1.
Then with probability at least 1 − δ over Y
[Equation]
The next lemma is quoted from (Arora et al., 2019, Lemma 5.4), giving an upper bound for the empirical Rademacher complexity if one has an accurate estimate for the distance with respect to each hidden unit.
Lemma D.11.
Given R > 0 , consider the following function class
[Equation]
with W 1 ,r ∈ R d the r -th row of W 1 .
Then for an i.i.d.
sample S = { x 1 , . . .
, x n } and every B > 0 , with probability at least 1 − δ over the random initialization, the empirical Rademacher complexity is bounded as:
[Equation]
Now we are ready to prove Theorem 5.
Proof of Theorem 5.
For Step 1.
By Lemma D.7, if m ≥ poly (cid:16) n, λ − 1min ( (cid:98) Θ) , δ − 1 (cid:17) , then with probability at least 1 − δ over the random initialization,
[Equation]
n
[Equation]
holds for any t ≥ 0 , which means that L n ( w ( t ) ) satisfies the Uniform-LGI for any t ≥ 0 with c n = (cid:113) λ min ( (cid:98) Θ) /n, θ n = 1 / 2 .
For Step 2.
By equation (13), we can directly get L n ( w ( t ) ) converges to zero with a linear convergence rate:
[Equation]
For Step 3.
By equation (12) and Lemma D.8, the length len ( w (0) , ∞ ) can be bounded as
[Equation]
By the property of the target function, we have
[Equation]
Since the entries of w 2 are drawn i.i.d.
from U {− 1 , 1 } , then by Lemma D.9, for each i ∈ [ n ] , with probability at least 1 − δ/ 6 n over w 2 ,
[Equation]
Taking the union bound over all i = 1 , 2 , . .
. , n , we have with probability at least 1 − δ over the random initialization,
[Equation]
For the Gaussian random matrix W (0) ∼ N (0 , 1 d I m × d ) , by Lemma D.10, we have with probability at least 1 − δ/ 6 over the random initialization,
[Equation]
Taking the union bound of equation (19), (20), (30) and (31), if m ≥ poly (cid:16) n, λ − 1min ( (cid:98) Θ) , δ − 1 (cid:17) , then with probability at least 1 − τ d − n +1 − τ d − 5 δ/ 6 over the samples and random initialization I , sup
[Equation]
Therefore, with probability at least 1 − τ d − n +1 − τ d − 5 δ/ 6 over the samples and random initialization,
[Equation]
n [Equation] [Equation] For the r -th row of W ( t ) 1 , we begin to bound the distance (cid:13)(cid:13)(cid:13) W ( t )
[Table]
[Table] [Table]
[Equation]
Now we apply Lemma D.11 with B = r n ( w (0) ) , R = sup w (0) ∈I , ( x,y ) ∈D (cid:113) 2 n mλ min ( (cid:98) Θ) r n ( w (0) ) , we get with probability at least 1 − δ/ 12 over the random initalization,
[Equation]
Then by equation (32), if m ≥ poly (cid:16) n, λ − 1min ( (cid:98) Θ) , δ − 1 (cid:17) , then with probability at least 1 − τ d − n +1 − τ d − 11 δ/ 12 over the samples and random initialization,
[Equation]
Finally, by Lemma C.1, we have with probability at least 1 − τ d − n +1 − τ d − δ over the samples and random initialization,
[Equation]
for some constant τ ∈ (0 , 1) that depend only on the subgaussian moment of the entries.
This completes the proof.