A N ON - PARAMETRIC R EGRESSION VIEWPOINT : G ENERALIZATION OF O VERPARAMETRIZED D EEP R E LU N ETWORK UNDER N OISY O BSERVATIONS
A BSTRACT
We study the generalization properties of the overparameterized deep neural network (DNN) with Rectiﬁed Linear Unit (ReLU) activations.
Under the nonparametric regression framework, it is assumed that the ground-truth function is from a reproducing kernel Hilbert space (RKHS) induced by a neural tangent kernel (NTK) of ReLU DNN, and a dataset is given with the noises.
Without a delicate adoption of early stopping, we prove that the overparametrized DNN trained by vanilla gradient descent does not recover the ground-truth function.
It turns out that the estimated DNN’s L 2 prediction error is bounded away from 0 .
As a complement of the above result, we show that the (cid:96) 2 -regularized gradient descent enables the overparametrized DNN to achieve the minimax optimal convergence rate of the L 2 prediction error, without early stopping.
Notably, the rate we obtained is faster than O ( n − 1 / 2 ) known in the literature.
I NTRODUCTION
Over the past few years, Neural Tangent Kernel (NTK) [Arora et al., 2019b; Jacot et al., 2018; Lee et al., 2018; Chizat & Bach, 2018] has been one of the most seminal discoveries in the theory of neural network.
The underpinning idea of the NTK-type theory comes from the observation that in a wide-enough neural net, model parameters updated by gradient descent (GD) stay close to their initializations during the training, so that the dynamics of the networks can be approximated by the ﬁrst-order Taylor expansion with respect to its parameters at initialization.
The linearization of learning dynamics on neural networks has been helpful in showing the linear convergence of the training error on both overparametrized shallow [Li & Liang, 2018; Du et al., 2018] and deep neural networks [Allen-Zhu et al., 2018;
Zou et al., 2018; 2020], as well as the characterizations of generalization error on both models [Arora et al., 2019a; Cao & Gu, 2019].
These ﬁndings clearly lead to the equivalence between learning dynamics of neural networks and the kernel methods in reproducing kernel Hilbert spaces (RKHS) associated with NTK.
Speciﬁcally, Arora et al.
[2019a] provided the O ( n − 1 / 2 ) generalization bound of shallow neural network, where n denotes the training sample size.
Recently, in the context of nonparametric regression, two papers, Nitanda & Suzuki [2020] and Hu et al.
[2021], showed that neural network can obtain the convergence rate faster than
O ( n − 1 / 2 ) by specifying the complexities of target function and hypothesis space.
Speciﬁcally, Nitanda & Suzuki [2020] showed that the shallow neural network with smoothly approximated ReLU (swish, see Ramachandran et al.
[2017]) activation trained via (cid:96) 2 -regularized averaged stochastic gradient descent (SGD) can recover the target function from RKHSs induced from NTK with swish activation.
Similarly, Hu et al.
[2021] showed that a shallow neural network with ReLU activation trained via (cid:96) 2 -regularized GD can generalize well, when the target function (i.e., f (cid:63)ρ ) is from H NTK.
Notably, the rate that the papers Nitanda & Suzuki [2020] and Hu et al.
[2021] obtained is minimax optimal, meaning that no estimators perform substantially better than the (cid:96) 2 -regularized GD or averaged SGD algorithms for recovering functions from respective function spaces.
Nevertheless, these results are restricted to shallow neural networks, and cannot explain the generalization abilities of deep neural network (DNN).
Similarly with Arora et al.
[2019a], Cao & Gu [2019] obtained the O ( n − 1 / 2 ) generalization bound, showing that the SGD generalize well for f (cid:63)ρ ∈ H NTK L , when f (cid:63)ρ has a bounded RKHS norm.
However, the rate they obtained is slower than the minimax rate we can actually achieve.
Furthermore, their results become vacuous under the presence of additive noises on the data set.
Motivated from these observations, the fundamental question in this study is as follows:
When the noisy dataset is generated from a function from H NTK L , does the overparametrized DNN obtained via ( (cid:96) 2 -regularized)
GD provably generalize well the unseen data?
We consider a neural network that has L ≥ 2 hidden layers with width m (cid:29) n .
(i.e., overparametrized deep neural network.)
We focus on the least-squares loss and assume that the activation function is ReLU.
A positivity assumption of NTK from ReLU DNN is imposed, meaning that λ ∞ > 0 , where λ ∞ denotes the minimum eigenvalue of the NTK.
We give a more formal mathematical deﬁnition of ReLU DNN in the following Subsection 2.2.
Under these settings, we provide an afﬁrmative answer to the above question by investigating the behavior of L 2 -prediction error of the obtained neural network with respect to GD iterations.
1. C ONTRIBUTIONS
Our derivations of algorithm-dependent prediction risk bound require the analysis on training dynamics of the estimated neural network through (regularized) GD algorithm.
We include these results as the
contributions of our paper, which can be of independent interests as well.
• In an unregulaized case, under the assumption λ ∞ > 0 , we show that the training loss converges to 0 at a linear rate.
As will be detailed in subsection 3.3, this is the different result from the seminal work of Allen-Zhu et al.
[2018], where they also prove a linear convergence of training loss of ReLU DNN, but under different data distribution assumption.
• We show that the DNN updated via vanilla GD does not recover the ground truth function f (cid:63)ρ ∈ H NTK L under noisy observations, if the DNN is trained for either too short or too long: that is, the prediction error is bounded away from 0 by some constant as n goes to inﬁnity.
In regularized case, we prove the mean-squared error (MSE) of DNN is upper bounded by some positive constant.
Additionally, we proved the dynamics of the estimated neural network get close to the solution of kernel ridge regression associated with NTK from ReLU DNN.
• We show that the (cid:96) 2 -regularization can be helpful in achieving the minimax optimal rate of the prediction risk for recovering f (cid:63)ρ ∈ H NTK L under the noisy data.
Speciﬁcally, it is shown that after some iterations of (cid:96) 2 -regularized GD, the minimax optimal rate (which is O (cid:0) n − d 2 d − 1 (cid:1) , where d is a feature dimension.) can be achieved.
Note that our paper is an extension of Hu et al.
[2021] to DNN model, showing that the (cid:96) 2 -regularized
DNN can achieve a minimax optimal rate of prediction error for recovering f (cid:63)ρ ∈ H NTK L .
However, we would like to emphasize that our work is not a trivial application of their work from at least two technical aspects.
These aspects are more detailed in the following subsection.
1. T ECHNICAL C OMPARISONS WITH H U ET AL . [2021]
Firstly, in the analysis of training loss of regularized shallow neural-net, Hu et al.
[2021] begin the proof by decomposing the difference between two individual predictions into two terms: one that is related with the gram matrix evaluated at each iteration of the algorithm and the perturbation term.
Henceforth, we name this decompostion as “Gram+Pert” decomposition.
This decomposition can be checked with the equality (E.2) in the supplementary PDF of Hu et al.
The key ingredients for the decomposition are (i) the simple gradient structure of the shallow neural net, and (ii) the partitioning of the nodes in the hidden-layer into two sets: a set of nodes whose activation patterns change from their initializations during training, and the complement of the set.
This construction of the sets peels off the ReLU activation in the difference so that the GD algorithm can be involved in the analysis.
However, because of the compositional structure of the network, the same nodes partitioning technique cannot be applied for obtaining the decomposition in the DNN setting with ReLU activation.
To avoid this difﬁculty, we employ a specially designed diagonal matrix (cid:101) Σ and this matrix can peel off the ReLU function for each layer of the network.
(See the deﬁnition of (cid:101) Σ in the proof of Theorem 3.5 in the Appendix.)
Recursive applications of this diagonal matrix across the entire hidden layers enable the Gram+Pert decomposition in our setting.
It should be noted that the diagnoal matrix (cid:101) Σ had been employed in Zou et al.
[2020], which analyzed the behavior of training loss of classiﬁcation problem via ReLU DNN under logistic loss.
However, since their result is dependent on different data distribution assumption under the different loss function from ours, they didn’t employ the Gram+Pert decomposition.
Thus their technical approaches are different from ours.
Secondly, Hu et al.
[2021] directly penalized the weight parameter W by adding (cid:107) W (cid:107) 2 F to the objective function.
The (cid:96) 2 -regularization solely on the W has an effect of pushing the weight towards the origin.
This makes (cid:107) W ( k ) − W (0) (cid:107) 2 ≤ O (cid:0) 1 (cid:1) 2 , allowing most activation patterns of the nodes in the hidden layer can change during the training, even in overparametrized setting.
Here, W ( k ) denotes the updated weight parameter at k th itertaion of algorithm, and (cid:107) · (cid:107) 2 denotes the spectral norm of the matrix.
Nonetheless, this doesn’t affect the analysis on obtaining the upper-bound of MSE in shallow neural net, since the network has only a single hidden layer.
In contrast, in the DNN setting, we allow the non-convex interactions of parameters across the hidden layers.
To the best of our knowledge, a technique for controlling the size of (cid:96) 2 -norm of network gradient has not been developed under this setting, yet.
We circumvent this difﬁculty by regularizing the distance between the updated and the initialized parameter, instead by directly regularizing the updated parameter.
This ensures that the updated parameter by (cid:96) 2 -regularized GD stays in a close neighborhood to its initialization, so that with heavy over-parametrization, the dynamics of network becomes linearized in parameter and we can ignore the non-convex interactions of parameters across the hidden layers.
Speciﬁcally, under suitable model parameter setting, we prove that (cid:107) W ( k ) (cid:96) − W (0) (cid:96) (cid:107) 2 ≤ (cid:101) O P (cid:0) 1 √ m (cid:1) 3 over all (cid:96) ∈ { 1 , . . .
, L } .
Here, (cid:101) O P ( · ) hides the dependencies on the model parameters; L , ω , and n .
This result allows us to adopt the so-called “Forward Stability” argument developed by Allen-Zhu et al.
[2018], and eventually leads to the control of network gradient under (cid:96) 2 sense.
1. A DDITIONAL R ELATED WORKS
There has been another line of work trying to characterize the generalizabilities of DNN under noisy observation settings.
Speciﬁcally, it has been shown that the neural network model can achieve minimax style optimal convergence rates of L 2 -prediction risk both in regression [Bauer & Kohler, 2019; Liu et al., 2019; Schmidt-Hieber, 2020] and classiﬁcation [Kim et al., 2021] problems.
Nonetheless, a limitation of the aforementioned papers is that they assume an adequate minimizer of the empirical risk can be obtained.
In other words, the mathematical proofs of their theorems do not correspond to implementable algortihms.
Recently, several papers, which study the generalization properties of neural network with algorithmic guarantees, appear online.
Speciﬁcally, Kohler & Krzyzak [2019] showed that the data interpolants obtained through DNN by vanilla GD is inconsistent.
This result is consistent with our result, but they consider the overparametrized DNN that is a linear combination of Ω( n 10 d 2 ) smaller neural network, and the activation function they consider is sigmoid function, which is smooth and differentiable.
Along this line of research, Kuzborskij & Szepesv´ari [2021] (regression) and Ji et al.
[2021] (classiﬁcation) showed that when training overparametrized shallow neural network, early stopping of vanilla GD enables us to obtain consistent estimators.
Notation.
We use the following notation for asymptotics:
For sufﬁciently large n , we write f ( n ) =
O ( g ( n )) , if there exists a constant K > 0 such that f ( n ) ≤ Kg ( n ) , and f ( n ) =
Ω( g ( n )) if f ( n ) ≥
K (cid:48) g ( n ) for some constant K (cid:48) > 0 .
The notation f ( n ) =
Θ( g ( n )) means that f ( n ) =
O ( g ( n )) and f ( n ) =
Ω( g ( n )) .
Let (cid:104) A, B (cid:105)
Tr :=
Tr ( A (cid:62) B ) for the two matrices A, B ∈ R d 1 × d 2 .
We adopt the shorthand notation denoting [ n ] := { 1 , 2 , . . . , n } for n ∈ N .
P ROBLEM F ORMULATION 2. N ON - PARAMETRIC R EGRESSION
Let X ⊂ R d and Y ⊂ R be the measureable feature space and output space.
We denote ρ as a joint probability measure on the product space X × Y , and let ρ X be the marginal distribution of the feature space X .
We assume that the noisy data-set D := { ( x i , y i ) } ni =1 are generated from the non-parametric regression model
y i
=
f (cid:63)ρ ( x i ) + ε i , where ε i i.i.d.
∼ N (0 , 1 2 ) for i = 1 , . . . , n .
Let (cid:98) f W ( k ) ( · ) be the value of neural network evaluated with the parameters W at the k -th iterations of GD update rule.
At k = 0 , we randomly initialize the weight parameters in the model following He initialization
[He et al., 2015] with a slight modiﬁcation.
Then, the L 2 prediction risk is deﬁned as the difference between two expected risks (i.e., excess risk)
R ( (cid:98) f W ( k ) ) :
= E ρ ∼ ( x , y )
[ (cid:0) y − (cid:98) f W ( k ) ( x ) (cid:1) 2 ] and R ( f (cid:63)ρ ) :
= E ρ ∼ ( x , y )
[ (cid:0) y − f
(cid:63)ρ ( x ) (cid:1) 2 ] , where f (cid:63)ρ ( x ) :=
E [ y | x ] .
Then, we can easily show the prediction risk has a following form:
[Equation]
Note that the expectation is taken over the marginal probability measure of feature space, ρ x , and the noise of the data, ε .
However, the (1) is still a random quantity due to the randomness of the initialized parameters (cid:0) W (0) (cid:96) (cid:1) (cid:96) =1 ,...,L .
2. D EEP N EURAL N ETWORK WITH R E LU ACTIVATION
Following the setting introduced in Allen-Zhu et al.
[2018], we consider a fully-connected deep neural networks with L hidden layers and m network width.
For L ≥ 2 , the output of the network f W ( · ) ∈ R with input data x ∈ X can be formally written as follows:
[Equation]
where S d − 1 is a unit sphere in d -dimensional euclidean space, σ ( · ) is an entry-wise activation function, W 1 ∈ R m × d , W 2 , . .
, W L ∈ R m × m denote the weight matrices for hidden layers and v ∈ R m × 1 denote the weight vector for the output layer.
Following the existing literature, we will consider ReLU activation function σ ( x ) = max( x, 0) , which is the most commonly used activation function by practitioners.
Random Initialization.
Each entries of weight matrices in hidden layers are assumed to be generated from (cid:0) W i,j (cid:1) (cid:96)
=1 ,...,L ∼ N (0 , 2 m ) , and entries of the output layer are drawn from v j ∼ N (0 , ωm ) .
This initialization scheme helps the forward propagation neither explode nor vanish at the initialization, seeing Allen-Zhu et al.
[2018]; Zou et al.
Note that we initialize the parameters in the last layer with variance ωm , where ω ≤ 1 is a model parameter to be chosen later for technical convenience.
Unregularized GD update rule.
We solve a following (cid:96) 2 -loss function with the given dataset
D :
[Equation]
Let W (0)1 , . . .
, W (0) L be the initialized weight matrices introduced above, and we consider a following gradient descent update rule:
[Equation]
[Equation]
where ∇ W (cid:96) (cid:0) L S ( · ) (cid:1) is a partial gradient of the loss function L S ( · ) with respect to the (cid:96) -th layer parameters W (cid:96) , and η > 0 is the learning rate of the gradient descent.
(cid:96) 2 -regularized GD update rule.
The estimator is obtained by minimizing a (cid:96) 2 -regularized function;
[Equation]
Naturally, we update the model parameters (cid:8) W D,(cid:96) (cid:9) (cid:96) =1 ,...,L via modiﬁed GD update rule:
[Equation]
The notations η 1 , η 2 are step sizes, and µ > 0 is a tuning parameter on regularization.
We adopt the different step sizes for the partial gradient and regularized term for the theoretical conveniences.
Furthermore, we add the additional subscript D to the update rule (6) to denote the variables are under the regularized GD update rule.
Recall that the W (0) D,(cid:96) are initialized parameters same with the unregularized case.
For simplicity, we ﬁx the output layer, and train L hidden layers for both unregularized and regularized cases.
M AIN T HEORY
First, we describe the neural tangent kernel (NTK) matrix of (2), which is ﬁrst proposed by Jacot et al.
[2018] and further studied by Arora et al.
[2019b]; Du et al.
[2019]; Lee et al.
[2018]; Yang [2019].
NTK matrix of DNN is a L -times recursively deﬁned n × n kernel matrix, whose entries are the inﬁnite-width limit of the gram matrix.
Let ∇ W (cid:96) (cid:2) f W (0) ( · ) (cid:3) be the gradient of the ReLU
DNN (2) with respect to the weight matrix in the (cid:96) th hidden layer at random initialization.
Note that when (cid:96) = 1 , ∇ W (cid:96) (cid:2) f W (0) ( · ) (cid:3) ∈
R m × d and when (cid:96) ∈ { 2 , . . .
, L } , ∇ W (cid:96) (cid:2) f W (0) ( · )
(cid:3) ∈
R m × m .
Then, as m → ∞ ,
[Equation]
where H ∞ L := (cid:8)
Ker ( x i , x j ) (cid:9) n i,j =1 .
Here, Ker ( · , · ) denotes a NTK function of (2) to be deﬁned as follows:
Deﬁnition 3.1.
(NTK function of (2)).
For any x , x (cid:48) ∈ X and (cid:96) ∈
[ L ] , deﬁne
[Equation]
and
[Equation]
where ˙ σ ( u ) = 1 (cid:0) u ≥ 0 (cid:1) .
Then, we can derive the ﬁnal expression of NTK function of (2) as follows:
[Equation]
The expression in (8) is adapted from Cao & Gu [2019].
As remarked in Cao & Gu [2019], a coefﬁcient 2 in Φ ( (cid:96) ) and ˙Φ ( (cid:96) ) remove the exponential dependence on the network depth L in the NTK function.
However, when compared with the NTK formula in Cao & Gu [2019], (8) is different from two aspects: (i)
An additional factor ω in (8)) comes from the difference in initialization settings of the output layer, in which Cao & Gu [2019] considers v j ∼ N (0 , 1 m ) , whereas we consider v j ∼ N (0 , ω m ) .
(ii) Φ ( L ) is not added in the ﬁnal expression of (8)), whereas it is added in the deﬁnition provided in Cao & Gu [2019].
This is because we only train the L hidden layers but ﬁx the output layer, while Cao & Gu [2019] train the entire layers of the network including the output layer.
As already been pointed by several papers, Cho & Saul [2009] and Jacot et al.
[2018], it can be proved that the NTK function (8) is a positive semi-deﬁnite kernel function.
Furthermore, Cho &
Saul [2009] prove that the expectations in Φ and ˙Φ have closed form solutions, when the covariance matrices have the form ( 1 t t 1 ) with | t | ≤ 1 :
[Equation]
Clearly, (8) is symmetric and continuous on the product space
X × X , from which it can be implied that Ker ( · , · ) is a Mercer kernel inducing an unique RKHS.
Following Ghorbani et al.
[2020], we deﬁne the RKHS induced by (8) as:
Deﬁnition 3.2.
(NTK induced RKHS).
For some integer p ∈ N , set of points { ˜ x j } p j =1 ⊂ X , and weight vector
α :=
{ α 1 , . . .
, α p } ∈ R p , deﬁne a complete vector space of functions, f : X → R ,
H NTK L := cl (cid:18)(cid:26) f ( · ) = p (cid:88)
j = α j Ker ( · , ˜ x j ) (cid:27)(cid:19) , (10) where cl ( · ) denotes closure.
In the remaining of our work, we assume the regression function f (cid:63)ρ ( x ) :=
E [ y | x ] belongs to H NTK L .
A SSUMPTIONS .
In this subsection, we state the assumptions imposed on the data distribution with some remarks.
(A1) ρ X is an uniform distribution on S d − 1 :=
{ x ∈ R d | (cid:107) x
(cid:107) 2 = 1 } , and noisy observations are assumed to be bounded.
(i
e., ρ x ∼ Unif (cid:0) S d − 1 (cid:1) , y i
=
O (1) , ∀ i ∈ [ n ] .)
(A2)
Draw n i.i.d.
samples { x i , f (cid:63)ρ ( x i ) } ni =1 from the joint measure ρ .
Then, with probability at least 1 − δ , we have λ min (cid:0) H ∞
L (cid:1) =
λ ∞ > 0 .
Remark 3.3.
• When the feature space is restricted on the unit sphere, the NTK function in (8) becomes rotationally invariant zonal kernel.
This setting allows to adopt the results of spectral decay of (8) in the basis of spherical harmonic polynomials for measuring the complexity of hypothesis space, H NTK L .
See the subsection 3.2 and references therein.
• Assumption (A2) is commonly employed in NTK related literature for proving global convergence of training error and generalization error of both deep and shallow neural network, Du et al.
[2018; 2019]; Arora et al.
[2019a].
Note that the (A2) holds as long as no two x i and x j are parallel to each other, which is true for most of the real-world distributions.
See the proof of this claim in Du et al.
3. M INIMAX RATE FOR RECOVERING f (cid:63)ρ ∈ H NTK L
The obtainable minimax rate of L 2 -prediction error is directly related with the complexity of function space of interest.
In our setting, the complexity of [Equation] can be characterized by the eigen-decay rate of the NTK function.
Since Ker ( x , x (cid:48) ) is deﬁned on the sphere, the decomposition can be given in the basis of spherical harmonics as follows:
[Equation]
where Y k,j , j = 1 , . .
, N ( d, k ) are spherical harmonic polynomials of degree k and { µ k } ∞ k
= are non-negative eigenvalues.
Recently, several researchers, both empirically [Basri et al., 2020] and theoretically [Chen & Xu, 2020; Geifman et al., 2020; Bietti & Bach, 2021], showed that, for large enough harmonic function frequency k , the decay rate of the eigenvalues µ k is in the order of Θ (cid:0) k − d (cid:1) 4 .
Given this result and the fact N ( d, k ) =
2 k + d − 3 k (cid:0) k
+ d − 3 d − 2 (cid:1) grows as k d − 2 for large k , it can be easily shown λ j = Θ (cid:0) j − dd − 1 (cid:1) , when Ker ( x , x (cid:48) )
= (cid:80) ∞
j =1 λ j φ j ( x ) φ j ( x (cid:48) ) , for eigen-values λ 1 ≥ λ 2 ≥ · · · ≥ 0 and orthonormal basis { φ j } ∞ j =1 .
Furthermore, it is a well known fact that if the eigenvalues decay at the rate λ j = Θ( j − 2 ν ) , then the corresponding minimax rate for estimating function in RKHS is O (cid:0) n − 2 ν 2 ν
+1 (cid:1) ,
[Raskutti et al., 2014; Yuan & Zhou, 2016; Hu et al., 2021].
By setting 2 ν = dd − 1 , we can see the minimax rate for recovering f (cid:63)ρ ∈ H NTK L is O (cid:0) n − d 2 d − 1 (cid:1) .
Remark 3.4.
We defer all the technical proofs of the Theorems in subsections 3 . 3 and 3 . 4 in the Appendix for conciseness of the paper.
We also provide numerical experiments which can corroborate our theoretical ﬁndings in the Appendix A.
3. A NALYSIS OF U NREGULARIZED DNN
In this subsection, we provide the results on the training loss of DNN estimator obtained via minimizing unregularized (cid:96) 2 -loss (3) and on the corresponding estimator’s L 2 -prediction risk R (cid:0) (cid:98) f k , f (cid:63)ρ (cid:1) .
Theorem 3.5.
(Optimization)
For some δ ∈ [0 , 1] , set the width of the network as m log 3 ( m ) ≥ Ω (cid:0) ω 7 n 8 L 18 λ 8 ∞ δ 2 (cid:1) , and set the step-size of gradient descent as η = O (cid:0) λ ∞ n 2 L 2 m (cid:1) .
Then, with probability at least 1 − δ over the randomness of initialized parameters W (0) :
= (cid:8) W (0) (cid:96) (cid:9)
L +1 (cid:96) =
1 with W (0) L +1 = v , we have for k = 0 , 1 , 2 , . . .
[Equation]
In other words, the training loss drops to 0 at a linear rate.
We acknowledge a series of past works Allen-Zhu et al.
[2018]; Du et al.
[2019] have similar spirits with those in Theorem 3.5.
However, it is worth noting that their results are not applicable in our problem settings and data assumptions.
Speciﬁcally, the result of Du et al.
[2019] is based on the smooth and differentiable activation function, whereas the Theorem 3.5 is about the training error of
ReLU activation function, which is not differentiable at 0 .
Furthermore, the result of Allen-Zhu et al.
[2018] relies on φ -separateness assumption stating that the every pair of feature vectors (cid:8) x i , x j (cid:9) n i (cid:54) =
j is apart from each other by some constant φ > 0 in a Euclidean norm.
In our work, the positivity assumption on the minimum eigenvalue of the NTK is imposed (i.e., λ ∞ > 0 ).
Remark 3.6.
Reducing the order of network width is deﬁnitely another line of interesting research direction.
We are aware of some works in literature, but we chose not to adopt the techniques since this can make the analysis overly complicated.
To the best of our knowledge, the paper that most neatly summarizes this line of literature is Zou & Gu [2019].
See the table in page 3 in their paper.
The order of width they obtained is Ω (cid:0) n 8 L 12 φ 8 (cid:1) , where they impose φ -separateness assumption.
Remark 3.7.
There has been an attempt to make a connection between the positivity and φseparateness assumptions.
Recently, Zou & Gu [2019] proved the relation λ ∞ =
Ω (cid:0) φn − 2 (cid:1) 5 in a shallow-neural net setting.
See Proposition 3 . 6 . of their work.
However, it is still an open question on whether this relation holds in DNN setting as well.
The results in Theorem 3.5 suggest a positive conjecture on this question.
Indeed, plugging the relation λ ∞ =
Ω (cid:0) φn − 2 (cid:1) in (11) and in the η = O (cid:0) λ ∞ n 2 L 2 m (cid:1) yield the discount factor (cid:0) 1 − Ω (cid:0) ηmφn 2 (cid:1)(cid:1) k and step-size η = O (cid:0) φ n 4 L 2 m (cid:1) , which are exactly the same orders as presented in Allen-Zhu et al.
See Theorem 1 of their ArXiv version paper for the clear comparison.
We leave the proof of this conjecture as a future work.
Theorem 3.8.
(Generalization) Let f (cid:63)ρ ∈ H NTK L .
Fix a failure probability δ ∈ [0 , 1] .
Set the width of the network as
[Equation]
, the step-size of gradient descent as η =
[Equation]
(cid:0)
[Equation]
(cid:1) , and the variance parameter ω ≤ O (cid:0)(cid:0) λ ∞ δ n (cid:1) 2 / 3 (cid:1) .
Then, if the GD iteration k ≥ Ω (cid:0) log( n ) ηmλ ∞ (cid:1) or k ≤ O (cid:0)
ηmωL (cid:1) , with probability at least 1 − δ over the randomness of initialized parameters W (0) , we have
[Equation]
This theorem states that if the network is trained for too long or too short, the L 2 -prediction error of (cid:98) f W ( k ) is bounded away from 0 by some constant factor.
Speciﬁcally, the former scenario indicates that the overﬁtting can be harmful for recovering f (cid:63)ρ ∈ H NTK L given the noisy observations.
Remark 3.9.
Readers should note that the Theorem 3.8 does not consider if the GD algorithm can achieve low prediction risk R (cid:0) (cid:98) f k , f (cid:63)ρ (cid:1) over the range of iterations ( ηmωL ) − 1 (cid:46)
k (cid:46) ( ηmλ ∞ ) − 1 log( n ) .
In the numerical experiment to be followed in Appendix A, we observe that for some algorithm iterations k ∗ , the risk indeed decreases to the same minimum as low as the (cid:96) 2 -regularized algorithm can achieve, and increases again.
This observation implies that the unregularized algorithm can achieve the minimax rate of prediction risk.
However, analytically deriving a data-dependent stopping time k ∗ in our scenario requires further studies, since we need a sharp characterization of eigen-distribution of NTK matrix of ReLU DNN, denoted as H ∞ L in this paper.
Readers can refer the Theorem 4 . 2 . of Hu et al.
[2021] in shallow-neural network and equation (6) in Raskutti et al.
[2014] in kernel regression context on how to compute k (cid:63) with the given eigenvalues of the associated kernel matrices.
3. A NALYSIS OF (cid:96) 2 - REGULARIZED DNN
In this subsection, we study the training dynamics of (cid:96) 2 -regularized DNN and the effects of the regularization for obtaining the minimax optimal convergence rate of L 2 -prediction risk.
In the results to be followed, we set the orders of model parameters µ , η 1 , η 2 in (6), and a variance parameter of output layer, ω as follows:
[Equation]
Theorem 3.10.
(Optimization) Suppose we minimize (cid:96) 2 -regularized objective function (5) via modiﬁed GD (6).
Set the network width
[Equation]
(cid:1) and model parameters as in (12).
Then, with probability at least 1 − δ , the mean-squared error follows
[Equation]
O P
[Equation]
[Equation] [Equation]
for k ≥ 0 .
Additionally, after k ≥ Ω (cid:0) ( η 2 µL ) − 1 log( n 3 / 2 ) (cid:1) iterations of (6), for some constant
C > 0 , we have
[Equation]
where we denote u D ( k ) :
=
[ (cid:98) f W ( k ) D ( x 1 ) , . .
, (cid:98) f W ( k ) D ( x n )] (cid:62) .
Several comments are in sequel.
Theorem 3.10 is, to our knowledge, the ﬁrst result that rigorously shows the training dynamics of (cid:96) 2 -regularized ReLU DNN in overparametrized setting.
Observe that the ﬁrst term on the right-hand side of the inequality (13) converges linearly to 0 , and the second term is some positive constant that is bounded away from 0 .
This implies that the MSE of regularized DNN is upper-bounded by some positive constant.
Note that we only provide the upper bound, but the results of our numerical experiments indicate that the MSE is lower-bounded by
O P (1) as well.
We leave the proof of this conjecture for the future work.
The inequality (14) states that the trained dynamics of the regularized neural network can approximate the optimal solution (denoted as g (cid:63)µ ) of the following kernel ridge regression problem:
[Equation]
15) where (cid:107) ·
(cid:107) H NTK L denotes a NTK-induced RKHS norm.
Note that the optimization problem in (15) is not normalized by sample size n .
The inequality (14) states that after approximately ( η 2 µL ) − 1 iterations of (6), the error rate becomes O P (cid:0) 1 n (cid:1) .
The approximation error is computed at the training data points under (cid:96) 2 norm.
This should be compared with the Theorem 5 .
1 of Hu et al.
[2021], where they showed that the similar approximation holds “within” a certain range of algorithm in shallow neural network setting.
In contrast, we show that the approximation holds “after” k ≥ Ω (cid:0) ( η 2 µL ) − 1 log( n 3 / 2 )) in deep neural network.
It should be noted that the difference of results comes from the regularization scheme, where we penalize the (cid:80) L(cid:96)
=1 (cid:107) W (cid:96) − W (0) (cid:96) (cid:107) 2 F , whereas Hu et al.
[2021] regularized the term (cid:107) W 1 (cid:107) 2 F .
As another important comparison, Hu et al.
[2019] showed the equivalence of a solution of kernel ridge regression associated with NTK and the ﬁrst order Taylor expansion of the regularized neural network dynamics; note, however, that the u D ( k ) in (14) is a full neural network dynamics.
Let R ( (cid:98) f W ( k ) D , f (cid:63)ρ ) be the L 2 -prediction risk of the regularized estimator (cid:98) f W ( k ) D via modiﬁed GD (6).
Next theorem states the result of generalization ability of (cid:98) f W ( k ) D .
Theorem 3.11.
(Generalization) Let f (cid:63)ρ ∈ H NTK L .
Suppose the network width
[Equation]
and model parameters are set as suggested in (12).
Then, with probability tending to 1 , we have
[Equation]
The resulting convergence rate is O (cid:0) n − d 2 d − 1 (cid:1) with respect to the training sample size n .
Note that the rate is always faster than O (cid:0) n − 1 / 2 (cid:1) and turns out to be the minimax optimal [Caponnetto &
De Vito, 2007; Blanchard & M ¨ ucke, 2018] for recovering f (cid:63)ρ ∈ H NTK L in the following sense:
[Equation]
(16) where ρ is a data distribution class satisfying the Assumptions (A1), (A2) and f (cid:63)ρ ∈ H NTK L , and inﬁmum is taken over all estimators D
→ (cid:98) f .
It is worth noting that the minimax rate in (16) is same with the minimax rate for recovering f (cid:63)ρ ∈ H NTK.
(i.e., Hu et al.
This result can be derived from the recent discovery of the equivalence between two function spaces , H NTK= H NTK L .
See Geifman et al.
[2020] and Chen & Xu [2020].
Remark 3.12.
A particular choice of µ = Θ (cid:0) n d − 1 2 d − 1 (cid:1) in (12) is for obtaining an optimal minimax rate for prediction error in Theorem 3.11.
Speciﬁcally, the order of µ determines the L 2 distance between the f (cid:63)ρ and the kernel regressor g (cid:63)µ .
That is, (cid:107) f
(cid:63)ρ − g (cid:63)µ (cid:107)
=
O P ( µn ) .
With the result H NTK=
H NTK L , the same proof of Lemma D.2.
in Hu et al.
[2021] can be applied for proving this result.
C ONCLUSION
We analyze the convergence rate of L 2 -prediction error of both the unregularized and the regularized gradient descent for overparameterized DNN with ReLU activation for a regression problem.
Under a positivity assumption of NTK, we show that without the adoption of early stopping, the
L 2 -prediction error of the estimated DNN via vanilla GD is bounded away from 0 (Theorem 3.5), whereas the prediction error of the DNN via (cid:96) 2 -regularized GD achieves the optimal minimax rate (Theorem 3.11).
The minimax rate O (cid:0) n − d 2 d − 1 (cid:1) is faster than the O ( n − 1 / 2 ) by specifying the complexities of target function and hypothesis space.
A CKNOWLEDGMENTS
This project is partially supported by the Transdisciplinary Research Institute for Advancing Data
Science (TRIAD), http://triad.gatech.edu, which is a part of the TRIPODS program at NSF and locates at Georgia Tech, enabled by the NSF grant CCF-1740776.
Authors are also partially supported by NSF grant DMS-2015363.
A N UMERICAL ILLUSTRATIONS
[Figure]
In this section, we use synthetic data to corroborate our theoretical ﬁndings.
We use the He initialization [He et al., 2015] and employ ( (cid:96) 2 -regularized)
GD as introduced in subsection 2.2.
For the experiments, we run 1000 epochs of GD and use a ﬁxed step size, setting η 1 = η 2 = 0 .
We uniformly generate n feature data x i train from S d − 1 with d = 2 and generate y i from f (cid:63)ρ ( x train i ) with ε i ∼ N (0 , 1) .
To create a function f (cid:63)ρ ∈ H NTK L , we use the deﬁnition in (10) with α ∈ Unif ( S p − 1 ) and with p ﬁxed points { ˜ x j } pj =1 ⊂ Unif ( S d − 1 ) , where p is simply set as 1 .
Note that Ker ( · , · ) in (10) can be calculated via the formulas (8) and (9) with speciﬁed network depth L .
We consider a scenario where we have a network with depth L = 8 and width m = 2000 .
The variance parameter of the output layer ( ω ) is set as 1 for unregularized and 0 .
001 for regularized cases.
In Fig 1.(a), we record the training errors of regularized networks over the GD epochs k ≤ 1000 , where we have n ∈ { 100 , 300 , 500 , 1000 , 5000 } training samples.
This aims to verify the inequality (13) that the MSE of regularized network is bounded away from 0 by some constant.
In Fig 1.(b), the prediction risks of both unregularized and regularized networks are displayed.
We approximate the risk with 1500 (cid:80) 500 j =1 (cid:0) (cid:98) f k ( x test j ) − f
(cid:63)ρ ( x test j ) (cid:1) 2 with a new test data set { x test j , f (cid:63)ρ ( x test j ) } 500 j =1 over k ≤ 1000 for both unregularized and regularized cases.
In both cases, they reach the same minimal risks, but the risk of unregularized network increase after it hits the minimal point, whereas the risk of regularized network stays stable.
Theorem 3 .
6 tells us that for the iteration less than the order O (cid:0) 1 ηmωL (cid:1) , the prediction error is bounded away from 0 .
In the experiment for unregularized case, we set η = 0 .
01 , m = 2000 , L = 8 , and ω = 1 .
Plugging in these parameters in the bound says that the minimum can be achieved within a very few iterations.
Note that the optimal risk is non-zero as long as we have ﬁnite sample sizes n , but converges to 0 at the rate O (cid:0) n − d 2 d − 1 (cid:1) .
In
Fig 1.(c), we verify that the more training sample sizes we have, the closer the risks of the regularized networks get to 0 .
The risk is evaluated at the sample sizes n = { 100 , 300 , 500 , 1000 , 5000 } .
We have to acknowledge that there is a discrepancy between our experiment setting and theory.
Speciﬁcally, due to the limited computing power, we could not run the experiment under the regime of width
[Equation]
But the prediction risk behaves similarly as expected by our theorems, which can be a partial evidence that the statement in theorems still holds in the narrower width of the network.
B P RELIMINARY N OTATIONS
Before presenting the formal proofs of Lemmas and main results, we introduce several notations used frequently throughout the proofs.
First, we denote x (cid:96),i the output of the (cid:96) th hidden layer with the input data x i after applying entry-wise ReLU activation function.
[Equation]
Denote f W ( k ) ( x ) a value of neural network (2) evaluated at the collection of network parameters
[Equation]
and
[Equation]
denotes the (cid:96) th hidden layer parameter updated by k th GD
[Table]
Partial gradient of f W ( k ) ( x ) .
We employ the following matrix product notation which was used in several other papers [Zou et al., 2018; Cao & Gu, 2019]:
[Equation]
otherwise
[Equation]
Then, the partial gradient of f W ( k ) ( x ) with respect to W ( k ) (cid:96) for 1 ≤ (cid:96) ≤ L has a following form: for i ∈ { 1 , . . . , n } ,
[Equation]
where Σ ( k ) (cid:96),i :=
[Equation]
∈
[Equation]
m and w ( k
[Equation]
,j denotes j th column of the matrix W ( k ) (cid:96) .
Gram matrix H ( k ) .
Each entries of empirical gram matrix evaluated at the k th GD update are deﬁned as follows:
[Equation]
Note that H (0) →
H ∞ L as m → ∞ which is proved in Jacot et al.
[2018]; Yang [2019]; Lee et al.
[2018]; Arora et al.
[2019b].
Perturbation region of weight matrices .
Consider a collection of weight matrices
[Equation]
such that
[Equation]
For all i ∈ { 1 , . . . , n } and (cid:96) = 1 , . . .
, L , we denote x (cid:96),i and (cid:101) x (cid:96),i as the outputs of the (cid:96) -th layer of the neural network with weight matrices W (0) and (cid:102) W , and Σ (cid:96),i and (cid:101) Σ (cid:96),i are diagonal matrices with (cid:0) Σ (cid:96),i (cid:1)
jj = 1 (cid:0) (cid:104) w (0) (cid:96),j , x (cid:96) − 1 ,i (cid:105) ≥ 0 (cid:1) and (cid:0)(cid:101) Σ (cid:96),i (cid:1)
[Equation]
= 1 (cid:0) (cid:104) (cid:101)
[Equation]
(cid:96)
[Equation]
(cid:101) x
(cid:96) − 1 ,i (cid:105) ≥ 0 (cid:1) , respectively.
C W HY IS IT HARD TO PROVE (cid:107)
[Equation]
In this subsection, we provide a heuristic argument on why it is hard to prove (cid:107) W ( k ) D,(cid:96) − W (0) D,(cid:96) (cid:107) 2 ≤
O (cid:0) 1 (cid:1) , where W ( k ) D,(cid:96) is the model parameter of (cid:96) th layer in k th iteration of algorithm.
Here, we regularize solely on the model parameter, instead on the relative to the initialization.
In this case, we can write the update rule as follows :
[Equation]
By recursively applying above equation (4 . 3) , we can write W ( k ) D,(cid:96) with respect to W (0) D,(cid:96) as follows:
[Equation]
Then
[Equation]
we can control the bound as follows:
[Equation]
We know under the initialization setting in our paper, (cid:107) W ( k ) D,(cid:96) (cid:107)
≤
O
(1) with high-probability (see Vershynin [2018]), and as long as we can prove the (cid:96) 2 -norm of gradient is bounded, then we can conclude (cid:107) W ( k ) D,(cid:96) − W (0) D,(cid:96) (cid:107) 2 ≤ O (cid:0) 1 (cid:1) .
However, we are not aware of works in which they control the size of (cid:107)∇ W (cid:96) (cid:2)
L S (cid:0) W ( k − (cid:96) − 1) D (cid:1)(cid:3) (cid:107) 2 where the non-convex interactions between model parameters across the hidden layers are allowed.
To the best of our knowledge, we know the work Allen-Zhu et al.
[2019] deals with the three layer case under this setting.
But we need further investigations on whether the techniques employed in their paper can be generalized to arbitrary L -hidden layer setting.
D U SEFUL L EMMAS
A simple fact.
Suppose v j i.i.d ∼ N (0 , ωm ) for j ∈ [ m ] .
Then, with probability at least 1 − exp [ − Ω( m )] , (cid:107) v (cid:107) 22 ≤ O ( ω ) .
Proof.
Since (cid:13)(cid:13) v 2 j (cid:13)(cid:13)
Ψ 1 ≤ O (cid:0) ωm (cid:1) for j ∈ [ m ] , where (cid:107) · (cid:107) Ψ 1 denotes a sub-exponential norm, Bernstein’s inequality for i.i.d.
centered sub-exponential random variables can be employed :
For any t ≥ 0 ,
P
[Equation]
m
[Equation]
j = (cid:18)
[Equation]
where c > 0 is an absolute constant.
Note that we used the fact centering does not hurt the subexponentiality of random variable.
Choosing t
= O (cid:0) ω (cid:1) concludes the proof.
Lemma 4.1 ( Lemma 7.1.
Allen-Zhu et al.
With probability at least 1 − O ( nL ) · exp[ − Ω( m/L )] , 3 / 4 ≤ (cid:107) x (0) (cid:96),i (cid:107) 2 ≤ 5 / 4 for all i ∈ { 1 , . . . , n } and (cid:96) ∈ { 1 , . . .
, L } .
Lemma 4.2 ( Lemma B.1.
Cao & Gu [2019] ) .
If τ ≤ O (cid:0) L − 9 / 2 [log( m )] − 3 (cid:1) , then with probability at least 1 − O ( nL ) · exp[ − Ω( mτ 2 / 3 L )] , 1 / 2 ≤ (cid:107) (cid:101) x (cid:96),i (cid:107) 2 ≤ 3 / 2 for all (cid:102) W ∈ B (cid:0) W (0) , τ (cid:1) , i ∈ { 1 , . . . , n } and (cid:96) ∈ { 1 , . . .
, L } .
Lemma 4.3 ( Allen-Zhu et al.
Uniformly over i ∈ { 1 , . . . , n } and 1 ≤ (cid:96) 1 ≤ (cid:96) 2 ≤ L , the following results hold:
( Lemma.7.3, Allen-Zhu et al.
[2018]) Suppose m ≥ Ω( nL log( nL )) , then with probability at least, 1 − O (cid:0) nL 2 (cid:1) · exp[ − Ω( mτ 2 / 3 L )] ,
[Equation] [Equation]
(cid:0) √ L (cid:1) .
( Lemma.7.4, Allen-Zhu et al.
[2018]) Suppose m ≥ Ω( nL log( nL )) , then with probability at least, 1 − O (cid:0) nL (cid:1) · exp[ − Ω( m/L )] ,
[Equation]
[Equation] [Equation]
[Equation]
(cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤ O (cid:0) √ w (cid:1) .
( Lemma.8.2, Allen-Zhu et al.
[2018]) Suppose τ ≤ O (cid:0) L − 9 / 2 [log( m )] − 3 (cid:1) .
For all (cid:102) W ∈ B (cid:0) W (0) , τ (cid:1) , with probability at least, 1 − O (cid:0) nL 2 (cid:1) · exp[ − Ω( mτ 2 / 3 L )] , (cid:13) (cid:13) (cid:13)(cid:101)
[Table] [Table]
( Lemma.8.7, Allen-Zhu et al.
For all (cid:96) ∈
[ L ] , let Σ (cid:48)(cid:48) (cid:96),i ∈ [ − 3 , 3] m ×
m be the diagonal matrices with at most s =
O ( mτ 2 / 3 L ) non-zero entries.
For all (cid:102) W ∈ B (cid:0) W (0) , τ (cid:1) , where
τ = O (cid:0) 1 L 1 . 5 (cid:1) , with probability at least 1 − O (cid:0) nL (cid:1) · exp[ − Ω( s log( m ))] ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Lemma 4.4 ( Lemma B.3.
Cao & Gu [2019] ) .
There exists an absolute constant κ such that, with probability at least 1 − O (cid:0) nL 2 (cid:1) · exp[ − Ω( mτ 2 / 3 L )] , i ∈ 1 , . . . , n and (cid:96) ∈ 1 , . . .
, L and for all (cid:102) W ∈ B (cid:0) W (0) , τ (cid:1) , with τ ≤ κL − 6 [log( m )] − 3 , it holds uniformly that
[Equation]
Lemma 4.5.
Suppose (cid:102) W ∈ B (cid:0) W (0) , τ (cid:1) and τ ≤ O (cid:0) L − 9 / 2 [log( m )] − 3 (cid:1) .
For all u ∈ R m with a cardinality (cid:107) u (cid:107) 0 ≤ s , for any 1 ≤ (cid:96) ≤ L and i ∈ { 1 , . . . , n } , with probability at least 1 − O ( nL ) · exp (cid:0) − Ω( s log( m )) (cid:1) − O ( nL ) · exp (cid:0) − Ω( mτ 2 / 3 L ) (cid:1) ,
[Equation]
≤
[Equation]
m ·
[Equation]
Proof.
Recall Lemma 4.2.
For any ﬁxed vector u ∈ R m , with probability at least 1 − O ( nL ) · exp[ − Ω( mτ 2 / 3 L )] for τ ≤ O (cid:0) L − 9 / 2 [log( m )] − 3 (cid:1) , for any 1 ≤ (cid:96) ≤ L and i ∈ { 1 , . . . , n } , we have the event T ,
[Equation]
Conditioned on this event happens, it is easy to see the random variable v (cid:62) (cid:0) (cid:81)
Lr = a (cid:101) Σ r,i (cid:102) W r,i (cid:1)
u ∼
SG (cid:0) 9 ωm
[Equation]
u (cid:107) 22 (cid:1) .
Based on this observation, we have the probability,
P
[Equation]
=
[Equation]
u
[Equation]
m
[Equation]
≤
[Equation]
(cid:102)
[Equation]
where in the last inequality, union bounds over the indices (cid:96) and i , and over the vector u ∈ R m with (cid:107) u (cid:107) 0 ≤ s are taken.
Lemma 4.6.
Suppose τ ≤ 1 CL 9 / 2 [log( m )] 3 for some constant C > 0 .
Then, for all i ∈ [ n ] and (cid:96) ∈
[ L ] , with probability at least 1 − O (cid:0) nL (cid:1) · exp[ − Ω( mτ 2 / 3 L )] , we have
[Equation]
Proof.
By using the results from Lemma 4.
[Equation]
we can control the term :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where, in the last inequality, we used the condition on τ ≤
[Equation]
Remark 4.7.
Note that the results in Lemmas 6 .
(second and ﬁfth items), 6 . 4 , 6 . 5 , 6 . 6 are in the setting of v j ∼ N (0 , ωm ) for j ∈ [ m ] .
For the notational convenience, in following Lemmas we denote f W ( k ) ( x i ) as u i ( k ) and let u ( k ) :
=
[ u 1 ( k ) , . . . , u n ( k )] (cid:62) for k ≥ 0 .
Lemma 4.8.
For some δ ∈ [0 , 1] , if m ≥ Ω (cid:0) L log( nL/δ ) (cid:1) , then with probability at least 1 − δ , (cid:107) u ( k ) (cid:107) 2 ≤ O (cid:0) √ nωδ (cid:1) for any k ≥ 0 .
Proof.
Recall the Lemma 4.2 stating that (cid:13)(cid:13)(cid:13) x ( k ) L,i (cid:13)(cid:13)(cid:13)
= O (1) for any input data x i for i ∈ [ n ] .
Also recall that v j ∼ N (0 , ωm ) for j ∈ [ m ] , x L,i ∈ R m and u i ( k ) = √ m v (cid:62) x
L,i ∼ N (cid:0) 0 , O ( ω ) (cid:1) .
Then, we have a following via simple Markov inequality: for any t ≥ 0 ,
[Equation]
Lemma 4.9.
For some δ ∈ [0 , 1] , if m ≥ Ω (cid:0) L log( nL/δ ) (cid:1) , then with probability at least 1 − δ , we have
[Equation]
Proof.
By Markov’s inequality, for any t ≥ 0 ,
[Equation]
Note that the expectation in the nominator of (22) is taken over the random noise ε and initialized parameter W (0) , v .
We can expand the nominator as follows:
[Equation]
(cid:107) u (0) − y (cid:107) 22 (cid:105) =
E W (0) , v (cid:107) u (0) (cid:107)
+ E ε (cid:107) y (cid:107)
− 2 E ε, W (0) , v (cid:104) y
(cid:62) u (0) (cid:105) .
For the convenience of notation, let y ∗ :=
[ f (cid:63)ρ ( x 1 ) , . . .
, f (cid:63)ρ ( x n )] (cid:62) and ε :=
[ ε 1 , . .
ε n ] (cid:62) .
Recall that we have y =
y ∗ + ε , and (cid:107) y ∗ (cid:107)
= O ( n ) .
Also note that by Lemma 4.1, with probability at least 1 − O ( nL ) · exp[ − Ω( m/L )] , for any i = 1 , . . . , n , (cid:107) x (0) L,i (cid:107) 2
= O (1) .
Then, we have a random variable u
i (0) = √ m v (cid:62) x
L,i ∼ N (0 , O ( ω )) .
Now, we are ready to derive the orders of three terms on the RHS of (23).
[Equation]
Combining the above three equalities, we conclude the proof.
Lemma 4.10.
Suppose τ = O (cid:0) n √ ω √ mδλ ∞ (cid:1) .
For some δ ∈ [0 , 1] such that δ ≥ O ( nL ) · exp[ − Ω( mτ 2 / 3 L )] , then with probability at least 1 − δ , we have
[Equation]
Proof.
By the deﬁnition of gram matrix H i,j ( k ) for any k ≥ 0 , we have
[Equation]
[Table]
[Equation]
≤ O (cid:32)
[Equation]
(cid:33)
[Equation]
In the second inequality, Lemmas 4.4 and 4.6 are used, and in the last inequality, τ
= O (cid:0) n √ ω √ mδλ ∞ (cid:1) is plugged in.
With this, using the fact that Frobenius norm of a matrix is bigger than the operator norm, we bound the term (cid:107) H ( k ) − H (0) (cid:107) 2 as follows:
[Equation]
Lemma 4.11.
For some δ ∈ [0 , 1] , with probability at least 1 − δ ,
[Equation]
Proof.
For some δ (cid:48) ∈ [0 , 1] , set ε
=
L 3 / 2 4 (cid:113) log( L/δ
[Equation]
) m from Theorem 3.1. of Arora et al.
[2019b].
For any ﬁxed i, j ∈ [ n ] , we have
[Equation]
After applying the union bound over all i, j ∈ [ n ] , setting δ = δ (cid:48) n 2 , and using the fact that Frobenius norm of a matrix is bigger than the operator norm, we conclude the proof.
For two positive semi-deﬁnite matrices A and B , if we write A (cid:23) B , then it means A − B is positive semi-deﬁnite matrix.
Similarly, if we write A (cid:31) B , then it means A − B is positive deﬁnite matrix.
With these notations, we introduce a following Lemma.
Lemma 4.12 ( Lemma D.6.
Hu et al.
For two positive semi-deﬁnite matrices A and B , 1.
Suppose A is non-singular, then A (cid:23)
B ⇐⇒ λ max ( BA − 1 ) ≤ 1 and A (cid:31) B ⇐⇒ λ max ( BA − 1 ) < 1 , where λ max ( · ) denotes the maximum eigenvalue of the input matrix.
Suppose A , B and Q are positive deﬁnite matrices, A and B are exchangeable, then
A
(cid:23) B =
⇒ AQA
(cid:23)
BQB .
E P ROOF OF T HEOREM 3.
For the convenience of notation, denote u i ( k ) =
f W ( k ) ( x i ) and let u ( k )
= (cid:2)
u 1 ( k ) , u 2 ( k ) , . .
, u n ( k ) (cid:3) (cid:62) .
In order to achieve linear convergence rate of the training error, (cid:107) u ( k ) − y (cid:107) 22 , we decompose the term as follows:
[Equation]
Equipped with this decomposition, the proof consists of the following steps:
Similarly with Du et al.
[2019], a term (cid:0) u ( k + 1) − u ( k ) (cid:1) is decomposed into two terms, where we denote them as I ( k )and I ( k ) 2 , respectively.
We note that the ﬁrst term I ( k )is related with a gram matrix H ( k ) and a second term I ( k ) 2 can be controlled small enough in (cid:96) 2 sense with proper choices of the step size and the width of network.
A term (cid:107) u ( k + 1) − u ( k ) (cid:107) 22 needs to be controlled small enough to ensure 2 (cid:0) u ( k ) − y (cid:1) (cid:62) (cid:0) u
( k + 1) − u ( k ) (cid:1) > (cid:107) u ( k + 1) − u ( k ) (cid:107) 22 so that the loss decreases.
It is shown that the distance between the gram matrix H ( k ) and the NTK matrix H ∞ L is close enough in terms of operator norm.
Lastly, we inductively show that the weights generated from gradient descent stay within a perturbation region B (cid:0) W (0) , τ (cid:1) , irrespective with the number of iterations of algorithm,
We start the proof by analyzing the term u ( k + 1) − u ( k ) .
Step 1.
Control on u ( k + 1 ) − u ( k ) .
Recall
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
≥ 0 (cid:1) and we introduce a diagonal matrix (cid:101) Σ ( k ) (cid:96),i , whose j th entry is deﬁned as follows:
[Equation]
With this notation, the difference x ( k +1)
L,i − x ( k ) L,i can be rewritten via the recursive applications of (cid:101) Σ ( k ) (cid:96),i : x ( k +
[Equation]
L,
i
[Equation]
Then, we introduce following notations :
[Equation]
Now
[Equation]
we can write u i ( k + 1) − u i ( k ) by noting that u i ( k ) = √ m · v T x ( k ) L,i :
[Equation]
k
[Equation]
Here, I ( k ) 1 ,i can be rewritten as follows:
[Equation]
(cid:0) u j (
[Equation]
y j (cid:1)
[Equation]
For I ( k ) 2 ,i , we need a more careful control.
First, we pay our attention on bounding the term (cid:107) v (cid:62) ( (cid:101) D ( k ) (cid:96),i − D ( k ) (cid:96),i ) (cid:107) 2 as follows: By triangle inequality, we have
[Equation]
[Equation]
We control the ﬁrst term of the right-hand side (R.H.S) in (26).
By the fourth item of the Lemma 4.3, we know (cid:107) Σ ( k ) r,i − Σ (0) r,i (cid:107) 0 ≤ O (cid:0) mτ
2 / 3 L (cid:1) and | (cid:0) Σ ( k ) r,i − Σ (0) r,i (cid:1) j,j | ≤ 1 for j ∈ [ m ] .
Then, we can plug Σ (cid:48)(cid:48) r,i = Σ ( k ) r,i − Σ (0) r,i in the inequality of the ﬁfth item of Lemma 4.3.
So, the ﬁrst term of the
R.H.S in (26) can be bounded by O (cid:0) τ 1 / 3 L 2 (cid:112) ω log( m ) (cid:1) .
The second term of the R.H.S in (26) can be similarly controlled as the ﬁrst term.
Observe that | (cid:0) Σ ( k ) r,i + (cid:101) Σ ( k ) r,i (cid:1) jj | ≤ 1 , then we have | (cid:0) Σ ( k ) r,i + (cid:101) Σ ( k ) r,i − Σ (0) r,i (cid:1) j,j | ≤ 2 for all j ∈ [ m ] .
Note that by the deﬁnition of (cid:101) Σ ( k ) r,i , we have (cid:107) (cid:101) Σ ( k ) r,i (cid:107)
= (cid:107) Σ ( k +1) r,i − Σ ( k ) r,i (cid:107) 0 ≤ (cid:107) Σ ( k +1) r,i − Σ (0) r,i (cid:107)
+ (cid:107) Σ ( k ) r,i − Σ (0) r,i (cid:107) 0 ≤
O (cid:0) mτ 2 / 3 L (cid:1) .
Thus, by the triangle inequality, we have (cid:107) Σ ( k ) r,i + (cid:101) Σ ( k ) r,i − Σ (0) r,i (cid:107)
≤ O (cid:0) mτ 2 / 3 L (cid:1) .
These observations enable us to plug Σ (cid:48)(cid:48) r,i = Σ ( k ) r,i + (cid:101) Σ ( k ) r,i − Σ (0) r,i in the inequality of the ﬁfth item of Lemma 4.3, and give the bound on the second
We have (cid:107) v (cid:62) (cid:0) (cid:101) D ( k ) (cid:96),i − D ( k ) (cid:96),i (cid:1) (cid:107) 2 ≤ O (cid:0) τ 1 / 3 L 2 (cid:112) ω log( m ) (cid:1) .
Now, we control the (cid:96) 2 -norm of the I ( k ) 2 as follows:
[Equation]
[Table]
[Equation]
[Table]
be written as follows:
[Equation]
It is worth noting that,
[Equation] [Equation]
Also, observe that | (cid:0) Σ ( k ) r,i + (cid:101) Σ ( k ) r,i (cid:1) jj | ≤ 1 for all j ∈ [ m ] , so by Lemma A.3 of Zou et al.
[2020], we know (cid:107) (cid:101) D ( k ) (cid:96),i (cid:107) 2 ≤ O (cid:0) √ L (cid:1) .
Combining all the facts, we can conclude:
[Equation]
where in the third inequality, we additionally used the fact (cid:107) v (cid:107)
= O ( ω ) with probability at least 1 − exp( − Ω( m )) , and the inequality (28).
In the last inequality, we used the assumption ω ≤ 1 .
Step 3.
λ min (cid:0) H ( k ) (cid:1) ≥ λ ∞ 2 with sufﬁciently large m .
Denote ρ ( A ) as a sprectral radius of a matrix A .
Then, we have
[Equation]
where, in the second inequality, we used a triangle inequality, λ min (cid:0) H ( k ) − H ∞
L (cid:1) + λ min (cid:0) H ∞
L (cid:1) ≤ λ min (cid:0) H ( k ) (cid:1) .
By Lemmas 4.10 and 4.11, setting m ≥ Ω (cid:18) ω 7 n 8 L 18 log 3 ( m ) λ 8 ∞ δ (cid:19) and ˜ O (cid:18)
λ 4 / 3 ∞ δ 1 / 3 n 4 / 3 L 4 (cid:19) ≤ ω ≤ 1 , we have
[Equation]
Thus, combining (30) and (31) yields that λ min (cid:0) H ( k ) (cid:1) ≥ λ ∞ 2 .
Step 4.
Concluding the proof.
Recall that
I ( k )
[Equation]
u
[Equation]
y ) .
Then observe that
[Equation]
[Equation]
We set the step size η , radius of perturbation region τ and network width m as follows,
[Equation]
With the above settings
[Equation]
we can control the (cid:107) u ( k + 1) − y (cid:107) 22 by combining (27)
[Equation]
(29) and (32) as follows,
[Equation]
So far, we have shown from Step 1 to Step 4 that given the radius of perturbation region τ has the order O (cid:0) n √ ω √ mδλ ∞ (cid:1) , then we can show the training error drops linearly to 0 with the discount factor (1 − ηmλ ∞) along with the proper choices of η and m .
It remains us to prove the iterates W ( k ) (cid:96) for all (cid:96) ∈
[ L ] generated by GD algorithm indeed stay in the perturbation region B (cid:0) W (0) , τ (cid:1) over k ≥ 0 with τ = O (cid:0) n √ ω √ mδλ ∞ (cid:1) .
Step 5.
The order of the radius of perturbation region.
We employ the induction process for the proof.
The induction hypothesis is : ∀ s ∈ [ k + 1] ,
[Equation]
First, it is easy to see it holds for s = 0 .
Now, suppose it holds for s = 0 , . .
, k , we consider s = k
+ 1 .
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where in the second inequality, we used Lemmas 4.4.
Note that since it is assumed that
W ( k ) (cid:96) ∈ B (cid:0) W (0) , τ (cid:1) , the Lemma is applicable with m ≥ Ω (cid:18)
ω 7 n 8 L 18 log 3 ( m ) λ 8 ∞
δ (cid:19) .
Similarly, since it is assumed that the induction hypothesis holds for s = 0 , . . .
, k , we can see
[Equation]
This inequality is plugged in the last inequality with
Lemma 4.9.
By combining the inequalities (33) for s ∈ [ k ] and (34), and triangle inequality, we conclude the proof:
[Equation]
Proposition 5.1.
For some δ ∈ [0 , 1] , set the width of the network as m ≥ Ω (cid:0) ω 7 n 8 L 18 log 3 ( m ) λ 8 ∞ δ 2 (cid:1) , and the step-size of gradient descent as η = O (cid:0) λ ∞ n 2 L 2 m (cid:1) .
Then, with probability at least 1 − δ over the randomness of initialized parameters W (0) , we have for k = 0 , 1 , 2 , . . . ,
[Equation]
where
[Equation]
∞ δ (cid:33) (cid:107) y − u (0) (cid:107) 2 .
Proof.
Deﬁne u
i ( k ) :
= f W ( k ) ( x i ) , then we have u ( k + 1) − u ( k ) = − ηm · H ( k ) (cid:0) u ( k ) − y (cid:1)
+ I ( k ) = − ηm · H ∞ L (cid:0) u ( k ) − y (cid:1) − ηm · (cid:0) H ( k ) − H ∞ L (cid:1)(cid:0) u ( k ) − y (cid:1) + I ( k )= − ηm · H ∞ L (cid:0) u ( k ) − y (cid:1) + e ( k ) .
By recursively applying the above equality
[Equation]
we can easily derive a following for any k ≥ 0 ,
[Equation]
Now [Equation] we want to show ξ ( k ) can be controlled in arbitrarily small number.
First, e ( k ) needs to be bounded in an (cid:96) 2 norm:
[Equation]
where, in the second inequality
[Equation]
τ =
O
[Equation]
n √ ω √
[Equation]
is plugged in (27). Equipped with the bound on (cid:107) e ( k ) (cid:107) 2 , we can easily bound the (cid:107) ξ ( k ) (cid:107) 2 as follows:
[Equation]
Note that in the third inequality, we used the result from Theorem 1 .
F P ROOF OF T HEOREM 3.
We begin the proof by decomposing the error (cid:98) f W ( k ) ( x ) − f ∗ ( x ) for any ﬁxed x ∈ Unif ( S d − 1 ) into two terms as follows: [Equation]
[Equation]
Here, we denote the solution of kernel regression with kernel H ∞
L as g ∗ ( x ) , which is a minimum RKHS norm interpolant of the noise-free data set { x i , f (cid:63)ρ ( x i ) } ni =1 .
To avoid the confusion of the notation, we write Ker ( x, X ) =
(cid:0) H ∞
L ( x, x 1 ) , . .
, H ∞
L ( x, x n ) (cid:1) n i =1 ∈ R n and let y ∗ =
[ f (cid:63)ρ ( x 1 ) , . . .
, f (cid:63)ρ ( x n )] (cid:62) .
Then, we have a following closed form solution g ∗ ( x ) as,
[Equation]
With the decomposition (37), the proof sketch of Theorem 3.8 is as follows.
Note that for any (cid:96) ∈
[ L ] , we have (cid:98) f W ( k ) ( x ) =
(cid:104) vec (cid:0) ∇ W (cid:96) (cid:2) f W ( k ) ( x ) (cid:3)(cid:1) , vec (cid:0) W ( k ) (cid:96) (cid:1) (cid:105) .
We can write the term vec (cid:0) W ( k ) (cid:96) (cid:1) with respect to vec (cid:0) W (0) (cid:96) (cid:1) ,
H ∞ L and the residual term via recursive applications of GD update rule and the result from proposition 2 . 1 .
Readers can refer (38).
Using the equality (38), we can further decompose ∆ 1 into three terms.
That is, ∆ 1 = ∆ 11 + ∆ 12 + ∆ 13 .
Then, using the boundedness of (cid:96) 2 -norm of network gradient and the fact that the size of (cid:107) ξ ( k ) (cid:107) 2 can be controlled with wide enough network, we can control the size of (cid:107) ∆ 12 (cid:107) 2 and (cid:107) ∆ 13 (cid:107) 2 aribtarily small.
In the term ∆ 2 , the g (cid:63) is an interpolant based on noiseless data.
For large enough data points, g (cid:63) converges fastly to f (cid:63) at the rate O P ( 1 √ n ) .
Lastly, the ∆ 11 is the only term that is involved with random error ε , and we show that (cid:107) ∆ 11 (cid:107) 2 is bounded away from 0 for small and large GD iteration index k .
Step 1.
Control on ∆ 1 .
For n data points (cid:0) x 1 , . . .
, x n (cid:1) and for the k th updated parameter W ( k ) , denote:
∇ W (cid:96)
[Equation]
Note that when (cid:96) = 1 , ∇ W (cid:96) (cid:2) f W ( k ) ( X ) (cid:3) ∈
R md × n and when (cid:96) = 2 , . . .
, L , ∇ W (cid:96) (cid:2) f W ( k ) ( X ) (cid:3) ∈
R m 2 × n .
With this notation, we can rewrite the Gradient Descent update rule as
[Equation]
Applying Proposition 3.
[Equation]
we can get :
[Equation]
=
[Equation] [Equation]
=
[Equation] [Equation]
First, we control (cid:96) 2 -norm of the ﬁrst term of ξ (cid:48)
( k ) as follows:
Note that (cid:107)∇ W (cid:96)
[Equation]
f W
[Equation]
j ) ( X ) (cid:3) (cid:107)
[Equation]
O (cid:0) √ nmω (cid:1) by Lemma 4.4 for 0 ≤ j ≤ k − 1 .
Then, we have
[Equation]
In the second inequality, (cid:80) ∞
[Equation]
1 j
[Equation] [Equation]
(cid:1) is used.
Then, we control (cid:96) 2 -norm of the second term of ξ (cid:48)
( k ) as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where in the second inequality, we used Lemmas 4.6 with τ = O (cid:0) n √ ω √ mδλ ∞ (cid:1) .
Now, we are ready to control ∆ 1 term.
By using the equality (38), we can decompose the term ∆as follows: Let us denote G k
= (cid:80) k − 1 j =0 ηm (cid:0)
I − ηm H ∞
L (cid:1) j .
Note that for any (cid:96) ∈
[ L ] , (cid:98) f W ( k ) ( x ) =
[Equation]
vec
[Equation]
x
[Equation]
, vec
[Equation]
k
[Equation]
(cid:105) and recall that y =
y ∗ + ε .
Then, for any ﬁxed (cid:96) (cid:48) ∈
[ L ] , we have:
[Equation]
Our goal in this step is to control (cid:107) ∆ 12 (cid:107) 2 and (cid:107) ∆ 13 (cid:107) 2 .
Then, in the third step, we will show (cid:107) ∆ 11 (cid:107)is the term, which governs the behavior of the prediction risk with respect to algorithm iteration k .
[Table]
we
[Table]
ﬁrst term
[Table]
∆
[Table]
[Equation]
where, in the second inequality, we plugged τ = O (cid:0) n √ ω √ mδλ ∞ (cid:1) in the result of Lemma 4.6 and used
Lemma 4.11.
In the last inequality, we used (cid:107) G k (cid:107) 2 ≤ O (cid:0) 1 λ ∞ (cid:1) .
Similarly, we can control the (cid:96)norm of the second term in ∆ 12 as follows:
[Equation]
We turn our attention to controlling (cid:107) ∆ 13 (cid:107) 2 .
The ﬁrst term in ∆ 13 ;
Recall that
[Equation]
≤
O
[Equation]
by Lemma 4.4.
Then, the random variable vec
[Equation]
k
[Equation]
x )
[Equation]
vec (cid:0) W (0) (cid:96) (cid:1) is simply a N (cid:0) 0 , O ( ω ) (cid:1) for 1 ≤ (cid:96) ≤ L .
A straightforward application of Chernoff bound for normal random variable and taking union bound over the layer 1 ≤ (cid:96) ≤ L yield that: with probability at least 1 − δ ,
[Equation] [Equation]
The (cid:96) 2 norm of the third term in ∆ 13 can be bounded as follows:
[Equation]
≤
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In the last inequality, we used the Lemma 4.8 and (cid:107) G k (cid:107)≤
O (cid:0)
[Equation]
(cid:1) .
By combining (39), (40), (44), (45) with (cid:13)(cid:13)(cid:13) ∇ W (cid:96) (cid:48) (cid:2) f W (0) ( x ) (cid:3)(cid:13)(cid:13)(cid:13) F ≤ O (cid:0) √ mω (cid:1) , we have a following :
[Equation]
(cid:13)(cid:13)(cid:13)(cid:0)
[Equation]
Step 2.
Control on ∆ 2 .
First, note that there is a recent ﬁnding that the reproducing kernel Hilbert spaces of NTKs with any number of layers (i.e., L ≥ 1 ) have the same set of functions, if kernels are deﬁned on S d − 1 .
See Chen & Xu [2020].
Along with this result, we can apply the proof used in
Lemma.
D.2.
in Hu et al.
[2021] for proving a following :
[Equation]
Step 3.
The behavior of L 2 risk is characterized by the term ∆ 11 .
Recall the decompositions (37) and (41), then we have:
[Equation]
Our goal in this step is mainly two-folded: (i) Control E ε (cid:107) Θ (cid:107) 22 arbitrarily small with proper choices of step-size of GD η and width of the network m .
(ii) Show that how E ε (cid:107) ∆ 11 (cid:107) 22 affect the behavior of prediction risk over the GD iterations k .
First, note that we have
[Equation] [Equation]
Second, recall Lemma 4.9 and note that over the random initialization, with probability at least 1 − δ ,
[Equation]
Now, by combining the bounds (42), (46) and (47), we have
E
[Equation]
log( nL/δ ) m · (cid:107) y
(cid:107)(cid:33) + O (cid:32) n 11 / 3 L 6 ω 13 / 3 (cid:107) y − u (0) (cid:107)m 1 / 3 λ 8 / 3 ∞
δ 1 /log( m ) (cid:33) + O (cid:32) n 5 / 3 L 4 ω 7 / 3 (cid:107) y − u (0) (cid:107)m 1 / 3 λ 14 / 3 ∞
δ 1 /log( m ) (cid:33)(cid:35) + O (cid:32) n 2 ωλ 2 ∞ δ(cid:33) + O (cid:18)n (cid:19)
≤
O (cid:18) ω
2 n 2 Lλ 2 ∞ (cid:114) log( nL/δ ) m (cid:19) + O (cid:32) n 14 / 3 L 6 ω 13 /m 1 / 3 λ 8 / 3 ∞ δ 4 / 3 log( m ) (cid:33) + O (cid:32) n 8 / 3 L 4 ω 7 /m 1 / 3 λ 14 / 3 ∞
δ 4 / 3 log( m ) (cid:33) + O (cid:32) n 2 ωλ 2 ∞ δ(cid:33) + O (cid:18)n (cid:19) , (51) where
Case 1.
When k is large, the L 2 risk is bounded away from zero by some constant.
Now we control E ε (cid:107) ∆ 11 (cid:107) 22 .
Recall the deﬁnitions (cid:107)
f
(cid:107):= (cid:82) x ∈S d − 1 | f ( x ) | 2 d x and
G k =
[Equation]
− 1 j =
[Equation]
I −
[Equation]
H ∞ L (cid:1) j .
Let us denote S =
y ∗ y ∗(cid:62) .
Then, we have
[Equation]
where
[Equation]
For the algorithm iterations k ≥ (cid:0) log( n ) ηmλ ∞ (cid:1) C 0 with some constant C 0 > 1 , we have
[Equation]
Since 1 + (cid:107) y (cid:107) 22 ≤ C 1 n for some constant C
[Equation]
we have
[Equation]
Using the ﬁrst item of Lemma (4.12) with the inequality (52)
[Equation]
we have
[Equation] [Equation]
The above inequality (53) lead to a following result :
[Equation]
It is obvious that both (cid:0) S
+ I (cid:1) − 1 − (cid:0)
I − ηm H ∞
L (cid:1) k and (cid:0) S
+ I (cid:1) − 1 − 1 n C 0 · I are positive deﬁnite matrices due to (54), and it is also easy to see that they are exchangeable.
By using the second item of Lemma (4.12), we have
[Equation]
Then, we have
E
[Equation]
where c 0 ∈ (0 , 1) is a constant and
A
[Equation]
and B
[Equation]
(cid:3)
[Equation]
dx
[Equation]
By triangle inequality with the decomposition (48) and the bound on E ε (cid:107) Θ (cid:107) 22 in (51), we have:
[Equation]
For the third term in (56), we can choose ω ≤ C 2 (cid:0) λ ∞ δ n (cid:1) 2 / 3 for some constant C 2 > 0 such that the term can be bounded by c 0 8 (cid:13)(cid:13)(cid:13)
Ker ( · , X ) (cid:0) H ∞ L (cid:1) − 1 (cid:13)(cid:13)(cid:13) 2 2 .
Similarly, the width m can be chosen large enough such that the fourth term in (56) is upper-bounded by c 0 8 (cid:107) Ker ( · , X ) (cid:0)
H ∞
L (cid:1) − 1 (cid:107) 22 .
Using the above choices of k , ω , and m , we can further bound (56):
[Equation]
Note that E ε (cid:107) (cid:98) f ∞ − g ∗ (cid:107)
= (cid:107)
Ker
[Equation]
X
[Equation]
H ∞ L (cid:1) − 1 (cid:107) 22 where g ∗ := 0 and (cid:98) f ∞ denotes the noise interpolator.
Then, by Theorem 4 . 2 . of Hu et al.
[2021], we know that E ε (cid:107) (cid:98) f ∞
− g ∗ (cid:107) 22 ≥ c 1 for some constant c 1 > 0 .
Then, we can take n large enough such that the term O (cid:0) 1 n (cid:1) is upper-bounded by c 0 c8 , and ﬁnish the proof.
Case 2.
When k is small, the L 2 risk is bounded away from zero by some constant.
Recall the deﬁnition of ∆ 11 in the decomposition (41),
[Equation]
We denote the eigen-decomposition of the matrix H ∞
L := (cid:80) ni
=
λ i v i
v i (cid:62) , then we can easily see a following:
[Equation]
By using the above inequality
[Equation]
we have
[Equation]
Recall the decompositions (37) and (41), then we have:
[Equation]
For some constant C (cid:48) 1 > 0 , let k ≤ C (cid:48) 1 (cid:0)
ηmnωL
cid:1)
such that the second term in the bound (59) can be bounded by 18 (cid:107) Ker ( · , X ) (cid:0)
H ∞ L (cid:1) − 1 y ∗ (cid:107) 22 .
For the fourth term in (59), we can choose ω ≤ C (cid:48) 2 (cid:0) λ ∞ δ n (cid:1) 2 / 3 for some constant C (cid:48) 2 > 0 such that the term can be bounded by 18 (cid:13)(cid:13)(cid:13)
Ker ( · , X ) (cid:0) H ∞ L (cid:1) − 1 y ∗ (cid:13)(cid:13)(cid:13) 2 2 .
Similarly, the width m can be chosen large enough such that the ﬁfth term in (59) is upper-bounded by 18 (cid:107) Ker ( · , X
[Equation]
Using the above choices of k , ω , and m , we can further bound (59):
[Equation] [Equation]
In the second inequality, we used (47) with triangle inequality.
In the third inequality, we can take n large enough such that the term O (cid:0) 1 n (cid:1) is upper-bounded by C
[Equation]
3 2 (cid:13)(cid:13)
f (cid:63)ρ (cid:13)(cid:13) 2 2 .
Lastly, by using the assumption that f (cid:63)ρ is a square-integrable function, we ﬁnish the proof.
G P ROOF OF T HEOREM 3.10-T RAINING ERROR
For the convenience of notation, we denote u D,i ( k ) =
f W ( k ) D ( x i ) and let u D ( k ) = (cid:2)
u 1 ,D ( k ) , . .
u n,D ( k ) (cid:3) (cid:62) .
In order to analyze the training error of (cid:96) 2 -regularized estimator, (cid:107) u D ( k ) − y (cid:107) 22 , we decompose the term as follows:
[Equation] [Equation]
Equipped with this decomposition, the proof consists of the following steps:
We decompose the decayed prediction difference u D ( k + 1) − (1 − η 2 µL ) u D ( k ) into two terms.
We note that the ﬁrst term is related with a gram matrix H D ( k ) and denote a second term as I ( k ) D . 2.
The term I ( k ) D
can
be
further
decomposed
into
three
terms, where we denote them as I ( k ) 2 ,D
I ( k ) 3 ,D and I ( k ) 5 ,D .
The crux for controlling the (cid:96) 2 -norm of the above
three terms is to utilize the results from the Appendix A.
The applications
of Lemmas in the Appendix
A
is possible, since we can inductively guarantee that (cid:107) W ( k ) D,(cid:96) − W (
) D,
[Equation]
(cid:107) 2 is sufﬁciently small enough for large enough m .
Given the decomposition (61), we further decompose it into four terms as follows:
[Equation]
In this step, we obtain the upper-bound of (cid:107) T i (cid:107) 2 for i = 1 , 2 , 3 , 4 obtained in Step 4 .
We combine the upper-bounds of (cid:107) T i (cid:107) 2 for i = 1 , 2 , 3 , 4 in step 3 and obtain the bound on (cid:107) u D ( k + 1) − y (cid:107) 22 with respect to (cid:107) u D ( k ) − y (cid:107)
and (cid:107) y (cid:107) 2 .
Lastly, we inductively show that the weights generated from regularized gradient descent stay within a perturbation region B (cid:0) W (0) , τ (cid:1) , irrespective with the number of iterations of algorithm.
We start the proof by analyzing the term u ( k + 1) − (1 − η 2 µL ) u ( k ) .
Step 1.
Dynamics of u D ( k + 1 ) − (1 − η 2 µL ) · u D ( k ) .
Recall (cid:0) Σ ( k ) D,(cid:96),i (cid:1)
jj = 1 (cid:0) (cid:104) w ( k ) D,(cid:96),j , x ( k ) D,(cid:96) − 1 ,i (cid:105) ≥ 0 (cid:1) and we introduce a diagonal matrix (cid:101) Σ ( k ) D,(cid:96),i , whose j th entry is deﬁned as follows:
[Equation]
With this notation, the difference x ( k +1) D,L,i − x ( k ) D,L,i can be rewritten via the recursive applications of (cid:101) Σ ( k ) D,(cid:96),i :
Then, we introduce following notations :
[Equation]
Now, we can write u D,i ( k + 1) − u D,i ( k ) by noting that u D,i ( k ) = √ m · v T x ( k ) D,L,i : u D,
i
[Equation] [Equation]
[Equation]
where in the second equality, we used the recursive relation (24), and in the third equality, modiﬁed
GD update rule (6) is applied.
Furthermore, I ( k ) 1 ,D,i can be rewritten as follows:
[Equation]
With I ( k ) 4 ,i = ( − η 2 µL ) · u D,i ( k ) and (64), we can rewrite (63) as follows: u D
[Equation]
i
[Equation]
k
[Equation] [Equation]
· n
[Equation]
[Table]
[Equation]
Let I ( k ) D
=
[ I ( k ) 2 ,D, 1 + I ( k ) 3 ,D,
+ I ( k ) 5 ,D, 1 , . . .
, I ( k ) 2 ,D,n + I ( k ) 3 ,D,n + I ( k ) 5 ,D, 1 ] (cid:62) .
Now, we control the bound on the (cid:13)(cid:13)(cid:13)
I ( k ) D (cid:13)(cid:13)(cid:13) 2 2 .
Recall that in Eq.
(27), we have
[Equation]
≤ O (cid:18)
[Equation]
nL
[Equation]
ωm (cid:112) log
[Equation]
(cid:107)
[Equation]
Similarly, (cid:13)(cid:13)(cid:13) I ( k ) 3 ,D (cid:13)(cid:13)(cid:13) 2 can be bounded: (cid:13)(cid:13)(cid:13) I ( k ) 3 ,D (cid:13)(cid:13)(cid:13) 2 ≤ n (cid:88) i = (cid:12)(cid:12)(cid:12) I ( k ) 3 ,D,i (cid:12)(cid:12)(cid:12) ≤ η 2 µ √ m · n (cid:88) i = (cid:34) L (cid:88) (cid:96) = (cid:13)(cid:13)(cid:13)(cid:13) v (cid:62) (cid:18) (cid:101) D ( k ) D,(cid:96),
[Table]
− D
[Table] [Table]
[Table] [Table]
≤O
[Table] [Table]
log
[Table]
m
[Table] [Table]
W
[Table]
W
[Table]
≤ O (cid:18)
[Table]
µnL
[Table] [Table]
( m
[Table]
Lastly (cid:13)(cid:13)(cid:13) I ( k ) 5 ,D (cid:13)(cid:13)(cid:13) 2 can be bounded: (cid:13)(cid:13)(cid:13) I ( k ) 5 ,D (cid:13)(cid:13)(cid:13) 2 ≤ n (cid:88) i = (cid:12)(cid:12)(cid:12) I ( k ) 5 ,D,i (cid:12)(cid:12)(cid:12) ≤ n (cid:88) i = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) η 2 µ √ m · v T
L (cid:88) (cid:96) =
D ( k ) D
[Equation]
(cid:96)
[Equation]
i W ( k ) D,
[Table]
[Equation]
O
[Equation]
where in the last inequality
[Equation]
we employed the same logic used in (44) with the Lemma 4.2 to obtain the upper-bound on the | u i,D ( k ) | .
We set the orders of the parameters µ , η 1 , η 2 , τ , and ω as follows:
[Equation]
Plugging the choices of parameters (69) with sufﬁciently large m in (66), (67) and (68) yields
[Equation]
Step 3.
Upper-bound of (cid:107) T i (cid:107) 2 on i = 1 , 2 , 3 , 4 .
First, we work on getting the upper-bound on λ max (cid:0) H D ( k ) (cid:1) .
By the Gershgorin’s circle theorem [Varga, 2004], we know the maximum eigenvalue of symmetric positive semi-deﬁnite matrix is upper-bounded by the maximum absolute column sum of the matrix.
Using this fact, we can bound the λ max (cid:0) H D
[Equation]
k
[Equation]
as :
[Equation]
Recall the decomposition (62).
Our goal is to obtain the upper-bound on T i for i = 1 , 2 , 3 , 4 .
Control on T 1 .
By using the inequality 2 η 2 µL (1 − η 2 µL ) y (cid:62) (cid:0) y − u D
[Equation]
k ) (cid:1)
[Equation]
η 2 µL (cid:107) y (cid:107)
+ η 2 µL (1 − η 2 µL ) 2 (cid:107)
y − u D ( k ) (cid:107) 2 2 , we have (cid:107) y
[Equation]
[Equation]
Control on T 2 .
Recall the equality (65).
Then, through applications of the Young’s inequality (cid:107) a + b (cid:107) 22 ≤ 2 (cid:107) a (cid:107) 22 + 2 (cid:107) b (cid:107) 22 for a, b ∈ R n , we have
[Equation]
Similarly with T 1 and T 2 , we can control T 3 and T 4 as follows:
Control on T 3 .
Recall H D ( k ) is a Gram matrix by deﬁnition.
Then, by using the fact λ min (cid:0) H D ( k ) (cid:1) ≥ 0 and Cauchy-Schwarz inequality, we have
[Equation]
Control on T 4 .
By a simple Cauchy-Schwarz and Young’s inequality
[Equation]
we have
[Equation] [Equation]
Step 4.
Upper-bound of the decomposition on training error (62).
Before getting the upper bound of the decomposition (62), we ﬁrst work on obtaining the bound of (76).
Set κ = O (cid:0) 1 n 2 (cid:1) and notice η 2 µL = O (cid:0) 1 n (cid:1) by (69), then we have
[Equation]
where in the second inequality, the Eq.
(70) is used with ( a + b ) 2 ≤ 2 a
+ 2 b 2 for a, b ∈ R , and in the last inequality, we used (cid:107) y (cid:107)
= O ( n ) and the sufﬁciently large m to control the order of the coefﬁcient terms of (cid:107) y − u D ( k ) (cid:107) 22 .
Speciﬁcally, we choose m ≥ Ω (cid:0) L 19 n 20 log 3 ( m )
δ
[Equation]
Now, by combining the inequalities (72), (73), (74), (75), (71) and (77), we obtain the upper-bound on the decomposition (62);
[Equation]
H D ( k ) (cid:1)(cid:19) · (cid:107) y
(cid:107)+ (cid:18)(cid:0) 1 + η 2 µL (cid:1)(cid:0) 1 − η 2 µL (cid:1) 2 + 2 m 2 η 21 λ max (cid:0) H D
( k ) (cid:1)
+ 4 mη 1 η 2 µL · λ max (cid:0) H D ( k ) (cid:1)(cid:19) · (cid:107) y − u D ( k ) (cid:107)
+ (cid:18)
(cid:13)(cid:13)(cid:13) I ( k ) D (cid:13)(cid:13)(cid:13)2 + 2 (cid:0) 1 − η 2 µL (cid:1) (cid:107)
y − u D ( k ) (cid:107)
cid:13)(cid:13)(cid:13) I ( k ) D (cid:13)(cid:13)(cid:13)
2 + η 2 µL (cid:13)(cid:13)(cid:13) I
( k ) D (cid:13)(cid:13)(cid:13) 2 2 (cid:19) ≤ (cid:40) 3 η 2 µL + η 22 µ 2 L 2 + O (cid:18) ωmnη 1 η 2 µL(cid:19)(cid:41) · (cid:107) y
(cid:107)+ (cid:40)(cid:0) 1 + η 2 µL + η 42 µ 4 L 4 (cid:1)(cid:0) 1 − η 2 µL (cid:1) 2 + O (cid:18) ω 2 m 2 n 2 η 21 L 2 (cid:19) + O (cid:18) ωmnη 1 η 2 µL(cid:19)(cid:41) · (cid:107) y − u D ( k ) (cid:107):= A · (cid:107) y (cid:107)
[Equation]
With the order choices of µ , η 1 and η 2 as in (69), it is easy to see the leading terms of both A and
B are same as η 2 µL =
o ( 1 n ) .
Then, by recursively applying the inequality (78), we can get the upper-bound on the training error.
[Equation]
In the last inequality, we used AB =
o (1) , B ≥ η 2 µL and (cid:107)
y (cid:107)
= O ( n ) .
Step 5.
The order of the radius of perturbation region.
It remains us to prove the radius of perturbation region τ has the order O P
[Equation]
First, recall that the (cid:96)
[Equation]
regularized GD update rule is as:
[Equation]
Similarly with the proof in the Theorem 3.
[Equation]
we employ the induction process for the proof.
The induction hypothesis is
[Equation]
It is easy to see it holds for s =
[Equation]
and suppose it holds for s = 0 , 1 , . .
, k , we consider k +1 .
Using the update rule (80), we have
[Equation]
In the ﬁrst inequality, we use the induction hypothesis for s = k , and Lemma 4.4.
In the second inequality, since the induction hypothesis holds for s = 0 , 1 , . . .
, k , we employ (cid:107)
y − u D ( k ) (cid:107) 2 ≤
O (cid:0) √
[Equation]
k 2 (cid:107) y − u D (0) (cid:107) 2 with the Lemma 4.9.
In the last inequality, we use η 2 µ < 1 .
By triangle inequality, the induction holds for s = k + 1 .
Plugging the proper choices of η 1 , η 2 and µ as suggested in (69) to O (cid:18) η 1 n √ mω √ δη 2 µ
[Equation]
yields (cid:107) W ( k ) D,(cid:96)
[Equation]
W (0) D,(cid:96) (cid:107)
[Equation]
O P (cid:18)
[Equation]
√ m n
[Equation] [Equation]
(cid:19)
[Equation]
H P ROOF OF T HEOREM 3.10-K ERNEL RIDGE REGRESSOR APPROXIMATION
We present a following proof sketch on the approximation of regularized DNN estimator to kernel ridge regressor.
The key idea for proving the second result in Theorem 3 . 8 is to write the distance between u i,D ( k ) (where D is to denote the prediction is obtained from regularized GD rule) and kernel regressor B :
=
H ∞
L (cid:0)
Cµ · I + H ∞
L (cid:1) − 1 y in terms of NTK matrix H ∞ L , which is as follows:
[Equation]
Above equality describes how the regularized estimator evolves to ﬁt the kernel regressor as iteration of algorithm goes by.
We can bound the (cid:96) 2 -norm of residual term e D ( k ) as O (1 /n ) , and show that the (cid:96) 2 norm of the ﬁrst term on the RHS of equation (4.3) decays at the rate O (cid:0) √ n (cid:0) 1 − η 2 µL (cid:1) k (cid:1) .
Here the √ n comes from the bound (cid:107) B (cid:107) 2 ≤ O ( √ n ) , since we know (cid:107) u (0) (cid:107) 2 has O ( √ nω ) with small ω ≤ 1 .
This yields the claim.
Recall the equality (65).
Then, we have
[Equation]
With τ = O (cid:18) L √ ω √ mδ n d 2 d − 1 (cid:19)
[Equation]
similarly with Lemma 4.10 and a direct employment of the result from
Lemma 4.
[Equation]
we can control the distance from H D ( k ) to H ∞
L under operator norm as follows:
[Equation]
where in the third inequality
[Equation]
ω
=
O (cid:0) 1 L 3 / 2 n − 5 d − 2 2 d − 1 (cid:1) is plugged-in.
The last inequality holds with d ≥ 2 with large enough n and the condition on width m ≥ Ω (cid:0) L 19 n 20 log 3 ( m ) δ(cid:1) .
Then, the (cid:96) 2 norm of ξ D ( k ) can be bounded as:
[Equation]
where in the second inequality, we used (83) with η
=
O
[Equation]
to control the ﬁrst term and employed Eq.
(70) to control the second term.
In the last equality, we used m ≥ Ω (cid:0) L 19 n 20 log 3 ( m )
[Equation]
(cid:1) .
Now, by setting B :=
[Equation]
−
[Equation]
H ∞
L y , we can easily convert the equality (82) as follows: for k ≥ 1 ,
[Equation]
The recursive applications of the equality (85) yields
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
term in (86) as:
[Equation]
[Table]
[Equation] [Equation]
where in the second inequality, we used (cid:107) u D (0) (cid:107) 2 ≤ O ( √ nω/δ ) and the fact that
[Equation]
By combining (87) and (88) and using a fact (1 − η 2 µL ) k ≤ exp( − η 2 µLk ) , we conclude that after k ≥ Ω (cid:0) ( η 2 µL ) − 1 log( n 3 / 2 ) (cid:1) , the error (cid:107) u D ( k ) − B (cid:107) 2 decays at the rate O (cid:0) 1 n (cid:1) .
I P ROOF OF T HEOREM 3.
We begin the proof by decomposing the error (cid:98)
f W ( k ) D ( x ) − f ∗ ( x ) for any ﬁxed x ∈ Unif ( S d − 1 ) into two terms as follows:
[Equation]
Here, we devise a solution of kernel ridge regression g ∗ µ ( x ) in the decomposition (89):
[Equation]
for some constant C > 0 .
Speciﬁcally, in the proof to follow, we choose η 1 and η 2 such that
C =
η 2 L η 1 m for the theoretical convenience.
Our goal is to show that all the terms (cid:107) ∆ D, 1 (cid:107) 22 , and (cid:107) ∆ D, 2 (cid:107) 22 have the order either equal to or smaller than O (cid:0) n − d 2 d − 1 (cid:1) with the proper choices on m , µ , η 1 and η 2 .
Since the high-level proof idea is similar with that of Theorem 3.8, we omit the step-by-step proof sketch of Theorem 3.11.
The most notable difference between the proof strategies of the two theorems is that the regularized DNN approximate the kernel ridge regressor of noisy data, whereas in Theorem 3.8, unregularized DNN approximate the interpolant based on noiseless data.
Step 1.
Control on ∆ D, 2 .
First, note that there is a recent ﬁnding that the reproducing kernel Hilbert spaces induced from NTKs with any number of layers (i.e., L ≥ 1 ) have the same set of functions, if kernels are deﬁned on S d − 1 .
See Chen & Xu [2020].
Along with this result, under the choice of model parameters as suggested in (69), we can apply exactly the same proof used in
Theorem.
3 . 2 in Hu et al.
[2021] for proving a following :
[Equation]
Step 2.
Control on ∆ D, 1 .
For n data points (cid:0) x 1 , . . .
, x n (cid:1) and for the k th updated parameter W ( k ) D , denote:
[Equation]
Note that when (cid:96) = 1 , ∇ W (cid:96) (cid:2) f W ( k ) D ( X ) (cid:3) ∈
R md × n and when (cid:96) = 2 , . . .
, L , ∇ W (cid:96) (cid:2) f W ( k ) D
( X ) (cid:3) ∈
R m 2 × n .
With this notation, we can write the vectorized version of the update rule (80) as:
[Equation]
∀ 1 ≤ (cid:96) ≤ L and ∀ k ≥ 1 .
Using the equality, we can get the decomposition :
[Equation]
−
[Equation] [Equation]
Let z D,k
( x ) := vec (cid:0) ∇ W (cid:96) (cid:2) f W ( k ) D ( x ) (cid:3)(cid:1) , and note that f W ( k ) D ( x )
= (cid:104)
z D,k ( x ) , vec (cid:0) W ( k ) D,(cid:96) (cid:1) (cid:105) .
Then, by the deﬁnition of ∆ D, 1 and the decomposition (91), we have
[Equation]
First, we focus on controlling the (cid:96) 2 bound on the ﬁrst two terms in (92).
Observe that the ﬁrst term can be bounded as:
[Equation]
Recall that (cid:107) z D,k
( x ) (cid:107)≤
O (cid:0) √ mω (cid:1) by Lemma 4.4.
Then, the random variable z D,k ( x ) (cid:62) vec (cid:0) W (0) D,(cid:96) (cid:1) |
z D,k ( x ) is simply a N (cid:0) 0 , O ( ω ) (cid:1) for 1 ≤ (cid:96) ≤ L .
A straightforward application of Chernoff bound for normal random variable and taking union bound over the layer 1 ≤ (cid:96) ≤ L yield that: with probability at least 1 − δ ,
[Equation]
The (cid:96) 2 norm of the second term in (92) can be similarly bounded as (93) in addition with the CauchySchwarz inequality:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
In the ﬁrst, second and third inequalities, we used a simple fact that for the matrix A ∈ R d 1 × d 2 with rank r , then (cid:107) A (cid:107) 2 ≤ (cid:107) A (cid:107) F ≤ √ r (cid:107) A (cid:107) 2 .
Recall that the rank of the matrix ∇ W (cid:96) (cid:2) f W D ( k − j − 1) ( x ) (cid:3) − ∇ W (cid:96) (cid:2) f W D (0) ( x ) (cid:3) is at most 2 .
In the second to the last inequality, we use the result of Lemma 4.
and the (cid:107) u D ( i ) − y (cid:107) 2 ≤ O ( √ n ) for any
i ≥ 1 .
In the last inequality, we plug the correct orders as set in (69) to τ , η 1 , η 2 and µ .
Back to the inequality (95), using the (cid:107) z D,k ( x ) (cid:107) 2 ≤ O (cid:0) √ mω (cid:1) and (96), we can get
[Equation]
Before controlling the (cid:96) 2 norm of C in (92), recall that we set B := (cid:18) η 2 µL η 1 m I +
H ∞
L (cid:19) −
H ∞
L y and the dynamics of u D ( k ) − B can be expressed in terms of H ∞
L as follows:
For any k ≥ 1 ,
[Equation]
(98) with (cid:107) e D ( k ) (cid:107) 2 ≤ O (cid:0) 1 n (cid:1) .
Using (98), we can further decompose the term E 2 in (91) as:
[Equation]
f W D (0) ( X ) (cid:3) k − 1 (cid:88)
j = (cid:0) 1 − η 2 µ ) j (cid:18)
u D ( k − j − 1) − y (cid:19) =
η 1 ∇ W (cid:96) (cid:2) f W D (0) ( X ) (cid:3) k − 1 (cid:88)
j = (cid:0) 1 − η 2 µ ) j (cid:18)(cid:0) 1 − η 2 µL
(cid:1) ·
I − mη 1 H ∞
L (cid:19) k − j −
B − η 1 ∇ W (cid:96) (cid:2) f W D (0) ( X ) (cid:3) k − 1 (cid:88)
j = (cid:0) 1 − η 2 µ ) j (cid:18)(cid:0) 1 − η 2 µL
(cid:1) ·
I − mη 1 H ∞
L (cid:19) k − j −u D (0) − η 1 ∇ W (cid:96) (cid:2) f W D (0) ( X ) (cid:3) k − 1 (cid:88)
j = (cid:0) 1 − η 2 µ ) j e D ( k − j − 1)
− η 1 ∇ W (cid:96) (cid:2) f W D (0) ( X ) (cid:3) k − 1 (cid:88)
j = (cid:0) 1 − η 2 µ ) j (cid:18)
B − y (cid:19) =
E 2 , 1 + E 2 , 2 + E 2 , 3 + E 2 , 4 .
Then, we can re-write the error term C in (92) as:
[Equation] [Equation]
Our goal is to control the (cid:96) 2 norm of each summand in the equality (100).
For the ﬁrst three terms in (100), a simple Cauchy-Schwarz inequality can be applied: for i = 1 , 2 , 3 :
[Equation]
We work on obtaining the bound of (cid:80) L (cid:96) =1 (cid:107) E 2 , 1 (cid:107) 2 2 .
Let T k be deﬁned as
[Equation]
Then, we have
[Equation]
To obtain the upper-bound on (101), we need to control the terms T (cid:62)
k H ∞
L T k and B (cid:62) T 2 k B .
Let us denote H ∞
L = (cid:80) ni
=
λ i v i v (cid:62) i be the eigen-decomposition of H ∞ L .
Using 1 − η 2 µL
[Equation] [Equation]
η 2 µ , note that
[Equation]
A similar logic can be applied to bound T (cid:62)
k H ∞
L T k :
[Equation]
Recall the deﬁnition of the notation B :=
H ∞
L
[Equation]
y .
Then, we can bound the term
[Equation]
where in the last equality
[Equation]
we used (cid:13)(cid:13) g ∗ µ (cid:13)(cid:13) 2 H
=
O P (1) in (90).
Now we turn our attention to bound the term B (cid:62) T 2 k B ,
[Equation]
105) where we used (cid:107) y (cid:107)
= O ( n ) in the last inequality.
Combining the bounds (104), (105) and the result from Lemma 4.
[Equation]
we can further bound (101) and have:
[Equation]
) where in the second inequality
[Equation]
we used m ≥ Ω
[Equation]
Similarly, we can bound
[Equation]
Here, in the second inequality, we used the inequalities (102) and (103) and Lemma 4.11.
In the third inequality, we used the Lemma 4.8, (cid:107) u (0) (cid:107)
=
O (cid:0) √ nωδ (cid:1) with probability at least 1 − δ .
In the last equality, we used m ≥ Ω (cid:18)
[Equation]
Next, we bound (cid:80) L(cid:96)
[Equation]
Now, we focus on obtaining the (cid:96) 2 norm bound on D in (100).
Recall the deﬁnition of the notation
B := H ∞
L
[Equation]
−
[Equation]
y .
A simple calculation yields that
[Equation]
Then, we can re-write the expression of the D as :
[Equation]
where in the second equality, (cid:80) k
[Equation] [Equation]
=
[Equation]
1 − η 2 µ
[Equation]
j = 1 − (1 − η
[Equation]
) k η 2 µ is used.
The (cid:96) 2 norm of ﬁrst term in the (109) can be bounded as:
[Equation]
· (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:18)
[Equation]
(cid:19) −
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:88)
[Equation]
[Equation]
(cid:13)(cid:13)(cid:13) (cid:13) (cid:13) (cid:18)
[Equation]
(cid:19)
[Equation]
(cid:13)(cid:13)(cid:13) (cid:13) (cid:13)≤ O
[Equation]
=
[Equation]
O
[Equation]
where, in the second inequality, we used Lemma 4.11, and also we used
[Equation]
The (cid:96) 2 norm of the second term in (109) can be easily bounded as:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Lastly
[Equation]
the (cid:96) 2 norm of the third term in (109) is bounded as:
[Equation]
[Table]
[Equation]
where in the fourth inequality
[Equation]
τ = O P
[Equation] [Equation]
is plugged in.
Combining the inequalities (110), (112) and (113), we get the bound on (cid:107)D(cid:107) 2 in (109):
[Equation] [Equation]
Step 3.
Combining all pieces.
Recall (cid:107) z D,k ( x ) (cid:107) 2 ≤ O (cid:0) √ mω (cid:1) .
With this fact, combining the bounds (94), (97), (106), (107), (108) and (114), we can bound the (cid:107) ∆ D, 1 (cid:107) 22 via the decomposition (92) as follows:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]