O PTIMAL R ATES FOR A VERAGED S TOCHASTIC G RADIENT D ESCENT UNDER N EURAL T ANGENT K ERNEL R EGIME
Anonymous authors
Paper under double-blind review
A BSTRACT
We analyze the convergence of the averaged stochastic gradient descent for overparameterized two-layer neural networks for regression problems.
It was recently found that a neural tangent kernel (NTK) plays an important role in showing the global convergence of gradient-based methods under the NTK regime, where the learning dynamics for overparameterized neural networks can be almost characterized by that for the associated reproducing kernel Hilbert space (RKHS).
However, there is still room for a convergence rate analysis in the NTK regime.
In this study, we show that the averaged stochastic gradient descent can achieve the minimax optimal convergence rate, with the global convergence guarantee, by exploiting the complexities of the target function and the RKHS associated with the NTK.
Moreover, we show that the target function speciﬁed by the NTK of a ReLU network can be learned at the optimal convergence rate through a smooth approximation of a ReLU network under certain conditions.
I NTRODUCTION
Recent studies have revealed why a stochastic gradient descent for neural networks converges to a global minimum and why it generalizes well under the overparameterized setting in which the number of parameters is larger than the number of given training examples.
One prominent approach is to map the learning dynamics for neural networks into function spaces and exploit the convexity of the loss functions with respect to the function.
The neural tangent kernel (NTK) (Jacot et al., 2018) has provided such a connection between the learning process of a neural network and a kernel method in a reproducing kernel Hilbert space (RKHS) associated with an NTK.
The global convergence of the gradient descent was demonstrated in Du et al.
(2019b); Allen-Zhu et al.
(2019a) ; Du et al. (2019a); Allen-Zhu et al. (2019b) through the development of a theory of NTK with the overparameterization.
In these theories, the positivity of the NTK on the given training examples plays a crucial role in exploiting the property of the NTK.
Speciﬁcally, the positivity of the Gram-matrix of the NTK leads to a rapid decay of the training loss, and thus the learning dynamics can be localized around the initial point of a neural network with the overparameterization, resulting in the equivalence between two learning dynamics for neural networks and kernel methods with the NTK through a linear approximation of neural networks.
Moreover, Arora et al. (2019a) provided a generalization bound of O
( T − 1 / 2 ) , where T is the number of training examples, on a gradient descent under the positivity assumption of the NTK.
These studies provided the ﬁrst steps in understanding the role of the NTK.
However, the eigenvalues of the NTK converge to zero as the number of examples increases, as shown in Su & Yang (2019) (also see Figure 1), resulting in the degeneration of the NTK.
This phenomenon indicates that the convergence rates in previous studies in terms of generalization are generally slower than O ( T − 1 / 2 ) owing to the dependence on the minimum eigenvalue.
Moreover, Bietti & Mairal (2019); Ronen et al.
; Cao et al.
(2019) also supported this observation by providing a precise estimation of the decay of the eigenvalues, and Ronen et al.
; Cao et al.
(2019) proved the spectral bias (Rahaman et al., 2019) for a neural network, where lower frequencies are learned ﬁrst using a gradient descent.
By contrast, several studies showed faster convergence rates of the (averaged) stochastic gradient descent in the RKHS in terms of the generalization (Cesa-Bianchi et al., 2004; Smale & Yao, 2006;
Ying & Zhou, 2006; Neu & Rosasco, 2018; Lin et al., 2020).
In particular, by extending the results in a ﬁnite-dimensional case (Bach & Moulines, 2013), Dieuleveut & Bach (2016); Dieuleveut et al.
(2017) showed convergence rates of O ( T − 2 rβ 2 rβ +1 ) depending on the complexity r ∈ [1 / 2 , 1] of the target functions and the decay rate β > 1 of the eigenvalues of the kernel (a.k.a.
the complexity of the hypothesis space).
In addition, extensions to the random feature settings (Rahimi & Recht, 2007; Rudi & Rosasco, 2017; Carratino et al., 2018), to the multi-pass variant (Pillaud-Vivien et al., 2018b), and to the tail-averaging and mini-batching variant (Mücke et al., 2019) have been developed.
Motivation.
The convergence rate of O ( T − 2 rβ 2 rβ +1 ) is always faster than O ( T − 1 / 2 ) and is known as the minimax optimal rate (Caponnetto & De Vito, 2007; Blanchard & Mücke, 2018).
Hence, a gap exists between the theories regarding NTK and kernel methods.
In other words, there is still room for an investigation into a stochastic gradient descent due to a lack of speciﬁcation of the complexities of the target function and the hypothesis space. That is, to obtain faster convergence rates, we should specify the eigenspaces of the NTK that mainly contain the target function (i.e., the complexity of the target function), and specify the decay rates of the eigenvalues of the NTK (i.e., the complexity of the hypothesis space), as studied in kernel methods (Caponnetto & De Vito, 2007; Steinwart et al., 2009;
Dieuleveut & Bach, 2016).
In summary, the fundamental question in this study is
Can stochastic gradient descent for overparameterized neural networks achieve the optimal rate in terms of the generalization by exploiting the complexities of the target function and hypothesis space?
In this study, we answer this question in the afﬁrmative, thereby bridging the gap between the theories of overparameterized neural networks and kernel methods.
[Figure]
1. C ONTRIBUTIONS
The connection between neural networks and kernel methods is being understood via the NTK, but it is still unknown whether the optimal convergence rate faster than O ( T − 1 / 2 ) is achievable by a certain algorithm for neural networks.
This is the ﬁrst paper to overcome technical challenges of achieving the optimal convergence rate under the NTK regime.
We obtain the minimax optimal convergence rates (Corollary 1), inherited from the learning dynamics in an RKHS, for an averaged stochastic gradient descent for neural networks.
That is, we show that smooth target functions efﬁciently speciﬁed by the NTK are learned rapidly at faster convergence rates than O (1 / √ T ) .
Moreover, we obtain an explicit optimal convergence rate of O (cid:16) T − 2 rd 2 rd + d − 1 (cid:17) for a smooth approximation of the
ReLU network (Corollary 2), where d is the dimensionality of the data space and r is the complexity of the target function speciﬁed by the NTK of the ReLU network.
1. T ECHNICAL C HALLENGE
The key to showing a global convergence (Theorem 1) is making the connection between kernel methods and neural networks in some sense.
Although this sort of analysis has been developed in several studies (Du et al., 2019b; Arora et al., 2019a; Weinan et al., 2019; Arora et al., 2019b; Lee et al., 2019; 2020), we would like to emphasize that our results cannot be obtained by direct application of their results.
A naive idea is to simply combine their results with the convergence analysis of the stochastic gradient descent for kernel methods, but it does not work.
The main reason is that we need the L 2 -bound weighted by a true data distribution on the gap between dynamics of stochastic gradient descent for neural networks and kernel methods if we try to derive a convergence rate of population risks for neural networks from that for kernel methods.
However, such a bound is not provided in related studies.
Indeed, to the best of our knowledge, all related studies make this kind of connection regarding the gap on training dataset or sample-wise high probability bound (Lee et al., 2019; Arora et al., 2019b).
That is, a statement “for every input data x with high probability | g ( t ) nn ( x ) − g
t ) ntk ( x ) | < (cid:15) ” cannot yield a desired statement “ (cid:107) g ( t ) nn − g ( t ) ntk (cid:107) L 2 ( ρ X ) < (cid:15) ” where g ( t ) nn and g ( t ) ntk are t -th iterate of gradient descent for a neural network and corresponding iterate described by NTK, and (cid:107)·(cid:107)
L 2 ( ρ X ) is the L 2 -norm weighted by a marginal data distribution ρ X over the input space.
Moreover, we note that we cannot utilize the positivity of the Gram-matrix of NTK which plays a crucial role in related studies because we consider the population risk with respect to (cid:107) · (cid:107) L 2 ( ρ X ) rather than the empirical risk.
To overcome these difﬁculties we develop a different strategy of the proof.
First, we make a bound on the gap between two dynamics of the averaged stochastic gradient descent for a two-layer neural network and its NTK with width M (Proposition A), and obtain a generalization bound for this intermediate NTK (Theorem A in Appendix).
Second, we remove the dependence on the width of M from the intermediate bound.
These steps are not obvious because we need a detailed investigation to handle the misspeciﬁcation of the target function by an intermediate NTK.
Based on detailed analyses, we obtain a faster and precise bound than those in previous results (Arora et al., 2019a).
The following is an informal version of Proposition A providing a new connection between a two-layer neural networks and corresponding NTK with width M .
Proposition 1 (Informal) .
Under appropriate conditions we simultaneously run averaged stochastic gradient descent for a neural network with width of M and for its NTK.
Assume they share the same hyper-parameters and examples to compute stochastic gradients.
Then, for arbitrary number of iterations T ∈ Z + and (cid:15) > 0 , there exists M ∈ Z + depending only on T and (cid:15) such that ∀ t ≤ T , [Equation] where g ( t ) nn and g ( t ) ntk are iterates obtained by averaged stochastic gradient descent. This proposition is the key because it connects two learning dynamics for a neural network and
its
NTK through overparameterization without the positivity of the NTK.
Instead of the positivity, this proposition says that overparameterization increases the time stayed in the NTK regime where the learning dynamics for neural networks can be characterized by the NTK.
As a result, the averaged stochastic gradient descent for the overparameterized two-layer neural networks can fully inherit preferable properties from learning dynamics in the NTK as long as the network width is sufﬁciently large.
See Appendix A for detail.
Besides the abovementioned studies, there are several works (Chizat & Bach, 2018b; Wu et al., 2019; Zou & Gu, 2019) that have shown the global convergence of (stochastic) gradient descent for overparameterized neural networks essentially relying on the positivity condition of NTK.
Moreover, faster convergence rates of the second-order methods such as the natural gradient descent and GaussNewton method have been demonstrated (Zhang et al., 2019; Cai et al., 2019) in the similar setting, and the further improvement of Gauss-Newton method with respect to the cost per iteration has been conducted in Brand et al.
There have been several attempts to improve the overparameterization size in the NTK theory.
For the regression problem, Song & Yang (2019) has succeeded in reducing the network width required in Du et al. (2019b) by utilizing matrix Chernoff bound. For the classiﬁcation problem, the positivity condition can be relaxed to a separability condition using another reference model (Cao & Gu, 2019a;b; Nitanda et al., 2019; Ji & Telgarsky, 2019), resulting in mild overparameterization and generalization bounds of O ( T − 1 / 2 ) or O
( T − 1 / 4 ) on classiﬁcation errors.
For an averaged stochastic gradient descent on classiﬁcation problems in RKHSs, linear convergence rates of the expected classiﬁcation errors have been demonstrated in Pillaud-Vivien et al.
(2018a);
Nitanda & Suzuki (2019).
Although our study focuses on regression problems, we describe how to combine their results with our theory in the Appendix.
The mean ﬁeld regime (Nitanda & Suzuki, 2017; Mei et al., 2018; Chizat & Bach, 2018a) that is a different limit of neural networks from the NTK is also important for the global convergence analysis of the gradient descent.
In the mean ﬁeld regime, the learning dynamics follows the Wasserstein gradient ﬂow which enables us to establish convergence analysis in the probability space.
Moreover, several studies (Allen-Zhu & Li, 2019; Bai & Lee, 2019; Ghorbani et al., 2019; Allen-Zhu & Li, 2020; Li et al., 2020) attempt to show the superiority of neural networks over kernel methods including the NTK.
Although it is also very important to study the conditions beyond the NTK regime, they do not affect our contribution and vice versa.
Indeed, which method is better depends on the assumption on the target function and data distribution, so it is important to investigate the optimal convergence rate and optimal method in each regime.
As shown in our study, the averaged stochastic gradient descent for learning neural network achieves the optimal convergence rate if the target function is included in RKHS associated with the NTK with the small norm.
It means there are no methods that outperform the averaged stochastic gradient descent under this setting.
P RELIMINARY
Let X ⊂ R d and Y ⊂ R be the measurable feature and label spaces, respectively.
We denote by ρ a probability measure on X × Y , by ρ X the marginal distribution on X , and by ρ ( ·| X ) the conditional distribution on Y , where ( X, Y ) ∼ ρ .
Let (cid:96) ( z, y ) ( z ∈ R , y ∈ Y ) be the squared loss function 12 ( z − y ) 2 , and let g : X → R be a hypothesis.
The expected risk function is deﬁned as follows:
[Equation]
The Bayes rule g ρ : X → R is a global minimizer of L over all measurable functions.
For the least squares regression, the Bayes rule is known to be g ρ ( X [Equation] =
E Y
[ Y | [Equation] ] and the excess risk of a hypothesis g (which is the difference between the expected risk of g and the expected risk of the Bayes rule g ρ ) is expressed as a squared L 2 ( ρ X ) -distance between g and g ρ (for details, see
Cucker & Smale (2002))
up to a constant:
[Equation]
where (cid:107) · (cid:107) L 2 ( ρ X ) is L 2 -norm weighted by ρ X deﬁned as (cid:107) g (cid:107) L 2 ( ρ X ) def = (cid:0)(cid:82) g 2 ( X )d ρ X ( X ) (cid:1) 1 /( g ∈ L 2 ( ρ X )) .
Hence, the goal of the regression problem is to approximate g ρ in terms of the L 2 ( ρ X ) -distance in a given hypothesis class.
Two-layer Neural Networks.
The hypothesis class considered in this study is the set of two-layer neural networks, which is formalized as follows.
Let
M ∈ Z
+ be the network width (number of hidden nodes).
Let a =
( a 1 , . .
. , a M ) (cid:62) ∈ R M ( a r ∈ R ) be the parameters of the output layer, B = ( b 1 , . . .
, b M ) ∈ R d × M ( b r ∈ R d ) be the parameters of the input layer, and c = ( c 1 , . . .
, c M ) (cid:62) ∈ R M ( c r ∈ R ) be the bias parameters.
We denote by Θ the collection of all parameters ( a, B, c ) , and consider two-layer neural networks:
[Equation]
where σ : R → R is an activation function and γ > 0 is a scale of the bias terms.
Symmetric initialization.
We adopt symmetric initialization for the parameters Θ .
Let a (0) =
( a (0)1 , . . .
, a (0) M ) (cid:62) , B (0) =
( b (0)1 , . . .
, b (0) M ) , and c (0) =
( c (0)1 , . . .
, c (0) M ) (cid:62) denote the initial values for a , B , and c , respectively.
Assume that the number of hidden units M ∈ Z + is even.
The parameters for the output layer are initialized as a (0) r
= R for r ∈ { 1 , . . .
, M 2 } and a (0) r = − R for r ∈ { M 2 + 1 , . . .
, M } , where R > 0 is a positive constant.
Let µ 0 be a uniform distribution on the sphere S d − 1 = { b ∈ R d
| (cid:107) b (cid:107) 2 = 1 } ⊂ R d used to initialize the parameters for the input layer.
The parameters for the input layer are initialized as b (0) r
= b (0) r
+ M 2 for r ∈ { 1 , . .
, M 2 } , where ( b (0) r ) M 2 r = are independently drawn from the distribution µ 0 .
The bias parameters are initialized as c (0) r = 0 for r ∈ { 1 , . .
, M } .
The aim of the symmetric initialization is to make an initial function g Θ (0)
= 0 , where Θ (0)
=
( a (0) , B (0) , c (0) ) .
This is just for theoretical simplicity.
Indeed, we can relax the symmetric initialization by considering an additional error stemming from the nonzero initialization in the function space.
Regularized Expected Risk Minimization.
Instead of minimizing the expected risk (1) itself, we consider the minimization problem of the regularized expected risk around the initial values:
[Equation]
where the last term is the L 2 -regularization at an initial point with a regularization parameter λ > 0 .
This regularization forces iterations obtained by optimization algorithms to stay close to the initial value, which enables us to utilize the better convergence property of regularized kernel methods.
Averaged Stochastic Gradient Descent.
Stochastic gradient descent is the most popular method for solving large-scale machine learning problems, and its averaged variant is also frequently used to stabilize and accelerate the convergence.
In this study, we analyze the generalization ability of an averaged stochastic gradient descent.
The update rule is presented in Algorithm 1.
Let Θ ( t ) =
( a ( t ) , B ( t ) , c ( t ) ) denote the collection of t -th iterates of parameters a ∈ R M ,
B ∈ R d × M , and c ∈ R M .
At t -th iterate, stochastic gradient descent using a learning rate η t for the problem (3) with respect to a, B, c is performed on lines 4–6 for a randomly sampled example ( x t , y t ) ∼ ρ . These updates can be rewritten in an element-wise fashion as follows. For r ∈ { 1 , . . . , M } ,
[Equation]
where a ( t )
= ( a ( t ) 1 , . .
. , a ( t ) M ) (cid:62) , B ( t ) =
( b ( t ) 1 , . . .
, b ( t ) M ) , and c ( t ) =
( c ( t ) 1 , . . .
, c ( t ) M ) (cid:62) .
Finally, a weighted average using weights α t of the history of parameters is computed on line 9.
In our theory, we consider the constant learning rate η t = η and uniform averaging α t = 1 / ( T + 1) .
Algorithm
1: Input: number of iterations T , regularization parameter λ , learning rates ( η t ) T − 1 t =0 , averaging weights ( α t )
Tt =0 , initial values Θ (0) =
( a (0) , B (0) , c (0) ) 2: for t = 0 to T − 1 do 3:
Randomly draw a sample ( x t , y t ) ∼ ρ 4: a ( t +1) ← a ( t ) − η t ∂ a (cid:96) ( g Θ ( t ) ( x t ) , y t ) − η t λ ( a ( t ) − a (0) ) 5: B ( t +1) ← B ( t ) − η t ∂ B (cid:96) ( g Θ ( t ) ( x t ) , y t ) − η t λ ( B ( t ) − B (0) ) 6: c ( t +1) ← c ( t ) − η t ∂ c (cid:96) ( g Θ ( t ) ( x t ) , y t ) − η t λ ( c ( t ) − c (0) ) 7: Θ ( t +1) ← ( a ( t +1) , B ( t +1) , c ( t +1) ) 8: end for 9: Θ ( T ) =
{+( (cid:80) Tt =
+}
{+α t a ( t ) , (cid:80) Tt =
+}
{+α t B ( t ) , (cid:80) Tt =
+}
α t c ( t ) ) 10: Return g Θ ( T )
Integral and Covariance Operators. The integral and covariance operators associated with the kernels, which are the limit of the Gram-matrix as the number of examples goes to inﬁnity, play a crucial role in determining the learning speed.
For a given Hilbert space H , we denote by ⊗ H the tensor product on H , that is, ∀
( f, g ) ∈ H 2 , f ⊗ H g deﬁnes a linear operator; h ∈ H (cid:55)→ ( f ⊗ H g )
h
= (cid:104) f, h (cid:105)
H g ∈ H .
Note that f ⊗ H g naturally induces a bilinear function: ( h, h (cid:48) ) ∈
H × H (cid:55)→ (cid:104) ( f ⊗ H g ) h, h (cid:48) (cid:105) H = (cid:104) f, h (cid:105) H (cid:104) g, h (cid:48) (cid:105) H .
When H is a reproducing kernel Hilbert space (RKHS) associated with a bounded kernel k : X × X → R , the covariance operator Σ : H (cid:55)→ H is deﬁned as follows:
Set K X def = k
( X, · ) and
[Equation]
Note that the covariance operator is a restriction of the integral operator on L 2 ( ρ X ) :
[Equation]
[-( (cid:80) Tt =
-]
[-α t a ( t ) , (cid:80) Tt =
-]
[-α t B ( t ) , (cid:80) Tt =
-]
We use the same symbol as above for convenience with a slight abuse of notation.
Because Σ is a compact self-adjoint operator on L 2 ( ρ X ) , Σ has the following eigendecomposition: Σ f = (cid:80) ∞ i =
λ i (cid:104) f, φ i (cid:105)
L 2 ( ρ X ) φ i for f ∈ L 2 ( ρ X ) , where { ( λ i , φ i ) } ∞
i =1 is a pair of eigenvalues and orthogonal eigenfunctions in L 2 ( ρ X ) .
For s ∈ R , the power Σ s is deﬁned as Σ s f = (cid:80) ∞ i [Equation] i (cid:104) f,
[Equation]
i
[Equation]
L
[Equation]
( ρ X ) φ i .
M AIN R ESULTS : M INIMAX O PTIMAL C ONVERGENCE R ATES
In this section, we present the main results regarding the fast convergence rates of the averaged stochastic gradient descent under a certain condition on the NTK and target function g ρ .
Neural tangent kernel.
The NTK is a recently developed kernel function and has been shown to be extremely useful in demonstrating the global convergence of the gradient descent method for neural networks (cf., Jacot et al.
; Chizat & Bach (2018b); Du et al. (2019b); Allen-Zhu et al.
(2019a;b); Arora et al. (2019a)).
The NTK in our setting is deﬁned as follows: ∀ x, ∀ x
[Equation]
where the expectation is taken with respect to b (0) ∼ µ 0 .
The NTK is the key to the global convergence of a neural network because it makes a connection between the (averaged) stochastic gradient descent for a neural network and the RKHS associated with k ∞ (see Proposition A).
Although this type of connection has been shown in previous studies (Arora et al., 2019b; Weinan et al., 2019; Lee et al., 2019; 2020), note that their results are inapplicable to our theory because we consider the population risk.
Indeed, our study is the ﬁrst to establish this connection for an (averaged) stochastic gradient descent in terms of the uniform distance on the support of the data distribution, enabling us to obtain faster convergence rates.
We note that an NTK k ∞ is the sum of two NTKs, that is, the ﬁrst and second terms in (4) are NTKs for the output and input layers with bias, respectively.
3. G LOBAL C ONVERGENCE A NALYSIS
Let H ∞ be an RKHS associated with NTK k ∞ , and let Σ ∞ be the corresponding integral operator.
Let { λ i } ∞ i =1 denote the eigenvalues of Σ ∞ sorted in decreasing order: λ 1 ≥ λ 2 ≥ · · · .
Assumption 1.
(A1) (Smooth activation function): There exists C > 0 such that (cid:107) σ (cid:48)(cid:48) (cid:107) ∞ ≤ C , (cid:107) σ (cid:48) (cid:107) ∞ ≤ 2 , and | σ ( u ) | ≤ 1 + | u | for ∀ u ∈ R . (A2) (Boundedness of data and hyperparameters): are bounded.
That is, supp( ρ X ) ⊂ { x ∈ R d | (cid:107) x (cid:107) 2 ≤ 1 } , Y ⊂ [ − 1 , 1] , R = 1 , and γ ∈ [0 , 1] .
(A3)
(Complexity of target function): There exists r ∈
[1 / 2 , 1] such that g ρ ∈ Σ r ∞ ( L 2 ( ρ X )) , i e., (cid:107) Σ − r ∞ g ρ (cid:107) L 2 ( ρ X ) < ∞ .
(A4)
(Complexity of hypothesis space): There exists β > 1 such that λ i =
Θ( i − β ) .
Remark.
• (A1): Typical smooth activation functions, such as sigmoid and tanh functions, and smooth approximations of the ReLU, such as swish (Ramachandran et al., 2017), which performs as well as or even better than the ReLU, satisfy Assumption (A1) .
This condition is used to relate the two learning dynamics between neural networks and kernel methods (see Proposition A).
• (A2): The boundedness (A2) of the feature space and label are often assumed for stochastic optimization and least squares regression for theoretical guarantees (see Steinwart et al.
Note that these constants in (A2) can be relaxed to arbitrary constants.
• (A3): Assumption (A3) measures the complexity of g ρ because Σ ∞ can be considered as a smoothing operator using a kernel k ∞ .
A larger r indicates a faster decay of the coefﬁcients of expansion of g ρ based on the eigenfunctions of Σ ∞ and smoothens g ρ .
In addition, Σ r ∞ ( L 2 ( ρ X )) shrinks with respect to r and Σ 1 / 2 ∞ ( L 2 ( ρ X ))
=
H ∞ , resulting in g ρ ∈ H ∞ .
This condition is used to control the bias of the estimators through L 2 -regularization.
The notation Σ − r ∞ g ρ represents any function G ∈ L 2 ( ρ X ) such that g ρ = Σ r ∞ G .
• (A4): Assumption (A4) controls the complexity of the hypothesis class H ∞ .
A larger β indicates a faster decay of the eigenvalues and makes H ∞ smaller.
This assumption is essentially needed to bound the variance of the estimators efﬁciently and derive a fast convergence rate.
Theorem 1 and Corollary 1, 2 hold even though the condition in (A4) is relaxed to λ
i =
O ( i − β ) and the lower bound λ
i =
Ω( i − β ) is necessary only for making obtained rates minimax optimal.
Under these assumptions, we derive the convergence rate of the averaged stochastic gradient descent for an overparameterized two-layer neural network, the proof is provided in the Appendix.
Theorem 1.
Suppose Assumptions (A1)-(A3) hold.
Run Algorithm 1 with a constant learning rate η satisfying 4(6 + λ ) η ≤ 1 .
Then, for any (cid:15) > 0 , (cid:107) Σ ∞ (cid:107) op ≥ λ > 0 , δ ∈ (0 , 1) , and T ∈ Z + , there exists M 0 ∈
Z
+ such that for any M ≥ M 0 , the following holds with high probability at least 1 − δ over the random choice of features Θ (0) :
[Equation]
where α > 0 is a universal constant and g Θ ( T ) is an iterate obtained through Algorithm 1.
Remark.
The ﬁrst term (cid:15) and second term λ 2 r (cid:107) Σ − r ∞ g ρ (cid:107) 2 L 2 ( ρ X ) are the approximation error and bias, which can be chosen to be arbitrarily small.
The ﬁrst term comes from the approximation of the NTK using ﬁnite-sized neural networks, and the second term comes from the L 2 -regularization, which coincides with a bias term in the theory of least squares regression (Caponnetto & De Vito, 2007).
The third and fourth terms come from the convergence of the averaged semi-stochastic gradient descent (which is considered in the proof) in terms of the optimization.
The appearance of an inverse dependence on λ in the fourth term is common because a smaller λ indicates a weaker strong convexity, which slows down the convergence speed of the optimization methods (Rakhlin et al., 2012).
The term Tr (cid:0) Σ ∞ (Σ ∞ + λI ) − 1 (cid:1) is the variance from the stochastic approximation of the gradient, and it is referred to as the degree of freedom or the effective dimension , which is known to be unavoidable in kernel regression problems (Caponnetto & De Vito, 2007; Dieuleveut & Bach, 2016; Rudi & Rosasco, 2017).
Global convergence in NTK regime.
This theorem shows the global convergence to the Bayes rule g ρ , which is a minimizer over all measurable maps because the approximation term (cid:15) can be arbitrarily small by taking a sufﬁciently large network width M .
The required value of M has an exponential dependence on T ; note, however, that reducing M is not the main focus of the present study.
The key technique is to relate two learning dynamics for two-layer neural networks and kernel methods in an RKHS approximating H ∞ up to a small error.
Unlike existing studies (Du et al., 2019b; Arora et al., 2019a;b; Weinan et al., 2019; Lee et al., 2019; 2020) showing such connections, our study establishes this connection in term of the L ∞ ( ρ X ) -norm, which is more useful in a generalization analysis.
Moreover, existing studies essentially rely on the strict positivity of the Gram-matrix to localize all iterates around an initial value, which can slow down the convergence rate in terms of the generalization because the convergence of the eigenvalues of the NTK to zero affects the Rademacher complexity.
By contrast, our theory succeeds in demonstrating the global convergence in the NTK regime with the overparameterization and without the positivity of the NTK.
3. O PTIMAL C ONVERGENCE R ATE
We derive the fast convergence rate from Theorem 1 by utilizing Assumption (A4) , which deﬁnes the complexity of the NTK.
The regularization parameter λ mainly controls the trade-off within the generalization bound, that is, a smaller value decreases the bias term but increases the variance term including the degree of freedom.
The degree of freedom
Tr (cid:0) Σ ∞ (Σ ∞ + λI ) − 1 (cid:1) can be speciﬁed by imposing Assumption (A4) because it determines the decay rate of the eigenvalues of Σ ∞ .
As a result, this trade-off between bias and variance depending on the choice of λ becomes clear, and we can determine the optimal value.
Concretely, by setting λ = T − β/ (2 rβ +1) , the sum of the bias and variance terms is minimized, and these terms become asymptotically equivalent.
Corollary 1.
Suppose Assumptions (A1)-(A4) hold.
Run Algorithm 1 with the constant learning rate η
= O (1) satisfying 4(6 + λ ) η ≤ 1 and λ
= T − β/ (2 rβ +1) .
Then, for any (cid:15) > 0 , δ ∈ (0 , 1) and
T ∈ Z + satisfying (cid:107) Σ ∞ (cid:107) op ≥ λ , there exists M 0 ∈
Z
+ such that for any M ≥ M 0 , the following holds with high probability at least 1 − δ over the random choice of random features Θ (0) :
[Equation]
where α > 0 is a universal constant and g Θ ( T ) is an iterate obtained by Algorithm 1.
The resulting convergence rate is O
( T − 2 rβ 2 rβ +1 ) with respect to T by considering a sufﬁciently large network width of M such that the error (cid:15) stemming from the approximation of NTK can be ignored.
Because T corresponds to the number of examples used to learn a predictor g Θ ( T ) , this convergence rate is simply the generalization error bound for the averaged stochastic gradient descent.
In general, this rate is always faster than T − 1 / 2 and is known to be the minimax optimal rate of estimation (Caponnetto & De Vito, 2007; Blanchard & Mücke, 2018) in H ∞ in the following sense.
Let P ( β, r ) be a data distribution class satisfying Assumptions (A2)-(A4) .
Then,
[Equation]
where ρ is taken in P ( β, r ) and h ( T ) is taken over all mappings ( x t , y t )
T − 1 t =
(cid:55)→ h ( T )
∈
H ∞ .
3. E XPLICIT O PTIMAL C ONVERGENCE R ATE FOR S MOOTH A PPROXIMATION OF R E LU
For smooth activation functions that sufﬁciently approximate the ReLU, an optimal explicit convergence rate can be derived under the setting in which the target function is speciﬁed by NTK with the ReLU, and the data are distributed uniformly on a sphere.
We denote the ReLU activation by σ ( u ) = max { 0 , u } and a smooth approximation of ReLU by σ ( s ) , which converges to ReLU, as s → ∞ in the following sense.
We make alternative assumptions to (A1) , (A2) , and (A3) :
Assumption 2.
(A1’)
σ ( s ) satisﬁes (A1)
σ ( s ) and σ ( s ) (cid:48) converge pointwise almost surely to σ and σ
(cid:48) as s → ∞ (A2’)
ρ X is a uniform distribution on { x ∈ R d | (cid:107) x (cid:107) 2 = 1 } .
Y ⊂
[ − 1 , 1] , R = 1 , and γ ∈ (0 , 1] .
(A3’)
The condition (A3) is satisﬁed by the NTK associated with the ReLU activation σ .
(A1’) and (A2’) are special cases of (A1) and (A2) .
There are several activation functions that satisfy this condition, including swish (Ramachandran et al., 2017): σ ( s ) ( u ) =
u 1+exp( − su ) .
Under these conditions, we can estimate the decay rate of the eigenvalues for the ReLU as β = 1 + 1 d − 1 , yielding the explicit optimal convergence rate by adapting the proof of Theorem 1 to the current setting.
Note that Algorithm 1 is run for a neural network with a smooth approximation σ ( s ) of the ReLU.
Corollary 2.
Suppose Assumptions (A1’) , (A2’) , and (A3’) hold.
Run Algorithm 1 with the constant learning rate η
= O (1) satisfying 4(6 + λ ) η ≤ 1 , and
λ =
T − d/ (2 rd + d − 1) .
Given any (cid:15) > 0 , δ ∈ (0 , 1) and T ∈ Z + satisfying (cid:107) Σ ∞ (cid:107) op ≥ 2 λ , let s be an arbitrary and sufﬁciently large positive value.
Then, there exists M 0 ∈
Z
+ such that for any M ≥ M 0 , the following holds with high probability at least 1 − δ over the random choice of random features Θ (0) :
[Equation]
where α > 0 is a universal constant and g Θ ( T ) is an iterate obtained by Algorithm 1.
E XPERIMENTS
We verify the importance of the speciﬁcation of target functions by showing the misspeciﬁcation signiﬁcantly slows down the convergence speed.
To evaluate the misspeciﬁcation, we consider single-layer learning as well as the two-layer learning, and we see the advantage of two-layer learning.
Here, note that, with evident modiﬁcation of the proofs, the counterparts of Corollaries 1 andfor learning a single layer also hold by replacing Σ ∞ with the covariance operator Σ a, ∞ ( Σ b, ∞ ) associated with k a, ∞ ( k b, ∞ ), where
[Equation]
] , which are components of k ∞ =
k a, ∞ + k b, ∞ corresponding to the output and input layers, respectively.
Then, from Corollaries 1 and 2, a Bayes rule g ρ is learned efﬁciently by optimizing the layer which has a small norm (cid:107) Σ − r g ρ (cid:107) L 2 ( ρ X ) for Σ ∈ { Σ a, ∞ , Σ b, ∞ , Σ ∞ } .
[Figure]
Experimental settings.
Figure 2 (Top) depicts norms (cid:107) Σ −
r g ρ (cid:107) L 2 ( ρ X ) for Σ ∈ { Σ a, ∞ , Σ b, ∞ , Σ ∞ } .
Bayes rules g ρ are averages of eigenfunctions of Σ a, ∞ (left), Σ b, ∞ (middle), and Σ ∞ (right) corresponding to the 10-largest eigenvalues excluding the ﬁrst and second, with the setting: R = 1 / (20 √ 2) , γ = 10 √ 2 , and ρ X is the uniform distribution on the unit sphere in R 2 .
To estimate eigenvalues and eigenfunctions, we draw 10 4 -samples from ρ X and M = 2 × 10 4 -hidden nodes of a two-layer ReLU.
Empirical observations.
We observe g ρ has the smallest norm with respect to the integral operator which speciﬁes g ρ and has a comparably small norm with respect to Σ ∞ even for the cases where g ρ is speciﬁed by Σ a, ∞ or Σ b, ∞ .
This observation suggests the efﬁciency of learning a corresponding layer to g ρ and learning both layers, and it is empirically veriﬁed.
We run Algorithm 1 10 -times with respect to output (blue), input (purple), and both layers (orange) of two-layer swish networks with s = 10 .
Figure 2 (Bottom) depicts the average and standard deviation of test errors.
From the ﬁgure, we see that learning a corresponding layer to g ρ and both layers exhibit faster convergence, and that misspeciﬁcation signiﬁcantly slows down the convergence speed in all cases.
C ONCLUSION
We analyzed the convergence of the averaged stochastic gradient descent for overparameterized twolayer neural networks for a regression problem.
Through the development of a new proof strategy that does not rely on the positivity of the NTK, we proved that the global convergence (Theorem 1) relies only on the overparameterization.
Moreover, we demonstrated the minimax optimal convergence rates (Corollary 1) in terms of the generalization error depending on the complexities of the target function and the hypothesis class and showed the explicit optimal rate for the smooth approximation of the ReLU.
Appendix A P ROOF S KETCH OF THE M AIN R ESULTS
We provide several key results and a proof sketch of Theorem 1 and Corollary 1.
We ﬁrst recall the deﬁnition of stochastic gradients of L in a general RKHS ( H , (cid:104) , (cid:105) H ) associated with a uniformly bounded real-valued kernel function k : X × X → R .
We set K X
= k
( X, · ) .
Then, it follows that for ∀ g, ∀ h ∈ H ,
[Equation]
g
[Equation]
+
[Equation] [Equation]
g
[Equation]
X
[Equation]
[Equation] [Equation]
(cid:105)
[Equation] [Equation]
h (cid:107) H ) , which is conﬁrmed by the following equations:
E [ l (( g + h )( X ) , Y )]
=
E [ l ( g ( X ) , Y ) + ∂ ζ l ( g ( X ) , Y ) h
( X )
+ o
( | h ( X ) | )] , h ( X ) =
(cid:104) h, k
( X, · ) (cid:105) H , and | h ( X ) | ≤ (cid:107) h (cid:107) H (cid:112) k
( X, X ) .
This means that the stochastic gradient of
L in H is given by ∂ ζ (cid:96) ( g ( X ) , Y ) k
( X, · ) for ( X, Y ) ∼ ρ .
In addition, the stochastic gradient of the
L 2 -regularized risk is given by ∂ ζ (cid:96) ( g ( X ) , Y ) k
( X, · ) + λg .
A.R EFERENCE A VERAGED S TOCHASTIC G RADIENT D ESCENT
We consider a random feature approximation of NTK k ∞ : for an initial value B (0) =
( b (0) r )
Mr =1 , ∀ x, ∀ x (cid:48) ∈ X ,
[Equation]
x
[Equation]
We can conﬁrm that k M is an approximation of NTK, that is, k M converges to k ∞ uniformly over supp( ρ X ) × supp( ρ X ) almost surely by the uniform law of large numbers.
We denote by ( H M , (cid:104) , (cid:105) H M ) an RKHS associated with k M .
By the assumptions, we see k M ( x, x (cid:48) ) ≤ 12 for ∀ ( x, x (cid:48) )
∈ supp( ρ X ) × supp( ρ X ) .
We introduce averaged stochastic gradient descent in H M (see Algorithm 2) as a reference for
Algorithm 1.
The notation G ( t ) represents a stochastic gradient at the t -th iterate:
[Equation]
end
[-( T ) = (cid:80) Tt =
 α t g ( t )-] The following proposition shows the equivalence between the averaged stochastic gradient descent for two-layer neural networks and that in H M up to a small constant depending on M .
Proposition A. Suppose Assumptions (A1) and (A2) hold.
Run Algorithms 1 and 2 with the constant learning rate η t = η satisfying ηλ < 1 and η ≤ 1 .
Moreover, assume that they share the same hyper-parameter settings and the same examples ( x t , y t ) T − 1 t =0 to compute stochastic gradient.
Then, for arbitrary T ∈ Z + and (cid:15) > 0 , there exists M ∈ Z + depending only on T and (cid:15) such that ∀ t ≤ T ,
[Equation]
(6) where g Θ ( t ) and g ( t ) are iterates obtained by Algorithm 1 and 2, respectively.
Remark.
Note that this proposition holds for non-averaged SGD too because it is a special case of averaged SGD by setting only one α t to 1 .
Key idea.
This proposition is the key because it connects two learning dynamics for neural networks and RKHS H M by utilizing overparameterization without the positivity of NTK unlike existing studies (Weinan et al., 2019; Arora et al., 2019b) that provide such a connection for continuous gradient ﬂow with the positive NTK.
Instead of the positivity of NTK, Proposition A says that overparameterization increases the time stayed in the NTK regime where the learning dynamics for neural networks can be characterized by the NTK.
As a result, because M is free from the other hyper-parameters, the averaged stochastic gradient descent for the overparameterized two-layer neural networks can fully inherit preferable properties from learning dynamics in H M with an appropriate choice of learning rates and regularization parameters as long as the network width is sufﬁciently large depending only on the number of iterations and the required accuracy.
A.C ONVERGENCE R ATE OF THE R EFERENCE ASGD
We give the convergence analysis of Algorithm 2 in H M , which will be a part of a bound in Theorem 1.
Proofs essentially rely on several techniques developed in serial studies (Bach & Moulines, 2013;
Dieuleveut & Bach, 2016; Dieuleveut et al., 2017; Pillaud-Vivien et al., 2018a; Rudi & Rosasco, 2017; Carratino et al., 2018) with several adaptations to our settings.
Let M ∈ Z + ∪ {∞} be a positive number or ∞ .
We set K M,X def = k M ( X, · ) and denote by Σ M the covariance operator deﬁned by k M :
[Equation]
We denote by g M,λ the minimizer of the regularized risk over H M :
[Equation]
(cid:27) . We remark that Σ M : L 2 ( ρ X ) → H M is isometric (Cucker & Smale, 2002), that is, ∀ ( f, g ) ∈
L 2 ( ρ X ) × L 2 ( ρ X ) ,
[Equation]
and we use this fact frequently.
It is known that g M,λ is represented as follows (Caponnetto & De Vito, 2007):
[Equation]
The following theorem provides a convergence rate of Algorithm 2 to the minimizer g M,λ .
Theorem A. Suppose Assumptions (A1) , (A2) and (A3) hold.
Run Algorithm 2 with the constant learning rate η t
=
η satisfying 4(6+ λ ) η ≤ 1 .
Then, for ∀ λ > 0 and ∀ δ ∈ (0 , 1) there exists M 0 >such that for ∀ M ≥ M 0 the following holds with high probability at least 1 − δ :
[Equation]
where g ( T ) is an iterate obtained by Algorithm 2.
Remark.
The ﬁrst and second terms stem from the optimization speed of a semi-stochastic part of averaged stochastic gradient descent.
The ﬁrst term has a better dependency on T , but it has a worse dependency on λ than the second one.
This kind of deterioration due to the weak strong convexity is common in ﬁrst-order optimization methods.
However, as conﬁrmed later, these two terms are dominated by the variance term corresponding to the third term by setting hyper-parameters appropriately.
To make the bound in Theorem A free from the size of M , we introduce the following proposition.
Proposition B. Suppose g ρ ∈ H ∞ holds.
Under Assumption (A1) and (A2) , for any δ ∈ (0 , 1) , there exists M 0 ∈
Z
+ such that for any M ≥ M 0 , the following holds with high probability at least 1 − δ
[Equation]
and if λ ≤ (cid:107) Σ ∞ (cid:107) op , then
[Equation]
Remark.
The last inequality on the degree of freedom was shown in Rudi & Rosasco (2017).
To show the convergence to g ρ , we utilize the following decomposition:
[Equation]
where g ∞ ,λ def = arg
[Equation] [Equation]
The ﬁrst term is the optimization speed evaluated in Theorem A, and the second and third terms are approximation errors from a random feature approximation of NTK and imposing L 2 -regularization, respectively.
These approximation terms can be evaluated by the following existing results.
The next proposition is a simpliﬁed version of Lemma 8 in Carratino et al.
Proposition C (Carratino et al.
Under Assumption (A1) , (A2) , and (A3) , for any (cid:15), λ >and δ ∈ (0 , 1] , there exists M 0 ∈
Z + depending on (cid:15), λ, δ such that for any M ≥ M 0 , the following holds with high probability at least 1 − δ :
[Equation]
Proposition D (Caponnetto & De Vito (2007)) .
Under Assumption (A3) , it follows that
[Equation]
By combining Theorem A, Proposition B, C, and D with the decomposition (8), we can establish the convergence rate of reference ASGD to reach g ρ , which is simply the generalization error bound.
Theorem B. Assume the same conditions as in Theorem A.
Then, for ∀ (cid:15) > 0 , (cid:107) Σ ∞ (cid:107) op ≥ ∀ λ > 0 , and ∀ δ ∈ (0 , 1) , there exists M 0 ∈
Z
+ such that for ∀ M ≥ M 0 , the following holds with high probability at least 1 − δ over the random choice of random features Θ (0) :
[Equation]
+
[Equation]
where g ( T ) is an iterate obtained by Algorithm
[Equation]
A.C ONVERGENCE R ATES OF ASGD FOR N EURAL N ETWORKS
As explained earlier, the generalization bound for the reference ASGD is inherited by that for twolayer neural networks through Proposition A with the following decomposition: for an iterate Θ T obtained by Algorithm 1,
[Equation]
That is, these two terms are bounded by Proposition A and Theorem B under Assumption (A1)-(A3) , resulting in Theorem 1, which exhibits comparable generalization error to Theorem B as long as the network width M is sufﬁciently large.
Theorem 1 immediately leads to the fast convergence rate in Corollary 1 by setting η t =
η = O (1) satisfying 4(6 + λ ) η ≤ 1 and λ
= T − β/ (2 rβ +1) with the bounds on (cid:107) g
ρ (cid:107) 2 H ∞
, (cid:107) g ρ (cid:107) 2 L 2 ( ρ X ) , and the degree of freedom.
Because β in Assumption (A4) controls the complexity of the hypothesis space
H ∞ , it derives a bound on the degree of freedom, as shown in Caponnetto & De Vito (2007):
[Equation]
In addition, the boundedness of (cid:107) Σ ∞ (cid:107) op ≤ O (1) gives
[Equation]
This ﬁnishes the proof of Corollary 1.
B P ROOF OF P ROPOSITION A
We ﬁrst show the Proposition A that says the equivalence between averaged stochastic gradient descent for two-layer neural networks and that in an RKHS associated with k M .
Proof.
Proof of Proposition A
Bound the growth of (cid:107) g Θ ( t ) (cid:107) L ∞ ( ρ X ) .
We ﬁrst show that there exist increasing functions d ( t ) and
M ( t ) depending only on t uniformly over the choice of the history of examples ( x t , y t ) ∞ t =1 used in
Algorithms such that (cid:107) g Θ ( s ) (cid:107) L ∞ ( ρ X ) ≤ d ( t ) for ∀ s ≤ t when M ≥ M ( t ) .
We show this statement by the induction.
Without loss of generality, we assume that there is no bias term, (cid:107) b (0) r (cid:107)
= 1 , and supp( ρ X )
⊂ { x ∈ R d + | (cid:107) x (cid:107) 2 ≤ 2 } by setting x ← ( x, γ ) (where γ ∈ (0 , 1) ).
Hence, we consider the update only for parameters a and B .
The above statement clearly holds for t = 0 .
Thus, we assume it holds for t .
We recall the speciﬁc update rules of the stochastic gradient descent:
[Equation]
Here, let us consider ∀ M ≥ M ( t ) .
Set d M b ( t ) = max s ≤ t, 1 ≤ r ≤ M (cid:107) b ( s ) r (cid:107) 2 .
Then, by expanding equation (9), we get
[Equation]
[Equation]
where we used (cid:107) σ ( u ) (cid:107) ≤ 1 + u and | y t | ≤ 1 .
As for the term | a ( s ) r | ( s ≤ t + 1 ), from the similar augment for s and the monotonicity, we have for s ≤ t + 1 ,
[Equation]
We next give a bound on (cid:107) b ( t +1)
r − b
[Equation]
0) r (cid:107) 2 .
By expanding equation (10), we get
[Equation]
[Equation]
where we used (cid:107) σ (cid:48) (cid:107) ∞ ≤ 2 and (cid:107) x t (cid:107) 2 ≤ 2 .
Here, we evaluate d Mb ( t ) .
From the similar augment for s ≤ t , the monotonicity, and (cid:107) b ( s ) r (cid:107) 2 ≤ 1 + (cid:107) b ( s ) r − b (0) r (cid:107) 2 , we get
[Equation]
Let M (cid:48)
( t + 1) be a positive integer depending on t and d ( t )
such that t +1 √ M ( d ( t ) + 1) ≤ 14 .
Let us reconsider ∀ M ≥ M (cid:48)
( t + 1) .
Then, since η ≤ 1 , we have
[Equation]
From the derivation of (11) and (12) and since η ≤ 1 , we have for 0 ≤ ∀ s ≤ t + 1 ,
[Equation]
[Equation]
where d 1 ( t + 1) and d 2 ( t + 1) are set to d
[Equation]
def = 6( t
[Equation]
d
[Equation] [Equation]
We next bound | g Θ ( t +1)
( x ) | for x ∈ ∀ supp( ρ X ) as follows.
Since g Θ (0) ≡ 0 , |
[Equation]
(cid:13)(cid:13)(cid:13) b ( t +1) r − b (0) r (cid:13)(cid:13)(cid:13)(cid:111) ≤ 2 d 1 ( t + 1) + 10 d 2 ( t + 1) .
In summary, by setting M ( t + 1) = max { M ( t ) , 16( t + 1) 2 ( d ( t ) + 1) 2 } and d ( t + 1) = 2 d 1 ( t + 1) + 10 d 2 ( t + 1) , we get (cid:107) g Θ ( t +1)
(cid:107) L ∞ ( ρ X ) ≤ d ( t + 1) when M ≥ M ( t + 1) .
We note that from the above construction, d ( t ) , d 1 ( t ) , d 2 ( t ) depend only on t and inequalities (13) are always hold for ∀ t
∈ Z + when M ≥ M ( t + 1) .
Linear approximation of the model.
For a given T ∈ Z + , we consider ∀ M ≥ M ( T ) and deﬁne the neighborhood of Θ (0) =
( a (0) r , b (0) r )
Mr =1 :
[Equation]
) r (cid:107)
[Equation]
d
[Equation]
T ) √ M (cid:27) .
From Taylor’s formula | σ ( b (cid:62) r x ) − σ ( b (0) (cid:62) r x ) − σ (cid:48)
( b (0) (cid:62) r x )( b r − b (0) r ) (cid:62) x | ≤ 2 (cid:107) σ (cid:48)(cid:48) (cid:107) ∞ (cid:107) b r − b (0) r
(cid:107)and the smoothness of σ , we get for Θ ∈ B T (Θ (0) ) and x ∈ supp( ρ X ) ,
[Equation]
a
[Equation]
We here deﬁne a linear model:
[Equation]
By taking the sum of (14) over r ∈ { 1 , . . .
, M } and by g Θ (0) ≡ 0 ,
[Equation]
We denote d 3 ( T ) def = 2 d 1 ( T ) d 2 ( T ) + 5 Cd 22 ( T ) .
Since iterates (Θ ( t ) )
Tt =0 obtained by Algorithmare contained in B T (Θ (0) ) , weighted averages (Θ ( t ) )
Tt =0 are also contained in B T (Θ (0) ) .
Thus, we get for ∀ t ∈ { 1 , . . .
, T } ,
[Equation]
Recursion of h Θ ( t ) using the random feature approximation of NTK.
We here derive a recursion of h Θ ( t ) using k M .
From the updates (9) and (10), we have
[Equation]
Note that for t ∈ { 0 , . .
, T }
[Equation]
+ 2( d ( T ) + 1) d 2 ( T )) , and | ( g Θ ( t ) ( x t ) − y t ) a ( t ) r σ (cid:48)
( b ( t ) (cid:62) r x t ) − ( h Θ ( t ) ( x t ) − y t ) a (0) r σ (cid:48)
( b (0) (cid:62) r x t ) | ≤ | ( g Θ ( t ) ( x t ) − h Θ ( t ) ( x t ))
a (0) r σ (cid:48)
( b (0) (cid:62) r x t ) | + | ( g Θ ( t ) ( x t ) − y t )( a ( t ) r σ (cid:48)
( b ( t ) (cid:62) r x t ) − a (0) r σ (cid:48)
( b (0) (cid:62) r x t ))
| ≤ 2 d 3 ( T ) √
M + ( d ( T ) + 1) | a ( t ) r σ (cid:48) ( b ( t ) (cid:62) r x t ) − a (0) r σ (cid:48)
( b (0) (cid:62) r x t ))
| ≤ 2 d 3 ( T ) √
M + ( d ( T ) + 1) (cid:110) | a (0) r
( σ (cid:48)
( b ( t ) (cid:62) r x t ) − σ (cid:48) ( b (0) (cid:62) r x t ))
| + | ( a ( t ) r − a (0) r ) σ (cid:48)
( b ( t ) (cid:62) r x t ) | (cid:111) ≤ 2 d 3 ( T ) √
M + 2( d ( T ) + 1) (cid:110) C (cid:107) b ( t )
r − b (0) r (cid:107) 2 + | a ( t ) r − a (0) r | (cid:111) ≤ 2 d 3 ( T ) √
M + 2( d ( T ) +
[Equation]
Plugging these two inequalities into (16), we have ∀ t ∈ { 1 , . .
, T − 1 } ,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where d
[Table]
T
[Equation]
[Table]
d
[Equation]
+
[Equation]
[Table]
[Equation]
( C + 2) d 2 ( T )) .
Clearly, the inverse inequality also holds:
[Equation]
Thus, we get
[Equation]
Equivalence between Algorithm 1 and 2.
We provide a bound between recursions of Algorithm 2 and (17).
Noting that h Θ (0) ≡ g (0) ≡ 0 , we have for ∀ t ∈ { 0 , . .
, T − 1 } ,
[Equation]
Noting (cid:107) k M (cid:107) L ∞ ( ρ X ) ≤ 12 and taking a supremum over x
[Equation]
x t ∈ supp( ρ X ) in both sides
[Equation]
we have (cid:107)
[Equation] [Equation]
Since h Θ is a linear model, we have h Θ ( T ) =
(cid:80) Tt =
α t h Θ ( t ) and
[Equation]
Combining this inequality with (15), we ﬁnally have
[Equation]
Because ( d 3 ( T ) + 13 T Td 4 ( T )) depends only on T and C from the construction, (cid:107) g Θ ( T ) − g
( T ) (cid:107) L ∞ ( ρ X ) → 0 as M → ∞ .
This ﬁnishes the proof of Proposition A.
C P ROOF OF T HEOREM A
In this section, we give the proof of the convergence theory for the reference ASGD
(Algorithm 2).
We introduce an auxiliary result for proving Theorem A.
Lemma A. Suppose Assumption ( A1 ), ( A2 ), and ( A3 ) hold.
Set ξ def = Y K M,X − ( K M,X ⊗ H M
K M,X + λI ) g M,λ .
Then, for ∀ λ > 0 and ∀ δ ∈ (0 , 1) there exists M 0 > 0 such that for ∀ M ≥ Mthe following holds with high probability at least 1 − δ :
[Equation]
Proof.
Since ξ = ( Y − g M,λ ( X ))
K M,X − λg M,λ , we get
[Equation]
We evaluate an expectation in the second and third terms in the right hand side of the above equation as follows:
[Equation]
Hence, we get
E
[Equation]
For h ∈ H M , (cid:10)
[Equation]
(18) where we used Assumption (A2) for the last inequality.
Finally
[Equation]
we provide an upper-bound on (cid:107) g M,λ (cid:107) L ∞ ( ρ X ) .
Since S − 1 − T − 1 = − S − 1 ( S − T ) T − 1 for arbitrary operators S and T
[Equation]
we get
[Equation]
Σ
[Equation]
+ λI
[Equation]
We denote F ∞ =
(Σ ∞ + λI ) − 1 g ρ and F M =
(Σ M + λI ) − 1 g ρ .
Noting g ∞ ,λ = Σ ∞ F ∞ and g M,λ = Σ M F M , we get for ∀ x ∈ supp( ρ X ) , | g ∞ ,λ ( x ) − g M,λ ( x ) | =
| Σ ∞ F ∞ ( x ) − Σ M F M ( x ) |
[Equation]
≤ (cid:107) K ∞ ,x − K M,x (cid:107) L 2 ( ρ X ) (cid:107) F ∞ (cid:107) L 2 ( ρ X ) + (cid:107) K M,x
[Equation]
L
[Equation]
ρ
[Equation] [Equation]
F M
[Equation]
F ∞
[Equation]
L
[Equation]
ρ X ) ≤λ
[Equation]
k ∞ −
[Equation]
M (cid:107) L ∞ ( ρ X ) 2 (cid:107) g ρ (cid:107)
L 2 ( ρ X )
[Equation]
12 λ
[Equation]
where we used k M ( x, x (cid:48) )
≤ 12 for ∀ ( x, x (cid:48) )
∈ supp( ρ X ) × supp( ρ X ) and inequality (19).
Moreover, we get
[Equation]
where we used Assumption ( A3 ) and the isometric map Σ 1 / 2 ∞ :
L 2 ( ρ X )
→ H ∞ .
Hence
[Equation]
we get
[Equation]
By the uniform law of large numbers (Theorem 3.1 in Mohri et al.
and the Bernstein’s inequality (Proposition 3 in Rudi & Rosasco (2017)) to random operators, (cid:107) k ∞ − k M (cid:107) L ∞ ( ρ X × ρ X ) and (cid:107) Σ ∞ − Σ M (cid:107) op converge to zero as M → ∞ in probability.
That is, for given λ > 0 and δ ∈ (0 , 1) , there exists M 0 such that for any M ≥ M 0 the following holds with high probability at least 1 − δ :
[Equation]
Combining with (18), we get
[Equation]
Proof of Theorem A. Since, stochastic gradient in H M is described as
[Equation]
the update rule of Algorithm 2 is
[Equation] [Equation]
,x [Equation]
Hence, we get
[Equation] [Equation]
This leads to the following stochastic recursion: t ∈ { 0 , . . .
, T − 1 } ,
[Equation]
By taking the average, we get
[Equation]
Thus, the average A T is composed of bias and noise terms.
We next bound these two terms, separately.
Bound the bias term.
Note that the bias term exactly corresponds to the recursion (21) with β t = 0 .
Hence, we consider the case of β t = 0 and consider the following stochastic recursion in
H M :
A
[Equation]
g M,λ ,
[Equation]
where we deﬁne H t = K M,x t ⊗ H M K M,x t .
In addition, we consider the deterministic recursion of this recursion: A (cid:48)
[Equation]
We set
[Equation]
Then, the bias term we want to evaluate is decomposed as follows: by Minkowski’s inequality,
[Equation]
We here bound the ﬁrst term in the right hand side of (23).
Note that from 4(6 + λ ) η ≤ 1 and (cid:107) k M (cid:107) L ∞ ( ρ X ) 2 ≤ 12 , we see 1 η (Σ M + λI ) (cid:52) η
(12+ λ ) I ≺ 12 I .
Since A (cid:48) t
= ( I − η Σ M − ηλI )
t g M,λ , its average is
[Equation]
Therefore,
[Equation]
We bound the second term in (23)
[Equation]
which measures the gap between A T and A (cid:48) T .
To do so, we consider the following recursion:
[Equation]
Hence
[Equation]
we have
[Equation]
Let ( F t ) T − 1 t =0 be a ﬁltration.
We take a conditional expectation given F t :
[Equation]
where we used (cid:107)
[Equation]
+
[Equation]
For g ∈ H M
[Equation]
we have
[Equation]
where we used k M ( x t , x t ) ≤ 12 which is conﬁrmed from the deﬁnition of k M and Assumption (A2)
[Equation]
This means that E [ H 2 t ] =
E [( K M,x t ⊗ H M K M,x t ) 2 ] (cid:52) 12Σ
M on H M × H M .
Hence
[Equation]
we get a bound on (25) as follows:
[Equation]
Next
[Equation]
we bound a term (26):
[Equation]
Combining these inequalities
[Equation]
we get
[Equation]
where for the last inequality we used 4 η (6 + λ ) ≤ 1 .
By taking the expectation and the average of (28) over t ∈ { 0 , . . .
, T − 1 } , we get
[Equation]
T
[Equation]
Since Σ 1 / 2 M : L 2 ( ρ X ) → H M is isometric, we see (cid:107) Σ 1 / 2 M ( A t − A (cid:48) t ) (cid:107) H M
=
(cid:107)
A t − A (cid:48) t (cid:107) L 2 ( ρ X ) .
Thus, the second term in (23) can be bounded as follows:
[Equation]
(29) where we used the convexity for the ﬁrst inequality and we used the following inequality for the last inequality: since (cid:107)
[Equation]
(cid:107)
[Equation]
and
[Equation] [Equation]
[Equation]
T
[Equation]
By plugging (24) and (29) into (23)
[Equation]
we get the bound on the bias term:
[Equation]
Bound the noise term.
Note that the noise term in (22) exactly corresponds to the recursion (21) with A 0 = 0 .
Hence, it is enough to consider the case of A 0 = 0 to evaluate the noise term.
In this case, the average A T can be rewritten as follows:
[Equation]
We here evaluate the noise term.
We set z s =
( x s , y s ) .
Note that since E z s [ β s ] = 0 , we have for s < s (cid:48) ,
E ( z s ,...,z T ) (cid:104) (cid:104)
Z s , Z s (cid:48) (cid:105) L 2 ( ρ X ) (cid:105) =
(cid:90)
X
E ( z s ,...,z T ) (cid:34)(cid:32) T (cid:88) t =
s t (cid:89) l = s + α l β s η (cid:33) (cid:32) T (cid:88) t = s (cid:48) t (cid:89) l = s (cid:48)
+ α l β s (cid:48)
η
(cid:33)(cid:35) d ρ X = (cid:90)
X E z s [
[Equation]
] E ( z s +1 ,...,z T ) (cid:34) β s (cid:48) (cid:32) T (cid:88) t = s t (cid:89) l
= s + α l η (cid:33) (cid:32) T (cid:88) t
= s (cid:48)
t (cid:89) l = s (cid:48)
+ α l η (cid:33)(cid:35) d ρ X = 0 .
Therefore, we have
[Equation]
Here
[Equation]
we apply Lemma 21 in Pillaud-Vivien et al.
(2018a)
with A = Σ M , H = Σ M + λI , C = 2(1 + (cid:107)
[Equation]
+
[Equation]
Σ
[Equation]
r ∞ g ρ
[Equation]
ρ X )
[Equation]
Σ M .
One of required conditions in this lemma is veriﬁed by
Lemma A. We verify the other required condition described below:
[Equation]
Indeed, we have
[Equation]
, where we used Σ M (Σ M + λI ) − 1 (cid:52) I and λ (Σ M + λI ) − 1 (cid:52) I .
Moreover
[Equation]
we see
E (cid:2) K M,X ⊗ H K M,X CH [Equation] K M,
[Equation]
Σ M
[Equation]
where we used (27) for the last inequality.
Hence, we get
[Equation]
Since
[Equation]
4 η (6 + λ ) ≤
[Equation]
the condition (32) is veriﬁed.
We apply Lemma 21 in Pillaud-Vivien et al.
(2018a) to (31), yielding the following inequality:
[Equation]
Convergence rate in terms of the optimization.
Finally, by combining (30) and (33) with (22)
[Equation]
we get the convergence rate of averaged stochastic gradient descent to g M,λ :
[Equation]
where g ( T ) def = 1 T +1 (cid:80)
Tt =
g ( t ) .
This ﬁnishes the proof.
D P ROOF OF P ROPOSITION B
We provide Proposition B which provides the bound on Theorem A.
Proof of Proposition B. From the Bernstein’s inequality (Proposition 3 in Rudi & Rosasco (2017)) to random operators
[Equation]
the covariance operator Σ M converges to Σ ∞ as M → ∞ in probability.
Especially, there exits M 0 ∈ Z
+ such that for any M ≥ M 0 , it follows that with high probability at least 1 − δ , Σ ∞ − Σ M (cid:52) 12 (Σ ∞ + λI ) in L 2 ( ρ X ) .
Thus, for ∀ f ∈ L 2 ( ρ X ) , we see
[Equation]
Hence
[Equation]
we have
[Equation]
Following the argument in Bach (2017b)
[Equation]
we have for ∀ f ∈
L 2 ( ρ X ) ,
[Equation]
Thus
[Equation]
we conﬁrm
that with high probability at least 1 − δ ,
[Equation]
Utilizing this inequality
[Equation]
we show the ﬁrst and second inequalities in Proposition B as follows.
It is sufﬁcient to prove the second inequality because of
[Equation]
Noting that g ρ ∈ H ∞ and g M
[Equation]
λ =
(Σ M + λI ) − 1 Σ M g ρ
[Equation]
we get
[Equation]
The third inequality on the degree of freedom is a result obtained by Rudi & Rosasco (2017).
E E IGENVALUE A NALYSIS OF N EURAL T ANGENT K ERNEL E.R EVIEW OF S PHERICAL H ARMONICS
We brieﬂy review the spherical harmonics which is useful in analyzing the eigenvalues of dot-product kernels.
For references, see Atkinson & Han (2012); Bach (2017a);
Bietti & Mairal (2019); Cao et al.
Here, we denote by τ d − 1 is the uniform distribution on the sphere S d − 1 ⊂ R d .
The surface area of
S d − 1 is ω d − 1 =
π d/ 2 Γ( d/ 2) where Γ is the Gamma function.
In L 2 ( τ d − 1 ) , there is an orthonomal basis consisting of a constant 1 and the spherical harmonics Y kj ( x ) , k ∈ Z ≥ 1 , j = 1 , . . .
N ( d, k ) , where
N ( d, k ) =
k + d − 2 k
[Equation]
That is, (cid:104)
Y ki , Y sj (cid:105)
L 2 ( τ d − 1 ) =
δ ks δ ij and (cid:104)
Y ki , 1 (cid:105)
L 2 ( τ d − 1 )
= 0 .
The spherical harmonics Y kj are homogeneous functions of degree k , and clearly Y kj have the same parity as k .
Legendre polynomial P k ( t ) of degree k and dimension d (a.k.a.
Gegenbauer polynomial ) is deﬁned as (Rodrigues’ formula):
[Equation]
Legendre polynomials have the same parity as k .
This polynomial is very useful in describing several formulas regarding the spherical harmonics.
Addition formula.
We have the following addition formula:
[Equation]
Hence, we see that P k ( x (cid:62) · ) is spherical harmonics of degree k .
Using the addition formula and the orthogonality of spherical harmonics, we have
[Equation]
[Equation]
Combining the following equation: for x = te d + √ 1 − t 2 x (cid:48) , ( x ∈ S d − 1 , x
(cid:48) ∈ S
d − 2 , t ∈ [ − 1 , 1]) ,
[Equation]
we see the orthogonality of Legendre polynomials in L 2 ([ − 1 , 1] , (1 − t 2 ) ( d − 3) / 2 d t ) and since
P k (1) =
1 we see
[Equation]
Recurrence relation.
We have the following relation:
[Equation] [Equation]
for k ≥ 1 , and for k =
we have tP 0 ( t ) =
P 1 ( t ) .
Funk-Hecke formula.
The following formula is useful in computing Fourier coefﬁcients with respect to spherical harmonics via Legendre polynomials.
For any linear combination Y k of Y kj , ( j ∈ { 1 , . . .
, N ( d, k ) } ) and any f ∈ L 2 ([ − 1 , 1] , (1 − t 2 ) ( d − 3) / 2 d t ) , we have for ∀ x ,
[Equation]
[Equation]
This formula say that spherical harmonics are eigenfunctions of the integral operator deﬁned by f ( x (cid:62) y ) and each eigen-space is spanned by spherical harmonics of the same degree.
Moreover, it also provides a way of computing corresponding eigenvalues.
E.E IGENVALUES OF D OT - PRODUCT K ERNELS
Let µ 0 be the uniform distribution on S d − 1 .
Note that although τ d − 1 and µ 0 are the same distribution, we use two distributions τ d − 1 and µ 0 depending on random variables.
First, we consider any activation function σ : R → R and a kernel function k ( x, x (cid:48) )
=
E b (0) ∼ µ
σ ( b (0) (cid:62) x ) σ ( b (0) (cid:62) x (cid:48) )
] on the sphere S d − 1 .
We show this kernel function is a type of dot-product kernels, that is, there is ˆ k : R → R such that k
( x, x (cid:48) )
=
ˆ k ( x (cid:62) x (cid:48) ) .
In fact, it can be conﬁrmed as follows.
For any x, x (cid:48) ∈ S d − 1 , we take θ ∈ [0 , π ] so that x (cid:62) x (cid:48)
=
cos θ , and an orthogonal matrix A ∈ R d × d so that
Ax = (1 , 0 , . . .
, 0) (cid:62) and Ax (cid:48)
= (cos θ, sin θ, 0 , . .
. , 0) (cid:62) because A preserves the value of x (cid:62) x (cid:48) .
Then, since µ 0 is rotationally invariant we see
[Equation]
where b (0) =
( b 01 , b (0)2 , . . .
, b (0) d ) .
In other words, we see k is a function of θ = arccos ( x (cid:62) x (cid:48) ) , and is a dot-product kernel k ( x, x (cid:48) )
=
ˆ k ( x (cid:62) x (cid:48) ) .
Hence, we can apply Funk-Hecke formula (38) to k ( x, · ) .
The derivation of eigenvalues of the integral operator follows a way developed by Bach (2017a);
Bietti & Mairal (2019); Cao et al.
In general, g ∈ L 2 ( τ d − 1 ) can be decomposed by spherical harmonics as follows.
[Equation]
where we used addition formula to the last equality.
Here, we apply this decomposition (39) to k ( x, · )
=
ˆ k ( x (cid:62) · ) .
Since P k ( Z (cid:62) · ) is a linear combination of spherical harmonics of degree k
(see addition formula), we get
[Equation]
where we used Funk-Hecke formula (38) and we set ˆ λ k = ω
d − 2 ω
d − 1 (cid:82) 1 − 1 ˆ k ( t ) P k ( t )(1 − t 2 ) ( d − 3) / 2 d t .
We note that ˆ λ k is eigenvalue with multiplicity N ( d, k ) of the integral operator deﬁned by k .
Next, we derive another expression of k .
In a similar way, we obtain the following equation:
[Equation]
where ˆ µ k = ω
d − 2 ω
d
[Equation] [Equation]
1 σ ( t ) P k ( t )(1 − t 2 ) ( d − 3) / 2 d t .
By the deﬁnition of k and the orthogonality of spherical harmonics, we get
[Equation]
S
[Equation]
where we used equation (36).
By the rotationally invariance
[Equation]
we can show
[Equation]
Thus, comparing (40) with (41)
[Equation]
we get ˆ λ k
= ˆ µ 2 k .
E.E IGENVALUES OF N EURAL T ANGENT K ERNELS
Utilizing a relation ˆ λ k
= ˆ µ 2 k
[Equation]
we derive a way of computing eigenvalues of the integral operator deﬁned by the integral operators Σ ∞ associated with the activation σ .
Recall the deﬁnition of the neural tangent kernel:
[Equation]
A neural tangent kernel consists of three kernels:
[Equation]
By the argument in the previous subsection, h 1 and h 2 are dot-product kernel, that is, there exist ˆ hand ˆ h 2 such that h 1 ( x, x (cid:48) )
=
ˆ h 1 ( x (cid:62) x (cid:48) ) and h 2 ( x, x (cid:48) )
=
ˆ h 2 ( x (cid:62) x (cid:48) ) .
Moreover, h 3 is a dot-product kernel as well because we get h 3 ( x, x (cid:48) )
=
ˆ h 3 ( x (cid:62) x (cid:48) ) by setting ˆ h 3 ( t ) =
t ˆ h 2 ( t ) .
Hence, theory explained earlier is applicable to these kernels.
Eigenvalues ˆ µ k for kernels h 1 and h 2 are described as follows:
[Equation]
yielding eigenvalues
ˆ λ (1) k
= (ˆ µ (1) k ) 2 and ˆ λ (2) k
= (ˆ µ (2) k ) 2 for h 1 and h 2 , respectively.
As for eigenvalues ˆ λ (3) k for h 3 , we have
[Equation]
where we used the recurrence relation (37).
Since, h 1 , h 2 , and h 3 have the same eigenfunctions, eigenvalues ˆ λ ∞ ,k of k ∞ is
[Equation]
[Equation]
Hence, calculation of { ˆ λ ∞ ,k } ∞ k =1 results in computing ˆ µ (1) k and ˆ µ (2) k for given activation σ .
Eigenvalues for ReLU and smooth approximations of ReLU.
As for ReLU activation, its eigenvalues were derived in Bach (2017a).
Let σ be ReLU.
Then, ˆ µ (1) k
= 0 and ˆ µ (2) k ∼ k − d/ 2 when k is odd and ˆ µ (1) k ∼ k − d/ 2 − 1 and ˆ µ (2) k
= 0 when k is even.
Consequently, we see ˆ λ ∞ ,k
= Θ( k − d ) .
We note that the multiplicity of ˆ λ ∞ ,k is N ( d, k ) , so that we should take into account the multiplicity to derive decay order of eigenvalues λ ∞ ,i of Σ ∞ .
Since 1+ (cid:80) kj =1 N ( d − 1 , j ) =
N ( d, k ) (for details see Atkinson & Han (2012)), we see λ ∞ ,N ( d +1 ,k ) =
Θ( k − d ) .
Moreover, N ( d + 1 , k ) =
Θ( k d − 1 ) yields λ ∞ ,N ( d +1 ,k )
=
Θ (cid:16) N ( d + 1 , k ) − 1 − 1 d − 1 (cid:17) .
As a result, Assumption (A4) is veriﬁed with β = 1 + 1 d − 1 for ReLU.
For the smooth approximation σ ( s ) of ReLU satisfying Assumption (A1’) , we can show that every eigenvalue of Σ ( s ) ∞ derived from σ ( s ) converges to that for ReLU as s → ∞ because of (42) and (43) with Lebesgue’s convergence theorem.
F E XPLICIT C ONVERGENCE R ATES FOR S MOOTH A PPROXIMATION OF R E LU
For convenience, we here list notations used in this section.
In this section, let σ and σ ( s ) be ReLU activation and its smooth approximation satisfying (A1’) , respectively, and for M ∈ Z + ∪ {∞} let k M , Σ M , g M,λ , k ( s ) M , Σ ( s ) M , g ( s ) M,λ be corresponding kernel, integral operators, and minimizers of the regularized expected risk functions.
Let g ( T ) be iterates obtained by the reference
ASGD (Algorithm 2) in the RKHS associated with k ( s ) M .
We consider the following decomposition:
[Equation]
These terms can be made arbitrarily small by taking large M and s .
As for (48) this property is a direct consequence of Proposition D. Note that Proposition C is not applicable to (46) because this proposition require the speciﬁcation of the target function by k ( s ) ∞ which does not hold in general.
In the following, we treat the remaining terms.
Proposition E. Suppose (A1’) and (A2’) hold.
Then, we have
plim M →∞ (cid:107) k
( s ) M − k ( s ) ∞ (cid:107) L ∞ ( ρ X ) 2 = 0 , lim s →∞ (cid:107) k
( s ) ∞ − k ∞ (cid:107) L ∞ ( ρ X ) 2 = 0 , 2.
plim M →∞ (cid:12)(cid:12)(cid:12) Tr (cid:16) Σ ( s ) M − Σ ( s ) ∞ (cid:17)(cid:12)(cid:12)(cid:12) = 0 , lim s →∞ (cid:12)(cid:12)(cid:12)
Tr (cid:16) Σ ( s ) ∞ − Σ ∞ (cid:17)(cid:12)(cid:12)(cid:12)
[Equation]
where plim denotes the convergence in probability.
Proof.
We show the ﬁrst statement.
By the uniform law of large numbers (Theorem [Equation] 1 in Mohri et al.
, we see the convergence in probability:
[Equation]
+ (1 + γ 2 )
[Equation]
[Table]
[Equation] where the limit is taken with respect to M → ∞ and the notation p −→ denotes the convergence in probability. Next
[Table]
we have the following convergence:
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where for the ﬁrst inequality we used the boundedness of σ, σ (cid:48) , σ ( s ) [Equation] and σ ( s ) (cid:48) on [ − 1 , 1] , for the equality we used the rotationally invariance of the measure µ
[Equation] and the limit is taken with respect to s → ∞ . For the ﬁnal convergence in the above expression we used Assumption (A1’) and boundedness with Lebesgue’s convergence theorem.
In general, for a kernel k and associated integral operator Σ with ρ X [Equation] we have Tr (Σ) =
(cid:82) S d [Equation] 1 k ( X, X )d ρ X .
Hence, (cid:12)(cid:12)(cid:12)
[Equation]
and the second statement holds immediately.
Finally, we show the third statement.
In the same manner as the derivation of inequality (19), we get
[Equation]
We denote F ( s ) M
=
(Σ ( s ) M + λI ) − 1 g ρ and F ( s ) ∞ =
(Σ ( s ) ∞ + λI ) − 1 g ρ .
Noting g ( s ) M,λ = Σ ( s ) M F ( s ) M and g ( s ) ∞ ,λ = Σ ( s ) ∞ F ( s ) ∞ , we get for x ∈ S
d − 1 ,
[Equation]
The both terms in the last expression (49) converge to 0 in probability because of the ﬁrst statement of this proposition and the Bernstein’s inequality (Proposition 3 in Rudi & Rosasco (2017)) to random operators.
This ﬁnishes the proof of the former of the third statement
[Equation]
In the same manner [Equation] we have
[Equation]
[Equation]
The ﬁrst term in the right hand side converges to 0 because of the ﬁrst statement of this proposition. We next show the convergence (cid:107) Σ ∞ − Σ ( s ) ∞ (cid:107) op → 0 as s → ∞ . As seen in the previous section, Σ ∞ and Σ ( s ) ∞ share the same eigenfunctions and every eigenvalue of Σ ( s ) ∞ converges to that of Σ ∞ as s → ∞ . Let { λ ( s ) ∞ ,i } ∞
i
=1 and { λ ∞ ,i } ∞ i =1 be eigenvalues of Σ ( s ) ∞ and Σ ∞ , respectively. For an arbitrary (cid:15) > 0 , we can take i (cid:15) such that (cid:80) ∞ i = i (cid:15) λ ∞ ,i < (cid:15) .
From the convergence λ ( s ) ∞ ,i → λ ∞ ,i and
Tr (cid:16) Σ ( s ) ∞ (cid:17) → Tr (Σ ∞ ) as s → ∞ , we see that for arbitrarily sufﬁciently large s , | λ ( s ) ∞ ,i − λ ∞ ,i | < (cid:15) ( i < i (cid:15) ) and (cid:80) ∞
i = i (cid:15) λ ( s ) ∞ ,i < 2 (cid:15) .
Clearly, for i ≥ i (cid:15) , | λ ( s ) ∞ ,i − λ ∞ ,i | ≤ (cid:80) ∞ i = i (cid:15) ( λ ( s ) ∞ ,i + λ ∞ ,i ) < 3 (cid:15) .
Therefore, we conclude the uniform convergence sup i ∈{ 1 ,..., ∞} | λ ( s ) ∞ ,i − λ ∞ ,i | → 0 as s → ∞ .
This implies (cid:107) Σ ∞ − Σ ( s ) ∞ (cid:107) op → 0 as s → ∞ .
So far, we have shown that (46), (47), and (48) can be made arbitrarily small by taking large s and M depending on λ .
The remaining problem is to show the convergence of (45).
To do so, we establish the counterpart of Theorem B by adapting Theorem A and Proposition B to the current setting.
The counterpart of Theorem B.
In Theorem A, the condition (A3) is required for NTK associated with the smooth activation σ ( s ) and it is not satisﬁed in general.
Note that (A3) is used for bounding (cid:107) g ( s ) M,λ [Equation] L ∞ ( ρ X ) uniformly as seen in the proof of Lemma A. Let us consider the decomposition:
[Equation]
Here, for the second inequality we used (20).
Note that the inequality (20) holds for Σ ∞ because the condition (A3) is supposed for ReLU.
For the last inequality we used Proposition E. Hence,
Theorem A can be applicable and the same convergence in Theorem A holds for σ ( s ) .
For arbitrarily sufﬁciently large s and M with high probability, we have
[Equation]
Next
[Equation]
we adapt Proposition B to the current setting.
By inequality (34) [Equation]
there exists M 0 ∈ Z + such that ∀ M ≥ M 0 , with high probability,
[Equation]
If (Σ ( s ) ∞ + λI ) − 1 (cid:52) 2(Σ ∞ + λI ) − 1 holds, then we have the counterpart of the second inequality in
Proposition B because
[Equation]
where we used the fact that g ρ is contained in H ∞ because of (A3’) .
Note that the ﬁrst inequality in Proposition B is a direct consequence of the second one.
We consider eigenvalues { λ ( s ) ∞ ,i } ∞ i = and { λ ∞ ,i } ∞ i =1 of Σ ( s ) ∞ and Σ ∞ , respectively.
Let i λ be an index such that for ∀
i > i λ , λ ∞ ,i ≤ λ 2 .
Since, every eigenvalue of { λ ( s ) ∞ ,i } ∞ i =1 converges to that of { λ ∞ ,i } ∞ i =1 as s → ∞ , for an arbitrarily sufﬁciently large s , we have | λ ( s ) ∞ ,i − λ ∞ ,i | ≤ λ 2 for ∀
i < i λ , leading to 1 / ( λ + λ ( s ) ∞ ,i ) ≤ 2 / ( λ + λ ∞ ,i ) .
As for the case i ≥ i λ , since 32 λ ≥ λ + λ ∞ ,i , we have 1 / ( λ + λ ( s ) ∞ ,i ) ≤ 1 /λ ≤ 3 / (2( λ + λ ∞ ,i )) .
Combining these, we obtain (Σ ( s ) ∞ + λI ) − 1 (cid:52) 2(Σ ∞ + λI ) − 1 and [Equation]
[Equation]
These are the counterpart of the ﬁrst and second inequalities in Proposition B.
Next, we consider the bound on the degree of freedom in this proposition.
Assume λ ≤ 12 (cid:107) Σ ∞ (cid:107) op .
As seen earlier, an operator Σ ( s ) ∞ converge to Σ ∞ in terms of the operator norm.
Hence, λ ≤ (cid:107) Σ ( s ) ∞ (cid:107) op for an arbitrarily sufﬁciently large s and the bound on the degree of freedom in Proposition B is applicable.
We get
[Equation]
Let us consider upper bounding the right hand side:
[Equation]
On the other hand
[Equation]
by the deﬁnition of i λ ,
[Equation]
Therefore, by inequality (55), the convergence of λ ( s ) ∞ ,i → λ ∞ ,i for i ∈ { 1 , . .
, i λ − 1 } as s → ∞ , and the second statement in Proposition E [Equation] we have
[Equation]
Combining (45)-(48) with (51), (53), (54), and (56), we establish the counterpart of Theorem B. For given (cid:15) , λ , and δ , there exist sufﬁciently large s and M such that with high probability 1 − δ , [Equation]
[Equation]
where α > 0 is a universal constant.
Proof of Corollary 2.
Since conditions (A1’) and (A2’) are special cases of (A1) and (A2) , we can apply Proposition A to Algorithm 1 for the neural network with the smooth approximation σ ( s ) of
ReLU.
Hence, by setting η t = η
=
O
(1) satisfying 4(6 + λ ) η ≤ 1 and λ
= T − β/ (2 rβ +1) where β = 1 + 1 d − 1 , and by applying Tr (cid:0) Σ ∞ (Σ ∞ + λI ) − 1 (cid:1) =
O ( λ − 1 /β )
(Caponnetto & De Vito, 2007) and
[Equation]
because of (cid:107) Σ ∞ (cid:107) op ≤ O (1) , we ﬁnish the proof of Corollary 2.
G A PPLICATION TO B INARY C LASSIFICATION P ROBLEMS
In this paper, we mainly focused on regression problems, but our idea can be applied to other applications.
We brieﬂy discuss its application to binary classiﬁcation problems.
A label space is set to Y = {− 1 , 1 } and a loss function is set to be the squared loss: (cid:96) ( z, y )
= 0 .
5( y − z ) 2 .
The ultimate goal of the binary classiﬁcation problem is to obtain the Bayes classiﬁer that minimizes the expected classiﬁcation error,
[Equation]
over all measurable maps.
It is known that the Bayes classiﬁer is expressed as sgn( g ρ ( X )) , where g ρ is the Bayes rule of L ( g ) =
E ρ [ l ( g ( X ) , Y )]
(see Zhang (2004); Bartlett et al.
Therefore, if g ρ satisﬁes a margin condition, i.e., | g ρ ( x ) | ≥ ∃ τ > 0 on supp( ρ X ) , then this goal is achieved by obtaining an τ/ 2 -accurate solution of g ρ in terms of the uniform norm on supp( ρ X ) .
That is, the required optimization accuracy on (cid:107) g Θ ( T ) − g ρ (cid:107) L ∞ ( ρ X ) to obtain the Bayes classiﬁer depends only on the margin τ unlike regression problems.
Due to this property, averaged stochastic gradient descent in RKHSs can achieve the linear convergence rate demonstrated in Pillaud-Vivien et al.
(2018a).
To leverage this theory to our problem setting, we consider the following decomposition:
[Equation]
The last term (61) can be made arbitrary small by λ → 0 as shown in Pillaud-Vivien et al.
(2018a).
A term (60) can be bounded in the same manner as the third statement of Proposition E, yielding the convergence to 0 as M → ∞ with high probability.
The convergence of (59) was shown in
Pillaud-Vivien et al.
(2018a)
and the convergence of (58) is guaranteed by Proposition A. As a result, we can show the following exponential convergence of the classiﬁcation error R ( g ) for two-layer neural networks with a sufﬁciently small λ as demonstrated in Pillaud-Vivien et al.
(2018a).
[Equation]
In Nitanda & Suzuki (2019), an exponential convergence was shown for the logistic loss (cid:96) ( z, y )
=
log(1 + exp( − yz )) as well.
Proposition A also holds for the logistic loss with an easier proof than the squared loss because of the boundedness of stochastic gradients of the loss.
Hence, their theory is also applicable to the reference ASGD in an RKHS.
In summary, (58), (59), and (61) can be bounded by the above argument.
However, we note that bounding (60) is not obvious and is left for future work.