B ENEFIT OF DEEP LEARNING WITH NON - CONVEX NOISY GRADIENT DESCENT : P ROVABLE EXCESS RISK BOUND AND SUPERIORITY TO KERNEL METHODS
A BSTRACT
Establishing a theoretical analysis that explains why deep learning can outperform shallow learning such as kernel methods is one of the biggest issues in the deep learning literature.
Towards answering this question, we evaluate excess risk of a deep learning estimator trained by a noisy gradient descent with ridge regularization on a mildly overparameterized neural network, and discuss its superiority to a class of linear estimators that includes neural tangent kernel approach, random feature model, other kernel methods, k -NN estimator and so on.
We consider a teacher-student regression model, and eventually show that any linear estimator can be outperformed by deep learning in a sense of the minimax optimal rate especially for a high dimension setting.
The obtained excess bounds are so-called fast learning rate which is faster than O (1 / √ n ) that is obtained by usual Rademacher complexity analysis.
This discrepancy is induced by the non-convex geometry of the model and the noisy gradient descent used for neural network training provably reaches a near global optimal solution even though the loss landscape is highly non-convex.
Although the noisy gradient descent does not employ any explicit or implicit sparsity inducing regularization, it shows a preferable generalization performance that dominates linear estimators.
I NTRODUCTION
In the deep learning theory literature, clarifying the mechanism by which deep learning can outperform shallow approaches has been gathering most attention for a long time.
In particular, it is quite important to show that a tractable algorithm for deep learning can provably achieve a better generalization performance than shallow methods.
Towards that goal, we study the rate of convergence of excess risk of both deep and shallow methods in a setting of a nonparametric regression problem.
One of the difﬁculties to show generalization ability of deep learning with certain optimization methods is that the solution is likely to be stacked in a bad local minimum, which prevents us to show its preferable performances.
Recent studies tackled this problem by considering optimization on overparameterized networks as in neural tangent kernel (NTK) (Jacot et al., 2018; Du et al., 2019a) and mean ﬁeld analysis (Nitanda & Suzuki, 2017; Chizat & Bach, 2018; Rotskoff &
Vanden-Eijnden, 2018; 2019; Mei et al., 2018; 2019), or analyzing the noisy gradient descent such as stochastic gradient Langevin dynamics (SGLD)
(Welling & Teh, 2011; Raginsky et al., 2017;
Erdogdu et al., 2018).
The NTK analysis deals with a relatively large scale initialization so that the model is well approximated by the tangent space at the initial solution, and eventually, all analyses can be reduced to those of kernel methods (Jacot et al., 2018; Du et al., 2019b; Allen-Zhu et al., 2019; Du et al., 2019a; Arora et al., 2019; Cao & Gu, 2019; Zou et al., 2020).
Although this regime is useful to show its global convergence, the obtained estimator looses large advantage of deep learning approaches because the estimation ability is reduced to the corresponding kernel methods.
To overcome this issue, there are several “beyond-kernel” type analyses.
For example, Allen-Zhu & Li (2019; 2020) showed beneﬁt of depth by analyzing ResNet type networks.
Li et al.
(2020) showed global optimality of gradient descent by reducing the optimization problem to a tensor decomposition problem for a speciﬁc regression problem, and showed the “ideal” estimator on a linear model has worse dependency on the input dimensionality.
Bai & Lee (2020) considered a second order Taylor expansion and showed that the sample complexity of deep approaches has better dependency on the input dimensionality than kernel methods.
Chen et al.
(2020) also derived a similar conclusion by considering a hierarchical representation.
The analyses mentioned above actually show some superiority of deep learning, but all of these bounds are essentially Ω(1 / √ n ) where n is the sample size, which is not optimal for regression problems with squared loss (Caponnetto & de Vito, 2007).
The reason why only such a sub-optimal rate is considered is that the target of their analyses is mostly the Rademacher complexity of the set in which estimators exist for bounding the generalization gap.
However, to derive a tight excess risk bound instead of the generalization gap, we need to evaluate so called local
Rademacher complexity (Mendelson, 2002; Bartlett et al., 2005; Koltchinskii, 2006) (see Eq.
(2) for the deﬁnition of excess risk).
Moreover, some of the existing analyses should change the target function class as the sample size n increases, for example, the input dimensionality is increased against the sample size, which makes it difﬁcult to see how the rate of convergence is affected by the choice of estimators.
Another promising approach is the mean ﬁeld analysis.
There are also some work that showed superiority of deep learning against kernel methods.
Ghorbani et al.
(2019) showed that, when the dimensionality d of input is polynomially increasing with respect to n , the kernel methods is outperformed by neural network approaches.
Although the situation of increasing d explains well the modern high dimensional situations, this setting blurs the rate of convergence.
Actually, we can show the superiority of deep learning even in a ﬁxed dimension setting.
There are several studies about approximation abilities of deep and shallow models.
Ghorbani et al.
(2020) showed adaptivity of kernel methods to the intrinsic dimensionality in terms of approximation error and discuss difference between deep and kernel methods.
Yehudai & Shamir (2019) showed that the random feature method requires exponentially large number of nodes against the input dimension to obtain a good approximation for a single neuron target function.
These are only for approximation errors and estimation errors are not compared.
Recently, the superiority of deep learning against kernel methods has been discussed also in the nonparametric statistics literature where the minimax optimality of deep learning in terms of excess risk is shown.
Especially, it is shown that deep learning achieves better rate of convergence than linear estimators in several settings (Schmidt-Hieber, 2020; Suzuki, 2019; Imaizumi & Fukumizu, 2019; Suzuki & Nitanda, 2019; Hayakawa & Suzuki, 2020).
Here, the linear estimators are a general class of estimators that includes kernel ridge regression, k -NN regression and Nadaraya-Watson estimator.
Although these analyses give clear statistical characterization on estimation ability of deep learning, they are not compatible with tractable optimization algorithms.
In this paper, we give a theoretical analysis that uniﬁes these analyses and shows the superiority of a deep learning method trained by a tractable noisy gradient descent algorithm.
We evaluate the excess risks of the deep learning approach and linear estimators in a nonparametric regression setting, and show that the minimax optimal convergence rate of the linear estimators can be dominated by the noisy gradient descent on neural networks.
In our analysis, the model is ﬁxed and no explicit sparse regularization is employed.
Our contributions can be summarized as follows:
• A reﬁned analysis of excess risks for a ﬁxed model with a ﬁxed input dimension is given to compare deep and shallow estimators.
Although several studies pointed out the curse of dimensionality is a key factor that separates shallow and deep approaches, we point out that such a separation appears in a rather low dimensional setting, and more importantly, the non-convexity of the model essentially makes the two regimes different.
• A lower bound of the excess risk which is valid for any linear estimator is derived.
The analysis is considerably general because the class of linear estimators includes kernel ridge regression with any kernel and thus it also includes estimators in the NTK regime.
• All derived convergence rate is a fast learning rate that is faster than O (1 / √ n ) .
We show that simple noisy gradient descent on a sufﬁciently wide two-layer neural network achieves a fast
learning rate by using a fact that the solution converges to a Bayes estimator with a Gaussian process prior, and the derived convergence rate can be faster than that of linear estimators.
This is much different from such existing work that compared only coefﬁcients with the same rate of convergence with respect to the sample size n .
Other related work Bach (2017) analyzed the model capacity of neural networks and its corresponding reproducing kernel Hilbert space (RKHS), and showed that the RKHS is much larger than the neural network model.
However, separation of the estimation abilities between shallow and deep is not proven.
Moreover, the analyzed algorithm is basically the Frank-Wolfe type method which is not typically used in practical deep learning.
The same technique is also employed by Barron (1993).
The Frank-Wolfe algorithm is a kind of sparsity inducing algorithm that is effective for estimating a function in a model with an L 1 -norm constraint.
It has been shown that explicit or implicit sparse regularization such as L 1 -regularization is beneﬁcial to obtain better performances of deep learning under certain situations (Chizat & Bach, 2020; Chizat, 2019; Gunasekar et al., 2018;
Woodworth et al., 2020; Klusowski & Barron, 2016).
For example, E et al.
(2019b;a) showed that the approximation error of a linear model suffers from the curse of dimensionality in a setting where the target function is in the Barron class (Barron, 1993), and showed an L 1 -type regularization avoids the curse of dimensionality.
However, our analysis goes in a different direction where a sparse regularization is not required.
P ROBLEM SETTING AND MODEL
In this section, we give the problem setting and notations that will be used in the theoretical analysis.
We consider the standard nonparametric regression problem where data are generated from the following model for an unknown true function f
o : R d → R :
[Equation]
where x i is independently identically distributed from P X whose support is included in Ω = [0 , 1] d , and (cid:15)
i is an observation noise that is independent of x i and satisﬁes E[ (cid:15) i ]
= 0 and (cid:15) i ∈
[ − U, U ] almost surely.
The n i.i.d.
observations are denoted by D n = ( x i , y i ) ni
=1 .
We want to estimate the true function f
o through the training data D n .
To achieve this purpose, we employ the squared loss (cid:96) ( y, f ( x ))
= ( y − f ( x )) 2 and accordingly we deﬁne the expected and empirical risks as L ( f ) :=
E Y,X
[ (cid:96) ( Y, f ( X ))] and (cid:98) L ( f ) :
= 1 n (cid:80) ni
=1 (cid:96) ( y i , f ( x i )) respectively.
Throughout this paper, we are interested in the excess (expected) risk of an estimator (cid:98) f deﬁned by
[Equation]
Since the loss function (cid:96) is the squared loss, the inﬁmum of inf f : measurable L ( f ) is achieved by f o : inf f : measurable L ( f )
=
L ( f o ) .
The population L 2 ( P X ) -norm is denoted by (cid:107) f (cid:107) L 2 ( P X ) :
= (cid:112)
E X ∼ P X
[ f ( X ) 2 ] and the sup-norm on the support of P X is denoted by (cid:107) f (cid:107) ∞ :
= sup x ∈ supp( P X ) | f ( x ) | .
We can easily check that for an estimator (cid:98) f , the L 2 -distance (cid:107) (cid:98) f − f o (cid:107) 2 L 2 ( P X ) between the estimator (cid:98) f and the true function f o is identical to the excess risk:
L ( (cid:98) f ) − L ( f o )
= (cid:107) (cid:98) f − f o (cid:107) 2 L 2 ( P X ) .
Note that the excess risk is different from the generalization gap
L ( (cid:98) f ) − (cid:98) L ( (cid:98) f ) .
Indeed, the generalization gap typically converges with the rate of O (1 / √ n ) which is optimal in a typical setting (Mohri et al., 2012).
On the other hand, the excess risk can be faster than O (1 / √ n ) , which is known as a fast learning rate (Mendelson, 2002; Bartlett et al., 2005; Koltchinskii, 2006;
Gin´e & Koltchinskii, 2006).
2. M ODEL OF TRUE FUNCTIONS
To analyze the excess risk, we need to specify a function class (in other words, model) in which the true function f
o is included.
In this paper, we only consider a two layer neural network model, whereas the techniques adapted in this paper can be directly extended to deeper neural network models.
We consider a teacher-student setting, that is, the true function f
o can be represented by a neural network deﬁned as follows.
For w ∈ R , let ¯ w be a “clipping” of w deﬁned as ¯ w :=
R × tanh( w/R ) where R ≥ 1 is a ﬁxed constant, and let [ x ; 1] :=
[ x (cid:62) , 1] (cid:62) for x ∈ R d .
Then, the teacher network is given by
[Equation]
where w 1 ,m ∈ R d
+1 and w 2 ,m ∈ R ( m ∈ N ) are the trainable parameters (where W = ( w 1 ,m , w 2 ,m ) ∞ m =1 ), a m ∈ R ( m ∈ N ) is a ﬁxed scaling parameter, and σ m :
R → R is an activation function for the m -th node.
The reason why we applied the clipping operation to the parameter of the second layer is just for a technical reason to ensure convergence of Langevin dynamics.
The dynamics is bounded in high probability in practical situations and the boundedness condition would be removed if further theoretical development of inﬁnite dimensional Langevin dynamics would be achieved.
Let H be a set of parameters W such that its squared norm is bounded: H :=
{ W = ( w 1 ,m , w 2 ,m ) ∞ m =1 | (cid:80) ∞ m =1 ( (cid:107) w 1 ,m (cid:107) 2 + w 22 ,m ) < ∞} .
Deﬁne (cid:107) W (cid:107) H :=
[ (cid:80) ∞ m =1 ( (cid:107) w 1 ,m (cid:107) 2 + w 22 ,m )] 1 / 2 for W ∈ H .
Let ( µ m ) ∞ m =1 be a regularization parameter such that µ m (cid:38) 0 .
Accordingly we deﬁne H γ :=
{ W ∈ H | (cid:107) W (cid:107)
H γ < ∞} where (cid:107) W (cid:107) H γ :=
[ (cid:80) ∞ m =1 µ − γm ( (cid:107) w 1 ,m (cid:107) 2 + w 22 ,m )] 1 /for a given 0 < γ .
Throughout this paper, we analyze an estimation problem in which the true function is included in the following model:
[Equation]
This is basically two layer neural network with inﬁnite width.
As assumed later, a m is assumed to decrease as m → ∞ .
Its decreasing rate controls the capacity of the model.
If the ﬁrst layer parameters ( w 1 ,m ) m are ﬁxed, this model can be regarded as a variant of the unit ball of some reproducing kernel Hilbert space (RKHS) with basis functions a m σ m ( w (cid:62) 1 ,m [ x ; 1]) .
However, since the ﬁrst layer ( w 1 ,m ) is also trainable, there appears signiﬁcant difference between deep and kernel approaches.
The Barron class (Barron, 1993; E et al., 2019b) is relevant to this function class.
Indeed, it is deﬁned as the convex hull of w 2 σ ( w (cid:62)
[ x ; 1]) with norm constraints on ( w 1 , w 2 ) where σ is an activation function.
On the other hand, we will put an explicit decay rate on a m and the parameter W has an L 2 -norm constraint, which makes the model F γ smaller than the Barron class.
E STIMATORS
We consider two classes of estimators and discuss their differences: linear estimators and deep learning estimator with noisy gradient descent (NGD).
Linear estimator A class of linear estimators, which we consider as a representative of “shallow” learning approach, consists of all estimators that have the following form:
[Equation]
Here, ( ϕ i ) ni =1 can be any measurable function (and L 2 ( P X ) -integrable
so that the excess risk can be deﬁned).
Thus, they could be selected as the “optimal” one so that the corresponding linear estimator minimizes the worst case excess risk.
Even if we chose such an optimal one, the worst case excess risk should be lower bounded by our lower bound given in Theorem 1.
It should be noted that the linear estimator does not necessarily imply “linear model.”
The most relevant linear estimator in the machine learning literature is the kernel ridge regression: (cid:98) f
( x ) =
Y (cid:62) ( K X + λ I) − 1 k ( x ) where K X
=
( k ( x i , x j ))
ni,j =1 ∈ R n × n , k ( x ) =
[ k ( x, x 1 ) , . .
k ( x, x n )] (cid:62)
∈ R n and Y =
[ y 1 , . . .
, y n ] (cid:62) ∈ R n for a kernel function k : R d × R d → R .
Therefore, the ridge regression estimator in the NTK regime or the random feature model is also included in the class of linear estimators.
The solution obtained in the early stopping criteria instead of regularization in the NTK regime under the squared loss is also included in the linear estimators.
Other examples include the k -NN estimator and the Nadaraya-Watson estimator.
All of them do not train the basis function in a nonlinear way, which makes difference from the deep learning approach.
In the nonparametric statistics literature, linear estimators have been studied for estimating a wavelet series model.
Donoho et al.
(1990; 1996) have shown that a wavelet shrinkage estimator can outperform any linear estimator by showing suboptimality of linear estimators.
Suzuki (2019) utilized such an argument to show superiority of deep learning but did not present any tractable optimization algorithm.
Noisy Gradient Descent with regularization As for the neural network approach, we consider a noisy gradient descent algorithm.
Basically, we minimize the following regularized empirical risk:
[Equation]
Here, we employ H 1 -norm as the regularizer.
We note that the constant γ controls the relative complexity of the true function f o compared to the typical solution obtained under the regularization.
Here, we deﬁne a linear operator A as λ (cid:107) W (cid:107) H 1 = W (cid:62) AW , that is, AW =
( λµ − 1 m w 1 ,m , λµ − 1 m w 2 ,m
) ∞ m =1 .
The regularized empirical risk can be minimized by noisy gradient descent as W k +
=
W k − η ∇ ( (cid:98) L ( f W k )
+ λ 2 (cid:107) W k (cid:107) 2 H 1 )
+ (cid:113) 2 ηβ ξ k , where η > 0 is a step size and ξ k = ( ξ k, (1 ,m ) , ξ k, (2 ,m ) )
∞ m =1 is an inﬁnite-dimensional Gaus- sian noise, i.e., ξ k, (1 ,m ) and ξ k, (2 ,m ) are independently identically distributed from the standard normal distribution (Da Prato & Zabczyk, 1996).
Here, ∇ (cid:98) L ( f W ) = 1 n (cid:80) ni
=1 2( f W ( x i ) − y i )( ¯ w 2 ,m a m [ x i ; 1] σ (cid:48) m ( w (cid:62) 1 ,m [ x i ; 1]) , a m tanh (cid:48) ( w 2 ,m /R )
σ m ( w (cid:62) 1 ,m [ x i ; 1]))
∞ m =1 .
However, since ∇(cid:107)
W k − 1 (cid:107) 2 H 1 is unbounded which makes it difﬁcult to show convergence, we employ the semiimplicit Euler scheme deﬁned by
[Equation]
where S η :=
(I+ ηA ) − 1 .
It is easy to check that this is equivalent to the following update rule: W k =
W k − 1 − η (cid:16) ∇ (cid:98) L ( f W k − 1 ) + S
η AW k − 1 + (cid:113) 2 ηβ ξ k − 1 (cid:17) .
Therefore, the implicit Euler scheme can be seen as a naive noisy gradient descent for minimizing the empirical risk with a slightly modiﬁed ridge regularization.
This can be interpreted as a discrete time approximation of the following inﬁnite dimensional Langevin dynamics :
[Equation]
where ( ξ t ) t ≥ 0 is the so-called cylindrical Brownian motion (see Da Prato & Zabczyk (1996) for the details).
Its application and analysis for machine learning problems with non-convex objectives have been recently studied by, for example, Muzellec et al.
Suzuki (2020).
The above mentioned algorithm is executed on an inﬁnite dimensional parameter space.
In practice, we should deal with a ﬁnite width network.
To do so, we approximate the solution by a ﬁnite dimensional one: W ( M ) =
( w 1 ,m , w 2 ,m )
Mm =1 where M corresponds to the width of the network.
We identify W ( M ) to the “zero-padded” inﬁnite dimensional one, W = ( w 1 ,m , w 2 ,m ) ∞ m =1 with w 1 ,m = 0 and w 2 ,m = 0 for all m > M .
Accordingly, we use the same notation f W ( M ) to indicate f W with zero padded vector W .
Then, the ﬁnite dimensional version of the update rule is given by
W ( M ) k +1 =
S ( M ) η (cid:16) W ( M ) k − η ∇ (cid:98) L ( f W ( M ) k )
+ (cid:113) 2 ηβ ξ ( M ) k (cid:17) , where ξ ( M ) k is the Gaussian noise vector obtained by projecting ξ k to the ﬁrst M components and S ( M ) η is also obtained in a similar way.
C ONVERGENCE RATE OF ESTIMATORS
In this section, we present the excess risk bounds for linear estimators and the deep learning estimator.
As for the linear estimators, we give its lower bound while we give an upper bound for the deep learning approach.
To obtain the result, we setup some assumptions on the model.
Assumption 1.
(i)
There exists a constant c µ such that µ m ≤ c µ m − 2 ( m ∈ N ) .
(ii) There exists α 1 > 1 / 2 such that a m ≤ µ α 1 m ( m ∈ N ) .
(iii)
The activation functions ( σ m ) m is bounded as (cid:107) σ m (cid:107) ∞ ≤ 1 .
Moreover, they are three times differentiable and their derivatives upto third order differentiation are uniformly bounded: ∃ C σ such that (cid:107) σ m (cid:107) 1 , 3 := max {(cid:107) σ (cid:48) m (cid:107) ∞ , (cid:107) σ (cid:48)(cid:48) m (cid:107) ∞ , (cid:107) σ (cid:48)(cid:48)(cid:48) m (cid:107) ∞ } ≤ C σ ( ∀ m ∈ N ) .
The ﬁrst assumption (i) controls the strength of the regularization, and combined with the second assumption (ii) and deﬁnition of the model F γ , complexity of the model is controlled.
If α 1 and γ are large, the model is less complicated.
Indeed, the convergence rate of the excess risk becomes faster if these parameters are large as seen later.
The decay rate µ m ≤ c µ m − 2 can be generalized as m − p with p >
but we employ this setting just for a technical simplicity for ensuring convergence of the Langevin dynamics.
The third assumption (iii) can be satisﬁed by several activation functions such as the sigmoid function and the hyperbolic tangent.
The assumption (cid:107) σ m (cid:107) ∞ ≤ 1 could be replaced by another one like (cid:107) σ m (cid:107) ∞ ≤ C , but we ﬁx this scaling for simple presentation.
4. M INIMAX LOWER BOUND FOR LINEAR ESTIMATORS
Here, we analyze a lower bound of excess risk of linear estimators, and eventually we show that any linear estimator suffers from curse of dimensionality.
To rigorously show that, we consider the following minimax excess risk over the class of linear estimators:
[Equation]
where inf is taken over all linear estimators and E D n [ · ] is taken with respect to the training data D n .
This expresses the best achievable worst case error over the class of linear estimators to estimate a function in F γ .
To evaluate it, we additionally assume the following condition.
Assumption 2.
We assume that µ m = m − 2 and a m = µ α 1 m ( m ∈ N ) (and hence c µ = 1 ).
There exists a monotonically decreasing sequence ( b m ) ∞ m =1 and s ≥ 3 such that b m = µ α 2 m ( ∀ m ) with α 2 > γ/ 2 and σ m ( u ) =
b sm σ ( b − 1 m u ) ( u ∈ R ) where σ is the sigmoid function: σ ( u )
=
1 / (1+ e − u ) .
Intuitively, the parameter s controls the “resolution” of each basis function σ m , and the relation between parameter α 1 and α 2 controls the magnitude of coefﬁcient for each basis σ m .
Note that the condition s ≥ 3 ensures (cid:107) σ m (cid:107) 1 , 3 is uniformly bounded and 0 < b m ≤ 1 ensures (cid:107) σ m (cid:107) ∞ ≤ 1 .
Our main strategy to obtain the lower bound is to make use of the so-called convex-hull argument .
That is, it is known that, for a function class F , the minimax risk R ( F ) over a class of linear estimators is identical to that for the convex hull of F (Hayakawa & Suzuki, 2020;
Donoho et al., 1990):
[Equation]
where conv( F ) = { (cid:80) Ni =
λ
i f i | f i ∈ F , (cid:80) Ni =
λ i = 1 , λ i ≥ 0 , N ∈ N } and conv( · ) is the closure of conv( · ) with respect to L 2 ( P X ) -norm.
Intuitively, since the linear estimator is linear to the observations ( y i ) ni
=1 of outputs, a simple application of Jensen’s inequality yields that its worst case error on the convex hull of the function class F does not increase compared with that on the original one F (see Hayakawa & Suzuki (2020) for the details).
This indicates that the linear estimators cannot distinguish the original hypothesis class F and its convex hull.
Therefore, if the class F is highly non-convex, then the linear estimators suffer from much slower convergence rate because its convex hull conv( F ) becomes much “fatter” than the original one F .
To make use of this argument, for each sample size n , we pick up appropriate m n and consider a subset generated by the basis function σ m n , i.e., F ( n ) γ [Equation] [Equation] ]) ∈
[Equation]
γ } . By applying the convex hull argument to this set, we obtain the relation R lin ( F γ ) ≥ R lin ( F ( n ) γ )
=
R lin ( conv( F ( n ) γ )) .
Since F ( n ) γ is highly non-convex, its convex hull conv( F ( n ) γ ) is much larger than the original set
F ( n ) γ and thus the minimax risk over the linear estimators would be much larger than that over all estimators including deep learning.
More intuitively, linear estimators do not adaptively select the basis functions and thus they should prepare redundantly large class of basis functions to approximate functions in the target function class.
The following theorem gives the lower bound of the minimax optimal excess risk over the class of linear estimators.
Theorem 1.
Suppose that Var( (cid:15) ) > 0 , P X is the uniform distribution on [0 , 1] d , and Assumptionis satisﬁed.
Let ˜ β
= α 1 +( s +1)
α 2 α 2 − γ/ 2 .
Then for arbitrary small κ (cid:48) > 0 , we have that
[Equation]
The proof is in Appendix A. We utilized the Irie-Miyake integral representation (Irie & Miyake, 1988;
Hornik et al., 1990) to show there exists a “complicated” function in the convex hull, and then we adopted the technique of Zhang et al.
(2002) to show the lower bound.
The lower bound is characterized by the decaying rate ( α 1 ) of a m relative to that ( α 2 ) of the scaling factor b m .
Indeed, the faster a m decays with increasing m , the faster the rate of the minimax lower bound becomes.
We can see that the minimax rate of linear estimators is quite sensitive to the dimension d .
Actually, for relatively high dimensional settings, this lower bound becomes close to a slow rate Ω(1 / √ n ) , which corresponds to the curse of dimensionality.
It has been pointed out that the sample complexity of kernel methods suffers from the curse of dimensionality while deep learning can avoid that by a tractable algorithms (e.g., Ghorbani et al.
; Bach (2017)).
Among them, Ghorbani et al.
(2019) showed that if the dimensionality d is polynomial against n , then the excess risk of kernel methods is bounded away from 0 for all n .
On the other hand, our analysis can be applied to any linear estimator including kernel methods, and it shows that even if the dimensionality d is ﬁxed, the convergence rate of their excess risk suffers from the curse of dimensionality.
This can be accomplished thanks to a careful analysis of the rate of convergence.
Bach (2017) derived an upper bound of the Rademacher complexity of the unit ball of the RKHS corresponding to a neural network model.
However, it is just an upper bound and there is still a large gap from excess risk estimates.
Allen-Zhu & Li (2019; 2020); Bai & Lee (2020); Chen et al.
(2020) also analyzed a lower bound of sample complexity of kernel methods.
However, their lower bound is not for the excess risk of the squared loss.
Eventually, the sample complexities of all methods including deep learning take a form of O ( C/ √ n ) and dependency of coefﬁcient C to the dimensionality or other factors such as magnitude of residual components is compared.
On the other hand, our lower bound properly involves the properties of squared loss such as strong convexity and smoothness and the bound shows the curse of dimensionality occurs even in the rate of convergence instead of just the coefﬁcient.
Finally, we would like to point out that several existing work (e.g.,
Ghorbani et al.
; Allen-Zhu & Li (2019)) considered a situation where the target function class changes as the sample size n increases.
However, our analysis reveals that separation between deep and shallow occurs even if the target function class F γ is ﬁxed.
4. U PPER BOUND FOR DEEP LEARNING
Here, we analyze the excess risk of deep learning trained by NGD and its algorithmic convergence rate.
Our analysis heavily relies on the weak convergence of the discrete time gradient Langevin dynamics to the stationary distribution of the continuous time one (Eq.
Under some assumptions, the continuous time dynamics has a stationary distribution (Da Prato & Zabczyk, 1992; Maslowski, 1989; Sowers, 1992; Jacquot & Royer, 1995; Shardlow, 1999; Hairer, 2002).
If we denote the probability measure on H corresponding to the stationary distribution by π ∞ , then it is given by
[Equation]
where ν β is the Gaussian measure in H with mean 0 and covariance ( βA ) − 1 (see Da Prato &
Zabczyk (1996) for the rigorous deﬁnition of the Gaussian measure on a Hilbert space).
Remarkably, this can be seen as the Bayes posterior for a prior distribution ν β and a “log-likelihood” function exp( − β (cid:98) L ( W )) .
Through this view point, we can obtain an excess risk bound of the solution W k .
The proofs of all theorems in this section are in Appendix B.
Under Assumption 1, the distribution of W k derived by the discrete time gradient Langevin synamics satisﬁes the following weak convergence property to the stationary distribution π ∞ .
This convergence rate analysis depends on the techniques by Br´ehier & Kopec (2016); Muzellec et al.
Proposition 1.
Assume Assumption 1 holds and β > η .
Then, there exist spectral gaps Λ ∗ η and Λ ∗ 0 (deﬁned in Eq.
(10) of Appendix B.1) and a constant C 0 such that, for any 0 < a < 1 / 4 , the following convergence bound holds for almost sure observation D n :
[Equation]
where C 1 is a constant depending only on c µ , R, α 1 , C σ , U, a (independent of η, k, β, λ, n ).
This proposition indicates that the expected risk of W k can be almost identical to that of the “Bayes posterior solution” obeying π ∞ after sufﬁciently large iterations k with sufﬁciently small step size η even though (cid:98) L ( f W ) is not convex.
The deﬁnition of Λ ∗ η can be found in Eq.
We should note that its dependency on β is exponential.
Thus, if we take β = Ω( n ) , then the computational cost until a sufﬁciently small error could be exponential with respect to the sample size n .
The same convergence holds also for ﬁnite dimensional one W ( M ) k with a modiﬁed stationary distribution.
The constants appearing in the bound are independent of the model size M (see the proof of Proposition 1 in Appendix B).
In particular, the convergence can be guaranteed even if W is inﬁnite dimensional.
This is quite different from usual ﬁnite dimensional analyses (Raginsky et al., 2017; Erdogdu et al., 2018; Xu et al., 2018) which requires exponential dependency on the dimension, but thanks to the regularization term, we can obtain the model size independent convergence rate.
Xu et al.
(2018) also analyzed a ﬁnite dimensional gradient Langevin dynamics and obtained a similar bound where
O ( η ) appears in place of the second term η 1 / 2 − a which corresponds to time discretization error.
In our setting the regularization term is (cid:107) W (cid:107) 2 H 1 = (cid:80) m
( (cid:107) w 1 ,m (cid:107) 2 + w 22 ,m ) /µ
m with µ m (cid:46) m − 2 , but if we employ (cid:107) W (cid:107) 2 H p/ 2 = (cid:80) m ( (cid:107) w 1 ,m (cid:107) 2 + w 22 ,m ) /µ
p/ 2 m for p > 1 , then the time discretization error term would be modiﬁed to η ( p − 1) /p − a
(Andersson et al., 2016).
We can interpret the ﬁnite dimensional setting as the limit of p → ∞ which leads to η ( p − 1) /p
→ η that recovers the ﬁnite dimensional result ( O ( η ) ) as shown by Xu et al.
In addition to the above algorithmic convergence, we also have the following convergence rate for the excess risk bound of the ﬁnite dimensional solution W ( M ) k .
Theorem 2.
Assume Assumption 1 holds, assume η < β ≤ min { n/ (2 U 2 ) , n } , and
0 < γ < 1 / 2 + α 1 .
Then, if the width satisﬁes M ≥ min (cid:8) λ 1 / 4 γ ( α 1 +1) β 1 / 2 γ , λ − 1 / 2( α 1 +1) , n 1 / 2 γ (cid:9) , the expected excess risk of W k is bounded as
[Equation]
where C is a constant independent of n, β, λ, η, k .
In particular, if we set β = min { n/ (2 U 2 ) , n } and λ = β − 1 , then for M ≥ n 1 / 2( α 1 +1) , we obtain
[Equation]
In addition to this theorem, if we further assume Assumption 2, we obtain a reﬁned bound as follows.
Corollary 1.
Assume Assumptions 1 and 2 hold and η < β , and let β = min { n/ (2 U 2 ) , n } and λ =
β − 1 .
Suppose that there exists 0 ≤ q ≤ s − 3 such that 0 < γ < 1 / 2 + α 1 + qα 2 .
Then, the excess risk bound of W ( M ) k for M ≥ n 1 / 2( α 1 + qα 2 +1) can be reﬁned as
[Equation]
These theorem and corollary shows that the tractable NGD algorithm achieves a fast convergence rate of the excess risk bound.
Indeed, if q is chosen so that γ > ( α 1 + qα 2 +1) / 2 , then the excess risk bound converges faster than O (1 / √ n ) .
Remarkably, the convergence rate is not affected by the input dimension d , which makes discrepancy from linear estimators.
The bound of Theorem 2 is tightest when γ is close to 1 / 2 + α 1 ( γ ≈ 1 /
+ α 1 + 3 α 2 for Corollary 1), and a smaller γ yields looser bound.
This relation between γ and α 1 reﬂects misspeciﬁcation of the “prior” distribution.
When γ is small, the regularization λ (cid:107) W (cid:107) 2 H 1 is not strong enough so that the variance of the posterior distribution becomes unnecessarily large for estimating the true function f o ∈ F γ .
Therefore, the best achievable bound can be obtained when the regularization is correctly speciﬁed.
The analysis of fast rate is in contrast to some existing work (Allen-Zhu & Li, 2019; 2020; Li et al., 2020; Bai & Lee, 2020) that basically evaluated the Rademacher complexity.
This is because we essentially evaluated a local Rademacher complexity instead.
4. C OMPARISON BETWEEN LINEAR ESTIMATORS AND DEEP LEARNING
Here, we compare the convergence rate of excess risks between the linear estimators and the neural network method trained by NGD using the bounds obtained in Theorem 1 and Corollary 1 respectively.
We write the lower bound (5) of the minimax excess risk of linear estimators as R ∗ lin and the excess risk of the neural network approach (7) as R ∗ NN .
To make the discussion concise, we consider a speciﬁc situation where s = 3 , α 1 = γ = 14 α 2 .
In this case, ˜ β
= 17 / 3 ≈ 5 . 667 , which gives
[Equation]
On the other hand, by setting q = 0 , we have
[Equation]
Thus, as long as α 1 > 11 . 3 /d + 1 ≈ 2˜ β/d
+ 1 , we have that
[Equation]
In particular, as d gets larger, R ∗ lin approaches to Ω( n − 1 / 2 ) while R ∗ NN is not affected by d and it gets close to O ( n − 1 ) as α 1 gets larger.
Moreover, the inequality α 1 > 11 .
3 /d + 1 can be satisﬁed by a relatively low dimensional setting; for example, d
= 10 is sufﬁcient when α 1 = 3 .
As α 1 becomes large, the model becomes “simpler” because ( a m ) ∞ m =1 decays faster.
However, the linear estimators cannot take advantage of this information whereas deep learning can.
From the convex hull argument, this discrepancy stems from the non-convexity of the model.
We also note that the superiority of deep learning is shown without sparse regularization while several existing work showed favorable estimation property of deep learning though sparsity inducing regularization (Bach, 2017; Chizat, 2019; Hayakawa & Suzuki, 2020).
However, our analysis indicates that sparse regularization is not necessarily as long as the model has non-convex geometry, i.e., sparsity is just one sufﬁcient condition for non-convexity but not a necessarily condition.
The parameter setting above is just a sufﬁcient condition and the lower bound R ∗ lin would not be tight.
The superiority of deep learning would hold in much wider situations.
C ONCLUSION
In this paper, we studied excess risks of linear estimators, as a representative of shallow methods, and a neural network estimator trained by a noisy gradient descent where the model is ﬁxed and no sparsity inducing regularization is imposed.
Our analysis revealed that deep learning can outperform any linear estimator even for a relatively low dimensional setting.
Essentially, non-convexity of the model induces this difference and the curse of dimensionality for linear estimators is a consequence of a fact that the geometry of the model becomes more “non-convex” as the dimension of input gets higher.
All derived bounds are fast rate because the analyses are about the excess risk with the squared loss, which made it possible to compare the rate of convergence.
The fast learning rate of the deep learning approach is derived through the fact that the noisy gradient descent behaves like a
Bayes estimator with model size independent convergence rate.
A CKNOWLEDGMENTS
TS was partially supported by JSPS Kakenhi (18K19793, 18H and 20H00576), Japan Digital
Design and JST-CREST.
A P ROOF OF T HEOREM
We basically combine the “convex hull argument” and the minimax optimal rate analysis for linear estimators developed by Zhang et al.
Zhang et al.
(2002) essentially showed the following statement in their Theorem 1.
Proposition 2 (Theorem 1 of Zhang et al.
Let µ be the Lebesgue measure.
Suppose that the space Ω has even partition A such that |A| = 2 K for an integer K ∈ N , each A has equivalent measure µ ( A ) =
2 − K for all A ∈ A , and A is indeed a partition of Ω , i.e., ∪ A ∈A =
Ω , A ∩ A (cid:48)
= ∅ for A,
A (cid:48) ∈ Ω and A (cid:54) =
A (cid:48) .
Then, if K is chosen as n − γ 1 ≤ 2 − K ≤ n − γ 2 for constants γ 1 , γ 2 >that are independent of n , then there exists an event E such that, for a constant C (cid:48) > 0 ,
P ( E ) ≥
+ o (1) and |{ x i | x i ∈ A ( i ∈ { 1 , . . . , n } ) }| ≤ C (cid:48)
n/ 2 K ( ∀ A ∈ A ) .
Moreover, suppose that, for a class F ◦ of functions on Ω , there exists ∆ > 0 that satisﬁes the following conditions:
There exists F > 0 such that, for any A ∈ A , there exists g ∈ F ◦ that satisﬁes g ( x ) ≥ 12 ∆ F for all x ∈ A , 2.
There exists K (cid:48) and C (cid:48)(cid:48) >
such that 1 n (cid:80) ni
=1 g ( x i ) 2 ≤ C (cid:48)(cid:48)
∆ 2 2 − K (cid:48) for any g ∈ F ◦ on the event E .
Then, there exists a constant F 1 such that at least one of the following inequalities holds:
[Equation]
for sufﬁciently large n .
Before we show the main assertion, we prepare some additional lemmas.
For a sigmoid function σ , let ˜ F ( σ ) C,τ := { x ∈ R d (cid:55)→ aσ ( τ ( w (cid:62) x + b )))
| | a | ≤ 2 C, (cid:107) w (cid:107) ≤ 1 , | b | ≤ 2 ( a, b ∈ R , w ∈ R d ) } for
C > 0 , τ > 0 .
Lemma 1.
Let ψ ( x ) = 12 ( σ ( x + 1) − σ ( x − 1)) and ˆ ψ be its Fourier transform: ˆ ψ ( ω ) :
= (2 π ) − 1 (cid:82) e − i ωx ψ ( x )d x .
Let h > 0 and D w > 0 .
Then, by setting τ = h − 1 (2 √ d + 1) D w and
C = (2 √ d +1)
D w πh
| ˆ ψ (1) | , the Gaussian RBF kernel can be approximated by
[Equation]
for any c ∈ [0 , 1] d , where C d is a constant depending only on d .
In particular, the right hand side is
O (exp( − n κ ))
if D w = n κ .
Proof of Lemma 1.
Let ψ h ( x ) =
ψ ( h − 1 x ) .
Suppose that, for f ∈ L 1 ( R d ) , its Fourier transform ˆ f ( ω ) =
(2 π ) − d (cid:82) e − i ω (cid:62) x f ( x )d x ( ω ∈ R d ) gives
[Equation]
for every x ∈ R d 1 .
Then the Irie-Miyake itegral representation (Irie & Miyake (1988); see also the proof of Theorem 3.1 in Hornik et al.
(1990)) gives
[Equation]
where d ν ( a, b ) is given by
[Equation]
[Equation]
Re
[Equation]
for any ω (cid:54) = 0 . Since the characteristic function of the multivariate normal distribution gives that
[Equation]
we have that
[Equation]
for all x ∈ R d . Since ψ h ( · ) = ψ ( h − 1 · ) and ˆ ψ h ( · ) = h ˆ ψ ( h · ) by its deﬁnition, the right hand side is equivalent to
[Equation]
e
[Equation]
[Equation]
d a d b.
Here, we set ω
= h − 1 .
Let N σ 2 be the probability measure corresponding to the multivariate normal with mean 0 and covariance σ 2 I , and let A D := { w ∈ R d | (cid:107) w (cid:107) ≤ D } .
Let D a > 0 and
D b = D a ( √ 2 d + 1) , and deﬁne
[Equation]
(cid:19)
d a d b. Then, we can see that, for any x ∈ [0 , 1] d , it holds that
[Equation]
where C d > 0 is a constant depending on only d , and we used | a (cid:62) ( x − c )
+ b | ≥ | b | − | a (cid:62) ( x − c ) | ≥ | b | − 2 √ dD a and ψ ( x ) ≤ 2 exp( −| x | ) .
Note that if D a = n κ , then the right hand side is
O ( h exp( − n κ )) .
Therefore, since N 1 ( A D a ) ≤ 1 , by setting τ = h − 1 D b , C =
D b πh | ˆ ψ (1) | , we have that
[Equation]
Hence, by rewriting D w ← D a , we obtain the assertion.
As noted above, the right hand is
O (exp( − n κ ))
if D a = n κ .
Proof of Theorem 1.
For a sample size n , we ﬁx m n which will be determined later and use
Proposition 2 with F ◦ = F ( n ) γ .
If w 2 ,m n = b (cid:112) µ γm n / 2 with | b | ≤ 1 and w 1 ,m = µ γ/ 2 m n [ u ; − u (cid:62) c ] / ( (cid:112) 2( d + 1)) for u ∈ R d such that (cid:107) u (cid:107) ≤ 1 and c ∈ [0 , 1] d , then (cid:107) ( w 1 ,m n , w 2 ,m n ) (cid:107) 2 ≤ µ γm n (1 / 2 + (1 + | u (cid:62) c | 2 ) /
2( d + 1)) ≤ µ γm n .
Therefore, ˜ ϕ u,c ( x ) =
a m n ¯ w 2 ,m n σ m n ( w (cid:62) 1 ,m n [ x ; 1]) =
µ α 1 m n ( bµ γ/ 2 m n / √ 2) µ sα 2 m n σ (cid:16) µ − α
+ γ/ 2 m n u (cid:62) ( x − c ) / (cid:112) {+
+}
( d + 1) (cid:17) ∈
F ( n ) γ ⊂ F γ for all b ∈ R with | b | ≤ 1 , u ∈ R d with (cid:107) u (cid:107) ≤ 1 , and c ∈ [0 , 1] d .
In other words, µ α 1 + γ/ 2+ sα 2 m n {+(2 C ) − 1 F ( σ ) C,τ ⊂ F ( n ) γ for any C > 0 and τ = 1 √ 2( d +1) µ − α 
+}
+ γ/ 2 m n . Therefore, by setting C = ( √ 2 d + 1) D w / ( πh | ˆ ψ (1) | ) for D w > 0 , Lemma 1 yields that for any c ∈ [0 , 1] d and given h > 0 , there exists g ∈ conv( F ( n ) γ ) such that
[Equation]
µ α
[Equation]
m n
[Equation]
∞ ≤ µ α 1 + γ/ [Equation] + sα 2 m n [Equation]
w
[Equation]
[Equation] = µ α 1 + γ/ [Equation] + sα 2 m n h
[Equation] 1) D w (cid:104) C
[Equation]
D
[Equation]
exp( − D 2 w / 2) + exp( − D w ) (cid:105) . We let D w = n κ for any κ > 0 and choose µ m n as τ (cid:39) µ − α 2 + γ/ 2 m n = D w h − 1 = h − 1 n κ .
We write ∆ := µ α 1 + γ/ 2+ sα 2 m n (2 C ) − 1 (cid:39) h
α 1+ sα 2+ γ/ 2 α 2 − γ/ 2 +1 n − κ ( α 1+ sα 2+ γ/ 2 α 2 − γ/ 2 +1) . Then, it holds that (cid:13)(cid:13)(cid:13)(cid:13) ∆ exp (cid:18) −(cid:107) · − c (cid:107)2 h(cid:19) − g (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Here, we set h as h = 2 − k with a positive integer k .
Accordingly, we deﬁne a partition A of Ω so that any A ∈ A can be represented as A = [2 − k j 1 , 2 − k
( j 1 + 1)
] × · · · × [2 − k j d , 2 − k
( j d + 1)] by non-negative integers 0 ≤ j i ≤ 2 k − 1 ( i = 1 , . . .
, d ).
Note that |A| = 2 dk =
h − d .
For each A ∈ A , we deﬁne c
A as c A
=
(2 − k
( j 1 + 1 / 2) , . .
, 2 − k
( j d + 1 / 2))
(cid:62) where ( j 1 , . . .
, j d ) is a set of indexes that satisﬁes A =
[2 − k j 1 , 2 − k
( j 1 + 1)
] × · · · × [2 − k j d , 2 − k
( j d + 1)] .
For each
A ∈ A , we deﬁne g A ∈ conv( F ( n ) γ ) as a function that satisﬁes Eq.
(9) for c =
c A .
Now, we apply Proposition 2 with F ◦ = conv( F ( n ) γ ) and K = K (cid:48)
= dk .
Let R ∗ :=
R lin (conv( F ( n ) γ )) .
First, we can see that there exits a constant F > 0 such that
[Equation]
where we used exp( − n κ ) (cid:28) 1 .
Second, in the event E introduced in the statement of Proposition 2, there exists C such that |{ i ∈ { 1 , . . . , n } | x i ∈
A (cid:48) }| ≤ Cn/ 2 − dk for all A (cid:48)
∈ A .
In this case, we can check that
[Equation]
by the uniform continuity of the Gaussian RBF.
Therefore, we also have
[Equation]
where c > 0 is a constant.
Thus, as long as h is polynomial to n like h =
Θ( n − a ) , the right hand side is O (∆ 2 h d ) .
Now, if we write
[Equation]
then we have ∆ (cid:39) h
˜ β n − κ ˜ β by its deﬁnition.
Here, we choose k as a maximum integer that satisﬁes F 3 32 ∆ 2 2 − dk > R ∗ .
In this situation, it holds that
[Equation]
Since Eq.
(8b) is not satisﬁed, Eq.
(8a) must hold, and hence we have
[Equation]
Therefore, we obtain that
[Equation]
by setting κ (cid:48)
= κ 2 d ˜ β
2˜
β +2 d .
This gives the assertion.
B P ROOFS OF P ROPOSITION 1, T HEOREM 2 AND C OROLLARY
Proposition 1, Theorem 2 and Corollary 1 can be shown by using Propositions 3 and 4 given in
Appendix B.1 shown below.
Let T α W
=
( µ αm w 1 ,m , µ αm w 2 ,m
) ∞ m =1 for W = ( w 1 ,m , w 2 ,m ) ∞ m =1 for α > 0 , and let us consider a model
h W :=
f T − α/ 2 W .
Then, the training error can be rewritten as
[Equation]
For notational simplicity, we let (cid:98) L ( W ) :
= (cid:98) L ( f W ) .
Let H ( M ) be { W ( M ) =
( w 1 ,m , w 2 ,m )
Mm =1 | w 1 ,m ∈ R d
+1 , w 2 ,m ∈ R , 1 ≤ m ≤ M } and ι : H ( M ) → H be the zero padding of W ( M ) , that is, ι ( W ( M ) )
= ( w (cid:48) 1 ,m , w (cid:48) 2 ,m ) ∞ m =1 ∈ H satisﬁes w (cid:48) 1 ,m = w 1 ,m , w (cid:48) 2 ,m =
w 2 ,m ( m ≤ M ) and w (cid:48) 1 ,m = 0 , w (cid:48) 2 ,m = 0 ( m > M ) .
Moreover, we deﬁne ι ∗ : H → H ( M ) as the map that extracts ﬁrst M components.
By abuse of notation, we write f W ( M ) for W ( M ) ∈ H ( M ) to indicate f ι ( W ( M ) ) .
Finally, let A ( M ) : H ( M ) → H ( M ) be a linear operator such that A ( M ) W ( M ) =
ι ∗ ( Aι ( W ( M ) )) , which is just a truncation of A .
Similarly, let
T aM W ( M ) for W ( M ) ∈ H ( M ) be the operator corresponding to T a W for W ∈ H , i.e., T aM W ( M ) =
ι ∗ ( T a ι ( W ( M ) )) .
B. A UXILIARY LEMMAS
First, we show some key propositions to show the main results.
To do so, we utilize the result by
Muzellec et al.
(2020) and Suzuki (2020).
Assumption 3.
(i)
There exists a constant c µ such that µ m ≤ c µ m − 2 .
(ii) There exist B, U > 0 such that the following two inequalities hold for some a ∈ (1 / 4 , 1) almost surely: (cid:107)∇ (cid:98) L ( W ) (cid:107) H ≤ B ( ∀ W ∈ H ) , (cid:107)∇ (cid:98) L ( W ) − ∇ (cid:98) L
[Table]
(iii)
For any data D n , (cid:98) L is three times differentiable.
Let ∇ 3 (cid:98) L ( W ) be the third-order derivative of (cid:98) L ( W ) .
This can be identiﬁed with a third-order linear form and ∇ 3 (cid:98) L ( W ) · ( h, k ) denotes the Riesz representor of l ∈ H (cid:55)→ ∇ 3 (cid:98) L ( W ) · ( h, k, l ) .
There exists α (cid:48) ∈
[0 , 1) , C α (cid:48) ∈ (0 , ∞ )
such that ∀ W, h, k ∈ H , (cid:107)∇ 3 (cid:98) L ( W )
[Table]
( h, k ) [Table] ( W )
[Table] h [Table]
Remark 1.
In the analysis of Br´ehier & Kopec (2016); Muzellec et al.
Suzuki (2020)
Assumption 3-(iii) is imposed for any ﬁnite dimensional projection L ( W ( M ) ) as a function on H ( M ) ) for all M ≥ 1 instead of L ( W ) as a function of H .
However, the condition on L ( W ) gives a sufﬁcient condition for any ﬁnite dimensional projection in our setting.
Thus, we employed the current version
Assumption 4.
For the loss function (cid:96) ( y, f ( x ))
= ( y − f ( x )) 2 , the following conditions holds:
(i)
There exists C > 0 such that for any f W ( W ∈ H ) , it holds that E X,Y [( (cid:96) ( Y, f W
[Equation]
X ))
[Equation]
Y, f [Equation] X )))
[Equation]
≤
[Equation]
( L ( f W ) − L ( f ∗ )) . (ii) β > 0 is chosen so that, for any h : R d → R and x ∈ supp( P X ) , it holds that [Equation]
(iii)
There exists L h > 0 such that (cid:107)∇ W
[Equation]
W − W (cid:48) (cid:107) H ( ∀ W, W (cid:48) ∈ H ) almost surely.
(iv) There exists C h such that (cid:107) h W − h W (cid:48) (cid:107) ∞ ≤ C h (cid:107) W − W (cid:48) (cid:107) H
( W, W (cid:48) ∈ H ) .
Proposition 3.
Assume Assumption 3 holds and β > η .
Suppose that ∃ ¯ R > 0 , 0 ≤ (cid:96) ( Y, f W ( X )) ≤ ¯
R for any W ∈ H (a.s.).
Let ρ = 1 1+ λη/µ 1 and b = µ 1 λ B + c µ βλ .
Accordingly, let ¯ b = max { b, 1 } , κ = ¯ b + 1 and ¯ V = 4¯ b/ ( √ (1+ ρ 1 /η ) /
2 − ρ 1 /η ) .
Then, the spectral gap of the dynamics is given by
[Equation]
where 0 < δ < 1 is a real number satisfying
[Equation]
poly( λ − 1 ) β ))) .
We deﬁne Λ ∗ 0 =
lim η → 0 Λ ∗ η (i.e., ¯ V is replaced by 4¯ b/ ( (cid:113) (1+exp( − λµ 1 )) / 2 − exp( − λµ 1 )) ).
We also deﬁne {+C W 
+}
= κ
[ ¯ V + 1] + √ 2( ¯ R + b ) √ δ .
Then, for any 0 < a < 1 / 4 , the following convergence bound holds for almost sure observation D n : for either L = L or L = (cid:98) L ,
[Equation]
where C 1 is a constant depending only on c µ , B, L, C α (cid:48) , a, ¯ R (independent of η, k, β, λ ).
Proposition 4.
Assume that Assumptions 3 and 4 hold.
Let ˜ α := 1 / { 2( α + 1) } for a given α > 0 and θ be an arbitrary real number satisfying 0 < θ < 1 − ˜ α .
Assume that the true function f o can be represented by h W ∗ =
f o for W ∗ ∈
H θ ( α +1) .
Then, if M ≥ min (cid:8) λ ˜ α/ 2[ θ
[Equation]
+1)] β 1 / 2[ θ ( α +1)] , λ − 1 / 2( α +1) , n 1 / 2[ θ ( α +1)] (cid:9) , the expected excess risk is bounded by
[Equation]
where C is a constant independent of n, β, λ, η, k .
Proof.
Repeating the same argument in Proposition 1 and using the same notation, Propositiongives
[Equation]
for any≤
M ≤ ∞ .
Therefore, we just need to bound the following quantity:
[Equation]
We deﬁne (cid:107) W ( M ) (cid:107) H ( M ) :
= (cid:107) ι ∗ ( W ( M ) )
(cid:107) H for W ( M ) ∈ H ( M ) .
For a > 0 , we deﬁne H ( M ) a be the projection of H a to the ﬁrst M components, H ( M ) a = { ι ( W ) | W ∈ H a } , and we deﬁne (cid:107) W ( M ) (cid:107) H ( M ) a := (cid:107) ι ∗ ( W ( M ) )
(cid:107) H a (note that since H ( M ) a is a ﬁnite dimensional linear space, it is same as H as a set).
Let ν ( M ) β be the Gaussian measure on H ( M ) with mean 0 and covariance ( βA ( M ) ) − 1 , and
˜ ν
( M ) β be the Gaussian measure corresponding to the random variable T α/ 2 M W ( M ) with W ( M ) ∼ ν ( M ) β .
Let the concentration function be
[Equation]
where, if there does not exist W ∈ H ( M )
α +1 that satisﬁes the condition inf , then we deﬁne φ ( M ) β,λ ( (cid:15) )
= ∞ , then Let (cid:15) ∗ > 0 be
[Equation]
Then, Suzuki (2020) showed the following bound:
[Equation]
They also showed that, for M = ∞ , it holds that
[Equation]
Substituting this bound of (cid:15) ∗ to Eq.
[Equation] we obtain Eq.
(13) for M = ∞ .
Moreover, in their proof, if M ≥ ( (cid:15) ∗ ) − 1 / [ θ ( α +1)] , then
[Equation]
Finally, since ˜ ν ( M ) β is a marginal distribution of ˜ ν
( ∞ ) β , it holds that − log
[Equation]
Therefore, as long as M ≥ ( (cid:15) ∗ ) − 1 / [ θ ( α +1)] , the rate of (cid:15) ∗ is not deteriorated from M = ∞ .
In other words, if M ≥ min (cid:8) λ ˜ α/ 2[ θ ( α +1)] β 1 / 2[ θ ( α +1)] , λ − θ/ 2[ θ ( α +1)] , n 1 / 2[ θ ( α +1)] (cid:9) , the bound (13) holds.
Remark 2.
Suzuki (2020) showed Proposition 4 under a condition α > 1 / 2 .
However, this is used only to ensure Assumption 3.
In our setting, we can show Assumption 3 holds directly and thus we may omit the condition α > 1 / 2 .
B. P ROOFS OF P ROPOSITION 1, T HEOREM 2 AND C OROLLARY
Here, we give the proofs of Proposition 1 and Theorem 2 simultaneously.
Proof of Proposition 1 and Theorem 2.
Let ¯ R =
(2 (cid:80) ∞ m =1 a m R + U ) 2 .
Then, we can easily check that ( y i − f W ( x i ))
2 ≤ ¯ R .
As stated above, we use Propositions 3 and 4 to show the statements.
First, we show Proposition 1 for the dynamics of W ( M ) k for any 1 ≤ M ≤ ∞ .
However, it sufﬁces to show the statement only for M = ∞ because the ﬁnite dimensional version can be seen as a speciﬁc case of the inﬁnite dimensional one.
Actually, the dynamics of W ( M ) k is same as that of ι ( ˜ W k ) where ˜ W k ∈ H obeys the following dynamics:
[Equation]
This is because f ι ( ˜ W k ) is determined by only the ﬁrst M components ι ( ˜ W k ) , ι ( ∇ (cid:98) L ( f ι ( ˜ W k ) ))
= ∇ W ( M ) (cid:98) L ( f W ( M ) )
| W ( M ) =
ι ( ˜ W k ) and S η is a diagonal operator.
Since the components of ˜ W k with indexes higher than M does not affect the objective, smoothness of the objective is not lost.
The stationary distribution π ( M ) ∞ of the continuous dynamics corresponding to W ( M ) is a probability measure on H ( M ) that satisﬁes
[Equation]
where ν ( M ) β is the Gaussian measure on R M × ( d +2) with mean 0 and covariance ( βA ( M ) ) − 1 .
We can notice that this is the marginal distribution of the stationary distribution of the continuous time counterpart of ˜ W k : d˜ π ∞ ( ˜ W ) ∝ exp( − β (cid:98) L ( f ι ( ˜ W ) ))
d ν β .
Therefore, we just need to consider an inﬁnite dimensional one.
For this reasoning, we show the convergence for the original inﬁnite dimensional dynamics ( W k ) ∞ k
=1 .
The convergence of the ﬁnite dimensional one ( W ( M ) k ) ∞ k
=1 can be shown by the same manner using the argument above.
To show Proposition 1, we use Propositions 3.
To do so, we need to check validity of Assumptions 3.
First, we check Assumption 3.
Assumption 3-(i) is ensured by Assumption 1.
Next, we check
Assumption 3-(ii).
The boundedness of the gradient can be shown as follows:
[Equation]
Similarly
[Equation]
we can show the Lipschitz continuity of the gradient as
[Equation]
We can also verify Assumption 3-(iii) in a similar way.
Then, we have veriﬁed Assumption 3.
Therefore, we may apply Proposition 3, and then we obtain Proposition 1.
Next, we show Theorem 2 by using Proposition 4.
For that purpose, we need to we verify Assumption 4.
The ﬁrst condition can be veriﬁed as
[Equation]
The second condition can be checked as follows.
Note that
[Equation]
Thus, under the condition β ≤ n/ (2 U 2 ) , the right hand side can be upper bounded by
[Equation]
Next, we check the third and fourth conditions.
Noting that
[Equation]
we have that
[Equation]
= m
[Equation]
, and (cid:107)∇ W
[Equation]
≤ ∞ (cid:88)
[Equation] = [Equation] ≤ ∞ (cid:88) m
[Equation]
≤
[Equation]
for a constant 0 < C 2 < ∞ .
Therefore, it holds that
[Equation]
which yields the forth condition
[Equation]
and we also have
[Equation]
which yields the third condition.
Since f o ∈ F γ , there exists W ∗ ∈ H γ such that f o = f W ∗ .
Therefore, applying Proposition 4 with α = α 1 ( ˜ α
= 1 / [2( α 1 + 1)] ) and θ = γ/ (1 + α 1 ) (since γ < 1 / 2 + α 1 , the condition θ < 1 − ˜ α is satisﬁed), we obtain that for M ≥ min (cid:8) λ 1 / 4 γ ( α 1 +1) β 1 / 2 γ , λ − 1 / 2( α 1 +1) , n 1 / 2 γ (cid:9) , the following excess risk bound holds:
[Equation]
Finally, by noting L ( W ( M
[Equation]
we obtain the assertion.
Finally, we give the proof of Corollary 1.
Proof of Corollary 1.
Note that
[Equation]
Therefore, we may redeﬁne α (cid:48) 1 ← α 1 + qα 2 and s (cid:48)
← s − q
so that we obtain another representation of the model F γ :
[Equation]
where ˇ σ m ( · ) = µ − s (cid:48)
α 2 m σ ( µ − α 2 m · ) .
Note that the condition 0 ≤ q ≤ s − 3 gives s − q ≥ 3 .
Therefore,
Assumptions 3 and 4 are valid even for the redeﬁned parameters α (cid:48) 1 , s (cid:48) and ˇ σ m instead of α 1 , s and σ m .
Therefore, we can apply Theorem 2 by simply replacing α 1 by α (cid:48)
= α 1 + qα 2 .