Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks
Abstract
While signiﬁcant theoretical progress has been achieved, unveiling the generalization mystery of overparameterized neural networks still remains largely elusive.
In this paper, we study the generalization behavior of shallow neural networks (SNNs) by leveraging the concept of algorithmic stability.
We consider gradient descent (GD) and stochastic gradient descent (SGD) to train SNNs, for both of which we develop consistent excess risk bounds by balancing the optimization and generalization via early-stopping.
As compared to existing analysis on GD, our new analysis requires a relaxed overparameterization assumption and also applies to SGD.
The key for the improvement is a better estimation of the smallest eigenvalues of the Hessian matrices of the empirical risks and the loss function along the trajectories of GD and SGD by providing a reﬁned estimation of their iterates.
Introduction
Neural networks have achieved remarkable success in solving large-scale machine learning problems in various application domains such as computer vision and natural language processing [33].
Firstorder methods such as gradient descent (GD) and stochastic gradient descent (SGD) are mainstream optimization algorithms for training neural networks due to their simplicity and efﬁciency [11, 33, 50].
Although the associated optimization problems are nonconvex and nonsmooth, GD/SGD can still ﬁnd a model with a very small or even zero training error [16, 20, 34, 39, 64, 69].
At the same time the models found by such ﬁrst-order methods has demonstrated good generalization performance on test data despite neural networks are often highly overparameterized in the sense that the number of parameters is much larger than the size of training examples [1, 2, 5]
These surprising phenomena have triggered a surge of research activities in understanding the generalization ability of neural networks.
Generalization analysis typically uses complexity measures such as VC dimension, covering numbers or Rademacher complexities to develop capacity-dependent bounds [8, 9, 25, 42, 48], which, however, may not explain well the generalization of overparameterized neural networks.
Impressive alternatives have been proposed which include the compression approach [4], the norm-based analysis [8, 25], the PAC-Bayes analysis [21] and the neural tangent kernel (NTK) approach [5, 28].
In particular, the NTK approach shows that the overparameterization pulls the dynamic of GD on neural networks close to its counterpart on a kernelized machine with the least-square loss [5, 20], which shows how overparameterization can help both optimization and generalization.
However, this approach often requires a very high overparameterization to gain useful results [6, 55, 60].
The recent appealing work [51] presents a kernel-free approach to study how overparameterization would improve the generalization for shallow neural networks (SNNs).
Their basic tool is the algorithmic stability [12], which measures how the replacement of an observation would change the algorithm output.
The authors showed the excess risk of GD is controlled by an interpolating network with the shortest GD path from the initialization, which is able to recover the existing NTK-based risk bounds as an application.
This result is achieved under an overparameterization assumption m (cid:38) ( ηT ) 5 , where m is the number of hidden nodes, η is the learning rate (step size) and T is the number of iterations.
While this result is very interesting and impressive, the overparameterization requirement m (cid:38) ( ηT ) 5 may still be more restrictive than that used in practice.
Furthermore, the analysis in [51] is restricted to the case of the full-batch GD.
One natural question thus arises:
Can we relax the overparameterization requirement for GD in [51] and further establish the stability and generalization of SGD for neural networks?
In this paper, we provide an afﬁrmative answer to the above question by establishing a reﬁned stability analysis for the gradient methods (GD and SGD) for training SNNs.
Our contributions are summarized as follows.
We develop excess risk bounds for GD on SNNs under a relaxed overparameterization.
In more details, we show that GD can achieve the excess risk bounds of the order O (1 / √ n ) if m (cid:38) ( ηT ) 3 , where n is the sample size.
This improves the existing overparameterization condition m (cid:38) ( ηT )
Under a low noise condition, our excess risk bounds improve to O (1 /n ) .
One key technical novelty in relaxing the overparameterization condition for GD in [51] is to improve the existing bounds on the norm of iterate sequence { W t } .
As we soon show in Section 4.1 below, this improvement is achieved by a better estimation of the smallest eigenvalue of the Hessian matrix of the empirical risk.
Speciﬁcally , the analysis [51] uses (cid:107) W t − {+W ( i ) t (cid:107) 
+}
= O ( √ ηt ) to lower-bound the smallest eigenvalue at α W t + (1 − α ) W
( i ) t by − 1 √ m
( (cid:107) W t − W ( i ) t (cid:107) 2 + 1) , where α ∈ (0 , 1) and { W ( i ) t } is an iterate sequence on a neighboring dataset.
As a comparison, we show (cid:107) W t − W ( i ) t (cid:107)
= O ( n − 1 ( ηt ) 32 ) which can be much better than O ( √ ηt ) if n is large.
Furthermore, our bound depends on the training errors and would improve in a low noise condition.
Under some speciﬁc cases, we can further show that E [ (cid:107) W t (cid:107) 22 ] = O (1) , which is independent of the iteration number.
We extend our analysis to SGD under the relaxed overparameterization condition m (cid:38) ( ηT ) 3 .
As compared to GD, SGD has a computational advantage in the sense that it can achieve the same risk bounds with a less computational cost.
The key analysis of SGD relies on the estimation of the Hessian spectrum of the loss over the individual training datum.
This is more challenging than estimating the counterpart of the empirical risk of GD since several properties of GD do not hold for SGD such as the monotonicity of the objective functions along the optimization process.
To overcome this technical hurdle, we provide a reﬁned analysis to control the bounds of the iterates of SGD which further leads to the estimation of the Hessian spectrum of the loss.
The remaining parts of the paper are organized as follows.
We present the related work in Sectionand illustrate the formulation of the problem in Section 3.
We present the main results in Sectionand sketch the idea of the proof in Section 5.
We conclude the paper in Section 6.
Related Work
In this section, we group the related work into two categories: the related work on stability analysis and the related work on generalization analysis of neural networks.
Stability and generalization .
As a fundamental concept in statistical learning theory, algorithmic stability considers how the perturbation of training examples would affect the output of an algorithm [53], which has a close connection to the learnability [46, 56].
The framework of using the concept of algorithmic stability to derive generalization bounds was established in an inﬂuential paper [12], where the uniform stability was introduced and was studied for regularization schemes.
Since then, various concepts of stability have been introduced to study the generalization gaps, including the hypothesis stability [12, 22], on-average stability [32, 56], Bayes stability [38], locally elastic stability [19] and argument/model stability [35, 43].
A very successful application of stability analysis is to use it to study SGD for smooth, Lipschitz and convex problems [26], which motivates a lot of follow-up studies on stochastic optimization [3, 10, 15, 31, 35, 36, 49].
The smoothness assumption in [26] was recently removed by taking very small step sizes [10, 35], while the convexity assumption was relaxed to a weak convexity assumption [52].
Under a Polyak-Lojasiewicz (PL) condition, it was shown that any algorithm converge to global minima would generalize without convexity conditions [15, 36].
The trade-off between stability and optimization was studied in [17].
Other than stochastic optimization, stability has found wide applications in structured prediction [44], meta learning [45], transfer learning [32], hyperparameter optimization [7], minimax problems [23, 37, 65] and adversarial training [61].
While most of the stability analysis imply generalization bounds in expectation, recent studies show that uniform stability can yield almost optimal high-probability bounds [13, 24, 30].
Generalization analysis of Neural Networks (NNs) .
Generalization analysis of NNs has attracted increasing attention to understand their great success in practice.
A popular approach to study the generalization of SNNs is via the uniform convergence approach, which studies the uniform generalization gaps in a hypothesis space [8, 25, 41, 48, 67].
However, this approach leads to capacity-based bounds which do not well explain why overparameterized models can still generalize well to testing examples [47].
To address this problem, researchers turn to other approaches such as the compression approach [4], the PAC-Bayes approach [21], the NTK approach [28] and the neural tangent random feature approach [14].
The key idea of the NTK approach is that, under sufﬁcient overparameterization and random initialization, the dynamics of GD on SNNs is close to the dynamics of GD on a least-squares problem associated to the NTK [5, 20].
This leads to generalization bounds based on a data-dependent complexity measure, which can distinguish the difference between learning with random labels and learning with true labels [5].
Meanwhile, recent studies suggest the connection to kernels might be only good at interpreting the performance of very wide networks [6, 55, 60], much more overparameterized than those used in reality [51].
The most related work is the recent analysis of GD for SNNs without either the NTK condition or the
PL condition [51].
They developed nontrivial generalization bounds under an overparameterization assumption m (cid:38) ( ηT ) 5 .
Furthermore, their analysis allows for improved bounds if there is no label noise, and shows an interesting connection to NTK-based risk bounds.
It should be mentioned that the analysis in [5] considers the ReLU activation function, while the discussions in [51] focus on smooth activation functions.
Problem Setup
Let P be a probability distribution deﬁned on a sample space Z :=
X × Y , where X ⊆ R d and
Y ⊆ R .
Let S
=
{ z i = ( x i , y i ) : i = 1 , . . . , n } be a sample drawn from P .
Based on S we wish to build a model f : X (cid:55)→ R .
The performance of f can be measured by the population risk deﬁned as
[Equation]
which is unknown and can be approximated by the empirical risk L S ( f ) = 12 n (cid:80) ni
=1 (cid:0) f
( x i ) − y i (cid:1) 2 .
A minimizer of the population risk is the regression function f ρ ( x ) =
E [ y | x ] , where E [ ·| x ] denotes the conditional expectation given x .
In this paper, we consider a shallow neural network of the form
[Equation]
) , where we ﬁx µ k ∈ { 1 √ m , − 1 √ m } , σ : R (cid:55)→ R is an activation function and W = ( w 1 , . . .
, w m ) ∈
R d × m is the weight matrix.
In the above formulation, w k denotes the weight of the edge connecting the input to the k -th hidden node, and µ k is the weight of the edge connecting the k -th hidden node to the output node.
Here m is the number of nodes in the hidden layer and (cid:104)· , ·(cid:105) denotes the inner product operator.
For simplicity, we denote
L ( W ) =
L ( f W ) and L S ( W ) =
L S ( f W ) .
Let W ∗ =
arg min W L ( W ) .
We choose a minimizer of L ( W ) with the smallest norm.
The relative behavior of a model W w.r.t.
W ∗ is quantiﬁed by the excess population risk L ( W ) − L ( W ∗ ) .
We denote by (cid:96) ( W ; z ) =
12 ( f W ( x ) − y )
the loss function of W on a single example z
=
( x , y ) .
Two representative algorithms to minimize the empirical risk are GD and SGD.
Deﬁnition 1 (Gradient Descent) .
Let W 0 ∈ R
d
× m be an initialization point.
GD updates { W t } by
[Equation]
where η > 0 is the step size and ∇ denotes the gradient operator.
Deﬁnition 2 (Stochastic Gradient Descent) .
Let W 0 ∈ R
d
× m be an initialization point.
SGD updates { W t } as follows
[Equation]
where i t is drawn from the uniform distribution over [ n ] := { 1 , . .
. , n } .
We are interested in the excess population risk of models trained by GD/SGD with T iterations.
We begin with the introduction of some assumptions on activations and loss functions.
Assumptions 1,were also imposed in [51].
We denote by (cid:107) · (cid:107) 2 the Frobenius norm.
Assumption 1 (Activation) .
The activation φ ( u ) is continuous and twice differentiable with constant
B φ , B φ (cid:48) , B φ (cid:48)(cid:48) > 0 bounding | φ ( u ) | ≤ B φ , | φ (cid:48) ( u ) | ≤ B φ (cid:48) and | φ (cid:48)(cid:48) ( u ) | ≤ B φ (cid:48)(cid:48) for any u ∈ R .
Activation functions satisfying Assumption 1 include sigmoid and hyperbolic tangent activations [51].
Assumption 2 (Inputs, labels, and the loss function) .
There exists constants C x , C y , C 0 > 0 such that (cid:107) x (cid:107) 2 ≤ C x , | y | ≤ C y and (cid:96) ( W 0 ; z ) ≤ C 0 for any x , y and z .
Our third assumption is on the regularity of the learning problems.
For any λ > 0 , we deﬁne
[Equation]
Note we use the asterisk to differentiate W ∗ λ and the GD iterate W t .
Assumption 3 (Regularity) .
Assume there exist α ∈ (0 , 1] and c α > 0 such that
[Equation]
Assumption 3 is related to the approximation error which characterize how well the SNNs approximate the least population risk, which is motivated from the approximation analysis in kernel learning.
In more details, a typical assumption in kernel learning is min f L ( f ) − L ( f ∗ )+ λ (cid:107) f (cid:107) 2 K =
O ( λ α ) , where α ∈ (0 , 1] depends on the regularity of a target function f ∗ and (cid:107) · (cid:107) K denotes the norm in a reproducing kernel Hilbert space.
If (cid:107) W ∗ (cid:107)
=
O (1) , then it is clear that
[Equation]
L [Equation] and therefore Assumption 3 holds with α = 1 .
Our analysis is based on the following error decomposition of the excess risk:
[Equation]
where we have used E
[ L S ( W ∗ 1 ηT )]
=
L ( W ∗ 1 ηT ) due to the independence between W ∗ 1 ηT and S .
We refer to the ﬁrst term E
[ L ( W T )] − E
[ L S ( W T )] as the generalization error (generalization gap) and the second term E (cid:2)
L S ( W T ) − L S ( W ∗ 1 ηT ) − 1 ηT (cid:107) W ∗ 1 ηT − W 0 (cid:107) 22 (cid:3) as the optimization error.
As in [51], we will use the on-average model stability to control the generalization error and tools in optimization theory to control the optimization error.
We will use Assumption 3 to control the last term L ( W ∗ 1 ηT ) + 1 ηT (cid:107) W ∗ 1 ηT − W 0 (cid:107) 22 − L ( W ∗ ) .
The on-average model stability considers the sensitivity of the output models up to the perturbation of a single example, and the sensitivity is averaged by traversing the single example throughout the sample set.
Let A ( S ) be the output model by applying an algorithm A to S .
Deﬁnition 3 (On-average Model Stability [35]) .
Let S
=
{ z 1 , . . .
, z n } and S (cid:48)
= { z (cid:48) 1 , . . .
, z (cid:48) n } be drawn independently from P .
For any i ∈ [ n ] , deﬁne S ( i ) =
{ z 1 , . . .
, z i − 1 , z (cid:48)
i , z i +1 , . . .
, z n } as the set formed from S by replacing the i -th element with z (cid:48)
i .
We say a randomized algorithm A is on-average model (cid:15)
-stable if E S,S (cid:48) ,A (cid:2)
[Table]
The connection between the generalization error and the on-average model stability was established in the following lemma.
We say a function W (cid:55)→ g ( W ) is ρ -smooth if, for any W and W (cid:48) , we have
[Equation]
Lemma 1 (Stability and Generalization [35]) .
Let A be an algorithm.
If for any z , the map W (cid:55)→ (cid:96) ( W ; z ) is ρ -smooth and nonnegative, then
[Equation]
Main Results
In this section, we present our main results on the risk bounds of GD and SGD which are summarized in Table 1.
We denote B (cid:16) B (cid:48) if there exist some universal constants c 1 and c 2 > 0 such that c 1 B ≤ B (cid:48) ≤ c 2 B .
We denote B (cid:38) B (cid:48)
if there exists a universal constant c > 0 such that B ≥ cB (cid:48) .
[Table]
GD [51]
O ( n
[Equation] [Equation]
α )
No m
[Equation]
ηT
[Equation]
n
[Equation]
O
[Equation]
n
[Equation]
O
[Equation]
n
[Equation]
Yes
[Equation]
n
[Equation]
O
[Equation]
n
[Equation]
GD
O
[Equation]
[Table]
This work
[Table]
SGD
[Table]
This work
[Table]
In particular, if α = 1 , our results indicate both GD and SGD for 2-layer SNNs with subquadratic overparametrization m (cid:38) n 3 / 2 can lead to optimal risk rate O ( n − 1 / 2 ) while the results in [46] need superquadratic overparametrization m (cid:38) n 5 / 2 .
4.1 Gradient Descent
We ﬁrst study the excess risk of the GD algorithm for SNNs.
Let e be the base of the natural logarithm.
Let ρ = C 2 x (cid:0) B 2 φ (cid:48)
+ B φ (cid:48)(cid:48)
B φ + B φ (cid:48)(cid:48)
C y √ m (cid:1) and b = C 2 x B φ (cid:48)(cid:48)
( B φ (cid:48)
C x + C 0 ) .
Theorem 2 (Generalization Error) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) and
[Equation]
then for any t ∈ [ T ] we have
[Equation]
Remark 1.
Under an assumption m (cid:38) ( ηT ) 3 , a bound similar to Theorem 2 was established in [51].
We relax this assumption to m (cid:38) ( ηT ) 5 /n 2 + η 2 T 2 in Eq.
As we will show, a typical choice is ηT (cid:16) n 11+ α .
In this case, the assumption in Eq.
(4.1) becomes m (cid:38) ( ηT ) 3 n − 2 α
+ α + η 2 T 2 , which is milder than the assumption m (cid:38) ( ηT ) 3 in [51].
This improvement is achieved by a better estimation of the smallest eigenvalue of a Hessian matrix.
Indeed, the smallest eigenvalue at α W t
+(1 − α ) W ( i ) t is lower bounded by − 1 √ m ( (cid:107) W t − W ( i ) t (cid:107) 2 + 1) (up to a constant factor), where α ∈ (0 , 1) and { W ( i ) t } is the SGD sequence on S ( i ) .
The analysis [51] uses (cid:107) W t − W ( i ) t (cid:107)
= O ( √ ηt ) to control the smallest eigenvalue.
Instead, we show (cid:107) W t − W ( i ) t (cid:107)
= O ( n − 1 ( ηt ) 32 ) (Lemma B.1).
A key step to relax the overparameterization is to build a bound on E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ] .
The existing analysis shows that (cid:107) W t − W 0 (cid:107)
= O ( ηt ) [51], which grows to inﬁnity as we run more and more iterations.
In the following lemma to be proved in Section B.1, we improve it to
E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ]
=
O ( η 2 Tn (cid:80) T − 1 j =
E [ L S ( w j )]
+ (cid:107) W ∗ 1 ηT − W 0 (cid:107) 22 ) .
In particular, if ηT = O ( √ n ) and (cid:107) W ∗ 1 ηT − W 0 (cid:107)
= O (1) , this bound becomes E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ]
= O (1) .
This explains why we relax the overparameterization assumption from m (cid:38) ( ηT ) 5 in [51] to m (cid:38) ( ηT ) 3 .
Furthermore, the bound involves (cid:80) T − 1 j =
E
[ L S ( W j )] which would improve if the training errors are small, which is critical to get fast rates in a low noise case.
Our basic idea to prove Lemma 3 is to ﬁrst control E
[ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ] in terms of training errors.
Our novelty is to replace these training errors with testing errors by using Theorem 2, which allows us to use Eq.
(4.2) to remove some terms.
The proof is given in Section B.2.
For simplicity we assume (cid:107) W ∗ 1 ηT − W 0 (cid:107) 2 ≥ 1 .
Lemma 3.
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) , Eq.
(4.1) holds,
[Equation]
and
[Equation]
then for any t ∈ [ T ] we have
[Equation]
Remark 2.
We impose the assumption E
[ L ( W s )] ≥ L ( W ∗ 1 ηT ) , ∀ s ∈ { 0 , 1 , . .
, T − 1 } .
If this assumption does not hold, then Assumption 3 implies further
[Equation]
This shows the violation of Eq.
(4.2) already implies a model W t , t ∈ [ T ] with a very small excess risk, and therefore the assumption Eq.
(4.2) does not essentially affect our results.
It should be mentioned that if (cid:107) W ∗ (cid:107) 2 =
O (1) we can derive similar results by replacing W ∗ 1 ηT in the analysis with W ∗ (note W ∗ already satisﬁes the inequality L ( W ∗ ) − L ( W ∗ )
+ 1 ηT (cid:107) W ∗ −
W 0 (cid:107)
= O (1 / ( ηT ))
and therefore can play the role of W ∗ 1 ηT ).
In this case, we no longer require the assumption (4.2).
Indeed, Eq.
(4.2) always holds with W ∗ 1 ηT replaced by W ∗ due to the inequality
L ( W s ) ≥ L ( W ∗ ) .
It should be mentioned that the bound in Lemma 3 is stated in expectation.
Therefore, we cannot directly combine this bound and the uniform convergence analysis to derive generalization bounds.
Now we present the optimization error bounds for GD.
Recall R T is deﬁned in Lemma 3.
The proof is given in Section B.2.
Theorem 4 (Optimization Error) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1) with η ≤ 1 / (2 ρ ) .
If Eq.
(4.1) , (4.2) and (4.3) hold, then
[Equation]
Remark 3.
The following optimization error bounds were established in [51]
[Equation]
A key difference between the above bound and Theorem 4 is that Eq.
(4.4) involves a term ( ηT ) 32 √ m , while Theorem 4 involves a term O (cid:16) √ ηTR T √ m (cid:17) .
If R T =
o ( ηT ) , then the optimization error bounds in Theorem 4 would be tighter than Eq.
Indeed, the analysis in [51] requires m (cid:38) ( ηT ) 5 to get the following optimization error bounds
[Equation]
As a comparison, if R T =
O (1) , Theorem 4 requires the assumption m (cid:38) ( ηT ) 3 to derive
[Equation]
We combine the above discussions on generalization and optimization error bounds together to derive the following excess risk bounds.
Note the right-hand side of Eq.
(4.1), (4.3) and Eq.
(4.5) are of the order of ( ηT )
if ηT = O ( n ) and (cid:107) W ∗ 1 ηT − W 0 (cid:107) 2 = O ( √ ηT ) .
The proofs of Theorem 5 and
Corollary 6 are given in Section B.3.
Theorem 5 (Excess Population Risk) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1) with η ≤ 1 / (2 ρ ) .
If ηT = O ( n ) , Eq.
(4.1) , (4.2) , (4.3) hold and
[Equation]
then
[Equation]
where Λ λ is deﬁned in Assumption 3.
The bound in Theorem 5 was also obtained in [51] under the assumption m (cid:38) ( ηT ) 5 .
As a direct corollary, we can use Assumption 3 to show that GD can achieve excess risk bounds of the order
O ( n − α 1+ α ) in the general case, and bounds of the order O ( n − α ) in the case L ( W ∗ ) =
which is due to the incorporation of empirical risks in the generalization bounds.
The basic idea is to balance the optimization and generalization via early-stopping [29, 39, 40, 58, 62, 63].
Similar bounds can be derived by the analysis in [51] under Assumption 3.
Corollary 6.
Let Assumption 3 hold and assumptions in Theorem 5 hold.
(a) If we choose ηT (cid:16) n 1 α +1 and m (cid:16) ( ηT ) 3 (cid:16) n 3 α +1 , then E [ L ( W T )] − L ( W ∗ ) =
O ( n − α 1+ α ) .
(b) If L ( W ∗ ) = 0 , choosing ηT (cid:16) n and m (cid:16) ( ηT ) 3 (cid:16) n 3 implies that E [ L ( W T )]
= O ( n − α ) .
Remark 4.
Other than the stability analysis [51], there are some discussions on the stability analysis for nonconvex functions that can be applied to SNNs [15, 26, 36, 68].
The discussions in [26] use step sizes η t =
O (1 /t ) to get meaningful stability bounds, which, however, is not sufﬁcient for a good convergence of optimization errors.
The discussions in [15, 36, 68] impose a PL condition, and their error bounds depend on a condition number which can be large in practice.
A recent paper [27] studies SGD for one-hidden-layer ReLU network with L 2 regularization from the NTK perspective and derives the appealing minimax optimal rate under the assumption that m is sufﬁciently large (e.g., m is at least larger than O ( n 8 ) ).
However, it is hard to derive a direct comparison since we study one-hidden-layer network with a smooth activation function.
Furthermore, our result holds if η ≤ 1 / (2 ρ ) , which is independent of m and n and is outside of the NTK regime.
As a comparison, the analysis based on NTK [34] requires η ≤ 2 /λ max (Θ) , where Θ ∈ R ( md ) × ( md ) is an neural tangent kernel and therefore the learning rate there is very small.
4.2 Stochastic Gradient Descent
As compared to GD, the analysis of SGD is more challenging since several properties of GD do not hold for SGD.
For example, the analysis in [51] relies critically on the monotonicity of the sequence { L S ( W t ) } , which does not hold for SGD.
Furthermore, the introduced randomness of { i t } increases the variance of the iterates, which increases the difﬁculty of controlling the norm of iterates.
We ﬁrst develop stability and generalization bounds of SGD.
In particular, we are interested in generalization bounds incorporating the training errors in the analysis [32, 35, 51].
This shows how good optimization would improve generalization, which is consistent with the analysis of
SGD in a convex setting [35].
Eq.
(4.7) gives on-average model stability bounds, which imply generalization bounds in Theorem 7.
The proof of Theorem 7 is given in Section C.2.
Without loss of generality we assume 4 TηC 0 ≥ 1 .
Let R (cid:48) T = max { 2 √ TηC 0 , (cid:107) W ∗ 1 ηT − W 0 (cid:107) 2 } and b (cid:48)
=
C 2 x B φ (cid:48)(cid:48)
[Equation]
C x
[Equation]
φ (cid:48)
+ √ 2 C 0 (cid:1) .
Let S ( i ) be deﬁned as in Deﬁnition 3.
Theorem 7 (Stability and Generalization) .
Let Assumptions 1, 2 hold.
Let { W t } t and { W ( i ) t } t be produced by SGD with η ≤ 1 / (2 ρ ) on S and S ( i ) , respectively.
If
[Equation]
then for any t ≤ T − 1 we have
[Equation]
Furthermore, we have the following generalization bounds
[Equation]
We now consider the optimization error bounds of SGD for SNNs.
In the following theorem, we give a bound on the average of the optimization errors for the sequence of SGD iterates.
Recall that R (cid:48)
T is deﬁned above Theorem 7.
Let ∆ t :=
max j =0 ,...,t E [ (cid:107) W j − W ∗ 1 ηT (cid:107) 22 ] for any t ∈ N .
Theorem 8 (Optimization Error) .
Let Assumptions 1, 2 hold.
Let { W t } t be produced by SGD with η ≤ 1 / (2 ρ ) .
If Eq.
(4.6) and Eq.
(4.2) hold, then
[Equation]
Finally, we develop the excess risk bounds for SGD on SNNs.
Note Eq.
(4.8) can be satisﬁed by choosing m (cid:16) ( ηT ) 3 since R (cid:48) T
= O ( √ ηT ) , which matches the overparameterization requirement of
GD and improves the requirement m (cid:38) ( ηT ) 5 in [51].
The proofs of Theorem 9 and Corollaryare given in Section C.3.
Theorem 9 (Excess Population Risk) .
Let Assumptions 1 and 2 hold.
Let { W t } be produced by (3.2) and Eq (4.2) hold.
If η ≤ 1 / (2 ρ ) ,
[Equation]
and T = O ( n ) then we have
[Equation]
Corollary 10.
Let Assumption 3 hold and assumptions in Theorem 9 hold.
We choose an appropriate m (cid:16) ( ηT ) 3 .
(a) We can choose η (cid:16) T − α
+ α and T (cid:16) n to get 1 T (cid:80) T − 1 t =
E [ L ( W t )] − L ( W ∗ ) =
O ( n − α 1+ α ) .
(b) If L ( W ∗ ) = 0 , we can choose T (cid:16) n and η (cid:16) 1 to get 1 T (cid:80) T − 1 t =
E [ L ( W t )]
= O ( n − α ) .
Remark 5.
By Corollary 10, SGD achieves excess risk bounds of the same order to that of GD in
Corollary 6.
An advantage of SGD over GD is that it requires less computation.
To illustrate this, let us consider the general case for example.
In this case, GD requires T (cid:16) n 11+ α to achieve the error bound O ( n − α 1+ α ) .
Since GD requires O ( n ) gradient computations per iteration and therefore the total gradient computation complexity is O ( n 2+ α 1+ α ) .
As a comparison, SGD requires O ( n ) gradient computations and therefore saves the computation by a factor of O ( n 11+ α ) .
Note Corollary 6 considers the risk for the last iterate, while Corollary 10 considers the average of risks for all iterates.
The underlying reason is that GD consistently decreases the training errors along the optimization process, while SGD does not enjoy this property.
Note that the overparameterization requirement becomes m (cid:16) n 3 α +1 and m (cid:16) n 3 in Part (a) and Part (b), respectively.
Main Idea of the Proof 5.1 Gradient Descent
In this subsection, we sketch our idea on the proof on gradient descent.
Generalization errors .
The starting point of our proof is the following bound given in Lemma A.
[Equation]
To apply the above inequality, we need to give a lower bound of 1 − η (cid:107) W t − W ( i ) t (cid:13)(cid:13) 2 √ m .
The analysis in [51] uses the crude bound (cid:107) W t − W ( i ) t (cid:13)(cid:13) 2 ≤ (cid:107) W t − W
(cid:13)(cid:13)
+ (cid:107) W 0 − W
( i )
t (cid:13)(cid:13) 2 (cid:46) √ ηt, which does not use the fact that W t +1 and W ( i ) t +1 are produced by SGD on neighboring datasets.
By the generation of W t +1 and W ( i ) t +1 , we show that (cid:13)(cid:13) W t − W ( i ) t (cid:13)(cid:13)
=
O
(( ηt ) 32 /n )
(Lemma B.1).
This explains why we get a relaxed overparameterization in the stability analysis as compared to [51].
Optimization errors .
The starting point of our proof is the following bound given in Eq.
(B.8)
[Equation]
The analysis in [51] controls (cid:107) W ∗ 1 ηT − W s (cid:107) 32 as follows
[Equation]
As a comparison
[Equation]
we use (cid:107) W s − W 0 (cid:107)
= O ( √ ηs ) in Eq.
(5.1) and show that
E [
[Equation]
W ∗ 1 ηT − W t (cid:107) 22 ] ηt can be bounded from above by
[Equation]
where we have used the overparameterization m (cid:38) ( ηT ) 3 .
It then follows that
[Equation]
Furthermore, we can apply stability analysis to relate E [ L S ( W s )] to E
[ L ( W s )] , and get (Lemma 3)
[Equation]
which is sharper than the bound (cid:107) W ∗ 1 ηT − W t (cid:107)
=
O ( √ ηt ) in [51].
This explains why we get a relaxed overparameterization in the optimization error analysis as compared to [51].
5.2 Stochastic Gradient Descent
Our starting point is to prove (cid:107) W t − W 0 (cid:107)
=
O ( √ ηT ) for t ∈ [ T ] .
This was shown for GD in [51].
However, the analysis there relies heavily on the following inequality L S ( W j +1 ) ≤
[Equation]
, which does not hold for SGD.
We use the induction strategy to show (cid:107) W t − W 0 (cid:107) =
O ( √ ηT ) .
If (cid:107) W t − W 0 (cid:107) =
O ( √ ηT ) , Lemma A.1 implies λ min ( ∇ 2 (cid:96) ( W t ; z )) (cid:38) − √ ηT √ m .
If m (cid:38) ( ηT ) 3 we can use the update strategy of SGD and the induction assumption to show (cid:107) W t +1 − W 0 (cid:107) =
O ( √ ηT ) .
The bound (cid:107) W t − W 0 (cid:107) =
O ( √ ηT ) is a crude estimate of the norm of iterates.
To get our results, we show the following sharper bound on the norm of iterates by considering bounds in expectation (Lemma C.2)
[Equation]
To show this, we use E [
[Equation]
− W
[Equation] [Equation]
[Equation]
W
[Equation]
W
[Equation]
[Equation] [Equation]
W
[Equation]
η
[Equation]
L
[Equation]
W
[Equation]
−
L S ( W t ) (cid:3) +
[Equation]
√
[Equation]
√ m E [ (cid:107) W
[Equation]
1 ηT − W t (cid:107) 22 ] (Eq. (C.5), up to a constant factor).
We take a summation of this inequality and use m (cid:38) ( ηT ) 3 to get
E [
[Equation]
, from which we get Eq.
The bound in Eq.
(5.2) requires to estimate (cid:80) tj =
E [ L S ( W j )] .
Our next step is then to control (cid:80) tj =
E [ L S ( W j )] as follows (Lemma C.3)
[Equation]
Conclusion
In this paper, we present stability and generalization analysis of both GD and SGD to train neural networks.
Under a regularity assumption, we show both GD and SGD can achieve excess risk bounds of the order O ( n − α α +1 ) , which further improve to the order O ( n − α ) under a low noise condition.
As compared to the existing stability analysis [51], we achieve our bounds under a relaxed overparameterization assumption and extend the existing analysis on GD to SGD.
Our improvement is achieved by developing sharper bounds on norm of the GD/SGD iterate sequences.
There remain several interesting questions for further discussion.
The ﬁrst question is whether the overparamterization requirement m (cid:38) ( ηT ) 3 can be further improved, and whether the overparameterization requirement can be independent of T .
Second, our analysis applies to SNNs with a smooth activation function.
It would be very interesting to extend our analysis to SNNs with the ReLU activation function.
A key challenge in this direction is to control the smallest eigenvalue of the associated Hessian matrix [51].
Third, our bounds are stated in expectation.
It would be useful to develop high-probability bounds to understand the robustness of the algorithm.
Finally, our analysis requires early-stopping in a low noise-setting.
It would be very interesting to develop risk bounds in a low-noise setting without early-stopping [54].
Acknowledgement.
The authors are grateful to the anonymous reviewers for their thoughtful comments and constructive suggestions.
Yiming’s work is supported by NSF grants (IIS-2103450,
IIS-2110546 and DMS-2110836)
Appendix for “Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks” A Lemmas
In this section, we collect several lemmas useful for our analysis.
The following lemma shows that the loss function is smooth and the loss function is weakly convex.
We develop a lower bound for the eigenvalue of the Hessian matrix which is slightly different from that in [51].
Let λ min ( A ) denote the smallest eigenvalue of a matrix A and ∇ 2 f denote the Hessian matrix of a function f .
We use a ∨ b = max { a, b } for any a, b ∈ R .
Lemma A.1 (Smoothness and Curvature [51]) .
Let z ∈ Z .
The function W (cid:55)→ (cid:96) ( W ; z ) is ρ -smooth.
For any W , we have
[Equation]
Proof.
The smoothness of the loss function was established in [51].
We only prove Eq.
(A.1).
The following inequality was established in [51]
[Equation]
We know
[Equation]
where we have used the following inequality established in [51]
[Equation]
It then follows that
[Equation]
The stated bound then follows directly.
The proof is completed.
Lemma A.2.
Let W , W (cid:48)
∈ R
d × m .
Then
[Equation]
where R = max { 1 , (cid:107) W − W 0 (cid:107) 2 , (cid:107) W (cid:48) − W 0 (cid:107) 2 } .
Proof.
According to Taylor’s theorem
[Equation]
there exists α ∈ [
[Equation]
1] such that
[Equation]
where W ( α ) =
α W + (1 − α ) W (cid:48)
and we have used Lemma A.1.
The proof is completed.
The following lemma shows the self-bounding property of smooth and nonnegative functions.
Lemma A.3 ([57]) .
Assume for all z , the function w (cid:55)→ (cid:96) ( w ; z ) is nonnegative and L -smooth.
Then (cid:107)∇ (cid:96) ( w ; z ) (cid:107) 22 ≤ 2 L(cid:96) ( w ; z ) .
The following recursive relationship on stability of GD was established in [51].
Note (cid:15) t deﬁned in Eq.
(A.4) is slightly different from that in [51].
Indeed, the discussions [51] derive the following lemma in their analysis.
The difference is that they further control (cid:107) W t − W ( i ) t (cid:107) 2 in Eq.
(A.4) as follows
[Equation]
Lemma A.
Let Assumptions 1, 2 hold.
Let { W t } t be produced by (3.1) .
If η ≤ 1 / (2 ρ ) , then for any t ∈ N we have
[Equation]
where
[Equation]
The following lemma shows how the GD iterate would deviate from the initial point.
Lemma A.5 ([51]) .
Let Assumptions 1, 2 hold and assume η ≤ 1 / (2 ρ ) .
Let { W t } be produced by
Eq.
Then for any t ∈ N we have
[Equation]
The following lemma shows an almost co-coercivity of the gradient operator associated with shallow neural networks, which plays an important role for the stability analysis.
Lemma A.6 (Almost Co-coercivity of the Gradient Operator [51]) .
Let Assumptions 1, 2 hold.
If η ≤ 1 / (2 ρ ) , then for any t ∈ N we have
[Equation]
where
[Equation]
Remark 6.
The above lemma can be proved in a way similar to Lemma 5 in [51] but using the following inequality to control the eigenvalue of Hessian matrix (see, e
[Equation]
g, (A.2))
[Equation]
where α ∈ [0 , 1] and
[Equation]
From the smoothness of (cid:96)
[Equation]
we further know that
[Equation]
Consequently, min
[Equation]
The remaining arguments in proving Lemma A.6 is the same as proving Lemma 5 in [51].
We omit the proof for simplicity.
As a comparison, the paper [51] uses the following inequality
[Equation]
and uses the following decomposition to estimate | f W ( α ) ( x ) − y |
[Equation]
However, the above estimation does not apply to SGD because we consider the loss function over a single datum instead of the empirical risk over the whole training data and one cannot guarantee |
f W ( i ) t ( x ) − y | ≤ √ 2 C 0 .
B Proofs on Gradient Descent B.1 Proofs on Generalization Bounds
We ﬁrst present a lemma on the uniform stability of GD, which will be used in lower bounding the smallest eigenvalue of Hessian matrices.
Lemma B.1.
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) and Eq.
(4.1) holds, then
[Equation]
Proof.
We can apply Lemma A.4 recursively and derive
[Equation]
n
[Equation] [Equation]
Furthermore
[Equation]
it follows from the ρ -smoothness of (cid:96) and Lemma A.5 that
[Equation]
In a similar way
[Equation]
we can show
[Equation]
We can combine the above three inequalities together and derive
[Equation]
We can choose p = 1 /t and use (1 + 1 /t )
t ≤ e to get (cid:13)(cid:13)
[Equation]
We now prove by induction to show that
[Equation]
Eq.
(B.3) with k = 0 holds trivially.
We now assume Eq.
(B.3) holds for all k ≤ t and want to show that it holds for k = t + 1 ≤ T .
Indeed, according to the induction hypothesis we know
[Equation]
It then follows from Eq.
(B.2) that
[Equation]
Furthermore, Eq.
(4.1) implies 2 η(cid:15) (cid:48) ≤ 1 / ( t + 1) and therefore (cid:16) 1 − 2 η(cid:15) (cid:48) (cid:17) t ≤ (cid:16)
[Equation]
B
[Equation]
It then follows that
[Equation]
This shows the induction hypothesis and completes the proof.
Proof of Theorem 2.
According to Eq.
(B.1) with p = 1 /t
and Eq.
(B.4)
we get
[Equation]
where we have used the self-bounding property of smooth functions
(Lemma A
[Equation]
We take an average over i ∈ [ n ] and get
[Equation]
where we have used E [ (cid:96) ( W j ; z i )]
=
E [ (cid:96) ( W ( i ) j ; z (cid:48) i )] due to the symmetry between z i and z (cid:48)
i .
According to Lemma 1 we further get
[Equation]
It then follows from L S ( W t ) ≤ 1 t (cid:80) t
[Equation]
1 j =
L S ( W j ) [51] that
[Equation]
The proof is completed.
B.2 Proofs on Optimization Error Bounds
Before giving the proof on optimization error bounds, we ﬁrst prove Lemma 3 on a bound of the GD iterates.
Proof of Lemma 3.
According to Theorem 2, we know
[Equation]
The following inequality was established in [51] for any W
[Equation]
We take expectation over both sides and choose W = W ∗ 1 ηT to get (note we do not have E [1 ∨ (cid:107) W −
W s (cid:107) 32 ] ≤ 1 ∨ E
[ (cid:107) W − W s (cid:107) 32 ] .
However, Eq.
(B.8) still holds if one check the analysis in [51].
Indeed
[Equation]
they upper bounded a sum of two terms by the maximum and one can exchange the sum and expectation.
We omit the details for simplicity)
[Equation]
According to Eq.
(B.6) we further get
[Equation]
− 1 (cid:88)
[Equation] [Equation]
] (cid:1)
[Equation]
Since E [ L ( W s )] ≥ L ( W ∗ 1 ηT ) we further get
[Equation] [Equation]
We can further use Lemma A.5 to derive
[Equation]
Let ∆ = max s ∈ [ T ] E [ (cid:107) W ∗ 1 ηT − W s (cid:107) 22 ] ∨ 1 .
The above inequality actually implies ∆ ≤ (cid:16)
[Equation]
According to the assumption m ≥ 4 b 2 ( ηT ) 2 (cid:0) √ 2 ηTC
[Equation]
+ E [ (cid:107) W ∗ 1 ηT − W 0 (cid:107) 2 ] (cid:1)
[Equation]
we further get
[Equation]
and therefore
[Equation]
The proof is completed.
Now we are ready to prove Theorem
[Equation]
Proof of Theorem 4.
According to Eq.
(B.7) with W = W ∗ 1 ηT we have
[Equation]
where we have used Lemma A.5 in the last step.
Since { L S ( w t ) } is monotonically decreasing [51], we derive
[Equation]
We then apply Lemma 3 to get the stated bound.
The proof is completed.
Both bounds in Theorem 2 and Lemma 3 depend on the term (cid:80) T − 1 s =0 E (cid:2)
L S ( W s ) (cid:3) , for which we provide a bound in the following lemma.
Lemma B.2.
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1) with η ≤ 1 / (2 ρ ) .
If
Eq.
(4.1) , (4.2) , (4.3) , (4.5) hold, then
[Equation]
Proof.
Taking expectation over both sides of Eq.
(B.9) we derive
T
[Equation]
It then follows from Lemma 3 that
[Equation]
By Eq.
(4.5), we have
T −
[Equation]
The stated bound follows directly.
The proof is completed.
Combined with Assumption 3, Lemma B.2 implies (if m (cid:38) η 3 T 3 )
T − 1 (cid:88)
[Equation]
If L ( W ∗ ) = 0 , we have (cid:80) T − 1 s =0 E (cid:2)
L S ( W s ) (cid:3) =
O ( T ( Tη ) − α ) , which explains why we can get improved bounds in a low noise case.
B.3 Proofs on Excess Risks Bounds
Proof of Theorem 5.
We have the following error decomposition
[Equation]
(B.10)
Theorem 2 implies
[Equation]
We can plug the above generalization bounds, the optimization bounds in Theorem 4 and the deﬁnition of Λ 1 ηT back into Eq.
(B.10), and derive
[Equation]
According to the deﬁnition of Λ 1 ηT , we know
[Equation]
and therefore R T deﬁned in Lemma 3 satisﬁes
[Equation]
According to Lemma B.
[Equation]
we know
[Equation]
It then follows that
[Equation]
We can plug the above bounds on R T and (cid:80) T − 1 s =0 E (cid:2)
L S ( W s ) (cid:3) back into Eq.
(B.11), which implies
E [
[Equation]
Since ηT = O ( n )
[Equation]
the above bound further translates to
[Equation]
Since m (cid:38) ( ηT ) 3 we further have
[Equation]
The stated bound then follows from L ( W ∗ 1 ηT ) + 1 ηT (cid:107) W ∗ 1 ηT − W 0 (cid:107)
= L ( W ∗ )
+ Λ 1 ηT .
The proof is completed.
Proof of Corollary 6.
According to Theorem 5 and Assumption 3, we know
[Equation]
We ﬁrst prove Part (a).
For the choice ηT (cid:16) n 1 α +1 , we have ηTL
[Equation]
∗
[Equation]
n
[Equation]
n
[Equation]
1+ α and
[Equation]
Part (b) follows directly from the choice Tη (cid:16) n .
Note these choices of ηT satisfy ηT = O ( n ) .
The proof is completed.
C Proofs on Stochastic Gradient Descent C.1 A Crude Bound on SGD Iterates
We ﬁrst provide a crude bound on the SGD iterates, which would be useful for our analysis.
Lemma C.1 (Iterate Bound) .
Let Assumptions
[Equation]
2 hold.
Let { W t } t be produced by SGD.
If η ≤ 1 / (2 ρ ) and m ≥ 64 C
( b (cid:48) )
2 ( Tη ) 3 , then for any t ∈
[ T ] we have
[Equation]
Proof.
According to Eq.
(3.2) we have the following inequality for any W ,
[Equation]
We now prove by induction to show the following inequality for all t ∈
[ T ]
[Equation]
It is clear that Eq.
(C.2) holds for t = 0 .
We now assume Eq.
(C.2) holds for all t ≤ j and want to prove it holds for t =
j + 1 ≤ T .
According to Lemma A.2 and the induction hypothesis we have the following inequality for all t ≤ j
[Equation]
We can combine the above inequality and Eq.
(C.1) with W = W 0 , which gives the following inequality for any t ≤ j
[Equation]
where we have used the self-bounding property and the assumption η ≤ 1 /ρ
[Equation]
We can take a summation of the above inequality and derive
[Equation]
where we have used the assumption m ≥ 64 C
( b (cid:48) )
2 ( Tη ) 3 .
This shows Eq.
(C.2) with t =
j
+ 1 .
The proof is completed.
C.2 Proofs on Generalization Bounds
Proof of Theorem 7.
We ﬁrst prove the stability of SGD.
We consider two cases.
If i t (cid:54) =
i , then according to the SGD update (3.2), we know
[Equation]
According to Lemma A.6, we further have
[Equation]
η
[Equation]
2 ηρ
[Table]
where (cid:15) (cid:48) t is deﬁned in Eq.
(A.5).
It then follows from η ≤ 1 / (2 ρ ) that
[Equation]
If i t (cid:54) =
i
[Equation]
we can use ( a + b ) 2 ≤ (1 + p ) a 2 + (1 + 1 /p )
b 2 to derive
[Equation]
where we have used the self-bounding property.
We can combine the above two cases to derive
[Equation]
We can apply the above inequality recursively and derive
[Equation]
E
[Equation]
W j
[Equation]
z i ) (cid:3) , where we have used the symmetry between z i and z (cid:48)
i and p = n/t .
Since (cid:107) W j − W 0 (cid:107) 2 ≤ R (cid:48) T and (cid:107) W ( i ) j − W 0 (cid:107) 2 ≤ R (cid:48) T , we know
[Equation]
√ m .
Furthermore, Eq.
(4.6) implies 2 η(cid:15)
(cid:48) s ≤ 1 / ( t + 1) and therefore t (cid:89)
j =
[Equation]
+ t
[Equation]
≤
[Equation] [Equation]
t
[Equation]
t
[Equation]
t ≤ e 2 .
It then follows that
[Equation]
We take an average over i ∈ [ n ] and get
[Equation]
Now we prove the generalization bounds for SGD.
According to Lemma
[Equation]
we have
[Equation]
It then follows from (4.7) that
[Equation]
The proof is completed.
The iterate bound in Lemma C.1 is a bit crude.
In the following lemma, we show this bound can be improved if we consider bounds in expectation.
Recall ∆ t :=
max j =0 ,...,t E [ (cid:107) W j − W ∗ 1 ηT (cid:107) 22 ] for any t ∈ N .
If tη 2 = O (1) and t = O ( n ) , Lemma C.2 shows ∆ t =
O ( (cid:107) W 0 − W ∗ 1 ηT (cid:107) 22 ) which is signiﬁcantly better than the bound O ( ηt ) in Lemma C.1.
This allows us to get excess risk bounds under a relaxed overparameterization.
Similar to the case with GD, this upper bound depends on the training errors of SGD iterates.
Lemma C.2.
Let Assumptions 1, 2 hold.
Let { W t } t be produced by SGD with η ≤ 1 / (2 ρ ) .
If Eq.
(4.6) and Eq.
(4.2) hold, then
[Equation]
Proof of Lemma C.2.
We take expectation w.r.t.
i t over both sides of Eq.
(C.1) and get
[Equation]
where the last step is due to Lemma A.2 and Lemma C.1.
Taking expectation over both sides of Eq.
(C.4)
[Equation]
we derive
[Equation]
This together with Theorem 7 implies
[Equation]
+ 2 ηb (cid:48) R (cid:48) T √ m E [ (cid:107) W ∗ 1 ηT − W t (cid:107) 22 ] + 8 e
ρ 2 (1 + t/n ) η 3 n t (cid:88) j
=
E [ L S ( W j )] + 8 eρη 2 (cid:16) (1 + t/n ) E
[ L S ( W t )] n t (cid:88) j
=
E [ L S ( W j )] (cid:17) 12 .
The assumption E (cid:2) L ( W t ) (cid:3) ≥ L ( W ∗ 1 ηT ) further implies
E [ (cid:107) W t +1 − W ∗ 1 ηT (cid:107) 22 ] ≤ E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ] + 2 ρη 2 E
[ L S ( W t )]
+ 2 ηb (cid:48) R (cid:48) T √ m E [ (cid:107) W ∗ 1 ηT − W t (cid:107) 22 ] + 8 e
ρ 2 (1 + t/n ) ηn t (cid:88)
j =
E [ L S ( W j )] + 8 eρη 2 (cid:16) (1 + t/n ) E
[ L S ( W t )] n t (cid:88) j
=
E [ L S ( W j )] (cid:17) 12 .
We take a summation of the above inequality and derive
[Equation]
According to the concavity of x (cid:55)→ √ x
[Equation]
we further get
[Equation]
It then follows that
[Equation]
Let ∆ t = max j =0 ,...,t E [
[Equation]
W
[Equation]
W
[Equation]
Then the above inequality actually implies (note it holds for any t )
[Equation]
where we have used 4( t + 1)
ηb (cid:48) R (cid:48)
T ≤ √ m .
It then follows that ∆
[Equation]
L
[Equation]
W
[Equation] [Equation]
The proof is completed.
Proof of Theorem 8.
According to (C.5) and Lemma C.2, we know
[Equation]
We take a summation of the above inequality and get the stated bound.
The proof is completed.
C.3 Proofs on Excess Risk Bounds
Before proving the excess risk bounds, we ﬁrst develop a useful lemma to control the term (cid:80) T −t =
E [ L S ( W t )] , which appears in our generalization bounds.
Lemma C.3.
Let Assumptions 1, 2 hold.
Let { W t } be produced by (3.2) with η ≤ 1 / (2 ρ ) .
If Eq.
(4.6) , Eq.
(4.2) hold and
[Equation]
Tρη 2 R (cid:48) T (cid:1) 2 (cid:16) 1 + 4 e
[Equation]
ηρT (1 + T/n )
[Equation]
C
[Equation]
then we have
[Equation]
Proof.
According to Eq.
(C.4)
[Equation]
we know
[Equation]
Since η ≤ 1 / (2 ρ )
[Equation]
we get
[Equation]
We take a summation of the above inequality and get
T − 1 (cid:88)
[Equation]
According to Lemma C.2 we further get
T
[Equation]
t =
[Equation]
By Eq.
(C.6), we further get
T
[Equation] [Equation]
This shows the stated bound.
The proof is completed.
Now we prove the excess generalization bounds for SGD.
Proof of Theorem 9.
By Theorem 8, we have 2 η
[Equation]
1 (cid:88) t =
[Equation]
where ∆ T := max j =0 ,...,T E [ (cid:107) W j − W ∗ 1 ηT (cid:107) 22 ] .
According to Theorem 7, we know
T − 1 (cid:88) t =
[Equation]
where we have used the concavity of x (cid:55)→ √ x .
We can combine the above two inequalities together and get
[Equation]
It then follows from the assumption m ≥ (4 Tηb (cid:48) R (cid:48) T )
that
[Equation]
According to Lemma C
[Equation] [Equation]
we know
[Equation]
We can combine the above two inequalities together to derive
[Equation]
It then follows Assumption 3 that
[Equation]
We can use Lemma C
[Equation]
3 to control (cid:80) T − 1 t =
E [ L S ( W t )] and get η
T − 1 (cid:88) t =
[Equation]
It then follows that
[Equation]
Since (cid:107) W 0 − W ∗ 1 ηT (cid:107) 22 ≤ ( ηT )Λ 1 ηT , we further get
[Equation]
The stated bound follows from m ≥ (4 Tηb (cid:48) R (cid:48) T ) 2 , T
= O ( n ) and L ( W ∗ 1 ηT )
≤ L ( W ∗ )
+ Λ 1 ηT .
The proof is completed.
Proof of Corollary 10.
According to Assumption 3 and Theorem 9, we know
[Equation]
We ﬁrst prove Part (a).
Since η (cid:16) T − α 1+ α and T (cid:16) n , we know ( Tη
[Equation]
α =
O
[Equation]
n
[Equation]
1+ α ) and η = O ( n − α 1+ α ) .
If L ( W ∗ ) = 0 , we know
[Equation]
In this case, we can choose T (cid:16) n and η (cid:16) 1 to get ( Tη ) − α
= O ( n − α ) .
The proof is completed.