Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
While signiﬁcant theoretical progress has been achieved, unveiling the generalization mystery of overparameterized neural networks still remains largely elusive.
In this paper, we study the generalization behavior of shallow neural networks(SNNs) by leveraging the concept of algorithmic stability.
We consider gradientdescent (GD) and stochastic gradient descent (SGD) to train SNNs, for both ofwhich we develop consistent excess risk bounds by balancing the optimizationand generalization via early-stopping.
As compared to existing analysis on GD,our new analysis requires a relaxed overparameterization assumption and alsoapplies to SGD.
The key for the improvement is a better estimation of the smallesteigenvalues of the Hessian matrices of the empirical risks and the loss functionalong the trajectories of GD and SGD by providing a reﬁned estimation of theiriterates.
Introduction
Neural networks have achieved remarkable success in solving large-scale machine learning problemsin various application domains such as computer vision and natural language processing [32].
Firstorder methods such as gradient descent (GD) and stochastic gradient descent (SGD) are mainstreamoptimization algorithms for training neural networks due to their simplicity and efﬁciency [11, 32, 46].
Although the associated optimization problems are nonconvex and nonsmooth, GD/SGD can stillﬁnd a model with a very small or even zero training error [16, 20, 33, 38, 58, 61].
At the same time,the models found by such ﬁrst-order methods has demonstrated good generalization performance ontest data despite neural networks are often highly overparameterized in the sense that the number ofparameters is much larger than the size of training examples [1, 2, 5].
These surprising phenomena have triggered a surge of research activities in understanding thegeneralization ability of neural networks.
Generalization analysis typically uses complexity measuressuch as VC dimension, covering numbers or Rademacher complexities to develop capacity-dependentbounds [8, 9], which, however, may not explain well the generalization of overparameterized neuralnetworks.
Impressive alternatives have been proposed which include the compression approach [4],the norm-based analysis [8, 25], the PAC-Bayes analysis [21] and the neural tangent kernel (NTK)approach [5, 27].
In particular, the NTK approach shows that the overparameterization pulls thedynamic of GD on neural networks close to its counterpart on a kernelized machine with theleast-square loss [5, 20], which shows how overparameterization can help both optimization andgeneralization.
However, this approach often requires a very high overparameterization to gain usefulresults [6, 50, 54].
The recent appealing work [47] presents a kernel-free approach to study how overparameterizationwould improve the generalization for shallow neural networks (SNNs).
Their basic tool is thealgorithmic stability [12], which measures how the replacement of an observation would change the algorithm output.
The authors showed the excess risk of GD is controlled by an interpolating networkwith the shortest GD path from the initialization, which is able to recover the existing NTK-basedrisk bounds as an application.
This result is achieved under an overparameterization assumptionm (cid:38) ( ηT ) 5 , where m is the number of hidden nodes, η is the learning rate (step size) and T is thenumber of iterations.
While this result is very interesting and impressive, the overparameterizationrequirement m (cid:38) ( ηT ) 5 may still be more restrictive than that used in practice.
Furthermore, theanalysis in [47] is restricted to the case of the full-batch GD.
One natural question thus arises:
Can we relax the overparameterization requirement for GD in [47] and furtherestablish the stability and generalization of SGD for neural networks?
In this paper, we provide an afﬁrmative answer to the above question by establishing a reﬁnedstability analysis for the gradient methods (GD and SGD) for training SNNs.
Our contributions aresummarized as follows.
We develop excess risk bounds for GD on SNNs under a relaxed overparameterization.
Inmore details, we show that GD can achieve the excess risk bounds of the order O (1 / √ n ) ifm (cid:38) ( ηT ) 3 , where n is the sample size.
This improves the existing overparameterizationcondition m (cid:38) ( ηT ) 5 [47].
Under a low noise condition, our excess risk bounds improve toO (1 /n ) .2.
One key technical novelty in relaxing the overparameterization condition for GD in [47] is to
improve the existing bounds on the norm of iterate sequence { W t } .
As we soon show in Section
4.1 below, this improvement is achieved by a better estimation of the smallest eigenvalue of the
Hessian matrix of the empirical risk.
Speciﬁcally, the analysis [47] uses
(cid:107)
[Equation]
to lower-bound
the smallest eigenvalue at α W t + (1 − α ) W ( i ) t by − 1 √ m ( (cid:107) W t − W ( i ) t (cid:107) 2 + 1) ,where α ∈ (0 , 1) and { W ( i ) t } is an iterate sequence on a neighboring dataset.
As a comparison,we show (cid:107) W t − W ( i ) t (cid:107)
= O ( n − 1 ( ηt ) 32 ) which can be much better than O ( √ ηt ) if n is large.
Furthermore, our bound depends on the training errors and would improve in a low noise condition.
Under some speciﬁc cases, we can further show that E [ (cid:107) W t (cid:107) 22 ] = O (1) , which is independent ofthe iteration number.3.
We extend our analysis to SGD under the relaxed overparameterization condition m (cid:38) ( ηT ) 3 .
Ascompared to GD, SGD has a computational advantage in the sense that it can achieve the samerisk bounds with a less computational cost.
The key analysis of SGD relies on the estimation ofthe Hessian spectrum of the loss over the individual training datum.
This is more challenging thanestimating the counterpart of the empirical risk of GD since several properties of GD do not holdfor SGD such as the monotonicity of the objective functions along the optimization process.
Toovercome this technical hurdle, we provide a reﬁned analysis to control the bounds of the iteratesof SGD which further leads to the estimation of the Hessian spectrum of the loss.
The remaining parts of the paper are organized as follows.
We present the related work in Section 2and illustrate the formulation of the problem in Section 3.
We present the main results in Section 4and conclude the paper in Section 5.
Related Work
In this section, we group the related work into two categories: the related work on stability analysisand the related work on generalization analysis of neural networks.
Stability and generalization .
As a fundamental concept in statistical learning theory, algorithmicstability considers how the perturbation of training examples would affect the output of an algorithm [49], which has a close connection to the learnability [43, 51].
The framework of using theconcept of algorithmic stability to derive generalization bounds was established in an inﬂuentialpaper [12], where the uniform stability was introduced and was studied for regularization schemes.
Since then, various concepts of stability have been introduced to study the generalization gaps,including the hypothesis stability [12, 22], on-average stability [31, 51], Bayes stability [37], locallyelastic stability [19] and argument/model stability [34, 40].
A very successful application of stabilityanalysis is to use it to study SGD for smooth, Lipschitz and convex problems [26], which motivates a lot of follow-up studies on stochastic optimization [3, 10, 15, 30, 34, 35].
The smoothness assumptionin [26] was recently removed by taking very small step sizes [10, 34], while the convexity assumptionwas relaxed to a weak convexity assumption [48].
Under a Polyak-Lojasiewicz (PL) condition,it was shown that any algorithm converge to global minima would generalize without convexityconditions [15, 35].
The trade-off between stability and optimization was studied in [17].
Otherthan stochastic optimization, stability has found wide applications in structured prediction [41], metalearning [42], transfer learning [31], hyperparameter optimization [7], minimax problems [23, 36, 59]and adversarial training [55].
While most of the stability analysis imply generalization bounds inexpectation, recent studies show that uniform stability can yield almost optimal high-probabilitybounds [13, 24, 29].
Generalization analysis of Neural Networks (NNs) .
Generalization analysis of NNs has attractedincreasing attention to understand their great success in practice.
A popular approach to studythe generalization of SNNs is via the uniform convergence approach, which studies the uniformgeneralization gaps in a hypothesis space [8, 25, 45].
However, this approach leads to capacitybased bounds which do not well explain why overparameterized models can still generalize wellto testing examples [44].
To address this problem, researchers turn to other approaches such as thecompression approach [4], the PAC-Bayes approach [21], the NTK approach [27] and the neuraltangent random feature approach [14].
The key idea of the NTK approach is that, under sufﬁcientoverparameterization and random initialization, the dynamics of GD on SNNs is close to the dynamicsof GD on a least-squares problem associated to the NTK [5, 20].
This leads to generalization boundsbased on a data-dependent complexity measure, which can distinguish the difference between learningwith random labels and learning with true labels [5].
Meanwhile, recent studies suggest the connectionto kernels might be only good at interpreting the performance of very wide networks [6, 50, 54], muchmore overparameterized than those used in reality [47].
The most related work is the recent analysis of
GD for SNNs without either the NTK condition or the PL condition [47].
They developed nontrivialgeneralization bounds under an overparameterization assumption m (cid:38) ( ηT ) 5 .
Furthermore, theiranalysis allows for improved bounds if there is no label noise, and shows an interesting connection to
NTK-based risk bounds.
It should be mentioned that the analysis in [5] considers the ReLU activationfunction, while the discussions in [47] focus on smooth activation functions.
Problem Setup
Let P be a probability distribution deﬁned on a sample space Z :=
X × Y , where X ⊆ R d and
Y ⊆ R .
Let S
=
{ z i = ( x i , y i ) : i = 1 , . . . , n } be a sample drawn from P .
Based on S we wish tobuild a model f : X (cid:55)→ R .
The performance of f can be measured by the population risk deﬁned as
[Equation]
which is unknown and can be approximated by the empirical risk L S ( f )
=
[Equation]
n (cid:80)
[Equation]
f
[Equation]
A minimizer of the population risk is the regression function f ρ ( x ) =
E [ y | x ] , where E [ ·| x ] denotesthe conditional expectation given x .
In this paper, we consider a shallow neural network of the form
[Equation]
[Equation] where we ﬁx µ k ∈ { 1 √ m , − 1 √ m } , σ : R (cid:55)→ R is an activation function and W = ( w 1 , . .
. , w m ) ∈
R d × m is the weight matrix.
In the above formulation, w k denotes the weight of the edge connectingthe input to the k -th hidden node, and µ k is the weight of the edge connecting the k -th hidden nodeto the output node.
Here m is the number of nodes in the hidden layer and (cid:104)· , ·(cid:105) denotes the innerproduct operator.
For simplicity, we denote
L ( W ) =
L ( f W ) and L S ( W ) =
L S ( f W ) .
Let W ∗ =
arg min W L ( W ) .
We choose a minimizer of L ( W ) with the smallest norm.
The relativebehavior of a model W w.r.t.
W ∗ is quantiﬁed by the excess population risk L ( W ) − L ( W ∗ ) .
Wedenote by (cid:96) ( W ; z ) = 12 ( f W ( x ) − y ) 2 the loss function of W on a single example z = ( x , y ) .
Tworepresentative algorithms to minimize the empirical risk are GD and SGD.
Deﬁnition 1 (Gradient Descent) .
Let W 0 ∈ R
d
× m be an initialization point.
GD updates { W t } asfollows
[Equation]
where η > 0 is the step size and ∇ denotes the gradient operator.
Deﬁnition 2 (Stochastic Gradient Descent) .
Let W 0 ∈ R
d
× m be an initialization point.
SGD updates{ W t } as follows
[Equation]
where i t is drawn from the uniform distribution over [ n ] := { 1 , . .
. , n } .
We are interested in the excess population risk of models trained by GD/SGD with T iterations.
Webegin with the introduction of some assumptions on activations and loss functions.
Assumptions 1, 2were also imposed in [47].
We denote by (cid:107) · (cid:107) 2 the Frobenius norm.
Assumption 1 (Activation) .
The activation φ ( u ) is continuous and twice differentiable with constant
B φ , B φ (cid:48) , B φ (cid:48)(cid:48) > 0 bounding | φ ( u ) | ≤ B φ , | φ (cid:48) ( u ) | ≤ B φ (cid:48) and | φ (cid:48)(cid:48) ( u ) | ≤ B φ (cid:48)(cid:48) for any u ∈ R .
Activation functions satisfying Assumption 1 include sigmoid and hyperbolic tangent activations [47].
Assumption 2 (Inputs, labels, and the loss function) .
There exists constants C x , C y , C 0 > 0 suchthat (cid:107) x (cid:107) 2 ≤ C x , | y | ≤ C y and (cid:96) ( W 0 ; z ) ≤ C 0 for any x , y and z .
Our third assumption is on the regularity of the learning problems.
For any λ > 0 , we deﬁne
[Equation]
Assumption 3 (Regularity) .
Assume there exist α ∈ (0 , 1] and c α > 0 such that
[Equation]
Assumption 3 is related to the approximation error which characterize how well the SNNs approximatethe least population risk [18, 53].
If (cid:107) W ∗ (cid:107)
=
O (1) , then it is clear that
[Equation]
and therefore Assumption 3 holds with α = 1 .
Our analysis is based on the following errordecomposition of the excess risk:
[Equation]
where we have used E
[ L S ( W ∗ 1 ηT )]
=
L ( W ∗ 1 ηT ) due to the independence between W ∗ 1 ηT and S .
Werefer to the ﬁrst term E
[ L ( W T )] − E
[ L S ( W T )] as the generalization error (generalization gap)and the second term E (cid:2)
L S ( W T ) − L
[Equation]
W
[Equation]
[Equation]
−
[Equation]
(cid:107) W ∗ 1 ηT − W 0 (cid:107) 22 (cid:3) as the optimization error.
As in [47], we will use the on-average model stability to control the generalization error and toolsin optimization theory to control the optimization error.
We will use Assumption 3 to control thelast term L ( W ∗ 1 ηT ) + 1 ηT (cid:107) W ∗ 1 ηT − W 0 (cid:107) 22 − L ( W ∗ ) .
The on-average model stability considers thesensitivity of the output models up to the perturbation of a single example, and the sensitivity isaveraged by traversing the single example throughout the sample set.
Let A ( S ) be the output modelby applying an algorithm A to S .
Deﬁnition 3 (On-average Model Stability [34]) .
Let S
=
{ z 1 , . . .
, z n } and S (cid:48)
= { z (cid:48) 1 , . . .
, z (cid:48) n } bedrawn independently from P .
For any i ∈ [ n ] , deﬁne S ( i ) =
{ z 1 , . . .
, z i − 1 , z (cid:48)
i , z i +1 , . . .
, z n } asthe set formed from S by replacing the i -th element with z (cid:48)
i .
We say a randomized algorithm A ison-average model (cid:15) -stable
if E S,S (cid:48) ,A (cid:2) 1 n (cid:80) ni =1 (cid:107) A ( S ) − A ( S ( i ) )
(cid:107) 22 (cid:3) ≤ (cid:15) 2 .
The connection between the generalization error and the on-average model stability was establishedin the following lemma.
Deﬁnition 4.
We say a function W (cid:55)→ g ( W ) is ρ -smooth if, for any W and W (cid:48) , we have
[Equation]
Lemma 1 (Stability and Generalization [34]) .
Let A be an algorithm.
If for any z , the map W (cid:55)→(cid:96) ( W ; z ) is ρ -smooth and nonnegative, then
[Equation]
[-(cid:16) 
-]
Main Results
In this section, we present our main results on the risk bounds of GD and SGD which are summarizedin Table 1.
[Table]
overparameterization
Computation
GD [47]
O ( n − α 1+ α )
No m (cid:38) ( ηT ) 5 (cid:16) n 5 α +
O ( n α +2 α +1 )
O ( n − α )
Yes m (cid:38) ( ηT ) 5 (cid:16) n
O ( n 2 )
GD
O ( n − α 1+ α )
No m (cid:38) ( ηT ) 3 (cid:16) n 3 α +
O ( n α +2 α +1 )
This work
O ( n − α )
Yes m (cid:38) ( ηT ) 3 (cid:16) n
O ( n 2 )
SGD
O ( n − α 1+ α )
No m (cid:38) ( ηT ) 3 (cid:16) n 3 α +
O ( n )
This work
O ( n − α )
Yes m (cid:38) ( ηT ) 3 (cid:16) n
O ( n )
In particular, if α = 1 , our results indicate both GD and SGD for 2-layer SNNs with subquadratic overparametrization m (cid:38) n 3 / 2 can lead to optimal risk rate O ( n − 1 / 2 ) while the results in [46] need superquadratic overparametrization m (cid:38) n 5 / 2 .
4.1 Gradient Descent
We ﬁrst study the excess risk of the GD algorithm for SNNs.
To this end, we ﬁrst develop itsgeneralization bounds.
We denote B (cid:16) B (cid:48) if there exist some universal constants c 1 and c 2 > 0such that c 1 B ≤ B (cid:48) ≤ c 2 B .
We denote B (cid:38) B (cid:48)
if there exists a universal constant c > 0 such that
B ≥ cB (cid:48) .
Let e be the base of the natural logarithm.
Let ρ
=
C 2 x (cid:0) B 2 φ (cid:48)
+ B φ (cid:48)(cid:48) B φ + B φ (cid:48)(cid:48) C y √ m (cid:1) andb = C 2 x B φ (cid:48)(cid:48)
( B φ (cid:48)
C x + C 0 ) .
Theorem 2 (Generalization Error) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) and
[Equation]
(4.1) then the following inequality holds for any t ∈
[ T ]
[Equation]
Remark 1.
Under an assumption m (cid:38) ( ηT ) 3 , a bound similar to Theorem 2 was recently establishedin [47].
We relax this assumption to m (cid:38) ( ηT ) 5 /n 2 + η 2 T 2 in Eq.
As we will show, a typicalchoice is ηT (cid:16) n 11+ α .
In this case, the assumption in Eq.
(4.1) becomes m (cid:38) ( ηT ) 3 n − 2 α
+ α + η 2 T 2 ,which is milder than the assumption m (cid:38) ( ηT ) 3 in [47].
This improvement is achieved by a better estimation of the smallest eigenvalue of a Hessian matrix.
Indeed, the smallest eigenvalue at α W t
+(1 − α ) W ( i ) t is lower bounded by − 1 √ m ( (cid:107) W t − W ( i ) t (cid:107) 2 +1)
(up to a constant factor), where α ∈ (0 , 1)and { W ( i ) t } is the SGD sequence on S ( i ) .
The analysis [47] uses (cid:107) W t − W ( i ) t (cid:107)
= O ( √ ηt ) tocontrol the smallest eigenvalue.
As a comparison, we show (cid:107) W t − W ( i ) t (cid:107)
= O ( n − 1 ( ηt ) 32 ) (Lemma
B.1).
A key step to relax the overparameterization is to build a bound on E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ] .
Theexisting analysis shows that (cid:107) W t − W 0 (cid:107)
= O ( ηt ) [47], which grows to inﬁnity as we runmore and more iterations.
In the following lemma to be proved in Section B.2, we improve it to
E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ]
=
O ( η 2 Tn (cid:80) T − 1 j =
E [ L S ( w j )]
+ (cid:107) W ∗ 1 ηT − W 0 (cid:107) 22 ) .
In particular, if ηT = O ( √ n )and (cid:107) W ∗ 1 ηT − W 0 (cid:107)
= O (1) , this bound becomes E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ]
= O (1) .
This explains whywe relax the overparameterization assumption from m (cid:38) ( ηT ) 5 in [47] to m (cid:38) ( ηT ) 3 .
Furthermore,the bound involves (cid:80) T − 1 j =
E
[ L S ( W j )] which would improve if the training errors are small, whichis critical to get fast rates in a low noise case.
Our basic idea to prove Lemma 3 is to ﬁrst control 199 E [ (cid:107) W t − W ∗ 1 ηT (cid:107) 22 ] in terms of training errors.
Our novelty is to replace these training errors withtesting errors by using Theorem 2, which allows us to use Eq.
(4.2) to remove some terms.
The proofis given in Section B.3.
For simplicity we assume (cid:107) W ∗ 1 ηT − W 0 (cid:107) 2 ≥ 1 .
Lemma 3.
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) , Eq.
(4.1)holds,
[Equation]
and
[Equation]
then for any t ∈ [ T ] we have
[Equation]
Remark 2.
We impose the assumption E
[ L ( W s )] ≥ L ( W ∗ 1 ηT ) , ∀ s ∈ { 0 , 1 , . .
, T − 1 } .
If thisassumption does not hold, then Assumption 3 implies further
[Equation]
This shows the violation of Eq.
(4.2) already implies a model W t , t ∈ [ T ] with a very small excessrisk, and therefore the assumption Eq.
(4.2) does not essentially affect our results.
Remark 3.
It should be mentioned that if (cid:107) W ∗ (cid:107)
=
O (1) we can derive similar results by replacing
W ∗ 1 ηT in the analysis with W ∗ (note W ∗ already satisﬁes the inequality L ( W ∗ ) − L ( W ∗ ) +1 ηT (cid:107) W ∗ − W 0 (cid:107)
= O (1 / ( ηT ))
and therefore can play the role of W ∗ 1 ηT ).
In this case, we no longerrequire the assumption (4.2).
Indeed, Eq.
(4.2) always holds with W ∗ 1 ηT replaced by W ∗ due to theinequality L ( W s ) ≥ L ( W ∗ ) .
Both bounds in Theorem 2 and Lemma 3 depend on the term (cid:80) T − 1 s =0 E (cid:2)
L S ( W s ) (cid:3) , for which weprovide a bound in the following lemma.
Combined with with Assumption 3, it implies (if m (cid:38) η 3 T 3 )
[Equation]
If L ( W ∗ ) = 0 , we have (cid:80) T − 1 s =0 E (cid:2)
L S ( W s ) (cid:3) =
O ( T ( Tη ) − α ) , which explains why we can getimproved bounds in a low noise case.
The proof is given in Section B.3.
Lemma 4.
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1) with η ≤ 1 / (2 ρ ) .
If Eq.(4.1) , (4.2) , (4.3) hold and
[Equation]
then
T − 1 (cid:88)
[Equation]
Now we present the optimization error bounds for GD.
Recall R T is deﬁned in Lemma 3.
The proofis given in Section B.3.
Theorem 5 (Optimization Error) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1)with η ≤ 1 / (2 ρ ) .
If Eq.
(4.1) , (4.2) and (4.3) hold, then
[Equation]
Remark 4.
The following optimization error bounds were established in [47]
[Equation]
A key difference between the above bound and Theorem 5 is that Eq.
(4.5) involves a term ( ηT ) 32 √ m ,while Theorem 5 involves a term O (cid:16) √ ηTR T √ m (cid:17) .
If R T =
o ( ηT ) , then the optimization error boundsin Theorem 5 would be tighter than Eq.
Indeed, the analysis in [47] requires m (cid:38) ( ηT ) 5 to getthe following optimization error bounds
[Equation]
As a comparison, if R T =
O (1) , Theorem 5 requires the assumption m (cid:38) ( ηT ) 3 to derive
[Equation]
We can combine the above discussions on generalization and optimization error bounds together toderive the following excess risk bounds.
Note the right-hand side of Eq.
(4.1), (4.3) and Eq.
(4.4) areof the order of ( ηT ) 3 if ηT = O ( n ) .
The proofs of Theorem 6 and Corollary 7 are given in Section
B.4.
Theorem 6 (Excess Population Risk) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.(3.1) with η ≤ 1 / (2 ρ ) .
If ηT = O ( n ) , Eq.
(4.1) , (4.2) , (4.3) and Eq.
(4.4) hold, then
[Equation]
where Λ λ is deﬁned in Assumption 3. The bound in Theorem 6 was also obtained in [47] under the assumption m (cid:38) ( ηT ) 5 . As a directcorollary, we show that GD can achieve excess risk bounds of the order O ( n − α 1+ α ) in the generalcase, and bounds of the order O ( n − α ) in the case L ( W ∗ ) = 0 which is due to the incorporationof empirical risks in the generalization bounds.
The basic idea is to balance the optimization andgeneralization via early-stopping [28, 38, 39, 56, 57].
Similar bounds can be derived by the analysisin [47] under Assumption 3.
Corollary 7.
Let Assumption 3 hold and assumptions in Theorem 6 hold.
(a) If we choose ηT (cid:16) n 1 α +1 and m (cid:16) ( ηT ) 3 (cid:16) n 3 α +1 , then E [ L ( W T )] − L ( W ∗ ) =
O ( n − α 1+ α ) .(b)
If L ( W ∗ ) = 0 , choosing ηT (cid:16) n and m (cid:16) ( ηT ) 3 (cid:16) n 3 implies that E [ L ( W T )]
= O ( n − α ) .
Remark 5.
Other than the stability analysis [47], there are some discussions on the stability analysisfor nonconvex functions that can be applied to SNNs [15, 26, 35, 60].
The discussions in [26] usestep sizes η t =
O (1 /t ) to get meaningful stability bounds, which, however, is not sufﬁcient for agood convergence of optimization errors.
The discussions in [15, 35, 60] impose a PL condition, andtheir error bounds depend on a condition number which can be large in practice.
4.2 Stochastic Gradient Descent
In this section, we study the excess risk bounds of SGD.
As compared to GD, the analysis of
SGD is more challenging since several properties of GD do not hold for SGD.
For example, theanalysis in [47] relies critically on the monotonicity of the sequence { L S ( W t ) } , which does nothold for SGD.
Furthermore, the introduced randomness of { i t } increases the variance of the iterates,which increases the difﬁculty of controlling the norm of iterates.
We ﬁrst provide a crude boundon the SGD iterates to be proved in Section C.2, which would be useful for our analysis.
Withoutloss of generality we assume 4 TηC 0 ≥ 1 .
Let R (cid:48) T = max { 2 √ TηC 0 , (cid:107) W
[Equation]
1 ηT − W 0 (cid:107) 2 } and
[Equation]
Lemma 8 (Iterate Bound) .
Let Assumptions 1, 2 hold.
Let { W t } t be produced by SGD.
If η ≤ 1 / (2 ρ )and
[Equation]
then
[Equation]
We now develop stability and generalization bounds of SGD.
In particular, we are interested ingeneralization bounds incorporating the training errors in the analysis [31, 34, 47].
This shows howgood optimization would improve generalization, which is consistent with the analysis of SGD in aconvex setting [34].
Theorem 9 gives on-average model stability bounds, which imply generalizationbounds in Theorem 10.
The proofs of Theorem 9 and Theorem 10 are given in Section C.3.
Let S ( i )be deﬁned as in Deﬁnition 3.
Theorem 9 (Stability Bound) .
Let Assumptions 1, 2 hold.
Let { W t } t and { W ( i ) t } t be produced by
SGD with η ≤ 1 / (2 ρ ) on S and S ( i ) , respectively.
If Eq.
(4.6) holds and
[Equation]
then for any t ≤ T − 1 we have
[Equation]
Theorem 10 (Generalization Error) .
Let assumptions in Theorem 9 hold.
Let { W t } t be produced by
SGD with η ≤ 1 / (2 ρ ) .
Then
[Equation]
The iterate bound in Lemma 8 is a bit crude.
In the following lemma to be proved in Section C.4, weshow this bound can be improved if we consider bounds in expectation.
Let
[Equation]
If tη 2 = O (1) and t = O ( n ) , Lemma 11 shows ∆ t
= O ( (cid:107) W 0 − W ∗ 1 ηT (cid:107) 22 ) which is signiﬁcantlybetter than the bound O ( ηt ) in Lemma 8.
This allows us to get excess risk bounds under a relaxedoverparameterization.
Similar to the case with GD, this upper bound depends on the training errorsof SGD iterates.
Lemma 11.
Let Assumptions 1, 2 hold.
Let { W t } t be produced by SGD with η ≤ 1 / (2 ρ ) .
If Eq.(4.6) , Eq.
(4.7) and Eq.
(4.2) hold, then
[Equation]
We now consider the optimization error bounds of SGD for SNNs.
In the following theorem, we givea bound on the average of the optimization errors for the sequence of SGD iterates.
Recall that R (cid:48)
T isdeﬁned above Lemma 8.
Theorem 12 (Optimization Error) .
Let assumptions in Lemma 11 hold.
Then
[Equation]
Finally, we develop the excess risk bounds for SGD on SNNs.
Note Eq.
(4.8) can be satisﬁed bychoosing m (cid:16) ( ηT ) 3 since R (cid:48)
T
= O ( √ ηT ) , which matches the overparameterization requirement of
GD and improves the requirement m (cid:38) ( ηT ) 5 in [47].
The proofs of Theorem 13 and Corollary 14are given in Section C.5.
Theorem 13 (Excess Population Risk) .
Let Assumptions 1 and 2 hold.
Let { W t } be produced by(3.2) and Eq (4.2) hold.
If η ≤ 1 / (2 ρ ) ,
[Equation]
and T = O ( n ) then we have
[Equation]
Corollary 14.
Let Assumption 3 hold and assumptions in Theorem 13 hold.
We choose an appropriatem (cid:16) ( ηT ) 3 .
(a) We can choose η (cid:16) T − α
+ α and T (cid:16) n to get 1 T (cid:80) T − 1 t =
E [ L ( W t )] − L ( W ∗ ) =
O ( n − α 1+ α ) .(b)
If L ( W ∗ ) = 0 , we can choose T (cid:16) n and η (cid:16) 1 to get 1 T (cid:80) T − 1 t =
E [ L ( W t )]
= O ( n − α ) .
Remark 6.
By Corollary 14, SGD achieves excess risk bounds of the same order to that of GD in
Corollary 7.
An advantage of SGD over GD is that it requires less computation.
To illustrate this, letus consider the general case for example.
In this case, GD requires T (cid:16) n 11+ α to achieve the errorbound O ( n − α 1+ α ) .
Since GD requires O ( n ) gradient computations per iteration and therefore thetotal gradient computation complexity is O ( n 2+ α 1+ α ) .
As a comparison, SGD requires O ( n ) gradientcomputations and therefore saves the computation by a factor of O ( n 11+ α ) .
Note Corollary 7 considersthe risk for the last iterate, while Corollary 14 considers the average of risks for all iterates.
Theunderlying reason is that GD consistently decreases the training errors along the optimization process,while SGD does not enjoy this property.
Note that the overparameterization requirement becomesm (cid:16) n 3 α +1 and m (cid:16) n 3 in Part (a) and Part (b), respectively.
Conclusion
In this paper, we present stability and generalization analysis of both GD and SGD to train neuralnetworks.
Under a regularity assumption, we show both GD and SGD can achieve excess riskbounds of the order O ( n − αα +1 ) , which further improve to the order O ( n − α ) under a low noisecondition.
As compared to the existing stability analysis [47], we achieve our bounds under a relaxedoverparameterization assumption and extend the existing analysis on GD to SGD.
Our improvementis achieved by developing sharper bounds on norm of the GD/SGD iterate sequences.
There remain several interesting questions for further discussion.
The ﬁrst question is whether theoverparamterization requirement m (cid:38) ( ηT ) 3 can be further improved, and whether the overparameterization requirement can be independent of T .
Second, our analysis applies to SNNs with a smoothactivation function.
It would be very interesting to extend our analysis to SNNs with the ReLUactivation function.
A key challenge in this direction is to control the smallest eigenvalue of theassociated Hessian matrix [47].
Third, our bounds are stated in expectation.
It would be useful todevelop high-probability bounds to understand the robustness of the algorithm.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’scontributions and scope?
[Yes](b) Did you describe the limitations of your work?
[Yes](c)
Did you discuss any potential negative societal impacts of your work?
[N/A](d)
Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]2.
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes](
b ) Did you include complete proofs of all theoretical results?
[Yes]3.
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A]
[Table]
[Table]
you
[Table]
all the training details (e g., [Table] splits, hyperparameters,
[Table]
they
[Table] chosen)?
[N [Table]
[Table] ) Did you report [Table]
(e g., with respect to the random seed after running experi- [Table] ments multiple times)?
[N/A [Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[N/A]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]