Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networks
Abstract
While significant theoretical progress has been achieved, unveiling the generalization mystery of overparameterized neural networks still remains largely elusive.
In this paper, we study the generalization behavior of shallow neural networks (SNNs) by leveraging the concept of algorithmic stability.
We consider gradient descent (GD) and stochastic gradient descent (SGD) to train SNNs, for both of which we develop consistent excess risk bounds by balancing the optimization and generalization via early-stopping.
As compared to existing analysis on GD, our new analysis requires a relaxed overparameterization assumption and also applies to SGD.
The key for the improvement is a better estimation of the smallest eigenvalues of the Hessian matrices of the empirical risks and the loss function along the trajectories of GD and SGD by providing a refined estimation of their iterates.
Introduction
Neural networks have achieved remarkable success in solving large-scale machine learning problems in various application domains such as computer vision and natural language processing [33].
Firstorder methods such as gradient descent (GD) and stochastic gradient descent (SGD) are mainstream optimization algorithms for training neural networks due to their simplicity and efficiency [11, 33, 50].
Although the associated optimization problems are nonconvex and nonsmooth
GD/SGD can still find a model with a very small or even zero training error [16, 20, 34, 39, 64, 69].
At the same time, the models found by such first-order methods has demonstrated good generalization performance on test data despite neural networks are often highly overparameterized in the sense that the number of parameters is much larger than the size of training examples [1, 2, 5].
These surprising phenomena have triggered a surge of research activities in understanding the generalization ability of neural networks.
Generalization analysis typically uses complexity measures such as VC dimension, covering numbers or Rademacher complexities to develop capacity-dependent bounds [8, 9, 25, 42, 48], which, however, may not explain well the generalization of overparameterized neural networks.
Impressive alternatives have been proposed which include the compression approach [4], the norm-based analysis [8, 25], the PAC-Bayes analysis [21] and the neural tangent kernel (NTK) approach [5, 28].
In particular, the NTK approach shows that the overparameterization pulls the dynamic of GD on neural networks close to its counterpart on a kernelized machine with the least-square loss [5, 20], which shows how overparameterization can help both optimization and generalization.
However, this approach often requires a very high overparameterization to gain useful results [6, 55, 60].
The recent appealing work [51] presents a kernel-free approach to study how overparameterization would improve the generalization for shallow neural networks (SNNs).
Their basic tool is the algorithmic stability [12], which measures how the replacement of an observation would change the algorithm output.
The authors showed the excess risk of GD is controlled by an interpolating network with the shortest GD path from the initialization, which is able to recover the existing NTK-based risk bounds as an application.
This result is achieved under an overparameterization assumption m ≳ ( ηT ) 5 , where m is the number of hidden nodes, η is the learning rate (step size) and T is the number of iterations.
While this result is very interesting and impressive, the overparameterization requirement m ≳ ( ηT ) 5 may still be more restrictive than that used in practice.
Furthermore, the analysis in [51] is restricted to the case of the full-batch GD.
One natural question thus arises:
Can we relax the overparameterization requirement for GD in [51] and further establish the stability and generalization of SGD for neural networks?
In this paper, we provide an affirmative answer to the above question by establishing a refined stability analysis for the gradient methods (GD and SGD) for training SNNs.
Our contributions are summarized as follows.
We develop excess risk bounds for GD on SNNs under a relaxed overparameterization.
In more details, we show that GD can achieve the excess risk bounds of the order O (1 / √ n ) if m ≳ ( ηT ) 3 , where n is the sample size.
This improves the existing overparameterization condition m ≳ ( ηT )
Under a low noise condition, our excess risk bounds improve to O (1 /n ) .
One key technical novelty in relaxing the overparameterization condition for GD in [51] is to improve the existing bounds on the norm of iterate sequence { W t } .
As we soon show in Section 4.1 below, this improvement is achieved by a better estimation of the smallest eigenvalue of the Hessian matrix of the empirical risk.
Specifically, the analysis [51] uses ∥ W t
−
[Equation] [Equation]
[Equation]
) to lower-bound the smallest eigenvalue at α W t + (1 − α )
W ( i ) t by − 1 √ m ( ∥ W t − W ( i ) t
∥ 2 + 1) , where α ∈ (0 , 1) and { W ( i ) t } is an iterate sequence on a neighboring dataset.
As a comparison, we show ∥ W t − W ( i ) t ∥ 2 = O ( n − 1 ( ηt ) 32 ) which can be much better than O ( √ ηt ) if n is large.
Furthermore, our bound depends on the training errors and would improve in a low noise condition.
Under some specific cases, we can further show that E [ ∥ W t ∥ 22 ] =
O (1) , which is independent of the iteration number.
We extend our analysis to SGD under the relaxed overparameterization condition m ≳ ( ηT ) 3 .
As compared to GD, SGD has a computational advantage in the sense that it can achieve the same risk bounds with a less computational cost.
The key analysis of SGD relies on the estimation of the Hessian spectrum of the loss over the individual training datum.
This is more challenging than estimating the counterpart of the empirical risk of GD since several properties of GD do not hold for SGD such as the monotonicity of the objective functions along the optimization process.
To overcome this technical hurdle, we provide a refined analysis to control the bounds of the iterates of SGD which further leads to the estimation of the Hessian spectrum of the loss.
The remaining parts of the paper are organized as follows.
We present the related work in Sectionand illustrate the formulation of the problem in Section 3.
We present the main results in Sectionand sketch the idea of the proof in Section 5.
We conclude the paper in Section 6.
Related Work
In this section, we group the related work into two categories: the related work on stability analysis and the related work on generalization analysis of neural networks.
Stability and generalization .
As a fundamental concept in statistical learning theory, algorithmic stability considers how the perturbation of training examples would affect the output of an algorithm [53], which has a close connection to the learnability [46, 56].
The framework of using the concept of algorithmic stability to derive generalization bounds was established in an influential paper [12], where the uniform stability was introduced and was studied for regularization schemes.
Since then, various concepts of stability have been introduced to study the generalization gaps, including the hypothesis stability [12, 22], on-average stability [32, 56], Bayes stability [38], locally elastic stability [19] and argument/model stability [35, 43].
A very successful application of stability analysis is to use it to study SGD for smooth, Lipschitz and convex problems [26], which motivates a lot of follow-up studies on stochastic optimization [3, 10, 15, 31, 35, 36, 49].
The smoothness assumption in [26] was recently removed by taking very small step sizes [10, 35], while the convexity assumption was relaxed to a weak convexity assumption [52].
Under a Polyak-Lojasiewicz (PL) condition, it was shown that any algorithm converge to global minima would generalize without convexity conditions [15, 36].
The trade-off between stability and optimization was studied in [17].
Other than stochastic optimization, stability has found wide applications in structured prediction [44], meta learning [45], transfer learning [32], hyperparameter optimization [7], minimax problems [23, 37, 65] and adversarial training [61].
While most of the stability analysis imply generalization bounds in expectation, recent studies show that uniform stability can yield almost optimal high-probability bounds [13, 24, 30].
Generalization analysis of Neural Networks (NNs) .
Generalization analysis of NNs has attracted increasing attention to understand their great success in practice.
A popular approach to study the generalization of SNNs is via the uniform convergence approach, which studies the uniform generalization gaps in a hypothesis space [8, 25, 41, 48, 67].
However, this approach leads to capacity-based bounds which do not well explain why overparameterized models can still generalize well to testing examples [47].
To address this problem, researchers turn to other approaches such as the compression approach [4], the PAC-Bayes approach [21], the NTK approach [28] and the neural tangent random feature approach [14].
The key idea of the NTK approach is that, under sufficient overparameterization and random initialization, the dynamics of GD on SNNs is close to the dynamics of GD on a least-squares problem associated to the NTK [5, 20].
This leads to generalization bounds based on a data-dependent complexity measure, which can distinguish the difference between learning with random labels and learning with true labels [5].
Meanwhile, recent studies suggest the connection to kernels might be only good at interpreting the performance of very wide networks [6, 55, 60], much more overparameterized than those used in reality [51].
The most related work is the recent analysis of GD for SNNs without either the NTK condition or the
PL condition [51].
They developed nontrivial generalization bounds under an overparameterization assumption m ≳ ( ηT ) 5 .
Furthermore, their analysis allows for improved bounds if there is no label noise, and shows an interesting connection to NTK-based risk bounds.
It should be mentioned that the analysis in [5] considers the ReLU activation function, while the discussions in [51] focus on smooth activation functions.
Problem Setup
Let P be a probability distribution defined on a sample space Z :=
X × Y , where X ⊆ R d and
Y ⊆ R .
Let S
=
{ z i = ( x i , y i ) : i = 1 , . . . , n } be a sample drawn from P .
Based on S we wish to build a model f : X (cid:55)→ R .
The performance of f can be measured by the population risk defined as
[Equation]
which is unknown and can be approximated by the empirical risk L S ( f ) = 12 n (cid:80) ni
=1 (cid:0) f
( x i ) − y i (cid:1) 2 .
A minimizer of the population risk is the regression function f ρ ( x ) =
E [ y | x ] , where E [ ·| x ] denotes the conditional expectation given x .
In this paper, we consider a shallow neural network of the form
[Equation]
where we fix µ k ∈ { 1 √ m , − 1 √ m } , σ : R (cid:55)→ R is an activation function and W = ( w 1 , . .
. , w m ) ∈
R d × m is the weight matrix.
In the above formulation, w k denotes the weight of the edge connecting the input to the k -th hidden node, and µ k is the weight of the edge connecting the k -th hidden node to the output node.
Here m is the number of nodes in the hidden layer and ⟨· , ·⟩ denotes the inner product operator.
For simplicity, we denote
L ( W ) =
L ( f W ) and L S ( W ) =
L S ( f W ) .
Let W ∗ =
arg min W L ( W ) .
We choose a minimizer of L ( W ) with the smallest norm.
The relative behavior of a model W w.r.t.
W ∗ is quantified by the excess population risk L ( W ) − L ( W ∗ ) .
We denote by ℓ ( W ; z )
= 12 ( f W ( x ) − y )
2 the loss function of W on a single example z
= ( x , y ) .
Two representative algorithms to minimize the empirical risk are GD and SGD.
Definition 1 (Gradient Descent) .
Let W 0 ∈ R
d
× m be an initialization point.
GD updates { W t } by
[Equation]
where η > 0 is the step size and ∇ denotes the gradient operator.
Definition 2 (Stochastic Gradient Descent) .
Let W 0 ∈ R
d
× m be an initialization point.
SGD updates { W t } as follows
[Equation]
where i t is drawn from the uniform distribution over [ n ] := { 1 , . .
. , n } .
We are interested in the excess population risk of models trained by GD/SGD with T iterations.
We begin with the introduction of some assumptions on activations and loss functions.
Assumptions 1,were also imposed in [51].
We denote by ∥ · ∥ 2 the Frobenius norm.
Assumption 1 (Activation) .
The activation ϕ ( u ) is continuous and twice differentiable with constant
B ϕ , B ϕ ′ , B ϕ ′′ > 0 bounding | ϕ ( u ) | ≤ B ϕ , | ϕ ′ ( u ) | ≤ B ϕ ′ and | ϕ ′′ ( u ) | ≤ B ϕ ′′ for any u ∈ R .
Activation functions satisfying Assumption 1 include sigmoid and hyperbolic tangent activations [51].
Assumption 2 (Inputs, labels, and the loss function) .
There exists constants C x , C y , C 0 > 0 such that ∥ x ∥ 2 ≤ C x , | y | ≤ C y and ℓ ( W 0 ; z ) ≤ C 0 for any x , y and z .
Our third assumption is on the regularity of the learning problems.
For any λ > 0 , we define
[Equation]
Note we use the asterisk to differentiate W ∗ λ and the GD iterate W t .
Assumption 3 (Regularity) .
Assume there exist α ∈ (0 , 1] and c α > 0 such that
[Equation]
Assumption 3 is related to the approximation error which characterize how well the SNNs approximate the least population risk, which is motivated from the approximation analysis in kernel learning.
In more details, a typical assumption in kernel learning is min f L ( f ) − L ( f ∗ )+ λ ∥ f ∥ 2 K =
O ( λ α ) , where α ∈ (0 , 1] depends on the regularity of a target function f ∗ and ∥ · ∥ K denotes the norm in a reproducing kernel Hilbert space.
If ∥ W ∗ ∥ 2 = O (1) , then it is clear that
[Equation]
and therefore Assumption 3 holds with α = 1 .
Our analysis is based on the following error decomposition of the excess risk:
[Equation]
where we have used E
[ L S ( W ∗ 1 ηT )]
=
L ( W ∗ 1 ηT ) due to the independence between W ∗ 1 ηT and S .
We refer to the first term E
[ L ( W T )] − E
[ L S ( W T )] as the generalization error (generalization gap) and the second term E (cid:2)
L S ( W T ) − L S ( W ∗ 1 ηT ) − 1 ηT ∥ W ∗ 1 ηT − W 0 ∥ 22 (cid:3) as the optimization error.
As in [51], we will use the on-average model stability to control the generalization error and tools in optimization theory to control the optimization error.
We will use Assumption 3 to control the last term L ( W ∗ 1 ηT ) + 1 ηT ∥ W ∗ 1 ηT − W
∥ 22 − L ( W ∗ ) .
The on-average model stability considers the sensitivity of the output models up to the perturbation of a single example, and the sensitivity is averaged by traversing the single example throughout the sample set.
Let A ( S ) be the output model by applying an algorithm A to S .
Definition 3 (On-average Model Stability [35]) .
Let S
=
{ z 1 , . . .
, z n } and S ′ =
{ z ′ 1 , . . .
, z ′ n } be drawn independently from P .
For any i ∈ [ n ] , define S ( i ) =
{ z 1 , . . .
, z i − 1 , z ′
i , z i +1 , . . .
, z n } as the set formed from S by replacing the i -th element with z ′ i .
We say a randomized algorithm A is on-average model
ϵ -stable if E S,S ′ ,A (cid:2)
1 n (cid:80) ni
=
∥
A ( S ) − A ( S ( i ) )
∥ 22 (cid:3) ≤ ϵ 2 .
The connection between the generalization error and the on-average model stability was established in the following lemma.
We say a function W (cid:55)→ g ( W ) is ρ -smooth if, for any W and W ′ , we have
[Equation]
Lemma 1 (Stability and Generalization [35]) .
Let A be an algorithm.
If for any z , the map
W (cid:55)→ ℓ ( W ; z ) is ρ -smooth and nonnegative, then
[Equation]
Main Results
In this section, we present our main results on the risk bounds of GD and SGD which are summarized in Table 1.
We denote B ≍ B ′ if there exist some universal constants c 1 and c 2 > 0 such that c 1 B ≤ B ′ ≤ c 2 B .
We denote B ≳ B ′ if there exists a universal constant c > 0 such that B ≥ cB ′ .
Algorithm
[Table]
GD [51]
[Equation]
O
[Equation]
GD
[Equation]
n
[Equation]
This work
O ( n − α )
Yes
[Equation]
O
[Equation]
n
[Equation]
SGD
O
[Equation]
No
[Equation]
O
[Equation]
n
[Equation]
This work
O ( n
[Equation]
In particular, if α = 1 , our results indicate both GD and SGD for 2-layer SNNs with subquadratic overparametrization m ≳ n 3 / 2 can lead to optimal risk rate O ( n − 1 / 2 ) while the results in [46] need superquadratic overparametrization m ≳ n 5 / 2 .
4.1 Gradient Descent
We first study the excess risk of the GD algorithm for SNNs.
Let e be the base of the natural logarithm.
Let ρ
=
C 2 x (cid:0) B 2 ϕ ′
+ B ϕ
′′ B ϕ + B ϕ ′′ C y √ m (cid:1) and b = C 2 x B ϕ ′′ ( B ϕ ′
C x + C 0 ) .
Theorem 2 (Generalization Error) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) and
[Equation]
then for any t ∈ [ T ] we have
[Equation]
j
[Equation]
Remark 1.
Under an assumption m ≳ ( ηT ) 3 , a bound similar to Theorem 2 was established in [51].
We relax this assumption to m ≳ ( ηT ) 5 /n 2 + η 2 T 2 in Eq.
As we will show, a typical choice is ηT ≍ n 11+ α .
In this case, the assumption in Eq.
(4.1) becomes m ≳ ( ηT ) 3 n − 2 α
+ α + η 2 T 2 , which is milder than the assumption m ≳
( ηT ) 3 in [51].
This improvement is achieved by a better estimation of the smallest eigenvalue of a Hessian matrix.
Indeed, the smallest eigenvalue at α W t
+(1 − α ) W ( i ) t is lower bounded by − 1 √ m ( ∥ W t − W ( i ) t ∥ 2 + 1) (up to a constant factor), where α ∈ (0 , 1) and { W ( i ) t } is the SGD sequence on S ( i ) .
The analysis [51] uses ∥ W t − W ( i ) t ∥
= O ( √ ηt ) to control the smallest eigenvalue.
Instead, we show ∥ W t − W ( i ) t ∥ 2 = O ( n − 1 ( ηt ) 32 ) (Lemma B.1).
A key step to relax the overparameterization is to build a bound on E [ ∥ W t − W ∗ 1 ηT ∥ 22 ] .
The existing analysis shows that ∥ W t − W 0 ∥
=
O ( ηt ) [51], which grows to infinity as we run more and more iterations.
In the following lemma to be proved in Section B.1, we improve it to
E [ ∥ W t − W
[Equation]
= O
[Equation]
Tn
[Equation] [Equation]
w
[Equation]
)] +
[Equation]
W
[Equation]
− W 0 ∥ 22 ) .
In particular, if ηT = O ( √ n ) and ∥ W ∗ 1 ηT − W 0 ∥ 22 = O (1) , this bound becomes E [ ∥ W t − W ∗ 1 ηT ∥ 22 ]
= O (1) .
This explains why we relax the overparameterization assumption from m ≳ ( ηT ) 5 in [51] to m ≳ ( ηT ) 3 .
Furthermore, the bound involves (cid:80) T − 1 j =
E
[ L S ( W j )] which would improve if the training errors are small, which is critical to get fast rates in a low noise case.
Our basic idea to prove Lemma 3 is to first control E [ ∥ W t − W ∗ 1 ηT ∥ 22 ] in terms of training errors.
Our novelty is to replace these training errors with testing errors by using Theorem 2, which allows us to use Eq.
(4.2) to remove some terms.
The proof is given in Section B.2.
For simplicity we assume ∥ W ∗ 1 ηT − W
∥ 2 ≥ 1 .
Lemma 3.
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
If η ≤ 1 / (2 ρ ) , Eq.
(4.1) holds,
[Equation] [Equation]
and
[Equation]
then for any t ∈ [ T ] we have
[Equation]
Remark 2.
We impose the assumption E
[ L ( W s )] ≥ L ( W ∗ 1 ηT ) , ∀ s ∈ { 0 , 1 , . .
, T − 1 } .
If this assumption does not hold, then Assumption 3 implies further
[Equation]
This shows the violation of Eq.
(4.2) already implies a model W t , t ∈ [ T ] with a very small excess risk, and therefore the assumption Eq.
(4.2) does not essentially affect our results.
It should be mentioned that if ∥ W ∗ ∥ 2 = O (1) we can derive similar results by replacing W ∗ 1 ηT in the analysis with W ∗ (note W ∗ already satisfies the inequality L ( W ∗ ) − L ( W ∗ ) + 1 ηT ∥ W ∗ −
W 0 ∥
= O (1 / ( ηT ))
and therefore can play the role of W ∗ 1 ηT ).
In this case, we no longer require the assumption (4.2).
Indeed, Eq.
(4.2) always holds with W ∗ 1 ηT replaced by W ∗ due to the inequality
L ( W s ) ≥ L ( W ∗ ) .
It should be mentioned that the bound in Lemma 3 is stated in expectation.
Therefore, we cannot directly combine this bound and the uniform convergence analysis to derive generalization bounds.
Now we present the optimization error bounds for GD.
Recall R T is defined in Lemma 3.
The proof is given in Section B.2.
Theorem 4 (Optimization Error) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1) with η ≤ 1 / (2 ρ ) .
If Eq.
(4.1) , (4.2) and (4.3) hold, then
[Equation]
Remark 3.
The following optimization error bounds were established in [51]
[Equation]
A key difference between the above bound and Theorem 4 is that Eq.
(4.4) involves a term ( ηT ) 32 √ m , while Theorem 4 involves a term O (cid:16) √ ηTR T √ m (cid:17) .
If R T =
o ( ηT ) , then the optimization error bounds in Theorem 4 would be tighter than Eq.
Indeed, the analysis in [51] requires m ≳ ( ηT ) 5 to get the following optimization error bounds
[Equation]
As a comparison, if R T =
O (1) , Theorem 4 requires the assumption m ≳
( ηT ) 3 to derive
E
[Equation]
We combine the above discussions on generalization and optimization error bounds together to derive the following excess risk bounds.
Note the right-hand side of Eq.
(4.1), (4.3) and Eq.
(4.5) are of the order of ( ηT )
if ηT = O ( n ) and ∥ W ∗ 1 ηT − W
∥
=
O ( √ ηT ) .
The proofs of Theorem 5 and
Corollary 6 are given in Section B.3.
Theorem 5 (Excess Population Risk) .
Let Assumptions 1, 2 hold.
Let { W t } be produced by Eq.
(3.1) with η ≤ 1 / (2 ρ ) .
If ηT = O ( n ) , Eq.
(4.1) , (4.2) , (4.3) hold and
[Equation]
then
[Equation]
where Λ λ is defined in Assumption 3.
The bound in Theorem 5 was also obtained in [51] under the assumption m ≳
( ηT ) 5 .
As a direct corollary, we can use Assumption 3 to show that GD can achieve excess risk bounds of the order
O ( n − α 1+ α ) in the general case, and bounds of the order O ( n − α ) in the case L ( W ∗ ) =
which is due to the incorporation of empirical risks in the generalization bounds.
The basic idea is to balance the optimization and generalization via early-stopping [29, 39, 40, 58, 62, 63].
Similar bounds can be derived by the analysis in [51] under Assumption 3.
Corollary 6.
Let Assumption 3 hold and assumptions in Theorem 5 hold.
(a) If we choose ηT ≍ n 1 α +1 and m ≍ ( ηT ) 3 ≍ n 3 α +1 , then E [ L ( W T )] − L ( W ∗ ) =
O ( n − α 1+ α ) .
(b) If L ( W ∗ ) = 0 , choosing ηT ≍ n and m ≍ ( ηT ) 3 ≍ n 3 implies that E [ L ( W T )]
= O ( n − α ) .
Remark 4.
Other than the stability analysis [51], there are some discussions on the stability analysis for nonconvex functions that can be applied to SNNs [15, 26, 36, 68].
The discussions in [26] use step sizes η t =
O (1 /t ) to get meaningful stability bounds, which, however, is not sufficient for a good convergence of optimization errors.
The discussions in [15, 36, 68] impose a PL condition, and their error bounds depend on a condition number which can be large in practice.
A recent paper [27] studies SGD for one-hidden-layer ReLU network with L 2 regularization from the NTK perspective and derives the appealing minimax optimal rate under the assumption that m is sufficiently large (e.g., m is at least larger than O ( n 8 ) ).
However, it is hard to derive a direct comparison since we study one-hidden-layer network with a smooth activation function.
Furthermore, our result holds if η ≤ 1 / (2 ρ ) , which is independent of m and n and is outside of the NTK regime.
As a comparison, the analysis based on NTK [34] requires η ≤ 2 /λ max (Θ) , where Θ ∈ R ( md ) × ( md ) is an neural tangent kernel and therefore the learning rate there is very small.
4.2 Stochastic Gradient Descent
As compared to GD, the analysis of SGD is more challenging since several properties of GD do not hold for SGD.
For example, the analysis in [51] relies critically on the monotonicity of the sequence { L S ( W t ) } , which does not hold for SGD.
Furthermore, the introduced randomness of { i t } increases the variance of the iterates, which increases the difficulty of controlling the norm of iterates.
We first develop stability and generalization bounds of SGD.
In particular, we are interested in generalization bounds incorporating the training errors in the analysis [32, 35, 51].
This shows how good optimization would improve generalization, which is consistent with the analysis of
SGD in a convex setting [35].
Eq.
(4.7) gives on-average model stability bounds, which imply generalization bounds in Theorem 7.
The proof of Theorem 7 is given in Section C.2.
Without loss of generality we assume 4 TηC 0 ≥ 1 .
Let R ′ T = max { 2 √ TηC 0 ,
∥ W ∗ 1 ηT − W
∥ 2 } and b ′ =
C 2 x B ϕ ′′ (cid:0)
C x B ϕ ′ + √ 2 C 0 (cid:1) .
Let S ( i ) be defined as in Definition 3.
Theorem 7 (Stability and Generalization) .
Let Assumptions 1, 2 hold.
Let { W t } t and { W ( i ) t } t be produced by SGD with η ≤ 1 / (2 ρ ) on S and S ( i ) , respectively.
If
[Equation]
then for any t ≤ T − 1 we have
[Equation]
Furthermore, we have the following generalization bounds
[Equation]
We now consider the optimization error bounds of SGD for SNNs.
In the following theorem, we give a bound on the average of the optimization errors for the sequence of SGD iterates.
Recall that R ′ T is defined above Theorem 7.
Let ∆ t :=
max j =0 ,...,t E [ ∥ W j − W ∗ 1 ηT ∥ 22 ] for any t ∈ N .
Theorem 8 (Optimization Error) .
Let Assumptions 1, 2 hold.
Let { W t } t be produced by SGD with η ≤ 1 / (2 ρ ) .
If Eq.
(4.6) and Eq.
(4.2) hold, then
[Equation]
Finally, we develop the excess risk bounds for SGD on SNNs.
Note Eq.
(4.8) can be satisfied by choosing m ≍ ( ηT ) 3 since R ′ T
= O ( √ ηT ) , which matches the overparameterization requirement of
GD and improves the requirement m ≳ ( ηT ) 5 in [51].
The proofs of Theorem 9 and Corollaryare given in Section C.3.
Theorem 9 (Excess Population Risk) .
Let Assumptions 1 and 2 hold.
Let { W t } be produced by (3.2) and Eq (4.2) hold.
If η ≤ 1 / (2 ρ ) ,
[Equation]
and T = O ( n ) then we have
[Equation]
+ ηL ( W ∗ ) (cid:1)
Corollary 10.
Let Assumption 3 hold and assumptions in Theorem 9 hold.
We choose an appropriate m ≍ ( ηT ) 3 .
(a) We can choose η ≍ T − α 1+ α and T ≍ n to get 1 T (cid:80) T − 1 t =
E [ L ( W t )] − L ( W ∗ ) =
O ( n − α 1+ α ) .
(b) If L ( W ∗ ) = 0 , we can choose T ≍ n and η ≍ 1 to get 1 T (cid:80) T − 1 t =
E [ L ( W t )]
= O ( n − α ) .
Remark 5.
By Corollary 10, SGD achieves excess risk bounds of the same order to that of GD in
Corollary 6.
An advantage of SGD over GD is that it requires less computation.
To illustrate this, let us consider the general case for example.
In this case, GD requires T ≍ n 11+ α to achieve the error bound O ( n − α 1+ α ) .
Since GD requires O ( n ) gradient computations per iteration and therefore the total gradient computation complexity is O ( n 2+ α 1+ α ) .
As a comparison, SGD requires O ( n ) gradient computations and therefore saves the computation by a factor of O ( n 11+ α ) .
Note Corollary 6 considers the risk for the last iterate, while Corollary 10 considers the average of risks for all iterates.
The underlying reason is that GD consistently decreases the training errors along the optimization process, while SGD does not enjoy this property.
Note that the overparameterization requirement becomes m ≍ n 3 α +1 and m ≍ n 3 in Part (a) and Part (b), respectively.
Main Idea of the Proof 5.1 Gradient Descent
In this subsection, we sketch our idea on the proof on gradient descent.
Generalization errors .
The starting point of our proof is the following bound given in Lemma A.
[Equation]
To apply the above inequality, we need to give a lower bound of 1 − η ∥ W t − W ( i ) t (cid:13)(cid:13) 2 √ m .
The analysis in [51] uses the crude bound ∥ W t − W ( i ) t (cid:13)(cid:13) 2 ≤ ∥ W t − W
(cid:13)(cid:13) 2 + ∥ W 0 − W
( i ) t (cid:13)(cid:13)
2 ≲ √ ηt, which does not use the fact that W t +1 and W ( i ) t +1 are produced by SGD on neighboring datasets.
By the generation of W t +1 and W ( i ) t +1 , we show that (cid:13)(cid:13) W t − W ( i ) t (cid:13)(cid:13)
=
O
(( ηt ) 32 /n )
(Lemma B.1).
This explains why we get a relaxed overparameterization in the stability analysis as compared to [51].
Optimization errors .
The starting point of our proof is the following bound given in Eq.
(B.8)
[Equation]
The analysis in [51] controls ∥ W ∗ 1 ηT − W s ∥ 32 as follows
[Equation]
As a comparison, we use ∥ W s − W 0 ∥ 2 = O ( √ ηs ) in Eq.
(5.1) and show that
[Equation]
can be bounded from above by
[Equation]
where we have used the overparameterization m ≳ ( ηT ) 3 .
It then follows that
[Equation]
Furthermore, we can apply stability analysis to relate E [ L S ( W s )] to E
[ L ( W s )] , and get (Lemma 3)
[Equation]
which is sharper than the bound ∥ W ∗ 1 ηT − W t ∥
= O ( √ ηt ) in [51].
This explains why we get a relaxed overparameterization in the optimization error analysis as compared to [51].
5.2 Stochastic Gradient Descent
Our starting point is to prove ∥ W t − W 0 ∥
=
O ( √ ηT ) for t ∈ [ T ] .
This was shown for GD in [51].
However, the analysis there relies heavily on the following inequality L S ( W j +1 ) ≤
[Equation]
, which does not hold for SGD.
We use the induction strategy to show ∥ W t − W 0 ∥ =
O ( √ ηT ) .
If ∥ W t − W 0 ∥ =
O ( √ ηT ) , Lemma A.1 implies λ min ( ∇ 2 ℓ ( W t ; z ))
≳ − √ ηT √ m .
If m ≳ ( ηT ) 3 we can use the update strategy of SGD and the induction assumption to show ∥ W t +1 − W
∥ =
O ( √ ηT ) .
The bound ∥ W t − W 0 ∥
=
O ( √ ηT ) is a crude estimate of the norm of iterates.
To get our results, we show the following sharper bound on the norm of iterates by considering bounds in expectation (Lemma C.2)
[Equation]
To show this, we use E [ ∥ W t +1 − W ∗ 1 ηT ∥ 22 ] ≤ E [ ∥ W t − W
[Equation]
1 ηT ∥ 22 ]+ η 2 E
[ L S ( W t )]
+ η E (cid:2)
L S ( W [Equation] 1 ηT ) −
L S ( W t ) (cid:3) + η √ ηT √ m E [ ∥ W ∗ 1 ηT − W t ∥ 22 ] (Eq.
(C.5), up to a constant factor).
We take a summation of this inequality and use m ≳ ( ηT ) 3 to get
[Equation]
from which we get Eq.
The bound in Eq.
(5.2) requires to estimate (cid:80) tj =
E [ L S ( W j )] .
Our next step is then to control (cid:80) tj =
E [ L S ( W j )] as follows (Lemma C.3)
[Equation]
Conclusion
In this paper, we present stability and generalization analysis of both GD and SGD to train neural networks.
Under a regularity assumption, we show both GD and SGD can achieve excess risk bounds of the order O ( n − α α +1 ) , which further improve to the order O ( n − α ) under a low noise condition.
As compared to the existing stability analysis [51], we achieve our bounds under a relaxed overparameterization assumption and extend the existing analysis on GD to SGD.
Our improvement is achieved by developing sharper bounds on norm of the GD/SGD iterate sequences.
There remain several interesting questions for further discussion.
The first question is whether the overparamterization requirement m ≳ ( ηT ) 3 can be further improved, and whether the overparameterization requirement can be independent of T .
Second, our analysis applies to SNNs with a smooth activation function.
It would be very interesting to extend our analysis to SNNs with the ReLU activation function.
A key challenge in this direction is to control the smallest eigenvalue of the associated Hessian matrix [51].
Third, our bounds are stated in expectation.
It would be useful to develop high-probability bounds to understand the robustness of the algorithm.
Finally, our analysis requires early-stopping in a low noise-setting.
It would be very interesting to develop risk bounds in a low-noise setting without early-stopping [54].
Acknowledgement.
The authors are grateful to the anonymous reviewers for their thoughtful comments and constructive suggestions.
Yiming’s work is supported by NSF grants (IIS-2103450,
IIS-2110546 and DMS-2110836).
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[N/A] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[N/A] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[N/A] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]