The alignment property of SGD noise and how it helps select ﬂat minima: A stability analysis
Abstract
The phenomenon that stochastic gradient descent (SGD) favors ﬂat minima has played a critical role in understanding the implicit regularization of SGD.
In this paper, we provide an explanation of this striking phenomenon by relating the particular noise structure of SGD to its linear stability (Wu et al., 2018).
Speciﬁcally, we consider training over-parameterized models with square loss.
We prove that if a global minimum θ ∗ is linearly stable for SGD, then it must satisfy (cid:107) H ( θ ∗ ) (cid:107) F (cid:54) O ( √ B/η ) , where (cid:107) H ( θ ∗ ) (cid:107) F , B, η denote the Frobenius norm of Hessian at θ ∗ , batch size, and learning rate, respectively.
Otherwise,
SGD will escape from that minimum exponentially fast.
Hence, for minima accessible to SGD, the sharpness—as measured by the Frobenius norm of the
Hessian—is bounded independently of the model size and sample size.
The key to obtaining these results is exploiting the particular structure of SGD noise: The noise concentrates in sharp directions of local landscape and the magnitude is proportional to loss value.
This alignment property of SGD noise provably holds for linear networks and random feature models (RFMs), and is empirically veriﬁed for nonlinear networks.
Moreover, the validity and practical relevance of our theoretical ﬁndings are also justiﬁed by extensive experiments on CIFAR-10 dataset.
Introduction
Modern machine learning (ML) models are often operated with far more unknown parameters than training examples, a regime referred to as over-parameterization.
In this regime, there are many global minima, all of which have zero training loss but their test performance can be signiﬁcantly different [47].
Fortunately, it is often observed that SGD converges to those generalizable ones, even without needing any explicit regularizations [49].
This suggests there must exist certain “implicit regularization” mechanism at work [32, 14, 47, 6].
More mysteriously, SGD solutions often generalize better than gradient descent (GD) solutions [21, 38].
Therefore, the SGD noise must play a critical role in implicit regularization.
The most popular explanation is that SGD favors ﬂatter minima [21] and ﬂatter minima generalize better [16].
This ﬂat-minima principle has been extensively and successfully adopted in practice to tune the hyperparameters of SGD [44, 21] and to design new optimizers [17, 11, 43] for improving generalization.
Therefore, understanding how SGD noise biases SGD towards ﬂatter minima is of paramount importance, which is the main focus of this paper.
The works [44, 10, 48] show that SGD noise is highly anisotropic; [30, 41]
ﬁnd the magnitude of
SGD noise to be loss dependent.
Both structures are shown to be critical for SGD picking ﬂat minima.
However, these works all make unrealistic (even wrong) over-simpliﬁcations of SGD noise (see the related work section for more details) in their analysis.
In addition, instead of studying SGD, they all consider the continuous-time stochastic differential equation (SDE), which is a good modeling of
SGD only in ﬁnite time and small learning rate (LR) regime [25].
It is generally unclear how this
SDE modeling is relevant for understanding SGD with a large LR—a regime preferred in practice.
Consequently, these works only provide intuitive and empirical analyses, lacking a quantitative characterization of when and how SGD favors ﬂat minima.
Another line of works [46, 29] relate the selection bias of SGD to the dynamical stability .
In overparameterized case, all global minima are ﬁxed points of SGD but their dynamical stabilities can be very different.
At unstable minima, a small perturbation will drive SGD to leave away, whereas, for stable minima, SGD can stay around and even converge back after initial perturbations.
Thus SGD prefers stable minima over unstable ones.
Speciﬁcally, [46, 29] analyze the linear stability [1] of
SGD, showing that a linearly stable minimum must be ﬂat and uniform.
Different from SDE-based analysis, this stability-based analysis is relevant for large-LR SGD and is even empirically accurate in predicting the properties of minima selected by SGD [46, 20, 8].
In this work, we follow the linear stability analysis in [46, 29] but take the particular geometry-aware structure of SGD noise into consideration.
We establish a direct connection between linear stability and ﬂatness, which allows us to obtain a quantitative characterization of how the learning rate and batch affect the ﬂatness of minima accessible to SGD.
In contrast, [46, 29] have to introduce the another quantity: non-uniformity together with ﬂatness to characterize linear stability because of neglecting the noise structure.
Setup Let { ( x i , y i ) } ni =1 with x i ∈ R d , y i ∈
R K be the training set and f ( · ; θ ) with θ ∈ R p be our model.
The model size is deﬁned to be the number of parameters p and in this paper.
For simplicity, we will always assume K = 1 and the extension to the case of K > 1 is straightforward.
Let
L i ( θ ) = 12 | f ( x i ; θ ) − y i | 2 be the ﬁtting error at the i -th sample and L ( θ ) =
1 n (cid:80) ni
=
L i ( θ ) be the empirical risk.
We will focus on the over-parameterized case in the sense that inf θ L ( θ ) = 0 .
To minimize L ( · ) , we consider the mini-batch SGD:
[Equation]
where η and B are the learning rate and batch size, respectively.
This SGD can be rewritten as θ t +
=
θ t − η ( ∇ L ( θ t )+ ξ t ) , where ξ t is the noise, satisfying E [ ξ t ] = 0 and E [ ξ t ξ Tt ]
= Σ( θ t ) /B.
Here the noise covariance Σ( θ )
= 1 n
[Equation]
L
[Equation]
θ ) T .
To characterize the local geometry of loss landscape, we consider the Gram matrix: G ( θ ) = 1 n
[Equation]
and the Hessian matrix: H ( θ ) =
G ( θ ) + 1 n (cid:80) ni
=
( f ( x i ; θ ) − y
[Equation]
) ∇ 2 f ( x i ; θ ) .
When the loss value is small, H ( θ ) ≈ G ( θ ) and in particular, if θ ∗ is an global minimum, H ( θ ∗ ) =
G ( θ ∗ ) .
Notations.
For a vector a , let (cid:107) a (cid:107) = √
a T a and (cid:107) a (cid:107) W = √ a T Wa .
For a matrix A , denote by { λ j ( A ) } the eigenvalue of A in a decreasing order.
For other notations, we refer to Appendix C.
Our main contributions are summarized as follows.
• We ﬁrst show that for many ML models, the SGD noise is geometry aware: 1) the noise magnitude is proportional to the loss value; 2) the noise covariance aligns well with the Gram matrix.
Speciﬁcally, to quantify the alignment strength, we deﬁne a loss-scaled alignment factor µ ( θ ) , which is proved to be bounded from below, i.e., there exists a size-independent positive constant µ 0 such that µ ( θ ) (cid:62) µ 0 , for linear networks (Proposition 2.3) and RFMs (Proposition 2.5), and is also empirically justiﬁed for nonlinear networks.
Moreover, we identify that it is the uniformity of model gradient norms {(cid:107)∇ f ( x i ; θ ) (cid:107) G ( θ ) } that mainly accounts for the alignment property of SGD noise.
• We then provide a thorough analysis of the linear stability of SGD by exploiting the alignment property of noise.
We prove in Theorem 3.3 that if a global minimum θ ∗ is linearly stable, then (cid:107) H ( θ ∗ ) (cid:107) F (cid:54) η −(cid:112) B/µ 0 .
Here the constant µ 0 quantiﬁes the alignment strength of SGD noise.
Hence, for minima accessible to SGD, the Hessian’s Frobenius norm—the ﬂatness perceived by SGD—is bounded independently of the model size and sample size.
Moreover, if a minimum is too sharp, violating the preceding stability condition, SGD will escape from it exponentially fast (Theorem 3.5).
Together, we obtain a quantitative characterization of when and how much SGD dislikes sharp minima.
• Our theoretical ﬁndings are also corroborated with well-designed experiments on a variety of models including linear networks, RFMs, convolutional networks, and fully-connected networks.
In particular, the practical relevance is demonstrated in Section 4 by extensive experiments on classifying full CIFAR-10 dataset with VGG nets and ResNets.
1.1 Related work
Noise structures.
[52, 19, 27] consider the Hessian-based approximation: Σ( θ )
≈ σ 2 H ( θ ) , where σ is a small constant.
[53] proposes an improved version: Σ( θ )
≈ 2 L ( θ ) H ( θ ) .
But these approximations in general cannot be accurate since Hessian is not semi-positive deﬁnite (SPD) in non-convex region.
More recently, [30] and [41] study SGD by assuming Σ( θ ) =
2 L ( θ ) H ( θ ∗ ) , where θ ∗ is a minimum of interest, and Σ( θ ) =
σ 2 L ( θ ) I p , respectively.
These assumptions completely ignore the state-dependence of noise direction.
In contrast, we assume Σ( θ ) =
2 L ( θ ) C ( θ ) with C ( θ ) having a nontrivial alignment with the Gram matrix G ( θ ) , which does not impose any explicit structural assumption on C ( θ ) .
As a result, our assumption is much weaker and can be rigorously justiﬁed for popular ML models both theoretically and empirically.
More importantly, we show that this weak alignment property is sufﬁcient for analyzing the linear stability of SGD.
We anticipate that our alignment assumption can be also adopted to analyze other properties of SGD.
Escape from sharp minima.
The escape behavior of SGD was ﬁrst studied in [52, 46], as an indicator of how much SGD dislikes sharp minima.
One of the most mysterious observation is that the escape happens in an unreasonably efﬁcient way.
However, the theoretical analysis there assumes the noise to be state-independent, and consequently, the derived escape time depends polynomially on the loss barrier.
Later [48, 30] attempt to study this issue using the classical diffusion-based framework [12] (Itô-SDE), which cannot explain the unreasonable escape efﬁciency at all since the resulting escape rates depend on the loss barrier exponentially.
See also [30] for an improved analysis.
[37, 51] argues that the SGD noise is heavy-tailed and thus SGD should be modeled as Lévy-SDE instead of
Itô-SDE.
Moreover, it is shown that the heavy-tailedness can ensure the escape rate depends on the basin volume instead of the loss barrier.
Unfortunately, the volume in high dimensions always scales with the dimension exponentially and consequently, this does not explain the escape efﬁciency in high dimensions.
Moreover, whether SGD noise is really heavy-tailed and whether the heavy-tailedness is really important for generalization are still debatable for neural networks [40, 26].
In contrast, we show that the unreasonable escape efﬁciency comes from the particular geometry-aware structure of
SGD noise, regardless of whether the noise is heavy- or light-tailed.
Flatness metrics In the literature, a variety of ﬂatness metrics have been adopted, such as the largest eigenvalue of Hessian [21], the trace of Hessian [9, 6], the basin volume [51], and the ones scaled by parameter norms [28, 39] in order to achieve the scaling-invariance for ReLU nets.
These metrics are proposed for either computation easiness or bounding generalization gaps.
It is unclear if they are perceivable to SGD, let alone how the boundedness of them depends on the batch size, learning rate, as well as the model size and sample size.
We show that for SGD solutions, the Frobenius norm of
Hessian—a ﬂatness perceived by SGD through the linear stability—is bounded by a size-independent quantity.
Note that a similar stability argument also applies to GD but only yielding the boundedness of the largest eigenvalue of Hessian [46, 31].
Lastly, we particularly mention the work [33], which provides a ﬁne-grained analysis of the implicit bias of training two-layer diagonal linear networks.
This work is related to ours since we both consider the magnitude and direction structure of SGD noise simultaneously.
However, the analysis in [33] is limited to the speciﬁc toy model but ours is relevant for general models.
The alignment property of SGD noise
Since ∇ L i ( θ ) = ( f ( x i ; θ ) − y i ) ∇ f
( x i ; θ ) , we have the following intuitive approximation [30]:
[Equation]
where ( i ) assumes that the full-batch gradient ∇ L ( θ ) to be negligible compared with the sample gradients {∇ L i ( θ ) } ; ( ii ) assumes that {∇ f ( x i ; θ ) } i and { L i ( θ ) } i are nearly decoupled.
This approximation cannot be true in general but tells us that 1)
The noise magnitude is proportional to the loss value; 2) the noise covariance aligns with the Gram matrix.
Motivated by (2), we deﬁne α ( θ ) =
[Equation]
F .
Here α ( θ ) quantiﬁes the similarity between Σ( θ ) and G ( θ ) , characterizing how much the noise concentrates in sharp directions of local landscape.
β ( θ ) characterizes the relative non-degeneracy of noise (with respect to the loss value).
Then we deﬁne the loss-scaled alignment factor:
[Equation]
which characterizes the direction and magnitude alignment simultaneously.
Intuitively speaking, if µ ( θ ) is bounded below, SGD noise is non-degenerate in sharp directions of local landscape.
In particular, α ( θ ) =
β ( θ )
= µ ( θ )
= 1 if the approximation (2) holds.
We say the noise satisﬁes the µ - alignment if µ ( θ ) > 0 .
Compared with the decoupling approximation (2), the µ -alignment is a much weaker condition.
Note that this speciﬁc weak quantiﬁcation of alignment is inspired for analyzing the linear stability of SGD, which is the focus of this paper.
Speciﬁcally, Theorem 3.3 shows that µ ( θ ) along with the Frobenius norm of Hessian determines the linear stability of SGD.
One may deﬁne other metrics to quantify alignment strength for studying other properties of SGD, but this is beyond the scope of this paper.
A relaxed alignment.
Let Σ 1 ( θ ) =
1 n
[Equation] [Equation]
=
[Equation]
θ
[Equation]
∇ L ( θ ) T .
Then Σ( θ ) =
Σ 1 ( θ ) − Σ 2 ( θ ) .
It is often believed that the full-batch gradient ∇ L is relatively small compared to the sample gradients {∇ L i } i .
As a result, the inﬂuence of Σ 2 ( θ ) should be negligible compared to Σ 1 ( θ ) .
To disentangle the inﬂuences of them, we deﬁne
[Equation]
Then µ ( θ ) = µ 1 ( θ ) − µ 2 ( θ ) .
Our linear stability analysis in Section 3 show that µ 1 ( θ ) (cid:62) µ 1 > 0 , a condition we refer to as µ 1 -alignment , is sufﬁcient to ensure that SGD only selects
ﬂat minima.
2.1 Why does
the alignment property holds?
Deﬁnition 2.1 (Norm uniformity of model gradients) .
Let g i ( θ ) =
∇ f
( x i ; θ ) , χ i ( θ ) :=
[Equation]
The quantity γ ( θ ) measures the uniformity of model gradient norms and this property can guarantee the µ 1 -alignment as stated below.
Lemma 2.2.
µ 1 ( θ ) (cid:62) γ ( θ )
Proof.
Noticing ¯ χ ( θ ) = 1 n (cid:80)
[Equation]
θ
[Equation]
2 F , we have
[Equation]
Thus µ 1 ( θ )
=
Tr(Σ 1 ( θ ) G ) / (2 L ( θ ) (cid:107) G (cid:107) 2 F ) (cid:62) γ ( θ ) .
The above proof suggests that the “decoupling” approximation in (2) holds in a weak sense if {(cid:107)∇ f ( x i ; θ ) (cid:107) G ( θ ) } i are uniform.
One can apply a similar argument by assuming the uniformity of the ﬁtting errors { L i ( θ ) } , which, unfortunately, we ﬁnd never hold in practice.
In contrast, we will show that the norm uniformity of model gradients provably holds for linear networks and RFMs, and can be empirically justiﬁed for nonlinear networks.
Over-parameterized linear models.
Consider an over-parameterized linear model (OLM): f ( x ; θ ) =
F ( θ ) T x , where F : Ω
(cid:55)→ R d denotes a general re-parameterization function.
Note that f ( · ; θ ) only represents linear functions but the corresponding landscape can be highly nonconvex.
Typical examples include the linear network: F ( θ )
=
W 1 W 2 · · · W L and the diagonal linear network: F ( θ ) =
( α− β 21 , . . .
, α 2 d − β 2 d ) T .
Both have attracted a lot of attention in analyzing the implicit bias of GD and SGD [3, 42, 33, 13, 4].
The following proposition provides a precise characterization of the noise covariance for OLM models, whose proof is deferred to Appendix D.
Proposition 2.3.
Denote by N (0 , S ) the Gaussian distribution with mean zero and covariance matrix S .
Suppose f ( · ; θ ) is a general OLM and x ∼ N (0 , S ) .
Consider the online SGD setting, i.e., n = ∞ .
Then, Σ( θ ) = ∇ L ( θ ) ∇ L ( θ ) T + 2 L ( θ ) G ( θ ) and µ 1 ( θ ) (cid:62) µ ( θ ) (cid:62)
This proposition shows that the alignment property holds in the entire parameter space and moreover, the alignment strength is independent of model size.
Here the alignment is only proved for the inﬁnite-sample case.
Similar results should hold for ﬁnite-sample cases by concentration inequalities as long as n is relatively large, but this straightforward extension does not bring any new insights.
It is more interesting to consider the low-sample regime (i.e., n < d ), where the alignments indeed hold (at least in typical regions explored by SGD) as demonstrated empirically in Figure 1.
In addition, the above proposition provides a closed-form expression of the noise covariance, which might be useful for analyzing other properties of SGD beyond the linear stability.
A comprehensive analysis of these issues is left to future work.
Feature-based models.
Consider a feature-based model f ( x ; θ ) = (cid:80) mj
=
[Equation]
j ϕ j
[Equation]
x ) = (cid:104)
[Equation]
, Φ( x ) (cid:105) .
In this case, the model gradients: g i =
g i ( θ ) =
Φ( x i ) and the Hessian and Gram matrix: G = H =n (cid:80) ni
=
g i g
Ti all are constant.
But the noise covariance Σ( θ ) is still state-dependent.
The norm uniformity of model gradients also becomes constant: γ = min i χ i / ( 1 n (cid:80)
[Equation]
i
[Equation]
with χ i = (cid:107) g
i (cid:107) 2 G .
[Equation]
The above lemma suggests that the µ 2 ( θ ) term is negligible as long as the Gram matrix is not nearly rank-1.
By bounding the γ and τ ( G ) , we can prove the µ 1 - and µ -alignment for random ReLU feature models as stated in the following proposition.
The proof is presented in Appendix F, where similar results for general RFMs are provided (see Proposition F.7).
Proposition 2.5.
If ϕ j ( x ) = ReLU( w Tj x ) with w j iid ∼ Unif( √ d S d − 1 ) and x ∼ Unif( S d − 1 ) .
Then, for any δ ∈ (0 , 1) , if n (cid:38) d 5 log(1 /δ )
, then w.p. at least 1 − δ , µ 1 ( θ ) (cid:62) 1 and µ ( θ ) (cid:38) d − 1 .
Although feature-based models are linear, their analysis is still applicable to understand nonlinear models, as long as the nonlinear model locally behaves like the linearized one: f lin ( x ; θ ) :
= f
( x ; θ ∗ )+ (cid:104) θ − θ ∗ , ∇ f ( x ; θ ∗ ) (cid:105) with ∇ f ( x ; θ ∗ ) learned from data.
Hence, Proposition 2.5 can explain why the alignment property holds in a local region around global minima.
Note that this is sufﬁcient for characterizing the linear stability of SGD, which is a local property in nature.
2.2 Empirical validations
Figure 1a reports the values of α ( θ t ) , β ( θ t ) , µ ( θ t ) during the SGD training of four types of models, including the linear networks and RFMs analyzed above, and fully-connected networks (FCN) and convolutional neural networks (CNN).
First, one can see that α ( θ t ) ’s are quite close to 1 during the entire training, suggesting the strong concentration of SGD noise in sharp directions of local landscape.
Second, β ( θ t ) ’s keep bounded below, implying the noise magnitudes are sufﬁciently large with respect to the training loss.
As a result, µ ( θ t ) ’s are signiﬁcantly positive for all models examined.
In particular, for the linear networks, the alignment holds in the low-sample regime, where n < d .
The size independence.
Figure 1b further examines how the extent of over-parameterization affects the alignment strength.
One can see clearly that for linear networks and RFMs, µ ( θ ) ’s are independent of the model size, which conﬁrms our theoretical ﬁndings proved above.
In addition, we also observe that for nonlinear networks, the alignment strength is also (nearly) independent of the model size.
For instance, for CNN, the value of µ ( θ ) only decreases from around 1 .
05 to 1 .
as the model size is increased by more than two orders of magnitude.
[Figure] [Figure]
The norm uniformity of model gradients.
Figure 2 shows the norm uniformity of model gradients, where we report the values of γ ( θ ) in the entire SGD trajectory.
On can see that γ ( θ ) is indeed bounded below, implying that the SGD noise satisﬁes the µ 1 -alignment according to Lemma 2.2.
[Figure]
The experiments above suggest that for nonlinear networks, the alignment property is satisﬁed in typical regions explored by SGD, including the random initialization and the convergent region.
In contrast, for OLMs and RFMs, we in fact prove that the alignment property holds for the entire parameter space.
The linear stability analysis
Let θ ∗ be a global minimum of L ( · ) .
When θ t is close to θ ∗ , the local dynamical behavior of SGD can be characterized by linearizing the dynamics around θ ∗ :
[Equation]
[Equation]
where ∇ 2 L i ( θ ∗ )
= ∇ f ( x i ; θ ∗ )
∇ f ( x i ; θ ∗ ) T .
This corresponds to the local quadratic approximation of the loss L ( · ) or the local linearization of the model around
[Equation]
Speciﬁcally, (4) is exactly the SGD of training the linearized model (5).
Deﬁnition 3.1 (Linear stability) .
A global minimum θ ∗ is said to be linearly stable if there exists a
C > 0 such that it holds for the linearized dynamics (4) that E [ L ( ˜ θ t )]
(cid:54) C E
[ L ( ˜ θ 0 )] , ∀ t (cid:62) 0 .
It is well-known in dynamical system that the local behavior of the original nonlinear dynamics can be characterized by the linearized one if the local quadratic approximation is non-degenerate.
However, in over-parameterized case, the local quadratic approximation is degenerate in ﬂat directions.
Consequently, one may be concerned about the relevance of local quadratic approximation and the resulting linear stability analysis.
Fortunately, the stability in Deﬁnition 3.1 is particularly measured with the change of loss value.
Thus the instability mostly comes from noise perturbations in sharp directions and the alignment property guarantees that noise mostly concentrates in sharp directions.
Consequently, the ﬂat directions contribute little to the instability.
In sharp directions, the local quadratic approximation is always valid, thereby explaining the relevance of linear stability analysis.
The rigorous formulation of this intuition is left to future work and we instead resort to numerical experiments to demonstrate the validity in this paper.
For simplicity, we will use θ t to denote ˜ θ t ; let θ ∗ = 0 and g i = ∇ f ( x i ; θ ∗ ) .
For the linearized model f lin ( · ; θ ) , we have L
[Equation]
θ
[Equation]
, where we omit the dependence on θ ∗ for simplicity.
Note that the Gram and Hessian matrix are constant but the noise covariance Σ( θ ) =
[Equation]
n
[Equation]
Before considering the speciﬁc linearized SGD (4), we ﬁrst have a general result.
Lemma 3.2.
Consider a general SGD: θ t +1 =
θ t − η ( ∇ L ( θ t ) + ξ t ) for the linearized model (5) , where ( ξ t ) t (cid:62) 1 are any noises satisfying E [ ξ t ]
= 0 , E [ ξ t ξ Tt ]
= S ( θ t ) .
Then we have
[Equation] [Equation]
where ν ( θ ) =
Tr( HS ( θ )) / 2 and r ( θ ) (cid:62) 0 .
Moreover, if η (cid:54) 2 /λ 1 ( H ) , then r ( θ ) (cid:54) 1 .
Proof.
Using the fact E [ ξ t ] = 0 and E [ ξ t ξ Tt ]
= S ( θ t ) , we have
[Equation]
where r ( θ ) = 1 − 2 η θ T H
2 θ θ T Hθ
+ η 2 θ T H
3 θ θ T Hθ since ∇ L ( θ ) =
Hθ .
By Lemma G.2, r ( θ ) (cid:62) 0 and if η (cid:54) 2 /λ 1 ( H ) , then r ( θ ) (cid:54) 1 .
The two terms r ( θ t ) L ( θ t ) and η 2 v ( θ t ) denote the contributions from the full-batch gradient ∇ L ( θ t ) and the noise ξ t , respectively.
The stability is affected by both terms simultaneously.
It is well-known that if θ ∗ is linearly stable for GD, then λ 1 ( H ( θ ∗ ))
(cid:54) 2 /η
(see, e.g., [46, 31]).
This also holds for
SGD but SGD imposes a stricter condition because of the extra η 2 ν ( θ t ) term.
Speciﬁcally,
[Equation]
It is implied that the more S ( θ ) aligns with H , the more unstable that minimum is .
3.1 The linear stability
imposes size independent ﬂatness
constraints
For the mini-batch SGD, the following theorem characterizes how the batch size and learning rate affect the ﬂatness—as measured by the Frobenius norm of Hessian—of minima accessible to SGD.
Theorem 3.3.
Let θ ∗ be a global minimum that is linearly stable.
Denote by µ ( θ ) the alignment factors for the linearized SGD (4) and model (5) .
If µ ( θ ) (cid:62) µ 0 , then (cid:107) H ( θ ∗ ) (cid:107) F (cid:54) 1 η (cid:113)
Bµ 0 .
Proof.
By (8) and the deﬁnition of µ ( θ ) , we have
[Equation]
To ensure the stability, we must have µ 0 η 2 (cid:107) H (cid:107) 2 F /B (cid:54) 1 , leading to (cid:107) H (cid:107) F (cid:54) (cid:112)
B/µ 0 /η .
We have shown in Section 2 that µ 0 is (nearly) size-independent, and thus the obtained upper bound of ﬂatness is also (nearly) size-independent.
As a comparison, for GD, the linear stability can only ensure λ max ( H ( θ ∗ )) (cid:54) 2 /η .
This gives a bound of the Hessian’s Frobenius norm: (cid:107) H ( θ ∗ ) (cid:107) F (cid:54) 2 √ p/η , depending on the model size explicitly.
The comparison of two bounds partially explains why SGD tends to select ﬂatter minima than GD.
We show below that the µ -alignment can be further relaxed to the µ 1 -alignment.
The proof is similar to the one of Theorem 3.3 and deferred to Appendix G.
Proposition 3.4.
Under the setting of Theorem 3.3, if the noise of linearized SGD satisﬁes µ 1 ( θ ) (cid:62) µ 1 , then (cid:107)
[Equation]
(cid:54) min
[Equation]
When B (cid:29) 1 , the bound becomes B/ ( η
[Equation]
( B
[Equation]
µ
[Equation]
) ≈ (cid:112)
B/µ 1 /η , which is the same as the case of µ -alignment.
Thus the inﬂuence of ∇ L is indeed negligible compared with {∇ L i } i .
Note that the linear stability is local in nature and hence our analysis essentially only needs the µ 1 -alignment to hold locally around minima of interest.
Experiments in Figure 2 shows that γ ( θ ∗ ) is always bounded below, i.e., the norm uniformity of model gradients is satisﬁed.
Combining with
Proposition F.7, we can conclude that the alignment property assumed in Proposition 3.4 holds.
Tightness of our analysis.
In the analysis above, we only inspect the instability caused by the noise, with the full-batch gradient completely ignored.
Therefore, we anticipate that our bound is tighter in small-batch regime, where the noise term dominates the full-batch term.
We will see that our numerical experiments indeed conﬁrm this tightness in small-scale regime.
However, for obtaining the tightest bound, one may need to consider both components simultaneously; this is much more complicated and left to future work.
Numerical validations.
Figure 3a numerically shows how the Frobenius norm of Hessian (not only the upper bound) changes with extent of over-parameterization, where the trace of Hessian is also plotted for comparison.
One can see that the Frobenius norm indeed keeps almost unchanged as increasing the model size but the Hessian trace increases signiﬁcantly.
Figure 3b further shows the ratio between the Frobenius norm and our upper bound in the training process and two batch sizes are examined.
We have two observations.
First, the correctness of our bound holds for the entire
SGD trajectory, suggesting that the linear stability analysis is relevant for the entire training process.
Second, as expected, the theoretical bound is indeed tighter for the case with a smaller batch size.
[Figure]
3.2 SGD escapes from sharp minima exponentially fast
The following theorem shows that the pure noise-driven escape from a sharp minimum is exponentially fast , whose proof follows trivially from (9).
Theorem 3.5.
Under the setting of Theorem 3.3, if (cid:107) H ( θ ∗ ) (cid:107) F > 1 η (cid:113)
Bµ 0 , then the linearized SGD satisﬁes E [ L ( θ t )] (cid:62)
γ t 0 E [ L ( θ 0 )] with γ 0 =
η 2 µ 0 B (cid:107)
[Equation]
Hence, linearized SGD takes roughly log γ 0 (1 /ε ) steps to escape from a O ( ε ) -loss region to a O (1) - loss region.
The escape time depends on the loss barrier only logarithmically and is independent of the parameter space dimension.
Due to the local closeness between linearized SGD and the original
SGD, this partially explains the unreasonable escape efﬁciency of SGD for training big models.
In contrast, the escape rates of existing works [48, 52, 51, 30] are either exponentially slow with respect to the loss barrier or suffer from the curse of dimensionality.
Figure 4 shows the trajectories of SGD escaping from sharp minima.
It is demonstrated that the escape is indeed exponentially fast and speciﬁcally, 10 steps are enough for SGD escaping to a high-loss region for all the models examined.
In addition, we observe that the escape is still exponentially fast in the high-loss region, although our analysis only applies to a local region.
How can we explain this nonlocal escape behavior?
We leave the study of this interesting phenomenon to future work
[Figure]
3.3 The importance of the noise structure
The magnitude structure.
Theorem 3.5 together with its proof suggests that the loss dependence of noise magnitude is critical for obtaining the exponentially fast escape.
The intuition is as follows.
When θ t is perturbed by noise to θ t +1 where L ( θ t +1 ) >
L ( θ t ) , the noise magnitude becomes larger there and thus θ t +1 is easier to be perturbed to a larger-loss region.
This positive feedback drives
SGD to leave exponentially fast.
On the contrary, the following lemma shows that if the noise is uniformly bounded, the noise-driven escape is at most linear in time.
Lemma 3.6.
Under the setting of Lemma 3.2, assume η (cid:54) 2 /λ 1 ( H ) and E [ HS ( θ )]
(cid:54) 2 σ 2 .
Then E [ L ( θ t ) − L ( θ 0 )] (cid:54) η 2 σ 2 t.
We set η (cid:54) 2 /λ 1 ( H ) to avoid the exponential escape caused by the full-batch component.
Proof.
By Lemma G.2, when η (cid:54) 2 /λ 1 ( H ) , r ( θ ) (cid:54) 1 .
Thus Lemma 3.2 implies E [ L ( θ t +1 )]
(cid:54) E
[ L ( θ t )]
+ η 2 σ 2 , which implies E [ L ( θ t )]
(cid:54) E
[ L ( θ 0 )]
+ η 2 σ 2 t .
The direction structure.
We now turn to consider the impact of direction structure.
Consider general SGDs: θ t +1 =
θ t − η ( ∇ L ( θ t ) + ξ t ) with E [ ξ t ξ t ]
= S ( θ t ) /B for the linearized model (5).
We compare two type of (unrealistic) noises:
• Geometry-aware noise: S 1 ( θ ) =
2 L ( θ ) H .
• Isotropic noise: S 2 ( θ ) = 2 σ 2 L ( θ )
I p with σ 2 =
Tr( H ) /p .
Here
the value of σ 2 is chosen to ensure two types of noises have the same total variance for a fair comparison [
Note that p denotes the model size.
Analogous to Theorem 3.3, for the second isotropic SGD,
[Equation]
Hence, the instability decreases with the parameter-space dimension and the resulting ﬂatness constraint is Tr( H ( θ ∗ ))
(cid:54) √ pB/η , depending on the model size explicitly.
In contrast, for the ﬁst noise, Theorem 3.3 implies (cid:107) H ( θ ∗ ) (cid:107) F (cid:54) √ B/η , independent of the model size.
This difference can be intuitively explained as follows.
The isotropic noise wastes most energy in perturbing SGD along ﬂat directions, which barely affects the instability.
In contrast, the geometry-aware noise focuses most energy on perturbing SGD along sharp directions, causing much more instability.
Larger-scale experiments
We have provided small-scale experiments to justify the validity of our theoretical ﬁndings for a variety of ML models.
Here we turn to demonstrate the practical relevance by consider the classiﬁcation of the CIFAR-10 dataset [22] with VGG nets [36] and ResNets [15].
In training, all explicit regularizations are removed to keep consistent with our theoretical analysis.
More details of the experimental setup can be found in Appendix A.
The alignment property and escaping behavior.
Figure 5a reports the alignment strength of
SGD noise during training for VGG-19 and ResNet-110.
One can see that the alignment factors are signiﬁcantly positive and similar results are also observed for a variety of VGG nets and ResNets of different depths and can be found in Figure 8 of Appendix B.
On can see that the alignment strength is nearly independent of the model size.
Figure 6a shows the behavior of escaping from sharp minima for VGG-19 and ResNet-110.
One can still observe that the escape is exponentially fast and similar observation for other ResNets and VGG nets can be found in Figure 9 in the appendix.
These observations suggest that our theoretical ﬁndings also hold for this practical setting
[Figure]
The ﬂatness and upper bound.
Figure 6a reports the ﬂatness of convergent solution and the corresponding upper bound predicted by our theory for ResNets and VGG nets.
It is again observed that the ﬂatness is nearly independent of the model size.
A surprising observation in Figure 6a is that our upper bounds are rather tight, see, e.g., VGG-16 and VGG-19.
This tightness suggests that
SGD runs (nearly) at the edge of stability [46, 8].
Moreover, as expected, Figure 6b shows that the our bound becomes tighter as decreasing batch size.
This is consistent with what we observe in small-scale experiment in Figure 3b.
[Figure]
Conclusion
We provide a stability-based explanation of why SGD selects ﬂat minima.
Our current understanding is as follows.
1) For popular ML models, the SGD noise aligns very well with local landscape. 2)
This alignment property ensures that the ﬂatness of stable minima must be size-independent.
This understanding is made rigorous and quantitative by introducing a loss-scaled alignment factor to characterize the alignment strength and analyze the linear stability.
Obviously, many questions remains open.
For example, can we understand what roles the stability plays in the whole dynamic process of SGD instead of only around global minima?
Can we establish the connection between the Hessian’s Frobenius norm and generalization?
Can we provide a ﬁne-grained characterization of the noise structure and how the structure is related to implicit regularization of SGD?
We leave the discussion of these important questions to future work.
Acknowledgements
We thank Zhiqin Xu and the anonymous reviewers for helpful suggestions.
The work of Lei Wu is supported by a startup fund
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information on how to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or [N/A] .
You are strongly encouraged to include a justiﬁcation to your answer , either by referencing the appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section ?? .
• Did you include the license to the code and datasets?
[No] The code and the data are proprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Local analysis (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix A. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Appendix A. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Estimated (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] (b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] (d) Did you discuss whether
and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Experiment setup
We consider training the following models in the over-parameterized regime.
In training, all explicit regularizations (including weight decay, dropout, data augmentation, batch normalization, learning rate decay) are removed, and a simple constant-LR SGD is used to train our models.
Small-scale models:
In this case, we set the sample size to be particularly small on purpose, which allows interested readers to quickly reproduce these experiments on their own computer.
Note that this choice does not impact our conclusions since we also discuss in details the inﬂuence of changing the extent of over-parameterization and provide larger-scale experiments.
• Random feature model (RFM).
The inputs { x i } ni =1 are drawn from N (0 , I d ) with d = 10 , n = 200 .
The labels are generated by
[Equation]
The model is
[Equation]
x ;
[Equation]
mj =1 θ j ReLU( w Tj x ) with m = 2000 and { w j } mj =1 independently drawn from N (0 , I d ) at initialization and ﬁxed during the training.
This model is trained by SGD with learning rate η = 0 .
003 and batch size B = 5 .
• Linear networks.
The inputs { x i } ni =1 drawn are drawn from N (0 , I d ) with d = 100 , n = 50 .
Here we set n <
d to examine the low-sample regime.
The labels are generated by
[Equation]
di
=1 x i /d .
The model is a four-layer linear network: d → m → m → m → 1 with m = 50 .
This model is trained by SGD with learning rate η = 0 .
1 and batch size B = 5 .
The default LeCun initialization is used.
• Fully-connected networks (FCN) .
We randomly sample n data from the MNIST training set and label { 1 , 2 , 3 , 4 , 5 } to 0 and { 5 , 6 , 7 , 8 , 9 , 10 } to 1 to form our new training set.
The model is a ReLU-activated fully-connected network with the architecture:→ m → m → 1 .
Except for studying the inﬂuence of over-parameterization, we always set m = 30 , n = 1000 , and the model size is p = 25441 .
This model is trained by SGD with η = 0 .
05 , B =
and the LeCun initialization is used.
• Convolutional neural networks (CNN).
This is a small LeNet-type CNN [23] whose architecture is given in Table 1.
The training set is the same as the one constructed above for FCN.
The LeCun initialization is used and the model is trained by SGD with η = 0 .
1 , B = 5 .
[Table]
Larger-scale models: For these experiments, the full CIFAR-10 dataset are used to train our model
• VGG networks.
The models are the standard VGG nets (including VGG-11, VGG-13, VGG-16, and VGG-19) for classifying CIFAR-10 proposed in [36].
• ResNets.
The residual networks for CIFAR-10 proposed in [15] are considered and the models include ResNets-26, ResNet-38, ResNet-50, and ResNet-100.
For ResNets, we follow [50] to use the ﬁxup
initialization in order to ensure that the model can be trained without batch normalization.
Both VGG networks and ResNets are trained by SGD with learning rate η = 0 .
1 and batch size
B = 64 until the training loss becomes smaller than 10 − 4 .
Hyperparameter choices.
In all experiments, the default model size, sample size, learning rate, and batch size described above are used unless explicitly speciﬁed, e.g., studying the inﬂuence of over-parameterization.
Efﬁcient computations of the alignment factors and ﬂatness.
Let g i = ∇ f
( x i ; θ ) , e i = f
( x i ; θ ) − y i .
Let Q
= ( g 1 , . .
. , g n ) ∈ R p × n and S =
( e 1 g 1 , . . .
, e n g n ) ∈
R p × n .
Then,
[Equation]
where P = I n × n − 1 n 11 T ∈ R n × n .
Here 1 ∈ R n denotes the all-one vector.
Notice that the computation of α ( θ ) , β ( θ ) , µ ( θ ) , (cid:107) H ( θ ) (cid:107) F can be reduced to computing (cid:107) G (cid:107) F , (cid:107) Σ (cid:107) F , and Tr( G Σ) .
The time complexity of naively computing them is on the order of O ( p 2 n ) , which is prohibitive for large-scale models where p (cid:29) n .
A more efﬁcient way is to use the following equations
[Equation]
where all the matrices are n × n .
Hence, using these equations, the computation complexity becomes
O ( n 2 p ) .
This is much smaller than O ( p 2 n ) when p (cid:29) n .
• For small-scale experiments, the equations in (10) are directly used.
• For the large-scale models, we need further approximations since the computation complexity O ( n 2 p ) is still prohibitive in this case.
Notice that the formulations in Eq.
(10) are all in the form of sample average, which allows us to perform Monte-Carlo approximation.
Speciﬁcally, we randomly choose B samples from x 1 , . . .
, x n and still use (10) to estimate these quantities.
But now the computation complexity becomes O ( B 2 p ) .
For the experiments on CIFAR-10, we test B ’s with different values and ﬁnd that B = 50 is sufﬁcient to obtain a reliable approximation of the original full-data quantity.
Hence, for all large-scale experiments in this paper, we use B = 50 to speed up the computation of alignment factors and ﬂatness.
We clarify that the models are still trained on the full dataset.
B Extra experiment results B.1 Small-scale experiments
Figure 7 shows the relative norm of model gradient, i.e., χ i ( θ ) / ¯ χ ( θ ) of each samples when the model convergences.
It is shown that γ ( θ ) = min i
χ i ( θ ) /
¯ χ ( θ ) is indeed bounded below.
This explains why the SGD noise satisﬁes the alignment property locally according to Lemma 2.4.
B.2 Larger-scale experiments
Figure 8 reports the values of alignment factors during the training for ResNet-26, ResNet-38, ResNet50, VGG-13, and VGG-19.
We see that the alignment factors are always signiﬁcantly positive.
In particular, for VGG-16, α ( θ ) is close to 1 and µ ( θ ) is roughly on the order of 0 . 1 .
In addition, Figure 9 shows that the noise-driven escape is indeed exponentially fast as suggested by our theoretical analysis.
[Figure]
[Figure] [Figure]
However
[Figure]
it should be stressed that the loss-scaled alignment factors for these larger-scale models are clearly smaller than the ones reported in small-scale experiments.
In particular, one can see that the values of µ ( θ ) for various ResNets are only the order of 0 .
To understand the underlying reason
[Figure]
we additionally examine the same models but for classifying a two-class subset of CIFAR-10 with the results reported in Figure 10.
It is shown that in this case, the loss-scaled alignment factors are roughly on the order of 0 .
1 for all the models, signiﬁcantly larger than the case of classifying the full CIFAR-10 dataset.
It is also very surprising to observe that for VGG networks, the standard alignment factor α ( θ ) ’s are nearly 1 , suggesting that the noise covariance completely aligns with the geometry of local landscape.
[Figure] [Figure]
C Notations.
For a vector v
[Figure]
let (cid:107) v (cid:107) p
=
( (cid:80) i v pi ) 1 /p
and ˆ v = v/ (cid:107) v (cid:107) 2 .
When p = 2 , we omit the subscript for simplicity.
For a matrix A
= ( a i,j ) , denote by (cid:107) A (cid:107) F = ( (cid:80)
i,j a 2 i,j ) 1 / 2 the Frobenius norm.
For a matrix or linear operator A , denote by { λ i ( A ) } i (cid:62) 1 the eigenvalues of A in a non-increasing order.
Let S d − 1 = { x ∈ R d | (cid:107) x
(cid:107) = 1 } , r S d − 1 = { x ∈ R d
| (cid:107) x
(cid:107) =
r } , and τ d − 1 =
Unif( S d − 1 ) .
For any distribution ρ , let (cid:107) f (cid:107)
L 2 ( ρ ) =
E x ∼ ρ [ f 2 ( x )] .
We use X (cid:46) Y to indicate X (cid:54) CY for an absolute constant C > 0 and X (cid:38) Y is deﬁned similarly.
Moreover, we use X ∼ Y to mean cY (cid:54) X (cid:62) CY for some absolute constants c, C > 0 .
We also use C to denote an absolute constant, whose value may change from line to line.
For a vector a and a SPD matrix W , let (cid:107) a (cid:107) W = √ a T Wa .
D Proof of Proposition 2.
We ﬁrst need the following lemma.
Lemma D.1.
Suppose z ∼ N (0 , S ) .
Then, E z [ | v T z | 2 zz T ]
= 2 Svv T S + (cid:107) S 1 / 2 v (cid:107) 2 S .
Proof.
First assume S = I d .
Notice that
[Equation]
if i
[Equation]
Therefore,
[Equation]
For general S , let x = S − 1 / 2 z .
Then x ∼ N (0 , I d ) .
Then we have
[Equation]
where the third step follows from (11).
Now we start to prove Proposition 2.3.
Let ∇ F : R p
(cid:55)→ R d be the Jacobian matrix of F .
Then ∇
[Equation]
x ; θ ) = ∇ F ( θ )
T x .
Recall that we assume y = f
( x ; θ ∗ ) , i.e., there is no label noise.
Thus, we have
[Equation]
where u ( θ ) =
F ( θ ) − F ( θ ∗ ) .
For the noise covariance, we have
[Equation]
and
[Equation]
where the third step follows from Lemma D.1.
Consequently, µ 1 ( θ ) (cid:62) µ ( θ ) (cid:62) 1 .
E Proof of Lemma 2.
Recall the deﬁnition of feature-based model: f ( x ; θ ) =
[Equation]
Here Φ( x ) ∈
R m denotes the feature of x .
In this case, the Hessian and Gram matrix are both constant but the
SGD noise is still state-dependent.
Let g i =
Φ( x i ) be the sample feature of x i .
Then,
[Equation]
Proof.
Noticing ¯ χ = 1 n (cid:80) ni
=
g Ti
[Equation]
i = (cid:107) G (cid:107) 2 F , then
[Equation]
Thus µ 1 ( θ ) (cid:62) γ .
In addition,
[Equation]
where the last step follows from Lemma G.1.
F Proof of Proposition
. F.1 Rademacher complexity
We shall use the Rademacher complexity to bound the difference between empirical quantities and the corresponding population ones.
Deﬁnition F.1 (Rademacher complexity) .
Let F be a set of functions.
The (empirical) Rademacher complexity of F is deﬁned as
[Equation]
, where { ξ i } ni =1 are i.i.d.
random variables satisfying
[Equation]
In particular, the following classic result will be frequently used, which is a restatement of [35,
Theorem 26.5].
Theorem F.2.
Consider a function class F and assume | f | (cid:54) B .
Then for any δ ∈ (0 , 1) , w.p. at least 1 − δ over the choice of ( z 1 , z 2 , . . .
, z n ) , we have,
[Equation]
Lemma F.3 (Contraction property [24]) .
Let φ : R (cid:55)→ R be β -Lispchitz continuous and φ ◦ F = { φ ◦ f : f ∈ F} .
Then,
[Equation]
Lemma F.4.
Let F = { u T x : u ∈ S d − 1 } be the linear class.
Then
[Equation]
Lemma F.5.
Let φ : X (cid:55)→ H be a feature map and H be a Hilbert space.
Deﬁne k
( x, y )
= (cid:104) φ x , φ y (cid:105) H and H =
{ f ( x ) =
(cid:104)
φ x , h (cid:105) H | (cid:107) h (cid:107) H (cid:54) 1 } .
Then, (cid:100)
Rad n ( H ) (cid:54) (cid:112)(cid:80) ni =1 k ( x i , x i
[Equation]
n .
Proof.
By the deﬁnition, we have
[Equation]
where ( i ) follows from the Jensen’s inequality.
For the proof of the above classic lemmas, we refer to [35].
The following lemma concerns the
Rademacher complexity of the product of two function classes.
Lemma F.6.
Let F and G be two function classes.
Suppose that sup f ∈F (cid:107) f (cid:107) ∞ (cid:54) A and sup g ∈G (cid:107) g (cid:107) ∞ (cid:54) B .
Deﬁne F ∗ G = { f ( x ) g ( x ) : X (cid:55)→ R : f ∈ F , g ∈ G} .
Then,
[Equation]
Proof.
By the deﬁnition of Rademacher complexity, n
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
where ( i ) follows from the Lemma F.3 and the fact that t 2 / 4 is ( A + B ) / 2 Lipschitz continuous since | f | (cid:54) A, | g | (cid:54) B .
F.2 Results for general random feature models
Consider a random feature model (RFM):
[Equation]
with { w j } mj =1 iid ∼ π and x ∼ ρ .
Here ϕ : X × Ω (cid:55)→ R is an arbitrary parametric feature function.
The scaling factor m − 1 / 2 is added only to ease taking the limit, which does not affect the noise structure.
The associated kernel and kernel operator are given by k ( x, x (cid:48) )
=
E w ∼ π [ σ ( w T x ) σ ( w T x (cid:48) )] and
K : L 2 ( ρ ) (cid:55)→ L 2 ( ρ ) , K u
= E x (cid:48) ∼ ρ [ k ( · , x (cid:48) )
u ( x (cid:48) )] .
In this case, g i = m − 1 / 2 ( ϕ ( x i ; w 1 ) , ϕ ( x i ; w 2 ) , . . .
, ϕ ( x i ; w m ))
T ∈ R m .
Deﬁne the empirical kernel matrix: ˆ k ( x, x (cid:48)
[Equation]
=
[Equation]
ms =1 ϕ ( x ; w s ) ϕ
( x (cid:48) ; w s ) .
Then g Ti g j = ˆ k
( x i , x j ) and as n, m → ∞ , we have
[Equation]
i
[Equation]
Therefore, in the limit, by our assumption χ i (cid:62) χ := inf x (cid:48)
E x [ k 2 ( x (cid:48) , x )] for any i ∈ [ n ] .
The remaining issue is to transfer this to a non-asymptotic one.
We ﬁrst make the following assumption on the feature function.
Assumption 1.
Let Ψ =
{ ϕ ( x ; · ) | x ∈ X} .
Assume sup x ∈X ,w ∈ Ω | ϕ
( x ; w ) | (cid:54) b and (cid:100)
Rad n (Ψ) (cid:54) b/ √ n .
Note the assumption that
[Equation]
Rad n (Ψ) =
O ( n − 1 / 2 ) is satisﬁed by the popular feature function ψ ( x ; w )
=
σ ( w T x ) with σ : R (cid:55)→ R being an activation function.
Proposition F.7.
Suppose m (cid:62) n .
Let χ ( x ) =
E x (cid:48) ∼ ρ [ k 2 ( x, x (cid:48) )] and ¯ χ = E x ∼ ρ
[ χ ( x )] .
For any δ ∈ (0 , 1 /e ) , w.p. larger than 1 − δ over the sampling of data and random features, we have µ 1 ( θ ) (cid:62) inf x χ ( x ) ¯ χ − (cid:15) n , µ 2 ( θ ) (cid:54) τ
( K ) + (cid:15) n , and µ ( θ ) (cid:62) inf x χ ( x ) ¯ χ − τ ( K ) − 2 (cid:15) n , where τ ( K ) =
λ 21 ( K ) / (cid:80)
j λ 2 j ( K ) and (cid:15) n = Cb(cid:113) log(1 /δ ) n .
By this proposition, ensuring the alignment property only needs inf x χ ( x ) / ¯ χ > 0 .
This condition is very mild and satisﬁed by most popular kernels, e.g., the dot-product kernel, which appears naturally in the analysis of neural nets [18].
Next we proceed to prove this proposition.
We ﬁrst need the following lemmas.
Lemma F.8.
Let Φ 1 = { k ( · , x ) | x ∈ X} .
Then,
[Equation]
Proof.
Denote by H k the reproducing kernel Hilbert space (RKHS).
Then, by the Moore-Aronsajn theorem [2],
[Equation]
Therefore, Φ 1 is a subset of the unit ball of H k , for which it is well-known that the Rademacher complexity is bounded by
[Equation]
Lemma F.9.
For any δ ∈ (0 , 1) , w.p. at least 1 − δ , we have
[Equation]
Proof.
Let Ψ 2 =
{ ϕ ( x ; · ) ϕ
( x (cid:48) ; · ) | x, x (cid:48) ∈ X} .
Notice that
[Equation]
Then by the same argument in the proof of Lemma F.6, we can obtain
[Equation]
b 2 / √ m , where the last step follows from Lemma 1.
Then, applying Theorem F.2, we complete the proof.
Prove the ﬁrst part of Proposition F.7 Notice that
[Equation]
Let Φ 2 = { k 2 ( x, · ) | x ∈ X} .
Since sup x,x (cid:48) | k
( x, x (cid:48)
[Equation]
| (cid:54) 1 , the Ledoux-Talagrand inequality (Lemma
F.3) implies that
[Equation]
Rad n
[Equation]
Φ
[Equation]
) (cid:54) 1 / √ n , where the last inequality follows from Lemma
F.8.
Then, applying Theorem F.2, for any δ ∈ (0 , 1 /e ) , we have w.p. 1 − δ that
[Equation]
Let ∆ =
sup x,x (cid:48) | k
( x, x (cid:48) ) − ˆ k
( x, x (cid:48) ) | .
Therefore, |
[Equation]
Applying Lemma F.9 and together with (14), we have
[Equation]
On the other hand, by the Hoeffding’s inequality, w.p. at least 1 − δ , we have
[Equation]
By Proposition 2.2, we have
[Equation]
The above proves the ﬁrst part of Proposition F.7.
We now turn to the second part.
We will frequently use the following McDiarmid’s inequality.
Theorem F.10 (McDiarmid’s inequality) .
Let X 1 , . . .
, X n are i.i.d.
random variables.
Assume for all i ∈ [ n ] and x 1 , . . .
, x n , ˜ x i ∈ X that
[Equation]
Let et σ 2 :=(cid:80) ni
=1 D 2 i .
Then, for any δ ∈ (0 , 1) , w.p. at least 1 − δ over the sampling of
X 1 , . . .
, X n , we have
[Equation]
We deﬁne two matrices
• The kernel matrix: K =
( K i,j ) ∈ R n × n with K i,j = 1 n k ( x i , x j ) ; • The approximate kernel matrix:
ˆ K =
( ˆ K i,j ) ∈ R n × n with
[Equation]
Note that the approximate kernel matrix is the normalized Gram matrix, i.e., ˆ K = G/n .
We will frequently use the following inequality:
[Equation]
Bounding the largest eigenvalue.
Lemma F.11.
There exists a constant c > 0 such that for any δ ∈ (0 , 1) , w.p. at least 1 − δ over the sampling of random features, we have λ 1 ( ˆ K ) (cid:54) λ
1 ( K ) + Cb(cid:112) log(2 /δ )
/m .
Proof.
Let ∆( w 1 , . . .
, w m )
= (cid:107) K − ˆ K (cid:107) F .
By Jensen’s inequality, we have
[Equation]
where the last inequality follows from (15).
Denote by ˆ K (cid:48) the approximate kernel matrix associated with ( w 1 , . .
w (cid:48) s , . . .
, w m ) .
Then,
[Equation]
Therefore σ 2 =(cid:80) ms =1 D 2 s (cid:46) b 4 /m .
By McDiarmid’s inequality (Theorem F.10), for any δ ∈ (0 , 1) , w.p. at least 1 − δ over the sampling of random features, we have
[Equation]
log(2 /δ )
m (cid:54) b(cid:114)
log(2 /δ )
m .
Plugging the above estimate to the Weyl’s inequality: λ 1 ( ˆ K ) (cid:54) λ 1 ( K ) + (cid:107) K − ˆ K (cid:107) 2 , we complete the proof.
Lemma F.12.
Suppose k ( · , · ) to be positive semi-deﬁnite kernel and let φ : X (cid:55)→ H be a feature map satisfying k ( x, y ) =
[Equation]
Then,
[Equation] [Equation]
Proof.
By the variational principle of eigenvalues, we have
[Equation]
Lemma F.13.
For any δ ∈ (0 , 1 /e ) , w.p. at least 1 − δ over the sampling of data, we have
[Equation]
Proof.
By Lemma F.12, we have
[Equation]
Let F 1 = { f ( x ) :=
[Equation]
h, σ x (cid:105) H
| (cid:107) h (cid:107) H (cid:54) 1 } and F 2 = { f 2 | f ∈ H 1 } .
Then, we have: (1) for any f ∈ H 1 , | f ( x ) |
[Equation]
by Lemma F.5.
Using contraction inequality (Lemma F.3), we have
[Equation]
Using Theorem F.2, for any δ ∈ (0 , 1 /e ) , w.p. larger than 1 − δ over the sampling of data, we have sup
[Equation]
For any ε > 0 , let ˆ h ∈ H such that λ 1 ( K ) (cid:54) ˆ E
[ (cid:104) φ x , ˆ h (cid:105) 2 H ] + ε .
Then, using (18), we have
[Equation]
Taking ε → 0 completes the proof.
Lemma F.14.
For any δ ∈ (0 , 1) , w.p. at least 1 − δ over the sampling of data and random features, we have
[Equation]
Proof.
The conclusion directly follows from the combination of Lemma F.13 and F.11.
Bounding the Frobenius norm.
The following lemma provide a lower bound of the Frobenius norm of the approximate kernel matrix.
Lemma F.15.
For any δ ∈ (0 , 1 /e ) , w.p. at least 1 − δ over the sampling of data and random features, we have
[Equation]
Proof.
Denote by { ( λ s , v s ) } s (cid:62) 1 the eigen-pairs of the kernel k ( · , · ) , and thus k ( x, x (cid:48) )
= (cid:80) s λ
s v s ( x ) v s ( x (cid:48) ) .
Let x, x (cid:48) be independently drawn from ρ .
Then,
[Equation]
Using the above equality, we have
[Equation]
Here, the last inequality is due to E [ k 2 ( x, x )] (cid:62) E
[ k 2 ( x, x (cid:48) )] as explained as follows.
Notice that for any x, x (cid:48) ∈ X , k 2 ( x, x (cid:48) )
(cid:54) k
( x, x ) k
( x (cid:48) , x (cid:48) )
(See, e.g., [34, Lemma 35]).
Therefore, E [ k 2 ( x, x (cid:48) )] (cid:54) E
[ k ( x, x )]
E [ k ( x (cid:48) , x (cid:48) )]
= ( E [ k ( x, x )]) 2 (cid:54) E
[ k 2 ( x, x )] .
The last inequality follows from that ( E [ X ])
(cid:54) E [ X 2 ] holds for any random variable X .
Let K (cid:48) be the kernel matrix corresponding to the ( x 1 , . . .
, ˜ x i , . . .
, x n ) .
Then,
[Equation]
Therefore, σ 2 =(cid:80) ni =1 | D i | 2 (cid:46) 1 /n .
Using Theorem F.10 and Eq.
(20), we have w.p. at least 1 − δ over the sampling of data that
[Equation]
In addition, following the proof of Lemma F.11, we have for any δ ∈ (0 , 1 /e ) , w.p. larger than 1 − δ over the sampling of random features that (cid:107) ˆ K − K (cid:107) F (cid:54) d (cid:112) log(2 /δ )
/m.
Thus,
[Equation]
Combing with (21), we complete the proof.
Prove the second part of Proposition F.7.
Recall (cid:15) n = (cid:112) log(1 /δ ) /n
+ b(cid:112) log(1 /δ )
/m .
Combining Lemma F.14 and F.15, we have
[Equation]
[Equation]
F.3 Proof of Proposition 2.
Denote by τ d − 1 the uniform distribution over the unit sphere S d − 1 = { x ∈ R
d : (cid:107) x (cid:107) 2 = 1 } .
According to [5], the eigenfunctions of
K are the spherical harmonics.
In particular, the eigenfunction corresponding to the largest eigenvalue is the ﬁrst spherical harmonics: Y 1 ( x ) ≡ 1 .
Therefore,
[Equation]
, where the last equality uses the rotational symmetry of τ d − 1 .
Moreover, by (19),
[Equation]
i λ 2 i
( K
[Equation]
=
E
[Equation]
[ κ
[Equation]
( x
[Equation]
=
E x [ κ 2 ( x 1 )] .
For the ReLU activation function, [7] shows that k ( x, x (cid:48) )
= κ ( x T x (cid:48) ) with
[Equation]
The eigenvalues of this kernel has been derived in [5, 45].
Speciﬁcally, we have
[Equation]
Hence,
[Equation]
Then, applying Proposition F.7, we complete the proof.
G Proofs of Section
We ﬁrst need the following technical lemma.
Lemma G.1.
For a, b > 0 , let g ( a, b, θ ) =
− a θ T H
2 θ θ T
Hθ
+ b θ Tt H
3 θ θ T Hθ .
Then, inf θ g ( a, b, θ ) (cid:62) − a 2 / (4 b ) .
Proof.
Let u
= H 1 / 2 θ/ (cid:107) H 1 / 2 θ (cid:107) and H = (cid:80)
j λ j e j e Tj the eigen-decomposition of H .
Suppose u
= (cid:80) j s j e j .
Then (cid:80) j s 2 j = 1 and −
[Equation]
We ﬁrst consider GD, for which the stability only imposes the ﬂatness constraint: λ 1 ( H ) (cid:54) 2 /η .
This means that the ﬂatness seen by GD is only the largest eigenvalue of Hessian.
Lemma G.2.
(1) inf θ r ( θ ) (cid:62) 0 ; (2) sup θ r ( θ ) (cid:54)
if η (cid:54) 2 /λ 1 ( H ) .
Proof.
Recall that r ( θ ) =
− 2 η θ T H
2 θ θ T Hθ
+ η 2 θ
Tt H
3 θ θ T Hθ .
Let u
= H 1 / 2 θ/ (cid:107) H 1 / 2 θ (cid:107) .
Then we have r ( θ ) = 1 − 2 ηu T Hu + η 2 u T H 2 u =
[Equation]
Let u
= (cid:80) j s j e j with { e j } j being the eigenvectors of H .
Then (cid:80) j s 2 j = 1 due to (cid:107) u (cid:107) = 1 .
By the assumption, ( ηλ j − 1) 2 (cid:54) 1 for all j ∈ [ n ] .
Then,
[Equation]
Proof of Proposition 3.4 Using µ 1 ( θ ) (cid:62) µ 1 and Σ( θ ) =
Σ 1 ( θ ) − Σ 2 ( θ ) , we have
[Equation]
where the contribution of Σ 2 ( θ ) is − θ T H 3 θ/ ( Bθ T Hθ ) .
Notice that there is a similar term in the contribution of mean gradient r ( θ ) .
Together we have E [ L ( θ t +1 )]
(cid:62) E
[ L ( θ t ) γ ( θ )] with γ
[Equation]
[Equation]
By Lemma G.1, we have
[Equation]
Let γ 0 (cid:54) 1 yields (cid:107) H (cid:107) 2 F (cid:54) B/ ( η (cid:112) ( B − 1) µ 1 ) .
This bound is trivial for the case B = 1 , where
[Equation]
Noticing that (cid:107) H (cid:107) 2 F = (cid:80) j λ 2 j ( H ) , the stability condition needs 1 − 2 ηλ 1 ( H ) + µ 1 η 2 B (cid:107) H (cid:107) F (cid:54) 1 , leading to (cid:88)
j λ
[Equation]
( H ) (cid:54)
[Equation]
u T Hu (cid:54) 2 B µ 1 η λ 1 ( H ) .
Thus, λ 1 ( H ) (cid:54) 2 B/ ( µ 1 η ) .
Consequently, (cid:107) H (cid:107) 2 F (cid:54)
Bµ 1 η λ 1 ( H ) (cid:54) ( 2 Bµ 1 η ) 2 .
Combing them, we complete the proof.