D IRECTION M ATTERS : O N THE I MPLICIT R EGULARIZATION E FFECT OF S TOCHASTIC G RADIENT D ESCENT WITH M ODERATE L EARNING R ATE
Anonymous authors
Paper under double-blind review
A BSTRACT
Understanding the algorithmic regularization effect of stochastic gradient descent (
SGD) is one of the key challenges in modern machine learning and deep learning theory.
Most of the existing works, however, focus on very small or even inﬁnitesimal learning rate regime and fail to cover practical scenarios where the learning rate is moderate and annealing
In this paper
, we make an initial attempt to characterize the particular regularization effect of SGD in the moderate learning rate regime by studying its behavior for optimizing an overparameterized linear regression problem
In this case
SGD and GD are known to converge to the unique minimum-norm solution however, with the moderate and annealing learning rate
, we show that they exhibit different directional bias
SGD converges along the large eigenvalue directions of the data matrix, while GD goes after the small eigenvalue directions.
Furthermore, we show that such directional bias does matter when early stopping is adopted where the SGD output is nearly optimal but the GD output is suboptimal
Finally
, our theory explains several folk arts in practice used for SGD hyperparameter tuning, such as (1) linearly scaling the initial learning rate with batch size; and (2) overrunning SGD with high learning rate even when the loss stops decreasing
I NTRODUCTION
Stochastic gradient descent (SGD) and its variants play a key role in training deep learning models.
From the optimization perspective, SGD is favorable in many aspects, e.g., scalability for largescale models (He et al., 2016), parallelizability with big training data (Goyal et al., 2017), and rich theory for its convergence (Ghadimi & Lan, 2013; Gower et al., 2019).
From the learning perspective, more surprisingly, overparameterized deep nets trained by SGD usually generalize well, even in the absence of explicit regularizers (Zhang et al., 2016; Keskar et al., 2016; Wu et al., 2020).
This suggests that SGD favors certain “good” solutions among the numerous global optima of the overparameterized model.
Such phenomenon is attributed to the implicit regularization effect of
SGD.
It remains one of the key theoretical challenges to characterize the algorithmic bias of SGD, especially with moderate and annealing learning rate as typically used in practice (He et al., 2016;
Keskar et al., 2016).
In the small learning rate regime , the regularization effect of SGD is relatively well understood, thanks to the recent advances on the implicit bias of gradient descent (GD) (Gunasekar et al., 2017; 2018a;b; Soudry et al., 2018;
Ma et al., 2018; Li et al., 2018; Ji & Telgarsky, 2019b;a; Ji et al., 2020; Nacson et al., 2019a; Ali et al., 2019; Arora et al., 2019; Moroshko et al., 2020; Chizat &
Bach, 2020).
According to classical stochastic approximation theory [(Kushner & Yin, 2003), with a sufﬁciently small learning rate, the randomness in SGD is negligible (which scales with learning rate), and as a consequence SGD will behave highly similar to its deterministic counterpart, i.e.,
GD.
Based on this fact, the regularization effect of SGD with small learning rate can be understood through that of GD.
Take linear models for example, GD has been shown to be biased towards maxmargin/minimum-norm solutions depending on the problem setups (Soudry et al., 2018; Gunasekar et al., 2018a; Ali et al., 2019); correspondingly, follow-ups show that SGD with small learning rate has the same bias (up to certain small uncertainty governed by the learning rate) (Nacson et al.,
[Figure]
2019b; Gunasekar et al., 2018a; Ali et al., 2020).
The analogy between SGD and GD in the small learning rate regime is also demonstrated in Figures 1(a) and 3.
However, the regularization theory for SGD with small learning rate cannot explain the beneﬁts of
SGD in the moderate learning rate regime , where the initial learning rate is moderate and followed by annealing (Li et al., 2019; Nakkiran, 2020; Leclerc & Madry, 2020; Jastrzebski et al., 2019).
In particular, empirical studies show that, in the moderate learning rate regime, (small batch) SGD generalizes much better than GD/large batch SGD (Keskar et al., 2016; Jastrz˛ebski et al., 2017; Zhu et al., 2019; Wu et al., 2020)
(see Figure 3).
This observation implies that, instead of imitating the bias of GD as in the small learning rate regime, SGD in the moderate learning rate regime admits superior bias than GD — it requires a dedicated characterization for the implicit regularization effect of SGD with moderate learning rate.
In this paper, we reveal a particular regularization effect of SGD with moderate learning rate that involves convergence direction .
In speciﬁc, we consider an overparameterized linear regression model learned by SGD/GD.
In this setting, SGD and GD are known to converge to the unique minimum-norm solution (Gunasekar et al., 2018a)
(see also Section 2.1).
However, with a moderate and annealing learning rate, we show that SGD and GD favor different convergence directions: SGD converges along the large eigenvalue directions of the data matrix ; in contrast, GD goes after the small eigenvalue directions.
The phenomenon is illustrated in Figure 1(b).
To sum up, we make the following contributions in this work:
For an overparameterized linear regression model, we show that SGD with moderate learning rate converges along the large eigenvalue directions of the data matrix, while GD goes after the small eigenvalue directions.
To our knowledge, this result initiates the regularization theory for SGD in the moderate learning rate regime, and complements existing results for the small learning rate.
Furthermore, we show the particular directional bias of SGD with moderate learning rate beneﬁts generalization when early stopping is used.
This is because converging along the large eigenvalue directions (SGD) leads to nearly optimal solutions, while converging
along the small eigenvalue directions (GD) can only give suboptimal solutions.
Finally, our results explain several folk arts for tuning SGD hyperparameters, such as (1) linearly scaling the initial learning rate with batch size (Goyal et al., 2017); and (2) overrunning SGD with high learning rate even when the loss stops decreasing (He et al., 2016).
P RELIMINARY
Let ( x, y ) ∈ R
d × R be a pair of d -dimensional feature vector and 1 -dimensional label.
We consider a linear regression problem with square loss deﬁned as (cid:96) ( x, y ; w ) :
= ( w (cid:62) x − y ) 2 , where w ∈ R d is the model parameter.
Let D be the population distribution over ( x, y ) , then the test loss is L D ( w ) :=
E
[Equation]
x,y )
∼D [ (cid:96) ( x, y ; w )] .
Let S :
= { ( x i , y i ) } ni =1 be a training set of n data points drawn i.i.d.
from the population distribution D .
Then the training/empirical loss is deﬁned as the average of the individual loss over all training data points,
[Equation]
where
[Equation]
We use { η k } to denote a learning rate scheme (LR).
Then gradient descent (GD) iteratively performs the following update:
[Equation]
Next we introduce mini-batch stochastic gradient descent (SGD).
1 Let b be the batch size.
For simplicity suppose n = mb for an integer m (number of mini-batches).
Then at each epoch, SGD ﬁrst randomly partitions the training set into m disjoint mini-batches with size b , and then sequentially performs m updates using the stochastic gradients calculated over the m mini-batches.
Specifically, at the k -th epoch, let the mini-batch index sets be B k 1 , B k 2 , . . .
, B km , where |B kj | =
b and (cid:83) m j =1 B k j
= { 1 , 2 , . . . , n } , then SGD takes m updates as follows
[Equation]
SGD)
We also write w k +1 =
w k,m +1 and w k = w k, 1 to be consistent with notations in (GD).
2. T HE MINIMUM - NORM BIAS
Before presenting our results on the directional bias, let us ﬁrst recap the well-known minimumnorm bias for SGD/GD optimizing linear regression problem (Gunasekar et al., 2018a; Belkin et al., 2019; Bartlett et al., 2020).
We rewrite the training loss as L S ( w ) = 1 n (cid:13)(cid:13)
X (cid:62) w − Y
(cid:13)(cid:13) 2 2 , where
X = ( x 1 , . . .
, x n ) ∈ R d × n and Y =
( y 1 , . . .
, y n ) (cid:62) ∈ R n .
Then its global minima are given by
W ∗ := (cid:8) w ∈ R d : Pw = PY (cid:9) , where P is the projection operator onto the data manifold, i.e., the column space of X .
We focus on overparameterized cases where W ∗ contains multiple elements.
Notice that every gradient ∇ (cid:96) i ( w ) = 2 x
i ( x (cid:62) i w − y i ) is spanned in the data manifold, thus (GD) and (SGD) can never move along the direction that is orthogonal to the data manifold.
In other words, (GD) and (SGD) implicitly admit the following hypothesis class :
[Equation]
where w 0 is the initialization and P ⊥
=
I − P is the projection operator onto the orthogonal complement to the column space of X .
Putting things together, for any global optimum w ∈ W ∗ (hence Pw = PY ), we have
[Equation]
where the right hand side is minimized when P ⊥ w = P ⊥ w 0 , i.e., w ∈ H S , thus w is the solution found by SGD/GD in the non-degenerated cases (when the learning rate is set properly so that the algorithms can ﬁnd a global optimum).
In sum, SGD/GD is biased to ﬁnd the global optimum that is closest to the initialization , which is referred as the “minimum-norm” bias in literature since the initialization is usually set to be zero.
W ARMING UP : A 2-D IMENSIONAL C ASE S TUDY
In this section we conduct a 2 -dimensional case study to motivate our understanding on the directional bias of SGD in the moderate learning rate regime.
Let us consider a training set consisting of two orthogonal points, S = { ( x 1 , y 1 = 0) , ( x 2 , y 2 = 0) } where
[Equation]
[Equation]
Clearly w ∗ = 0 is the unique minimum of L S ( w ) .
The Hessian of the empirical loss is ∇ 2 L S ( w ) =
x 1 x (cid:62) 1 + x 2 x (cid:62) 2 = diag ( κ, 1) , which has two eigenvalues: the smaller one 1 is contributed by data x 2 , and the larger one κ contributed by data x 1 .
Hence L S ( w ) is κ -smooth.
Similarly the Hessian of the individual losses are ∇ 2 (cid:96) 1 ( w ) = 2 x 1 x (cid:62) 1 = diag (2 κ, 0) and ∇ 2 (cid:96) 2 ( w ) = 2 x 2 x (cid:62)
2 = diag (0 , 2) .
Thus (cid:96) 2 ( w ) is 2 -smooth, but (cid:96) 1 ( w ) , the individual loss for data x 1 , is only 2 κ -smooth, which is more ill-conditioned compared to L S ( w ) and (cid:96) 2 ( w ) .
Next we consider a moderate initial learning rate η ∈ (cid:0) 1 κ , 21+ κ (cid:1) .
According to convex optimization theory (Boyd et al., 2004), gradient step with such learning rate is convergent for L S ( w ) and (cid:96) 2 ( w ) , but oscillating for (cid:96) 1 ( w ) .
In other words, (GD) is convergent; and (SGD) is convergent along direction x 2 (or e 2 ), but oscillating along direction x 1 (or e 1 ).
We also see this by analytically solving (GD) and (SGD) for this example:
[Equation]
where | 1 − ηκ | < | 1 − η | < 1 and |
[Equation]
By Eq.
(3), with moderate learning rate GD is convergent for both directions e 1 and e 2 .
Moreover,
GD ﬁts e 1 faster since the contraction parameter is smaller, i.e., | 1 − ηκ | < | 1 − η | < 1 .
Thus observing the entire optimization path, GD approaches the minimum w ∗ = 0 along e 2 , which corresponds to the smaller eigenvalue direction of ∇ 2 L S ( w ) .
This is veriﬁed by the blue dots in
Figure 1(b).
We note this directional bias for GD also holds in the small learning rate regime, as shown in Figure 1(a).
As for SGD in the initial phase where the learning rate is moderate, Eq.
(3) shows it converges along e 2 but oscillates along e 1 since | 1 − 2 η | < 1 <
| 1 − 2 ηκ | .
In other words, SGD cannot ﬁt e 1 before the learning rate decays; however when this happens, e 2 is already well ﬁtted.
Overall, SGD ﬁts eﬁrst then ﬁts e 1 , i.e., SGD converges to the minimum w ∗ = 0 along e 1 , which corresponds to the larger eigenvalue direction of ∇ 2 L S ( w ) .
This is veriﬁed by the red dots in Figure 1(b).
We note this particular directional bias for SGD is dedicated to the moderate learning rate regime; in the small learning rate regime, as discussed before, SGD behaves similar to GD thus goes after the smaller eigenvalue direction, which is illustrated in Figure 1(a).
The above idea can be carried over to more general cases: the training loss usually has relatively smooth curvature because of the empirical averaging; yet some individual losses can possess bad smoothness condition, corresponding to the data points that contribute to the large eigenvalues of the Hessian/data matrix.
Then with a moderate learning rate, while GD is convergent, SGD is convergent for the smooth individual losses but oscillating for the ill-conditioned individual losses.
Thus SGD can only ﬁt the latter losses after the learning rate anneals.
Therefore, in the moderate learning rate regime, SGD tends to converge along the large eigenvalue directions while GD tends to go after the small eigenvalue directions.
We will rigorously justify the above intuitions in the following section.
M AIN T HEORY
In this section we present our main theoretical results.
The proofs are deferred to Appendix B.
We specify the population distribution of ( x, y ) ∈
R d × R in the following manner: (1) we consider the feature vector given by x = ζ · ξ , where ζ ∈ R 1 is a magnitude random variable that takes value in (0 , 1] , and ξ ∈ R d is an angle random variable that follows a sphere uniform distribution, i.e., ξ ∼ U ( S d − 1 ) ; (2) we consider a realizable setting where the label is given by y = w (cid:62)∗ x , i.e., there exists a true parameter
w ∗ ∈ R d that generates the label from the feature vector.
Then the test loss is
[Equation]
22 , where µ = E [ ζ ] /d .
For an i.i.d.
generated training set S =
{ ( x i , y i ) } n i =1 , the training loss and the individual losses are
[Equation]
where X = ( x 1 , .
x n ) .
We denote by P the projection operator onto the column space of X (the data manifold).
For i ∈ [ n ] , we denote λ
i := (cid:107) x i (cid:107) 22 .
By the previous data generalization process, we have λ i ∈
Without loss of generality, we assume { λ i } i ∈ [ n ] are sorted in a descending order, i.e., λ 1 ≥ λ 2 ≥ · · · ≥ λ n .
With the these preparations, we are ready to state our main theorems.
4. T HE DIRECTIONAL BIAS OF SGD
We ﬁrst present Theorems 1 and 2 that characterize the different directional biases of SGD and GD in the moderate learning rate regime.
Theorem 1 (The directional bias of SGD with moderate LR, informal) .
Suppose d ≥ poly ( n ) , λ 1 > λ 2 + o (1) and λ n > o (1) 2 , where o (1) is a small positive constant depending on n/d , and the initialization is away from w ∗ .
Consider (SGD) with the following moderate learning rate scheme
[Equation]
Then for (cid:15) =
o (1) , there exist k 1 > O (cid:0) log 1 (cid:15) (cid:1) and k 2 , such that with high probability the output of
SGD w sgd satisﬁes
[Equation]
where γ 1 is the largest eigenvalue of the data matrix XX (cid:62) .
Theorem 2 (The directional bias of GD with moderate or small LR, informal) .
Suppose d ≥ poly ( n ) , λ n − 1 > λ n + o (1) and λ n > o (1) , where o (1) is a small positive constant depending on n/d , and the initialization is away from w ∗ .
Consider (GD) with the following moderate or small learning rate scheme
[Equation]
∈
[Equation] [Equation]
[Equation]
Then for any (cid:15) > 0 , if k 2 > O (cid:0) log 1 (cid:15) (cid:1) , then with high probability the output of GD w gd satisﬁes
[Equation]
where γ n is the smallest eigenvalue of the data matrix XX (cid:62) restricted in the column space of X .
A few remarks on Theorems 1 and 2 are in order.
Remark 1.
As the Rayleigh quotient (5) (resp.
(7) ) converges to its maximum (resp. minimum), the vector gets closer to the eigenvector of the largest (resp.
smallest) eigenvalue (Trefethen & Bau III, 1997).
Thus Theorem 1 and 2 suggest that, when projected onto the data manifold, SGD and GD converge to the optimum along the largest and smallest eigenvalue direction respectively.
Here we are only interested in the projection onto the data manifold, since SGD/GD cannot move along the direction that is orthogonal to the data manifold as discussed in Section 2.1.
Remark 2.
For the ease of presentation, we make strong gap assumptions ( λ 1 > λ
+ o (1) and λ n − 1 > λ n + o (1) ), and show SGD/GD converges along the largest/smallest eigenvalue direction.
Nonetheless, one can weaken these requirements and extend our analysis to show a reﬁned result where SGD/GD converges along the subspace spanned by the eigenvectors of the top/bottom eigenvalues.
Remark 3.
We legitimately assume b < n/ 2 − o (1) since it is not very meaningful to discuss
SGD that uses more than (roughly) half of the training set as a mini-batch.
Then the learning rate schedule in (4) intersects with that in (6) , i.e., (6) covers both moderate and small learning rate schemes .
Their intersection determines a moderate learning rate scheme, where SGD converges along the large eigenvalue directions while GD goes after the small eigenvalue directions.
This justiﬁes the regularization effect of SGD with moderate learning rate.
Remark 4.
Technically, in Theorem 1 one can set b = n to include GD as a special case, so that GD also follows the large eigenvalue directions.
However, the initial learning rate in (4) needs to be at least nλ 1 ≥ n ignoring the small order term, which is way too large to be even numerically stable in practical big data circumstances.
This observation is also directly supported by Theorem 2, where (6) speciﬁes the range of learning rate such that GD converges along small eigenvalue directions.
Note the upper bound in (6) linearly scales with n , and is large enough to include all learning rate that can be adopted in practice.
Thus one cannot have a legitimate learning rate for GD to converge along the large eigenvalue directions as SGD does with moderate learning rate.
Note GD with small learning rate also converges along the small eigenvalue directions, since the learning rate schedule in (6) covers the small learning rate scheme.
In complement, the following
Theorem 3 shows that in the small learning rate regime, SGD is imitating GD and converges along the small eigenvalue directions as well.
Theorems 1, 2 and 3 together show that, converging along the large eigenvalue directions is a distinct regularization effect that is unique to SGD with moderate learning rate .
Theorem 3 (The directional bias of SGD with small LR, informal) .
Theorem 2 applies to (SGD) with the following small learning rate scheme
[Equation]
Experiment Figure 2 shows two experiments for verifying Theorems 1, 2 and 3.
The details of the experimental setup are deferred to Appendix D. In Figure 2(a), we run SGD and GD in both moderate and small learning rate regimes, and directly compare their Rayleigh quotients as deﬁned in (5) and (7).
We can see that the Rayleigh quotient reaches its maximum for SGD with moderate learning rate, and reaches its minimum for both GD and SGD with small learning rate, which veriﬁes Theorems 1, 2 and 3.
In Figure 2(b), we run the algorithms to optimize a neural network on a subset of the FashionMNIST dataset.
Since the neural network is non-convex and have multiple local minima, we compare the relative Rayleigh quotients , i.e., the Rayleigh quotients of the convergence directions divided by the maximum absolute eigenvalue of the Hessian (see Appendix D.3).
Figure 2(b) shows that SGD with moderate learning rate converges along relatively large eigenvalue directions while GD/SGD with small learning rate converges along relatively small eigenvalue directions.
This distinguishes the directional bias of SGD and GD in the moderate learning rate regime and provides evidence from neural network training to support our theory.
4. E FFECTS OF THE DIRECTIONAL BIAS
Next we justify the beneﬁt of the particular directional bias of SGD with moderate learning rate.
Recall the hypothesis class H S (Eq.
(2)) for SGD and GD.
Then for an algorithm output w alg , we have the following generalization error decomposition (Shalev-Shwartz & Ben-David, 2014),
[Equation]
The approximate error is an intrinsic error determined by the hypothesis class, and is not improvable unless enlarging the hypothesis class.
In contrast, the estimation error ∆( w alg ) is determined by the algorithm as well as its hyperparameters.
Thus, in the following theorem, we use the estimation error to compare the generalization performance of the SGD and GD outputs in different learning rate regimes.
Theorem 4 (Effects of the directional bias, informal) .
Let W α :=
(cid:8) w ∈ H S :
L S ( w ) = α (cid:9) be an α -level set of the training loss L S ( w ) .
Let ∆ ∗ α :=
inf w ∈W α ∆( w ) be the minimum estimation error within the α -level set W α .
Under the same conditions as Theorems 1 and 2, if b < n/ 2 − o (1) , then there exists k 1 > 0 and k 2 > 0 such that with high probability:
• The output of (SGD) with moderate LR (4) satisﬁes ∆( w sgd ) <
(1 + (cid:15) ) · ∆ ∗ α , where α is the training loss of w sgd , and (cid:15)
=
o (1) is a small constant; • The output of (GD) with moderate or small LR (6) satisﬁes ∆( w gd ) > M · ∆ ∗ α , where α is the training loss of w gd , and M = γ 1 /γ n − o (1) is a constant larger than 1 + (cid:15) ;
[Figure]
by ( η, η (cid:48) , k 1 , k 2 ) =
(10 − 2 , 10 − 3 , 2 . 5 × 10 3 , 10 4 ) .
Since neural network is non-convex, we compare the relative Rayleigh quotient of the concerned algorithms, i e., the Rayleigh quotient of the convergence directions divided by the maximum absolute eigenvalue of the Hessian (see Appendix D.3).
• The output of (SGD) with small LR (8) satisﬁes ∆( w sgd ) > M · ∆ ∗ α , where α is the training loss of w sgd , and M = γ 1 /γ n − o (1) > 1 is a constant larger than 1 + (cid:15) .
[Figure]
Theorem 4 suggests that: (1) in the moderate learning rate regime, there is a separation between the test error of
SGD and that of GD.
In detail, early stopped SGD ﬁnds a nearly optimal solution thanks to its particular directional bias.
In contrast, early stopped GD can only ﬁnd a suboptimal one; and (2) in the small learning rate regime, however, SGD no longer admits the dedicated directional bias for moderate learning rate.
Instead it behaves similarly as
GD, and hence outputs suboptimal solutions when early stopping is adopted.
Remark 5.
In practice it is usually intractable and unnecessary to achieve the exact global minima of the training loss; instead we often early stop the algorithm once obtaining a small enough training loss, i.e., reaching an α -level set.
In this spirit, Theorem 4 compares the generalization ability of SGD with moderate learning rate vs.
GD/SGD with small learning rate within a level set.
Remark 6.
We note the second conclusion in Theoremis also obtained by Nakkiran (2020) for a different purpose.
In speciﬁc, Nakkiran (2020) show the separation between the test error of GD with “large” and annealing learning rate, and test error of GD with small learning rate.
However, the “large” learning rate for GD in their analysis is linear in the training sample size and is not practical as we have discussed in Remark 4.
In contrast, we show that under the practically used moderate learning rate, there is a separation between the generalization abilities of SGD and GD.
To our knowledge, our work gives the ﬁrst theoretical justiﬁcation of the phenomenon that SGD outperforms GD when the learning rate is moderate.
Experiment Figure 3 shows the generalization performance of a neural network trained by SGD and GD in both moderate and small learning rate regimes.
The setup details are included in Appendix D. We can observe that (1) SGD with moderate learning rate generalizes the best, and (2)
GD and SGD with small learning rate perform similarly, but both are worse than SGD with moderate learning rate.
The empirical results suggest SGD with moderate learning has certain benign regularization effect.
This is explained by the distinct directional bias for SGD with moderate learning rate shown in previous theorems.
R ELATED W ORK
In this section, we review related works and discuss their similarities and differences to our work.
The implicit bias of GD The implicit bias of GD has been extensively studied in recent years.
We summarize several representative results as follows.
For homogeneous model with exponentiallytailed loss, GD converges along the max-margin direction (Gunasekar et al., 2017; 2018a;b; Soudry et al., 2018;
Ma et al., 2018; Ji & Telgarsky, 2019a; Ji et al., 2020; Nacson et al., 2019a).
For least square problem and its variants, GD is biased towards minimum-norm solution (Gunasekar et al., 2018a; Ali et al., 2019; Suggala et al., 2018).
Note that this is also the foundation for the learning theory in the interpolation regime such as double descent (Belkin et al., 2019) and benign overﬁtting (Bartlett et al., 2020).
For matrix factorization and linear network, Gunasekar et al.
Li et al.
(2018) show the GD solution minimizes nuclear norm in special cases; more generally,
Arora et al.
; Ji & Telgarsky (2019b) show GD balances/aligns layers; Moroshko et al.
(2020) suggest the GD bias relies on initialization scale and training accuracy.
For inﬁnite-width network,
Chizat & Bach (2020) show GD ﬁnds a max-margin classiﬁer in a functional space.
The regularization theory for GD is fruitful; While some of them can be applied to SGD with small learning rate as have discussed (Nacson et al., 2019b; Gunasekar et al., 2018a; Ali et al., 2020), none of them can be carried over to SGD in the moderate learning rate regime.
As far as we know, our paper is the ﬁrst to study the regularization effect of SGD with moderate learning rate.
The stability of SGD Stability is another approach to justify the generalization ability of SGD, where a stable algorithm is guaranteed to have small generalization error (Bousquet & Elisseeff, 2002).
Along this line, several works show that SGD is stable under certain assumptions and therefore generalizes well (Hardt et al., 2016; Kuzborskij & Lampert, 2018; Charles & Papailiopoulos, 2018; Bassily et al., 2020).
More interestingly, Charles & Papailiopoulos (2018) show a simple example where SGD is stable but GD is not, which partly explains the empirically superior performance of SGD.
We also aim to justify the beneﬁts of SGD, but take a different approach from the algorithmic regularization.
The escaping behavior of SGD A popular theory (Jastrz˛ebski et al., 2017; Zhu et al., 2019; Simsekli et al., 2019) attributes the regularization effect of SGD to its behavior of escaping from sharp minima.
These works are built upon the continuous approximation of SGD via stochastic differential equations (Li et al., 2017;
Hu et al., 2017; Xu et al., 2018; Simsekli et al., 2019).
However it requires a small learning rate for the approximation to hold.
In contrast, our main interest is in the moderate learning rate regime.
Another related work in this line is (Wu et al., 2018), where they study the dynamical stability of minima and how SGD chooses them, but they do not show a directional bias introduced in our work.
Non-small learning rate The regularization effect of non-small learning rate has received increasing attentions recently.
Theoretically, Li et al.
; HaoChen et al.
(2020) study the generalization of certain stochastic dynamics equipped with annealing learning rate scheme.
However, their results cannot cover vanilla SGD.
Lewkowycz et al.
(2020) study the role of initial large learning rate for inﬁnity-width network.
Our work is motivated by Nakkiran (2020), which shows that a large initial learning rate helps GD generalize.
This result can be recovered from our theorems; more importantly, we characterize the directional bias of SGD.
Empirically, Jastrzebski et al.
; Leclerc &
Madry (2020) investigate the impact of two-phase learning rate for training with SGD, but they do not provide any theoretical analysis.
D ISCUSSIONS
Linear scaling rule
Linearly enlarging the initial learning rate according to batch size, or the linear scaling rule (Goyal et al., 2017), is an important folk art for paralleling SGD with large batch size while maintaining a good generalization performance.
Interestingly, the linear scaling rule arises naturally from Theorem 1, where LR (4) suggests the initial learning rate η to scale linearly with batch size b to guarantee the desired directional bias.
Our theory thus partly explains the mechanism of the linear scaling rule.
Overrunning SGD with high learning rate Besides the moderate initial learning rate, another key ingredient in Theorem 1 is a sufﬁciently large k 1 , i.e., SGD needs to run with moderate learning rate for sufﬁciently long time (to ﬁt small eigenvalue directions).
This requirement coincidentally agrees with the folk art to overrun SGD with high learning rate .
For example, see Figure 4 in (He et al., 2016): from the ∼ 2 × 10 5 -th to the ∼ 3 × 10 5 -th iteration, even the training error seems to make no progress, practitioners let SGD run with a relatively high learning rate for nearly 10 5 iterates.
Our theory sheds light on understanding the hidden beneﬁts in this “overrunning” phase: indeed the loss is not decreasing since SGD with high learning rate cannot ﬁt the large eigenvalue directions, but on the other hand the overrunning lets SGD ﬁt the small eigenvalue directions better, which in the end leads to the directional bias that SGD converges along the large eigenvalue directions according to
Theorem 1.
Thus overrunning SGD with high learning rate is helpful.
Key technical challenges With non-small learning rate, analyzing SGD is usually hard since measure concentration turns vacuous and as a result one cannot relate the SGD iterates to that of GD.
Alternatively, we control the SGD iterates epoch by epoch, then bound their composition to characterize the long run behavior of SGD.
However, controlling the epoch-wise update of SGD is highly non-trivial, since in different epochs the sequence of stochastic gradient steps varies, and they do not commute due to the issue of matrix multiplication.
To overcome this difﬁculty, we adopt techniques from matrix perturbation theory (Horn & Johnson, 2012) and conduct an analysis in the overparameterized regime.
We believe these techniques are of independent interest.
C ONCLUSION AND F UTURE W ORK
We characterize a distinct directional regularization effect of SGD with moderate learning rate, where SGD converges along the large eigenvalue directions of the data matrix.
In contrast, neither
GD nor SGD with small learning rate can achieve this effect.
Moreover, we show this directional bias beneﬁts generalization when early stopping is adopted.
Finally, our theory explains several folk arts used in practice for SGD hyperparameter tuning.
As an initial attempt our results are limited to overparameterized linear models, and we ignore other factors that may contribute to the good generalization of SGD for training neural networks, e.g., network structures and explicit regularization.
It is left as a future work to extend our results to nonlinear/nonconvex models (with explicit regularization)
A P RELIMINARY FOR A PPENDIX A. A DDITIONAL NOTATIONS
We adopt the notations and settings in main text.
In addition we make the following notations.
For a vector x ∈ R d , denote its direction as ¯ x := x (cid:107) x (cid:107) 2 .
For simplicity assume the training data { x 1 , . . .
, x n } are linear independent.
For training data x i , i ∈ [ n ] , we denote λ i
= (cid:107) x
i (cid:107) 22 , then by construction we have λ i ∈ (0 , 1] .
Without loss of generality let λ 1 ≥ · · · ≥ λ n .
We deﬁne
[Equation]
Then based on the above deﬁnitions, we deﬁne the following two projection operators
[Equation]
Clearly for any v ∈ R d , Pv projects v onto subspace span { x 1 , . .
, x n } , while P ⊥ v projects v onto the orthogonal complement of span { x 1 , . .
, x n } .
Furthermore we introduce two more projection operators
[Equation]
For any v ∈ R d , P − 1 v projects v onto subspace span { x 2 , . . .
, x n } , while P 1 v projects v into the orthogonal complement of span { x 2 , . . .
, x n } with respect to span { x 1 , . . .
, x n } .
In the following, we often write the column space of P , which refers to (cid:8) Pv : v ∈ R d (cid:9) , similarly for P − 1 , P 1 and
P ⊥ as well.
Clearly the column space of P is also span { x 1 , . . .
, x n } ; the column space of P − 1 is also the data manifold span { x 2 , . . .
, x n } .
We highlight that the total space R d can be decomposed as the direct sum of the column space of P − 1 , P 1 and P ⊥ , i.e., I =
P − 1 + P 1 + P ⊥ .
By deﬁnition, it is easy to verify that
[Equation]
Then we deﬁne the following matrices which will be repeatedly used in the subsequent proof.
[Equation]
Based on the above deﬁnitions, it is easy to show that
[Equation]
A. L EMMAS
We present the following theorems and lemmas as preparation for our analysis.
Theorem (Gershgorin circle theorem, restated for symmetric matrix) .
Let A ∈ R n × n be a symmetric matrix.
Let A ij be the entry in the i -th row and the j -th column.
Let
[Equation]
| A ij | , i = 1 , . .
n.
Consider n Gershgorin discs
[Equation]
The eigenvalues of A are in the union of Gershgorin discs
[Equation]
Furthermore, if the union of k of the n discs that comprise G ( A ) forms a set G k ( A ) that is disjoint from the remaining n − k discs, then G k ( A ) contains exactly k eigenvalues of A , counted according to their algebraic multiplicities.
Proof.
See, e.g., Horn & Johnson (2012), Chap 6.1, Theorem 6.1.1.
Theorem (Hoffman-Wielandt theorem, restated for symmetric matrix) .
Let A, E ∈ R n × n be symmetric.
Let λ 1 , . . .
λ n be the eigenvalues of A , arranged in decreasing order.
Let ˆ λ 1 , . . .
, ˆ λ n be the eigenvalues of A + E , arranged in decreasing order.
Then
[Equation]
Proof.
See, e.g., Horn & Johnson (2012), Chapter 6.3, Theorem 6.3.5 and Corollary 6.3.8.
Lemma 1.
Let d ≥ 4 log(2 n 2 /δ ) for some δ ∈ (0 , 1) .
Then with probability at least 1 − δ , we have
[Equation]
Proof.
See Section C.1.
By Lemma 1 we can assume d ≥ poly ( n ) such that nι is sufﬁciently small depends on requirements.
The following two lemmas characterize the projected components of each training data onto the column space of P 1 , P − 1 , and P ⊥ .
Lemma 2.
For x j (cid:54) = x 1 , we have
• P − 1 x j = x j ; • P 1 x j = 0; • P ⊥ x j = 0 .
Proof.
These are by the construction of the projection operators.
Lemma 3.
Assume √ nι ≤ 1 / 4 .
With probability at least 1 − δ , we have
• 0 ≤ (cid:107) P − 1 ¯ x 1 (cid:107) 2 ≤ 2 √ nι ; • √ 1 − 4 nι 2 ≤ (cid:107) P 1 ¯ x 1 (cid:107) 2 ≤ 1; • P ⊥ x 1 = 0 .
Proof.
See Section C.2.
The following four lemmas characterize the spectrum of the matrices H , H − 1 , H 1 and H c .
Lemma 4.
Let γ 1 , . .
. , γ n be the n non-zero eigenvalues of H :
=
XX (cid:62) in decreasing order.
then λ n − nι ≤ γ 1 , . .
, γ n ≤ λ 1 + nι.
Furthermore, if there exist λ r and λ r +1 such that λ r > λ r +
+ 2 nι , then λ
n − nι ≤ γ r +1 , . .
γ n ≤ λ r +
+ nι < λ r − nι ≤ γ 1 , . . .
, γ r ≤ λ 1 + nι.
Proof.
See Section C.3.
Lemma 5.
Assume λ n ≥ 3 nι .
Consider the symmetric matrix H − 1 :=
P − 1 X ( P − 1 X ) (cid:62)
∈ R d × d .
• 0 is an eigenvalue of H − 1 with algebraic multiplicity being d − n + 1 , and its corresponding eigenspace is the column space of P 1 + P ⊥ .
• Restricted in the column space of P − 1 , the n − 1 eigenvalues of H − 1 belong to ( λ n − nι, λ 2 + nι ) .
Proof.
See Section C.4.
Lemma 6.
Consider matrix H 1 :
= P 1 X ( P 1 X ) (cid:62) ∈ R
d × d .
We have H 1 has only one non-zero eigenvalue, which belongs to
[Equation]
Moreover, the corresponding eigenspace is the column space of P 1 , which is 1 -dim.
Proof.
Clearly H 1 is rank- 1 since the column space of P 1 is 1 -dim.
Thus it has only one non-zero eigenvalue, which is given by
[Equation]
where the last equality follows from Lemma 3.
Lemma 7.
Consider matrix H
[Equation]
:=
( P
[Equation]
+
[Equation]
P
[Equation]
d × d .
[Equation]
Proof.
[Equation]
[Table]
where the last equality follows from Lemma 3 and λ 1 ≤ 1 .
B M ISSING P ROOFS FOR THE T HEOREMS IN M AIN T EXT B. T HE DIRECTIONAL BIAS OF SGD WITH MODERATE LEARNING RATE
Reloading notations Let π k := (cid:8) B k 1 , . . .
, B km (cid:9) be a randomly chosen uniform m -partition of [ n ] , where n = mb .
Then the SGD iterates at the k -th epoch can be formulated as:
[Equation]
where we assume that the learning rate is ﬁxed within each epoch.
Note here π k is independently and randomly chosen at each epoch.
For simplicity we often ignore the epoch-indicator k , and write the uniform partition as π :=
{B 1 , . . .
, B m } .
It is clear from context that π is random over epochs.
For a mini-batch B j ∈ π , we denote H ( B j ) :
= (cid:80) i ∈B j x i x (cid:62) i .
Considering translating the variable by
[Equation]
then we can reformulate the SGD update rule as
[Equation]
Let
M
[Equation]
Here the matrix production over a sequence of matrices (cid:8) M i ∈ R
d × d (cid:9) m j =1 is deﬁned from the left to right with descending index,
[Equation]
Let v k +
= v k,m +
and v k = v k, 1 .
Then we can further reformulate Eq.
(9) and obtain the epochwise update of SGD as
[Equation]
In light of the notion of v , the following lemma restates the related notations of loss functions, hypothesis class, level set, and estimation error deﬁned in Section 2 and 4.
Lemma 8 (Reloading SGD notations) .
Regarding repramaterization v = w − w ∗ , we can reload the following related notations:
• Empirical loss and population loss are
[Equation]
• The hypothesis class is
[Equation]
• The α -level set is
[Equation]
• For v ∈ H S , the estimation error is
[Equation]
Moreover,
[Equation]
Proof.
See Section C.5.
Based on the above deﬁnitions
[Equation]
the following lemma characterizes the one-step update of SGD.
Lemma 9 (One step SGD update) .
Consider the j -th SGD update at the k -th epoch as given by
Eq.
Set the learning rate be constant η during that epoch.
Then for j = 1 , . .
, m we have
[Equation]
Moreover, if 1 / ∈ B j , i
[Equation]
e., x 1 is not used in the j -the step, then
[Equation]
Proof.
See Section C.6.
Eq.
(10) indicates the key to analyze the convergence of v k +1 is to characterize the spectrum of the matrix M π .
In particular the following lemma bounds the spectrum of M π when projected onto the column space of P − 1 .
Lemma 10.
Suppose 3 nι < λ n .
Suppose 0 < η < b λ 2 +3 nι .
Let π :=
{B 1 , . . .
, B m } be a uniform m partition of index set [ n ] , where n = mb .
Consider the following d × d matrix
[Equation]
[Equation]
Then for the spectrum of M (cid:62)− 1 M − 1 we have:
• 1 is an eigenvalue of M (cid:62)− 1 M − 1 with multiplicity being d − n + 1 ; moreover, the corresponding eigenspace is the column space of P 1 + P ⊥ .
• Restricted in the column space of P − 1 , the eigenvalues of M (cid:62)− 1 M − 1 are upper bounded by ( q − 1 ( η )) 2 < 1 , where
[Equation]
Proof.
See Section C.7.
Consider the projections of v k onto the column space of P − 1 and P 1 .
For simplicity let
[Equation]
The following lemma controls the update of A k and B k .
Lemma 11 (Update rules for A k and B k ) .
Suppose 3 nι < λ n .
Suppose 0 < η < b λ 2 +3 nι .
Consider the k -th epoch of SGD iterates given by Eq.
Set the learning rate in this epoch to be constant η .
Denote
[Equation] [Equation]
Then we have the following:
• A k +1 ≤ q − 1 ( η ) ·
A k + ξ ( η ) · B k .
• B k +1 ≤ q 1 ( η ) · B k + ξ ( η ) · A k . • B k +1 ≥ q 1 ( η ) · B k − ξ ( η ) · A k .
Proof.
See Section C.8.
Note we can rephrase the update rules for A k and B k as
[Equation]
, where “ ≤ ” means “entry-wisely smaller than”.
The following two lemmas characterize the long run behaviors of A k and B k with different learning rate.
Lemma 12 (The long run behavior of SGD with moderate LR) .
Suppose 3 nι < λ n , and λ 2 +4 nι < λ 1 .
Suppose v 0 is far away from 0 .
Consider the ﬁrst k 1 epochs of SGD iterates given by Eq.
Set the learning rate during this stage to be constant, i.e., η k = η for 0 ≤ k < k 1 .
Suppose
[Equation]
Then for 0 < (cid:15) < 1 and 0 < β < β 0 < B 0 satisfying √ nι ≤ poly ( (cid:15)β ) , there exists k 1 ≥
O (cid:16) log 1 (cid:15)β (cid:17) such that
A k 1 ≤ (cid:15) · β. • B k 1 ≤ (cid:107) Pv 0 (cid:107) 2 · ρ k 1 1 + (cid:15) 2 · β = poly (cid:16) 1 (cid:15)β (cid:17) .
• For all k = 0 , 1 , . . .
, k 1 , B k > β 0 .
Proof.
See Section C.9.
Lemma
(The long run behavior of SGD with small LR) .
Suppose 3 nι < λ n , and λ 2 +4 nι < λ 1 .
Suppose v 0 is far away from 0 .
Consider another k 2 − k 1 epochs of SGD iterates given by Eq.
Set the learning rate to be constant during the updates, i.e., η k = η (cid:48) for k 1 ≤ k < k 2 .
Suppose 0 < η (cid:48)
<
b 2 λ 1 .
Consider the (cid:15) and β given in Lemma 12.
Then for k ≥ k 1 , we have
• A k ≤ (cid:15) · β. • B k ≤ (cid:26) q · B k − 1 , B k − 1 > β, β, B k − 1 < β.
where q ∈ (0 , 1) is a constant.
Proof.
See Section C.10.
Theorem 5 (Theorem 1, formal version) .
Suppose 3 nι < λ n and λ 2 + 4 nι < λ 1 .
Suppose v 0 is away from 0 .
Consider the SGD iterates given by Eq.
(10) with the following moderate learning rate scheme
[Equation]
Then for 0 < (cid:15) < 1 such that √ nι ≤ poly ( (cid:15) ) , there exist k 1 > O (cid:0) log 1 (cid:15) (cid:1) and k 2 such that
[Equation]
Proof.
We choose k 1 and k 2 as in Lemma 12 and Lemma 13 with β set as a small constant, then we are guaranteed to have
[Equation]
from where we have
[Equation]
Then we have
[Equation] [Equation]
[Equation] [Equation]
nι ( since γ 1 ≤ λ 1 + nι by Lemma 4 ) =
γ 1 (1 − 4 nι
[Equation]
1 − (cid:15)
[Equation]
nι (1 − 4 nι 2 )(1 − (cid:15) 2 ) − 4 √ nι
[Equation] [Equation]
( since √ nι ≤ poly ( (cid:15) )) =
[Equation]
1 (1 − (cid:15) )
[Equation]
Theorem 6 (Theorem 4 ﬁrst part, formal version) .
Suppose 3 nι < λ n and λ 2 + 4 nι < λ 1 .
Suppose v 0 is away from 0 .
Consider the SGD iterates given by Eq.
(10) with the following moderate learning rate schedule
[Equation]
Then for 0 < (cid:15) < 1 satisfying √ nι ≤ poly ( (cid:15) ) , there exist k 1 and k 2 such that SGD outputs an (cid:15) -optimal solution.
Proof.
We set
[Equation]
and apply Lemma 12 to choose a k 1 such that
[Equation]
Thus for all 0 ≤ k ≤ k 1 ,
[Equation]
( γ n is the smallest eigenvalue of XX (cid:62) in the column space of P )
[Equation]
>
[Equation]
Eq
[Equation]
14) ) which implies SGD cannot reach the α -level set during the iteration of ﬁrst stage, i.e., SGD does not terminate in this stage.
We thus consider the second stage.
From Lemma 13 we know (cid:107) P 1 v kn (cid:107) 2 will keep decreasing before being smaller than β , and (cid:107) P − 1 v k (cid:107) 2 stays small during this period, i.e., SGD ﬁts P 1 v while in the same time does not mess up P − 1 v .
Mathematically speaking, there exists k 2 and α such that
[Equation]
which implies SGD terminates at the k 2 -th epoch.
Then by Lemmas 3 and 8, we have
[Equation]
which yields
[Equation]
Then we can bound the estimation error as
[Equation]
where we use the fact that ∆ ∗ = µnα/γ 1 from Lemma 8.
Hence SGD is (cid:15) -near optimal.
B.
T HE DIRECTIONAL BIAS OF GD WITH MODERATE OR SMALL LEARNING RATE
Reloading notations Denote the eigenvalue decomposition of XX (cid:62) as
[Equation]
G = ( g 1 , . . . , g n , . . . , g d ) , where G ∈ R d × d is orthonormal, and γ 1 , . . .
, γ n are given by Lemma 4.
Clearly, span { g 1 , . . . , g n } = span { x 1 , . . . , x n } .
Let
G (cid:107) =
( g 1 , . . . , g n ) ,
G ⊥
[Equation]
g n +1 , . .
. , g d ) , then
[Equation]
Recall the GD iterates at the k -th epoch:
[Equation]
Considering translating then rotating the variable as,
[Equation]
then we can reformulate the GD iterates as
[Equation]
We present the following lemma to reload the related notations regarding the parameterization u =
G (cid:62) ( w − w ∗ ) .
Lemma 14 (Reloading GD notations) .
Regarding reparametrization u = G (cid:62) ( w − w ∗ ) , we can reload the following related notations:
• Empirical loss and population loss are
[Equation]
• The hypothesis class is
[Equation]
• The α -level set is
[Equation]
• For u ∈ H S , the estimation error is
[Equation]
Moreover,
[Equation]
Proof.
See Section C.11.
The following lemma sovles GD iterates in Eq.
Lemma 15.
For t = 0 , . . .
, T ,
[Equation]
Proof.
This is by directly solving Eq.
(15) where Γ is diagonal.
Theorem 7 (Theorem 2, formal version) .
Suppose λ n + 2 nι < λ n − 1 .
Suppose u 0 is away from 0 .
Consider the GD iterates given by Eq.
(15) with learning rate scheme
[Equation]
Then for (cid:15) ∈ (0 , 1) , if k ≥ O (cid:0) log 1 (cid:15) (cid:1) , then we have
[Equation]
Proof.
For i = 1 , . . . , n , denote q i ( η ) = 1 − 2 γ i n · η, where η ∈ (cid:16) 0 ,
[Equation]
1 +2 nι (cid:17) .
Then we have 0 <
q i ( η ) < 1 since η <
[Equation]
where the second inequality follows from γ 1 < λ 1 + nι by Lemma 4.
Furthermore, since λ n + nι < λ n − 1 − nι , Lemma 4 gives us
[Equation]
16) which implies
[Equation]
Moreover,
[Equation]
is increasing, let q = max
η< n
[Equation]
1+2 nι f ( η ) , then q < 1 by our assumption on the learning rate.
From Lemma 15 we have
[Equation]
By the assumption that
[Equation]
we have
[Equation]
which further yields
[Equation]
By the above inequalities we have
[Equation]
Finally we note that
[Equation]
ni
[Equation]
≥ γ n since γ n is the smallest in { γ i } n i =1 .
Theorem 8 (Theorem 4 second part, formal version) .
Suppose λ n + 2 nι < λ n − 1 .
Suppose u 0 is away from 0 .
Consider the GD iterates given by Eq.
(15) with learning rate scheme
[Equation]
Then for (cid:15) ∈ (0 , 1) , if k ≥ O (cid:0) log 1 (cid:15) (cid:1) , , then GD outputs an M -suboptimal solution, where M = γγ n (1 − (cid:15) ) > 1 is a constant.
Proof.
Consider an α -level set where
[Equation]
From Lemma 15 we know L S ( u k ) is monotonic decreasing thus GD cannot terminate before the k -epoch, i.e., the output of GD is u k .
Thus
[Equation]
where we have M > 1 by letting (cid:15) <
1 − γ n γ 1 .
B. T HE DIRECTIONAL BIAS OF SGD WITH SMALL LEARNING RATE
We analyze SGD with small learning rate by repeating the arguments in previous two sections.
Let us denote X − n := ( x 1 , x 2 , . . .
, x n − 1 ) and
[Equation]
That is, P − n is the projection onto the column space of X − n and P n is the projection onto the orthogonal complement of the column space of X − n with respect to the column space of X .
Let us reload
[Equation]
Then
[Equation]
Following a routine check we can reload the following lemmas.
Lemma 16 (Variant of Lemma 10) .
Suppose 3 nι < λ n .
Suppose 0 < η < b λ 1 +3 nι .
Let π :=
{B 1 , . . .
, B m } be a uniform m partition of index set [ n ] , where n = mb .
Consider the following d × d matrix
[Equation]
Then for the spectrum of M (cid:62)− n M − n we have:
• 1 is an eigenvalue of M (cid:62)− n M − n with multiplicity being d − n + 1 ; moreover, the corresponding eigenspace is the column space of P n + P ⊥ .
• Restricted in the column space of P − n , the eigenvalues of M (cid:62)− n M − n are upper bounded by ( q − n ( η )) 2 < 1 , where
[Equation]
Proof.
This is by a routine check of the proof of Lemma 10.
Consider the projections of v k onto the column space of P − n and P n .
For simplicity we reload the following notations
[Equation]
The following lemma controls the update of A k and B k .
Lemma 17 (Variant of Lemma 11) .
Suppose 3 nι < λ n .
Suppose 0 < η < b λ 1 +3 nι .
Consider the k -th epoch of SGD iterates given by Eq.
Set the learning rate in this epoch to be constant η .
Denote
[Equation]
[Table]
nηι
[Table]
Then we have the following:
• A k +1 ≤ q − n ( η ) ·
A k + ξ ( η ) · B k .
• B k +1 ≤ q n ( η ) · B k + ξ ( η ) ·
A k . • B k +1 ≥ q n ( η ) · B k − ξ ( η ) · A k .
Proof.
This is by a routine check of the proof of Lemma 11.
Lemma 18 (Variant of Lemma 13) .
Suppose 3 nι < λ n and λ n + 4 nι < λ n − 1 .
Consider the SGD iterates given by Eq.
(10) with the following small learning rate scheme
[Equation]
Then for 0 < (cid:15) < 1 satisfying √ nι ≤ poly ( (cid:15) ) , if k 2 ≥ O (cid:0) log 1 (cid:15) (cid:1) , then A k 2 B k 2 ≤ (cid:15).
Proof.
From the assumption we have η (cid:48) <
b 2( λ 1 + nι ) and η (cid:48) <
b 2 λ n , thus ξ := ξ
[Equation]
nι
[Equation]
b (cid:107)
[Equation]
(cid:107)≤
[Equation] [Equation]
n (1 − 4 nι 2 ) b η (cid:48) , ( since (cid:107) P n ¯ x n (cid:107) 22 ≥ 1 − 4 nι 2 by reloading Lemma 3 ) <
[Equation]
(cid:48)
[Equation]
Moreover, by the gap assumption λ n + 4 nι < λ n − 1 we have
[Equation]
Therefore 0 < q − n < q n < 1 .
Thus we can set ξ = 4 η (cid:48) √ nι b to be small such that
[Equation]
Moreover, since √ nι ≤ poly ( (cid:15) ) and ξ = 4 η (cid:48) √ nι b , we have
[Equation]
Now we recursively show A k B k ≤ A 0 B 0 .
Clearly it holds for k = 0 .
Suppose A k B k ≤ A 0 B 0 , we consider
A k +1 B k +1 in the following
A k +
[Equation]
Lemma
[Equation]
assumption
[Equation]
( by Eq.
(22) and (
[Equation]
where in the last inequality we assume (cid:15) 2 <
A 0 B 0 .
Moreover, from the above we have
[Equation]
which implies
[Equation]
where we set k 2 ≥ O (cid:0) log 1 (cid:15) (cid:1) .
Next we prove the directional bias of SGD with small learning rate.
Theorem 9 (Theorem 3, formal version) .
Suppose 3 nι < λ n and λ n + 4 nι < λ n − 1 .
Suppose vis away from 0 .
Consider the SGD iterates given by Eq.
(10) with the following small learning rate scheme
[Equation]
Then for 0 < (cid:15) < 1 satisfying √ nι ≤ poly ( (cid:15) ) , if k 2 ≥ O (cid:0) log 1 (cid:15) (cid:1) , then
[Equation]
Proof.
First by Lemma 18 we have
[Equation]
Next by H = H n + H − n + H c we obtain
[Equation]
reloading
[Equation] [Equation]
by reloading Lemma 4 and Eq.
[Equation]
24) ≤ γ
[Equation]
γ n ·
[Equation]
since √ nι ≤ poly ( (cid:15) )
Finally, we note v (cid:62) k 2 Hv k 2 (cid:107) Pv k 2 (cid:107) 2 2 ≥ γ n since γ n is the smallest eigenvalue of H restricted in the column space of P .
Theorem 10 (Theorem 4 third part, formal version) .
Suppose 3 nι < λ n and λ n + 4 nι < λ n − 1 .
Suppose v 0 is away from 0 .
Consider the SGD iterates given by Eq.
(10) with the following small learning rate scheme
[Equation]
Then for 0 < (cid:15) < 1 such that √ nι ≤ poly ( (cid:15) ) , if k 2 ≥ O (cid:0) log 1 (cid:15) (cid:1) , then SGD outputs an Msuboptimal solution where M = γ 1 γ n (1 − (cid:15) ) > 1 is a constant.
Proof.
From Eq.
(10) and η (cid:48) <
12 λ 1 we know that restricted in the column space of P , the eigenvalues of M π is smaller than 1 , thus v k indeed converges to 0 .
Consider an α -level set where
[Equation]
Then
[Equation]
where we have M > 1 by letting (cid:15) <
1 − γ n γ 1 .
C P ROOF OF A UXILIARY L EMMAS IN S ECTIONS A AND B C. P ROOF OF L EMMA
Proof of Lemma 1.
Note that ¯ x i follows uniform distribution on the sphere S d − 1 .
Therefore, let ξ be a random variable following distribution χ 2 d distribution and deﬁne z i = ξ · ¯ x i , we have z i follows standard normal distribution in the d -dimensional space.
Then it sufﬁces to prove that |(cid:104) z i , z j (cid:105)| / ( (cid:107) z
i (cid:107) 2 (cid:107) z
j (cid:107) 2 ) ≤ ι for all i (cid:54) =
j .
First we will bound the inner product (cid:104)
z i , z j (cid:105) .
Note that we have each entry in z i is 1 -subgaussian, it can be direcly deduced that
[Equation]
is d -subexponential, where z ( k ) i denotes the k -th of the vector z i .
Then if follows that
[Equation]
Next we will lower bound (cid:107) z
i (cid:107) 2 .
Note that
[Equation]
Since z ( k )
i is 1 -subgaussian, we have (cid:107)
z i (cid:107) 2 − d is d -subexpoential, then
[Equation]
Finally, applying the union bound for all possible i, j ∈ [ n ] , we have with probability at least 1 − δ , the following holds for all i (cid:54) =
j ,
[Equation]
Assume d ≥ 4 log(2 n 2 /δ ) , we have (cid:107) z
i (cid:107) 2 ≥ d/ 2 .
Then it follows that
[Equation]
This completes the proof
[Equation]
C. P ROOF OF L EMMA
Proof of Lemma 3.
Similar to the proof of Lemma 1, we consider translating x 1 , . . .
, x n to z 1 , . . . ,
z n by introducing χ 2 d random variables.
Let Z − 1 =
( z 2 , . . .
, z n ) ∈ R d × ( n − 1) , in which each entry is i.i.d.
generated from Gaussian distribution N (0 , 1) .
Then we have
[Equation]
Then conditioned on ¯ x 1 , we have each entry in Z (cid:62)− 1 ¯ x 1 i.i.d.
follows N (0 , 1) .
Then it is clear that
[Equation]
Z
[Equation]
1 ¯ x
[Equation]
22 follows from χ 2 n − 1 distribution, implying that with probability at least 1 − δ (cid:48) , we have
[Equation]
Then by Corollary 5.35 in Vershynin (2010), we know that with probability at least 1 − δ (cid:48) it holds that √
[Equation]
Therefore, assume (cid:112) ( n − 1) + (cid:112) 2 log(2 /δ (cid:48) )
≤ √ d/ 8 , we have with probability at least 1 − δ (cid:48) (cid:13) (cid:13)
Z − 1 ( Z (cid:62)−
[Equation]
Combining with the upper bound of (cid:107) Z (cid:62)− 1 ¯ x 1 (cid:107) 2 , set δ (cid:48)
=
δ/ 2 , we have with probability at least 1 − δ that
[Equation]
where the last inequality follows from the deﬁnition of ι .
Then assume √ nι ≤ 1 / 4 , we are able to completes the proof of the ﬁrst argument.
Note that (cid:107) P 1 ¯ x 1 (cid:107)
+ (cid:107) P − 1 ¯ x 1 (cid:107)
= (cid:107) ¯ x 1 (cid:107)
= 1 , we have (cid:107) P − 1 ¯
[Equation]
This completes the proof of the second argument.
The third argument holds trivially by the construction of P ⊥ .
C. P ROOF OF L EMMA
Proof of Lemma 4.
Clearly XX (cid:62) ∈ R
d × d is of rank n and symmetric, thus XX (cid:62) has n real, nonzero (potentially repeated) eigenvalues, denoted as γ 1 , . .
, γ n in non-decreasing order.
Moreover, γ 1 , . . . , γ n are also eigenvalues of X (cid:62) X ∈ R n × n , thus it is sufﬁcient to locate the eigenvalues of
X (cid:62) X , where (cid:0) X (cid:62) X (cid:1) ij = x (cid:62) i x j .
We ﬁrst calculate the diagonal entry
[Equation]
Then we bound the off diagonal entries.
For j (cid:54) = i ,
[Equation]
where we use 0 < λ 1 , .
λ n ≤ 1 .
Thus we have
[Equation]
Finally our conclusions hold by applying Gershgorin circle theorem.
C. P ROOF OF L EMMA
Proof of Lemma 5.
The ﬁrst conclusion is clear since by construction, we have P − 1 P 1 =
P − 1 P ⊥ = 0 .
Note that H − 1 is a rank n − 1 symmetric matrix.
Let τ 2 , . . .
, τ n be the n − 1 non-zero eigenvalues of H − 1 .
Clearly, τ 2 , . .
. , τ n with τ 1 := 0 give the spectrum of
H
[Equation]
1 :=
[Equation]
P
[Equation]
We then bound τ 2 , . . .
, τ n by analyzing H (cid:48)− 1 .
From Lemma 3 we have (cid:107) P − 1 ¯ x 1 (cid:107) 2 ≤ 2 √ nι.
From Lemma 2 we have
[Equation]
Then we calculate the diagonal entries:
[Equation]
Then we bound the off diagonal entries.
Let j (cid:54) =
i .
Then at least one of them is not 1 .
Without loss of generality let i (cid:54) = 1 , which yields x i =
P − 1 x i by Lemma (2).
Thus (cid:104) x i , P 1 x j (cid:105) =
(cid:104) P − 1 x i , P 1 x j (cid:105) =
Thus we have
[Equation]
Thus we have
[Equation]
Finally, we set 4 nι 2 + 2 nι < λ n , so that the ﬁrst Geoshgorin disc does not intersect with the others, then Gershgorin circle theorem gives our second conclusion.
C. P ROOF OF L EMMA
Proof of Lemma 8.
For the empirical loss, it is clear that
[Equation]
where we use Lemma 4, Lemma 5, and Lemma 6.
For the population loss,
[Equation]
For the hypothesis class H S = (cid:8) w ∈ R d : P ⊥ w = P ⊥ w 0 (cid:9) , applying w − w ∗ = v and w 0 − w ∗ = v 0 , we obtain
[Equation]
For the α -level set, we note the optimal training loss is L ∗ S = inf v ∈H S L S ( v ) = 0 .
As for the estimation error, we note that inf v ∈H
[Equation]
L
[Equation]
v
[Equation]
inf
[Equation]
thus for v ∈ H S , we have
[Equation]
Finally, consider v ∈ V , i.e., nα = v (cid:62) XX (cid:62) v, thus
[Equation]
where γ 1 is the largest eigenvalue of the matrix XX (cid:62) and the inferior is attended by setting v parallel to the ﬁrst eigenvector of XX (cid:62) .
C. P ROOF OF L EMMA
Proof of Lemma 9.
From Eq.
(9) we have
[Equation]
Recall the following property of projection operators:
[Equation]
Moreover since x (cid:62) i P ⊥ v = 0 , we have
[Equation]
Applying P 1 to Eq.
(26) we have
[Equation]
Similarly applying P − 1 to Eq.
(26) we have
[Equation]
To sum up we have
[Equation]
Notice that if 1 / ∈ B j , i.e., x 1 is not used in the j -th step, then we claim
[Equation]
since H ( B j )
=
(cid:80) i ∈B j x i x (cid:62) i is composed by the data belonging to the column space of P − 1 .
Therefore if 1 / ∈ B j we have
[Equation]
C. P ROOF OF L EMMA
Proof of Lemma 10.
Clearly for each component in the production, the column space of P 1 + P ⊥ , which is ( n − d + 1) -dimensional, belongs to its eigenspace of eigenvalue 1 , which yields the ﬁrst claim.
In the following, we restrict ourselves in the column space of P − 1 .
Let us expand M − 1 :
[Equation]
We ﬁrst analyze matrix H − 1 .
Since H ( B j ) = (cid:80) i ∈B j x i x (cid:62) i and π = {B 1 , . . .
, B m } is a partition for index set [ n ] , we have
[Equation]
which is exactly the matrix we studied in Lemma 5, and from where we have H − 1 has eigenvalue zero (with multiplicity being n − d +1 ) in the column space of P 1 + P ⊥ , and restricted in the column space of P − 1 , the eigenvalues of H − 1 belong to ( λ n − nι, λ 2 + nι ) .
Then we analyze matrix C .
[Equation]
Remember that B i ∩ B j = ∅ for i (cid:54) =
j , thus x i (cid:48) (cid:54) =
x j (cid:48) for i (cid:48)
∈ B i and j (cid:48)
∈ B j .
Then from Lemmawe have,
[Equation]
Inserting this into Eq.
(27) we obtain
[Equation]
We can bound the Frobenius norm of the higher degree terms in matrix C in a similar manner; in sum for the Frobenius norm of C , we have
[Equation]
where for the second to the last inequality we notice that for f ( t )
= (1 + t ) m and t ∈ [0 , 12 n ] , we have f (cid:48)(cid:48) ( t ) = m ( m − 1)(1+ t ) m − 2 ≤ m ( m − 1)(1+ 12 m ) m − 2 ≤ m ( m − 1) ·√ e, which implies f ( t ) is ( m 2 √ e ) -smooth for t ∈ [0 , 12 m ] ; moreover, by the assumption that 3 nι < λ n and η <
[Equation]
λ n +3 nι , we can indeed verify that
[Equation]
Now we rephrase M (cid:62)− 1 M − 1 as
[Equation]
Restricting ourselves in the column space of P − 1 , the eigenvalues of H − 1 belong to ( λ n − nι, λ 2 + nι ) , thus the eigenvalues of (cid:0) I − 2 ηb H − 1 (cid:1) 2 are upper bounded by
[Equation]
where the last inequality is guaranteed by our assumptions on η and ι .
For simplicity we defer the veriﬁcation to the end of the proof.
Consider the following eigen decomposition I − 2 ηH −
= U diag ( µ 1 , . . . , µ n − 1 , 1 , . . .
, 1) U (cid:62) , where µ 1 , . . . , µ n − 1 ∈ ( − 1 , 1) by Eq.
Then we have
[Equation]
Therefore we can bound the Frobenius norm of D by
[Equation]
where the last inequality follows from 2 ηι ≤ 1 / (2 m ) proved in Eq.
Finally, applying Hoffman-Wielandt theorem with Eq.
(29), (30) and (31)
[Equation]
we conclude that, restricted in the column space of P − 1 , the eigenvalues of M (cid:62)− 1 M − 1 are upper bounded by
[Equation]
At this point we left to verify Eq.
(30) and
[Equation]
Clearly it sufﬁces to verify Eq.
(cid:12)(cid:12)(cid:12)(cid:12) 1 − 2 η b ( λ 2 + nι ) (cid:12)(cid:12)(cid:12)(cid:12) + 3 nηι b <⇔ 3 nι b η − 1 < 1 − 2( λ 2 + nι )
b η < 1 − 3 nι b η ⇔ (cid:26) 2 λ 2 − nι b η > 0 2 λ 2 +5 nι b η <⇐   η >2 λ 2 − nι >
η < b λ 2 +2 .
5 nι ⇐ (cid:40) 3 nι < λ n (since λ 2 ≥ λ n ) 0 < η < b λ 2 +3 nι
Similarly, we verify that (cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
λ n − nι
[Equation]
+ 3 nηι b <⇔ 3 nι b η − 1 < 1 − 2( λ n − nι )
b η < 1 − 3 nι b η ⇔ (cid:26) 2 λ n − 5 nι b η > 0 2 λ n + nι b η <⇐   η >2 λ n − 5 nι >η < b λ n +0 .
5 nι ⇐ (cid:40) 3 nι < λ n 0 < η < b λ 2 +
nι
(since λ 2 ≥ λ n )
These complete our proof.
C. P ROOF OF L EMMA
Proof of Lemma 11.
Note that during one epoch of SGD updates, x 1 is used for only once.
Without loss of generality, assume SGD uses x 1 at the l -th step, i.e., 1 ∈ B l and 1 / ∈ B j for j (cid:54)
=
l .
Recursively applying Lemma 9, we have (cid:18) P 1 v k,m +1 P − 1 v k,m + (cid:19) =
(cid:18) I  (cid:81) mj
= l +1 (cid:0)
I − 2 ηb P − 1 H ( B j ) P −(cid:1)(cid:19)
× (cid:18)
I − 2 ηb P 1 H ( B l ) P− 2 ηb P 1 H ( B l ) P −− 2 ηb P −
[Equation]
H ( B
[Equation]
) P
I − 2 ηb P −
[Equation]
H ( B
[Equation]
) P −(cid:19) × (cid:18)
I  (cid:81) l − 1 j =1 (cid:0) I − 2 ηb P − 1 H ( B j ) P − 1 (cid:1)(cid:19) ×
(cid:18) P 1 v k,
P − 1 v k,(cid:19)
Let v k +
= v k,m +1 , v k
= v k, 1 and
M l := I − 2 η b P − 1 H ( B l ) P −
M >l := m (cid:89)
j = l + (cid:18)
I − 2 η b P − 1 H ( B j ) P −(cid:19)
M <l :=
l − 1 (cid:89)
j = (cid:18)
I − 2 η b P − 1 H ( B j ) P −(cid:19)
M − 1 := M >l · M l · M <l = m (cid:89)
j = (cid:18)
I − 2 η b P − 1 H ( B j ) P −(cid:19) then we have (cid:18) P 1 v k +
P − 1 v k + (cid:19) =
(cid:18)
I
M >l (cid:19)
[Equation]
In the following we bound the norm of each entries in the above coefﬁcient matrix.
According to Lemma 5, we have the eigenvalues of P − 1 H ( B j ) P − 1 are upper bounded by λ 2 + nι.
Thus the assumption η < b λ 2 +2 nι yields
[Equation]
which further yields
[Equation]
On the other hand notice that P 1 x i = 0 for i (cid:54) = 1 , thus
[Equation]
=
P − 1 x 1 x (cid:62) 1 P 1 , which yield
[Equation]
36) where the last inequality is from Lemma 3 and λ 1 = (cid:107) x 1 (cid:107) 2 ≤ 1 .
Eq.
(35) and (36) imply
[Equation]
Next, by P 1 x i = 0 for i (cid:54) = 1 we have
[Equation]
from where we know (cid:107) P 1 x 1 (cid:107) 22 is the only non-zero eigenvalue of the rank- 1 matrix P 1 H ( B l ) P 1 , and the corresponding eigenspace is the column space of P 1 .
Therefore 1 − 2 ηb (cid:107) P 1 x 1 (cid:107) 22 is an eigenvalue of the matrix I − 2 ηb P 1 H ( B l ) P 1 , and the corresponding eigenspace is the column space of P 1 , which implies
[Equation]
Finally, according to Lemma 10, we have, restricted in the column space of P − 1 , the right eigenvalues of M − 1 is upper bounded by ( q − 1 ( η )) 2 , which implies
[Equation]
Note we have q − 1 ( η ) < 1 by Lemma 10.
Combining Eq.
(34) with Eq.
(37), (38), (39), and letting B k := (cid:107) P 1 v k (cid:107) 2 , A k := (cid:107) P − 1 v k (cid:107) 2 , we obtain
[Equation]
C. P ROOF OF L EMMA
Proof of Lemma 12.
Let
[Equation]
Then for 0 < k ≤ k 1 , Lemma 11 gives us
[Equation]
[Equation]
where “ ≤ ” means “entry-wisely smaller than”.
Let θ, ρ − 1 , ρ 1 determine the eigen decomposition of the coefﬁcient matrix, i.e.,
[Equation]
Then Eq.
(42) and Eq.
(43) yield
[Equation]
We claim the following inequalities hold by our assumptions:
[Equation]
The veriﬁcation of Eq.
(45) is left later.
In the following we prove the conclusions using Eq.
We ﬁrst bound A k 1 using Eq.
(44) and Eq.
[Equation]
which justiﬁes the ﬁrst conclusion.
In addition we can obtain an uniform upper bound for A k for k = 0 , 1 , . . .
, k 1 :
[Equation]
Next we bound B k 1 using Eq.
(44) and Eq.
[Equation]
which justiﬁes the second conclusion.
We proceed to derive the uniform lower bound for B k for k = 0 , 1 , . . .
k 1 .
We do it by induction.
For k = 0 , by assumption we have B 0 ≥ β 0 .
Suppose B k − 1 ≥ β 0 , then by Eq.
(41), (45) and (46) we have
[Equation]
which justiﬁes the third conclusion
[Equation]
Veriﬁcation of Eq. (45)
From Eq.
(43) and Gershgorin circle theorem we have
[Equation]
Moreover, reformatting Eq.
(43) as
[Equation]
we then have
[Equation]
For Eq.
(45a)
[Equation]
using Eq.
(47) it sufﬁces to show
[Equation]
(49a
[Equation]
Notice the deﬁnitions of q 1 , q − 1 and ξ are given in Eq.
Firstly, Eq. (49c) holds trivially when √ n > 4 / 3 .
Secondly, for Eq.
(49b), noticing that ξ = 4 η √ nι b ≤ ηnιb when n ≥ 16 , it sufﬁces to show
[Equation]
⇔
[Equation]
⇔  
[Equation]
b
[Equation]
b η <
[Equation]
⇐   η >λ 2 − nι >λ n − 3 nι >η < b λ 2 +3 nι η < b λ n + nι ⇐ (cid:40) 3 nι < λ n 0 < η < b λ 2 +3 nι which are given in assumptions.
Thirdly, for Eq. (49c) it sufﬁces to show
[Equation]
( since nι < 1) which are given in assumptions.
For Eq.
(45b), it sufﬁces to show set
[Equation]
as given in assumptions.
For Eq.
(45c), using Eq.
(45b) it sufﬁces to show
[Equation]
⇐ √ nι ≤ poly ( (cid:15)β ) .
( by Eq.
For Eq.
(45c), it sufﬁces to show ξ
[Equation]
A
[Equation] [Equation]
(cid:15)β⇐ √ nι
[Equation]
O (1) .
( by Eq.
C. P ROOF OF L EMMA
Proof of Lemma 13.
Let ξ (cid:48) :
= ξ ( η (cid:48) )
= 4 η (cid:48) √ nι b , q (cid:48) 1 :=
q 1 ( η (cid:48) )
= (cid:12)(cid:12)(cid:12)(cid:12) 1 − 2 η (cid:48)
λ 1 b (cid:107) P 1 ¯ x 1 (cid:107) 22 (cid:12)(cid:12)(cid:12)(cid:12) , q (cid:48)− 1 :=
q − 1 ( η (cid:48) )
= max (cid:26)(cid:12)(cid:12)(cid:12)(cid:12) 1 − 2 η (cid:48) b ( λ 2 + nι ) (cid:12)(cid:12)(cid:12)(cid:12)
+ 3 nη (cid:48)
ι b , (cid:12)(cid:12)(cid:12)(cid:12) 1 − 2 η (cid:48) b ( λ n − nι ) (cid:12)(cid:12)(cid:12)(cid:12) + 3 nη (cid:48)
ι b (cid:27) .
Then for k 1 < k ≤ k 2 , Lemma 11 gives us (cid:18)
A k
B k (cid:19) ≤ (cid:18) q (cid:48) −ξ (cid:48) ξ (cid:48) q (cid:48)(cid:19) · (cid:18)
A k −
B k −(cid:19)
[Equation]
where “ ≤ ” means “entry-wisely smaller than”.
Denote
[Equation]
We claim the following inequalities hold by our assumptions:
[Equation]
The veriﬁcation of Eq.
(53) is left later.
In the following we prove the main conclusions in the lemma using Eq.
We proceed by induction.
Clearly the conclusions are true for k = k 1 .
Suppose for k 1 , . . .
, k − 1 , the conclusions are also true.
Then the induction assumptions give us
[Equation]
where the last inequality is due to B ≥ B k 1 ≥ β 0 > β .
Then by Eq.
(51) we have
[Equation]
Also by Eq.
(51) we have
[Equation]
[Equation]
Veriﬁcation of Eq.
Notice the deﬁnitions of q (cid:48) 1 , q (cid:48)− 1 and ξ (cid:48) are given in Eq.
Recall q (cid:48)− 1 < 1 is already justiﬁed by the choice of learning rate η (cid:48) <
1 λ 2 +
nι
(e.g., see Lemma 10), thus for Eq.
(53a), it sufﬁces to show
[Equation]
which are given in assumptions.
For Eq.
(53b), it sufﬁces to show
[Equation]
which is implied by nι ≤ poly ( (cid:15)β ) .
For Eq.
(53c), it sufﬁces to show
[Equation]
by Eq.
We complete our proof.
C. P ROOF OF L EMMA
Proof of Lemma 14.
For the empirical loss,
[Equation]
For the population loss,
[Equation]
For the hypothesis class H S = (cid:8) w ∈ R d : P ⊥ w = P ⊥ w 0 (cid:9) , Note P ⊥ G = diag (0 , . . .
Apply w − w ∗ =
Gu and notice w 0 − w ∗ =
Gu 0 , then we obtain
[Equation]
= u
[Equation]
i ) 0 , for i = n + 1 , . . .
, d (cid:9) .
For the level set, we only need to note that L ∗ S = inf u ∈H S L S ( u ) = 0 .
As for the estimation error, we note that
[Equation]
, thus for u ∈ U , we have
[Equation]
=
[Equation]
n
[Equation]
i = (cid:16) u ( i ) (cid:17) 2 + µ d (cid:88) i = n + (cid:16) u ( i ) (cid:17) 2 − µ d (cid:88) i = n + (cid:16) u ( i )
(cid:17)= µ n (cid:88) i
= (cid:16) u
( i ) (cid:17) 2 .
Now consider u ∈ U , i.e., 1 n (cid:80) ni
=
γ i (cid:0) u ( i ) (cid:1)
= α, then
[Equation]
(cid:88)
i
[Equation]
(cid:16)
[Equation] [Equation]
) (cid:17)
[Equation]
γ 1 , where the inferior is attended when, e.g., u (1) = ± (cid:113) nαγ 1 and u (2) = · · · = u ( n ) = 0 .
D E XPERIMENTS D ETAILS
In this section, we describe the details for our experiments.
D. 2-D EXAMPLE
This part corresponds to Section 3 and Figure 1.
The two training data points are
[Equation]
and the corresponding individual losses are
[Equation]
Then the training loss is
[Equation]
We initialize the algorithms from w 0 = (0 . 6 , 0 . 6) (cid:62) .
Two kinds of learning rate regime are considered.
In the small learning rate regime, the learning rate is
[Equation]
=
[Equation] [Equation]
In the moderate learning rate regime, the learning rate is η
[Equation]
(cid:26) 1 .
[Equation]
κ, k =
[Equation]
[Equation] [Equation]
1 /κ, k = 101 , . . .
For SGD, the mini-batch size is 1 .
D. L INEAR REGRESSION ON SYNTHETIC DATA
This part corresponds to Section 4 and Figure 2(a).
The model is an overparameterized linear model, with d = 10 4 and n = 100 .
The true parameter w ∗ is randomly drawn from an d -dimensional Gaussian distribution, N (0 , 0 .
1 · I d × d ) .
We then randomly draw n = 100 samples from the d -dimensional space as described in Section 4, where ζ ∼ U ([0 . 5 , 1]) .
We initialize the algorithms from zero.
We consider two kinds of learning rate regimes.
The small learning rate scheme is speciﬁed by
[Equation]
The moderate learning rate scheme is speciﬁed by
[Equation]
For SGD, the mini-batch size is 1 .
D. N EURAL NETWORK ON A SUBSET OF F ASHION MNIST
This part corresponds to Figure 2(b) and Figure 3.
Model We use a LeNet-alike convolutional network:
[Equation]
⇒ linear ⇒ output .
The ﬁrst convolutional layer uses 5 × 5 kernels with 10 channels and no padding and the second convolutional layer uses 5 × 5 kernels with 16 channels and no padding.
The number of hidden units between the two fully connected layers are 60 .
Dataset https://github.com/zalandoresearch/fashion-mnist
We randomly choose 2 , 000 original test data as our training set, and use the 60 , 000 original training data as our test set.
Thus we have 2 , 000 training data and 60 , 000 test data.
We scale the image data to [0 , 1] .
Algorithms We randomly initialize the algorithms from a Gaussian distribution with zero mean and standard deviation 0 .
We consider two kinds of learning rate regimes.
The small learning rate scheme is speciﬁed by
[Equation]
The moderate learning rate scheme is speciﬁed by
[Equation]
For SGD, the mini-batch size is 25 .
For both GD and SGD, the weight decay parameter is set as 0 . 002 .
[Figure]
Relative Rayleigh quotient We discuss the relative Rayleigh quotients calculated in Figure 2(b).
Unlike the linear regression model where the Hessian is a constant, for neural networks the loss function is non-convex.
In other words, not only there are multiple local minima, but also
the
Hessian varies at different points.
Therefore, a direct comparison in terms of the Rayleigh quotients for the iterates of different algorithms makes little sense.
Instead, we consider the relative Rayleigh quotient,
i
[Figure]
e., the Rayleigh quotient normalized by the maximum absolute eigenvalue of the Hessian at that point.
Mathematically, the relative Rayleigh quotient is deﬁned as
[Equation]
[Table]
where ∇ L ( w ) / (cid:107)∇ L ( w ) (cid:107) 2 is the convergence direction of gradient methods, and (cid:107)∇ 2 L ( w ) (cid:107) 2 is the operator norm of the Hessian, i.e., its maximum absolute eigenvalue.
Note that it is computationally hard in practice to project a parameter onto the data manifold, thus we use the vanilla convergence direction instead of the projected one in the above deﬁnition of the relative Rayleigh quotient.
Since our goal is to compare the relative Rayleigh quotient between different algorithms, this simpliﬁcation would not affect our conclusions.
We obtain the maximum absolute eigenvalue of the Hessian by running power method for 5 iterates.
Additional experiments: GD and SGD with increasing learning rates We conduct numerical experiments of training neural networks on a subset of FashionMNIST dataset for SGD and GD with different learning rates η ∈ { 0 .
The test accuracy results are displayed in Figure 4.
Several conclusions can be drawn from the plots.
(1) For a learning rate over η = 0 .
08 , SGD cannot converge (thus only gives about 10% test accuracy).
SGD generalizes best with a moderate learning rate η = 0 .
(2) GD fails to converge when η = 0 .
Also, as the learning rate increases, the test accuracy of GD ﬁrst increases then decreases; but even at its peak ( η = 0 . 01 ), GD performs worse than SGD with a moderate learning rate.
{+Paired t-test for Figure 
+}
We also conduct statistical test to show SGD with moderate learning rate is signiﬁcantly better than the other baselines.
Recall that the experiments are repeated for 10 runs, at each run, we ﬁrst ﬁx a random seed, then run the four algorithms (GD/SGD with small/moderate learning rate) under the same seed.
In Table 1, we report the complete results from the 10 runs.
By running a paired t-test at the 5% signiﬁcance level, we ﬁnd that SGD with moderate learning rate is signiﬁcantly better than GD with moderate learning rate ( p -value = 0 . 0043 ).
Similarly, SGD with moderate learning rate is signiﬁcantly better than SGD with small learning rate ( p -value = 0 . 012 ), and is also signiﬁcantly better than GD with small learning rate ( p -value = 0 . 0095 ).
[Table]