S ELF -S TABILIZATION : T HE I MPLICIT B IAS OF G RADIENT D ESCENT AT THE E DGE OF S TABILITY
A BSTRACT
Traditional analyses of gradient descent show that when the largest eigenvalue of the Hessian, also known as the sharpness S ( θ ) , is bounded by 2 /η , training is "stable" and the training loss decreases monotonically.
Recent works, however, have observed that this assumption does not hold when training modern neural networks with full batch or large batch gradient descent.
Most recently, Cohen et al.
(2021) detailed two important phenomena.
The first, dubbed progressive sharpening , is that the sharpness steadily increases throughout training until it reaches the instability cutoff 2 /η .
The second, dubbed edge of stability is that the sharpness hovers at 2 /η for the remainder of training while the loss continues decreasing, albeit non-monotonically.
We demonstrate that, far from being chaotic, the dynamics of gradient descent at the edge of stability can be captured by a cubic Taylor expansion: as the iterates diverge in direction of the top eigenvector of the Hessian due to instability, the cubic term in the local Taylor expansion of the loss function causes the curvature to decrease until stability is restored.
This property, which we call self-stabilization is a general property of gradient descent and explains its behavior at the edge of stability.
A key consequence of self-stabilization is that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint S ( θ ) ≤ 2 /η .
Our analysis provides precise predictions for the loss, sharpness and deviation from the PGD trajectory throughout training, which we verify both empirically in a number of standard settings and theoretically under mild conditions.
Our analysis uncovers the mechanism for gradient descent’s implicit bias towards stability
I NTRODUCTION 1. G RADIENT D ESCENT AT THE E DGE OF S TABILITY
Almost all neural networks are trained using a variant of gradient descent, most commonly stochastic gradient descent (SGD) or ADAM (Kingma & Ba, 2015).
When deciding on an initial learning rate, many practitioners rely on intuition drawn from classical optimization.
In particular, the following classical lemma, known as the "descent lemma," provides a common heuristic for choosing a learning rate in terms of the sharpness of the loss function:
Definition 1.
Given a loss function L ( θ ) , the sharpness at θ is defined to be S ( θ ) :
=
λ max ( ∇ 2 L ( θ )) .
When this eigenvalue is unique, the associated eigenvector is denoted by u ( θ ) .
Lemma 1 (Descent Lemma) .
Assume that S ( θ ) ≤ ℓ for all θ .
If θ t +1 =
θ t − η ∇ L ( θ t ) ,
[Equation]
Here, the loss decrease is proportional to the squared gradient, and is controlled by the quadratic η (2 − ηℓ ) in η .
This function is maximized at η = 1 /ℓ , a popular learning rate criterion.
For any η < 2 /ℓ , the descent lemma guarantees that the loss will decrease.
As a result, learning rates below 2 /ℓ are considered "stable" while those above 2 /ℓ are considered "unstable."
For quadratic loss functions, e.g. from linear regression, this is tight.
Any learning rate above 2 /ℓ provably leads to exponentially increasing loss.
However, it has recently been observed that in neural networks, the descent lemma is not predictive of the optimization dynamics.
Recently, Cohen et al.
(2021) observed two important phenomena for gradient descent, which made more precise similar observations in Jastrz˛ebski et al. Jastrzebski et al. (2020) for SGD:
Progressive Sharpening Throughout most of the optimization trajectory, the gradient of the loss is negatively aligned with the gradient of sharpness, i.e. ∇ L ( θ ) · ∇ S ( θ ) < 0 .
As a result, for any reasonable learning rate η , the sharpness increases throughout training until it reaches S ( θ ) = 2 /η .
Edge of Stability Once the sharpness reaches 2 /η (the “break-even” point in Jastrzebski et al. , it ceases to increase and remains around 2 /η for the rest of training.
The descent lemma no longer guarantees the loss decreases but the loss still continues decreasing, albeit non-monotonically.
[Figure]
1. S ELF - STABILIZATION : T HE I MPLICIT B IAS OF I NSTABILITY
In this work we explain the second stage, "edge of stability.
" We identify a new implicit bias of gradient descent which we call self-stabilization .
Self-stabilization is the mechanism by which the sharpness remains bounded around 2 /η , despite the continued force of progressive sharpening, and by which the gradient descent dynamics do not diverge, despite instability.
Unlike progressive sharpening, which is only true for specific loss functions (e.g. those resulting from neural network optimization (Cohen et al., 2021)), self stabilization is a general property of gradient descent.
Traditional non-convex optimization analyses involve Taylor expanding the loss function to second order around θ to prove loss decrease when η ≤ 2 /S ( θ ) .
When this is violated, the iterates diverge exponentially in the top eigenvector direction, u , thus leaving the region in which the loss function is locally quadratic.
Understanding the dynamics thus necessitates a cubic Taylor expansion.
Our key insight is that the missing term in the Taylor expansion of the gradient after diverging in the u direction is ∇ 3 L ( θ )( u, u ) , which is conveniently equal to the gradient of the sharpness at θ :
Lemma 2 (Self-Stabilization Property) .
If the top eigenvalue of ∇ 2 L ( θ ) is unique, then the sharpness S ( θ ) is differentiable at θ and ∇ S ( θ )
= ∇ 3 L ( θ )( u ( θ ) , u ( θ )) .
As the iterates move in the negative gradient direction, this term has the effect of decreasing the sharpness .
The story of self-stabilization is thus that as the iterates diverge in the u direction, the strength of this movement in the −∇ S ( θ ) direction grows until it forces the sharpness below 2 /η , at which point the iterates in the u direction shrink and the dynamics re-enter the quadratic regime.
This negative feedback loop prevents both the sharpness S ( θ ) and the movement in the top eigenvector direction, u , from growing out of control.
As a consequence, we show that gradient descent implicitly solves the constrained minimization problem :
[Equation]
Specifically, if the stable set M is defined by M :=
{ θ : S ( θ ) ≤ 2 /η and ∇ L ( θ ) · u ( θ ) = 0 } 1 then the gradient descent trajectory { θ t } tracks the following projected gradient descent trajectory { θ † t } which solves the constrained problem (Barber & Ha, 2017):
[Equation]
Our main contributions are as follows.
First, we explain self-stabilization as a generic property of gradient descent for a large class of loss functions, and provide precise predictions for the loss, sharpness, and deviation from the constrained trajectory { θ † t } throughout training (Section 4).
Next, we prove that under mild conditions on the loss function (which we verify empirically for standard architectures and datasets), our predictions track the true gradient descent dynamics up to higher order error terms (Section 5).
Finally, we verify our predictions by replicating the experiments in
Cohen et al.
and show that they model the true gradient descent dynamics (Section 6).
R ELATED W ORK
Xing et al.
(2018) observed that for some neural networks trained by full-batch gradient descent, the loss is not monotonically decreasing. Wu et al. (2018) remarked that gradient descent cannot converge to minima where the sharpness exceeds 2 /η but did not give a mechanism for avoiding such minima.
Lewkowycz et al.
(2020) observed that when the initial sharpness is larger than 2 /η , gradient descent "catapults" into a stable region and eventually converges.
Jastrz˛ebski et al.
(2019) studied the sharpness along stochastic gradient descent trajectories and observed an initial increase (i.e. progressive sharpening) followed by a peak and eventual decrease.
They also observed interesting relationships between the dynamics in the top eigenvector direction and the sharpness.
Jastrzebski et al.
(2020) conjectured a general characterization of stochastic gradient descent dynamics asserting that the sharpness tends to grow but cannot exceed a stability criterion given by their eq (1), which reduces to S ( θ ) ≤ 2 /η in the case of full batch training.
Cohen et al. (2021) demonstrated that for the special case of (full batch) gradient descent training, the optimization dynamics exhibit a simple characterization. First, the sharpness rises until it reaches S ( θ ) = 2 /η at which point the dynamics transition into an “edge of stability” (EOS) regime where the sharpness oscillates around 2 /η
and the loss continues to decrease, albeit non-monotonically.
Recent works have sought to provide theoretical analyses for the EOS phenomenon.
Ma et al.
(2022) analyzes EOS when the loss satisfies a "subquadratic growth" assumption.
Ahn et al.
(2022) argues that unstable convergence is possible when there exists a "forward invariant subset" near the set of minimizers.
Arora et al.
(2022) analyzes progressive sharpening and the EOS phenomenon for normalized gradient descent close to the manifold of global minimizers.
Lyu et al.
(2022) uses the EOS phenomenon to analyze the effect of normalization layers on sharpness for scale-invariant loss functions.
Chen & Bruna (2022) show global convergence despite instability for certain 2D toy problems and in a 1-neuron student-teacher setting.
The concurrent work Li et al.
(2022b) proves progressive sharpening for a two-layer network and analyzes the EOS dynamics through four stages similar to ours using the norm of the output layer as a proxy for sharpness.
Beyond the EOS phenomenon itself, prior work has also shown that SGD with large step size or small batch size will lead to a decrease in sharpness (Keskar et al., 2017; Jastrzebski et al., 2017;
Jastrz˛ebski et al., 2019; Jastrzebski et al., 2020).
Gilmer et al.
(2021) also describes connections between EOS, learning rate warm-up, and gradient clipping.
At a high level, our proof relies on the idea that oscillations in an unstable direction prescribed by the quadratic approximation of the loss cause a longer term effect arising from the third-order
Taylor expansion of the dynamics.
This overall idea has also been used to analyze the implicit regularization of SGD (Blanc et al., 2020;
Damian et al., 2021;
Li et al., 2022a).
In those settings, oscillations come from the stochasticity, while in our setting the oscillations stem from instability.
S ETUP
We denote the loss function by L ∈ C 3 ( R d ) .
Let θ ∈ R
d
follow gradient descent with learning rate η , i.e. θ t +1 :
= θ t − η ∇ L ( θ t ) .
Recall that M :=
{ θ : S ( θ ) ≤ 2 /η and ∇ L ( θ ) · u ( θ ) = 0 } is the set of stable points and proj M :=
arg min θ ′ ∈M
∥
θ − θ ′ ∥ is the orthogonal projection onto M .
For notational simplicity, we will shift time so that θ 0 is the first point such that S (proj M ( θ ))
= 2 /η .
The constrained trajectory θ † is initialized with θ † 0 :=
proj M ( θ 0 ) after which it follows eq.
Our key assumption is the existence of progressive sharpening along the constrained trajectory, which is captured by the progressive sharpening coefficient α
( θ ) :
= −∇ L ( θ ) · ∇ S ( θ ) :
Assumption 1 (Progressive Sharpening) .
Let α ( θ ) :
= −∇ L ( θ ) · ∇ S ( θ ) .
Then α ( θ † t ) > 0 . [Figure]
We focus on the regime in which there is a single unstable eigenvalue, and we leave understanding multiple unstable eigenvalues to future work.
We thus make the following assumption on ∇ 2 L ( θ † t ) :
Assumption 2 (Eigengap) .
For some absolute constant c < 2 we have λ 2 ( ∇ 2 L ( θ † t ))
<
c/η .
T HE S ELF - STABILIZATION P ROPERTY OF G RADIENT D ESCENT
In this section, we derive a set of equations that predict the displacement between the gradient descent trajectory { θ t } and the constrained trajectory { θ † t } .
Viewed as a dynamical system, these equations give rise to a negative feedback loop, which prevents both the sharpness and the displacement in the unstable direction from diverging.
These equations also allow us to predict the values of the sharpness and the loss throughout the gradient descent trajectory.
4. T HE F OUR S TAGES OF E DGE OF S TABILITY : A H EURISTIC D ERIVATION
The analysis in this section proceeds by a cubic Taylor expansion around a fixed reference point θ ⋆ :
= θ † 0 .
For notational simplicity, we will define the following quantities at θ ⋆ :
[Equation]
where α = −∇ L ·∇ S > 0 is the progressive sharpening coefficient at θ ⋆ .
For simplicity, in Sectionwe assume that ∇ S ⊥ u and ∇ L, ∇ S ∈ ker( ∇ 2 L ) , and ignore higher order error terms.
3 Our main argument in Section 5 does not require these assumptions and tracks all error terms explicitly.
We want to track the movement in the unstable direction u and the direction of changing sharpness ∇ S , and thus define x t := u · ( θ t − θ ⋆ ) and y t := ∇ S · ( θ t − θ ⋆ ) .
Note that y t is approximately equal to the change in sharpness from θ ⋆ to θ t , since Taylor expanding the sharpness yields
[Equation]
At a high level, the mechanism for edge of stability can be described in 4 stages (see Figure 1):
Stage 1: Progressive Sharpening While x, y are small, ∇ L ( θ t ) ≈ ∇ L .
In addition, because ∇ L · ∇ S < 0 , gradient descent naturally increases the sharpness at every step.
In particular,
[Equation]
The sharpness therefore increases linearly with rate ηα .
Stage 2: Blowup As x t measures the deviation from θ ⋆ in the u direction, the dynamics of x t can be modeled by gradient descent on a quadratic with sharpness S ( θ t ) ≈
2 /η + y t .
In particular, the rule for gradient descent on a quadratic gives
[Equation]
When the sharpness exceeds 2 /η , i.e. when y t > 0 , | x t | begins to grow exponentially.
[Figure]
Stage 3: Self-Stabilization Once the movement in the u direction is sufficiently large, the loss is no longer locally quadratic.
Understanding the dynamics necessitates a third order Taylor expansion.
The missing cubic term in the Taylor expansion of ∇ L ( θ t ) is ∇ 3 L ( u, u ) x 2 t 2 = ∇ S x 2 t 2 by Lemma 2.
This biases the optimization trajectory in the −∇ S direction, which decreases sharpness.
Recalling β
= ∥∇ S ∥ 2 , the new update for y becomes:
[Equation]
Therefore once x t > (cid:112) 2 α/β , the sharpness begins to decrease and continues to do so until the sharpness goes below 2 /η and the dynamics return to stability.
Stage 4: Return to Stability At this point | x t | is still large from stages 1 and 2.
However, the selfstabilization of stage 3 eventually drives the sharpness below 2 /η so that y t < 0 .
Because the rule for gradient descent on a quadratic with sharpness S ( θ t ) = 2 /η
+ y t < 2 /η is x t +1 ≈ − (1+ ηy t ) x t , | x t | begins to shrink exponentially and the process returns to stage 1.
Combining the update for x t , y t in all four stages, we obtain the following simplified dynamics:
[Equation]
where we recall α = −∇ L · ∇ S is the progressive sharpening coefficient and β = ∥∇ S ∥ 2 .
A NALYZING THE SIMPLIFIED DYNAMICS
We now analyze the dynamics in eq.
First, note that x t changes sign at every iteration, and that, x t +1 ≈ − x t due to the instability in the u direction.
While eq.
(3) cannot be directly modeled by an
ODE due to these rapid oscillations, we can instead model | x t | , y t , whose update is controlled by η .
As a consequence, we can couple the dynamics of | x t | , y t to the following ODE X ( t ) , Y ( t ) :
[Equation]
This system has the unique fixed point ( X, Y ) =
( δ, 0) where δ :=
(cid:112) 2 α/β .
We also note that this
ODE can be written as a Lotka-Volterra predator-prey model after a change of variables, which is a classical example of a negative feedback loop.
In particular, the following quantity is conserved:
Lemma 3.
Let h ( z ) :
= z − log z − 1 .
Then g ( X ( t ) , Y ( t )) :
= h (cid:16) βX ( t ) 2 2 α (cid:17) + Y ( t ) 2 α is conserved.
Proof.
[Equation]
As a result we can use the conservation of g to explicitly bound the size of the trajectory:
Corollary 1.
For all t ,
The fluctuations in sharpness are ˜ O ( √ α ) , while the fluctuations in the unstable direction are ˜ O ( δ ) .
Moreover, the normalized displacement in the ∇ S direction, i.e. ∇ S ∥∇ S ∥ · ( θ − θ ⋆ ) is also bounded by ˜ O
( δ ) , so the entire process remains bounded by ˜ O ( δ ) .
Note that the fluctuations increase as the progressive sharpening constant α grows, and decrease as the self-stabilization force β grows.
4. R ELATIONSHIP WITH THE CONSTRAINED TRAJECTORY θ † t
Equation (3) completely determines the displacement θ t − θ ⋆ in the u, ∇ S directions and Section 4.
shows that these dynamics remain bounded by ˜ O
( δ ) where δ = (cid:112) 2 α/β .
However, progress is still made in all other directions.
Indeed, θ t evolves in these orthogonal directions by − ηP ⊥ u, ∇ S ∇ L at every step where P ⊥ u, ∇ S is the projection onto this orthogonal subspace.
This can be interpreted as first taking a gradient step of − η ∇ L and then projecting out the ∇ S direction to ensure the sharpness does not change.
Lemma 13, given in the Appendix, shows that this is precisely the update for θ † t (eq.
up to higher order terms.
The preceding derivation thus implies that ∥ θ t − θ † t ∥ ≤ ˜ O
( δ ) and that this ˜ O ( δ ) error term is controlled by the self-stabilizing dynamics in eq.
T HE P REDICTED D YNAMICS AND T HEORETICAL R ESULTS
We now present the equations governing edge of stability for general loss functions.
5. N OTATION
Our general approach Taylor expands the gradient of each iterate θ t around the corresponding iterate θ † t of the constrained trajectory.
We define the following Taylor expansion quantities at θ † t :
Definition 2 (Taylor Expansion Quantities at θ † t ) .
[Equation]
( θ † t ) , u t := u ( θ † t ) .
Furthermore, for any vector-valued function v ( θ ) , we define v ⊥ t :=
P ⊥ u t v ( θ † t ) where P ⊥ u t is the projection onto the orthogonal complement of u t .
We also define the following quantities which govern the dynamics near θ ⋆t .
Definition 3.
Let α t := −∇ L t · ∇ S t , β t :=
(cid:13)(cid:13) ∇ S ⊥ t (cid:13)(cid:13) 2 , and δ t :=
(cid:113)
α t β t .
Furthermore, we define
[Equation]
Recall that α t is the progressive sharpening force, β t is the strength of the stabilization force, and δ t controls the size of the deviations from θ † t and was the fixed point in the x direction in Section 4.2.
The scalars β s → t capture the effect of the interactions between ∇ S and the Hessian.
5. T HE EQUATIONS GOVERNING EDGE OF STABILITY
We now introduce the equations governing edge of stability.
We track the following quantities:
[Equation]
Our predicted dynamics directly predict the displacement v t and the full definition is deferred to
Appendix C. However, they have a relatively simple form in the u t , ∇ S ⊥ t directions:
Lemma 4 (Predicted Dynamics for x, y ) .
Let ⋆ v t denote our predicted dynamics (defined in Appendix C).
Letting ⋆ x t = u t · ⋆ v t and ⋆ y t = ∇ S ⊥ t · ⋆ v t , we have
[Equation]
⋆ x t and
[Equation]
Note that when β s → t are constant, our update reduces to the simple case discussed in Section 4, which we analyze fully.
When x t is large, eq.
(5) demonstrates that there is a self-stabilization force which acts to decrease y t ; however, unlike in Section 4, the strength of this force changes with t .
5. C OUPLING T HEOREM
We now show that, under a mild set of assumptions which we verify to hold empirically in Appendix E, the true dynamics are accurately governed by the predicted dynamics.
This lets us use the predicted dynamics to predict the loss, sharpness, and the distance to the constrained trajectory θ † t .
Our errors depend on the unitless quantity ϵ , which we verify is small in Appendix E.
Definition 5.
Let ϵ t :
=
η √ α t and ϵ :=
sup t ϵ t .
To control Taylor expansion errors, we require upper bounds on ∇ 3 L and its Lipschitz constant:
Assumption 3.
Let ρ 3 , ρ 4 to be the minimum constants such that for all θ , (cid:13)(cid:13) ∇ 3 L ( θ ) (cid:13)(cid:13) op ≤ ρ 3 and ∇ 3 L is ρ 4 -Lipschitz with respect to ∥·∥ op .
Then we assume that ρ 4 = O ( ηρ 23 ) .
Next, we require the following generalization of Assumption 1:
Assumption 4.
For all t ,
[Equation]
Finally, we require a set of “non-worst-case" assumptions, which are that the quantities ∇ 2 L, ∇ 3 L, and λ min ( ∇ 2 L ) are nicely behaved in the directions orthogonal to u t , which generalizes the eigengap assumption.
We verify the assumptions on ∇ 2 L and ∇ 3 L empirically in Appendix E.
Assumption 5.
For all t and v, w ⊥ u t ,
[Equation]
With these assumptions in place, we can state our main theorem which guarantees ⋆ x, ⋆ y, ⋆ v predict the loss, sharpness, and deviation from the constrained trajectory up to higher order terms:
Theorem 1.
Let T :
= O ( ϵ − 1 ) and assume that min t ≤ T | ⋆ x t | ≥ c 1 δ.
Then for any t ≤ T , we have
[Equation]
The sharpness is controlled by the slowly evolving quantity ⋆ y t and the period-2 oscillations of ( ∇ S ·
U )
⋆ x t .
This combination of gradual and rapid periodic behavior was observed by Cohen et al.
and appears in our experiments.
Theorem 1 also shows that the loss at θ t spikes whenever ⋆ x t is large.
On the other hand, when ⋆ x t is small, L ( θ t ) approaches the loss of the constrained trajectory.
E XPERIMENTS
We verify that the predicted dynamics defined in eq.
(5) accurately capture the dynamics of gradient descent at the edge of stability by replicating the experiments in (Cohen et al., 2021) and tracking the deviation of gradient descent from the constrained trajectory.
In Figure 3, we evaluate our theory on a 3-layer MLP and a 3-layer CNN trained with mean squared error (MSE) on a 5k subset of CIFAR and a 2-layer Transformer (Vaswani et al., 2017) trained with MSE on SST2 Socher et al.
We provide additional experiments varying the learning rate and loss function in Appendix G, which use the generalized predicted dynamics described in Section 7.2.
For additional details, see Appendix D.
Figure 3 confirms that the predicted dynamics eq.
(5) accurately predict the loss, sharpness, and distance from the constrained trajectory.
In addition, while the gradient flow trajectory diverges from the gradient descent trajectory at a linear rate, the gradient descent trajectory and the constrained trajectories remain close throughout training .
In particular, the dynamics converge to the fixed point ( | x t | , y t ) =
( δ t , 0) described in Section 4.2 and ∥ θ t − θ † t ∥ → δ t .
This confirms our claim that gradient descent implicitly follows the constrained trajectory eq.
In Section 5, various assumptions on the model were made to obtain the EOS behavior.
In Appendix E, we numerically verify these assumptions to ensure the validity of our theory.
[Figure]
MLP+MSE on CIFAR10, η = 0 .
[Figure]
CNN+MSE on CIFAR10, η = 0 .
[Figure]
Transformer+MSE on SST2, η = 0 .
A key consequence of the predicted dynamics is that the loss and sharpness only depend on ( ⋆ x t , ⋆ y t ) , which are governed by the 2D dynamical system eq.
Therefore, understanding the EOS dynamics only requires analyzing this dynamical system, which is generally well behaved (Figure 3).
Furthermore, we expect ⋆ x t , ⋆ y t to converge to ( ± δ t , 0) , the fixed point of eq.
In fact, Figureshows that after a few cycles, ( ⋆ x t , ⋆ y t ) indeed converges to this fixed point.
We can accurately predict its location as well as the loss increase from the constrained trajectory due to ⋆ x t ̸ = 0 .
7. G ENERALIZED P REDICTED D YNAMICS
In order for our cubic Taylor expansions to track the true gradients, we require a bound on the fourth derivative of the loss (Assumption 3).
This is usually sufficient to capture the EOS dynamics as demonstrated by Figure 3 and Appendix E.
However, this condition was violated in some of our experiments, especially when using logistic loss.
To overcome this challenge, we developed a generalized form of the predicted dynamics whose definition we defer to
Appendix F. These generalized predictions are qualitatively similar to those given by the predicted dynamics in Section 5; however, they precisely track the dynamics of gradient descent in a wider range of settings (see Appendix G).
7. I MPLICATIONS FOR N EURAL N ETWORK T RAINING
Non-Monotonic Loss Decrease An important property of EOS is that the loss decreases over long time scales, albeit non-monotonically.
Our theory provides a clear explanation for this phenomenon.
We show that the gradient descent trajectory remains close to the constrained trajectory (Sectionsand 5).
Since the constrained trajectory is stable , it satisfies a descent lemma (Lemma 14), and its loss monotonically decreases.
Over short time periods, the loss is dominated by the rapid fluctuations of x t described in Section 4.
Over longer time periods, the loss decrease of the constrained trajectory overpowers the bounded fluctuations of x t , leading to an overall loss decrease.
Generalization & the Role of Large Learning Rates Prior work has shown that in neural networks, both decreasing sharpness of the learned solution (Keskar et al., 2017; Dziugaite & Roy, 2017; Neyshabur et al., 2017; Jiang et al., 2020) and increasing the learning rate (Smith et al., 2018;
Li et al., 2019; Lewkowycz et al., 2020) are correlated with better generalization.
Our analysis shows that gradient descent implicitly constrains the sharpness to stay below 2 /η , which suggests larger learning may improve generalization by reducing the sharpness.
In Figure 4 we confirm that in a standard setting, full-batch gradient descent generalizes better with large learning rates.
Training Speed Additional experiments in (Cohen et al., 2021, Appendix F) show that, despite the instability in the training process, larger learning rates lead to faster convergence.
This phenomenon is explained by our analysis.
Gradient descent is coupled to the constrained trajectory which minimizes the loss while constraining movement in the u t , ∇ S ⊥ t directions.
Since only two directions are “off limits," the constrained trajectory can still move quickly in the orthogonal directions, using the large learning rate to accelerate convergence.
We demonstrate this empirically in Figure 4.
We defer additional discussion of our work, including the effect of multiple unstable eigenvalues and connections to Sharpness Aware Minimization (Foret et al., 2021), warm-up (Gilmer et al., 2021), and scale-invariant loss functions
(Lyu et al., 2022) to Appendix H.
7. F UTURE W ORK
An important direction for future work is understanding the EOS dynamics when there are multiple unstable eigenvalues, which we briefly discuss in Appendix H. Another interesting direction is understanding the global convergence properties at EOS, including convergence to a KKT point of the constrained update eq.
Next, our analysis focused on the EOS dynamics but left open the question of why neural networks exhibit progressive sharpening.
Finally, we would like to understand the role of self-stabilization in stochastic -gradient descent it interacts A CKNOWLEDGEMENTS AD acknowledges support from a NSF Graduate Research Fellowship.
EN acknowledges support from a National Defense Science & Engineering Graduate Fellowship, and NSF grants CIFand DMS-2014279. JDL, AD, EN acknowledge support of the Sloan Research Fellowship, NSF CCF 2002272, NSF IIS 2107304, NSF CIF 2212262, and NSF-CAREER under award #2144994. The authors would like to thank Jeremy Cohen, Kaifeng Lyu and Lei Chen for helpful discussions throughout the course of this project.
We would especially like to thank Jeremy Cohen for suggesting the term “self-stabilization” to describe the negative feedback loop derived in this paper
T ABLE OF C ONTENTS
1 Introduction 1.
Gradient Descent at the Edge of Stability . .
Self-stabilization: The Implicit Bias of Instability . . . . . . . . . . . . . . . . . .
2 Related Work 3 Setup
The Self-stabilization Property of Gradient Descent 4.
The Four Stages of Edge of Stability: A Heuristic Derivation . . . . . . . . . . . .
Analyzing the simplified dynamics . . . . . . . . . . . . . . . . . . . . . . . . . .
Relationship with the constrained trajectory θ † t . . . . . . . . . . . . . . . . . . . .
5 The Predicted Dynamics and Theoretical Results 5.
Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The equations governing edge of stability . .
Coupling Theorem . .
6 Experiments 7 Discussion 7.
Takeaways from the Predicted Dynamics . . . . . . . . . . . . . . . . . . . . . . .
Generalized Predicted Dynamics . .
Implications for Neural Network Training . . . . . . . . . . . . . . . . . . . . . .
Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A Notation
B A Toy Model for Self-Stabilization
C Definition of the Predicted Dynamics
D Experimental Details
D.1 Architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
D.2 Data . .
D.3 Experimental Setup . .
E Empirical Verification of the Assumptions
F The Generalized Predicted Dynamics
F.
Deriving the Generalized Predicted Dynamics . . . . . . . . . . . . . . . . . . . .
F.
Properties of the Generalized Predicted Dynamics . . . . . . . . . . . . . . . . . .
G Additional Experiments
G.1 The Benefit of Large Learning Rates:
Training Time and Generalization . .
G.2 Experiments with the Generalized Predicted Dynamics . . . . . . . . . . . . . . .
H Additional Discussion
H.1 Scale Invariant Lemmas . .
I Proofs
I.
Properties of the Constrained Trajectory . . . . . . . . . . . . . . . . . . . . . . .
I.
Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
I.
Proof of Auxiliary Lemmas . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A N OTATION
We denote by ∇ k L ( θ ) the k -th order derivative of the loss L at θ .
Note that ∇ k L ( θ ) is a symmetric k -tensor in ( R d ) ⊗ k when θ ∈ R d .
For a symmetric k -tensor T , and vectors u 1 , . . .
u
j ∈ R
d we will use T ( u 1 , . . .
, u j ) to denote the tensor contraction of T with u 1 , . .
. , u j , i.e.
[Equation]
[Equation]
We use P u 1 ,...,u k to denote the orthogonal projection onto span( u 1 , . .
. , u k ) and P ⊥ u 1 ,...,u k is the projection onto the corresponding orthogonal complement.
For matrices A 1 , . . .
A k , we define t (cid:89) k
=
A k :=
A 1 . . .
A t and 1 (cid:89) k
= t
A k :=
A t . . .
A 1 .
B A T OY M ODEL FOR S ELF -S TABILIZATION
For α, β > 0 , consider the function
[Equation]
initialized at the point ( x 0 , 0 , 0) .
Note that the constrained trajectory will follow x † t = 0 , y † t = 0 , z † t = − ηt as it cannot decrease y without increasing the sharpness past 2 /η .
We therefore have:
[Equation]
Note that this satisfies all of the assumptions in Section 4 and it satisfies α
= −∇
L t · ∇ S t and β = ∥∇ S t ∥ 2 .
This process will then follow eq.
(4) in the x, y directions while it tracks the constrained trajectory θ † t moving linearly in the − P ⊥ u, ∇ S ∇ L = [0 , 0 , − 1] direction.
C D EFINITION OF THE P REDICTED D YNAMICS
Below, we present the full definition of the predicted dynamics:
Definition 6 (Predicted Dynamics, full) .
Define ⋆ v 0 = v 0 , and let ⋆ x t =
⋆ v t · u t , ⋆ y t = ∇ S ⊥ · ⋆ v t .
Then
[Equation]
For convenience, we will define the map step t : R d → R d as follows:
Definition 7.
Given a vector v and a timestep t , define step t ( v ) by
[Equation]
where x = u t · v and y = ∇ S ⊥ t · v .
It is easy to see that ⋆ v t +1 =
step t ( ⋆ v t ) .
Proof of Lemma 4.
Defining A t = ( I − η ∇ 2 L t )
P ⊥ u t , we can unfold the recursion in eq.
(6) to obtain the following formula for ⋆ v t .
[Equation]
It is then immediate to see that ⋆ x t =
⋆ v t · u t , ⋆ y t = ∇ S ⊥ t · ⋆ v t have the following simple update: x ∗ t +1 = −
[Equation]
ηy
[Equation]
t ) x
[Equation]
t and y ∗ t +1 =
η t (cid:88) s = β s → t (cid:20)
{+δ 
+}
s
[Equation]
x ∗ s
[Equation]
, where we recall that we have defined
[Equation]
D E XPERIMENTAL D ETAILS D. A RCHITECTURES
We evaluated our theory on four different architectures.
The 3-layer MLP and CNN are exact copies of the MLP and CNN used in (Cohen et al., 2021).
The MLP has width 200 , the CNN has width 32 , and both are using the swish activation (Ramachandran et al., 2017).
We also evaluate on a ResNet with progressive widths 16 , 32 , 64 , 128 and on a 2-layer Transformer with hidden dimension 64 and two attention heads.
D. D ATA
We evaluated our theory on three primary tasks: CIFAR10 multi-class classification with both categorical MSE loss and cross-entropy loss, CIFAR10 binary classification (cats vs dogs) with binary
MSE loss and logistic loss, and SST2 (Socher et al., 2013) with binary MSE loss and logistic loss
. D. E XPERIMENTAL S ETUP
For every experiment, we tracked the gradient descent dynamics until they reached instability and then began tracking the constrained trajectory, gradient descent, gradient flow, and both our predicted dynamics (Section 5) and our generalized predicted dynamics (Appendix F).
In addition, we tracked the various quantities on which we made assumptions for Section 5 in order to validate these assumptions.
We also tracked the second eigenvalue of the Hessian at the constrained trajectory throughout training and stopped training once it reached 1 .
9 /η , to ensure the existence of a single unstable eigenvalue.
Finally, as the edge of stability dynamics are very sensitive to small perturbation when | x | is small (see Figure 2), we switched to computing gradients with 64-bit precision after first reaching instability to avoid propagating floating point errors.
Eigenvalues were computed using the LOBPCG sparse eigenvalue solver in JAX (Bradbury et al., 2018).
To compute the constrained trajectory, we computed a linearized approximation for proj M inspired by Lemma 13 along with a Newton step in the u t direction to ensure that ∇ L · u = 0 .
Each linearized approximation step required recomputing the sharpness and top eigenvector and each projection step then consisted of three linearized projection steps, for a total of three eigenvalue computations per projection step.
Our experiments were conducted in JAX (Bradbury et al., 2018), using https://github.com/ locuslab/edge-of-stability as a reference for replicating the experimental setup used in (Cohen et al., 2021).
All experiments were conducted on two servers, each with 10 NVIDIA GPUs.
Our code can be found at https://github.com/adamian98/EOS .
E E MPIRICAL V ERIFICATION OF THE A SSUMPTIONS
For each of the experimental settings considered (MLP+MSE, CNN+MSE, CNN+Logistic,
ResNet18+MSE, Transformer+MSE, Transformer+Logistic), we plot a number of quantities along the constrained trajectory to verify that the assumptions made in the main text hold.
For each learning rate η we have 8 plots tracking various quantities, which verify the assumptions as follows:
Assumption 1 is verified by the 1st plot, ϵ being small is verified by the 2nd plot, Assumptionis verified by the 3rd and 4th plots, Assumption 3 is verified by the 5th plot, and Assumption 5 is verified by the last 3 plots.
As described in the experimental setup, training is stopped once the second eigenvalue is 1 .
9 /η , so Assumption 2 always
MLP+MSE on CIFAR
[Figure]
t
[Equation]
CNN+MSE on CIFAR
[Figure]
CNN+Logistic on CIFAR10 (cats vs dogs)
[Figure]
ResNet18+MSE on CIFAR10 (cats vs dogs)
[Figure]
Transformer+MSE on SST
[Figure]
− ∇
[Figure]
: = − ∇ L t · ∇ S t
[Figure]
η √ α
[Figure]
Transformer+Logistic on SST
[Figure]
F T HE G ENERALIZED P REDICTED D YNAMICS
Our analysis relies on a cubic Taylor expansion of the gradient.
However, in order for this Taylor expansion to accurately track the gradients we need a bound on the fourth derivative of the loss (Assumption 3).
Section 6 and Appendix E show that this approximation is sufficient to capture the dynamics of gradient descent at the edge of stability for many standard models when the loss criterion is the mean squared error.
However, for certain architectures and loss functions, including
ResNet18 and models trained with the logistic loss, this condition is often violated.
In these situations, the loss function in the top eigenvector direction is either sub-quadratic , meaning that the quadratic Taylor expansion overestimates the loss and sharpness 6 , or super-quadratic , meaning that the quadratic Taylor expansion underestimates the loss and sharpness.
To capture this phenomenon, we derive a more general form of the predicted dynamics which reduces to the standard predicted dynamics in Section 5 when the loss in the top eigenvector direction is approximately quadratic.
In addition, Appendix G shows that the generalized predicted dynamics capture the dynamics of gradient descent at the edge of stability for both mean squared error and cross-entropy in all settings we tested.
F. D ERIVING THE G ENERALIZED P REDICTED D YNAMICS
To derive the generalized predicted dynamics, we will abstract away the dynamics in the top eigenvector direction.
Specifically, for every t we define
[Equation]
We say that L is sub-quadratic at t if F t ( x ) < 0 and super-quadratic if F t ( x ) > 0 .
Note that knowing F t is not sufficient to capture the dynamics in the u t direction.
Specifically,
[Equation]
It is still critically important to track the effect that the movement in the ∇ S ⊥ t direction has on the dynamics of x .
As in Section 4.1, the effect of the movement in the ∇ S ⊥ t direction on the dynamics of x is changing the sharpness by y t .
This gives us the generalized predicted dynamics update:
[Equation]
where x ⋆t +1 = − (1 + ηy ⋆t ) x
⋆t − ηF ′
( x ⋆t ) .
Note that when F t ( x ) = 0 is exactly quadratic, this reduces to the standard predicted dynamics update in eq.
Note that the update for y is completely unchanged:
Lemma 5.
Restricted to the u t , ∇ S t directions, the generalized predicted dynamics v ⋆ t imply:
[Equation]
The proof is identical to the proof of Lemma 4.
F. P ROPERTIES OF THE G ENERALIZED P REDICTED D YNAMICS
Note that due to the sign flipping argument in Appendix I, we can assume that F is an even function as the odd part will only influence the dynamics through additional oscillations of period 2, so throughout the remainder of this section we will assume that F t ( x ) =
F t ( − x ) .
Otherwise, we can simply redefine F by its even part.
Next, note that the fixed point of eq.
(11) is still when x t = δ t , regardless of the shape of F t , due to the need to stabilize the ∇ S ⊥ t direction.
This contradicts previous 1-dimensional analyses of edge of stability in which the fixed point in the top eigenvector direction strongly depends on the shape of F t , the loss in the u t direction.
The limiting value of y t can therefore be read from the update for x t .
If ( δ t , y ) is an orbit of period 2 of eq.
(11), then
[Equation]
In addition, note that the sharpness can no longer be approximated as S ( θ t )
≈ 2 /η + y t as the sharpness now changes along the u t direction.
In particular, it changes by F ′′ ( x ) so that
[Equation]
Therefore, the limiting sharpness of eq.
(11) is
[Equation]
When F t = 0 and the loss is exactly quadratic in the u direction, this update reduces to fixed point predictions in Section 4.1.
One interesting phenomenon observed by Cohen et al.
(2021) is that with cross-entropy loss, the sharpness was never exactly 2 /η , but usually hovered above it.
This contradicts the predictions of the standard predicted dynamics which predict that the fixed point has sharpness 0 .
However, using the generalized predicted dynamics eq.
(11), we can give a clear explanation.
When the loss is sub-quadratic, e.g. when F t ( x ) = − ρ 4 x 4 24 , we have
[Equation]
so the sharpness will converge to a value below 2 /η .
On the other hand if the loss is super-quadratic, the sharpness converges to a value above 2 /η .
More generally, whether the loss converges to a value above or below 2 /η depends on the sign of F ′′ t ( δ
[Equation]
In our experiments in Appendix G, we observed both sub-quadratic and super-quadratic loss functions.
In particular, the loss was usually sub-quadratic when it first reached instability but gradually became super-quadratic as training progressed at the edge of stability.
G A DDITIONAL E XPERIMENTS G. T HE B ENEFIT OF L ARGE L EARNING R ATES : T RAINING T IME AND G ENERALIZATION
We trained ResNet18 with full batch gradient descent on the full 50k training set of CIFAR10 with various learning rates, in addition to the commonly proposed learning rate schedule η t := 1 /S ( θ t ) .
We show that despite entering the edge of stability, large learning rates converge much faster.
In addition, due to the self-stabilization effect of gradient descent, the final sharpness is bounded by 2 /η which is smaller for larger learning rates and leads to better generalization (see Figure 4).
[Figure]
ac
[Figure]
η = 0 .η
=
0 .η
= 0 .η = 0 .η
= 0 .η
t = 1 / S ( θ t )
G. E XPERIMENTS WITH THE G ENERALIZED P REDICTED D YNAMICS
MLP+MSE on CIFAR
Gradient Descent
[Figure]
i s t a n ce t o θ † η = 0 .
[Figure]
CNN+MSE on CIFAR
[Figure]
CNN+Logistic on CIFAR10 (cats vs dogs)
[Figure]
D i s t a n ce t o θ † η = 0 .
ResNet18+MSE on CIFAR10 (cats vs dogs)
[Figure]
Transformer+MSE on SST
Gradient Descent
[Figure]
D i s t a n ce t o θ † η = 0 .
[Figure]
Transformer+Logistic on SST
Gradient Descent
[Figure]
i s t a n ce t o θ † η = 0 .
H A DDITIONAL D ISCUSSION
A Precise Criterion for Self-Stabilization Our theoretical results in Sections 4 and 5 give sufficient conditions for self-stabilization as evidenced by Theorem 1.
However, these assumptions may not be strictly necessary and we believe that self-stabilization may hold under significantly weaker assumptions.
An important open question is understanding the precise conditions on the loss function and learning rate that enable self-stabilization.
Furthermore, in Theorem 1 in Section 5, we give a quantitative bound on the time for which the EOS trajectory and the constrained trajectory can be coupled. It is an interesting future direction to understand whether coupling can be strengthened to hold for longer time periods to allow for convergence to a KKT point of the constrained trajectory eq. Multiple Unstable Eigenvalues Our work focuses on explaining edge of stability in the presence of a single unstable eigenvalue (Assumption 2).
However, Cohen et al.
(2021) observed that progressive sharpening appears to apply to all eigenvalues, even after the largest eigenvalue has become unstable.
As a result, all of the top eigenvalues will successively enter edge of stability (see
Figure 5).
In particular, Figure 5 shows that the dynamics are fairly well behaved in the period when only a single eigenvalue is unstable, yet appear to be significantly more chaotic when multiple eigenvalues are unstable.
[Figure]
One technical challenge with dealing with multiple eigenvalues is that, when the top eigenvalue is not unique, the sharpness is no longer differentiable and it is unclear how to generalize our analysis.
However, one might expect that gradient descent can still be coupled to projected gradient descent under the non-differentiable constraint S ( θ † T ) ≤ 2 /η .
When there are k unstable eigenvalues, with corresponding eigenvectors u 1 t , . .
, u kt , the constrained update is roughly equivalent to projecting out the subspace span {∇ 3 L t ( u it , u jt ) :
i, j ∈ [ k ] } from the gradient update − η ∇ L t .
Demonstrating self-stabilization thus requires analyzing the dynamics in the subspace span (cid:16) { u it : i ∈
[ k ] } ∪ {∇ 3 L t ( u it , u jt ) : i, j ∈ [ k ] } (cid:17) .
We leave investigating the dynamics of multiple unstable eigenvalues for future work.
Connection to Sharpness Aware Minimization (SAM)
Foret et al.
(2021) introduced the sharpness-aware minimization (SAM) algorithm, which aims to control sharpness by solving the optimization problem min θ max ∥ δ ∥≤ ϵ L ( θ + δ ) .
This is roughly equivalent to minimizing S ( θ ) over all global minimizers, and thus SAM tries to explicitly minimize the sharpness.
Our analysis shows that gradient descent implicitly minimizes the sharpness, and for a fixed η looks to minimize L ( θ ) subject to S ( θ ) = 2 /η .
Connections to Warmup.
Gilmer et al.
(2021) demonstrated that learning rate warmup , which consists of gradually increasing the learning rate, empirically leads to being able to train with a larger learning rate.
The self-stabilization property of gradient descent provides a plausible explanation for this phenomenon.
If too large of an initial learning rate η 0 is chosen (so that S ( θ 0 ) is much greater than 2 /η 0 ), then the iterates may diverge before self stabilization can decrease the sharpness to 2 /η 0 .
On the other hand, if the learning rate is chosen that S ( θ 0 ) is only slightly greater than 2 /η 0 , selfstabilization will decrease the sharpness to 2 /η 0 .
Repeatedly increasing the learning rate slightly could then lead to small decreases in sharpness without the iterates diverging, thus allowing training to proceed with a large learning rate.
Connection to Weight Decay and Sharpness Reduction.
Lyu et al.
(2022) proved that when the loss function is scale-invariant, gradient descent with weight decay and sufficiently small learning rate converges leads to reduction of the normalized sharpness S ( θ/ ∥ θ ∥ ) .
In fact, the mechanism behind the sharpness reduction is exactly the self-stabilization force described in this paper restricted to the setting in (Lyu et al., 2022).
We present here a heuristic derivation of this equivalence.
First, we show that any scale invariant satisfies our assumptions when trained with weight decay.
Lemma 6.
Let f be a scale invariant loss function, i.e. f ( θ ) =
f ( cθ ) .
Let L ( θ ) =
f ( θ ) + λ 2 ∥
θ ∥ 2 .
Then for any local minimizer θ of f ( θ ) such that S ( θ ) = 2 /η ,
• ∇ L ( θ ) ⊥ u ( θ ) • ρ 4 =
O ( ηρ 23 ) • α ( θ ) >
• α ( θ ) ∥∇ L ( θ ) ∥∥∇ S ( θ ) ∥
= Θ(1) • ∥∇ S ( θ ) ∥
=
Θ( ρ 3 )
Our primary result is that gradient descent solves the constrained problem min θ L ( θ ) such that
S ( θ ) ≤ 2 /η .
Let S f ( θ ) denote the largest eigenvalue of ∇ 2 f ( θ ) .
To prove equivalence to the sharpness reduction, we will need the following lemma from (Lyu et al., 2022) which follows from the scale invariance of the f :
[Equation]
Let θ := θ ∥ θ ∥ .
Then we have the following equality between minimization problems:
min θ L ( θ ) such that S ( θ ) ≤ 2 /η ⇐⇒ min θ f
[Equation]
θ
[Equation]
+ [Equation] ∥  such that S f ( θ ) ≤ 2 /η − λ ⇐⇒ min θ, ∥ [Equation]
∥
[Equation]
θ
[Equation]
+
[Equation]
∥ θ ∥ such that
[Equation]
where the last line follows from the scale-invariance of the loss function.
In particular if ηλ is sufficiently small and the dynamics are initialized near a global minimizer of the loss, this will converge to the solution of the constrained problem:
[Equation]
H. S CALE I NVARIANT L EMMAS
Let θ denote an arbitrary parameter and let θ = θ/ ∥ θ ∥ .
Throughout this section, let f be a scale invariant function with non-vanishing Hessian.
Lemma 7.
[Equation]
Proof.
We start with the equality: f
( θ ) = f ( θ ) .
Differentiating with respect to θ and using that ∇ θ θ =
P ⊥ θ ∥ θ ∥ gives,
[Equation]
Differentiating this again gives:
[Equation]
A few corollaries immediately follow:
Corollary 2.
For any critical point θ of f ,
[Equation]
Corollary 3.
For any critical point θ of f ,
[Equation]
Proof.
Note that from Corollary 2, the top eigenvector of ∇ 2 f is perpendicular to θ .
In addition,
[Equation]
so this is also the top eigenvector of ∇ 2 L ( θ ) , i.e. u ( θ ) .
Finally,
[Equation]
which is parallel to θ and concludes the proof.
Lemma 8.
[Equation]
Proof.
Let S f ( θ ) denote the largest eigenvalue of ∇ 2 f ( θ ) .
Then by scale invariance, ∇ 2 f ( θ ) = ∇ 2 f ( θ ) /
∥ θ ∥ 2 .
This implies that
[Equation]
Differentiating this gives:
[Equation]
Finally, we have from ∇ 2 L ( θ ) =
∇ 2 f ( θ ) + λI that S ( θ )
=
S f ( θ ) + λ so
[Equation]
Lemma 9.
Let ρ 3 = sup S ( θ ) ≤ 4 /η
(cid:13)(cid:13)
∇ 3 L ( θ ) (cid:13)(cid:13) op and ρ 4 = sup S ( θ ) ≤ 4 /η
(cid:13)(cid:13) ∇ 4 L ( θ ) (cid:13)(cid:13) op .
Then ρ 3 = Θ( [Equation] [Equation]
and ρ 4 = O ( η − 2 ) .
In particular, ρ 4 = O ( ηρ 23 ) .
Proof.
Note that S ( θ ) < 4 /η implies that
[Equation]
Therefore, ∇
[Equation]
and ∇
[Equation]
by compactness of S d − 1 .
In addition, note that (cid:13)(cid:13) ∇ 3 L ( u, u ) (cid:13)(cid:13)
=
∥∇ S ( θ ) ∥ ≥ S ( θ ) − λ ∥ θ ∥= Θ( η
[Equation]
so ρ 3 = Θ( η − 3 / 2 ) .
Lemma 10.
At any second order stationary point θ of f ,
[Equation]
Proof.
[Equation]
Lemma 11.
At any second order stationary point θ of f ,
[Equation]
Proof.
[Equation]
where the last step follows from compactness of S d − 1 and the fact that ∇ 2 f is non-vanishing.
Proof of Lemma 6.
The lemma is simply a restatement of Corollary 3, Lemma 10, Lemma 9, and
Lemma 11.
I P ROOFS I. P ROPERTIES OF THE C ONSTRAINED T RAJECTORY
We next prove several nice properties of the constrained trajectory.
First, we require the following auxiliary lemma, which shows that several quantities are Lipschitz in a neighborhood around the constrained trajectory:
Definition 8 (Lipschitz Sets) .
S t := B ( θ † t , 2 − c 4 ηρ 3 ) where c is the constant in Assumption 2 and
B ( x, r ) denotes the ball of radius r centered at x .
Lemma 12 (Lipschitz Properties) .
θ → ∇ L ( θ ) is O ( η − 1 ) -Lipschitz
in each set S t . 2.
θ → ∇ 2 L ( θ ) is ρ 3 -Lipschitz with respect to ∥·∥ 2 . 3.
θ → λ i ( ∇ 2 L ( θ )) is ρ 3 -Lipschitz.
θ → u ( θ ) is O ( ηρ 3 ) -Lipschitz in each set S t .
θ → ∇ S ( θ ) is O ( ηρ 23 ) -Lipschitz
in each set S t .
Proof.
The Lipschitzness of ∇ 2 L ( θ ) follows immediately from the bound (cid:13)(cid:13) ∇ 3 L ( θ ) (cid:13)(cid:13) op ≤ ρ 3 .
Weil’s inequality then immediately implies the desired bound on the Lipschitz constant of the eigenvalues of ∇ 2 L ( θ ) .
Therefore for any t , we have for all θ ∈ S t :
[Equation]
Next, from the derivative of eigenvector formula:
[Equation]
which implies the bound on the Lipschitz constant of u restricted to S t .
Finally, because ∇ S ( θ ) = ∇ 3 L ( θ )( u ( θ ) , u ( θ )) ,
[Equation]
where the second to last inequality follows from the bound on ∥∇ u ( θ ) ∥ 2 restricted to S t and the last inequality follows from Assumption 3.
Lemma 13 (First-order approximation of the constrained trajectory update { θ † t } ) .
For all t ≤ T ,
[Equation]
Proof.
We will prove by induction that S t = 2 /η for all t .
The base case follows from the definitions of θ 0 , θ † 0 .
Next, assume S ( θ † t ) = 0 for some t ≥ 0 .
Let θ ′ =
θ † t − η ∇ L t .
Then because θ † t ∈ M we have (cid:13)(cid:13)(cid:13) θ † t +1 − θ ′ (cid:13)(cid:13)(cid:13) ≤ (cid:13)(cid:13)(cid:13) θ † t − θ ′ (cid:13)(cid:13)(cid:13) =
η ∥∇ L t ∥ .
Then because θ † t +1 = proj
M ( θ ′ ) , the KKT conditions for this minimization problem imply that there exist x, y with y ≥ 0 such that
[Equation]
Next, note that we can decompose ∇ S t = u t [Equation]
[Equation]
Let s t = ∇ S ⊥ t ∥ ∇ S ⊥ t ∥ .
We can now perform the change of variables
[Equation]
to get
[Equation]
Note that
O
[Equation]
for sufficiently small ϵ so because (cid:13) (cid:13)(cid:13) θ † t +1 − θ ′ (cid:13) (cid:13)(cid:13) ≤ η ∥∇ L t ∥ we have [Equation] ≤ [Equation]
θ †
[Equation]
+
[Equation]
− θ ′ (cid:13)(cid:13)(cid:13) ≤ η ∥∇ L t ∥
so x, y =
O ( η ∥∇ L t ∥ ) .
Therefore,
[Equation]
Then Taylor expanding ∇ L t +1 around θ † t gives
[Equation]
so x ′ =
O ( ϵ 2 · η ∥∇ L t ∥ ) .
We can also Taylor expand S t +1 around θ † t and use that S t = 2 /η to get
[Equation]
Now note that for ϵ sufficiently small we have
[Equation]
Therefore if y ′ = 0 , we would have S t +1 > 2 /η which contradicts θ † t +1 ∈ M .
Therefore y ′ >and therefore y > 0 , which by complementary slackness implies S t +1 = 2 /η .
This then implies that
[Equation]
Putting it all together gives
[Equation]
where the last line follows from u t · ∇ L t = 0 .
Lemma 14 (Descent Lemma for θ † ) .
For all t ≤ T ,
[Equation]
Proof.
Taylor expanding L ( θ † t +1 ) around L ( θ † t ) and using Lemma 13 gives
[Equation]
Next, note that because γ t = Θ(1) we have ∥∇ L [Equation]
∥
[Equation] O ( [Equation]
P ⊥ u t , [Equation] S t ∇ L t (cid:13)(cid:13) ) .Therefore
for ϵ sufficiently small,
[Equation]
Therefore,
[Equation]
which completes the proof.
Corollary
[Equation]
Let L ⋆ =
min
θ L ( θ ) .
Then there exists t ≤ T such that
[Equation]
Proof.
Inductively applying Lemma 14 we have that there exists an absolute constant c such that
[Equation]
which implies that
[Equation]
I. P ROOF OF T HEOREM
We first require the following three lemmas, whose proofs are deferred to Appendix I.3.
Lemma 15 (2-Step Lemma) .
Let
[Equation]
Assume that ∥ v t ∥ ≤ ϵ − 1 δ .
Then
[Equation]
Lemma 16.
Assume that there exists constants c 1 , c 2 such that for all t ≤ T , ∥
⋆ v t ∥ ≤ c 2 δ , | ⋆ x t | ≥ c 1 δ .
Then, for all t ≤ T , we have
[Equation]
Lemma 17.
For t ≤ T , ∥
⋆ v t ∥ ≤ O ( δ ) .
With these lemmas in hand, we can prove Theorem 1.
Proof of Theorem 1.
First, by Lemma 17, we have ∥ ⋆ v t ∥
≤ O ( δ ) .
Next, by Lemma 16, we have
[Equation]
Next, we Taylor expand to calculate S ( θ t ) :
[Equation]
Finally [Equation] we Taylor expand the loss:
[Equation]
where the last line follows from Assumption
[Equation]
I. P ROOF OF A UXILIARY L EMMAS
Proof of Lemma 15.
Taylor expanding the update for θ t +1 about θ † t , we get
[Equation]
Additionally, recall that the update for θ † t +1 is
[Equation]
Subtracting the previous 2 equations and expanding out ∇ 3 L ( v t , v t ) via the non-worst-case bounds, we obtain
[Equation]
We would first like to compute the magnitude of v t +1 .
[Equation]
Observe that by definition of ϵ and δ , and since ∥ v t ∥ ≤ ϵ − 1 δ
[Equation]
Hence
[Equation]
Note that we can bound
[Equation]
Therefore
[Equation]
the one-step update in the u t direction is:
[Equation]
where we have defined the error term E t as
[Equation]
The update in the v ⊥ direction is
[Equation]
First, observe that
[Equation]
Therefore we can control the first of the error terms as
[Equation]
As for the second error term, we can decompose
[Equation]
ηx
[Equation]
P
[Equation]
1 ∇ 3 L t
[Equation]
u
[Equation]
[Equation]
− P ⊥
[Equation]
+
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∇ 3 L t ( u t , v ⊥ t ) (cid:13)(cid:13)(cid:17) .
By Assumption 5, we have (cid:13)(cid:13)
P ⊥ u t ∇ 3 L t ( u t , v ⊥ t ) (cid:13)(cid:13) ≤ O ( ϵρ 3 ∥ v t ∥ ) .
Additionally, (cid:13)(cid:13)(cid:13) P ⊥ u t − P ⊥ u t +1 (cid:13)(cid:13)(cid:13) ≤
O ( ∥ u t − u t +1 ∥ ) .
Therefore (cid:13)(cid:13)(cid:13)
[Equation]
P
[Equation]
L
[Equation]
( u
[Equation]
, v ⊥ t
[Equation]
where we used ηρ 3 ∥ v t ∥
=
O (1) .
Altogether [Equation] we have v ⊥
[-+
-][Equation] We next compute the two-step update for x t : x [Equation]
=
[Equation] ∥ [Equation] ∥
[Equation]
+ E [Equation]
+1 )
[Equation]
We previously obtained ηρ 3 ∥ v t ∥ =
O (1) .
Furthermore,
[Equation]
Hence
[Equation]
The first of these two error terms can be bounded as
[Equation]
As for the second term [Equation] we can bound
[Equation]
Additionally, we have
[Equation]
+ E
[Equation]
Therefore
[Equation]
Altogether, the two-step update for x t is
[Equation]
Additionally, the two-step update for v ⊥ t is
[Equation]
Define v t +1 = step t ( v t ) , v t +2 = step t +1 (
v t ) , and x i = v i · u i , y i = ∇ S ⊥ i · v i for i ∈ { t +1 , t +2 }
By the definition of step , one sees that
[Equation]
and
[Equation]
The update for x after applying step is
[Equation]
Therefore
[Equation]
Additionally, the update for v ⊥ is
[Equation]
Therefore
[Equation]
Altogether, we get that
[Equation]
as desired.
Proof of Lemma 16.
Define w t
=
(cid:26)t if is even r t −t if is odd and define the auxiliary trajectory (cid:98) v by (cid:98) v 0 = v 0 and (cid:98) v[-t +
-]
{+t +
+} = step( (cid:98) v t ) + w t .
I first claim that (cid:98) v t = v t for all even t ≤ T , which we will prove by induction on t .
The base case is given by assumption so assume the result for some even t ≥ 0 .
Then,
[Equation]
which completes the induction.
Next, we will prove by induction that for t ≤ T ,
[Equation]
By definition, (cid:98) v 0 = v 0 =
⋆ v 0 , so the claim is clearly true for t = 0 .
Next, assume the claim holds for t .
If t is even then ∥ w t ∥ = 0 ; otherwise ∥ v t ∥ ≤ 2 c 2 δ [Equation] and thus
[Equation]
First observe that
[Equation]
Next, note that
[Equation]
Therefore (cid:12)(cid:12)(cid:12)(cid:12) (cid:98) x
t +1 − ⋆ x t +1 ⋆ x t + (cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12) (cid:98) x
t − ⋆ x t ⋆ x t (cid:12)(cid:12)(cid:12)(cid:12)
+ O ( ϵ
+ ϵ δ (cid:13)(cid:13)(cid:98) v ⊥ t − ⋆ v ⊥ t (cid:13)(cid:13) ) .
Let d t = max (cid:16)(cid:13)(cid:13)(cid:98) v ⊥ t − ⋆ v ⊥ t (cid:13)(cid:13) , δ (cid:12)(cid:12)(cid:12) (cid:98) x
t − ⋆ x t ⋆ x t (cid:12)(cid:12)(cid:12)(cid:17) .
Then (cid:13)(cid:13)(cid:98) v ⊥ t +1 − ⋆ v ⊥ t
[Equation]
Therefore
[Equation]
so for t ≤ T we have d t +1 ≤ O ( ϵδ ) .
Therefore
[Equation]
so the induction is proven.
Altogether, we get ∥ (cid:98) v t − ⋆ v t ∥ ≤ O ( ϵδ ) for all such t , as desired.
Proof of Lemma 17.
Recall that
[Equation]
and y
[Equation]
t + [Equation] = η t (cid:88) s = β s → t (cid:20) δ 2 s − x ∗ s (cid:21) , Since t ≤ 1 η max t | λ min ( ∇ 2 L t ) | , we have that β s → t = O ( ρ 23 ) , and thus ⋆ y t ≤ O ( ρ 23 ) tηδ 2 = O ( √ ρ 1 ρ 3 ) .
Therefore
[Equation]
x [Equation] | . Since t ≤ O ( ϵ − 1 ) , | ⋆ x t | grows by at most a constant factor, and thus | ⋆ x t | ≤ O ( δ ) .
Finally, recall that
[Equation]
By the triangle inequality,
[Equation]
Therefore ∥ ⋆ v t ∥ ≤ O ( δ ) .