Original: U NDERSTANDING E DGE - OF -S TABILITY T RAINING D YNAMICS WITH A M INIMALIST E XAMPLE
Revised: U NDERSTANDING E DGE - OF -S TABILITY T RAINING D YNAMICS WITH A M INIMALIST E XAMPLE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, researchers observed that gradient descent for deep neural networks operates in an “edge-of-stability” (EoS) regime: the sharpness (maximum eigenvalue of the Hessian) is often larger than stability threshold 2/ η (where η is the step size).
Revised: Recently, researchers observed that gradient descent for deep neural networks operates in an “edge-of-stability” (EoS) regime: the sharpness (maximum eigenvalue of the Hessian) is often larger than stability threshold 2 /η (where η is the step size).

Original: Despite this, the loss oscillates and converges in the long run, and the sharpness at the end is just slightly below 2 /η .
Revised: Despite this, the loss oscillates and converges in the long run, and the sharpness at the end is just slightly below 2 /η .

Original: While many other well-understood nonconvex objectives such as matrix factorization or two-layer networks can also converge despite large sharpness, there is often a larger gap between sharpness of the endpoint and 2 /η .
Revised: While many other well-understood nonconvex objectives such as matrix factorization or two-layer networks can also converge despite large sharpness, there is often a larger gap between sharpness of the endpoint and 2 /η .

Original: In this paper, we study EoS phenomenon by constructing a simple function that has the same behavior.
Revised: In this paper, we study EoS phenomenon by constructing a simple function that has the same behavior.

Original: We give rigorous analysis for its training dynamics in a large local region and explain why the final converging point has sharpness close to 2 /η .
Revised: We give rigorous analysis for its training dynamics in a large local region and explain why the final converging point has sharpness close to 2 /η .

Original: Globally we observe that the training dynamics for our example has an interesting bifurcating behavior, which was also observed in the training of neural nets.
Revised: Globally we observe that the training dynamics for our example have an interesting bifurcating behavior, which was also observed in the training of neural nets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many works tried to understand how simple gradient-based methods can optimize complicated neural network objectives.
Revised: Many works tried to understand how simple gradient-based methods can optimize complicated neural network objectives.

Original: However, recently some empirical observations show that optimization for deep neural networks may operate in a more surprising regime.
Revised: However, recently some empirical observations show that optimization for deep neural networks may operate in a more surprising regime.

Original: In particular, Cohen et al.
Revised: In particular, Cohen et al.

Original: (2021) observed that when running gradient descent on neural networks with a fixed step-size η , the sharpness (largest eigenvalue of the Hessian) of the training trajectory often oscillates around the stability threshold  2 /η 1 , while the loss still continues to decrease in the long run.
Revised: (2021) observed that when running gradient descent on neural networks with a fixed step-size η , the sharpness (largest eigenvalue of the Hessian) of the training trajectory often oscillates around the stability threshold of 2 /η 1 , while the loss still continues to decrease in the long run.

Original: This phenomenon is called “edge-of-stability” and has received a lot of attention (see Section 1.2 for related work).
Revised: This phenomenon is called “edge-of-stability” and has received a lot of attention (see Section 1.2 for related works).

Original: While many works try to understand why (variants of) gradient descent can still converge despite that the sharpness is larger than 2 /η , empirically gradient descent for deep neural networks has even stronger properties.
Revised: While many works try to understand why (variants of) gradient descent can still converge despite that the sharpness is larger than 2 /η , empirically gradient descent for deep neural networks has even stronger properties.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 1a, for a fixed initialization, if one changes the step size η , the final converging point has sharpness very close to  2 /η .
Revised: 1a, for a fixed initialization, if one changes the step size η , the final converging point has sharpness very close to the corresponding 2 /η .

Original: We call this phenomenon “sharpness adaptivity”.
Revised: We call this phenomenon “sharpness adaptivity”.

Original: Another perspective on the same phenomenon is that for a wide range of initializations, for a fixed step-size η , their final converging points all have sharpness very close to 2 /η , we call this phenomenon “sharpness concentration”.
Revised: Another perspective on the same phenomenon is that for a wide range of initializations, for a fixed step-size η , their final converging points all have sharpness very close to 2 /η . We call this phenomenon “sharpness concentration”.

Original: Surprisingly, both sharpness adaptivity and sharpness concentration happen on deeper networks, while for shallower models of non-convex optimization such as matrix factorization or 2-layer neural networks the gap between sharpness and 2 /η is often much larger (see Fig.
Revised: Surprisingly, both sharpness adaptivity and sharpness concentration happen on deeper networks, while for shallower models of non-convex optimization such as matrix factorization or 2-layer neural networks, the gap between sharpness and 2 /η is often much larger (see Fig.

Original: 1b).
Revised: 1b).

Original: This suggests that these phenomena are related to network depth.
Revised: This suggests that these phenomena are related to network depth.

Original: What is the mechanism for sharpness adaptivity and concentration, and how does that relate to the number of layers?
Revised: What is the mechanism for sharpness adaptivity and concentration, and how does that relate to the number of layers?

Original: To answer these questions, in this paper we consider a minimalist example of edge-of-stability.
Revised: To answer these questions, in this paper we consider a minimalist example of edge-of-stability.

Original: More specifically, we construct an objective function (4-layer scalar network with coupling entries), such that gradient descent on this objective has similar empirical behavior as deeper networks.
Revised: More specifically, we construct an objective function (4-layer scalar network with coupling entries), such that gradient descent on this objective has similar empirical behavior as deeper networks.

Original: We give a rigorous analysis for the training dynamics of this objective function in a large local region, which proves that the dynamics satisfy both sharpness adaptivity and sharpness concentration.
Revised: We give a rigorous analysis for the training dynamics of this objective function in a large local region, which proves that the dynamics satisfy both sharpness adaptivity and sharpness concentration.

Original: The global training dynamics for our objective exhibit a complicated fractal behavior (which is also why our rigorous results are local), and such behavior has been observed in training of neural networks.
Revised: The global training dynamics for our objective exhibit a complicated fractal behavior (which is also why our rigorous results are local), and such behavior has been observed in training of neural networks.

Original: [Figure]
Revised: [Figure]

Original: and (c), the sharpness converges very close to 2 /η with loss continuing to decrease.
Revised: , the sharpness converges very close to 2 /η with loss continuing to decrease.

Original: For (b), the sharpness decreases to be significantly lower than 2 /η .
Revised: For (b), the sharpness decreases to be significantly lower than 2 /η .

Original: 1. O UR R ESULTS
Revised: 1. O UR R ESULTS

Original: The objective function we consider is very simple:
Revised: The objective function we consider is very simple:

Original: L ( x, y, z, w ) ≜ 12 (1 − xyzw ) 2 .
Revised: L ( x, y, z, w ) ≜ 12 (1 − xyzw ) 2 .

Original: One can view this as a 4-layer scalar network (each layer has a single neuron).
Revised: One can view this as a 4-layer scalar network (each layer has a single neuron).

Original: We even couple the initialization so that x
Revised: We even couple the initialization so that x

Original: = z, y =
Revised: = z, y =

Original: w so effectively it becomes an objective on two variables L ( x, y ) ≜
Revised: w so effectively it becomes an objective on two variables L ( x, y ) ≜

Original: 14 (1 − x 2 y 2 ) 2 .
Revised: 14 (1 − x 2 y 2 ) 2 .

Original: For this objective function we prove its convergence and sharpness concentration properties:
Revised: For this objective function we prove its convergence and sharpness concentration properties:

Original: Theorem 1.1 (Sharpness Concentration, Informal) .
Revised: Theorem 1.1 (Sharpness Concentration, Informal) .

Original: For any learning rate η smaller than some constant, there is a constant size region S η such that the GD trajectory with step size η from all initializations in S η converge to a global minimum with sharpness within (2 /η − 203 η, 2 /η ) .
Revised: For any learning rate η smaller than some constant, there is a constant size region S η such that the GD trajectory with step size η from all initializations in S η converge to a global minimum with sharpness within (2 /η − 203 η, 2 /η ) .

Original: As a direct corollary, we can also prove that it has the sharpness adaptivity property.
Revised: As a direct corollary, we can also prove that it has the sharpness adaptivity property.

Original: Corollary 1.1 (Sharpness Adaptivity, Informal) .
Revised: Corollary 1.1 (Sharpness Adaptivity, Informal) .

Original: There exists a constant size region S and a corresponding range of step sizes K that for all η ∈ K , the GD trajectory with step size η from any initialization in S converges to a global minimum with sharpness within (2 /η − 203 η, 2 /η ) .
Revised: There exists a constant size region S and a corresponding range of step sizes K that for all η ∈ K , the GD trajectory with step size η from any initialization in S converges to a global minimum with sharpness within (2 /η − 203 η, 2 /η ) .

Original: The training dynamics are illustrated in Fig.
Revised: The training dynamics are illustrated in Fig.

Original: To analyze the training dynamics, we reparametrize the objective function and show that the 2-step dynamics of gradient descent roughly follow a parabola trajectory.
Revised: To analyze the training dynamics, we reparametrize the objective function and show that the 2-step dynamics of gradient descent roughly follow a parabola trajectory.

Original: The extreme point of this parabola is the final converging point which has sharpness very close to 2 /η .
Revised: The extreme point of this parabola is the final converging point which has sharpness very close to 2 /η .

Original: Intuitively, the parabola trajectory comes from a cubic term in the approximation of the training dynamics (see Section 3.1 for detailed discussions).
Revised: Intuitively, the parabola trajectory comes from a cubic term in the approximation of the training dynamics (see Section 3.1 for detailed discussions).

Original: We can also extend our result to a setting where x, y are replaced by vectors, see Theorem 3.2 in Section 3.3.
Revised: We can also extend our result to a setting where x, y are replaced by vectors, see Theorem 3.2 in Section 3.3.

Original: In Section 4 we explain the difference between the dynamics of our degree-4 model with degreemodels (which are more similar to matrix factorizations or 2-layer neural networks).
Revised: In Section 4 we explain the difference between the dynamics of our degree-4 model with degreemodels (which are more similar to matrix factorizations or 2-layer neural networks).

Original: We show that the dynamics for degree-2 models do not have the higher order terms, and their trajectories form an ellipse instead of a parabola.
Revised: We show that the dynamics for degree-2 models do not have the higher order terms, and their trajectories form an ellipse instead of a parabola.

Original: In Section 5 we show why it is difficult to extend Theorem 3.1 to global convergence – the training trajectory exhibits fractal behavior globally.
Revised: In Section 5 we show why it is difficult to extend Theorem 3.1 to global convergence – the training trajectory exhibits fractal behavior globally.

Original: Such behaviors can be qualitatively approximated by simple low-degree nonlinear dynamics standard in chaos theory, but are still very difficult to analyze.
Revised: Such behaviors can be qualitatively approximated by simple low-degree nonlinear dynamics standard in chaos theory, but are still very difficult to analyze.

Original: Finally, in Section 6 we present the similarity between our minimalist model and the GD trajectory of some over-parameterized deep neural networks trained on a real-world dataset.
Revised: Finally, in Section 6 we present the similarity between our minimalist model and the GD trajectory of some over-parameterized deep neural networks trained on a real-world dataset.

Original: Toward the end of convergence, the trajectory of the deep networks mostly lies on a 2-dimensional subspace and can be well characterized by a parabola as in the scalar case.
Revised: Toward the end of convergence, the trajectory of the deep networks mostly lies on a 2-dimensional subspace and can be well characterized by a parabola as in the scalar case.

Original: 1. R ELATED W ORKS
Revised: 1. R ELATED W ORKS

Original: The phenomenon of gradient descent on the Edge of Stability (EoS) was first formalized and empirically demonstrated in Cohen et al.
Revised: The phenomenon of gradient descent on the Edge of Stability (EoS) was first formalized and empirically demonstrated in Cohen et al.

Original: They show that the loss can non-monotonically decrease even when the sharpness λ > 2 /η .
Revised: They show that the loss can non-monotonically decrease even when the sharpness λ > 2 /η .

Original: The non-monotone property of the loss has also been observed in many other settings (Jastrzebski et al., 2020;
Revised: The non-monotone property of the loss has also been observed in many other settings (Jastrzebski et al., 2020;

Original: Xing et al., 2018; Lewkowycz et al., 2020; Wang et al., 2022; Arora et al., 2018; Li et al., 2022a).
Revised: Xing et al., 2018; Lewkowycz et al., 2020; Wang et al., 2022; Arora et al., 2018; Li et al., 2022a).

Original: Recently several works try to understand the mechanism behind EoS with different loss functions under various assumptions (Ahn et al., 2022; Ma et al., 2022; Arora et al., 2022; Lyu et al., 2022; Li et al., 2022b).
Revised: Recently several works try to understand the mechanism behind EoS with different loss functions under various assumptions (Ahn et al., 2022; Ma et al., 2022; Arora et al., 2022; Lyu et al., 2022; Li et al., 2022b).

Original: Ahn et al.
Revised: Ahn et al.

Original: (2022) studied the non-monotonic decreasing behavior of gradient descent (which they call unstable convergence ) and discussed the possible causes of this phenomenon.
Revised: (2022) studied the non-monotonic decreasing behavior of gradient descent (which they call unstable convergence ) and discussed the possible causes of this phenomenon.

Original: From a landscape perspective, Ma et al.
Revised: From a landscape perspective, Ma et al.

Original: (2022) defined a special subquadratic property of the loss function, and proved that EoS occurs based on this assumption.
Revised: (2022) defined a special subquadratic property of the loss function, and proved that EoS occurs based on this assumption.

Original: Despite the simplicity, their model displayed the EoS phenomenon without sharpness adaptivity.
Revised: Despite the simplicity, their model displayed the EoS phenomenon without sharpness adaptivity.

Original: Instead, our model focuses on a minimalist scalar network and proves the convergence results together with the sharpness adaptive phenomenon.
Revised: Instead, our model focuses on a minimalist scalar network and proves the convergence results together with the sharpness adaptive phenomenon.

Original: Arora et al.
Revised: Arora et al.

Original: and Lyu et al.
Revised: and Lyu et al.

Original: (2022) studied the implicit bias on the sharpness of gradient descent in some general loss function.
Revised: (2022) studied the implicit bias on the sharpness of gradient descent in some general loss function.

Original: Both works focus on the regime where the parameter is close to the manifold of minimum loss.
Revised: Both works focus on the regime where the parameter is close to the manifold of minimum loss.

Original: Arora et al.
Revised: Arora et al.

Original: (2022) proved that with a modified loss √ L or using normalized GD, gradient descent enters the EoS regime and has a sharpness reduction effect around the manifold of minima.
Revised: (2022) proved that with a modified loss √ L or using normalized GD, gradient descent enters the EoS regime and has a sharpness reduction effect around the manifold of minima.

Original: Lyu et al.
Revised: Lyu et al.

Original: (2022) provably showed how GD enters EoS regime and keeps reducing spherical sharpness on a scale-invariant objective.
Revised: (2022) provably showed how GD enters EoS regime and keeps reducing spherical sharpness on a scale-invariant objective.

Original: In both works, the effective step-size η changes throughout the training process, so sharpness adaptivity and concentration do not apply.
Revised: In both works, the effective step-size η changes throughout the training process, so sharpness adaptivity and concentration do not apply.

Original: Our results start from a simpler example without normalization, whereas the above works focus on general functions with normalized gradient or scale-invariance property.
Revised: Our results start from a simpler example without normalization, whereas the above works focus on general functions with normalized gradient or scale-invariance property.

Original: Another line of works (Lewkowycz et al., 2020; Wang et al., 2022) focuses on the implicit bias introduced by a large learning rate.
Revised: Another line of works (Lewkowycz et al., 2020; Wang et al., 2022) focuses on the implicit bias introduced by a large learning rate.

Original: Lewkowycz et al.
Revised: Lewkowycz et al.

Original: (2020) first proposed “catapult phase”, a regime similar to the EoS, where loss does not diverge even if sharpness is larger than 2 /η .
Revised: (2020) first proposed “catapult phase”, a regime similar to the EoS, where loss does not diverge even if sharpness is larger than 2 /η .

Original: Wang et al.
Revised: Wang et al.

Original: (2022) provided a convergence analysis on the matrix factorization problem for large learning rate beyond 2 /λ where λ is the sharpness.
Revised: (2022) provided a convergence analysis on the matrix factorization problem for large learning rate beyond 2 /λ where λ is the sharpness.

Original: Their results include two stages: in the first phase, the loss may oscillate but never diverge; the sharpness decreases to enter the second phase, where the loss decreases monotonically.
Revised: Their results include two stages: in the first phase, the loss may oscillate but never diverge; the sharpness decreases to enter the second phase, where the loss decreases monotonically.

Original: Recently Li et al.
Revised: Recently Li et al.

Original: (2022b) provided a theoretical analysis on sharpness along the GD trajectory in a two-layer linear network setting under some assumptions during the training process.
Revised: (2022b) provided a theoretical analysis on sharpness along the GD trajectory in a two-layer linear network setting under some assumptions during the training process.

Original: These works mostly focus on the degree-2 setting which does not have the sharpness adaptivity and sharpness concentration properties.
Revised: These works mostly focus on the degree-2 setting which does not have the sharpness adaptivity and sharpness concentration properties.

Original:  P RELIMINARIES AND N OTATIONS
Revised:  P RELIMINARIES AND N OTATIONS

Original: In this section, we introduce the minimalist model which exhibits both sharpness adaptivity and sharpness concentration.
Revised: In this section, we introduce the minimalist model which exhibits both sharpness adaptivity and sharpness concentration.

Original: 2. G RADIENT D ESCENT ON P RODUCT OF 4 S CALARS
Revised: 2. G RADIENT D ESCENT ON P RODUCT OF 4 S CALARS

Original: We focus on the simple objective L ( x, y, z, w ) ≜ 12 (1 − xyzw ) 2 .
Revised: We focus on the simple objective L ( x, y, z, w ) ≜ 12 (1 − xyzw ) 2 .

Original: Let the learnable parameters x, y, z, w ∈ R to be trained using gradient descent with a fixed step size η ∈ R + that
Revised: Let the learnable parameters x, y, z, w ∈ R to be trained using gradient descent with a fixed step size η ∈ R + that

Original: [Equation]
Revised: [Equation]

Original: Here x t denotes the value of parameter x after the t -th update.
Revised: Here x t denotes the value of parameter x after the t -th update.

Original: To further simplify the problem, we consider the symmetric initialization of z 0 = x 0 , w 0 = y 0 .
Revised: To further simplify the problem, we consider the symmetric initialization of z 0 = x 0 , w 0 = y 0 .

Original: Note that due to symmetry of objective, the identical entries will remain identical throughout the training process, so the training dynamics reduces to two dimensional and the 1-step update of x and y follows
Revised: Note that due to symmetry of objective, the identical entries will remain identical throughout the training process, so the training dynamics reduces to two dimensional and the 1-step update of x and y follows

Original: [Equation]
Revised: [Equation]

Original: It’s easy to show that the set of global minima for this function form the hyperbola xy = 1 .
Revised: It’s easy to show that the set of global minima for this function form the hyperbola xy = 1 .

Original: Without loss of generality we focus on the case when x, y > 0 , and in most of the analysis we also focus on the side where x > y .
Revised: Without loss of generality we focus on the case when x, y > 0 , and in most of the analysis we also focus on the side where x > y .

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, with GD running on such a minimal model, we observe convergence on EoS for a wide range of initializations.
Revised: 2, with GD running on such a minimal model, we observe convergence on EoS for a wide range of initializations.

Original: Eventually all such trajectories converge to minima that are just slightly flatter than the “EoS minima” (the minima whose sharpness is exactly 2 /η , see Definition 1).
Revised: Eventually all such trajectories converge to minima that are just slightly flatter than the “EoS minima” (the minima whose sharpness is exactly 2 /η , see Definition 1).

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: In the 2D trajectory, the 2-step movement quickly converges to some smooth curves ending very close to the EoS minimum.

Original: In (b) we demonstrate sharpness concentration by running GD with constant learning rate η = 0 .

Original: 2 for 50000 iterations from a dense grid of initializations and plot the sharpness of their converging minima.

Original: Initializations in the red shaded area all converge to a minima with sharpness in (2 /η − 0 .

Original: 1 , 2 /η ) .

Original:  E O S M INIMA AND R EPARAMETERIZATION
Revised: 2. E O S M INIMA AND R EPARAMETERIZATION

Original: Given that a wide range of initializations all converge very close to the “EoS minima” with sharpness 2 /η , we want to concretely characterize those points.
Revised: Given that a wide range of initializations all converge very close to the “EoS minima” with sharpness 2 /η , we want to concretely characterize those points.

Original: The complete calculations are deferred to
Revised: The complete calculations are deferred to

Original: Appendix B.1.
Revised: Appendix B.1.

Original: Denote γ = xy , the Hessian of the objective L at ( x, x, y, y ) admits eigenvalues
Revised: Denote γ = xy , the Hessian of the objective L at ( x, x, y, y ) admits eigenvalues

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and λ 3 = x 2 (1 − γ ) , λ
Revised: and λ 3 = x 2 (1 − γ ) , λ

Original: 4 =
Revised: 4 =

Original: y 2 (1 − γ ) .
Revised: y 2 (1 − γ ) .

Original: When ( x, y ) converges to any minimum, γ = xy = 1 , so λ 2 , λ 3 , λ 4 all vanishes.
Revised: When ( x, y ) converges to any minimum, γ = xy = 1 , so λ 2 , λ 3 , λ 4 all vanishes.

Original: Therefore it is λ 1 that corresponds to the EoS phenomenon people observe.
Revised: Therefore it is λ 1 that corresponds to the EoS phenomenon people observe.

Original: When η < 12 , solving λ 1 = 2 /η with x 2 y 2 = 1 gives x = ± 1 √ 2 (( − 4 + η − 2 ) 12 + η − 1 ) 12 , y = ±√ 2(( − 4 + η − 2 ) 12 + η − 1 ) − 12 and their multiplicative inverses.
Revised: When η < 12 , solving λ 1 = 2 /η with x 2 y 2 = 1 gives x = ± 1 √ 2 (( − 4 + η − 2 ) 12 + η − 1 ) 12 , y = ±√ 2(( − 4 + η − 2 ) 12 + η − 1 ) − 12 and their multiplicative inverses.

Original: These solutions correspond to the minima with sharpness exactly equal to the EoS threshold of 2 /η .
Revised: These solutions correspond to the minima with sharpness exactly equal to the EoS threshold of 2 /η .

Original: Since they are all symmetric with each other, without loss of generality we pick the minimum of interest as follows.
Revised: Since they are all symmetric with each other, without loss of generality we pick the minimum of interest as follows.

Original: Definition 1 ( η -EoS Minimum) .
Revised: Definition 1 ( η -EoS Minimum) .

Original: For any step size η ∈ (0 , 12 ) , the η -EoS minimum under the ( x, y ) - parameterization is
Revised: For any step size η ∈ (0 , 12 ) , the η -EoS minimum under the ( x, y ) - parameterization is

Original: [Equation]
Revised: [Equation]

Original: Though we are able to obtain a closed-form expression for the EoS minimum, its x - y coordinate could still be tricky to analyze.
Revised: Though we are able to obtain a closed-form expression for the EoS minimum, its x - y coordinate could still be tricky to analyze.

Original: Thus we consider the following reparameterization:
Revised: Thus we consider the following reparameterization:

Original: For any ( x, y ) ∈ { ( x, y ) ∈ R + × R + : x > y } , define c ≜ ( x 2 − y 2 ) 12 and d ≜ xy .
Revised: For any ( x, y ) ∈ { ( x, y ) ∈ R + × R + : x > y } , define c ≜ ( x 2 − y 2 ) 12 and d ≜ xy .

Original: This gives a bijective continuous mapping between { ( x, y ) ∈ R + × R + : x > y } and { ( c, d ) ∈ R + × R + } .
Revised: This gives a bijective continuous mapping between { ( x, y ) ∈ R + × R + : x > y } and { ( c, d ) ∈ R + × R + } .

Original: This is a natural reparameterization since intuitively the basis in the new coordinate system are the two orthogonal family of hyperbolas
Revised: This is a natural reparameterization since intuitively the basis in the new coordinate system are the two orthogonal family of hyperbolas

Original: xy = C and x 2 −
Revised: xy = C and x 2 −

Original: y 

Revised: y

Original: = C .
Revised: = C .

Original: The former captures the movement orthogonal to the manifold of minima xy = 1 while the latter captures the movement along the manifold of minima.
Revised: The former captures the movement orthogonal to the manifold of minima xy = 1 while the latter captures the movement along the manifold of minima.

Original: Note that a similar separation of dynamics was also used in Arora et al. (2022).
Revised: Note that a similar separation of dynamics was also used in Arora et al. (2022).

Original: With c, d as defined, the η -EoS minimum simplifies to (˘ c, ˘ d ) ≜
Revised: With c, d as defined, the η -EoS minimum simplifies to (˘ c, ˘ d ) ≜

Original: ( η − 2 − 4) 14 , 1) .
Revised: ( η − 2 − 4) 14 , 1) .

Original: To expand the dynamics near the η -EoS minimum, we let a ≜ c − ( η − 2 − 4) 14 and b ≜ d − 1 to be the offset from (˘ c, ˘ d ) .
Revised: To expand the dynamics near the η -EoS minimum, we let a ≜ c − ( η − 2 − 4) 14 and b ≜ d − 1 to be the offset from (˘ c, ˘ d ) .

Original: Our analysis will primarily be using the ( a, b ) -parameterization.
Revised: Our analysis will primarily be using the ( a, b ) -parameterization.

Original: Definition 2 ( η -EoS Reparameterization) .
Revised: Definition 2 ( η -EoS Reparameterization) .

Original: For any step size η > 0 , for any ( x, y ) ∈ R + × R + such that x > y , the ( a, b ) reparameterization of ( x, y ) are respectively given by
Revised: For any step size η > 0 , for any ( x, y ) ∈ R + × R + such that x > y , the ( a, b ) reparameterization of ( x, y ) are respectively given by

Original: [Equation]
Revised: [Equation]

Original: Let κ ≜ √ η , following Eq.
Revised: Let κ ≜ √ η , following Eq.

Original: (2), the 1-step update under the reparameterization becomes
Revised: (2), the 1-step update under the reparameterization becomes

Original: [Equation]
Revised: [Equation]

Original: Now we can proceed to analyze the dynamics of this simple example.
Revised: Now we can proceed to analyze the dynamics of this simple example.

Original:  D YNAMICS OF G RADIENT D ESCENT ON P RODUCT OF 4 S CALARS
Revised:  D YNAMICS OF G RADIENT D ESCENT ON D EGREE -4 M ODEL

Original: In this section, we will rigorously analyze the training dynamics characterized by Eq.
Revised: In this section, we will rigorously analyze the training dynamics characterized by Eq.

Original: First we will introduce the approximation of one and two-step update and build up intuition on the dynamics.
Revised: First we will introduce the approximation of one and two-step update and build up intuition on the dynamics.

Original: Then we will present our main theoretical results that the 4 scalar example exhibits both characterizations of EoS training.
Revised: Then we will present our main theoretical results that the degree-4 model exhibits both characterizations of EoS training.

Original: 3. A PPROXIMATING 1-S TEP AND 2-S TEP U PDATES
Revised: 3. A PPROXIMATING 1-S TEP AND 2-S TEP U PDATES

Original: Here we introduce the informal approximation on Eq.
Revised: Here we introduce the informal approximation on Eq.

Original: (6) and the corresponding two-step updates.
Revised: (6) and the corresponding two-step updates.

Original: For cleanness of presentation we will use ≈ to hide all dominated terms.
Revised: For cleanness of presentation we will use ≈ to hide all dominated terms.

Original: The rigorous statements of the approximations and corresponding proofs are deferred to Appendix B.3.
Revised: The rigorous statements of the approximations and corresponding proofs are deferred to Appendix B.3.

Original: When we are only describing the one/two-step dynamics, we use a, a ′ , a ′′ to denote a t , a t +1 , a t +2 and b, b ′ , b ′′ to denote b t , b t +1 , b t +2 .
Revised: When we are only describing the one/two-step dynamics, we use a, a ′ , a ′′ to denote a t , a t +1 , a t +2 and b, b ′ , b ′′ to denote b t , b t +1 , b t +2 .

Original: Denoting κ ≜ √ η , when κ, | a | , | b | are all not too large (see precise ranges in condition B.1), we have
Revised: Denoting κ ≜ √ η , when κ, | a | , | b | are all not too large (see precise ranges in condition B.1), we have

Original: [Equation]
Revised: [Equation]

Original: In the approximation, a is monotonically decreasing at a steady rate of 2 b 3 κ 3 per step.
Revised: In the approximation, a is monotonically decreasing at a steady rate of 2 b 3 κ 3 per step.

Original: The one step update of b is flipping signs and contains second and third order terms of b .
Revised: The one step update of b is flipping signs and contains second and third order terms of b .

Original: For the two-step approximation however, the oscillation behavior and the even-order terms of b all cancels.
Revised: For the two-step approximation however, the oscillation behavior and the even-order terms of b all cancels.

Original: This is consistent with the analysis in (Arora et al., 2022) that the two step dynamics travels along a sharpness reducing flow.
Revised: This is consistent with the analysis in (Arora et al., 2022) that the two step dynamics travels along a sharpness reducing flow.

Original: Before proceeding to analyze the discrete GD movement, we first get intuition by approximating the two-step dynamics with a simple ODE
Revised: Before proceeding to analyze the discrete GD movement, we first get intuition by approximating the two-step dynamics with a simple ODE

Original: [Equation]
Revised: [Equation]

Original: This would be the limit when κ is going to 0 and the movement of two-step dynamics become very small.
Revised: This would be the limit when κ is going to 0 and the movement of two-step dynamics become very small.

Original: The general solution for Eq.
Revised: The general solution for Eq.

Original: (8) is given by
Revised: (8) is given by

Original: [Equation]
Revised: [Equation]

Original: for some constant C ∈ R .
Revised: for some constant C ∈ R .

Original: As a decreases following
Revised: As a decreases following

Original: Eq.
Revised: Eq.

Original: (7), the trajectory converges toward the parabola b 2 = 12 aκ + 116 κ 4 .
Revised: (7), the trajectory converges toward the parabola b 2 = 12 aκ + 116 κ 4 .

Original: Note that the convergence to the parabola is exponential with respect to a , so if a is initialized positive and not too small, it will converge to a minima that is very close to a = − 18 κ 3 as shown in Fig.
Revised: Note that the convergence to the parabola is exponential with respect to a , so if a is initialized positive and not too small, it will converge to a minima that is very close to a = − 18 κ 3 as shown in Fig.

Original: This is a minimum that is just slightly flatter than the κ 2 -EoS minimum.
Revised: This is a minimum that is just slightly flatter than the κ 2 -EoS minimum.

Original: [Figure]
Revised: [Figure]

Original: 3. C ONVERGENCE ON E O S FOR P RODUCT OF 4 S CALARS
Revised: 3. C ONVERGENCE ON E O S FOR THE D EGREE -4 M ODEL

Original: Now we state our convergence result on the 4 scalar objective under ( a, b ) -parameterization.
Revised: Now we state our convergence result on the 4 scalar objective under ( a, b ) -parameterization.

Original: Theorem 3.1 (Sharpness Concentration) .
Revised: Theorem 3.1 (Sharpness Concentration) .

Original: For a large enough absolute constant K , suppose κ < 1 2000 √ 2 K − 1 , and the initialization ( a 0 , b 0 ) satisfies a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .
Revised: For a large enough absolute constant K , suppose κ < 1 2000 √ 2 K − 1 , and the initialization ( a 0 , b 0 ) satisfies a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .

Original: Consider the GD trajectory characterized in Eq.
Revised: Consider the GD trajectory characterized in Eq.

Original: (6) with fixed step size κfrom ( a 0 , b 0 ) , for any ϵ > 0 there exists T
Revised: (6) with fixed step size κfrom ( a 0 , b 0 ) , for any ϵ > 0 there exists T

Original: = O
Revised: = O

Original: ( K − 2 κ − 152 + log( ϵ − 1 ) + log( | b 0 | − 1 ) κ − 72 ) such that for all t > T , | b t | <
Revised: ( K − 2 κ − 152 + log( ϵ − 1 ) + log( | b 0 | − 1 ) κ − 72 ) such that for all t > T , | b t | <

Original: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: Under the context of x, y coordinate and sharpness, Theorem 3.1 gives the following corollary:
Revised: Under the context of x, y coordinate and sharpness, Theorem 3.1 gives the following corollary:

Original: Corollary 3.1 (Sharpness Concentration under ( x, y ) -Parameterization) .
Revised: Corollary 3.1 (Sharpness Concentration under ( x, y ) -Parameterization) .

Original: For a large enough absolute constant K , suppose η < 1 8000000 K − 2 , and the initialization ( x 0 , y 0 ) satisfies x 0 ∈ (˘ x + 13 η 54 ,
Revised: For a large enough absolute constant K , suppose η < 1 8000000 K − 2 , and the initialization ( x 0 , y 0 ) satisfies x 0 ∈ (˘ x + 13 η 54 ,

Original: ˘ x + 15 K − 2 η − 12 ) and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) where (˘ x, ˘ y ) is the η -EoS minima de- fined in Definition 1.
Revised: ˘ x + 15 K − 2 η − 12 ) and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) where (˘ x, ˘ y ) is the η -EoS minima de- fined in Definition 1.

Original: The GD trajectory characterized in Eq.
Revised: The GD trajectory characterized in Eq.

Original: (2) with fixed step size η from ( x 0 , y 0 ) will converge to a global minimum with sharpness λ ∈ ( 2 η − 203 η, 2 η ) .
Revised: (2) with fixed step size η from ( x 0 , y 0 ) will converge to a global minimum with sharpness λ ∈ ( 2 η − 203 η, 2 η ) .

Original: Note that when the step size η (and hence κ ) is relatively small, the final sharpness is very close to 2 /η .
Revised: Note that when the step size η (and hence κ ) is relatively small, the final sharpness is very close to 2 /η .

Original: The range of initialization that satisfies the requirement is quite large: in the original ( x, y ) - parameterization it contains a box of width Θ( K − 2 η − 12 ) and height Θ( K −1 η12 ) .
Revised: The range of initialization that satisfies the requirement is quite large: in the original ( x, y ) - parameterization it contains a box of width Θ( K − 2 η − 12 ) and height Θ( K − η

Original: Many of the
Revised: Many of the

Original: initial
Revised: initial

Original: points can be far from the EoS-minimum.
Revised: points can be far from the EoS-minimum.

Original: The complete proofs are deferred to Appendix B.6.
Revised: The complete proofs are deferred to Appendix B.6.

Original: Here we discuss the proof sketch of Theorem 3.1.
Revised: Here we discuss the proof sketch of Theorem 3.1.

Original: Our convergence analysis focuses on the 2-step update.
Revised: Our convergence analysis focuses on the 2-step update.

Original: It contains two phases:
Revised: It contains two phases:

Original: Phase 1.
Revised: Phase 1.

Original: (Convergence to near parabola)
Revised: (Convergence to near parabola)

Original: We consider initializations in region I, II, and III.
Revised: We consider initializations in region I, II, and III.

Original: • In I, b ′′ − b is dominated by − b 3 and ( a, b ) follows an exponential trajectory.
Revised: • In I, b ′′ − b is dominated by − b 3 and ( a, b ) follows an exponential trajectory.

Original: We show that | b | decreases exponen- tially with respect to a and enters region II (Lemma 8).
Revised: We show that | b | decreases exponen- tially with respect to a and enters region II (Lemma 8).

Original: • In III, b ′′ − b is dominated by abκ and ( a, b ) follows an elliptic trajectory centered at (0 , 0) .
Revised: • In III, b ′′ − b is dominated by abκ and ( a, b ) follows an elliptic trajectory centered at (0 , 0) .

Original: We show that | b | increases at superlinearly with respect to a and enters II (Lemma 9).
Revised: We show that | b | increases at superlinearly with respect to a and enters II (Lemma 9).

Original: • We also show that once ( a, b ) enters II, it will stay in II until it exits from the left and enters IV (Lemma 11).
Revised: • We also show that once ( a, b ) enters II, it will stay in II until it exits from the left and enters IV (Lemma 11).

Original: Thus after Phase 1, all initializations will be in IV.
Revised: Thus after Phase 1, all initializations will be in IV.

Original: Phase 2.
Revised: Phase 2.

Original: (Convergence along parabola) • After ( a, b ) enters IV, we show that it will further converge to the parabola that | b 2 − 12 aκ − 116 κ 4 | < 1200 κ 4 will be satisfied before a decreases to κ 52 and enters V (Lemma 13).
Revised: (Convergence along parabola) • After ( a, b ) enters IV, we show that it will further converge to the parabola that | b 2 − 12 aκ − 116 κ 4 | < 1200 κ 4 will be satisfied before a decreases to κ 52 and enters V (Lemma 13).

Original: • Then we show that the inequality will be preserved in V while it moves left until it enters VI (Lemma 14).
Revised: • Then we show that the inequality will be preserved in V while it moves left until it enters VI (Lemma 14).

Original: • In VI, the dynamics is again similar to III, but with a being negative.
Revised: • In VI, the dynamics is again similar to III, but with a being negative.

Original: We conclude our proof by showing | b | will converge to 0 superlinearly with respect to a (Lemma 15).
Revised: We conclude our proof by showing | b | will converge to 0 superlinearly with respect to a (Lemma 15).

Original: [Figure]
Revised: [Figure]

Original: arrows indicate the directions of local 2-step movement
Revised: indicate the directions of local 2-step movement.

Original: This diagram is only for demonstration purpose and ratios are 
Revised: This diagram is only for demonstration purpose and ratios are not exact.

Original: not exact. Following Theorem 3.1, we can also formally characterize the sharpness adaptive phenomenon for a local region using the following corollary.
Revised: Following Theorem 3.1, we can also formally characterize the sharpness adaptive phenomenon for a local region using the following corollary.

Original: The proof is deferred to Appendix B.6.2.
Revised: The proof is deferred to Appendix B.6.2.

Original: Corollary 3.2 (Sharpness Adaptivity) .
Revised: Corollary 3.2 (Sharpness Adaptivity) .

Original: For a large enough constant K , fix any α < 1 2000 √ 2 K − 1 .
Revised: For a large enough constant K , fix any α < 1 2000 √ 2 K − 1 .

Original: For all initialization ( x 0 , y 0 ) in the region characterized by
Revised: For all initialization ( x 0 , y 0 ) in the region characterized by

Original: [Equation]
Revised: [Equation]

Original: and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) , the GD trajectory from ( x 0 , y 0 ) characterized by Eq.
Revised: and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) , the GD trajectory from ( x 0 , y 0 ) characterized by Eq.

Original: (2) with any step size η ∈ ( α 2 − 1 10 K − 2 α 2 , α 2 ) will converge to a minima with sharpness λ ∈ ( 2 η − 20 3 η, 2 η ) .
Revised: (2) with any step size η ∈ ( α 2 − 1 10 K − 2 α 2 , α 2 ) will converge to a minima with sharpness λ ∈ ( 2 η − 20 3 η, 2 η ) .

Original: 3. C ONVERGENCE ON E O S FOR R ANK -1 F ACTORIZATION OF I SOTROPIC M ATRIX
Revised: 3. C ONVERGENCE ON E O S FOR R ANK -1 F ACTORIZATION OF I SOTROPIC M ATRIX

Original: Inspired by the scalar factorization problem, we extend it to a rank-1 factorization of an isotropic matrix.
Revised: Inspired by the scalar factorization problem, we extend it to a rank-1 factorization of an isotropic matrix.

Original: In particular, we consider the following optimization problem:
Revised: In particular, we consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: Similar to the under-parameterized case in Wang et al.
Revised: Similar to the under-parameterized case in Wang et al.

Original: (2022), this problem also guarantees the alignment between x and y if ( x , y ) is a global minimum, i.e., x = c y for some c ∈ R .
Revised: (2022), this problem also guarantees the alignment between x and y if ( x , y ) is a global minimum, i.e., x = c y for some c ∈ R .

Original: To prove the convergence for Eq.
Revised: To prove the convergence for Eq.

Original: (11) at the edge of stability, we first prove the alignment can be soon achieved.
Revised: (11) at the edge of stability, we first prove the alignment can be soon achieved.

Original: After the alignment, we prove the equivalence between this problem and the case of the product of 4 scalars, and prove the convergence of this problem.
Revised: After the alignment, we prove the equivalence between this problem and the degreescalar model, and prove the convergence of this problem.

Original: We directly give the final theorem and the proof is deferred to Appendix C.
Revised: We directly give the final theorem and the proof is deferred to Appendix C.

Original: The experiments demonstrate similar EoS phenomenon (See Appendix A.5).
Revised: The experiments demonstrate similar EoS phenomenon (See Appendix A.5).

Original: Theorem 3.2.
Revised: Theorem 3.2.

Original: For a large enough absolute constant K , with all the initialization ( x 0 , y 0 ) satisfying x 0 ∼ δ x Unif ( S d − 1 ) , y 0 ∼ δ y Unif( S d − 1 ) 2 , δ x δ y
Revised: For a large enough absolute constant K , with all the initialization ( x 0 , y 0 ) satisfying x 0 ∼ [Equation] Unif

Original: =
Revised: [Equation]

Original: 12 , δ x ∈ (˘ x + 180 K − 2 η − 12 , ˘ x + 18 K − 2 η − 12 ) , if step size η < min { K − 4 8000000 ,
Revised: [Equation] [Equation] if step size η < min { K − 4 8000000 ,

Original: K − 2 20000+2000(log( d ) − log( δ 0 )) } , and a multiplicative perturbation y ′ t =
Revised: K − 2 20000+2000(log( d ) − log( δ 0 )) } , and a multiplicative perturbation y ′ t =

Original: y t (1 + 2 K − 1 ) is performed at time t = t p for some t p >
Revised: y t (1 + 2 K − 1 ) is performed at time t = t p for some t p >

Original: O ( − log( η ) + log( d ) − log( δ 0 )
Revised: O ( − log( η ) + log( d ) − log( δ 0 )

Original: + K 3 ) , then for any ϵ > 0 , with probability p > 1 − 2 δ 0 − 2 exp {− Ω( d ) } there exists T
Revised: + K 3 ) , then for any ϵ > 0 , with probability p > 1 − 2 δ 0 − 2 exp {− Ω( d ) } there exists T

Original: = O
Revised: = O

Original: ( K − 2 κ − 152 − log( ϵ ) − log( δ 0 ))
Revised: ( K − 2 κ − 152 − log( ϵ ) − log( δ 0 ))

Original: such that for all t > T , L ( x, y ) <
Revised: such that for all t > T , L ( x, y ) <

Original: ϵ and ∥ x t ∥ 2 + ∥
Revised: ϵ and ∥ x t ∥ 2 + ∥

Original: y t ∥ 2 ∈ ( 1 η − 10 3 η, 1 η ) .
Revised: y t ∥ 2 ∈ ( 1 η − 10 3 η, 1 η ) .

Original: Note that we require an additional perturbation because we need to guarantee that the trajectory does not converge to an unstable point (where sharpness λ > 2 /η ).
Revised: Note that we require an additional perturbation because we need to guarantee that the trajectory does not converge to an unstable point (where sharpness λ > 2 /η ).

Original: This was proved without perturbation for the scalar case but is more challenging in higher dimensions.
Revised: This was proved without perturbation for the scalar case but is more challenging in higher dimensions.

Original: The objective will still converge to a minimum very close to an η -EoS minimum.
Revised: The objective will still converge to a minimum very close to an η -EoS minimum.

Original: The experiment results are available in Appendix A.5.
Revised: The experiment results are available in Appendix A.5.

Original:  D IFFERENCES IN D EGREE -2 AND H IGHER D EGREE M ODELS
Revised:  D IFFERENCES IN D EGREE -2 AND H IGHER D EGREE M ODELS

Original: In this section, we will look at some similar models of lower degree, and explain why for degreemodels the sharpness of final converging point is often farther from 2 /η compared to higher degree models.
Revised: In this section, we will look at some similar models of lower degree, and explain why for degreemodels the sharpness of final converging point is often farther from 2 /η compared to higher degree models.

Original: We will use similar methods as in Section 2 and Section 3 to gain intuition for the dynamics.
Revised: We will use similar methods as in Section 2 and Section 3 to gain intuition for the dynamics.

Original: Previous works including (Chen & Bruna, 2022) and (Wang et al., 2022) have studied the dynamics of beyond EoS training on the problem of factorizing a single scalar or an isotropic matrix into two components.
Revised: Previous works including (Chen & Bruna, 2022) and (Wang et al., 2022) have studied the dynamics of beyond EoS training on the problem of factorizing a single scalar or an isotropic matrix into two components.

Original: The objectives studied includes min x , y ∈ R d ( µ − x ⊤ y ) 2 , min x , y ∈ R d ∥ µ I d − xy ⊤ ∥ 2 F , and the corresponding scalar case min x,y ∈ R ( µ − xy ) 2 .
Revised: The objectives studied includes min x , y ∈ R d ( µ − x ⊤ y ) 2 , min x , y ∈ R d ∥ µ I d − xy ⊤ ∥ 2 F , and the corresponding scalar case min x,y ∈ R ( µ − xy ) 2 .

Original: They were able to show that for initializations with sharpness greater than 2 /η , GD with constant learning rate η provably converges to a global minimum with sharpness less or equal to 2 /η .
Revised: They were able to show that for initializations with sharpness greater than 2 /η , GD with constant learning rate η provably converges to a global minimum with sharpness less or equal to 2 /η .

Original: Empirically, the sharpness reduction process on these 2-component objectives will usually “overshoot” the EoS threshold and converge to a minima that is significantly flatter than the EoS minimum, and one does not observe the oscillation of sharpness around the EoS threshold (see Appendix A.3).
Revised: Empirically, the sharpness reduction process on these 2-component objectives will usually “overshoot” the EoS threshold and converge to a minima that is significantly flatter than the EoS minimum, and one does not observe the oscillation of sharpness around the EoS threshold (see Appendix A.3).

Original: In this section we consider the scalar objective min x,y ∈ R (1 − xy ) 2 since it is able to captures the major dynamical properties of those more complex objectives as discussed in Wang et al.
Revised: In this section we consider the scalar objective min x,y ∈ R (1 − xy ) 2 since it is able to captures the major dynamical properties of those more complex objectives as discussed in Wang et al.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 5, initializations with sharpness exceeding the EoS threshold will converge to a minima that is distinguishably flatter than the EoS minimum, and globally there is not a region of initialization that gives EoS convergence.
Revised: 5, initializations with sharpness exceeding the EoS threshold will converge to a minima that is distinguishably flatter than the EoS minimum, and globally there is not a region of initialization that gives EoS convergence.

Original: Unlike the parabola for the degree-4 case, the two step update travels in a roughly circular trajectory centered at the κ 2 -EoS minimum as shown in Fig.
Revised: Unlike the parabola for the degree-4 case, the 2-step update travels in a roughly circular trajectory centered at the κ 2 -EoS minimum as shown in Fig.

Original: 5a (right).
Revised: 5a (right).

Original: Therefore locally we observe that sharper initializations tend to converge to flatter minima.
Revised: Therefore locally we observe that sharper initializations tend to converge to flatter minima.

Original: The difference between the degree-2 and degree-4 case can be easily explained by a local expansion.
Revised: The difference between the degree-2 and degree-4 case can be easily explained by a local expansion.

Original: Using the same ( c, d ) -reparameterization and setting ( a, b ) to be the offset of ( c, d ) from the EoS minimum, the two step update of ( a, b ) under learning rate κ 2 can be approximated by
Revised: Using the same ( c, d ) -reparameterization and setting ( a, b ) to be the offset of ( c, d ) from the EoS minimum, the two step update of ( a, b ) under learning rate κ 2 can be approximated by

Original: [Equation]
Revised: [Equation]

Original: This is very similar to Eq.
Revised: This is very similar to Eq.

Original: (7) except that we no longer have the − b 3 term for the 2-step update on b which was attracting b close to 0.
Revised: (7) except that we no longer have the − b 3 term for the 2-step update on b which was attracting b close to 0.

Original: In this case, the ODE approximation d b/ d a = 4 a/bκ 2 gives the general solution b 2 = 4( C − a 2 ) /κ
Revised: In this case, the ODE approximation d b/ d a = 4 a/bκ 2 gives the general solution b 2 = 4( C − a 2 ) /κ

Original: 2 for C ∈ R
Revised: 2 for C ∈ R

Original: + , which corresponds to the family of ellipses centered at (0 , 0)
Revised: + , which corresponds to the family of ellipses centered at (0 , 0)

Original: and matches the two step trajectory in Fig.
Revised: and matches the two step trajectory in Fig.

Original: 5a.
Revised: 5a.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: minima. In Appendix A.2.3, we discuss a degree-3 model exhibiting mixed behavior around different EoS minima, which further verifies our explanation above.
Revised: In Appendix A.2.3, we discuss a degree-3 model exhibiting mixed behavior around different EoS minima, which further verifies our explanation above.

Original: We also empirically note that the coupling of entries will naturally arise when training general scalar networks from non-coupling initializations.
Revised: We also empirically note that the coupling of entries will naturally arise when training general scalar networks from non-coupling initializations.

Original: Thus it is not an artifact we have to impose on the model to observe EoS (see Appendix A.4).
Revised: Thus it is not an artifact we have to impose on the model to observe EoS (see Appendix A.4).

Original:  G LOBAL T RAJECTORY AND C HAOS
Revised:  G LOBAL T RAJECTORY AND C HAOS

Original: There exists very limited global convergence analysis for constant step size gradient descent training beyond EoS on complicated non-convex objectives.
Revised: There exists very limited global convergence analysis for constant step size gradient descent training beyond EoS on complicated non-convex objectives.

Original: Even for the product of 4 scalars, the boundary separating converging and diverging initializations (Fig.
Revised: Even for the product of 4 scalars, the boundary separating converging and diverging initializations (Fig.

Original: 6a) exhibits complicated fractal structures.
Revised: 6a) exhibits complicated fractal structures.

Original: Moreover, we observe that for initializations close to such boundary, their GD training trajectories usually begin with a phase of chaotic oscillation which eventually “de-bifurcates” and converges to the parabolic two-step trajectory as discussed in Section 3.
Revised: Moreover, we observe that for initializations close to such boundary, their GD training trajectories usually begin with a phase of chaotic oscillation which eventually “de-bifurcates” and converges to the parabolic two-step trajectory as discussed in Section 3.

Original: Similar oscillation phenomenon has also been empirically observed by Ruiz-Garcia et al.
Revised: Similar oscillation phenomenon has also been empirically observed by Ruiz-Garcia et al.

Original: (2021) in neural networks when they increase the learning rate and destabilize the network from a local trajectory.
Revised: (2021) in neural networks when they increase the learning rate and destabilize the network from a local trajectory.

Original: So what is causing the bifurcation?
Revised: So what is causing the bifurcation?

Original: Previously, Ruiz-Garcia et al.
Revised: Previously, Ruiz-Garcia et al.

Original: (2021) attributed the phenomenon to the cascading effect of oscillation along multiple large eigendirections of the network.
Revised: (2021) attributed the phenomenon to the cascading effect of oscillation along multiple large eigendirections of the network.

Original: Yet this explanation is quite unsatisfying for our simple model as there is only one oscillating direction.
Revised: Yet this explanation is quite unsatisfying for our simple model as there is only one oscillating direction.

Original: [Figure]
Revised: [Figure]

Original: ) Converging and diverging initialization ( η = 0 . 2 ) [Figure] [Figure] [Figure]
Revised: (a) Converging and diverging initialization ( η = 0 . 2 ).

Original: [Figure]

Original: a − 0 .

Original: [Figure]

Original: − 0 .− 0 .

Original: 4 − 0 .0 .0 .0 .

Original: [Figure]
Revised: [Figure]

Original: b

Original: [Figure]

Original: a

Original: [Figure]

Original: [Figure] Parameterized [Figure] Looking closely to the trajectory (Fig. 6b), one will find it very similar to the bifurcation diagram of self-recurrent polynomial maps (such as the famous logistic map x t +1 = rx t (1 − x t ) parameterized by r ).
Revised: Looking closely to the trajectory (Fig. 6b), one will find it very similar to the bifurcation diagram of self-recurrent polynomial maps (such as the famous logistic map x t +1 = rx t (1 − x t ) parameterized by r ).

Original: In the degree, the existence of such self-recurrent map is explicit since following Eq.
Revised: In the degree-4 model, the existence of such self-recurrent map is explicit since following Eq.

Original: (7), the approximate 2-step update of b can be rewritten as b ′′ =
Revised: (7), the approximate 2-step update of b can be rewritten as b ′′ =

Original: b (1 + 8 aκ − 16 b 2 ) .
Revised: b (1 + 8 aκ − 16 b 2 ) .

Original: If we consider a to be relatively stationary, the trajectory of b will be locally characterized by the self-recurrent 1D nonlinear dynamical system b t +1 =
Revised: If we consider a to be relatively stationary, the trajectory of b will be locally characterized by the self-recurrent 1D nonlinear dynamical system b t +1 =

Original: b t (1 + 8 aκ − 16 b 2 t ) parameterized by a .
Revised: b t (1 + 8 aκ − 16 b 2 t ) parameterized by a .

Original: In
Revised: In

Original: Fig.
Revised: Fig.

Original: 6c, we compute the bifurcation diagram for the recurrent map numerically and see that they are qualitatively similar.
Revised: 6c, we compute the bifurcation diagram for the recurrent map numerically and see that they are qualitatively similar.

Original: Following this analogy, one may instantly relate the first bifurcating point with the EoS minima that the trajectory eventually converges to, and the non-bifurcating regime for the polynomial maps with the “sub-EoS regime” on the left (in Fig.
Revised: Following this analogy, one may instantly relate the first bifurcating point with the EoS minima that the trajectory eventually converges to, and the non-bifurcating regime for the polynomial maps with the “sub-EoS regime” on the left (in Fig.

Original: 6b) of the EoS minima.
Revised: 6b) of the EoS minima.

Original:  C ONNECTION TO R EAL -W ORLD M ODELS
Revised:  C ONNECTION TO R EAL -W ORLD M ODELS

Original: In this section we show how the degree-4 model analyzed above resembles the converging dynamics of over-parameterized regression models trained on real-world dataset.
Revised: In this section we show how the degree-4 model analyzed above resembles the converging dynamics of over-parameterized regression models trained on real-world dataset.

Original: We train a 5-layer ELUactivated fully connected network on a 2-class small subset of CIFAR-10 (Krizhevsky et al., 2009) with GD.
Revised: We train a 5-layer ELUactivated fully connected network on a 2-class small subset of CIFAR-10 (Krizhevsky et al., 2009) with GD.

Original: The loss converges to 0 and the sharpness converges to just slightly below 2 /η .
Revised: The loss converges to 0 and the sharpness converges to just slightly below 2 /η .

Original: We visualize the dynamics by projecting the trajectory onto the subspace spanned by the top eigenvector of minimum (oscillation direction) and the movement direction of parameters orthogonal to oscillation (see Definition 4 in Appendix A.6.1 for exact characterization).
Revised: We visualize the dynamics by projecting the trajectory onto the subspace spanned by the top eigenvector of minimum (oscillation direction) and the movement direction of parameters orthogonal to oscillation (see Definition 4 in Appendix A.6.1 for exact characterization).

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 7 (mid), after some initial bifurcation-like oscillation, the 2-step trajectory stabilizes and moves along some smooth curves toward the minimum.
Revised: 7 (mid), after some initial bifurcation-like oscillation, the 2-step trajectory stabilizes and moves along some smooth curves toward the minimum.

Original: Near the minimum (Fig. 7, right), the trajectory in fact lies mostly in this 2-dimensional subspace (see Fig. 23c in Appendix) and can be very well-captured by a parabola, which is very similar to our minimalist example.
Revised: Near the minimum (Fig. 7, right), the trajectory in fact lies mostly in this 2-dimensional subspace (see Fig. 23c in Appendix) and can be very well-captured by a parabola, which is very similar to our minimalist example.

Original: More experimental results on realworld models are available in Appendix A.
Revised: More experimental results on realworld models are available in Appendix A.

Original: [Figure]
Revised: [Figure]

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: In this paper we proposed a simple degree-4 model that captures the sharpness adaptivity and sharpness concentration phenomena that happen in gradient descent training of deep neural networks.
Revised: In this paper we proposed a simple degree-4 model that captures the sharpness adaptivity and sharpness concentration phenomena that happen in gradient descent training of deep neural networks.

Original: The simplicity of the model allowed us to perform rigorous analysis on the training dynamics for a large local region.
Revised: The simplicity of the model allowed us to perform rigorous analysis on the training dynamics for a large local region.

Original: The analysis gives new insights on why the training dynamics of the degreemodel is inherently different from the training dynamics of degree-2 models.
Revised: The analysis gives new insights on why the training dynamics of the degreemodel is inherently different from the training dynamics of degree-2 models.

Original: Finally we show that the over-paramterized deep networks trained on real data exhibits a similar parabolic converging trajectory as the scalar example.
Revised: Finally we show that the over-paramterized deep networks trained on real data exhibits a similar parabolic converging trajectory as the scalar example.

Original: We hope many of these observations can be generalized to highlight the difference between training dynamics of deeper networks and the shallower models.
Revised: We hope many of these observations can be generalized to highlight the difference between training dynamics of deeper networks and the shallower models.

Original: There are still many open problems.
Revised: There are still many open problems.

Original: Can we identify the hidden dynamics of the real world model that yields the parabolic converging trajectory?
Revised: Can we identify the hidden dynamics of the real world model that yields the parabolic converging trajectory?

Original: Can we theoretically understand the automatic coupling of small entries as discussed in Appendix A.4?
Revised: Can we theoretically understand the automatic coupling of small entries as discussed in Appendix A.4?

Original: Is there a way to understand and leverage the fractal/bifurcation behavior in Section 5 toward global dynamics analysis?
Revised: Is there a way to understand and leverage the fractal/bifurcation behavior in Section 5 toward global dynamics analysis?

Revised:  A CKNOWLEDGMENTS

Revised: This work is supported by NSF Award DMS-2031849, CCF-1845171 (CAREER), CCF- (Tripods) and a Sloan Research Fellowship.

Original: Supplementary Materials for Understanding Edge-of-Stability Training Dynamics with a Minimalist Example A A DDITIONAL E XPERIMENTS
Revised: Supplementary Materials for Understanding Edge-of-Stability Training Dynamics with a Minimalist Example A A DDITIONAL E XPERIMENTS

Original: In this section, we provide more empirical evidences supporting the main text.
Revised: In this section, we provide more empirical evidences supporting the main text.

Original: In Appendix A.1, we will first introduce our experiment setup including the structure and initialization for the neural network models as well as the generative model for the synthetic datasets.
Revised: In Appendix A.1, we will first introduce our experiment setup including the structure and initialization for the neural network models as well as the generative model for the synthetic datasets.

Original: In Appendix A.2, we will present some additional figures demonstrating the training dynamics near the EoS minima for the degree-2 and degree-4 examples discussed in Section 3 and Section 4.
Revised: In Appendix A.2, we will present some additional figures demonstrating the training dynamics near the EoS minima for the degree-2 and degree-4 examples discussed in Section 3 and Section 4.

Original: We will also discuss a degree-3 example exhibiting different behavior around different EoS minima.
Revised: We will also discuss a degree-3 example exhibiting different behavior around different EoS minima.

Original: We will explain the phenomenon using our understanding of the 2 and degree-4 models.
Revised: We will explain the phenomenon using our understanding of the 2 and degree-4 models.

Original: In Appendix A.3, we will provide additional empirical evidence that shallow neural networks usually does not converge to the exact EoS threshold.
Revised: In Appendix A.3, we will provide additional empirical evidence that shallow neural networks usually does not converge to the exact EoS threshold.

Original: In Appendix A.4, we will present some results on the training dynamics of scalar networks without the coupling initialization.
Revised: In Appendix A.4, we will present some results on the training dynamics of scalar networks without the coupling initialization.

Original: We will empirically show that the coupling of entries will arise along the training process.
Revised: We will empirically show that the coupling of entries will arise along the training process.

Original: In Appendix A.5, we will demonstrate the EoS phenomenon on the rank-1 factorization.
Revised: In Appendix A.5, we will demonstrate the EoS phenomenon on the rank-1 factorization.

Original: In Appendix A.6, we will introduce the experiment on learning real-world images (as presented in Section 6) in more detail.
Revised: In Appendix A.6, we will introduce the experiment on learning real-world images (as presented in Section 6) in more detail.

Original: We will also present additional experiment results on networks with different activations and local trajectory with perturbation.
Revised: We will also present additional experiment results on networks with different activations and local trajectory with perturbation.

Original: In Appendix A.7, we will present some experiments on the edge of stability phenomenon when the model is optimized with stochastic gradient descent.
Revised: In Appendix A.7, we will present some experiments on the edge of stability phenomenon when the model is optimized with stochastic gradient descent.

Original: A. E XPERIMENT S ETTINGS A.1. C ALCULATION OF N UMERICAL S HARPNESS
Revised: A. E XPERIMENT S ETTINGS A.1. C ALCULATION OF N UMERICAL S HARPNESS

Original: For the scalar network examples the closed-form Hessian is simple.
Revised: For the scalar network examples the closed-form Hessian is simple.

Original: We compute the exact parameter
Revised: We compute the exact parameter

Original: Hessian and use numerical packages to compute its top eigenvalue.
Revised: Hessian and use numerical packages to compute its top eigenvalue.

Original: For neural networks, we use the PyHessian package by (Yao et al., 2020), which compute the top eigenvector eigenvalue pair by inferencing the Hessian vector product and do power iteration.
Revised: For neural networks, we use the PyHessian package by (Yao et al., 2020), which compute the top eigenvector eigenvalue pair by inferencing the Hessian vector product and do power iteration.

Original: For all numerical sharpness computed for neural networks, we set tol=1e-6 and max iter=10000 .
Revised: For all numerical sharpness computed for neural networks, we set tol=1e-6 and max iter=10000 .

Original: A.1. S YNTHETIC E XPERIMENTS
Revised: A .1. S YNTHETIC E XPERIMENTS

Original: For all experiments involving neural networks on synthetic datasets (Fig.
Revised: For all experiments involving neural networks on synthetic datasets (Fig.

Original: 1a, Fig.
Revised: 1a, Fig.

Original: 1b, Fig. 15), we use fully connected networks with the same dimension for input, output, and all hidden layers.
Revised: 1b, Fig. 15), we use fully connected networks with the same dimension for input, output, and all hidden layers.

Original: The bias of all layers are fixed to 0.
Revised: The bias of all layers are fixed to 0.

Original: Formally, a L -layer width d network can be modeled by f : R d → R d such that for input vector x ∈ R d ,[Equation]
Revised: Formally, a L -layer width d network can be modeled by f : R d → R d such that for input vector x ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: where σ : R d → R d is some entry-wise activation function and W l ∈ R
Revised: where σ : R d → R d is some entry-wise activation function and W l ∈ R

Original: d × d for all l ∈ [ L ] .
Revised: d × d for all l ∈ [ L ] .

Original: For this paper we only considered σ being the ReLU activation σ ( x ) = x 1 x ≥ 0 or the identity σ ( x ) = x .
Revised: For this paper we only considered σ being the ReLU activation σ ( x ) = x 1 x ≥ 0 or the identity σ ( x ) = x .

Original: Initialization of Neural Networks
Revised: Initialization of Neural Networks

Original: We use Xavier initialization (Glorot & Bengio, 2010) with gain of 1 to initialize the all weight matrices.
Revised: We use Xavier initialization (Glorot & Bengio, 2010) with gain of 1 to initialize the all weight matrices.

Original: For shallow two-layer networks that will not enter the EoS regime if using completely random initialization, we will asymmetrically re-scale the layers after random initialization by multiplying a constant to all entries of the same layer.
Revised: For shallow two-layer networks that will not enter the EoS regime if using completely random initialization, we will asymmetrically re-scale the layers after random initialization by multiplying a constant to all entries of the same layer.

Original: When we present results for the re-scaled experiments, we will state the re-scale factor.
Revised: When we present results for the re-scaled experiments, we will state the re-scale factor.

Original: Synthetic Dataset and Loss Function
Revised: Synthetic Dataset and Loss Function

Original: For the experiments involving neural networks, we use synthetic datasets very similar to the linear network experiment in section L.3 of Cohen et al.
Revised: For the experiments involving neural networks, we use synthetic datasets very similar to the linear network experiment in section L.3 of Cohen et al.

Original: For a neural network as described above with dimension d , we consider the problem of mapping n inputs x 1 , . . .
Revised: For a neural network as described above with dimension d , we consider the problem of mapping n inputs x 1 , . . .

Original: , x n ∈ R d to n outputs y 1 , . . . , y n ∈ R d .
Revised: , x n ∈ R d to n outputs y 1 , . . . , y n ∈ R d .

Original: Let X ∈ R
Revised: Let X ∈ R

Original: d × d and Y ∈ R
Revised: d × d and Y ∈ R

Original: d × d denote the vertically stack inputs and outputs respectively (Here n = d ).
Revised: d × d denote the vertically stack inputs and outputs respectively (Here n = d ).

Original: We generate X as a whitened matrix such that XX T =
Revised: We generate X as a whitened matrix such that XX T =

Original: d I d and generate Y by Y = XA where A = diag (1 , d − 1 d , . .
Revised: d I d and generate Y by Y = XA where A = diag (1 , d − 1 d , . .

Original: , 2 d , 1 d ) .
Revised: , 2 d , 1 d ) .

Original: For all experiments with neural networks on synthetic datasets, we consider the simple squared loss
Revised: For all experiments with neural networks on synthetic datasets, we consider the simple squared loss

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which matches our analysis on scalar networks.
Revised: which matches our analysis on scalar networks.

Original: A.1. R EAL -W ORLD D ATA E XPERIMENTS
Revised: A.1. R EAL -W ORLD D ATA E XPERIMENTS

Original: Here we provide the detailed setting for the experiment results shown in Section 6 in the main text as well as Appendix A.6.
Revised: Here we provide the detailed setting for the experiment results shown in Section 6 in the main text as well as Appendix A.6.

Original: We consider a binary classification problem on a subset of CIFAR-10 image classification dataset (Krizhevsky et al., 2009).
Revised: We consider a binary classification problem on a subset of CIFAR-10 image classification dataset (Krizhevsky et al., 2009).

Original: Dataset
Revised: Dataset

Original: To study the training process in an over-parameterized setting (in which the loss can converge close to 0), we take a binary 50-sample subset from CIFAR-10 containing the first 25 samples of class(airplane) and class 1 (automobile).
Revised: To study the training process in an over-parameterized setting (in which the loss can converge close to 0), we take a binary 50-sample subset from CIFAR-10 containing the first 25 samples of class(airplane) and class 1 (automobile).

Original: Then we label samples from class 0 by -1 and samples from class 1 by +1.
Revised: Then we label samples from class 0 by -1 and samples from class 1 by +1.

Original: Here we are consider a binary classification problem since for networks with output dimension larger than 2, there is typically not a strong eigengap between the first eigenvalue and the other eigenvalues (Sagun et al., 2016; Papyan, 2018;
Revised: Here we are consider a binary classification problem since for networks with output dimension larger than 2, there is typically not a strong eigengap between the first eigenvalue and the other eigenvalues (Sagun et al., 2016; Papyan, 2018;

Original: Wu et al., 2020).
Revised: Wu et al., 2020).

Original: The dynamics with multiple eigenvalues around the stability threshold may exhibits cascading oscillation along different eigendirections (Ruiz-Garcia et al., 2021), and could be complicated to analyze.
Revised: The dynamics with multiple eigenvalues around the stability threshold may exhibits cascading oscillation along different eigendirections (Ruiz-Garcia et al., 2021), and could be complicated to analyze.

Original: Network Structure
Revised: Network Structure

Original: We conduct the experiment on fully-connected neural networks with four hidden layers of width 200.
Revised: We conduct the experiment on fully-connected neural networks with four hidden layers of width 200.

Original: We consider tanh and ELU as activations.
Revised: We consider tanh and ELU as activations.

Original: In Table 1 we provide the structure of a fully-connected
Revised: In Table 1 we provide the structure of a fully-connected

Original: ELU-activated architecture.
Revised: ELU-activated architecture.

Original: This architecture follows the experiments in Li et al.
Revised: This architecture follows the experiments in Li et al.

Original: (2022b)
Revised: (2022b)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Loss Function For the CIFAR-10 subset experiment { ( x 1 , y i ) } ni =1 where n = 50 , x i ∈ R 3072 , and y i ∈ { 1 , − 1 } , we consider the mean squared loss
Revised: Loss Function For the CIFAR-10 subset experiment { ( x 1 , y i ) } ni =1 where n = 50 , x i ∈ R 3072 , and y i ∈ { 1 , − 1 } , we consider the mean squared loss

Original: [Equation]
Revised: [Equation]

Original: (15) which matches our theoretical analysis on the scalar example
Revised: (15) which matches our theoretical analysis on the scalar example

Original: [Table]
Revised: [Table]

Original: A. A DDITIONAL E XPERIMENTS FOR S CALAR N ETWORK E XAMPLES
Revised: A. A DDITIONAL E XPERIMENTS FOR S CALAR N ETWORK E XAMPLES

Original: In this section we show some additional figures demonstrating the local training dynamics and convergence boundary for the two cases we analyzed in Section 3 and Section 4.
Revised: In this section we show some additional figures demonstrating the local training dynamics and convergence boundary for the two cases we analyzed in Section 3 and Section 4.

Original: A.2. 4-L AYER S CALAR N ETWORK
Revised: A.2. 4-L AYER S CALAR N ETWORK

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure][Figure]
Revised: [Figure] [Figure]

Original: A

Original: [Figure]
Revised: [Figure]

Original: 2-L

Original:  S CALAR N ETWORK
Revised: L AYER S CALAR N ETWORK

Original: [Figure]
Revised: [Figure]

Revised: Trajectory

Original: [Figure]
Revised: [Figure]

Revised: 100 Iterations)

Original: [Figure] 
Revised: [Figure] [Figure] [Figure] [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Revised: y

Original: Normalized Two-Step Update
Revised: Normalized Two-Step Update

Original: [Figure]
Revised: [Figure]

Original: C o n v e r g i n g
Revised: C o n v e r g i n g

Original: Sh a r pn e ss
Revised: Sh a r pn e ss

Original: A.2. 3-L AYER S CALAR N ETWORK
Revised: A.2. 3-L AYER S CALAR N ETWORK

Original: Now we look into an interesting example with different behaviors around different EoS minima.
Revised: Now we look into an interesting example with different behaviors around different EoS minima.

Original: We consider a 3-layer scalar network with objective
Revised: We consider a 3-layer scalar network with objective

Original: [Equation]
Revised: [Equation]

Original: To make the dynamics two dimensional, we consider the initialization with z = y .
Revised: To make the dynamics two dimensional, we consider the initialization with z = y .

Original: The equality of the last two entries will be preserved through training so the dynamics is two dimensional in terms of x and y .
Revised: The equality of the last two entries will be preserved through training so the dynamics is two dimensional in terms of x and y .

Original: In the positive quadrant, the global minima is √ xy = 

Revised: In the positive quadrant, the global minima is √ xy =

Original: and there are two EoS minima.
Revised: and there are two EoS minima.

Original: In Fig. 13, we plot the converging sharpness from different initializations in comparison with Fig.
Revised: In Fig. 13, we plot the converging sharpness from different initializations in comparison with Fig.

Original: 2b and Fig.
Revised: 2b and Fig.

Original: 5b in the main text.
Revised: 5b in the main text.

Original: Around the EoS minimum that the single entry x is small and the duplicated entries y are large (upper left of Fig. 13), the behavior is similar to the 2 scalar case (Fig. 5b) with no sharpness concentration.
Revised: Around the EoS minimum that the single entry x is small and the duplicated entries y are large (upper left of Fig. 13), the behavior is similar to the 2 scalar case (Fig. 5b) with no sharpness concentration.

Original: Around the EoS minima with large single entry and small duplicating entries (lower right of Fig. 13), we have a region of initialization (the red shaded area) with sharpness concentration similar to the 4 scalar case (Fig. 2b).
Revised: Around the EoS minima with large single entry and small duplicating entries (lower right of Fig. 13), we have a region of initialization (the red shaded area) with sharpness concentration similar to the 4 scalar case (Fig. 2b).

Original: [Figure]
Revised: [Figure]

Original: A heuristic explanation to this difference lies in the difference in the degree of the small entries.
Revised: A heuristic explanation to this difference lies in the difference in the degree of the small entries.

Original: Around the EoS minima that the single entry is small, the local two-step approximation is similar to Eq.
Revised: Around the EoS minima that the single entry is small, the local two-step approximation is similar to Eq.

Original: (12) and gives us elliptical two-step trajectories.
Revised: (12) and gives us elliptical two-step trajectories.

Original: Around the minima with small duplicating entries, the two-step approximation would contain the cubic term as in Eq.
Revised: Around the minima with small duplicating entries, the two-step approximation would contain the cubic term as in Eq.

Original: (7), which gives us both sharpness concentration and adaptivity.
Revised: (7), which gives us both sharpness concentration and adaptivity.

Original: Such heuristics can be further verified by visualizing the local dynamics around the minima.
Revised: Such heuristics can be further verified by visualizing the local dynamics around the minima.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 14, the local dynamics around the minima with small duplicating entries is similar to the case of the degree-4 example with convergence toward a parabolic trajectory.
Revised: 14, the local dynamics around the minima with small duplicating entries is similar to the case of the degree-4 example with convergence toward a parabolic trajectory.

Original: On the other hand, the local dynamics around the minima with only one small entry is similar to the case of the degree-2 example where parameters follow an locally elliptic trajectory centered at the EoS minima.
Revised: On the other hand, the local dynamics around the minima with only one small entry is similar to the case of the degree-2 example where parameters follow an locally elliptic trajectory centered at the EoS minima.

Original: [Figure]
Revised: [Figure]

Original: y
Revised: y

Original: Normalized Two-Step Update (EoS)
Revised: Normalized Two-Step Update (EoS)

Original: EoS Minima Global Minima
Revised: EoS Minima Global Minima

Original: C o n v e r g i n g
Revised: C o n v e r g i n g

Original: Sh a r pn e ss 0 .0 .0 .0 .0 .0 .
Revised: Sh a r pn e ss 0 .0 .0 .0 .0 .0 .

Original: 8 √ x 1 .1 .1 .1 .1 .2 .y
Revised: 8 √ x 1 .1 .1 .1 .1 .2 .y

Original: Normalized Two-Step Update (No EoS)
Revised: Normalized Two-Step Update (No EoS)

Original: EoS Minima Global Minima
Revised: EoS Minima Global Minima

Original: C o n v e r g i n g
Revised: C o n v e r g i n g

Original: Sh a r pn e ss
Revised: Sh a r pn e ss

Original: [Figure]
Revised: [Figure]

Original: A. A DDITIONAL E XPERIMENTS FOR 2-C OMPONENT S CALAR F ACTORIZATION
Revised: A. A DDITIONAL E XPERIMENTS FOR 2-C OMPONENT S CALAR F ACTORIZATION

Original: In this section we present the experiment results for 2-component scalar factorization deferred from
Revised: In this section we present the experiment results for 2-component scalar factorization deferred from

Original: Section 4.
Revised: Section 4.

Original: The dynamics as shown in Fig.
Revised: The dynamics as shown in Fig.

Original: 15 is very similar to the degree-2 example in Fig.
Revised: 15 is very similar to the degree-2 example in Fig.

Original: [Figure]
Revised: [Figure]

Original: A. E XPERIMENTS FOR G ENERAL S CALAR N ETWORKS
Revised: A. E XPERIMENTS FOR G ENERAL S CALAR N ETWORKS

Original: In this section
Revised: In this section

Original: [Figure]
Revised: [Figure]

Original: we will present some empirical observations on training dynamics of more general scalar networks related to the sharpness concentration and adaptation phenomena.
Revised: we will present some empirical observations on training dynamics of more general scalar networks related to the sharpness concentration and adaptation phenomena.

Original: A n -layer scalar network is defined to be the model parameterized by n entries x 1 , . . .
Revised: A n -layer scalar network is defined to be the model parameterized by n entries x 1 , . . .

Original: , x n ∈ R with objectiveL
Revised: , x n ∈ R with objective

Original: [Equation]
Revised: [Equation]

Original: A.4. I NITIALIZATION WITHOUT D UPLICATED E NTRIES
Revised: A.4. I NITIALIZATION WITHOUT D UPLICATED E NTRIES

Original: We first consider a variant of the degree-3 example as discussed in Appendix A.2.3.
Revised: We first consider a variant of the degree-3 example as discussed in Appendix A.2.3.

Original: In particular, we initialize the two small entries differently and record their values throughout the training trajectory
Revised: In particular, we initialize the two small entries differently and record their values throughout the training trajectory

Original: [Figure]
Revised: [Figure]

Original: As we can see in Fig.
Revised: As we can see in Fig.

Original: 16, at the very beginning of the training, the second entry x 2 converges to x 3 geometrically, then the dynamics is reduced to the case of duplicated entries, which we know that the sharpness concentration behavior would happen for sufficiently asymmetric initialization.
Revised: 16, at the very beginning of the training, the second entry x 2 converges to x 3 geometrically, then the dynamics is reduced to the case of duplicated entries, which we know that the sharpness concentration behavior would happen for sufficiently asymmetric initialization.

Original: In Fig.
Revised: In Fig.

Original: 17 we consider a 7-layer scalar network with 3 different large entries and 4 different small entries.
Revised: 17 we consider a 7-layer scalar network with 3 different large entries and 4 different small entries.

Original: We observe similar behavior as x 4 , x 5 , x 6 all converges to x 7 geometrically, and we observe concentration of sharpness with 4 duplicated small entries.
Revised: We observe similar behavior as x 4 , x 5 , x 6 all converges to x 7 geometrically, and we observe concentration of sharpness with 4 duplicated small entries.

Original: [Figure]
Revised: [Figure]

Original: To probe into the detailed training dynamics of general scalar networks
Revised: To probe into the detailed training dynamics of general scalar networks

Original: [Figure]
Revised: [Figure]

Original: we plot the pairwise dynamics of the entries as shown in Fig.
Revised: we plot the pairwise dynamics of the entries as shown in Fig.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: A.4. O N “L ARGE ” AND “S MALL ” I NITIALIZATIONS
Revised: A.4. O N “L ARGE ” AND “S MALL ” I NITIALIZATIONS

Original: In the experiments shown above, we have seen that there are mainly two classes of behaviors for the entries: the entries that were initialized to be large moves slowly with little oscillation while the entries that were initialized to be small has significant oscillation along the trajectory.
Revised: In the experiments shown above, we have seen that there are mainly two classes of behaviors for the entries: the entries that were initialized to be large moves slowly with little oscillation while the entries that were initialized to be small has significant oscillation along the trajectory.

Original: Intuitively, it is the decreasing large entry that decreases the sharpness and stabilizes the oscillating small entries and result in the final convergence close to the EoS minimum.
Revised: Intuitively, it is the decreasing large entry that decreases the sharpness and stabilizes the oscillating small entries and result in the final convergence close to the EoS minimum.

Original: A natural question to ask is whether there exists a clear boundary separating the “small” and “large” entries.
Revised: A natural question to ask is whether there exists a clear boundary separating the “small” and “large” entries.

Original: We consider a 4-layer scalar network with initialization (6 , 0 . 7 , 0 . 3 , 0 . 2) optimized with GD with step size η = 0 .
Revised: We consider a 4-layer scalar network with initialization (6 , 0 . 7 , 0 . 3 , 0 . 2) optimized with GD with step size η = 0 .

Original: In Fig.
Revised: In Fig.

Original: 20 and Fig.
Revised: 20 and Fig.

Original: 21 we visualize the training loss, sharpness, and the pairwise dynamics.
Revised: 21 we visualize the training loss, sharpness, and the pairwise dynamics.

Original: The mixed behaviors suggests a clear boundary between the “large” and “small” entries does not exists
Revised: The mixed behaviors suggests a clear boundary between the “large” and “small” entries does not exists

Original: [Figure]
Revised: [Figure]

Original: and the complexity of this problem is beyond this simple heuristics
Revised: and the complexity of this problem is beyond this simple heuristics

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: A. A DDITIONAL E XPERIMENTS FOR R ANK -1 F ACTORIZATION OF I SOTROPIC M ATRIX
Revised: A. A DDITIONAL E XPERIMENTS FOR R ANK -1 F ACTORIZATION OF I SOTROPIC M ATRIX

Original: In this section, we will demonstrate the EoS phenomenon on the rank-1 factorization of isotropic matrix in Section 3.3.
Revised: In this section, we will demonstrate the EoS phenomenon on the rank-1 factorization of isotropic matrix in Section 3.3.

Original: We first show that the loss, the sharpness and the trajectory of GD is very similar to the degreescalar network case.
Revised: We first show that the loss, the sharpness and the trajectory of GD is very similar to the degreescalar network case.

Original: For each different learning rate, the sharpness concentrates to a tiny interval close to the stability threshold 2 /η when trained with gradient descent.
Revised: For each different learning rate, the sharpness concentrates to a tiny interval close to the stability threshold 2 /η when trained with gradient descent.

Original: Also, we consider the 2D trajectory of the two vectors x , y .
Revised: Also, we consider the 2D trajectory of the two vectors x , y .

Original: We plot the the trajectory in the norm of each vector, i.e. ∥ x ∥∥ y ∥ , and get a similar figure to the scalar case.
Revised: We plot the the trajectory in the norm of each vector, i.e. ∥ x ∥∥ y ∥ , and get a similar figure to the scalar case.

Original: Actually, we can prove that the dynamics of this training objective will eventually be reduced to our degree-4 scalar network.
Revised: Actually, we can prove that the dynamics of this training objective will eventually be reduced to our degree-4 scalar network.

Original: That is because all the global minima of this optimization problem requires that x is aligned with y , i.e. x = c y .
Revised: That is because all the global minima of this optimization problem requires that x is aligned with y , i.e. x = c y .

Original: After the two vectors are aligned, the training dynamics of ∥ x ∥ ,
Revised: After the two vectors are aligned, the training dynamics of ∥ x ∥ ,

Original: ∥ y ∥ will be exactly equivalent to those of the scalar network.
Revised: ∥ y ∥ will be exactly equivalent to those of the scalar network.

Original: The following figure shows how GD enters EoS on the rank-1 factorization problem, and how fast the alignment of the two vectors is achieved.
Revised: The following figure shows how GD enters EoS on the rank-1 factorization problem, and how fast the alignment of the two vectors is achieved.

Original: Here we consider an alignment indicator ∥ x ∥ 2 ∥ y ∥ 2 − ( x ⊤ y ) 2 showing how the two vectors are aligned.
Revised: Here we consider an alignment indicator ∥ x ∥ 2 ∥ y ∥ 2 − ( x ⊤ y ) 2 showing how the two vectors are aligned.

Original: If x is parallel to y , i.e. x = c y , then the variable becomes 0.
Revised: If x is parallel to y , i.e. x = c y , then the variable becomes 0.

Original: Detailed analysis for this problem is deferred to Appendix C.
Revised: Detailed analysis for this problem is deferred to Appendix C.

Original: [Figure]
Revised: [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: x

Original: [Figure][Figure] [Figure] [Figure]
Revised: [Figure]

Original: k y k

Original: [Figure]
Revised: [Figure]

Original: GlobalMinima
Revised: GlobalMinima

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: Sha
Revised: Sh

Original: [Figure]
Revised: [Figure]

Original: pn e ss (λ )
Revised: pn e ss (

Original: [Figure]
Revised: [Figure]

Original: [Figure] 
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: A. E O S C ONVERGENCE FOR D EEP N EURAL N ETWORKS T RAINED ON R EAL D ATA
Revised: A. E O S C ONVERGENCE FOR D EEP N EURAL N ETWORKS T RAINED ON R EAL D ATA

Original: In this section we present a more comprehensive description and additional results for the experiment of learning a 2-class small subset of CIFAR-10 with 50 images in an over-parameterized regression setting.
Revised: In this section, we present a more comprehensive description and additional results for the experiment of learning a 2-class small subset of CIFAR-10 with 50 images in an over-parameterized regression setting.

Original: The details for the network structures and dataset construction are available in
Revised: The details for the network structures and dataset construction are available in

Original: Appendix A.1.3.
Revised: Appendix A.1.3.

Original: In this experiment, we train two 5-layer fully connected (fc) networks of width 200 with ELU and tanh activation using (full-batch) gradient descent on the binary dataset with mean squared loss.
Revised: In this experiment, we train two 5-layer fully connected (fc) networks of width 200 with ELU and tanh activation using (full-batch) gradient descent on the binary dataset with mean squared loss.

Original: We chose these two activation functions following the empirical experiments in Cohen et al.
Revised: We chose these two activation functions following the empirical experiments in Cohen et al.

Original: ReLU is not being used since its training dynamics as the loss converges to 0 is very unstable.
Revised: ReLU is not being used since its training dynamics as the loss converges to 0 is very unstable.

Original: We record the training loss and sharpness of the two training processes.
Revised: We record the training loss and sharpness of the two training processes.

Original: To better visualize the training trajectory, we consider the following projection mechanism.
Revised: To better visualize the training trajectory, we consider the following projection mechanism.

Original: A.6. T RAJECTORY P ROJECTION
Revised: A.6. T RAJECTORY P ROJECTION

Original: Inspired by the observation on the scalar example, we note that the dynamics toward the end of the convergence has two distinctive directions: an “oscillation direction” which is aligned with the first eigenvector of the Hessian, and an “movement direction” which the 2-step average of the model moves along and converges to the final minimum.
Revised: Inspired by the observation on the scalar example, we note that the dynamics toward the end of the convergence has two distinctive directions: an “oscillation direction” which is aligned with the first eigenvector of the Hessian, and an “movement direction” which the 2-step average of the model moves along and converges to the final minimum.

Original: In the context of our ( a, b ) -reparameterization for theoretical analysis (Definition 2), the oscillation direction corresponds to b and the movement direction corresponds to a .
Revised: In the context of our ( a, b ) -reparameterization for theoretical analysis (Definition 2), the oscillation direction corresponds to b and the movement direction corresponds to a .

Original: In a local region around the converging minima, ( a, b ) constitutes a parabolic trajectory that can be well captured by the solution of the ODE in Eq.
Revised: In a local region around the converging minima, ( a, b ) constitutes a parabolic trajectory that can be well captured by the solution of the ODE in Eq.

Original: In a high-dimensional setting, the oscillation direction is still naturally the top eigenvector at minimum, but we have to manually pick a movement direction to project onto.
Revised: In a high-dimensional setting, the oscillation direction is still naturally the top eigenvector at minimum, but we have to manually pick a movement direction to project onto.

Original: To be concrete, consider a trajectory of the parameters { θ 1 , θ 2 , . .
Revised: To be concrete, consider a trajectory of the parameters { θ 1 , θ 2 , . .

Original: , θ T − 1 , θ T } , where θ t ∈ R
Revised: , θ T − 1 , θ T } , where θ t ∈ R

Original: d is the parameter vector for the model after the t -th iteration.
Revised: d is the parameter vector for the model after the t -th iteration.

Original: We define the oscillation direction v osc as the first eigenvector of the parameter Hessian H
Revised: We define the oscillation direction v osc as the first eigenvector of the parameter Hessian H

Original: ( θ T ) and the movement direction v move (ˆ t ) as follows:
Revised: ( θ T ) and the movement direction v move (ˆ t ) as follows:

Original: Definition 3 (Movement direction) .
Revised: Definition 3 (Movement direction) .

Original: For some iteration ˆ t ∈ [ T ] , define v move (ˆ t ) as
Revised: For some iteration ˆ t ∈ [ T ] , define v move (ˆ t ) as

Original: [Equation]
Revised: [Equation]

Original: Fix some iteration ˆ t , v move (ˆ t ) captures the non-oscillatory movement of the parameters from step ˆ t to step T .
Revised: Fix some iteration ˆ t , v move (ˆ t ) captures the non-oscillatory movement of the parameters from step ˆ t to step T .

Original: We orthonormalize the basis by projecting v osc off from v move (ˆ t ) and get
Revised: We orthonormalize the basis by projecting v osc off from v move (ˆ t ) and get

Original: [Equation]
Revised: [Equation]

Revised: v

Original: [Equation]
Revised: [Equation]

Original:  Now with the orthonormal basis, we define the movement-oscillation projection of θ t to be the projection of its offset from the minima (which we approximate by the mean of the last two steps in the trajectory) onto ¯ v move (ˆ t ) and ¯ v osc .
Revised: [Equation] Now with the orthonormal basis, we define the movement-oscillation projection of θ t to be the projection of its offset from the minima (which we approximate by the mean of the last two steps in the trajectory) onto ¯ v move (ˆ t ) and ¯ v osc .

Original: Definition 4 (Movement-Oscillation Projection) .
Revised: Definition 4 (Movement-Oscillation Projection) .

Original: Fix an iteration ˆ t for determining the movement direction, the movement-oscillation projection of θ t is
Revised: Fix an iteration ˆ t for determining the movement direction, the movement-oscillation projection of θ t is

Original: [Equation]
Revised: [Equation]

Original: When doing the projection in practice (as in Fig.
Revised: When doing the projection in practice (as in Fig.

Original: 24 and Fig. 26), we fix ˆ t = 5000 , which is when the 2-step trajectory becomes relatively stable.
Revised: 24 and Fig. 26), we fix ˆ t = 5000 , which is when the 2-step trajectory becomes relatively stable.

Original: We also record the norm of the component of the offset θ t − 12 ( θ T − 1 + θ T ) that is orthogonal to the subspace spanned by ¯ v move (ˆ t ) and ¯ v osc .
Revised: We also record the norm of the component of the offset θ t − 12 ( θ T − 1 + θ T ) that is orthogonal to the subspace spanned by ¯ v move (ˆ t ) and ¯ v osc .

Original: These results are shown in Fig. 23c and Fig. 25c.
Revised: These results are shown in Fig. 23c and Fig. 25c.

Original: A.6. ELU-A CTIVATED F ULLY C ONNECTED N ETWORK
Revised: A.6. ELU-A CTIVATED F ULLY C ONNECTED N ETWORK

Original: Here we present the experiment results for training a 5-layer ELU-activated FC network on the binary subset of CIFAR-10.
Revised: Here we present the experiment results for training a 5-layer ELU-activated FC network on the binary subset of CIFAR-10.

Original: In Fig.
Revised: In Fig.

Original: 23, we show the evolution of loss and sharpness along the training process.
Revised: 23, we show the evolution of loss and sharpness along the training process.

Original: The sharpness eventually converge to just slightly below the 2 /η threshold.
Revised: The sharpness eventually converge to just slightly below the 2 /η threshold.

Original: We also observe that the dynamics toward the end of the converging process is mainly happening in the 2-dimensional subspace spanned by the oscillation and movement directions (Fig. 23c).
Revised: We also observe that the dynamics toward the end of the converging process is mainly happening in the 2-dimensional subspace spanned by the oscillation and movement directions (Fig. 23c).

Original: In Fig.
Revised: In Fig.

Original: 24 we plot the projected trajectory of the training process.
Revised: 24 we plot the projected trajectory of the training process.

Original: Toward the end of the training process, the trajectory can be very accurately characterized by a parabola and the converging sharpness is just slightly below the stability threshold.
Revised: Toward the end of the training process, the trajectory can be very accurately characterized by a parabola and the converging sharpness is just slightly below the stability threshold.

Original: This is identical to what we observe (and proved) for the scalar network case.
Revised: This is identical to what we observe (and proved) for the scalar network case.

Original: [Figure]
Revised: [Figure]

Original: (a) is identical to Fig.
Revised: ) is identical to Fig.

Original: 7 (left) in the main text.
Revised: 7 (left) in the main text.

Original: We see that the model is capable of memorizing all data as the loss decreases exponentially to 0.
Revised: We see that the model is capable of memorizing all data as the loss decreases exponentially to 0.

Original: Toward convergence, the sharpness oscillates very close to the stability threshold and eventually converges to 199.97.
Revised: Toward convergence, the sharpness oscillates very close to the stability threshold and eventually converges to 199.97.

Original: In (b) we show a section of (a) between iteration 5000 and 5030.
Revised: In (b) we show a section of (a) between iteration 5000 and 5030.

Original: We can clearly observe two distinctive features of the EoS regime: the loss decreases non-monotonically and the sharpness oscillates around 2 /η .
Revised: We can clearly observe two distinctive features of the EoS regime: the loss decreases non-monotonically and the sharpness oscillates around 2 /η .

Original: In (c) we plot the norm of the offset from minima that is orthogonal to the movement-oscillation projection.
Revised: In (c) we plot the norm of the offset from minima that is orthogonal to the movement-oscillation projection.

Original: After 3000 iterations the residual becomes very small, suggesting that dynamics is mainly happening in the 2 dimensional subspace and hence the projection captures the dynamics quite well.
Revised: After 3000 iterations the residual becomes very small, suggesting that dynamics is mainly happening in the 2 dimensional subspace and hence the projection captures the dynamics quite well.

Original: [Figure]
Revised: [Figure]

Original: A.6. T ANH -A CTIVATED F ULLY C ONNECTED N ETWORK
Revised: A.6. T ANH -A CTIVATED F ULLY C ONNECTED N ETWORK

Original: Here we show the results for the same experiment on a tanh-activated 5-layer FC network.
Revised: Here we show the results for the same experiment on a tanh-activated 5-layer FC network.

Original: The phenomena are qualitatively identical to the ELU case described above.
Revised: The phenomena are qualitatively identical to the ELU case described above.

Original: [Figure]
Revised: [Figure]

Original: Please refer to the caption of Fig.
Revised: Please refer to the caption of Fig.

Original: 23 for detailed explanation
Revised: 23 for detailed explanation

Original: [Figure]
Revised: [Figure]

Revised: i

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: Please refer to the caption of Fig.
Revised: Please refer to the caption of Fig.

Original: 24 for detailed explanation.
Revised: 24 for detailed explanation.

Original: A. E DGE OF S TABILITY AND S TOCHASTIC G RADIENT D ESCENT
Revised: A. E DGE OF S TABILITY AND S TOCHASTIC G RADIENT D ESCENT

Original: In this section, we will briefly discuss some empirical observations of EoS in stochastic gradient descent (SGD).
Revised: In this section, we will briefly discuss some empirical observations of EoS in stochastic gradient descent (SGD).

Original: In Appendix A.7.1, we will first empirically present the effects of different forms of noise on our scalar model.
Revised: In Appendix A.7.1, we will first empirically present the effects of different forms of noise on our scalar model.

Original: Then in Appendix A.7.2 we will compare it with the observations made on real world models trained with mini-batch gradient descent.
Revised: Then in Appendix A.7.2 we will compare it with the observations made on real world models trained with mini-batch gradient descent.

Original: Finally, we will discuss the limitations of our scalar model in explaining what people observe about EoS when the model is trained with SGD.
Revised: Finally, we will discuss the limitations of our scalar model in explaining what people observe about EoS when the model is trained with SGD.

Original: A.7. GD WITH N OISE ON S CALAR N ETWORK
Revised: A.7. GD WITH N OISE ON S CALAR N ETWORK

Original: We first look into the training trajectory of our degree-4 scalar network example with noise injected to the gradient descent process.
Revised: We first look into the training trajectory of our degree-4 scalar network example with noise injected to the gradient descent process.

Original: We consider label noise , which perturbs the target by a small amount per iteration, and gradient noise , which perturbs the gradient by a small amount per iteration before updating the parameter according to it.
Revised: We consider label noise , which perturbs the target by a small amount per iteration, and gradient noise , which perturbs the gradient by a small amount per iteration before updating the parameter according to it.

Original: Label Noise:
Revised: Label Noise:

Original: To simulate the existence of label noise, at each iteration we compute the gradient for the objective
Revised: To simulate the existence of label noise, at each iteration we compute the gradient for the objective

Original: [Equation]
Revised: [Equation]

Original: where δ is sampled from a zero-mean Gaussian N (0 , σ 2 ) for each iteration.
Revised: where δ is sampled from a zero-mean Gaussian N (0 , σ 2 ) for each iteration.

Original: This is equivalent to adding a perturbation of δ to the label (which is 1 in our original model).
Revised: This is equivalent to adding a perturbation of δ to the label (which is 1 in our original model).

Original: We start from the same initialization as in Fig.
Revised: We start from the same initialization as in Fig.

Original: 8 and plot the trajectory in Fig.
Revised: 8 and plot the trajectory in Fig.

Original: As shown in Fig. 27a and Fig. 27b, the trajectory first roughly follows a parabolic boundary and reaches close to the set of global minima near the EoS-minimum relatively quickly (for arounditerations).
Revised: As shown in Fig. 27a and Fig. 27b, the trajectory first roughly follows a parabolic boundary and reaches close to the set of global minima near the EoS-minimum relatively quickly (for arounditerations).

Original: This part of the trajectory resembles our analysis for the case without label noise.
Revised: This part of the trajectory resembles our analysis for the case without label noise.

Original: After the model reaches the tip of the parabola, the dynamics is mainly dominated by the label noise.
Revised: After the model reaches the tip of the parabola, the dynamics is mainly dominated by the label noise.

Original: The gradient is dominated by its noise component of δxy ( y, x ) , which is orthogonal to the manifold of global minima xy = 1 , thus the model starts oscillating around the global minima.
Revised: The gradient is dominated by its noise component of δxy ( y, x ) , which is orthogonal to the manifold of global minima xy = 1 , thus the model starts oscillating around the global minima.

Original: As shown in
Revised: As shown in

Original: Fig. 27c and Fig.
Revised: Fig. 27c and Fig.

Original: 27d, the sharpness further decreases very slowly (for 10 6 iterations) and eventually reaches the flattest global minimum at (1 , 1) with sharpness of 4.
Revised: 27d, the sharpness further decreases very slowly (for 10 6 iterations) and eventually reaches the flattest global minimum at (1 , 1) with sharpness of 4.

Original: We believe this is within the regime of the sharpness reduction flow near the manifold of minima, which is comprehensively studied by
Revised: We believe this is within the regime of the sharpness reduction flow near the manifold of minima, which is comprehensively studied by

Original: Damian et al.
Revised: Damian et al.

Original: ; Li et al.
Revised: ; Li et al.

Original: (2021; 2022b);
Revised: (2021; 2022b);

Original: Lyu et al. (2022).
Revised: Lyu et al. (2022).

Original: [Figure]
Revised: [Figure]

Original: We plot the loss, sharpness, and training trajectory of the label noise model with σ = 0 .
Revised: We plot the loss, sharpness, and training trajectory of the label noise model with σ = 0 .

Original: The label noise model first follows a trajectory similar to the original GD training trajectory and reaches near the EoS minimum as shown in (a, b) .
Revised: The label noise model first follows a trajectory similar to the original GD training trajectory and reaches near the EoS minimum as shown in (a, b) .

Original: Then it follows the sharpness reduction flow along the manifold of minima and reaches the flattest minima as shown in (c, d) .
Revised: Then it follows the sharpness reduction flow along the manifold of minima and reaches the flattest minima as shown in (c, d) .

Original: Gradient Noise:
Revised: Gradient Noise

Original: For gradient noise model, we sample a perturbation of ( δ, δ ′ ) from a 2-dimensional spherical Gaussian N (0 , σ 2 I 2 ) at each iteration and apply this perturbation to the gradient before we update the parameter.
Revised: For gradient noise model, we sample a perturbation of ( δ, δ ′ ) from a 2-dimensional spherical Gaussian N (0 , σ 2 I 2 ) at each iteration and apply this perturbation to the gradient before we update the parameter.

Original: The one-step dynamics with gradient noise ( δ, δ ′ ) is then:
Revised: The one-step dynamics with gradient noise ( δ, δ ′ ) is then:

Original: [Equation]
Revised: [Equation]

Original: With gradient noise, the initial parabolic trajectory can still be observed as shown in Fig. 28b.
Revised: With gradient noise, the initial parabolic trajectory can still be observed as shown in Fig. 28b.

Original: As the model reaches close to the manifold of global minima near the tip of the parabola, it no longer follows a monotone sharpness reduction flow (as in Fig. 27c for the label noise case) but instead randomly oscillates along the manifold of global minima between the two EoS minima.
Revised: As the model reaches close to the manifold of global minima near the tip of the parabola, it no longer follows a monotone sharpness reduction flow (as in Fig. 27c for the label noise case) but instead randomly oscillates along the manifold of global minima between the two EoS minima.

Original: We believe this is due to the component of the gradient noise parallel to the minima manifold, which dominates the sharpness reduction effect.
Revised: We believe this is due to the component of the gradient noise parallel to the minima manifold, which dominates the sharpness reduction effect.

Original: [Figure]
Revised: [Figure]

Original: We plot the loss, sharpness [Figure] and training trajectory of the gradient noise model with σ = 0 .
Revised: We plot the loss, sharpness, and training trajectory of the gradient noise model with σ = 0 .

Original: Like the label noise model, the gradient noise model first follows a trajectory similar to the original GD training trajectory and reaches near the EoS minimum as shown in (a, b) .
Revised: Like the label noise model, the gradient noise model first follows a trajectory similar to the original GD training trajectory and reaches near the EoS minimum as shown in (a, b) .

Original: After getting around the EoS-minimum, the parameter begins to randomly oscillate and traverse around the manifold of global minima between the two EoS minima as shown in (c,d) .
Revised: After getting around the EoS-minimum, the parameter begins to randomly oscillate and traverse around the manifold of global minima between the two EoS minima as shown in (c,d) .

Original: It is likely that the gradient noise finally converges to some distribution along the manifold of global minima.
Revised: It is likely that the gradient noise finally converges to some distribution along the manifold of global minima.

Original: A.7. M INIBATCH SGD FOR O VER - PARAMETERIZED M ODELS
Revised: A.7. M INIBATCH SGD FOR O VER - PARAMETERIZED M ODELS

Original: In this section we empirically investigate what happens to the converging sharpness when overparameterized network are trained with minibatch SGD.
Revised: In this section we empirically investigate what happens to the converging sharpness when overparameterized network are trained with minibatch SGD.

Original: We use the same 5-layer FC models and dataset as used in Section 6.
Revised: We use the same 5-layer FC models and dataset as used in Section 6.

Original: Other than full-batch gradient descent, we also train the models with mini-batch gradient descent with varying batchsizes and record their converging sharpness.
Revised: Other than full-batch gradient descent, we also train the models with mini-batch gradient descent with varying batchsizes and record their converging sharpness.

Original: [Figure]
Revised: [Figure]

Original: Instead of going to the flattest minima (as in the label noise case) or randomly oscillating below the EoS threshold (as in the gradient noise case) the converging minima of overparameterized deep networks trained with mini-batch SGD have highly concentrated sharpness that is correlated to the batch size.
Revised: Instead of going to the flattest minima (as in the label noise case) or randomly oscillating below the EoS threshold (as in the gradient noise case) the converging minima of overparameterized deep networks trained with mini-batch SGD have highly concentrated sharpness that is correlated to the batch size.

Original: We note that a key difference between the over-parameterized mini-batch SGD and the noisy GD experiment discussed in Appendix A.7.1 is that the loss for mini-batch SGD can converge to a fixed point with loss 0 (i.e. the model can memorize all training data) while the models with fixed additive noise will not converge to a fixed point.
Revised: We note that a key difference between the over-parameterized mini-batch SGD and the noisy GD experiment discussed in Appendix A.7.1 is that the loss for mini-batch SGD can converge to a fixed point with loss 0 (i.e. the model can memorize all training data) while the models with fixed additive noise will not converge to a fixed point.

Original: Currently the minimalist scalar model example we analyzed can only memorize one data point.
Revised: Currently the minimalist scalar model example we analyzed can only memorize one data point.

Original: We believe it is an interesting future direction to generalize the model to memorize more data and understand why mini-batch SGD converges below the EoS threshold.
Revised: We believe it is an interesting future direction to generalize the model to memorize more data and understand why mini-batch SGD converges below the EoS threshold.

Original: B T HEORETICAL A NALYSIS ON THE D EGREE -4 E XAMPLE
Revised: B T HEORETICAL A NALYSIS ON THE D EGREE -4 E XAMPLE

Original: In this section we present the complete rigorous analysis on the training dynamics of the degreeexample discussed in Section 3.
Revised: In this section we present the complete rigorous analysis on the training dynamics of the degreeexample discussed in Section 3.

Original: The section will be organized in the following way:
Revised: The section will be organized in the following way:

Original: In Appendix B.1 we will first define the problem and two reparameterizations we used for analysis, this serves as a more comprehensive version of Section 2 in the main text.
Revised: In Appendix B.1 we will first define the problem and two reparameterizations we used for analysis, this serves as a more comprehensive version of Section 2 in the main text.

Original: In Appendix B.2 we will first restate our main theorem for the degree-4 example (Theorem 3.1) along with two corollaries (Corollary 3.1, and Corollary 3.2).
Revised: In Appendix B.2 we will first restate our main theorem for the degree-4 example (Theorem 3.1) along with two corollaries (Corollary 3.1, and Corollary 3.2).

Original: These theoretical results characterized the sharpness concentration and sharpness adaptivity phenomenon of the degree-4 example.
Revised: These theoretical results characterized the sharpness concentration and sharpness adaptivity phenomenon of the degree-4 example.

Original: Then we will provide a more comprehensive proof sketch for Theorem 3.1 that is similar to the discussion in
Revised: Then we will provide a more comprehensive proof sketch for Theorem 3.1 that is similar to the discussion in

Original: Section 3.2 of the main text.
Revised: Section 3.2 of the main text.

Original: Then we will provide the lemmas for dynamics approximation (Appendix B.3), phase I convergence (Appendix B.4), and phase II convergence (Appendix B.5).
Revised: Then we will provide the lemmas for dynamics approximation (Appendix B.3), phase I convergence (Appendix B.4), and phase II convergence (Appendix B.5).

Original: Finally, in Appendix B.6 we will use the lemmas to complete the proof for the main theorem along with its corollaries.
Revised: Finally, in Appendix B.6 we will use the lemmas to complete the proof for the main theorem along with its corollaries.

Original: B. P RELIMINARIES
Revised: B. P RELIMINARIES

Original: We consider a simple objective function L ( x, y, z, w ) =
Revised: We consider a simple objective function L ( x, y, z, w ) =

Original: 12 ( xyzw − 1) 2 .
Revised: 12 ( xyzw − 1) 2 .

Original: Denote γ :=
Revised: Denote γ :=

Original: xyzw , then ∇L
Revised: xyzw , then ∇L

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Let the parameter [ x, y, z, w ] to be optimized by gradient descent with step size η , that
Revised: Let the parameter [ x, y, z, w ] to be optimized by gradient descent with step size η , that

Original: [Equation]
Revised: [Equation]

Original: To further simplify the problem, we consider the symmetric initialization of z 0 = x 0 , w 0 = y 0 .
Revised: To further simplify the problem, we consider the symmetric initialization of z 0 = x 0 , w 0 = y 0 .

Original: Note that due to symmetry of objective, the identical entries will remain identical throughout the training process, so the training dynamics reduces to two dimensional, and the global minima is simply S
Revised: Note that due to symmetry of objective, the identical entries will remain identical throughout the training process, so the training dynamics reduces to two dimensional, and the global minima is simply S

Original: = { ( x, y ) ∈ R 2 : x 2 y 2 = 1 } .
Revised: = { ( x, y ) ∈ R 2 : x 2 y 2 = 1 } .

Original: Computing the closed-form of the gradient, we know the 1-step update of x and y follows
Revised: Computing the closed-form of the gradient, we know the 1-step update of x and y follows

Original: [Equation]
Revised: [Equation]

Original: Denote γ =
Revised: Denote γ =

Original: xy , the parameter Hessian of the objective L at ( x, y, x, y ) admits eigenvalues λ
Revised: xy , the parameter Hessian of the objective L at ( x, y, x, y ) admits eigenvalues λ

Original: = x 2 (1 − γ ) , λ
Revised: = x 2 (1 − γ ) , λ

Original: 2 =
Revised: 2 =

Original: y 2 (1 − γ ) and λ 3 = 12 (cid:16)(cid:0)
Revised: y 2 (1 − γ ) and λ 3 = 12 (cid:16)(cid:0)

Original: x 2 + y 2 (cid:1) (cid:0) 3 γ 2 − 1 (cid:1) − (cid:112) ( x 2 + y 2 ) 2 (1 − 3 γ 2 ) 2 + 4 γ 

Revised: x 2 + y 2 (cid:1) (cid:0) 3 γ 2 − 1 (cid:1) − (cid:112) ( x 2 + y 2 ) 2 (1 − 3 γ 2 ) 2 + 4 γ

Original: (3 − 10 γ 2 + 7 γ 4 ) (cid:17) , λ 4 = 12 (cid:16)(cid:0) x
Revised: (3 − 10 γ 2 + 7 γ 4 ) (cid:17) , λ 4 = 12 (cid:16)(cid:0) x

Original: 2 + y 2 (cid:1) (cid:0) 3 γ 2 − 1 (cid:1) + (cid:112) ( x 2 + y 2 ) 
Revised: 2 + y 2 (cid:1) (cid:0) 3 γ 2 − 1 (cid:1) + (cid:112) ( x 2 + y 2 ) 2 (

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: When ( x, y ) converges to any minimum, γ = x 2 y 2 = 1 , so λ 1 , λ 2 , λ 3 all vanishes.
Revised: When ( x, y ) converges to any minimum, γ = x 2 y 2 = 1 , so λ 1 , λ 2 , λ 3 all vanishes.

Original: Therefore it is λ 4 that corresponds to the EoS phenomenon people observe.
Revised: Therefore it is λ 4 that corresponds to the EoS phenomenon people observe.

Original: When η < 12 , solving λ 4 = 2 /η with x 2 y 2 = 1 gives
Revised: When η < 12 , solving λ 4 = 2 /η with x 2 y 2 = 1 gives

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and their multiplicative inverses.
Revised: and their multiplicative inverses.

Original: These solutions correspond to the minima with sharpness exactly equal to the EoS threshold of 2 /η .
Revised: These solutions correspond to the minima with sharpness exactly equal to the EoS threshold of 2 /η .

Original: Since they are all symmetric with each other, without loss of generality we pick the minimum of interest as
Revised: Since they are all symmetric with each other, without loss of generality we pick the minimum of interest as

Original: [Equation]
Revised: [Equation]

Original: To better analyze the dynamics under a more natural coordinate, we consider the reparameterization that
Revised: To better analyze the dynamics under a more natural coordinate, we consider the reparameterization that

Original: For any ( x, y ) ∈ { ( x, y ) ∈ R + × R + : x > y } , define
Revised: For any ( x, y ) ∈ { ( x, y ) ∈ R + × R + : x > y } , define

Original: [Equation]
Revised: [Equation]

Original: This gives a bijective continuous mapping between { ( x, y ) ∈ R + × R + : x > y } and { ( c, d ) ∈ R + × R + } .
Revised: This gives a bijective continuous mapping between { ( x, y ) ∈ R + × R + : x > y } and { ( c, d ) ∈ R + × R + } .

Original: Intuitively, we are taking the lower half of y = 1 /x on the positive quadrant as d = 1 .
Revised: Intuitively, we are taking the lower half of y = 1 /x on the positive quadrant as d = 1 .

Original: With c, d as defined, the η -EoS minimum simplifies to (˘ c, ˘ d ) ≜
Revised: With c, d as defined, the η -EoS minimum simplifies to (˘ c, ˘ d ) ≜

Original: ( η − 2 − 4) 14 , 1) .
Revised: ( η − 2 − 4) 14 , 1) .

Original: The inverse map can be computed as
Revised: The inverse map can be computed as

Original: [Equation]
Revised: [Equation]

Original: To expand the dynamics near the η -EoS minimum, we define
Revised: To expand the dynamics near the η -EoS minimum, we define

Original: [Equation]
Revised: [Equation]

Original: to be the offset from (˘ c, ˘ d ) .
Revised: to be the offset from (˘ c, ˘ d ) .

Original: Our analysis will primarily be using the ( a, b ) -parameterization.
Revised: Our analysis will primarily be using the ( a, b ) -parameterization.

Original: To summarize, the ( c, d ) and ( a, b ) reparameterization of ( x, y ) are respectively given by
Revised: To summarize, the ( c, d ) and ( a, b ) reparameterization of ( x, y ) are respectively given by

Original: [Equation]
Revised: [Equation]

Original: Let κ ≜ √ η , under the reparameterization Eq.
Revised: Let κ ≜ √ η , under the reparameterization Eq.

Original: (24) becomes.
Revised: (24) becomes.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B. T HEORETICAL R ESULTS AND P ROOF S KETCH
Revised: B. T HEORETICAL R ESULTS AND P ROOF S KETCH

Original: Now with the reparameterization defined, we restate our convergence result on the 4 scalar objective and discuss the proof sketch.
Revised: Now with the reparameterization defined, we restate our convergence result on the 4 scalar objective and discuss the proof sketch.

Original: Theorem 3.1 (Sharpness Concentration) .
Revised: Theorem 3.1 (Sharpness Concentration) .

Original: For a large enough absolute constant K , suppose κ < 1 2000 √ 2 K − 1 , and the initialization ( a 0 , b 0 ) satisfies a 0 ∈ (12 κ 52 , 1 4 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .
Revised: For a large enough absolute constant K , suppose κ < 1 2000 √ 2 K − 1 , and the initialization ( a 0 , b 0 ) satisfies a 0 ∈ (12 κ 52 , 1 4 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .

Original: Consider the GD trajectory characterized in Eq.
Revised: Consider the GD trajectory characterized in Eq.

Original: (6) with fixed step size κfrom ( a 0 , b 0 ) , for any ϵ > 0 there exists T
Revised: (6) with fixed step size κfrom ( a 0 , b 0 ) , for any ϵ > 0 there exists T

Original: = O
Revised: = O

Original: ( K − 2 κ − 152 + log( ϵ − 1 ) + log( | b 0 | − 1 ) κ − 72 ) such that for all t > T , | b t | <
Revised: ( K − 2 κ − 152 + log( ϵ − 1 ) + log( | b 0 | − 1 ) κ − 72 ) such that for all t > T , | b t | <

Original: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: B.2. P ROOF S KETCH OF T HEOREM 3. 
Revised: B.2. P ROOF S KETCH OF T HEOREM 3.

Original: Our analysis begins with approximating the local movement using primarily Taylor expansion around the κ 2 -EoS Minimum (Appendix B.3).
Revised: Our analysis begins with approximating the local movement using primarily Taylor expansion around the κ 2 -EoS Minimum (Appendix B.3).

Original: We show that for initialization within a local region of width 2 K − 2 κ − 1 and height 2 K − 1 centered at the κ 2 -EoS minimum (condition B.1), the local two-step update of a and b can be characterized by
Revised: We show that for initialization within a local region of width 2 K − 2 κ − 1 and height 2 K − 1 centered at the κ 2 -EoS minimum (condition B.1), the local two-step update of a and b can be characterized by

Original: [Equation]
Revised: [Equation]

Original: Where R a , R b are remainders that we can effectively bound (Corollary B.1).
Revised: Where R a , R b are remainders that we can effectively bound (Corollary B.1).

Original: We note that in the region we are considering, a is always monotonically decreasing at b 2 κ 3 per 2 steps (Lemma 5).
Revised: We note that in the region we are considering, a is always monotonically decreasing at b 2 κ 3 per 2 steps (Lemma 5).

Original: With the approximation ready, we will conduct our convergence analysis with 2 phases.
Revised: With the approximation ready, we will conduct our convergence analysis with 2 phases.

Original: [Figure]
Revised: [Figure]

Original: I
Revised: I

Original: II
Revised: II

Original: III
Revised: III

Original: IV
Revised: IV

Original: [Figure]
Revised: [Figure]

Original: In Phase 1 (Appendix B.4), we consider all possible initializations ( a 0 , b 0 ) such that a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .
Revised: In Phase 1 (Appendix B.4), we consider all possible initializations ( a 0 , b 0 ) such that a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .

Original: We partition the region of initializations into three parts separated by b 2 = 2 aκ and b 2 = 14 aκ (shown as region I, II, III in Fig. 30).
Revised: We partition the region of initializations into three parts separated by b 2 = 2 aκ and b 2 = 14 aκ (shown as region I, II, III in Fig. 30).

Original: • For initializations in region I where b 2 > 2 aκ (condition B.3), we show that the cubic term b 3 in the expression of b ′′ in Eq.
Revised: • For initializations in region I where b 2 > 2 aκ (condition B.3), we show that the cubic term b 3 in the expression of b ′′ in Eq.

Original: (33) dominates the abκ term as well as the remainder, so that the two step update on | b | is monotonically decreasing with at least an additive update of −| b 3 | .
Revised: (33) dominates the abκ term as well as the remainder, so that the two step update on | b | is monotonically decreasing with at least an additive update of −| b 3 | .

Original: Combining with slow movement of a , we show that initializations in region I will quickly enter region II (Lemma 8).
Revised: Combining with slow movement of a , we show that initializations in region I will quickly enter region II (Lemma 8).

Original: • For initializations in region III where b 2 ∈ (0 , 14 aκ ) (condition B.4), we show that the abκ term will dominate the b 3 term and other remainders.
Revised: • For initializations in region III where b 2 ∈ (0 , 14 aκ ) (condition B.4), we show that the abκ term will dominate the b 3 term and other remainders.

Original: Thus the two-step update of | b | will monotonically increase with a multiplicative rate of at least (1+ aκ ) .
Revised: Thus the two-step update of | b | will monotonically increase with a multiplicative rate of at least (1+ aκ ) .

Original: Combining with slow movement of a , we show that initializations in region III will also quickly enter region II (Lemma 9).
Revised: Combining with slow movement of a , we show that initializations in region III will also quickly enter region II (Lemma 9).

Original: • For the last step in Phase 1, we show that the 2-step trajectories entering region II will stay in the region in the sense that b 2 ∈ ( 14 aκ, 2 aκ ) .
Revised: • For the last step in Phase 1, we show that the 2-step trajectories entering region II will stay in the region in the sense that b 2 ∈ ( 14 aκ, 2 aκ ) .

Original: (Lemma 11)
Revised: (Lemma 11)

Original: We also show that a will keep decreasing and enter ( 32 κ 52 , 2 κ 52 ) .
Revised: We also show that a will keep decreasing and enter ( 32 κ 52 , 2 κ 52 ) .

Original: In the diagram (Fig. 30) this corresponds to the phase of entering region IV from II.
Revised: In the diagram (Fig. 30) this corresponds to the phase of entering region IV from II.

Original: At the end of Phase I, we would have shown that all trajectories starting from the required initialization will converge to near the parabola[-b 2 = 
 aκ enter-]
Revised: At the end of Phase I, we would have shown that all trajectories starting from the required initialization will converge to near the parabola[-b 2 = 
 aκ enter-]

Original: IV from the right.

Original: In Phase 2 (Appendix B.5), we begin with initialization ( a 0 , b 0 ) such that a 0 ∈ ( 32 κ 52 , 2 κ 52 ) and b 20 ∈ ( 14 a 0 κ, 2 a 0 κ ) .
Revised: In Phase 2 (Appendix B.5), we begin with initialization ( a 0 , b 0 ) such that a 0 ∈ ( 32 κ 52 , 2 κ 52 ) and b 20 ∈ ( 14 a 0 κ, 2 a 0 κ ) .

Original: In phase 1 we have shown a rough convergence result close to the parabola b 2 = 12 aκ with the extreme point of (0 , 0) .
Revised: In phase 1 we have shown a rough convergence result close to the parabola b 2 = 12 aκ with the extreme point of (0 , 0) .

Original: In phase 2 we will change the parabola of interest to be b 2 = 12 aκ + 116 κ 4 which is characterized by the ODE approximation as discussed in Section 3.1.
Revised: In phase 2 we will change the parabola of interest to be b 2 = 12 aκ + 116 κ 4 which is characterized by the ODE approximation as discussed in Section 3.1.

Original: In particular we will focus on the residual ξ ≜ b 2 − 12 aκ − 116 κ 4 .
Revised: In particular we will focus on the residual ξ ≜ b 2 − 12 aκ − 116 κ 4 .

Original: The phase 2 convergence has the following three stages.
Revised: The phase 2 convergence has the following three stages.

Original: Throughout the analysis we fix a small constant δ = 0 .
Revised: Throughout the analysis we fix a small constant δ = 0 .

Original: • Stage 1.
Revised: • Stage 1.

Original: After the trajectory enters a ∈ ( 3 2 κ 5 2 , 2 κ 5 2 ) and b 2 ∈ ( 1 4 aκ 5 2 , 2 aκ 5 2 ) , we will show that the two-step update of ξ follows | ξ ′′ − (1 − 32 c 2 ) ξ | < δb 2 κ 4 .
Revised: After the trajectory enters a ∈ ( 3 2 κ 5 2 , 2 κ 5 2 ) and b 2 ∈ ( 1 4 aκ 5 2 , 2 aκ 5 2 ) , we will show that the two-step update of ξ follows | ξ ′′ − (1 − 32 c 2 ) ξ | < δb 2 κ 4 .

Original: (Lemma 12).
Revised: (Lemma 12).

Original: Then we show that | ξ | will further decrease to less than 18 δκ 4 before a decreases to less than κ 52 , and the trajectory will enter region V from the right (Lemma 13).
Revised: Then we show that | ξ | will further decrease to less than 18 δκ 4 before a decreases to less than κ 52 , and the trajectory will enter region V from the right (Lemma 13).

Original: • Stage 2.
Revised: • Stage 2.

Original: After the trajectory enters region V, we will show that it will remain close to the parabola
Revised: After the trajectory enters region V, we will show that it will remain close to the parabola

Original: ξ ≜ b 2 − 1 2 aκ − 1 16 κ 4 that | ξ | will remain less than 1 8 δκ 4 while a decreases
Revised: ξ ≜ b 2 − 1 2 aκ − 1 16 κ 4 that | ξ | will remain less than 1 8 δκ 4 while a decreases

Original: into the interval a ∈ ( − 18 (1 − 3 δκ 3 ) , − 110 (1 + 2 δ ) κ 3 ) and the trajectory enters region VI (Lemma 14).
Revised: into the interval a ∈ ( − 18 (1 − 3 δκ 3 ) , − 110 (1 + 2 δ ) κ 3 ) and the trajectory enters region VI (Lemma 14).

Original: • Stage 3.
Revised: • Stage 3.

Original: Finally we conclude the proof by a convergence analysis in region VI.
Revised: Finally we conclude the proof by a convergence analysis in region VI.

Original: The twostep dynamics approximation in region VI is very similar to region III, that the abκ term in the two-step update of b will dominate.
Revised: The twostep dynamics approximation in region VI is very similar to region III, that the abκ term in the two-step update of b will dominate.

Original: Since a is now negative, | b | will follow the multiplicative update | b ′′ | < (1 + aκ ) | b | .
Revised: Since a is now negative, | b | will follow the multiplicative update | b ′′ | < (1 + aκ ) | b | .

Original: We will also show that the movement of a will be small, and the final converging minima will not be far from the extreme point a = 18 κ(Lemma 15).
Revised: We will also show that the movement of a will be small, and the final converging minima will not be far from the extreme point a = 18 κ(Lemma 15).

Original: B. 1 AND 2-S TEP D YNAMICS OF a AND b
Revised: B. 1 AND 2-S TEP D YNAMICS OF a AND b

Original: Now we begin our rigorous analysis on the dynamics of ( a t , b t ) .
Revised: Now we begin our rigorous analysis on the dynamics of ( a t , b t ) .

Original: For simplicity of notations, when analyzing the 1-step and 2-step dynamics of ( a t , b t ) , we use a, a ′ , a ′′ to denote a t , a t +1 , a t +2 and b, b ′ , b ′′ to denote b t , b t +1 , b t +2 .
Revised: For simplicity of notations, when analyzing the 1-step and 2-step dynamics of ( a t , b t ) , we use a, a ′ , a ′′ to denote a t , a t +1 , a t +2 and b, b ′ , b ′′ to denote b t , b t +1 , b t +2 .

Original: For simplicity of calculation, we consider the change of variable κ = √ η .
Revised: For simplicity of calculation, we consider the change of variable κ = √ η .

Original: In the following analysis, use operator O ( · ) to only hides absolute constants that are independent of ϵ, κ, a, b and no asymptotic limits are taken.
Revised: In the following analysis, use operator O ( · ) to only hides absolute constants that are independent of ϵ, κ, a, b and no asymptotic limits are taken.

Original: Concretely, for monomial x and polynomial y of some variables, we denote y
Revised: Concretely, for monomial x and polynomial y of some variables, we denote y

Original: = O ( x ) if there exists some absolute constant K independent of the variables such that for any parameterization of the variables, | y | < K | x | .
Revised: = O ( x ) if there exists some absolute constant K independent of the variables such that for any parameterization of the variables, | y | < K | x | .

Original: Note that this is stronger than the usual big-
Revised: Note that this is stronger than the usual big-

Original: O notations.
Revised: O notations.

Original: Throughout the analysis we will use K to represent the absolute constant that uniformly upper bounds all the absolute constants of the O ( · ) terms.
Revised: Throughout the analysis we will use K to represent the absolute constant that uniformly upper bounds all the absolute constants of the O ( · ) terms.

Original: This is well defined as we will only be considering finite number of O ( · ) terms.
Revised: This is well defined as we will only be considering finite number of O ( · ) terms.

Original: B.3. O NE S TEP D YNAMICS A PPROXIMATION OF a AND b
Revised: B.3. O NE S TEP D YNAMICS A PPROXIMATION OF a AND b

Original: Lemma 1.
Revised: Lemma 1.

Original: Fix any positive ϵ that ϵ < 0 .
Revised: Fix any positive ϵ that ϵ < 0 .

Original: 5 , for any κ ∈ (0 , ϵ 14 ) , for all ( a, b ) such that | a | < ϵκ −and | b | < min { 1 , 15 ϵκ − 2 } , we have
Revised: 5 , for any κ ∈ (0 , ϵ 14 ) , for all ( a, b ) such that | a | < ϵκ −and | b | < min { 1 , 15 ϵκ − 2 } , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: Recall from Eq.
Revised: Recall from Eq.

Original: (32) that
Revised: (32) that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since (cid:0) κ − 4 − 4 (cid:1) 14 will approach infinity as κ goes to 0, we instead analyze
Revised: Since (cid:0) κ − 4 − 4 (cid:1) 14 will approach infinity as κ goes to 0, we instead analyze

Original: [Equation]
Revised: [Equation]

Original: [Equation] Note that (cid:0) 1 − b 2 (1 + b ) 2 (2 + b ) 2 κ 4 (cid:1) 12 will be close to 1 − 2 b 2 (1 + b ) 2 (2 + b ) 2 κ 4 for not so large b and 1 − 4 κ 4 will be close to 1 for small κ , we will leverage these two properties to approximate a ′ .
Revised: Note that (cid:0) 1 − b 2 (1 + b ) 2 (2 + b ) 2 κ 4 (cid:1) 12 will be close to 1 − 2 b 2 (1 + b ) 2 (2 + b ) 2 κ 4 for not so large b and 1 − 4 κ 4 will be close to 1 for small κ , we will leverage these two properties to approximate a ′ .

Original: First observe that for any x ∈ (0 , 8 ϵ/ (1 + 2 ϵ ) 2 ) we have 1 − x 2 − ϵx < √ 1 − x < 1 − x 2 .
Revised: First observe that for any x ∈ (0 , 8 ϵ/ (1 + 2 ϵ ) 2 ) we have 1 − x 2 − ϵx < √ 1 − x < 1 − x 2 .

Original: Since ϵ < 0 .
Revised: Since ϵ < 0 .

Original: 5 , (1 + 2 ϵ ) 2 < 4 , so it is sufficient to let x < 2 ϵ for the inequality to hold.
Revised: 5 , (1 + 2 ϵ ) 2 < 4 , so it is sufficient to let x < 2 ϵ for the inequality to hold.

Original: Now we substitute x by b 2 (1 + b ) 2 (2 + b ) 2 κ 4 .
Revised: Now we substitute x by b 2 (1 + b ) 2 (2 + b ) 2 κ 4 .

Original: Since | b | < 1 , (1 + b ) 2 (2 + b ) 2 < 36 .
Revised: Since | b | < 1 , (1 + b ) 2 (2 + b ) 2 < 36 .

Original: Thus when | b | < 15 ϵκ − 2 we have b 2 (1 + b ) 2 (2 + b ) 2 κ 4 < 36 b 2 κ 4 < 36( 15 ϵκ − 2 ) 2 κ 4 < 2 ϵ , and therefore
Revised: Thus when | b | < 15 ϵκ − 2 we have b 2 (1 + b ) 2 (2 + [Equation] ) 2 κ 4 < 36 b 2 κ 4 < 36( 15 ϵκ − 2 ) 2 κ 4 < 2 ϵ , and therefore

Original: [Equation]
Revised: [Equation]

Original: Since 1 − 4 κ 4 > 0 , by Bernoulli inequality we have (1 − κ 4 ) 4 ≥ 1 − 4 κ 4 , combining with the requirement of κ < ϵ 14 , we have (1 − 4 κ 4 ) 14 ≥ 1 − κ 4 > 1 − ϵ .
Revised: Since 1 − 4 κ 4 > 0 , by Bernoulli inequality we have (1 − κ 4 ) 4 ≥ 1 − 4 κ 4 , combining with the requirement of κ < ϵ 14 , we have (1 − 4 κ 4 ) 14 ≥ 1 − κ 4 > 1 − ϵ .

Original: Meanwhile since we required | d | < ϵκ − 1 , | κd | < ϵ , so
Revised: Meanwhile since we required | d | < ϵκ − 1 , | κd | < ϵ , so

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.

Original: (37) and Eq.

Original: (38) we have

Original: [Equation]
Revised: [Equation]

Original: Dividing both sides by κ completes the proof
Revised: Dividing both sides by κ completes the proof

Original: [Equation]
Revised: [Equation]

Original: Lemma 2.
Revised: Lemma 2.

Original: For any κ ∈ (0 , 0 . 1) , for all ( a, b ) such that | a | < κ − 1 and | b | < 1 ,
Revised: For any κ ∈ (0 , 0 . 1) , for all ( a, b ) such that | a | < κ − 1 and | b | < 1 ,

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 2.
Revised: Proof of Lemma 2.

Original: For simplicity of notation
Revised: For simplicity of notation

Original: [Equation]
Revised: [Equation]

Original: let 
Revised: let δ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Plugging Eq.
Revised: Plugging Eq.

Original: (41) into Eq.
Revised: (41) into Eq.

Original: (32), since | b | < 

Revised: (32), since | b | <

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Since κ < 0 .
Revised: Since κ < 0 .

Original: 1 , | a | < κ − 1 and | b | < 1 , by Lemma 18 we have
Revised: 1 , | a | < κ − 1 and | b | < 1 , by Lemma 18 we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Combining the conditions required by Lemma 1 and Lemma 2 we have
Revised: Combining the conditions required by Lemma 1 and Lemma 2 we have

Original: Condition B.1 (One-step Dynamics Approximation Condition) .
Revised: Condition B.1 (One-step Dynamics Approximation Condition) .

Original: [Equation]
Revised: [Equation]

Original: When condition B.1 is satisfied
Revised: When condition B.1 is satisfied

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: B.3. T WO S TEP D YNAMICS A PPROXIMATION OF a AND b
Revised: B.3. T WO S TEP D YNAMICS A PPROXIMATION OF a AND b

Original: Now we approximate the 2-step dynamics for ( a, b ) .
Revised: Now we approximate the 2-step dynamics for ( a, b ) .

Original: Lemma 3.
Revised: Lemma 3.

Original: Fix some positive constant ϵ < 0 .
Revised: Fix some positive constant ϵ < 0 .

Original: 1 , with κ, a, b satisfying condition B.1,
Revised: 1 , with κ, a, b satisfying condition B.1,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Combining the one step dynamics characterized in Lemma 1 and Lemma 2 we have
Revised: Combining the one step dynamics characterized in Lemma 1 and Lemma 2 we have

Original: [Equation]
Revised: [Equation]

Original: We will analyze these terms one by one.
Revised: We will analyze these terms one by one

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that b ′
Revised: Note that b ′

Original: =
Revised: =

Original: O ( b ) + O ( abκ ) and a ′ =
Revised: O ( b ) + O ( abκ ) and a ′ =

Original: a + O ( b 2 κ 3 )
Revised: a + O ( b 2 κ 3 )

Original: , so
Revised: , so

Original: [Equation]
Revised: [Equation]

Original: Hence
Revised: Hence

Original: O
Revised: O

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining above we have
Revised: Combining above we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 4.
Revised: Lemma 4.

Original: Fix some positive constant ϵ <
Revised: Fix some positive constant ϵ <

Original: [Equation]
Revised: [Equation]

Original: 5 , with κ, a, b satisfying condition B
Revised: 5 , with κ, a, b satisfying condition B

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: Combining Lemma 1 and Lemma 2 we have
Revised: Combining Lemma 1 and Lemma 2 we have

Original: [Equation]
Revised: [Equation]

Original: From Eq.
Revised: From Eq.

Original: (51) in the proof of Lemma 3 we have
Revised: (51) in the proof of Lemma 3 we have

Original: [Equation]
Revised: [Equation]

Original: Hence the last three terms of Eq.
Revised: Hence the last three terms of Eq.

Original: (57) can be calculated that
Revised: (57) can be calculated that

Original: [Equation]
Revised: [Equation]

Original: Combining above we have that
Revised: Combining above we have that

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: Combining Lemma 4 and Lemma 3, the 2-step dynamics approximation can be summarized as:
Revised: Combining Lemma 4 and Lemma 3, the 2-step dynamics approximation can be summarized as:

Original: Corollary B.1.
Revised: Corollary B.1.

Original: For any positive constant ϵ < 0 .
Revised: For any positive constant ϵ < 0 .

Original: 5 , with κ, a, b satisfying condition B.1 we haveb
Revised: 5 , with κ, a, b satisfying condition B.1 we have

Original: [Equation]
Revised: [Equation]

Original: P HASE I: C ONVERGENCE TO N EAR P ARABOLA
Revised: P HASE I: C ONVERGENCE TO N EAR P ARABOLA

Original: Now we show that under the ( a, b ) parameterization, any initialization ( a 0 , b 0 ) such that a 0 ∈ (3 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) will converge near the parabola very fast.
Revised: Now we show that under the ( a, b ) parameterization, any initialization ( a 0 , b 0 ) such that a 0 ∈ (3 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) will converge near the parabola very fast.

Original: As mentioned above, there are mainly three regimes of interest.
Revised: As mentioned above, there are mainly three regimes of interest.

Original: We will first determine the region in which the two step update b ′′ − b is solely dominated by − b 3 or abκ .
Revised: We will first determine the region in which the two step update b ′′ − b is solely dominated by − b 3 or abκ .

Original: To formally analyze the different dynamics and characterizing the regimes for them, in this section we use K to denote the uniform upper bound over the absolute constants hidden by the O ( · ) operator in the 2-step dynamics approximation characterized by Corollary B.1.
Revised: To formally analyze the different dynamics and characterizing the regimes for them, in this section we use K to denote the uniform upper bound over the absolute constants hidden by the O ( · ) operator in the 2-step dynamics approximation characterized by Corollary B.1.

Original: Since there is only finite terms with O -notation, such constant K is well defined and independent of ϵ, κ, a, b .
Revised: Since there is only finite terms with O -notation, such constant K is well defined and independent of ϵ, κ, a, b .

Original: Without loss of generality we assume K > 512 .
Revised: Without loss of generality we assume K > 512 .

Original: Rewriting Corollary B.1 with the uniform upper bound K , we have the following corollary:
Revised: Rewriting Corollary B.1 with the uniform upper bound K , we have the following corollary:

Original: Corollary B.2.
Revised: Corollary B.2.

Original: (2-step dynamics approximation of a and b ) There exists some absolute constants
Revised: (2-step dynamics approximation of a and b ) There exists some absolute constants

Original: K such that for any constant ϵ < 0 . 5 , for all κ, a, b satisfying condition B.1, the 2-step update of ( a, b ) can be characterized by
Revised: K such that for any constant ϵ < 0 . 5 , for all κ, a, b satisfying condition B.1, the 2-step update of ( a, b ) can be characterized by

Original: [Equation]
Revised: [Equation]

Original: where the remainder R a and R b have upper bound:
Revised: where the remainder R a and R b have upper bound:

Original: [Equation]
Revised: [Equation]

Original: Now we establish the conditions to characterize the work zones in order for the analysis to be more tractable in different regimes.
Revised: Now we establish the conditions to characterize the work zones in order for the analysis to be more tractable in different regimes

Revised: [Equation]

Original: B.4. W HEN a ′′ − a IS CLOSE TO − 4 b 2 κ
Revised: B.4. W HEN a ′′ − a IS CLOSE TO − 4 b 2 κ

Original: Here we formalize the observation that when b and κ are not large, the two-step movement of a is monotone and always close to 4 b 2 κ 3 .
Revised: Here we formalize the observation that when b and κ are not large, the two-step movement of a is monotone and always close to 4 b 2 κ 3 .

Original: Concretely we have the following lemma:
Revised: Concretely we have the following lemma:

Original: Lemma 5.
Revised: Lemma 5.

Original: Fix ϵ
Revised: Fix ϵ

Original: = K − 1 , for all κ, a, b satisfying condition B.1 as well as the extra condition of | κ | < K − 1 and | b | < K − 1 , we have | a ′′ − ( a − 4 b 2 κ 3 ) | < 3 b 2 κ 3 .
Revised: = K − 1 , for all κ, a, b satisfying condition B.1 as well as the extra condition of | κ | < K − 1 and | b | < K − 1 , we have | a ′′ − ( a − 4 b 2 κ 3 ) | < 3 b 2 κ 3 .

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: Since we assume K > 512 , we have ϵ
Revised: Since we assume K > 512 , we have ϵ

Original: =
Revised: =

Original: K − 1 < 0 .
Revised: K − 1 < 0 .

Original: 5 as required by Corollary B.2.
Revised: 5 as required by Corollary B.2.

Original: Since κ, a, b satisfies condition B.1, by Corollary B.2 we have a ′′ =
Revised: Since κ, a, b satisfies condition B.1, by Corollary B.2 we have a ′′ =

Original: a − 4 b 2 κ 3 + R a where
Revised: a − 4 b 2 κ 3 + R a where

Original: [Equation]
Revised: [Equation]

Original: Thus to prove the lemma we only need to bound every term on RHS of Eq.
Revised: Thus to prove the lemma we only need to bound every term on RHS of Eq.

Original: (66) by b 2 κ 3 .
Revised: (66) by b 2 κ 3 .

Original: (i)
Revised: (i)

Original: Sincewe
Revised: Since

Original:  fixed ϵ=
Revised: we fixed ϵ

Original:  K − 1 , K (cid:12)(cid:12) ϵb 2 κ 3 (cid:12)(cid:12) =
Revised: = K − 1 , K (cid:12)(cid:12) ϵb 2 κ 3 (cid:12)(cid:12) =

Original: b 2 κ 3 .
Revised: b 2 κ 3 .

Original: (ii)
Revised: (ii)

Original: Since | b | < K − 1 , K (cid:12)(cid:12)
Revised: Since | b | < K − 1 , K (cid:12)(cid:12)

Original: b
Revised: b

Original: 3 κ 3 (cid:12)(cid:12) <
Revised: 3 κ 3 (cid:12)(cid:12) <

Original: K (cid:12)(cid:12) K − 1 b 2 κ 3 (cid:12)(cid:12) <
Revised: K (cid:12)(cid:12) K − 1 b 2 κ 3 (cid:12)(cid:12) <

Original: b 2 κ 3 .
Revised: b 2 κ 3 .

Original: (iii)
Revised: (iii)

Original: Since κ < K − 1 , K (cid:12) (cid:12) b 2 κ 4 (cid:12) (cid:12) <
Revised: Since κ < K − 1 , K (cid:12) (cid:12) b 2 κ 4 (cid:12) (cid:12) <

Original: K (cid:12) (cid:12)
Revised: K (cid:12) (cid:12)

Original: [Equation]
Revised: [Equation]

Original: Therefore | R a | < 3 b 2 κ 3 which completes the proof.
Revised: Therefore | R a | < 3 b 2 κ 3 which completes the proof.

Original: Note that when we fix ϵ = K − 1 , condition B.1 becomes
Revised: Note that when we fix ϵ = K − 1 , condition B.1 becomes

Original: [Equation]
Revised: [Equation]

Original: Combining with the additional condition of Lemma 5, we can summarize the condition for Lemmato hold as
Revised: Combining with the additional condition of Lemma 5, we can summarize the condition for Lemmato hold as

Original: [Equation]
Revised: [Equation]

Original: With K > 512 , K − 1 < 0 .
Revised: With K > 512 , K − 1 < 0 .

Original: 1 , so (1) can be reduced to κ < K − 1 .
Revised: 1 , so (1) can be reduced to κ < K − 1 .

Original: Since κ < K − 1 , 15 κ − 2 > 15 K 2 , so 15 K − 1 κ − 2 > 15 K > 1 > K − 1 , and thus (3) can be reduced to | b | < K − 1 .
Revised: Since κ < K − 1 , 15 κ − 2 > 15 K 2 , so 15 K − 1 κ − 2 > 15 K > 1 > K − 1 , and thus (3) can be reduced to | b | < K − 1 .

Original: In conclusion, the following condition is sufficient for | a ′′ − ( a − 4 b 2 κ 3 ) | < 3 b 2 κ 3 .
Revised: In conclusion, the following condition is sufficient for | a ′′ − ( a − 4 b 2 κ 3 ) | < 3 b 2 κ 3 .

Original: Condition B.2 (Condition for (4 ± 3) b 2 κ 3 movement of a ) .
Revised: Condition B.2 (Condition for (4 ± 3) b 2 κ 3 movement of a ) .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: B.4. W HEN − b 3 D OMINATES THE D YNAMICS OF b
Revised: B.4. W HEN − b 3 D OMINATES THE D YNAMICS OF b

Original: Lemma 6.
Revised: Lemma 6.

Original: For all κ, a, b satisfying
Revised: For all κ, a, b satisfying

Original: [Equation]
Revised: [Equation]

Original: We have | b ′′ − ( b − 16 b 3 ) | ≤ 14 | b 3 | .
Revised: We have | b ′′ − ( b − 16 b 3 ) | ≤ 14 | b 3 | .

Original: Proof of Lemma 6.
Revised: Proof of Lemma 6.

Original: First it is straightforward to check that the condition on κ, a, b is stronger than condition B.1 if we set ϵ = 0 .
Revised: First it is straightforward to check that the condition on κ, a, b is stronger than condition B.1 if we set ϵ = 0 .

Original: 1 , thus by Corollary B.2 we have b ′′ =
Revised: 1 , thus by Corollary B.2 we have b ′′ =

Original: b − 16 b 3 + 8 abκ + R b where
Revised: b − 16 b 3 + 8 abκ + R b where

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Thus to prove the claim, it is sufficient to bound | 8 abκ | by 8 | b 3 | and every term on RHS of Eq.
Revised: Thus to prove the claim, it is sufficient to bound | 8 abκ | by 8 | b 3 | and every term on RHS of Eq.

Original: (71) by | b 3 | .
Revised: (71) by | b 3 | .

Original: We will now bound them term by term.
Revised: We will now bound them term by term.

Original: (i) Since (cid:112) | a | κ < | b | , | aκ | < b 2 , so 8 | abκ | = 8 | b || aκ | < 8 | b 3 | .
Revised: (i) Since (cid:112) | a | κ < | b | , | aκ | < b 2 , so 8 | abκ | = 8 | b || aκ | < 8 | b 3 | .

Original: (ii) Since | b | < K − 1 , K | b 4 | < K | K − 1 b 3 | = | b 3 | .
Revised: (ii) Since | b | < K − 1 , K | b 4 | < K | K − 1 b 3 | = | b 3 | .

Original: (iii) Since | a | < K − 2 κ − 1 , multiplying K 2 | aκ 2 | on both side gives K 2 a 2 κ 2 < | a | κ .
Revised: (iii) Since | a | < K − 2 κ − 1 , multiplying K 2 | aκ 2 | on both side gives K 2 a 2 κ 2 < | a | κ .

Original: it follows by taking square root that K | a | κ < (cid:112) | a | κ .
Revised: it follows by taking square root that K | a | κ < (cid:112) | a | κ .

Original: Since | b | > (cid:112) | a | κ , we have K | a | κ < | b | .
Revised: Since | b | > (cid:112) | a | κ , we have K | a | κ < | b | .

Original: Multiply b 2 on both side gives K | ab 2 κ | < | b 3 | .
Revised: Multiply b 2 on both side gives K | ab 2 κ | < | b 3 | .

Original: (iv) Since | a | < K − 2 κ − 1 < K − 1 κ − 1 , multiply K | a | κ on both side gives Ka 2 κ 2 < | a | κ and hence √ K | a | κ < (cid:112) | a | κ < | b | .
Revised: (iv) Since | a | < K − 2 κ − 1 < K − 1 κ − 1 , multiply K | a | κ on both side gives Ka 2 κ 2 < | a | κ and hence √ K | a | κ < (cid:112) | a | κ < | b | .

Original: Squaring both sides and multiply by b gives K | a 2 bκ 2 | < | b 3 | .
Revised: Squaring both sides and multiply by b gives K | a 2 bκ 2 | < | b 3 | .

Original: (v) Since | a | > κ 3 and | b | > (cid:112) | a | κ , we have | b | > κ 2 > Kκ 4 .
Revised: (v) Since | a | > κ 3 and | b | > (cid:112) | a | κ , we have | b | > κ 2 > Kκ 4 .

Original: Multiplying b 2 on both sides, we have K | b 2 κ 4 | < | b 3 | .
Revised: Multiplying b 2 on both sides, we have K | b 2 κ 4 | < | b 3 | .

Original: (vi) In (v) we observe that | b | > κ 2 , so | b | > √ Kκ 52 and hence b 2 >
Revised: (vi) In (v) we observe that | b | > κ 2 , so | b | > √ Kκ 52 and hence b 2 >

Original: Kκ 5 .
Revised: Kκ 5 .

Original: Multiplying | b | on both side gives K | bκ 3 | < | b 3 | .
Revised: Multiplying | b | on both side gives K | bκ 3 | < | b 3 | .

Original: (vii) Since we fixed ϵ = 0 .
Revised: (vii) Since we fixed ϵ = 0 .

Original: while κ < K − 1 < 0 .
Revised: while κ < K − 1 < 0 .

Original: 1 , we have ϵκ < 1 .
Revised: 1 , we have ϵκ < 1 .

Original: Since | b | > κ 2 , we have | b | > κ 3 ϵ > Kϵκ 3 .
Revised: Since | b | > κ 2 , we have | b | > κ 3 ϵ > Kϵκ 3 .

Original: Multiply b 2 on both side gives K | ϵc 2 κ 3 | < | c 3 | .
Revised: Multiply b 2 on both side gives K | ϵc 2 κ 3 | < | c 3 | .

Original: Therefore we have | 8 abκ | + | R b | < 14 | c 3 | , which completes the proof.
Revised: Therefore we have | 8 abκ | + | R b | < 14 | c 3 | , which completes the proof.

Original: We restate the sufficient condition for | b ′′ − ( b − 16 b 3 ) | ≤ 14 | b 3 | as follow
Revised: We restate the sufficient condition for | b ′′ − ( b − 16 b 3 ) | ≤ 14 | b 3 | as follow

Original: Condition B.3 (Condition for − b 3 Dominated b Movement) .
Revised: Condition B.3 (Condition for − b 3 Dominated b Movement) .

Original: [Equation]
Revised: [Equation]

Original: B.4. W HEN abκ D OMINATES THE b
Revised: B.4. W HEN abκ D OMINATES THE b

Original: [Equation]
Revised: [Equation]

Original: − b
Revised: − b

Original: Lemma 7.
Revised: Lemma 7.

Original: For any κ, a, b satisfying
Revised: For any κ, a, b satisfying

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: We have | b ′′ − ( b + 8 abκ ) | ≤ 7 | abκ | .
Revised: We have | b ′′ − ( b + 8 abκ ) | ≤ 7 | abκ | .

Original: Proof of Lemma 7.
Revised: Proof of Lemma 7.

Original: First fix ϵ = 0 .
Revised: First fix ϵ = 0 .

Original: It is straightforward to check that for all κ, a, b satisfying the given condition, condition B.1 is also satisfied, so by Corollary B.2 we have b ′′ =
Revised: It is straightforward to check that for all κ, a, b satisfying the given condition, condition B.1 is also satisfied, so by Corollary B.2 we have b ′′ =

Original: b − 16 b 3 +8 abκ +
Revised: b − 16 b 3 +8 abκ +

Original: R b where
Revised: R b where

Original: [Equation]
Revised: [Equation]

Original: Thus to prove the claim, it is sufficient to bound 16 | b 3 | by 2 | abκ | , two terms in RHS of Eq.
Revised: Thus to prove the claim, it is sufficient to bound 16 | b 3 | by 2 | abκ | , two terms in RHS of Eq.

Original: (74) by 12 | b 3 |
Revised: (74) by 12 | b 3 |

Original: and the remaining 4 terms by | b 3 | .
Revised: and the remaining 4 terms by | b 3 | .

Original: We will now bound them term by term.
Revised: We will now bound them term by term.

Revised: [Equation]

Original: (i)
Revised: i)

Original: Since | b | < 12 √ 2 (cid:112) | a | κ , b 2 < 18 | a | κ .
Revised: Since | b | < 12 √ 2 (cid:112) | a | κ , b 2 < 18 | a | κ .

Original: Multiply 16 | b | on both side gives 16 | b 3 | < 2 | abκ | .
Revised: Multiply 16 | b | on both side gives 16 | b 3 | < 2 | abκ | .

Original: (ii) Since | a | < K − 2 κ , we have | a | κ < K − 2 .
Revised: (ii) Since | a | < K − 2 κ , we have | a | κ < K − 2 .

Original: Multiplying a 2 κ 2 on both side gives | a 3 | κ 3 < K − 2 a 2 κ 3 .
Revised: Multiplying a 2 κ 2 on both side gives | a 3 | κ 3 < K − 2 a 2 κ 3 .

Original: If we take the 6 -th root, we have (cid:112) | a | κ < ( K − 1 | a | κ ) 13 .
Revised: If we take the 6 -th root, we have (cid:112) | a | κ < ( K − 1 | a | κ ) 13 .

Original: Since | b | < (cid:112) | a | κ , we have | b | < ( K − 1 | a | κ ) 13 and hence | b 3 | < K − 1 | a | κ .
Revised: Since | b | < (cid:112) | a | κ , we have | b | < ( K − 1 | a | κ ) 13 and hence | b 3 | < K − 1 | a | κ .

Original: Multiply K | b | on both side gives K | b 4 | < | abκ | .
Revised: Multiply K | b | on both side gives K | b 4 | < | abκ | .

Original: (iii)
Revised: (iii)

Original: Since | b | < K − 1 , multiply both side by K | abκ | gives K | ab 2 κ | < | abκ | .
Revised: Since | b | < K − 1 , multiply both side by K | abκ | gives K | ab 2 κ | < | abκ | .

Original: (iv) Since | a | < K − 2 κ − 1 < 12 K − 1 κ − 1 (as we assumed K > 512 ), multiply both side by | ab | κgives K | a 2 bκ 2 | < 1 2 | abκ | .
Revised: (iv) Since | a | < K − 2 κ − 1 < 12 K − 1 κ − 1 (as we assumed K > 512 ), multiply both side by | ab | κgives K | a 2 bκ 2 | < 1 2 | abκ | .

Original: (v) Since | a | > κ 3 and κ < K − 1 , we have | a | > κ 3 > K − 2 κ 5 > K 2 κ 7 .
Revised: (v) Since | a | > κ 3 and κ < K − 1 , we have | a | > κ 3 > K − 2 κ 5 > K 2 κ 7 .

Original: Multiplying K − 2 | a | κ −on both side gives K − 2 a 2 κ − 6 > | a | κ .
Revised: Multiplying K − 2 | a | κ −on both side gives K − 2 a 2 κ − 6 > | a | κ .

Original: It follows by taking the square root that (cid:112) | a | κ < K − 1 | a | κ − 3 .
Revised: It follows by taking the square root that (cid:112) | a | κ < K − 1 | a | κ − 3 .

Original: Since we have | b | < 12 √ 2 (cid:112) | a | κ < 12 (cid:112) | a | κ , combining with above gives | b | < 12 K − 1 | a | κ − 3 .
Revised: Since we have | b | < 12 √ 2 (cid:112) | a | κ < 12 (cid:112) | a | κ , combining with above gives | b | < 12 K − 1 | a | κ − 3 .

Original: Multiply K | b | κ 4 on both side gives K | b 2 κ 4 | < 12 | abκ | .
Revised: Multiply K | b | κ 4 on both side gives K | b 2 κ 4 | < 12 | abκ | .

Original: (vi) Since | a | > κ 3 , we have | a | > Kκ 4 .
Revised: (vi) Since | a | > κ 3 , we have | a | > Kκ 4 .

Original: Multiplying | b | κ on both side gives K | bκ 5 | < | abκ | .
Revised: Multiplying | b | κ on both side gives K | bκ 5 | < | abκ | .

Original: (vii) Since | a | > κ 3 and ϵ = 0 .
Revised: (vii) Since | a | > κ 3 and ϵ = 0 .

Original: 5 , we have | a | > K 2 ϵ 2 κ 5 .
Revised: 5 , we have | a | > K 2 ϵ 2 κ 5 .

Original: Taking the multiplicative inverse and multiply a 2 κ on both side gives | a | κ < K − 2 ϵ − 2 κ − 6 a 2 .
Revised: Taking the multiplicative inverse and multiply a 2 κ on both side gives | a | κ < K − 2 ϵ − 2 κ − 6 a 2 .

Original: It follows by taking the square root that (cid:112) | a | κ < K − 1 ϵ − 1 κ − 3 | a | .
Revised: It follows by taking the square root that (cid:112) | a | κ < K − 1 ϵ − 1 κ − 3 | a | .

Original: Since | b | < (cid:112) | a | κ , we have | b | < K − 1 ϵ − 1 κ − 3 | a | .
Revised: Since | b | < (cid:112) | a | κ , we have | b | < K − 1 ϵ − 1 κ − 3 | a | .

Original: Finally multiply Kϵ | b | κ 3 on both side, we have K | ϵb 2 κ 3 | < | abκ | .
Revised: Finally multiply Kϵ | b | κ 3 on both side, we have K | ϵb 2 κ 3 | < | abκ | .

Original: From (i) we have 16 | b 3 | < 2 | abκ | , from (ii) - (vii) we have R b =
Revised: From (i) we have 16 | b 3 | < 2 | abκ | , from (ii) - (vii) we have R b =

Original: K (cid:12)(cid:12) b 4 (cid:12)(cid:12)
Revised: K (cid:12)(cid:12) b 4 (cid:12)(cid:12)

Original: + K (cid:12)(cid:12) ab 2 κ (cid:12)(cid:12) + K (cid:12)(cid:12) a 2 bκ 2 (cid:12)(cid:12)
Revised: + K (cid:12)(cid:12) ab 2 κ (cid:12)(cid:12) + K (cid:12)(cid:12) a 2 bκ 2 (cid:12)(cid:12)

Original: +
Revised: +

Original: K (cid:12)(cid:12) b
Revised: K (cid:12)(cid:12) b

Original: 2 κ 4 (cid:12)(cid:12) + K (cid:12)(cid:12) bκ 5 (cid:12)(cid:12) + K (cid:12)(cid:12) ϵb 2 κ 3 (cid:12)(cid:12) <
Revised: 2 κ 4 (cid:12)(cid:12) + K (cid:12)(cid:12) bκ 5 (cid:12)(cid:12) + K (cid:12)(cid:12) ϵb 2 κ 3 (cid:12)(cid:12) <

Original: 5 | abκ | .
Revised: 5 | abκ | .

Original: Therefore | b ′′ − ( b + 8 abκ ) | ≤ 16 | b 3 | + R b ≤ 7 | abκ | , which completes the proof.
Revised: Therefore | b ′′ − ( b + 8 abκ ) | ≤ 16 | b 3 | + R b ≤ 7 | abκ | , which completes the proof.

Original: We restate the sufficient condition for | b ′′ − ( b + 8 abκ ) | ≤ 7 | abκ | as follow:
Revised: We restate the sufficient condition for | b ′′ − ( b + 8 abκ ) | ≤ 7 | abκ | as follow:

Original: Condition B.4 (Condition for abκ Dominated b ′′ − b Movement) .
Revised: Condition B.4 (Condition for abκ Dominated b ′′ − b Movement) .

Original: [Equation]
Revised: [Equation]

Original: C ONVERGENCE WHEN − b 3 DOMINATES abκ
Revised: C ONVERGENCE WHEN − b 3 DOMINATES abκ

Original: Now we are ready to analyze the dynamics when c 3 dominates the movement of c .
Revised: Now we are ready to analyze the dynamics when c 3 dominates the movement of c .

Original: For t ∈ N , we abuse the notation to let ( c t , d t ) denote c and d after the t -th 2-step update with step size η
Revised: For t ∈ N , we abuse the notation to let ( c t , d t ) denote c and d after the t -th 2-step update with step size η

Original: = κfrom the initialization ( c 0 , d 0 ) .
Revised: = κfrom the initialization ( c 0 , d 0 ) .

Original: Lemma 8 (Convergence to near parabola from large b ) .
Revised: Lemma 8 (Convergence to near parabola from large b ) .

Original: For any κ < K − 1 , for any initialization ( a 0 , b 0 ) satisfying a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and | b 0 | ∈
Revised: For any κ < K − 1 , for any initialization ( a 0 , b 0 ) satisfying a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and | b 0 | ∈

Original: [2 (cid:112) | a 0 | κ, K − 1 ) , there exists some T < κ −such that a T ∈ (2 κ 52 , K − 2 κ − 1 ) and | b T | ∈ ( √ a T κ, 2 √ a T κ ) .
Revised: [2 (cid:112) | a 0 | κ, K − 1 ) , there exists some T < κ −such that a T ∈ (2 κ 52 , K − 2 κ − 1 ) and | b T | ∈ ( √ a T κ, 2 √ a T κ ) .

Original: Proof of Lemma 8.
Revised: Proof of Lemma 8.

Original: We will prove the claim using induction.
Revised: We will prove the claim using induction.

Original: Consider the inductive hypothesis for k ∈ { 0 , 1 , · · · , ⌊ κ − 4 ⌋} that
Revised: Consider the inductive hypothesis for k ∈ { 0 , 1 , · · · , ⌊ κ − 4 ⌋} that

Original: P ( k ) : | a k | ∈ (2 κ 52 , 14 K − 2 κ − 1 ) and | b k | ∈ (2 √ a k κ, min { K − 1 , ( k + 1) − 12 ) } .
Revised: P ( k ) : | a k | ∈ (2 κ 52 , 14 K − 2 κ − 1 ) and | b k | ∈ (2 √ a k κ, min { K − 1 , ( k + 1) − 12 ) } .

Original: Since | b 0 | < K − 1 < 1 = (0 + 1) − 12 and a 0 ∈ (3 κ 52 , 14 K − 2 κ − 1 ) ⊂
Revised: Since | b 0 | < K − 1 < 1 = (0 + 1) − 12 and a 0 ∈ (3 κ 52 , 14 K − 2 κ − 1 ) ⊂

Original: (2 κ 52 , 14 K − 2 κ − 1 ) as required by the initialization, the base case P (0) holds trivially.
Revised: (2 κ 52 , 14 K − 2 κ − 1 ) as required by the initialization, the base case P (0) holds trivially.

Original: Now we can proceed to the inductive step.
Revised: Now we can proceed to the inductive step.

Original: Assume P ( l ) holds for all l ≤ k , we want to show that P ( k + 1) holds unless | b k +1 | < 2 (cid:112) | a k +1 | κ .
Revised: Assume P ( l ) holds for all l ≤ k , we want to show that P ( k + 1) holds unless | b k +1 | < 2 (cid:112) | a k +1 | κ .

Original: By the inductive hypothesis, ( a k , b k ) satisfies condition B.2 and condition B.3.
Revised: By the inductive hypothesis, ( a k , b k ) satisfies condition B.2 and condition B.3.

Original: Thus by Lemmaand Lemma 6 we have
Revised: Thus by Lemmaand Lemma 6 we have

Original: [Equation]
Revised: [Equation]

Original: By the strong inductive hypothesis, these properties also holds when substituting k by any l < k .
Revised: By the strong inductive hypothesis, these properties also holds when substituting k by any l < k .

Original: We first check the lower bound for a k +1 under the assumption that k < κ − 4 .
Revised: We first check the lower bound for a k +1 under the assumption that k < κ − 4 .

Original: Since a l +1 > a l − 7 b 2 l κ 3 for all l ≤ k , we have
Revised: Since a l +1 > a l − 7 b 2 l κ 3 for all l ≤ k , we have

Original: [Equation]
Revised: [Equation]

Original: b l < ( l 
Revised: b l < ( l + 1)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: IH
Revised: IH

Original: [Equation]
Revised: [Equation]

Original: Under the assumption that k < κ − 4 and κ < K − 1 < 1 512 , it is not hard to check that
Revised: Under the assumption that k < κ − 4 and κ < K − 1 < 1 512 , it is not hard to check that

Original: [Equation]
Revised: [Equation]

Original: [Equation] The last inequality holds since when κ = 1512 we have 2 − 4 log( 1512 ) − 107 ( 1512 ) − 12 < − 5 and 2 + log( κ − 4 )
Revised: The last inequality holds since when κ = 1512 we have 2 − 4 log( 1512 ) − 107 ( 1512 ) − 12 < − 5 and 2 + log( κ − 4 )

Original: = 2 − 4 log( κ ) is monotonically increasing when κ < 1512 .
Revised: = 2 − 4 log( κ ) is monotonically increasing when κ < 1512 .

Original: Plugging back into
Revised: Plugging back into

Original: Eq.
Revised: Eq.

Original: (77), we know that if P ( l ) holds for all l ≤ k and k < κ − 4 , then
Revised: (77), we know that if P ( l ) holds for all l ≤ k and k < κ − 4 , then

Original: [Equation]
Revised: [Equation]

Original: The upper bound of a k < 14 K − 2 κ − 1 always holds since a is monotonically decreasing by Eq.
Revised: The upper bound of a k < 14 K − 2 κ − 1 always holds since a is monotonically decreasing by Eq.

Original: Now we check the upper bound for | b k +1 | .
Revised: Now we check the upper bound for | b k +1 | .

Original: Consider f ( x ) = x − 12 , we have f ′ ( x ) = − 12 x − 32 = − 12 f ( x ) 3 and f ′′
Revised: Consider f ( x ) = x − 12 , we have f ′ ( x ) = − 12 x − 32 = − 12 f ( x ) 3 and f ′′

Original: ( x ) = 34 x − 52 .
Revised: ( x ) = 34 x − 52 .

Original: Note that f ′′ ( x ) > 0 for all x > 0 , so by a first order Taylor expansion around x = k
Revised: Note that f ′′ ( x ) > 0 for all x > 0 , so by a first order Taylor expansion around x = k

Original: + 

Revised: +

Original: we have f ( k + 2) >
Revised: we have f ( k + 2) >

Original: f ( k + 1) − 12 f ( k + 1) 3 .
Revised: f ( k + 1) − 12 f ( k + 1) 3 .

Original: Combined with | b k +1 | <
Revised: Combined with | b k +1 | <

Original: | b k | − | b 3 k | , it follows
Revised: | b k | − | b 3 k | , it follows

Original: [Equation]
Revised: [Equation]

Original: Since f ( k + 1) ≤ 1 and | b k | < | b 0 | < K − 1 < 12 .
Revised: Since f ( k + 1) ≤ 1 and | b k | < | b 0 | < K − 1 < 12 .

Original: for all k ≥ 1 , we have b 2 k + | b k | f
Revised: for all k ≥ 1 , we have b 2 k + | b k | f

Original: ( k + 1) + f ( k + 1) 2 < 2 , and hence (cid:0) 1 − 12 (cid:0) b 2 k + |
Revised: ( k + 1) + f ( k + 1) 2 < 2 , and hence (cid:0) 1 − 12 (cid:0) b 2 k + |

Original: b k | f
Revised: b k | f

Original: ( k + 1) + f ( k + 1) 2 (cid:1)(cid:1) > 0 .
Revised: ( k + 1) + f ( k + 1) 2 (cid:1)(cid:1) > 0 .

Original: Since | b k | < ( k + 1) − 12 = f ( k + 1) by the induction hypothesis, we have f ( k + 2) − | b t +1 | > 0 , so | b k +1 | <
Revised: Since | b k | < ( k + 1) − 12 = f ( k + 1) by the induction hypothesis, we have f ( k + 2) − | b t +1 | > 0 , so | b k +1 | <

Original: ( k + 2) − 12 .
Revised: ( k + 2) − 12 .

Original: The other upper bound | b k +1 | < K − 1 holds trivially since | b 0 | <
Revised: The other upper bound | b k +1 | < K − 1 holds trivially since | b 0 | <

Original: K − 1 as required by the initialization and | b k | is monotonically decreasing according to Eq.
Revised: K − 1 as required by the initialization and | b k | is monotonically decreasing according to Eq.

Original: Now we will show that there exists some τ < κ − 4 that | b τ | < 2 (cid:112) | a τ | κ .
Revised: Now we will show that there exists some τ < κ − 4 that | b τ | < 2 (cid:112) | a τ | κ .

Original: Assume toward contradiction that there is no such τ , then the induction may proceed to h ≜
Revised: Assume toward contradiction that there is no such τ , then the induction may proceed to h ≜

Original: ⌊ κ − 4 ⌋ so that a h > 2 κ 52 , | b h | <
Revised: ⌊ κ − 4 ⌋ so that a h > 2 κ 52 , | b h | <

Original: ( h + 1) 12 < ( κ − 4 ) 12 < κ 2 and | b h | > 2 √ a h κ > 2 (cid:112) 2 κ 52 κ = 2 √ 2 κ 74 .
Revised: ( h + 1) 12 < ( κ − 4 ) 12 < κ 2 and | b h | > 2 √ a h κ > 2 (cid:112) 2 κ 52 κ = 2 √ 2 κ 74 .

Original: The last two inequalities lead to contradiction as 2 √ 2 κ 7 4 > κ 2 , so the assumption does not hold and there exists some τ < κ − 4 for | b τ | < 2 √ a τ κ .
Revised: The last two inequalities lead to contradiction as 2 √ 2 κ 7 4 > κ 2 , so the assumption does not hold and there exists some τ < κ − 4 for | b τ | < 2 √ a τ κ .

Original: Let T <
Revised: Let T <

Original: ⌊ κ − 4 ⌋ be the smallest such τ , then the induction may proceed to k
Revised: ⌊ κ − 4 ⌋ be the smallest such τ , then the induction may proceed to k

Original: = T − 1 , which guarantees P ( t ) for all t < T .
Revised: = T − 1 , which guarantees P ( t ) for all t < T .

Original: Thus for all t < T , a t > (2 κ 52 , K − 2 κ − 1 ) and | b t | ∈ (2 √ a T κ, K − 1 ) .
Revised: Thus for all t < T , a t > (2 κ 52 , K − 2 κ − 1 ) and | b t | ∈ (2 √ a T κ, K − 1 ) .

Original: Since P ( T − 1) holds, following Eq.
Revised: Since P ( T − 1) holds, following Eq.

Original: (77) we also have a T > 2 κ 5 2 .
Revised: (77) we also have a T > 2 κ 5 2 .

Original: Now we still need to show | b T | > √ a T κ .
Revised: Now we still need to show | b T | > √ a T κ .

Original: Since a k is monotonically decreasing, it is sufficient to show | b T | > √ a T − 1 κ.
Revised: Since a k is monotonically decreasing, it is sufficient to show | b T | > √ a T − 1 κ.

Original: From Eq.
Revised: From Eq.

Original: (76) we have | b T | > | b T − 1 | − 30 | b 3 T − 1 | =
Revised: (76) we have | b T | > | b T − 1 | − 30 | b 3 T − 1 | =

Original: | b T − 1 | (1 − 30 b 2 T − 1 ) .
Revised: | b T − 1 | (1 − 30 b 2 T − 1 ) .

Original: Since b T − 1 < K − 1 < 1512 , (1 − 30 b 2 T − 1 ) > 12 .
Revised: Since b T − 1 < K − 1 < 1512 , (1 − 30 b 2 T − 1 ) > 12 .

Original: Combined with | b T − 1 | > 2 √ a T − 1 κ as shown above, we have | b T | > 12 (2 √ a T − 1 κ ) > √ a T − 1 κ , which completes the proof.
Revised: Combined with | b T − 1 | > 2 √ a T − 1 κ as shown above, we have | b T | > 12 (2 √ a T − 1 κ ) > √ a T − 1 κ , which completes the proof.

Original: B.4. C ONVERGENCE WHEN abκ DOMINATES b
Revised: B.4. C ONVERGENCE WHEN abκ DOMINATES b

Original: Lemma 9 (Convergence to near parabola from small b ) .
Revised: Lemma 9 (Convergence to near parabola from small b ) .

Original: For any κ < K − 1 , for any initialization ( a 0 , b 0 ) satisfying a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and | b 0 | ∈ (0 , 14 √ a 0 κ ] , there exists some
Revised: For any κ < K − 1 , for any initialization ( a 0 , b 0 ) satisfying a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and | b 0 | ∈ (0 , 14 √ a 0 κ ] , there exists some

Original: T < 12 log( | b 0 | − 1 ) κ − 72 such that | b T | ∈ ( 14 √ a T κ, 12 √ a T κ ) and a T ∈ (2 κ 52 , 14 K − 2 κ − 1 ) .
Revised: T < 12 log( | b 0 | − 1 ) κ − 72 such that | b T | ∈ ( 14 √ a T κ, 12 √ a T κ ) and a T ∈ (2 κ 52 , 14 K − 2 κ − 1 ) .

Original: Proof of Lemma Lemma 9.
Revised: Proof of Lemma Lemma 9.

Original: We will prove this claim using induction.
Revised: We will prove this claim using induction.

Original: Consider the inductive hypothesis for k ∈ N that
Revised: Consider the inductive hypothesis for k ∈ N that

Original: [Equation]
Revised: [Equation]

Original: and if
Revised: and if

Original: [Equation]
Revised: [Equation]

Original: The base case when k = 0 holds from the initialization, so we proceed to the inductive step.
Revised: The base case when k = 0 holds from the initialization, so we proceed to the inductive step.

Original: Assume
Revised: Assume

Original: P ( l ) holds for all l ≤ k , we want to show P ( k + 1) holds unless | b k | > 14 √ a k κ .
Revised: P ( l ) holds for all l ≤ k , we want to show P ( k + 1) holds unless | b k | > 14 √ a k κ .

Original: First note that by the inductive hypothesis, since a k < 14 K − 2 κ − 1 , we have | b k | < 14 √ a k κ < 14 √ K − 2 κ − 1 κ < K − 1 .
Revised: First note that by the inductive hypothesis, since a k < 14 K − 2 κ − 1 , we have | b k | < 14 √ a k κ < 14 √ K − 2 κ − 1 κ < K − 1 .

Original: Combining with conditions on | a k | and | b k | from the inductive hypothesis we have ( a k , b k ) satisfying condition B.2 and condition B.4.
Revised: Combining with conditions on | a k | and | b k | from the inductive hypothesis we have ( a k , b k ) satisfying condition B.2 and condition B.4.

Original: Thus by Lemma 5 and Lemma 7 we have
Revised: Thus by Lemma 5 and Lemma 7 we have

Original: [Equation]
Revised: [Equation]

Original: Observe that when a is not too small, the movement of b is significantly larger than the movement of a .
Revised: Observe that when a is not too small, the movement of b is significantly larger than the movement of a .

Original: From Eq.
Revised: From Eq.

Original: (81) we have a k − a k +1 < 8 b 2 k κ 3 and | b k +1 | − | b k | > 2 a k | b k | κ , so
Revised: (81) we have a k − a k +1 < 8 b 2 k κ 3 and | b k +1 | − | b k | > 2 a k | b k | κ , so

Original: [Equation]
Revised: [Equation]

Original: since | b t | < 14 √ a t κ )
Revised: | [Equation] t | [Equation]

Original: [Equation]
Revised: [Equation]

Revised: a t

Original: [Equation]
Revised: [Equation]

Original: ( since a k > κ 52 ) (82)
Revised: ( since a k > κ 52 ) (82)

Original: When k = 0 , we directly have ( | b 1 | − | b 0 | ) / ( a 0 − a 1 ) > κ − 54 .
Revised: When k = 0 , we directly have ( | b 1 | − | b 0 | ) / ( a 0 − a 1 ) > κ − 54 .

Original: When k ≥ 1 , from the inductive hypothesis we have ( | b k | − | b 0 | ) / ( a 0 − a k ) > κ − 54 , combining with ( | b k +1 | − | b k | ) /
Revised: When k ≥ 1 , from the inductive hypothesis we have ( | b k | − | b 0 | ) / ( a 0 − a k ) > κ − 54 , combining with ( | b k +1 | − | b k | ) /

Original: ( a k − a k +1 ) >
Revised: ( a k − a k +1 ) >

Original: κ − 54 , we have ( | b k +1 | − | b 0 | ) /
Revised: κ − 54 , we have ( | b k +1 | − | b 0 | ) /

Original: ( a 0 − a k +1 ) >
Revised: ( a 0 − a k +1 ) >

Original: κ − 54 by the mediant inequality.
Revised: κ − 54 by the mediant inequality.

Original: Now we check the bounds on | a k +1 | and | b k +1 | .
Revised: Now we check the bounds on | a k +1 | and | b k +1 | .

Original: Since a is monotonically decreasing from Eq.
Revised: Since a is monotonically decreasing from Eq.

Original: (81), we have a k +1 < a 0 < 14 K − 2 κ − 1 .
Revised: (81), we have a k +1 < a 0 < 14 K − 2 κ − 1 .

Original: Since ( | b k | − | b 0 | ) / ( a 0 − a k ) > κ − 54 according to the inductive hypothesis and ( a 0 − a k ) > 0 by monotonicity of a , a 0 − a k < ( | b k | − | b 0 | ) κ 54 <
Revised: Since ( | b k | − | b 0 | ) / ( a 0 − a k ) > κ − 54 according to the inductive hypothesis and ( a 0 − a k ) > 0 by monotonicity of a , a 0 − a k < ( | b k | − | b 0 | ) κ 54 <

Original: | b k | κ 54 < 14 √ a k κκ 54 = 14 √ a k κ 74 < 14 √ a 0 κ 74 .
Revised: | b k | κ 54 < 14 √ a k κκ 54 = 14 √ a k κ 74 < 14 √ a 0 κ 74 .

Original: Here the last step holds again by monotonicity of a .
Revised: Here the last step holds again by monotonicity of a .

Original: Reorganizing the inequality we have a k > a 0 − 14 √ a 0 κ 74 = √ a 0 ( √ a 0 − 14 κ 74 ) .
Revised: Reorganizing the inequality we have a k > a 0 − 14 √ a 0 κ 74 = √ a 0 ( √ a 0 − 14 κ 74 ) .

Original: Since a 0 > 12 κ 52 by the initialization condition, √ a 0 > 3 κand √ a 0 − 14 κ 74 > 2 κ 54 .
Revised: Since a 0 > 12 κ 52 by the initialization condition, √ a 0 > 3 κand √ a 0 − 14 κ 74 > 2 κ 54 .

Original: Thus a k > √ a 0 ( √ a 0 − 14 κ 74 ) > 6 κ 52 .
Revised: Thus a k > √ a 0 ( √ a 0 − 14 κ 74 ) > 6 κ 52 .

Original: Note that since | b k | < K − 1 < 1512 , 7 b 2 k κ 3 < 7 K − 2 κ 3 < κ 3 .
Revised: Note that since | b k | < K − 1 < 1512 , 7 b 2 k κ 3 < 7 K − 2 κ 3 < κ 3 .

Original: From Eq.
Revised: From Eq.

Original: (81) we have a k +1 > a k − 8 b
Revised: (81) we have a k +1 > a k − 8 b

Original: 2 k κ 3 > 6 κ 5 2 − κ 3 > 2 κ 52 .
Revised: 2 k κ 3 > 6 κ 5 2 − κ 3 > 2 κ 52 .

Original: This gives the desired lower bound for a k +
Revised: This gives the desired lower bound for a k +

Original: Since | b k +1 | > | b k | + 2 a k | b k | κ from Eq.
Revised: Since | b k +1 | > | b k | + 2 a k | b k | κ from Eq.

Original: (81), combining with a k > 2 κ 52 we have | b k +1 | > | b k | (1 + 2 a k κ ) > | b k | (1 + 4 κ 52 κ ) .
Revised: (81), combining with a k > 2 κ 52 we have | b k +1 | > | b k | (1 + 2 a k κ ) > | b k | (1 + 4 κ 52 κ ) .

Original: Since | b k | > | b 0 | (1 + 4 κ 72 ) k by the inductive hypothesis, we have | b
Revised: Since | b k | > | b 0 | (1 + 4 κ 72 ) k by the inductive hypothesis, we have | b

Original: k
Revised: k

Original: +

Revised: +

Original: | > | b 0 | (1 + 4 κ 7 2 ) k
Revised: | > | b 0 | (1 + 4 κ 7 2 ) k

Original: +1 .
Revised: +1 .

Original: With the guarantees on | a k +1 | , | b k +1 | , and ( | b k |−| b 0 | ) /
Revised: With the guarantees on | a k +1 | , | b k +1 | , and ( | b k |−| b 0 | ) /

Original: ( a 0 − a k ) as shown above, if we additionally assume that | b k +1 | < 14 √ a k +1 κ , P ( k + 1) will hold and the induction can proceed.
Revised: ( a 0 − a k ) as shown above, if we additionally assume that | b k +1 | < 14 √ a k +1 κ , P ( k + 1) will hold and the induction can proceed.

Original: Now claim that there exists some τ < 12 log( | b 0 | − 1 ) κ − 72 that | b τ | ≥ 14 √ a τ κ .
Revised: Now claim that there exists some τ < 12 log( | b 0 | − 1 ) κ − 72 that | b τ | ≥ 14 √ a τ κ .

Original: Assume toward contradiction that there is no such τ , then the induction can proceed for any t ∈ N .
Revised: Assume toward contradiction that there is no such τ , then the induction can proceed for any t ∈ N .

Original: Consider t ≥ 12 log( | b 0 | − 1 ) κ − 72 , we have
Revised: Consider t ≥ 12 log( | b 0 | − 1 ) κ − 72 , we have

Original: [Equation]
Revised: [Equation]

Original: ( Since κ − 72 > 12 ) > (cid:0) 
Revised: ( Since κ − 72 > 12 ) > (cid:0) 12 log(

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: +
Revised: +

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original:  (cid:1) (
Revised: ) (cid:1) (

Original: [Equation]
Revised: [Equation]

Original: ) ( Since log( K − 2 ) < 0)
Revised: ) ( Since log( K − 2 ) < 0)

Original: = (cid:0) 12 log(( K − 2 κ − 1 ) κ ) + log( | b 0 | − 1 ) (cid:1) (1 + 14 κ − 72 ) > (cid:0) 12 log( a k κ ) + log( | b 0 | − 1 ) (cid:1) (1 + 14 κ − 72 )
Revised: = (cid:0) 12 log(( K − 2 κ − 1 ) κ ) + log( | b 0 | − 1 ) (cid:1) (1 + 14 κ − 72 ) > (cid:0) 12 log( a k κ ) + log( | b 0 | [Equation] 1 ) (cid:1) ( [Equation]

Revised: [Equation]

Revised: − 72 )

Original: ( Since K − 2 κ − 1 > a k ) = (cid:0) 12 log( ak κ ) + log( |
Revised: ( Since K − 2 κ − 1 > a k ) = (cid:0) [Equation] ( a

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: ( |

Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: 1 ) (cid:1) (1 +
Revised: ) (cid:1) (

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: = (cid:0) 12 log( a k
Revised: = (cid:0) 12 log( [Equation] k

Original: [Equation]
Revised: [Equation]

Original: + log( |
Revised: + log( |

Original: [Equation]
Revised: [Equation]

Original: 0 | − 1 ) (cid:1) log(1 +
Revised: | [Equation] 1 ) (cid:1) log(

Original: [Equation]
Revised: [Equation]

Original: 72 ) −( Since log(1 + x ) ≥ x/ (1 + x )) .
Revised: [Equation] ( Since log(1 + x ) ≥ x/ (1 + x )) .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Original: Hence (1 + 4 κ 72 ) t > 14 √ a t κ/ | b 0 | , and therefore | b t | > | b 0 | (1 + 4 κ 72 ) t > 14 √ a t κ , which contradicts that P ( t ) holds and lead to contradiction.
Revised: Hence (1 + 4 κ 72 ) t > 14 √ a t κ/ | b 0 | , and therefore | b t | > | b 0 | (1 + 4 κ 72 ) t > 14 √ a t κ , which contradicts that P ( t ) holds and lead to contradiction.

Original: Therefore there must exists some τ such that | b τ | > 14 √ a τ κ .
Revised: Therefore there must exists some τ such that | b τ | > 14 √ a τ κ .

Original: Let T < 12 log( | b 0 | − 1 ) κ − 72 be the smallest such τ , then the induction will proceed to k
Revised: Let T < 12 log( | b 0 | − 1 ) κ − 72 be the smallest such τ , then the induction will proceed to k

Original: =
Revised: =

Original: T − 1 .
Revised: T − 1 .

Original: Moreover since P ( T − 1) holds, following the previous analysis, the bounds on | a T | also holds, so we have | a T | ∈ (2 κ 52 , 14 K − 2 κ − 1 ) .
Revised: Moreover since P ( T − 1) holds, following the previous analysis, the bounds on | a T | also holds, so we have | a T | ∈ (2 κ 52 , 14 K − 2 κ − 1 ) .

Original: Now to complete the proof we only need to show that | b T | < 12 √ a T κ .
Revised: Now to complete the proof we only need to show that | b T | < 12 √ a T κ .

Original: From Eq.
Revised: From Eq.

Original: (81) we have | b T | <
Revised: (81) we have | b T | <

Original: | b T − 1 | + 14 a T − 1 | b T − 1 | κ = (1 + 14 a T − 1 κ ) | b T − 1 | .
Revised: | b T − 1 | + 14 a T − 1 | b T − 1 | κ = (1 + 14 a T − 1 κ ) | b T − 1 | .

Original: Since a T − 1 < 14 K − 2 κ − 1 and K > 512 , (1 + 14 a T − 1 κ ) < (1 + 14 K − 2 ) < 1 . 1 .
Revised: Since a T − 1 < 14 K − 2 κ − 1 and K > 512 , (1 + 14 a T − 1 κ ) < (1 + 14 K − 2 ) < 1 . 1 .

Original: So | b T | < 1 . 1 | b T − 1 | < 1 . 1( 14 √ a T − 1 κ ) .
Revised: So | b T | < 1 . 1 | b T − 1 | < 1 . 1( 14 √ a T − 1 κ ) .

Original: [Equation]
Revised: [Equation]

Original: On the other side, note that a T > a T − 1 − 7 b 2 T − 1 κ 3 by Eq.
Revised: On the other side, note that a T > a T − 1 − 7 b 2 T − 1 κ 3 by Eq.

Original: (81), where b 2 T − 1 < ( 14 √ a T − 1 κ ) 2 = 116 a T − 1 κ .
Revised: (81), where b 2 T − 1 < ( 14 √ a T − 1 κ ) 2 = 116 a T − 1 κ .

Original: We have a T > a T − 1 − 716 a T − 1 κ > (1 − κ ) a T − 1 .
Revised: We have a T > a T − 1 − 716 a T − 1 κ > (1 − κ ) a T − 1 .

Original: Since κ < K − 1 < 1512 , we have a
Revised: Since κ < K − 1 < 1512 , we have a

Original: T > 0 .
Revised: T > 0 .

Original: 99 a T − 1 and hence
Revised: 99 a T − 1 and hence

Original: [Equation]
Revised: [Equation]

Original: Combining Eq.
Revised: Combining Eq.

Original: (85) and Eq.
Revised: (85) and Eq.

Original: (86) we have | b T | < 1 . 21( 14 √ a T κ ) < 12 √ a T κ .
Revised: (86) we have | b T | < 1 . 21( 14 √ a T κ ) < 12 √ a T κ .

Original: This concludes the proof for this lemma.
Revised: This concludes the proof for this lemma.

Original: B.4. 
Revised: B.4. | b | S TAYS IN

Original: | b | S TAYS IN ( 14 √ aκ, 2 √ aκ ) AS a D ECREASES W HEN a IS N OT T OO S MALL
Revised: ( 14 √ aκ, 2 √ aκ ) AS a D ECREASES W HEN a IS N OT T OO S MALL

Original: After showing that b will enter ( 14 √ aκ, 2 √ aκ ) , we now show that it will not leave this region unless a is very small.
Revised: After showing that b will enter ( 14 √ aκ, 2 √ aκ ) , we now show that it will not leave this region unless a is very small.

Original: To do so, we first determine a regime in which we can effectively bound the two-step movement of b .
Revised: To do so, we first determine a regime in which we can effectively bound the two-step movement of b .

Original: Lemma 10.
Revised: Lemma 10.

Original: For any κ, a, b satisfying
Revised: For any κ, a, b satisfying

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: we have | b ′′ − b | < 116 √ aκ.
Revised: we have | b ′′ − b | < 116 √ aκ.

Original: Proof of Lemma 10.
Revised: Proof of Lemma 10.

Original: Fix ϵ
Revised: Fix ϵ

Original: = 0 .
Revised: = 0 .

Original: 1 , it is straightforward to check that κ, a, b satisfies condition B.1.
Revised: 1 , it is straightforward to check that κ, a, b satisfies condition B.1.

Original: Hence from Eq.
Revised: Hence from Eq.

Original: (64) and Eq.
Revised: (64) and Eq.

Original: (65) we have that
Revised: (65) we have that

Original: [Equation]
Revised: [Equation]

Original: To prove the claim it is sufficient to bound all terms on RHS under 1128 √ aκ .
Revised: To prove the claim it is sufficient to bound all terms on RHS under 1128 √ aκ .

Original: Note that with b < 2 √ aκ and a < K − 2 κ − 1 , √ aκ < √ K − 2 =
Revised: Note that with b < 2 √ aκ and a < K − 2 κ − 1 , √ aκ < √ K − 2 =

Original: K − 1 < 1128 and hence b < 2 K − 1 .
Revised: K − 1 < 1128 and hence b < 2 K − 1 .

Original: (i) Since K > 512 , aκ < K − 2 < 1 16384 .
Revised: (i) Since K > 512 , aκ < K − 2 < 1 16384 .

Original: Multiplying 128 √ aκ on both sides we have 128( aκ ) 32 < 1128 √ aκ .
Revised: Multiplying 128 √ aκ on both sides we have 128( aκ ) 32 < 1128 √ aκ .

Original: Since | b | < 2 √ aκ , | b 3 | < 8( aκ ) 32 , thus 16 | b 3 | < 128( aκ ) 32 < 1128 √ aκ.
Revised: Since | b | < 2 √ aκ , | b 3 | < 8( aκ ) 32 , thus 16 | b 3 | < 128( aκ ) 32 < 1128 √ aκ.

Original: (ii)
Revised: (ii)

Original: Since | b | < 2 √ aκ , 8 | abκ | < 16( aκ ) 32 < 1 1024 √ aκ .
Revised: Since | b | < 2 √ aκ , 8 | abκ | < 16( aκ ) 32 < 1 1024 √ aκ .

Original: The last inequality holds directly from (i).
Revised: The last inequality holds directly from (i).

Original: (iii)
Revised: (iii)

Original: Since | b | < 2 √ aκ , Kb 4 < 16 Ka 2 κ 2 = 16 K ( aκ ) 32 √ aκ .
Revised: Since | b | < 2 √ aκ , Kb 4 < 16 Ka 2 κ 2 = 16 K ( aκ ) 32 √ aκ .

Original: Since √ aκ < K −
Revised: Since √ aκ < K −

Original: 16 K ( aκ ) 32 √ aκ < 16 K − 2 √ aκ < 16 16284 √ aκ < 1128 √ aκ.
Revised: 16 K ( aκ ) 32 √ aκ < 16 K − 2 √ aκ < 16 16284 √ aκ < 1128 √ aκ.

Original: Therefore Kb 4 < 1128 √ abκ .
Revised: Therefore Kb 4 < 1128 √ abκ .

Original: (iv) Since | b | < 2 K − 1 , K | ab 2 κ | < 2 | abκ | , which is less than 1512 √ aκ from (ii).
Revised: (iv) Since | b | < 2 K − 1 , K | ab 2 κ | < 2 | abκ | , which is less than 1512 √ aκ from (ii).

Original: (v) Since | aκ | <
Revised: (v) Since | aκ | <

Original: K − 1 , K | ab 2 κ 2 | < | abκ | < 1 1024 √ abκ.
Revised: K − 1 , K | ab 2 κ 2 | < | abκ | < 1 1024 √ abκ.

Original: (vi) Note that as κ < 0 .
Revised: (vi) Note that as κ < 0 .

Original: 1 in condition B.1, κ 4 < 1512 .
Revised: 1 in condition B.1, κ 4 < 1512 .

Original: Since | b | < 2 √ aκ < 2 K − 1 , b 2 < 4 K − 1 √ aκ , and thus Kb 2 κ 4 < 4 √ aκκ 4 < 1128 √ aκ.
Revised: Since | b | < 2 √ aκ < 2 K − 1 , b 2 < 4 K − 1 √ aκ , and thus Kb 2 κ 4 < 4 √ aκκ 4 < 1128 √ aκ.

Original: (vii)
Revised: (vii)

Original: Since b < 2 √ aκ , K | bκ 5 | < 2 Kκ 5 √ aκ .
Revised: Since b < 2 √ aκ , K | bκ 5 | < 2 Kκ 5 √ aκ .

Original: Given κ < K − 1 < 14 K − 15 , we have κ 5 < 1 1024 K − 1 , and hence 2 Kκ 5 √ aκ < 1512 √ aκ .
Revised: Given κ < K − 1 < 14 K − 15 , we have κ 5 < 1 1024 K − 1 , and hence 2 Kκ 5 √ aκ < 1512 √ aκ .

Original: Thus K | bκ 5 | < 1512 √ abκ.
Revised: Thus K | bκ 5 | < 1512 √ abκ.

Original: (viii)
Revised: (viii)

Original: Since we fixed ϵ = 0 .
Revised: Since we fixed ϵ = 0 .

Original: 1 , κ < K − 1 < 1512 <
Revised: 1 , κ < K − 1 < 1512 <

Original: 18 ϵ − 13 , κ 3 < 1512 ϵ − 1 .
Revised: 18 ϵ − 13 , κ 3 < 1512 ϵ − 1 .

Original: Multiplying 4 ϵ √ aκ on both sides we have 4 ϵ √ aκκ 3 < 1128 √ aκ.
Revised: Multiplying 4 ϵ √ aκ on both sides we have 4 ϵ √ aκκ 3 < 1128 √ aκ.

Original: Since b 2 < 4 K − 1 √ aκ from (vi), we have 
Revised: Since b 2 < 4 K − 1 √ aκ from (vi), we have Kϵb

Original: [Equation]
Revised: [Equation]

Original: Now that we have bounded every monomial term on RHS of Eq.
Revised: Now that we have bounded every monomial term on RHS of Eq.

Original: (88) by 1128 √
Revised: (88) by 1128 √

Original: aκ , we have | b ′′ − b | < 116 √ aκ , which completes the proof.
Revised: aκ , we have | b ′′ − b | < 116 √ aκ , which completes the proof.

Original: Here we restate the condition for Lemma
Revised: Here we restate the condition for Lemma

Original: Condition B.5 (Condition for small b movement) .
Revised: Condition B.5 (Condition for small b movement) .

Original: [Equation]
Revised: [Equation]

Original: With the two-step movement of c bounded above, we may proceed to state the lemma that guarantees c will not leave ( 14 √ dκ, 2 √ dκ ) unless d is very small.
Revised: With the two-step movement of c bounded above, we may proceed to state the lemma that guarantees c will not leave ( 14 √ dκ, 2 √ dκ ) unless d is very small.

Original: Lemma 11.
Revised: Lemma 11.

Original: For any κ and initialization a 0 , b 0 satisfying
Revised: For any κ and initialization a 0 , b 0 satisfying

Original: [Equation]
Revised: [Equation]

Original: There exists some T ≤ 16 a 0 κ − 132 such that a T < κ 52 and for all t < T , a t > κ 
Revised: There exists some T ≤ 16 a 0 κ − 132 such that a T < κ 52 and for all t < T , a t > κ 52 and b t ∈ ( 1 4 √ a t κ, 2 √ a t κ ) .

Original: Proof of Lemma 11.
Revised: Proof of Lemma 11.

Original: First we check that the region defined is not empty.
Revised: First we check that the region defined is not empty.

Original: This is true since given κ < 116 K − 1 , we have 14 K − 2 κ − 1 > 4 K − 1 > K − 52 > κ 52 .
Revised: This is true since given κ < 116 K − 1 , we have 14 K − 2 κ − 1 > 4 K − 1 > K − 52 > κ 52 .

Original: To prove the claim we consider the inductive hypothesis
Revised: To prove the claim we consider the inductive hypothesis

Original: [Equation]
Revised: [Equation]

Original: Assume that P ( k ) holds for some k .
Revised: Assume that P ( k ) holds for some k .

Original: Since a k < a 0 <
Revised: Since a k < a 0 <

Original: 14 K − 2 κ − 1 , we have | b k | < 2 √ a k κ < 2 (cid:113) 14 K − 2 κ − 1 κ = K − 1 .
Revised: 14 K − 2 κ − 1 , we have | b k | < 2 √ a k κ < 2 (cid:113) 14 K − 2 κ − 1 κ = K − 1 .

Original: With κ < 116 K − 1 and a k < a 0 < 14 K − 2 κ − 1 , we have κ, b k , a k satisfying condition B.2 and condition B.5.
Revised: With κ < 116 K − 1 and a k < a 0 < 14 K − 2 κ − 1 , we have κ, b k , a k satisfying condition B.2 and condition B.5.

Original: Thus a k +1 <
Revised: Thus a k +1 <

Original: a k − b 2 k κ 3 , a k +1 > a k − 8 b
Revised: a k − b 2 k κ 3 , a k +1 > a k − 8 b

Original: 2 k κ 3 (by Lemma 5), and | b k +1 − b k | < 116 √ a k κ (by Lemma 10).
Revised: 2 k κ 3 (by Lemma 5), and | b k +1 − b k | < 116 √ a k κ (by Lemma 10).

Original: Observe that since a k +1 > a k − 8 b 2 k κ 3 ,
Revised: Observe that since a k +1 > a k − 8 b 2 k κ 3 ,

Original: [Equation]
Revised: [Equation]

Original: Note that with κ < 116 K − 1 where we assume K > 128 , we have √ 1 − 32 κ 4 > 0 . 99 and hence √ a k +1 κ > 0 .
Revised: Note that with κ < 116 K − 1 where we assume K > 128 , we have √ 1 − 32 κ 4 > 0 . 99 and hence √ a k +1 κ > 0 .

Original: 99 √ a k κ.
Revised: 99 √ a k κ.

Original: Now we will show that b k +1 will not leave ( 14 √ a k +1 κ, 2 √ a k +1 κ ) .
Revised: Now we will show that b k +1 will not leave ( 14 √ a k +1 κ, 2 √ a k +1 κ ) .

Original: There are three cases to consider:
Revised: There are three cases to consider:

Original: When | b k | ∈ ( √ a k κ, 2 √ a k 
Revised: When | b k | ∈ ( √ a k κ, 2 √ a k κ

Original: κ ) , along with | b k | < K − 1 we have condition B.3 satisfied and thus | b k +1 | < | b k | (1 − b 2 k ) by Lemma 6.
Revised: ) , along with | b k | < K − 1 we have condition B.3 satisfied and thus | b k +1 | < | b k | (1 − b 2 k ) by Lemma 6.

Original: Since κ < 116 K − 1 < 1 2048 , we have κ 14 < 14 √ 2 , and thus κ 74 > 4 √ 2 κ 2 .
Revised: Since κ < 116 K − 1 < 1 2048 , we have κ 14 < 14 √ 2 , and thus κ 74 > 4 √ 2 κ 2 .

Original: Moreover, since a k > κ 52 , we have | b k | > √ a k κ > κ 74 > 4 √ 2 κ 2 .
Revised: Moreover, since a k > κ 52 , we have | b k | > √ a k κ > κ 74 > 4 √ 2 κ 2 .

Original: Squaring both sides we have b 2 k > 32 κ 4 .
Revised: Squaring both sides we have b 2 k > 32 κ 4 .

Original: Hence 1 − 32 κ 4 > 1 − b 2 k > 1 − 2 b 2 k + b 4 k
Revised: Hence 1 − 32 κ 4 > 1 − b 2 k > 1 − 2 b 2 k + b 4 k

Original: =
Revised: =

Original: 1 − b 2 k ) 2 .
Revised: 1 − b 2 k ) 2 .

Original: The last inequality holds since b 2 k > b 4 k as | b k | < 1 .
Revised: The last inequality holds since b 2 k > b 4 k as | b k | < 1 .

Original: Now taking the square root on both sides we have 1 − b 2 k < √ 1 − 32 κ 4 .
Revised: Now taking the square root on both sides we have 1 − b 2 k < √ 1 − 32 κ 4 .

Original: Since | b k | < 2 √ a k κ , combining with Eq.
Revised: Since | b k | < 2 √ a k κ , combining with Eq.

Original: (91) and 1 − b 2 k < √ 1 − 32 κ 4 we have | b k +1 | < 2 √ a k κ (1 − b 2 k ) < 2 √ 1 − 32 κ 4 √ a k κ < 2 √ a k +1 κ , which gives the desired upper bound to | b k +1 | .
Revised: (91) and 1 − b 2 k < √ 1 − 32 κ 4 we have | b k +1 | < 2 √ a k κ (1 − b 2 k ) < 2 √ 1 − 32 κ 4 √ a k κ < 2 √ a k +1 κ , which gives the desired upper bound to | b k +1 | .

Original: Now we prove the lower bound for | b k +1 | .
Revised: Now we prove the lower bound for | b k +1 | .

Original: Since we know | b k +1 − b k | < 116 √ a k κ , by triangle inequality, | b k | > √ aκ implies | b k +1 | > 1516 √ a k κ > 14 √ a k κ > 14 √ a k +1 κ .
Revised: Since we know | b k +1 − b k | < 116 √ a k κ , by triangle inequality, | b k | > √ aκ implies | b k +1 | > 1516 √ a k κ > 14 √ a k κ > 14 √ a k +1 κ .

Original: When | b k | ∈
Revised: When | b k | ∈

Original: [ 12 √ 2 √ a k κ, √ a k κ ] , since | b k +1 − b k | < 116 √ a k κ , by triangle inequality we have | b k +1 | ∈
Revised: [ 12 √ 2 √ a k κ, √ a k κ ] , since | b k +1 − b k | < 116 √ a k κ , by triangle inequality we have | b k +1 | ∈

Original: [( 1 2 √ 2 − 116 ) √ a k κ, 1716 √ a k κ ] .
Revised: [( 1 2 √ 2 − 116 ) √ a k κ, 1716 √ a k κ ] .

Original: Since ( 1 2 √ 2 − 116 ) √ a k κ > 14 √ a k κ > 14 √ a k +1 κ and 1716 √ a k κ < 2 √ a k +1 κ as √ a k +1 κ > 0 .
Revised: Since ( 1 2 √ 2 − 116 ) √ a k κ > 14 √ a k κ > 14 √ a k +1 κ and 1716 √ a k κ < 2 √ a k +1 κ as √ a k +1 κ > 0 .

Original: 99 √ a k κ , we have | b k +
Revised: 99 √ a k κ , we have | b k +

Original: 1 | ∈ ( 14 √ a k κ, 2 √ a k κ ) .
Revised: | ∈ ( 14 √ a k κ, 2 √ a k κ ) .

Original: When | b k | ∈ ( 14 √ a k κ, 12 √ 2 √ a k κ ) , along with | b k | < K − 1 we have condition B.4 satisfied, and thus | b k +1 | > | b k | +
Revised: When | b k | ∈ ( 14 √ a k κ, 12 √ 2 √ a k κ ) , along with | b k | < K − 1 we have condition B.4 satisfied, and thus | b k +1 | > | b k | +

Original: | b k a k κ | > | b k | .
Revised: | b k a k κ | > | b k | .

Original: Since √ a k +1 κ < √ a k κ , | b k +1 | > 14 √ a k +1 κ .
Revised: Since √ a k +1 κ < √ a k κ , | b k +1 | > 14 √ a k +1 κ .

Original: On the other side, since | b k +1 − b k | < 116 √ a k κ , and | b k | < 12 √ 2 √ a k κ , by triangle inequality we have | b k +1 | < ( 116 + 12 √ 2 ) √ a k κ < 2 √ a k +1 κ.
Revised: On the other side, since | b k +1 − b k | < 116 √ a k κ , and | b k | < 12 √ 2 √ a k κ , by triangle inequality we have | b k +1 | < ( 116 + 12 √ 2 ) √ a k κ < 2 √ a k +1 κ.

Original: The last step holds since √ a k +1 κ > 0 .
Revised: The last step holds since √ a k +1 κ > 0 .

Original: 99 √ a k κ .
Revised: 99 √ a k κ .