Original: Summarizing the three cases, we know that b k +1 ∈ ( 14 √ a k +1 κ, 2 √ a k +1 κ ) .
Revised: Summarizing the three cases, we know that b k +1 ∈ ( 14 √ a k +1 κ, 2 √ a k +1 κ ) .

Original: By the assumption of P ( k ) we also have a k > κ 52 and a k < a 0 − 116 ( k − 1) κ 132 .
Revised: By the assumption of P ( k ) we also have a k > κ 52 and a k < a 0 − 116 ( k − 1) κ 132 .

Original: Since we know b k > 14 √ a k κ , we have b 2 k κ 3 > 116 a k κ 4 > 116 κ 132 .
Revised: Since we know b k > 14 √ a k κ , we have b 2 k κ 3 > 116 a k κ 4 > 116 κ 132 .

Original: Thus a k +1 <
Revised: Thus a k +1 <

Original: a k − b 2 k κ 3 < a k − 

Revised: a k − b 2 k κ 3 < a k −

Original: κ 132 < a 0 − 116 kκ 132 .
Revised: κ 132 < a 0 − 116 kκ 132 .

Original: Therefore unless b k +1 < κ 52 , P ( k + 1) holds.
Revised: Therefore unless b k +1 < κ 52 , P ( k + 1) holds.

Original: Note that there must be some t such that a t < κ 5 2 since when t > 16 a 0 κ − 13 2 + 1 , a 0 − 1 16 ( t − 1) κ 13 2 < 0 .
Revised: Note that there must be some t such that a t < κ 5 2 since when t > 16 a 0 κ − 13 2 + 1 , a 0 − 1 16 ( t − 1) κ 13 2 < 0 .

Original: We induct on k from 1 , the base case holds by the initialization of b 0 and a 0 .
Revised: We induct on k from 1 , the base case holds by the initialization of b 0 and a 0 .

Original: Let T be the smallest t such that a t < κ 52 , at which we terminate the induction.
Revised: Let T be the smallest t such that a t < κ 52 , at which we terminate the induction.

Original: Then for all t < T , a t ∈ ( κ 52 , a 0 ] and b t ∈ ( 14 √ a t κ, 2 √ a t κ ) .
Revised: Then for all t < T , a t ∈ ( κ 52 , a 0 ] and b t ∈ ( 14 √ a t κ, 2 √ a t κ ) .

Original: This concludes the proof.
Revised: This concludes the proof.

Original: Corollary B.3.
Revised: Corollary B.3.

Original: Following the initialization condition and notation of Lemma 11, if a 0 > 2 κ 52 , there exists some τ < T such that a t ∈ ( 32 κ 52 , 2 κ 52 ) .
Revised: Following the initialization condition and notation of Lemma 11, if a 0 > 2 κ 52 , there exists some τ < T such that a t ∈ ( 32 κ 52 , 2 κ 52 ) .

Original: Proof of Corollary B.3.
Revised: Proof of Corollary B.3.

Original: We will follow the notations defined in the proof of Lemma 11.
Revised: We will follow the notations defined in the proof of Lemma 11.

Original: By definition of T , P ( t ) holds for all t < T .
Revised: By definition of T , P ( t ) holds for all t < T .

Original: Then for all t < T , we have a t > κ 52 , b t < K − 1 , and a t
Revised: Then for all t < T , we have a t > κ 52 , b t < K − 1 , and a t

Original: +1 <
Revised: +1 <

Original: a t − b 2 t κ 3 .
Revised: a t − b 2 t κ 3 .

Original: Hence | a t +1 − a t | < K − 2 κ 3 < 12 κ 52 since we assumed K > 128 .
Revised: Hence | a t +1 − a t | < K − 2 κ 3 < 12 κ 52 since we assumed K > 128 .

Original: Since a 0 > 2 κ 52 and a T < κ 52 < 32 κ 52 , combining with | a t +1 − a t | < 12 κ 52 we know that there must exist some τ such that a τ ∈ ( 32 κ 52 , 2 κ 52 ) .
Revised: Since a 0 > 2 κ 52 and a T < κ 52 < 32 κ 52 , combining with | a t +1 − a t | < 12 κ 52 we know that there must exist some τ such that a τ ∈ ( 32 κ 52 , 2 κ 52 ) .

Original: Corollary B.4.
Revised: Corollary B.4.

Original: Following Lemma 11, if a 0 ∈ ( 32 κ 52 , 2 κ 52 ) , T > 1128 κ −
Revised: Following Lemma 11, if a 0 ∈ ( 32 κ 52 , 2 κ 52 ) , T > 1128 κ −

Original: Proof of Corollary B.4.
Revised: Proof of Corollary B.4.

Original: We will follow the notations defined in the proof of Lemma 11.
Revised: We will follow the notations defined in the proof of Lemma 11.

Original: By definition of T , P ( t ) holds for all t < T .
Revised: By definition of T , P ( t ) holds for all t < T .

Original: Then for all t < T , we have a t <
Revised: Then for all t < T , we have a t <

Original: a 0 = 2 κ 5and b t < 2 √ a t κ < 2 √ 2 κ 74 .
Revised: a 0 = 2 κ 5and b t < 2 √ a t κ < 2 √ 2 κ 74 .

Original: It follows that a t +1 > a t − 8 b 2 t κ 3 > a t − 64 κ 132 .
Revised: It follows that a t +1 > a t − 8 b 2 t κ 3 > a t − 64 κ 132 .

Original: Since a 0 − a T > 32 κ 52 − κ 

Revised: Since a 0 − a T > 32 κ 52 − κ

Original: = 12 κ 52 , we must have T > 12 κ 52 / 64 κ 132 = 1128 κ − 4 .
Revised: = 12 κ 52 , we must have T > 12 κ 52 / 64 κ 132 = 1128 κ − 4 .

Original: B. 
Revised: B. P HASE II: C ONVERGENCE A LONG THE P ARABOLA

Original: P HASE II: C ONVERGENCE A LONG THE P ARABOLA

Original: In Appendix B.4 we have shown that for a certain range of initializations, ( a, b ) converges close to the parabola 2 b 2 = aκ very fast.
Revised: In Appendix B.4 we have shown that for a certain range of initializations, ( a, b ) converges close to the parabola 2 b 2 = aκ very fast.

Original: In this section, we will show that ( a, b ) will slowly move along the parabola, and will eventually converge to a point with sharpness just below the EoS threshold 2 /η
Revised: In this section, we will show that ( a, b ) will slowly move along the parabola, and will eventually converge to a point with sharpness just below the EoS threshold 2 /η

Original: = 2 /κ 2 .
Revised: = 2 /κ 2 .

Original: To facilitate the analysis, we define the residual ξ ≜ b 2 − 12 aκ − 116 κ 4 and consider a small perturbation constant threshold δ = 0 . 04 .
Revised: To facilitate the analysis, we define the residual ξ ≜ b 2 − 12 aκ − 116 κ 4 and consider a small perturbation constant threshold δ = 0 . 04 .

Original: Follow from Corollary B.1 we have that for any ϵ < 0 .
Revised: Follow from Corollary B.1 we have that for any ϵ < 0 .

Original: 5 , for any κ, a, b satisfying condition B.1,
Revised: 5 , for any κ, a, b satisfying condition B.1,

Original: [Equation]
Revised: [Equation]

Original: When | b | < κ , the above expression can be further reduced to
Revised: When | b | < κ , the above expression can be further reduced to

Original: [Equation]
Revised: [Equation]

Original: Hence there exists absolute constants K such that for all ϵ < 0 .
Revised: Hence there exists absolute constants K such that for all ϵ < 0 .

Original: 5 , for all a, b, κ satisfying condition B.1 and | b | < κ , we have ξ ′′ =
Revised: 5 , for all a, b, κ satisfying condition B.1 and | b | < κ , we have ξ ′′ =

Original: (1 − 32 b 2 ) ξ + R ξ where
Revised: (1 − 32 b 2 ) ξ + R ξ where

Original: [Equation]
Revised: [Equation]

Original: Now fix δ = 0 .
Revised: Now fix δ = 0 .

Original: 04 , we will determine the regime such that | R ξ | is less than δb 2 κ 4 .
Revised: 04 , we will determine the regime such that | R ξ | is less than δb 2 κ 4 .

Original: Lemma 12.
Revised: Lemma 12.

Original: For any κ, a, b satisfying
Revised: For any κ, a, b satisfying

Original: [Equation]
Revised: [Equation]

Original: where δ = 0 .
Revised: where δ = 0 .

Original: 04 , we have
Revised: 04 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 12.
Revised: Proof of Lemma 12.

Original: Fix ϵ
Revised: Fix ϵ

Original: = 15 δK − 1 , claim that κ, b, a in the given regime satisfies condition B.1.
Revised: = 15 δK − 1 , claim that κ, b, a in the given regime satisfies condition B.1.

Original: We check the conditions one by one:
Revised: We check the conditions one by one:

Original: (i) Since we assume K > 128 , fixing ϵ = 15 δK − 1 satisfies ϵ < 0 .(ii)
Revised: (i) Since we assume K > 128 , fixing ϵ = 15 δK − 1 satisfies ϵ < 0 .(ii)

Original: With both δ and K − 1 less than 0, κ < 1 80 √ 2 δK − 1 < 0 .
Revised: With both δ and K − 1 less than 0, κ < 1 80 √ 2 δK − 1 < 0 .

Original: Also
Revised: Also

Original: [Equation]
Revised: [Equation]

Original: ϵ 14 .
Revised: ϵ 14 .

Original: Thus κ < min { 0 . 1 , ϵ 14 } .
Revised: Thus κ < min { 0 . 1 , ϵ 14 } .

Original: (iii) With ϵ = 1 5 δK − 1 and κ < 1 80 √ 2 δK − 1 , ϵκ − 1 > 16 √ 2 > 2 κ 5 2 > | a | .
Revised: (iii) With ϵ = 1 5 δK − 1 and κ < 1 80 √ 2 δK − 1 , ϵκ − 1 > 16 √ 2 > 2 κ 5 2 > | a | .

Original: Thus | a | < ϵκ − 1 .
Revised: Thus | a | < ϵκ − 1 .

Original: (iv) Since ϵκ − 1 > 16 √ 2 and κ < 1 , ϵκ − 2 / 5 > 165 √ 2 > 1 > 2 √ 2 κ 74 .
Revised: (iv) Since ϵκ − 1 > 16 √ 2 and κ < 1 , ϵκ − 2 / 5 > 165 √ 2 > 1 > 2 √ 2 κ 74 .

Original: Thus | b | < min { 1 , ϵκ − 2 / 5 } .
Revised: Thus | b | < min { 1 , ϵκ − 2 / 5 } .

Original: Thus Eq.
Revised: Thus Eq.

Original: (94) applies, and we only need to bound every term on its RHS by 15 δb 2 κ 4 to complete the proof.
Revised: (94) applies, and we only need to bound every term on its RHS by 15 δb 2 κ 4 to complete the proof.

Original: We will do that term by term.
Revised: We will do that term by term.

Original: (i) 
Revised: (i) Since

Original: Since κ < 1 80 √ 2 δK − 1 < 1 , we have κ 54 < 1 80 √ 2 δK − 1 .
Revised: κ < 1 80 √ 2 δK − 1 < 1 , we have κ 54 < 1 80 √ 2 δK − 1 .

Original: Multiplying 16 √ 2 κ 4 on both sides we have 16 √ 2 κ 214 < 15 δK − 1 κ 4 .
Revised: Multiplying 16 √ 2 κ 4 on both sides we have 16 √ 2 κ 214 < 15 δK − 1 κ 4 .

Original: Note that since | b | < 2 √ 2 κ 72 , | b 3 | < 16 √ 2 κ 214 , so | b 3 | < 15 δK − 1 κ 4 .
Revised: Note that since | b | < 2 √ 2 κ 72 , | b 3 | < 16 √ 2 κ 214 , so | b 3 | < 15 δK − 1 κ 4 .

Original: Multiplying Kb 2 on both sides gives K | b 5 | < 15 δb 2 κ 4 .
Revised: Multiplying Kb 2 on both sides gives K | b 5 | < 15 δb 2 κ 4 .

Original: (ii)
Revised: (ii)

Original: Since κ 54 < 1 80 √ 2 δK − 1 < 1 20 √ 2 δK − 1 , multiplying 4 √ 2 κ 3 on both sides we have 4 √ 2 κ 174 < 15 δK − 1 κ 3 .
Revised: Since κ 54 < 1 80 √ 2 δK − 1 < 1 20 √ 2 δK − 1 , multiplying 4 √ 2 κ 3 on both sides we have 4 √ 2 κ 174 < 15 δK − 1 κ 3 .

Original: Note that since | b | < 2 √ 2 κ 74 and | a | < 2 κ 52 , | ab | < 4 √ 2 κ 174 , we have | ab | < 15 δK − 1 κ 3 .
Revised: Note that since | b | < 2 √ 2 κ 74 and | a | < 2 κ 52 , | ab | < 4 √ 2 κ 174 , we have | ab | < 15 δK − 1 κ 3 .

Original: Multiplying Kb 2 κ on both sides gives K | ab 3 κ | < 15 δb 2 κ 4 .
Revised: Multiplying Kb 2 κ on both sides gives K | ab 3 κ | < 15 δb 2 κ 4 .

Original: (iii)
Revised: (iii)

Original: Since κ < 1 80 √ 2 δK − 1 < 1 , we have κ 3 < κ < 120 δK − 1 .
Revised: Since κ < 1 80 √ 2 δK − 1 < 1 , we have κ 3 < κ < 120 δK − 1 .

Original: Multiplying 4 κ 2 on both side gives 4 κ 5 < 15 K − 1 δκ 2 .
Revised: Multiplying 4 κ 2 on both side gives 4 κ 5 < 15 K − 1 δκ 2 .

Original: Since | a | < 2 κ 52 , we have a 2 < 4 κ 5 < 15 K − 1 δκ 2 .
Revised: Since | a | < 2 κ 52 , we have a 2 < 4 κ 5 < 15 K − 1 δκ 2 .

Original: Multiplying Kb 2 κ 2 on both side, we have Ka 2 b 2 κ 2 < 15 δb 2 κ 4 .
Revised: Multiplying Kb 2 κ 2 on both side, we have Ka 2 b 2 κ 2 < 15 δb 2 κ 4 .

Original: (iv) Since κ < 12 , 2 κ 5 2 < κ .
Revised: (iv) Since κ < 12 , 2 κ 5 2 < κ .

Original: Thus | b | < 2 κ 5 2 < κ < 1 80 √ 2 δK − 1 < 15 δK − 1 .
Revised: Thus | b | < 2 κ 5 2 < κ < 1 80 √ 2 δK − 1 < 15 δK − 1 .

Original: Multiplying Kb 2 κon both side gives K | b 3 κ 4 | < 15 δb 2 κ 4 .
Revised: Multiplying Kb 2 κon both side gives K | b 3 κ 4 | < 15 δb 2 κ 4 .

Original: (v) Since we fixed ϵ = 15 δK − 1 , Kϵb 2 κ 4 = 15 δb 2 κ 4 .
Revised: (v) Since we fixed ϵ = 15 δK − 1 , Kϵb 2 κ 4 = 15 δb 2 κ 4 .

Original: Therefore we have
Revised: Therefore we have

Original: [Equation]
Revised: [Equation]

Original: Plugging back to ξ ′′ =
Revised: Plugging back to ξ ′′ =

Original: (1 − 32 b 2 ) ξ + R ξ completes the proof.
Revised: (1 − 32 b 2 ) ξ + R ξ completes the proof.

Original: Here we restate the condition for Lemma 13:
Revised: Here we restate the condition for Lemma 13:

Original: Condition B.6.
Revised: Condition B.6.

Original: With δ = 0 . 04 and K > 512 ,
Revised: With δ = 0 . 04 and K > 512 ,

Original: [Equation]
Revised: [Equation]

Original: Corollary B.5.
Revised: Corollary B.5.

Original: For any κ, a, b satisfying condition B.6, if | ξ | > 1 16 δκ 4 , we have | ξ ′′ | < (1 − 4 κ 7 2 ) | ξ | .
Revised: For any κ, a, b satisfying condition B.6, if | ξ | > 1 16 δκ 4 , we have | ξ ′′ | < (1 − 4 κ 7 2 ) | ξ | .

Original: Proof of Corollary B.5.
Revised: Proof of Corollary B.5.

Original: Since condition B.6 holds, by Lemma 13 we have
Revised: Since condition B.6 holds, by Lemma 13 we have

Original: [Equation]
Revised: [Equation]

Original: Thus if | ξ | > 116 δκ 4 , it follows that
Revised: Thus if | ξ | > 116 δκ 4 , it follows that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Since 
Revised: Since |

Original: [Equation]
Revised: [Equation]

Original: Since 
Revised: Since | b | [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: B.5. P HASE II S TAGE
Revised: B.5. P HASE II S TAGE

Original: In this stage we will show that after | b | gets close to (cid:112) aκ/ 2 while a decreases to around 2 κ 52 from
Revised: In this stage we will show that after | b | gets close to (cid:112) aκ/ 2 while a decreases to around 2 κ 52 from

Original: Phase I of the convergence, the residual of ( b, a ) to the parabola will further decrease to below 18 δκ 4 .
Revised: Phase I of the convergence, the residual of ( b, a ) to the parabola will further decrease to below 18 δκ 4 .

Original: Lemma 13.
Revised: Lemma 13.

Original: For any κ < 1 80 √ 2 δK − 1 , for all initialization ( b 0 , a 0 ) such that a 0 ∈ ( 32 κ 52 , 2 κ 52 ) and | b 0 | ∈ ( 14 √ a 0 κ, 2 √ a 0 κ ) .
Revised: For any κ < 1 80 √ 2 δK − 1 , for all initialization ( b 0 , a 0 ) such that a 0 ∈ ( 32 κ 52 , 2 κ 52 ) and | b 0 | ∈ ( 14 √ a 0 κ, 2 √ a 0 κ ) .

Original: Let T be the time that a exits ( κ 52 , 2 κ 52 ) as characterized in Lemmaand Corollary B.4.
Revised: Let T be the time that a exits ( κ 52 , 2 κ 52 ) as characterized in Lemmaand Corollary B.4.

Original: There exists some τ < T such that ξ τ < 18 δκ 4 .
Revised: There exists some τ < T such that ξ τ < 18 δκ 4 .

Original: Proof of Lemma 13.
Revised: Proof of Lemma 13.

Original: First note that since κ < 1 80 √ 2 δK − 1 < 116 K − 1 , the initialization condition given is a subset of the valid initialization for Lemma 11.
Revised: First note that since κ < 1 80 √ 2 δK − 1 < 116 K − 1 , the initialization condition given is a subset of the valid initialization for Lemma 11.

Original: Thus for all t < T , we have a t ∈ ( κ 52 , 2 κ 52 ) and | b t | ∈ ( 14 √ a t κ, 2 √ a t κ ) .
Revised: Thus for all t < T , we have a t ∈ ( κ 52 , 2 κ 52 ) and | b t | ∈ ( 14 √ a t κ, 2 √ a t κ ) .

Original: Hence | b t | < 2 (cid:112) 2 κ 52 κ = 2 √ 2 κ 74 and | b t | > (cid:113) 14 κ 52 κ = 12 κ 74 .
Revised: Hence | b t | < 2 (cid:112) 2 κ 52 κ = 2 √ 2 κ 74 and | b t | > (cid:113) 14 κ 52 κ = 12 κ 74 .

Original: Therefore ( b t , a t ) satisfies condition B.6 and (cid:12)(cid:12)
Revised: Therefore ( b t , a t ) satisfies condition B.6 and (cid:12)(cid:12)

Original: ξ t +1 − (1 − 32 b 2 t ) ξ t (cid:12)(cid:12) <
Revised: ξ t +1 − (1 − 32 b 2 t ) ξ t (cid:12)(cid:12) <

Original: δb 2 t κ 4 .
Revised: δb 2 t κ 4 .

Original: Also note that with K > 512 and δ = 0 .
Revised: Also note that with K > 512 and δ = 0 .

Original: 05 , b 2 t < 8 κ 72 < 8 κ < 1 10 √ 2 δK − 1 < 1 .
Revised: 05 , b 2 t < 8 κ 72 < 8 κ < 1 10 √ 2 δK − 1 < 1 .

Original: For all t < T that | ξ t | > 116 δκ 4 , by Corollary B.5 we have | ξ t +1 | < (1 − 4 κ 72 )
Revised: For all t < T that | ξ t | > 116 δκ 4 , by Corollary B.5 we have | ξ t +1 | < (1 − 4 κ 72 )

Original: | ξ t | .
Revised: | ξ t | .

Original: At the initialization, we have | ξ 1 | ≤ | b 20 | + | 12 a 0 κ | + 116 κ 4 < 92 a 0 κ + 116 κ 4 < 9 κ 72 + 116 κ 4 < 10 κ 72 .
Revised: At the initialization, we have | ξ 1 | ≤ | b 20 | + | 12 a 0 κ | + 116 κ 4 < 92 a 0 κ + 116 κ 4 < 9 κ 72 + 116 κ 4 < 10 κ 72 .

Original: Thus for all τ < T such that for all t < τ, | ξ t | > 116 δκ 4 , we have | ξ τ | < (1 − 4 κ 72 ) τ 10 κ 72 .
Revised: Thus for all τ < T such that for all t < τ, | ξ t | > 116 δκ 4 , we have | ξ τ | < (1 − 4 κ 72 ) τ 10 κ 72 .

Original: Now we only need to show that | ξ τ | will decrease sufficiently fast.
Revised: Now we only need to show that | ξ τ | will decrease sufficiently fast.

Original: Consider
Revised: Consider

Original: [Equation]
Revised: [Equation]

Original: Since log(1 − 4 κ 72 ) < − 4 κ 72 by a second order Taylor expansion, we have τ <
Revised: Since log(1 − 4 κ 72 ) < − 4 κ 72 by a second order Taylor expansion, we have τ <

Original: log(80 δ − 1 κ − 12 ) / 4 κ 72 .
Revised: log(80 δ − 1 κ − 12 ) / 4 κ 72 .

Original: Substituting δ = 0 .
Revised: Substituting δ = 0 .

Original: 04 in the expression, log(80 δ − 1 κ − 12 ) = log(2000) + log( κ − 12 ) <
Revised: 04 in the expression, log(80 δ − 1 κ − 12 ) = log(2000) + log( κ − 12 ) <

Original: 8 + log( κ − 12 ) .
Revised: 8 + log( κ − 12 ) .

Original: Observe that for all x > 175 , we have 8 + log( x ) < √ x .
Revised: Observe that for all x > 175 , we have 8 + log( x ) < √ x .

Original: Since κ < 1 80 √ 2 δK − 1 < 1 512 × 2000 √ 2 < 1 175 2 , let x = κ − 12 , we have 8 + log( κ − 12 ) < κ − 14 .
Revised: Since κ < 1 80 √ 2 δK − 1 < 1 512 × 2000 √ 2 < 1 175 2 , let x = κ − 12 , we have 8 + log( κ − 12 ) < κ − 14 .

Original: Hence log(80 δ − 1 κ − 12 ) < κ − 14 and τ < κ − 14 / 4 κ 72 = 14 κ − 154 .
Revised: Hence log(80 δ − 1 κ − 12 ) < κ − 14 and τ < κ − 14 / 4 κ 72 = 14 κ − 154 .

Original: Recall from Corollary B.4 we have
Revised: Recall from Corollary B.4 we have

Original: T > 1 512 κ − 4 .
Revised: T > 1 512 κ − 4 .

Original: If we assume K > 512 , then κ < 1 512 × 2000 √ 2 < 1 32 4 , and thus κ − 1 4 > 32 and
Revised: If we assume K > 512 , then κ < 1 512 × 2000 √ 2 < 1 32 4 , and thus κ − 1 4 > 32 and

Original: T > 1512 κ − 4 > 14 κ − 154 > τ .
Revised: T > 1512 κ − 4 > 14 κ − 154 > τ .

Original: Since τ < T , a τ > κ 52 .
Revised: Since τ < T , a τ > κ 52 .

Original: If for all t < τ , | ξ t | > 116 δκ 4 , then following the analysis above, by definition of τ we have | ξ τ | < 18 δκ 4 .
Revised: If for all t < τ , | ξ t | > 116 δκ 4 , then following the analysis above, by definition of τ we have | ξ τ | < 18 δκ 4 .

Original: If there exists some t < τ that | ξ t | ≤ 116 δκ 4 , then setting τ = t directly completes the proof.
Revised: If there exists some t < τ that | ξ t | ≤ 116 δκ 4 , then setting τ = t directly completes the proof.

Original: B.5. 
Revised: B.5. P HASE II S TAGE

Original: P HASE II S TAGE

Original: In this phase, we show that once | ξ | is smaller than 18 δκ 4 , a will decrease slowly while | ξ | does not increase beyond 18 δκ 4 .
Revised: In this phase, we show that once | ξ | is smaller than 18 δκ 4 , a will decrease slowly while | ξ | does not increase beyond 18 δκ 4 .

Original: Lemma 14.
Revised: Lemma 14.

Original: For all κ < 1 80 √ 2 δK − 1 , for all initialization ( a 0 , b 0 ) satisfying a 1 ∈ ( κ 52 , 2 κ 52 ) and | ξ 1 | < 18 δκ 4 , there exists some T < 48 δ − 1 κ − 92 +1 such that a T < − 18 (1 − 3 δ ) κ 3 and for all t < T , a t > − 18 (1 − 3 δ ) κ 3 and | ξ k | < 18 δκ 4 .
Revised: For all κ < 1 80 √ 2 δK − 1 , for all initialization ( a 0 , b 0 ) satisfying a 1 ∈ ( κ 52 , 2 κ 52 ) and | ξ 1 | < 18 δκ 4 , there exists some T < 48 δ − 1 κ − 92 +1 such that a T < − 18 (1 − 3 δ ) κ 3 and for all t < T , a t > − 18 (1 − 3 δ ) κ 3 and | ξ k | < 18 δκ 4 .

Original: We will prove the claim using induction.
Revised: We will prove the claim using induction.

Original: Consider the inductive hypothesis
Revised: Consider the inductive hypothesis

Original: P 
Revised: P ( [Equation] : | ξ [Equation]

Revised: | <

Original: [Equation]
Revised: [Equation]

Original:  a
Revised: and a k ∈ ( − [Equation] [Equation] , a

Original: [Equation]
Revised: [Equation]

Original: Note that P (0) holds directly from construction, so we proceed to the inductive step.
Revised: Note that P (0) holds directly from construction, so we proceed to the inductive step.

Original: Assuming
Revised: Assuming

Original: P ( k ) holds, We will show that either a k < − 18 (1 − 3 δ ) κ 3 or P ( k + 1) holds.
Revised: P ( k ) holds, We will show that either a k < − 18 (1 − 3 δ ) κ 3 or P ( k + 1) holds.

Original: First we verify that b k , a k satisfies condition B.6.
Revised: First we verify that b k , a k satisfies condition B.6.

Original: Since δ = 0 .
Revised: Since δ = 0 .

Original: 05 , |− 18 (1 − 3 δ ) κ 3 | < | 18 κ 3 | < 2 κ 52 .
Revised: 05 , |− 18 (1 − 3 δ ) κ 3 | < | 18 κ 3 | < 2 κ 52 .

Original: Also since a 1 < 2 κ 5 2 as required, we have | a t | < max {| − 1 8 (1 − 3 δ ) κ 3 | , | a 1 − 116 δκ 7 k |} < 2 κ 5 2 .
Revised: Also since a 1 < 2 κ 5 2 as required, we have | a t | < max {| − 1 8 (1 − 3 δ ) κ 3 | , | a 1 − 116 δκ 7 k |} < 2 κ 5 2 .

Original: Since | ξ k | < 18 δκ 4 , we have b 2 k < 12 a k κ + 116 κ 4 + 18 δκ 4 < 12 (2 κ 52 )+ κ 4 < 2 κ 72 .
Revised: Since | ξ k | < 18 δκ 4 , we have b 2 k < 12 a k κ + 116 κ 4 + 18 δκ 4 < 12 (2 κ 52 )+ κ 4 < 2 κ 72 .

Original: Hence | b t | < 2 √ 2 κas required.
Revised: Hence | b t | < 2 √ 2 κas required.

Original: Therefore we have | ξ k +1 − (1 − 32 b 2 ) ξ k | < δb 2 k κ 4 .
Revised: Therefore we have | ξ k +1 − (1 − 32 b 2 ) ξ k | < δb 2 k κ 4 .

Original: Next we establish the lower bounds for | b k | , which will give lower bound for the movement of a .
Revised: Next we establish the lower bounds for | b k | , which will give lower bound for the movement of a .

Original: Since a k > − 18 (1 − 3 δ ) κ 3 and | ξ k | = | b 2 k − ( 12 a k κ + 116 κ 4 ) | < 18 δκ 4 , we have
Revised: Since a k > − 18 (1 − 3 δ ) κ 3 and | ξ k | = | b 2 k − ( 12 a k κ + 116 κ 4 ) | < 18 δκ 4 , we have

Original: [Equation]
Revised: [Equation]

Original: Since | b k | < K − 1 and | a k | < K − 1 , condition B.2 is satisfied and we have a k +1 < a k − b 2 k κ 3 < a k − 116 δκ 4 κ 3 .
Revised: Since | b k | < K − 1 and | a k | < K − 1 , condition B.2 is satisfied and we have a k +1 < a k − b 2 k κ 3 < a k − 116 δκ 4 κ 3 .

Original: Given that a k ≤ a 1 − 116 δκ 7 k by the inductive hypothesis, a k +1 ≤ a 1 − 116 δκ 7 ( k +1) .
Revised: Given that a k ≤ a 1 − 116 δκ 7 k by the inductive hypothesis, a k +1 ≤ a 1 − 116 δκ 7 ( k +1) .

Original: What remains to show for the inductive step is that | ξ k +1 | < 18 δκ 3 .
Revised: What remains to show for the inductive step is that | ξ k +1 | < 18 δκ 3 .

Original: There are two cases to consider.
Revised: There are two cases to consider.

Original: When | ξ k | ∈ ( 116 δκ 4 , 18 δκ 4 ) , by Corollary B.5 we know | ξ k +1 | < (1 − 4 κ 72 )
Revised: When | ξ k | ∈ ( 116 δκ 4 , 18 δκ 4 ) , by Corollary B.5 we know | ξ k +1 | < (1 − 4 κ 72 )

Original: | ξ k | < | ξ k | < 18 δκ 4 .
Revised: | ξ k | < | ξ k | < 18 δκ 4 .

Original: When | ξ k | ≤ 116 δκ 4 , since | ξ k +1 − (1 − 32 b 2 ) ξ k | < δb 2 k κ 4 , we have[Equation] [Equation] [Equation]
Revised: When | ξ k | ≤ 116 δκ 4 , since | ξ k +1 − (1 − 32 b 2 ) ξ k | < δb 2 k κ 4 , we have

Original: [Equation]
Revised: [Equation]

Original: Since b 2 k < 2 κ 72 < 148 , | ξ k +1 − ξ k | < 148 3 δκ 4 = 116 δκ 4 .
Revised: Since b 2 k < 2 κ 72 < 148 , | ξ k +1 − ξ k | < 148 3 δκ 4 = 116 δκ 4 .

Original: Since | ξ k | ≤ 116 δκ 4 , we have | ξ k +1 | ≤ | ξ k +1 − ξ k | + | ξ k | ≤ 18 δκ 4 as desired.
Revised: Since | ξ k | ≤ 116 δκ 4 , we have | ξ k +1 | ≤ | ξ k +1 − ξ k | + | ξ k | ≤ 18 δκ 4 as desired.

Original: In summary we have P ( k ) implies P ( k + 1) unless a k +1 < − 18 (1 − 3 δ ) κ 3 .
Revised: In summary we have P ( k ) implies P ( k + 1) unless a k +1 < − 18 (1 − 3 δ ) κ 3 .

Original: Note that there must exists some t such that a t +1 < − 18 (1 − 3 δ ) κ 3 since for any τ ≥ 48 δ − 1 κ − 92 + 1 , if the induction proceed to P ( τ ) , then a 1 − 116 δκ 7 ( τ − 1) <
Revised: Note that there must exists some t such that a t +1 < − 18 (1 − 3 δ ) κ 3 since for any τ ≥ 48 δ − 1 κ − 92 + 1 , if the induction proceed to P ( τ ) , then a 1 − 116 δκ 7 ( τ − 1) <

Original: a 1 − 3 κ 52 < − κ 52 < − 18 (1 − 3 δ ) κ 3 , which violates
Revised: a 1 − 3 κ 52 < − κ 52 < − 18 (1 − 3 δ ) κ 3 , which violates

Original: P ( τ ) .
Revised: P ( τ ) .

Original: Let T < 48 δ − 1 κ − 92 + 1 be the first t such that a t < − 18 (1 − 3 δ ) κ 3 , then by construction we have for all t < T , P ( t ) holds.
Revised: Let T < 48 δ − 1 κ − 92 + 1 be the first t such that a t < − 18 (1 − 3 δ ) κ 3 , then by construction we have for all t < T , P ( t ) holds.

Original: This completes the proof of the lemma.
Revised: This completes the proof of the lemma.

Original: Corollary B.6.
Revised: Corollary B.6.

Original: Following Lemma 14, a T − 1 ∈ ( − 18 (1 − 3 δ ) κ 3 , − 110 (1 + 2 δ ) κ 3 ) .
Revised: Following Lemma 14, a T − 1 ∈ ( − 18 (1 − 3 δ ) κ 3 , − 110 (1 + 2 δ ) κ 3 ) .

Original: Proof of Corollary B.6.
Revised: Proof of Corollary B.6.

Original: Denote T − 1 by τ , by definition of T , we know P ( τ ) holds and therefore we have b 2 τ < 2 κ 72 , a τ > − 18 (1 − 3 δ ) κ 3 and a T > a τ − 5 b 2 τ κ 3 .
Revised: Denote T − 1 by τ , by definition of T , we know P ( τ ) holds and therefore we have b 2 τ < 2 κ 72 , a τ > − 18 (1 − 3 δ ) κ 3 and a T > a τ − 5 b 2 τ κ 3 .

Original: Combining above we have a T > a τ − 10 κ 72 κ 3 , so a τ < a T + 10 κ 132 < − 18 (1 − 3 δ ) κ 3 + 10 κ 132 .
Revised: Combining above we have a T > a τ − 10 κ 72 κ 3 , so a τ < a T + 10 κ 132 < − 18 (1 − 3 δ ) κ 3 + 10 κ 132 .

Original: Note that since we set δ = 0 .
Revised: Note that since we set δ = 0 .

Original: 04 , we have
Revised: 04 , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] Since κ − 72 > K 72 > 5000 , we have 10 κ 132 < 1500 κ 3 = − 110 (1 + 2 δ ) κ 3 − ( − 18 (1 − 3 δ ) κ 3 ) .
Revised: Since κ − 72 > K 72 > 5000 , we have 10 κ 132 < 1500 κ 3 = − 110 (1 + 2 δ ) κ 3 − ( − 18 (1 − 3 δ ) κ 3 ) .

Original: Adding − 18 (1 − 3 δ ) κ 3 on both sides, we have a τ < − 18 (1 − 3 δ ) κ 3 + 10 κ 132 < − 110 (1 + 2 δ ) κ 3 .
Revised: Adding − 18 (1 − 3 δ ) κ 3 on both sides, we have a τ < − 18 (1 − 3 δ ) κ 3 + 10 κ 132 < − 110 (1 + 2 δ ) κ 3 .

Original: Combining with a τ > − 1 8 (1 − 3 δ ) κ 3 concludes the proof.
Revised: Combining with a τ > − 1 8 (1 − 3 δ ) κ 3 concludes the proof.

Original: B.5. P HASE II S TAGE
Revised: B.5. P HASE II S TAGE

Original: Here we state the lemma which proves the final convergence of the two step trajectory.
Revised: Here we state the lemma which proves the final convergence of the two step trajectory.

Original: The proof is very similar to that of Lemma 7 except a is negative now and | b | is decreasing.
Revised: The proof is very similar to that of Lemma 7 except a is negative now and | b | is decreasing.

Original: Lemma 15 (Final Convergence) .
Revised: Lemma 15 (Final Convergence) .

Original: For all κ < 116 K − 1 , for all a 0 , b 0 satisfying a 0 ∈ ( − 18 (1 − 3 δ ) κ 3 , − 110 (1 + 2 δ ) κ 3 ) and | ξ 0 | = | b 20 − 12 a 0 κ − 116 κ 4 | < 18 δκ 4 , for all ϵ > 0 , there exists some
Revised: For all κ < 116 K − 1 , for all a 0 , b 0 satisfying a 0 ∈ ( − 18 (1 − 3 δ ) κ 3 , − 110 (1 + 2 δ ) κ 3 ) and | ξ 0 | = | b 20 − 12 a 0 κ − 116 κ 4 | < 18 δκ 4 , for all ϵ > 0 , there exists some

Original: T < 25 log( ϵ − 1 ) such that for all t ≥ T , | b t | < ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: T < 25 log( ϵ − 1 ) such that for all t ≥ T , | b t | < ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: Proof.
Revised: Proof.

Original: We will prove the claim using induction.
Revised: We will prove the claim using induction.

Original: Consider the inductive hypothesis
Revised: Consider the inductive hypothesis

Original: [Equation]
Revised: [Equation]

Original: When k = 0 , the statement holds trivially, so we proceed to the inductive step.
Revised: When k = 0 , the statement holds trivially, so we proceed to the inductive step.

Original: Assume P ( k ) holds for some k ∈ N , we want to show that P ( k + 1) holds as well.
Revised: Assume P ( k ) holds for some k ∈ N , we want to show that P ( k + 1) holds as well.

Original: First we check that ( a k , b k ) with κ < 116 K − 1 satisfies condition B.4.
Revised: First we check that ( a k , b k ) with κ < 116 K − 1 satisfies condition B.4.

Original: Since (1 − 1 10 (1 + 2 δ )) < 1 , by the inductive hypothesis and the initialization condition on | b 0 | we have | b k | < | b 0 | .
Revised: Since (1 − 1 10 (1 + 2 δ )) < 1 , by the inductive hypothesis and the initialization condition on | b 0 | we have | b k | < | b 0 | .

Original: It is also from the inductive hypothesis that | a k | > | a 0 | , thus to show | b k | < 12 √ 2 (cid:112) | a k | κ , one only need to show for k = 0 case, which is equivalent to b 20 < − 18 a 0 κ .
Revised: It is also from the inductive hypothesis that | a k | > | a 0 | , thus to show | b k | < 12 √ 2 (cid:112) | a k | κ , one only need to show for k = 0 case, which is equivalent to b 20 < − 18 a 0 κ .

Original: From the initialization condition, ξ 0 = | b 20 − 12 a 0 κ − 116 κ 4 | < 18 δκ 4 .
Revised: From the initialization condition, ξ 0 = | b 20 − 12 a 0 κ − 116 κ 4 | < 18 δκ 4 .

Original: Since a 0 < − 110 (1 + 2 δ ) κ 3 , we must have 12 a 0 + 116 κ 4 < 0 ≤ b 20 .
Revised: Since a 0 < − 110 (1 + 2 δ ) κ 3 , we must have 12 a 0 + 116 κ 4 < 0 ≤ b 20 .

Original: It follows that
Revised: It follows that

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  ( Since a 0 < − 
Revised: 0 κ ( Since a 0 < − 110 (1 + 2 δ )

Original: [Equation]
Revised: [Equation]

Original: 1 + 2 δ )

Original: [Equation]
Revised: [Equation]

Original: From the initialization condition we have a 0 < − 110 (1 + 2 δ ) κ 3 = − 0 .
Revised: From the initialization condition we have a 0 < − 110 (1 + 2 δ ) κ 3 = − 0 .

Original: 108 κ 3 < − 116 κ 3 , so | a k | > | a 0 | > 116 κ 3 .
Revised: 108 κ 3 < − 116 κ 3 , so | a k | > | a 0 | > 116 κ 3 .

Original: For upper-bound on | a k | we note that a 0 > − 18 (1 − 3 δ ) κ 3 > − κ 3 by initialization, combining with the inductive hypothesis we have a k > − κ 3 − 2 κ ( | b 0 | − | b t | ) >
Revised: For upper-bound on | a k | we note that a 0 > − 18 (1 − 3 δ ) κ 3 > − κ 3 by initialization, combining with the inductive hypothesis we have a k > − κ 3 − 2 κ ( | b 0 | − | b t | ) >

Original: − κ 3 − 2 | b 0 | .
Revised: − κ 3 − 2 | b 0 | .

Original: Since b 20 <| a 0 | κ < 164 (1 − 3 δ ) κ 4 <
Revised: Since b 20 <| a 0 | κ < 164 (1 − 3 δ ) κ 4 <

Original: ( 18 κ 2 ) 2 , | b 0 | < 18 κ 2 , so a k > − κ 3 − 14 κ 3 and hence | a k | < 54 κ 3 < K − 2 κ − 1 since we assumed κ < 116 K − 1 .
Revised: ( 18 κ 2 ) 2 , | b 0 | < 18 κ 2 , so a k > − κ 3 − 14 κ 3 and hence | a k | < 54 κ 3 < K − 2 κ − 1 since we assumed κ < 116 K − 1 .

Original: Therefore we have shown that κ, a k , b
Revised: Therefore we have shown that κ, a k , b

Original: k satisfies condition B.4 and by Lemma 7 we have | b k +1 | < | b k + a k b k κ | .
Revised: k satisfies condition B.4 and by Lemma 7 we have | b k +1 | < | b k + a k b k κ | .

Original: Since 1 + a k κ > 0 as | a k | < 18 κ 3 and κ < K − 1 < 1512 , we may write the update of b k as | b k +1 | < | b k | (1+ a k κ ) .
Revised: Since 1 + a k κ > 0 as | a k | < 18 κ 3 and κ < K − 1 < 1512 , we may write the update of b k as | b k +1 | < | b k | (1+ a k κ ) .

Original: Since a k < a 0 < − 1 10 (1+2 δ ) κ 3 , we have | b k +1 | < | b k | (1 − 1 10 (1+2 δ ) κ 3 ) .
Revised: Since a k < a 0 < − 1 10 (1+2 δ ) κ 3 , we have | b k +1 | < | b k | (1 − 1 10 (1+2 δ ) κ 3 ) .

Original: Combining with the inductive hypothesis that | b k | < | b 0 | (1 − 110 (1 + 2 δ ) κ 3 ) k , we have | b k +1 | < | b 0 | (1 − 110 (1 + 2 δ ) κ 3 )
Revised: Combining with the inductive hypothesis that | b k | < | b 0 | (1 − 110 (1 + 2 δ ) κ 3 ) k , we have | b k +1 | < | b 0 | (1 − 110 (1 + 2 δ ) κ 3 )

Original: k +1 .
Revised: k +1 .

Original: Since condition B.4 is stronger than condition B.2, by Lemma 5 we have a k − a k +1 < 8 b 2 k κ 3 .
Revised: Since condition B.4 is stronger than condition B.2, by Lemma 5 we have a k − a k +1 < 8 b 2 k κ 3 .

Original: Combining with | b k | − | b k +1 | > | a k || b k | κ , we have
Revised: Combining with | b k | − | b k +1 | > | a k || b k | κ , we have

Original: [Equation]
Revised: [Equation]

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Revised: Since

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Since the inductive hypothesis gives ( a 0 − a k ) / ( | b 0 | − | b k | ) < 8 √ 2 κ , it follows by the mediant inequality that ( a 0 − a k +1 ) /
Revised: Since the inductive hypothesis gives ( a 0 − a k ) / ( | b 0 | − | b k | ) < 8 √ 2 κ , it follows by the mediant inequality that ( a 0 − a k +1 ) /

Original: ( | b 0 | − | b k +1 | ) < 8 √ 2 κ .
Revised: ( | b 0 | − | b k +1 | ) < 8 √ 2 κ .

Original: Thus we have shown P ( k ) → P ( k +1) , and by induction we know P ( k ) holds for any k ∈ N .
Revised: Thus we have shown P ( k ) → P ( k +1) , and by induction we know P ( k ) holds for any k ∈ N .

Original: Now we we can wrap up the convergence analysis leveraging this property.
Revised: Now we we can wrap up the convergence analysis leveraging this property.

Original: Since for all t , | b t | ≤ | b 0 | (1 − 110 (1+2 δ ))
Revised: Since for all t , | b t | ≤ | b 0 | (1 − 110 (1+2 δ ))

Original: k , for any ϵ > 0 we may pick T > log (1 − 110 (1+2 δ ))
Revised: k , for any ϵ > 0 we may pick T > log (1 − 110 (1+2 δ ))

Original: ( ϵ/ | b 0 | ) such that for all t > T , | b t | < ϵ .
Revised: ( ϵ/ | b 0 | ) such that for all t > T , | b t | < ϵ .

Original: Note that since | b 0 | < 18 κ 2 < 1 and 1 − 110 (1 + 2 δ ) , we have
Revised: Note that since | b 0 | < 18 κ 2 < 1 and 1 − 110 (1 + 2 δ ) , we have

Original: [Equation]
Revised: [Equation]

Original: For the region of final convergence, for any t we know that from P ( t ) that a 0 − a t ∈ [0 , 8 √ 2 κ ( | b 0 |− | b t | )) , so we have a t > a 0 − 8 √ 2 κ | b 0 | .
Revised: For the region of final convergence, for any t we know that from P ( t ) that a 0 − a t ∈ [0 , 8 √ 2 κ ( | b 0 |− | b t | )) , so we have a t > a 0 − 8 √ 2 κ | b 0 | .

Original: Since we know | b 0 | < 18 κ 2 and a 0 > − 18 κ 3 by initialization, it follows that a t > − 18 − √ 2 κ 2 > − 53 κ 3 .
Revised: Since we know | b 0 | < 18 κ 2 and a 0 > − 18 κ 3 by initialization, it follows that a t > − 18 − √ 2 κ 2 > − 53 κ 3 .

Original: The upper bound of a t < − 110 (1 + 2 δ ) κ 3 < 110 κ 3 is trivial since a is monotonically decreasing.
Revised: The upper bound of a t < − 110 (1 + 2 δ ) κ 3 < 110 κ 3 is trivial since a is monotonically decreasing.

Original: Thus in summary we have shown that for any ϵ > 0 , there exists some T < 25 log( ϵ − 1 ) such that for all t > T , | b t | <
Revised: Thus in summary we have shown that for any ϵ > 0 , there exists some T < 25 log( ϵ − 1 ) such that for all t > T , | b t | <

Original: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: B. P ROOF OF T HEOREM 3.1 AND ITS C OROLLARIES
Revised: B. P ROOF OF T HEOREM 3.1 AND ITS C OROLLARIES

Original: With all the lemmas ready, we may now prove Theorem 3.1 and its corollaries.
Revised: With all the lemmas ready, we may now prove Theorem 3.1 and its corollaries.

Original: We first restate Theorem 3.1 here.
Revised: We first restate Theorem 3.1 here.

Original: Theorem 3.1 (Sharpness Concentration) .
Revised: Theorem 3.1 (Sharpness Concentration) .

Original: For a large enough absolute constant K , suppose κ < 1 2000 √ 2 K − 1 , and the initialization ( a 0 , b 0 ) satisfies a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .
Revised: For a large enough absolute constant K , suppose κ < 1 2000 √ 2 K − 1 , and the initialization ( a 0 , b 0 ) satisfies a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .

Original: Consider the GD trajectory characterized in Eq.
Revised: Consider the GD trajectory characterized in Eq.

Original: (6) with fixed step size κfrom ( a 0 , b 0 ) , for any ϵ > 0 there exists T
Revised: (6) with fixed step size κfrom ( a 0 , b 0 ) , for any ϵ > 0 there exists T

Original: = O
Revised: = O

Original: ( K − 2 κ − 152 + log( ϵ − 1 ) + log( | b 0 | − 1 ) κ − 72 ) such that for all t > T , | b t | <
Revised: ( K − 2 κ − 152 + log( ϵ − 1 ) + log( | b 0 | − 1 ) κ − 72 ) such that for all t > T , | b t | <

Original: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: The proof for the main theorem is very simple after we have all the lemmas as discussed above.
Revised: The proof for the main theorem is very simple after we have all the lemmas as discussed above.

Original: Proof of Theorem 3.1.
Revised: Proof of Theorem 3.1.

Original: We consider any initialization ( a 0 , b 0 ) satisfying a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .
Revised: We consider any initialization ( a 0 , b 0 ) satisfying a 0 ∈ (12 κ 52 , 14 K − 2 κ − 1 ) and b 0 ∈ ( − K − 1 , K − 1 ) \{ 0 } .

Original: We abuse the notation to let a t and b t be the value of a and b after the t -th two step update from a 0 and b 0 .
Revised: We abuse the notation to let a t and b t be the value of a and b after the t -th two step update from a 0 and b 0 .

Original: If | b 0 | ≥ 2 √ a 0 κ , then by Lemma 8 there exists some τ 1 < κ − 4 such that a τ 1 ∈ (2 κ 52 , 14 K − 2 κ − 1 ) and | b τ 1 | ∈ ( √ a τ 1 κ, 2 √ a τ 1 κ ) .
Revised: If | b 0 | ≥ 2 √ a 0 κ , then by Lemma 8 there exists some τ 1 < κ − 4 such that a τ 1 ∈ (2 κ 52 , 14 K − 2 κ − 1 ) and | b τ 1 | ∈ ( √ a τ 1 κ, 2 √ a τ 1 κ ) .

Original: If | b 0 | ≤ 1 4 (cid:112) | a 0 κ | , by Lemma 9 there exists some τ 2 < 12 log( | b 0 | − 1 ) κ − 72 such that | b τ 2 | ∈
Revised: If | b 0 | ≤ 1 4 (cid:112) | a 0 κ | , by Lemma 9 there exists some τ 2 < 12 log( | b 0 | − 1 ) κ − 72 such that | b τ 2 | ∈

Original: ( 14 √ a τ 2 κ, 12 √ a τ 2 κ ) and a τ 2 ∈ (2 κ 52 , 14 K − 2 κ − 1 ) .
Revised: ( 14 √ a τ 2 κ, 12 √ a τ 2 κ ) and a τ 2 ∈ (2 κ 52 , 14 K − 2 κ − 1 ) .

Original: Thus there exists some T 1 =
Revised: Thus there exists some T 1 =

Original: O ( κ − 4 + log( | b 0 | − 1 ) κ − 72 ) such that a T 1 ∈
Revised: O ( κ − 4 + log( | b 0 | − 1 ) κ − 72 ) such that a T 1 ∈

Original: (2 κ 52 , 14 K − 2 κ − 1 ) and b T 1 ∈ ( 14 √ a T 1 κ, 2 √ a T 1 κ ) .
Revised: (2 κ 52 , 14 K − 2 κ − 1 ) and b T 1 ∈ ( 14 √ a T 1 κ, 2 √ a T 1 κ ) .

Original: Now by Lemma 11 and Corollary B.3, we know that there exists some τ 3 ≤ a 0 κ − 132 such that within τ 3 two-step updates from ( a T 1 , b T 1 ) we have a T 1 + τ 3 ∈ ( 32 κ 52 , 2 κ 52 ) and b T 1 + τ 3 ∈ ( 14 √ a T 1 + τ 3 κ, 2 √ a T 1 + τ 3 κ ) .
Revised: Now by Lemma 11 and Corollary B.3, we know that there exists some τ 3 ≤ a 0 κ − 132 such that within τ 3 two-step updates from ( a T 1 , b T 1 ) we have a T 1 + τ 3 ∈ ( 32 κ 52 , 2 κ 52 ) and b T 1 + τ 3 ∈ ( 14 √ a T 1 + τ 3 κ, 2 √ a T 1 + τ 3 κ ) .

Original: Let T 2 =
Revised: Let T 2 =

Original: T 1 + τ 3 .
Revised: T 1 + τ 3 .

Original: This completes phase 1 of convergence.
Revised: This completes phase 1 of convergence.

Original: For phase 2, since a T 2 ∈ ( 32 κ 52 , 2 κ 52 ) and | b T 2 | ∈ ( 14 √ a T 2 κ, 2 √ a T 2 κ ) , by Lemma 13, there exists τ 4 < | a T 2 | κ − 13 2 < 2 κ − 6 such that within τ 4 steps from ( a T 2 , b T 2 ) , we have a T 2 + τ 4 ∈ ( κ − 5 2 , 2 κ − 5 2 ) and | ξ T 2 + τ 4 | < 18 δκ 4 where we fixed δ = 0 .
Revised: For phase 2, since a T 2 ∈ ( 32 κ 52 , 2 κ 52 ) and | b T 2 | ∈ ( 14 √ a T 2 κ, 2 √ a T 2 κ ) , by Lemma 13, there exists τ 4 < | a T 2 | κ − 13 2 < 2 κ − 6 such that within τ 4 steps from ( a T 2 , b T 2 ) , we have a T 2 + τ 4 ∈ ( κ − 5 2 , 2 κ − 5 2 ) and | ξ T 2 + τ 4 | < 18 δκ 4 where we fixed δ = 0 .

Original: Let T 3 =
Revised: Let T 3 =

Original: T 2 + τ 4 .
Revised: T 2 + τ 4 .

Original: After the residual | ξ | decreases to less than 18 δκ 4 with T 3 two-step updates, by Lemma 14 and
Revised: After the residual | ξ | decreases to less than 18 δκ 4 with T 3 two-step updates, by Lemma 14 and

Original: Corollary B.6 there exists some τ 5 < 48 δ − 1 κ − 92 such that a T 3 + τ 5 ∈ ( − 18 (1 − 3 δ ) κ 3 , − 110 (1+2 δ ) κ 3 ) while | ξ T 3 + τ 5 | < 1 8 δκ 4 .
Revised: Corollary B.6 there exists some τ 5 < 48 δ − 1 κ − 92 such that a T 3 + τ 5 ∈ ( − 18 (1 − 3 δ ) κ 3 , − 110 (1+2 δ ) κ 3 ) while | ξ T 3 + τ 5 | < 1 8 δκ 4 .

Original: Let T 4 =
Revised: Let T 4 =

Original: T 3 + τ 5 .
Revised: T 3 + τ 5 .

Original: Finally, by Lemma 15 we have that starting from ( a T 4 , b T 4 ) , there exists some τ 6 < 25 log( ϵ − 1 )
Revised: Finally, by Lemma 15 we have that starting from ( a T 4 , b T 4 ) , there exists some τ 6 < 25 log( ϵ − 1 )

Original: that for any t > T 4 + τ 6 , | b t | <
Revised: that for any t > T 4 + τ 6 , | b t | <

Original: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: ϵ and a t ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: Thus we have the trajectory converging to some minima with a ∈ ( − 53 κ 3 , − 110 κ 3 ) .
Revised: Thus we have the trajectory converging to some minima with a ∈ ( − 53 κ 3 , − 110 κ 3 ) .

Original: Finally we bound the total number of steps required for convergence.
Revised: Finally we bound the total number of steps required for convergence.

Original: Since | a 0 | < 14 K − 2 κ − 1 , we have τ 3 < 14 K − 2 κ − 152 .
Revised: Since | a 0 | < 14 K − 2 κ − 1 , we have τ 3 < 14 K − 2 κ − 152 .

Original: Moreover, since κ < K − 1 , we have κ − 4 =
Revised: Moreover, since κ < K − 1 , we have κ − 4 =

Original: O ( K − 2 κ − 152 ) .
Revised: O ( K − 2 κ − 152 ) .

Original: Since
Revised: Since

Original: T 5 = T 1 + τ 3 + τ 4 + τ 5 + τ 6 , we have
Revised: T 5 = T 1 + τ 3 + τ 4 + τ 5 + τ 6 , we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: This completes the proof of the theorem.
Revised: This completes the proof of the theorem.

Original: B.6. P ROOF OF C OROLLARY 3. 
Revised: B.6. P ROOF OF C OROLLARY 3.

Original: Before we proceed to prove Corollary 3.1, we first show a simple lemma on the approximity of x and c ( x, y ) ≜ (cid:112) x 2 − y 2 when x is large and | 1 − xy | is small.
Revised: Before we proceed to prove Corollary 3.1, we first show a simple lemma on the approximity of x and c ( x, y ) ≜ (cid:112) x 2 − y 2 when x is large and | 1 − xy | is small.

Original: Recall that c was previously defined in Eq.
Revised: Recall that c was previously defined in Eq.

Original: (31) and the ( a, b ) coordinate that we have been focusing on is the offset from the κ 2 -EoS minima in the ( c, d ) coordinate.
Revised: (31) and the ( a, b ) coordinate that we have been focusing on is the offset from the κ 2 -EoS minima in the ( c, d ) coordinate.

Original: Lemma 16 (Approximity of c to x ) .
Revised: Lemma 16 (Approximity of c to x ) .

Original: For any large constant K > 512 , fix any κ < 1 2000 √ 2 K − 1 .
Revised: For any large constant K > 512 , fix any κ < 1 2000 √ 2 K − 1 .

Original: For any x ∈ ( √ 22 κ − 1 , 2 κ − 1 ) and any y such that | 1 − xy
Revised: For any x ∈ ( √ 22 κ − 1 , 2 κ − 1 ) and any y such that | 1 − xy

Original: | < K − 1 , we have c ( x, y ) ≜ (cid:112) x 2 − y 2 ∈
Revised: | < K − 1 , we have c ( x, y ) ≜ (cid:112) x 2 − y 2 ∈

Original: ( x − 32 κ 3 , x ) .
Revised: ( x − 32 κ 3 , x ) .

Original: Proof of Lemma 16.
Revised: Proof of Lemma 16.

Original: Since xy < 1 + K − 1 and x > √ 22 κ − 1 , we must have y <
Revised: Since xy < 1 + K − 1 and x > √ 22 κ − 1 , we must have y <

Original: 2(1 + K − 1 ) κ < 2 √ 2 κ , where the last step holds since we assumed K > 512 .
Revised: 2(1 + K − 1 ) κ < 2 √ 2 κ , where the last step holds since we assumed K > 512 .

Original: Meanwhile 1 − xy <
Revised: Meanwhile 1 − xy <

Original: K − 1 also implies xy > 1 − K − 1 > 0 , so y > 0 .
Revised: K − 1 also implies xy > 1 − K − 1 > 0 , so y > 0 .

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: where the last two inequality holds since x > √ 22 κ − 1 and √ x > x when x ∈ (0 , 1) .
Revised: where the last two inequality holds since x > √ 22 κ − 1 and √ x > x when x ∈ (0 , 1) .

Original: Thus c ( x, y ) > x − 16 xκ 4 > x − 32 κ 3 since we assume x < 2 κ − 1 .
Revised: Thus c ( x, y ) > x − 16 xκ 4 > x − 32 κ 3 since we assume x < 2 κ − 1 .

Original: Since y > 0 , (cid:112) x
Revised: Since y > 0 , (cid:112) x

Original: 2 − y 2 < x , so c ( x, y ) ∈
Revised: 2 − y 2 < x , so c ( x, y ) ∈

Original: ( x − 32 κ 3 , x ) .
Revised: ( x − 32 κ 3 , x ) .

Original: Now we can proceed to proving Corollary 3.1.
Revised: Now we can proceed to proving Corollary 3.1.

Original: We first restate the result here:
Revised: We first restate the result here:

Original: Corollary 3.1 (Sharpness Concentration under ( x, y ) -Parameterization) .
Revised: Corollary 3.1 (Sharpness Concentration under ( x, y ) -Parameterization) .

Original: For a large enough absolute constant K , suppose η < 1 8000000 K − 2 , and the initialization ( x 0 , y 0 ) satisfies x 0 ∈ (˘ x + 13 η 54 ,
Revised: For a large enough absolute constant K , suppose η < 1 8000000 K − 2 , and the initialization ( x 0 , y 0 ) satisfies x 0 ∈ (˘ x + 13 η 54 ,

Original: ˘ x + 15 K − 2 η − 12 ) and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) where (˘ x, ˘ y ) is the η -EoS minima de- fined in Definition 1.
Revised: ˘ x + 15 K − 2 η − 12 ) and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) where (˘ x, ˘ y ) is the η -EoS minima de- fined in Definition 1.

Original: The GD trajectory characterized in Eq.
Revised: The GD trajectory characterized in Eq.

Original: (2) with fixed step size η from ( x 0 , y 0 ) will converge to a global minimum with sharpness λ ∈ ( 2 η − 203 η, 2 η ) .
Revised: (2) with fixed step size η from ( x 0 , y 0 ) will converge to a global minimum with sharpness λ ∈ ( 2 η − 203 η, 2 η ) .

Original: Proof of Corollary 3.1.
Revised: Proof of Corollary 3.1.

Original: Following the convergence proof for Theorem 3.1, to prove this corollary we only need to show that all initializations x 0 , y 0 satisfies the initialization condition of Theorem 3.
Revised: Following the convergence proof for Theorem 3.1, to prove this corollary we only need to show that all initializations x 0 , y 0 satisfies the initialization condition of Theorem 3.

Original: after re-parameterized to ( a, b ) , and the sharpness λ of the minima will satisfy λ ∈ ( 2 η − 203 η, 2 η ) .
Revised: after re-parameterized to ( a, b ) , and the sharpness λ of the minima will satisfy λ ∈ ( 2 η − 203 η, 2 η ) .

Original: We first check the initialization in the ( x, y ) coordinate satisfies the initialization condition in Theorem 3.1.
Revised: We first check the initialization in the ( x, y ) coordinate satisfies the initialization condition in Theorem 3.1.

Original: Due to the different contexts, we will use κ = √ η and η itself interchangeably.
Revised: Due to the different contexts, we will use κ = √ η and η itself interchangeably.

Original: Recall from Eq.
Revised: Recall from Eq.

Original: (27) that ˘ x = √ 2 

Revised: (27) that ˘ x = √ 2

Original: (( − 4 + η − 2 ) 12 + η − 1 ) 12 .
Revised: (( − 4 + η − 2 ) 12 + η − 1 ) 12 .

Original: It is not hard to check that ˘ x > √ 2 2 κ −and ˘ x < κ − 1 where κ = √ η .
Revised: It is not hard to check that ˘ x > √ 2 2 κ −and ˘ x < κ − 1 where κ = √ η .

Original: Since | x 0 y 0 − 1 | < K − 1 as required by the initialization, from
Revised: Since | x 0 y 0 − 1 | < K − 1 as required by the initialization, from

Original: Lemma 16 we have c 0 ≜ c ( x 0 , y 0 ) ∈
Revised: Lemma 16 we have c 0 ≜ c ( x 0 , y 0 ) ∈

Original: ( x 0 − 32 κ 3 , x 0 ) .
Revised: ( x 0 − 32 κ 3 , x 0 ) .

Original: By the same reasoning we also have ˘ c ≜ c (˘ x, ˘ y ) ∈
Revised: By the same reasoning we also have ˘ c ≜ c (˘ x, ˘ y ) ∈

Original: (˘ x − 32 κ 3 , ˘ x ) .
Revised: (˘ x − 32 κ 3 , ˘ x ) .

Original: Thus x 0 > ˘ x + 13 κ 52 implies c 0 > ˘ c + 13 κ 52 − 32 κ 3 .
Revised: Thus x 0 > ˘ x + 13 κ 52 implies c 0 > ˘ c + 13 κ 52 − 32 κ 3 .

Original: Since we assume κ < 1 2000 √ 2 K − 1 where K > 512 , 32 κ 3 < κ 52 , and hence c 0 > ˘ c + 12 κ 52 .
Revised: Since we assume κ < 1 2000 √ 2 K − 1 where K > 512 , 32 κ 3 < κ 52 , and hence c 0 > ˘ c + 12 κ 52 .

Original: Therefore a 0 ≜ c 0 − ˘ c > 12 κ 5 2 .
Revised: Therefore a 0 ≜ c 0 − ˘ c > 12 κ 5 2 .

Original: On the other end, since ˘ x < κ − 1 as shown above, we have ˘ x + 14 K − 2 κ − 1 < 2 κ − 1 .
Revised: On the other end, since ˘ x < κ − 1 as shown above, we have ˘ x + 14 K − 2 κ − 1 < 2 κ − 1 .

Original: So we can again apply Lemma 16 so that c 0 < c (˘ x + 14 K − 2 η − 12 , y 0 ) <
Revised: So we can again apply Lemma 16 so that c 0 < c (˘ x + 14 K − 2 η − 12 , y 0 ) <

Original: ˘ x + 14 K − 2 η − 12 .
Revised: ˘ x + 14 K − 2 η − 12 .

Original: Since ˘ c > ˘ x − 32 κ 3 , we have c 0 < ˘ c + 15 K − 2 η − 12 + 32 κ 3 < ˘ c + 14 K − 2 η − 12 .
Revised: Since ˘ c > ˘ x − 32 κ 3 , we have c 0 < ˘ c + 15 K − 2 η − 12 + 32 κ 3 < ˘ c + 14 K − 2 η − 12 .

Original: The last step holds since 32 κ 3 < K − 2 η − 12 .
Revised: The last step holds since 32 κ 3 < K − 2 η − 12 .

Original: Therefore a 0 ≜ c 0 − ˘ c < 120 14 K − 2 η − 12 .
Revised: Therefore a 0 ≜ c 0 − ˘ c < 120 14 K − 2 η − 12 .

Original: Since | b 0 | = | 1 − xy | ∈ (0 , K − 1 ) by construction, we know ( a 0 , b 0 ) satisfies the initialization condition of Theorem 3.1, and we can have the trajectory converging to a global minima with a ∈ [ − 53 , − 110 ] and b = 0 .
Revised: Since | b 0 | = | 1 − xy | ∈ (0 , K − 1 ) by construction, we know ( a 0 , b 0 ) satisfies the initialization condition of Theorem 3.1, and we can have the trajectory converging to a global minima with a ∈ [ − 53 , − 110 ] and b = 0 .

Original: Now we show that for global minima with satisfies a k ∈ [ − 53 , − 110 ] and b k = 0 , the sharpness λ satisfies λ ∈ ( 2 η − 203 η, 2 η ) .
Revised: Now we show that for global minima with satisfies a k ∈ [ − 53 , − 110 ] and b k = 0 , the sharpness λ satisfies λ ∈ ( 2 η − 203 η, 2 η ) .

Original: Recall from Eq.
Revised: Recall from Eq.

Original: (3) that the sharpness of ( x, y ) near the global minima is given by
Revised: (3) that the sharpness of ( x, y ) near the global minima is given by

Original: [Equation]
Revised: [Equation]

Original: 110) where γ ≜ xy .
Revised: 110) where γ ≜ xy .

Original: When ( x, y ) is a global minima, γ = 1 , and Eq.
Revised: When ( x, y ) is a global minima, γ = 1 , and Eq.

Original: (110) reduces to
Revised: (110) reduces to

Original: [Equation]
Revised: [Equation]

Original: Since c = ( κ − 4 − 4) 14 + a , where a ∈ [ − 53 κ 3 , − 110 κ 3 ] , we have c < ( κ − 4 − 4) 14 and hence λ = 2 √ 4 + c 4 < 2 √ 4 + κ − 4 − 4 = 2 κ − 2 = 2 η .
Revised: Since c = ( κ − 4 − 4) 14 + a , where a ∈ [ − 53 κ 3 , − 110 κ 3 ] , we have c < ( κ − 4 − 4) 14 and hence λ = 2 √ 4 + c 4 < 2 √ 4 + κ − 4 − 4 = 2 κ − 2 = 2 η .

Original: Now we prove the lower bound for λ .
Revised: Now we prove the lower bound for λ .

Original: Follow from c > ( κ − 4 − 4) 14 − 53 κ 3 , we have
Revised: Follow from c > ( κ − 4 − 4) 14 − 53 κ 3 , we have

Original: [Equation]
Revised: [Equation]

Original:  Bernoulli inequality )
Revised: ( Bernoulli inequality )

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Hence in conclusion, the converging minima has sharpness λ ∈ ( 2 η − 203 η, 2 η ) , which completes the proof for the corollary.
Revised: Hence in conclusion, the converging minima has sharpness λ ∈ ( [Equation] η − 203 η, 2 η ) , which completes the proof for the corollary.

Original: B.6. P ROOF OF C OROLLARY 3. 
Revised: B.6. P ROOF OF C OROLLARY 3.

Original: Before proving Corollary 3.2, we first show a simple lemma on the approximity of ˘ x and κ − 1 where ˘ x is the x -coordinate for the κ 2 -EoS minima.
Revised: Before proving Corollary 3.2, we first show a simple lemma on the approximity of ˘ x and κ − 1 where ˘ x is the x -coordinate for the κ 2 -EoS minima.

Original: Lemma 17 (Approximity of ˘ x and κ − 1 ) .
Revised: Lemma 17 (Approximity of ˘ x and κ − 1 ) .

Original: For any large constant K > 512 , fix any κ < 1 2000 √ 2 K − 1 .
Revised: For any large constant K > 512 , fix any κ < 1 2000 √ 2 K − 1 .

Original: With ˘ x ≜ 1 √ 2 (( − 4 + η − 2 ) 12 + η − 1 ) 12 , we have | ˘ x − κ − 1 | < 36 κ 3 .
Revised: With ˘ x ≜ 1 √ 2 (( − 4 + η − 2 ) 12 + η − 1 ) 12 , we have | ˘ x − κ − 1 | < 36 κ 3 .

Original: Proof of Lemma 17.
Revised: Proof of Lemma 17.

Original: Since the condition for κ is identical to that of Lemma 16, we have | ˘ x − ˘ c | < 32 κ 3 from Lemma 16 where ˘ c = ( κ − 4 − 4) 14 from the calculation in Appendix B.1.
Revised: Since the condition for κ is identical to that of Lemma 16, we have | ˘ x − ˘ c | < 32 κ 3 from Lemma 16 where ˘ c = ( κ − 4 − 4) 14 from the calculation in Appendix B.1.

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original: It is straightforward that ˘ c < κ − 1 , so | ˘ c − κ − 1 | < 4 κ 3 .
Revised: It is straightforward that ˘ c < κ − 1 , so | ˘ c − κ − 1 | < 4 κ 3 .

Original: Combining with | ˘ x − ˘ c | < 32 κ 3 , we have | ˘ x − κ − 1 | <
Revised: Combining with | ˘ x − ˘ c | < 32 κ 3 , we have | ˘ x − κ − 1 | <

Original: 36 κ 3 , which completes the proof.
Revised: 36 κ 3 , which completes the proof.

Original: Now we can proceed to prove Corollary 3.2.
Revised: Now we can proceed to prove Corollary 3.2.

Original: We first restate the corollary.
Revised: We first restate the corollary.

Original: Corollary 3.2 (Sharpness Adaptivity) .
Revised: Corollary 3.2 (Sharpness Adaptivity) .

Original: For a large enough constant K , fix any α <
Revised: For a large enough constant K , fix any α <

Original: [Equation]
Revised: [Equation]

Original: [Equation] K − [Equation]
Revised: √ 2 K − 1 .

Original: For all initialization ( x 0 , y 0 ) in the region characterized by
Revised: For all initialization ( x 0 , y 0 ) in the region characterized by

Original: [Equation]
Revised: [Equation]

Original: and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) , the GD trajectory from ( x 0 , y 0 ) characterized by Eq.
Revised: and | x 0 y 0 − 1 | ∈ (0 , K − 1 ) , the GD trajectory from ( x 0 , y 0 ) characterized by Eq.

Original: (2) with any step size η ∈ ( α 2 − 110 K − 2 α 2 , α 2 ) will converge to a minima with sharpness λ ∈ ( 2 η − 203 η, 2 η ) .
Revised: (2) with any step size η ∈ ( α 2 − 110 K − 2 α 2 , α 2 ) will converge to a minima with sharpness λ ∈ ( 2 η − 203 η, 2 η ) .

Original: Proof of Corollary 3.2.
Revised: Proof of Corollary 3.2.

Original: To prove this corollary, we only need to show that for all η in the required range, the initialization region characterized by the corollary is a subset of the initialization region required by Corollary 3.1 for that particular η .
Revised: To prove this corollary, we only need to show that for all η in the required range, the initialization region characterized by the corollary is a subset of the initialization region required by Corollary 3.1 for that particular η .

Original: For the ease of derivation, we will use κ 2 to substitute for η .
Revised: For the ease of derivation, we will use κ 2 to substitute for η .

Original: Since we are dealing with different step sizes, we augment our notation to let (˘ x κ 2 , ˘ y κ 2 ) denote the κ 2 -EoS minimum.
Revised: Since we are dealing with different step sizes, we augment our notation to let (˘ x κ 2 , ˘ y κ 2 ) denote the κ 2 -EoS minimum.

Original: Note that the condition for y 0 , namely | x 0 y 0 − 1 | ∈ (0 , K − 1 ) is identical to what is required by Corollary 3.1 so we only need to show for any learning rate κ
Revised: Note that the condition for y 0 , namely | x 0 y 0 − 1 | ∈ (0 , K − 1 ) is identical to what is required by Corollary 3.1 so we only need to show for any learning rate κ

Original: [Equation]
Revised: [Equation]

Original: We will first show ˘ x η + 13 κ 52 < α − 1 + 115 K − 2 α − 1 .
Revised: We will first show ˘ x η + 13 κ 52 < α − 1 + 115 K − 2 α − 1 .

Original: Since κ 2 > α 2 − 110 K − 2 α 2 , we have κ > (1 − 110 K − 2 ) 12 α > (1 − 19 K − 2 + 1324 K − 4 ) 12 α = (1 − 116 K − 2 )
Revised: Since κ 2 > α 2 − 110 K − 2 α 2 , we have κ > (1 − 110 K − 2 ) 12 α > (1 − 19 K − 2 + 1324 K − 4 ) 12 α = (1 − 116 K − 2 )

Original: α where the last inequality holds since we may assume K > 512 .
Revised: α where the last inequality holds since we may assume K > 512 .

Original: Taking the multiplicative inverse, we have κ − 1 <
Revised: Taking the multiplicative inverse, we have κ − 1 <

Original: (1 − 118 K − 2 ) − 1 α − 1 .
Revised: (1 − 118 K − 2 ) − 1 α − 1 .

Original: Now note that since K − 1 < 1 , (1 + 116 K − 2 )(1 − 118 K − 2 ) = 1 + 1144 K − 2 − 1288 K − 4 > 1 , so (1 − 118 K − 2 ) − 1 < 1 + 116 K − 2 and hence κ − 1 < (1 − 118 K − 2 ) − 1 α − 1 <
Revised: Now note that since K − 1 < 1 , (1 + 116 K − 2 )(1 − 118 K − 2 ) = 1 + 1144 K − 2 − 1288 K − 4 > 1 , so (1 − 118 K − 2 ) − 1 < 1 + 116 K − 2 and hence κ − 1 < (1 − 118 K − 2 ) − 1 α − 1 <

Original: (1 + 116 K − 2 )
Revised: (1 + 116 K − 2 )

Original: α − 1 .
Revised: α − 1 .

Original: From
Revised: From

Original: Lemma 17 we know ˘ x κ 2 < κ − 1 + 36 κ 3 , so ˘ x κ 2 + 13 κ 52 < κ − 1 + 36 κ 3 + 13 κ 52 <
Revised: Lemma 17 we know ˘ x κ 2 < κ − 1 + 36 κ 3 , so ˘ x κ 2 + 13 κ 52 < κ − 1 + 36 κ 3 + 13 κ 52 <

Original: (1 +
Revised: (1 +

Original: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that since κ < α < 1 2000 √ 2 K − 1 where K > 512 , we have
Revised: Note that since κ < α < 1 2000 √ 2 K − 1 where K > 512 , we have

Original: [Equation]
Revised: [Equation]

Original: Thus combining with ˘ x κ 2 + 13 κ 52 < (1 + 116 K − 2 )
Revised: Thus combining with ˘ x κ 2 + 13 κ 52 < (1 + 116 K − 2 )

Original: α − 1 + 36 κ 3 + 13 κ 52 , we have ˘ x
Revised: α − 1 + 36 κ 3 + 13 κ 52 , we have ˘ x

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The other side is much simpler to show.
Revised: The other side is much simpler to show.

Original: Since κ < α , we have κ − 1 > α − 1 .
Revised: Since κ < α , we have κ − 1 > α − 1 .

Original: From Lemmawe have ˘ x κ 2 > κ − 1 − 36 κ 3 , so ˘ x κ 

Revised: From Lemmawe have ˘ x κ 2 > κ − 1 − 36 κ 3 , so ˘ x κ

Original: + 15 K − 2 κ − 1 > α − 1 − 36 κ 3 + 15 K − 2 α − 1 .
Revised: + 15 K − 2 κ − 1 > α − 1 − 36 κ 3 + 15 K − 2 α − 1 .

Original: From Eq.
Revised: From Eq.

Original: (117) we know 36 κ 3 < 1480 K − 2 <
Revised: (117) we know 36 κ 3 < 1480 K − 2 <

Original: 130 K − 2 α − 1 = ( 15 − 16 )
Revised: 130 K − 2 α − 1 = ( 15 − 16 )

Original: K − 2 α − 1 .
Revised: K − 2 α − 1 .

Original: Therefore ˘ x κ 2 + 15 K − 2 κ − 1 > α − 1 + 1 6 K − 2 α − 1 .
Revised: Therefore ˘ x κ 2 + 15 K − 2 κ − 1 > α − 1 + 1 6 K − 2 α − 1 .

Original: This concludes the proof for the corollary.
Revised: This concludes the proof for the corollary.

Original: B. O THER A UXILIARY L EMMAS B.7. C ONSTANT BOUND ON R δ ( κ )
Revised: B. O THER A UXILIARY L EMMAS B.7. C ONSTANT BOUND ON R δ ( κ )

Original: Lemma 18.
Revised: Lemma 18.

Original: For all κ < 0 .
Revised: For all κ < 0 .

Original: 1 , for all ( a, b ) ∈ ( − κ − 1 , κ − 1 ) × ( − 1 , 1) , there exists some absolute constant
Revised: 1 , for all ( a, b ) ∈ ( − κ − 1 , κ − 1 ) × ( − 1 , 1) , there exists some absolute constant

Original: K independent of a, b, κ such that
Revised: K independent of a, b, κ such that

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 18.
Revised: Proof of Lemma 18.

Original: By explicitly computing the derivatives for δ with respect to κ , we know that δ is C 6 with respect to κ and have the Taylor expansion
Revised: By explicitly computing the derivatives for δ with respect to κ , we know that δ is C 6 with respect to κ and have the Taylor expansion

Original: [Equation]
Revised: [Equation]

Original: where R δ ( κ ) is the Lagrangian remainder that R δ ( κ ) =
Revised: where R δ ( κ ) is the Lagrangian remainder that R δ ( κ ) =

Original: ∂
Revised: ∂

Original: δ ∂κ 5 ( x ) / 120 for some x ∈ [0 , κ ] .
Revised: δ ∂κ 5 ( x ) / 120 for some x ∈ [0 , κ ] .

Original: To prove the lemma we only need to bound the R δ by some absolute constants.
Revised: To prove the lemma we only need to bound the R δ by some absolute constants.

Original: For simplicity of notation, denote α ≜ (1 − 4 x 4 ) 14 , β
Revised: For simplicity of notation, denote α ≜ (1 − 4 x 4 ) 14 , β

Original: = α + ax , γ = 1 + b , and ϕ = (cid:0) 4 γ 2 x 4 + β 4 (cid:1) 12 .
Revised: = α + ax , γ = 1 + b , and ϕ = (cid:0) 4 γ 2 x 4 + β 4 (cid:1) 12 .

Original: Moreover, let
Revised: Moreover, let

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: By doing some tedious calculation we have
Revised: By doing some tedious calculation we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Since κ < 0 .
Revised: Since κ < 0 .

Original: 1 , we have α =
Revised: 1 , we have α =

Original: (1 − 4 x 4 ) 1 4 > 0 .
Revised: (1 − 4 x 4 ) 1 4 > 0 .

Original: 99 and ϕ = (cid:0) 4 γ 2 x 4 + β 4 (cid:1) 12 ≥ β 2 = ( α + dx ) 2 ≥ α 2 > 0 .
Revised: 99 and ϕ = (cid:0) 4 γ 2 x 4 + β 4 (cid:1) 12 ≥ β 2 = ( α + dx ) 2 ≥ α 2 > 0 .

Original: Thus α and ϕ are bounded from below by some constants.
Revised: Thus α and ϕ are bounded from below by some constants.

Original: Since κ is bounded from above, we have ρ 1 , . . ρ
Revised: Since κ is bounded from above, we have ρ 1 , . . .

Original: Now we give upper bounds for β and γ .
Revised: Now we give upper bounds for β and γ .

Original: Since | x | < 0 . 1 , we have (1 − 4 x 4 ) 14 ≤ 1 .
Revised: Since | x | < 0 . 1 , we have (1 − 4 x 4 ) 14 ≤ 1 .

Original: Since | a | < κ − 1 , we have β = α + ax < α + aκ < α + 1 < 2 .
Revised: Since | a | < κ − 1 , we have β = α + ax < α + aκ < α + 1 < 2 .

Original: Since | b | < 1 , we have γ = 1 + b < 2 .
Revised: Since | b | < 1 , we have γ = 1 + b < 2 .

Original: Note that it is straightforward from construction that all these terms are positive.
Revised: Note that it is straightforward from construction that all these terms are positive.

Original: From Eq.
Revised: From Eq.

Original: (122) we have ∂ 

Revised: (122) we have ∂

Original: δ ∂κ 6 ( x ) as some finite degree polynomial of β, γ, κ, ρ 1 , . . .
Revised: δ ∂κ 6 ( x ) as some finite degree polynomial of β, γ, κ, ρ 1 , . . .

Original: , ρ 6 , and ϕ − 1 .
Revised: , ρ 6 , and ϕ − 1 .

Original: Since the absolute value for all of these terms are bounded above by some constants independent of x , we have (cid:12)(cid:12)(cid:12) ∂ 6 δ ∂κ 6 ( x ) (cid:12)(cid:12)(cid:12) uniformly bounded above by some constant K for all x ∈ [0 , κ ] .
Revised: Since the absolute value for all of these terms are bounded above by some constants independent of x , we have (cid:12)(cid:12)(cid:12) ∂ 6 δ ∂κ 6 ( x ) (cid:12)(cid:12)(cid:12) uniformly bounded above by some constant K for all x ∈ [0 , κ ] .

Original: This completes the proof of the lemma.
Revised: This completes the proof of the lemma.

Original: C T HEORETICAL A NALYSIS ON R ANK -1 A PPROXIMATION OF I SOTROPIC M ATRIX
Revised: C T HEORETICAL A NALYSIS ON R ANK -1 A PPROXIMATION OF I SOTROPIC M ATRIX

Original: In this section we prove the convergence of the vector case.
Revised: In this section we prove the convergence of the vector case.

Original: C. P RELIMINARIES
Revised: C. P RELIMINARIES

Original: Model: We consider a generalized model from the scalar product case.
Revised: Model: We consider a generalized model from the scalar product case.

Original: [Equation]
Revised: [Equation]

Original: The normalization factor 14 is added to show the equivalence of this rank-1 isotropic matrix factorization problem and the scalar case considered in Appendix B.6.
Revised: The normalization factor 14 is added to show the equivalence of this rank-1 isotropic matrix factorization problem and the scalar case considered in Appendix B.6.

Original: We will show how the equivalence is achieved due to the alignment in the next section.
Revised: We will show how the equivalence is achieved due to the alignment in the next section.

Original: Update Rule: We use gradient descent to optimize this problem.
Revised: Update Rule: We use gradient descent to optimize this problem.

Original: By computing the closed-form of the gradient, we know the 1-step update of x and y follows:
Revised: By computing the closed-form of the gradient, we know the 1-step update of x and y follows:

Original: [Equation]
Revised: [Equation]

Original: With the gradient descent update above
Revised: With the gradient descent update above

Original: [Equation]
Revised: [Equation]

Original: we can have the dynamics of the following four quantities:
Revised: we can have the dynamics of the following four quantities:

Original: ∥ x ∥ 2 , ∥ y ∥ 2 , and ( x ⊤ y ) 2 .
Revised: ∥ x ∥ 2 , ∥ y ∥ 2 , and ( x ⊤ y ) 2 .

Original: [Equation]
Revised: [Equation]

Original: Furthermore
Revised: Furthermore

Original: [Equation]
Revised: [Equation]

Original: we define an alignment notation for the variable: ξ t := ∥ x t ∥ 2 ∥∥ y t ∥ 2 − ( x ⊤ t y t ) 2 .
Revised: we define an alignment notation for the variable: ξ t := ∥ x t ∥ 2 ∥∥ y t ∥ 2 − ( x ⊤ t y t ) 2 .

Original: By Eq.
Revised: By Eq.

Original: (126) and Eq.
Revised: (126) and Eq.

Original: [Equation]

Original:  we can derive the following update rule of ξ .
Revised: (125), we can derive the following update rule of ξ .

Original: [Equation]
Revised: [Equation]

Original: ) (cid:19)(cid:19)(127)
Revised: (cid:19)(cid:19)(127)

Original: C. P ROOF OF T HEOREM 3. 
Revised: C. P ROOF OF T HEOREM 3.

Original: We restate the theorem of the vector case.
Revised: We restate the theorem of the vector case.

Original: Theorem C.1.
Revised: Theorem C.1.

Original: For a large enough absolute constant K , with all the initialization ( x 0 , y 0 ) satisfying x 0 ∼ δ x Unif ( S d − 1 ) , y 0 ∼ δ y Unif ( S d − 1 ) , δ x δ y
Revised: For a large enough absolute constant K , with all the initialization ( x 0 , y 0 ) satisfying x 0 ∼ δ x Unif ( S d − 1 ) , y 0 ∼ δ y Unif ( S d − 1 ) , δ x δ y

Original: =
Revised: =

Original: 2 , δ x ∈ (˘ x + 1 80 K − 2 η − 1 2 , ˘ x + 1 8 K − 2 η − 1 2 ) , if step size η < min { K − 4 8000000 ,
Revised: 2 , δ x ∈ (˘ x + 1 80 K − 2 η − 1 2 , ˘ x + 1 8 K − 2 η − 1 2 ) , if step size η < min { K − 4 8000000 ,

Original: K − 2 20000+2000(log( d ) − log( δ 0 )) } , and a multiplicative perturbation y ′ t =
Revised: K − 2 20000+2000(log( d ) − log( δ 0 )) } , and a multiplicative perturbation y ′ t =

Original: y t (1 + 2 K − 1 ) 3 is performed at time t = t p for some t p >
Revised: y t (1 + 2 K − 1 ) 3 is performed at time t = t p for some t p >

Original: O ( − log( η ) + log( d ) − log( δ 0 )
Revised: O ( − log( η ) + log( d ) − log( δ 0 )

Original: + K 3 ) , then for any ϵ > 0 , with probability p > 1 − 2 δ 0 − 2 exp {− Ω( d ) } there exists T
Revised: + K 3 ) , then for any ϵ > 0 , with probability p > 1 − 2 δ 0 − 2 exp {− Ω( d ) } there exists T

Original: = O
Revised: = O

Original: ( K − 2 κ − 152 − log( ϵ ) − log( δ 0 ))
Revised: ( K − 2 κ − 152 − log( ϵ ) − log( δ 0 ))

Original: such that for all t > T , L ( x, y ) <
Revised: such that for all t > T , L ( x, y ) <

Original: ϵ and ∥ x t ∥ 2 + ∥
Revised: ϵ and ∥ x t ∥ 2 + ∥

Original: y t ∥ 2 ∈ ( 1 η − 103 η, 1 η ) .
Revised: y t ∥ 2 ∈ ( 1 η − 103 η, 1 η ) .

Original: In the following analysis, we still use the operator O ( · ) to only hides absolute constants, and K to represent the absolute constant that uniformly upper bounds all the absolute constants of the O ( · ) terms.
Revised: In the following analysis, we still use the operator O ( · ) to only hides absolute constants, and K to represent the absolute constant that uniformly upper bounds all the absolute constants of the O ( · ) terms.

Original: Also K > 512 .
Revised: Also K > 512 .

Original: We still use the notation of η -EoS minimum (˘ x, ˘ y ) of the scalar case in the vector case.
Revised: We still use the notation of η -EoS minimum (˘ x, ˘ y ) of the scalar case in the vector case.

Original: We first prove some properties of the global minimizers.
Revised: We first prove some properties of the global minimizers.

Original: The following lemma guarantees the alignment of the two vectors x , y at any global minimizer ( x , y ) .
Revised: The following lemma guarantees the alignment of the two vectors x , y at any global minimizer ( x , y ) .

Original: Lemma 19.
Revised: Lemma 19.

Original: (Global minimizers)
Revised: (Global minimizers)

Original: For all global minimizers ( x , y ) of the optimization problem
Revised: For all global minimizers ( x , y ) of the optimization problem

Original: Eq.
Revised: Eq.

Original: (123) , we have x = c y for some c ∈ R , and ∥ x ∥∥ y ∥
Revised: (123) , we have x = c y for some c ∈ R , and ∥ x ∥∥ y ∥

Original: = 1 .
Revised: = 1 .

Original: Proof.
Revised: Proof.

Original: We directly consider the objective (cid:13)(cid:13)
Revised: We directly consider the objective (cid:13)(cid:13)

Original: I − [Equation] ⊤ [Equation]
Revised: I − xy ⊤ xy ⊤ (cid:13)(cid:13) 2 F .

Original: ⊤

Original: [Equation]
Revised: [Equation]

Original: F .

Original: [Equation]

Original: The global minimizer takes value d − 14 and the equality holds when
Revised: The global minimizer takes value d − 14 and the equality holds when

Original: [Equation]
Revised: [Equation]

Original: which is equivalent to x = c y for some c ∈ R , and ∥ x ∥∥ y ∥
Revised: which is equivalent to x = c y for some c ∈ R , and ∥ x ∥∥ y ∥

Original: = 1 .
Revised: = 1 .

Original: Now we consider the formal proof.
Revised: Now we consider the formal proof.

Original: To prove the theorem we have four steps: (i) we prove the two vectors x , y will decay geometrically after T
Revised: To prove the theorem we have four steps: (i) we prove the two vectors x , y will decay geometrically after T

Original: = O ( − log( η ) − log( δ 0 )
Revised: = O ( − log( η ) − log( δ 0 )

Original: + log( d )) time (Lemma 20,
Revised: + log( d )) time (Lemma 20,

Original: Lemma 21); (ii) we prove the norm of the two vectors ∥ x ∥ , ∥ y ∥ will satisfy the initialization condi- tion in T ′ <
Revised: Lemma 21); (ii) we prove the norm of the two vectors ∥ x ∥ , ∥ y ∥ will satisfy the initialization condi- tion in T ′ <

Original: 6 K 3 + 4 K time (Lemma 22);(iii)
Revised: 6 K 3 + 4 K time (Lemma 22);(iii)

Original: To escape from some sharp minima, we add some deterministic perturbation, and we prove that after the perturbation gradient descent will re-enter the feasible regime, while at the same time keep a constant distance to the manifold of the minimizers; (iv) after entering the feasible region and ξ is small enough, the re-parameterized dynamics of ∥ x ∥ and ∥
Revised: To escape from some sharp minima, we add some deterministic perturbation, and we prove that after the perturbation gradient descent will re-enter the feasible regime, while at the same time keep a constant distance to the manifold of the minimizers; (iv) after entering the feasible region and ξ is small enough, the re-parameterized dynamics of ∥ x ∥ and ∥

Original: y ∥ can be captured by the scalar dynamics (Lemma 25).
Revised: y ∥ can be captured by the scalar dynamics (Lemma 25).

Original: Then, we can reduce this vector case to the scalar product case and finish the proof.
Revised: Then, we can reduce this vector case to the scalar product case and finish the proof.

Original: Here we present the following key lemmas to prove the convergence at the edge of stability.
Revised: Here we present the following key lemmas to prove the convergence at the edge of stability.

Original: For simplicity, we denote α ( t ) :
Revised: For simplicity, we denote α ( t ) :

Original: = η ( ∥ x t ∥ 2 + ∥ y t ∥ 2 ) − 1 .
Revised: = η ( ∥ x t ∥ 2 + ∥ y t ∥ 2 ) − 1 .

Original: Lemma 20.
Revised: Lemma 20.

Original: (Alignment) Assume 

Revised: (Alignment) Assume

Original: < α (0) < 18 , 0 < x ⊤ 0 y 0 <
Revised: < α (0) < 18 , 0 < x ⊤ 0 y 0 <

Original: ∥ x 0 ∥∥ y 

Revised: ∥ x 0 ∥∥ y

Original: ∥ ≤ 12 , η < 0 .
Revised: ∥ ≤ 12 , η < 0 .

Original: Then as long as − 1100 <
Revised: Then as long as − 1100 <

Original: α ( t ) < 18 holds for t ∈ [0 , t 1 ] for some t 1 , then for all t ∈ [0 , t 1 + 1] , 0 < ( x ⊤ t y t ) 2 < 1310 and ξ t +1 < ξ t ; Moreover, there exists some t 0 < log 2 ( 7 20 x ⊤ 0 y 0 ) , for all t ∈ [ t 0 , t 1 + 1] , 720 < ( x ⊤ t y t ) 2 < 1310 and ξ t +1 < 0 .
Revised: α ( t ) < 18 holds for t ∈ [0 , t 1 ] for some t 1 , then for all t ∈ [0 , t 1 + 1] , 0 < ( x ⊤ t y t ) 2 < 1310 and ξ t +1 < ξ t ; Moreover, there exists some t 0 < log 2 ( 7 20 x ⊤ 0 y 0 ) , for all t ∈ [ t 0 , t 1 + 1] , 720 < ( x ⊤ t y t ) 2 < 1310 and ξ t +1 < 0 .

Original: 7 ξ t .
Revised: 7 ξ t .

Original: Proof.
Revised: Proof.

Original: We use induction.
Revised: We use induction.

Original: For iteration 0, the induction basis holds.
Revised: For iteration 0, the induction basis holds.

Original: Now we consider iteration t +when assuming the conclusion holds at iteration 0 , 1 , 2 , ..., t .
Revised: Now we consider iteration t +when assuming the conclusion holds at iteration 0 , 1 , 2 , ..., t .

Original: Consider Equation (126).
Revised: Consider Equation (126).

Original: We denote c t =
Revised: We denote c t =

Original: ( x ⊤ t y t ) 2 .
Revised: ( x ⊤ t y t ) 2 .

Original: And from the induction hypothesis, ξ t < ∥ x 0 ∥ 2 ∥ y 

Revised: And from the induction hypothesis, ξ t < ∥ x 0 ∥ 2 ∥ y

Original: ∥ 2 < 1 / 4 for all time.
Revised: ∥ 2 < 1 / 4 for all time.

Original: Thus ∥ x t ∥ 

Revised: Thus ∥ x t ∥

Original: ∥
Revised: ∥

Original: y t ∥ 2 =
Revised: y t ∥ 2 =

Original: c t + ξ t < 3120 .
Revised: c t + ξ t < 3120 .

Original: Then we have c t +1 <
Revised: Then we have c t +1 <

Original: c t (1 + (1 + α ( t ))
Revised: c t (1 + (1 + α ( t ))

Original: (1 − c t )
Revised: (1 − c t )

Original: + 20 η 2 ) 2 < c t ( 1001 

Revised: + 20 η 2 ) 2 < c t ( 1001

Original: + α ( t ) − (1 + α ( t ))
Revised: + α ( t ) − (1 + α ( t ))

Original: c t ) 2 .
Revised: c t ) 2 .

Original: Since 0 < c t < 13 / 10 and − 0 .
Revised: Since 0 < c t < 13 / 10 and − 0 .

Original: 01 <
Revised: 01 <

Original: α < 1 / 8 , this function takes maximal value when c t =
Revised: α < 1 / 8 , this function takes maximal value when c t =

Original: [Equation]
Revised: [Equation]

Original: 1+ α [Equation] t )) .
Revised: 1+ α ( t )) .

Original: [Equation]
Revised: [Equation]

Original: The lower bound 0 of c t is straightforward since we have
Revised: The lower bound 0 of c t is straightforward since we have

Original: [Equation]
Revised: [Equation]

Original: due to c t + ξ t / 2 < 31 / 20 and − 0 .
Revised: due to c t + ξ t / 2 < 31 / 20 and − 0 .

Original: 01 <
Revised: 01 <

Original: α ( t ) < 1 / 8 .
Revised: α ( t ) < 1 / 8 .

Original: Then we consider the tighter bound and a faster decaying rate after some t 0 .
Revised: Then we consider the tighter bound and a faster decaying rate after some t 0 .

Original: If c 0 > 7 / 20 and the lower bound holds at t = 0 , then t 0 = 0 < log 2 ( 7 20 x ⊤ 0 y 0 ) .
Revised: If c 0 > 7 / 20 and the lower bound holds at t = 0 , then t 0 = 0 < log 2 ( 7 20 x ⊤ 0 y 0 ) .

Original: Then we begin the induction from t 0 .
Revised: Then we begin the induction from t 0 .

Original: Consider t ≥ t 0 , 
Revised: Consider t ≥ t 0 , ξ

Original: [Equation]
Revised: [Equation]

Original: and the lower bound of c t +1 becomes (since 7 / 20 < c t < 1 . 3 )
Revised: and the lower bound of c t +1 becomes (since 7 / 20 < c t < 1 . 3 )

Original: c 
Revised: c t +1 > min { c t ( [Equation]

Original: [Equation]
Revised: [Equation]

Original: >min
Revised: >

Original: [Equation]
Revised: [Equation]

Original: > 7 / 20 .
Revised: > 7 / 20 .

Original: The last inequality is because of the monotonic decrement of the last function, which takes minimal value at c t = 1310 .
Revised: The last inequality is because of the monotonic decrement of the last function, which takes minimal value at c t = 1310 .

Original: This proves the first statement.
Revised: This proves the first statement.

Original: Then consider the alignment dynamics by Equation (127).
Revised: Then consider the alignment dynamics by Equation (127).

Original: The factor can be bounded as follow:
Revised: The factor can be bounded as follow:

Original: [Equation]
Revised: [Equation]

Original: By induction, we finish the proof.
Revised: By induction, we finish the proof.

Original: If c 0 ≤ 7 / 20 , then we prove that c t will eventually become larger than 7 / 20 after some time t 0 , and then apply the same induction process above to finish the proof.
Revised: If c 0 ≤ 7 / 20 , then we prove that c t will eventually become larger than 7 / 20 after some time t 0 , and then apply the same induction process above to finish the proof.

Original: Still we consider the lower bound of the dynamics of c t .
Revised: Still we consider the lower bound of the dynamics of c t .

Original: If we have c t < 7 / 20 , 
Revised: If we have c t < 7 / 20 , c

Original: [Equation]
Revised: [Equation]

Revised: >

Revised: [Equation]

Revised: 1 +

Revised: [Equation]

Revised: [Equation]

Original: Then it at most takes t 0 = ⌈ log 2 ( 7 20 x
Revised: t − [Equation] [Equation] >

Original: [Equation]
Revised: [Equation]

Revised: 1 +

Original:  y 0 )
Revised: [Equation] [Equation] c t − 1 / 8)) 2 > 2 c t Then it at most takes t 0 = ⌈ log 2 ( 7 20 x ⊤ 0 y 0 )

Original: ⌉ to satisfy the condition.
Revised: ⌉ to satisfy the condition.

Original: Now we finish the proof.
Revised: Now we finish the proof.

Original: We first consider the time ξ takes to become smaller than η 2 .
Revised: We first consider the time ξ takes to become smaller than η 2 .

Original: Lemma 21.
Revised: Lemma 21.

Original: (Alignment convergence time)
Revised: (Alignment convergence time)

Original: Suppose all the conditions in Theorem C.1 holds.
Revised: Suppose all the conditions in Theorem C.1 holds.

Original: Then with probability 1 − 2 δ 0 − 2 exp {− Ω( d ) } , there exists some time T 0 = log 0 .
Revised: Then with probability 1 − 2 δ 0 − 2 exp {− Ω( d ) } , there exists some time T 0 = log 0 .

Original: 7 ( η 2 ) + log 2 ( 21 d 20 δ 20 ) =
Revised: 7 ( η 2 ) + log 2 ( 21 d 20 δ 20 ) =

Original: O ( − log( η ) + log( d ) − log( δ 0 ))
Revised: O ( − log( η ) + log( d ) − log( δ 0 ))

Original: such that ξ t < η 2 and ∥ x t ∥ ∈ (˘ x + 1200 K − 2 η − 12 , ˘ x + 16 K − 2 η − 12 ) for t ∈ [ T 0 , T 0 + 6 K 3 + 4 K ] .
Revised: such that ξ t < η 2 and ∥ x t ∥ ∈ (˘ x + 1200 K − 2 η − 12 , ˘ x + 16 K − 2 η − 12 ) for t ∈ [ T 0 , T 0 + 6 K 3 + 4 K ] .

Original: Proof.
Revised: Proof.

Original: We first prove that the bound of ( x ⊤ y ) 2 can be reached with probability 1 − 2 δ 0 − 2 exp {− Ω( d ) } , and meanwhile the alignment ξ begins to shrink after t 0 = ⌈ log 2 ( 21 d 20 δ 20 ) ⌉ .
Revised: We first prove that the bound of ( x ⊤ y ) 2 can be reached with probability 1 − 2 δ 0 − 2 exp {− Ω( d ) } , and meanwhile the alignment ξ begins to shrink after t 0 = ⌈ log 2 ( 21 d 20 δ 20 ) ⌉ .

Original: For the initialization, δ x δ y = 12 , and by symmetry we know
Revised: For the initialization, δ x δ y = 12 , and by symmetry we know

Original: [Equation]
Revised: [Equation]

Original: It is equivalent to consider sampling from N (0 , I ) and then divide it by its norm.
Revised: It is equivalent to consider sampling from N (0 , I ) and then divide it by its norm.

Original: By the initialization condition, we apply the Gaussian concentration bound and Theorem 3.1.1 in Vershynin (2018) and get
Revised: By the initialization condition, we apply the Gaussian concentration bound and Theorem 3.1.1 in Vershynin (2018) and get

Original: [Equation]
Revised: [Equation]

Original: Then with probability p > 1 − 2 δ 0 − 2 exp {− Ω( d ) } , δ 20 d ≤ ( x ⊤ 0 y 0 ) 2 < 12 .
Revised: Then with probability p > 1 − 2 δ 0 − 2 exp {− Ω( d ) } , δ 20 d ≤ ( x ⊤ 0 y 0 ) 2 < 12 .

Original: During t ∈ [0 , T +6 K 3 + 4 K ] , we can apply the first argument in Lemma 20 and induction to prove that ( x ⊤ t y t ) 2 < 1310 and 0 < α ( t ) <
Revised: During t ∈ [0 , T +6 K 3 + 4 K ] , we can apply the first argument in Lemma 20 and induction to prove that ( x ⊤ t y t ) 2 < 1310 and 0 < α ( t ) <

Original: 13 K − 2 50 .
Revised: 13 K − 2 50 .

Original: For t = 0 , since δ x ∈ (˘ x + 180 K − 2 η − 12 , ˘ x + 18 K − 2 η − 12 ) , 140 K − 2 < α (0) <
Revised: For t = 0 , since δ x ∈ (˘ x + 180 K − 2 η − 12 , ˘ x + 18 K − 2 η − 12 ) , 140 K − 2 < α (0) <

Original: 14 K − 2 < 18 and ( x ⊤ 0 y 0 ) 2 <
Revised: 14 K − 2 < 18 and ( x ⊤ 0 y 0 ) 2 <

Original: ∥ x 0 ∥ 2 ∥ y 

Revised: ∥ x 0 ∥ 2 ∥ y

Original: ∥ 2 = 14 < 1310 .
Revised: ∥ 2 = 14 < 1310 .

Original: Then we suppose for time t ∈ [0 , t 1 − 1] the statement is correct.
Revised: Then we suppose for time t ∈ [0 , t 1 − 1] the statement is correct.

Original: By the induction hypothesis, we know for t ∈ [0 , t 1 − 1] the condition of Lemma 20 holds.
Revised: By the induction hypothesis, we know for t ∈ [0 , t 1 − 1] the condition of Lemma 20 holds.

Original: Therefore, by Lemma 20, for all t
Revised: Therefore, by Lemma 20, for all t

Original: [Equation]
Revised: [Equation]

Original: With the upper bound of ( x ⊤ t y t ) 2 , the one step movement of ∥ x ∥ 2 + ∥ y ∥ 2 can be bounded:
Revised: With the upper bound of ( x ⊤ t y t ) 2 , the one step movement of ∥ x ∥ 2 + ∥ y ∥ 2 can be bounded:

Original: [Equation]
Revised: [Equation]

Original: For t < T 0 + 6 K 3 + 4 K , the total movement of ∥ x t ∥ 2 + ∥
Revised: For t < T 0 + 6 K 3 + 4 K , the total movement of ∥ x t ∥ 2 + ∥

Original: y t ∥ 2 is smaller than 5 η ( T + 6 K 3 + 4 K ) , and the corresponding movement of α ( t ) is smaller than 5 η 2 ( T + 6 K 3 + 4 K ) <
Revised: y t ∥ 2 is smaller than 5 η ( T + 6 K 3 + 4 K ) , and the corresponding movement of α ( t ) is smaller than 5 η 2 ( T + 6 K 3 + 4 K ) <

Original: K − 2 200 (since η < min { K − 4 8000000 ,
Revised: K − 2 200 (since η < min { K − 4 8000000 ,

Revised: [Equation]

Revised: log

Original: [Equation]
Revised: [Equation]

Original: Thus by induction, ( x ⊤ t y t ) 2 < 1310 and 0 < α ( t ) <
Revised: Thus by induction, ( x ⊤ t y t ) 2 < 1310 and 0 < α ( t ) <

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Revised: +

Revised: [Equation]

Original: < 13 K − 2 50 .
Revised: < 13 K − 2 50 .

Original: Then by the second argument of Lemma 20, we have 720 < ( x ⊤ t y t ) 2 <and ξ t +1 < 0 .
Revised: Then by the second argument of Lemma 20, we have 720 < ( x ⊤ t y t ) 2 <and ξ t +1 < 0 .

Original: 7 ξ t for t > log 2 ( 21 d 20 δ 20 )
Revised: 7 ξ t for t > log 2 ( 21 d 20 δ 20 )

Original: =
Revised: =

Original: O (log( d ) − log( δ 0 )) .
Revised: O (log( d ) − log( δ 0 )) .

Original: After t = ⌈ log 2 ( 21 d 20 δ 20 ) ⌉ , we can calculate the time when ξ t < η 2 .
Revised: After t = ⌈ log 2 ( 21 d 20 δ 20 ) ⌉ , we can calculate the time when ξ t < η 2 .

Original: It needs at most log 0 .
Revised: It needs at most log 0 .

Original: 7 ( η 2 ) =
Revised: 7 ( η 2 ) =

Original: −O (log( η )) for ξ t to become smaller than η 2 .
Revised: −O (log( η )) for ξ t to become smaller than η 2 .

Original: Now we know with T 0 = log 0 .
Revised: Now we know with T 0 = log 0 .

Original: 7 ( η 2 ) + log 2 ( 21 d 20 δ 20 ) =
Revised: 7 ( η 2 ) + log 2 ( 21 d 20 δ 20 ) =

Original: O ( − log( η ) + log( d ) − log( δ 0 )) , within t ∈ [ T 0 , T 0 + 6 K 3 + 4 K ] , ξ t < η 2 always holds.
Revised: O ( − log( η ) + log( d ) − log( δ 0 )) , within t ∈ [ T 0 , T 0 + 6 K 3 + 4 K ] , ξ t < η 2 always holds.

Original: On the other hand, we can also have the lower bound of the ∥ x t ∥ 2 + ∥
Revised: On the other hand, we can also have the lower bound of the ∥ x t ∥ 2 + ∥

Original: y t ∥ 2 .
Revised: y t ∥ 2 .

Original: The one step movement has the lower bound
Revised: The one step movement has the lower bound

Original: [Equation]
Revised: [Equation]

Original: − 1)+ (2 −3 4 ∥ x t ∥ 2 ∥ y t ∥ 2 )( ∥ x t ∥ 2 ∥∥ y t ∥
Revised: 1)+ (2 −

Original: [Equation]
Revised: [Equation]

Original: − ( x⊤ t y t ) 2 ))) +
Revised: x

Original: [Equation]
Revised: [Equation]

Original: η ( x ⊤
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original:  y [Equation]
Revised: ∥ y t ∥

Original: [Equation]
Revised: [Equation]

Original: x

Original:  [Equation]
Revised: ∥∥ y t ∥ [Equation] [Equation]

Revised: +

Revised: [Equation]

Original: ∥
Revised: ∥

Original: [Equation]
Revised: [Equation]

Original: ∥ y t ∥ 2 ) > 0 − 2 η > − 5 η.
Revised: ) > 0 − 2 η > − 5 η.

Original: So for t < T 0 + 6 K 3 + 4 K , the total decrement of ∥ x t ∥ 2 + ∥
Revised: So for t < T 0 + 6 K 3 + 4 K , the total decrement of ∥ x t ∥ 2 + ∥

Original: y t ∥ 2 is larger than − 5 η
Revised: y t ∥ 2 is larger than − 5 η

Original: ( T 0 + 6 K 3 + 4 K ) > − K − 2 200 η − 12 .
Revised: ( T 0 + 6 K 3 + 4 K ) > − K − 2 200 η − 12 .

Original: Because x ⊤ t y t ∈ ( 720 , 1310 ) , ∥ y t ∥ 2 < 1310 / ∥ x t ∥ 2 < 1 1000 K − 2 η − 12 .
Revised: Because x ⊤ t y t ∈ ( 720 , 1310 ) , ∥ y t ∥ 2 < 1310 / ∥ x t ∥ 2 < 1 1000 K − 2 η − 12 .

Original: Therefore, ∥ x t ∥ 2 > 1200 K − 2 η − 12 .
Revised: Therefore, ∥ x t ∥ 2 > 1200 K − 2 η − 12 .

Original: After we have the alignment guarantee, we can prove that we will enter the feasible regime for the scalar case with high probability.
Revised: After we have the alignment guarantee, we can prove that we will enter the feasible regime for the scalar case with high probability.

Original: Still, we denote c t =
Revised: Still, we denote c t =

Original: ( x ⊤ t y t ) 2 .
Revised: ( x ⊤ t y t ) 2 .

Original: Lemma 22.
Revised: Lemma 22.

Original: (Feasible regime guarantee)
Revised: (Feasible regime guarantee)

Original: If ξ t < η 2 , η <
Revised: If ξ t < η 2 , η <

Original: [Equation]
Revised: [Equation]

Original: for all t ∈ [ t 1 , t 1 + 6 K 3 + 4 K ] , then there exists some T < 6 K 3 + 4 K such that for all t ∈ [ t 1 + T, t 1 + 6 K 3 + 4 K ] ,
Revised: for all t ∈ [ t 1 , t 1 + 6 K 3 + 4 K ] , then there exists some T < 6 K 3 + 4 K such that for all t ∈ [ t 1 + T, t 1 + 6 K 3 + 4 K ] ,

Original: |∥ x t ∥∥ y t
Revised: |∥ x t ∥∥ y t

Original: ∥ − 1 | < K − 1 .
Revised: ∥ − 1 | < K − 1 .

Original: Proof.
Revised: Proof.

Original: To prove the statement, we try to prove a stronger statement:
Revised: To prove the statement, we try to prove a stronger statement:

Original: [Equation]
Revised: [Equation]

Original: If this statement holds, because ξ t < η 2 , the product of the norms are
Revised: If this statement holds, because ξ t < η 2 , the product of the norms are

Original: [Equation]
Revised: [Equation]

Original: So we have ∥ x t ∥∥ y t ∥ ∈ (1 − 23 K − 1 , 1 + 12 K − 2 ) and the lemma is proved.
Revised: So we have ∥ x t ∥∥ y t ∥ ∈ (1 − 23 K − 1 , 1 + 12 K − 2 ) and the lemma is proved.

Original: So in the rest of the proof, we will prove this stronger statement.
Revised: So in the rest of the proof, we will prove this stronger statement.

Original: First we prove that there exists some time t < 4 K + 1 that the square of the inner product c t will be larger than 1 − K − 1 .
Revised: First we prove that there exists some time t < 4 K + 1 that the square of the inner product c t will be larger than 1 − K − 1 .

Original: If c t ∈ ( 720 , 1 / 2) , then
Revised: If c t ∈ ( 720 , 1 / 2) , then

Original: [Equation]
Revised: [Equation]

Original: Thus c t will provably enter c t > 1 / 2 in one step.
Revised: Thus c t will provably enter c t > 1 / 2 in one step.

Original: And if 1 / 2 < c t < 1 − K − 1 , we have
Revised: And if 1 / 2 < c t < 1 − K − 1 , we have

Original: [Equation]
Revised: [Equation]

Original: which means it takes at most 4 K steps for c t to become larger than 1 − K − 1 .
Revised: which means it takes at most 4 K steps for c t to become larger than 1 − K − 1 .

Original: Now we prove if c t > 1 − K − 1 , then it will take at most T = 6 K 3 steps s.t. for t ′ > t + T , c t ′ ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) , which finishes the proof.
Revised: Now we prove if c t > 1 − K − 1 , then it will take at most T = 6 K 3 steps s.t. for t ′ > t + T , c t ′ ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) , which finishes the proof.

Original: We first prove there exists some t , c t ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) ; afterwards we prove that once c t gets in, it will never get out of the region.
Revised: We first prove there exists some t , c t ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) ; afterwards we prove that once c t gets in, it will never get out of the region.

Original: If at first c t ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) , the proof is done.
Revised: If at first c t ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) , the proof is done.

Original: Otherwise, we suppose c t > 1 + K − 1 − K − 2 .
Revised: Otherwise, we suppose c t > 1 + K − 1 − K − 2 .

Original: Next, we consider the two step dynamics of c t and prove it will decay by a constant factor every two steps.
Revised: Next, we consider the two step dynamics of c t and prove it will decay by a constant factor every two steps.

Original: First we pick out the relatively small terms, and find out the main part of the two step dynamics.
Revised: First we pick out the relatively small terms, and find out the main part of the two step dynamics.

Original: If α ( t ) <
Revised: If α ( t ) <

Original: α 0 < 1 / 8 , and 1 + K − 1 − K − 2 <
Revised: α 0 < 1 / 8 , and 1 + K − 1 − K − 2 <

Original: c t < 1 . 3 , we have:
Revised: c t < 1 . 3 , we have:

Original: [Equation]
Revised: [Equation]

Original: Then we upper bound the difference α ( t + 1) − α ( t ) in one step.
Revised: Then we upper bound the difference α ( t + 1) − α ( t ) in one step.

Original: Consider the dynamics of ∥ x ∥ 2 + ∥ y ∥ 2 .
Revised: Consider the dynamics of ∥ x ∥ 2 + ∥ y ∥ 2 .

Original: The difference each step is at most
Revised: The difference each step is at most

Original: [Equation]
Revised: [Equation]

Original: Then the corresponding update of α ( t ) will be less than 5 η 2 .
Revised: Then the corresponding update of α ( t ) will be less than 5 η 2 .

Original: That means we havec t
Revised: That means we have

Original: [Equation]
Revised: [Equation]

Original: <[Equation] < c t (1 + (1 + α ( t ))
Revised: < c t (1 + (1 + α ( t ))

Original: (1 − c t )) 2 (1 + (1 + α ( t ))
Revised: (1 − c t )) 2 (1 + (1 + α ( t ))

Original: (1 − c t +1 )) 2 + 100 η< c t (1 + (1 + α ( t ))
Revised: (1 − c t +1 )) 2 + 100 η< c t (1 + (1 + α ( t ))

Original: (1 − c t )) 2 (1 + (1 + α ( t ))
Revised: (1 − c t )) 2 (1 + (1 + α ( t ))

Original: (1 − c t (1 + (1 + α ( t ))(1 − c t ) − 2 η 2 ) 2 )) 2 + 100 η< c t (1 + (1 + α ( t ))(1 − c t )) 2 (1 + (1 + α ( t ))
Revised: (1 − c t (1 + (1 + α ( t ))(1 − c t ) − 2 η 2 ) 2 )) 2 + 100 η< c t (1 + (1 + α ( t ))(1 − c t )) 2 (1 + (1 + α ( t ))

Original: (1 − c t (1 + (1 + α ( t ))(1 − c t )) 2 )) 2 + 200 η(128)
Revised: (1 − c t (1 + (1 + α ( t ))(1 − c t )) 2 )) 2 + 200 η(128)

Original: Similarly, we can have the lower bound of the 2-step dynamics.
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: After bound all small terms to 200 η [Equation] we consider the main part
Revised: After bound all small terms to 200 η 2 , we consider the main part

Original: [Equation]
Revised: [Equation]

Original: To prove it will decrease
Revised: To prove it will decrease

Original: [Equation]
Revised: [Equation]

Original: we need to prove the factor
Revised: we need to prove the factor

Original: [Equation]
Revised: [Equation]

Original: when α 
Revised: when α ∈

Original: [Equation]
Revised: [Equation]

Original:  We first prove that the function f ( α, c ) monotonically increases when α > − 1100 increases, i.e. ∂f ( α,c ) ∂α > 0 .
Revised: − [Equation] We first prove that the function f ( α, c ) monotonically increases when α > − 1100 increases, i.e. ∂f ( α,c ) ∂α > 0 .

Original: We directly give the simplified expression of this partial derivative.
Revised: We directly give the simplified expression of this partial derivative.

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Therefore, f ( α, c ) < f ( 12 K − 2 , c ) for all 1+ K − 1 − K − 2 < c < 1 . 3 .
Revised: Therefore, f ( α, c ) < f ( 12 K − 2 , c ) for all 1+ K − 1 − K − 2 < c < 1 . 3 .

Original: Denote β = 12 ( K − 2 − 2 K − 3 +
Revised: Denote β = 12 ( K − 2 − 2 K − 3 +

Original: K − 4 ) and suppose c = 1 + t √ β for some t > √ 2 (since c > 1 + K − 1 − K − 2 ).
Revised: K − 4 ) and suppose c = 1 + t √ β for some t > √ 2 (since c > 1 + K − 1 − K − 2 ).

Original: Meanwhile, since t √ β < 0 . 3 , t < 0 .
Revised: Meanwhile, since t √ β < 0 . 3 , t < 0 .

Original: 3 β − 1 / 2 .
Revised: 3 β − 1 / 2 .

Original: Then we plug in c = 1 + t √ β and expand f ( β, c ) .
Revised: Then we plug in c = 1 + t √ β and expand f ( β, c ) .

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Thus we have
Revised: Thus we have

Original: [Equation]
Revised: [Equation]

Original: Thus it takes at most 2 × 0 .
Revised: Thus it takes at most 2 × 0 .

Original: 3 / ( 110 K − 3 ) <
Revised: 3 / ( 110 K − 3 ) <

Original: 6 K 3 steps for c t to get into the region c t ∈ (1 − K − 1 , 1+
Revised: 6 K 3 steps for c t to get into the region c t ∈ (1 − K − 1 , 1+

Original: K − 1 − K − 2 ) .
Revised: K − 1 − K − 2 ) .

Original: Finally we prove that if c s ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) for some s , then for all t > s ,
Revised: Finally we prove that if c s ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) for some s , then for all t > s ,

Original: [Equation]
Revised: [Equation]

Original: We use induction and suppose c t satisfies the condition above.
Revised: We use induction and suppose c t satisfies the condition above.

Original: Note that we have the upper and lower bound of c t +1 above:
Revised: Note that we have the upper and lower bound of c t +1 above:

Original: [Equation]
Revised: [Equation]

Original: Meanwhile c t (1 + (1 + α ( t ))
Revised: Meanwhile c t (1 + (1 + α ( t ))

Original: (1 − c t )) 2 is monotonically decreasing with c t when − 1100 <
Revised: (1 − c t )) 2 is monotonically decreasing with c t when − 1100 <

Original: α ( t ) <
Revised: α ( t ) <

Original: 12 K − 2 and c t ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) .
Revised: 12 K − 2 and c t ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) .

Original: Thus we have:
Revised: Thus we have:

Original: [Equation]
Revised: [Equation]

Original: Therefore, by induction we prove that for all t > s , c t ∈(
Revised: Therefore, by induction we prove that for all t > s , c t ∈

Original: [Equation]
Revised: [Equation]

Original: − K − 2 ) .
Revised: K − 2 ) .

Original: In this way, |∥ x t ∥∥ y t ∥ − 1 | < K − 1 for t > t 1 + T .
Revised: In this way, |∥ x t ∥∥ y t ∥ − 1 | < K − 1 for t > t 1 + T .

Original: Entering the feasible region is not enough for establishing the equivalence to the scalar case.
Revised: Entering the feasible region is not enough for establishing the equivalence to the scalar case.

Original: Furthermore, we need the alignment variable ξ t to be small enough, such that some O ( · ) notation term can contain all the terms with ξ in the ( a, b ) -parameterization dynamics.
Revised: Furthermore, we need the alignment variable ξ t to be small enough, such that some O ( · ) notation term can contain all the terms with ξ in the ( a, b ) -parameterization dynamics.

Original: However, we need to guarantee that the trajectory does not converge to an unstable point near the minima.
Revised: However, we need to guarantee that the trajectory does not converge to an unstable point near the minima.

Original: We can prove that for the scalar case if the initialization is not exactly on the manifold of minimizers, but it becomes more challenging in higher dimensions.
Revised: We can prove that for the scalar case if the initialization is not exactly on the manifold of minimizers, but it becomes more challenging in higher dimensions.

Original: Therefore, we require an additional perturbation to escape from any unstable point.
Revised: Therefore, we require an additional perturbation to escape from any unstable point.

Original: We pick the time t p =
Revised: We pick the time t p =

Original: T 

Revised: T

Original: +6 K 3 +4 K =
Revised: +6 K 3 +4 K =

Original: O ( − log( η )+log( d ) − log( δ 0 )
Revised: O ( − log( η )+log( d ) − log( δ 0 )

Original: + K 3 ) to guarantee that when the perturbation happens, x and y are aligned and ( x ⊤ y ) 2 is not large enough to cause instability.
Revised: + K 3 ) to guarantee that when the perturbation happens, x and y are aligned and ( x ⊤ y ) 2 is not large enough to cause instability.

Original: After the perturbation at t p , the gradient descent dynamics prevent the objective from hitting the manifold of minimizers.
Revised: After the perturbation at t p , the gradient descent dynamics prevent the objective from hitting the manifold of minimizers.

Original: The following lemma proves that the properties of the bound of c t :=
Revised: The following lemma proves that the properties of the bound of c t :=

Original: ( x ⊤ y ) 2 , α = η ( ∥ x ∥ 2 + ∥ y ∥ 2 ) − 1 and ξ are still valid after the perturbation.
Revised: ( x ⊤ y ) 2 , α = η ( ∥ x ∥ 2 + ∥ y ∥ 2 ) − 1 and ξ are still valid after the perturbation.

Original: Lemma 23.
Revised: Lemma 23.

Original: After the perturbation at time t p , we have the following properties hold in t ∈ [ t p , 2 t p + 2] : (i) After the perturbation c ′ t p ∈
Revised: After the perturbation at time t p , we have the following properties hold in t ∈ [ t p , 2 t p + 2] : (i) After the perturbation c ′ t p ∈

Original: (1 + K − 1 − 2 K − 2 , 1 + 3 K − 1 + 2 K − 2 ) , and c t ∈ ( 720 , 1310 ) ; (ii)
Revised: (1 + K − 1 − 2 K − 2 , 1 + 3 K − 1 + 2 K − 2 ) , and c t ∈ ( 720 , 1310 ) ; (ii)

Original: K − 2 100 < α ( t ) < 12 K − 2 ; (iii) ξ t < (0 . 7) t − t p η 2 .
Revised: K − 2 100 < α ( t ) < 12 K − 2 ; (iii) ξ t < (0 . 7) t − t p η 2 .

Original: Proof.
Revised: Proof.

Original: We prove this property by induction.
Revised: We prove this property by induction.

Original: Firstly, we prove the basis of induction at t = t p .
Revised: Firstly, we prove the basis of induction at t = t p .

Original: For (i), before perturbation we have c t p ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) , so after the perturbation we have 
Revised: For (i), before perturbation we have c t p ∈ (1 − K − 1 , 1 + K − 1 − K − 2 ) , so after the perturbation we have c ′

Original: [Equation]
Revised: [Equation]

Original: For (ii), the initial value of α ( t ) satisfies that 140 K − 2 < α (0) <
Revised: For (ii), the initial value of α ( t ) satisfies that 140 K − 2 < α (0) <

Original: 14 K − 2 .
Revised: 14 K − 2 .

Original: But the total movement of α ( t ) before perturbation is bounded within ( − 5 η 2 t p , 5 η 2 t p ) ⊂ ( − K − 2 200 , K − 2 200 ) , and the perturbation introduce a movement of ∥ y ∥ 2 by ∥
Revised: But the total movement of α ( t ) before perturbation is bounded within ( − 5 η 2 t p , 5 η 2 t p ) ⊂ ( − K − 2 200 , K − 2 200 ) , and the perturbation introduce a movement of ∥ y ∥ 2 by ∥

Original: y t ∥ 2 < 1310 /
Revised: y t ∥ 2 < 1310 /

Original: ∥ x t ∥ 2 < 1 1000 K − 2 η − 12 .
Revised: ∥ x t ∥ 2 < 1 1000 K − 2 η − 12 .

Original: Due to the upper bound of η , α ( t ) ∈ ( K − 2 100 , K − 2 2 ) .
Revised: Due to the upper bound of η , α ( t ) ∈ ( K − 2 100 , K − 2 2 ) .

Original: For (iii), since K > 512 , thus at t p , ξ t p < (0 . 7) 6 K 3 ξ t < (0 . 7) 6 K 3 η 2 .
Revised: For (iii), since K > 512 , thus at t p , ξ t p < (0 . 7) 6 K 3 ξ t < (0 . 7) 6 K 3 η 2 .

Original: After the perturbation ξ ′ t p = (1 + 2 K − 1 ) 2 ξ t p < η 2 .
Revised: After the perturbation ξ ′ t p = (1 + 2 K − 1 ) 2 ξ t p < η 2 .

Original: So all three statement holds for t =
Revised: So all three statement holds for t =

Original: t p .
Revised: t p .

Original: Then we suppose for t ∈ [ t p , t p + t 1 ] all three statement holds and prove them for t p + t 1 + 1 .
Revised: Then we suppose for t ∈ [ t p , t p + t 1 ] all three statement holds and prove them for t p + t 1 + 1 .

Original: First by the dynamics of c t with condition c t ∈ ( 720 , 1310 ) and 0 < α ( t ) <
Revised: First by the dynamics of c t with condition c t ∈ ( 720 , 1310 ) and 0 < α ( t ) <

Original: K − 2 2 , we have c t +1 <
Revised: K − 2 2 , we have c t +1 <

Original: c t (1 + (1 + K − 2 2 )(1 − c t )) 2 < 13 / 10 and c t +1 > c t (1 + (1 − c t ) − 10 η 2 ) 2 > 720 .
Revised: c t (1 + (1 + K − 2 2 )(1 − c t )) 2 < 13 / 10 and c t +1 > c t (1 + (1 − c t ) − 10 η 2 ) 2 > 720 .

Original: Statement (i) is proved.
Revised: Statement (i) is proved.

Original: For (ii), since c t and ξ t are bounded for t ≤ t p + t 1 + 1 , we can still have the total movement of α ( t ) lies in ( − 5 η 2 t p , 5 η 2 t p ) ⊂ ( − K − 2 200 , K − 2 200 ) .
Revised: For (ii), since c t and ξ t are bounded for t ≤ t p + t 1 + 1 , we can still have the total movement of α ( t ) lies in ( − 5 η 2 t p , 5 η 2 t p ) ⊂ ( − K − 2 200 , K − 2 200 ) .

Original: Combine with the movement of y at perturbation ( < 1 1000 K − 2 η − 12 ), the total movement is still bounded by ( − 6 K − 2 1000 , 6 K − 2 1000 ) , which proves the bound in (ii).
Revised: Combine with the movement of y at perturbation ( < 1 1000 K − 2 η − 12 ), the total movement is still bounded by ( − 6 K − 2 1000 , 6 K − 2 1000 ) , which proves the bound in (ii).

Original: Finally for (iii), as long as c t < 1310 , ξ t
Revised: Finally for (iii), as long as c t < 1310 , ξ t

Original: +

Revised: +

Original: < 0 .
Revised: < 0 .

Original: 7 ξ t < 0 .
Revised: 7 ξ t < 0 .

Original: 7 t 1 +1 η 2 .
Revised: 7 t 1 +1 η 2 .

Original: Therefore all three statements are proved by induction.
Revised: Therefore all three statements are proved by induction.

Original: After reclaiming all the bounds after the perturbation, we need to prove that the alignment variable ξ will be small enough to approximate the scalar case.
Revised: After reclaiming all the bounds after the perturbation, we need to prove that the alignment variable ξ will be small enough to approximate the scalar case.

Original: The following lemma proves that b stays at a constant level when α ( t ) is some constant.
Revised: The following lemma proves that b stays at a constant level when α ( t ) is some constant.

Original: Lemma 24.
Revised: Lemma 24.

Original: Suppose ξ t < η 2 , η < [Equation] K 
Revised: Suppose ξ t < η 2 , η < 1 8000000 K −

Original: [Equation]
Revised: [Equation]

Original: , K
Revised: , K

Original: [Equation]
Revised: [Equation]

Original:  < α ( t ) <12 K
Revised: 100 < α ( t ) <

Original: [Equation] 2 for all t ∈ [ t 1 , 2 t 1 + 2] for some t 1 .
Revised: 12 K − 2 for all t ∈ [ t 1 , 2 t 1 + 2] for some t 1 .

Original: If ( x ⊤ t 1 y t 1 ) 2 ∈ (1 + K − 1 − 2 K − 2 , 1 + 3 K − 1 + 2 K − 2 ) , then for all t ∈ [ t 1 , 2 t 1 ] , ( x ⊤ t 1 +2 k y t 1 +2 k ) 2 ∈ (1 + 120 K − 1 , 1 + 3 K − 1 + 2 K − 2 ) , ( x ⊤ t 1 +2 k − 1 y t 1 +2 k − 1 ) 2 < 1 − 120 K − 1 , k ∈ N , k < t 1 / 2 .
Revised: If ( x ⊤ t 1 y t 1 ) 2 ∈ (1 + K − 1 − 2 K − 2 , 1 + 3 K − 1 + 2 K − 2 ) , then for all t ∈ [ t 1 , 2 t 1 ] , ( x ⊤ t 1 +2 k y t 1 +2 k ) 2 ∈ (1 + 120 K − 1 , 1 + 3 K − 1 + 2 K − 2 ) , ( x ⊤ t 1 +2 k − 1 y t 1 +2 k − 1 ) 2 < 1 − 120 K − 1 , k ∈ N , k < t 1 / 2 .

Original: Proof.
Revised: Proof.

Original: Denote c t :=
Revised: Denote c t :=

Original: ( x ⊤ t y t ) 2 .
Revised: ( x ⊤ t y t ) 2 .

Original: Here we consider the two step dynamics of the inner product (Eq.
Revised: Here we consider the two step dynamics of the inner product (Eq.

Original: (128) and Eq.
Revised: (128) and Eq.

Original: [Equation]
Revised: [Equation]

Original: We first consider the lower bound of the ( x ⊤ t 1 +2 k y t 1 +2 k ) 2 .
Revised: We first consider the lower bound of the ( x ⊤ t 1 +2 k y t 1 +2 k ) 2 .

Original: We prove the sequence of the c t +2 k , k = 0 , 1 , 2 , ... by induction, and then use this conclusion to prove the upper bound of c t +2 k − 1 .
Revised: We prove the sequence of the c t +2 k , k = 0 , 1 , 2 , ... by induction, and then use this conclusion to prove the upper bound of c t +2 k − 1 .

Original: We know that k
Revised: We know that k

Original: = 

Revised: =

Original: the statement holds.
Revised: the statement holds.

Original: Then suppose the lower bound holds for k ≤ k 0 , and we prove it for k = k 0 + 1 .
Revised: Then suppose the lower bound holds for k ≤ k 0 , and we prove it for k = k 0 + 1 .

Original: Here we consider the function as we do in Lemma 22:
Revised: Here we consider the function as we do in Lemma 22:

Original: [Equation]
Revised: [Equation]

Original: Denote γ = 1 100 K − 2 .
Revised: Denote γ = 1 100 K − 2 .

Original: By Eq.
Revised: By Eq.

Original: (130), we know
Revised: (130), we know

Original: [Equation]
Revised: [Equation]

Original: > f ( γ, c )
Revised: f ( γ, c )

Original: Suppose c t 1 +2 k
Revised: Suppose c t 1 +2 k

Original: = 1 + q √ γ , q ∈ ( 12 , 30 + 40 K − 1 ) .
Revised: = 1 + q √ γ , q ∈ ( 12 , 30 + 40 K − 1 ) .

Original: Then the expression f ( γ, c ) becomes: f ( γ, [Equation] ) =f
Revised: Then the expression f ( γ, c ) becomes: f ( γ, c ) =

Original: [Equation]

Original:  γ, 1 + q √ γ )
Revised: f ( γ, 1 + q √ γ )

Original: [Equation]
Revised: [Equation]

Original: Notice that when c > 1 , f ( γ, c ) decrease as c increase.
Revised: Notice that when c > 1 , f ( γ, c ) decrease as c increase.

Original: Now we consider the range of q : If q ∈ ( 12 , 23 ] , we have:
Revised: Now we consider the range of q : If q ∈ ( 12 , 23 ] , we have:

Original: [Equation]
Revised: [Equation]

Original: + 16 γ81 + 64 γ81 − 4 γ27 − 152 γ − 92 γ81 +> 1 + 10 27 γ 3 / 2 .
Revised: + 16 γ81 + 64 γ81 − 4 γ27 − 152 γ − 92 γ81 +> [Equation] 27 γ 3 / 2 .

Original: ( since K > 512 , γ 1 / 2 = K − 1 10 )
Revised: ( since K > 512 , γ 1 / 2 = K − 1 10 )

Original: That means c t 1 +2 k +2 >
Revised: That means c t 1 +2 k +2 >

Original: ((1 + 1027 K − 2 ) 2 − 200 η 2 ) c
Revised: ((1 + 1027 K − 2 ) 2 − 200 η 2 ) c

Original: t 1 +2 k > c t 1 +2 k > (1 + K − 1 20 ) and the proof is done.
Revised: t 1 +2 k > c t 1 +2 k > (1 + K − 1 20 ) and the proof is done.

Original: Otherwise, we have q ∈ ( 23 , 30 + 40 K − 1 ) , also we have the lower bound of this function:
Revised: Otherwise, we have q ∈ ( 23 , 30 + 40 K − 1 ) , also we have the lower bound of this function:

Original: [Equation]
Revised: [Equation]

Original: ( since K > 512 , γ 1 / 2 = K − 1 10 )
Revised: ( since K > 512 , γ 1 / 2 = K − 1 10 )

Original: That means
Revised: That means

Original: [Equation]
Revised: [Equation]

Original: Then the lower bound of c t 1 +2 k is proved.
Revised: Then the lower bound of c t 1 +2 k is proved.

Original: As for the upper bound, since when t = t 1 + 2 k , x , y satisfies all the conditions in Lemma 22.
Revised: As for the upper bound, since when t = t 1 + 2 k , x , y satisfies all the conditions in Lemma 22.

Original: If c t 1 +2 k < 1 + K − 1 − K − 2 , we have proved that it will never be larger than 1 + K − 1 − K − 2 .
Revised: If c t 1 +2 k < 1 + K − 1 − K − 2 , we have proved that it will never be larger than 1 + K − 1 − K − 2 .

Original: Otherwise if c t 1 +2 k ∈ (1 + K − 1 − K − 2 , 1 + 3 K − 1 + 2 K − 2 )
Revised: Otherwise if c t 1 +2 k ∈ (1 + K − 1 − K − 2 , 1 + 3 K − 1 + 2 K − 2 )

Original: We apply the inequality Eq.
Revised: We apply the inequality Eq.

Original: and know c t 1 +2 k +2 <
Revised: and know c t 1 +2 k +2 <

Original: c t 1 +2 k .
Revised: c t 1 +2 k .

Original: In this way, by induction we prove the bound ( x ⊤ t 1 +2 k y t 1 +2 k ) 2 ∈ (1 + 120 K − 1 , 1 + 3 K − 1 + 2 K − 2 ) .
Revised: In this way, by induction we prove the bound ( x ⊤ t 1 +2 k y t 1 +2 k ) 2 ∈ (1 + 120 K − 1 , 1 + 3 K − 1 + 2 K − 2 ) .

Original: As for the upper bound of ( x ⊤ t 1 +2 k − 1 y t 1 +2 k − 1 ) 2 , we directly apply the upper bound of 1-step dynamics (if c t > 1 + 120 K − 1 ):
Revised: As for the upper bound of ( x ⊤ t 1 +2 k − 1 y t 1 +2 k − 1 ) 2 , we directly apply the upper bound of 1-step dynamics (if c t > 1 + 120 K − 1 ):

Original: [Equation]
Revised: [Equation]

Original: Therefore the upper bound ( x ⊤ t 1 +2 k − 1 y t 1 +2 k − 1 ) 2 < 1 − 120 K − 1 holds for all k < t 1 / 2 .
Revised: Therefore the upper bound ( x ⊤ t 1 +2 k − 1 y t 1 +2 k − 1 ) 2 < 1 − 120 K − 1 holds for all k < t 1 / 2 .

Original: Finally, we denote a := (cid:112)
Revised: Finally, we denote a := (cid:112)

Original: ∥ x ∥ 2 − ∥ y ∥ 2 − ( η − 2 − 4) 12 , b := ∥
Revised: ∥ x ∥ 2 − ∥ y ∥ 2 − ( η − 2 − 4) 12 , b := ∥

Original: x ∥∥ y ∥ − 1 .
Revised: x ∥∥ y ∥ − 1 .

Original: With all the lemmas above, we prove the equivalence between the dynamics of ( a, b ) and the one step dynamics of in the scalar case (Lemma 1, Lemma 2).
Revised: With all the lemmas above, we prove the equivalence between the dynamics of ( a, b ) and the one step dynamics of in the scalar case (Lemma 1, Lemma 2).

Original: For simplicity of notations, when analyzing the 1-step and 2-step dynamics of ( a t , b t ) , we use a, a ′ to denote a t , a t +1 and b, b ′ to denote b t , b t +1 , etc.
Revised: For simplicity of notations, when analyzing the 1-step and 2-step dynamics of ( a t , b t ) , we use a, a ′ to denote a t , a t +1 and b, b ′ to denote b t , b t +1 , etc.

Original: For simplicity of calculation, we consider the change of variable κ = √ η .
Revised: For simplicity of calculation, we consider the change of variable κ = √ η .

Original: Lemma 25.
Revised: Lemma 25.

Original: (Equivalence with scalar updates)
Revised: (Equivalence with scalar updates)

Original: If ξ t < min { η 2 , ηb 4 t } , ∥ x t ∥ ∈ (˘ x + 1200 K − 2 η − 12 , ˘ x + 14 K − 2 η − 12 ) and |∥ x t ∥∥ y t
Revised: If ξ t < min { η 2 , ηb 4 t } , ∥ x t ∥ ∈ (˘ x + 1200 K − 2 η − 12 , ˘ x + 14 K − 2 η − 12 ) and |∥ x t ∥∥ y t

Original: ∥ − 1 | < K − 1 , the following equations hold for some fixed constant ϵ = min { 0 . 5 , 1125 K − 1 } :
Revised: ∥ − 1 | < K − 1 , the following equations hold for some fixed constant ϵ = min { 0 . 5 , 1125 K − 1 } :

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For simplicity, we denote x = ∥ x ∥ and y = ∥
Revised: For simplicity, we denote x = ∥ x ∥ and y = ∥

Original: y ∥ .
Revised: y ∥ .

Original: We first prove that:
Revised: We first prove that:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We suppose y =
Revised: We suppose y =

Original: c x + θ for some θ ∈ R d and ⟨ θ , x ⟩ = 0 .
Revised: c x + θ for some θ ∈ R d and ⟨ θ , x ⟩ = 0 .

Original: In this way, we have x ⊤ y = cx 2 , y 

Revised: In this way, we have x ⊤ y = cx 2 , y

Original: = c 2 x 2 + ∥ θ ∥ 2 , ξ = x 2 ∥ θ ∥ 2 .
Revised: = c 2 x 2 + ∥ θ ∥ 2 , ξ = x 2 ∥ θ ∥ 2 .

Original: Since ξ < ηb 4 and x = O ( κ − 1 ) ,
Revised: Since ξ < ηb 4 and x = O ( κ − 1 ) ,

Original: ∥ θ ∥ =
Revised: ∥ θ ∥ =

Original: O ( b 2 κ 2 ) .
Revised: O ( b 2 κ 2 ) .

Original: Then check the dynamics of x and plug in y =
Revised: Then check the dynamics of x and plug in y =

Original: c x + θ .
Revised: c x + θ .

Original: [Equation]
Revised: [Equation]

Original: Since ( x ⊤ y ) and xy are both bounded as constant, we can directly take the norm of both sides and with triangle inequality we have:
Revised: Since ( x ⊤ y ) and xy are both bounded as constant, we can directly take the norm of both sides and with triangle inequality we have:

Original: [Equation]
Revised: [Equation]

Original: Similarly, we have the dynamics of y .
Revised: Similarly, we have the dynamics of y .

Original: [Equation]
Revised: [Equation]

Original: We now reparameterize the dynamics in ( a, b ) -parameterization.
Revised: We now reparameterize the dynamics in ( a, b ) -parameterization.

Original: [Equation]
Revised: [Equation]

Original: where δ ≜ (cid:114)
Revised: where δ ≜ (cid:114)

Original: [Equation]
Revised: [Equation]

Original: And by the bound of ∥ x t ∥ and |∥ x t ∥∥ y t ∥ − 1 | , we can have | a | < ϵκ, | b | < min { 1 , 15 ϵκ − 2 } , which satisfies the condition in Lemma 1 and Lemma 2.
Revised: And by the bound of ∥ x t ∥ and |∥ x t ∥∥ y t ∥ − 1 | , we can have | a | < ϵκ, | b | < min { 1 , 15 ϵκ − 2 } , which satisfies the condition in Lemma 1 and Lemma 2.

Original: Then we follow the proof of Lemma 1 and
Revised: Then we follow the proof of Lemma 1 and

Original: Lemma 2 to finish the proof (since the only difference is the two O ( · ) notation).
Revised: Lemma 2 to finish the proof (since the only difference is the two O ( · ) notation).

Original: After we can reduce the dynamics of vector case to the scalar case dynamics, we need to prove that in the scalar case (in all stages), for any t ≥ 0 , ( b t +1 b t ) 4 > 0 . 7 , which means ξ t shrinks faster than b 4 t .
Revised: After we can reduce the dynamics of vector case to the scalar case dynamics, we need to prove that in the scalar case (in all stages), for any t ≥ 0 , ( b t +1 b t ) 4 > 0 . 7 , which means ξ t shrinks faster than b 4 t .

Original: In that case, we can conclude that ξ t > ηb 4 t always holds along the scalar case trajectory.
Revised: In that case, we can conclude that ξ t > ηb 4 t always holds along the scalar case trajectory.

Original: Lemma 26.
Revised: Lemma 26.

Original: Suppose all conditions in Theorem 3.1 hold.
Revised: Suppose all conditions in Theorem 3.1 hold.

Original: Then for all t > 0 , ( b t +1 b t ) 4 > 0 .
Revised: Then for all t > 0 , ( b t +1 b t ) 4 > 0 .

Original: Proof.
Revised: Proof.

Original: From the proof of Theorem 3.1, we know that for all t > 0 , | b t | < K − 1 and | a t | < κ − 1 .
Revised: From the proof of Theorem 3.1, we know that for all t > 0 , | b t | < K − 1 and | a t | < κ − 1 .

Original: Thus Lemma 2 holds and for all t ≥ 

Revised: Thus Lemma 2 holds and for all t ≥

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Since we know all constants hidden by the O ( · ) operator are upper bounded by K , we have the following lower bound on the multiplicative update of b :
Revised: Since we know all constants hidden by the O ( · ) operator are upper bounded by K , we have the following lower bound on the multiplicative update of b :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Finally we conclude the proof of Theorem C.1.
Revised: Finally we conclude the proof of Theorem C.1.

Original: Proof of Theorem C.1.
Revised: Proof of Theorem C.1.

Original: By Lemma 21, we know with probability 1 − 2 δ 0 − 2 exp {− Ω( d ) } , there exists some time T 

Revised: By Lemma 21, we know with probability 1 − 2 δ 0 − 2 exp {− Ω( d ) } , there exists some time T

Original: = O ( − log( η ) − log( δ 0 )
Revised: = O ( − log( η ) − log( δ 0 )

Original: + log( d ))
Revised: + log( d ))

Original: that ξ t < η 2 for all t > T 0 .
Revised: that ξ t < η 2 for all t > T 0 .

Original: Also for t ∈ [ T 0 , T 0 +6 K 3 +4 K ] , ∥ x t ∥ ∈ (˘ x + 1200 K − 2 η − 12 , ˘ x + 16 K − 2 η − 12 ) .
Revised: Also for t ∈ [ T 0 , T 0 +6 K 3 +4 K ] , ∥ x t ∥ ∈ (˘ x + 1200 K − 2 η − 12 , ˘ x + 16 K − 2 η − 12 ) .

Original: This bound of ∥ x ∥ guarantees that for t ∈ [ T 0 , T 0 + 6 K 3 + 4 K ] , α ( t ) ∈ (0 , K − 2 2 ) , which satisfies the condition of Lemma 22.
Revised: This bound of ∥ x ∥ guarantees that for t ∈ [ T 0 , T 0 + 6 K 3 + 4 K ] , α ( t ) ∈ (0 , K − 2 2 ) , which satisfies the condition of Lemma 22.

Original: Then by Lemma 22, we know for some t ∗ < T 0 + 6 K 3 + 4 K < t p , |∥ x t ∥∥ y t
Revised: Then by Lemma 22, we know for some t ∗ < T 0 + 6 K 3 + 4 K < t p , |∥ x t ∥∥ y t

Original: ∥ − 1 | < K − 1 for t ∈ [ t ∗ , t p ] .
Revised: ∥ − 1 | < K − 1 for t ∈ [ t ∗ , t p ] .

Original: After entering and staying in the region, we add the perturbation at t p .
Revised: After entering and staying in the region, we add the perturbation at t p .

Original: By Lemma 23, we have the bounds before perturbation still hold: (i) After the perturbation c ′ t p ∈
Revised: By Lemma 23, we have the bounds before perturbation still hold: (i) After the perturbation c ′ t p ∈

Original: (1+ K − 1 − 2 K − 2 , 1+ 3 K − 1 + 2 K − 2 ) , and c t ∈ ( 7 20 , 13 10 ) ; (ii) K − 2 100 < α ( t ) < 1 2 K − 2 ; (iii) ξ t < (0 . 7) t − t p η 2 .
Revised: (1+ K − 1 − 2 K − 2 , 1+ 3 K − 1 + 2 K − 2 ) , and c t ∈ ( 7 20 , 13 10 ) ; (ii) K − 2 100 < α ( t ) < 1 2 K − 2 ; (iii) ξ t < (0 . 7) t − t p η 2 .

Original: Therefore, the condition of Lemma 22 and Lemma 24 are both met.
Revised: Therefore, the condition of Lemma 22 and Lemma 24 are both met.

Original: So after another 6 K 3 + 4 K steps, we have | b t | = |∥ x t ∥∥ y t
Revised: So after another 6 K 3 + 4 K steps, we have | b t | = |∥ x t ∥∥ y t

Original: ∥ − 1 | < K − 1 and | ( x ⊤ t y t ) 2 − 1 | > 1 − K − 1 20 .
Revised: ∥ − 1 | < K − 1 and | ( x ⊤ t y t ) 2 − 1 | > 1 − K − 1 20 .

Original: The second expression can lead to | b t | > K − 1 20 / (1 + ∥ x t ∥∥ y t ∥ )
Revised: The second expression can lead to | b t | > K − 1 20 / (1 + ∥ x t ∥∥ y t ∥ )

Original: > K − 1 41 .
Revised: > K − 1 41 .

Original: This means ξ t < (0 . 7) [Equation] K η [Equation]
Revised: This means ξ t < (0 . 7) 4 K η 2 < (0 . 7) 2000 η · K − 4 8000000 <

Original: < (0 . 7) 2000 η

Original: [Equation] K [Equation] < η | b 
Revised: η | b t | 4 .

Original: [Equation] By Lemma 25, we know the dynamics of the norm of the vectors ( ∥ x ∥ , ∥ y ∥ ) can be captured by the scalar case (including the initialization condition and the one step update rules).
Revised: By Lemma 25, we know the dynamics of the norm of the vectors ( ∥ x ∥ , ∥ y ∥ ) can be captured by the scalar case (including the initialization condition and the one step update rules).

Original: And by Lemma 26, we know the alignment will be kept and the dynamics of the vectors will always be true.
Revised: And by Lemma 26, we know the alignment will be kept and the dynamics of the vectors will always be true.

Original: Finally, we apply Theorem 3.1 and finish the proof.
Revised: Finally, we apply Theorem 3.1 and finish the proof.