Original: Asdiscussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: As discussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: When ρ 1 , ρ 2 , ρ 3 achieve their upper boundat the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .
Revised: When ρ 1 , ρ 2 , ρ 3 achieve their upper bound at the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: However, ρ ishighly dependent on the problem instance f ( x ) and training process.
Revised: However, ρ is highly dependent on the problem instance f ( x ) and training process.

Original: Our experiments in Appendix
Revised: Our experiments in Appendix

Original: B.1 shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lowerthan it appears to be.
Revised: B shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lower than it appears to be.

Original: We further remark that γ 1 ( n ) ∝ (1 − β n 1 ) .
Revised: We further remark that γ 1 ( n ) ∝ (1 − β n 1 ) .

Original: That is to say, as β 1 increases, the threshold of β 2 alsoslowly increases (to ensure the convergence).
Revised: That is to say, as β 1 increases, the threshold of β 2 also slowly increases (to ensure the convergence).

Original: However, such phenomenon is not observed in ourexperiments.
Revised: However, such phenomenon is not observed in our experiments.

Original: We conjecture γ 1 ( n ) should be independent of β 1 , removing the dependence (1 − β n 1 )is considered as an interesting future direction.
Revised: We conjecture γ 1 ( n ) should be independent of β 1 , removing the dependence (1 − β n 1 ) is considered as an interesting future direction.

Original: Remark G.8 .
Revised: Remark G.8 .

Original: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.
Revised: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.

Original: Thiscan be seen in the proof in Appendix G.9 (the definition of F 4 ) and Remark G.14.
Revised: This can be seen in the proof in Appendix G.9 (the deﬁnition of F 4 ) and Remark G.14.

Original: Based on Lemma G.6, we can further rewrite (30) as follows.
Revised: Based on Lemma G.6, we can further rewrite (31) as follows.

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 3.1 is concluded by plugging (34) into Descent Lemma (24) and then takingtelescope some from k = t 0 to k = T .
Revised: The proof of Theorem 3.1 is concluded by plugging (35) into Descent Lemma (25) and then taking telescope some from k = t 0 to k = T .

Original: These steps are quite standard in non-convex optimization.
Revised: These steps are quite standard in non-convex optimization.

Original: We finish these calculation in Lemma [-G.
-]
Revised: We ﬁnish these calculation in Lemma [-G.
-]G.

Original: Lemma G.9.
Revised: Lemma G.9.

Original: When inequality (34) holds, we have the following results based on Descent Lemma(24) :
Revised: When inequality (35) holds, we have the following results based on Descent Lemma (25) :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma G.9 can be seen in Appendix G.10.
Revised: Proof of Lemma G.9 can be seen in Appendix G.10.

Original: Now, the whole proof of Theorem 3.1 iscompleted.
Revised: Now, the whole proof of Theorem 3.1 is completed.

Original: G.2 Proof of Lemma G. 
Revised: G.2 Proof of Lemma G.

Original: To prove Lemma G.1, we only prove (27).
Revised: To prove Lemma G [Equation] [Equation] we only prove (28).

Original: The proof for (26) can be analogized from the proof of(27).
Revised: The proof for (27) can be analogized from the proof of (28).

Original: We discuss the following two cases:
Revised: We discuss the following two cases:

Original: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have∂ l f ( x k, 0 )
Revised: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have

Original: [Equation]
Revised: [Equation]

Revised: where ( a ) is because of √ v l,k,i ≥ (cid:112) β

Original: m l,k ,i [Equation]
Revised: i 2 √ v l,k, 0 .

Original:  ∂ l f i 
Revised: Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have ∂ l f ( x k, 0 ) ( m l,k,i

Original: [Equation]
Revised: [Equation]

Original: x k,
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is because of √ v l,k,i ≥ (cid:112) β i 2 √ v l,k, 0 . Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have∂ l f ( x k,
Revised: i ( x k, 0 )) √

Original: [Equation]
Revised: [Equation]

Original: ( m l,k,i − ∂ l f i ( x k,
Revised: f [Equation] √ v [Equation]

Revised: (cid:18)

Revised: [Equation]

Revised: v

Original: [Equation]
Revised: [Equation]

Revised: v

Original:  where ( ∗ ) is due to 1 √ 1+ x ≥ 1 − x 2 .
Revised: [Equation] | 2 v l [Equation] (cid:19) (36) where ( ∗ ) is due to 1 √ 1+ x ≥ 1 − x 2 .

Original: We further have
Revised: We further have

Original: [Table]
Revised: [Table]

Original: v

Original: [Table]
Revised: [Equation]

Original: where ( ∗∗ ) is due to the condition in the Lemma max i | ∂ l f i ( x τ k, 0 )
Revised: where ( ∗∗ ) is due to the condition in the Lemma max i | ∂ l f i ( x τ k, 0 )

Original: | ≥ Q k ≥ n △ nk .
Revised: | ≥ Q k ≥ n (cid:52) nk .

Original: Plug the aboveresult into (35) we have
Revised: Plug the above result into (36) we have

Original: [Equation]
Revised: [Equation]

Original: In the last step ( ∗ ∗ ∗ ) , we use the following Lemma G.10.
Revised: In the last step ( ∗ ∗ ∗ ) , we use the following Lemma G.10.

Original: Lemma G.10.
Revised: Lemma G.10.

Original: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfies
Revised: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisﬁes max i | [Equation] i

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof idea of Lemma G.10 is similar as Lemma F.1 in (Shi et al., 2020), but with thefollowing differences:
Revised: The proof idea of Lemma G.10 is similar as Lemma F.1 in (Shi et al., 2020), but with the following differences:

Original: The condition of Lemma F.1 in (Shi et al., 2020)
Revised: The condition of Lemma F.1 in (Shi et al., 2020)

Original: both require the full batch gradient | ∂ l f ( x k, 0 ) | ≥nQ k , which is different from our condition.
Revised: both require the full batch gradient | ∂ l f ( x k, 0 ) | ≥ nQ k , which is different from our condition.

Original: Here, we choose the condition of Lemma G.10because it meets the need of our decomposition strategy in (30) .2.
Revised: Here, we choose the condition of Lemma G. because it meets the need of our decomposition strategy in (31) .

Original: The constant Q k is different.
Revised: The constant Q k is different.

Original: The difference is due to the different update formula of RMSPropand general Adam.
Revised: The difference is due to the different update formula of RMSProp and general Adam.

Original: Anyhow, Lemma G.10 can be easily proved following most of the steps in Lemma F.1 in (Shi et al.,2020).
Revised: Anyhow, Lemma G.10 can be easily proved following most of the steps in Lemma F.1 in (Shi et al., 2020).

Original: We omit the proof for brevity.
Revised: We omit the proof for brevity.

Original: Combining the above Case 1 and Case 2 , we have
Revised: Combining the above Case 1 and Case 2 , we have

Original: [Equation]
Revised: [Equation]

Original: i ( x k, 0 ) | , where δ 1 = (1 − β 2 )4 nρ 2 β n+
Revised: i ( x k, 0 ) | , where δ 1 = (1 − β 2 )4 nρ 2 β n+

Original: (cid:18)
Revised: [Equation]

Original: 1 √ β n 2 −(cid:19) .
Revised: √ β n 2 −(cid:19) .

Original: Proof is completed.
Revised: Proof is completed.

Original: G.3 Proof of Lemma G. 
Revised: G.3 Proof of Lemma G.

Original: We first prove the result for j = 1 .
Revised: We ﬁrst prove the result for j = 1 .

Original: We discuss the following two cases.
Revised: We discuss the following two cases.

Original: Case 1: when ∂ l f ( x k, 0 ) √ v l,k, 0 ≥ ∂ l f ( x k − 1 , 0 ) √ v l,k − 1 , 0 : when ∂ l f ( x k, 0 ) ≤ 0 , we have
Revised: Case 1: when ∂ l f ( x k, 0 ) √ v l,k, 0 ≥ ∂ l f ( x k − 1 , 0 ) √ v l,k − 1 , 0 : when ∂ l f ( x k, 0 ) ≤ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) > 0 , we have
Revised: When ∂ l f ( x k, 0 ) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In conclusion, we have:
Revised: In conclusion, we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , where δ 1 =
Revised: where δ 1 =

Original: [Equation]
Revised: [Equation]

Original: is a constant that goes to 0 when β 2 goes to 1.
Revised: is a constant that goes to 0 when β 2 goes to 1.

Original: Therefore, wehave
Revised: Therefore, we have

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: Case 2: when∂ l f
Revised: Case 2: when

Original: [Equation]
Revised: [Equation]

Original:  x [Equation]
Revised: : when ∂ l f ( x k, 0 ) ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: : when ∂ l f ( x k, 0 ) ≥ 0 , we have
Revised: When ∂ l f ( x k, 0 ) <

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) < 0 , we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Following the same strategy as in Case 1, we can show that
Revised: Following the same strategy as in Case [Equation] we can show that

Original: [Equation]
Revised: [Equation]

Original: which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Original: Case 1 and Case 2 together, we have
Revised: Case 1 and Case 2 together, we have

Original: [Equation]
Revised: [Equation]

Original: Now we consider the case when j > 1 .
Revised: Now we consider the case when j > 1 .

Original: Based on the above inequality, we have
Revised: Based on the above inequality, we have

Original: [Table]
Revised: [Table]

Original: The proof is completed.
Revised: The proof is completed.

Original: G.4 Proof of Lemma G. 
Revised: G.4 Proof of Lemma G.

Original: To prove Lemma G.4, we need to further decompose E [ (cid:80) l large ( b 1 )] .
Revised: To prove Lemma G.4, we need to further decompose E [ (cid:80) l large ( b 1 )] .

Original: First and foremost, we write(cid:80) n − 1 i =

Revised: First and foremost, we write (cid:80) n − 1 i =

Original: m l,k,i in an explicit form.m l,k,i = (1 − β 
Revised: m l,k,i in an explicit form. m l,k,i = (1 − β 1 ) { ∂ l f τ k

Original: [Equation]
Revised: [Equation]

Original: ∂ l f ( x 1 , 0 ) (37)

Original: Since ∂ l f ( x 1 , 0 ) =
Revised: Since ∂ l f ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =
Revised: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =

Original: (cid:80) n − 1 i =

Revised: (cid:80) n − 1 i =

Original: ∂ l f i ( x 1 , 0 ) , wehaveβ (
Revised: ∂ l f i ( x 1 , 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: [-− 1) n + i +
-]
Revised: [-− 1) n + i +
-]Plugging (39) into (38), we have m l,

Original: [Equation]
Revised: [Equation]

Original: Plugging (38) into (37), we have
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ∂ l f τ k, 0 ( x k, 0 ) + β i +11 f τ k − 1 ,n − 1 ( x k − 1 ,n − 1 ) + · · · + β i 
Revised: +11 f τ k − 1 ,n − 1 ( x k − 1 ,n − 1 ) + · · · + β i + nf τ k −

Revised: [Equation]

Original: + nf τ k − 1 , 0 ( x k − 1 , 0 ) + · · · + β ( k − 2) n + i +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + i∂ l f τ 1 , 0 ( x 1 , 0 )
Revised: 0 ( x k − 1 , 0 ) + · · · + β ( k − 2) n + i +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + i∂ l f τ 1 , 0 ( x 1 , 0 )

Original: + β ( k − 1) n
Revised: + β ( k − 1) n

Original: + i +1∂ 
Revised: + i +1∂ l f 0 (

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation][Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original:  ( k − 1) n
Revised: β ( k − 1) n

Original: [Equation] i +
 
Revised: + i +
 [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ( x
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: [-1) n + i +
-]
Revised: [-1) n + i +
-]

Original: [Equation]
Revised: [Equation]

Original: l f 0 ( x 1 , 0 ) + · · ·
Revised: 1) n

Original: [Equation]
Revised: [Equation]

Original: β ( k − 1) n + i[-+
-]
Revised: i[-+
-]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: l

Original: [Equation]
Revised: [Equation]

Original: Using (39), we have (For each ‘epoch’, we suggest readers to read from bottom to the top.)
Revised: n

Original: [Equation]
Revised: [Equation]

Original: thepoch

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [-( x 1 ,n − 1 ) + · · · + β ( k − 1) n + n − 2∂ l f τ 1 , 0 ( x 1 , 0 ) + · · · + β ( k − 2) n +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 
 -thepoch +-]
Revised: [-( x 1 ,n − 1 ) + · · · + β ( k − 1) n + n − 2∂ l f τ 1 , 0 ( x 1 , 0 ) + · · · + β ( k − 2) n +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 
 -thepoch +-][Equation]

Original: [Equation]
Revised: [Equation]

Original: ( k [Equation] [Equation] f
Revised: Using (40), we have (For each ‘epoch’, we suggest readers to read from bottom to the top

Original: [Equation]
Revised: [Equation]

Original: ) n

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: [Equation]

Original: l f n

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ∂ l f [Equation] [Equation] [Equation]
Revised: −

Original: [Equation]

Original: [Equation]

Original: ( k

Original: [Equation]
Revised: [Equation]

Original: 1) n

Original: [Equation]
Revised: [Equation]

Original: 1∂ l f n − 1 ( x 1 , 0 )

Original: [Equation]
Revised: [Equation]

Original: [-(cid:124) (cid:123)(cid:122) (cid:125) 
 -thepoch ... } (40)-] Note that in (40), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition willcause trouble in bounding the (cid:80) n − 1 i =

Revised: [-(cid:124) (cid:123)(cid:122) (cid:125) 
 -thepoch ... } (40)-] Note that in (41), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition will cause trouble in bounding the (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) .

Original: Therefore, we need to define an auxillarysequence M ′ l,k which does not involve such a phase transition.
Revised: Therefore, we need to deﬁne an auxillary sequence M (cid:48) l,k which does not involve such a phase transition.

Original: We define M ′ l,k as follows.
Revised: We deﬁne M (cid:48) l,k as follows.

Original: (For each‘epoch’, we suggest readers to read from bottom to the top.)
Revised: (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: 2( n −

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: ∂ l f n −

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: x

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: [Equation] [Equation]

Original: n
Revised: k −

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ∂ l f n
Revised: ∂ l f n

Original: [Equation]
Revised: [Equation]

Original: ∂{+(cid:125) 
+}
Revised: {+(cid:125) 
+}

Original: [Equation] [Equation]
Revised: -thepoch ... } (42)

Original: Now we bound (cid:80) n − 1 i =

Revised: Now we bound (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M ′ l,k .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M (cid:48) l,k .

Original: Denoting F l,k :=(cid:80) n − 1 i =

Revised: Denoting F l,k := (cid:80) n − 1 i =

Original: ∂ l f i ( x k, 0 ) , we have:
Revised: ∂ l f i ( x k, 0 ) , we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k .F l
Revised: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k .

Original: [Equation]

Original: k := n − 1 (cid:88) i =
Revised: F l,k := n − 1 (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: + ...

Original: +
Revised: +

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: +

Original: [Equation]
Revised: [Equation]

Original: For the sake of better presentation
Revised: For the sake of better presentation

Original: [Equation]
Revised: [Equation]

Original: we rewrite F l [Equation] k and M ′ l [Equation]
Revised: we rewrite F l,k and M (cid:48) l,k as follows:

Original: k as follows:

Original: [Equation]
Revised: [Equation]

Original: · , where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .
Revised: where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .

Original: Similarly for
Revised: Similarly for

Original: M ′ l,k .
Revised: M (cid:48) l,k .

Original: Now, we separate M ′ l,k − F l,k into two parts.
Revised: Now, we separate M (cid:48) l,k − F l,k into two parts.

Original: [Equation]
Revised: [Equation]

Original: Now, we rewrite E [ (cid:80) l large ( b 1 )] :
Revised: Now, we rewrite E [ (cid:80) l large ( b 1 )] :

Original: [Equation]
Revised: [Equation]

Original: l large [Equation] [Equation] l large
Revised: E

Original: [Equation]
Revised: [Equation]

Original: l large
Revised: l large

Original: [Equation]

Original: (cid:88) l large

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) (1)

Original: [Equation]
Revised: [Equation]

Original: We bound (1) , (2) and (3) respectively.
Revised: We bound (1) , (2) and (3) respectively.

Original: We bound (1) and (3) in the following Lemma G.11 and
Revised: We bound (1) and (3) in the following Lemma G.11 and

Original: Lemma G.12.
Revised: Lemma G.12.

Original: Since the difference of M l,k − M ′ l,k only occurs in the high order terms of the infinitesum sequence, (1) is expected to vanish as k grows.
Revised: Since the difference of M l,k − M (cid:48) l,k only occurs in the high order terms of the inﬁnite sum sequence, (1) is expected to vanish as k grows.

Original: Similarly for (3) .
Revised: Similarly for (3) .

Original: Lemma G.11.
Revised: Lemma G.11.

Original: When k is large enough such that: β ( k − 1) n
Revised: When k is large enough such that: β ( k − 1) n

Original: [Equation]
Revised: [Equation]

Original: 1 and k ≥ 4 , then we have [Equation]
Revised: √

Original: [Equation]
Revised: [Equation]

Original: where G 4 = d (cid:113)
Revised: 1 and k ≥ 4 , then we have

Original: [Equation]
Revised: [Equation]

Original: β n 2 β n

Original: [Equation]
Revised: [Equation]

Original: √ 2 n ( n [Equation] (cid:80)
Revised: where G 4 = d (cid:113) 2 ρ 23 β

Original: [Equation]
Revised: [Equation]

Original: Lemma G.12.
Revised: Lemma G.12.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have
Revised: n≤ [Equation] [Equation]

Original: E
Revised: k

Original: [Equation]
Revised: [Equation]

Original:  where G 5 =d (cid:113)
Revised: 1 and k ≥ 4 , then we have [Equation] where G 5 =

Original: [Equation]
Revised: [Equation]

Original: 23 β n 2 (cid:16) β 2 n 1 2 n 3 △
Revised: (cid:113)

Original: [Equation]
Revised: [Equation]

Original: We relegate the proof of Lemma G.11 and G.12 to Appendix G.5 and G.6.
Revised: We relegate the proof of Lemma G.11 and G.12 to Appendix G.5 and G.6.

Original: Now we bound (2) .
Revised: Now we bound (2) .

Original: This part involves the difficulties (i), (ii) and (iii) mentioned in Appendix G.1.
Revised: This part involves the difﬁculties (i), (ii) and (iii) mentioned in Appendix G.1.

Original: We bound (2) in Lemma G.13.
Revised: We bound (2) in Lemma G.13.

Original: Lemma G.13.
Revised: Lemma G.13.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have 
Revised: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have E

Original: [Equation]
Revised: [Equation]

Original: The constant terms G 6 and G 7 are specified in Appendix G.7.
Revised: The constant terms G 6 and G 7 are speciﬁed in Appendix G.7.

Original: Proof is shown in Appendix G.7.
Revised: Proof is shown in Appendix G.

Revised: [Equation]

Original: Combining Lemma G.11, G.12 and G.13 together, we conclude the proof.
Revised: Combining Lemma G.11, G.12 and G.13 together, we conclude the proof.

Original: [Equation]
Revised: [Equation]

Original: G.5 Proof of Lemma G. 
Revised: G.5 Proof of Lemma G.

Original: By definition of M l,k and M ′ l,k (see (40) and (41)), they only differ when k ≤ 0 .
Revised: By deﬁnition of M l,k and M (cid:48) l,k (see (41) and (42)), they only differ when k ≤ 0 .

Original: More specifically,
Revised: More speciﬁcally,

Original: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)
Revised: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: M l,k

Original: [Equation]
Revised: [Equation]

Original: l,k [Equation] + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + (cid:16) β ( k − 1) n +2− β kn +1(cid:17)
Revised: 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + (cid:16) β ( k − 1) n +2− β kn +1(cid:17)

Original: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) -

Revised: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) -

Original: -th epoch + (cid:16) β
Revised: -th epoch + (cid:16) β

Original: ( k − 1) n + n +2− β
Revised: ( k − 1) n + n +2− β

Original: ( k +1) n + n(cid:17) ∂
Revised: ( k +1) n + n(cid:17) ∂

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +2− β
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +2− β

Original: ( k +1) n + n + n − 1(cid:17)
Revised: ( k +1) n + n + n − 1(cid:17)

Original: ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n + n +1(cid:17)
Revised: ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n + n +1(cid:17)

Original: ∂ l f n − 1 ( x 1 , 0 )
Revised: ∂ l f n − 1 ( x 1 , 0 )

Original: + (cid:16) β
Revised: + (cid:16) β

Original: ( k − 1) n +3− β
Revised: ( k − 1) n +3− β

Original: ( k +1) n +1(cid:17) ∂
Revised: ( k +1) n +1(cid:17) ∂

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β ( k +1) n + n(cid:17) ∂ l f n − 1 ( x 1 ,0 ) (cid:124) [Equation]
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β ( k +1) n + n(cid:17) ∂ l f n − 1 ( x 1 ,

Original: [Equation]
Revised: [Equation]

Original: We start with the 1st column in the ‘0-th epoch’
Revised: We start with the 1st column in the ‘0-th epoch’

Original: (from the bottom to the top).
Revised: (from the bottom to the top).

Original: The 1st column in the ‘0-th epoch’ 
Revised: The 1st column in the ‘0-th epoch’ =

Revised: β (

Original: [Equation]
Revised: [Equation]

Original:  Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can bebounded as follows.
Revised: − [Equation] Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can be bounded as follows.

Original: The last column in the ‘0-th epoch’ =
Revised: The last column in the ‘0-th epoch’ =

Original: β ( k − 1) n +

Revised: β ( k − 1) n +

Original: [Equation]
Revised: [Equation]

Original: Summing up all the columns, we have
Revised: Summing up all the columns, we have

Original: The ‘0-thepoch’
Revised: The ‘0-th

Original: [Equation]
Revised: [Equation]

Original: Using the same technique, it can be shown that The
Revised: Using the same technique

Original: ‘-1-th epoch’ ≤ β

Original: [Equation]
Revised: [Equation]

Original: k −
Revised: it can be shown that

Original: [Equation]
Revised: [Equation]

Original: The ‘-2-th epoch’ ≤ β k −
Revised: th

Original: [Equation]
Revised: [Equation]

Revised: Plugging all these results in (48)

Original: [Equation]
Revised: [Equation]

Original: Plugging all these results in (47), we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: That is to say, when k is large enough such that β ( k − 1)
Revised: That is to say, when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 we have 
Revised: n≤ β n 1 √ k − 1 we have (cid:12)(cid:12) [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma G.10, the proof is completed.
Revised: Combining with Lemma G.10, the proof is completed.

Original: G.6 Proof of Lemma G. 
Revised: G.6 Proof of Lemma G.

Original: We start with (cid:16) M ′ l,k (cid:17) 0 − ( F l,k ) 0 . 
Revised: We start with (cid:16) M (cid:48) l,k (cid:17) 0 − ( F l,k ) 0 . (cid:0) M (cid:48) l

Original: [Equation]
Revised: [Equation]

Original: M ′ l
Revised: k

Original: [Equation]
Revised: [Equation]

Original: k (cid:1)

Original: [Equation]
Revised: [Equation]

Original: F

Original: [Equation]
Revised: [Equation]

Revised: (cid:125)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We further have
Revised: We further have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Using the same calculation
Revised: Using the same calculation

Original: [Equation]
Revised: [Equation]

Original: we can get the following result:
Revised: we can get the following result:

Original: [Equation]

Original: [Equation]

Original: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: (cid:1) n −

Original: [Equation]

Original: i =

Original: | ∂ l f i ( x 1 , 0 ) | .

Original: Repeat this calculation, we have
Revised: Repeat this calculation, we have

Original: [Equation]
Revised: [Equation]

Original: When β ( k − 1)
Revised: When β ( k − 1)

Original: n≤ β n 1 √ k − 1 , we further have:
Revised: n≤ β n 1 √ k − 1 , we further have:

Original: [Equation]
Revised: [Equation]

Original: k, where ( ∗ ) : 1 √ n ( k − 1) − 1 ≤ (cid:113) 2 nk when k ≥ 4 .
Revised: where ( ∗ ) : [Equation] ≤ (cid:113) 2 nk when k ≥ 4 .

Original: Therefore, we have
Revised: Therefore, we have

Original: (cid:12)(cid:12)(cid:12)(cid:0)

Original: [Equation]
Revised: [Equation]

Original: ˜

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma G.10, the proof is completed with constant G 5 defined in Lemma G.12.
Revised: Combining with Lemma G.10, the proof is completed with constant G 5 deﬁned in Lemma G.12.

Original: G.7 Proof of Lemma G. 
Revised: G.7 Proof of Lemma G.

Original: In this section, we derive a lower bound for E (cid:104)(cid:80)
Revised: In this section, we derive a lower bound for E (cid:104)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).
Revised: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).

Original: We first rewrite “ l large" intoindicator function as follows:
Revised: We ﬁrst rewrite “ l large" into indicator function as follows:

Original: [Table]
Revised: [Table]

Original: where I k :
Revised: where I k :

Original: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := △ 1 n √ n √ k
Revised: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := (cid:52) 1 n √ n √ k

Original: [Table]
Revised: [Table]

Revised: [Table]

Original:  n 2 (cid:17) .
Revised: − β 2 ) n β n 2 (cid:17) .

Original: We also define I k − j :=
Revised: We also deﬁne I k − j :=

Original: I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , it will be used later.
Revised: I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , it will be used later.

Original: We take the conditional expectation over the history before x k, 0 .
Revised: We take the conditional expectation over the history before x k, 0 .

Original: We first focus on
Revised: We ﬁrst focus on

Original: E k (cid:104)(cid:80) dl
Revised: E k (cid:104)(cid:80) dl

Original: =1{+=
+} I k ∂ [Table] f ( x k, 0 ) √ v k, 0 (cid:16)(cid:16) 
Revised: {+=
+} I k ∂ l f ( x k, 0 ) √ v k, 0 (cid:16)(cid:16) M (cid:48)

Original: M ′ l,k (cid:17) k − ( F l,k ) k (cid:17)(cid:105)
Revised: l,k (cid:17) k − ( F l,k ) k (cid:17)(cid:105)

Original: and we delegate the history part for later analysis.
Revised: and we delegate the history part for later analysis.

Original: For each possible (cid:16) M ′ l,k (cid:17) k − ( F l,k ) k , we first convert all x k,i into x k, 0 using Lemma F.2.
Revised: For each possible (cid:16) M (cid:48) l,k (cid:17) k − ( F l,k ) k , we ﬁrst convert all x k,i into x k, 0 using Lemma F.2.

Original: Since(cid:16)
Revised: Since [Equation]

Revised: M

Original: M ′ l,k (cid:17) k contains ( n − 1)+ · ·
Revised: [Equation] l,k [Equation] k contains ( n − 1)

Revised: +

Revised: [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Original:  terms of x k,i (with i ̸ = 0 ), we have
Revised: n [Equation] [Equation] terms of x k,i (with i (cid:54) = 0 ), we have

Revised: [Equation]

Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Using the same strategy for analyzing the color-ball toy example
Revised: Using the same strategy for analyzing the color-ball toy example, we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the followingderivation.
Revised: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the following derivation.

Original: Now we move one step further exert E k − 1 ( · ) .
Revised: Now we move one step further exert E k − 1 ( · ) .

Original: In particular, we need to calculate
Revised: In particular, we need to calculate

Original: E k
Revised: E k

Original: [Equation]
Revised: [Equation]

Original: (cid:17)(cid:35)(cid:41)
Revised: 

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised:  

Revised: [Equation]

Original: The blue term is the residue from the k -th epoch.
Revised: The blue term is the residue from the k -th epoch.

Original: The red term is the main component in the ( k − 1) -thepoch.
Revised: The red term is the main component in the ( k − 1) -th epoch.

Original: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitzproperty.
Revised: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitz property.

Original: First of all, we work on ( a ) .
Revised: First of all, we work on ( a ) .

Original: [Figure]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: LemmaG.2 ≥

Original: [Table]
Revised: [Table]

Original: where I k,k − j :=
Revised: where I k,k − j :=

Original: I (cid:16) max i | ∂ l f i ( x k, 0 )
Revised: I (cid:16) max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and˜ I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and ˜ I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .

Original: By Lemma G.3,we know I k,k − 1 =
Revised: By Lemma G.3, we know I k,k − 1 =

Original: I k − 1 , so we have:
Revised: I k − 1 , so we have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now we derive a lower bound for (b).
Revised: Now we derive a lower bound for (b).

Original: Similarly as before, we rewrite (cid:16) M ′ l,k (cid:17) k − 1 and ( F l,k ) k − 1 asfollows.
Revised: Similarly as before, we rewrite (cid:16) M (cid:48) l,k (cid:17) k − 1 and ( F l,k ) k − 1 as follows.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We now calculate ( b ) .
Revised: We now calculate ( b ) .

Original: Using the same idea as in ( a ) , we have
Revised: Using the same idea as in ( a ) , we have

Original: [Equation]
Revised: [Equation]

Revised: b

Original: [Table] [Equation] 
Revised: [Equation] [Table]

Original: To proceed, we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:12)(cid:12)(cid:12) .
Revised: To proceed, we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:12)(cid:12)(cid:12) .

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .
Revised: [Table] [Equation] [Table] The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .

Original: Now we have derived lower bounds for both ( a ) and ( b ) .
Revised: Now we have derived lower bounds for both ( a ) and ( b ) .

Original: Combining together, we have:
Revised: Combining together, we have:

Original: [Table]
Revised: [Table]

Original: 

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: For the first term in the above inequality
Revised: For the ﬁrst term in the above inequality

Original: [Equation]
Revised: [Equation]

Original: we can calculate it using the idea in the color-ball toyexample:
Revised: we can calculate it using the idea in the color-ball toy example:

Original: [Equation]
Revised: [Equation]

Original: To proceed
Revised: To proceed

Original: [Equation]
Revised: [Equation]

Original: we further take E k − 2 ( · ) and bound
Revised: we further take E k − 2 ( · ) and bound

Original: E k − d
Revised: E k − 

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original:   .
Revised:   .

Original: Repeat this process until k = 1 , we have:
Revised: Repeat this process until k = 1 , we have:

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: [Table]

Revised: Error

Original: [Equation]
Revised: [Equation]

Original: [Table]

Revised: Error

Original: [Equation]
Revised: [Equation]

Original: + Error 1 + Error 2 + Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .
Revised: + Error 1 + Error 2 + Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .

Original: We specify Error 1 , Error 2 , Error 3 as follows.
Revised: We specify Error 1 , Error 2 , Error 3 as follows.

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: − d
Revised: − d

Original: [Table] [Table]
Revised: [Table] [Table]

Original: β
Revised: β

Original: [Table]
Revised: [Table]

Original: ) △ n
Revised: 2 | J 1 | n [ Q k + Q k − 1 + β n

Original: [Table]
Revised: [Table]

Original: d
Revised: Q k + Q k −

Original: [Table]
Revised: [Table]

Original: β n 2 | J 1 | n [ Q k + Q k − 1 + β n 1 ( Q k + Q k − 1 + Q k − 2 )
Revised: + Q k − [Table]

Original:  + 
Revised: ) + · ·

Original: [Table]
Revised: [Table]

Original: Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a finite constant, we have (cid:80) ∞
Revised: Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a ﬁnite constant, we have (cid:80) ∞

Original: j =0 β jn 1 △ n ( k − j ) = {+(cid:52) 
+}
Revised: j =0 β jn 1 (cid:52) n ( k − j ) = {+(cid:52) 
+}

Original: △ 1 δ 2 √ nk .
Revised: δ 2 √ nk .

Original: Inaddition, we have[ Q
Revised: In addition, we have

Original: [Equation]
Revised: [Equation]

Original: + Q k − 1 + β n 1 ( Q k + Q k − 1 + Q k − 2 ) + · · · ] ( ∗ ) ≤ Q k (1 + β n 1 + · · · ) + Q k − 1 + β n 1 ( Q k − 1 + Q k −
Revised: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj =

Original: [Equation]
Revised: [Equation]

Original: [-where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj = 
-]
Revised: [-where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj = 
-]√

Original: √ k − j ≤ 2 J √ k − 1 .
Revised: [Equation] − 1 .

Original: This inequality can be simply proved by taking the integral over 1 √ k .
Revised: This inequality can be simply proved by taking the integral over 1 √ k .

Original: The final inequality is due to
Revised: The ﬁnal inequality is due to

Original: Q k − 1 < 2 Q k .
Revised: Q k − 1 < 2 Q k .

Original: Now, we have
Revised: Now, we have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Based on the calculation in Lemma F.
Revised: Based on the calculation in Lemma F.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have
Revised: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have

Original: [Equation]
Revised: [Equation]

Original: Using the fact that
Revised: Using the fact that

Original: [Equation]
Revised: [Equation]

Original: for k ≥ 2 )

Original: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: +
Revised: d

Original: [Equation]
Revised: [Equation]

Original: + [Equation] + Error
Revised: n

Original: [Equation]
Revised: [Equation]

Original: where
Revised: i = |

Original: [Equation]
Revised: [Equation]

Original: where J 1 =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: β
Revised: where

Original: [Equation]
Revised: [Equation]

Original: − · · [Equation] , δ
Revised: where J

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: kk − j (if needed, wecan further bound J 1 by n for simplicity).
Revised: (if needed, we can further bound J 1 by n for simplicity).

Original: This conclude the proof.
Revised: This conclude the proof.

Original: G.8 Proof of Lemma G. 
Revised: G.8 Proof of Lemma G.

Original: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .
Revised: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .

Original: The upper bound for (cid:80) l large ( a 2 ) isvery straightforward using inequality (23).(cid:88)
Revised: The upper bound for (cid:80) l large ( a 2 ) is very straightforward using inequality (24).

Original: [Equation]
Revised: [Equation]

Original: Now we shift gear to (cid:80)l large ( b 2 ) : = δ
Revised: Now we shift gear to (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ρ 23 β
Revised: large

Original: [Equation]
Revised: [Equation]

Original: m
Revised: b

Original: [Equation]

Original: − ∂

Original: [Equation]
Revised: [Equation]

Original: f

Original: [Equation]
Revised: [Equation]

Original: ) | .
Revised: ) | .

Original: To proceed, weneed an upper bound for (cid:80) l large (cid:80) n − 1 i =

Revised: To proceed, we need an upper bound for (cid:80) l large (cid:80) n − 1 i =

Original: | m l,k,i − ∂ l f i ( x k, 0 ) | .
Revised: | m l,k,i − ∂ l f i ( x k, 0 ) | .

Original: For each i , we perform the followingdecomposition.
Revised: For each i [Equation] we perform the following decomposition. [Equation]

Original: [Equation]
Revised: [Equation]

Original: To start, we bound ( d 1 ) .
Revised: To start, we bound ( d 1 ) .

Original: [Equation]
Revised: [Equation]

Original: Now, we bound ( d 2 ) .
Revised: Now, we bound ( d 2 ) .

Original: Recall (cid:80) n − 1 i =

Revised: Recall (cid:80) n − 1 i =

Original: ∂ l f i ( x 1 , 0 ) =
Revised: ∂ l f i ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 ) , we have[Equation] l
Revised: ∂ l f ( x 1 , 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.[Equation] [Table] [Equation]
Revised: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80)Jj =
Revised: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80)

Original: [Equation]
Revised: [Equation]

Original: ≤
Revised: =

Original: [Equation]
Revised: [Equation]

Original: k − 1 .

Original: Plugging (62) into (61) and re-arranging the index, we have
Revised: Plugging (63) into (62) and re-arranging the index, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: ) |

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: d

Original: [Table]
Revised: [Table]

Original: |

Original: [Table]
Revised: [Equation]

Original: f

Original: [Table]

Original: ) | + (cid:114)

Original: [Table]

Original: (cid:33) + 2 qd

Original: [Table]

Original: (cid:112) n ( k [Table] 1) −(cid:35) + β (
Revised: where the second last inequality holds because: when k ≥ 4 ,

Original: [Table]

Original: [Table]

Original: d (cid:88) l

Original: [Table]

Original: 1 (cid:88)

Original: [Table]

Original: | ∂

Original: [Table]

Original: f i (

Original: [Table]

Original: [Table]

Original: ) |

Original: [Table]

Original: n

Original: [Table] | [Table]
Revised: [Equation]

Original: f

Original: [Table]

Original: |

Original: [Table]

Original: (cid:114)

Original: [Table]

Original: D

Original: [Table]

Original: d

Original: [Table]

Original: + 1 −

Original: [Table]

Original: 2 d [Table] (cid:112) n ( k − 1) −+ β [Table]
Revised: i ) holds when k is large enough such that β ( k − 1) n

Original: [Table]

Original: d

Original: [Table]
Revised: [Equation]

Original: l

Original: [Table]

Original: n
Revised: k

Original: [Table]

Original: i

Original: [Table]

Original: | ∂

Original: [Table]

Original: f

Original: [Table]

Original: [Table]

Original: [Table]

Original: ) |

Original: When

Original: [Table]

Original: √ D

Original: [Table]

Original: d

Original: [Table]

Original: | ∂

Original: [Table]

Original: f

Original: [Table]

Original: [Table]

Original: ) | + (cid:114) D

Original: [Table]

Original: D

Original: [Table]

Original: d (cid:33) + 1 − β2 √ 2 d △ 1 √ nk + β ( k

Original: [Table]

Original: [Equation]

Original: where the second last inequality holds because: when k ≥ 4 ,

Original: [Equation]
Revised: [Equation]

Original: √ nk . ( i ) holds when kis large enough such that β ( k − 1) n [Equation] Further
Revised: Further, β n 1 √ k − 1 ≤ √

Original: [Equation]
Revised: [Equation]

Original:  k when k ≥ 2 .
Revised: 1 √ k when k ≥ 2 .

Original: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .
Revised: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .

Original: Plugging (63) and (60) into (59), we concludethe proof.
Revised: Plugging (64) and (61) into (60), we conclude the proof.

Original: [Table]
Revised: [Table]

Original: ∂

Original: [Equation]
Revised: [Equation]

Original: The proof is concluded by adding expectation on both sides of the inequality.
Revised: The proof is concluded by adding expectation on both sides of the inequality

Revised: [Equation]

Original: G.9 Proof of Lemma G. 
Revised: G.9 Proof of Lemma G.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.
Revised: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.

Original: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have 
Revised: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: Now, we discuss two cases.
Revised: Now, we discuss two cases.

Original: Case (a): when | ∂ α f ( x k, 0 )
Revised: Case (a): when | ∂ α f ( x k, 0 )

Original: | ≥ 4{+
+} √∆ 1 (1 − β 2 ) √ D 1 nkd .
Revised: | ≥ {+
+} √∆ 1 (1 − [Equation] 2 ) √ D 1 nkd .

Original: In this case, we have the following result.
Revised: In this case, we have the following result.

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ≥
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original: Now we consider the following two sub-cases.
Revised: Now we consider the following two sub-cases.

Original: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d :
Revised: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d :

Original: [Equation]
Revised: [Equation]

Original: • When ∂ α f
Revised: • When ∂ α f

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: x k

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Combining together
Revised: Combining together

Original: [Equation]
Revised: [Equation]

Original: we have the following results for Case (a) 
Revised: we have the following results for Case (a) .

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: Case (b): When | ∂ α f ( x k, 0 ) | < 4 √
Revised: Case (b): When | ∂ α f ( x k 0 ) | < 4 √

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original:  we have
Revised: , we have

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where G 1 :
Revised: where G 1 :

Original: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .
Revised: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .

Original: Now, the following two claims are both true.
Revised: Now, the following two claims are both true.

Original: • Claim 1:
Revised: • Claim 1:

Original: [Equation]
Revised: [Equation]

Original: Claim
Revised: Claim

Original: [Equation]
Revised: [Equation]

Original: Combining Claim 1 and Claim
Revised: Combining Claim 1 and Claim

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where G 2 :=
Revised: where G 2 :=

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Combining Case (a) and Case (b) together, we have
Revised: Combining Case (a) and Case (b) together, we have

Original: [Equation]
Revised: [Equation]

Revised: where the last inequality is because of (cid:107)∇ f ( x k, 0 ) (cid:107) 22 ≤ d∂ α f

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because of ∥∇ f ( x k, 0 )
Revised: f [Equation] x [Equation]

Original: ∥ 22 ≤ d∂
Revised: (cid:107) [Equation] d [Equation]

Revised: α f ( x k

Original: [Equation]
Revised: [Equation]

Original:  Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − ϵ, 1] , such that 1 √ 5 D 1 d − F 3 ≥ 1 √ 10 D 1 d ,or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma
Revised: 0 ) | . Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − (cid:15), 1] , such that [Equation] or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma

Original: G.6).
Revised: G.6).

Original: With such a choice of β 2 , we have the following results by changing all the F 3 into 
Revised: With such a choice of β 2 , we have the following results by changing all the F 3 into 1 √ 10 D 1 d :

Original: [Equation]
Revised: E   (cid:88) l large

Original: [Equation]
Revised: [Equation]

Original: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .
Revised: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .

Original: This is because: first, F 4 = F 3 (cid:113) D 0 D 1 d ;second, F 3 ≤ 1 √ 10 D 1 d .
Revised: This is because: ﬁrst, F 4 = F 3 (cid:113) D 0 D 1 d ; second, F 3 ≤ 1 √ 10 D 1 d .

Original: The proof of Lemma G.6 is completed.
Revised: The proof of Lemma G.6 is completed.

Original: We restate all the constants as follows ( G 4 , G 5 , G 6 are specified in Appendix G.7):
Revised: We restate all the constants as follows ( G 4 , G 5 , G 6 are speciﬁed in Appendix G.7):

Original: [Equation]
Revised: [Equation]

Original: Remark G.14 .
Revised: Remark G.14 .

Original: We comment that we can always replace F 5 in the final result by F 4 , i.e., we can choosenot to apply the last inequality ( ∗ ) in the proof.
Revised: We comment that we can always replace F 5 in the ﬁnal result by F 4 , i.e., we can choose not to apply the last inequality ( ∗ ) in the proof.

Original: The benefit of using F 4 is that F 4 monotonouslydecrease to 0 when increasing β 2 to 1.
Revised: The beneﬁt of using F 4 is that F 4 monotonously decrease to 0 when increasing β 2 to 1.

Original: This monotone property is not shown in the notation of F 5 .
Revised: This monotone property is not shown in the notation of F 5 .

Original: Nevertheless, we choose to use F 5 since it is a much cleaner constant.
Revised: Nevertheless, we choose to use F 5 since it is a much cleaner constant.

Original: G.10 Proof of Lemma G. 
Revised: G.10 Proof of Lemma G.

Original: Based on Descent Lemma, we have
Revised: Based on Descent Lemma, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging in the r.h.s.
Revised: Plugging in the r

Original: of (34), we have the following relation after rearranging.
Revised: [Equation]

Original: T
Revised: h

Original: [Equation]
Revised: [Equation]

Original: Recall we have 2 (cid:16) √ T −
Revised: s. of (35)

Original: [Equation]
Revised: [Equation]

Original: t
Revised: we have the following relation after rearranging.

Original: [Equation]
Revised: [Equation]

Original: −
Revised:  .

Original: [Equation] (cid:80)[Equation]
Revised: Recall we have 2 (cid:16) √ T − √ t 0 − 1 (cid:17) ≤ (cid:80)

Original: [-= t 
-]
Revised: [-= t 
-]Tk

Original: [Equation] k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 .
Revised: = t 0 1 √ k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 .

Original: Further, since ∥·∥ 1 ≥ ∥·∥ 2 ,we get the following relation when t 0 = 1 .
Revised: Further, since (cid:107)·(cid:107) 1 ≥ (cid:107)·(cid:107) 2 , we get the following relation when t 0 = 1 .

Original: [Equation]
Revised: [Equation]

Original: We specify all the constant as follows.
Revised: We specify all the constant as follows.

Original: For all the following constants, We keep the same notation astheir appearance in their corresponding lemmas.
Revised: For all the following constants, We keep the same notation as their appearance in their corresponding lemmas.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: where J 1 = (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − [-
-]
Revised: where J 1 = (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − [Equation] n β n − [-
-]1(cid:1) , δ

Original: [Equation]
Revised: [Equation]

Revised: = lim k

Revised: [Equation]

Revised: (cid:80)

Revised: [Equation]

Original: Ifneeded, we can further bound J 1 by n for simplicity.
Revised: = [Equation] β 1 . If needed, we can further bound J 1 by n for simplicity.

Original: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .
Revised: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .

Original: Usually, theseconstants can be different for different problems.
Revised: Usually, these constants can be different for different problems.

Original: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.[Equation] [Equation]
Revised: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n . ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma G.9 is now completed.
Revised: The proof of Lemma G.9 is now completed.

Original: This also concludes the whole proof of Theorem 3.1.
Revised: This also concludes the whole proof of Theorem 3.1.