Original: we have the following result when k is large enough such that β ( k − 1)
Revised: we have the following result when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 4 :
Revised: n≤ β n 1 √ k − 1 and k ≥ 4 :

Original: [Equation]
Revised: [Equation]

Original: (cid:40)(cid:114)

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] Proof of Lemma G.7 can be seen in Appendix G.9.
Revised: Proof of Lemma G.6 can be seen in Appendix G.9.

Original: Remark G.8 .
Revised: Remark G.7 .

Original: Condition (32) specifies the smallest threshold of β 2 to ensure the convergence.
Revised: Condition (33) specifies the smallest threshold of β 2 to ensure the convergence.

Original: Thiscondition can be translated into the threshold funtion γ 1 ( n ) mentioned in Theorem 3.1.
Revised: Thiscondition can be translated into the threshold funtion γ 1 ( n ) mentioned in Theorem 3.1.

Original: As a roughestimate, Lemma G.7 requires β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) / ( n 2 ρ ) (cid:1) , where ρ = ρ 1 ρ 2 ρ 3 .
Revised: As a roughestimate, Lemma G.6 requires β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) / ( n 2 ρ ) (cid:1) , where ρ = ρ 1 ρ 2 ρ 3 .

Original: Asdiscussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: Asdiscussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: When ρ 1 , ρ 2 , ρ 3 achieve their upper boundat the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .
Revised: When ρ 1 , ρ 2 , ρ 3 achieve their upper boundat the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: However, ρ ishighly dependent on the problem instance f ( x ) and training process.
Revised: However, ρ ishighly dependent on the problem instance f ( x ) and training process.

Original: Our experiments in Appendix
Revised: Our experiments in Appendix

Original: D.2 shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lowerthan it appears to be.
Revised: B.1 shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lowerthan it appears to be.

Original: We further remark that γ 1 ( n ) ∝ (1 − β n 1 ) .
Revised: We further remark that γ 1 ( n ) ∝ (1 − β n 1 ) .

Original: That is to say, as β 1 increases, the threshold of β 2 alsoslowly increases (to ensure the convergence).
Revised: That is to say, as β 1 increases, the threshold of β 2 alsoslowly increases (to ensure the convergence).

Original: However, such phenomenon is not observed in ourexperiments.
Revised: However, such phenomenon is not observed in ourexperiments.

Original: We conjecture γ 1 ( n ) should be independent of β 1 , removing the dependence (1 − β n 1 )is considered as an interesting future direction.
Revised: We conjecture γ 1 ( n ) should be independent of β 1 , removing the dependence (1 − β n 1 )is considered as an interesting future direction.

Original: Remark G.9 .
Revised: Remark G.8 .

Original: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.
Revised: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.

Original: Thiscan be seen in the proof in Appendix G.9 (the definition of F 4 ) and Remark G.17.
Revised: Thiscan be seen in the proof in Appendix G.9 (the definition of F 4 ) and Remark G.14.

Original: Based on Lemma G.7, we can further rewrite (28) as follows.
Revised: Based on Lemma G.6, we can further rewrite (30) as follows.

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 3.1 is concluded by plugging (33) into Descent Lemma (22) and then takingtelescope some from k = t 0 to k = T .
Revised: The proof of Theorem 3.1 is concluded by plugging (34) into Descent Lemma (24) and then takingtelescope some from k = t 0 to k = T .

Original: These steps are quite standard in non-convex optimization.
Revised: These steps are quite standard in non-convex optimization.

Original: We finish these calculation in Lemma G.

Revised: We finish these calculation in Lemma G.

Original: Lemma G.10.
Revised: Lemma G.9.

Original: When inequality (33) holds, we have the following results based on Descent Lemma(22) :
Revised: When inequality (34) holds, we have the following results based on Descent Lemma(24) :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma G.10 can be seen in Appendix G.10.
Revised: Proof of Lemma G.9 can be seen in Appendix G.10.

Original: Now, the whole proof of Theorem 3.1 iscompleted.
Revised: Now, the whole proof of Theorem 3.1 iscompleted.

Original: G.2 Proof of Lemma G. 
Revised: G.2 Proof of Lemma G.

Original: To prove Lemma G.1, we first show the following result.

Original: Lemma G.11.

Original: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfiesmax i | ∂ l f i ( x k, 0 ) |

Original: [Equation]

Original: Q k :=

Original: [Equation]

Original: n 2 we have

Original: [Equation]

Original: The proof idea of Lemma G.11 is similar as Lemma F.3 in (Shi et al., 2020), but with the followingdifferences:

Original: The condition of Lemma F.3 in (Shi et al.,

Original: requires the full batch gradient| ∂ l f ( x k, 0 ) | ≥ nQ k , which is different from our condition.

Original: Here, we choose the condition of Lemma G.11 because it meets the need of our later decomposition strategy in (28).2.

Original: The constant Q k is different.

Original: The difference is due to the different update formula ofRMSProp and general Adam.

Original: Anyhow, Lemma G.11 can be easily proved following most of the steps in Lemma F.3 (Shi et al.

Original: We omit the proof for brevity.

Original: To prove Lemma G.1, we only prove (25).
Revised: To prove Lemma G.1, we only prove (27).

Original: The proof for (24) can be analogized from the proof of(25).
Revised: The proof for (26) can be analogized from the proof of(27).

Original: We discuss the following two cases:
Revised: We discuss the following two cases:

Original: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have∂ l f ( x k, 0 )( m l,k,i − ∂ l f i ( x k, 0 )) √ v l,k,i
Revised: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have∂ l f ( x k, 0 )

Original: [Equation]
Revised: [Equation]

Original: ∂ l f ( x k, 0 ) ( m l
Revised: m l,k ,i

Original: [Equation]
Revised: [Equation]

Original: k [Equation]
Revised: ∂ l f

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x 
Revised: x k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  √ v 
Revised: where ( a ) is because of √ v l,k,i ≥ (cid:112) β i 2 √ v l,k, 0 . Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have∂ l f ( x k,

Original: [Equation]
Revised: [Equation]

Original: Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k,0 )) ≤ 0 , we have [Figure]
Revised: ( m l,k,i − ∂ l f i ( x k,

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is because of √ v l,k,i ≥ (cid:112) β i 2√ v l,k, 0 .
Revised: where ( ∗ ) is due to 1 √ 1+ x ≥ 1 − x 2 .

Original: Combining these two cases, we have
Revised: We further have

Original: [Figure]
Revised: [Table]

Revised: v

Original: [Equation]
Revised: [Table]

Original:  i ( x  k, 0 )) √ v l,k,(cid:18) − 1 − β (cid:18) − 1 + 4 ρ 2 i β n (cid:19)(cid:19) ≥ n − 1 (cid:88) i =
Revised: where ( ∗∗ ) is due to the condition in the Lemma max i | ∂ l f i ( x τ k, 0 ) | ≥ Q k ≥ n △ nk .

Original: ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) √ v l,k,− δ 1 (cid:12)(cid:12)(cid:12)(cid:12) ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:12)(cid:12)(cid:12)(cid:12) n − 1 (cid:88) i = | m l,k,i − ∂ l f i ( x k, 0 ) | , where δ 1 = (1 − β 2 ) (cid:16) − 1+ 4 nρ 2 βn 2 (cid:17) + (cid:18) 1 √ β n 2 −(cid:19) . Proof is completed.
Revised: Plug the aboveresult into (35) we have

Original: To prove Lemma G.3, we first show the following result.
Revised: [Equation] In the last step ( ∗ ∗ ∗ ) , we use the following Lemma G.10.

Original: Lemma G.12.
Revised: Lemma G.10.

Original: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfiesmax
Revised: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfies

Original: [Equation]
Revised: [Equation]

Original: 2 , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof idea of Lemma G.12 is silimar with Lemma F.1 in (Shi et al., 2020), but with thefollowing differences:
Revised: The proof idea of Lemma G.10 is similar as Lemma F.1 in (Shi et al., 2020), but with thefollowing differences:

Original: The condition of Lemma F.1 in (Shi et al., 2020)requires the full batch gradient
Revised: The condition of Lemma F.1 in (Shi et al., 2020)

Original:  | ∂ l f ( x k, 0 ) | ≥ nQ k , which is different from our condition.
Revised: both require the full batch gradient | ∂ l f ( x k, 0 ) | ≥nQ k , which is different from our condition.

Original: Here, we choose the condition of Lemma G.11 because it meets the need of ourlater decomposition strategy in(28).2.
Revised: Here, we choose the condition of Lemma G.10because it meets the need of our decomposition strategy in (30) .2.

Original: The constant Q k is different.
Revised: The constant Q k is different.

Original: The difference is due to the different update formula of RMSProp and general Adam.
Revised: The difference is due to the different update formula of RMSPropand general Adam.

Original:  Lemma G.12 can be  proved following most of the steps of Lemma F.1 in (Shi et al., 2020).
Revised: Anyhow, Lemma G.10 can be easily proved following most of the steps in Lemma F.1 in (Shi et al.,2020).

Original: We omitthe proof for brevity.
Revised: We omit the proof for brevity.

Revised: Combining the above Case 1 and Case 2 , we have

Revised: [Equation]

Revised: i ( x k, 0 ) | , where δ 1 = (1 − β 2 )4 nρ 2 β

Revised: n+

Original: Now we prove Lemma G.3. 
Revised: (cid:18) 1 √ β n 2 −(cid:19) . Proof is completed. G.3 Proof of Lemma G.

Original: We first prove the result for j = 1 .
Revised: We first prove the result for j = 1 .

Original: We discuss the following two cases.
Revised: We discuss the following two cases.

Original: Case 1: when ∂ l f ( x k, 0 ) √ v k,≥ [Equation] : when ∂ l f ( x k, 0 ) ≤ 0 , we have[Equation]
Revised: Case 1: when ∂ l f ( x k, 0 ) √ v l,k, 0 ≥ ∂ l f ( x k − 1 , 0 ) √ v l,k − 1 , 0 : when ∂ l f ( x k, 0 ) ≤ 0 , we have

Original: Lemma

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) > 0 , we have
Revised: When ∂ l f ( x k, 0 ) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In conclusion, we have:
Revised: In conclusion, we have:

Original: [Equation]
Revised: [Equation]

Original:  δ 1 =[Equation]
Revised: [Equation] , where δ 1 =

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:17)(cid:19) is a constant that goes to 0 when β 2 goes to 1.
Revised: is a constant that goes to 0 when β 2 goes to 1.

Original: Therefore, we have
Revised: Therefore, wehave

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  Case 2: when ∂ l f ( xk, 0 ) √ v k,
Revised: Note that Case 2: when ∂ l [Equation] x

Original: [Equation]
Revised: [Equation]

Original: : when ∂ l f ( x k, 0 ) ≥ 0 , we have
Revised: : when ∂ l f ( x k, 0 ) ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) < 0 , we have
Revised: When ∂ l f ( x k, 0 ) < 0 , we have

Original: ∂ l f ( x k, 0 ) √ v k,≥ ∂ l f ( x k, 0 ) √ v k − 1 , (cid:112) β n 2 .
Revised: [Equation]

Original: Following the same strategy as in Case 1, we can show that
Revised: Following the same strategy as in Case 1, we can show that

Original: [Equation]
Revised: [Equation]

Original: which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: Case 1 and Case 2 together, we have
Revised: Case 1 and Case 2 together, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: Now we consider the case when j > 1 .
Revised: Now we consider the case when j > 1 .

Original: Based on the above inequality, we have
Revised: Based on the above inequality, we have

Original: [Table]

Original: ∂ l f ( x k, 0 ) √

Original: [Table]

Original: k,

Original: [Table]
Revised: [Table]

Original: The proof is completed.
Revised: The proof is completed.

Original: G.4 Proof of Lemma G. 
Revised: G.4 Proof of Lemma G.

Original: To prove Lemma G.5, we need to further decompose E [ (cid:80) l large ( b 1 )] .
Revised: To prove Lemma G.4, we need to further decompose E [ (cid:80) l large ( b 1 )] .

Original: First and foremost, we write{+write(cid:80) n − 1 i =
 m l,k,i in an explicit form.m l,k,i = (1 − β [Equation]+}
Revised: First and foremost, we {+write(cid:80) n − 1 i =
 m l,k,i in an explicit form.m l,k,i = (1 − β [Equation]+}

Revised: +

Original: [Equation]
Revised: [Equation]

Revised: −

Revised: [Equation]

Original:  Since ∂ l f ( x 1 , 0 ) =
Revised: + [Equation] ∂ l f ( x 1 , 0 ) (37) Since ∂ l f ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =
Revised: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =

Original: (cid:80) [Equation]{+n − 1 i =
+}
Revised: (cid:80) {+n − 1 i =
+}

Original:  f i ( x 1 , 0 ) , wehave
Revised: ∂ l f i ( x 1 , 0 ) , wehaveβ (

Original: [Equation]
Revised: [Equation]

Original: Plugging (36) into (35), we have [Equation] i {++
+}
Revised: − 1) n + i {++
+}

Original: [Equation]
Revised: [Equation]

Original: Using (37), we have(For each ‘epoch’, we suggest readers to read from bottom to the top.) [Equation] [Table] [Equation]
Revised: Plugging (38) into (37), we have

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: ∂ l f τ k, 0 ( x k, 0 )

Original: [Table] [Equation] [Table] [Equation]
Revised: +

Original: [Table] [Equation] [Table] [Equation]
Revised: β

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: i +11 f τ k − 1 ,n − 1 ( x k − 1 ,n − 1 ) + · · · + β i

Original: [Equation] [Table] [Equation] [Table]
Revised: + nf τ k − 1 , 0 ( x k − 1 , 0 ) + · · · + β ( k − 2) n

Original: [Equation] [Table] [Equation] [Table]
Revised: + i +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + i∂ l f τ 1 , 0 ( x 1 , 0 )

Original: [Equation] [Table] [Equation] [Table]
Revised: + β ( k − 1) n

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: + i +1∂

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation][Table] 0 ( x 1 , 0 ) + · · · + β ( k − 2) n +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1)
Revised: [Equation]

Original: n∂ l f τ 1 , 0 ( [-x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 
-]
Revised: [Equation] [Equation] ( [-x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 
-]k − 1) n [Equation]

Original: {+i +
+}
Revised: {+i +
+}

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: ( x

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: {+1) n + i +
+}
Revised: {+1) n + i +
+}

Revised: [Equation]

Revised: l f 0 ( x 1 , 0 ) + · · ·

Revised: [Equation]

Revised: β ( k − 1) n

Original: {++ i +
+}
Revised: {++ i +
+}

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: Using (39), we have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Revised: [Equation]

Revised: [Equation]

Original: {+( x 1 ,n − 1 ) + · · · + β ( k − 1) n + n − 2∂ l f τ 1 , 0 ( x 1 , 0 ) + · · · + β ( k − 2) n +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 
+}
Revised: {+( x 1 ,n − 1 ) + · · · + β ( k − 1) n + n − 2∂ l f τ 1 , 0 ( x 1 , 0 ) + · · · + β ( k − 2) n +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 
+}

Original: -thepoch
Revised: -thepoch

Original: + β ( k − 1) n + n∂ l f 0 ( x 1 , 0 )
Revised: + β ( k [Equation] [Equation]

Revised: ∂

Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: [Equation]

Revised: ( k

Revised: [Equation]

Revised: l f n

Revised: [Equation]

Revised: [Equation]

Revised: l f

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: ( k

Revised: [Equation]

Revised: 1) n + n − 1∂ l f n

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Original: + · · · + β ( k − 1) n + n∂ l fn − 1 ( x 1 , 0 )
Revised: [Equation] ) n [Equation] ∂ l f 0 (

Revised: [Equation]

Revised: [Equation]

Original: + β ( k − 1) n + n − 1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n − 1∂ l f n − 1 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f n − 1 ( x 1 , 0 ) 
Revised: [Equation] [Equation] ( k − 1) n +1∂ l f n − 1 ( x 1 , 0 ) [Equation]

Original: + (cid:124) (cid:123)(cid:122) (cid:125) 

Revised: (cid:124) (cid:123)(cid:122) (cid:125)

Original: -thepoch ... } (38)
Revised: -thepoch ... } (40)

Original: Note that in (38), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition willcause trouble in bounding the (cid:80) n − 1 i =

Revised: Note that in (40), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition willcause trouble in bounding the (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) .

Original: Therefore, we need to define an auxillarysequence M ′ l,k which does not involve such a phase transition.
Revised: Therefore, we need to define an auxillarysequence M ′ l,k which does not involve such a phase transition.

Original: We define M ′ l,k as follows.
Revised: We define M ′ l,k as follows.

Original: (For each‘epoch’, we suggest readers to read from bottom to the top.)
Revised: (For each‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation]
Revised: [Equation]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [-) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 1 -thepoch + β ( k − 1) n + n∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n + n − 1∂ l f n − 1 ( x 1 , 0 ) + β ( k − 1) n + n − 1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n +2( n − 1)∂ l f n − 1 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n∂ l f n − 1 ( x 1 , 0 )+ (cid:124) (cid:123)(cid:122) (cid:125) 
-]
Revised: [-) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 1 -thepoch + β ( k − 1) n + n∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n + n − 1∂ l f n − 1 ( x 1 , 0 ) + β ( k − 1) n + n − 1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n +2( n − 1)∂ l f n − 1 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n∂ l f n − 1 ( x 1 , 0 )+ (cid:124) (cid:123)(cid:122) (cid:125) 
-][Equation] [Table] [Equation] ∂

Original: -thepoch ... } (39)
Revised: [Equation] [Equation]

Original: Now we bound (cid:80) n − 1 i =

Revised: Now we bound (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M ′ l,k .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M ′ l,k .

Original: Denoting F l,k :=(cid:80) n − 1 i =

Revised: Denoting F l,k :=(cid:80) n − 1 i =

Original: ∂ l f i ( x k, 0 ) , we have:n
Revised: ∂ l f i ( x k, 0 ) , we have:

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k . 
Revised: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k . F l [Equation] k :

Original: [Equation]
Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: (cid:88) i =

Revised: [Equation]

Revised: + ...

Revised: +

Revised: [Equation]

Original:  For the sake of better presentation, we rewrite F l,k and M ′ l,k as follows:
Revised: [Equation] For the sake of better presentation [Equation] we rewrite F l

Original: [Equation]
Revised: [Equation]

Revised: k and M ′ l

Original:  where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .
Revised: [Equation] k as follows: [Equation] · , where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .

Original: Similarly for
Revised: Similarly for

Original: M ′ l,k .
Revised: M ′ l,k .

Original: Now, we separate M ′ l,k − F l,k into two parts.
Revised: Now, we separate M ′ l,k − F l,k into two parts.

Original: [Equation]
Revised: [Equation]

Original: Now, we rewrite E [ (cid:80) l large ( b 1 )] : 
Revised: Now, we rewrite E [ (cid:80) l large ( b 1 )] : [Equation] l large [Equation]

Revised: [Equation]

Revised: l large

Original: [Equation]
Revised: [Equation]

Revised: large

Revised: [Equation]

Revised: l large

Revised: [Equation]

Revised: (cid:124) (cid:123)(cid:122) (cid:125) (1)

Revised: [Equation]

Original: We bound (1) , (2) and (3) respectively.
Revised: We bound (1) , (2) and (3) respectively.

Original: We bound (1) and (3) in the following Lemma G.13 and
Revised: We bound (1) and (3) in the following Lemma G.11 and

Original: Lemma G.14.
Revised: Lemma G.12.

Original: Since the difference of M l,k − M ′ l,k only occurs in the high order terms of the infinitesum sequence, (1) is expected to vanish as k grows.
Revised: Since the difference of M l,k − M ′ l,k only occurs in the high order terms of the infinitesum sequence, (1) is expected to vanish as k grows.

Original: Similarly for (3) .
Revised: Similarly for (3) .

Original: Lemma G.13.
Revised: Lemma G.11.

Original: When k is large enough such that: β ( k − 1) 
Revised: When k is large enough such that: β ( k − 1) n

Original: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have 
Revised: [Equation] 1 and k ≥ 4 , then we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: where G 4 = d (cid:113) 2 ρ 2 3 β n 2 β n [-1 √ 2 n ( n − 1) (cid:80) n − 1 i =
-]
Revised: where G 4 = d (cid:113) [Equation] β n [-1 √ 2 n ( n − 1) (cid:80) n − 1 i =
-][Equation]

Original: E ( | ∂ α f i ( x 1 , 0 ) | ) .
Revised: β n [Equation] √ 2 n ( n [Equation]

Revised: (cid:80)

Revised: [Equation]

Original: Lemma G.14.
Revised: Lemma G.12.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have
Revised: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: 44) where G 5 = d (cid:113) 2 ρ 23 β n 2 (cid:16) β 
Revised: where G 5 = d (cid:113) [Equation] 23 β n 2 (cid:16) β 2 n 1 2 n 3 △

Original: [Equation]
Revised: [Equation]

Original: We relegate the proof of Lemma G.13 and G.14 to Appendix G.5 and G.6.
Revised: We relegate the proof of Lemma G.11 and G.12 to Appendix G.5 and G.6.

Original: Now we bound (2) .
Revised: Now we bound (2) .

Original: This part involves the difficulties (i), (ii) and (iii) mentioned in Appendix G.1.
Revised: This part involves the difficulties (i), (ii) and (iii) mentioned in Appendix G.1.

Original: We bound (2) in Lemma G.15.
Revised: We bound (2) in Lemma G.13.

Original: Lemma G.15.
Revised: Lemma G.13.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 2 , then we haveE
Revised: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have

Original: [Equation]
Revised: [Equation]

Original: The constant terms G 6 and G 7 are specified in Appendix G.7.
Revised: The constant terms G 6 and G 7 are specified in Appendix G.7.

Original: Proof is shown in Appendix G.7.
Revised: Proof is shown in Appendix G.7.

Original: Combining Lemma G.13, G.14 and G.15 together, we conclude the proof.[Equation] l large
Revised: Combining Lemma G.11, G.12 and G.13 together, we conclude the proof.

Original: [Equation]
Revised: [Equation]

Original: G.5 Proof of Lemma G. 
Revised: G.5 Proof of Lemma G.

Original: By definition of M l,k and M ′ l,k (see (38) and (39)), they only differ when k ≤ 0 .
Revised: By definition of M l,k and M ′ l,k (see (40) and (41)), they only differ when k ≤ 0 .

Original: More specifically,
Revised: More specifically,

Original: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)
Revised: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation] l
Revised: M l,

Original: [Equation]
Revised: [Equation]

Original: l

Original: [Equation]
Revised: [Equation]

Original:  k [Equation]
Revised: + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + (cid:16) β ( k − 1) n +2− β kn +1(cid:17)

Original: +2− β ( k − 1) n +3(cid:17) ∂ l f 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β ( k − 1) n + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) {+(cid:124) (cid:123)(cid:122) (cid:125) -
+}
Revised: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) {+(cid:124) (cid:123)(cid:122) (cid:125) -
+}

Original:  + (cid:16) β
Revised: -th epoch + (cid:16) β

Original: ( k − 1) n +1− β ( k − 1) n +2(cid:17) ∂
Revised: ( k − 1) n + n +2− β ( k +1) n + n(cid:17) ∂

Original: l f 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n+1− β ( k − 1) n +n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 0-th epoch + (cid:16) β
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +2− β

Original: ( k − 1) n + n+1− β kn + n(cid:17) ∂
Revised: ( k +1) n + n + n − 1(cid:17)

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +1− β kn + n + n − 1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17)
Revised: ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n + n +1(cid:17)

Original: ∂ l f n − 1 ( x 1 , 0 )
Revised: ∂ l f n − 1 ( x 1 , 0 )

Original: + (cid:16) β
Revised: + (cid:16) β

Original: ( k − 1) n +2− β kn +1(cid:17) 
Revised: ( k − 1) n +3− β ( k +1) n +1(cid:17) ∂

Original: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) [-(cid:123)(cid:122) (cid:125) -
 -th epoch + (cid:16)-]
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β ( k +1) n + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) [-(cid:123)(cid:122) (cid:125) -
 -th epoch + (cid:16)-][Equation]

Original: [Equation]
Revised: [Equation]

Original: We start with the 1st column in the ‘0-th epoch’
Revised: We start with the 1st column in the ‘0-th epoch’

Original: (from the bottom to the top).
Revised: (from the bottom to the top).

Original: The 1st column in the ‘0-th epoch’=
Revised: The 1st column in the ‘0-th epoch’

Original: [-β ( k − 1) n +
-]
Revised: [-β ( k − 1) n +
-]

Original: [Equation]

Original: [Equation]

Original: | ∂

Original: [Equation]
Revised: [Equation]

Original: k − [Equation] [Equation] [Equation]
Revised: Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can bebounded as follows.

Original: k −

Original: [Equation] n [Equation] | ≤ β
Revised: The last column in the ‘0-th epoch’ =

Original: [Equation]

Original:  k − [Equation] n {++
+}
Revised: β ( k − 1) n {++
+}

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation]

Original: [Equation]

Original: |

Original: [Equation]

Original: l [Equation] | (46) Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can bebounded as follows.
Revised: Summing up all the columns, we have

Original: Thelast column in the ‘0-th epoch’=
Revised: The ‘0-th epoch’

Original: [Equation]
Revised: [Equation]

Original: Summing up all the columns, we have

Original: The ‘0-th epoch’ ≤ β ( k − 1) n +1n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | ≤ β ( k − 1) n +1n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | (48)

Original: Using the same technique, it can be shown that
Revised: Using the same technique, it can be shown that

Original: The ‘-1-th epoch’ ≤ β ( k −1) n +22 n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | , (49)
Revised: The ‘-1-th epoch’ ≤ β [Equation] k −

Original:  The ‘-2-th epoch’ ≤ β ( k −1) n +33 n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | (50) · · ·
Revised: [Equation] The ‘-2-th epoch’ ≤ β ( k −

Original:  Plugging all these results in (45), we have(cid:12)(cid:12) M
Revised: [Equation] [Equation] Plugging all these results in (47), we have

Original: [Equation]
Revised: [Equation]

Original: That is to say, when k is large enough such that β ( k − 1)
Revised: That is to say, when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 we have
Revised: n≤ β n 1 √ k − 1 we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma G.12, the proof is completed.
Revised: Combining with Lemma G.10, the proof is completed.

Original: G.6 Proof of Lemma G. 
Revised: G.6 Proof of Lemma G.

Original: We start with 
Revised: We start with (cid:16) M ′ l,k (cid:17) 0 − ( F l,k ) 0 . [Equation] M ′ l

Original: [Equation]
Revised: [Equation]

Original: whereδ i = n (cid:88)
Revised: k (cid:1) 0 −

Original: [Equation]
Revised: [Equation]

Original: =
Revised: F [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∂
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We further have
Revised: We further have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: ≤ n
Revised: Using the same calculation

Original: [Equation]
Revised: [Equation]

Original: =
Revised: we can get the following result: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Using the same calculation
Revised: (cid:0)

Original: [Equation]
Revised: [Equation]

Original: we can get the following result:
Revised: (cid:1)

Original: [Equation]
Revised: [Equation]

Revised: i = |

Revised: [Equation]

Revised: l f i ( x 1 , 0 ) | .

Original: Repeat this calculation
Revised: Repeat this calculation, we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: we have
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: When β ( k − 1)
Revised: When β ( k − 1)

Original: n≤ β n 1 √ k − 1 , we further have:
Revised: n≤ β n 1 √ k − 1 , we further have:

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) :
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original:  (cid:113) 2 nk when k ≥ 4 .
Revised: k, where ( ∗ ) : 1 √ n ( k − 1) − 1 ≤ (cid:113) 2 nk when k ≥ 4 .

Original: Therefore, we have
Revised: Therefore, we have

Revised: (cid:12)(cid:12)(cid:12)(cid:0)

Original: [Equation]
Revised: [Equation]

Revised: ˜

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma G.12, the proof is completed with constant G 5 defined in Lemma G.14.
Revised: Combining with Lemma G.10, the proof is completed with constant G 5 defined in Lemma G.12.

Original: G.7 Proof of Lemma G.15 
Revised: G.7 Proof of Lemma G.

Original: In this section, we derive a lower bound for E (cid:104)(cid:80)
Revised: In this section, we derive a lower bound for E (cid:104)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).
Revised: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).

Original: We first rewrite “ l large" intoindicator function as follows:
Revised: We first rewrite “ l large" intoindicator function as follows:

Original: [Equation]
Revised: [Table]

Original: where I 
Revised: where I k :

Revised: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := △ 1 n √ n √ k

Original: [Equation]
Revised: [Table]

Revised: n 2 (cid:17) .

Original: We also define I [Equation] = [Equation]
Revised: We also define I k − j :=

Original:  | [Equation] (cid:17) , it will be used later.[Equation]
Revised: I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , it will be used later.

Original: We take the conditional expectation over the history before x k, 0 .
Revised: We take the conditional expectation over the history before x k, 0 .

Original: We first focus on
Revised: We first focus on

Original: E 
Revised: E k (cid:104)(cid:80) dl

Original: [Equation] l
Revised: =1 I k ∂

Original: [Equation]
Revised: [Table]

Original:  k [Equation]
Revised: f ( x k, 0 ) √ v k, 0 (cid:16)(cid:16) M ′ l,k (cid:17) k − ( F l,k ) k (cid:17)(cid:105)

Original: and we delegate the history part for later analysis.
Revised: and we delegate the history part for later analysis.

Original: For each possible (cid:16) M ′ l,k (cid:17) k − ( F l,k ) k , we first convert all x k,i into x k, 0 using Lemma F.3.
Revised: For each possible (cid:16) M ′ l,k (cid:17) k − ( F l,k ) k , we first convert all x k,i into x k, 0 using Lemma F.2.

Original: Since(cid:16) M ′ l,k (cid:17) k contains ( n − 1) +· · · + 1 = n ( n − 1)terms of x k,i (with i ̸ = 0 ), we have
Revised: Since(cid:16) M ′ l,k (cid:17) k contains ( n − 1) +

Original: [Equation]
Revised: [Equation]

Original: Using the same strategy for analyzing the color-ball toy example (Figure 15), we have

Original: [Equation]
Revised: [Equation]

Original: 1 − [Equation] n
Revised: terms of x k,i (with i ̸ = 0 ), we have

Original: [Equation]
Revised: [Equation]

Original: 21 −
Revised: Using the same strategy for analyzing the color-ball toy example

Original: [Equation]
Revised: [Equation]

Original: n
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the followingderivation.
Revised: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the followingderivation.

Original: Now we move one step further exert E k − 1 ( · ) .
Revised: Now we move one step further exert E k − 1 ( · ) .

Original: In particular, we need to calculate
Revised: In particular, we need to calculate

Revised: E

Revised: [Equation]

Revised: (cid:17)(cid:35)(cid:41)

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: The blue term is the residue from the k -th epoch.
Revised: The blue term is the residue from the k -th epoch.

Original: The red term is the main component in the ( k − 1)
Revised: The red term is the main component in the ( k − 1)

Original: -thepoch.
Revised: -thepoch.

Original: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitzproperty.
Revised: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitzproperty.

Original: First of all, we work on ( a ) .
Revised: First of all, we work on ( a ) .

Revised: [Figure]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 2 ρβ n 2 | J 1 | n 2 △ n ( k − 1) − (1 − β 1 ) d (cid:115) 2 ρβ n 2 | J 1 | nQ k , where I k,k −:=
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: I (cid:16) max i | ∂ l f i ( x k, 0 )
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Table]

Revised: [Equation]

Original:  | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and˜ I k,k − 1 := I (cid:16) max i | ∂ l f i ( x k, 0 )
Revised: [Table] where I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and˜ I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .

Original: By Lemma G.4,we know I k,k − 1 =
Revised: By Lemma G.3,we know I k,k − 1 =

Original: I k − 1 , so we have:
Revised: I k − 1 , so we have:

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The blue term will be handled using color-ball method when taking conditional expectation E k − 1 ( · ) .

Original: Now we derive a lower bound for (b).
Revised: Now we derive a lower bound for (b).

Original: Similarly as before, we rewrite (cid:16) M ′ l,k (cid:17) k − 1 and ( F l,k ) k − 1 asfollows.
Revised: Similarly as before, we rewrite (cid:16) M ′ l,k (cid:17) k − 1 and ( F l,k ) k − 1 asfollows.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We now calculate ( b ) .
Revised: We now calculate ( b ) .

Original: Using the same idea as in ( a ) , we have
Revised: Using the same idea as in ( a ) , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] To proceed [Table] we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,
Revised: To proceed, we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:12)(cid:12)(cid:12) .

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: n

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .
Revised: The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .

Original: Now we have derived lower bounds for both ( a ) and ( b ) .
Revised: Now we have derived lower bounds for both ( a ) and ( b ) .

Original: Combining together, we have:
Revised: Combining together, we have:

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: F

Original: [Equation]

Original: [Table]

Original: [Equation]

Original:  

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]
Revised: [Table]

Original: (cid:32)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Original: [Table]
Revised: [Equation]

Original: For the first term in the above inequality
Revised: For the first term in the above inequality

Original: [Table]
Revised: [Equation]

Original: we can calculate it using the idea in the color-ball toyexample:
Revised: we can calculate it using the idea in the color-ball toyexample:

Original: E k −(cid:40)
Revised: [Equation]

Original: d (cid:88) l = I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 (1 − β 1 ) Jn − 1 (cid:88) i = ∂ l f i ( x k − 1 , 0 ) + d (cid:88) l =
Revised: To proceed

Original: I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 ,(cid:0) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:1)(cid:41) = d (cid:88) l = I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 (1 − β 1 ) β n 1 Jn − 1 (cid:88) i = ∂ l f i ( x k − 1 , 0 ) .
Revised: [Equation]

Original: To proceed, we further take E k − 2 ( · ) and bound
Revised: we further take E k − 2 ( · ) and bound

Original: E k −  d
Revised: E k −  d

Original: (cid:88) l =
Revised: (cid:88) [Equation]

Original: I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 (1 − β 1 ) β n 1 Jn − 1 (cid:88) i = ∂ l f i ( x k − 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) ( a ) + d (cid:88) l = I k ∂ l f ( x k, 0 ) √ v k,(cid:16)(cid:0) M ′ l,k (cid:1) k − 2 − ( F l,k ) k − 2 (cid:17) (cid:124) (cid:123)(cid:122) (cid:125) ( b )   .
Revised:   .

Original: Repeat this process until k = 1 , we have:
Revised: Repeat this process until k = 1 , we have:

Original: E   (cid:88) l large ∂ l f ( x k, 0 ) √ v k,(cid:16)(cid:0) M ′ l,k

Original: [Equation]

Original: n
Revised: E

Original: [Equation]
Revised: [Equation]

Original: 1 (cid:88)

Original: [Equation] = [Equation] +
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Equation] Error [Equation] Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .
Revised: + Error 1 + Error 2 + Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .

Original: We specifiy Error 1 , Error 2 , Error 3 as follows.
Revised: We specify Error 1 , Error 2 , Error 3 as follows.

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: (cid:114) [Equation] (cid:32)
Revised: n

Original: [Equation]
Revised: [Equation]

Original: (cid:88) [Equation] = [Equation]
Revised: [Table]

Original: − 
Revised: − d

Original: [Equation]
Revised: [Table]

Original: (cid:88) l =

Original: [Equation] − [Equation] [Equation]
Revised: [Table]

Original: (cid:32)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n− 1 (cid:88) i = ∂ l f i ( x k [Equation] 0 ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:33)
Revised: △ n

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation]
Revised: d

Original: [Table]
Revised: [Table]

Original:  Q [Equation] [Table] [Equation]
Revised: β n 2 | J 1 | n [ Q k + Q k − 1 + β n 1 ( Q k + Q k − 1 + Q k − 2 )

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: +

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Table] Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a finite constant, we have (cid:80) ∞
Revised: Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a finite constant, we have (cid:80) ∞

Original: j =0 β jn 1 △ n ( k − j ) =
Revised: j =0 β jn 1 △ n ( k − j ) =

Original: △ 1 δ 2 √ nk .
Revised: △ 1 δ 2 √ nk .

Original: Inaddition, we [-have(cid:2) Q k + β n 
 (-] Q k + Q k − 1) + β2 n 1 ( Q k + Q k − 1 + Q k − 2 ) 
Revised: Inaddition, we [-have(cid:2) Q k + β n 
 (-]have[ Q k + Q k − 1 + β n 1 ( Q k + Q k − 1 + Q k − 2 ) + · · · ] ( ∗ ) ≤

Original:  + · · · (cid:3) Lemma F.1 ≤ Q k  +2 (cid:0) β n 1 2 Q k  + [-β 2 n 
 [Equation] Therefore-]
Revised: Q k (1 + β n 1 + · · · ) + Q k − 1 + β n 1 ( Q k − 1 + [-β 2 n 
 [Equation] Therefore-]Q k −

Original: [Equation]
Revised: [Equation]

Original:  we have Error [Equation] [Table]{+(cid:80) Jj = 
+}
Revised: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have {+(cid:80) Jj = 
+}

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: √ k − j ≤ 2 J √ k − 1 .

Original: [Equation] [Table] [Equation] [Table]
Revised: This inequality can be simply proved by taking the integral over 1 √ k .

Original: [Equation] [Table] [Equation] [Table]
Revised: The final inequality is due to

Original: [Equation] [Table] [Equation] [Table]
Revised: Q k − 1 < 2 Q k .

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: Now, we have

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: Based on the calculation in Lemma F.
Revised: Based on the calculation in Lemma F.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: Error
Revised: Error

Original: [Equation]

Original: 1 −

Original: [Equation]

Original: (cid:16)

Original: [Equation]

Original: [Equation]

Original: 4 ρ

Original: [Equation]

Original: n

Original: [Equation]

Original: (cid:17)(cid:17)

Original: δ

Original: [Equation]

Original: (1 −

Original: [Equation]

Original: [Equation]

Original: (cid:32) n

Original: [Equation]

Original: (cid:88)

Original: i

Original: [Equation]

Original: | ∂ l f i ( x

Original: [Equation]

Original: 0 ) | (cid:33) (1 −

Original: [Equation]

Original: n 1 )

Original: [Equation]

Original: Error

Original: [Equation]
Revised: [Equation]

Original: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have
Revised: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have

Original: [Equation]
Revised: [Equation]

Original: Using the fact that
Revised: Using the fact that

Original: [Equation]
Revised: [Equation]

Original: for k [Equation]
Revised: (for k ≥ 2 )

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: +Error 1 + Error 2 + Error≥ − G 6 √ k − G 7 δ 1 E (cid:32) d (cid:88) l = n − 1 (cid:88) i = | ∂ l f i ( x k, 0 ) | (cid:33) , where G:
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n [Equation] [Equation] | (cid:115)
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n
Revised: + Error

Original: [Equation]
Revised: [Equation]

Original: (cid:34) d (cid:88) l = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) In − 1 (cid:88) i = ∂ l f i ( x 1 , 0 ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) [Equation] [Equation]
Revised: where

Original: [Equation]

Original: (cid:115)

Original: [Equation]
Revised: [Equation]

Original: + d   1 − 1 √ β n(cid:114)
Revised: where J 1 =

Original: [Equation]
Revised: [Equation]

Original: where J 1 = (cid:0) −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: − · ·
Revised: − · ·

Original: [Equation]
Revised: [Equation]

Original: , δ
Revised: , δ

Original: [Equation]
Revised: [Equation]

Revised: =

Original:  kk − j (if needed, wecan further bound J 1 by n for simplicity).
Revised: [Equation] j [Equation] kk − j (if needed, wecan further bound J 1 by n for simplicity).

Original: This conclude the proof.
Revised: This conclude the proof.

Original: G.8 Proof of Lemma G. 
Revised: G.8 Proof of Lemma G.

Original: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .
Revised: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .

Original: The upper bound for (cid:80) l large ( a 2 ) isvery straightforward using inequality (21).(cid:88) l large
Revised: The upper bound for (cid:80) l large ( a 2 ) isvery straightforward using inequality (23).(cid:88)

Original: [Equation]
Revised: [Equation]

Original: Now we shift gear to (cid:80) l large( b 2 ) :
Revised: Now we shift gear to (cid:80) l large

Original: = δ 1 (cid:113) 2 ρ 23 β n

Original: [Equation]
Revised: [Equation]

Original: m [Equation]
Revised: b 2 ) :

Original: − ∂
Revised: = δ

Original: [Equation]
Revised: [Equation]

Original: f

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: m

Original: [Equation]
Revised: [Equation]

Original: ) | .
Revised: | .

Original: To proceed, weneed an upper bound for (cid:80) l large (cid:80) n − 1 i =

Revised: To proceed, weneed an upper bound for (cid:80) l large (cid:80) n [Equation] 1 i =

Original: | m l,k,i − ∂ l f i ( x k, 0 ) | .
Revised: | m l,k,i − ∂ l f i ( x k, 0 ) | .

Original: For each i , we perform the followingdecomposition.
Revised: For each i , we perform the followingdecomposition.

Original: [Equation]
Revised: [Equation]

Original: l large

Original: [Equation]

Original: To start, we bound ( d 1 ) .
Revised: To start, we bound ( d 1 ) .

Original: [Equation]
Revised: [Equation]

Original: Now, we bound ( d 2 ) .
Revised: Now, we bound ( d 2 ) .

Original: Recall (cid:80) n − 1 i =

Revised: Recall (cid:80) n − 1 i =

Original: ∂ l f i ( x 1 , 0 ) =
Revised: ∂ l f i ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 ) , we have
Revised: ∂ l f ( x 1 , 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: l,k − 1 , [Equation]
Revised: l

Original: [Equation]
Revised: [Equation]

Original: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.
Revised: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj =

Original: [Equation]
Revised: [Equation]

Revised: ≤

Revised: [Equation]

Revised: k − 1 .

Original: Plugging (60) into (59) and re-arranging the index, we haved
Revised: Plugging (62) into (61) and re-arranging the index, we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Table]

Revised: ) |

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: d

Revised: [Table]

Revised: |

Revised: [Table]

Revised: f

Revised: [Table]

Revised: ) | + (cid:114)

Revised: [Table]

Revised: (cid:33) + 2 qd

Revised: [Table]

Revised: (cid:112) n ( k

Revised: [Table]

Original: =
Revised: 1) −(cid:35) + β ( [Table] [Table]

Revised: d (cid:88) l

Revised: [Table]

Revised: 1 (cid:88)

Revised: [Table]

Revised: | ∂

Original: [Equation]
Revised: [Table] f i ( [Table] [Table]

Revised: ) |

Revised: [Table]

Revised: n

Original: [Table]
Revised: [Table]

Revised: |

Revised: [Table]

Revised: f

Original: [Equation]
Revised: [Table] | [Table]

Revised: (cid:114)

Revised: [Table]

Revised: D

Revised: [Table]

Revised: d

Revised: [Table]

Revised: + 1 −

Revised: [Table]

Original: where the second last inequality holds because: when k ≥ 4 , 1 √ n 
Revised: 2 d [Table] (cid:112) n ( k − 1) −+ β [Table]

Revised: [Table]

Revised: d

Revised: [Table]

Revised: l

Revised: [Table]

Revised: n

Revised: [Table]

Revised: i

Revised: [Table]

Revised: | ∂

Revised: [Table]

Revised: f

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: ) |

Revised: When

Revised: [Table]

Revised: √ D

Revised: [Table]

Revised: d

Revised: [Table]

Revised: | ∂

Revised: [Table]

Revised: f

Revised: [Table]

Revised: [Table]

Revised: ) | + (cid:114) D

Revised: [Table]

Revised: D

Original: [Equation]
Revised: [Table]

Original: k − 
Revised: d (cid:33) + 1 − β2 √ 2 d △ 1 √ nk + β ( k [Table] [Equation] where the second last inequality holds because: when k ≥ 4 ,

Original: [Equation]
Revised: [Equation]

Original: − 1 ≤ √ 2 √ nk .
Revised: √ nk .

Original: ( i ) holds when kis large enough such that β ( k − 1) 
Revised: ( i ) holds when kis large enough such that β ( k − 1) n

Original: n≤ β n 1 √ k − 1 .
Revised: [Equation]

Original: Further, β n 1 √ k − 1 ≤ √
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: k when k ≥ 2 .
Revised: k when k ≥ 2 .

Original: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .
Revised: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .

Original: Plugging (61) and (58) into (57), we concludethe proof.
Revised: Plugging (63) and (60) into (59), we concludethe proof.

Original: [Table]
Revised: [Table]

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: The proof is concluded by adding expectation on both sides of the inequality.
Revised: The proof is concluded by adding expectation on both sides of the inequality.

Original: G.9 Proof of Lemma G. 
Revised: G.9 Proof of Lemma G.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ≥

Original: [Equation]

Original: large

Original: [Equation]
Revised: [Equation]

Original: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.
Revised: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.

Original: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have
Revised: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have

Original: [Equation]
Revised: [Equation]

Original: To proceed, we introduce the following Lemma G.16, which is Lemma F.2 in (Shi et al., 2020). Lemma G.16. (Lemma F.2 in (Shi et al., 2020)) Given k , we set α as the index of the coordinatewith the greatest gradient:
Revised: Now, we discuss two cases.

Original: [Equation] Under Assumption 2.1 and 2.2, if k ≥ 4 and (cid:113) | ∂ α f ( x k, [Equation]
Revised: Case (a): when | ∂ α f ( x k, 0 ) | ≥ 4 √∆ 1 (1 − β 2 ) √ D 1 nkd .

Original: [Equation] , the following holds:
Revised: In this case, we have the following result.

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma G.16 can be seen in (Shi et al., 2020), Lemma F.2, Page 26.
Revised: [Table]

Original: Now, we discusstwo cases. Case (a): when | ∂ α f ( x k, 0 )
Revised: [Equation]

Original: | ≥ 4 √∆ 1 (1 − β 2 ) √ D 1 nkd .
Revised: [Table]

Original: In this case, the condition of Lemma G.16is satisfied.
Revised: [Equation] [Table] [Equation] [Table]

Original: We have we have v α,k, 0 ≤ 52 D 1 d (cid:16) | ∂
Revised: [Equation] [Table] [Equation] [Table]

Original: [-α f ( x k, 0 ) | 
-]
Revised: [-α f ( x k, 0 ) | 
-][Equation] [Table] [Equation] [Table]

Original: + D 0 D 1 d (cid:17) .
Revised: [Equation] [Table] [Equation] [Table]

Original: Therefore, ∂ α f ( x k, 0 ) 2 √ v α,k,≥
Revised: ≥ [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: To proceed, we consider the following two sub-cases.
Revised: Now we consider the following two sub-cases.

Original: [Figure]

Original: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d :[Figure] [Equation] ( x
Revised: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d :

Original: [Equation]
Revised: [Equation]

Original: • When ∂ α f
Revised: • When ∂ α f

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: x k

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Combining together
Revised: Combining together

Original: [Equation]
Revised: [Equation]

Original: we have the following results for Case (a)
Revised: we have the following results for Case (a)

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Case (b
Revised: [Table] [Equation] [Table] [Equation]

Original: : When |
Revised: [Table]

Original: ∂
Revised: [Equation]

Original: α f ( x k,
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: | < 4 √
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: Case (b): When | ∂ α f ( x k, 0 ) | < 4 √

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: , we have
Revised: we have

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where G 1 :
Revised: where G 1 :

Original: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .
Revised: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .

Original: Now, the following two claims are both true.• Claim 1:
Revised: Now, the following two claims are both true. • Claim 1:

Original: [Equation]
Revised: [Equation]

Original: • Claim2:
Revised: Claim

Original: [Equation]
Revised: [Equation]

Original: Combining Claim 1 and Claim
Revised: Combining Claim 1 and Claim

Original: [Equation]
Revised: [Equation]

Original: we have[
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 65) where G 2 :=
Revised: where G 2 :=

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Combining Case (a) and Case (b) together, we have
Revised: Combining Case (a) and Case (b) together, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because of ∥∇ f ( x k, 0 )
Revised: where the last inequality is because of ∥∇ f ( x k, 0 )

Original: ∥ 22 ≤ d∂
Revised: ∥ 22 ≤ d∂

Original: [Equation]
Revised: [Equation]

Original: Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − ϵ, 1] , such that 1 √ 5 D 1 d − F 3 ≥ 1 √ 10 D 1 d ,or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma
Revised: Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − ϵ, 1] , such that 1 √ 5 D 1 d − F 3 ≥ 1 √ 10 D 1 d ,or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma

Original: G.7).
Revised: G.6).

Original: With such a choice of β 2 , we have the following results by changing all the F 3 into
Revised: With such a choice of β 2 , we have the following results by changing all the F 3 into

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .
Revised: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .

Original: This is because: first, F 4 = F 3 (cid:113) D 0 D 1 d ;second, F 3 ≤ 1 √ 10 D 1 d .
Revised: This is because: first, F 4 = F 3 (cid:113) D 0 D 1 d ;second, F 3 ≤ 1 √ 10 D 1 d .

Original: The proof of Lemma G.7 is completed.
Revised: The proof of Lemma G.6 is completed.

Original: We restate all the constants as follows ( G 4 , G 5 , G 6 are specified in Appendix G.7):
Revised: We restate all the constants as follows ( G 4 , G 5 , G 6 are specified in Appendix G.7):

Original: [Equation]
Revised: [Equation]

Original: Remark G.17 .
Revised: Remark G.14 .

Original: We comment that we can always replace F 5 in the final result by F 4 , i.e., we can choosenot to apply the last inequality ( ∗ ) in the proof.
Revised: We comment that we can always replace F 5 in the final result by F 4 , i.e., we can choosenot to apply the last inequality ( ∗ ) in the proof.

Original: The benefit of using F 4 is that F 4 monotonouslydecrease to 0 when increasing β 2 to 1.
Revised: The benefit of using F 4 is that F 4 monotonouslydecrease to 0 when increasing β 2 to 1.

Original: This monotone property is erased in the notation of F 5 .
Revised: This monotone property is not shown in the notation of F 5 .

Original: Nevertheless, we choose to use F 5 since it is a much cleaner constant.
Revised: Nevertheless, we choose to use F 5 since it is a much cleaner constant.

Original: G.10 Proof of Lemma G. 
Revised: G.10 Proof of Lemma G.

Original: Based on Descent Lemma, we have
Revised: Based on Descent Lemma, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging in the r.h.s.
Revised: Plugging in the r.h.s.

Original: of (33), we have the following relation after rearranging.
Revised: of (34), we have the following relation after rearranging.

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Recallin Lemma F.1, we have 
Revised: Recall we have 2 (cid:16) √ T − [Equation] t [Equation]

Revised: −

Revised: [Equation]

Revised: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: Further,since ∥ · ∥{+= t 
 [Equation] k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 . Further, since ∥·∥+} 1 ≥ ∥ · ∥ 2 , we get the following relation when t 0 = 1 .
Revised: {+= t 
 [Equation] k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 . Further, since ∥·∥+} 1 ≥ ∥·∥ 2 ,we get the following relation when t 0 = 1 .

Original: [Equation]
Revised: [Equation]

Original: We specify all the constant as follows.
Revised: We specify all the constant as follows.

Original: For all the following constants, We keep the same notation astheir appearance in their corresponding lemmas.
Revised: For all the following constants, We keep the same notation astheir appearance in their corresponding lemmas.

Original: [Equation]
Revised: [Equation]

Revised: [Figure]

Original: [Equation]  [Equation]
Revised: [Equation] [Figure] [Equation]

Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: ;where J 1 = (cid:0) − 1 n β 1 − 2 n
Revised: [Figure]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  1  n {+− 
+}
Revised: where J 1 = (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n {+− 
+}

Original: [Equation]
Revised: [Equation]

Original: Ifneeded, we can further bound J 1 by n for simplicity.
Revised: Ifneeded, we can further bound J 1 by n for simplicity.

Original: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .
Revised: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .

Original: Usually, theseconstants can be different for different problems.
Revised: Usually, theseconstants can be different for different problems.

Original: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.
Revised: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Equation]

Original: The proof of Lemma G.10 is now completed.
Revised: The proof of Lemma G.9 is now completed.

Original: This also concludes the whole proof of Theorem 3.1.
Revised: This also concludes the whole proof of Theorem 3.1.

Original: [Figure]