Original: More Specifically, A ( β 2 ) needs to satisfy ( ρ 1 , ρ 2and ρ 3 are defined in Appendix F.1).[Figure]
Revised: More Specifically, A ( β 2 ) needs to satisfy ( ρ 1 , ρ 2and ρ 3 are defined in Appendix F.1).

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have the following result when k is large enough such that β ( k − 1)
Revised: we have the following result when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 4 :
Revised: n≤ β n 1 √ k − 1 and k ≥ 4 :

Original: [Equation]
Revised: [Equation]

Revised: (cid:40)(cid:114)

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original:  Proof of Lemma G.6 can be seen in Appendix G.9.
Revised: [Equation] [Table] [Equation] Proof of Lemma G.7 can be seen in Appendix G.9.

Original: Remark G.7 .
Revised: Remark G.8 .

Original: Condition (33) specifies the smallest threshold of β 2 to ensure the convergence.
Revised: Condition (32) specifies the smallest threshold of β 2 to ensure the convergence.

Original: Thiscondition can be translated into the threshold funtion γ 1 ( n ) mentioned in Theorem 3.1.
Revised: Thiscondition can be translated into the threshold funtion γ 1 ( n ) mentioned in Theorem 3.1.

Original: As a roughestimate, Lemma G.6 requires β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) / ( n 2 ρ ) (cid:1) , where ρ = ρ 1 ρ 2 ρ 3 .
Revised: As a roughestimate, Lemma G.7 requires β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) / ( n 2 ρ ) (cid:1) , where ρ = ρ 1 ρ 2 ρ 3 .

Original: Asdiscussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: Asdiscussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: When ρ 1 , ρ 2 , ρ 3 achieve their upper boundat the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .
Revised: When ρ 1 , ρ 2 , ρ 3 achieve their upper boundat the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: However, ρ ishighly dependent on the problem instance f ( x ) and training process.
Revised: However, ρ ishighly dependent on the problem instance f ( x ) and training process.

Original: Our experiments in Appendix
Revised: Our experiments in Appendix

Original: B.1 shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lowerthan it appears to be.
Revised: D.2 shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lowerthan it appears to be.

Original: We further remark that γ 1 ( n ) ∝ (1 − β n 1 ) .
Revised: We further remark that γ 1 ( n ) ∝ (1 − β n 1 ) .

Original: That is to say, as β 1 increases, the threshold of β 2 alsoslowly increases (to ensure the convergence).
Revised: That is to say, as β 1 increases, the threshold of β 2 alsoslowly increases (to ensure the convergence).

Original: However, such phenomenon is not observed in ourexperiments.
Revised: However, such phenomenon is not observed in ourexperiments.

Original: We conjecture γ 1 ( n ) should be independent of β 1 , removing the dependence (1 − β n 1 )is considered as an interesting future direction.
Revised: We conjecture γ 1 ( n ) should be independent of β 1 , removing the dependence (1 − β n 1 )is considered as an interesting future direction.

Original: Remark G.8 .
Revised: Remark G.9 .

Original: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.
Revised: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.

Original: Thiscan be seen in the proof in Appendix G.9 (the definition of F 4 ) and Remark G.14.
Revised: Thiscan be seen in the proof in Appendix G.9 (the definition of F 4 ) and Remark G.17.

Original: Based on Lemma G.6, we can further rewrite (30) as follows.
Revised: Based on Lemma G.7, we can further rewrite (28) as follows.

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 3.1 is concluded by plugging (34) into Descent Lemma (24) and then takingtelescope some from k = t 0 to k = T .
Revised: The proof of Theorem 3.1 is concluded by plugging (33) into Descent Lemma (22) and then takingtelescope some from k = t 0 to k = T .

Original: These steps are quite standard in non-convex optimization.
Revised: These steps are quite standard in non-convex optimization.

Original: We finish these calculation in Lemma G.

Revised: We finish these calculation in Lemma G.

Original: Lemma G.9.
Revised: Lemma G.10.

Original: When inequality (34) holds, we have the following results based on Descent Lemma(24) :
Revised: When inequality (33) holds, we have the following results based on Descent Lemma(22) :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma G.9 can be seen in Appendix G.10.
Revised: Proof of Lemma G.10 can be seen in Appendix G.10.

Original: Now, the whole proof of Theorem 3.1 iscompleted.
Revised: Now, the whole proof of Theorem 3.1 iscompleted.

Original: G.2 Proof of Lemma G. 
Revised: G.2 Proof of Lemma G.

Original: To prove Lemma G.1, we only prove (27).
Revised: To prove Lemma G.1, we first show the following result. Lemma G.11. Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfiesmax i | ∂ l f i ( x k, 0 ) | [Equation]

Revised: Q k :=

Revised: [Equation]

Revised: n 2 we have

Revised: [Equation]

Revised: The proof idea of Lemma G.11 is similar as Lemma F.3 in (Shi et al., 2020), but with the followingdifferences:

Revised: The condition of Lemma F.3 in (Shi et al.,

Revised: requires the full batch gradient| ∂ l f ( x k, 0 ) | ≥ nQ k , which is different from our condition.

Revised: Here, we choose the condition of Lemma G.11 because it meets the need of our later decomposition strategy in (28).2.

Revised: The constant Q k is different.

Revised: The difference is due to the different update formula ofRMSProp and general Adam.

Revised: Anyhow, Lemma G.11 can be easily proved following most of the steps in Lemma F.3 (Shi et al.

Revised: We omit the proof for brevity.

Revised: To prove Lemma G.1, we only prove (25).

Original: The proof for (26) can be analogized from the proof of(27).
Revised: The proof for (24) can be analogized from the proof of(25).

Original: We discuss the following two cases:
Revised: We discuss the following two cases:

Original: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have∂ l f ( x k, 0 ) 
Revised: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) √ v l,k,i

Original: [Equation]
Revised: [Equation]

Original:  m l,k ,i
Revised: ∂ l f ( x k, 0 ) ( m l

Original: [Equation]
Revised: [Equation]

Original: ∂ l f
Revised: k [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: xk,
Revised: x

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is because of √ v l,k,i ≥ (cid:112) β
Revised: [Equation]

Original: i 2 √ vl,k, 0 .
Revised: √ v

Original:  Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have∂ l f ( x k,
Revised: [Equation] Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have

Original: [Equation]
Revised: [Figure]

Original: ( m l,k,i − ∂ l f i ( x k,

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) is due to 1 √ 1+ x ≥ 1 − x 2  .
Revised: where ( a ) is because of √ v l,k,i ≥ (cid:112) β i 2 √ v l,k, 0 .

Original: We further have
Revised: Combining these two cases, we have

Original: [Table]
Revised: [Figure]

Original: v

Original: [Table]
Revised: [Equation]

Original: where ( ∗∗ ) is due to the condition in the Lemma max i | ∂ l f i ( xτ k, 0 ) | ≥ Q k ≥ n △ nk .
Revised: i ( x k, 0 )) √ v l,k,(cid:18) − 1 − β (cid:18) − 1 + 4 ρ 2 i β n (cid:19)(cid:19) ≥ n − 1 (cid:88) i =

Original: Plug the aboveresult into (35) we have
Revised: ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) √ v l,k,− δ 1 (cid:12)(cid:12)(cid:12)(cid:12) ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:12)(cid:12)(cid:12)(cid:12) n − 1 (cid:88) i = | m l,k,i − ∂ l f i ( x k, 0 ) | , where δ 1 = (1 − β 2 ) (cid:16) − 1+ 4 nρ 2 βn 2 (cid:17) + (cid:18) 1 √ β n 2 −(cid:19) . Proof is completed.

Original: [Equation] In the last step ( ∗ ∗ ∗ ) , we use the following Lemma G.10.
Revised: To prove Lemma G.3, we first show the following result.

Original: Lemma G.10.
Revised: Lemma G.12.

Original: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfies
Revised: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisfiesmax

Original: [Equation]
Revised: [Equation]

Original:  , we have
Revised: 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof idea of Lemma G.10 is similar as Lemma F.1 in (Shi et al., 2020), but with thefollowing differences:
Revised: The proof idea of Lemma G.12 is silimar with Lemma F.1 in (Shi et al., 2020), but with thefollowing differences:

Original: The condition of Lemma F.1 in (Shi et al., 2020) 
Revised: The condition of Lemma F.1 in (Shi et al., 2020) requires the full batch gradient

Original: both require the full batch gradient | ∂ l f ( x k, 0 ) | ≥nQ k , which is different from our condition.
Revised: | ∂ l f ( x k, 0 ) | ≥ nQ k , which is different from our condition.

Original: Here, we choose the condition of Lemma G.10because it meets the need of our  decomposition strategy in (30) .2.
Revised: Here, we choose the condition of Lemma G.11 because it meets the need of our later decomposition strategy in(28).2.

Original: The constant Q k is different.
Revised: The constant Q k is different.

Original: The difference is due to the different update formula of RMSPropand general Adam.
Revised: The difference is due to the different update formula of RMSProp and general Adam.

Original: Anyhow, Lemma G.10 can beeasily proved following most of the steps in Lemma F.1 in (Shi et al.,2020).
Revised: Lemma G.12 can be proved following most of the steps of Lemma F.1 in (Shi et al., 2020).

Original: We omit the proof for brevity.
Revised: We omitthe proof for brevity.

Original: Combining the above Case 1 and Case 2 , we have

Original: [Equation]

Original: i ( x k, 0 ) | , where δ 1 = (1 − β 2 )4 nρ 2 β

Original: n+

Original: (cid:18)

Original: 1 √ β n 2 −(cid:19) .

Original: Proof is completed.

Original: G.3 Proof of Lemma G.
Revised: Now we prove Lemma G.3.

Original: We first prove the result for j = 1 .
Revised: We first prove the result for j = 1 .

Original: We discuss the following two cases.
Revised: We discuss the following two cases.

Original: Case 1: when ∂ l f ( x k, 0 ) √ v l,k, 0 ≥ ∂ l f ( x k − 1 , 0 ) √ v l,k − 1 , 0 : when ∂ l f ( x k, 0 ) ≤ 0 , we have 
Revised: Case 1: when ∂ l f ( x k, 0 ) √ v k,≥ [Equation] : when ∂ l f ( x k, 0 ) ≤ 0 , we have [Equation]

Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) > 0 , we have
Revised: When ∂ l f ( x k, 0 ) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In conclusion, we have:
Revised: In conclusion, we have:

Original: [Equation]
Revised: [Equation]

Original: [Equation] , where δ 1 = 
Revised: δ 1 = [Equation]

Original: [Equation]
Revised: [Equation]

Original:  is a constant that goes to 0 when β 2 goes to 1.
Revised: 2 (cid:17)(cid:19) is a constant that goes to 0 when β 2 goes to 1.

Original: Therefore, wehave
Revised: Therefore, we have

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: Note that

Original: Case 2: when ∂ l f [Equation] x [Equation]
Revised: Case 2: when ∂ l f ( x k, 0 ) √ v k,

Original: [Equation]
Revised: [Equation]

Original: : when ∂ l f ( x k, 0 ) ≥ 0 , we have
Revised: : when ∂ l f ( x k, 0 ) ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) < 0 , we have
Revised: When ∂ l f ( x k, 0 ) < 0 , we have

Original: [Equation]
Revised: ∂ l f ( x k, 0 ) √ v k,≥ ∂ l f ( x k, 0 ) √ v k − 1 , (cid:112) β n 2 .

Original: Following the same strategy as in Case 1, we can show that
Revised: Following the same strategy as in Case 1, we can show that

Original: [Equation]
Revised: [Equation]

Original: which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: Case 1 and Case 2 together, we have
Revised: Case 1 and Case 2 together, we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Now we consider the case when j > 1 .
Revised: Now we consider the case when j > 1 .

Original: Based on the above inequality, we have
Revised: Based on the above inequality, we have

Revised: [Table]

Revised: ∂ l f ( x k, 0 ) √

Revised: [Table]

Revised: k,

Original: [Table]
Revised: [Table]

Original: The proof is completed.
Revised: The proof is completed.

Original: G.4 Proof of Lemma G. 
Revised: G.4 Proof of Lemma G.

Original: To prove Lemma G.4, we need to further decompose E [ (cid:80) l large ( b 1 )] .
Revised: To prove Lemma G.5, we need to further decompose E [ (cid:80) l large ( b 1 )] .

Original: First and foremost, we [-write(cid:80) n − 1 i =
-]
Revised: First and foremost, we [-write(cid:80) n − 1 i =
-]write

Original: m l,k,i in an explicit form.m l,k,i = (1 − β

Original: [Equation]
Revised: [Equation]

Original: ∂ l f ( x 1 , 0 ) (37)

Original: Since ∂ l f ( x 1 , 0 ) =
Revised: Since ∂ l f ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =
Revised: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =

Original: (cid:80) [-n − 1 i =
-]
Revised: (cid:80) [-n − 1 i =
-][Equation]

Original: ∂ l f i ( x 1 , 0 ) , wehaveβ (
Revised: f i ( x 1 , 0 ) , wehave [Equation]

Revised: Plugging (36) into (35), we have

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Revised: Using (37), we have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [-− 1) n + i +
-]
Revised: [-− 1) n + i +
-][Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: Plugging (38) into (37), we have
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: ∂ l f τ k, 0 ( x k, 0 ) + β i +11 f τ k − 1 ,n − 1 ( x k − 1 ,n − 1 ) + · · · + β i
Revised: [Table]

Original: + nf τ k − 1 , 0 ( x k − 1 , 0 ) + · · · + β ( k − 2) n + i +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + i∂ l f τ 1 , 0 ( x 1 , 0 )
Revised: [Equation]

Original: + β ( k − 1) n
Revised: [Table]

Original: + i +1∂ [Equation] [Equation] 
Revised: [Equation] [Table]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original:  ( k − 1) n [-[Equation] i +
 [Equation]-]
Revised: 0 ( x 1 , 0 ) + · · · + β ( k − 2) n [-[Equation] i +
 [Equation]-]+1∂

Original: n

Original: [Equation]

Original: ( x

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [-1) n + i +
-]
Revised: [-1) n + i +
-]

Original: [Equation]

Original: l f 0 ( x 1 , 0 ) + · · · [Equation] β ( k − 1)[-n + i +
-]
Revised: l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1)[-n + i +
-]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: Using (39), we have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation]

Original: thepoch [Equation] [Equation] ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + n − 2∂ l f τ 1 , 0 ( x 1 , 0 ) + · · · + β ( k − 2) n +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 

Revised: n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125)

Original: -thepoch
Revised: -thepoch

Original: + [Equation] ( k [Equation]
Revised: + β ( k − 1) n + n∂ l f 0 ( x 1 , 0 )

Original: [Equation]

Original: f

Original: [Equation]

Original: ) n

Original: [Equation]

Original: n

Original: [Equation]

Original: [Equation]

Original: l f n

Original: [Equation]

Original: [Equation] [Equation] [Equation] ∂ l f 
Revised: + · · · + β ( k − 1) n + n∂ l f n − 1 ( x 1 , 0 )

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: ( k

Original: [Equation]

Original:  1) n [Equation] 1∂ l f n − 1 ( x 1 , 0 ) [Equation]
Revised: + β ( k − 1) n + n − 1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n − 1∂ l f n − 1 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f n − 1 ( x 1 , 0 )

Original:  (cid:124) (cid:123)(cid:122) (cid:125) 

Revised: + (cid:124) (cid:123)(cid:122) (cid:125)

Original: -thepoch ... } (40)
Revised: -thepoch ... } (38)

Original: Note that in (40), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition willcause trouble in bounding the (cid:80) n − 1 i =

Revised: Note that in (38), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition willcause trouble in bounding the (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) .

Original: Therefore, we need to define an auxillarysequence M ′ l,k which does not involve such a phase transition.
Revised: Therefore, we need to define an auxillarysequence M ′ l,k which does not involve such a phase transition.

Original: We define M ′ l,k as follows.
Revised: We define M ′ l,k as follows.

Original: (For each‘epoch’, we suggest readers to read from bottom to the top.)
Revised: (For each‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Equation] [Table][Equation]
Revised: [Table]

Original: [Table] [Equation] [Table] [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: n

Original: [Equation]
Revised: [Equation]

Original:  n [Equation] ∂ l f n [Equation]{+− 1 ( x 1 , 0 )+ (cid:124) (cid:123)(cid:122) (cid:125) 
+}
Revised: ) + · · · + β ( k − 1) n∂ l f τ 1 , 0 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 1 -thepoch + β ( k − 1) n + n∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n + n − 1∂ l f n − 1 ( x 1 , 0 ) + β ( k − 1) n + n − 1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n +2( n − 1)∂ l f n − 1 ( x 1 , 0 ) + · · · + β ( k − 1) n +1∂ l f 0 ( x 1 , 0 ) + · · · + β ( k − 1) n + n∂ l f n {+− 1 ( x 1 , 0 )+ (cid:124) (cid:123)(cid:122) (cid:125) 
+}

Original: ∂

Original: [Equation] [Equation]
Revised: -thepoch ... } (39)

Original: Now we bound (cid:80) n − 1 i =

Revised: Now we bound (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M ′ l,k .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M ′ l,k .

Original: Denoting F l,k :=(cid:80) n − 1 i =

Revised: Denoting F l,k :=(cid:80) n − 1 i =

Original: ∂ l f i ( x k, 0 ) , we have:
Revised: ∂ l f i ( x k, 0 ) , we have:n

Original: [Equation]

Original: [Equation]

Original: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k .

Original: F l

Original: [Equation]
Revised: [Equation]

Original: k := n − 1 (cid:88)i = [Equation] + ... +
Revised: (cid:88)

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: For the sake of better presentation [Equation] we  rewrite F l
Revised: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k .

Original: [Equation]
Revised: [Equation]

Original: k and M ′ l [Equation] k as follows:
Revised: For the sake of better presentation, we rewrite F l,k and M ′ l,k as follows:

Original: [Equation]
Revised: [Equation]

Original: · , where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .
Revised: where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .

Original: Similarly for
Revised: Similarly for

Original: M ′ l,k .
Revised: M ′ l,k .

Original: Now, we separate M ′ l,k − F l,k into two parts.
Revised: Now, we separate M ′ l,k − F l,k into two parts.

Original: [Equation]
Revised: [Equation]

Original: Now, we rewrite E [ (cid:80) l large ( b 1 )] :[Equation] l large [Equation]
Revised: Now, we rewrite E [ (cid:80) l large ( b 1 )] :

Original: [Equation]

Original: l large

Original: [Equation]

Original: l large

Original: [Equation]

Original: (cid:88) l large

Original: [Equation]
Revised: [Equation]

Original: (cid:124) (cid:123)(cid:122) (cid:125) (1)

Original: [Equation]

Original: We bound (1) , (2) and (3) respectively.
Revised: We bound (1) , (2) and (3) respectively.

Original: We bound (1) and (3) in the following Lemma G.11 and
Revised: We bound (1) and (3) in the following Lemma G.13 and

Original: Lemma G.12.
Revised: Lemma G.14.

Original: Since the difference of M l,k − M ′ l,k only occurs in the high order terms of the infinitesum sequence, (1) is expected to vanish as k grows.
Revised: Since the difference of M l,k − M ′ l,k only occurs in the high order terms of the infinitesum sequence, (1) is expected to vanish as k grows.

Original: Similarly for (3) .
Revised: Similarly for (3) .

Original: Lemma G.11.
Revised: Lemma G.13.

Original: When k is large enough such that: β ( k − 1)n
Revised: When k is large enough such that: β ( k − 1)

Original: [Equation] 1 and k ≥ 4 , then we have[Equation]
Revised: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have

Original: [Equation]
Revised: [Equation]

Original: where G 4 = d (cid:113) [Equation] β n 2 β n [Equation]{+1 √ 2 n ( n − 1) (cid:80) n − 1 i =
+}
Revised: where G 4 = d (cid:113) 2 ρ 2 3 β n 2 β n {+1 √ 2 n ( n − 1) (cid:80) n − 1 i =
+}

Original: √ 2 n ( n [Equation] (cid:80) [Equation]
Revised: E ( | ∂ α f i ( x 1 , 0 ) | ) .

Original: Lemma G.12.
Revised: Lemma G.14.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have
Revised: n≤ β n 1 √ k − 1 and k ≥ 4 , then we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original:  where G 5 = d (cid:113) [Equation] 23 β n 2 (cid:16) β2 n 1 2 n 3 △
Revised: 44) where G 5 = d (cid:113) 2 ρ 23 β n 2 (cid:16) β

Original: [Equation]
Revised: [Equation]

Original: We relegate the proof of Lemma G.11 and G.12 to Appendix G.5 and G.6.
Revised: We relegate the proof of Lemma G.13 and G.14 to Appendix G.5 and G.6.

Original: Now we bound (2) .
Revised: Now we bound (2) .

Original: This part involves the difficulties (i), (ii) and (iii) mentioned in Appendix G.1.
Revised: This part involves the difficulties (i), (ii) and (iii) mentioned in Appendix G.1.

Original: We bound (2) in Lemma G.13.
Revised: We bound (2) in Lemma G.15.

Original: Lemma G.13.
Revised: Lemma G.15.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have 
Revised: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have E

Original: [Equation]
Revised: [Equation]

Original: The constant terms G 6 and G 7 are specified in Appendix G.7.
Revised: The constant terms G 6 and G 7 are specified in Appendix G.7.

Original: Proof is shown in Appendix G.7.
Revised: Proof is shown in Appendix G.7.

Original: Combining Lemma G.11, G.12 and G.13 together, we conclude the proof. 
Revised: Combining Lemma G.13, G.14 and G.15 together, we conclude the proof. [Equation] l large

Original: [Equation]
Revised: [Equation]

Original: G.5 Proof of Lemma G. 
Revised: G.5 Proof of Lemma G.

Original: By definition of M l,k and M ′ l,k (see (40) and (41)), they only differ when k ≤ 0 .
Revised: By definition of M l,k and M ′ l,k (see (38) and (39)), they only differ when k ≤ 0 .

Original: More specifically,
Revised: More specifically,

Original: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)
Revised: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: M l,k
Revised: [Equation] l

Original: [Equation]
Revised: [Equation]

Original: l,k
Revised: l

Original: [Equation]
Revised: [Equation]

Original: + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + (cid:16) β ( k − 1) n +2− β kn +1(cid:17)
Revised: k [Equation]

Original:  ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 )[-(cid:124) (cid:123)(cid:122) (cid:125) -
-]
Revised: +2− β ( k − 1) n +3(cid:17) ∂ l f 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β ( k − 1) n + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 )[-(cid:124) (cid:123)(cid:122) (cid:125) -
-]

Original: -th epoch + (cid:16) β
Revised: + (cid:16) β

Original: ( k − 1) n + n +2− β ( k +1) n + n(cid:17) ∂
Revised: ( k − 1) n +1− β ( k − 1) n +2(cid:17) ∂

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n  +  n +2− β
Revised: l f 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +1− β ( k − 1) n + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) 0-th epoch + (cid:16) β

Original: ( k +1) n + n  + n − 1(cid:17)
Revised: ( k − 1) n + n +1− β kn + n(cid:17) ∂

Original:  ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n + n +1(cid:17)
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +1− β kn + n + n − 1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17)

Original: ∂ l f n − 1 ( x 1 , 0 )
Revised: ∂ l f n − 1 ( x 1 , 0 )

Original: + (cid:16) β
Revised: + (cid:16) β

Original: ( k − 1) n +3− β ( k +1) n +1(cid:17)∂
Revised: ( k − 1) n +2− β kn +1(cid:17)

Original:  l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β ( k +1) n + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) [Equation]{+(cid:123)(cid:122) (cid:125) -
 -th epoch + (cid:16)+}
Revised: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) {+(cid:123)(cid:122) (cid:125) -
 -th epoch + (cid:16)+}

Original: [Equation]
Revised: [Equation]

Original: We start with the 1st column in the ‘0-th epoch’
Revised: We start with the 1st column in the ‘0-th epoch’

Original: (from the bottom to the top).
Revised: (from the bottom to the top).

Original: The 1st column in the ‘0-th epoch’ 
Revised: The 1st column in the ‘0-th epoch’ =

Original: {+β ( k − 1) n +
+}
Revised: {+β ( k − 1) n +
+}

Revised: [Equation]

Revised: [Equation]

Revised: | ∂

Original: [Equation]
Revised: [Equation]

Revised: k −

Revised: [Equation]

Revised: [Equation]

Revised: [Equation]

Revised: k −

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: | ≤ β

Revised: [Equation]

Revised: k −

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: [Equation]

Revised: |

Revised: [Equation]

Original:  Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can bebounded as follows.
Revised: l [Equation] | (46) Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can bebounded as follows.

Original: The last column in the ‘0-th epoch’ =
Revised: The last column in the ‘0-th epoch’ =

Original: [-β ( k − 1) n +
-]
Revised: [-β ( k − 1) n +
-]

Original: [Equation]
Revised: [Equation]

Original: Summing up all the columns, we have
Revised: Summing up all the columns, we have

Original: The ‘0-th epoch’ [Equation]
Revised: The ‘0-th epoch’ ≤ β ( k − 1) n +1n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | ≤ β ( k − 1) n +1n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | (48)

Original: Using the same technique, it can be shown thatThe
Revised: Using the same technique, it can be shown that

Original:  ‘-1-th epoch’ ≤ β [Equation] k − [Equation]
Revised: The ‘-1-th epoch’ ≤ β ( k − 1) n +22 n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | , (49)

Original: The ‘-2-th epoch’ ≤ β  k − 
Revised: The ‘-2-th epoch’ ≤ β ( k − 1) n +33 n ( n − 1)(1 − β 1 ) n − 1 (cid:88) i = | ∂ l f i ( x 1 , 0 ) | (50) · · ·

Original: [Equation] [Equation] Plugging all these results in (47), we have
Revised: Plugging all these results in (45), we have(cid:12)(cid:12) M

Original: [Equation]
Revised: [Equation]

Original: That is to say, when k is large enough such that β ( k − 1)
Revised: That is to say, when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 we have
Revised: n≤ β n 1 √ k − 1 we have

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma G.10, the proof is completed.
Revised: Combining with Lemma G.12, the proof is completed.

Original: G.6 Proof of Lemma G. 
Revised: G.6 Proof of Lemma G.

Original: We start with(cid:16) M ′ l,k (cid:17) 0 − ( F l,k ) 0 .
Revised: We start with

Original: [Equation]
Revised: [Equation]

Original: M ′ l [Equation] k (cid:1)
Revised: whereδ i = n (cid:88)

Original: [Equation]
Revised: [Equation]

Original: F [Equation] [Equation]
Revised: =

Original: [Equation]
Revised: [Equation]

Original: where
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: We further have
Revised: We further have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Revised: ≤ n

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: Using the same calculation
Revised: Using the same calculation

Original: [Equation]
Revised: [Equation]

Original: we can get the following result:
Revised: we can get the following result:

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:0) [Equation] (cid:1) n −
Revised: Repeat this calculation

Original: [Equation]
Revised: [Equation]

Original: i = | ∂ l f i ( x 1 , 0 ) | . Repeat this calculation, we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: When β ( k − 1)
Revised: When β ( k − 1)

Original: n≤ β n 1 √ k − 1 , we further have:
Revised: n≤ β n 1 √ k − 1 , we further have:

Original: [Equation]
Revised: [Equation]

Original: k, where ( ∗ ) : 1 √ n ( k − 1) − 1 ≤ (cid:113) 2 nk when k ≥ 4 .
Revised: where ( ∗ ) : [Equation] (cid:113) 2 nk when k ≥ 4 .

Original: Therefore, we have
Revised: Therefore, we have

Original: (cid:12)(cid:12)(cid:12)(cid:0)

Original: [Equation]
Revised: [Equation]

Original: ˜

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma G.10, the proof is completed with constant G 5 defined in Lemma G.12.
Revised: Combining with Lemma G.12, the proof is completed with constant G 5 defined in Lemma G.14.

Original: G.7 Proof of Lemma G.
Revised: G.7 Proof of Lemma G.15

Original: In this section, we derive a lower bound for E (cid:104)(cid:80)
Revised: In this section, we derive a lower bound for E (cid:104)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).
Revised: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).

Original: We first rewrite “ l large" intoindicator function as follows:
Revised: We first rewrite “ l large" intoindicator function as follows:

Original: [Table]
Revised: [Equation]

Original: where Ik :
Revised: where I

Original: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := △ 1 n √ n √ k

Original: [Table]
Revised: [Equation]

Original: n 2 (cid:17) .

Original: We also define I k − j :=
Revised: We also define I [Equation] = [Equation]

Original: I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , it will be used later. 
Revised: | [Equation] (cid:17) , it will be used later. [Equation]

Original: We take the conditional expectation over the history before x k, 0 .
Revised: We take the conditional expectation over the history before x k, 0 .

Original: We first focus on
Revised: We first focus on

Original: Ek (cid:104)(cid:80) dl
Revised: E

Original: =1 I k ∂
Revised: [Equation] l

Original: [Table]
Revised: [Equation]

Original: f ( x k, 0 ) √ v k, 0 (cid:16)(cid:16) M ′ l,k (cid:17) k − ( F l,k ) k (cid:17)(cid:105)
Revised: k [Equation]

Original: and we delegate the history part for later analysis.
Revised: and we delegate the history part for later analysis.

Original: For each possible (cid:16) M ′ l,k (cid:17) k − ( F l,k ) k , we first convert all x k,i into x k, 0 using Lemma F.2.
Revised: For each possible (cid:16) M ′ l,k (cid:17) k − ( F l,k ) k , we first convert all x k,i into x k, 0 using Lemma F.3.

Original: Since(cid:16) M ′ l,k (cid:17) k contains ( n − 1) + · · 
Revised: Since(cid:16) M ′ l,k (cid:17) k contains ( n − 1) + · · · + 1 = n ( n − 1)terms of x k,i (with i ̸ = 0 ), we have

Original: [Equation]
Revised: [Equation]

Original: terms of x k,i (with i ̸ = 0 ), we have 
Revised: Using the same strategy for analyzing the color-ball toy example (Figure 15), we have [Equation] 1 −

Original: [Equation]
Revised: [Equation]

Original: Using the same strategy for analyzing the color-ball toy example
Revised: n

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: 21 −

Original: [Equation]
Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the followingderivation.
Revised: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the followingderivation.

Original: Now we move one step further exert E k − 1 ( · ) .
Revised: Now we move one step further exert E k − 1 ( · ) .

Original: In particular, we need to calculate
Revised: In particular, we need to calculate

Original: E k

Original: [Equation]

Original: (cid:17)(cid:35)(cid:41)

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: The blue term is the residue from the k -th epoch.
Revised: The blue term is the residue from the k -th epoch.

Original: The red term is the main component in the ( k − 1)
Revised: The red term is the main component in the ( k − 1)

Original: -thepoch.
Revised: -thepoch.

Original: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitzproperty.
Revised: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitzproperty.

Original: First of all, we work on ( a ) .
Revised: First of all, we work on ( a ) .

Original: [Figure]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: 2 ρβ n 2 | J 1 | n 2 △ n ( k − 1) − (1 − β 1 ) d (cid:115) 2 ρβ n 2 | J 1 | nQ k , where I k,k −:=

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: I (cid:16) max i | ∂ l f i ( x k, 0 )

Original: [Table]

Original: [Table]

Original: [Equation]

Original: [Table] where I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and˜ I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 )
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and˜ I k,k − 1 := I (cid:16) max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .

Original: By Lemma G.3,we know I k,k − 1 =
Revised: By Lemma G.4,we know I k,k − 1 =

Original: I k − 1 , so we have:
Revised: I k − 1 , so we have:

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: The blue term will be handled using color-ball method when taking conditional expectation E k − 1 ( · ) .

Original: Now we derive a lower bound for (b).
Revised: Now we derive a lower bound for (b).

Original: Similarly as before, we rewrite (cid:16) M ′ l,k (cid:17) k − 1 and ( F l,k ) k − 1 asfollows.
Revised: Similarly as before, we rewrite (cid:16) M ′ l,k (cid:17) k − 1 and ( F l,k ) k − 1 asfollows.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We now calculate ( b ) .
Revised: We now calculate ( b ) .

Original: Using the same idea as in ( a ) , we have
Revised: Using the same idea as in ( a ) , we have

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  To proceed, we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:12)(cid:12)(cid:12) .
Revised: [Table] To proceed [Table] we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Revised: n

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .
Revised: [Table] [Equation] [Table] The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .

Original: Now we have derived lower bounds for both ( a ) and ( b ) .
Revised: Now we have derived lower bounds for both ( a ) and ( b ) .

Original: Combining together, we have:
Revised: Combining together, we have:

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: [Equation]

Revised: F

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original:  
Revised:  

Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: 

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: (cid:32)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)

Revised: [Table]

Original: For the first term in the above inequality
Revised: For the first term in the above inequality

Original: [Equation]
Revised: [Table]

Original: we can calculate it using the idea in the color-ball toyexample:
Revised: we can calculate it using the idea in the color-ball toyexample:

Original: [Equation]
Revised: E k −(cid:40)

Original: To proceed
Revised: d (cid:88) l = I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 (1 − β 1 ) Jn − 1 (cid:88) i = ∂ l f i ( x k − 1 , 0 ) + d (cid:88) l =

Original: [Equation]
Revised: I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 ,(cid:0) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:1)(cid:41) = d (cid:88) l = I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 (1 − β 1 ) β n 1 Jn − 1 (cid:88) i = ∂ l f i ( x k − 1 , 0 ) .

Original:  we further take E k − 2 ( · ) and bound
Revised: To proceed, we further take E k − 2 ( · ) and bound

Original: E k −  d
Revised: E k −  d

Original: (cid:88) [Equation]
Revised: (cid:88) l =

Original:    .
Revised: I k − 1 ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 (1 − β 1 ) β n 1 Jn − 1 (cid:88) i = ∂ l f i ( x k − 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) ( a ) + d (cid:88) l = I k ∂ l f ( x k, 0 ) √ v k,(cid:16)(cid:0) M ′ l,k (cid:1) k − 2 − ( F l,k ) k − 2 (cid:17) (cid:124) (cid:123)(cid:122) (cid:125) ( b )   .

Original: Repeat this process until k = 1 , we have:
Revised: Repeat this process until k = 1 , we have:

Revised: E   (cid:88) l large ∂ l f ( x k, 0 ) √ v k,(cid:16)(cid:0) M ′ l,k

Revised: [Equation]

Original: E
Revised: n

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: 1 (cid:88) [Equation] =

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: + [Equation] [Equation] Error

Original: [Equation]
Revised: [Equation]

Original: + Error 1 + Error 2 + Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .
Revised: Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .

Original: We specify Error 1 , Error 2 , Error 3 as follows.
Revised: We specifiy Error 1 , Error 2 , Error 3 as follows.

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: n
Revised: (cid:114)

Original: [Equation]
Revised: [Equation]

Revised: (cid:32)

Original: [Table]
Revised: [Equation]

Original: − d
Revised: (cid:88)

Original: [Table]
Revised: [Equation]

Revised: =

Original: [Table]
Revised: [Equation]

Original: β
Revised: −

Original: [Table]
Revised: [Equation]

Original: ) △ n
Revised: (cid:88) l =

Original: [Table]
Revised: [Equation]

Original: d
Revised: −

Original: [Table]
Revised: [Equation]

Original: β n 2 | J 1 | n [ Q k + Q k − 1 + β n 1 ( Q k + Q k − 1 + Q k − 2 )

Original: + [Table]
Revised: [Equation]

Revised: (cid:32)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n − 1 (cid:88) i =

Revised: ∂ l f i ( x k

Revised: [Equation]

Revised: 0 ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:33)

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: Q

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original:  Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a finite constant, we have (cid:80) ∞
Revised: [Equation] [Table] [Table] Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a finite constant, we have (cid:80) ∞

Original: j =0 β jn 1 △ n ( k − j ) =
Revised: j =0 β jn 1 △ n ( k − j ) =

Original: △ 1 δ 2 √ nk .
Revised: △ 1 δ 2 √ nk .

Original: Inaddition, we have[ Q [Equation]{+k + β n 
+}
Revised: Inaddition, we have(cid:2) Q {+k + β n 
+}

Original:  + Q k − 1  + β  n 1 ( Q k + Q k − 1 + Q k − 2 )+ · · · ] ( ∗ ) ≤
Revised: ( Q k + Q k − 1 ) + β 2 n 1 ( Q k + Q k − 1 + Q k − 2 )

Original: Q k (1 + β n 1 + · · · ) + Q k− 1 +  β n 1 ( Q k− 1 + Q k −{+β 2 n 
 [Equation] Therefore+}
Revised: + · · · (cid:3) Lemma F.1 ≤ Q k + 2 (cid:0) β n 1 2 Q k + {+β 2 n 
 [Equation] Therefore+}

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have [-(cid:80) Jj = 
-]
Revised: we have [-(cid:80) Jj = 
-]Error [Equation] [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: √ k − j ≤ 2 J √ k − 1 .
Revised: [Equation] [Table] [Equation] [Table]

Original: This inequality can be simply proved by taking the integral over 1 √ k .
Revised: [Equation] [Table] [Equation] [Table]

Original: The final inequality is due to
Revised: [Equation] [Table] [Equation] [Table]

Original: Q k − 1 < 2 Q k .
Revised: [Equation] [Table] [Equation] [Table]

Revised: [Equation]

Original: Now, we have
Revised: [Table] [Equation] [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: Based on the calculation in Lemma F.
Revised: Based on the calculation in Lemma F.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: Error
Revised: Error

Revised: [Equation]

Revised: 1 −

Revised: [Equation]

Revised: (cid:16)

Revised: [Equation]

Revised: [Equation]

Revised: 4 ρ

Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: (cid:17)(cid:17)

Revised: δ

Revised: [Equation]

Revised: (1 −

Revised: [Equation]

Revised: [Equation]

Revised: (cid:32) n

Revised: [Equation]

Revised: (cid:88)

Revised: i

Revised: [Equation]

Revised: | ∂ l f i ( x

Revised: [Equation]

Revised: 0 ) | (cid:33) (1 −

Revised: [Equation]

Revised: n 1 )

Revised: [Equation]

Revised: Error

Original: [Equation]
Revised: [Equation]

Original: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have
Revised: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have

Original: [Equation]
Revised: [Equation]

Original: Using the fact that
Revised: Using the fact that

Original: [Equation]
Revised: [Equation]

Original: for k ≥ 2 )
Revised: for k [Equation]

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: + 
Revised: + Error 1 + Error 2 + Error≥ − G 6 √ k − G 7 δ 1 E (cid:32) d (cid:88) l = n − 1 (cid:88) i = | ∂ l f i ( x k, 0 ) | (cid:33) , where G:

Original: [Equation]
Revised: [Equation]

Original: +
Revised: n [Equation] [Equation] | (cid:115)

Original: [Equation]
Revised: [Equation]

Original: + Error
Revised: n

Original: [Equation]
Revised: [Equation]

Original: where
Revised: (cid:34) d (cid:88) l = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) In − 1 (cid:88) i = ∂ l f i ( x 1 , 0 ) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:35) [Equation] [Equation]

Revised: [Equation]

Revised: (cid:115)

Original: [Equation]
Revised: [Equation]

Original: where J 1 =
Revised: + d   1 − 1 √ β n(cid:114)

Original: [Equation]
Revised: [Equation]

Original:  −
Revised: where J 1 = (cid:0) −

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: − · ·
Revised: − · ·

Original: [Equation]
Revised: [Equation]

Original: , δ
Revised: , δ

Original: [Equation]
Revised: [Equation]

Original: = [Equation] kk − j (if needed, wecan further bound J 1 by n for simplicity).
Revised: kk − j (if needed, wecan further bound J 1 by n for simplicity).

Original: This conclude the proof.
Revised: This conclude the proof.

Original: G.8 Proof of Lemma G. 
Revised: G.8 Proof of Lemma G.

Original: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .
Revised: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .

Original: The upper bound for (cid:80) l large ( a 2 ) isvery straightforward using inequality (23).(cid:88)
Revised: The upper bound for (cid:80) l large ( a 2 ) isvery straightforward using inequality (21).(cid:88) l large

Original: [Equation]
Revised: [Equation]

Original: Now we shift gear to (cid:80) l large ( b 2 ) :
Revised: Now we shift gear to (cid:80) l large ( b 2 ) :

Original: = δ [Equation] ρ 23 β 
Revised: = δ 1 (cid:113) 2 ρ 23 β n

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: − ∂
Revised: − ∂

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Revised: x k

Revised: [Equation]

Original: ) | .
Revised: ) | .

Original: To proceed, weneed an upper bound for (cid:80) l large (cid:80) n − 1 i =

Revised: To proceed, weneed an upper bound for (cid:80) l large (cid:80) n − 1 i =

Original: | m l,k,i − ∂ l f i ( x k, 0 ) | .
Revised: | m l,k,i − ∂ l f i ( x k, 0 ) | .

Original: For each i , we perform the followingdecomposition.
Revised: For each i , we perform the followingdecomposition.

Original: [Equation]
Revised: [Equation]

Revised: l large

Revised: [Equation]

Original: To start, we bound ( d 1 ) .
Revised: To start, we bound ( d 1 ) .

Original: [Equation]
Revised: [Equation]

Original: Now, we bound ( d 2 ) .
Revised: Now, we bound ( d 2 ) .

Original: Recall (cid:80) n − 1 i =

Revised: Recall (cid:80) n − 1 i =

Original: ∂ l f i ( x 1 , 0 ) =
Revised: ∂ l f i ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 ) , we have
Revised: ∂ l f ( x 1 , 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l,k − 1 ,

Original: [Equation]
Revised: [Equation]

Original: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation] [Table] [Equation] where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj =
Revised: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.

Original: [Equation]
Revised: [Equation]

Original: ≤

Original: [Equation]
Revised: [Equation]

Original: k − 1 .

Original: Plugging (62) into (61) and re-arranging the index, we have
Revised: Plugging (60) into (59) and re-arranging the index, we haved

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: =

Original: [Table] [Equation][Table] [Table]
Revised: [Equation]

Original: ) |

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: [Table]

Original: d

Original: [Table]
Revised: [Table]

Original: |

Original: [Table]
Revised: [Equation]

Original: f

Original: [Table]

Original: ) | + (cid:114)

Original: [Table]

Original: (cid:33) + 2 qd

Original: [Table]

Original: (cid:112) n ( k [Table] 1) −(cid:35) + β (
Revised: where the second last inequality holds because: when k ≥ 4 , 1 √ n

Original: [Table]

Original: [Table]

Original: d (cid:88) l

Original: [Table]

Original: 1 (cid:88)

Original: [Table]

Original: | ∂

Original: [Table]

Original: f i (

Original: [Table]

Original: [Table]

Original: ) |

Original: [Table]

Original: n

Original: [Table] | [Table]
Revised: [Equation]

Original: f [Table] | [Table]
Revised: k −

Original: (cid:114)

Original: [Table]

Original: D

Original: [Table]
Revised: [Equation]

Original: d

Original: [Table] + 1 − [Table] 2 d
Revised: − 1 ≤ √ 2 √ nk .

Original: [Table]

Original: (cid:112) n ( k − 1)−+ β [Table] [Table] d
Revised: ( i ) holds when kis large enough such that β ( k − 1)

Original: [Table]

Original: l

Original: [Table]

Original: n

Original: [Table]

Original: i

Original: [Table]

Original: | ∂

Original: [Table]

Original: f

Original: [Table]

Original: [Table]

Original: [Table]

Original: ) |

Original: When

Original: [Table]

Original: √ D

Original: [Table]

Original: d

Original: [Table]

Original: | ∂

Original: [Table]

Original: f

Original: [Table]

Original: [Table]

Original: ) | + (cid:114) D

Original: [Table]

Original: D

Original: [Table]

Original: d (cid:33) + 1 − β2 √ 2 d △ 1 √ nk + β ( k

Original: [Table] [Equation] where the second last inequality holds because: when k ≥ 4 , [Equation]
Revised: n≤ β n 1 √ k − 1 .

Original: √ nk . ( i ) holds when kis large enough such that β ( k − 1) n [Equation] Further
Revised: Further, β n 1 √ k − 1 ≤ √

Original: [Equation]
Revised: [Equation]

Original: k when k ≥ 2 .
Revised: k when k ≥ 2 .

Original: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .
Revised: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .

Original: Plugging (63) and (60) into (59), we concludethe proof.
Revised: Plugging (61) and (58) into (57), we concludethe proof.

Original: [Table]
Revised: [Table]

Original: ∂
Revised: ∂

Original: [Equation]
Revised: [Equation]

Original: The proof is concluded by adding expectation on both sides of the inequality.
Revised: The proof is concluded by adding expectation on both sides of the inequality.

Original: G.9 Proof of Lemma G. 
Revised: G.9 Proof of Lemma G.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Revised: ≥

Revised: [Equation]

Revised: large

Original: [Equation]
Revised: [Equation]

Original: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.
Revised: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.

Original: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have
Revised: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Now, we discuss two cases.
Revised: To proceed, we introduce the following Lemma G.16, which is Lemma F.2 in (Shi et al., 2020). Lemma G.16. (Lemma F.2 in (Shi et al., 2020)) Given k , we set α as the index of the coordinatewith the greatest gradient:

Revised: [Equation]

Original: Case (a): when | ∂ α f ( x k, 0 ) | ≥ 4 √∆ 1 (1 − β 2 ) √ D 1 nkd .
Revised: Under Assumption 2.1 and 2.2, if k ≥ 4 and (cid:113) | ∂ α f ( x k, [Equation] [Equation]

Original: In this case, we have the following result.
Revised: , the following holds:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: The proof of Lemma G.16 can be seen in (Shi et al., 2020), Lemma F.2, Page 26.

Original: [Equation]
Revised: Now, we discusstwo cases. Case (a): when | ∂ α f ( x k, 0 ) | ≥ 4 √∆ 1 (1 − β 2 ) √ D 1 nkd . In this case, the condition of Lemma G.16is satisfied.

Original: [Table] [Equation] [Table] [Equation]
Revised: We have we have v α,k, 0 ≤ 52 D 1 d (cid:16) | ∂

Original: [Table] ≥{+α f ( x k, 0 ) | 
+}
Revised: {+α f ( x k, 0 ) | 
+}

Original: [Table]
Revised: + D 0 D 1 d (cid:17) . Therefore, ∂ α f ( x k, 0 ) 2 √ v α,k,≥

Original: [Equation]
Revised: [Equation]

Original: Now we consider the following two sub-cases.
Revised: To proceed, we consider the following two sub-cases.

Revised: [Figure]

Original: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d : 
Revised: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d : [Figure] [Equation] ( x

Original: [Equation]
Revised: [Equation]

Original: • When ∂ α f
Revised: • When ∂ α f

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: x k

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Combining together
Revised: Combining together

Original: [Equation]
Revised: [Equation]

Original: we have the following results for Case (a)
Revised: we have the following results for Case (a)

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation]
Revised: Case (b

Original: [Table]
Revised: : When | ∂ α f ( x k,

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table]
Revised: | < 4 √

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Case (b): When | ∂ α f ( x k, 0 ) | < 4 √

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original:  we have
Revised: , we have

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where G 1 :
Revised: where G 1 :

Original: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .
Revised: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .

Original: Now, the following two claims are both true. • Claim 1:
Revised: Now, the following two claims are both true.• Claim 1:

Original: [Equation]
Revised: [Equation]

Original:  Claim 
Revised: • Claim 2:

Original: [Equation]
Revised: [Equation]

Original: Combining Claim 1 and Claim
Revised: Combining Claim 1 and Claim

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have[

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where G 2 :=
Revised: 65) where G 2 :=

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Combining Case (a) and Case (b) together, we have
Revised: Combining Case (a) and Case (b) together, we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: where the last inequality is because of ∥∇ f ( x k, 0 )
Revised: where the last inequality is because of ∥∇ f ( x k, 0 )

Original: ∥ 22 ≤ d∂
Revised: ∥ 22 ≤ d∂

Original: [Equation]
Revised: [Equation]

Original: Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − ϵ, 1] , such that 1 √ 5 D 1 d − F 3 ≥ 1 √ 10 D 1 d ,or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma
Revised: Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − ϵ, 1] , such that 1 √ 5 D 1 d − F 3 ≥ 1 √ 10 D 1 d ,or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma

Original: G.6).
Revised: G.7).

Original: With such a choice of β 2 , we have the following results by changing all the F 3 into
Revised: With such a choice of β 2 , we have the following results by changing all the F 3 into

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .
Revised: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .

Original: This is because: first, F 4 = F 3 (cid:113) D 0 D 1 d ;second, F 3 ≤ 1 √ 10 D 1 d .
Revised: This is because: first, F 4 = F 3 (cid:113) D 0 D 1 d ;second, F 3 ≤ 1 √ 10 D 1 d .

Original: The proof of Lemma G.6 is completed.
Revised: The proof of Lemma G.7 is completed.

Original: We restate all the constants as follows ( G 4 , G 5 , G 6 are specified in Appendix G.7):
Revised: We restate all the constants as follows ( G 4 , G 5 , G 6 are specified in Appendix G.7):

Original: [Equation]
Revised: [Equation]

Original: Remark G.14 .
Revised: Remark G.17 .

Original: We comment that we can always replace F 5 in the final result by F 4 , i.e., we can choosenot to apply the last inequality ( ∗ ) in the proof.
Revised: We comment that we can always replace F 5 in the final result by F 4 , i.e., we can choosenot to apply the last inequality ( ∗ ) in the proof.

Original: The benefit of using F 4 is that F 4 monotonouslydecrease to 0 when increasing β 2 to 1.
Revised: The benefit of using F 4 is that F 4 monotonouslydecrease to 0 when increasing β 2 to 1.

Original: This monotone property is not shown in the notation of F 5 .
Revised: This monotone property is erased in the notation of F 5 .

Original: Nevertheless, we choose to use F 5 since it is a much cleaner constant.
Revised: Nevertheless, we choose to use F 5 since it is a much cleaner constant.

Original: G.10 Proof of Lemma G. 
Revised: G.10 Proof of Lemma G.

Original: Based on Descent Lemma, we have
Revised: Based on Descent Lemma, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging in the r.h.s.
Revised: Plugging in the r.h.s.

Original: of (34), we have the following relation after rearranging.
Revised: of (33), we have the following relation after rearranging.

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: Recall  we have2 (cid:16) √ T − [Equation] t [Equation]
Revised: Recall in Lemma F.1, we have

Original: −

Original: [Equation]

Original: (cid:80)

Original: [Equation]
Revised: [Equation]

Original: [-= t 
 [Equation] k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 . Further, since ∥·∥-] 1 ≥ ∥·∥ 2 ,we get the following relation when t 0 = 1 .
Revised: [-= t 
 [Equation] k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 . Further, since ∥·∥-]Further,since ∥ · ∥ 1 ≥ ∥ · ∥ 2 , we get the following relation when t 0 = 1 .

Original: [Equation]
Revised: [Equation]

Original: We specify all the constant as follows.
Revised: We specify all the constant as follows.

Original: For all the following constants, We keep the same notation astheir appearance in their corresponding lemmas.
Revised: For all the following constants, We keep the same notation astheir appearance in their corresponding lemmas.

Original: [Equation]
Revised: [Equation]

Original: [Figure]

Original: [Equation][Figure] [Equation]
Revised: [Equation] [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: ;where J 1 = (cid:0) − 1 n β 1 − 2 n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where J 1= (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n[-− 
-]
Revised: 1 n[-− 
-]

Original: [Equation]
Revised: [Equation]

Original: Ifneeded, we can further bound J 1 by n for simplicity.
Revised: Ifneeded, we can further bound J 1 by n for simplicity.

Original: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .
Revised: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .

Original: Usually, theseconstants can be different for different problems.
Revised: Usually, theseconstants can be different for different problems.

Original: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.
Revised: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Figure]

Original: The proof of Lemma G.9 is now completed.
Revised: The proof of Lemma G.10 is now completed.

Original: This also concludes the whole proof of Theorem 3.1.
Revised: This also concludes the whole proof of Theorem 3.1.

Revised: [Figure]