Original: [Equation]
Revised: [Equation]

Original: 2 −(cid:19) , which goes to 0 when β 2 goes to 1.
Revised: 2 −(cid:19) , which goes to 0 when β 2 goes to 1.

Original: Proof can be seen in Appendix G.4.
Revised: Proof can be seen in Appendix G.4.

Original: Now we bound the error terms: E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .
Revised: Now we bound the error terms: E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .

Original: Since they are multiplied by δ 1 , these two terms vanish when β 2 → 1 .
Revised: Since they are multiplied by δ 1 , these two terms vanish when β 2 → 1 .

Original: Bounding E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .
Revised: Bounding E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .

Original: We bound these two terms in the following Lemma G.5.
Revised: We bound these two terms in the following Lemma G.5.

Original: Lemma G.5.
Revised: Lemma G.5.

Original: Given all the history up to x k, 0 , we denote α as the index of the coordinate with the greatest gradient: α = arg max l =
Revised: Given all the history up to x k, 0 , we denote α as the index of the coordinate with the greatest gradient: α = arg max l =

Original: [Equation]
Revised: [Equation]

Original: , 2 , ··· ,d | ∂ l f ( x k, 0 ) | .
Revised: , 2 , ··· ,d | ∂ l f ( x k, 0 ) | .

Original: Under Assumption 2.1 and 2.2, consider β 1 < √ β 2 , when k is large such that: k ≥ 4 ; β ( k − 1) n
Revised: Under Assumption 2.1 and 2.2, consider β 1 < √ β 2 , when k is large such that: k ≥ 4 ; β ( k − 1) n

Original: [Equation]
Revised: [Equation]

Original: , we have the following results:
Revised: , we have the following results:

Original: [Equation]
Revised: [Equation]

Original: where δ 1 is deﬁned in (17) , D 0 is deﬁned in Assumption 2.2.
Revised: where δ 1 is deﬁned in (17) , D 0 is deﬁned in Assumption 2.2.

Original: Detailed proof of Lemma G.5 is shown in Appendix G.8.
Revised: Detailed proof of Lemma G.5 is shown in Appendix G.8.

Original: Bounding E (cid:110)(cid:80) l large ( a 1 ) + (cid:80) l large ( b 1 ) − (cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:111) .
Revised: Bounding E (cid:110)(cid:80) l large ( a 1 ) + (cid:80) l large ( b 1 ) − (cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:111) .

Original: With the help of Lemma
Revised: With the help of Lemma

Original: G.4 and G.5, we can bound E (cid:110)(cid:80) l large ( a 1 ) + (cid:80) l large ( b 1 ) − (cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:111) .
Revised: G.4 and G.5, we can bound E (cid:110)(cid:80) l large ( a 1 ) + (cid:80) l large ( b 1 ) − (cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:111) .

Original: The intuition is as follows:
Revised: The intuition is as follows:

Original: • ( a 1 ) is in the order of | ∂ l f ( x k, 0 ) | , multiplied by some positive constant; • (cid:80) l large { ( a 2 )
Revised: • ( a 1 ) is in the order of | ∂ l f ( x k, 0 ) | , multiplied by some positive constant; • (cid:80) l large { ( a 2 )

Original: + ( b 2 ) } vanishes
Revised: + ( b 2 ) } vanishes

Original: when β 2 → 1 . • E (cid:110)(cid:80) l large ( b 1 ) (cid:111) vanishes when β 2 → 1 and k goes to inﬁnity.
Revised: when β 2 → 1 . • E (cid:110)(cid:80) l large ( b 1 ) (cid:111) vanishes when β 2 → 1 and k goes to inﬁnity.

Original: Therefore, E (cid:110)(cid:80) l large ( a 1 ) + (cid:80) l large ( b 1 ) − (cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:111) is still in the order of | ∂ l f ( x k, 0 ) | (multiplied by a positive constant when β 2 is large).
Revised: Therefore, E (cid:110)(cid:80) l large ( a 1 ) + (cid:80) l large ( b 1 ) − (cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:111) is still in the order of | ∂ l f ( x k, 0 ) | (multiplied by a positive constant when β 2 is large).

Original: More formal results are shown in Lemma G.6.
Revised: More formal results are shown in Lemma G.6.

Original: Lemma G.6.
Revised: Lemma G.6.

Original: Under Assumption 2.1 and 2.2, when the hyperparameters satisfy: i) β 1 < √ β 2 < 1 , ii) β 2 is large enough such that A ( β 2 ) is small enough to satisfy (34) , where A ( β 2 ) is a non-negative constant that approaches 0 when β 2 approaches 1 .
Revised: Under Assumption 2.1 and 2.2, when the hyperparameters satisfy: i) β 1 < √ β 2 < 1 , ii) β 2 is large enough such that A ( β 2 ) is small enough to satisfy (34) , where A ( β 2 ) is a non-negative constant that approaches 0 when β 2 approaches 1 .

Original: More Speciﬁcally, A ( β 2 ) needs to satisfy ( ρ 1 , ρand ρ 3 are deﬁned in Appendix F.1).
Revised: More Speciﬁcally, A ( β 2 ) needs to satisfy ( ρ 1 , ρand ρ 3 are deﬁned in Appendix F.1).

Original: [Equation]
Revised: [Equation]

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: we have the following result when k is large enough such that β ( k − 1)
Revised: we have the following result when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 4 :
Revised: n≤ β n 1 √ k − 1 and k ≥ 4 :

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Proof of Lemma G.6 can be seen in Appendix G.9.
Revised: Proof of Lemma G.6 can be seen in Appendix G.9.

Original: Remark G.7 .
Revised: Remark G.7 .

Original: Condition (34) speciﬁes the smallest threshold of β 2 to ensure the convergence.
Revised: Condition (34) speciﬁes the smallest threshold of β 2 to ensure the convergence.

Original: This condition can be translated into the threshold funtion γ 1 ( n ) mentioned in Theorem 3.1.
Revised: This condition can be translated into the threshold funtion γ 1 ( n ) mentioned in Theorem 3.1.

Original: As a rough estimate, Lemma G.6 requires β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) / ( n 2 ρ ) (cid:1) , where ρ = ρ 1 ρ 2 ρ 3 .
Revised: As a rough estimate, Lemma G.6 requires β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) / ( n 2 ρ ) (cid:1) , where ρ = ρ 1 ρ 2 ρ 3 .

Original: As discussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: As discussed in Appendix F.1, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: When ρ 1 , ρ 2 , ρ 3 achieve their upper bound at the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .
Revised: When ρ 1 , ρ 2 , ρ 3 achieve their upper bound at the same time, we get the worst case bound β 2 ≥ γ 1 ( n ) = 1 − O (cid:0) (1 − β n 1 ) /n 4 .

Original: 5 (cid:1) .
Revised: 5 (cid:1) .

Original: However, ρ is highly dependent on the problem instance f ( x ) and training process.
Revised: However, ρ is highly dependent on the problem instance f ( x ) and training process.

Original: Our experiments in Appendix B shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lower than it appears to be.
Revised: Our experiments in Appendix B shows that ρ is often much smaller than its worst case bound, making the threshold of β 2 lower than it appears to be.

Original: Note that for adaptive gradient methods, we are not the ﬁrst to provide the threshold on β 2 for convergence guarantee: a similar threshold on β 2 was ﬁrstly provided for RMSProp by (Shi et al., 2020).
Revised: Note that for adaptive gradient methods, we are not the ﬁrst to provide the threshold on β 2 for convergence guarantee: a similar threshold on β 2 was ﬁrstly provided for RMSProp by (Shi et al., 2020).

Original: We prove that the threshold also exists for Adam, but with extra dependence on β 1 .
Revised: We prove that the threshold also exists for Adam, but with extra dependence on β 1 .

Original: Remark G.8 .
Revised: Remark G.8 .

Original: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.
Revised: We emphasize that the constant term O ( √ D 0 ) will vanish to 0 as β 2 goes to 1.

Original: This can be seen in the proof in Appendix G.9 (the deﬁnition of F 4 ) and Remark G.14.
Revised: This can be seen in the proof in Appendix G.9 (the deﬁnition of F 4 ) and Remark G.14.

Original: Based on Lemma G.6, we can further rewrite (31) as follows.
Revised: Based on Lemma G.6, we can further rewrite (31) as follows.

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 3.1 is concluded by plugging (35) into Descent Lemma (25) and then taking telescope some from k = t 0 to k = T .
Revised: The proof of Theorem 3.1 is concluded by plugging (35) into Descent Lemma (25) and then taking telescope some from k = t 0 to k = T .

Original: These steps are quite standard in non-convex optimization.
Revised: These steps are quite standard in non-convex optimization.

Original: We ﬁnish these calculation in Lemma G.
Revised: We ﬁnish these calculation in Lemma G.

Original: Lemma G.9.
Revised: Lemma G.9.

Original: When inequality (35) holds, we have the following results based on Descent Lemma (25) :
Revised: When inequality (35) holds, we have the following results based on Descent Lemma (25) :

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma G.9 can be seen in Appendix G.10.
Revised: Proof of Lemma G.9 can be seen in Appendix G.10.

Original: Now, the whole proof of Theorem 3.1 is completed.
Revised: Now, the whole proof of Theorem 3.1 is completed.

Original: G.2 Proof of Lemma G. 
Revised: G.2 Proof of Lemma G.

Original: To prove Lemma G.1, we only prove (28).
Revised: To prove Lemma G.1, we only prove (28).

Original: The proof for (27) can be analogized from the proof of (28).
Revised: The proof for (27) can be analogized from the proof of (28).

Original: We discuss the following two cases:
Revised: We discuss the following two cases:

Original: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have
Revised: Case 1: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≤ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: where ( a ) is because of √ v l,k,i ≥ (cid:112) β
Revised: where ( a ) is because of √ v l,k,i ≥ (cid:112) β

Original: i 2 √ v l,k, 0 .
Revised: i 2 √ v l,k, 0 .

Original: Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have ∂
Revised: Case 2: When ∂ l f ( x k, 0 ) ( m l,k,i − ∂ l f i ( x k, 0 )) ≥ 0 , we have ∂

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) is due to 1 √ 1+ x ≥ 1 − x 2 .
Revised: where ( ∗ ) is due to 1 √ 1+ x ≥ 1 − x 2 .

Original: We further have
Revised: We further have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where ( ∗∗ ) is due to the condition in the Lemma max i | ∂ l f i ( x τ k, 0 )
Revised: where ( ∗∗ ) is due to the condition in the Lemma max i | ∂ l f i ( x τ k, 0 )

Original: | ≥ Q k ≥ n (cid:52) nk .
Revised: | ≥ Q k ≥ n (cid:52) nk .

Original: Plug the above result into (36) we have
Revised: Plug the above result into (36) we have

Original: [Equation]
Revised: [Equation]

Original: In the last step ( ∗ ∗ ∗ ) , we use the following Lemma G.10, which is based on (Shi et al., 2020).
Revised: In the last step ( ∗ ∗ ∗ ) , we use the following Lemma G.10, which is based on (Shi et al., 2020).

Original: Lemma G.10.
Revised: Lemma G.10.

Original: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisﬁes max i
Revised: Under Assumption 2.1, if the l -th component of ∇ f i ( x k, 0 ) satisﬁes max i

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof idea of Lemma G.10 is similar as Lemma F.1 in (Shi et al., 2020), but with the following differences:
Revised: The proof idea of Lemma G.10 is similar as Lemma F.1 in (Shi et al., 2020), but with the following differences:

Original: The condition of Lemma F.1 in (Shi et al., 2020)
Revised: The condition of Lemma F.1 in (Shi et al., 2020)

Original: both require the full batch gradient | ∂ l f ( x k, 0 )
Revised: both require the full batch gradient | ∂ l f ( x k, 0 )

Original: | ≥ nQ k , which is different from our condition.
Revised: | ≥ nQ k , which is different from our condition.

Original: Here, we choose the condition of Lemma G. because it meets the need of our decomposition strategy in (31) .
Revised: Here, we choose the condition of Lemma G. because it meets the need of our decomposition strategy in (31) .

Original: The constant Q k is different.
Revised: The constant Q k is different.

Original: The difference is due to the different update formula of RMSProp and general Adam.
Revised: The difference is due to the different update formula of RMSProp and general Adam.

Original: Anyhow, Lemma G.10 can be easily proved following most of the steps in Lemma F.1 in (Shi et al., 2020).
Revised: Anyhow, Lemma G.10 can be easily proved following most of the steps in Lemma F.1 in (Shi et al., 2020).

Original: We omit the proof for brevity.
Revised: We omit the proof for brevity.

Original: Combining the above Case 1 and Case 2 , we have
Revised: Combining the above Case 1 and Case 2 , we have

Original: [Equation]
Revised: [Equation]

Original: i ( x k, 0 ) | , where δ 1 = (1 − β 2 )4 nρ 2 β n+
Revised: i ( x k, 0 ) | , where δ 1 = (1 − β 2 )4 nρ 2 β n+

Original: [Equation]
Revised: [Equation]

Original: √ β n 2 −(cid:19) .
Revised: √ β n 2 −(cid:19) .

Original: Proof is completed.
Revised: Proof is completed.

Original: G.3 Proof of Lemma G. 
Revised: G.3 Proof of Lemma G.

Original: We ﬁrst prove the result for j = 1 .
Revised: We ﬁrst prove the result for j = 1 .

Original: We discuss the following two cases.
Revised: We discuss the following two cases.

Original: Case 1: when ∂ l f ( x k, 0 ) √ v l,k, 0 ≥ ∂ l f ( x k − 1 , 0 ) √ v l,k − 1 , 0 : when ∂ l f ( x k, 0 ) ≤ 0 , we have
Revised: Case 1: when ∂ l f ( x k, 0 ) √ v l,k, 0 ≥ ∂ l f ( x k − 1 , 0 ) √ v l,k − 1 , 0 : when ∂ l f ( x k, 0 ) ≤ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) > 0 , we have
Revised: When ∂ l f ( x k, 0 ) > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In conclusion, we have:
Revised: In conclusion, we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where δ 1 =
Revised: where δ 1 =

Original: [Equation]
Revised: [Equation]

Original: is a constant that goes to 0 when β 2 goes to 1.
Revised: is a constant that goes to 0 when β 2 goes to 1.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Note that
Revised: Note that

Original: Case 2: when
Revised: Case 2: when

Original: [Equation]
Revised: [Equation]

Original: : when ∂ l f ( x k, 0 ) ≥ 0 , we have
Revised: : when ∂ l f ( x k, 0 ) ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: When ∂ l f ( x k, 0 ) <
Revised: When ∂ l f ( x k, 0 ) <

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Following the same strategy as in Case
Revised: Following the same strategy as in Case

Original: [Equation]
Revised: [Equation]

Original: we can show that
Revised: we can show that

Original: [Equation]
Revised: [Equation]

Original: which further implies
Revised: which further implies

Original: [Equation]
Revised: [Equation]

Original: Case 1 and Case 2 together, we have
Revised: Case 1 and Case 2 together, we have

Original: [Equation]
Revised: [Equation]

Original: Now we consider the case when j > 1 .
Revised: Now we consider the case when j > 1 .

Original: Based on the above inequality, we have
Revised: Based on the above inequality, we have

Original: [Table]
Revised: [Table]

Original: The proof is completed.
Revised: The proof is completed.

Original: G.4 Proof of Lemma G. 
Revised: G.4 Proof of Lemma G.

Original: To prove Lemma G.4, we need to further decompose E [ (cid:80) l large ( b 1 )] .
Revised: To prove Lemma G.4, we need to further decompose E [ (cid:80) l large ( b 1 )] .

Original: First and foremost, we write (cid:80) n − 1 i =

Revised: First and foremost, we write (cid:80) n − 1 i =

Original: m l,k,i in an explicit form.
Revised: m l,k,i in an explicit form.

Original: m l,k,i = (1 − β 1 ) { ∂ l f τ k
Revised: m l,k,i = (1 − β 1 ) { ∂ l f τ k

Original: [Equation]
Revised: [Equation]

Original: Since ∂ l f ( x 1 , 0 ) =
Revised: Since ∂ l f ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =
Revised: ∂ l f ( x 1 , 0 )(1 − β 1 )(1 + β 1 + · · · + β ∞ 1 ) and ∂ l f ( x 1 , 0 ) =

Original: (cid:80) n − 1 i =

Revised: (cid:80) n − 1 i =

Original: ∂ l f i ( x 1 , 0 ) , we have
Revised: ∂ l f i ( x 1 , 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Plugging (39) into (38), we have m l,
Revised: Plugging (39) into (38), we have m l,

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: +11 f τ k − 1 ,n − 1 ( x k − 1 ,n − 1 ) + · · · + β i + nf τ k − 1 , 0 ( x k − 1 , 0 ) + · · · + β ( k − 2) n + i +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + i∂ l f τ 1 , 0 ( x 1 , 0 )
Revised: +11 f τ k − 1 ,n − 1 ( x k − 1 ,n − 1 ) + · · · + β i + nf τ k − 1 , 0 ( x k − 1 , 0 ) + · · · + β ( k − 2) n + i +1∂ l f τ 1 ,n − 1 ( x 1 ,n − 1 ) + · · · + β ( k − 1) n + i∂ l f τ 1 , 0 ( x 1 , 0 )

Original: + β ( k − 1) n
Revised: + β ( k − 1) n

Original: + i +1∂ l f 0 ( x 1 , 0 )
Revised: + i +1∂ l f 0 ( x 1 , 0 )

Original: + · · · + β ( k − 1) n
Revised: + · · · + β ( k − 1) n

Original: + i +1∂ l f n − 1 ( x 1 , 0 )
Revised: + i +1∂ l f n − 1 ( x 1 , 0 )

Original: + β ( k − 1) n
Revised: + β ( k − 1) n

Original: + i +2∂ l f 0 ( x 1 , 0 )
Revised: + i +2∂ l f 0 ( x 1 , 0 )

Original: + · · · + β ( k − 1) n
Revised: + · · · + β ( k − 1) n

Original: + i +2∂ l f n − 1 ( x 1 , 0 )
Revised: + i +2∂ l f n − 1 ( x 1 , 0 )

Original: + · · · +
Revised: + · · · +

Original: [Equation]
Revised: [Equation]

Original: 1 ∂ l f 0 ( x 1 , 0 ) + · · · +
Revised: 1 ∂ l f 0 ( x 1 , 0 ) + · · · +

Original: [Equation]
Revised: [Equation]

Original: ∞ 1 ∂ l f n − 1 ( x 1 , 0 ) } (40)
Revised: ∞ 1 ∂ l f n − 1 ( x 1 , 0 ) } (40)

Original: Using (40), we have (For each ‘epoch’, we suggest readers to read from bottom to the top
Revised: Using (40), we have (For each ‘epoch’, we suggest readers to read from bottom to the top

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: −
Revised: −

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that in (41), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition will cause trouble in bounding the (cid:80) n − 1 i =

Revised: Note that in (41), the power of β 1 grows slower in the blue part (when k = 0 ), such a transition will cause trouble in bounding the (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) .

Original: Therefore, we need to deﬁne an auxillary sequence M (cid:48) l,k which does not involve such a phase transition.
Revised: Therefore, we need to deﬁne an auxillary sequence M (cid:48) l,k which does not involve such a phase transition.

Original: We deﬁne M (cid:48) l,k as follows.
Revised: We deﬁne M (cid:48) l,k as follows.

Original: (For each ‘epoch’, we suggest readers to read from bottom to the top.)
Revised: (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: 2( n −
Revised: 2( n −

Original: [Equation]
Revised: [Equation]

Original: ∂ l f n −
Revised: ∂ l f n −

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k −
Revised: k −

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: ∂ l f n
Revised: ∂ l f n

Original: [Equation]
Revised: [Equation]

Original: (cid:125) 

Revised: (cid:125)

Original: -thepoch ... } (42)
Revised: -thepoch ... } (42)

Original: Now we bound (cid:80) n − 1 i =

Revised: Now we bound (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M (cid:48) l,k .
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) with the help of M (cid:48) l,k .

Original: Denoting F l,k := (cid:80) n − 1 i =

Revised: Denoting F l,k := (cid:80) n − 1 i =

Original: ∂ l f i ( x k, 0 ) , we have:
Revised: ∂ l f i ( x k, 0 ) , we have:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k .
Revised: To proceed, we use the relation 1 = (1 − β 1 )(1 + β 1 + β 21 + · · · ) to rewrite F l,k .

Original: F l,k := n − 1 (cid:88) i =
Revised: F l,k := n − 1 (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: For the sake of better presentation
Revised: For the sake of better presentation

Original: [Equation]
Revised: [Equation]

Original: we rewrite F l,k and M (cid:48) l,k as follows:
Revised: we rewrite F l,k and M (cid:48) l,k as follows:

Original: [Equation]
Revised: [Equation]

Original: where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .
Revised: where ( F l,k ) j contains the summand of F l,k in the j -th epoch, j = k, k − 1 , · · · − ∞ .

Original: Similarly for
Revised: Similarly for

Original: M (cid:48) l,k .
Revised: M (cid:48) l,k .

Original: Now, we separate M (cid:48) l,k − F l,k into two parts.
Revised: Now, we separate M (cid:48) l,k − F l,k into two parts.

Original: [Equation]
Revised: [Equation]

Original: Now, we rewrite E [ (cid:80) l large ( b 1 )] :
Revised: Now, we rewrite E [ (cid:80) l large ( b 1 )] :

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: l large
Revised: l large

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We bound (1) , (2) and (3) respectively.
Revised: We bound (1) , (2) and (3) respectively.

Original: We bound (1) and (3) in the following Lemma G.11 and
Revised: We bound (1) and (3) in the following Lemma G.11 and

Original: Lemma G.12.
Revised: Lemma G.12.

Original: Since the difference of M l,k − M (cid:48) l,k only occurs in the high order terms of the inﬁnite sum sequence, (1) is expected to vanish as k grows.
Revised: Since the difference of M l,k − M (cid:48) l,k only occurs in the high order terms of the inﬁnite sum sequence, (1) is expected to vanish as k grows.

Original: Similarly for (3) .
Revised: Similarly for (3) .

Original: Lemma G.11.
Revised: Lemma G.11.

Original: When k is large enough such that: β ( k − 1) n
Revised: When k is large enough such that: β ( k − 1) n

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: 1 and k ≥ 4 , then we have
Revised: 1 and k ≥ 4 , then we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where G 4 = d (cid:113) 2 ρ 23 β
Revised: where G 4 = d (cid:113) 2 ρ 23 β

Original: [Equation]
Revised: [Equation]

Original: Lemma G.12.
Revised: Lemma G.12.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤
Revised: n≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: 1 and k ≥ 4 , then we have
Revised: 1 and k ≥ 4 , then we have

Original: [Equation]
Revised: [Equation]

Original: where G 5 =
Revised: where G 5 =

Original: [Equation]
Revised: [Equation]

Original: (cid:113)
Revised: (cid:113)

Original: [Equation]
Revised: [Equation]

Original: We relegate the proof of Lemma G.11 and G.12 to Appendix G.5 and G.6.
Revised: We relegate the proof of Lemma G.11 and G.12 to Appendix G.5 and G.6.

Original: Now we bound (2) .
Revised: Now we bound (2) .

Original: This part involves the difﬁculties (i), (ii) and (iii) mentioned in Appendix G.1.
Revised: This part involves the difﬁculties (i), (ii) and (iii) mentioned in Appendix G.1.

Original: We bound (2) in Lemma G.13.
Revised: We bound (2) in Lemma G.13.

Original: Lemma G.13.
Revised: Lemma G.13.

Original: When k is large enough such that: β ( k − 1)
Revised: When k is large enough such that: β ( k − 1)

Original: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have
Revised: n≤ β n 1 √ k − 1 and k ≥ 2 , then we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: The constant terms G 6 and G 7 are speciﬁed in Appendix G.7.
Revised: The constant terms G 6 and G 7 are speciﬁed in Appendix G.7.

Original: Proof is shown in Appendix G.
Revised: Proof is shown in Appendix G.

Original: [Equation]
Revised: [Equation]

Original: Combining Lemma G.11, G.12 and G.13 together, we conclude the proof.
Revised: Combining Lemma G.11, G.12 and G.13 together, we conclude the proof.

Original: [Equation]
Revised: [Equation]

Original: G.5 Proof of Lemma G. 
Revised: G.5 Proof of Lemma G.

Original: By deﬁnition of M l,k and M (cid:48) l,k (see (41) and (42)), they only differ when k ≤ 0 .
Revised: By deﬁnition of M l,k and M (cid:48) l,k (see (41) and (42)), they only differ when k ≤ 0 .

Original: More speciﬁcally,
Revised: More speciﬁcally,

Original: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)
Revised: We have (For each ‘epoch’, we suggest readers to read from bottom to the top.)

Original: [Equation]
Revised: [Equation]

Original: 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂
Revised: 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn +2(cid:17) ∂

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + (cid:16) β ( k − 1) n +2− β kn +1(cid:17)
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β kn + n +1(cid:17) ∂ l f n − 1 ( x 1 , 0 ) + (cid:16) β ( k − 1) n +2− β kn +1(cid:17)

Original: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) -

Revised: ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +2− β kn + n(cid:17) ∂ l f n − 1 ( x 1 , 0 ) (cid:124) (cid:123)(cid:122) (cid:125) -

Original: -th epoch + (cid:16) β
Revised: -th epoch + (cid:16) β

Original: ( k − 1) n + n +2− β
Revised: ( k − 1) n + n +2− β

Original: ( k +1) n + n(cid:17) ∂
Revised: ( k +1) n + n(cid:17) ∂

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +2− β
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n + n +2− β

Original: ( k +1) n + n + n − 1(cid:17)
Revised: ( k +1) n + n + n − 1(cid:17)

Original: ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n + n +1(cid:17)
Revised: ∂ l f n − 1 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n +2(cid:17) ∂ l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +4− β ( k +1) n + n +1(cid:17)

Original: ∂ l f n − 1 ( x 1 , 0 )
Revised: ∂ l f n − 1 ( x 1 , 0 )

Original: + (cid:16) β
Revised: + (cid:16) β

Original: ( k − 1) n +3− β
Revised: ( k − 1) n +3− β

Original: ( k +1) n +1(cid:17) ∂
Revised: ( k +1) n +1(cid:17) ∂

Original: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β ( k +1) n + n(cid:17) ∂ l f n − 1 ( x 1 ,
Revised: l f 0 ( x 1 , 0 ) + · · · + (cid:16) β ( k − 1) n +3− β ( k +1) n + n(cid:17) ∂ l f n − 1 ( x 1 ,

Original: [Equation]
Revised: [Equation]

Original: We start with the 1st column in the ‘0-th epoch’
Revised: We start with the 1st column in the ‘0-th epoch’

Original: (from the bottom to the top).
Revised: (from the bottom to the top).

Original: The 1st column in the ‘0-th epoch’ =
Revised: The 1st column in the ‘0-th epoch’ =

Original: β (
Revised: β (

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can be bounded as follows.
Revised: Similarly, we can bound every column in the in the ‘0-th epoch’, e.g. last ( n − 1 -th) column can be bounded as follows.

Original: The last column in the ‘0-th epoch’ =
Revised: The last column in the ‘0-th epoch’ =

Original: β ( k − 1) n +

Revised: β ( k − 1) n +

Original: [Equation]
Revised: [Equation]

Original: Summing up all the columns, we have
Revised: Summing up all the columns, we have

Original: The ‘0-th
Revised: The ‘0-th

Original: [Equation]
Revised: [Equation]

Original: Using the same technique
Revised: Using the same technique

Original: [Equation]
Revised: [Equation]

Original: it can be shown that
Revised: it can be shown that

Original: [Equation]
Revised: [Equation]

Original: th
Revised: th

Original: [Equation]
Revised: [Equation]

Original: Plugging all these results in (48)
Revised: Plugging all these results in (48)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: That is to say, when k is large enough such that β ( k − 1)
Revised: That is to say, when k is large enough such that β ( k − 1)

Original: n≤ β n 1 √ k − 1 we have (cid:12)(cid:12)
Revised: n≤ β n 1 √ k − 1 we have (cid:12)(cid:12)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining with Lemma G.10, the proof is completed.
Revised: Combining with Lemma G.10, the proof is completed.

Original: G.6 Proof of Lemma G. 
Revised: G.6 Proof of Lemma G.

Original: We start with (cid:16) M (cid:48) l,k (cid:17) 0 − ( F l,k ) 0 .
Revised: We start with (cid:16) M (cid:48) l,k (cid:17) 0 − ( F l,k ) 0 .

Original: (cid:0) M (cid:48)
Revised: (cid:0) M (cid:48)

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:125)
Revised: (cid:125)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We further have
Revised: We further have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Using the same calculation
Revised: Using the same calculation

Original: [Equation]
Revised: [Equation]

Original: we can get the following result:
Revised: we can get the following result:

Original: [Equation]
Revised: [Equation]

Original: Repeat this calculation, we have
Revised: Repeat this calculation, we have

Original: [Equation]
Revised: [Equation]

Original: When β ( k − 1)
Revised: When β ( k − 1)

Original: n≤ β n 1 √ k − 1 , we further have:
Revised: n≤ β n 1 √ k − 1 , we further have:

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) :
Revised: where ( ∗ ) :

Original: [Equation]
Revised: [Equation]

Original: ≤ (cid:113) 2 nk when k ≥ 4 .
Revised: ≤ (cid:113) 2 nk when k ≥ 4 .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Combining with Lemma G.10, the proof is completed with constant G 5 deﬁned in Lemma G.12.
Revised: Combining with Lemma G.10, the proof is completed with constant G 5 deﬁned in Lemma G.12.

Original: G.7 Proof of Lemma G. 
Revised: G.7 Proof of Lemma G.

Original: In this section, we derive a lower bound for E (cid:104)(cid:80)
Revised: In this section, we derive a lower bound for E (cid:104)(cid:80)

Original: [Equation]
Revised: [Equation]

Original: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).
Revised: We will use the ideas mentioned in Appendix G.1 (i.e., Step 1,2 and 3).

Original: We ﬁrst rewrite “ l large" into indicator function as follows:
Revised: We ﬁrst rewrite “ l large" into indicator function as follows:

Original: [Table]
Revised: [Table]

Original: where I k :
Revised: where I k :

Original: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := (cid:52) 1 n √ n √ k
Revised: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := (cid:52) 1 n √ n √ k

Original: [Table] [Table]
Revised: [Table] [Table]

Original: − β 2 ) n β n 2 (cid:17) .
Revised: − β 2 ) n β n 2 (cid:17) .

Original: We also deﬁne I k − j :=
Revised: We also deﬁne I k − j :=

Original: I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , it will be used later.
Revised: I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , it will be used later.

Original: We take the conditional expectation over the history before x k, 0 .
Revised: We take the conditional expectation over the history before x k, 0 .

Original: We ﬁrst focus on
Revised: We ﬁrst focus on

Original: E k (cid:104)(cid:80) dl
Revised: E k (cid:104)(cid:80) dl

Original: =

Revised: =

Original: I k ∂ l f ( x k, 0 ) √ v k, 0 (cid:16)(cid:16) M (cid:48)
Revised: I k ∂ l f ( x k, 0 ) √ v k, 0 (cid:16)(cid:16) M (cid:48)

Original: l,k (cid:17) k − ( F l,k ) k (cid:17)(cid:105)
Revised: l,k (cid:17) k − ( F l,k ) k (cid:17)(cid:105)

Original: and we delegate the history part for later analysis.
Revised: and we delegate the history part for later analysis.

Original: For each possible (cid:16) M (cid:48) l,k (cid:17) k − ( F l,k ) k , we ﬁrst convert all x k,i into x k, 0 using Lemma F.2.
Revised: For each possible (cid:16) M (cid:48) l,k (cid:17) k − ( F l,k ) k , we ﬁrst convert all x k,i into x k, 0 using Lemma F.2.

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: M
Revised: M

Original: [Equation]
Revised: [Equation]

Original: l,k
Revised: l,k

Original: [Equation]
Revised: [Equation]

Original: k contains ( n − 1)
Revised: k contains ( n − 1)

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: terms of x k,i (with i (cid:54) = 0 ), we have
Revised: terms of x k,i (with i (cid:54) = 0 ), we have

Original: [Equation]
Revised: [Equation]

Original: Lemma
Revised: Lemma

Original: [Equation]
Revised: [Equation]

Original: Using the same strategy for analyzing the color-ball toy example, we have
Revised: Using the same strategy for analyzing the color-ball toy example, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the following derivation.
Revised: We denote J 1 := (cid:0) − 1 n β 1 − 2 n β 21 − · · · − n − 1 n β n − 1(cid:1) , it will be used repeatedly in the following derivation.

Original: Now we move one step further exert E k − 1 ( · ) .
Revised: Now we move one step further exert E k − 1 ( · ) .

Original: In particular, we need to calculate
Revised: In particular, we need to calculate

Original: E k
Revised: E k

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: The blue term is the residue from the k -th epoch.
Revised: The blue term is the residue from the k -th epoch.

Original: The red term is the main component in the ( k − 1) -th epoch.
Revised: The red term is the main component in the ( k − 1) -th epoch.

Original: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitz property.
Revised: Before calculating E k − 1 ( · ) , we need to convert all the variable x into x k − 1 , 0 using Lipschitz property.

Original: First of all, we work on ( a ) .
Revised: First of all, we work on ( a ) .

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] 
Revised: [Table] [Equation] [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: LemmaG.2 ≥
Revised: LemmaG.2 ≥

Original: [Table]
Revised: [Table]

Original: where I k,k − j :=
Revised: where I k,k − j :=

Original: I (cid:16) max i | ∂ l f i ( x k, 0 )
Revised: I (cid:16) max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and ˜ I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) and ˜ I k,k − j := I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) | ≤ (cid:80) kp = k − j Q p (cid:17) .

Original: By Lemma G.3, we know I k,k − 1 =
Revised: By Lemma G.3, we know I k,k − 1 =

Original: I k − 1 , so we have
Revised: I k − 1 , so we have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now we derive a lower bound for (b).
Revised: Now we derive a lower bound for (b).

Original: Similarly as before, we rewrite (cid:16) M (cid:48) l,k (cid:17) k − 1 and ( F l,k ) k − 1 as follows.
Revised: Similarly as before, we rewrite (cid:16) M (cid:48) l,k (cid:17) k − 1 and ( F l,k ) k − 1 as follows.

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We now calculate ( b ) .
Revised: We now calculate ( b ) .

Original: Using the same idea as in ( a ) , we have
Revised: Using the same idea as in ( a ) , we have

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: To proceed, we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:12)(cid:12)(cid:12) .
Revised: To proceed, we derive an upper bound for (cid:12)(cid:12)(cid:12) M l,k − 1 − ( F l,k − 1 ) k − 1 (cid:12)(cid:12)(cid:12) .

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .
Revised: The red term will be handled using color-ball method when taking conditional expectation Ex k − 1 ( · ) .

Original: Now we have derived lower bounds for both ( a ) and ( b ) .
Revised: Now we have derived lower bounds for both ( a ) and ( b ) .

Original: Combining together, we have:
Revised: Combining together, we have:

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: For the ﬁrst term in the above inequality
Revised: For the ﬁrst term in the above inequality

Original: [Equation]
Revised: [Equation]

Original: we can calculate it using the idea in the color-ball toy example:
Revised: we can calculate it using the idea in the color-ball toy example:

Original: [Equation]
Revised: [Equation]

Original: To proceed
Revised: To proceed

Original: [Equation]
Revised: [Equation]

Original: we further take E k − 2 ( · ) and bound
Revised: we further take E k − 2 ( · ) and bound

Original: E k − 
Revised: E k − 

Original: [Equation]
Revised: [Equation]

Original:   .
Revised:   .

Original: Repeat this process until k = 1 , we have:
Revised: Repeat this process until k = 1 , we have:

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: + Error 1 + Error 2 + Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .
Revised: + Error 1 + Error 2 + Errorwhere ( ∗ ) holds for large k such that β ( k − 1) n ≤ β n 1 √ k − 1 .

Original: We specify Error 1 , Error 2 , Error 3 as follows.
Revised: We specify Error 1 , Error 2 , Error 3 as follows.

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: − d
Revised: − d

Original: [Table] [Table]
Revised: [Table] [Table]

Original: β
Revised: β

Original: [Table]
Revised: [Table]

Original: 2 | J 1 | n [ Q k + Q k − 1 + β n
Revised: 2 | J 1 | n [ Q k + Q k − 1 + β n

Original: [Table]
Revised: [Table]

Original: Q k + Q k −
Revised: Q k + Q k −

Original: [Table]
Revised: [Table]

Original: + Q k −
Revised: + Q k −

Original: [Table]
Revised: [Table]

Original: ) + · ·
Revised: ) + · ·

Original: [Table]
Revised: [Table]

Original: Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a ﬁnite constant, we have (cid:80) ∞
Revised: Since δ 2 = lim k →∞ (cid:80) k − 1 j =1 ( β n 1 ) j (cid:113) kk − j is a ﬁnite constant, we have (cid:80) ∞

Original: j =0 β jn 1 (cid:52) n ( k − j )
Revised: j =0 β jn 1 (cid:52) n ( k − j )

Original: = (cid:52) 

Revised: = (cid:52)

Original: δ 2 √ nk .
Revised: δ 2 √ nk .

Original: In addition, we have
Revised: In addition, we have

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj =
Revised: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80) Jj =

Original: [Equation]
Revised: [Equation]

Original: This inequality can be simply proved by taking the integral over 1 √ k .
Revised: This inequality can be simply proved by taking the integral over 1 √ k .

Original: The ﬁnal inequality is due to
Revised: The ﬁnal inequality is due to

Original: Q k − 1 < 2 Q k .
Revised: Q k − 1 < 2 Q k .

Original: Now, we have
Revised: Now, we have

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Based on the calculation in Lemma F.
Revised: Based on the calculation in Lemma F.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: Error
Revised: Error

Original: [Equation]
Revised: [Equation]

Original: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have
Revised: Since 1 − β 1 1 − β n 1 ≤ 1 and | J 1 | ≤ n , we have

Original: [Equation]
Revised: [Equation]

Original: Using the fact that
Revised: Using the fact that

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i = |
Revised: i = |

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:33)
Revised: (cid:33)

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where J
Revised: where J

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (if needed, we can further bound J 1 by n for simplicity).
Revised: (if needed, we can further bound J 1 by n for simplicity).

Original: This conclude the proof.
Revised: This conclude the proof.

Original: G.8 Proof of Lemma G. 
Revised: G.8 Proof of Lemma G.

Original: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .
Revised: We now derive upper bounds for (cid:80) l large ( a 2 ) and (cid:80) l large ( b 2 ) .

Original: The upper bound for (cid:80) l large ( a 2 ) is very straightforward using inequality (24).
Revised: The upper bound for (cid:80) l large ( a 2 ) is very straightforward using inequality (24).

Original: [Equation]
Revised: [Equation]

Original: Now we shift gear to (cid:80)
Revised: Now we shift gear to (cid:80)

Original: [Equation]
Revised: [Equation]

Original: large
Revised: large

Original: [Equation]
Revised: [Equation]

Original: b
Revised: b

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) | .
Revised: ) | .

Original: To proceed, we need an upper bound for (cid:80) l large (cid:80) n − 1 i =

Revised: To proceed, we need an upper bound for (cid:80) l large (cid:80) n − 1 i =

Original: | m l,k,i − ∂ l f i ( x k, 0 ) | .
Revised: | m l,k,i − ∂ l f i ( x k, 0 ) | .

Original: For each i
Revised: For each i

Original: [Equation]
Revised: [Equation]

Original: we perform the following decomposition.
Revised: we perform the following decomposition.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: To start, we bound ( d 1 ) .
Revised: To start, we bound ( d 1 ) .

Original: [Equation]
Revised: [Equation]

Original: Now, we bound ( d 2 ) .
Revised: Now, we bound ( d 2 ) .

Original: Recall (cid:80) n − 1 i =

Revised: Recall (cid:80) n − 1 i =

Original: ∂ l f i ( x 1 , 0 ) =
Revised: ∂ l f i ( x 1 , 0 ) =

Original: ∂ l f ( x 1 , 0 ) , we have
Revised: ∂ l f ( x 1 , 0 ) , we have

Original: [Equation]
Revised: [Equation]

Original: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.
Revised: Note that for any i ∈ [0 , n − 1] , j ∈ [0 , n − 1] , t ∈ [1 , k − 1] , we have the following result.

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80)
Revised: where ( ∗ ) uses the following fact: consider integers k > J > 0 , we have (cid:80)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Plugging (63) into (62) and re-arranging the index, we have
Revised: Plugging (63) into (62) and re-arranging the index, we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the second last inequality holds because: when k ≥ 4 ,
Revised: where the second last inequality holds because: when k ≥ 4 ,

Original: [Equation]
Revised: [Equation]

Original: i ) holds when k is large enough such that β ( k − 1) n
Revised: i ) holds when k is large enough such that β ( k − 1) n

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: Further, β n 1 √ k − 1 ≤ √
Revised: Further, β n 1 √ k − 1 ≤ √

Original: [Equation]
Revised: [Equation]

Original: 1 √ k when k ≥ 2 .
Revised: 1 √ k when k ≥ 2 .

Original: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .
Revised: Now, we have derived upper bounds for ( d 1 ) and ( d 2 ) .

Original: Plugging (64) and (61) into (60), we conclude the proof.
Revised: Plugging (64) and (61) into (60), we conclude the proof.

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: The proof is concluded by adding expectation on both sides of the inequality
Revised: The proof is concluded by adding expectation on both sides of the inequality

Original: [Equation]
Revised: [Equation]

Original: G.9 Proof of Lemma G. 
Revised: G.9 Proof of Lemma G.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.
Revised: where constant terms G 4 , G 5 , G 6 , G 7 can be seen at the end of Appendix G.7.

Original: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have
Revised: Since α = arg max l =1 , 2 , ··· ,d | ∂ l f ( x k, 0 ) | and (cid:80) l large ∂ l f ( x k, 0 ) 2 √ v l,k,≥ ∂ α f ( x k, 0 ) 2 √ v α,k, 0 , we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Now, we discuss two cases.
Revised: Now, we discuss two cases.

Original: Case (a): when | ∂ α f ( x k, 0 )
Revised: Case (a): when | ∂ α f ( x k, 0 )

Original: | ≥ 

Revised: | ≥

Original: √∆ 1 (1 −
Revised: √∆ 1 (1 −

Original: [Equation]
Revised: [Equation]

Original: 2 ) √ D 1 nkd .
Revised: 2 ) √ D 1 nkd .

Original: In this case, we have the following result.
Revised: In this case, we have the following result.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation] 
Revised: [Equation] [Table] [Equation] [Table]

Original:  [Table]
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Now we consider the following two sub-cases.
Revised: Now we consider the following two sub-cases.

Original: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d :
Revised: • When ∂ α f ( x k, 0 ) 2 ≤ D 0 D 1 d :

Original: [Equation]
Revised: [Equation]

Original: • When ∂ α f
Revised: • When ∂ α f

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: x k

Original: [Equation]
Revised: [Equation]

Original: Combining together
Revised: Combining together

Original: [Equation]
Revised: [Equation]

Original: we have the following results for Case (a) .
Revised: we have the following results for Case (a) .

Original: [Equation]
Revised: [Equation]

Original: Case (b): When | ∂ α f ( x k
Revised: Case (b): When | ∂ α f ( x k

Original: 0 ) | < 4 √
Revised: 0 ) | < 4 √

Original: [Equation]
Revised: [Equation]

Original: , we have
Revised: , we have

Original: [Equation]
Revised: [Equation]

Original: where G 1 :
Revised: where G 1 :

Original: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .
Revised: = F 2 + F 3 4 √∆ 1 (1 − β 2 ) √ D 1 nd .

Original: Now, the following two claims are both true.
Revised: Now, the following two claims are both true.

Original: • Claim
Revised: • Claim

Original: [Equation]
Revised: [Equation]

Original: Claim
Revised: Claim

Original: [Equation]
Revised: [Equation]

Original: Combining Claim 1 and Claim
Revised: Combining Claim 1 and Claim

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where G 2 :=
Revised: where G 2 :=

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: Combining Case (a) and Case (b) together, we have
Revised: Combining Case (a) and Case (b) together, we have

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is because of (cid:107)∇ f ( x k, 0 ) (cid:107) 22 ≤ d∂ α f
Revised: where the last inequality is because of (cid:107)∇ f ( x k, 0 ) (cid:107) 22 ≤ d∂ α f

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: x
Revised: x

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: α f ( x k
Revised: α f ( x k

Original: [Equation]
Revised: [Equation]

Original: 0 ) | .
Revised: 0 ) | .

Original: Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − (cid:15), 1] , such that
Revised: Recall F 3 → 0 when β 2 → 1 , so there exists an interval (1 − (cid:15), 1] , such that

Original: [Equation]
Revised: [Equation]

Original: or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma
Revised: or equivalently F 3 ≤ 1 √ 10 D 1 d (note that F 3 is the same as “ A ( β 2 ) ” stated in the condition of Lemma

Original: G.6).
Revised: G.6).

Original: With such a choice of β 2 , we have the following results by changing all the F 3 into 1 √ 10 D 1 d :
Revised: With such a choice of β 2 , we have the following results by changing all the F 3 into 1 √ 10 D 1 d :

Original: E   (cid:88) l large
Revised: E   (cid:88) l large

Original: [Equation]
Revised: [Equation]

Original: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .
Revised: In inequality ( ∗ ) , we change F 4 into F 5 := (cid:113) D 0 D 1 d 1 √ 10 D 1 d .

Original: This is because: ﬁrst, F 4 = F 3 (cid:113) D 0 D 1 d ; second, F 3 ≤ 1 √ 10 D 1 d .
Revised: This is because: ﬁrst, F 4 = F 3 (cid:113) D 0 D 1 d ; second, F 3 ≤ 1 √ 10 D 1 d .

Original: The proof of Lemma G.6 is completed.
Revised: The proof of Lemma G.6 is completed.

Original: We restate all the constants as follows ( G 4 , G 5 , G 6 are speciﬁed in Appendix G.7):
Revised: We restate all the constants as follows ( G 4 , G 5 , G 6 are speciﬁed in Appendix G.7):

Original: [Equation]
Revised: [Equation]

Original: Remark G.14 .
Revised: Remark G.14 .

Original: We comment that we can always replace F 5 in the ﬁnal result by F 4 , i.e., we can choose not to apply the last inequality ( ∗ ) in the proof.
Revised: We comment that we can always replace F 5 in the ﬁnal result by F 4 , i.e., we can choose not to apply the last inequality ( ∗ ) in the proof.

Original: The beneﬁt of using F 4 is that F 4 monotonously decrease to 0 when increasing β 2 to 1.
Revised: The beneﬁt of using F 4 is that F 4 monotonously decrease to 0 when increasing β 2 to 1.

Original: This monotone property is not shown in the notation of F 5 .
Revised: This monotone property is not shown in the notation of F 5 .

Original: Nevertheless, we choose to use F 5 since it is a much cleaner constant.
Revised: Nevertheless, we choose to use F 5 since it is a much cleaner constant.

Original: G.10 Proof of Lemma G. 
Revised: G.10 Proof of Lemma G.

Original: Based on Descent Lemma, we have
Revised: Based on Descent Lemma, we have

Original: [Equation]
Revised: [Equation]

Original: Plugging in the r
Revised: Plugging in the r

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation]
Revised: [Equation]

Original: s. of (35)
Revised: s. of (35)

Original: [Equation]
Revised: [Equation]

Original: we have the following relation after rearranging.
Revised: we have the following relation after rearranging.

Original: [Equation]
Revised: [Equation]

Original:  .
Revised:  .

Original: Recall we have 2 (cid:16) √ T − √ t 0 − 1 (cid:17) ≤ (cid:80)
Revised: Recall we have 2 (cid:16) √ T − √ t 0 − 1 (cid:17) ≤ (cid:80)

Original: Tk
Revised: Tk

Original: = t 0 1 √ k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 .
Revised: = t 0 1 √ k , (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 .

Original: Further, since (cid:107)·(cid:107) 1 ≥ (cid:107)·(cid:107) 2 , we get the following relation when t 0 = 1 .
Revised: Further, since (cid:107)·(cid:107) 1 ≥ (cid:107)·(cid:107) 2 , we get the following relation when t 0 = 1 .

Original: [Equation]
Revised: [Equation]

Original: We specify all the constant as follows.
Revised: We specify all the constant as follows.

Original: For all the following constants, We keep the same notation as their appearance in their corresponding lemmas.
Revised: For all the following constants, We keep the same notation as their appearance in their corresponding lemmas.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where J 1 = (cid:0) − 1 n β 1 − 2 n β 21 −
Revised: where J 1 = (cid:0) − 1 n β 1 − 2 n β 21 −

Original: [Equation]
Revised: [Equation]

Original: δ
Revised: δ

Original: [Equation]
Revised: [Equation]

Original: = lim k
Revised: = lim k

Original: [Equation]
Revised: [Equation]

Original: (cid:80) k
Revised: (cid:80) k

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation]
Revised: [Equation]

Original: If needed, we can further bound J 1 by n for simplicity.
Revised: If needed, we can further bound J 1 by n for simplicity.

Original: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .
Revised: Further, ρ 1 , ρ 2 , ρ 3 are constants satisfying the following conditions for ∀ l = 1 , · · · , d .

Original: Usually, these constants can be different for different problems.
Revised: Usually, these constants can be different for different problems.

Original: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: In the worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.
Revised: ρ 3 is larger when ∂ l f i ( x k, 0 ) are more aligned.

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma G.9 is now completed.
Revised: The proof of Lemma G.9 is now completed.

Original: This also concludes the whole proof of Theorem 3.1.
Revised: This also concludes the whole proof of Theorem 3.1.

Original: G.11 Dissussion on Bias Correction Terms and Non-Zero (cid:15) (Hyperparameter for Numerical Stability)
Revised: G.11 Dissussion on Bias Correction Terms and Non-Zero (cid:15) (Hyperparameter for Numerical Stability)

Original: In the above analysis, we focus on Adam without bias correction terms and we consider (cid:15) = 

Revised: In the above analysis, we focus on Adam without bias correction terms and we consider (cid:15) =

Original: ( (cid:15) is the hyperparameter for numerical stability in Algorithm 1).
Revised: ( (cid:15) is the hyperparameter for numerical stability in Algorithm 1).

Original: For completeness, we now brieﬂy discuss how to incorperate the bias correction terms and non-zero (cid:15) into our analysis above.
Revised: For completeness, we now brieﬂy discuss how to incorperate the bias correction terms and non-zero (cid:15) into our analysis above.

Original: Based on the current convergence proof, we only requires several additional simple changes.
Revised: Based on the current convergence proof, we only requires several additional simple changes.

Original: We list them as follows.
Revised: We list them as follows.

Original: Adam with bias correction terms: This bias correction terms are introduced by (Kingma & Ba, 2014)).
Revised: Adam with bias correction terms: This bias correction terms are introduced by (Kingma & Ba, 2014)).

Original: It can be implemented by (1) changing the stepsize η k into ˆ η k = √ 1 − β k1 − β k 1 η k = √ 1 − β k1 − β k 1 η 0 √ k ; (2) change the initialization of Algorithm 1 into m 1 , − 1 = v 1 , − 1 = 0 .
Revised: It can be implemented by (1) changing the stepsize η k into ˆ η k = √ 1 − β k1 − β k 1 η k = √ 1 − β k1 − β k 1 η 0 √ k ; (2) change the initialization of Algorithm 1 into m 1 , − 1 = v 1 , − 1 = 0 .

Original: More details can be seen in (Kingma & Ba, 2014)).
Revised: More details can be seen in (Kingma & Ba, 2014)).

Original: We now explain how to inlcude this two changes into our analysis.
Revised: We now explain how to inlcude this two changes into our analysis.

Original: (1) Change of stepsize: We observe that the new stepsize ˆ η k is well bounded around the old stepsize η k , i.e., ˆ η k ∈ [ √ 1 − β 2 η k , 1 1 − β 1 η k ] .
Revised: (1) Change of stepsize: We observe that the new stepsize ˆ η k is well bounded around the old stepsize η k , i.e., ˆ η k ∈ [ √ 1 − β 2 η k , 1 1 − β 1 η k ] .

Original: Therefore, to prove the convergence of Adam with ˆ η k , we add the follwing steps into the current proof.
Revised: Therefore, to prove the convergence of Adam with ˆ η k , we add the follwing steps into the current proof.

Original: • Whenever we need an upper bound on ˆ η k , we use ˆ η k ≤ 1 1 − β 1 η k .
Revised: • Whenever we need an upper bound on ˆ η k , we use ˆ η k ≤ 1 1 − β 1 η k .

Original: Then we follow the original analysis with an extra constant 1 1 − β 1 .
Revised: Then we follow the original analysis with an extra constant 1 1 − β 1 .

Original: This step will appear in Lemma F.2.
Revised: This step will appear in Lemma F.2.

Original: It turns out we only need to change the constant (cid:52) nk in Lemma F.2 into
Revised: It turns out we only need to change the constant (cid:52) nk in Lemma F.2 into

Original: [Table]
Revised: [Table]

Original: The rest of the analysis remains the same.
Revised: The rest of the analysis remains the same.

Original: Whenver we need a lower bound on ˆ η k , we use ˆ η k ≥ √ 1 − β 2 η k .
Revised: Whenver we need a lower bound on ˆ η k , we use ˆ η k ≥ √ 1 − β 2 η k .

Original: Then we follow the original analysis with an extra constant √ 1 − β 2 .
Revised: Then we follow the original analysis with an extra constant √ 1 − β 2 .

Original: This step will appear in Lemma G.9 and we only need to change the constant terms in the ﬁnal result.
Revised: This step will appear in Lemma G.9 and we only need to change the constant terms in the ﬁnal result.

Original: The rest of the analysis remains the same.
Revised: The rest of the analysis remains the same.

Original: (2) Change of initialization: In our current analysis, we use initialization m 1 , − 1 = ∇ f ( x 0 ) and v 1 , − 1 = max i ∇ f i ( x 0 ) ◦
Revised: (2) Change of initialization: In our current analysis, we use initialization m 1 , − 1 = ∇ f ( x 0 ) and v 1 , − 1 = max i ∇ f i ( x 0 ) ◦

Original: ∇ f i ( x 0 ) .
Revised: ∇ f i ( x 0 ) .

Original: Now we explain how to prove convergence with initialization m 1 , − 1 = v 1 , − 1 = 0 .
Revised: Now we explain how to prove convergence with initialization m 1 , − 1 = v 1 , − 1 = 0 .

Original: • We use m 1 , − 1 = ∇ f ( x 0 ) at Lemma G.11, in which we bound the difference between M l,k and M (cid:48) l,k .
Revised: • We use m 1 , − 1 = ∇ f ( x 0 ) at Lemma G.11, in which we bound the difference between M l,k and M (cid:48) l,k .

Original: The goal of this lemma is to control M l,k − M (cid:48) l,k
Revised: The goal of this lemma is to control M l,k − M (cid:48) l,k

Original: = O ( β ( k − 1) n) .
Revised: = O ( β ( k − 1) n) .

Original: As explained in the proof of Lemma G.11, after expanding M l,k and M (cid:48) l,k into serieses, they
Revised: As explained in the proof of Lemma G.11, after expanding M l,k and M (cid:48) l,k into serieses, they

Original: only differ when “ k ≤ 0 ” (as shown in equation (48)).
Revised: only differ when “ k ≤ 0 ” (as shown in equation (48)).

Original: When we use m 1 , − 1 = 0 , half of the terms in equation (48) will become 0.
Revised: When we use m 1 , − 1 = 0 , half of the terms in equation (48) will become 0.

Original: However, it does not affect the result of Lemma G.11 since all the terms in equation (48) are (at least) weighted by β ( k − 1) n.
Revised: However, it does not affect the result of Lemma G.11 since all the terms in equation (48) are (at least) weighted by β ( k − 1) n.

Original: So even if half of them to be 0, the rest of the terms is still in the order of β ( k − 1)
Revised: So even if half of them to be 0, the rest of the terms is still in the order of β ( k − 1)

Original: n.
Revised: n.

Original: Therefore, Lemma G.11 still holds with a few changes on the constant terms.
Revised: Therefore, Lemma G.11 still holds with a few changes on the constant terms.

Original: • We use v 1 , − 1 = max i ∇
Revised: • We use v 1 , − 1 = max i ∇

Original: f
Revised: f

Original: i ( x 0 ) ◦ ∇ f i ( x 0 ) at Lemma G.1,which is mainly based on Lemma F.1 in (Shi et al., 2020).
Revised: i ( x 0 ) ◦ ∇ f i ( x 0 ) at Lemma G.1,which is mainly based on Lemma F.1 in (Shi et al., 2020).

Original: According to (Shi et al., 2020): to include bias correction terms into analysis, we just need to add one more constraint k > 8 √ 2 1 − β n 2 + 

Revised: According to (Shi et al., 2020): to include bias correction terms into analysis, we just need to add one more constraint k > 8 √ 2 1 − β n 2 +

Original: and then we can reach the same conclusion.
Revised: and then we can reach the same conclusion.

Original: Adam with non-zero (cid:15) :
Revised: Adam with non-zero (cid:15) :

Original: In our current analysis, we consider (cid:15) = 0 .
Revised: In our current analysis, we consider (cid:15) = 0 .

Original: In practice, (cid:15) is often set to be a small postive number such as 10 − 8 .
Revised: In practice, (cid:15) is often set to be a small postive number such as 10 − 8 .

Original: Proving convergence with (cid:15) > 0 is strictly simpler.
Revised: Proving convergence with (cid:15) > 0 is strictly simpler.

Original: It only requires a few simple changes based on the current proof.
Revised: It only requires a few simple changes based on the current proof.

Original: We explain as below.
Revised: We explain as below.

Original: When (cid:15) (cid:54) = 0 , the new 2nd-order momentum becomes ˆ v k, 0 := √ v k, 0 + (cid:15) .
Revised: When (cid:15) (cid:54) = 0 , the new 2nd-order momentum becomes ˆ v k, 0 := √ v k, 0 + (cid:15) .

Original: This brings the following changes:
Revised: This brings the following changes:

Original: • Whenever we want an upper bound on ˆ v k, 0 , we can choose one of the following upper bound.
Revised: • Whenever we want an upper bound on ˆ v k, 0 , we can choose one of the following upper bound.

Original: – When √ v k, 0 ≥ (cid:15) , we have ˆ v k, 0 ≤ 2 √ v k, 0 .
Revised: – When √ v k, 0 ≥ (cid:15) , we have ˆ v k, 0 ≤ 2 √ v k, 0 .

Original: Then, we follow the same steps in the current proof with minor changes on the constant.
Revised: Then, we follow the same steps in the current proof with minor changes on the constant.

Original: – When √ v k, 0 < (cid:15) , we have ˆ v k, 0 ≤ 2 (cid:15) .
Revised: – When √ v k, 0 < (cid:15) , we have ˆ v k, 0 ≤ 2 (cid:15) .

Original: This step, again, decouple the statistical dependency between ∇ f τk, 0 ( x ) and √ v k, 0 .
Revised: This step, again, decouple the statistical dependency between ∇ f τk, 0 ( x ) and √ v k, 0 .

Original: It changes Adam into SGD and thus simpliﬁes the proof.
Revised: It changes Adam into SGD and thus simpliﬁes the proof.

Original: Many technical lemmas could be skipped in this case.
Revised: Many technical lemmas could be skipped in this case.

Original: • Whenever we want a lower bound on ˆ v k, 0 , we have ˆ v k, 0 ≥ (cid:15) .
Revised: • Whenever we want a lower bound on ˆ v k, 0 , we have ˆ v k, 0 ≥ (cid:15) .

Original: This step decouples the statistical dependency between ∇ f τk, 0 ( x ) and √ v k, 0 .
Revised: This step decouples the statistical dependency between ∇ f τk, 0 ( x ) and √ v k, 0 .

Original: It changes Adam into SGD and thus simpliﬁes the proof.
Revised: It changes Adam into SGD and thus simpliﬁes the proof.

Original: Many technical lemmas could be skipped in this case.
Revised: Many technical lemmas could be skipped in this case.