Original: Adam Can Converge Without Any Modiﬁcation On Update Rules
Revised: Adam Can Converge Without Any Modiﬁcation On Update Rules

Original: Abstract
Revised: Abstract

Original: Ever since Reddi et al.
Revised: Ever since Reddi et al.

Original: (2018) pointed out the divergence issue of Adam, many new variants have been designed to obtain convergence.
Revised: (2018) pointed out the divergence issue of Adam, many new variants have been designed to obtain convergence.

Original: However, vanilla Adam remains exceptionally popular and it works well in practice.
Revised: However, vanilla Adam remains exceptionally popular and it works well in practice.

Original: Why is there a gap between theory and practice?
Revised: Why is there a gap between theory and practice?

Original: We point out there is a mismatch between the settings of theory and practice: Reddi et al.
Revised: We point out there is a mismatch between the settings of theory and practice: Reddi et al.

Original: (2018) pick the problem after picking the hyperparameters of Adam, i.e., ( β 1 , β 2 ) ; while practical applications often ﬁx the problem ﬁrst and then tune ( β 1 , β 2 ) .
Revised: (2018) pick the problem after picking the hyperparameters of Adam, i.e., ( β 1 , β 2 ) ; while practical applications often ﬁx the problem ﬁrst and then tune ( β 1 , β 2 ) .

Original: Due to this observation, we conjecture that the empirical convergence can be theoretically justiﬁed, only if we change the order of picking the problem and hyperparameter.
Revised: Due to this observation, we conjecture that the empirical convergence can be theoretically justiﬁed, only if we change the order of picking the problem and hyperparameter.

Original: In this work, we conﬁrm this conjecture.
Revised: In this work, we conﬁrm this conjecture.

Original: We prove that, when the 2nd-order momentum parameter β 2 is large and 1st-order momentum parameter β 1 < √ β 2 < 1 , Adam converges to the neighborhood of critical points.
Revised: We prove that, when the 2nd-order momentum parameter β 2 is large and 1st-order momentum parameter β 1 < √ β 2 < 1 , Adam converges to the neighborhood of critical points.

Original: The size of the neighborhood is propositional to the variance of stochastic gradients.
Revised: The size of the neighborhood is propositional to the variance of stochastic gradients.

Original: Under an extra condition (strong growth condition), Adam converges to critical points.
Revised: Under an extra condition (strong growth condition), Adam converges to critical points.

Original: As β 2 increases, our convergence result can cover any β 1 ∈ [0 , 1) including β 1 = 0 . 9 , which is the default setting in deep learning libraries.
Revised: As β 2 increases, our convergence result can cover any β 1 ∈ [0 , 1) including β 1 = 0 . 9 , which is the default setting in deep learning libraries.

Original: To our knowledge, this is the ﬁrst result showing that Adam can converge under a wide range of hyperparameters without any modiﬁcation on its update rules.
Revised: To our knowledge, this is the ﬁrst result showing that Adam can converge under a wide range of hyperparameters without any modiﬁcation on its update rules.

Original: Further, our analysis does not require assumptions of bounded gradients or bounded 2nd-order momentum.
Revised: Further, our analysis does not require assumptions of bounded gradients or bounded 2nd-order momentum.

Original: When β 2 is small, we further point out a large region of
Revised: When β 2 is small, we further point out a large region of

Original: ( β 1 , β 2 ) combinations where Adam can diverge to inﬁnity.
Revised: ( β 1 , β 2 ) combinations where Adam can diverge to inﬁnity.

Original: Our divergence result considers the same setting (ﬁxing the optimization problem ahead) as our convergence result, indicating that there is a phase transition from divergence to convergence when increasing β 2 .
Revised: Our divergence result considers the same setting (ﬁxing the optimization problem ahead) as our convergence result, indicating that there is a phase transition from divergence to convergence when increasing β 2 .

Original: These positive and negative results provide suggestions on how to tune Adam hyperparameters: for instance, when Adam does not work well, we suggest tuning up β 2 and trying β 1 < √ β 2 .
Revised: These positive and negative results provide suggestions on how to tune Adam hyperparameters: for instance, when Adam does not work well, we suggest tuning up β 2 and trying β 1 < √ β 2 .

Original:  Introduction
Revised:  Introduction

Original: Modern machine learning tasks often aim to solve the following ﬁnite-sum problem.
Revised: Modern machine learning tasks often aim to solve the following ﬁnite-sum problem.

Original: [Equation]
Revised: [Equation]

Original: where n is the number of samples or mini-batches and x denotes the trainable parameters.
Revised: where n is the number of samples or mini-batches and x denotes the trainable parameters.

Original: In deep learning, Adam (Kingma & Ba, 2014) is one of the most popular algorithms for solving (1).
Revised: In deep learning, Adam (Kingma & Ba, 2014) is one of the most popular algorithms for solving (1).

Original: It has been applied to various machine learning domains such as natural language processing (NLP) (Vaswani et al., 2017; Brown et al., 2020; Devlin et al., 2018), generative adversarial networks (GANs) (Radford et al., 2015; Isola et al., 2017; Zhu et al., 2017) and computer vision (CV) (Dosovitskiy et al., 2021).
Revised: It has been applied to various machine learning domains such as natural language processing (NLP) (Vaswani et al., 2017; Brown et al., 2020; Devlin et al., 2018), generative adversarial networks (GANs) (Radford et al., 2015; Isola et al., 2017; Zhu et al., 2017) and computer vision (CV) (Dosovitskiy et al., 2021).

Original: Despite its prevalence, Reddi et al.
Revised: Despite its prevalence, Reddi et al.

Original: (2018) point out that Adam can diverge with a wide range of hyperparameters.
Revised: (2018) point out that Adam can diverge with a wide range of hyperparameters.

Original: A main result in (Reddi et al., 2018) states that 2 :
Revised: A main result in (Reddi et al., 2018) states that 2 :

Original: For any β 1 , β 2 s.t. 

Revised: For any β 1 , β 2 s.t.

Original: ≤ β 1 < √ β 2 < 1 , there exists a problem such that Adam diverges.
Revised: ≤ β 1 < √ β 2 < 1 , there exists a problem such that Adam diverges.

Original: Here, β 1 and β 2 are the hyperparameter to control Adam’s 1st-order and 2nd-order momentum.
Revised: Here, β 1 and β 2 are the hyperparameter to control Adam’s 1st-order and 2nd-order momentum.

Original: More description of Adam can be seen in Algorithm 1 (presented later in Section 2.1).
Revised: More description of Adam can be seen in Algorithm 1 (presented later in Section 2.1).

Original: Ever since (Reddi et al., 2018) pointed out the divergence issue, many new variants have been designed.
Revised: Ever since (Reddi et al., 2018) pointed out the divergence issue, many new variants have been designed.

Original: For instance, AMSGrad (Reddi et al., 2018) enforced the adaptor v t (deﬁned later in Algorithm 1) to be non-decreasing; AdaBound (Luo et al., 2019) imposed constraint v t ∈ [ C l , C u ] to ensure the boundedness on effective stepsize.
Revised: For instance, AMSGrad (Reddi et al., 2018) enforced the adaptor v t (deﬁned later in Algorithm 1) to be non-decreasing; AdaBound (Luo et al., 2019) imposed constraint v t ∈ [ C l , C u ] to ensure the boundedness on effective stepsize.

Original: We introduce more variants in Appendix D.1.
Revised: We introduce more variants in Appendix D.1.

Original: On the other hand, counter-intuitively, vanilla Adam remains exceptionally popular (see evidence at (Scholar)).
Revised: On the other hand, counter-intuitively, vanilla Adam remains exceptionally popular (see evidence at (Scholar)).

Original: Without any modiﬁcation on its update rules, Adam works well in practice.
Revised: Without any modiﬁcation on its update rules, Adam works well in practice.

Original: Even more mysteriously, we ﬁnd that the commonly reported hyperparameters actually satisfy the divergence condition stated earlier.
Revised: Even more mysteriously, we ﬁnd that the commonly reported hyperparameters actually satisfy the divergence condition stated earlier.

Original: For instance, Kingma & Ba (2014) claimed that ( β 1 , β 2 ) =
Revised: For instance, Kingma & Ba (2014) claimed that ( β 1 , β 2 ) =

Original: (0 . 9 , 0 . 999) is a “good choice for the tested machine learning problems" and it is indeed the default setting in deep learning libraries.
Revised: (0 . 9 , 0 . 999) is a “good choice for the tested machine learning problems" and it is indeed the default setting in deep learning libraries.

Original: In super-large models GPT-3 and Megatron (Brown et al., 2020; Smith et al., 2022), ( β 1 , β 2 ) is chosen to be (0 . 9 , 0 . 95) .
Revised: In super-large models GPT-3 and Megatron (Brown et al., 2020; Smith et al., 2022), ( β 1 , β 2 ) is chosen to be (0 . 9 , 0 . 95) .

Original: GAN researchers (e.g. Radford et al.
Revised: GAN researchers (e.g. Radford et al.

Original: ; Isola et al.
Revised: ; Isola et al.

Original: (2017)) use ( β 1 , β 2 ) =
Revised: (2017)) use ( β 1 , β 2 ) =

Original: All these hyperparameters live in the divergence region β 1 < √ β 2 .
Revised: All these hyperparameters live in the divergence region β 1 < √ β 2 .

Original: Surprisingly, instead of observing the divergence issue, these hyperparameters achieve good performances and they actually show the sign of convergence.
Revised: Surprisingly, instead of observing the divergence issue, these hyperparameters achieve good performances and they actually show the sign of convergence.

Original: Why does Adam work well despite its theoretical divergence issue?
Revised: Why does Adam work well despite its theoretical divergence issue?

Original: Is there any mismatch between deep learning problems and the divergent example?
Revised: Is there any mismatch between deep learning problems and the divergent example?

Original: We take a closer look into the divergence example and ﬁnd out the mismatch does exist.
Revised: We take a closer look into the divergence example and ﬁnd out the mismatch does exist.

Original: In particular, we notice an important (but often ignored) characteristic of the divergence example: (Reddi et al., 2018) picks ( β 1 , β 2 ) before picking the sample size n .
Revised: In particular, we notice an important (but often ignored) characteristic of the divergence example: (Reddi et al., 2018) picks ( β 1 , β 2 ) before picking the sample size n .

Original: Put in another way, to construct the divergence example, they change n for different ( β 1 , β 2 ) .
Revised: Put in another way, to construct the divergence example, they change n for different ( β 1 , β 2 ) .

Original: For instance, for ( β 1 , β 2 ) =
Revised: For instance, for ( β 1 , β 2 ) =

Original: (0 , 0 . 99) , they use one n to construct the divergent example; for ( β 1 , β 2 )
Revised: (0 , 0 . 99) , they use one n to construct the divergent example; for ( β 1 , β 2 )

Original: = (0 , 0 . 9999) , they use another n to construct another divergent example.
Revised: = (0 , 0 . 9999) , they use another n to construct another divergent example.

Original: On the other hand, in practical applications of Adam listed above, practitioners tune the hyperparameters ( β 1 , β 2 ) after the sample size n is ﬁxed.
Revised: On the other hand, in practical applications of Adam listed above, practitioners tune the hyperparameters ( β 1 , β 2 ) after the sample size n is ﬁxed.

Original: So there is a gap between the setting of theory and practice: the order of picking n and ( β 1 , β 2 ) is different.
Revised: So there is a gap between the setting of theory and practice: the order of picking n and ( β 1 , β 2 ) is different.

Original: Considering the good performance of Adam under ﬁxed n , we conjecture that Adam can converge in this setting.
Revised: Considering the good performance of Adam under ﬁxed n , we conjecture that Adam can converge in this setting.

Original: Unfortunately, the behavior of vanilla Adam is far less studied than its variants (perhaps due to the criticism of divergence).
Revised: Unfortunately, the behavior of vanilla Adam is far less studied than its variants (perhaps due to the criticism of divergence).

Original: To verify this conjecture, we run experiments for different choices of ( β 1 , β 2 ) on a few tasks.
Revised: To verify this conjecture, we run experiments for different choices of ( β 1 , β 2 ) on a few tasks.

Original: First, we run Adam for a convex function (2) with ﬁxed n (see the deﬁnition in Section 3.2).
Revised: First, we run Adam for a convex function (2) with ﬁxed n (see the deﬁnition in Section 3.2).

Original: Second, we run Adam for the classiﬁcation problem on data MNIST and
Revised: Second, we run Adam for the classiﬁcation problem on data MNIST and

Original: CIFAR-10 with ﬁxed batchsize.
Revised: CIFAR-10 with ﬁxed batchsize.

Original: We observe some interesting phenomena in Figure 1 (a), (b) and (c).
Revised: We observe some interesting phenomena in Figure 1 (a), (b) and (c).

Original: First, when β 2 is large, the optimization error is small for almost all values of β 1 .
Revised: First, when β 2 is large, the optimization error is small for almost all values of β 1 .

Original: Second, when β 1 , β 2 are both small, there is a red region with relatively large error.
Revised: Second, when β 1 , β 2 are both small, there is a red region with relatively large error.

Original: On MNIST, CIFAR-10, the error in the red region is increased by 1.4 times than that in the blue region.
Revised: On MNIST, CIFAR-10, the error in the red region is increased by 1.4 times than that in the blue region.

Original: The situation is a lot worse on function (2) (deﬁned later in Section 3.2): the error in the red region is 70 times higher.
Revised: The situation is a lot worse on function (2) (deﬁned later in Section 3.2): the error in the red region is 70 times higher.

Original: [Table]
Revised: [Table]

Original: (a) Function (2) 1 

Revised: (a) Function (2) 1

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Figure]
Revised: [Figure]

Original: While Adam’s performances seem unstable in the red region, we ﬁnd that Adam always performs well in the top blue region in Figure 1.
Revised: While Adam’s performances seem unstable in the red region, we ﬁnd that Adam always performs well in the top blue region in Figure 1.

Original: This seems to suggest that Adam can converge without any algorithmic modiﬁcation as long as β 1 and β 2 are chosen properly.
Revised: This seems to suggest that Adam can converge without any algorithmic modiﬁcation as long as β 1 and β 2 are chosen properly.

Original: We ask the following question:
Revised: We ask the following question:

Original: Can Adam provably converge without any modiﬁcation on its update rules?
Revised: Can Adam provably converge without any modiﬁcation on its update rules?

Original: In this work we theoretically explore this question.
Revised: In this work we theoretically explore this question.

Original: Our contributions are visualized in Figure 1 (d).
Revised: Our contributions are visualized in Figure 1 (d).

Original: We prove the following results when n is ﬁxed (or more rigorously, when the function class is ﬁxed):
Revised: We prove the following results when n is ﬁxed (or more rigorously, when the function class is ﬁxed):

Original: • We prove that when β 2 is large enough and β 1 < √ β 2 , Adam converges to the neighborhood of critical points.
Revised: • We prove that when β 2 is large enough and β 1 < √ β 2 , Adam converges to the neighborhood of critical points.

Original: The size of the neighborhood is propositional to the variance of stochastic gradients.
Revised: The size of the neighborhood is propositional to the variance of stochastic gradients.

Original: With an extra condition (so-called strong growth condition), we prove that Adam can converge to critical points.
Revised: With an extra condition (so-called strong growth condition), we prove that Adam can converge to critical points.

Original: As β 2 increases, these results can cover any momentum parameter β 1 ∈ [0 , 1) including the default setting β 1 = 0 .
Revised: As β 2 increases, these results can cover any momentum parameter β 1 ∈ [0 , 1) including the default setting β 1 = 0 .

Original: In particular, our analysis does not require bounded gradient assumption.
Revised: In particular, our analysis does not require bounded gradient assumption.

Original: • We study the divergence issue of small- β 2 Adam.
Revised: • We study the divergence issue of small- β 2 Adam.

Original: We prove that: for any ﬁxed n (or more rigorously, for any ﬁxed function class), there exists a function such that, Adam diverges to inﬁnity when ( β 1 , β 2 ) is picked in the red region in Figure 1 (d).
Revised: We prove that: for any ﬁxed n (or more rigorously, for any ﬁxed function class), there exists a function such that, Adam diverges to inﬁnity when ( β 1 , β 2 ) is picked in the red region in Figure 1 (d).

Original: The size of the red region increases with n .
Revised: The size of the red region increases with n .

Original: The shape of the region follows the solution to our analytic conditions.
Revised: The shape of the region follows the solution to our analytic conditions.

Original: • We emphasize a few characteristics of our results.
Revised: • We emphasize a few characteristics of our results.

Original: (1) phase transition .
Revised: (1) phase transition .

Original: The divergence result considers the same setting as our convergence result, indicating that there is a phase transition from divergence to convergence when changing β 2 .
Revised: The divergence result considers the same setting as our convergence result, indicating that there is a phase transition from divergence to convergence when changing β 2 .

Original: (2) problem-dependent bounds .
Revised: (2) problem-dependent bounds .

Original: Our convergence and divergence regions of ( β 1 , β 2 ) are problem-dependent, which is drastically different from (Reddi et al., 2018) which established the problem-independent worst-case choice of ( β 1 , β 2 ) .
Revised: Our convergence and divergence regions of ( β 1 , β 2 ) are problem-dependent, which is drastically different from (Reddi et al., 2018) which established the problem-independent worst-case choice of ( β 1 , β 2 ) .

Original: (3) non-asymptotic characterization .
Revised: (3) non-asymptotic characterization .

Original: the “divergence region” of ( β 1 , β 2 ) expands as n increases and converges to the whole region [0 , 1) 2 as n goes to inﬁnity, which recovers (actually stronger than) the problem-independent divergence result of (Reddi et al., 2018) that requires β 1 < √ β 2 .
Revised: the “divergence region” of ( β 1 , β 2 ) expands as n increases and converges to the whole region [0 , 1) 2 as n goes to inﬁnity, which recovers (actually stronger than) the problem-independent divergence result of (Reddi et al., 2018) that requires β 1 < √ β 2 .

Original: In this sense, we can view the divergence result of (Reddi et al., 2018) as an asymptotic characterization of the divergence region (as n → ∞ ) and our divergence result as a non-asymptotic characterization (for any ﬁxed n ).
Revised: In this sense, we can view the divergence result of (Reddi et al., 2018) as an asymptotic characterization of the divergence region (as n → ∞ ) and our divergence result as a non-asymptotic characterization (for any ﬁxed n ).

Original: We provide more discussion in Section 4.
Revised: We provide more discussion in Section 4.

Original: • Our positive and negative results can provide suggestions for tuning β 1 and β 2 : for instance,when Adam does not work well, we suggest tuning up β 2 and trying β 1 < √ β 2 .
Revised: • Our positive and negative results can provide suggestions for tuning β 1 and β 2 : for instance,when Adam does not work well, we suggest tuning up β 2 and trying β 1 < √ β 2 .

Original: We provide more tuning suggestions in Appendix C.
Revised: We provide more tuning suggestions in Appendix C.

Original: We believe our results can boost new understandings for Adam.
Revised: We believe our results can boost new understandings for Adam.

Original: While Reddi et al.
Revised: While Reddi et al.

Original: (2018) reveal that “Adam can diverge", our results show the other side of the coin: when n is ﬁxed (or when function class is ﬁxed), Adam can still converge without any modiﬁcation on its update rules.
Revised: (2018) reveal that “Adam can diverge", our results show the other side of the coin: when n is ﬁxed (or when function class is ﬁxed), Adam can still converge without any modiﬁcation on its update rules.

Original: Our results suggest that Adam is still a theoretically justiﬁed algorithm and practitioners can use it conﬁdently.
Revised: Our results suggest that Adam is still a theoretically justiﬁed algorithm and practitioners can use it conﬁdently.

Original: We further emphasize that our convergence results can cover any β 1 ∈ [0 , 1) , which allows the algorithm to bring arbitrarily heavy momentum signals.
Revised: We further emphasize that our convergence results can cover any β 1 ∈ [0 , 1) , which allows the algorithm to bring arbitrarily heavy momentum signals.

Original: It turns out that large-momentum Adam is not easy to analyze.
Revised: It turns out that large-momentum Adam is not easy to analyze.

Original: Even with stronger assumptions like bounded gradient ( (cid:107)∇ f ( x ) (cid:107) <
Revised: Even with stronger assumptions like bounded gradient ( (cid:107)∇ f ( x ) (cid:107) <

Original: C , ∀ x ), its convergence is not well understood (see related works in Section 2.2).
Revised: C , ∀ x ), its convergence is not well understood (see related works in Section 2.2).

Original: To our best knowledge, this is the ﬁrst result that proves vanilla Adam with any β 1 can converge without any assumption of bounded gradient or bounded 2nd-order momentum.
Revised: To our best knowledge, this is the ﬁrst result that proves vanilla Adam with any β 1 can converge without any assumption of bounded gradient or bounded 2nd-order momentum.

Original: The proof contains a new method to handle unbounded momentum in the stochastic non-linear dynamics system.
Revised: The proof contains a new method to handle unbounded momentum in the stochastic non-linear dynamics system.

Original: We will highlight our technical novelties in
Revised: We will highlight our technical novelties in

Original: Section 5.
Revised: Section 5.

Original:  Preliminaries 2.1 Review of Adam
Revised:  Preliminaries 2.1 Review of Adam

Original: We consider ﬁnite-sum problem (1).
Revised: We consider ﬁnite-sum problem (1).

Original: We use x to denote the optimization variable.
Revised: We use x to denote the optimization variable.

Original: We denote ∇ f j as the gradient of f j and let ◦ be the component-wise product.
Revised: We denote ∇ f j as the gradient of f j and let ◦ be the component-wise product.

Original: The division and square-root operator are component-wise as well.
Revised: The division and square-root operator are component-wise as well.

Original: We present randomly shufﬂed Adam in Algorithm 1.
Revised: We present randomly shufﬂed Adam in Algorithm 1.

Original: In Algorithm 1, m denotes the 1st-order momentum and v denotes the 2nd-order momentum.
Revised: In Algorithm 1, m denotes the 1st-order momentum and v denotes the 2nd-order momentum.

Original: they are weighted averaged by hyperparameter β 1 , β 2 , respectively.
Revised: they are weighted averaged by hyperparameter β 1 , β 2 , respectively.

Original: Larger β 1 , β 2 will adopt more history information.
Revised: Larger β 1 , β 2 will adopt more history information.

Original: We denote x k,i , m k,i , v k,i ∈
Revised: We denote x k,i , m k,i , v k,i ∈

Original: R d as the value of x, m, v at the k -th outer loop (epoch) and i -th inner loop (batch), respectively.
Revised: R d as the value of x, m, v at the k -th outer loop (epoch) and i -th inner loop (batch), respectively.

Original: We choose η k = η 1 √ nk as the stepsize.
Revised: We choose η k = η 1 √ nk as the stepsize.

Original: In practice, (cid:15) is adopted for numerical stability and it is often chosen to be 10 − 8 .
Revised: In practice, (cid:15) is adopted for numerical stability and it is often chosen to be 10 − 8 .

Original: In our theory, we allow (cid:15) to be an arbitrary non-negative constant including 0 .
Revised: In our theory, we allow (cid:15) to be an arbitrary non-negative constant including 0 .

Original: In the original version of Adam in (Kingma & Ba, 2014), it has an additional “bias correction” step.
Revised: In the original version of Adam in (Kingma & Ba, 2014), it has an additional “bias correction” step.

Original: This “bias correction” step can be implemented by changing the stepsize η k into ˆ η k = √ 1 −
Revised: This “bias correction” step can be implemented by changing the stepsize η k into ˆ η k = √ 1 −

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: η k
Revised: η k

Original: Algorithm
Revised: Algorithm

Original: Initialize x 1 , 0 = x 0 , m 1 , − 1 = ∇ f ( x 0 ) and v 1 , − 1 = max i ∇ f i ( x 0 ) ◦ ∇ f i ( x 0 ) .
Revised: Initialize x 1 , 0 = x 0 , m 1 , − 1 = ∇ f ( x 0 ) and v 1 , − 1 = max i ∇ f i ( x 0 ) ◦ ∇ f i ( x 0 ) .

Original: for k = 1 → ∞ do
Revised: for k = 1 → ∞ do

Original: Sample { τ k, 0 , τ k, 1 , · · · , τ k,n − 1 } as a random permutation of { 0 , 1 , 2 , · · · , n − 1 } for i = 

Revised: Sample { τ k, 0 , τ k, 1 , · · · , τ k,n − 1 } as a random permutation of { 0 , 1 , 2 , · · · , n − 1 } for i =

Original: → n − 1 do m k,i =
Revised: → n − 1 do m k,i =

Original: β 1 m k,i − 1 + (1 − β 1 ) ∇ f τ k,i ( x k,i ) v k,i =
Revised: β 1 m k,i − 1 + (1 − β 1 ) ∇ f τ k,i ( x k,i ) v k,i =

Original: β 2 v k,i − 1 + (1 − β 2 ) ∇ f τ k,i ( x k,i ) ◦ ∇ f τ k,i ( x k,i ) x
Revised: β 2 v k,i − 1 + (1 − β 2 ) ∇ f τ k,i ( x k,i ) ◦ ∇ f τ k,i ( x k,i ) x

Original: k,i +1 =
Revised: k,i +1 =

Original: x k,i − η k √ v k,i + (cid:15) ◦ m k
Revised: x k,i − η k √ v k,i + (cid:15) ◦ m k

Original: i end for x k +1 , 

Revised: i end for x k +1 ,

Original: = x k,n ; v k +1 , − 1 = v k,n − 1 ; m k +1 , − 1 = m k,n − 1 end for and using zero initialization.
Revised: = x k,n ; v k +1 , − 1 = v k,n − 1 ; m k +1 , − 1 = m k,n − 1 end for and using zero initialization.

Original: In Algorithm 1, the “bias correction” step is replaced by a special initialization, which also corrects the bias.
Revised: In Algorithm 1, the “bias correction” step is replaced by a special initialization, which also corrects the bias.

Original: Note that ˆ η k ∈ [ √ 1 − β 2 η k , 1 1 − β 1 η k ] is well-bounded near η k , so η k and ˆ η k brings the same convergence rate.
Revised: Note that ˆ η k ∈ [ √ 1 − β 2 η k , 1 1 − β 1 η k ] is well-bounded near η k , so η k and ˆ η k brings the same convergence rate.

Original: In addition, as the effect of initialization becomes negligible when the training progresses, Adam with zero & our initialization will have the same asymptotic behavior.
Revised: In addition, as the effect of initialization becomes negligible when the training progresses, Adam with zero & our initialization will have the same asymptotic behavior.

Original: We follow the form of Algorithm 1, which makes results cleaner.
Revised: We follow the form of Algorithm 1, which makes results cleaner.

Original: In our analysis, we make the assumptions below.
Revised: In our analysis, we make the assumptions below.

Original: Assumption 2.1.
Revised: Assumption 2.1.

Original: we consider x ∈ R d and f i ( x ) satisﬁes gradient Lipschitz continuous with constant
Revised: we consider x ∈ R d and f i ( x ) satisﬁes gradient Lipschitz continuous with constant

Original: L .
Revised: L .

Original: We assume f ( x ) is lower bounded by a ﬁnite constant f ∗ .
Revised: We assume f ( x ) is lower bounded by a ﬁnite constant f ∗ .

Original: Assumption 2.2.
Revised: Assumption 2.2.

Original: f i ( x ) and f ( x ) satisfy: (cid:80) n − 1 i =

Revised: f i ( x ) and f ( x ) satisfy: (cid:80) n − 1 i =

Original: (cid:107)∇ f i ( x ) (cid:107) 22 ≤ D 1 (cid:107)∇ f
Revised: (cid:107)∇ f i ( x ) (cid:107) 22 ≤ D 1 (cid:107)∇ f

Original: [Equation]
Revised: [Equation]

Original: x )
Revised: x )

Original: [Equation]
Revised: [Equation]

Original: 22 + D 0 , ∀ x ∈ R d .
Revised: 22 + D 0 , ∀ x ∈ R d .

Original: Assumption 2.2 is quite general.
Revised: Assumption 2.2 is quite general.

Original: When D 1 = 1 /n , it becomes the “constant variance” with constant
Revised: When D 1 = 1 /n , it becomes the “constant variance” with constant

Original: D 0 /n .
Revised: D 0 /n .

Original: “constant variance" condition is commonly used in both SGD and Adam analysis (e.g.
Revised: “constant variance" condition is commonly used in both SGD and Adam analysis (e.g.

Original: (Ghadimi et al., 2016; Zaheer et al., 2018; Huang et al., 2021)).
Revised: (Ghadimi et al., 2016; Zaheer et al., 2018; Huang et al., 2021)).

Original: Assumption 2.2 allows more ﬂexible choices of D 1 (cid:54) = n and thus it is weaker than “constant variance”.
Revised: Assumption 2.2 allows more ﬂexible choices of D 1 (cid:54) = n and thus it is weaker than “constant variance”.

Original: When D 0 > 0 , the problem instance is sometimes called “non-realizable" (Shi et al., 2020).
Revised: When D 0 > 0 , the problem instance is sometimes called “non-realizable" (Shi et al., 2020).

Original: In this case, adaptive gradient methods are not guaranteed to reach the exact critical points.
Revised: In this case, adaptive gradient methods are not guaranteed to reach the exact critical points.

Original: Instead, they only converge to a bounded region (near critical points)
Revised: Instead, they only converge to a bounded region (near critical points)

Original: (Zaheer et al., 2018; Shi et al., 2020).
Revised: (Zaheer et al., 2018; Shi et al., 2020).

Original: This phenomenon indeed occurs for Adam in experiments, even with diminishing stepsize (see Figure(a)).
Revised: This phenomenon indeed occurs for Adam in experiments, even with diminishing stepsize (see Figure(a)).

Original: The behavior of SGD is similar: constant stepsize SGD converges to a bounded region with its size propositional to the noise level D 

Revised: The behavior of SGD is similar: constant stepsize SGD converges to a bounded region with its size propositional to the noise level D

Original: (Yan et al., 2018; Yu et al., 2019; Liu et al., 2020b).
Revised: (Yan et al., 2018; Yu et al., 2019; Liu et al., 2020b).

Original: When D 0 = 0 , Assumption 2.2 is often called “strong growth condition" (SGC) (Vaswani et al., 2019).
Revised: When D 0 = 0 , Assumption 2.2 is often called “strong growth condition" (SGC) (Vaswani et al., 2019).

Original: When (cid:107)∇ f ( x ) (cid:107) = 0 , under SGC we have (cid:107)∇ f j ( x ) (cid:107) = 

Revised: When (cid:107)∇ f ( x ) (cid:107) = 0 , under SGC we have (cid:107)∇ f j ( x ) (cid:107) =

Original: for all j . SGC is increasingly popular recently e.g.(Schmidt & Roux, 2013; Vaswani et al., 2019).
Revised: for all j . SGC is increasingly popular recently e.g.(Schmidt & Roux, 2013; Vaswani et al., 2019).

Original: This condition is known to be reasonable in the overparameterized regime where neural networks can interpolate all data points (Vaswani et al., 2019).
Revised: This condition is known to be reasonable in the overparameterized regime where neural networks can interpolate all data points (Vaswani et al., 2019).

Original: We will show that Adam can converge to critical points if SGC holds.
Revised: We will show that Adam can converge to critical points if SGC holds.

Original: When n, f ∗ , L, D 0 , D 1 are ﬁxed a priori, we use F n,f ∗ L,D 0 ,D 1 ( R d ) to denote the function class containing f ( x ) satisfying Assumption 2.1 and 2.2 with constant n, f ∗ , etc.. Since n is ﬁxed when the function class F n,f ∗ L,D 0 ,D 1 ( R d ) , we introduce this notation to clearly present the divergence result in
Revised: When n, f ∗ , L, D 0 , D 1 are ﬁxed a priori, we use F n,f ∗ L,D 0 ,D 1 ( R d ) to denote the function class containing f ( x ) satisfying Assumption 2.1 and 2.2 with constant n, f ∗ , etc.. Since n is ﬁxed when the function class F n,f ∗ L,D 0 ,D 1 ( R d ) , we introduce this notation to clearly present the divergence result in

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: Without this pre-deﬁned function class, the claim of divergence might be confusing.
Revised: Without this pre-deﬁned function class, the claim of divergence might be confusing.

Original: 2.2 Related Works
Revised: 2.2 Related Works

Original: Ever since Reddi et al.
Revised: Ever since Reddi et al.

Original: (2018) pointed out the divergence issue, there are many attempts on designing new variants of Adam.
Revised: (2018) pointed out the divergence issue, there are many attempts on designing new variants of Adam.

Original: Since we focus on understanding Adam without modiﬁcation on its update rules, we introduce more variants later in Appendix D.1.
Revised: Since we focus on understanding Adam without modiﬁcation on its update rules, we introduce more variants later in Appendix D.1.

Original: Compared with proposing new variants, the convergence of vanilla Adam is far less studied than its variants (perhaps due to the criticism of divergence).
Revised: Compared with proposing new variants, the convergence of vanilla Adam is far less studied than its variants (perhaps due to the criticism of divergence).

Original: There are only a few works analyzing vanilla Adam and they require extra assumptions.
Revised: There are only a few works analyzing vanilla Adam and they require extra assumptions.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2018b) analyze the counter-example in (Reddi et al., 2018) and ﬁnd certain hyperparameter can work.
Revised: (2018b) analyze the counter-example in (Reddi et al., 2018) and ﬁnd certain hyperparameter can work.

Original: However, their analysis is restricted to the counter-example.
Revised: However, their analysis is restricted to the counter-example.

Original: Zaheer et al.
Revised: Zaheer et al.

Original: (2018) study the relation between mini-batch sizes and (non- )convergence of Adam.
Revised: (2018) study the relation between mini-batch sizes and (non- )convergence of Adam.

Original: However, this work require β 1 = 0 and Adam is reduced to RMSProp (Hinton et al., 2012).
Revised: However, this work require β 1 = 0 and Adam is reduced to RMSProp (Hinton et al., 2012).

Original: De et al.
Revised: De et al.

Original: (2018) analyze RMSProp and non-zero- β 1 Adam, but they assume the sign of all stochastic gradients to keep the same.
Revised: (2018) analyze RMSProp and non-zero- β 1 Adam, but they assume the sign of all stochastic gradients to keep the same.

Original: It seems unclear how to check this condition a priori.
Revised: It seems unclear how to check this condition a priori.

Original: Additionally, they require β 1 to be inversely related to the upper bound of gradient, which forces βto be small (as a side note, this result only applies to full-batch Adam).
Revised: Additionally, they require β 1 to be inversely related to the upper bound of gradient, which forces βto be small (as a side note, this result only applies to full-batch Adam).

Original: Défossez et al.
Revised: Défossez et al.

Original: (2020) analyze
Revised: (2020) analyze

Original: Adam with β 1 < β 2 and provide some insights on the momentum mechanisms.
Revised: Adam with β 1 < β 2 and provide some insights on the momentum mechanisms.

Original: However, their bound is inversely proportional to (cid:15) (the hyperparameter for numerical stability) and the bound goes to inﬁnity when (cid:15) goes to 0.
Revised: However, their bound is inversely proportional to (cid:15) (the hyperparameter for numerical stability) and the bound goes to inﬁnity when (cid:15) goes to 0.

Original: This is different from practical application since small (cid:15) such as 10 −often works well.
Revised: This is different from practical application since small (cid:15) such as 10 −often works well.

Original: Further, using large (cid:15) is against the nature of adaptive gradient methods because √ v no longer dominates in the choice of stepsize.
Revised: Further, using large (cid:15) is against the nature of adaptive gradient methods because √ v no longer dominates in the choice of stepsize.

Original: In this case, Adam is essentially transformed back to SGD.
Revised: In this case, Adam is essentially transformed back to SGD.

Original: Two recent works (Huang et al., 2021) and (Guo et al., 2021) propose novel and simple frameworks to analyze Adam-family with large β 1 .
Revised: Two recent works (Huang et al., 2021) and (Guo et al., 2021) propose novel and simple frameworks to analyze Adam-family with large β 1 .

Original: Yet, they require the effective stepsize of Adam to be bounded in certain interval, i.e.,
Revised: Yet, they require the effective stepsize of Adam to be bounded in certain interval, i.e.,

Original: [Equation]
Revised: [Equation]

Original: This boundedness condition changes
Revised: This boundedness condition changes

Original: Adam into AdaBound (Luo et al., 2019) and thus they cannot explain the observations on original
Revised: Adam into AdaBound (Luo et al., 2019) and thus they cannot explain the observations on original

Original: Adam in Section 1.
Revised: Adam in Section 1.

Original: To summarize, all these works require at least one strong assumption (e.g. large (cid:15) ).
Revised: To summarize, all these works require at least one strong assumption (e.g. large (cid:15) ).

Original: Additionally, they all (including those for new variants) require bounded gradient assumptions.
Revised: Additionally, they all (including those for new variants) require bounded gradient assumptions.

Original: A recent work (Shi et al., 2020) takes the ﬁrst attempt to analyze RMSProp without bounded gradient assumption.
Revised: A recent work (Shi et al., 2020) takes the ﬁrst attempt to analyze RMSProp without bounded gradient assumption.

Original: They show that RMSProp can converge to the neighborhood of critical points.
Revised: They show that RMSProp can converge to the neighborhood of critical points.

Original: We believe it is important to study Adam rather than RMSProp: Numerically, Adam often outperforms
Revised: We believe it is important to study Adam rather than RMSProp: Numerically, Adam often outperforms

Original: RMSProp on complicated tasks (e.g. on Atari games, the mean reward is improved from 88% to 110% (Agarwal et al., 2020)).
Revised: RMSProp on complicated tasks (e.g. on Atari games, the mean reward is improved from 88% to 110% (Agarwal et al., 2020)).

Original: Theoretically, literature on RMSProp cannot reveal the interaction between β 1 and β 2 ; or how these hyperparameters jointly affect (or jeopardize) the convergence of Adam.
Revised: Theoretically, literature on RMSProp cannot reveal the interaction between β 1 and β 2 ; or how these hyperparameters jointly affect (or jeopardize) the convergence of Adam.

Original: However, it is non-trivial to jointly analyze the effect of β 1 and β 2 .
Revised: However, it is non-trivial to jointly analyze the effect of β 1 and β 2 .

Original: We point out there are at least three challenges.
Revised: We point out there are at least three challenges.

Original: First, it seems unclear how to control the massive momentum m t of
Revised: First, it seems unclear how to control the massive momentum m t of

Original: Adam.
Revised: Adam.

Original: Second, m t is multiplied by 1 / √ v t , causing non-linear perturbation.
Revised: Second, m t is multiplied by 1 / √ v t , causing non-linear perturbation.

Original: Third, m t and 1 / √ v t are statistically dependent and cannot be decoupled.
Revised: Third, m t and 1 / √ v t are statistically dependent and cannot be decoupled.

Original: We propose new methods to resolve these issues.
Revised: We propose new methods to resolve these issues.

Original: We highlight our technical novelties in Section 5.
Revised: We highlight our technical novelties in Section 5.

Original: 2.3 The Importance and Difﬁculties of Removing Bounded Gradient Assumptions
Revised: 2.3 The Importance and Difﬁculties of Removing Bounded Gradient Assumptions

Original: Here, we emphasize the importance to remove bounded gradient assumption.
Revised: Here, we emphasize the importance to remove bounded gradient assumption.

Original: First, unlike the assumptions in Section 2.1, bounded gradient is not common in SGD analysis.
Revised: First, unlike the assumptions in Section 2.1, bounded gradient is not common in SGD analysis.

Original: So it is of theoretical interests to remove this condition for Adam.
Revised: So it is of theoretical interests to remove this condition for Adam.

Original: Second, bounded gradient condition rules out the chances of gradient divergence a priori.
Revised: Second, bounded gradient condition rules out the chances of gradient divergence a priori.

Original: However, there are numerical evidences showing that Adam’s gradient can diverge (see Section 6 and Appendix B).
Revised: However, there are numerical evidences showing that Adam’s gradient can diverge (see Section 6 and Appendix B).

Original: Removing the boundedness assumption helps us point out the divergence and convergence phase transition in the ( β 1 , β 2 ) diagram.
Revised: Removing the boundedness assumption helps us point out the divergence and convergence phase transition in the ( β 1 , β 2 ) diagram.

Original: However, it is often difﬁcult to analyze convergence without bounded gradient assumption.
Revised: However, it is often difﬁcult to analyze convergence without bounded gradient assumption.

Original: First, it is non-trivial to control stochastic momentum.
Revised: First, it is non-trivial to control stochastic momentum.

Original: Even for SGD, this task is challenging.
Revised: Even for SGD, this task is challenging.

Original: For instance,
Revised: For instance,

Original: An early paper Bertsekas & Tsitsiklis (2000) analyzed SGD-type methods without any boundedness condition.
Revised: An early paper Bertsekas & Tsitsiklis (2000) analyzed SGD-type methods without any boundedness condition.

Original: But it is not until recently that Yu et al.
Revised: But it is not until recently that Yu et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2020b)
Revised: (2020b)

Original: ; Jin et al.
Revised: ; Jin et al.

Original: (2022) prove
Revised: (2022) prove

Original: SGDM (SGD with momentum) converges without bounded gradient assumption.
Revised: SGDM (SGD with momentum) converges without bounded gradient assumption.

Original: Such attempts of removing boundedness assumptions are often appreciated for general optimization problems where “bounded-assumption-free" is considered as a major contribution.
Revised: Such attempts of removing boundedness assumptions are often appreciated for general optimization problems where “bounded-assumption-free" is considered as a major contribution.

Original: Secondly, for Adam, the role of momentum m t is even more intricate since it is multiplied by 1 / √ v t .
Revised: Secondly, for Adam, the role of momentum m t is even more intricate since it is multiplied by 1 / √ v t .

Original: Combined with v t , the impact of previous signals not only affect the update direction, but also change the stepsize for each component.
Revised: Combined with v t , the impact of previous signals not only affect the update direction, but also change the stepsize for each component.

Original: Further, both momentum m t and stepsize 1 / √ v t are random variables and they are highly correlated.
Revised: Further, both momentum m t and stepsize 1 / √ v t are random variables and they are highly correlated.

Original: Such statistical dependency causes trouble for analysis.
Revised: Such statistical dependency causes trouble for analysis.

Original: In summary, the role of momentum in Adam could be much different from that in SGDM or GDM.
Revised: In summary, the role of momentum in Adam could be much different from that in SGDM or GDM.

Original: Even with boundedness conditions, the convergence of large- β 1 Adam is still not well understood (see related works in Section 2.2).
Revised: Even with boundedness conditions, the convergence of large- β 1 Adam is still not well understood (see related works in Section 2.2).

Original: In this work, we propose new techniques to handle Adam’s momentum under any large β 1 , regardless of the gradient magnitude.
Revised: In this work, we propose new techniques to handle Adam’s momentum under any large β 1 , regardless of the gradient magnitude.

Original: These techniques are not revealed in any existing works.
Revised: These techniques are not revealed in any existing works.

Original: We introduce our technical contribution in Section 5.
Revised: We introduce our technical contribution in Section 5.

Original:  Main Results 3.1 Convergence Results
Revised:  Main Results 3.1 Convergence Results

Original: Here, we give the convergence results under large β 2 .
Revised: Here, we give the convergence results under large β 2 .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any f ( x ) ∈ F n,f ∗ L,D 0 ,D 1 ( R d ) , we assume the hyperparameters in Algorithmsatisfy: β 1 < √ β 2 < 1 ; β 2 is greater or equal to a threshold γ 1 ( n ) ; and η k = η 1 √ nk .
Revised: For any f ( x ) ∈ F n,f ∗ L,D 0 ,D 1 ( R d ) , we assume the hyperparameters in Algorithmsatisfy: β 1 < √ β 2 < 1 ; β 2 is greater or equal to a threshold γ 1 ( n ) ; and η k = η 1 √ nk .

Original: Let k m ∈
Revised: Let k m ∈

Original: N
Revised: N

Original: satisﬁes k m ≥ 4 and β ( k m − 1)
Revised: satisﬁes k m ≥ 4 and β ( k m − 1)

Original: n≤
Revised: n≤

Original: [Equation]
Revised: [Equation]

Original: n 1 √ k m − 1 , 5 we have the following results for any T > k m :
Revised: n 1 √ k m − 1 , 5 we have the following results for any T > k m :

Original: [Equation]
Revised: [Equation]

Original: Remark 1: the choice of β 2 .
Revised: Remark 1: the choice of β 2 .

Original: Our theory suggests that large β 2 should be used to ensure convergence.
Revised: Our theory suggests that large β 2 should be used to ensure convergence.

Original: This message matches our experimental ﬁndings in Figure 1.
Revised: This message matches our experimental ﬁndings in Figure 1.

Original: We would like to point out that the requirement of “large β 2 " is neccessary, because small β 2 will indeed lead to divergence (shown later in Section 3.2).
Revised: We would like to point out that the requirement of “large β 2 " is neccessary, because small β 2 will indeed lead to divergence (shown later in Section 3.2).

Original: We here comment a bit on the the threshold γ 1 ( n ) .
Revised: We here comment a bit on the the threshold γ 1 ( n ) .

Original: γ 1 ( n ) satisﬁes β 2 ≥ 1 − O (cid:16) 1 − β n 1 n 2 ρ (cid:17) (see inequality (34) and Remark G.7), where ρ is a constant that depends on the training trajectory.
Revised: γ 1 ( n ) satisﬁes β 2 ≥ 1 − O (cid:16) 1 − β n 1 n 2 ρ (cid:17) (see inequality (34) and Remark G.7), where ρ is a constant that depends on the training trajectory.

Original: In worst cases, ρ is upper bounded by n 2 . 5 , but we ﬁnd the practical ρ to be much smaller.
Revised: In worst cases, ρ is upper bounded by n 2 . 5 , but we ﬁnd the practical ρ to be much smaller.

Original: In Appendix B, we estimate ρ on MNIST and CIFAR-10.
Revised: In Appendix B, we estimate ρ on MNIST and CIFAR-10.

Original: In practical training process, we empirically observe that ρ ≈ O ( n ) , thus the required γ 1 ( n ) ≈ 1 − O (cid:0) n − 3 (cid:1) .
Revised: In practical training process, we empirically observe that ρ ≈ O ( n ) , thus the required γ 1 ( n ) ≈ 1 − O (cid:0) n − 3 (cid:1) .

Original: Note that our threshold of β 2 is a sufﬁcient condition for convergence, so there may be a gap between the practical choices and the theoretical bound of β 2 .
Revised: Note that our threshold of β 2 is a sufﬁcient condition for convergence, so there may be a gap between the practical choices and the theoretical bound of β 2 .

Original: Closing the gap will be an interesting future direction.
Revised: Closing the gap will be an interesting future direction.

Original: We ﬁnd that γ 1 ( n ) increases with n .
Revised: We ﬁnd that γ 1 ( n ) increases with n .

Original: This property suggests that larger β 2 should be used when n is large.
Revised: This property suggests that larger β 2 should be used when n is large.

Original: This phenomenon is also veriﬁed by our experiments in Appendix B. We also remark that γ 1 ( n ) slowly increases with β 1 .
Revised: This phenomenon is also veriﬁed by our experiments in Appendix B. We also remark that γ 1 ( n ) slowly increases with β 1 .

Original: This property is visualized in Figure 1 (d) where the lower boundary of blue region slightly lifts up when β 1 increases.
Revised: This property is visualized in Figure 1 (d) where the lower boundary of blue region slightly lifts up when β 1 increases.

Original: Remark 2: the choice of β 1 .
Revised: Remark 2: the choice of β 1 .

Original: Theorem 3.1 requires β 1 < √ β 2 .
Revised: Theorem 3.1 requires β 1 < √ β 2 .

Original: Since β 2 is suggested to be large, our convergence result can cover ﬂexible choice of β 1 ∈ [0 , 1) .
Revised: Since β 2 is suggested to be large, our convergence result can cover ﬂexible choice of β 1 ∈ [0 , 1) .

Original: For instance, β 2 = 0 . 999 brings the threshold of β 1 < 0 .
Revised: For instance, β 2 = 0 . 999 brings the threshold of β 1 < 0 .

Original: 9995 , which covers basically all practical choices of β 1 reported in the literature (see Section 1), including the default setting β 1 = 0 .
Revised: 9995 , which covers basically all practical choices of β 1 reported in the literature (see Section 1), including the default setting β 1 = 0 .

Original: This result is much stronger than those in the
Revised: This result is much stronger than those in the

Original: RMSProp literature (e.g. (Shi et al., 2020; Zaheer et al., 2018)).
Revised: RMSProp literature (e.g. (Shi et al., 2020; Zaheer et al., 2018)).

Original: To our knowledge, we are the ﬁrst to prove convergence of Adam under any β 1 ∈ [0 , 1) without bounded gradient assumption.
Revised: To our knowledge, we are the ﬁrst to prove convergence of Adam under any β 1 ∈ [0 , 1) without bounded gradient assumption.

Original: Remark 3: convergence to a bounded region.
Revised: Remark 3: convergence to a bounded region.

Original: When D 0 > 0 , Adam converges to a bounded region near critical points.
Revised: When D 0 > 0 , Adam converges to a bounded region near critical points.

Original: As discussed in Section 2.1, converging to bounded region is common for stochastic methods including constant-stepsize SGD (Yan et al., 2018;
Revised: As discussed in Section 2.1, converging to bounded region is common for stochastic methods including constant-stepsize SGD (Yan et al., 2018;

Original: Yu et al., 2019; Liu et al., 2020b) and diminishing-stepsize RMSProp (Zaheer et al., 2018; Shi et al., 2020).
Revised: Yu et al., 2019; Liu et al., 2020b) and diminishing-stepsize RMSProp (Zaheer et al., 2018; Shi et al., 2020).

Original: This phenomenon is also observed in practice: even for convex quadratic function with D 0 > 0 , Adam with diminishing stepsize cannot reach exactly zero gradient (see Figure 4 (a) in Section 6).
Revised: This phenomenon is also observed in practice: even for convex quadratic function with D 0 > 0 , Adam with diminishing stepsize cannot reach exactly zero gradient (see Figure 4 (a) in Section 6).

Original: This is because: even though η k is decreasing, the effective stepsize η k / √ v k,i might not decay.
Revised: This is because: even though η k is decreasing, the effective stepsize η k / √ v k,i might not decay.

Original: The good news is that, the constant O ( √ D 0 ) vanishes to 0 as β 2 goes to 1 (both in theory and experiments).
Revised: The good news is that, the constant O ( √ D 0 ) vanishes to 0 as β 2 goes to 1 (both in theory and experiments).

Original: The relation between β 2 and constant O ( √ D 0 ) are introduced in Remark G.14 in Appendix G.9.
Revised: The relation between β 2 and constant O ( √ D 0 ) are introduced in Remark G.14 in Appendix G.9.

Original: The size shrinks to 0 because the movement of √ v k,i shrinks as β 2 increases.
Revised: The size shrinks to 0 because the movement of √ v k,i shrinks as β 2 increases.

Original: As a corollary of Theorem 3.1, we have the following result under SGC (i.e., D 0 = 0 ).
Revised: As a corollary of Theorem 3.1, we have the following result under SGC (i.e., D 0 = 0 ).

Original: Corollary 3.2.
Revised: Corollary 3.2.

Original: Under the setting in Theorem 3.1.
Revised: Under the setting in Theorem 3.1.

Original: When D 0 = 0 for Assumption 2.2, we have
Revised: When D 0 = 0 for Assumption 2.2, we have

Original: [Equation]
Revised: [Equation]

Original: Under SGC (i.e. D 0 = 0 ), Corollary 3.2 states that Adam can converge to critical points.
Revised: Under SGC (i.e. D 0 = 0 ), Corollary 3.2 states that Adam can converge to critical points.

Original: This is indeed the case in practice.
Revised: This is indeed the case in practice.

Original: For instance, function (2) satisﬁes SGC and we observe 0 gradient norm after Adam converges (see Section 6 and Appendix B).
Revised: For instance, function (2) satisﬁes SGC and we observe 0 gradient norm after Adam converges (see Section 6 and Appendix B).

Original: The convergence rate in Corollary 3.2 is comparable to that of SGD under the same condition in (Vaswani et al., 2019).
Revised: The convergence rate in Corollary 3.2 is comparable to that of SGD under the same condition in (Vaswani et al., 2019).

Original: 3.2 Divergence Results
Revised: 3.2 Divergence Results

Original: Theorem 3.1 shows that when β 2 is large, any β 1 < √ β 2 ensures convergence.
Revised: Theorem 3.1 shows that when β 2 is large, any β 1 < √ β 2 ensures convergence.

Original: Now we consider the case where β 2 is small.
Revised: Now we consider the case where β 2 is small.

Original: We will show that in this case, a wide range of β 1 is facing the risk of diverging to inﬁnity.
Revised: We will show that in this case, a wide range of β 1 is facing the risk of diverging to inﬁnity.

Original: The divergence of small- β 2 Adam suggests that “large β 2 " is necessary in
Revised: The divergence of small- β 2 Adam suggests that “large β 2 " is necessary in

Original: When the convergence result Theorem 3.1.
Revised: When the convergence result Theorem 3.1.

Original: We construct a counter-example in F n,f ∗ L,D 0 ,D 1 ( R d ) .
Revised: We construct a counter-example in F n,f ∗ L,D 0 ,D 1 ( R d ) .

Original: Consider f ( x ) = (cid:80) n − 1 i =

Revised: Consider f ( x ) = (cid:80) n − 1 i =

Original: f i ( x ) for x ∈ R , we deﬁne f i ( x ) as:
Revised: f i ( x ) for x ∈ R , we deﬁne f i ( x ) as:

Original: [Equation]
Revised: [Equation]

Original: Summing up all the f i ( x ) , we can see that
Revised: Summing up all the f i ( x ) , we can see that

Original: [Equation]
Revised: [Equation]

Original: is a lower bounded convex smooth function with optimal solution x ∗ = − 2 .
Revised: is a lower bounded convex smooth function with optimal solution x ∗ = − 2 .

Original: Function (2) allows both iterates and gradients to diverge to inﬁnity.
Revised: Function (2) allows both iterates and gradients to diverge to inﬁnity.

Original: As shown in Figure 1 (a), when running Adam on (2), there exists a red large-error region.
Revised: As shown in Figure 1 (a), when running Adam on (2), there exists a red large-error region.

Original: This shows the sign of divergence.
Revised: This shows the sign of divergence.

Original: We further theoretically verify the conjecture in Proposition 3.3.
Revised: We further theoretically verify the conjecture in Proposition 3.3.

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: For any function class F n,f ∗ L,D 0 ,D 1 ( R d ) , there exists a f ( x ) ∈ F n,f ∗ L,D 0 ,D 1 ( R d ) , s.t.
Revised: For any function class F n,f ∗ L,D 0 ,D 1 ( R d ) , there exists a f ( x ) ∈ F n,f ∗ L,D 0 ,D 1 ( R d ) , s.t.

Original: when ( β 1 , β 2 ) satisﬁes analytic condition (12) , (13) , (14) in Appendix E, Adam’s iterates and function values diverge to inﬁnity.
Revised: when ( β 1 , β 2 ) satisﬁes analytic condition (12) , (13) , (14) in Appendix E, Adam’s iterates and function values diverge to inﬁnity.

Original: By solving these conditions in NumPy , we plot the orange region in Figure 2.
Revised: By solving these conditions in NumPy , we plot the orange region in Figure 2.

Original: The size of the region depends on n and it expands to the whole region when n goes to inﬁnity.
Revised: The size of the region depends on n and it expands to the whole region when n goes to inﬁnity.

Original: [Equation]
Revised: [Equation]

Original: The proof can be seen in Appendix E. We ﬁnd the “divergence region" always stays below the “convergence threshold" γ 1 ( n ) in
Revised: The proof can be seen in Appendix E. We ﬁnd the “divergence region" always stays below the “convergence threshold" γ 1 ( n ) in

Original: Theorem 3.1, so the two results are self-consistent (see the remark in Appendix E).
Revised: Theorem 3.1, so the two results are self-consistent (see the remark in Appendix E).

Original: Proposition 3.3 states the divergence of iterates and function values.
Revised: Proposition 3.3 states the divergence of iterates and function values.

Original: Consistently, our experiments also show the divergence of gradient (see Section 6 and Appendix B).
Revised: Consistently, our experiments also show the divergence of gradient (see Section 6 and Appendix B).

Original: These results characterize Adam’s divergence behavior both numerically and theoretically.
Revised: These results characterize Adam’s divergence behavior both numerically and theoretically.

Original: We emphasize that the orange region is not discussed in (Reddi et al., 2018) because we consider n ﬁxed while they allow n changing.
Revised: We emphasize that the orange region is not discussed in (Reddi et al., 2018) because we consider n ﬁxed while they allow n changing.

Original: When n is allowed to increase, our orange region will expand to the whole region and thus we can derive a similar (actually stronger) result as (Reddi et al., 2018).
Revised: When n is allowed to increase, our orange region will expand to the whole region and thus we can derive a similar (actually stronger) result as (Reddi et al., 2018).

Original: We provide more explanation in
Revised: We provide more explanation in

Original: Section 4.
Revised: Section 4.

Original: Combining Theorem 3.1 and Proposition 3.3, we establish a clearer image on the relation between ( β 1 , β 2 ) and qualitative behavior of Adam.
Revised: Combining Theorem 3.1 and Proposition 3.3, we establish a clearer image on the relation between ( β 1 , β 2 ) and qualitative behavior of Adam.

Original:  Reconciling Our Results with (Reddi et al., 2018)
Revised:  Reconciling Our Results with (Reddi et al., 2018)

Original: We discuss more on the relation between (Reddi et al., 2018) and our results.
Revised: We discuss more on the relation between (Reddi et al., 2018) and our results.

Original: The divergence result shown in Section 1 does not contradict with our convergence results in Theorem 3.1.
Revised: The divergence result shown in Section 1 does not contradict with our convergence results in Theorem 3.1.

Original: Further, it is different from our divergence result in Proposition 3.3.
Revised: Further, it is different from our divergence result in Proposition 3.3.

Original: The key difference lies in whether ( β 1 , β 2 ) is picked before or after picking the function class F n,f ∗ L,D 0 ,D 1 ( R d ) .
Revised: The key difference lies in whether ( β 1 , β 2 ) is picked before or after picking the function class F n,f ∗ L,D 0 ,D 1 ( R d ) .

Original: We discuss the following two cases.
Revised: We discuss the following two cases.

Original: Case I: When ( β 1 , β 2 ) is picked before picking F n,f ∗ L,D 0 ,D 1 ( R d ) .
Revised: Case I: When ( β 1 , β 2 ) is picked before picking F n,f ∗ L,D 0 ,D 1 ( R d ) .

Original: As discussed in Section 1, the divergence result requires different n for different ( β 1 , β 2 ) .
Revised: As discussed in Section 1, the divergence result requires different n for different ( β 1 , β 2 ) .

Original: In this sense, the considered function class is constantly changing.
Revised: In this sense, the considered function class is constantly changing.

Original: It does not contradict with our Theorem 3. which considers a ﬁxed function class with ﬁxed n .
Revised: It does not contradict with our Theorem 3. which considers a ﬁxed function class with ﬁxed n .

Original: For Case I , we illustrate Adam’s behavior in Figure 3.
Revised: For Case I , we illustrate Adam’s behavior in Figure 3.

Original: The red region is proved by (Reddi et al., 2018).
Revised: The red region is proved by (Reddi et al., 2018).

Original: For completeness, we remove the condition “ β 1 < √ β 2 " and further prove that Adam will diverge to inﬁnity for any ( β 1 , β 2 ) ∈ [0 , 1) 2 .
Revised: For completeness, we remove the condition “ β 1 < √ β 2 " and further prove that Adam will diverge to inﬁnity for any ( β 1 , β 2 ) ∈ [0 , 1) 2 .

Original: The result is shown in the following Corollary 4.1.
Revised: The result is shown in the following Corollary 4.1.

Original: Corollary 4.1.
Revised: Corollary 4.1.

Original: For any ( β 1 , β 2 ) ∈ [0 , 1) 2 , there exists a function satisfying Assumption 2.1 and 2.
Revised: For any ( β 1 , β 2 ) ∈ [0 , 1) 2 , there exists a function satisfying Assumption 2.1 and 2.

Original: that the Adam’s iterates and function values diverge to inﬁnity.
Revised: that the Adam’s iterates and function values diverge to inﬁnity.

Original: Proof of Corollary 4.1 can be seen in the ﬁnal paragraph in Appendix E.
Revised: Proof of Corollary 4.1 can be seen in the ﬁnal paragraph in Appendix E.

Original: In the proof, we also require different n to cause divergence for different ( β 1 , β 2 ) .
Revised: In the proof, we also require different n to cause divergence for different ( β 1 , β 2 ) .

Original: So the function class is constantly changing.
Revised: So the function class is constantly changing.

Original: As a result, in Case I , we cannot prove any convergence result.
Revised: As a result, in Case I , we cannot prove any convergence result.

Original: [Table]
Revised: [Table]

Original: Case II: When ( β 1 , β 2 ) is picked after picking F n,f ∗ L,D 0 ,D 1 ( R d ) .
Revised: Case II: When ( β 1 , β 2 ) is picked after picking F n,f ∗ L,D 0 ,D 1 ( R d ) .

Original: When the function class is picked in advance, sample size n will also be ﬁxed.
Revised: When the function class is picked in advance, sample size n will also be ﬁxed.

Original: This case is closer to most practical applications.
Revised: This case is closer to most practical applications.

Original: In this case, we ﬁnd that Adam’s behavior changes signiﬁcantly in the different region of Figure 3.
Revised: In this case, we ﬁnd that Adam’s behavior changes signiﬁcantly in the different region of Figure 3.

Original: First, ∀ f ( x ) ∈ F n,f ∗ L,D 0 ,D 1 ( R d ) will converge when β 1 < √ β 2 and β 2 is large.
Revised: First, ∀ f ( x ) ∈ F n,f ∗ L,D 0 ,D 1 ( R d ) will converge when β 1 < √ β 2 and β 2 is large.

Original: Second, ∃ f ( x ) ∈
Revised: Second, ∃ f ( x ) ∈

Original: F n,f ∗ L,D 0 ,D 1 ( R d ) will diverge to inﬁnity when ( β 1 , β 2 ) are in the orange region in Figure 2.
Revised: F n,f ∗ L,D 0 ,D 1 ( R d ) will diverge to inﬁnity when ( β 1 , β 2 ) are in the orange region in Figure 2.

Original: Since Case
Revised: Since Case

Original: II is closer to practical scenarios, these results can provide better guidance for hyperparameter tuning for Adam users.
Revised: II is closer to practical scenarios, these results can provide better guidance for hyperparameter tuning for Adam users.

Original: We provide some suggestions for practitioners in Appendix C.
Revised: We provide some suggestions for practitioners in Appendix C.

Original: For Case II , we summarize the possible behaviors of Adam in Table 1.
Revised: For Case II , we summarize the possible behaviors of Adam in Table 1.

Original: We also illustrate our convergence and divergence results in Figure 1 (d).
Revised: We also illustrate our convergence and divergence results in Figure 1 (d).

Original: Note that there are some blanket areas where
Revised: Note that there are some blanket areas where

Original: Adam’s behavior remains unknown, this part will be left as interesting future work.
Revised: Adam’s behavior remains unknown, this part will be left as interesting future work.

Original:  Proof Ideas for the Convergence Result
Revised:  Proof Ideas for the Convergence Result

Original: We now (informally) introduce our proof ideas for the convergence result in Theorem 3.1.
Revised: We now (informally) introduce our proof ideas for the convergence result in Theorem 3.1.

Original: Simply put, we want to control the update direction m k,i / √ v k,i inside the dual cone of gradient direction.
Revised: Simply put, we want to control the update direction m k,i / √ v k,i inside the dual cone of gradient direction.

Original: Namely:
Revised: Namely:

Original: [Equation]
Revised: [Equation]

Original: However, directly proving (3) could be difﬁcult because both m k,i and v k,i distort the trajectory.
Revised: However, directly proving (3) could be difﬁcult because both m k,i and v k,i distort the trajectory.

Original: To start with, we try to control the movement of v k,i .
Revised: To start with, we try to control the movement of v k,i .

Original: We use a standard approach for Adam-type algorithms: use large or increasing β 2 (e.g. (Shi et al., 2020;
Revised: We use a standard approach for Adam-type algorithms: use large or increasing β 2 (e.g. (Shi et al., 2020;

Original: Zou et al., 2019; Chen et al., 2018b)).
Revised: Zou et al., 2019; Chen et al., 2018b)).

Original: Recall v k,i =
Revised: Recall v k,i =

Original: [Equation]
Revised: [Equation]

Original: β
Revised: β

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: k,j
Revised: k,j

Original: [Equation]
Revised: [Equation]

Original: x k,j ) ◦ ∇ f τ k,j ( x k,j ) +
Revised: x k,j ) ◦ ∇ f τ k,j ( x k,j ) +

Original: β i 2 v k, 0 , we have v k,i ≈ v k, 0 when β 2 is large.
Revised: β i 2 v k, 0 , we have v k,i ≈ v k, 0 when β 2 is large.

Original: In this case, we have:
Revised: In this case, we have:

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst “ ≈ " is due to the large β 2 and the second “ ≈ " is our goal.
Revised: where the ﬁrst “ ≈ " is due to the large β 2 and the second “ ≈ " is our goal.

Original: Now we need to show:
Revised: Now we need to show:

Original: [Equation]
Revised: [Equation]

Original: where ∂ l f ( x k, 0 ) is the l -th component of ∇ f ( x k, 0 ) , similarly for m l,k, 

Revised: where ∂ l f ( x k, 0 ) is the l -th component of ∇ f ( x k, 0 ) , similarly for m l,k,

Original: and v l,k, 0 .
Revised: and v l,k, 0 .

Original: ( ∗ ) is due to the ﬁnite-sum structure.
Revised: ( ∗ ) is due to the ﬁnite-sum structure.

Original: However, it is not easy to prove (4).
Revised: However, it is not easy to prove (4).

Original: We point out some technical issues below.
Revised: We point out some technical issues below.

Original: Issue I: massive momentum.
Revised: Issue I: massive momentum.

Original: Directly proving (4) is still not easy.
Revised: Directly proving (4) is still not easy.

Original: We need to ﬁrst consider a simpliﬁed problem: for every l ∈ [ d ] , assume we treat ∂ l f ( x k, 0 ) / √ v l,k, 0 as a constant, how to bound
Revised: We need to ﬁrst consider a simpliﬁed problem: for every l ∈ [ d ] , assume we treat ∂ l f ( x k, 0 ) / √ v l,k, 0 as a constant, how to bound

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: It turns out that this simpliﬁed problem is still non-trivial.
Revised: It turns out that this simpliﬁed problem is still non-trivial.

Original: When β 1 is large, m l,k,i contains heavy historical signals which signiﬁcantly distort the trajectory from gradient direction.
Revised: When β 1 is large, m l,k,i contains heavy historical signals which signiﬁcantly distort the trajectory from gradient direction.

Original: Existing literature (Zaheer et al., 2018; De et al., 2018; Shi et al., 2020) take a naive approach: they set β 1 ≈ 0 so that m l,k,i ≈ ∂ l f τ k,i ( x k,i ) .
Revised: Existing literature (Zaheer et al., 2018; De et al., 2018; Shi et al., 2020) take a naive approach: they set β 1 ≈ 0 so that m l,k,i ≈ ∂ l f τ k,i ( x k,i ) .

Original: Then we get (4) ≈ 0 .
Revised: Then we get (4) ≈ 0 .

Original: However, this method cannot be applied here since we are interested in practical cases where β 1 is large in [0 , 1) .
Revised: However, this method cannot be applied here since we are interested in practical cases where β 1 is large in [0 , 1) .

Original: Issue II: stochastic non-linear dynamics.
Revised: Issue II: stochastic non-linear dynamics.

Original: Even if we solve Issue I , it is still unclear how to prove (4).
Revised: Even if we solve Issue I , it is still unclear how to prove (4).

Original: This is because: for every l ∈ [ d ] , ∂ l f ( x k, 0 ) / √ v l,k, 0 is a r.v. instead of a constant.
Revised: This is because: for every l ∈ [ d ] , ∂ l f ( x k, 0 ) / √ v l,k, 0 is a r.v. instead of a constant.

Original: With this term involved, we are facing with a stochastic non-linear dynamics, which could be difﬁcult to analyze.
Revised: With this term involved, we are facing with a stochastic non-linear dynamics, which could be difﬁcult to analyze.

Original: Further, ∂ l f ( x k, 0 ) / √ v l,k, 0 is statistically dependent with (cid:0) m l,k,i − ∂ l f τ k,i ( x k, 0 ) (cid:1) , so we are not allowed to handle the expectation E ( ∂ l f ( x k, 0 ) / √ v l,k, 0 ) separately .
Revised: Further, ∂ l f ( x k, 0 ) / √ v l,k, 0 is statistically dependent with (cid:0) m l,k,i − ∂ l f τ k,i ( x k, 0 ) (cid:1) , so we are not allowed to handle the expectation E ( ∂ l f ( x k, 0 ) / √ v l,k, 0 ) separately .

Original: Unfortunately, even with additional assumptions like bounded gradient, there is no general approach to tackle the above issues.
Revised: Unfortunately, even with additional assumptions like bounded gradient, there is no general approach to tackle the above issues.

Original: In this work, we propose solutions regardless of gradient magnitude.
Revised: In this work, we propose solutions regardless of gradient magnitude.

Original: Solution to Issue I.
Revised: Solution to Issue I.

Original: We prove the following Lemma to resolve Issue I .
Revised: We prove the following Lemma to resolve Issue I .

Original: Lemma 5.1.
Revised: Lemma 5.1.

Original: (Informal) Consider Algorithm 1.
Revised: (Informal) Consider Algorithm 1.

Original: For every l ∈ [ d ] and any β 1 ∈ [0 , 1) , we have the following result under Assumption 2.1.
Revised: For every l ∈ [ d ] and any β 1 ∈ [0 , 1) , we have the following result under Assumption 2.1.

Original: [Equation]
Revised: [Equation]

Original: where ∂ l f ( x k, 0 ) is the l -th component of ∇ f ( x k, 0 ) ; m l,k,i
Revised: where ∂ l f ( x k, 0 ) is the l -th component of ∇ f ( x k, 0 ) ; m l,k,i

Original: = (1 − β 1 ) ∂ l f τ k,i ( x k,i ) + β 1 m l,k,i − 1 .
Revised: = (1 − β 1 ) ∂ l f τ k,i ( x k,i ) + β 1 m l,k,i − 1 .

Original: We present the proof idea in Appendix A. Simply put, we construct a simple toy example called “color-ball" model (of the 1st kind).
Revised: We present the proof idea in Appendix A. Simply put, we construct a simple toy example called “color-ball" model (of the 1st kind).

Original: This toy model shows a special property of δ ( β 1 ) .
Revised: This toy model shows a special property of δ ( β 1 ) .

Original: We ﬁnd out: for Algorithm 1, error terms from successive epochs can be canceled, which keeps the momentum roughly in the descent direction.
Revised: We ﬁnd out: for Algorithm 1, error terms from successive epochs can be canceled, which keeps the momentum roughly in the descent direction.

Original: This important property is not revealed in any existing work.
Revised: This important property is not revealed in any existing work.

Original: Remark 4: When assuming bounded gradient (cid:107)∇ f ( x ) (cid:107) ≤ G , a naive upper bound would be δ ( β 1 )
Revised: Remark 4: When assuming bounded gradient (cid:107)∇ f ( x ) (cid:107) ≤ G , a naive upper bound would be δ ( β 1 )

Original: =
Revised: =

Original: O ( G ) .
Revised: O ( G ) .

Original: However, such constant upper bound does not imply δ ( β 1 ) is close to 0.
Revised: However, such constant upper bound does not imply δ ( β 1 ) is close to 0.

Original: It will not help prove the convergence.
Revised: It will not help prove the convergence.

Original: This might be partially the reason why large- β 1 Adam is hard to analyze even under bounded gradient (see related works in Section 2.2).
Revised: This might be partially the reason why large- β 1 Adam is hard to analyze even under bounded gradient (see related works in Section 2.2).

Original: We emphasize Lemma 5.1 holds true regardless of gradient norm, so it could be deployed in both bounded or unbounded gradient analysis.
Revised: We emphasize Lemma 5.1 holds true regardless of gradient norm, so it could be deployed in both bounded or unbounded gradient analysis.

Original: Solution to Issue II.
Revised: Solution to Issue II.

Original: We try to show (4) by adopting Lemma 5.1.
Revised: We try to show (4) by adopting Lemma 5.1.

Original: However, the direct application cannot work since ∂ l f ( x k, 0 ) √ v l,k, 0 is random.
Revised: However, the direct application cannot work since ∂ l f ( x k, 0 ) √ v l,k, 0 is random.

Original: Despite its randomness, we ﬁnd out that when β 2 is large, the changes of ∂ l f ( x k, 0 ) √ v l,k, 0 shrinks along iteration.
Revised: Despite its randomness, we ﬁnd out that when β 2 is large, the changes of ∂ l f ( x k, 0 ) √ v l,k, 0 shrinks along iteration.

Original: As such, although ∂ l f ( x k, 0 ) √ v l,k, 0 brings extra perturbation, the quantity in (4) share the similar asymptotic behavior as δ ( β 1 ) .
Revised: As such, although ∂ l f ( x k, 0 ) √ v l,k, 0 brings extra perturbation, the quantity in (4) share the similar asymptotic behavior as δ ( β 1 ) .

Original: We prove the following Lemma 5.2.
Revised: We prove the following Lemma 5.2.

Original: Lemma 5.2.
Revised: Lemma 5.2.

Original: (Informal) Under Assumption 2.1 and 2.2, consider Algorithm 1 with large β 2 and β 1 < √ β 2 .
Revised: (Informal) Under Assumption 2.1 and 2.2, consider Algorithm 1 with large β 2 and β 1 < √ β 2 .

Original: For those l with gradient component larger than certain threshold, we have:
Revised: For those l with gradient component larger than certain threshold, we have:

Original: [Equation]
Revised: [Equation]

Original: In Appendix A, we introduce how to derive (6) from (5).
Revised: In Appendix A, we introduce how to derive (6) from (5).

Original: To do so, we introduce a new type of “color-ball" model (we call it color-ball of the 2nd kind) which adopts the random perturbation of ∂ l f ( x k, 0 ) √ v l,k, 0 .
Revised: To do so, we introduce a new type of “color-ball" model (we call it color-ball of the 2nd kind) which adopts the random perturbation of ∂ l f ( x k, 0 ) √ v l,k, 0 .

Original: Understanding color-ball model of the 2nd kind is crucial for proving Lemma 5.2.
Revised: Understanding color-ball model of the 2nd kind is crucial for proving Lemma 5.2.

Original: We conclude the proof of (4) by some additional analysis on “those l with small gradient component".
Revised: We conclude the proof of (4) by some additional analysis on “those l with small gradient component".

Original: This case is a bit easier since it reduces to bounded gradient case.
Revised: This case is a bit easier since it reduces to bounded gradient case.

Original: For readers who wants to learn more about the idea of tackling Issue I and II , please refer to Appendix A where we formally introduce the 1st and 2nd kind of color-ball models.
Revised: For readers who wants to learn more about the idea of tackling Issue I and II , please refer to Appendix A where we formally introduce the 1st and 2nd kind of color-ball models.

Original: Since the whole proof is quite long, we provide a proof sketch in Appendix G.1.
Revised: Since the whole proof is quite long, we provide a proof sketch in Appendix G.1.

Original: The whole proof is presented in Appendix G.
Revised: The whole proof is presented in Appendix G.

Original:  Experiments
Revised:  Experiments

Original: To support our theory, we provide more simulations and real-data experiments.
Revised: To support our theory, we provide more simulations and real-data experiments.

Original: All the experimental settings and hyperparameters are presented in Appendix B.1.
Revised: All the experimental settings and hyperparameters are presented in Appendix B.1.

Original: We aim to show:
Revised: We aim to show:

Original: (I).
Revised: (I).

Original: When β 2 is large, a large range of β 1 gives good performance, including all β 1 < √ β 2 .
Revised: When β 2 is large, a large range of β 1 gives good performance, including all β 1 < √ β 2 .

Original: (II).
Revised: (II).

Original: When β 2 is small, a large range of β 1 performs relatively badly.
Revised: When β 2 is small, a large range of β 1 performs relatively badly.

Original: [Figure]
Revised: [Figure]

Original: Convergence to bounded region when D 0 > 0 .
Revised: Convergence to bounded region when D 0 > 0 .

Original: In Figure 4, we run large- β 2 Adam on function (9) (deﬁned later in Appendix B).
Revised: In Figure 4, we run large- β 2 Adam on function (9) (deﬁned later in Appendix B).

Original: This function satisﬁes with D 0 > 0 .
Revised: This function satisﬁes with D 0 > 0 .

Original: We ﬁnd that even with diminishing stepsize η k = 1 / √ k , Adam may not converge to an exact critical point.
Revised: We ﬁnd that even with diminishing stepsize η k = 1 / √ k , Adam may not converge to an exact critical point.

Original: Instead, it converges to a bounded region.
Revised: Instead, it converges to a bounded region.

Original: This is because: even though η k is decreasing, the effective stepsize η k / √ v k,i might not decay.
Revised: This is because: even though η k is decreasing, the effective stepsize η k / √ v k,i might not decay.

Original: Further, the size of the region shrinks when β 2 increases.
Revised: Further, the size of the region shrinks when β 2 increases.

Original: This is because the movement of √ v k,i shrinks as β 2 increases.
Revised: This is because the movement of √ v k,i shrinks as β 2 increases.

Original: These phenomena match Remark 3 and claim (I) .
Revised: These phenomena match Remark 3 and claim (I) .

Original: Convergence to critical points when D 0 = 

Revised: Convergence to critical points when D 0 =

Original: Since function (2) satisﬁes D 0 = 0 , we run more experiments on (2) with initialization x = − 5 and n = 5 , 10 , 15 , 20 .
Revised: Since function (2) satisﬁes D 0 = 0 , we run more experiments on (2) with initialization x = − 5 and n = 5 , 10 , 15 , 20 .

Original: We show the result of n = 20 in
Revised: We show the result of n = 20 in

Original: Figure 4 (a), (b); the rest are shown in Appendix B.
Revised: Figure 4 (a), (b); the rest are shown in Appendix B.

Original: We ﬁnd
Revised: We ﬁnd

Original: that: when β 2 is large, Adam converges to critical points for β 1 < √ β 2 .
Revised: that: when β 2 is large, Adam converges to critical points for β 1 < √ β 2 .

Original: These phenomena match claim (I) .
Revised: These phenomena match claim (I) .

Original: Gradient norm of iterates can be unbounded when β 2 is small.
Revised: Gradient norm of iterates can be unbounded when β 2 is small.

Original: On function (2), We further run Adam with small β 2 at initialization x = − 5 .
Revised: On function (2), We further run Adam with small β 2 at initialization x = − 5 .

Original: In this case, gradient norms of iterates increase dramatically.
Revised: In this case, gradient norms of iterates increase dramatically.

Original: This emphasizes the importance of discarding bounded gradient assumptions.
Revised: This emphasizes the importance of discarding bounded gradient assumptions.

Original: These phenomena match claim (II) .
Revised: These phenomena match claim (II) .

Original: MNIST and CIFAR-10.
Revised: MNIST and CIFAR-10.

Original: As shown in Figure 1 (b)& (c) in Section 1, the training results match both claim (I) and (II).
Revised: As shown in Figure 1 (b)& (c) in Section 1, the training results match both claim (I) and (II).

Original: In addition, there is a convex-shaped boundary on the transition from low loss to higher loss, this boundary roughly matches the condition in Theorem 3.1.
Revised: In addition, there is a convex-shaped boundary on the transition from low loss to higher loss, this boundary roughly matches the condition in Theorem 3.1.

Original: NLP.
Revised: NLP.

Original: We use Adam to train Transformer XL (Dai et al., 2019) on the WikiText-103 dataset (Merity et al., 2016).
Revised: We use Adam to train Transformer XL (Dai et al., 2019) on the WikiText-103 dataset (Merity et al., 2016).

Original: This architecture and dataset is widely used in NLP tasks (e.g. (Howard & Ruder, 2018;
Revised: This architecture and dataset is widely used in NLP tasks (e.g. (Howard & Ruder, 2018;

Original: See et al., 2017)).
Revised: See et al., 2017)).

Original: As shown in Figure 4 (d), the training results match both claim (I) and (II).
Revised: As shown in Figure 4 (d), the training results match both claim (I) and (II).

Original:  Conclusions
Revised:  Conclusions

Original: In this work, we explore the (non-)convergence of Adam.
Revised: In this work, we explore the (non-)convergence of Adam.

Original: When β 2 is large, we prove that Adam can converge with any β 1 < √ β 2 .
Revised: When β 2 is large, we prove that Adam can converge with any β 1 < √ β 2 .

Original: When β 2 is small, we further show that Adam might diverge to inﬁnity for a wide range of β 1 .
Revised: When β 2 is small, we further show that Adam might diverge to inﬁnity for a wide range of β 1 .

Original: One interesting question is to verify the advantage of Adam over
Revised: One interesting question is to verify the advantage of Adam over

Original: SGD.
Revised: SGD.

Original: In this work, we focus on the fundamental issue of convergence.
Revised: In this work, we focus on the fundamental issue of convergence.

Original: Proving faster convergence of
Revised: Proving faster convergence of

Original: Adam would be our future work.
Revised: Adam would be our future work.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: Yushun Zhang would like to thank Bohan Wang, Reviewer xyuf and Reviewer UR9H for the careful proof reading.
Revised: Yushun Zhang would like to thank Bohan Wang, Reviewer xyuf and Reviewer UR9H for the careful proof reading.

Original: Yushun Zhang would like to thank Bohan Wang for the valuable discussions around
Revised: Yushun Zhang would like to thank Bohan Wang for the valuable discussions around

Original: Lemma G.3.
Revised: Lemma G.3.

Original: Yushun Zhang would also like to thank Reviewer UR9H for the valuable discussions around Lemma G.13.
Revised: Yushun Zhang would also like to thank Reviewer UR9H for the valuable discussions around Lemma G.13.

Revised: Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s contributions and scope?

Original: Negative Social Impact
Revised: [Yes] (b) Did you describe the limitations of your work? [Yes] (c) Did you discuss any potential negative societal impacts of your work? [Yes] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [No] (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [No] (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [No] 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] (b) Did you mention the license of the assets?

Revised: [No] (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [No] (d) Did you discuss whether

Revised: and how consent was obtained from people whose data you’re using/curating?

Revised: [No] (e) Did you discuss whether the data you are using/curating contains personally identiﬁable information or offensive content?

Revised: [No] 5.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]

Revised: Negative Social Impact

Original: This script may provide better guidance for neural nets training.
Revised: This script may provide better guidance for neural nets training.

Original: It would have certain negative social impact if neural nets are deployed for illegal usage.
Revised: It would have certain negative social impact if neural nets are deployed for illegal usage.

Original: Appendix Organization
Revised: Appendix Organization

Original: The Appendix is organized as follows.
Revised: The Appendix is organized as follows.

Original: • Appendix A introduces two “color-ball" models and their applications in proving Lemma 5.1 and Lemma 5.2.
Revised: • Appendix A introduces two “color-ball" models and their applications in proving Lemma 5.1 and Lemma 5.2.

Original: This part is important for proving Theorem 3.1.
Revised: This part is important for proving Theorem 3.1.

Original: • Appendix B introduces more experiments to support our theory.
Revised: • Appendix B introduces more experiments to support our theory.

Original: • Appendix C provide some suggestions for hyperparameter tuning of Adam.
Revised: • Appendix C provide some suggestions for hyperparameter tuning of Adam.

Original: • Appendix D.1 provide more discussions on some recent related works.
Revised: • Appendix D.1 provide more discussions on some recent related works.

Original: Appendix D.2 re-state the non-convergence results in (Reddi et al., 2018).
Revised: Appendix D.2 re-state the non-convergence results in (Reddi et al., 2018).

Original: • Appendix E provides detailed proof for Proposition 3.3.
Revised: • Appendix E provides detailed proof for Proposition 3.3.

Original: • Appendix F provides some more notations and technical lemmas that serve for the proof of Theorem 3.1.
Revised: • Appendix F provides some more notations and technical lemmas that serve for the proof of Theorem 3.1.

Original: • Appendix G provides detailed proof for Theorem 3.1.
Revised: • Appendix G provides detailed proof for Theorem 3.1.

Original: Especially, Appendix G.1 provide a proof roadmap.
Revised: Especially, Appendix G.1 provide a proof roadmap.

Original: A Introduction to the “Color-Ball” Models: the Key Ingredients to Prove Theorem 3. 
Revised: A Introduction to the “Color-Ball” Models: the Key Ingredients to Prove Theorem 3.

Original: We now introduce two “color-ball" models and their applications in tackling Issue I and Issue II mentioned in Section 5.
Revised: We now introduce two “color-ball" models and their applications in tackling Issue I and Issue II mentioned in Section 5.

Original: These two color-ball models are important for proving Theorem 3.1.
Revised: These two color-ball models are important for proving Theorem 3.1.

Original: Solution to Issue I.
Revised: Solution to Issue I.

Original: As discussed in Section 5, we wish to show that δ ( β 1 ) vanishes with k .
Revised: As discussed in Section 5, we wish to show that δ ( β 1 ) vanishes with k .

Original: Formally, we wish to get the following equation (7) for every l ∈ [ d ] .
Revised: Formally, we wish to get the following equation (7) for every l ∈ [ d ] .

Original: [Equation]
Revised: [Equation]

Original: When k is large, O ( β nk 1 ) vanishes faster than O ( 1 √ k ) and thus Lemma 5.1 can be proved.
Revised: When k is large, O ( β nk 1 ) vanishes faster than O ( 1 √ k ) and thus Lemma 5.1 can be proved.

Original: In the following context, we will carefully quantify the mismatch between m l,k,i and the ∂ l f ( x k, 0 ) .
Revised: In the following context, we will carefully quantify the mismatch between m l,k,i and the ∂ l f ( x k, 0 ) .

Original: We ﬁnd out that the error terms from successive epochs can be cancelled, which keeps the momentum roughly in the descent direction.
Revised: We ﬁnd out that the error terms from successive epochs can be cancelled, which keeps the momentum roughly in the descent direction.

Original: To help readers understand our idea, we introduce the “color-ball " model (of the 1st kind) as follows.
Revised: To help readers understand our idea, we introduce the “color-ball " model (of the 1st kind) as follows.

Original: The color-ball model of the 1st kind. Consider a box containing two balls labeled with constant c 0 , c 1 ∈ R , respectively.
Revised: The color-ball model of the 1st kind. Consider a box containing two balls labeled with constant c 0 , c 1 ∈ R , respectively.

Original: In each round (epoch), we randomly sample balls from the box without replacement, then we put them back.
Revised: In each round (epoch), we randomly sample balls from the box without replacement, then we put them back.

Original: We denote the 1st sampled label in the k -th epoch as a k and the 2nd sampled one as b k ; a k , b k ∈ { c 0 , c 1 } .
Revised: We denote the 1st sampled label in the k -th epoch as a k and the 2nd sampled one as b k ; a k , b k ∈ { c 0 , c 1 } .

Original: We deﬁne two random variables m 0 and m 1 as follows (assume β ∈ [0 , 1) ):
Revised: We deﬁne two random variables m 0 and m 1 as follows (assume β ∈ [0 , 1) ):

Original: [Equation]
Revised: [Equation]

Original: where m 0 ,k denotes the summand of m 0 in k -th epoch, similarly for m 1 ,k .
Revised: where m 0 ,k denotes the summand of m 0 in k -th epoch, similarly for m 1 ,k .

Original: Note that in each epoch, m 0 and m 1 share the same sample order.
Revised: Note that in each epoch, m 0 and m 1 share the same sample order.

Original: Further, we introduce the following deterministic constants.
Revised: Further, we introduce the following deterministic constants.

Original: [Equation]
Revised: [Equation]

Original: where f 0 ,k denotes the summand of f 0 in k -th epoch, similarly for f 1 ,k .
Revised: where f 0 ,k denotes the summand of f 0 in k -th epoch, similarly for f 1 ,k .

Original: Now we prove the following
Revised: Now we prove the following

Original: Lemma A.1. Lemma A.1.
Revised: Lemma A.1. Lemma A.1.

Original: In the color-ball model of the 1st kind, we have
Revised: In the color-ball model of the 1st kind, we have

Original: [Equation]
Revised: [Equation]

Original: where the expectation is taken on all the possible draws.
Revised: where the expectation is taken on all the possible draws.

Original: For the color-ball example with n ≥ 2 balls, we have
Revised: For the color-ball example with n ≥ 2 balls, we have

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: i = m
Revised: i = m

Original: [Equation]
Revised: [Equation]

Original: − n − 1 (cid:88) i = f
Revised: − n − 1 (cid:88) i = f

Original: [Equation]
Revised: [Equation]

Original: How is Lemma A.1 related to (7) ?
Revised: How is Lemma A.1 related to (7) ?

Original: In this color-ball toy example, m i mimics the possible realization of momentum up to the i -th inner loop in k -th epoch.
Revised: In this color-ball toy example, m i mimics the possible realization of momentum up to the i -th inner loop in k -th epoch.

Original: f i mimics the stochastic gradient ∇ f i ( x k, 0 ) .
Revised: f i mimics the stochastic gradient ∇ f i ( x k, 0 ) .

Original: This is because we can expand ∇ f i ( x k, 0 ) into an inﬁnite-sum sequence ∇ f i ( x k, 0 ) =
Revised: This is because we can expand ∇ f i ( x k, 0 ) into an inﬁnite-sum sequence ∇ f i ( x k, 0 ) =

Original: (1 − β 1 ) ∇ f i ( x k, 0 ) (cid:80) ∞
Revised: (1 − β 1 ) ∇ f i ( x k, 0 ) (cid:80) ∞

Original: j =0 β
Revised: j =0 β

Original: j 1 , which shares a similar structure as f i .
Revised: j 1 , which shares a similar structure as f i .

Original: In this sense, Lemma A.1 may provide ideas to prove (7).
Revised: In this sense, Lemma A.1 may provide ideas to prove (7).

Original: Nevertheless, there are still gap between these two, we will explain the gap later.
Revised: Nevertheless, there are still gap between these two, we will explain the gap later.

Original: Proof.
Revised: Proof.

Original: We use E k [ · ] to denote the conditional expectation given all the history up to the beginning of k -th epoch.
Revised: We use E k [ · ] to denote the conditional expectation given all the history up to the beginning of k -th epoch.

Original: Since E [ · ] = E [ E k [ · ]] , we ﬁrst calculate E k (cid:104)(cid:80) 1 i =

Revised: Since E [ · ] = E [ E k [ · ]] , we ﬁrst calculate E k (cid:104)(cid:80) 1 i =

Original: m i (cid:105) .
Revised: m i (cid:105) .

Original: Since all the history before k -th epoch is ﬁxed, we relegate this part to later discussion and ﬁrst focus on the expectation of (cid:80)i =

Revised: Since all the history before k -th epoch is ﬁxed, we relegate this part to later discussion and ﬁrst focus on the expectation of (cid:80)i =

Original: m i,k .
Revised: m i,k .

Original: As shown in Figure 5 (upper part), there are 2 possible realization.
Revised: As shown in Figure 5 (upper part), there are 2 possible realization.

Original: With the help of Figure 5 (upper part), we have the following result.
Revised: With the help of Figure 5 (upper part), we have the following result.

Original: [Equation]
Revised: [Equation]

Original: Now we move one step further to calculate E k − 1 E k (cid:104)(cid:80) 1 i =

Revised: Now we move one step further to calculate E k − 1 E k (cid:104)(cid:80) 1 i =

Original: m i − (cid:80) 1 i =

Revised: m i − (cid:80) 1 i =

Original: f i (cid:105) .
Revised: f i (cid:105) .

Original: Using the similar strategy as (8), we have the following result.
Revised: Using the similar strategy as (8), we have the following result.

Original: The calculation is illustrated in Figure 5 (lower part) and Figure 6.
Revised: The calculation is illustrated in Figure 5 (lower part) and Figure 6.

Original: [Figure]
Revised: [Figure]

Original: (cid:88)
Revised: (cid:88)

Original: [Figure]
Revised: [Figure]

Original: (cid:88)
Revised: (cid:88)

Original: [Figure]
Revised: [Figure]

Original: (cid:35)(cid:41)
Revised: (cid:35)(cid:41)

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: We observe that only the highest order term remains in the calculation.
Revised: We observe that only the highest order term remains in the calculation.

Original: Repeat this process until k = 1 , we will get the results in Lemma A.1.
Revised: Repeat this process until k = 1 , we will get the results in Lemma A.1.

Original: The above analysis also holds for general n ≥ 2 .
Revised: The above analysis also holds for general n ≥ 2 .

Original: The gap between Lemma A.1 and Equation (7) .
Revised: The gap between Lemma A.1 and Equation (7) .

Original: Lemma A.1 shows the key idea of proving equation (7).
Revised: Lemma A.1 shows the key idea of proving equation (7).

Original: However, due to its idealized setting, the color-ball toy example is still far from our real goal (7).
Revised: However, due to its idealized setting, the color-ball toy example is still far from our real goal (7).

Original: We list some of the gaps here.
Revised: We list some of the gaps here.

Original: • In each possible trajectory: x k,i is changing with k and i , while the balls are ﬁxed in the color-ball example.
Revised: • In each possible trajectory: x k,i is changing with k and i , while the balls are ﬁxed in the color-ball example.

Original: • When taking expectation: x is changing in different trajectory, while the balls is ﬁxed in the color-ball example.
Revised: • When taking expectation: x is changing in different trajectory, while the balls is ﬁxed in the color-ball example.

Original: • x is a vector in R d while the label of the balls are constant in R .
Revised: • x is a vector in R d while the label of the balls are constant in R .

Original: It requires extra technical lemmas to handle these differences.
Revised: It requires extra technical lemmas to handle these differences.

Original: We provide more discussions in
Revised: We provide more discussions in

Original: Appendix G.1.
Revised: Appendix G.1.

Original: Solution to Issue II.
Revised: Solution to Issue II.

Original: We now discuss how to resolve Issue II mentioned in Section 5.
Revised: We now discuss how to resolve Issue II mentioned in Section 5.

Original: The solution contains two parts.
Revised: The solution contains two parts.

Original: First, we need to prove (5).
Revised: First, we need to prove (5).

Original: Second, we derive (6) from (5).
Revised: Second, we derive (6) from (5).

Original: Due to the limited space, we relegate the ﬁrst part to Appendix G.1 (related to Lemma G.2).
Revised: Due to the limited space, we relegate the ﬁrst part to Appendix G.1 (related to Lemma G.2).

Original: Now, we discuss the second part: Assume we have (5), how do we use it to prove (6)?
Revised: Now, we discuss the second part: Assume we have (5), how do we use it to prove (6)?

Original: To answer this question, we introduce the color-ball model of the 2nd kind.
Revised: To answer this question, we introduce the color-ball model of the 2nd kind.

Original: The color-ball model of the 2nd kind. Consider the same setting as the color-ball model in Step 1 .
Revised: The color-ball model of the 2nd kind. Consider the same setting as the color-ball model in Step 1 .

Original: We deﬁne a sequence of real random variable { r j } kj =1 with the following relation.
Revised: We deﬁne a sequence of real random variable { r j } kj =1 with the following relation.

Original: [Equation]
Revised: [Equation]

Original: Further
Revised: Further

Original: [Equation]
Revised: [Equation]

Original: we assume r j is ﬁxed when ﬁxing the history up to j -th round.
Revised: we assume r j is ﬁxed when ﬁxing the history up to j -th round.

Original: The sequence { r j } kj = mimics the sequence { ∂ l f ( x k, 0 ) √ v l,k, 0 } ∞ k =1 in (5).
Revised: The sequence { r j } kj = mimics the sequence { ∂ l f ( x k, 0 ) √ v l,k, 0 } ∞ k =1 in (5).

Original: Now
Revised: Now

Original: [Equation]
Revised: [Equation]

Original: we consider the following quantities.
Revised: we consider the following quantities.

Original: [Equation]
Revised: [Equation]

Original: (cid:123)(cid:122)
Revised: (cid:123)(cid:122)

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: (cid:123)(cid:122) (cid:125)
Revised: (cid:123)(cid:122) (cid:125)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) 
Revised: ) 

Original: [Equation]
Revised: [Equation]

Original: We now prove the following Lemma A.2.
Revised: We now prove the following Lemma A.2.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: Consider the color-ball model of the 2nd kind, we have
Revised: Consider the color-ball model of the 2nd kind, we have

Original: [Equation]
Revised: [Equation]

Original: For general n = 1 , 2 , 3 , · · · , we have
Revised: For general n = 1 , 2 , 3 , · · · , we have

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We only describe the proof idea here.
Revised: We only describe the proof idea here.

Original: The proof contains the following 4 steps.
Revised: The proof contains the following 4 steps.

Original: Step 2.1.
Revised: Step 2.1.

Original: We ﬁrstly take E k [ · ] and thus r k can be viewed as a constant.
Revised: We ﬁrstly take E k [ · ] and thus r k can be viewed as a constant.

Original: We use the color-ball procedure as in Figure 7 to calculate E k (cid:104) r k (cid:80) 1 i =

Revised: We use the color-ball procedure as in Figure 7 to calculate E k (cid:104) r k (cid:80) 1 i =

Original: m i,k − r k (cid:80) 1 i =

Revised: m i,k − r k (cid:80) 1 i =

Original: f i (cid:105) .
Revised: f i (cid:105) .

Original: Step 2.2.
Revised: Step 2.2.

Original: We change E k (cid:104) r k (cid:80) 1 i =

Revised: We change E k (cid:104) r k (cid:80) 1 i =

Original: m i,k − 1 (cid:105) into E k (cid:104) r k − 1 (cid:80) 1 i =

Revised: m i,k − 1 (cid:105) into E k (cid:104) r k − 1 (cid:80) 1 i =

Original: m i,k − 1 (cid:105) + Error; where Error =
Revised: m i,k − 1 (cid:105) + Error; where Error =

Original: O (1 / √ k ) .
Revised: O (1 / √ k ) .

Original: Step 2.3.
Revised: Step 2.3.

Original: We calculate E k − 1 E k (cid:104) r k − 1 (cid:80) 1 i =

Revised: We calculate E k − 1 E k (cid:104) r k − 1 (cid:80) 1 i =

Original: m i,k − 1 (cid:105) =
Revised: m i,k − 1 (cid:105) =

Original: E k − 1 (cid:104) r k − 1 (cid:80) 1 i =

Revised: E k − 1 (cid:104) r k − 1 (cid:80) 1 i =

Original: m i,k − 1 (cid:105) .
Revised: m i,k − 1 (cid:105) .

Original: This part is illustrated in the top row in Figure
Revised: This part is illustrated in the top row in Figure

Original: Step 2.4.
Revised: Step 2.4.

Original: For the leftovers in Step 2.1 , we change all r k into r k − 1 .
Revised: For the leftovers in Step 2.1 , we change all r k into r k − 1 .

Original: Then we do the cancellation to calculate E k − 1 E k (cid:104) r k − 1 (cid:80) 1 i =

Revised: Then we do the cancellation to calculate E k − 1 E k (cid:104) r k − 1 (cid:80) 1 i =

Original: m i,k − 1 + r k − 1 (cid:80) 1 i =
 m i,k − r k − 1 (cid:80) 1 i =

Revised: m i,k − 1 + r k − 1 (cid:80) 1 i =
 m i,k − r k − 1 (cid:80) 1 i =

Original: f i (cid:105) .
Revised: f i (cid:105) .

Original: This step is shown in the second and third row in Figure 8.
Revised: This step is shown in the second and third row in Figure 8.

Original: Repeat this process to the 1st epoch.
Revised: Repeat this process to the 1st epoch.

Original: We can prove the Lemma A.2.
Revised: We can prove the Lemma A.2.

Original: We emphasize that here are still some gap between Lemma A.2 and our goal in (6) in Lemma 5.2.
Revised: We emphasize that here are still some gap between Lemma A.2 and our goal in (6) in Lemma 5.2.

Original: First, we have the similar gap as discussed at the end of the Solution to Issue I .
Revised: First, we have the similar gap as discussed at the end of the Solution to Issue I .

Original: Second, the condition in Lemma 5.2 has requirement on the gradient norm, while this requirement is temporarily ignored in the color-ball method of the 2nd kind.
Revised: Second, the condition in Lemma 5.2 has requirement on the gradient norm, while this requirement is temporarily ignored in the color-ball method of the 2nd kind.

Original: It requires some technical lemmas to handle these gaps.
Revised: It requires some technical lemmas to handle these gaps.

Original: For more details, please refer to the complete proof in Appendix G.
Revised: For more details, please refer to the complete proof in Appendix G.

Original: B More experiments
Revised: B More experiments

Original: Estimation on ρ in Theorem 3.1.
Revised: Estimation on ρ in Theorem 3.1.

Original: To ensure convergence, Theorem 3.1 requires β 2 ≥ γ 1 ( n ) = 1 − O ((1 − β n 1 ) /n 2 ρ ) .
Revised: To ensure convergence, Theorem 3.1 requires β 2 ≥ γ 1 ( n ) = 1 − O ((1 − β n 1 ) /n 2 ρ ) .

Original: Now we estimate the constant ρ .
Revised: Now we estimate the constant ρ .

Original: According to our deﬁnition in Remark G. in Appendix G.1, ρ = ρ 1 ρ 2 ρ 3 , where ρ 1 , ρ 2 , ρ 3 are deﬁned as follows.
Revised: According to our deﬁnition in Remark G. in Appendix G.1, ρ = ρ 1 ρ 2 ρ 3 , where ρ 1 , ρ 2 , ρ 3 are deﬁned as follows.

Original: [Equation]
Revised: [Equation]

Original: These deﬁnitions can also be seen in Appendix F.1.
Revised: These deﬁnitions can also be seen in Appendix F.1.

Original: In worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: In worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: However, ρ is highly dependent on the problem instance f ( x ) and training process.
Revised: However, ρ is highly dependent on the problem instance f ( x ) and training process.

Original: We now estimate how ρ changes with Adam’s trajectory on MNIST and CIFAR-10.
Revised: We now estimate how ρ changes with Adam’s trajectory on MNIST and CIFAR-10.

Original: We use β 1 = 0 .
Revised: We use β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: On both datasets, we set batchsize to be 64, which brings n = 937 on MNIST and n = 781 on
Revised: On both datasets, we set batchsize to be 64, which brings n = 937 on MNIST and n = 781 on

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: CIFAR-10.
Revised: CIFAR-10.

Original: We collect ρ 1 , ρ 2 , ρ 3 along the training process and estimate their distribution density.
Revised: We collect ρ 1 , ρ 2 , ρ 3 along the training process and estimate their distribution density.

Original: The results are shown in Figure 9 and 10.
Revised: The results are shown in Figure 9 and 10.

Original: On both CIFAR-10 and MNIST, we observe that the maximal ρ 1 < 25 ≈ O ( √ n ) , ρ 2 < 

Revised: On both CIFAR-10 and MNIST, we observe that the maximal ρ 1 < 25 ≈ O ( √ n ) , ρ 2 <

Original: ≈ O ( n ) , ρ 3 < 0 .
Revised: ≈ O ( n ) , ρ 3 < 0 .

Original: 1 ≈ O (1 / √ n ) .
Revised: 1 ≈ O (1 / √ n ) .

Original: Therefore, ρ = ρ 1 ρ 2 ρ 3 ≈ O ( n ) .
Revised: Therefore, ρ = ρ 1 ρ 2 ρ 3 ≈ O ( n ) .

Original: Batchsize and β 2 .
Revised: Batchsize and β 2 .

Original: As shown in Figure 11: on MNIST, smaller batchsize requires larger β 2 to reach small loss.
Revised: As shown in Figure 11: on MNIST, smaller batchsize requires larger β 2 to reach small loss.

Original: Since batchsize equals to (number of total sample)/(number of batches).
Revised: Since batchsize equals to (number of total sample)/(number of batches).

Original: In the context of ﬁnite-sum setting with n summand, n usually stands for the number of batches (e.g., In the extreme case when batchsize = 1 , n equals to the number of total samples).
Revised: In the context of ﬁnite-sum setting with n summand, n usually stands for the number of batches (e.g., In the extreme case when batchsize = 1 , n equals to the number of total samples).

Original: Therefore, smaller batchsize brings larger n .
Revised: Therefore, smaller batchsize brings larger n .

Original: As such, Figure 11 matches the message by Theorem 3.1: the threshold of β 2 increases with n .
Revised: As such, Figure 11 matches the message by Theorem 3.1: the threshold of β 2 increases with n .

Original: [Figure]
Revised: [Figure]

Original: More experiments on function (2) .
Revised: More experiments on function (2) .

Original: Figure 1 (a) shows the optimality gap after 50k iterations when n = 10 and initialization x = 1 .
Revised: Figure 1 (a) shows the optimality gap after 50k iterations when n = 10 and initialization x = 1 .

Original: Here, we provide more relevant experiments.
Revised: Here, we provide more relevant experiments.

Original: First, when initialized at x = 1 , we run experiments with n = 5 , 10 , 15 , 20 .
Revised: First, when initialized at x = 1 , we run experiments with n = 5 , 10 , 15 , 20 .

Original: The results are shown in Figure 12.
Revised: The results are shown in Figure 12.

Original: We observe that the blue region shrinks as n increases.
Revised: We observe that the blue region shrinks as n increases.

Original: This matches our conclusion in Theorem 3.1: when n increases, the convergent threshold of β 2 increases, which means we need larger β 2 to ensure convergence.
Revised: This matches our conclusion in Theorem 3.1: when n increases, the convergent threshold of β 2 increases, which means we need larger β 2 to ensure convergence.

Original: It also matches the conclusion in Theorem 3.3: when n increases, the divergence region will expand (more evidence can be seen in Appendix E).
Revised: It also matches the conclusion in Theorem 3.3: when n increases, the divergence region will expand (more evidence can be seen in Appendix E).

Original: When initializing at x = − 5 , we further demonstrate that the gradient norm of f ( x ) can dramatically increase.
Revised: When initializing at x = − 5 , we further demonstrate that the gradient norm of f ( x ) can dramatically increase.

Original: All the setting is the same as that in Figure 12 except for the change of initialization.
Revised: All the setting is the same as that in Figure 12 except for the change of initialization.

Original: [Figure]
Revised: [Figure]

Original: Starting at x = −
Revised: Starting at x = −

Original: [Figure]
Revised: [Figure]

Original: the algorithm can touch the “quadratic" side of function (2) where the gradient is unbounded.
Revised: the algorithm can touch the “quadratic" side of function (2) where the gradient is unbounded.

Original: The results are shown in Figure 13.
Revised: The results are shown in Figure 13.

Original: We observe a similar pattern as that in Figure 12.
Revised: We observe a similar pattern as that in Figure 12.

Original: As a result
Revised: As a result

Original: [Figure]
Revised: [Figure]

Original: the gradient norm of f ( x ) is large in the left bottom corner.
Revised: the gradient norm of f ( x ) is large in the left bottom corner.

Original: We further plot the change of gradient norm along the iterations.
Revised: We further plot the change of gradient norm along the iterations.

Original: We pick β 1 = 0 .
Revised: We pick β 1 = 0 .

Original: 1 and β 2 = 0 , 0 . 1 , 0 . 3 , 0 . 5 , 0 .
Revised: 1 and β 2 = 0 , 0 . 1 , 0 . 3 , 0 . 5 , 0 .

Original: 99 to see the phase transition when increasing β 2 .
Revised: 99 to see the phase transition when increasing β 2 .

Original: The result is shown in
Revised: The result is shown in

Original: Figure 14.
Revised: Figure 14.

Original: When β 2 is small, the gradient norm of f ( x ) increases rapidly along the iteration.
Revised: When β 2 is small, the gradient norm of f ( x ) increases rapidly along the iteration.

Original: Most of them are even much larger than the upper bound of color bar in Figure 13.
Revised: Most of them are even much larger than the upper bound of color bar in Figure 13.

Original: As a result, there is a phase transition from diverge to converge when increasing β 2 from 0 to
Revised: As a result, there is a phase transition from diverge to converge when increasing β 2 from 0 to

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: Adam converges to a exact critical point when D 0 = 0 .
Revised: Adam converges to a exact critical point when D 0 = 0 .

Original: Since function (2) satisﬁes D 0 = 0 , we further provide empirical evidence that Adam converges to a exact critical point when D 0 = 0 .
Revised: Since function (2) satisﬁes D 0 = 0 , we further provide empirical evidence that Adam converges to a exact critical point when D 0 = 0 .

Original: We use function (2) n = 20 and initialization x = − 5 .
Revised: We use function (2) n = 20 and initialization x = − 5 .

Original: We choose some large enough β 2 to ensure the convergence.
Revised: We choose some large enough β 2 to ensure the convergence.

Original: As shown in Figure 15, We observe 0 gradient norm after Adam converges.
Revised: As shown in Figure 15, We observe 0 gradient norm after Adam converges.

Original: All the hyperparameter setting is the same as before.
Revised: All the hyperparameter setting is the same as before.

Original: Adam converges to a bounded region when D 0 > 0 .
Revised: Adam converges to a bounded region when D 0 > 0 .

Original: Now we show that Adam may not converge to an exact critical point when D 0 > 0 .
Revised: Now we show that Adam may not converge to an exact critical point when D 0 > 0 .

Original: Instead, it converges to a bounded region near the critical point.
Revised: Instead, it converges to a bounded region near the critical point.

Original: For this part, we re-state the example from (Shi et al., 2020).
Revised: For this part, we re-state the example from (Shi et al., 2020).

Original: Consider the following function.
Revised: Consider the following function.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: Summing up f j ( x ) we get
Revised: Summing up f j ( x ) we get

Original: [Equation]
Revised: [Equation]

Original: This f ( x ) is lower bounded by − a 2 9 with optimal solution x ∗ = 0 .
Revised: This f ( x ) is lower bounded by − a 2 9 with optimal solution x ∗ = 0 .

Original: At the optimal point x ∗ = 0 , ∇ f j ( x ∗ ) (cid:54) = 

Revised: At the optimal point x ∗ = 0 , ∇ f j ( x ∗ ) (cid:54) =

Original: so we have D 0 > 0 .
Revised: so we have D 0 > 0 .

Original: When running Algorithm 1 on this function, we observe that
Revised: When running Algorithm 1 on this function, we observe that

Original: Adam with diminishing stepsize does not converge to 0 gradient norm.
Revised: Adam with diminishing stepsize does not converge to 0 gradient norm.

Original: Instead, it converge to a bounded region.
Revised: Instead, it converge to a bounded region.

Original: Further, the size of the region shrinks when β 2 increases.
Revised: Further, the size of the region shrinks when β 2 increases.

Original: These phenomena matches our discussion in Remark 3.
Revised: These phenomena matches our discussion in Remark 3.

Original: The result is shown in Figure 4 (a
Revised: The result is shown in Figure 4 (a

Original: In the experiment, we choose
Revised: In the experiment, we choose

Original: B.1 Experimental Settings
Revised: B.1 Experimental Settings

Original: Here, we introduce our experimental settings.
Revised: Here, we introduce our experimental settings.

Original: • Experiments on function (2) .
Revised: • Experiments on function (2) .

Original: We use Algorithm 1 with cyclic order f 0 , f 1 , f 2 and so on.
Revised: We use Algorithm 1 with cyclic order f 0 , f 1 , f 2 and so on.

Original: We report the optimality gap x − x ∗ after 50k iteration, or equivalently 50000 /n epochs.
Revised: We report the optimality gap x − x ∗ after 50k iteration, or equivalently 50000 /n epochs.

Original: We use (cid:15) =
Revised: We use (cid:15) =

Original: 10 − 8 for numerical stability.
Revised: 10 − 8 for numerical stability.

Original: We use diminishing stepsize η k = 0 .
Revised: We use diminishing stepsize η k = 0 .

Original: 1 / √ k , where k is the index of epoch.
Revised: 1 / √ k , where k is the index of epoch.

Original: Unless otherwise stated, this setting applies to all the other experiments on function (2).
Revised: Unless otherwise stated, this setting applies to all the other experiments on function (2).

Original: In Figure 1 (a)
Revised: In Figure 1 (a)

Original: we use n = 10 and initialization x = 1 .
Revised: we use n = 10 and initialization x = 1 .

Original: We will report more results with different n and different initialization in Appendix B. • MNIST (Deng, 2012).
Revised: We will report more results with different n and different initialization in Appendix B. • MNIST (Deng, 2012).

Original: We use one-hidden-layer neural network with width =16.
Revised: We use one-hidden-layer neural network with width =16.

Original: We set batchsize =1, weight decay =0, stepsize =0.0001 and train for 20 epochs.
Revised: We set batchsize =1, weight decay =0, stepsize =0.0001 and train for 20 epochs.

Original: We use (cid:15) =
Revised: We use (cid:15) =

Original: 10 − 8 for numerical stability.
Revised: 10 − 8 for numerical stability.

Original: • CIFAR-10 (Krizhevsky et al., 2009).
Revised: • CIFAR-10 (Krizhevsky et al., 2009).

Original: We use ResNet-18 (He et al., 2016) as the architecture.
Revised: We use ResNet-18 (He et al., 2016) as the architecture.

Original: We choose batchsize =16, weight decay =5e-4 , initial stepsize=1e-3.
Revised: We choose batchsize =16, weight decay =5e-4 , initial stepsize=1e-3.

Original: We use a stage-wise constant learning rate scheduling with a multiplicative factor of 0.1 on epoch 30, 60 and 90.
Revised: We use a stage-wise constant learning rate scheduling with a multiplicative factor of 0.1 on epoch 30, 60 and 90.

Original: We use (cid:15) =
Revised: We use (cid:15) =

Original: 10 − 8 for numerical stability.
Revised: 10 − 8 for numerical stability.

Original: For MNIST and CIFAR-10, larger batchsize will bring similar pattern as that in Figure 1, but the phase transition will occur at some smaller β 2 .
Revised: For MNIST and CIFAR-10, larger batchsize will bring similar pattern as that in Figure 1, but the phase transition will occur at some smaller β 2 .

Original: • NLP.
Revised: • NLP.

Original: The WikiText-103 dataset is a collection of over 100 million tokens extracted from the set of veriﬁed ‘Good’ and ‘Featured’ articles on Wikipedia.
Revised: The WikiText-103 dataset is a collection of over 100 million tokens extracted from the set of veriﬁed ‘Good’ and ‘Featured’ articles on Wikipedia.

Original: The base model of Transformer XL contains 16 self-attention layers.
Revised: The base model of Transformer XL contains 16 self-attention layers.

Original: In each self-attention layer, there are 10 heads.
Revised: In each self-attention layer, there are 10 heads.

Original: The encoding dimension of each head is set to be 41.
Revised: The encoding dimension of each head is set to be 41.

Original: We set batchsize = 60, number of iteration =
Revised: We set batchsize = 60, number of iteration =

Original: 200k, and initial stepsize = 0.00025.
Revised: 200k, and initial stepsize = 0.00025.

Original: We use cosine learning rate scheduler, which is a popular choice for training Transformer.
Revised: We use cosine learning rate scheduler, which is a popular choice for training Transformer.

Original: We use (cid:15) =
Revised: We use (cid:15) =

Original: 10 − 8 for numerical stability.
Revised: 10 − 8 for numerical stability.

Original: C Some Potential Implications for Practical Users
Revised: C Some Potential Implications for Practical Users

Original: Theorem 3.1 and Proposition 3.3 establish a clearer image on the relation between ( β 1 , β 2 ) and qualitative behavior of Adam, which may have certain implication for practical users.
Revised: Theorem 3.1 and Proposition 3.3 establish a clearer image on the relation between ( β 1 , β 2 ) and qualitative behavior of Adam, which may have certain implication for practical users.

Original: Many practitioners are still doing grid or random search over ( β 1 , β 2 ) , which could be costly.
Revised: Many practitioners are still doing grid or random search over ( β 1 , β 2 ) , which could be costly.

Original: The following advice may cut down a large portion of search space.
Revised: The following advice may cut down a large portion of search space.

Original: Suppose we start with some random point ( β ∗ 1 , β ∗ 2 ) , we provide the following suggestions for tuning β 1 and β 2 .
Revised: Suppose we start with some random point ( β ∗ 1 , β ∗ 2 ) , we provide the following suggestions for tuning β 1 and β 2 .

Original: Case 1: If ( β ∗ 1 , β ∗ 2 ) lies above the blue curve in Figure 2.
Revised: Case 1: If ( β ∗ 1 , β ∗ 2 ) lies above the blue curve in Figure 2.

Original: We point out two sub-cases.
Revised: We point out two sub-cases.

Original: First, if
Revised: First, if

Original: Adam with ( β ∗ 1 , β ∗ 2 ) diverges, then any points with “ β 1 < √ β 2 " and “ β 2 < β ∗ 2 " shall not be tried due to the risk of divergence.
Revised: Adam with ( β ∗ 1 , β ∗ 2 ) diverges, then any points with “ β 1 < √ β 2 " and “ β 2 < β ∗ 2 " shall not be tried due to the risk of divergence.

Original: Second, if Adam with ( β ∗ 1 , β ∗ 2 ) converges, then any points above this point will converge.
Revised: Second, if Adam with ( β ∗ 1 , β ∗ 2 ) converges, then any points above this point will converge.

Original: You may either ﬁx β 1 = β ∗ 1 and increase β 2 , or ﬁx β 2 = β ∗ 2 and try smaller β 1 .
Revised: You may either ﬁx β 1 = β ∗ 1 and increase β 2 , or ﬁx β 2 = β ∗ 2 and try smaller β 1 .

Original: Both ways have convergence guarantee.
Revised: Both ways have convergence guarantee.

Original: Case 2: If ( β ∗ 1 , β ∗ 2 ) lies below the blue curve in Figure 2.
Revised: Case 2: If ( β ∗ 1 , β ∗ 2 ) lies below the blue curve in Figure 2.

Original: We still discuss two sub-cases.
Revised: We still discuss two sub-cases.

Original: First, if you observe divergence at ( β ∗ 1 , β ∗ 2 ) , then do not further try any point on the left.
Revised: First, if you observe divergence at ( β ∗ 1 , β ∗ 2 ) , then do not further try any point on the left.

Original: We do not suggest exploring the points in below either, since the majority of them will still face the risk of divergence.
Revised: We do not suggest exploring the points in below either, since the majority of them will still face the risk of divergence.

Original: Instead, we suggest ﬁx β 1 = β ∗ 1 and increase β 2 .
Revised: Instead, we suggest ﬁx β 1 = β ∗ 1 and increase β 2 .

Original: Since our Theorem 3.1 applies for any β 1 < √ β 2 , algorithm will converge when β 2 is large enough.
Revised: Since our Theorem 3.1 applies for any β 1 < √ β 2 , algorithm will converge when β 2 is large enough.

Original: Second, if you observe convergence at ( β ∗ 1 , β ∗ 2 ) (which is also possible according to Figure 1 (a)), then we suggest: (i) either ﬁx β 1 = β ∗ 1 and increase β 2 , or (ii) ﬁx β 2 = β ∗ 2 and try smaller β 1 .
Revised: Second, if you observe convergence at ( β ∗ 1 , β ∗ 2 ) (which is also possible according to Figure 1 (a)), then we suggest: (i) either ﬁx β 1 = β ∗ 1 and increase β 2 , or (ii) ﬁx β 2 = β ∗ 2 and try smaller β 1 .

Original: Both ways push ( β 1 , β 2 ) into the region of Theorem 3.1 with convergence guarantee.
Revised: Both ways push ( β 1 , β 2 ) into the region of Theorem 3.1 with convergence guarantee.

Original: D More Discussion on Related Works. D.1 More Related Works on the Convergence Analysis of Adam Family
Revised: D More Discussion on Related Works. D.1 More Related Works on the Convergence Analysis of Adam Family

Original: New variants of Adam.
Revised: New variants of Adam.

Original: Ever since Reddi et al.
Revised: Ever since Reddi et al.

Original: (2018) pointed out the non-convergence issue of
Revised: (2018) pointed out the non-convergence issue of

Original: Adam, one active line of work has tried to design new variants of Adam that can be proved to converge.
Revised: Adam, one active line of work has tried to design new variants of Adam that can be proved to converge.

Original: For instance, Zou et al.
Revised: For instance, Zou et al.

Original: (2019); Gadat & Gavra (2020)
Revised: (2019); Gadat & Gavra (2020)

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2018b, 2021) replace the constant hyperparameters by iterate-dependent ones e.g. β 1 t or β 2 t .
Revised: (2018b, 2021) replace the constant hyperparameters by iterate-dependent ones e.g. β 1 t or β 2 t .

Original: AMSGrad (Reddi et al., 2018) and
Revised: AMSGrad (Reddi et al., 2018) and

Original: AdaFom (Chen et al., 2018b) modify { v t } to be an non-decreasing sequence.
Revised: AdaFom (Chen et al., 2018b) modify { v t } to be an non-decreasing sequence.

Original: Similarly, AdaBound (Luo et al., 2019) impose lower and upper bounds on { v t } to prevent the effective stepsize from vanishing or exploding.
Revised: Similarly, AdaBound (Luo et al., 2019) impose lower and upper bounds on { v t } to prevent the effective stepsize from vanishing or exploding.

Original: Zhou et al.
Revised: Zhou et al.

Original: (2018b) also adopt a new estimate of v t to correct the bias.
Revised: (2018b) also adopt a new estimate of v t to correct the bias.

Original: There are also attempts to combine Adam with Nesterov momentum (Dozat, 2016) as well as warm-up techniques (Liu et al., 2020a).
Revised: There are also attempts to combine Adam with Nesterov momentum (Dozat, 2016) as well as warm-up techniques (Liu et al., 2020a).

Original: Padam (Chen et al., 2018a) also introduce a partial adaptive parameter to improve the generalization performance.
Revised: Padam (Chen et al., 2018a) also introduce a partial adaptive parameter to improve the generalization performance.

Original: There are also some works providing theoretical analysis on the variants of Adam.
Revised: There are also some works providing theoretical analysis on the variants of Adam.

Original: For instance, Zhou et al.
Revised: For instance, Zhou et al.

Original: (2018a) study the convergence of AdaGrad and
Revised: (2018a) study the convergence of AdaGrad and

Original: AMSGrad under bounded gradient condition.
Revised: AMSGrad under bounded gradient condition.

Original: Gadat & Gavra (2020) study the asymptotic behavior of a subclass of adaptive gradient methods from landscape point of view.
Revised: Gadat & Gavra (2020) study the asymptotic behavior of a subclass of adaptive gradient methods from landscape point of view.

Original: Their analysis applies to Adam-variants with β 1 = 0 and β 2 increasing along the iterates (it could also be understood as
Revised: Their analysis applies to Adam-variants with β 1 = 0 and β 2 increasing along the iterates (it could also be understood as

Original: RMSProp with increasing β 2 ).
Revised: RMSProp with increasing β 2 ).

Original: When this script is under review, a new work (Iiduka, 2022) appear on line.
Revised: When this script is under review, a new work (Iiduka, 2022) appear on line.

Original: Iiduka (2022) analyze the convergence of AMSGrad by relaxing the Lipschitz-gradient condition.
Revised: Iiduka (2022) analyze the convergence of AMSGrad by relaxing the Lipschitz-gradient condition.

Original: However, their analysis requires extra conditions on both bounded gradient and bounded domain.
Revised: However, their analysis requires extra conditions on both bounded gradient and bounded domain.

Original: Some more discussions on (Guo et al., 2021) and (Huang et al., 2021).
Revised: Some more discussions on (Guo et al., 2021) and (Huang et al., 2021).

Original: Here, we discuss more on two recent works Guo et al.
Revised: Here, we discuss more on two recent works Guo et al.

Original: and Huang et al.
Revised: and Huang et al.

Original: As mentioned in Section 2.2, they both require some extra conditions.
Revised: As mentioned in Section 2.2, they both require some extra conditions.

Original: First, both Guo et al.
Revised: First, both Guo et al.

Original: and Huang et al.
Revised: and Huang et al.

Original: (2021) requires bounded gradient assumption.
Revised: (2021) requires bounded gradient assumption.

Original: This can be seen in Assumption 2 in (Guo et al., 2021).
Revised: This can be seen in Assumption 2 in (Guo et al., 2021).

Original: In (Huang et al., 2021), they require bounded iterates ( their Theorem 1) or change Adam into AdaBound (Luo et al., 2019) by clipping (their Remark 2, Corollary 1).
Revised: In (Huang et al., 2021), they require bounded iterates ( their Theorem 1) or change Adam into AdaBound (Luo et al., 2019) by clipping (their Remark 2, Corollary 1).

Original: Both settings inherent boundedness on gradient.
Revised: Both settings inherent boundedness on gradient.

Original: Besides bounded gradient, both (Huang et al., 2021) and (Guo et al., 2021) requires 1 / ( √ v t + (cid:15) ) ≤ C u .
Revised: Besides bounded gradient, both (Huang et al., 2021) and (Guo et al., 2021) requires 1 / ( √ v t + (cid:15) ) ≤ C u .

Original: This condition is stated in Assumption 2 in (Guo et al., 2021) and Assumption 3 in (Huang et al., 2021) (they presented it as H t (cid:23)
Revised: This condition is stated in Assumption 2 in (Guo et al., 2021) and Assumption 3 in (Huang et al., 2021) (they presented it as H t (cid:23)

Original: ρI
Revised: ρI

Original: d (cid:31) 0 , where matrix H t = diag ( √ v t + (cid:15) ) ).
Revised: d (cid:31) 0 , where matrix H t = diag ( √ v t + (cid:15) ) ).

Original: Combining these two conditions, the effective stepsize of Adam will be bounded in certain interval 1 √ v t + (cid:15) ∈
Revised: Combining these two conditions, the effective stepsize of Adam will be bounded in certain interval 1 √ v t + (cid:15) ∈

Original: [ C l , C u ] .
Revised: [ C l , C u ] .

Original: Such boundedness condition changes Adam into AdaBound (Luo et al., 2019) and thus cannot explain the observations of Adam in Section 1.
Revised: Such boundedness condition changes Adam into AdaBound (Luo et al., 2019) and thus cannot explain the observations of Adam in Section 1.

Original: D.2 A Brief Introduction to (Reddi et al., 2018)
Revised: D.2 A Brief Introduction to (Reddi et al., 2018)

Original: Here, we re-state two counter examples by (Reddi et al., 2018).
Revised: Here, we re-state two counter examples by (Reddi et al., 2018).

Original: For the consistence of notation, we will re-state their results under our notation in the full script.
Revised: For the consistence of notation, we will re-state their results under our notation in the full script.

Original: They consider the convex problem ((Reddi et al., 2018)): min (cid:80) n − 1 i =

Revised: They consider the convex problem ((Reddi et al., 2018)): min (cid:80) n − 1 i =

Original: f i ( x ) where x ∈ [ − 1 , 1] , n ≥ 3 : f i ( x ) = (cid:26) nx, for i =− x, otherwise, (10)
Revised: f i ( x ) where x ∈ [ − 1 , 1] , n ≥ 3 : f i ( x ) = (cid:26) nx, for i =− x, otherwise, (10)

Original: Note that (10) satisfy both Assumption 2.1 and 2.2 (with D 1 = n 2 + n − 1 and D 0 = 0 ), so our assumptions do not rule out this counter-example a priori.
Revised: Note that (10) satisfy both Assumption 2.1 and 2.2 (with D 1 = n 2 + n − 1 and D 0 = 0 ), so our assumptions do not rule out this counter-example a priori.

Original: This is a constrained problem with feasible set x ∈ [ − 1 , 1] , the optimal solution is x ∗ = − 1 .
Revised: This is a constrained problem with feasible set x ∈ [ − 1 , 1] , the optimal solution is x ∗ = − 1 .

Original: Since they consider constrained problems, their claimed “divergence" actually means the iterates will stay in a huge region with the size of whole feasible set.
Revised: Since they consider constrained problems, their claimed “divergence" actually means the iterates will stay in a huge region with the size of whole feasible set.

Original: Here, we call it “non-convergence" to distinguish from our result of “diverge to inﬁnity" in Proposition 3.3.
Revised: Here, we call it “non-convergence" to distinguish from our result of “diverge to inﬁnity" in Proposition 3.3.

Original: They consider sampling f i in the cyclic order: f 0 , f 1 , f 2 .
Revised: They consider sampling f i in the cyclic order: f 0 , f 1 , f 2 .

Original: In (Reddi et al., 2018), Function (10) is presented as an “online optimization problem with non-zero average regret".
Revised: In (Reddi et al., 2018), Function (10) is presented as an “online optimization problem with non-zero average regret".

Original: We choose to use the form of (10) since it is more consistent with our notation in Algorithm 1.
Revised: We choose to use the form of (10) since it is more consistent with our notation in Algorithm 1.

Original: We re-state their results as follows.
Revised: We re-state their results as follows.

Original: Theorem D.1 (Theorem 2 in (Reddi et al., 2018)) .
Revised: Theorem D.1 (Theorem 2 in (Reddi et al., 2018)) .

Original: For any ﬁxed ( β 1 , β 2 ) s.t. β 1 < √ β 2 , there exists function (10) with large enough n , s.t.
Revised: For any ﬁxed ( β 1 , β 2 ) s.t. β 1 < √ β 2 , there exists function (10) with large enough n , s.t.

Original: Adam will converge to highly sub-optimal solution x = 1 .
Revised: Adam will converge to highly sub-optimal solution x = 1 .

Original: We brieﬂy re-state the non-convergent condition for this Theorem.
Revised: We brieﬂy re-state the non-convergent condition for this Theorem.

Original: As stated in Equation (7), Appendix
Revised: As stated in Equation (7), Appendix

Original: B in (Reddi et al., 2018), for every ﬁxed ( β 1 , β 2 ) , they need a “constant n that depends on β 1 and β 2 ".
Revised: B in (Reddi et al., 2018), for every ﬁxed ( β 1 , β 2 ) , they need a “constant n that depends on β 1 and β 2 ".

Original: As such, they require different n to cause non-convergence on different ( β 1 , β 2 ) .
Revised: As such, they require different n to cause non-convergence on different ( β 1 , β 2 ) .

Original: So the considered function class is constantly changing.
Revised: So the considered function class is constantly changing.

Original: For completeness, we further re-state Theorem 1 in (Reddi et al., 2018).
Revised: For completeness, we further re-state Theorem 1 in (Reddi et al., 2018).

Original: Theorem D.2 (Theorem 1 in (Reddi et al., 2018)) .
Revised: Theorem D.2 (Theorem 1 in (Reddi et al., 2018)) .

Original: For function (10) , when β 1 = 0 and β 2 = 1 / ( n 2 + 1) , Adam will converge to highly sub-optimal solution x = 1 .
Revised: For function (10) , when β 1 = 0 and β 2 = 1 / ( n 2 + 1) , Adam will converge to highly sub-optimal solution x = 1 .

Original: This theorem considers choosing ( β 1 , β 2 ) after n .
Revised: This theorem considers choosing ( β 1 , β 2 ) after n .

Original: However, this result only show non-convergence on a single point ( β 1 , β 2 ) =
Revised: However, this result only show non-convergence on a single point ( β 1 , β 2 ) =

Original: (0 , 1 / ( n 2 + 1)) .
Revised: (0 , 1 / ( n 2 + 1)) .

Original: This point lies somewhere on the left boundary of Figure 3.
Revised: This point lies somewhere on the left boundary of Figure 3.

Original: It seems unclear how Adam’s behavior would change as we change the ( β 1 , β 2 ) to anywhere else.
Revised: It seems unclear how Adam’s behavior would change as we change the ( β 1 , β 2 ) to anywhere else.

Original: E Proof of Proposition 3. 
Revised: E Proof of Proposition 3.

Original: We restate our counter-example here.
Revised: We restate our counter-example here.

Original: Consider f ( x ) = (cid:80) n − 1 i =

Revised: Consider f ( x ) = (cid:80) n − 1 i =

Original: f i ( x ) for x ∈ R , we deﬁne f i ( x ) as follows:
Revised: f i ( x ) for x ∈ R , we deﬁne f i ( x ) as follows:

Original: [Equation]
Revised: [Equation]

Original: Summing up all the f i ( x ) , we can see that
Revised: Summing up all the f i ( x ) , we can see that

Original: [Equation]
Revised: [Equation]

Original: is a convex smooth function with optimal solution x ∗ = − 2 and optimal value f ( x ∗ ) = − 3 / 2 .
Revised: is a convex smooth function with optimal solution x ∗ = − 2 and optimal value f ( x ∗ ) = − 3 / 2 .

Original: However, we are going to show that, for any ﬁxed n > 2 , there exists an orange region shown in
Revised: However, we are going to show that, for any ﬁxed n > 2 , there exists an orange region shown in

Original: Figure 2, s.t., Adam with any ( β 1 , β 2 ) combination in the yellow region diverge to x = ∞ rather than the optimal solution x = − 2 , causing the divergence.
Revised: Figure 2, s.t., Adam with any ( β 1 , β 2 ) combination in the yellow region diverge to x = ∞ rather than the optimal solution x = − 2 , causing the divergence.

Original: Now we introduce the formal statement of
Revised: Now we introduce the formal statement of

Original: Proposition 3.3.
Revised: Proposition 3.3.

Original: Proposition E.1.
Revised: Proposition E.1.

Original: (Formal statement of Proposition 3.3.) Consider the convex function (11) for a ﬁxed n .
Revised: (Formal statement of Proposition 3.3.) Consider the convex function (11) for a ﬁxed n .

Original: Starting at the initialization x = 1 and initial stepsize η 1 , the iterates of Adam diverge to inﬁnity if the following holds:
Revised: Starting at the initialization x = 1 and initial stepsize η 1 , the iterates of Adam diverge to inﬁnity if the following holds:

Original: [Equation]
Revised: [Equation]

Original: The analysis is motivated by that in (Reddi et al., 2018, Theorem 1).
Revised: The analysis is motivated by that in (Reddi et al., 2018, Theorem 1).

Original: However, (Reddi et al., 2018,
Revised: However, (Reddi et al., 2018,

Original: Theorem 1) considers a simpliﬁed case with β 1 = 0 .
Revised: Theorem 1) considers a simpliﬁed case with β 1 = 0 .

Original: Here, we consider non-zero β 1 , especially for those β 1 > √ β 2 .
Revised: Here, we consider non-zero β 1 , especially for those β 1 > √ β 2 .

Original: To show the divergence, we aim to prove the following claim: (we denote x k,i as the value of x at the k -th outer loop and i -th inner loop )
Revised: To show the divergence, we aim to prove the following claim: (we denote x k,i as the value of x at the k -th outer loop and i -th inner loop )

Original: Claim: for any ﬁxed n > 2 , there exists an orange region shown in Figure 2 s.t., Adam with any β 1 - β 2 combination in the orange region gives x k +1 , 0 > 1 as long as x k, 0 = 1 .
Revised: Claim: for any ﬁxed n > 2 , there exists an orange region shown in Figure 2 s.t., Adam with any β 1 - β 2 combination in the orange region gives x k +1 , 0 > 1 as long as x k, 0 = 1 .

Original: Since the gradient stays constant when x > 1 , so x will go to inﬁnity if the claim holds.
Revised: Since the gradient stays constant when x > 1 , so x will go to inﬁnity if the claim holds.

Original: To prove this claim, we only need to analyze the trajectory of Adam within one particular outer loop, e.g., the k -th outer loop.
Revised: To prove this claim, we only need to analyze the trajectory of Adam within one particular outer loop, e.g., the k -th outer loop.

Original: We will show that x k +1 , 0 > 1 if this outer loop is initialized with x k, 0 = 1 .
Revised: We will show that x k +1 , 0 > 1 if this outer loop is initialized with x k, 0 = 1 .

Original: Similarly as (Reddi et al., 2018), we assume f i ( x ) are sampled in the order of f 0 ( x ) , f 1 ( x ) , · · · , f n − 1 ( x ) within the k -th outer loop.
Revised: Similarly as (Reddi et al., 2018), we assume f i ( x ) are sampled in the order of f 0 ( x ) , f 1 ( x ) , · · · , f n − 1 ( x ) within the k -th outer loop.

Original: Now let us prove the claim.
Revised: Now let us prove the claim.

Original: For function (11), the update rule of Adam is shown as follows.
Revised: For function (11), the update rule of Adam is shown as follows.

Original: [Equation]
Revised: [Equation]

Original: where δ k,
Revised: where δ k,

Original: [Equation]
Revised: [Equation]

Original: We decompose the total movement (cid:80) n − 1 i =

Revised: We decompose the total movement (cid:80) n − 1 i =

Original: δ k,i into three terms as follows.
Revised: δ k,i into three terms as follows.

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: i =
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: We will show that for some β 1 and β 2 : ( I ) , ( II ) > 0 and ( III ) < 0 .
Revised: We will show that for some β 1 and β 2 : ( I ) , ( II ) > 0 and ( III ) < 0 .

Original: However, ( I ) and ( II ) outweigh ( III ) , causing the divergence.
Revised: However, ( I ) and ( II ) outweigh ( III ) , causing the divergence.

Original: First, we show that m k − 1 ,n − 1 < 0 when β 1 is small.
Revised: First, we show that m k − 1 ,n − 1 < 0 when β 1 is small.

Original: [Equation]
Revised: [Equation]

Original: when β 1 is small,we have (1 − β n − 1) > (1 − β 1 ) β n − 1n , which implies − m k − 1 ,n − 1 > 0 .
Revised: when β 1 is small,we have (1 − β n − 1) > (1 − β 1 ) β n − 1n , which implies − m k − 1 ,n − 1 > 0 .

Original: For these choices of β 1 , we have ( I ) > 0 .
Revised: For these choices of β 1 , we have ( I ) > 0 .

Original: Now we derive a lower bound for ( II ) .
Revised: Now we derive a lower bound for ( II ) .

Original: [Equation]
Revised: [Equation]

Original: The inequality is due to the fact that v k, 0 ≤ n 2 .
Revised: The inequality is due to the fact that v k, 0 ≤ n 2 .

Original: Since β j 2 n 2 is small when β 2 is small and j is close to n , there exists some small β 2 such that β j 2 n 2 ≤ 0 .
Revised: Since β j 2 n 2 is small when β 2 is small and j is close to n , there exists some small β 2 such that β j 2 n 2 ≤ 0 .

Original: 1 for at least one j < n .
Revised: 1 for at least one j < n .

Original: For these small enough β 2 , we keep the summand with j ≥ log β 2 (0 .
Revised: For these small enough β 2 , we keep the summand with j ≥ log β 2 (0 .

Original: 1 /n 2 ) and drop the rest.
Revised: 1 /n 2 ) and drop the rest.

Original: We have the following lower bound for ( II ) .
Revised: We have the following lower bound for ( II ) .

Original: [Equation]
Revised: [Equation]

Original: However, this lower bound only holds for the small β 2 .
Revised: However, this lower bound only holds for the small β 2 .

Original: With simple modiﬁcation, we derive a universal lower bound of ( II ) for any β 2 ∈ (0 , 1) .
Revised: With simple modiﬁcation, we derive a universal lower bound of ( II ) for any β 2 ∈ (0 , 1) .

Original: [Equation]
Revised: [Equation]

Original: Now we derive a lower bound for ( III ) .
Revised: Now we derive a lower bound for ( III ) .

Original: [Equation]
Revised: [Equation]

Original: The remaining step is to show for small enough step size η 1 , the iterates will stay in the linear region, thus the above gradient holds for all iterates in the trajectory.
Revised: The remaining step is to show for small enough step size η 1 , the iterates will stay in the linear region, thus the above gradient holds for all iterates in the trajectory.

Original: As we initial x as x 0 = 1 , if for all m < n and k > 0 , (cid:80) mi =0 δ k,i ≥ − 2 , we can conclude all iterates in the trajectory stay in the linear region.
Revised: As we initial x as x 0 = 1 , if for all m < n and k > 0 , (cid:80) mi =0 δ k,i ≥ − 2 , we can conclude all iterates in the trajectory stay in the linear region.

Original: Because it holds that m k − 1 ,n − 1 ≤ 0 and v k − 1 ,n − 1 ≥ 0 , we have the following result after dropping most negative terms in the deﬁnition of δ k,i :
Revised: Because it holds that m k − 1 ,n − 1 ≤ 0 and v k − 1 ,n − 1 ≥ 0 , we have the following result after dropping most negative terms in the deﬁnition of δ k,i :

Original: [Equation]
Revised: [Equation]

Original: Therefore, to make (cid:80) mi =

Revised: Therefore, to make (cid:80) mi =

Original: δ k,i ≥
Revised: δ k,i ≥

Original: [Equation]
Revised: [Equation]

Original: mi =0 − η
Revised: mi =0 − η

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: √ (1 − β 2 ) β n 2 ≥ − 2 , we have
Revised: √ (1 − β 2 ) β n 2 ≥ − 2 , we have

Original: [Equation]
Revised: [Equation]

Original: To show the divergence, we want to show that there exists some β 1 and β 2 s.t.
Revised: To show the divergence, we want to show that there exists some β 1 and β 2 s.t.

Original: the both of the following conditions hold:
Revised: the both of the following conditions hold:

Original: [Equation]
Revised: [Equation]

Original: The proof of Theorem 3.3 is completed.
Revised: The proof of Theorem 3.3 is completed.

Original: With the help of Python, we visualize the region where ( C1 ) and ( C2 ) hold.
Revised: With the help of Python, we visualize the region where ( C1 ) and ( C2 ) hold.

Original: The results are shown in Figure 16.
Revised: The results are shown in Figure 16.

Original: We use orange color to indicate the region where ( C1 ) holds.
Revised: We use orange color to indicate the region where ( C1 ) holds.

Original: White color is used for the counter part.
Revised: White color is used for the counter part.

Original: As for ( C2 ) , we use the gray vertical line to indicate the line where (1 − β n − 1) =
Revised: As for ( C2 ) , we use the gray vertical line to indicate the line where (1 − β n − 1) =

Original: [Equation]
Revised: [Equation]

Original: 1 − β 1 ) β n − 1n .
Revised: 1 − β 1 ) β n − 1n .

Original: Note that there are two solutions to this equation: one solution is β 1 = 1 and the other solution lies in 0 < β 1 < 1 , this is why there are two vertical lines in the ﬁgure.
Revised: Note that there are two solutions to this equation: one solution is β 1 = 1 and the other solution lies in 0 < β 1 < 1 , this is why there are two vertical lines in the ﬁgure.

Original: ( C2 ) holds on the left hand side of the left gray vertical line.
Revised: ( C2 ) holds on the left hand side of the left gray vertical line.

Original: The intersection of two regions will be the region where Adam diverges, which is actually the orange region in Figure 16.
Revised: The intersection of two regions will be the region where Adam diverges, which is actually the orange region in Figure 16.

Original: As n increases, ( C2 ) holds for a wide range of β 1 , so the grey vertical lines move towards β 1 = 1 and ﬁnally get overlapped.
Revised: As n increases, ( C2 ) holds for a wide range of β 1 , so the grey vertical lines move towards β 1 = 1 and ﬁnally get overlapped.

Original: In addition, the size of divergence region increases with n
Revised: In addition, the size of divergence region increases with n

Original: Relation with γ 1 ( n ) in Theorem 3.1 According to Theorem 3.1, γ 1 ( n ) is at least in the order of 1 − O ( n − 2 ) .
Revised: Relation with γ 1 ( n ) in Theorem 3.1 According to Theorem 3.1, γ 1 ( n ) is at least in the order of 1 − O ( n − 2 ) .

Original: Combining with Figure 16.
Revised: Combining with Figure 16.

Original: It is not hard to see that γ 1 ( n ) is always larger than the upper boundary of the orange region, so there is no contradiction.
Revised: It is not hard to see that γ 1 ( n ) is always larger than the upper boundary of the orange region, so there is no contradiction.

Original: Proof of Corollary 4.1.
Revised: Proof of Corollary 4.1.

Original: For any ( β 1 , β 2 ) ∈ [0 , 1) 2 , condition (C1), (C2) and (C3) can be satisﬁed by some sufﬁciently large n .
Revised: For any ( β 1 , β 2 ) ∈ [0 , 1) 2 , condition (C1), (C2) and (C3) can be satisﬁed by some sufﬁciently large n .

Original: Therefore, Adam will diverge and the proof is concluded.
Revised: Therefore, Adam will diverge and the proof is concluded.

Original: F Some More Notations and Useful Lemmas for Convergence Analysis F.1 More notations
Revised: F Some More Notations and Useful Lemmas for Convergence Analysis F.1 More notations

Original: • We denote x k,i , m k,i , v k,i ∈
Revised: • We denote x k,i , m k,i , v k,i ∈

Original: R
Revised: R

Original: d as the value of x, m, v at the k -th outer loop and i -th inner loop.
Revised: d as the value of x, m, v at the k -th outer loop and i -th inner loop.

Original: Further, we denote x l,k,i , m l,k,i , v l,k,i ∈ R as the l -th component of x k,i , m k,i , v k,i .
Revised: Further, we denote x l,k,i , m l,k,i , v l,k,i ∈ R as the l -th component of x k,i , m k,i , v k,i .

Original: • We denote η k as the stepsize at the k -th epoch (outer loop).
Revised: • We denote η k as the stepsize at the k -th epoch (outer loop).

Original: We will focus mainly on diminishing step size, especially η k = η 1 √ nk , where n is the number of batches (inner loop).
Revised: We will focus mainly on diminishing step size, especially η k = η 1 √ nk , where n is the number of batches (inner loop).

Original: • We denote ∂ l f ( x ) =
Revised: • We denote ∂ l f ( x ) =

Original: ∂∂x l f ( x ) , ∂ l f j ( x ) =
Revised: ∂∂x l f ( x ) , ∂ l f j ( x ) =

Original: ∂∂x l f j ( x ) .
Revised: ∂∂x l f j ( x ) .

Original: Further, we will use τ k,i to index the i -th randomly chosen batch in the k -th epoch.
Revised: Further, we will use τ k,i to index the i -th randomly chosen batch in the k -th epoch.

Original: In this sense, we denote ∂ l f τ k,i ( x ) as ∂∂x l f τ k,i ( x ) .
Revised: In this sense, we denote ∂ l f τ k,i ( x ) as ∂∂x l f τ k,i ( x ) .

Original: • Given an epoch k , we denote α as the index of the coordinate with the greatest gradient:
Revised: • Given an epoch k , we denote α as the index of the coordinate with the greatest gradient:

Original: [Equation]
Revised: [Equation]

Original: 0.8 beta 0.
Revised: 0.8 beta 0.

Original: b e t a(a) n =0.
Revised: b e t a(a) n =0.

Original: 0.8 beta 0.
Revised: 0.8 beta 0.

Original: b e t a(b) n =0.
Revised: b e t a(b) n =0.

Original: 0.8 beta 0.
Revised: 0.8 beta 0.

Original: b e t a(c) n =0.
Revised: b e t a(c) n =0.

Original: 0.8 beta 0.
Revised: 0.8 beta 0.

Original: b e t a
Revised: b e t a

Original: [Figure]
Revised: [Figure]

Original: • We deﬁne ρ 1 , ρ 2 , ρ 3 as constants satisfying the following condition for any l = 1 , · · · , d :
Revised: • We deﬁne ρ 1 , ρ 2 , ρ 3 as constants satisfying the following condition for any l = 1 , · · · , d :

Original: [Equation]
Revised: [Equation]

Original: , 0 ) |.
Revised: , 0 ) |.

Original: ρ 1 , ρ 2 , ρ 3 are problem-dependent constants.
Revised: ρ 1 , ρ 2 , ρ 3 are problem-dependent constants.

Original: In worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .
Revised: In worst case, we have 0 ≤ ρ 3 ≤ √ nρ 1 ≤ n .

Original: • We deﬁne the constant (cid:52) x :=
Revised: • We deﬁne the constant (cid:52) x :=

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: x L
Revised: x L

Original: [Equation]
Revised: [Equation]

Original: This constant will be used repeatedly, especially in Lemma F.2.
Revised: This constant will be used repeatedly, especially in Lemma F.2.

Original: • We deﬁne Q k :=
Revised: • We deﬁne Q k :=

Original: (cid:52) 1 n
Revised: (cid:52) 1 n

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: This constant will be used repeatedly.
Revised: This constant will be used repeatedly.

Original: • We deﬁne
Revised: • We deﬁne

Original: [Equation]
Revised: [Equation]

Original: This constant will be used repeatedly.
Revised: This constant will be used repeatedly.

Original: Note that δ 1 → 0 when β 2 → 1 . • E ( · ) means taking expectation over the whole trajectory.
Revised: Note that δ 1 → 0 when β 2 → 1 . • E ( · ) means taking expectation over the whole trajectory.

Original: • We will use E k [ · ] as a shorthand for E [ · (cid:12)(cid:12) x k, 0 , x k − 1 ,n − 1 , · · · , x 1 , 0 ] , i.e. the conditional expectation given all the history up to x k, 0 .
Revised: • We will use E k [ · ] as a shorthand for E [ · (cid:12)(cid:12) x k, 0 , x k − 1 ,n − 1 , · · · , x 1 , 0 ] , i.e. the conditional expectation given all the history up to x k, 0 .

Original: Similarly, E k − 1 stands for the conditional expectation given all the history up to x k − 1 , 0 .
Revised: Similarly, E k − 1 stands for the conditional expectation given all the history up to x k − 1 , 0 .

Original: We further introduce some ‘index’ notation of the stochastic gradients, these notations will be useful in the proof.
Revised: We further introduce some ‘index’ notation of the stochastic gradients, these notations will be useful in the proof.

Original: For all the stochastic gradient, we have
Revised: For all the stochastic gradient, we have

Original: • True index: every stochastic gradient has its own true index: f 1 ( · ) , f 2 ( · ) , · · · , f n − 1 ( · ) .
Revised: • True index: every stochastic gradient has its own true index: f 1 ( · ) , f 2 ( · ) , · · · , f n − 1 ( · ) .

Original: All these { f i ( · ) } n − 1 i =0 are ﬁxed once the optimization problem is formulated.
Revised: All these { f i ( · ) } n − 1 i =0 are ﬁxed once the optimization problem is formulated.

Original: Note that for a ﬁxed x , we have the following relationship between f i and f :
Revised: Note that for a ﬁxed x , we have the following relationship between f i and f :

Original: [Equation]
Revised: [Equation]

Original: • Random-shufﬂe index: At the k -th epoch, all the stochastic gradients are sampled in the order of f τ k, 0 ( · ) , f
Revised: • Random-shufﬂe index: At the k -th epoch, all the stochastic gradients are sampled in the order of f τ k, 0 ( · ) , f

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Since Algorithm 1 is sampling without replacement, there is an implicit bijective mapping between
Revised: Since Algorithm 1 is sampling without replacement, there is an implicit bijective mapping between

Original: { f i ( · ) } n − 1 i =0 and { f τ k
Revised: { f i ( · ) } n − 1 i =0 and { f τ k

Original: i ( · ) } n −
Revised: i ( · ) } n −

Original: 1 i =
Revised: 1 i =

Original: Further, for a ﬁxed x , we have the following useful property:
Revised: Further, for a ﬁxed x , we have the following useful property:

Original: [Equation]
Revised: [Equation]

Original: • Index for m and v : As shown in Algorithm 1, we denote :
Revised: • Index for m and v : As shown in Algorithm 1, we denote :

Original: [Equation]
Revised: [Equation]

Original: F.2 Some useful lemmas
Revised: F.2 Some useful lemmas

Original: We begin with proving some useful lemmas.
Revised: We begin with proving some useful lemmas.

Original: Lemma F.1.
Revised: Lemma F.1.

Original: For any β (cid:54) = 1 , we have
Revised: For any β (cid:54) = 1 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: Proof only involves basic calculation, we omit the proof here.
Revised: Proof only involves basic calculation, we omit the proof here.

Original: Lemma F.2.
Revised: Lemma F.2.

Original: Under Assumption 2.1, if β 1 < √ β 2 , we have
Revised: Under Assumption 2.1, if β 1 < √ β 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We start with bounding | x l,k,i +1 − x l,k,i | .
Revised: We start with bounding | x l,k,i +1 − x l,k,i | .

Original: By the update formula of Adam, we have | x l,k,i
Revised: By the update formula of Adam, we have | x l,k,i

Original: [Equation]
Revised: [Equation]

Original: 1 − x l,
Revised: 1 − x l,

Original: [Equation]
Revised: [Equation]

Original: ,i
Revised: ,i

Original: [Equation]
Revised: [Equation]

Original: Now, by Assumption 2.1, we have
Revised: Now, by Assumption 2.1, we have

Original: [Equation]
Revised: [Equation]

Original: Proof is completed.
Revised: Proof is completed.

Original: Lemma F.3.
Revised: Lemma F.3.

Original: Under Assumption 2.1 and 2.2, for any integer i ∈ [0 , n − 1] , we have the following two results,
Revised: Under Assumption 2.1 and 2.2, for any integer i ∈ [0 , n − 1] , we have the following two results,

Original: [Equation]
Revised: [Equation]

Original: where | ∂ α f ( x k, 0 ) | = max l ∈ [1 ,d ] | ∂ l f ( x k, 0 ) | .
Revised: where | ∂ α f ( x k, 0 ) | = max l ∈ [1 ,d ] | ∂ l f ( x k, 0 ) | .

Original: Proof.
Revised: Proof.

Original: | ∂ l f τ k,j ( x k,j ) |
Revised: | ∂ l f τ k,j ( x k,j ) |

Original: [Table]
Revised: [Table]

Original: 2 ≤
Revised: 2 ≤

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: (cid:88)
Revised: (cid:88)

Original: [Table]
Revised: [Table]

Original: =
Revised: =

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Assumption
Revised: Assumption

Original: [Equation]
Revised: [Equation]

Original: where ( ∗ ) is because of the deﬁnition (see Appendix F.1): ρ 1 is a constant satisfying ρ 1 ≥ (cid:80) ni
Revised: where ( ∗ ) is because of the deﬁnition (see Appendix F.1): ρ 1 is a constant satisfying ρ 1 ≥ (cid:80) ni

Original: =1 | ∂ l f i ( x k, 0 ) | √ (cid:80) ni
Revised: =1 | ∂ l f i ( x k, 0 ) | √ (cid:80) ni

Original: =1 | ∂ l f i ( x k, 0 ) | 2 ; ( ∗∗ ) is due to (cid:107)∇ f ( x k, 0 ) (cid:107) 22 ≤ d | ∂ α f ( x k, 0 ) | 2 .
Revised: =1 | ∂ l f i ( x k, 0 ) | 2 ; ( ∗∗ ) is due to (cid:107)∇ f ( x k, 0 ) (cid:107) 22 ≤ d | ∂ α f ( x k, 0 ) | 2 .

Original: The proof of (24) follows the same procedure as above.
Revised: The proof of (24) follows the same procedure as above.

Original: G Proof of Theorem 3. G.1 A roadmap of the proof
Revised: G Proof of Theorem 3. G.1 A roadmap of the proof

Original: Our proof is based on the Descent Lemma:
Revised: Our proof is based on the Descent Lemma:

Original: [Equation]
Revised: [Equation]

Original: The expectation E ( · ) is taken on the whole trajectory.
Revised: The expectation E ( · ) is taken on the whole trajectory.

Original: Summing both sides from the initialization k
Revised: Summing both sides from the initialization k

Original: = t 0 to k = T , we have the following re-arranged inequality: (usually we set t 0 = 1 .)
Revised: = t 0 to k = T , we have the following re-arranged inequality: (usually we set t 0 = 1 .)

Original: [Equation]
Revised: [Equation]

Original: To prove the convergence, we need an upper bound for L 2 E (cid:107) x k +1 , 0 − x k, 0 (cid:107) 22 , as well as a lower bound for E (cid:104)∇ f ( x k, 0 ) , x k, 0 − x k +1 , 0 (cid:105) (and such a lower bound should be in the order of 1 √ k E (cid:107)∇ f ( x k, 0 ) (cid:107) ).
Revised: To prove the convergence, we need an upper bound for L 2 E (cid:107) x k +1 , 0 − x k, 0 (cid:107) 22 , as well as a lower bound for E (cid:104)∇ f ( x k, 0 ) , x k, 0 − x k +1 , 0 (cid:105) (and such a lower bound should be in the order of 1 √ k E (cid:107)∇ f ( x k, 0 ) (cid:107) ).

Original: The upper bound for L 2 E (cid:107) x k +1 , 0 − x k, 0 (cid:107) 22 is relatively easy to get: according to the update rule of
Revised: The upper bound for L 2 E (cid:107) x k +1 , 0 − x k, 0 (cid:107) 22 is relatively easy to get: according to the update rule of

Original: Adam, this term is in the order of O ( η 2 k m l,k,i /v l,k,i )
Revised: Adam, this term is in the order of O ( η 2 k m l,k,i /v l,k,i )

Original: =
Revised: =

Original: O ( η 2 k (cid:107)∇ f ( x k, 0 ) (cid:107) /
Revised: O ( η 2 k (cid:107)∇ f ( x k, 0 ) (cid:107) /

Original: (cid:107)∇ f ( x k, 0 ) (cid:107) )
Revised: (cid:107)∇ f ( x k, 0 ) (cid:107) )

Original: =
Revised: =

Original: O ( 1 k ) .
Revised: O ( 1 k ) .

Original: Further, recall (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 , so L 2 E (cid:107) x k +1 , 0 − x k, 0 (cid:107) 22 contributes to the log term in Theorem 3.1.
Revised: Further, recall (cid:80) Tk = t 0 1 k ≤ log T +1 t 0 , so L 2 E (cid:107) x k +1 , 0 − x k, 0 (cid:107) 22 contributes to the log term in Theorem 3.1.

Original: The proof is shown in Lemma F.2.
Revised: The proof is shown in Lemma F.2.

Original: However, the lower bound for E (cid:104)∇ f ( x k, 0 ) , x k, 0 − x k +1 , 0 (cid:105) =
Revised: However, the lower bound for E (cid:104)∇ f ( x k, 0 ) , x k, 0 − x k +1 , 0 (cid:105) =

Original: E (cid:104)(cid:80) dl
Revised: E (cid:104)(cid:80) dl

Original: =1 (cid:80) n − 1 i =

Revised: =1 (cid:80) n − 1 i =

Original: ∂ l f ( x k, 0 ) m l,k,i √ v l,k,i (cid:105) requires sophisticated derivation, we explain as follows.
Revised: ∂ l f ( x k, 0 ) m l,k,i √ v l,k,i (cid:105) requires sophisticated derivation, we explain as follows.

Original: Before taking the expectation, we ﬁrst work on every possible realization of (cid:80) dl
Revised: Before taking the expectation, we ﬁrst work on every possible realization of (cid:80) dl

Original: =1 (cid:80) n − 1 i =

Revised: =1 (cid:80) n − 1 i =

Original: ∂ l f ( x k, 0 ) m l,k,i √ v l,k,i .
Revised: ∂ l f ( x k, 0 ) m l,k,i √ v l,k,i .

Original: We perform the following decomposition for every l ∈ [1 , d ] .
Revised: We perform the following decomposition for every l ∈ [1 , d ] .

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: First of all, we introduce the following Lemma G.1 to further decompose ( a ) and ( b ) .
Revised: First of all, we introduce the following Lemma G.1 to further decompose ( a ) and ( b ) .

Original: The intuition of this decomposition is as follows: by increasing β 2 , we can control the movement of the moving average factor v l,k,i , so the trajectory of Adam is easier to control.
Revised: The intuition of this decomposition is as follows: by increasing β 2 , we can control the movement of the moving average factor v l,k,i , so the trajectory of Adam is easier to control.

Original: Lemma G.1.
Revised: Lemma G.1.

Original: Under Assumption 2.1, for those l satisfying max i | ∂ l f i ( x k, 0 )
Revised: Under Assumption 2.1, for those l satisfying max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k := (cid:52) 1 n √ n √ k
Revised: | ≥ Q k := (cid:52) 1 n √ n √ k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: n β n 2 , we have the following lower bound for ( a ) in (26) :
Revised: n β n 2 , we have the following lower bound for ( a ) in (26) :

Original: [Equation]
Revised: [Equation]

Original: | ∂ l f i ( x k, 0 ) | , (27) where δ 1 = (1 − β 2 )4 nρ 2 β
Revised: | ∂ l f i ( x k, 0 ) | , (27) where δ 1 = (1 − β 2 )4 nρ 2 β

Original: n+ (cid:18) 1 √ β n 2 −(cid:19) .
Revised: n+ (cid:18) 1 √ β n 2 −(cid:19) .

Original: ρ 2 , ρ 3 are constants satisfying ρ 2 ≥ | max i ∂ l
Revised: ρ 2 , ρ 3 are constants satisfying ρ 2 ≥ | max i ∂ l

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: x k

Original: [Equation]
Revised: [Equation]

Original: 2 1 n
Revised: 2 1 n

Original: [Equation]
Revised: [Equation]

Original: ni =
Revised: ni =

Original: [Equation]
Revised: [Equation]

Original: | ∂
Revised: | ∂

Original: [Equation]
Revised: [Equation]

Original: f i ( x k
Revised: f i ( x k

Original: [Equation]
Revised: [Equation]

Original: | 2 , ρ 3 ≥ | (cid:80) ni =1 ∂ l f i ( x k, 0 ) | √n
Revised: | 2 , ρ 3 ≥ | (cid:80) ni =1 ∂ l f i ( x k, 0 ) | √n

Original: [Equation]
Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Original: | 2 .
Revised: | 2 .

Original: Note that δ 1 → 0 when β 2 → 1 .
Revised: Note that δ 1 → 0 when β 2 → 1 .

Original: Under the same condition, we also have a lower bound for ( b ) in (26) :
Revised: Under the same condition, we also have a lower bound for ( b ) in (26) :

Original: [Equation]
Revised: [Equation]

Original: The proof can be seen in Appendix G.2.
Revised: The proof can be seen in Appendix G.2.

Original: Lemma G.1 decomposes both ( a ) and ( b ) : in the denominator, we approximate v l,k,i (which is changing with i ) by v l,k, 0 (which is ﬁxed).
Revised: Lemma G.1 decomposes both ( a ) and ( b ) : in the denominator, we approximate v l,k,i (which is changing with i ) by v l,k, 0 (which is ﬁxed).

Original: Accordingly, the approximation error can be controlled by increasing β 2 (since δ 1 → 0 when β 2 → 1 ).
Revised: Accordingly, the approximation error can be controlled by increasing β 2 (since δ 1 → 0 when β 2 → 1 ).

Original: However,
Revised: However,

Original: Lemma G.1 can only be applied to those l with max i | ∂ l f i ( x k, 0 )
Revised: Lemma G.1 can only be applied to those l with max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k , so we need to discuss the following two cases to complete the whole picture.
Revised: | ≥ Q k , so we need to discuss the following two cases to complete the whole picture.

Original: Case 1: unbounded gradient.
Revised: Case 1: unbounded gradient.

Original: Given x k, 0 , consider those l with max i | ∂ l f i ( x k, 0 )
Revised: Given x k, 0 , consider those l with max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k , we have the following decomposition:
Revised: | ≥ Q k , we have the following decomposition:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ,k
Revised: ,k

Original: [Equation]
Revised: [Equation]

Original: n −
Revised: n −

Original: [Equation]
Revised: [Equation]

Original: i = ( m
Revised: i = ( m

Original: [Equation]
Revised: [Equation]

Original: (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) (
Revised: (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) (

Original: [Equation]
Revised: [Equation]

Original: − (cid:34)
Revised: − (cid:34)

Original: [Equation]
Revised: [Equation]

Original: (cid:115)
Revised: (cid:115)

Original: [Equation]
Revised: [Equation]

Original: β n
Revised: β n

Original: [Equation]
Revised: [Equation]

Original: n − 1 (cid:88)
Revised: n − 1 (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: | m l
Revised: | m l

Original: [Equation]
Revised: [Equation]

Original: i − ∂
Revised: i − ∂

Original: [Equation]
Revised: [Equation]

Original: f i (
Revised: f i (

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: | (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) ( b
Revised: | (cid:35) (cid:124) (cid:123)(cid:122) (cid:125) ( b

Original: [Equation]
Revised: [Equation]

Original: Case 2: bounded gradient.
Revised: Case 2: bounded gradient.

Original: Given x k, 0 , consider those l with max i | ∂ l f i ( x k, 0 ) | ≤ Q k , the analysis degenerates to the “bounded gradient” scenario, we have the following lower bound:
Revised: Given x k, 0 , consider those l with max i | ∂ l f i ( x k, 0 ) | ≤ Q k , the analysis degenerates to the “bounded gradient” scenario, we have the following lower bound:

Original: [Equation]
Revised: [Equation]

Original: where F 1 :=
Revised: where F 1 :=

Original: [Equation]
Revised: [Equation]

Original: n , ( ∗ ) is due to the fact that | ∂ l f ( x k, 0 ) | ≤ n max i | ∂ l f i ( x k, 0 ) | ≤ nQ k .
Revised: n , ( ∗ ) is due to the fact that | ∂ l f ( x k, 0 ) | ≤ n max i | ∂ l f i ( x k, 0 ) | ≤ nQ k .

Original: Combining Case 1 & 2 together, we have the following result (note that ( a 1 ) , ( a 2 ) , ( b 1 ) , ( b 2 ) vary with l .
Revised: Combining Case 1 & 2 together, we have the following result (note that ( a 1 ) , ( a 2 ) , ( b 1 ) , ( b 2 ) vary with l .

Original: A more precise notation will be ( a 1 ) l , etc.
Revised: A more precise notation will be ( a 1 ) l , etc.

Original: We drop the subscript for brevity).
Revised: We drop the subscript for brevity).

Original: [Equation]
Revised: [Equation]

Original: l large
Revised: l large

Original: [Equation]
Revised: [Equation]

Original: l small
Revised: l small

Original: [Equation]
Revised: [Equation]

Original: large
Revised: large

Original: [Equation]
Revised: [Equation]

Original: ≥
Revised: ≥

Original: [Equation]
Revised: [Equation]

Original: l large
Revised: l large

Original: [Equation]
Revised: [Equation]

Original: l large
Revised: l large

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: l large
Revised: l large

Original: [Equation]
Revised: [Equation]

Original: (30) where “ l large” stands for the gradient component in Case 1 and “ l small” indicates those in Case 2 .
Revised: (30) where “ l large” stands for the gradient component in Case 1 and “ l small” indicates those in Case 2 .

Original: We assume “ l large” is not an empty set, otherwise the analysis degenerates to the proof with bounded gradient assumption.
Revised: We assume “ l large” is not an empty set, otherwise the analysis degenerates to the proof with bounded gradient assumption.

Original: Now we take expectation on (30).
Revised: Now we take expectation on (30).

Original: The expectation is taken on all the possible trajectories up to the i -th iteration in k -th epoch.
Revised: The expectation is taken on all the possible trajectories up to the i -th iteration in k -th epoch.

Original: [Equation]
Revised: [Equation]

Original: In the following context, we will discuss how to bound all terms in (31), respectively.
Revised: In the following context, we will discuss how to bound all terms in (31), respectively.

Original: First and foremost, we derive a lower bound for E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .
Revised: First and foremost, we derive a lower bound for E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .

Original: Since E (cid:104)(cid:80) l large ( b 1 ) (cid:105) contains all the historical gradient information, it involves great effort to handle it.
Revised: Since E (cid:104)(cid:80) l large ( b 1 ) (cid:105) contains all the historical gradient information, it involves great effort to handle it.

Original: We will show that the lower bound of E (cid:104)(cid:80) l large ( b 1 ) (cid:105) will vanish when β 2 is large and k goes to inﬁnity.
Revised: We will show that the lower bound of E (cid:104)(cid:80) l large ( b 1 ) (cid:105) will vanish when β 2 is large and k goes to inﬁnity.

Original: Recall E (cid:104)(cid:80) l large ( b 1 ) (cid:105) =
Revised: Recall E (cid:104)(cid:80) l large ( b 1 ) (cid:105) =

Original: E (cid:104)(cid:80) l large ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Revised: E (cid:104)(cid:80) l large ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) .
Revised: (cid:105) .

Original: When β 1 is large, m l,k,i contains heavy historical signals.
Revised: When β 1 is large, m l,k,i contains heavy historical signals.

Original: It seems unclear how large E (cid:104)(cid:80) l large ( b 1 ) (cid:105) would be when β 1 goes to 1.
Revised: It seems unclear how large E (cid:104)(cid:80) l large ( b 1 ) (cid:105) would be when β 1 goes to 1.

Original: Existing literatures (Zaheer et al., 2018; De et al., 2018; Shi et al., 2020) take a naive approach: they set β 1 ≈ 0 so that m k,i ≈ ∇ f τ k,i ( x k,i ) .
Revised: Existing literatures (Zaheer et al., 2018; De et al., 2018; Shi et al., 2020) take a naive approach: they set β 1 ≈ 0 so that m k,i ≈ ∇ f τ k,i ( x k,i ) .

Original: Then we get δ ( β 1 ) ≈ 0 .
Revised: Then we get δ ( β 1 ) ≈ 0 .

Original: However, this naive method cannot be applied here since we are interested in practical cases where β 1 is large in [0 , 1) .
Revised: However, this naive method cannot be applied here since we are interested in practical cases where β 1 is large in [0 , 1) .

Original: We emphasize the following technical difﬁculties in bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) for any β 1 ∈ [0 , 1) :
Revised: We emphasize the following technical difﬁculties in bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) for any β 1 ∈ [0 , 1) :

Original: • Issue (i)
Revised: • Issue (i)

Original: In order to bound E (cid:104)(cid:80) l large ( b 1 ) (cid:105) , we need to ﬁrst know how to bound its simpler version: E (cid:104)(cid:80) n − 1 i =

Revised: In order to bound E (cid:104)(cid:80) l large ( b 1 ) (cid:105) , we need to ﬁrst know how to bound its simpler version: E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) .
Revised: (cid:105) .

Original: This term measures the difference between the current gradient and weighted historical gradients.
Revised: This term measures the difference between the current gradient and weighted historical gradients.

Original: It seems unclear that how large this term could be when β 1 goes to 1.
Revised: It seems unclear that how large this term could be when β 1 goes to 1.

Original: • Issue (ii)
Revised: • Issue (ii)

Original: Even if we can bound E (cid:104)(cid:80) n − 1 i =

Revised: Even if we can bound E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) , it is still unclear how to bound E (cid:104) ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Revised: (cid:105) , it is still unclear how to bound E (cid:104) ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) , which is further multiplied by a random variable ∂ l f ( x k, 0 ) √ v l,k, 0 .
Revised: (cid:105) , which is further multiplied by a random variable ∂ l f ( x k, 0 ) √ v l,k, 0 .

Original: • Issue (iii)
Revised: • Issue (iii)

Original: Even if we can bound E (cid:104) ∂
Revised: Even if we can bound E (cid:104) ∂

Original: l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Revised: l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) , it is still different from E (cid:104)(cid:80) l large ( b 1 ) (cid:105) which contains additional operation “ (cid:80) l large " inside the expectation.
Revised: (cid:105) , it is still different from E (cid:104)(cid:80) l large ( b 1 ) (cid:105) which contains additional operation “ (cid:80) l large " inside the expectation.

Original: Note that the set “ l large" is a random variable which changes along different trajectories, so there is still non-negligible gap between bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) and
Revised: Note that the set “ l large" is a random variable which changes along different trajectories, so there is still non-negligible gap between bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) and

Original: E (cid:104)
Revised: E (cid:104)

Original: [Equation]
Revised: [Equation]

Original: x k
Revised: x k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: ,k
Revised: ,k

Original: [Equation]
Revised: [Equation]

Original: To our best knowledge, there is no general approach to tackle the above issues.
Revised: To our best knowledge, there is no general approach to tackle the above issues.

Original: In the following content, we will overcome difﬁculties (i), (ii) and (iii) in Step 1, 2 and 3 respectively.
Revised: In the following content, we will overcome difﬁculties (i), (ii) and (iii) in Step 1, 2 and 3 respectively.

Original: In Step 1, we will discuss how to bound E (cid:104)(cid:80) n − 1 i =

Revised: In Step 1, we will discuss how to bound E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) , which is a simpliﬁed version of E [( b 1 )] .
Revised: (cid:105) , which is a simpliﬁed version of E [( b 1 )] .

Original: Bounding this term will shed light on bounding the whole term E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .
Revised: Bounding this term will shed light on bounding the whole term E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .

Original: Then in Step 2 and 3, we will prove several technical lemmas to handle the effect of ∂ l f ( x k, 0 ) √ v l,k, 0 and “ (cid:80) l large ", by which we can tackle (ii) and (iii) .
Revised: Then in Step 2 and 3, we will prove several technical lemmas to handle the effect of ∂ l f ( x k, 0 ) √ v l,k, 0 and “ (cid:80) l large ", by which we can tackle (ii) and (iii) .

Original: Combining all together we can bound E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .
Revised: Combining all together we can bound E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .

Original: Bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) :
Revised: Bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) :

Original: Step 1.
Revised: Step 1.

Original: We now introduce the key idea of bounding
Revised: We now introduce the key idea of bounding

Original: E (cid:104)(cid:80) n − 1 i =

Revised: E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) .
Revised: (cid:105) .

Original: In Appendix A, we distill our idea into a toy example called "the color-ball model of the 1st kind" and thus we prove Lemma A.1.
Revised: In Appendix A, we distill our idea into a toy example called "the color-ball model of the 1st kind" and thus we prove Lemma A.1.

Original: This lemma is crucial for bounding
Revised: This lemma is crucial for bounding

Original: E (cid:104)(cid:80) n − 1 i =

Revised: E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) .
Revised: (cid:105) .

Original: We refer the readers to Appendix A for more explanation.
Revised: We refer the readers to Appendix A for more explanation.

Original: Lemma A.1 can provide insights in bounding E (cid:104)(cid:80) n − 1 i =

Revised: Lemma A.1 can provide insights in bounding E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) .
Revised: (cid:105) .

Original: However, there is still certain gap between E (cid:104)(cid:80) n − 1 i =

Revised: However, there is still certain gap between E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) and the quantities in Lemma A.1.
Revised: (cid:105) and the quantities in Lemma A.1.

Original: We elaborate as follows:
Revised: We elaborate as follows:

Original: • To mimic the color-ball example, we need to expand (cid:80) n − 1 i =

Revised: • To mimic the color-ball example, we need to expand (cid:80) n − 1 i =

Original: ∂ l f i ( x k, 0 ) into an inﬁnite sum sequence: (cid:80) n − 1 i =

Revised: ∂ l f i ( x k, 0 ) into an inﬁnite sum sequence: (cid:80) n − 1 i =

Original: ∂ l f i ( x k, 0 )(1 − β 1 )(1 + β 1 + · · · β ∞ 1 ) .
Revised: ∂ l f i ( x k, 0 )(1 − β 1 )(1 + β 1 + · · · β ∞ 1 ) .

Original: However, (cid:80) n − 1 i =

Revised: However, (cid:80) n − 1 i =

Original: m l,k,i is a ﬁnite sum sequence up to the order of β kn .
Revised: m l,k,i is a ﬁnite sum sequence up to the order of β kn .

Original: In contrast, both sequences in the color-ball example are “ﬁnite sum”.
Revised: In contrast, both sequences in the color-ball example are “ﬁnite sum”.

Original: As such, there is an error term caused by “ﬁnite sum v.s. inﬁnite sum”.
Revised: As such, there is an error term caused by “ﬁnite sum v.s. inﬁnite sum”.

Original: • When taking the expectation, the variable x in each possible trajectory is different.
Revised: • When taking the expectation, the variable x in each possible trajectory is different.

Original: In contrast, in the color-ball example, { a i } 2 i =0 are ﬁxed in all shufﬂing order (so it is much easier to calculate the expectation by summing them up).
Revised: In contrast, in the color-ball example, { a i } 2 i =0 are ﬁxed in all shufﬂing order (so it is much easier to calculate the expectation by summing them up).

Original: To mimic the color-ball example, we repeatedly take conditional expectation at the beginning of each k -th epoch.
Revised: To mimic the color-ball example, we repeatedly take conditional expectation at the beginning of each k -th epoch.

Original: In this way, x k, 0 will be ﬁx.
Revised: In this way, x k, 0 will be ﬁx.

Original: Despite x k,i is still changing across the trajectory, we can transform x k,i into x k, 0 by using Lipschitz property.
Revised: Despite x k,i is still changing across the trajectory, we can transform x k,i into x k, 0 by using Lipschitz property.

Original: In each trajectory of (cid:80) n − 1 i =

Revised: In each trajectory of (cid:80) n − 1 i =

Original: m l,k,i , the variable x in the summand of m l,k,i varies with k and i ; while the variable in ∂ l f i ( x k, 0 ) is ﬁxed to be x k, 0 .
Revised: m l,k,i , the variable x in the summand of m l,k,i varies with k and i ; while the variable in ∂ l f i ( x k, 0 ) is ﬁxed to be x k, 0 .

Original: In contrast, in the color-ball example, { a i } 2 i = are the same across the epoch.
Revised: In contrast, in the color-ball example, { a i } 2 i = are the same across the epoch.

Original: To mimic the ﬁnal step in the color-ball example (Figure 6), at each step of conditional expectation, we need to simultaneously move the variables in m l,k,i and ∂ l f i ( x k, 0 ) (using Lipschitz property) so that they can match and cancel out with each other.
Revised: To mimic the ﬁnal step in the color-ball example (Figure 6), at each step of conditional expectation, we need to simultaneously move the variables in m l,k,i and ∂ l f i ( x k, 0 ) (using Lipschitz property) so that they can match and cancel out with each other.

Original: This operation will introduce new error terms and it is our duty to put them under control.
Revised: This operation will introduce new error terms and it is our duty to put them under control.

Original: We omit the proof for bounding E (cid:104)(cid:80) n − 1 i =

Revised: We omit the proof for bounding E (cid:104)(cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) since this is not our actual goal.
Revised: (cid:105) since this is not our actual goal.

Original: Instead we will directly use the above ideas to bound E [( b 1 )] .
Revised: Instead we will directly use the above ideas to bound E [( b 1 )] .

Original: To do so, we need to further tackle the issue (ii) and (iii) mentioned before.
Revised: To do so, we need to further tackle the issue (ii) and (iii) mentioned before.

Original: We explain as follows.
Revised: We explain as follows.

Original: Bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) :
Revised: Bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) :

Original: Step 2.
Revised: Step 2.

Original: We now resolve issue (ii) , i.e., handle the effect of ∂ l f ( x k, 0 ) √ v l,k, 0 .
Revised: We now resolve issue (ii) , i.e., handle the effect of ∂ l f ( x k, 0 ) √ v l,k, 0 .

Original: The key idea is as follows: when we calculate E [( b 1 )] , we sequentially take conditional expectation E k ( · ) , E k − 1 ( · ) , etc.. When taking E k ( · ) , we will ﬁx all the historical information up to k -th epoch, so ∂ l f ( x k, 0 ) √ v l,k, 0 can be regarded as a constant.
Revised: The key idea is as follows: when we calculate E [( b 1 )] , we sequentially take conditional expectation E k ( · ) , E k − 1 ( · ) , etc.. When taking E k ( · ) , we will ﬁx all the historical information up to k -th epoch, so ∂ l f ( x k, 0 ) √ v l,k, 0 can be regarded as a constant.

Original: In this sense, E k (cid:104) ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =0 ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: In this sense, E k (cid:104) ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =0 ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) can be calculated following the same idea as color-ball toy example.
Revised: (cid:105) can be calculated following the same idea as color-ball toy example.

Original: However, when taking E k − 1 ( · ) , ∂ l f ( x k, 0 ) √ v l,k, 0 will become a random variable which changes with different trajectories.
Revised: However, when taking E k − 1 ( · ) , ∂ l f ( x k, 0 ) √ v l,k, 0 will become a random variable which changes with different trajectories.

Original: In this case, the color-ball method cannot be applied.
Revised: In this case, the color-ball method cannot be applied.

Original: To ﬁx this issue, we introduce the following lemma to change ∂ l f ( x k, 0 ) √ v l,k, 0 into ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 , which can again be regarded as a ﬁxed constant when taking E k − 1 ( · ) .
Revised: To ﬁx this issue, we introduce the following lemma to change ∂ l f ( x k, 0 ) √ v l,k, 0 into ∂ l f ( x k − 1 , 0 ) √ v k − 1 , 0 , which can again be regarded as a ﬁxed constant when taking E k − 1 ( · ) .

Original: Lemma G.2.
Revised: Lemma G.2.

Original: Suppose Assumption 2.1 holds and β 1 < √ β 2 .
Revised: Suppose Assumption 2.1 holds and β 1 < √ β 2 .

Original: For any integer j ∈ [0 , k ] , if max i
Revised: For any integer j ∈ [0 , k ] , if max i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , max
Revised: , max

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , max i |
Revised: , max i |

Original: [Equation]
Revised: [Equation]

Original: l f i ( x k
Revised: l f i ( x k

Original: [Equation]
Revised: [Equation]

Original: | ≥ Q k
Revised: | ≥ Q k

Original: [Equation]
Revised: [Equation]

Original: j (where
Revised: j (where

Original: Q k
Revised: Q k

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ), then we have the following result:
Revised: ), then we have the following result:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: ) √
Revised: ) √

Original: [Table]
Revised: [Table]

Original: where δ 1 is deﬁned in (17) .
Revised: where δ 1 is deﬁned in (17) .

Original: The proof of Lemma G.2 can be seen in Appendix G.3.
Revised: The proof of Lemma G.2 can be seen in Appendix G.3.

Original: To proceed, we combine Lemma G.2 and the “color-ball method of the 2nd kind", and thus we can prove Lemma A.2.
Revised: To proceed, we combine Lemma G.2 and the “color-ball method of the 2nd kind", and thus we can prove Lemma A.2.

Original: This lemma is crucial for our current goal: bounding E ( b 1 ) .
Revised: This lemma is crucial for our current goal: bounding E ( b 1 ) .

Original: We refer the readers to Appendix A for more information.
Revised: We refer the readers to Appendix A for more information.

Original: We emphasize that here are still the following gap between Lemma A.2 and our goal E ( b 1 ) .
Revised: We emphasize that here are still the following gap between Lemma A.2 and our goal E ( b 1 ) .

Original: • We have the similar gap as discussed at the end of the Step 1 .
Revised: • We have the similar gap as discussed at the end of the Step 1 .

Original: • The condition in Lemma G.2 has requirement on the gradient norm, while this requirement is temporarily ignored in the color-ball method of the 2nd kind.
Revised: • The condition in Lemma G.2 has requirement on the gradient norm, while this requirement is temporarily ignored in the color-ball method of the 2nd kind.

Original: • The result in Lemma G.2 has additional error terms other than O (1 / √ k ) .
Revised: • The result in Lemma G.2 has additional error terms other than O (1 / √ k ) .

Original: This is slightly different from the setting in Lemma A.2.
Revised: This is slightly different from the setting in Lemma A.2.

Original: It requires some technical lemmas to handle these gaps.
Revised: It requires some technical lemmas to handle these gaps.

Original: We ﬁll in these gaps in Lemma G.4.
Revised: We ﬁll in these gaps in Lemma G.4.

Original: The technical details can be seen Appendix G.4.
Revised: The technical details can be seen Appendix G.4.

Original: Bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) :
Revised: Bounding E (cid:104)(cid:80) l large ( b 1 ) (cid:105) :

Original: Step 3.
Revised: Step 3.

Original: Now we shift gear to tackle (iii): handling the random variable “ l large".
Revised: Now we shift gear to tackle (iii): handling the random variable “ l large".

Original: We rewrite “ l large" into the indicator function as follows:
Revised: We rewrite “ l large" into the indicator function as follows:

Original: [Equation]
Revised: [Equation]

Original: l large
Revised: l large

Original: [Equation]
Revised: [Equation]

Original: x k, 0 ) √ v l,k,n − 1 (cid:88) i =
Revised: x k, 0 ) √ v l,k,n − 1 (cid:88) i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 )) 
Revised: ( m l,k,i − ∂ l f i ( x k, 0 )) 

Original:  = E (cid:34) d (cid:88) l =
Revised:  = E (cid:34) d (cid:88) l =

Original: I k ∂ l f ( x k, 0 ) √ v l,k,n − 1 (cid:88) i = ( m l,k,i − ∂ l f i ( x k, 0 )) (cid:35) , where I k :
Revised: I k ∂ l f ( x k, 0 ) √ v l,k,n − 1 (cid:88) i = ( m l,k,i − ∂ l f i ( x k, 0 )) (cid:35) , where I k :

Original: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := (cid:52) 1 n √ n √ k 32 √ 2 (1 − β 2 ) n β n 2 (cid:17) is the indicator function ( I ( A )
Revised: = I (cid:16) max i | ∂ l f i ( x k, 0 ) | ≥ Q k := (cid:52) 1 n √ n √ k 32 √ 2 (1 − β 2 ) n β n 2 (cid:17) is the indicator function ( I ( A )

Original: =when event A holds and I ( A ) = 0 otherwise.)
Revised: =when event A holds and I ( A ) = 0 otherwise.)

Original: Similarly as before, when taking E k ( · ) , “ I k " can be regarded as a constant index.
Revised: Similarly as before, when taking E k ( · ) , “ I k " can be regarded as a constant index.

Original: Therefore, E k (cid:104)(cid:80) dl
Revised: Therefore, E k (cid:104)(cid:80) dl

Original: =

Revised: =

Original: I k ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Revised: I k ∂ l f ( x k, 0 ) √ v l,k, 0 (cid:80) n − 1 i =

Original: ( m l,k,i − ∂ l f i ( x k, 0 ))
Revised: ( m l,k,i − ∂ l f i ( x k, 0 ))

Original: (cid:105) can be calculated following the same idea as color-ball model of the 1st kind.
Revised: (cid:105) can be calculated following the same idea as color-ball model of the 1st kind.

Original: However, when taking E k − 1 ( · ) , I k will become a random variable which changes with different trajectories.
Revised: However, when taking E k − 1 ( · ) , I k will become a random variable which changes with different trajectories.

Original: In this case, the color-ball method cannot be applied.
Revised: In this case, the color-ball method cannot be applied.

Original: Similarly as in Step 2 , we introduce the following lemma to change I k into I k − 1 (deﬁned later), which can again be regarded as a ﬁxed when taking E k − 1 ( · ) .
Revised: Similarly as in Step 2 , we introduce the following lemma to change I k into I k − 1 (deﬁned later), which can again be regarded as a ﬁxed when taking E k − 1 ( · ) .

Original: Lemma G.3.
Revised: Lemma G.3.

Original: Suppose Assumption 2.1 holds and β 1 < √ β 2 .
Revised: Suppose Assumption 2.1 holds and β 1 < √ β 2 .

Original: For 0 ≤ j ≤ k , we deﬁne
Revised: For 0 ≤ j ≤ k , we deﬁne

Original: I k − j := I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , where Q k :=
Revised: I k − j := I (cid:16) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:17) , where Q k :=

Original: (cid:52)
Revised: (cid:52)

Original: [Equation]
Revised: [Equation]

Original: 32 √
Revised: 32 √

Original: [Equation]
Revised: [Equation]

Original: 2 , then we have the following results.
Revised: 2 , then we have the following results.

Original: I k
Revised: I k

Original: = I   max i | ∂ l f i ( x k, 0 )
Revised: = I   max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 )
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 )

Original: | ≥ k (cid:88) p =
Revised: | ≥ k (cid:88) p =

Original: k − j
Revised: k − j

Original: Q p 
Revised: Q p 

Original: 
Revised: 

Original: + I   max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) |
Revised: + I   max i | ∂ l f i ( x k, 0 ) | ≥ Q k and max i | ∂ l f i ( x k − j, 0 ) |

Original: ≤ k (cid:88) p =
Revised: ≤ k (cid:88) p =

Original: k − j
Revised: k − j

Original: Q p 
Revised: Q p 

Original:  , (32)
Revised:  , (32)

Original: I   max i | ∂ l f i ( x k, 0 )
Revised: I   max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 )
Revised: | ≥ Q k and max i | ∂ l f i ( x k − j, 0 )

Original: | ≥ k (cid:88) p =
Revised: | ≥ k (cid:88) p =

Original: k − j
Revised: k − j

Original: Q p 
Revised: Q p 

Original:  =
Revised:  =

Original: I   max i | ∂ l f i ( x k − j, 0 )
Revised: I   max i | ∂ l f i ( x k − j, 0 )

Original: | ≥ k (cid:88) p =
Revised: | ≥ k (cid:88) p =

Original: k − j
Revised: k − j

Original: Q p 
Revised: Q p 

Original:  =
Revised:  =

Original: I k − j .
Revised: I k − j .

Original: Proof.
Revised: Proof.

Original: Equation (32) is straightforward, we only prove (33) here.
Revised: Equation (32) is straightforward, we only prove (33) here.

Original: Under Assumption 2.1, we have | ∂ l f i ( x k, 0 ) − ∂ l f i ( x k − j, 0 ) | ≤ n (cid:52) n ( k − 1) + · · · + n (cid:52) n ( k − j ) ≤ Q k − 1 + · · · + Q k − j .
Revised: Under Assumption 2.1, we have | ∂ l f i ( x k, 0 ) − ∂ l f i ( x k − j, 0 ) | ≤ n (cid:52) n ( k − 1) + · · · + n (cid:52) n ( k − j ) ≤ Q k − 1 + · · · + Q k − j .

Original: To show the second inequality, it requires comparing the value between Q k and n (cid:52) n ( k ) , which are both problemdependent constants.
Revised: To show the second inequality, it requires comparing the value between Q k and n (cid:52) n ( k ) , which are both problemdependent constants.

Original: Here, the inequality holds when n (cid:52) n ( k ) ≤ Q k .
Revised: Here, the inequality holds when n (cid:52) n ( k ) ≤ Q k .

Original: If otherwise, we can always deﬁne ˜ Q k :=
Revised: If otherwise, we can always deﬁne ˜ Q k :=

Original: max { Q k , n (cid:52) n ( k ) } and the inequality still holds by changing all the Q k into ˜ Q k .
Revised: max { Q k , n (cid:52) n ( k ) } and the inequality still holds by changing all the Q k into ˜ Q k .

Original: We temporarily omit this step for now.
Revised: We temporarily omit this step for now.

Original: Since | ∂ l f
Revised: Since | ∂ l f

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤
Revised: ≤

Original: Q k
Revised: Q k

Original: [Equation]
Revised: [Equation]

Original: + · · · +
Revised: + · · · +

Original: Q k
Revised: Q k

Original: [Equation]
Revised: [Equation]

Original: j , the event (cid:110) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:111) implies the event { max i | ∂ l f i ( x k, 0 )
Revised: j , the event (cid:110) max i | ∂ l f i ( x k − j, 0 ) | ≥ (cid:80) kp = k − j Q p (cid:111) implies the event { max i | ∂ l f i ( x k, 0 )

Original: | ≥ Q k } , so the proof is completed.
Revised: | ≥ Q k } , so the proof is completed.

Original: Now we are ready to bound E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .
Revised: Now we are ready to bound E (cid:104)(cid:80) l large ( b 1 ) (cid:105) .

Original: Combining Step 1, 2 and 3 together, we prove the following Lemma G.4.
Revised: Combining Step 1, 2 and 3 together, we prove the following Lemma G.4.

Original: Lemma G.4.
Revised: Lemma G.4.

Original: Under Assumption 2.1, consider β 1 < √ β 2 , when k is large such that: k ≥ 4 ; β ( k − 1)
Revised: Under Assumption 2.1, consider β 1 < √ β 2 , when k is large such that: k ≥ 4 ; β ( k − 1)

Original: n≤ β n 1 √ k − 1 , we have the following result:
Revised: n≤ β n 1 √ k − 1 , we have the following result:

Original: [Equation]
Revised: [Equation]

Original: where δ 1 = (1 − β 2 )4 nρ 2 β n+ (cid:18) 1 √
Revised: where δ 1 = (1 − β 2 )4 nρ 2 β n+ (cid:18) 1 √

Original: [Equation]
Revised: [Equation]

Original: 2 −(cid:19) , which goes to 0 when β 2 goes to 1.
Revised: 2 −(cid:19) , which goes to 0 when β 2 goes to 1.

Original: Proof can be seen in Appendix G.4.
Revised: Proof can be seen in Appendix G.4.

Original: Now we bound the error terms: E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .
Revised: Now we bound the error terms: E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .

Original: Since they are multiplied by δ 1 , these two terms vanish when β 2 → 1 .
Revised: Since they are multiplied by δ 1 , these two terms vanish when β 2 → 1 .

Original: Bounding E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .
Revised: Bounding E (cid:104)(cid:80) l large { ( a 2 ) + ( b 2 ) } (cid:105) .

Original: We bound these two terms in the following Lemma G.5.
Revised: We bound these two terms in the following Lemma G.5.

Original: Lemma G.5.
Revised: Lemma G.5.

Original: Given all the history up to x k, 0 , we denote α as the index of the coordinate with the greatest gradient: α = arg max l =
Revised: Given all the history up to x k, 0 , we denote α as the index of the coordinate with the greatest gradient: α = arg max l =

Original: [Equation]
Revised: [Equation]

Original: , 2 , ··· ,d | ∂ l f ( x k, 0 ) | .
Revised: , 2 , ··· ,d | ∂ l f ( x k, 0 ) | .

Original: Under Assumption 2.1 and 2.2, consider β 1 < √ β 2 , when k is large such that: k ≥ 4 ; β ( k − 1) n
Revised: Under Assumption 2.1 and 2.2, consider β 1 < √ β 2 , when k is large such that: k ≥ 4 ; β ( k − 1) n

Original: [Equation]
Revised: [Equation]