Original: P ROVABLE  A DAPTIVITY IN A DAM
Revised: P ROVABLE B ENEFIT OF A DAPTIVITY IN A DAM

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Adaptive Moment Estimation (Adam) has been observed to converge faster than stochastic gradient descent (SGD) in practice.
Revised: Adaptive Moment Estimation (Adam) has been observed to converge faster than stochastic gradient descent (SGD) in practice.

Original: However, such an advantage has not been theoretically characterized – the existing convergence rate of Adam is no better than SGD.
Revised: However, such an advantage has not been theoretically characterized – the existing convergence rate of Adam is no better than SGD.

Original: We attribute this mismatch between theory and practice to a commonly used assumption: the smoothness is globally upper bounded by some constant L (called L -smooth condition ).
Revised: We attribute this mismatch between theory and practice to a commonly used assumption: the smoothness is globally upper bounded by some constant L (called L -smooth condition ).

Original: Specifically, compared to SGD, Adam adaptively chooses a learning rate better suited to the local smoothness .
Revised: Specifically, compared to SGD, Adam adaptively chooses a learning rate better suited to the local smoothness .

Original: This advantage becomes prominent when the local smoothness varies drastically across the domain, which however is hided under L -smooth condition.
Revised: This advantage becomes prominent when the local smoothness varies drastically across the domain, which however is hided under L -smooth condition.

Original: In this paper, we analyze the convergence of Adam under a condition called ( L 0 , L 1 ) -smooth condition, which allows the local smoothness to grow with the gradient norm.
Revised: In this paper, we analyze the convergence of Adam under a condition called ( L 0 , L 1 ) -smooth condition, which allows the local smoothness to grow with the gradient norm.

Original: This condition has been empirically verified to be more realistic for deep neural networks (Zhang et al., 2019a) than the L -smooth condition.
Revised: This condition has been empirically verified to be more realistic for deep neural networks (Zhang et al., 2019a) than the L -smooth condition.

Original: Under ( L 0 , L 1 ) -smooth condition, we establish the convergence for Adam with practical hyperparameters.
Revised: Under ( L 0 , L 1 ) -smooth condition, we establish the convergence for Adam with practical hyperparameters.

Original: As such, we argue that Adam can adapt to the local smoothness, justifying Adam’s  adaptivity .
Revised: As such, we argue that Adam can adapt to the local smoothness, justifying Adam’s benefit of adaptivity .

Original: In contrast, SGD can be arbitrarily slow under this condition.
Revised: In contrast, SGD can be arbitrarily slow under this condition.

Original: Hence, we theoretically characterize the advantage of Adam over SGD.
Revised: Hence, we theoretically characterize the advantage of Adam over SGD.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Machine learning tasks are often formulated as solving the following finite-sum problem.
Revised: Machine learning tasks are often formulated as solving the following finite-sum problem.

Original: [Equation]
Revised: [Equation]

Original: where { f i ( w ) } n − 1 i =0 is lower bounded, n denotes the number of samples or mini-batches, and w denotes the trainable parameters.
Revised: where { f i ( w ) } n − 1 i =0 is lower bounded, n denotes the number of samples or mini-batches, and w denotes the trainable parameters.

Original: Among various gradient-based optimizers, Stochastic Gradient
Revised: Among various gradient-based optimizers, Stochastic Gradient

Original: Descent (SGD) is a simple and popular method to solve Eq.
Revised: Descent (SGD) is a simple and popular method to solve Eq.

Original: However, Adaptive gradient methods including Adaptive Moment estimation (Adam) (Kingma & Ba, 2014) is recently observed to outperform SGD in modern deep neural tasks including GANs (Brock et al., 2018), BERTs (Kenton & Toutanova, 2019), GPTs (Brown et al., 2020) and ViTs (Dosovitskiy et al., 2020).
Revised: However, Adaptive gradient methods including Adaptive Moment estimation (Adam) (Kingma & Ba, 2014) is recently observed to outperform SGD in modern deep neural tasks including GANs (Brock et al., 2018), BERTs (Kenton & Toutanova, 2019), GPTs (Brown et al., 2020) and ViTs (Dosovitskiy et al., 2020).

Original: For instance, as reported in Figure 1.
Revised: For instance, as reported in Figure 1.

Original: (a): SGD converges much slower than Adam during the training of Transformers.
Revised: (a): SGD converges much slower than Adam during the training of Transformers.

Original: Similar phenomena are also reported in BERT training (Zhang et al., 2019b).
Revised: Similar phenomena are also reported in BERT training (Zhang et al., 2019b).

Original: The empirical success of Adam comes from its special update rules.
Revised: The empirical success of Adam comes from its special update rules.

Original: Firstly, it uses the heavy-ball momentum mechanism controlled by a hyperparameter β 1 .
Revised: Firstly, it uses the heavy-ball momentum mechanism controlled by a hyperparameter β 1 .

Original: Second, it uses an adaptive learning rate strategy.
Revised: Second, it uses an adaptive learning rate strategy.

Original: In particular, the learning rate of Adam contains exponential moving averages of past squared gradients, which is weighted by β 2 .
Revised: In particular, the learning rate of Adam contains exponential moving averages of past squared gradients, which is weighted by β 2 .

Original: Larger β 1 and β 2 will bring more history information  into the update.
Revised: Larger β 1 and β 2 will bring more gradient information of historical steps into the update.

Original: The update rule of Adam is given in Eq.
Revised: The update rule of Adam is given in Eq.

Original: (2) (presented later in Section 3).
Revised: (2) (presented later in Section 3).

Original: Despite its practical success, the theoretical understanding of Adam is limited.
Revised: Despite its practical success, the theoretical understanding of Adam is limited.

Original: For instance, the existing convergence rates of Adam are no better than that of SGD (Zhang et al., 2022; Shi et al., 2021; Défossez et al., 2020;
Revised: For instance, the existing convergence rates of Adam are no better than that of SGD (Zhang et al., 2022; Shi et al., 2021; Défossez et al., 2020;

Original: Zou et al., 2019; De et al., 2018;
Revised: Zou et al., 2019; De et al., 2018;

Original: Guo et al., 2021).
Revised: Guo et al., 2021).

Original: As such, there is a mismatch between Adam’s superior empirical performance and its theoretical understanding.
Revised: As such, there is a mismatch between Adam’s superior empirical performance and its theoretical understanding.

Original: To close the gap between theory and practice, we revisit the existing analyses for Adam.
Revised: To close the gap between theory and practice, we revisit the existing analyses for Adam.

Original: Current setups for the analysis of Adam fail to model the real-world applications: all of the convergence analyses of Adam are based on L -smooth condition, i.e., the Lipschitz coefficient of the gradient is globally bounded.
Revised: Current setups for the analysis of Adam fail to model the real-world applications: all of the convergence analyses of Adam are based on L -smooth condition, i.e., the Lipschitz coefficient of the gradient is globally bounded.

Original: However, it has been recently observed that L -smooth condition does not hold in many deep learning tasks (Zhang et al., 2019a; Crawshaw et al., 2022).
Revised: However, it has been recently observed that L -smooth condition does not hold in many deep learning tasks (Zhang et al., 2019a; Crawshaw et al., 2022).

Original: This gap in setting can obscure Adam’s superiority: different local Lipschitz coefficients of the gradient (local smoothness) may require different optimal learning rates in terms of convergence.
Revised: This gap in setting can obscure Adam’s superiority: different local Lipschitz coefficients of the gradient (local smoothness) may require different optimal learning rates in terms of convergence.

Original: However, the learning rate of
Revised: However, the learning rate of

Original: SGD is ignorant of the local smoothness along the training trajectory.
Revised: SGD is ignorant of the local smoothness along the training trajectory.

Original: If the local smoothness does not change sharply along the trajectory, the optimal learning rate does not change much.
Revised: If the local smoothness does not change sharply along the trajectory, the optimal learning rate does not change much.

Original: Then SGD can work well by selecting a learning rate through grid search.
Revised: Then SGD can work well by selecting a learning rate through grid search.

Original: However, when the local smoothness varies dramatically, a learning rate fits some points well may fit other points along the trajectory arbitrarily badly, which indicates SGD may converge arbitrarily slow (detailed discussions can be found in [Theorem 4, (Zhang et al., 2019a)] and Section 4.3 in this paper).
Revised: However, when the local smoothness varies dramatically, a learning rate fits some points well may fit other points along the trajectory arbitrarily badly, which indicates SGD may converge arbitrarily slow (detailed discussions can be found in [Theorem 4, (Zhang et al., 2019a)] and Section 4.3 in this paper).

Original: In contrast, Adam adapts the update according to the local information and does not suffer from such an issue.
Revised: In contrast, Adam adapts the update according to the local information and does not suffer from such an issue.

Original: Following the above methodology, we analyze the convergence of Adam under ( L 0 , L 1 )
Revised: Following the above methodology, we analyze the convergence of Adam under ( L 0 , L 1 )

Original: -smooth condition, which assumes the local smoothness (the spectral norm of the local Hesssian) to be upper bounded by L 1 · ( local gradient norm )
Revised: -smooth condition, which assumes the local smoothness (the spectral norm of the local Hesssian) to be upper bounded by L 1 · ( local gradient norm )

Original: + L 0 (Assumption 1).
Revised: + L 0 (Assumption 1).

Original: As the gradient norm can be unbounded, ( L 0 , L 1 ) -smooth condition allows the local smoothness to grow fiercely with the gradient norm.
Revised: As the gradient norm can be unbounded, ( L 0 , L 1 ) -smooth condition allows the local smoothness to grow fiercely with the gradient norm.

Original: Moreover, it has been demonstrated by (Zhang et al., 2019a; 2020a; Crawshaw et al., 2022) that for many practical neural networks, ( L 0 , L 1 ) -smooth condition more closely characterizes the optimization landscape (as illustrated by Figure 1 (b)) along the optimization trajectories.
Revised: Moreover, it has been demonstrated by (Zhang et al., 2019a; 2020a; Crawshaw et al., 2022) that for many practical neural networks, ( L 0 , L 1 ) -smooth condition more closely characterizes the optimization landscape (as illustrated by Figure 1 (b)) along the optimization trajectories.

Original: [Figure]
Revised: [Figure]

Original:  It can be observed that  the training process nearly satisfies local smoothness = e × gradient norm, which thus  obeys ( L 0 , L
Revised: ( gradient norm ) + 1 . 4 . It can be observed that all the (log( gradient norm ) , log( local smoothness )) points lie under this line, and thus the training process obeys (0 , e 1 . 4 )

Original: ) -smooth condition.
Revised: -smooth condition.

Original: Under ( L 0 , L 1 ) -smooth condition, we successfully establish the convergence of Adam.
Revised: Under ( L 0 , L 1 ) -smooth condition, we successfully establish the convergence of Adam.

Original: Meanwhile, in contrast, under the same assumption, it is proved that SGD can be arbitrarily slow in (Zhang et al., 2019a) or even diverge (see Section 4.3).
Revised: Meanwhile, in contrast, under the same assumption, it is proved that SGD can be arbitrarily slow in (Zhang et al., 2019a) or even diverge (see Section 4.3).

Original: Therefore, our theory demonstrates that Adam can provably converge faster than SGD.
Revised: Therefore, our theory demonstrates that Adam can provably converge faster than SGD.

Original: The main contribution of this paper is summarized as follows.
Revised: The main contribution of this paper is summarized as follows.

Original: We derive the first convergence result of Adam under  ( L 0 , L 1 )
Revised: We derive the first convergence result of Adam under the more realistic ( L 0 , L 1 )

Original: -smooth condition (Theorem 1).
Revised: -smooth condition (Theorem 1).

Original: First, our convergence result is established under the mildest assumption so far:
Revised: First, our convergence result is established under the mildest assumption so far:

Original: • ( L 0 , L 1 ) -smooth condition is strictly weaker than L -smooth condition. 
Revised: • ( L 0 , L 1 ) -smooth condition is strictly weaker than L -smooth condition. More importantly, ( L 0 , L 1 )

Revised: -smooth condition is observed to hold in practical deep learning tasks.

Original: Relaxing the smoothness condition is important to characterize the advantage of Adam over SGD.
Revised: Relaxing the smoothness condition is important to characterize the advantage of Adam over SGD.

Original: • Our result does not require the bounded gradient assumption (i.e. ∥∇ f ( x ) ∥ ≤ C ).
Revised: • Our result does not require the bounded gradient assumption (i e. ∥∇ f ( x ) ∥ ≤ C ).

Original: Removing such a condition is necessary, as otherwise ( L 0 , L 1 ) -smooth condition will degenerate to L -smooth condition.
Revised: Removing such a condition is necessary, as otherwise ( L 0 , L 1 ) -smooth condition will degenerate to L -smooth condition.

Original: Our result does not need other strong assumptions like bounded adaptor or large ε (see Eq.
Revised: Our result does not need other strong assumptions like bounded adaptor or large ε (see Eq.

Original: (2)), either.
Revised: (2)), either.

Original: Furthermore, the conclusion of our convergence result is among the strongest.
Revised: Furthermore, the conclusion of our convergence result is among the strongest.

Original: • Our convergence result holds for every possible trajectory.
Revised: • Our convergence result holds for every possible trajectory.

Original: This is much stronger than the common results of “convergence in expectation" and is technically challenging.
Revised: This is much stronger than the common results of “convergence in expectation" and is technically challenging.

Original: In our convergence results, the setting of hyperparameters ( β 1 , β 2 ) is close to practice.
Revised: In our convergence results, the setting of hyperparameters ( β 1 , β 2 ) is close to practice.

Original: Specifically, our result holds for any β 1 and any β 2 close to 1 , which matches the practical settings (for example, 0 . 9 and 0 . 999 in deep learning libraries).
Revised: Specifically, our result holds for any β 1 and any β 2 close to 1 , which matches the practical settings (for example, 0 . 9 and 0 . 999 in deep learning libraries).

Original: We also provide counter-examples where SGD diverges under the same setting as  Theorem 1.
Revised: We also provide counter-examples where SGD diverges under the same setting as in Theorem 1.

Original: Therefore, our results can shed new light on why Adam outperforms SGD in deep learning tasks.
Revised: Therefore, our results can shed new light on why Adam outperforms SGD in deep learning tasks.

Original: The rest of this paper is organized as follows: Section 2 summarizes the related works; Section 3, introduces notations and assumptions; Section 4 presents our convergence result of Adam along with the proof ideas; finally, Section 5 presents experiments to support our theoretical claims.
Revised: The rest of this paper is organized as follows: Section 2 summarizes the related works; Section 3, introduces notations and assumptions; Section 4 presents our convergence result of Adam along with the proof ideas.

Original:  R ELATED WORKS
Revised:  R ELATED WORKS

Original: Convergence analysis for Adam.
Revised: Convergence analysis for Adam.

Original: When Adam is firstly proposed in (Kingma & Ba, 2015), the authors provide a convergence proof.
Revised: When Adam is firstly proposed in (Kingma & Ba, 2015), the authors provide a convergence proof.

Original: However, Reddi et al.
Revised: However, Reddi et al.

Original: (2019) point out that the proof in (Kingma & Ba, 2015) has flaws.
Revised: (2019) point out that the proof in (Kingma & Ba, 2015) has flaws.

Original: They further provide simple counterexamples with which Adam diverges.
Revised: They further provide simple counterexamples with which Adam diverges.

Original: Ever since then, there have been many attempts to modify the update rules of Adam to ensure convergence: e.g., AMSGrad (Reddi et al., 2019) and AdaBound (Luo et al., 2019).
Revised: Ever since then, there have been many attempts to modify the update rules of Adam to ensure convergence: e.g., AMSGrad (Reddi et al., 2019) and AdaBound (Luo et al., 2019).

Original: Due to the limited space, we introduce more details of the Adam-variants in Appendix A.
Revised: Due to the limited space, we introduce more details of the Adam-variants in Appendix A.

Original: On the other hand, vanilla Adam works well in practice, and the divergence is not reported unless for carefully constructed examples.
Revised: On the other hand, vanilla Adam works well in practice, and the divergence is not reported unless for carefully constructed examples.

Original: This empirical phenomenon motivates researchers to rethink the counterexamples.
Revised: This empirical phenomenon motivates researchers to rethink the counterexamples.

Original: The counterexamples states “for every β 1 < √ β 2 , there exists a problem that
Revised: The counterexamples states “for every β 1 < √ β 2 , there exists a problem that

Original: Adam diverges".
Revised: Adam diverges".

Original: That is to say, the divergence statement requires picking ( β 1 , β 2 ) before fixing the problem, while in practice, the algorithmic parameters are often picked according to the problem.
Revised: That is to say, the divergence statement requires picking ( β 1 , β 2 ) before fixing the problem, while in practice, the algorithmic parameters are often picked according to the problem.

Original: Based on this observation, a recent work (Zhang et al., 2022) proves that Adam can converge with ( β 1 , β 2 ) picked after the problem is given.
Revised: Based on this observation, a recent work (Zhang et al., 2022) proves that Adam can converge with ( β 1 , β 2 ) picked after the problem is given.

Original: Similar analyses on vanilla Adam and RMSProp are also shown under various conditions in (Zaheer et al., 2018b; Zou et al., 2019; Défossez et al., 2020; De et al., 2018;
Revised: Similar analyses on vanilla Adam and RMSProp are also shown under various conditions in (Zaheer et al., 2018b; Zou et al., 2019; Défossez et al., 2020; De et al., 2018;

Original: Guo et al., 2021; Shi et al., 2021).
Revised: Guo et al., 2021; Shi et al., 2021).

Original: We list and compare these results in Table 1 for convenience.
Revised: We list and compare these results in Table 1 for convenience.

Original: In summary, we emphasize that all the above works (including those for Adam-variants) require L -smooth condition.
Revised: In summary, we emphasize that all the above works (including those for Adam-variants) require L -smooth condition.

Original: In addition, most of them require stronger assumptions such as bounded gradient or large ϵ . 
Revised: In addition, most of them require stronger assumptions such as bounded gradient or large ϵ . ( L 0 , L 1 ) -Smoothness Allow β 1 ̸ =

Original: [Figure]
Revised: [Table]

Original: ( a ) Allow {+ε = 
+}
Revised: ( a ) Allow {+ε = 
+}

Original: [Table] [Table] Zaheer et al.
Revised: ( b ) Allow unbounded gradient Trajectory-wise convergence Zaheer et al.

Original: (2018b)
Revised: (2018b)

Original: × × × × × De et al.
Revised: × × × × × De et al.

Original: (2018) ( c ) × × × × × Défossez et al.
Revised: (2018) ( c ) × × × × × Défossez et al.

Original: × ✓ × × × Guo et al.
Revised: × ✓ × × × Guo et al.

Original: (2021) ( d ) × ✓ × × ×
Revised: (2021) ( d ) × ✓ × × ×

Original: Huang et al.
Revised: Huang et al.

Original: (2021) ( d ) × ✓ × × × Shi et al.
Revised: (2021) ( d ) × ✓ × × × Shi et al.

Original: (2021) × × ✓ ✓ ✓ Zhang et al.
Revised: (2021) × × ✓ ✓ ✓ Zhang et al.

Original: × ✓ ✓ ✓ ×
Revised: × ✓ ✓ ✓ ×

Original: This work ✓ ✓ ✓ ✓ ✓
Revised: This work ✓ ✓ ✓ ✓ ✓

Original: In this work, we focus on the convergence of vanilla Adam.
Revised: In this work, we focus on the convergence of vanilla Adam.

Original: In particular, our result is the first to relax L -smooth condition for Adam.
Revised: In particular, our result is the first to relax L -smooth condition for Adam.

Original: Moreover our convergence analysis of Adam is proved under the mildest assumptions so far and with the strongest conclusions, which can be easily extended to other Adam-variants as well.
Revised: Moreover our convergence analysis of Adam is proved under the mildest assumptions so far and with the strongest conclusions, which can be easily extended to other Adam-variants as well.

Original: Generalized smoothness assumption.
Revised: Generalized smoothness assumption.

Original: There are several attempts on relaxing L -smooth condition.
Revised: There are several attempts on relaxing L -smooth condition.

Original: Zhang et al. (2019a) propose ( L 0 , L 1 ) -smooth condition to theoretically explain the acceleration effect of clipped SGD over SGD.
Revised: Zhang et al. (2019a) propose ( L 0 , L 1 ) -smooth condition to theoretically explain the acceleration effect of clipped SGD over SGD.

Original: Similar results are also extended to clipped SGD with momentum (Zhang et al., 2020a) and differentially-private SGD (DP SGD)
Revised: Similar results are also extended to clipped SGD with momentum (Zhang et al., 2020a) and differentially-private SGD (DP SGD)

Original: (Yang et al., 2022).
Revised: (Yang et al., 2022).

Original: Through extensive experiments, these works empirically showed that ( L 0 , L 1 ) -smooth condition holds when Adam outperforms SGD.
Revised: Through extensive experiments, these works empirically showed that ( L 0 , L 1 ) -smooth condition holds when Adam outperforms SGD.

Original: However, they did not theoretically analyze Adam in this setting.
Revised: However, they did not theoretically analyze Adam in this setting.

Original: Theoretical comparison between adaptive gradient methods and SGD.
Revised: Theoretical comparison between adaptive gradient methods and SGD.

Original: The comparison between adaptive gradient methods and SGD is a popular topic.
Revised: The comparison between adaptive gradient methods and SGD is a popular topic.

Original: There are several theoretical works from different perspectives.
Revised: There are several theoretical works from different perspectives.

Original: Wardet al.
Revised: Ward

Original: ; Défossez et  al.
Revised: ); Défossez et et al.

Original: Faw et al.

Original: (2022) show that AdaGrad can converge with any constant learning rate.
Revised: ) show that AdaGrad can converge with any constant learning rate.

Original: They argue that “tuning-free" is one advantage of
Revised: They argue that “tuning-free" is one advantage of

Original: AdaGrad over SGD.
Revised: AdaGrad over SGD.

Original: This line of works has the following differences with us.
Revised: This line of works has the following differences with us.

Original: First, they analyze
Revised: First, they analyze

Original: AdaGrad while we focus on Adam.
Revised: AdaGrad while we focus on Adam.

Original: Second, they require L -smooth, while we do not have this condition.
Revised: Second, they require L -smooth, while we do not have this condition.

Original: Third, their proposed convergence rate is no better than SGD, while we show Adam can converge faster.
Revised: Third, their proposed convergence rate is no better than SGD, while we show Adam can converge faster.

Original: Xie et al.
Revised: Xie et al.

Original: (2022) study the escaping rate from saddle points of the continuous dynamics approximation of Adam and SGD.
Revised: (2022) study the escaping rate from saddle points of the continuous dynamics approximation of Adam and SGD.

Original: They show that adaptive learning rate can help escape saddle points efficiently.
Revised: They show that adaptive learning rate can help escape saddle points efficiently.

Original: This work is orthogonal to ours: they compare the behavior of Adam and SGD after a certain stationary point is reached, while we focus on comparing the iteration complexity of approaching the stationary points.
Revised: This work is orthogonal to ours: they compare the behavior of Adam and SGD after a certain stationary point is reached, while we focus on comparing the iteration complexity of approaching the stationary points.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2020b) prove that the (adaptive) clipped gradient method converges faster than SGD when the stochastic noises are heavy-tailed.
Revised: (2020b) prove that the (adaptive) clipped gradient method converges faster than SGD when the stochastic noises are heavy-tailed.

Original: As such, they argue that the heavy-tailed noise is one cause of SGD’s poor performance.
Revised: As such, they argue that the heavy-tailed noise is one cause of SGD’s poor performance.

Original: However, a recent work (Chen et al., 2021b) shows that (S)GD still performs poorly even when the stochastic noise is removed in the training process (by using the full-batch gradient), which shows that the effect of the stochastic noise may not be crucial.
Revised: However, a recent work (Chen et al., 2021b) shows that (S)GD still performs poorly even when the stochastic noise is removed in the training process (by using the full-batch gradient), which shows that the effect of the stochastic noise may not be crucial.

Original: There are also recent works (Zhou et al., 2020; Zou et al., 2021) trying to compare the generalization behaviors of adaptive gradient methods and SGD.
Revised: There are also recent works (Zhou et al., 2020; Zou et al., 2021) trying to compare the generalization behaviors of adaptive gradient methods and SGD.

Original: These works are orthogonal to the topic of this work.
Revised: These works are orthogonal to the topic of this work.

Original: More detailed discussion is deferred to Appendix A.
Revised: More detailed discussion is deferred to Appendix A.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: This section introduces notations, definitions, and assumptions that are used throughout this work.
Revised: This section introduces notations, definitions, and assumptions that are used throughout this work.

Original: Notations.
Revised: Notations.

Original: We list the notations that are used in the formal definition of the randomly-shuffled Adam and its convergence analysis.
Revised: We list the notations that are used in the formal definition of the randomly-shuffled Adam and its convergence analysis.

Original: • (Vector) We define a ⊙ b as the Hadamard product (i
Revised: • (Vector) We define a ⊙ b as the Hadamard product (i

Original: e., component-wise product) between two vectors a and b with the same dimension.
Revised: e., component-wise product) between two vectors a and b with the same dimension.

Original: We also define ⟨ a , b ⟩ as the ℓ 2 inner product between a and b , and ∥ a ∥ p as the ℓ p norm of a (specifically, we abbreviate the ℓ 2 norm of a as ∥ a ∥ ).
Revised: We also define ⟨ a , b ⟩ as the ℓ 2 inner product between a and b , and ∥ a ∥ p as the ℓ p norm of a (specifically, we abbreviate the ℓ 2 norm of a as ∥ a ∥ ).

Original: We define 1 d as an all-one vector with dimension d . • (Derivative) For a function f ( w ) :
Revised: We define 1 d as an all-one vector with dimension d . • (Derivative) For a function f ( w ) :

Original: R d → R , we define ∇ f ( w ) as the gradient of f at point w , and ∂ l f ( w ) as the l -th partial derivative of f at point w , i.e., ∂ l f ( w ) = ( ∇ f ( w ))
Revised: R d → R , we define ∇ f ( w ) as the gradient of f at point w , and ∂ l f ( w ) as the l -th partial derivative of f at point w , i.e., ∂ l f ( w ) = ( ∇ f ( w ))

Original: l . • (Array)
Revised: l . • (Array)

Original: We define [ m 1 , m 2 ] ≜ { m 1 , · · · , m 2 } , ∀ m 1 , m 2 ∈ N , m 1 ≤ m 2 .
Revised: We define [ m 1 , m 2 ] ≜ { m 1 , · · · , m 2 } , ∀ m 1 , m 2 ∈ N , m 1 ≤ m 2 .

Original: Specifically, we use [ m ] ≜ { 1 , · · · , m } .
Revised: Specifically, we use [ m ] ≜ { 1 , · · · , m } .

Original: Formal Definition of Adam.
Revised: Formal Definition of Adam.

Original: Based on the n -sum problem Eq.
Revised: Based on the n -sum problem Eq.

Original: (1), we provide the update rule of
Revised: (1), we provide the update rule of

Original: Adam as follows.
Revised: Adam as follows.

Original: We initialize w 1 , 0 , m 1 , − 1 , and ν 1 , − 1 as any point in R d .
Revised: We initialize w 1 , 0 , m 1 , − 1 , and ν 1 , − 1 as any point in R d .

Original: At the beginning of every outer loop k ∈ N + , we sample { τ k, 0 , · · · , τ k,n − 1 } as a random permutation of { 0 , 1 , · · · , n − 1 } .
Revised: At the beginning of every outer loop k ∈ N + , we sample { τ k, 0 , · · · , τ k,n − 1 } as a random permutation of { 0 , 1 , · · · , n − 1 } .

Original: Then, in each inner loop i ∈ [0 , n − 1] , we respectively calculate the 1 st -order momentum m k,i , the 2 st -order momentum ν k,i , and the parameter w k,i +1 as 
Revised: Then, in each inner loop i ∈ [0 , n − 1] , we respectively calculate the 1 st -order momentum m k,i , the 2 st -order momentum ν k,i , and the parameter w k,i +1 as [Equation]

Original: [Equation]
Revised: [Equation]

Original: In the end of the outer loop k , Adam updates w k +1 , 0 = w k,n , ν k +1 , − 1 = ν k,n − 1 , m k +1 , − 

Revised: In the end of the outer loop k , Adam updates w k [Equation] [Equation] k,n − 1 , m k +1 , −

Original: = m k,n − 1 as a preparation for the next outer loop.
Revised: = m k,n − 1 as a preparation for the next outer loop.

Original: m k,i and ν k,i are weighted averages with hyperparamter β 1 ∈ [0 , 1) and β 2 ∈ [0 , 1) , respectively.
Revised: m k,i and ν k,i are weighted averages with hyperparamter β 1 ∈ [0 , 1) and β 2 ∈ [0 , 1) , respectively.

Original: We choose η k to be diminishing learning rate η k = η 1 √ k.
Revised: We choose η k to be diminishing learning rate η k = η 1 √ k

Original: In practice, ε is adopted for numerical stability and it is often chosen to be 10 − 8 in practice.
Revised: In practice, ε is adopted for numerical stability and it is often chosen to be 10 − 8 in practice.

Original: In our theory, we allow ε to be an arbitrary non-negative constant including 0.
Revised: In our theory, we allow ε to be an arbitrary non-negative constant including 0.

Original: We further emphasize that thatrandom shuffling is not a key factor in our analysis.
Revised: We further emphasize that that our analysis holds for any random permutation order used to generate τ k,i .

Original: The proof also works for other settings.
Revised: In the default setting in deep learning libraries, the pure Adam is implemented in the random shuffling fashion and it is the default setting for computer vision, NLP, and generative models.

Original: We make two mild assumptions on the objective function (Eq.
Revised: We make two mild assumptions on the objective function (Eq.

Original: Assumption 1 ( ( L 0 , L 1 ) -smooth condition) .
Revised: Assumption 1 ( ( L 0 , L 1 ) -smooth condition) .

Original: f i ( w ) satisfies ( L 0 , L 1 ) -smooth condition, i.e., there exists positive constants ( L 0 , L 1 ), such that, ∀ 
Revised: f i ( w ) satisfies ( L 0 , L 1 ) -smooth condition, i.e., there exist positive constants ( L 0 , L 1 ), such that, ∀ w 1 , w 2 ∈ R d satisfying ∥ w 1 − w 2 ∥ ≤ 1 L 1 ,

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: (3) is firstly introduced by Zhang et al. (2019a).
Revised: (3) is firstly introduced by Zhang et al. (2019a).

Original: It is strictly weaker than the classical Lsmooth condition (i.e., L 1 = 0 in Assumption 3, see [Remark 2.3, (Zhang et al., 2020a)] for details).
Revised: It is strictly weaker than the classical Lsmooth condition (i.e., L 1 = 0 in Assumption 3, see [Remark 2.3, (Zhang et al., 2020a)] for details).

Original: For instance, Eq.
Revised: For instance, Eq.

Original: (3) holds for a wide range of polynomials and even the exponential functions, while L -smooth condition does not hold for polynomials with degree larger than 2.
Revised: (3) holds for a wide range of polynomials and even the exponential functions, while L -smooth condition does not hold for polynomials with degree larger than 2.

Original: More importantly, empirical observation in (Zhang et al., 2019a; 2020a) demonstrates that Eq.
Revised: More importantly, empirical observation in (Zhang et al., 2019a; 2020a) demonstrates that Eq.

Original: (3) is a better characterization of the loss landscape of neural networks than L -smooth condition, especially in the tasks where Adam significantly outperforms SGD.
Revised: (3) is a better characterization of the loss landscape of neural networks than L -smooth condition, especially in the tasks where Adam significantly outperforms SGD.

Original: Assumption 2 (Affine Noise Variance) .
Revised: Assumption 2 (Affine Noise Variance) .

Original: ∀ w ∈ R d , the gradients of { f i ( w ) } n − 1 i =0 has the following connection with the gradient of f ( w ) :
Revised: ∀ w ∈ R d , the gradients of { f i ( w ) } n − 1 i =0 has the following connection with the gradient of f ( w ) :

Original: [Equation]
Revised: [Equation]

Original: Assumption 2 is quite general. When D 1 = 1 /n , it becomes the “bounded variance” with constant
Revised: Assumption 2 is quite general, which covers the "bounded variance" assumption (Ghadimi et al., 2016; Zaheer et al., 2018a;

Original: D 0 /n . The “bounded variance” condition is common in both SGD and Adam analysis (e.g. (Ghadimi et al., 2016; Zaheer et al., 2018a; Huang et al., 2021)). Assumption 2 allows more flexible choices of
Revised: Huang et al., 2021) and the "strongly growth condition" in existing literature (Schmidt & Roux, 2013; Vaswani et al., 2019).

Original: D 1 ̸ = n and thus it is strictly weaker than bounded variance. When D 0 > 0 , theobjective function is often called “non-realizable” (Shi et al., 2021; Zhang et al., 2022). In this case, adaptive gradient methods, even withdiminishing learning rate, cannot reach the exact stationary points. Instead, they only converge to a bounded region containing stationary points (Zaheer et al., 2018a;
Revised: Compare to the "bounded variance" assumption where the noise diminishes when the gradient is large, Assumption 2 allows the noise variance to grow with the gradient norm, and thus applies to a rich variety of problems (e.g., (Khani & Liang, 2020)).

Original: Shi et al., 2021; Zhang et al., 2022). This phenomenon is also observed empirically (see Figure 2). When D 0 = 0 , Assumption 2 becomes (cid:80) n − 1 j =0 ∥∇ f j ( x ) ∥ 22 ≤ D 1∥∇ f ( x ) ∥ 22 .
Revised: Furthermore, combining Assumptions 1 and 2, we conclude that f satisfies ( nL 0 + L 1 √ n √ D 0 , L 1 √ n √ D 1 ) -smooth condition. The proof is immediate and we defer it to

Original: This is often called “strong growth condition” (SGC) (Schmidt & Roux, 2013; Vaswani et al., 2019). SGC is increasingly popular recently as it is known to be reasonable in the overparameterized regime where neural networks can interpolate all data points (Vaswani et al., 2019). We will show that Adam can converge to stationary points if D 0 = 0 .
Revised: Lemma 2 in the appendix.

Original:  M AIN RESULT 4. A DAM CONVERGES WITH ( L 0 , L 1 ) - SMOOTH CONDITION
Revised:  M AIN RESULT 4. A DAM CONVERGES WITH ( L 0 , L 1 ) - SMOOTH CONDITION

Original: We formally present our main result as follows:
Revised: We formally present our main result as follows:

Original: Theorem 1.
Revised: Theorem 1.

Original: Consider Adam defined as Eq.
Revised: Consider Adam defined as Eq.

Original: (2) with diminishing learning rate η k = η 1 √ k .
Revised: (2) with diminishing learning rate η k = η 1 √ k .

Original: Let
Revised: Let

Original: Assumptions 1 and 2 hold.
Revised: Assumptions 1 and 2 hold.

Original: Suppose that β 1 and β 2 satisfies β 21 < β [-
-] and
Revised: Suppose that β 1 and β 2 satisfies β 21 < β [-
-]2 < 1 and

Original: [Equation]
Revised: [Equation]

Original: Then, we have that
Revised: Then, we have that

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 shows that Adam converges to the neighborhood of stationary points under ( L 0 , L 1 ) - smooth condition.
Revised: Theorem 1 shows that Adam converges to the neighborhood of stationary points under ( L 0 , L 1 ) - smooth condition.

Original: In Section 4.2, we present a proof sketch to illustrate our proof idea
Revised: In Section 4.2, we present a proof sketch to illustrate our proof idea.

Original: [Equation] Remark 1 (The choice of hyperparameters ( β 1 , β 2 ) ) .
Revised: Remark 1 (The choice of hyperparameters ( β 1 , β 2 ) ) .

Original: In Theorem 1, the feasible range of ( β 1 , β 2 ) subject to the two constraints: β 21 < β 2  and Inequality (4).
Revised: In Theorem 1, the feasible range of ( β 1 , β 2 ) subject to the two constraints: β 21 < β 2 < 1 and Inequality (4).

Original: We emphasize that the intersection of these two constraints is not empty.
Revised: We emphasize that the intersection of these two constraints is not empty.

Original: One can easily observe that δ ( β 2 ) → 0 as β 2 → 1 , and thus the constraints hold when β 1 is fixed and β 2 approaches 1 .
Revised: One can easily observe that δ ( β 2 ) → 0 as β 2 → 1 , and thus the constraints hold when β 1 is fixed and β 2 approaches 1 .

Original: Therefore, Theorem 1 indicates that Adam can work when β 2 close enough to 1 .
Revised: Therefore, Theorem 1 indicates that Adam can work when β 2 close enough to 1 .

Revised: On the other hand, there is little restriction on β 1 except β 21 < β 2 (discussed in Appendix B).

Original: This agrees with β 2 selection in deep learning libraries (e.g., 0 .
Revised: This agrees with β 2 selection in deep learning libraries (e.g., 0 .

Original: 999 in Pytorch).
Revised: 999 in Pytorch).

Original: We discuss a bit more on β 1 . First, The requirement β 21 < β 2 is a standard condition in Adam-family literature (Reddi et al., 2018; Zou et al., 2019; Défossez et al., 2020; Shi et al., 2021). Since βis suggested to be large, this condition covers flexible choice of β  1 ∈ [0 , 1) .
Revised: On the other hand, there are counterexamples in (Zhang et al., 2022), where Adam is showed to diverge if β 2 are chosen improperly (i.e., not close to 1 ).

Original: Second, although the dependency between β 1 and β 2in Inequality (4) seems complicated, the effect of β 1 on Inequality (4) can be omitted when adopting the practical value such as β 1 = 0 . This is because the term “ 1+ β 1 1 − β 1" is way smaller than the term “ ( n − 1) " in the right-hand-side of Inequality (4). For example, when β 1 = 0 . 9 (the default setting in deep learning libraries), 1+ β 1 1 − β 1 = 19 .
Revised: Therefore, the condition that β 2 is close to 1 is both sufficient and necessary for the convergence of Adam.

Original: This number is much smaller than n on real datasets (for instance, on CIFAR-10 (He et al., 2016), n = 50 k/ 128 ≈ 390 ; on ImageNet (You et al., 2017), n = 1 . 2 m/ 8 k = 150 ).

Original: As a result, constraint (4) is inactive for practical choice of β 1 .

Original: Remark 2 (Convergence to the neighborhood of stationary points) .
Revised: Remark 2 (Convergence to the neighborhood of stationary points) .

Original: When D 0 ̸ = 0 , our theorem can only ensure that Adam converges to a neighborhood of stationary points.
Revised: When D 0 ̸ = 0 , our theorem can only ensure that Adam converges to a neighborhood of stationary points.

Original: As discussed under
Revised: As discussed under

Original: Assumption 2, convergence to the bounded region is common for Adam analysis: even with L -smooth condition, Zhang et al.
Revised: Assumption 2, convergence to the bounded region is common for Adam analysis: even with L -smooth condition, Zhang et al.

Original: (2022) can only show that Adam with diminishing learning rates converges to a bounded region.
Revised: (2022) can only show that Adam with diminishing learning rates converges to a bounded region.

Original: This is because even with diminishing η k , the effective learning rate η k ε 1 d + √ ν k,i may not decay.
Revised: This is because even with diminishing η k , the effective learning rate [Equation] may not decay.

Original: We further conduct experiments and observe that Adam cannot reach the stationary point over a simple quadratic function with D 0 ̸ = [-
-]
Revised: We further conduct experiments and observe that Adam cannot reach the stationary point over a simple quadratic function with D 0 ̸ = [-
-]0 . Specifically, we run Adam on a synthetic optimization target which is given as follows:

Original: (see Figure 2).
Revised: [Equation] Such an example is proposed by (Shi et al., 2021). One can easily observe that f ( x ) and { f i ( x ) } 9 i = satisfies Assumption 1 and Assumption 2 with D 1 = 208 , D 0 = 75 59 , thus Theorem 1 applies. With the diminishing learning rate η t = 1 √ t , we plot the distance between Adam’s trajectory and the unique minimum 0 of f in Figure 2.

Original:  The good news isthat the neighborhood shrinks with β 2 → 1 . This also explains why large β 2 is preferred in practice. We further comment on a bit on the size of the region.
Revised: It can be observed that Adam does not converge to the unique minimum 0 , but gets closer to the minimum if choosing β 2 closer to 1 , which supports Theorem 1. The good news is the neighborhood shrinks as β 2 → 1 , which explains the practical use of large β 2 .

Original: We refine the bounded region { w : min { ∥∇f ( w ))
Revised: Furthermore, we refine the bounded region { w : min { ∥∇

Original: ∥ √ D, ∥∇ f ( w )∥ 2 √ D} ≤ O ( √ D 0 δ ( β 2 )) } in existing literature (Shi et al., 2021; Zhang et al., 2022) to { w : min { ∥∇ f ( w ))
Revised: [Equation] ) } in existing literature (Shi et al., 2021; Zhang et al., 2022) to { w : min { ∥∇ f ( w ))

Original: ∥ √ D, ∥∇ f ( w ) ∥ 2 √ D} ≤ O ( √ D 0 min { δ ( β 2 ) , δ ( β 2 ) 2 } ) } through a more careful analysis.
Revised: ∥ √ D, ∥∇ f ( w ) ∥ 2 √ D} ≤ O ( √ D 0 min { δ ( β 2 ) , δ ( β 2 ) 2 } ) } through a more careful analysis.

Original: We emphasize that this change contributes to a sharp improvement since δ ( β 2 ) is required to be close to 0.
Revised: We emphasize that this change contributes to a sharp improvement since δ ( β 2 ) is required to be close to 0.

Original: When D 0 [-= 
-]
Revised: Remark 3 (Dependence on the dimension d ) . The convergence rate and the neighborhood have polynomial dependence over the dimension d , which is not desired as the convergence rate of SGD under L -smooth condition does not have such a dependence. We conjecture that such a dependence may be due to the limit of proof techniques. However, such a result suffices to provide a convergence rate separation between Adam and SGD since we will latter show that SGD may converge arbitrarily slow under ( L 0 [-= 
-], L 1 )

Original: (i.e., under strong growth condition), Theorem 1 further indicates Adam will converge to stationary points.
Revised: -smooth condition. Furthermore to the best of our knowledge, all the existing convergence analysis of Adam has the dependence of d even under L -smooth condition e.g., (Défossez et al., 2020; Shi et al., 2021; Zhang et al., 2022).

Original: We present this result in Corollary 1.
Revised: Though we believe that removing the dependence on d is possible and worthy studying, it is technically difficult and deserves an independent work. Remark 4 (Constant learning rate) . Following the same proof strategy of Theorem 1, one can show that with constant learning rate η , the conclusion (Eq. (5)) of Theorem 1 changes into min k ∈ [1 ,T ] { ∥∇ f ( w k, 0 ) ∥

Original: Corollary 1.
Revised: √ D, ∥∇ f ( w k, 0 ) ∥ 2 √ D} ≤ O ( log T ηT )+ O ( √ D 0 min { δ ( β 2 ) , δ ( β 2 ) 2 } ) + O ( η ) . In other words, Adam will converge faster to the neighborhood (with rate 1 / √ t → 1 /t ), but the size of neighborhood is enlarged by an additional term O ( η ) due to the constant step-size.

Original: Let all the conditions in Theorem 1 hold and [-further assume D 
-]
Revised: Remark 5. Theorem 1 allows both β 1 = 0 (RMSprop) and [-further assume D 
-]β 1 > 0 (Adam).

Original: =
Revised: However, the convergence rate of these two cases are not distinguished, showing no benefit of momentum so far. At the current stage, we take a first step to compare RMSProp & Adam to SGD. A more detailed comparison between RMSprop and Adam will be considered as future work.

Original: 0 , we have [Equation]
Revised: For completeness, we briefly discuss on the difficulty in analyzing the effect of momentum in Appendix B.

Original: 4. P ROOF IDEAS
Revised: 4. P ROOF IDEAS

Original: Here, we briefly explain our proof idea for the convergence result.
Revised: Here, we briefly explain our proof idea for the convergence result.

Original: Specifically, we identify the challenges and  solve them one by one.
Revised: We will first prove Theorem 1 for RMSProp (i.e., β 1 = 0 ) to show the challenge brought by ( L 0 , L 1 ) -smooth condition and how we tackle it. We then show the additional difficulty when extending the result to general Adam and our corresponding intuition to solve it.

Original: Similar to other analyses of gradient-based optimizers, we start with the following descent lemma:
Revised: Stage I: Convergence of RMSProp. We start with the following descent lemma:

Original: [Equation]
Revised: [Equation]

Original: Second [Equation] where L loc is the local smoothness, and is bounded by  ( L 0 + L 1 ∥∇ f ( w k, 0 ) ∥ ) due to ( L 0 , L 1 )
Revised: where L loc is the local smoothness. We first bound the second order term by noticing that the norm of the epoch- k update of RMSProp is in order O ( η k )

Original: {+when 
 = β 21 < β 2 < 1 (a standard property with proof deferred to Lemma 3 in the appendix). When k is large enough, the epoch- k update is small enough and ( L 0 , L 1 )+} -smooth condition (Assumption 1).
Revised: {+when 
 = β 21 < β 2 < 1 (a standard property with proof deferred to Lemma 3 in the appendix). When k is large enough, the epoch- k update is small enough and ( L 0 , L 1 )+} -smooth condition applies, leading to L loc =

Original: We would like to show that (i) the first order term is negative andthat (ii) the first order term dominates the change of loss f ( wk +1 , 0 ) − f ( w k, 0 ) .
Revised: O (1) + O ( ∥∇ f ( w ) ∥ ) and the second order term = O ( η 2 k ) + O ( η 2 k ∥∇ f ( w ) ∥ ) . Therefore, in order to show f ( w k, 0 ) decreases, we need to prove the first order is negative and dominates the term

Revised: O ( η 2 k )

Revised: + O ( η 2 k ∥∇ f ( w ) ∥ ) .

Original: Proving (i) is equivalent to proving the alignment between the update w k +1 , 0 − w k, 0 = − η k (cid:80) i [-
-]
Revised: In other words, we need to lower bound the alignment between the update w k +1 , 0 − w k, 0 = − η k (cid:80) i [-
-]1 ε 1 d + √ ν k,i ⊙ ∇ f τ k,i ( w k,i ) and the negative gradient −∇ f ( w k, 0 ) . It is easy to prove such a property when ν k,i does not change, i.e., if we are analyzing SGD with preconditioning, in which case [Equation] ⊙∇ f ( w k, 0 ) .

Revised: Can we extend the above methodology to RMSProp?

Revised: We give an affirmative answer when β 2 is close to 1 and the gradient is large by providing the following lemma.

Original: ε 1  d + √ ν k,i ⊙ m k,i and the negative gradient −∇ f ( w k, 0 ) .
Revised: Lemma 1 (Informal) . For any l ∈ [ d ] and i ∈ [0 , n − 1] , if max p ∈ [0 ,n − 1] | ∂ l f p [Equation] w k, 0 ) | =

Revised: Ω

Revised: [Equation]

Revised: (cid:80)

Original: As the momentum m k,i and the adaptor 1 ε 1 d + √ ν k,i are both stochastic and they are highly statistically correlated, we first decouple the correlation between the momentum and the adaptor following the routine approach: we control the change of ν k,i within one epoch by setting β 2 close to 1 .
Revised: [Equation] , then | ν l [Equation] k

Revised: [Equation]

Revised: i −

Revised: [Equation]

Revised: l

Original: Specifically, as ν k,i = 
Revised: [Equation] k [Equation] = O

Revised: [Equation]

Revised: δ

Revised: [Equation]

Revised: β

Original: β 2 ν k,i − 1 + (1 − β 2 ) ∇ f τ k,i ( w k,i ) ⊙ ∇ f τ k,i ( w k,i ) , setting β 2 close enough to 1 leads to ν k,i ≈ ν 
Revised: [Equation] ν l [Equation] k

Original: k,i − 1 ≈ · · · ≈ ν k, 0 .
Revised: [Equation] The proof idea is that (1). the squared maximum gradient can be bounded by ν l,k, 0 with an error term.

Original: Consequently, Adam with β 2 close enough to 1 canthen be viewed as momentum SGD with coordinate-wise but constant learning rate{+only bounded by ν l,k, 0 ; (3). thus ν l,k,i = β i 2 ν l,k, 0 + (1 − β 2 ) β i − 
+}
Revised: Thus when the maximum gradient is larger than the error term, it can be {+only bounded by ν l,k, 0 ; (3). thus ν l,k,i = β i 2 ν l,k, 0 + (1 − β 2 ) β i − 
+}

Original: [Equation]
Revised: [Equation]

Original: within the k -th epoch (with close-enough-to- 1 β 2 ), i.e.,
Revised: 0 ) gets close to ν l,k, 0 when β 2 is close to 1 . The detailed proof is sophisticated due to the presence of ( L 0 , L 1 ) -smooth condition and we defer it to Corollary 2 for details. Therefore, if we denote those dimensions with large gradients (i.e., satisfying the requirement of Lemma 1) as L klarge and the rest as L ksmall , Lemma 1 indicates

Original: [Equation]
Revised: [Equation]

Revised: Here in the second " ≈ ", we use again the squared maximum gradient is close to ν l,k, 0 if it is large.

Revised: A formal derivation of the above result can be seen in Appendix D.2.

Revised: How about the dimensions in L ksmall ?

Revised: We will treat them as error terms.

Revised: Specifically, l ∈ L ksmall indicates that ∂ l f ( w k, 0 ) =

Revised: O ( (cid:80) k − 1 r =1 β ( k − 1 − r ) 2η r

Revised: ∥∇ f ( w r, 0 ) ∥ + η k ) .

Revised: One can easily observe that ∂ l f τk,i ( w k,i ) √ ν l,k,i + ε is bounded, and thus (cid:80) l ∈ L ksmall ( w l,k +1 , 0 − w l,k, 0 )

Revised: ∂

Original: It remains toshow − η k ⟨ (cid:80) i m k,i √ ν k, 0 + ε , −∇ f ( w k, 0 ) ⟩ > 0 .
Revised: l f ( w k, 0 ) equals to − η k (cid:88) l ∈ L klarge ∂ l f ( w k, 0 ) √ [Equation] k [Equation]

Original: Nevertheless, such a claim is not necessarily correct, as the momentum m k,i contains a heavy historical signal, and may push the update away from the negative gradient direction.
Revised: i [Equation] ε (cid:88) i

Revised: [Equation]

Revised: l f τ k,i

Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: ,i

Revised: [Equation]

Revised: O (cid:32)

Revised: [Equation]

Revised: k −

Revised: [Equation]

Revised: r = β ( k − 1 − r ) 2η r ∥∇ f ( w r, 0 ) ∥

Revised: + η k (cid:33)(cid:33) .

Original: We consider this problem as our first technical challenge.
Revised: Putting all the estimates together, we conclude that f ( w k +1 , 0 ) − f ( w k, 0 ) is smaller than − Ω( η k min { ∥∇ f ( w k, 0 ) ∥ √ D, ∥∇ f ( w k, 0 ) ∥ 2 √ D} ) + O ( η k ( (cid:80) kr =1 β ( k − 1 − r ) 2η r ∥∇ f ( w r, 0 ) ∥ ))

Revised: + O ( η 2 k ) .

Original: Challenge I: How to deal with the  historical information in the momentum?
Revised: The accumulation of O ( η 2 k ) is in order log T and grows slowly. To derive the convergence result, we need to show the first term dominates. However, the second term contains historical gradient information, which is not necessarily smaller than the first term in absolute value. How to deal with it?

Revised: We observe that while the first and the second terms are not comparable for single k , summing over k leads to f ( w T +1 , 0 ) − f ( w 1 , 0 ) is smaller than

Revised: [Equation]

Revised: With a sum order change, one can easily observe that the last term equals to O

Revised: ( (cid:80) Tk =

Revised: η 2 k ∥∇ f

Revised: ( w k, 0 ) ∥ ) , which is also O ( (cid:80) Tk =1 η k min { ∥∇ f ( w k, 0 ) ∥

Revised: √ D, ∥∇

Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: √

Revised: [Equation]

Revised: + O ( η 2 k ) due to the mean value inequality η 2 k ∥∇ f ( w k, 0 )

Revised: ∥ ≤ O ( η 2 k )

Revised: + O ( η 2 k (cid:113) D 1 D 0 ∥∇ f ( w k, 0 )

Revised: ∥ 2 ) .

Revised: Thus the last term is dominated by the last to second term.

Revised: This completes the proof for RMSProp.

Revised: Remark 6 (Difficulty compared to L -smooth case) .

Original: Even after we disentangle the adaptor and the momentum, the  historical  information  in the momentum prevents the alignment between − η  k(cid:80) i m k,i √ ν k, 0 + ε and −∇ f ( w k, 0 ) , and thus the first order term may be non-negative.
Revised: First of all, the change of ν k,i is easier to bound without the historical gradient term. Secondly, under L -smooth condition, the error does not contain historical gradient information and is only in order of O ( η 2 k ) , which is easy to bound. Remark 7 (Intuition why SGD converges slowly) . If we replace Adam with SGD in the above analysis, the first order term may no longer dominates the second order term.

Revised: This explains why SGD converges slowly.

Revised: A detailed discussion can be found in Appendix B.

Original: Insight I: A new potential function.
Revised: Stage II: Extend the proof to general Adam. For Adam, the update norm is still bounded and the second order term can be bounded similarly. However, the analysis of the first order term becomes more challenging even though we still have ν k,i ≈ ν k, 0 . Specifically, even with constant ν k,i =

Original:  We overcome thischallenge by proposing a new potential function  and using a new descent lemma to replace Eq.
Revised: ν k, 0 , − η k ⟨ (cid:80) i m k,i √ ν k,i + ε , −∇ f ( w k, 0 ) ⟩ > 0 is not necessarily correct, as the momentum m k,i contains a heavy historical signal, and may push the update away from the negative gradient direction. How to deal with this challenge? We observe that the alignment of w k +1 , 0 − w k, 0 and −∇ f ( w k, 0 ) is required due to that our analysis is based on the potential function f ( w k, 0 ) . However, while this potential function is suitable for the analysis of RMSProp, it is no longer appropriate for Adam based on the above discussion and we need to construct another potential function.

Original: Our construction of the potential function is based on the following observation: we revisit the update rule in Eq.
Revised: Our construction of the potential function is based on the following observation: we revisit the update rule in Eq.

Original: (2) and rewrite it in the following manner:
Revised: (2) and rewrite it in the following manner:

Original: [Equation]
Revised: [Equation]

Original: Notice that the right-hand-side of the above equation contains no historical gradients but only the gradient of the current step!
Revised: Notice that the right-hand-side of the above equation contains no historical gradients but only the gradient of the current step!

Original: We then get 1 / √ ν k,i into the play, leading to
Revised: We then get 1 / √ ν k,i into the play, leading to

Original: [Equation]
Revised: [Equation]

Original: k,i +1 − β

Original: [Equation] k,i [Equation] One can see that the sequence {u k,i ≜ w k,i − β 1 w k,i
Revised: One can see that the sequence {

Original: [Equation]
Revised: [Equation]

Original: 1 1 − [Equation] } are (approximately) doing SGD within one epoch (with coordinate-wise but constant learning rate ν k,i )!
Revised: } are (approximately) doing SGD within one epoch (with coordinate-wise but constant learning rate ν k,i )!

Original: Further notice that u k,i = w k,i + β 1 w k,i − w k,i − 1 1 − βis close to w k,i .
Revised: Further notice that u k,i = w k,i + β 1 w k,i − w k,i − 1 1 − βis close to w k,i .

Original: Therefore, we choose our potential function as f ( u k,i ) .
Revised: Therefore, we choose our potential function as f ( u k,i ) .

Original: The Taylor’s expansion of f at u k, 0 then provides a new descent lemma, i.e.,[Equation] [Equation] k
Revised: The Taylor’s expansion of f at u k, 0 then provides a new descent lemma, i.e.,

Original: [Equation]
Revised: [Equation]

Original: By noticing w k,i ≈ u k,i ≈ u k, 0 , the first order term can be further approximated as follows,
Revised: By noticing w k,i ≈ u k,i ≈ u k, 0 , the first order term can be further approximated as follows,

Original: [Equation]
Revised: [Equation]

Original: which is negative, and the challenge is addressed.
Revised: which is negative, and the challenge is addressed.

Original: Remark 3.
Revised: Remark 8.

Original: Zhang et al.
Revised: Zhang et al.

Original: (2022) also encounterthis challenge. They overcome this challenge by additionally assuming that the random permutation to generate { τ k, 0  , · · · , τ k,n − 1 } is uniformly sampled from all possible permutation orders. Based on this additional assumption, the expectation of the momentum can be shown close to the gradient and thus they derivean in-expectation result. In contrast, we do not need such an assumption.
Revised: (2022) also encounter the misalignment between w k +1 , 0 − w k, 0 and −∇ f ( w k, 0 ) , but they do not use the potential function and can only derive in-expectation convergence.

Original: Further, their proof is more complex than ours as they need to deal with the historical gradient signal in the momentum. Our potential function allows us to offer a trajectory-wise convergence result with simplified proof. To complete the proof, we move on to show that the second order term in Eq. (8) is minor compared to the first order term, which we identify as the second challenge.
Revised: A detailed discussion can be found in Appendix B.

Original: We note that such a property does not hold for SGD under ( L 0 , L 1 )

Original: -smooth condition: the first order term in Eq.

Original: (6) is (approximately) − η k ∥∇ f ( w k, 0 ) ∥ 2 for SGD, while the corresponding second order term is

Original: [Equation] In other words, the second order term has one more η k ∥∇ f ( w k, 0 ) ∥ factor than the first order. Without any boundedness condition on the gradient, this factor could be unboundedly large and thus makes the second order term dominate the dynamics of the objective function f .
Revised: Remark 9.

Original: This eventually leads to slow convergence or even divergence for SGD under ( L 0 , L 1 )

Original: -smooth condition (see counter-examples in

Original: Section 4.3).

Original: Challenge II: How to show the second order term in Eq. (8) is minor for Adam under ( L 0 , L 1 ) - smooth condition? Insight II: Adaptivity in Adam. We overcome this challenge by leveraging the adaptivity in Adam.
Revised: There are similar potential functions in existing works for other optimizers, but none of them succeeds to derive the result for Adam.

Original: We first revisit the above discussion for SGD, and find the additional ∥∇ f ( w k, 0 ) ∥ is because the update of SGD has the dependency on gradient with degree 1 .

Original: Viewing the first order and the second order term in Eq.

Original: (6) respectively as linear and quadratic functions of w k +1 , 0 − w k, 0 , their coefficients also contain dependency on the gradient with order 1 .

Original: This leads to a higher-degree dependency on the gradient of the second order term ( 1 + 2 × 1 ) than the first order ( 1 + 1 ).

Original: On the other hand, Adam does not suffer from this problem: both the momentum and the adaptor have the same unit as the gradient, and thus the update (equals momentum/adaptor) has a dependency on the gradient with degree 0 , which leads to the same degree of dependency on the gradient for the first order ( 1 + 2 × 0 ) and second order term ( 1 + 2 × 0 ).

Original: Formally, for the first order term, a calculation for Eq.

Original: (9) yields

Original: [Equation]

Original: Meanwhile, due to the hyperparameter setting β 1 < √ β 2 , ∥ m k,i / ν k,i ∥ is bounded, and thus the second order term can be bounded by

Original: [Equation]

Original: , 0 ) ∥ √ D, ∥∇ f ( w k, 0 ) ∥ 2 √

Original: D(cid:27)(cid:19) .

Original: [-Here the last equality is due to the mean value inequality η 2 k ∥∇ f ( w k, 0 ) ∥ ≤ 12 η 2 k + 
-]
Revised: [-Here the last equality is due to the mean value inequality η 2 k ∥∇ f ( w k, 0 ) ∥ ≤ 12 η 2 k + 
-]

Original: [-η 2 k ∥∇ f ( w k, 
 ∥and we drop the minor term O ( η 2 k )-]
Revised: [-η 2 k ∥∇ f ( w k, 
 ∥and we drop the minor term O ( η 2 k )-]We defer a detailed discussion to Appendix B.

Original: =

Original: O (1 /k ) .

Original: One can now easily observe that the first order term is larger than the second order by order of 1 /η k .

Original: This addresses the second challenge.

Original: 4. SGD MAY CONVERGE ARBITRARILY SLOW WITH ( L 0 , L 1 ) - SMOOTH CONDITION
Revised: 4. SGD MAY CONVERGE ARBITRARILY SLOW WITH ( L 0 , L 1 ) - SMOOTH CONDITION

Original: As the ( L 0 , L 1 ) -smooth condition appears to be a more precise characterization of the landscape of neural networks than L -smooth condition, one may wonder how SGD performs under such a circumstance.
Revised: As the ( L 0 , L 1 ) -smooth condition appears to be a more precise characterization of the landscape of neural networks than L -smooth condition, one may wonder how SGD performs under such a circumstance.

Original: Here, we find examples that satisfy ( L 0 , L 1 ) -smooth condition, but GD and SGD diverges.
Revised: Here, we find examples that satisfy ( L 0 , L 1 ) -smooth condition, but GD and SGD diverges.

Original: This implies that under this realistic smoothness assumption, (S)GD can be arbitrarily slow.
Revised: This implies that under this realistic smoothness assumption, (S)GD can be arbitrarily slow.

Original: GD may converge arbitrarily slow with ( L 0 , L 1 )
Revised: GD may converge arbitrarily slow with ( L 0 , L 1 )

Original: -smooth condition.
Revised: -smooth condition.

Original: To begin with, we notice that
Revised: To begin with, we notice that

Original: Zhang et al. (2019a) have already provided counterexamples over which GD converges arbitrarily slowly.
Revised: Zhang et al. (2019a) have already provided counterexamples over which GD converges arbitrarily slowly.

Original: However, their examples use constant learning rate GD, which, unfortunately, does NOT match the setting of Theorem 1 (i.e., diminishing learning rate), and thus does not suffice to provide a fair comparison between GD and (full-batch) Adam.
Revised: However, their examples use constant learning rate GD, which, unfortunately, does NOT match the setting of Theorem 1 (i.e., diminishing learning rate and stochastic update), and thus does not suffice to provide a fair comparison between GD and (full-batch) Adam.

Original: To fill this gap in setting, we provide the following example:
Revised: To fill this gap in setting, we provide the following example:

Original: Example 1.
Revised: Example 1.

Original: There exists a domain X and a target function f : R X
Revised: There exists a domain X and a target function f : R X

Original: → R , such that for every initial learning rate η 1 , there exists a region E ⊂ X with infinite measure, such that GD with diminishing learning rate η t = η 1 √ t and initialized in E will drive f ( w t ) and ∥∇ f ( w t ) ∥ to infinity.
Revised: → R , such that for every initial learning rate η 1 , there exists a region E ⊂ X with infinite measure, such that GD with diminishing learning rate η t = η 1 √ t and initialized in E will drive f ( w t ) and ∥∇ f ( w t ) ∥ to infinity.

Original: We defer a concrete construction of the example to Appendix B.1.
Revised: We defer a concrete construction of the example to Appendix C.1.

Revised: We emphasize here that it is a classification problem with linear model, which shares similar properties with practical classification problems.

Original: The intuition of this example is that with a fixed initial learning rate, a bad initialization may lead to arbitrarily worse local smoothness due to globally unbounded smoothness, and make the first-step stepsize relatively large.
Revised: The intuition of this example is that with a fixed initial learning rate, a bad initialization may lead to arbitrarily worse local smoothness due to globally unbounded smoothness, and make the first-step stepsize relatively large.

Original: This leads to an even worse (larger) local smoothness at the next step.
Revised: This leads to an even worse (larger) local smoothness at the next step.

Original: Also, the increase of the local smoothness can be rapid enough to offset the decrease in learning rate, and eventually drive the loss to divergence.
Revised: Also, the increase of the local smoothness can be rapid enough to offset the decrease in learning rate, and eventually drive the loss to divergence.

Original: On the contrary, the update norm of Adam is bounded regardless of the initialization, and thus Adam does not suffer from the same problem and will converge.
Revised: On the contrary, the update norm of Adam is bounded regardless of the initialization, and thus Adam does not suffer from the same problem.

Original: One may notice that the divergence issue in the above example can be equivalently viewed as “given initialization, a large learning rate will bring divergence".
Revised: One may notice that the divergence issue in the above example can be equivalently viewed as “given initialization, a large learning rate will bring divergence".

Original: A natural question is that whether reducing the learning rate will lead to convergence.
Revised: A natural question is that whether reducing the learning rate will lead to convergence.

Original: The answer could be yes, but it is true only if some additional assumption is provided (for example, Assumption 4 in (Zhang et al., 2019a), which assumes the gradient is bounded over a loss sub-level set).
Revised: The answer could be yes, but it is true only if some additional assumption is provided (for example, Assumption 4 in (Zhang et al., 2019a), which assumes the gradient is bounded over a loss sub-level set).

Original: However, in this case, the feasible initial learning rate depends on the local smoothness over the trajectory.
Revised: However, in this case, the feasible initial learning rate depends on the local smoothness over the trajectory.

Original: Since the initial gradient can be arbitrarily large, the initial learning rate needs to be arbitrarily small, which thus leads to a slow convergence rate if bad initialization is picked.
Revised: Since the initial gradient can be arbitrarily large, the initial learning rate needs to be arbitrarily small, which thus leads to a slow convergence rate if bad initialization is picked.

Original: This is exactly the intuition of the lower bound in (Zhang et al., 2019a).
Revised: This is exactly the intuition of the lower bound in (Zhang et al., 2019a).

Revised: Please refer to a detailed discussion in Appendix B.

Original: Similar negative results also apply to SGD.
Revised: Similar negative results also apply to SGD.

Original: Actually, the situation is even worse: SGD still suffer from divergence for learning rate 1 /L , which can lead to convergence for GD.
Revised: Actually, the situation is even worse: SGD still suffer from divergence for learning rate 1 /L , which can lead to convergence for GD.

Original: Example 2.
Revised: Example 2.

Original: There exists a domain X , a initial point w 0 ∈ X and a target function f = (cid:80) n − 1 i =

Revised: There exists a domain X , a initial point w 0 ∈ X and a target function f = (cid:80) n − 1 i =

Original: f i , over which SGD with initial learning rate η 1 = 1 L and diminishing learning rate η t = η 1 √ t diverges while GD with the same setting converges, where L is the smoothness upper bound of the loss sub-level set { w ∈ X : f ( w ) ≤ f ( w 0 ) } .
Revised: f i , over which SGD with initial learning rate η 1 = 1 L and diminishing learning rate η t = η 1 √ t diverges while GD with the same setting converges, where L is the smoothness upper bound of the loss sub-level set { w ∈ X : f ( w ) ≤ f ( w 0 ) } .

Original: We defer the concrete construction to Appendix B.2.
Revised: We defer the concrete construction to Appendix C.2.

Original: Note that GD converges with the initial learning rate 1 L , while SGD diverges in this setting.

Original: The intuition of this example is that even withsmall learning rate, a non-full-batch update can step out the loss sub-level set and increase the local smoothness coefficient.E XPERIMENTS
Revised: The intuition of this example is that even with learning rate 1 /L , a non-full-batch update can step out the loss sub-level set and increase the local smoothness coefficient.

Original: We conduct numerical experiments to support our theoretical findings. We illustrate through the experiment that with D 0 ̸ = 0 , Adam may not converge to the stationary point. Specifically, we run Adam on a synthetic optimization target which is given as follows:
Revised: Similarly as before, keep reducing the learning rate might lead to convergence.

Original: [Equation]

Original: Such an example is [-proposed by Shi et al. One can easily observe that f ( x ) and { f i ( x ) } 9 i = satisfies Assumption 1 and Assumption 2 with non-zero D 0 ̸ = 
 ( D 1 = 208 , D 0 = 75 59 ), thus Theorem 1 applies.-]
Revised: But this only happens when additional assumptions is [-proposed by Shi et al. One can easily observe that f ( x ) and { f i ( x ) } 9 i = satisfies Assumption 1 and Assumption 2 with non-zero D 0 ̸ = 
 ( D 1 = 208 , D 0 = 75 59 ), thus Theorem 1 applies.-]imposed.

Original: With diminishing learning rates η t = 10 √ t , η t = 1 √ t , and η t = 0 . 1 √ t , we plot the distance between Adam’s trajectory and the unique minimum 0 of f in Figure 2. It can be observed that Adam does not converge to the unique minimum 0 , but gets closer to the minimum if choosing β 2 closer to 1 , which supports Theorem 1. [Figure]
Revised: Even if so, the convergence rate could be arbitrarily slow.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we take the first step to theoretically understand the adaptivity in Adam.
Revised: In this paper, we take the first step to theoretically understand the adaptivity in Adam.

Original: Specifically, we prove that Adam can converge arbitrarily faster than SGD .
Revised: Specifically, we prove that Adam can converge arbitrarily faster than SGD .

Original: Our assumption is realistic and close to practical settings.
Revised: Our assumption is realistic and close to practical settings.

Original: On the other hand, there is still a lot to explore for Adam’s performance.
Revised: On the other hand, there is still a lot to explore for Adam’s performance.

Original: For example, it is interesting to analyze how momentum helps in Adam’s optimization.
Revised: For example, it is interesting to analyze how momentum helps in Adam’s optimization.

Revised: Further, it is interesting to investigate whether Adam can handle more sharp smooth conditions, e.g., smoothness is bounded by a high order polynomial of the gradient norm.

Original: A A DDITIONAL R ELATED W ORKS
Revised: A A DDITIONAL R ELATED W ORKS

Original: In this section, we provide discussions on more related works.
Revised: In this section, we provide discussions on more related works.

Original: New variants of Adam.
Revised: New variants of Adam.

Original: Ever since Reddi et al.
Revised: Ever since Reddi et al.

Original: (2019) pointed out the non-convergence issue of
Revised: (2019) pointed out the non-convergence issue of

Original: Adam, many new variants of Adam are designed.
Revised: Adam, many new variants of Adam are designed.

Original: For instance, Zou et al.
Revised: For instance, Zou et al.

Original: (2019); Gadat & Gavra (2020)
Revised: (2019); Gadat & Gavra (2020)

Original: ; Chen et al.
Revised: ; Chen et al.

Original: (2018b; 2021a) replaced the constant hyperparameters by iterate-dependent ones e.g. β 1 t or β 2 t .
Revised: (2018b; 2021a) replaced the constant hyperparameters by iterate-dependent ones e.g. β 1 t or β 2 t .

Original: AMSGrad (Reddi et al., 2019) and AdaFom (Chen et al., 2018b) enforced { v t } to be an non-decreasing.
Revised: AMSGrad (Reddi et al., 2019) and AdaFom (Chen et al., 2018b) enforced { v t } to be an non-decreasing.

Original: Similarly, AdaBound (Luo et al., 2019) imposed constraints v t ∈
Revised: Similarly, AdaBound (Luo et al., 2019) imposed constraints v t ∈

Original: [ C l , C u ] to prevent the learning rate from vanishing or exploding.
Revised: [ C l , C u ] to prevent the learning rate from vanishing or exploding.

Original: Similarly, Zhou et al.
Revised: Similarly, Zhou et al.

Original: (2018b) adopted a new estimate of v t to correct the bias.
Revised: (2018b) adopted a new estimate of v t to correct the bias.

Original: In addition, there are attempts to combine Adam with Nesterov momentum (Dozat, 2016) as well as warm-up techniques (Liu et al., 2020).
Revised: In addition, there are attempts to combine Adam with Nesterov momentum (Dozat, 2016) as well as warm-up techniques (Liu et al., 2020a).

Original: There are also some works providing theoretical analysis on the variants of Adam.
Revised: There are also some works providing theoretical analysis on the variants of Adam.

Original: For instance, Zhou et al.
Revised: For instance, Zhou et al.

Original: (2018a) studied the convergence of AdaGrad and AMSGrad.
Revised: (2018a) studied the convergence of AdaGrad and AMSGrad.

Original: Gadat & Gavra (2020) studied the asymptotic behavior of a subclass of adaptive gradient methods from landscape point of view.
Revised: Gadat & Gavra (2020) studied the asymptotic behavior of a subclass of adaptive gradient methods from landscape point of view.

Original: Their analysis applies to Adam-variants with β 1 = 0 and β 2 increasing along the iterates.
Revised: Their analysis applies to Adam-variants with β 1 = 0 and β 2 increasing along the iterates.

Original: In summary, all these works require L -smooth condition.
Revised: In summary, all these works require L -smooth condition.

Original: Though the new variants of Adam can probably converge, the convergence rate is no better than SGD.
Revised: Though the new variants of Adam can probably converge, the convergence rate is no better than SGD.

Original: Generalization ability of Adaptive gradient methods.
Revised: Generalization ability of Adaptive gradient methods.

Original: The generalization ability of Adam is a hot debate topic.
Revised: The generalization ability of Adam is a hot debate topic.

Original: For instance, Wang et al.
Revised: For instance, Wang et al.

Original: (2021) studied the implicit bias of adaptive optimization algorithms on homogeneous neural networks.
Revised: (2021) studied the implicit bias of adaptive optimization algorithms on homogeneous neural networks.

Original: They proved that the convergent direction of Adam and
Revised: They proved that the convergent direction of Adam and

Original: RMSProp is the same as SGD.
Revised: RMSProp is the same as SGD.

Original: Zhou et al.
Revised: Zhou et al.

Original: Xie et al.
Revised: Xie et al.

Original: ; Zou et al.
Revised: ; Zou et al.

Original: (2021) argue that Adam preferred sharp local-min while GD prefers the wide ones.
Revised: (2021) argue that Adam preferred sharp local-min while GD prefers the wide ones.

Original: As such, they argue that Adam generalizes worse than SGD.
Revised: As such, they argue that Adam generalizes worse than SGD.

Original: Zou et al.
Revised: Zou et al.

Original: (2021) prove that Adam generalizes worse than SGD over a specific model.
Revised: (2021) prove that Adam generalizes worse than SGD over a specific model.

Original: There are also several attempts to improve the generalization ability of Adam.
Revised: There are also several attempts to improve the generalization ability of Adam.

Original: For instance,
Revised: For instance,

Original: Padam (Chen et al., 2018a) introduced a partial adaptive parameter to improve the generalization performance.
Revised: Padam (Chen et al., 2018a) introduced a partial adaptive parameter to improve the generalization performance.

Original: AdamW (Loshchilov & Hutter, 2017) improved regularization in Adam by decoupling the weight decay from the gradient-based update.
Revised: AdamW (Loshchilov & Hutter, 2017) improved regularization in Adam by decoupling the weight decay from the gradient-based update.

Original: B D IVERGING E XAMPLE FOR (S)GD
Revised: B A DDITIONAL DISCUSSIONS B. R ESTRICTION OF β 1 IN T HEOREM

Revised: We discuss a bit more on β 1 . First, The requirement β 21 < β 2 is a standard condition in Adam-family literature (Reddi et al., 2018; Zou et al., 2019; Défossez et al., 2020; Shi et al., 2021).

Revised: Since βis suggested to be large, this condition covers flexible choice of β 1 ∈ [0 , 1) .

Revised: Second, although the dependency between β 1 and β 2 in Inequality (4) seems complicated, the effect of β 1 on Inequality (4) can be omitted when adopting the practical value such as β 1 = 0 .

Revised: This is because the term “ 1+ β 1 1 − β 1 " is way smaller than the term “ ( n − 1) " in the right-hand-side of Inequality (4).

Revised: For example, when β 1 = 0 .

Revised: 9 (the default setting in deep learning libraries), 1+ β 1 1 − β 1 = 19 .

Revised: This number is much smaller than n on real datasets (for instance, on CIFAR-10 (He et al., 2016), n = 50 k/ 128 ≈ 390 ; on ImageNet (You et al., 2017), n = 1 . 2 m/ 8 k = 150 ).

Revised: As a result, constraint (4) is inactive for practical choice of β 1 .

Revised: B. C OMPARISONS OF OPTIMIZERS OVER THE FINE - TUNING TASK

Revised: For the case where the gradient along the trajectory is small, ( L 0 , L 1 ) -smooth condition will degenerate to L -smooth condition, and thus SGD works well.

Revised: This may explain the phenomenon that SGD is also adopted in some finetuning tasks, as pretraining can be viewed as selecting a good initialization (and we can expect that the gradient is small along the trajectory).

Revised: While the above discussion is highly intuitive, it is an interesting future work to formally prove it.

Revised: B. O N THE EFFECT OF MOMENTUM

Revised: Since Adam and RMSProp both converge under ( L 0 , L 1 ) smooth condition, a natural question is what the effect of momentum is in Adam.

Revised: However, this is a highly non-trivial question.

Revised: The effect of momentum is not clear even for momentum SGD in non-convex optimization, let alone for Adam.

Revised: We believe that this question is interesting yet beyond the scope of this paper and leave it as a future work.

Revised: B. C ONVERGENCE RESULT OF RMSP ROP

Revised: As mentioned in Section 4.2, we provide the convergence result of RMSProp as a corollary of

Revised: Theorem 1 here.

Revised: Corollary 1.

Revised: Let all the conditions in Theorem 1 hold.

Revised: Then, for RMSProp, if

Revised: [Equation]

Revised: then, we have that

Revised: [Equation]

Revised: Since Adam and RMSProp both converge under ( L 0 , L 1 ) -smooth condition, a natural question is what the effect of momentum is in Adam.

Revised: However, this is a highly non-trivial question.

Revised: Theoretically, the effect of momentum is not clear even for momentum SGD in non-convex optimization, let alone for Adam.

Revised: Practically, there are existing experiments suggesting that the performance of well-tuned

Revised: RMSProp can match that of well-tuned Adam (Choi et al., 2019).

Revised: We believe that this question is interesting but beyond the scope of this paper and leave it as a future work.

Revised: B. A DVANTAGE OF A DAM OVER THE

Revised: GD/SGD

Revised: WITH

Revised: GRADIENT

Revised: CLIPPING

Revised: (Zhang et al., 2019a) shows that GD/SGD with gradient clipping converges under ( L 0 , L 1 ) smooth condition.

Revised: A natural question is that what is the benefit of Adam over GD/SGD with gradient clipping.

Revised: Honestly, we are not able to answer this question yet, but one potential advantage may be that Adam can handle more complex noise.

Revised: It is not known whether SGD with gradient clipping can converge in our setting, as the existing analyses of GD/SGD with gradient clipping under ( L 0 , L 1 ) -smooth condition all assume that the distance between stochastic gradient and true gradient are bounded with probability 1 ([Zhang et al., 2019] and [Zhang et al. 2020]), which is strictly stronger than our affine variance noise assumption.

Revised: It will be interesting to either provide counterexample that SGD with clipping does not converge under ( L 0 , L 1 )

Revised: -smooth condition with affine noise assumption, or prove that SGD with clipping does converge and find other perspective to demonstrate the advantage of

Revised: Adam over SGD with clipping.

Revised: B. I NTUITION WHY A DAM CONVERGES FASTER THAN SGD

Revised: We first illustrate this from the proof.

Revised: If we analyze SGD following the same methodology of Section 4.2 for SGD, the first order term in Eq.

Revised: (6) for SGD is ⟨− η k (cid:80) n − 1 i =0 ∇ f τ k,i ( w k,i ) , ∇ f τ k,i ( w k, 0 ) ⟩ ≈ ⟨− η k (cid:80) n − 1 i =0 ∇ f τ k,i ( w k, 0 ) , ∇ f τ k,i ( w k, 0 ) ⟩ = − η k ∥∇ f τ k,i ( w k, 0 )

Revised: ∥ 2 .

Revised: However, when it comes to bound the second order term, we will encounter two problems.

Revised: First, it is not known whether the update of SGD is bounded, and thus whether we can bound L loc using ( L 0 , L 1 ) -smooth condition.

Revised: Second, even if we succeed to bound L loc by O ( ∥∇ f ( w k,i ))

Revised: ∥ + O (1) , the second order term has the form O

Revised: η 2 k ∥∇ f

Revised: ( w k,i ) ∥ 3 ) +

Revised: O ( η 2 k ∥∇ f

Revised: ( w k,i ) ∥ 2 ) , and may be larger than the first order term when the gradient is large.

Revised: We invite the readers to see the counterexample in Section 4.3 to learn more.

Revised: We then provide an example.

Revised: When the local smoothness constant coefficientL varies drastically in the domain, it is difficult to decide a pre-decide stepsize that guarantees to work along SGD trajectories.

Revised: There are two cases:

Revised: When entering a sharp local region, the pre-determined steps might be too large and cause divergence of SGD.

Revised: When entering a flat region, the pre-determined steps might be too small to

Revised: make progress.

Revised: It will cause slow convergence of SGD.

Revised: Situation gets worse when using diminishing steps.

Revised: For Adam, the effective stepsize involves

Revised: [Equation]

Revised: , which is adaptive changing along the trajectory.

Revised: Adam can better handle the above two cases: 1.

Revised: When entering the sharp local region , the gradient is usually large and 1 √ ν k,i + ε 1 d will gradually decrease in this region.

Revised: Finally it will reach a small enough stepsize to slip into the sharp region.

Revised: When entering the flat region, the gradient is consistently small and thus 1 √ ν k,i + ε 1 d will gradually increase, leading to larger stepsize and thus Adam will converge faster.

Revised: B. D ISCUSSION ON THE POTENTIAL FUNCTION

Revised: We notice that similar potential functions have already been applied in the analysis of other momentum-based optimizers, e.g., momentum (S)GD in (Ghadimi et al., 2015) and (Liu et al., 2020b) and Adam-type optimizers (except Adam) in (Chen et al., 2018b).

Revised: However, extending the proof to Adam is highly-nontrivial and fails to provide a convergence result for Adam.

Revised: The key difficulty lies in showing that the first order expansion of f ( u k, 0 ) is positive, which further requires that the adaptive learning rate does not change much within one epoch.

Revised: This is hard for Adam as the adaptive learning rate of Adam can be nonmonotonic.

Revised: The lack of L-smooth condition makes the proof even challenging due to the unbounded error brought by gradient norms.

Revised: B. D ISCUSSION ON THE IN - EXPECTATION RESULT IN (Z HANG ET AL ., 2022)

Revised: Zhang et al.

Revised: (2022) handle the misalignment between w k +1 , 0 − w k, 0 and −∇ f ( w k, 0 ) by additionally assuming that the random permutation to generate { τ k, 0 , · · · , τ k,n − 1 } is uniformly sampled from all possible permutation orders.

Revised: Based on this additional assumption, the expectation of the momentum can be shown close to the gradient and thus they derive an in-expectation result.

Revised: In contrast, we do not need such an assumption.

Revised: Further, their proof is more complex than ours as they need to deal with the historical gradient signal in the momentum.

Revised: Our potential function allows us to offer a trajectory-wise convergence result with simplified proof.

Revised: B. I NSIGHT FOR PRACTITIONERS

Revised: First, Adam receives great popularity among practitioners (with more than 100k citations).

Revised: It is important to theoretically understand this current algorithm.

Revised: Second, we provide new suggestions for practitioners : When running experiments on tasks such as

Revised: Transformers and LSTM training, we suggest using Adam instead of SGD.

Revised: Though this is a folk result based on engineering experience.

Revised: It is firstly theoretically justified.

Revised: Third, we provide suggestions for hyperparameter tuning (based on the convergence conditions in Theorem 1): when running Adam, we suggest tune up beta2 and try different β 1 < √ β 2 .

Revised: This suggestion would save much effort of grid-searching the ( β 1 , β 2 ) combination.

Original: In this section, we discuss the divergence issue for (S)GD under the ( L 0 , L 1 ) -smooth condition.
Revised: In this section, we discuss the divergence issue for (S)GD under the ( L 0 , L 1 ) -smooth condition.

Original: This section is organized as follows: we will first consider the simple full-batch case, i.e., n =and f ( w ) = f 0 ( w ) (therefore there is no random reshuffling and no randomness in the process of training), in which case we will show GD may suffer from divergence under the ( L 0 , L 1 ) assumption while Adam converges to the stationary point.
Revised: This section is organized as follows: we will first consider the simple full-batch case, i.e., n =and f ( w ) = f 0 ( w ) (therefore there is no random reshuffling and no randomness in the process of training), in which case we will show GD may suffer from divergence under the ( L 0 , L 1 ) assumption while Adam converges to the stationary point.

Original: We then step further to the random reshuffling case and show that the randomness may brought additional divergence issues.
Revised: We then step further to the random reshuffling case and show that the randomness may brought additional divergence issues.

Original: B. C OMPARE GD AND F ULL - BATCH A DAM
Revised: C. C OMPARE GD AND F ULL - BATCH A DAM

Original: To start with, we notice there is an example in Zhang et al. (2019a) over which GD will converge arbitrarily slow.
Revised: To start with, we notice there is an example in Zhang et al. (2019a) over which GD will converge arbitrarily slow.

Original: We re-state their result as follows:
Revised: We re-state their result as follows:

Original: Property 1 (Theorem 4, Zhang et al. (2019a)) .
Revised: Property 1 (Theorem 4, Zhang et al. (2019a)) .

Original: For every fixed learning rate η ∈ R + and tolerate error bound ε ∈ R + , there exists a domain X , an optimization problem f : X → R , and an initialization point w 1 ∈ X satisfying f is lower with ( L 0 , L 1 ) smoothness and satisfying M ≜ sup {∥∇ f ( w ) ∥| w such that f ( w ) ≤ f ( w 0 ) } < ∞ , such that GD over f starting from w 1 with constant learning rate η , such that any t ∈ Z + smaller than L 1 M ( f ( w 1 ) − min w ∈X f ( w ) − 5 ε/ 8) 8 ε 2 (log M +1) satisfies ∥∇ f ( w t ) ∥ > 
Revised: For every fixed learning rate η ∈ R + and tolerate error bound ε ∈ R + , there exists a domain X , an optimization problem f : X → R , and an initialization point w 1 ∈ X satisfying f is lower with ( L 0 , L 1 ) smoothness and satisfying M ≜ sup {∥∇ f ( w ) ∥| w such that f ( w ) ≤ f ( w 0 ) } < ∞ , such that GD over f starting from w 1 with constant learning rate η , such that any t ∈ Z + smaller than L 1 M ( f ( w 1 ) − min w ∈X f ( w ) − 5 ε/ 8) 8 ε 2 (log M +1) satisfies ∥∇ f ( w t ) ∥ > ε .

Original: ε By Property 1, we know that the convergence of GD can be arbitrarily slow depending on the gradient upper bound M of the loss sub-level set { w : f ( w ) ≤ f ( w 1 ) } , which further depends on the initialization.
Revised: By Property 1, we know that the convergence of GD can be arbitrarily slow depending on the gradient upper bound M of the loss sub-level set { w : f ( w ) ≤ f ( w 1 ) } , which further depends on the initialization.

Original: In other words, the performance of GD sensitively relies on the initialization.
Revised: In other words, the performance of GD sensitively relies on the initialization.

Original: However, Property 1 does not suffice to show Adam performs better than GD, as in Property 1 the learning rate is constant while in Theorem 1, it is decaying.
Revised: However, Property 1 does not suffice to show Adam performs better than GD, as in Property 1 the learning rate is constant while in Theorem 1, it is decaying.

Original: To fill this gap in setting, we provide the following example satisfying Assumptions 2 and 1, on which GD with decaying learning rate diverges if initialized badly.
Revised: To fill this gap in setting, we provide the following example satisfying Assumptions 2 and 1, on which GD with decaying learning rate diverges if initialized badly.

Original: Example 3 (Example 1, restated) .
Revised: Example 3 (Example 1, restated) .

Original: Let n = 1 and p = 2 with f 0 (( w 1 , w 2 ))
Revised: Let n = 1 and p = 2 with f 0 (( w 1 , w 2 ))

Original: =
Revised: =

Original: e − √ 32 w 1 − 12 w 2 + e √ 32 w 1 − 12 w 

Revised: e − √ 32 w 1 − 12 w 2 + e √ 32 w 1 − 12 w

Original: and f (( w 1 , w 2 ))
Revised: and f (( w 1 , w 2 ))

Original: =
Revised: =

Original: f 0 (( w 1 , w 2 )) .
Revised: f 0 (( w 1 , w 2 )) .

Original: Then, f satisfies Assumptions 1 and 2 .
Revised: Then, f satisfies Assumptions 1 and 2 .

Original: However, for every initial learning rate η 1 , there exists a region E with infinite measure over R 2 such that, with
Revised: However, for every initial learning rate η 1 , there exists a region E with infinite measure over R 2 such that, with

Original: GD with decaying learning rate η k = η 1 √ k and start point in E , f and ∇ f diverge to infinity.
Revised: GD with decaying learning rate η k = η 1 √ k and start point in E , f and ∇ f diverge to infinity.

Original: Proof.
Revised: Proof.

Original: Notations.
Revised: Notations.

Original: To simplify the notation, we define x 1 = ( √ 32 , 12 ) and x 2 = ( − √ 32 , 12 ) .
Revised: To simplify the notation, we define x 1 = ( √ 32 , 12 ) and x 2 = ( − √ 32 , 12 ) .

Original: It can be observed that ∥ x 1 ∥ =
Revised: It can be observed that ∥ x 1 ∥ =

Original: ∥ x 2 ∥ = 1 and ⟨ x 1 , x 2 ⟩ = − 12 .
Revised: ∥ x 2 ∥ = 1 and ⟨ x 1 , x 2 ⟩ = − 12 .

Original: We also denote the parameter given by the k -th iteration of GD as w k .
Revised: We also denote the parameter given by the k -th iteration of GD as w k .

Original: By definition, we have
Revised: By definition, we have

Original: [Equation]
Revised: [Equation]

Original: We further define g ( x ) ≜ η 1 e 38 x − 17 x − 16 η 1 e − 1516 x .
Revised: We further define g ( x ) ≜ η 1 e 38 x − 17 x − 16 η 1 e − 1516 x .

Original: As g ( x ) → ∞ as x → ∞ , there exists a large enough positive constant C > 1 , such that for every x > C , g ( x ) > 0 .
Revised: As g ( x ) → ∞ as x → ∞ , there exists a large enough positive constant C > 1 , such that for every x > C , g ( x ) > 0 .

Original: Verify the assumptions.
Revised: Verify the assumptions.

Original: Assumption 2 immediately follows as n = 1 .
Revised: Assumption 2 immediately follows as n = 1 .

Original: We then check Assumption 1.
Revised: We then check Assumption 1.

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: Therefore, ∥∇ 2 f 0 ( w ) ∥ 2 ≤ e −⟨ w , x 1 ⟩ + e −⟨ w , x 2 ⟩ , while∥∇ f ( w ) [Equation] ≥
Revised: Therefore, ∥∇ 2 f 0 ( w ) ∥ 2 ≤ e −⟨ w , x 1 ⟩ + e −⟨ w , x 2 ⟩ , while

Original: [Equation]
Revised: [Equation]

Original: As a conclusion, Assumption 1 is satisfied with L 0 = 

Revised: As a conclusion, Assumption 1 is satisfied with L 0 =

Original: and L 1 = 2 .
Revised: and L 1 = 2 .

Original: Initialization.
Revised: Initialization.

Original: We define region E as
Revised: We define region E as

Original: [Equation]
Revised: [Equation]

Original: It should be noticed that E has infinite measure.
Revised: It should be noticed that E has infinite measure.

Original: Iteration.
Revised: Iteration.

Original: As { a 1 x 1 + b 0 : a 1 > C, a 1 > 16 ∥ b 1 ∥} is symmetric with { a 1 x 2 + b 1 : a 1 >
Revised: As { a 1 x 1 + b 0 : a 1 > C, a 1 > 16 ∥ b 1 ∥} is symmetric with { a 1 x 2 + b 1 : a 1 >

Original: C, a 1 > 16 ∥ b 1 ∥} , we analyze the case when w 1 is in the former set without loss of generality, i.e., w 1 = a 1 x 1 + b 1 with a 1 > 0 and a 1 > 16 ∥ b 1 ∥ .
Revised: C, a 1 > 16 ∥ b 1 ∥} , we analyze the case when w 1 is in the former set without loss of generality, i.e., w 1 = a 1 x 1 + b 1 with a 1 > 0 and a 1 > 16 ∥ b 1 ∥ .

Original: We will prove by induction that ∀ k ≥ 1 , w k = a k x i k + b k , where i k equals to 1 if k is odd and 2 if k is even.
Revised: We will prove by induction that ∀ k ≥ 1 , w k = a k x i k + b k , where i k equals to 1 if k is odd and 2 if k is even.

Original: a k and b k satisfies a k ≥ 

Revised: a k and b k satisfies a k ≥

Original: ∥ b k ∥ and a k ≥ 17 k − 1 C .
Revised: ∥ b k ∥ and a k ≥ 17 k − 1 C .

Original: For k = 1 , the proof directly follows by the initialization.
Revised: For k = 1 , the proof directly follows by the initialization.

Original: For k ̸ = 1 , suppose that the above property holds for all the iterations before the k -th iteration.
Revised: For k ̸ = 1 , suppose that the above property holds for all the iterations before the k -th iteration.

Original: Suppose k is even.
Revised: Suppose k is even.

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: , which leads to
Revised: which leads to

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Consequently,
Revised: Consequently,

Original: [Equation]
Revised: [Equation]

Original: w

Original: [Equation]

Original: −

Original: [Equation]

Original: k −

Original: [Equation]

Original: x [Equation] , w k − 1 ⟩ x 1 . The norm of w k − 1 + η k − 1 e −⟨ x 1 , w k − 1 ⟩ x 1 can be bounded as
Revised: The norm of w k − 1 + η k − 1 e −⟨ x 1 , w k − 1 ⟩ x 1 can be bounded as

Original: [Equation]

Original: [Equation]

Original: [Equation]

Original: k −

Original: [Equation]
Revised: [Equation]

Original: while the coefficient of x 2 can be lower bounded as
Revised: while the coefficient of x 2 can be lower bounded as

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original:  As by the induction condition a k − 1 ≥ 17 k − 1 C ≥ C , we have that g ( a k − 1 ) ≥ 0 .
Revised: ≥ g ( a k − 1 ) . As by the induction condition a k − 1 ≥ 17 k − 1 C ≥ C , we have that g ( a k − 1 ) ≥ 0 .

Original: Therefore, choosing a k = η k − 1 e −⟨ x 2 , w k − 1 ⟩ and b k = ∥
Revised: Therefore, choosing a k = η k − 1 e −⟨ x 2 , w k − 1 ⟩ and b k = ∥

Original: w k − 1 + η k − 1 e −⟨ x 1 , w k − 1 ⟩ x 1 ∥ completes the proof for k .
Revised: w k − 1 + η k − 1 e −⟨ x 1 , w k − 1 ⟩ x 1 ∥ completes the proof for k .

Original: The proof of the case that k is odd follows the same routine by simply exchanging x 1 and x 2 in the above proof.
Revised: The proof of the case that k is odd follows the same routine by simply exchanging x 1 and x 2 in the above proof.

Original: Consequently, for ∥∇ f ( w k ) ∥ , we have that when k is even
Revised: Consequently, for ∥∇ f ( w k ) ∥ , we have that when k is even

Original: [Equation]
Revised: [Equation]

Original: Similar claim holds for the case k is odd.
Revised: Similar claim holds for the case k is odd.

Original: We then complete the proof that ∥∇ f ( w k ) ∥ → ∞ as k → ∞ .
Revised: We then complete the proof that ∥∇ f ( w k ) ∥ → ∞ as k → ∞ .

Original: The claim that f ( w k ) → ∞ as k → ∞ can be derived following the same routine.
Revised: The claim that f ( w k ) → ∞ as k → ∞ can be derived following the same routine.

Original: The proof is completed.
Revised: The proof is completed.

Original: On the other hand, as f in Example 1 satisfies Assumption 1 and Assumption 2 (with D 0 = 0 ), Theorem 1 applies and full batch Adam will converge to a stationary point regardless of the initialization.
Revised: On the other hand, as f in Example 1 satisfies Assumption 1 and Assumption 2 (with D 0 = 0 ), Theorem 1 applies and full batch Adam will converge to a stationary point regardless of the initialization.

Original: Example 1 indicates that even for the full-batch case, Adam appears to be less sensitive to the initialization compared with GD under ( L 0 , L 1 ) -smooth condition.
Revised: Example 1 indicates that even for the full-batch case, Adam appears to be less sensitive to the initialization compared with GD under ( L 0 , L 1 ) -smooth condition.

Original: However, under the same setting and fixing the initialization point, tuning down the initial learning rate η 1 may help GD converge.
Revised: However, under the same setting and fixing the initialization point, tuning down the initial learning rate η 1 may help GD converge.

Original: Specifically, when gradient norm of the loss sub-level set { w : f ( w ) ≤ f ( w 0 ) } is upper bounded, if we choose the learning rate to be sufficiently small (for example, smaller than 2 / ( L 0 + L 1 M ) 1 where M is the gradient norm upper bound of the loss sub-level set), we can make the loss keeps decreasing as GD iterates, and the parameter stays in the loss sub-level set.
Revised: Specifically, when gradient norm of the loss sub-level set { w : f ( w ) ≤ f ( w 0 ) } is upper bounded, if we choose the learning rate to be sufficiently small (for example, smaller than 2 / ( L 0 + L 1 M ) 1 where M is the gradient norm upper bound of the loss sub-level set), we can make the loss keeps decreasing as GD iterates, and the parameter stays in the loss sub-level set.

Original: However, such a learning rate can be rather small when M is large and leads to a slow convergence of GD, which is exactly the intuition behind Property 1.
Revised: However, such a learning rate can be rather small when M is large and leads to a slow convergence of GD, which is exactly the intuition behind Property 1.

Original: Furthermore, when n ≥ 2 and SGD is adopted, tuning down the learning rate may not help any more.
Revised: Furthermore, when n ≥ 2 and SGD is adopted, tuning down the learning rate may not help any more.

Original: We will discuss such an issue in the following section.
Revised: We will discuss such an issue in the following section.

Original: B. C OMPARE SGD AND A DAM
Revised: C. C OMPARE SGD AND A DAM

Original: First of all, Example 3 can be simply extended to provide a divergent example for SGD.
Revised: First of all, Example 3 can be simply extended to provide a divergent example for SGD.

Revised: 1 2 / ( L 0 + L 1 M ) serves as the largest reasonable setting of the learning rate for GD, as f over the loss sub-level set has smoothness

Original: Example 4 (Example 1, restated) .
Revised: Example 4 (Example 1, restated) .

Original: Let n = 2 and p = 2 with f 0 (( w 1 , w 2 ))
Revised: Let n = 2 and p = 2 with f 0 (( w 1 , w 2 ))

Original: =
Revised: =

Original: e − √ 32 w 1 − 12 w 2 , f 1 ( x ) =
Revised: e − √ 32 w 1 − 12 w 2 , f 1 ( x ) =

Original: e √ 32 w 1 − 12 w 2 , and f (( w 1 , w 2 ))
Revised: e √ 32 w 1 − 12 w 2 , and f (( w 1 , w 2 ))

Original: =
Revised: =

Original: f 0 (( w 1 , w 2 )) .
Revised: f 0 (( w 1 , w 2 )) .

Original: Then, f satisfies Assumptions 1 and 2 (with
Revised: Then, f satisfies Assumptions 1 and 2 (with

Original: D 0 = 0 ).
Revised: D 0 = 0 ).

Original: However, for every initial learning rate η 1 , there exists a region E with infinite measure over R 2 such that, with SGD with decaying learning rate
Revised: However, for every initial learning rate η 1 , there exists a region E with infinite measure over R 2 such that, with SGD with decaying learning rate

Original: η k = η 1 √ k and start point in E , f and ∇ f diverge to infinity.
Revised: η k = η 1 √ k and start point in E , f and ∇ f diverge to infinity.

Original: The proof follows the similar routine as Example 3 and we omit it here.
Revised: The proof follows the similar routine as Example 3 and we omit it here.

Original: Note as D 0 = 0 in this case,
Revised: Note as D 0 = 0 in this case,

Original: Adam still converges to the stationary point.
Revised: Adam still converges to the stationary point.

Original: On the other hand, as discussed in the previous section, for SGD over ( L 0 , L 1 ) -smooth condition, a small learning rate may no longer help and SGD will still diverge.
Revised: On the other hand, as discussed in the previous section, for SGD over ( L 0 , L 1 ) -smooth condition, a small learning rate may no longer help and SGD will still diverge.

Original: We illustrate this idea by consider the following example.
Revised: We illustrate this idea by consider the following example.

Original: Example 5 (Diverging Example for SGD with Small Learning Rate) .
Revised: Example 5 (Diverging Example for SGD with Small Learning Rate) .

Original: Specifically, we rescale the x 1 and x 2 in Example 3 as ˜ x 1 = x 1 and ˜ x 2 = 50 x 2 .
Revised: Specifically, we rescale the x 1 and x 2 in Example 3 as ˜ x 1 = x 1 and ˜ x 2 = 50 x 2 .

Original: We choose initialization point w 1 = ˜ x 1 , and choose learning rate as η 1 = 1 /L with decaying learning rate η t = η 1 √ t , where L is defined as the smoothness upper bound of the loss sub-level set, i.e.,
Revised: We choose initialization point w 1 = ˜ x 1 , and choose learning rate as η 1 = 1 /L with decaying learning rate η t = η 1 √ t , where L is defined as the smoothness upper bound of the loss sub-level set, i.e.,

Original: [Equation]
Revised: [Equation]

Original: By the definition of the previous section, GD on such an example will converge to the stationary point, as the iterates will stay in the sub-level set, the smoothness will be small, and the loss will decrease.
Revised: By the definition of the previous section, GD on such an example will converge to the stationary point, as the iterates will stay in the sub-level set, the smoothness will be small, and the loss will decrease.

Original: However, we run SGD on this example and find that the loss and gradient norm will explode in three steps (Figure 3).
Revised: However, we run SGD on this example and find that the loss and gradient norm will explode in three steps (Figure 3).

Original: [Figure]
Revised: [Figure]

Revised: learning rate and small initial learning rate diverges.

Original: We only plot the first two steps because the values exceed the math range in python 
Revised: We only plot the first two steps because the values exceed the math range in python in the third step.

Original: The intuition behind Example 5 is that the update calculate from a batch in SGD does not essentially reduce the loss and may no longer keeps the iterates stay in the loss sub-level set.
Revised: The intuition behind Example 5 is that the update calculate from a batch in SGD does not essentially reduce the loss and may no longer keeps the iterates stay in the loss sub-level set.

Original: This can eventually make the loss diverging, or keep the parameter away from the saddle point in the training of neural networks.
Revised: This can eventually make the loss diverging, or keep the parameter away from the saddle point in the training of neural networks.

Original: C P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: This appendix provides the formal proof of Theorem 1.
Revised: This appendix provides the formal proof of Theorem 1.

Original: Specifically, we will first make some preparations by ① showing notations and ② proving lemmas which characterize several basic properties of the Adam optimizer, and then prove Theorem 1 based on the lemmas.
Revised: Specifically, we will first make some preparations by ① showing notations and ② proving lemmas which characterize several basic properties of the Adam optimizer, and then prove Theorem 1 based on the lemmas.

Original: Remark 4.
Revised: Remark 10.

Original: In the remaining proof of this paper, we assume without the loss of generality that η 1 is small enough , such that the following requirements are fulfilled (with notations explained latter):
Revised: In the remaining proof of this paper, we assume without the loss of generality that η 1 is small enough , such that the following requirements are fulfilled (with notations explained latter):

Original: • 2 C 2 √ dη 1 ≤ 1 L 1 .
Revised: • 2 C 2 √ dη 1 ≤ 1 L 1 .

Original: This will latter ensure that we can directly apply the definition of ( L  L 1 ) - smooth condition (Assumption 1) to parameter sequence { w k,i } k,i ; • 1 4(2 √ 2+1) ≥√ D 1 C 11 η 1 .
Revised: This will latter ensure that we can directly apply the definition of ( L 0 , L 1 ) - smooth condition (Assumption 1) to parameter sequence { w k,i } k,i ; • 1 4(2 √ 2+1) ≥

Revised: √

Original:  This will latter ensure the second-order term is smaller than the first-order term at the end of the proof.
Revised: D 1 C 11 η 1 . This will latter ensure the second-order term is smaller than the first-order term at the end of the proof.

Original: The proof can be easily extended to general cases (while certainly more cumbersome) by selecting large enough K and using the epoch K as a new start point and derive the results after epoch K
Revised: The proof can be easily extended to general cases (while certainly more cumbersome) by selecting large enough K and using the epoch K as a new start point and derive the results after epoch K

Original: (this is due to η k is decaying, and K is finite, the epochs before epoch k can be uniformly bounded and we then derive the desired result for all epochs).
Revised: (this is due to η k is decaying, and K is finite, the epochs before epoch k can be uniformly bounded and we then derive the desired result for all epochs).

Original: Without the loss of generality, we also take the following initialization: w 1 , 0 = w 0 , m 1 , − 1 = ∇ f τ 1 , − 1 ( w 0 ) ( τ 1 , − 1 can be any integer in [0 , n − 1] ), and ν l, 1 , − 1 = max j { ∂ l f j ( w 0 ) 2 } , ∀ l (here the maximum is taken component-wisely).
Revised: Without the loss of generality, we also take the following initialization: w 1 , 0 = w 0 , m 1 , − 1 = ∇ f τ 1 , − 1 ( w 0 ) ( τ 1 , − 1 can be any integer in [0 , n − 1] ), and ν l, 1 , − 1 = max j { ∂ l f j ( w 0 ) 2 } , ∀ l (here the maximum is taken component-wisely).

Original: We take the initialization to have a more concise proof, while the proof can be easily extended to all the initialization as the information of the initialization in the exponentially decayed average of Adam (both in m k,i and ν k,i ) decays rapidly with k increasing.
Revised: We take the initialization to have a more concise proof, while the proof can be easily extended to all the initialization as the information of the initialization in the exponentially decayed average of Adam (both in m k,i and ν k,i ) decays rapidly with k increasing.

Original: C. P RELIMINARIES C.1. N OTATIONS
Revised: D. P RELIMINARIES D.1. N OTATIONS

Original: Here we provide a complete list of notations used in the appendix for a clear reference.
Revised: Here we provide a complete list of notations used in the appendix for a clear reference.

Original: • We use ( k 1 , i 1 ) ≤ ( < )( k 2 , i 2 ) for ∀ k 1 , k 2 ∈ N +
Revised: • We use ( k 1 , i 1 ) ≤ ( < )( k 2 , i 2 ) for ∀ k 1 , k 2 ∈ N +

Original: and i 1 , i 2 ∈ { 0 , · · · , n − 1 } , if either k 1 < k 2 or k 

Revised: and i 1 , i 2 ∈ { 0 , · · · , n − 1 } , if either k 1 < k 2 or k

Original: = k 

Revised: = k

Original: and i 1 ≤ ( < ) i• We define function g ( x ) :
Revised: and i 1 ≤ ( < ) i• We define function g ( x ) :

Original: [0 , 1] → R / − as g ( β 2 ) ≜ max 
Revised: [0 , 1] → R / − as g ( β 2 ) ≜ max   1 (cid:113) β n − 1− 1 , 1 − 1 (cid:114) β n − 1+ 8 n 1 − β n − 1 2 β n, 1 − (cid:112) β 2 , (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original: • We define constants { C i } 

Revised: • We define constants { C i }

Original: i
Revised: i

Original: =1 as follows:
Revised: =1 as follows:

Original: [Equation] 
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation] 
Revised: [Equation] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: C.1. A UXILIARY L EMMAS
Revised: D.1. A UXILIARY L EMMAS

Original: Here we provide auxiliary lemmas describing basic properties of Adam and the descent lemma under ( L 0 , L 1 ) -smooth condition. 
Revised: Here we provide auxiliary lemmas describing basic properties of Adam and the descent lemma under ( L 0 , L 1 ) -smooth condition. Smoothness of f Lemma 2.

Revised: With Assumptions 2 and 1, f satisfies ( nL 0 + L 1 √ n √ D 0 , L 1 √ n √ D 1 ) -smooth condition.

Revised: Proof.

Original:  Basic Properties of Adam
Revised: ∀ w 1 , w 2 ∈ R d satisfying [Equation] The proof is completed. Basic Properties of Adam

Original: The following lemma characterizes the update norm of Adam.
Revised: The following lemma characterizes the update norm of Adam.

Original: Lemma 1 (Bounded Update) .
Revised: Lemma 3 (Bounded Update) .

Original: If β 1 < √ β 2 , we have ∀ k ∈ N + , i ∈ { 0 , · · · , n − 1 } ,
Revised: If β 1 < √ β 2 , we have ∀ k ∈ N + , i ∈ { 0 , · · · , n − 1 } ,

Original: [Equation]
Revised: [Equation]

Original: m l,k,i | √
Revised: i

Original: [Equation]
Revised: [Equation]

Original: l,k,i + ε ≤ C 1 , where
Revised: 1 , where

Original: [Equation]
Revised: [Equation]

Original: Furthermore, we have | w l,k,i +1 − w l,k,i | ≤ C 1 η k , and thus ∥ w k,i +1 − w k,i ∥ ≤ C 1 η k √ d .
Revised: Furthermore, we have | w l,k,i +1 − w l,k,i | ≤ C 1 η k , and thus ∥ w k,i +1 − w k,i ∥ ≤ C 1 η k √ d .

Original: Proof.
Revised: Proof.

Original: By the definition of m k,i , we have
Revised: By the definition of m k,i , we have

Original: [Equation]
Revised: [Equation]

Original: m l,k
Revised: m l,k

Original: ,i
Revised: ,i

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = 

Original: [Equation]
Revised: [Equation]

Original: ≤ 
Revised: ≤ 

Revised: [Equation]

Revised: m

Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Original: max
Revised: max

Original: [Equation]
Revised: [Equation]

Original: ( ⋆ )

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Revised: ( ⋆ ) ≤   (

Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: j

Revised: [Equation]

Revised: [Equation]

Original: |
Revised: |

Original: [Equation]
Revised: [Equation]

Revised: f

Original: [Equation]
Revised: [Equation]

Revised: + (

Original: [Equation] 
Revised: [Equation] [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation] 
Revised: [Equation] [Equation] [Equation]

Revised: ) n

Original: [Equation]
Revised: [Equation]

Original: n
Revised: i

Original: [Equation]
Revised: [Equation]

Revised: (( m

Original: [Equation]
Revised: [Equation]

Revised: ) n

Original: [Equation]
Revised: [Equation]

Original: |
Revised: j )|

Original: [Equation]
Revised: [Equation]

Original: l f
Revised: l f

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Revised: j (

Revised: [Equation]

Revised: m

Original: [Equation]
Revised: [Equation]

Revised: j )

Revised: [Equation]

Revised: [Equation]

Revised: ( k

Revised: [Equation]

Original: where Eq.
Revised: where Eq.

Original: ( ⋆ ) is due to the Cauchy-Schwartz’s Inequality.
Revised: ( ⋆ ) is due to the Cauchy-Schwartz’s Inequality [Equation]

Revised: and Eq.

Revised: ( ∗ ) is due to the definition of ν l, 1 , − 1 .

Original: We complete the proof of the first claim.
Revised: We complete the proof of the first claim.

Original: The second claim then follows directly from the update rule
Revised: The second claim then follows directly from the update rule

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: Based on Lemma
Revised: Based on Lemma

Original: [Equation]
Revised: [Equation]

Original: we then provide estimations for the norms of the momentum and the adaptor.
Revised: we then provide estimations for the norms of the momentum and the adaptor.

Original: Lemma 2 (Estimation of the norm of the momentum) .
Revised: Lemma 4 (Estimation of the norm of the momentum) .

Original: We have for all l ∈ [ d ] , k ∈ Z
Revised: We have for all l ∈ [ d ] , k ∈ Z

Original: + , i ∈ [ n ] ,| m l,k,i | ≤
Revised: + , i ∈ [ n ] ,

Original: [Equation]
Revised: [Equation]

Original: i ′
Revised: 1 (cid:112) D

Original: [Equation]
Revised: [Equation]

Revised: η k i − 1 (cid:88)

Original:  ∥∇ f ( w k,j ) ∥
Revised: j = ∥∇ f ( w k,j ) ∥

Original: + L 1 C 1 (cid:112)
Revised: + L 1 C 1 (cid:112)

Original: Dk − 1 (cid:88) t =
Revised: Dk − 1 (cid:88) t =

Original: η k − t n − 1 (cid:88)
Revised: η k − t n − 1 (cid:88)

Original: j = β tn
Revised: j = β tn

Original: + i − j∥∇
Revised: + i − j∥∇

Original: f
Revised: f

Original: ( w k − t,j ) ∥ .
Revised: ( w k − t,j ) ∥ .

Original: Similarly, l ∈ [ d ] , k ∈ Z + / { 1 } , | m l,k − 1 ,n − 1 | ≤ max i ′ ∈ [ n ] 
Revised: Similarly, l ∈ [ d ] , k ∈ Z + / { 1 } , | m l,k − 1 ,n − 1 | ≤ max i ′ ∈ [ n ] | ∂ l f i ′ ( w k, 0 ) | +

Original: [Equation]
Revised: [Equation]

Original: i ′
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: w
Revised: (cid:88) [Equation]

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: To begin with, for any t ∈ [ k − 1] and any j ∈ [
Revised: To begin with, for any t ∈ [ k − 1] and any j ∈ [

Original: [Equation]
Revised: [Equation]

Original: n − 1]
Revised: n − 1]

Original: [Equation]
Revised: [Equation]

Original: we have the following estimation for ∂ l f i ( w k − t,j ) :
Revised: we have the following estimation for ∂ l f i ( w k − t,j ) :

Original: [Equation]
Revised: [Equation]

Original: k
Revised: η

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: t [Equation] d [Equation]
Revised: where Inequality ( ⋆ ) is due to ( L 0 , L 1 )

Original: 2 √ [Equation] where Inequality ( ∗ ) is due to ∀ a, b ∈ N + , a > b , (cid:80) bi = 
Revised: -smooth condition, and Inequality ( ∗ ) is due to ∀ a, b ∈ N + , a > b , (cid:80) bi = 1 √ a − i ≤

Original: [Equation]
Revised: [Equation]

Original:  Similarly, we have that for any j ∈ [0 , n − 1] ,
Revised: b +1 a . Similarly, we have that for any j ∈ [0 , n − 1] ,

Original: [Equation]
Revised: [Equation]

Original: Therefore, the norm of m l,k,i can be bounded as | m l,k,i | [Equation] i [Equation]
Revised: Therefore, the norm of m l,k,i can be bounded as | m l,k,i | ≤ (1 − β 1 ) i (cid:88)

Original: [Equation] [Equation] [Equation] j )
Revised: j = β ( k − 1) n

Original: [Equation] [Equation] 1 ) k [Equation]
Revised: + i − (( k − 1) n + j )| ∂ l f τ k,j ( w k,j ) |

Original: [Equation] t = n − 1 (cid:88)[Equation]
Revised: + (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)

Original: [Equation] j [Equation]
Revised: j = β tn + i − j| ∂

Original:  f [Equation] k [Equation]
Revised: l f τ k − t,j ( w k − t,j ) | + β ( k − 1) n + i +1| ∂

Original:  ) | [Equation] [Equation]
Revised: l f τ 1 , 0 ( w 1 , 0 ) | ≤ (1 − β 1 ) i (cid:88)

Original:  k [Equation]
Revised: j = β ( k − 1) n

Original: [Equation]
Revised: + i − (( k − 1) n + j )| ∂ l f τ k,j ( w k, 0 ) |

Original:  t = n − 1 (cid:88)j [Equation] [Equation]
Revised: + (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)

Original: i [Equation] j 
Revised: j = β tn

Original: [Equation] l f [Equation]
Revised: + i − j| ∂

Original:  k [Equation] t [Equation]
Revised: l f τ k − t,j ( w k, 0 )

Original: | [Equation] [Equation] [Equation]
Revised: | + β ( k − 1) n + i +1| ∂

Original: C [Equation] dL [Equation]
Revised: l f τ 1 , 0 ( w k, 0 ) | + (1 − β 1 ) i (cid:88)

Original: [Equation] D [Equation] C
Revised: j = β ( k − 1) n

Original: [Equation] η k √ dj (cid:33) [Equation] [Equation]
Revised: + i − (( k − 1) n + j )(cid:32)

Original: t

Original: [Equation]

Original: n [Equation] (cid:88) j 
Revised: j − 1 (cid:88) p =

Original: [Equation] β tn [Equation]
Revised: C 1 η k √ dL 1 (cid:112) D 1 ∥∇ f ( w k,p ) ∥

Original: i [Equation]
Revised: + ( L 0 + L 1 (cid:112) D 0 ) C 1 η k √ dj (cid:33) + (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)

Original: j [Equation]
Revised: j = β tn

Original:   n [Equation] (cid:88) p =
Revised: + i − j  n − 1 (cid:88) p =

Original: j
Revised: j

Original: C 1 η k − t √ dL
Revised: C 1 η k − t √ dL

Original: [Equation]
Revised: [Equation]

Original: +

Original: [Equation]

Original: =

Original: [Equation] + β ( [Equation] [Equation]
Revised: where Inequality ( ⋆ ) is due to a exchange in the sum order.

Original: ≤ max

Original: [Equation] Following the same routine, we have | m l,k, − 1 | ≤ (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88) j
Revised: Following the same routine

Original: [Equation]
Revised: [Equation]

Original: β
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: j
Revised: m l

Original: [Equation]
Revised: [Equation]

Original: t
Revised: k

Original: [Equation]
Revised: [Equation]

Original: j )
Revised: − 1 |

Original: [Equation]
Revised: [Equation]

Original:  ≤ (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)
Revised: 1 , 0 ) | ≤ (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)

Original: j = β tn − 1 − j| ∂
Revised: j = β tn − 1 − j| ∂

Original: l f τ k − t,j ( w k, 0 )
Revised: l f τ k − t,j ( w k, 0 )

Original: | + β ( k − 1) n| ∂ l f τ 1 , 0 ( w k, 0 ) | + (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)
Revised: | + β ( k − 1) n| ∂ l f τ 1 , 0 ( w k, 0 ) | + (1 − β 1 ) k − 1 (cid:88) t = n − 1 (cid:88)

Original: j = β tn − 1 − j
Revised: j = β tn − 1 − j

Original: C√ d   n − 1 (cid:88) p =
Revised: C√ d   n − 1 (cid:88) p =

Original: j
Revised: j

Original: L 1 (cid:112) D 1 ∥∇ f ( w k − t,p ) ∥
Revised: L 1 (cid:112) D 1 ∥∇ f ( w k − t,p ) ∥

Original: η k − t + t − 1 (cid:88) r
Revised: η k − t + t − 1 (cid:88) r

Original: = n − 1 (cid:88) p =
Revised: = n − 1 (cid:88) p =

Original: L 1 (cid:112) D 1 ∥∇ f ( w k − r,p ) ∥
Revised: L 1 (cid:112) D 1 ∥∇ f ( w k − r,p ) ∥

Original: η k − r + 2 √ 

Revised: η k − r + 2 √

Original: ( L 0 + L 1 (cid:112) D 0 ) C 1 √ dη k ( tn − j ) (cid:17) + β ( k − 1) n(cid:32)
Revised: ( L 0 + L 1 (cid:112) D 0 ) C 1 √ dη k ( tn − j ) (cid:17) + β ( k − 1) n(cid:32)

Original: k − 1 (cid:88) t = n − 1 (cid:88) p =
Revised: k − 1 (cid:88) t = n − 1 (cid:88) p =

Original: L 1 (cid:112) D 1 ∥∇ f ( w k − r,p ) ∥
Revised: L 1 (cid:112) D 1 ∥∇ f ( w k − r,p ) ∥

Original: C 1 η k − r √ d
Revised: C 1 η k − r √ d

Original: + 2 √ 2( L 0 + L 1 (cid:112) D 0 ) C 1 √ dη k ( k − 1) n (cid:33) ≤ max i ∈ [ n ] | ∂ l f i ( w k, 0 )
Revised: + 2 √ 2( L 0 + L 1 (cid:112) D 0 ) C 1 √ dη k ( k − 1) n (cid:33) ≤ max i ∈ [ n ] | ∂ l f i ( w k, 0 )

Original: | + k − 1 (cid:88) t
Revised: | + k − 1 (cid:88) t

Original: = n − 1 (cid:88)
Revised: = n − 1 (cid:88)

Original: j = β tn − 1 − j
Revised: j = β tn − 1 − j

Original: C 1 η k − t √ dL 1 (cid:112) D 1 ∥∇ f ( w k − t,j ) ∥ +√ 

Revised: C 1 η k − t √ dL 1 (cid:112) D 1 ∥∇ f ( w k − t,j ) ∥ +√

Original: ( L 0 + L 1 √ D 0 ) C 1 √ dη k 1 − β.
Revised: ( L 0 + L 1 √ D 0 ) C 1 √ dη k 1 − β.

Original: The proof is completed.
Revised: The proof is completed.

Original: Lemma 3 (Estimation of the norm of the adaptor) .
Revised: Lemma 5 (Estimation of the norm of the adaptor) .

Original: We have for all l ∈ [ d ] , k ∈ Z + , | 
Revised: We have for all l ∈ [ d ] , k ∈ Z + , | ν l,k, 0 | ≥ β n 2 1 − β

Revised: [Equation]

Revised: 1 − β n 2 (cid:88) i ∈ [ n ]

Original: [Equation]
Revised: [Equation]

Revised: f

Revised: [Equation]

Revised: [Equation]

Revised: (cid:88) i ∈ [ n ]

Revised: [Equation]

Revised: −

Revised: [Equation]

Original: l
Revised: n

Original: [Equation]
Revised: [Equation]

Original: |
Revised: √ [Equation] − [Equation]

Revised: [Equation]

Revised:  

Original: [Equation]
Revised: [Equation]

Original: and | ν
Revised: (cid:88) t = β n [Equation] (cid:112) [Equation]

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: ) n

Original: [Equation]
Revised: [Equation]

Revised: − t n

Revised: [Equation]

Revised: (cid:88)

Revised: j = ( (cid:112) D

Revised: [Equation]

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: [Equation]

Revised: (cid:112) D 0 )

Revised: 

Original:  max 
Revised:    , and | ν l,k, 0 | ≤ [Equation] max i ∈ [ n ]

Original: [Equation]
Revised: [Equation]

Revised: w

Revised: [Equation]

Revised: [Equation]

Revised: (cid:18)

Revised: [Equation]

Revised: [Equation]

Revised: (cid:88)

Revised: [Equation]

Revised: = (cid:112)

Revised: [Equation]

Revised: −

Revised: [Equation]

Revised: 

Original:  [Equation]
Revised:  [Equation]

Original: Proof.
Revised: Proof.

Original: By the definition of ν l,k, 0 , we have ν l,k,=
Revised: By the definition of ν l,k, 0 , we have ν l,k,=(1 − β 2 ) ∂

Original: [Equation]{+l f τ k, 0 ( w k, 0 ) 
 + k − 1 (cid:88) t = n − 1 (cid:88)+}
Revised: {+l f τ k, 0 ( w k, 0 ) 
 + k − 1 (cid:88) t = n − 1 (cid:88)+}

Original:  (1 − β 2 ) β tn − j∂ l f τ k − t,j ( w k − t,j )+ β ( k − 1) n +1max i ∈ [ n ] ∂ l f i ( w 1 , 0 )≥ (1 − β 2 ) ∂ l f τ k, 0 ( w k, 0 ) 

Revised: j = (1 − β 2 ) β tn − j∂ l f τ k − t,j ( w k − t,j )+ β ( k − 1) n +1max i ∈ [ n ] ∂ l f i ( w 1 , 0 )≥ (1 − β 2 ) ∂ l f τ k, 0 ( w k, 0 )

Original: + k − 1 (cid:88) t
Revised: + k − 1 (cid:88) t

Original: = n − 1 (cid:88)
Revised: = n − 1 (cid:88)

Original: j = (1 − β 2 ) β tn 2 ∂ l f τ k − t,j ( w k − t,j )+ β ( k − 1)
Revised: j = (1 − β 2 ) β tn 2 ∂ l f τ k − t,j ( w k − t,j )+ β ( k − 1)

Original: n +1 n n (cid:88) i =
Revised: n +1 n n (cid:88) i =

Original: ∂ l f i ( w 1 , 0 )=(1 − β 2 )
Revised: ∂ l f i ( w 1 , 0 )=(1 − β 2 )

Original: ∂
Revised: ∂

Original: l f τ k, 0 ( w k, 0 ) 

Revised: l f τ k, 0 ( w k, 0 )

Original: + k − 1 (cid:88) t
Revised: + k − 1 (cid:88) t

Original: = n − 1 (cid:88)
Revised: = n − 1 (cid:88)

Original: j
Revised: j

Original: = (1 − β 2 ) β tn 2 ( ∂ l f τ k − t,j ( w k, 0 )
Revised: = (1 − β 2 ) β tn 2 ( ∂ l f τ k − t,j ( w k, 0 )

Original: + ∂ l f τ k − t,j ( w k − t,j ) − ∂ l f τ k − t,j ( w k, 0 ))
Revised: + ∂ l f τ k − t,j ( w k − t,j ) − ∂ l f τ k − t,j ( w k, 0 ))

Original: + β ( k − 1) n +1 n n (cid:88) i = ( ∂ l f i ( w k, 0 ) + ∂ l f i ( w 1 , 0 ) − ∂ l f i ( w k, 0 ))≥ (1 − β 2 ) ∂
Revised: + β ( k − 1) n +1 n n (cid:88) i = ( ∂ l f i ( w k, 0 ) + ∂ l f i ( w 1 , 0 ) − ∂ l f i ( w k, 0 ))≥ (1 − β 2 ) ∂

Original: l f τ k, 0 ( w k, 0 ) 

Revised: l f τ k, 0 ( w k, 0 )

Original: + k − 1 (cid:88) t
Revised: + k − 1 (cid:88) t

Original: = n − 1 (cid:88)j = (1 − β 2 ) β tn 2 ∂ l f τ k − t,j ( w k, 0 ) + β ( k − 1) n +1 n n (cid:88) i = ∂ l f i ( w k, 0 )− k − 1 (cid:88) t = n − 1 (cid:88)
Revised: = n − 1 (cid:88)

Original: j

Original:  = (1 − β 2 ) β tn 2| ∂ l f τ k − t,j ( w k, 0 )|| ∂ l f τ k − t,j ( w k, 0 ) − ∂ l f τ k − t,j ( w [Table] t,
Revised: j = (1 − β 2 ) β tn 2 ∂ l f τ k − t,j ( w k, 0 )

Original: [Table] ) | − β ( k [Table] 1) n +
Revised: + β ( k − 1) n +1 n n (cid:88) i =

Original: [Table]

Original: n n (cid:88) i = | ∂ l f i ( w k, 0 ) || ∂ l f i ( w k, 0 ) −∂ l f i ( w 1 , [Table]
Revised: ∂ l f i ( w k, 0 )− k − 1 (cid:88) t = n − 1 (cid:88)

Original: ) | (

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: [Equation]

Original: n [Equation] [Equation] ( k
Revised: j = (1 − β 2 ) β tn 2 |

Original: [Equation]

Original: n

Original: [Equation]

Original: n (cid:33)

Original: [Table]
Revised: [Table]

Original: i

Original: [Equation]

Original: [Table]

Original: [Table]

Original: f
Revised: f

Original: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: [Table]

Original: |

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Table]

Original: C

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: ) C

Original: [Table]

Original: [Equation]

Original: dη

Original: [Table]

Original: (cid:33)

Original: [Table]

Original: n

Original: [Table]

Original: i =

Original: [Table]

Original: |

Original: [Table]
Revised: [Table]

Original: f
Revised: f

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: ) | (cid:32)

Original: [Table]

Original: r =

Original: [Table]
Revised: [Table]

Original: n

Original: [Table]
Revised: [Table]

Original: p

Original: [Equation]

Original: [Table]

Original: L

Original: [Equation]

Original: [Equation]

Original: [Table]

Original: r [Equation] [Equation] C
Revised: The first claim is proved

Original: [Table]
Revised: [Table]

Original: η k [Table] √
Revised: As for the upper bound, we have

Original: [Equation]
Revised: [Equation]

Original: √

Original: [Equation]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: D

Original: [Table]
Revised: [Table]

Original: ) C [Table] dη [Table]
Revised: [Equation]

Original: ) n (cid:33) [Table]The first claim is proved. As for the upper bound, we have
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: [Equation] [Table] 
Revised: [Equation] [Table] [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: We then immediately have the following corollary when max i ∈ [ n ] | ∂ l f i ( w k, 0 ) | is large enough.
Revised: We then immediately have the following corollary when max i ∈ [ n ] | ∂ l f i ( w k, 0 ) | is large enough.

Original: Corollary 2.
Revised: Corollary 2 (Lemma 1, formal) [Equation]

Original: If
Revised: If

Original: [Equation]
Revised: [Equation]

Original: then
Revised: then

Original: [Equation]
Revised: [Equation]

Original: Furthermore, if Eq.
Revised: Furthermore, if Eq.

Original: (10) holds, we have ∀ i ∈ { 0 , · · · , n − 1 } ,
Revised: (11) holds, we have ∀ i ∈ { 0 , · · · , n − 1 } ,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The first claim is derived by directly applying the range of max i ∈ [ n ] | ∂ l f i ( w k, 0 ) | into Lemma 3.
Revised: The first claim is derived by directly applying the range of max i ∈ [ n ] | ∂ l f i ( w k, 0 ) | into Lemma 5.

Original: As for the second claim
Revised: As for the second claim

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: On the other hand, since ∀ j ∈ {
Revised: On the other hand, since ∀ j ∈ {

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: n − 1 }
Revised: n − 1 }

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Therefore, if Eq.
Revised: Therefore, if Eq.

Original: (10) holds
Revised: (11) holds

Original: [Equation]
Revised: [Equation]

Original: we then have
Revised: we then have

Original: [Equation]
Revised: [Equation]

Original: Following the same routine
Revised: Following the same routine

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: and if Eq.
Revised: and if Eq.

Original: (10) holds,
Revised: (11) holds,

Original: [Equation]
Revised: [Equation]

Original: The proof of the second claim is completed
Revised: The proof of the second claim is completed

Original: [Equation]
Revised: [Equation]

Original: Remark 5.
Revised: Remark 11.

Original: For brevity, we denote
Revised: For brevity, we denote

Original: [Equation]
Revised: [Equation]

Original: The right-hand-size of Eq.
Revised: The right-hand-size of Eq.

Original: (10) is smaller than
Revised: (11) is smaller than

Original: [Equation]
Revised: [Equation]

Revised: j

Original: [Equation] 
Revised: [Equation] [Equation]

Original: n
Revised: n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: j =
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: f (
Revised: f (

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: j
Revised: j

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Furthermore, we define g ( β 2 ) as g
Revised: Furthermore, we define g ( β 2 ) as g

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [Equation]
Revised: [Equation]

Original: and the conclusion of Corollary 2 can be translated into that if Eq.
Revised: and the conclusion of Corollary 2 can be translated into that if Eq.

Original: (11) holds,
Revised: (12) holds,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: In the end of this section, we draw the connection of the gradients across the epoch.
Revised: In the end of this section, we draw the connection of the gradients across the epoch.

Original: Lemma 4.
Revised: Lemma 6.

Original: ∀ k ∈ N + , i ∈ { 0 , · · · , n − 1 } , ∥∇ 
Revised: ∀ k ∈ N + , i ∈ { 0 , · · · , n − 1 } , ∥∇ f ( w

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By Assumption 1, we have ∥∇ f ( w k,i )
Revised: By Assumption 1, we have ∥∇ f ( w k,i )

Original: ∥ ≤∥∇ f ( w k, 0 ) ∥
Revised: ∥ ≤∥∇ f ( w k, 0 ) ∥

Original: + (cid:32) nL 

Revised: + (cid:32) nL

Original: + Ln (cid:88)
Revised: + Ln (cid:88)

Original: i =
Revised: i =

Original: ∥∇ f i ( w k, 0 ) ∥ (cid:33)
Revised: ∥∇ f i ( w k, 0 ) ∥ (cid:33)

Original: ∥ w k,i − w k, 0 ∥ ≤∥∇ f ( w k, 0 ) ∥
Revised: ∥ w k,i − w k, 0 ∥ ≤∥∇ f ( w k, 0 ) ∥

Original: + (cid:32) nL 

Revised: + (cid:32) nL

Original: + Ln (cid:88)
Revised: + Ln (cid:88)

Original: i =
Revised: i =

Original: ∥∇ f i ( w k, 0 ) ∥ (cid:33) i √
Revised: ∥∇ f i ( w k, 0 ) ∥ (cid:33) i √

Original: dC 1 η k ≤∥∇ f ( w k, 0 )
Revised: dC 1 η k ≤∥∇ f ( w k, 0 )

Original: ∥ +   nL 0 + L 1 √ n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)
Revised: ∥ +   nL 0 + L 1 √ n (cid:118)(cid:117)(cid:117)(cid:116) n (cid:88)

Original: i =
Revised: i =

Original: ∥∇ f i ( w k, 0 )
Revised: ∥∇ f i ( w k, 0 )

Original: ∥  i √ dC 1 η k ≤∥∇ f ( w k, 0 )
Revised: ∥  i √ dC 1 η k ≤∥∇ f ( w k, 0 )

Original: ∥ + (cid:16) nL 0 + L 1 √ n (cid:112) D 1 ∥∇ f ( w k, 0 ) ∥
Revised: ∥ + (cid:16) nL 0 + L 1 √ n (cid:112) D 1 ∥∇ f ( w k, 0 ) ∥

Original: + L 1 √ n (cid:112) D 0 (cid:17)
Revised: + L 1 √ n (cid:112) D 0 (cid:17)

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: Descent Lemma Under ( L 0 , L 1 ) -smooth
Revised: Descent Lemma Under ( L 0 , L 1 ) -smooth

Original: condition We need a descent lemma assuming ( L 0 , L 1 ) - smooth condition similar as the case assuming L smoothness.
Revised: condition We need a descent lemma assuming ( L 0 , L 1 ) - smooth condition similar as the case assuming L smoothness.

Original: Specifically, for a function h : X → R satisfying L -smooth condition and two points w and v in the domain X , by Taylor’s expansion, we have
Revised: Specifically, for a function h : X → R satisfying L -smooth condition and two points w and v in the domain X , by Taylor’s expansion, we have

Original: [Equation]
Revised: [Equation]

Original: This is called "Descent Lemma" by existing literature Sra (2014), as it guarantees that the loss decreases with proper parameter update.
Revised: This is called "Descent Lemma" by existing literature Sra (2014), as it guarantees that the loss decreases with proper parameter update.

Original: Parallel to the above inequality, we have the following
Revised: Parallel to the above inequality, we have the following

Original: Descent Lemma under ( L 0 , L 1 ) smoothness.
Revised: Descent Lemma under ( L 0 , L 1 ) smoothness.

Original: Lemma 5.
Revised: Lemma 7.

Original: Assume that function h : X → R satisfies ( L 0 , L 1 )
Revised: Assume that function h : X → R satisfies ( L 0 , L 1 )

Original: -smooth condition, i.e., ∀ w , v ∈ X satisfying
Revised: -smooth condition, i.e., ∀ w , v ∈ X satisfying

Original: [Equation]
Revised: [Equation]

Original: Then, for any three points u , w , v ∈ X satisfying ∥ w − u ∥ ≤ 1 L 1 and ∥ v − u ∥ ≤ 1 L 1 .
Revised: Then, for any three points u , w , v ∈ X satisfying ∥ w − u ∥ ≤ 1 L 1 and ∥ v − u ∥ ≤ 1 L 1 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By the Fundamental theorem of calculus, we have
Revised: By the Fundamental theorem of calculus, we have

Original: [Equation]
Revised: [Equation]

Original: where Inequality ( ⋆ ) is due to ∥
Revised: where Inequality ( ⋆ ) is due to ∥

Original: [Equation]
Revised: [Equation]

Original: [Equation] and thus the definition of ( L 0 , L 1 ) -smooth condition can be applied.
Revised: and thus the definition of ( L 0 , L 1 ) -smooth condition can be applied [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: C. P ROOF OF A DAM ’ S C ONVERGENCE
Revised: D. P ROOF OF A DAM ’ S C ONVERGENCE

Original: Proof of Theorem 1.
Revised: Proof of Theorem 1.

Original: We define u k ≜ w k, 0 − β 1 w k, − 1 1 − β(with w 1 , − 1 ≜ w 1 , 0 = w 0 ), and let u l,k be the i -th component of u k , ∀ k ∈ N + , l ∈ [ d ] .
Revised: We define u k ≜ w k, 0 − β 1 w k, − 1 1 − β(with w 1 , − 1 ≜ w 1 , 0 = w 0 ), and let u l,k be the i -th component of u k [Equation] ∀ k ∈ N + [Equation]

Revised: l ∈ [ d ] .

Original: Then, by Lemma
Revised: Then, by Lemma

Original: [Equation]
Revised: [Equation]

Original: we immediately have ∀ l ∈
Revised: we immediately have ∀ l ∈

Original: [ d ] , | u l,k − w l,k, 0 | is bounded as
Revised: [ d ] , | u l,k − w l,k, 0 | is bounded as

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where C 2 is defined as C 2 ≜ nC
Revised: where C 2 is defined as C 2 ≜ nC

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: We then analyze the change of the Lyapunov function f ( u k ) along the iterations.
Revised: We then analyze the change of the Lyapunov function f ( u k ) along the iterations.

Original: Specifically, by
Revised: Specifically, by

Original: Lemma 5, we have
Revised: Lemma 7, we have

Original: [Equation]
Revised: [Equation]

Original: Here in Eq.
Revised: Here in Eq.

Original: ( ∗ ) , L large and L small are respectively defined as
Revised: ( ∗ ) , L large and L small are respectively defined as

Original: L klarge =
Revised: L klarge =

Original: { l : l ∈ [ d ] , s.t. Eq.
Revised: { l : l ∈ [ d ] , s.t. Eq.

Original: (11) holds } ,
Revised: (12) holds } ,

Original: L ksmall =
Revised: L ksmall =

Original: { l : l ∈ [ d ] , s.t. Eq.
Revised: { l : l ∈ [ d ] , s.t. Eq.

Original: (11) doesn’t hold } .
Revised: (12) doesn’t hold } .

Original: Apparently, L large ∪ L small =
Revised: Apparently, L large ∪ L small =

Original: [ d ] .
Revised: [ d ] .

Original: We than tackle (cid:80) l ∈ L klarge ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) and (cid:80) l ∈ L ksmall ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) respectively.{++
+}
Revised: We then tackle (cid:80) l ∈ L klarge ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) and (cid:80) l ∈ L ksmall ∂ l f ( w k, 0 )( u l,k {++
+}

Original:  ① Analysis for (cid:80) l ∈ L ksmall ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) :
Revised: − u l,k ) respectively. ① Analysis for (cid:80) l ∈ L ksmall ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) :

Original: By directly applying the range of max i ∈ [ n ] | ∂ l f i ( w k, 0 ) | , we have
Revised: By directly applying the range of max i ∈ [ n ] | ∂ l f i ( w k, 0 ) | , we have

Original: [Equation]
Revised: [Equation]

Original: ≤ dC
Revised: ≤ dC

Original: [Equation]
Revised: [Equation]

Original: Summing over k from 1 to t then leads to
Revised: Summing over k from 1 to t then leads to

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: which by Lemma 4 further leads to
Revised: which by Lemma 6 further leads to

Original: T
Revised: T

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: We further define
Revised: We further define

Original: [Equation]
Revised: [Equation]

Original: and thus
Revised: and thus

Original: T (cid:88) k
Revised: T (cid:88) k

Original: = (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:88) l ∈ L ksmall
Revised: = (cid:12) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:88) l ∈ L ksmall

Original: [Equation]
Revised: [Equation]

Original: ② Analysis for (cid:80) l ∈ L klarge ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) :
Revised: ② Analysis for (cid:80) l ∈ L klarge ∂ l f ( w k, 0 )( u l,k +1 − u l,k ) :

Original: This term requires a more sophisticated analysis.
Revised: This term requires a more sophisticated analysis.

Original: To begin with, we provide a decomposition of u k +1 − u k .
Revised: To begin with, we provide a decomposition of u k +1 − u k .

Original: According to the definition of u k , we have
Revised: According to the definition of u k , we have

Original: [Equation]
Revised: [Equation]

Original: Here equation ( ⋆ ) is due to a direct application of the update rule of w k,i .
Revised: Here equation ( ⋆ ) is due to a direct application of the update rule of w k,i .

Original: We then analyze the above three terms respectively, namely, we define
Revised: We then analyze the above three terms respectively, namely, we define

Original: [Equation]
Revised: [Equation]

Original: One can then easily observe that by Eq.
Revised: One can then easily observe that by Eq.

Original: (16), (cid:88)
Revised: (17), (cid:88)

Original: l ∈ L klarge
Revised: l ∈ L klarge

Original: [Equation]
Revised: [Equation]

Original: ② .
Revised: ② .

Original: [Equation]
Revised: [Equation]

Original:  Tackling
Revised: A) Tackling

Original: [Equation]
Revised: [Equation]

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: where Eq.
Revised: where Eq.

Original: ( ⋆ ) is due to
Revised: ( ⋆ ) is due to

Original: [Table]
Revised: [Table]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: n
Revised: n

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: w k
Revised: w k

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ∥ 
Revised: ∥ .

Original: [Equation]

Original: Here Eq.
Revised: Here Eq.

Original: ( ∗ ) is due to Corollary 2, Eq.
Revised: ( ∗ ) is due to Corollary 2, Eq.

Original: ( ◦ ) is due to f i is ( L 0 , L 1 ) -smooth, ∀ i , and Eq.
Revised: ( ◦ ) is due to f i is ( L 0 , L 1 ) -smooth, ∀ i , and Eq.

Original: ( • ) is due to
Revised: ( • ) is due to

Original: Lemma 1.
Revised: Lemma 3.

Original: ② .
Revised: ② .

Original: (B) Tackling Term (cid:80)
Revised: (B) Tackling Term (cid:80)

Original: l ∈ L klarge ∂ l f ( w
Revised: l ∈ L klarge ∂ l f ( w

Original: [Equation]
Revised: [Equation]

Original:  We have for any l ∈ L max ,
Revised: l : We have for any l ∈ L max ,

Original: [Equation]
Revised: [Equation]

Original: where Inequality ( ⋆ ) is due to Corollary
Revised: where Inequality ( ⋆ ) is due to Corollary

Original: [Equation]
Revised: [Equation]

Original: and g ( β 2 ) is defined in Lemma
Revised: and g ( β 2 ) is defined in Lemma

Original: [Equation]
Revised: [Equation]

Original: and Inequality ( ∗ ) is due to Lemma2, by which we have ∀ i ∈ {− 1 , · · · , n − 1 } | m l,k,i | ≤ max i ′ ∈ [ n ]
Revised: and Inequality ( ∗ ) is due to Lemma

Original: [Equation]
Revised: [Equation]

Original:  i 
Revised: by which we have ∀ i ∈ {− 1 , · · · , n − 1 } | m l,k,i | ≤

Original: [Equation]
Revised: [Equation]

Original: w
Revised: i ′ ∈ [ n

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, summing over L klarge and k leads to
Revised: Therefore, summing over L klarge and k leads to

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where Inequality ( ⋆ ) is due to Lemma 4.
Revised: where Inequality ( ⋆ ) is due to Lemma 6.

Original: [Equation]
Revised: [Equation]

Original: For any l ∈ L klarge ,
Revised: For any l ∈ L klarge ,

Original: [Equation]
Revised: [Equation]

Original: Summing over k and L klarge then leads to
Revised: Summing over k and L klarge then leads to

Original: [Equation]
Revised: [Equation]

Original: Put ② .(A), ② .(B), and ② .(C) together.
Revised: Put ② .(A), ② .(B), and ② .(C) together.

Original: We have
Revised: We have

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: klarge
Revised: klarge

Original: [Equation]
Revised: [Equation]

Original: klarge
Revised: klarge

Original: [Equation]
Revised: [Equation]

Original: where C 5 , C 6 , and C 7 are constants defined as
Revised: where C 5 , C 6 , and C 7 are constants defined as

Original: [Equation]
Revised: [Equation]

Original: We then analyze the first two terms in Eq.
Revised: We then analyze the first two terms in Eq.

Original: (17) here.
Revised: (18) here.

Original: Specifically, we have
Revised: Specifically, we have

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where Inequality ( ⋆ ) is due to Corollary 2 and Equality ( ◦ ) is due to
Revised: where Inequality ( ⋆ ) is due to Corollary 2 and Equality ( ◦ ) is due to

Original: [Equation]
Revised: [Equation]

Original: Parallel to Eq.
Revised: Parallel to Eq.

Original: and summing the right-hand-side of the above inequality over k from 1 to t , we have
Revised: and summing the right-hand-side of the above inequality over k from 1 to t , we have

Original: [Equation]
Revised: [Equation]

Original: Suppose now there does not exist an iteration k ∈
Revised: Suppose now there does not exist an iteration k ∈

Original: [ T ]
Revised: [ T ]

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: since otherwise
Revised: since otherwise

Original: [Equation]
Revised: [Equation]

Original: the proof has been completed.
Revised: the proof has been completed.

Original: By Lemma
Revised: By Lemma

Original: [Equation]
Revised: [Equation]

Original: we then have
Revised: we then have

Original: [Equation]
Revised: [Equation]

Original: Putting ① and ② together and summing over k
Revised: Putting ① and ② together and summing over k

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where C 11 , C12 , and C 13 is defined as
Revised: where C 11 , C

Original: [Table]
Revised: [Table]

Original: [Equation] [Table] [Equation] [Table]
Revised: and C 13 is defined as

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: On the other hand, as for ∀ k ∈ [ T ] ,
Revised: On the other hand, as for ∀ k ∈ [ T ] ,

Original: [Equation]
Revised: [Equation]

Original: we have that
Revised: we have that

Original: [Equation]
Revised: [Equation]

Original: and thus,
Revised: and thus,

Original: [Equation]
Revised: [Equation]

Original: k =
Revised: [Table] [Equation] [Table] [Equation]

Original:  [Equation]
Revised: [Table] [Equation] [Table] [Equation]

Original:  [Equation] 
Revised: [Table] [Equation] [Table] (cid:19) .

Original: The proof is completed.
Revised: The proof is completed.

Revised: Remark 12.

Revised: By the definitions of C 11 , C 12 , and C 13 , one can easily observe that the hidden coefficient of O ( ln T √ T ) is in the order of 32 for d and 32 for n .

Revised: For completeness, we would like to emphasize that our contribution is “providing the first convergence bound of Adam without the L-smooth condition”, but we agree the bound itself can be tighten.

Revised: Proving tighter bound is an interesting topic and we leave it as a future work.

Original: Lemma 6.
Revised: Lemma 8.

Original: Let Assumptions 1 and 2 hold.
Revised: Let Assumptions 1 and 2 hold.

Original: Let β 21 < β 2 and Eq.
Revised: Let β 21 < β 2 and Eq.

Original: (4) hold.
Revised: (4) hold.

Original: Then, either there exists a iteration k ∈
Revised: Then, either there exists a iteration k ∈

Original: [ T ] , such that either
Revised: [ T ] , such that either

Original: [Equation]
Revised: [Equation]

Original: or for all iteration k ∈ [1 , T ] , we have that
Revised: or for all iteration k ∈ [1 , T ] , we have that

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: Proof.
Revised: Proof.

Original: To begin with, we have
Revised: To begin with, we have

Original: [Equation]
Revised: [Equation]

Original: [Table] [Equation] [Table] [Equation]
Revised: where Inequality ( ⋆ ) is due to that

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where Inequality ( ⋆ ) is due to that
Revised: k,

Original: [Equation]
Revised: [Equation]

Original:  We respectively consider the case ε ≤ √ D 0 and ε > √ D 0 .
Revised: 2 + D 0 . We respectively consider the case ε ≤ √ D 0 and ε > √ D 0 .

Original: Case I: ε ≤ √ D 0 .
Revised: Case I: ε ≤ √ D 0 .

Original: In this case, we have that
Revised: In this case [Equation] we have that

Original: [Equation]
Revised: [Equation]

Original: We further discuss the case depending on whether ∥∇ f ( w k, 0 ) ∥ 2 ≤ D 0 D 1 or not.
Revised: We further discuss the case depending on whether ∥∇ f ( w k, 0 ) ∥ 2 ≤ D 0 D 1 or not.

Original: Case I.1: ∥∇ f ( w k, 0 )
Revised: Case I.1: ∥∇ f ( w k, 0 )

Original: ∥ 2 ≤ D 0 D 1 .
Revised: ∥ 2 ≤ D 0 D 1 .

Original: In this case, the last line of the above equations can be further lower bounded by
Revised: In this case, the last line of the above equations can be further lower bounded by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where Inequality ( ∗ ) is due to the constraint on β 2 .
Revised: where Inequality ( ∗ ) is due to the constraint on β 2 .

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have either (1).
Revised: we have either (1).

Original: there exists a iteration k ∈
Revised: there exists a iteration k ∈

Original: [ T ]
Revised: [ T ]

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: or (2).for all k ∈ [1 , T ] ,
Revised: or (2).for all k ∈ [1 , T ] ,

Original: [Equation]
Revised: [Equation]

Original: l ∈
Revised: l ∈

Original: [Equation]
Revised: [Equation]

Original: d
Revised: d

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Case II: ε > √ D 0 .
Revised: Case II: ε > √ D 0 .

Original: In this case
Revised: In this case

Original: [Table]
Revised: [Table]

Original: we have that
Revised: we have that

Original: [Table]
Revised: [Table]

Original: Similar as Case I , we further divides the case regarding the value of ∥∇ f ( w k, 0 ) ∥ .
Revised: Similar as Case I , we further divides the case regarding the value of ∥∇ f ( w k, 0 ) ∥ .

Original: Case II.1: D 1 ∥∇ f ( w k, 0 ) ∥ 2 ≤ ε 2 .
Revised: Case II.1: D 1 ∥∇ f ( w k, 0 ) ∥ 2 ≤ ε 2 .

Original: In this case, we have
Revised: In this case, we have

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have either (1).
Revised: Therefore, we have either (1).

Original: there exists a iteration k ∈
Revised: there exists a iteration k ∈

Original: [ T ] , such that
Revised: [ T ] , such that

Original: [Equation]
Revised: [Equation]

Original: or (2).
Revised: or (2).

Original: for all k ∈ [1 , T ] ,
Revised: for all k ∈ [1 , T ] ,

Original: [Equation]
Revised: [Equation]

Original: As a conclusion of Case I and Case II
Revised: As a conclusion of Case I and Case II

Original: [Equation]
Revised: [Equation]

Original: we have that either there exists a iteration k ∈
Revised: we have that either there exists a iteration k ∈

Original: [ T ]
Revised: [ T ]

Original: [Equation]
Revised: [Equation]

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: or for all iteration k ∈ [1 , T ] , we have that
Revised: or for all iteration k ∈ [1 , T ] , we have that

Original: [Equation]
Revised: [Equation]

Original: [Table]

Original: |

Original: [Table]
Revised: [Table]

Original: β n[-
-]
Revised: β n[-
-]

Original: [Table]
Revised: [Table]

Original: i ∈ [ n ] |
Revised: [Table] ] |

Original: [Table]
Revised: [Table]

Original: l
Revised: l

Original: [Table]
Revised: [Table]

Original: i (

Original: [Table]

Original: [Table]
Revised: [Table]

Original: |
Revised: |

Original: [Table]
Revised: [Table]

Original: (cid:18)
Revised: (cid:18)

Original: [Table]
Revised: [Table]

Original: i ∈ [ n ] |
Revised: i ∈ [ n ] |

Original: [Table]
Revised: [Table]

Original: l f i (
Revised: l f i (

Original: [Table]
Revised: [Table]

Original: k, 0 ) |(cid:19)
Revised: k, 0 ) |

Original: [Table]
Revised: [Table]

Original: The proof is completed.
Revised: The proof is completed.

Original: D E XPERIMENT D ETAILS
Revised: E E XPERIMENT D ETAILS

Original: This section collects experiments and their corresponding settings, and is arranged as follows: to begin with, we show that Adam works well under the different reshuffling order; we then provide the experiment settings of Figure 1.
Revised: This section collects experiments and their corresponding settings, and is arranged as follows: to begin with, we show that Adam works well under the different reshuffling order; we then provide the experiment settings of Figure 1.

Original: D. A DAM WORKS WELL UNDER DIFFERENT RESHUFFLING ORDER
Revised: E. A DAM WORKS WELL UNDER DIFFERENT RESHUFFLING ORDER

Original: We run Adam on ResNet 110 for CIFAR 10 across different random seeds and plot the 10-run mean and variance in Figure 4.
Revised: We run Adam on ResNet 110 for CIFAR 10 across different random seeds and plot the 10-run mean and variance in Figure 4.

Original: One can observe that the performance of Adam is robust with respect to random seed, and support Theorem 1 in terms of trajectory-wise convergence.
Revised: One can observe that the performance of Adam is robust with respect to random seed, and support Theorem 1 in terms of trajectory-wise convergence.

Original: The experiment is based on this repo, where we adopt the default hyperparameters settings.
Revised: The experiment is based on this repo, where we adopt the default hyperparameters settings.

Original: D. L OCAL SMOOTHNESS VS . GRADIENT NORM
Revised: E. L OCAL SMOOTHNESS VS . GRADIENT NORM

Original: In this section, we provide the models and hyperparameter settings of Figures 1.
Revised: In this section, we provide the models and hyperparameter settings of Figures 1.

Original: We will also illustrate how we evaluate the local smoothness.
Revised: We will also illustrate how we evaluate the local smoothness.

Original: Models and hyper-parameter settings in Figures 1.
Revised: Models and hyper-parameter settings in Figures 1.

Original: In Figure 1, we use exactly the same setting as
Revised: In Figure 1, we use exactly the same setting as

Original: Vaswani et al.
Revised: Vaswani et al.

Original: (2017) on WMT 2014 dataset, based on this repo.
Revised: (2017) on WMT 2014 dataset, based on this repo.

Original: How we evaluate the local smoothness.
Revised: How we evaluate the local smoothness.

Original: We use the same method as Zhang et al. (2019a).
Revised: We use the same method as Zhang et al. (2019a).

Original: Specifically, with a finite-difference step α , we calculate the smoothness at w k as local smoothness =
Revised: Specifically, with a finite-difference step α , we calculate the smoothness at w k as local smoothness =

Original: [Equation]
Revised: [Equation]