Original: SA DAM : A V ARIANT OF A DAM FOR S TRONGLY C ONVEX F UNCTIONS
Revised: SA DAM : A V ARIANT OF A DAM FOR S TRONGLY C ONVEX F UNCTIONS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The Adam algorithm has become extremely popular for large-scale machine learning.
Revised: The Adam algorithm has become extremely popular for large-scale machine learning.

Original: Under convexity condition, it has been proved to enjoy a data-dependent
Revised: Under convexity condition, it has been proved to enjoy a data-dependent

Original: O ( √ T ) regret bound where T is the time horizon.
Revised: O ( √ T ) regret bound where T is the time horizon.

Original: However, whether strong convexity can be utilized to further improve the performance remains an open problem.
Revised: However, whether strong convexity can be utilized to further improve the performance remains an open problem.

Original: In this paper, we give an afﬁrmative answer by developing a variant of Adam (referred to as SAdam ) which achieves a data-dependent O (log T ) regret bound for strongly convex functions.
Revised: In this paper, we give an afﬁrmative answer by developing a variant of Adam (referred to as SAdam ) which achieves a data-dependent O (log T ) regret bound for strongly convex functions.

Original: The essential idea is to maintain a faster decaying yet under controlled step size for exploiting strong convexity.
Revised: The essential idea is to maintain a faster decaying yet under controlled step size for exploiting strong convexity.

Original: In addition, under a special conﬁguration of hyperparameters, our SAdam reduces to SC-RMSprop, a recently proposed variant of RMSprop for strongly convex functions, for which we provide the ﬁrst data-dependent logarithmic regret bound.
Revised: In addition, under a special conﬁguration of hyperparameters, our SAdam reduces to SC-RMSprop, a recently proposed variant of RMSprop for strongly convex functions, for which we provide the ﬁrst data-dependent logarithmic regret bound.

Original: Empirical results on optimizing strongly convex functions and training deep networks demonstrate the effectiveness of our method.
Revised: Empirical results on optimizing strongly convex functions and training deep networks demonstrate the effectiveness of our method.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Online Convex Optimization (OCO) is a well-established learning framework which has both theoretical and practical appeals (Shalev-Shwartz et al., 2012).
Revised: Online Convex Optimization (OCO) is a well-established learning framework which has both theoretical and practical appeals (Shalev-Shwartz et al., 2012).

Original: It is performed in a sequence of consecutive rounds:
Revised: It is performed in a sequence of consecutive rounds:

Original: In each round t , ﬁrstly a learner chooses a decision x t from a convex set D ⊆ R d , at the same time, an adversary reveals a loss function f t ( · ) : D (cid:55)→ R , and consequently the learner suffers a loss f t ( x t ) .
Revised: In each round t , ﬁrstly a learner chooses a decision x t from a convex set D ⊆ R d , at the same time, an adversary reveals a loss function f t ( · ) : D (cid:55)→ R , and consequently the learner suffers a loss f t ( x t ) .

Original: The goal is to minimize regret, deﬁned as the difference between the cumulative loss of the learner and that of the best decision in hindsight (Hazan et al., 2016):
Revised: The goal is to minimize regret, deﬁned as the difference between the cumulative loss of the learner and that of the best decision in hindsight (Hazan et al., 2016):

Original: R ( T ) :=
Revised: R ( T ) :=

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: f t ( x t ) − min x ∈D
Revised: f t ( x t ) − min x ∈D

Original: T (cid:88) t =
Revised: T (cid:88) t =

Original: f t ( x ) .
Revised: f t ( x ) .

Original: The most classic algorithm for OCO is Online Gradient Descent (OGD)
Revised: The most classic algorithm for OCO is Online Gradient Descent (OGD)

Original: (Zinkevich, 2003), which attains an O ( √ T ) regret.
Revised: (Zinkevich, 2003), which attains an O ( √ T ) regret.

Original: OGD iteratively performs descent step towards gradient direction with a predetermined step size, which is oblivious to the characteristics of the data being observed.
Revised: OGD iteratively performs descent step towards gradient direction with a predetermined step size, which is oblivious to the characteristics of the data being observed.

Original: As a result, its regret bound is data-independent , and can not beneﬁt from the structure of data.
Revised: As a result, its regret bound is data-independent , and can not beneﬁt from the structure of data.

Original: To address this limitation, various of adaptive gradient methods, such as Adagrad (Duchi et al., 2011), RMSprop (Tieleman & Hinton, 2012) and Adadelta (Zeiler, 2012) have been proposed to exploit the geometry of historical data.
Revised: To address this limitation, various of adaptive gradient methods, such as Adagrad (Duchi et al., 2011), RMSprop (Tieleman & Hinton, 2012) and Adadelta (Zeiler, 2012) have been proposed to exploit the geometry of historical data.

Original: Among them, Adam (Kingma & Ba, 2015), which dynamically adjusts the step size and the update direction by exponential average of the past gradients, has been extensively popular and successfully applied to many applications (Xu et al., 2015; Gregor et al., 2015; Kiros et al., 2015; Denkowski & Neubig, 2017; Bahar et al., 2017).
Revised: Among them, Adam (Kingma & Ba, 2015), which dynamically adjusts the step size and the update direction by exponential average of the past gradients, has been extensively popular and successfully applied to many applications (Xu et al., 2015; Gregor et al., 2015; Kiros et al., 2015; Denkowski & Neubig, 2017; Bahar et al., 2017).

Original: Despite the outstanding performance, Reddi et al.
Revised: Despite the outstanding performance, Reddi et al.

Original: (2018) pointed out that Adam suffers the non-convergence issue, and developed two modiﬁed versions, namely AMSgrad and AdamNC.
Revised: (2018) pointed out that Adam suffers the non-convergence issue, and developed two modiﬁed versions, namely AMSgrad and AdamNC.

Original: These variants are equipped with data-dependent regret bounds, which are O ( √ T ) in the worst case and become tighter when gradients are sparse 1 .
Revised: These variants are equipped with data-dependent regret bounds, which are O ( √ T ) in the worst case and become tighter when gradients are sparse 1 .

Original: While the theoretical behavior of Adam in convex case becomes clear, it remains an open problem whether strong convexity can be exploited to achieve better performance.
Revised: While the theoretical behavior of Adam in convex cases becomes clear, it remains an open problem whether strong convexity can be exploited to achieve better performance.

Original: Such property arises, for instance, in support vector machines as well as other regularized learning problems, and it is well-known that the vanilla OGD with  appropriately chosen step size enjoys a much better O (log T ) regret bound for strongly convex functions (Hazan et al., 2007).
Revised: Such property arises, for instance, in support vector machines as well as other regularized learning problems, and it is wellknown that the vanilla OGD with an appropriately chosen step size enjoys a much better O (log T ) regret bound for strongly convex functions (Hazan et al., 2007).

Original: In this paper, we propose a variant of Adam adapted to strongly convex functions, referred to as SAdam .
Revised: In this paper, we propose a variant of Adam adapted to strongly convex functions, referred to as SAdam .

Original: Our algorithm follows the general framework of Adam, yet keeping a faster decaying step size controlled by time-variant heperparameters to exploit strong convexity.
Revised: Our algorithm follows the general framework of Adam, yet keeping a faster decaying step size controlled by time-variant heperparameters to exploit strong convexity.

Original: Theoretical analysis demonstrates that SAdam achieves a data-dependent O (log T ) regret bound for strongly convex functions, which means that it converges faster than AMSgrad and AdamNC in such cases, and also enjoys a huge gain in the face of sparse gradients.
Revised: Theoretical analysis demonstrates that SAdam achieves a data-dependent O (log T ) regret bound for strongly convex functions, which means that it converges faster than AMSgrad and AdamNC in such cases, and also enjoys a huge gain in the face of sparse gradients.

Original: Furthermore, under a special conﬁguration of heperparameters, the proposed algorithm reduces to the SC-RMSprop (Mukkamala & Hein, 2017), which is a variant of RMSprop algorithm for strongly convex functions.
Revised: Furthermore, under a special conﬁguration of heperparameters, the proposed algorithm reduces to the SC-RMSprop (Mukkamala & Hein, 2017), which is a variant of RMSprop algorithm for strongly convex functions.

Original: We provide an alternative proof for SC-RMSprop, and establish the ﬁrst data-dependent logarithmic regret bound.
Revised: We provide an alternative proof for SC-RMSprop, and establish the ﬁrst data-dependent logarithmic regret bound.

Original: Finally, we evaluate the proposed algorithm on strongly convex problems as well as deep networks, and the empirical results demonstrate the effectiveness of our method.
Revised: Finally, we evaluate the proposed algorithm on strongly convex problems as well as deep networks, and the empirical results demonstrate the effectiveness of our method.

Original: Notation.
Revised: Notation.

Original: Throughout the paper, we use lower case bold face letters to denote vectors, lower case letters to denote scalars, and upper case letters to denote matrices.
Revised: Throughout the paper, we use lower case bold face letters to denote vectors, lower case letters to denote scalars, and upper case letters to denote matrices.

Original: We use (cid:107) · (cid:107) to denote the (cid:96) 2 -norm and (cid:107) · (cid:107) ∞ the inﬁnite norm.
Revised: We use (cid:107) · (cid:107) to denote the (cid:96) 2 -norm and (cid:107) · (cid:107) ∞ the inﬁnite norm.

Original: For a positive deﬁnite matrix H ∈ R d × d , the weighted (cid:96) 2 -norm is deﬁned by (cid:107) x (cid:107) 2 H
Revised: For a positive deﬁnite matrix H ∈ R d × d , the weighted (cid:96) 2 -norm is deﬁned by (cid:107) x (cid:107) 2 H

Original: =
Revised: =

Original: x
Revised: x

Original: (cid:62) H x .
Revised: (cid:62) H x .

Original: The H -weighted projection Π H D ( x ) of x onto D is deﬁned by Π H D ( x ) =
Revised: The H -weighted projection Π H D ( x ) of x onto D is deﬁned by Π H D ( x ) =

Original: arg min y ∈D (cid:107)
Revised: arg min y ∈D (cid:107)

Original: y − x (cid:107) 2 H .
Revised: y − x (cid:107) 2 H .

Original: We use g t to denote the gradient of f t ( · ) at x t .
Revised: We use g t to denote the gradient of f t ( · ) at x t .

Original: For vector sequence { v t } Tt =1 , we denote the i -th element of v t by v t,i .
Revised: For vector sequence { v t } Tt =1 , we denote the i -th element of v t by v t,i .

Original: For diagonal matrix sequence { M t } Tt =1 , we use m t,i to denote the i -th element in the diagonal of M t .
Revised: For diagonal matrix sequence { M t } Tt =1 , we use m t,i to denote the i -th element in the diagonal of M t .

Original: We use g 1: t,i =
Revised: We use g 1: t,i =

Original: [ g 1 ,i , · · · , g t,i ] to denote the vector obtained by concatenating the i -th element of the gradient sequence { g t } Tt =1 .
Revised: [ g 1 ,i , · · · , g t,i ] to denote the vector obtained by concatenating the i -th element of the gradient sequence { g t } Tt =1 .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In this section, we brieﬂy review related work in online convex and strongly convex optimization.
Revised: In this section, we brieﬂy review related work in online convex and strongly convex optimization.

Original: In the literature, most studies are devoted to the minimization of regret for convex functions.
Revised: In the literature, most studies are devoted to the minimization of regret for convex functions.

Original: Under the assumptions that the inﬁnite norm of gradients and the diameter of the decision set are bounded, OGD with step size on the order of O (1 / √ t ) (referred to as convex OGD) achieves a data-independent
Revised: Under the assumptions that the inﬁnite norm of gradients and the diameter of the decision set are bounded, OGD with step size on the order of O (1 / √ t ) (referred to as convex OGD) achieves a data-independent

Original: O ( d √ T ) regret (Zinkevich, 2003), where d is the dimension.
Revised: O ( d √ T ) regret (Zinkevich, 2003), where d is the dimension.

Original: To conduct more informative updates,
Revised: To conduct more informative updates,

Original: Duchi et al.
Revised: Duchi et al.

Original: (2011) introduce Adagrad algorithm, which adjusts the step size of OGD in a perdimension basis according to the geometry of the past gradients.
Revised: (2011) introduce Adagrad algorithm, which adjusts the step size of OGD in a perdimension basis according to the geometry of the past gradients.

Original: In particular, the diagonal version of the algorithm updates decisions as
Revised: In particular, the diagonal version of the algorithm updates decisions as

Original: [Equation]
Revised: [Equation]

Original: where α > 0 is a constant factor, V t is a d × d diagonal matrix, and ∀ i ∈
Revised: where α > 0 is a constant factor, V t is a d × d diagonal matrix, and ∀ i ∈

Original: [ d ] , v t,i =
Revised: [ d ] , v t,i =

Original: ( (cid:80) tj =1 g 2 j,i ) /t is the arithmetic average of the square of the i -th elements of the past gradients.
Revised: ( (cid:80) tj =1 g 2 j,i ) /t is the arithmetic average of the square of the i -th elements of the past gradients.

Original: Intuitively, while the step size of Adagrad, i.e., ( α/ √ t ) V − 1 / 2 t , decreases generally on the order of O (1 / √ t ) as that in convex OGD, the additional matrix V − 1 / 2 t will automatically increase step sizes for sparse dimensions in order to seize the infrequent yet valuable information therein.
Revised: Intuitively, while the step size of Adagrad, i.e., ( α/ √ t ) V − 1 / 2 t , decreases generally on the order of O (1 / √ t ) as that in convex OGD, the additional matrix V − 1 / 2 t will automatically increase step sizes for sparse dimensions in order to seize the infrequent yet valuable information therein.

Original: For convex functions, Adagrad enjoys an O ( (cid:80) di =1 (cid:107) g 1: T,i (cid:107) ) regret, which is O ( d √ T ) in the worst case and becomes tighter when gradients are sparse.
Revised: For convex functions, Adagrad enjoys an O ( (cid:80) di =1 (cid:107) g 1: T,i (cid:107) ) regret, which is O ( d √ T ) in the worst case and becomes tighter when gradients are sparse.

Original: Although Adagrad works well in sparse cases, its performance has been found to deteriorate when gradients are dense due to the rapid decay of the step size since it uses all the past gradients in the update (Zeiler, 2012).
Revised: Although Adagrad works well in sparse cases, its performance has been found to deteriorate when gradients are dense due to the rapid decay of the step size since it uses all the past gradients in the update (Zeiler, 2012).

Original: To tackle this issue, Tieleman & Hinton (2012) propose RMSprop, which alters the arithmetic average procedure with Exponential Moving Average (EMA), i.e.,
Revised: To tackle this issue, Tieleman & Hinton (2012) propose RMSprop, which alters the arithmetic average procedure with Exponential Moving Average (EMA), i.e.,

Original: [Equation]
Revised: [Equation]

Original: where β ∈ [0 , 1] is a hyperparameter, and diag( · ) denotes extracting the diagonal matrix.
Revised: where β ∈ [0 , 1] is a hyperparameter, and diag( · ) denotes extracting the diagonal matrix.

Original: In this way, the weights assigned to past gradients decay exponentially so that the reliance of the update is essentially limited to recent few gradients.
Revised: In this way, the weights assigned to past gradients decay exponentially so that the reliance of the update is essentially limited to recent few gradients.

Original: Since the invention of RMSprop, many EMA variants of
Revised: Since the invention of RMSprop, many EMA variants of

Original: Adagrad have been developed (Zeiler, 2012; Kingma & Ba, 2015; Dozat, 2016).
Revised: Adagrad have been developed (Zeiler, 2012; Kingma & Ba, 2015; Dozat, 2016).

Original: One of the most popular algorithms is Adam (Kingma & Ba, 2015), where the ﬁrst-order momentum acceleration, shown in (2), is incorporated into RMSprop to boost the performance:
Revised: One of the most popular algorithms is Adam (Kingma & Ba, 2015), where the ﬁrst-order momentum acceleration, shown in (2), is incorporated into RMSprop to boost the performance:

Original: [Equation]
Revised: [Equation]

Original: While it has been successfully applied to various practical applications, a recent study by Reddi et al.
Revised: While it has been successfully applied to various practical applications, a recent study by Reddi et al.

Original: (2018) shows that Adam could fail to converge to the optimal decision even in some simple one-dimensional convex scenarios due to the potential rapid ﬂuctuation of the step size.
Revised: (2018) shows that Adam could fail to converge to the optimal decision even in some simple one-dimensional convex scenarios due to the potential rapid ﬂuctuation of the step size.

Original: To resolve this issue, they design two modiﬁed versions of Adam.
Revised: To resolve this issue, they design two modiﬁed versions of Adam.

Original: The ﬁrst one is AMSgrad,
Revised: The ﬁrst one is AMSgrad,

Original: [Equation]
Revised: [Equation]

Original: where an additional element-wise maximization procedure is employed before the update of x t to ensure a stable step size.
Revised: where an additional element-wise maximization procedure is employed before the update of x t to ensure a stable step size.

Original: The other is AdamNC, where the framework of Adam remains unchanged, yet a time-variant β 2 (i.e., β 2 t ) is adopted to keep the step size under control.
Revised: The other is AdamNC, where the framework of Adam remains unchanged, yet a time-variant β 2 (i.e., β 2 t ) is adopted to keep the step size under control.

Original: Theoretically, the two algorithms achieve data-dependent O ( √ T (cid:80) di =1 v T,i + (cid:80) di =1 (cid:107) g 1: T,i (cid:107) log T ) and O ( √ T (cid:80) di
Revised: Theoretically, the two algorithms achieve data-dependent O ( √ T (cid:80) di =1 v T,i + (cid:80) di =1 (cid:107) g 1: T,i (cid:107) log T ) and O ( √ T (cid:80) di

Original: =1 v T,i + (cid:80) di =1 (cid:107) g 1: T,i (cid:107) ) regrets respectively.
Revised: =1 v T,i + (cid:80) di =1 (cid:107) g 1: T,i (cid:107) ) regrets respectively.

Original: In the worst case, they suffer O ( d √ T log T ) and O ( d √ T ) regrets respectively, and enjoy a huge gain when data is sparse.
Revised: In the worst case, they suffer O ( d √ T log T ) and O ( d √ T ) regrets respectively, and enjoy a huge gain when data is sparse.

Original: Note that the aforementioned algorithms are mainly analysed in general convex settings and suffer at least O ( d √ T ) regret in the worst case.
Revised: Note that the aforementioned algorithms are mainly analysed in general convex settings and suffer at least O ( d √ T ) regret in the worst case.

Original: For online strongly convex optimization, the classical
Revised: For online strongly convex optimization, the classical

Original: OGD with step size proportional to O (1 /t ) (referred to as strongly convex OGD) achieves a dataindependent O ( d log T ) regret (Hazan et al., 2007).
Revised: OGD with step size proportional to O (1 /t ) (referred to as strongly convex OGD) achieves a dataindependent O ( d log T ) regret (Hazan et al., 2007).

Original: Inspired by this, Mukkamala & Hein (2017) modify the update rule of Adagrad in (1) as follows
Revised: Inspired by this, Mukkamala & Hein (2017) modify the update rule of Adagrad in (1) as follows

Original: [Equation]
Revised: [Equation]

Original: so that the step size decays approximately on the order of O (1 /t ) , which is similar to that in strongly convex OGD.
Revised: so that the step size decays approximately on the order of O (1 /t ) , which is similar to that in strongly convex OGD.

Original: The new algorithm, named SC-Adagrad, is proved to enjoy a data-dependent regret bound of O ( (cid:80) di =1 log( (cid:107) g 1: T,i (cid:107) 2 )) , which is O ( d log T ) in the worst case.
Revised: The new algorithm, named SC-Adagrad, is proved to enjoy a data-dependent regret bound of O ( (cid:80) di =1 log( (cid:107) g 1: T,i (cid:107) 2 )) , which is O ( d log T ) in the worst case.

Original: They further extend this idea to RMSprop, and propose an algorithm named SC-RMSprop.
Revised: They further extend this idea to RMSprop, and propose an algorithm named SC-RMSprop.

Original: However, as pointed out in Section 3, their regret bound for SC-RMSprop is in fact data-independent , and our paper  provide the ﬁrst data-dependent regret bound for this algorithm.
Revised: However, as pointed out in Section 3, their regret bound for SC-RMSprop is in fact data-independent , and in this paper we provide the ﬁrst data-dependent regret bound for this algorithm.

Original: Very recently, several modiﬁcations of Adam adapted to non-convex settings have been developed (Chen et al., 2019;
Revised: Very recently, several modiﬁcations of Adam adapted to non-convex settings have been developed (Chen et al., 2019;

Original: Basu et al., 2018; Zhang et al., 2018; Shazeer & Stern, 2018).
Revised: Basu et al., 2018; Zhang et al., 2018; Shazeer & Stern, 2018).

Original: However, to our knowledge, none of these algorithms are particularly designed for strongly convex functions, nor enjoys a logarithmic regret bound.
Revised: However, to our knowledge, none of these algorithms are particularly designed for strongly convex functions, nor enjoy a logarithmic regret bound.

Original:  SA DAM
Revised:  SA DAM

Original: In this section, we ﬁrst describe the proposed algorithm, then state its theoretical guarantees, and ﬁnally compare it with  SC-RMSprop algorithm.
Revised: In this section, we ﬁrst describe the proposed algorithm, then state its theoretical guarantees, and ﬁnally compare it with the SC-RMSprop algorithm.

Original: 3. T HE A LGORITHM
Revised: 3. T HE A LGORITHM

Original: Before proceeding to our algorithm, following previous studies, we introduce some standard deﬁnitions (Boyd & Vandenberghe, 2004) and assumptions (Reddi et al., 2018).
Revised: Before proceeding to our algorithm, following previous studies, we introduce some standard deﬁnitions (Boyd & Vandenberghe, 2004) and assumptions (Reddi et al., 2018).

Original: Deﬁnition 1.
Revised: Deﬁnition 1.

Original: A function f ( · ) : D (cid:55)→ R is λ -strongly convex if ∀ x 1 , x [Equation] D ,
Revised: A function f ( · ) : D (cid:55)→ R is λ -strongly convex if ∀ x 1 , x 2 ∈ D ,

Original: [Equation]
Revised: [Equation]

Original: Assumption 1.
Revised: Assumption 1.

Original: The inﬁnite norm of the gradients of all loss functions are bounded by G ∞ , i.e., their exists a constant G ∞ > 0 such that max x ∈D (cid:107)∇ f t ( x ) (cid:107) ∞
Revised: The inﬁnite norm of the gradients of all loss functions are bounded by G ∞ , i.e., their exists a constant G ∞ > 0 such that max x ∈D (cid:107)∇ f t ( x ) (cid:107) ∞

Original: ≤ G ∞ holds for all t ∈ [ T ] .
Revised: ≤ G ∞ holds for all t ∈ [ T ] .

Original: Assumption 2.
Revised: Assumption 2.

Original: The decision set D is bounded.
Revised: The decision set D is bounded.

Original: Speciﬁcally, their exists a constant D ∞ > 0 such that max x 1 , x 2 ∈D (cid:107) x 1 − x 2 (cid:107) ∞
Revised: Speciﬁcally, their exists a constant D ∞ > 0 such that max x 1 , x 2 ∈D (cid:107) x 1 − x 2 (cid:107) ∞

Original: ≤ D ∞ .
Revised: ≤ D ∞ .

Original: We are now ready to present our algorithm, which follows the general framework of Adam and is summarized in Algorithm 1.
Revised: We are now ready to present our algorithm, which follows the general framework of Adam and is summarized in Algorithm 1.

Original: In each round t , we ﬁrstly observe the gradient at x t (Step 4), then compute the ﬁrst-order momentum ˆ g t (Step 5).
Revised: In each round t , we ﬁrstly observe the gradient at x t (Step 4), then compute the ﬁrst-order momentum ˆ g t (Step 5).

Original: Here β 1 t a time-variant hyperparameter.
Revised: Here β 1 t a time-variant hyperparameter.

Original: Next, we calculate the second-order momentum V t by EMA of the square of past gradients (Step 6).
Revised: Next, we calculate the second-order momentum V t by EMA of the square of past gradients (Step 6).

Original: This procedure is controlled by β 2 t , whose value will be discussed later.
Revised: This procedure is controlled by β 2 t , whose value will be discussed later.

Original: After that, we add a vanishing factor δt to the diagonal of V t and get ˆ V t (Step 7), which is a standard technique for avoiding too large steps caused by small gradients in the beginning iterations.
Revised: After that, we add a vanishing factor δt to the diagonal of V t and get ˆ V t (Step 7), which is a standard technique for avoiding too large steps caused by small gradients in the beginning iterations.

Original: Finally, we update the decision by ˆ g t and ˆ V t , which is then projected onto the decision set (Step 8).
Revised: Finally, we update the decision by ˆ g t and ˆ V t , which is then projected onto the decision set (Step 8).

Original: While SAdam is inspired by Adam, there exist two key differences: One is the update rule of x t in Step 8,  the other is the conﬁguration of β 2 t in Step 6.
Revised: While SAdam is inspired by Adam, there exist two key differences: One is the update rule of x t in Step 8, and the other is the conﬁguration of β 2 t in Step 6.

Original: Intuitively, both modiﬁcations stem from strongly convex OGD, and jointly result in a faster decaying yet under controlled step size which helps utilize the strong convexity while preserving the practical beneﬁts of Adam.
Revised: Intuitively, both modiﬁcations stem from strongly convex OGD, and jointly result in a faster decaying yet under controlled step size which helps utilize the strong convexity while preserving the practical beneﬁts of Adam.

Original: Speciﬁcally, in the ﬁrst modiﬁcation, we remove the square root operation in (4) of Adam, and update x t at Step 8 as follows
Revised: Speciﬁcally, in the ﬁrst modiﬁcation, we remove the square root operation in (4) of Adam, and update x t at Step 8 as follows

Original: [Equation]
Revised: [Equation]

Original: In this way, the step size used to update the i -th element of x t is αt ˆ v − 1 t,i , which decays in general on the order of O (1 /t ) , and can still be automatically tuned in a per-feature basis via the EMA of the historical gradients.
Revised: In this way, the step size used to update the i -th element of x t is αt ˆ v − 1 t,i , which decays in general on the order of O (1 /t ) , and can still be automatically tuned in a per-feature basis via the EMA of the historical gradients.

Original: The second modiﬁcation is made to β 2 t , which determines the value of V t and thus also controls the decaying rate of the step size.
Revised: The second modiﬁcation is made to β 2 t , which determines the value of V t and thus also controls the decaying rate of the step size.

Original: To help understand the motivation behind our algorithm, we ﬁrst revisit
Revised: To help understand the motivation behind our algorithm, we ﬁrst revisit

Original: Adam, where β 2 t is simply set to be constant, which, however, could cause rapid ﬂuctuation of the step size, and further leads to the non-convergence issue.
Revised: Adam, where β 2 t is simply set to be constant, which, however, could cause rapid ﬂuctuation of the step size, and further leads to the non-convergence issue.

Original: To ensure convergence, Reddi et al.
Revised: To ensure convergence, Reddi et al.

Original: (2018) propose that β 2 t should satisfy the following two conditions:
Revised: (2018) propose that β 2 t should satisfy the following two conditions:

Original: Condition 1.
Revised: Condition 1.

Original: ∀ t ∈ [ T ] and i ∈ [ d ] , 
Revised: ∀ t ∈ [ T ] and i ∈ [ d ] , √

Original: [Equation]
Revised: [Equation]

Original: Condition 2.
Revised: Condition 2.

Original: For some ζ > 0 and ∀ t ∈ [ T ] , i ∈ [ d ] ,
Revised: For some ζ > 0 and ∀ t ∈ [ T ] , i ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst condition implies that the difference between the inverses of step sizes in two consecutive rounds is positive.
Revised: The ﬁrst condition implies that the difference between the inverses of step sizes in two consecutive rounds is positive.

Original: It is inherently motivated by convex OGD (i.e., OGD with step size α √ t , where α > 0 is a constant factor), in which 
Revised: It is inherently motivated by convex OGD (i.e., OGD with step size α √ t , where α > 0 is a constant factor), in which √

Original: [Equation]
Revised: [Equation]

Original: is a key condition used in the analysis.
Revised: is a key condition used in the analysis.

Original: We ﬁrst modify Condition 1 by mimicking the behavior of strongly convex OGD as we are devoted to minimizing regret for strongly convex functions.
Revised: We ﬁrst modify Condition 1 by mimicking the behavior of strongly convex OGD as we are devoted to minimizing regret for strongly convex functions.

Original: In strongly convex OGD (Hazan et al., 2007), the step size at each round t is set as αt with α ≥ 1 λ for λ -strongly convex functions.
Revised: In strongly convex OGD (Hazan et al., 2007), the step size at each round t is set as αt with α ≥ 1 λ for λ -strongly convex functions.

Original: Under this conﬁguration, we have
Revised: Under this conﬁguration, we have

Original: [Equation]
Revised: [Equation]

Original: Motivated by this, we propose the following condition for our SAdam, which is an analog to (7).
Revised: Motivated by this, we propose the following condition for our SAdam, which is an analog to (7).

Original: Condition 3.
Revised: Condition 3.

Original: Their exists a constant C > 0 such that for any α ≥ Cλ , we have ∀ t ∈
Revised: Their exists a constant C > 0 such that for any α ≥ Cλ , we have ∀ t ∈

Original: [ T ] and i ∈ [ d ] ,
Revised: [ T ] and i ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: Note that the extra (1 − β 1 ) in the righthand side of (8) is necessary because SAdam involves the ﬁrst-order momentum in its update.
Revised: Note that the extra (1 − β 1 ) in the righthand side of (8) is necessary because SAdam involves the ﬁrst-order momentum in its update.

Original: Finally, since the step size of SAdam scales with 1 /t rather than 1 / √ t in Adam, we modify Condition 2 accordingly as follows:
Revised: Finally, since the step size of SAdam scales with 1 /t rather than 1 / √ t in Adam, we modify Condition 2 accordingly as follows:

Original: Condition 4.
Revised: Condition 4.

Original: For some ζ > 0 , ∀ t ∈ [ T ] and i ∈ [ d ] ,
Revised: For some ζ > 0 , ∀ t ∈ [ T ] and i ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: 3. T HEORETICAL G UARANTEES
Revised: 3. T HEORETICAL G UARANTEES

Original: In the following, we give a general regret bound when the two conditions are satisﬁed.
Revised: In the following, we give a general regret bound when the two conditions are satisﬁed.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are λstrongly convex.
Revised: , f T ( · ) are λstrongly convex.

Original: Let δ > 0 , β 1 t = β 1 ν t − 1 , where β 1 ∈ [0 , 1) , ν ∈ [0 , 1) , and { β 2 t } Tt =1 ∈
Revised: Let δ > 0 , β 1 t = β 1 ν t − 1 , where β 1 ∈ [0 , 1) , ν ∈ [0 , 1) , and { β 2 t } Tt =1 ∈

Original: [0 , 1] T be a parameter sequence such that Conditions 3 and 4 are satisﬁed.
Revised: [0 , 1] T be a parameter sequence such that Conditions 3 and 4 are satisﬁed.

Original: Let α ≥ Cλ .
Revised: Let α ≥ Cλ .

Original: The regret of SAdam satisﬁes
Revised: The regret of SAdam satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: The above theorem implies that our algorithm enjoys an O ( (cid:80) di =1 log( (cid:107) g 1: T,i (cid:107) 2 ))
Revised: The above theorem implies that our algorithm enjoys an O ( (cid:80) di =1 log( (cid:107) g 1: T,i (cid:107) 2 ))

Original: regret bound, which is O ( d log T ) in the worst case, and automatically becomes tighter whenever the gradients are small or sparse such that (cid:107) g 1: T,i (cid:107) 

Revised: regret bound, which is O ( d log T ) in the worst case, and automatically becomes tighter whenever the gradients are small or sparse such that (cid:107) g 1: T,i (cid:107)

Original: (cid:28) G 2 ∞ T for some
Revised: (cid:28) G 2 ∞ T for some

Original: i ∈
Revised: i ∈

Original: [ d ] .
Revised: [ d ] .

Original: The superiority of data-dependent bounds have been witnessed by a long list of literature, such as Duchi et al.
Revised: The superiority of data-dependent bounds have been witnessed by a long list of literature, such as Duchi et al.

Original: Mukkamala & Hein (2017); Reddi et al.
Revised: Mukkamala & Hein (2017); Reddi et al.

Original: In the following, we give some concrete examples:
Revised: In the following, we give some concrete examples:

Original: • Consider a one-dimensional sparse setting where non-zero gradient appears with probability c/T and c > 1 is a constant.
Revised: • Consider a one-dimensional sparse setting where non-zero gradient appears with probability c/T and c > 1 is a constant.

Original: Then E (cid:104) log (cid:16)(cid:80)
Revised: Then E (cid:104) log (cid:16)(cid:80)

Original: Tt =1 g 2 t, 1 (cid:17)(cid:105) =
Revised: Tt =1 g 2 t, 1 (cid:17)(cid:105) =

Original: O (log( c )) , which is a constant factor.
Revised: O (log( c )) , which is a constant factor.

Original: • Consider a high-dimensional sparse setting where in each dimension of gradient non-zero element appears with probability p =
Revised: • Consider a high-dimensional sparse setting where in each dimension of gradient non-zero element appears with probability p =

Original: T ( m − d ) /d with m ∈ [1 , d ) being a constant.
Revised: T ( m − d ) /d with m ∈ [1 , d ) being a constant.

Original: Then, E (cid:104)(cid:80) di =1 log (cid:16)(cid:80)
Revised: Then, E (cid:104)(cid:80) di =1 log (cid:16)(cid:80)

Original: Tj =1 g 2 j,i (cid:17)(cid:105) =
Revised: Tj =1 g 2 j,i (cid:17)(cid:105) =

Original: O ( m log T ) , which is much tighter than O ( d log T ) .
Revised: O ( m log T ) , which is much tighter than O ( d log T ) .

Original: Remark 2.
Revised: Remark 2.

Original: In practice, ﬁrst-order momentum is a powerful technique that can signiﬁcantly boost the performance (Reddi et al., 2018), and our paper is the ﬁrst to show that algorithms equipped with such technique can achieve logarithmic regret bound for strongly convex functions.
Revised: In practice, ﬁrst-order momentum is a powerful technique that can signiﬁcantly boost the performance (Reddi et al., 2018), and our paper is the ﬁrst to show that algorithms equipped with such technique can achieve logarithmic regret bound for strongly convex functions.

Original: However, since the regret bound of SAdam is data-dependent, it is difﬁcult to rigorously analyse the inﬂuence of the ﬁrst-order momentum parameter β 1 as it affects all the gradients appearing in the last term of the regret of Theorem 1.
Revised: However, since the regret bound of SAdam is data-dependent, it is difﬁcult to rigorously analyse the inﬂuence of the ﬁrst-order momentum parameter β 1 as it affects all the gradients appearing in the last term of the regret of Theorem 1.

Original: We will further investigate this in the feature work.
Revised: We will further investigate this in the feature work.

Original: We note that the regret bounds of adaptive algorithms with ﬁrst-order momentum (e.g., Reddi et al., 2018; Chen et al., 2019) all share a similar structure as our regret bound with respect to β 1 .
Revised: We note that the regret bounds of adaptive algorithms with ﬁrst-order momentum (e.g., Reddi et al., 2018; Chen et al., 2019) all share a similar structure as our regret bound with respect to β 1 .

Original: Next, we provide an instantiation of { β 2 t } Tt =1 such that Conditions 3 and 4 hold, and derive the following Corollary.
Revised: Next, we provide an instantiation of { β 2 t } Tt =1 such that Conditions 3 and 4 hold, and derive the following Corollary.

Original: Corollary 2.
Revised: Corollary 2.

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are λstrongly convex.
Revised: , f T ( · ) are λstrongly convex.

Original: Let δ > 0 , β 1 t = β 1 ν t − 1 where ν, β 1 ∈ [0 , 1) , and 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] .
Revised: Let δ > 0 , β 1 t = β 1 ν t − 1 where ν, β 1 ∈ [0 , 1) , and 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] .

Original: Then we have:
Revised: Then we have:

Original: For any α ≥ (2 − γ ) G 2 ∞ λ (1 − β 1 ) , ∀ t ∈ [ T ] and i ∈
Revised: For any α ≥ (2 − γ ) G 2 ∞ λ (1 − β 1 ) , ∀ t ∈ [ T ] and i ∈

Original: [ d ] ,
Revised: [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: t,i
Revised: t,i

Original: [Equation]
Revised: [Equation]

Original: For all t ∈ [ T ] and j ∈ [ d ] ,
Revised: For all t ∈ [ T ] and j ∈ [ d ] ,

Original: [Equation]
Revised: [Equation]

Original: Moreover, let α ≥ (2 − γ ) G 2 ∞ λ (1 − β 1 ) , and the regret of SAdam satisﬁes:
Revised: Moreover, let α ≥ (2 − γ ) G 2 ∞ λ (1 − β 1 ) , and the regret of SAdam satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Furthermore, as a special case, by setting β 1 t = 0 and 1 − 1 t ≤ β 2 t ≤ 1 − γt , our algorithm reduces to SC-RMSprop (Mukkamala & Hein, 2017), which is a variant of RMSprop for strongly convex functions.
Revised: Furthermore, as a special case, by setting β 1 t = 0 and 1 − 1 t ≤ β 2 t ≤ 1 − γt , our algorithm reduces to SC-RMSprop (Mukkamala & Hein, 2017), which is a variant of RMSprop for strongly convex functions.

Original: Although Mukkamala & Hein (2017) have provided theoretical guarantees for this algorithm, we note that their regret bound is in fact data-independent .
Revised: Although Mukkamala & Hein (2017) have provided theoretical guarantees for this algorithm, we note that their regret bound is in fact data-independent .

Original: Speciﬁcally, the regret bound provided by Mukkamala & Hein (2017) takes the following form:[Equation]
Revised: Speciﬁcally, the regret bound provided by Mukkamala & Hein (2017) takes the following form:

Original: [Equation]
Revised: [Equation]

Original: Focusing on the denominator of the last term in (11), we have
Revised: Focusing on the denominator of the last term in (11), we have

Original: [Equation]
Revised: [Equation]

Original: ∈

Original: [Equation]

Original: thus 
Revised: thus α

Original: [Equation]
Revised: [Equation]

Original: which implies that their regret bound is of order
Revised: which implies that their regret bound is of order

Original: O ( d log T ) , and thus data-independent.
Revised: O ( d log T ) , and thus data-independent.

Original: In contrast, based on Corollary 2, we present a new regret bound for SC-RMSprop in the following, which is
Revised: In contrast, based on Corollary 2, we present a new regret bound for SC-RMSprop in the following, which is

Original: O ( (cid:80) di =1 log( (cid:107) g 1: T,i (cid:107) 2 )) , and thus data-dependent.
Revised: O ( (cid:80) di =1 log( (cid:107) g 1: T,i (cid:107) 2 )) , and thus data-dependent.

Original: Corollary 3.
Revised: Corollary 3.

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are λstrongly convex.
Revised: , f T ( · ) are λstrongly convex.

Original: Let δ > 0 , β 1 t = 0 , and 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] .
Revised: Let δ > 0 , β 1 t = 0 , and 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] .

Original: Let α ≥ (2 − γ ) G 2 ∞ λ .
Revised: Let α ≥ (2 − γ ) G 2 ∞ λ .

Original: Then SAdam reduces to SC-RMSprop, and its regret satisﬁes
Revised: Then SAdam reduces to SC-RMSprop, and its regret satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Finally, we note that Mukkamala & Hein (2017) also consider a more general version of SC-RMSprop which uses a time-variant non-increasing δ for each dimension i .
Revised: Finally, we note that Mukkamala & Hein (2017) also consider a more general version of SC-RMSprop which uses a time-variant non-increasing δ for each dimension i .

Original: In Appendix D we introduce the δ -variant technique to our SAdam, and provide the corresponding theoretical guarantee.
Revised: In Appendix D we introduce the δ -variant technique to our SAdam, and provide the corresponding theoretical guarantee.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we present empirical results on optimizing strongly convex functions and training deep networks.
Revised: In this section, we present empirical results on optimizing strongly convex functions and training deep networks.

Original: More results can be found Appendix.
Revised: More results can be found Appendix.

Original: Algorithms.
Revised: Algorithms.

Original: In both experiments, we compare the following algorithms:
Revised: In both experiments, we compare the following algorithms:

Original: [Figure]
Revised: [Figure]

Original: • SC-Adagrad (Mukkamala & Hein, 2017), with step size α t = α/t .
Revised: • SC-Adagrad (Mukkamala & Hein, 2017), with step size α t = α/t .

Original: • SC-RMSprop (Mukkamala & Hein, 2017), with step size α t = α/t and β t = 1 − 0 .
Revised: • SC-RMSprop (Mukkamala & Hein, 2017), with step size α t = α/t and β t = 1 − 0 .

Original: 9 t .
Revised: 9 t .

Original: • Adam (Kingma & Ba, 2015) and AMSgrad (Reddi et al., 2018), both with β 1 = 0 .
Revised: • Adam (Kingma & Ba, 2015) and AMSgrad (Reddi et al., 2018), both with β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , α t = α/ √ t for convex problems and time-invariant α t
Revised: 999 , α t = α/ √ t for convex problems and time-invariant α t

Original: = α for non-convex problems.
Revised: = α for non-convex problems.

Original: • AdamNC
Revised: • AdamNC

Original: (Reddi et al., 2018), with β 1 = 0 .
Revised: (Reddi et al., 2018), with β 1 = 0 .

Original: 9 , β 2 t = 1 − 1 /t , and α t = α/ √ t for convex problems and a time-invariant α t = α for non-convex problems.
Revised: 9 , β 2 t = 1 − 1 /t , and α t = α/ √ t for convex problems and a time-invariant α t = α for non-convex problems.

Original: • Online Gradient Descent
Revised: • Online Gradient Descent

Original: (OGD), with step size α t
Revised: (OGD), with step size α t

Original: = α/t for strongly convex problems and a time-invariant α t = α for non-convex problems.
Revised: = α/t for strongly convex problems and a time-invariant α t = α for non-convex problems.

Original: • Our proposed SAdam, with β 1 = 0 .
Revised: • Our proposed SAdam, with β 1 = 0 .

Original: 9 , β 2 t = 1 − 0 .
Revised: 9 , β 2 t = 1 − 0 .

Original: 9 t .
Revised: 9 t .

Original: For Adam, AdamNC and AMSgrad, we choose δ = 10 − 8 according to the recommendations in their papers.
Revised: For Adam, AdamNC and AMSgrad, we choose δ = 10 − 8 according to the recommendations in their papers.

Original: For SC-Adagrad and SC-RMSprop, following Mukkamala & Hein (2017), we choose a time-variant δ t,i = ξ 2 e − ξ 1 tv t,i for each dimension i , with ξ 1 = 0 . 1 , ξ 2 = 1 for convex problems and ξ 1 = 0 .
Revised: For SC-Adagrad and SC-RMSprop, following Mukkamala & Hein (2017), we choose a time-variant δ t,i = ξ 2 e − ξ 1 tv t,i for each dimension i , with ξ 1 = 0 . 1 , ξ 2 = 1 for convex problems and ξ 1 = 0 .

Original: 1 , ξ 2 = 0 .
Revised: 1 , ξ 2 = 0 .

Original: 1 for non-convex problems.
Revised: 1 for non-convex problems.

Original: For our SAdam, since the removing of the square root procedure and very small gradients may cause too large step sizes in the beginning iterations, we use a rather large δ = 10 − 2 to avoid this problem.
Revised: For our SAdam, since the removing of the square root procedure and very small gradients may cause too large step sizes in the beginning iterations, we use a rather large δ = 10 − 2 to avoid this problem.

Original: To conduct a fair comparison, for each algorithm, we choose α from the set { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } and report the best results.
Revised: To conduct a fair comparison, for each algorithm, we choose α from the set { 0 . 1 , 0 . 01 , 0 . 001 , 0 . 0001 } and report the best results.

Original: Datasets.
Revised: Datasets.

Original: In both experiments, we examine the performances of the aforementioned algorithms on three widely used datasets: MNIST (60000 training samples, 10000 test samples), CIFAR10 ( training samples, 10000 test samples), and CIFAR100 (50000 training samples, 10000 test samples).
Revised: In both experiments, we examine the performances of the aforementioned algorithms on three widely used datasets: MNIST (60000 training samples, 10000 test samples), CIFAR10 ( training samples, 10000 test samples), and CIFAR100 (50000 training samples, 10000 test samples).

Original: We refer to LeCun (1998) and Krizhevsky (2009) for more details of the three datasets.
Revised: We refer to LeCun (1998) and Krizhevsky (2009) for more details of the three datasets.

Original: 4. O PTIMIZING S TRONGLY C ONVEX F UNCTIONS
Revised: 4. O PTIMIZING S TRONGLY C ONVEX F UNCTIONS

Original: In the ﬁrst experiment, we consider the problem of mini-batch (cid:96) 2 -regularized softmax regression, which belongs to the online strongly convex optimization framework.
Revised: In the ﬁrst experiment, we consider the problem of mini-batch (cid:96) 2 -regularized softmax regression, which belongs to the online strongly convex optimization framework.

Original: Let K be the number of classes and m be the batch size.
Revised: Let K be the number of classes and m be the batch size.

Original: In each round t , ﬁrstly a mini-batch of training samples { ( x m , y m ) } mi = arrives, where y i ∈
Revised: In each round t , ﬁrstly a mini-batch of training samples { ( x m , y m ) } mi = arrives, where y i ∈

Original: [ K ] , ∀ i ∈ [ m ] .
Revised: [ K ] , ∀ i ∈ [ m ] .

Original: Then, the algorithm predicts parameter vectors { w i , b i } Ki =1 , and suffers a loss which takes the following form:
Revised: Then, the algorithm predicts parameter vectors { w i , b i } Ki =1 , and suffers a loss which takes the following form:

Original: [Equation]
Revised: [Equation]

Original: The value of λ 1 and λ 2 are set to be 10 − 2 for all experiments.
Revised: The value of λ 1 and λ 2 are set to be 10 − 2 for all experiments.

Original: The regret (in log scale) v.s. dataset proportion is shown in Fig.
Revised: The regret (in log scale) v.s. dataset proportion is shown in Fig.

Original: It can be seen that our SAdam outperforms other methods across all the considered datasets.
Revised: It can be seen that our SAdam outperforms other methods across all the considered datasets.

Original: Besides, we observe that data-dependent strongly convex methods such as
Revised: Besides, we observe that data-dependent strongly convex methods such as

Original: SC-Adagrad, SC-RMSprop and SAdam preform better than algorithms for general convex functions such as Adam, AMSgrad and AdamNC.
Revised: SC-Adagrad, SC-RMSprop and SAdam preform better than algorithms for general convex functions such as Adam, AMSgrad and AdamNC.

Original: Finally, OGD has the overall highest regret on all three datasets.
Revised: Finally, OGD has the overall highest regret on all three datasets.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: T RAINING D EEP N ETWORKS
Revised: T RAINING D EEP N ETWORKS

Original: Following Mukkamala & Hein (2017)
Revised: Following Mukkamala & Hein (2017)

Original: [Figure]
Revised: [Figure]

Original: we also conduct experiments on a 4-layer CNN, which consists of two convolutional layers (each with 32 ﬁlters of size 3 × 3), one max-pooling layer (with a 2 ×window and 0.25 dropout), and one fully connected layer (with 128 hidden units and 0.5 dropout).
Revised: we also conduct experiments on a 4-layer CNN, which consists of two convolutional layers (each with 32 ﬁlters of size 3 × 3), one max-pooling layer (with a 2 ×window and 0.25 dropout), and one fully connected layer (with 128 hidden units and 0.5 dropout).

Original: We employ ReLU function as the activation function for convolutional layers and softmax function as the activation function for the fully connected layer.
Revised: We employ ReLU function as the activation function for convolutional layers and softmax function as the activation function for the fully connected layer.

Original: The loss function is the cross-entropy.
Revised: The loss function is the cross-entropy.

Original: The training loss v.s. epoch is shown in Fig.
Revised: The training loss v.s. epoch is shown in Fig.

Original: 2, and the testing accuracy v.s. epoch is presented in Fig.
Revised: 2, and the testing accuracy v.s. epoch is presented in Fig.

Original: As can be seen, our SAdam achieves the lowest training loss on the three data sets.
Revised: As can be seen, our SAdam achieves the lowest training loss on the three data sets.

Original: Moreover, this performance gain also translates into good performance on testing accuracy.
Revised: Moreover, this performance gain also translates into good performance on testing accuracy.

Original: The experimental results show that although our proposed SAdam is designed for strongly convex functions, it could lead to superior practical performance even in some highly non-convex cases such as deep learning tasks.
Revised: The experimental results show that although our proposed SAdam is designed for strongly convex functions, it could lead to superior practical performance even in some highly non-convex cases such as deep learning tasks.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this paper, we provide a variant of Adam adapted to strongly convex functions.
Revised: In this paper, we provide a variant of Adam adapted to strongly convex functions.

Original: The proposed algorithm, namely SAdam, follows the general framework of Adam, while keeping a step size decaying in general on the order of O (1 /t ) and controlled by data-dependent heperparameters to exploit strong convexity.
Revised: The proposed algorithm, namely SAdam, follows the general framework of Adam, while keeping a step size decaying in general on the order of O (1 /t ) and controlled by data-dependent heperparameters to exploit strong convexity.

Original: Theoretical analysis shows that SAdam achieves a data-dependent
Revised: Theoretical analysis shows that SAdam achieves a data-dependent

Original: O ( d log T ) regret bound for strongly convex functions, which means that it converges much faster than Adam, AdamNC, and AMSgrad in such cases, and can enjoy a huge gain in the face of sparse gradients.
Revised: O ( d log T ) regret bound for strongly convex functions, which means that it converges much faster than Adam, AdamNC, and AMSgrad in such cases, and can enjoy a huge gain in the face of sparse gradients.

Original: In addition, we also provide the ﬁrst data-dependent logarithmic regret bound for SCRMSprop.
Revised: In addition, we also provide the ﬁrst data-dependent logarithmic regret bound for SCRMSprop.

Original: Finally, we test the proposed algorithm on optimizing strongly convex functions as well as training deep networks, and the empirical results demonstrate the effectiveness of our method.
Revised: Finally, we test the proposed algorithm on optimizing strongly convex functions as well as training deep networks, and the empirical results demonstrate the effectiveness of our method.

Original: Since SAdam enjoys a data-dependent O ( d log T ) regret for online strongly convex optimization, it can be easily translated into a data-dependent O ( d log T/T ) convergence rate for stochastic strongly convex optimization (SSCO) by using the online-to-batch conversion (Kakade & Tewari, 2009).
Revised: Since SAdam enjoys a data-dependent O ( d log T ) regret for online strongly convex optimization, it can be easily translated into a data-dependent O ( d log T/T ) convergence rate for stochastic strongly convex optimization (SSCO) by using the online-to-batch conversion (Kakade & Tewari, 2009).

Original: However, this rate is not optimal for SSCO, and it is sill an open problem how to achieve a datadependent
Revised: However, this rate is not optimal for SSCO, and it is sill an open problem how to achieve a datadependent

Original: O ( d/T ) convergence rate for SSCO.
Revised: O ( d/T ) convergence rate for SSCO.

Original: Recent development on adaptive gradient method (Chen et al., 2018) has proved that Adagrad combined with  multi-stage scheme (Hazan & Kale, 2014) can achieve this rate, but it is highly non-trivial to extend this technique to SAdam, and we leave it as a future work.
Revised: Recent development on adaptive gradient method (Chen et al., 2018) has proved that Adagrad combined with the multi-stage scheme (Hazan & Kale, 2014) can achieve this rate, but it is highly non-trivial to extend this technique to SAdam, and we leave it as a future work.

Revised:  A CKNOWLEDGEMENT

Revised: This work was partially supported by NSFC (61976112), NSFC-NRF Joint Research Project (61861146001), and the Collaborative Innovation Center of Novel Software Technology and Industrialization.

Original: A P ROOF OF T HEOREM
Revised: A P ROOF OF T HEOREM

Original: From Deﬁnition [Equation] we can upper bound regret as
Revised: From Deﬁnition 1, we can upper bound regret as

Original: [Equation]
Revised: [Equation]

Original: ) where x ∗ := min x ∈D (cid:80) Tt =1 f t ( x ) is the best decision in hindsight.
Revised: ) where x ∗ := min x ∈D (cid:80) Tt =1 f t ( x ) is the best decision in hindsight.

Original: On the other hand, by the update rule of x t +1 in Algorithm
Revised: On the other hand, by the update rule of x t +1 in Algorithm

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: 14) where α t := α/t , and the inequality is due to the following lemma, which implies that the weighed projection procedure is non-expansive.
Revised: 14) where α t := α/t [Equation] and the inequality is due to the following lemma [Equation]

Revised: which implies that the weighted projection procedure is non-expansive.

Original: Lemma 1.
Revised: Lemma 1.

Original: (McMahan & Streeter, 2010) Let A ∈ R d × d be a positive deﬁnite matrix and D
Revised: (McMahan & Streeter, 2010) Let A ∈ R d × d be a positive deﬁnite matrix and D

Original: ⊆ R d be a convex set.
Revised: ⊆ R d be a convex set.

Original: Then we have, ∀ x , y ∈ R d ,
Revised: Then we have, ∀ x , y ∈ R d ,

Original: [Equation]
Revised: [Equation]

Original: Rearranging (14), we have
Revised: Rearranging (14), we have

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the last inequality is due to β 1 t ≤ β
Revised: where the last inequality is due to β 1 t ≤ β

Original: [Equation]
Revised: [Equation]

Original: We proceed to bound the second term of the inequality above.
Revised: We proceed to bound the second term of the inequality above.

Original: When t ≥
Revised: When t ≥

Original: [Equation]
Revised: [Equation]

Original: by Young’s inequality and the deﬁnition of β 1 t
Revised: by Young’s inequality and the deﬁnition of β 1 t

Original: [Equation]
Revised: [Equation]

Original: we get
Revised: we get

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: When t = 1 , this term becomes 0 since ˆ g 0 = 0 in Algorithm 1.
Revised: When t = 1 , this term becomes 0 since ˆ g 0 = 0 in Algorithm 1.

Original: Plugging (16) and (17) into (13)
Revised: Plugging (16) and (17) into (13)

Original: [Equation]
Revised: [Equation]

Original: we get the following inequality, of which we divide the righthand side into three parts and upper bound each of them one by one.
Revised: we get the following inequality, of which we divide the righthand side into three parts and upper bound each of them one by one.

Original: [Equation]
Revised: [Equation]

Original: To bound P 1 , we haveP
Revised: To bound P 1 , we have

Original: [Table]

Original:  =[Equation] [Table] [Equation]
Revised: P 1 =

Original: [Table]
Revised: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where the inequality is derived from β 1( t − 1) ≥ β  t .
Revised: where the inequality is derived from β 1( t − 1) ≥ β 1 t .

Original: For the ﬁrst term in the last equality of (18)
Revised: For the ﬁrst term in the last equality of (18)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where the second equality is because ˆ v t,i = v t,i + δt
Revised: where the second equality is because ˆ v t,i = v t,i + δt

Original: [Equation]
Revised: [Equation]

Original: and the second inequality is due to 1 − β 1 ≤ 1 − β 1 t .
Revised: and the second inequality is due to 1 − β 1 ≤ 1 − β 1 t .

Original: For the second term of (18)
Revised: For the second term of (18)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where ﬁrst inequality is due to Condition 3, and the second inequality follows from Assumption 2.
Revised: where ﬁrst inequality is due to Condition 3, and the second inequality follows from Assumption 2.

Original: Combining (18), (19) and (20), we get
Revised: Combining (18), (19) and (20), we get

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: To bound P 2 , we introduce the following lemma.
Revised: To bound P 2 , we introduce the following lemma.

Original: Lemma 2.
Revised: Lemma 2.

Original: The following inequality holds
Revised: The following inequality holds

Original: [Equation]
Revised: [Equation]

Original: By Lemma 2, we have
Revised: By Lemma 2, we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we turn to upper bound P 3 :
Revised: Finally [Equation] we turn to upper bound P 3 :

Original: [Equation]
Revised: [Equation]

Original: To further bound P (cid:48) 3 ,  we have 
Revised: To further bound P (cid:48) 3 , following Bock et al. (2019), we have [Equation]

Original: [Equation]
Revised: [Equation]

Original: 24) where the inequality follows from ν T ≥ ν T +1 .
Revised: where the inequality follows from ν T ≥ ν T +1 .

Original: Thus,
Revised: Thus,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We complete the proof by combining (21), (23) and (25).
Revised: We complete the proof by combining (21), (23) and (25).

Original: B P ROOF OF C OROLLARY
Revised: B P ROOF OF C OROLLARY

Original: For the ﬁrst condition, we have
Revised: For the ﬁrst condition, we have

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst inequality is derived from the deﬁnition of β 2 t , the second and the third inequalities are due to Assumption 1.
Revised: where the ﬁrst inequality is derived from the deﬁnition of β 2 t , the second and the third inequalities are due to Assumption 1.

Original: Based on (26), for any α ≥ (2 − γ ) G 2 ∞ λ (1 − β 1 ) , we have tv t,i α − ( t − 1) v t − 1 ,i α ≤ λ (1 − β 1 ) holds for all t ∈
Revised: Based on (26), for any α ≥ (2 − γ ) G 2 ∞ λ (1 − β 1 ) , we have tv t,i α − ( t − 1) v t − 1 ,i α ≤ λ (1 − β 1 ) holds for all t ∈

Original: [ T ] and i ∈ [ d ] .
Revised: [ T ] and i ∈ [ d ] .

Original: For the second condition, we have
Revised: For the second condition, we have

Original: [Equation]
Revised: [Equation]

Original: j 
Revised: j =

Original: [Equation]
Revised: [Equation]

Original: t 
Revised: t −

Original: [Equation]
Revised: [Equation]

Original: k = (cid:18) 1 − t − k +(cid:19) γ j g 2 j,i =
Revised: k = (cid:18) 1 − t − k +(cid:19) γ j g 2 j,i =

Original: t t (cid:88)
Revised: t t (cid:88)

Original: j = t − j (cid:89) k
Revised: j = t − j (cid:89) k

Original: = t − k t − k
Revised: = t − k t − k

Original: +γ j g 2 j,i =
Revised: +γ j g 2 j,i =

Original: t t (cid:88) j
Revised: t t (cid:88) j

Original: = j t γ j g 2 j,i =
Revised: = j t γ j g 2 j,i =

Original: γ t (cid:88) j = g 2 j [Equation] i (27) where the inequality follows from β t ≥ 1 − 1 t and 1 − β t ≥ γt .
Revised: γ t (cid:88) j = g 2 j,i (27) where the inequality follows from β t ≥ 1 − 1 t and 1 − β t ≥ γt .

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: We begin with the following lemma that is central to our analysis.
Revised: We begin with the following lemma that is central to our analysis.

Original: Lemma 3.
Revised: Lemma 3.

Original: For all i ∈ [ d ] and t ∈ [ T ] , we have
Revised: For all i ∈ [ d ] and t ∈ [ T ] , we have

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For any a ≥ b > 0 , the inequality 1 + x ≤ e x implies that
Revised: For any a ≥ b > 0 , the inequality 1 + x ≤ e x implies that

Original: [Equation]
Revised: [Equation]

Original: Let m 

Revised: Let m

Original: = ζδ , and m j = (cid:80) jk
Revised: = ζδ , and m j = (cid:80) jk

Original: =1 g 2 k,i + ζδ > 0 .
Revised: =1 g 2 k,i + ζδ > 0 .

Original: By (29), we have
Revised: By (29), we have

Original: [Equation]
Revised: [Equation]

Original: Summing over 1 to t , we have
Revised: Summing over 1 to T , we have

Original: [Equation]
Revised: [Equation]

Original: Now we turn to the proof of Lemma 2.
Revised: Now we turn to the proof of Lemma 2.

Original: First, expending the last term in the summation by the update rule of Algorithm 
Revised: First, expending the last term in the summation by the update rule of Algorithm 1, we get

Original: [Equation]
Revised: [Equation]

Original: we get [Equation]
Revised: The above equality can be further bounded as

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: (31) where the ﬁrst inequality is due to 1 − β 1 j ≤ 1 , the second inequality follows from Cauchy-Schwarz inequality, the third inequality is due to β 1 t ≤ β 1 .
Revised: 31) where the ﬁrst inequality is due to 1 − β 1 j ≤ 1 , the second inequality follows from Cauchy-Schwarz inequality, the third inequality is due to β 1 t ≤ β 1 .

Original: Let r j = g 2 j,i / ( (cid:80) jk
Revised: Let r j = g 2 j,i / ( (cid:80) jk

Original: =1 g 2 k,i + ζδ ) .
Revised: =1 g 2 k,i + ζδ ) .

Original: Using similar arguments for all time steps and summing over 1 to T , we have
Revised: Using similar arguments for all time steps and summing over 1 to T , we have

Original: T (cid:88)

Original: [Equation]

Original: ≤

Original: [Equation]

Original: [Equation]

Original: i =

Original: T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original: t (cid:88) j

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: β

Original: [Equation]
Revised: [Equation]

Original: =αζ ( [Equation] 1 ) d (cid:88) i = [Equation]
Revised: =

Original: (cid:88)

Original: j

Original: [Equation]
Revised: [Equation]

Original: (cid:88)

Original: [Equation]

Original: [Equation]

Original: = αζ (

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =

Original: T (cid:88) j = (cid:80) T − j [Equation] =
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: l

Original: [Equation]
Revised: [Equation]

Original: g
Revised: g

Original: [Equation]

Original: i (cid:80) jk

Original: [Equation]
Revised: [Equation]

Original: 1 g

Original: [Equation]
Revised: [Equation]

Original: k

Original: [Equation]

Original: ≤ αζ (1 − β 1 )d (cid:88) i= T (cid:88) j = g 2 j,i (cid:80) jk =1 g 2 k
Revised: αζ (1 − β 1 )d (cid:88) i

Original: [Equation]

Original: (28) ≤ αζ (1 − β 1 )d (cid:88)

Original: i = log (cid:32)(cid:80)
Revised: = log (cid:32)(cid:80)

Original: Tj =1 g 2 j,i ζδ +(cid:33) .
Revised: Tj =1 g 2 j,i ζδ +(cid:33) .

Original: D SA DAM WITH  DECAYING REGULARIZATION FACTOR
Revised: D SA DAM WITH A DECAYING REGULARIZATION FACTOR

Original:  5: ˆ g t = β 1 t ˆ g t − 1 + (1 − β 1 t ) g t 6:
Revised: ∇ 5: ˆ g t = β 1 t ˆ g t − 1 + (1 − β 1 t ) g t 6:

Original: V t = β 2 t V t − 1 + (1 − β 2 t )diag( g t g (cid:62) t ) 7: ˆ V t =
Revised: V t = β 2 t V t − 1 + (1 − β 2 t )diag( g t g (cid:62) t ) 7: ˆ V t =

Original: V t + diag (cid:0) δ t t (cid:1) 8: x t +1 =
Revised: V t + diag (cid:0) δ t t (cid:1) 8: x t +1 =

Original: Π ˆ V t D (cid:16) x t − αt ˆ V − 1 t ˆ g t (cid:17) 9: end for
Revised: Π ˆ V t D (cid:16) x t − αt ˆ V − 1 t ˆ g t (cid:17) 9: end for

Original: In this section, we establish a generalized version of SAdam, which employs a time-variant regularization factor δ t,i for each dimension i , instead of a ﬁxed one for all
Revised: In this section, we establish a generalized version of SAdam, which employs a time-variant regularization factor δ t,i for each dimension i , instead of a ﬁxed one for all

Original: i ∈
Revised: i ∈

Original: [ d ] and t ∈ [ T ] as in the original
Revised: [ d ] and t ∈ [ T ] as in the original

Original: SAdam.
Revised: SAdam.

Original: The algorithm is referred to as SAdamD and summarized in Algorithm 2.
Revised: The algorithm is referred to as SAdamD and summarized in Algorithm 2.

Original: It can be seen that our SAdamD reduces to SC-RMSprop with time-variant δ t when β 1 t = 0 and 1 − 1 t ≤ β 2 t ≤ 1 − γt .
Revised: It can be seen that our SAdamD reduces to SC-RMSprop with time-variant δ t when β 1 t = 0 and 1 − 1 t ≤ β 2 t ≤ 1 − γt .

Original: For SAdamD, we prove the following theoretical guarantee:
Revised: For SAdamD, we prove the following theoretical guarantee:

Original: Theorem 4.
Revised: Theorem 4.

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are λ -strongly convex.
Revised: , f T ( · ) are λ -strongly convex.

Original: Let { δ t,i } Tt =1 ∈ (0 , 1] T be a non-increasing sequence for all i ∈
Revised: Let { δ t,i } Tt =1 ∈ (0 , 1] T be a non-increasing sequence for all i ∈

Original: [ d ] , β 1 t = β 1 ν t − 1 where β 1 ∈ [0 , 1) , ν ∈ [0 , 1) , and { β 2 t } Tt =1 ∈
Revised: [ d ] , β 1 t = β 1 ν t − 1 where β 1 ∈ [0 , 1) , ν ∈ [0 , 1) , and { β 2 t } Tt =1 ∈

Original: [0 , 1] T be a parameter sequence such that Conditions 3 and 4 are satisﬁed.
Revised: [0 , 1] T be a parameter sequence such that Conditions 3 and 4 are satisﬁed.

Original: Let α ≥ Cλ .
Revised: Let α ≥ Cλ .

Original: The regret of SAdamD satisﬁes
Revised: The regret of SAdamD satisﬁes

Original: [Equation]
Revised: [Equation]

Original: By setting β 1 t = 0 and 1 − 1 t ≤ β 2 t ≤ 1 − γt , we can derive the following regret bound for
Revised: By setting β 1 t = 0 and 1 − 1 t ≤ β 2 t ≤ 1 − γt , we can derive the following regret bound for

Original: SC-RMSprop:
Revised: SC-RMSprop:

Original: Corollary 5.
Revised: Corollary 5.

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are λstrongly convex.
Revised: , f T ( · ) are λstrongly convex.

Original: Let { δ t,i } Tt =1 ∈ (0 , 1] T be a non-increasing sequence for all i ∈
Revised: Let { δ t,i } Tt =1 ∈ (0 , 1] T be a non-increasing sequence for all i ∈

Original: [ d ] , and 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] .
Revised: [ d ] , and 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] .

Original: Let α ≥ (2 − γ ) G 2 ∞ λ .
Revised: Let α ≥ (2 − γ ) G 2 ∞ λ .

Original: Then SAdamD reduces to SC-RMSprop, and the regret satisﬁes
Revised: Then SAdamD reduces to SC-RMSprop, and the regret satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Finally, we provide an instantiation of δ t and derive the following Corollary.
Revised: Finally [Equation] we provide an instantiation of δ t and derive the following Corollary.

Original: Corollary 6.
Revised: Corollary 6.

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are λstrongly convex.
Revised: , f T ( · ) are λstrongly convex.

Original: Let δ t,i =
Revised: Let δ t,i =

Original: [Equation]
Revised: [Equation]

Original: 2 1+ ξ 1 (cid:80) tj =1 g 2 j,i , where ξ 2 ∈ (0 , 1] and ξ 1 ≥ 0 are hypeparameters.
Revised: 1+ ξ 1 (cid:80) tj =1 g 2 j,i , where ξ 2 ∈ (0 , 1] and ξ 1 ≥ 0 are hypeparameters.

Original: Then we have δ t,i ∈ (0 , 1] and is non-increasing ∀
Revised: Then we have δ t,i ∈ (0 , 1] and is non-increasing ∀

Original: i ∈
Revised: i ∈

Original: [ d ] , t ∈ [ T ] .
Revised: [ d ] , t ∈ [ T ] .

Original: Let 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] , and α ≥ (2 − γ ) G 2 ∞ λ .
Revised: Let 1 − 1 t ≤ β 2 t ≤ 1 − γt , where γ ∈ (0 , 1] , and α ≥ (2 − γ ) G 2 ∞ λ .

Original: Then SAdamD reduces to SC-RMSprop, and the regret satisﬁes
Revised: Then SAdamD reduces to SC-RMSprop, and the regret satisﬁes

Original: [Equation]
Revised: [Equation]

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: By similar arguments as in the proof of Theorem 1, we can upper bound regret as
Revised: By similar arguments as in the proof of Theorem [Equation] we can upper bound regret as

Original: R ( T ) ≤
Revised: R ( T ) ≤

Original: [Equation]
Revised: [Equation]

Original: To bound P 1 , based on (18), we have
Revised: To bound P 1 , based on (18), we have

Original: P 1 ≤
Revised: P 1 ≤

Original: T (cid:88) t
Revised: T (cid:88) t

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For the ﬁrst term in (36), we have
Revised: For the ﬁrst term in (36), we have

Original: [Equation]
Revised: [Equation]

Original: For the second term of (36), we have
Revised: For the second term of (36), we have

Original: [Equation]
Revised: [Equation]

Original: where the inequality follows from Condition 3.
Revised: where the inequality follows from Condition 3.

Original: Combining (36), (37) and (38)
Revised: Combining (36), (37) and (38)

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: To bound P
Revised: To bound P

Original: [Equation]
Revised: [Equation]

Original: we ﬁrst introduce the following lemma.
Revised: we ﬁrst introduce the following lemma.

Original: Lemma 4.
Revised: Lemma 4.

Original: The following inequality holds
Revised: The following inequality holds

Original: [Equation]
Revised: [Equation]

Original: The proof of Lemma 4 can be found in Appendix F. Based on Lemma 4, we have
Revised: The proof of Lemma 4 can be found in Appendix F. Based on Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we turn to upper bound P 3 :
Revised: Finally, we turn to upper bound P 3 :

Original: [Equation]
Revised: [Equation]

Original: We ﬁnish the proof by combining (39), (41) and (42).
Revised: We ﬁnish the proof by combining (39), (41) and (42).

Original: F P ROOF OF L EMMA
Revised: F P ROOF OF L EMMA

Original: Expending the last term in the summation by using the update rule of Algorithm 2, we have α T (cid:107) ˆ g T (cid:107) 2ˆ V − 1 T = α T d (cid:88)
Revised: Expending the last term in the summation by using the update rule of Algorithm 2, we have α T (cid:107) ˆ g T (cid:107) 2ˆ V − 1 T = α T d (cid:88)

Original: i
Revised: i

Original: =
Revised: =

Original: ˆ g 2 T, i
Revised: ˆ g 2 T,i v T

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original:  T
Revised: δ T

Original: [Equation]
Revised: [Equation]

Original: i T
Revised: i T

Original: [Equation]
Revised: [Equation]

Original:  d (cid:88) i 
Revised: α d (cid:88) i =

Original: [Equation]
Revised: [Equation]

Original: (cid:16)(cid:80)

Original: Tj
Revised: Tj

Original: [Equation]
Revised: [Equation]

Original: ) g

Original: [Equation]
Revised: [Equation]

Original:  T(cid:80)
Revised: (cid:81) T

Revised: [Equation]

Revised: T

Original: [Equation]
Revised: [Equation]

Original: The above equality can be further bounded as
Revised: The above equality can be further bounded as

Original: [Equation]
Revised: [Equation]

Original: ≤ 
Revised: ≤ αζ

Original: [Equation]
Revised: [Equation]

Original:  − 
Revised: 1 − β

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst inequality follows from Cauchy-Schwarz inequality and 1 − β 1 t ≤ 1 , and the second inequality is due to β 1 t ≤ β 1 .
Revised: The ﬁrst inequality follows from Cauchy-Schwarz inequality and 1 − β 1 t ≤ 1 , and the second inequality is due to β 1 t ≤ β 1 .

Original: Let r j = [Equation] j
Revised: Let r j = g 2 j,

Original: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: k =1 g 2 k,i + ζδ T,i .
Revised: k =1 g 2 k,i + ζδ T,i .

Original: By using similar arguments as in (32), we have
Revised: By using similar arguments as in (32), we have

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: G O N THE CONVERGENCE OF A MSGRAD
Revised: G O N THE CONVERGENCE OF A MSGRAD

Original: In this section, we ﬁrstly provide the AMSgrad algorithm and its theoretical guarantees (Reddi et al., 2018), then state a theoretical ﬂaw in their analysis revealed by Tran et al.
Revised: In this section, we ﬁrstly provide the AMSgrad algorithm and its theoretical guarantees (Reddi et al., 2018), then state a theoretical ﬂaw in their analysis revealed by Tran et al.

Original: (2019), and ﬁnally propose a simple solution to ﬁx this problem.
Revised: (2019), and ﬁnally propose a simple solution to ﬁx this problem.

Original: The AMSgrad algorithm developed in Reddi et al.
Revised: The AMSgrad algorithm developed in Reddi et al.

Original: (2018) is summarized in Algorithm 3.
Revised: (2018) is summarized in Algorithm 3.

Original: ˆ g t = β 1 t ˆ g t − 1 + (1 − β 1 t ) g t
Revised: ˆ g t = β 1 t ˆ g t − 1 + (1 − β 1 t ) g t

Original: V t = β 2 V t − 1 + (1 − β 2 )diag( g t g (cid:62) t )
Revised: V t = β 2 V t − 1 + (1 − β 2 )diag( g t g (cid:62) t )

Original: ˆ V t = max { V t , ˆ V t − 1 }
Revised: ˆ V t = max { V t , ˆ V t − 1 }

Original: 1 = Π √ˆ Vt x t − α t
Revised: = Π √ V − α t

Original: ˆ V −
Revised: V −

Original: For AMSgrad, Reddi et al.
Revised: For AMSgrad, Reddi et al.

Original: (2018) provide the following regret bound.
Revised: (2018) provide the following regret bound.

Original: Theorem 7 (Theorem 4 in Reddi et al.
Revised: Theorem 7 (Theorem 4 in Reddi et al.

Original: (2018), problematic) .
Revised: (2018), problematic) .

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are convex.
Revised: , f T ( · ) are convex.

Original: Let δ > 0 , β 1 > 0 , β 1 t ≤ β 1 , and γ = β 1 √ β 2 ≤ 1 .
Revised: Let δ > 0 , β 1 > 0 , β 1 t ≤ β 1 , and γ = β 1 √ β 2 ≤ 1 .

Original: The regret of AMSgrad satisﬁes
Revised: The regret of AMSgrad satisﬁes

Original: [Equation]
Revised: [Equation]

Revised: d

Revised: [Equation]

Revised: i

Revised: [Equation]

Original: {+(cid:107) g 
+}
Revised: {+(cid:107) g 
+}

Revised: : T

Revised: [Equation]

Revised: i (cid:107) .

Original: Recently, Tran et al.
Revised: Recently, Tran et al.

Original: (2019) point out a mistake in the proof of Theorem 7.
Revised: (2019) point out a mistake in the proof of Theorem 7.

Original: Speciﬁcally, in Reddi et al.
Revised: Speciﬁcally, in Reddi et al.

Original: (2018), the following inequality is utilized (Proof of Lemma 2, Page 18):[Equation]
Revised: (2018), the following inequality is utilized (Proof of Lemma 2, Page 18):

Original: [Equation]
Revised: [Equation]

Original: which, however, may not hold.
Revised: which, however, may not hold.

Original: To see this, we note that essentially (47) uses
Revised: To see this, we note that essentially (47) uses

Original: [Equation]
Revised: [Equation]

Original: which holds only if β 1 t ≤ β 1 and (cid:107) ˆ V1 / 4 t (
Revised: which holds only if β 1 t ≤ β 1 and (cid:107) ˆ V

Original: [Equation]
Revised: [Equation]

Original: is
Revised: t

Original: [Equation]

Original: -negative

Original: [Equation]
Revised: [Equation]

Original: However, as empirically shown by Tran et al.
Revised: However, as empirically shown by Tran et al.

Original: (2019), the letter requirement can be violated in some counterexamples.
Revised: (2019), the letter requirement can be violated in some counterexamples.

Original: Note that similar problems exist in many recent proposed methods, such as Padam (Chen & Gu, 2018) and Adabound (Luo et al., 2019; Savarese, 2019).
Revised: Note that similar problems exist in many recent proposed Adam variants.

Original: To address this issue, Tran et al.
Revised: To address this issue, Tran et al.

Original: (2019) establish a new convergence proof of AMSGrad, which indicates an O ( d √ T ) data-independent regret bound.
Revised: (2019) establish a new convergence proof of AMSGrad, which indicates an O ( d √ T ) data-independent regret bound.

Original: Morover, they  propose a variant of AMSgrad, called AdamX, which alters the stricture of AMSgrad to force the inequality been satisﬁed.
Revised: Moreover, as an alternative, they also propose a variant of AMSgrad, called AdamX, which alters the stricture of AMSgrad to force the inequality been satisﬁed.

Original: For AdamX, they also give a new theoretical analysis and an O ( d √ T ) data-independent regret bound.
Revised: For AdamX, they also give a new theoretical analysis and an O ( d √ T ) data-independent regret bound.

Original: In this paper, we ﬁnd out that the above problem can be solved by simply conﬁguring β 1 t  in a non-increasing manner, i.e., ∀ t ≥ 2 , β 1 t ≤ β 1( t − 1) .
Revised: In this paper, we ﬁnd out that the above problem can be solved by simply conﬁguring β 1 t of AMSgrad in a non-increasing manner, i.e., ∀ t ≥ 2 , β 1 t ≤ β 1( t − 1) .

Original: Speciﬁcally, when β 1 t is non-increasing, we can rewrite (47) asT
Revised: Speciﬁcally, when β 1 t is non-increasing, we can rewrite (47) as

Original: [Equation]
Revised: [Equation]

Original: i

Original: [Equation]

Original: [Equation]

Original: T

Original: [Equation]
Revised: [Equation]

Original: i (cid:107) ≤ 2 α

Original: [Equation]

Original: [Equation]

Original: where the second inequality is derived from β 1 t ≤ β 1( t − 1) , and the last inequality is due to the fact that β 1 t ≤ β 1 and
Revised: where the second inequality is derived from β 1 t ≤ β 1( t − 1) , and the last inequality is due to the fact that β 1 t ≤ β 1 and

Original: [Equation]
Revised: [Equation]

Original: is non-negative.
Revised: is non-negative.

Original: In this way, the proof of
Revised: In this way, the proof of

Original: AMSgrad can proceed, and the algorithm structure as well as the conclusion in Theorem 7 remain unchanged.
Revised: AMSgrad can proceed, and the algorithm structure as well as the conclusion in Theorem 7 remain unchanged.

Original: To summarize, we restate Theorem 7 as follows.
Revised: To summarize, we restate Theorem 7 as follows.

Original: Theorem 8 (Fixed theoretical guarantee of AMSgrad) .
Revised: Theorem 8 (Fixed theoretical guarantee of AMSgrad) .

Original: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .
Revised: Suppose Assumptions 1 and 2 hold, and all loss functions f 1 ( · ) , . . .

Original: , f T ( · ) are convex.
Revised: , f T ( · ) are convex.

Original: Let δ > 0 , β 1 > 0 , β 1 t ≤ β 1( t − 1) where t ≥ 2 , β 11 = β 1 , and γ = β 1 √ β 2 ≤ 1 .
Revised: Let δ > 0 , β 1 > 0 , β 1 t ≤ β 1( t − 1) where t ≥ 2 , β 11 = β 1 , and γ = β 1 √ β 2 ≤ 1 .

Original: The regret of AMSgrad satisﬁes
Revised: The regret of AMSgrad satisﬁes

Original: R ( T ) ≤ D 2 ∞ √ T α (1 − β 1 ) d (cid:88)
Revised: R ( T ) ≤ D 2 ∞ √ T α (1 − β 1 ) d (cid:88)

Original: i
Revised: i

Original: = ˆ v 1 / 2 T,i +
Revised: = ˆ v 1 / 2 T,i +

Original: D 2 ∞ 2(1 − β 1 )
Revised: D 2 ∞ 2(1 − β 1 )

Original: T (cid:88) t = d (cid:88) i =
Revised: T (cid:88) t = d (cid:88) i =

Original: β 1 t ˆ v 1 / 2 t,i α t + α √ 

Revised: β 1 t ˆ v 1 / 2 t,i α t + α √

Original: + log T (1 − β 1 ) 2 (1 − γ ) (cid:112) (1 − β 2 ) d (cid:88) i
Revised: + log T (1 − β 1 ) 2 (1 − γ ) (cid:112) (1 − β 2 ) d (cid:88) i

Original: = (cid:107) g 1: T,i (cid:107) .
Revised: = (cid:107) g 1: T,i (cid:107) .

Original: H E XPERIMENTS ON R ES N ET
Revised: H E XPERIMENTS ON R ES N ET

Original: [Figure]
Revised: [Figure]

Original: In this section, we conduct the image classiﬁcation task on CIFAR10 dataset using ResNet18 (He et al., 2016).
Revised: In this section, we conduct the image classiﬁcation task on CIFAR10 dataset using ResNet18 (He et al., 2016).

Original: The parameter conﬁguration of algorithms is the same as that in Section 4.
Revised: The parameter conﬁguration of algorithms is the same as that in Section 4.

Original: We repeat each experiment 10 times and take their average.
Revised: We repeat each experiment 10 times and take their average.

Original: The training loss v.s. epoch is shown in Fig. 4a, and the testing accuracy v.s. epoch is shown in Fig.
Revised: The training loss v.s. epoch is shown in Fig. 4a, and the testing accuracy v.s. epoch is shown in Fig.

Original: 4b.
Revised: 4b.

Original: As can be seen, our proposed SAdam outperforms other algorithms.
Revised: As can be seen, our proposed SAdam outperforms other algorithms.