Original: A CUTUM : W HEN G ENERALIZATION M EETS A DAPTABILITY
Revised: A CUTUM : W HEN G ENERALIZATION M EETS A DAPTABILITY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In spite of the slow convergence, stochastic gradient descent (SGD) is still the most practical optimization method due to its outstanding generalization ability and simplicity.
Revised: In spite of the slow convergence, stochastic gradient descent (SGD) is still the most widely used optimization method due to its outstanding generalization ability and simplicity.

Original: On the other hand, adaptive methods have attracted much more attention of optimization and machine learning communities, both for the leverage of life-long information and for the deep and fundamental mathematical theory.
Revised: On the other hand, adaptive methods have attracted much more attention of optimization and machine learning communities, both for the leverage of life-long information and for the deep and fundamental mathematical theory.

Original: Taking the best of both worlds is the most exciting and challenging question in the ﬁeld of optimization for machine learning.
Revised: Taking the best of both worlds is the most exciting and challenging question in the ﬁeld of optimization for machine learning.

Original: In this paper, wetake a small step towards such ultimate goal. We revisit existing adaptive  methods from a novel point of view, which reveals a fresh understanding of momentum.
Revised: In this paper, we revisit existing adaptive gradient methods from a novel point of view, which reveals a fresh understanding of momentum.

Original: Our new intuition empowers us to remove the second moment in Adam without the loss of performance.
Revised: Our new intuition empowers us to remove the second moment in Adam without the loss of performance.

Original: Based on our view, we propose a new method, named acu te adaptive momen tum ( Acutum ).
Revised: Based on our view, we propose a novel optimization method, the acute adaptive momentum ( Acutum ).

Original: To the best of our knowledge, Acutum is the ﬁrst adaptive gradient method without second moments.
Revised: To the best of our knowledge, Acutum is the ﬁrst adaptive gradient method without second moments.

Original: Experimentally, we demonstrate that our method has a faster convergence rate than Adam/Amsgrad, and generalizes as well as SGD with momentum.
Revised: Experimentally, we demonstrate that our method has a faster convergence rate than Adam/Amsgrad, and generalizes as well as SGD with momentum.

Original: We also provide a convergence analysis of our proposed method to complement our intuition.
Revised: We also provide a convergence analysis of our proposed method to complement our intuition.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the rapid development of neural network architectures (Goodfellow et al., 2016), training algorithms have attracted much more attention in the modern machine learning community.
Revised: With the rapid development of neural network architectures (Goodfellow et al., 2016), training algorithms have attracted much more attention in the modern machine learning community.

Original: Due to the network size and data amount increasing dramatically, calculating the full gradient of data and implementing the full gradient descent (GD) become computationally expensive.
Revised: Due to the network size and data amount increasing dramatically, calculating the full gradient of data and implementing the full gradient descent (GD) become computationally expensive.

Original: Therefore, stochastic gradient descent (SGD) (Robbins & Monro, 1951) becomes the most practical optimization method for training deep neural networks (DNNs).
Revised: Therefore, stochastic gradient descent (SGD) (Robbins & Monro, 1951) becomes the most practical optimization method for training deep neural networks (DNNs).

Original: In each iteration, SGD samples mini-batch data and computes the gradient corresponding to the mini-batch.
Revised: In each iteration, SGD samples mini-batch data and computes the gradient corresponding to the mini-batch.

Original: Although SGD is computationally affordable, it needs a mechanism of ﬁne-tune the learning rate, e.g., linear decay or exponential decay, to converge efﬁciently.
Revised: Although SGD is computationally affordable, it needs a mechanism of ﬁne-tune the learning rate, e.g., linear decay or exponential decay, to converge efﬁciently.

Original: In fact, it is pretty brittle to tune the learning rate dynamically with SGD.
Revised: In fact, it is pretty brittle to tune the learning rate dynamically with SGD.

Original: To release the learning rate tuning burden of SGD and accelerate its convergence, several adaptive variants of SGD were proposed, including Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012),
Revised: To release the learning rate tuning burden of SGD and accelerate its convergence, several adaptive variants of SGD were proposed, including Adagrad (Duchi et al., 2011), Adadelta (Zeiler, 2012),

Original: RMSProp (Hinton et al., 2012), Adam (Kingma & Ba, 2014), etc.
Revised: RMSProp (Hinton et al., 2012), Adam (Kingma & Ba, 2014), etc.

Original: The descent direction is elementwise automatically adapted by the ﬁrst moment and the second moment with an exponential average of gradients.
Revised: The descent direction is elementwise automatically adapted by the ﬁrst moment and the second moment with an exponential average of gradients.

Original: Such a descent direction has achieved great improvements in practice, while the fundamental interpretation is still unclear.
Revised: Such a descent direction has achieved great improvements in practice, while the fundamental interpretation is still unclear.

Original: Besides, the theoretical regret analysis of these online learning algorithms has become completed gradually in convex objective settings.
Revised: Besides, the theoretical regret analysis of these online learning algorithms has become completed gradually in convex objective settings.

Original: Due to the fast decay of exponential moving average, Adam cannot converge even in simple convex cases as shown in (Reddi et al., 2019).
Revised: Due to the fast decay of exponential moving average, Adam cannot converge even in simple convex cases as shown in (Reddi et al., 2019).

Original: Amsgrad (Reddi et al., 2019) addressed this issue by keeping an extra non-decreasing sequence to buffer the fast decay.
Revised: Amsgrad (Reddi et al., 2019) addressed this issue by keeping an extra non-decreasing sequence to buffer the fast decay.

Original: On the other hand, although adaptive methods are easy to use and fast to converge, the generalization results cannot be as good as SGD.
Revised: On the other hand, although adaptive methods are easy to use and fast to converge, the generalization results cannot be as good as SGD (Wilson et al., 2017).

Original: Various works (Chen & Gu, 2018;
Revised: Various works (Chen & Gu, 2018;

Original: Luo et al., 2019) have been proposed to make algorithms not only converge faster but also has good generalization.
Revised: Luo et al., 2019) have been proposed to make algorithms not only converge faster but also has good generalization.

Original: However, these algorithms still inherit the second moment following Adam.
Revised: However, these algorithms still inherit the second moment following Adam.

Original: Different from existing works, we provide a novel view for this problem by revisiting adaptive algorithms Through the view of direction angle between the descent direction and the momentum, we are able to demonstrate a fresh understanding of momentum.
Revised: Different from existing works, we provide a novel view for this problem by revisiting adaptive algorithms Through the view of direction angle between the descent direction and the momentum, we are able to demonstrate a fresh understanding of momentum.

Original: In summary, we propose a new algorithm, called acu te adaptive momen tum
Revised: In summary, we propose a new algorithm, called acu te adaptive momen tum

Original: ( Acutum ), to ﬁnd adaptive gradients with only the ﬁrst moment.
Revised: ( Acutum ), to ﬁnd adaptive gradients with only the ﬁrst moment.

Original: To remove the second moments safely, we investigate their properties by reformulating the proximal subproblem of Adagrad.
Revised: To remove the second moments safely, we investigate their properties by reformulating the proximal subproblem of Adagrad.

Original: The main observation is that second moments essentially penalize the projection of the current descent direction on the exponential moving average of previous gradients.
Revised: The main observation is that second moments essentially penalize the projection of the current descent direction on the exponential moving average of previous gradients.

Original: The new intuition allows us to remove second moments in
Revised: The new intuition allows us to remove second moments in

Original: Adam without any loss of performance.
Revised: Adam without any loss of performance.

Original: To the best of our knowledge, it is the ﬁrst time to remove the second moments for adaptive gradient methods.
Revised: To the best of our knowledge, it is the ﬁrst time to remove the second moments for adaptive gradient methods.

Original: As shown by our experiment, the proposed algorithm ( Acutum ) outperforms Adam/Amsgrad in convergence speed while almost preserving the generalization ability of SGD.
Revised: As shown by our experiment, the proposed algorithm ( Acutum ) outperforms Adam/Amsgrad in wall-clock time while almost preserving the generalization ability of SGD.

Original: In particular, we make the following contributions:
Revised: In particular, we make the following contributions:

Original: • We propose a new optimization algorithm called Acutum , which doesn’t need to compute the second moments.
Revised: • We propose a new optimization algorithm called Acutum , which doesn’t need to compute the second moments.

Original: To the best of our knowledge, it is the ﬁrst time to obtain the adaptive gradients with only the ﬁrst moments.
Revised: To the best of our knowledge, it is the ﬁrst time to obtain the adaptive gradients with only the ﬁrst moments.

Original: The intuition behind our algorithm is to remove second moments by attaching their purpose to the ﬁrst moment through Adagrad’s proximal subproblem inspiration.
Revised: The intuition behind our algorithm is to remove second moments by attaching their purpose to the ﬁrst moment through Adagrad’s proximal subproblem inspiration.

Original: • We provide a regret convergence analysis of Acutum on the convex setting, based on the analysis of (Kingma & Ba, 2014; Reddi et al., 2019; Chen & Gu, 2018), and prove a datadependent regret bound, complementing our intuition.
Revised: • We provide a regret convergence analysis of Acutum on the convex setting, based on the analysis of (Kingma & Ba, 2014; Reddi et al., 2019; Chen & Gu, 2018), and prove a datadependent regret bound, complementing our intuition.

Original: • We also provide various experiments about our proposed Acutum on training modern deep models.
Revised: • We also provide various experiments about our proposed Acutum on training modern deep models.

Original: The experimental results empirically show that Acutum takes the best from both Adam-type methods and SGD.
Revised: The experimental results empirically show that Acutum takes the best from both Adam-type methods and SGD.

Original: Notations.
Revised: Notations.

Original: In the rest of this paper, we assume any decision variables θ ∈ R d where d denotes the dimension of parameters.
Revised: In the rest of this paper, we assume any decision variables θ ∈ R d where d denotes the dimension of parameters.

Original: We denote the l 2 norm of a given vector θ by (cid:107) θ (cid:107) =
Revised: We denote the l 2 norm of a given vector θ by (cid:107) θ (cid:107) =

Original: (cid:113)(cid:80) di
Revised: (cid:113)(cid:80) di

Original: =

Revised: =

Original: θ 2 i .
Revised: θ 2 i .

Original: With slightly abuse of notation, we write arithmetic symbols as element-wise operations for vectors, e.g., a 2 = [ a 21 , a 22 , . .
Revised: With slightly abuse of notation, we write arithmetic symbols as element-wise operations for vectors, e.g., a 2 = [ a 21 , a 22 , . .

Original: T , a / b = [ a 1 / b 1 , a 2 / b 2 , . .
Revised: T , a / b = [ a 1 / b 1 , a 2 / b 2 , . .

Original: T .
Revised: T .

Original: We denote by (cid:98) x
Revised: We denote by (cid:98) x

Original: (cid:99) the greatest integer less than or equal to the real number x .
Revised: (cid:99) the greatest integer less than or equal to the real number x .

Original: Given any integers x, y , where y (cid:54) = 0 , we denote by x (mod y )
Revised: Given any integers x, y , where y (cid:54) = 0 , we denote by x (mod y )

Original: the remainder of the Euclidean division of x by y .
Revised: the remainder of the Euclidean division of x by y .

Original: In the optimization setting, we denote by f i ( · ) the loss function when we feed the i -th training data into our model, we also denote f S ( · ) when we feed a collection of samples, i.e., f S ( · ) :
Revised: In the optimization setting, we denote by f i ( · ) the loss function when we feed the i -th training data into our model, we also denote f S ( · ) when we feed a collection of samples, i.e., f S ( · ) :

Original: = (cid:80) i ∈ S
Revised: = (cid:80) i ∈ S

Original: f i ( · ) .
Revised: f i ( · ) .

Original: Given any point θ ∈ R d and any convex set
Revised: Given any point θ ∈ R d and any convex set

Original: X ⊆ R d , we denote by Π X ( θ ) the projection of θ onto X , i.e., arg min µ ∈X (cid:107) µ − θ (cid:107) .
Revised: X ⊆ R d , we denote by Π X ( θ ) the projection of θ onto X , i.e., arg min µ ∈X (cid:107) µ − θ (cid:107) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The increasing size of deep neural networks and the amount of train data have dramatically induced the difﬁculty of training neural networks.
Revised: The increasing size of deep neural networks and the amount of train data have dramatically induced the difﬁculty of training neural networks.

Original: The huge parameter size makes the landscape of neural networks more non-convex, and the amount of train data requires more computation resources to calculate the gradient.
Revised: The huge parameter size makes the landscape of neural networks more non-convex, and the amount of train data requires more computation resources to calculate the gradient.

Original: Therefore, all the facts urge researchers to design faster optimization algorithms for deep neural networks, while maintaining a reasonable generalization ability.
Revised: Therefore, all the facts urge researchers to design faster optimization algorithms for deep neural networks, while maintaining a reasonable generalization ability.

Original: SGD-momentum is widely used in training large-scale neural networks while the excellent generalization ability of SGD is vulnerable to its learning rate.
Revised: SGD-momentum is widely used in training large-scale neural networks while the excellent generalization ability of SGD is vulnerable to its learning rate.

Original: Meanwhile, researchers began to focus on the design of adaptive gradient methods for a fast and simple optimizer.
Revised: Meanwhile, researchers began to focus on the design of adaptive gradient methods for a fast and simple optimizer.

Original: Adagrad proposed in (Duchi et al., 2011) introduced the second moment to obtain a self-adaptive learning rate, and hence freed researchers’ hands of parameter tuning.
Revised: Adagrad proposed in (Duchi et al., 2011) introduced the second moment to obtain a self-adaptive learning rate, and hence freed researchers’ hands of parameter tuning.

Original: The update rules of Adagrad can be formulated as follows:
Revised: The update rules of Adagrad can be formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: where g t denotes the stochastic gradient, v t is the accumulation of gradient’s second moments, i.e., v t = 1 t (cid:80) t τ =1 g 2 τ , and α t is the decreasing learning rate with α t
Revised: where g t denotes the stochastic gradient, v t is the accumulation of gradient’s second moments, i.e., v t = 1 t (cid:80) t τ =1 g 2 τ , and α t is the decreasing learning rate with α t

Original: = O (1 / √ t ) .
Revised: = O (1 / √ t ) .

Original: Theoretically, Adagrad improves the convergence of regret from O ( √ d/ √ T ) to O (1 / √ T ) for the convex objectives with sparse gradients.
Revised: Theoretically, Adagrad improves the convergence of regret from O ( √ d/ √ T ) to O (1 / √ T ) for the convex objectives with sparse gradients.

Original: However, in practice, people realize that adaptive gradient of Adagrad, i.e., g t / √ v t goes to zero very quickly due to v t accumulating to large number quickly as the algorithm proceeds.
Revised: However, in practice, people realize that adaptive gradient of Adagrad, i.e., g t / √ v t goes to zero very quickly due to v t accumulating to large number quickly as the algorithm proceeds.

Original: To make it through, RMSProp (Hinton et al., 2012) introduced the exponential decay in the second moment to control the accumulation speed of second moment in Adagrad.
Revised: To make it through, RMSProp (Hinton et al., 2012) introduced the exponential decay in the second moment to control the accumulation speed of second moment in Adagrad.

Original: Furthermore,
Revised: Furthermore,

Original: (Kingma & Ba, 2014) incorporated the momentum into RMSProp and proposed Adam.
Revised: (Kingma & Ba, 2014) incorporated the momentum into RMSProp and proposed Adam.

Original: The detailed procedure of Adam can be formulated as:
Revised: The detailed procedure of Adam can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: where m t is the exponential decay of momentum, i.e., m t = (cid:80) tτ
Revised: where m t is the exponential decay of momentum, i.e., m t = (cid:80) tτ

Original: =1 (1 − β 1 )
Revised: =1 (1 − β 1 )

Original: β t − τg τ , and v t is the exponential decay of second moment, i.e., v t
Revised: β t − τg τ , and v t is the exponential decay of second moment, i.e., v t

Original: =
Revised: =

Original: (cid:80) tτ
Revised: (cid:80) tτ

Original: =1 (1 − β 2 ) β t − τg 2 τ .
Revised: =1 (1 − β 2 ) β t − τg 2 τ .

Original: The faster convergence, robust hyper-parameters and good performance on CV and NLP’s tasks make Adam become one of the most successful optimizers these years.
Revised: The faster convergence, robust hyper-parameters and good performance on CV and NLP’s tasks make Adam become one of the most successful optimizers these years.

Original: However, (Reddi et al., 2019) showed Adam does not converge in certain convex cases, and proposed Amsgrad to correct the direction of Adam with imposing an increasing sequence of the second moment.
Revised: However, (Reddi et al., 2019) showed Adam does not converge in certain convex cases, and proposed Amsgrad to correct the direction of Adam with imposing an increasing sequence of the second moment.

Original: Although convergence, in theory, is ﬁxed and the training speed is still fast, the generalization ability of adaptive algorithms is still worse than SGD with momentum.
Revised: Although convergence, in theory, is ﬁxed and the training speed is still fast, the generalization ability of adaptive algorithms is still worse than SGD with momentum.

Original: Thus, there are a lot of studies to improve the generalization performance of Adam types algorithms.
Revised: Thus, there are a lot of studies to improve the generalization performance of Adam types algorithms.

Original: Almost all of these works try to make some connections between Adam and SGD-momentum such that the proposed algorithm converges faster like Adam while generalizing better as SGD.
Revised: Almost all of these works try to make some connections between Adam and SGD-momentum such that the proposed algorithm converges faster like Adam while generalizing better as SGD.

Original: For example, (Zhang et al., 2017) proposed a normalized direction-preserving Adam (ND-Adam).
Revised: For example, (Zhang et al., 2017) proposed a normalized direction-preserving Adam (ND-Adam).

Original: To make the direction of the gradients preserved, the authors adjusted the learning rate to each weight vector as a whole, instead of each individual weight.
Revised: To make the direction of the gradients preserved, the authors adjusted the learning rate to each weight vector as a whole, instead of each individual weight.

Original: It could be thought as an adjustment of learning rate dynamically in each iteration, instead of the learning rate decay.
Revised: It could be thought as an adjustment of learning rate dynamically in each iteration, instead of the learning rate decay.

Original: (Loshchilov & Hutter, 2017) proposed to modify
Revised: (Loshchilov & Hutter, 2017) proposed to modify

Original: Adam to decay all weights the same regardless of the gradient variances.
Revised: Adam to decay all weights the same regardless of the gradient variances.

Original: (Keskar & Socher, 2017) proposed to improve the generalization performance by investigating a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate.
Revised: (Keskar & Socher, 2017) proposed to improve the generalization performance by investigating a hybrid strategy that begins training with an adaptive method and switches to SGD when appropriate.

Original: The experimental results show that it obtains a better generalization with the convergence speed sacriﬁcing.
Revised: The experimental results show that it obtains a better generalization with the convergence speed sacriﬁcing.

Original: With a similar insight, (Luo et al., 2019) introduced the element-wise clipping technique to the second moment and proposed AdaBound, which allows the algorithm behavior switched from a clipped Adam to
Revised: With a similar insight, (Luo et al., 2019) introduced the element-wise clipping technique to the second moment and proposed AdaBound, which allows the algorithm behavior switched from a clipped Adam to

Original: SGD-momentum as the algorithm proceeds.
Revised: SGD-momentum as the algorithm proceeds.

Original: Besides, (Chen & Gu, 2018) proposed Padam which tried to ﬁnd the balanced state between Adam and SGD-momentum through tuning the order of the second moment in the update paradigm.
Revised: Besides, (Chen & Gu, 2018) proposed Padam which tried to ﬁnd the balanced state between Adam and SGD-momentum through tuning the order of the second moment in the update paradigm.

Original: All the aforementioned methods try to ﬁnd the connection between Adam and SGD-momentum, thus all these algorithms take the second moment adaptation as a grant.
Revised: All the aforementioned methods try to ﬁnd the connection between Adam and SGD-momentum, thus all these algorithms take the second moment adaptation as a grant.

Original: The difference between previous works and ours is that we suggest to remove the second moment and propose an angle based algorithm, by revisiting the original idea of second moment adaptation.
Revised: The difference between previous works and ours is that we suggest to remove the second moment and propose an angle based algorithm, by revisiting the original idea of second moment adaptation.

Original: Such intuition could be of independent interest in the ﬁeld of the optimization community.
Revised: Such intuition could be of independent interest in the ﬁeld of the optimization community.

Original:  A CUTUM : A CUTE A DAPTIVE M OMENTUM A LGORITHM
Revised:  T HE A CUTE A DAPTIVE M OMENTUM A LGORITHM

Original: In this section, we ﬁrst revisit existing adaptive algorithms from a novel point of view, then introduce the intuition and provide the algorithm procedure of our method Acutum .
Revised: In this section, we ﬁrst revisit existing adaptive algorithms from a novel point of view, then introduce the intuition and provide the algorithm procedure of our method Acutum .

Original: The performance guarantees established in Section 4 and evaluation results in Section 5 validate our insights.
Revised: The performance guarantees established in Section 4 and evaluation results in Section 5 validate our insights.

Original: From the above section, one may notice that almost all the adaptive methods utilize second moments to adjust the individual magnitude of the updates.
Revised: From the above section, one may notice that almost all the adaptive methods utilize second moments to adjust the individual magnitude of the updates.

Original: However, the efﬁcient calculation of second moments, see Equation (1), is only proposed in Adagrad (Duchi et al., 2011).
Revised: However, the efﬁcient calculation of second moments, see Equation (1), is only proposed in Adagrad (Duchi et al., 2011).

Original: Let’s take a look at the original update paradigm of Adagrad:
Revised: Let’s take a look at the original update paradigm of Adagrad:

Original: [Equation]
Revised: [Equation]

Original: where G t = (cid:80) tτ
Revised: where G t = (cid:80) tτ

Original: =1 g τ g Tτ , and g τ denotes the stochastic gradient calculated at iteration τ .
Revised: =1 g τ g Tτ , and g τ denotes the stochastic gradient calculated at iteration τ .

Original: Recall that in each iteration, SGD seeks the optimal solution of a subproblem constructed from a quadratic approximation of the objective.
Revised: Recall that in each iteration, SGD seeks the optimal solution of a subproblem constructed from a quadratic approximation of the objective.

Original: Similarly, we can write the update procedure of Adagrad in Equation (2) as
Revised: Similarly, we can write the update procedure of Adagrad in Equation (2) as

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst-order penalty T 1 leads the parameter θ t to the opposite direction to the current gradient g t .
Revised: where the ﬁrst-order penalty T 1 leads the parameter θ t to the opposite direction to the current gradient g t .

Original: Note that Adagrad can adaptively adjust each individual dimension of θ t through introducing
Revised: Note that Adagrad can adaptively adjust each individual dimension of θ t through introducing

Original: G t in the quadratic term T 2 , while SGD cannot.
Revised: G t in the quadratic term T 2 , while SGD cannot.

Original: If we substitute the deﬁnition of G t back into Equation (3), T 2 can be formulated as:
Revised: If we substitute the deﬁnition of G t back into Equation (3), T 2 can be formulated as:

Original: [Equation]
Revised: [Equation]

Original: By Young’s inequality (Young, 1912)
Revised: By Young’s inequality (Young

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: we can obtain an upper bound of T 2 as follows:
Revised: we can obtain an upper bound of T 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: Combining Equation (5) and Equation (3)
Revised: Combining Equation (5) and Equation (3)

Original: [Equation]
Revised: [Equation]

Original: we can approximate the update rule of Adagrad as follows:
Revised: we can approximate the update rule of Adagrad as follows:

Original: [Equation]
Revised: [Equation]

Original: We can think the upper bound of the Adagrad subproblem as the combination of the standard ﬁrst and second order penalty as T 1 and T 3 , with a penalty for the projections of current descent θ − θ t on the previous gradients g τ , i.e., T 4 .
Revised: We can think the upper bound of the Adagrad subproblem as the combination of the standard ﬁrst and second order penalty as T 1 and T 3 , with a penalty for the projections of current descent θ − θ t on the previous gradients g τ , i.e., T 4 .

Original: With these observations in mind, a natural question comes out: how can we understand T 4 in neural network training tasks or in general online learning tasks?
Revised: With these observations in mind, a natural question comes out: how can we understand T 4 in neural network training tasks or in general online learning tasks?

Original: Before answering this question, we ﬁrst describe the practical usage of training samples.
Revised: Before answering this question, we ﬁrst describe the practical usage of training samples.

Original: The training procedure of DNNs can be stated as follows.
Revised: The training procedure of DNNs can be stated as follows.

Original: The algorithm randomly shufﬂes the whole training dataset A and partitions A into mini-batch of equal size { A 0 , A 1 , . . .
Revised: The algorithm randomly shufﬂes the whole training dataset A and partitions A into mini-batch of equal size { A 0 , A 1 , . . .

Original: , A p − 1 } , then the algorithm is fed with the sample in a ﬁxed order, say, ﬁrst A 0 , then A 1 , and so on.
Revised: , A p − 1 } , then the algorithm is fed with the sample in a ﬁxed order, say, ﬁrst A 0 , then A 1 , and so on.

Original: The whole procedure repeats after each pass of the whole dataset.
Revised: The whole procedure repeats after each pass of the whole dataset.

Original: To make our description more clear, we take the ﬁrst pass as an example.
Revised: To make our description more clear, we take the ﬁrst pass as an example.

Original: Let ∇ f A t ( θ t ) be the gradient calculated on iteration t by using the sample in subset A t , where 0 ≤ t < p due to the ﬁrst pass.
Revised: Let ∇ f A t ( θ t ) be the gradient calculated on iteration t by using the sample in subset A t , where 0 ≤ t < p due to the ﬁrst pass.

Original: Note that the loss function is the sum of whole samples, e.g., 1 n (cid:80) ni
Revised: Note that the loss function is the sum of whole samples, e.g., 1 n (cid:80) ni

Original: =

Revised: =

Original: f i ( θ ) , if we utilize ∇ f A i ( θ t ) to directly update parameters like SGD, e.g., θ t +

Revised: f i ( θ ) , if we utilize ∇ f A i ( θ t ) to directly update parameters like SGD, e.g., θ t +

Original: =
Revised: =

Original: θ t − α ∇ f A t ( θ t ) , the batch loss (cid:80) i ∈
Revised: θ t − α ∇ f A t ( θ t ) , the batch loss (cid:80) i ∈

Original: A t f i ( θ ) will decrease since it aligns with the opposite direction of its gradient, e.g., ( θ t +1 − θ t ) T ∇ f
Revised: A t f i ( θ ) will decrease since it aligns with the opposite direction of its gradient, e.g., ( θ t +1 − θ t ) T ∇ f

Original: A t ( θ t ) =
Revised: A t ( θ t ) =

Original: −(cid:107)∇
Revised: −(cid:107)∇

Original: f
Revised: f

Original: A t ( θ t ) (cid:107) 2 < 0 .
Revised: A t ( θ t ) (cid:107) 2 < 0 .

Original: However, for loss corresponding to the sample not in A t , it is highly possible that ( θ t +1 − θ t ) T ∇ f A i ( θ t ) > 0 .
Revised: However, for loss corresponding to the sample not in A t , it is highly possible that ( θ t +1 − θ t ) T ∇ f A i ( θ t ) > 0 .

Original: On other words, only using −∇ f A i ( θ t ) as update direction will decrease the loss function corresponding to A t but increase the total loss function except A t .
Revised: On other words, only using −∇ f A i ( θ t ) as update direction will decrease the loss function corresponding to A t but increase the total loss function except A t .

Original: Ideally, if there exists a direction θ t +1 − θ t which is orthogonal to ∇ f A i ( θ t ) for any i (cid:54) =
Revised: Ideally, if there exists a direction θ t +1 − θ t which is orthogonal to ∇ f A i ( θ t ) for any i (cid:54) =

Original: t (mod p ) , and forms an acute angle with ∇ f A t ( θ t ) , then θ t +1 can guarantee a sufﬁcient descent for the loss function corresponding to A t while not increase the loss function outside A t .
Revised: t (mod p ) , and forms an acute angle with ∇ f A t ( θ t ) , then θ t +1 can guarantee a sufﬁcient descent for the loss function corresponding to A t while not increase the loss function outside A t .

Original: Then a natural update of θ t +1 to achieve the above goal can be formulated as follows:
Revised: Then a natural update of θ t +1 to achieve the above goal can be obtained by solving the following sub-problem:

Original: [Equation]
Revised: [Equation]

Original: where ( θ − θ t ) T ∇ f A i ( θ t )

Original: [-= 
-]
Revised: [-= 
-]

Original: , i (cid:54) = t (mod p ) . In practice, however, the computational cost of computing ∇ f A i ( θ t ) for all i is expensive in each iteration.
Revised: In practice, however, the computational cost of computing ∇ f A i ( θ t ) for all i is expensive in each iteration.

Original: As a natural solution, we can approximate ∇ f A i ( θ t ) with previous gradients f
Revised: As a natural solution, we can approximate ∇ f A i ( θ t ) with previous gradients f

Original: A i ( θ i ) , which is exactly g i calculated in Adagrad in iteration i before iteration t , where i (cid:54) =
Revised: A i ( θ i ) , which is exactly g i calculated in Adagrad in iteration i before iteration t , where i (cid:54) =

Original: t (mod p ) .
Revised: t (mod p ) .

Original: Hence a soft margin variant of Equation (6) can be formulated as
Revised: Hence a soft margin variant of Equation (6) can be formulated as

Original: [Equation]
Revised: [Equation]

Original: which is exactly the formulation in Equation (5) and Equation (3)!
Revised: which is exactly the formulation in Equation (5) and Equation (3)!

Original: Thus, one can reasonably consider that Adagrad decreases the cumulative loss by guaranteeing a descent on the current batch loss while does not increase previous batch loss.
Revised: Thus, one can reasonably consider that Adagrad decreases the cumulative loss by guaranteeing a descent on the current batch loss while does not increase previous batch loss.

Original: Although the approximation of ∇ f A i ( θ t ) using previous gradients g i in Adagrad performs well in practice, there is still a large accuracy gap.
Revised: Although the approximation of ∇ f A i ( θ t ) using previous gradients g i in Adagrad performs well in practice, there is still a large accuracy gap.

Original: For example, let g t − 1 and g t − 2 denote the gradients ∇ f A
Revised: For example, let g t − 1 and g t − 2 denote the gradients ∇ f A

Original: i − 1 ( θ t − 1 ) and ∇ f A
Revised: i − 1 ( θ t − 1 ) and ∇ f A

Original: i − 2 ( θ t − 2 ) , respectively.
Revised: i − 2 ( θ t − 2 ) , respectively.

Original: It is often the case that the approximation error between ∇ f A i − 1 ( θ t ) and g t − 1 is smaller than that between ∇ f A i − 2 ( θ t ) and g t − 2 , assuming the objective is local smoothness 1 .
Revised: It is often the case that the approximation error between ∇ f A i − 1 ( θ t ) and g t − 1 is smaller than that between ∇ f A i − 2 ( θ t ) and g t − 2 , assuming the objective is local smoothness 1 .

Original: In other words, the approximation accuracy of g τ to ∇ f A i ( θ t ) decays as τ decreases.
Revised: In other words, the approximation accuracy of g τ to ∇ f A i ( θ t ) decays as τ decreases.

Original: Inspired by this observation, we substitute the arithmetic average in Equation (7) to exponential moving average and reassign the coefﬁcients of g τ like Adam/momentum methods, then Equation (7) becomes
Revised: Inspired by this observation, we substitute the arithmetic average in Equation (7) to exponential moving average and reassign the coefﬁcients of g τ like Adam/momentum methods, then Equation (7) becomes

Original: [Equation]
Revised: [Equation]

Original: This is what RMSProp actually does.
Revised: This is what RMSProp actually does.

Original: Therefore, we obtain the same optimization intuition of RMSProp as Adagrad but add weight of the approximate gradient according to its approximation accuracy.
Revised: Therefore, we obtain the same optimization intuition of RMSProp as Adagrad but add weight of the approximate gradient according to its approximation accuracy.

Original: Enlightened by the above analysis, we are ready to propose a new adaptive momentum method.
Revised: Enlightened by the above analysis, we are ready to propose a new adaptive momentum method.

Original: Let m t denote the approximated gradient of the previous batch loss at current parameters.
Revised: Let m t denote the approximated gradient of the previous batch loss at current parameters.

Original: With the following observation: (i) if θ t − θ t +1 forms acute angles with both g t and m t , rather than penalizing the projection like what Adagrad and RMSProp do, we can obtain both descent on current and previous batches; (ii) to handle the case when the estimation of previous gradients using m t is not accurate, we expect the current gradient g t to dominate the descent direction, which is denoted as ˆ m t .
Revised: With the following observation: (i) if θ t − θ t +1 forms acute angles with both g t and m t , rather than penalizing the projection like what Adagrad and RMSProp do, we can obtain both descent on current and previous batches; (ii) to handle the case when the estimation of previous gradients using m t is not accurate, we expect the current gradient g t to dominate the descent direction, which is denoted as ˆ m t .

Original: It can be guaranteed by requiring the projection length of ˆ m t on g t is less than the length of g t if properly regularizing m t .
Revised: It can be guaranteed by requiring the projection length of ˆ m t on g t is less than the length of g t if properly regularizing m t .

Original: Then, we propose a new iterative subproblem in the following:
Revised: Then, we propose a new iterative subproblem in the following:

Original: [Equation]
Revised: [Equation]

Original: where ˆ m t

Original: +1 =

Original: g

Original: [Equation]

Original: With such formulation, the following properties hold:
Revised: With such formulation, the following properties hold:

Original: • We ensure the above two observations, speciﬁcally, with the update of momentum ˆ m t , we can guarantee that (cid:107) ˆ m t +1 (cid:107)
Revised: • We ensure the above two observations, speciﬁcally, with the update of momentum ˆ m t , we can guarantee that (cid:107) ˆ m t +1 (cid:107)

Original: =
Revised: =

Original: O ( (cid:107) g t (cid:107) ) .
Revised: O ( (cid:107) g t (cid:107) ) .

Original: • We attach the properties of the second moment to the ﬁrst-order operation, inspired by Adagrad proximal subproblem.
Revised: • We attach the properties of the second moment to the ﬁrst-order operation, inspired by Adagrad proximal subproblem.

Original: The proposed algorithm Acutum is summarized in Algorithm 1.Algorithm
Revised: The proposed algorithm Acutum is summarized in Algorithm 1.

Original: 1: Input: initial point θ 0 ∈ X ; step size { α t } , parameters { β } 2: set ˆ m 3: for to do 4: g  ∇ f 
Revised: g t = ∇ f t ( θ t )

Original: 5: ˆ m t =
Revised: ˆ m t =

Original: g t + β 1 ,t (cid:107) g t (cid:107) (cid:14) ( (cid:107) ˆ m t − 1 (cid:107) + (cid:15) )
Revised: g t + β 1 ,t (cid:107) g t (cid:107) (cid:14) ( (cid:107) ˆ m t − 1 (cid:107) + (cid:15) )

Original: · ˆ m t −6: θ t +1 =
Revised: · ˆ m t − θ t +1 =

Original: Π X ( θ t − α t · ˆ m t )7: end for 8: Return: x
Revised: Π X ( θ t − α t · ˆ m t )

Original: Note that, at step 5 we use (cid:15) to prevent precision overﬂow caused by a tiny (cid:107) ˆ m t − 1 (cid:107) .
Revised: Note that, at step 5 we use (cid:15) to prevent precision overﬂow caused by a tiny (cid:107) ˆ m t − 1 (cid:107) .

Original: β 1 ,t is the momentum weight.
Revised: β 1 ,t is the momentum weight.

Original: We specify its value in the theoretical analysis in Section 4.
Revised: We specify its value in the theoretical analysis in Section 4.

Original: In practice, we can simply set β 1 ,t to be 1 .
Revised: In practice, we can simply set β 1 ,t to be 1 .

Original: Compared with other adaptive gradient methods, Acutum has fewer hyper-parameters, and is much easier to tune deep models.
Revised: Compared with other adaptive gradient methods, Acutum has fewer hyper-parameters, and is much easier to tune deep models.

Original: Besides, it is also computation-friendly (for both time and space complexities) since the algorithm does not need second moments.
Revised: Besides, it is also computation-friendly (for both time and space complexities) since the algorithm does not need second moments.

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: In this section, we analyze the convergence rate of our proposed algorithm Acutum in an online convex setting, i.e., we assume the convexity of a sequence of loss functions f 1 , f 2 , . . .
Revised: In this section, we analyze the convergence rate of our proposed algorithm Acutum in an online convex setting, i.e., we assume the convexity of a sequence of loss functions f 1 , f 2 , . . .

Original: , f T , as shown in the following:
Revised: , f T , as shown in the following:

Original: Assumption 4.1 (Convexity Assumption) .
Revised: Assumption 4.1 (Convexity Assumption) .

Original: For each loss function f t ( θ ) where 1 ≤ t ≤ T , we assume it is convex.
Revised: For each loss function f t ( θ ) where 1 ≤ t ≤ T , we assume it is convex.

Original: That is, for any x , y ∈ X , we have
Revised: That is, for any x , y ∈ X , we have

Original: [Equation]
Revised: [Equation]

Original: where X is the feasible set of function f 1 , f 2 , . .
Revised: where X is the feasible set of function f 1 , f 2 , . .

Original: , f T .
Revised: , f T .

Original: Notice that this assumption is widely used for theoretical analysis, in both stochastic optimization methods (Johnson & Zhang, 2013; Defazio et al., 2014; Shalev-Shwartz & Zhang, 2013) and online learning methods (Duchi et al., 2011; Kingma & Ba, 2014; Reddi et al., 2019).
Revised: Notice that this assumption is widely used for theoretical analysis, in both stochastic optimization methods (Johnson & Zhang, 2013; Defazio et al., 2014; Shalev-Shwartz & Zhang, 2013) and online learning methods (Duchi et al., 2011; Kingma & Ba, 2014; Reddi et al., 2019).

Original: Rather than consider the objective as a ﬁnite-sum function, here we analyze the regret from a online learning perspective.
Revised: Rather than consider the objective as a ﬁnite-sum function, here we analyze the regret from a online learning perspective.

Original: That is to say, at each time step t , we are given the loss f
Revised: That is to say, at each time step t , we are given the loss f

Original: ( θ t ) after selecting some feasible solution θ t .
Revised: ( θ t ) after selecting some feasible solution θ t .

Original: Then we continue to pick the next point θ t +1 based on the previous losses.
Revised: Then we continue to pick the next point θ t +1 based on the previous losses.

Original: We adopt online regret as the convergence metric, which equals to the relative difference between the algorithm’s cumulative loss and the ofﬂine minimum value.
Revised: We adopt online regret as the convergence metric, which equals to the relative difference between the algorithm’s cumulative loss and the ofﬂine minimum value.

Original: The deﬁnition of regret is straightforward, i.e.,
Revised: The deﬁnition of regret is straightforward, i.e.,

Original: R ( 
Revised: R ( T ) := [Equation] (cid:88)

Original: [Equation]
Revised: [Equation]

Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: f
Revised: f

Original: [Equation]
Revised: [Equation]

Original: θ 
Revised: θ )

Original: [Equation]
Revised: [Equation]

Original: where X is the feasible set over all steps.
Revised: where X is the feasible set over all steps.

Original: Our purpose is to ﬁnd a sequence of points { θ t } 1 ≤ t ≤ T to minimize the overall regret R ( T ) .
Revised: Our purpose is to ﬁnd a sequence of points { θ t } 1 ≤ t ≤ T to minimize the overall regret R ( T ) .

Original: We state the main theoretical results in the following theorem.
Revised: We state the main theoretical results in the following theorem.

Original: Theorem 4.2.
Revised: Theorem 4.2.

Original: Suppose Assumption 4.1 holds.
Revised: Suppose Assumption 4.1 holds.

Original: We set  α t = α (cid:112) d/t , β 1 ,t = βλ t − 1 , where β, λ ∈ (0 , 1) .
Revised: We set (cid:15) > 0 , α t = α/ √ t , β 1 ,t = βλ t − 1 , where β, λ ∈ (0 , 1) .

Original: If the diameter of the convex feasible set X are bounded, i.e., (cid:107) θ − θ ∗ (cid:107) ∞ ≤ D ∞
Revised: If the diameter of the convex feasible set X are bounded, i.e., (cid:107) θ − θ ∗ (cid:107) ∞ ≤ D ∞

Original: for all θ ∈ X , and f t has bounded gradients, i.e., (cid:107)∇ f t ( θ ) (cid:107) ∞ ≤ G ∞ for all θ ∈ X , 1 ≤ t ≤ T , then the regret of Algorithm 1 satisﬁes:
Revised: for all θ ∈ X , and f t has bounded gradients, i.e., (cid:107)∇ f t ( θ ) (cid:107) ∞ ≤ G ∞ for all θ ∈ X , 1 ≤ t ≤ T , then the regret of Algorithm 1 satisﬁes:

Original: [Equation]
Revised: [Equation]

Original: Theorem 4.2 shows that our proposed algorithm has a similar order of step size to online gradient descent, i.e., O ( (cid:112) d/t ) , because of removing the second moments.
Revised: Theorem 4.2 shows that our proposed algorithm enjoys a similar order of step size to most of adaptive gradient methods, i.e., O (1 / √ t ) .

Original: In this situation, weight decay and appropriate learning rate decay strategies have much more inﬂuence on Acutum than Adamtype optimizers.
Revised: In practice, removing the second moments usually allows a larger learning rate, which make weight decay and appropriate learning rate decay strategies have much more inﬂuence on Acutum than Adam-type optimizers.

Original: Also, from the proof of Theorem 4.2 in appendix A.2, we ﬁnd that the same order of regret bound can also be obtained even with a more modest momentum decay β 1 ,t = β/t .
Revised: Also, from the proof of Theorem 4. in appendix A.2, we ﬁnd that the same order of regret bound can also be obtained even with a more modest momentum decay β 1 ,t = β/t .

Original: Furthermore, we show in the following corollary that our Acutum enjoys a regret bound of ˜ O ( √ T ) , which is comparable to the best of known bound for general convex online learning problems.
Revised: Furthermore, we show in the following corollary that our Acutum enjoys a regret bound of ˜ O ( √ T ) , which is comparable to the best of known bound for general convex online learning problems.

Original: Corollary 4.3.
Revised: Corollary 4.3.

Original: Frame the hypotheses of Theorem 4.2, for all T ≥ 1 , the regret of Algorithmsatisﬁes R ( T ) =
Revised: Frame the hypotheses of Theorem 4.2, for all T ≥ 1 , the regret of Algorithmsatisﬁes R ( T ) =

Original: ˜ O ( √ T ) .
Revised: ˜ O ( √ T ) .

Original: Corollary 4.3 illustrates that Acutum obtains R T
Revised: Corollary 4.3 illustrates that Acutum obtains R T

Original: = o ( T ) for all situations which is independent of the sparsity of the data features.
Revised: = o ( T ) for all situations which is independent of the sparsity of the data features.

Original: Besides, since lim T →∞ R T /T → 0 , Acutum will converge to the optimal solution when the objectives are convex.
Revised: Besides, since lim T →∞ R T /T → 0 , Acutum will converge to the optimal solution when the objectives are convex.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct extensive experiments on image classiﬁcation tasks.
Revised: In this section, we conduct extensive experiments on image classiﬁcation tasks.

Original: We want to validate that Acutum with only ﬁrst moments can not only obtain a better generalization performance than adaptive gradient methods, but also converges faster than SGD-momentum.
Revised: We want to validate that Acutum with only ﬁrst moments can not only obtain a better generalization performance than adaptive gradient methods, but also converges faster than SGD-momentum.

Original: First, note that introducing weight decay is equivalent to adding l 2 regularization to the objectives, and has a signiﬁcant impact on the generalization ability of optimizers.
Revised: First, note that introducing weight decay is equivalent to adding l 2 regularization to the objectives, and has a signiﬁcant impact on the generalization ability of optimizers.

Original: To better characterize generalization performance, we conduct experiments from two perspectives: (i)
Revised: To better characterize generalization performance, we conduct experiments from two perspectives: (i)

Original: In Section 5.1, we record
Revised: In Section 5.1, we record

Original: [Figure]
Revised: [Figure]

Original: the convergence of training loss and the test accuracy of all optimizers with ﬁxed weight decays (they optimize the same objective function).
Revised: the convergence of training loss and the test accuracy of all optimizers with ﬁxed weight decays (they optimize the same objective function).

Original: (ii)
Revised: (ii)

Original: In Section 5.2, we adopt the optimal weight decays for all optimizers, then investigate their generalization ability based on the test accuracy.
Revised: In Section 5.2, we adopt the optimal weight decays for all optimizers, then investigate their generalization ability based on the test accuracy.

Original: Now we introduce the experimental settings of our experiments, which are general and commonly used.
Revised: Now we introduce the experimental settings of our experiments, which are general and commonly used.

Original: Speciﬁcally [Figure] we implement all of our codes in Pytorch platform version 1.0+ within Python 3.6+.
Revised: Speciﬁcally, we implement all of our codes in Pytorch platform version 1.0+ within Python 3.6+.

Original: To obtain sufﬁcient comparisons with adaptive gradient variants, we choose various baselines, including SGD-momentum, Adam (Kingma & Ba, 2014), Amsgrad (Reddi et al., 2019),
Revised: To obtain sufﬁcient comparisons with adaptive gradient variants, we choose various baselines, including SGD-momentum, Adam (Kingma & Ba, 2014), Amsgrad (Reddi et al., 2019),

Original: Adamw (Loshchilov & Hutter, 2017),  Padam (Chen & Gu, 2018) and Adabound (Luo et al., 2019).
Revised: Adamw (Loshchilov & Hutter, 2017), Yogi (Zaheer et al., 2018), Padam (Chen & Gu, 2018), Radam (Liu et al., 2019) and Adabound (Luo et al., 2019).

Original: For the image classiﬁcation tasks, we use two datasets CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and test three different CNN architectures including VGGNet (Simonyan & Zisserman, 2014), ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) .
Revised: For the image classiﬁcation tasks, we use two datasets CIFAR-10, CIFAR-100 (Krizhevsky et al., 2009), and test three different CNN architectures including VGGNet (Simonyan & Zisserman, 2014), ResNet (He et al., 2016) and DenseNet (Huang et al., 2017) .

Original: To obtain stable convergence, we run 200 epochs, and decay the learning rate by 0 .
Revised: To obtain stable convergence, we run 200 epochs, and decay the learning rate by 0 .

Original: 1 every 50 epochs.
Revised: 1 every 50 epochs.

Original: We perform cross-validation to choose the best learning rates for all optimizers and second moment parameters β 2 for all adaptive gradient methods.
Revised: We perform cross-validation to choose the best learning rates for all optimizers and second moment parameters β 2 for all adaptive gradient methods.

Original: 5. E XPERIMENT WITH F IXED W EIGHT D ECAY
Revised: 5. E XPERIMENT WITH F IXED W EIGHT D ECAY

Original: We ﬁrst evaluate CIFAR-100 dataset.
Revised: We ﬁrst evaluate CIFAR-100 dataset.

Original: In this experiment, the values of weight decay in all optimizers are ﬁxed, and are chosen to be the weight decay in SGD-momentum when it achieves the maximum test accuracy.
Revised: In this experiment, the values of weight decay in all optimizers are ﬁxed, and are chosen to be the weight decay in SGD-momentum when it achieves the maximum test accuracy.

Original: From the ﬁrst row of Figure 1, i.e, the training curves of three tests, we observe that our proposed
Revised: From the ﬁrst row of Figure 1, i.e, the training curves of three tests, we observe that our proposed

Original: Acutum signiﬁcantly outperforms Adam, and has comparable rate with Padam and Adabound when we ﬁx the weight decay.
Revised: Acutum signiﬁcantly outperforms Adam, and has comparable rate with Padam and Adabound when we ﬁx the weight decay.

Original: Also, in the second row of Figure 1, i,e, the test accuracy, our Acutum outperforms all benchmarks except only SGD-momentum.
Revised: Also, in the second row of Figure 1, i,e, the test accuracy, our Acutum outperforms all benchmarks except only SGD-momentum.

Original: However, the training loss converges much slower when implemented with SGD-momentum which is also recorded in other literature.
Revised: However, the training loss converges much slower when implemented with SGD-momentum which is also recorded in other literature.

Original: The above evaluation results validate that our method achieves both fast convergence and good generalization when the weight decay is ﬁxed.
Revised: The above evaluation results validate that our method achieves both fast convergence and good generalization when the weight decay is ﬁxed.

Original: Note that the AdamW method reduces training loss quickly, however, there is a large generalization gap between it and other optimizers, including
Revised: Note that the AdamW method reduces training loss quickly, however, there is a large generalization gap between it and other optimizers, including

Original: SGD-momentum, Acutum, Padam.
Revised: SGD-momentum, Acutum, Padam.

Original: 5. E XPERIMENT WITH O PTIMAL W EIGHT D ECAY
Revised: 5. E XPERIMENT WITH O PTIMAL W EIGHT D ECAY

Original: In this section, we conduct experiments on CIFAR-10 dataset.
Revised: In this section, we conduct experiments on CIFAR-10 dataset.

Original: Speciﬁcally, we ﬁnd the optimal weight decay for each optimizer which can obtain the maximum test accuracy.
Revised: Speciﬁcally, we ﬁnd the optimal weight decay for each optimizer which can obtain the maximum test accuracy.

Original: [Figure]
Revised: [Figure]

Original: As shown in Figure

Original: [Figure] Adam [Figure] AdamW and Amsgrad perform worse than the other optimizers in the plots of test accuracy.
Revised: As shown in Figure 2, Adam, AdamW and Amsgrad perform worse than the other optimizers in the plots of test accuracy.

Original: In this optimal weight decay setting, Padam, Adabound and Acutum can even have better test performance than SGD-momentum on VGGNet if sacriﬁcing some convergence rate.
Revised: In this optimal weight decay setting, Padam, Adabound and Acutum can even have better test performance than SGD-momentum on VGGNet if sacriﬁcing some convergence rate.

Original: Besides, they outperform Adam by more than 2 percentage points for the test accuracy.
Revised: Besides, they outperform Adam by more than 2 percentage points for the test accuracy.

Original: The experimental results demonstrate that optimizers which converge faster than Acutum generalize worse.
Revised: The experimental results demonstrate that optimizers which converge faster than Acutum generalize worse.

Original: Similar to results in previous section, Acutum can effectively bridge the generalization gap, and has a comparable performance with the state-of-the-art Adam’s variants while only requiring ﬁrst moments.
Revised: Similar to results in previous section, Acutum can effectively bridge the generalization gap, and has a comparable performance with the state-of-the-art Adam’s variants while only requiring ﬁrst moments.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we revisited the existing adaptive optimization methods from a novel point of view.
Revised: In this paper, we revisited the existing adaptive optimization methods from a novel point of view.

Original: We found that the widely used second moments essentially penalize the projection of the current descent direction on the exponential moving average of previous gradients.
Revised: We found that the widely used second moments essentially penalize the projection of the current descent direction on the exponential moving average of previous gradients.

Original: In other words, it aims to decrease the current batch loss while does not increase previous batch loss.
Revised: In other words, it aims to decrease the current batch loss while does not increase previous batch loss.

Original: Following such new idea, we investigate how to obtain descent on both current and previous batches.
Revised: Following such new idea, we investigate how to obtain descent on both current and previous batches.

Original: Speciﬁcally, we proposed a new method, acute adaptive momentum ( Acutum ).
Revised: Speciﬁcally, we proposed a new method, acute adaptive momentum ( Acutum ).

Original: It removes the computationcomplicated second moments and constructed a decent direction by forming acute angles with both current and (approximated) previous gradients.
Revised: It removes the computationcomplicated second moments and constructed a decent direction by forming acute angles with both current and (approximated) previous gradients.

Original: We analyzed its convergence property in the online convex setting.
Revised: We analyzed its convergence property in the online convex setting.

Original: The extensive evaluations demonstrate that our Acutum can effectively bridge the gap between fast convergence and good generalization, i.e., the advantages of both adaptive gradient methods and SGD-momentum.
Revised: The extensive evaluations demonstrate that our Acutum can effectively bridge the gap between fast convergence and good generalization, i.e., the advantages of both adaptive gradient methods and SGD-momentum.

Original: For future work, it would be interesting to evaluate our method in other types of tasks and deep models, especially NLP and GAN problems.
Revised: For future work, it would be interesting to evaluate our method in other types of tasks and deep models, especially NLP and GAN problems.

Original: Besides, explaining the generalization ability of SGD is one of the most signiﬁcant open problems in this ﬁeld.
Revised: Besides, explaining the generalization ability of SGD is one of the most signiﬁcant open problems in this ﬁeld.

Original: We also hope that one could improve the convergence rate of SGD by only scheduling the learning rate carefully and dynamically.
Revised: We also hope that one could improve the convergence rate of SGD by only scheduling the learning rate carefully and dynamically.

Original: A P ROOF OF THE M AIN R ESULTS A. S OME I MPORTANT L EMMAS
Revised: A P ROOF OF THE M AIN R ESULTS A. S OME I MPORTANT L EMMAS

Original: In this section, we give several important deﬁnitions and lemmas which will be used in the proof of the regret convergence.
Revised: In this section, we give several important deﬁnitions and lemmas which will be used in the proof of the regret convergence.

Original: Lemma A.1.
Revised: Lemma A.1.

Original: For some convex feasible set X ⊂ R d , suppose ˆ θ 1 = min θ ∈X (cid:107) θ − θ 1 (cid:107) and ˆ θ 2 = min
Revised: For some convex feasible set X ⊂ R d , suppose ˆ θ 1 = min θ ∈X (cid:107) θ − θ 1 (cid:107) and ˆ θ 2 = min

Original: θ ∈X (cid:107) θ − θ 2 (cid:107) then we have (cid:13)(cid:13)(cid:13)
Revised: θ ∈X (cid:107) θ − θ 2 (cid:107) then we have (cid:13)(cid:13)(cid:13)

Original: ˆ θ 1 − ˆ θ 2 (cid:13)(cid:13)(cid:13) ≤ (cid:107) θ 1 − θ 2 (cid:107) .
Revised: ˆ θ 1 − ˆ θ 2 (cid:13)(cid:13)(cid:13) ≤ (cid:107) θ 1 − θ 2 (cid:107) .

Original: Proof.
Revised: Proof.

Original: Here, we only provide the proof for completeness.
Revised: Here, we only provide the proof for completeness.

Original: Combining the deﬁnition ˆ θ 1 = min θ ∈X (cid:107) θ − θ 1 (cid:107) , ˆ θ 2 = min θ ∈X (cid:107) θ − θ 2 (cid:107) with the properties of projection operator, we have (cid:16) ˆ θ 1 − θ 1 (cid:17) T (cid:16) ˆ θ 2 − ˆ θ 1 (cid:17) ≥ 0 and (cid:16) ˆ θ 2 − θ 2 (cid:17) T (cid:16) ˆ θ 1 − ˆ θ 2 (cid:17) ≥ 0 .
Revised: Combining the deﬁnition ˆ θ 1 = min θ ∈X (cid:107) θ − θ 1 (cid:107) , ˆ θ 2 = min θ ∈X (cid:107) θ − θ 2 (cid:107) with the properties of projection operator, we have (cid:16) ˆ θ 1 − θ 1 (cid:17) T (cid:16) ˆ θ 2 − ˆ θ 1 (cid:17) ≥ 0 and (cid:16) ˆ θ 2 − θ 2 (cid:17) T (cid:16) ˆ θ 1 − ˆ θ 2 (cid:17) ≥ 0 .

Original: From the above inequalities, we can obtain (cid:16) ˆ θ1 −
Revised: From the above inequalities, we can obtain (cid:16) ˆ θ

Original: [Equation]
Revised: [Equation]

Original: (cid:17)
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ˆ θ 2 − ˆ θ 1 (cid:17) + (cid:16)
Revised: ˆ θ 2 − ˆ [Equation] (cid:17) + (cid:16)

Original: [Equation]
Revised: [Equation]

Original: 2 − ˆ
Revised: 2 − ˆ

Original: [Equation]
Revised: [Equation]

Original: 2 (cid:17)
Revised: 2 (cid:17)

Original: [Equation]
Revised: [Equation]

Original: (cid:16) ˆθ 2 − ˆ θ 1 (cid:17) ≥ 0 ⇔ ( θ 2 − θ 1 )
Revised: (cid:16) ˆ

Original: [Equation]
Revised: [Equation]

Original:  (cid:16) ˆ θ 2 − ˆ θ 1 (cid:17) ≥(cid:13) (cid:13)(cid:13)  ˆ θ 2 − ˆ θ 1(cid:13) (cid:13)(cid:13)  2 .
Revised: 2 − ˆ θ 1 (cid:17) ≥ 0 ⇔ ( θ 2 − θ 1 ) T (cid:16) ˆ θ 2 − ˆ θ 1 (cid:17) ≥ (cid:13)(cid:13) (cid:13) ˆ θ 2 − ˆ θ 1 (cid:13)(cid:13) (cid:13) 2 .

Original: With the property that (cid:13)(cid:13)(cid:13) ( θ 2 − θ 1 ) − (cid:16) ˆ θ 2 − ˆ θ 1 (cid:17)(cid:13)(cid:13)(cid:13) 2 ≥ 0 , we have
Revised: With the property that (cid:13)(cid:13)(cid:13) ( θ 2 − θ 1 ) − (cid:16) ˆ θ 2 − ˆ θ 1 (cid:17)(cid:13)(cid:13)(cid:13) 2 ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Thus, the proof has been completed.
Revised: Thus, the proof has been completed.

Original: Lemma A.2.
Revised: Lemma A.2.

Original: In the Algorithm 1, we have (cid:107) ˆ m t (cid:107) ≤ 2 (cid:107) g t (cid:107) at each iteration t if the momentum parameter satisﬁes β 1 ,t ≤ 1 .
Revised: In the Algorithm 1, we have (cid:107) ˆ m t (cid:107) ≤ 2 (cid:107) g t (cid:107) at each iteration t if the momentum parameter satisﬁes β 1 ,t ≤ 1 .

Original: Proof.
Revised: Proof.

Original: With the deﬁnition of ˆ m t , we have
Revised: With the deﬁnition of ˆ m t , we have

Original: [Equation]
Revised: [Equation]

Original: With the condition that β 1 ,t ≤ 1 , we have (cid:107) ˆ m t (cid:107) ≤ 2 (cid:107) g t (cid:107) .
Revised: With the condition that β 1 ,t ≤ 1 , we have (cid:107) ˆ m t (cid:107) ≤ 2 (cid:107) g t (cid:107) .

Original: T HEOREM  
Revised: T HEOREM 4.

Original: Proof.
Revised: Proof.

Original: Consider the update rule of Algorithm 1, parameters θ ∗ and θ t satisfy Π X 
Revised: Consider the update rule of Algorithm [Equation] parameters θ ∗ and θ t satisfy Π X ( θ

Original: [Equation]
Revised: [Equation]

Original:  ∗ and 
Revised: ) = [Equation] ∗ and Π X ( θ t −

Original: [Equation]
Revised: [Equation]

Revised: t ˆ m t )

Revised: = θ t +1 .

Original: With Lemma A.
Revised: With Lemma A.

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where 1 follows from the deﬁnition of ˆ m t .
Revised: where 1 follows from the deﬁnition of ˆ m t .

Original: Rearrange the items in Equation (10), we obtain 
Revised: Rearrange the items in Equation (10) [Equation] we obtain [Equation]

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where 1 establishes since the Young’s inequality.
Revised: where 1 holds by the Young’s inequality.

Original: By Assumption 4.1, with the convexity of all f i , we have
Revised: By Assumption 4.1, with the convexity of all f i , we have

Original: [Equation]
Revised: [Equation]

Original: Submit Equation (11) to the above inequalities, we obtain
Revised: Submit Equation (A.2) to the above inequalities, we obtain

Original: T (cid:88)
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: whereit should be notice that β 1 ,t = βλ t − 1 is monotonically decreasing with t. For T
Revised: where β 1 ,t = βλ t − 1 is monotonically decreasing with t

Original: [Equation]
Revised: [Equation]

Original:  we haveT
Revised: For T 1 , we have

Original: [Equation] 
Revised: [Equation] where we set α t = α/ √ t

Original: [Equation]
Revised: [Equation]

Original: For T 2 in Equation (12)
Revised: For T 2 in Equation (11), we obtain

Revised: T (cid:88) t = d (cid:88)

Original: [Equation]
Revised: [Equation]

Original: we obtain T(cid:88)
Revised: T

Original: [Equation]
Revised: [Equation]

Original: 14) where we introducethe deﬁnition of α t = α (cid:112) d/t and Lemma A.2 in inequality 1 .
Revised: (13) where we introduce Lemma A.2 in inequality 1 .

Original: Besides, we get 2 through Cauchy-Schwarz inequality.
Revised: Besides, we get 2 through Cauchy-Schwarz inequality.

Original: With similar proof techniques, we have T 3 satisﬁed
Revised: With similar proof techniques, we have T 3 satisﬁed

Original: T (cid:88)t
Revised: T (cid:88)

Original: [Equation]
Revised: [Equation]

Original: [Equation] i [Equation]
Revised: i =

Original: [Equation]
Revised: [Equation]

Original: i
Revised: t

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:118)(cid:117)(cid:117)(cid:116)T (cid:88) t =
Revised: (cid:118)(cid:117)(cid:117)(cid:116)

Original: [Equation]
Revised: [Equation]

Original:  Since Equation (15) is sufﬁcient for our regret convergence analysis [Equation] we have not tried to optimize it further.
Revised: [Equation] Since Equation (14) is sufﬁcient for our regret convergence analysis, we have not tried to optimize it further.

Original: Then, for T 4 , we have
Revised: Then, for T 4 , we have

Original: [Equation]
Revised: [Equation]

Original: where 1 follows from the deﬁnition of β 1 ,t and α t , and 2 establishes because of the geometric series summation rule.
Revised: where 1 follows from the deﬁnition of β 1 ,t and α t , and 2 establishes because of the geometric series summation rule.

Original: As a result, submitting Equation (13), Equation (14), Equation (15) and
Revised: As a result, submitting Equation (12), Equation (13), Equation (14) and

Original: Equation (16) back into Equation (12), we have
Revised: Equation (15) back into Equation (11), we have

Original: [Equation]
Revised: [Equation]

Original: This completes the proof.
Revised: This completes the proof.

Original: A. P ROOF OF C OROLLARY 4. 
Revised: A. P ROOF OF C OROLLARY 4.

Original: Proof.
Revised: Proof.

Original: From Theorem 4.2, we have
Revised: From Theorem 4.2, we have

Original: [Equation]
Revised: [Equation]

Original: For the second term, we have
Revised: For the second term, we have

Original: [Equation]
Revised: [Equation]

Original: Combine the above inequality with Equation (18), we obtain
Revised: Combining the above inequality with Equation (17), we obtain

Original: [Equation]
Revised: [Equation]