Original: W IN : W EIGHT -D ECAY -I NTEGRATED N ESTEROV A CCELERATION FOR A DAPTIVE G RADIENT A LGORITHMS
Revised: W IN : W EIGHT -D ECAY -I NTEGRATED N ESTEROV A CCELERATION FOR A DAPTIVE G RADIENT A LGORITHMS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training deep networks on increasingly large-scale datasets is computationally challenging.
Revised: Training deep networks on increasingly large-scale datasets is computationally challenging.

Original: In this work, we explore the problem of “ how to accelerate the convergence of adaptive gradient algorithms in a general manner ”, and aim at providing practical insights to boost the training efﬁciency.
Revised: In this work, we explore the problem of “ how to accelerate the convergence of adaptive gradient algorithms in a general manner ”, and aim at providing practical insights to boost the training efﬁciency.

Original: To this end, we propose an effective and general Weight-decay-Integrated Nesterov acceleration (Win) for adaptive algorithms to enhance their convergence speed.
Revised: To this end, we propose an effective and general Weight-decay-Integrated Nesterov acceleration (Win) for adaptive algorithms to enhance their convergence speed.

Original: Taking AdamW and
Revised: Taking AdamW and

Original: Adam as examples, we minimize a dynamical loss per iteration which combines the vanilla training loss and a dynamic regularizer inspired by proximal point method (PPM) to improve the convexity of the problem.
Revised: Adam as examples, we minimize a dynamical loss per iteration which combines the vanilla training loss and a dynamic regularizer inspired by proximal point method (PPM) to improve the convexity of the problem.

Original: To introduce Nesterovalike-acceleration into AdamW and Adam, we respectively use the ﬁrst- and second-order Taylor approximations of vanilla loss to update the variable twice while ﬁxing the above dynamic regularization brought by PPM.
Revised: To introduce Nesterovalike-acceleration into AdamW and Adam, we respectively use the ﬁrst- and second-order Taylor approximations of vanilla loss to update the variable twice while ﬁxing the above dynamic regularization brought by PPM.

Original: In this way, we arrive at our Win acceleration (like Nesterov acceleration) for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.
Revised: In this way, we arrive at our Win acceleration (like Nesterov acceleration) for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.

Original: Next, we extend this Win acceleration to LAMB and SGD.
Revised: Next, we extend this Win acceleration to LAMB and SGD.

Original: Our transparent acceleration derivation could provide insights for other accelerated methods and their integration into adaptive algorithms.
Revised: Our transparent acceleration derivation could provide insights for other accelerated methods and their integration into adaptive algorithms.

Original: Besides, we prove the convergence of Win-accelerated adaptive algorithms and justify their convergence superiority over their non-accelerated counterparts by taking AdamW and Adam as examples.
Revised: Besides, we prove the convergence of Win-accelerated adaptive algorithms and justify their convergence superiority over their non-accelerated counterparts by taking AdamW and Adam as examples.

Original: Experimental results testify the faster convergence speed and superior performance of our Win-accelerated AdamW, Adam,
Revised: Experimental results testify the faster convergence speed and superior performance of our Win-accelerated AdamW, Adam,

Original: LAMB and SGD over their non-accelerated counterparts on vision classiﬁcation tasks and language modeling tasks with both CNN and Transformer backbones.
Revised: LAMB and SGD over their non-accelerated counterparts on vision classiﬁcation tasks and language modeling tasks with both CNN and Transformer backbones.

Original: We hope Win acceleration shall be a default acceleration option for all popular optimizers in deep learning community to improve the training efﬁciency.
Revised: We hope Win acceleration shall be a default acceleration option for all popular optimizers in deep learning community to improve the training efﬁciency.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) are effective to model realistic data and have been successfully applied to many applications, e.g. image classiﬁcation (He et al., 2016) and speech recognition (Sainath et al., 2013).
Revised: Deep neural networks (DNNs) are effective to model realistic data and have been successfully applied to many applications, e.g. image classiﬁcation (He et al., 2016) and speech recognition (Sainath et al., 2013).

Original: Typically, their training models can be formulated as a nonconvex problem:
Revised: Typically, their training models can be formulated as a nonconvex problem:

Original: [Equation]
Revised: [Equation]

Original: where z ∈ R d is the model parameter; sample ζ is drawn from a data distribution D ; the loss f is differentiable; λ is a constant.
Revised: where z ∈ R d is the model parameter; sample ζ is drawn from a data distribution D ; the loss f is differentiable; λ is a constant.

Original: Though many algorithms, e.g. gradient descent (Cauchy et al., 1847) and variance-reduced algorithms (Johnson & Zhang, 2013), can solve problem (1), SGD (Robbins & Monro, 1951) uses the compositional structure in (1) to efﬁciently estimate gradient via minibatch data, and has become a dominant algorithm to train DNNs in practice because of its higher efﬁciency and effectiveness.
Revised: Though many algorithms, e.g. gradient descent (Cauchy et al., 1847) and variance-reduced algorithms (Johnson & Zhang, 2013), can solve problem (1), SGD (Robbins & Monro, 1951) uses the compositional structure in (1) to efﬁciently estimate gradient via minibatch data, and has become a dominant algorithm to train DNNs in practice because of its higher efﬁciency and effectiveness.

Original: However, on sparse data or ill-conditioned problems, SGD suffers from slow convergence speed (Kingma & Ba, 2014), as it scales the gradient uniformly in all parameter coordinate and ignores the problem properties on each coordinate.
Revised: However, on sparse data or ill-conditioned problems, SGD suffers from slow convergence speed (Kingma & Ba, 2014), as it scales the gradient uniformly in all parameter coordinate and ignores the problem properties on each coordinate.

Original: To resolve this issue, recent work has proposed a variety of adaptive methods, e.g. Adam (Kingma & Ba, 2014) and AdamW (Loshchilov &
Revised: To resolve this issue, recent work has proposed a variety of adaptive methods, e.g. Adam (Kingma & Ba, 2014) and AdamW (Loshchilov &

Original: Hutter, 2018), that scale each gradient coordinate according to the current geometry curvature of the loss F ( z ) .
Revised: Hutter, 2018), that scale each gradient coordinate according to the current geometry curvature of the loss F ( z ) .

Original: This coordinate-wise scaling greatly accelerates the optimization convergence and helps them, e.g. Adam and AdamW, become more popular in DNN training, especially for transformers.
Revised: This coordinate-wise scaling greatly accelerates the optimization convergence and helps them, e.g. Adam and AdamW, become more popular in DNN training, especially for transformers.

Original: Unfortunately, along with the increasing scale of both datasets and models, efﬁcient DNN training even with SGD or adaptive algorithms has become very challenging.
Revised: Unfortunately, along with the increasing scale of both datasets and models, efﬁcient DNN training even with SGD or adaptive algorithms has become very challenging.

Original: In this work, we are particularly interested in the problem of “ how to accelerate the convergence of adaptive algorithms in a general manner ” because of their dominant popularity across many DNNs.
Revised: In this work, we are particularly interested in the problem of “ how to accelerate the convergence of adaptive algorithms in a general manner ” because of their dominant popularity across many DNNs.

Original: Heavy ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are widely used in SGD but are rarely studied in adaptive algorithms.
Revised: Heavy ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are widely used in SGD but are rarely studied in adaptive algorithms.

Original: Among the very few, NAdam (Dozat, 2016) simpliﬁes Nesterov acceleration to estimate the ﬁrst moment of gradient in Adam while totally ignoring the second-order moments, which is not exact Nesterov acceleration and may not inherit its full acceleration merit.
Revised: Among the very few, NAdam (Dozat, 2016) simpliﬁes Nesterov acceleration to estimate the ﬁrst moment of gradient in Adam while totally ignoring the second-order moments, which is not exact Nesterov acceleration and may not inherit its full acceleration merit.

Original: Contributions:
Revised: Contributions:

Original: In this work, based on a recent Nesterov-type acceleration formulation (Nesterov et al., 2018) and proximal point method (PPM) (Moreau, 1965), we propose a new Weight-decayIntegrated Nesterov acceleration (Win for short) to accelerate adaptive algorithms, and also further analyze the convergence of Win-accelerated adaptive algorithms to justify their convergence superiority by taking AdamW and Adam as examples.
Revised: In this work, based on a recent Nesterov-type acceleration formulation (Nesterov et al., 2018) and proximal point method (PPM) (Moreau, 1965), we propose a new Weight-decayIntegrated Nesterov acceleration (Win for short) to accelerate adaptive algorithms, and also further analyze the convergence of Win-accelerated adaptive algorithms to justify their convergence superiority by taking AdamW and Adam as examples.

Original: Our main contributions are highlighted below.
Revised: Our main contributions are highlighted below.

Original: Firstly, we use PPM to rigorously derive our Win acceleration for accelerating adaptive algorithms.
Revised: Firstly, we use PPM to rigorously derive our Win acceleration for accelerating adaptive algorithms.

Original: By taking AdamW and Adam as examples, at the k -iteration, we follow PPM spirit and minimize a dynamically regularized loss F ( z )+ 12 η k (cid:107)
Revised: By taking AdamW and Adam as examples, at the k -iteration, we follow PPM spirit and minimize a dynamically regularized loss F ( z )+ 12 η k (cid:107)

Original: z − x k (cid:107) 2 √ v k
Revised: z − x k (cid:107) 2 √ v k

Original: + ν with the second-order gradient moment v k and the stabilizing constant ν in AdamW and Adam.
Revised: + ν with the second-order gradient moment v k and the stabilizing constant ν in AdamW and Adam.

Original: Then to introduce Nesterov-alike acceleration and also make the problem solvable iteratively, we respectively approximate F ( z ) by its ﬁrstand second-order Taylor expansions to update the variable z twice while always ﬁxing the above dynamic regularization and also an extra regularizer 12 η k (cid:107) z (cid:107) 2 √ v k + ν induced by the weight decay in
Revised: Then to introduce Nesterov-alike acceleration and also make the problem solvable iteratively, we respectively approximate F ( z ) by its ﬁrstand second-order Taylor expansions to update the variable z twice while always ﬁxing the above dynamic regularization and also an extra regularizer 12 η k (cid:107) z (cid:107) 2 √ v k + ν induced by the weight decay in

Original: AdamW. As a result, we achieve at our Win acceleration, a Nesterov-alike acceleration, for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.
Revised: AdamW. As a result, we achieve at our Win acceleration, a Nesterov-alike acceleration, for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.

Original: Then we extend this Win acceleration to LAMB (You et al., 2019) and SGD.
Revised: Then we extend this Win acceleration to LAMB (You et al., 2019) and SGD.

Original: The above acceleration derivation is transparent and general which could motivate other accelerations and provide examples to introduce other accelerations into adaptive algorithms.
Revised: The above acceleration derivation is transparent and general which could motivate other accelerations and provide examples to introduce other accelerations into adaptive algorithms.

Original: Secondly, we prove the convergence of our Win-accelerated AdamW and Adam.
Revised: Secondly, we prove the convergence of our Win-accelerated AdamW and Adam.

Original: For both, to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, their stochastic gradient complexity is O (cid:0) c 2 .
Revised: For both, to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, their stochastic gradient complexity is O (cid:0) c 2 .

Original: 5 ∞ ν 0 .
Revised: 5 ∞ ν 0 .

Original: 5 (cid:15) 4 (cid:1) and matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same conditions, where c ∞ upper bounds the (cid:96) ∞ norm of stochastic gradient.
Revised: 5 (cid:15) 4 (cid:1) and matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same conditions, where c ∞ upper bounds the (cid:96) ∞ norm of stochastic gradient.

Original: Moreover, this complexity improves a factor O (cid:0) 1 ν 0 .
Revised: Moreover, this complexity improves a factor O (cid:0) 1 ν 0 .

Original: 75 (cid:1) over the complexity O ( c 2 .
Revised: 75 (cid:1) over the complexity O ( c 2 .

Original: 5 ∞ ν 1 .
Revised: 5 ∞ ν 1 .

Original: 25 (cid:15) 4 ) of Adam-type optimizers in (Zhou et al., 2018; Guo et al., 2021), e.g. Adam, AdaGrad (Duchi et al., 2011), AdaBound (Luo et al., 2018), since ν is often small, e.g. O ( ν ) =
Revised: 25 (cid:15) 4 ) of Adam-type optimizers in (Zhou et al., 2018; Guo et al., 2021), e.g. Adam, AdaGrad (Duchi et al., 2011), AdaBound (Luo et al., 2018), since ν is often small, e.g. O ( ν ) =

Original: 10 − 4 in practice.
Revised: 10 − 4 in practice.

Original: Indeed, Win-accelerated Adam and AdamW also enjoy superior complexity than other Adam variants, e.g. Adabelief (Zhuang et al., 2020) with compelxity O ( c 62 ν 2 (cid:15) 4 ) and RMSProp (Zhou et al., 2018) with comeplexity O ( c 0 .
Revised: Indeed, Win-accelerated Adam and AdamW also enjoy superior complexity than other Adam variants, e.g. Adabelief (Zhuang et al., 2020) with compelxity O ( c 62 ν 2 (cid:15) 4 ) and RMSProp (Zhou et al., 2018) with comeplexity O ( c 0 .

Original: 5 ∞ d 0 .
Revised: 5 ∞ d 0 .

Original: 5 ν(cid:15) 4 ) , especially on over-parameterized networks, where c 2 is the maximum (cid:96) 2 -norm of stochastic gradient.
Revised: 5 ν(cid:15) 4 ) , especially on over-parameterized networks, where c 2 is the maximum (cid:96) 2 -norm of stochastic gradient.

Original: Finally, experimental results on both vision classiﬁcation tasks and language modeling tasks show that our Win-accelerated algorithms, i.e. accelerated AdamW, Adam, LAMB and SGD, can accelerate the convergence speed and also improve the performance of their corresponding non-accelerated counterparts by a remarkable margin on both CNN and transformer architectures.
Revised: Finally, experimental results on both vision classiﬁcation tasks and language modeling tasks show that our Win-accelerated algorithms, i.e. accelerated AdamW, Adam, LAMB and SGD, can accelerate the convergence speed and also improve the performance of their corresponding non-accelerated counterparts by a remarkable margin on both CNN and transformer architectures.

Original: All these results show the strong compatibility, generalization and superiority of our acceleration technique.
Revised: All these results show the strong compatibility, generalization and superiority of our acceleration technique.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In the context of deep learning, when considering efﬁciency and generalization, one often prefers to adopt SGD and adaptive gradient algorithms (Bernstein et al., 2020), e.g. Adam, instead of other algorithms, e.g. variance-reduced algorithms (Johnson & Zhang, 2013), to solve problem (1).
Revised: In the context of deep learning, when considering efﬁciency and generalization, one often prefers to adopt SGD and adaptive gradient algorithms (Bernstein et al., 2020), e.g. Adam, instead of other algorithms, e.g. variance-reduced algorithms (Johnson & Zhang, 2013), to solve problem (1).

Original: But, in practice, adaptive algorithms often suffer from inferior generalization performance than SGD.
Revised: But, in practice, adaptive algorithms often suffer from inferior generalization performance than SGD.

Original: To solve this issue, AdamW (Loshchilov & Hutter, 2018) proposes a decoupled weight decay which introduces an (cid:96) 2 -alike regularization into Adam to decay network weight iteratively, and its effectiveness is widely validated on ViTs (Touvron et al., 2021) and CNNs (Touvron et al., 2021).
Revised: To solve this issue, AdamW (Loshchilov & Hutter, 2018) proposes a decoupled weight decay which introduces an (cid:96) 2 -alike regularization into Adam to decay network weight iteratively, and its effectiveness is widely validated on ViTs (Touvron et al., 2021) and CNNs (Touvron et al., 2021).

Original: Later,
Revised: Later,

Original: LAMB (You et al., 2019) scales the update in AdamW to the weight magnitude for getting rid of too large or small update, but suffers from unsatisfactory performance on small batch.
Revised: LAMB (You et al., 2019) scales the update in AdamW to the weight magnitude for getting rid of too large or small update, but suffers from unsatisfactory performance on small batch.

Original: In this work, we aim to design a general acceleration approach to accelerate these adaptive algorithms.
Revised: In this work, we aim to design a general acceleration approach to accelerate these adaptive algorithms.

Original: Heavy-ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are two classical acceleration techniques, and their effectiveness in SGD is well testiﬁed.
Revised: Heavy-ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are two classical acceleration techniques, and their effectiveness in SGD is well testiﬁed.

Original: Later, NAdam (Dozat, 2016) integrates Nesterov acceleration into the ﬁrst-order gradient moment estimation but ignores the second-order gradient moments which harms the acceleration effect.
Revised: Later, NAdam (Dozat, 2016) integrates Nesterov acceleration into the ﬁrst-order gradient moment estimation but ignores the second-order gradient moments which harms the acceleration effect.

Original: Recently, for full gra- dient decent algorithm, a new general Nesterov-type acceleration (Nesterov et al., 2018) directly interpolates two variables to look ahead for correction, and is more ﬂexible than vanilla Nesterov acceleration (Nesterov, 2003) which interpolates the variable and gradient.
Revised: Recently, for full gra- dient decent algorithm, a new general Nesterov-type acceleration (Nesterov et al., 2018) directly interpolates two variables to look ahead for correction, and is more ﬂexible than vanilla Nesterov acceleration (Nesterov, 2003) which interpolates the variable and gradient.

Original: See more discussion in
Revised: See more discussion in

Original: Sec. 3.2.
Revised: Sec. 3.2.

Original: In this work, we use proximal point method to introduce this new acceleration into adaptive algorithms by a rigorous and transparent derivation and also necessary adaptations (or tailors).
Revised: In this work, we use proximal point method to introduce this new acceleration into adaptive algorithms by a rigorous and transparent derivation and also necessary adaptations (or tailors).

Original:  W EIGHT - DECAY -I NTEGRATED N ESTEROV A CCELERATION
Revised:  W EIGHT - DECAY -I NTEGRATED N ESTEROV A CCELERATION

Original: To accelerate full gradient descent algorithm, given a full gradient ∇ F ( z k ) of problem (1) at the k -th iteration, Nesterov-type acceleration (Nesterov et al., 2018) generally uses a conservative step η k and a reckless step ¯ η k to update two sequences x k +1 and y k +1 respectively, and then linearly combines them to update the variable z k +1 of the problem.
Revised: To accelerate full gradient descent algorithm, given a full gradient ∇ F ( z k ) of problem (1) at the k -th iteration, Nesterov-type acceleration (Nesterov et al., 2018) generally uses a conservative step η k and a reckless step ¯ η k to update two sequences x k +1 and y k +1 respectively, and then linearly combines them to update the variable z k +1 of the problem.

Original: Similar formulations are also observed and proved in many recent works, e.g. (Allen-Zhu & Orecchia, 2014; Bansal & Gupta, 2019; Ahn & Sra, 2022).
Revised: Similar formulations are also observed and proved in many recent works, e.g. (Allen-Zhu & Orecchia, 2014; Bansal & Gupta, 2019; Ahn & Sra, 2022).

Original: In general, their acceleration formulation can be formally formulated as
Revised: In general, their acceleration formulation can be formally formulated as

Original: [Equation]
Revised: [Equation]

Original: This acceleration enjoys provably faster convergence rate for full gradient descent method on convex problems (Nesterov et al., 2018), and is then empirically validated in many convex and nonconvex cases, e.g. (Wilson et al., 2017; Nado et al., 2021).
Revised: This acceleration enjoys provably faster convergence rate for full gradient descent method on convex problems (Nesterov et al., 2018), and is then empirically validated in many convex and nonconvex cases, e.g. (Wilson et al., 2017; Nado et al., 2021).

Original: Despite its effectiveness, such acceleration is rarely explored in adaptive gradient algorithms, especially for network training.
Revised: Despite its effectiveness, such acceleration is rarely explored in adaptive gradient algorithms, especially for network training.

Original: In deterministic optimization setting, another widely used optimization-stabilizing and acceleration approach is proximal point method (PPM)
Revised: In deterministic optimization setting, another widely used optimization-stabilizing and acceleration approach is proximal point method (PPM)

Original: (Moreau, 1965; Rockafellar, 1976).
Revised: (Moreau, 1965; Rockafellar, 1976).

Original: At the k -th iteration, PPM optimizes an (cid:96) 2 -regularized loss F ( z )+ 12 η k (cid:107) z − z k − 1 (cid:107) 22 instead of the vanilla loss F ( z ) .
Revised: At the k -th iteration, PPM optimizes an (cid:96) 2 -regularized loss F ( z )+ 12 η k (cid:107) z − z k − 1 (cid:107) 22 instead of the vanilla loss F ( z ) .

Original: This small change enhances the convexity of the problem, which accelerates and stabilizes optimization process (Kim et al., 2022; Zhou et al., 2021).
Revised: This small change enhances the convexity of the problem, which accelerates and stabilizes optimization process (Kim et al., 2022; Zhou et al., 2021).

Original: To make the (cid:96) 2 -regularized problem solvable iteratively,
Revised: To make the (cid:96) 2 -regularized problem solvable iteratively,

Original: PPM approximates the loss F ( z ) by its ﬁrst- or second-order Taylor expansion so that each iteration has a close-form solution (see below).
Revised: PPM approximates the loss F ( z ) by its ﬁrst- or second-order Taylor expansion so that each iteration has a close-form solution (see below).

Original: In the following, we borrow the PPM idea to induce a Weightdecay-Integrated Nesterov acceleration (Win) for adaptive algorithms by using AdamW and Adam as examples in Sec. 3.1, and extend this acceleration technique to LAMB and SGD in Sec. 3.2.
Revised: In the following, we borrow the PPM idea to induce a Weightdecay-Integrated Nesterov acceleration (Win) for adaptive algorithms by using AdamW and Adam as examples in Sec. 3.1, and extend this acceleration technique to LAMB and SGD in Sec. 3.2.

Original: W IN -A CCELERATED A DAM W AND A DAM
Revised: W IN -A CCELERATED A DAM W AND A DAM

Original: To begin with, following most adaptive gradient algorithms, e.g. Adam and AdamW, we estimate the ﬁrst- and second-order moments m k and v k of gradient as follows:
Revised: To begin with, following most adaptive gradient algorithms, e.g. Adam and AdamW, we estimate the ﬁrst- and second-order moments m k and v k of gradient as follows:

Original: [Equation]
Revised: [Equation]

Original: where g k is the average gradient on a minibatch data of size b , β 1 ∈ [0 , 1] and β 2 ∈ [0 , 1] .
Revised: where g k is the average gradient on a minibatch data of size b , β 1 ∈ [0 , 1] and β 2 ∈ [0 , 1] .

Original: For the initialization, we set m 0 = g 0 , v 0 = g 20 .
Revised: For the initialization, we set m 0 = g 0 , v 0 = g 20 .

Original: For brevity, with a small scaler ν > 0 , we deﬁne
Revised: For brevity, with a small scaler ν > 0 , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then following the spirit of PPM, at the k -th iteration, we minimize a regularized loss F ( x )+ 12 η k (cid:107) x − x k (cid:107) 2 s k .
Revised: Then following the spirit of PPM, at the k -th iteration, we minimize a regularized loss F ( x )+ 12 η k (cid:107) x − x k (cid:107) 2 s k .

Original: Here we use the regularizer (cid:107) x − x k (cid:107) 2 s k instead of the (cid:96) 2 -regularization (cid:107) x − x k (cid:107) 22 , since 1) this new regularization allows us to handle adaptive algorithms as shown below Eqn.
Revised: Here we use the regularizer (cid:107) x − x k (cid:107) 2 s k instead of the (cid:96) 2 -regularization (cid:107) x − x k (cid:107) 22 , since 1) this new regularization allows us to handle adaptive algorithms as shown below Eqn.

Original: (5), and 2) it also helps increase the convexity of the problem and actually further considers the different sharpness properties of each coordinates of the loss because of different elements in s k , thus also speeding up the convergence.
Revised: (5), and 2) it also helps increase the convexity of the problem and actually further considers the different sharpness properties of each coordinates of the loss because of different elements in s k , thus also speeding up the convergence.

Original: To make the problem solvable iteratively, we approximate the vanilla loss F ( z ) by its ﬁrst-order Taylor expansion at the point z k and update x k +1 as
Revised: To make the problem solvable iteratively, we approximate the vanilla loss F ( z ) by its ﬁrst-order Taylor expansion at the point z k and update x k +1 as

Original: [Equation]
Revised: [Equation]

Original: where (cid:107) x (cid:107) s k
Revised: where (cid:107) x (cid:107) s k

Original: = (cid:112) (cid:104) x , s k ∗ x (cid:105) with an element-wise product operation ∗ , m k is used to approximate the full gradient ∇ F ( z k ) for Taylor expansion.
Revised: = (cid:112) (cid:104) x , s k ∗ x (cid:105) with an element-wise product operation ∗ , m k is used to approximate the full gradient ∇ F ( z k ) for Taylor expansion.

Original: We add a small regularization λ 2 (cid:107) x (cid:107) 2 s k , since 1) it can largely improve the generalization performance in practice (Loshchilov & Hutter, 2018; Touvron et al., 2021); 2) it allows us to derive Adam ( λ =0 ) and AdamW ( λ> 0 ).
Revised: We add a small regularization λ 2 (cid:107) x (cid:107) 2 s k , since 1) it can largely improve the generalization performance in practice (Loshchilov & Hutter, 2018; Touvron et al., 2021); 2) it allows us to derive Adam ( λ =0 ) and AdamW ( λ> 0 ).

Original: Speciﬁcally, when λ =0 , the updating (5) becomes the exact Adam algorithm.
Revised: Speciﬁcally, when λ =0 , the updating (5) becomes the exact Adam algorithm.

Original: If λ> 0 , the updating (5) is an approximation to the updating rule x k +1 =(1 − λη k ) x
Revised: If λ> 0 , the updating (5) is an approximation to the updating rule x k +1 =(1 − λη k ) x

Original: k − η k u k of AdamW.
Revised: k − η k u k of AdamW.

Original: This is because consider λη k is small in practice, we can approximate (1+ λη k ) − 1 = 1 − λη k
Revised: This is because consider λη k is small in practice, we can approximate (1+ λη k ) − 1 = 1 − λη k

Original: + O ( λ 2 η 2 k ) and thus
Revised: + O ( λ 2 η 2 k ) and thus

Original: [Equation]
Revised: [Equation]

Original: − λη k + O ( λ 2 η 2 k )] x k − [ η k −O ( λη 2 k )
Revised: − λη k + O ( λ 2 η 2 k )] x k − [ η k −O ( λη 2 k )

Original: + O ( λ 3 η 3 k )]
Revised: + O ( λ 3 η 3 k )]

Original: u k which becomes AdamW by ignoring the ignorable terms related to O ( η 2 k ) or O ( η 3 k ) .
Revised: u k which becomes AdamW by ignoring the ignorable terms related to O ( η 2 k ) or O ( η 3 k ) .

Original: This is also one reason that we adopt the regularizer (cid:107) x − x k (cid:107) 2 s k in (5) instead of the (cid:96) 2 -regularization in PPM, since we can ﬂexibly derive Adam and AdamW.
Revised: This is also one reason that we adopt the regularizer (cid:107) x − x k (cid:107) 2 s k in (5) instead of the (cid:96) 2 -regularization in PPM, since we can ﬂexibly derive Adam and AdamW.

Original: Input: initialization x 0 =
Revised: Input: initialization x 0 =

Original: z 

Revised: z

Original: = 0 , step size { ( η k , ¯ η k ) } Tk =0 , moment parameters { β 1 , β 2 } .
Revised: = 0 , step size { ( η k , ¯ η k ) } Tk =0 , moment parameters { β 1 , β 2 } .

Original: Output: (¯ x , ¯ z ) uniformly seleted from { ( x k , z k ) } Tk =0 .
Revised: Output: (¯ x , ¯ z ) uniformly seleted from { ( x k , z k ) } Tk =0 .

Original: 1 while k < T do
Revised: 1 while k < T do

Original: [Equation]
Revised: [Equation]

Original: LAMB
Revised: LAMB

Original: [Equation]
Revised: [Equation]

Original: with τ
Revised: with τ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: 8 end while
Revised: 8 end while

Original: Similarly, we minimize a regularized loss objective F ( z ) + 12 η k (cid:107) z − x t +1 (cid:107) 

Revised: Similarly, we minimize a regularized loss objective F ( z ) + 12 η k (cid:107) z − x t +1 (cid:107)

Original: s k again, and further approximate F ( z ) by its second-order approximation F ( z k ) + (cid:104) m k ,
Revised: s k again, and further approximate F ( z ) by its second-order approximation F ( z k ) + (cid:104) m k ,

Original: z − z k (cid:105) + 12¯ η k (cid:107) z − z k (cid:107) 2 s k
Revised: z − z k (cid:105) + 12¯ η k (cid:107) z − z k (cid:107) 2 s k

Original: z k +1 = argmin z F ( z k )+ (cid:104) m k , z − z k (cid:105) + 1 2¯ η k (cid:107)
Revised: z k +1 = argmin z F ( z k )+ (cid:104) m k , z − z k (cid:105) + 1 2¯ η k (cid:107)

Original: z − z k
Revised: z − z k

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: η k
Revised: η k

Original: [Equation]
Revised: [Equation]

Original: where τ k = 1 η k +¯ η k
Revised: where τ k = 1 η k +¯ η k

Original: + λη k ¯ η k , m k is used to approximate ∇ F ( x k ) as guaranteed by Theorem 1 in
Revised: + λη k ¯ η k , m k is used to approximate ∇ F ( x k ) as guaranteed by Theorem 1 in

Original: Sec. 4, ¯ η k approximates the inverse of the local smoothness parameter of F ( z ) around z k .
Revised: Sec. 4, ¯ η k approximates the inverse of the local smoothness parameter of F ( z ) around z k .

Original: Here we use a regularizer (cid:107)
Revised: Here we use a regularizer (cid:107)

Original: z − x k +1 (cid:107) 2 s k with the latest update x k +1 instead of x k as an anchor point, since the latest update x k +1 could often provide better regularization for the concurrent optimization.
Revised: z − x k +1 (cid:107) 2 s k with the latest update x k +1 instead of x k as an anchor point, since the latest update x k +1 could often provide better regularization for the concurrent optimization.

Original: Now we have used PPM to rigorously derive our Win-accelerated AdamW and Adam in Eqns.
Revised: Now we have used PPM to rigorously derive our Win-accelerated AdamW and Adam in Eqns.

Original: (3), (5) and (6).
Revised: (3), (5) and (6).

Original: For more clarity, we summarize their algorithmic steps in Algorithm 1 in which we omit the bias-correction term for simplicity.
Revised: For more clarity, we summarize their algorithmic steps in Algorithm 1 in which we omit the bias-correction term for simplicity.

Original: When λ = 0 , it is Win-accelerated Adam; if λ > 0 , it gives Win-accelerated AdamW.
Revised: When λ = 0 , it is Win-accelerated Adam; if λ > 0 , it gives Win-accelerated AdamW.

Original: Generally, AdamW can greatly improve the generalization performance of Adam by simply adding a weight decay (i.e. the regularizer λ 2 (cid:107) · (cid:107) 2 s k ) into Adam as observed in many works, e.g. (Loshchilov & Hutter, 2018; Touvron et al., 2021).
Revised: Generally, AdamW can greatly improve the generalization performance of Adam by simply adding a weight decay (i.e. the regularizer λ 2 (cid:107) · (cid:107) 2 s k ) into Adam as observed in many works, e.g. (Loshchilov & Hutter, 2018; Touvron et al., 2021).

Original: Our Winacceleration is quit simple and efﬁcient, since our accelerated AdamW/Adam only adds one extra simple algorithmic step, i.e. the seventh step in Algorithm 1, on vanilla AdamW/Adam, and brings negligible extra computational overhead into vanilla optimizer, e.g. about 2% extra training time per iteration on AdamW evaluated on ViT-small and ViT-base.
Revised: Our Winacceleration is quit simple and efﬁcient, since our accelerated AdamW/Adam only adds one extra simple algorithmic step, i.e. the seventh step in Algorithm 1, on vanilla AdamW/Adam, and brings negligible extra computational overhead into vanilla optimizer, e.g. about 2% extra training time per iteration on AdamW evaluated on ViT-small and ViT-base.

Original: Moreover, for the only extra hyperparameter, the reckless step ¯ η k , in Algorithm 1 over AdamW/Adam, we always set it 2 × larger than the conservative step η k for all iterations, i.e. ¯ η k =2 η k , working well in our all experiments.
Revised: Moreover, for the only extra hyperparameter, the reckless step ¯ η k , in Algorithm 1 over AdamW/Adam, we always set it 2 × larger than the conservative step η k for all iterations, i.e. ¯ η k =2 η k , working well in our all experiments.

Original: Now we discuss the relations between Nesterov-type acceleration (2) and our Win acceleration (6).
Revised: Now we discuss the relations between Nesterov-type acceleration (2) and our Win acceleration (6).

Original: For comparison, we introduce a virtual sequence y k
Revised: For comparison, we introduce a virtual sequence y k

Original: +

Revised: +

Original: =
Revised: =

Original: z k − ¯ η k u k in Win, and rewrite (6) as
Revised: z k − ¯ η k u k in Win, and rewrite (6) as

Original: [Equation]
Revised: [Equation]

Original: where u k is deﬁned in (4).
Revised: where u k is deﬁned in (4).

Original: By comparing Nesterov-type acceleration (2) with our Win acceleration (7), one can observe some similarity and also differences as well.
Revised: By comparing Nesterov-type acceleration (2) with our Win acceleration (7), one can observe some similarity and also differences as well.

Original: For similarity, both acceleration use a conservative step η k and a reckless step ¯ η k to update x k +1 and y k +1 respectively, and then linearly combine x k +1 and y k +1 to obtain z k +1 .
Revised: For similarity, both acceleration use a conservative step η k and a reckless step ¯ η k to update x k +1 and y k +1 respectively, and then linearly combine x k +1 and y k +1 to obtain z k +1 .

Original: For the differences, the ﬁrst one is that Win has a weight-decay-alike factor
Revised: For the differences, the ﬁrst one is that Win has a weight-decay-alike factor

Original: [Equation]
Revised: [Equation]

Original: k in (7) which slightly decays the variable x k like AdamW and also the update u k , while Nesterov acceleration has no such important factor.
Revised: k in (7) which slightly decays the variable x k like AdamW and also the update u k , while Nesterov acceleration has no such important factor.

Original: Note, this weight decay can greatly beneﬁt generalization in practice as shown in many works, e.g. (Loshchilov & Hutter, 2018;
Revised: Note, this weight decay can greatly beneﬁt generalization in practice as shown in many works, e.g. (Loshchilov & Hutter, 2018;

Original: Touvron et al., 2021; Liu et al., 2021).
Revised: Touvron et al., 2021; Liu et al., 2021).

Original: Another difference is that for almost all acceleration techniques, including Nesterov-type acceleration in (2), the sum of the linear combination factors ( e.g. ρ k and 1 − ρ k in (2)) is always one.
Revised: Another difference is that for almost all acceleration techniques, including Nesterov-type acceleration in (2), the sum of the linear combination factors ( e.g. ρ k and 1 − ρ k in (2)) is always one.

Original: In contrast, in Eqn.
Revised: In contrast, in Eqn.

Original: (7), Win uses ¯ η k τ k + η
Revised: (7), Win uses ¯ η k τ k + η

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: when λ > 0 , which indeed gives a second weigh decay.
Revised: when λ > 0 , which indeed gives a second weigh decay.

Original: Since these two differences are caused by the weight decay, we call our acceleration “weight-decay-integrated Nesterov acceleration” (Win).
Revised: Since these two differences are caused by the weight decay, we call our acceleration “weight-decay-integrated Nesterov acceleration” (Win).

Original: 3. E XTENSION TO LAMB AND SGD
Revised: 3. E XTENSION TO LAMB AND SGD

Original: Here we generalize Win acceleration to LAMB (You et al., 2019) and SGD (Robbins & Monro, 1951).
Revised: Here we generalize Win acceleration to LAMB (You et al., 2019) and SGD (Robbins & Monro, 1951).

Original: For LAMB, it scales the update u k of AdamW in Eqn.
Revised: For LAMB, it scales the update u k of AdamW in Eqn.

Original: (4) so that u k is at the same magnitude of the network weight x k .
Revised: (4) so that u k is at the same magnitude of the network weight x k .

Original: That is, it changes the update rule x k +

Revised: That is, it changes the update rule x k +

Original: =
Revised: =

Original: (1 − λη k ) x
Revised: (1 − λη k ) x

Original: k − η k m k / s k in
Revised: k − η k m k / s k in

Original: AdamW to x k +

Revised: AdamW to x k +

Original: =
Revised: =

Original: x
Revised: x

Original: k − η k (cid:107) x k (cid:107) 2 (cid:107) r k + λ x k (cid:107) 2 ( r k + λ x k ) where r k = m k / s k .
Revised: k − η k (cid:107) x k (cid:107) 2 (cid:107) r k + λ x k (cid:107) 2 ( r k + λ x k ) where r k = m k / s k .

Original: This modiﬁcation is to avoid too large or small update, improving optimization efﬁciency.
Revised: This modiﬁcation is to avoid too large or small update, improving optimization efﬁciency.

Original: To extend Win acceleration to
Revised: To extend Win acceleration to

Original: LAMB, we inherit this scaling spirit, and scale the update u k in (4) to the following one:
Revised: LAMB, we inherit this scaling spirit, and scale the update u k in (4) to the following one:

Original: [Equation]
Revised: [Equation]

Original: We scale m k / s k instead of ( m k / s k + λ x k ) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .
Revised: We scale m k / s k instead of ( m k / s k + λ x k ) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .

Original: Next, we can respectively follow Eqn.
Revised: Next, we can respectively follow Eqn.

Original: (5) and (6) to update the two sequences x k and z k .
Revised: (5) and (6) to update the two sequences x k and z k .

Original: See the detailed steps of Win-accelerated LAMB in Algorithm 1, and the detailed comparison between LAMB and Win-accelerated LAMB in Appendix A.
Revised: See the detailed steps of Win-accelerated LAMB in Algorithm 1, and the detailed comparison between LAMB and Win-accelerated LAMB in Appendix A.

Original: For SGD, applying Win acceleration to it is quite direct.
Revised: For SGD, applying Win acceleration to it is quite direct.

Original: Speciﬁcally, the only algorithmic difference between SGD and AdamW on the (cid:96) 2 -regularized problems is that SGD has no second-order moment v k while AdamW has.
Revised: Speciﬁcally, the only algorithmic difference between SGD and AdamW on the (cid:96) 2 -regularized problems is that SGD has no second-order moment v k while AdamW has.

Original: So we can borrow the acceleration framework of AdamW in Sec.
Revised: So we can borrow the acceleration framework of AdamW in Sec.

Original: 3.1 to accelerate SGD by setting s k = 1 ∈ R d in Eqn.
Revised: 3.1 to accelerate SGD by setting s k = 1 ∈ R d in Eqn.

Original: (4), (5) and (6), and obtain WIN-accelerated SGD:
Revised: (4), (5) and (6), and obtain WIN-accelerated SGD:

Original: [Equation]
Revised: [Equation]

Original: ) where β (cid:48) 1 ∈ [0 , 1] is dampening parameter.
Revised: ) where β (cid:48) 1 ∈ [0 , 1] is dampening parameter.

Original: Here we slightly modify the moment m k to accord with the one used in Nesterov-accelerated SGD ( e.g. SGD-M in Pytorch) whose updating steps are
Revised: Here we slightly modify the moment m k to accord with the one used in Nesterov-accelerated SGD ( e.g. SGD-M in Pytorch) whose updating steps are

Original: [Equation]
Revised: [Equation]

Original: By comparing Win-accelerated SGD and SGD-M in (10), one can ﬁnd their big differences mainly caused by their different acceleration strategies and ways to handle weight decay.
Revised: By comparing Win-accelerated SGD and SGD-M in (10), one can ﬁnd their big differences mainly caused by their different acceleration strategies and ways to handle weight decay.

Original: Win-accelerated
Revised: Win-accelerated

Original: SGD is derived from PPM and a recently proposed acceleration (2), while SGD-M modiﬁes another previous Nesterov-type acceleration (Nesterov, 2003) (with formulation m k =
Revised: SGD is derived from PPM and a recently proposed acceleration (2), while SGD-M modiﬁes another previous Nesterov-type acceleration (Nesterov, 2003) (with formulation m k =

Original: β 1 m k − 1 − η k b (cid:80) bi =1 ∇ f
Revised: β 1 m k − 1 − η k b (cid:80) bi =1 ∇ f

Original: ( x k + η k m k − 1 ; ζ i ) and x k +

Revised: ( x k + η k m k − 1 ; ζ i ) and x k +

Original: = x k + m k ) to better train networks.
Revised: = x k + m k ) to better train networks.

Original: See more mechanisms of previous Nesterov acceleration and (10) in (Sutskever et al., 2013; Bengio et al., 2013).
Revised: See more mechanisms of previous Nesterov acceleration and (10) in (Sutskever et al., 2013; Bengio et al., 2013).

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: Here we investigate the convergence performance of Win-accelerated algorithms by taking accelerated AdamW, Adam and SGD as examples, as these algorithms are more preferably used in deep learning ﬁeld.
Revised: Here we investigate the convergence performance of Win-accelerated algorithms by taking accelerated AdamW, Adam and SGD as examples, as these algorithms are more preferably used in deep learning ﬁeld.

Original: Moreover, since we aim to accelerate deep network training which is highly nonconvex problems, we focus on analyzing nonconvex problems to accord with the practical setting.
Revised: Moreover, since we aim to accelerate deep network training which is highly nonconvex problems, we focus on analyzing nonconvex problems to accord with the practical setting.

Original: For analysis, we follow previous works on optimizer, e.g. (Kingma & Ba, 2014; Reddi et al., 2019;
Revised: For analysis, we follow previous works on optimizer, e.g. (Kingma & Ba, 2014; Reddi et al., 2019;

Original: Luo et al., 2019; Duchi et al., 2011; Xie et al., 2022), and introduce necessary assumptions.
Revised: Luo et al., 2019; Duchi et al., 2011; Xie et al., 2022), and introduce necessary assumptions.

Original: Assumption 1 ( L -smoothness) .
Revised: Assumption 1 ( L -smoothness) .

Original: We say a function f
Revised: We say a function f

Original: ( z , · ) to be L -smooth w.r.t.
Revised: ( z , · ) to be L -smooth w.r.t.

Original: z , if for ∀ z 1 , z 2 and ∀ ζ ∼ D , we have (cid:107)∇ f ( z 1 , ζ ) − ∇ f
Revised: z , if for ∀ z 1 , z 2 and ∀ ζ ∼ D , we have (cid:107)∇ f ( z 1 , ζ ) − ∇ f

Original: ( z 2 , ζ ) (cid:107) 2 ≤ L (cid:107) z 1 − z 2 (cid:107) 2 with a universal constant L .
Revised: ( z 2 , ζ ) (cid:107) 2 ≤ L (cid:107) z 1 − z 2 (cid:107) 2 with a universal constant L .

Original: Assumption 2 (Unbiased and bounded gradient estimation) .
Revised: Assumption 2 (Unbiased and bounded gradient estimation) .

Original: The gradient estimation g k is unbiased, i.e. for ∀ k , E [ g k ] = ∇ F ( z k ) , and its magnitude and variance are bounded, namely, for ∀ k , (cid:107) g k (cid:107) ∞ ≤ c ∞ and E [ (cid:107)∇ F ( z k ) − g k (cid:107) 2 ] ≤ σ with two universal constants c ∞ and σ .
Revised: The gradient estimation g k is unbiased, i.e. for ∀ k , E [ g k ] = ∇ F ( z k ) , and its magnitude and variance are bounded, namely, for ∀ k , (cid:107) g k (cid:107) ∞ ≤ c ∞ and E [ (cid:107)∇ F ( z k ) − g k (cid:107) 2 ] ≤ σ with two universal constants c ∞ and σ .

Original: Next, we ﬁrst deﬁne a dynamic function F k ( z ) at the k -th iteration which is real loss minimized by our algorithms.
Revised: Next, we ﬁrst deﬁne a dynamic function F k ( z ) at the k -th iteration which is real loss minimized by our algorithms.

Original: It combines the vanilla loss F ( z ) in (1) and a dynamic regularization λ 2 (cid:107) z (cid:107) 2 s k :
Revised: It combines the vanilla loss F ( z ) in (1) and a dynamic regularization λ 2 (cid:107) z (cid:107) 2 s k :

Original: [Equation]
Revised: [Equation]

Original: (11) where s k is given in (4).
Revised: (11) where s k is given in (4).

Original: To obtain (11), following PPM spirit and Eqn.
Revised: To obtain (11), following PPM spirit and Eqn.

Original: (5), one can approximate
Revised: (5), one can approximate

Original: F ( z ) by its ﬁrst-order Taylor expansion, and obtain Eqn.
Revised: F ( z ) by its ﬁrst-order Taylor expansion, and obtain Eqn.

Original: (5) with x replaced by z to update z k +

Revised: (5) with x replaced by z to update z k +

Original: =
Revised: =

Original: + λη k
Revised: + λη k

Original: ( z k − η k m k / s k ) .
Revised: ( z k − η k m k / s k ) .

Original: Since λη k is very small, one can follow the discussion below Eqn.
Revised: Since λη k is very small, one can follow the discussion below Eqn.

Original: (5) and approximate z k +1 as z k +

Revised: (5) and approximate z k +1 as z k +

Original: =
Revised: =

Original: (1 − λη k )
Revised: (1 − λη k )

Original: z k − η k m k / s k which becomes the update rule of
Revised: z k − η k m k / s k which becomes the update rule of

Original: AdamW. This is the reason that our analysis on Win-accelerated AdamW involves a dynamic loss
Revised: AdamW. This is the reason that our analysis on Win-accelerated AdamW involves a dynamic loss

Original: F k ( z ) in (11).
Revised: F k ( z ) in (11).

Original: Note, for Win-accelerated Adam ( λ =0 ), F k ( z ) degenerates to the vanilla loss F ( z ) .
Revised: Note, for Win-accelerated Adam ( λ =0 ), F k ( z ) degenerates to the vanilla loss F ( z ) .

Original: With these assumptions, we analyze the convergence behaviors of our accelerated algorithms on general nonconvex problems, and summarize our main results in Theorem 1 with proof in Appendix E.
Revised: With these assumptions, we analyze the convergence behaviors of our accelerated algorithms on general nonconvex problems, and summarize our main results in Theorem 1 with proof in Appendix E.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .
Revised: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .

Original: By setting ¯ η k = γη k , γ > 1 , η k = η ≤
Revised: By setting ¯ η k = γη k , γ > 1 , η k = η ≤

Original: [Equation]
Revised: [Equation]

Original: iterations with minibatch size b and ∆ =
Revised: iterations with minibatch size b and ∆ =

Original: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } T k =0 generated by Win-accelerated AdamW and Adam in Algorithm 1 satisﬁes the following four properties.
Revised: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } T k =0 generated by Win-accelerated AdamW and Adam in Algorithm 1 satisﬁes the following four properties.

Original: a)
Revised: a)

Original: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by
Revised: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: b) The gradient moment m k can well estimate the full gradient ∇ F ( x k ) and ∇ F ( z k ) :
Revised: b) The gradient moment m k can well estimate the full gradient ∇ F ( x k ) and ∇ F ( z k ) :

Original: [Equation]
Revised: [Equation]

Original: c)
Revised: c)

Original: The sequence { x k , z k } satisﬁes
Revised: The sequence { x k , z k } satisﬁes

Original: [Equation]
Revised: [Equation]

Original: d) The total stochastic gradient complexity to achieve the above three properties is O
Revised: d) The total stochastic gradient complexity to achieve the above three properties is O

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 guarantees the convergence of Win-accelerated AdamW and Adam in Algorithm 1 on nonconvex problems.
Revised: Theorem 1 guarantees the convergence of Win-accelerated AdamW and Adam in Algorithm 1 on nonconvex problems.

Original: When λ > 0 , Algorithm 1 corresponds to Win-accelerated AdamW; and if λ = 0 , it becomes Win-accelerated Adam.
Revised: When λ > 0 , Algorithm 1 corresponds to Win-accelerated AdamW; and if λ = 0 , it becomes Win-accelerated Adam.

Original: For both cases, Theorem 1 holds.
Revised: For both cases, Theorem 1 holds.

Original: Theorem 1 a) shows that by running at most T
Revised: Theorem 1 a) shows that by running at most T

Original: = O (cid:0)
Revised: = O (cid:0)

Original: [Equation]
Revised: [Equation]

Original: iterations, the average gradient
Revised: iterations, the average gradient

Original: [Equation]
Revised: [Equation]

Original: T
Revised: T

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: E
Revised: E

Original: [Equation]
Revised: [Equation]

Original: F k ( x k )
Revised: F k ( x k )

Original: [Equation]
Revised: [Equation]

Original: (cid:3) is upper bounded by (cid:15) 2 , guaranteeing the algorithmic convergence.
Revised: (cid:3) is upper bounded by (cid:15) 2 , guaranteeing the algorithmic convergence.

Original: Theorem 1 b) indicates the gradient moment m k can well estimate the full gradient ∇ F ( z k ) and also ∇ F ( x k ) because of their small distances, guaranteeing the good Taylor approximation used in Eqn.
Revised: Theorem 1 b) indicates the gradient moment m k can well estimate the full gradient ∇ F ( z k ) and also ∇ F ( x k ) because of their small distances, guaranteeing the good Taylor approximation used in Eqn.

Original: (5) and (6).
Revised: (5) and (6).

Original: Moreover, in
Revised: Moreover, in

Original: Theorem 1 c), one can ﬁnd that although Algorithm 1 uses a conservative step η k and a reckless step ¯ η k = γη k ( ∀ γ > 1) to update, the two sequences x k +1 and z k +1 can converge to each other, which could be the key for the good convergence behavior of both Win-accelerated AdamW and Adam.
Revised: Theorem 1 c), one can ﬁnd that although Algorithm 1 uses a conservative step η k and a reckless step ¯ η k = γη k ( ∀ γ > 1) to update, the two sequences x k +1 and z k +1 can converge to each other, which could be the key for the good convergence behavior of both Win-accelerated AdamW and Adam.

Original: Now we discuss the stochastic gradient complexity of Win-accelerated Adam and AdamW. Theorem 1 d) shows that to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, both Win-accelerated Adam and AdamW have the complexity O (cid:0) c 2 .
Revised: Now we discuss the stochastic gradient complexity of Win-accelerated Adam and AdamW. Theorem 1 d) shows that to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, both Win-accelerated Adam and AdamW have the complexity O (cid:0) c 2 .

Original: 5 ∞ σ 2 L ν 0 .
Revised: 5 ∞ σ 2 L ν 0 .

Original: 5 (cid:15) 4 (cid:1) which matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same Assumptions 1 and 2.
Revised: 5 (cid:15) 4 (cid:1) which matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same Assumptions 1 and 2.

Original: Our accelerated Adam and AdamW enjoy superior complexity over Adam-type optimizers, e.g. Adam, AdaGrad (Duchi et al., 2011),
Revised: Our accelerated Adam and AdamW enjoy superior complexity over Adam-type optimizers, e.g. Adam, AdaGrad (Duchi et al., 2011),

Original: AdaBound (Luo et al., 2018), whose previously best known complexity under the same assumptions is O (
Revised: AdaBound (Luo et al., 2018), whose previously best known complexity under the same assumptions is O (

Original: [Equation]
Revised: [Equation]

Original: 4 ) in (Zhou et al., 2018;
Revised: 4 ) in (Zhou et al., 2018;

Original: Chen et al., 2021; Guo et al., 2021).
Revised: Chen et al., 2021; Guo et al., 2021).

Original: By comparison, both accelerated Adam and AdamW improve their complexity by a factor O (cid:0) 1 ν 0 .
Revised: By comparison, both accelerated Adam and AdamW improve their complexity by a factor O (cid:0) 1 ν 0 .

Original: 75 (cid:1) , where ν is often very small in practice, e.g. O ( ν ) =
Revised: 75 (cid:1) , where ν is often very small in practice, e.g. O ( ν ) =

Original: 10 − 4 .
Revised: 10 − 4 .

Original: Since the convergence of AdamW has not been proved yet in the literatures, here we cannot directly compare with it.
Revised: Since the convergence of AdamW has not been proved yet in the literatures, here we cannot directly compare with it.

Original: Moreover, the complexity of Win-accelerated Adam and AdamW is also lower than O ( c 62 σ 2 L ν 2 (cid:15) 4 ) of Adabelief (Zhuang et al., 2020) and O ( c 0 .
Revised: Moreover, the complexity of Win-accelerated Adam and AdamW is also lower than O ( c 62 σ 2 L ν 2 (cid:15) 4 ) of Adabelief (Zhuang et al., 2020) and O ( c 0 .

Original: 5 ∞ d 0 .
Revised: 5 ∞ d 0 .

Original: 5 σ 2 L ν(cid:15)) of RMSProp (Tijmen & Geoffrey, 2012;
Revised: 5 σ 2 L ν(cid:15)) of RMSProp (Tijmen & Geoffrey, 2012;

Original: Zhou et al., 2018), especially on over-parameterized networks, since for a d -dimensional gradient, its (cid:96) 2 -norm upper bound c 2 is often much larger than the (cid:96) ∞ -norm c ∞ and can be √ d × larger for worse case.
Revised: Zhou et al., 2018), especially on over-parameterized networks, since for a d -dimensional gradient, its (cid:96) 2 -norm upper bound c 2 is often much larger than the (cid:96) ∞ -norm c ∞ and can be √ d × larger for worse case.

Original: Now we discuss the convergence performance of Win-accelerated SGD in Theorem 2.
Revised: Now we discuss the convergence performance of Win-accelerated SGD in Theorem 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .
Revised: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .

Original: By setting ¯ η k = γη k , γ > 1 , η k = η ≤ b(cid:15) 2 36( γ +1) 

Revised: By setting ¯ η k = γη k , γ > 1 , η k = η ≤ b(cid:15) 2 36( γ +1)

Original: Lσ 2 , β 1 ≤ b(cid:15) 2 6 σ 2 and β (cid:48) 1 = 1 − β 1 , after T
Revised: Lσ 2 , β 1 ≤ b(cid:15) 2 6 σ 2 and β (cid:48) 1 = 1 − β 1 , after T

Original: = O (cid:0) ∆
Revised: = O (cid:0) ∆

Original: σ 2 L ν 0 .
Revised: σ 2 L ν 0 .

Original: 5 b(cid:15) 4 (cid:1) iterations with minibatch size b and ∆ =
Revised: 5 b(cid:15) 4 (cid:1) iterations with minibatch size b and ∆ =

Original: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by Win-accelerated SGD in (9) satisﬁes the four properties in Theorem 1 with ν = c ∞ =
Revised: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by Win-accelerated SGD in (9) satisﬁes the four properties in Theorem 1 with ν = c ∞ =

Original: c = 1 and s k = 1 ∈ R d .
Revised: c = 1 and s k = 1 ∈ R d .

Original: See its proof in Appendix F. Theorem 2 also guarantees the convergence of Win-accelerated SGD.
Revised: See its proof in Appendix F. Theorem 2 also guarantees the convergence of Win-accelerated SGD.

Original: By using the hyper-parameter settings in Theorem 2, the sequence { ( x k , z k ) } Tk =0 generated by Winaccelerated SGD satisﬁes the four properties in Theorem 1 with ν = c ∞ =
Revised: By using the hyper-parameter settings in Theorem 2, the sequence { ( x k , z k ) } Tk =0 generated by Winaccelerated SGD satisﬁes the four properties in Theorem 1 with ν = c ∞ =

Original: c = 1 and s k = 1 .
Revised: c = 1 and s k = 1 .

Original: It shows the complexity
Revised: It shows the complexity

Original: O
Revised: O

Original: ( Lσ 2 (cid:15) 4 ) of Win-accelerated SGD which also matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same Assumptions 1 and 2.
Revised: ( Lσ 2 (cid:15) 4 ) of Win-accelerated SGD which also matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same Assumptions 1 and 2.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Here we investigate the performance of our accelerated algorithms on two representative tasks, including vision classiﬁcation tasks and also natural language modeling tasks.
Revised: Here we investigate the performance of our accelerated algorithms on two representative tasks, including vision classiﬁcation tasks and also natural language modeling tasks.

Original: For vision tasks, we test accelerated algorithms on both CNNs, e.g. ResNet (He et al., 2016), and vision transformers
Revised: For vision tasks, we test accelerated algorithms on both CNNs, e.g. ResNet (He et al., 2016), and vision transformers

Original: [Table]
Revised: [Table]

Original: (ViTs), such as ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).
Revised: (ViTs), such as ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).

Original: For natural language modeling tasks, we use LSTM (Schmidhuber et al., 1997) and Transformer-XL (Dai et al., 2019) to evaluate the proposed accelerated algorithms for sequence modeling.
Revised: For natural language modeling tasks, we use LSTM (Schmidhuber et al., 1997) and Transformer-XL (Dai et al., 2019) to evaluate the proposed accelerated algorithms for sequence modeling.

Original: For clarity
Revised: For clarity

Original: [Table]
Revised: [Table]

Original: we call our accelerated algorithm “X-Win”, where “X” denotes vanilla optimizers, e.g. Adam.
Revised: we call our accelerated algorithm “X-Win”, where “X” denotes vanilla optimizers, e.g. Adam.

Original: In all experiments, we do not change model architectures and data augmentations
Revised: In all experiments, we do not change model architectures and data augmentations

Original: [Table]
Revised: [Table]

Original: and only replace the default optimizer with ours.
Revised: and only replace the default optimizer with ours.

Original: Moreover, for all experiments, our accelerated algorithms, e
Revised: Moreover, for all experiments, our accelerated algorithms, e

Original: [Table]
Revised: [Table]

Original: g. AdamW-Win, always use the default optimizer-inherent hyper-parameters of the vanilla optimizers, e
Revised: g. AdamW-Win, always use the default optimizer-inherent hyper-parameters of the vanilla optimizers, e

Original: [Table]
Revised: [Table]

Original: g. ﬁrst- and second-order moment parameters β 1 and β 2 in AdamW; and their reckless step ¯ η k always satisﬁes ¯ η k = 2 η k .
Revised: g. ﬁrst- and second-order moment parameters β 1 and β 2 in AdamW; and their reckless step ¯ η k always satisﬁes ¯ η k = 2 η k .

Original: These settings well reduce the parameter-tuning cost of our algorithms.
Revised: These settings well reduce the parameter-tuning cost of our algorithms.

Original: In the experiments, same with other optimizers, we only slightly tune other widely tuned hyper-parameters around the vanilla ones, e
Revised: In the experiments, same with other optimizers, we only slightly tune other widely tuned hyper-parameters around the vanilla ones, e

Original: [Table]
Revised: [Table]

Original: g. step size and warm-up epochs, etc , which is reasonable, as our accelerated algorithms have two step sizes and the vanilla ones are not very suitable.
Revised: g. step size and warm-up epochs, etc , which is reasonable, as our accelerated algorithms have two step sizes and the vanilla ones are not very suitable.

Original: 5. R ESULTS ON V ISION C LASSIFICATION T ASKS
Revised: 5. R ESULTS ON V ISION C LASSIFICATION T ASKS

Original: [Table]
Revised: [Table]

Original: Results on ResNet18.
Revised: Results on ResNet18.

Original: Here we follow the conventional supervised training setting used in ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Deng et al., 2009).
Revised: Here we follow the conventional supervised training setting used in ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Deng et al., 2009).

Original: Due to limited space, we defer the hyper-parameter settings of the four accelerated algorithms in Table 1 into Appendix B.
Revised: Due to limited space, we defer the hyper-parameter settings of the four accelerated algorithms in Table 1 into Appendix B.

Original: Table 1 shows that our accelerated algorithms can improve the corresponding non-accelerated versions by a remarkable margin.
Revised: Table 1 shows that our accelerated algorithms can improve the corresponding non-accelerated versions by a remarkable margin.

Original: For instance, AdamW-Win, Adam-Win and
Revised: For instance, AdamW-Win, Adam-Win and

Original: LAMB-Win respectively make 3 .
Revised: LAMB-Win respectively make 3 .

Original: 1% , 2 . 2% and 2 .
Revised: 1% , 2 . 2% and 2 .

Original: 6% improvement over their corresponding non-accelerated counterparts, AdamW, Adam and LAMB.
Revised: 6% improvement over their corresponding non-accelerated counterparts, AdamW, Adam and LAMB.

Original: Moreover, SGD-Win improves SGD-H (i.e. SGD + heavy ball) by 3 . 4% , and also surpasses SGD-M ( Nesterov-accelerated SGD in Sec. 3.2) by 0 .
Revised: Moreover, SGD-Win improves SGD-H (i.e. SGD + heavy ball) by 3 . 4% , and also surpasses SGD-M ( Nesterov-accelerated SGD in Sec. 3.2) by 0 .

Original: 5% , also validating the superiority of our Win acceleration.
Revised: 5% , also validating the superiority of our Win acceleration.

Original: Besides, our accelerated algorithms, i.e. SGD-Win,
Revised: Besides, our accelerated algorithms, i.e. SGD-Win,

Original: AdamW-Win and LAMB-Win, beat several other optimizers, e.g. AdaBound, Radam (Liu et al., 2019), Nadam, Padam (Chen et al., 2021), AdaBelief, Yogi (Zaheer et al., 2018) and MSVAG (Balles & Hennig, 2018), in which Nadam uses Nesterov acceleration to estimate its ﬁrst-order gradient moment.
Revised: AdamW-Win and LAMB-Win, beat several other optimizers, e.g. AdaBound, Radam (Liu et al., 2019), Nadam, Padam (Chen et al., 2021), AdaBelief, Yogi (Zaheer et al., 2018) and MSVAG (Balles & Hennig, 2018), in which Nadam uses Nesterov acceleration to estimate its ﬁrst-order gradient moment.

Original: Actually, LAMB-Win sets a new SoTA top-1 accuracy on ResNet18.
Revised: Actually, LAMB-Win sets a new SoTA top-1 accuracy on ResNet18.

Original: All these results show the strong compatibility and superiority of our Win-acceleration in adaptive algorithms.
Revised: All these results show the strong compatibility and superiority of our Win-acceleration in adaptive algorithms.

Original: Results on ResNet50&101.
Revised: Results on ResNet50&101.

Original: Here we adopt the training setting in (Wightman et al., 2021) to train
Revised: Here we adopt the training setting in (Wightman et al., 2021) to train

Original: ResNet50&101, as this setting uses stronger data augmentation and largely improves CNNs’ performance.
Revised: ResNet50&101, as this setting uses stronger data augmentation and largely improves CNNs’ performance.

Original: See augmentation details and our algorithmic hyper-parameter settings in Appendix B. Here
Revised: See augmentation details and our algorithmic hyper-parameter settings in Appendix B. Here

Original: LAMB is the default optimizer because of its higher performance than other optimizers caused by the stronger augmentations (Wightman et al., 2021).
Revised: LAMB is the default optimizer because of its higher performance than other optimizers caused by the stronger augmentations (Wightman et al., 2021).

Original: All optimizers in Table 2 are under this setting.
Revised: All optimizers in Table 2 are under this setting.

Original: Table 2 reports the top-1 accuracy of the compared optimizers on ImageNet.
Revised: Table 2 reports the top-1 accuracy of the compared optimizers on ImageNet.

Original: By comparison, one can observe that our accelerated algorithms consistently outperform their corresponding non-accelerated version.
Revised: By comparison, one can observe that our accelerated algorithms consistently outperform their corresponding non-accelerated version.

Original: For example, across the three training epoch settings on ResNet50 and ResNet101, LAMBWin always achieves remarkable improvement over LAMB which is the ofﬁcial optimizer for this training recipe.
Revised: For example, across the three training epoch settings on ResNet50 and ResNet101, LAMBWin always achieves remarkable improvement over LAMB which is the ofﬁcial optimizer for this training recipe.

Original: Speciﬁcally, LAMB-Win makes 0 .
Revised: Speciﬁcally, LAMB-Win makes 0 .

Original: 7% average improvement over LAMB on both
Revised: 7% average improvement over LAMB on both

Original: [Table]
Revised: [Table]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: ResNet50 and ResNet101.
Revised: ResNet50 and ResNet101.

Original: For AdamW-Win and Adam-Win, they also respectively improve their counterparts by 0 .
Revised: For AdamW-Win and Adam-Win, they also respectively improve their counterparts by 0 .

Original: 7% and 0 .
Revised: 7% and 0 .

Original: 6% on ResNet50, 1 . 2% and 0.6% on ResNet101.
Revised: 6% on ResNet50, 1 . 2% and 0.6% on ResNet101.

Original: SGD-Win also makes 2 .
Revised: SGD-Win also makes 2 .

Original: 5% and 0 .
Revised: 5% and 0 .

Original: 8% overall improvement over heavy-ball accelerated SGD (SGD-H) and Nesterov accelerated SGD (SGD-M) on ResNet50, and also has similar advantage on ResNet101.
Revised: 8% overall improvement over heavy-ball accelerated SGD (SGD-H) and Nesterov accelerated SGD (SGD-M) on ResNet50, and also has similar advantage on ResNet101.

Original: These improvements are not trivial because of the following two reasons.
Revised: These improvements are not trivial because of the following two reasons.

Original: 1) Since the performance is already high and may approach the model limit, it is already very hard to make very large improvement.
Revised: 1) Since the performance is already high and may approach the model limit, it is already very hard to make very large improvement.

Original: This can be testiﬁed by the fact that in (Wightman et al., 2021), using LAMB to train ResNet50 for even 600 epochs only gives 80.4% top-1 accuracy.
Revised: This can be testiﬁed by the fact that in (Wightman et al., 2021), using LAMB to train ResNet50 for even 600 epochs only gives 80.4% top-1 accuracy.

Original: In contrast, our accelerated LAMB-Win uses 300 epochs (only half computational cost) to achieve 80 . 2% . 2)
Revised: In contrast, our accelerated LAMB-Win uses 300 epochs (only half computational cost) to achieve 80 . 2% . 2)

Original: By comparing the previous optimizers, including SAM, SGD-M, Adam, AdamW and LAMB, one can observe smaller accuracy gap ( ≤ 0 .
Revised: By comparing the previous optimizers, including SAM, SGD-M, Adam, AdamW and LAMB, one can observe smaller accuracy gap ( ≤ 0 .

Original: 2% ) between the best optimizer and the runner-up.
Revised: 2% ) between the best optimizer and the runner-up.

Original: For example, on ResNet101, the SoTA optimizer, i.e. SAM, only makes 0 .
Revised: For example, on ResNet101, the SoTA optimizer, i.e. SAM, only makes 0 .

Original: 1% average improvement over the runner-up, namely LAMB.
Revised: 1% average improvement over the runner-up, namely LAMB.

Original: All these comparisons show the non-travail improvement of our accelerated algorithms over their counterparts, and also the superiority of our acceleration technique.
Revised: All these comparisons show the non-travail improvement of our accelerated algorithms over their counterparts, and also the superiority of our acceleration technique.

Original: Results on ViTs.
Revised: Results on ViTs.

Original: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021;
Revised: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021;

Original: Yu et al., 2021).
Revised: Yu et al., 2021).

Original: To evaluate the performance of our accelerated algorithms, we select two popular and representative ViT architures, including ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).
Revised: To evaluate the performance of our accelerated algorithms, we select two popular and representative ViT architures, including ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).

Original: See the training setting and our hyper-parameter settings in Appendix B.
Revised: See the training setting and our hyper-parameter settings in Appendix B.

Original: We test our accelerated algorithms under different model sizes and different training epochs, and report the results in Table 3.
Revised: We test our accelerated algorithms under different model sizes and different training epochs, and report the results in Table 3.

Original: One can ﬁnd that since AdamW and LAMB use the decoupled weight decay, they enjoy better performance than SGD and Adam, which is also observed in other works, e.g. (Xiao et al., 2021;
Revised: One can ﬁnd that since AdamW and LAMB use the decoupled weight decay, they enjoy better performance than SGD and Adam, which is also observed in other works, e.g. (Xiao et al., 2021;

Original: Nado et al., 2021).
Revised: Nado et al., 2021).

Original: Moreover, under different training settings, our accelerated algorithms consistently outperform the corresponding non-accelerated counterparts.
Revised: Moreover, under different training settings, our accelerated algorithms consistently outperform the corresponding non-accelerated counterparts.

Original: Speciﬁcally, compared the default AdamW optimizer on both ViT and PoolFormer, our accelerated AdamW-Win respectively makes about 1.0%, 0.9%, 1.0% average improvement under the two training epoch settings on ViT-S, ViT-B and PoolFormer-S12.
Revised: Speciﬁcally, compared the default AdamW optimizer on both ViT and PoolFormer, our accelerated AdamW-Win respectively makes about 1.0%, 0.9%, 1.0% average improvement under the two training epoch settings on ViT-S, ViT-B and PoolFormer-S12.

Original: For Adam-Win and LAMB-Win, one can also observe their remarkable improvements on the three ViT backbones.
Revised: For Adam-Win and LAMB-Win, one can also observe their remarkable improvements on the three ViT backbones.

Original: Moreover, our accelerated SGD-Win also outperforms the Nesterov-accelerated SGD denoted as “SGD-M” by non-trivial margins under all settings.
Revised: Moreover, our accelerated SGD-Win also outperforms the Nesterov-accelerated SGD denoted as “SGD-M” by non-trivial margins under all settings.

Original: All these results are consistent with the observations on ResNets, and they together demonstrate the advantage of our accelerated optimizers for deep network training.
Revised: All these results are consistent with the observations on ResNets, and they together demonstrate the advantage of our accelerated optimizers for deep network training.

Original: [Figure]
Revised: [Figure]

Original: Results Analysis.
Revised: Results Analysis.

Original: Here we investigate the convergence behaviors of our accelerated algorithms, and aim to explain their better test performance over their non-accelerated counterparts.
Revised: Here we investigate the convergence behaviors of our accelerated algorithms, and aim to explain their better test performance over their non-accelerated counterparts.

Original: In Fig.
Revised: In Fig.

Original: 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their non-accelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesterove-accelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig.
Revised: 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their non-accelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesterove-accelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig.

Original: 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their nonaccelerated versions.
Revised: 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their nonaccelerated versions.

Original: Fig.
Revised: Fig.

Original: 3 in Appendix B also reveals SGD-Win and
Revised: 3 in Appendix B also reveals SGD-Win and

Original: Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.
Revised: Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.

Original: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated counterparts under the same computational cost.
Revised: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated counterparts under the same computational cost.

Original: [Table]
Revised: [Table]

Original: Robust Analysis.
Revised: Robust Analysis.

Original: For the only extra hyperparameter ¯ η k in our accelerated algorithms over their non-accelerated counterparts, in experiments, we always set ¯ η k = γη k , where γ =2 determines the relation between the reckless step ¯ η k and the conservative step η k .
Revised: For the only extra hyperparameter ¯ η k in our accelerated algorithms over their non-accelerated counterparts, in experiments, we always set ¯ η k = γη k , where γ =2 determines the relation between the reckless step ¯ η k and the conservative step η k .

Original: Here we investigate the effects of γ to the accelerated algorithms on ResNet50 by taking AdamW-Win and LAMB-Win as examples because of their superior performance.
Revised: Here we investigate the effects of γ to the accelerated algorithms on ResNet50 by taking AdamW-Win and LAMB-Win as examples because of their superior performance.

Original: Tableshows the stable performance of AdamW-Win and LAMB-Win when tuning γ in a relatively large range, thus testifying the robustness of AdamW-Win and LAMB-Win to the hyper-parameter γ .
Revised: Tableshows the stable performance of AdamW-Win and LAMB-Win when tuning γ in a relatively large range, thus testifying the robustness of AdamW-Win and LAMB-Win to the hyper-parameter γ .

Original: 5. R ESULTS ON N ATURAL L ANGUAGE M ODELING T ASKS
Revised: 5. R ESULTS ON N ATURAL L ANGUAGE M ODELING T ASKS

Original: [Table]
Revised: [Table]

Original: Results on LSTM.
Revised: Results on LSTM.

Original: We follow AdaBelief to test our accelerated algorithms via training three-layered LSTM (Schmidhuber et al., 1997) on the Penn TreeBank dataset (Marcinkiewicz, 1994) for 200 epochs.
Revised: We follow AdaBelief to test our accelerated algorithms via training three-layered LSTM (Schmidhuber et al., 1997) on the Penn TreeBank dataset (Marcinkiewicz, 1994) for 200 epochs.

Original: See optimization and training details in Appendix B.
Revised: See optimization and training details in Appendix B.

Original: From Table 5, one can observe that our Win-accelerated algorithms consistently surpass the corresponding nonaccelerated counterparts
Revised: From Table 7, one can observe that our Win-accelerated algorithms consistently surpass the corresponding nonaccelerated counterparts

Original: [Table]
Revised: [Table]

Original: and actually bring 1.2 overall average perplexity improvement over the four nonaccelerated counterparts.
Revised: and actually bring 1.2 overall average perplexity improvement over the four nonaccelerated counterparts.

Original: [Table]
Revised: [Table]

Original: Results on Transformer-XL.
Revised: Results on Transformer-XL.

Original: We adopt a widely used language sequence model, i.e. Transformer-XL (Dai et al., 2019), to further evaluate the performance of our accelerated algorithms.
Revised: We adopt a widely used language sequence model, i.e. Transformer-XL (Dai et al., 2019), to further evaluate the performance of our accelerated algorithms.

Original: Since 1) Adam is the most popular and used optimizer in NLP models, including Transformer-XL, and 2) our limited resource cannot well tune the hyperparameters of other optimizers in Sec.
Revised: Since 1) Adam is the most popular and used optimizer in NLP models, including Transformer-XL, and 2) our limited resource cannot well tune the hyperparameters of other optimizers in Sec.

Original: 5.1, we take Adam as an example to show the superiority of our accelerated algorithms.
Revised: 5.1, we take Adam as an example to show the superiority of our accelerated algorithms.

Original: Follow the ofﬁcial setting of Transformer-XL-base, we use Adam-Win with the default hyper-parameters of Adam on the WikiText-103 dataset.
Revised: Follow the ofﬁcial setting of Transformer-XL-base, we use Adam-Win with the default hyper-parameters of Adam on the WikiText-103 dataset.

Original: See more details in Appendix B.
Revised: See more details in Appendix B.

Original: Table 6 shows that under different training steps, our accelerated Adam-Win always achieves lower test PPL than the ofﬁcial Adam optimizer.
Revised: Table 8 shows that under different training steps, our accelerated Adam-Win always achieves lower test PPL than the ofﬁcial Adam optimizer.

Original: Speﬁcally, it improves 1.5 average test PPL over Adam on the three test cases.
Revised: Speﬁcally, it improves 1.5 average test PPL over Adam on the three test cases.

Original: All these results are consistent with observations on vision tasks, and they together demonstrate the advantages of our accelerated algorithms.
Revised: All these results are consistent with observations on vision tasks, and they together demonstrate the advantages of our accelerated algorithms.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we adopt proximal point method to derive a weight-decay-integrated Nesterov acceleration for AdamW and Adam, and extend it to LAMB and SGD.
Revised: In this work, we adopt proximal point method to derive a weight-decay-integrated Nesterov acceleration for AdamW and Adam, and extend it to LAMB and SGD.

Original: Moreover, we prove the convergence of our accelerated algorithms, i.e. accelerated AdamW, Adam and SGD, and observe the superiority of the accelerated Adam-type algorithm over the vanilla ones in terms of stochastic gradient complexity.
Revised: Moreover, we prove the convergence of our accelerated algorithms, i.e. accelerated AdamW, Adam and SGD, and observe the superiority of the accelerated Adam-type algorithm over the vanilla ones in terms of stochastic gradient complexity.

Original: Finally, experimental results validate the advantages of our accelerated algorithms.
Revised: Finally, experimental results validate the advantages of our accelerated algorithms.

Original: A PPENDIX
Revised: A PPENDIX

Original: The appendix contains the technical proofs of convergence results and some additional experimental details of the paper entitled “Win: Weight-Decay-Integrated Nesterov Acceleration for Adaptive
Revised: The appendix contains the technical proofs of convergence results and some additional experimental details of the paper entitled “Win: Weight-Decay-Integrated Nesterov Acceleration for Adaptive

Original: Gradient Algorithms”.
Revised: Gradient Algorithms”.

Original: It is structured as follows.
Revised: It is structured as follows.

Original: Appendix B provides more experimental details, such as hyper-parameter settings of the four accelerated algorithms and the ofﬁcial data augmentations.
Revised: Appendix B provides more experimental details, such as hyper-parameter settings of the four accelerated algorithms and the ofﬁcial data augmentations.

Original: In Appendix C, we deﬁne some necessary notations for our analysis.
Revised: In Appendix C, we deﬁne some necessary notations for our analysis.

Original: Then Appendix D provides some auxiliary lemmas throughout this document.
Revised: Then Appendix D provides some auxiliary lemmas throughout this document.

Original: Then Appendix E presents the proof of the convergence results in Sec. 1, i.e., the proof of Theorems 1 and 2.
Revised: Then Appendix E presents the proof of the convergence results in Sec. 1, i.e., the proof of Theorems 1 and 2.

Original: Finally, Appendix G provides the proofs of some auxiliary lemmas in Appendix D.
Revised: Finally, Appendix G provides the proofs of some auxiliary lemmas in Appendix D.

Original: A M ORE D ETAILS OF LAMB AND W IN - ACCELERATED LAMB
Revised: A M ORE D ETAILS OF LAMB AND W IN - ACCELERATED LAMB

Original: Here we introduce more details of vanilla LAMB (You et al., 2019) and our Win-accelerated LAMB.
Revised: Here we introduce more details of vanilla LAMB (You et al., 2019) and our Win-accelerated LAMB.

Original: Speciﬁcally, Algorithm 2 and 3 respectively summarize the algorithmic steps of LAMB and Winaccelerated LAMB.
Revised: Speciﬁcally, Algorithm 2 and 3 respectively summarize the algorithmic steps of LAMB and Winaccelerated LAMB.

Original: By comparison, we can ﬁnd two differences.
Revised: By comparison, we can ﬁnd two differences.

Original: The ﬁrst one is that LAMB (You et al., 2019) scales (cid:16) m k √ v k
Revised: The ﬁrst one is that LAMB (You et al., 2019) scales (cid:16) m k √ v k

Original: +
Revised: +

Original: ν
Revised: ν

Original: + λ x k (cid:17) via u k = (cid:107)
Revised: + λ x k (cid:17) via u k = (cid:107)

Original: x k (cid:107) 2 (cid:107) m
Revised: x k (cid:107) 2 (cid:107) m

Original: k / √ v k
Revised: k / √ v k

Original: + ν + λ x k (cid:107) 2 (cid:16) m
Revised: + ν + λ x k (cid:107) 2 (cid:16) m

Original: k √ v k
Revised: k √ v k

Original: +
Revised: +

Original: ν
Revised: ν

Original: + λ x k (cid:17) , while Winaccelerated LAMB scales
Revised: + λ x k (cid:17) , while Winaccelerated LAMB scales

Original: m k √ v k + ν through u k = (cid:107) x k (cid:107) 2 (cid:107) m k / √ v k + ν (cid:107) 2 m k √ v k
Revised: m k √ v k + ν through u k = (cid:107) x k (cid:107) 2 (cid:107) m k / √ v k + ν (cid:107) 2 m k √ v k

Original: + ν .
Revised: + ν .

Original: We scale m k / √ v k + ν instead of (cid:16) m k √ v k
Revised: We scale m k / √ v k + ν instead of (cid:16) m k √ v k

Original: +
Revised: +

Original: ν
Revised: ν

Original: + λ x k (cid:17) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .
Revised: + λ x k (cid:17) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .

Original: The second difference is that Win-accelerated LAMB has one extra acceleration step, i.e. the 7-th step in Algorithm 3, while LAMB has not.
Revised: The second difference is that Win-accelerated LAMB has one extra acceleration step, i.e. the 7-th step in Algorithm 3, while LAMB has not.

Original: B M ORE E XPERIMENTAL D ETAILS
Revised: B M ORE E XPERIMENTAL D ETAILS

Original: Due to space limitation, we defer the experimental details, such as hyper-parameter settings of the four accelerated algorithms, and their ofﬁcial augmentations in (He et al., 2016) and (Wightman et al., 2021), to this section.
Revised: Due to space limitation, we defer the experimental details, such as hyper-parameter settings of the four accelerated algorithms, and their ofﬁcial augmentations in (He et al., 2016) and (Wightman et al., 2021), to this section.

Original: For accelerated algorithms, including AdamW-Win, LAMB-Win, Adam-Win and SGD-Win, always share the default optimizer-inherent hyper-parameters of the vanilla optimizers and its reckless step ¯ η k is always 2 × larger than its conservative step η k for all iterations, i.e. ¯ η k = 2 η k .
Revised: For accelerated algorithms, including AdamW-Win, LAMB-Win, Adam-Win and SGD-Win, always share the default optimizer-inherent hyper-parameters of the vanilla optimizers and its reckless step ¯ η k is always 2 × larger than its conservative step η k for all iterations, i.e. ¯ η k = 2 η k .

Original: For AdamWWin, LAMB-Win, Adam-Win, their ﬁrst- and second-order moment parameters β 1 and β 2 are set to the default values β 1 = 0 .
Revised: For AdamWWin, LAMB-Win, Adam-Win, their ﬁrst- and second-order moment parameters β 1 and β 2 are set to the default values β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 used in AdamW, LAMB and Adam.
Revised: 999 used in AdamW, LAMB and Adam.

Original: For LAMBWin, its other key parameters, such as “grad averaging” and “trust clip”, also adopt the default ones in vanilla LAMB.
Revised: For LAMBWin, its other key parameters, such as “grad averaging” and “trust clip”, also adopt the default ones in vanilla LAMB.

Original: For SGD-Win, it uses the default momentum parameter 0.9 and set dampening parameter as 0.0 used in vanilla SGD.
Revised: For SGD-Win, it uses the default momentum parameter 0.9 and set dampening parameter as 0.0 used in vanilla SGD.

Original: Settings on ResNet18.
Revised: Settings on ResNet18.

Original: Here we follow the conventional supervised training setting used in
Revised: Here we follow the conventional supervised training setting used in

Original: ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Deng et al., 2009).
Revised: ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Deng et al., 2009).

Original: For data augmentation in (He et al., 2016), it uses random crop and horizontal ﬂipping with probability 0.5.
Revised: For data augmentation in (He et al., 2016), it uses random crop and horizontal ﬂipping with probability 0.5.

Original: For warm-up epochs, for all four accelerated algorithms, we set it as 5.0.
Revised: For warm-up epochs, for all four accelerated algorithms, we set it as 5.0.

Original: For base learning rate, we respectively set it as 3 × 10 − 3 , 5 × 10 − 3 , 3 × 10 − 3 , and 1.2 for AdamW-Win,
Revised: For base learning rate, we respectively set it as 3 × 10 − 3 , 5 × 10 − 3 , 3 × 10 − 3 , and 1.2 for AdamW-Win,

Original: LAMB-Win, Adam-Win and SGD-Win.
Revised: LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: For weight decay, we respectively set it as 5 × 10 − 2 , 5 × 10 − 2 , 10 − 6 , and 10 −for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: For weight decay, we respectively set it as 5 × 10 − 2 , 5 × 10 − 2 , 10 − 6 , and 10 −for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: On ResNet18, all algorithms are trained for 90 epochs with minibatch size 512 by following the conventional setting.
Revised: On ResNet18, all algorithms are trained for 90 epochs with minibatch size 512 by following the conventional setting.

Original: Settings on ResNet50&101.
Revised: Settings on ResNet50&101.

Original: For these two networks, we use “A2 training recipe” in (Wightman et al., 2021) to train them, since this training setting uses stronger data augmentation and largely improves CNNs’ performance.
Revised: For these two networks, we use “A2 training recipe” in (Wightman et al., 2021) to train them, since this training setting uses stronger data augmentation and largely improves CNNs’ performance.

Original: Speciﬁcally, the data augmentation in (Wightman et al., 2021) uses random crop, horizontal ﬂipping with probability, Mixup with parameter 0.1 (Zhang et al., 2018),
Revised: Speciﬁcally, the data augmentation in (Wightman et al., 2021) uses random crop, horizontal ﬂipping with probability, Mixup with parameter 0.1 (Zhang et al., 2018),

Original: CutMix with parameter 1.0 and probability 0 .
Revised: CutMix with parameter 1.0 and probability 0 .

Original: 5 (Yun et al., 2019), and RandAugment (Cubuk et al., 2020) with M = 7 , N = 2 and MSTD = 0 .
Revised: 5 (Yun et al., 2019), and RandAugment (Cubuk et al., 2020) with M = 7 , N = 2 and MSTD = 0 .

Original: Moreover, it often use binary cross-entropy (BCE) loss for training.
Revised: Moreover, it often use binary cross-entropy (BCE) loss for training.

Original: On both ResNet50 and ResNet101, for base learning rate, we respectively set it as 2 × 10 − 3 , 8 × 10 − 3 , 1 × 10 − 3 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for base learning rate, we respectively set it as 2 × 10 − 3 , 8 × 10 − 3 , 1 × 10 − 3 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.

Original: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 5, 10, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 5, 10, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Settings on ViT and PoolFormer.
Revised: Settings on ViT and PoolFormer.

Original: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021; Yu et al., 2021).
Revised: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021; Yu et al., 2021).

Original: For this setting, data augmentation includes random crop, horizontal ﬂipping with probability, Mixup with parameter 0.8 (Zhang et al., 2018), CutMix with parameter 1.0 and probability 0 .
Revised: For this setting, data augmentation includes random crop, horizontal ﬂipping with probability, Mixup with parameter 0.8 (Zhang et al., 2018), CutMix with parameter 1.0 and probability 0 .

Original: 5 (Yun et al., 2019), RandAugment (Cubuk et al., 2020) with M = 9 , N =and MSTD = 0 .
Revised: 5 (Yun et al., 2019), RandAugment (Cubuk et al., 2020) with M = 9 , N =and MSTD = 0 .

Original: 5 , and Random Erasing with parameter p = 0 .
Revised: 5 , and Random Erasing with parameter p = 0 .

Original: For training loss, we use cross entropy loss.
Revised: For training loss, we use cross entropy loss.

Original: On both ViT-S and ViT-B, for base learning rate, we respectively set it as 2 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 4 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ViT-S and ViT-B, for base learning rate, we respectively set it as 2 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 4 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.

Original: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 60, 30, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 60, 30, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: For
Revised: For

Original: AdamW-Win, following the default setting in AdamW, its minibatch size is 1024 for ViT-S andfor ViT-B.
Revised: AdamW-Win, following the default setting in AdamW, its minibatch size is 1024 for ViT-S andfor ViT-B.

Original: For all other accelerated optimizer, their minibatch sizes are always 1024.
Revised: For all other accelerated optimizer, their minibatch sizes are always 1024.

Original: Settings on LSTM.
Revised: Settings on LSTM.

Original: On LSTM, for base learning rate, we respectively set it as 1 × 10 − 3 , 1 × 10 − 2 , 1 × 10 − 2 , and 15.0 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On LSTM, for base learning rate, we respectively set it as 1 × 10 − 3 , 1 × 10 − 2 , 1 × 10 − 2 , and 15.0 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and divide the learning rate by 10 at epoch 100 and 145.
Revised: Moreover, we follow the default setting and divide the learning rate by 10 at epoch 100 and 145.

Original: For weight decay, we respectively set it as 2 × 10 − 2 , 5 × 10 − 2 , 1 . 8 × 10 − 6 , and 2 × 10 − 5 for AdamW-Win, LAMB-Win,
Revised: For weight decay, we respectively set it as 2 × 10 − 2 , 5 × 10 − 2 , 1 . 8 × 10 − 6 , and 2 × 10 − 5 for AdamW-Win, LAMB-Win,

Original: Adam-Win and SGD-Win.
Revised: Adam-Win and SGD-Win.

Original: We do not utilize the warmup strategy in this experiment.
Revised: We do not utilize the warmup strategy in this experiment.

Original: Following the default setting, we set minibatch size as 20.
Revised: Following the default setting, we set minibatch size as 20.

Original: Settings on Transformer-XL.
Revised: Settings on Transformer-XL.

Original: On Transformer-XL, for base learning rate, we set it as 4 × 10 − 4 for
Revised: On Transformer-XL, for base learning rate, we set it as 4 × 10 − 4 for

Original: Adam-Win.
Revised: Adam-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: For weight decay, we set it as 10 − 6 for Adam-Win.
Revised: For weight decay, we set it as 10 − 6 for Adam-Win.

Original: For warm-up steps, we set it as 2000.
Revised: For warm-up steps, we set it as 2000.

Original: Following the default setting, we set minibatch size as 60 × 4 .
Revised: Following the default setting, we set minibatch size as 60 × 4 .

Original: Training Epochs
Revised: Training Epochs

Original: T e s t
Revised: T e s t

Original: T o pA cc u r a c y
Revised: T o pA cc u r a c y

Original: SGD-M
Revised: SGD-M

Original: SGD-Win
Revised: SGD-Win

Original: Adam
Revised: Adam

Original: Adam-Win
Revised: Adam-Win

Original: Test accuracy curves of SGD-Win and Adam-Win on ResNet18.
Revised: Test accuracy curves of SGD-Win and Adam-Win on ResNet18.

Original: Here we investigate the convergence behaviors of our accelerated algorithms and hope to explain their better test performance over their non-accelerated counterparts.
Revised: Here we investigate the convergence behaviors of our accelerated algorithms and hope to explain their better test performance over their non-accelerated counterparts.

Original: In each sub-ﬁgure pair of Fig. 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their nonaccelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesteroveaccelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig. 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their corresponding non-accelerated versions.
Revised: In each sub-ﬁgure pair of Fig. 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their nonaccelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesteroveaccelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig. 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their corresponding non-accelerated versions.

Original: Fig.
Revised: Fig.

Original: 3 in Appendix B also reveals SGD-Win and Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.
Revised: 3 in Appendix B also reveals SGD-Win and Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.

Original: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated algorithms under the same computational cost.
Revised: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated algorithms under the same computational cost.

Original: C N OTATIONS
Revised: C N OTATIONS

Original: Here we ﬁrst give some important notations used in this document.
Revised: Here we ﬁrst give some important notations used in this document.

Original: For brevity, we let
Revised: For brevity, we let

Original: [Equation]
Revised: [Equation]

Original: Since we have (cid:107) m k (cid:107) ∞ ≤ c ∞ and ν ≤ (cid:107)
Revised: Since we have (cid:107) m k (cid:107) ∞ ≤ c ∞ and ν ≤ (cid:107)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ν (cid:107) ∞ ≤ c 2 ∞ + ν in Lemma 3 (see Appendix D), for brevity, let
Revised: ν (cid:107) ∞ ≤ c 2 ∞ + ν in Lemma 3 (see Appendix D), for brevity, let

Original: [Equation]
Revised: [Equation]

Original: Also we deﬁne
Revised: Also we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Next, we introduce an virtual sequence { y k } into the algorithm.
Revised: Next, we introduce an virtual sequence { y k } into the algorithm.

Original: In this way, we can rewrite the update steps in Algorithm 1 in the manuscript as its equivalent form (12):
Revised: In this way, we can rewrite the update steps in Algorithm 1 in the manuscript as its equivalent form (12):

Original: [Equation]
Revised: [Equation]

Original: where m 0 =
Revised: where m 0 =

Original: g 0 and v 0 = g 2 0 .
Revised: g 0 and v 0 = g 2 0 .

Original: For analysis, we further deﬁne
Revised: For analysis, we further deﬁne

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where λ k = λ 2 (cid:80) ki =1 (cid:0) 1 − µ 2 (cid:1) i ( k > 0) and λ 0 = 0 in which µ = β 2 c 2 ∞ ν .
Revised: where λ k = λ 2 (cid:80) ki =1 (cid:0) 1 − µ 2 (cid:1) i ( k > 0) and λ 0 = 0 in which µ = β 2 c 2 ∞ ν .

Original: In the following, we mainly use these notations to ﬁnish our proofs.
Revised: In the following, we mainly use these notations to ﬁnish our proofs.

Original: D A UXILIARY L EMMAS
Revised: D A UXILIARY L EMMAS

Original: Before giving our analysis, we ﬁrst provide some important lemmas.
Revised: Before giving our analysis, we ﬁrst provide some important lemmas.

Original: Lemma 3.
Revised: Lemma 3.

Original: Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 = 

Revised: That is, x k +1 =

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: Then { ( m k , s k ) } satisﬁes Assume c s, ∞ ≤ (cid:107) g k (cid:107) ∞ ≤ c ∞ , then we have
Revised: Then { ( m k , s k ) } satisﬁes Assume c s, ∞ ≤ (cid:107) g k (cid:107) ∞ ≤ c ∞ , then we have

Original: [Equation]
Revised: [Equation]

Original: See its proof in Appendix G.1.
Revised: See its proof in Appendix G.1.

Original: Lemma 4.
Revised: Lemma 4.

Original: (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 =
Revised: That is, x k +1 =

Original: [Equation]
Revised: [Equation]

Original: √ v k + ν .
Revised: √ v k + ν .

Original: Then { x k } satisﬁes
Revised: Then { x k } satisﬁes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Lemma 5.
Revised: Lemma 5.

Original: (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 =
Revised: That is, x k +1 =

Original: [Equation]
Revised: [Equation]

Original: √ v k + ν .
Revised: √ v k + ν .

Original: Then { z k } satisﬁes
Revised: Then { z k } satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Lemma 6.
Revised: Lemma 6.

Original: Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 = 1 1+ λη k
Revised: That is, x k +1 = 1 1+ λη k

Original: [Equation]
Revised: [Equation]

Original: By setting η k = η , ¯ η k = ¯ η , β 1 ,k
Revised: By setting η k = η , ¯ η k = ¯ η , β 1 ,k

Original: =
Revised: =

Original: β
Revised: β

Original: and β 2 ,k = β 2 , then we have y
Revised: and β 2 ,k = β 2 , then we have y

Original: [Equation]
Revised: [Equation]

Original: w i s i (cid:107)
Revised: w i s i (cid:107)

Original: y k +1 − (1 + λ ¯ η ) x k +1 (cid:107) 2 ≤ ρ k +1 (¯ η − η )(1 − ητ )
Revised: y k +1 − (1 + λ ¯ η ) x k +1 (cid:107) 2 ≤ ρ k +1 (¯ η − η )(1 − ητ )

Original: (1 + λη )k (cid:88) i =  ρ
Revised: (1 + λη )k (cid:88) i =  ρ

Original: i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i + (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13)
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13)

Original: (cid:107)
Revised: (cid:107)

Original: z k +1 − x k +1 (cid:107) 2 ≤ ρ k +1 ητ (¯ η − η ) 2 (1 − ητ )
Revised: z k +1 − x k +1 (cid:107) 2 ≤ ρ k +1 ητ (¯ η − η ) 2 (1 − ητ )

Original: (1 + λη )k (cid:88) i =  ρ
Revised: (1 + λη )k (cid:88) i =  ρ

Original: i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i + (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , (cid:107)
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , (cid:107)

Original: z k +1 − z k (cid:107) 2 ≤ 2¯ η(1 + λη )(cid:13) (cid:13)(cid:13) (cid:13) w k s k
Revised: z k +1 − z k (cid:107) 2 ≤ 2¯ η(1 + λη )(cid:13) (cid:13)(cid:13) (cid:13) w k s k

Original: (cid:13) (cid:13)(cid:13) (cid:13) + 2 ρ k
Revised: (cid:13) (cid:13)(cid:13) (cid:13) + 2 ρ k

Original: +

Revised: +

Original: ¯
Revised: ¯

Original: η 2 τ 2 (¯ η − η )(1 − ητ ) k
Revised: η 2 τ 2 (¯ η − η )(1 − ητ ) k

Original: (cid:88) i =
Revised: (cid:88) i =

Original:  ρ
Revised:  ρ

Original: i + (cid:13) (cid:13)(cid:13) (cid:13)
Revised: i + (cid:13) (cid:13)(cid:13) (cid:13)

Original: w i s i (cid:13) (cid:13)(cid:13) (cid:13) , where ρ k +

Revised: w i s i (cid:13) (cid:13)(cid:13) (cid:13) , where ρ k +

Original: =
Revised: =

Original: ητρ k and ρ 1 = 1 .
Revised: ητρ k and ρ 1 = 1 .

Original: See its proof in Appendix G.2.
Revised: See its proof in Appendix G.2.

Original: Lemma 7.
Revised: Lemma 7.

Original: Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 = 1 1+ λη k
Revised: That is, x k +1 = 1 1+ λη k

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: s k (cid:17) , y k +1 =
Revised: s k (cid:17) , y k +1 =

Original: z k − ¯ η k m k s k , z k +1 =
Revised: z k − ¯ η k m k s k , z k +1 =

Original: ¯ η k τ k x k +1 + η k τ k y k
Revised: ¯ η k τ k x k +1 + η k τ k y k

Original: +1 , s k = √ v k
Revised: +1 , s k = √ v k

Original: + ν .
Revised: + ν .

Original: By setting η k = η , ¯ η k = ¯ η , β 1 ,k
Revised: By setting η k = η , ¯ η k = ¯ η , β 1 ,k

Original: =
Revised: =

Original: β
Revised: β

Original: and β 2 ,k = β 2 , then we have
Revised: and β 2 ,k = β 2 , then we have

Original: E (cid:104) (cid:107) m k − ∇ F ( x k ) (cid:107) 2 (cid:105) ≤ 2(1 − β 1 ) E (cid:104) (cid:107) m k − 1 − ∇ F ( z k − 1 ) (cid:107) 2 (cid:105)
Revised: E (cid:104) (cid:107) m k − ∇ F ( x k ) (cid:107) 2 (cid:105) ≤ 2(1 − β 1 ) E (cid:104) (cid:107) m k − 1 − ∇ F ( z k − 1 ) (cid:107) 2 (cid:105)

Original: + 2Π k (1 − β 1 ) 

Revised: + 2Π k (1 − β 1 )

Original: L 2 β + 2 β 21 σb + 2 L Π (cid:48) k , where Π k := 2¯ η(1 + λη ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 

Revised: L 2 β + 2 β 21 σb + 2 L Π (cid:48) k , where Π k := 2¯ η(1 + λη ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13)

Original: + 2 ρ k ¯ η 2 τ 2 (¯ η − η )(1 − ητ )
Revised: + 2 ρ k ¯ η 2 τ 2 (¯ η − η )(1 − ητ )

Original: k − 1 (cid:88) i =
Revised: k − 1 (cid:88) i =

Original: ρ
Revised: ρ

Original: i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i + (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i
Revised: w i s i

Original: (cid:13)(cid:13)(cid:13)(cid:13) Π (cid:48)
Revised: (cid:13)(cid:13)(cid:13)(cid:13) Π (cid:48)

Original: k :
Revised: k :

Original: = ρ k ητ (¯ η − η )(1 − ητ )(1 + λη )k − 1 (cid:88)
Revised: = ρ k ητ (¯ η − η )(1 − ητ )(1 + λη )k − 1 (cid:88)

Original: i =  ρ
Revised: i =  ρ

Original: i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i + (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , where ρ k +1 =
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , where ρ k +1 =

Original: ητρ k and ρ 1 = 1 .
Revised: ητρ k and ρ 1 = 1 .

Original: see its proof in Appendix G.3.
Revised: see its proof in Appendix G.3.

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Recall our deﬁnition (cid:101) F
Revised: Recall our deﬁnition (cid:101) F

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: + λ
Revised: + λ

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: E ζ [ f
Revised: E ζ [ f

Original: [Equation]
Revised: [Equation]

Original: z ;
Revised: z ;

Original: [Equation]
Revised: [Equation]

Original: ] +
Revised: ] +

Original: [Equation]
Revised: [Equation]

Original: k
Revised: k

Original: [Equation]
Revised: [Equation]

Original: z
Revised: z

Original: [Equation]
Revised: [Equation]

Original: 2 s k , in the (13).
Revised: 2 s k , in the (13).

Original: By using the smoothness of f ( θ ; ζ )
Revised: By using the smoothness of f ( θ ; ζ )

Original: [Equation]
Revised: [Equation]

Original: we can obtain (cid:101)
Revised: we can obtain (cid:101)

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds since Lemma 3 proves (cid:13)(cid:13)(cid:13) s k s k
Revised: where (cid:172) holds since Lemma 3 proves (cid:13)(cid:13)(cid:13) s k s k

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ∞ ∈ [1 − µ, 1 + µ ] ( ∀ p ∈
Revised: ∞ ∈ [1 − µ, 1 + µ ] ( ∀ p ∈

Original: [0 , 1]) in which µ = β 2 c 2 ∞ ν ; (cid:173) holds because in Lemma 4, we have
Revised: [0 , 1]) in which µ = β 2 c 2 ∞ ν ; (cid:173) holds because in Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: (cid:174) holds, since we set η k ≤ c 21 (1+ λη k ) 2 c 2 ( L + λc 1 )
Revised: (cid:174) holds, since we set η k ≤ c 21 (1+ λη k ) 2 c 2 ( L + λc 1 )

Original: such that
Revised: such that

Original: [Equation]
Revised: [Equation]

Original: From Lemma 7, by setting η k = η , ¯ η k = ¯ η and β 1 ,k = β 1 , we have
Revised: From Lemma 7, by setting η k = η , ¯ η k = ¯ η and β 1 ,k = β 1 , we have

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Here ρ k +

Revised: Here ρ k +

Original: =
Revised: =

Original: ητρ k and ρ 1 = 1 .
Revised: ητρ k and ρ 1 = 1 .

Original: By considering c 2 ≥ (cid:107) s k (cid:107) ∞ ≥ c 1 , we have Π k ≤
Revised: By considering c 2 ≥ (cid:107) s k (cid:107) ∞ ≥ c 1 , we have Π k ≤

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: From Lemma
Revised: From Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we use the results in Lemma 6 that
Revised: where we use the results in Lemma 6 that

Original: [Equation]
Revised: [Equation]

Original: Then we add Eqn.
Revised: Then we add Eqn.

Original: (18) and α × (17) as follows:
Revised: (18) and α × (17) as follows:

Original: [Equation]
Revised: [Equation]

Original: Then by setting α = η (1 − β 1 ) c 1 β 1 (1+ λη ) and G ( x k +1 ) =
Revised: Then by setting α = η (1 − β 1 ) c 1 β 1 (1+ λη ) and G ( x k +1 ) =

Original: [Equation]
Revised: [Equation]

Original: F k
Revised: F k

Original: [Equation]
Revised: [Equation]

Original: η
Revised: η

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: Then summing the above inequality from k = 0 to k = T − 1 gives
Revised: Then summing the above inequality from k = 0 to k = T − 1 gives

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where ∆ = F ( x 0 ) − F ( x (cid:63) ) ;
Revised: where ∆ = F ( x 0 ) − F ( x (cid:63) ) ;

Original: x − 1 and m − 1 are two virtual points which satisfy m − 1 = ∇ F ( x − 1 ) .
Revised: x − 1 and m − 1 are two virtual points which satisfy m − 1 = ∇ F ( x − 1 ) .

Original: Now we try to bound (cid:80) T − 1 k =

Revised: Now we try to bound (cid:80) T − 1 k =

Original: ¯Π
Revised: ¯Π

Original: k
Revised: k

Original: and
Revised: and

Original: (cid:80) T − 1 k =

Revised: (cid:80) T − 1 k =

Original: ¯Π k .
Revised: ¯Π k .

Original: Firstly
Revised: Firstly

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w k − 1 (cid:107)
Revised: (cid:107) w k − 1 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: + 2¯ η
Revised: + 2¯ η

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: τ

Original: [Equation]
Revised: [Equation]

Original: (¯ η − η )c
Revised: (¯ η − η )c

Original: [Equation]
Revised: [Equation]

Original: (1 − ητ )
Revised: (1 − ητ )

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: = ρ k (cid:34) k − 1 (cid:88) i =  ρ
Revised: = ρ k (cid:34) k − 1 (cid:88) i =  ρ

Original: i +1 (cid:107)
Revised: i +1 (cid:107)

Original: w i (cid:107)(cid:35) =
Revised: w i (cid:107)(cid:35) =

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: = (cid:104)
Revised: = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: (cid:107)
Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: η )
Revised: η )

Original: [Equation]
Revised: [Equation]

Original: c
Revised: c

Original: [Equation]
Revised: [Equation]

Original: (1 − ητ )
Revised: (1 − ητ )

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: =  ρ k + (cid:34) T − 1 (cid:88) i =
Revised: =  ρ k + (cid:34) T − 1 (cid:88) i =

Original: k ρ i (cid:107) w k (cid:107)(cid:35) (cid:172) ≤ 2¯ ηc
Revised: k ρ i (cid:107) w k (cid:107)(cid:35) (cid:172) ≤ 2¯ ηc

Original: [Equation]
Revised: [Equation]

Original: (1 + λη )
Revised: (1 + λη )

Original: [Equation]
Revised: [Equation]

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: = (cid:104) (cid:107)
Revised: = (cid:104) (cid:107)

Original: [Equation]
Revised: [Equation]

Original: k − 1 (cid:107)
Revised: k − 1 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: (cid:105)
Revised: (cid:105)

Original: + 2¯ η 2 τ (¯ η − η )c 1 η (1 − ητ )
Revised: + 2¯ η 2 τ (¯ η − η )c 1 η (1 − ητ )

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) (cid:173)
Revised: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) (cid:173)

Original: [Equation]
Revised: [Equation]

Original: γ
Revised: γ

Original: [Equation]
Revised: [Equation]

Original: ηc
Revised: ηc

Original: [Equation]
Revised: [Equation]

Original: (cid:18) (1 + λη )
Revised: (cid:18) (1 + λη )

Original: [Equation]
Revised: [Equation]

Original: + ητ ( γ − 1)(1 − ητ )(cid:19) T − 1 (cid:88) k
Revised: + ητ ( γ − 1)(1 − ητ )(cid:19) T − 1 (cid:88) k

Original: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) where (cid:172) holds since 1) (cid:80) T − 1 i =
Revised: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) where (cid:172) holds since 1) (cid:80) T − 1 i =

Original: k ρ i =
Revised: k ρ i =

Original: ρ k (1 − η T − k τ T − k ) 1 − ητ ≤ ρ k 1 − ητ
Revised: ρ k (1 − η T − k τ T − k ) 1 − ητ ≤ ρ k 1 − ητ

Original: and 2) ρ k +

Revised: and 2) ρ k +

Original: =
Revised: =

Original: ητρ k and ρ 1 = 1 ; (cid:173) holds by setting ¯ η = γη .
Revised: ητρ k and ρ 1 = 1 ; (cid:173) holds by setting ¯ η = γη .

Original: Similarly, we can bound
Revised: Similarly, we can bound

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: T
Revised: T

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds since we choose proper η and β 1 such that
Revised: where (cid:172) holds since we choose proper η and β 1 such that

Original: [Equation]
Revised: [Equation]

Original: Now we discuss how to select η and β 1 such that (19) holds.
Revised: Now we discuss how to select η and β 1 such that (19) holds.

Original: To begin with, we have 1 (1+ λη ) 2 ≤and ητ ( γ − 1) 2 (1 − ητ ) 2 = 1+ γ + λγη γ (1+ λη ) ( γ − 1) 2 ≤ 1+ γγ ( γ − 1) 2 < γ 2 in which ητ
Revised: To begin with, we have 1 (1+ λη ) 2 ≤and ητ ( γ − 1) 2 (1 − ητ ) 2 = 1+ γ + λγη γ (1+ λη ) ( γ − 1) 2 ≤ 1+ γγ ( γ − 1) 2 < γ 2 in which ητ

Original: = η η + γη + λγη 2 .
Revised: = η η + γη + λγη 2 .

Original: Then we only need
Revised: Then we only need

Original: [Equation]
Revised: [Equation]

Original: So we arrive at
Revised: So we arrive at

Original: [Equation]
Revised: [Equation]

Original: where we set T ≥ 6 c 2 (1+ λη )∆ η(cid:15)and β 1 ≤ c 1 b(cid:15) 2 6 c 2 σ 2 .
Revised: where we set T ≥ 6 c 2 (1+ λη )∆ η(cid:15)and β 1 ≤ c 1 b(cid:15) 2 6 c 2 σ 2 .

Original: This result directly bounds
Revised: This result directly bounds

Original: [Equation]
Revised: [Equation]

Original: Moreover, from Lemma 6, we have
Revised: Moreover, from Lemma 6, we have

Original: [Table]
Revised: [Table]

Original: (1 − ητ )
Revised: (1 − ητ )

Original: k (cid:88) i =  ρ
Revised: k (cid:88) i =  ρ

Original: i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i + (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) (cid:173) =
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:35) (cid:173) =

Original: T
Revised: T

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: = Π k where (cid:172) holds by using Lemma 6; (cid:173) holds by using the deﬁnition in Eqn.
Revised: = Π k where (cid:172) holds by using Lemma 6; (cid:173) holds by using the deﬁnition in Eqn.

Original: Now remaining task is to upper bound 1 T (cid:80) T − 1 k =

Revised: Now remaining task is to upper bound 1 T (cid:80) T − 1 k =

Original: Π k and 1 T (cid:80) T − 1 k =

Revised: Π k and 1 T (cid:80) T − 1 k =

Original: Π (cid:48) k .
Revised: Π (cid:48) k .

Original: From the bound in Eqn.
Revised: From the bound in Eqn.

Original: (15) and the following bound on 1 T (cid:80) T − 1 k
Revised: (15) and the following bound on 1 T (cid:80) T − 1 k

Original: =

Revised: =

Original: ¯Π k and 1 T (cid:80) T − 1 k =

Revised: ¯Π k and 1 T (cid:80) T − 1 k =

Original: ¯Π
Revised: ¯Π

Original: (cid:48)
Revised: (cid:48)

Original: k , we have
Revised: k , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:172) ≤
Revised: (cid:172) ≤

Original: [Equation]
Revised: [Equation]

Original: (cid:15)
Revised: (cid:15)

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds, since 1) 1 T (cid:80) T − 1 k =0 E (cid:107) w k (cid:107) 2 ≤ 4 (cid:15) 2 ; 2) we use the results in Eqn.
Revised: where (cid:172) holds, since 1) 1 T (cid:80) T − 1 k =0 E (cid:107) w k (cid:107) 2 ≤ 4 (cid:15) 2 ; 2) we use the results in Eqn.

Original: (19) to obtain
Revised: (19) to obtain

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Besides, we have
Revised: Besides, we have

Original: [Equation]
Revised: [Equation]

Original: where in (cid:172) we use w k = m k + λ x k ∗ s k .
Revised: where in (cid:172) we use w k = m k + λ x k ∗ s k .

Original: In this way
Revised: In this way

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: From Eqn.
Revised: From Eqn.

Original: (17), we have
Revised: (17), we have

Original: [Equation]
Revised: [Equation]

Original: By setting ρ k +1 =
Revised: By setting ρ k +1 =

Original: (1 − β 1 ) ρ k and ρ 0 =
Revised: (1 − β 1 ) ρ k and ρ 0 =

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For all hyper-parameters
Revised: For all hyper-parameters

Original: [Equation]
Revised: [Equation]

Original: we put their constrains together:
Revised: we put their constrains together:

Original: [Equation]
Revised: [Equation]

Original: where c 1 = ν 0 .
Revised: where c 1 = ν 0 .

Original: 5 ≤ (cid:107) s k
Revised: 5 ≤ (cid:107) s k

Original: [Equation]
Revised: [Equation]

Original: ∞ ≤
Revised: ∞ ≤

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: For η , it should satisfy
Revised: For η , it should satisfy

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: Considering (1 − ητ )(1 + λη ) =
Revised: Considering (1 − ητ )(1 + λη ) =

Original: γ (1+ λη ) 2 1+ γ + λγη ≥
Revised: γ (1+ λη ) 2 1+ γ + λγη ≥

Original: γγ
Revised: γγ

Original: +1 , γ ( γ − 1)( γ +1) ≥ 1 γ and c 1 = ν 0 .
Revised: +1 , γ ( γ − 1)( γ +1) ≥ 1 γ and c 1 = ν 0 .

Original: 5 << 1 , then we have η ≤
Revised: 5 << 1 , then we have η ≤

Original: [Equation]
Revised: [Equation]

Original: where ν is often much smaller than one, and β 1 is very small.
Revised: where ν is often much smaller than one, and β 1 is very small.

Original: For T , we have
Revised: For T , we have

Original: [Equation]
Revised: [Equation]

Original: Now we compute the stochastic gradient complexity.
Revised: Now we compute the stochastic gradient complexity.

Original: For T iterations, the complexity is
Revised: For T iterations, the complexity is

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: F P ROOFS OF T HEOREM
Revised: F P ROOFS OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Recall our deﬁnition (cid:101) F k ( θ k )
Revised: Recall our deﬁnition (cid:101) F k ( θ k )

Original: = F ( θ ) + λ k 2 (cid:107) θ (cid:107) 

Revised: = F ( θ ) + λ k 2 (cid:107) θ (cid:107)

Original: =
Revised: =

Original: E ζ [ f ( θ ; ζ )]
Revised: E ζ [ f ( θ ; ζ )]

Original: + λ k 2 (cid:107) θ (cid:107) 22 in the (13).
Revised: + λ k 2 (cid:107) θ (cid:107) 22 in the (13).

Original: By setting β (cid:48) 1 = 1 − β 1 , then we have (cid:107) m k (cid:107) ∞ ≤ c ∞ by using Lemma 3 (see Appendix D).
Revised: By setting β (cid:48) 1 = 1 − β 1 , then we have (cid:107) m k (cid:107) ∞ ≤ c ∞ by using Lemma 3 (see Appendix D).

Original: Also we deﬁne
Revised: Also we deﬁne

Original: [Equation]
Revised: [Equation]

Original: By using the smoothness of f ( θ ; ζ ) , we can obtain (cid:101) F k +1 ( x k +1 )
Revised: By using the smoothness of f ( θ ; ζ ) , we can obtain (cid:101) F k +1 ( x k +1 )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ≤
Revised: ≤

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds because in Lemma 4, we have
Revised: where (cid:172) holds because in Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: where s k could be 1 ; (cid:172) holds, since we set η k ≤ (1+ λη k ) 2( L + λ ) such that
Revised: where s k could be 1 ; (cid:172) holds, since we set η k ≤ (1+ λη k ) 2( L + λ ) such that

Original: [Equation]
Revised: [Equation]

Original: Then in the following, we can directly follow the proof of Theorem 1.
Revised: Then in the following, we can directly follow the proof of Theorem 1.

Original: This is because the only difference between accelerated SGD and AdamW is that SGD has no the second-order moment v k , while AdamW has.
Revised: This is because the only difference between accelerated SGD and AdamW is that SGD has no the second-order moment v k , while AdamW has.

Original: By let s k
Revised: By let s k

Original: = 1 in accelerated AdamW and setting β (cid:48) 1 = 1 − β 1 in accelerated
Revised: = 1 in accelerated AdamW and setting β (cid:48) 1 = 1 − β 1 in accelerated

Original: SGD, then they share the exact the same updating rules.
Revised: SGD, then they share the exact the same updating rules.

Original: So after setting β (cid:48) 1 = 1 − β 1 in accelerated
Revised: So after setting β (cid:48) 1 = 1 − β 1 in accelerated

Original: SGD, to follow the proofs of Theorem 1, we only need to verify whether the auxiliary lemmas and the proof process of Theorem 1 hold for s k
Revised: SGD, to follow the proofs of Theorem 1, we only need to verify whether the auxiliary lemmas and the proof process of Theorem 1 hold for s k

Original: = 1 .
Revised: = 1 .

Original: This is the true case.
Revised: This is the true case.

Original: Please check our auxiliary lemmas, including Lemma 3 ∼ 7, and the proof process of Theorem 1.
Revised: Please check our auxiliary lemmas, including Lemma 3 ∼ 7, and the proof process of Theorem 1.

Original: Consider s k =in accelerated SGD, we have c 1 :
Revised: Consider s k =in accelerated SGD, we have c 1 :

Original: = 1 ≤ (cid:107) s k (cid:107) ∞ ≤ c 2 := 1 .
Revised: = 1 ≤ (cid:107) s k (cid:107) ∞ ≤ c 2 := 1 .

Original: In this way, by setting η ≤
Revised: In this way, by setting η ≤

Original: [Equation]
Revised: [Equation]

Original: 2 , β
Revised: 2 , β

Original: [Equation]
Revised: [Equation]

Original: b(cid:15) 2 6 σ 2 and β (cid:48) 1 = 1 − β 1 , after T
Revised: b(cid:15) 2 6 σ 2 and β (cid:48) 1 = 1 − β 1 , after T

Original: = O (cid:0)
Revised: = O (cid:0)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: iterations with minibatch size b and ∆ =
Revised: iterations with minibatch size b and ∆ =

Original: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by accelerated SGD satisﬁes the following four properties.
Revised: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by accelerated SGD satisﬁes the following four properties.

Original: a)
Revised: a)

Original: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by
Revised: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: b) The gradient moment m k can well estimate the full gradient ∇ F ( z k ) at the point
Revised: b) The gradient moment m k can well estimate the full gradient ∇ F ( z k ) at the point

Original: z k :
Revised: z k :

Original: [Equation]
Revised: [Equation]

Original: c)
Revised: c)

Original: The sequence { x k , z k } satisﬁes
Revised: The sequence { x k , z k } satisﬁes

Original: [Equation]
Revised: [Equation]

Original: d) The total stochastic gradient complexity to achieve the above three properties is O (cid:0) ∆ σ 2 L (cid:15) 4 (cid:1) .
Revised: d) The total stochastic gradient complexity to achieve the above three properties is O (cid:0) ∆ σ 2 L (cid:15) 4 (cid:1) .

Original: The proof is completed.
Revised: The proof is completed.

Original: G P ROOFS OF A UXILIARY L EMMAS G. P ROOF OF L EMMA
Revised: G P ROOFS OF A UXILIARY L EMMAS G. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: To begin with
Revised: To begin with

Original: [Equation]
Revised: [Equation]

Original: we assume that ∀ t ≤ k
Revised: we assume that ∀ t ≤ k

Original: [Equation]
Revised: [Equation]

Original: it holds
Revised: it holds

Original: [Equation]
Revised: [Equation]

Original: Then we consider the case where t = k + 1 as follows
Revised: Then we consider the case where t = k + 1 as follows

Original: [Equation]
Revised: [Equation]

Original: Then we derive the second results as follows:
Revised: Then we derive the second results as follows:

Original: [Equation]
Revised: [Equation]

Original: ∞ .
Revised: ∞ .

Original: Therefore, we have 1 −
Revised: Therefore, we have 1 −

Original: [Equation]
Revised: [Equation]

Original: We complete the proof.
Revised: We complete the proof.

Original: G.
Revised: G.

Original: P ROOF OF L EMMA
Revised: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: To begin with, we have
Revised: To begin with, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds since w k :
Revised: where (cid:172) holds since w k :

Original: = m k + λ x k ∗ s k ; (cid:173) holds since we set all η k = η and ¯ η k = ¯ η which gives τ k = τ =
Revised: = m k + λ x k ∗ s k ; (cid:173) holds since we set all η k = η and ¯ η k = ¯ η which gives τ k = τ =

Original: [Equation]
Revised: [Equation]

Original: +¯ η + λη ¯ η .
Revised: +¯ η + λη ¯ η .

Original: Therefore, by deﬁning ρ k +1 =
Revised: Therefore, by deﬁning ρ k +1 =

Original: ητρ
Revised: ητρ

Original: k
Revised: k

Original: and ρ 1 = 1 , then we have
Revised: and ρ 1 = 1 , then we have

Original: [Equation]
Revised: [Equation]

Original: For k = 0 , we have
Revised: For k = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In this way, one can obtain
Revised: In this way, one can obtain

Original: [Equation]
Revised: [Equation]

Original: s
Revised: s

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) hold since z 0 = x 0 and ρ 1 = 1 .
Revised: where (cid:172) hold since z 0 = x 0 and ρ 1 = 1 .

Original: Then we can upper bound (cid:13)(cid:13)(cid:13)(cid:13)
Revised: Then we can upper bound (cid:13)(cid:13)(cid:13)(cid:13)

Original: y k +1 − (1 + λ ¯ η ) x k + ρ k
Revised: y k +1 − (1 + λ ¯ η ) x k + ρ k

Original: + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: + (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: s i (cid:13)(cid:13)(cid:13)(cid:13) where (cid:172) holds since 1) (cid:80) ki =0 1 − ητρ i
Revised: s i (cid:13)(cid:13)(cid:13)(cid:13) where (cid:172) holds since 1) (cid:80) ki =0 1 − ητρ i

Original: =
Revised: =

Original: (cid:80) ki
Revised: (cid:80) ki

Original: [Equation]
Revised: [Equation]

Original: 0 ( 1 ρ i
Revised: 0 ( 1 ρ i

Original: [Equation]
Revised: [Equation]

Original: 1 −
Revised: 1 −

Original: [Equation]
Revised: [Equation]

Original: ρ i ) =
Revised: ρ i ) =

Original: ρ k +1 , and 2) Jensen’ inequality.
Revised: ρ k +1 , and 2) Jensen’ inequality.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Moreover, we can also bound
Revised: Moreover, we can also bound

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) .
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) .

Original: On the other hand, we have
Revised: On the other hand, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) we plug in y k +

Revised: where (cid:172) we plug in y k +

Original: =
Revised: =

Original: z k − ¯ η k m k s k .
Revised: z k − ¯ η k m k s k .

Original: Then we can upper bound
Revised: Then we can upper bound

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: G. P ROOF OF L EMMA
Revised: G. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: From Lemma 5, we have
Revised: From Lemma 5, we have

Original: [Equation]
Revised: [Equation]

Original: where in (cid:172)
Revised: where in (cid:172)

Original: [Equation]
Revised: [Equation]

Original: we use the results in Lemma 6 that
Revised: we use the results in Lemma 6 that

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where in (cid:172) , we use the results in Lemma 6 that
Revised: where in (cid:172) , we use the results in Lemma 6 that

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.