Original: W IN : W EIGHT -D ECAY -I NTEGRATED N ESTEROV A CCELERATION FOR A DAPTIVE G RADIENT A LGORITHMS
Revised: W IN : W EIGHT -D ECAY -I NTEGRATED N ESTEROV A CCELERATION FOR A DAPTIVE G RADIENT A LGORITHMS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Training deep networks onincreasingly large-scale datasets is computationally challenging.
Revised: Training deep networks on large-scale datasets is computationally challenging.

Original: In this work, we explore the problem of “ how to acceleratethe convergence of adaptive gradient algorithms in a general manner ”, and aimat providing practical insights to boost the training efﬁciency.
Revised: In this work, we explore the problem of “ how to accelerate adaptive gradient algorithms in a general manner ”, and aim to provide practical efﬁciency-boosting insights.

Original: To this end, we propose an effective and general Weight-decay-Integrated Nesterov acceleration (Win)for adaptive algorithms to enhance their convergence speed.
Revised: To this end, we propose an effective and general Weight-decay-Integrated Nesterov acceleration (Win) to accelerate adaptive algorithms.

Original: Taking AdamW and Adam as examples, we minimize a dynamical loss per iteration which combines the vanilla training loss and a dynamic regularizer inspired by proximal point method (PPM) to improve the convexity of the problem.
Revised: Taking AdamW and Adam as examples, we minimize a dynamical loss per iteration which combines the vanilla training loss and a dynamic regularizer inspired by proximal point method (PPM) to improve the convexity of the problem.

Original: To introduce Nesterovalike-acceleration into AdamW and Adam, we respectively use the ﬁrst- and second-order Taylor approximations of vanilla loss to update the variable twice while ﬁxing the above dynamic regularization brought by PPM.
Revised: To introduce Nesterov-alike-acceleration into AdamW and Adam, we respectively use the ﬁrstand second-order Taylor approximations of vanilla loss to update the variable twice.

Original: In this way, we arrive at our Win acceleration(like Nesterov acceleration) for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.
Revised: In this way, we arrive at our Win acceleration for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.

Original: Next, we extendthis Win acceleration to LAMB and SGD.
Revised: Next, we extend Win acceleration to LAMB and SGD.

Original: Our transparent acceleration derivation could provide insights for other accelerated methods and their integration into adaptive algorithms.
Revised: Our transparent acceleration derivation could provide insights for other accelerated methods and their integration into adaptive algorithms.

Original: Besides, we prove the convergence of Win-accelerated adaptive algorithms and justify their convergence superiority over their non-accelerated counterparts by taking AdamW and Adam as examples.
Revised: Besides, we prove the convergence of Win-accelerated adaptive algorithms and justify their convergence superiority over their non-accelerated counterparts by taking AdamW and Adam as examples.

Original: Experimental results testify  the faster convergence speed and superior performance of our Win-accelerated AdamW, Adam,
Revised: Experimental results testify to the faster convergence speed and superior performance of our Win-accelerated AdamW, Adam,

Original: LAMB and SGD over their non-accelerated counterparts on vision classiﬁcation tasks and language modeling tasks with both CNN and Transformer backbones.
Revised: LAMB and SGD over their non-accelerated counterparts on vision classiﬁcation tasks and language modeling tasks with both CNN and Transformer backbones.

Original: We hope Winacceleration shall be a default acceleration option forall popular optimizers in deep learning community to improve the training efﬁciency. 
Revised: We hope Win shall be a default acceleration option for popular optimizers in deep learning community to improve the training efﬁciency. Code will be released at https://github.com/sail-sg/win .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep neural networks (DNNs) are effective to model realistic data and have been successfully applied to many applications, e.g. image classiﬁcation (He et al., 2016) and speech recognition (Sainath et al., 2013).
Revised: Deep neural networks (DNNs) are effective to model realistic data and have been successfully applied to many applications, e.g. image classiﬁcation (He et al., 2016) and speech recognition (Sainath et al., 2013).

Original: Typically, their training models can be formulated as a nonconvex problem:
Revised: Typically, their training models can be formulated as a nonconvex problem:

Original: [Equation]
Revised: [Equation]

Original: where z ∈ R d is the model parameter; sample ζ is drawn from a data distribution D ; the loss f is differentiable; λ is a constant.
Revised: where z ∈ R d is the model parameter; sample ζ is drawn from a data distribution D ; the loss f is differentiable; λ is a constant.

Original: Though many algorithms, e.g. gradient descent (Cauchy et al., 1847) and variance-reduced algorithms (Johnson & Zhang, 2013), can solve problem (1), SGD (Robbins & Monro, 1951) uses the compositional structure in (1) to efﬁciently estimate gradient via minibatch data, and has become a dominant algorithm to train DNNs in practice because of its higher efﬁciency and effectiveness.
Revised: Though many algorithms, e.g. gradient descent (Cauchy et al., 1847) and variance-reduced algorithms (Rie Johnson, 2013), can solve problem (1), SGD (Robbins & Monro, 1951) uses the compositional structure in (1) to efﬁciently estimate gradient via minibatch data, and has become a dominant algorithm to train DNNs in practice because of its higher efﬁciency and effectiveness.

Original: However, on sparse data or ill-conditioned problems, SGD suffers from slow convergence speed (Kingma & Ba, 2014), as it scales the gradient uniformly in all parameter coordinate and ignores the problem properties on each coordinate.
Revised: However, on sparse data or ill-conditioned problems, SGD suffers from slow convergence speed (Kingma & Ba, 2014), as it scales the gradient uniformly in all parameter coordinate and ignores the problem properties on each coordinate.

Original: To resolve this issue, recent work has proposed a variety of adaptive methods, e.g. Adam (Kingma & Ba, 2014) and AdamW (Loshchilov &
Revised: To resolve this issue, recent work has proposed a variety of adaptive methods, e.g. Adam (Kingma & Ba, 2014) and AdamW (Loshchilov &

Original: Hutter, 2018), that scale each gradient coordinate according to the current geometry curvature of the loss F ( z ) .
Revised: Hutter, 2018), that scale each gradient coordinate according to the current geometry curvature of the loss F ( z ) .

Original: This coordinate-wise scaling greatly accelerates the optimization convergence and helps them, e.g. Adam and AdamW, become more popular in DNN training, especially for transformers.
Revised: This coordinate-wise scaling greatly accelerates the optimization convergence and helps them, e.g. Adam and AdamW, become more popular in DNN training, especially for transformers.

Original: Unfortunately, along with the increasing scale of both datasets and models, efﬁcient DNN training even with SGD or adaptive algorithms has become very challenging.
Revised: Unfortunately, along with the increasing scale of both datasets and models, efﬁcient DNN training even with SGD or adaptive algorithms has become very challenging.

Original: In this work, we are particularly interested in the problem of “ how to accelerate the convergence of adaptive algorithms in a general manner ” because of their dominant popularity across many DNNs.
Revised: In this work, we are particularly interested in the problem of “ how to accelerate the convergence of adaptive algorithms in a general manner ” because of their dominant popularity across many DNNs.

Original: Heavy ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are widely used in SGD but are rarely studied in adaptive algorithms.
Revised: Heavy ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are widely used in SGD but are rarely studied in adaptive algorithms.

Original: Among the very few, NAdam (Dozat, 2016) simpliﬁes Nesterov acceleration to estimate the ﬁrst moment of gradient in Adam while totally ignoring the second-order moments, which is not exact Nesterov acceleration and may not inherit its full acceleration merit.
Revised: Among the very few, NAdam (Dozat, 2016) simpliﬁes Nesterov acceleration to estimate the ﬁrst moment of gradient in Adam while totally ignoring the second-order moments, which is not exact Nesterov acceleration and may not inherit its full acceleration merit.

Original: Contributions:
Revised: Contributions:

Original: In this work, based on a recent Nesterov-type acceleration formulation (Nesterov et al., 2018) and proximal point method (PPM) (Moreau, 1965), we propose a new Weight-decayIntegrated Nesterov acceleration (Win for short) to accelerate adaptive algorithms, and also further analyze the convergence of Win-accelerated adaptive algorithms to justify their convergence superiority by taking AdamW and Adam as examples.
Revised: In this work, based on a recent Nesterov-type acceleration formulation (Nesterov et al., 2018) and proximal point method (PPM) (Moreau, 1965), we propose a new Weight-decayIntegrated Nesterov acceleration (Win for short) to accelerate adaptive algorithms, and also further analyze the convergence of Win-accelerated adaptive algorithms to justify their convergence superiority by taking AdamW and Adam as examples.

Original: Our main contributions are highlighted below.
Revised: Our main contributions are highlighted below.

Original: Firstly, we use PPM to rigorously derive our Win acceleration for accelerating adaptive algorithms.
Revised: Firstly, we use PPM to rigorously derive our Win acceleration for accelerating adaptive algorithms.

Original: By taking AdamW and Adam as examples, at the k -iteration, we follow PPM spirit and minimize a dynamically regularized loss F ( z )+ 12 η k (cid:107)
Revised: By taking AdamW and Adam as examples, at the k -iteration, we follow PPM spirit and minimize a dynamically regularized loss F ( z )+ 12 η k (cid:107)

Original: z − x k (cid:107) 2 √ v k
Revised: z − x k (cid:107) 2 √ v k

Original: + ν with the second-order gradient moment v k and the stabilizing constant ν in AdamW and Adam.
Revised: + ν with the second-order gradient moment v k and the stabilizing constant ν in AdamW and Adam.

Original: Then to introduce Nesterov-alike acceleration and also make the problem solvable iteratively, we respectively approximate F ( z ) by its ﬁrstand second-order Taylor expansions to update the variable z twice while always ﬁxing the above dynamic regularization and also an extra regularizer 12 η k (cid:107) z (cid:107) 2 √ v k + ν induced by the weight decay in
Revised: Then to introduce Nesterov-alike acceleration and also make the problem solvable iteratively, we respectively approximate F ( z ) by its ﬁrstand second-order Taylor expansions to update the variable z twice while always ﬁxing the above dynamic regularization and also an extra regularizer 12 η k (cid:107) z (cid:107) 2 √ v k + ν induced by the weight decay in

Original: AdamW. As a result, we achieve at our Win acceleration, a Nesterov-alike acceleration, for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.
Revised: AdamW. As a result, we arrive at our Win acceleration, a Nesterov-alike acceleration, for AdamW and Adam that uses a conservative step and a reckless step to update twice and then linearly combines these two updates for acceleration.

Original: Then we extend this Win acceleration to LAMB (You et al., 2019) and SGD.
Revised: Then we extend this Win acceleration to LAMB (You et al., 2019) and SGD.

Original: The above acceleration derivation is transparent and general which could motivate other accelerations and provide examples to introduce other accelerations into adaptive algorithms.
Revised: The above acceleration derivation is transparent and general which could motivate other accelerations and provide examples to introduce other accelerations into adaptive algorithms.

Original: Secondly, we prove the convergence of our Win-accelerated AdamW and Adam.
Revised: Secondly, we prove the convergence of our Win-accelerated AdamW and Adam.

Original: For both, to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, their stochastic gradient complexity is O (cid:0) c 2 .
Revised: For both, to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, their stochastic gradient complexity is O (cid:0) c 2 .

Original: 5 ∞ ν 0 .
Revised: 5 ∞ ν 1 .

Original: 5 (cid:15) 4 (cid:1) and matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same conditions, where c ∞ upper bounds the (cid:96) ∞ norm of stochastic gradient.
Revised: 25 (cid:15) 4 (cid:1) and matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same conditions, where c ∞ upper bounds the (cid:96) ∞ norm of stochastic gradient.

Original: Moreover, this complexity improves a factor O (cid:0) 1 ν 0 .
Revised: Moreover, this complexity improves a factor O ( dc 0 .

Original: 75 (cid:1) over the complexity O ( c2 .
Revised: 5 ∞ ) over the complexity O ( c

Original: 5 ∞  ν 1 .
Revised: 2 ∞ dσ 2 L ν 1 .

Original: 25 (cid:15) 4 ) of Adam-type optimizers in (Zhou et al., 2018; Guo et al., 2021), e.g. Adam, AdaGrad (Duchi et al., 2011), AdaBound (Luo et al., 2018), since ν is often small, e.g. O ( ν ) =
Revised: 25 (cid:15) 4 ) of Adam-type optimizers in (Zhou et al., 2018; Guo et al., 2021), e.g. Adam, AdaGrad (Duchi et al., 2011), AdaBound (Luo et al., 2018), since the network parameter dimension d is often much larger than c 0 .

Original: 10 − 4 in practice.
Revised: 5 ∞ , especially for over-parameterized modern networks.

Original: Indeed, Win-accelerated Adam and AdamW also enjoy superior complexity than other Adam variants, e.g. Adabelief (Zhuang et al., 2020) with compelxity O ( c 62 ν 2 (cid:15) 4 ) and RMSProp (Zhou et al., 2018) with comeplexity O ( c 0 .
Revised: Indeed, Win-accelerated Adam and AdamW also enjoy superior complexity than other Adam variants, e.g. Adabelief (Zhuang et al., 2020) with compelxity O ( c 62 ν 2 (cid:15) 4 ) and RMSProp (Zhou et al., 2018) with comeplexity O ( c 0 .

Original: 5 ∞ d 0 .
Revised: 5 ∞ d 0 .

Original: 5 ν(cid:15) 4 ) , especially on over-parameterized networks, where c 2 is the maximum (cid:96) 2 -norm of stochastic gradient.
Revised: 5 ν(cid:15) 4 ) , especially on overparameterized networks, where c 2 is the maximum (cid:96) 2 -norm of stochastic gradient.

Original: Finally, experimental results on both vision classiﬁcation tasks and language modeling tasks show that our Win-accelerated algorithms, i.e. accelerated AdamW, Adam, LAMB and SGD, can accelerate the convergence speed and also improve the performance of their corresponding non-accelerated counterparts by a remarkable margin on both CNN and transformer architectures.
Revised: Finally, experimental results on both vision classiﬁcation tasks and language modeling tasks show that our Win-accelerated algorithms, i.e. accelerated AdamW, Adam, LAMB and SGD, can accelerate the convergence speed and also improve the performance of their corresponding non-accelerated counterparts by a remarkable margin on both CNN and transformer architectures.

Original: All these results show the strong compatibility, generalization and superiority of our acceleration technique.
Revised: All these results show the strong compatibility, generalization and superiority of our acceleration technique.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: In the context of deep learning, when considering efﬁciency and generalization, one often prefers to adopt SGD and adaptive gradient algorithms (Bernstein et al., 2020), e.g. Adam, instead of other algorithms, e.g. variance-reduced algorithms (Johnson & Zhang, 2013), to solve problem (1).
Revised: In the context of deep learning, when considering efﬁciency and generalization, one often prefers to adopt SGD and adaptive gradient algorithms, e.g. Adam, instead of other algorithms, e.g. variancereduced algorithms (Rie Johnson, 2013), to solve problem (1).

Original: But, in practice, adaptive algorithms often suffer from inferior generalization performance than SGD.
Revised: But, in practice and theory, adaptive algorithms often suffer from inferior generalization performance than SGD (Zhou et al., 2020a;b).

Original: To solve this issue, AdamW (Loshchilov & Hutter, 2018) proposes a decoupled weight decay which introduces an (cid:96) 2 -alike regularization into Adam to decay network weight iteratively, and its effectiveness is widely validated on ViTs (Touvron et al., 2021) and CNNs (Touvron et al., 2021).
Revised: To solve this issue, AdamW (Loshchilov & Hutter, 2018) proposes a decoupled weight decay which introduces an (cid:96) 2 -alike regularization into Adam to decay network weight iteratively, and its effectiveness is widely validated on ViTs (Touvron et al., 2021) and CNNs (Touvron et al., 2021).

Original: Later,
Revised: Later,

Original: LAMB (You et al., 2019) scales the update in AdamW to the weight magnitude for getting rid of too large or small update, but suffers from unsatisfactory performance on small batch.
Revised: LAMB (You et al., 2019) scales the update in AdamW to the weight magnitude for avoiding too large or small update, but suffers from unsatisfactory performance on small batch.

Original: In this work, we aim to design a general accelerationapproach to accelerate these adaptive algorithms.
Revised: In this work, we aim to design a general acceleration to accelerate these adaptive algorithms.

Original: Heavy-ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are two classical acceleration techniques, and their effectiveness in SGD is well testiﬁed.
Revised: Heavy-ball acceleration (Polyak, 1964) and Nesterov acceleration (Nesterov, 2003) are two classical acceleration techniques, and their effectiveness in SGD is well testiﬁed.

Original: Later, NAdam (Dozat, 2016) integrates Nesterov acceleration into the ﬁrst-order gradient moment estimation but ignores the second-order gradient moments which harms the acceleration effect.
Revised: Later, NAdam (Dozat, 2016) integrates Nesterov acceleration into the ﬁrst-order gradient moment estimation but ignores the second-order gradient moments which harms the acceleration effect.

Revised: Some works (Anil et al., 2022; 2020) also explore Nesterov acceleration for second-order algorithms, e.g. shampoo (Gupta et al., 2018).

Original: Recently, for full gra- dient decent algorithm, a new general Nesterov-type acceleration (Nesterov et al., 2018) directly interpolates two variables to look ahead for correction, and is more ﬂexible than vanilla Nesterov acceleration (Nesterov, 2003) which interpolates the variable and gradient.
Revised: Recently, for full gradient decent algorithm, a new general Nesterov-type acceleration (Nesterov et al., 2018) directly interpolates two variables to look ahead for correction, and is more ﬂexible than vanilla Nesterov acceleration (Nesterov, 2003) which interpolates the variable and gradient.

Original: Seemore discussion in Sec. 3.2.
Revised: See discussion in Sec. 3.2.

Original: In this work, we use proximal point method to introduce this new acceleration into adaptive algorithms by a rigorous and transparent derivation andalso necessary adaptations (or tailors).
Revised: Here we use proximal point method to introduce this new acceleration into adaptive algorithms by a rigorous and transparent derivation and necessary tailors.

Original:  W EIGHT - DECAY -I NTEGRATED N ESTEROV A CCELERATION
Revised:  W EIGHT - DECAY -I NTEGRATED N ESTEROV A CCELERATION

Original: To accelerate full gradient descent algorithm, given a full gradient ∇ F ( z k ) of problem (1) at the k -th iteration, Nesterov-type acceleration (Nesterov et al., 2018) generally uses a conservative step η k and a reckless step ¯ η k to update two sequences x k +1 and y k +1 respectively, and then linearly combines them to update the variable z k +1 of the problem.
Revised: To accelerate full gradient descent algorithm, given a full gradient ∇ F ( z k ) of problem (1) at the k -th iteration, Nesterov-type acceleration (Nesterov et al., 2018) generally uses a conservative step η k and a reckless step ¯ η k to update two sequences x k +1 and y k +1 respectively, and then linearly combines them to update the variable z k +1 of the problem.

Original: Similar formulations are also observed and proved inmany recent works, e.g. (Allen-Zhu & Orecchia, 2014; Bansal & Gupta, 2019; Ahn & Sra, 2022).
Revised: Similar formulations are also observed and proved in recent works, e.g. (Allen-Zhu & Orecchia, 2014; Bansal & Gupta, 2019; Ahn & Sra, 2022).

Original: In general, their acceleration formulation can be formally formulated as
Revised: In general, their acceleration formulation can be formally formulated as

Original: [Equation]
Revised: [Equation]

Original: This acceleration enjoys provably faster convergence rate for full gradient descent method on convex problems (Nesterov et al., 2018), and is then empirically validated in many convex and nonconvex cases, e.g. (Wilson et al., 2017; Nado et al., 2021).
Revised: This acceleration enjoys provably faster convergence rate for full gradient descent method on convex problems (Nesterov et al., 2018), and is then empirically validated in many convex and nonconvex cases, e.g. (Wilson et al., 2017; Nado et al., 2021).

Original: Despite its effectiveness, such acceleration is rarely explored in adaptive gradient algorithms, especially for network training.
Revised: Despite its effectiveness, such acceleration is rarely explored in adaptive gradient algorithms, especially for network training.

Original: In deterministic optimization setting, another widely used optimization-stabilizing and acceleration approach is proximal point method (PPM)
Revised: In deterministic optimization setting, another widely used optimization-stabilizing and acceleration approach is proximal point method (PPM)

Original: (Moreau, 1965; Rockafellar, 1976).
Revised: (Moreau, 1965; Rockafellar, 1976).

Original: At the k -th iteration, PPM optimizes an (cid:96) 2 -regularized loss F ( z )+ 12 η k (cid:107) z − z k − 1 (cid:107) 22 instead of the vanilla loss F ( z ) .
Revised: At the k -th iteration, PPM optimizes an (cid:96) 2 -regularized loss F ( z )+ 12 η k (cid:107) z − z k − 1 (cid:107) 22 instead of the vanilla loss F ( z ) .

Original: This small change enhances the convexity of the problem, which accelerates and stabilizes optimization process (Kim et al., 2022; Zhou et al., 2021).
Revised: This small change enhances the convexity of the problem, accelerating and also stabilizing optimization process (Kim et al., 2022; Zhou et al., 2021c).

Original: To make the (cid:96) 2 -regularized problem solvable iteratively,
Revised: To make the (cid:96) 2 -regularized problem solvable iteratively,

Original: PPM approximates the loss F ( z ) by its ﬁrst- or second-order Taylor expansion so that each iteration has a close-form solution (see below).
Revised: PPM approximates the loss F ( z ) by its ﬁrst- or second-order Taylor expansion so that each iteration has a close-form solution (see below).

Original: In the following, we borrow thePPM idea  to induce a Weightdecay-Integrated Nesterov acceleration (Win) for adaptive algorithms by using AdamW and Adam as examples in Sec. 3.1, and  extend this acceleration technique to LAMB and SGD in Sec. 3.2.
Revised: At below, we borrow the idea in PPM to induce a Weightdecay-Integrated Nesterov acceleration (Win) for adaptive algorithms by using AdamW and Adam as examples in Sec. 3.1, and then extend this acceleration technique to LAMB and SGD in Sec. 3.2.

Original:  W IN -A CCELERATED A DAM W AND A DAM
Revised: 3. W IN -A CCELERATED A DAM W AND A DAM

Original: To begin with, following most adaptive gradient algorithms, e.g. Adam and AdamW, we estimate the ﬁrst- and second-order moments m k and v k of gradient as follows:
Revised: To begin with, following most adaptive gradient algorithms, e.g. Adam and AdamW, we estimate the ﬁrst- and second-order moments m k and v k of gradient as follows:

Original: [Equation]
Revised: [Equation]

Original: where g k is the average gradient on a minibatch data of size b , β 1 ∈ [0 , 1] and β 2 ∈ [0 , 1] .
Revised: where g k is the average gradient on a minibatch data of size b , β 1 ∈ [0 , 1] and β 2 ∈ [0 , 1] .

Original: For the initialization, we set m 0 = g 0 , v 0 = g 20 .
Revised: For the initialization, we set m 0 = g 0 , v 0 = g 20 .

Original: For brevity, with a small scaler ν > 0 , we deﬁne
Revised: For brevity, with a small scaler ν > 0 , we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Then following the spirit of PPM, at the k -th iteration, we minimize a regularized loss F ( x )+ 12 η k (cid:107) x − x k (cid:107) 2 s k .
Revised: Then following the spirit of PPM, at the k -th iteration, we minimize a regularized loss F ( x )+ 12 η k (cid:107) x − x k (cid:107) 2 s k , where (cid:107) x

Original:  Here we use the regularizer (cid:107) x − x k (cid:107) 2 s k instead of the (cid:96) 2 -regularization (cid:107) x − x k (cid:107) 22 , since 1) this new regularization allows us to handle adaptive algorithms as shown below Eqn.
Revised: (cid:107) s k = (cid:112) (cid:104) x , s k ∗ x (cid:105) with an element-wise product operation ∗ . Here we use the regularizer (cid:107) x − x k (cid:107) 2 s k instead of the (cid:96) 2 -regularization (cid:107) x − x k (cid:107) 22 , since 1) this new regularization can induce adaptive algorithms as shown below Eqn.

Original: (5), and 2) it also helps increase the convexity of the problem andactually further considersthe different sharpness properties of each coordinates of the loss because of different elements in s k , thus also speeding up the convergence.
Revised: (5), and 2) it increases the convexity of the problem and further considers different sharpness property of each coordinate because of different elements in s k , accelerating convergence.

Original: To make the problem solvable iteratively, we approximate the vanilla loss F ( z ) by its ﬁrst-order Taylor expansion at the point z k and update x k +1 as
Revised: To make the problem solvable iteratively, we approximate the vanilla loss F ( z ) by its ﬁrst-order Taylor expansion at the point z k and update x k +1 as

Original: [Equation]
Revised: [Equation]

Original: where(cid:107) x (cid:107) s k = (cid:112) (cid:104) x , s k ∗ x (cid:105) with an element-wise product operation ∗ , m k is used to approximate the full gradient ∇ F ( z k ) for Taylor expansion.
Revised: where m k is used to approximate the full gradient ∇ F ( z k ) .

Original: We add a small regularization λ  2 (cid:107) x (cid:107) 2 s k , since 1) it can largely improve the generalization performance in practice (Loshchilov & Hutter, 2018; Touvron et al., 2021); 2) it allows us to derive Adam ( λ =0 ) and AdamW ( λ> 0 ). 
Revised: We add a small regularization λ k 2 (cid:107) x (cid:107) 2 s k , since 1) it can largely improve the generalization performance in practice (Loshchilov & Hutter, 2018; Touvron et al., 2021); 2) it allows us to derive Adam ( λ k = 0 ) and AdamW ( λ k > 0 ). Here λ k can be ﬁxed as a constant or evolved along iteration number k . But in practice, a evolving λ k often enjoys better performance than a ﬁxed one (Caron et al., 2021; Zhou et al., 2022).

Original: Speciﬁcally, when λ =0 , the updating (5) becomes the exact Adam algorithm.
Revised: When λ k = 0 , the updating (5) becomes the exact Adam.

Original: If λ> 0 , the updating (5) is an approximation to the updating rule x k +1 =(1{++
 = (1+} − λη k ) x
Revised: If λ k > 0 , the updating (5) can approximate the updating rule x k {++
 = (1+} − λ k η k ) x

Original: k − η k u k of AdamW.
Revised: k − η k u k of AdamW.

Original: This is because consider λη k is small in practice, we can approximate (1+ λη k ) − 1 = 1 − λη k + O ( λ 2  η 2 k ) and thus
Revised: This is because consider λ k η k is small in practice, we can approximate (1+ λ k η k ) − 1 =1 − λ k η k + O ( λ 2 k η 2 k ) and thus

Original: [Equation]
Revised: [Equation]

Original:  − λη k + O ( λ 2  η 2 k )] x k − [ η k −O ( λη 2 k )
Revised: [1 − λ k η k + O ( λ 2 k η 2 k )] x k − [ η k − O ( λ k η 2 k )

Original: + O ( λ 3  η 3 k )]
Revised: + O ( λ 3 k η 3 k )]

Original: u k which becomes AdamW by ignoring the ignorable termsrelated to O ( η 2 k ) or O ( η 3 k ) .
Revised: u k which becomes AdamW by ignoring the ignorable terms O ( η 2 k ) or O ( η 3 k ) .

Original: This is also one reason that we adopt the regularizer (cid:107) x − x k (cid:107) 2 s k in (5) instead of the (cid:96) 2 -regularization in PPM, since we can ﬂexibly derive Adam and AdamW.
Revised: This is also one reason that we adopt the regularizer (cid:107) x − x k (cid:107) 2 s k in (5) instead of the (cid:96) 2 -regularization in PPM, since we can ﬂexibly derive Adam and AdamW.

Original: [-Input: initialization x 0 = z 
 = 0 , step size {-] (η k , ¯ η k ) } Tk =0 , moment parameters { β 1 , β 2 } . Output: (¯ x , ¯ z ) uniformly seleted from { ( x k , z k ) } Tk =0 .{+− x t +1 (cid:107) 
+}
Revised: [-Input: initialization x 0 = z 
 = 0 , step size {-]Similarly, we minimize a regularized loss F ( z ) + 12 η k (cid:107) z {+− x t +1 (cid:107) 
+}

Original: 1 while k < T do
Revised: s k again, and further approximate

Original: [Equation] LAMB [Equation] with τ
Revised: F ( z ) by its second-order approximation F ( z k )

Original: [Equation]
Revised: + (cid:104) m

Original: =
Revised: k , z

Original: [Equation]
Revised: [Equation]

Original: 8 end while

Original: [-Similarly, we minimize a regularized loss objective F ( z ) + 12 η k (cid:107) z − x t +1 (cid:107) 
-]
Revised: [-Similarly, we minimize a regularized loss objective F ( z ) + 12 η k (cid:107) z − x t +1 (cid:107) 
-]

Original: s k again, and further approximate F ( z ) by its second-order approximation F ( z k ) + (cid:104) m k , z − z k (cid:105) + 12¯ η k (cid:107) z − z k (cid:107) 2 s k z k +1 = argmin z F ( z k )+ (cid:104) m k , z − z k (cid:105) + 1 2¯ η k (cid:107) z − zk
Revised: z

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: η k
Revised: =

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where τ k = 1 η k +¯ η k
Revised: where τ k = 1 η k +¯ η k

Original: + λη k ¯ η k , m k is used to approximate ∇ F ( x k ) as guaranteed by Theorem 1 in
Revised: + λ k η k ¯ η k , m k is used to approximate ∇ F ( x k ) as guaranteed by Theorem 1 in

Original: Sec. 4, ¯ η k approximates the inverse of the local smoothness parameter of F ( z ) around z k .
Revised: Sec. 4, ¯ η k approximates the inverse of the local smoothness parameter of F ( z ) around z k .

Original: Here we use a regularizer (cid:107)
Revised: Here we use a regularizer (cid:107)

Original: z − x k +1 (cid:107) 2 s k with the latest update x k +1 instead of x k as an anchor point, since the latest update x k +1 could often provide better regularization for the concurrent optimization.
Revised: z − x k +1 (cid:107) 2 s k with the latest update x k +1 instead of x k as an anchor point, since the latest update x k +1 could often provide better regularization for the concurrent optimization.

Original: Now we have used PPM to rigorously derive our Win-accelerated AdamW and Adam in Eqns.
Revised: Now we have used PPM to rigorously derive our Win-accelerated AdamW and Adam in Eqns.

Original: (3), (5) and (6).
Revised: (3), (5) and (6).

Original: For more clarity, we summarize their algorithmic steps in Algorithm 1 in which we omit the bias-correction term for simplicity.
Revised: For more clarity, we summarize their algorithmic steps in Algorithm 1 in which we omit the bias-correction term for simplicity.

Original: When λ = 0 , it is Win-accelerated Adam; if λ > 0 , it gives Win-accelerated AdamW.
Revised: When λ = 0 , it is Win-accelerated Adam; if λ > 0 , it gives Win-accelerated AdamW.

Original: Generally, AdamW can greatly improve the generalization performance of Adam by simply adding a weight decay (i.e. the regularizer λ 2 (cid:107) · (cid:107) 2 s k ) into Adam as observed in many works, e.g. (Loshchilov & Hutter, 2018; Touvron et al., 2021).
Revised: Generally, AdamW can greatly improve the generalization performance of Adam by simply adding a weight decay (i.e. the regularizer λ 2 (cid:107) · (cid:107) 2 s k ) into Adam as observed in many works, e.g. (Loshchilov & Hutter, 2018; Touvron et al., 2021).

Original: Our Winacceleration is quit simple and efﬁcient, since our accelerated AdamW/Adam only adds one extra simple algorithmic step, i.e. the seventh step in Algorithm 1, on vanilla AdamW/Adam, and brings negligible extra computational overhead into vanilla optimizer, e.g. about 2%  extra training time per iteration on AdamW evaluated on ViT-small and ViT-base.
Revised: Our Winacceleration is quit simple and efﬁcient, since our accelerated AdamW/Adam only adds one extra simple algorithmic step, i.e. the seventh step in Algorithm 1, on vanilla AdamW/Adam, and brings negligible extra computational overhead into vanilla optimizer, e.g. about 2% ∼ 5% extra training time per iteration on AdamW evaluated on ViT-small and ViT-base.

Original: Moreover, for the only extra hyperparameter, the reckless step ¯ η k , in Algorithm 1 over AdamW/Adam, we always set it 2 × larger than the conservative step η k for all iterations, i.e. ¯ η k =2 η k , working well in our all experiments.
Revised: Moreover, for the only extra hyper-parameter, the reckless step ¯ η k , in Algorithm 1 over AdamW/Adam, we always set it 2 × larger than the conservative step η k for all iterations, i.e. ¯ η k =2 η k , working well in our all experiments.

Original: Now we discuss the relations between Nesterov-type acceleration (2) and our Win acceleration (6).
Revised: Now we discuss the relations between Nesterov-type acceleration (2) and our Win acceleration (6).

Original: For comparison, we introduce a virtual sequence y k
Revised: For comparison, we introduce a virtual sequence y k

Original: +

Revised: +

Original: =
Revised: =

Original: z k − ¯ η k u k in Win, and rewrite (6) as
Revised: z k − ¯ η k u k in Win, and rewrite (6) as

Original: [Equation]
Revised: [Equation]

Original: where u k is deﬁned in (4).
Revised: where u k is deﬁned in (4).

Original: By comparing Nesterov-type acceleration (2) with our Win acceleration (7), one can observe some similarity and also differences as well.
Revised: By comparing Nesterov-type acceleration (2) with our Win acceleration (7), one can observe some similarity and also differences as well.

Original: For similarity, both acceleration use a conservative step η k and a reckless step ¯ η k to update x k +1 and y k +1 respectively, and then linearly combine x k +1 and y k +1 to obtain z k +1 .
Revised: For similarity, both methods use a conservative step η k and a reckless step ¯ η k to update x k +1 and y k +1 respectively, and then linearly combine x k +1 and y k +1 to obtain z k +1 .

Original: For the differences, the ﬁrst one is that Win has a weight-decay-alike factor [Equation] k in (7) which slightly decays the variable x k like AdamW and also the update u k , while Nesterov acceleration has no such important factor.
Revised: For the differences, the ﬁrst one is that Win has a weight-decay-alike factor 1 1+ λ k η k in (7) which slightly decays the variable x k like AdamW and also the update u k , while Nesterov acceleration does not have.

Original: Note,this weight decay can greatly beneﬁt generalization in practice as shown in many works, e.g. (Loshchilov & Hutter, 2018; Touvron et al., 2021; Liu et al., 2021).
Revised: Note, weight decay can greatly beneﬁt generalization in practice as shown in many works, e.g. (Loshchilov & Hutter, 2018; Touvron et al., 2021; Liu et al., 2021).

Original: Another difference is that for almost all acceleration techniques, including Nesterov-type accelerationin (2), the sum of the linear combination factors ( e.g. ρ k and 1 − ρ k in (2)) is always one.
Revised: Another difference is that for almost all acceleration techniques, including Nesterov-type acceleration (2), the sum of their linear combination factors ( e.g. ρ k and 1 − ρ k in (2)) is always one.

Original: In contrast, in Eqn.
Revised: In contrast, in Eqn.

Original: (7), Win uses ¯ η k τ k +η
Revised: (7), Win uses ¯ η k τ [Equation] +

Original: [Equation]
Revised: [Equation]

Original: −
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: when λ > 0 , which indeed gives a second weigh decay.
Revised: > 0 , which further gives a second weigh decay.

Original: Since these two differences are caused by the weight decay, we call our acceleration “weight-decay-integrated Nesterov acceleration” (Win).
Revised: Since these two differences are caused by the weight decay, we call our acceleration “weight-decay-integrated Nesterov acceleration” (Win for short).

Original: 3. E XTENSION TO LAMB AND SGD
Revised: 3. E XTENSION TO LAMB AND SGD

Original: Here we generalize Win acceleration to LAMB (You et al., 2019) and SGD (Robbins & Monro, 1951).
Revised: Here we generalize Win acceleration to LAMB (You et al., 2019) and SGD (Robbins & Monro, 1951).

Original: For LAMB, it scales the update u k of AdamW in Eqn.
Revised: For LAMB, it scales the update u k of AdamW in Eqn.

Original: (4) so that u k is at the same magnitude of the network weight x k .
Revised: (4) so that u k is at the same magnitude of the network weight x k .

Original: That is, it changes the update rule x k +

Revised: That is, it changes the update rule x k +

Original: =
Revised: =

Original: (1 − λη k ) x
Revised: (1 − λ k η k ) x

Original: k − η k m k / s k in AdamW to x k [-+
-] =
Revised: k − η k m k / s k in AdamW to x k [-+
-]+1 =

Original: x
Revised: x

Original: k − η k (cid:107) x k (cid:107) 2 (cid:107) r k + λ  x k (cid:107) 2{+
+} ( r k + λ  x k ) where r k = m k / s k .
Revised: k − η k (cid:107) x k (cid:107) 2 (cid:107) r k + λ k x k (cid:107) {+
+} ( r k + λ k x k ) where r k = m k / s k .

Original: This modiﬁcation is to avoid too large or small update, improving optimization efﬁciency.
Revised: This modiﬁcation is to avoid too large or small update, improving optimization efﬁciency.

Original: To extend Win acceleration to
Revised: To extend Win acceleration to

Original: LAMB, we inherit this scaling spirit, and scale the update u k in (4) to the following one:
Revised: LAMB, we inherit this scaling spirit, and scale the update u k in (4) to the following one:

Original: [Equation]
Revised: [Equation]

Original: We scale m k / s k instead of ( m k / s k + λ  x k ) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .
Revised: We scale m k / s k instead of ( m k / s k + λ k x k ) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .

Original: Next, we can respectively follow Eqn.
Revised: Next, we can respectively follow Eqn.

Original: (5) and (6) to update the two sequences x k and z k .
Revised: (5) and (6) to update the two sequences x k and z k .

Original: See the detailed steps of Win-accelerated LAMB in Algorithm 1, and the detailed comparison between LAMB and Win-accelerated LAMB in Appendix A. {+Input: initialization x 0 = z 
+}
Revised: See the detailed steps of Win-accelerated LAMB in Algorithm 1, and the detailed comparison between LAMB and Win-accelerated LAMB in Appendix A. {+Input: initialization x 0 = z 
+}

Revised: = 0 , step size { ( η k , ¯ η k ) } Tk =0 , moment parameters { β 1 , β 2 } .

Original: For SGD, applying Win acceleration to it is quite direct.
Revised: Output: (¯ x , ¯ z ) uniformly seleted from { ( x k , z k ) } Tk =0 . 1 while k < T do g k = 1 b (cid:80) bi =1 ∇ f ( z k ; ζ i ) m k = ( [Equation]

Revised: 1 ) m

Revised: [Equation]

Revised: g k /* m 0 = g 0 */ v k = (

Revised: [Equation]

Revised: ) v

Revised: [Equation]

Revised: +

Revised: [Equation]

Revised: g

Revised: [Equation]

Revised: k /* v 0 = g 20 */ u k = m k √ v k + ν for AdamW and Adam, u k = (cid:107) x k (cid:107) 2 (cid:107) m k / √ v k

Revised: + ν + λ k x k (cid:107) 2 (cid:0) m k √ v k

Original: {++ ν + λ k x k (cid:1) for LAMB x k +
+}
Revised: {++ ν + λ k x k (cid:1) for LAMB x k +
+}

Original: {+= 1 
+}
Revised: {+= 1 
+}

Revised: + λ k η k

Revised: ( x k − η k u k )

Revised: z k +1 =

Revised: ¯ η k τ k x k +1 + η k τ k

Revised: ( z k − ¯ η k u k ) with τ k = 1 η k +¯ η k + λ k η k ¯ η k 8 end while

Revised: For SGD, applying Win acceleration to it is quite direct.

Original: Speciﬁcally, the only algorithmic difference between SGD and AdamW on the (cid:96) 2 -regularized problems is that SGD has no second-order moment v k while AdamW has.
Revised: Speciﬁcally, the only algorithmic difference between SGD and AdamW on the (cid:96) 2 -regularized problems is that SGD has no second-order moment v k while AdamW has.

Original: So we can borrow the acceleration framework of AdamW in Sec.
Revised: So we can borrow the acceleration framework of AdamW in Sec.

Original: 3.1 to accelerate SGD by setting s k = 1 ∈ R d in Eqn.
Revised: 3.1 to accelerate SGD by setting s k = 1 ∈ R d in Eqn.

Original: (4), (5) and (6), and obtain WIN-accelerated SGD:
Revised: (4), (5) and (6), and obtain Win-accelerated SGD:

Original: [Equation]
Revised: [Equation]

Original: ) where β (cid:48) 1 ∈ [0 , 1] is dampening parameter.
Revised: where β (cid:48) 1 ∈ [0 , 1] is dampening parameter.

Original: Here we slightly modify the moment m k to accord with the one used in Nesterov-accelerated SGD ( e.g. SGD-M in Pytorch) whose updating steps are
Revised: Here we slightly modify the moment m k to accord with the one used in Nesterov-accelerated SGD ( e.g. SGD-M in Pytorch) whose updating steps are

Original: [Equation]
Revised: [Equation]

Original: By comparing Win-accelerated SGD and SGD-M in (10), one can ﬁnd their big differences mainly caused by their different acceleration strategies and ways to handle weight decay.
Revised: By comparing Win-accelerated SGD and SGD-M in (10), one can ﬁnd their big differences mainly caused by their different acceleration strategies and ways to handle weight decay.

Original: Win-accelerated
Revised: Win-accelerated

Original: SGD is derived from PPM and a recently proposed acceleration (2), while SGD-M modiﬁes another previous Nesterov-type acceleration (Nesterov, 2003) (with formulation m k =
Revised: SGD is derived from PPM and a recently proposed acceleration (2), while SGD-M modiﬁes another previous Nesterov-type acceleration (Nesterov, 2003) (of formulation m k =

Original: β 1 m k − 1 − η k b (cid:80) bi =1 ∇ f
Revised: β 1 m k − 1 − η k b (cid:80) bi =1 ∇ f

Original: ( x k + η k m k − 1 ; ζ i ) and x k +

Revised: ( x k + η k m k − 1 ; ζ i ) and x k +

Original: = x k + m k ) to better train networks.
Revised: = x k + m k ) to better train networks.

Original: See more mechanisms of previous Nesterov acceleration and (10) in (Sutskever et al., 2013; Bengio et al., 2013).
Revised: See more mechanisms of previous Nesterov acceleration and (10) in (Sutskever et al., 2013; Bengio et al., 2013).

Original:  C ONVERGENCE A NALYSIS
Revised:  C ONVERGENCE A NALYSIS

Original: Here we investigate the convergence performance of Win-accelerated algorithms by taking accelerated AdamW, Adam and SGD as examples, as these algorithms are more preferably used in deep learning ﬁeld.
Revised: Here we investigate the convergence performance of Win-accelerated algorithms by taking accelerated AdamW, Adam and SGD as examples, as these algorithms are more preferably used in deep learning ﬁeld.

Original: Moreover, since we aim to accelerate deep network training which is highly nonconvex problems, we focus on analyzing nonconvex problems to accord with the practical setting.
Revised: Moreover, since we aim to accelerate deep network training which is highly nonconvex problems, we focus on analyzing nonconvex problems to accord with the practical setting.

Original: For analysis, we follow previous works on optimizer, e.g. (Kingma & Ba, 2014; Reddi et al., 2019;
Revised: For analysis, we follow previous optimization works, e.g. (Kingma & Ba, 2014; Reddi et al., 2019;

Original: Luo et al., 2019; Duchi et al., 2011;  Xie et al., 2022), and introduce necessary assumptions.
Revised: Duchi et al., 2011; Zhou et al., 2020b; 2021a;b; Xie et al., 2022), to introduce necessary assumptions.

Original: Assumption 1 ( L -smoothness) .
Revised: Assumption 1 ( L -smoothness) .

Original: We say a function f
Revised: We say a function f

Original: ( z , · ) to be L -smooth w.r.t.
Revised: ( z , · ) to be L -smooth w.r.t.

Original: z , if for ∀ z 1 , z 2 and ∀ ζ ∼ D , we have (cid:107)∇ f ( z 1 , ζ ) − ∇ f
Revised: z , if for ∀ z 1 , z 2 and ∀ ζ ∼ D , we have (cid:107)∇ f ( z 1 , ζ ) − ∇ f

Original: ( z 2 , ζ ) (cid:107) 2 ≤ L (cid:107) z 1 − z 2 (cid:107) 2 with a universal constant L .
Revised: ( z 2 , ζ ) (cid:107) 2 ≤ L (cid:107) z 1 − z 2 (cid:107) 2 with a universal constant L .

Original: Assumption 2 (Unbiased and bounded gradient estimation) .
Revised: Assumption 2 (Unbiased and bounded gradient estimation) .

Original: The gradient estimation g k is unbiased, i.e. for ∀ k , E [ g k ] = ∇ F ( z k ) , and its magnitude and variance are bounded, namely, for ∀ k , (cid:107) g k (cid:107) ∞ ≤ c ∞ and E [ (cid:107)∇ F ( z k ) − g k (cid:107) 2 ] ≤ σ with two universal constants c ∞ and σ .
Revised: The gradient estimation g k is unbiased, i.e. for ∀ k , E [ g k ] = ∇ F ( z k ) , and its magnitude and variance are bounded, namely, for ∀ k , (cid:107) g k (cid:107) ∞ ≤ c ∞ and E [ (cid:107)∇ F ( z k ) − g k (cid:107) 2 ] ≤ σ with two universal constants c ∞ and σ .

Original: Next, we ﬁrst deﬁne a dynamic function F k ( z ) at the k -th iteration which is real loss minimized by our algorithms.
Revised: Next, we ﬁrst deﬁne a dynamic function F k ( z ) at the k -th iteration which is real loss minimized by our algorithms.

Original: It combines the vanilla loss F ( z ) in (1) and a dynamic regularization λ  2 (cid:107) z (cid:107) 2 s k :
Revised: It combines the vanilla loss F ( z ) in (1) and a dynamic regularization λ k 2 (cid:107) z (cid:107) 2 s k :

Original: [Equation]
Revised: [Equation]

Original: (11) where s k is given in (4).
Revised: (11) where s k is given in (4).

Original: To obtain (11), following PPM spirit and Eqn.
Revised: To obtain (11), following PPM spirit and Eqn.

Original: (5), one can approximate
Revised: (5), one can approximate

Original: F ( z ) by its ﬁrst-order Taylor expansion, and obtain Eqn.
Revised: F ( z ) by its ﬁrst-order Taylor expansion, and obtain Eqn.

Original: (5) with x replaced by z to update z k +

Revised: (5) with x replaced by z to update z k +

Original: =
Revised: =

Original: + λη k
Revised: + λ k η k

Original: ( z k − η k m k / s k ) .
Revised: ( z k − η k m k / s k ) .

Original: Since λη k is very small, one can follow the discussion below Eqn.
Revised: Since λ k η k is very small, one can follow the discussion below Eqn.

Original: (5) and approximate z k +1 as z k [-+
 = (1-] − λη k )
Revised: (5) and approximate z k +1 as z k [-+
 = (1-]+1 =(1 − λ k η k )

Original: z k − η k m k / s k which becomes the update rule of 
Revised: z k − η k m k / s k which becomes the update rule of AdamW.

Original: AdamW. This is the reason that our analysis on Win-accelerated AdamW involves a dynamic loss F k ( z ) in (11).
Revised: This is the reason that our analysis on Win-accelerated AdamW involves a dynamic loss F k ( z ) in (11).

Original: Note, for Win-accelerated Adam ( λ  =0 ), F k ( z ) degenerates to the vanilla loss F ( z ) .
Revised: Note, for Win-accelerated Adam ( λ k =0 ), F k ( z ) degenerates to the vanilla loss F ( z ) .

Original: With these assumptions, we analyze the convergence behaviors of our accelerated algorithms on general nonconvex problems, and summarize our main results in Theorem 1 with proof in Appendix E.
Revised: With these assumptions, we analyze the convergence behaviors of our accelerated algorithms on general nonconvex problems, and summarize our main results in Theorem 1 with proof in Appendix E.

Original: Theorem 1.
Revised: Theorem 1.

Original: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .
Revised: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .

Original: By setting ¯ η k = γη k , γ > 1 , η k = η ≤ {+O (cid:0) ν 1 . 25 b(cid:15) 2 c 1 . 5 γ 2 . 5 σ 2 L (cid:1) , β 1 ≤ O (cid:0) ν 0 . 5 b(cid:15) 
+}
Revised: Let ¯ η k = γη k , γ > 1 , η k = η ≤ {+O (cid:0) ν 1 . 25 b(cid:15) 2 c 1 . 5 γ 2 . 5 σ 2 L (cid:1) , β 1 ≤ O (cid:0) ν 0 . 5 b(cid:15) 
+}

Original: [Equation]
Revised: cσ 2 (cid:1) , β 2 ∈ (0 , 1) , c = ( c 2 ∞ + ν ) 0 . 5 , λ k = λ (1 − β 2 c 2 ∞ ν ) k ( k > 0) and λ 0 = 0 with a constant λ > 0 .

Revised: Then after T = O (cid:0) c 2 .

Original:  iterations with minibatch size b and ∆ =
Revised: 5 ∞ γ 2 . 5 σ 2 L ∆ ν 1 . 25 b(cid:15)(cid:1) iterations with minibatch size b and ∆ =

Original: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } T k =0 generated by Win-accelerated AdamW and Adam in Algorithm 1 satisﬁes the following four properties.
Revised: F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by Win-accelerated AdamW and Adam in Algorithm 1 satisﬁes the following four properties.

Original: a)
Revised: a)

Original: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by
Revised: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: b) The gradient moment m k can well estimate the full gradient ∇ F ( x k ) and ∇ F ( z k ) :
Revised: b) The gradient moment m k can well estimate the full gradient ∇ F ( x k ) and ∇ F ( z k ) :

Original: [Equation]
Revised: [Equation]

Original: c)
Revised: c)

Original: The sequence { x k , z k } satisﬁes
Revised: The sequence { x k , z k } satisﬁes

Original: [Equation]
Revised: [Equation]

Original: d) The total stochastic gradient complexity to achieve the above three properties isO
Revised: d) The total stochastic gradient complexity to achieve the above three properties is

Original: [Equation]
Revised: [Equation]

Original: Theorem 1 guarantees the convergence of Win-accelerated AdamW and Adam in Algorithm 1 on nonconvex problems.
Revised: Theorem 1 guarantees the convergence of Win-accelerated AdamW and Adam in Algorithm 1 on nonconvex problems.

Original: When λ  > 0 , Algorithm 1 corresponds to Win-accelerated AdamW; and if λ = 0 , it becomes Win-accelerated Adam.
Revised: When λ k > 0 ( λ k =0 ), Algorithm 1 corresponds to Win-accelerated AdamW (Adam).

Original: For both cases, Theorem 1 holds.
Revised: For both cases, Theorem 1 holds.

Original: Theorem 1 a) shows that by running at most T 
Revised: Theorem 1 a) shows that by running at most T =

Original: = O (cid:0)
Revised: O (cid:0)

Original: [Equation]
Revised: [Equation]

Original: iterations, the average gradient
Revised: iterations, the average gradient

Original: [Equation]
Revised: [Equation]

Original: T [Equation] k [Equation]
Revised: (cid:80)

Original: E

Original: [Equation]
Revised: [Equation]

Original: F k ( x k ) [Equation] (cid:3) is upper bounded by (cid:15) 2 , guaranteeing the algorithmic convergence.
Revised: F k ( x k ) (cid:107) 22 (cid:3) is upper bounded by (cid:15) 2 , guaranteeing the algorithmic convergence.

Original: Theorem 1 b) indicates the gradient moment m k can well estimate the full gradient ∇ F ( z k ) and also ∇ F ( x k ) because of their small distances, guaranteeing the good Taylor approximation used in Eqn.
Revised: Theorem 1 b) indicates the gradient moment m k can well estimate the full gradient ∇ F ( z k ) and also ∇ F ( x k ) because of their small distances, guaranteeing the good Taylor approximation used in Eqns.

Original: (5) and (6).
Revised: (5) and (6).

Original: Moreover, in Theorem 1 c), one can ﬁnd that although Algorithm 1 uses a conservative step η k and a reckless step ¯ η k = γη k ( ∀ γ > 1) to update, the two sequences x k +1 and z k +1 can converge to each other, which could be the key for the good convergence behavior of both Win-accelerated AdamW and Adam.
Revised: Moreover, in Theorem 1 c), one can ﬁnd that although Algorithm 1 uses a conservative step η k and a reckless step ¯ η k = γη k ( ∀ γ > 1) to update, the two sequences x k +1 and z k +1 can converge to each other, which could be the key for the good convergence behavior of both Win-accelerated AdamW and Adam.

Original: Now we discuss the stochastic gradient complexity of Win-accelerated Adam and AdamW. Theorem 1 d) shows that to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, both Win-accelerated Adam and AdamW have the complexity O (cid:0) c 2 .
Revised: Now we discuss the stochastic gradient complexity of Win-accelerated Adam and AdamW. Theorem 1 d) shows that to ﬁnd an (cid:15) -approximate ﬁrst-order stationary point, both Win-accelerated Adam and AdamW have the complexity O (cid:0) c 2 .

Original: 5 ∞ σ 2 L ν 0 .
Revised: 5 ∞ σ 2 L ν 1 .

Original: 5 (cid:15) 4 (cid:1) which matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same Assumptions 1 and 2.
Revised: 25 (cid:15) 4 (cid:1) which matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under the same Assumptions 1 and 2.

Original: Our accelerated Adam and AdamW enjoy superior complexity over Adam-type optimizers, e.g. Adam,AdaGrad (Duchi et al., 2011),
Revised: Our accelerated Adam and AdamW enjoy superior complexity over Adam-type optimizers, e.g. Adam,

Original:  AdaBound (Luo et al., 2018), whose previously best known complexity under the same assumptions is O ( [Equation]
Revised: AdaGrad (Duchi et al., 2011), AdaBound (Luo et al., 2018), whose previously best known complexity under the same assumptions is O ( c 2 ∞ dσ 2 L ν 1 .

Original:  4 ) in (Zhou et al., 2018; Chen et al., 2021; Guo et al., 2021).
Revised: 25 (cid:15) 4 ) in (Zhou et al., 2018; Chen et al., 2021; Guo et al., 2021).

Original: By comparison, both accelerated Adam and AdamW improve their complexity by a factor O (cid:0) 1 ν 0 .
Revised: By comparison, both accelerated Adam and AdamW improve their complexity by a factor O ( dc 0 .

Original: 75 (cid:1) , where ν is often very small in practice, e.g. O ( ν ) =
Revised: 5 ∞ ) , where the network parameter dimension d is often much larger than c 0 .

Original: 10 − 4 .
Revised: 5 ∞ , especially for over-parameterized modern networks.

Original: Since the convergence of AdamW has not been proved yet in the literatures, here we cannot directly compare with it.
Revised: Since the convergence of AdamW has not been proved yet in the literatures, here we cannot directly compare with it.

Original: Moreover, the complexity of Win-accelerated Adam and AdamW is also lower than O ( c 62 σ 2 L ν 2 (cid:15) 4 ) of Adabelief (Zhuang et al., 2020) and O ( c 0 .
Revised: Moreover, the complexity of Win-accelerated Adam and AdamW is also lower than O ( c 62 σ 2 L ν 2 (cid:15) 4 ) of Adabelief (Zhuang et al., 2020) and O ( c 0 .

Original: 5 ∞ d 0 .
Revised: 5 ∞ d 0 .

Original: 5 σ 2 L ν(cid:15)) of RMSProp (Tijmen & Geoffrey, 2012;
Revised: 5 σ 2 L ν(cid:15)) of RMSProp (Tijmen & Geoffrey, 2012;

Original: Zhou et al., 2018), especially on over-parameterized networks, since for a d -dimensional gradient, its (cid:96) 2 -norm upper bound c 2 is often much larger than the (cid:96) ∞ -norm c ∞ and can be √ d × larger for worse case.
Revised: Zhou et al., 2018), especially on over-parameterized networks, since for a d -dimensional gradient, its (cid:96) 2 -norm upper bound c 2 is often much larger than the (cid:96) ∞ -norm c ∞ and can be √ d × larger for worse case.

Original: Now we discuss the convergence performance of Win-accelerated SGD in Theorem 2.
Revised: Now we discuss the convergence performance of Win-accelerated SGD in Theorem 2.

Original: Theorem 2.
Revised: Theorem 2.

Original: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .
Revised: Suppose Assumptions 1 and 2 hold, and x (cid:63) ∈ argmin x F ( x ) .

Original: By setting ¯ η k = γη k , γ > 1 , η k = η ≤ b(cid:15) [-2 36( γ +1) 
-]
Revised: Let ¯ η k = γη k , γ > 1 , η k = η ≤O (cid:0) b(cid:15) [-2 36( γ +1) 
-][Equation]

Original: Lσ 2  , β 1 ≤ b(cid:15) 2 6 σ 2 and β (cid:48) 1 = 1{+
 =1+} − β 1 , after T = O (cid:0) ∆
Revised: 5 γ 2 . 5 σ 2 L (cid:1) , β 1 ≤O (cid:0) b(cid:15) 2 cσ 2 (cid:1) , β (cid:48) {+
 =1+} − β 1 , λ k = λ , λ 0 =0 .

Original: σ 2 L ν 0 .
Revised: After T

Original: 5 b(cid:15) 4 (cid:1) iterations with minibatch size b and ∆ = F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by Win-accelerated SGD in (9) satisﬁes the four properties in Theorem 1 with ν = c ∞ =
Revised: = O (cid:0) ∆ σ 2 L b(cid:15) 4 (cid:1) iterations with minibatch size b and ∆= F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by Win-accelerated SGD in (9) satisﬁes the four properties in Theorem 1 with ν = c ∞ =

Original: c = 1 and s k = 1 ∈ R d .
Revised: c =1 and s k = 1 ∈ R d .

Original: See its proof in Appendix F. Theorem 2 also guarantees the convergence of Win-accelerated SGD.
Revised: See its proof in Appendix F. Theorem 2 also guarantees the convergence of Win-accelerated SGD.

Original: By using the hyper-parameter settings in Theorem 2, the sequence { ( x k , z k ) } Tk =0 generated by Winaccelerated SGD satisﬁes the four properties in Theorem 1 with ν = c ∞ =
Revised: By using the hyper-parameter settings in Theorem 2, the sequence { ( x k , z k ) } Tk =0 generated by Winaccelerated SGD satisﬁes the four properties in Theorem 1 with ν = c ∞ =

Original: c = 1 and s k = 1 .
Revised: c = 1 and s k = 1 .

Original: It shows the complexity
Revised: It shows the complexity

Original: O
Revised: O

Original: ( Lσ 2 (cid:15) 4 ) of Win-accelerated SGD which also matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) underthe same Assumptions 1 and 2. 
Revised: ( Lσ 2 (cid:15) 4 ) of Win-accelerated SGD which also matches the lower bound Ω( 1 (cid:15) 4 ) in (Arjevani et al., 2019; 2020) (up to constant factors) under Assumptions 1 and 2. [Table]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Here we investigate the performance of our accelerated algorithms on two representative tasks, including vision classiﬁcation tasks andalso natural language modeling tasks.
Revised: Here we evaluate our accelerated algorithms on two representative tasks, including vision classiﬁcation tasks and natural language modeling tasks.

Original: For vision tasks, we test accelerated algorithms on both CNNs, e.g. ResNet (He et al., 2016), and vision transformers[Table] (ViTs), such as ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).
Revised: For vision tasks, we test accelerated algorithms on both CNNs, e.g. ResNet (He et al., 2016), and vision transformers (ViTs), e.g. ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021; 2022).

Original: Fornatural language modeling tasks, we use LSTM (Schmidhuber et al., 1997) and Transformer-XL (Dai et al., 2019)to evaluate the proposed accelerated algorithms for sequence modeling. For clarity [Table]
Revised: For language modeling tasks, we use LSTM (Schmidhuber et al., 1997) and Transformer-XL (Dai et al., 2019) for evaluation.

Original:  we call our accelerated algorithm “X-Win”, where “X” denotes vanilla optimizers, e.g. Adam.
Revised: For clarity, we call our accelerated algorithm “X-Win”, where “X” denotes vanilla optimizers, e.g. Adam.

Original: In all experiments, we do not change model architectures and data augmentations [Table] and only replace the default optimizer with ours.Moreover, for all experiments, our accelerated algorithms, e
Revised: In all experiments, we do not change model architectures and data augmentations, and only replace the default optimizer with ours.

Original: [Table] g. AdamW-Win, always use the default optimizer-inherent hyper-parameters of the vanilla optimizers, e [Table] g. ﬁrst- and second-order moment parameters β 1 and β 2 in AdamW; and their reckless step ¯ η k always satisﬁes ¯ η k = 2 η k .
Revised: Moreover, for all experiments, our accelerated algorithms, e.g. AdamW-Win, always use the default optimizer-inherent hyper-parameters of the vanilla optimizers, e.g. ﬁrst- and second-order moment parameters β 1 and β 2 in AdamW; and their reckless step ¯ η k always satisﬁes ¯ η k = 2 η k .

Original: These settings well reduce the parameter-tuning cost of our algorithms.
Revised: These settings well reduce the parameter-tuning cost of our algorithms.

Original: In the experiments, same with other optimizers, we only slightly tune other widely tuned hyper-parameters around the vanilla ones, e [Table] g. step size and warm-up epochs, etc , which is reasonable, as our accelerated algorithms have two step sizes and the vanilla ones are not very suitable.
Revised: In the experiments, same with other optimizers, we only slightly tune other widely tuned hyper-parameters around the vanilla ones, e.g. step size and warm-up epochs, etc , which is reasonable, as our accelerated algorithms have two step sizes and the vanilla ones are not very suitable.

Original: 5. R ESULTS ON V ISION C LASSIFICATION T ASKS
Revised: 5. R ESULTS ON V ISION C LASSIFICATION T ASKS

Original: [Table]
Revised: [Table]

Original: Results on ResNet18.
Revised: Results on ResNet18.

Original: Here we follow the conventional supervised training setting used in ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Deng et al., 2009).
Revised: Here we follow the conventional supervised training setting used in ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Fei-Fei, 2009).

Original: Due to limited space, we defer the hyper-parameter settings of the four accelerated algorithms in Table 1 into Appendix B.
Revised: Due to limited space, we defer the hyper-parameter settings of the four accelerated algorithms in Table 1 into Appendix B.

Original: Table 1 shows that our accelerated algorithms can improve the corresponding non-accelerated versions by a remarkable margin.
Revised: Table 1 shows that our accelerated algorithms can improve the corresponding non-accelerated versions by a remarkable margin.

Original: For instance, AdamW-Win, Adam-Win and
Revised: For instance, AdamW-Win, Adam-Win and

Original: LAMB-Win respectively make 3 .
Revised: LAMB-Win respectively make 3 .

Original: 1% , 2 . 2% and 2 .
Revised: 1% , 2 . 8% and 2 .

Original: 6% improvement over their corresponding non-accelerated counterparts, AdamW, Adam and LAMB.
Revised: 6% improvement over their corresponding non-accelerated counterparts, AdamW, Adam and LAMB.

Original: Moreover, SGD-Win improves SGD-H (i.e. SGD + heavy ball) by 3 . 4% , and also surpasses SGD-M ( Nesterov-accelerated SGD in Sec. 3.2) by 0 .
Revised: Moreover, SGD-Win improves SGD-H (i.e. SGD + heavy ball) by 3 . 4% , and also surpasses SGD-M ( Nesterov-accelerated SGD in Sec. 3.2) by 0 .

Original: 5% , also validating the superiority of our Win acceleration.
Revised: 5% , also validating the superiority of our Win acceleration.

Original: Besides, our accelerated algorithms, i.e. SGD-Win,
Revised: Besides, our accelerated algorithms, i.e. SGD-Win,

Original: AdamW-Win and LAMB-Win, beat several other optimizers, e.g. AdaBound, Radam (Liu et al., 2019), Nadam, Padam (Chen et al., 2021), AdaBelief, Yogi (Zaheer et al.,2018) and MSVAG (Balles & Hennig, 2018), in which Nadam uses Nesterov acceleration to estimate its ﬁrst-order gradient moment.
Revised: AdamW-Win and LAMB-Win, beat several other optimizers, e.g. AdaBound, Radam (Liu et al., 2019), Nadam, Padam (Chen et al., 2021), AdaBelief, Yogi (Zaheer et al., 2018), in which Nadam uses Nesterov acceleration to estimate its ﬁrst-order gradient moment.

Original: Actually, LAMB-Win sets a new SoTA top-1 accuracy on ResNet18.
Revised: Actually, LAMB-Win sets a new SoTA top-1 accuracy on ResNet18.

Original: All these results show the strong compatibility and superiority of our Win-acceleration in adaptive algorithms.
Revised: All these results show the strong compatibility and superiority of our Win-acceleration in adaptive algorithms.

Original: Results on ResNet50&101.
Revised: Results on ResNet50&101.

Original: Here we adopt the training setting in (Wightman et al., 2021) to train
Revised: Here we adopt the training setting in (Wightman et al., 2021) to train

Original: ResNet50&101, as this setting uses stronger data augmentation and largely improves CNNs’ performance.
Revised: ResNet50&101, as this setting uses stronger data augmentation and largely improves CNNs’ performance.

Original: See augmentation details and our algorithmic hyper-parameter settings in Appendix B. Here
Revised: See augmentation details and our algorithmic hyper-parameter settings in Appendix B. Here

Original: LAMB is the default optimizer because of its higher performance than other optimizers caused by the stronger augmentations (Wightman et al., 2021).
Revised: LAMB is the default optimizer because of its higher performance than other optimizers caused by the stronger augmentations (Wightman et al., 2021).

Original: All optimizers in Table 2 are under this setting.
Revised: All optimizers in Table 2 are under this setting.

Original: Table 2 reports the top-1 accuracy of the compared optimizers on ImageNet.

Original: By comparison, one can observe that our accelerated algorithms consistently outperform their corresponding non-accelerated version.
Revised: Table 2 shows that our accelerated algorithms consistently outperform their corresponding nonaccelerated version.

Original: For example, across the three training epoch settings on ResNet50 and ResNet101, LAMBWin always achieves remarkable improvement overLAMB which is the ofﬁcial optimizer  for this training recipe.
Revised: For example, across the three training epoch settings on ResNet50 / ResNet101, [Table] LAMB-Win always achieves remarkable improvement over the ofﬁcial optimizer LAMB for this training recipe.

Original: Speciﬁcally, LAMB-Win makes 0 .
Revised: Speciﬁcally, LAMB-Win makes 0 .

Original: 7% average improvement over LAMB on both
Revised: 7% average improvement over LAMB on both

Original: [Table]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: ResNet50 and ResNet101.
Revised: ResNet50 / ResNet101.

Original: For AdamW-Win and Adam-Win, they also respectively improve their counterparts by 0 .
Revised: For AdamW-Win and Adam-Win, they also respectively improve their counterparts by 0 .

Original: 7% and 0 .
Revised: 7% and 0 .

Original: 6% on ResNet50, 1 . 2% and 0.6% on ResNet101.
Revised: 4% on ResNet50, 1 . 2% and 0.6% on ResNet101.

Original: SGD-Win also makes 2 .
Revised: SGD-Win also makes 2 .

Original: 5% and 0 .
Revised: 5% and 0 .

Original: 8% overall improvement over heavy-ball accelerated SGD (SGD-H) and Nesterov accelerated SGD (SGD-M) on ResNet50, and also has similar advantage on ResNet101.
Revised: 8% overall improvement over heavy-ball accelerated SGD (SGD-H) and Nesterov accelerated SGD (SGD-M) on ResNet50, and also has similar advantage on ResNet101.

Original: These improvements are not trivial because of the following two reasons.
Revised: These improvements are not trivial because of the following two reasons.

Original: 1) Since the performance is already high and may approach the model limit, it is already very hard to make very large improvement.
Revised: 1) Since the performance is already high and may approach the model limit, it is already very hard to make very large improvement.

Original: This can be testiﬁed by the fact that in (Wightman et al., 2021), using LAMB to train ResNet50 foreven 600 epochs only gives 80.4% top-1 accuracy.
Revised: This is testiﬁed by the fact that in (Wightman et al., 2021), using LAMB to train ResNet50 for 600 epochs only gives 80.4% top-1 accuracy.

Original: In contrast, our accelerated LAMB-Win uses 300 epochs (only half computational cost) to achieve 80 . 2% . 2)
Revised: In contrast, our accelerated LAMB-Win uses 300 epochs (half training cost) to achieve 80 . 2% . 2)

Original: By comparing the previous optimizers, including SAM, SGD-M, Adam, AdamW and LAMB, one can observe smaller accuracy gap ( ≤ 0 .
Revised: By comparing the previous optimizers, including SAM, SGD-M, Adam, AdamW and LAMB, one can observe smaller accuracy gap ( ≤ 0 .

Original: 2% ) between the best optimizer and the runner-up.
Revised: 2% ) between the best optimizer and the runner-up.

Original: For example, on ResNet101, the SoTA optimizer, i.e. SAM, only makes 0 .
Revised: For example, on ResNet101, the SoTA optimizer, i.e. SAM, only makes 0 .

Original: 1% average improvement over the runner-up, namely LAMB.
Revised: 1% average improvement over the runner-up LAMB.

Original: All these comparisons show the non-travail improvement of our accelerated algorithms over their counterparts, and also the superiority of our acceleration technique.
Revised: All these comparisons show the non-travail improvement of our accelerated algorithms over their counterparts.

Original: Results on ViTs.
Revised: Results on ViTs.

Original: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021;
Revised: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021;

Original: Yu et al., 2021).
Revised: Yu et al., 2021).

Original: To evaluate the performance of our accelerated algorithms, we select two popular and representative ViT architures, including ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).
Revised: To evaluate the performance of our accelerated algorithms, we select two popular and representative ViT architures, including ViT (Dosovitskiy et al., 2020) and PoolFormer (Yu et al., 2021).

Original: See the training setting and our hyper-parameter settings in Appendix B.
Revised: See the training setting and our hyper-parameter settings in Appendix B.

Original: We test our accelerated algorithms under different model sizes and different training epochs, and report the results in Table 3.
Revised: We test our accelerated algorithms under different model sizes and different training epochs, and report the results in Table 3.

Original: One can ﬁnd that since AdamW and LAMB use the decoupled weight decay, they enjoy better performance than SGD and Adam, which is also observed in other works, e.g. (Xiao et al., 2021;
Revised: One can ﬁnd that since AdamW and LAMB use the decoupled weight decay, they enjoy better performance than SGD and Adam, which is also observed in other works, e.g. (Xiao et al., 2021;

Original: Nado et al., 2021).
Revised: Nado et al., 2021).

Original: Moreover, under different training settings, our accelerated algorithms consistently outperform the corresponding non-accelerated counterparts.
Revised: Moreover, under different training settings, our accelerated algorithms consistently outperform the corresponding non-accelerated counterparts.

Original: Speciﬁcally, compared the default AdamW optimizer on both ViT and PoolFormer, our accelerated AdamW-Win respectively makes about 1.0%, 0.9%, 1.0% average improvement under the two training epoch settings on ViT-S, ViT-B and PoolFormer-S12.
Revised: Speciﬁcally, compared the default AdamW optimizer on both ViT and PoolFormer, our accelerated AdamW-Win respectively makes about 1.0%, 0.9%, 1.0% average improvement under the two training epoch settings on ViT-S, ViT-B and PoolFormer-S12.

Original: For Adam-Win and LAMB-Win, one can also observe their remarkable improvements on the three ViT backbones.
Revised: For Adam-Win and LAMB-Win, one can also observe their remarkable improvements on the three ViT backbones.

Original: Moreover, our accelerated SGD-Win also outperforms the Nesterov-accelerated SGD denoted as “SGD-M” by non-trivial margins under all settings.
Revised: Moreover, our accelerated SGD-Win also outperforms the Nesterov-accelerated SGD denoted as “SGD-M” by non-trivial margins under all settings.

Original: All these results are consistent with the observations on ResNets, and they together demonstrate the advantage of our accelerated optimizers for deep network training.
Revised: All these results are consistent with the observations on ResNets, and they together demonstrate the advantage of our accelerated optimizers for deep network training.

Original: [Figure]
Revised: [Figure]

Original: Results Analysis.
Revised: Results Analysis.

Original: Here we investigate the convergence behaviors of our accelerated algorithms, and aim to explain their better test performance over their non-accelerated counterparts.
Revised: Here we investigate the convergence behaviors of our accelerated algorithms, and aim to explain their better test performance over their non-accelerated counterparts.

Original: In Fig.
Revised: In Fig.

Original: 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their non-accelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesterove-accelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig.
Revised: 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their non-accelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesterove-accelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig.

Original: 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their nonaccelerated versions.
Revised: 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their nonaccelerated versions.

Original: Fig.
Revised: Fig.

Original: 3 in Appendix B also reveals SGD-Win and
Revised: 3 in Appendix B also reveals SGD-Win and

Original: Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.
Revised: Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.

Original: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated counterparts under the same computational cost. 
Revised: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated counterparts under the same computational cost. [Figure]

Original: [Table]
Revised: [Table]

Original: Robust Analysis.
Revised: Robust Analysis.

Original: For the only extra hyperparameter ¯ η k in our accelerated algorithms over their non-accelerated counterparts, in experiments, we always set ¯ η k = γη k , where γ =2 determines the relation between the reckless step ¯ η k and the conservative step η k .
Revised: For the only extra hyperparameter ¯ η k in our accelerated algorithms over their non-accelerated counterparts, in experiments, we always set ¯ η k = γη k , where γ = 2 .

Original: Here we investigate the effects of γ to the accelerated algorithms on ResNet50 by taking AdamW-Win and LAMB-Win as examples because of their superior performance.
Revised: Here we investigate the effects of γ to the accelerated algorithms on ResNet50 by taking AdamW-Win and LAMB-Win as examples because of their superior performance.

Original: Tableshows the stable performance of AdamW-Win and LAMB-Win when tuning γ in a relatively large range, thus testifying the robustnessof AdamW-Win and LAMB-Win to the hyper-parameter γ .
Revised: Table 4 shows the stable performance of AdamW-Win and LAMB-Win when tuning γ in a relatively large range, thus validating their robustness to the hyper-parameter γ .

Original: 5. R ESULTS ON N ATURAL L ANGUAGE M ODELING T ASKS
Revised: 5. R ESULTS ON N ATURAL L ANGUAGE M ODELING T ASKS

Original: [Table]
Revised: [Table]

Original: Results on LSTM.
Revised: Results on LSTM.

Original: We follow AdaBelief to test our accelerated algorithms via training three-layered LSTM (Schmidhuber et al., 1997) on the Penn TreeBank dataset (Marcinkiewicz, 1994) for 200 epochs.
Revised: We follow AdaBelief to test our accelerated algorithms via training three-layered LSTM (Schmidhuber et al., 1997) on the Penn TreeBank dataset (Marcinkiewicz, 1994) for 200 epochs.

Original: See optimization and training details in Appendix B.
Revised: See optimization and training details in Appendix B.

Original: From Table 7, one can observe that our Win-accelerated algorithms consistently surpass the corresponding nonaccelerated counterparts [Table] and actually bring 1.2 overall average perplexity improvement over the four nonaccelerated counterparts.
Revised: From Table 5, one can observe that our Win-accelerated algorithms consistently surpass the corresponding nonaccelerated counterparts, and actually bring 1.2 overall average perplexity improvement over the four nonaccelerated counterparts.

Original: [Table]
Revised: [Table]

Original: Results on Transformer-XL.
Revised: Results on Transformer-XL.

Original: We adopt a widely used language sequence model, i.e. Transformer-XL (Dai et al., 2019), to further evaluate the performance of our accelerated algorithms.
Revised: We adopt a widely used language sequence model, i.e. Transformer-XL (Dai et al., 2019), to further evaluate the performance of our accelerated algorithms.

Original: Since 1) Adam is the most popular and used optimizer in NLP models, including Transformer-XL, and 2) our limited resource cannot well tune the hyperparameters of other optimizers in Sec.
Revised: Since 1) Adam is the most popular and used optimizer in NLP models, including Transformer-XL, and 2) our limited resource cannot well tune the hyperparameters of other optimizers in Sec.

Original: 5.1, we take Adam as an example to show the superiority of our accelerated algorithms.
Revised: 5.1, we take Adam as an example to show the superiority of our accelerated algorithms.

Original: Follow the ofﬁcial setting of Transformer-XL-base, we use Adam-Win with the default hyper-parameters of Adam on the WikiText-103 dataset.
Revised: Follow the ofﬁcial setting of Transformer-XL-base, we use Adam-Win with the default hyper-parameters of Adam on the WikiText-103 dataset.

Original: See more details in Appendix B.
Revised: See more details in Appendix B.

Original: Table 8 shows that under different training steps, our accelerated Adam-Win always achieves lower test PPL than the ofﬁcial Adam optimizer.
Revised: Table 6 shows that under different training steps, our accelerated Adam-Win always achieves lower test PPL than the ofﬁcial Adam optimizer.

Original: Speﬁcally, it improves 1.5 average test PPL over Adam on the three test cases.
Revised: Speﬁcally, it improves 1.5 average test PPL over Adam on the three test cases.

Original: All these results are consistent with observations on vision tasks, and they together demonstrate the advantages of our accelerated algorithms.
Revised: All these results are consistent with observations on vision tasks, and they together demonstrate the advantages of our accelerated algorithms.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we adopt proximal point method to derive a weight-decay-integrated Nesterov acceleration for AdamW and Adam, and extend it to LAMB and SGD.
Revised: In this work, we adopt proximal point method to derive a weight-decay-integrated Nesterov acceleration for AdamW and Adam, and extend it to LAMB and SGD.

Original: Moreover, we prove the convergence of our accelerated algorithms, i.e. accelerated AdamW, Adam and SGD, and observe the superiority of the accelerated Adam-type algorithm over the vanilla ones in terms of stochastic gradient complexity.
Revised: Moreover, we prove the convergence of our accelerated algorithms, i.e. accelerated AdamW, Adam and SGD, and observe the superiority of the accelerated Adam-type algorithm over the vanilla ones in terms of stochastic gradient complexity.

Original: Finally, experimental results validate the advantages of our accelerated algorithms.
Revised: Finally, experimental results validate the advantages of our accelerated algorithms.

Original: A PPENDIX
Revised: A PPENDIX

Original: The appendix contains the technical proofs of convergence results and some additional experimental details of the paper entitled “Win: Weight-Decay-Integrated Nesterov Acceleration for Adaptive

Original: Gradient Algorithms”.

Original: It is structured as follows.
Revised: The appendix is structured as follows.

Revised: In Appendix A, we provide more details of LAMB and Winaccelerated LAMB.

Original:  Appendix B provides more experimental details, such as hyper-parameter settings of the four accelerated algorithms and the ofﬁcial data augmentations.
Revised: Then, Appendix B provides more experimental details, such as hyper-parameter settings of the four accelerated algorithms and the ofﬁcial data augmentations.

Original: In Appendix C, we deﬁne some necessary notations for our analysis.
Revised: In Appendix C, we deﬁne some necessary notations for our analysis.

Original: Then Appendix D provides some auxiliary lemmas throughout this document.
Revised: Then Appendix D provides some auxiliary lemmas throughout this document.

Original: Then Appendix E presents the proof of the convergence results in Sec. 1, i.e., the proof of Theorems 1 and 2.
Revised: Then Appendix E presents the proof of the convergence results in Sec. 1, i.e., the proof of Theorems 1 and 2.

Original: Finally, Appendix G provides the proofs of some auxiliary lemmas in Appendix D.
Revised: Finally, Appendix G provides the proofs of some auxiliary lemmas in Appendix D.

Original: A M ORE D ETAILS OF LAMB AND W IN - ACCELERATED LAMB
Revised: A M ORE D ETAILS OF LAMB AND W IN - ACCELERATED LAMB

Original: Here we introduce more details of vanilla LAMB (You et al., 2019) and our Win-accelerated LAMB.
Revised: Here we introduce more details of vanilla LAMB (You et al., 2019) and our Win-accelerated LAMB.

Original: Speciﬁcally, Algorithm 2 and 3 respectively summarize the algorithmic steps of LAMB and Winaccelerated LAMB.
Revised: Speciﬁcally, Algorithm 2 and 3 respectively summarize the algorithmic steps of LAMB and Winaccelerated LAMB.

Original: By comparison, we can ﬁnd two differences.

Original: The ﬁrst one is that LAMB (You et al., 2019) scales (cid:16) m k √ v k

Original: +

Original: ν

Original: + λ x k (cid:17) via u k = (cid:107)

Original: x k (cid:107) 2 (cid:107) m

Original: k / √ v k

Original: + ν + λ x k (cid:107) 2 (cid:16) m

Original: k √ v k

Original: +

Original: ν

Original: + λ x k (cid:17) , while Winaccelerated LAMB scales

Original: m k √ v k + ν through u k = (cid:107) x k (cid:107) 2 (cid:107) m k / √ v k + ν (cid:107) 2 m k √ v k

Original: + ν .

Original: We scale m k / √ v k + ν instead of (cid:16) m k √ v k

Original: +

Original: ν

Original: + λ x k (cid:17) in LAMB, as our scaling can be repeatedly used to update our two sequences x k and z k .

Original: The second difference is that Win-accelerated LAMB has one extra acceleration step, i.e. the 7-th step in Algorithm 3, while LAMB has not.

Original: B M ORE E XPERIMENTAL D ETAILS
Revised: B M ORE E XPERIMENTAL D ETAILS

Original: Due to space limitation, we defer the experimental details, such as hyper-parameter settings of the four accelerated algorithms, and their ofﬁcial augmentations in (He et al., 2016) and (Wightman et al., 2021), to this section.
Revised: Due to space limitation, we defer the experimental details, such as hyper-parameter settings of the four accelerated algorithms, and their ofﬁcial augmentations in (He et al., 2016) and (Wightman et al., 2021), to this section.

Original: For accelerated algorithms, including AdamW-Win, LAMB-Win, Adam-Win and SGD-Win, always share the default optimizer-inherent hyper-parameters of the vanilla optimizers and its reckless step ¯ η k is always 2 × larger than its conservative step η k for all iterations, i.e. ¯ η k = 2 η k .
Revised: For accelerated algorithms, including AdamW-Win, LAMB-Win, Adam-Win and SGD-Win, always share the default optimizer-inherent hyper-parameters of the vanilla optimizers and its reckless step ¯ η k is always 2 × larger than its conservative step η k for all iterations, i.e. ¯ η k = 2 η k .

Original: For AdamWWin, LAMB-Win, Adam-Win, their ﬁrst- and second-order moment parameters β 1 and β 2 are set to the default values β 1 = 0 .
Revised: For AdamWWin, LAMB-Win, Adam-Win, their ﬁrst- and second-order moment parameters β 1 and β 2 are set to the default values β 1 = 0 .

Original: 9 and β 2 = 0 .
Revised: 9 and β 2 = 0 .

Original: 999 used in AdamW, LAMB and Adam.
Revised: 999 used in AdamW, LAMB and Adam.

Original: For LAMBWin, its other key parameters, such as “grad averaging” and “trust clip”, also adopt the default ones in vanilla LAMB.
Revised: For LAMBWin, its other key parameters, such as “grad averaging” and “trust clip”, also adopt the default ones in vanilla LAMB.

Original: For SGD-Win, it uses the default momentum parameter 0.9 and set dampening parameter as 0.0 used in vanilla SGD. {+Algorithm 2: al. Input: initialization x 0 = z 
+}
Revised: For SGD-Win, it uses the default momentum parameter 0.9 and set dampening parameter as 0.0 used in vanilla SGD. {+Algorithm 2: al. Input: initialization x 0 = z 
+}

Revised: = 0 , step size { ( η k , ¯ η k ) } Tk =0 , moment parameters { β 1 , β 2 } .

Original: Settings on ResNet18.
Revised: Output: ¯ x uniformly seleted from { x k } Tk =0 . 1 while k < T do g k = 1 b (cid:80) bi =1 ∇ f ( z k ; ζ i ) m k = (1 − β 1 ) m k − 1 + β

Original: {+1 g k /* m 0 = g 
+}
Revised: {+1 g k /* m 0 = g 
+}

Revised: */ v k = (1 − β 2 ) v k − 1 + β 2 g 2 k

Revised: * v 0 = g 20 */ u k = (cid:107) x k (cid:107) 2 (cid:107) m k / √ v k + ν

Revised: + λ x k (cid:107) 2 (cid:16) m

Revised: k √ v k

Revised: +

Revised: ν

Original: {++ λ x k (cid:17) x k +
+}
Revised: {++ λ x k (cid:17) x k +
+}

Revised: = x

Revised: k − η k u k 7 end while

Revised: Input: initialization x 0 =

Original: {+z 
+}
Revised: {+z 
+}

Revised: = 0 , step size { ( η k , ¯ η k ) } Tk =0 , moment parameters { β 1 , β 2 } .

Revised: Output: (¯ x , ¯ z ) uniformly seleted from { ( x k , z k ) } Tk =0 .

Revised: 1 while k < T do g k = 1 b (cid:80) bi =1 ∇ f

Revised: ( z k ; ζ i ) m k

Revised: = (1 − β 1 ) m k − 1 + β

Original: {+1 g k /* m 0 = g 
+}
Revised: {+1 g k /* m 0 = g 
+}

Revised: */ v k = (1 − β 2 ) v k − 1 + β 2 g 2 k

Original: {+* v 0 = g 2 
+}
Revised: {+* v 0 = g 2 
+}

Revised: * / u k = (cid:107) x k (cid:107) 2 (cid:107) m k / √ v k

Revised: + ν + λ k x k (cid:107) 2 (cid:0) m k √ v k

Revised: + ν + λ k x k (cid:1) x

Revised: Settings on ResNet18.

Original: Here we follow the conventional supervised training setting used in
Revised: Here we follow the conventional supervised training setting used in

Original: ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Deng et al., 2009).
Revised: ResNets (He et al., 2016) and evaluate our accelerated algorithms on ImageNet (Fei-Fei, 2009).

Original: For data augmentation in (He et al., 2016), it uses random crop and horizontal ﬂipping with probability 0.5.
Revised: For data augmentation in (He et al., 2016), it uses random crop and horizontal ﬂipping with probability 0.5.

Original: For warm-up epochs, for all four accelerated algorithms, we set it as 5.0.
Revised: For warm-up epochs, for all four accelerated algorithms, we set it as 5.0.

Original: For base learning rate, we respectively set it as 3 × 10 − 3 , 5 × 10 − 3 , 3 × 10 − 3 , and 1.2 for AdamW-Win,
Revised: For base learning rate, we respectively set it as 3 × 10 − 3 , 5 × 10 − 3 , 3 × 10 − 3 , and 1.2 for AdamW-Win,

Original: LAMB-Win, Adam-Win and SGD-Win.
Revised: LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: For weight decay, we respectively set it as 5 × 10 − 2 , 5 × 10 − 2 , 10 − 6 , and 10 −for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: For weight decay, we respectively set it as 5 × 10 − 2 , 5 × 10 − 2 , 10 − 6 , and 10 −for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: On ResNet18, all algorithms are trained for 90 epochs with minibatch size 512 by following the conventional setting.
Revised: On ResNet18, all algorithms are trained for 90 epochs with minibatch size 512 by following the conventional setting.

Original: Settings on ResNet50&101.
Revised: Settings on ResNet50&101.

Original: For these two networks, we use “A2 training recipe” in (Wightman et al., 2021) to train them, since this training setting uses stronger data augmentation and largely improves CNNs’ performance.
Revised: For these two networks, we use “A2 training recipe” in (Wightman et al., 2021) to train them, since this training setting uses stronger data augmentation and largely improves CNNs’ performance.

Original: Speciﬁcally, the data augmentation in (Wightman et al., 2021) uses random crop, horizontal ﬂipping with probability, Mixup with parameter 0.1 (Zhang et al., 2018),
Revised: Speciﬁcally, the data augmentation in (Wightman et al., 2021) uses random crop, horizontal ﬂipping with probability, Mixup with parameter 0.1 (Zhang et al., 2018),

Original: CutMix with parameter 1.0 and probability 0 .
Revised: CutMix with parameter 1.0 and probability 0 .

Original: 5 (Yun et al., 2019), and RandAugment (Cubuk et al., 2020) with M = 7 , N = 2 and MSTD = 0 .
Revised: 5 (Yun et al., 2019), and RandAugment (Cubuk et al., 2020) with M = 7 , N = 2 and MSTD = 0 .

Original: Moreover, it often use binary cross-entropy (BCE) loss for training.
Revised: Moreover, it often use binary cross-entropy (BCE) loss for training.

Original: On both ResNet50 and ResNet101, for base learning rate, we respectively set it as 2 × 10 − 3 , 8 × 10 − 3 , 1 × 10 − 3 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for base learning rate, we respectively set it as 2 × 10 − 3 , 8 × 10 − 3 , 1 × 10 − 3 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.

Original: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 5, 10, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 5, 10, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Settings on ViT and PoolFormer.
Revised: Settings on ViT and PoolFormer.

Original: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021; Yu et al., 2021).
Revised: We follow the widely used ofﬁcial training setting of ViTs (Touvron et al., 2021; Yu et al., 2021).

Original: For this setting, data augmentation includes random crop, horizontal ﬂipping with probability, Mixup with parameter 0.8 (Zhang et al., 2018), CutMix with parameter 1.0 and probability 0 .
Revised: For this setting, data augmentation includes random crop, horizontal ﬂipping with probability, Mixup with parameter 0.8 (Zhang et al., 2018), CutMix with parameter 1.0 and probability 0 .

Original: 5 (Yun et al., 2019), RandAugment (Cubuk et al., 2020) with M = 9 , N =and MSTD = 0 .
Revised: 5 (Yun et al., 2019), RandAugment (Cubuk et al., 2020) with M = 9 , N =and MSTD = 0 .

Original: 5 , and Random Erasing with parameter p = 0 .
Revised: 5 , and Random Erasing with parameter p = 0 .

Original: For training loss, we use cross entropy loss.
Revised: For training loss, we use cross entropy loss.

Original: On both ViT-S and ViT-B, for base learning rate, we respectively set it as 2 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 4 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ViT-S and ViT-B, for base learning rate, we respectively set it as 2 × 10 − 3 , 5 × 10 − 3 , 1 × 10 − 4 , and 0.8 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for weight decay, we respectively set it as 5 × 10 − 2 , 2 × 10 − 2 , 10 − 5 , and 5 × 10 − 4 for AdamW-Win, LAMBWin, Adam-Win and SGD-Win.

Original: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 60, 30, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On both ResNet50 and ResNet101, for warm-up epoch number, we respectively set it as 5, 60, 30, 5 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: For
Revised: For

Original: AdamW-Win, following the default setting in AdamW, its minibatch size is 1024 for ViT-S andfor ViT-B.
Revised: AdamW-Win, following the default setting in AdamW, its minibatch size is 1024 for ViT-S andfor ViT-B.

Original: For all other accelerated optimizer, their minibatch sizes are always 1024.
Revised: For all other accelerated optimizer, their minibatch sizes are always 1024.

Original: Settings on LSTM.
Revised: Settings on LSTM.

Original: On LSTM, for base learning rate, we respectively set it as 1 × 10 − 3 , 1 × 10 − 2 , 1 × 10 − 2 , and 15.0 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.
Revised: On LSTM, for base learning rate, we respectively set it as 1 × 10 − 3 , 1 × 10 − 2 , 1 × 10 − 2 , and 15.0 for AdamW-Win, LAMB-Win, Adam-Win and SGD-Win.

Original: Moreover, we follow the default setting and divide the learning rate by 10 at epoch 100 and 145.
Revised: Moreover, we follow the default setting and divide the learning rate by 10 at epoch 100 and 145.

Original: For weight decay, we respectively set it as 2 × 10 − 2 , 5 × 10 − 2 , 1 . 8 × 10 − 6 , and 2 × 10 − 5 for AdamW-Win, LAMB-Win,
Revised: For weight decay, we respectively set it as 2 × 10 − 2 , 5 × 10 − 2 , 1 . 8 × 10 − 6 , and 2 × 10 − 5 for AdamW-Win, LAMB-Win,

Original: Adam-Win and SGD-Win.
Revised: Adam-Win and SGD-Win.

Original: We do not utilize the warmup strategy in this experiment.
Revised: We do not utilize the warmup strategy in this experiment.

Original: Following the default setting, we set minibatch size as 20.
Revised: Following the default setting, we set minibatch size as 20.

Original: Settings on Transformer-XL.
Revised: Settings on Transformer-XL.

Original: On Transformer-XL, for base learning rate, we set it as 4 × 10 − 4 for
Revised: On Transformer-XL, for base learning rate, we set it as 4 × 10 − 4 for

Original: Adam-Win.
Revised: Adam-Win.

Original: Moreover, we follow the default setting and use cosine learning rate decay.
Revised: Moreover, we follow the default setting and use cosine learning rate decay.

Original: For weight decay, we set it as 10 − 6 for Adam-Win.
Revised: For weight decay, we set it as 10 − 6 for Adam-Win.

Original: For warm-up steps, we set it as 2000.
Revised: For warm-up steps, we set it as 2000.

Original: Following the default setting, we set minibatch size as 60 × 4 .
Revised: Following the default setting, we set minibatch size as 60 × 4 .

Revised: Test accuracy curves of SGD-Win and Adam-Win on ResNet18.

Revised: Here we investigate the convergence behaviors of our accelerated algorithms and hope to explain their better test performance over their non-accelerated counterparts.

Revised: In each sub-ﬁgure pair of Fig. 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their nonaccelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesteroveaccelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig. 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their corresponding non-accelerated versions.

Revised: Fig.

Revised: 3 in Appendix B also reveals SGD-Win and Adam-Win enjoy faster convergence than

Original: Training Epochs
Revised: Training Epochs

Original: T e s t
Revised: T e s t

Original: T o pA cc u r a c y
Revised: T o pA cc u r a c y

Original: SGD-M
Revised: SGD-M

Original: SGD-Win
Revised: SGD-Win

Original: Adam
Revised: Adam

Original: Adam-Win Test accuracy curves of SGD-Win and Adam-Win on ResNet18.
Revised: Adam-Win their non-accelerated counterparts in terms of test accuracy.

Original: Here we investigate the convergence behaviors of our accelerated algorithms and hope to explain their better test performance over their non-accelerated counterparts.

Original: In each sub-ﬁgure pair of Fig. 1, we plot the curves of training and test losses along with the training epochs on ResNet18 and ViT-B. One can ﬁnd that our accelerated algorithms, e.g. AdamW-Win, show much faster convergence behaviors than their nonaccelerated counterparts, e.g. AdamW. Moreover, SGD-Win also converges faster than Nesteroveaccelerated SGD, i.e. SGD-M. We also plot the curves of test accuracy in Fig. 2, showing the superior convergence speed of AdamW-Win and LAMB-Win over their corresponding non-accelerated versions.

Original: Fig.

Original: 3 in Appendix B also reveals SGD-Win and Adam-Win enjoy faster convergence than their non-accelerated counterparts in terms of test accuracy.

Original: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated algorithms under the same computational cost.
Revised: So these faster convergence behaviors could contribute to our accelerated algorithms for their higher performance over non-accelerated algorithms under the same computational cost.

Original: C N OTATIONS
Revised: C N OTATIONS

Original: Here we ﬁrst give some important notations used in this document.
Revised: Here we ﬁrst give some important notations used in this document.

Original: For brevity, we let
Revised: For brevity, we let

Original: [Equation]
Revised: [Equation]

Original: Since we have (cid:107) m k (cid:107) ∞ ≤ c ∞ and ν ≤ (cid:107) 
Revised: Since we have (cid:107) m k (cid:107) ∞ ≤ c ∞ and ν ≤ (cid:107) v

Revised: i + ν (cid:107) ∞ ≤ c 2 ∞ + ν in Lemma 3 (see Appendix D)

Original: [Equation]
Revised: [Equation]

Original: i
Revised: for brevity

Original: [Equation]
Revised: [Equation]

Original: ν (cid:107) ∞ ≤ c 2 ∞ + ν in Lemma 3 (see Appendix D), for brevity, let
Revised: let

Original: [Equation]
Revised: [Equation]

Original: Also we deﬁne
Revised: Also we deﬁne

Original: [Equation]
Revised: [Equation]

Original: Next, we introduce an virtual sequence { y k } into the algorithm.
Revised: Next, we introduce an virtual sequence { y k } into the algorithm.

Original: In this way, we can rewrite the update steps in Algorithm 1 in the manuscript as its equivalent form (12): 
Revised: In this way, we can rewrite the update steps in Algorithm 1 in the manuscript as its equivalent form (12): [Equation]

Original: [Equation]
Revised: [Equation]

Original: where m 0 =
Revised: where m 0 =

Original: g 0 and v 0 = g 2 0 .
Revised: g 0 and v 0 = g 20 .

Original: For analysis, we further deﬁne
Revised: For analysis, we further deﬁne

Original: [Equation]
Revised: [Equation]

Original: [Equation] where λ k = λ 2 (cid:80) ki =1 (cid:0) 1 − µ 2 (cid:1) i ( k> 0) and λ 0 = 0 in which µ = β 2 c 2 ∞ ν .
Revised: where λ k = λ (1 − µ ) k in which µ = β 2 c 2 ∞ δ .

Original: In the following, we mainly use these notations to ﬁnish our proofs.
Revised: In the following, we mainly use these notations to ﬁnish our proofs.

Original: D A UXILIARY L EMMAS
Revised: D A UXILIARY L EMMAS

Original: Before giving our analysis, we ﬁrst provide some important lemmas.
Revised: Before giving our analysis, we ﬁrst provide some important lemmas.

Original: Lemma 3.
Revised: Lemma 3.

Original: Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1{++
+} =[-
-]
Revised: That is, x k {++
+} =[-
-]

Original: [Equation]{++ λ k η k (cid:16) x k − η k m k s k (cid:17) , y k +
 = z k − ¯ η k m k s k , z k +1 = ¯ η k τ k x k +1 + η k τ k y k+}
Revised: {++ λ k η k (cid:16) x k − η k m k s k (cid:17) , y k +
 = z k − ¯ η k m k s k , z k +1 = ¯ η k τ k x k +1 + η k τ k y k+}

Original:  k
Revised: +1 , s k = √ v k

Original: [Equation]
Revised: + ν .

Original: Then { ( m k , s k ) } satisﬁes Assume c s, ∞ ≤ (cid:107) g k (cid:107) ∞ ≤ c ∞ , then we have 
Revised: Then { ( m k , s k ) } satisﬁes Assume c s, ∞ ≤ (cid:107) g k (cid:107) ∞ ≤ c ∞ , then we have (cid:107) m k (cid:107) ∞ ≤ c ∞ , (cid:107) v i + ν (cid:107) ∞ ≤ c 2 ∞ + ν, β 2 c 2 ∞ 2( c 2 s, ∞ + ν ) ≤ (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: s k s k + (cid:13)(cid:13)(cid:13)(cid:13) ∞ < 1 + β 2 c 2 ∞ 2( c 2 s, ∞ + ν ) .

Original: See its proof in Appendix G.1.
Revised: See its proof in Appendix G.1.

Original: Lemma 4.
Revised: Lemma 4.

Original: (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 =
Revised: That is, x k +1 =

Original: [Equation]
Revised: [Equation]

Original: √ v k + ν . Then { x k } satisﬁes [Equation] [Equation]
Revised: k (cid:16) x k − η k m k s k (cid:17) , y k +1 =

Original: Lemma 5. (Xie et al., 2022) Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: z k − ¯ η k m k s k , z k +1 = ¯ η k τ k x k +1 + η k τ k y k

Original: That is, x k +1  = 
Revised: +1 , s k = √ v k

Original: [Equation] √ v k + ν .
Revised: + ν .

Original: Then { z k } satisﬁes
Revised: Then { z k } satisﬁes

Original: [Equation]
Revised: [Equation]

Original: Lemma 6.
Revised: Lemma 5.

Original: Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1{++
+} = 1 1+ λη k 
Revised: That is, x k {++
+} = + λ k η

Revised: k

Original: [Equation]
Revised: [Equation]

Original: By setting η k = η , ¯ η k = ¯ η , β 1 ,k
Revised: By setting η k = η , ¯ η k = ¯ η , β 1 ,k

Original: =
Revised: =

Original: β
Revised: β

Original: and β 2 ,k = β 2 , then we havey
Revised: and β 2 ,k = β 2 , then we have

Original: [Equation]
Revised: [Equation]

Original: w i s i (cid:107) y k +1 − (1 + λ ¯ η ) x k +1 (cid:107) 2 ≤ ρ k +1 (¯ η − η )(1 − ητ ) (1 + λη )k (cid:88) i =  ρ i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: [Equation]

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) 
Revised: [Equation] (cid:13)(cid:13)(cid:13)(cid:13) w

Original: (cid:107) z k +1 − x k +1 (cid:107) 2 ≤ ρ k +1 ητ (¯ η − η ) 2 (1 − ητ ) (1 + λη )k (cid:88) i =  ρ i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: [Equation]

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , (cid:107)
Revised: (cid:13)(cid:13)(cid:13)(cid:13) , (cid:107)

Original: z k +1 − z k (cid:107) 2 ≤ 2¯ η(1 + λη )(cid:13) (cid:13)(cid:13) (cid:13) w k s k
Revised: [Equation] ≤ 2¯ [Equation]

Original: (cid:13) (cid:13)(cid:13) (cid:13) + 2 ρ k
Revised: [Equation] (cid:13)(cid:13)(cid:13)(cid:13)

Original: [-+
 ¯-]
Revised: [-+
 ¯-][Equation]

Original: η 2 τ 2 (¯ η − η )(1 − ητ ) k 
Revised: k [Equation] (cid:13)(cid:13)(cid:13)(cid:13) [Equation]

Original: (cid:88) i =
Revised: (cid:88) i =

Original:  ρ
Revised:  ρ

Original: i + (cid:13) (cid:13)(cid:13) (cid:13)
Revised: [Equation] + [Equation] (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13) (cid:13)(cid:13) (cid:13) , where ρ k +

Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , where ρ k +

Original: =
Revised: =

Original: ητρ k and ρ 1 = 1 {+and ρ 
 = 0+} .
Revised: ητ k − 1 ρ k , ρ 1 = 1 {+and ρ 
 = 0+} .

Original: See its proof in Appendix G.2.
Revised: See its proof in Appendix G.2.

Original: Lemma 7.
Revised: Lemma 6.

Original: Suppose the sequence { x k , y k , z k } are updated by Eqn.
Revised: Suppose the sequence { x k , y k , z k } are updated by Eqn.

Original: That is, x k +1 = 1 1+ λη k
Revised: That is, x k +1 = 1 1+ λη k

Original: [Equation]
Revised: [Equation]

Original: η

Original: [Equation]
Revised: [Equation]

Original: m

Original: [Equation]

Original: s k (cid:17) , y k +1 =

Original: z k − ¯ η k m k s k , z k +1 =

Original: ¯ η k τ k x k +1 + η k τ k y k

Original: +1 , s k = √ v k

Original: + ν .

Original: By setting η k = η , ¯ η k = ¯ η , β 1 ,k
Revised: By setting η k = η , ¯ η k = ¯ η , β 1 ,k

Original: =
Revised: =

Original: β
Revised: β

Original: and β 2 ,k = β 2 , then we have
Revised: and β 2 ,k = β 2 , then we have

Original: E (cid:104) (cid:107) m k − ∇ F ( x k ) (cid:107) 2 (cid:105) ≤ 2(1 − β 1 ) E (cid:104) (cid:107) m k − 1 − ∇ F ( z k − 1 ) (cid:107) 2 (cid:105)
Revised: E (cid:104) (cid:107) m [Equation]

Original: + 2Π k (1 − [-β 1 ) 
-]
Revised: − [-β 1 ) 
-][Equation] x [Equation]

Original: L 2 β + 2 β 21 σb + 2 L Π (cid:48) k , where Π k := 2¯ η(1 + λη ) [-2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 
-]
Revised: ) [-2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 
-](cid:107) [Equation] ≤ [Equation]

Original: + 2 ρ k ¯ η 2 τ 2 (¯ η − η )(1 − ητ )
Revised: 1 − β [Equation] β [Equation]

Original: k − 1 (cid:88) i =
Revised: b [Equation] Π (cid:48)

Original: {+[Equation] where Π k := 2¯ η(1 + λ k − 1 η ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 
 + 2 ρ k ¯ η 2 (¯ η − η ) 2 τ 2 k − 1 (1 + λ k − 1 η )k − 1 (cid:88) i =+}  ρ 
Revised: {+[Equation] where Π k := 2¯ η(1 + λ k − 1 η ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 
 + 2 ρ k ¯ η 2 (¯ η − η ) 2 τ 2 k − 1 (1 + λ k − 1 η )k − 1 (cid:88) i =+}  ρ i +1 (1 − ητ

Original: i  +  (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13)  Π (cid:48)
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , Π [Table]

Original: k :
Revised: k :=

Revised: [Table]

Revised: [Table]

Original: = ρ k ητ (¯ η − η)(1 − ητ )(1 + λη )k − 1 (cid:88)
Revised: η ( [Table] η − η )k [Table]

Original: i = ρ
Revised: i [Table] ρ

Original: i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: [Table] [Table] − [Table]

Revised: [Table]

Revised: λ

Original:  w i s i (cid:13)(cid:13)(cid:13)(cid:13) , where ρ k +1 ={++
+}
Revised: [Table] η ) [Table] w i s i (cid:13)(cid:13)(cid:13)(cid:13) , where ρ k {++
+}

Original: ητρ k and ρ 1 = 1 {+and ρ 
 = 0+} .
Revised: = ητ k − 1 ρ k , ρ 1 = 1 {+and ρ 
 = 0+} .

Original: see its proof in Appendix G.3.
Revised: see its proof in Appendix G.3.

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Recall our deﬁnition(cid:101) F
Revised: Recall our deﬁnition F

Original: [Equation]
Revised: [Equation]

Original: z [Equation] ) = [Equation]
Revised: , in the (13).

Original: + λ [Equation] = E ζ [ f 
Revised: By using the smoothness of f ( θ ; ζ )

Original: [Equation]
Revised: [Equation]

Original: z ;
Revised: we can obtain

Original: [Equation] ] + [Equation] k 
Revised: F k +1 ( x k +

Original: [Equation]
Revised: [Equation]

Original: z [Equation] 2 s k , in the (13). By using the smoothness of f ( θ ; ζ )
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: we can obtain (cid:101)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: ≤

Revised: [Equation]

Revised: +

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds since Lemma 3 proves (cid:13)(cid:13)(cid:13) s ks k
Revised: where (cid:172) holds since Lemma 3 proves (cid:13)(cid:13)(cid:13) s k

Original: [Equation]
Revised: [Equation]

Revised: k +

Original: [Equation]
Revised: [Equation]

Original: ∞ ∈ [1 − µ, 1 + µ ] ( ∀ p ∈
Revised: ∞ ∈ [1 − µ, 1 + µ ] ( ∀ p ∈

Original: [0 , 1]) in which µ = β 2 c 2 ∞ ν ; (cid:173) holds because in Lemma 4, we have
Revised: [0 , 1]) in which µ = β 2 c 2 ∞ ν ; (cid:173) holds because λ k = λ k +1 1 − µ and (cid:107) [Equation] =

Original: [Equation]
Revised: [Equation]

Original: λ
Revised: + [Equation] (cid:174) holds, because

Original: [Equation]
Revised: [Equation]

Original: (cid:174) holds, since we set η k ≤ c 21 (1+ λη k ) 2 c 2 ( L+ λc 1 )
Revised: ≤ c 2 := ( c 2 ∞ + ν ) 0 . (cid:175) holds, since we set η k ≤ c 21 (1+ λ k η k ) 2 c 2 ( L

Original:  such that
Revised: + λ k c 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: From Lemma 7, by setting η k = η , ¯ η k = ¯ η and β 1 ,k = β 1 , we have
Revised: From Lemma 6, by setting η k = η , ¯ η k = ¯ η and β 1 ,k = β 1 , we have

Revised: E (cid:104) (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  where {+Π k := 2¯ η(1 + λ k − 1 η ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 
 + 2 ρ k ¯ η 2 (¯ η − η ) 2 τ 2 k − 1 (1 + λ k − 1 η )k − 1 (cid:88) i =  ρ i +1 (1 − ητ i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)+}
Revised: ) where {+Π k := 2¯ η(1 + λ k − 1 η ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w k −s k −(cid:13)(cid:13)(cid:13)(cid:13) 
 + 2 ρ k ¯ η 2 (¯ η − η ) 2 τ 2 k − 1 (1 + λ k − 1 η )k − 1 (cid:88) i =  ρ i +1 (1 − ητ i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)+}

Original: [Equation]
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) , Π (cid:48) k := τ k − 1 ρ k η (¯ η − η )k − 1 (cid:88) i =  ρ i +1 (1 − ητ

Original:  (cid:13)(cid:13)(cid:13)(cid:13) 
Revised: i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w i

Original: [Equation]
Revised: [Equation]

Original: Here ρ k +

Revised: Here ρ k +

Original: =
Revised: =

Original: ητρ k and ρ 1 = 1 {+and ρ 
 = 0+} .
Revised: ητ k − 1 ρ k , ρ 1 = 1 {+and ρ 
 = 0+} .

Original: By considering c 2 ≥ (cid:107) s k (cid:107) ∞ ≥ c 1 , we have Πk ≤
Revised: By considering c 2 ≥ (cid:107) s k (cid:107) ∞ ≥ c 1 , we have Π

Original: [Equation]
Revised: [Equation]

Revised: Therefore, by plugging the results in Eqn.

Revised: (14) into the upper bound of F k +1 ( x k +1 ) , we have

Revised: F k +

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have
Revised: k

Original: [Equation]
Revised: [Equation]

Revised: 17) where (cid:172) uses the fact that 0 < λ k ≤ λ .

Original: From Lemma
Revised: Then, from Lemma

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where we use the results in Lemma 6 that
Revised: where we use the results in Lemma 5 that

Original: [Equation]
Revised: [Equation]

Original: Then we add Eqn.
Revised: Then we add Eqn.

Original: (18) and α × (17) as follows:
Revised: (17) and α × (18) as follows:

Original: [Equation]
Revised: [Equation]

Original: Then by setting α = η (1 − β 1 ) c 1 β 1 (1+ λη ) and G( x k +1 ) =
Revised: Then by setting α = η ( [Equation] [Equation] 1 βand G

Original: [Equation]
Revised: [Equation]

Original: F k
Revised: E ζ [ f [Equation] [Equation] ] +

Revised: [Equation]

Revised: (cid:107)

Original: [Equation]
Revised: [Equation]

Original:  η 
Revised: k + η [Equation] [Equation] E

Original: [Equation]
Revised: [Equation]

Original:  we can obtain
Revised: (cid:107) m k − ∇ F ( x k ) (cid:107) 2 (cid:105) , we can obtain

Original: [Equation]
Revised: [Equation]

Revised: where (cid:172) uses the fact that 0 < λ k ≤ λ .

Original: Then summing the above inequality from k = 0 to k = T − 1 gives
Revised: Then summing the above inequality from k = 0 to k = T −and using 0 < λ k ≤ λ give

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original:  where ∆ = F ( x 0 ) − F ( x (cid:63) ) ;
Revised: [Table] where ∆ = F ( x 0 ) − F ( x (cid:63) ) ;

Original: x − 1 and m − 1 are two virtual points which satisfy m − 1 = ∇ F ( x − 1 ) .
Revised: x − 1 and m − 1 are two virtual points which satisfy m − 1 = ∇ F ( x − 1 ) .

Original: Now we try to bound (cid:80) T − 1 k =

Revised: Now we try to bound (cid:80) T − 1 k =

Original: ¯Π
Revised: ¯Π

Original: k
Revised: k

Original: and
Revised: and

Original: (cid:80) T − 1 k =

Revised: (cid:80) T − 1 k =

Original: ¯Π k .
Revised: ¯Π k .

Original: Firstly
Revised: Firstly, we have

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: [Table] [Equation] [Table] [Equation]

Original: T − 1 (cid:88) k
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: =
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: (cid:107) w k − 1 (cid:107)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: (cid:105)
Revised: [Equation] [Table] [Equation] , where (cid:172) holds since 0 ≤ λ k ≤ λ ; (cid:173) holds, since 1) for i

Original: {+≥ k we have 
+} + 2¯ η
Revised: {+≥ k we have 
+} + λ i − 1 η 1+ λ k η ≤ + λ k − 1 η 1+ λ k η = 1+ λ k − 1 η

Original: [Equation]
Revised: [Equation]

Original: τ
Revised: ≤

Original: [Equation] 
Revised: [Equation] and 2) [Equation] 1 1 − ητ whose minimum is at λ i − 1 = 0 and τ = 1 η +¯ η ; (cid:174) holds, since (cid:80) T − 1 i =

Original: (¯ η − η )c
Revised: k ρ

Original: [Equation]{+i τ 2 i − 1 = 1 η (cid:80) T − 1 i = k ρ i +
+}
Revised: {+i τ 2 i − 1 = 1 η (cid:80) T − 1 i = k ρ i +
+}

Original: (1 − ητ ) T − 1 (cid:88) k
Revised: τ i − 1 ≤ τη (cid:80) T − 1 i = k

Original: = ρ k (cid:34) k  −  1 (cid:88) i =  ρ
Revised: ρ i +1 ≤ τη ρ k +1 (1 − η T − k τ T − k ) 1 − ητ ≤ τρ k +1 η

Original: i +1 (cid:107)
Revised: (1 − ητ ) ; (cid:175) holds by setting ¯ η = γη . Similarly, we can bound T [Equation]

Original: w i (cid:107)(cid:35) =
Revised: k [Equation]

Original: [Equation]
Revised: [Equation]

Original: c

Original: [Equation] (cid:88) [Equation] =  (cid:104)
Revised: ρ k (cid:34) k − 1 (cid:88) i =  ρ i +1 (cid:107) w

Original: [Equation]
Revised: i

Original: (cid:107) [Equation] (cid:105) [Equation]
Revised: cid:107)(cid:35) ≤ τη (¯ η − η )c 21 (1 − ητ )

Original: [Equation] η )
Revised: T − 1 (cid:88) k

Original: [Equation] c
Revised: = (cid:107) w k

Original: [Equation] (1 − ητ ) T − 1 (cid:88) k
Revised: (cid:107)ρ k + (cid:34) T − 1 (cid:88) i =

Original: =  ρ k+ (cid:34) T − 1 (cid:88) i =
Revised: k

Original: k ρ i (cid:107) w k (cid:107)(cid:35) (cid:172) ≤ 2¯ ηc [Equation] (1 + λη )[Equation]
Revised: ρ i (cid:35) (cid:172) ≤ (¯ η − η )c 21 (1 − ητ )

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: = (cid:104) (cid:107) 
Revised: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) ≤

Original: [Equation]
Revised: η 2 γ 2 ( γ − 1)c 21 (1 + γ )

Original: k − 1 (cid:107)
Revised: T − 1 (cid:88) k

Original: [Equation] (cid:105) + 2¯ η 2 τ (¯ η − η )c 1 η (1 − ητ )
Revised: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) ≤ η 2 ( γ − 1)c

Original: T − 1 (cid:88) k
Revised: T − 1 (cid:88) k

Original: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) (cid:173) [Equation] γ [Equation]
Revised: = (cid:104) (cid:107) w k (cid:107) 2 (cid:105) (20) where (cid:172) holds since 1)

Original: ηc [Equation] (cid:18) (1 + λη ) [Equation]
Revised: ρ k +1 =

Original: + ητ ( γ − 1)(1 − ητ )(cid:19) T − 1 (cid:88) k
Revised: ητ k − 1 ρ k ≤ ητρ k and ρ 1 = 1 and 2) (cid:80) T − 1 i

Original: =(cid:104) (cid:107) w k (cid:107) 2 (cid:105) where (cid:172) holds since 1) (cid:80) T − 1 i = k ρ i = ρ k (1 − η T − k τ T − k ) 1 − ητ ≤ ρ k 1 − ητ and 2) ρ k [-+
-]
Revised: = k ρ i ≤ ρ k (1 − η T − k τ T − k ) 1 − ητ ≤ ρ k 1 − ητ which together give 1 ρ k [-+
-]+1 (cid:104)(cid:80) T − 1 i =

Original: =
Revised: k

Original: ητρ k and ρ  1 = 1 ; (cid:173) holds by setting ¯ η = γη .Similarly, we can bound [Equation]
Revised: ρ i (cid:105) ≤ 1 ρ k +1 ρ k 1 − ητ ≤ 1 ητ 1 1 − ητ ≤ 1 ητ (1 − ητ ) .

Original: Therefore [Equation] we haveT
Revised: Therefore, we have

Original: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds since we choose proper η and β 1 such that
Revised: where (cid:172) holds since we choose proper η and β 1 such that

Original: [Equation]
Revised: [Equation]

Original: Now we discuss how to select η and β 1 such that (19) holds. To begin with, we have 1 (1+ λη ) 2 ≤and ητ ( γ − 1) 2 (1 − ητ ) 2 = 1+ γ  + λγη γ (1+ λη ) ( γ − 1) 2 ≤ 1+ γγ ( γ − 1) 2  < γ2 in which ητ = η η + γη + λγη 2 . 
Revised: (21) where (cid:172) uses γ 3 ( γ − 1) 2 (1+ λτ ) (1+ γ )≤ 1 + γτ = 1 + γη ( γ + 1) and (cid:173) uses γ 3 ( γ − 1) 2 (1+ γ ) 4 < γ . Now we select η and β 1 such that (21) holds:

Original: Then we only need

Original: [Equation]
Revised: [Equation]

Original: So we arrive at
Revised: So we arrive at

Original: [Equation]
Revised: [Equation]

Original: where we set T ≥ 6 c 2 (1+ λη )∆ η(cid:15)and β 1 ≤ c 1 b(cid:15) 2 6 c 2  σ 2 .
Revised: where we set T ≥ 4 c 2 (1+ λη )∆ η(cid:15)and β 1 ≤ c 1 b(cid:15) 2 4 c 2 (1+ λη ) σ 2 .

Original: This result directly bounds
Revised: This result directly bounds

Original: [Equation]
Revised: [Equation]

Original: Moreover, from Lemma6, we have
Revised: Moreover [Equation] from Lemma

Original: [Table]
Revised: [Equation]

Original: (1 − ητ )
Revised: we have [Equation] w i s i (cid:13) (cid:13)(cid:13)(cid:13) 2 (cid:174) = T

Original: k (cid:88) i = ρ
Revised: T − 1 (cid:88) k = Π (cid:48)(cid:48) k , T T − 1 (cid:88) k

Original:  i + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: = (cid:107) z k − x k (cid:107) 2 (cid:172) ≤ 1 T T − 1 (cid:88) k = τ k − 1 ρ k η (¯ η − η )k − 1 (cid:88) i =

Original:  w i s i (cid:13)(cid:13)(cid:13)(cid:13) 2(cid:35) (cid:173) =
Revised: ρ i +1 (1 − ητ i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13) w i s i (cid:13)(cid:13)(cid:13)(cid:13) 2 (cid:173) =

Original: T
Revised: T

Revised: T − 1 (cid:88) k

Revised: = Π (cid:48) k ,

Original: T  − 1 (cid:88) k
Revised: T T − 1 (cid:88) k

Revised: = (cid:107)

Original: {+z k +1 − z k (cid:107) 
+}
Revised: {+z k +1 − z k (cid:107) 
+}

Revised: (cid:172) ≤ 1 T

Revised: T − 1 (cid:88) k

Original: {+= (cid:34) 2¯ η(1 + λ k η ) 
+}
Revised: {+= (cid:34) 2¯ η(1 + λ k η ) 
+}

Revised: + 2 ρ k

Revised: +1 ¯ η 2 (¯ η − η ) 2 τ 2 k (1 + λ k η )k

Revised: (cid:88)

Revised: i =  ρ

Revised: i +1 (1 − ητ

Revised: i − 1 )(1 + λ i η )(cid:35) (cid:13)(cid:13)(cid:13)(cid:13) w k s k

Revised: (cid:13)(cid:13)(cid:13)(cid:13) (cid:173) ≤

Revised: T

Revised: T − 1 (cid:88) k

Original: {+= Π k where ρ k +
+}
Revised: {+= Π k where ρ k +
+}

Revised: =

Revised: ητ k − 1 ρ k , ρ 1 = 1 and ρ 0 = 0 .

Original: = Π k where (cid:172) holds by using Lemma 6; (cid:173) holds by using the deﬁnition in Eqn. 
Revised: where (cid:172) holds by using Lemma 5; (cid:173) holds by using the deﬁnition in Eqn. (15); (cid:174) holds by deﬁning: [Equation] [Table]

Revised: [Equation]

Revised: 23) where (cid:172) holds since

Original: Now remaining task is to upper bound 1 T (cid:80) T − 1  k [-=
-]
Revised: [Equation] whose minimum is at λ i − 1 = 0 and τ = 1 η +¯ η ; (cid:173) holds since 1) ρ k +1 = ητ k − 1 ρ k [-=
-]≤ ητρ k and ρ 1 = 1 and

Revised: [Equation]

Revised: k

Revised: [Equation]

Revised: which together give

Original: Π k and 1 T (cid:80) T − 1 k =
 
Revised: [Equation] ; (cid:174) holds by using 1) 1 T (cid:80) T − 1 k =
 E [Equation]

Original: Π (cid:48) k .
Revised: w [Equation] (cid:107) 2 ≤ 4 (cid:15) 2 in Eqn. (22); 2) we use the results in Eqn.

Revised: (21) to obtain

Revised: [Equation]

Original: From the bound in Eqn.
Revised: From the bound in Eqn.

Original: (15) and the following bound on 1 T (cid:80) T − 1 k {+=
+}
Revised: (16) and the following bound on 1 T (cid:80) T − 1 k {+=
+}

Original: [-=
-]
Revised: [-=
-]¯Π

Original: ¯Π k and 1 T (cid:80) T − 1 k [-=
-] ¯Π
Revised: k and 1 T (cid:80) T − 1 k [-=
-]=0 ¯Π

Original: (cid:48)
Revised: [Equation]

Original: k , we have
Revised: k , we have

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:172) ≤ [Equation] (cid:15)
Revised: c

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds, since 1) 1 T (cid:80) T − 1 k =0 E (cid:107) w k (cid:107) 2 ≤ 4 (cid:15) 2 ; 2) we use the results in Eqn.
Revised: where (cid:172) holds, since 1) 1 T (cid:80) T − 1 k =0 E (cid:107) w k (cid:107) 2 ≤ 4 (cid:15) 2 ; 2) we use the results in Eqn.

Original: (19) to obtain 
Revised: (21) to obtain 2 γ 2 η

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: Besides, we have
Revised: Besides, we have

Original: [Equation]
Revised: [Equation]

Original: where in (cid:172) we use w k = m k + λ  x k ∗ s k .
Revised: where in (cid:172) we use w k = m k + λ k x k ∗ s k .

Original: In this way
Revised: In this way

Original: [Equation]
Revised: [Equation]

Original: we have

Original: [Equation]

Original: From Eqn.

Original: (17), we have

Original: [Equation]

Original: By setting ρ k +1 =

Original: (1 − β 1 ) ρ k and ρ 0 =

Original: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: For all hyper-parameters [Equation] we put their constrains together:
Revised: For all hyper-parameters, we put their constrains together:

Original: [Equation]
Revised: [Equation]

Original: where c 1 = ν 0 .
Revised: where c 1 = ν 0 .

Original: 5 ≤ (cid:107) s k
Revised: 5 ≤ (cid:107) s k

Original: [Equation]
Revised: [Equation]

Original: ∞ ≤
Revised: ∞ ≤

Original: [Equation]
Revised: [Equation]

Original: =
Revised: c 2 ∞ + ν

Original: [Equation]
Revised: [Equation]

Revised: 5 =

Revised: c 2 .

Original: For η , it should satisfy 
Revised: For η , it should satisfy η ≤ min

Original: [Equation]
Revised: [Equation]

Revised: c

Original: [Equation]
Revised: [Equation]

Revised: c

Revised: [Equation]

Revised: λc

Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: Considering(1 − ητ )(1 + λη ) = γ (1+ λη ) 2 1+ γ + λγη ≥ γγ +1 , γ ( γ − 1)( γ +1) ≥ 1 γ and c 1 = ν 0 .
Revised: Considering λη << 1 , 1+ λη 1+(1 − µ ) λη = a ≤ 1 1 − µ , µ is a constant, and c 1 = ν 0 .

Original: 5 << 1 , then we have η≤
Revised: 5 << 1 , then we have η

Original: [Equation]
Revised: [Equation]

Original: where ν is often much smaller than one, and β 1 is very small.
Revised: where ν is often much smaller than one, and β 1 is very small.

Original: For T , we have
Revised: For T , we have

Original: [Equation]
Revised: [Equation]

Original: Now we compute the stochastic gradient complexity.
Revised: Now we compute the stochastic gradient complexity.

Original: For T iterations, the complexity is
Revised: For T iterations, the complexity is

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed.

Original: F P ROOFS OF T HEOREM
Revised: F P ROOFS OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Recall our deﬁnition(cid:101) F k ( θ k )
Revised: Recall our deﬁnition F k ( θ k )

Original: = F ( θ ) + λ k 2 (cid:107) θ (cid:107) 

Revised: = F ( θ ) + λ k 2 (cid:107) θ (cid:107)

Original: =
Revised: =

Original: E ζ [ f ( θ ; ζ )]
Revised: E ζ [ f ( θ ; ζ )]

Original: + λ k 2 (cid:107) θ (cid:107) 22 in the (13).
Revised: + λ k 2 (cid:107) θ (cid:107) 22 in the (13).

Original: By setting β (cid:48) 1 = 1 − β 1 , then we have (cid:107) m k (cid:107) ∞ ≤ c ∞ by using Lemma 3 (see Appendix D).
Revised: By setting β (cid:48) 1 = 1 − β 1 , then we have (cid:107) m k (cid:107) ∞ ≤ c ∞ by using Lemma 3 (see Appendix D).

Original: Also we deﬁne
Revised: Also we deﬁne

Original: [Equation]
Revised: [Equation]

Original:  By using the smoothness of f ( θ ; ζ ) , we can obtain(cid:101) F k +1 ( x k +1 ) [Equation] [Equation]
Revised: Note in the following, we set all λ k = λ. By using the smoothness of f ( θ ; ζ ) , we can obtain

Original: ≤
Revised: F k

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds because in Lemma 4, we have
Revised: where (cid:172) holds because [Equation] (cid:173) holds, because

Original: [Equation]
Revised: [Equation]

Original: where s k could be 1 ; (cid:172) holds, since we set η k ≤  (1+ λη k ) 2( L + λ ) such that
Revised: (cid:175) holds, since we set η k ≤ c 21 (1+ λη k ) 2 c 2 ( L + λc 1 ) such that

Original: [Equation]
Revised: [Equation]

Original: Then in the following, we can directly follow the proof of Theorem 1.
Revised: Then in the following, we can directly follow the proof of Theorem 1.

Original: This is because the only difference between accelerated SGD and AdamW is that SGD has no the second-order moment v k , while AdamW has.
Revised: This is because the only difference between accelerated SGD and AdamW is that SGD has no the second-order moment v k , while AdamW has.

Original: By let s k
Revised: By let s k

Original: = 1 in accelerated AdamW and setting β (cid:48) 1 = 1 − β 1 in accelerated
Revised: = 1 in accelerated AdamW and setting β (cid:48) 1 = 1 − β 1 in accelerated

Original: SGD, then they share the exact the same updating rules.
Revised: SGD, then they share the exact the same updating rules.

Original: So after setting β (cid:48) 1 = 1 − β 1 in accelerated
Revised: So after setting β (cid:48) 1 = 1 − β 1 in accelerated

Original: SGD, to follow the proofs of Theorem 1, we only need to verify whether the auxiliary lemmas and the proof process of Theorem 1 hold for s k
Revised: SGD, to follow the proofs of Theorem 1, we only need to verify whether the auxiliary lemmas and the proof process of Theorem 1 hold for s k

Original: = 1 .
Revised: = 1 .

Original: This is the true case.
Revised: This is the true case.

Original: Please check our auxiliary lemmas, including Lemma 3 ∼ 7, and the proof process of Theorem 1.
Revised: Please check our auxiliary lemmas, including Lemma 3 ∼ 6, and the proof process of Theorem 1.

Original: Consider s k =in accelerated SGD, we have c 1 :
Revised: Consider s k =in accelerated SGD, we have c 1 :

Original: = 1 ≤ (cid:107) s k (cid:107) ∞ ≤ c 2 := 1 .
Revised: = 1 ≤ (cid:107) s k (cid:107) ∞ ≤ c 2 := 1 .

Original: In this way, by setting  η ≤
Revised: In this way, by setting ¯ η k = γη k , γ > 1 , η k = η ≤O (cid:0)

Original: [Equation]
Revised: [Equation]

Original: 2 , β [Equation] b(cid:15) 2 6 σ 2 and β (cid:48) 1 = 1{+
 =1+} − β 1 ,  after T
Revised: 1 ≤O (cid:0) b(cid:15) 2 cσ 2 (cid:1) , β (cid:48) {+
 =1+} − β 1 , λ k = λ , λ 0 =0 , after T

Original: = O (cid:0) 
Revised: = O (cid:0) ∆

Original: [Equation] [Equation] iterations with minibatch size b and ∆ = F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by accelerated SGD satisﬁes the following four properties.
Revised: σ 2 L b(cid:15) 4 (cid:1) iterations with minibatch size b and ∆= F ( x 0 ) − F ( x (cid:63) ) , the sequence { ( x k , z k ) } Tk =0 generated by accelerated SGD satisﬁes the following four properties

Original: a)
Revised: a)

Original: The gradient ∇ F k ( x k ) of the sequence { x k } Tk =0 can be upper bounded by
Revised: The gradient ∇ F k ( x k ) of the sequence { x k } T k =0 can be upper bounded by

Original: [Equation]
Revised: [Equation]

Original: b) The gradient moment m k can well estimate the full gradient ∇ F ( z k ) at the point z k  :
Revised: b) The gradient moment m k can well estimate the full gradient ∇ F ( x k ) and ∇ F ( z k ) :

Original: [Equation]
Revised: [Equation]

Original: c)
Revised: c)

Original: The sequence { x k , z k } satisﬁes
Revised: The sequence { x k , z k } satisﬁes

Original: [Equation]
Revised: [Equation]

Original: d) The total stochastic gradient complexity to achieve the above three properties is O (cid:0)  ∆ σ 2 L  (cid:15) 4 (cid:1) .
Revised: d) The total stochastic gradient complexity to achieve the above three properties is O (cid:0) c 2 . 5 ∞ ∆ σ 2 L ν 1 . 25 (cid:15) 4 (cid:1) .

Original: The proof is completed.
Revised: The proof is completed.

Original: G P ROOFS OF A UXILIARY L EMMAS G. P ROOF OF L EMMA
Revised: G P ROOFS OF A UXILIARY L EMMAS G. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: To begin with
Revised: To begin with

Original: [Equation]
Revised: [Equation]

Original: we assume that ∀ t ≤ k
Revised: we assume that ∀ t ≤ k

Original: [Equation]
Revised: [Equation]

Original: it holds
Revised: it holds

Original: [Equation]
Revised: [Equation]

Original: Then we consider the case where t = k + 1 as follows
Revised: Then we consider the case where t = k + 1 as follows

Original: [Equation]
Revised: [Equation]

Original: Then we derive the second results as follows: 
Revised: Then we derive the second results as follows: (cid:13)(cid:13)(cid:13)(cid:13)(cid:114) [Equation] (cid:13)(cid:13)(cid:13)(cid:13) [Equation]

Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:114)

Original: [Equation]
Revised: [Equation]

Original:  ∞ .
Revised: (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:115) [Equation] (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ∞ .

Original: Therefore, we have1 −
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: We complete the proof.
Revised: We complete the proof.

Original: G. 
Revised: G. P ROOF OF L EMMA

Original: P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: To begin with, we have
Revised: To begin with, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) holds since w k :
Revised: where (cid:172) holds since w k :

Original: = m k + λ  x k ∗ s k ; (cid:173) holds since we set all η k = η and ¯ η k = ¯ η which gives τ k = τ = [Equation]
Revised: = m k + λ k x k ∗ s k ; (cid:173) holds since we set all η k = η and ¯ η k = ¯ η which gives τ k = τ = 1 η +¯ η

Original: +¯ η + λη ¯ η .
Revised: + λ k η ¯ η .

Original: Therefore, by deﬁning ρ k +1 =
Revised: Therefore, by deﬁning ρ k +1 =

Original: ητρ k and ρ 1 = 1  , then we have
Revised: ητ k − 1 ρ k , ρ 1 = 1 and ρ 0 = 0 , then we have

Original: [Equation]
Revised: [Equation]

Original: For k = 0 , we have
Revised: For k = 0 , we have

Original: [Equation]
Revised: [Equation]

Original: In this way, one can obtain
Revised: In this way

Original: [Equation]
Revised: [Equation]

Original: s
Revised: one can obtain

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) hold since z 0 = x 0 and ρ 1 = 1 .
Revised: where (cid:172) hold since z 0 = x 0 and ρ 1 = 1 .

Original: Then we can upper bound (cid:13)(cid:13)(cid:13)(cid:13)
Revised: Then we can upper bound [Equation]

Original: y k +1 − (1 +λ ¯ η ) x k + ρ k + (cid:13)(cid:13)(cid:13)(cid:13)
Revised: +

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ρ
Revised: ρ

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: w
Revised: ρ

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)
Revised: (cid:13)(cid:13)(cid:13)(cid:13)

Revised: [Equation]

Original: w
Revised: w

Original: [Equation]
Revised: [Equation]

Original: s i (cid:13)(cid:13)(cid:13)(cid:13) where (cid:172) holds since 1) (cid:80) ki =0{+=
+} 1 − ητρ i {+− 
+}
Revised: s [Equation] (cid:13)(cid:13)(cid:13)(cid:13) where (cid:172) holds since 1) (cid:80) ki {+=
+} 1 − ητ i {+− 
+}

Original: =
Revised: ρ

Original: (cid:80) ki [Equation] 0 ( 1 ρ i 
Revised: i +

Original: [Equation]
Revised: [Equation]

Original: 1 −

Original: [Equation]
Revised: [Equation]

Original: ρ i ) =

Original:  ρ k +1 , and 2) Jensen’ inequality.
Revised: 1 ρ k +1 , and 2) Jensen’ inequality.

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]

Original: (cid:13)(cid:13)(cid:13)(cid:13)

Original: [Equation]
Revised: [Equation]

Original: Moreover, we can also bound
Revised: Moreover, we can also bound

Original: [Equation]
Revised: [Equation]

Original: + [Equation] i 
Revised: ρ i +1 (1 − ητ

Original: [Equation] ρ [Equation] (cid:13)(cid:13)(cid:13)(cid:13)
Revised: i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)

Original: w i s i (cid:13)(cid:13)(cid:13)(cid:13) .
Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13) .

Original: On the other hand, we have
Revised: On the other hand, we have

Original: [Equation]
Revised: [Equation]

Original: where (cid:172) we plug in y k +

Revised: where (cid:172) we plug in y k +

Original: =
Revised: =

Original: {+z k − ¯ η k m k s k ; in (cid:173) we plug in x k +
+}
Revised: {+z k − ¯ η k m k s k ; in (cid:173) we plug in x k +
+}

Original: {+= 1 
+}
Revised: {+= 1 
+}

Revised: + λ k η (cid:16) x k − η m k s k (cid:17) = 1 1+ λ k η (cid:16) x k − η w k − λ k x k ∗

Revised: s k s k (cid:17) =

Revised: x k − η 1+ λ k η w k s k ; and (cid:174) we have ¯ ητ k +¯ ηλ k =

Original: z k  − ¯ η k m k s k  .
Revised: ¯ ητ k (1+¯ ηλ k )(1+ ηλ k ) and (1 − ητ k ) = ¯ ητ k (1 + ηλ k ) .

Original: Then we can upper bound 
Revised: Then we can upper bound (cid:107)

Revised: z k +1 − z k (cid:107) 2 ≤ 2¯ η 2 τ 2 k

Revised: (1 + λ k η ) 2 (cid:107) (1 + λ k ¯ η ) x k +1 −

Revised: y k +1 (cid:107) 2 + 2¯ η(1 + λ k η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)

Revised: w k s k (cid:13)(cid:13)(cid:13)(cid:13) ≤ 2¯ η(1 + λ k η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)

Revised: w k s k (cid:13)(cid:13)(cid:13)(cid:13)

Original: {++ 
+}
Revised: {++ 
+}

Revised: ρ k

Revised: +1 ¯ η 2 (¯ η − η ) 2 τ 2 k (1 + λ k η )k

Original: [Equation]
Revised: (cid:88) i =  ρ i +1 (1 − ητ i − 1 )(1 + λ i η ) 2 (cid:13)(cid:13)(cid:13)(cid:13)

Revised: w i s i (cid:13)(cid:13)(cid:13)(cid:13)

Original: The proof is completed.
Revised: The proof is completed.

Original: G. P ROOF OF L EMMA
Revised: G. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: From Lemma 5, we have
Revised: From Lemma 4, we have

Original: [Equation]
Revised: [Equation]

Original: where in (cid:172) 
Revised: where in (cid:172) , we use the results in Lemma 5 that

Original: [Equation]
Revised: [Equation]

Original: we use the results in Lemma 6 that
Revised: with Π k := 2¯ η

Original: [Equation]
Revised: [Equation]

Original: Then we have
Revised: Then we have

Original: [Equation]
Revised: [Equation]

Original: where in (cid:172) , we use the results in Lemma 6 that
Revised: where in (cid:172) [Equation] we use the results in Lemma 4 that

Revised: [Equation]

Revised: and also the results in Lemma 5 that

Original: [Equation]
Revised: [Equation]

Original: The proof is completed.
Revised: The proof is completed

Revised: [Equation]