Original: D ECOUPLED W EIGHT D ECAY R EGULARIZATION
Revised: D ECOUPLED W EIGHT D ECAY R EGULARIZATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: L 2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam.
Revised: L 2 regularization and weight decay regularization are equivalent for standard stochastic gradient descent (when rescaled by the learning rate), but as we demonstrate this is not the case for adaptive gradient algorithms, such as Adam.

Original: While common implementations of these algorithms employ L 2 regularization (often calling it “weight decay” in what may be misleading due to the inequivalence we expose), we propose a simple modiﬁcation to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t.
Revised: While common implementations of these algorithms employ L 2 regularization (often calling it “weight decay” in what may be misleading due to the inequivalence we expose), we propose a simple modiﬁcation to recover the original formulation of weight decay regularization by decoupling the weight decay from the optimization steps taken w.r.t.

Original: the loss function.
Revised: the loss function.

Original: We provide empirical evidence that our proposed modiﬁcation (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam’s generalization performance, allowing it to compete with
Revised: We provide empirical evidence that our proposed modiﬁcation (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam and (ii) substantially improves Adam’s generalization performance, allowing it to compete with

Original: SGD with momentum on image classiﬁcation datasets (on which it was previously typically outperformed by the latter).
Revised: SGD with momentum on image classiﬁcation datasets (on which it was previously typically outperformed by the latter).

Original: Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments will be available after the review process.
Revised: Our proposed decoupled weight decay has already been adopted by many researchers, and the community has implemented it in TensorFlow and PyTorch; the complete source code for our experiments is available at https://github.com/loshchil/AdamW-and-SGDW

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adaptive gradient methods, such as AdaGrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2014) and most recently AMSGrad (Reddi et al., 2018) have become a default method of choice for training feed-forward and recurrent neural networks (Xu et al., 2015;
Revised: Adaptive gradient methods, such as AdaGrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2014) and most recently AMSGrad (Reddi et al., 2018) have become a default method of choice for training feed-forward and recurrent neural networks (Xu et al., 2015;

Original: Radford et al., 2015).
Revised: Radford et al., 2015).

Original: Nevertheless, state-of-the-art results for popular image classiﬁcation datasets, such as CIFAR-10 and CIFAR-100 Krizhevsky (2009), are still obtained by applying SGD with momentum (Gastaldi, 2017; Cubuk et al., 2018).
Revised: Nevertheless, state-of-the-art results for popular image classiﬁcation datasets, such as CIFAR-10 and CIFAR-100 Krizhevsky (2009), are still obtained by applying SGD with momentum (Gastaldi, 2017; Cubuk et al., 2018).

Original: Furthermore, Wilson et al.
Revised: Furthermore, Wilson et al.

Original: (2017) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks, such as image classiﬁcation, character-level language modeling and constituency parsing.
Revised: (2017) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks, such as image classiﬁcation, character-level language modeling and constituency parsing.

Original: Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., 2016;
Revised: Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., 2016;

Original: Dinh et al., 2017) and inherent problems of adaptive gradient methods (Wilson et al., 2017).
Revised: Dinh et al., 2017) and inherent problems of adaptive gradient methods (Wilson et al., 2017).

Original: In this paper, we investigate whether it is better to use L 2 regularization or weight decay regularization to train deep neural networks with SGD and Adam.
Revised: In this paper, we investigate whether it is better to use L 2 regularization or weight decay regularization to train deep neural networks with SGD and Adam.

Original: We show that a major factor of the poor generalization of the most popular adaptive gradient method, Adam, is due to the fact that L 2 regularization is not nearly as effective for it as for SGD.
Revised: We show that a major factor of the poor generalization of the most popular adaptive gradient method, Adam, is due to the fact that L 2 regularization is not nearly as effective for it as for SGD.

Original: Speciﬁcally, our analysis of Adam leads to the following observations:
Revised: Speciﬁcally, our analysis of Adam leads to the following observations:

Original: L 2 regularization and weight decay are not identical.
Revised: L 2 regularization and weight decay are not identical.

Original: Contrary to a belief which seems popular among some practitioners, the two techniques are not equivalent.
Revised: Contrary to a belief which seems popular among some practitioners, the two techniques are not equivalent.

Original: For SGD, they can be made equivalent by a reparameterization of the weight decay factor based on the learning rate; this is not the case for Adam.
Revised: For SGD, they can be made equivalent by a reparameterization of the weight decay factor based on the learning rate; this is not the case for Adam.

Original: In particular, when combined with adaptive gradients, L 2 regularization leads to weights with large parameter and/or gradient amplitudes being regularized less than they would be when using weight decay.
Revised: In particular, when combined with adaptive gradients, L 2 regularization leads to weights with large parameter and/or gradient amplitudes being regularized less than they would be when using weight decay.

Original: L 2 regularization is not effective in Adam.
Revised: L 2 regularization is not effective in Adam.

Original: One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that common deep learning libraries 
Revised: One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that common deep learning libraries only implement L 2 regularization, not the original weight decay.

Original: only implement L  regularization 
Revised: Therefore, on tasks/datasets where the use of L 2 regularization is beneﬁcial for SGD (e

Original: , not the original weight decay.
Revised: g.,

Original: Therefore, on tasks/datasets where the use of L 2 regularization is beneﬁcial for SGD (e.g., on many popular image classiﬁcation datasets), Adam leads to worse results than SGD with momentum (for which L 2 regularization behaves as expected).
Revised: on many popular image classiﬁcation datasets), Adam leads to worse results than SGD with momentum (for which L 2 regularization behaves as expected).

Original: Weight decay is equally effective in both SGD and Adam.
Revised: Weight decay is equally effective in both SGD and Adam.

Original: For SGD, it is equivalent to Lregularization, while for Adam it is not.
Revised: For SGD, it is equivalent to Lregularization, while for Adam it is not.

Original: Optimal weight decay depends on the total number of batch passes/weight updates.
Revised: Optimal weight decay depends on the total number of batch passes/weight updates.

Original: Our empirical analysis of SGD and Adam suggests that the larger the runtime/number of batch passes to be performed, the smaller the optimal weight decay.
Revised: Our empirical analysis of SGD and Adam suggests that the larger the runtime/number of batch passes to be performed, the smaller the optimal weight decay.

Original: This effect tends to be neglected because hyperparameters are often tuned for a ﬁxed number of training epochs.
Revised: This effect tends to be neglected because hyperparameters are often tuned for a ﬁxed number of training epochs.

Original: As a result, the values of the weight decay found to perform best for short runs do not generalize to much longer runs.
Revised: As a result, the values of the weight decay found to perform best for short runs do not generalize to much longer runs.

Original: The main contribution of this paper is to improve regularization in Adam by decoupling the weight decay from the gradient-based update .
Revised: The main contribution of this paper is to improve regularization in Adam by decoupling the weight decay from the gradient-based update .

Original: In a comprehensive analysis, we show that Adam generalizes substantially better with decoupled weight decay than with L 2 regularization, achieving 15% relative improvement in test error (see Figures 2 and 3); this holds true for various image recognition datasets (CIFAR-10 and ImageNet32x32), training budgets (ranging from 100 to 1800 epochs), and learning rate schedules (ﬁxed, drop-step, and cosine annealing; see Figure 1).
Revised: In a comprehensive analysis, we show that Adam generalizes substantially better with decoupled weight decay than with L 2 regularization, achieving 15% relative improvement in test error (see Figures 2 and 3); this holds true for various image recognition datasets (CIFAR-10 and ImageNet32x32), training budgets (ranging from 100 to 1800 epochs), and learning rate schedules (ﬁxed, drop-step, and cosine annealing; see Figure 1).

Original: We demonstrate that our decoupled weight decay renders the optimal settings of the learning rate and the weight decay factor much more independent, thereby easing hyperparameter optimization (see Figure 2).
Revised: We demonstrate that our decoupled weight decay renders the optimal settings of the learning rate and the weight decay factor much more independent, thereby easing hyperparameter optimization (see Figure 2).

Original: The main motivation of this paper is to improve Adam to make it competitive w.r.t.
Revised: The main motivation of this paper is to improve Adam to make it competitive w.r.t.

Original: SGD with momentum even for those problems where it did not use to be competitive.
Revised: SGD with momentum even for those problems where it did not use to be competitive.

Original: We hope that as a result, practitioners do not need to switch between Adam and SGD anymore, which in turn should reduce the common issue of selecting dataset/task-speciﬁc training algorithms and their hyperparameters.
Revised: We hope that as a result, practitioners do not need to switch between Adam and SGD anymore, which in turn should reduce the common issue of selecting dataset/task-speciﬁc training algorithms and their hyperparameters.

Original:  D ECOUPLING THE W EIGHT D ECAY FROM THE G RADIENT - BASED U PDATE
Revised:  D ECOUPLING THE W EIGHT D ECAY FROM THE G RADIENT - BASED U PDATE

Original: In the weight decay described by Hanson & Pratt (1988), the weights θ decay exponentially as
Revised: In the weight decay described by Hanson & Pratt (1988), the weights θ decay exponentially as

Original: [Equation]
Revised: [Equation]

Original:  where λ deﬁnes the rate of the weight decay per step and ∇ f t ( θ t ) is the t -th batch gradient to be multiplied by a learning rate α .
Revised: ) where λ deﬁnes the rate of the weight decay per step and ∇ f t ( θ t ) is the t -th batch gradient to be multiplied by a learning rate α .

Original: For standard SGD, it is equivalent to standard L 2 regularization:
Revised: For standard SGD, it is equivalent to standard L 2 regularization:

Original: Proposition 1 (Weight decay =
Revised: Proposition 1 (Weight decay =

Original: L 2 reg for standard SGD) .
Revised: L 2 reg for standard SGD) .

Original: Standard SGD with base learning rate α executes the same steps on batch loss functions f t ( θ ) with weight decay λ (deﬁned in Equation 1) as it executes without weight decay on f reg t ( θ ) =
Revised: Standard SGD with base learning rate α executes the same steps on batch loss functions f t ( θ ) with weight decay λ (deﬁned in Equation 1) as it executes without weight decay on f reg t ( θ ) =

Original: f t ( θ ) + λ (cid:48) 2 (cid:107) θ (cid:107) 22 , with λ (cid:48) =
Revised: f t ( θ ) + λ (cid:48) 2 (cid:107) θ (cid:107) 22 , with λ (cid:48) =

Original: λα .
Revised: λα .

Original: The proofs of this well-known fact, as well as our other propositions, are given in the Appendix A.
Revised: The proofs of this well-known fact, as well as our other propositions, are given in the Appendix A.

Original: Due to this equivalence, L 2 regularization is very frequently referred to as weight decay, including in popular deep learning libraries.
Revised: Due to this equivalence, L 2 regularization is very frequently referred to as weight decay, including in popular deep learning libraries.

Original: However, as we will demonstrate later in this section, this equivalence does not hold for adaptive gradient methods.
Revised: However, as we will demonstrate later in this section, this equivalence does not hold for adaptive gradient methods.

Original: One fact that is often overlooked already for the simple case of SGD is that in order for the equivalence to hold, the L 2 regularizer λ (cid:48) has to be set to λα , i.e., if there is an overall best weight decay value λ , the best value of λ (cid:48) is tightly coupled with the learning rate α .
Revised: One fact that is often overlooked already for the simple case of SGD is that in order for the equivalence to hold, the L 2 regularizer λ (cid:48) has to be set to λα , i.e., if there is an overall best weight decay value λ , the best value of λ (cid:48) is tightly coupled with the learning rate α .

Original: In order to decouple the effects of these two hyperparameters, we advocate to decouple the weight decay step as proposed by Hanson & Pratt (1988) (Equation 1).
Revised: In order to decouple the effects of these two hyperparameters, we advocate to decouple the weight decay step as proposed by Hanson & Pratt (1988) (Equation 1).

Original: Looking ﬁrst at the case of SGD, we propose to decay the weights simultaneously with the update of θ t based on gradient information in Line 9 of Algorithm 1.
Revised: Looking ﬁrst at the case of SGD, we propose to decay the weights simultaneously with the update of θ t based on gradient information in Line 9 of Algorithm 1.

Original: This yields our proposed variant of
Revised: This yields our proposed variant of

Original: SGD with momentum using decoupled weight decay ( SGDW ).
Revised: SGD with momentum using decoupled weight decay ( SGDW ).

Original: This simple modiﬁcation explicitly decouples λ and α (although some problem-dependent implicit coupling may of course remain as for any two hyperparameters).
Revised: This simple modiﬁcation explicitly decouples λ and α (although some problem-dependent implicit coupling may of course remain as for any two hyperparameters).

Original: In order to account for a possible scheduling of both α and λ , we introduce a scaling factor η t delivered by a user-deﬁned procedure SetScheduleMultiplier ( t ) .
Revised: In order to account for a possible scheduling of both α and λ , we introduce a scaling factor η t delivered by a user-deﬁned procedure SetScheduleMultiplier ( t ) .

Original: Now, let’s turn to adaptive gradient algorithms like the popular optimizer Adam Kingma & Ba (2014), which scale gradients by their historic magnitudes.
Revised: Now, let’s turn to adaptive gradient algorithms like the popular optimizer Adam Kingma & Ba (2014), which scale gradients by their historic magnitudes.

Original: Intuitively, when Adam is run on a loss function f plus L 2 regularization, weights that tend to have large gradients in f do not get regularized as much as they would with decoupled weight decay, since the gradient of the regularizer gets scaled along with the gradient of f .
Revised: Intuitively, when Adam is run on a loss function f plus L 2 regularization, weights that tend to have large gradients in f do not get regularized as much as they would with decoupled weight decay, since the gradient of the regularizer gets scaled along with the gradient of f .

Original: This leads to an inequivalence of L 2 and decoupled weight decay regularization for adaptive gradient algorithms:
Revised: This leads to an inequivalence of L 2 and decoupled weight decay regularization for adaptive gradient algorithms:

Original: Proposition 2 (Weight decay (cid:54) =
Revised: [Table] [Table] [Table] [Table]

Original: L 2 reg for adaptive gradients) .
Revised: [Table]

Original: Let O denote an optimizer that has iterates θ t +1 ← θ t − α M t ∇ f t ( θ t ) when run on batch loss function f t ( θ ) without weight decay,
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: β 2 ) g 2 t 9: 
Revised: [Table] 2 ) g 2 t 9: ˆ m [Table] ← m t /

Revised: [Table]

Original: ˆ m t ← m t / (1 − β t 1 ) (cid:46) β 1 is taken to the power of t 10: ˆ v t ← v t / (1 − β t 2 ) (cid:46) β 2 is taken to the power of t 11: η t ← SetScheduleMultiplier ( t ) (cid:46) can be ﬁxed, decay, or also be used for warm restarts 12: θ t ← θ t − 1 − η t (cid:16) α ˆ m t / ( √ ˆ v t + (cid:15) )
Revised: β t 1 ) [Table] ˆ [Table]

Revised: ←

Revised: [Table]

Revised: [Table]

Revised: ←

Revised: [Table]

Revised: [Table]

Original: + λ θ t − 1 (cid:17) 13: until stopping criterion is met 14: return optimized parameters θ t and θ t +1 ← (1 − λ )
Revised: + [Table] : until stopping criterion is met 14: return optimized parameters θ t Proposition 2 (Weight decay (cid:54) =

Original:  θ t − α M t ∇ f t ( θ t ) when run on f t ( θ ) with weight decay, respectively, with
Revised: L 2 reg for adaptive gradients) . Let O denote an optimizer that has iterates θ t +1 ← θ t − α M t ∇ f t ( θ t ) when run on batch loss function f t ( θ ) without weight decay, and θ t +1 ← (1 − λ ) θ t − α M t ∇ f t ( θ t ) when run on f t ( θ ) with weight decay, respectively, with

Original: M t (cid:54) =
Revised: M t (cid:54) =

Original: k I (where k ∈ R ).
Revised: k I (where k ∈ R ).

Original: Then, for O there exists no L 2 coefﬁcient λ (cid:48)
Revised: Then, for O there exists no L 2 coefﬁcient λ (cid:48)

Original: such that running O on batch loss f reg t ( θ ) =
Revised: such that running O on batch loss f reg t ( θ ) =

Original: f t ( θ )+ λ (cid:48) 2 (cid:107) θ (cid:107) 22 without weight decay is equivalent to running O on f t ( θ ) with decay λ ∈ R + .
Revised: f t ( θ )+ λ (cid:48) 2 (cid:107) θ (cid:107) 22 without weight decay is equivalent to running O on f t ( θ ) with decay λ ∈ R + .

Original: We decouple weight decay and loss-based gradient updates in Adam as shown in line 12 of Algorithm 2; this gives rise to our variant of Adam with decoupled weight decay ( AdamW ).
Revised: We decouple weight decay and loss-based gradient updates in Adam as shown in line 12 of Algorithm 2; this gives rise to our variant of Adam with decoupled weight decay ( AdamW ).

Original: Having shown that L 2 regularization and weight decay regularization differ for adaptive gradient algorithms raises the question of how they differ and how to interpret their effects.
Revised: Having shown that L 2 regularization and weight decay regularization differ for adaptive gradient algorithms raises the question of how they differ and how to interpret their effects.

Original: Their equivalence for standard SGD remains very helpful for intuition: both mechanisms push weights closer to zero, at the same rate.
Revised: Their equivalence for standard SGD remains very helpful for intuition: both mechanisms push weights closer to zero, at the same rate.

Original: However, for adaptive gradient algorithms they differ: with L 2 regularization, the sums of the gradient of the loss function and the gradient of the regularizer (i.e., the L 2 norm of the weights) are adapted, whereas with weight decay, only the gradients of the loss function are adapted (with the weight decay step separated from the adaptive gradient mechanism).
Revised: However, for adaptive gradient algorithms they differ: with L 2 regularization, the sums of the gradient of the loss function and the gradient of the regularizer (i.e., the L 2 norm of the weights) are adapted, whereas with weight decay, only the gradients of the loss function are adapted (with the weight decay step separated from the adaptive gradient mechanism).

Original: With L 2 regularization both types of gradients are normalized by their typical (summed) magnitudes, and therefore weights x with large typical gradient magnitude s are regularized by a smaller relative amount than other weights.
Revised: With L 2 regularization both types of gradients are normalized by their typical (summed) magnitudes, and therefore weights x with large typical gradient magnitude s are regularized by a smaller relative amount than other weights.

Original: In contrast, weight decay regularizes all weights with the same rate λ , effectively regularizing weights x with large s more than standard L 2 regularization does.
Revised: In contrast, weight decay regularizes all weights with the same rate λ , effectively regularizing weights x with large s more than standard L 2 regularization does.

Original: We demonstrate this formally for a simple special case of adaptive gradient algorithm with a ﬁxed preconditioner:
Revised: We demonstrate this formally for a simple special case of adaptive gradient algorithm with a ﬁxed preconditioner:

Original: Proposition 3 (Weight decay = scale-adjusted L 2 reg for adaptive gradient algorithm with ﬁxed preconditioner) .
Revised: Proposition 3 (Weight decay = scale-adjusted L 2 reg for adaptive gradient algorithm with ﬁxed preconditioner) .

Original: Let O denote an algorithm with the same characteristics as in Proposition 2, and using a ﬁxed preconditioner matrix M t = diag ( s ) − 1 (with s i > 0 for all i ).
Revised: Let O denote an algorithm with the same characteristics as in Proposition 2, and using a ﬁxed preconditioner matrix M t = diag ( s ) − 1 (with s i > 0 for all i ).

Original: Then, O with base learning rate α executes the same steps on batch loss functions f t ( θ ) with weight decay λ as it executes without weight decay on the scale-adjusted regularized batch loss
Revised: Then, O with base learning rate α executes the same steps on batch loss functions f t ( θ ) with weight decay λ as it executes without weight decay on the scale-adjusted regularized batch loss

Original: [Equation]
Revised: [Equation]

Original: where (cid:12) and √· denote element-wise multiplication and square root, respectively, and λ (cid:48) =
Revised: where (cid:12) and √· denote element-wise multiplication and square root, respectively, and λ (cid:48) =

Original: λα .
Revised: λα .

Original:  J USTIFICATION OF D ECOUPLED W EIGHT D ECAY VIA A V IEW OF A DAPTIVE G RADIENT M ETHODS AS B AYESIAN F ILTERING
Revised:  J USTIFICATION OF D ECOUPLED W EIGHT D ECAY VIA A V IEW OF A DAPTIVE G RADIENT M ETHODS AS B AYESIAN F ILTERING

Original: We now discuss a justiﬁcation of decoupled weight decay in the framework of Bayesian ﬁltering for a uniﬁed theory of adaptive gradient algorithms due to Aitchison (2018).
Revised: We now discuss a justiﬁcation of decoupled weight decay in the framework of Bayesian ﬁltering for a uniﬁed theory of adaptive gradient algorithms due to Aitchison (2018).

Original: After we posted a preliminary version of our current paper on arXiv, Aitchison noted that his theory “gives us a theoretical framework in which we can understand the superiority of this weight decay over L 2 regularization, because it is weight decay, rather than L 2 regularization that emerges through the straightforward application of Bayesian ﬁltering.
Revised: After we posted a preliminary version of our current paper on arXiv, Aitchison noted that his theory “gives us a theoretical framework in which we can understand the superiority of this weight decay over L 2 regularization, because it is weight decay, rather than L 2 regularization that emerges through the straightforward application of Bayesian ﬁltering.

Original: ”(Aitchison, 2018).
Revised: ”(Aitchison, 2018).

Original: While full credit for this theory goes to Aitchison, we summarize it here to shed some light on why weight decay may be favored over L 2 regularization.
Revised: While full credit for this theory goes to Aitchison, we summarize it here to shed some light on why weight decay may be favored over L 2 regularization.

Original: Aitchison (2018) views stochastic optimization of n parameters x 1 , . .
Revised: Aitchison (2018) views stochastic optimization of n parameters x 1 , . .

Original: , x n as a Bayesian ﬁltering problem with the goal of inferring a distribution over the optimal values of each of the parameters x i given the current values of the other parameters θ − i ( t ) at time step t .
Revised: , x n as a Bayesian ﬁltering problem with the goal of inferring a distribution over the optimal values of each of the parameters x i given the current values of the other parameters θ − i ( t ) at time step t .

Original: When the other parameters do not change this is an optimization problem, but when they do change it becomes one of “tracking” the optimizer using Bayesian ﬁltering as follows.
Revised: When the other parameters do not change this is an optimization problem, but when they do change it becomes one of “tracking” the optimizer using Bayesian ﬁltering as follows.

Original: One is given a probability distribution P ( θ t | y 1: t ) of the optimizer at time step t that takes into account the data y 1: t from the ﬁrst t mini batches, a state transition prior P ( θ t +1 | θ t ) reﬂecting a (small) data-independent change in this distribution from one step to the next, and a likelihood P ( y t +1 | θ t +1 ) derived from the mini batch at step t + 1 .
Revised: One is given a probability distribution P ( θ t | y 1: t ) of the optimizer at time step t that takes into account the data y 1: t from the ﬁrst t mini batches, a state transition prior P ( θ t +1 | θ t ) reﬂecting a (small) data-independent change in this distribution from one step to the next, and a likelihood P ( y t +1 | θ t +1 ) derived from the mini batch at step t + 1 .

Original: The posterior distribution P ( θ t +1 | y 1: t +1 ) of the optimizer at time step t +can then be computed (as usual in Bayesian ﬁltering) by marginalizing over θ t to obtain the onestep ahead predictions P ( θ t +1 | y 1: t ) and then applying Bayes’ rule to incorporate the likelihood
Revised: The posterior distribution P ( θ t +1 | y 1: t +1 ) of the optimizer at time step t +can then be computed (as usual in Bayesian ﬁltering) by marginalizing over θ t to obtain the onestep ahead predictions P ( θ t +1 | y 1: t ) and then applying Bayes’ rule to incorporate the likelihood

Original: P ( y t +1 | θ t +1 ) .
Revised: P ( y t +1 | θ t +1 ) .

Original: Aitchison (2018) assumes a Gaussian state transition distribution P ( θ t +1 | θ t ) and an approximate conjugate likelihood P ( y t +1 | θ t +1 ) , leading to the following closed-form update of the ﬁltering distribution’s mean:
Revised: Aitchison (2018) assumes a Gaussian state transition distribution P ( θ t +1 | θ t ) and an approximate conjugate likelihood P ( y t +1 | θ t +1 ) , leading to the following closed-form update of the ﬁltering distribution’s mean:

Original: [Equation]
Revised: [Equation]

Original: where g is the gradient of the log likelihood of the mini batch at time t .
Revised: where g is the gradient of the log likelihood of the mini batch at time t .

Original: This result implies a preconditioner of the gradients that is given by the posterior uncertainty Σ post of the ﬁltering distribution: updates are larger for parameters we are more uncertain about and smaller for parameters we are more certain about.
Revised: This result implies a preconditioner of the gradients that is given by the posterior uncertainty Σ post of the ﬁltering distribution: updates are larger for parameters we are more uncertain about and smaller for parameters we are more certain about.

Original: Aitchison (2018) goes on to show that popular adaptive gradient methods, such as Adam and RMSprop, as well as Kronecker-factorized methods are special cases of this framework.
Revised: Aitchison (2018) goes on to show that popular adaptive gradient methods, such as Adam and RMSprop, as well as Kronecker-factorized methods are special cases of this framework.

Original: Decoupled weight decay very naturally ﬁts into this uniﬁed framework can express weight decay as part of the state-transition distribution: Aitchison (2018) assumes a slow change of the optimizer according to the following Gaussian:
Revised: Decoupled weight decay very naturally ﬁts into this uniﬁed framework can express weight decay as part of the state-transition distribution: Aitchison (2018) assumes a slow change of the optimizer according to the following Gaussian:

Original: [Equation]
Revised: [Equation]

Original: where Q is the covariance of Gaussian perturbations of the weights, and A is a regularizer to avoid values growing unboundedly over time.
Revised: where Q is the covariance of Gaussian perturbations of the weights, and A is a regularizer to avoid values growing unboundedly over time.

Original: When instantiated as A = λ × I , this regularizer A plays exactly the role of decoupled weight decay as described in Equation 1, since this leads to multiplying the current mean estimate θ t by (1 − λ ) at each step.
Revised: When instantiated as A = λ × I , this regularizer A plays exactly the role of decoupled weight decay as described in Equation 1, since this leads to multiplying the current mean estimate θ t by (1 − λ ) at each step.

Original: Notably, this regularization is also directly applied to the prior and does not depend on the uncertainty in each of the parameters (which would be required for L 2 regularization).
Revised: Notably, this regularization is also directly applied to the prior and does not depend on the uncertainty in each of the parameters (which would be required for L 2 regularization).

Original:  E XPERIMENTAL V ALIDATION
Revised:  E XPERIMENTAL V ALIDATION

Original: We now evaluate the performance of decoupled weight decay under various training budgets and learning rate schedules.
Revised: We now evaluate the performance of decoupled weight decay under various training budgets and learning rate schedules.

Original: Our experimental setup follows that of Gastaldi (2017), who proposed, in addition to L 2 regularization, to apply the new Shake-Shake regularization to a 3-branch residual DNN that allowed to achieve new state-of-the-art results of 2.86% on the CIFARdataset (Krizhevsky, 2009).
Revised: Our experimental setup follows that of Gastaldi (2017), who proposed, in addition to L 2 regularization, to apply the new Shake-Shake regularization to a 3-branch residual DNN that allowed to achieve new state-of-the-art results of 2.86% on the CIFARdataset (Krizhevsky, 2009).

Original: We always used a batch size of 128.
Revised: We always used a batch size of 128.

Original: The regular data augmentation procedure used for the CIFAR datasets was applied.
Revised: The regular data augmentation procedure used for the CIFAR datasets was applied.

Original: We used the same model/source code based on fb.resnet.torch 1 .
Revised: We used the same model/source code based on fb.resnet.torch 1 .

Original: The base networks are a 26 2x64d ResNet (i.e. the network has a depth of 26,residual branches and the ﬁrst residual block has a width of 64) and a 26 2x96d ResNet with 11.6M and 25.6M parameters, respectively.
Revised: The base networks are a 26 2x64d ResNet (i.e. the network has a depth of 26,residual branches and the ﬁrst residual block has a width of 64) and a 26 2x96d ResNet with 11.6M and 25.6M parameters, respectively.

Original: For a detailed description of the network and the Shake-Shake method, we refer the interested reader to Gastaldi (2017).
Revised: For a detailed description of the network and the Shake-Shake method, we refer the interested reader to Gastaldi (2017).

Original: We also perform experiments on the ImageNet32x32 dataset (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset with 1.2 million 32 × 32 pixels images.
Revised: We also perform experiments on the ImageNet32x32 dataset (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset with 1.2 million 32 × 32 pixels images.

Original: 4. E VALUATING D ECOUPLED W EIGHT D ECAY W ITH D IFFERENT L EARNING R ATE S CHEDULES
Revised: 4. E VALUATING D ECOUPLED W EIGHT D ECAY W ITH D IFFERENT L EARNING R ATE S CHEDULES

Original: In our ﬁrst experiment, we compare Adam with L 2 regularization to Adam with decoupled weight decay (AdamW), using three different learning rate schedules: a ﬁxed learning rate, a drop-step schedule, and a cosine annealing schedule (Loshchilov & Hutter, 2016).
Revised: In our ﬁrst experiment, we compare Adam with L 2 regularization to Adam with decoupled weight decay (AdamW), using three different learning rate schedules: a ﬁxed learning rate, a drop-step schedule, and a cosine annealing schedule (Loshchilov & Hutter, 2016).

Original: For each learning rate schedule and weight decay variant, we trained a 2x64d ResNet for 100 epochs, using different settings of the initial learning rate α and the weight decay factor λ .
Revised: For each learning rate schedule and weight decay variant, we trained a 2x64d ResNet for 100 epochs, using different settings of the initial learning rate α and the weight decay factor λ .

Original: Figure 1 shows that decoupled weight decay outperforms L 2 regularization for all learning rate schedules, with larger differences for better learning rate schedules.
Revised: Figure 1 shows that decoupled weight decay outperforms L 2 regularization for all learning rate schedules, with larger differences for better learning rate schedules.

Original: We also note that decoupled weight decay leads to a more separable hyperparameter search space, especially when a learning rate schedule, such as step-drop and cosine annealing is applied.
Revised: We also note that decoupled weight decay leads to a more separable hyperparameter search space, especially when a learning rate schedule, such as step-drop and cosine annealing is applied.

Original: The ﬁgure also shows that cosine annealing clearly outperforms the other learning rate schedules; we thus used cosine annealing for the remainder of the experiments.
Revised: The ﬁgure also shows that cosine annealing clearly outperforms the other learning rate schedules; we thus used cosine annealing for the remainder of the experiments.

Original: 4. D ECOUPLING THE W EIGHT D ECAY AND I NITIAL L EARNING R ATE P ARAMETERS
Revised: 4. D ECOUPLING THE W EIGHT D ECAY AND I NITIAL L EARNING R ATE P ARAMETERS

Original: In order to verify our hypothesis about the coupling of α and λ , in Figure 2 we compare the performance of L 2 regularization vs. decoupled weight decay in SGD (SGD vs. SGDW, top row) and in
Revised: In order to verify our hypothesis about the coupling of α and λ , in Figure 2 we compare the performance of L 2 regularization vs. decoupled weight decay in SGD (SGD vs. SGDW, top row) and in

Original: Adam (Adam vs. AdamW, bottom row).
Revised: Adam (Adam vs. AdamW, bottom row).

Original: In SGD (Figure 2, top left), L 2 regularization is not decoupled from the learning rate (the common way as described in Algorithm 1) and the ﬁgure clearly shows that the basin of best hyperparameter settings (depicted by color and top-10 hyperparameter settings by black circles) is not aligned with the x-axis or y-axis but lies on the diagonal.
Revised: In SGD (Figure 2, top left), L 2 regularization is not decoupled from the learning rate (the common way as described in Algorithm 1), and the ﬁgure clearly shows that the basin of best hyperparameter settings (depicted by color and top-10 hyperparameter settings by black circles) is not aligned with the x-axis or y-axis but lies on the diagonal.

Original: This suggests that the two hyperparameters are interdependent and need to be changed simultaneously, while only changing one of them might substantially worsen results.
Revised: This suggests that the two hyperparameters are interdependent and need to be changed simultaneously, while only changing one of them might substantially worsen results.

Original: Consider, e.g., the setting at the top left black circle ( α = 1 / 2 , λ
Revised: Consider, e.g., the setting at the top left black circle ( α = 1 / 2 , λ

Original: = 1 / 8 ∗ 0 .
Revised: = 1 / 8 ∗ 0 .

Original: 001 ); only changing either α or λ by itself would worsen results, while changing both of them could still yield clear improvements.
Revised: 001 ); only changing either α or λ by itself would worsen results, while changing both of them could still yield clear improvements.

Original: We note that this coupling of initial learning rate and L 2 regularization factor might have contributed to SGD’s reputation of being very sensitive to its hyperparameter settings.
Revised: We note that this coupling of initial learning rate and L 2 regularization factor might have contributed to SGD’s reputation of being very sensitive to its hyperparameter settings.

Original: In contrast, the results for SGD with decoupled weight decay (SGDW) in Figure 2 (top right) show that weight decay and initial learning rate are decoupled.
Revised: In contrast, the results for SGD with decoupled weight decay (SGDW) in Figure 2 (top right) show that weight decay and initial learning rate are decoupled.

Original: The proposed approach renders the two hyperparameters more separable: even if the learning rate is not well tuned yet (e.g., consider the value of 1/1024 in Figure 2, top right), leaving it ﬁxed and only optimizing the weight decay factor would yield a good value (of 1/4*0.001).
Revised: The proposed approach renders the two hyperparameters more separable: even if the learning rate is not well tuned yet (e.g., consider the value of 1/1024 in Figure 2, top right), leaving it ﬁxed and only optimizing the weight decay factor would yield a good value (of 1/4*0.001).

Original: This is not the case for SGD with L 2 regularization (see
Revised: This is not the case for SGD with L 2 regularization (see

Original: Figure 2, top left).
Revised: Figure 2, top left).

Original: The results for Adam with L 2 regularization are given in Figure 2 (bottom left).
Revised: The results for Adam with L 2 regularization are given in Figure 2 (bottom left).

Original: Adam’s best hyperparameter settings performed clearly worse than SGD’s best ones (compare Figure 2, top left).
Revised: Adam’s best hyperparameter settings performed clearly worse than SGD’s best ones (compare Figure 2, top left).

Original: While both methods used L 2 regularization, Adam did not beneﬁt from it at all: its best results obtained for non-zero L 2 regularization factors were comparable to the best ones obtained without the
Revised: While both methods used L 2 regularization, Adam did not beneﬁt from it at all: its best results obtained for non-zero L 2 regularization factors were comparable to the best ones obtained without the

Original: L 2 regularization, i.e., when λ = 0 .
Revised: L 2 regularization, i.e., when λ = 0 .

Original: Similarly to the original SGD, the shape of the hyperparameter landscape suggests that the two hyperparameters are coupled.
Revised: Similarly to the original SGD, the shape of the hyperparameter landscape suggests that the two hyperparameters are coupled.

Original: In contrast, the results for our new variant of Adam with decoupled weight decay (AdamW) in
Revised: In contrast, the results for our new variant of Adam with decoupled weight decay (AdamW) in

Original: Figure 2 (bottom right) show that AdamW largely decouples weight decay and learning rate.
Revised: Figure 2 (bottom right) show that AdamW largely decouples weight decay and learning rate.

Original: The results for the best hyperparameter settings were substantially better than the best ones of Adam with L 2 regularization and rivaled those of SGD and SGDW.
Revised: The results for the best hyperparameter settings were substantially better than the best ones of Adam with L 2 regularization and rivaled those of SGD and SGDW.

Original: In summary, the results in Figure 2 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simpliﬁes the problem of hyperparameter tuning in SGD and improves Adam’s performance to be competitive w.r.t.
Revised: In summary, the results in Figure 2 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simpliﬁes the problem of hyperparameter tuning in SGD and improves Adam’s performance to be competitive w.r.t.

Original: SGD with momentum.
Revised: SGD with momentum.

Original: 4. B ETTER G ENERALIZATION OF A DAM W
Revised: 4. B ETTER G ENERALIZATION OF A DAM W

Original: While the previous experiment suggested that the basin of optimal hyperparameters of AdamW is broader and deeper than the one of Adam, we next investigated the results for much longer runs of 1800 epochs to compare the generalization capabilities of AdamW and Adam.
Revised: While the previous experiment suggested that the basin of optimal hyperparameters of AdamW is broader and deeper than the one of Adam, we next investigated the results for much longer runs of 1800 epochs to compare the generalization capabilities of AdamW and Adam.

Original: We ﬁxed the initial learning rate to 0.001 which represents both the default learning rate for Adam and the one which showed reasonably good results in our experiments.
Revised: We ﬁxed the initial learning rate to 0.001 which represents both the default learning rate for Adam and the one which showed reasonably good results in our experiments.

Original: Figure 3 shows the results for 12 settings of the L 2 regularization of Adam and 7 settings of the normalized weight decay of
Revised: Figure 3 shows the results for 12 settings of the L 2 regularization of Adam and 7 settings of the normalized weight decay of

Original: AdamW (the normalized weight decay represents a rescaling formally deﬁned in the Appendix B.1, it amounts to a multiplicative factor which depends on the number of bath passes).
Revised: AdamW (the normalized weight decay represents a rescaling formally deﬁned in the Appendix B.1, it amounts to a multiplicative factor which depends on the number of bath passes).

Original: Interestingly, while the dynamics of the learning curves of Adam and AdamW often coincided for the ﬁrst half of the training run, AdamW often led to lower training loss and test errors (see Figure 3 top left and top right, respectively).
Revised: Interestingly, while the dynamics of the learning curves of Adam and AdamW often coincided for the ﬁrst half of the training run, AdamW often led to lower training loss and test errors (see Figure 3 top left and top right, respectively).

Original: Importantly, the use of weight decay in Adam did not yield as good results as in AdamW (see also Figure 3, bottom left).
Revised: Importantly, the use of weight decay in Adam did not yield as good results as in AdamW (see also Figure 3, bottom left).

Original: Next, we investigated whether AdamW’s better results were only due to better convergence or due to better generalization.
Revised: Next, we investigated whether AdamW’s better results were only due to better convergence or due to better generalization.

Original: The results in
Revised: The results in

Original: Figure 3 (bottom right) for the best settings of Adam and AdamW suggest that AdamW did not only yield better training loss but also yielded better generalization performance for similar training loss values .
Revised: Figure 3 (bottom right) for the best settings of Adam and AdamW suggest that AdamW did not only yield better training loss but also yielded better generalization performance for similar training loss values .

Original: The results on ImageNet32x32 (see SuppFigure 4 in the Appendix) lead to the same conclusion of substantially improved generalization performance.
Revised: The results on ImageNet32x32 (see SuppFigure 4 in the Appendix) lead to the same conclusion of substantially improved generalization performance.

Original: 4. A DAM WR WITH W ARM R ESTARTS FOR B ETTER A NYTIME P ERFORMANCE
Revised: 4. A DAM WR WITH W ARM R ESTARTS FOR B ETTER A NYTIME P ERFORMANCE

Original: In order to improve anytime performance of SGDW and AdamW we extended them with warm restarts of (Loshchilov & Hutter, 2016) to obtain SGDWR and AdamWR, respectively (see section
Revised: In order to improve anytime performance of SGDW and AdamW we extended them with warm restarts of (Loshchilov & Hutter, 2016) to obtain SGDWR and AdamWR, respectively (see section

Original: B.2 in the Appendix).
Revised: B.2 in the Appendix).

Original: As Figure 4 shows, AdamWR greatly sped up AdamW on CIFAR-10 and
Revised: As Figure 4 shows, AdamWR greatly sped up AdamW on CIFAR-10 and

Original: ImageNet32x32, up to a factor of 10 (see the results at the ﬁrst restart).
Revised: ImageNet32x32, up to a factor of 10 (see the results at the ﬁrst restart).

Original: For the default learning rate of 0.001, AdamW achieved 15% relative improvement in test errors compared to Adam both on
Revised: For the default learning rate of 0.001, AdamW achieved 15% relative improvement in test errors compared to Adam both on

Original: CIFAR-10 (also see Figure 3) and ImageNet32x32 (also see SuppFigure 5).
Revised: CIFAR-10 (also see Figure 3) and ImageNet32x32 (also see SuppFigure 5).

Original: AdamWR achieved the

Original: For a better resolution and with training loss curves, see SuppFigure 5 and SuppFigure 6 in the supplementary material.

Original:  same improved results but with a much better anytime performance.
Revised: AdamWR achieved the same improved results but with a much better anytime performance.

Original: These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on
Revised: These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on

Original: ImageNet32x32.
Revised: ImageNet32x32.

Original: 4. U SE OF A DAM W ON OTHER DATASETS AND ARCHITECTURES
Revised: 4. U SE OF A DAM W ON OTHER DATASETS AND ARCHITECTURES

Original: Several other research groups have already successfully applied AdamW in citable works.
Revised: Several other research groups have already successfully applied AdamW in citable works.

Original: For example, AnonymizedA (2018) used AdamW to train a novel architecture for face detection on the standard WIDER FACE dataset (Yang et al., 2016), obtaining almost 10x faster predictions than the previous state of the art algorithms while achieving comparable performance. 
Revised: For example, Wang et al. (2018) used AdamW to train a novel architecture for face detection on the standard WIDER FACE dataset (Yang et al., 2016), obtaining almost 10x faster predictions than the previous state of the art algorithms while achieving comparable performance. V ¨ olker et al.

Original: AnonymizedB (2018) employed AdamW with cosine annealing to train convolutional neural networks to classify and characterize error-related brain signals measured from intracranial electroencephalography (EEG) recordings.
Revised: (2018) employed AdamW with cosine annealing to train convolutional neural networks to classify and characterize error-related brain signals measured from intracranial electroencephalography (EEG) recordings.

Original: While their paper does not provide a comparison to Adam,upon email inquiry, they kindly provided us with a direct comparison of the two on their best-performing problem-speciﬁc network architecture Deep4Net and a variant of ResNet.
Revised: While their paper does not provide a comparison to Adam, they kindly provided us with a direct comparison of the two on their best-performing problem-speciﬁc network architecture Deep4Net and a variant of ResNet.

Original: AdamW with the same hyperparameter setting as Adam yielded higher test set accuracy on Deep4Net (73.68% versus 71.37%) and statistically signiﬁcantly higher test set accuracy on ResNet (72.04% versus 61.34%). 
Revised: AdamW with the same hyperparameter setting as Adam yielded higher test set accuracy on Deep4Net (73.68% versus 71.37%) and statistically signiﬁcantly higher test set accuracy on ResNet (72.04% versus 61.34%). Radford et al.

Original: AnonymizedC (2018) employed AdamW to train Transformer (Vaswani et al., 2017) architectures to obtain new state-ofthe-art results on a wide range of benchmarks for natural language understanding. 
Revised: (2018) employed AdamW to train Transformer (Vaswani et al., 2017) architectures to obtain new state-of-the-art results on a wide range of benchmarks for natural language understanding. Zhang et al.

Original: AnonymizedD (2018) compared L 2 regularization vs. weight decay for SGD, Adam and the Kronecker-Factored Approximate Curvature (K-FAC) optimizer (Martens & Grosse, 2015) on the CIFAR datasets with ResNet and VGG architectures, reporting that decoupled weight decay consistently outperformed L 2 regularization in cases where they differ.
Revised: compared L 2 regularization vs. weight decay for SGD, Adam and the Kronecker-Factored Approximate Curvature (K-FAC) optimizer (Martens & Grosse, 2015) on the CIFAR datasets with ResNet and VGG architectures, reporting that decoupled weight decay consistently outperformed L 2 regularization in cases where they differ.

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: Following suggestions that adaptive gradient methods such as Adam might lead to worse generalization than SGD with momentum (Wilson et al., 2017), we identiﬁed and exposed the inequivalence of L 2 regularization and weight decay for Adam.
Revised: Following suggestions that adaptive gradient methods such as Adam might lead to worse generalization than SGD with momentum (Wilson et al., 2017), we identiﬁed and exposed the inequivalence of L 2 regularization and weight decay for Adam.

Original: We empirically showed that our version of Adam with decoupled weight decay yields substantially better generalization performance than the common implementation of Adam with L 2 regularization.
Revised: We empirically showed that our version of Adam with decoupled weight decay yields substantially better generalization performance than the common implementation of Adam with L 2 regularization.

Original: We also proposed to use warm restarts for
Revised: We also proposed to use warm restarts for

Original: Adam to improve its anytime performance.
Revised: Adam to improve its anytime performance.

Original: Our results obtained on image classiﬁcation datasets must be veriﬁed on a wider range of tasks, especially ones where the use of regularization is expected to be important.
Revised: Our results obtained on image classiﬁcation datasets must be veriﬁed on a wider range of tasks, especially ones where the use of regularization is expected to be important.

Original: It would be interesting to integrate our ﬁndings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., 2017).
Revised: It would be interesting to integrate our ﬁndings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., 2017).

Original: While we focused our experimental analysis on Adam, we believe that similar results also hold for other adaptive gradient methods, such as AdaGrad (Duchi et al., 2011) and AMSGrad (Reddi et al., 2018).  
Revised: While we focused our experimental analysis on Adam, we believe that similar results also hold for other adaptive gradient methods, such as AdaGrad (Duchi et al., 2011) and AMSGrad (Reddi et al., 2018).  A CKNOWLEDGMENTS

Revised: This work was supported by the European Research Council (ERC) under the European Union’s

Revised: Horizon 2020 research and innovation programme under grant no. 716721, by the German Research Foundation (DFG), under the BrainLinksBrainTools Cluster of Excellence (grant number

Revised: EXC 1086) and through grant no.

Original: A F ORMAL A NALYSIS OF W EIGHT D ECAY VS L 2 R EGULARIZATION Proof of Proposition
Revised: INST 37/935-1 FUGG, and by the German state of BadenW¨urttemberg through bwHPC. We thank Patryk Chrabaszcz for helping running experiments with ImageNet32x32. We thank Matthias Feurer and Robin Schirrmeister for providing valuable feedback on this paper in several iterations.

Revised: We thank Martin V¨olker, Robin Schirrmeister, and Tonio

Revised: Ball for providing us with a comparison of AdamW and Adam on their EEG data.

Revised: Finally, we thank the following members of the deep learning community for implementing decoupled weight decay in various deep learning libraries:

Revised: • Jingwei Zhang, Lei Tai, Robin Schirrmeister

Revised: and Kashif Rasul for their implementations in PyTorch (see https://github.com/pytorch/pytorch/pull/4429 ) • Phil Jund for his implementation in TensorFlow described at https://www.tensorflow.org/api_docs/python/tf/contrib/opt/ DecoupledWeightDecayExtension • Sylvain Gugger, Anand Saha, Jeremy Howard and other members of fast.ai for their implementation available at https://github.com/sgugger/Adam-experiments • Guillaume Lambard for his implementation in Keras available at https://github. com/GLambard/AdamW_Keras • Yagami Lin for his implementation in Caffe available at https://github.com/ Yagami123/Caffe-AdamW-AdamWR

Revised: A F ORMAL A NALYSIS OF W EIGHT D ECAY VS L 2 R EGULARIZATION Proof of Proposition

Original: The proof for this well-known fact is straight-forward.
Revised: The proof for this well-known fact is straight-forward.

Original: SGD without weight decay has the following iterates on f reg t ( θ) =
Revised: SGD without weight decay has the following iterates on f reg t [Equation] θ

Revised: [Equation]

Revised: =

Revised: [Equation]

Original: f t ( θ ) + λ (cid:48) 2 (cid:107) θ(cid:107) 22 :
Revised: λ [Equation] [Equation] θ

Original: [Equation]
Revised: [Equation]

Original: SGD with weight decay has the following iterates on f t ( θ ) :
Revised: SGD with weight decay has the following iterates on f t ( θ ) :

Original: [Equation]
Revised: [Equation]

Original: These iterates are identical since λ (cid:48)
Revised: These iterates are identical since λ (cid:48)

Original: =
Revised: =

Original: λα .
Revised: λα .

Original: Proof of Proposition
Revised: Proof of Proposition

Original: Similarly to the Proof of Proposition 1, the iterates of O without weight decay on f reg t ( θ ) =
Revised: Similarly to the Proof of Proposition 1, the iterates of O without weight decay on f reg t ( θ ) =

Original: f t ( θ )+ 12 λ (cid:48) (cid:107) θ (cid:107) 

Revised: f t ( θ )+ 12 λ (cid:48) (cid:107) θ (cid:107)

Original: and O with weight decay λ on f t are, respectively:
Revised: and O with weight decay λ on f t are, respectively:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The equality of these iterates for all θ t would imply λ θ t = αλ (cid:48)
Revised: The equality of these iterates for all θ t would imply λ θ t = αλ (cid:48)

Original: M t θ t .
Revised: M t θ t .

Original: This can only hold for all θ t if M t =
Revised: This can only hold for all θ t if M t =

Original: k
Revised: k

Original: I , with k ∈ R , which is not the case for O .
Revised: I , with k ∈ R , which is not the case for O .

Original: Therefore, no L 2 regularizer λ (cid:48) (cid:107) θ (cid:107) 22 exists that makes the iterates equivalent.
Revised: Therefore, no L 2 regularizer λ (cid:48) (cid:107) θ (cid:107) 22 exists that makes the iterates equivalent.

Original: Proof of Proposition
Revised: Proof of Proposition

Original: O without weight decay has the following iterates on f sreg t
Revised: O without weight decay has the following iterates on f sreg t

Original: [Equation]
Revised: [Equation]

Original: where the division by s is element-wise.
Revised: where the division by s is element-wise.

Original: O with weight decay has the following iterates on f t ( θ ) :
Revised: O with weight decay has the following iterates on f t ( θ ) :

Original: [Equation]
Revised: [Equation]

Original: These iterates are identical since λ (cid:48)
Revised: These iterates are identical since λ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: B A DDITIONAL P RACTICAL I MPROVEMENTS OF A DAM
Revised: B A DDITIONAL P RACTICAL I MPROVEMENTS OF A DAM

Original: Having discussed decoupled weight decay for improving Adam’s generalization, in this section we introduce two additional components to improve Adam’s performance in practice.
Revised: Having discussed decoupled weight decay for improving Adam’s generalization, in this section we introduce two additional components to improve Adam’s performance in practice.

Original: B. N ORMALIZED W EIGHT D ECAY
Revised: B. N ORMALIZED W EIGHT D ECAY

Original: Our preliminary experiments showed that different weight decay factors are optimal for different computational budgets (deﬁned in terms of the number of batch passes).
Revised: Our preliminary experiments showed that different weight decay factors are optimal for different computational budgets (deﬁned in terms of the number of batch passes).

Original: Relatedly, Li et al.
Revised: Relatedly, Li et al.

Original: (2017) demonstrated that a smaller batch size (for the same total number of epochs) leads to the shrinking effect of weight decay being more pronounced.
Revised: (2017) demonstrated that a smaller batch size (for the same total number of epochs) leads to the shrinking effect of weight decay being more pronounced.

Original: Here, we propose to reduce this dependence by normalizing the values of weight decay.
Revised: Here, we propose to reduce this dependence by normalizing the values of weight decay.

Original: Speciﬁcally, we replace the hyperparameter λ by a new (more robust) normalized weight decay hyperparameter λ norm , and use this to set λ as λ
Revised: Speciﬁcally, we replace the hyperparameter λ by a new (more robust) normalized weight decay hyperparameter λ norm , and use this to set λ as λ

Original: = λ norm (cid:113) bBT , where b is the batch size, B is the total number of training points and T is the total number of epochs.
Revised: = λ norm (cid:113) bBT , where b is the batch size, B is the total number of training points and T is the total number of epochs.

Original: Thus, λ norm can be interpreted as the weight decay used if only one batch pass is allowed.
Revised: Thus, λ norm can be interpreted as the weight decay used if only one batch pass is allowed.

Original: We emphasize that our choice of normalization is merely one possibility informed by few experiments; a more lasting conclusion we draw is that using some normalization can substantially improve results.
Revised: We emphasize that our choice of normalization is merely one possibility informed by few experiments; a more lasting conclusion we draw is that using some normalization can substantially improve results.

Original: B. A DAM WITH C OSINE A NNEALING AND W ARM R ESTARTS
Revised: B. A DAM WITH C OSINE A NNEALING AND W ARM R ESTARTS

Original: We now apply cosine annealing and warm restarts to Adam, following the recent work of Loshchilov & Hutter (2016).
Revised: We now apply cosine annealing and warm restarts to Adam, following the recent work of Loshchilov & Hutter (2016).

Original: There, the authors proposed Stochastic Gradient Descent with Warm Restarts (SGDR) to improve anytime performance of SGD by quickly cooling down the learning rate according to a cosine schedule and periodically increasing it.
Revised: There, the authors proposed Stochastic Gradient Descent with Warm Restarts (SGDR) to improve anytime performance of SGD by quickly cooling down the learning rate according to a cosine schedule and periodically increasing it.

Original: SGDR has been successfully adopted to lead to new state-of-the-art results for popular image classiﬁcation benchmarks (Huang et al., 2017;
Revised: SGDR has been successfully adopted to lead to new state-of-the-art results for popular image classiﬁcation benchmarks (Huang et al., 2017;

Original: Gastaldi, 2017; Zoph et al., 2017), and we therefore tried extending it to Adam.
Revised: Gastaldi, 2017; Zoph et al., 2017), and we therefore tried extending it to Adam.

Original: However, while our initial version of Adam with warm restarts had better anytime performance than Adam, it was not competitive with SGD with warm restarts, precisely because L 2 regularization was not working as well as in SGD.
Revised: However, while our initial version of Adam with warm restarts had better anytime performance than Adam, it was not competitive with SGD with warm restarts, precisely because L 2 regularization was not working as well as in SGD.

Original: Now, having ﬁxed this issue by means of the original weight decay regularization (Section 2) and also having introduced normalized weight decay (Section B.1), the original work on cosine annealing and warm restarts by Loshchilov & Hutter (2016) directly carries over to Adam.
Revised: Now, having ﬁxed this issue by means of the original weight decay regularization (Section 2) and also having introduced normalized weight decay (Section B.1), the original work on cosine annealing and warm restarts by Loshchilov & Hutter (2016) directly carries over to Adam.

Original: In the interest of keeping the presentation self-contained, we brieﬂy describe how SGDR schedules the change of the effective learning rate in order to accelerate the training of DNNs.
Revised: In the interest of keeping the presentation self-contained, we brieﬂy describe how SGDR schedules the change of the effective learning rate in order to accelerate the training of DNNs.

Original: Here, we decouple the initial learning rate α and its multiplier η t used to obtain the actual learning rate at iteration t (see, e.g., line 8 in Algorithm 1).
Revised: Here, we decouple the initial learning rate α and its multiplier η t used to obtain the actual learning rate at iteration t (see, e.g., line 8 in Algorithm 1).

Original: In SGDR, we simulate a new warm-started run/restart of
Revised: In SGDR, we simulate a new warm-started run/restart of

Original: SGD once T i epochs are performed, where i is the index of the run.
Revised: SGD once T i epochs are performed, where i is the index of the run.

Original: Importantly, the restarts are not performed from scratch but emulated by increasing η t while the old value of θ t is used as an initial solution.
Revised: Importantly, the restarts are not performed from scratch but emulated by increasing η t while the old value of θ t is used as an initial solution.

Original: The amount by which η t is increased controls to which extent the previously acquired information (e.g., momentum) is used.
Revised: The amount by which η t is increased controls to which extent the previously acquired information (e.g., momentum) is used.

Original: Within the i -th run, the value of η t decays according to a cosine annealing (Loshchilov & Hutter, 2016) learning rate for each batch as follows:
Revised: Within the i -th run, the value of η t decays according to a cosine annealing (Loshchilov & Hutter, 2016) learning rate for each batch as follows:

Original: [Equation]
Revised: [Equation]

Original: where η ( i ) min and η ( i ) max are ranges for the multiplier and T cur accounts for how many epochs have been performed since the last restart.
Revised: where η ( i ) min and η ( i ) max are ranges for the multiplier and T cur accounts for how many epochs have been performed since the last restart.

Original: T cur is updated at each batch iteration t and is thus not constrained to integer values.
Revised: T cur is updated at each batch iteration t and is thus not constrained to integer values.

Original: Adjusting (e.g., decreasing) η
Revised: Adjusting (e.g., decreasing) η

Original: ( i ) min and η ( i ) max at every i -th restart (see also Smith (2016)) could potentially improve performance, but we do not consider that option here because it would involve additional hyperparameters.
Revised: ( i ) min and η ( i ) max at every i -th restart (see also Smith (2016)) could potentially improve performance, but we do not consider that option here because it would involve additional hyperparameters.

Original: For η ( i ) max = 1 and η ( i ) min = 0 , one can simplify
Revised: For η ( i ) max = 1 and η ( i ) min = 0 , one can simplify

Original: Eq.
Revised: Eq.

Original: (14) to
Revised: (14) to

Original: [Equation]
Revised: [Equation]

Original: In order to achieve good anytime performance, one can start with an initially small T i (e.g., from 1% to 10% of the expected total budget) and multiply it by a factor of T mult (e.g., T mult = 2 ) at every restart.
Revised: In order to achieve good anytime performance, one can start with an initially small T i (e.g., from 1% to 10% of the expected total budget) and multiply it by a factor of T mult (e.g., T mult = 2 ) at every restart.

Original: The ( i + 1) -th restart is triggered when T cur = T i by setting T cur to 0.
Revised: The ( i + 1) -th restart is triggered when T cur = T i by setting T cur to 0.

Original: An example setting of the schedule multiplier is given in C.
Revised: An example setting of the schedule multiplier is given in C.

Original: Our proposed AdamWR algorithm represents AdamW (see Algorithm 2) with η t following Eq.
Revised: Our proposed AdamWR algorithm represents AdamW (see Algorithm 2) with η t following Eq.

Original: (15) and λ computed at each iteration using normalized weight decay described in the previous section.
Revised: (15) and λ computed at each iteration using normalized weight decay described in the previous section.

Original: We note that normalized weight decay allowed us to use a constant parameter setting across short and long runs performed within AdamWR and SGDWR (SGDW with warm restarts).
Revised: We note that normalized weight decay allowed us to use a constant parameter setting across short and long runs performed within AdamWR and SGDWR (SGDW with warm restarts).

Original: C A N E XAMPLE S ETTING OF THE S CHEDULE M ULTIPLIER
Revised: C A N E XAMPLE S ETTING OF THE S CHEDULE M ULTIPLIER

Original: An example schedule of the schedule multiplier η t is given in SuppFigure 1 for T i =

Revised: An example schedule of the schedule multiplier η t is given in SuppFigure 1 for T i =

Original: = 100 and
Revised: = 100 and

Original: T mult = 2 .
Revised: T mult = 2 .

Original: After the initial 100 epochs the learning rate will reach 0 because η t =100 = 0 .
Revised: After the initial 100 epochs the learning rate will reach 0 because η t =100 = 0 .

Original: Then, since T cur = T i =0 , we restart by resetting T cur = 0 , causing the multiplier η t to be reset to 1 due to Eq.
Revised: Then, since T cur = T i =0 , we restart by resetting T cur = 0 , causing the multiplier η t to be reset to 1 due to Eq.

Original: This multiplier will then decrease again from 1 to 0, but now over the course ofepochs because T i =1 = T i =

Revised: This multiplier will then decrease again from 1 to 0, but now over the course ofepochs because T i =1 = T i =

Original: T mult = 200 .
Revised: T mult = 200 .

Original: Solutions obtained right before the restarts, when η t =(e.g., at epoch indexes 100, 300, 700 and 1500 as shown in SuppFigure 1) are recommended by the optimizer as the solutions, with more recent solutions prioritized.
Revised: Solutions obtained right before the restarts, when η t =(e.g., at epoch indexes 100, 300, 700 and 1500 as shown in SuppFigure 1) are recommended by the optimizer as the solutions, with more recent solutions prioritized.

Original: D A DDITIONAL R ESULTS
Revised: D A DDITIONAL R ESULTS

Original: We investigated whether the use of much longer runs (1800 epochs) of “standard Adam” (Adam with L 2 regularization and a ﬁxed learning rate) makes the use of cosine annealing unnecessary.
Revised: We investigated whether the use of much longer runs (1800 epochs) of “standard Adam” (Adam with L 2 regularization and a ﬁxed learning rate) makes the use of cosine annealing unnecessary.

Original: [Figure]
Revised: [Figure]

Original: Lea r n i ng r a t e m u l t
Revised: Lea r n i ng r a t e m u l t

Original: i p li e r η
Revised: i p li e r η

Original: T 0 =100, T mult =
Revised: T 0 =100, T mult =

Original: SuppFigure 2 shows the results of standard Adam for a 4 by 4 logarithmic grid of hyperparameter settings (the coarseness of the grid is due to the high computational expense of runs forepochs).
Revised: SuppFigure 2 shows the results of standard Adam for a 4 by 4 logarithmic grid of hyperparameter settings (the coarseness of the grid is due to the high computational expense of runs forepochs).

Original: Even after taking the low resolution of the grid into account, the results appear to be at best comparable to the ones obtained with AdamW with 18 times less epochs and a smaller network (see
Revised: Even after taking the low resolution of the grid into account, the results appear to be at best comparable to the ones obtained with AdamW with 18 times less epochs and a smaller network (see

Original: SuppFigure 3, top row, middle).
Revised: SuppFigure 3, top row, middle).

Original: These results are not very surprising given Figure 2 in the main paper (which demonstrates the effectiveness of AdamW) and SuppFigure 1 (which demonstrates the necessity to use some learning rate schedule such as cosine annealing).
Revised: These results are not very surprising given Figure 2 in the main paper (which demonstrates the effectiveness of AdamW) and SuppFigure 1 (which demonstrates the necessity to use some learning rate schedule such as cosine annealing).

Original: Our experimental results with Adam and SGD suggested that the total runtime in terms of the number of epochs affect the basin of optimal hyperparameters (see SuppFigure 3).
Revised: Our experimental results with Adam and SGD suggested that the total runtime in terms of the number of epochs affect the basin of optimal hyperparameters (see SuppFigure 3).

Original: More speciﬁcally, the greater the total number of epochs the smaller the values of the weight decay should be.
Revised: More speciﬁcally, the greater the total number of epochs the smaller the values of the weight decay should be.

Original: SuppFigure 4 shows that our remedy for this problem, the normalized weight decay deﬁned in Eq.
Revised: SuppFigure 4 shows that our remedy for this problem, the normalized weight decay deﬁned in Eq.

Original: (15), simpliﬁes hyperparameter selection because the optimal values observed for short runs are similar to the ones for much longer runs.
Revised: (15), simpliﬁes hyperparameter selection because the optimal values observed for short runs are similar to the ones for much longer runs.

Original: We used our initial experiments on CIFAR-10 to suggest the square root normalization we proposed in Eq.
Revised: We used our initial experiments on CIFAR-10 to suggest the square root normalization we proposed in Eq.

Original: (15) and double-checked that this is not a coincidence on the
Revised: (15) and double-checked that this is not a coincidence on the

Original: ImageNet32x32 dataset (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset with 1.2 million 32 × 32 pixels images, where an epoch is 24 times longer than on CIFAR-10.
Revised: ImageNet32x32 dataset (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset with 1.2 million 32 × 32 pixels images, where an epoch is 24 times longer than on CIFAR-10.

Original: This experiment also supported the square root scaling: the best values of the normalized weight decay observed on CIFAR-10 represented nearly optimal values for ImageNet32x32 (see SuppFigure 3).
Revised: This experiment also supported the square root scaling: the best values of the normalized weight decay observed on CIFAR-10 represented nearly optimal values for ImageNet32x32 (see SuppFigure 3).

Original: In contrast, had we used the same raw weight decay values λ for ImageNet32x32 as for CIFAR10 and for the same number of epochs, without the proposed normalization, λ would have been roughly 5 times too large for ImageNet32x32, leading to much worse performance .
Revised: In contrast, had we used the same raw weight decay values λ for ImageNet32x32 as for CIFAR10 and for the same number of epochs, without the proposed normalization, λ would have been roughly 5 times too large for ImageNet32x32, leading to much worse performance .

Original: The optimal normalized weight decay values were also very similar (e.g., λ norm = 0 .
Revised: The optimal normalized weight decay values were also very similar (e.g., λ norm = 0 .

Original: 025 and λ norm = 0 . 05 ) across SGDW and AdamW.
Revised: 025 and λ norm = 0 . 05 ) across SGDW and AdamW.

Original: SuppFigure 4 is the equivalent of Figure 3 in the main paper, but for ImageNet32x32 instead of for
Revised: SuppFigure 4 is the equivalent of Figure 3 in the main paper, but for ImageNet32x32 instead of for

Original: CIFAR-10.
Revised: CIFAR-10.

Original: The qualitative results are identical: weight decay leads to better training loss (crossentropy) than L 2 regularization, and to an even greater improvement of test error.
Revised: The qualitative results are identical: weight decay leads to better training loss (crossentropy) than L 2 regularization, and to an even greater improvement of test error.

Original: SuppFigure 5 and SuppFigure 6 are the equivalents of Figure 4 in the main paper but supplemented with training loss curves in its bottom row.
Revised: SuppFigure 5 and SuppFigure 6 are the equivalents of Figure 4 in the main paper but supplemented with training loss curves in its bottom row.

Original: The results show that Adam and its variants with decoupled weight decay converge faster (in terms of training loss) on CIFAR-10 than the corresponding
Revised: The results show that Adam and its variants with decoupled weight decay converge faster (in terms of training loss) on CIFAR-10 than the corresponding

Original: SGD variants (the difference for ImageNet32x32 is small).
Revised: SGD variants (the difference for ImageNet32x32 is small).

Original: As is discussed in the main paper, when the same values of training loss are considered, AdamW demonstrates better values of test error than Adam.
Revised: As is discussed in the main paper, when the same values of training loss are considered, AdamW demonstrates better values of test error than Adam.

Original: Interestingly, SuppFigure 5 and SuppFigure 6 show that restart variants AdamWR and
Revised: Interestingly, SuppFigure 5 and SuppFigure 6 show that restart variants AdamWR and

Original: SGDWR also demonstrate better generalization than AdamW and SGDW, respectively.
Revised: SGDWR also demonstrate better generalization than AdamW and SGDW, respectively.