Original: F IXING W EIGHT D ECAY R EGULARIZATION IN A DAM
Revised: F IXING W EIGHT D ECAY R EGULARIZATION IN A DAM

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential beneﬁt of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.
Revised: We note that common implementations of adaptive gradient algorithms, such as Adam, limit the potential beneﬁt of weight decay regularization, because the weights do not decay multiplicatively (as would be expected for standard weight decay) but by an additive constant factor.

Original: We propose a simple way to resolve this issue by decouplingthe weight decay and the optimization steps taken w.r.t.
Revised: We propose a simple way to resolve this issue by decoupling weight decay and the optimization steps taken w.r.t.

Original: the loss function.
Revised: the loss function.

Original: We provide empirical evidence that the proposed decoupling (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam’s generalization performance, allowing it to compete with SGD with momentum on image classiﬁcation datasets (on which it was previously typically outperformed by the latter).
Revised: We provide empirical evidence that our proposed modiﬁcation (i) decouples the optimal choice of weight decay factor from the setting of the learning rate for both standard SGD and Adam, and (ii) substantially improves Adam’s generalization performance, allowing it to compete with SGD with momentum on image classiﬁcation datasets (on which it was previously typically outperformed by the latter).

Original: We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.
Revised: We also demonstrate that longer optimization runs require smaller weight decay values for optimal results and introduce a normalized variant of weight decay to reduce this dependence.

Original: Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-of-the-art results on CIFAR-10 and ImageNet32x32. 
Revised: Finally, we propose a version of Adam with warm restarts (AdamWR) that has strong anytime performance while achieving state-ofthe-art results on CIFAR-10 and ImageNet32x32. Our source code will become available after the review process.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Adaptive gradient methods, such as AdaGrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), and Adam (Kingma & Ba, 2014) have become a default method of choice for training feedforward and recurrent neural networks (Xu et al., 2015; Gregor et al., 2015; Radford et al., 2015).
Revised: Adaptive gradient methods, such as AdaGrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), and Adam (Kingma & Ba, 2014) have become a default method of choice for training feedforward and recurrent neural networks (Xu et al., 2015; Gregor et al., 2015; Radford et al., 2015).

Original: Nevertheless, state-of-the-art results for popular image classiﬁcation datasets, such as CIFARand CIFAR-100, are still obtained by applying SGD with momentum (Huang et al., 2016; 2017; Loshchilov & Hutter, 2016; Gastaldi, 2017).
Revised: Nevertheless, state-of-the-art results for popular image classiﬁcation datasets, such as CIFARand CIFAR-100 Krizhevsky (2009), are still obtained by applying SGD with momentum (Huang et al., 2016; 2017; Loshchilov & Hutter, 2016; Gastaldi, 2017).

Original: Furthermore, Wilson et al.
Revised: Furthermore, Wilson et al.

Original: (2017) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks such as image classiﬁcation, character-level language modeling and constituency parsing.
Revised: (2017) suggested that adaptive gradient methods do not generalize as well as SGD with momentum when tested on a diverse set of deep learning tasks such as image classiﬁcation, character-level language modeling and constituency parsing.

Original: Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., 2016;
Revised: Different hypotheses about the origins of this worse generalization have been investigated, such as the presence of sharp local minima (Keskar et al., 2016;

Original: Dinh et al., 2017) and inherent problems of adaptive gradient methods (Wilson et al., 2017).
Revised: Dinh et al., 2017) and inherent problems of adaptive gradient methods (Wilson et al., 2017).

Original: In this paper, we show that a major factor in Adam’s poor generalization  lies in its dysfunctional implementation of weight decay.
Revised: In this paper, we show that a major factor in the poor generalization of the most popular adaptive gradient method, Adam, lies in its dysfunctional implementation of weight decay; the issue we identify in Adam also pertains to other adaptive gradient methods.

Original: Speciﬁcally, our analysis of Adam given in this paper leads to the following observations:
Revised: Speciﬁcally, our analysis of Adam given in this paper leads to the following observations:

Original: The standard way to implement L regularization / weight decay in Adam is dysfunctional
Revised: The standard way to implement L 2 regularization/weight decay in Adam is dysfunctional.

Original: One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that L 2 regularization / weight decay are implemented suboptimally in common deep learning libraries.
Revised: One possible explanation why Adam and other adaptive gradient methods might be outperformed by SGD with momentum is that L 2 regularization/weight decay are implemented suboptimally in common deep learning libraries.

Original: Therefore, on tasks / datasets where the use of L 2 regularization is beneﬁcial (e.g., on many popular image classiﬁcation datasets), Adam leads to worse results than SGD with momentum, where the L 2 regularization behaves as expected.
Revised: Therefore, on tasks/datasets where the use of L 2 regularization is beneﬁcial (e.g., on many popular image classiﬁcation datasets), Adam leads to worse results than SGD with momentum (for which L 2 regularization behaves as expected).

Original: L 2 regularization and weight decay are not the same thing.
Revised: L 2 regularization and weight decay are not the same thing.

Original: Contrary to common belief, the two techniques are not equivalent.
Revised: Contrary to common belief, the two techniques are not equivalent.

Original: For SGD, they can be made equivalent by a reparameterization of the weight decay factor based on the learning rate; thisreparameterization is not available in closed form foradaptive gradient methods, such as Adam. 
Revised: For SGD, they can be made equivalent by a reparameterization of the weight decay factor based on the learning rate; this is not the case for Adam. In particular, when combined with adaptive gradients, L 2 regularization leads to weights with large gradients being regularized less than they would be when using weight decay.

Original: Optimal weight decay is a function (among other things) of the total number ofepochs / batch passes.
Revised: Optimal weight decay is a function (among other things) of the total number of batch passes/weight updates.

Original: Our empirical analysis of Adam suggests that the longer the runtime / number of batch passes to be performed, the smaller the optimal weight decay.
Revised: Our empirical analysis of Adam suggests that the longer the runtime/number of batch passes to be performed, the smaller the optimal weight decay.

Original: This effect tends to be neglected because hyperparameters are often tuned for a ﬁxed or a comparable number of training epochs.
Revised: This effect tends to be neglected because hyperparameters are often tuned for a ﬁxed or a comparable number of training epochs.

Original: As a result, the values of the weight decay found to perform best for short runs do not generalize to much longer runs.
Revised: As a result, the values of the weight decay found to perform best for short runs do not generalize to much longer runs.

Original: Our contributions are aimed at ﬁxing the issues described above:
Revised: Our contributions are aimed at ﬁxing the issues described above:

Original: Decoupling weight decay from the gradient-based update (Section 2).
Revised: Decoupling weight decay from the gradient-based update (Section 2).

Original: We suggest to decouple the gradient-based update from weight decay for both SGD and Adam.
Revised: We suggest to decouple the gradient-based update from weight decay for both SGD and Adam.

Original: The resulting SGD version SGDW decouples optimal settings of the learning rate and the weight decay factor, and the resulting Adam version AdamW generalizes substantially better than Adam.
Revised: The resulting SGD version SGDW decouples optimal settings of the learning rate and the weight decay factor, and the resulting Adam version AdamW generalizes substantially better than Adam.

Original: Normalizing the values of weight decay (Section 3).
Revised: Normalizing the values of weight decay (Section 3).

Original: We propose to parameterize the weight decay factor as a function of the total number of batch passes.
Revised: We propose to parameterize the weight decay factor as a function of the total number of batch passes.

Original: This leads to a greater invariance of the hyperparameter settings in the sense that the values found to perform best for short runs also perform well for many times longer runs.
Revised: This leads to a greater invariance of the hyperparameter settings in the sense that the values found to perform best for short runs also perform well for many times longer runs.

Original: Adam with warm restarts and normalized weight decay (Section 4).
Revised: Adam with warm restarts and normalized weight decay (Section 4).

Original: After we ﬁx the weight decay in Adam and design AdamW, we design AdamWR to obtain great anytime performance by performing warm restarts.
Revised: After we ﬁx the weight decay in Adam and design AdamW, we introduce AdamWR to obtain strong anytime performance by performing warm restarts.

Original: The main motivation of this paper is to ﬁx the weight decay in Adam to make it competitive w.r.t.
Revised: The main motivation of this paper is to ﬁx the weight decay in Adam to make it competitive w.r.t.

Original: SGD with momentum even for those problems where it did not use to be competitive.
Revised: SGD with momentum even for those problems where it did not use to be competitive.

Original: We hope that as a result, practitioners do not need to switch between Adam and SGD anymore, which in turn should help to reduce the common issue of selecting dataset / task-speciﬁc training algorithms and their hyperparameters.
Revised: We hope that as a result, practitioners do not need to switch between Adam and SGD anymore, which in turn should help to reduce the common issue of selecting dataset/task-speciﬁc training algorithms and their hyperparameters.

Original:  D ECOUPLING THE W EIGHT D ECAY FROM THE GRADIENT - BASED UPDATE
Revised:  D ECOUPLING THE W EIGHT D ECAY FROM THE GRADIENT - BASED UPDATE

Original: In the weight decay described by Hanson & Pratt (1988), the weights  decay exponentially as
Revised: In the weight decay described by Hanson & Pratt (1988), the weights x decay exponentially as

Original: [Equation]
Revised: [Equation]

Original: [Equation] ) where  the  t -th batch gradient ∇ f t ( x t ) is weighted by a learning rate α t .
Revised: where w t deﬁnes the rate of the weight decay at time-step t and ∇ f t ( x t ) is the t -th batch gradient multiplied by a learning rate α t .

Original: Following Hanson & Pratt (1988), one can  consider a bias term (also referred to as the regularization term) accounting for “costs” on weights which are, e.g., quadratic with weight values as forthe L 2 regularization:
Revised: Following Hanson & Pratt (1988), one can also modify the original batch loss f t ( x t ) and consider a bias term (also referred to as the regularization term) accounting for “costs” on weights which are, e.g., quadratic in the weight values as for L 2 regularization:

Original: [Equation]
Revised: [Equation]

Original: where w t deﬁnes the impact of the L 2 regularization.
Revised: where w t deﬁnes the impact of the L 2 regularization.

Original: In order to consider the weight decay regularization, one can reformulate the objective function as in Eq.
Revised: In order to consider the weight decay regularization, one can reformulate the objective function as in Eq.

Original: (2) or directly adjust ∇ f  ( x t ) as
Revised: (2) or directly adjust ∇ f t ( x t ) as

Original: [Equation]
Revised: [Equation]

Original: where w t deﬁnes the rate of the weight decay at time-step t .

Original: Historically, stochastic gradient descent methods inherited this way of implementing the weight decay regularization.
Revised: Historically, stochastic gradient descent methods inherited this way of implementing the weight decay regularization.

Original: The currently most common way (e.g., in popular libraries such as TensorFlow, Keras, PyTorch,
Revised: The currently most common way (e.g., in popular libraries such as TensorFlow, Keras, PyTorch,

Original: Torch, and Lasagne) to introduce the weight decay regularization is to introduce the L 2 regularization term as in Eq.
Revised: Torch, and Lasagne) to introduce the weight decay regularization is to use the L 2 regularization term as in Eq.

Original: (2) or, often equivalently, to directly modify the gradient as in Eq.
Revised: (2) or, often equivalently, to directly modify the gradient as in Eq.

Original: Let’s ﬁrst consider the simple case of SGD with momentum; Algorithm 1 demonstrates modifying the gradients directly in this method (see line 6).
Revised: Let’s ﬁrst consider the simple case of SGD with momentum; Algorithm 1 demonstrates modifying the gradients directly in this method (see line 6).

Original: The weight decay term w t x t − 1 will ﬁrst modify g t (see line 6) and then affect the momentum term m t (see line 8).
Revised: The weight decay term w t x t − 1 will ﬁrst modify g t (see line 6) and then affect the momentum term m t (see line 8).

Original: While the smoothing of the weight decay factor by β1 (see line 8) might be a feature, we note (for simplicity, we omit η t ) that x t will decay by αw t x t − 1 (see line 9) and not w t x t − 1 as one could expect according to the deﬁnition of the weight decay given by Eq. Practically, if one wants to keep the actual weight decay α t w t ﬁxed while changing α t to α (cid:48) t , then w t should be modiﬁed to w (cid:48) t
Revised: While the smoothing of the weight decay factor by β

Original: = α t w t α (cid:48) t . This renders the problem of hyperparameter selection of α t and w t non-separable .
Revised: [Equation]

Original: 1: given learning rate α  ∈ IR , momentum factor β 1 ∈ IR , weight decay factor w ∈ IR 2: initialize time step t ← 0 , parameter vector x t =0 ∈ IR n , ﬁrst moment vector m t =0 ← 0 , schedule multiplier η t =0 ∈ IR 3: repeat 4: t ← t +5: ∇ f t ( x t − 1 ) ← SelectBatch ( x t − 1 ) (cid:46) select batch and return the corresponding gradient 6: g t ← ∇ f t ( x t − 1 ) + w t x t −7: η t ← SetScheduleMultiplier ( t ) (cid:46) can be ﬁxed, decay, be used for warm restarts 8: m
Revised: 1: given learning rate α t ∈ IR , momentum factor β 1 ∈ IR , weight decay factor w ∈ IR 2: initialize time step t ← 0 , parameter vector x t =0 ∈ IR n , ﬁrst moment vector m t =0 ← 0 , schedule multiplier η t =0 ∈ IR 3: repeat 4: t ← t +5: ∇ f t ( x t − 1 ) ← SelectBatch ( x t − 1 ) (cid:46) select batch and return the corresponding gradient 6: g t ← ∇ f t ( x t − 1 ) + w t x t −7: η t ← SetScheduleMultiplier ( t ) (cid:46) can be ﬁxed, decay, be used for warm restarts 8: m

Original: t ← β 1 m t − 1 + η t α t g t 9: x t ← x t − 1 − m t − η t w t x t −10: until stopping criterion is met 11: return optimized parameters x t 1: given
Revised: t ← β 1 m t − 1 + η t α t g t 9: x t ← x t − 1 − m t − η t w t x t −10: until stopping criterion is met 11: return optimized parameters x t moment vector m

Original: IR time t ← 0 , parameter vector 
Revised: 0 ← 0 , second moment vector v t 0 ← 0 , schedule multiplier η

Original: =0 ∈ IR n , ﬁrst moment vector m t =0 ← 0 , second moment vector v t =0 ← 0, schedule multiplier η t =0 ∈ IR 3: repeat 4: t ← t +5: ∇ f t ( x t − 1 ) ← SelectBatch ( x t − 1 ) (cid:46)select batch and returnthe corresponding gradient 6: g t ← ∇ f t ( x t − 1 ) + w t x t −7: m t ← β 1 m t − 1 + (1 − β 1 ) g t (cid:46) here and below all operations are element-wise 8: v t ← β 2 v t − 1 + (1 − β 2 ) g 2 t 9: ˆ m t ← m t / (1 − β t 1 ) (cid:46) here, β 1 is taken to the power of t 10: ˆ v t ← v t / (1 − β t 2 ) (cid:46) here, β 2 is taken to the power of t 11: η t ← SetScheduleMultiplier ( t ) (cid:46) can be ﬁxed, decay, be used for warm restarts 12: x t ← x t − 1 − η t (cid:16) α  ˆ m t / ( √ ˆ v t + (cid:15) ) + w t x t − 1 (cid:17) 13: until stopping criterion is met 14: return optimized parameters x 
Revised: 0 ∈ IR 3: repeat 4: t ← t +5: ∇ f t ( x t − 1 ) ← SelectBatch ( x t − 1 ) (cid:46) return corresponding gradient 6: g t ← ∇ f t ( x t − 1 ) + w t x t −7: m t ← β 1 m t − 1 + (1 − β 1 ) g t (cid:46) here and below all operations are element-wise 8: v t ← β 2 v t − 1 + (1 − β 2 ) g 2 t 9: ˆ m t ← m t / (1 − β t 1 ) (cid:46) here, β 1 is taken to the power of t 10: ˆ v t ← v t / (1 − β t 2 ) (cid:46) here, β 2 is taken to the power of t 11: η t ← SetScheduleMultiplier ( t ) (cid:46) can be ﬁxed, decay, be used for warm restarts 12: x t ← x t − 1 − η t (cid:16) α t ˆ m t / ( √ ˆ v t + (cid:15) ) + w t x t − 1 (cid:17) 13: until stopping criterion is met 14: return optimized parameters x t (see line 8) might be a feature, we note (for simplicity, we omit η t ) that x t will decay by α t w t x t −(see line 9) and not w t x t − 1 as one could expect according to the deﬁnition of the weight decay given by Eq.

Original:  t 
Revised: Practically, if one wants to keep the actual weight decay α t w t ﬁxed while changing α t to α (cid:48) t , then w t should be modiﬁed to w (cid:48) t = α t w t α (cid:48) t . This renders the problem of hyperparameter selection of α t and w t non-separable .

Original: We propose to ﬁx this problem by following the original deﬁnition of weight decay given by Eq.
Revised: We propose to ﬁx this problem by following the original deﬁnition of weight decay given by Eq.

Original: (1) and decay the weights simultaneously with the update of x t based on gradient information in
Revised: (1) and decay the weights simultaneously with the update of x t based on gradient information in

Original: Line 9 of Algorithm 1.
Revised: Line 9 of Algorithm 1.

Original: This yields our proposed SGD variant SGDW with momentum.
Revised: This yields our proposed SGD variant SGDW with momentum.

Original: Although the proposed simple modiﬁcation explicitly decouples w t and α t , some problem-dependent implicit coupling is likely to remain.
Revised: Although the proposed simple modiﬁcation explicitly decouples w t and α t , some problem-dependent implicit coupling is likely to remain.

Original: In order to account for a possible scheduling of both α t and w t , we introduce a scaling factor η t delivered by a user-deﬁned procedure SetScheduleMultiplier ( t ) .
Revised: In order to account for a possible scheduling of both α t and w t , we introduce a scaling factor η t delivered by a user-deﬁned procedure SetScheduleMultiplier ( t ) .

Revised: It should be noted that when L 2 regularization is used, weight decay contributes to the batch gradient and thus effectively is scheduled in the same way as the learning rate.

Original:  Having shown that using L 2 regularization instead of weight decay already couples regularization and learning rate in the simple case of SGD with momentum, we now consider adaptive gradient optimizers, such as the Adam algorithm proposed by Kingma & Ba (2014), in which the coupling leads to even more unintended behavior.
Revised: Now, since we decouple the two we should also remember to schedule both of them with η t . Having shown that using L 2 regularization instead of weight decay already couples regularization and learning rate in the simple case of SGD with momentum, we now consider adaptive gradient optimizers, such as the Adam algorithm proposed by Kingma & Ba (2014), in which the coupling leads to even more unintended behavior.

Original: As an adaptive gradient method, Adam maintains a vector v t responsible for storing smoothed amplitudes of parameter-wise gradients g 2 t (see line 8 in Algorithm 2).
Revised: As an adaptive gradient method, Adam maintains a vector v t responsible for storing smoothed amplitudes of parameter-wise gradients g 2 t (see line 8 in Algorithm 2).

Original: These factors are used to control parameter-wise learning rates by normalizing parameter-wise gradients by √ ˆ v t + (cid:15) in line 12 of Algorithm 2.
Revised: These factors are used to control parameter-wise learning rates by normalizing parameter-wise gradients by √ ˆ v t + (cid:15) in line 12 of Algorithm 2.

Original: The common way to introduce the weight decay w t x t − 1 to Adam results in an update which only distantly resembles the original weight decay given by Eq.
Revised: The common way to introduce the weight decay w t x t − 1 to Adam results in an update which only distantly resembles the original weight decay given by Eq.

Original: (1) because the v t vectors are not only responsible for the parameter-wise amplitudes of g t but also for the parameter-wise amplitudes of weights x t .
Revised: (1) because the v t vectors are not only responsible for the parameter-wise amplitudes of g t but also for the parameter-wise amplitudes of weights x t .

Original: The amplitudes are then used to renormalize ˆ m t as given in line 12 of Algorithm 2.
Revised: The amplitudes are then used to renormalize ˆ m t as given in line 12 of Algorithm 2.

Original: To gain a bit of intuition, let us consider the case when t is large, causing β t 1 and β t 2 to go to zero andx t ←
Revised: To gain a bit of intuition, let us consider the case when t is large, causing β t 1 and β t 2 to go to zero and

Original: [Equation]
Revised: [Equation]

Original: , with g t = ∇ f t ( x t

Original: [Equation]

Original: [Equation]

Original: where operations are performed parameter-wise.
Revised: where operations are performed parameter-wise.

Original: Not only the batch gradient ∇ f t ( x t − 1 ) is normalized but also the weight decay w t x t − 1 itself.
Revised: Not only the batch gradient ∇ f t ( x t − 1 ) is normalized but also the weight decay w t x t − 1 itself.

Original: Since this formula normalizes updates by their typical amplitudes, the decay of weights does not account for amplitudes anymore, leading to the relative decay being weaker for weights with large gradients.
Revised: Since this formula normalizes updates by their typical amplitudes, the decay of weights does not account for amplitudes anymore, leading to the relative decay being weaker for weights with large gradients.

Original: This is a correct implementation of Lregularization, but not of weight decay.
Revised: This is a correct implementation of Lregularization, but not of weight decay.

Original: Therefore, it might be misleading to use the two terms interchangeably, as is commonly done in the literature.
Revised: Therefore, it might be misleading to use the two terms interchangeably, as is commonly done in the literature.

Original: We note that this difference between the two mechanisms for Adam has not been investigated and / or described before.
Revised: We note that this difference between the two mechanisms for Adam has not been investigated and/or described before.

Original: As in the case of SGDW, we propose to follow the original deﬁnition of weight decay and perform it simultaneously with the gradient-based update as shown in line 12 of Algorithm 2 for AdamW . 
Revised: As in the case of SGDW, we propose to follow the original deﬁnition of weight decay and perform it simultaneously with the gradient-based update as shown in line 12 of Algorithm 2 for AdamW . As we will demonstrate experimentally (in Section 5.2), AdamW generalizes much better than Adam.

Original:  N ORMALIZED W EIGHT D ECAY
Revised:  N ORMALIZED W EIGHT D ECAY

Original: Since our preliminary experiments showed that different weight decay factors are optimal for different computational budgets (deﬁned in terms of the number of batch passes), we introduce a normalized weight decay to reduce this dependence.
Revised: Since our preliminary experiments showed that different weight decay factors are optimal for different computational budgets (deﬁned in terms of the number of batch passes), we introduce a normalized weight decay to reduce this dependence.

Original: At iteration t , w t is set as follows:
Revised: At iteration t , w t is set as follows:

Original: [Equation]
Revised: [Equation]

Original: where b t is the batch size, B is the total number of training points to be used in one epoch and T i is the total number of epochs within the i -th run / restart of the algorithm.
Revised: where b t is the batch size, B is the total number of training points to be used in one epoch and T i is the total number of epochs within the i -th run/restart of the algorithm.

Original: Thus, w norm can be interpreted as the weight decay to be used if only one batch pass is allowed.
Revised: Thus, w norm can be interpreted as the weight decay to be used if only one batch pass is allowed.

Revised: We note a recent relevant observation of Li et al.

Revised: (2017) who demonstrated that a smaller batch size (for the same total number of epochs) leads to the shrinking effect of weight decay being more pronounced.

Revised: Here, we propose to address that effect with normalized weight decay.

Original:  A DAM WITH W ARM R ESTARTS AND N ORMALIZED W EIGHT D ECAY
Revised:  A DAM WITH W ARM R ESTARTS AND N ORMALIZED W EIGHT D ECAY

Original: In order to accelerate the training of DNNs, one can schedule the change of  the learning rate α t .
Revised: We now apply warm restarts to Adam, following the recent work of Loshchilov & Hutter (2016). There, the authors proposed Stochastic Gradient Descent with Warm Restarts (SGDR) to improve anytime performance of SGD by quickly cooling down the learning rate and periodically increasing it. SGDR has been successfully adopted to lead to new state-of-the-art results for popular image classiﬁcation benchmarks (Huang et al., 2017; Gastaldi, 2017), and we therefore tried extending it to Adam. However, while our initial version of Adam with warm restarts had better anytime performance than Adam, it was not competitive with SGD with warm restarts, precisely because of Adam’s dysfunctional weight decay.

Original:  Here, we decouple the initial learning rate and its multiplier η t used to obtain the actual learning rate at iteration t (see, e.g., line 8 in Algorithm 1).Among multiple possible ways to schedule η t , we employ Stochastic Gradient Descent with Warm Restarts (SGDR) by Loshchilov & Hutter (2016) because of its good anytime performance and its successful adoption which has led to new stateof-the-art results (Huang et al., 2017; Gastaldi, 2017).
Revised: Now, having ﬁxed weight decay regularization (Section 2) and also having introduced normalized weight decay (Section 3), the work of Loshchilov & Hutter (2016) on warm restarts directly carries over, and we use it to construct AdamWR to fully beneﬁt from warm restarts. In the interest of keeping the presentation self-contained, we brieﬂy describe how SGDR schedules the change of the effective learning rate in order to accelerate the training of DNNs. Here, we decouple the initial learning rate and its multiplier η t used to obtain the actual learning rate at iteration t (see, e.g., line 8 in Algorithm 1).

Original: In SGDR, we simulate a new warm-started run / restart of SGD once T i epochs are performed, where i is the index of the run.
Revised: In SGDR, we simulate a new warm-started run/restart of SGD once T i epochs are performed, where i is the index of the run.

Original: Importantly, the restarts are not performed from scratch but emulated by increasing η t while the old value of x t is used as an initial solution.
Revised: Importantly, the restarts are not performed from scratch but emulated by increasing η t while the old value of x t is used as an initial solution.

Original: The amount of this increase controls to which extent the previously acquired information (e.g., momentum) is used.
Revised: The amount by which η t is increases controls to which extent the previously acquired information (e.g., momentum) is used.

Original: Within the i -th run, the value of η t decays according to the cosine annealing (Loshchilov & Hutter, 2016) for each batch as follows:
Revised: Within the i -th run, the value of η t decays according to the cosine annealing (Loshchilov & Hutter, 2016) for each batch as follows:

Original: [Equation]
Revised: [Equation]

Original: where η ( i ) min and η ( i ) max are ranges for the multiplier and T cur accounts for how many epochs have been performed since the last restart.
Revised: where η ( i ) min and η ( i ) max are ranges for the multiplier and T cur accounts for how many epochs have been performed since the last restart.

Original: Since T cur is updated at each batch iteration t , it can take discretized values such as 0.1, 0.2, etc.
Revised: T cur is updated at each batch iteration t and is thus not constrained to integer values.

Original: As discussed by Loshchilov & Hutter (2016), one can adjust (e.g., decrease) η
Revised: Adjusting (e.g., decreasing) η

Original: ( i ) min and η ( i ) max at every i -th restart (see also Smith (2016)). While this could potentially improvethe performance, following Loshchilov & Hutter (2016), we do not consider that option in our experiments because it would involve additional hyperparameters.
Revised: ( i ) min and η ( i ) max at every i -th restart (see also Smith (2016)) could potentially improve performance, but we do not consider that option in our experiments because it would involve additional hyperparameters.

Original: For η ( i ) max =and η ( i ) min = 0 , one can simplify Eq.
Revised: For η ( i ) max = 1 and η ( i ) min = 0 , one can simplify Eq.

Original: (6) to
Revised: (6) to

Original: [Equation]
Revised: [Equation]

Original: In order to maintain a good anytime performance, one can start with an initially small T i (e.g., from 1% to 10% of the expected total budget) and multiply it by a factor of T mult (e.g., T mult = 2 ) at every restart.
Revised: In order to maintain a good anytime performance, one can start with an initially small T i (e.g., from 1% to 10% of the expected total budget) and multiply it by a factor of T mult (e.g., T mult = 2 ) at every restart.

Original: The ( i + 1) -th restart is triggered when T cur = T i by setting T cur to 0.
Revised: The ( i + 1) -th restart is triggered when T cur = T i by setting T cur to 0.

Original: An example setting of the schedule multiplier is given in Section 1.1 of the supplementary material.
Revised: An example setting of the schedule multiplier is given in Section 1.1 of the supplementary material.

Revised: Note that the effective learning rate is controlled by η t α t where α t is set to the initial learning rate and stays constant in our experimental setup.

Revised: The reason why we employ α t and not simply α is to account for possible practical extensions, e.g., to adapt α t as a function of batch size in (scheduled) large-batch settings.

Original: Our proposed AdamWR algorithm represents AdamW given in Algorithm 2 with η t following Eq.
Revised: Our proposed AdamWR algorithm represents AdamW given in Algorithm 2 with η t following Eq.

Original: (7) and w t computed at each iteration using normalized weight decay according to Eq.
Revised: (7) and w t computed at each iteration using normalized weight decay according to Eq.

Revised: We note that normalized weight decay allowed us to use a constant parameter setting across short and long runs performed within AdamWR.

Revised: Equivalently to AdamWR, we deﬁne SGDWR as SGDW with warm restarts.

Original:  E XPERIMENTAL V ALIDATION
Revised:  E XPERIMENTAL V ALIDATION

Original: Our experimental setup follows that of Gastaldi (2017), who proposed, in addition to L 2 regularization, to apply the new Shake-Shake regularization to a 3-branch residual neural network.
Revised: Our experimental setup follows that of Gastaldi (2017), who proposed, in addition to L 2 regularization, to apply the new Shake-Shake regularization to a 3-branch residual neural network.

Original: Gastaldi (2017) showed that this regularization allowed to achieve new state-of-the-art results of 2.86% on the CIFAR-10 dataset  and of 15.85% on CIFAR-100.
Revised: Gastaldi (2017) showed that this regularization allowed to achieve new state-of-the-art results of 2.86% on the CIFAR-10 dataset (Krizhevsky, 2009) and of 15.85% on CIFAR-100.

Original: The network was trained by SGDR with batch size 128 for 1800 epochs ( T 0 = 1800 ) without restarts with the learning rate scheduled by Eq
Revised: The network was trained by SGDR with batch size 128 for 1800 epochs ( T 0 = 1800 ) without restarts with the learning rate scheduled by Eq.

Original: The regular data augmentation procedure used for the CIFAR datasets was applied.
Revised: The regular data augmentation procedure used for the CIFAR datasets was applied.

Original: We used the same model / source code based on fb.resnet.torch 1 .
Revised: We used the same model/source code based on fb.resnet.torch 1 .

Original: The base networks are a 26 2x64d ResNet (i.e. the network has a depth of 26, 2 residual branches and the ﬁrst residual block has a width of 64) and 26 2x96d ResNet with 11.6M and 25.6M parameters, respectively.
Revised: The base networks are a 26 2x64d ResNet (i.e. the network has a depth of 26, 2 residual branches and the ﬁrst residual block has a width of 64) and 26 2x96d ResNet with 11.6M and 25.6M parameters, respectively.

Original: For a detailed description of the network and the Shake-Shake method, we refer the interested reader to Gastaldi (2017).
Revised: For a detailed description of the network and the Shake-Shake method, we refer the interested reader to Gastaldi (2017).

Original: 5. D ECOUPLING THE W EIGHT D ECAY FROM THE GRADIENT - BASED UPDATE
Revised: 5. D ECOUPLING T HE W EIGHT D ECAY AND I NITIAL L EARNING R ATE P ARAMETERS

Original: In order to verify our hypothesis about the coupling of the initial learning rate α t and the weight decay factor w t , we trained a 2x64d ResNet with cosine annealing for 100 epochs with different settings of α t and w t .
Revised: In order to verify our hypothesis about the coupling of the initial learning rate α t and the weight decay factor w t , we trained a 2x64d ResNet with cosine annealing for 100 epochs with different settings of α t and w t .

Original: Throughout this paper, we scheduled the learning rate with cosine annealing because it leads to better results than a ﬁxed learning rate (see SuppFigure 4 in the supplementary material).
Revised: Throughout this paper, we scheduled the learning rate with cosine annealing because it leads to better results than a ﬁxed learning rate (see SuppFigure 1 in the supplementary material).

Original: Figure 1 compares SGD vs. SGDW (top row) and Adam vs. AdamW (bottom row).
Revised: Figure 1 compares SGD vs. SGDW (top row) and Adam vs. AdamW (bottom row).

Original: For the case of SGD (Figure 1, top left), weight decay is not decoupled from the learning rate (the common way as described in Algorithm 1), and the ﬁgure clearly shows that the basin of best hyperparameter settings (depicted by color and top-10 hyperparameter settings by black circles) is not aligned with the x-axis or y-axis but lies on the diagonal.
Revised: For the case of SGD (Figure 1, top left), weight decay is not decoupled from the learning rate (the common way as described in Algorithm 1), and the ﬁgure clearly shows that the basin of best hyperparameter settings (depicted by color and top-10 hyperparameter settings by black circles) is not aligned with the x-axis or y-axis but lies on the diagonal.

Original: This suggests that the two hyperparameters are interdependent and need to be changed simultaneously, while only changing one of them might substantially worsen results.
Revised: This suggests that the two hyperparameters are interdependent and need to be changed simultaneously, while only changing one of them might substantially worsen results.

Original: Consider, e.g., the setting at the top left black circle ( α t = 1 / 2 , w t = 1 / 8 ∗ 0 . 001 ); only changing either α t or w t by itself would worsen results, while changing both of them could still yield clear improvements.
Revised: Consider, e.g., the setting at the top left black circle ( α t = 1 / 2 , w t = 1 / 8 ∗ 0 . 001 ); only changing either α t or w t by itself would worsen results, while changing both of them could still yield clear improvements.

Original: We note that this coupling of initial learning rate and weight decay factor might have contributed to SGD’s reputation of being very sensitive to its hyperparameter settings.
Revised: We note that this coupling of initial learning rate and weight decay factor might have contributed to SGD’s reputation of being very sensitive to its hyperparameter settings.

Original: In contrast, the results for our new SGDW in Figure 1 (top right) show that SGDW decouples weight decay and initial learning rate.
Revised: In contrast, the results for our new SGDW in Figure 1 (top right) show that SGDW decouples weight decay and initial learning rate.

Original: The proposed approach renders the two hyperparameters more separable: even if the learning rate is not well tuned yet (e.g., consider the value of 1/1024 in
Revised: The proposed approach renders the two hyperparameters more separable: even if the learning rate is not well tuned yet (e.g., consider the value of 1/1024 in

Original: Figure 1, top right), leaving it ﬁxed and only optimizing the weight decay factor would yield a good value (of 1/4*0.001).
Revised: Figure 1, top right), leaving it ﬁxed and only optimizing the weight decay factor would yield a good value (of 1/4*0.001).

Original: This is not the case for the original SGD shown in Figure 1 (top left).
Revised: This is not the case for the original SGD shown in Figure 1 (top left).

Original: The results for different hyperparameter settings of the original Adam are given in Figure 1 (bottom left).
Revised: The results for different hyperparameter settings of the original Adam are given in Figure 1 (bottom left).

Original: Adam’s best hyperparameter settings performed clearly worse than SGD’s best ones (compare
Revised: Adam’s best hyperparameter settings performed clearly worse than SGD’s best ones (compare

Original: Figure 1, top left).
Revised: Figure 1, top left).

Original: While both methods use the original way to employ weight decay, the original
Revised: While both methods use the original way to employ weight decay, the original

Original: Adam did not beneﬁt from it at all: its best results obtained for non-zero weight decay values were comparable to the best ones obtained without the weight decay regularization, i.e., when w t = 0 .
Revised: Adam did not beneﬁt from it at all: its best results obtained for non-zero weight decay values were comparable to the best ones obtained without the weight decay regularization, i.e., when w t = 0 .

Original: Similarly to the original SGD, the shape of the hyperparameter landscape suggests that the two hyperparameters are coupled.
Revised: Similarly to the original SGD, the shape of the hyperparameter landscape suggests that the two hyperparameters are coupled.

Original: In contrast, the results for our new AdamW in Figure 1 (bottom right) show that AdamW largely decouples weight decay and learning rate.
Revised: In contrast, the results for our new AdamW in Figure 1 (bottom right) show that AdamW largely decouples weight decay and learning rate.

Original: The results for the best hyperparameter settings were substantially better than the best ones of the original Adam and rivaled those of SGD and SGDW.
Revised: The results for the best hyperparameter settings were substantially better than the best ones of the original Adam and rivaled those of SGD and SGDW.

Original: In summary, the experimental results in Figure 1 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simpliﬁes the problem of hyperparameter tuning in SGD and improves Adam’s performance to be competitive w.r.t.
Revised: In summary, the experimental results in Figure 1 support our hypothesis that the weight decay and learning rate hyperparameters can be decoupled, and that this in turn simpliﬁes the problem of hyperparameter tuning in SGD and improves Adam’s performance to be competitive w.r.t.

Original: SGD with momentum.
Revised: SGD with momentum.

Original: 5. B ETTER G ENERALIZATION OF A DAM W
Revised: 5. B ETTER G ENERALIZATION OF A DAM W

Original: While the previous experiment suggested that the basin of optimal hyperparameters of AdamW is broader and deeper than the one of Adam, we next investigated the results for much longer runs of 1800 epochs to compare the generalization capabilities of AdamW and Adam.
Revised: While the previous experiment suggested that the basin of optimal hyperparameters of AdamW is broader and deeper than the one of Adam, we next investigated the results for much longer runs of 1800 epochs to compare the generalization capabilities of AdamW and Adam.

Original: We ﬁxed the initial learning rate to 0.001 which represents both the default learning rate for Adam and the one which showed reasonably good results in our experiments.
Revised: We ﬁxed the initial learning rate to 0.001 which represents both the default learning rate for Adam and the one which showed reasonably good results in our experiments.

Original: Figure 2 shows the results for 12 settings of the weight decay of Adam and 7 settings of the normalized weight decay of AdamW.
Revised: Figure 2 shows the results for 12 settings of the weight decay of Adam and 7 settings of the normalized weight decay of AdamW.

Original: Interestingly, while the dynamics of the learning curves of Adam and AdamW often coincided for the ﬁrst half of the training run, AdamW often led to lower training loss and test errors (see Figure 2 top left and top right, respectively).
Revised: Interestingly, while the dynamics of the learning curves of Adam and AdamW often coincided for the ﬁrst half of the training run, AdamW often led to lower training loss and test errors (see Figure 2 top left and top right, respectively).

Original: Importantly, the use of weight decay in Adam did not yield as good results as in AdamW (see also Figure 2, bottom left).
Revised: Importantly, the use of weight decay in Adam did not yield as good results as in AdamW (see also Figure 2, bottom left).

Original: Next, we investigated whether AdamW’s better results were only due to better convergence or due to better generalization.
Revised: Next, we investigated whether AdamW’s better results were only due to better convergence or due to better generalization.

Original: The results in
Revised: The results in

Original: Figure 2 (bottom right) for the best settings of Adam and AdamW suggest that AdamW did not only yield better training loss but also yielded better generalization performance for similar training loss values .
Revised: Figure 2 (bottom right) for the best settings of Adam and AdamW suggest that AdamW did not only yield better training loss but also yielded better generalization performance for similar training loss values .

Original: The results on ImageNet32x32 (see SuppFigure 3 in the supplementary material) lead to the same conclusion of substantially improved generalization performance.
Revised: The results on ImageNet32x32 (see SuppFigure 4 in the supplementary material) lead to the same conclusion of substantially improved generalization performance.

Original: 5. E ASIER H YPERPARAMETER S ELECTION DUE TO N ORMALIZED W EIGHT DECAY
Revised: 5. E ASIER H YPERPARAMETER S ELECTION DUE TO N ORMALIZED W EIGHT DECAY

Original: Our experimental results with Adam and SGD suggested that the total runtime in terms of the number of epochs affect the basin of optimal hyperparameters (see SuppFigure 2 in the supplementary material).
Revised: Our experimental results with Adam and SGD suggested that the total runtime in terms of the number of epochs affect the basin of optimal hyperparameters (see SuppFigure 3 in the supplementary material).

Original: More speciﬁcally, the greater the total number of epochs the smaller the values of the weight decay should be.
Revised: More speciﬁcally, the greater the total number of epochs the smaller the values of the weight decay should be.

Original: SuppFigure 2 shows that our remedy for this problem, the normalized weight decay deﬁned in Eq.
Revised: SuppFigure 3 shows that our remedy for this problem, the normalized weight decay deﬁned in Eq.

Original: (7), simpliﬁes hyperparameter selection because the optimal values observed for short runs are similar to the ones for much longer runs.
Revised: (7), simpliﬁes hyperparameter selection because the optimal values observed for short runs are similar to the ones for much longer runs.

Original: While our initial experiments on CIFAR-10 suggested the square root
Revised: While our initial experiments on CIFAR-10 suggested the square root

Original: ﬁt we proposed in Eq.
Revised: ﬁt we proposed in Eq.

Original: (7), to double-check that this is not a coincidence, we also performed experiments on the ImageNet32x32 dataset (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset with 1.2 million 32 × 32 pixels images, where an epoch is 24 times longer than on CIFAR-10.
Revised: (7), to double-check that this is not a coincidence, we also performed experiments on the ImageNet32x32 dataset (Chrabaszcz et al., 2017), a downsampled version of the original ImageNet dataset with 1.2 million 32 × 32 pixels images, where an epoch is 24 times longer than on CIFAR-10.

Original: This experiment also supported the square root scaling: the best values of the normalized weight decay observed on CIFAR-10 represented nearly optimal values for ImageNet32x32 (see SuppFigure 2).
Revised: This experiment also supported the square root scaling: the best values of the normalized weight decay observed on CIFAR-10 represented nearly optimal values for ImageNet32x32 (see SuppFigure 3).

Original: In contrast, had we used the same raw weight decay values w t for ImageNet32x32 as for CIFAR-10 and for the same number of epochs, without the proposed normalization, w t would have been roughly 5 greater than optimal for ImageNet32x32, leading to much worse performance .
Revised: In contrast, had we used the same raw weight decay values w t for ImageNet32x32 as for CIFAR-10 and for the same number of epochs, without the proposed normalization, w t would have been roughly 5 greater than optimal for ImageNet32x32, leading to much worse performance .

Original: The optimal normalized weight decay values were also very similar  across SGDW and AdamW.
Revised: The optimal normalized weight decay values were also very similar (e.g., w norm = 0 . 025 and w norm = 0 . 05 ) across SGDW and AdamW.

Original: 5. A DAM WR WITH W ARM R ESTARTS FOR BETTER ANYTIME PERFORMANCE
Revised: We investigated whether the use of much longer runs (1800 epochs) of the original Adam with L 2 regularization makes the use of cosine annealing unnecessary. The results of Adam without cosine annealing (i.e., with ﬁxed learning rate) for a 4 by 4 logarithmic grid of hyperparameter settings are given in SuppFigure 5 in the supplementary material.

Original: Finally, following Loshchilov & Hutter (2016), who proposed to improve anytime performance of 
Revised: Even after taking into account the low resolution of the grid, the results appear to be at best comparable to the ones obtained with AdamW with 18 times less epochs and a smaller network (see SuppFigure 2). These results are not very surprising given Figure 1 (which demonstrates the effectiveness of AdamW) and SuppFigure(which demonstrates the necessity to use some learning rate schedule such as cosine annealing). 5. A DAM WR WITH W ARM R ESTARTS FOR BETTER ANYTIME PERFORMANCE

Original: SGD by performing warm restarts, we investigated  AdamWR as an extension of AdamW (using normalized weight decay to avoid the need for a different weight decay factor for restarts with longer annealing schedules).
Revised: Finally, we investigated the strong anytime performance AdamWR obtains from warm restarts (using normalized weight decay to avoid the need for a different weight decay factor for restarts with longer annealing schedules).

Original: As Figure 3 shows, AdamWR greatly sped up AdamW on CIFAR-10 and ImageNet32x32, up to a factor of 10 (see the results at the ﬁrst restart).
Revised: As Figure 3 shows, AdamWR greatly sped up AdamW on CIFAR10 and ImageNet32x32, up to a factor of 10 (see the results at the ﬁrst restart).

Original: For the default learning rate of 0.001, AdamW achieved 15% relative improvement in test errors compared to Adam both on CIFAR-10 (see Figure 2) and ImageNet32x32 (see SuppFigure 3).
Revised: For the default learning rate of 0.001, AdamW achieved 15% relative improvement in test errors compared to Adam both on CIFAR-10 (also see Figure 2) and ImageNet32x32 (also see SuppFigure 4).

Original: AdamWR achieved the same improved results but with a much better anytime performance.
Revised: AdamWR achieved the same improved results but with a much better anytime performance.

Original: These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on ImageNet32x32.
Revised: These improvements closed most of the gap between Adam and SGDWR on CIFAR-10 and yielded comparable performance on ImageNet32x32.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: Following suggestions that adaptive gradient methods such as Adam might lead to worse generalization than SGD with momentum (Wilson et al., 2017), we identiﬁed at least one possible explanation to this phenomenon: the dysfunctional use of L 2 regularization and weight decay.
Revised: Following suggestions that adaptive gradient methods such as Adam might lead to worse generalization than SGD with momentum (Wilson et al., 2017), we identiﬁed at least one possible explanation to this phenomenon: the dysfunctional use of L 2 regularization and weight decay.

Original: We proposed a simple ﬁx to deal with this issue, yielding substantially better generalization performance in our
Revised: We proposed a simple ﬁx to deal with this issue, yielding substantially better generalization performance in our

Original: AdamW variant.
Revised: AdamW variant.

Original: We also proposed normalized weight decay and warm restarts for Adam, showing that a more robust hyperparameteer selection and a better anytime performance can be achieved in our new AdamWR variant.
Revised: We also proposed normalized weight decay and warm restarts for Adam, showing that a more robust hyperparameteer selection and a better anytime performance can be achieved in our new AdamWR variant.

Original: Our preliminary results obtained with AdamW and AdamWR on image classiﬁcation datasets must be veriﬁed on a wider range of tasks, especially the ones where the use of regularization is expected to be important.
Revised: Our preliminary results obtained with AdamW and AdamWR on image classiﬁcation datasets must be veriﬁed on a wider range of tasks, especially the ones where the use of regularization is expected to be important.

Original: It would be interesting to integrate our ﬁndings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., 2017). 
Revised: It would be interesting to integrate our ﬁndings on weight decay into other methods which attempt to improve Adam, e.g, normalized direction-preserving Adam (Zhang et al., 2017). While we focussed our experimental analysis on Adam, we believe that similar results also hold for other adaptive gradient methods, such as AdaGrad (Duchi et al., 2011) and RMSProp (Tieleman & Hinton, 2012).

Revised: The results shown in Figure 2 suggest that Adam and AdamW follow very similar curves most of the time until the third phase of the run where AdamW starts to branch out to outperform Adam.

Original: Advani & Saxe (2017) analytically showed that in the limited data regime ofdeep networks the presence of eigenvalues that are zero forms a frozen subspace in  which no learning occurs and thus smaller (e.g., zero) initial weight norms should be used to achieve best generalization results.
Revised: As pointed out by an anonymous reviewer, it would be interesting to investigate what causes this branching and whether the desired effects are observed at the bottom of the landscape. One could investigate this using the approach of Im et al. (2016) to switch from Adam to AdamW at a given epoch index. Since it is quite possible that the effect of regularization is not that pronounced in the early stages of training, one could think of designing a version of Adam which exploits this by being fast in the early stages and well-regularized in the late stages of training.

Revised: The latter might be achieved with a custom schedule of the weight decay factor.

Revised: In this paper, we argue that the popular interpretation that weight decay =

Revised: L 2 regularization is not precise.

Revised: Instead, the difference between the two leads to the following important consequences.

Revised: Two algorithms as different as SGD and Adam will exhibit different effective rates of weight decay even if the same regularization coefﬁcient is used to include L 2 regularization in the objective function.

Revised: Moreover, when decoupled weight decay is applied, two algorithms as different as SGDW and

Revised: AdamW will optimize two effectively different objective functions even if the same weight decay factor is used.

Revised: Our ﬁndings suggest that the original Adam algorithm with L 2 regularization affects effective rates of weight decay in a way that precludes effective regularization and that effective regularization is achievable by decoupling the weight decay.

Revised: Advani & Saxe (2017) analytically showed that in the limited data regime of deep networks the presence of eigenvalues that are zero forms a frozen subspace in which no learning occurs and thus smaller (e.g., zero) initial weight norms should be used to achieve best generalization results.

Original: Our future work shall consider adapting initial weight norms or weight norm constraints (Salimans &
Revised: Our future work shall consider adapting initial weight norms or weight norm constraints (Salimans &

Original: Kingma, 2016) at each warm restart.
Revised: Kingma, 2016) at each warm restart.

Original: Kawaguchi et al.
Revised: Kawaguchi et al.

Original: (2017) proposed a family of regularization techniques which are speciﬁc to the current batch and its size.
Revised: (2017) proposed a family of regularization techniques which are speciﬁc to the current batch and its size.

Original: Similarly to L 2 regularization and weight decay, the latter techniques might be attempted to be transformed to act directly on weights.
Revised: Similarly to L 2 regularization and weight decay, the latter techniques might be attempted to be transformed to act directly on weights.

Original:  S UPPLEMENTARY M ATERIAL 1. A N EXAMPLE SETTING OF THE SCHEDULE MULTIPLIER
Revised:  S UPPLEMENTARY M ATERIAL 1. A N EXAMPLE SETTING OF THE SCHEDULE MULTIPLIER

Original: An example schedule of the schedule multiplier η t is given in SuppFigure 4 for T i =

Revised: An example schedule of the schedule multiplier η t is given in SuppFigure 1 for T i =

Original: = 100 and
Revised: = 100 and

Original: T mult = 2 .
Revised: T mult = 2 .

Original: After the initial 100 epochs the learning rate will reach 0 because η t =100 = 0 .
Revised: After the initial 100 epochs the learning rate will reach 0 because η t =100 = 0 .

Original: Then, since T cur = T i =0 ,  T cur will be set to 0 thus setting the multiplier η t to 1 due to Eq.
Revised: Then, since T cur = T i =0 , we restart by resetting T cur = 0 , causing the multiplier η t to be reset todue to Eq.

Original: The latter will  decrease again from 1 to 0 but now during 200 epochs because T i =1 = T i =

Revised: This multiplier will then decrease again from 1 to 0, but now over the course ofepochs because T i =1 = T i =

Original: T mult = 200 .
Revised: T mult = 200 .

Original: Solutions obtained  when η t [-= 
 (e g.,-] at epoch indexes 100, 300, 700 and 1500 as shown in SuppFigure 4) are recommended by the optimizer as the solutions (the more recent solutions are prioritized) of the problem at hand. [Figure]
Revised: Solutions obtained right before the restarts, when η t [-= 
 (e g.,-]=(e.g., at epoch indexes 100, 300, 700 and 1500 as shown in SuppFigure 1) are recommended by the optimizer as the solutions, with more recent solutions prioritized.

Original: r n i

Original: [Figure]
Revised: [Figure]

Original: r a t e [Figure] u l t i p li e r η [Figure]
Revised: is scheduled to epoch T is doubled