Original: D OMAIN - INDEPENDENT
Revised: D OMAIN - INDEPENDENT

Original: D OMINANCE OF A DAPTIVE
Revised: D OMINANCE OF A DAPTIVE

Original: M ETHODS
Revised: M ETHODS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: From a simpliﬁed analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned.
Revised: From a simpliﬁed analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned.

Original: We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search.
Revised: We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search.

Original: In light of this observation, we demonstrate that, against conventional wisdom,
Revised: In light of this observation, we demonstrate that, against conventional wisdom,

Original: Adam can also outperform SGD on vision tasks, as long as the coupling between its learning rate and adaptability is taken into account.
Revised: Adam can also outperform SGD on vision tasks, as long as the coupling between its learning rate and adaptability is taken into account.

Original: In practice, AvaGrad matches the best results, as measured by generalization accuracy, delivered by any existing optimizer (SGD or adaptive) across image classiﬁcation (CIFAR, ImageNet) and character-level language modelling (Penn Treebank) tasks.
Revised: In practice, AvaGrad matches the best results, as measured by generalization accuracy, delivered by any existing optimizer (SGD or adaptive) across image classiﬁcation (CIFAR, ImageNet) and character-level language modelling (Penn Treebank) tasks.

Original: This later observation, alongside of AvaGrad’s decoupling of hyperparameters, could make it the preferred optimizer for deep learning, replacing both SGD and Adam.
Revised: This later observation, alongside of AvaGrad’s decoupling of hyperparameters, could make it the preferred optimizer for deep learning, replacing both SGD and Adam.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Deep network architectures are becoming increasingly complex, often containing parameters that can be grouped according to multiple functionalities, such as gating, attention, convolution, and generation.
Revised: Deep network architectures are becoming increasingly complex, often containing parameters that can be grouped according to multiple functionalities, such as gating, attention, convolution, and generation.

Original: Such parameter groups should arguably be treated differently during training, as their gradient statistics might be highly distinct.
Revised: Such parameter groups should arguably be treated differently during training, as their gradient statistics might be highly distinct.

Original: Adaptive gradient methods designate parameter-wise learning rates based on gradient histories, treating such parameters groups differently and, in principle, promise to be better suited for training complex neural network architectures.
Revised: Adaptive gradient methods designate parameter-wise learning rates based on gradient histories, treating such parameters groups differently and, in principle, promise to be better suited for training complex neural network architectures.

Original: Nonetheless, advances in neural architectures have not been matched by progress in adaptive gradient descent algorithms.
Revised: Nonetheless, advances in neural architectures have not been matched by progress in adaptive gradient descent algorithms.

Original: Vanilla SGD is still prevalent, in spite of the development of seemingly more sophisticated adaptive alternatives, such as RMSProp (Dauphin et al., 2015) and Adam (Kingma & Ba, 2015).
Revised: SGD is still prevalent, in spite of the development of seemingly more sophisticated adaptive alternatives, such as RMSProp (Dauphin et al., 2015) and Adam (Kingma & Ba, 2015).

Original: Such adaptive methods have been observed to yield poor generalization compared to SGD in classiﬁcation tasks (Wilson et al., 2017), and hence have been mostly adopted for training complex models (Vaswani et al., 2017; Arjovsky et al., 2017).
Revised: Such adaptive methods have been observed to yield poor generalization compared to SGD in classiﬁcation tasks (Wilson et al., 2017), and hence have been mostly adopted for training complex models (Vaswani et al., 2017; Arjovsky et al., 2017).

Original: For relatively simple architectures, such as ResNets (He et al., 2016a) and DenseNets (Huang et al., 2017), SGD is still the dominant choice.
Revised: For relatively simple architectures, such as ResNets (He et al., 2016a) and DenseNets (Huang et al., 2017), SGD is still the dominant choice.

Original: At a theoretical level, concerns have also emerged about the current crop of adaptive methods.
Revised: At a theoretical level, concerns have also emerged about the current crop of adaptive methods.

Original: Recently, Reddi et al.
Revised: Recently, Reddi et al.

Original: (2018) has identiﬁed cases, even in the stochastic convex setting, where Adam (Kingma & Ba, 2015) fails to converge.
Revised: (2018) has identiﬁed cases, even in the stochastic convex setting, where Adam (Kingma & Ba, 2015) fails to converge.

Original: Modiﬁcations to Adam that provide convergence guarantees have been formulated, but have shortcomings.
Revised: Modiﬁcations to Adam that provide convergence guarantees have been formulated, but have shortcomings.

Original: AMSGrad (Reddi et al., 2018) requires non-increasing learning rates, while AdamNC (Reddi et al., 2018) and AdaBound (Luo et al., 2019) require that adaptivity be gradually eliminated during training.
Revised: AMSGrad (Reddi et al., 2018) requires non-increasing learning rates, while AdamNC (Reddi et al., 2018) and AdaBound (Luo et al., 2019) require that adaptivity be gradually eliminated during training.

Original: Moreover, while most of the recently proposed variants do not provide formal guarantees for non-convex problems, the ones that do (Zaheer et al., 2018; Chen et al., 2019) fail to deliver a convergence rate which matches that of SGD.
Revised: Moreover, while most of the recently proposed variants do not provide formal guarantees for non-convex problems, the few current convergence rate analyses in the literature (Zaheer et al., 2018; Chen et al., 2019) do not match SGD’s.

Original: Section 3 fully details the convergence rates of the most popular Adam variants, along with their shortcomings.
Revised: Section 3 fully details the convergence rates of the most popular Adam variants, along with their shortcomings.

Original: Our contribution is marked improvements to adaptive optimizers, from both theoretical and practical perspectives.
Revised: Our contribution is marked improvements to adaptive optimizers, from both theoretical and practical perspectives.

Original: At the theoretical level, we focus on convergence guarantees, deriving new algorithms:
Revised: At the theoretical level, we focus on convergence guarantees, deriving new algorithms:

Original: • Delayed Adam.
Revised: • Delayed Adam.

Original: Inspired by Zaheer et al.
Revised: Inspired by Zaheer et al.

Original: (2018)’s analysis of Adam, Section 4 proposes a simple modiﬁcation for adaptive gradient methods which yields a provable convergence rate of O (1 / √ T ) in the stochastic non-convex setting – the same as SGD.
Revised: (2018)’s analysis of Adam, Section 4 proposes a simple modiﬁcation for adaptive gradient methods which yields a provable convergence rate of O (1 / √ T ) in the stochastic non-convex setting – the same as SGD.

Original: Our modiﬁcation can be implemented by swapping two lines of code and preserves adaptivity without incurring extra memory costs.
Revised: Our modiﬁcation can be implemented by swapping two lines of code and preserves adaptivity without incurring extra memory costs.

Original: To illustrate these results, we present a non-convex problem where Adam fails to converge to
Revised: To illustrate these results, we present a non-convex problem where Adam fails to converge to

Original: a stationary point, while Delayed Adam – Adam with our proposed modiﬁcation – provably converges with a rate of O (1 / √ T ) .
Revised: a stationary point, while Delayed Adam – Adam with our proposed modiﬁcation – provably converges with a rate of O (1 / √ T ) .

Original: • AvaGrad.
Revised: • AvaGrad.

Original: Inspecting the convergence rate of Delayed Adam, we show that it would improve with an adaptive global learning rate, which self-regulates based on global statistics of the gradient second moments.
Revised: Inspecting the convergence rate of Delayed Adam, we show that it would improve with an adaptive global learning rate, which self-regulates based on global statistics of the gradient second moments.

Original: Following this insight, Section 5 proposes a new adaptive method, AvaGrad, whose hyperparameters decouple learning rate and adaptability.
Revised: Following this insight, Section 5 proposes a new adaptive method, AvaGrad, whose hyperparameters decouple learning rate and adaptability.

Original: Through extensive experiments, Section 6 demonstrates that AvaGrad is not merely a theoretical exercise.
Revised: Through extensive experiments, Section 6 demonstrates that AvaGrad is not merely a theoretical exercise.

Original: AvaGrad performs as well as both SGD and Adam in their respectively favored usage scenarios.
Revised: AvaGrad performs as well as both SGD and Adam in their respectively favored usage scenarios.

Original: Along this experimental journey, we happen to disprove some conventional wisdom, ﬁnding adaptive optimizers, including Adam, to be superior to SGD for training CNNs.
Revised: Along this experimental journey, we happen to disprove some conventional wisdom, ﬁnding adaptive optimizers, including Adam, to be superior to SGD for training CNNs.

Original: The caveat is that, excepting AvaGrad, these methods are sensitive to hyperparameter values.
Revised: The caveat is that, excepting AvaGrad, these methods are sensitive to hyperparameter values.

Original: AvaGrad is a uniquely attractive adaptive optimizer, yielding near best results over a wide range of hyperparameters.
Revised: AvaGrad is a uniquely attractive adaptive optimizer, yielding near best results over a wide range of hyperparameters.

Original:  P RELIMINARIES 2. N OTATION
Revised:  P RELIMINARIES 2. N OTATION

Original: For vectors a = [ a 1 , a 2 , . .
Revised: For vectors a = [ a 1 , a 2 , . .

Original: , b = [ b 1 , b 2 , . . . ]
Revised: , b = [ b 1 , b 2 , . . . ]

Original: ∈ R d , we use the following notation: 1 a for elementwise division ( 1 a = [ 1 a 1 , 1 a 2 , . . . ] ), √ a for element-wise square root ( √ a = [ √ a 1 , √ a 2 , . . . ] ), a + b for element-wise addition ( a + b = [ a 1 + b 1 , a 2 + b 2 , . . . ] ), a (cid:12) b for element-wise multiplication ( a (cid:12) b =
Revised: ∈ R d , we use the following notation: 1 a for elementwise division ( 1 a = [ 1 a 1 , 1 a 2 , . . . ] ), √ a for element-wise square root ( √ a = [ √ a 1 , √ a 2 , . . . ] ), a + b for element-wise addition ( a + b = [ a 1 + b 1 , a 2 + b 2 , . . . ] ), a (cid:12) b for element-wise multiplication ( a (cid:12) b =

Original: [ a 1 b 1 , a 2 b 2 , . . . ] ).
Revised: [ a 1 b 1 , a 2 b 2 , . . . ] ).

Original: Moreover, (cid:107) a (cid:107) is used to denote the (cid:96) 2 -norm: other norms will be speciﬁed whenever used ( e.g., (cid:107) a (cid:107) ∞ ).
Revised: Moreover, (cid:107) a (cid:107) is used to denote the (cid:96) 2 -norm: other norms will be speciﬁed whenever used ( e.g., (cid:107) a (cid:107) ∞ ).

Original: For subscripts and vector indexing, we adopt the following convention: the subscript t is used to denote an object related to the t -th iteration of an algorithm ( e.g., w t ∈ R d denotes the iterate at time step t ); the subscript i is used for indexing: w i ∈ R denotes the i -th coordinate of w ∈ R d .
Revised: For subscripts and vector indexing, we adopt the following convention: the subscript t is used to denote an object related to the t -th iteration of an algorithm ( e.g., w t ∈ R d denotes the iterate at time step t ); the subscript i is used for indexing: w i ∈ R denotes the i -th coordinate of w ∈ R d .

Original: When used together, t precedes i : w t,
Revised: When used together, t precedes i : w t,

Original: i ∈ R denotes the i -th coordinate of w t ∈ R d .
Revised: i ∈ R denotes the i -th coordinate of w t ∈ R d .

Original: 2. S TOCHASTIC N ON -C ONVEX O PTIMIZATION
Revised: 2. S TOCHASTIC N ON -C ONVEX O PTIMIZATION

Original: In the stochastic non-convex setting, we are concerned with the optimization problem:
Revised: In the stochastic non-convex setting, we are concerned with the optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where D is a probability distribution over a set S of “data points”.
Revised: where D is a probability distribution over a set S of “data points”.

Original: We also assume that f is M -smooth in w , as is typically done in non-convex optimization:
Revised: We also assume that f is M -smooth in w , as is typically done in non-convex optimization:

Original: [Equation]
Revised: [Equation]

Original: Methods for stochastic non-convex optimization are evaluated in terms of number of iterations or gradient evaluations required to achieve small loss gradients.
Revised: Methods for stochastic non-convex optimization are evaluated in terms of number of iterations or gradient evaluations required to achieve small loss gradients.

Original: This differs from the stochastic convex setting where convergence is measured w.r.t.
Revised: This differs from the stochastic convex setting where convergence is measured w.r.t.

Original: suboptimality f ( w ) − min w ∈ R d f ( w ) .
Revised: suboptimality f ( w ) − min w ∈ R d f ( w ) .

Original: We assume that the algorithm takes a sequence of data points S
Revised: We assume that the algorithm takes a sequence of data points S

Original: =
Revised: =

Original: ( s 1 , . . .
Revised: ( s 1 , . . .

Original: , s T ) from which it deterministically computes a sequence of parameter settings w 1 , . . .
Revised: , s T ) from which it deterministically computes a sequence of parameter settings w 1 , . . .

Original: , w T together with a distribution P over { 1 , . . .
Revised: , w T together with a distribution P over { 1 , . . .

Original: , T } .
Revised: , T } .

Original: We say an algorithm has a convergence rate of O ( g ( T ))
Revised: We say an algorithm has a convergence rate of O ( g ( T ))

Original: if E
Revised: if E

Original: [Equation]
Revised: [Equation]

Original: O ( g ( T )) where, as deﬁned above, f ( w ) =
Revised: O ( g ( T )) where, as deﬁned above, f ( w ) =

Original: E s ∼D
Revised: E s ∼D

Original: [ f s ( w )] .
Revised: [ f s ( w )] .

Original: We also assume that the functions f s have bounded gradients: there exists some G ∞ such that (cid:107)∇ f s ( w ) (cid:107) ∞ ≤ G ∞ for all s ∈ S and w ∈ R d .
Revised: We also assume that the functions f s have bounded gradients: there exists some G ∞ such that (cid:107)∇ f s ( w ) (cid:107) ∞ ≤ G ∞ for all s ∈ S and w ∈ R d .

Original: Throughout the paper, we also let G 2 denote an upper bound on (cid:107)∇ f s ( w ) (cid:107) .
Revised: Throughout the paper, we also let G 2 denote an upper bound on (cid:107)∇ f s ( w ) (cid:107) .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Here we present a brief overview of optimization methods commonly used for training neural networks, along with their convergence rate guarantees for stochastic smooth non-convex problems.
Revised: Here we present a brief overview of optimization methods commonly used for training neural networks, along with their convergence rate guarantees for stochastic smooth non-convex problems.

Original: We consider methods which, at each iteration t , receive or compute a gradient estimate:
Revised: We consider methods which, at each iteration t , receive or compute a gradient estimate:

Original: [Equation]
Revised: [Equation]

Original: and perform an update of the form:
Revised: and perform an update of the form:

Original: [Equation]
Revised: [Equation]

Original: where α t ∈ R is the global learning rate , η t ∈ R
Revised: where α t ∈ R is the global learning rate , η t ∈ R

Original: d are the parameter-wise learning rates , and m t ∈ R
Revised: d are the parameter-wise learning rates , and m t ∈ R

Original: d is the update direction, typically deﬁned as:
Revised: d is the update direction, typically deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: Non-momentum methods such as SGD, AdaGrad, and RMSProp (Dauphin et al., 2015; Duchi et al., 2011) have m t =
Revised: Non-momentum methods such as SGD, AdaGrad, and RMSProp (Dauphin et al., 2015; Duchi et al., 2011) have m t =

Original: g t ( i.e., β 1 ,t = 0 ), while momentum SGD and Adam (Kingma & Ba, 2015) have β 1 ,t ∈ (0 , 1) .
Revised: g t ( i.e., β 1 ,t = 0 ), while momentum SGD and Adam (Kingma & Ba, 2015) have β 1 ,t ∈ (0 , 1) .

Original: Note that while α t can always be absorbed into η t , representing the update in this form will be convenient throughout the paper.
Revised: Note that while α t can always be absorbed into η t , representing the update in this form will be convenient throughout the paper.

Original: SGD uses the same learning rate for all parameters, i.e., η t = (cid:126) 1 .
Revised: SGD uses the same learning rate for all parameters, i.e., η t = (cid:126) 1 .

Original: Although SGD is simple and offers no adaptation, it has a convergence rate of O (1 / √ T ) with either constant, increasing, or decreasing learning rates (Ghadimi & Lan, 2013), and is widely used when training deep networks, especially
Revised: Although SGD is simple and offers no adaptation, it has a convergence rate of O (1 / √ T ) with either constant, increasing, or decreasing learning rates (Ghadimi & Lan, 2013), and is widely used when training deep networks, especially

Original: CNNs (He et al., 2016a; Huang et al., 2017).
Revised: CNNs (He et al., 2016a; Huang et al., 2017).

Original: At the heart of its convergence proof is the fact that E s t [ α t · η t (cid:12) g t ] =
Revised: At the heart of its convergence proof is the fact that E s t [ α t · η t (cid:12) g t ] =

Original: α t · ∇ f ( w t ) .
Revised: α t · ∇ f ( w t ) .

Original: Popular adaptive methods such as RMSProp (Dauphin et al., 2015), AdaGrad (Duchi et al., 2011), and Adam (Kingma & Ba, 2015) have η t = 1 √ v t + (cid:15) , where v t ∈ R d is given by:
Revised: Popular adaptive methods such as RMSProp (Dauphin et al., 2015), AdaGrad (Duchi et al., 2011), and Adam (Kingma & Ba, 2015) have η t = 1 √ v t + (cid:15) , where v t ∈ R d is given by:

Original: [Equation]
Revised: [Equation]

Original: and v 0 = 0 .
Revised: and v 0 = 0 .

Original: As v t is an estimate of the second moments of the gradients, the optimizer designates smaller learning rates for parameters with larger uncertainty in their stochastic gradients.
Revised: As v t is an estimate of the second moments of the gradients, the optimizer designates smaller learning rates for parameters with larger uncertainty in their stochastic gradients.

Original: However, in this setting η t and s t are no longer independent, hence E s t [ α t · η t (cid:12) g t ] (cid:54) =
Revised: However, in this setting η t and s t are no longer independent, hence E s t [ α t · η t (cid:12) g t ] (cid:54) =

Original: α t · E s t [ η t ] (cid:12) ∇ f ( w t ) .
Revised: α t · E s t [ η t ] (cid:12) ∇ f ( w t ) .

Original: This “bias” can cause RMSProp and Adam to present convergence issues, even in the stochastic convex setting (Reddi et al., 2018).
Revised: This “bias” can cause RMSProp and Adam to present convergence issues, even in the stochastic convex setting (Reddi et al., 2018).

Original: Recently, Zaheer et al.
Revised: Recently, Zaheer et al.

Original: (2018) showed that RMSProp and Adam have a convergence rate of O ( σ 2 + 1 /T ) , where σ 2 = sup w ∈ R d E s ∼D (cid:104) (cid:107)∇ f s ( w ) − ∇ f ( w ) (cid:107) 2 (cid:105) , hence convergence is generally not guaranteed.
Revised: (2018) showed that, with a constant learning rate, RMSProp and Adam have a convergence rate of O ( σ 2 + 1 /T ) , where σ 2 = sup w ∈ R d E s ∼D (cid:104) (cid:107)∇ f s ( w ) − ∇ f ( w ) (cid:107) 2 (cid:105) , hence their result does not generally guarantee convergence.

Original: Chen et al.
Revised: Chen et al.

Original: (2019) showed that AdaGrad and AMSGrad enjoy a convergence rate of O (log T/ √ T ) .
Revised: (2019) showed that AdaGrad and AMSGrad enjoy a convergence rate of O (log T/ √ T ) when a decaying learning rate is used.

Original: Note that both methods constrain η t in some form, the former with β 2 ,t = 1 − 1 /t
Revised: Note that both methods constrain η t in some form, the former with β 2 ,t = 1 − 1 /t

Original: (adaptability diminishes with t ), and the latter explicitly enforces v t ≥ v j for all j < t ( η t is pointwise non-increasing).
Revised: (adaptability diminishes with t ), and the latter explicitly enforces v t ≥ v j for all j < t ( η t is point-wise non-increasing).

Original: In both cases, the method is less adaptive than Adam, and yet fails to match the convergence rate of SGD.
Revised: In both cases, the method is less adaptive than Adam, and yet analyses so far have not delivered a convergence rate that matches SGD’s.

Original:  SGD- LIKE C ONVERGENCE WITHOUT C ONSTRAINED R ATES
Revised:  SGD- LIKE C ONVERGENCE WITHOUT C ONSTRAINED R ATES

Original: Algorithm Input: w 1 ∈ R d , α t , (cid:15) > 0 , β 1 ,t , β 2 ,t ∈ [0 , 1) 1: Set m 

Revised: Input: w 1 ∈ R d , α t , (cid:15) > 0 , β 1 ,t , β 2 ,t ∈ [0 , 1) 1: Set m

Original: = 0 , v 0 =2: for t = 1 to T do 3:
Revised: = 0 , v 0 =2: for t = 1 to T do 3:

Original: Draw s t ∼ D 4:
Revised: Draw s t ∼ D 4:

Original: Compute g t = ∇ f s t ( w t ) 5: m t = β 1 ,t m t − 1 + (1 − β 1 ,t) g t 6: η t = 1 √ v t − 1 + (cid:15) 7: w
Revised: Compute g t = ∇ f s t ( w t ) 5: m t = β 1 ,t m t − 1 + (1 − β 1 ,t

Original: [Equation]
Revised: [Equation]

Original: 8: v
Revised: [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: = β 2 ,t v t − 1 + (1 − β 2 ,t ) g 2 t 9: end for
Revised: 9: end for

Original: We ﬁrst take a step back to note the following: to show that Adam might not converge in the stochastic convex setting, Reddi et al.
Revised: We ﬁrst take a step back to note the following: to show that Adam might not converge in the stochastic convex setting, Reddi et al.

Original: (2018) provide a stochastic linear problem where Adam fails to converge w.r.t.
Revised: (2018) provide a stochastic linear problem where Adam fails to converge w.r.t.

Original: suboptimality.
Revised: suboptimality.

Original: Since non-convex optimization is evaluated w.r.t.
Revised: Since non-convex optimization is evaluated w.r.t.

Original: norm of the gradients, a different instance is required to characterize Adam’s behavior in this setting.
Revised: norm of the gradients, a different instance is required to characterize Adam’s behavior in this setting.

Original: The following result shows that even for a quadratic problem, Adam indeed does not converge to a stationary point:
Revised: The following result shows that even for a quadratic problem, Adam indeed does not converge to a stationary point:

Original: Theorem 1.
Revised: Theorem 1.

Original: For any (cid:15) ≥ 0 and constant β 2 ,t = β 2 ∈ [0 , 1) , there is a stochastic convex optimization problem for which Adam does not converge to a stationary point.
Revised: For any (cid:15) ≥ 0 and constant β 2 ,t = β 2 ∈ [0 , 1) , there is a stochastic convex optimization problem for which Adam does not converge to a stationary point.

Original: Proof.
Revised: Proof.

Original: The full proof is given in Appendix A.
Revised: The full proof is given in Appendix A.

Original: The argument follows closely from Reddi et al.
Revised: The argument follows closely from Reddi et al.

Original: (2018), where we explicitly present a stochastic optimization problem:
Revised: (2018), where we explicitly present a stochastic optimization problem:

Original: [Equation]
Revised: [Equation]

Original: We show that, for large enough C (as a function of δ, (cid:15), β 2 ), Adam will move towards w = 1 where ∇ f (1)
Revised: We show that, for large enough C (as a function of δ, (cid:15), β 2 ), Adam will move towards w = 1 where ∇ f (1)

Original: = δ , and that the constraint w ∈ [0 , 1] does not make w = 1 a stationary point.
Revised: = δ , and that the constraint w ∈ [0 , 1] does not make w = 1 a stationary point.

Original: This result, like the one in Reddi et al.
Revised: This result, like the one in Reddi et al.

Original: (2018), relies on the fact that η t and s t are correlated: upon a draw of the rare sample C w 2 2 , the learning rate η t decreases signiﬁcantly and Adam takes a small step in the correct direction.
Revised: (2018), relies on the fact that η t and s t are correlated: upon a draw of the rare sample C w 2 2 , the learning rate η t decreases signiﬁcantly and Adam takes a small step in the correct direction.

Original: On the other hand, a sequence of common samples increases η t and
Revised: On the other hand, a sequence of common samples increases η t and

Original: Adam moves faster towards w = 1 .
Revised: Adam moves faster towards w = 1 .

Original: Instead of enforcing η t to be point-wise non-increasing in t (Reddi et al., 2018), which forces the optimizer to take small steps even for a long sequence of common samples, we propose to simply have η t be independent of s t .
Revised: Instead of enforcing η t to be point-wise non-increasing in t (Reddi et al., 2018), which forces the optimizer to take small steps even for a long sequence of common samples, we propose to simply have η t be independent of s t .

Original: As an extra motivation for this approach, note that successful proof strategies (Zaheer et al., 2018) to analyzing adaptive methods include the following step:
Revised: As an extra motivation for this approach, note that successful proof strategies (Zaheer et al., 2018) to analyzing adaptive methods include the following step:

Original: [Equation]
Revised: [Equation]

Original: where bounding E s t [( η t − η t − 1 ) (cid:12) g t ] , seen as a form of bias, is a key part of recent convergence analyses.
Revised: where bounding E s t [( η t − η t − 1 ) (cid:12) g t ] , seen as a form of bias, is a key part of recent convergence analyses.

Original: Replacing η t by η t − 1 in the update equation of Adam removes this bias and can be implemented by simply swapping lines of code (updating η after w ),also yielding a simpleand short convergence analysis. Additionally, it does not hinder the adaptability of the method in any way.
Revised: Replacing η t by η t − 1 in the update equation of Adam removes this bias and can be implemented by simply swapping lines of code (updating η after w ), yielding a simple convergence analysis without hindering the adaptability of the method in any way.

Original: Algorithm 1 provides pseudo-code when applying this modiﬁcation to Adam, yielding Delayed Adam.
Revised: Algorithm 1 provides pseudocode when applying this modiﬁcation, highlighted in red, to Adam, yielding Delayed Adam.

Original: The following Theorem shows that this modiﬁcation is enough to guarantee a SGD-like convergence rate of O (1 / √ T ) in the stochastic non-convex setting for general adaptive gradient methods.
Revised: The following Theorem shows that this modiﬁcation is enough to guarantee a SGD-like convergence rate of O (1 / √ T ) in the stochastic non-convex setting for general adaptive gradient methods.

Original: Theorem 2.
Revised: Theorem 2.

Original: Consider any optimization method which updates parameters as follows:
Revised: Consider any optimization method which updates parameters as follows:

Original: [Equation]
Revised: [Equation]

Original: where g t := ∇ f s t ( w t ) , s t ∼ D , and α t , η t are independent of s t .
Revised: where g t := ∇ f s t ( w t ) , s t ∼ D , and α t , η t are independent of s t .

Original: Assume that f ( w 1 ) − f ( w (cid:63) ) ≤ D , f ( w ) =
Revised: Assume that f ( w 1 ) − f ( w (cid:63) ) ≤ D , f ( w ) =

Original: E s ∼D
Revised: E s ∼D

Original: [ f s ( w )] is M -smooth, and (cid:107)∇ f s ( w ) (cid:107) ∞
Revised: [ f s ( w )] is M -smooth, and (cid:107)∇ f s ( w ) (cid:107) ∞

Original: ≤ G ∞ for all s ∈ S , w ∈ R d .
Revised: ≤ G ∞ for all s ∈ S , w ∈ R d .

Original: Moreover, let Z = (cid:80) Tt =
Revised: Moreover, let Z = (cid:80) Tt =

Original: 1 α t min i η t,i .
Revised: 1 α t min i η t,i .

Original: For α t = γ t (cid:113) 2 D TMG 2 ∞ , if p ( Z | s t ) =
Revised: For α t = γ t (cid:113) 2 D TMG 2 ∞ , if p ( Z | s t ) =

Original: p ( Z ) for all s t ∈ S , then:
Revised: p ( Z ) for all s t ∈ S , then:

Original: [Equation]
Revised: [Equation]

Original: where P assigns probabilities p ( t )
Revised: where P assigns probabilities p ( t )

Original: ∝ α t · min i η t,i .
Revised: ∝ α t · min i η t,i .

Original: Proof.
Revised: Proof.

Original: The full proof is given in Appendix B, along with analysis for the case with momentum β 1 ,t ∈ (0 , 1) in Appendix B.1, and in particular β 1 ,t = β 1 / √ t , which yields a similar rate.
Revised: The full proof is given in Appendix B, along with analysis for the case with momentum β 1 ,t ∈ (0 , 1) in Appendix B.1, and in particular β 1 ,t = β 1 / √ t , which yields a similar rate.

Original: The convergence rate depends on (cid:107) η t (cid:107) and min i η t,i , which are random variables for Adam-like algorithms.
Revised: The convergence rate depends on (cid:107) η t (cid:107) and min i η t,i , which are random variables for Adam-like algorithms.

Original: However, if there are constants H and L such that 0 <
Revised: However, if there are constants H and L such that 0 <

Original: L ≤ η t,i ≤ H < ∞ for all i and t , then a rate of O (1 / √ T ) is guaranteed.
Revised: L ≤ η t,i ≤ H < ∞ for all i and t , then a rate of O (1 / √ T ) is guaranteed.

Original: This is the case for Delayed Adam, where 1 / ( G 2 + (cid:15) )
Revised: This is the case for Delayed Adam, where 1 / ( G 2 + (cid:15) )

Original: ≤ η t,i ≤ 1 /(cid:15) for all t and i .
Revised: ≤ η t,i ≤ 1 /(cid:15) for all t and i .

Original: Theorem 2 also requires that α t and η t are independent of s t , which can be assured to hold by applying a “delay” to their respective computations, if necessary ( i.e., replacing η t by η t − 1 , as in Delayed Adam).
Revised: Theorem 2 also requires that α t and η t are independent of s t , which can be assured to hold by applying a “delay” to their respective computations, if necessary ( i.e., replacing η t by η t − 1 , as in Delayed Adam).

Original: Additionally, the assumption that p ( Z | s t ) =
Revised: Additionally, the assumption that p ( Z | s t ) =

Original: p ( Z ) , meaning that a single sample should not affect the distribution of Z = (cid:80) T t
Revised: p ( Z ) , meaning that a single sample should not affect the distribution of Z = (cid:80) T t

Original: =1 α t min i η t,i , is required since P is conditioned on the samples S
Revised: =1 α t min i η t,i , is required since P is conditioned on the samples S

Original: (unlike in standard analysis, where Z = (cid:80) Tt =1 α t and α t is deterministic), and is expected to hold as T → ∞ .
Revised: (unlike in standard analysis, where Z = (cid:80) Tt =1 α t and α t is deterministic), and is expected to hold as T → ∞ .

Original: Practitioners typically use the last iterate w T or perform early-stopping: in this case, whether the assumption holds or not does not affect the behavior of the algorithm.
Revised: Practitioners typically use the last iterate w T or perform early-stopping: in this case, whether the assumption holds or not does not affect the behavior of the algorithm.

Original: Nonetheless, we also show in Appendix B.2 a similar rate that does not require this assumption to hold, which also yields a
Revised: Nonetheless, we also show in Appendix B.2 a similar rate that does not require this assumption to hold, which also yields a

Original: O (1 / √ T ) convergence rate taken that the parameter-wise learning rates are bounded from above and below.
Revised: O (1 / √ T ) convergence rate taken that the parameter-wise learning rates are bounded from above and below.

Original:  A VA G RAD : A N A DAPTIVE M ETHOD WITH A DAPTIVE V ARIANCE Algorithm
Revised:  A VA G RAD : A N A DAPTIVE M ETHOD WITH A DAPTIVE V ARIANCE Algorithm

Original: A

Original: Input: w 1 ∈ R d , α t , (cid:15) > 0 , β 1 ,t , β 2 ,t ∈ [0 , 1) 1: Set m 

Revised: Input: w 1 ∈ R d , α t , (cid:15) > 0 , β 1 ,t , β 2 ,t ∈ [0 , 1) 1: Set m

Original: = 0 , v 0 =2: for t = 1 to T do 3:
Revised: = 0 , v 0 =2: for t = 1 to T do 3:

Original: Draw s t ∼ D 4:
Revised: Draw s t ∼ D 4:

Original: Compute g t = ∇ f s t ( w t ) 5: m t = β 1 ,t m t − 1 + (1 − β 1 ,t ) g t 6: η t = 1 √ v t − 1 + (cid:15) 7: w t +1 =
Revised: Compute g t = ∇ f s t ( w t ) 5: m t = β 1 ,t m t − 1 + (1 − β 1 ,t ) g t 6: η t = 1 √ v t − 1 + (cid:15) 7: w t +1 =

Original: w t − α t · η t (cid:107) η t / √ d (cid:107) 2 (cid:12) m t 8: v t = β 2 ,t v t − 1 + (1 − β 2 ,t )
Revised: w t − α t · η t (cid:107) η t / √ d (cid:107) 2 (cid:12) m t 8: v t = β 2 ,t v t − 1 + (1 − β 2 ,t )

Original: g 2 t 9: end for
Revised: g 2 t 9: end for

Original: Now, we consider the implications of Theoremfor Delayed Adam, where η t = 1 √ v t − 1 + (cid:15) , and hence 1 / ( G 2 + (cid:15) )
Revised: Now, we consider the implications of Theoremfor Delayed Adam, where η t = 1 √ v t − 1 + (cid:15) , and hence 1 / ( G 2 + (cid:15) )

Original: ≤ η t,i ≤ 1 /(cid:15) for all t and i .
Revised: ≤ η t,i ≤ 1 /(cid:15) for all t and i .

Original: For a ﬁxed γ t = γ , chosen a-priori (that is, without knowledge of the realization of { η t } Tt =1 ), we can optimize γ to minimize the worst-case rate using (cid:107) η t (cid:107) 2 ≤ d/(cid:15) 2 and min i η t,i ≥ G 2 + (cid:15) .
Revised: For a ﬁxed γ t = γ , chosen a-priori (that is, without knowledge of the realization of { η t } Tt =1 ), we can optimize γ to minimize the worst-case rate using (cid:107) η t (cid:107) 2 ≤ d/(cid:15) 2 and min i η t,i ≥ G 2 + (cid:15) .

Original: This yields γ ∗ =
Revised: This yields γ ∗ =

Original: O ( (cid:15) ) , and a convergence rate of O (1 /(cid:15) ) , suggesting that, at least in the worst case, (cid:15) should be chosen to be as large as possible, and the learning rate α should scale linearly with (cid:15) (hence, also being large).
Revised: O ( (cid:15) ) , and a convergence rate of O (1 /(cid:15) ) , suggesting that, at least in the worst case, (cid:15) should be chosen to be as large as possible, and the learning rate α should scale linearly with (cid:15) (hence, also being large).

Original: What if we allow γ t to vary in each time step?
Revised: What if we allow γ t to vary in each time step?

Original: For example, choosing γ t = √ d/ (cid:107) η t (cid:107) yields a convergence rate with a linear dependence on (cid:16) 1 T (cid:80)
Revised: For example, choosing γ t = √ d/ (cid:107) η t (cid:107) yields a convergence rate with a linear dependence on (cid:16) [Equation] = [Equation]

Original: Tt =1 min i
Revised: [Equation] t

Original: [Equation]
Revised: [Equation]

Original: ,i
Revised: (cid:17) − 1 ≤

Original: [Equation]
Revised: [Equation]

Original: t (cid:107) (cid:17) − 1 ≤ (cid:16) 1 T √ d (cid:80) Tt =1 min i η t,i max i η t,i (cid:17) − 1 .
Revised: 1 T √ d [Equation] Tt =1 min i η t,i max i η t,i (cid:17) − 1 .

Original: While using 1 / ( G 2 + (cid:15) ) ≤ η t,i ≤ 1 /(cid:15)
Revised: While using 1 / ( G 2 + (cid:15) ) ≤ η t,i ≤ 1 /(cid:15)

Original: we see that in the worst-case this is also O (1 /(cid:15) ) , this dependence differs from the one with ﬁxed γ t = γ in a few aspects.
Revised: we see that in the worst-case this is also O (1 /(cid:15) ) , this dependence differs from the one with ﬁxed γ t = γ in a few aspects.

Original: Most notably, if we consider different scalings of η t = 1 √ v t − 1 + (cid:15) ( e.g., small (cid:15) and varying v t − 1 ), the convergence rate with ﬁxed γ can get arbitrarily worse, while for γ t = √ d/ (cid:107) η t (cid:107) it remains unchanged.
Revised: Most notably, if we consider different scalings of η t = 1 √ v t − 1 + (cid:15) ( e.g., small (cid:15) and varying v t − 1 ), the convergence rate with ﬁxed γ can get arbitrarily worse, while for γ t = √ d/ (cid:107) η t (cid:107) it remains unchanged.

Original: In particular, for the case d = 1 , we have (cid:16) 1 T (cid:80) Tt
Revised: In particular, for the case d = 1 , we have (cid:16) 1 T (cid:80) Tt

Original: =1 min i η t,i (cid:107) η t (cid:107) (cid:17) − 1 = 1 , while for ﬁxed γ we get a dependence on (cid:80)
Revised: =1 min i η t,i (cid:107) η t (cid:107) (cid:17) − 1 = 1 , while for ﬁxed γ we get a dependence on (cid:80)

Original: Tt =1 1+ γ 2 η 2 t (cid:80)
Revised: Tt =1 1+ γ 2 η 2 t (cid:80)

Original: Tt =1 γη t , which again can be large if η t is either large or small.
Revised: Tt =1 γη t , which again can be large if η t is either large or small.

Original: Lastly, multiplying the learning rate by √ d/ (cid:107) η t (cid:107) removes its dependence on (cid:15) in the worst-case setting, making the two hyperparameters more separable.
Revised: Lastly, multiplying the learning rate by √ d/ (cid:107) η t (cid:107) removes its dependence on (cid:15) in the worst-case setting, making the two hyperparameters more separable.

Original: The choice of γ t = √ d/ (cid:107) η t (cid:107) , motivated by the above facts, yields a method which we name AvaGrad – A daptive VA riance Grad ients, presented as pseudo-code in Algorithm 2.
Revised: The choice of γ t = √ d/ (cid:107) η t (cid:107) , motivated by the above facts, yields a method which we name AvaGrad – A daptive VA riance Grad ients, presented as pseudo-code in Algorithm 2 with the proposed scaling highlighted in red.

Original: We call it an “adaptive variance” method since, if we scale up or down the variance of the gradients, and hence also v t and η t , the convergence guarantee in Theorem 2 does not change, while for a ﬁxed learning rate (as is typically done in practice, except for discrete decays during training)
Revised: We call it an “adaptive variance” method since, if we scale up or down the variance of the gradients, and hence also v t and η t , the convergence guarantee in Theorem 2 does not change, while for a ﬁxed learning rate (as is not uncommonly done in practice, except for discrete decays during training (Zagoruyko & Komodakis, 2016; Merity et al., 2018))

Original: it can get arbitrarily bad.
Revised: it can get arbitrarily bad.

Original:  E XPERIMENTS 6. S YNTHETIC D ATA
Revised:  E XPERIMENTS 6. S YNTHETIC D ATA

Original: To illustrate empirically the implications of Theorem 1 and Theorem 2, we set up a synthetic stochastic optimization problem with the same form as the one used in the proof of Theorem 1:
Revised: To illustrate empirically the implications of Theorem 1 and Theorem 2, we set up a synthetic stochastic optimization problem with the same form as the one used in the proof of Theorem 1:

Original: [Equation]
Revised: [Equation]

Original: This function has a stationary point w (cid:63) = 1 − 0 .
Revised: This function has a stationary point w (cid:63) = 1 − 0 .

Original: 002 ≈ 12 , and it satisﬁes Theorem 1 for β 1 = 0 , β 2 = 0 .
Revised: 002 ≈ 12 , and it satisﬁes Theorem 1 for β 1 = 0 , β 2 = 0 .

Original: 99 , (cid:15) =
Revised: 99 , (cid:15) =

Original: 10 − 8 .
Revised: 10 − 8 .

Original: We proceed to perform stochastic optimization with Adam, AMSGrad, and Delayed
Revised: We proceed to perform stochastic optimization with Adam, AMSGrad, and Delayed

Original: Adam, with constant learning rate α t = 10 − 5 .
Revised: Adam, with constant learning rate α t = 10 − 5 .

Original: For simplicity, we let P be uniform over (1 , . . .
Revised: For simplicity, we let P be uniform over (1 , . . .

Original: , T ) , since α t is constant.
Revised: , T ) , since α t is constant.

Original: Figure 1 shows the progress of 1 t (cid:80) tt (cid:48)
Revised: Figure 1 shows the progress of 1 t (cid:80) tt (cid:48)

Original: =1 w t (cid:48) and 1 t (cid:80) tt (cid:48)
Revised: =1 w t (cid:48) and 1 t (cid:80) tt (cid:48)

Original: =
Revised: =

Original: 1 (cid:107)∇ f ( w t (cid:48) )
Revised: 1 (cid:107)∇ f ( w t (cid:48) )

Original: (cid:107) 2 for each iteration t : as expected, Adam fails to converge to the stationary point w (cid:63) , while both AMSGrad and Delayed
Revised: (cid:107) 2 for each iteration t : as expected, Adam fails to converge to the stationary point w (cid:63) , while both AMSGrad and Delayed

Original: Adam converge.
Revised: Adam converge.

Original: Note that Delayed Adam converges signiﬁcantly faster, likely because it has no constraint on the learning rates.
Revised: Note that Delayed Adam converges signiﬁcantly faster, likely because it has no constraint on the learning rates.

Original: 5 t 1e 0.
Revised: 5 t 1e 0.

Original: 1 t t t 0 =w tw
Revised: 1 t t t 0 =w tw

Original: Adam AMSGrad
Revised: Adam AMSGrad

Original: Delayed Adam     5 t 1e 0.
Revised: Delayed Adam     5 t 1e 0.

Original: 1 t t t 0 = 1 || f ( w t 0 )
Revised: 1 t t t 0 = 1 || f ( w t 0 )

Original: ||
Revised: ||

Original: Adam AMSGrad
Revised: Adam AMSGrad

Original: Delayed Adam
Revised: [Figure]

Original: 6. I MAGE C LASSIFICATION WITH CNN S
Revised: 6. I MAGE C LASSIFICATION WITH CNN S

Revised: Our theory suggests that, in the worst case, (cid:15) should be chosen as large as possible, at which point the learning rate α should scale linearly with it.

Original: As a ﬁrst experiment, we train a Wide ResNet 28-4 (Zagoruyko & Komodakis, 2016) on the CIFAR10 dataset (Krizhevsky, 2009).
Revised: As a ﬁrst experiment to assess this hypothesis, we analyze the interaction between α and (cid:15) when training a Wide ResNet 28-4 (Zagoruyko & Komodakis, 2016) on the CIFAR-10 dataset (Krizhevsky, 2009).

Original: Our training follows Zagoruyko & Komodakis (2016): images are channel-wise normalized, randomly cropped, and horizontally ﬂipped during training.
Revised: Our training follows Zagoruyko & Komodakis (2016): images are channel-wise normalized, randomly cropped, and horizontally ﬂipped during training.

Original: The learning rate is decayed by a factor of 5 at epochs 60, 120 and 160, and the model is trained for a total ofepochs with a weight decay of 0 . 0005 .
Revised: The learning rate is decayed by a factor of 5 at epochs 60, 120 and 160, and the model is trained for a total of 200 epochs with a weight decay of 0 . 0005 .

Original: Appendix C describes additional experimental details.
Revised: Appendix C describes additional experimental details.

Original: We  evaluate the performance of SGD and different adaptive gradient methods: Adam, AMSGrad, 
Revised: We use a validation set of 5,000 images to evaluate the performance of SGD and different adaptive gradient methods: Adam, AMSGrad, AdaBound (Luo et al., 2019),

Original: AdaBound, and our proposed algorithm, AvaGrad.
Revised: AdaShift (Zhou et al., 2019), and our proposed algorithm, AvaGrad.

Original: Additionally, we also assess whether performing weight decay as proposed in Loshchilov & Hutter (2019) instead of standard L 2 regularization positively impacts the performance of adaptive methods: we do this by also evaluating AdamW and AvaGradW.
Revised: Additionally, we also assess whether performing weight decay as proposed in Loshchilov & Hutter (2019) instead of standard L 2 regularization positively impacts the performance of adaptive methods: we do this by also evaluating AdamW and AvaGradW.

Original: We run each adaptive method with different powers of 10 forthe (cid:15) hyperparameter, from its default value (cid:15) = 10 − 8 up to (cid:15) = 100 , which is large enough such that adaptability should be almost completely removed from the algorithm.
Revised: We run each adaptive method with different powers of 10 for (cid:15) , from its default value (cid:15) = 10 − 8 up to (cid:15) = 100 , which is large enough such that adaptability should be almost completely removed from the algorithm.

Original: We also vary the learning rate α of each method with different powers of 10 , multiplied by 1 and 5 ( e.g., 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 , . . . ).
Revised: We also vary the learning rate α of each method with different powers of 10 , multiplied by 1 and 5 ( e.g., 0 . 05 , 0 . 1 , 0 . 5 , 1 . 0 , . . . ).

Original: Figure 2 show the results for Adam and AvaGrad.
Revised: Figure 2 shows the results for Adam and AvaGrad.

Original: Our main ﬁndings are twofold:
Revised: Our main ﬁndings are twofold:

Original: • The optimal (cid:15) for every adaptive method is considerably larger than the values typically used in practice, ranging from 0 .
Revised: • The optimal (cid:15) for every adaptive method is considerably larger than the values typically used in practice, ranging from 0 .

Original: 1 (Adam, AMSGrad, AvaGradW) to 10 . 

Revised: 1 (Adam, AMSGrad, AvaGradW) to 10 .

Original: (AvaGrad, AdamW).
Revised: (AvaGrad, AdamW).

Original: For Adam and AMSGrad, the optimal learning rate is α = (cid:15) = 0 .
Revised: For Adam and AMSGrad, the optimal learning rate is α = (cid:15) = 0 .

Original: 1 , a value 100 times larger than the default.
Revised: 1 , a value 100 times larger than the default.

Original: • All adaptive methods, except for AdaBound, outperform SGD in terms of validation performance.
Revised: • All adaptive methods, except for AdaBound, outperform SGD in terms of validation performance.

Original: Note that for SGD the optimal learning rate is α = 0 . 1 , matching the value used in work such as He et al.
Revised: Note that for SGD the optimal learning rate is α = 0 . 1 , matching the value used in work such as He et al.

Original: (2016a)
Revised: (2016a)

Original: ; Zagoruyko & Komodakis (2016); Xie et al.
Revised: ; Zagoruyko & Komodakis (2016); Xie et al.

Original: (2017), which presented state-of-the-art results at time of publication.
Revised: (2017), which presented state-of-the-art results at time of publication.

Original: However, the fact that adaptive methods outperform SGD in this setting is not conclusive, since they are executed with more hyperparameter settings (varying (cid:15) as well as α ).
Revised: However, the fact that adaptive methods outperform SGD in this setting is not conclusive, since they are executed with more hyperparameter settings (varying (cid:15) as well as α ).

Original: Moreover, the main motivation for adaptive methods is to be less sensitive to hyperparameter values; performing an extensive grid search defeats their purpose.
Revised: Moreover, the main motivation for adaptive methods is to be less sensitive to hyperparameter values; performing an extensive grid search defeats their purpose.

Original: Aiming for a fair comparison, we also train a Wide ResNet 28-10 on both CIFAR-10 and CIFAR-100, evaluating the  performance of each  method with its optimal values for α and (cid:15) found in the previous experiment. 
Revised: Aiming for a fair comparison between SGD and adaptive methods, we also train a Wide ResNet 28-10 on both CIFAR-10 and CIFAR-100, evaluating the test performance of each adaptive method with its optimal values for α and (cid:15) found in the previous experiment. For SGD, we conﬁrmed that [Figure] [Table]

Revised: the learning rate α = 0 .

Revised: 1 still yielded the best validation performance with the new architecture, hence the fact that we transfer hyperparameters from the Wide ResNet 28-4 runs does not unfairly advantage adaptive methods in the comparison with SGD.

Original: With a larger network and a different task (CIFAR-100), this experiment should also capture how hyperparameters of adaptive methods transfer between tasks and models.
Revised: With a larger network and a different task (CIFAR-100), this experiment should also capture how hyperparameters of adaptive methods transfer between tasks and models.

Original: On CIFAR-10, SGD achieves 3 .
Revised: On CIFAR-10, SGD achieves 3 .

Original: 86% test error (reported as 4% in Zagoruyko & Komodakis (2016)) and is outperformed by both Adam ( 3 . 64% ) and AvaGrad ( 3 . 80% ).
Revised: 86% test error (reported as 4% in Zagoruyko & Komodakis (2016)) and is outperformed by both Adam ( 3 . 64% ) and AvaGrad ( 3 . 80% ).

Original:  CIFAR-100, SGD ( 19 . 05% )[Figure] [Table] is outperformed by Adam ( 18 . 96% ), AMSGrad ( 18 . 97% ),AvaGrad ( 18 .
Revised: On CIFAR-100, SGD ( 19 . 05% ) is outperformed by Adam ( 18 . 96% ), AMSGrad ( 18 . 97% ),

Original: [Table]
Revised: AdaShift ( 18 . 88% ), AvaGrad ( 18 .

Original:  and AvaGradW ( 19 . 04% ).
Revised: 76% ), and AvaGradW ( 19 . 04% ).

Original: We believe these results are surprising, as they suggest that adaptive methods might yield state-of-the-art performance when training CNNs as long as their adaptability is correctly controlled with (cid:15) .
Revised: We believe these results are surprising, as they show that adaptive methods can yield state-of-the-art performance when training CNNs as long as their adaptability is correctly controlled with (cid:15) .

Original: As a ﬁnal evaluation of the role of adaptability when training convolutional networks, we repeat the previous experiment on the ImageNet dataset (Russakovsky et al., 2015), training a ResNet 50 (He et al., 2016b) with SGD and different adaptive methods, transferring the hyperparameters from our original CIFAR-10 results.
Revised: As a ﬁnal evaluation of the role of adaptability when training convolutional networks, we repeat the previous experiment on the ImageNet dataset (Russakovsky et al., 2015), training a ResNet 50 (He et al., 2016b) with SGD and different adaptive methods, transferring the hyperparameters from our original CIFAR-10 results.

Original: Training follows Gross & Wilber (2016): the network is trained forepochs with a batch size of 256 , the learning rate is decayed by a factor of 10 at epochs 30, 60 and 90, and a weight decay of 0 . 0001 is applied.
Revised: Training follows Gross & Wilber (2016): the network is trained forepochs with a batch size of 256 , the learning rate is decayed by a factor of 10 at epochs 30, 60 and 90, and a weight decay of 0 . 0001 is applied.

Original: SGD yields 24 .
Revised: SGD yields 24 .

Original: 01% top-1 validation error, underperforming
Revised: 01% top-1 validation error, underperforming

Original: Adam ( 23 . 45% ), AMSGrad ( 23 . 46% ), AvaGrad ( 23 .
Revised: Adam ( 23 . 45% ), AMSGrad ( 23 . 46% ), AvaGrad ( 23 .

Original: 58% ) and AvaGradW ( 23 .
Revised: 58% ) and AvaGradW ( 23 .

Original: Again, note that the hyperparameters used for SGD match the ones in He et al.
Revised: Again, note that the hyperparameters used for SGD match the ones in He et al.

Original: (2016a),
Revised: (2016a),

Original: He et al.
Revised: He et al.

Original: (2016b) and
Revised: (2016b) and

Original: Gross & Wilber (2016): an initial learning rate of 0 . 1 with a momentum of 0 . 9 .
Revised: Gross & Wilber (2016): an initial learning rate of 0 . 1 with a momentum of 0 . 9 .

Original: Table 1 summarizes.
Revised: Table 1 summarizes.

Revised: [Figure]

Original: 6. L ANGUAGE M ODELLING WITH RNN S
Revised: 6. L ANGUAGE M ODELLING WITH RNN S

Original: It is perhaps not very surprising that to perform optimally in the image classiﬁcation tasks studied previously, adaptive gradient methods required large values of (cid:15) , and hence were barely adaptive.
Revised: It is perhaps not very surprising that to perform optimally in the image classiﬁcation tasks studied previously, adaptive gradient methods required large values of (cid:15) , and hence were barely adaptive.

Original: Here, we consider a task where state-of-the-art results are typically achieved by adaptive methods with low values for (cid:15) : language modelling with recurrent networks.
Revised: Here, we consider a task where state-of-the-art results are typically achieved by adaptive methods with low values for (cid:15) : language modelling with recurrent networks.

Original: In particular, we perform character-level language modelling on Penn Treebank dataset (Marcus et al., 1994; Mikolov et al., 2010) with LSTMs, following Merity et al.
Revised: In particular, we perform character-level language modelling on Penn Treebank dataset (Marcus et al., 1994; Mikolov et al., 2010) with LSTMs, following Merity et al.

Original: More speciﬁcally, the model is trained for [Figure] epochs, and the learning rate is decayed by 10 at epochs 300 and 400.
Revised: The model is trained for 500 epochs, and the learning rate is decayed by 10 at epochs 300 and 400.

Original: A batch size of 128, a BPTT length of 150, and weight decay of 1 . 2 × 10 − 6 are used, along with dropout.
Revised: A batch size of 128, a BPTT length of 150, and weight decay of 1 . 2 × 10 − 6 are used, along with dropout.

Original: We evaluate the  performance of SGD, Adam, AMSGrad,  AdaBound, AdamW, AvaGrad and AvaGradW with varying learning rate α and adaptability parameter (cid:15) , when training a 3-layer LSTM with 300 hidden units in each layer.
Revised: We evaluate the validation performance of SGD, Adam, AMSGrad, AdaShift, AdaBound, AdamW, AvaGrad and AvaGradW with varying learning rate α and adaptability parameter (cid:15) , when training a 3-layer LSTM with 300 hidden units in each layer.

Original: Figure 3 shows that, in this task, smaller values for (cid:15) are indeed optimal: Adam, AMSGrad and AvaGrad performed best with (cid:15)
Revised: Figure 3 shows that, in this task, smaller values for (cid:15) are indeed optimal: Adam, AMSGrad and AvaGrad performed best with (cid:15)

Original: = 10 − 8 .
Revised: = 10 − 8 .

Revised: The optimal learning rates for both Adam and AMSGrad, α = 0 .

Revised: 002 , agree with the value used in Merity et al.

Revised: Both AvaGrad and AvaGradW performed best with α = 200 : the former with (cid:15) = 10 − 8 , the latter with (cid:15) = 10 − 5 .

Original: Next, we train a 3-layer LSTM with 1000 hidden units per layer (the same model used in Merity et al.
Revised: Next, we train a 3-layer LSTM with 1000 hidden units per layer (the same model used in Merity et al.

Original: (2018), where it was trained with Adam, α = 0 . 002 and (cid:15) = 10 − 8 ), choosing  α, (cid:15) which yielded the best validation performance in the previous experiment. 
Revised: (2018), where it was trained with Adam), choosing values for α, (cid:15) which yielded the best validation performance in the previous experiment. For SGD, we again conﬁrmed that a learning rate ofperformed best on the validation set.

Original: Table 1 (right column) reports all results.
Revised: Table 1 (right column) reports all results.

Original: In this setting, AvaGrad and AvaGradW outperform Adam, achieving bit-per-characters of 1 . 179 and 1 . 175 compared to 1 . 182 .
Revised: In this setting, AvaGrad and AvaGradW outperform Adam, achieving bit-per-characters of 1 . 179 and 1 . 175 compared to 1 . 182 .

Original: 6. H YPERPARAMETER S EPARABILITY 
Revised: 6. H YPERPARAMETER S EPARABILITY AND D OMAIN -I NDEPENDENCE

Original: One of the main motivations behind AvaGrad is that it removes the dependence between the learning rate α and the adaptability parameter (cid:15) , at least in the worst-case rate of Theorem 2.
Revised: One of the main motivations behind AvaGrad is that it removes the dependence between the learning rate α and the adaptability parameter (cid:15) , at least in the worst-case rate of Theorem 2.

Original: Observing the heatmaps in Figure 2 and 3, we can see that indeed AvaGrad offers more separability between α and (cid:15) , when compared to Adam.
Revised: Observing the heatmaps in Figure 2 and 3, we can see that indeed AvaGrad offers more separability between α and (cid:15) , when compared to Adam.

Original: For example, for (cid:15) ≥ 0 . 0001 , it has little to no interaction with the learning rate α , as opposed to Adam where the optimal α increases linearly with (cid:15) .
Revised: For example, for (cid:15) ≥ 0 . 0001 , it has little to no interaction with the learning rate α , as opposed to Adam where the optimal α increases linearly with (cid:15) .

Original: For language modelling on Penn Treebank, the optimal learning rate for AvaGrad was α = 200 for every choice of (cid:15) , while for image classiﬁcation on CIFAR-10, we had α = 1 . 0 for all values of (cid:15) except for (cid:15) ∈ { 10 − 5 , 100 } .
Revised: For language modelling on Penn Treebank, the optimal learning rate for AvaGrad was α = 200 for every choice of (cid:15) , while for image classiﬁcation on CIFAR-10, we had α = 1 . 0 for all values of (cid:15) except for (cid:15) ∈ { 10 − 5 , 100 } .

Original: This suggests that AvaGrad enables a grid search over α and (cid:15) (with quadratic complexity) to be broken into two line searches over α and (cid:15) separately (linear complexity).
Revised: This suggests that AvaGrad enables a grid search over α and (cid:15) (with quadratic complexity) to be broken into two line searches over α and (cid:15) separately (linear complexity).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: As neural architectures become more complex, with parameters having highly heterogeneous roles, parameter-wise learning rates are often necessary for training.
Revised: As neural architectures become more complex, with parameters having highly heterogeneous roles, parameter-wise learning rates are often necessary for training.

Original: However, adaptive methods have both theoretical and empirical gaps, with SGD outperforming them in some tasks and having stronger theoretical convergence guarantees.
Revised: However, adaptive methods have both theoretical and empirical gaps, with SGD outperforming them in some tasks and having stronger theoretical convergence guarantees.

Original: In this paper, we close this gap, by ﬁrst providing a convergence rate guarantee that matches SGD’s, and by showing that, with proper hyperparameter tuning, adaptive methods can dominate in both computer vision and natural language processing tasks.
Revised: In this paper we close this gap by ﬁrst providing a convergence rate guarantee that matches SGD’s and by showing that, with proper hyperparameter tuning, adaptive methods can dominate in both computer vision and natural language processing tasks.

Original: Key to our ﬁnding is AvaGrad, our proposed optimizer whose adaptability is decoupled from its learning rate.
Revised: Key to our ﬁnding is AvaGrad, our proposed optimizer whose adaptability is decoupled from its learning rate.

Original: We hope our work will motivate further development in adaptive methods, both theoretical and practical. Our ﬁndings indicate that hyperparameter coupling should be accounted for when designing adaptive optimization methods. Our  results demonstrate that awareness of this coupling should trigger re-examination of what optimizers are best to apply in practice for training deep neural networks.
Revised: Our experimental results show that proper tuning of the learning rate together with the adaptability of the method is necessary to achieve optimal results in different domains, where distinct neural network architectures are used across tasks. By enabling this tuning to be performed in linear time, AvaGrad takes a leap towards efﬁcient domain-agnostic training of general neural architectures

Original: Appendix A P ROOF OF T HEOREM
Revised: Appendix A P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Consider the following stochastic optimization problem:
Revised: Consider the following stochastic optimization problem:

Original: [Equation]
Revised: [Equation]

Original: with probability
Revised: with probability

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: otherwise

Original: [Equation]
Revised: [Equation]

Original: 12) where C > 1 − pp > 1 + (cid:15) w 1 √ 1 − β 2 .
Revised: 12) where C > 1 − pp > 1 + (cid:15) w 1 √ 1 − β 2 .

Original: Note that ∇ f ( w ) =
Revised: Note that ∇ f ( w ) =

Original: pCw − (1 − p ) , and f is minimized at w (cid:63) =
Revised: pCw − (1 − p ) , and f is minimized at w (cid:63) =

Original: [Equation]
Revised: [Equation]

Original: The proof follows closely from Reddi’s linear example for convergence in suboptimality.
Revised: The proof follows closely from Reddi’s linear example for convergence in suboptimality.

Original: We assume w.l.o.g.
Revised: We assume w.l.o.g.

Original: that β 1 = 0 .
Revised: that β 1 = 0 .

Original: Consider:
Revised: Consider:

Original: [Equation]
Revised: [Equation]

Original: E [∆ t ]
Revised: E [∆ t ]

Original: [Equation]
Revised: [Equation]

Original: where the expectation is over all the randomness in the algorithm up to time t , as all expectations to follow in the proof.
Revised: where the expectation is over all the randomness in the algorithm up to time t , as all expectations to follow in the proof.

Original: Note that T 1 = 0 for w t = 0 .
Revised: Note that T 1 = 0 for w t = 0 .

Original: For w t > 0 we bound T 1 as follows:
Revised: For w t > 0 we bound T 1 as follows:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, T 1 ≥ min(0 ,
Revised: Hence, T 1 ≥ min(0 ,

Original: [Equation]
Revised: [Equation]

Original: As for T 2 , we have, from Jensen’s inequality:
Revised: As for T 2 , we have, from Jensen’s inequality:

Original: [Equation]
Revised: [Equation]

Original: Now, remember that v t − 1 = (1 − β
Revised: Now, remember that v t − 1 = (1 − β

Original: [Equation]
Revised: [Equation]

Original: (cid:80)
Revised: (cid:80)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =1 β t − i − 

Revised: =1 β t − i −

Original: [Equation]
Revised: [Equation]

Original: 2 i , hence:
Revised: 2 i , hence:

Original: [Equation]
Revised: [Equation]

Original: 17) and thus:
Revised: 17) and thus:

Original: [Equation]
Revised: [Equation]

Original: Plugging in the bounds for T 1 and T 2 in Equation 14:
Revised: Plugging in the bounds for T 1 and T 2 in Equation 14:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Hence, for large enough C , and C (cid:29) δ , w (cid:63) ≈ 11+ δ while the above quantity becomes nonnegative, and hence E [ w t ] ≥ w 1 .
Revised: Hence, for large enough C , and C (cid:29) δ , w (cid:63) ≈ 11+ δ while the above quantity becomes nonnegative, and hence E [ w t ] ≥ w 1 .

Original: In other words, Adam will, in expectation, drift away from the stationary point, towards w = 1 , at which point (cid:107)∇ f (1) (cid:107) 2 = δ .
Revised: In other words, Adam will, in expectation, drift away from the stationary point, towards w = 1 , at which point (cid:107)∇ f (1) (cid:107) 2 = δ .

Original: For example, δ = 1 implies that
Revised: For example, δ = 1 implies that

Original: [Equation]
Revised: [Equation]

Original: = 1 .
Revised: = 1 .

Original: To see that w = 1 is not a stationary point due to the feasibility constraints, check that ∇ f (1) = 1 > 0 : that is, the negative gradient points towards the feasible region.
Revised: To see that w = 1 is not a stationary point due to the feasibility constraints, check that ∇ f (1) = 1 > 0 : that is, the negative gradient points towards the feasible region.

Original: B P ROOF OF T HEOREM
Revised: B P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Throughout the proof we use the following notation for clarity:
Revised: Throughout the proof we use the following notation for clarity:

Original: [Equation]
Revised: [Equation]

Original: We start from the fact that f is M -smooth:
Revised: We start from the fact that f is M -smooth:

Original: [Equation]
Revised: [Equation]

Original: and use the update w t +

Revised: and use the update w t +

Original: =
Revised: =

Original: w t − α t · η t (cid:12) m t :
Revised: w t − α t · η t (cid:12) m t :

Original: [Equation]
Revised: [Equation]

Original: (cid:12)
Revised: (cid:12)

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table][Equation] [Table] [Equation]
Revised: [Table]

Original: [Table]

Original: [Equation][Table] [Equation] [Table]
Revised: [Equation]

Original: [Equation] [Table] [Equation]
Revised: [Table] [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] 
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁrst step we used the fact that (cid:107) m t (cid:12) η t (cid:107) 

Revised: where in the ﬁrst step we used the fact that (cid:107) m t (cid:12) η t (cid:107)

Original: = (cid:80) di
Revised: = (cid:80) di

Original: =1 m 2 t
Revised: =1 m 2 t

Original: [Equation]
Revised: [Equation]

Original: i η 2 t,i ≤ max j m 2 t
Revised: i η 2 t,i ≤ max j m 2 t

Original: [Equation]
Revised: [Equation]

Original: j (cid:80) di
Revised: j (cid:80) di

Original: =1 η 2 t,i ≤
Revised: =1 η 2 t,i ≤

Original: G 2 ∞ (cid:107) η t (cid:107) 2 , in the second we used m t = β 1 ,t m t − 1 +(1 − β 1 ,t ) g t , in the third we used Cauchy-Schwarz, and in the fourth we used (cid:107)∇ f ( w t ) (cid:107) ≤ G 2 , along with (cid:107) m t − 1 (cid:12) η t (cid:107) =
Revised: G 2 ∞ (cid:107) η t (cid:107) 2 , in the second we used m t = β 1 ,t m t − 1 +(1 − β 1 ,t ) g t , in the third we used Cauchy-Schwarz, and in the fourth we used (cid:107)∇ f ( w t ) (cid:107) ≤ G 2 , along with (cid:107) m t − 1 (cid:12) η t (cid:107) =

Original: (cid:113)(cid:80) d i =

Revised: (cid:113)(cid:80) d i =

Original: m
Revised: m

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: 1 ,i η 2 t,i ≤ max j η t,j (cid:113)(cid:80) di =1 m 2 t
Revised: 1 ,i η 2 t,i ≤ max j η t,j (cid:113)(cid:80) di =1 m 2 t

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: t .
Revised: t .

Original: Now, taking the expectation over s t , and using the fact that E s t [ g t ] = ∇ f ( w t ) , and that η t , α t are both independent of s t :
Revised: Now, taking the expectation over s t , and using the fact that E s t [ g t ] = ∇ f ( w t ) , and that η t , α t are both independent of s t :

Original: [Equation]
Revised: [Equation]

Original: (23) where in the second step we used β 1 ,t ≤ β 1 and (cid:104)∇ f ( w t ) , ∇ f ( w t ) (cid:12) η t (cid:105) =
Revised: (23) where in the second step we used β 1 ,t ≤ β 1 and (cid:104)∇ f ( w t ) , ∇ f ( w t ) (cid:12) η t (cid:105) =

Original: (cid:80) d i =1 ∇ f ( w ) 2 i η t,i ≥ min j η t,j (cid:80) di
Revised: (cid:80) d i =1 ∇ f ( w ) 2 i η t,i ≥ min j η t,j (cid:80) di

Original: =1 ∇ f ( w ) 

Revised: =1 ∇ f ( w )

Original: i
Revised: i

Original: =
Revised: =

Original: L t (cid:107)∇ f ( w ) (cid:107) 2 .
Revised: L t (cid:107)∇ f ( w ) (cid:107) 2 .

Original: Re-arranging, we get:
Revised: Re-arranging, we get:

Original: [Equation]
Revised: [Equation]

Original: Now, deﬁning p ( t ) =
Revised: Now, deﬁning p ( t ) =

Original: α t L t Z , where Z = (cid:80) Tt =
Revised: α t L t Z , where Z = (cid:80) Tt =

Original: 1 α t L t , dividing by Z (1 − β 1 ) and summing over t :
Revised: 1 α t L t , dividing by Z (1 − β 1 ) and summing over t :

Original: T 
Revised: T (cid:88) [Equation] =

Original: [Equation]
Revised: [Equation]

Original: Now, taking the conditional expectation over all samples S given Z :
Revised: Now, taking the conditional expectation over all samples S given Z :

Original: [Equation]
Revised: [Equation]

Original: 26) where in the second step we used E S [ E s t [ f ( w t +1 )] | Z ]=
Revised: where in the second step we used E S [ E s t [ f ( w t +1 )] | Z ]

Original:  E S
Revised: = E S

Original: [ f ( w t +1 )] which follows from the assumption that p ( Z | s t ) =
Revised: [ f ( w t +1 )] which follows from the assumption that p ( Z | s t ) =

Original: p ( Z ) ,  along with telescoping sum.
Revised: p ( Z ) , and the third step follows from a telescoping sum, along with the fact that E S [ f ( w 1 )] = f ( w 1 ) .

Original: Now, using f ( w 1 ) − E S [ f ( w T +1 ) | Z ] ≤ f [Equation] f 
Revised: Now, using f ( w 1 ) − E S [ f ( w T +1 ) | Z ] ≤ f ( w 1 ) − f ( w

Original: [Equation]
Revised: [Equation]

Original:  D :
Revised: ) ≤ D :

Original: E S
Revised: E S

Original: [Equation]
Revised: [Equation]

Original: Then, taking the expectation over Z :
Revised: Then, taking the expectation over Z :

Original: [Equation]
Revised: [Equation]

Revised: Now, let α t = γ t (cid:113) 2 D TMG 2 ∞ :

Original: [Equation]
Revised: [Equation]

Original: Now, let α t = γ t (cid:113) 2 D TMG 2  ∞ :
Revised: where we used the fact that G 2 ≤ G ∞ √ d . [Table] [Table] [Equation]

Original: E S (cid:34)
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Revised: [Table]

Original: t = p
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation]
Revised: [Equation]

Original: t
Revised: [Table] [Equation] [Table] [Equation]

Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Original: (cid:35)
Revised: [Equation] [Table] [Equation] [Table]

Original: [Equation][Equation]
Revised: [Equation]

Original: where we used the fact that G 2 ≤ G ∞ √ d .

Original: Now, recall that Z = (cid:80) Tt = 1 α t L t = (cid:113) 2 D TMG 2 ∞ (cid:80) Tt =1 γ t L t :
Revised: [Table]

Original: E S (cid:34) T (cid:88) t =

Original: [Equation]
Revised: [Equation]

Original:  (30) Setting β
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: = β
Revised: [Table]

Original: [Equation][-= 
-]
Revised: [Equation][-= 
-]

Original: and checking that (cid:80)

Original: [Equation]
Revised: [Table]

Original: Recalling that L t = min i η t,i proves the claim
Revised: Recalling that L t = min i η t,i proves the claim.

Original: [Equation]

Original: B. T HE CASE WITH FIRST - ORDER MOMENTUM
Revised: B. T HE CASE WITH FIRST - ORDER MOMENTUM

Original: For the case β 1 ,t > 0 , assume that β 1 ,t = β 1 √ t in Equation 30:E S
Revised: For the case β 1 ,t > 0 , assume that β 1 ,t = β 1 √ t in Equation 30:

Original: [Equation]
Revised: [Equation]

Original: where in the last step we used (cid:80)
Revised: where in the last step we used (cid:80)

Original: Tt =1 1 √ t ≤ 2 √ T .
Revised: Tt =1 1 √ t ≤ 2 √ T .

Original: Similarly to the guarantee in Equation 31, we can show a O (1 / √ T ) convergence rate if we further assume that there exist constants H and L such that
Revised: Similarly to the guarantee in Equation 31, we can show a O (1 / √ T ) convergence rate if we further assume that there exist constants H and L such that

Original: 0 <
Revised: 0 <

Original: L ≤ η t,i ≤ H < ∞ for all i and t ( i.e., the parameter-wise learning rates are bounded away from zero and also from above), and that γ t is bounded similarly.
Revised: L ≤ η t,i ≤ H < ∞ for all i and t ( i.e., the parameter-wise learning rates are bounded away from zero and also from above), and that γ t is bounded similarly.

Original: For example, having γ t = γ yields:
Revised: For example, having γ t = γ yields:

Original: [Equation]
Revised: [Equation]

Original: where in the ﬁrst step we used (cid:107) η t (cid:107) 2 ≤ dH 2 , H t ≤ H , and L t ≥ L .
Revised: where in the ﬁrst step we used (cid:107) η t (cid:107) 2 ≤ dH 2 , H t ≤ H , and L t ≥ L .

Original: Note that for any constant γ the above is O (1 / √ T ) .
Revised: Note that for any constant γ the above is O (1 / √ T ) .

Original: B. T HE CASE WITH UNCONDITIONAL DISTRIBUTION OVER ITERATES
Revised: B. T HE CASE WITH UNCONDITIONAL DISTRIBUTION OVER ITERATES

Original: To show a similar bound without the assumption that p ( Z | s t )
Revised: To show a similar bound without the assumption that p ( Z | s t )

Original: = p ( Z ) , we can alternatively bound L t and (cid:107) η t (cid:107) using the worst case over possible samples S .
Revised: = p ( Z ) , we can alternatively bound L t and (cid:107) η t (cid:107) using the worst case over possible samples S .

Original: From (24) we have, with β 1 ,t = 0 :
Revised: From (24) we have, with β 1 ,t = 0 :

Original: [Equation]
Revised: [Equation]

Original: inf
Revised: inf

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]

Original: Now, deﬁne p ( t ) =
Revised: Now, deﬁne p ( t ) =

Original: α t (inf S L t ) /Z with Z = (cid:80) Tt =1 α t (inf S L t ) instead.
Revised: α t (inf S L t ) /Z with Z = (cid:80) Tt =1 α t (inf S L t ) instead.

Original: As long as α t does not depend on S , Z is no longer a random variable.
Revised: As long as α t does not depend on S , Z is no longer a random variable.

Original: Following the same steps as above leads to the following:
Revised: Following the same steps as above leads to the following:

Original: [Equation]
Revised: [Equation]

Original: In particular, if there are constants H and L such that 0 <
Revised: In particular [Equation] if there are constants H and L such that 0 <

Original: L ≤ η t,i ≤ H < ∞ for all i and t , can bound sup S (cid:107) η t (cid:107) 2 ≤ dH 2 and inf S L t ≥ L , yielding:
Revised: L ≤ η t,i ≤ H < ∞ for all i and t [Equation] can bound sup S (cid:107) η t (cid:107) 2 ≤ dH 2 and inf S L t ≥ L , yielding:

Original: [Equation]
Revised: [Equation]

Original: Hence a O (1 / √ T ) follows as long as γ t can be upper and lower bounded accordingly by constants.
Revised: Hence a O (1 / √ T ) follows as long as γ t can be upper and lower bounded accordingly by constants.

Original: C E XPERIMENTAL D ETAILS C. CIFAR
Revised: C E XPERIMENTAL D ETAILS C. CIFAR

Original: The CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009) consist of 60,000 RGB images with 32 × 32 pixels, with a standard split of 50,000 and 10,000 training and test images, sampled from 10 and 100 classes, respectively.
Revised: The CIFAR-10 and CIFAR-100 datasets (Krizhevsky, 2009) consist of 60,000 RGB images with 32 × 32 pixels, with a standard split of 50,000 and 10,000 training and test images, sampled from 10 and 100 classes, respectively.

Original: We pre-process the dataset with channel-wise normalization using statistics from the training set, and for data-augmentation we ﬂip each image horizontally with 50% probability, along with a random cropping, achieved by ﬁrst padding 4 black pixels to each edge of the image and then extracting a random 32 × 32 crop.
Revised: We pre-process the dataset with channel-wise normalization using statistics from the training set, and for data-augmentation we ﬂip each image horizontally with 50% probability, along with a random cropping, achieved by ﬁrst padding 4 black pixels to each edge of the image and then extracting a random 32 × 32 crop.

Revised: For the hyperparameter exploration with a Wide ResNet 28-4, we measure performance on a validation set of 5,000 images sampled from the training set, while for the ﬁnal results with the Wide ResNet 28-10, we report results on the test set.

Original: Each Wide ResNet is trained with a batch size of 128 for a total of 200 epochs on a single GPU, with the learning rate being decayed by a factor of 5 at epochs 60, 120 and 160.
Revised: Each Wide ResNet is trained with a batch size of 128 for a total of 200 epochs on a single GPU, with the learning rate being decayed by a factor of 5 at epochs 60, 120 and 160.

Original: A weight decay of 0 .is used with every optimization method.
Revised: A weight decay of 0 .is used with every optimization method.

Original: For SGD, we use a momentum of 0 .
Revised: For SGD, we use a momentum of 0 .

Original: 9 , while for adaptive methods, we use β 1 = 0 .
Revised: 9 , while for adaptive methods, we use β 1 = 0 .

Original: 9 , β 2 = 0 .
Revised: 9 , β 2 = 0 .

Original: 999 , as suggested in their respective papers.
Revised: 999 , as suggested in their respective papers.

Original: For AdaBound, we use the default α ∗ = 0 .
Revised: For AdaBound, we use the default α ∗ = 0 .

Original: 1 and γ = 10 − 3 .
Revised: 1 and γ = 10 − 3 .

Revised: For AdaShift, we use the default n = 10 .

Original: C. I MAGE N ET
Revised: C. I MAGE N ET

Original: The ILSVRC 2012 datasets (Russakovsky et al., 2015) is composed of 1.2M training and 50, validation RGB images, sampled from a total of 1,000 classes.
Revised: The ILSVRC 2012 datasets (Russakovsky et al., 2015) is composed of 1.2M training and 50, validation RGB images, sampled from a total of 1,000 classes.

Original: We train a ResNet 50 (He et al., 2016b), and report results using single-crops of 224 × 224 images for evaluation.
Revised: We train a ResNet 50 (He et al., 2016b), and report results using single-crops of 224 × 224 images for evaluation.

Original: We follow Gross & Wilber (2016) for data-augmentation and training speciﬁcs: in particular, we decay the learning rate by a factor of 10 at epochs 30, 60 and 90, and use a weight decay of 0 . 0001 .
Revised: We follow Gross & Wilber (2016) for data-augmentation and training speciﬁcs: in particular, we decay the learning rate by a factor of 10 at epochs 30, 60 and 90, and use a weight decay of 0 . 0001 .

Original: We use 4 GPUs for training, with a total batch size of 256 (64 per GPU).
Revised: We use 4 GPUs for training, with a total batch size of 256 (64 per GPU).

Original: C. P ENN T REEBANK
Revised: C. P ENN T REEBANK

Original: We follow Merity et al.
Revised: We follow Merity et al.

Original: (2018) for both the model and optimization details for the Penn Treebank dataset (Marcus et al., 1994) for character-level language modelling (Mikolov et al., 2010).
Revised: (2018) for both the model and optimization details for the Penn Treebank dataset (Marcus et al., 1994) for character-level language modelling (Mikolov et al., 2010).

Revised: Again, we measure validation performance for the small-scale experiments, and test performance for the large-scale ones.

Original: More speciﬁcally, we train a 3-layer LSTM (Hochreiter & Schmidhuber, 1997) with an embedding size of 200, and decay the learning rate by a factor of 10 at epochs 300 and 400, while the model is trained for a total of 500 epochs.
Revised: More speciﬁcally, we train a 3-layer LSTM (Hochreiter & Schmidhuber, 1997) with an embedding size of 200, and decay the learning rate by a factor of 10 at epochs 300 and 400, while the model is trained for a total of 500 epochs.

Original: We use a batch size of 128 and BPTT length of 150.
Revised: We use a batch size of 128 and BPTT length of 150.

Original: The model is regularized with a weight decay of 1 . 2 × 10 − 6 , a weight dropout (Merity et al., 2017) of p = 0 .
Revised: The model is regularized with a weight decay of 1 . 2 × 10 − 6 , a weight dropout (Merity et al., 2017) of p = 0 .

Original: 5 , and a dropout (Srivastava et al., 2014) of p = 0 .
Revised: 5 , and a dropout (Srivastava et al., 2014) of p = 0 .

Original: 1 for the input and output layers and p = 0 .
Revised: 1 for the input and output layers and p = 0 .

Original: 25 for the hidden-layer.
Revised: 25 for the hidden-layer.