Original: O N
Revised: O N

Original: THE INSUFFICIENCY OF EXISTING MOMENTUM SCHEMES FOR S TOCHASTIC O PTIMIZATION
Revised: THE INSUFFICIENCY OF EXISTING MOMENTUM SCHEMES FOR S TOCHASTIC O PTIMIZATION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov’s accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide signiﬁcant improvements over stochastic gradient descent (SGD).
Revised: Momentum based stochastic gradient methods such as heavy ball (HB) and Nesterov’s accelerated gradient descent (NAG) method are widely used in practice for training deep networks and other supervised learning models, as they often provide signiﬁcant improvements over stochastic gradient descent (SGD).

Original: Theoretically, these “fast gradient” methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact.
Revised: In general, “fast gradient” methods have provable improvements over gradient descent only for the deterministic case, where the gradients are exact.

Original: In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case , they partially mimic their exact gradient counterparts resulting in some practical gain
Revised: In the stochastic case, the popular explanations for their wide applicability is that when these fast gradient methods are applied in the stochastic case, they partially mimic their exact gradient counterparts, resulting in some practical gain.

Original: This work provides a counterpoint to this belief by proving that there are simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters.
Revised: This work provides a counterpoint to this belief by proving that there are simple problem instances where these methods cannot outperform SGD despite the best setting of its parameters.

Original: These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances.
Revised: These negative problem instances are, in an informal sense, generic; they do not look like carefully constructed pathological instances.

Original: These results suggest (along with empirical evidence) that HB or NAG’s practical performance gains are a by-product of minibatching
Revised: These results suggest (along with empirical evidence) that HB or NAG’s practical performance gains are a by-product of minibatching.

Original: Furthermore this work provides a viable (and provable) alternative which , on the same set of problem instances signiﬁcantly improves over HB NAG, and SGD’s performance.
Revised: Furthermore, this work provides a viable (and provable) alternative, which, on the same set of problem instances, signiﬁcantly improves over HB, NAG, and SGD’s performance.

Original: This algorithm, denoted as ASGD, is a simple to implement stochastic algorithm, based on a relatively less popular version of Nesterov’s AGD.
Revised: This algorithm, referred to as Accelerated Stochastic Gradient Descent (ASGD), is a simple to implement stochastic algorithm, based on a relatively less popular variant of Nesterov’s Acceleration.

Original: Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD.
Revised: Extensive empirical results in this paper show that ASGD has performance gains over HB, NAG, and SGD.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: First order optimization methods, which access a function (to be optimized) through its gradient or an unbiased approximation of its gradient, are the workhorses for modern large scale optimization problems, which include training the current state-of-the-art deep neural networks.
Revised: First order optimization methods, which access a function (to be optimized) through its gradient or an unbiased approximation of its gradient, are the workhorses for modern large scale optimization problems, which include training the current state-of-the-art deep neural networks.

Original: Gradient descent (Cauchy, 1847) is the simplest ﬁrst order method that is used heavily in practice.
Revised: Gradient descent (Cauchy, 1847) is the simplest ﬁrst order method that is used heavily in practice.

Original: However, it is known that for the class of smooth convex functions, gradient descent is suboptimal (Nesterov, 2004) and there exists a class of algorithms called fast gradient/momentum based methods which achieve optimal convergence guarantees.
Revised: However, it is known that for the class of smooth convex functions as well as some simple non-smooth problems (Nesterov, 2012a)), gradient descent is suboptimal (Nesterov, 2004) and there exists a class of algorithms called fast gradient/momentum based methods which achieve optimal convergence guarantees.

Original: The heavy ball method (Polyak, 1964) and Nesterov’s accelerated gradient descent (Nesterov, 1983) are two of the most popular methods in this category.
Revised: The heavy ball method (Polyak, 1964) and Nesterov’s accelerated gradient descent (Nesterov, 1983) are two of the most popular methods in this category.

Original: On the other hand, training deep neural networks on large scale datasets have been possible through the use of Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951), which samples a random subset of training data to compute gradient estimates that are then used to optimize the objective function.
Revised: On the other hand, training deep neural networks on large scale datasets have been possible through the use of Stochastic Gradient Descent (SGD) (Robbins & Monro, 1951), which samples a random subset of training data to compute gradient estimates that are then used to optimize the objective function.

Original: The advantages of SGD for large scale optimization and the related issues of tradeoffs between computational and statistical efﬁciency was highlighted in Bottou & Bousquet (2007).
Revised: The advantages of SGD for large scale optimization and the related issues of tradeoffs between computational and statistical efﬁciency was highlighted in Bottou & Bousquet (2007).

Original: The above mentioned theoretical advantages of fast gradient methods (Polyak, 1964; Nesterov, 1983) (albeit for smooth convex problems) coupled with cheap to compute stochastic gradient estimates led to the inﬂuential work of Sutskever et al.
Revised: The above mentioned theoretical advantages of fast gradient methods (Polyak, 1964; Nesterov, 1983) (albeit for smooth convex problems) coupled with cheap to compute stochastic gradient estimates led to the inﬂuential work of Sutskever et al.

Original: (2013), which demonstrated the empirical advantages possessed by SGD when augmented with the momentum machinery.
Revised: (2013), which demonstrated the empirical advantages possessed by SGD when augmented with the momentum machinery.

Original: This work has led to the wide spread adoption of momentum methods for training deep neural networks; so much so that, in the context of neural network training, gradient descent often refers to momentum methods.
Revised: This work has led to the wide spread adoption of momentum methods for training deep neural networks; so much so that, in the context of neural network training, gradient descent often refers to momentum methods.

Original: But, there is a subtle difference between classical momentum methods and their implementation in practice – classical momentum methods work in the exact ﬁrst order oracle model (Nesterov, 2004), i.e., they employ exact gradients (computed on the full training dataset), while in practice (Sutskever et al., 2013), they are implemented with stochastic gradients (estimated from a randomly sampled subset/mini-batch of the training data).
Revised: But, there is a subtle difference between classical momentum methods and their implementation in practice – classical momentum methods work in the exact ﬁrst order oracle model (Nesterov, 2004), i.e., they employ exact gradients (computed on the full training dataset), while in practice (Sutskever et al., 2013), they are implemented with stochastic gradients (estimated from a randomly sampled subset/mini-batch of the training data).

Original: This leads to a natural question:
Revised: This leads to a natural question:

Original: Are momentum methods optimal even in the stochastic ﬁrst order oracle (SFO) model, where we access stochastic gradients computed on a small constant sized minibatches (or a batchsize of 1 ?)
Revised: Are momentum methods optimal even in the stochastic ﬁrst order oracle (SFO) model, where we access stochastic gradients computed on a small constant sized minibatches (or a batchsize of 1 ?)

Original: Even disregarding the question of optimality of momentum methods in the SFO model, it is not even known if momentum methods (say, Polyak (1964); Nesterov (1983)) provide any provable improvement over SGD in this model.
Revised: Even disregarding the question of optimality of momentum methods in the SFO model, it is not even known if momentum methods (say, Polyak (1964); Nesterov (1983)) provide any provable improvement over SGD in this model.

Original: While these are open questions, a recent effort of Jain et al.
Revised: While these are open questions, a recent effort of Jain et al.

Original: (2017) showed that improving upon SGD (in the stochastic ﬁrst order oracle) is rather subtle as there exists problem instances in SFO model where it is not possible to improve upon SGD, even information theoretically.
Revised: (2017) showed that improving upon SGD (in the stochastic ﬁrst order oracle) is rather subtle as there exists problem instances in SFO model where it is not possible to improve upon SGD, even information theoretically.

Original: Jain et al.
Revised: Jain et al.

Original: (2017) studied a variant of Nesterov’s accelerated gradient updates (Nesterov, 2012) for stochastic linear regression and show that their method improves upon SGD wherever it is information theoretically admissible.
Revised: (2017) studied a variant of Nesterov’s accelerated gradient updates (Nesterov, 2012b) for stochastic linear regression and show that their method improves upon SGD wherever it is information theoretically admissible.

Original: Through out this paper, we refer to the algorithm of (Jain et al., 2017) as Accelerated Stochastic Gradient Method (ASGD) while we refer to a stochastic version of the most widespread form of Nesterov’s method (Nesterov, 1983) as NAG; HB denotes a stochastic version of the heavy ball method.
Revised: Through out this paper, we refer to the algorithm of Jain et al. (2017) as Accelerated Stochastic Gradient Method (ASGD) while we refer to a stochastic version of the most widespread form of Nesterov’s method (Nesterov, 1983) as NAG; HB denotes a stochastic version of the heavy ball method.

Original: Critically, while the result of
Revised: Critically, while the result of

Original: Jain et al.
Revised: Jain et al.

Original: (2017) shows that ASGD improves on SGD in any information-theoretically admissible regime, it is still not known whether HB and NAG can achieve a similar performance gain.
Revised: (2017) shows that ASGD improves on SGD in any information-theoretically admissible regime, it is still not known whether HB and NAG can achieve a similar performance gain.

Original: A key contribution of this work is to show that HB does not provide similar performance gain to ASGD even when it is informationally-theoretically admissible.
Revised: A key contribution of this work is to show that HB does not provide similar performance gain over SGD even when it is informationally-theoretically admissible.

Original: That is, we provide a problem instance where it is indeed possible to improve upon SGD (and in fact ASGD achieves this improvement), but HB cannot achieve any improvement over SGD ˙We validate this claim empirically as well.
Revised: That is, we provide a problem instance where it is indeed possible to improve upon SGD (and in fact ASGD achieves this improvement), but HB cannot achieve any improvement over SGD. We validate this claim empirically as well.

Original: In fact, we provide empirical evidence to the claim that NAG also do not achieve any improvement over SGD for several problems where ASGD can still achieve better rates of convergence.
Revised: In fact, we provide empirical evidence to the claim that NAG also do not achieve any improvement over SGD for several problems where ASGD can still achieve better rates of convergence.

Original: This raises a question about why HB and NAG provide better performance than SGD in practice (Sutskever et al., 2013), especially for training deep networks.
Revised: This raises a question about why HB and NAG provide better performance than SGD in practice (Sutskever et al., 2013), especially for training deep networks.

Original: Our conclusion (that is well supported by our theoretical result) is that HB and NAG’s improved performance is attributed to mini-batching and hence, these methods will often struggle to improve over SGD with small constant batch sizes.
Revised: Our conclusion (that is well supported by our theoretical result) is that HB and NAG’s improved performance is attributed to mini-batching and hence, these methods will often struggle to improve over SGD with small constant batch sizes.

Original: Note that this result indicates that there is a natural tension between the gains offered by momentum methods with larger batches and the superior generalization properties offered by training with small mini-batches (Keskar et al., 2016), which is a regime that may not be amenable for HB/NAG to improve upon SGD.

Original: This is in stark contrast to methods like ASGD, which are designed to improve over SGD across small or large mini-batch sizes.
Revised: This is in stark contrast to methods like ASGD, which is designed to improve over SGD across small or large mini-batch sizes.

Original: In fact, based on our experiments, we observe that on the task of training deep residual networks (He et al., 2016a) on the cifar-10 dataset, we note that ASGD offers noticeable improvements by achieving 5 − 7% better test error over HB and NAG even with commonly used batch sizes like 128 during the initial stages of the optimization.
Revised: In fact, based on our experiments, we observe that on the task of training deep residual networks (He et al., 2016a) on the cifar-10 dataset, we note that ASGD offers noticeable improvements by achieving 5 − 7% better test error over HB and NAG even with commonly used batch sizes like 128 during the initial stages of the optimization.

Original: 1. C ONTRIBUTIONS
Revised: 1. C ONTRIBUTIONS

Original: The contributions of this paper are as follows.
Revised: The contributions of this paper are as follows.

Original: In Section 3, we prove that HB is not optimal in the SFO model.
Revised: In Section 3, we prove that HB is not optimal in the SFO model.

Original: In particular, there exist linear regression problems for which the performance of HB (with any step size and momentum) is either the same or worse than that of SGD while ASGD improves upon both of them.
Revised: In particular, there exist linear regression problems for which the performance of HB (with any step size and momentum) is either the same or worse than that of SGD while ASGD improves upon both of them.

Original: Experiments on several different linear regression problems suggest that the suboptimality of HB in the SFO model is not restricted to special cases – it is rather widespread.
Revised: Experiments on several different linear regression problems suggest that the suboptimality of HB in the SFO model is not restricted to special cases – it is rather widespread.

Original: Empirically, the same holds true for NAG as well (Section 5).
Revised: Empirically, the same holds true for NAG as well (Section 5).

Original: The above observations suggest that the only reason for the superiority of momentum methods in practice is minibatching, which reduces the variance in stochastic gradients and moves  closer to  exact ﬁrst order oracle.
Revised: The above observations suggest that the only reason for the superiority of momentum methods in practice is minibatching, which reduces the variance in stochastic gradients and moves the SFO closer to the exact ﬁrst order oracle.

Original: This conclusion is supported by empirical evidence through training deep residual networks on cifar-10, with a batch size of 8 (see Section 5.3).
Revised: This conclusion is supported by empirical evidence through training deep residual networks on cifar-10, with a batch size of 8 ( see Section 5.3)

Original: We present an intuitive and easier to tune version of ASGD (see Section4) and show that ASGD can provide signiﬁcantly faster convergence to a reasonable accuracy than SGD, HB, NAG,while still providing asymptotically optimal accuracy.
Revised: We present an intuitive and easier to tune version of ASGD (see Section and show that ASGD can provide signiﬁcantly faster convergence to a reasonable accuracy than SGD, HB, NAG,

Revised: while

Original: Hence, the take-home message of this paper is: HB and NAG are not optimal in the SFO model.
Revised: still providing favorable or comparable asymptotic

Revised: accuracy

Revised: as these methods.

Revised: Hence, the take-home message of this paper is: HB and NAG are not optimal in the SFO model.

Original: The only reason for the superiority of momentum methods in practice is minibatching.
Revised: The only reason for the superiority of momentum methods in practice is minibatching.

Original: ASGD provides a distinct advantage in training deep networks over SGD, HB and NAG.
Revised: ASGD provides a distinct advantage in training deep networks over SGD, HB and NAG.

Original: Require:
Revised: Require:

Original: Initial w 0 , stepsize δ , momentum α 1: w − 1 ← w 0 ; t ←/*Set w − 1 to w 0 */ 2: while w t not converged do 3: w t +1 ← w t − δ · (cid:98) ∇ f t ( w t )+ α · ( w t − w t − 1 ) /*Sum
Revised: Initial w 0 , stepsize δ , momentum α 1: w − 1 ← w 0 ; t ←/*Set w − 1 to w 0 */ 2: while w t not converged do 3: w t +1 ← w t − δ · (cid:98) ∇ f t ( w t )+ α · ( w t − w t − 1 ) /*Sum

Original: of stochastic gradient step and momentum*/ 4: t ← t +
Revised: of stochastic gradient step and momentum*/ 4: t ← t +

Original: Ensure: w t /*Return the last iterate*/
Revised: Ensure: w t /*Return the last iterate*/

Original: Algorithm
Revised: Algorithm

Original: Require:
Revised: Require:

Original: Initial w 0 , stepsize δ , momentum α 1: v 0 ← w 0 ; t ←/*Set v 0 to w 0 */ 2: while w t not converged do 3: v t +1 ← w t − δ · (cid:98) ∇ f t ( w t ) /*SGD
Revised: Initial w 0 , stepsize δ , momentum α 1: v 0 ← w 0 ; t ←/*Set v 0 to w 0 */ 2: while w t not converged do 3: v t +1 ← w t − δ · (cid:98) ∇ f t ( w t ) /*SGD

Original: step*/ 4: w t +1 =
Revised: step*/ 4: w t +1 =

Original: (1 + α ) v t +1 − αv t /*Sum of SGD step and previous iterate*/ 5: t ← t +
Revised: (1 + α ) v t +1 − αv t /*Sum of SGD step and previous iterate*/ 5: t ← t +

Original: Ensure: w t /*Return the last iterate*/
Revised: Ensure: w t /*Return the last iterate*/

Original:  N OTATION
Revised:  N OTATION

Original: We denote matrices by bold-face capital letters and vectors by lower-case letters.
Revised: We denote matrices by bold-face capital letters and vectors by lower-case letters.

Original: f ( w ) =
Revised: f ( w ) =

Original: 1 /n (cid:80) i f i ( w ) denotes the function to optimize w.r.t.
Revised: 1 /n (cid:80) i f i ( w ) denotes the function to optimize w.r.t.

Original:  w .
Revised: model parameters w .

Original: ∇ f ( w ) denote gradient of f at w while (cid:98) ∇ f t ( w ) denote a stochastic gradient of f .
Revised: ∇ f ( w ) denotes exact gradient of f at w while (cid:98) ∇ f t ( w ) denotes a stochastic gradient of f .

Original: That is, (cid:98) ∇ f t ( w t ) =
Revised: That is, (cid:98) ∇ f t ( w t ) =

Original: ∇ f i t ( w ) where i t is sampled uniformly at random from [1 , . .
Revised: ∇ f i t ( w ) where i t is sampled uniformly at random from [1 , . .

Original: . , n ] .
Revised: . , n ] .

Original: For linear regression, f i ( w ) =
Revised: For linear regression, f i ( w ) =

Original: ( y −(cid:104) w, x i (cid:105) ) 2 where y  is the targetvariable and x  ∈ (cid:60) d is the covariate, and (cid:98) ∇ f t ( w t )
Revised: ( y i − (cid:104) w, x i (cid:105) ) 2 where y i ∈ (cid:60) is the target and x i ∈ (cid:60) d is the covariate, and (cid:98) ∇ f t ( w t )

Original: = − ( y t −(cid:104)
Revised: = − ( y t −(cid:104)

Original: w t , x t (cid:105) )
Revised: w t , x t (cid:105) )

Original: x t .
Revised: x t .

Original: In this case, H = E (cid:2) xx (cid:62) (cid:3) denotes the Hessian of f and κ = λ 1 ( H ) λ
Revised: In this case, H = E (cid:2) xx (cid:62) (cid:3) denotes the Hessian of f and κ = λ 1 ( H ) λ

Original: d ( H ) denotes it’s condition number.
Revised: d ( H ) denotes it’s condition number.

Original: Algorithm 1 provides a pseudo-code of HB method (Polyak, 1964).
Revised: Algorithm 1 provides a pseudo-code of HB method (Polyak, 1964).

Original: w t − w t − 1 is the momentum term and α denotes the momentum parameter.
Revised: w t − w t − 1 is the momentum term and α denotes the momentum parameter.

Original: Next iterate w t +1 is obtained by a linear combination of the SGD update and the momentum term.
Revised: Next iterate w t +1 is obtained by a linear combination of the SGD update and the momentum term.

Original: Algorithm 2 provides pseudo-code of a stochastic version of the most commonly used form of Nesterov’s accelerated gradient descent (Nesterov, 1983)
Revised: Algorithm 2 provides pseudo-code of a stochastic version of the most commonly used form of Nesterov’s accelerated gradient descent (Nesterov, 1983)

Original:  S UBOPTIMALITY OF H EAVY B ALL M ETHOD
Revised:  S UBOPTIMALITY OF H EAVY B ALL M ETHOD

Original: In this section, we show that there exist linear regression problems where the performance of HB (Algorithm 1) is no better than that of SGD, while ASGD signiﬁcantly improves upon this performance.
Revised: In this section, we show that there exists linear regression problems where the performance of HB (Algorithm 1) is no better than that of SGD, while ASGD signiﬁcantly improves upon SGD’s performance.

Original: Let us now describe the problem instance.
Revised: Let us now describe the problem instance.

Original: Fix w ∗ ∈ R 2 and let ( x , y ) ∼ D be the following distribution over R 2 :
Revised: Fix w ∗ ∈ R 2 and let ( x , y ) ∼ D be the following distribution over R 2 :

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original:  where e 1 , e 2 ∈ R 2 are canonical basis vectors, σ 1 > σ 2 > 0 .
Revised: = [Equation] where e 1 , e 2 ∈ R 2 are canonical basis vectors, σ 1 > σ 2 > 0 .

Original: Let z be a random variable such that E (cid:2)
Revised: Let z be a random variable such that E (cid:2)

Original: z 2 (cid:3)
Revised: z 2 (cid:3)

Original: = 2 and E (cid:2) z 4 (cid:3) = 2 c ≥ 4 .
Revised: = 2 and E (cid:2) z 4 (cid:3) = 2 c ≥ 4 .

Original: Hence, we have: E (cid:2) ( x ( i ) ) 2 (cid:3)
Revised: Hence, we have: E (cid:2) ( x ( i ) ) 2 (cid:3)

Original: =
Revised: =

Original: σ 2 i , E (cid:2) ( x ( i ) ) 4 (cid:3) = cσ 4 i , for i
Revised: σ 2 i , E (cid:2) ( x ( i ) ) 4 (cid:3) = cσ 4 i , for i

Original: = 1 , 2 .
Revised: = 1 , 2 .

Original: Now, our goal is to minimize:
Revised: Now, our goal is to minimize:

Original: [Equation]
Revised: [Equation]

Original: Let κ and ˜ κ denote the computational and statistical condition numbers – see Jain et al.
Revised: Let κ and ˜ κ denote the computational and statistical condition numbers – see Jain et al.

Original: (2017) for deﬁnitions.
Revised: (2017) for deﬁnitions.

Original: For the problem above, we have κ = 2 cσ 21 σ 22 and ˜ κ = 2 c .
Revised: For the problem above [Equation] we have κ = 2 cσ 21 σ 22 and ˜ κ = 2 c .

Original: Then we obtain following convergence rates for SGD and ASGD when applied to the above given problem instance:
Revised: Then we obtain following convergence rates for SGD and ASGD when applied to the above given problem instance:

Original: Corollary 1 (of Theorem 1 of Jain et al.
Revised: Corollary 1 (of Theorem 1 of Jain et al.

Original: Let w SGDt be the t th iterate of SGD on the above problem with starting point w 0 and stepsize 1 cσ 21 .
Revised: Let w SGDt be the t th iterate of SGD on the above problem with starting point w 0 and stepsize 1 cσ 21 .

Original: The error of w SGDt can be bounded as,
Revised: The error of w SGDt can be bounded as,

Original: [Equation]
Revised: [Equation]

Original: On the other hand, ASGD achieves the following superior rate.
Revised: On the other hand [Equation] ASGD achieves the following superior rate.

Original: Corollary 2 (of Theorem 1 of Jain et al.
Revised: Corollary 2 (of Theorem 1 of Jain et al.

Original: Let w ASGDt be the t th iterate of ASGD on the above problem with starting point w 0 and appropriate parameters.
Revised: Let w ASGDt be the t th iterate of ASGD on the above problem with starting point w 0 and appropriate parameters.

Original: The error of w ASGDt can be bounded as,
Revised: The error of w ASGDt can be bounded as,

Original: [Equation]
Revised: [Equation]

Original: Input: Initial w 0 , short step δ , long step parameter κ ≥ 1 , statistical advantage parameter ξ ≤ √ κ 1: ¯ w 0 ← w 0 ; t ←/*Set running average to  0 */ 2: α ← 1 − ξκ /*Set momentum value*/ 3: while  converged do 4: ¯ w t +1 ← α · ¯ w [Equation] + (1 − 
Revised: Input: Initial w 0 , short step δ , long step parameter κ ≥ 1 , statistical advantage parameter ξ ≤ √ κ 1: ¯ w 0 ← w 0 ; t ←/*Set running average to w 0 */ 2: α ← 1 − ξκ /*Set momentum value*/ 3: while w t not converged do 4: ¯ w t +1 ← α · ¯ w t + (1 − α ) · (cid:16) w t − κ · δ 0 .

Original: [Equation]

Original: ) · (cid:16) w t − κ · δ 0 .

Original: 7 · (cid:98) ∇ f t ( w t ) (cid:17) /*Update
Revised: 7 · (cid:98) ∇ f t ( w t ) (cid:17) /*Update

Original: the running average as a weighted average of previous running average and a long step gradient */ 5: w t +1 ← 0 .
Revised: the running average as a weighted average of previous running average and a long step gradient */ 5: w t +1 ← 0 .

Original: 7+(1 − α ) · (cid:16) w t − δ · (cid:98) ∇ f t ( w t ) (cid:17) + 1 − α 0 .
Revised: 7+(1 − α ) · (cid:16) w t − δ · (cid:98) ∇ f t ( w t ) (cid:17) + 1 − α 0 .

Original: 7+(1 − α ) · ¯ w t + /*Update the iterate as weighted average of current running average and short step gradient*/ 6: t ← t +
Revised: 7+(1 − α ) · ¯ w t + /*Update the iterate as weighted average of current running average and short step gradient*/ 6: t ← t +

Original: Output: w t /*Return the last iterate*/
Revised: Output: w t /*Return the last iterate*/

Original: Note that  ˜ κ[-= 
-]
Revised: Note that for a given problem/input distribution ˜ κ[-= 
-]

Original:  c is a constant while κ = 2 cσ 21 σ 22 can be arbitrarily large.
Revised: = 2 c is a constant while κ = 2 cσ 21 σ 22 can be arbitrarily large.

Original: Hence, ASGD improves upon rate of SGD by a factor of √ κ .
Revised: Hence, ASGD improves upon rate of SGD by a factor of √ κ .

Original: The following proposition, which is the main result of this section, establishes that HB (Algorithm 1) cannot provide a similar improvement.
Revised: The following proposition, which is the main result of this section, establishes that HB (Algorithm 1) cannot provide a similar improvement over SGD as what ASGD offers.

Original: In fact, we show that despite selecting best parameters for HB, it’s convergence rate is exactly same as that SGD (Corollary 1), up to constants.
Revised: In fact, we show no matter the choice of parameters of HB, its performance does not improve over SGD by more than a constant.

Original: Proposition 3.
Revised: Proposition 3.

Original: Let w HBt be the t th iterate of HB (Algorithm 1) on the above problem with starting point w 0 .
Revised: Let w HBt be the t th iterate of HB (Algorithm 1) on the above problem with starting point w 0 .

Original: For any choice of stepsize δ and momentum α ∈ [0 , 1] , ∃ T large enough such that ∀ t ≥ T , we have,
Revised: For any choice of stepsize δ and momentum α ∈ [0 , 1] , ∃ T large enough such that ∀ t ≥ T , we have,

Original: [Equation]
Revised: [Equation]

Original: where C ( κ, δ, α ) depends on κ, δ and α (but not on t ).
Revised: where C ( κ, δ, α ) depends on κ, δ and α (but not on t ).

Original: That is, to obtain (cid:98) w s.t.
Revised: That is, to obtain (cid:98) w s.t.

Original: (cid:107) (cid:98) w − w ∗ (cid:107) ≤
Revised: (cid:107) (cid:98) w − w ∗ (cid:107) ≤

Original: (cid:15) , HB requires Ω( κ log 1 (cid:15) ) samples and iterations.
Revised: (cid:15) , HB requires Ω( κ log 1 (cid:15) ) samples and iterations.

Original: On the other hand, ASGD can obtain (cid:15) -approximation to w ∗ in O ( √ κ log  1 (cid:15) ) iterations.
Revised: On the other hand, ASGD can obtain (cid:15) -approximation to w ∗ in O ( √ κ log κ log 1 (cid:15) ) iterations.

Original: While we do not prove it theoretically, we observe empirically that for the same problem instance, NAG also obtains nearly same rate as HB and SGD.
Revised: While we do not prove it theoretically, we observe empirically that for the same problem instance, NAG also obtains nearly same rate as HB and SGD.

Revised: We conjecture that a possible lower bound for NAG can be established using a similar proof technique as that of HB (i.e. Proposition 3).

Original:  A LGORITHM
Revised:  A LGORITHM

Original: In this section, we will  present and explain an intuitive version of ASGD. Algorithm 3 presents the pseudocode.
Revised: We will now present and explain an intuitive version of ASGD (pseudo code in Algorithm 3).

Original: The algorithm takes three inputs: short step δ , long step parameter κ and statistical advantage parameter ξ .
Revised: The algorithm takes three inputs: short step δ , long step parameter κ and statistical advantage parameter ξ .

Original: The short step δ is precisely the same as the step size in SGD, HB or NAG.
Revised: The short step δ is precisely the same as the step size in SGD, HB or NAG.

Original: For convex problems, this scales inversely with the smoothness of the function.
Revised: For convex problems, this scales inversely with the smoothness of the function.

Original: The long step parameter κ is intended to give an estimate of the ratio of the largest and smallest curvatures of the function; for convex functions, this is just the condition number.
Revised: The long step parameter κ is intended to give an estimate of the ratio of the largest and smallest curvatures of the function; for convex functions, this is just the condition number.

Original: The statistical advantage parameter ξ captures trade off between statistical and computational condition numbers – in the deterministic case, ξ = √ κ and ASGD is equivalent to NAG, while in the high stochasticity regime, ξ is much smaller.
Revised: The statistical advantage parameter ξ captures trade off between statistical and computational condition numbers – in the deterministic case, ξ = √ κ and ASGD is equivalent to NAG, while in the high stochasticity regime, ξ is much smaller.

Original: The algorithmitself maintains two iterates: descent iterate w t and a running average ¯ w t .
Revised: The algorithm maintains two iterates: descent iterate w t and a running average ¯ w t .

Original: The running average is a weighted average of the previous average and a long gradient step from the descent iterate, while the descent iterate is updated as a convex combination of short gradient step from the descent iterate and the running average.
Revised: The running average is a weighted average of the previous average and a long gradient step from the descent iterate, while the descent iterate is updated as a convex combination of short gradient step from the descent iterate and the running average.

Original: The idea is that since the algorithm takes a long step as well as short step and an appropriate average of both of them, it can make progress on different directions at a similar pace.
Revised: The idea is that since the algorithm takes a long step as well as short step and an appropriate average of both of them, it can make progress on different directions at a similar pace.

Original: Appendix B shows the equivalence between Algorithmand ASGD as proposed in Jain et al.
Revised: Appendix B shows the equivalence between Algorithm 3 and ASGD as proposed in Jain et al.

Original: Note that the constant 0 .
Revised: Note that the constant 0 .

Original: 7 appearing in Algorithmhas no special signiﬁcance.
Revised: 7 appearing in Algorithm 3 has no special signiﬁcance.

Original: Jain et al.
Revised: Jain et al.

Original: (2017) require it to be smaller than (cid:112) 1 / 6 but any constant smaller than 1 seems to work in experiments.
Revised: (2017) require it to be smaller than (cid:112) 1 / 6 but any constant smaller than 1 seems to work in practice.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We now present our experimental results exploring performance of SGD, HB, NAG and ASGD.
Revised: We now present our experimental results exploring performance of SGD, HB, NAG and ASGD.

Original: Our experiments are geared towards answering the following questions
Revised: Our experiments are geared towards answering the following questions:

Original: : • Even for linear regression, is the suboptimality of HB restricted to speciﬁc distributions given in Section 3 or does it hold for more general distributions as well?
Revised: • Even for linear regression, is the suboptimality of HB restricted to speciﬁc distributions given in Section 3 or does it hold for more general distributions as well?

Original: Is the same true of NAG?
Revised: Is the same true of NAG?

Original: [Table]
Revised: [Table]

Original: • What is the reason for the superiority of HB and NAG in practice?
Revised: • What is the reason for the superiority of HB and NAG in practice?

Original: Is it because momentum methods have better performance that SGD for stochastic gradients or due to minibatching?
Revised: Is it because momentum methods have better performance that SGD for stochastic gradients or due to minibatching?

Original: Does this superiority hold even for small minibatches?
Revised: Does this superiority hold even for small minibatches?

Original: • How does the performance of ASGD compare to that of SGD, HB and NAG, especially while training deep networks?
Revised: • How does the performance of ASGD compare to that of SGD, HB and NAG, especially while training deep networks?

Original: Section 5.1 and parts of Section 5.2 address the ﬁrst two questions.
Revised: Section 5.1 and parts of Section 5.2 address the ﬁrst two questions.

Original: Section 5.2 and 5.3 address
Revised: Section 5.2 and 5.3 address

Original: Question 2 partially and the last question.
Revised: Question 2 partially and the last question.

Original: We use Matlab to conduct experiments presented in
Revised: We use Matlab to conduct experiments presented in

Original: Section 5.1 and use PyTorch pyt for our deep networks related experiments.
Revised: Section 5.1 and use PyTorch (pytorch, 2017) for our deep networks related experiments.

Original: 5. L INEAR R EGRESSION
Revised: L INEAR R EGRESSION

Original: In this section, we will present results on performance of the four optimization methods (SGD,
Revised: In this section, we will present results on performance of the four optimization methods (SGD,

Original: HB, NAG, and ASGD) for linear regression problems.
Revised: HB, NAG, and ASGD) for linear regression problems.

Original: We consider two different class of linear regression problems, both of them in two dimensions.
Revised: We consider two different class of linear regression problems, both of them in two dimensions.

Original: Given κ which stands for condition number, we consider the following two distributions:
Revised: Given κ which stands for condition number, we consider the following two distributions:

Original: Discrete : x = e 1 w.p. 0 .
Revised: Discrete : x = e 1 w.p. 0 .

Original: 5 and x = 2 κ · e 2 with 0 . 5 ; e i is the
Revised: 5 and x = 2 κ · e 2 with 0 . 5 ; e i is the

Original: i th standard basis vector.
Revised: i th standard basis vector.

Original: Gaussian : x ∈ R 2 is distributed as a Gaussian random vector with covariance matrix (cid:20) 1 κ (cid:21) .
Revised: Gaussian : x ∈ R 2 is distributed as a Gaussian random vector with covariance matrix (cid:20) [Equation] (cid:21) .

Original: We ﬁx a randomly generated 
Revised: We ﬁx a randomly generated w ∗ ∈ R 2 and for both the distributions above, we let y = (cid:104) w ∗ , x (cid:105) .

Original: We vary κ from { 2 4 , 2 5 , ..., 2 12 } and for each κ in this set, we run 100 independent runs of all four methods, each for a total of t = 5 κ iterations.
Revised: We vary κ from { 2 4 , 2 5 , ..., 2 12 } and for each κ in this set, we run 100 independent runs of all four methods, each for a total of t = 5 κ iterations.

Original: We deﬁne that the algorithm converges if there is no error in the second half (i.e. after 2 .
Revised: We deﬁne that the algorithm converges if there is no error in the second half (i.e. after 2 .

Original: 5 κ updates)
Revised: 5 κ updates)

Original: that exceeds the starting error - this is reasonable since we expect geometric convergence of the initial error.
Revised: that exceeds the starting error - this is reasonable since we expect geometric convergence of the initial error.

Original: Unlike ASGD and SGD, we do not know optimal learning rate and momentum parameters for NAG and HB in the stochastic gradient model.
Revised: Unlike ASGD and SGD, we do not know optimal learning rate and momentum parameters for NAG and HB in the stochastic gradient model.

Original: So, we perform a grid search over the values of the learning rate and momentum parameters.
Revised: So, we perform a grid search over the values of the learning rate and momentum parameters.

Original: In particular, we lay a 10 × 10 grid in [0 , 1] ×
Revised: In particular, we lay a 10 × 10 grid in [0 , 1] ×

Original: [0 , 1] for learning rate and momentum and run NAG and HB.
Revised: [0 , 1] for learning rate and momentum and run NAG and HB.

Original: Then, for each grid point, we consider the subset of 100 trials that converged and computed the ﬁnal error using these.
Revised: Then, for each grid point, we consider the subset of 100 trials that converged and computed the ﬁnal error using these.

Original: Finally, the parameters that yield the minimal error are chosen for NAG and HB, and these numbers are reported.
Revised: Finally, the parameters that yield the minimal error are chosen for NAG and HB, and these numbers are reported.

Original: We measure convergence performance of a method using:
Revised: We measure convergence performance of a method using:

Original: [Equation]
Revised: [Equation]

Original: We compute the rate (1) for all the algorithms with varying condition number κ .
Revised: We compute the rate (1) for all the algorithms with varying condition number κ .

Original: Given a rate vs κ plot for a method, we compute it’s slope (denoted as γ ) using linear regression.
Revised: Given a rate vs κ plot for a method, we compute it’s slope (denoted as γ ) using linear regression.

Original: Table 1 presents the estimated slopes (i.e. γ ) for various methods for both the discrete and the Gaussian case.
Revised: Table 1 presents the estimated slopes (i.e. γ ) for various methods for both the discrete and the Gaussian case.

Original: The slope values clearly show that the rate of SGD, HB and NAG have a nearly linear dependence on κ while that of ASGD seems to scale linearly with √ κ .
Revised: The slope values clearly show that the rate of SGD, HB and NAG have a nearly linear dependence on κ while that of ASGD seems to scale linearly with √ κ .

Original: 5. D EEP A UTOENCODERS FOR MNIST
Revised: 5. D EEP A UTOENCODERS FOR MNIST

Original: In this section, we present experimental results on training deep autoencoders for the mnist dataset, and we closely follow the setup of Hinton & Salakhutdinov (2006).
Revised: In this section, we present experimental results on training deep autoencoders for the mnist dataset, and we closely follow the setup of Hinton & Salakhutdinov (2006).

Original: This problem is a standard benchmark for evaluating the performance of different optimization algorithms e.g., Martens (2010);
Revised: This problem is a standard benchmark for evaluating the performance of different optimization algorithms e.g., Martens (2010);

Original: Sutskever et al.
Revised: Sutskever et al.

Original: ; Martens & Grosse (2015); Reddi et al.
Revised: ; Martens & Grosse (2015); Reddi et al.

Original: The network architecture follows previous work Hinton & Salakhutdinov (2006) and is represented as 784 − 1000 − 500 − 250 − 30 − 250 − 500 − 1000 − 784 with the ﬁrst and last 784 nodes representing the input and output respectively.
Revised: The network architecture follows previous work (Hinton & Salakhutdinov, 2006) and is represented as 784 − 1000 − 500 − 250 − 30 − 250 − 500 − 1000 − 784 with the ﬁrst and last 784 nodes representing the input and output respectively.

Original: All hidden/output nodes employ sigmoid activations except for the layer with 30 nodes which employs linear activations and we use MSE loss.
Revised: All hidden/output nodes employ sigmoid activations except for the layer with 30 nodes which employs linear activations and we use MSE loss.

Original: Initialization follows the scheme of Martens (2010), also employed in Sutskever et al.
Revised: Initialization follows the scheme of Martens (2010), also employed in Sutskever et al.

Original: ; Martens & Grosse (2015).
Revised: ; Martens & Grosse (2015).

Original: We perform training with two minibatch sizes − 1 and 8 .
Revised: We perform training with two minibatch sizes − 1 and 8 .

Original: The runs with minibatch size of 1 were run for 30 epochs while the runs with minibatch size of 8 were run for 50 epochs.
Revised: The runs with minibatch size of 1 were run for 30 epochs while the runs with minibatch size of 8 were run for 50 epochs.

Original: For each of SGD, HB, NAG and ASGD, a grid search over learning rate, momentum and long step parameter (whichever is applicable) was done and best parameters were chosen based on achieving the smallest training error in the same protocol followed by (say,)Sutskever et al.
Revised: For each of SGD, HB, NAG and ASGD, a grid search over learning rate, momentum and long step parameter (whichever is applicable) was done and best parameters were chosen based on achieving the smallest training error in the same protocol followed by Sutskever et al.

Original: The grid was extended whenever the best parameter fell at the edge of a grid.
Revised: The grid was extended whenever the best parameter fell at the edge of a grid.

Original: For the parameters chosen by grid search, we perform 10 runs with different seeds and averaged the results.
Revised: For the parameters chosen by grid search, we perform 10 runs with different seeds and averaged the results.

Original: The results are presented in Figures 2 and 3.
Revised: The results are presented in Figures 2 and 3.

Original: Note that the ﬁnal loss values reported here are suboptimal compared to those in published literature e.g., Sutskever et al.
Revised: Note that the ﬁnal loss values reported here are suboptimal compared to those in published literature e.g., Sutskever et al.

Original: (2013); while Sutskever et al.
Revised: (2013); while Sutskever et al.

Original: (2013) report results after 750000 updates with a large batch size of 200 (which implies a total of 750000 × 200 = 150 M gradient evaluations), whereas, our results are after 1 .
Revised: (2013) report results after 750000 updates with a large batch size of 200 (which implies a total of 750000 × 200 = 150 M gradient evaluations), whereas, our results are after 1 .

Original: 8 M updates of SGD with a batch size 1 (which is just 1 .
Revised: 8 M updates of SGD with a batch size 1 (which is just 1 .

Original: 8 M gradient evaluations).
Revised: 8 M gradient evaluations).

Original: Effect of minibatch sizes : While HB and NAG decay the loss faster compared to SGD for a minibatch size of 8 (Figure 2), this superior decay rate does not hold for a minibatch size of 1 (Figure 3).
Revised: Effect of minibatch sizes : While HB and NAG decay the loss faster compared to SGD for a minibatch size of 8 (Figure 2), this superior decay rate does not hold for a minibatch size of 1 (Figure 3).

Original: This supports our intuitions from the stochastic linear regression setting, where we demonstrate that HB and NAG are suboptimal in the stochastic
Revised: This supports our intuitions from the stochastic linear regression setting, where we demonstrate that HB and NAG are suboptimal in the stochastic

Original: ﬁrst order oracle model.
Revised: ﬁrst order oracle model.

Original: Comparison of ASGD with momentum methods : While ASGD performs slightly better than
Revised: Comparison of ASGD with momentum methods : While ASGD performs slightly better than

Original: NAG for batch size 8 in the training error (Figure 2), ASGD decays the error at a faster rate compared to all the three other methods for a batch size of 1 (Figure 3).
Revised: NAG for batch size 8 in the training error (Figure 2), ASGD decays the error at a faster rate compared to all the three other methods for a batch size of 1 (Figure 3).

Original: 5. D EEP R ESIDUAL N ETWORKS FOR CIFAR- 
Revised: 5. D EEP R ESIDUAL N ETWORKS FOR CIFAR-

Original: In this section, we will  present experimental results on training deep residual networks He et al. (2016b) with pre-activation blocksas introduced in He et al.
Revised: We will now present experimental results on training deep residual networks (He et al., 2016b) with pre-activation blocks He et al.

Original: (2016a)
Revised: (2016a)

Original: for classifying images in cifar10 Krizhevsky & Hinton (2009); the network we use has 44 layers (dubbed preresnet-44).
Revised: for classifying images in cifar-10 (Krizhevsky & Hinton, 2009); the network we use has 44 layers (dubbed preresnet-44).

Original: The code for these experiments was downloaded from pre.
Revised: The code for this section was downloaded from preresnet (2017).

Original: One of the most distinct characteristics of this experiment compared to our previous experiments is learning rate decay.
Revised: One of the most distinct characteristics of this experiment compared to our previous experiments is learning rate decay.

Original: We use a validation set based decay scheme, wherein, after every 3 epochs, we decay the learning rate by a certain factor (which we grid search on) if the validation zero one error does not decrease by at least a certain amount (precise numbers are provided in the appendix since they vary across batch sizes).
Revised: We use a validation set based decay scheme, wherein, after every 3 epochs, we decay the learning rate by a certain factor (which we grid search on) if the validation zero one error does not decrease by at least a certain amount (precise numbers are provided in the appendix since they vary across batch sizes).

Original: Due to space constraints, we present only a subset of training error plots.
Revised: Due to space constraints, we present only a subset of training error plots.

Original: Please see Appendix C.3 for some more plots on training errors.
Revised: Please see Appendix C.3 for some more plots on training errors.

Original: Effect of minibatch sizes : Our ﬁrst experiment tries to understand how the performance of HB and NAG compare with that of SGD and how it varies with minibatch sizes.
Revised: Effect of minibatch sizes : Our ﬁrst experiment tries to understand how the performance of HB and NAG compare with that of SGD and how it varies with minibatch sizes.

Original: Figure 4 presents the test zero one error for minibatch sizes of 8 and 128 .
Revised: Figure 4 presents the test zero one error for minibatch sizes of 8 and 128 .

Original: While training with batch size 8 was done for 40 epochs, with batch size 128 , it was done for 120 epochs.
Revised: While training with batch size 8 was done for 40 epochs, with batch size 128 , it was done for 120 epochs.

Original: We perform a grid search over all parameters for each of these algorithms.
Revised: We perform a grid search over all parameters for each of these algorithms.

Original: See Appendix C.3 for details on the grid search parameters.
Revised: See Appendix C.3 for details on the grid search parameters.

Original: We observe that ﬁnal error achieved by SGD, HB and NAG are all very close for both batch sizes.
Revised: We observe that ﬁnal error achieved by SGD, HB and NAG are all very close for both batch sizes.

Original: While NAG exhibits a superior rate of convergence compared to SGD and HB for batch size 128 , this superior rate of convergence disappears for a batch size of 8 .
Revised: While NAG exhibits a superior rate of convergence compared to SGD and HB for batch size 128 , this superior rate of convergence disappears for a batch size of 8 .

Original: Comparison of ASGD with momentum methods : The next experiment tries to understand how ASGD compares with HB and NAG.
Revised: Comparison of ASGD with momentum methods : The next experiment tries to understand how ASGD compares with HB and NAG.

Original: The errors achieved by various methods when we do grid search over all parameters are presented in Table 2.
Revised: The errors achieved by various methods when we do grid search over all parameters are presented in Table 2.

Original: Note that the ﬁnal test errors for batch size 128 are better than those for batch size 8 since the former was run for 120 epochs while the latter was run only for 40 epochs (due to time constraints). [Figure]
Revised: Note that the ﬁnal test errors for batch size 128 are better than those for batch size 8 since the former was run for 120 epochs while the latter was run only for 40 epochs (due to time constraints). Algorithm

Original: [Table]
Revised: [Table]

Original: While the ﬁnal error achieved by ASGD is similar tothat of all other methods, we are more interested in understanding whether ASGD has a superior convergence speed.
Revised: While the ﬁnal error achieved by ASGD is similar/favorable compared to all other methods, we are also interested in understanding whether ASGD has a superior convergence speed.

Original: In order to do this experiment however, we need to address the issue of different learning rates used by various algorithms and different places where they decay learning rate.
Revised: For this experiment, we need to address the issue of differing learning rates used by various algorithms and different iterations where they decay learning rates.

Original: So, for each of HB and NAG, we choose the learning rate and decay factors by grid search, use these values for ASGD and do grid search only over long step parameter κ and momentum α for ASGD.
Revised: So, for each of HB and NAG, we choose the learning rate and decay factors by grid search, use these values for ASGD and do grid search only over long step parameter κ and momentum α for ASGD.

Original: The results are presented in Figures 5 and 6.
Revised: The results are presented in Figuresand 6.

Original: For batch size 128 , ASGD decays error at a faster rate compared to both HB and NAG.
Revised: For batch size 128 , ASGD decays error at a faster rate compared to both HB and NAG.

Original: For batch size 8 , while we see a superior convergence of ASGD compared to NAG, we do not see this superiority over HB.
Revised: For batch size 8 , while we see a superior convergence of ASGD compared to NAG, we do not see this superiority over HB.

Original: The reason for this turns out to be that the learning rate for HB, which we also use for ASGD, turns out to be quite suboptimal for ASGD.
Revised: The reason for this turns out to be that the learning rate for HB, which we also use for ASGD, turns out to be quite suboptimal for ASGD.

Original: So, for batch size 8 , we also compare fully optimized (i.e., grid search over learning rate as well)
Revised: So, for batch size 8 , we also compare fully optimized (i.e., grid search over learning rate as well)

Original: ASGD with HB.
Revised: ASGD with HB.

Original: The superiority of ASGD over HB is clear from this comparison.
Revised: The superiority of ASGD over HB is clear from this comparison.

Original: These results suggest that ASGD decays error at a faster rate compared to HB and NAG across different batch sizes.
Revised: These results suggest that ASGD decays error at a faster rate compared to HB and NAG across different batch sizes.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: First order oracle methods : The primary method in this family is Gradient Descent (GD) (Cauchy, 1847).
Revised: First order oracle methods : The primary method in this family is Gradient Descent (GD) (Cauchy, 1847).

Original: As mentioned previously, GD is suboptimal for smooth convex optimization (Nesterov, 2004), and this is addressed using momentum methods such as the Heavy Ball method (Polyak, 1964) (for quadratics), and Nesterov’s Accelerated gradient descent (Nesterov, 1983).
Revised: As mentioned previously, GD is suboptimal for smooth convex optimization (Nesterov, 2004), and this is addressed using momentum methods such as the Heavy Ball method (Polyak, 1964) (for quadratics), and Nesterov’s Accelerated gradient descent (Nesterov, 1983).

Original: Stochastic ﬁrst order methods : The simplest method employing the SFO is Stochastic Gradient
Revised: Stochastic ﬁrst order methods : The simplest method employing the SFO is SGD (Robbins &

Original: Descent (Robbins & Monro, 1951)(SGD); the effectiveness of SGD has been immense, and its applicability goes well beyond optimizing convex objectives.
Revised: Monro, 1951); the effectiveness of SGD has been immense, and its applicability goes well beyond optimizing convex objectives.

Original: Accelerating SGD is a tricky proposition given the instability of fast gradient methods in dealing with noise, as evidenced by several negative results which consider both statistical (Proakis, 1974; Polyak, 1987; Roy & Shynk, 1990) and adversarial errors (Devolder et al., 2014).
Revised: Accelerating SGD is a tricky proposition given the instability of fast gradient methods in dealing with noise, as evidenced by several negative results which consider both statistical (Proakis, 1974; Polyak, 1987; Roy & Shynk, 1990) and adversarial errors (d’Aspremont, 2008; Devolder et al., 2014).

Original: A result of Jain et al.
Revised: A result of Jain et al.

Original: (2017) developed the ﬁrst provably accelerated SGD method for linear regression  inspired by a method of Nesterov (2012).
Revised: (2017) developed the ﬁrst provably accelerated SGD method for linear regression which achieved minimax rates, inspired by a method of Nesterov (2012b).

Original: Other schemes such as Ghadimi & Lan (2012; 2013); Dieuleveut et al.
Revised: Schemes of Ghadimi & Lan (2012; 2013); Dieuleveut et al.

Original: (2016), which indicate acceleration  algorithms  are  in 
Revised: (2016), which indicate acceleration is possible with noisy gradients do not hold in the SFO model satisﬁed by algorithms that are run in practice (see Jain et al.

Original: practice

Original: (see Jain

Original: et al for more details).
Revised: (2017) for more details).

Original: While HB (Polyak, 1964) and NAG (Nesterov, 1983) are known to be effective in case of exact ﬁrst order oracle, for the SFO, the theoretical performance of HB and NAG is not well understood.
Revised: While HB (Polyak, 1964) and NAG (Nesterov, 1983) are known to be effective in case of exact ﬁrst order oracle, for the SFO, the theoretical performance of HB and NAG is not well understood.

Revised: Understanding Stochastic Heavy Ball: Understanding HB’s performance with inexact gradients has been considered in efforts spanning several decades, in many communities like controls, optimization and signal processing.

Original: Polyak (1987) describes HBto be rather brittle when provided with noisy gradient estimates.
Revised: Polyak (1987) considered HB with noisy gradients and concluded that the improvements offered by HB with inexact gradients vanish unless strong assumptions on the inexactness was considered; an instance of this is when the variance of inexactness decreased as the iterates approach the minimizer. Proakis (1974); Roy & Shynk (1990); Sharma et al. (1998) suggest that the improved rates offered by stochastic HB arose at the cost of worse asymptotic behavior. We resolve these unquantiﬁed improvements on rates as being just constant factors over SGD, in stark contrast to the gains offered by ASGD.

Revised: Loizou & Richt´arik (2017) state their method as Stochastic

Original: Practical methods for training deep networks : Momentum based methods employed with stochastic gradients (Sutskever et al., 2013) have become standard and very popular in practice.
Revised: HB but require stochastic gradients that nearly behave as exact gradients; indeed, their rates match that of the standard HB method (Polyak, 1964). Such rates are not information theoretically possible (see Jain et al. (2017)), especially with a batch size of 1 or even with constant sized minibatches. Accelerated and Fast Methods for ﬁnite-sums: There have been developments pertaining to faster methods for ﬁnite-sums (also known as ofﬂine stochastic optimization): amongst these are methods such as SDCA (Shalev-Shwartz & Zhang, 2012), SAG (Roux et al., 2012), SVRG (Johnson &

Revised: Zhang, 2013), SAGA (Defazio et al., 2014), which offer linear convergence rates for strongly convex ﬁnite-sums, improving over SGD’s sub-linear rates (Rakhlin et al., 2012).

Revised: These methods have been improved using accelerated variants (Shalev-Shwartz & Zhang, 2014; Frostig et al., 2015a; Lin et al., 2015; Defazio, 2016; Allen-Zhu, 2016).

Revised: Note that these methods require storing the entire training set in memory and taking multiple passes over the same for guaranteed progress.

Revised: Furthermore, these methods require computing a batch gradient or require memory requirements (typically Ω( | training data points | ) ).

Revised: For deep learning problems, data augmentation is often deemed necessary for achieving good performance; this implies computing quantities such as batch gradient (or storage necessities) over this augmented dataset is often infeasible.

Revised: Such requirements are mitigated by the use of simple streaming methods such as SGD, HB, NAG, ASGD.

Revised: For other distinctions between the ofﬂine and online stochastic methods refer to Frostig et al.

Revised: (2015b).

Revised: Practical methods for training deep networks : Momentum based methods employed with stochastic gradients (Sutskever et al., 2013) have become standard and very popular in practice.

Original: These schemes tend to outperform standard SGD on several important practical problems.
Revised: These schemes tend to outperform standard SGD on several important practical problems.

Original: As previously mentioned, we attribute this improvement to effect of minibatching rather than improvement  by HB or NAG when working with stochastic gradients.
Revised: As previously mentioned, we attribute this improvement to effect of minibatching rather than improvement offered by HB or NAG in the SFO model.

Original: Other schemes such as Adagrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2014) represent an important and useful class of algorithms.
Revised: Schemes such as Adagrad (Duchi et al., 2011), RMSProp (Tieleman & Hinton, 2012), Adam (Kingma & Ba, 2014) represent an important and useful class of algorithms.

Original: The advantages offered by these methods are orthogonal to the advantages offered by fast gradient methods; it is an important direction to explore augmenting these methods with ASGD.
Revised: The advantages offered by these methods are orthogonal to the advantages offered by fast gradient methods; it is an important direction to explore augmenting these methods with ASGD as opposed to standard HB or NAG based acceleration schemes. Chaudhari et al.

Revised: (2017) proposed Entropy-SGD, which is an altered objective that adds a local strong convexity term to the actual empirical risk objective, with an aim to improve generalization.

Original: C ONCLUSIONS AND F UTURE D IRECTIONS
Revised: However, we do not understand convergence rates for convex problems or the generalization ability of this technique in a rigorous manner. Chaudhari et al. (2017) propose to use SGD in their procedure but mention that they employ the HB/NAG method in their implementation for achieving better performance. Naturally, we can use ASGD in this context.

Revised: Path normalized SGD (Neyshabur et al., 2015) is a variant of SGD that alters the metric on which the weights are optimized.

Revised: As noted in their paper, path normalized SGD could be improved using HB/NAG (or even the ASGD method).

Revised:  C ONCLUSIONS AND F UTURE D IRECTIONS

Original: In this paper, we show that the performance gain of HB over SGD in stochastic setting is attributable to minibatching rather than the algorithm’s ability to accelerate with stochastic gradients.
Revised: In this paper, we show that the performance gain of HB over SGD in stochastic setting is attributable to minibatching rather than the algorithm’s ability to accelerate with stochastic gradients.

Original: Concretely, we provide a formal proof that for several easy problem instances, HB does not outperform
Revised: Concretely, we provide a formal proof that for several easy problem instances, HB does not outperform

Original: SGD despite large condition number of the problem; we observe this trend for NAG in our experiments.
Revised: SGD despite large condition number of the problem; we observe this trend for NAG in our experiments.

Original: In contrast, ASGD (Jain et al., 2017) provides signiﬁcant improvement over SGD for the same problem instances.
Revised: In contrast, ASGD (Jain et al., 2017) provides signiﬁcant improvement over SGD for these problem instances.

Original: We observe similar trend when training a resnet on cifar-10 and an autoencoder on mnist.
Revised: We observe similar trends when training a resnet on cifar-10 and an autoencoder on mnist.

Original: This work motivates several directions such as understanding the behavior of ASGD onother domains such as NLP,combining ASGD with adagrad (Duchi et al. , 2011)/adam (Kingma & Ba, 2014) andpossibly developing automatic  tuning schemessimilar to (Zhang et al., 2017).
Revised: This work motivates several directions such as understanding the behavior of ASGD on domains such as NLP, and developing automatic momentum tuning schemes (Zhang et al., 2017).

Original: A S UBOPTIMALITY OF HB: P ROOF OF P ROPOSITION
Revised: A S UBOPTIMALITY OF HB: P ROOF OF P ROPOSITION

Original: Before proceeding to the proof, we introduce some additional notation.
Revised: Before proceeding to the proof we introduce some additional notation.

Original: Let θ ( j ) t +1 denote the concatenated and centered estimates in the j th direction for j = 1 , 2 .
Revised: Let θ ( j ) t +1 denote the concatenated and centered estimates in the j th direction for j = 1 , 2 .

Original: [Equation]
Revised: [Equation]

Original: Since the distribution over x is such that the coordinates are decoupled, we see that θ ( j ) t +1 can be written in terms of θ ( j ) t as:θ ( j ) t +1 = (cid:98) A ( j ) t +1 θ ( j ) t , with (cid:98) A ( j ) t +1 =
Revised: Since the distribution over x is such that the coordinates are decoupled we see that θ ( j ) t +1 can be written in terms of θ ( j ) t as:

Original: [-(cid:20) 
-]
Revised: [-(cid:20) 
-][Equation]

Original: + α − δ ( x ( j ) t +1 )
Revised: [Equation] − [Equation]

Original: − α  (cid:21) .
Revised: [Equation]

Original: Let Φ ( j ) t +1 def
Revised: Let Φ ( j ) t +1 def

Original: = E (cid:104) θ ( j ) t +1 ⊗ θ ( j ) t +1 (cid:105) denote the covariance matrix of θ ( j ) t +1 .
Revised: = E (cid:104) θ ( j ) t +1 ⊗ θ ( j ) t +1 (cid:105) denote the covariance matrix of θ ( j ) t +1 .

Original: We have Φ ( j ) t +1 = B ( j ) Φ ( j ) t with, B ( j ) deﬁned asB ( j
Revised: We have Φ ( j ) t +1 = B ( j ) Φ ( j ) t with, B ( j ) deﬁned as

Original: [Equation]
Revised: [Equation]

Original: def =  
Revised: x [Equation] j )

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: − α
Revised: − α

Original: [Equation]
Revised: [Equation]

Original: −
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  
Revised:  

Original: [- =   (1 + α − δσ 2 j ) 
-]
Revised: [- =   (1 + α − δσ 2 j ) 
-][Equation]

Original: + ( c − 1)( δσ 2 j )− α (1 + α − δσ 2 j ) − α (1 + α − δσ 2 j ) α(1 + α − δσ 2 j ) − α (1 + α − δσ 2 j ) − α  
Revised:  

Original:  .
Revised: [Equation]

Original: We prove Proposition 3 by showing that for any choice of stepsize and momentum, either of the two holds:
Revised: We prove Proposition 3 by showing that for any choice of stepsize and momentum, either of the two holds:

Original: • B (1) has an eigenvalue larger than 1 , or, • the largest eigenvalue of B (2) is greater than 1 − 500 κ .
Revised: • B (1) has an eigenvalue larger than 1 , or, • the largest eigenvalue of B (2) is greater than 1 − 500 κ .

Original: This is formalized in the following two lemmas.
Revised: This is formalized in the following two lemmas.

Original: Lemma 4.
Revised: Lemma 4.

Original: If the stepsize δ is such that δσ 21 ≥ 2 ( 1 − α 2 ) c
Revised: If the stepsize δ is such that δσ 21 ≥ 2 ( 1 − α 2 ) c

Original: +( c − 2) α , then B (1) has an eigenvalue ≥ 1 .
Revised: +( c − 2) α , then B (1) has an eigenvalue ≥ 1 .

Original: Lemma 5.
Revised: Lemma 5.

Original: If the stepsize δ is such that δσ 21 < 2 ( 1 − α 2 ) c
Revised: If the stepsize δ is such that δσ 21 < 2 ( 1 − α 2 ) c

Original: +( c − 2) α , then B (2) has an eigenvalue of magnitude ≥ 1 − 500 κ .
Revised: +( c − 2) α , then B (2) has an eigenvalue of magnitude ≥ 1 − 500 κ .

Original: Given this notation, we can now consider the j th dimension without the superscripts; when needed, they will be made clear in the exposition.
Revised: Given this notation, we can now consider the j th dimension without the superscripts; when needed, they will be made clear in the exposition.

Original: Denoting x def = δσ 2 and t def = 

Revised: Denoting x def = δσ 2 and t def =

Original: + α − x , we have:
Revised: + α − x , we have:

Original: [Equation]
Revised: [Equation]

Revised: 

Original: A. P ROOF
Revised: A. P ROOF

Original: The analysis goes via computation of the characteristic polynomial of B and evaluating at different values to obtain bounds on its roots.
Revised: The analysis goes via computation of the characteristic polynomial of B and evaluating at different values to obtain bounds on its roots.

Original: Lemma 6.
Revised: Lemma 6.

Original: The characteristic polynomial of B is:
Revised: The characteristic polynomial of B is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst begin by writing out the expression for the determinant:
Revised: We ﬁrst begin by writing out the expression for the determinant:

Original: [Equation]
Revised: [Equation]

Original: t − z

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: t − α − z    − z (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .

Original: expanding along the ﬁrst column, we have:
Revised: expanding along the ﬁrst column, we have:

Original: [Equation]
Revised: [Equation]

Original: Expanding the terms yields the expression in the lemma.
Revised: Expanding the terms yields the expression in the lemma.

Original: The next corollary follows by some simple arithmetic manipulations.
Revised: The next corollary follows by some simple arithmetic manipulations.

Original: Corollary 7.
Revised: Corollary 7.

Original: Substituting z
Revised: Substituting z

Original: = 1 − τ in the characteristic equation of Lemma 6, we have:
Revised: = 1 − τ in the characteristic equation of Lemma 6, we have:

Original: [Equation]
Revised: [Equation]

Original: Proof of Lemma 4.
Revised: Proof of Lemma 4.

Original: The ﬁrst observation necessary to prove the lemma is that the characteristic polynomial D ( z ) approaches ∞ as z → ∞ , i.e., lim z →∞ D ( z ) =
Revised: The ﬁrst observation necessary to prove the lemma is that the characteristic polynomial D ( z ) approaches ∞ as z → ∞ , i.e., lim z →∞ D ( z ) =

Original: + ∞ .
Revised: + ∞ .

Original: Next, we evaluate the characteristic polynomial at 1 , i.e. compute D (1) .
Revised: Next, we evaluate the characteristic polynomial at 1 , i.e. compute D (1) .

Original: This follows in a straightforward manner from corollary (7) by substituting τ = 0 in equation (2), and this yields,
Revised: This follows in a straightforward manner from corollary (7) by substituting τ = 0 in equation (2), and this yields,

Original: [Equation]
Revised: [Equation]

Original: As α < 1 , x = δσ 2 > 0 , we have the following by setting D (1) ≤ 0 and solving for x :
Revised: As α < 1 , x = δσ 2 > 0 , we have the following by setting D (1) ≤ 0 and solving for x :

Original: [Equation]
Revised: [Equation]

Original: Since D (1) ≤ 0 and D ( z ) ≥ 0 as z → ∞ , there exists a root of D ( · ) which is ≥ 1 .
Revised: Since D (1) ≤ 0 and D ( z ) ≥ 0 as z → ∞ , there exists a root of D ( · ) which is ≥ 1 .

Original: Remark 8.
Revised: Remark 8.

Original: The above characterization is striking in the sense that for any c > 1 , increasing the momentum parameter α naturally requires the reduction in the step size δ to permit the convergence of the algorithm, which is not observed when fast gradient methods are employed in deterministic optimization.
Revised: The above characterization is striking in the sense that for any c > 1 , increasing the momentum parameter α naturally requires the reduction in the step size δ to permit the convergence of the algorithm, which is not observed when fast gradient methods are employed in deterministic optimization.

Original: For instance, in the case of deterministic optimization, setting c = 1 yields δσ 21 < 2(1 + α ) .
Revised: For instance, in the case of deterministic optimization, setting c = 1 yields δσ 21 < 2(1 + α ) .

Original: On the other hand, when employing the stochastic heavy ball method with x ( j ) = 2 σ 2 j , we have the condition that c = 2 , and this implies, δσ 21 < 2(1 − α 2 )
Revised: On the other hand, when employing the stochastic heavy ball method with x ( j ) = 2 σ 2 j , we have the condition that c = 2 , and this implies, δσ 21 < 2(1 − α 2 )

Original: = 1 − α 2 .
Revised: [Equation]

Original: We now prove Lemma 5.
Revised: We now prove Lemma 5.

Original: We ﬁrst consider the large momentum setting.
Revised: We ﬁrst consider the large momentum setting.

Original: Lemma 9.
Revised: Lemma 9.

Original: When the momentum parameter α is set such that 1 − 450 /κ ≤ α ≤ 1 , B has an eigenvalue of magnitude ≥ 1 − 450 κ .
Revised: When the momentum parameter α is set such that 1 − 450 /κ ≤ α ≤ 1 , B has an eigenvalue of magnitude ≥ 1 − 450 κ .

Original: Proof.
Revised: Proof.

Original: This follows easily from the fact that det ( B ) =
Revised: This follows easily from the fact that det ( B ) =

Original: α 4 = (cid:81) 

Revised: α 4 = (cid:81)

Original: j =1 λ j ( B ) ≤ ( λ max ( B )) 4 , thus implying 1 − 450 /κ ≤ α ≤ | λ max ( B ) | .
Revised: j =1 λ j ( B ) ≤ ( λ max ( B )) 4 , thus implying 1 − 450 /κ ≤ α ≤ | λ max ( B ) | .

Original: Remark 10.
Revised: Remark 10.

Original: Note that the above lemma holds for any value of the learning rate δ , and holds for every eigen direction of H .
Revised: Note that the above lemma holds for any value of the learning rate δ , and holds for every eigen direction of H .

Original: Thus, for “large” values of momentum, the behavior of stochastic heavy ball does degenerate to the behavior of stochastic gradient descent.
Revised: Thus, for “large” values of momentum, the behavior of stochastic heavy ball does degenerate to the behavior of stochastic gradient descent.

Original: We now consider the setting where momentum is bounded away from 1 .
Revised: We now consider the setting where momentum is bounded away from 1 .

Original: Corollary 11.
Revised: Corollary 11.

Original: Consider B (2) , by substituting τ = l/κ , x = δλ min =
Revised: Consider B (2) , by substituting τ = l/κ , x = δλ min =

Original: c ( δσ 21 ) /κ
Revised: c ( δσ 21 ) /κ

Original: in equation (2) and accumulating terms in varying powers of 1 /κ , we obtain:
Revised: in equation (2) and accumulating terms in varying powers of 1 /κ , we obtain:

Original: [Equation]
Revised: [Equation]

Original: Lemma 12.
Revised: Lemma 12.

Original: Let 2 < c < 3000 , 0 ≤ α ≤ 1 − 450 κ , l = 1 + 2 c ( δσ 21 ) 1 − α .
Revised: Let 2 < c < 3000 , 0 ≤ α ≤ 1 − 450 κ , l = 1 + 2 c ( δσ 21 ) 1 − α .

Original: Then, G ( l ) ≤ 0 .
Revised: Then, G ( l ) ≤ 0 .

Original: Proof.
Revised: Proof.

Original: Since ( δσ 21 ) ≤ 2(1 − α 2 ) c
Revised: Since ( δσ 21 ) ≤ 2(1 − α 2 ) c

Original: +( c − 2) α , this implies ( δσ 21 ) 1 − α ≤ 2(1+ α ) c
Revised: +( c − 2) α , this implies ( δσ 21 ) 1 − α ≤ 2(1+ α ) c

Original: +( c − 2) α ≤ 4 c , thus implying, 1 ≤ l ≤ 9 .
Revised: +( c − 2) α ≤ 4 c , thus implying, 1 ≤ l ≤ 9 .

Original: Substituting the value of l in equation (3), the coefﬁcient of O (1 /κ ) is − (1 − α ) 3 (1 + α ) .
Revised: Substituting the value of l in equation (3), the coefﬁcient of O (1 /κ ) is − (1 − α ) 3 (1 + α ) .

Original: We will bound this term along with (3 − 4 α − α 

Revised: We will bound this term along with (3 − 4 α − α

Original: + 2 α 3 )
Revised: + 2 α 3 )

Original: l 2 /κ 

Revised: l 2 /κ

Original: =
Revised: =

Original: (1 − α ) 2 (3 + 2 α ) l 2 /κ 2 to obtain: −(
Revised: (1 − α ) 2 (3 + 2 α ) l 2 /κ 2 to obtain: −

Original: [Equation]
Revised: [Equation]

Original: where
Revised: +

Original: [Equation]
Revised: [Equation]

Original:  we use the fact that α < [Equation] l ≤ 9 .
Revised: where, we use the fact that α < 1 , l ≤ 9 .

Original: The natural implication of this bound is that the terms that are lower order, such as O (1 /κ 4 ) and O (1 /κ 5 ) will be negative owing to the large constant above.
Revised: The natural implication of this bound is that the terms that are lower order, such as O (1 /κ 4 ) and O (1 /κ 5 ) will be negative owing to the large constant above.

Original: Let us verify that this is indeed the case by considering the terms having powers of O (1 /κ 4 ) and
Revised: Let us verify that this is indeed the case by considering the terms having powers of O (1 /κ 4 ) and

Original: O (1 /κ 5 ) from equation (3):
Revised: O (1 /κ 5 ) from equation (3):

Original: [Equation]
Revised: [Equation]

Original: The expression above evaluates to ≤ 0 given an upperbound on the value of c .
Revised: The expression above evaluates to ≤ 0 given an upperbound on the value of c .

Original: The expression above follows from the fact that l ≤ 9 , κ ≥ 1 .
Revised: The expression above follows from the fact that l ≤ 9 , κ ≥ 1 .

Original: Next, consider the terms involving O (1 /κ 3 ) and O (1 /κ 2 ) , in particular,
Revised: Next, consider the terms involving O (1 /κ 3 ) and O (1 /κ 2 ) , in particular,

Original: [Equation]
Revised: [Equation]

Original: κ − (1 − α )

Original: ( c + ( c − 2) α ) (cid:1) ≤ c 2 ( δσ 21 )κ(cid:0) 5 cl κ − (1 − α )

Original: ( c + ( c − 2) α ) (cid:1) ≤ c 2 ( δσ 2 1 )κ(cid:0) 5 cl κ − (1 − α ) c (cid:1) ≤ c 3 ( δσ 21 )κ(cid:0) 5 l κ − κ (cid:1) ≤ c 3 ( δσ 21 )κ· −κ ≤ 0 .

Original: Next,
Revised: Next,

Original: [Equation]
Revised: [Equation]

Original: In both these cases, we used the fact that α ≤ 1 − 450 κ implying − (1 − α ) ≤ − 450 κ .
Revised: In both these cases, we used the fact that α ≤ 1 − 450 κ implying − (1 − α ) ≤ − 450 κ .

Original: Finally, other remaining terms are negative.
Revised: Finally, other remaining terms are negative.

Original: We are now ready to prove Lemma 5.
Revised: We are now ready to prove Lemma 5.

Original: Proof of Lemma 5.
Revised: Proof of Lemma 5.

Original: Combining Lemmas 9 and 12, we see that no matter what stepsize and momentum we choose, B ( j ) has an eigenvalue of magnitude at least 1 − 500 κ for some j ∈ { 1 , 2 } .
Revised: Combining Lemmas 9 and 12, we see that no matter what stepsize and momentum we choose, B ( j ) has an eigenvalue of magnitude at least 1 − 500 κ for some j ∈ { 1 , 2 } .

Original: This proves the lemma.
Revised: This proves the lemma.

Original: B E QUIVALENCE OF A LGORITHM 3 AND ASGD
Revised: B E QUIVALENCE OF A LGORITHM 3 AND ASGD

Original: We begin by writing out the updates of ASGD as written out in Jain et al.
Revised: We begin by writing out the updates of ASGD as written out in Jain et al.

Original: (2017), which starts with two iterates a 0 and d 0 , and from time t = 0 , 1 , ...
Revised: (2017), which starts with two iterates a 0 and d 0 , and from time t = 0 , 1 , ...

Original: T − 1 implements the following updates:
Revised: T − 1 implements the following updates:

Original: [Equation]
Revised: [Equation]

Original: Next, we specify the step sizes β 1 = c 23 / √ κ [Equation] κ , α 1 = c 3 / ( c 3 + β ) , γ 1 = β/ ( c 3 λ min ) and δ 1 = 1 /R 2 , where κ = R 2 /λ min .
Revised: Next, we specify the step sizes β 1 = c 23 / √ κ (cid:101) κ , α 1 = c 3 / ( c 3 + β ) , γ 1 = β/ ( c 3 λ min ) and δ 1 = 1 /R 2 , where κ = R 2 /λ min .

Original: Note that the step sizes in the paper of Jain et al.
Revised: Note that the step sizes in the paper of Jain et al.

Original: (2017) with c 1 in their paper set to 1 yields the step sizes above.
Revised: (2017) with c 1 in their paper set to 1 yields the step sizes above.

Original: Now, substituting equation 6 in equation 7 and substituting the value of γ 1 , we have:
Revised: Now, substituting equation 6 in equation 7 and substituting the value of γ 1 , we have:

Original: [Equation]
Revised: [Equation]

Original: We see that d t +1 is precisely the update of the running average ¯ w t +1 in the ASGD method employed in this paper.
Revised: We see that d t +1 is precisely the update of the running average ¯ w t +1 in the ASGD method employed in this paper.

Original: We now update b t to become b t +

Revised: We now update b t to become b t +

Original: and this can be done by writing out equation 4 at t + 1 , i.e:
Revised: and this can be done by writing out equation 4 at t + 1 , i.e:

Original: [Equation]
Revised: [Equation]

Original: By substituting the value of α 1 we note that this is indeed the update of the iterate as a convex combination of the current running average and a short gradient step as written in this paper.
Revised: By substituting the value of α 1 we note that this is indeed the update of the iterate as a convex combination of the current running average and a short gradient step as written in this paper.

Original: In this paper, we set c 3 to be equal to 0 .
Revised: In this paper, we set c 3 to be equal to 0 .

Original: 7 , and any constant less than 1 works.
Revised: 7 , and any constant less than 1 works.

Original: In terms of variables, we note that α in this paper’s algorithm description maps to 1 − β 1 .
Revised: In terms of variables, we note that α in this paper’s algorithm description maps to 1 − β 1 .

Original: C M ORE DETAILS ON EXPERIMENTS
Revised: C M ORE DETAILS ON EXPERIMENTS

Original: In this section, we will present more details on our experimental setup.
Revised: In this section, we will present more details on our experimental setup.

Original: C. L INEAR R EGRESSION
Revised: C. L INEAR R EGRESSION

Original: In this section, we will present some more results on our experiments on the linear regression problem.
Revised: In this section, we will present some more results on our experiments on the linear regression problem.

Original: Just as in Appendix A, it is indeed possible to compute the expected error of all the algorithms among SGD, HB, NAG and ASGD, by tracking certain covariance matrices which evolve as linear systems.
Revised: Just as in Appendix A, it is indeed possible to compute the expected error of all the algorithms among SGD, HB, NAG and ASGD, by tracking certain covariance matrices which evolve as linear systems.

Original: For SGD, for instance, denoting Φ SGDt def = E (cid:2)(cid:0) w SGDt − w ∗ (cid:1) ⊗ (cid:0) w SGDt − w ∗ (cid:1)(cid:3) , we see that Φ SGDt + = B ◦ Φ SGDt , where B is a linear operator acting on d × d matrices such that
Revised: For SGD, for instance, denoting Φ SGDt def = E (cid:2)(cid:0) [Equation] , we see that Φ SGDt + = B ◦ Φ SGDt , where B is a linear operator acting on d × d matrices such that

Original: B ◦ M def =
Revised: B ◦ M def =

Original: M − [Equation] − δMH + δ 2 E (cid:2) (cid:104) x, Mx (cid:105) xx (cid:62) (cid:3) .
Revised: M − δHM − δMH + δ 2 E (cid:2) (cid:104) x, Mx (cid:105) xx (cid:62) (cid:3) .

Original: Similarly, HB, NAG and ASGD also have corresponding operators (see Appendix A for more details on the operator corresponding to HB).
Revised: Similarly, HB, NAG and ASGD also have corresponding operators (see Appendix A for more details on the operator corresponding to HB).

Original: The largest magnitude of the eigenvalues of these matrices indicate the rate of decay achieved by the particular algorithm – smaller it is compared to 1 , faster the decay.
Revised: The largest magnitude of the eigenvalues of these matrices indicate the rate of decay achieved by the particular algorithm – smaller it is compared to 1 , faster the decay.

Original: We now detail the range of parameters explored for these results: the condition number κ was varied from { 2 4 , 2 5 , .., 2 28 } for all the optimization methods and for both the discrete and gaussian problem.
Revised: We now detail the range of parameters explored for these results: the condition number κ was varied from { 2 4 , 2 5 , .., 2 28 } for all the optimization methods and for both the discrete and gaussian problem.

Original: For each of these experiments, we draw 1000 samples and compute the empirical estimate of the fourth moment tensor.
Revised: For each of these experiments, we draw 1000 samples and compute the empirical estimate of the fourth moment tensor.

Original: For NAG and HB, we did a very ﬁne grid search by samplingvalues in the interval (0 , 1] for both the learning rate and the momentum parameter and chose the parameter setting that yielded the smallest λ max ( B ) that is less than 1 (so that it falls in the range of convergence of the algorithm).
Revised: For NAG and HB, we did a very ﬁne grid search by samplingvalues in the interval (0 , 1] for both the learning rate and the momentum parameter and chose the parameter setting that yielded the smallest λ max ( B ) that is less than 1 (so that it falls in the range of convergence of the algorithm).

Original: As for SGD and ASGD, we employed a learning rate of 1 / 3 for the Gaussian case and a step size of 0 .
Revised: As for SGD and ASGD, we employed a learning rate of 1 / 3 for the Gaussian case and a step size of 0 .

Original: 9 for the discrete case.
Revised: 9 for the discrete case.

Original: The statistical advantage parameter of
Revised: The statistical advantage parameter of

Original: ASGD was chosen to be (cid:112) 3 κ/ 2 for the Gaussian case and (cid:112) 2 κ/ 3 for the Discrete case, and the a long step parameters of 3 κ and 2 κ were chosen for the Gaussian and Discrete case respectively.
Revised: ASGD was chosen to be (cid:112) 3 κ/ 2 for the Gaussian case and (cid:112) 2 κ/ 3 for the Discrete case, and the a long step parameters of 3 κ and 2 κ were chosen for the Gaussian and Discrete case respectively.

Original: The reason it appears as if we choose a parameter above the theoretically maximal allowed value of the advantage parameter is because the deﬁnition of κ is different in this case.
Revised: The reason it appears as if we choose a parameter above the theoretically maximal allowed value of the advantage parameter is because the deﬁnition of κ is different in this case.

Original: The κ we speak about for this experiment is λ max /λ min unlike the condition number for the stochastic optimization problem.
Revised: The κ we speak about for this experiment is λ max /λ min unlike the condition number for the stochastic optimization problem.

Original: In a manner similar to actually running the algorithms (the results of whose are presented in the main paper), we also note that we can compute the rate as in equation 1 and join all these rates using a curve and estimate its slope (in the log scale).
Revised: In a manner similar to actually running the algorithms (the results of whose are presented in the main paper), we also note that we can compute the rate as in equation 1 and join all these rates using a curve and estimate its slope (in the log scale).

Original: This result is indicated in table 3.
Revised: This result is indicated in table 3.

Original: Figure 7 presents these results, where for each method, we did grid search over all parameters and chose parameters that give smallest λ max .
Revised: Figure 7 presents these results, where for each method, we did grid search over all parameters and chose parameters that give smallest λ max .

Original: We see the same pattern as in Figure 1 from actual runs – SGD,HB and NAG all have linear dependence on condition number κ , while ASGD has a dependence of √ κ .
Revised: We see the same pattern as in Figure 1 from actual runs – SGD,HB and NAG all have linear dependence on condition number κ , while ASGD has a dependence of √ κ .

Original: [Table]
Revised: [Table]

Original: C
Revised: C. A UTOENCODERS FOR MNIST

Original: A UTOENCODERS FOR MNIST

Original: We begin by noting that the learning rates tend to vary as we vary batch sizes, which is something that is known in theory (Jain et al., 2016).
Revised: We begin by noting that the learning rates tend to vary as we vary batch sizes, which is something that is known in theory (Jain et al., 2016).

Original: Furthermore, we extend the grid especially whenever our best parameters of a baseline method tends to land at the edge of a grid.
Revised: Furthermore, we extend the grid especially whenever our best parameters of a baseline method tends to land at the edge of a grid.

Original: The parameter ranges explored by our grid search are:
Revised: The parameter ranges explored by our grid search are:

Original: Batch Size 1 : (parameters chosen by running for 20 epochs) • SGD: learning rate: { 0 . 01 , 0 . 01 √ 10 , 0 . 1 , 0 . 1 √ 10 , 1 , √ 10 , 5 , 10 , 20 , 10 √ 10 , 40 , 60 , 80 , 100 .
Revised: Batch Size 1 : (parameters chosen by running for 20 epochs) • SGD: learning rate: { 0 . 01 , 0 . 01 √ 10 , 0 . 1 , 0 . 1 √ 10 , 1 , √ 10 , 5 , 10 , 20 , 10 √ 10 , 40 , 60 , 80 , 100 .

Original: • NAG/HB: learning rate: { 0 . 01 √ 10 , 0 . 1 , 0 .
Revised: • NAG/HB: learning rate: { 0 . 01 √ 10 , 0 . 1 , 0 .

Original: 1 √ 10 , 1 , √ 10 , 10 } , momentum 
Revised: 1 √ 10 , 1 , √ 10 , 10 } , momentum { 0 , 0 . 5 , 0 .

Original: • ASGD:
Revised: • ASGD:

Original: learning rate: { 2 . 5 , 5 } , long step { 100 . 0 , 1000 . 0 } , advantage parameter { 2 . 5 , 5 . 0 , 10 . 0 , 20 . 0 } .
Revised: learning rate: { 2 . 5 , 5 } , long step { 100 . 0 , 1000 . 0 } , advantage parameter { 2 . 5 , 5 . 0 , 10 . 0 , 20 . 0 } .

Original: Batch Size 8 : (parameters chosen by running for 50 epochs) • SGD: learning rate: { 0 .
Revised: Batch Size 8 : (parameters chosen by running for 50 epochs) • SGD: learning rate: { 0 .

Original: 001 √ 10 . 0 , 0 . 01 , 0 . 01 √ 10 , 0 .
Revised: 001 √ 10 . 0 , 0 . 01 , 0 . 01 √ 10 , 0 .

Original: 1 , 0 . 1 √ 10 , 1 , √ 10 , 5 ,, 10 √ 10 , 40 , 60 , 80 , 100 , 120 , 140 } .
Revised: 1 , 0 . 1 √ 10 , 1 , √ 10 , 5 ,, 10 √ 10 , 40 , 60 , 80 , 100 , 120 , 140 } .

Original: • NAG/HB: learning rate: { 5 . 0 , 10 . 0 , 20 . 0 , 10 √ 10 , 40 , 60 } , momentum { 0 , 0 . 25 , 0 . 5 , 0 .
Revised: • NAG/HB: learning rate: { 5 . 0 , 10 . 0 , 20 . 0 , 10 √ 10 , 40 , 60 } , momentum { 0 , 0 . 25 , 0 . 5 , 0 .

Original: • ASGD:
Revised: • ASGD:

Original: learning rate { 40 , 60 } .
Revised: learning rate { 40 , 60 } .

Original: For a long step of 100 , swept over advantage parameters of { 1 . 5 , 2 , 2 . 5 , 5 , 10 , 20 } .
Revised: For a long step of 100 , swept over advantage parameters of { 1 . 5 , 2 , 2 . 5 , 5 , 10 , 20 } .

Original: For a long step of 1000 , we swept over advantage parameters of { 2 . 5 , 5 , 10 } .
Revised: For a long step of 1000 , we swept over advantage parameters of { 2 . 5 , 5 , 10 } .

Original: C. D EEP R ESIDUAL N ETWORKS FOR CIFAR- 
Revised: C. D EEP R ESIDUAL N ETWORKS FOR CIFAR-

Original: In this section, we will provide more details on our experiments on cifar-10, as well as present some additional results.
Revised: In this section, we will provide more details on our experiments on cifar-10, as well as present some additional results.

Original: We used a weight decay of 0 .
Revised: We used a weight decay of 0 .

Original: in all our experiments.
Revised: in all our experiments.

Original: The grid search parameters we used for various algorithms are as follows.
Revised: The grid search parameters we used for various algorithms are as follows.

Original: Note that the ranges in which parameters such as learning rate need to be searched differ based on batch size (Jain et al., 2016).
Revised: Note that the ranges in which parameters such as learning rate need to be searched differ based on batch size (Jain et al., 2016).

Original: Furthermore, we tend to extrapolate the grid search whenever a parameter (except for the learning rate decay factor) at the edge of the grid has been chosen; this is done so that we always tend to lie in the interior of the grid that we have searched on.
Revised: Furthermore, we tend to extrapolate the grid search whenever a parameter (except for the learning rate decay factor) at the edge of the grid has been chosen; this is done so that we always tend to lie in the interior of the grid that we have searched on.

Original: Note that for the purposes of the grid search, we choose a hold out set from the training data and add it in to the training data after the parameters are chosen, for the ﬁnal run.
Revised: Note that for the purposes of the grid search, we choose a hold out set from the training data and add it in to the training data after the parameters are chosen, for the ﬁnal run.

Original: Batch Size 8 : Note: (i) parameters chosen by running for 40 epochs and picking the grid search parameter that yields the smallest validation 0 / 1 error.
Revised: Batch Size 8 : Note: (i) parameters chosen by running for 40 epochs and picking the grid search parameter that yields the smallest validation 0 / 1 error.

Original: (ii)
Revised: (ii)

Original: The validation set decay scheme that we use is that if the validation error does not decay by at least 1% every three passes over the data, we cut the learning rate by a constant factor (which is grid searched as described below).
Revised: The validation set decay scheme that we use is that if the validation error does not decay by at least 1% every three passes over the data, we cut the learning rate by a constant factor (which is grid searched as described below).

Original: The minimal learning rate to use is ﬁxed to be 6 .
Revised: The minimal learning rate to use is ﬁxed to be 6 .

Original: 25 × 10 − 5 , so that we do not decay far too many times and curtail progress prematurely.
Revised: 25 × 10 − 5 , so that we do not decay far too many times and curtail progress prematurely.

Original: • SGD: learning rate: { 0 . 0033 , 0 . 01 , 0 . 033 , 0 . 1 , 0 .
Revised: • SGD: learning rate: { 0 . 0033 , 0 . 01 , 0 . 033 , 0 . 1 , 0 .

Original: 33 } , learning rate decay factor { 5 , 10 } .
Revised: 33 } , learning rate decay factor { 5 , 10 } .

Original: • NAG/HB: learning rate: { 0 . [Table] [Table] [Table]
Revised: • NAG/HB: learning rate: { 0 . 001 , 0 . 0033 , 0 . 01 , 0 . 033 } , momentum { 0 . 8 , 0 . 9 , 0 .

Original: . 033 } , momentum { 0 . 8 , 0 . 9 , 0 .

Original: 95 , 0 . 97 } , learning rate decay factor { 5 , 10 } .
Revised: 95 , 0 . 97 } , learning rate decay factor { 5 , 10 } .

Original: • ASGD:
Revised: • ASGD:

Original: learning rate { 0 . 01 , 0 . 0330 , 0 . 1 } , long step { 1000 , 10000 , 50000 } , advantage parameter { 5 , 10 } , learning rate decay factor { 5 , 10 } .
Revised: learning rate { 0 . 01 , 0 . 0330 , 0 . 1 } , long step { 1000 , 10000 , 50000 } , advantage parameter { 5 , 10 } , learning rate decay factor { 5 , 10 } .

Original: Batch Size 128 :
Revised: Batch Size 128 :

Original: Note: (i) parameters chosen by running for 120 epochs and picking the grid search parameter that yields the smallest validation 0 / 1 error.
Revised: Note: (i) parameters chosen by running for 120 epochs and picking the grid search parameter that yields the smallest validation 0 / 1 error.

Original: (ii)
Revised: (ii)

Original: The validation set decay scheme that we use is that if the validation error does not decay by at least 0 .
Revised: The validation set decay scheme that we use is that if the validation error does not decay by at least 0 .

Original: 2% every four passes over the data, we cut the learning rate by a constant factor (which is grid searched as describedbelow).
Revised: 2% every four passes over the data, we cut the learning rate by a constant factor ( which is grid searched as described

Original:  The minimal learning rate to use is ﬁxed to be 1 ×10 − 3 , so that we do not decay far too many times and curtail progress prematurely.
Revised: below). The minimal learning rate to use is ﬁxed to be 1 × − 3 , so that we do not decay far too many times and curtail progress prematurely

Original: • SGD: learning rate: { 0 . 01 , 0 .  27 , 0 . 81 } , learning rate decay factor { 2 , √ 10 , 5 } .
Revised: • SGD: learning rate: { 0 . 01 , 0 . 03 , 0 . 09 , 0 . 27 , 0 . 81 } , learning rate decay factor { 2 , √ 10 , 5 } .

Original: • NAG/HB: learning rate: { 0 . 01 , 0 . 03 , 0 . 09 , 0 . 27 } , momentum { 0 . 5 , 0 . 8 , 0 . 9 , 0 .
Revised: • NAG/HB: learning rate: { 0 . 01 , 0 . 03 , 0 . 09 , 0 . 27 } , momentum { 0 . 5 , 0 . 8 , 0 . 9 , 0 .

Original: 95 , 0 . 97 } , learning rate decay factor { 2 , √ 10 , 5 } .
Revised: 95 , 0 . 97 } , learning rate decay factor { 2 , √ 10 , 5 } .

Original: • ASGD:
Revised: • ASGD:

Original: learning rate { 0 . 01 , 0 . 03 , 0 . 09 , 0 . 27 } , long step { 100 , 1000 , 10000 } , advantage parameter { 5 , 10 , 20 } , learning rate decay factor { 2 , √ 10 , 5 } .
Revised: learning rate { 0 . 01 , 0 . 03 , 0 . 09 , 0 . 27 } , long step { 100 , 1000 , 10000 } , advantage parameter { 5 , 10 , 20 } , learning rate decay factor { 2 , √ 10 , 5 } .

Original: As a ﬁnal remark, for any comparison across algorithms, such as, (i) ASGD vs. NAG, (ii) ASGD vs HB, we ﬁx the starting learning rate, learning rate decay factor and decay schedule chosen by the best grid search run of NAG/HB respectively and perform a grid search over the long step and advantage parameter of ASGD.
Revised: As a ﬁnal remark, for any comparison across algorithms, such as, (i) ASGD vs. NAG, (ii) ASGD vs HB, we ﬁx the starting learning rate, learning rate decay factor and decay schedule chosen by the best grid search run of NAG/HB respectively and perform a grid search over the long step and advantage parameter of ASGD.

Original: In a similar manner, when we compare (iii) SGD vs NAG or, (iv)
Revised: In a similar manner, when we compare (iii) SGD vs NAG or, (iv)

Original: SGD vs. HB, we choose the learning rate, learning rate decay factor and decay schedule of SGD and simply sweep over the momentum parameter of NAG or HB and choose the momentum that offers the best validation error.
Revised: SGD vs. HB, we choose the learning rate, learning rate decay factor and decay schedule of SGD and simply sweep over the momentum parameter of NAG or HB and choose the momentum that offers the best validation error.

Original: We now present plots of training function value for different algorithms and batch sizes.
Revised: We now present plots of training function value for different algorithms and batch sizes.

Original: Effect of minibatch sizes : Figure 8 plots training function value for batch sizes of 128 andfor SGD, HB and NAG.
Revised: Effect of minibatch sizes : Figure 8 plots training function value for batch sizes of 128 andfor SGD, HB and NAG.

Original: We notice that in the initial stages of training, NAG obtains substantial improvements compared to SGD and HB for batch size 128 but not for batch size 8 .
Revised: We notice that in the initial stages of training, NAG obtains substantial improvements compared to SGD and HB for batch size 128 but not for batch size 8 .

Original: Towards the end of training however, NAG starts decreasing the training function value rapidly for both the batch sizes.
Revised: Towards the end of training however, NAG starts decreasing the training function value rapidly for both the batch sizes.

Original: The reason for this phenomenon is not clear.
Revised: The reason for this phenomenon is not clear.

Original: Note however, that at this point, the test error has already stabilized and the algorithms are just overﬁtting to the data.
Revised: Note however, that at this point, the test error has already stabilized and the algorithms are just overﬁtting to the data.

Original: Comparison of ASGD with momentum methods : We now present the training error plots for ASGD compared to HB and NAG in Figures 9 and 10 respectively.
Revised: Comparison of ASGD with momentum methods : We now present the training error plots for ASGD compared to HB and NAG in Figures 9 and 10 respectively.

Original: As mentioned earlier, in order to see a clear trend, we constrain the learning rate and decay schedule of ASGD to be the same as that of HB and NAG respectively, which themselves were learned using grid search.
Revised: As mentioned earlier, in order to see a clear trend, we constrain the learning rate and decay schedule of ASGD to be the same as that of HB and NAG respectively, which themselves were learned using grid search.

Original: We see similar trends as in the validation error plots from Figures 5 and 6.
Revised: We see similar trends as in the validation error plots from Figures 5 and 6.

Original: Please see the ﬁgures and their captions for more details.
Revised: Please see the ﬁgures and their captions for more details.