Original: Y ELLOW F IN AND THE A RT OF M OMENTUM T UNING
Revised: Y ELLOW F IN AND THE A RT OF M OMENTUM T UNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Hyperparameter tuning is one of the most time-consuming workloads in deep learning.
Revised: Hyperparameter tuning is one of the most time-consuming workloads in deep learning.

Original: State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable.
Revised: State-of-the-art optimizers, such as AdaGrad, RMSProp and Adam, reduce this labor by adaptively tuning an individual learning rate for each variable.

Original: Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results.
Revised: Recently researchers have shown renewed interest in simpler methods like momentum SGD as they may yield better results.

Original: Motivated by this trend, we ask: can simple adaptive methods based on SGD perform as well or better?
Revised: Motivated by this trend, we ask: can simple adaptive methods based on SGD perform as well or better?

Original: We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.
Revised: We revisit the momentum SGD algorithm and show that hand-tuning a single learning rate and momentum makes it competitive with Adam.

Original: We then analyze its robustness to learning rate misspeciﬁcation and objective curvature variation.
Revised: We then analyze its robustness to learning rate misspeciﬁcation and objective curvature variation.

Original: Based on these insights, we design Y ELLOW F IN , an automatic tuner for momentum and learning rate in SGD.
Revised: Based on these insights, we design Y ELLOW F IN , an automatic tuner for momentum and learning rate in SGD.

Original: Y ELLOW F IN optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the ﬂy.
Revised: Y ELLOW F IN optionally uses a negative-feedback loop to compensate for the momentum dynamics in asynchronous settings on the ﬂy.

Original: We empirically show Y ELLOW F IN can converge in fewer iterations than Adam on ResNets and
Revised: We empirically show Y ELLOW F IN can converge in fewer iterations than Adam on ResNets and

Original: LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to 3 .
Revised: LSTMs for image recognition, language modeling and constituency parsing, with a speedup of up to 3 .

Original: 28 x in synchronous and up to 2 .
Revised: 28 x in synchronous and up to 2 .

Original: 69 x in asynchronous settings.
Revised: 69 x in asynchronous settings.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Accelerated forms of stochastic gradient descent (SGD), pioneered by Polyak (1964) and Nesterov (1983), are the de-facto training algorithms for deep learning.
Revised: Accelerated forms of stochastic gradient descent (SGD), pioneered by Polyak (1964) and Nesterov (1983), are the de-facto training algorithms for deep learning.

Original: Their use requires a sane choice for their hyperparameters : typically a learning rate and momentum parameter (Sutskever et al., 2013).
Revised: Their use requires a sane choice for their hyperparameters : typically a learning rate and momentum parameter (Sutskever et al., 2013).

Original: However, tuning hyperparameters is arguably the most time-consuming part of deep learning, with many papers outlining best tuning practices written (Bengio, 2012;
Revised: However, tuning hyperparameters is arguably the most time-consuming part of deep learning, with many papers outlining best tuning practices written (Bengio, 2012;

Original: Orr and Müller, 2003; Bengio et al., 2012; Bottou, 2012).
Revised: Orr and Müller, 2003; Bengio et al., 2012; Bottou, 2012).

Original: Deep learning researchers have proposed a number of methods to deal with hyperparameter optimization, ranging from grid-search and smart black-box methods (Bergstra and Bengio, 2012; Snoek et al., 2012) to adaptive optimizers.
Revised: Deep learning researchers have proposed a number of methods to deal with hyperparameter optimization, ranging from grid-search and smart black-box methods (Bergstra and Bengio, 2012; Snoek et al., 2012) to adaptive optimizers.

Original: Adaptive optimizers aim to eliminate hyperparameter search by tuning on the ﬂy for a single training run: algorithms like AdaGrad (Duchi et al., 2011), RMSProp (Tieleman and Hinton, 2012) and Adam (Kingma and Ba, 2014) use the magnitude of gradient elements to tune learning rates individually for each variable and have been largely successful in relieving practitioners of tuning the learning rate.
Revised: Adaptive optimizers aim to eliminate hyperparameter search by tuning on the ﬂy for a single training run: algorithms like AdaGrad (Duchi et al., 2011), RMSProp (Tieleman and Hinton, 2012) and Adam (Kingma and Ba, 2014) use the magnitude of gradient elements to tune learning rates individually for each variable and have been largely successful in relieving practitioners of tuning the learning rate.

Original: [Figure]
Revised: [Figure]

Original: Recently some researchers have started favoring simple momentum SGD over the previously mentioned adaptive methods (Chen et al., 2016;
Revised: Recently some researchers have started favoring simple momentum SGD over the previously mentioned adaptive methods (Chen et al., 2016;

Original: Gehring et al., 2017), often reporting better test scores (Wilson et al., 2017).
Revised: Gehring et al., 2017), often reporting better test scores (Wilson et al., 2017).

Original: Motivated by this trend, we ask the question: can simpler adaptive methods based on momentum SGD perform as well or better?
Revised: Motivated by this trend, we ask the question: can simpler adaptive methods based on momentum SGD perform as well or better?

Original: We empirically show that, with hand-tuned learning rate,
Revised: We empirically show that, with hand-tuned learning rate,

Original: Polyak’s momentum SGD achieves faster convergence than Adam for a large class of models.
Revised: Polyak’s momentum SGD achieves faster convergence than Adam for a large class of models.

Original: We then formulate the optimization update as a dynamical system and study certain robustness properties of the momentum operator.
Revised: We then formulate the optimization update as a dynamical system and study certain robustness properties of the momentum operator.

Original: Inspired by our analysis, we design Y ELLOW F IN , an automatic hyperparameter tuner for momentum SGD.
Revised: Inspired by our analysis, we design Y ELLOW F IN , an automatic hyperparameter tuner for momentum SGD.

Original: Y ELLOW F IN simultaneously tunes the learning rate and momentum on the ﬂy, and can handle the complex dynamics of asynchronous execution.
Revised: Y ELLOW F IN simultaneously tunes the learning rate and momentum on the ﬂy, and can handle the complex dynamics of asynchronous execution.

Original: Speciﬁcally: • In Section 2, we demonstrate examples where momentum offers convergence robust to learning rate misspeciﬁcation and curvature variation in a class of non-convex objectives.
Revised: Speciﬁcally: • In Section 2, we demonstrate examples where momentum offers convergence robust to learning rate misspeciﬁcation and curvature variation in a class of non-convex objectives.

Original: This robustness is desirable for deep learning.
Revised: This robustness is desirable for deep learning.

Original: It stems from a known but obscure fact: the momentum operator’s spectral radius is constant in a large subset of the hyperparameter space.
Revised: It stems from a known but obscure fact: the momentum operator’s spectral radius is constant in a large subset of the hyperparameter space.

Original: • In Section 3, we use these robustness insights and a simple quadratic model analysis to motivate the design of Y ELLOW F IN , an automatic tuner for momentum SGD.
Revised: • In Section 3, we use these robustness insights and a simple quadratic model analysis to motivate the design of Y ELLOW F IN , an automatic tuner for momentum SGD.

Original: Y ELLOW F IN uses on-theﬂy measurements from the gradients to tune both a single learning rate and momentum.
Revised: Y ELLOW F IN uses on-theﬂy measurements from the gradients to tune both a single learning rate and momentum.

Original: • In Section 3.3, we discuss common stability concerns related to the phenomenon of exploding gradients (Pascanu et al., 2013).
Revised: • In Section 3.3, we discuss common stability concerns related to the phenomenon of exploding gradients (Pascanu et al., 2013).

Original: We present a natural extension to our basic tuner, using adaptive gradient clipping, to stabilize training for objectives with exploding gradients.
Revised: We present a natural extension to our basic tuner, using adaptive gradient clipping, to stabilize training for objectives with exploding gradients.

Original: • In Section 4 we present closed-loop Y ELLOW F IN , suited for asynchronous training.
Revised: • In Section 4 we present closed-loop Y ELLOW F IN , suited for asynchronous training.

Original: It uses a novel component for measuring the total momentum in a running system, including any asynchrony-induced momentum, a phenomenon described in Mitliagkas et al.
Revised: It uses a novel component for measuring the total momentum in a running system, including any asynchrony-induced momentum, a phenomenon described in Mitliagkas et al.

Original: This measurement is used in a negative feedback loop to control the value of algorithmic momentum.
Revised: This measurement is used in a negative feedback loop to control the value of algorithmic momentum.

Original: We provide a thorough empirical evaluation of the performance and stability of our tuner.
Revised: We provide a thorough empirical evaluation of the performance and stability of our tuner.

Original: In Section 5, we demonstrate empirically that on ResNets and LSTMs Y ELLOW F IN can converge in fewer iterations compared to: (i) hand-tuned momentum SGD (up to 1 .
Revised: In Section 5, we demonstrate empirically that on ResNets and LSTMs Y ELLOW F IN can converge in fewer iterations compared to: (i) hand-tuned momentum SGD (up to 1 .

Original: 75 x speedup); and (ii) default Adam ( 0 .
Revised: 75 x speedup); and (ii) default Adam ( 0 .

Original: 8 x to 3 .
Revised: 8 x to 3 .

Original: 3 x speedup).
Revised: 3 x speedup).

Original: Under asynchrony, the closed-loop control architecture speeds up Y ELLOW F IN , making it up to 2 .
Revised: Under asynchrony, the closed-loop control architecture speeds up Y ELLOW F IN , making it up to 2 .

Original: 69 x faster than Adam.
Revised: 69 x faster than Adam.

Original: Our experiments include runs on 7 different models, randomized over at least 5 different random seeds.
Revised: Our experiments include runs on 7 different models, randomized over at least 5 different random seeds.

Original: Y ELLOW F IN is stable and achieves consistent performance: the normalized sample standard deviation of test metrics varies from 0 .
Revised: Y ELLOW F IN is stable and achieves consistent performance: the normalized sample standard deviation of test metrics varies from 0 .

Original: 05% to 0 .
Revised: 05% to 0 .

Original: We
Revised: We

Original: released PyTorch and TensorFlow implementations that can be used as drop-in replacements
Revised: released PyTorch and TensorFlow implementations that can be used as drop-in replacements

Original: for any
Revised: for any

Original: optimizer.
Revised: optimizer.

Original: Y ELLOW F IN has also been implemented in various other packages.
Revised: Y ELLOW F IN has also been implemented in various other packages.

Original: Its large-scale deployment in industry has taught us important lessons about stability; we discuss those challenges and our solution in Section 3.3.
Revised: Its large-scale deployment in industry has taught us important lessons about stability; we discuss those challenges and our solution in Section 3.3.

Original: We conclude with related work and discussion in Section 6 and
Revised: We conclude with related work and discussion in Section 6 and

Original:  T HE MOMENTUM OPERATOR
Revised:  T HE MOMENTUM OPERATOR

Original: In this section we identify the main technical insight behind the design of Y ELLOW F IN : gradient descent with momentum can exhibit linear convergence robust to learning rate misspeciﬁcation and to curvature variation.
Revised: In this section we identify the main technical insight behind the design of Y ELLOW F IN : gradient descent with momentum can exhibit linear convergence robust to learning rate misspeciﬁcation and to curvature variation.

Original: The robustness to learning rate misspeciﬁcation means tolerance to a less-carefully-tuned learning rate.
Revised: The robustness to learning rate misspeciﬁcation means tolerance to a less-carefully-tuned learning rate.

Original: On the other hand, the robustness to curvature variation means empirical linear convergence on a class of non-convex objectives with varying curvatures.
Revised: On the other hand, the robustness to curvature variation means empirical linear convergence on a class of non-convex objectives with varying curvatures.

Original: After preliminary on momentum, we discuss these two properties desirable for deep learning objectives.
Revised: After preliminary on momentum, we discuss these two properties desirable for deep learning objectives.

Original: 2. P RELIMINARIES
Revised: 2. P RELIMINARIES

Original: We aim to minimize some objective f ( x ) .
Revised: We aim to minimize some objective f ( x ) .

Original: In machine learning, x is referred to as the model and the objective is some loss function .
Revised: In machine learning, x is referred to as the model and the objective is some loss function .

Original: A low loss implies a well-ﬁt model.
Revised: A low loss implies a well-ﬁt model.

Original: Gradient descent-based procedures use the gradient of the objective function, ∇ f ( x ) , to update the model iteratively.
Revised: Gradient descent-based procedures use the gradient of the objective function, ∇ f ( x ) , to update the model iteratively.

Original: These procedures can be characterized by the convergence rate with respect to the distance to a minimum.
Revised: These procedures can be characterized by the convergence rate with respect to the distance to a minimum.

Original: Deﬁnition 1 (Convergence rate) .
Revised: Deﬁnition 1 (Convergence rate) .

Original: Let x ∗ be a local minimum of f ( x ) and x t denote the model after t steps of an iterative procedure.
Revised: Let x ∗ be a local minimum of f ( x ) and x t denote the model after t steps of an iterative procedure.

Original: The iterates converge to x ∗ with linear rate β , if
Revised: The iterates converge to x ∗ with linear rate β , if

Original: [Equation]
Revised: [Equation]

Original: Polyak’s momentum gradient descent (Polyak, 1964) is one of these iterative procedures, given by
Revised: Polyak’s momentum gradient descent (Polyak, 1964) is one of these iterative procedures, given by

Original: [Equation]
Revised: [Equation]

Original: where α denotes a single learning rate and µ a single momentum for all model variables.
Revised: where α denotes a single learning rate and µ a single momentum for all model variables.

Original: Momentum’s main appeal is its established ability to accelerate convergence (Polyak, 1964).
Revised: Momentum’s main appeal is its established ability to accelerate convergence (Polyak, 1964).

Original: On a γ -strongly convex δ -smooth function with condition number κ = δ/γ , the optimal convergence rate of gradient descent without momentum is O ( κ − 1 κ +1 )
Revised: On a γ -strongly convex δ -smooth function with condition number κ = δ/γ , the optimal convergence rate of gradient descent without momentum is O ( κ − 1 κ +1 )

Original: (Nesterov, 2013).
Revised: (Nesterov, 2013).

Original: On the other hand, for certain classes of strongly convex and smooth functions, like quadratics, the optimal momentum value,
Revised: On the other hand, for certain classes of strongly convex and smooth functions, like quadratics, the optimal momentum value,

Original: [Equation]
Revised: [Equation]

Original: yields the optimal accelerated linear convergence rate O ( √ κ −
Revised: yields the optimal accelerated linear convergence rate O ( √ κ −

Original: [Equation]
Revised: [Equation]

Original: √ κ +1 ) .
Revised: √ κ +1 ) .

Original: This guarantee does not generalize to arbitrary strongly convex smooth functions (Lessard et al., 2016).
Revised: This guarantee does not generalize to arbitrary strongly convex smooth functions (Lessard et al., 2016).

Original: Nonetheless, this linear rate can often be observed in practice even on non-quadratics (cf.
Revised: Nonetheless, this linear rate can often be observed in practice even on non-quadratics (cf.

Original: Section 2.2).
Revised: Section 2.2).

Original: The main insight we expose has been hidden away in proofs: Quadratic objectives with condition number κ > 1 have different curvatures along different axes.
Revised: The main insight we expose has been hidden away in proofs: Quadratic objectives with condition number κ > 1 have different curvatures along different axes.

Original: On these quadratics, Polyak’s momentum gradient descent, with µ ≥ µ ∗ , achieves the same linear convergence rate √ µ along all axes with different curvatures .
Revised: On these quadratics, Polyak’s momentum gradient descent, with µ ≥ µ ∗ , achieves the same linear convergence rate √ µ along all axes with different curvatures .

Original: Speciﬁcally, let x i,t and x ∗ i be the i-th coordinates of x t and x ∗ .
Revised: Speciﬁcally, let x i,t and x ∗ i be the i-th coordinates of x t and x ∗ .

Original: For any µ ≥ µ ∗ with a proper learning rate, the update in (1) can achieve | x i,t − x ∗ i | ≤ √ µ t | x i, 0 − x ∗ i | simultaneously for all axes i .
Revised: For any µ ≥ µ ∗ with a proper learning rate, the update in (1) can achieve | x i,t − x ∗ i | ≤ √ µ t | x i, 0 − x ∗ i | simultaneously for all axes i .

Original: This hidden insight reveals linear convergence behavior on quadratics, where the curvature is different across different axes, but constant on the one dimensional slice along each axis.
Revised: This hidden insight reveals linear convergence behavior on quadratics, where the curvature is different across different axes, but constant on the one dimensional slice along each axis.

Original: In the next section, we extend and show with examples that similar linear convergence rate √ µ can be achieved in a class of one dimensional non-convex objectives where curvature varies .
Revised: In the next section, we extend and show with examples that similar linear convergence rate √ µ can be achieved in a class of one dimensional non-convex objectives where curvature varies .

Original: This linear convergence behavior is robust to learning rate misspeciﬁcation and to the varying curvature; these robustness properties implies a tuning rule for learning rate and momentum in Section 2.2.
Revised: This linear convergence behavior is robust to learning rate misspeciﬁcation and to the varying curvature; these robustness properties implies a tuning rule for learning rate and momentum in Section 2.2.

Original: We extend this rule to handle SGD noise and generalize it to multidimensional objectives in Section 3.
Revised: We extend this rule to handle SGD noise and generalize it to multidimensional objectives in Section 3.

Original: 2. R OBUSTNESS PROPERTIES OF THE MOMENTUM OPERATOR
Revised: 2. R OBUSTNESS PROPERTIES OF THE MOMENTUM OPERATOR

Original: In this section, we analyze the dynamics of momentum on a class of one-dimensional, non-convex objectives.
Revised: In this section, we analyze the dynamics of momentum on a class of one-dimensional, non-convex objectives.

Original: We ﬁrst introduce the notion of generalized curvature and use it to describe the momentum operator.
Revised: We ﬁrst introduce the notion of generalized curvature and use it to describe the momentum operator.

Original: Then we discuss the robustness properties of the momentum operator.
Revised: Then we discuss the robustness properties of the momentum operator.

Original: Curvature along different directions is encoded in the different eigenvalues of the Hessian.
Revised: Curvature along different directions is encoded in the different eigenvalues of the Hessian.

Original: It is the only feature of a quadratic needed to characterize the convergence of gradient descent.
Revised: It is the only feature of a quadratic needed to characterize the convergence of gradient descent.

Original: Speciﬁcally, gradient descent achieves a linear convergence rate | 1 − αh c | on one-dimensional quadratics with constant curvature h c .
Revised: Speciﬁcally, gradient descent achieves a linear convergence rate | 1 − αh c | on one-dimensional quadratics with constant curvature h c .

Original: On non-quadratic objectives with varying curvature , this neat characterization is lost.
Revised: On non-quadratic objectives with varying curvature , this neat characterization is lost.

Original: We can recover it by deﬁning a new kind of “curvature” with respect to a speciﬁc minimum.
Revised: We can recover it by deﬁning a new kind of “curvature” with respect to a speciﬁc minimum.

Original: Deﬁnition 2 (Generalized curvature) .
Revised: Deﬁnition 2 (Generalized curvature) .

Original: Let x ∗ be a local minimum of f ( x ) :
Revised: Let x ∗ be a local minimum of f ( x ) :

Original: R → R .
Revised: R → R .

Original: Generalized curvature with respect to x ∗ , denoted by h ( x ) , satisﬁes the following.
Revised: Generalized curvature with respect to x ∗ , denoted by h ( x ) , satisﬁes the following.

Original: [Equation]
Revised: [Equation]

Original: Generalized curvature describes, in some sense, non-local curvature with respect to minimum x ∗ .
Revised: Generalized curvature describes, in some sense, non-local curvature with respect to minimum x ∗ .

Original: It coincides with curvature on quadratics.
Revised: It coincides with curvature on quadratics.

Original: On non-quadratic objectives, it characterizes the convergence behavior of gradient descent-based algorithms.
Revised: On non-quadratic objectives, it characterizes the convergence behavior of gradient descent-based algorithms.

Original: Speciﬁcally, we recover the fact that starting at point x t , distance from minimum x ∗ is reduced by | 1 − αh ( x t ) | in one step of gradient descent.
Revised: Speciﬁcally, we recover the fact that starting at point x t , distance from minimum x ∗ is reduced by | 1 − αh ( x t ) | in one step of gradient descent.

Original: Using a state-space augmentation, we can rewrite the momentum update of (1), and deﬁne the momentum operator A t at time t as
Revised: Using a state-space augmentation, we can rewrite the momentum update of (1), and deﬁne the momentum operator A t at time t as

Original: [Equation]
Revised: [Equation]

Original: Lemma 3 (Robustness of the momentum operator) .
Revised: Lemma 3 (Robustness of the momentum operator) .

Original: Assume that generalized curvature h and hyperparameters α, µ satisfy
Revised: Assume that generalized curvature h and hyperparameters α, µ satisfy

Original: [Equation]
Revised: [Equation]

Original: Then as proven in Appendix A, the spectral radius of the momentum operator at step t depends solely on the momentum parameter: ρ ( A t ) = √ µ .
Revised: Then as proven in Appendix A, the spectral radius of the momentum operator at step t depends solely on the momentum parameter: ρ ( A t ) = √ µ .

Original: The inequalities in (5) deﬁne the robust region , the set of learning rate α and momentum µ achieving this √ µ spectral radius.
Revised: The inequalities in (5) deﬁne the robust region , the set of learning rate α and momentum µ achieving this √ µ spectral radius.

Original: The spectral radius of A t can describe the evolution of distance | x t − x ∗ | in (4).
Revised: The spectral radius of A t can describe the evolution of distance | x t − x ∗ | in (4).

Original: Consequently, the spectral radius of the composition A t · · · A 1 is closely related 1 to the convergence behavior, namely the asymptotic relationship between | x t − x ∗ | and the initial distance | x 0 − x ∗ | .
Revised: Consequently, the spectral radius of the composition A t · · · A 1 is closely related 1 to the convergence behavior, namely the asymptotic relationship between | x t − x ∗ | and the initial distance | x 0 − x ∗ | .

Original: Unfortunately given time-homogenous spectral radii √ µ , the spectral radius of the composition of A t · · · A 1 does not always follow the asymptotics of √ µ t .
Revised: Unfortunately given time-homogenous spectral radii √ µ , the spectral radius of the composition of A t · · · A 1 does not always follow the asymptotics of √ µ t .

Original: However, the homogeneous spectral radii often suggest empirical linear convergence in practice.
Revised: However, the homogeneous spectral radii often suggest empirical linear convergence in practice.

Original: In other words, this linear convergence rate is not always guaranteed .
Revised: In other words, this linear convergence rate is not always guaranteed .

Original: Instead, we demonstrate examples, in the two following paragraphs, to expose the robustness properties : given learning rate α and momentum µ are in the robust region, homogeneous spectral radius √ µ of A t suggests empirical linear convergence; this behavior is robust with respect to learning rate mispeciﬁcation and to varying curvature .
Revised: Instead, we demonstrate examples, in the two following paragraphs, to expose the robustness properties : given learning rate α and momentum µ are in the robust region, homogeneous spectral radius √ µ of A t suggests empirical linear convergence; this behavior is robust with respect to learning rate misspeciﬁcation and to varying curvature .

Original: [Figure]
Revised: [Figure]

Original: Momentum is robust to learning rate misspeciﬁcation For a onedimensional quadratic with curvature h , we have generalized curvature h ( x )
Revised: Momentum is robust to learning rate misspeciﬁcation For a onedimensional quadratic with curvature h , we have generalized curvature h ( x )

Original: = h for all x .
Revised: = h for all x .

Original: Lemma 3 implies the spectral radius ρ ( A t )= √ µ if
Revised: Lemma 3 implies the spectral radius ρ ( A t )= √ µ if

Original: [Equation]
Revised: [Equation]

Original: In Figure 2, we plot ρ ( A t ) for different α and µ when h = 1 .
Revised: In Figure 2, we plot ρ ( A t ) for different α and µ when h = 1 .

Original: As we increase the value of momentum, a linear rate of convergence, √ µ , is robustly achieved by an ever-widening range of learning rates.
Revised: As we increase the value of momentum, a linear rate of convergence, √ µ , is robustly achieved by an ever-widening range of learning rates.

Original: This suggests that, for objectives with large condition number, higher values of momentum can be more robust to learning rate mispeciﬁcation.
Revised: This suggests that, for objectives with large condition number, higher values of momentum can be more robust to learning rate mispeciﬁcation.

Original: This property inﬂuences the design of our tuner: more generally for a class of one-dimensional non-convex objectives , as long as learning rate α and momentum µ satisﬁes (5) in Lemma 3 at every step, α and µ are in the robust region and all momentum operators have the same spectral radius .
Revised: This property inﬂuences the design of our tuner: more generally for a class of one-dimensional non-convex objectives , as long as learning rate α and momentum µ satisﬁes (5) in Lemma 3 at every step, α and µ are in the robust region and all momentum operators have the same spectral radius .

Original: In the case of quadratics, this implies a convergence rate of √ µ , independent of the learning rate.
Revised: In the case of quadratics, this implies a convergence rate of √ µ , independent of the learning rate.

Original: Having established that, we can just focus on optimally tuning momentum.
Revised: Having established that, we can just focus on optimally tuning momentum.

Original: Momentum is robust to varying curvatures As discussed in Section 2.1, the intuition hidden in classic results is that for certain strongly convex smooth objectives, momentum larger than the value in (2) can achieve the same rate of linear convergence along all axes with different curvatures.
Revised: Momentum is robust to varying curvatures As discussed in Section 2.1, the intuition hidden in classic results is that for certain strongly convex smooth objectives, momentum larger than the value in (2) can achieve the same rate of linear convergence along all axes with different curvatures.

Original: We extend this intuition to certain one-dimensional non-convex functions with varying curvatures along their domains; we discuss the generalization to multidimensional cases in Section 3.1.
Revised: We extend this intuition to certain one-dimensional non-convex functions with varying curvatures along their domains; we discuss the generalization to multidimensional cases in Section 3.1.

Original: Lemmaguarantees constant, time-homogeneous spectral radii for momentum operators
Revised: Lemmaguarantees constant, time-homogeneous spectral radii for momentum operators

Original: A t assuming (5) is satisﬁed at every step.
Revised: A t assuming (5) is satisﬁed at every step.

Original: This assumption motivates a “long-range” extension of the condition number.
Revised: This assumption motivates a “long-range” extension of the condition number.

Original: Deﬁnition 4 (Generalized condition number) .
Revised: Deﬁnition 4 (Generalized condition number) .

Original: We deﬁne the generalized condition number (GCN) with respect to a local minimum x ∗ of a scalar function, f ( x ) :
Revised: We deﬁne the generalized condition number (GCN) with respect to a local minimum x ∗ of a scalar function, f ( x ) :

Original: R → R , to be the dynamic range of its generalized curvature h ( x ) :
Revised: R → R , to be the dynamic range of its generalized curvature h ( x ) :

Original: [Equation]
Revised: [Equation]

Original: The GCN captures variations in generalized curvature along a scalar slice.
Revised: The GCN captures variations in generalized curvature along a scalar slice.

Original: From Lemma 3 we get
Revised: From Lemma 3 we get

Original: [Equation]
Revised: [Equation]

Original: as the description of the robust region.
Revised: as the description of the robust region.

Original: The momentum and learning rate satisfying (8) guarantees a homogeneous spectral radius of √ µ for all A t .
Revised: The momentum and learning rate satisfying (8) guarantees a homogeneous spectral radius of √ µ for all A t .

Original: Speciﬁcally, µ ∗ is the smallest momentum value that allows for homogeneous spectral radii.
Revised: Speciﬁcally, µ ∗ is the smallest momentum value that allows for homogeneous spectral radii.

Original: We demonstrate with examples that homogeneous spectral radii suggest an empirical linear convergence behavior on a class of non-convex objectives .
Revised: We demonstrate with examples that homogeneous spectral radii suggest an empirical linear convergence behavior on a class of non-convex objectives .

Original: In
Revised: In

Original: Figure 3(a), the non-convex objective, composed of two quadratics with curvatures 1 and 1000 , has a GCN of 1000 .
Revised: Figure 3(a), the non-convex objective, composed of two quadratics with curvatures 1 and 1000 , has a GCN of 1000 .

Original: Using the tuning rule of (8), and running the momentum
Revised: Using the tuning rule of (8), and running the momentum

Original: algorithm (Figure 3(b)) practically yields the linear convergence predicted by Lemma 3.
Revised: algorithm (Figure 3(b)) practically yields the linear convergence predicted by Lemma 3.

Original: In Figures 3(c,d), we demonstrate an LSTM as another example.
Revised: In Figures 3(c,d), we demonstrate an LSTM as another example.

Original: As we increase the momentum value (the single one for all variables in the model), more model variables follow a √ µ convergence rate.
Revised: As we increase the momentum value (the single one for all variables in the model), more model variables follow a √ µ convergence rate.

Original: In these examples, the linear convergence is robust to the varying curvature of the objectives .
Revised: In these examples, the linear convergence is robust to the varying curvature of the objectives .

Original: This property inﬂuences our tuner design: in the next section, we extend the tuning rules of (8) to handle SGD noise; we generalize the extended rule to multidimensional cases as the tuning rule in Y ELLOW F IN .
Revised: This property inﬂuences our tuner design: in the next section, we extend the tuning rules of (8) to handle SGD noise; we generalize the extended rule to multidimensional cases as the tuning rule in Y ELLOW F IN .

Original: [Equation]
Revised: [Equation]

Original: F IN TUNER
Revised: F IN TUNER

Original: In this section we describe Y ELLOW F IN , our tuner for momentum SGD.
Revised: In this section we describe Y ELLOW F IN , our tuner for momentum SGD.

Original: We ﬁrst introduce a noisy quadratic model f ( x ) as the local approximation of an arbitrary one-dimensional objective in stochastic cases.
Revised: We ﬁrst introduce a noisy quadratic model f ( x ) as the local approximation of an arbitrary one-dimensional objective in stochastic cases.

Original: Based on this approximation, we extend the tuning rule of (8) to SGD.
Revised: Based on this approximation, we extend the tuning rule of (8) to SGD.

Original: In section 3.1, we generalize the discussion to multidimensional objectives; it yields the Y ELLOW F IN tuning rule .
Revised: In section 3.1, we generalize the discussion to multidimensional objectives; it yields the Y ELLOW F IN tuning rule .

Original: Noisy quadratic model We consider minimizing the one-dimensional quadratic
Revised: Noisy quadratic model We consider minimizing the one-dimensional quadratic

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: local quadratic approximations of multidimensional objectives.
Revised: local quadratic approximations of multidimensional objectives.

Original: Next we get exact expression for the mean square error after running momentum SGD on the scalar quadratic in (9) for t steps.
Revised: Next we get exact expression for the mean square error after running momentum SGD on the scalar quadratic in (9) for t steps.

Original: Lemma 5.
Revised: Lemma 5.

Original: Let f ( x ) be deﬁned as in (9) , x 1 = x 0 and x t follow the momentum update (1) with stochastic gradients ∇ f S t ( x t − 1 ) for t ≥ 2 .
Revised: Let f ( x ) be deﬁned as in (9) , x 1 = x 0 and x t follow the momentum update (1) with stochastic gradients ∇ f S t ( x t − 1 ) for t ≥ 2 .

Original: Let e 1 =
Revised: Let e 1 =

Original: [1 , 0] T , the expectation of squared distance to the optimum x ∗ is
Revised: [1 , 0] T , the expectation of squared distance to the optimum x ∗ is

Original: [Equation]
Revised: [Equation]

Original: where the ﬁrst and second term correspond to squared bias and variance, and their corresponding momentum dynamics are captured by operators
Revised: where the ﬁrst and second term correspond to squared bias and variance, and their corresponding momentum dynamics are captured by operators

Original: [Equation]
Revised: [Equation]

Original: Even though it is possible to numerically work on (10) directly, we use a scalar, asymptotic surrogate in (12) based on the spectral radii of operators to simplify analysis and expose insights.
Revised: Even though it is possible to numerically work on (10) directly, we use a scalar, asymptotic surrogate in (12) based on the spectral radii of operators to simplify analysis and expose insights.

Original: This decision is supported by our ﬁndings in Section 2: the spectral radii can capture empirical convergence rate.
Revised: This decision is supported by our ﬁndings in Section 2: the spectral radii can capture empirical convergence rate.

Original: [Equation]
Revised: [Equation]

Original: One of our design decisions for Y ELLOW F IN is to always work in the robust region of Lemma 3.
Revised: One of our design decisions for Y ELLOW F IN is to always work in the robust region of Lemma 3.

Original: We know that this implies a spectral radius √ µ of the momentum operator, A , for the bias.
Revised: We know that this implies a spectral radius √ µ of the momentum operator, A , for the bias.

Original: Lemmashows that under the exact same condition, the variance operator B has spectral radius µ .
Revised: Lemmashows that under the exact same condition, the variance operator B has spectral radius µ .

Original: Lemma 6.
Revised: Lemma 6.

Original: The spectral radius of the variance operator, B is µ , if (1 − √ µ ) 2 ≤ αh ≤
Revised: The spectral radius of the variance operator, B is µ , if (1 − √ µ ) 2 ≤ αh ≤

Original: [Equation]
Revised: [Equation]

Original: 1 + √ µ ) 2 .
Revised: 1 + √ µ ) 2 .

Original: As a result, the surrogate objective of (12), takes the following form in the robust region.
Revised: As a result, the surrogate objective of (12), takes the following form in the robust region.

Original: [Equation]
Revised: [Equation]

Original: We extend this surrogate to multidimensional cases to extract a noisy tuning rule for Y ELLOW F IN .
Revised: We extend this surrogate to multidimensional cases to extract a noisy tuning rule for Y ELLOW F IN .

Original: 3. T UNING RULE
Revised: 3. T UNING RULE

Original: In this section, we present S INGLE S TEP , the tuning rule of YellowFin (Algorithm 1).
Revised: In this section, we present S INGLE S TEP , the tuning rule of YellowFin (Algorithm 1).

Original: Based on the surrogate in (13), S INGLE S TEP is a multidimensional SGD version of the noiseless tuning rule in (8).
Revised: Based on the surrogate in (13), S INGLE S TEP is a multidimensional SGD version of the noiseless tuning rule in (8).

Original: We ﬁrst generalize (8) and (13) to multidimensional cases, and then discuss S INGLE S TEP in details.
Revised: We ﬁrst generalize (8) and (13) to multidimensional cases, and then discuss S INGLE S TEP in details.

Original: As discussed in Section 2.2, GCN ν captures the dynamic range of generalized curvatures in a onedimensional objective with varying curvature.
Revised: As discussed in Section 2.2, GCN ν captures the dynamic range of generalized curvatures in a onedimensional objective with varying curvature.

Original: The consequent robust region described by (8) implies homogeneous spectral radii.
Revised: The consequent robust region described by (8) implies homogeneous spectral radii.

Original: On a multidimensional non-convex objective, each one-dimensional slice passing a minimum x ∗ can have varying curvature .
Revised: On a multidimensional non-convex objective, each one-dimensional slice passing a minimum x ∗ can have varying curvature .

Original: As we use a single µ and α for the entire model , if ν simultaneously captures the dynamic range of generalized curvature over all these slices, µ and α in (8) are in the robust region for all these slices.
Revised: As we use a single µ and α for the entire model , if ν simultaneously captures the dynamic range of generalized curvature over all these slices, µ and α in (8) are in the robust region for all these slices.

Original: This implies homogeneous spectral radii √ µ according to Lemma 3, empirically facilitating convergence at a common rate along all the directions.
Revised: This implies homogeneous spectral radii √ µ according to Lemma 3, empirically facilitating convergence at a common rate along all the directions.

Original: Given homogeneous spectral radii √ µ along all directions, the surrogate in (13) generalizes on the local quadratic approximation of multiple dimensional objectives.
Revised: Given homogeneous spectral radii √ µ along all directions, the surrogate in (13) generalizes on the local quadratic approximation of multiple dimensional objectives.

Original: On this approximation with minimum x ∗ , the expectation of squared distance to x ∗ , E (cid:107) x 0 − x ∗ (cid:107) 2 , decomposes into independent scalar components along the eigenvectors of the Hessian.
Revised: On this approximation with minimum x ∗ , the expectation of squared distance to x ∗ , E (cid:107) x 0 − x ∗ (cid:107) 2 , decomposes into independent scalar components along the eigenvectors of the Hessian.

Original: We deﬁne gradient variance C as the sum of gradient variance along these eigenvectors.
Revised: We deﬁne gradient variance C as the sum of gradient variance along these eigenvectors.

Original: The one-dimensional surrogates in (13) for the independent components sum to µ t (cid:107) x 0 − x ∗ (cid:107) 

Revised: The one-dimensional surrogates in (13) for the independent components sum to µ t (cid:107) x 0 − x ∗ (cid:107)

Original: + (1 − µ t )
Revised: + (1 − µ t )

Original: α 2 C/ (1 − µ ) , the multidimensional surrogate corresponding to the one in (13).
Revised: α 2 C/ (1 − µ ) , the multidimensional surrogate corresponding to the one in (13).

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: end function
Revised: end function

Original: Let D be an estimate of the current model’s distance to a local quadratic approximation’s minimum, and C denote an estimate for gradient variance.
Revised: Let D be an estimate of the current model’s distance to a local quadratic approximation’s minimum, and C denote an estimate for gradient variance.

Original: S INGLE S TEP minimizes the multidimensional surrogate after a single step (i.e. t = 1 ) while ensuring µ and α in the robust region for all directions.
Revised: S INGLE S TEP minimizes the multidimensional surrogate after a single step (i.e. t = 1 ) while ensuring µ and α in the robust region for all directions.

Original: A single instance of S INGLE S TEP solves a single momentum and learning rate for the entire model at each iteration.
Revised: A single instance of S INGLE S TEP solves a single momentum and learning rate for the entire model at each iteration.

Original: Speciﬁcally, the extremal curvatures h min and h max denote estimates for the largest and smallest generalized curvature respectively.
Revised: Speciﬁcally, the extremal curvatures h min and h max denote estimates for the largest and smallest generalized curvature respectively.

Original: They are meant to capture both generalized curvature variation along all different directions (like the classic condition number) and also variation that occurs as the landscape evolves .
Revised: They are meant to capture both generalized curvature variation along all different directions (like the classic condition number) and also variation that occurs as the landscape evolves .

Original: The constraints keep the global learning rate and momentum in the robust region (deﬁned in Lemma 3) for slices along all directions.
Revised: The constraints keep the global learning rate and momentum in the robust region (deﬁned in Lemma 3) for slices along all directions.

Original: S INGLE S TEP can be solved in closed form; we refer to Appendix D for relevant details on the closed form solution.
Revised: S INGLE S TEP can be solved in closed form; we refer to Appendix D for relevant details on the closed form solution.

Original: Y ELLOW F IN uses functions C URVATURE R ANGE , V ARIANCE and D ISTANCE to measure quantities h max , h min ,
Revised: Y ELLOW F IN uses functions C URVATURE R ANGE , V ARIANCE and D ISTANCE to measure quantities h max , h min ,

Original: C and D respectively.
Revised: C and D respectively.

Original: These measurement functions can be designed in different ways.
Revised: These measurement functions can be designed in different ways.

Original: We present the implementations we used for our experiments, based completely on gradients, in Section 3.2.
Revised: We present the implementations we used for our experiments, based completely on gradients, in Section 3.2.

Original: 3. M EASUREMENT FUNCTIONS IN Y ELLOW F IN
Revised: 3. M EASUREMENT FUNCTIONS IN Y ELLOW F IN

Original: This section describes our implementation of the measurement oracles used by Y ELLOW F IN : C URVATURE R ANGE , V ARIANCE , and D ISTANCE .
Revised: This section describes our implementation of the measurement oracles used by Y ELLOW F IN : C URVATURE R ANGE , V ARIANCE , and D ISTANCE .

Original: We design the measurement functions with the assumption of a negative log-probability objective; this is in line with typical losses in machine learning, e.g. cross-entropy for neural nets and maximum likelihood estimation in general.
Revised: We design the measurement functions with the assumption of a negative log-probability objective; this is in line with typical losses in machine learning, e.g. cross-entropy for neural nets and maximum likelihood estimation in general.

Original: Under this assumption, the Fisher information matrix—i.e. the expected outer product of noisy gradients— approximates the Hessian of the objective (Duchi, 2016; Pascanu and Bengio, 2013).
Revised: Under this assumption, the Fisher information matrix—i.e. the expected outer product of noisy gradients— approximates the Hessian of the objective (Duchi, 2016; Pascanu and Bengio, 2013).

Original: This allows for measurements purely from minibatch gradients with overhead linear to model dimensionality.
Revised: This allows for measurements purely from minibatch gradients with overhead linear to model dimensionality.

Original: These implementations are not guaranteed to give accurate measurements.
Revised: These implementations are not guaranteed to give accurate measurements.

Original: Nonetheless, their use in our experiments in Section 5 shows that they are sufﬁcient for Y ELLOW F IN to outperform the state of the art on a variety of objectives.
Revised: Nonetheless, their use in our experiments in Section 5 shows that they are sufﬁcient for Y ELLOW F IN to outperform the state of the art on a variety of objectives.

Original: We also refer to Appendix E for details on zero-debias (Kingma and
Revised: We also refer to Appendix E for details on zero-debias (Kingma and

Original: Ba, 2014), slow start (Schaul et al., 2013) and smoothing for curvature range estimation.
Revised: Ba, 2014), slow start (Schaul et al., 2013) and smoothing for curvature range estimation.

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: end function
Revised: end function

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: end function
Revised: end function

Original: 0k 1k 2k 3k Iterations 10 −

Revised: 0k 1k 2k 3k Iterations 10 −

Original: G r a d i e n t n o r m
Revised: G r a d i e n t n o r m

Original: Without clipping With clipping
Revised: Without clipping With clipping

Original: Clipping thresh.
Revised: Clipping thresh.

Original: 0k 1k 2k 3k Iterations 10 −

Revised: 0k 1k 2k 3k Iterations 10 −

Original: T r a i n i n g l o ss
Revised: T r a i n i n g l o ss

Original: Without clipping With clipping
Revised: Without clipping With clipping

Original: Loss
Revised: Loss

Original: BLEU
Revised: BLEU

Original: Default w/o clip.
Revised: Default w/o clip.

Original: diverge
Revised: diverge

Original: Default w/ clip.
Revised: Default w/ clip.

Original: YF 2.
Revised: YF 2.

Original: Curvature range Let g t be a noisy gradient, we estimate the range of curvatures in Algorithm 2.
Revised: Curvature range Let g t be a noisy gradient, we estimate the range of curvatures in Algorithm 2.

Original: We notice that the outer product g t g Tt has an eigenvalue h t = (cid:107) g t (cid:107) 2 with eigenvector g t .
Revised: We notice that the outer product g t g Tt has an eigenvalue h t = (cid:107) g t (cid:107) 2 with eigenvector g t .

Original: Thus under our negative log-likelihood assumption, we use h t to approximate the curvature of Hessian along gradient direction g t .
Revised: Thus under our negative log-likelihood assumption, we use h t to approximate the curvature of Hessian along gradient direction g t .

Original: Speciﬁcally, we maintain h min and h max as running averages of extreme curvature h min ,t and h max ,t , from a sliding window of width 20.
Revised: Speciﬁcally, we maintain h min and h max as running averages of extreme curvature h min ,t and h max ,t , from a sliding window of width 20.

Original: As gradient directions evolve, we estimate curvatures along different directions.
Revised: As gradient directions evolve, we estimate curvatures along different directions.

Original: Thus h min and h max capture the curvature variations.
Revised: Thus h min and h max capture the curvature variations.

Original: Gradient variance To estimate the gradient variance in Algorithm 3, we use running averages g and g 2 to keep track of g t and g t (cid:12) g t , the ﬁrst and second order moment of the gradient.
Revised: Gradient variance To estimate the gradient variance in Algorithm 3, we use running averages g and g 2 to keep track of g t and g t (cid:12) g t , the ﬁrst and second order moment of the gradient.

Original: As
Revised: As

Original: Var( g t ) =
Revised: Var( g t ) =

Original: E g 2 t − ( E g t ) 2 , we estimate the gradient variance C in (14) using C = (cid:107) g 2 − g 2 (cid:107) 1 .
Revised: E g 2 t − ( E g t ) 2 , we estimate the gradient variance C in (14) using C = (cid:107) g 2 − g 2 (cid:107) 1 .

Original: Distance to optimum In Algorithm 4, we estimate the distance to the optimum of the local quadratic approximation.
Revised: Distance to optimum In Algorithm 4, we estimate the distance to the optimum of the local quadratic approximation.

Original: Inspired by the fact that (cid:107)∇ f ( x ) (cid:107) ≤ (cid:107) H (cid:107)(cid:107) x − x (cid:63) (cid:107) for a quadratic f ( x ) with Hessian H and minimizer x ∗ , we ﬁrst maintain h and (cid:107) g (cid:107) as running averages of curvature h t and gradient norm (cid:107) g t (cid:107) .
Revised: Inspired by the fact that (cid:107)∇ f ( x ) (cid:107) ≤ (cid:107) H (cid:107)(cid:107) x − x (cid:63) (cid:107) for a quadratic f ( x ) with Hessian H and minimizer x ∗ , we ﬁrst maintain h and (cid:107) g (cid:107) as running averages of curvature h t and gradient norm (cid:107) g t (cid:107) .

Original: Then the distance is approximated using (cid:107) g (cid:107) /h .
Revised: Then the distance is approximated using (cid:107) g (cid:107) /h .

Original: 3. S TABILITY ON NON - SMOOTH OBJECTIVES
Revised: 3. S TABILITY ON NON - SMOOTH OBJECTIVES

Original: Neural network objectives can involve arbitrary non-linearities, and large Lipschitz constants (Szegedy et al., 2013).
Revised: Neural network objectives can involve arbitrary non-linearities, and large Lipschitz constants (Szegedy et al., 2013).

Original: Furthermore, the process of training them is inherently non-stationary, with the landscape abruptly switching from ﬂat to steep areas.
Revised: Furthermore, the process of training them is inherently non-stationary, with the landscape abruptly switching from ﬂat to steep areas.

Original: In particular, the objective functions of RNNs with hidden units can exhibit occasional but very steep slopes (Pascanu et al., 2013).
Revised: In particular, the objective functions of RNNs with hidden units can exhibit occasional but very steep slopes (Pascanu et al., 2013).

Original: To deal with this issue, we use adaptive gradient clipping heuristics as a very natural addition to our basic tuner.
Revised: To deal with this issue, we use adaptive gradient clipping heuristics as a very natural addition to our basic tuner.

Original: It is discussed with extensive details in Appendix F.
Revised: It is discussed with extensive details in Appendix F.

Original: In Figure 4, we present an example of an LSTM that exhibits the ’exploding gradient’ issue.
Revised: In Figure 4, we present an example of an LSTM that exhibits the ’exploding gradient’ issue.

Original: The proposed adaptive clipping can stabilize the training process using Y ELLOW F IN and prevent large catastrophic gradient spikes.
Revised: The proposed adaptive clipping can stabilize the training process using Y ELLOW F IN and prevent large catastrophic gradient spikes.

Original: We validate the proposed adaptive clipping on the convolutional sequence to sequence learning model (Gehring et al., 2017) for IWSLT 2014 German-English translation.
Revised: We validate the proposed adaptive clipping on the convolutional sequence to sequence learning model (Gehring et al., 2017) for IWSLT 2014 German-English translation.

Original: The default optimizer (Gehring et al., 2017) uses learning rate 0 .
Revised: The default optimizer (Gehring et al., 2017) uses learning rate 0 .

Original: 25 and Nesterov’s momentum 0 . 99 , diverging to loss overﬂow due to ’exploding gradient’.
Revised: 25 and Nesterov’s momentum 0 . 99 , diverging to loss overﬂow due to ’exploding gradient’.

Original: It requires, as in Gehring et al.
Revised: It requires, as in Gehring et al.

Original: (2017), strict manually set gradient norm threshold 0 .
Revised: (2017), strict manually set gradient norm threshold 0 .

Original: 1 to stabilize.
Revised: 1 to stabilize.

Original: In Table 1, we can see YellowFin, with adaptive clipping, outperforms the default optimizer using manually set clipping, with 0.84 higher validation BLEU4 after 120 epochs.
Revised: In Table 1, we can see YellowFin, with adaptive clipping, outperforms the default optimizer using manually set clipping, with 0.84 higher validation BLEU4 after 120 epochs.

Original:  C LOSED - LOOP Y ELLOW F IN
Revised:  C LOSED - LOOP Y ELLOW F IN

Original: Asynchrony is a parallelization technique that avoids synchronization barriers (Niu et al., 2011).
Revised: Asynchrony is a parallelization technique that avoids synchronization barriers (Niu et al., 2011).

Original: It yields better hardware efﬁciency, i.e. faster steps, but can increase the number of iterations to a given metric, i.e. statistical efﬁciency, as a tradeoff (Zhang and Ré, 2014).
Revised: It yields better hardware efﬁciency, i.e. faster steps, but can increase the number of iterations to a given metric, i.e. statistical efﬁciency, as a tradeoff (Zhang and Ré, 2014).

Original: Mitliagkas et al.
Revised: Mitliagkas et al.

Original: (2016) interpret asynchrony as added momentum dynamics.
Revised: (2016) interpret asynchrony as added momentum dynamics.

Original: We design closed-loop Y ELLOW F IN , a variant of Y ELLOW F IN to automatically control algorithmic momentum, compensate for asynchrony and accelerate convergence.
Revised: We design closed-loop Y ELLOW F IN , a variant of Y ELLOW F IN to automatically control algorithmic momentum, compensate for asynchrony and accelerate convergence.

Original: We use the formula in (14) to model the dynamics in the system, where the total momentum, µ T , includes both asynchrony-induced and algorithmic momentum, µ , in (1).
Revised: We use the formula in (14) to model the dynamics in the system, where the total momentum, µ T , includes both asynchrony-induced and algorithmic momentum, µ , in (1).

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst use (14) to design an robust estimator ˆ µ T for the value of total momentum at every iteration.
Revised: We ﬁrst use (14) to design an robust estimator ˆ µ T for the value of total momentum at every iteration.

Original: Then we use a simple negative feedback control loop to adjust the value of algorithmic momentum so that ˆ µ T matches the target momentum decided by Y ELLOW F IN in Algorithm 1.
Revised: Then we use a simple negative feedback control loop to adjust the value of algorithmic momentum so that ˆ µ T matches the target momentum decided by Y ELLOW F IN in Algorithm 1.

Original: In Figure 5, we demonstrate momentum dynamics in an asynchronous training system.
Revised: In Figure 5, we demonstrate momentum dynamics in an asynchronous training system.

Original: As directly using the target
Revised: As directly using the target

Original: [Figure]
Revised: [Figure]

Original: value as algorithmic momentum, Y ELLOW F IN (middle) presents total momentum ˆ µ T strictly larger than the target momentum, due to asynchrony-induced momentum.
Revised: value as algorithmic momentum, Y ELLOW F IN (middle) presents total momentum ˆ µ T strictly larger than the target momentum, due to asynchrony-induced momentum.

Original: Closed-loop Y ELLOW F IN automatically brings down algorithmic momentum, match measured total momentum ˆ µ T to target value and, as we will see, signiﬁcantly speeds up convergence comparing to Y ELLOW F IN .
Revised: Closed-loop Y ELLOW F IN automatically brings down algorithmic momentum, match measured total momentum ˆ µ T to target value and, as we will see, signiﬁcantly speeds up convergence comparing to Y ELLOW F IN .

Original: We refer to Appendix G for details on estimator ˆ µ T and Closed-loop Y ELLOW F IN in Algorithm 5.
Revised: We refer to Appendix G for details on estimator ˆ µ T and Closed-loop Y ELLOW F IN in Algorithm 5.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically validate the importance of momentum tuning and evaluate Y ELLOW F IN in both synchronous (single-node) and asynchronous settings.
Revised: In this section, we empirically validate the importance of momentum tuning and evaluate Y ELLOW F IN in both synchronous (single-node) and asynchronous settings.

Original: In synchronous settings, we ﬁrst demonstrate that, with hand-tuning, momentum SGD is competitive with Adam, a state-of-the-art adaptive method.
Revised: In synchronous settings, we ﬁrst demonstrate that, with hand-tuning, momentum SGD is competitive with Adam, a state-of-the-art adaptive method.

Original: Then, we evaluate Y ELLOW F IN without any hand tuning in comparison to handtuned Adam and momentum SGD.
Revised: Then, we evaluate Y ELLOW F IN without any hand tuning in comparison to handtuned Adam and momentum SGD.

Original: In asynchronous settings, we show that closed-loop Y ELLOW F IN accelerates with momentum closed-loop control, performing signiﬁcantly better than Adam.
Revised: In asynchronous settings, we show that closed-loop Y ELLOW F IN accelerates with momentum closed-loop control, performing signiﬁcantly better than Adam.

Original: We evaluate on convolutional neural networks (CNN) and recurrent neural networks (RNN).
Revised: We evaluate on convolutional neural networks (CNN) and recurrent neural networks (RNN).

Original: For CNN, we train ResNet (He et al., 2016) for image recognition on CIFAR10 and CIFAR100 (Krizhevsky et al., 2014).
Revised: For CNN, we train ResNet (He et al., 2016) for image recognition on CIFAR10 and CIFAR100 (Krizhevsky et al., 2014).

Original: For RNN, we train LSTMs for character-level language modeling with the TinyShakespeare (TS) dataset
Revised: For RNN, we train LSTMs for character-level language modeling with the TinyShakespeare (TS) dataset

Original: (Karpathy et al., 2015), word-level language modeling with the Penn TreeBank (PTB) (Marcus et al., 1993), and constituency parsing on the Wall Street Journal (WSJ) dataset (Choe and
Revised: (Karpathy et al., 2015), word-level language modeling with the Penn TreeBank (PTB) (Marcus et al., 1993), and constituency parsing on the Wall Street Journal (WSJ) dataset (Choe and

Original: Charniak).
Revised: Charniak).

Original: We refer to Table 3 in Appendix H for model speciﬁcations.
Revised: We refer to Table 3 in Appendix H for model speciﬁcations.

Original: To eliminate inﬂuences of a speciﬁc random seed, in our synchronous and asynchronous experiments, the training loss and validation metrics are averaged from 3 runs using different random seeds.
Revised: To eliminate inﬂuences of a speciﬁc random seed, in our synchronous and asynchronous experiments, the training loss and validation metrics are averaged from 3 runs using different random seeds.

Original: 5. S YNCHRONOUS EXPERIMENTS
Revised: 5. S YNCHRONOUS EXPERIMENTS

Original: We tune Adam and momentum SGD on learning rate grids with prescribed momentum 0 .
Revised: We tune Adam and momentum SGD on learning rate grids with prescribed momentum 0 .

Original: 9 for SGD.
Revised: 9 for SGD.

Original: We ﬁx the parameters of Algorithm 1 in all experiments, i.e. Y ELLOW F IN runs without any hand tuning .
Revised: We ﬁx the parameters of Algorithm 1 in all experiments, i.e. Y ELLOW F IN runs without any hand tuning .

Original: We provide full speciﬁcations, including the learning rate (grid) and the number of iterations we train on each model in Appendix I. For visualization purposes, we smooth training losses with a uniform window of width 1000 .
Revised: We provide full speciﬁcations, including the learning rate (grid) and the number of iterations we train on each model in Appendix I. For visualization purposes, we smooth training losses with a uniform window of width 1000 .

Original: For Adam and momentum SGD on each model, we pick the conﬁguration achieving the lowest averaged smoothed loss.
Revised: For Adam and momentum SGD on each model, we pick the conﬁguration achieving the lowest averaged smoothed loss.

Original: To compare two algorithms, we record the lowest smoothed loss achieved by both.
Revised: To compare two algorithms, we record the lowest smoothed loss achieved by both.

Original: Then the speedup is reported as the ratio of iterations to achieve this loss.
Revised: Then the speedup is reported as the ratio of iterations to achieve this loss.

Original: We use this setup to validate our claims.
Revised: We use this setup to validate our claims.

Original: [Table]
Revised: [Table]

Original: Momentum SGD is competitive with adaptive methods
Revised: Momentum SGD is competitive with adaptive methods

Original: In Table 2, we compare tuned momentum SGD and tuned
Revised: In Table 2, we compare tuned momentum SGD and tuned

Original: Adam on ResNets with training losses shown in Figure 9 in
Revised: Adam on ResNets with training losses shown in Figure 9 in

Original: Appendix J. We can observe that momentum SGD achieves 1 .
Revised: Appendix J. We can observe that momentum SGD achieves 1 .

Original: 71 x and 1 .
Revised: 71 x and 1 .

Original: 87 x speedup to tuned Adam on CIFAR10 and
Revised: 87 x speedup to tuned Adam on CIFAR10 and

Original: CIFAR100 respectively.
Revised: CIFAR100 respectively.

Original: In Figure 6 and Table 2, with the exception of PTB LSTM, momentum SGD also produces better training loss, as well as better validation perplexity in language modeling and validation F1 in parsing.
Revised: In Figure 6 and Table 2, with the exception of PTB LSTM, momentum SGD also produces better training loss, as well as better validation perplexity in language modeling and validation F1 in parsing.

Original: For the parsing task, we also compare with tuned Vanilla SGD and
Revised: For the parsing task, we also compare with tuned Vanilla SGD and

Original: AdaGrad, which are used in the NLP community.
Revised: AdaGrad, which are used in the NLP community.

Original: Figure 6 (right) shows that ﬁxed momentum 0.
Revised: Figure 6 (right) shows that ﬁxed momentum 0.

Original: can already speedup Vanilla SGD by 2 .
Revised: can already speedup Vanilla SGD by 2 .

Original: 73 x, achieving observably better validation F1 .
Revised: 73 x, achieving observably better validation F1 .

Original: We refer to
Revised: We refer to

Original: Appendix J.2 for further discussion on the importance of momentum adaptivity in Y ELLOW F IN .
Revised: Appendix J.2 for further discussion on the importance of momentum adaptivity in Y ELLOW F IN .

Original: [Figure]
Revised: [Figure]

Original: Y ELLOW F IN can match hand-tuned momentum SGD and can outperform hand-tuned Adam
Revised: Y ELLOW F IN can match hand-tuned momentum SGD and can outperform hand-tuned Adam

Original: In our experiments, Y ELLOW F IN , without any hand-tuning, yields training loss matching hand-tuned momentum SGD for all the ResNet and LSTM models in Figure 6 and 9.
Revised: In our experiments, Y ELLOW F IN , without any hand-tuning, yields training loss matching hand-tuned momentum SGD for all the ResNet and LSTM models in Figure 6 and 9.

Original: When comparing to tuned
Revised: When comparing to tuned

Original: Adam in Table 2, except being slightly slower on PTB LSTM, Y ELLOW F IN achieves 1 .
Revised: Adam in Table 2, except being slightly slower on PTB LSTM, Y ELLOW F IN achieves 1 .

Original: 38 x to 3 .
Revised: 38 x to 3 .

Original: 28 x speedups in training losses on the other four models.
Revised: 28 x speedups in training losses on the other four models.

Original: More importantly, Y ELLOW F IN consistently shows better validation metrics than tuned Adam in Figure 6 .
Revised: More importantly, Y ELLOW F IN consistently shows better validation metrics than tuned Adam in Figure 6 .

Original: It demonstrates that Y ELLOW F IN can match tuned momentum SGD and outperform tuned state-of-the-art adaptive optimizers.
Revised: It demonstrates that Y ELLOW F IN can match tuned momentum SGD and outperform tuned state-of-the-art adaptive optimizers.

Original: In
Revised: In

Original: Appendix J.4, we show Y ELLOW F IN further speeding up with ﬁner-grain manual learning rate tuning.
Revised: Appendix J.4, we show Y ELLOW F IN further speeding up with ﬁner-grain manual learning rate tuning.

Original: 5. A SYNCHRONOUS EXPERIMENTS
Revised: 5. A SYNCHRONOUS EXPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: In this section, we evaluate closed-loop Y ELLOW F IN with focus on the number of iterations to reach a certain solution.
Revised: In this section, we evaluate closed-loop Y ELLOW F IN with focus on the number of iterations to reach a certain solution.

Original: To that end, we run 16 asynchronous workers on a single machine and force them to update the model in a round-robin fashion, i.e. the gradient is delayed for 15 iterations.
Revised: To that end, we run 16 asynchronous workers on a single machine and force them to update the model in a round-robin fashion, i.e. the gradient is delayed for 15 iterations.

Original: Figure 7 presents training losses on the CIFAR100 ResNet, using Y ELLOW F IN in Algorithm 1, closedloop Y ELLOW F IN in Algorithm 5 and Adam with the learning rate achieving the best smoothed loss in Section 5.1.
Revised: Figure 7 presents training losses on the CIFAR100 ResNet, using Y ELLOW F IN in Algorithm 1, closedloop Y ELLOW F IN in Algorithm 5 and Adam with the learning rate achieving the best smoothed loss in Section 5.1.

Original: We can observe closed-loop Y ELLOW F IN achieves 20 .
Revised: We can observe closed-loop Y ELLOW F IN achieves 20 .

Original: 1 x speedup to Y ELLOW F IN , and consequently a 2 .
Revised: 1 x speedup to Y ELLOW F IN , and consequently a 2 .

Original: 69 x speedup to Adam.
Revised: 69 x speedup to Adam.

Original: This demonstrates that (1) closed-loop Y ELLOW F IN accelerates by reducing algorithmic momentum to compensate for asynchrony and (2) can converge in less iterations than Adam in asynchronous-parallel training.
Revised: This demonstrates that (1) closed-loop Y ELLOW F IN accelerates by reducing algorithmic momentum to compensate for asynchrony and (2) can converge in less iterations than Adam in asynchronous-parallel training.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Many techniques have been proposed on tuning hyperparameters for optimizers.
Revised: Many techniques have been proposed on tuning hyperparameters for optimizers.

Original: General hyperparameter tuning approaches, such as random search (Bergstra and Bengio, 2012) and Bayesian approaches (Snoek et al., 2012; Hutter et al., 2011), directly applies to optimizer tuning.
Revised: General hyperparameter tuning approaches, such as random search (Bergstra and Bengio, 2012) and Bayesian approaches (Snoek et al., 2012; Hutter et al., 2011), directly applies to optimizer tuning.

Original: As another trend, adaptive methods, including AdaGrad (Duchi et al., 2011), RMSProp (Tieleman and Hinton, 2012) and Adam (Chilimbi et al., 2014), uses per-dimension learning rate.
Revised: As another trend, adaptive methods, including AdaGrad (Duchi et al., 2011), RMSProp (Tieleman and Hinton, 2012) and Adam (Chilimbi et al., 2014), uses per-dimension learning rate.

Original: Schaul et al.
Revised: Schaul et al.

Original: (2013) use a noisy quadratic model similar to ours to extract learning rate tuning rule in Vanilla SGD.
Revised: (2013) use a noisy quadratic model similar to ours to extract learning rate tuning rule in Vanilla SGD.

Original: However they do not use momentum which is essential in training modern neural networks.
Revised: However they do not use momentum which is essential in training modern neural networks.

Original: Existing adaptive momentum approach either consider the deterministic setting (Graepel and Schraudolph, 2002; Rehman and Nawi, 2011; Hameed et al., 2016; Swanston et al., 1994; Ampazis and Perantonis, 2000; Qiu et al., 1992) or only analyze stochasticity with O (1 /t ) learning rate (Leen and Orr, 1994).
Revised: Existing adaptive momentum approach either consider the deterministic setting (Graepel and Schraudolph, 2002; Rehman and Nawi, 2011; Hameed et al., 2016; Swanston et al., 1994; Ampazis and Perantonis, 2000; Qiu et al., 1992) or only analyze stochasticity with O (1 /t ) learning rate (Leen and Orr, 1994).

Original: In contrast, we aim at practical momentum adaptivity for stochastically training neural networks.
Revised: In contrast, we aim at practical momentum adaptivity for stochastically training neural networks.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We presented Y ELLOW F IN , the ﬁrst optimization method that automatically tunes momentum as well as the learning rate of momentum SGD.
Revised: We presented Y ELLOW F IN , the ﬁrst optimization method that automatically tunes momentum as well as the learning rate of momentum SGD.

Original: Y ELLOW F IN outperforms the state-of-the-art adaptive optimizers on a large class of models both in synchronous and asynchronous settings.
Revised: Y ELLOW F IN outperforms the state-of-the-art adaptive optimizers on a large class of models both in synchronous and asynchronous settings.

Original: It estimates statistics purely from the gradients of a running system and then tunes the hyperparameters of momentum SGD based on noisy, local quadratic approximations.
Revised: It estimates statistics purely from the gradients of a running system and then tunes the hyperparameters of momentum SGD based on noisy, local quadratic approximations.

Original: As future work, we believe that more accurate curvature estimation methods, like the bbprop method (Martens et al., 2012) can further improve Y ELLOW F IN .
Revised: As future work, we believe that more accurate curvature estimation methods, like the bbprop method (Martens et al., 2012) can further improve Y ELLOW F IN .

Original: We also believe that our closed-loop momentum control mechanism in
Revised: We also believe that our closed-loop momentum control mechanism in

Original: Section 4 could accelerate convergence for other adaptive methods in asynchronous-parallel settings.
Revised: Section 4 could accelerate convergence for other adaptive methods in asynchronous-parallel settings.

Original: A P ROOF OF L EMMA
Revised: A P ROOF OF L EMMA

Original: To prove Lemma 3, we ﬁrst prove a more generalized version in Lemma 7.
Revised: To prove Lemma 3, we ﬁrst prove a more generalized version in Lemma 7.

Original: By restricting f to be a one dimensional quadratics function, the generalized curvature h t itself is the only eigenvalue.
Revised: By restricting f to be a one dimensional quadratics function, the generalized curvature h t itself is the only eigenvalue.

Original: We can prove Lemma 3 as a straight-forward corollary.
Revised: We can prove Lemma 3 as a straight-forward corollary.

Original: Lemma 7 also implies, in the multiple dimensional correspondence of (4), the spectral radius ρ ( A t ) = √ µ if the curvature on all eigenvector directions (eigenvalue) satisﬁes (5).
Revised: Lemma 7 also implies, in the multiple dimensional correspondence of (4), the spectral radius ρ ( A t ) = √ µ if the curvature on all eigenvector directions (eigenvalue) satisﬁes (5).

Original: Lemma 7.
Revised: Lemma 7.

Original: Let the gradients of a function f be described by
Revised: Let the gradients of a function f be described by

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with H ( x t ) ∈ R n (cid:55)→ R n × n .
Revised: with H ( x t ) ∈ R n (cid:55)→ R n × n .

Original: Then the momentum update can be expressed as a linear operator:
Revised: Then the momentum update can be expressed as a linear operator:

Original: [Equation]
Revised: [Equation]

Original: where y t (cid:44) x t − x ∗ .
Revised: where y t (cid:44) x t − x ∗ .

Original: Now, assume that the following condition holds for all eigenvalues λ ( H ( x t )) of H ( x t ) :
Revised: Now, assume that the following condition holds for all eigenvalues λ ( H ( x t )) of H ( x t ) :

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: then the spectral radius of A t is controlled by momentum with ρ ( A t ) = √ µ.
Revised: then the spectral radius of A t is controlled by momentum with ρ ( A t ) = √ µ.

Original: Proof.
Revised: Proof.

Original: Let λ t be an eigenvalue of matrix
Revised: Let λ t be an eigenvalue of matrix

Original: A t , it gives det ( A t − λ t I ) = 0 .
Revised: A t , it gives det ( A t − λ t I ) = 0 .

Original: We deﬁne the blocks in
Revised: We deﬁne the blocks in

Original: A t as C =
Revised: A t as C =

Original: I − α H t + µ
Revised: I − α H t + µ

Original: I − λ t
Revised: I − λ t

Original: I , D = − µ I , E
Revised: I , D = − µ I , E

Original: = I and F = − λ t I which gives
Revised: = I and F = − λ t I which gives

Original: [Equation]
Revised: [Equation]

Original: assuming generally F is invertible.
Revised: assuming generally F is invertible.

Original: Note we use H t (cid:44) H ( x t ) for simplicity in writing.
Revised: Note we use H t (cid:44) H ( x t ) for simplicity in writing.

Original: The equation det (cid:0) C − DF − 1 E (cid:1) = 0 implies that
Revised: The equation det (cid:0) C − DF − 1 E (cid:1) = 0 implies that

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: with M t = ( I − α H t + µ I ) .
Revised: with M t = ( I − α H t + µ I ) .

Original: In other words, λ t satisﬁed that λ 2 t − λ t λ ( M t ) + µ = 0 with λ ( M t ) being one eigenvalue of M t .
Revised: In other words, λ t satisﬁed that λ 2 t − λ t λ ( M t ) + µ = 0 with λ ( M t ) being one eigenvalue of M t .

Original: I.e.
Revised: I.e.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: On the other hand, (17) guarantees that (1 − αλ
Revised: On the other hand, (17) guarantees that (1 − αλ

Original: ( H t )+ µ ) 2 ≤ 4 µ .
Revised: ( H t )+ µ ) 2 ≤ 4 µ .

Original: We know both H t and I − α H t + µ
Revised: We know both H t and I − α H t + µ

Original: I are symmetric.
Revised: I are symmetric.

Original: Thus for all eigenvalues λ ( M t ) of M t , we have λ ( M t ) 

Revised: Thus for all eigenvalues λ ( M t ) of M t , we have λ ( M t )

Original: = (1 − αλ ( H t )+ µ ) 2 ≤ 4 µ which guarantees | λ t | = √ µ for all λ t .
Revised: = (1 − αλ ( H t )+ µ ) 2 ≤ 4 µ which guarantees | λ t | = √ µ for all λ t .

Original: As the spectral radius is equal to the magnitude of the largest eigenvalue of A t , we have the spectral radius of A t being √ µ .
Revised: As the spectral radius is equal to the magnitude of the largest eigenvalue of A t , we have the spectral radius of A t being √ µ .

Original: B P ROOF OF L EMMA
Revised: B P ROOF OF L EMMA

Original: We ﬁrst prove Lemma 8 and Lemma 9 as preparation for the proof of Lemma 5.
Revised: We ﬁrst prove Lemma 8 and Lemma 9 as preparation for the proof of Lemma 5.

Original: After the proof for one dimensional case, we discuss the trivial generalization to multiple dimensional case.
Revised: After the proof for one dimensional case, we discuss the trivial generalization to multiple dimensional case.

Original: Lemma 8.
Revised: Lemma 8.

Original: Let the h be the curvature of a one dimensional quadratic function f and x t = E x t .
Revised: Let the h be the curvature of a one dimensional quadratic function f and x t = E x t .

Original: We assume, without loss of generality, the optimum point of f is x (cid:63) = 0 .
Revised: We assume, without loss of generality, the optimum point of f is x (cid:63) = 0 .

Original: Then we have the following recurrence
Revised: Then we have the following recurrence

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: From the recurrence of momentum SGD, we have
Revised: From the recurrence of momentum SGD, we have

Original: [Equation]
Revised: [Equation]

Original: By putting the equation in to matrix form, (20) is a straight-forward result from unrolling the recurrence for t times.
Revised: By putting the equation in to matrix form, (20) is a straight-forward result from unrolling the recurrence for t times.

Original: Note as we set x 1 = x 0 with no uncertainty in momentum SGD, we have [ x 0 , x 1 ] =
Revised: Note as we set x 1 = x 0 with no uncertainty in momentum SGD, we have [ x 0 , x 1 ] =

Original: [ x 0 , x 1 ] .
Revised: [ x 0 , x 1 ] .

Original: Lemma 9.
Revised: Lemma 9.

Original: Let U t =
Revised: Let U t =

Original: E ( x t − x t ) 2 and V t =
Revised: E ( x t − x t ) 2 and V t =

Original: E ( x t − x t )( x t − 1 − x t − 1 ) with x t being the expectation of x t .
Revised: E ( x t − x t )( x t − 1 − x t − 1 ) with x t being the expectation of x t .

Original: For quadratic function f ( x ) with curvature h ∈ R , We have the following recurrence
Revised: For quadratic function f ( x ) with curvature h ∈ R , We have the following recurrence

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: and C = E ( ∇ f S t ( x t ) − ∇ f ( x t )) 2 is the variance of gradient on minibatch S t .
Revised: and C = E ( ∇ f S t ( x t ) − ∇ f ( x t )) 2 is the variance of gradient on minibatch S t .

Original: Proof.
Revised: Proof.

Original: We prove by ﬁrst deriving the recurrence for U t and V t respectively and combining them in to a matrix form.
Revised: We prove by ﬁrst deriving the recurrence for U t and V t respectively and combining them in to a matrix form.

Original: For U t , we have
Revised: For U t , we have

Original: [Equation]
Revised: [Equation]

Original: where the cross terms cancels due to the fact E S t [ ∇ f ( x t ) − ∇ f S t ( x t )]
Revised: where the cross terms cancels due to the fact E S t [ ∇ f ( x t ) − ∇ f S t ( x t )]

Original: = 0 in the third equality.
Revised: = 0 in the third equality.

Original: For V t , we can similarly derive
Revised: For V t , we can similarly derive

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Again, the term involving ∇ f ( x t ) − ∇ f S t ( x t ) cancels in the third equality as a results of E
Revised: Again, the term involving ∇ f ( x t ) − ∇ f S t ( x t ) cancels in the third equality as a results of E

Original: S t [ ∇ f ( x t ) − ∇ f S t ( x t )]
Revised: S t [ ∇ f ( x t ) − ∇ f S t ( x t )]

Original: = 0 .
Revised: = 0 .

Original: (23) and (24) can be jointly expressed in the following matrix form (cid:32)
Revised: (23) and (24) can be jointly expressed in the following matrix form (cid:32)

Original: U t +
Revised: U t +

Original: [Equation]
Revised: [Equation]

Original: Note the second term in the second equality is zero because x 0 and x 1 are deterministic.
Revised: Note the second term in the second equality is zero because x 0 and x 1 are deterministic.

Original: Thus
Revised: Thus

Original: U 1 =
Revised: U 1 =

Original: U 

Revised: U

Original: = V 1 =0 .
Revised: = V 1 =0 .

Original: According to Lemma 8 and 9, we have E ( x t − x ∗ ) 

Revised: According to Lemma 8 and 9, we have E ( x t − x ∗ )

Original: =
Revised: =

Original: ( e (cid:62) 

Revised: ( e (cid:62)

Original: A t [ x 1 , x 0 ] (cid:62) ) 2 and E ( x t − x t ) 2 = α 2 C e (cid:62) 

Revised: A t [ x 1 , x 0 ] (cid:62) ) 2 and E ( x t − x t ) 2 = α 2 C e (cid:62)

Original: ( I − B t )( I − B ) − 1 e 1 where e 1 ∈ R n has all zero entries but the ﬁrst dimension.
Revised: ( I − B t )( I − B ) − 1 e 1 where e 1 ∈ R n has all zero entries but the ﬁrst dimension.

Original: Combining these two terms, we prove Lemma 5.
Revised: Combining these two terms, we prove Lemma 5.

Original: Though the proof here is for one dimensional quadratics, it trivially generalizes to multiple dimensional quadratics.
Revised: Though the proof here is for one dimensional quadratics, it trivially generalizes to multiple dimensional quadratics.

Original: Speciﬁcally, we can decompose the quadratics along the eigenvector directions, and then apply Lemma 5 to each eigenvector direction using the corresponding curvature h (eigenvalue).
Revised: Speciﬁcally, we can decompose the quadratics along the eigenvector directions, and then apply Lemma 5 to each eigenvector direction using the corresponding curvature h (eigenvalue).

Original: By summing quantities in (10) for all eigenvector directions, we can achieve the multiple dimensional correspondence of (10).
Revised: By summing quantities in (10) for all eigenvector directions, we can achieve the multiple dimensional correspondence of (10).

Original: C P ROOF OF L EMMA
Revised: C P ROOF OF L EMMA

Original: Again we ﬁrst present a proof of a multiple dimensional generalized version of Lemma 6.
Revised: Again we ﬁrst present a proof of a multiple dimensional generalized version of Lemma 6.

Original: The proof of Lemma 6 is a one dimensional special case of Lemma 10.
Revised: The proof of Lemma 6 is a one dimensional special case of Lemma 10.

Original: Lemma 10 also implies that for multiple dimension quadratics, the corresponding spectral radius ρ ( B ) = µ if (1 −√ µ ) 2 α
Revised: Lemma 10 also implies that for multiple dimension quadratics, the corresponding spectral radius ρ ( B ) = µ if (1 −√ µ ) 2 α

Original: [Equation]
Revised: [Equation]

Original: on all the eigenvector directions with h being the eigenvalue (curvature).
Revised: on all the eigenvector directions with h being the eigenvalue (curvature).

Original: Lemma 10.
Revised: Lemma 10.

Original: Let H ∈ R n × n be a symmetric matrix and ρ ( B ) be the spectral radius of matrix
Revised: Let H ∈ R n × n be a symmetric matrix and ρ ( B ) be the spectral radius of matrix

Original: [Equation]
Revised: [Equation]

Original: We have ρ ( B ) = µ if all eigenvalues λ ( H ) of H satisﬁes
Revised: We have ρ ( B ) = µ if all eigenvalues λ ( H ) of H satisﬁes

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Proof.
Revised: Proof.

Original: Let λ be an eigenvalue of matrix B , it gives det ( B − λ I )
Revised: Let λ be an eigenvalue of matrix B , it gives det ( B − λ I )

Original: = 0 which can be alternatively expressed as
Revised: = 0 which can be alternatively expressed as

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: assuming F is invertible, i.e. λ + µ (cid:54) = 0 , where the blocks in B
Revised: assuming F is invertible, i.e. λ + µ (cid:54) = 0 , where the blocks in B

Original: [Equation]
Revised: [Equation]

Original: with M =
Revised: with M =

Original: I − α H + µ I .
Revised: I − α H + µ I .

Original: (28) can be transformed using straight-forward algebra as
Revised: (28) can be transformed using straight-forward algebra as

Original: [Equation]
Revised: [Equation]

Original: Using similar simpliﬁcation technique as in (28), we can further simplify into
Revised: Using similar simpliﬁcation technique as in (28), we can further simplify into

Original: [Equation]
Revised: [Equation]

Original: if λ (cid:54) = µ , as ( λ + µ ) 2 I − λ M (cid:62) M is diagonalizable, we have ( λ + µ ) 2 − λλ ( M ) 2 = 0 with λ ( M ) being an eigenvalue of symmetric M .
Revised: if λ (cid:54) = µ , as ( λ + µ ) 2 I − λ M (cid:62) M is diagonalizable, we have ( λ + µ ) 2 − λλ ( M ) 2 = 0 with λ ( M ) being an eigenvalue of symmetric M .

Original: The analytic solution to the equation can be explicitly expressed as
Revised: The analytic solution to the equation can be explicitly expressed as

Original: [Equation]
Revised: [Equation]

Original: When the condition in (27) holds, we have λ ( M ) 2 =
Revised: When the condition in (27) holds, we have λ ( M ) 2 =

Original: (1 − αλ ( H ) + µ ) 2 ≤ 4 µ .
Revised: (1 − αλ ( H ) + µ ) 2 ≤ 4 µ .

Original: One can verify that
Revised: One can verify that

Original: [Equation]
Revised: [Equation]

Original: Thus the roots in (31) are conjugate with | λ |
Revised: Thus the roots in (31) are conjugate with | λ |

Original: = µ .
Revised: = µ .

Original: In conclusion, the condition in (27) can guarantee all the eigenvalues of B has magnitude µ .
Revised: In conclusion, the condition in (27) can guarantee all the eigenvalues of B has magnitude µ .

Original: Thus the spectral radius of B is controlled by µ .
Revised: Thus the spectral radius of B is controlled by µ .

Original: D A NALYTICAL SOLUTION TO (14)
Revised: D A NALYTICAL SOLUTION TO (14)

Original: The problem in (14) does not need iterative solver but has an analytical solution.
Revised: The problem in (14) does not need iterative solver but has an analytical solution.

Original: Substituting only the second constraint, the objective becomes p ( x ) =
Revised: Substituting only the second constraint, the objective becomes p ( x ) =

Original: x 2 D 2 +(1 − x ) 4 /h 2min C with x = √ µ ∈ [0 , 1) .
Revised: x 2 D 2 +(1 − x ) 4 /h 2min C with x = √ µ ∈ [0 , 1) .

Original: By setting the gradient of p ( x ) to 0, we can get a cubic equation whose root x = √ µ p can be computed in closed form using Vieta’s substitution.
Revised: By setting the gradient of p ( x ) to 0, we can get a cubic equation whose root x = √ µ p can be computed in closed form using Vieta’s substitution.

Original: As p ( x ) is uni-modal exactly maximum
Revised: As p ( x ) is uni-modal exactly maximum

Original: [Figure]
Revised: [Figure]

Original: E P RACTICAL IMPLEMENTATION
Revised: E P RACTICAL IMPLEMENTATION

Original: In Section 3.2, we discuss estimators for learning rate and momentum tuning in Y ELLOW F IN .
Revised: In Section 3.2, we discuss estimators for learning rate and momentum tuning in Y ELLOW F IN .

Original: In our experiment practice, we have identiﬁed a few practical implementation details which are important for improving estimators.
Revised: In our experiment practice, we have identiﬁed a few practical implementation details which are important for improving estimators.

Original: Zero-debias is proposed by Kingma and Ba (2014), which accelerates the process where exponential average adapts to the level of original quantity in the beginning.
Revised: Zero-debias is proposed by Kingma and Ba (2014), which accelerates the process where exponential average adapts to the level of original quantity in the beginning.

Original: We applied zero-debias to all the exponential average quantities involved in our estimators.
Revised: We applied zero-debias to all the exponential average quantities involved in our estimators.

Original: In some
Revised: In some

Original: LSTM models, we observe that our estimated curvature may decrease quickly along the optimization process.
Revised: LSTM models, we observe that our estimated curvature may decrease quickly along the optimization process.

Original: In order to better estimate extremal curvature h max and h min with fast decreasing trend, we apply zero-debias exponential average on the logarithmic of h max ,t and h min ,t , instead of directly on h max ,t and h min ,t .
Revised: In order to better estimate extremal curvature h max and h min with fast decreasing trend, we apply zero-debias exponential average on the logarithmic of h max ,t and h min ,t , instead of directly on h max ,t and h min ,t .

Original: Except from the above two techniques, we also implemented the slow start heuristic proposed by (Schaul et al., 2013).
Revised: Except from the above two techniques, we also implemented the slow start heuristic proposed by (Schaul et al., 2013).

Original: More speciﬁcally, we use α = min { α t , t · α t / (10 · w ) } as our learning rate with w as the size of our sliding window in h max and h min estimation.
Revised: More speciﬁcally, we use α = min { α t , t · α t / (10 · w ) } as our learning rate with w as the size of our sliding window in h max and h min estimation.

Original: It discount the learning rate in the ﬁrst 10 · w steps and helps to keep the learning rate small in the beginning when the exponential averaged quantities are not accurate enough.
Revised: It discount the learning rate in the ﬁrst 10 · w steps and helps to keep the learning rate small in the beginning when the exponential averaged quantities are not accurate enough.

Original: F A DAPTIVE GRADIENT CLIPPING IN Y ELLOW F IN
Revised: F A DAPTIVE GRADIENT CLIPPING IN Y ELLOW F IN

Original: Gradient clipping has been established in literature as a standard—almost necessary—tool for training such objectives (Pascanu et al., 2013; Goodfellow et al., 2016;
Revised: Gradient clipping has been established in literature as a standard—almost necessary—tool for training such objectives (Pascanu et al., 2013; Goodfellow et al., 2016;

Original: Gehring et al., 2017).
Revised: Gehring et al., 2017).

Original: However, the classic tradeoff between adaptivity and stability applies: setting a clipping threshold that is too low can hurt performance; setting it to be high, can compromise stability.
Revised: However, the classic tradeoff between adaptivity and stability applies: setting a clipping threshold that is too low can hurt performance; setting it to be high, can compromise stability.

Original: Y ELLOW F IN , keeps running estimates of extremal gradient magnitude squares, h max and h min in order to estimate a generalized condition number.
Revised: Y ELLOW F IN , keeps running estimates of extremal gradient magnitude squares, h max and h min in order to estimate a generalized condition number.

Original: We posit that √ h max is an ideal gradient norm threshold for adaptive clipping.
Revised: We posit that √ h max is an ideal gradient norm threshold for adaptive clipping.

Original: In order to ensure robustness to extreme gradient spikes, like the ones in Figure 4, we also limit the growth rate of the envelope h max in Algorithm 2 as follows:
Revised: In order to ensure robustness to extreme gradient spikes, like the ones in Figure 4, we also limit the growth rate of the envelope h max in Algorithm 2 as follows:

Original: [Equation]
Revised: [Equation]

Original: Our heuristics follows along the lines of classic recipes like Pascanu et al.
Revised: Our heuristics follows along the lines of classic recipes like Pascanu et al.

Original: However, instead of using the average gradient norm to clip, it uses a running estimate of the maximum norm h max .
Revised: However, instead of using the average gradient norm to clip, it uses a running estimate of the maximum norm h max .

Original: In Section 3.3, we saw that adaptive clipping stabilizes the training on objectives that exhibit exploding gradients.
Revised: In Section 3.3, we saw that adaptive clipping stabilizes the training on objectives that exhibit exploding gradients.

Original: In Figure 8, we demonstrate that the adaptive clipping does not hurt performance on models that do not exhibit instabilities without clipping.
Revised: In Figure 8, we demonstrate that the adaptive clipping does not hurt performance on models that do not exhibit instabilities without clipping.

Original: Speciﬁcally, for both PTB LSTM and CIFAR
Revised: Speciﬁcally, for both PTB LSTM and CIFAR

Original: ResNet, the difference between Y ELLOW F IN with and without adaptive clipping diminishes quickly.
Revised: ResNet, the difference between Y ELLOW F IN with and without adaptive clipping diminishes quickly.

Original: G C LOSED - LOOP Y ELLOW F IN FOR ASYNCHRONOUS TRAINING
Revised: G C LOSED - LOOP Y ELLOW F IN FOR ASYNCHRONOUS TRAINING

Original: In Section 4, we brieﬂy discuss the closed-loop momentum control mechanism in closed-loop
Revised: In Section 4, we brieﬂy discuss the closed-loop momentum control mechanism in closed-loop

Original: Y ELLOW F IN .
Revised: Y ELLOW F IN .

Original: In this section, after presenting more preliminaries on asynchrony, we show with details on the mechanism: it measures the dynamics on a running system and controls momentum with a negative feedback loop.
Revised: In this section, after presenting more preliminaries on asynchrony, we show with details on the mechanism: it measures the dynamics on a running system and controls momentum with a negative feedback loop.

Original: Preliminaries Asynchrony is a popular parallelization technique (Niu et al., 2011) that avoids synchronization barriers.
Revised: Preliminaries Asynchrony is a popular parallelization technique (Niu et al., 2011) that avoids synchronization barriers.

Original: When training on M asynchronous workers, staleness (the number of model updates between a worker’s read and write operations) is on average τ =
Revised: When training on M asynchronous workers, staleness (the number of model updates between a worker’s read and write operations) is on average τ =

Original: M − 1 , i.e., the gradient in the SGD update is delayed by τ iterations as ∇ f S t − τ ( x t − τ ) .
Revised: M − 1 , i.e., the gradient in the SGD update is delayed by τ iterations as ∇ f S t − τ ( x t − τ ) .

Original: Asynchrony yields faster steps, but can increase the number of iterations to achieve the same solution, a tradeoff between hardware and statistical efﬁciency (Zhang and Ré, 2014).
Revised: Asynchrony yields faster steps, but can increase the number of iterations to achieve the same solution, a tradeoff between hardware and statistical efﬁciency (Zhang and Ré, 2014).

Original: Mitliagkas et al.
Revised: Mitliagkas et al.

Original: (2016) interpret asynchrony as added momentum dynamics.
Revised: (2016) interpret asynchrony as added momentum dynamics.

Original: Experiments in Hadjis et al.
Revised: Experiments in Hadjis et al.

Original: (2016) support this ﬁnding, and demonstrate that reducing algorithmic momentum can compensate for asynchrony-induced momentum and signiﬁcantly reduce the number of iterations for convergence.
Revised: (2016) support this ﬁnding, and demonstrate that reducing algorithmic momentum can compensate for asynchrony-induced momentum and signiﬁcantly reduce the number of iterations for convergence.

Original: Motivated by that result, we use the model in (34), where the total momentum, µ T , includes both asynchrony-induced and algorithmic momentum, µ , in (1).
Revised: Motivated by that result, we use the model in (34), where the total momentum, µ T , includes both asynchrony-induced and algorithmic momentum, µ , in (1).

Original: [Equation]
Revised: [Equation]

Original: We will use this expression to design an estimator for the value of total momentum, ˆ µ T .
Revised: We will use this expression to design an estimator for the value of total momentum, ˆ µ T .

Original: This estimator is a basic building block of closed-loop Y ELLOW F IN , that removes the need to manually compensate for the effects of asynchrony .
Revised: This estimator is a basic building block of closed-loop Y ELLOW F IN , that removes the need to manually compensate for the effects of asynchrony .

Original: Measuring the momentum dynamics Closed-loop Y ELLOW F IN estimates total momentum µ T on a running system and uses a negative feedback loop to adjust algorithmic momentum accordingly.
Revised: Measuring the momentum dynamics Closed-loop Y ELLOW F IN estimates total momentum µ T on a running system and uses a negative feedback loop to adjust algorithmic momentum accordingly.

Original: Equation (14) gives an estimate of ˆ µ T on a system with staleness τ , based on (14).
Revised: Equation (14) gives an estimate of ˆ µ T on a system with staleness τ , based on (14).

Original: [Equation]
Revised: [Equation]

Original: median
Revised: median

Original: [Equation]
Revised: [Equation]

Original: (cid:19) (35)
Revised: (cid:19) (35)

Original: We use τ -stale model values to match the staleness of the gradient, and perform all operations in an elementwise fashion.
Revised: We use τ -stale model values to match the staleness of the gradient, and perform all operations in an elementwise fashion.

Original: This way we get a total momentum measurement from each variable; the median combines them into a more robust estimate.
Revised: This way we get a total momentum measurement from each variable; the median combines them into a more robust estimate.

Original: Closing the asynchrony loop Given a reliable measurement of µ T , we can use it to adjust the value of algorithmic momentum so that the total momentum matches the target momentum as decided by Y ELLOW F IN in Algorithm 1.
Revised: Closing the asynchrony loop Given a reliable measurement of µ T , we can use it to adjust the value of algorithmic momentum so that the total momentum matches the target momentum as decided by Y ELLOW F IN in Algorithm 1.

Original: Closed-loop Y ELLOW F IN in Algorithm 5 uses a simple negative feedback loop to achieve the adjustment.
Revised: Closed-loop Y ELLOW F IN in Algorithm 5 uses a simple negative feedback loop to achieve the adjustment.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: H M ODEL SPECIFICATION
Revised: H M ODEL SPECIFICATION

Original: The model speciﬁcation is shown in Table 3 for all the experiments in Section 5.
Revised: The model speciﬁcation is shown in Table 3 for all the experiments in Section 5.

Original: CIRAR

Revised: CIRAR

Original: ResNet uses the regular ResNet units while CIFAR100 ResNet uses the bottleneck units.
Revised: ResNet uses the regular ResNet units while CIFAR100 ResNet uses the bottleneck units.

Original: Only the convolutional layers are shown with ﬁlter size, ﬁlter number as well as the repeating count of the units.
Revised: Only the convolutional layers are shown with ﬁlter size, ﬁlter number as well as the repeating count of the units.

Original: The layer counting for ResNets also includes batch normalization and Relu layers.
Revised: The layer counting for ResNets also includes batch normalization and Relu layers.

Original: The LSTM models are also diversiﬁed for different tasks with different vocabulary sizes, word embedding dimensions and number of layers.
Revised: The LSTM models are also diversiﬁed for different tasks with different vocabulary sizes, word embedding dimensions and number of layers.

Original: I S PECIFICATION FOR SYNCHRONOUS EXPERIMENTS
Revised: I S PECIFICATION FOR SYNCHRONOUS EXPERIMENTS

Original: In Section 5.1, we demonstrate the synchronous experiments with extensive discussions.
Revised: In Section 5.1, we demonstrate the synchronous experiments with extensive discussions.

Original: For the reproducibility, we provide here the speciﬁcation of learning rate grids.
Revised: For the reproducibility, we provide here the speciﬁcation of learning rate grids.

Original: The number of iterations as
Revised: The number of iterations as

Original: [Table]
Revised: [Table]

Original: well as epochs
Revised: well as epochs

Original: , i
Revised: , i

Original: [Table]
Revised: [Table]

Original: e. the number of passes over the full training sets, are also listed for completeness.
Revised: e. the number of passes over the full training sets, are also listed for completeness.

Original: For Y ELLOW F IN in all the experiments in Section 5, we uniformly use sliding window size 20 for extremal curvature estimation and β = 0 .
Revised: For Y ELLOW F IN in all the experiments in Section 5, we uniformly use sliding window size 20 for extremal curvature estimation and β = 0 .

Original: 999 for smoothing.
Revised: 999 for smoothing.

Original: For momentum SGD and Adam, we use the following conﬁgurations.
Revised: For momentum SGD and Adam, we use the following conﬁgurations.

Original: • CIFAR

Revised: • CIFAR

Original: ResNet – 40 k iterations ( ∼ 114 epochs) –
Revised: ResNet – 40 k iterations ( ∼ 114 epochs) –

Original: Momentum SGD learning rates { 0 . 001 , 0 .
Revised: Momentum SGD learning rates { 0 . 001 , 0 .

Original: 01 (best) , 0 . 1 , 1 . 0 } , momentum 0.
Revised: 01 (best) , 0 . 1 , 1 . 0 } , momentum 0.

Original: – Adam learning rates { 0 . 0001 , 0 .
Revised: – Adam learning rates { 0 . 0001 , 0 .

Original: 001 (best) , 0 . 01 , 0 . 1 } • CIFAR100 ResNet – 120 k iterations ( ∼ 341 epochs) –
Revised: 001 (best) , 0 . 01 , 0 . 1 } • CIFAR100 ResNet – 120 k iterations ( ∼ 341 epochs) –

Original: Momentum SGD learning rates { 0 . 001 , 0 .
Revised: Momentum SGD learning rates { 0 . 001 , 0 .

Original: 01 (best) , 0 . 1 , 1 . 0 } , momentum 0.
Revised: 01 (best) , 0 . 1 , 1 . 0 } , momentum 0.

Original: – Adam learning rates { 0 . 00001 , 0 . 0001 (best) , 0 . 001 , 0 .
Revised: – Adam learning rates { 0 . 00001 , 0 . 0001 (best) , 0 . 001 , 0 .

Original: 01 } • PTB LSTM – 30k iterations ( ∼ 13 epochs) –
Revised: 01 } • PTB LSTM – 30k iterations ( ∼ 13 epochs) –

Original: Momentum SGD learning rates { 0 . 01 , 0 . 1 , 1 . 

Revised: Momentum SGD learning rates { 0 . 01 , 0 . 1 , 1 .

Original: (best) , 10 . 0 } , momentum 0.
Revised: (best) , 10 . 0 } , momentum 0.

Original: – Adam learning rates { 0 . 0001 , 0 .
Revised: – Adam learning rates { 0 . 0001 , 0 .

Original: 001 (best) , 0 . 01 , 0 . 1 } • TS LSTM – ∼ 21 k iterations ( 50 epochs) –
Revised: 001 (best) , 0 . 01 , 0 . 1 } • TS LSTM – ∼ 21 k iterations ( 50 epochs) –

Original: Momentum SGD learning rates { 0 . 05 , 0 . 1 , 0 . 5 , 1 . 

Revised: Momentum SGD learning rates { 0 . 05 , 0 . 1 , 0 . 5 , 1 .

Original: (best) , 5 . 0 } , momentum 0.
Revised: (best) , 5 . 0 } , momentum 0.

Original: – Adam learning rates { 0 . 0005 , 0 . 001 , 0 .
Revised: – Adam learning rates { 0 . 0005 , 0 . 001 , 0 .

Original: 005 (best) , 0 . 01 , 0 . 05 } – Decrease learning rate by factor 0.97 every epoch for all optimizers, following the design by Karpathy et al. (2015).
Revised: 005 (best) , 0 . 01 , 0 . 05 } – Decrease learning rate by factor 0.97 every epoch for all optimizers, following the design by Karpathy et al. (2015).

Original: • WSJ LSTM – ∼ 120 k iterations ( 50 epochs) –
Revised: • WSJ LSTM – ∼ 120 k iterations ( 50 epochs) –

Original: Momentum SGD learning rates { 0 . 05 , 0 . 1 , 0 .
Revised: Momentum SGD learning rates { 0 . 05 , 0 . 1 , 0 .

Original: 5 (best) , 1 . 0 , 5 . 0 } , momentum 0.
Revised: 5 (best) , 1 . 0 , 5 . 0 } , momentum 0.

Original: – Adam learning rates { 0 . 0001 , 0 . 0005 , 0 .
Revised: – Adam learning rates { 0 . 0001 , 0 . 0005 , 0 .

Original: 001 (best) , 0 .
Revised: 001 (best) , 0 .

Original: 01 } – Vanilla SGD learning rates { 0 . 05 , 0 . 1 , 0 . 5 , 1 . 

Revised: 01 } – Vanilla SGD learning rates { 0 . 05 , 0 . 1 , 0 . 5 , 1 .

Original: (best) , 5 . 0 }
Revised: (best) , 5 . 0 }

Original: – Adagrad learning rates { 0 . 05 , 0 . 1 , 0 .
Revised: – Adagrad learning rates { 0 . 05 , 0 . 1 , 0 .

Original: 5( best ) , 1 . 0 , 5 . 0 } – Decrease learning rate by factor 0.9 every epochs after 14 epochs for all optimizers, following the design by Choe and Charniak.
Revised: 5( best ) , 1 . 0 , 5 . 0 } – Decrease learning rate by factor 0.9 every epochs after 14 epochs for all optimizers, following the design by Choe and Charniak.

Original: J A DDITIONAL EXPERIMENT RESULTS J. T RAINING LOSSES ON CIFAR10 AND CIFAR100 R ES N ET
Revised: J A DDITIONAL EXPERIMENT RESULTS J. T RAINING LOSSES ON CIFAR10 AND CIFAR100 R ES N ET

Original: In Figure we demonstrate the training loss on CIFAR10 ResNet and CIFAR100 ResNet.
Revised: In Figure we demonstrate the training loss on CIFAR10 ResNet and CIFAR100 ResNet.

Original: Speciﬁcally, Y ELLOW F IN can match the performance of hand-tuned momentum SGD, and achieves 1.93x and 1.38x speedup comparing to hand-tuned Adam respectively on CIFAR10 and CIFAR100 ResNet.
Revised: Speciﬁcally, Y ELLOW F IN can match the performance of hand-tuned momentum SGD, and achieves 1.93x and 1.38x speedup comparing to hand-tuned Adam respectively on CIFAR10 and CIFAR100 ResNet.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: J. I MPORTANCE OF MOMENTUM ADAPTIVITY
Revised: J. I MPORTANCE OF MOMENTUM ADAPTIVITY

Original: To further emphasize the importance of momentum adaptivity in Y ELLOW F IN
Revised: To further emphasize the importance of momentum adaptivity in Y ELLOW F IN

Original: [Figure]
Revised: [Figure]

Original: we run YF on
Revised: we run YF on

Original: CIFAR100 ResNet and TS LSTM.
Revised: CIFAR100 ResNet and TS LSTM.

Original: In the experiments, Y ELLOW F IN tunes the learning rate.
Revised: In the experiments, Y ELLOW F IN tunes the learning rate.

Original: Instead of also using the momentum tuned by YF, we continuously feed prescribed momentum value 0 .and 0 .
Revised: Instead of also using the momentum tuned by YF, we continuously feed prescribed momentum value 0 .and 0 .

Original: 9 to the underlying momentum SGD optimizer which YF is tuning.
Revised: 9 to the underlying momentum SGD optimizer which YF is tuning.

Original: In Figure
Revised: In Figure

Original: [Figure]
Revised: [Figure]

Original: when comparing to Y ELLOW F IN with prescribed momentum 0.0 or 0.9, Y ELLOW F IN with adaptively tuned momentum achieves observably faster convergence on both TS LSTM and CIFAR100 ResNet.
Revised: when comparing to Y ELLOW F IN with prescribed momentum 0.0 or 0.9, Y ELLOW F IN with adaptively tuned momentum achieves observably faster convergence on both TS LSTM and CIFAR100 ResNet.

Original: It empirically demonstrates the essential role of momentum adaptivity in Y ELLOW F IN .
Revised: It empirically demonstrates the essential role of momentum adaptivity in Y ELLOW F IN .

Original: J. 
Revised: J.

Original: T UNING MOMENTUM CAN IMPROVE A DAM IN ASYNCHRONOUS
Revised: T UNING MOMENTUM CAN IMPROVE A DAM IN ASYNCHRONOUS

Original: [Figure]
Revised: [Figure]

Original: We conduct experiments on PTB LSTM with 16 asynchronous workers using Adam using the same protocol as in Section 5.2.
Revised: We conduct experiments on PTB LSTM with 16 asynchronous workers using Adam using the same protocol as in Section 5.2.

Original: Fixing the learning rate to the value achieving the lowest smoothed loss in Section 5.1, we sweep the smoothing parameter β 1 (Kingma and Ba, 2014) of the ﬁrst order moment estimate in grid {− 0 . 2 , 0 . 0 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } .
Revised: Fixing the learning rate to the value achieving the lowest smoothed loss in Section 5.1, we sweep the smoothing parameter β 1 (Kingma and Ba, 2014) of the ﬁrst order moment estimate in grid {− 0 . 2 , 0 . 0 , 0 . 3 , 0 . 5 , 0 . 7 , 0 . 9 } .

Original: β 1 serves the same role as momentum in SGD and we call it the momentum in Adam.
Revised: β 1 serves the same role as momentum in SGD and we call it the momentum in Adam.

Original: Figure 11 shows tuning momentum for Adam under asynchrony gives measurably better training loss.
Revised: Figure 11 shows tuning momentum for Adam under asynchrony gives measurably better training loss.

Original: This result emphasizes the importance of momentum tuning in asynchronous settings and suggests that state-of-the-art adaptive methods can perform sub-optimally when using prescribed momentum.
Revised: This result emphasizes the importance of momentum tuning in asynchronous settings and suggests that state-of-the-art adaptive methods can perform sub-optimally when using prescribed momentum.

Original: [Figure]
Revised: [Figure]

Original: Epochs
Revised: Epochs

Original: V a li d a t i o n p e r p l e x i t y
Revised: V a li d a t i o n p e r p l e x i t y

Original: YellowFin
Revised: YellowFin

Original: Adam default
Revised: Adam default

Original: YF searched
Revised: YF searched

Original: Adam searched
Revised: Adam searched

Original: Epochs
Revised: Epochs

Original: V a li d a t i o n a cc u r a c y
Revised: V a li d a t i o n a cc u r a c y

Original: YellowFin
Revised: YellowFin

Original: Adam default
Revised: Adam default

Original: YF searched
Revised: YF searched

Original: Adam searched
Revised: Adam searched

Original: J. A CCELERATING Y ELLOW F IN WITH FINER GRAIN LEARNING RATE TUNING
Revised: J. A CCELERATING Y ELLOW F IN WITH FINER GRAIN LEARNING RATE TUNING

Original: As an adaptive tuner, Y ELLOW F IN does not involve manual tuning.
Revised: As an adaptive tuner, Y ELLOW F IN does not involve manual tuning.

Original: It can present faster development iterations on model architectures than grid search on optimizer hyperparameters.
Revised: It can present faster development iterations on model architectures than grid search on optimizer hyperparameters.

Original: In deep learning practice for computer vision and natural language processing, after ﬁxing the model architecture, extensive optimizer tuning (e.g. grid search or random search) can further improve the performance of a model.
Revised: In deep learning practice for computer vision and natural language processing, after ﬁxing the model architecture, extensive optimizer tuning (e.g. grid search or random search) can further improve the performance of a model.

Original: A natural question to ask is can we also slightly tune Y ELLOW F IN to accelerate convergence and improve the model performance.
Revised: A natural question to ask is can we also slightly tune Y ELLOW F IN to accelerate convergence and improve the model performance.

Original: Speciﬁcally, we can manually multiply a positive number, the learning rate factor, to the auto-tuned learning rate in Y ELLOW F IN to further accelerate.
Revised: Speciﬁcally, we can manually multiply a positive number, the learning rate factor, to the auto-tuned learning rate in Y ELLOW F IN to further accelerate.

Original: In this section, we empirically demonstrate the effectiveness of learning rate factor on a 29layer ResNext (2x64d) (Xie et al., 2016) on CIFAR10 and a Tied LSTM model (Press and
Revised: In this section, we empirically demonstrate the effectiveness of learning rate factor on a 29layer ResNext (2x64d) (Xie et al., 2016) on CIFAR10 and a Tied LSTM model (Press and

Original: Wolf, 2016) with 650 dimensions for word embedding and two hidden units layers on the PTB dataset.
Revised: Wolf, 2016) with 650 dimensions for word embedding and two hidden units layers on the PTB dataset.

Original: When running Y ELLOW F IN , we search for the optimal learning rate factor in grid { 13 , 0 . 5 , 1 , 2( best for ResNext ) , 3( best for Tied LSTM ) , 10 } .
Revised: When running Y ELLOW F IN , we search for the optimal learning rate factor in grid { 13 , 0 . 5 , 1 , 2( best for ResNext ) , 3( best for Tied LSTM ) , 10 } .

Original: Similarly, we search the same learning rate factor grid for Adam, multiplying the factor to its default learning rate 0 .
Revised: Similarly, we search the same learning rate factor grid for Adam, multiplying the factor to its default learning rate 0 .

Original: To further strengthen the performance of Adam as a baseline, we also run it on conventional logarithmic learning rate grid { 5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 } for ResNext and { 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2 } for Tied
Revised: To further strengthen the performance of Adam as a baseline, we also run it on conventional logarithmic learning rate grid { 5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 } for ResNext and { 1 e − 4 , 5 e − 4 , 1 e − 3 , 5 e − 3 , 1 e − 2 } for Tied

Original: LSTM.
Revised: LSTM.

Original: We report the best metric from searching the union of learning rate factor grid and logarithmic learning rate grid as searched Adam results.
Revised: We report the best metric from searching the union of learning rate factor grid and logarithmic learning rate grid as searched Adam results.

Original: Empirically, learning factor 13 and 1 . 0 works best for
Revised: Empirically, learning factor 13 and 1 . 0 works best for

Original: Adam respectively on ResNext and Tied LSTM.
Revised: Adam respectively on ResNext and Tied LSTM.

Original: As shown in Figure 12, with the searched best learning rate factor, Y ELLOW F IN can improve validation perplexity on Tied LSTM from 88 .
Revised: As shown in Figure 12, with the searched best learning rate factor, Y ELLOW F IN can improve validation perplexity on Tied LSTM from 88 .

Original: 7 to 80 .
Revised: 7 to 80 .

Original: 5 , an improvement of more than 9% .
Revised: 5 , an improvement of more than 9% .

Original: Similarly, the searched learning rate factor can improve test accuracy from 92 .
Revised: Similarly, the searched learning rate factor can improve test accuracy from 92 .

Original: 63 to 94 .
Revised: 63 to 94 .

Original: 75 on ResNext.
Revised: 75 on ResNext.

Original: More importantly, we can observe, with learning rate factor search on the two models, Y ELLOW F IN can achieve better validation metric than the searched Adam results.
Revised: More importantly, we can observe, with learning rate factor search on the two models, Y ELLOW F IN can achieve better validation metric than the searched Adam results.

Original: It demonstrates that ﬁner-grain learning rate tuning, i.e. the learning rate factor search, can be effectively applied on Y ELLOW F IN to improve the performance of deep learning models.
Revised: It demonstrates that ﬁner-grain learning rate tuning, i.e. the learning rate factor search, can be effectively applied on Y ELLOW F IN to improve the performance of deep learning models.