Original: F LUCTUATION - DISSIPATION RELATIONS FOR STOCHASTIC GRADIENT DESCENT
Revised: F LUCTUATION - DISSIPATION RELATIONS FOR STOCHASTIC GRADIENT DESCENT

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics.
Revised: The notion of the stationary equilibrium ensemble has played a central role in statistical mechanics.

Original: In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity.
Revised: In machine learning as well, training serves as generalized equilibration that drives the probability distribution of model parameters toward stationarity.

Original: Here, we derive stationary ﬂuctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm.
Revised: Here, we derive stationary ﬂuctuation-dissipation relations that link measurable quantities and hyperparameters in the stochastic gradient descent algorithm.

Original: These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule.
Revised: These relations hold exactly for any stationary state and can in particular be used to adaptively set training schedule.

Original: We can further use the relations to efﬁciently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity.
Revised: We can further use the relations to efﬁciently extract information pertaining to a loss-function landscape such as the magnitudes of its Hessian and anharmonicity.

Original: Our claims are empirically veriﬁed.
Revised: Our claims are empirically veriﬁed

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Equilibration rules the long-term fate of many macroscopic dynamical systems.
Revised: Equilibration rules the long-term fate of many macroscopic dynamical systems.

Original: For instance, as we pour water into a glass and let it be, the stationary state of tranquility is eventually attained.
Revised: For instance, as we pour water into a glass and let it be, the stationary state of tranquility is eventually attained.

Original: Zooming into the tranquil water with a microscope would reveal, however, a turmoil of stochastic ﬂuctuations that maintain the apparent stationarity in balance.
Revised: Zooming into the tranquil water with a microscope would reveal, however, a turmoil of stochastic ﬂuctuations that maintain the apparent stationarity in balance.

Original: This is vividly exempliﬁed by the
Revised: This is vividly exempliﬁed by the

Original: Brownian motion (Brown, 1828): a pollen immersed in water is constantly bombarded by jittery molecular movements, resulting in the macroscopically observable diffusive motion of the solute.
Revised: Brownian motion (Brown, 1828): a pollen immersed in water is constantly bombarded by jittery molecular movements, resulting in the macroscopically observable diffusive motion of the solute.

Original: Out of the effort in bridging microscopic and macroscopic realms through the Brownian movement came a prototype of ﬂuctuation-dissipation relations (Einstein, 1905; Von Smoluchowski, 1906).
Revised: Out of the effort in bridging microscopic and macroscopic realms through the Brownian movement came a prototype of ﬂuctuation-dissipation relations (Einstein, 1905; Von Smoluchowski, 1906).

Original: These relations quantitatively link degrees of noisy microscopic ﬂuctuations to smooth macroscopic dissipative phenomena and have since been codiﬁed in the linear response theory for physical systems (Onsager, 1931; Green, 1954; Kubo, 1957), a cornerstone of statistical mechanics.
Revised: These relations quantitatively link degrees of noisy microscopic ﬂuctuations to smooth macroscopic dissipative phenomena and have since been codiﬁed in the linear response theory for physical systems (Onsager, 1931; Green, 1954; Kubo, 1957), a cornerstone of statistical mechanics.

Original: Machine learning begets another form of equilibration.
Revised: Machine learning begets another form of equilibration.

Original: As a model learns patterns in data, its performance ﬁrst improves and then plateaus, again reaching apparent stationarity.
Revised: As a model learns patterns in data, its performance ﬁrst improves and then plateaus, again reaching apparent stationarity.

Original: This dynamical process naturally comes equipped with stochastic ﬂuctuations as well: often given data too gigantic to consume at once, training proceeds in small batches and random selections of these mini-batches consequently give rise to the noisy dynamical excursion of the model parameters in the loss-function landscape, reminiscent of the Brownian motion.
Revised: This dynamical process naturally comes equipped with stochastic ﬂuctuations as well: often given data too gigantic to consume at once, training proceeds in small batches and random selections of these mini-batches consequently give rise to the noisy dynamical excursion of the model parameters in the loss-function landscape, reminiscent of the Brownian motion.

Original: It is thus natural to wonder if there exist analogous ﬂuctuation-dissipation relations that quantitatively link the noise in mini-batched data to the observable evolution of the model performance and that in turn facilitate the learning process.
Revised: It is thus natural to wonder if there exist analogous ﬂuctuation-dissipation relations that quantitatively link the noise in mini-batched data to the observable evolution of the model performance and that in turn facilitate the learning process.

Original: Here, we derive such ﬂuctuation-dissipation relations for the stochastic gradient descent algorithm.
Revised: Here, we derive such ﬂuctuation-dissipation relations for the stochastic gradient descent algorithm.

Original: The only assumption made is stationarity of the probability distribution that governs the model parameters at sufﬁciently long time.
Revised: The only assumption made is stationarity of the probability distribution that governs the model parameters at sufﬁciently long time.

Original: Our results thus apply to generic cases with non-Gaussian mini-batch noises and nonconvex loss-function landscapes.
Revised: Our results thus apply to generic cases with non-Gaussian mini-batch noises and nonconvex loss-function landscapes.

Original: Practically, the ﬁrst relation (FDR1) offers the metric for assessing equilibration and yields an adaptive algorithm that sets learning-rate schedule on the ﬂy.
Revised: Practically, the ﬁrst relation (FDR1) offers the metric for assessing equilibration and yields an adaptive algorithm that sets learning-rate schedule on the ﬂy.

Original: The second relation (FDR2) further helps us determine the properties of the lossfunction landscape, including the strength of its Hessian and the degree of anharmonicity, i.e., the deviation from the idealized harmonic limit of a quadratic loss surface and a constant noise matrix.
Revised: The second relation (FDR2) further helps us determine the properties of the lossfunction landscape, including the strength of its Hessian and the degree of anharmonicity, i.e., the deviation from the idealized harmonic limit of a quadratic loss surface and a constant noise matrix.

Original: Our approach should be contrasted with recent attempts to import the machinery of stochastic differential calculus into the study of the stochastic gradient descent algorithm
Revised: Our approach should be contrasted with recent attempts to import the machinery of stochastic differential calculus into the study of the stochastic gradient descent algorithm

Original: (Mandt et al., 2015;
Revised: (Mandt et al., 2015;

Original: Li et al., 2015; Mandt et al., 2017; Li et al., 2017; Smith & Le, 2018; Chaudhari & Soatto, 2017;
Revised: Li et al., 2015; Mandt et al., 2017; Li et al., 2017; Smith & Le, 2018; Chaudhari & Soatto, 2017;

Original: Jastrzebski et al., 2017; Zhu et al., 2018; An et al., 2018).
Revised: Jastrzebski et al., 2017; Zhu et al., 2018; An et al., 2018).

Original: This line of work all assumes Gaussian noises and sometimes additionally employs the quadratic harmonic approximation for loss-function landscapes.
Revised: This line of work all assumes Gaussian noises and sometimes additionally employs the quadratic harmonic approximation for loss-function landscapes.

Original: The more severe drawback, however, is the usage of the analogy with continuous-time stochastic differential equations, which is inconsistent in general (see Section 2.3.3).
Revised: The more severe drawback, however, is the usage of the analogy with continuous-time stochastic differential equations, which is inconsistent in general (see Section 2.3.3).

Original: Instead, the stochastic gradient descent algorithm can be properly treated within the framework of the KramersMoyal expansion (Van Kampen, 1992; Gardiner, 2009; Risken, 1984; Radons et al., 1990;
Revised: Instead, the stochastic gradient descent algorithm can be properly treated within the framework of the KramersMoyal expansion (Van Kampen, 1992; Gardiner, 2009; Risken, 1984; Radons et al., 1990;

Original: Leen &
Revised: Leen &

Original: Moody, 1993).
Revised: Moody, 1993).

Original: The paper is organized as follows.
Revised: The paper is organized as follows.

Original: In Section 2, after setting up notations and deriving a stationary ﬂuctuation-dissipation theorem (FDT), we derive two speciﬁc ﬂuctuation-dissipation relations.
Revised: In Section 2, after setting up notations and deriving a stationary ﬂuctuation-dissipation theorem (FDT), we derive two speciﬁc ﬂuctuation-dissipation relations.

Original: The ﬁrst relation (FDR1) can be used to check stationarity and the second relation (FDR2) to delineate the shape of the loss-function landscape, as empirically borne out in Section 3.
Revised: The ﬁrst relation (FDR1) can be used to check stationarity and the second relation (FDR2) to delineate the shape of the loss-function landscape, as empirically borne out in Section 3.

Original: An adaptive scheduling method is proposed and tested in Section 3.3.
Revised: An adaptive scheduling method is proposed and tested in Section 3.3.

Original: We conclude in Section 4 with future outlooks.
Revised: We conclude in Section 4 with future outlooks.

Original:  F LUCTUATION - DISSIPATION RELATIONS
Revised:  F LUCTUATION - DISSIPATION RELATIONS

Original: A model is parametrized by a weight coordinate, θ = { θ i } i =1 ,...,P .
Revised: A model is parametrized by a weight coordinate, θ = { θ i } i =1 ,...,P .

Original: The training set of N s examples is utilized by the model to learn patterns in the data and the model’s overall performance is evaluated by a full-batch loss function, f ( θ ) ≡ 1 N s (cid:80)
Revised: The training set of N s examples is utilized by the model to learn patterns in the data and the model’s overall performance is evaluated by a full-batch loss function, f ( θ ) ≡ 1 N s (cid:80)

Original: N s α
Revised: N s α

Original: =

Revised: =

Original: f α ( θ ) , with f α ( θ ) quantifying the performance of the model on a particular sample α : the smaller the loss is, the better the model is expected to perform.
Revised: f α ( θ ) , with f α ( θ ) quantifying the performance of the model on a particular sample α : the smaller the loss is, the better the model is expected to perform.

Original: The learning process can thus be cast as an optimization problem of minimizing the loss function.
Revised: The learning process can thus be cast as an optimization problem of minimizing the loss function.

Original: One of the most commonly used optimization schemes is the stochastic gradient descent (SGD) algorithm (Robbins & Monro, 1951) in which a mini-batch B ⊂ { 1 , 2 , . . .
Revised: One of the most commonly used optimization schemes is the stochastic gradient descent (SGD) algorithm (Robbins & Monro, 1951) in which a mini-batch B ⊂ { 1 , 2 , . . .

Original: , N s } of size |B| is stochastically chosen for training at each time step.
Revised: , N s } of size |B| is stochastically chosen for training at each time step.

Original: Speciﬁcally, the update equation is given by
Revised: Speciﬁcally, the update equation is given by

Original: [Equation]
Revised: [Equation]

Original: where η > 0 is a learning rate and a mini-batch loss f B [Equation] θ ) ≡ 1 |B| [Equation]
Revised: where η > 0 is a learning rate and a mini-batch loss f B ( θ ) ≡ 1 |B| (cid:80) α ∈B f α ( θ ) .

Original: ∈B f α (

Original: [Equation]

Original: Note that
Revised: Note that

Original: [Equation]
Revised: [Equation]

Original:  with (cid:74) . . .
Revised: ) with (cid:74) . . .

Original: (cid:75) m .
Revised: (cid:75) m .

Original: b .
Revised: b .

Original: denoting the average over mini-batch realizations.
Revised: denoting the average over mini-batch realizations.

Original: For later purposes, it is convenient to deﬁne a full two-point noise matrix (cid:101) C through
Revised: For later purposes, it is convenient to deﬁne a full two-point noise matrix (cid:101) C through

Original: [Equation]
Revised: [Equation]

Original: and, more generally, higher-point noise tensors
Revised: and, more generally, higher-point noise tensors

Original: [Equation]
Revised: [Equation]

Original: Below, we shall not make any assumptions on the distribution of the noise vector ∇ f B – other than that a mini-batch is independent and identically distributed from the N s training samples at each time step – and the noise distribution is therefore allowed to have nontrivial higher connected moments indicative of non-Gaussianity.
Revised: Below, we shall not make any assumptions on the distribution of the noise vector ∇ f B – other than that a mini-batch is independent and identically distributed from the N s training samples at each time step – and the noise distribution is therefore allowed to have nontrivial higher connected moments indicative of non-Gaussianity.

Original: It is empirically often observed that the performance of the model plateaus after some training through SGD.
Revised: It is empirically often observed that the performance of the model plateaus after some training through SGD.

Original: It is thus natural to hypothesize the existence of a stationary-state distribution, p ss ( θ ) , that dictates the SGD sampling at long time (see Section 2.3.4 for discussion on this assumption).
Revised: It is thus natural to hypothesize the existence of a stationary-state distribution, p ss ( θ ) , that dictates the SGD sampling at long time (see Section 2.3.4 for discussion on this assumption).

Original: For any observable quantity, O ( θ ) , – something that can be measured during training such as θand f ( θ ) – its stationary-state average is then deﬁned as
Revised: For any observable quantity, O ( θ ) , – something that can be measured during training such as θand f ( θ ) – its stationary-state average is then deﬁned as

Original: [Equation]
Revised: [Equation]

Original: In general the probability distribution of the model parameters evolves as p ( θ , t + 1)
Revised: In general the probability distribution of the model parameters evolves as p ( θ , t + 1)

Original: = (cid:113) (cid:82) d θ (cid:48)
Revised: = (cid:113) (cid:82) d θ (cid:48)

Original: p
Revised: [Equation] [Equation]

Original: θ (cid:48) , t )
Revised: (cid:48) , [Equation]

Original: δ
Revised: [Equation] (cid:8)

Original: [Equation]
Revised: [Equation]

Original: θ − (cid:2) θ
Revised: − (cid:2) [Equation] (cid:48)

Original: [Equation]
Revised: [Equation]

Original: f B ( θ (cid:48) )
Revised: B ( θ (cid:48) )

Original: (cid:3)(cid:9) (cid:121) m .
Revised: (cid:3)(cid:9) (cid:121) m .

Original: b .
Revised: b .

Original: and in particular for the stationary state (cid:90)
Revised: and in particular for the stationary state (cid:90)

Original: [Equation]
Revised: [Equation]

Original: p

Original: [Equation]

Original: [Equation]

Original: Thus follows the master equation
Revised: Thus follows the master equation

Original: [Equation]
Revised: [Equation]

Original: In the next two subsections, we apply this general formula to simple observables in order to derive various stationary ﬂuctuation-dissipation relations.
Revised: In the next two subsections, we apply this general formula to simple observables in order to derive various stationary ﬂuctuation-dissipation relations.

Original: Incidentally, the discrete version of the FokkerPlanck equation can be derived through the Kramers-Moyal expansion, considering the more general nonstationary version of the above equation and performing the Taylor expansion in η and repeated integrations by parts (Van Kampen, 1992; Gardiner, 2009; Risken, 1984; Radons et al., 1990; Leen & Moody, 1993).
Revised: Incidentally, the discrete version of the FokkerPlanck equation can be derived through the Kramers-Moyal expansion, considering the more general nonstationary version of the above equation and performing the Taylor expansion in η and repeated integrations by parts (Van Kampen, 1992; Gardiner, 2009; Risken, 1984; Radons et al., 1990; Leen & Moody, 1993).

Original: 2. F IRST FLUCTUATION - DISSIPATION RELATION
Revised: 2. F IRST FLUCTUATION - DISSIPATION RELATION

Original: Applying the master equation (FDT) to the linear observable,
Revised: Applying the master equation (FDT) to the linear observable,

Original: [Equation]
Revised: [Equation]

Original: We thus have
Revised: We thus have

Original: [Equation]
Revised: [Equation]

Original: This is natural because there is no particular direction that the gradient picks on average as the model parameter stochastically bounces around the local minimum or, more generally, wanders around the loss-function landscape according to the stationary distribution.
Revised: This is natural because there is no particular direction that the gradient picks on average as the model parameter stochastically bounces around the local minimum or, more generally, wanders around the loss-function landscape according to the stationary distribution.

Original: Performing similar algebra for the quadratic observable (cid:104)
Revised: Performing similar algebra for the quadratic observable (cid:104)

Original: θ i θ j (cid:105) yields
Revised: θ i θ j (cid:105) yields

Original: [Equation]
Revised: [Equation]

Original: In particular, taking the trace of this matrix-form relation, we obtain
Revised: In particular, taking the trace of this matrix-form relation, we obtain

Original: [Equation]
Revised: [Equation]

Original: More generally, in the case of SGD with momentum µ and dampening ν , whose update equation is given by
Revised: More generally, in the case of SGD with momentum µ and dampening ν , whose update equation is given by

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original:  a similar derivation yields (see Appendix A)
Revised: 11) a similar derivation yields (see Appendix A)

Original: [Equation]
Revised: [Equation]

Original: The last equation reduces to the equation (FDR1) when µ = ν = 0 with v = − ∇ f B .
Revised: The last equation reduces to the equation (FDR1) when µ = ν = 0 with v = − ∇ f B .

Original: Also note that (cid:104) θ · ( ∇ f ) (cid:105) =
Revised: Also note that (cid:104) θ · ( ∇ f ) (cid:105) =

Original: (cid:104) ( θ − θ c ) · ( ∇ f ) (cid:105) for an arbitrary constant vector θ c because of the equation (8).
Revised: (cid:104) ( θ − θ c ) · ( ∇ f ) (cid:105) for an arbitrary constant vector θ c because of the equation (8).

Original: This ﬁrst ﬂuctuation-dissipation relation is easy to evaluate on the ﬂy during training, exactly holds without any approximation if sampled well from the stationary distribution, and can thus be used as the standard metric to check if learning has plateaued, just as similar relations can be used to check equilibration in Monte Carlo simulations of physical systems (Santen & Krauth, 2000).
Revised: This ﬁrst ﬂuctuation-dissipation relation is easy to evaluate on the ﬂy during training, exactly holds without any approximation if sampled well from the stationary distribution, and can thus be used as the standard metric to check if learning has plateaued, just as similar relations can be used to check equilibration in Monte Carlo simulations of physical systems (Santen & Krauth, 2000).

Original: [It should be cautioned, however, that the ﬂuctuation-dissipation relations are necessary but not sufﬁcient to ensure stationarity (Odriozola & Berthier, 2011).
Revised: [It should be cautioned, however, that the ﬂuctuation-dissipation relations are necessary but not sufﬁcient to ensure stationarity (Odriozola & Berthier, 2011).

Original: Such a metric can in turn be used to schedule changes in hyperparameters, as shall be demonstrated in Section 3.3.
Revised: Such a metric can in turn be used to schedule changes in hyperparameters, as shall be demonstrated in Section 3.3.

Original: 2. S ECOND FLUCTUATION - DISSIPATION RELATION
Revised: 2. S ECOND FLUCTUATION - DISSIPATION RELATION

Original: Applying the master equation (FDT) on the full-batch loss function and Taylor-expanding it in the learning rate η yields the closed-form expression
Revised: Applying the master equation (FDT) on the full-batch loss function and Taylor-expanding it in the learning rate η yields the closed-form expression

Original: [Equation]
Revised: [Equation]

Original: where we recalled the equation (4) and introduced
Revised: where we recalled the equation (4) and introduced

Original: [Equation]
Revised: [Equation]

Original: In particular, H i,j ( θ ) ≡ F i,j ( θ ) is the Hessian matrix.
Revised: In particular, H i,j ( θ ) ≡ F i,j ( θ ) is the Hessian matrix.

Original: Reorganizing terms, we obtain
Revised: Reorganizing terms, we obtain

Original: [Equation]
Revised: [Equation]

Original:  In the case of SGD with momentum and dampening, the left-hand side is replaced by (1 − ν ) (cid:68) ( ∇ f ) 2 (cid:69) − µ (cid:104) v · ∇ f (cid:105) and (cid:101) C
Revised: (cid:43)  . (FDR2) In the case of SGD with momentum and dampening, the left-hand side is replaced by (1 − ν ) (cid:68) ( ∇ f ) 2 (cid:69) − µ (cid:104) v · ∇ f (cid:105) and (cid:101) C

Original: i 1 ,i 2 ,...,i k by more hideous expressions (see Appendix A).
Revised: i 1 ,i 2 ,...,i k by more hideous expressions (see Appendix A).

Original: We can extract at least two types of information on the loss-function landscape by evaluating the dependence of the left-hand side, G ( η ) ≡ (cid:68) ( ∇ f ) 2 (cid:69) , on the learning rate η .
Revised: We can extract at least two types of information on the loss-function landscape by evaluating the dependence of the left-hand side, G ( η ) ≡ (cid:68) ( ∇ f ) 2 (cid:69) , on the learning rate η .

Original: First, in the small learning rate regime, the value of 2 G ( η ) /η approximates Tr (cid:16) H (cid:101) C (cid:17) around a local ravine.
Revised: First, in the small learning rate regime, the value of 2 G ( η ) /η approximates Tr (cid:16) H (cid:101) C (cid:17) around a local ravine.

Original: Second, nonlinearity of G ( η ) at higher η indicates discernible effects of anharmonicity.
Revised: Second, nonlinearity of G ( η ) at higher η indicates discernible effects of anharmonicity.

Original: In such a regime, the
Revised: In such a regime, the

Original: Hessian matrix H cannot be approximated as constant (which also implies that { F i 1 ,i 2 ,...,i k } k> 2 are nontrivial) and/or the noise two-point matrix (cid:101) C cannot be regarded as constant.
Revised: Hessian matrix H cannot be approximated as constant (which also implies that { F i 1 ,i 2 ,...,i k } k> 2 are nontrivial) and/or the noise two-point matrix (cid:101) C cannot be regarded as constant.

Original: Such nonlinearity especially indicates the breakdown of the harmonic approximation, that is, the quadratic truncation of the loss-function landscape, often used to analyze the regime explored at small learning rates.
Revised: Such nonlinearity especially indicates the breakdown of the harmonic approximation, that is, the quadratic truncation of the loss-function landscape, often used to analyze the regime explored at small learning rates.

Original: 2. R EMARKS 2.3. I NTUITION WITHIN THE HARMONIC APPROXIMATION
Revised: 2. R EMARKS 2.3. I NTUITION WITHIN THE HARMONIC APPROXIMATION

Original: In order to gain some intuition about the ﬂuctuation-dissipation relations, let us momentarily employ the harmonic approximation, i.e., assume that there is a local minimum of the loss function at θ = θ (cid:63) and retain only up to quadratic terms of the Taylor expansions around it: f ( θ )
Revised: In order to gain some intuition about the ﬂuctuation-dissipation relations, let us momentarily employ the harmonic approximation, i.e., assume that there is a local minimum of the loss function at θ = θ (cid:63) and retain only up to quadratic terms of the Taylor expansions around it: f ( θ )

Original: ≈ f 

Revised: ≈ f

Original: + 12 (cid:80) Pi,j =1 h i,j
Revised: + 12 (cid:80) Pi,j =1 h i,j

Original: ( θ i − θ (cid:63)i )( θ j − θ (cid:63)j ) .
Revised: ( θ i − θ (cid:63)i )( θ j − θ (cid:63)j ) .

Original: Within this approximation, (cid:104) θ · ( ∇ f ) (cid:105) =
Revised: Within this approximation, (cid:104) θ · ( ∇ f ) (cid:105) =

Original: (cid:104) ( θ − θ (cid:63) ) · ( ∇ f ) (cid:105)
Revised: (cid:104) ( θ − θ (cid:63) ) · ( ∇ f ) (cid:105)

Original: ≈ 2 (cid:104) f − f 0 (cid:105) .
Revised: ≈ 2 (cid:104) f − f 0 (cid:105) .

Original: The relation (FDR1) then becomes (cid:104)
Revised: The relation (FDR1) then becomes (cid:104)

Original: f − f 0 (cid:105)
Revised: f − f 0 (cid:105)

Original: ≈ 14 η (cid:68)
Revised: ≈ 14 η (cid:68)

Original: Tr (cid:101) C (cid:69) , linking the height of the noise ball to the noise amplitude.
Revised: Tr (cid:101) C (cid:69) , linking the height of the noise ball to the noise amplitude.

Original: This is in line with, for instance, the theorem 4.6 of the reference Bottou et al.
Revised: This is in line with, for instance, the theorem 4.6 of the reference Bottou et al.

Original: and substantiates the analogy between SGD and simulated annealing, with the learning rate η – multiplied by Tr (cid:101) C – playing the role of temperature (Bottou, 1991).
Revised: and substantiates the analogy between SGD and simulated annealing, with the learning rate η – multiplied by Tr (cid:101) C – playing the role of temperature (Bottou, 1991).

Original: 2.3. H IGHER - ORDER RELATIONS
Revised: 2.3. H IGHER - ORDER RELATIONS

Original: Additional relations can be derived by repeating similar calculations for higher-order observables.
Revised: Additional relations can be derived by repeating similar calculations for higher-order observables.

Original: For example, at the cubic order,
Revised: For example, at the cubic order,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The systematic investigation of higher-order relations is relegated to future work.
Revised: The systematic investigation of higher-order relations is relegated to future work.

Original: 2.3. SGD (cid:54) = SDE
Revised: [Equation]

Original: There is no limit in which SGD asymptotically reduces to the stochastic differential equation (SDE).
Revised: There is no limit in which SGD asymptotically reduces to the stochastic differential equation (SDE).

Original: In order to take such a limit with continuous time differential d t → 0 + , each SGD update must become inﬁnitesimal.
Revised: In order to take such a limit with continuous time differential d t → 0 + , each SGD update must become inﬁnitesimal.

Original: One may thus try d t ≡ η → 0 + , as in recent work adapting the view that
Revised: One may thus try d t ≡ η → 0 + , as in recent work adapting the view that

Original: SGD = SDE (Mandt et al., 2015; Li et al., 2015; Mandt et al., 2017; Li et al., 2017; Smith & Le, 2018; Chaudhari & Soatto, 2017; Jastrzebski et al., 2017; Zhu et al., 2018;
Revised: SGD = SDE (Mandt et al., 2015; Li et al., 2015; Mandt et al., 2017; Li et al., 2017; Smith & Le, 2018; Chaudhari & Soatto, 2017; Jastrzebski et al., 2017; Zhu et al., 2018;

Original: An et al., 2018).
Revised: An et al., 2018).

Original: But this in turn forces the noise vector with zero mean, ∇ f B − ∇ f , to be multiplied by d t .
Revised: But this in turn forces the noise vector with zero mean, ∇ f B − ∇ f , to be multiplied by d t .

Original: This is in contrast to the scaling √ d t needed for the standard machinery of SDE – It ˆ o-Stratonovich calculus and all that – to apply; the additional factor of d t 1 / 2 makes the effective noise covariance be suppressed by d t and the resulting equation in the continuous-time limit, if anything, would just be an ordinary differential equation without noise 2 [unless noise with the proper scaling is explicitly added as in stochastic gradient Langevin dynamics (Welling & Teh, 2011;
Revised: This is in contrast to the scaling √ d t needed for the standard machinery of SDE – Itˆo-Stratonovich calculus and all that – to apply; the additional factor of d t 1 / 2 makes the effective noise covariance be suppressed by d t and the resulting equation in the continuous-time limit, if anything, would just be an ordinary differential equation without noise 2 [unless noise with the proper scaling is explicitly added as in stochastic gradient Langevin dynamics (Welling & Teh, 2011;

Original: Teh et al., 2016) and natural Langevin dynamics (Marceau-Caron & Ollivier, 2017;
Revised: Teh et al., 2016) and natural Langevin dynamics (Marceau-Caron & Ollivier, 2017;

Original: Nado et al., 2018)].
Revised: Nado et al., 2018)].

Original: In short, the recent work views η = √ η √ d t and sends d t → 0 + while pretending that η is ﬁnite, which is inconsistent.
Revised: In short, the recent work views η = √ η √ d t and sends d t → 0 + while pretending that η is ﬁnite, which is inconsistent.

Original: This is not just a technical subtlety.
Revised: This is not just a technical subtlety.

Original: When unjustiﬁably passing onto the continuous-time Fokker-Planck equation, the diffusive term is incorrectly governed by the connected two-point noise matrix C i,j ( θ ) ≡ (cid:101) C i,j ( θ ) − [ ∂ i f ( θ )]
Revised: When unjustiﬁably passing onto the continuous-time Fokker-Planck equation, the diffusive term is incorrectly governed by the connected two-point noise matrix C i,j ( θ ) ≡ (cid:101) C i,j ( θ ) − [ ∂ i f ( θ )]

Original: [ ∂ j f ( θ )] rather than the full two-point noise matrix (cid:101) C
Revised: [ ∂ j f ( θ )] rather than the full two-point noise matrix (cid:101) C

Original: i,j ( θ ) that appears herein.
Revised: i,j ( θ ) that appears herein.

Original: 3 We must instead employ the discrete-time version of the
Revised: 3 We must instead employ the discrete-time version of the

Original: Fokker-Planck equation derived in references Van Kampen (1992); Gardiner (2009); Risken (1984);
Revised: Fokker-Planck equation derived in references Van Kampen (1992); Gardiner (2009); Risken (1984);

Original: Radons et al.
Revised: Radons et al.

Original: ; Leen & Moody (1993), as has been followed in the equation (6).
Revised: ; Leen & Moody (1993), as has been followed in the equation (6).

Original: 2.3. O N STATIONARITY
Revised: 2.3. O N STATIONARITY

Original: In contrast to statistical mechanics where an equilibrium state is dictated by a handful of thermodynamic variables, in machine learning a stationary state generically depends not only on hyperparameters but also on a part of its learning history.
Revised: In contrast to statistical mechanics where an equilibrium state is dictated by a handful of thermodynamic variables, in machine learning a stationary state generically depends not only on hyperparameters but also on a part of its learning history.

Original: The stationarity assumption made herein, which is codiﬁed in the equation (6), is weaker than the typicality assumption underlying statistical mechanics and can hold even in the presence of lingering memory.
Revised: The stationarity assumption made herein, which is codiﬁed in the equation (6), is weaker than the typicality assumption underlying statistical mechanics and can hold even in the presence of lingering memory.

Original: In the full-batch limit |B| =
Revised: In the full-batch limit |B| =

Original: N s , for instance, any distribution delta-peaked at a local minimum is stationary.
Revised: N s , for instance, any distribution delta-peaked at a local minimum is stationary.

Original: For sufﬁciently small learning rates η as well, it is natural to expect multiple stationary distributions that form disconnected ponds around these minima, which merge upon increasing η and fragment upon decreasing η .
Revised: For sufﬁciently small learning rates η as well, it is natural to expect multiple stationary distributions that form disconnected ponds around these minima, which merge upon increasing η and fragment upon decreasing η .

Original: It is beyond the scope of the present paper to formulate conditions under which stationary distributions exist.
Revised: It is beyond the scope of the present paper to formulate conditions under which stationary distributions exist.

Original: Indeed, if the formulation were too generic, there could be counterexamples to such a putative existence statement.
Revised: Indeed, if the formulation were too generic, there could be counterexamples to such a putative existence statement.

Original: A case in point is a model with the unregularized cross entropy loss, whose model parameters keep cascading toward inﬁnity in order to sharpen its softmax output (Neyshabur et al., 2014; 2017) with logarithmically diverging θ 2 (Soudry et al., 2018).
Revised: A case in point is a model with the unregularized cross entropy loss, whose model parameters keep cascading toward inﬁnity in order to sharpen its softmax output (Neyshabur et al., 2014; 2017) with logarithmically diverging θ 2 (Soudry et al., 2018).

Original: It would be interesting to see if there are any other nontrivial caveats.
Revised: It would be interesting to see if there are any other nontrivial caveats.

Original:  E MPIRICAL TESTS
Revised:  E MPIRICAL TESTS

Original: In this section we empirically bear out our theoretical claims in the last section.
Revised: In this section we empirically bear out our theoretical claims in the last section.

Original: To this end, two simple models of supervised learning are used (see Appendix B for full speciﬁcations): a multilayer perceptron (MLP) learning patterns in the MNIST training data (LeCun et al., 1998) through SGD without momentum and a convolutional neural network (CNN) learning patterns in the CIFAR10 training data (Krizhevsky & Hinton, 2009) through SGD with momentum µ = 0 .
Revised: To this end, two simple models of supervised learning are used (see Appendix B for full speciﬁcations): a multilayer perceptron (MLP) learning patterns in the MNIST training data (LeCun et al., 1998) through SGD without momentum and a convolutional neural network (CNN) learning patterns in the CIFAR10 training data (Krizhevsky & Hinton, 2009) through SGD with momentum µ = 0 .

Original: For both models, the mini-batch size is set to be |B| = 100 , and the training data are shufﬂed at each epoch t = N s |B| ˆ t epoch with ˆ t epoch ∈ N .
Revised: For both models, the mini-batch size is set to be |B| = 100 , and the training data are shufﬂed at each epoch t = N s |B| ˆ t epoch with ˆ t epoch ∈ N .

Original: In order to avoid the overﬁtting cascade mentioned in Section 2.3.4, the L 2 -regularization term 12 λ θ 2 with the weight decay λ = 0 .
Revised: In order to avoid the overﬁtting cascade mentioned in Section 2.3.4, the L 2 -regularization term 12 λ θ 2 with the weight decay λ = 0 .

Original: 01 is included in the loss function f .
Revised: 01 is included in the loss function f .

Original: Before proceeding further, let us deﬁne the half-running average of an observable O as
Revised: Before proceeding further, let us deﬁne the half-running average of an observable O as

Revised: O ( t ) ≡ t − tt (cid:88) t (cid:48)

Original: [Equation]{+= t 
 + O ( t (cid:48) ) with t 
+}
Revised: {+= t 
 + O ( t (cid:48) ) with t 
+}

Revised: = (cid:98) t/ 2 (cid:99) .

Original: This is the average of the observable up to the time step t , with the initial half discarded as containing transient.
Revised: This is the average of the observable up to the time step t , with the initial half discarded as containing transient.

Original: If SGD drives the distribution of the model parameters to stationarity at long time, then [Equation]
Revised: If SGD drives the distribution of the model parameters to stationarity at long time, then lim t →∞ O ( t ) = (cid:104)O(cid:105) .

Original: 3. F IRST FLUCTUATION - DISSIPATION RELATION AND EQUILIBRATION
Revised: 3. F IRST FLUCTUATION - DISSIPATION RELATION AND EQUILIBRATION

Original: In order to assess the proximity to stationarity, deﬁne
Revised: In order to assess the proximity to stationarity, deﬁne

Original: [Equation]
Revised: [Equation]

Original:  (with v replaced by − ∇ f B for SGD without momentum).
Revised: and O [Equation] (17) (with v replaced by − ∇ f B for SGD without momentum).

Original: 4 Both of these observables can easily be measured on the ﬂy at each time step during training and, according to the relation (FDR1’), the running averages of these two observables should converge to each other upon equilibration.
Revised: 4 Both of these observables can easily be measured on the ﬂy at each time step during training and, according to the relation (FDR1’), the running averages of these two observables should converge to each other upon equilibration.

Original: [Figure]
Revised: [Figure]

Revised: O R = (1+ µ ) 2(1 − ν )

Revised: η v 2 , whose stationary-state averages should agree according to the relation (FDR1’).

Original: In order to verify this claim, we ﬁrst train the model with the learning rate η = 0 .
Revised: In order to verify this claim, we ﬁrst train the model with the learning rate η = 0 .

Original: 1 for ˆ t totalepoch =
Revised: 1 for ˆ t totalepoch =

Original: epochs, that is, for t total =
Revised: epochs, that is, for t total =

Original: N s |B| ˆ t totalepoch = 100 N s |B| time steps.
Revised: N s |B| ˆ t totalepoch = 100 N s |B| time steps.

Original: As shown in the ﬁgure 1, the observables
Revised: As shown in the ﬁgure 1, the observables

Original: O L ( t ) and O R ( t ) converge to each other.
Revised: O L ( t ) and O R ( t ) converge to each other.

Original: We then take the model at the end of the initial 100 - [Figure] = 0 . epoch training and sequentially train it further at various learning rates 
Revised: We then take the model at the end of the initial 100epoch training and sequentially train it further at various learning rates η

Original: η (see Appendix B).
Revised: (see Appendix B).

Original: The observables O L ( t ) and O R ( t ) again converge to each other, as plotted in the ﬁgure 2.
Revised: The observables O L ( t ) and O R ( t ) again converge to each other, as plotted in the ﬁgure 2.

Original: Note that the smaller the learning rate is, the longer it takes to equilibrate.
Revised: Note that the smaller the learning rate is, the longer it takes to equilibrate.

Original: 3. S ECOND FLUCTUATION - DISSIPATION RELATION AND SHAPE OF LOSS - FUNCTION LANDSCAPE
Revised: 3. S ECOND FLUCTUATION - DISSIPATION RELATION AND SHAPE OF LOSS - FUNCTION LANDSCAPE

Original: In order to assess the loss-function landscape information from the relation (FDR2), deﬁne
Revised: In order to assess the loss-function landscape information from the relation (FDR2), deﬁne

Original: O

Original: [Equation]
Revised: [Equation]

Original: (18) (with the second term nonexistent for SGD without momentum).
Revised: [Figure] (with the second term nonexistent for SGD without momentum).

Original: 5 Note that ( ∇ f ) 2 is a full-batch – not mini-batch – quantity.
Revised: 5 Note that ( ∇ f ) 2 is a full-batch – not mini-batch – quantity.

Original: Given its computational cost, here we measure this ﬁrst term only at the end of each epoch and take the half-running average over these sparse sample points, discarding the initial half of the run.
Revised: Given its computational cost, here we measure this ﬁrst term only at the end of each epoch and take the half-running average over these sparse sample points, discarding the initial half of the run.

Original: The half-running average of the full-batch observable O FB at the end of sufﬁciently long training, which is a good proxy for (cid:104)O FB (cid:105) , is plotted in the ﬁgure 3 as a function of the learning rate η .
Revised: The half-running average of the full-batch observable O FB at the end of sufﬁciently long training, which is a good proxy for (cid:104)O FB (cid:105) , is plotted in the ﬁgure 3 as a function of the learning rate η .

Original: As predicted by the relation (FDR2), at small learning rates η , the observable (cid:104)O FB (cid:105) approaches zero; its slope – divided by (cid:68) Tr (cid:101) C (cid:69) if preferred – measures the magnitude of the Hessian matrix, component-wise averaged over directions in which the noise preferentially ﬂuctuates.
Revised: As predicted by the relation (FDR2), at small learning rates η , the observable (cid:104)O FB (cid:105) approaches zero; its slope – divided by (cid:68) Tr (cid:101) C (cid:69) if preferred – measures the magnitude of the Hessian matrix, component-wise averaged over directions in which the noise preferentially ﬂuctuates.

Original: Meanwhile, nonlinearity at higher learning rates η measures the degree of anharmonicity experienced over the distribution p ss ( θ ) .
Revised: Meanwhile, nonlinearity at higher learning rates η measures the degree of anharmonicity experienced over the distribution p ss ( θ ) .

Original: We see that anharmonic effects are pronounced especially for the CNN on the
Revised: We see that anharmonic effects are pronounced especially for the CNN on the

Original: CIFAR-10 data even at moderately small learning rates.
Revised: CIFAR-10 data even at moderately small learning rates.

Original: This invalidates the use of the quadratic harmonic approximation for the loss-function landscape and/or the assumption of the constant noise matrix for this model except at very small learning rates.[Figure] on MNIST, µ =0. [Figure]
Revised: This invalidates the use of the quadratic harmonic approximation for the loss-function landscape and/or the assumption of the constant noise matrix for this model except at very small learning rates.

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: CIFAR

Original: [Figure]

Original: 3. F IRST FLUCTUATION - DISSIPATION RELATION AND LEARNING - RATE SCHEDULES
Revised: 3. F IRST FLUCTUATION - DISSIPATION RELATION AND LEARNING - RATE SCHEDULES

Original: Saturation of the relation (FDR1) suggests the learning stationarity, at which point it might be wise to decrease the learning rate η .
Revised: Saturation of the relation (FDR1) suggests the learning stationarity, at which point it might be wise to decrease the learning rate η .

Original: Such scheduling is often carried out in an ad hoc manner but we can now algorithmize this procedure as follows:
Revised: Such scheduling is often carried out in an ad hoc manner but we can now algorithmize this procedure as follows:

Revised: Evaluate the half-running averages O L ( t ) and O R ( t ) at the end of each epoch.

Original: If (cid:12)(cid:12)(cid:12) O L ( t ) O R ( t ) − 1 (cid:12)(cid:12)(cid:12) <
Revised: If (cid:12)(cid:12)(cid:12) O L ( t ) O R ( t ) − 1 (cid:12)(cid:12)(cid:12) <

Original: X , then decrease the learning rate as η → (1 − Y ) η and also set t =for the purpose of evaluating half-running averages.
Revised: X , then decrease the learning rate as η → (1 − Y ) η and also set t =for the purpose of evaluating half-running averages.

Original: Here, two scheduling hyperparameters X and Y are introduced, which control the threshold for saturation of the relation (FDR1) and the amount of decrease in the learning rate, respectively.
Revised: Here, two scheduling hyperparameters X and Y are introduced, which control the threshold for saturation of the relation (FDR1) and the amount of decrease in the learning rate, respectively.

Original: Plotted in the ﬁgure 4 are results for SGD without momentum, with the Xavier initialization (Glorot & Bengio, 2010) and training through (i) preset training schedule with decrease of the learning rate by a factor of 10 for each 100 epochs, (ii) an adaptive scheduler with X = 0 .
Revised: Plotted in the ﬁgure 4 are results for SGD without momentum, with the Xavier initialization (Glorot & Bengio, 2010) and training through (i) preset training schedule with decrease of the learning rate by a factor of 10 for each 100 epochs, (ii) an adaptive scheduler with X = 0 .

Original: 01 ( 1% threshold) and 
Revised: 01 ( 1% threshold) and second term, ensure · ∇ , we measure the half-running ( t )

Revised: [Figure]

Original: Y = 0 .
Revised: Y = 0 .

Original: 1 ( 10% decrease), and (iii) the AMSGrad algorithm (J. Reddi et al., 2018) with the default hyperparameters.
Revised: 1 ( 10% decrease), and (iii) the AMSGrad algorithm (J. Reddi et al., 2018) with the default hyperparameters.

Original: The adaptive scheduler attains comparable accuracies with the preset scheduling at long time and outperforms AMSGrad (see Appendix C for additional simulations).
Revised: The adaptive scheduler attains comparable accuracies with the preset scheduling at long time and outperforms AMSGrad (see Appendix C for additional simulations).

Original: [Figure]
Revised: [Figure]

Original: These two scheduling methods span different subspaces of all the possible schedules.
Revised: These two scheduling methods span different subspaces of all the possible schedules.

Original: The adaptive scheduling method proposed herein has a theoretical grounding and in practice much less dimensionality for tuning of scheduling hyperparameters than the presetting method, thus ameliorating the optimization of scheduling hyperparameters.
Revised: The adaptive scheduling method proposed herein has a theoretical grounding and in practice much less dimensionality for tuning of scheduling hyperparameters than the presetting method, thus ameliorating the optimization of scheduling hyperparameters.

Original: The systematic comparison between the two scheduling methods for state-of-the-arts architectures, and also the comparison with the AMSGrad algorithm for natural language processing tasks, could be a worthwhile avenue to pursue in the future.
Revised: The systematic comparison between the two scheduling methods for state-of-the-arts architectures, and also the comparison with the AMSGrad algorithm for natural language processing tasks, could be a worthwhile avenue to pursue in the future.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we have derived the ﬂuctuation-dissipation relations with no assumptions other than stationarity of the probability distribution.
Revised: In this paper, we have derived the ﬂuctuation-dissipation relations with no assumptions other than stationarity of the probability distribution.

Original: These relations hold exactly even when the noise is nonGaussian and the loss function is nonconvex.
Revised: These relations hold exactly even when the noise is nonGaussian and the loss function is nonconvex.

Original: The relations have been empirically veriﬁed and used to probe the properties of the loss-function landscapes for the simple models.
Revised: The relations have been empirically veriﬁed and used to probe the properties of the loss-function landscapes for the simple models.

Original: The relations further have resulted in the algorithm to adaptively set learning-rate schedule on the ﬂy rather than presetting it in an ad hoc manner.
Revised: The relations further have resulted in the algorithm to adaptively set learning-rate schedule on the ﬂy rather than presetting it in an ad hoc manner.

Original: In addition to systematically testing the performance of this adaptive scheduling algorithm, it would be interesting to investigate non-Gaussianity and noncovexity in more details through higher-point observables, both analytically and numerically.
Revised: In addition to systematically testing the performance of this adaptive scheduling algorithm, it would be interesting to investigate non-Gaussianity and noncovexity in more details through higher-point observables, both analytically and numerically.

Original: It would also be interesting to further elucidate the physics of machine learning by extending our formalism to incorporate nonstationary dynamics, linearly away from stationarity (Onsager, 1931; Green, 1954;
Revised: It would also be interesting to further elucidate the physics of machine learning by extending our formalism to incorporate nonstationary dynamics, linearly away from stationarity (Onsager, 1931; Green, 1954;

Original: Kubo, 1957) and beyond (Jarzynski, 1997; Crooks, 1999), so that it can in particular properly treat overﬁtting cascading dynamics and time-dependent sample distributions. 
Revised: Kubo, 1957) and beyond (Jarzynski, 1997; Crooks, 1999), so that it can in particular properly treat overﬁtting cascading dynamics and time-dependent sample distributions. A CKNOWLEDGMENTS

Revised: The author thanks Ludovic Berthier, L ´ eon Bottou, Guy Gur-Ari, Kunihiko Kaneko, Ari Morcos,

Revised: Dheevatsa Mudigere, Yann Ollivier, Yuandong Tian, and Mark Tygert for discussions.

Revised: Special thanks go to Daniel Adam Roberts who prompted the practical application of the ﬂuctuationdissipation relations, leading to the adaptive method in Section 3.3.

Original: A SGD WITH MOMENTUM AND DAMPENING
Revised: A SGD WITH MOMENTUM AND DAMPENING

Original: For SGD with momentum µ and dampening ν , the update equation is given by
Revised: For SGD with momentum µ and dampening ν , the update equation is given by

Original: [Equation]
Revised: [Equation]

Original: Here v = { v i } i =1 ,...,P is the velocity and η > 0 the learning rate; SGD without momentum is the special case with µ = 0 .
Revised: Here v = { v i } i =1 ,...,P is the velocity and η > 0 the learning rate; SGD without momentum is the special case with µ = 0 .

Original: Again hypothesizing the existence of a stationary-state distribution p ss ( θ , v ) , the stationary-state average of an observable O ( θ , v ) is deﬁned as
Revised: Again hypothesizing the existence of a stationary-state distribution p ss ( θ , v ) , the stationary-state average of an observable O ( θ , v ) is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: Just as in the main text, from the assumed stationarity follows the master equation for SGD with momentum and dampening
Revised: Just as in the main text, from the assumed stationarity follows the master equation for SGD with momentum and dampening

Original: [Equation]
Revised: [Equation]

Original: For the linear observables,
Revised: For the linear observables,

Original: [Equation]
Revised: [Equation]

Original: 23) and
Revised: 23) and

Original: [Equation]
Revised: [Equation]

Original: 24) thus
Revised: 24) thus

Original: [Equation]
Revised: [Equation]

Original: For the quadratic observables
Revised: For the quadratic observables

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Note that the relations (26) and (27) are trivially satisﬁed at each time step if the left-hand side observables are evaluated at one step ahead and thus their being satisﬁed for running averages has nothing to do with equilibration [the same can be said about the relation (23)]; the only nontrivial relation is the equation (28), which is a consequence of setting (cid:104)
Revised: Note that the relations (26) and (27) are trivially satisﬁed at each time step if the left-hand side observables are evaluated at one step ahead and thus their being satisﬁed for running averages has nothing to do with equilibration [the same can be said about the relation (23)]; the only nontrivial relation is the equation (28), which is a consequence of setting (cid:104)

Original: θ i θ j
Revised: θ i θ j

Original: (cid:105) constant of time.
Revised: (cid:105) constant of time.

Original: After taking traces and some rearrangement, we obtain the relation (FDR1’) in the main text.
Revised: After taking traces and some rearrangement, we obtain the relation (FDR1’) in the main text.

Original: For the full-batch loss function, the algebra similar to the one in the main text yields
Revised: For the full-batch loss function, the algebra similar to the one in the main text yields

Original: [Equation]
Revised: [Equation]

Original: B M ODELS AND SIMULATION PROTOCOLS B. MLP ON MNIST THROUGH SGD WITHOUT MOMENTUM
Revised: B M ODELS AND SIMULATION PROTOCOLS B. MLP ON MNIST THROUGH SGD WITHOUT MOMENTUM

Original: The MNIST training data consist of N s = 60000 black-white images of hand-written digits with 28 -by- 28 pixels (LeCun et al., 1998).
Revised: The MNIST training data consist of N s = 60000 black-white images of hand-written digits with 28 -by- 28 pixels (LeCun et al., 1998).

Original: We preprocess the data through an afﬁne transformation such that their mean and variance (over both the training data and pixels) are zero and one, respectively.
Revised: We preprocess the data through an afﬁne transformation such that their mean and variance (over both the training data and pixels) are zero and one, respectively.

Original: Our multilayer perceptron (MLP) consists of a 784 -dimensional input layer followed by a hidden layer of 200 neurons with ReLU activations, another hidden layer of 200 neurons with ReLU activations, and a 10 -dimensional output layer with the softmax activation.
Revised: Our multilayer perceptron (MLP) consists of a 784 -dimensional input layer followed by a hidden layer of 200 neurons with ReLU activations, another hidden layer of 200 neurons with ReLU activations, and a 10 -dimensional output layer with the softmax activation.

Original: The model performance is evaluated by the cross-entropy loss supplemented by the L 2 -regularization term 12 λ θ 2 with the weight decay λ = 0 .
Revised: The model performance is evaluated by the cross-entropy loss supplemented by the L 2 -regularization term 12 λ θ 2 with the weight decay λ = 0 .

Original: Throughout the paper, the MLP is trained on the MNIST data through SGD without momentum.
Revised: Throughout the paper, the MLP is trained on the MNIST data through SGD without momentum.

Original: The data are shufﬂed at each epoch with the mini-batch size |B| = 100 .
Revised: The data are shufﬂed at each epoch with the mini-batch size |B| = 100 .

Original: The MLP is initialized through the Xavier method (Glorot & Bengio, 2010) and trained for ˆ t totalepoch = 100 epochs with the learning rate η = 0 .
Revised: The MLP is initialized through the Xavier method (Glorot & Bengio, 2010) and trained for ˆ t totalepoch = 100 epochs with the learning rate η = 0 .

Original: We then sequentially train it with ( η, ˆ t total epoch )
Revised: We then sequentially train it with ( η, ˆ t total epoch )

Original: =
Revised: =

Original: (0 . 05 , 500) → (0 . 02 , 500)
Revised: (0 . 05 , 500) → (0 . 02 , 500)

Original: → (0 . 01 , 500) → (0 . 005 , 1000) → (0 . 003 , 1000) .
Revised: → (0 . 01 , 500) → (0 . 005 , 1000) → (0 . 003 , 1000) .

Original: This sequential-run protocol is carried out with 4 distinct seeds for the random-number generator used in data shufﬂing, all starting from the common model parameter attained at the end of the initial 100epoch run.
Revised: This sequential-run protocol is carried out with 4 distinct seeds for the random-number generator used in data shufﬂing, all starting from the common model parameter attained at the end of the initial 100epoch run.

Original: The ﬁgure 2 depicts trajectories for one particular seed, while the ﬁgure 3 plots means and error bars over these distinct seeds.
Revised: The ﬁgure 2 depicts trajectories for one particular seed, while the ﬁgure 3 plots means and error bars over these distinct seeds.

Original: B. CNN ON CIFAR-10 THROUGH SGD WITH MOMENTUM
Revised: B. CNN ON CIFAR-10 THROUGH SGD WITH MOMENTUM

Original: The CIFAR-10 training data consist of N s = 50000 color images of objects – divided into ten categories – with 32 -by- 32 pixels in each of 3 color channels, each pixel ranging in [0 , 1] (Krizhevsky & Hinton, 2009).
Revised: The CIFAR-10 training data consist of N s = 50000 color images of objects – divided into ten categories – with 32 -by- 32 pixels in each of 3 color channels, each pixel ranging in [0 , 1] (Krizhevsky & Hinton, 2009).

Original: We preprocess the data through uniformly subtracting 0 .
Revised: We preprocess the data through uniformly subtracting 0 .

Original: 5 and multiplying by 2 so that each pixel ranges in [ − 1 , 1] .
Revised: 5 and multiplying by 2 so that each pixel ranges in [ − 1 , 1] .

Original: In order to describe the architecture of our convolutional neural network (CNN) in detail, let us associate a tuple [ F, C, S, P ; M ] to a convolutional layer with ﬁlter width F , a number of channels
Revised: In order to describe the architecture of our convolutional neural network (CNN) in detail, let us associate a tuple [ F, C, S, P ; M ] to a convolutional layer with ﬁlter width F , a number of channels

Original: C , stride S , and padding P , followed by ReLU activations and a max-pooling layer of width M .
Revised: C , stride S , and padding P , followed by ReLU activations and a max-pooling layer of width M .

Original: Then, as in the demo at Karpathy (2014), our CNN consists of a (32 , 32 , 3) input layer followed by a convolutional layer with [5 , 16 , 1 , 2; 2] , another convolutional layer with [5 , 20 , 1 , 2; 2] , yet another convolutional layer with [5 , 20 , 1 , 2; 2] , and ﬁnally a fully-connected 10 -dimensional output layer with the softmax activation.
Revised: Then, as in the demo at Karpathy (2014), our CNN consists of a (32 , 32 , 3) input layer followed by a convolutional layer with [5 , 16 , 1 , 2; 2] , another convolutional layer with [5 , 20 , 1 , 2; 2] , yet another convolutional layer with [5 , 20 , 1 , 2; 2] , and ﬁnally a fully-connected 10 -dimensional output layer with the softmax activation.

Original: The model performance is evaluated by the cross-entropy loss supplemented by the L 2 -regularization term 12 λ θ 2 with the weight decay λ = 0 .
Revised: The model performance is evaluated by the cross-entropy loss supplemented by the L 2 -regularization term 12 λ θ 2 with the weight decay λ = 0 .

Original: Throughout the paper (except in Section 3.3 where the adaptive scheduling method is tested for
Revised: Throughout the paper (except in Section 3.3 where the adaptive scheduling method is tested for

Original: SGD without momentum), the CNN is trained on the CIFAR-10 data through SGD with momentum µ = 0 .
Revised: SGD without momentum), the CNN is trained on the CIFAR-10 data through SGD with momentum µ = 0 .

Original: 9 and dampening ν = 0 .
Revised: 9 and dampening ν = 0 .

Original: The data are shufﬂed at each epoch with the mini-batch size |B| = 100 .
Revised: The data are shufﬂed at each epoch with the mini-batch size |B| = 100 .

Original: The CNN is initialized through the Xavier method (Glorot & Bengio, 2010) and trained for ˆ t totalepoch = 100 epochs with the learning rate η = 0 .
Revised: The CNN is initialized through the Xavier method (Glorot & Bengio, 2010) and trained for ˆ t totalepoch = 100 epochs with the learning rate η = 0 .

Original: We then sequentially train it with ( η, ˆ t totalepoch ) =
Revised: We then sequentially train it with ( η, ˆ t totalepoch ) =

Original: (0 . 05 , 200) → (0 . 02 , 200) → (0 . 01 , 200) → (0 .
Revised: (0 . 05 , 200) → (0 . 02 , 200) → (0 . 01 , 200) → (0 .

Original: → (0 . 003 , 400) → (0 . 002 , 400) → (0 . 0015 , 400) → (0 .
Revised: → (0 . 003 , 400) → (0 . 002 , 400) → (0 . 0015 , 400) → (0 .

Original: → (0 . 0005 , 800) → (0 . 00025 , 800) → (0 . 0001 , 800) .
Revised: → (0 . 0005 , 800) → (0 . 00025 , 800) → (0 . 0001 , 800) .

Original: At each junction of the sequence, the velocity v is zeroed.
Revised: At each junction of the sequence, the velocity v is zeroed.

Original: This sequential-run protocol is carried out withdistinct seeds for the random-number generator used in data shufﬂing, all starting from the common model parameter attained at the end of the initial 100 -epoch run.
Revised: This sequential-run protocol is carried out withdistinct seeds for the random-number generator used in data shufﬂing, all starting from the common model parameter attained at the end of the initial 100 -epoch run.

Original: The ﬁgure 2 depicts trajectories for one particular seed, while the ﬁgure 3 plots means and error bars over these distinct seeds.
Revised: The ﬁgure 2 depicts trajectories for one particular seed, while the ﬁgure 3 plots means and error bars over these distinct seeds.

Original: C A DDITIONAL SIMULATIONS C. A DAM VERSUS AMSG RAD
Revised: C A DDITIONAL SIMULATIONS C. A DAM VERSUS AMSG RAD

Original: Plotted in the ﬁgure S1 are the comparisons between Adam (Kingma & Ba, 2014) and AMSGrad (J. Reddi et al., 2018) algorithms with the default hyperparameters α = 10 − 3 , ( β 1 , β 2 )
Revised: Plotted in the ﬁgure S1 are the comparisons between Adam (Kingma & Ba, 2014) and AMSGrad (J. Reddi et al., 2018) algorithms with the default hyperparameters α = 10 − 3 , ( β 1 , β 2 )

Original: = (0 . 9 , 0 . 999) , and (cid:15) = 10 − 8 .
Revised: = (0 . 9 , 0 . 999) , and (cid:15) = 10 − 8 .

Original: The AMSGrad algorithm marginally outperforms the Adam algorithm for the tasks at hand and thus the results with the AMSGrad are presented in the main text.
Revised: The AMSGrad algorithm marginally outperforms the Adam algorithm for the tasks at hand and thus the results with the AMSGrad are presented in the main text.

Original: [Figure]
Revised: [Figure]

Original: C. I NITIAL ACCURACY GAIN WITH DIFFERENT SCHEDULING HYPERPARAMETERS
Revised: C. I NITIAL ACCURACY GAIN WITH DIFFERENT SCHEDULING HYPERPARAMETERS

Original: In the ﬁgure 4(a) for the MNIST classiﬁcation task with the MLP, the proposed adaptive method with the scheduling hyperparameters X = 0 .
Revised: In the ﬁgure 4(a) for the MNIST classiﬁcation task with the MLP, the proposed adaptive method with the scheduling hyperparameters X = 0 .

Original: 01 and Y = 0 .
Revised: 01 and Y = 0 .

Original: 1 outperforms the AMSGrad algorithm in terms of accuracy attained at long time and also exhibits a quick initial convergence.
Revised: 1 outperforms the AMSGrad algorithm in terms of accuracy attained at long time and also exhibits a quick initial convergence.

Original: In the ﬁgure 4(b) for the CIFAR-10 classiﬁcation task with the CNN, however, while the proposed adaptive method attains better accuracy at long time, its initial accuracy gain is visibly slower than the AMSGrad algorithm.
Revised: In the ﬁgure 4(b) for the CIFAR-10 classiﬁcation task with the CNN, however, while the proposed adaptive method attains better accuracy at long time, its initial accuracy gain is visibly slower than the AMSGrad algorithm.

Original: This lag in initial accuracy gain can be ameliorated by choosing another combination of the scheduling hyperparameters, e.g., X = 0 .
Revised: This lag in initial accuracy gain can be ameliorated by choosing another combination of the scheduling hyperparameters, e.g., X = 0 .

Original: 1 and Y = 0 .
Revised: 1 and Y = 0 .

Original: 3 , at the expense of degradation in generalization accuracy with respectto the original choice X = and
Revised: 3 , at the expense of degradation in generalization accuracy with respect

Original: [Figure]
Revised: [Figure]