Original: T HE BREAK - EVEN POINT ON THE OPTIMIZATION TRAJECTORIES OF DEEP NEURAL NETWORKS
Revised: T HE BREAK - EVEN POINT ON THE OPTIMIZATION TRAJECTORIES OF DEEP NEURAL NETWORKS

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Understanding the optimization trajectory is critical for understanding training of deep neural networks.
Revised: The early phase of training of deep neural networks is critical for their ﬁnal performance.

Original: We show how the hyperparameters of stochastic gradient descent inﬂuence the covariance of  the gradients ( K ) and the Hessian of thetraining loss ( H ) along the trajectory. Based on a theoretical model, we conjecture that using a high learning rate or a small batch size leads SGD to regions of loss landscape, typically early during training, characterized by (1) reduced spectral norm of K and (2) improved conditioning of K and H .
Revised: In this work, we study how the hyperparameters of stochastic gradient descent (SGD) used in the early phase of training affect the rest of the optimization trajectory. We argue for the existence of the “break-even" point on this trajectory, beyond which the curvature of the loss surface and noise in the gradient are implicitly regularized by SGD. In particular, we demonstrate on multiple classiﬁcation tasks that using a large learning rate in the initial phase of training reduces the variance of gradients, and improves the conditioning of the covariance of gradients (a matrix that describes noise in the mini-batch gradient).

Original: We refer to the point on the training trajectory afterwhich these effects hold as the break-even point .
Revised: These effects are beneﬁcial from the optimization perspective and become visible after the breakeven point.

Original: We demonstrate these effects empirically for a range of deep neural networks applied to different tasks. Finally, we apply our analysis to networks with batch normalization (BN) layers and ﬁnd that it is necessary to use a higher learning rate to improve loss surface conditioning compared to a network without BN layers
Revised: Complementing prior work, we also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers. In short, our work shows that key properties of the loss surface are strongly inﬂuenced by SGD in the early phase of training. We argue that studying the impact of the identiﬁed effects on generalization is a promising future direction.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The choice of the optimization method implicitly regularizes deep neural networks (DNNs) by inﬂuencing the optimization trajectory in the loss surface (Neyshabur, 2017; Arora, 2019). In this work, we theoretically and empirically investigate how the learning rate and the batch size used at the beginning of  training determine properties of the entire optimization trajectory.
Revised: The connection between optimization and generalization of deep neural networks (DNNs) is not fully understood. For instance, using a large initial learning rate often improves generalization, which can come at the expense of the initial training loss reduction (Goodfellow et al., 2016; Li et al., 2019; Jiang et al., 2020). In contrast, using batch normalization layers typically improves both generalization and training speed of deep neural networks (Luo et al., 2019; Bjorck et al., 2018).

Original: [Figure]
Revised: These simple examples illustrate fundamental limitations of our understanding of DNNs. Understanding the early phase of training has recently emerged as a promising avenue for studying the link between optimization and generalization of DNNs. It has been observed that regularization in the early phase of training is necessary to arrive at a well generalizing ﬁnal solution (Keskar et al., 2017; Sagun et al., 2017; Achille et al., 2017).

Original: We focusour analysis on two objects that quantify different properties of the optimization trajectory: the covariance ofgradients ( K ) 1 , and the Hessian of the training loss ( H ).
Revised: Another phenomenon is that the local shape of the loss surface changes rapidly in the beginning of training (LeCun et al., 2012; Keskar et al., 2017; Achille et al., 2017; Jastrzebski et al., 2018; Fort & Ganguli, 2019). Theoretical approaches to understanding deep networks also increasingly focus on the early part of the optimization trajectory (Li et al., 2019; Arora et al., 2019). In this work, we study the dependence of the optimization trajectory on the early phase of training.

Original: The matrix K quantiﬁes noise induced by noisy estimate of the full-batch gradient, andhas been linked to the generalization error (Roux et al., 2008; Fort et al., 2019).
Revised: We investigate noise in the mini-batch gradients using the covariance of gradients, 1 and the local We deﬁne it as K = N (cid:80) Ni = ( g i − g ) T ( g i − g )

Original: The matrix H describes the curvature of the loss surface.
Revised: g i = g ( x i y i ; θ is training curvature of the loss surface using the Hessian.

Original: Better conditioning of H has been attributed as the main reason behind the efﬁcacy of batch normalization (Bjorck et al., 2018; Ghorbani et al., 2019).
Revised: These two matrices capture important and complementary aspects of optimization (Roux et al., 2008; Ghorbani et al., 2019) and generalization performance of DNNs (Jiang et al., 2020; Keskar et al., 2017; Bjorck et al., 2018; Fort et al., 2019).

Original:  Our ﬁrstand main contribution is predicting, and empirically demonstrating two effects in the early phase of  training inﬂuenced by the choice of the hyperparameters in stochastic gradient descent (SGD): (1) reduced spectral norms of K and H and (2) improved conditioning of K and H .
Revised: We include a more detailed discussion in Sec. 2. [Figure] Our ﬁrst contribution is a simpliﬁed model of the early part of the training trajectory of DNNs. Based on prior empirical work (Sagun et al., 2017), we assume that the local curvature of the loss surface (the spectral norm of the Hessian) increases or decreases monotonically along the optimization trajectory.

Original: These effects manifest themselves after acertain point on the optimization trajectory, to which we refer to as the break-even point .
Revised: Under this model, gradient descent reaches a point in the early phase of training at which it oscillates along the most curved direction of the loss surface. We call this point the break-even point and show empirical evidence of its existence in the training of actual DNNs.

Original: See Fig.
Revised: Our main contribution is to state and present empirical evidence for two conjectures about the dependence of the optimization trajectory on the early phase of training. Speciﬁcally, we conjecture that the hyperparameters of stochastic gradient descent (SGD) used before reaching the break-even point control: (1) the spectral norms of K and H , and (2) the conditioning of K and H . In particular, using a larger learning rate prior to reaching the break-even point reduces the spectral norm of K along the optimization trajectory (see Fig.

Original: 1 for an illustration of this phenomenon.
Revised: 1 for an illustration of this phenomenon).

Original: We make our predictions based on a theoretical model of the initial phase of training, which incorporates recent observations on the instability and oscillations in the parameter space that characterize the learning dynamics of neural networks (Masters & Luschi, 2018; Xing et al., 2018; Lan et al., 2019).
Revised: Reducing the spectral norm of K decreases the variance of the mini-batch gradient, which is linked to improved convergence speed (Johnson & Zhang, 2013).

Original: As our second contribution, we apply our analysis to a network with batch normalization (BN) layers and ﬁnd that our predictions are valid in this case too.
Revised: Finally, we apply our analysis to a network with batch normalization (BN) layers and ﬁnd that our predictions are valid in this case as well.

Original: Delving deeper in this direction of investigation, we show that using a large learning rate is necessary to reach better-conditioned, relatively to a network without BN layers, regions of the loss surface, which was previously attributed to BN alone (Bjorck et al., 2018; Ghorbani et al., 2019; Page, 2019).
Revised: Delving deeper in this line of investigation, we show that using a large learning rate is necessary to reach better-conditioned (relatively to a network without BN layers) regions of the loss surface, which was previously attributed to BN alone (Bjorck et al., 2018; Ghorbani et al., 2019; Page, 2019).

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Learning dynamics and the early phase of training.
Revised: Implicit regularization induced by the optimization method.

Original: Our theoretical model is motivated by recent work on the learning dynamics of  neural networks  (Goodfellow et al., 2014; Masters & Luschi, 2018; Wu et al., 2018;
Revised: The choice of the optimization method implicitly affects generalization performance of deep neural networks (Neyshabur, 2017). In particular, using a large initial learning rate is known to improve generalization (Goodfellow et al., 2016; Li et al., 2019).

Original: Yao et al., 2018;
Revised: A classical approach to study these questions is to bound the generalization error based on measures such as the norm of the parameters at the ﬁnal minimum (Bartlett et al., 2017; Jiang et al., 2020).

Original: Xing et al., 2018; Jastrzebski et al., 2018; Lan et al., 2019).We are directly inspired by Xing et al. ; Jastrzebski et al.
Revised: An emerging approach is to study the properties of the whole optimization trajectory (Arora et al., 2019).

Original: ; Lan et al.
Revised: Arora et al.

Original: (2019) who show that training oscillates, which can be characterized by a negative cosine between consecutive steps.
Revised: (2019) suggest it is necessary to study the optimization trajectory to understand optimization and generalization of deep networks.

Original: Our theoretical approach is closely related to Wu et al.

Original: While they study how SGD selects a minimum from a stability perspective, we apply their methodology to the early phase of training.

Original: Based on their theoretical results and other recent studies, we make predictions about the whole training trajectory.

Original: In our work we argue that the initial phase of training determines the rest of the trajectory.

Original: This is directly related to Erhan et al. ; Achille et al.
Revised: Relatedly, Erhan et al. (2010); Achille et al.

Original: (2017)who show the existence of the critical period of learning.
Revised: (2017) show the existence of a critical period of learning.

Original: Erhan et al.
Revised: Erhan et al.

Original: (2010) argue that training, unless pre-training is used, is sensitive to shufﬂing of examples in the ﬁrst epochs of training.
Revised: (2010) argue that training, unless pre-training is used, is sensitive to shufﬂing of examples in the ﬁrst epochs of training.

Original: Achille et al.
Revised: Achille et al.

Original: (2017); Golatkar et al.
Revised: (2017); Golatkar et al.

Original:  Sagun et al.
Revised: ; Sagun et al.

Original: ; Keskar et al.
Revised: ; Keskar et al.

Original: (2017) demonstrate that adding regularization in the beginning of training affects the ﬁnal generalization disproportionately more compared to doing so later.
Revised: (2017) demonstrate that adding regularization in the beginning of training affects the ﬁnal generalization disproportionately more compared to doing so later.

Revised: We continue research in this direction and ask how the choice of hyperparameters in SGD in the early phase of training affects the optimization trajectory in terms of the covariance of gradients, and the Hessian.

Original: The covariance ofthe gradients and the Hessian.
Revised: The covariance of gradients and the Hessian.

Revised: The Hessian quantiﬁes the local curvature of the loss surface.

Revised: Recent work has shown that the largest eigenvalues of H can grow quickly in the early phase of training (Keskar et al., 2017; Sagun et al., 2017; Fort & Scherlis, 2019; Jastrzebski et al., 2018).

Revised: Keskar et al.

Revised: Jastrzebski et al.

Original:  The covariance of  the  gradients, which we denote by K , encapsulates the geometry and  magnitude of variation in gradients across different samples (Thomas et al., 2019).
Revised: (2017) studied the dependence of the Hessian (at the ﬁnal minima) on the optimization hyperparameters. The Hessian can be decomposed into two terms, where the dominant term (at least at the end of training) is the uncentered covariance of gradients G (Sagun et al., 2017; Papyan, 2019). The covariance of gradients, which we denote by K , encapsulates the geometry and the magnitude of variation in gradients across different samples.

Original: The matrix K was related to the generalization error in Roux et al.
Revised: The matrix K was related to the generalization error in Roux et al.

Original: A similar quantity, cosine alignment between gradients computed on different examples, was recently shown to explain some aspects of deep networks generalization (Fort et al., 2019).
Revised: ; Jiang et al. Closely related quantities, such as the cosine alignment between gradients computed on different examples, were recently shown to explain some aspects of deep networks generalization (Fort et al., 2019;

Original: The second object that we study is the Hessian that quantiﬁes the loss surface shape (LeCun et al., 2012).
Revised: Liu et al., 2020; He & Su, 2020).

Original: Recent work has shown that the largest eigenvalues of H grow quickly in the early phase of training, and then stabilize at a value dependent on the learning rate and the batch size (Keskar et al., 2017; Sagun et al., 2017; Fort & Scherlis, 2019; Jastrzebski et al., 2018).
Revised: Zhang et al.

Original: The Hessian can be decomposed into a sum of two terms, where the dominant term (at least at the end of training) is theuncentered covariance of gradients G (Sagun et al., 2017; Papyan, 2019).
Revised: (2019) argues that in DNNs the Hessian and the covariance of gradients are close in terms of the largest eigenvalues. Learning dynamics of deep neural networks.

Original: While we study K , the centered version of G , K and G are typically similar due to the dominance of noise in training (Zhu et al., 2018;
Revised: Our theoretical model is motivated by recent work on learning dynamics of neural networks (Goodfellow et al., 2014; Masters & Luschi, 2018; Wu et al., 2018; Yao et al., 2018;

Original: Thomas et al., 2019).
Revised: Xing et al., 2018; Jastrzebski et al., 2018; Lan et al., 2019).

Original: Implicit regularization induced by optimization method. Multiple prior works study the regularization effects that are attributed only to the  optimization method (Neyshabur, 2017).
Revised: We are directly inspired by Xing et al. (2018) who show that for popular classiﬁcation task, the cosine of the angle between consecutive optimization steps in SGD is negative. Similar observations can be found in Lan et al.

Original: A popular research direction is  to bound the generalization error based on the properties of the ﬁnal minimum such as the norm of  the parameter vector or the Hessian (Bartlett et al., 2017; Keskar et al., 2017). Perhaps the most related work to ours is (Arora et al., 2019; Arora, 2019).
Revised: Our theory is closely related to Wu et al. (2018) who study how SGD selects the ﬁnal minimum from a stability perspective. We apply their methodology to the early phase of training, and make predictions about the whole training trajectory.

Revised:  B REAK - EVEN POINT AND TWO CONJECTURES ABOUT SGD TRAJECTORY

Original: They suggest it isnecessary to study the  optimization trajectory to understand generalization of deep networks.
Revised: Our overall motivation is to better understand the connection between optimization and generalization of DNNs.

Original: In this vein, but in contrast to most of the previous work, we focus (1) on the implicit regularization effects that can be attributed to the GD dynamics at the beginning of training, and (2) on the covariance of gradients.
Revised: In this section we study how the covariance of gradients ( K ) and the Hessian ( H ) depend on the early phase of training. We are inspired by recent empirical observations showing their importance for optimization and generalization of DNNs (see Sec.

Original: T WO CONJECTURES ABOUT SGD TRAJECTORY
Revised: 2 for a detailed discussion). Recent work has shown that in the early phase of training the gradient norm (Goodfellow et al., 2016; Fort & Ganguli, 2019; Liu et al., 2020) and the local curvature of the loss surface (Jastrzebski et al., 2018; Fort & Ganguli, 2019) can rapidly increase.

Original: In this section we make two conjectures about the optimization trajectory induced by SGD based on a theoretical model of the learning dynamics in the early stage of training.
Revised: Informally speaking, one scenario we study here is when this initial growth is rapid enough to destabilize training. Inspired by Wu et al. (2018), we formalize this intuition using concepts from dynamical stability. Based on the developed theory, we state two conjectures about the dependence of K and H on hyperparameters of SGD, which we investigate empirically in Sec. 4.

Original: Deﬁnitions.
Revised: Deﬁnitions.

Original:  Let us denote the loss on an example ( x , y ) by L ( x , y ; θ ) , where θ is a D -dimensional parameter vector.
Revised: We begin by introducing the notation. Let us denote the loss on an example ( x , y ) by L ( x , y ; θ ) , where θ is a D -dimensional parameter vector.

Original: A key object we study is the Hessian of the training loss ( H ). The second key object we study is the covariance ofthe gradients K = 1 N (cid:80) Ni =

Revised: The two key objects we study are the Hessian of the training loss ( H ), and the covariance of gradients K = 1 N (cid:80) Ni =

Original: ( g i − g ) T ( g i − g ) ,
Revised: ( g i − g ) T ( g i − g ) ,

Original: where g i = g ( x i , y i ; θ ) is the gradient of L with respect to θ calculated on i -th example, N is the number of training examples, and g is the full-batch gradient.
Revised: where g i = g ( x i , y i ; θ ) is the gradient of L with respect to θ calculated on i -th example, N is the number of training examples, and g is the full-batch gradient.

Original: We denote the i -th normalized eigenvector and eigenvalue of a matrix A by e iA and λ iA .
Revised: We denote the i -th normalized eigenvector and eigenvalue of a matrix A by e iA and λ iA .

Original: Both H and K are computed at a given θ , but we omit this dependence in the notation.
Revised: Both H and K are computed at a given θ , but we omit this dependence in the notation.

Original: Let t index steps of optimization, and let θ ( t ) denote the parameter vector at optimization step t .
Revised: Let t index steps of optimization, and let θ ( t ) denote the parameter vector at optimization step t .

Original: Inspired by Wu et al.
Revised: Inspired by Wu et al.

Original: we study stability of optimization, in our case restricted to e 1 H .
Revised: we introduce the following condition to quantify stability at a given θ ( t ) .

Original: Let us call the projection of parameters θ onto e 1 H by ψ = (cid:104) θ, e 1 H (cid:105) .
Revised: Let us denote the projection of parameters θ onto e 1 H by ψ = (cid:104) θ, e 1 H (cid:105) .

Original: With a slight abuse of notation let g ( ψ ) =
Revised: With a slight abuse of notation let g ( ψ ) =

Original: (cid:104) g ( θ ) , e 1 H (cid:105) .
Revised: (cid:104) g ( θ ) , e 1 H (cid:105) .

Original: Similarly to Wu et al.

Original: (2018), we say SGD is unstable along e 1 H at θ ( t ) if the norm of elements of sequence ψ ( τ + 1) =
Revised: We say SGD is unstable along e 1 H at θ ( t ) if the norm of elements of sequence ψ ( τ + 1) =

Original: ψ ( τ ) − ηg ( ψ ( τ )) diverges when τ → ∞ , where ψ (0)
Revised: ψ ( τ ) − ηg ( ψ ( τ )) diverges when τ → ∞ , where ψ (0)

Original: =
Revised: =

Original: θ ( t ) .
Revised: θ ( t ) .

Original: The sequence ψ ( τ ) represents optimization trajectory restricted from θ ( t ) only to the direction of e 1 H .
Revised: The sequence ψ ( τ ) represents optimization trajectory restricted from θ ( t ) only to the direction of e 1 H .

Original: Assumptions.
Revised: Assumptions.

Original: We make the following assumptions to construct our model.
Revised: Based on recent empirical studies, we make the following assumptions.

Original: The loss surfacefor each example projected onto e 1 H is a quadratic function.
Revised: The loss surface projected onto e 1 H is a quadratic one-dimensional function of the form f ( ψ ) =

Revised: (cid:80) Ni =1 ( ψ − ψ ∗ ) 2 H

Original: This assumption was also used in Wu et al.
Revised: i . The same assumption was made in Wu et al.

Revised: (2018), but along all directions in the weight space.

Original: Alain et al.
Revised: Alain et al.

Original: (2019) show empirically that the loss averaged over all training examples is well approximated by a quadratic function along e 1 H .
Revised: (2019) show empirically that the loss averaged over all training examples is well approximated by a quadratic function along e 1 H .

Original: The eigenvectors e 1 H and e 1 K are co-linear, i.e. e 1 H = ±
Revised: The eigenvectors e 1 H and e 1 K are co-linear, i.e. e 1 H = ±

Original: e 1 K , and λ 1 K
Revised: e 1 K , and λ 1 K

Original: = αλ 1 H for some α ∈ R .
Revised: = αλ 1 H for some α ∈ R .

Original: This is inspired by Papyan (2019) who show that  H can be  approximated by G (non-centered K ).
Revised: This is inspired by the fact that the top eigenvalues of H can be well approximated using G (non-centered K ) Papyan (2019); Sagun et al.

Revised: Zhang et al.

Revised: (2019) shows direct evidence for co-linearity of the largest eigenvalues of K and H .

Original: When in a region that is not stable along e 1 H , λ 1 H decreases along the training trajectory, and the distance to the minimizer along e 1 H increases.
Revised: When in a region that is not stable along e 1 H , λ 1 H decreases along the training trajectory, and the distance to the minimizer along e 1 H increases.

Original: This is inspired by recent work showing training can escape a region with too large curvature compared to the learning rate (Zhu et al., 2018;
Revised: This is inspired by recent work showing training can escape a region with too large curvature compared to the learning rate (Zhu et al., 2018;

Original: Wu et al., 2018; Jastrzebski et al., 2018).
Revised: Wu et al., 2018; Jastrzebski et al., 2018).

Original: The spectral norm of H , λ 1 H , increases during training and the distance to the minimizer along e 1 H decreases, unless increasing λ 1 H would lead to entering a region where training is not stable along e 1 H .
Revised: The spectral norm of H , λ 1 H , increases during training and the distance to the minimizer along e 1 H decreases, unless increasing λ 1 H would lead to entering a region where training is not stable along e 1 H .

Original: This is inspired by (Keskar et al., 2017; 
Revised: This is inspired by (Keskar et al., 2017; Goodfellow et al., 2016;

Original: Sagun et al., 2017; Jastrzebski et al., 2018; Fort & Scherlis, 2019; Fort & Ganguli, 2019) who show that in many settings λ 1 H  increases in the beginning of training, while at the same time the 
Revised: Sagun et al., 2017; Jastrzebski et al., 2018; Fort & Scherlis, 2019; Fort & Ganguli, 2019) who show that in many settings λ 1 H or gradient norm increases in the beginning of training, while at the same time the overall training loss decreases.

Original: overall training loss decreases.
Revised: Finally, we also assume that S (cid:29) N , i.e. that the batch size is small compared to the number of training examples.

Original: These assumptions are only used to build a theoretical model for the early phase of training.
Revised: These assumptions are only used to build a theoretical model for the early phase of training.

Original: Its main purpose is to make predictions about the training procedure that we verify empirically.
Revised: Its main purpose is to make predictions about the training procedure that we test empirically in Sec. 4.

Original: Reaching the break-even point earlier for a larger learning rate or a smaller batch size.
Revised: Reaching the break-even point earlier for a larger learning rate or a smaller batch size.

Original: Let us restrict ourselves to the case when training is initialized at θ (0) at which SGD is stable along e 1 H (0) .
Revised: Let us restrict ourselves to the case when training is initialized at θ (0) at which SGD is stable along e 1 H (0) .

Original: 2 We aim to show that the learning rate ( η ) and the batch size ( S ) determine H and K in our model, and conjecture that the same holds empirically for real neural networks.
Revised: 2 We aim to show that the learning rate ( η ) and the batch size ( S ) determine H and K in our model, and conjecture that the same holds empirically for actual neural networks.

Original: Consider two optimization trajectories for η 1 and η 2 , where η 1 > η 2 , that are initialized at the same θ 0 , where optimization is stable along e 1 H ( t ) and λ 1 H ( t ) > 0 .
Revised: Consider two optimization trajectories for η 1 and η 2 , where η 1 > η 2 , that are initialized at the same θ 0 , where optimization is stable along e 1 H ( t ) and λ 1 H ( t ) > 0 .

Original: Under Assumption 1 the loss surface along e 1 H ( t ) can be expressed as f ( ψ ) =
Revised: Under Assumption 1 the loss surface along e 1 H ( t ) can be expressed as f ( ψ ) =

Original: (cid:80) Ni =1 ( ψ − ψ ∗ ) 2 H
Revised: (cid:80) Ni =1 ( ψ − ψ ∗ ) 2 H

Original: i ( t ) , where H i ( t ) ∈ R .
Revised: i ( t ) , where H i ( t ) ∈ R .

Original: It can be shown that at any iteration t the necessary and sufﬁcient condition for SGD to be stable along e 1 H ( t ) is: 
Revised: It can be shown that at any iteration t the necessary and sufﬁcient condition for SGD to be stable along e 1 H ( t ) is: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where N is the training set size and s ( t ) 2 =
Revised: ) where N is the training set size and s ( t ) 2 =

Original: Var[ H i ( t )] over the training examples.
Revised: Var[ H i ( t )] over the training examples.

Original: A proof can be found in (Wu et al., 2018).
Revised: A proof can be found in (Wu et al., 2018).

Original: We call this point on the trajectory on which the LHS of Eq.
Revised: We call this point on the trajectory on which the LHS of Eq.

Original: 1 becomes equal to 1 for the ﬁrst time the break-even point .
Revised: 1 becomes equal to 1 for the ﬁrst time the break-even point .

Original: By deﬁnition, there exists only a single break-even point on the trajectory.
Revised: By deﬁnition, there exists only a single break-even point on the trajectory.

Original: Underthe Assumption 3, λ 1 H ( t ) and λ 1 K ( t ) increase over time.
Revised: Under Assumption 3, λ 1 H ( t ) and λ 1 K ( t ) increase over time.

Original: If s = N , the break-even point is reached at λ 1 H ( t ) = 2 η .
Revised: If S = N , the break-even point is reached at λ 1 H ( t ) = 2 η .

Original: More generally, it can be shown that for η 1 , the break-even point is reached for a lower magnitude of λ 1 H ( t ) than for η 2 .
Revised: More generally, it can be shown that for η 1 , the break-even point is reached for a lower magnitude of λ 1 H ( t ) than for η 2 .

Original: The same reasoning can be applied to S .
Revised: The same reasoning can be repeated for S (in which case we assume N (cid:29) S ).

Original: We state this formally and prove in App. A.
Revised: We state this formally and prove in App. B.

Original: From this point on the trajectory, under Assumption 4, SGD does not enter regions where either λ 1 H( t (cid:48) ) or λ 1 K( t (cid:48) ) is larger than at the break-even point, as otherwise it would lead to increasing one of the terms in LHS of Eq. 1, and hence losing stability along e 1 H ( t (cid:48) ) .
Revised: From the break-even point on the trajectory, under Assumption 4, SGD does not enter regions where either λ 1 H or λ 1 K is larger than at the break-even point, as otherwise it would lead to increasing one of the terms in LHS of Eq. 1, and hence losing stability along e 1 H ( t (cid:48) ) .

Original: The two conjectures about real DNNs.
Revised: Two conjectures about real DNNs.

Original: Assuming that real DNNs reach the break-even point, we make the following two conjectures, arising from our theoretical model above, about their optimization trajectory.
Revised: Assuming that real DNNs reach the break-even point, we make the following two conjectures about their optimization trajectory.

Original: The most direct implication of reaching the break-even point is that λ 1 K and λ 1 H at the break-even point depend on η and S , which we formalize as:
Revised: The most direct implication of reaching the break-even point is that λ 1 K and λ 1 H at the break-even point depend on η and S , which we formalize as:

Original: Conjecture 1 (Variance reduction effect of SGD) .
Revised: Conjecture 1 (Variance reduction effect of SGD) .

Original: Along the SGD trajectory, the maximum attained values of λ 1 H and λ 1 K are smaller for a larger learning rate or a smaller batch size.
Revised: Along the SGD trajectory, the maximum attained values of λ 1 H and λ 1 K are smaller for a larger learning rate or a smaller batch size.

Original: We refer to Con. 1 as variance reduction effect of SGD, because reducing λ 1 K can be shown to reduce the L 2 distance between the full-batch gradient, and the mini-batch gradient.
Revised: We refer to Conjecture 1 as variance reduction effect of SGD because reducing λ 1 K can be shown to reduce the L 2 distance between the full-batch gradient, and the mini-batch gradient.

Revised: We expect that similar effects should exist for other optimization or regularization choices, which we leave for future work.

Original: Next, we make another, stronger, conjecture.
Revised: Next, we make another, stronger, conjecture.

Original: It is plausible to assume that reaching the break-even point does not affect the λ iH and λ iK for i (cid:54) = 1, because increasing their values does not impact stability along e 1 H in our theoretical model.
Revised: It is plausible to assume that reaching the break-even point affects to a lesser degree λ iH and λ iK for i (cid:54) = 1 because increasing their values does not impact stability along e 1 H in our theoretical model.

Original: It is also well known that initially a large number of eigenvalues of H increase (Fort & Scherlis, 2019; Sagun et al., 2017).

Original: Based on these remarks we conjecture that:
Revised: Based on this we conjecture that:

Original: Conjecture 2 (Pre-conditioning effect of SGD) .
Revised: Conjecture 2 (Pre-conditioning effect of SGD) .

Original: Along the SGD trajectory, the maximum attained values of λ ∗ K λ 1 K and λ ∗ H λ 1 H are larger for a larger learning rate or a smaller batch size, where λ∗ K  and λ∗ H  are the smallest nonzero eigenvalues ofH and K  , respectively.
Revised: Along the SGD trajectory, the maximum attained values of λ ∗ K λ 1 K and λ ∗ H λ 1 H are larger for a larger learning rate or a smaller batch size, where λ K ∗ and λ H ∗ are the smallest non-zero eigenvalues of K and H , respectively.

Original: Furthermore, the maximum attained values of Tr( K ) , Tr( H ) are smaller for a larger learning rate or a smaller batch size.
Revised: Furthermore, the maximum attained values of Tr( K ) and Tr( H ) are smaller for a larger learning rate or a smaller batch size.

Original: We consider non-zero eigenvalues in Con. 2, because K has  N − D non-zero eigenvalues, where N is the number of training points.
Revised: We consider non-zero eigenvalues in the conjecture, because K has at most N − 1 non-zero eigenvalues, where N is the number of training points, which can be drastically smaller than D in overparametrized DNNs.

Original: Both conjectures are only valid for learning rates and batch sizesin the range that guarantee that training is stable after reaching the break-even point.
Revised: Both conjectures are only valid for learning rates and batch sizes that guarantee that training converges.

Original: It is also important to stress that while our predictions are related to the curvature of the ﬁnal minimum (Wu et al., 2018; Jastrzebski et al., 2017), in contrast to prior work we are interested in the properties of the whole training trajectory.
Revised: Finally, from the optimization perspective the effects discussed above are desirable.

Original: Finally, from the optimization perspective, the effects discussed above are desirable; many papers in the optimization literature focused on either reducing variance of the mini-batch gradients (Johnson & Zhang, 2013), or on improving conditioning  (Roux et al., 2008). [Figure] [Figure]
Revised: Many papers in the optimization literature underline the importance of reducing variance of the mini-batch gradient (Johnson & Zhang, 2013), or of conditioning of the covariance of gradients (Roux et al., 2008). There are also connection between these effects and generalization (Jiang et al., 2020), which we discuss towards the end of the paper.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section we ﬁrst analyse learning dynamics in the early phase of training.
Revised: In this section we ﬁrst analyse learning dynamics in the early phase of training.

Original: Next, we empirically validate the two conjectures.
Revised: Next, we empirically investigate the two conjectures.

Original: In the ﬁnal part we extend our analysis to a neural network with batch normalization layers.
Revised: In the ﬁnal part we extend our analysis to a neural network with batch normalization layers.

Original: Due to the space constraint we take the following approach to reporting results.
Revised: Due to the space constraint we take the following approach to reporting results.

Original: In the main body of the paper we focus on the CIFAR-10 dataset (Krizhevsky, 2009) and the IMDB dataset (Maas et al., 2011), to which we apply three architectures: a vanilla CNN (SimpleCNN) following 
Revised: In the main body of the paper, we focus on the CIFAR-10 dataset (Krizhevsky, 2009) and the IMDB dataset (Maas et al., 2011), to which we apply three architectures: a vanilla CNN (SimpleCNN) following Keras example

Original: Keras example (Chollet et al., 2015), ResNet-32 (He et al., 2015) and LSTM (Hochreiter & Schmidhuber, 1997).
Revised: (Chollet et al., 2015), ResNet-32 (He et al., 2015), and LSTM (Hochreiter & Schmidhuber, 1997).

Original: We also validate the two conjectures for DenseNet (Huang et al., 2016)  on the ImageNet (Deng et al., 2009) dataset,  BERT (Devlin et al., 2018) ﬁne-tuned on the MNLI dataset (Williams et al., 2017), and a multi-layer perceptron  on the FashionMNIST dataset (Xiao et al., 2017).
Revised: We also test the two conjectures for DenseNet (Huang et al., 2016) trained on the ImageNet (Deng et al., 2009) dataset, and BERT (Devlin et al., 2018) ﬁne-tuned on the MNLI dataset (Williams et al., 2018). We include in the Supplement experiments using a multi-layer perceptron trained on the FashionMNIST dataset (Xiao et al., 2017), and experiments using SGD with momentum and learning rate decay.

Original: These results and all experimental details are  in App.
Revised: All experimental details are described in App.

Original: C.
Revised: D.

Original: Following Dauphin et al.
Revised: Following Dauphin et al.

Original: Alain et al.
Revised: Alain et al.

Original: (2019), we estimate the top eigenvalues and eigenvectors of Husing the Lanczos algorithm on a random subset of5% of the training set on CIFAR-10.
Revised: (2019), we estimate the top eigenvalues and eigenvectors of H on a small subset of the training set (e.g. 5% in the case of CIFAR-10) using the Lanczos [Figure]

Original: We estimate the top eigenvalues and eigenvectors of K using (in most cases) batch size of 128 and approximately 5% of the training set on CIFAR-10. Furthermore, under the assumption that noise due to sampling data is normally distributed, our approximation biases the smallest eigenvalue towards 0 .
Revised: [Figure]

Original: To counteract this, we use the 5 th smallest non-zero eigenvalue in place ofλ ∗ H and λ ∗ K .
Revised: algorithm (Lanczos, 1950). Computing the full eigenspace of K is infeasible for real DNNs.

Original: We describe the procedure in more details, as well as examine the impact of using mini-batching in computing the eigenspectrum of K , in App. 
Revised: Instead, we compute the covariance using mini-batch gradients. In App. C we argue empirically that (after normalization) this approximates well the largest eigenvalue

Original: B.
Revised: [Figure] and we include other details on computing the eigenspaces.

Original: 4. A CLOSER LOOK AT THE EARLY PHASE OF TRAINING
Revised: 4. A CLOSER LOOK AT THE EARLY PHASE OF TRAINING

Original: In this section we examine the learning dynamics in the early phase of training.
Revised: First, we examine the learning dynamics in the early phase of training.

Original: Our goal is to verify some of the assumptions made in Sec.
Revised: Our goal is to verify some of the assumptions made in Sec.

Original: We analyse the evolution of λ 1 H and λ 1 K for η = 0 .
Revised: We analyse the evolution of λ 1 H and λ 1 K when using η = 0 .

Original: 02 and η = 0 .
Revised: 02 and η = 0 .

Original: 2 using the SimpleCNN on the CIFAR-10 dataset.
Revised: 002 to train SimpleCNN on the CIFAR-10 dataset.

Revised: We repeat this experiment 5 times using different initialization.

Revised: Visualizing the break-even point.

Revised: We visualize the early part of the optimization trajectory in

Revised: Fig.

Revised: Following Erhan et al.

Revised: (2010), we embed the test set predictions at each step of training of SimpleCNN using UMAP (McInnes et al., 2018).

Revised: Background color indicates λ 1 K (left) and the training accuracy at the iteration with the closest embedding in Euclidean distance (right).

Revised: We observe that the trajectory corresponding to the lower learning rate reaches regions of the loss surface characterized by larger λ 1 K , compared to regions reached at the same training accuracy in the second trajectory.

Revised: Additionally, in Fig.

Revised: 3 we plot the spectrum of K (left) and H (right) at the iterations when λ K and λ H respectively reach the highest values.

Revised: We observe more outliers with the lower learning rate in the distributions of both λ K and λ H .

Original: Are λ 1 K and λ 1 H correlated in the beginning of training?
Revised: Are λ 1 K and λ 1 H correlated in the beginning of training?

Original: The key assumption behind our theoretical model is that λ 1 K and λ 1 H are correlated, at least prior to reaching the break-even point.
Revised: The key assumption behind our theoretical model is that λ 1 K and λ 1 H are correlated, at least prior to reaching the break-even point.

Original: We conﬁrm this in Fig.
Revised: We conﬁrm this in Fig.

Original: The highest achieved λ 1 K and λ 1 H are larger for the smaller η .
Revised: The highest achieved λ 1 K and λ 1 H are larger for the smaller η .

Original: Additionally, we observe that after achieving the highest value of λ 1 H , further growth of λ 1 K does not translate into an increase of λ 1 H .
Revised: Additionally, we observe that after achieving the highest value of λ 1 H , further growth of λ 1 K does not translate into an increase of λ 1 H .

Original: This is expected: λ 1 H decays to 0 when the mean loss decays to 0 for cross entropy loss (Martens, 2016). 
Revised: This is expected: λ 1 H decays to 0 when the mean loss decays to 0 for cross entropy loss (Martens, 2016). [Figure] [Figure] [Figure]

Revised: in training speeds.

Original:  Does training become increasingly unstable in the early phase of training?
Revised: By default we use S = 128 in all experiments, η = 0 . for the CIFAR-10 dataset, and η = 1 . 0 for the IMDB dataset. Does training become increasingly unstable in the early phase of training?

Original: A key aspect of our model is that an increase of λ 1 K and λ 1 H translates into a decrease in stability, which we formalized as stability along e 1 H .
Revised: According to Assumption 3, an increase of λ 1 K and λ 1 H translates into a decrease in stability, which we formalized as stability along e 1 H .

Original: Computingstability directly  along e 1 H is computationally expensive.
Revised: Computing directly stability along e 1 H is computationally expensive.

Original: Instead, we measure a more tractable proxy.
Revised: Instead, we measure a more tractable proxy.

Original: At each iteration we measure the loss on the training set before, and after taking the step, which we denote as ∆ L (a positive value indicates  reduction of the training loss).
Revised: At each iteration we measure the loss on the training set before and after taking the step, which we denote as ∆ L (a positive value indicates a reduction of the training loss).

Original: In Fig.
Revised: In Fig.

Original: 2 we observe that training becomes unstable when λ 1 K reaches the maximum value.Visualizing the break-even point.
Revised: 2 we observe that training becomes unstable ( ∆ L starts to be increasingly oscillating) around the time λ 1 K reaches the maximum value.

Original: Finally, to understand the break-even point phenomenon better, we visualize the learning dynamics leading to reaching the break-even point in our model in

Original: Fig.

Original: Following Erhan et al.

Original: (2010), we embed the test set predictions at each step of training of

Original: SimpleCNN, using UMAP (McInnes et al., 2018).

Original: Background color indicates λ 1 K (left) and the training accuracy at the closest iteration (right).

Original: We observe that early in training the trajectory corresponding to a lower learning rate of η = 0 .

Original: 01 reaches regions of the loss surface corresponding to the same accuracy as η = 0 . 1 , but characterized by larger λ 1 K .

Original: Additionally, in Fig.

Original: 3 we plot the spectrum of K (left) and H at the iteration when λ K and λ H , respectively, reach the highest value.

Original: We can observe that for the lower learning rate the distribution has more outliers.

Original: Summary.
Revised: Summary.

Original: We have shown that thedynamics of the early phase of training is consistent with the assumptions made in our theoretical model.
Revised: We have shown that the early phase of training is consistent with the assumptions made in our theoretical model.

Original: That is, λ 1 K and λ 1 H increase approximately proportionally to each other, which is also correlated with a decrease of a proxy of stability.
Revised: That is, λ 1 K and λ 1 H increase approximately proportionally to each other, which is also correlated with a decrease of a proxy of stability.

Original: Finally, we have shown qualitatively reaching the break-even point.
Revised: Finally, we have shown qualitatively reaching the break-even point.

Original: 4. V ARIANCE REDUCTION AND  PRE - CONDITIONING EFFECT OF SGD
Revised: 4. T HE VARIANCE REDUCTION AND THE PRE - CONDITIONING EFFECT OF SGD

Original: In this section we validate empirically Con. 1 and Con.
Revised: In this section we test empirically Conjecture 1 and Conjecture 2.

Original: 2 in three settings.

Original: For each model we pick manually a suitable range of learning rates and batch sizes to ensure that the properties of K and H that we  converged in a reasonable computational budget and training is stable.
Revised: For each model we pick manually a suitable range of learning rates and batch sizes to ensure that the properties of K and H that we study have converged under a reasonable computational budget.

Original: We use 200 epochs on CIFAR-10 and 50 epochs on IMDB.
Revised: We mainly focus on studying the covariance of gradients ( K ), and leave a closer investigation of the Hessian for future work.

Original: In ResNet-32 we remove batch normalization.
Revised: We use batch size of 128 to compute K when we vary the batch size for training. When we vary the learning rate instead, we use the same batch size as the one used to train the model. App. C describes the remaining details on how we approximate the eigenspaces of K and H .

Original:  We summarize the results for SimpleCNN, ResNet-32,  and LSTM in Fig.
Revised: [Figure] We summarize the results for SimpleCNN, ResNet-32, LSTM, BERT, and DenseNet in Fig.

Original: 4 and Fig.
Revised: 4, Fig. 5, and Fig.

Original: Curves are smoothed with a moving average.
Revised: Curves are smoothed using moving average for visual clarity.

Original: The training curves, as well as experiments for other architectures and datasets (including a DenseNet on ImageNet and BERT on MNLI) can be found in App.
Revised: Training curves and additional experiments are reported in App.

Original: D.
Revised: E.

Original: Null hypothesis.

Original: A natural assumption is that the choice of η or S does not inﬂuence K and H along the optimization trajectory.

Original: In particular, it is not self-evident that using a high η , or a small

Original: S , would steer optimization towards better conditioned regions of the loss surface.

Original:  Conjecture 1.
Revised: Testing Conjecture 1.

Original: To validate Conjecture 1 we examine the highest value of λ 1 K observed along the optimization trajectory.
Revised: To test Conjecture 1, we examine the highest value of λ 1 K observed along the optimization trajectory.

Original: As visible in Fig. 4 using a higher η results in λ 1 K achieving a lower maximum.
Revised: As visible in Fig. 4, using a higher η results in λ 1 K achieving a lower maximum during training.

Original:  For instance max( λ 1 K ) = 0 . 87 and max( λ 1 K )= 3 .
Revised: Similarly, we observe that using a higher S in SGD leads to reaching a higher maximum value of λ 1 K . For instance, for SimpleCNN (top row of Fig. 4) we observe max( λ 1 K ) = 0 . 68 and max( λ 1 K )

Original: 01 for η = 0 .
Revised: = 3 . 30 for η = 0 .

Original: 1 and η = 0 .
Revised: 1 and η = 0 .

Original: 01 , respectively.
Revised: 01 , respectively.

Original: Similarly, we can conclude that using a higher S in SGD leads to reaching a higher value of λ 1 K .

Original: Recall that we compute λ 1 K using a constant batch size of 128 .

Original: While we say that low S leads to variance reduction (lower maximum λ 1 K ), this is not contradictory to the fact that increasing S generally decreases variance of mini-batch gradients.

Original:  Conjecture 2.
Revised: Testing Conjecture 2.

Original: To test Conjecture 2 we compute the maximum value of λ ∗ K /λ 1 K along the optimization trajectory; as discussed before we use the 5 th smallest eigenvalue in place of λ ∗ K .
Revised: To test Conjecture 2, we compute the maximum value of λ ∗ K /λ 1 K along the optimization trajectory.

Original: It is visible in Fig.
Revised: It is visible in Fig.

Original: 4 that using a higher η results in  a larger maximum value of λ ∗ K /λ 1 K along the trajectory.
Revised: 4 that using a higher η results in reaching a larger maximum value of λ ∗ K /λ 1 K along the trajectory.

Original: For instance, in the case of SimpleCNN(top row of Fig. 4) max( λ ∗ K /λ 1 K ) = 0 . 45 and max( λ ∗ K /λ 1 K ) = 0 .
Revised: For instance, in the case of SimpleCNN max( λ ∗ K /λ 1 K ) = 0 . 37 and max( λ ∗ K /λ 1 K ) = 0 .

Original: 295 for η = 0 .
Revised: 24 for η = 0 .

Original: 1 and η = 0 .
Revised: 1 and η = 0 .

Original: 01 , respectively.
Revised: 01 , respectively.

Original: Similarly, we can conclude from these plots that using a higher S leads to lower max( λ ∗ K /λ 1 K )  . 
Revised: A counter-intuitive effect of decreasing the batch size. Consistently with Conjecture 2, we observe that the maximum value of Tr( K ) is smaller for the smaller batch size. In the case of SimpleCNN max(Tr( K )) = 5 . 56 and max(Tr( K ))

Original: [Figure]
Revised: = 10 .

Original: better conditioning).
Revised: 86 for S = 10 and S = 100 , respectively.

Original: These effects occur early in training.
Revised: Due to space constraints we report the effect of η and S on Tr( K ) in other settings in App. E.

Original: The vertical lines marks the ﬁrst epoch at which training accuracy is above 60% for CIFAR-10, and above for 75% for IMDB.
Revised: This effect is counter-intuitive because Tr( K ) is proportional to the variance of the mini-batch gradient (see also App. C).

Original: We use base S = 128 in all experiments. We use base η = 0 . for CIFAR-10 and η = 0 .
Revised: Naturally, using a lower batch size generally increases the variance of the mini-batch gradient, and Tr( K ) .

Original: 1 for IMDB.
Revised: This seeming contradiction is explained by the fact that we measure

Original: Due to space constraints we move Figures showing the effect of η and S on Tr( K )  to App. D. We observe that the maximum of Tr( K ) depends on η and S in  the same way as λ 1 K .
Revised: Tr( K ) using a different batch size ( 128 ) than the one used to train the model. Hence, decreasing the batch size both increases (due to approximating the gradient using fewer examples) and decreases (as predicted in Conjecture 2) the variance of the mini-batch gradient along the optimization trajectory.

Original: How early in training is the break-even point reached?
Revised: How early in training is the break-even point reached?

Original: How λ 1 H and λ 1 K depend on η and S at the end of training was already studied by Jastrzebski et al.

Original: ; Keskar et al. ; Wu et al. Importantly, we ﬁnd that λ 1 K and λ 1 H reach the highest value early in training: close to reaching 60% training accuracy on CIFAR-10, and 75% training accuracy on IMDB. 
Revised: We ﬁnd that λ 1 K and λ 1 H reach their highest values early in training, close to reaching 60% training accuracy on CIFAR-10, and 75% training accuracy on IMDB. The training and validation accuracies are reported for all the experiments in App.

Original: See also the vertical lines in Fig.
Revised: E. This suggests that the break-even point is reached early in training.

Original: Other experiments.
Revised: The Hessian.

Original: We report how λ 1 H depends on η and S for ResNet-32 and SimpleCNN in
Revised: In the above, we have focused on the covariance of gradients.

Original:  Fig.
Revised: In Fig.

Original: We observe that the conclusions also apply to λ 1 H , which is consistent with experiments in Jastrzebski et al.
Revised: 5 we report how λ 1 H depends on η and S for ResNet-32 and SimpleCNN. Consistently with prior work (Keskar et al., 2017; Jastrzebski et al., 2018), we observe that using a smaller η or using a larger S coincides with a larger peak value of λ 1 H . For instance, for SimpleCNN we observe max( λ 1 H ) = 26 . 27 and max( λ 1 H ) = 211 .

Original: We found the effect of η and S on λ ∗ H /λ 1H of η and S to be weaker.
Revised: 17 for η = 0 . 1 and η = 0 . 01 , respectively. We leave for future work testing predictions made in Conjecture 2 about the Hessian.

Revised: Larger scale studies.

Original: This might be because, in contrast to K , we approximate the spectrum of H using only  the top ﬁve eigenvalues (see App.
Revised: Finally, we test the two conjectures on two larger scale settings: BERT ﬁne-tuned on the MNLI dataset, and DenseNet trained on the ImageNet dataset. Due to memory constraints, we only vary the learning rate. We report results in Fig. We observe that both

Original: B for details).
Revised: [Figure] conjectures hold in these two settings. It is worth noting that DenseNet uses batch normalization layers. In the next section we investigate closer batch-normalized networks.

Original: Summary
Revised: Summary.

Original: In this section we have demonstrated the variance reduction (Conjecture 1) and the preconditioning effect (Conjecture 2) of SGD.
Revised: In this section we have shown evidence supporting the variance reduction (Conjecture 1) and the pre-conditioning effect (Conjecture 2) of SGD on a wide range of classiﬁcation tasks

Original: Furthermore, we have shown these effects occur early in training.
Revised: [Figure]

Original: We also found that  conclusions also apply to other settings including BERT on MNLI and
Revised: We also found that the above conclusions hold for MLP trained on the Fashion MNIST dataset, SGD with momentum

Original: DenseNet on ImageNet (see App. D).
Revised: [Figure] and SGD with learning rate decay. We include these results in App. E-G.

Original: 4. I MPORTANCE OF η FOR CONDITIONING IN BATCH NORMALIZED NETWORKS
Revised: 4. I MPORTANCE OF LEARNING RATE FOR CONDITIONING IN BATCH NORMALIZED NETWORKS

Original:  The loss surface of deep  networks has been widely reported to be ill-conditioned, which is the key motivation behind using second order optimization methods in deep learning (LeCun et al., 2012; Martens & Grosse, 2015).
Revised: [Figure] [Figure] The loss surface of deep neural networks has been widely reported to be ill-conditioned (LeCun et al., 2012; Martens, 2016).

Original: Recently, Ghorbani et al.
Revised: Recently, Ghorbani et al.

Original: (2019); Page (2019) have argued that the key reason behind the efﬁcacy of batch normalization (Ioffe & Szegedy, 2015) is improving conditioning of the loss surface.
Revised: (2019); Page (2019) have argued that the key reason behind the efﬁcacy of batch normalization (Ioffe & Szegedy, 2015) is improving conditioning of the loss surface.

Original: Our Conjecture 2 is that using a high η or a low S  resultsas well in improved [Figure] [Figure] [Figure]
Revised: In Conjecture 2 we propose that using a high η (or a small S ) results in improving the condition number of K and H .

Original: || ||

Original: [Figure] conditioning. A natural question that we investigate in this section is how the two phenomena are related. 
Revised: A natural question that we investigate in this section is how the two phenomena are related. We study here the effect of learning rate, and report in App.

Revised: H a similar study for batch size.

Original: Are the two conjectures valid in batch-normalized networks?
Revised: Are the two conjectures valid in batch-normalized networks?

Original: First, to  investigate whether our conjectures hold in batch-normalized network, we run similar experiments as in Sec. 
Revised: First, to further investigate whether our conjectures hold in batch-normalized network, we run similar experiments as in Sec. 4.

Original: 4.2 on a SimpleCNN model with batch normalization layers inserted after eachconvolutional layer (SimpleCNNBN), using the CIFAR-10 dataset.
Revised: on a SimpleCNN model with batch normalization layers inserted after each layer (SimpleCNN-BN), on the CIFAR-10 dataset.

Original: We test η ∈ { 0 .
Revised: We test η ∈ { 0 .

Original: 001 , 0 . 01 , 0 . 1 , 1 . 0 } ( η = 1 . 0 leads to divergence of SimpleCNN without BN).
Revised: 001 , 0 . 01 , 0 . 1 , 1 . 0 } ( η = 1 . 0 leads to divergence of SimpleCNN without BN).

Original: We summarize the results in Fig.
Revised: We summarize the results in Fig.

Original: The evolution of λ ∗ K /λ 1 K and λ 1 K show that both Conjecture 1 and Conjecture 2 hold in this setting.
Revised: We observe that the evolution of λ ∗ K /λ 1 K and λ 1 K is consistent with both Conjecture 1 and Conjecture 2.

Original: A closer look at the early phase of training.
Revised: A closer look at the early phase of training.

Original: To further corroborate that our analysis applies to BN networks, we study the early phase of training of a network with batch normalization layers, complementing the results in Sec. 4.1.
Revised: To further corroborate that our analysis applies to batch-normalized networks, we study the early phase of training of SimpleCNN-BN, complementing the results in Sec. 4.1.

Original: We observe in Fig.
Revised: We observe in Fig.

Original: 6 (bottom) that training of SimpleCNN-BN starts in a region characterized by  relatively high λ 1 K .
Revised: 7 (bottom) that training of SimpleCNN-BN starts in a region characterized by a relatively high λ 1 K .

Original: This is consistent with prior work showing that batch-normalized networks can exhibit gradient explosion in the ﬁrst iteration (Yang et al., 2019).
Revised: This is consistent with prior work showing that batch-normalized networks can exhibit gradient explosion in the ﬁrst iteration (Yang et al., 2019).

Original:  λ 1 K then decays for all but the lowest η .
Revised: The value of λ 1 K then decays for all but the lowest η .

Original: This behavior is consistent with our theoretical model.
Revised: This behavior is consistent with our theoretical model.

Original: We also track the norm of the scaling factor in BN, (cid:107) γ (cid:107) , in the last layer of the network in Fig.
Revised: We also track the norm of the scaling factor in batch normalization layers, (cid:107) γ (cid:107) , in the last layer of the network in Fig.

Original: 6 (bottom).
Revised: 7 (bottom).

Original: It is visible that η = 1 .and{+. 
 and+} η = 0 .
Revised: It is visible that η = 1 {+. 
 and+} η = 0 .

Original: 1 initially decrease the value of (cid:107) γ (cid:107) , which we hypothesize to be one of the mechanisms by which high η steers optimization towards better conditioned regions of the loss surface in BN networks.
Revised: 1 initially decrease the value of (cid:107) γ (cid:107) , which we hypothesize to be one of the mechanisms by which high η steers optimization towards better conditioned regions of the loss surface in batch-normalized networks.

Revised: Interestingly, this seems consistent with Luo et al.

Revised: (2019) who argue that using mini-batch statistics in batch normalization acts as an implicit regularizer by reducing (cid:107) γ (cid:107) .

Original: BN requires using a high learning rate.
Revised: Using batch normalization requires using a high learning rate.

Original: As our conjectures hold for BN network, a natural question is if learning can be ill-conditioned with a low learning rate even when BN is used.
Revised: As our conjectures hold for SimpleCNN-BN, a natural question is if learning can be ill-conditioned with a low learning rate even when batch normalization is used.

Original: Ghorbani et al.
Revised: Ghorbani et al.

Original: (2019) show that without BN, mini-batch gradients are largely contained in the subspace spanned by the top eigenvectors of noncentered K .
Revised: (2019) show that without batch normalization, mini-batch gradients are largely contained in the subspace spanned by the top eigenvectors of noncentered K .

Original: To answer this question we track (cid:107) g
Revised: To answer this question we track (cid:107) g

Original: (cid:107) / (cid:107) g 5 (cid:107) , where g denotes the mini-batch gradient, and g 5 denotes the mini-batch gradient projected onto the top 5 eigenvectors of K .
Revised: (cid:107) / (cid:107) g 5 (cid:107) , where g denotes the mini-batch gradient, and g 5 denotes the mini-batch gradient projected onto the top 5 eigenvectors of K .

Original: A value of (cid:107) g (cid:107) / (cid:107) g 5 (cid:107) close to 1 implies that the mini-batch gradient is mostly contained in the subspace spanned by the top 5 eigenvectors of K .
Revised: A value of (cid:107) g (cid:107) / (cid:107) g 5 (cid:107) close to 1 implies that the mini-batch gradient is mostly contained in the subspace spanned by the top 5 eigenvectors of K .

Original: We compare two settings: (1) SimpleCNN-BN optimized with η = 0 .
Revised: We compare two settings: (1) SimpleCNN-BN optimized with η = 0 .

Original: 001 , and (2) SimpleCNN optimized with η = 0 .
Revised: 001 , and (2) SimpleCNN optimized with η = 0 .

Original: We make three observations.
Revised: We make three observations.

Original: First, the maximum (minimum) value of (cid:107) g (cid:107) / (cid:107) g 5 (cid:107) is 1 .
Revised: First, the maximum (minimum) value of (cid:107) g (cid:107) / (cid:107) g 5 (cid:107) is 1 .

Original: 90 ( 1 . 37 ) and 1 .[-
-]
Revised: 90 ( 1 . 37 ) and 2 .[-
-]

Original:  ( 1 . 12 ), respectively.
Revised: 02 ( 1 . 09 ), respectively.

Original: Second, the maximum value of λ 1 K is 10 .and 16 , respectively.
Revised: Second, the maximum value of λ 1 K is 12 .and 3 . 30 , respectively.

Original: Finally, λ ∗ K /λ 1 K reaches 0 .
Revised: Finally, λ ∗ K /λ 1 K reaches 0 .

Original: 368 in the ﬁrst setting, and 0 .
Revised: 343 in the ﬁrst setting, and 0 .

Original: 295 in the second setting.
Revised: 24 in the second setting.

Original: Comparing these differences to differences that are induced by using the highest η = 1 . 0 in SimpleCNN-BN, we can conclude that using a large learning rate is necessary to observe the effect of loss smoothing which was previously attributed to BN alone (Ghorbani et al., 2019; Page, 2019; 
Revised: Comparing these differences to differences that are induced by using the highest η = 1 .in SimpleCNN-BN, we can conclude that using a large learning rate is necessary to observe the effect of loss smoothing which was previously attributed to batch normalization alone (Ghorbani et al., 2019; Page, 2019; Bjorck et al., 2018).

Original: Bjorck et al., 2018). This might be directly related to the result that  a high learning rate is necessary to achieve good generalization when using BN (Bjorck et al., 2018).
Revised: This might be directly related to the result that using a high learning rate is necessary to achieve good generalization when using batch normalization layers (Bjorck et al., 2018).

Original: Summary.
Revised: Summary.

Original: We have shown that the effects ofchanging the learning rate described in Con. 1 and
Revised: We have shown that the effects of the learning rate predicted in Conjecture 1 and

Original: Con. 2also hold for a network with batch normalization layers, and that using a high learning rate is necessary in a batch-normalized network to improve conditioning of the loss surface, compared to conditioning of the loss surface in the same network without batch-normalization.
Revised: Conjecture 2 hold for a network with batch normalization layers, and that using a high learning rate is necessary in a batch-normalized network to improve conditioning of the loss surface, compared to conditioning of the loss surface in the same network without batch normalization layers.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Based on a theoretical model weconjectured and empirically argued for the existence of the breakeven point on the optimization trajectory induced by SGD.
Revised: Based on our theoretical model, we argued for the existence of the break-even point on the optimization trajectory induced by SGD.

Original: Next, we demonstrated that using a high learning rate or a small batch size in SGD has two effects on K  and H along the trajectory that we referred to as (1) variance reduction and (2) pre-conditioning.
Revised: We presented evidence that hyperparameters used in the early phase of training control the spectral norm and the conditioning of K (a matrix describing noise in the mini-batch gradients) and H (a matrix describing local curvature of the loss surface) after reaching the break-even point.

Original: There are many potential implications of the existence of the break-even point. We investigated one in particular and demonstrated that using a high learning rate is necessary to achieve the loss smoothing effects previously attributed to batch normalization alone.
Revised: In particular, using a large initial learning rate steers optimization to better conditioned regions of the loss surface, which is beneﬁcial from the optimization point of view. A natural direction for the future is connecting our observations to recent studies on the relation of measures, such as gradient variance, to the generalization of deep networks (Li et al., 2019; Jiang et al., 2020; Fort et al., 2019). Our work shows that the hyperparameters of SGD control these measures after the break-even point.

Original: Additionally, the break-even occurs typically early during training, which might be related to the recently discovered phenomenon of the critical learning period in training of deep networks (Achille et al., 2017; Golatkar et al.
Revised: Another interesting direction is to understand the connection between the existence of the break-even point and the existence of the critical learning period in training of DNNs (Achille et al., 2017). A CKNOWLEDGMENTS KC thanks the support by NVIDIA and eBay. SJ thanks Amos Storkey and Luke Darlow for fruitful discussions.

Original: We plan to investigate this connection in the future
Revised: We include here a similar plot as Fig. 1, but visualizing the conditioning of the covariance of gradients ( λ ∗ K /λ 1 K , left), the trace of the covariance of gradients ( Tr( K ) , middle), and the spectral norm of the Hessian ( λ 1 H , right). The choice of the learning rate impacts the regions of the loss surface reached by SGD in terms of these quantities. [Figure]

Original: A P ROOFS
Revised: B P ROOFS

Original: In this section wewill formally state and prove the theorem usedinformally in Sec. 3.
Revised: In this section we formally state and prove the theorem used in Sec. 3.

Original: With the deﬁnitions introduced in Sec. 3 in mind, we state the following:
Revised: With the deﬁnitions introduced in Sec. 3 in mind, we state the following:

Original: Theorem 1.
Revised: Theorem 1.

Original: Assuming that training is stable along e 1 H ( t ) at t = 0 , then λ 1 H ( t ) and λ 1 K ( t ) at which
Revised: Assuming that training is stable along e 1 H ( t ) at t = 0 , then λ 1 H ( t ) and λ 1 K ( t ) at which

Original: SGD becomes unstable along e 1 H ( t ) are smaller for a larger η or a smaller S .
Revised: SGD becomes unstable along e 1 H ( t ) are smaller for a larger η or a smaller S .

Original: Proof.
Revised: Proof.

Original: This theorem is a straightforward application of Theorem 1 from Wu et al.
Revised: This theorem is a straightforward application of Theorem 1 from Wu et al.

Original: (2018) to the early phase of training.
Revised: (2018) to the early phase of training.

Original: First, let us consider two optimization trajectories corresponding to using two different learning rates η 1 and η 2 ( η 1 > η 2 ).
Revised: First, let us consider two optimization trajectories corresponding to using two different learning rates η 1 and η 2 ( η 1 > η 2 ).

Original: For both trajectories, Theorem 1 of Wu et al.
Revised: For both trajectories, Theorem 1 of Wu et al.

Original: (2018) states that SGD is stable at an iteration t if the following inequality is satisﬁed:
Revised: (2018) states that SGD is stable at an iteration t if the following inequality is satisﬁed:

Original: [Equation]
Revised: [Equation]

Original: where s ( t ) =
Revised: where s ( t ) =

Original: Var[H i (t)] .
Revised: Var[H i (t)] .

Original: Using Assumption 1 we get Var[ g i ( ψ )] =
Revised: Using Assumption 1 we get Var[ g i ( ψ )] =

Original: Var[ ψH i ] =
Revised: Var[ ψH i ] =

Original: ψ 2 s ( t ) 2 .
Revised: ψ 2 s ( t ) 2 .

Original: Using this we can rewrite inequality (2) as:
Revised: Using this we can rewrite inequality (2) as:

Original: [Equation]
Revised: [Equation]

Original: At the  iteration t ∗ at which training becomes unstable along e 1 H ( t ∗ ) , λ 1 H ( t ∗ ) can be calculated from inequality (3) as
Revised: To calculate the spectral norm of the Hessian at iteration t ∗ at which training becomes unstable we equate the two sides of inequality (3) and solve for λ 1 H ( t ∗ ) , which gives

Original: [Equation]
Revised: [Equation]

Original: where α denotes the proportionality constant from Assumption 2.
Revised: where α denotes the proportionality constant from Assumption 2.

Original: Note that if n = S the right hand side degenerates to 2 η , which completes the proof in the case n = S .
Revised: Note that if n = S the right hand side degenerates to 2 η , which completes the proof for n = S .

Original: To prove the general case, let denote by λ 1 H ( t ∗ 1 ) and λ 1 H
Revised: To prove the general case, let denote by λ 1 H ( t ∗ 1 ) and λ 1 H

Original: ( t ∗ 2 ) the value of λ 1 H at which training becomes unstable along e 1 H for η 1 and η 2 , respectively.
Revised: ( t ∗ 2 ) the value of λ 1 H at which training becomes unstable along e 1 H for η 1 and η 2 , respectively.

Original: Similarly, let us denote by ψ ( t ∗ 1 ) and ψ ( t ∗ 2 ) the value of ψ at which training becomes unstable.
Revised: Similarly, let us denote by ψ ( t ∗ 1 ) and ψ ( t ∗ 2 ) the corresponding values of ψ .

Original: Let us assume by contradiction that λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) .
Revised: Let us assume by contradiction that λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) .

Original: A necessary condition for this inequality to hold is that ψ ( t ∗ 1 ) > ψ ( t ∗ 2 ) .
Revised: A necessary condition for this inequality to hold is that ψ ( t ∗ 1 ) > ψ ( t ∗ 2 ) .

Original: However,  ψ ( t )by Assumption 4 decreases  with increasing λ 1 H  , which corresponds to reducingthe distance to the minimizer along e 1 H. Hence, that would imply λ 1 H ( t ∗ 1 ) < λ 1 H ( t ∗ 2 ) , which contradicts λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) .
Revised: However, Assumption 4 implies that prior to reaching the break-even point ψ ( t ) decreases monotonically with increasing λ 1 H ( t ∗ 1 ) , which corresponds to reducing distance to the minimizer along e 1 H , which contradicts λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) .

Original: Repeating the same argument for  batch size completes the proof.
Revised: Repeating the same argument for two trajectories optimized using two different batch sizes completes the proof.

Original: It is also straightforward to extend the argument to the case when training is initialized at an unstable region along e 1 H (0) , which we formalize as follows.
Revised: It is also straightforward to extend the argument to the case when training is initialized at an unstable region along e 1 H (0) , which we formalize as follows.

Original: Theorem 2.
Revised: Theorem 2.

Original: If training is unstable along e 1 H ( t ) at t = 0 , then λ 1 H ( t ) and λ 1 K ( t ) at which SGD becomes for the ﬁrst time stable along e 1 H ( t ) are smaller for a larger η or a smaller S .
Revised: If training is unstable along e 1 H ( t ) at t = 0 , then λ 1 H ( t ) and λ 1 K ( t ) at which SGD becomes for the ﬁrst time stable along e 1 H ( t ) are smaller for a larger η or a smaller S .

Original: Proof.
Revised: Proof.

Original: We will use a similar argument as in the proof of Th.
Revised: We will use a similar argument as in the proof of Th.

Original: Let us consider two optimization trajectories corresponding to using two different learning rates η 1 and η 2 ( η 1 > η 2 ).
Revised: Let us consider two optimization trajectories corresponding to using two different learning rates η 1 and η 2 ( η 1 > η 2 ).

Original: Let λ 1 H ( t ∗ 1 ) and λ 1 H ( t ∗ 2 ) denote the spectral norm of H at the iteration at which training is for the ﬁrst time stable along e 1 H for η 1 and η 2 , respectively.
Revised: Let λ 1 H ( t ∗ 1 ) and λ 1 H ( t ∗ 2 ) denote the spectral norm of H at the iteration at which training is for the ﬁrst time stable along e 1 H for η 1 and η 2 , respectively.

Original: Following the steps in the proof of Th.
Revised: Following the steps in the proof of Th.

Original: 1 we get that
Revised: 1 we get that

Original: [Equation]
Revised: [Equation]

Original: Using the same notation as in the proof of Th.
Revised: Using the same notation as in the proof of Th.

Original: 1, let us assume by contradiction that λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) .
Revised: 1, let us assume by contradiction that λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) .

Original: Assumption 3 implies that ψ ( t ) increases with decreasing λ 1 H ( t ) .
Revised: Assumption 3 implies that ψ ( t ) increases with decreasing λ 1 H ( t ) .

Original: Hence, if λ 1 H ( t ∗ 2 ) has reached a smaller value, it means ψ ( t ∗ 2 )has increased to a larger value, i.e. ψ ( t ∗ 2 ) >ψ
Revised: Hence, if λ 1 H ( t ∗ 2 ) is smaller, it means ψ ( t ∗ 2 ) increased to a larger value, i.e. ψ ( t ∗ 2 ) >

Original:  ( t ∗ 1 ) .
Revised: ψ ( t ∗ 1 ) .

Original: However,  a necessary condition for λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) inequality to hold is that ψ ( t ∗ 1 ) > ψ ( t ∗ 2 ) , which leads to a contradiction.
Revised: However, if ( n − S ) ( n − 1) ≈ 1 , a necessary condition for λ 1 H ( t ∗ 1 ) > λ 1 H ( t ∗ 2 ) inequality to hold is that ψ ( t ∗ 1 ) > ψ ( t ∗ 2 ) , which leads to a contradiction.

Original: Repeating the same argument for batch size completes the proof.
Revised: Repeating the same argument for batch size completes the proof.

Original: B A PPROXIMATING THE EIGENSPACE OF K AND H
Revised: C A PPROXIMATING THE EIGENSPACE OF K AND H

Original: A small subset of the training set approximates well the largest eigenvalues of thetrue Hessian on the CIFAR-10 dataset (Alain et al., 2019).
Revised: Using a small subset of the training set sufﬁces to approximate well the largest eigenvalues of the Hessian on the CIFAR-10 dataset (Alain et al., 2019).

Original: This might be due to the hierarchical structure of the

Original: Hessian (Papyan, 2019; Fort & Ganguli, 2019); the largest eigenvalues are connected to the class structure in the data.

Original: Following Alain et al.
Revised: Following Alain et al.

Original: we use approximately the same 5% fraction of the dataset in CIFAR-10 experiments, and SCIPY Python package. 
Revised: we use approximately the same 5% fraction of the dataset in CIFAR-10 experiments, and SCIPY Python package. We use the same setting on the other datasets as well.

Original: Evaluating the eigenspace of K isperhaps less common in deep learning.
Revised: Analysing the eigenspace of K is less common in deep learning.

Original: We sample L mini-batch gradient of size M . 
Revised: For the purposes of this paper, we are primarily interested in estimating λ K ∗ and λ K . We also estimate Tr( K ) . It is worth noting that Tr( K ) is related to the variance in gradients as 1 D Tr( K )

Revised: = 1 D 1 N (cid:80)

Original: {+Ni =
+}
Revised: {+Ni =
+}

Revised: ||

Original: Then following Papyan (2019) we compute  the corresponding Gram matrixˆ K  that has entries ˆK ij =  (cid:104) g
Revised: g − g i || 2 , where g is the full-batch gradient, D is the number of parameters and N is the number of datapoints. The naive computation of the eigenspace of K is infeasible for realistically large deep networks due to the quadratic cost in the number of parameters. Instead, we compute K using mini-batches. To avoid storing a D × D matrix in memory, we ﬁrst sample L mini-batch gradient of size M and compute the corresponding Gram matrix K M that has entries K Mij = 1 L (cid:104) g

Original: i − g, g j − g (cid:105) , where g is the full batch gradient.
Revised: i − g, g j − g (cid:105) , where g is the full-batch gradient, which we estimate based on the L mini-batches.

Original: Note that ˆK ij is only L × L dimensional.
Revised: To compute the eigenspace of K M we use SVD routine from the NumPy package. We take the ( L − 1) th smallest eigenvalue as the smallest non-zero eigenvalue of K M (covariance matrix computed using L observations has by deﬁnition L − 1 non-zero eigenvalues). Papyan (2019); Fort & Ganguli (2019) show that the top eigenvalues of H emerge due to clustering of gradients of the logits. See also Fort & Ganguli (2019).

Original: It can be shown thatin expectation ofmini-batches ˆ K has the same eigenspectrum as
Revised: Based on the proximity of the largest eigenvalues of H and K , it suggests that the top eigenvalue of K might be similar to that of K M . To investigate this we run the following experiment on the CIFAR-10 dataset using SimpleCNN. We estimate λ 1 K using M = 1 and M = 128 , in both cases using the whole training set. We subsample the dataset to 10% to speed up the computation.

Original: K .
Revised: Fig.

Original:  In all experiments we ﬁx L = 25. When comparing different batch sizes we use M = 128 , otherwise we use the same batch size as the one used to train the model.
Revised: 9 shows a strong correlation between λ 1 K computed with M = 1 and with M = 128 , for three different learning rates. [Figure] In all experiments we use L = 25 , and for the IMDB dataset we increase the number of mini-batch gradients to L = 200 to reduce noise (same conclusion hold for L = 25 ).

Original: For instance on the CIFAR-10 dataset this amounts to using approximately 5% of the training set.
Revised: For instance, on the CIFAR-10 dataset this amounts to using approximately 5% of the training set.

Original: To compute Tr( K ) we compute the trace of ˆK . Due to large computational cost of estimating top eigenvalues using the Lanczos algorithm, we approximate in most experiments  the Tr( H ) using only the top 5 eigenvalues of H . A natural question is whether using M = 128 approximates well the underlying K . To investigate this we compare λ 1 K evaluated using either M = 128or M = 1 .
Revised: In experiments that vary the batch size we use M = 128 .

Original: We adapt L to keep the same number of N = LM of used training examples in the computation. We test this on SimpleCNN model with η = 0 . We show that the two are strongly correlated in Fig. [Figure]
Revised: Otherwise, we use the value M as the batch size used to train the model.

Original: C E XPERIMENTAL DETAILS  
Revised: D E XPERIMENTAL DETAILS FOR S EC . 4.

Original: In this section we describe all the details for experiments in the main text and in the Appendix.
Revised: In this section we describe all the details for experiments in Sec. 4.2.

Original: ResNet-32 on CIFAR The model trained is the ResNet-32 (He et al., 2015). The network is trained for 200 epochs with a batch size equal to 128. The dataset used is CIFAR-10.
Revised: ResNet-32 on CIFAR. ResNet-32 (He et al., 2015) is trained for 200 epochs with a batch size equal to 128 on the CIFAR-10 dataset.

Original: Standard data augmentation and preprocessing is applied. 
Revised: Standard data augmentation and preprocessing is applied. Following He et al.

Revised: , we regularize the model using weight decay 0 . 0001 .

Original: The default learning rate is 0 . 05 , and the default batch size is 128 .Weight decay 0 . 0001 is applied to all convolutional layers.
Revised: We apply it to all convolutional kernels. When varying the batch size, we use learning rate of 0 . 05 . When varying the learning rate, we use batch size of 128 .

Original: SimpleCNN on CIFAR
Revised: SimpleCNN on CIFAR.

Original: The network used is a simple convolutional network based on example from Keras  repository (Chollet et al., 2015). 
Revised: The network used is a simple convolutional network with 4 convolutional layers from Keras examples repository (Chollet et al., 2015). The architecture is as follows.

Original: First, the data is passed through two convolutional layers with 32 ﬁlters,both using same padding, ReLU activations, and  the kernel size of 3x3. Next, the second pair of layers is used with 64 ﬁlters (the rest of parameters are the same). Each pair ends with a max-pooling layer with a windowof size  2. 
Revised: After the ﬁrst two convolutional layers with 32 ﬁlters, and after the last two convolutional layers withﬁlters, we include max pooling layer with a window size of 2. After each layer we include ReLU nonlinearity.

Original: Before the classiﬁcation layer, a densely connected layer with 128 units is used.
Revised: Before the classiﬁcation layer, a densely connected layer with 128 units is used.

Original: The data follows the same scheme as in the ResNetexperiment.

Original: The default learning rate is 0 . 05 , and the default batch size is 128 .
Revised: When varying the batch size, we use a learning rate of 0 . 05 . When varying the learning rate, we use a batch size of 128 .

Original: BERT on MNLI
Revised: BERT on MNLI.

Original: The model used in this experiment is BERT-base (Devlin et al., 2018), pretrained on multilingual data 4 .
Revised: The model used in this experiment is BERT-base (Devlin et al., 2018), pretrained on multilingual data 4 .

Original: The model is trained on  MultiNLI dataset (Williams et al., 2018) with the maximum sentence length equal to 40.
Revised: The model is trained on the MultiNLI dataset (Williams et al., 2018) with the maximum sentence length equal to 40.

Original: The network is trained for 20 epochs with a batch size of 32.
Revised: The network is trained for 20 epochs with a batch size of 32.

Original:  MLP on FMNIST This experiment is using a Multi Layer Perceptron network with two hidden layers of size 300 and 100, both with ReLU activations.
Revised: Experiments are repeated with three different seeds that control initialization and data shufﬂing. MLP on FMNIST. This experiment is using a Multi Layer Perceptron network with two hidden layers of size 300 and 100, both with ReLU activations.

Original: The data is normalized to the [0 , 1] range and no augmentation is being used.
Revised: The data is normalized to the [0 , 1] range.

Original: The network is trained with a batch size of 64 for 200 epochs. 
Revised: The network is trained with a batch size of 64 for 200 epochs. [Table]

Original: LSTM on IMDB
Revised: LSTM on IMDB.

Original: The network used in this experiment consists of an embedding layer followed by an LSTM with 100 hidden units.
Revised: The network used in this experiment consists of an embedding layer followed by an LSTM with 100 hidden units.

Original: We use vocabulary size of 20000 words and the maximum length of the sequence equal to 80.
Revised: We use vocabulary size of 20000 words and the maximum length of the sequence equal to 80.

Original: The model is trained for 100 epochs with a  batch size of 128.
Revised: The model is trained for 100 epochs. When varying the learning rate, we use a constant batch size of 128.

Revised: When varying the batch size, we use a learning rate of 1.0.

Revised: Experiments are repeated with two different seeds that control initialization and data shufﬂing.

Original: DenseNet on ImageNet
Revised: DenseNet on ImageNet.

Original: The network used is the DenseNet-121 (Huang et al., 2016).
Revised: The network used is the DenseNet-121 (Huang et al., 2016).

Original: The dataset used is the ILSVRC 2012 (Russakovsky et al., 2015).
Revised: The dataset used is the ILSVRC 2012 (Russakovsky et al., 2015).

Original: The images are centered, but no augmentation is being used.
Revised: The images are centered, but no augmentation is being used.

Original: Due to large computational cost, the network is trained  for 10 epochs with a batch size of 32.
Revised: Due to large computational cost, the network is trained only for 10 epochs with a batch size of 32.

Original: Neither dropout nor weight decay is used for training.

Original: D A DDITIONAL EXPERIMENTS FOR S EC . 4.
Revised: E A DDITIONAL EXPERIMENTS FOR S EC . 4.2.

Original: In this section we repeat experiments from Sec.
Revised: In this section we include additional data for experiments in Sec.

Original: 4.2 in various other settings, as well as include additional data from settings already included in the main text.
Revised: 4.2, as well as include experiments using MLP trained on the Fashion MNIST dataset.

Original: ResNet-32 on CIFAR-10.
Revised: SimpleCNN on CIFAR-10.

Original: In Fig.
Revised: In Fig.

Original: 8 and Fig.
Revised: 12 and Fig.

Original: 9 we report accuracy on the training set and the validation set, λ 1 H , and Tr( K ) for all runs on the ResNet-32 model and the CIFAR-10 dataset.
Revised: 13 we report accuracy on the training set and the validation set, λ 1 H , and Tr( K ) for all experiments with SimpleCNN model on the CIFAR-10 dataset

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: SimpleCNN on CIFAR-10.
Revised: ResNet-32 on CIFAR-10.

Original: In Fig.
Revised: In Fig.

Original: 8 and Fig.
Revised: 12 and Fig.

Original: 9 we report accuracy on the training set and the validation set, λ 1 H , and Tr( K ) for all runs on theSimpleCNN model and the CIFAR-10 dataset .
Revised: 13 we report accuracy on the training set and the validation set, λ 1 H , and Tr( K ) for all experiments with ResNet-32 on the CIFAR-10 dataset .

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: LSTM on IMDB.
Revised: LSTM on IMDB.

Original: In Fig.
Revised: In Fig.

Original: 12 and Fig.
Revised: 14 and Fig.

Original: 13 we report accuracy on the training set and the validation set, λ 1 H , and Tr( K ) for all runs on the LSTM model and the IMDB dataset.
Revised: 15 we report accuracy on the training set and the validation set, λ 1 H [Figure] and Tr( K ) for all experiments with LSTM on the IMDB dataset.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: BERT on MNLI.
Revised: BERT on MNLI.

Original: In Fig.
Revised: In Fig.

Original: 14 and Fig. 15 we report results for the  BERT model on the MNLI dataset.
Revised: 16 we report accuracy on the training set and the validation set and Tr( K ) for BERT model on the MNLI dataset.

Original: [Figure]
Revised: [Figure]

Revised: DenseNet on ImageNet.

Revised: In Fig.

Revised: 17 we report accuracy on the training set and the validation set and Tr( K ) for DenseNet on the ImageNet dataset.

Original: [Figure]
Revised: [Figure]

Original: MLP on FMNIST.
Revised: MLP on FMNIST.

Original: In Fig.
Revised: In Fig.

Original: 16 and Fig.
Revised: 18 and Fig.

Original: 17 we report results forthe MLP model on the FMNIST dataset.
Revised: 19 we report results for MLP model on the FMNIST dataset.

Revised: We observe that all conclucions carry over to this setting

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: DenseNet on ImageNet.
Revised: MOMENTUM Here, we run the same experiment as in Sec. 4.2 using SimpleCNN on the CIFAR-10 dataset.

Original: In Fig.
Revised: Instead of varying the learning rate, we test different values of the momentum β parameter in the range of 0 . 1 , 0 . 5 and 0 . We can observe that Conjecture 1 and Conjecture 2 generalize to momentum in the sense that using a higher momentum has an analogous effect to using a higher learning rate, or using a smaller batch size in SGD. We report the results in Fig.

Original: 18 and Fig. 
Revised: 20 and Fig. [Figure] [Figure]

Original: 19 we report results for  the DenseNet-121 model on the ImageNet dataset
Revised: Next, to study whether our Conjectures are also valid for SGD with momentum, but held constant, we run the same experiment as in Sec. 4.2 using SimpleCNN on the CIFAR-10 dataset. For all runs, we set momentum to 0 . Learning rate 0 .

Revised: 1 diverged training, so we include only 0 . 01 and 0 . 001 .

Revised: We can observe that both Conjectures generalize to this setting.

Revised: We report the results in Fig.

Revised: 22 and

Revised: Fig. 23.

Revised: [Figure]

Revised: [Figure]

Revised: G A DDITIONAL EXPERIMENTS FOR SGD WITH LEARNING RATE DECAY

Revised: To understand the effect of learning rate decay, we run the same experiment as in Sec.

Revised: 4.2 using

Revised: SimpleCNN on the CIFAR-10 dataset.

Revised: Additionally, we divide the learning rate by the factor ofafter 100th epoch.

Revised: We can observe that Conjecture 1 and Conjecture 2 generalize to scenario with learning rate schedule in the sense that changing learning rate doesn’t change the relative ordering of the maximum λ 1 K and λ ∗ K /λ 1 K .

Revised: We report the results in Fig.

Revised: 24 and Fig.

Revised: [Figure]

Revised: [Figure]

Revised: H A DDITIONAL EXPERIMENTS FOR S IMPLE CNN-BN

Revised: To further explore the connection between our conjectures and the effects of batch-normalization layers on the conditioning of the loss surface

Original: [Figure]
Revised: [Figure]

Revised: we repeat here experiments from Sec. 4.

Revised: [Figure]

Revised: but varying the batch size.

Revised: Fig.

Revised: 26 summarizes the results.

Revised: On the whole

Original: [Figure]
Revised: [Figure]

Revised: conclusions from Sec.

Revised: 4.3 carry over to this setting in the sense that decreasing the batch size has a similar effect on the studied metrics as increasing the learning rate.

Revised: One exception is the experiment using the smallest batch size of 10 .

Revised: In this case, the maximum values of (cid:107) g (cid:107) (cid:107) g 5 (cid:107) and λ ∗ K /λ 1 K are smaller than in the experiments using larger batch sizes

Revised: [Figure]

Revised: [Figure]