Original: Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
Revised: Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

Original: Anonymous Author(s)

Original: AffiliationAddress email

Original: Abstract
Revised: Abstract

Original: A fundamental property of deep learning normalization techniques, such as batchnormalization, is making the pre-normalization parameters scale invariant.
Revised: A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant.

Original: Theintrinsic domain of such parameters is the unit sphere, and therefore their gradientoptimization dynamics can be represented via spherical optimization with varyingeffective learning rate (ELR), which was studied previously. 
Revised: The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously. However, the varying

Revised: ELR may obscure certain characteristics of the intrinsic loss landscape structure.

Original: In this work, weinvestigate the properties of training scale-invariant neural networks directly on thesphere using a fixed ELR.
Revised: In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR.

Original: We discover three regimes of such training dependingon the ELR value: convergence, chaotic equilibrium, and divergence.
Revised: We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence.

Original: We studythese regimes in detail both on a theoretical examination of a toy example and ona thorough empirical analysis of real scale-invariant deep learning models.
Revised: We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models.

Original: Eachof the regimes possesses its own unique features and has strong parallels withprevious research on both general andspecific scale-invariant neural networkstraining.
Revised: Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training.

Original: Finally, we demonstrate how the discovered regimes are reflected in theconventional training of normalized networks.
Revised: Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.

Original:  Introduction
Revised:  Introduction

Original: Most modern neural network architectures contain some type of normalization layers, such as Batch
Revised: Most modern neural network architectures contain some type of normalization layers, such as Batch

Original: Normalization (BN) [9] or Layer Normalization [2].
Revised: Normalization (BN) [9] or Layer Normalization [2].

Original: Normalization makes networks partially scaleinvariant (SI), i.e., multiplication of their parameters preceding the normalization layers by a positivescalar does not change the model’s output.
Revised: Normalization makes networks partially scaleinvariant (SI), i e., multiplication of their parameters preceding the normalization layers by a positive scalar does not change the model’s output.

Original: In general, the training dynamics of model parameters canbe viewed from two perspectives: the direction dynamics , i.e., the dynamics of the projection of theparameters onto the unit sphere, and the norm dynamics .
Revised: In general, the training dynamics of model parameters can be viewed from two perspectives: the direction dynamics , i e., the dynamics of the projection of the parameters onto the unit sphere and the norm dynamics .

Original: For SI neural networks, the former seemsto be more important since the direction alone determines the output of the scale-invariant function.
Revised: For SI neural networks, the former seems to be more important since the direction alone determines the output of the scale-invariant function.

Original: Nevertheless, the latter influences the optimization by changing the effective learning rate (ELR), i.e.,the learning rate  a scale-invariant model would have if it were optimized on the unit sphere.
Revised: However, the latter influences the optimization by changing the effective learning rate (ELR), i.e., the learning rate (LR) a scale-invariant model would have if it were optimized on the unit sphere.

Original: Many works have studied the effect of scale invariance on training dynamics through the prism ofnorm or, equivalently, ELR dynamics [28, 8, 1,  32, 21, 22, 25, 29, 23].
Revised: Many works have studied the effect of scale invariance on training dynamics through the prism of norm or, equivalently, ELR dynamics [31, 8, 1, 35, 23, 24, 27, 32, 25].

Original: They discovered that duringstandard training with weight decay the ELR can change in a non-trivial way and lead to differentoptimization dynamics: periodic behavior [23], destabilization [19, 21], or stabilization on somesphere [22, 29].
Revised: They discovered that during standard training with weight decay the ELR can change in a non-trivial way and lead to different optimization dynamics: periodic behavior [25], destabilization [21, 23], or stabilization on some sphere [24, 32].

Original: However, considering only the ELR dynamics is not enough for a comprehensivestudy of the specifics of training normalized neural networks.
Revised: However, considering only the ELR dynamics is not enough for a comprehensive study of the specifics of training normalized neural networks.

Original: In particular, the impact of the varying
Revised: In particular, the impact of the varying

Original: ELR on training may obscure certain important properties of SI neural networks intrinsic domain, i.e.,the unit sphere, thus narrowing our intuition about thegeneral loss landscape structure.
Revised: ELR on training may obscure certain important properties of SI neural networks’ intrinsic domain, i.e., the unit sphere, thus narrowing our intuition about the loss landscape structure.

Original: Therefore, inthis work, we are focused on the direction dynamics.
Revised: Therefore, in this work, we are focused on the direction dynamics.

Original: To eliminate the influence of the changing normon training, we fix the total norm of all scale-invariant parameters of the normalized neural networkand train it with a constant effective learning rate on the sphere.
Revised: To eliminate the influence of the changing norm on training, we fix the total norm of all scale-invariant parameters of the normalized neural network and train it with a constant effective learning rate on the sphere.

Original: We discover three regimes of such an optimization procedure depending on the ELR value (see Figure 1).
Revised: We discover three regimes of such an optimization procedure depending on the ELR value (see Figure 1).

Original: The first regime (small ELRs) can be considered as a typical convergence to a minimum withmonotonically decaying training loss.
Revised: The first regime (small ELRs) can be considered as a typical convergence to a minimum with monotonically decaying training loss.

Original: The second one (medium ELRs) demonstrates an oscillatingbehavior of the loss around a certain value separated from both the global minimum and  random guessbehavior.
Revised: The second one (medium ELRs) demonstrates a sustained oscillating behavior of the loss around a certain value separated from both the global minimum and the random guess behavior.

Original: We call this regime a chaotic equilibrium , reminiscent of the equilibrium state reportedin several previous works [22, 29].
Revised: We call this regime a chaotic equilibrium , reminiscent of the equilibrium state reported in several previous works [24, 32].

Original: The last regime (high ELRs) represents destabilized, divergedtraining associated with an excessively large optimization step size.
Revised: The last regime (high ELRs) represents destabilized, diverged training associated with an excessively large optimization step size.

Original: We thoroughly investigate thethree regimes of training SI neural networks on the sphere. We analyzethe underlying mechanisms behind each of these regimes on a model example and conduct anextensive empirical study with neural networks trained on the sphere.
Revised: We analyze the underlying mechanisms behind each of these regimes on a toy example and conduct an extensive empirical study with neural networks trained on the sphere using projected stochastic gradient descent (SGD). We point out the main features of each regime and study how they relate to each other and what peculiarities of the optimization dynamics and intrinsic structure of the loss landscape they reveal.

Revised: Our findings both reconfirmed some of the previous beliefs, like that training with larger (E)LR generally ends up in better optima, or that high-sharpness zones are present in the optimization trajectory, and introduced novel insights, like that different (E)LRs of the first regime even with the same initialization and batch ordering actually lead to distinguishable minima with different sharpness/generalization profiles depending on the specific (E)LR value, or that the high-sharpness zones are responsible for the boundary between the first and the second regimes and have close ties with the Double Descent phenomenon.

Original: We also provide connectionsbetween the three regimes and prior work on neural networks training, both general and scale-invariantspecific, and demonstrate how they are manifested inthe conventional neural networks training.
Revised: Finally, we demonstrate how our results are reflected in conventional neural network training and suggest several practical implications that help achieve optimal solutions in terms of sharpness and generalization.

Original:  Background and related work
Revised:  Background and related work

Original: As mentioned in the introduction, for every function F ( θ ) optimized w.r.t.
Revised: As mentioned in the introduction, for every function F ( θ ) optimized w.r.t.

Original: its parameters θ we canseparate the dynamics of the parameters direction θ / ∥ θ ∥ from the dynamics of their norm ∥ θ ∥ .
Revised: its parameters θ we can separate the dynamics of the parameters direction θ / ∥ θ ∥ from the dynamics of their norm ∥ θ ∥ .

Original: Ifthe function F ( θ ) is scale-invariant, i.e., F ( c θ )
Revised: If the function F ( θ ) is scale-invariant, i.e., F ( c θ )

Original: = F ( θ ) , ∀ c > 0 , the former is the only factor thatdetermines the evolution of the function, since F ( θ / ∥ θ ∥ )
Revised: = F ( θ ) , ∀ c > 0 , the former is the only factor that determines the value of the function, since F ( θ / ∥ θ ∥ )

Original: = F ( θ ) .
Revised: = F ( θ ) .

Original: However, even though the SIfunction’s value does not depend on the parameters norm, the latter greatly influences its gradients:∇ F ( c θ ) 
Revised: However, even though the SI function’s value does not depend on the parameters norm, the latter greatly influences its gradients: ∇ F ( c θ ) =

Original: = ∇ F ( θ ) /c, ∀ c > 0 and thus the optimization speed.
Revised: ∇ F ( θ ) /c, ∀ c > 0 and thus the optimization speed.

Original: This argument can be equivalentlyreformulated in terms of how the norm dynamics control the direction dynamics’ pace via the notionof effective learning rate.
Revised: This argument can be equivalently reformulated in terms of how the norm dynamics control the direction dynamics’ speed via the notion of effective learning rate.

Original: For the standard (S)GD optimization, ELR is defined as the learning rate(LR) divided by the squared parameters norm, and it represents how the learning rate should beadapted to mimic the same direction dynamics if the function were optimized directly on the unitsphere [25].
Revised: For the standard (S)GD optimization, ELR is defined as the learning rate divided by the squared parameters norm, and it represents how the learning rate should be adapted to mimic the same direction dynamics if the function were optimized directly on the unit sphere [27].

Original: Most of the previous work has dealt with the implications of using normalization in neural networks,i.e., making them (partially) scale-invariant, from the perspective of the ELR dynamics.
Revised: Most of the previous work has dealt with the implications of using normalization in neural networks, i.e., making them (partially) scale-invariant, from the perspective of the ELR dynamics.

Original: For example, Arora et al.
Revised: For example, Arora et al.

Original: [1] study the convergence properties of batch normalized neural networks and explainwhy they allow much higher learning rates compared to their non-BN counterparts.
Revised: [1] study the convergence properties of batch normalized neural networks and explain why they allow much higher learning rates compared to their non-BN counterparts.

Original: Many works havealso focused on training scale-invariant neural networks with weight decay [28, 32, 21, 22, 29, 23].
Revised: Many works have also focused on training scale-invariant neural networks with weight decay [31, 35, 23, 24, 32, 25].

Original: They all discovered the non-trivial dynamics of ELR induced by the interplay between normalizationand weight decay but came to different conclusions.
Revised: They all discovered the non-trivial dynamics of ELR induced by the interplay between normalization and weight decay but came to different conclusions.

Original: Some studies have reported training instabilitycaused by this interplay [19, 21], while others have argued that, on the contrary, the equilibrium state iseventually reached, i.e., ELR becomes stable after some time [28, 4, 22, 17, 29].
Revised: Some studies have reported training instability caused by this interplay [21, 23], while others have argued that, on the contrary, the equilibrium state is eventually reached, i.e., ELR becomes stable after some time [31, 4, 24, 19, 32].

Original: Recently, Lobachevaet al.
Revised: Recently, Lobacheva et al.

Original: [23] demonstrated that training with weight decay and constant LR may experience regularperiodic behavior for a practical range of LRs.
Revised: [25] demonstrated that training with weight decay and constant LR may experience regular periodic behavior for a practical range of LRs.

Original: However, while all of these works mainly study the direction dynamics with varying rate, they lackexploring the properties of optimization of SI models with a fixed effective learning rate.
Revised: However, while all of these works mainly study the direction dynamics with varying rate, they lack exploring the properties of optimization of SI models with a fixed effective learning rate.

Original: Decouplingthe direction dynamics from the changing norm effects could reveal particular features of the intrinsicdomain of scale-invariant neural networks that were potentially obscured by the dynamical ELR.
Revised: Decoupling the direction dynamics from the changing norm effects could reveal particular features of the intrinsic domain of scale-invariant neural networks that were potentially obscured by the dynamical ELR.

Original: To cope with it, we optimize our models directly on the sphere using projected stochastic gradient descent (SGD) with a constant learning rate, which is equivalent to fixing the ELR.
Revised: To cope with it, we optimize our models directly on the sphere using projected SGD with a constant learning rate, which is equivalent to fixing the ELR.

Original: Our methodreveals some interesting results about SI neural networks and allows to draw parallels with scatteredprevious deep learning work.
Revised: Our method reveals some interesting results about SI neural networks and allows to draw parallels with previous scattered deep learning work.

Original: We note that similar experiments were conducted by, e.g., Arora et al.[1] and Lobacheva et al.
Revised: We note that similar experiments were conducted by, e.g., Arora et al. [1] and Lobacheva et al.

Original: [23] to highlight certain features of standard training when ELR is dynamical.
Revised: [25] to highlight certain features of standard training when ELR is dynamical.

Original: Also, Cho and Lee [5] optimized scale-invariant parameters in normalized neural networks directlyon their intrinsic domain using Riemannian gradient descent, but they proposed an optimizationmethod rather than investigating the properties of such training.
Revised: Also, Cho and Lee [5] optimized scale-invariant parameters in normalized neural networks directly on their intrinsic domain using Riemannian gradient descent, but they proposed an optimization method rather than investigating the properties of such training.

Original:  Theoretical analysis
Revised:  Theoretical analysis

Original: To explain the observed behavior, we proceed with the theoretical analysis of scale-invariant functionsoptimization with a fixed effective learning rate.
Revised: To explain the observed behavior, we proceed with the theoretical analysis of scale-invariant functions optimization with a fixed effective learning rate.

Original: At first, we derive several important generalproperties of such a process that will help to clarify some of its empirical aspects.
Revised: At first, we derive several important general properties of such a process that will help to clarify some of its empirical aspects.

Original: Then we considera concrete example of a function with multiple scale-invariant parameter groups, alike normalizedneural networks, and study its convergence depending on the ELR value to shed more light on theprime causes of the three regimes.
Revised: Then we consider a concrete example of a function with multiple scale-invariant parameter groups, similar to normalized neural networks, and study its convergence depending on the ELR value to shed more light on the prime causes of the three regimes.

Original: We provide all formal derivations and proofs in Appendix A.
Revised: We provide all formal derivations and proofs in Appendix A.

Original: 3.1 General properties
Revised: 3.1 General properties

Original: We begin with an analysis of common features of all SI functions optimization with a fixed learningrate on the sphere.
Revised: We begin with an analysis of common features of all SI functions optimization with a fixed learning rate on the sphere.

Original: For that, consider a function F ( θ ) of a vector parameter θ ∈ R P that can bedivided into n groups: θ = ( θ 1 , . .
Revised: For that, consider a function F ( θ ) of a vector parameter θ ∈ R P that can be divided into n groups: θ = ( θ 1 , . .

Original: , θ n ) , where each θ i ∈ R p i and (cid:80) ni
Revised: , θ n ) , where each θ i ∈ R p i and (cid:80) ni

Original: =1 p i = P .
Revised: =1 p i = P .

Original: We say thateach of these groups is scale-invariant, i.e., we can multiply it by a positive scalar value, with theothers fixed, and the output of the function does not change.
Revised: We say that each of these groups is scale-invariant, i.e., we can multiply it by a positive scalar value, with the others fixed, and the output of the function does not change.

Original: This is typical for neural networkswith multiple normalized layers: each of these layers (and even each column/filter in each layer) isscale-invariant.
Revised: This is typical for neural networks with multiple normalized layers: each of these layers (and even each column/filter in a layer) is scale-invariant.

Original: Naturally, if a function is scale-invariant w.r.t.
Revised: Naturally, if a function is scale-invariant w.r.t.

Original: several parameter groups, it is alsoscale-invariant w.r.t.
Revised: several parameter groups, it is also scale-invariant w.r.t.

Original: their union, therefore the whole vector θ is scale-invariant too.
Revised: their union, therefore the whole vector θ is scale-invariant too.

Original: Since scale-invariant functions are effectively defined on the sphere, we try to minimize F ( θ ) on thesphere of radius ρ :
Revised: Since scale-invariant functions are effectively defined on the sphere, we try to minimize F ( θ ) on the sphere of radius ρ :

Original: [Equation]
Revised: [Equation]

Revised: where S P − 1 ( ρ ) = { θ ∈ R P :

Revised: ∥ θ ∥ = ρ } .

Original: We do that using the projected gradient descent method with a fixed learning rate η :
Revised: We do that using the projected gradient descent method with a fixed learning rate η :

Original: [Equation]
Revised: [Equation]

Original: Of course, since the function is scale-invariant w.r.t.
Revised: Of course, since the function is scale-invariant w.r.t.

Original: each  group θ i , we could optimize it on the productof n independent spheres  instead of the single mutual sphere, as we do in eq.
Revised: each individual group θ i , we could optimize it on the product of n independent spheres S p i − 1 instead of the single mutual sphere, as we do in eq.

Original: However, thatwould either require introducing too many hyperparameters in the model ( n different learning rates)or overconstrainted the problem (one learning rate for all  spheres).
Revised: However, that would either require introducing too many hyperparameters in the model ( n different learning rates for n spheres) or overconstrained the problem (one learning rate for all n spheres).

Original: We also argue that constrainingthe total parameters norm is much more natural and closer to the standard neural networks training(see discussion in Appendix B).
Revised: We also argue that constraining the total parameters norm is much more natural and closer to the standard neural networks training with weight decay (see discussion in Appendix A.1).

Original: We now define the notions of effective gradient and effective learning rate for each SI group θ i .
Revised: We now define the notions of effective gradient and effective learning rate for each SI group θ i .

Original: Effective gradient is the gradient w.r.t.
Revised: Effective gradient is the gradient w.r.t.

Original: θ i measured at the same point but with the group θ i projectedon its unit sphere, i.e., at point ˜ θ
Revised: θ i measured at the same point but with the group θ i projected on its unit sphere, i.e., at point ˜ θ

Original: =
Revised: =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: , θ i /ρ
Revised: , θ i /ρ

Original: i , . .
Revised: i , . .

Original: , θ n ) , where ρ i ≡ ∥ θ i ∥ .
Revised: , θ n ) , where ρ i ≡ ∥ θ i ∥ .

Original: Due to scaleinvariance w.r.t.
Revised: Due to scale invariance w.r.t.

Original: θ i , the function’s value is equal at both points θ and ˜ θ
Revised: θ i , the function’s value is equal at both points θ and ˜ θ

Original: but, as discussed earlier, thenorm of the effective gradient, which we denote as ˜ g i , is ρ i times the norm of the regular gradientw.r.t.
Revised: but, as discussed earlier, the norm of the effective gradient, which we denote as ˜ g i , is ρ i times the norm of the regular gradient w.r.t.

Original: θ i : ˜ g
Revised: θ i : ˜ g

Original: i =
Revised: i =

Original: g
Revised: g

Original: i ρ i , where g i ≡ ∥∇ θ i F ( θ ) ∥ .
Revised: i ρ i , where g i ≡ ∥∇ θ i F ( θ ) ∥ .

Original: Effective learning rate ˜ η
Revised: Effective learning rate ˜ η

Original: i is the learning rate requiredto make a gradient step w.r.t.
Revised: i is the learning rate required to make a gradient step w.r.t.

Original: θ i equivalent to the original but started from the point ˜ θ .
Revised: θ i equivalent to the original but started from the point ˜ θ .

Original: It is knownfrom the previous literature on scale-invariant functions [25, 29, 23] that ELR for the group θ i equals˜ η i =
Revised: It is known from the previous literature on scale-invariant functions [27, 32, 25] that ELR for the group θ i equals ˜ η i =

Original: η/ρ 2 i .
Revised: η/ρ 2 i .

Original: We also define the total effective gradient norm ˜ g
Revised: We also define the total effective gradient norm ˜ g

Original: = ρg , where g ≡ ∥∇ θ F ( θ ) ∥ , and thetotal ELR ˜ η = η/ρ 2 , which is fixed in the considered setup .
Revised: = gρ , where g ≡ ∥∇ θ F ( θ ) ∥ , and the total ELR ˜ η = η/ρ 2 , which is fixed in the considered setup.

Original: From the constraint on the total norm (cid:80) ni =1 ρ 2 i =
Revised: From the constraint on the total norm (cid:80) ni =1 ρ 2 i =

Original: ρ 2 we obtain the following fundamental equationrelating the ELRs of individual SI groups to the total ELR value:
Revised: ρ 2 we obtain the following fundamental equation relating the ELRs of individual SI groups to the total ELR value:

Original: [Equation]
Revised: [Equation]

Original: Another important notion is the effective step size (ESS) which is the product of the effective gradientlength by the effective learning rate: ˜
Revised: Another important notion is the effective step size (ESS), which is the product of the effective gradient length by the effective learning rate: ˜

Original: η i ˜ g i .
Revised: η i ˜ g i .

Original: This value defines how far we effectively move thescale-invariant parameters θ i after one gradient step.
Revised: This value defines how far we effectively move the scale-invariant parameters θ i after one gradient step.

Original: It can be shown that the total squared ESS canbe expressed as a convex combination of squared ESS values of individual SI groups:
Revised: It can be shown that the total squared ESS can be expressed as a convex combination of squared ESS values of individual SI groups:

Original: [Equation]
Revised: [Equation]

Original: The process (2) and eq.
Revised: The process (2) and eq.

Original: 3 yield the following update rule for each group’s individual ELR:
Revised: 3 yield the following update rule for each group’s individual ELR:

Original: [Equation]
Revised: [Equation]

Original: from what follows that for a given SI group the higher (lower) the ESS at the current iteration,the lower (higher) the ELR becomes after it .
Revised: from which it follows that for a given SI group the higher (lower) the ESS at the current iteration, the lower (higher) the ELR becomes after it .

Original: Since, by definition, ESS and ELR values are highlycorrelated, we conclude that at each iteration the largest ELRs tend to become smaller and viceversa.
Revised: Since, by definition, ESS and ELR values are highly correlated, we conclude that at each iteration the largest ELRs tend to become smaller and vice versa.

Original: This “negative feedback” principle becomes very important in distinguishing between the threeregimes of optimization.
Revised: This “negative feedback” principle becomes very important in distinguishing between the three regimes of optimization.

Original: 3.2 Explaining the regimes
Revised: 3.2 Explaining the regimes

Original: To provide a clear and easy-to-follow explanation of the differences between the three regimes oftraining on the sphere, we construct a simple yet illustrative example of a function with several groupsof SI parameters and analyze its optimization properties depending on the total ELR value.
Revised: To provide a clear and easy-to-follow explanation of the differences between the three regimes of training on the sphere, we construct a simple yet illustrative example of a function with several groups of SI parameters and analyze its optimization properties depending on the total ELR value.

Original: The simplest working example of a scale-invariant function is the following one:
Revised: The simplest working example of a scale-invariant function is the following one:

Original: [Equation]
Revised: [Equation]

Original: This function has also been used in previous works to visually demonstrate various properties ofgeneral scale-invariant functions [22, 23].
Revised: This function has also been used in previous works to visually demonstrate various properties of general scale-invariant functions [24, 25].

Original: Based on it, we develop an elucidative example of afunction that contains more than one group of scale-invariant parameters — just as real multilayernormalized neural networks do.
Revised: Based on it, we develop an elucidative example of a function that contains more than one group of scale-invariant parameters — just as real multilayer normalized neural networks do.

Original: For that, we take a conical combination of n > 1 functions (6) eachdepending on its own pair of variables:
Revised: For that, we take a conical combination of n > 1 functions (6) each depending on its own pair of variables:

Original: [Equation]
Revised: [Equation]

Original: where x = ( x 1 , . .
Revised: where x = ( x 1 , . .

Original: , x n ) ,
Revised: , x n ) ,

Original: y =
Revised: y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) , and each α i > 0 is a fixed positive coefficient.
Revised: , y n ) , and each α i > 0 is a fixed positive coefficient.

Original: Inthis function, there are n groups of SI parameters: ( x i , y i ) , i = 1 , . . . , n .
Revised: In this function, there are n groups of SI parameters: ( x i , y i ) , i = 1 , . . . , n .

Original: In accordance withthe setup described previously, we intend to optimize this function on the unit sphere using theprojected gradient descent method with a fixed (effective) learning rate, i.e., we plug θ =
Revised: In accordance with the setup described previously, we intend to optimize this function on the unit sphere using the projected gradient descent method with a fixed (effective) learning rate, i.e., we plug θ =

Original: ( x , y ) =(( x 1 , y 1 ) , . .
Revised: ( x , y ) = (( x 1 , y 1 ) , . .

Original: , ( x n , y n )) , P = 2 n , ρ = 1 into equations (1) and (2).
Revised: , ( x n , y n )) , P = 2 n , ρ = 1 into equations (1) and (2).

Original: Now we would like to study the behavior of optimization of F ( x , y ) on the unit sphere depending onthe learning rate value η , which equals to the total ELR ˜
Revised: Now we would like to study the behavior of optimization of F ( x , y ) on the unit sphere depending on the learning rate value η , which equals to the total ELR ˜

Original: η due to ρ = 1 .
Revised: η due to ρ = 1 .

Original: But first we need to dwell onthe convergence properties of each of its subfunctions α i f
Revised: But first we need to dwell on the convergence properties of each of its subfunctions α i f

Original: ( x i , y i ) .
Revised: ( x i , y i ) .

Original: The next proposition, which werigorously formulate and prove in the appendix, answers this question.
Revised: The next proposition, which we rigorously formulate and prove in Appendix A.3, answers this question.

Original: Proposition 1 For the SI function f α ( x, y ) =
Revised: Proposition 1 For the SI function f α ( x, y ) =

Original: α x 2 x 2 + y 2 minimized with ELR ˜ η :
Revised: α x 2 x 2 + y 2 minimized with ELR ˜ η :

Original: if ˜ η < 1 α , it linearly converges to zero, i.e., its global minimum; 
Revised: if ˜ η < 1 α , it linearly converges to zero, i.e., its global minimum; 2.

Original: if ˜ η > 1 α , it [Table]
Revised: if ˜ η > 1 α , it stabilizes at value 12 (cid:16) α − 1˜ η (cid:17) .

Original: Knowing that each of the subfunctions of F ( x , y ) can converge to the minimum only when its ELRstays belowthe 1 /α ivalue and considering the fundamental constraint on ELRs (3), we state that thefirst regime (convergence) is only possible when the total ELR is below a certain threshold:
Revised: Knowing that each of the subfunctions of F ( x , y ) can converge to the minimum only when its ELR stays below 1 /α i and considering the fundamental constraint on ELRs (3), we state that the first regime (convergence) is only possible when the total ELR is below a certain threshold:

Original: [Equation]
Revised: [Equation]

Original: If the total ELR exceeds that threshold, the overall convergence is impossible, since at least oneof individual ELRs must exceed its convergence threshold value.
Revised: What if the total ELR exceeds that threshold? The first regime cannot be observed: convergence becomes impossible, since at least one of the individual ELRs exceeds its convergence threshold value, according to eq.

Original: By Proposition 1, the value ofeach such subfunction isalso separated from the minimum (zero) by some quantity depending onits ELR.
Revised: By Proposition 1 the value of the corresponding subfunction is separated from zero by some quantity depending on its ELR.

Original: This means that the total function value, and therefore the total effective gradient, is alsonon-vanishing (we give a more formal argument in the appendix).
Revised: This means that the total function value, and therefore the total effective gradient, is also nonvanishing (we give a more formal argument in Appendix A.4).

Original: This brings us to the “undamped”dynamics of ELRs (5), where the effective step sizes do not tend to zero.
Revised: This brings us to the “undamped” dynamics of ELRs (5), where the effective step sizes do not tend to zero.

Original: In that case, the dynamicsstabilizes only when all the ESS values become equal , which is possible if and only if each individual 
Revised: In that case, the dynamics stabilize only when all the ESS values become equal, which is possible if and only if each individual ELR ˜

Original: ELR ˜ η i takes its corresponding equilibrium value:
Revised: η i takes its corresponding equilibrium value:

Original: [Equation]
Revised: [Equation]

Original: Due to the negative feedback principle of eq.
Revised: Due to the negative feedback principle of eq.

Original: (5), the system will naturally tend towardsthis state.
Revised: (5), the system will naturally tend towards this state when

Original:  Note that, in contrast, in the first regime ESS values decay too rapidly, andthe model may not have enough time to reach equilibrium.
Revised: ESS values are non-negligible. Note that, in contrast, in the first regime, ESS values decay too rapidly due to convergence, and the model may fail to reach equilibrium.

Original: In sum, when the total ELRexceeds the convergence threshold and each individual ELR (in average) equals its equilibrium value corresponding to the equalized ESS, the optimization process enters the secondregime, i.e., the chaotic equilibrium .
Revised: In sum, when the total ELR exceeds the convergence threshold and each individual ELR (in average) equals its equilibrium value corresponding to the equalized ESS, the optimization process enters the second regime, i.e., the chaotic equilibrium .

Original: Finally, if the total ELR is set too high, the chaoswill dominate equilibration, and in that case we observe the third regime, or divergence.
Revised: Finally, if the total ELR is set too high, the chaos will dominate equilibration and the optimization dynamics will resemble a random walk. In that case, we observe the third regime, or divergence

Original: In Figure 2, we plot the evolution of the function (7) during optimizationon the unit sphere with three different effective learning rates.
Revised: In Figure 2, we plot the evolution of the function (7) during optimization on the unit sphere with three different effective learning rates.

Original: We set n =3 , α i =
Revised: We set n = 3 , α i =

Original: i , i = 0 , 1 , 2 , and start from a point randomly selected on the 6dimensional unit sphere.
Revised: i , i = 0 , 1 , 2 , and start from a point randomly selected on the 6-dimensional unit sphere.

Original: In this case, the threshold condition (8) becomes˜ η < 1 / 7 ≈ 0 .
Revised: In this case, the threshold condition (8) becomes ˜ η < 1 / 7 ≈ 0 .

Original: 143 , and thus we select three ELRs corresponding to thethree optimization regimes, respectively: 0 . 1 , 0 . 2 , and 0 .
Revised: 143 , and thus we select three ELRs corresponding to the three optimization regimes, respectively: 0 . 1 , 0 . 2 , and 0 .

Original: The lineson the plot directly demonstrate the anticipated behavior: the smallest ELR leads to a fast decay of the function’s value, the medium one makesthe function stabilize around a certain level, and the largest ELR depictsthe most chaotic behavior.
Revised: The lines on the plot directly demonstrate the anticipated behavior: the smallest ELR leads to a fast decay of the function’s value, the medium one makes the function stabilize around a certain level, and the largest ELR depicts the most chaotic behavior.

Original: In the appendix, we also provide plots of theindividual ELRs dynamics that show that in the chaotic equilibrium ˜ η iin average by iterations closely match the predicted values (9), while inthe first regime they are not reached due to quick convergence, and in thethird one the optimization is too unstable to distinguish any equilibration.
Revised: In Appendix A.5, we also provide plots of the individual ELRs dynamics that show that in the chaotic equilibrium ˜ η i in average by iterations closely match the predicted values (9), while in the first regime they are not reached due to quick convergence, and in the third one the optimization is too unstable to distinguish any equilibration.

Original:  Experimental setup
Revised:  Experimental setup

Original: After discussing the theoretical substantiation of the different regimes of learning on the sphere in thecase of a simple SI function, let us move on to an empirical analysis of the learning dynamics of neuralnetworks equipped with Batch Normalization.
Revised: After discussing the theoretical substantiation of the different regimes of learning on the sphere in the case of a simple SI function, let us move on to an empirical analysis of the learning dynamics of neural networks equipped with Batch Normalization.

Original: We conduct experiments with a simple 3-layer BNconvolutional neural network (ConvNet) and a ResNet-181 on CIFAR-10 [15] and CIFAR-100 [16]datasets.
Revised: We conduct experiments with a simple 3-layer BN convolutional neural network (ConvNet) and a ResNet-18 on CIFAR-10 [17] and CIFAR-100 [18] datasets.

Revised: Both networks are in the implementation of Lobacheva et al.

Original: We train all networks by optimizing cross-entropy loss using stochastic gradient descentwith batch size 128 for 1000 epochs.
Revised: [25] and we train them by optimizing cross-entropy loss using stochastic gradient descent with batch size 128 for 1000 epochs.

Original: In Section 5, we analyze the training regimes of fully scale-invariant neural networks, trained onthe sphere with a fixed ELR.
Revised: In Section 5, we analyze the training regimes of fully scale-invariant neural networks, trained on the sphere with a fixed ELR.

Original: To obtain fully scale-invariant versions of the networks we follow thesame approach as in Lobacheva et al.
Revised: To obtain fully scale-invariant versions of the networks we follow the same approach as in Lobacheva et al.

Original: We fix the non-scale-invariant weights, i.e., we use zeromean and unit variance in BN layers and freeze the weights of the last layer at random initializationwith increased norm equal to 10.
Revised: We fix the non-scale-invariant weights, i.e., we use zero shift and unit scale in BN layers and freeze the weights of the last layer at random initialization with increased norm equal to 10.

Original: To train a network with ELR ˜ η , we fix the norm of the weightsat the initial value ρ and use the projected SGD with a fixed learning rate η = ˜ ηρ 2 .
Revised: To train a network with ELR ˜ η , we fix the norm of the weights at the initial value ρ and use the projected SGD with a fixed learning rate η = ˜ ηρ 2 .

Original: We considera wide range of ELRs: { 10 − k , 2 · 10 − k , 5 · 10 − k } k =0 , 7 , with additional values close to the regime transitions.
Revised: We consider a wide range of ELRs: { 10 − k , 2 · 10 − k , 5 · 10 − k } k =0 , 7 , which more than covers the values of ELR usually encountered during training of regular networks. We also used a more fine grained ELR grid closer to the boundaries between the regimes.

Original: In Section 6, we investigate how the observed training regimes transfer to more practicalscenarios.
Revised: In Section 6, we investigate how the observed training regimes transfer to more practical scenarios.

Original: We trainthe networks in the whole parameter space, not only on the sphere, with a fixedlearning rate and weight decay as in Lobacheva et al.
Revised: We train networks in the whole parameter space with a fixed learning rate and weight decay as in Lobacheva et al.

Original: [23], Wan et al.
Revised: [25], Wan et al.

Original: We use weight decay of1 e − 4 / 5 e − 4 for ConvNet / ResNet-18 in all the experiments and a range of LRs, specific for eachscenario.
Revised: We use weight decay of 1 e − 4 / 5 e − 4 for ConvNet / ResNet-18 in all the experiments and a range of LRs, specific for each scenario.

Original: In some experiments, we also use momentum of 0.9, standard CIFAR data augmentation,and cosine LR schedule of 200 epochs.
Revised: In some experiments, we also use momentum of 0.9, standard CIFAR data augmentation, and cosine LR schedule for 200 epochs.

Original: More details can be found in Appendix D.
Revised: More details can be found in Appendix C.

Original: To make the experiments with different ELRs/LRs fully comparable, we train the networks fromthe same random initialization and with the same optimizer random seed.
Revised: To make the experiments with different ELRs/LRs fully comparable, we train the networks from the same random initialization and with the same optimizer random seed.

Original: However, we show in
Revised: However, we show in

Original: Appendix F that the results are consistent if we vary both of them.
Revised: Appendix E that the results are consistent if we vary both of them.

Original: For each visualization we choosethe most representative subset of ELRs/LRs due to the difficulties of distinguishing between manynoisy lines on the plots.
Revised: For each visualization we choose the most representative subset of ELRs/LRs due to the difficulties of distinguishing between many noisy lines on the plots.

Original: At each training epoch, we log standard train / test statistics and the mean(over mini-batches) norm of the stochastic effective gradients for fully scale-invariant networks or themean norm of the regular stochastic gradients in the case of standard networks.
Revised: At each training epoch, we log standard train / test statistics and the mean (over mini-batches) norm of the stochastic effective gradients for fully scale-invariant networks or the mean norm of the regular stochastic gradients in the case of standard networks.

Original: We choose the lattermetric for the following two reasons.
Revised: We choose this metric for the following two reasons.

Original: First, it can serve as a measure of sharpness of the minimum towhich we converge in the first regime, since the mean stochastic gradient norm is strongly correlatedwith the trace of the gradient covariance matrix, which in turn is closely tied with the loss Hessianand the Fisher Information Matrix [12, 31, 27] (see Appendix C).
Revised: First, it can serve as a measure of sharpness of the minimum to which we converge in the first regime, since the mean stochastic gradient norm is strongly correlated with the trace of the gradient covariance matrix, which in turn is closely tied with the loss Hessian and the Fisher Information Matrix [12, 34, 30] (see Appendix B).

Original: Second, it represents the average(effective) step size up to multiplication by (E)LR and thus is also related to optimization dynamics.
Revised: Second, it represents the average (effective) step size up to multiplication by (E)LR and thus is also related to optimization dynamics.

Original:  Three regimes of training on the sphere
Revised:  Three regimes of training on the sphere

Original: In this section, we investigate the three training regimes of fully scale-invariant neural networks,trained on the sphere with a fixed ELR.
Revised: In this section, we investigate the three training regimes of fully scale-invariant neural networks, trained on the sphere with a fixed ELR.

Original: We first analyze the learning dynamics in each regime,including the properties of the resulting solutions and the loss landscape around them.
Revised: We first analyze the learning dynamics in each regime, including the properties of the resulting solutions and the loss landscape around them.

Original: After that,we discuss transitions between the regimes in more detail.
Revised: After that, we discuss transitions between the regimes in more detail.

Original: We show the results for ConvNet on
Revised: We show the results for ConvNet on

Original: CIFAR-10, the results for other dataset-architecture pairs are consistent and presented in Appendix E.
Revised: CIFAR-10, the results for other dataset-architecture pairs are consistent and presented in Appendix D.

Original: 5.1 Regime one: convergence
Revised: 5.1 Regime one: convergence

Original: Training with the smallest ELRs results in the first training regime, which we call the convergenceregime .
Revised: Training with the smallest ELRs results in the first training regime, which we call the convergence regime .

Original: As shown in Figure 1, optimization in this regime experiences a typical convergence behavior:after a number of epochs the model is able to achieve a very low training lossregion and continuesconverging to the minimum.
Revised: As shown in Figure 1, optimization in this regime experiences a typical convergence behavior: after a number of epochs the model is able to reach regions with very low training loss and continues converging to the minimum.

Original: The speed of convergence depends on the ELR value — the higher the ELR, the faster the optimization, and the lower training loss is achieved after a fixed number of epochs.
Revised: The speed of convergence depends on the ELR value — the higher the ELR, the faster the optimization, and the lower training loss is achieved after a fixed number of epochs.

Original: Training with different ELRsalso results in solutions with different sharpness and generalization. Wemeasure sharpness of a solution as the average norm of stochastic effective gradients at that point andgeneralization as its test error.
Revised: Also, training with different ELRs results in solutions with different sharpness (mean norm of stochastic effective gradients) and generalization (test error): higher

Original: Higher ELRs lead to less sharp solution with better generalization.
Revised: ELRs lead to less sharp solution with better generalization.

Original: Similar results are well known in the literature [12, 30, 20, 14, 18, 6, 3, 26, 11, 7]: training neuralnetworks using larger learning rates typically results in flatter and better generalizing optima.
Revised: Similar results are well known in the literature [12, 33, 22, 28, 14, 20, 6, 3, 29, 11, 7], i.e., training neural networks using larger learning rates typically results in flatter and better generalizing optima.

Revised: We discovered that the optima achieved after training with different ELRs not only vary in sharpness and generalization but also reside in different basins, i.e., there is a significant loss barrier on the linear interpolation between them (see

Revised: Appendix I for more details).

Original: Furthermore, ELR  not onlyinfluences the rate of convergence and the properties of the final solutionbut its local neighborhood and the whole training trajectory as well.
Revised: The value of ELR affects not only the rate of convergence and the properties of the final solution; it may have an effect on the whole training trajectory as well.

Original: To analyze the training trajectoriesindependently from the training speed, we look at the evolution of sharpness and generalization vs.training loss during the optimization.
Revised: To analyze the training trajectories independently from the training speed, we look at the evolution of sharpness and generalization vs. training loss during the optimization.

Original: We show the corresponding plots for different ELR values inthe first regime in Figure 3, left.
Revised: We show the corresponding plots for different ELR values of the first regime in Figure 3, left.

Original: For the lowest ELRs the trajectories coincide, but the training is too[Figure] achieved with different ELRs are not linearly connected. slow to converge to the low-loss region.
Revised: For the lowest ELRs the trajectories coincide, but the training is too slow to converge to the low-loss region.

Original: For the remaining values, as we increase ELR, we obtain alower trajectory, and in the low-loss region trajectories of different ELRs appear like parallel lines.
Revised: For the remaining values, as we increase ELR, we obtain a lower trajectory, and in the low-loss region trajectories of different ELRs appear as parallel lines.

Original: Such differences in trajectories show that training with a higher ELR not only results in a flatter finalsolution but also traverses less sharp regions the whole time: for any fixed value of the training loss,the point corresponding to a higher ELR has lower sharpness and better generalization. This is alsoconsistent with recent results on the impact of higher learning rates on better optimization trajectoryconditioning [14, 6, 11, 7]. The optima achieved after training with different ELRs not only vary in sharpness and generalizationbut also lie in different loss landscape regions even if optimization starts from the same initializationand uses the same order of batches. In Figure 4 we check whether the solutions achieved with different
Revised: [Figure]

Original: ELRs are linearly connected. For each pair ofneighboring ELRs we connect the corresponding finalweight vectors with a line segment and look at the training loss along this segment. The results showthat thesolutions that converged to a low-loss region lie in different basins and there exist barriers ofhigh loss between them. The resulting basins  have different  sharpness and generalization profiles, which are reflected asparallel lines in the low-loss region in Figure 3, left.
Revised: The parallelism of the trajectories in the low-loss region is particularly interesting and leads us to the following question: do the basins to which training with different ELRs converges have different overall sharpness and generalization profiles?

Original: Moreover, each basin has its own specific profilein a sense that  SGD cannot find a solution with significantly different sharpness and generalizationin that basin.
Revised: That is, can we hypothesize that all solutions reachable with SGD in a given basin have similar sharpness and generalization at the same training loss?

Original: To confirm that, we take a solution achieved with a specific ELR and finetune it withlower and higher ELR values (see Figure 3, right).
Revised: To answer that, we take a solution achieved with a specific ELR and fine-tune it with lower and higher ELR values (see Figure 3, right).

Original: Finetuning with  lower ELR  continues to movealong the same trajectory as the initial experiment, hencethere are no sharper sub-minima observedinside the flat basin.
Revised: Fine-tuning with a lower ELR stays in the same basin and continues to move along the same trajectory as the initial experiment, hence no sharper sub-minima can be found inside the flat basin.

Original: Finetuning with  higher ELR also maintains the same trajectory at first butaftersome time jumps out of the low-loss region  and converges to a new basin with a profile typical forhigher ELRs.
Revised: Fine-tuning with a higher ELR also maintains the same trajectory at first but jumps out of the low-loss region after a while and converges to a new basin with a profile attributed to that ELR.

Original: It means that the initial basin is too sharp for the higher ELR to converge.
Revised: That means that the initial basin is too sharp for the higher ELR to converge.

Original: The resultsare consistent for different ELR values, see Appendix  I.
Revised: The results are consistent for different ELR values, see Appendix H; we also provide results on linear connectivity of pre-trained and fine-tuned solutions in Appendix I.

Original: As can be seen from Figure 3, left, the range of sharpness and generalization profiles reachable withthe fixed ELR training in the first regime is limited.
Revised: As can be seen from Figure 3, left, the range of sharpness and generalization profiles reachable with the fixed ELR training in the first regime is limited.

Original: For low ELRs the trajectories coincide andoptimization converges to the sharpest reachable basin.
Revised: For small ELRs, the trajectories coincide and optimization converges to the sharpest available basin.

Original: Increasing ELR results in flatter optima, butafter some ELR optimization stops converging to low-loss regions and demonstrates the behavior ofthe second training regime, which we describe next.
Revised: Increasing ELR results in flatter optima, but at some value the optimization stops converging to low-loss regions and demonstrates the behavior of the second training regime, which we describe next.

Original: 5.2 Regimes two and three: chaotic equilibrium and divergence
Revised: 5.2 Regimes two and three: chaotic equilibrium and divergence

Original: A further increase in ELR in the first regime leads to a sharp switch to another regime, the chaoticequilibrium .
Revised: A further increase of ELR in the first regime leads to a sharp switch to another regime, the chaotic equilibrium .

Original: Figure 1 shows that after a certain threshold ELR, the optimization is no longer able toreach low-loss regions and gets stuck at some  level.
Revised: Figure 1 shows that after a certain threshold ELR, the optimization is no longer able to reach low-loss regions and gets stuck at some loss level.

Original: We see that the effective gradients norm alsostabilizes, and higher ELRs correspond to a higher loss but lower sharpness.
Revised: We observe that the effective gradients norm also stabilizes, and higher ELRs correspond to a higher loss but lower sharpness.

Original: As we determined in Section 3.2, the second regime corresponds to the state when all individualeffective step sizes become equal, which also leads to  stabilization of individual ELRs.
Revised: As we determined in Section 3.2, the second regime corresponds to the state when all individual effective step sizes become equal, which also leads to the stabilization of individual ELRs.

Original: In Figure 5,left, we depict the values (averaged over last 200 epochs) of individual ELRs and effective gradientnorms w.r.t.
Revised: In Figurewe depict the values (averaged over last 200 epochs) of individual ELRs and effective gradient norms w.r.t.

Original: each SI parameters group of each layer of the neural network trained with ˜ η
Revised: each SI parameters group of each layer of the neural network trained with ˜ η

Original: = 10 − 3 .
Revised: = 10 − 3 .

Original: We see that the effective step sizes (ELR times effective gradient norm) indeed concentrate around acertain equilibrium value denoted as a dotted black line on the plot.
Revised: We see that the effective step sizes (ELR times effective gradient norm) indeed concentrate around a certain equilibrium value denoted as a dashed black line on the plot.

Original: In Appendix J, we provide results for other ELRs and additional plots demonstrating the stabilization of individual ELRs and effectivegradient norms, which corroborates our definition of the second regime as the chaotic equilibrium. 
Revised: In Appendix J, we provide results for other ELRs and additional plots demonstrating the stabilization of individual ELRs and effective gradient norms, which corroborates our definition of the second regime as the chaotic equilibrium. In the second regime, the optimization localizes a certain region in the loss landscape and “hops along its walls”. For moderate ELRs, this region is locally convex in the sense that the loss on the linear path [Figure]

Original: Now, what if we finetune a model from the  second regime  with a smaller ELR corresponding to thefirst regime?
Revised: between different checkpoints from the same trajectory is convex and generally much lower than at its edges. However, for the highest ELRs, the dynamics become more chaotic and distinguished barriers may appear on the linear path. We provide the corresponding plots in Appendix K. Each ELR of the second regime determines its own loss level — the height at which the optimization hops. Changing the ELR value during training appropriately changes the training dynamics: increasing/decreasing the ELR directly leads to the trajectory with a loss level corresponding to the new ELR value (see

Revised: Figure 5, left).

Revised: This significantly distinguishes the second regime from the first, where fine-tuning with a different ELR does not affect the learning trajectory, unless the new ELR is too large, in which case the optimization suddenly jumps out of the basin.

Revised: Now, what if we fine-tune a model from the second regime with a smaller ELR corresponding to the first regime?

Original: As can be seen from Figure 5, right, the resulting optima highly depend on the starting
Revised: As can be seen from Figure 5, right, the resulting optima highly depend on the starting

Original: ELR (more results in Appendix K).
Revised: ELR (more results in Appendix L).

Original: Finetuning networks, obtained with low second regime ELRs,with any first regime ELR converges to basins with the same sharpness/generalization profile thatcorresponds to the maximum ELR in the first regime.
Revised: Fine-tuning the models, pre-trained with low second regime ELRs, using any first regime ELR converges to basins with the same sharpness/generalization profile that is attributed to the maximum ELR in the first regime.

Original: For larger second regime ELRs, however,finetuning results in a variety of trajectories depending on the chosen  first regime ELR.
Revised: For larger second regime ELRs, however, fine-tuning results in a variety of trajectories depending on the chosen ELR of the first regime.

Original: It means thattraining with low second regime ELR resides in regions containing only flat optima.
Revised: That means that training with low second regime ELR takes place in regions containing only flat minima.

Original: Meanwhile,basins obtained with high ELRs contain optima of different sharpness.
Revised: Meanwhile, regions discovered with high ELRs contain optima of different sharpness.

Original: Moreover, finetuning leads tothe same range of sharpness/generalization profiles that was observed in the first regime.
Revised: Moreover, fine-tuning from these regions results in the same range of sharpness/generalization profiles that was observed in the first regime.

Original: For the highest considered ELR values, we observe the most unstable optimization behavior corresponding to the random guess accuracy (see Figure 1).
Revised: For the highest considered ELR values, we observe the most unstable optimization behavior corresponding to the random guess accuracy (see Figure 1).

Original: This is the third, divergent training regime .
Revised: We call it the third, divergent training regime .

Original: We  compare it with random walk in Appendix L.
Revised: We explicitly compare the third regime with random walk in Appendix M.

Original: 5.3 Transitions between the regimes
Revised: 5.3 Transitions between the regimes

Original: In the end, we would like to discuss the connections between the threeregimes and our global view of training on the sphere.
Revised: In the end, we would like to discuss the connections between the three regimes and our global view of training on the sphere.

Original: In Figure 6, weplot the training loss for several ELR values that correspond to the threeregimes and transitions between them.
Revised: In Figure 6, we plot the training loss for several ELR values that correspond to the three regimes and transitions between them.

Original: When the ELR is too high (purpleline), the total ESS becomes so large that the model is unable to evendetect any region in the parameters space with non-trivial quality, andwe observe the third regime.
Revised: When the ELR is too high (purple line), the total ESS becomes so large that the model is unable to even detect any region in the parameters space with non-trivial quality, and we encounter the third regime.

Original: With a smaller ELR (light blue line), themodel starts occasionally hitting the edge of the region with a (relatively)low loss but then quickly escapes it and returns back to the random guessbehavior; this could be attributed to the border regime between the secondand the third regimes.
Revised: With a smaller ELR (light blue line), the model starts occasionally hitting the edge of the region with a (relatively) low loss but then quickly escapes it and returns back to the random guess behavior; this could be attributed to the border regime between the second and the third regimes.

Original: Setting the ELR to an even smaller value (greenline) results in the chaotic equilibrium, when the loss is stabilized neara certain value, which is lower the lower the ELR.
Revised: Setting the ELR to an even smaller value (green line) results in the chaotic equilibrium, when the loss is stabilized near a certain value, which is lower the lower the ELR.

Original: For ELR values between the highest ELRs ofthe first regime and the lowest ones of the second regime (yellow line), we can witness a suddenchange from equilibrium behavior to convergence.
Revised: For ELR values between the highest ELRs of the first regime and the lowest ones of the second regime (yellow line), we can witness a sudden change from equilibrium behavior to convergence.

Original: We conjecture that the major difference betweenthe first and second optimization regimes and such a sharp transition between them, in particular,is due to the presence of zones of  increased sharpness in the optimization trajectories of neuralnetworks [13, 14, 18, 11, 7].
Revised: We conjecture that the major difference between the first and second optimization regimes and such a sharp transition between them, in particular, is due to the presence of zones of [Figure] increased sharpness in the optimization trajectories of neural networks [13, 14, 20, 11, 7].

Original: In the gradient norm vs. training loss diagram in Figure 1, we can seethat the sharpness reaches its peak precisely at the transition point between the first two regimes;this leads us to a hypothesis that only training with sufficiently small ELR values allows gettingthrough this bottleneck and enter the convergence regime.
Revised: In the gradient norm vs. training loss diagram in Figure 1, we can observe that the sharpness reaches its peak precisely at the transition point between the first two regimes; this leads us to a hypothesis that only training with sufficiently small ELR values allows getting through this bottleneck and enter the convergence regime.

Original: In Appendix H, we discuss the relationshipbetween this transition and the epoch-wise double descent of the test error [24]: we discover thatthe sharpness peak correlates with the test error peak and also becomes much more noticeable at thepresence of label noise in the training data.
Revised: In Appendix G, we discuss the relationship between this transition and the epoch-wise double descent of the test error [26]: we discover that the sharpness peak correlates with the test error peak and also becomes much more noticeable at the presence of label noise in the training data.

Original: Finally, the smallest ELRs (orange line) correspond toa typical convergence to the optimum, i.e., the first regime, and the lower the ELR the sharper theminimum we converge to.
Revised: Finally, the smallest ELRs (orange line) correspond to a typical convergence to the optimum, i.e., the first regime, and the lower the ELR the sharper the minimum we converge to.

Original:  Regimes in standard training
Revised:  Regimes in standard training

Original: In this section, we study how different training regimes of SI neural networks on the sphere arerepresented in more practical scenarios.
Revised: In this section, we study how different training regimes of SI neural networks on the sphere are represented in more practical scenarios and what clues they can give to achieve a good minimum in terms of sharpness and generalization.

Original: First, we experiment with training SI networks in a wholeparameter space, where both direction and norm dynamics influence the optimization trajectory.
Revised: First, we experiment with training SI networks in a whole parameter space, where both direction and norm dynamics influence the optimization trajectory.

Original: Wetrain SI networks with weight decay and vary LR instead of ELR.
Revised: We train SI networks with weight decay and vary LR instead of ELR.

Original: All three regimes are also presentin such training (see Figure 7, left).
Revised: All three regimes are also present in such training (see Figure 7, left).

Original: We argue that the difference between the first and the secondregimes explains why disparate results on training dynamics are reported in the literature.
Revised: We argue that the difference between the first and the second regimes explains why disparate results on training dynamics are reported in the literature.

Original: Lobachevaet al.
Revised: Lobacheva et al.

Original: [23] have discovered that SI neural networks experience convergence with periodic jumpsduring optimization.
Revised: [25] have discovered that SI neural networks experience convergence with periodic jumps during optimization.

Original: We observe the same behavior for high LR values in the first regime, wherethe trajectories periodically jump out of the low-loss region (see yellow trajectory for LR 2.0).
Revised: We observe the same behavior for high LR values in the first regime, where the trajectories periodically jump out of the low-loss region (see yellow trajectory for LR 2.0).

Original: Onthe other hand, experiments of Li et al.
Revised: On the other hand, experiments of Li et al.

Original: [22] and Wan et al.
Revised: [24] and Wan et al.

Original: [29] with fixed LRs demonstrate that theequilibrium state is reached, which we also observe in the second training regime.[Figure]
Revised: [32] with fixed LRs demonstrate that the equilibrium state is reached, which we also observe in the second training regime.

Original: Now, let us considerthe conventional neural network training, which implies optimization of allnetwork parameters, including the non-scale-invariant ones, in the entire parameter space withmomentum and data augmentation.
Revised: Now, let us consider conventional neural network training, which implies optimization of all network parameters, including the non-scale-invariant ones, in the entire parameter space with momentum and data augmentation.

Original: Here we use ResNet-18 network of standard size with weightdecay and try both constant and the standard cosine LR schedule (see Figure 7, middle and right).
Revised: Here we use ResNet-18 network of standard size with weight decay and try both constant and the standard cosine LR schedule (see Figure 7, middle and right).

Original: In conventional training, only the first two training regimes are present because LR also affectsnon-scale-invariant parameters, and so training with very high LR values results in NaN values inthe loss.
Revised: In conventional training, only the first two training regimes are present because LR also affects the non-scale-invariant parameters, and so training with very high LR values results in NaN values in the loss, since the weights of the last layer can become large enough to lead to numerical overflows.

Original: The transition between the first two regimes is not as easy to locate, as in the previousexperiments.
Revised: The transition between the first two regimes is not as easy to locate as in the previous experiments.

Original: All the training trajectories end up in regions with relatively high training loss, mainlybecause of the data augmentation, which makes the task much harder to learn and requires more timeto converge to the low-loss region.
Revised: All the training trajectories end up in regions with relatively high training loss, mainly because of the data augmentation, which makes the task much harder to learn and requires more time to converge to the low-loss region.

Original: Moreover, the task may become too hard for some models toachieve a low training loss at all.
Revised: Moreover, the task may become too hard for some models to achieve a low training loss at all.

Original: We show in Appendix G, that similar results are observed in case oftraining SI networks that are too small for the task in hand.
Revised: We show in Appendix F, that similar results are observed in case of training SI networks that are too small for the task in hand.

Original: In the experiments with constant LRs, training with lower LRs  passes the region with the highestsharpness along the trajectory and starts converging to a flatter region (LRs less then 0.01), whiletraining with higher LRs  gets stuck in the high-sharpness zone (LRs higher or equal to 0.01).
Revised: In the experiments with constant LRs, training with lower LRs ( < 0 . 01 ) passes the region with the highest sharpness along the trajectory and starts converging to a flatter region, while training with higher LRs ( ≥ 0 . 01 ) gets stuck in the high-sharpness zone.

Original: Theformer may be assigned to the first training regime, and the latter to the second one.
Revised: The former can be attributed to the first training regime, and the latter to the second one.

Original: The best testaccuracy is achieved for the highest LRs of the first regime, similarly to the results in Section 5.
Revised: Note that, in accordance with Section 5, the best test accuracy is achieved with the largest LR of the first regime.

Original: In the experiments with cosine LR schedule, we decrease LR from some starting LR value to zeroduring 200 epochs of training.
Revised: In the experiments with cosine LR schedule, we decrease LR from some starting LR value to zero during 200 epochs of training.

Original: If we start with  low LR ( ≤ 0 . 02 ), optimization happens with LRs ofthe first regime all or almost all time, and the first regime trajectory is observed.
Revised: If we start with a low LR ( ≤ 0 . 02 ), optimization is carried out all or almost all the time with LRs of the first regime, and the first regime trajectory is observed.

Original: Moreover, for higherstarting LRs, the solution with better sharpness profile and generalization is achieved.
Revised: Moreover, for higher starting LRs, the solution with better sharpness profile and generalization is achieved.

Original: If we start withhigh LR ( ≥ 0 . 5 ), then most of the training is done with LRs of the second regime, and optimizationdoes not have enough time with low LR values to converge to the low-loss region.
Revised: If we start with a high LR ( ≥ 0 . 5 ), then most of the training is done with LRs of the second regime, and optimization does not have enough time with low LR values to converge to the low-loss region.

Original: The mediumstarting LR values are of the most interest here (between 0 .
Revised: The medium starting LR values are of the most interest here (between 0 .

Original: 02 and 0 . 5 ).
Revised: 02 and 0 . 5 ).

Original: In this case, training first hassubstantial time with gradually decreasing LRs of the second regime, which helps it to localize theregion with the flattest optima, and then has just enough time with the first regime LRs to pass thehigh-sharpness zone and start converging to the minimum.
Revised: In this case, training first has substantial time with gradually decreasing LRs of the second regime, which helps it to localize the region with the flattest optima, and then has just enough time with the first regime LRs to pass the high-sharpness zone and converge to the minimum.

Original: With such training, solutions with bettersharpening/generalization profiles are achieved, which is consistent with the results of Section 5.2and the existing literature on pre-training with high LR values [20, 10].
Revised: With such training, solutions with better sharpening/generalization profiles are achieved, which is consistent with the results of Section 5.2 and the existing literature on pre-training with high LR values [22, 10].

Original:  Conclusion 
Revised:  Conclusion and Discussion

Original: In this work, we investigated the properties of training scale-invariant neural networks on the sphereusing projected SGD.
Revised: In this work, we investigated the properties of training scale-invariant neural networks on the sphere using projected SGD.

Original: This method allowed us to study the intrinsic domain of SI models moreaccurately.
Revised: This method allowed us to study the intrinsic domain of SI models more accurately.

Original: We discovered three regimes of such training depending on the effective learning ratevalue: convergence, chaotic equilibrium, and divergence.
Revised: We discovered three regimes of such training depending on the effective learning rate value: convergence, chaotic equilibrium, and divergence.

Revised: They uncover multiple properties of the intrinsic loss landscape of scale-invariant neural networks: some are related to diverse previous results in deep learning, and some appear as novel.

Original: We showed that these regimes arerelated todiverse previous results in deep learning and present in conventional neural network training as well.
Revised: We showed that these regimes are present in conventional neural network training as well and can provide intuition on how to achieve better optima.

Original: Limitations and 
Revised: Possible future directions Speaking about the potential prospects for further research, three directions can be distinguished. First, it would be interesting to develop more solid and general theoretical groundings for the observed phenomena. For instance, a closer look at the negative feedback principle in eq. (5) could potentially shed more light on the ESS/ELR dynamics in the first and second regimes compared to the current decay vs. alignment dilemma and maybe better explain occasional transitions from the second to the first regime for certain ELRs in Section 5.3.

Original: societal impactThe main limitations of our work are applying a single optimization method on the sphere (projected SGD), experimenting only with BN networks as SI models, andconsidering only one of the symmetries that may conceal the actual properties of neural networksintrinsic domain [17]. To our knowledge, our work does not have any negative societal impact.
Revised: Second, the analysis could be extended to other tasks, architectures, and especially optimizers like Adam [15], which can be challenging due to their indirect impact on the effective direction and effective learning rate in SI models [27]. Third, it is very tempting to investigate other potential practical implications of the three regimes for conventional training.

Original: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope? [Yes](b) Did you describe the limitations of your work?
Revised: The analysis of various LR schedules from the perspective of the discovered training regimes is intriguing and might help in designing more efficient and more explainable LR schedules.

Original: [Yes] See the end of Section 7.(c)

Original: Did you discuss any potential negative societal impacts of your work? [Yes] See the end of Section 7. (d) Have you read the ethics review guidelines and ensured that your paper conforms to 
Revised: For example, from our experiments we observe that decreasing (E)LR in the first regime during training does not influence the final solution but only slows down the convergence; at the same time, for the optimal final performance, optimization should run through the low second regime (E)LRs, which allows to achieve basins containing solutions with the best sharpness/generalization properties.

Original: them? [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
Revised: Limitations and societal impact The main limitations of our work are applying a single optimization method on the sphere (projected SGD), experimenting only with BN networks as SI models, and considering only one of the symmetries that may conceal the actual properties of neural networks intrinsic domain [19].

Original: [Yes] See Section 3. (b) Did you include complete proofs of all theoretical results?
Revised: To our knowledge, our work does not have any negative societal impact.

Original: [Yes] See Appendix 3. If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)?
Revised: Acknowledgments and Disclosure of Funding

Original: [Yes] See Supple-

Original: mentary materials. (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Revised: We would like to thank Mikhail Burtsev for the valuable discussions.

Original: [Yes] See Section‘4 and Appendix D.

Original: (c) Did you report error bars (e.g., with respect to the random seed after running experi-

Original: ments multiple times)?

Original: [Yes] We show that the described training regimes are consistent for different random seeds inAppendix F (d) Did you include the total amount of compute and thetype of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
Revised: The publication was supported by the grant for research centers in the field of AI provided by the Analytical Center for the

Original: [Yes] See Appendix D.

Original: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] See Section 4. (b) Did you mention the license of the assets? [Yes] See Appendix D. CIFAR datasets are distributed under MIT license and code is under Apache-2.0 License license (we
Revised: Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE

Original: provide the link to the GitHub in Section 4.)

Original: (c) Did you include any new assets either in the supplemental material or as a URL?

Original: [Yes]We provide our code in Supplementary materials. (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [No] We train networks on the CIFAR datasets which are widely usedin NeurIPS papers.
Revised: University № 70-2021-00139.

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We train networks on the CIFAR datasets which are widely used in NeurIPS papers.
Revised: The empirical results were supported in part through the computational resources of HPC facilities at HSE University [16].

Original: If you used crowdsourcing or conducted research with human subjects.. (a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
Revised: Additional revenues of the authors for the last three years: laboratory sponsorship by Samsung Research, Samsung Electronics and Huawei

Original: (b) Did you describe any potential participant risks, with links to Institutional Review

Original: Board (IRB) approvals, if applicable? [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
Revised: Technologies; honorarium for teaching by MSU; internship at Yandex.

Original: [N/A]