Original: Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes
Revised: Training Scale-Invariant Neural Networks on the Sphere Can Happen in Three Regimes

Original: Abstract
Revised: Abstract

Original: A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant.
Revised: A fundamental property of deep learning normalization techniques, such as batch normalization, is making the pre-normalization parameters scale invariant.

Original: The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously.
Revised: The intrinsic domain of such parameters is the unit sphere, and therefore their gradient optimization dynamics can be represented via spherical optimization with varying effective learning rate (ELR), which was studied previously.

Original: However, the varying
Revised: However, the varying

Original: ELR may obscure certain characteristics of the intrinsic loss landscape structure.
Revised: ELR may obscure certain characteristics of the intrinsic loss landscape structure.

Original: In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR.
Revised: In this work, we investigate the properties of training scale-invariant neural networks directly on the sphere using a fixed ELR.

Original: We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence.
Revised: We discover three regimes of such training depending on the ELR value: convergence, chaotic equilibrium, and divergence.

Original: We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models.
Revised: We study these regimes in detail both on a theoretical examination of a toy example and on a thorough empirical analysis of real scale-invariant deep learning models.

Original: Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training.
Revised: Each regime has unique features and reflects specific properties of the intrinsic loss landscape, some of which have strong parallels with previous research on both regular and scale-invariant neural networks training.

Original: Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.
Revised: Finally, we demonstrate how the discovered regimes are reflected in conventional training of normalized networks and how they can be leveraged to achieve better optima.

Original:  Introduction
Revised:  Introduction

Original: Most modern neural network architectures contain some type of normalization layers, such as Batch
Revised: Most modern neural network architectures contain some type of normalization layers, such as Batch

Original: Normalization (BN) [9] or Layer Normalization [2].
Revised: Normalization (BN) [10] or Layer Normalization [2].

Original: Normalization makes networks partially scaleinvariant (SI), i e., multiplication of their parameters preceding the normalization layers by a positive scalar does not change the model’s output.
Revised: Normalization makes networks partially scaleinvariant (SI), i e., multiplication of their parameters preceding the normalization layers by a positive scalar does not change the model’s output.

Original: In general, the training dynamics of model parameters can be viewed from two perspectives: the direction dynamics , i e., the dynamics of the projection of the parameters onto the unit sphere and the norm dynamics .
Revised: In general, the training dynamics of model parameters can be viewed from two perspectives: the direction dynamics , i e., the dynamics of the projection of the parameters onto the unit sphere and the norm dynamics .

Original: For SI neural networks, the former seems to be more important since the direction alone determines the output of the scale-invariant function.
Revised: For SI neural networks, the former seems to be more important since the direction alone determines the output of the scale-invariant function.

Original: However, the latter influences the optimization by changing the effective learning rate (ELR), i.e., the learning rate (LR) a scale-invariant model would have if it were optimized on the unit sphere.
Revised: However, the latter influences the optimization by changing the effective learning rate (ELR), i.e., the learning rate (LR) a scale-invariant model would have if it were optimized on the unit sphere.

Original: Many works have studied the effect of scale invariance on training dynamics through the prism of norm or, equivalently, ELR dynamics [31, 8, 1, 35, 23, 24, 27, 32, 25].
Revised: Many works have studied the effect of scale invariance on training dynamics through the prism of norm or, equivalently, ELR dynamics [33, 9, 1, 38, 24, 26, 29, 35, 27].

Original: They discovered that during standard training with weight decay the ELR can change in a non-trivial way and lead to different optimization dynamics: periodic behavior [25], destabilization [21, 23], or stabilization on some sphere [24, 32].
Revised: They discovered that during standard training with weight decay the ELR can change in a non-trivial way and lead to different optimization dynamics: periodic behavior [27], destabilization [22, 24], or stabilization on some sphere [26, 35].

Original: However, considering only the ELR dynamics is not enough for a comprehensive study of the specifics of training normalized neural networks.
Revised: However, considering only the ELR dynamics is not enough for a comprehensive study of the specifics of training normalized neural networks.

Original: In particular, the impact of the varying
Revised: In particular, the impact of the varying

Original: ELR on training may obscure certain important properties of SI neural networks’ intrinsic domain, i.e., the unit sphere, thus narrowing our intuition about the loss landscape structure.
Revised: ELR on training may obscure certain important properties of SI neural networks’ intrinsic domain, i.e., the unit sphere, thus narrowing our intuition about the loss landscape structure.

Original: Therefore, in this work, we are focused on the direction dynamics.
Revised: Therefore, in this work, we are focused on the direction dynamics.

Original: To eliminate the influence of the changing norm on training, we fix the total norm of all scale-invariant parameters of the normalized neural network and train it with a constant effective learning rate on the sphere.
Revised: To eliminate the influence of the changing norm on training, we fix the total norm of all scale-invariant parameters of the normalized neural network and train it with a constant effective learning rate on the sphere.

Original: We discover three regimes of such an optimization procedure depending on the ELR value (see
Revised: We discover three regimes of such an optimization procedure depending on the ELR value (see

Original: Figure 1).
Revised: Figure 1).

Original: The first regime (small ELRs) can be considered as a typical convergence to a minimum with monotonically decaying training loss.
Revised: The first regime (small ELRs) can be considered as a typical convergence to a minimum with monotonically decaying training loss.

Original: The second one (medium ELRs) demonstrates a sustained oscillating behavior of the loss around a certain value separated from both the global minimum and the random guess behavior.
Revised: The second one (medium ELRs) demonstrates a sustained oscillating behavior of the loss around a certain value separated from both the global minimum and the random guess behavior.

Original: We call this regime a chaotic equilibrium , reminiscent of the equilibrium state reported in several previous works [24, 32].
Revised: We call this regime a chaotic equilibrium , reminiscent of the equilibrium state reported in several previous works [26, 35].

Original: The last regime (high ELRs) represents destabilized, diverged training associated with an excessively large optimization step size.
Revised: The last regime (high ELRs) represents destabilized, diverged training associated with an excessively large optimization step size.

Original: We analyze the underlying mechanisms behind each of these regimes on a toy example and conduct an extensive empirical study with neural networks trained on the sphere using projected stochastic gradient descent (SGD).
Revised: We analyze the underlying mechanisms behind each of these regimes on a toy example and conduct an extensive empirical study with neural networks trained on the sphere using projected stochastic gradient descent (SGD).

Original: We point out the main features of each regime and study how they relate to each other and what peculiarities of the optimization dynamics and intrinsic structure of the loss landscape they reveal.
Revised: We point out the main features of each regime and study how they relate to each other and what peculiarities of the optimization dynamics and intrinsic structure of the loss landscape they reveal.

Original: Our findings both reconfirmed some of the previous beliefs, like that training with larger (E)LR generally ends up in better optima, or that high-sharpness zones are present in the optimization trajectory, and introduced novel insights, like that different (E)LRs of the first regime even with the same initialization and batch ordering actually lead to distinguishable minima with different sharpness/generalization profiles depending on the specific (E)LR value, or that the high-sharpness zones are responsible for the boundary between the first and the second regimes and have close ties with the Double Descent phenomenon.
Revised: Our findings both reconfirmed some of the previous beliefs, like that training with larger (E)LR generally ends up in better optima, or that high-sharpness zones are present in the optimization trajectory, and introduced novel insights, like that different (E)LRs of the first regime even with the same initialization and batch ordering actually lead to distinguishable minima with different sharpness/generalization profiles depending on the specific (E)LR value, or that the high-sharpness zones are responsible for the boundary between the first and the second regimes and have close ties with the Double Descent phenomenon.

Original: Finally, we demonstrate how our results are reflected in conventional neural network training and suggest several practical implications that help achieve optimal solutions in terms of sharpness and generalization.
Revised: Finally, we demonstrate how our results are reflected in conventional neural network training and suggest several practical implications that help achieve optimal solutions in terms of sharpness and generalization.

Revised: Our code is available at https://github.com/tipt0p/three_regimes_on_the_sphere .

Original:  Background and related work
Revised:  Background and related work

Original: As mentioned in the introduction, for every function F ( θ ) optimized w.r.t.
Revised: As mentioned in the introduction, for every function F ( θ ) optimized w.r.t.

Original: its parameters θ we can separate the dynamics of the parameters direction θ / ∥ θ ∥ from the dynamics of their norm ∥ θ ∥ .
Revised: its parameters θ we can separate the dynamics of the parameters direction θ / ∥ θ ∥ from the dynamics of their norm ∥ θ ∥ .

Original: If the function F ( θ ) is scale-invariant, i.e., F ( c θ )
Revised: If the function F ( θ ) is scale-invariant, i.e., F ( c θ )

Original: = F ( θ ) , ∀ c > 0 , the former is the only factor that determines the value of the function, since F ( θ / ∥ θ ∥ )
Revised: = F ( θ ) , ∀ c > 0 , the former is the only factor that determines the value of the function, since F ( θ / ∥ θ ∥ )

Original: = F ( θ ) .
Revised: = F ( θ ) .

Original: However, even though the SI function’s value does not depend on the parameters norm, the latter greatly influences its gradients: ∇ F ( c θ ) =
Revised: However, even though the SI function’s value does not depend on the parameters norm, the latter greatly influences its gradients: ∇ F ( c θ ) =

Original: ∇ F ( θ ) /c, ∀ c > 0 and thus the optimization speed.
Revised: ∇ F ( θ ) /c, ∀ c > 0 and thus the optimization speed.

Original: This argument can be equivalently reformulated in terms of how the norm dynamics control the direction dynamics’ speed via the notion of effective learning rate.
Revised: This argument can be equivalently reformulated in terms of how the norm dynamics control the direction dynamics’ speed via the notion of effective learning rate.

Original: For the standard (S)GD optimization, ELR is defined as the learning rate divided by the squared parameters norm, and it represents how the learning rate should be adapted to mimic the same direction dynamics if the function were optimized directly on the unit sphere [27].
Revised: For the standard (S)GD optimization, ELR is defined as the learning rate divided by the squared parameters norm, and it represents how the learning rate should be adapted to mimic the same direction dynamics if the function were optimized directly on the unit sphere [29].

Original: Most of the previous work has dealt with the implications of using normalization in neural networks, i.e., making them (partially) scale-invariant, from the perspective of the ELR dynamics.
Revised: Most of the previous work has dealt with the implications of using normalization in neural networks, i.e., making them (partially) scale-invariant, from the perspective of the ELR dynamics.

Original: For example, Arora et al.
Revised: For example, Arora et al.

Original: [1] study the convergence properties of batch normalized neural networks and explain why they allow much higher learning rates compared to their non-BN counterparts.
Revised: [1] study the convergence properties of batch normalized neural networks and explain why they allow much higher learning rates compared to their non-BN counterparts.

Original: Many works have also focused on training scale-invariant neural networks with weight decay [31, 35, 23, 24, 32, 25].
Revised: Many works have also focused on training scale-invariant neural networks with weight decay [33, 38, 24, 26, 35, 27].

Original: They all discovered the non-trivial dynamics of ELR induced by the interplay between normalization and weight decay but came to different conclusions.
Revised: They all discovered the non-trivial dynamics of ELR induced by the interplay between normalization and weight decay but came to different conclusions.

Original: Some studies have reported training instability caused by this interplay [21, 23], while others have argued that, on the contrary, the equilibrium state is eventually reached, i.e., ELR becomes stable after some time [31, 4, 24, 19, 32].
Revised: Some studies have reported training instability caused by this interplay [22, 24], while others have argued that, on the contrary, the equilibrium state is eventually reached, i.e., ELR becomes stable after some time [33, 4, 26, 20, 35].

Original: Recently, Lobacheva et al.
Revised: Recently, Lobacheva et al.

Original: [25] demonstrated that training with weight decay and constant LR may experience regular periodic behavior for a practical range of LRs.
Revised: [27] demonstrated that training with weight decay and constant LR may experience regular periodic behavior for a practical range of LRs.

Original: However, while all of these works mainly study the direction dynamics with varying rate, they lack exploring the properties of optimization of SI models with a fixed effective learning rate.
Revised: However, while all of these works mainly study the direction dynamics with varying rate, they lack exploring the properties of optimization of SI models with a fixed effective learning rate.

Original: Decoupling the direction dynamics from the changing norm effects could reveal particular features of the intrinsic domain of scale-invariant neural networks that were potentially obscured by the dynamical ELR.
Revised: Decoupling the direction dynamics from the changing norm effects could reveal particular features of the intrinsic domain of scale-invariant neural networks that were potentially obscured by the dynamical ELR.

Original: To cope with it, we optimize our models directly on the sphere using projected SGD with a constant learning rate, which is equivalent to fixing the ELR.
Revised: To cope with it, we optimize our models directly on the sphere using projected SGD with a constant learning rate, which is equivalent to fixing the ELR.

Original: Our method reveals some interesting results about SI neural networks and allows to draw parallels with previous scattered deep learning work.
Revised: Our method reveals some interesting results about SI neural networks and allows to draw parallels with previous scattered deep learning work.

Original: We note that similar experiments were conducted by, e.g., Arora et al.
Revised: We note that similar experiments were conducted by, e.g., Arora et al.

Original: [1] and Lobacheva et al.
Revised: [1] and Lobacheva et al.

Original: [25] to highlight certain features of standard training when ELR is dynamical.
Revised: [27] to highlight certain features of standard training when ELR is dynamical.

Original: Also, Cho and Lee [5] optimized scale-invariant parameters in normalized neural networks directly on their intrinsic domain using Riemannian gradient descent, but they proposed an optimization method rather than investigating the properties of such training.
Revised: Also, Cho and Lee [5] optimized scale-invariant parameters in normalized neural networks directly on their intrinsic domain using Riemannian gradient descent, but they proposed an optimization method rather than investigating the properties of such training.

Original:  Theoretical analysis
Revised:  Theoretical analysis

Original: To explain the observed behavior, we proceed with the theoretical analysis of scale-invariant functions optimization with a fixed effective learning rate.
Revised: To explain the observed behavior, we proceed with the theoretical analysis of scale-invariant functions optimization with a fixed effective learning rate.

Original: At first, we derive several important general properties of such a process that will help to clarify some of its empirical aspects.
Revised: At first, we derive several important general properties of such a process that will help to clarify some of its empirical aspects.

Original: Then we consider a concrete example of a function with multiple scale-invariant parameter groups, similar to normalized neural networks, and study its convergence depending on the ELR value to shed more light on the prime causes of the three regimes.
Revised: Then we consider a concrete example of a function with multiple scale-invariant parameter groups, similar to normalized neural networks, and study its convergence depending on the ELR value to shed more light on the prime causes of the three regimes.

Original: We provide all formal derivations and proofs in Appendix A.
Revised: We provide all formal derivations and proofs in Appendix A.

Original: 3.1 General properties
Revised: 3.1 General properties

Original: We begin with an analysis of common features of all SI functions optimization with a fixed learning rate on the sphere.
Revised: We begin with an analysis of common features of all SI functions optimization with a fixed learning rate on the sphere.

Original: For that, consider a function F ( θ ) of a vector parameter θ ∈ R P that can be divided into n groups: θ = ( θ 1 , . .
Revised: For that, consider a function F ( θ ) of a vector parameter θ ∈ R P that can be divided into n groups: θ = ( θ 1 , . .

Original: , θ n ) , where each θ i ∈ R p i and (cid:80) ni
Revised: , θ n ) , where each θ i ∈ R p i and (cid:80) ni

Original: =1 p i = P .
Revised: =1 p i = P .

Original: We say that each of these groups is scale-invariant, i.e., we can multiply it by a positive scalar value, with the others fixed, and the output of the function does not change.
Revised: We say that each of these groups is scale-invariant, i.e., we can multiply it by a positive scalar value, with the others fixed, and the output of the function does not change.

Original: This is typical for neural networks with multiple normalized layers: each of these layers (and even each column/filter in a layer) is scale-invariant.
Revised: This is typical for neural networks with multiple normalized layers: each of these layers (and even each column/filter in a layer) is scale-invariant.

Original: Naturally, if a function is scale-invariant w.r.t.
Revised: Naturally, if a function is scale-invariant w.r.t.

Original: several parameter groups, it is also scale-invariant w.r.t.
Revised: several parameter groups, it is also scale-invariant w.r.t.

Original: their union, therefore the whole vector θ is scale-invariant too.
Revised: their union, therefore the whole vector θ is scale-invariant too.

Original: Since scale-invariant functions are effectively defined on the sphere, we try to minimize F ( θ ) on the sphere of radius ρ :
Revised: Since scale-invariant functions are effectively defined on the sphere, we try to minimize F ( θ ) on the sphere of radius ρ :

Original: [Equation]
Revised: [Equation]

Original: where S P − 1 ( ρ ) = { θ ∈ R P :
Revised: where S P − 1 ( ρ ) = { θ ∈ R P :

Original: ∥ θ ∥ = ρ } .
Revised: ∥ θ ∥ = ρ } .

Original: We do that using the projected gradient descent method with a fixed learning rate η :
Revised: We do that using the projected gradient descent method with a fixed learning rate η :

Original: [Equation]
Revised: [Equation]

Original: Of course, since the function is scale-invariant w.r.t.
Revised: Of course, since the function is scale-invariant w.r.t.

Original: each individual group θ i , we could optimize it on the product of n independent spheres
Revised: each individual group θ i , we could optimize it on the product of n independent spheres

Original: S p
Revised: S p

Original: i − 1 instead of the single mutual sphere, as we do in eq.
Revised: i − 1 instead of the single mutual sphere, as we do in eq.

Original: However, that would either require introducing too many hyperparameters in the model ( n different learning rates for n spheres) or overconstrained the problem (one learning rate for all n spheres).
Revised: However, that would either require introducing too many hyperparameters in the model ( n different learning rates for n spheres) or overconstrained the problem (one learning rate for all n spheres).

Original: We also argue that constraining the total parameters norm is much more natural and closer to the standard neural networks training with weight decay (see discussion in Appendix A.1).
Revised: We also argue that constraining the total parameters norm is much more natural and closer to the standard neural networks training with weight decay (see discussion in Appendix A.1).

Original: We now define the notions of effective gradient and effective learning rate for each SI group θ i .
Revised: We now define the notions of effective gradient and effective learning rate for each SI group θ i .

Original: Effective gradient is the gradient w.r.t.
Revised: Effective gradient is the gradient w.r.t.

Original: θ i measured at the same point but with the group θ i projected on its unit sphere, i.e., at point ˜ θ
Revised: θ i measured at the same point but with the group θ i projected on its unit sphere, i.e., at point ˜ θ

Original: =
Revised: =

Original: ( θ 1 , . .
Revised: ( θ 1 , . .

Original: , θ i /ρ
Revised: , θ i /ρ

Original: i , . .
Revised: i , . .

Original: , θ n ) , where ρ i ≡ ∥ θ i ∥ .
Revised: , θ n ) , where ρ i ≡ ∥ θ i ∥ .

Original: Due to scale invariance w.r.t.
Revised: Due to scale invariance w.r.t.

Original: θ i , the function’s value is equal at both points θ and ˜ θ
Revised: θ i , the function’s value is equal at both points θ and ˜ θ

Original: but, as discussed earlier, the norm of the effective gradient, which we denote as ˜ g i , is ρ i times the norm of the regular gradient w.r.t.
Revised: but, as discussed earlier, the norm of the effective gradient, which we denote as ˜ g i , is ρ i times the norm of the regular gradient w.r.t.

Original: θ i : ˜ g
Revised: θ i : ˜ g

Original: i =
Revised: i =

Original: g
Revised: g

Original: i ρ i , where g i ≡ ∥∇ θ i F ( θ ) ∥ .
Revised: i ρ i , where g i ≡ ∥∇ θ i F ( θ ) ∥ .

Original: Effective learning rate ˜ η
Revised: Effective learning rate ˜ η

Original: i is the learning rate required to make a gradient step w.r.t.
Revised: i is the learning rate required to make a gradient step w.r.t.

Original: θ i equivalent to the original but started from the point ˜ θ .
Revised: θ i equivalent to the original but started from the point ˜ θ .

Original: It is known from the previous literature on scale-invariant functions [27, 32, 25] that ELR for the group θ
Revised: It is known from the previous literature on scale-invariant functions [29, 35, 27] that ELR for the group θ

Original: i equals ˜ η i =
Revised: i equals ˜ η i =

Original: η/ρ 2 i .
Revised: η/ρ 2 i .

Original: We also define the total effective gradient norm ˜
Revised: We also define the total effective gradient norm ˜

Original: g
Revised: g

Original: = gρ , where g ≡ ∥∇ θ F ( θ ) ∥ , and the total ELR ˜ η = η/ρ 2 , which is fixed in the considered setup.
Revised: = gρ , where g ≡ ∥∇ θ F ( θ ) ∥ , and the total ELR ˜ η = η/ρ 2 , which is fixed in the considered setup.

Original: From the constraint on the total norm (cid:80) ni =1 ρ 2 i =
Revised: From the constraint on the total norm (cid:80) ni =1 ρ 2 i =

Original: ρ 2 we obtain the following fundamental equation relating the ELRs of individual SI groups to the total ELR value:
Revised: ρ 2 we obtain the following fundamental equation relating the ELRs of individual SI groups to the total ELR value:

Original: [Equation]
Revised: [Equation]

Original: Another important notion is the effective step size (ESS), which is the product of the effective gradient length by the effective learning rate: ˜
Revised: Another important notion is the effective step size (ESS), which is the product of the effective gradient length by the effective learning rate: ˜

Original: η i ˜ g i .
Revised: η i ˜ g i .

Original: This value defines how far we effectively move the scale-invariant parameters θ i after one gradient step.
Revised: This value defines how far we effectively move the scale-invariant parameters θ i after one gradient step.

Original: It can be shown that the total squared ESS can be expressed as a convex combination of squared ESS values of individual SI groups:
Revised: It can be shown that the total squared ESS can be expressed as a convex combination of squared ESS values of individual SI groups:

Original: [Equation]
Revised: [Equation]

Original: The process (2) and eq.
Revised: The process (2) and eq.

Original: 3 yield the following update rule for each group’s individual ELR:
Revised: 3 yield the following update rule for each group’s individual ELR:

Original: [Equation]
Revised: [Equation]

Original:  from which it follows that for a given SI group the higher (lower) the ESS at the current iteration, the lower (higher) the ELR becomes after it .
Revised: 5) from which it follows that for a given SI group the higher (lower) the ESS at the current iteration, the lower (higher) the ELR becomes after it .

Original: Since, by definition, ESS and ELR values are highly correlated, we conclude that at each iteration the largest ELRs tend to become smaller and vice versa.
Revised: Since, by definition, ESS and ELR values are highly correlated, we conclude that at each iteration the largest ELRs tend to become smaller and vice versa.

Original: This “negative feedback” principle becomes very important in distinguishing between the three regimes of optimization.
Revised: This “negative feedback” principle becomes very important in distinguishing between the three regimes of optimization.

Original: 3.2 Explaining the regimes
Revised: 3.2 Explaining the regimes

Original: To provide a clear and easy-to-follow explanation of the differences between the three regimes of training on the sphere, we construct a simple yet illustrative example of a function with several groups of SI parameters and analyze its optimization properties depending on the total ELR value.
Revised: To provide a clear and easy-to-follow explanation of the differences between the three regimes of training on the sphere, we construct a simple yet illustrative example of a function with several groups of SI parameters and analyze its optimization properties depending on the total ELR value.

Original: The simplest working example of a scale-invariant function is the following one:
Revised: The simplest working example of a scale-invariant function is the following one:

Original: [Equation]
Revised: [Equation]

Original: This function has also been used in previous works to visually demonstrate various properties of general scale-invariant functions [24, 25].
Revised: This function has also been used in previous works to visually demonstrate various properties of general scale-invariant functions [26, 27].

Original: Based on it, we develop an elucidative example of a function that contains more than one group of scale-invariant parameters — just as real multilayer normalized neural networks do.
Revised: Based on it, we develop an elucidative example of a function that contains more than one group of scale-invariant parameters — just as real multilayer normalized neural networks do.

Original: For that, we take a conical combination of n > 1 functions (6) each depending on its own pair of variables:
Revised: For that, we take a conical combination of n > 1 functions (6) each depending on its own pair of variables:

Original: [Equation]
Revised: [Equation]

Original: where x = ( x 1 , . .
Revised: where x = ( x 1 , . .

Original: , x n ) ,
Revised: , x n ) ,

Original: y =
Revised: y =

Original: ( y 1 , . . .
Revised: ( y 1 , . . .

Original: , y n ) , and each α i > 0 is a fixed positive coefficient.
Revised: , y n ) , and each α i > 0 is a fixed positive coefficient.

Original: In this function, there are n groups of SI parameters: ( x i , y i ) , i = 1 , . . . , n .
Revised: In this function, there are n groups of SI parameters: ( x i , y i ) , i = 1 , . . . , n .

Original: In accordance with the setup described previously, we intend to optimize this function on the unit sphere using the projected gradient descent method with a fixed (effective) learning rate, i.e., we plug θ =
Revised: In accordance with the setup described previously, we intend to optimize this function on the unit sphere using the projected gradient descent method with a fixed (effective) learning rate, i.e., we plug θ =

Original: ( x , y ) =
Revised: ( x , y ) =

Original: (( x 1 , y 1 ) , . .
Revised: (( x 1 , y 1 ) , . .

Original: , ( x n , y n )) , P = 2 n , ρ = 1 into equations (1) and (2).
Revised: , ( x n , y n )) , P = 2 n , ρ = 1 into equations (1) and (2).

Original: Now we would like to study the behavior of optimization of F ( x , y ) on the unit sphere depending on the learning rate value η , which equals to the total ELR ˜
Revised: Now we would like to study the behavior of optimization of F ( x , y ) on the unit sphere depending on the learning rate value η , which equals to the total ELR ˜

Original: η due to ρ = 1 .
Revised: η due to ρ = 1 .

Original: But first we need to dwell on the convergence properties of each of its subfunctions α i f
Revised: But first we need to dwell on the convergence properties of each of its subfunctions α i f

Original: ( x i , y i ) .
Revised: ( x i , y i ) .

Original: The next proposition, which we rigorously formulate and prove in Appendix A.3, answers this question.
Revised: The next proposition, which we rigorously formulate and prove in Appendix A.3, answers this question.

Original: Proposition 1 For the SI function f α ( x, y ) =
Revised: Proposition 1 For the SI function f α ( x, y ) =

Original: α x 2 x 2 + y 2 minimized with ELR ˜ η :
Revised: α x 2 x 2 + y 2 minimized with ELR ˜ η :

Original: if ˜ η < 1 α , it linearly converges to zero, i.e., its global minimum; 2.
Revised: if ˜ η < 1 α , it linearly converges to zero, i.e., its global minimum; 2.

Original: if ˜ η > 1 α , it stabilizes at value 12 (cid:16) α − 1˜ η (cid:17) .
Revised: if ˜ η > 1 α , it stabilizes at value 12 (cid:16) α − 1˜ η (cid:17) .

Original: Knowing that each of the subfunctions of F ( x , y ) can converge to the minimum only when its ELR stays below 1 /α i and considering the fundamental constraint on ELRs (3), we state that the first regime (convergence) is only possible when the total ELR is below a certain threshold:
Revised: Knowing that each of the subfunctions of F ( x , y ) can converge to the minimum only when its ELR stays below 1 /α i and considering the fundamental constraint on ELRs (3), we state that the first regime (convergence) is only possible when the total ELR is below a certain threshold:

Original: [Equation]
Revised: [Equation]

Original: What if the total ELR exceeds that threshold?
Revised: What if the total ELR exceeds that threshold?

Original: The first regime cannot be observed: convergence becomes impossible, since at least one of the individual ELRs exceeds its convergence threshold value, according to eq.
Revised: The first regime cannot be observed: convergence becomes impossible, since at least one of the individual ELRs exceeds its convergence threshold value, according to eq.

Original: By Proposition 1 the value of the corresponding subfunction is separated from zero by some quantity depending on its ELR.
Revised: By Proposition 1 the value of the corresponding subfunction is separated from zero by some quantity depending on its ELR.

Original: This means that the total function value, and therefore the total effective gradient, is also nonvanishing (we give a more formal argument in Appendix A.4).
Revised: This means that the total function value, and therefore the total effective gradient, is also nonvanishing (we give a more formal argument in Appendix A.4).

Original: This brings us to the “undamped” dynamics of ELRs (5), where the effective step sizes do not tend to zero.
Revised: This brings us to the “undamped” dynamics of ELRs (5), where the effective step sizes do not tend to zero.

Original: In that case, the dynamics stabilize only when all the ESS values become equal, which is possible if and only if each individual ELR ˜
Revised: In that case, the dynamics stabilize only when all the ESS values become equal, which is possible if and only if each individual ELR ˜

Original: η i takes its corresponding equilibrium value:
Revised: η i takes its corresponding equilibrium value:

Original: [Equation]
Revised: [Equation]

Original: Due to the negative feedback principle of eq.
Revised: Due to the negative feedback principle of eq.

Original: (5), the system will naturally tend towards this state when
Revised: (5), the system will naturally tend towards this state when

Original: ESS values are non-negligible.
Revised: ESS values are non-negligible.

Original: Note that, in contrast, in the first regime, ESS values decay too rapidly due to convergence, and the model may fail to reach equilibrium.
Revised: Note that, in contrast, in the first regime, ESS values decay too rapidly due to convergence, and the model may fail to reach equilibrium.

Original: In sum, when the total ELR exceeds the convergence threshold and each individual ELR (in average) equals its equilibrium value corresponding to the equalized ESS, the optimization process enters the second regime, i.e., the chaotic equilibrium .
Revised: In sum, when the total ELR exceeds the convergence threshold and each individual ELR (in average) equals its equilibrium value corresponding to the equalized ESS, the optimization process enters the second regime, i.e., the chaotic equilibrium .

Original: Finally, if the total ELR is set too high, the chaos will dominate equilibration and the optimization dynamics will resemble a random walk.
Revised: Finally, if the total ELR is set too high, the chaos will dominate equilibration and the optimization dynamics will resemble a random walk.

Original: In that case, we observe the third regime, or divergence
Revised: In that case, we observe the third regime, or divergence

Original: In Figure 2, we plot the evolution of the function (7) during optimization on the unit sphere with three different effective learning rates.
Revised: In Figure 2, we plot the evolution of the function (7) during optimization on the unit sphere with three different effective learning rates.

Original: We set n = 3 , α i =
Revised: We set n = 3 , α i =

Original: i , i = 0 , 1 , 2 , and start from a point randomly selected on the 6-dimensional unit sphere.
Revised: i , i = 0 , 1 , 2 , and start from a point randomly selected on the 6-dimensional unit sphere.

Original: In this case, the threshold condition (8) becomes ˜ η < 1 / 7 ≈ 0 .
Revised: In this case, the threshold condition (8) becomes ˜ η < 1 / 7 ≈ 0 .

Original: 143 , and thus we select three ELRs corresponding to the three optimization regimes, respectively: 0 . 1 , 0 . 2 , and 0 .
Revised: 143 , and thus we select three ELRs corresponding to the three optimization regimes, respectively: 0 . 1 , 0 . 2 , and 0 .

Original: The lines on the plot directly demonstrate the anticipated behavior: the smallest ELR leads to a fast decay of the function’s value, the medium one makes the function stabilize around a certain level, and the largest ELR depicts the most chaotic behavior.
Revised: The lines on the plot directly demonstrate the anticipated behavior: the smallest ELR leads to a fast decay of the function’s value, the medium one makes the function stabilize around a certain level, and the largest ELR depicts the most chaotic behavior.

Original: In Appendix A.5, wealso provide plots of the individual ELRs dynamics that show
Revised: In Appendix A.5, we provide plots of the individual ELRs dynamics that show

Original: that in the chaotic equilibrium ˜ η i in average by iterations closely match the predicted values (9), while in the first regime they are not reached due to quick convergence, and in the thirdone the optimization is too unstable to distinguish any equilibration.
Revised: that in the chaotic equilibrium ˜ η i in average by iterations closely match the predicted values (9), while in the first regime they are not reached due to quick convergence, and in the third the optimization is too unstable to distinguish any equilibration.

Original:  Experimental setup
Revised:  Experimental setup

Original: After discussing the theoretical substantiation of the different regimes of learning on the sphere in the case of a simple SI function, let us move on to an empirical analysis of the learning dynamics of neural networks equipped with Batch Normalization.
Revised: After discussing the theoretical substantiation of the different regimes of learning on the sphere in the case of a simple SI function, let us move on to an empirical analysis of the learning dynamics of neural networks equipped with Batch Normalization.

Original: We conduct experiments with a simple 3-layer BN convolutional neural network (ConvNet) and a ResNet-18 on CIFAR-10[17] and CIFAR-100 [18]  datasets.
Revised: We conduct experiments with a simple 3-layer BN convolutional neural network (ConvNet) and a ResNet-18 on CIFAR-10 [18] and CIFAR100 [19] datasets.

Original: Both networks are in the implementation of Lobacheva et al.
Revised: Both networks are in the implementation of Lobacheva et al.

Original: [25] and we train them by optimizing cross-entropy loss using stochastic gradient descent with batch size 128 for 1000 epochs.
Revised: and we train them by optimizing cross-entropy loss using stochastic gradient descent with batch size 128 forepochs.

Revised: In Appendix O, we also show that three training regimes may be observed for other neural architectures, such as multilayer perceptron and Transformer [34].

Original: In Section 5, we analyze the training regimes of fully scale-invariant neural networks, trained on the sphere with a fixed ELR.
Revised: In Section 5, we analyze the training regimes of fully scale-invariant neural networks, trained on the sphere with a fixed ELR.

Original: To obtain fully scale-invariant versions of the networks we follow the same approach as in Lobacheva et al.
Revised: To obtain fully scale-invariant versions of the networks we follow the same approach as in Lobacheva et al.

Original: We fix the non-scale-invariant weights, i.e., we use zero shift and unit scale in BN layers and freeze the weights of the last layer at random initialization with increased norm equal to 10.
Revised: We fix the non-scale-invariant weights, i.e., we use zero shift and unit scale in BN layers and freeze the weights of the last layer at random initialization with increased norm equal to 10.

Original: To train a network with ELR ˜ η , we fix the norm of the weights at the initial value ρ and use the projected SGD with a fixed learning rate η = ˜ ηρ 2 .
Revised: To train a network with ELR ˜ η , we fix the norm of the weights at the initial value ρ and use the projected SGD with a fixed learning rate η = ˜ ηρ 2 .

Original: We consider a wide range of ELRs: { 10 − k , 2 · 10 − k , 5 · 10 − k }  k =0 ,7 , which more than covers the values of ELR usually encountered during training of regular networks.
Revised: We consider a wide range of ELRs: { 10 − k , 2 · 10 − k , 5 · 10 − k } 7 k =0 , which more than covers the values of ELR usually encountered during training of regular networks.

Original: We also used a more fine grained ELR grid closer to the boundaries between the regimes.
Revised: We also used a more fine grained ELR grid closer to the boundaries between the regimes.

Original: In Section 6, we investigate how the observed training regimes transfer to more practical scenarios.
Revised: In Section 6, we investigate how the observed training regimes transfer to more practical scenarios.

Original: We train networks in the whole parameter space with a fixed learning rate and weight decay as in Lobacheva et al.
Revised: We train networks in the whole parameter space with a fixed learning rate and weight decay as in Lobacheva et al.

Original: [25], Wan et al.
Revised: [27], Wan et al.

Original: We use weight decay of 1 e − 4 / 5 e − 4 for ConvNet / ResNet-18 in all the experiments and a range of LRs, specific for each scenario.
Revised: We use weight decay of 1e-4 / 5e-4 for ConvNet / ResNet-18 in all the experiments and a range of LRs, specific for each scenario.

Original: In some experiments, we also use momentum of 0.9, standard CIFAR data augmentation, and cosine LR schedule for 200 epochs.
Revised: In some experiments, we also use momentum of 0.9, standard CIFAR data augmentation, and cosine LR schedule for 200 epochs.

Original: More details can be found in Appendix C.
Revised: More details can be found in Appendix C.

Original: To make the experiments with different ELRs/LRs fully comparable, we train the networks from the same random initialization and with the same optimizer random seed.
Revised: To make the experiments with different ELRs/LRs fully comparable, we train the networks from the same random initialization and with the same optimizer random seed.

Original: However, we show in
Revised: However, we show in

Original: Appendix E that the results are consistent if we vary both of them.
Revised: Appendix E that the results are consistent if we vary both of them.

Original: For each visualization we choose the most representative subset of ELRs/LRs due to the difficulties of distinguishing between many noisy lines on the plots.
Revised: For each visualization we choose the most representative subset of ELRs/LRs due to the difficulties of distinguishing between many noisy lines on the plots.

Original: At each training epoch, we log standard train / test statistics and the mean (over mini-batches) norm of the stochastic effective gradients for fully scale-invariant networks or the mean norm of the regular stochastic gradients in the case of standard networks.
Revised: At each training epoch, we log standard train / test statistics and the mean (over mini-batches) norm of the stochastic effective gradients for fully scale-invariant networks or the mean norm of the regular stochastic gradients in the case of standard networks.

Original: We choose this metric for the following two reasons.
Revised: We choose this metric for the following two reasons.

Original: First, it can serve as a measure of sharpness of the minimum to which we converge in the first regime, since the mean stochastic gradient norm is strongly correlated with the trace of the gradient covariance matrix, which in turn is closely tied with the loss Hessian and the Fisher Information Matrix [12, 34, 30] (see Appendix B).
Revised: First, it can serve as a measure of sharpness of the minimum to which we converge in the first regime, since the mean stochastic gradient norm is strongly correlated with the trace of the gradient covariance matrix, which in turn is closely tied with the loss Hessian and the Fisher Information Matrix [13, 37, 32] (see Appendix B).

Original: Second, it represents the average (effective) step size up to multiplication by (E)LR and thus is also related to optimization dynamics.
Revised: Second, it represents the average (effective) step size up to multiplication by (E)LR and thus is also related to optimization dynamics.

Original:  Three regimes of training on the sphere
Revised:  Three regimes of training on the sphere

Original: In this section, we investigate the three training regimes of fully scale-invariant neural networks, trained on the sphere with a fixed ELR.
Revised: In this section, we investigate the three training regimes of fully scale-invariant neural networks, trained on the sphere with a fixed ELR.

Original: We first analyze the learning dynamics in each regime, including the properties of the resulting solutions and the loss landscape around them.
Revised: We first analyze the learning dynamics in each regime, including the properties of the resulting solutions and the loss landscape around them.

Original: After that, we discuss transitions between the regimes in more detail.
Revised: After that, we discuss transitions between the regimes in more detail.

Original: We show the results for ConvNet on
Revised: We show the results for ConvNet on

Original: CIFAR-10, the results for other dataset-architecture pairs are consistent and presented in Appendix D.
Revised: CIFAR-10, the results for other dataset-architecture pairs are consistent and presented in Appendix D.

Original: 5.1 Regime one: convergence
Revised: 5.1 Regime one: convergence

Original: Training with the smallest ELRs results in the first training regime, which we call the convergence regime .
Revised: Training with the smallest ELRs results in the first training regime, which we call the convergence regime .

Original: As shown in Figure 1, optimization in this regime experiences a typical convergence behavior: after a number of epochs the model is able to reach regions with very low training loss and continues converging to the minimum.
Revised: As shown in Figure 1, optimization in this regime experiences a typical convergence behavior: after a number of epochs the model is able to reach regions with very low training loss and continues converging to the minimum.

Original: The speed of convergence depends on the ELR value — the higher the ELR, the faster the optimization, and the lower training loss is achieved after a fixed number of epochs.
Revised: The speed of convergence depends on the ELR value — the higher the ELR, the faster the optimization, and the lower training loss is achieved after a fixed number of epochs.

Original: Also, training with different ELRs results in solutions with different sharpness (mean norm of stochastic effective gradients) and generalization (test error): higher
Revised: Also, training with different ELRs results in solutions with different sharpness (mean norm of stochastic effective gradients) and generalization (test error): higher

Original: ELRs lead to less sharp solution with better generalization.
Revised: ELRs lead to less sharp solution with better generalization.

Original: Similar results are well known in the literature [12, 33, 22, 28, 14, 20, 6, 3, 29, 11, 7], i.e., training neural networks using larger learning rates typically results in flatter and better generalizing optima.
Revised: Similar results are well known in the literature [13, 36, 23, 30, 15, 21, 6, 3, 31, 12, 8], i.e., training neural networks using larger learning rates typically results in flatter and better generalizing optima.

Original: We discovered that the optima achieved after training with different ELRs not only vary in sharpness and generalization but also reside in different basins, i.e., there is a significant loss barrier on the linear interpolation between them (see Appendix I for more details).
Revised: Furthermore, we discovered that the optima achieved after training with different ELRs not only vary in sharpness and generalization but also reside in different basins, i.e., there is a significant loss barrier on the linear interpolation between them (see Appendix I for more details).

Original: The value of ELR affects not only the rate of convergence and the properties of the final solution; it may have an effect on the whole training trajectory as well.
Revised: The value of ELR affects not only the rate of convergence and the properties of the final solution; it may have an effect on the whole training trajectory as well.

Original: To analyze the training trajectories independently from the training speed, we look at the evolution of sharpness and generalization vs. training loss during the optimization.
Revised: To analyze the training trajectories independently from the training speed, we look at the evolution of sharpness and generalization vs. training loss during the optimization.

Original: We show the corresponding plots for different ELR values of the first regime in Figure 3, left.
Revised: We show the corresponding plots for different ELR values of the first regime in Figure 3, left.

Original: For the lowest ELRs the trajectories coincide, but the training is too slow to converge to the low-loss region.
Revised: For the lowest ELRs the trajectories coincide, but the training is too slow to converge to the low-loss region.

Original: For the remaining values, as we increase ELR, we obtain a lower trajectory, and in the low-loss region trajectories of different ELRs appear as parallel lines.
Revised: For the remaining values, as we increase ELR, we obtain a lower trajectory, and in the low-loss region trajectories of different ELRs appear as parallel lines.

Original: [Figure]
Revised: [Figure]

Original: The parallelism of the trajectories in the low-loss region is particularly interesting and leads us to the following question: do the basins to which training with different ELRs converges have different overall sharpness and generalization profiles?
Revised: The parallelism of the trajectories in the low-loss region is particularly interesting and leads us to the following question: do the basins to which training with different ELRs converges have different overall sharpness and generalization profiles?

Original: That is, can we hypothesize that all solutions reachable with SGD in a given basin have similar sharpness and generalization at the same training loss?
Revised: That is, can we hypothesize that all solutions reachable with SGD in a given basin have similar sharpness and generalization at the same training loss?

Original: To answer that, we take a solution achieved with a specific ELR and fine-tune it with lower and higher
Revised: To answer that, we take a solution achieved with a specific ELR and fine-tune it with lower and higher

Original: ELR values (see Figure 3, right).
Revised: ELR values (see Figure 3, right).

Original: Fine-tuning with a lower ELR stays in the same basin and continues to move along the same trajectory as the initial experiment, hence no sharper sub-minima can be found inside the flat basin.
Revised: Fine-tuning with a lower ELR stays in the same basin and continues to move along the same trajectory as the pre-trained model, hence no sharper sub-minima can be found inside the flat basin.

Original: Fine-tuning with a higher ELR also maintains the same trajectory at first but jumps out of the low-loss region after a while and converges to a new basin with a profile attributed to that  ELR.
Revised: Fine-tuning with a higher ELR also maintains the same trajectory at first but jumps out of the low-loss region after a while and converges to a new basin with a profile attributed to that higher ELR.

Original: That means that the initial basin is too sharp for the higher ELR to converge.
Revised: That means that the initial basin is too sharp for the higher ELR to converge.

Original: The results are consistent for different ELR values, see Appendix H; we also provide results on linear connectivity of pre-trained and fine-tuned solutions in Appendix I.
Revised: The results are consistent for different ELR values, see Appendix H; we also provide results on linear connectivity of pre-trained and fine-tuned solutions in Appendix I.

Original: As can be seen from Figure 3, left, the range of sharpness and generalization profiles reachable with the fixed ELR training in the first regime is limited.
Revised: As can be seen from Figure 3, left, the range of sharpness and generalization profiles reachable with the fixed ELR training in the first regime is limited.

Original: For small ELRs, the trajectories coincide and optimization converges to the sharpest available basin.
Revised: For small ELRs, the trajectories coincide and optimization converges to the sharpest available basin.

Original: Increasing ELR results in flatter optima, but at some value the optimization stops converging to low-loss regions and demonstrates the behavior of the second training regime, which we describe next.
Revised: Increasing ELR results in flatter optima, but at some value the optimization stops converging to low-loss regions and demonstrates the behavior of the second training regime, which we describe next.

Original: 5.2 Regimes two and three: chaotic equilibrium and divergence
Revised: 5.2 Regimes two and three: chaotic equilibrium and divergence

Original: A further increase of ELR in the first regime leads to a sharp switch to another regime, the chaotic equilibrium .
Revised: A further increase of ELR in the first regime leads to a sharp switch to another regime, the chaotic equilibrium .

Original: Figure 1 shows that after a certain threshold ELR, the optimization is no longer able to reach low-loss regions and gets stuck at some loss level.
Revised: Figure 1 shows that after a certain threshold ELR, the optimization is no longer able to reach low-loss regions and gets stuck at some loss level.

Original: We observe that the effective gradients norm also stabilizes, and higher ELRs correspond to a higher loss but lower sharpness.
Revised: We observe that the effective gradients norm also stabilizes, and higher ELRs correspond to a higher loss but lower sharpness.

Original: As we determined in Section 3.2, the second regime corresponds to the state when all individual effective step sizes become equal, which also leads to the stabilization of individual ELRs.
Revised: As we determined in Section 3.2, the second regime corresponds to the state when all individual effective step sizes become equal, which also leads to the stabilization of individual ELRs.

Original: In Figurewe depict the values (averaged over last 200 epochs) of individual
Revised: In Figurewe depict the values (averaged over last 200 epochs) of individual

Original: ELRs and effective gradient norms w.r.t.
Revised: ELRs and effective gradient norms w.r.t.

Original: each SI parameters group of each layer of the neural network trained with ˜ η
Revised: each SI parameters group of each layer of the neural network trained with ˜ η

Original: = 10 − 3 .
Revised: = 10 − 3 .

Original: We see that the effective step sizes (ELR times effective gradient norm) indeed concentrate around a certain equilibrium value denoted as a dashed black line on the plot.
Revised: We see that the effective step sizes (ELR times effective gradient norm) indeed concentrate around a certain equilibrium value denoted as a dashed black line on the plot.

Original: In Appendix J, we provide results for other ELRs and additional plots demonstrating the stabilization of individual ELRs and effective gradient norms, which corroborates our definition of the second regime as the chaotic equilibrium.
Revised: In Appendix J, we provide results for other ELRs and additional plots demonstrating the stabilization of individual ELRs and effective gradient norms, which corroborates our definition of the second regime as the chaotic equilibrium.

Original: In the second regime, the optimization localizes a certain region in the loss landscape and “hops along its walls”.
Revised: In the second regime, the optimization localizes a certain region in the loss landscape and “hops along its walls”.

Original: For moderate ELRs, this region is locally convex in the sense that the loss on the linear path
Revised: For moderate ELRs, this region is locally convex in the sense that the loss on the linear path

Original: [Figure]
Revised: [Figure]

Original: between different checkpoints from the same trajectory is convex and generally much lower than at its edges.
Revised: between different checkpoints from the same trajectory is convex and generally much lower than at its edges.

Original: However, for the highest ELRs, the dynamics become more chaotic and distinguished barriers may appear on the linear path.
Revised: However, for the highest ELRs, the dynamics become more chaotic and distinguished barriers may appear on the linear path.

Original: We provide the corresponding plots in Appendix K. Each ELR of the second regime determines its own loss level — the height at which the optimization hops.
Revised: We provide the corresponding plots in Appendix K. Each ELR of the second regime determines its own loss level — the height at which the optimization “hops”.

Original: Changing the ELR value during training appropriately changes the training dynamics: increasing/decreasing the ELR directly leads to the trajectory with a loss level corresponding to the new ELR value (see
Revised: Changing the ELR value during training appropriately changes the training dynamics: increasing/decreasing the ELR directly leads to the trajectory with a loss level corresponding to the new ELR value (see

Original: Figure 5, left).
Revised: Figure 5, left).

Original: This significantly distinguishes the second regime from the first, where fine-tuning with a different ELR does not affect the learning trajectory, unless the new ELR is too large, in which case the optimization suddenly jumps out of the basin.
Revised: This significantly distinguishes the second regime from the first, where fine-tuning with a different ELR does not affect the learning trajectory, unless the new ELR is too large, in which case the optimization suddenly jumps out of the basin.

Original: Now, what if we fine-tune a model from the second regime with a smaller ELR corresponding to the first regime?
Revised: Now, what if we fine-tune a model from the second regime with a smaller ELR corresponding to the first regime?

Original: As can be seen from Figure 5, right, the resulting optima highly depend on the starting
Revised: As can be seen from Figure 5, right, the resulting optima highly depend on the starting

Original: ELR (more results in Appendix L).
Revised: ELR (more results in Appendix L).

Original: Fine-tuning the models, pre-trained with low second regime ELRs, using any first regime ELR converges to basins with thesame sharpness/generalization profilethat is attributed to the maximum ELR in the first regime.
Revised: Fine-tuning the models, pre-trained with low second regime ELRs, using any first regime ELR converges to basins with the sharpness/generalization profile attributed to the maximum ELR in the first regime.

Original: For larger second regime ELRs,however, fine-tuning results in a variety of trajectories depending on the chosen ELR of the first regime.
Revised: For larger second regime ELRs, fine-tuning results in a variety of trajectories depending on the chosen ELR of the first regime.

Original: That means that training with low second regime ELR takes place in regions containing only flat minima.
Revised: That means that training with low second regime ELRs discovers regions containing only flat minima.

Original: Meanwhile, regions discovered with high ELRs contain optima of different sharpness.
Revised: Meanwhile, regions discovered with high ELRs contain optima of different sharpness.

Original: Moreover, fine-tuning from these regions results in the same range of sharpness/generalization profiles that was observed in the first regime.
Revised: Moreover, fine-tuning from these regions results in the same range of sharpness/generalization profiles that was observed in the first regime.

Original: For the highest considered ELR values, we observe the most unstable optimization behavior corresponding to the random guess accuracy (see Figure 1).
Revised: For the highest considered ELR values, we observe the most unstable optimization behavior corresponding to the random guess accuracy (see Figure 1).

Original: We call it the third, divergent training regime .
Revised: We call it the third, divergent training regime .

Original: We explicitly compare the third regime with random walk in Appendix M.
Revised: We explicitly compare the third regime with random walk in Appendix M.

Original: 5.3 Transitions between the regimes
Revised: 5.3 Transitions between the regimes

Original: In the end, we would like to discuss the connections between the three regimes and our global view of training on the sphere.
Revised: In the end, we would like to discuss the connections between the three regimes and our global view of training on the sphere.

Original: In
Revised: In

Original: Figure 6, we plot the training loss for several ELR values that correspond to the three regimes and transitions between them.
Revised: Figure 6, we plot the training loss for several ELR values that correspond to the three regimes and transitions between them.

Original: When the ELR is too high (purple line), the total ESS becomes so large that the model is unable to even detect any region in the parameters space with non-trivial quality, and we encounter the third regime.
Revised: When the ELR is too high (purple line), the total ESS becomes so large that the model is unable to even detect any region in the parameters space with non-trivial quality, and we encounter the third regime.

Original: With a smaller ELR (light blue line), the model starts occasionally hitting the edge of the region with a (relatively) low loss but then quickly escapes it and returns back to the random guess behavior; this could be attributed to the border regime between the second and the third regimes.
Revised: With a smaller ELR (light blue line), the model starts occasionally hitting the edge of the region with a (relatively) low loss but then quickly escapes it and returns back to the random guess behavior; this could be attributed to the border regime between the second and the third regimes.

Original: Setting the ELR to an even smaller value (green line) results in the chaotic equilibrium, when the loss is stabilized near a certain value, which is lower the lower the ELR.
Revised: Setting the ELR to an even smaller value (green line) results in the chaotic equilibrium, when the loss is stabilized near a certain value, which is lower the lower the ELR.

Original: For ELR values between the highest ELRs of the first regime and the lowest ones of the second regime (yellow line), we can witness a sudden change from equilibrium behavior to convergence.
Revised: For ELR values between the highest ELRs of the first regime and the lowest ones of the second regime (yellow line), we can witness a sudden change from equilibrium behavior to convergence.

Original: We conjecture that the major difference between the first and second optimization regimes and such a sharp transition between them, in particular, is due to the presence of zones of[Figure] increased sharpness in the optimization trajectories of neural networks [13, 14, 20, 11, 7].
Revised: We conjecture that the major difference between the first and second optimization regimes and such a sharp transition between them, in particular, is due to the presence of zones of increased sharpness in the optimization trajectories of neural networks [14, 15, 21, 12, 8].

Original: In the gradient norm vs. training loss diagram in Figure 1, we can observe that the sharpness reaches its peak precisely at the transition point between the first two regimes; this leads us to a hypothesis that only training with sufficiently small ELR values allows getting through this bottleneck and enter the convergence regime.
Revised: In the gradient norm vs. training loss diagram in Figure 1, we can observe that the sharpness reaches its peak precisely at the transition point between the first two regimes; this leads us to a hypothesis that only training with sufficiently small ELR values allows getting through this bottleneck and enter the convergence regime.

Original: In Appendix G, we discuss the relationship between this transition and the epoch-wise double descent of the test error [26]: we discover that the sharpness peak correlates with the test error peak and also becomes much more noticeable at the presence of label noise in the training data.
Revised: In Appendix G, we discuss the relationship between this transition and the epoch-wise double descent of the test error [28]: we discover that the sharpness peak correlates with the test error peak and also becomes much more noticeable at the presence of label noise in the training data.

Original: Finally, the smallest ELRs (orange line) correspond to a typical convergence to the optimum, i.e., the first regime, and the lower the ELR the sharper the minimum we converge to.
Revised: Finally, the smallest ELRs (orange line) correspond to a typical convergence to the optimum, i.e., the first regime, and the lower the ELR the sharper the minimum we converge to.

Original:  Regimes in standard training
Revised:  Regimes in standard training

Original: In this section, we study how different training regimes of SI neural networks on the sphere are represented in more practical scenarios and what clues they can give to achieve a good minimum in terms of sharpness and generalization.
Revised: In this section, we study how different training regimes of SI neural networks on the sphere are represented in more practical scenarios and what clues they can give to achieve a good minimum in terms of sharpness and generalization.

Original: First, we experiment with training SI networks in a whole parameter space, where both direction and norm dynamics influence the optimization trajectory.
Revised: First, we experiment with training SI networks in a whole parameter space, where both direction and norm dynamics influence the optimization trajectory.

Original: We train SI networks with weight decay and vary LR instead of ELR.
Revised: We train SI networks with weight decay and vary LR instead of ELR.

Original: All three regimes are also present in such training (see Figure 7, left).
Revised: All three regimes are also present in such training (see Figure 7, left).

Original: We argue that the difference between the first and the second regimes explains why disparate results on training dynamics are reported in the literature.
Revised: We argue that the difference between the first and the second regimes explains why disparate results on training dynamics are reported in the literature.

Original: Lobacheva et al.
Revised: Lobacheva et al.

Original: [25] have discovered that SI neural networks experience convergence with periodic jumps during optimization.
Revised: [27] have discovered that SI neural networks experience convergence with periodic jumps during optimization.

Original: We observe the same behavior for high LR values in the first regime, where the trajectories periodically jump out of the low-loss region (see yellow trajectory for LR 2.0).
Revised: We observe the same behavior for high LR values in the first regime, where the trajectories periodically jump out of the low-loss region (the yellow trajectory for LR = 2 . 0 ).

Original: On the other hand, experiments of Li et al.
Revised: On the other hand, experiments of Li et al.

Original: [24] and Wan et al.
Revised: [26] and Wan et al.

Original: [32] with fixed LRs demonstrate that the equilibrium state is reached, which we also observe in the second training regime.
Revised: [35] with fixed LRs demonstrate that the equilibrium state is reached, which we can observe in the second training regime.

Original: Now, let us consider conventional neural network training, which implies optimization of all network parameters, including the non-scale-invariant ones, in the entire parameter space with momentum and data augmentation.
Revised: Now, let us consider conventional neural networks training, which implies optimization of all network’s weights (including the non-scale-invariant ones) in the entire parameter space with momentum and data augmentation.

Original: Here we use ResNet-18 network of standard size with weight decay and try both constant and the standard cosine LR schedule (see Figure 7, middle and right).
Revised: Here we use ResNet-18 network of standard size with weight decay and try both constant and the standard cosine LR schedule (see Figure 7, middle and right).

Original: In conventional training, only the first two training regimes are present because LR also affects the non-scale-invariant parameters, and so training with very high LR values results in NaN values in the loss, since the weights of the last layer can become large enough to lead to numerical overflows.
Revised: In conventional training, only the first two training regimes are present because LR also affects the non-scale-invariant parameters, and so training with very high LR values results in NaN values in the loss, since the weights of the last layer can become large enough to lead to numerical overflows.

Original: The transition between the first two regimes is not as easy to locate as in the previous experiments.
Revised: The transition between the first two regimes is not as easy to locate as in the previous experiments.

Original: All the training trajectories end up in regions with relatively high training loss, mainly because of the data augmentation, which makes the task much harder to learn and requires more time to converge to the low-loss region.
Revised: All the training trajectories end up in regions with relatively high training loss, mainly because of the data augmentation, which makes the task much harder to learn and requires more time to converge to the low-loss region.

Original: Moreover, the task may become too hard for some models to achieve a low training loss at all.
Revised: Moreover, the task may become too hard for some models to achieve a low training loss at all.

Original: We show in Appendix F, that similar results are observed in case of training SI networks that are too small for the task in hand.
Revised: We show in Appendix F, that similar results are observed in case of training SI networks that are too small for the task in hand.

Original: In the experiments with constant LRs, training with lower LRs ( < 0 . 01 ) passes the region with the highest sharpness along the trajectory and starts converging to a flatter region, while training with higher LRs ( ≥ 0 .
Revised: In the experiments with constant LRs, training with lower LRs ( < 0 . 01 ) passes the region with the highest sharpness along the trajectory and starts converging to a flatter region, while training with higher LRs ( ≥ 0 .

Original: 01 ) gets stuck in the high-sharpness zone.
Revised: 01 ) gets stuck in the high-sharpness zone.

Original: The former can be attributed to the first training regime, and the latter to the second one.
Revised: The former can be attributed to the first training regime, and the latter to the second one.

Original: Note that, in accordance with Section 5, the best test accuracy is achieved with the largest LR of the first regime.
Revised: Note that, in accordance with Section 5, the best test accuracy is achieved with the largest LR of the first regime.

Original: In the experiments with cosine LR schedule, we decrease LR from some starting LR value to zero during 200 epochs of training.
Revised: In the experiments with cosine LR schedule, we decrease LR from some starting LR value to zero during 200 epochs of training.

Original: If we start with a low LR ( ≤ 0 . 02 ), optimization is carried out all or almost all the time with LRs of the first regime, and the first regime trajectory is observed.
Revised: If we start with a low LR ( ≤ 0 . 02 ), optimization is carried out all or almost all the time with LRs of the first regime, and the first regime trajectory is observed.

Original: Moreover, for higher starting LRs, the solution with better sharpness profile and generalization is achieved.
Revised: Moreover, for higher starting LRs, the solution with better sharpness profile and generalization is achieved.

Original: If we start with a high LR ( ≥ 0 . 5 ), then most of the training is done with LRs of the second regime, and optimization does not have enough time with low LR values to converge to the low-loss region.
Revised: If we start with a high LR ( ≥ 0 . 5 ), then most of the training is done with LRs of the second regime, and optimization does not have enough time with low LR values to converge to the low-loss region.

Original: The medium starting LR values are of the most interest here (between 0 .
Revised: The medium starting LR values are of the most interest here (between 0 .

Original: 02 and 0 . 5 ).
Revised: 02 and 0 . 5 ).

Original: In this case, training first has substantial time with gradually decreasing LRs of the second regime, which helps it to localize the region with the flattest optima, and then has just enough time with the first regime
Revised: In this case, training first has substantial time with gradually decreasing LRs of the second regime, which helps it to localize the region with the flattest optima, and then has just enough time with the first regime

Original: LRs to pass the high-sharpness zone and converge to the minimum.
Revised: LRs to pass the high-sharpness zone and converge to the minimum.

Original: With such training, solutions with better sharpening/generalization profiles are achieved, which is consistent with the results of
Revised: With such training, solutions with better sharpening/generalization profiles are achieved, which is consistent with the results of

Original: Section 5.2 and the existing literature on pre-training with high LR values [22, 10].
Revised: Section 5.2 and the existing literature on pre-training with high LR values [23, 11].

Original:  Conclusion and Discussion
Revised:  Conclusion and Discussion

Original: In this work, we investigated the properties of training scale-invariant neural networks on the sphere using projected SGD.
Revised: In this work, we investigated the properties of training scale-invariant neural networks on the sphere using projected SGD.

Original: This method allowed us to study the intrinsic domain of SI models more accurately.
Revised: This method allowed us to study the intrinsic domain of SI models more accurately.

Original: We discovered three regimes of such training depending on the effective learning rate value: convergence, chaotic equilibrium, and divergence.
Revised: We discovered three regimes of such training depending on the effective learning rate value: convergence, chaotic equilibrium, and divergence.

Original: They uncover multiple properties of the intrinsic loss landscape of scale-invariant neural networks: some are related to diverse previous results in deep learning, and some appear as novel.
Revised: They uncover multiple properties of the intrinsic loss landscape of scale-invariant neural networks: some are related to diverse previous results in deep learning, and some appear as novel.

Original: We showed that these regimes are present in conventional neural network training as well and can provide intuition on how to achieve better optima.
Revised: We showed that these regimes are present in conventional neural network training as well and can provide intuition on how to achieve better optima.

Original: Possible future directions Speaking about the potential prospects for further research, three directions can be distinguished.
Revised: Possible future directions Speaking about the potential prospects for further research, three directions can be distinguished.

Original: First, it would be interesting to develop more solid and general theoretical groundings for the observed phenomena.
Revised: First, it would be interesting to develop more solid and general theoretical groundings for the observed phenomena.

Original: For instance, a closer look at the negative feedback principle in eq.
Revised: For instance, a closer look at the negative feedback principle in eq.

Original: (5) could potentially shed more light on the ESS/ELR dynamics in the first and second regimes compared to the current decay vs. alignment dilemma and maybe better explain occasional transitions from the second to the first regime for certain ELRs in Section 5.3.
Revised: (5) could potentially shed more light on the ESS/ELR dynamics in the first and second regimes compared to the current decay vs. alignment dilemma and maybe better explain occasional transitions from the second to the first regime for certain ELRs in Section 5.3.

Original: Second, the analysis could be extended to other tasks, architectures, and especially optimizers like
Revised: Second, the analysis could be extended to other tasks, architectures, and especially optimizers like

Original: Adam [15], which can be challenging due to their indirect impact on the effective direction and effective learning rate in SI models [27].
Revised: Adam [16], which can be challenging due to their indirect impact on the effective direction and effective learning rate in SI models [29].

Original: Third, it is very tempting to investigate other potential practical implications of the three regimes for conventional training.
Revised: Third, it is very tempting to investigate other potential practical implications of the three regimes for conventional training.

Original: The analysis of various LR schedules from the perspective of the discovered training regimes is intriguing and might help in designing more efficient and more explainable LR schedules.
Revised: The analysis of various LR schedules from the perspective of the discovered training regimes is intriguing and might help in designing more efficient and more explainable LR schedules.

Original: For example, from our experiments we observe that decreasing (E)LR in the first regime during training does not influence the final solution but only slows down the convergence; at the same time, for the optimal final performance, optimization should run through the low second regime (E)LRs, which allows to achieve basins containing solutions with the best sharpness/generalization properties.
Revised: For example, from our experiments we observe that decreasing (E)LR in the first regime during training does not influence the final solution but only slows down the convergence; at the same time, for the optimal final performance, optimization should run through the low second regime (E)LRs, which allows to achieve basins containing solutions with the best sharpness/generalization properties.

Original: Limitations and societal impact The main limitations of our work are applying a single optimization method on the sphere (projected SGD), experimenting only with BN networks as SI models, and considering only one of the symmetries that may conceal the actual properties of neural networks intrinsic domain [19].
Revised: Limitations and societal impact The main limitations of our work are applying a single optimization method on the sphere (projected SGD), experimenting only with BN networks as SI models, and considering only one of the symmetries that may conceal the actual properties of neural networks intrinsic domain [20].

Original: To our knowledge, our work does not have any negative societal impact.
Revised: To our knowledge, our work does not have any negative societal impact.

Original: Acknowledgments and Disclosure of Funding
Revised: Acknowledgments and Disclosure of Funding

Original: We would like to thank Mikhail Burtsev for the valuable discussions.
Revised: We would like to thank Mikhail Burtsev for the valuable discussions.

Original: The publication was supported by the grant for research centers in the field of AI provided by the Analytical Center for the
Revised: The publication was supported by the grant for research centers in the field of AI provided by the Analytical Center for the

Original: Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE
Revised: Government of the Russian Federation (ACRF) in accordance with the agreement on the provision of subsidies (identifier of the agreement 000000D730321P5Q0002) and the agreement with HSE

Original: University № 70-2021-00139.
Revised: University № 70-2021-00139.

Original: The empirical results were supported in part through the computational resources of HPC facilities at HSE University [16].
Revised: The empirical results were supported in part through the computational resources of HPC facilities at HSE University [17].

Original: Additional revenues of the authors for the last three years: laboratory sponsorship by Samsung Research, Samsung Electronics and Huawei
Revised: Additional revenues of the authors for the last three years: laboratory sponsorship by Samsung Research, Samsung Electronics and Huawei

Original: Technologies; honorarium for teaching by MSU; internship at Yandex. 
Revised: Technologies; honorarium for teaching by MSU; internship at Yandex. Checklist

Revised: For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

Revised: [Yes] (b) Did you describe the limitations of your work?

Revised: [Yes] See the end of Section 7.

Revised: (c) Did you discuss any potential negative societal impacts of your work?

Revised: [Yes] See the end of Section 7.

Revised: (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?

Revised: [Yes] 2.

Revised: If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?

Revised: [Yes] See Section 3.

Revised: (b) Did you include complete proofs of all theoretical results?

Revised: [Yes] See Appendix 3. 3.

Revised: If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

Revised: [Yes] URL at the end of Introduction.

Revised: (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

Revised: [Yes] See Section‘4 and Appendix C. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

Revised: [Yes] We show that the described training regimes are consistent for different random seeds in Appendix E (d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

Revised: [Yes] See Appendix C. 4.

Revised: If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?

Revised: [Yes] See Section 4.

Revised: (b) Did you mention the license of the assets?

Revised: [Yes] See Appendix C. CIFAR datasets are distributed under MIT license and code is under Apache-2.0 License license (we provide the link to the GitHub in Section 4.)

Revised: (c) Did you include any new assets either in the supplemental material or as a URL?

Revised: [Yes] We provide our code, URL at the end of Introduction.

Revised: (d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

Revised: [No] We train networks on the CIFAR datasets which are widely used in NeurIPS papers.

Revised: (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

Revised: [No] We train networks on the CIFAR datasets which are widely used in NeurIPS papers.

Revised: If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?

Revised: [N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

Revised: [N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

Revised: [N/A]