Toward Equation of Motion for Deep Neural Networks: Continuous-time Gradient Descent and Discretization Error Analysis
Abstract
We derive and solve an “Equation of Motion” (EoM) for deep neural networks (DNNs), a differential equation that precisely describes the discrete learning dynamics of DNNs.
Differential equations are continuous but have played a prominent role even in the study of discrete optimization (gradient descent (GD) algorithms).
However, there still exist gaps between differential equations and the actual learning dynamics of DNNs due to discretization error .
In this paper, we start from gradient flow (GF) and derive a counter term that cancels the discretization error between GF and GD.
As a result, we obtain EoM , a continuous differential equation that precisely describes the discrete learning dynamics of GD.
We also derive discretization error to show to what extent EoM is precise.
In addition, we apply EoM to two specific cases: scale- and translation-invariant layers.
EoM highlights differences between continuous-time and discrete-time GD, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD.
Our experimental results support our theoretical findings.
Introduction
[Figure]
Learning dynamics
Let us first explain our primary motivation for the present paper.
In physics , one of the fundamental goals is to predict the dynamics of matter and its fundamental constituents.
Specifically, “predict” here means to construct differential equations that best describe the physical system under consideration and to solve them.
Such differential equations are called Equations of Motion (EoM).
An interesting question here may be “What is the EoM for deep neural networks (DNNs)?”
That is, to what extent can we predict the discrete learning dynamics of DNNs by constructing differential equations?
This is our research question.
Differential equations have played a prominent role in studying discrete optimization (gradient descent (GD) algorithms), although they are continuous [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].
In the context of deep learning, gradient flow (GF) and stochastic differential equations (SDEs) are used to analyze (stochastic) gradient descent ((S)GD).
Research targets include: convergence [6, 7, 8, 12, 13, 9, 14, 17], stability of optimization [19], optimization with constraints [19], convergent states [17, 20], flatness of loss landscapes [17], empirical risk bounds [15], and online PCA [11].
Various techniques for continuous analysis have been imported to the analysis of discrete GD algorithms.
However, there still exist gaps between differential equations and actual learning dynamics due to discretization error , which is the main interest of the present paper and is often missing in the literature above.
To be specific, we focus on GF ˙ θ ( t ) =
− g ( θ ( t )) as a continuous approximation of GD θ k +
=
θ k − η g ( θ ( t )) ,
where θ ( t ) ∈ R d and θ k ∈ R
d are the weight parameters of a DNN at time t ∈ R and step k ∈ Z , respectively, and g ∈ R d is a gradient vector.
η ∈ R is a learning rate and is regarded as the discretization step size when GF is discretized with the Euler method [21]:
[Equation]
Due to this approximation, discretization error (or “continuation error”) is introduced, and thus GF cannot fully explain the dynamics of GD.
For instance, we show that according to GF, the weight norm of a scale-invariant layer collapses to zero when we use weight decay, while GD does not show such behavior (Section 5.1).
To fill the critical gap between GF and GD, we propose modifying GF to describe the learning dynamics of GD more precisely; i.e., we add a counter term ξ ∈ R d to the gradient g of GF that cancels the discretization error (Figure 1).
This idea is motivated by backward error analysis in numerical analysis [21].
We derive a functional integral equation that determines the counter term and solve it (Section 3).
As a result, we obtain a more reliable differential equation, called EoM here, that describes the discrete learning dynamics of GD.
Using the counter term, we derive the leading order of discretization error (Section 4.1) to show to what extent GF and EoM are precise in describing
GD’s dynamics.
This point is often missed in the literature on the continuous approximation of discrete GD algorithms [22, 23, 24, 11, 25, 26, 27, 28].
We further derive a sufficient condition for learning rates for the discretization error to be small (Section 4.2).
We show that EoM well explains empirical results.
Furthermore, to show the benefits of EoM, we apply it to two specific cases: scale-invariant layers [29, 30] and translation-invariant layers [31, 32] (Section 5).
For scale-invariant layers, we show that a better description of GD’s discrete dynamics requires modifications to the decay rate of weight norms that is previously derived in the continuous regime (SDEs) [33].
In addition, we show that
EoM successfully reproduces the limiting dynamics ( t → ∞ ) of weight norms and angular update [34] that are previously derived in the discrete regime, while GF cannot reproduce this result.
For translation-invariant layers, we show that EoM rather than GF dramatically matches empirical results, indicating the importance of the counter term.
To the best of our knowledge, no study analyzes the temporal evolution of translation-invariant layers except for [31] and [32], where only the sum of weights is their focus, while we derive the dynamics of the whole weights.
Our contribution is four-fold.
Our code 1 and detailed experimental results are given as supplementary materials.
To fill the critical gap between GF and GD, we derive a counter term for GF that cancels the discretization error, and as a result, we obtain EoM, a continuous differential equation that precisely describes the discrete learning dynamics of GD.
To show to what extent GF and EoM are precise in describing discrete GD dynamics, we derive the leading order of discretization error, as is often missed in the literature on the continuous approximation of discrete GD algorithms.
We further derive a sufficient condition for learning rates for the discretization error to be small.
We apply EoM to two specific cases: scale-invariant layers and translation-invariant layers, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD.
Our experimental results support our theoretical findings.
Our work is the first step toward answering this research question: to what extent can we predict the discrete learning dynamics of DNNs by constructing differential equations (EoM for DNNs)?
Also, our work helps researchers import continuous analysis to the discrete analysis of GD algorithms.
In this sense, our work bridges discrete and continuous analyses of GD algorithms.
Related Work
The idea of approximating discrete-time stochastic algorithms with continuous equations dates back to stochastic approximation theory [1, 2, 3, 4, 5].
Their primary focus is convergence analysis for discrete-time algorithms, while our focus is to predict the learning dynamics (temporal evolution) of weight parameters, such as the decay rates of weight norms and effective learning rate of scaleinvariant layers.
Our idea of the counter term is inspired by the backward error analysis developed for numerical analysis [35].
This idea is now used to analyze discrete optimization [22, 23, 24, 11, 25, 26, 27, 28].
[18] is a pioneering work on discretization error analysis between GF and GD that is based on the numerical analysis of the Euler method [21].
They derive a sufficient condition for learning rates for the discretization error to be small.
This analysis is based on a bound (inequality), while we derive an explicit relationship between learning rates and discretization error as an equality.
Neural mechanics and Noether’s learning dynamics [31, 32] provide a solution to a part of the aforementioned problem: to what extent can we predict the learning dynamics of DNNs by constructing differential equations?
They derive (the breaking of) conservation laws of weight parameters using differential equations and provide the temporal evolution of the conserved quantities.
The present work is inspired by these studies but has crucial differences: 1) our focus is on the temporal evolution of all of the network parameters, not only the conserved quantities, 2) the gradient’s correction for canceling the discretization error is not limited to the first order, but all orders, and 3) the discretization error is explicitly provided in the present paper.
See Appendix G for more related studies.
Equation of Motion for Deep Neural Networks
In the following sections, we define EoM by modifying GF (Section 3.1).
We show that the counter term satisfies a functional integral equation (Section 3.2), and then we solve it (Section 3.3).
3.1 Our Approach and Definitions
We begin with a simple idea: add a counter term to GF to cancel discretization error, i.e.,
[Equation]
[Equation]
) where θ ( t ) ∈
R
d is the vectorized weight parameters of a DNN at time t ∈ R , d ∈ N is the dimension of the weight, and ˙ θ ( t ) denotes d θ ( t ) /dt .
Gradient g ( θ ( t )) is defined as g ( θ ( t )) :
= ∇ f ( θ ( t ))
+ λ θ ( t ) , which consists of a loss function f
( θ ( t )) and weight decay term λ θ ( t ) , where λ > 0 controls the strength of weight decay.
η > 0 is a small learning rate, and ξ ( θ ( t ))
∈ R d is the counter term.
Throughout this paper, we assume all functions are sufficiently smooth.
We call
Equation (1) the Equation of Motion (EoM) for DNNs, or simply EoM.
Our aim is to find ξ that makes Equation (1) more reliable to precisely approximate GD θ k +
=
θ k − η g ( θ k ) , where θ k ∈ R
d is the weight at step k ∈ Z ≥ 0 .
To do so, we first define the discretization error between GF (1) and GD at step k :
[Equation]
) and find ξ that makes e k small.
Throughout this paper, we use the standard Euler method to discretize
GF: ˙ θ ( t ) ≒ ( θ ( t + η ) − θ ( t )) /η and t = kη ; thus, η is identified with the discretization step size.
3.2 How to Determine Counter Term
We show that the leading order of e k with respect to η is controlled by the counter term (Theorem 3.2), and as a result, the counter term is determined via a functional integral equation (Equation (6)).
Our first theorem shows what the counter term should cancel.
Theorem 3.1 (Recursive formula for discretization error) .
Discretization error e k satisfies:
[Equation]
Here, we defined Λ ( θ ( kη )) :
=
η 2 (cid:82) 1 0 ds ¨ θ ( η ( k + s ))
(1 − s ) − η 2 ξ ( θ ( kη ))
∈ R d .
The proof is based on Taylor’s theorem and is given in Appendix A.1.
The right-hand side of Equation (3) tells us that the counter term (third term) should cancel the first and second terms.
However, the following theorem states that the first term gives only subleading contributions with respect to η .
Theorem 3.2 (Leading order of discretization error) .
Suppose that Λ ( θ ( kη ))
=
O ( η γ ) and e 0 =
O ( η γ ) for some γ > 0 .
Then e k
= O ( η γ )
and − η ( g ( θ ( kη ))
− g ( θ ( kη ) − e k ))
=
O ( η γ +1 ) .
Therefore, the first term in the right-hand side of Equation (3) is negligible compared with Λ :
[Equation]
The proof is by induction and given in Appendix A.2.
Therefore, the leading order of discretization error is O ( η γ ) and given by:
[Equation]
This is a functional equation of ξ because ¨ θ ( t ) contains ξ via Equation (1).
A solution to Equation (6) for a large γ gives a small Λ and thus gives a small e k via Equation (5).
3.3 Solution to Equation
How can we solve Equation (6)?
It is not easy to find an exact solution because Equation (6) is a functional integral equation [36, 37, 38, 39, 40]; therefore, we assume a power series solution with respect to η :
[Equation]
In the following theorem, we successfully find a solution for all orders of η .
Theorem 3.3 (Solution of Equation 6) .
The solution to Equation (6) of form (7) is given by
[Equation]
for α = 0 , 1 , 2 , ... , where we use differential operators (Lie derivatives) D α :=
ξ α − 1 ( θ ) · ∇ ( α = 1 , 2 , ... ) and D 0 :=
g ( θ ) · ∇ and also defined Ξ α ( θ ) :
= ξ α − 1 ( θ ) ( α = 1 , 2 , ... ) and Ξ 0 ( θ ) :
= g ( θ ) .
The proof follows from the definition of the Lie derivative and is given in Appendix A.3.
The first two orders of the solution are given by:
[Equation]
Discussions.
As can be inferred from Equations (8–10), ˜ ξ α contains the α + 2 nd -order derivative of the loss function.
Therefore, the higher-order counter terms cancel the higher-order smoothness of the discretization error.
Here, we note that Equation (8) can be found, e.g., in [35], as a higher-order backward error analysis.
However, our derivation above has independent contributions: 1) we clarify that the counter term cancels the leading order of discretization error (Theorem 3.2), and 2) we find that the discretization error itself is also given by the counter term (Corollary 4.1 in the next section).
Equation (9) often appears in the literature on backward error analysis [21, 35] and its related topics in machine learning, e.g., [41, 23, 24, 27, 28, 31].
Typically, ˜ ξ 0 is added to gradients of continuous equations (e.g., SDE) to close the gap between continuous equations and discrete algorithms (e.g.,
SGD) by canceling (at least first-order) discretization error.
However, higher-order discretization error is neglected in these studies.
In contrast, our solution (8) cancels all orders of discretization error.
Discretization Error
The question here is to what extent the continuous approximation (1, 8) is precise; this point is often missed in the literature on continuous approximation [22, 23, 24, 11, 25, 26, 27, 28].
In this section, we use the counter term (8) and quantify discretization error as a function of the loss function and its derivatives (Section 4.1).
We find that our result well explains empirical results.
We further derive a sufficient condition for learning rates for the discretization error to be small (Section 4.2).
4.1 Counter Term Gives Leading Order of Discretization Error
We show that the counter term gives the leading order of discretization error between GD vs. GF and
EoM. The proof follows from Theorem 3.2 and 3.3 and is given in Appendix A.4.
Corollary 4.1 (Leading order of discretization error is given by ˜ ξ α ) .
Suppose that we use ξ up to
O ( η γ − 1 ) , i.e., ξ = ˜
ξ
+ η ˜ ξ 1 + · · · + η γ − 1 ˜ ξ γ − 1 for γ ∈ Z > 0 ( ξ := 0 for γ = 0 ).
Then,
[Equation]
First, Corollary 4.1 implies that the higher the orders of the counter term we use (large γ ), the more precise EoM (1) is (small e k ).
Thus, GF ( ξ = 0 ) gives larger discretization error than EoM ( ξ ̸ = 0 ).
Second, Corollary 4.1 gives the equality of the leading order of discretization error at arbitrary steps.
This is not a bound [18] nor an asymptotic analysis ( k → ∞ ).
Third, let us give an intuition by considering ξ = 0 (GF).
Then, Corollary 4.1 gives:
[Equation]
where H ( θ ) ∈ R
d × d is the Hessian of the loss function f with respect to θ
and I ∈ R
d × d is the identity matrix.
Equation (12) suggests that 1) large learning rates lead to a large discretization error and 2) steep loss functions (along the trajectory) lead to a large discretization error.
𝒆𝒆
Empirical result.
We find Equation (12) well explains our empirical result.
We compare Equation (12) (up to O ( η 2 ) ) with the actual discretization error of GD and GF in Figure 2.
First, the gap between our theoretical prediction of discretization error (orange curve) and the actual discretization error (red curve) is small because the range of relative error ( || e k
|| /
|| θ
k || ) in this plot is only 0–0.01 (see also Figure 11 in Appendix F).
Second, most of the discretization error for Theory (orange curve) and Experiment (red curve) is produced within the first 100 steps.
We can understand this phenomenon with the help of Equation (12).
It suggests that discretization error can be enhanced when the loss function is non-smooth along the learning trajectory, which is likely to occur at the beginning of training due to random initialization.
Therefore, a large part of discretization error is produced in the early stage of training.
Third, we see that most of the gap between Theory (orange curve) and Experiment (red curve) also comes from the first 100 steps; in fact, the green curve shows that there is a much smaller enhancement of the gap after the 100th step.
The source of the gap is the higher-order term O ( η 3 ) in Equation (12).
It consists of higher-order derivatives of the loss function (Theorem 8 and Corollary 4.1) and thus can be large when the loss function is non-smooth along the learning trajectory.
Therefore, by the same logic as above, the early stage of training tends to produce a gap between Theory (orange curve) and Experiment (red curve).
4.2 Discretization Error Bounds
We provide a sufficient condition (an upper bound for η ) for GF and EoM to follow GD up to a given step k , which helps us infer desired learning rates (step sizes) for the discretization error to be small.
We first consider ξ = 0 (GF).
Corollary 4.2 (Learning rate bound for ξ = 0 ) .
Let ξ = 0 and assume that e
= O ( η 3 ) .
Let ϵ and t be arbitrary positive numbers.
If the step size satisfies
[Equation]
for some k ∈ { 1 , 2 , ..., ⌊ tη ⌋} , then the discretization error can be arbitrarily small:
[Equation]
The proof follows from Equation (12) and is given in Appendix A.5.
We see that 1) there is no guarantee that the discretization error is small unless the learning rate is sufficiently small, 2) we need small learning rates to keep the discretization error small for a long period, and 3) we need small learning rates to keep the discretization error small for non-smooth loss landscapes.
This is consistent with our empirical results in Figure 3 and 4; in fact, 1) the discretization error blows up for a large learning rate ( η = 10 − 1 in Figure 3), 2) it increases as the number of steps increases (Figure 4), and 3) most of it is produced in the early phase of training, where the objective function tends to be non-smooth, and the gradients tend to be large.
We compare our bound (13) with a bound given in [18] because, to our knowledge, only [18] provides a bound for the step size with respect to discretization error in the context of deep learning.
In [18], it is proved that in essence, η ≲ ϵ/β tϵ γ tϵ c t , where β tϵ and γ tϵ measure the non-smoothness of the loss function, and c t depends on the spectrum of the Hessian.
These factors are hard to compute analytically unless the loss function and network are simple, but the qualitative behavior of this bound is the same as ours (13); i.e., both bounds become tight when the loss function is non-smooth.
We also derive a learning rate bound for ξ = ˜ ξ [-
-]0 (EoM) and the full statement is given in Corollary
A.1 in Appendix A.6, which states that if η < O ( 3 (cid:112) ϵk ) , then || e k || <
ϵ
+ O ( ϵ 43 ) .
Therefore, larger step sizes are now allowed compared with Corollary 4.2 (GF) because of the non-zero counter term.
Furthermore, we can show larger bounds for higher-order counter terms in a similar way.
Weight decay is 10 − 3 .
Curves include both GF and
EoM. Relative discretization error is also shown in Appendix F. See Section 6 for experimental settings.
[Figure]
Figure 3 is magnified.
The counter term reduces discretization error as expected, and smaller learning rates give smaller discretization errors
Application: Scale- and Translation-invariant Layers
To show the benefits of EoM, we finally apply our theory to two specific cases: scale-invariant layers [29, 30] and translation-invariant layers [31, 32].
Additionally, Appendix B provides an application to broken conservation laws [31].
In the following, we simply focus on ξ = 0 and ξ = ˜
ξ 0 to analyze the differences between ξ = 0 and ξ ̸ = 0 .
Definitions Let us first introduce our notation.
A transformation ψ of θ ∈ R d with parameter α ∈ R is said to be a symmetry transformation of loss function f if f ( ψ ( θ , α ))
=
f ( θ ) .
1 A ∈ { 0 , 1 } d denotes the indicator vector of subspace A ⊂ R d
(e.g., A is a linear layer in the DNN).
For a scalar α ∈ R , we define α A :=
α 1 A + 1 A c ∈ R , where A c is the complement of A .
For a vector θ ∈ R d , we define θ A :
= θ ⊙ 1 A ∈ R d , where ⊙ is the Hadamard element-wise product.
For the gradient operator ∇ = ( ∂/∂θ 1 , ..., ∂/∂θ d ) ⊤ , we define ∇ A := 1 A ⊙ ∇ .
We also define r A :
=
|| θ
A || and ˆ θ A :
= θ
A /r A .
5.1 Learning Dynamics of Scale-invariant Layers
In this section, we focus on scale-invariant layers.
A scale-invariant layer A is defined as a subspace that is invariant under the scale transformation ψ ( θ , α ) :
= α A θ = α θ
A + θ A c ( α > 0 ).
For example, a linear layer immediately before a batch normalization layer is scale-invariant.
We see that for a better description of GD’s discrete dynamics, we need modifications to the decay rate of r A that is previously derived in the continuous regime [33].
In addition, we show that EoM successfully reproduces the limiting dynamics of r A and angular update [34] at t → ∞ that are previously derived in the discrete regime, while GF cannot.
In Appendix C, we additionally show that there are crucial differences between GD and GF via the effective learning rate of scale-invariant layers [29, 42, 30, 43, 44, 33, 45, 34, 46, 47].
EoM for r
We construct the EoM for r A (the EoM for ˆ θ A is given in Appendix C for completeness).
Theorem 5.1 (EoM for r A and solution) .
EoM (1) gives ˙ r 2 A ( t )
= − 2 λr 2 A ( t [Equation] − [Equation]
θ A ( t
[Equation] [Equation] Specifically, this is equivalent to:
[Equation]
for ξ = 0 (GF) and
[Equation]
for ξ = ˜ ξ 0 (EoM).
The proof is based on Equations (1, 9) and given in Appendix A.7.
Equation (15) gives r 2 A ( kη )
=
r 2 A (0) e − 2 ηλk ( k ∈ Z ≥ 0 ) at discretization; therefore, ηλ is regarded as the decay rate of r A ( intrinsic learning rate [33]).
This is originally discussed in the continuous regime (SDE) [33]; however, we find that for a better description of the discrete dynamics of GD, the decay rate needs to be modified from ηλ to ηλ (1 + ηλ 2 ) (see the exponent of Equation (17)).
This means that r A in GD decays faster than expected from a naive continuous dynamics (GF (15) and SDE [33]).
See Appendix G for higher-order corrections.
Limiting dynamics.
We next derive the limiting dynamics ( t → ∞ ) of r A .
Corollary 5.1 ( r A at equilibrium) .
When ξ = 0 (GF), r A collapses to zero as t → ∞ .
When ξ = ˜ ξ(EoM), assume that there exist two constants r A∗ ≥ 0 and c ∗ ≥ 0 such that r A ( t ) t →∞ −−−→ r A∗ and ||∇
A f (ˆ θ A ( t ) + θ A c ( t ))
||
t →∞ −−−→ c ∗ .
Then r 2 A∗ = (cid:113)
η 2 λ + ηλ 2 c ∗ .
The proof follows from Theorem 5.1 and is given in Appendix A.8.
The non-zero counter term successfully reproduces r 2 A∗ ∼ (cid:112) η/ 2 λ c ∗ [29, 34], which is originally derived in the discrete regime (SGD), although our approach is continuous (EoM (1)).
Without the counter term, we cannot explain this behavior because GF gives r A ( t ) t →∞ −−−→ 0( ̸ = (cid:112) η/ 2 λ c ∗ ) .
We next derive the limiting dynamics of angular update [34], which is designed to measure the temporal evolution of scale-invariant networks.
It is originally defined in the discrete regime:
cos ∆ k := ˆ θ A k · ˆ θ A k +1 , where ˆ θ A k :=
A ⊙ θ k ||
A ⊙ θ k || .
That is, ∆ k represents a single-step angular change in the weight parameters of the scale-invariant layers A .
In the continuous regime, we can define cos ∆( t ) :
= ˆ θ A ( t ) · ˆ θ A ( t + η ) .
Corollary 5.2 ( ∆( t ) at equilibrium) .
Let us use ξ = ˜ ξ 0 .
Suppose that the assumptions in Corollary 5.1 are satisfied.
The angular update at equilibrium, denoted by ∆ ∗ , is given by cos ∆ ∗ = 1 − ηλ 1 − η 2 λ 2 / 2 +
O ( η 3 ) , and thus, ∆ ∗ = √ 2 ηλ + O (( ηλ ) 3 / 2 ) .
The proof is based on Corollary 5.1 and is given in Appendix A.10.
EoM successfully reproduces ∆ ∗ ∼ √ 2 ηλ [34], which is originally derived in the discrete regime (SGD), although EoM is continuous itself.
On the other hand, GF cannot explain the limiting dynamics of ∆( t ) because when ξ = 0 , r ( t ) goes to zero as t → ∞ (Equation (15)), and thus, cos ∆( t ) =
θ A ( t ) r A ( t ) · θ A ( t + η ) r A ( t + η ) is ill-defined.
In summary, there are gaps between GF and GD, and our discussion above indicates that the counter term is inevitable to describe the actual dynamics of GD.
5.2 Learning Dynamics of Translation-invariant Layers
Next, we apply EoM to translation-invariant layers.
To the best of our knowledge, no study analyzes the temporal evolution of translation-invariant layers except for [31] and [32], where only the sum of weights is their focus, while we derive the dynamics of the whole weights.
A translation-invariant layer A is defined as a layer that is invariant under the translation transformation ψ ( θ , α ) :
= θ + α 1 A ( α ∈ R ).
For example, a linear layer immediately before the softmax layer is translation-invariant.
In the following, we derive EoM and show that its theoretical prediction of decay rates dramatically matches empirical results, indicating the importance of the counter term.
In Appendix D, we additionally discuss the differences between GF and GD in translation-invariant layers.
For convenience, we first decompose θ A to two vectors (Figure 5); θ A⊥ is orthogonal to ∇ f ( θ ) , and θ A∥ is orthogonal to θ A⊥ .
Here, note that ∇ f ( θ ) is orthogonal to 1 A because of translation invariance; in fact, differentiating both sides of f ( θ + α 1 A ) =
f ( θ ) with respect to α and setting α = 0 , we have 1 A · ∇ f ( θ ) =
(see also Lemma A.7 in Appendix A.11).
Formally, we define θ A⊥ , θ A∥ , and the projection matrix P as θ A⊥ := P θ A = 1 A · θ A d A 1 A , θ A∥ :=
( I − P )
θ
A = θ A − θ A⊥ , and P :=
1 d A 1 A 1 ⊤A , where d A is the dimension of A .
We construct the EoM for θ A⊥ (the EoM for θ A∥ is given in Appendix D for completeness).
Theorem 5.2 (EoM for θ A⊥ ) .
EoM (1) gives ˙ θ A⊥ ( t ) =
− λ θ A⊥ ( t ) − ηP ξ
( θ ( t )) .
Specifically, this is equivalent to ˙ θ A⊥ ( t ) =
− λ θ A⊥ ( t ) ⇐⇒ θ A⊥ ( t )
=
θ
A⊥ (0)
e − λt for ξ = 0 (GF) and ˙ θ A⊥ ( t ) =
− ( λ + ηλ 2 2 ) θ A⊥ ( t )
⇐⇒ θ A⊥ ( t )
=
θ
A⊥ (0) e − ( λ + ηλ 2 2 ) t for ξ = ˜ ξ 0 (EoM).
The proof is based on Equations (1, 9) and is given in Appendix A.11.
θ A⊥ monotonically collapses to zero as t → ∞ in either case of ξ = 0 or ξ ̸ = 0 ; thus, as t increases, the dynamics is restricted onto the subspace orthogonal to θ A⊥ (Figure 5).
The decay rate is corrected by the counter term from ηλ to ηλ + η 2 λ 2 2 , as is also done for r A in Section 5.1.
Therefore, the θ A⊥ of GD decays faster than that of GF.
Figure 6 and Table 1 support our findings.
In particular, Table 1 shows that the decay rates predicted by EoM dramatically match those of GD, indicating the importance of the counter term
[Table]
[Table] [Figure]
Here, θ A = ( θ A 1 , θ A 2 ) ⊤ .
θ A⊥ decays to(also shown in Figure 6). The decay of
GD is faster than that of GF (Theorem 5.2).
As t increases, the dynamics is restricted onto the subspace orthogonal to θ A⊥ .
|| 𝜽 𝐴 ⊥ ||
Experiment
We explain our experimental settings for Figures 2–6 and Table 1.
Our network consists of a first linear layer, swish activation [48], second linear layer, batch normalization [49], third linear layer, and last softmax layer.
Cross-entropy is used for the loss function.
We note that the second linear layer is scale-invariant, and the last linear layer is translation invariant.
The batch normalization uses fixed statistics to keep the scale invariance of the second linear layer.
Swish is chosen to ensure differentiability.
None of the linear layers have a bias term.
The dataset is the training set of MNIST [50], and thus, the batch size is 60,000.
Gradient descent is used for the optimizer.
We use 64-bits of precision for all computations.
To simulate GF and EoM, we use a sufficiently small learning rate ( 10 − 5 ).
The results are produced from only one random seed to save on computational costs, but we confirm that different random seeds lead to similar results.
More detailed information is given in Appendix E and our code.
In all experiments, we use ξ = ˜ ξ 0 for EoM.
We do not include higher-order counter terms, such as ˜ ξ 1 , because they require third and higher order derivatives of the loss function and are thus extremely memory-consuming.
We could circumvent this issue, e.g., by applying Hessian-free optimization [51], but this is out of our current scope.
Conclusion and Limitations
In this work, to fill the critical gap between GF and GD, we add a counter term to GF and obtain
EoM, a continuous differential equation that precisely describes the discrete learning dynamics of
GD.
To show to what extent GF and EoM are precise in describing GD’s discrete dynamics, we derive the leading order of discretization error, as is often missed in the literature on the continuous approximation of discrete GD algorithms.
We further derive a sufficient condition for learning rates for the discretization error to be small.
We apply our theory to two specific cases, scale- and translation-invariant layers, indicating the importance of the counter term for a better description of the discrete learning dynamics of GD.
Our experimental results support our theoretical findings.
Throughout this paper, we focus only on GD and GF to expose the ideas simply, and our study does not include stochasticity (e.g., SGD and SDE), acceleration methods (e.g., momentum and Nesterov [52]), or adaptive optimizers (e.g., Adam [53]).
Nonetheless, they could be combined with our analysis, for example, using error analysis of SDEs [23, 24], continuous-time accelerated methods [7, 54, 9, 13, 14, 55, 16], and continuous-time Adam [56].
See Appendix G for more discussions.
Therefore, our study could be extended to import continuous analysis to the discrete analysis of various GD algorithms.
In this sense, our work bridges discrete and continuous analyses of GD algorithms.
Acknowledgment
We thank Shuhei M. Yoshida for his insightful comments on the dynamics of scale-invariant layers and the experimental settings.
We also thank Hidenori Tanaka for his discussion that inspired us to start this study. Checklist For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes] (b) Did you describe the limitations of your work?
[Yes] See Sections 6 and 7 and Appendix G. (c) Did you discuss any potential negative societal impacts of your work?
[N/A] (d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] (b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix A. 3.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] See the code in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6 and Appendix E. (c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No] To save computational costs, we do not run experiments with multiple random seeds, but we confirm that different random seeds give similar results, as stated in Section 6.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix E. 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] See our code.
(b) Did you mention the license of the assets?
[Yes] See our code.
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes] See our code.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] We do not use such data.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A]
The data we are using do not include such information.
If you used crowdsourcing or conducted research with human subjects... (a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendices A Proofs A.1 Proof of Theorem 3.
Proof.
The integral form of Taylor’s theorem gives
[Equation]
Remember the definition of the discrete gradient descent:
[Equation]
Subtracting Equation (19) from Equation (18), we have
[Equation]
A.2 Proof of Theorem 3.
Proof.
The proof is by induction.
For k = 0 , e
= O ( η γ ) by assumption.
If e k
= O ( η γ ) for k ≥ 1 ,
Theorem 3.1 gives e
[Equation]
(22) η ( g ( θ ( kη )) − g ( θ ( kη ) − e k ))
=
O ( η γ +1 ) follows from Taylor’s expansion of g ( θ ( kη ) − e k ) around θ ( kη ) and from assumption e k
= O [Equation]
[Equation]
A.3 Proof of Theorem 3.
Proof.
The proof of Theorem 3.3 consists of the following three Lemmas, all of which are proved in the following sections.
Lemma A.1.
[Equation]
Lemma A.2.
For n ≥ 1 ,
[Equation]
where D k 1 · · · D k n − 1 := 1 for n = 1 .
Lemma A.3.
[Equation]
Theorem 3.3 follows by comparing both sides of Equation (6) order-by-order with using Equation (26) and the expansion of ξ (7).
Proof of Lemma [-A.
-]A.
Proof.
[Equation]
From Line (27) to (28), we used s ′ :
= s − kη and the Taylor expansion of ¨ θ ( kη + s ′ ) around kη .
From Line (28) to (29), we used (cid:82) η 0 ds ′
( η − s ′ )
s ′ n =
[Equation]
+
[Equation] n + [Equation] ( n +2) for n ≥ 0 .
A.3.2 Proof of Lemma A.
Proof.
Note that given ˙ θ ( t ) = − g ( θ ( t ))
− η ξ ( θ ( t )) , we have
[Equation]
where d 0 θ /dt {+
+}
:= θ .
Therefore,
[Equation]
≥ [Equation] Thus, by definition of D , D α , and Ξ α (Theorem 3.3 in Section 3.3), we have
[Equation]
A.3.3 Proof of Lemma A.
Proof.
From Lemma A.1 and A.2, we have
[Equation]
On the last line, we replaced n + 2 and n + k 1 + · · · + k n +2 with i and j , respectively.
A.4 Proof of Corollary 4.
Proof.
By assumption, we use
[Equation]
From Theorem 3.3, we have
[Equation]
Therefore [Equation] Theorem 3.2 gives
[Equation]
A.5 Proof of Corollary 4.
Proof.
From Equation (12), we have
[Equation]
Because e 0 = O ( η 3 ) by assumption
[Equation]
we have
[Equation] [Equation]
[Equation]
Let t > 0 be a given arbitrary number.
Then, for k ∈ { 1 , 2 , ..., ⌊ tη ⌋} ,
[Equation]
Therefore, if η < (cid:112) ϵ/k (cid:112) 2 / max 0 ≤ t ′ ≤ t {|| ( H ( θ ( t ′ )
+ λI ) g
( θ ( t ′ ))
||} , then || e k || <
[Equation] + O ( [Equation]
A.6 Proof of Corollary A.
Corollary A.1 (Learning rate bound when ξ = ˜ ξ 0 ) .
Let ξ = ˜ ξ 0 and assume that e
= O ( η 4 ) .
Let ϵ and t be arbitrary positive numbers.
If the step size satisfies
[Equation]
for some k ∈ { 1 , 2 , ..., ⌊ tη ⌋} , then the discretization error can be arbitrarily small:
[Equation]
Proof.
From Equation (10) and Corollary 4.1 and by assumption, we have
[Equation]
Because e 0 = O ( η 4 ) by assumption
[Equation]
we have e
[Equation]
[Equation] [Equation]
Therefore,
[Equation]
Let t > 0 be a given arbitrary number.
Then, for k ∈ { 1 , 2 , ..., ⌊ tη ⌋} ,
[Equation]
Therefore
[Equation]
if
[Equation]
then ||
e k
|| <
ϵ
+ O ( ϵ 4 / 3 ) [Equation]
A.7 Proof of Theorem 5.
We use the following Lemmas.
Lemma A.4.
For scale-invariant layers A , the following equations hold:
[Equation]
where H A ( θ ) :
= ( 1 A ⊙ ∇ )( 1 A ⊙ ∇ )
⊤ f ( θ ) .
Proof.
Differentiating both sides of f ( α A ⊙ θ )
=
f ( θ ) with respect to α , we have
[Equation]
where ∇ A f ( α A ⊙ θ ) means ( ∇ A f ( θ ))
| θ =
α A ⊙ θ .
For α = 1 , we have
[Equation] [Equation]
Applying ∇ , we have
[Equation]
Multiplying by 1 A c ⊙ , we have
[Equation]
Therefore,
[Equation]
Lemma A.5.
For scale-invariant layers A , the following equations hold:
[Equation]
where ∇ A f (ˆ θ A + θ A c ) :
= ( ∇ A f ( θ ))
| θ
=
ˆ θ A + θ A c .
Proof.
Note that f ( θ ) =
f ( α A ⊙ θ )
=
f ( α θ A + θ A c ) .
Differentiating both sides with respect to θ , we have
[Equation]
[Equation]
[Equation]
For α = 1 /r A , we have
[Equation]
Therefore,
[Equation]
Lemma A.6.
For scale-invariant layers A , the following equations hold for all α > 0 :
[Equation]
where H A ( α A ⊙ θ ) :
= (( 1 A ⊙ ∇ )( 1 A ⊙ ∇ )
⊤ f ( θ ))
| θ =
α A ⊙ θ .
Proof.
Because ∇ f ( θ ) =
α ∇ A f ( α θ A ) + ∇ A c f
( θ A c ) (Equation 75),
[Equation]
Therefore,
[Equation]
For α = 1 , we have
[Equation]
We now prove Theorem [Equation] [Equation]
Proof.
We use Lemmas A.4, A.
[Equation]
and A.6.
[Equation]
For ξ = 0 ,
[Equation]
For ξ = ˜ ξ 0 ,
[Equation]
We used
[Equation]
Using ˙ x ( t )
=
− a x + y ( t ) ⇔ x ( t [Equation] = x [Equation]
e − at
[Equation]
t 0 dτe − a [Equation] t − τ ) y ( τ ) [Equation]
we can show the remaining equations [Equation]
A.8 Proof of Corollary 5.
Proof.
When ξ = 0 , r A t →∞ −−−→ 0 is obvious from the EoM for r A (Theorem 5.1).
When ξ = ˜ ξ 0 ,
EoM is given by
[Equation]
At equilibrium, ˙ r A = 0 and ||∇
A f (ˆ θ A
+ θ A c )
||
=
c ∗ by assumption; thus, we have
[Equation]
A.9 Proof of Theorem C.
Proof.
We use Lemmas A.4, A.5, and A.6:
[Equation]
where ξ A := 1 A ⊙ ξ .
We used ˙
r A = − λr A − η ˆ θ A · ξ ( θ ) (Theorem 5.1).
Note that ηr A ((ˆ θ A · ξ A ( θ ))
ˆ θ A − ξ A ( θ )) has no ˆ θ A component; i.e., it is orthogonal to ˆ θ A .
When ξ = 0 , Equation (107) is equivalent to ˙ˆ θ A = − 1 r 2 A ∇ A f (ˆ θ A ) .
When ξ = ˜ ξ 0 , note that from Equation (99),
[Equation]
Therefore,
[Equation]
Also,
[Equation]
Therefore,
[Equation]
A.10 Proof of Corollary 5.
Proof.
We use Lemmas A.4 and A [Equation]
The angular update is defined as
[Equation]
We evaluate the higher order terms in θ A ( t + η ) and r A ( t + η ) .
First,
[Equation]
The second derivative ¨ θ ( t ) is given by
[Equation] [Equation]
c + λ 2 θ
A + O ( η ) .
Therefore,
[Equation]
Next,
[Equation]
Because ˙ r A = − λr A − η ˆ θ A · ξ (use Equation (91) and ˙ r 2 A = 2 r A ˙ r A ),
[Equation]
In addition
[Equation]
because ¨ r A = − λ ˙
r
A + O ( η ) =
λ
[Equation]
r
[Equation]
O ( η ) ,
[Equation]
Therefore
[Equation]
cos [Equation] A [Equation]
t
[Equation]
t
[Equation]
− η
[Equation]
ˆ θ A ( t [Equation] [Equation] ( θ ( t
[Equation]
+
[Equation]
+ O ( η 3 ) .
[Equation]
Substituting ξ A = ˜ ξ 0 A , and using
[Equation]
137) we have
[Equation]
[Equation] At equilibrium [Equation] we have r 2 A t →∞ −−−→
[Equation] =
[Equation]
η
[Equation]
+
[Equation]
c ∗ and ||∇ A
f (ˆ θ A ( t ) + θ
A
[Equation]
( t )
[Equation] || t →∞ −−−→ c
[Equation]
because of Corollary 5.1.
Thus,
[Equation]
[Equation]
and we have shown the first statement of the theorem.
The second statement follows from Equation (141).
By definition of cosine and tangent, we have tan
[Equation]
[Equation]
Therefore, using Taylor’s series of the tangent function, we have
[Equation]
This concludes the proof.
A.11 Proof of Theorem 5.
We use the following Lemma:
Lemma A.7.
For translation-invariant layers A , the following equations hold:
[Equation]
Proof.
Note that P ⊤ =
P , P 2 = P , and thus, P ⊤ ( I − P ) =
P ( I − P ) =
P − P = 0 .
Therefore, θ A⊥ · θ A∥ = θ ⊤A P ⊤ ( I − P ) θ A = 0 .
[Equation]
Next, differentiating f ( θ ) =
f ( θ + α 1 A ) with respect to α , we have
[Equation]
For α = 0 , we have
[Equation]
Therefore,
[Equation]
156) and
[Equation]
Next, differentiating Equation 155 with respect to θ , we have
[Equation]
Therefore,
[Equation]
and
[Equation]
Next, differentiating f ( θ ) =
f ( θ + α 1 A ) with respect to θ , we have
[Equation]
161) and
[Equation]
For α = − 1 A · θ A d A , we have
[Equation]
and
[Equation]
We begin the proof of Theorem 5.2.
Proof.
We use Lemma A.7.
[Equation]
When θ = 0 , EoM is
[Equation]
When ξ = ˜ ξ 0 A , note that
[Equation]
167) and
[Equation]
Thus,
[Equation]
Therefore,
[Equation]
Using ˙ v ( t )
= − a v ( t ) ⇔ v ( t ) = v (0) e − at , we can show the remaining equations.
A.12 Proof of Theorem D.
Proof.
We use Lemma A.7.
First, note that
[Equation]
Because ˙
[Equation]
172) and
[Equation]
[Equation]
we have
[Equation]
Note that ˙ θ A∥ is orthogonal to θ A∥ because θ A⊥ · ˙ θ A∥ = − θ A⊥ ·∇ A f ( θ ) − λ θ A⊥ · θ A∥ −
[Equation]
⊤A⊥ ( I [Equation]
P )˜
ξ
A = 0 − 0 − 0 =
(we used θ ⊤A⊥ ( I − P )
= θ ⊤A P ⊤ ( I − P )
= θ ⊤A ( P − P ) = 0 ).
When ξ = 0 , we have
[Equation]
[Equation]
When ξ = ˜ ξ 0 , we have
[Equation]
A.13 Proof of Theorem B.
Proof.
First, note that ∇
[Equation]
(183) which can be shown by differentiating f ( θ ) =
f ( G ( θ , α )) with respect to α .
Thus, assuming θ · (( ∇ f ( θ ) · ∇ ) G ( θ , α )) and using ˙ θ ( t ) =
−∇ f
( θ ( t ))
− λ θ ( t ) − η ξ ( θ ( t )) , we have d dt ( θ ( t ) · G ( θ ( t ) , α ))
(184) = ˙
θ · G ( θ , α ) + θ · ( ˙ θ · ∇ G ( θ , α )) (185) = − ∇ f ( θ ) · G ( θ , α ) − λ θ · G ( θ , α ) − η ξ ( θ ) · G ( θ , α ) + θ · ( − ( ∇ f ( θ ) · ∇ ) − λ ( θ · ∇ ) − η ξ ( θ ) · ∇ ) G ( θ , α ) (186) = − λ ( θ · G ( θ , α ) + θ · (( θ · ∇ ) G ( θ , α )))
− η ξ ( θ ) · G ( θ , α ) − θ · (( η ξ ( θ ) · ∇ ) G ( θ , α )) .
Using ˙ v ( t )
= − a v ( t ) + u t ⇔ v ( t ) = v (0) e − at + (cid:82) t 0 dτe − a ( t − τ ) u ( τ ) , we have θ ( t ) · G ( θ ( t ) , α ) (188) = θ (0) · G ( θ (0) , α ) (
[Equation]
B Learning Dynamics Induced by Symmetry Breaking: Neural Mechanics
To show the benefits of the counter term, we apply it to broken conservation laws [31].
In [31], the authors build relationships between the symmetries of weights and conserved quantities (i.e.,
Noether’s theorem [57, 58] for DNNs), and they also investigate the dynamics of DNNs under symmetry breaking.
We address three shortcomings of their analysis: 1) it includes a counter term only up to order one, 2) a discretization error analysis is missing, and 3) their experiment makes too optimistic an assumption on gradients.
First, we generalize broken conservation laws (Equations (18–20) in [31]) by adding all orders of the counter term.
Let G ( θ , α ) :
=
∂ α ψ ( θ , α ) , which is called the generator of symmetry transformation ψ .
Theorem B.1 (Generalized broken conservation law) .
Let f be symmetric under transformation ψ .
Assume that G satisfies θ ( t ) · { ( ∇ f ( θ ( t )) · ∇ ) G ( θ ( t ) , α ) } = 0 .
Then,
[Equation]
Note that the assumption holds for translation, scale, and rescale transformation [31].
Furthermore,
Equation (192) can be formally solved:
[Equation]
The proof is given in Appendix A.13.
Now, Equation (193) includes all orders of the counter term ξ = (cid:80) ∞
α =0 ˜ ξ α .
We can reproduce [31] by setting ξ = ˜ ξ 0 .
In addition, we already know the discretization error (Corollary 4.1), which is lacking in [31].
We also provide empirical results on
Equation (193) in the following sections.
B.1 Scale-invariant Layers
For scale transformation, G ( θ , α ) =
α A θ , and thus, the left hand side of Equation (193) becomes || θ
A || 2 .
Therefore, Equation 193 describes the temporal evolution of the weight norm of scaleinvariant layers.
Figure 7 shows the temporal evolution of ||
θ A || 2 for the network explained in
Section 6.
Figure 8 shows the gap of ||
θ A || 2 between GD and its theoretical predictions (GF and
EoM)
(Equation 193).
We see that the counter term reduces the gap.
There is an improvement in the experimental settings compared with [31].
As described in [31], they substitute the gradients computed in GD for the gradients used for GF’s simulation instead of using small learning rates to simulate continuous trajectories of GF.
This approximation reduces computational costs, but it causes an additional gap between the surrogate gradients and the true gradients of GF along the continuous trajectories.
Therefore, we avoid this approximation; we use a small learning rate ( η = 10 − 5 ) to simulate GF and EoM, as explained in Section 6.
+
B.2 Translation-invariant Layers
We also provide an empirical result for translation-invariant layers.
For translation transformation,
G ( θ , α ) =
α 1 A and thus the left hand side of Equation (193) becomes 1 A · θ A (sum of weights).
Therefore, Equation (193) describes the temporal evolution of the sum of weights of translationinvariant layers.
Figure 9 shows the temporal evolution of 1 A · θ A for the network described in
Section 6.
Figure 10 shows the gap of 1 A · θ A between GD and its theoretical predictions (GF and
EoM)
(Equation 193).
We see that the counter term reduces the gap.
+
C Equation of Motion for ˆ θ A
For completeness, we construct the EoM for ˆ θ
A for scale-invariant layers A .
See Section 5.1 for the
EoM for r A .
Theorem C.1 (EoM for ˆ θ A ) .
EoM (1) gives ˙ˆ θ A ( t ) =
−
[Equation] [Equation] ˆ
[Equation] + [Equation] ξ ( θ ( t )))
ˆ θ A ( t ) − ξ ( θ ( t ))) .
Specifically, this is equivalent to:
[Equation]
194) for ξ = 0 (GF) and
[Equation]
for ξ = ˜ ξ 0 (EoM), where H A (ˆ θ A ) :
= ( 1 A ⊙ ∇ )( 1 A ⊙ ∇ )
⊤ f ( θ ) | θ
=
ˆ θ A .
The proof is given in Appendix A.9.
Effective learning rate.
This result highlights the differences between GD and GF on scaleinvariant layers.
The factor 1 r 2 A (Equation (194)), which is ηr 2 A at discretization, is called the effective learning rate [29, 42, 30, 43, 44, 33, 45, 34, 46, 47].
The dynamics of ˆ θ A is induced by ∇ A f
(ˆ θ A + θ A c ) with the effective learning rate ηr 2 A , not η .
We find that the counter term corrects the effective learning rate to a matrix operator form (Equation (195)).
Let us see the meaning of each correction in order.
First, I (identity matrix) corresponds to the original effective learning rate.
Second, η 2 H A directs the gradient ∇ A f
(ˆ θ A + θ A c ) toward the maximum eigenvector of H A , i.e., a flat direction.
Therefore, GD tends to go through flatter regions than GF.
Third, η 2 I (( ∇ A c f ( θ ) + λ
θ A c ) · ∇ A c ) involves ∇ A c f into the learning dynamics of A ; therefore, A is explicitly affected by A c in GD, unlike in GF.
This point is often missing in the literature on scale-invariant networks because it is often assumed that the whole network is scale-invariant.
Fourth, η 2 ˆ θ A ∇ ⊤A f ( θ ) cancels the ˆ θ A component of the right hand side of Equation (195), which may not seem obvious but can be seen from the proof of Theorem C.1 (see Appendix A.9), and thus, ˙ˆ θ A is orthogonal to ˆ θ A , which should be satisfied anyway because ||
ˆ θ A || 2 ≡ 1 = ⇒
2 ˙ˆ θ A
· ˆ θ A = 0 .
D Equation of Motion for θ A∥
For completeness, we provide the EoM for θ A∥ .
The proof is given in Appendix A.12.
Theorem D.1 (EoM for θ A∥ ) .
EoM (1) gives
[Equation]
Specifically, this is equivalent to:
[Equation]
for ξ = 0 (GF) and
[Equation]
for ξ = ˜ ξ 0 (EoM).
This result highlights the differences between the dynamics of GD and GF.
The two factors ηλ 2 I in
Equation (198) mean that the existence of weight decay increases the learning rate (increases the velocity ˙ θ A∥ ).
The factor η 2 H means that, as mentioned in Appendix C, GD tends to go along sharper paths than GF.
Note that velocity ˙ θ A∥ is orthogonal to θ A⊥ because ∇ f , θ A∥ , and H ( ∇ f + λ θ A∥ ) are orthogonal to θ A⊥ .
H ( ∇ f + λ θ A∥ ) ⊥
θ
A⊥ follows because H v ⊥ θ A⊥ for arbitrary non-zero vector v ∈ R d ( ∵ H 1 A
= H θ A⊥ = 0 ) (see Lemma A.7).
η 2 I (( ∇ A c f ( θ A∥ + θ A c ) + λ
θ A c ) · ∇ A c ) involves ∇ A c f into the learning dynamics of A .
We see that the dynamics of θ A∥ is also independent of that of θ A⊥ , and thus, they are completely separable.
A summary of Theorems 5.2 and D.1 is given in Figure 5.
E Details of Experiment
We provide detailed experimental settings (see also Section 6).
Our computational infrastructure is a
DGX-1 server.
The fundamental libraries used in the experiment are TensorFlow 2.3 [59], Numpy 1.18 [60], and Python 3.6.8 [61].
The random seeds used for TensorFlow and Numpy are both 7.
The input image is first divided by 127.5 and subtracted by 1.
The maximum total number of iterations ismillion steps for GF and EoM.
The total runtime is approximately a month.
We use least square fitting ( np.polyfit ) to calculate the decay rates in Table 1.
More information and detailed experimental results can be found in our code.
In Figures 2 and 12, the theoretical prediction of discretization error is defined as || e k || =
η 2 2 || (cid:80) k − 1 s =
( H ( θ ( sη ))
+ λI ) g
( θ ( sη ))
||
(Equation (12)).
To reduce computational costs, we approximate the r.h.s.
: ( H ( θ ( t ))
+ λI ) g
( θ ( t )) ∼ g ( θ ( t )+ ϵ g ( θ ( t )))
− g ( θ ( t ) − ϵ g
( θ ( t )))
2 ϵ , where ϵ is set to 10 − 7 .
The green curve in Figure 2 is defined as e k = ˜
e 100 + η 2 2 (cid:80) k − 1 s =100 ( H ( θ ( sη ))
+ λI ) g
( θ ( sη ))
(compare this with Equation (12)), where ˜ e 100 is the actual discretization error at the 100th step that is obtained from GD.
Therefore, the green curve represents the theoretical prediction of discretization error after the 100th step, given ˜ e 100 .
F Supplementary Experiment F.1 Relative Discretization Error
We provide the relative discretization error, which is defined as || e k || /
||
θ k ||
( k ∈ Z ≥ 0 ).
See Figure 11.
We can see that a large learning rate ( η = 10 − 1 ) leads to a large discretization error (Figure(a) and (c)).
We also see that the counter term reduces the discretization error as expected (Figure(b) and (d)).
[Figure]
F.2 Theoretical Prediction Vs. Experimental Result of Discretization Error
We compare the theoretical prediction of discretization error between GF and GD (Equation (12)) with the actual discretization error obtained in the experiment.
The green curve is defined as e k = ˜
e 100 + η 2 2 (cid:80) k − 1 s =100 ( H ( θ ( sη ))
+ λI ) g
( θ ( sη ))
+ O ( η 3 ) (compare this with Equation (12)), where ˜ e 100 is the actual discretization error at the 100th step.
Therefore, the green curve represents the theoretical prediction of discretization error after the 100th step given ˜ e 100 .
[Figure]
G Supplementary Discussion
Supplementary related work (Section 2).
To show the benefits of EoM, we focus on scaleinvariant layers [29, 42, 30, 43, 44, 33, 45, 34, 46, 47] and translation-invariant layers [31, 32] in
Section 5.
To carry over the stability of a continuous optimization algorithm to a discretized system, the authors of [19] add a feedback term to the optimization, and after that, they apply a discretization method to it.
The authors’ primary motivation is to keep the orthogonality of the weight parameters of DNNs, which is different from ours.
Convergence of ξ (Section 3.3).
Note that the expansion of ξ in terms of η is not necessarily convergent, as is also pointed out in [35].
Thus, we have to truncate the expansion at a suitable order.
The discretization error at the truncation is given in Theorem 4.1.
Beyond leading order of discretization error (Theorem 3.2 and Section 4.1).
In this work, we analyze the leading order of discretization error.
However, higher-order terms cannot always be negligible.
We discuss in Section 4.1 that the higher-order terms are important at the beginning of training.
Existence of A c (Section 5).
In our theoretical analysis of scale- and translation-invariant layers, the network contains both invariant ( A ) and non-invariant layers ( A c ), while previous works assume the whole network is invariant for simplicity [29, 42, 30, 43, 44, 33, 45, 34, 46, 47].
We avoid this assumption and show that such mixed networks require appropriate modifications to analyses of invariant networks.
For example, ∇ f ( θ ) =
1 || θ || ∇ f
(ˆ θ ) for invariant networks, while ∇ A f ( θ ) =
1 || θ
A || ∇ A f (ˆ θ
A
+ θ A c ) for mixed networks (Lemma A.5), not 1 ||
θ
A || ∇ A f (ˆ θ A ) .
Such a naive replacement is not allowed.
Higher-order corrections to decay rate of r A (Section 5.1).
We can compute more corrections to the decay rate of r A ( A is a scale-invariant layer), using more counter terms.
For example, a long algebra gives decay rate ηλ (1 + ηλ 2 + η 2 λ 2 3 ) for ξ = ˜
ξ 0 + η ˜ ξ 1 .
The proof is similar to Appendix A.7.
On equilibrium assumptions in Corollaries 5.1 and 5.2 (Section 5.1).
We make assumptions in
Corollaries 5.1 and 5.2; there exist two constants r A∗ ≥ 0 and c ∗ ≥ 0 such that r A ( t ) t →∞ −−−→ r A∗ and ||∇
A f (ˆ θ A ( t ) + θ A c ( t ))
||
t →∞ −−−→ c ∗ .
These assumptions are similar to those given in previous studies [29, 34].
However, whether the assumptions are valid in the actual learning dynamics of DNNs is of independent interest.
In fact, the equilibrium assumption ( r A∗ ( t ) and ||∇ A f
(ˆ θ A ( t )
+ θ A c ( t ))
|| t →∞ −−−→ constant) could not be satisfied even at one million steps of GD, and potentially because of it, r A∗ and ∆ ∗ have a large discrepancy between the empirical results and theoretical predictions.
Deeper analyses on this point are needed.
Under what conditions are the equilibrium assumptions valid?
Can we relax the equilibrium assumptions and obtain realistic limiting dynamics of scale-invariant layers?
This is exciting future work.
In contrast to our empirical result mentioned above, in [34], their experiments dramatically match their theoretical prediction.
This is potentially because of differences in experimental settings; in [34], SGD is used (ours is GD) and variance is induced, ResNet-50 [62, 63] is used (ours is a fully-connected network with three layers), ImageNet [64, 65] and MSCOCO [66] are used (ours is
MNIST [50]), and large learning rates ( ∼ 10 − 1 ) and small weight decays ( ∼ 10 − 4 ) are used (ours are given in Appendix E).
Extension of EoM to general settings (Section 7).
While we focus on GD and GF for simplicity, our counter-term-based approach and discretization error analysis can be extended to more general settings, such as SGD, acceleration methods (e.g., momentum SGD), and adaptive optimizers (e.g.,
Adam [53]).
First, to extend our analysis to SGD, discretization error analysis of the Euler-Maruyama method, e.g., [67], can be used.
SDE’s error analysis [23, 24] is also relevant.
Second, we can extend our counter-term-based approach and discretization error analysis to acceleration methods by modifying the analysis for different differential equations from GF and different discretization schemes from the Euler method, as is discussed in [7, 14, 12].
Third, [56] is the first work that provides a continuous approximation of Adam.
However, its counter term and discretization error are open questions.