S PHERICAL M OTION D YNAMICS : L EARNING D YNAMICS OF N EURAL N ETWORK WITH N ORMALIZATION , W EIGHT D ECAY , AND SGD
Anonymous authors
Paper under double-blind review
A BSTRACT
In this work, we comprehensively reveal the learning dynamics of neural network with normalization, weight decay (WD), and SGD (with momentum), named as
Spherical Motion Dynamics (SMD).
Most related works study SMD by focusing on “effective learning rate” in “equilibrium” condition, i.e. assuming the convergence of weight norm.
However, their discussions on why this equilibrium condition can be reached in SMD is either absent or less convincing.
Our work investigates SMD by directly exploring the cause of equilibrium condition.
Specifically, 1) we introduce the assumptions that can lead to equilibrium condition in
SMD, and prove that weight norm can approach its theoretical value in a linear rate regime with given assumptions; 2) we propose “angular update” as a substitute for effective learning rate to measure the evolving of neural network in SMD, and prove angular update can also approach to its theoretical value in a linear rate regime; 3) we verify our assumptions and theoretical results on various computer vision tasks including ImageNet and MSCOCO with standard settings.
Experiment results show our theoretical ﬁndings agree well with empirical observations.
I NTRODUCTION AND B ACKGROUND
Normalization techniques (e.g. Batch Normalization (Ioffe & Szegedy, 2015) or its variants) are one of the most commonly adopted techniques for training deep neural networks (DNN).
A typical normalization can be formulated as following: consider a single unit in a neural network, the input is X , the weight of linear layer is w (bias is included in w ), then its output is
[Equation]
where g is a nonlinear activation function like ReLU or sigmoid, µ , σ are mean and standard deviation computed across speciﬁc dimension of Xw (like Batch Normalization (Ioffe & Szegedy, 2015), Layer Normalization Ba et al.
(2016), Group Normalization (Wu & He, 2018), etc.).
β , γ are learnable parameters to remedy for the limited range of normalized feature map.
Aside from normalizing feature map, Salimans & Kingma (2016) normalizes weight by l 2 norm instead:
[Equation]
where || · || 2 denotes l 2 norm of a vector.
Characterizing evolving of networks during training.
Though formulated in different manners, all normalization techniques mentioned above share an interesting property: the weight w afﬁliated with a normalized unit is scale-invariant : ∀ α ∈ R + , y
( X ; α W ; γ, β ) =
y ( X ; w ; γ, β ) .
Due to the scale-invariant property of weight, the Euclidean distance deﬁned in weight space completely fails to measure the evolving of DNN during learning process.
As a result, original deﬁnition of learning rate η cannot sufﬁciently represent the update efﬁciency of normalized DNN.
To deal with such issue, van Laarhoven (2017); Hoffer et al.
; Zhang et al.
(2019) propose “effective learning rate” as a substitute for learning rate to measure the update efﬁciency of normalized neural network with stochastic gradient descent (SGD), deﬁned as
[Equation]
Joint effects of normalization and weight decay.
van Laarhoven (2017) explores the joint effect of normalization and weight decay (WD), and obtains the magnitudes of weight by assuming the convergence of weight, i.e. if w t = w t +1 , the weight norm can be approximated as || w t || 2 =
O ( 4 (cid:112) η/λ ) , where λ is WD coefﬁcient.
Combining with Eq.(3), we have η eff = √ ηλ .
A more intuitive demonstration about relationship between normalization and weight decay is presented in
Chiley et al.
(see Figure 1): due to the fact that the gradient of scale invariant weight ∂ L /∂ w ( L is the loss function of normalized network without WD part) is always perpendicular to weight w , one can infer that gradient component ∂ L /∂ w always tends to increase the weight norm, while the gradient component provided by WD always tends to reduce weight norm.
Thus if weight norm remains unchanged, or “equilibrium has been reached” 1 , one can obtain
[Equation]
Eq.(4) implies the magnitude of update is scale-invariant of gradients, and effective learning rate should be √ 2 ηλ .
Li & Arora (2020) manages to estimate the magnitude of update in SGDM, their result is presented in limit and accumulation manner: if both lim T −→∞ R T = 1 /T (cid:80)
Tt =
||
w t
||
and lim
[Equation]
= 1 /T (cid:80)
[Equation]
=
[Equation]
||
[Equation]
t
[Equation]
w t +1 || exist, then we have
[Equation]
Though not rigorously, one can easily speculate from Eq.(5)
the magnitude of update in SGDM cases should be (cid:112) 2 ηλ/ (1 + α ) in equilibrium condition.
But proof of Eq.(5) requires more strong assumptions: not only convergence of weight norm, but also convergence of update norm (cid:107) w t +1 − w t (cid:107) 2 (both in accumulation manner).
As discussed above, all previous qualitative results about “effective learning rate” (van
Laarhoven, 2017; Chiley et al., 2019; Li &
Arora, 2020) highly rely on equilibrium condition, but none of them explores why such equilibrium condition can be achieved.
Only van
Laarhoven (2017) simply interprets the occurrence of equilibrium as a natural result of convergence of optimization, i.e. when optimization is close to be ﬁnished, w t = w t +1 , resulting in equilibrium condition.
However, this interpretation has an apparent contradiction: according to Eq.(4) and (5), when equilibrium condition is reached, the magnitude of update is constant, only determined by hyper-parameters, which means optimization process has not converged yet.
Li & Arora (2020) also notices the non-convergence of SGD with BN and WD, so they do not discuss reasonableness of assumptions adopted by Eq.(5).
In a word, previous results about “effective learning rate” in equilibrium condition can only provide vague insights, they are difﬁcult to be connected with empirical observation.
In this work, we comprehensively reveal the learning dynamics of normalized neural network using stochastic gradient descent without/with momentum (SGD/SGDM) and weight decay, named as
Spherical Motion Dynamics (SMD ).
Our investigation aims to answer the following question:
1 ”Weight norm remains unchanged” means || w t
{+|| 
 ≈ || w t+} Why and how equilibrium condition can be reached in Spherical Motion Dynamics?
Speciﬁcally, our contributions are
• We introduce the assumptions that can lead to equilibrium condition in SMD, and justify their reasonableness by sufﬁcient experiments.
We also prove under given assumptions, equilibrium condition can be reached as weight norm approach to its theoretical value in a linear rate regime in SMD.
Our assumptions show the equilibrium condition can occur long before the ﬁnish of the whole optimization; • We deﬁne a novel index, angular update , to measure the change of normalized neural network in a single iteration, and derive its theoretical value under equilibrium condition in SMD.
We also prove angular update can approach to its theoretical value in a linear regime along with behavior of weight norm.
Our results imply that the update efﬁciency of SGD/SGDM on normalized neural network only depends on pre-deﬁned hyper-parameters in both SGD and SGDM case; • We verify our theorems on different computer vision tasks (including one of most challenging datasets ImageNet (Russakovsky et al., 2015) and
MSCOCO (Lin et al., 2014)) with various networks structures and normalization techniques.
Experiments show the theoretical value of angular update and weight norm agree well with empirical observation.
Recently, a parallel work (Li et al., 2020b) is published to analyze the equilibrium condition of normalized neural network with weight decay in SGD case.
They model the learning dynamic of normalized network as a SDE, and propose the concept “intrinsic learning rate” ( λ e = λη ) to measure the converge rate of weight norm and effective learning rate.
Their main result, in which converging time to equilibrium in SGD case is O
(1 / ( λη )) , is consistent with our theory.
The major difference between Li et al.
(2020b)
and our work is that all results from Li et al.
(2020b) are qualitatively results or conjectures limited on SGD case.
While our work derives the quantitative results on both SGD/SGDM cases, our analysis allows us to precisely predict the value of weight norm/angular update in large scale data experiments, and clarify the difference of approaching rate between SGD and SGDM case.
Besides, Li et al.
(2020b) conducts empirical study to discuss the connection between the equilibrium state and accuracy of trained model.
Our work mainly focuses on the equilibrium condition itself, connection between equilibrium condition and performance of trained model is beyond our discussion in this paper.
The experiments we present in main text are only for veriﬁcation of our theory.
Our theory on equilibrium condition in Spherical Motion Dynamics implies equilibrium condition mostly relies on the update rules of SGD/SGDM with WD, and scale-invariant property.
The cause of equilibrium condition is independent of decrease of loss or trajectory of optimization, but equilibrium condition turns out to signiﬁcantly affect update efﬁciency of normalized network by controlling the relative update (Eq.(4)).
We believe equilibrium condition is one of the key reason why learning dynamic of normalized neural network is not consistent with traditional optimization theory (Li et al., 2020b).
We think it has great potential to study the leaning dynamic of normalized network or develop novel efﬁcient learning strategy under the view of Spherical Motion Dynamics.
R ELATED W ORK
Normalization techniques Batch normalization(BN (Ioffe & Szegedy, 2015)) is proposed to deal with gradient vanishing/explosion, and accelerate the training of DNN.
Rapidly, BN has been widely used in almost all kinds of deep learning tasks.
Aside from BN, more types of normalization techniques have been proposed to remedy the defects of BN (Ioffe, 2017; Wu & He, 2018; Chiley et al., 2019; Yan et al., 2020) or to achieve better performance (Ba et al., 2016; Ulyanov et al., 2016; Salimans & Kingma, 2016; Shao et al., 2019; Singh & Krishnan, 2020).
Though extremely effective, the mechanism of BN still remains as a mystery.
Existing works attempt to analyze the function of
BN: Ioffe & Szegedy (2015) claims BN can reduce the Internal Covariance Shift (ICS) of DNN;
Santurkar et al.
(2018) argue that the effectiveness of BN is not related to ICS, but the smoothness of normalized network; Luo et al.
(2019) shows BN can be viewed as an implicit regularization technique;
Cai et al.
(2019) proves that with BN orthogonal least square problem can converge at linear rate; Dukler et al.
(2020) proves weight normalization can speed up training in a two-layer
ReLU network.
Weight decay Weight decay (WD) is well-known as l 2 regularization, or ridge regression, in statistics.
WD is also found to be extreme effective when applied in deep learning tasks.
Krizhevsky &
Geoffrey (2009) shows WD sometimes can even improve training accuracy not just generalization performance; Zhang et al.
(2019) show WD can regularize the input-output Jacobian norm and reduce the effective damping coefﬁcient;
Li et al.
(2020a) discusses the disharmony between WD and weight normalization.
A more recent work Lewkowycz & Gur-Ari (2020) empirically ﬁnds the number of SGD steps T until a model achieves maximum performance satisﬁes T ∝ 1 λη , where λ, η are weight decay factor and learning rate respectively, they interpret this phenomenon under the view of Neural Tangent Kernel (Jacot et al., 2018), showing that weight decay can accelerate the training process.
Notice their result has no connection with equilibrium condition discussed in this work.
Our results shows the cause of equilibrium condition can be reached long before neural network can get its highest performance.
Effective learning rate Due to the scale invariant property caused by normalization, researchers start to study the behavior of effective learning rate.
van Laarhoven (2017); Chiley et al.
(2019) estimate the magnitude of effective learning rate under equilibrium assumptions in SGD case; Hoffer et al.
(2018) quantify effective learning rate without equilibrium assumptions, so their results are much weaker; Arora et al.
(2019) proves that without WD, normalized DNN can still converge with ﬁxed/decaying learning rate in GD/SGD cases respectively; Zhang et al.
(2019) shows WD can increase effective learning rate; Li & Arora (2020) proves standard multi-stage learning rate schedule with BN and WD is equivalent to an exponential increasing learning rate schedule without WD.
As a proposition, Li & Arora (2020) quantiﬁes the magnitude of effective learning rate in SGDM case.
But none of them have ever discussed why equilibrium condition can be reached.
A recent work Li et al.
(2020b) studies the convergence of effective learning rate by SDE, proving that the convergence time is of O (1 / ( λη )) , where λ, η are weight decay factor and learning rate respectively.
Their result can only provide intuitive understanding, and is limited on SGD case.
P RELIMINARY ON S PHERICAL M OTION D YNAMICS
First of all, we review the property of scale invariant weight, and depict Spherical Motion Dynamics (SMD) in SGD case.
Notice except deﬁnitions, all intuitive statements or derivations in this section mostly comes from previous literature, they are not mathematically rigorous.
We summarize them to provide background of our topic and preliminary knowledge for readers.
Lemma 1.
If w is scale-invariant with respect to L ( w ) , then for all k > 0 , we have:
[Equation]
Proof can be seen in Appendix B.1.
Lemma 1 is also discussed in Hoffer et al.
van Laarhoven (2017); Li & Arora (2020).
Eq.(7) implies gradient norm is inﬂuenced by weight norm, but weight norm does not affect the output of DNN, thus we deﬁne unit gradient to eliminate the effect of weight norm.
Deﬁnition 1 ( Unit Gradient ) .
If w t (cid:54) = 0 , ˜ w = w / || w || 2 , the unit gradient of ∂ L /∂ w | w = w t is ∂ L /∂ w | w = ˜ w t .
Setting k as 1 / || w t || 2 inEq.(7), we have
[Equation]
A typical SGD update rule without WD is
[Equation]
if || w t ||
= ||
w t
+
|| 2 , dividing both side of Eq.(9) by || w t
|| 2 , we have
[Equation]
Eq.(10) shows effective learning rate can be viewed as learning rate of SGD on unit sphere
S p − 1 (Hoffer et al., 2018).
But effective learning rate still cannot properly represent the magnitude of update, since unit gradient norm is unknown.
Therefore we propose the angular update deﬁned below.
Deﬁnition 2 ( Angular Update ) .
Assuming w t is a scale-invariant weight from a neural network at iteration t , then the angular update ∆ t is deﬁned as
[Equation]
where (cid:93) ( · , · ) denotes the angle between two vectors, (cid:104)· , ·(cid:105) denotes the inner product.
According to Eq.(6), ∂ L /∂ w is perpendicular to weight w .
Therefore, if angular update ∆ t is small enough, it can be approximated by ﬁrst order Taylor series expansion of tan ∆ t , then we can see its connection between effective learning rate and unit gradient norm
[Equation]
Another deduction from Eq.(6) is that weight norm always increases because
[Equation]
From Eq.(7) we can infer that increasing weight norm can lead to smaller gradient norm if unit gradient norm is unchanged.
Zhang et al.
(2019) states the potential risk that GD/SGD without WD but BN will converge to a stationary point not by reducing loss but by reducing effective learning rate due to increasing weight norm.
Arora et al.
(2019) proves that full gradient descent can avoid the risk and converge to a stationary point deﬁned in S p − 1 , but their results still require sophisticated learning rate decay schedule in SGD case.
Besides, practical implementation suggests training DNN without WD always suffers from poor generalization (Zhang et al., 2019; Bengio & LeCun, 2007;
Lewkowycz & Gur-Ari, 2020).
Now considering the update rule of SGD with WD:
[Equation]
We can approximate the update of weight norm by
[Equation]
[Equation]
The derivation of Eq.(15) is presented in Appendix A.1.
Eq.(15) implies WD can provide direction to reduce weight norm, hence Chiley et al.
; Zhang et al.
(2019) points out the possibility that weight norm can be steady, but do not explain this clearly.
Here we demonstrate the mechanism deeper (see Figure 1): if unit gradient norm remains unchanged, note “centripetal force” ( − λη || w t || 2 ) is proportional to weight norm, while “centrifugal force” (
[Equation]
||
∂ L ∂ w (cid:12)(cid:12)(cid:12) w = ˜
w t || 22 ) is inversely proportional to cubic weight norm.
As a result, the dynamics of weight norm is like a spherical motion in physics: overly large weight norm makes centripetal force larger than centrifugal force, leading to decreasing weight norm; while too small weight norm makes centripetal force smaller than centrifugal force, resulting in the increase of weight norm.
Intuitively, equilibrium condition tends to be reached if the number of iterations is sufﬁciently large.
Notice that the core assumption mentioned above is “unit gradient norm is unchanged”.
In fact this assumption can solve the contradiction we present in Section 1: convergence of w t leads to nonzero || w t +1 − w t || 2 / ||
w t
|| 2 .
Our analysis implies the convergence of weight norm is not equivalent to convergence of weight, steady unit gradient norm can also make weight norm converge, steady unit gradient norm does not rely on the fact that optimization has reached an optimum solution.
But a problem about unit gradient assumption rises: unit gradient norm cannot remain unchanged during training in practice, it is not a reasonable assumption.
In next section, we formulate this assumption in a reasonable manner, and rigorously prove the existence of equilibrium condition in
SGD case.
Discussion about SGD cannot be easily extended to SGDM case, because momentum is not always perpendicular to weight as unit gradient.
But we can still prove the existence of equilibrium condition in SGDM case under modiﬁed assumptions.
M AIN R ESULTS
First of all, we prove the existence of equilibrium condition in SGD case and provide the approaching rate of weight norm.
Theorem 1.
(Equilibrium condition in SGD) Assume the loss function is L ( X ; w ) with scaleinvariant weight w , denote g t =
∂
L ∂ w (cid:12)(cid:12)
X t , w t , ˜ g t =
g t · || w t || 2 .
Considering the update rule of SGD with weight decay,
[Equation]
(16) where λ, η ∈ (0 , 1) .
If the following assumptions hold: 1) λη << 1 ; 2) ∃ L, V, ∈ R + ,
E [ || ˜ g
t || 22 |
w t ] =
L , E
[( || ˜ g t || 22 − L ) 2 | w t ] ≤ V ; 3) ∃ l ∈ R
+ , || ˜
g t || 22 > l , l > 2[ 2 λη 1 − 2 λη ] 2 L .
Then ∃ B > 0 , w ∗ = 4 (cid:112) Lη/ (2 λ ) , we have
[Equation]
Remark 1.
The theoretical value of weight norm w ∗ in Theorem 1 is consistent with the derivation of weight norm in equilibrium in van Laarhoven (2017), though van Laarhoven (2017) assumes the equilibrium condition has been reached in advance, hence van Laarhoven (2017) cannot provide the approaching rate and scale of bias/variance.
The vanishing term ( (1 − 4 λη ) T B ) in Eq.(17) is consistent with the convergence time
O (1 / ( λη )) presented in Li et al.
(2020b).
Proof of Theorem 1 can be seen in Appendix B.2.
It shows the square of weight norm can approach to its theoretical value in a linear rate regime (when vanishing term is larger than noisy term in Eq.(17)), and its variance is bounded by V η 2 l , which is empirically small.
Now we discuss the reasonableness of assumptions in Theorem 1.
Assumption 1 is consistent with commonly used settings; assumptions 2 and 3 imply the E || ˜ g t || 22 remains unchanged within several iterations, and its lower bound cannot be far from its expectation.
We need to clarify assumption 2 further: we do not require the E ||
˜
g t
|| 22 must be constant across the whole training process, but only remains unchanged locally.
On one hand, small learning rate( η ) can guarantee E ||
˜
g t || 22 changes slowly; on the other hand, when E || ˜
g t || 22 changes, it means that square of weight norm will approach to its new theoretical value as Theorem 1 describes.
Experiment result in
Figure 2 strongly justiﬁes our analysis.
We also conduct extensive experiments to verify our claim further, please refer to Appendix C.1.
Now we extend Theorem 1 to SGDM case.
SGDM is more complex than SGD since momentum is not always perpendicular to the weight, therefore we need to modify assumptions.
Theorem 2.
(Equilibrium condition in SGDM)
Considering the update rule of SGDM (heavy ball method (Polyak, 1964)):
[Equation] [Equation]
(19) where λ, η, α ∈ (0 , 1) .
If following assumptions hold: 4) λη << 1 , λη < (1 − √ α ) 2 ; 5) Deﬁne h t =
||
g t ||
+ 2 α (cid:104) v t − 1 , g t (cid:105) ,
˜ h t
= h t · || w t || 22 .
∃ L, V, ∈ R + , E
[˜ h t | w t ]
=
L ,
E [(˜ h t − L ) 2 | w t ] ≤ V ; 6) ∃ l ∈ R + , h t > l , l > 2[ 6 λη (1 − α ) 3 (1+ α ) − 8 λη (1 − α ) ] 2 L .
then ∃ B > 0 , w ∗ = 4 (cid:112) Lη/ ( λ (1 − α )(2 − λη/ (1 + α ))) , we have
[Equation]
Remark 2.
So far, no other work rigorously prove equilibrium condition can be reached in SGDM cases.
Even the most relevant work (Li et al., 2020b) only provides their conjecture on approaching rate of weight norm in SGDM, they speculate that the time of approaching to equilibrium should be O (1 / ( λη )) , same order as approaching time in SGD case, their conjecture cannot provide further insight.
While our results (vanishing terms in Eq.(17), (20) respectively) can clearly reﬂect the difference: the approaching rate of SGDM should be 1 / (1 − α ) times faster than rate of SGD with same ηλ .
α is usually set as 0 .
9 , so SGDM can reach equilibrium condition much faster than SGD.
Proof can be seen in Appendix B.3.
Like assumption 1, assumption 4 is also satisﬁed for commonly used hyper-parameter settings.
Besides, λη < (1 − √ α ) 2 is also mentioned in Li & Arora (2020) for other purpose; Assumption 5 shows not unit gradient gradient norm ||
˜ g t ||
but an adjusted value ˜ h t dominates the expectation and variance of the weight norm square.
We empirically ﬁnd the expectation of (cid:104) v t − 1 , g t (cid:105) is very close to 0 , therefore the behavior of ˜ h t is similar to that of || ˜ g t ||(see (d) in Figure 2), making square of weight norm approaching its theoretical value in SGDM case.
We leave theoretical analysis on ˜ h t as future work.
As for assumption 6, commonly used settings ( ηλ << 1 ) can make 2[
[Equation]
an very small lower bound for
[Equation]
The experiments on justiﬁcation of assumptions 4,5,6 can be seen in Figure 2 and appendix C.1.
Comparing with Eq.(17) and Eq.(20), it implies that with same η, λ , SGDM can reach equilibrium condition much faster than SGD, but may have a larger variance, our experiments also verify that(see (b), (e) in Figure 2).
Since we have proven weight norm will approach to its theoretical value in SMD in a linear rate regime, we can derive the theoretical value of angular update ∆ t and its variance.
Theorem 3.
(Theoretical value of Angular Update)
In SGD case, if assumptions in theoremholds, then ∃ C > 0 , we have
[Equation]
In SGDM case, if assumptions in theorem 2 holds, ∃ C > 0 , we have
[Equation] [Equation]
Remark 3.
Theoretical value of angular update in Theorem 3 is consistent with Eq.(4, 5) from
Chiley et al.
; Li & Arora (2020) respectively.
Notice the variance term in Eq.(21,22) is of
O ( V/Ll ) , it is too large comparing with its empirical value, we leave it as a future work to improve the bound of variance term.
Though connection between performance of neural network and equilibrium condition is beyond the main discussion of this paper, the ﬁndings of a parallel work (Li et al., 2020b) can inspire us the possible advantage of momentum:
Li et al.
(2020b) interprets that smaller λη can get higher performance when training DNN, but the approaching rate ( O ( λη )) is slow; larger λη has faster approaching rate but it can only get bad performance.
Inspired by Li et al.
(2020b), we suspect that ∆ t ∝ √ λη is highly correlated to the performance of DNN: smaller angular update in equilibrium condition can lead to better performance of DNN.
According to theorem 3, with same λη , momentum method has faster approaching rate( λη/ (1 − α ) ) and smaller angular update ( (cid:112) 2 λη/ (1 + α )) than pure SGD.
This means momentum method can simultaneously accelerate training process and improve the ﬁnal performance, comparing pure SGD method.
We leave this conjecture as a future work.
Proof of theorem 3 is shown in Appendix B.4.
According to theorem 3, the theoretical value of angular update and its approaching rate almost only depends on hyper-parameters: weight decay factor λ , learning rate η , (and momentum factor α ).
It implies update efﬁciency of scale-invariant weights within a single step is totally controlled by predeﬁned hyper-parameters in equilibrium condition, regardless other attributes of the weights (shape, size, position in network structure, or effects from other weights).
That’s the key reasons why we propose Angular Update (Eq.(11)) to replace effective learning rate (Eq.(3)): effective learning rate can only reﬂect the inﬂuence of weight norm, it cannot reveal how unit gradient norm affect the relative update (see Eq.(10)), while angular update in equilibrium condition implies that both weight norm and unit gradient norm do not affect the scale of update within a single iteration, only hyper-parameters (learning rate, weight decay factor, momentum factor) matter.
Our experiment results strongly prove our claim (see Figure 3(a), 3(b), 3(d), 3(e)).
E XPERIMENTS
In this section, we show that our theorems can agree well with empirical observations on ImageNet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014).
We conduct experiments in two cases.
In the ﬁrst case we train neural network with ﬁxed learning rate to verify our theorems in
SGD and SGDM, respectively; in the second case we investigate SMD with more commonly used settings, multi-stage learning rate schedule.
5. F IXED L EARNING R ATE
With ﬁxed learning rate, we train Resnet
(He et al., 2016) with SGD/SGDM on ImageNet.
Learning rate is ﬁxed as η = 0 .
2 ; WD factor is λ = 10 − 4 ; with SGDM, the momentum factor is α = 0 .
Figure 2 presents the unit gradient norm square, weight norm, and angular update of the weights from LAYER .2.0.
CONV 2 of Resnet50 in SGD and SGDM cases, respectively.
It can be inferred from Figure 2 the behavior of || ˜
g t || 22 , ˜ h t and hyper-parameter settings satisfy our assumptions in
Theorem 1 and 2, therefore theoretical value of weight norm and angular update agree with empirical value very well.
We also can observe SGDM can achieve equilibrium more quickly than SGD.
According to Eq.(17),(20), the underlying reason might be with same learning rate η and WD factor λ , approaching rate of SGDM ( 1 − λη 1 − α ) is smaller than that of SGD ( 1 − λη ).
[Figure]
5. M ULTI - STAGE L EARNING R ATE S CHEDULE
Now we turn to study the behavior of angular update with SGDM and multi-stage leanring rate schedule on Imagenet (Russakovsky et al., 2015) and MSCOCO (Lin et al., 2014).
In ImageNet classiﬁcation task, we still adopt Resnet50 as baseline for it is a widely recognized network structure.
The training settings rigorously follow Goyal et al.
(2017): learning rate is initialized as 0 . 1 , and divided by 10 at 30 , 60 , 80 -th epoch; the WD factor is 10 − 4 ; the momentum factor is 0 .
In
MSCOCO experiment, we conduct experiments on Mask-RCNN (He et al., 2017) benchmark using a Feature Pyramid Network (FPN)
(Lin et al., 2017), ResNet50 backbone and SyncBN (Peng et al., 2018) following the 4x setting in He et al.
(2019): total number of iteration is 360 , 000 , learning rate is initialized as 0 .
02 , and divided by 10 at iteration 300 , 000 , 340 , 000 ; WD coefﬁcient is 10 − 4 .
There appears to be some mismatch between theorems and empirical observations in (a), (b) of
Figure 3: angular update in the last two stages is smaller than its theoretical value.
This phenomenon can be well interpreted by our theory: according to Theorem 1, 2, when equilibrium condition is reached, theoretical value of weight norm satisﬁes || w t
|| 2 ∝ 4 (cid:112) η λ , therefore when learning rate is
[Figure]
divided by k , equilibrium condition is broken, theoretical value of weight norm in new equilibrium condition should get 4 (cid:112) 1 /k smaller.
But new equilibrium condition cannot be reached immediately (see (c), (f) in Figure 3), following corollary gives the least number of iterations to reach the new equilibrium condition.
Corollary 3.1.
In SGD case with learning rate η , WD coefﬁcient λ , if learning rate is divided by k , and unit gradient norm remains unchanged, then at least (cid:100)
[log( k )] / (4 λη ) (cid:101) iterations are required to reach the new equilibrium condition; In SGDM case with momentum coefﬁcient α , then at least (cid:100) [log( k )(1 − α )] / (4 λη ) (cid:101) iterations are required to reach the new equilibrium condition.
Corollary 3.1 also implies SGD/SGDM with smaller learning rate requires more iterations to reach new equilibrium condition.
Hence, in second learning rate stage in Imagenet experiments, angular update can reach its new theoretical value within 15 epochs, but during last two learning rate stages of Imagenet/MSCOCO experiments, SGDM cannot completely reach new equilibrium by the end of training procedure.
As a result, we observe empirical value of angular update seems smaller than its theoretical value.
Based on our theorem, we can bridge the gap by skipping the intermediate process from old equilibrium to new one.
Speciﬁcally, when learning rate is divided by k , norm of scale-invariant weight is also divided by 4 √ k , SGDM can reach in new equilibrium immediately.
Experiments((b),(e) in Figure 3) show this simple strategy can make angular update always approximately equal to its theoretical value across the whole training process though learning rate changes.
C ONCLUSION
In this paper, we comprehensively reveal the learning dynamics of DNN with normalization,
WD, and SGD/SGDM, named as Spherical Motion Dynamics (SMD).
Different from most related works (van Laarhoven, 2017; Hoffer et al., 2018; Chiley et al., 2019; Li & Arora, 2020), we directly explore the cause of equilibrium.
Speciﬁcally, we introduce the assumptions that can lead to equilibrium condition, and show these assumptions are easily satisﬁed by practical implementation of
DNN; Under given assumptions, we prove equilibrium condition can be reached at linear rate, far before optimization has converged.
Most importantly, we show our theorem is widely valid, they can be veriﬁed on one of most challenging computer vision tasks, beyond synthetic datasets.
We believe our theorems on SMD can bridge the gap between current theoretical progress and practical usage on deep learning techniques.
A M ISSING DERIVATIONS A. A PPROXIMATION OF W EIGHT N ORM U PDATE
Now considering the update rule of SGD with WD:
[Equation]
Since ∂ L ∂ w (cid:12)(cid:12)(cid:12) w = w t is perpendicular to w t , we have
[Equation]
Therefore
[Equation]
Now assume η, λ is extremely small
[Equation]
so that
[Equation]
Therefore, we have
[Equation]
where
∂ L ∂ w (cid:12)(cid:12)(cid:12) w = ˜
w t is the unit gradient deﬁned in Deﬁnition 1.
B P ROOF OF T HEOREMS
Remark 4.
In the following context, we will use the following conclusion multiple times: ∀ δ, ε ∈ R , if | δ | (cid:28) 1 , | ε | (cid:28) 1 , then we have:
[Equation]
B. P ROOF OF L EMMA
Proof.
Given w 0 ∈ R p \{ 0 } , since ∀ k > 0 , L ( w 0 ) =
L ( k w 0 ) , then we have ∂ L ( w )
[Equation]
=
[Equation]
w
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
∂k
[Equation]
B. P ROOF OF T HEOREM
Lemma 2.
If the sequence { x t } ∞ t =1 satisﬁes
[Equation]
where x 1 > 0 , L >
Then, we have
[Equation]
Proof.
If x t ≥ (cid:113)
L 1 − α , since (cid:112) L/ (1 − α ) ≥ 2 √ αL then we have
[Equation]
36) which means ∀ k > t, x k ≥ (cid:112) L/ (1 − α ) .
If x t < (cid:112) L/ (1 − α ) , then we have
[Equation]
Therefore, by deduction method we have if x T < (cid:112) L/ (1 − α ) , then ∀ t ∈ [1 , T − 1] , we have 0 < x t < x t +
[Equation] [Equation]
In summary, we have
[Equation] [Equation]
Proof of Theorem 1.
since (cid:104) w
[Equation]
, g t (cid:105) = 0 , then we have:
[Equation]
Denote x t as || w t || 22 , L t as || ˜ g t || 22 and omit O (( ηλ ) 2 ) part.
Then L t > l , E
[ L t | x t ] =
L ,
V ar ( L t | x t ) =
E [( L t − L ) 2 | x t ] < V . Eq.(41) is equivalent to
[Equation]
According to Lemma 2, we have
[Equation]
Eq.(43) implies when t > 1 + log (( √
[Equation] [Equation]
, we have x t > (cid:113) lη 4 λ .
Now, denote x ∗ as the (cid:113)
Lη 2 λ , then we have
[Equation] [Equation]
Combining with assumption 3 in Theorem
[Equation]
we have 1 − 2 λη − (cid:113) 2 Ll · 2 λη > 0 , which means
[Equation]
Combining with Eq.(44), (43), (46), if t > T ( α, λ, η, l, x 0 ) , we have
[Equation]
[Equation]
Considering the expectation with respect to the distribution of x t
[Equation]
we have
[Equation]
Approximate (1 − 2 λη ) 2 = 1 − 4 λη , and iterate Eq.(48) for t − T ( α, λ, η, l, x 0 ) times, we have
[Equation]
T ( α, λ, η, l, x 0 ) is ﬁnite, and only depends on α, λ, η, l, x 0 .
Hence we can set B = max { (1 −
[Equation]
[Equation] [Equation]
, x 0 ) } , note B is ﬁnite by
[Equation]
(50) therefore, ∀ t > 0 , we have
[Equation]
B. P ROOF OF T HEOREM
Lemma 3.
Assume α, β, ε ∈ (0 , 1) , where β << 1 .
Denote diag (1 − 2 β 1 − α , α, α 2 +
α 2 1 − α β as Λ , k = ( 1 (1 − α ) 2 , − 2 α (1 − α
[Equation]
2 , α
[Equation]
2 ) T , e = (1 , 1 , 1) T .
If ε < 13 [ 1 − α 2 β − 8 1 − α ] , then ∀ d ∈ R p , we have
[Equation]
Proof.
Omit O ( β 2 ) part, we have || (
[Equation] [Equation]
=
[Equation]
Then we have
[Equation] [Equation]
Since ε < 13 [ 1 − α 2 β − 8 1 − α ]
[Equation]
we have
[Equation]
Hence, we have
[Equation]
Proof of Theorem 2.
The update rule is
[Equation]
Then we have
[Equation] [Equation]
Let X t , A , e denote:
[Equation] [Equation]
65) respectively.
The Eq.(60), (61) is formulated as a iterative map:
[Equation]
When λη < (1 − √ α ) 2 , the eigen value of A are all real number:
[Equation]
and they satisfy
[Equation]
Therefore, A can be formulated as
[Equation]
where Λ is a diagonal matrix whose diagonal elements are the eigen value of A ; the column vector of S is the eigen vectors of A , note the fromulation of S , Λ are not unique.
Speciﬁcally, we set Λ , S as
[Equation]
Moreover, the inverse of S exists, and can be explicitly expressed as
[Equation]
λ 1 − α )( α − λ 3 ) 2 α
α + λ 1 )( α + λ 3 ) ( λ 1 − α )
( α − λ 3 )(1+ α − β ) − 2 αλ 1 λ 3 ( λ 1 − α )( α − λ 3 ) ( α + λ 3 ) λ( α − λ 3 )( λ 1 − λ 3 ) − 2 λ 3 ( α + λ 3 )( α + λ 1 ) ( λ 1 − λ 3 )( α − λ 3 )(1+ α − β ) ( α + λ 3 ) λ 1 λ 3 ( α − λ 3 )( λ 1 − λ 3 )   .
(74) let Y t
=
S − 1 X t , combining with Eq.(66), we have
[Equation]
Combining with Eq.(73) and Eq.(74), and set Y t
= (˜ a t , ˜ b t , ˜ c t ) T , we rewrite Eq.(75) as
[Equation]
[Equation]
Note || w t ||
= ˜ a t + ˜ b t + ˜ c t .
Now we prove the following inequations by mathematical induction
[Equation] [Equation]
[Equation] [Equation]
Since the start point X 1 = ( a 1 , a 1 , a 1 ) T ( a 1 > 0 ), the start point Y 1 =
S − 1 X 1 .
Combining with
Eq.(74), we have
[Equation]
85) by which we have ( α − λ 1 )˜ b 1 > ( λ 1 − λ 3 )˜ c
Besides ˜ a 1 + ˜ b 1 + ˜ c
= e T SY 1 =
e T X 1 =
a 1 > 0 .
Suppose for t =
T , Eq.
(79), (80), (81), (82) hold, combining with Eq.(77), (78), we can derive ˜ b T +1 < 0 , ˜ a T +1 > 0 , so Eq.(79), (80) hold for t =
T + 1 ; Besides, we have
[Equation]
thus Eq.(81) holds for t =
T + 1 .
Sum Eq.(76), Eq.(77), Eq.(78), due to Eq.(81) we have ˜
[Equation]
+
[Equation]
˜
[Equation]
Eq.(83) holds for t =
T + 1 , combining with the fact that ˜ a T + ˜ b T + ˜ c T > 0 , we have ˜ a T +1 + ˜ b T +1 + ˜ c T +1 > 0 .
According to Lemma 2, we can estimate the lower bound of ˜ a t + ˜ b t + ˜ c t : when t > 1 +
[Equation]
Now we can analyze the expectation of distance( l 2 norm) between Y t =
(˜ a t , ˜ b t , ˜ c t ) T and the ﬁxed point Y ∗ =
(˜ a ∗ , ˜ b ∗ , ˜ c ∗ ) T which satisﬁes
[Equation]
Assume x t = ˜
[Equation]
t + ˜ b
[Equation]
˜
[Equation]
t , x ∗ = ˜
a
[Equation]
b
[Equation]
+ ˜ c ∗ > 0 , then we have:
[Equation]
where k = ( k 1 , k 2 , k 3 ) T = S − 1 e .
In the following context, we will omit the O ( λ 2 η 2 ) part since λη (cid:28) 1 .
k 1 , k 2 , k 3 can be approximated as
[Equation] [Equation]
Then we have
[Equation]
The ﬁxed point of Eq.(89) is computed as x ∗ =
[Equation]
, and we have known that if t > 1 + log
[Equation]
√
[Equation]
,, therefore we have
[Equation]
According to assumption
[Equation]
we can prove
[Equation]
combining with Lemma
[Equation]
we have
[Equation]
which implies
[Equation]
where
[Equation]
Therefore, similar to the proof of theorem 1, ∃ B > 0 ,
[Equation]
[Equation]
Recall || w t ||
=
e T Y t , therefore
[Equation] [Equation]
Remark 5.
By Eq.(101), the variance of || w t || 22 is bounded by 3 V η 2 (1+4 α
+ α 4 ) l (1 − α ), which is not small enough.
But we somehow can reduce it: according to Eq.(96), if x t is close to its theoretical value, then E [(˜ h t − L ) 2 | Y t ] xt η 4 < 4 V η 3 λ (1 − α ) l , hence variance of || w t
|| 22 can be bounded by 3 V η 2 (1+4 α 2 + α 4 ) 2 L (1 − α ).
B.
P ROOF OF T HEOREM
Proof.
In SGD case, we have
[Equation]
(102) then we have
[Equation]
According to the deﬁnition of ∆ t , ∆ t ≥ 0 , and ∆ t is very close to 0 , hence we have
[Equation]
[Equation] [Equation]
where x t , x t +1 denotes || w t
|| 22 , ||
w t
+1 ||
22 respectively as in Eq.
Assume t is sufﬁciently large so that x t , x t +1 are close to x ∗
= (cid:113)
Lη 2 λ , the ﬁrst order of Taylor series expansion of Eq.(106) at x t = x t +
= x ∗ is
[Equation]
Reorganizing Eq.(107), and applying Cauchy Inequality, we have
[Equation]
Combining with Eq.(51) and Eq.(108), we have
[Equation]
where C = 1 2 Lη 2 · B , B is deﬁned in Eq.(51).
In SGDM case, the angular update can be computed by
[Equation] [Equation]
115) where ( a t , b t , c t ) =
( || w t
|| 22 , (cid:104) w t , w t +1 (cid:105) ,
|| w t +1 || 22 ) .
According to the proof of theorem 2, when t is sufﬁciently large, ( a t , b t , c t ) will be close to the ﬁxed point of Eq.(66), ( a ∗ , b ∗ , c ∗ ) , where a ∗ =
c ∗ =
( (cid:114)
Lη λ (1 − α )(2 − λη 1+ α ) , b ∗ =
+ α − λ 1+ α a ∗ .
Then the ﬁrst order Taylor series expansion of Eq.(115) at ( a t = a ∗ , b t = b ∗ , c t = c ∗ ) is
[Equation]
[Equation] [Equation]
Now substituting ( a t , b t , c t ) with (˜ a t , ˜ b t , ˜ c t ) deﬁned in Eq.(75, 76,77, 78), Eq.(107) is rewritten as
[Equation]
[Equation]
where (˜ a ∗ , ˜ b ∗ , ˜ c ∗ ) is the ﬁxed point of Eq.(89).
Omit O ( λη ) , we have
[Equation]
where Y t , Y ∗ is deﬁned in Eq.(75, 89) respectively.
According to Eq.(100), the mean square error
E | ∆ t − (cid:113) 2 λη 1+ α | 2 can be bounded by
[Equation]
where C = (1 − α 2 )(1 − α ) 4 4 Lη 2 α· B , B is deﬁned in Eq.(101).
B. P ROOF OF C OROLLARY 3.
Proof of Corollary 3.1.
In SGD case, and Eq.(42), we have
[Equation]
) which means
[Equation]
On the other hand, we know that when η is divided by k , || w t
|| 22 should be divided by √ k to reach the new equilibrium condition, therefore we have
[Equation]
Since λη (cid:28) 1 , log(1 − 2 λη ) ≈ − 2 λη , thus
T > log( k ) 4 λη .
In SGDM case, by Eq.(87), we have
[Equation]
Similar to SGD case, we have
[Equation]
C E XPERIMENTS C. E XPERIMENTS ON S YTHETIC D ATA
In this section we apply experiments on sythetic data to prove our claim in section 4: theoremonly requires expected square norm of unit gradient is locally steady, the expectations do not need to remain unchanged across the whole training process.
The proof of theorem implies square norm of weight is determined by the following iterative map:
[Equation]
130) where λ, η ∈ (0 , 1) , L t denotes the square of unit gradient norm.
Hence we simulate x t with different type of { L t } ∞ t =1 .
Results in Figure 4 shows as long as the local variance of square norm of unit gradient is not too much, and expectation of L t changes smoothly, weight norm can quickly converge to its theoretical value base on expectation of square norm of unit gradient.
We also simulate SGDM case by following iteration map
[Equation] [Equation]
where A , X t , e is deﬁned as Eq.(62), (63), (64).
Simulation results is shown in Figure 5.
C. C OMPLEMENTARY IN M ULTI -S TAGE L EARNING R ATE S CHEDULE
In this section we present complementary results in Multi-Stage Learning Rate Schedule experiment.
The plots of weight norm (empirical and predicted values) in multi-learning rate stage is shown in
Figure 6.
We also present the test performance of resent50/maskrcnn on Imagenet/MSCOCO with multi-stage learning rate schedule mentioned in Section 5.2.
We only provide complementary results for reference only.
We do not intend to prove the advantages or disadvantages of rescaling strategy here, it is beyond the discussion of this paper .
[Table] [Table]
C. R ETHINKING L INEAR S CALING P RINCIPLE IN S PHERICAL M OTION D YNAMICS
In this section, we will discuss the effect of Linear Scaling Principle (LSP) under the view of SMD.
Linear Scaling Principle is proposed by Goyal et al.
(2017) to tune the learning rate η with batch size B by η ∝ B .
The intuition of LSP is if weights do not change too much within k iterations, then k iterations of SGD with learning rate η and minibatch size B (Eq.(132)) can be approximated by a single iteration of SGD with learning rate kη and minibatch size kB (Eq.(133):
[Equation]
Goyal et al.
(2017) shows that combining with gradual warmup, LSP can enlarge the batch size up to 8192(256 × 32) without severe degradation on ImageNet experiments.
LSP has been proven extremely effective in a wide range of applications.
However, from the perspective of SMD, the angular update mostly relies on the pre-deﬁned hyper-parameters, and it is hardly affected by batch size.
To clarify the connection between LSP and SMD, we explore the learning dynamics of DNN with different batch size by conducting extensive experiments with ResNet on ImageNet, the training settings rigorously follow Goyal et al.
(2017): momentum coefﬁcient is α = 10 − 4 ; WD coefﬁcient is λ = 10 − 4 ; Batch size is denoted by B ; learning rate is initialized as B 256 · 0 . 1 ; Total training epoch is 90 epoch, and learning rate is divided by 10 at 30 , 60 , 80 epoch respectively.
The results of experiments(Figure 7, 8) suggests that the assumption of LSP does not always hold in practice because of three reasons: ﬁrst, the approximate equivalence between a single iteration in large batch setting, and multiple iterations in small batch setting can only hold in pure SGD formulation, but momentum method is far more commonly used; Second, according Theorem 2, the enlargement ratio of angular update is only determined by the increase factor of learning rate.
Figure 7 shows in practice, the accumulated angular update (cid:93) ( w t , w t + k ) in small batch batch setting is much larger than angular update (cid:93) ( w t , w t +1 ) of a single iteration in larger batch setting when using Linear Scaling Principle; Third, even in pure SGD cases, the enlargement of angular update still relies on the increase of learning rate, and has no obvious connection to the enlargement of gradient’s norm when equilibrium condition is reached (see Figure 8).
In conclusion, though LSP usually works well in practical applications, SMD suggests we can ﬁnd more sophisticated and reasonable schemes to tune the learning rate when batch size increases.
C. S PHERICAL M OTION D YNAMICS WITH D IFFERENT N ETWORK S TRUCTURES
We also verify our theory on other commonly used network structures (MobileNet-V2 (Sandler et al., 2018), ShufﬂeNet-V2+ (Ma et al., 2018)) with standard training settings.
The results is shown in Figure 9.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]