A N E XPONENTIAL L EARNING R ATE S CHEDULE FOR D EEP L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Intriguing empirical evidence exists that deep learning can work well with exotic schedules for varying the learning rate.
This paper suggests that the phenomenon may be due to Batch Normalization or BN(Ioffe & Szegedy, 2015), which is ubiquitous and provides beneﬁts in optimization and generalization across all standard architectures.
The following new results are shown about BN with weight decay and momentum (in other words, the typical use case which was not considered in earlier theoretical analyses of stand-alone BN (Ioffe & Szegedy, 2015; Santurkar et al., 2018; Arora et al., 2018)
• Training can be done using SGD with momentum and an exponentially increasing learning rate schedule, i
e., learning rate increases by some (1 + α ) factor in every epoch for some α > 0 .
(Precise statement in the paper.)
To the best of our knowledge this is the ﬁrst time such a rate schedule has been successfully used, let alone for highly successful architectures.
As expected, such training rapidly blows up network weights, but the network stays wellbehaved due to normalization.
• Mathematical explanation of the success of the above rate schedule: a rigorous proof that it is equivalent to the standard setting of BN + SGD + Standard Rate Tuning + Weight Decay + Momentum.
This equivalence holds for other normalization layers as well, Group Normalization(Wu & He, 2018), Layer Normalization(Ba et al., 2016), Instance Norm(Ulyanov et al., 2016), etc.
• A worked-out toy example illustrating the above linkage of hyperparameters.
Using either weight decay or BN alone reaches global minimum, but convergence fails when both are used.
I NTRODUCTION
Batch Normalization (BN) offers signiﬁcant beneﬁts in optimization and generalization across architectures, and has become ubiquitous.
Usually best performance is attained by adding weight decay and momentum in addition to BN.
Usually weight decay is thought to improve generalization by controlling the norm of the parameters.
However, it is fallacious to try to separately think of optimization and generalization because we are dealing with a nonconvex objective with multiple optima.
Even slight changes to the training surely lead to a different trajectory in the loss landscape, potentially ending up at a different solution!
One needs trajectory analysis to have a hope of reasoning about the effects of such changes.
In the presence of BN and other normalization schemes, including GroupNorm, LayerNorm, and
InstanceNorm, the optimization objective is scale invariant to the parameters, which means rescaling parameters would not change the prediction, except the parameters that compute the output which do not have BN.
However, Hoffer et al.
(2018b) shows that ﬁxing the output layer randomly doesn’t harm the performance of the network.
So the trainable parameters satisfy scale invariance.(See more in Appendix E)
The current paper introduces new modes of analysis for such settings.
This rigorous analysis yields the surprising conclusion that the original learning rate (LR) schedule and weight decay(WD) can be folded into a new exponential schedule for learning rate: in each iteration multiplying it by (1 + α ) for some α > 0 that depends upon the momentum and weight decay rate.
Theorem 1.1 (Main, Informal) .
SGD on a scale-invariant objective with initial learning rate η , weight decay factor λ , and momentum factor γ is equivalent to SGD with momentum factor γ where at iteration t , the learning rate ˜ η t in the new exponential learning rate schedule is deﬁned as ˜ η t = α − 2 t − 1 η without weight decay( ˜ λ = 0 ) where α is a non-zero root of equation
[Equation]
Speciﬁcally, when momentum γ = 0 , the above schedule can be simpliﬁed as ˜ η t =
(1 − λη ) − 2 t − 1 η .
The above theorem requires that the product of learning rate and weight decay factor, λη , is small than (1 − √ γ ) 2 , which is almost always satisﬁed in practice.
The rigorous and most general version of above theorem is Theorem 2.12, which deals with multi-phase LR schedule, momentum and weight decay.
There are other recently discovered exotic LR schedules, e.g. Triangular LR schedule(Smith, 2017) and Cosine LR schedule(Loshchilov & Hutter, 2016), and our exponential LR schedule is an extreme example of LR schedules that become possible in presence of BN.
Such an exponential increase in learning rate seems absurd at ﬁrst sight and to the best of our knowledge, no deep learning success has been reported using such an idea before.
It does highlight the above-mentioned viewpoint that in deep learning, optimization and regularization are not easily separated.
Of course, the exponent trumps the effect of initial lr very fast (See Figure 4), which explains why training with BN and
WD is not sensitive to the scale of initialization, since with BN, tuning the scale of initialization is equivalent to tuning the initial LR η while ﬁxing the product of LR and WD, ηλ (See Lemma 2.7).
Note that it is customary in BN to switch to a lower LR upon reaching a plateau in the validation loss.
According to the analysis in the above theorem, this corresponds to an exponential growth with a smaller exponent, except for a transient effect when a correction term is needed for the two processes to be equivalent (see discussion around Theorem 2.12).
Thus the ﬁnal training algorithm is roughly as follows:
Start from a convenient LR like 0 .
1 , and grow it at an exponential rate with a suitable exponent.
When validation loss plateaus, switch to an exponential growth of LR with a lower exponent.
Repeat the procedure until the training loss saturates.
In Section 3, we demonstrate on a toy example how weight decay and normalization are inseparably involved in the optimization process.
With either weight decay or normalization alone, SGD will achieve zero training error.
But with both turned on, SGD fails to converge to global minimum.
In Section 4, we experimentally verify our theoretical ﬁndings on CNNs and ResNets.
We also construct better exponential LR schedules by incorporating the Cosine LR schedule on CIFAR10, which opens the possibility of even more general theory of rate schedule tuning towards better performance.
1. R ELATED W ORK
There have been other theoretical analyses of training models with scale-invariance.
(Cho & Lee, 2017) proposed to run Riemanian gradient descent on Grassmann manifold G (1 , n ) since the weight matrix is scaling invariant to the loss function.
observed that the effective stepsize is proportional to η w (cid:107) w t (cid:107) 2 .
(Arora et al., 2019) show the gradient is always perpendicular to the current parameter vector which has the effect that norm of each scale invariant parameter group increases monotonically, which has an auto-tuning effect.
(Wu et al., 2018) proposes a new adaptive learning rate schedule motivated by scale-invariance property of Weight Normalization.
Previous work for understanding Batch Normalization.
(Santurkar et al., 2018) suggested that the success of BNhas does not derive from reduction in Internal Covariate Shift, but by making landscape smoother.
(Kohler et al., 2018) essentially shows linear model with BN could achieve exponential convergence rate assuming gaussian inputs, but their analysis is for a variant of GD with an inner optimization loop rather than GD itself.
(Bjorck et al., 2018) observe that the higher learning rates enabled by BN empirically improves generalization.
(Arora et al., 2019) prove that with certain mild assumption, (S)GD with BN ﬁnds approximate ﬁrst order stationary point with any ﬁxed learning rate.
None of the above analyses incorporated weight decay, but (Zhang et al., 2019;
Hoffer et al., 2018a; van Laarhoven, 2017) argued qualitatively that weight decay makes parameters have smaller norms, and thus the effective learning rate, η w (cid:107) w t (cid:107) 2 is larger.
They described experiments showing this effect but didn’t have a closed form theoretical analysis like ours.
None of the above analyses deals with momentum rigorously.
1. P RELIMINARIES AND N OTATIONS
For batch B = { x i } Bi =1 , network parameter θ , we denote the network by f θ and the loss function at iteration t by L t ( f θ ) =
L ( f θ , B t ) .
When there’s no ambiguity, we also use L t ( θ ) for convenience.
We say a loss function L ( θ ) is scale invariant to its parameter θ is for any c ∈ R + , L ( θ ) =
L ( c θ ) .
In practice, the source of scale invariance is usually different types of normalization layers, including Batch Normalization (Ioffe & Szegedy, 2015), Group Normalization (Wu & He, 2018),
Layer Normalization (Ba et al., 2016), Instance Norm (Ulyanov et al., 2016), etc.
Implementations of SGD with Momentum/Nesterov comes with subtle variations in literature.
We adopt the variant from Sutskever et al.
(2013), also the default in PyTorch (Paszke et al., 2017).
Lregularization (a.k.a.
Weight Decay ) is another common trick used in deep learning.
Combining them together, we get the one of the mostly used optimization algorithms below.
Deﬁnition 1.2.
[SGD with Momentum and Weight Decay] At iteration t , with randomly sampled batch B t , update the parameters θ t and momentum v t as following:
[Equation]
where η t is the learning rate at epoch t , γ is the momentum coefﬁcient, and λ is the factor of weight decay.
Usually, v 0 is initialized to be 0 .
For ease of analysis, we will use the following equivalent of Deﬁnition 1.2.
[Equation]
where η − 1 and θ − 1 must be chosen in a way such that v
= θ 0 − θ − 1 η −is satisﬁed, e.g. when v 0 = 0 , θ − 1 = θ 0 and η − 1 could be arbitrary.
A key source of intuition is the following simple lemma about scale-invariant networks Arora et al.
The ﬁrst property ensures GD (with momentum)
always increases the norm of the weight.(See Lemma C.1 in Appendix C) and the second property says that the gradients are smaller for parameteres with larger norm, thus stabilizing the trajectory from diverging to inﬁnity.
Lemma 1.3 (Scale Invariance) .
If for any c ∈ R + , L ( θ ) =
L ( c θ ) , then (1).
(cid:104)∇ θ L, θ (cid:105)
= 0 ; (2).
∇
[Equation]
, for any
[Equation]
>
D ERIVING E XPONENTIAL L EARNING R ATE S CHEDULE
As a warm-up in Section 2.1 we show that if momentum is turned off then Fixed LR + Fixed WD can be translated to an equivalent Exponential LR.
In Section 2.2 we give a more general analysis on the equivalence between Fixed LR + Fixed WD + Fixed Momentum Factor and Exponential
LR + Fixed Momentum Factor .
While interesting, this still does completely apply to real-life deep learning where reaching full accuracy usually requires multiple phases in training where LR is ﬁxed within a phase and reduced by some factor from one phase to the next.
Section 2.3 shows how to interpret such a multi-phase LR schedule + WD + Momentum as a certain multi-phase exponential
LR
schedule with Momentum
2. R EPLACING WD BY E XPONENTIAL LR IN M OMENTUM -F REE SGD
We use notation of Section 1.2 and assume LR is ﬁxed over iterations, i.e. η t = η 0 , and γ (momentum factor) is set as 0 .
We also use λ to denote WD factor and θ 0 to denote the initial parameters.
The intuition should be clear from Lemma 1.3, which says that shrinking parameter weights by factor ρ (where ρ < 1 ) amounts to making the gradient ρ − 1 times larger without changing its direction.
Thus in order to restore the ratio between original parameter and its update (LR × Gradient), the easiest way would be scaling LR by ρ 2 .
This suggests that scaling the parameter θ by ρ at each step is equivalent to scaling the LR η by ρ − 2 .
To prove this formally we use the following formalism.
We’ll refer to the vector ( θ , η ) the state of a training algorithm and study how this evolves under various combinations of parameter changes.
We will think of each step in training as a mapping from one state to another.
Since mappings can be composed, any ﬁnite number of steps also correspond to a mapping.
The following are some basic mappings used in the proof.
Run GD with WD for a step:
[Equation]
Scale the parameter θ :
[Equation]
Scale the LR η : Π
[Equation] [Equation]
, η
[Equation]
θ , cη ) .
For example, when ρ = 1 , GD 1 t is vanilla GD update without WD, also abbreviated as GD t .
When ρ = 1 − λη 0 , GD 1 − λη 0 t is GD update with WD λ and LR η 0 .
Here L t is the loss function at iteration t , which is decided by the batch of the training samples B t in t th iteration.
Below is the main result of this subsection, showing our claim that GD + WD ⇔ GD+
Exp LR (when Momentum is zero).
It will be proved after a series of lemmas.
Theorem 2.1 (WD ⇔ Exp LR) .
For every ρ < 1 and positive integer t following holds:
[Equation]
With WD being λ , ρ is set as 1 − λη 0 and thus the scaling factor of LR per iteration is ρ − 2 =
(1 − λη 0 ) − 2 , except for the ﬁrst iteration it’s ρ − 1 = (1 − λη 0 ) − 1 .
We ﬁrst show how to write GD update with WD as a composition of above deﬁned basic maps.
Lemma 2.2.
[Equation]
Below we will deﬁne the proper notion of equivalence such that (1).
Π ρ 1 ∼ Π ρ − 2 2 , which implies
GD
[Equation]
∼
[Equation]
◦ GD
[Equation]
Π
[Equation]
1 2 ; (2) the equivalence is preserved under future GD updates.
We ﬁrst extend the equivalence between weights (same direction) to that between states, with additional requirement that the ratio between the size of GD update and that of parameter are the same among all equivalent states, which yields the notion of Equivalent Scaling .
Deﬁnition 2.3 (Equivalent States) .
( θ , η ) is equivalent to ( θ (cid:48) , η (cid:48) )
iff ∃ c > 0 , ( (cid:101) θ , (cid:101) η ) =
[Π c 1 ◦ Π c 2 2 ]( θ , η ) =
( c θ , c 2 η ) , which is also denoted by ( (cid:101) θ , (cid:101) η ) c ∼ ( θ , η ) .
Π c 1 ◦ Π c 2 2 is called Equivalent Scaling for all c > 0 .
The following lemma shows that equivalent scaling commutes with GD update with WD, implying that equivalence is preserved under GD update (Lemma 2.4).
This anchors the notion of equivalence — we could insert equivalent scaling anywhere in a sequence of basic maps(GD update,
LR/parameter scaling)
, without changing the ﬁnal network.
Lemma 2.4.
For any constant c, ρ > 0 and t ≥ 0 , GD ρt ◦ [Π c 1 ◦ Π c 2 2 ] =
[Π c 1 ◦ Π c 2 2 ] ◦ GD ρt .
In other words, ( θ , η ) c ∼ ( θ (cid:48) , η (cid:48) )
=
⇒ GD
[Equation]
GD
[Equation] [Equation]
[Equation]
Now we formally deﬁne equivalence relationship between maps using equivalent scalings.
Deﬁnition 2.5 (Equivalent Maps) .
Two maps F, G are equivalent iff ∃ c > 0 , F = Π c 1 ◦ Π c 2 2 ◦ G , which is also denoted by F c ∼ G .
Proof of Theorem 2.1.
By Lemma 2.2,, GD ρt ρ ∼ Π ρ − 1◦ GD t ◦ Π ρ − 1 2 .
By Lemma 2.4, GD update preserves map equivalence, i.e. F c ∼ G ⇒ GD ρt ◦ F c ∼ GD ρt ◦ G, ∀ c, ρ > 0 .
Thus,
[Equation]
2. R EPLACING WD BY E XPONENTIAL LR: C ASE OF CONSTANT LR WITH MOMENTUM
In this subsection the setting is the same to that in Subsection 2.1 except that the momentum factor is γ instead of 0.
Suppose the initial momentum is v 0 , we set θ − 1 = θ 0 − v 0 η .
Presence of momentum requires representing the state of the algorithm with four coordinates, ( θ , η, θ (cid:48) , η (cid:48) ) , which stand respectively for the current parameters/LR and the buffered parameters/LR (from last iteration) respectively.
Similarly
[Equation]
we deﬁne the following basic maps and equivalence relationships.
Run GD with WD for a step:
[Equation]
Scale Current parameter θ
[Equation]
Scale Current LR η :
[Equation]
Scale Buffered parameter θ
[Equation]
[Equation]
Scale Buffered parameter η (cid:48)
[Equation]
Deﬁnition 2.6 (Equivalent States) .
( θ , η, θ (cid:48) , η (cid:48) ) is equivalent to ( (cid:101) θ , (cid:101) η, (cid:101) θ (cid:48) , (cid:101) η (cid:48) )
iff ∃ c > 0 , ( θ , η, θ (cid:48) , η
[Equation]
) =
[Equation] [Equation]
[Equation] [Equation]
η (cid:48) ) , which is also denoted by ( θ , η, θ (cid:48) , η (cid:48) )
c ∼ ( (cid:101) θ , (cid:101) η, (cid:101) θ (cid:48) , (cid:101) η (cid:48) ) .
We call Π c 1 ◦ Π c 2 2 ◦ Π c 3 ◦ Π c 2 4 Equivalent Scalings for all c > 0 .
Again by expanding the deﬁnition, we show equivalent scalings commute with GD update.
Lemma
[Equation]
Similarly, we can rewrite GD ρt as a composition of vanilla GD update and other scalings by expanding the deﬁnition, when the current and buffered LR are the same in the input of GD ρt .
Lemma 2.8.
For any input ( θ , η, θ (cid:48) , η ) , if α > 0 is a root of α + γα − 1 = ρ + γ , then
GD
[Equation] [Equation]
[Equation] [Equation]
θ , η, θ
[Equation]
, η ) .
In other words,
[Equation]
Though looking complicated, the RHS of Equation 5 is actually the desired Π α −
[Equation]
conjugated with some scaling on momentum part Π α 3 ◦
Π α 4 , and Π α
[Equation]
1◦ Π α
[Equation]
in the current update cancels with the Π α 3 ◦
Π α 4 in the next update.
Now we are ready to show the equivalence between
WD and Exp LR schedule when momentum is turned on for both.
Theorem 2.9 (GD + WD ⇔ GD+
Exp LR; With Momentum) .
The following deﬁned two sequences of parameters , { θ t } ∞ t =0 and { (cid:101) θ t } ∞ t =0 , satisfy (cid:101) θ t = α t θ t , thus they correspond to the same networks in function space, i.e. f θ t = f (cid:101) θ t , ∀ t ∈ N , given (cid:101) θ
= θ 0 , (cid:101) θ − 1 = θ − 1 α , and (cid:101) η t = η 0 α − 2 t − 1 .
[Equation]
where α is a positive root of equation x 2 − (1 + γ − λη 0 ) x
+ γ = 0 , which is always smaller than 1(See Appendix B.1).
When γ = 0 , α = 1 − λη 0 is the unique non-zero solution.
Remark 2.10.
Above we implicitly assume that λη 0 ≤ (1 − √ γ ) 2 such that the roots are real and this is always true in practice.
For instance of standard hyper-parameters where γ = 0 .
9 , η 0 = 0 .
1 , λ = 0 .
[Equation]
(cid:28) 1 .
Proof.
Note that ( (cid:101) θ 0 ,
[Equation]
η
[Equation]
, (cid:101)
[Equation]
, (cid:101)
[Equation]
= (cid:104)
[Equation]
◦
[Equation]
Π
[Equation]
(cid:105) ( θ 0 , η 0 , θ 0 , η 0 ) , it sufﬁces to show that
[Equation]
which follows immediately from Lemma 2.7 and Lemma 2.8 by induction.
2. R EPLACING WD BY E XPONENTIAL LR: C ASE OF MULTIPLE LR PHASES
Usual practice in deep learning shows that reaching full training accuracy requires reducing the learning rate a few times.
Deﬁnition 2.11.
Step Decay is the (standard) learning rate schedule, where training has K phases
I = 0 , 1 , . .
, K − 1 , where phase I starts at iteration T
I
( T 0 = 0 ), and all iterations in phase I use a ﬁxed learning rate
of η ∗
I
The algorithm state in Section 2.2, consists of 4 components including buffered and current LR.
When LR changes, the buffered and current LR are not equal, and thus Lemma 2.8 cannot be applied any more.
In this section we show how to ﬁx this issue by adding extra momentum correction.
In detail, we show the below deﬁned Exp LR schedule leads the same trajectory of networks in function space, with one-time momentum correction at the start of each phase.
We empirically ﬁnd on CIFAR10 that ignoring the correction term does not change performance much.
Theorem 2.12 (Tapered-Exponential LR Schedule) .
There exists a way to correct the momentum only at the ﬁrst iteration of each phase, such that the following Tapered-Exponential LR schedule (TEXP) { (cid:101) η t } with momentum factor γ and no WD, leads the same sequence networks in function space as that of Step Decay LR schedule(Deﬁnition 2.11) with momentum factor γ and WD λ .
[Equation]
where α
[Equation]
The analysis in previous subsection give the equivalence within each phase, where the same LR is used throughout the phase.
To deal with the difference between buffered LR and current LR when entering new phases, the idea is to pretend η t − 1 =
η t and θ t − 1 becomes whatever it needs to maintain θ t − θ t − 1 η t −such that we can again apply Lemma 2.8, which requires the current LR of the input state is equal to its buffered LR.
Because scaling α in RHS of Equation 5 is different in different phases, so unlike what happens within each phase, they don’t cancel with each other at phase transitions, thus remaining as a correction of the momentum.
The proofs are delayed to
Appendix B, where we proves a more general statement allowing phase-dependent WD, { λ I } K − 1 I =0 .
Alternative interpretation of Step Decay to exponential LR schedule: Below we present a new
LR schedule, TEXP++ , which is exactly equivalent to Step Decay without the need of one-time correction of momentum when entering each phase.
We further show in Appendix B.1 that when translating from Step Decay , the TEXP++ we get is very close to the original TEXP (Equation 9), i.e. the ratio between the LR growth per round, (cid:101)
η t +1 (cid:101) η t / (cid:101) η (cid:48)
t +1 (cid:101) η (cid:48)
t converges to 1 exponentially each phase.
For example, with WD 0.0005, max LR 0.1, momentum factor 0.9, the ratio is within 1 ± 0 .
0015 ∗ 0 .
9 t − T I , meaning TEXP and TEXP++ are very close for Step Decay with standard hyperparameters.
Theorem 2.13.
The following two sequences of parameters , { θ t } ∞ t =0 and { (cid:101) θ t } ∞ t =0 , deﬁne the same sequence of network functions, i.e. f θ t = f (cid:101) θ t , ∀ t ∈ N , given the initial conditions, (cid:101) θ
0 =
P 0 θ 0 , (cid:101) θ −
[Equation]
=
[Equation]
[Equation]
, for
[Equation]
= 1 , 2 ,
[Equation]
) , for t = 1 , 2 , . . . , where (cid:101) η t =
P t P t +1 η t , P t = t (cid:81)
[Equation]
[Equation]
and α t recursively deﬁned as
[Equation]
The LR schedule { (cid:101) η t } ∞ t =0 is called Tapered Exponential ++ , or TEXP++ .
E XAMPLE ILLUSTRATING INTERPLAY OF WD AND BN
The paper so far has shown that effects of different hyperparameters in training are not easily separated, since their combined effect on the trajectory is complicated.
We give a simple example to illustrate this, where convergence is guaranteed if we use either BatchNorm or weight decay in isolation, but convergence fails if both are used.
(Momentum is turned off for clarity of presentation)
Setting: Suppose we are ﬁne-tuning the last linear layer of the network, where the input of the last layer is assumed to follow a standard Gaussian distribution N (0 , I m ) , where m is the input dimension of last layer.
We also assume this is a binary classiﬁcation task with logistic loss, l ( u, y )
=
ln(1 + exp( − uy )) , where label y ∈ {− 1 , 1 } and u ∈ R is the output of the neural network.
For simplicity we assume the the input of the last layer are already separable, and
w.l.o.g.
we assume the label is equal to the sign of the ﬁrst coordinate of x ∈ R m , namely sign ( x 1 ) .
Thus the training loss and training error are simply
[Equation]
Case 1: WD alone: Since both the above function and L2 regularization are convex w ,vanilla
SGD with suitably small learning rate could get arbitrarily close to the global minimum for this regularized objective, which has 100% training accuracy.
Case 2: BN alone: Add a BN layer after the linear layer(here we use global batch statistics for simplicity), and ﬁx scalar and bias term to 1 and 0.
The objective becomes
[Equation]
From Appendix B.6, there’s some constant C , such that ∀ w ∈ R m with constant probability, (cid:107)∇ w L BN ( w , x ) (cid:107) ≥ C
[Equation]
w (cid:107) . By Pythagorean Theorem, (cid:107) w t +1 (cid:107)
[Equation]
w t (cid:107)
[Equation]
+ η
[Equation]
w L BN ( w t , x ) (cid:107) 2 ) 2 ≥ (cid:107) w t (cid:107) 4 + 2 η 2 (cid:107) w t (cid:107) 2 (cid:107)∇ w L BN ( w t , x ) (cid:107) 2 .
As a result, for any ﬁxed learning rate , (cid:107) w t +1 (cid:107) 4 ≥ 2 (cid:80) ti =1 η 2 (cid:107) w (cid:107) 2 (cid:107)∇ w L BN ( w i , x ) (cid:107) 2 grows at least linearly with high probability.
Following the analysis of Arora et al.
(2019), this is like reducing the effective learning rate, and when (cid:107) w t (cid:107) is large enough, the effective learning rate is small enough, and thus SGD can ﬁnd the local minimum, which is the unique global minimum.
Case 3: Both BN and WD: Suppose WD factor is λ , LR is η , we have the following theorem:
Theorem 3.1.
[Nonconvergence] Starting from iteration any T 0 , with probability 1 − δ over the randomness of samples, the training error will be larger than επ at least once for the following consecutive
[Equation]
Sketch.
(See full proof in Appendix B.)
The high level idea of this proof is that if the test error is low, the weight is restricted in a small cone around the global minimum, and thus the amount of the gradient update is bounded by the size of the cone.
In this case, the growth of the norm of the weight by Pythagorean Theorem is not large enough to cancel the shrinkage brought by weight decay.
As a result, the norm of the weight converges to 0 geometrically.
Again we need to use the lower bound for size of the gradient, that (cid:107)∇ w L t (cid:107) =
Θ( η √ m (cid:107) w t (cid:107) ) holds with constant probability.
Thus the size of the gradient will grow along with the shrinkage of (cid:107) w t (cid:107) until they’re comparable, forcing the weight to leave the cone in next iteration.
E XPERIMENTS
The translation to exponential LR schedule is exact except for one-time momentum correction term entering new phases.
The experiments explore the effect of this correction term.
The Tapered
Exponential(TEXP) LR schedule contains two parts when entering a new phase I: an instant LR decay ( η I η I − 1 ) and an adjustment of the growth factor ( α ∗ I − 1 → α ∗ I ).
The ﬁrst part is relative small compared to the huge exponential growing.
Thus a natural question arises: Can we simplify TEXP
LR schedule by dropping the part of instant LR decay?
Also, previously we have only veriﬁed our equivalence theorem in Step Decay LR schedules.
But it’s not sure how would the Exponential LR schedule behave on more rapid time-varying LR schedules such as Cosine LR schedule.
Settings: We train PreResNet32 on CIFAR10.
The initial learning rate is 0.1 and the momentum is 0.9 in all settings.
We ﬁx all the scalar and bias of BN, because otherwise they together with the following conv layer grow exponentially, sometimes exceeding the range of Float32 when trained with large growth rate for a long time.
We ﬁx the parameters in the last fully connected layer for scale invariance of the objective.
4. T HE BENEFIT OF INSTANT LR DECAY
We tried the following LR schedule (we call it TEXP-- ).
Interestingly, up to correction of momentum when entering a new phase, this schedule is equivalent to a constant LR schedule, but with the weight decay coefﬁcient reduced correspondingly at the start of each phase.
(See Theorem B.2 and Figure 6)
[Equation]
where
[Equation]
4. B ETTER E XPONENTIAL LR S CHEDULE WITH C OSINE LR
We applied the TEXP LR schedule (Theorem 2.12) on the Cosine LR schedule (Loshchilov &
Hutter, 2016), where the learning rate changes every epoch, and thus correction terms cannot be ignored.
The LR at epoch t ≤ T is deﬁned as: η t = η 0 1+cos( tT π ).
Our experiments show this hybrid schedule with Cosine LR performs better on CIFAR10 than Step Decay, but this ﬁnding needs to be veriﬁed on other datasets.
C ONCLUSIONS
The paper shows rigorously how BN allows a host of very exotic learning rate schedules in deep learning and veriﬁes these effects in experiments.
The lr increases exponentially in almost every iteration during training.
The exponential increase derives from use of weight decay but the precise expression involves momentum as well.
We suggest that the efﬁcacy of this rule may be hard to explain with canonical frameworks in optimization.
Our analyses of BN is a substantial improvement over earlier theoretical analyses, since it accounts for weight decay and momentum, which are always combined in practice.
Our tantalising experiments with a hybrid of exponential and cosine rates suggest that more surprises may lie out there.
Our theoretical analysis of interrelatedness of hyperparameters could also lead to faster hyperparameter search.
A V IEWING EXP LR VIA C ANONICAL O PTIMIZATION F RAMEWORK
This section tries to explain why the efﬁcacy of exponential LR in deep learning is mysterious to us, at least as viewed in the canonical framework of optimization theory.
Canonical framework for analysing 1st order methods This focuses on proving that each —or most—steps of GD noticeably reduce the objective, by relying on some assumption about the spectrum norm of the hessian of the loss, and most frequently, the smoothness , denoted by β .
Speciﬁcally, for GD update θ t +1 =
θ t − η ∇ L ( θ t ) , we have
[Equation]
When β < 2 η , the ﬁrst order term is larger than the second order one, guaranteeing the loss value decreases.
Since the analysis framework treats the loss as a black box (apart from the assumed bounds on the derivative norms), and the loss is non-convex, the best one can hope for is to prove speedy convergence to a stationary point (where gradient is close to 0 ).
An increasing body of work proves such results.
Now we turn to difﬁculties in understanding the exponential LR in context of the above framework and with scale-invariance in the network.
Since loss is same for θ and c · θ for all c > 0 a simple calculation shows that along any straight line through the origin, smoothness is a decreasing function of c , and is very high close to origin.
(Note: it is also possible to one can show the following related fact:
In any ball containing the origin, the loss is nonconvex.)
Thus if one were trying to apply the canonical framework to argue convergence to a stationary point, the natural idea would be to try to grow the norm of the parameters until smoothness drops enough that the above-
mentioned Canonical Framework starts to apply.
Arora et al.
(2019) showed this happens in GD with ﬁxed LR (WD turned off), and furthermore the resulting convergence rate to stationary point is asymptotically similar to analyses of nonconvex optimization with learning rate set as in the Canonical framework .
Santurkar et al.
(2018) observed similar phenomenon in experiments, which they described as a smoothening of the objective due to BN. 2.
The Canonical Framework can be thought of as a discretization of continuous gradient descent (i.e., gradient ﬂow): in principle it is possible to use arbitrarily small learning rate, but one uses ﬁnite learning rate merely to keep the number of iterations small.
The discrete process approximates the continuous process due to smoothness being small.
In case of gradient ﬂow with weight decay (equivalently, with exponential LR schedule) the discrete process cannot track the continuous process for very long, which suggests that any explanation of the beneﬁts of exponential LR may need to rely on discrete process being somehow better.
The reason being that for gradient ﬂow one can decouple the speed of the θ t into the tangential and the radial components, where the former one has no effect on the norm and the latter one has no effect on the objective but scales the tangential gradient exponentially.
Thus the Gradient Flow with WD gives exactly the same trajectory as vanilla Gradient Flow does, excepting a exponential reparametrization with respect to time t . 3.
It can be shown that if the local smoothness is upperbounded by 2 η (as stipulated in Canonical Framework ) during a sequence θ t ( t = 1 , 2 , . . . )
of GD updates with WD and constant LR
then such sequence satisﬁes θ t → 0 .
This contrasts with the usual experimental observation that θ t stays bounded away from 0 .
One should thus conclude that in practice, with constant LR and WD, smoothness doesn’t always stay small (unlike the above analyses where WD is turned off).
B O MITTED P ROOFS B. O MITTED P ROOF IN S ECTION
Lemma B.1 (Some Facts about Equation 1) .
Suppose z 1 , z 2 ( z 1 ≥ z 2 ) are the two real roots of the the following equation, we have
[Equation]
z 1 = 1+ γ − λη + √ (1 − γ ) 2 − 2(1+ γ ) λη + λ 2 η 2, z 2 = 1+ γ − λη − √ (1 − γ ) 2 − 2(1+ γ ) λη + λ 2 η 22.
z 1 , z 2 are real ⇐⇒ λη ≤ (1 − √ γ ) 2 ; 3.
z 1 z 2 = γ, z 1 + z
=
(1 + γ − λη ) ; 4.
γ ≤ z 2 ≤ z 1 ≤ 1 ; 5.
Let t = λη 1 − γ , we have z 1 ≥ 11+ t ≥ 1 − t = 1 − λη 1 − γ .
if we view z 1 ( λη ) , z 2 ( λη ) as functions of λη , then z 1 ( λη ) is monotone decreasing, z 2 ( η ) is monotone increasing.
Proof.
Let f ( x ) =
x 2 − (1 + γ − λη ) x + γ , we have f (1) =
f ( γ ) = λη ≥ 0 .
Note the minimum of f is taken at x = 1+ γ − λη∈ [0 , 1] , the both roots of f ( x ) = 0 must lie between 0 and 1 , if exists.
[Equation]
Note that ( z 1 − z 2 )
= ( z 1 + z 2 ) 2 − 4 z
z
=
(1 + γ − λη ) 2 − 4 γ is monotone decreasing, since z 1 ( λη ) +
z 2 ( λη ) is constant, z 1 ( λη ) ≥ z 2 ( λη ) , z 1 ( λη ) must be decreasing and z 2 ( λη ) must be increasing
B. O MITTED PROOFS IN S ECTION 2.
Proof of Lemma 2.2.
For any ( θ , η ) , we have
[Equation]
Proof of Lemma 2.4.
For any ( θ , η ) , we have
[Equation]
B. O MITTED PROOFS IN S ECTION 2.
Proof of Lemma 2.7.
For any input ( θ , η, θ (cid:48) , η (cid:48) ) , it’s easy to check both composed maps have the same outputs on the 2,3,4th coordinates, namely ( c 2 η, c θ , c 2 η (cid:48) ) .
For the ﬁrst coordinate, we have
[Equation] [Equation]
[Equation] [Equation]
Invariance, Lemma 1.
Proof of Lemma 2.8.
For any input ( θ , η, θ (cid:48) , η (cid:48) )
[Equation]
it’s easy to check both composed maps have the same outputs on the 2,3,4th coordinates, namely ( η, θ , η ) .
For the ﬁrst coordinate
[Equation]
we have
[Equation]
B. O MITTED PROOFS OF T HEOREM 2.
In this subsection we will prove a stronger version of Theorem 2.12(restated below), allowing the
WD, λ I changing each phase.
Theorem B.2 (A stronger version of Theorem 2.12) .
There exists a way to correct the momentum only at the ﬁrst iteration of each phase, such that the following Tapered-Exponential LR schedule (TEXP) { (cid:101) η t } with momentum factor γ and no WD, leads the same sequence networks in function space compared to that of Step Decay LR schedule(Deﬁnition 2.11) with momentum factor γ and phase-dependent WD λ ∗ I in phase I , where phase I lasts from iteration T I to iteration T
I +1 , T 0 = 0 .
[Equation]
where α ∗ I = 1+ γ
[Equation] [Equation]
Towards proving Theorem 2.12, we need the following lemma which holds by expanding the deﬁnition, and we omit its proof.
Lemma B.3 (Canonicalization) .
We deﬁne the Canonicalization map as N ( θ , η, θ (cid:48) , η (cid:48) )
= ( θ , η, θ − η η (cid:48) ( θ − θ (cid:48) ) , η ) , and it holds that
GD ρt ◦ N = GD ρt , ∀ ρ > 0 , t ≥ 0 .
N ◦ (cid:104)
Π c 1 ◦ Π c 2 2 ◦ Π c 3 ◦ Π c 2 4 (cid:105) =
(cid:104)
Π c 1 ◦ Π c 2 2 ◦ Π c 3 ◦ Π c 2 4 (cid:105) ◦ N , ∀ c > 0 .
Similar to the case of momentum-free SGD, we deﬁne the notion of equivalent map below
Deﬁnition B.4 (Equivalent Maps) .
For two maps F and G , we say F is equivalent to G iff ∃ c > 0 ,
[Equation]
=
[Equation]
(cid:105) ◦ G , which is also denoted by F c ∼ G .
Note that for any ( θ , η, θ (cid:48) , η (cid:48) ) , [ N ( θ , η, θ (cid:48) , η (cid:48) )]
2 =
[ N ( θ , η, θ (cid:48) , η (cid:48) )] 4 .
Thus as a direct consequence of Lemma 2.8, the following lemma holds.
[Equation]
Proof of Theorem2.12.
Starting with initial state ( θ 0 , η 0 , θ − 1 , η − 1 ) where η − 1 = η 0 and a given LR schedule { η t } t ≥ 0 , the parameters generated by GD with WD and momentum satisﬁes the following relationship:
[Equation]
Deﬁne b (cid:13) t = a F t = F
[Equation]
◦ F
[Equation]
◦ . . .
◦ F a , for a ≤ b .
By Lemma B.3 and Lemma B.5, letting α t be the root of x 2 − ( γ + 1 − η t − 1 λ t
[Equation]
1 ) x
+ γ = 0 , we have
[Equation]
where
[Equation]
i =
α i ∼ is because of Lemma B.5, and H t is deﬁned as
[Equation]
Since the canonicalization map N only changes the momentum part of the state, it’s easy to check that H t doesn’t touch the current parameter θ and the current LR η .
Thus H t only changes the momentum part of the input state.
Now we claim that H t ◦ GD t − 1 =
GD t − 1 whenever η t =
η t − 1 .
This is because when η t =
η t − 1 , α t = α t +1 , thus H t ◦ GD t − 1 =
GD t − 1 .
In detail,
[Equation]
where ∗ = is because GD update GD t sets η (cid:48) the same as η , and thus ensures the input of N has the same momentum factor in buffer as its current momentum factor, which makes N an identity map.
Thus we could rewrite Equation 10 with a “sloppy”version of H t , H (cid:48) t =
[Equation]
=
[Equation] [Equation]
Now we construct the desired sequence of parameters achieved by using the Tapered Exp LR schedule 9 and the additional one-time momentum correction per phase.
Let ( (cid:101) θ 0 , (cid:101) η 0 , (cid:101) θ − 1 , (cid:101) η − 1 ) =
( θ 0 , η 0 , θ − 1 , η 0 ) , and
[Equation]
we claim { (cid:101) θ t } t =0 is the desired sequence of parameters.
We’ve already shown that θ t ∼ (cid:101) θ t , ∀ t .
Clearly { (cid:101) θ t } t =0 is generated using only vanilla GD, scaling LR and modifying the momentum part of the state.
When t (cid:54) =
T I for any I , η t =
η t − 1 and thus H (cid:48) t
= Id .
Thus the modiﬁcation on the momentum could only happen at T I
( I ≥ 0) .
Also it’s easy to check that α t = α ∗
I , if
T I + 1 ≤ t ≤ T
I +1 .
B. O MITTED PROOFS OF T HEOREM 2.
Theorem B.6.
The following two sequences of parameters , { θ t } ∞ t =0 and { (cid:101) θ t } ∞ t =0 , deﬁne the same sequence of network functions, i.e. f θ t = f (cid:101) θ t , ∀ t ∈ N , given the initial conditions, (cid:101) θ
0 =
P 0 θ 0 , (cid:101) θ − 1 =
P − 1 θ − 1 .
[Equation]
where (cid:101) η t =
P t P t +1 η t , P t = t
[Equation]
i =
[Equation]
− 1 i , ∀ t ≥ − 1 and α t recursively deﬁned as
[Equation]
12) needs to be always positive.
Here α 0 , α − 1 are free parameters.
Different choice of α 0 , α − 1 would lead to different trajectory for { (cid:101) θ t } , but the equality that (cid:101) θ t =
P t θ t is always satisﬁed.
If the initial condition is given via v 0 , then it’s also free to choose η − 1 , θ − 1 , as long as θ 0 − θ
[Equation]
1 η
[Equation] [Equation]
Proof of Theorem 2.13.
We will prove by induction.
By assumption S ( t ) : P t θ t = (cid:101) θ t for t = − 1 , 0 .
Now we will show that S ( t ) = ⇒ S ( t + 1)
[Equation]
t
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
Rescaling
[Equation]
Simplfying
[Equation]
η
[Equation]
To conclude that P t θ t = (cid:101) θ t , it sufﬁces to show that the coefﬁcients before (cid:101) θ t − 1 is the same to that in (2) .
In other words, we need to show
[Equation]
which is equivalent to the deﬁnition of α t , (Equation 12).
Lemma B.7 (Sufﬁcient Conditions for positivity of α t ) .
Let λ max = max t λ t , η max = max t η t .
Deﬁne z min is the larger root of the equation x 2 − (1 + γ − λ max η max ) x
+ γ = 0 .
To guarantee the existence of z max we also assume η max λ max ≤ (1 − √ γ ) 2 .
Then we have
[Equation]
Proof.
We will prove the above theorem with a strengthened induction —
[Equation]
Since α 0 = 1 , S (0) is obviously true.
Now suppose S ( t ) is true for some t ∈ N , we will prove
S ( t + 1) .
First, since 0 < α t ≤ 1 , α t +1 = − η t λ t + 1 + η t η t − 1 γ (1 − α − 1 t ) ≤ 1 .
Again by Equation 12, we have
[Equation]
which shows α t +1 ≥ z min .
Here the last step is by deﬁnition of z min .
Because of α t +1 ≥ z min , we have
[Equation]
Now we are ready to give the formal statement about the closeness of Equation 9 and the reduced
LR schedule by Theorem 2.13.
Theorem B.8.
Given a Step Decay LR schedule with { T I } K − 1 I =0 , { η ∗ I } K − 1 I =0 , { λ ∗ I } K − 1 I =0 , the TEXP++
LR schedule in Theorem 2.13 is the following( α 0 =
α − 1 = 1 , T 0 = 0 ):
[Equation]
It’s the same as the TEXP LR schedule( { ˜ η t } ) in Theorem 2.12 throughout each phase I , in the sense that
[Equation]
where z min is the larger root of x 2 − (1 + γ − λ max η max ) x + γ = 0 .
In Appendix B, we show that z − 1 min ≤ 1 + η max λ max 1 − γ .
When λ max η max is small compared to 1 − γ , which is usually the case in practice, one could approximate z min by 1.
For example, when γ = 0 .
9 , λ max = 0 .
0005 , η max = 0 .
1 , the above upper bound becomes
[Equation]
Proof of Theorem B.8.
Assuming z 1 I and z 2 I ( z 1 I ≥ z 2 I ) are the roots of Equation 1 with η = η I and λ
= λ
I , we have γ ≤ z 2 I (cid:48) ≤ √ γ ≤ z min ≤ z 1 I ≤ 1 , ∀
I, I (cid:48) ∈
[ K − 1] by Lemma B.1.
We can rewrite the recursion in Theorem 2.13 as the following:
[Equation] [Equation]
In other words, we have
[Equation]
By Lemma B.7, we have α t ≥ z min , ∀ t ≥ 0 .
Thus | α
[Equation]
γ ) 2 | α t − 1 z 1 I | , which means α t geometrically converges to its stable ﬁxed point z 1 I .
and (cid:101) η t − 1 (cid:101) η t =
( z 1 I ) 2 .
Since that z min ≤ α t ≤ 1 , z min ≤ z 1 I ≤ 1 , we have | α TI z 1 I − 1 | ≤ 1 − z min z min = λ max η max 1 − γ ≤ 1 , and thus | α t
z
I − 1 | ≤ λ max η max 1 − γ ( γ z 2 min ) t − T I − 1 ≤ 1 , ∀ T I + 1 ≤ t ≤ T
I +1 .
Note that α ∗ I =
z
I , ˆ η t − 1 ˆ η t = α t α t +1 By deﬁnition of TEXP and TEXP++, we have
[Equation]
Thus we have when t =
T I , (cid:12)(cid:12)(cid:12)
[Equation]
ˆ η t −ˆ η t (cid:30) (cid:101)
η
[Equation]
(cid:101) η t −
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12) ≤ (cid:12)(cid:12)(cid:12) (cid:12)
[Equation]
(cid:12)(cid:12)(cid:12) (cid:12) ≤ (cid:12)(cid:12)(cid:12) (cid:12) α T
I +1 z
I − 1 (cid:12)(cid:12)(cid:12) (cid:12) + (cid:12)(cid:12)(cid:12) (cid:12) α T
I z
I −− 1 (cid:12)(cid:12)(cid:12) (cid:12) + (cid:12)(cid:12)(cid:12) (cid:12) α T
I +1 z
I − 1 (cid:12)(cid:12)(cid:12) (cid:12) (cid:12)(cid:12)(cid:12) (cid:12) α T
I z
I −− 1 (cid:12)(cid:12)(cid:12) (cid:12) ≤
[Equation]
λ max η max 1 − γ .
When T I + 1 ≤ t ≤ T I +1 , we have (cid:12)(cid:12)(cid:12)(cid:12)
ˆ η t −ˆ η t (cid:30)
[Equation]
η
[Equation]
−(cid:101)
[Equation]
= (cid:12)(cid:12)(cid:12)(cid:12)
α
[Equation]
+1 z
[Equation]
1 α
[Equation]
z
[Equation]
1 − 1 (cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)
α t +1 z 1 I − 1 − 1 (cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)
α t z 1 I − 1 − 1 (cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12) α t +1 z 1 I − 1 − 1 (cid:12)(cid:12)(cid:12)(cid:12) (cid:12)(cid:12)(cid:12)(cid:12)
α t z 1 I − 1 − 1 (cid:12)(cid:12)(cid:12)(cid:12) ≤ 3 λ
[Equation]
η max 1 − γ ( γ z 2 min ) t − T
I − 1 .
Thus we conclude ∀
I ∈
[ K − 1] , T I + 1 ≤ t ≤ T I +1 , we have (cid:12)(cid:12)(cid:12)(cid:12)
ˆ η t −ˆ η t (cid:30) (cid:101)
η t −(cid:101) η t − 1 (cid:12)(cid:12)(cid:12)(cid:12) ≤ 3 λ max η max 1 − γ (cid:18) γ z
[Equation]
min (cid:19)
[Equation]
−
[Equation]
−≤
[Equation]
λ
[Equation]
η max 1 − γ
[Equation]
γ )
[Equation]
B. O MITTED P ROOFS IN S ECTION
We will need the following lemma when lower bounding the norm of the stochastic gradient.
Lemma B.9 (Concentration of Chi-Square) .
Suppose X 1 , . . .
, X k i.i.d.
∼ N (0 , 1) , then
[Equation]
Proof.
This Chernoff-bound based proof is a special case of Dasgupta & Gupta (2003).
[Equation]
The last equality uses the fact that E (cid:2) tX
i (cid:3) = 1 √ 1 − 2 t for t < 12 .
The proof is completed by taking t = 1 − β 2 β .
Setting for Theorem B.6: Suppose WD factor is λ , LR is η , the width of the last layer is m ≥ 2 ,
Now the SGD updates have the form
[Equation]
(1 − λη ) w t − η y t 1 + exp( x (cid:62) t w t (cid:107) w t (cid:107) y t )
Π ⊥ w t x t (cid:107) w t (cid:107) , where x t i.i.d.
∼ N (0 , I m ) , y t
= sign ( x t, 1 ) , and Π ⊥ w t
=
I − w t w (cid:62) t (cid:107) w t (cid:107) 2 .
Proof of Theorem B.6.
Step 1: We will use ˆ w to denote w (cid:107) w (cid:107) and ∠ uw to arccos(ˆ u (cid:62) ˆ w ) .
Note that training error ≤ επ is equivalent to ∠ e 1 w t < ε .
Let T 1 =
[Equation]
, and T 2 = 6 ln 1 δ .
Thus if we assume the training error is smaller than ε from iteration T 0 to T 0 + T 1 + T 2 , then by spherical triangle inequality, ∠ w t w t (cid:48) ≤ ∠ e 1 w t (cid:48)
+ ∠ e 1 w t = 2 ε , for T 0 ≤ t, t (cid:48)
≤ T
+ T 1 + T 2 .
Now let’s deﬁne w (cid:48) t
= (1 − ηλ ) w t and for any vector w , and we have the following two relationships:
[Equation]
(cid:107) w t +1 (cid:107) ≤ (cid:107) w ‘ t
[Equation]
cos2 ε [Equation] The second property is because by Lemma 1.3, ( w t +1 − w (cid:48) t ) ⊥ w (cid:48) t and by assumption of small error, ∠ w t +1 w (cid:48)
t ≤ 2 ε .
Therefore
[Equation]
[Equation]
In other word, (cid:107) w T 0 + T 1 (cid:107) 2 ≤ η √ m 32 ε .
Since (cid:107) w T 0 + t (cid:107) is monotone decreasing, (cid:107) w T 0 + t (cid:107) 2 ≤ η √ m 32 ε holds for any t =
T 1 , . .
, T 1 + T 2 .
Step 2: We show that the norm of the stochastic gradient is lower bounded with constant probability.
Note that x (cid:62) t w t (cid:107) w t (cid:107) and Π ⊥ w t x t are independent gaussian r.v., where x (cid:62) t w t (cid:107) w t (cid:107) ∼ N (0 , 1) and (cid:107) Π ⊥ w t x t (cid:107) 2 ∼ χ 2 ( m − 1) .
[Equation]
and by Lemma B.9,
[Equation]
Thus w.p. at least 16 , equation 21 and equation 22 happen together, which implies
[Equation]
Step 3 .
To stay in the cone { w | ∠ we 1 ≤ ε } , the SGD update (cid:107)
[Equation]
= (cid:107)∇ ln(1 + exp( − x (cid:62) t w t (cid:107) w t (cid:107) y t )) (cid:107) has to be smaller than (cid:107) w t (cid:107) sin 2 ε for any t
= T 0 + T 1 , . . .
, T
+ T 1 + T 2 .
However, step 1 and 2 together show that (cid:107)∇ ln(1+exp( − x (cid:62) t w t (cid:107) w t (cid:107) y t )) (cid:107) ≥ 2 (cid:107) w t (cid:107) ε w.p. 16 per iteration.
Thus the probability that w t always stays in the cone for every t
= T 0 + T 1 , . . .
, T
+ T 1 + Tis less than (cid:0) 56 (cid:1) T
≤ δ .
Remark B.10.
It’s interesting that the only property of the global minimum we use is that the if both w t , w t +1 are (cid:15) optimal, then the angle between w t and w t +1 is at most 2 (cid:15) .
Thus we indeed have proved a stronger statement: At least once in every
[Equation]
the angle between w t and w t +1 will be larger than 2 (cid:15) .
In other words, if the the amount of the update stablizes in terms of angle, then this angle must be larger than √ 2 ηλ for this simple model.
B. O MITTED P ROOFS IN S ECTION A
Lemma B.11.
Suppose loss L is scale invariant, then L is non-convex in the following two sense:
The domain is non-convex: scale invariant loss can’t be deﬁned at origin; 2.
There exists no ball containing origin such that the loss is locally convex, unless the loss is constant function.
Proof.
Suppose L ( θ ∗ ) =
sup θ ∈ B L ( θ ) .
W.L.O.G, we assume (cid:107) θ ∗ (cid:107) < 1 .
By convexity, every line segment passing θ ∗ must have constant loss, which implies the loss is constant over set B−{ c θ ∗ (cid:107) θ ∗ (cid:107) | − 1 ≤ c ≤ 0 } .
Applying the above argument on any other maximum point θ (cid:48) implies the loss is constant over B − { 0 } .
Theorem B.12.
Suppose the momentum factor γ = 0 , LR η t = η is constant, and the loss function
L is lower bounded.
If ∃ c > 0 and T ≥ 0 such that ∀ t ≥ T , f ( θ t +1 ) − f ( θ t ) ≤ − cη (cid:107)∇ L ( θ t ) (cid:107) 2 , then lim t →∞ (cid:107) θ t (cid:107) = 0 .
Proof in Item 3.
By Lemma 1.3 and the update rule of GD with WD, we have
[Equation]
, which implies
[Equation]
Thus for any T (cid:48) > T ,
T (cid:48) (cid:88) t =
T (cid:107) θ t (cid:107) 2 ≤ 1 − (1 − λη ) 
T (cid:48) − 1 (cid:88) t = T (cid:107)∇ L ( θ t ) (cid:107)
[Equation]
+ (cid:107) θ T (cid:107)  ≤ 1 λη  
T (cid:48) − 1 (cid:88) t = T (cid:107)∇ L (
[Equation]
t ) (cid:107)
[Equation]
(cid:107)
[Equation]
T (cid:107)  .
Note that by assumption we have (cid:80) T (cid:48) − 1 t
= T (cid:107)∇ L ( θ t ) (cid:107) 2 = 1 cη f ( θ T ) − f
( θ T (cid:48) ) .
As a conclusion, we have (cid:80) ∞ t = T (cid:107) θ t (cid:107) 2 ≤ f ( θ T ) − min θ f ( θ ) cη 2 λ
[Equation]
(cid:107)
[Equation]
, which implies lim t →∞ (cid:107)
[Equation]
(cid:107)
[Equation]
= 0 .
C O THER R ESULTS
Now we rigorously analyze norm growth in this algorithm.
This greatly extends previous analyses of effect of normalization schemes (Wu et al., 2018; Arora et al., 2018) for vanilla SGD.
Theorem C.1.
Under the update rule 1.2 with λ t = 0 , the norm of scale invariant parameter θ t satisﬁes the following property:
• Almost Monotone
Increasing: (cid:107) θ t +1 (cid:107) 2 − (cid:107) θ t (cid:107) 2 ≥ − γ t +1 η t η 0 ( (cid:107) θ 0 (cid:107) 2 − (cid:107) θ − 1 (cid:107) 2 ) .
• Assuming η t = η
is a constant, then
[Equation]
Proof.
Let’s use R t , D t , C t to denote (cid:107) θ t (cid:107) 2 , (cid:107) θ t +1 − θ t (cid:107) 2 , θ (cid:62) t ( θ t +1 − θ t ) respectively.
The only property we will use about loss is ∇ θ L (cid:62) t θ t = 0 .
Expanding the square of (cid:107) θ t +1 (cid:107) 2 = (cid:107) ( θ t +1 − θ t ) + θ t (cid:107) 2 , we have ∀ t ≥ − 1 S ( t ) :
R t +1 −
[Equation]
t =
D t + 2 C t .
We also have
[Equation]
namely, ∀ t ≥ 0 P ( t ) :
C
[Equation]
Simplify S ( t ) η t − γS ( t − 1) η t −+ P ( t ) , we have
R t +
[Equation]
R t η t − γ R t − R t −η t −= D t η t
[Equation]
γ
D t −η t − 1 −
[Equation]
λ t R
[Equation] [Equation]
When λ t = 0 , we have
[Equation]
Further if η t = η is a constant, we have
[Equation]
which covers the result without momentum in (Arora et al., 2019) as a special case:
[Equation]
For general deep nets, we have the following result, suggesting that the mean square of the update are constant compared to the mean square of the norm.
The constant is mainly determined by ηλ , explaining why the usage of weight decay prevents the parameters to converge in direction.
Theorem C.2.
For SGD with constant LR η , weight decay λ and momentum γ , when the limits
[Equation]
exist, we have
[Equation]
Proof of Theorem C.2.
Take average of Equation 24 over t , when the limits R ∞ =
lim T
[Equation]
T (cid:80)
[Equation]
(cid:107)
[Equation]
(cid:107)
[Equation]
D ∞ =
lim
[Equation]
(cid:80) T − 1 t =
[Equation]
(cid:107) w t +1 − w t (cid:107) 2 exists, we have
[Equation]
D A DDITIONAL E XPERIMENTAL F IGURES E S CALE I NVARIANCE IN M ODERN N ETWORK A RCHITECTURES
In this section, we will discuss how Normalization layers make the output of the network scaleinvariant to its parameters.
Viewing a neural network as a DAG, we give a sufﬁcient condition for the scale invariance which could be checked easily by topological order, and apply this on several standard network architectures such as Fully Connected(FC)
Networks, Plain CNN, ResNet(He et al., 2016a), and PreResNet(He et al., 2016b).
For simplicity, we restrict our discussions among networks with ReLU activation only.
Throughout this section, we assume the linear layers and the bias after last normalization layer are ﬁxed to its random initialization, which doesn’t harm the performance of the network empirically(Hoffer et al., 2018b).
E. N OTATIONS
Deﬁnition E.1 (Degree of Homogeneity) .
Suppose k is an integer and θ is all the parameters of the network, then f is said to be homogeneous of degree k , or k -homogeneous, if ∀ c > 0 , f ( c θ ) =
c k f ( θ ) .
The output of f can be multi-dimensional.
Speciﬁcally, scale invariance means degree of homogeneity is 0.
Suppose the network only contains following modules, and we list the degree of homogeneity of these basic modules, given the degree of homogeneity of its input.
(I) Input (L) Linear Layer, e.g. Convolutional Layer or Fully Connected Layer (B)
Bias Layer(Adding Trainable Bias to the output of the previous layer)
(+) Addition Layer (adding the outputs of two layers with the same dimension 2 .)
(N) Normalization Layer without afﬁne transformation(including BN, GN, LN, IN etc.)
(NA) Normalization Layer with afﬁne transformation
[Table]
Remark E.2.
For the purpose of deciding the degree of homogeneity of a network, there’s no difference among convolutional layers, fully connected layer and the diagonal linear layer in the afﬁne transformation of Normalization layer, since they’re all linear and the degree of homogeneity is increased by 1 after applying them.
On the other hand, BN and IN has some beneﬁt which GN and LN doesn’t have, namely the bias term (per channel) immediately before BN or IN has zero effect on the network output and thus can be removed.
(See Figure 15)
We also demonstrate the homogeneity of the output of the modules via the following ﬁgures, which will be reused to later to deﬁne network architectures.
Theorem E.3.
For a network only consisting of modules deﬁned above and ReLU activation, we can view it as a Directed acyclic graph and check its scale invariance by the following algorithm.
[Figure]
E. N ETWORKS WITHOUT A FFINE T RANSFORMATION AND B IAS
We start with the simple cases where all bias term(including that of linear layer and normalization layer) and the scaling term of normalization layer are ﬁxed to be 0 and 1 element-wise respectively, which means the bias and the scaling could be dropped from the network structure.
We empirically ﬁnd this doesn’t affect the performance of network in a noticeable way.
We will discuss the full case in the next subsection.
Plain CNN/FC networks: See Figure 8.
ResNet: See Figure 10.
To ensure the scaling invariance, we add an additional normalizaiton layer in the shortcut after downsampling.
This implementation is sometimes used in practice and doesn’t affect the performance in a noticeable way.
[Figure]
Preactivation ResNet: See Figure 11.
Preactivation means to change the order between convolutional layer and normalization layer.
For similar reason, we add an additional normalizaiton layer in the shortcut before downsampling.
[Figure]
E. N ETWORKS WITH A FFINE T RANSFORMATION
Now we discuss the full case where the afﬁne transformation part of normalization layer is trainable.
Due to the reason that the bias of linear layer (before BN) has 0 gradient as we mentioned in E.2, the bias term is usually dropped from network architecture in practice to save memory and accelerate training( even with other normalization methods)(See
PyTorch Implementation (Paszke et al., 2017)).
However, when LN or GN is used, and the bias term of linear layer is trainable, the network could be scale variant (See Figure 15).
Plain CNN/FC networks: See Figure 12.
ResNet: See Figure 13.
To ensure the scaling invariance, we add an additional normalizaiton layer in the shortcut after downsampling.
This implementation is sometimes used in practice and doesn’t affect the performance in a noticeable way.
[Figure]
Preactivation ResNet: See Figure 14.
Preactivation means to change the order between convolutional layer and normalization layer.
For similar reason, we add an additional normalizaiton layer in the shortcut before downsampling
[Figure]