Does Momentum Change the Implicit Regularization on Separable Data?
Anonymous Author(s)
AffiliationAddress
Abstract
The momentum acceleration technique is widely adopted in many optimizationalgorithms.
However, there is no theoretical answer on how the momentum affectsthe generalization performance of the optimization algorithms.
This paper studiesthis problem by analyzing the implicit regularization of momentum-based optimization.
We prove that on the linear classification problem with separable data andexponential-tailed loss, gradient descent with momentum (GDM) converges to the
L 2 max-margin solution, which is the same as vanilla gradient descent.
That meansgradient descent with momentum acceleration still converges to a low-complexitymodel, which guarantees their generalization.
We then analyze the stochastic andadaptive variants of GDM (i e., SGDM and deterministic Adam) and show they alsoconverge to the L 2 max-margin solution.
Technically, to overcome the difficulty ofthe error accumulation in analyzing the momentum, we construct new potentialfunctions to analyze the gap between the model parameter and the max-marginsolution.
Numerical experiments are conducted and support our theoretical results.
Introduction
It is widely believed that the optimizers have the implicit regularization in terms of selecting outputparameters among all the minima on the landscape [22, 14, 39].
Parallel to the analysis of coordinatedescent ([27, 35]), [29] shows that gradient descent would converge to the L 2 max-margin solutionfor the linear classification task with exponential-tailed loss, which mirrors its good generalizationproperty in practice.
Since then, many efforts have been taken on analyzing the implicit regularizationof various local-search optimizers, including stochastic gradient descent [20], steepest descent [6],
AdaGrad [25] and optimizers for homogeneous neural networks [18, 11, 37].
However, though the momentum acceleration technique is widely adopted in the optimizationalgorithms in both convex and non-convex learning tasks [32, 36, 33], the understanding of howthe momentum would affect the generalization performance of the optimization algorithms is stillunclear, as the historical gradients in the momentum may significantly change the searching directionof the optimization dynamics.
A natural question is:
Can we theoretically analyze the implicit regularization of momentum-based optimizers?
In this paper, we take the first step to analyze the convergence of momentum-based optimizersand unveil their implicit regularization.
Our study starts from the classification problem with thelinear model and exponential-tailed loss using Gradient Descent with Momentum (GDM) optimizer.
Then the variants of GDM such as Stochastic Gradient Descent with Momentum (SGDM) anddeterministic Adam are also analyzed.
We consider the optimizers with constant learning rateand constant momentum hyper-parameters, which are widely adopted in practice, e.g., the default
[Table]
setting in popular machine learning frameworks [23] and in experiments [40].
Our main results aresummarized in Theorem 1.
Theorem 1 (informal) .
With linearly separable dataset S , linear model and exponential-tailed loss:
• For GDM with a constant learning rate, the parameter norm diverges to infinity, with its
direction converging to the L 2 max-margin solution.
The same conclusion holds for SGDMwith a constant learning rate.•
For deterministic Adam with a constant learning rate and stochastic RMSProp (i
e., Adamwithout momentum) with a decaying learning rate, the same conclusion holds.
Theorem 1 states that GDM and its variants converge to the L 2 max-margin solution, which is thesame as their without-momentum versions, indicating that momentum does not affect the convergentdirection.
Therefore, the good generalization behavior of the output parameters of these optimizers iswell validated as the margin of a classifier is positively correlated with its generalization error [13]and is supported by existing experimental observations (e.g., [29, 19, 37]).
Our contributions are significant in terms of the following aspects:
• We establish the implicit regularization of the momentum-based optimizers, an open problemsince the initial work [29].
The momentum-based optimizers are widely used in practice, andour theoretical characterization deepens the understanding of their generalization property,which is important on its own.• Technically, we design a two-stage framework to analyze the momentum-based optimizers,
which generalizes the proof techniques in [29] and [20].
The first stage shows the conver-
gence of the loss.
New potential functions for SGDM and Adam are proposed and can be of
independent interest for convergence analysis of momentum-based optimizers.
The second
stage shows the convergence of the parameter.
We propose an easy-to-check condition of whether the difference between learned parameters and the scaled max-margin solution is
bounded.
This condition can be generalized to implicit regularization analyses of other
momentum-based optimizers.
• We further verify our theory through numerical experiments.
Organization of This Paper.
Section 2 collects further related works on the implicit regularizationof the first order optimizers and the convergence of momentum-based optimizers.
Section 3 showsbasic settings and assumptions which will be used throughout this paper.
Section 4 studies theimplicit regularization of GDM, while Section 5 and Section 6 explore respectively the implicitregularization of SGDM and Adam.
Discussions of these results are put in Section 7.
Detailed proofsand experiments can be found in the appendix.
Further related works
Implicit Regularization of First-order Optimization Methods.
[29] prove that gradient descent onlinear classification problem with exponential-tailed loss converges to the direction of the max L 2margin solution of the corresponding hard-margin Support Vector Machine.
[20] extend the results in[29] to the stochastic case, proving that the convergent direction of SGD is the same as GD almostsurely.
[25] go beyond the vanilla gradient descent methods and consider the AdaGrad optimizerinstead.
They prove that the convergent direction of AdaGrad has a dependency on the optimizingtrajectory, which varies according to the initialization.
[12] propose a primal-dual analysis frameworkfor the linear classification models and prove a faster convergent rate of the margin by increasingthe learning rate according to the loss.
Based on [12], [9] design another algorithm with an evenfaster convergent rate of margin by applying the Nesterov’s Acceleration Method on the dual space.
However, the corresponding form of the algorithm on the primal space is no longer a Nesterov’s
Acceleration Method nor GDM, which is significantly different from our settings.
On the other hand, another line of work is trying to extend the linear case result to deep neuralnetworks.
[10, 7] study the deep linear network and [29] study the two-layer neural networkwith ReLU activation.
[18] propose a framework to analyze the asymptotic direction of GD onhomogeneous neural networks, proving that given there exists a time the network achieves 100%training accuracy, GD will converge to some KKT point of the L 2 max-margin problem.
[37] extendthe framework of [18] to adaptive optimizers and prove RMSProp and Adam without momentumhave the same convergent direction as GD, while AdaGrad does not.
The results [18, 37] indicatethat results in the linear model can be extended to deep homogeneous neural networks and suggestthat the linear model is an appropriate starting point to study the implicit bias.
Except for the exponential-tailed loss, there are also works on the implicit bias with squared loss.
Interesting readers can refer to [26, 16, 1] etc. for details.
Convergence of Momentum-Based Optimization Methods.
For convex optimization problems,the convergence rate of Nesterov’s Acceleration Method [21] has been proved in various approaches(e.g.
In contrast, although GDM (Polyak’s Heavy-Ball Method) was proposed in [24]before the Nesterov’s Acceleration Method, the convergence of GDM on convex loss with Lipschitzgradient was not solved until [5] provides an ergodic convergent result for GDM, i.e., the convergentresult for the running average of the iterates.
[31] provide a non-ergodic analysis when the trainingloss is coercive (the training loss goes to infinity whenever parameter norm goes to infinity), convex,and globally smooth.
However, all existing results cannot be directly applied to exponential-tailedloss, which is non-coercive.
There are also works on the convergence of SGDM under various settings.
[41] prove SGDMconverges to a bounded region assuming both bounded gradient norm and bounded gradient variance.
The bounded gradient norm assumption is further removed by [42, 17].
Nevertheless, a convergingto-stationary-point analysis is required in the implicit regularization analysis.
Thus their results cannot be directly applied.
[34] analyze a particular case when the momentum parameter increases overiterations, which, however, does not agree with the practice where the momentum parameter is fixed.
As for (stochastic)
Adam, its convergence analysis is still an open problem, and the current analysesare restricted to specific settings (e.g., bounded gradient, dynamical momentum hyperparameters).
We recommend interested readers to refer to [4, 3, 28, 8] for details.
Preliminaries
This paper focuses on the linear model with the exponential-tailed loss.
We mainly investigate binaryclassification.
However, the methodology can be easily extended to the multi-class classificationproblem (please refer to Appendix E.3 for details).
Problem setting.
The dataset used for training is denoted as S =
( x i , y i )
Ni
=1 , where x i ∈ R
d is thei -th input feature, and
y i ∈ R is the i -th label ( i = 1 , 2 , · · · , N ).
We will use the linear model to fitthe label: for any feature x ∈ R d and parameter w ∈ R d , the prediction is given by ⟨ w , x ⟩ .
For binary classification, given any data z i =
( x i , y i ) ∈ S , the individual loss for parameter wis given as ℓ ( y i ⟨ w , x i ⟩ ) .
As only y i x i is used in the loss, we then ensemble the feature and labeltogether and assume y i = 1 ( ∀ i ∈ { 1 , · · · , N } ) without the loss of generality.
We then drop y i for brevity and redefine S =
( x i ) Ni =1 .
The spectral norm of the data matrix ( x 1 , · · · , x N ) is defined asσ max .
We use ˜ ℓ ( w , x ) ≜ ℓ ( ⟨ w , x ⟩ ) for brevity.
The optimization target is defined as the averaged loss:
L ( w ) =
(cid:80)
Ni =1 ˜ ℓ ( w , x i ) N .
Optimizer.
Here we will introduce the update rules of GDM, SGDM and deterministic Adam.
123 GDM’s update rule is
[Equation]
SGDM can be viewed as a stochastic version of GDM by randomly choosing a subset of the dataset 125 to update.
Specifically, SGDM changes the update of m ( t ) into
[Equation]
where B ( t ) is a subset of S with size b which can be sampled either with replacement ( abbreviatedas "w/. r" ) or without replacement (i.e., with random shuffling, abbreviated as "w/o. r" ), and
L B ( t ) is defined as L B ( t ) ( w ) =
(cid:80) [Equation] ∈ B ( t
[Equation]
( w [Equation] b .
We also define F t as the sub-sigma algebra over themini-batch sampling, such that ∀ t ∈ N , w ( t ) is adapted with respect to the sigma algebra flow F t .
The Adam optimizer can be viewed as a variant of SGDM in which the preconditioner is adopted, 131 whose form is characterized as follows:
[Equation]
(Update Rule) : w ( t ) =
w ( t − 1)
[Equation]
η
[Equation]
where 1 √ ˆ ν ( t − 1)+ ε 1 d is called the preconditioner.
Assumptions:
The analysis of this paper are based on three common assumptions in existingliterature (first proposed by [29]).
They are respectively on the separability of the dataset, theindividual loss behavior at the tail, and the smoothness of the individual loss.
We list them as follows:
Assumption 1 (Linearly Separable Dataset) .
There exists one parameter w ∈ R d , such that [Equation]
Assumption 2 (Exponential-tailed Loss) .
The individual loss ℓ is exponential-tailed, i.e.,
• Differentiable and monotonically decreasing to zero, with its derivative converging to zero at
positive infinity and to non-zero at negative infinity, i.e., lim x →∞ ℓ ( x )
[Equation]
x →∞ ℓ ′ ( x ) = [Equation] [Equation]
ℓ
[Equation]
x ) < 0 , and ℓ ′ ( x ) < 0 , ∀ x ∈ R ;
• Close to exponential loss when x is large enough,
i
e., there exist positive constants c, a, µ + , µ − , x + , and x − , such that,
[Equation]
Assumption 3 (Smooth Loss) .
Either of the following assumptions holds regarding the case: (D): (Without Stochasticity)
The individual loss ℓ is locally smooth, i. e., for any s 0 ∈ R , there exists apositive real H s 0 , such that ∀ x, y ≥ s 0 , | ℓ ′ ( x ) − ℓ ′ ( y ) | ≤ H s 0 | x − y | .(S
): (With Stochasticity)
The individual loss ℓ is globally smooth, i.e., there exists a positive real H ,such that ∀ x, y ∈ R , | ℓ ′ ( x ) − ℓ ′ ( y ) | ≤ H | x − y | .
We provide explanations of these three assumptions, respectively.
Based on Assumption 1, we canformally define the margin and the maximum margin solution of an optimization problem:
Definition 1.
Let the margin ˆ γ ( w ) of parameter w defined as the lowest score of the prediction of w 151 over the dataset S , i.e., ˆ γ ( w ) =
min x ∈ S ⟨ w , x ⟩ .
We then define the maximum margin solution ˆ wand the L 2 max margin γ of the dataset S as follows:
[Equation]
Since ∥ · ∥ 2 is strongly convex and set { w : ˆ γ ( w ) ≥ 1 } is convex, ˆ w is uniquely defined.
Assumption 2 constraints the loss to be exponential-tailed, which is satisfied by many popular choicesof ℓ , including the exponential loss ( ℓ exp ( x ) =
e − x ) and the logistic loss ( ℓ log ( x ) =
log(1 + e − x ) ).
Also, as c and a can be respectively absorbed by resetting the learning rate and data as η = cη andx i
= a x i , without loss of generality, in this paper we only analyze the case that c = a = 1 .
The globally smooth assumption (Assumption 3. (S)) is strictly stronger than the locally smoothassumption (Assumption 3.
(D)).
One can easily verify that both the exponential loss and the logisticloss meet Assumption 3.
( D ), and the logistic loss also meets Assumption 3.
( S ).
The implicit regularization of GDM
In this section, we analyze the implicit regularization of GDM with a two-stage framework 1 .
Later,we will use this framework to investigate SGDM further and deterministic Adam.
The formaltheorem of the implicit regularization of GDM is as follows:
Theorem 2.
Let Assumptions 1, 2, and 3.
( D ) hold.
Let β ∈
[0 , 1) and η <
N σ 2 max H ℓ − 1 ( N L ( w 1 ))
( ℓ − 1is the inverse function of ℓ ).
Then, for almost every data set S , with arbitrary initialization point w (1) , GDM (Eq.
equation 1) satisfies that w ( t ) − ln( t ) ˆ w is bounded as t → ∞ , and lim t →∞ w ( t ) ∥
w ( t ) ∥
= ˆ w ∥ ˆ w ∥ .
Theorem 2 shows that the implicit regularization of GDM agrees with GD in linear classification withexponential-tailed loss (c.f.
[29] for results on GD).
This consistency can be verified by existing andour experiments (c.f. Section 7 for detailed discussions).
Remark 1 (On the hyperparameter setting) .
Firstly, the learning rate upper bound
N σ 2 max H ℓ − 1 ( N L ( w 1 )) agrees with that of GD exactly [29], indicating our analysis is tight.
Secondly, Theorem 2 adopts a constant momentum hyper-parameter, which agrees with the practical use(e.g.
, β is fixed to be 0 .
Also, Theorem 2 puts no restriction on the range of β , which allowswider choices of hyper-parameter tuning.
We then present a proof sketch of Theorem 2, which is divided into two parts: we first prove that thesum of squared gradients is bounded, which indicates both the loss and the norm of gradient convergeto 0 and the parameter diverges to infinity; these properties will then be applied to show the differencebetween w ( t ) and ln( t ) ˆ w is bounded, and therefore, the direction of ˆ w dominates as t → ∞ .
Stage I: Loss Dynamics.
The goal of this stage is to characterize the dynamics of the loss and provethe convergence of GDM.
The core of this stage is to select a proper potential function ξ ( t ) , which isrequired to correlate with the training loss L and be non-increasing along the optimization trajectory.
For GD, since L is non-increasing with a properly chosen learning rate, we can pick ξ ( t ) =
L ( t ) .
However, as the update of GDM does not align with the direction of the negative gradient, trainingloss L ( t ) in GDM is no longer monotonously decreasing, and the potential function requires specialconstruction.
Inspired by [31], we choose the following ξ ( t ) :
Lemma 1.
Let all conditions in Theorem 2 hold.
Define ξ ( t ) △ =
L ( w ( t ))
+ 1 − β 2 βη
∥ w ( t ) − w ( t − 1) ∥ 2 .[-
-]
Define C 1 as a positive real with C 1 ≜ σ 2 max H ℓ − 1 ( N L ( w 1 )) 2 N η .
We then have
[Equation]
Remark 2.
Although this potential function is obtained by [31] by directly examining Taylor’sexpansion at w ( t ) , the proof here is non-trivial as we only require the loss to be locally smoothinstead of globally smooth in [31].
We need to prove that the smoothness parameter along thetrajectory is upper bounded.
We defer the detailed proof to Appendix B.1.1.
By Lemma 1, we have that ξ ( t ) is monotonously decreasing by gap 1 − C 1 η ∥ w ( t + 1) − w ( t ) ∥ 2 .
As ξ (1) =
L ( w (1)) is a finite number, we have (cid:80) ∞ t =1 ∥ w ( t + 1) − w ( t ) ∥ 2 < ∞ .
By that (1 − β ) η ∇L ( w ( t ))
= ( w ( t + 1) − w ( t ))
− β ( w ( t ) − w ( t − 1)) , it immediately follows that (cid:80) ∞ t
=1∥∇L ( w ( t ))
∥ 2 < ∞ .
Stage II.
Parameter Dynamics.
The goal of this stage is to characterize the dynamics of theparameter and show that GDM asymptotically converges (in direction) to the max-margin solutionˆ w .
To see this, we define a residual term r ( t ) △ = w ( t ) − ln( t ) ˆ w − ˜ w with some constant vector ˜ w(specified in Appendix B.1.2).
If we can show the norm of r ( t ) is bounded over the iterations, wecomplete the proof as ln( t ) ˆ w will then dominates the dynamics of w ( t ) .
For simplicity, we use the continuous dynamics approximation of GDM [31] to demonstrate why 203 r ( t ) is bounded:
[Equation]
We start by directly examining the evolution of ∥ r ( t ) ∥ , i.e.,
[Equation]
which by integration by part leads to
[Equation]
[Table]
We then check the terms one by one:
• (cid:82) T 1 (cid:10) r
( s ) , − η ∇L ( w ( s ))
− 1 s ˆ w (cid:11) d s : This term also occurs in the analysis of GD [29] andhas been proved to be bounded;
• ⟨ r ( T ) , − d w ( T ) d t ⟩ : as shown in Stage I, d w ( T ) d t → 0 (i
e., w ( T ) − w ( T − 1) → 0 in thediscrete case) as T → ∞ .
Thus, this term is o ( ∥ r ( T ) ∥ ) ;
• (cid:82) T 1 (cid:68) d r ( s ) d s , d w ( s ) d s (cid:69) d s : finite due to d r ( s ) d s = d w ( s ) d
s
− 1 s ˆ w and (cid:82) ∞ 1 ∥ d w ( s ) d s ∥ 2 d s is finite (i e., (cid:80) ∞ t =1 ∥ w ( t + 1) − w ( t ) ∥ 2 < ∞ in the discrete case) by Stage I.
Putting them together, we show that ∥ r ( T ) ∥ 2 + o ( ∥ r ( T ) ∥ ) is upper bounded over the iterations,which immediately leads to that ∥ r ( T ) ∥ is bounded.
Applying similar methodology to the discreteupdate rule, we have the following lemma (the proof can be found in Appendix B.1.2).
Lemma 2.
Define potential function g : Z + → R as
[Equation]
g ( t ) is upper bounded, which further indicates ∥ r ( t ) ∥ is upper bounded.
Remark 3.
Our technique for analyzing GDM here is essentially more complex and elaborate thanthat for GD in [29] due to the historical information of gradients GDM.
The approach in [29]cannot be directly applied.
It is worth mentioning that we provide a more easy-to-check condition forwhether r ( t ) is bounded, i.e., "is g ( t ) upper-bounded?".
This condition can be generalized for othermomentum-based implicit regularization analyses.
E.g., SGDM and Adam later in this paper.
Tackle the difficulty brought by random sampling
In this section, we analyze the implicit regularization of SGDM.
Parallel to GDM, we establish thefollowing implicit regularization result for SGDM:
Theorem 3.
Let Assumption 1, 2, and 3.
( S ) hold.
Let β ∈
[0 , 1) and η <
[Figure]
[Equation]
[Figure]
[Equation]
Then,with arbitrary initialization point w (1) , SGDM (w/. r) satisfies w ( t ) − ln
( t ) ˆ w is bounded as t → ∞and lim t →∞ w ( t ) ∥ w ( t ) ∥
= ˆ w ∥ ˆ w ∥ , almost surely (a.s.).
Similar to the GDM case, Theorem 3 shows that the implicit regularization of SGDM under thissetting is consistent with SGD (c.f.
[20] for the implicit regularization of SGD).
This matches theobservations in practice (c.f. Section 7 for details), and is later supported by our experiments (e.g.,
Figure 1).
We add two remarks on the learning rate upper bound and extension to SGDM (w/. r).
Remark 4 (On the learning rate) .
Firstly, our learning rate upper bound
[Equation]
exactlymatches that of SGD 2 bγ 2 Hσ 4 max
[20] when β
= 0 , and matches that of SGD in terms of the orderof σ max , H , and b
when β ̸ = 0 .
This indicates our analysis is tight.
Secondly, as the bound ismonotonously increasing with respect to batch size b , Theorem 3 also sheds light on the learning ratetuning, i.e., the larger the batch size is, the larger the learning rate is.
Remark 5.
(On SGDM (w/o. r))
Theorem 3 can be similarly extended to SGDM (w/o. r).
We deferthe detailed description of the corresponding theorem together with the proof to Appendix E.2.
Next, we show the proof sketch for Theorem 3.
The proof also contains two stages, where Stage II is 241 similar to that for GDM.
However, we highlight that Stage I for SGDM is not a trivial extensionof that for GDM and has its merit for other optimization analyses for SGDM .
Specifically, the 243 methodology used to construct GDM’s potential function fails for SGDM due to the random sampling.
We defer a detailed discussion to Appendix B.2.3.
We then need to find a proper potential function 245 for SGDM.
Inspired by SGD’s simple update rule, we rearrange the update rule of SGDM such thatonly the gradient information of the current step is contained, i.e.,
[Equation]
β − η ∇L B ( t ) ( w ( t )) .
By defining u ( t ) ≜ w ( t ) − β w ( t − 1) 1 − β = w ( t ) + β 1 − β ( w ( t ) − w ( t − 1)) , we have that u ( t ) is close tow ( t ) (differs by order of one-step update w ( t ) − w ( t − 1) ), and the update rule of u ( t ) only containsthe current-step-gradient information ∇L ( w ( t )) .
We then select potential function as L ( u ( t )) , and asimple Taylor’s expansion directly leads to:
[Equation]
i.e., L ( u ( t )) is a proper potential function.
We formalize the above discussion into the followinglemma.
Lemma 3.
Let all conditions in Theorem 3 hold.
Then, there exists a positive constant C 2 , such that
[Equation]
By letting T → ∞ in the second claim of Lemma 3, we have (cid:80) ∞ s =1 E ∥∇L ( w ( s ))
∥ 2 < ∞ , which isindeed what we need in Stage I.
Analyze the effect of preconditioners
6.1 Implicit regularization of deterministic Adam
This section presents the implicit regularization of deterministic Adam, i.e., Adam without randomsampling.
Theorem 4.
Let Assumption 1, 2, and 3.
( D ) hold.
Let 1 > β 2 > β 41 ≥ 0 , and the learning rateη is a small enough constant (The upper bound of learning rate is complex, and we defer it to
Appendix C.1).
Then, with arbitrary initialization point w (1) , deterministic Adam (Eq. 3) satisfiesthat w ( t ) − ln( t ) ˆ w is bounded as t → ∞ , and lim t →∞ w ( t ) ∥ w ( t ) ∥
= ˆ w ∥ ˆ w ∥ .
Remark 6 (On the β 1 and β 2 range) .
Almost all existing literature assume a time-decaying hyperparameter choice of β 1 or β 2 (c.f., [15, 2]).
On the other hand, our result proves that deterministic
Adam converges with constant settings of β 1 and β 2 , which agrees with the practical use.
Remark 7 ((Discussion on the results in [29])) .
[29] observe that, on a synthetic dataset, the directionof the output parameter by Adam still does not converge to the max-margin direction after 2 × 10 6iterations (but is getting closer).
At the same time, GD seems to converge to the max-margin direction.
This seems to contradict with Theorem 4. We reconduct the experiments (please refer to Appendix
F.1.2 for details) and found (1).
such a phenomenon occurs because a large learning rate is selected,using which GD will also stick in a direction close to the max-margin direction (but not equal); (2).the constructed synthetic dataset is ill-posed, as the non-support data is larger than the support databy order of magnitude (which is also rare in practice).
On a well-posed dataset, we observe that both Adam and GD will converge to the max-margin solution rapidly (Figure 1).
We simply introduce the proof idea here and put the full proof in Appendix C. The proof of determin- 277 istic Adam is very similar to that of GDM with minor changes in potential functions.
Specifically,ξ ( t ) in Lemma 1 is changed to
[Equation]
and g ( t ) in Lemma 2 is changed to
[Equation]
The rest of the proof then flows similarly to the GDM case.
6.2 What if random sampling is added?
We have obtained the implicit regularization for GDM, SGDM, and deterministic Adam.
One maywonder whether the implicit regularization of stochastic Adam can be obtained.
Unfortunately, thegap can not be closed yet.
This is because an implicit regularization analysis requires the knowledgeof the loss dynamics, little of which, however, has been ever known even for stochastic RMSProp(i.e.
, Adam with β 1 = 0 in Eq.
(3)) with constant learning rates.
Specifically, the main difficulty liesin bounding the change of conditioner 1 √
[Equation]
1 d + [Equation] ν ( t ) across iterations, which is required to make thedrift term ⟨∇L ( w ( t )) , E ( w ( t + 1) − w ( t )) ⟩ (derived by Taylor’s expansion of the epoch start from
Kt ) negative to ensure a non-increasing loss.
On the other hand, if we adopt decaying learning rates η t = 1 √ t , [28] shows β 2 close enough to 1 , thefollowing equation holds for stochastic RMSProp (w/o. r) (recall that K ≜
Nb is the epoch size)
[Equation]
Based on this result, we have the following theorem for stochastic RMSProp (w/o. r):
Theorem 5.
Let Assumptions 1, 2, and 3.
(S) hold.
Let β 2 be close enough to 1 .
Then, with arbitraryinitialization point w (1) , stochastic RMSProp (w/o. r) converges to the L 2 max-margin solution.
Remark 8 (On the decaying learning rate) .
The decaying learning rate is a "stronger" settingcompared to the constant learning rate, both in the sense that GDM, SGDM, and deterministic Adamcan be shown to converge to the max-margin solution following the same routine as Theorems 2, 3,and 4, and in the sense that we usually adopt constant learning rate in practice.
The proof is on the grounds of a novel characterization of the loss convergence rate derived from Eq. 300 (8), and readers can find the details in Appendix D. Furthermore
, the proof can be easily extended tothe Stochastic Adaptive Heavy Ball (SAHB) algorithm (proposed by [34]), which can be viewed as a 302 momentum version of RMSProp (but different from Adam) with the following update rule [Equation] The proof of Theorem 5 can be easily extended to SAHB, based on the fact that u ( t ) = w ( t ) − β 1 w ( t − 1) 1 − β 1has a simple update, i.e., [Equation]
and L ( u ( t )) can be used as a potential function just as the SGDM case.
[Figure]
all optimizers converge to the
the asymptotic behaviors with & without momentum are similar.
The experimental observation support our theoretical results.
Discussions
Consistency with the Experimental Results.
We conduct experiments to verify our theoreticalfindings.
Specifically, we (1).
run GD, GDM, SGD, SGDM, and Adam on a synthetic dataset toobserve their implicit regularization; (2).
run GD and Adam on ill-posed dataset proposed in [29]to verify Theorem 4; (3).
run SGD and SGDM on neural networks to classify the MNIST datasetand compare their implicit regularization.
The experimental observations stand with our theoreticalresults.
Furthermore, it is worth mentioning that experimental phenomenons that adding momentumwill not change the implicit regularization have also been observed by existing literature [29, 19, 37].
Influence of hyperparameters on convergence rates.
Our results can be further extended to providea precise characterization of the influence of the hyperparameters η and β on the convergence rate of(S)GDM.
Specifically, in Appendix E.4, we show that the asymptotic convergence rate of (S)GDM is C 1 η 1 t , where C is some constant independent of β and η .
Therefore, increase η can lead to a fasterconvergence rate (with learning rate requirements in Theorems 2 and 3 satisfied).
However, changingβ does not affect the convergence rate, which is also observed in our experiments (e.g., Figure 1).
While this seems weird, as replacing gradient with momentum in deep learning often accelerates thetraining, we hypothesize that such an acceleration appears as the landscape of neural networks ishighly non-convex and thus can not be observed in the case considered by this paper.
Gap Between The Linear Model and Deep Neural Networks.
While our results only hold forthe linear classification problem, extending the results to the deep neural networks is possible.
Specifically, existing literature [18, 37] provide a framework for deriving implicit regularization fordeep homogeneous neural networks.
However, the approach in [18, 37] can not be trivially appliedto the momentum-based optimizers, as their proofs require the specific gradient-based updates tolower bound a smoothed margin (c.f., Theorem 4.1, [18]).
It remains an exciting work to see how ourresults can be expanded to GDM and Adam for deep neural networks.
Conclusion
This paper studies the implicit regularization of momentum-based optimizers in linear classificationwith exponential-tailed loss.
Our results indicate that for SGD and the deterministic version of Adam,adding momentum will not influence the implicit regularization, and the direction of the parameterconverges to the L 2 max-margin solution.
Our theoretical results stand with existing experimentalobservations, and developed techniques such as the potential functions may inspire the analyses onother momentum-based optimizers.
Motivated by the results and techniques for linear cases in thispaper, it has the potential to extend them to the homogeneous neural network in the future.
Anothertopic left for future work is to derive the implicit regularization of constant learning rate stochastic
Adam.
As discussed in Section 6.2, this topic is non-trivial, and it requires new techniques andassumptions to be developed.