Does Momentum Change the Implicit Regularization on Separable Data?
Abstract
The momentum acceleration technique is widely adopted in many optimization algorithms.
However, there is no theoretical answer on how the momentum affects the generalization performance of the optimization algorithms.
This paper studies this problem by analyzing the implicit regularization of momentum-based optimization.
We prove that on the linear classification problem with separable data and exponential-tailed loss, gradient descent with momentum (GDM) converges to the
L 2 max-margin solution, which is the same as vanilla gradient descent.
That means gradient descent with momentum acceleration still converges to a low-complexity model, which guarantees their generalization.
We then analyze the stochastic and adaptive variants of GDM (i.e., SGDM and deterministic Adam) and show they also converge to the L 2 max-margin solution.
Technically, the implicit regularization of SGDM is established based on a novel convergence analysis of SGDM under a general noise condition called affine noise variance condition. To the best of our knowledge, we are the first to derive SGDM’s convergence under such an assumption.
Numerical experiments are conducted to support our theoretical results.
Introduction
It is widely believed that the optimizers have the implicit regularization in terms of selecting output parameters among all the minima on the landscape [26, 17, 45].
Parallel to the analysis of coordinate descent ([31, 40]), [34] shows that gradient descent would converge to the L 2 max-margin solution for the linear classification task with exponential-tailed loss, which mirrors its good generalization property in practice.
Since then, many efforts have been taken on analyzing the implicit regularization of various local-search optimizers, including stochastic gradient descent [24], steepest descent [8],
AdaGrad [29] and optimizers for homogeneous neural networks [21, 14, 43].
However, though the momentum acceleration technique is widely adopted in the optimization algorithms in both convex and non-convex learning tasks [37, 41, 38], the understanding of how [Table] the momentum would affect the generalization performance of the optimization algorithms is still unclear, as the historical gradients in the momentum may significantly change the searching direction of the optimization dynamics.
A natural question is:
Can we theoretically analyze the implicit regularization of momentum-based optimizers?
In this paper, we take the first step to analyze the convergence of momentum-based optimizers and unveil their implicit regularization.
Our study starts from the classification problem with the linear model and exponential-tailed loss using Gradient Descent with Momentum (GDM) optimizer.
Then the variants of GDM such as Stochastic Gradient Descent with Momentum (SGDM) and deterministic Adam are also analyzed.
We consider the optimizers with constant learning rate and constant momentum hyper-parameters, which are widely adopted in practice, e.g., the default setting in popular machine learning frameworks [27] and in experiments [46].
Our main results are summarized in Theorem 1.
Theorem 1 (informal) .
With linearly separable dataset S , linear model and exponential-tailed loss:
• For GDM with a constant learning rate, the parameter norm diverges to infinity, with its direction converging to the L 2 max-margin solution.
The same conclusion holds for SGDM with a constant learning rate.
• For deterministic Adam with a constant learning rate and stochastic RMSProp (i.e., Adam without momentum) with a decaying learning rate, the same conclusion holds.
Theorem 1 states that GDM and its variants converge to the L 2 max-margin solution, which is the same as their without-momentum versions, indicating that momentum does not affect the convergent direction.
Therefore, the good generalization behavior of the output parameters of these optimizers is well validated as the margin of a classifier is positively correlated with its generalization error [16] and is supported by existing experimental observations (e.g., [34, 23, 43]).
Our contributions are significant in terms of the following aspects
: • We establish the implicit regularization of the momentum-based optimizers, an open problem since the initial work [34].
The momentum-based optimizers are widely used in practice, and our theoretical characterization deepens the understanding of their generalization property, which is important on its own. Technically, we design a two-stage framework to analyze the momentum-based optimizers, which generalizes the proof techniques in [34] and [24].
The first stage shows the convergence of the loss.
Specifically, we derive the first convergence analysis of SGDM under affine noise variance condition [2], generalizing the results under bounded noise variance condition in literature [47, 20].
The second stage shows the convergence of the parameters.
We propose an easy-to-check condition of whether the difference between learned parameters and the scaled max-margin solution is bounded.
This condition can be generalized to implicit regularization analyses of other momentum-based optimizers.
• We further verify our theory through numerical experiments.
The technical difficulty to analyzing the implicit regularization of the momentum-based optimizers mainly come from the historical information of training trajectory contained by the momentum term.
Specifically, the foundation of the analysis is to derive the convergence of the loss.
It is not trivial because, for momentum-based optimizers, the loss may not monotonously decrease due to the mismatch between the direction of the gradient and the momentum.
Furthermore, even if the loss is proved to converge, the converging direction of the parameters is still hard to characterize as it’s difficult to track the training trajectory due to the momentum term.
Organization of This Paper.
Section 2 collects further related works on the implicit regularization of the first-order optimizers and the convergence of momentum-based optimizers.
Section 3 shows basic settings and assumptions which will be used throughout this paper.
Section 4 studies the implicit regularization of GDM, while Section 5 and Section 6 explore respectively the implicit regularization of SGDM and Adam.
Discussions of these results are put in Section 7.
Detailed proofs and experiments can be found in the appendix.
Further related works
Implicit Regularization of First-order Optimization Methods.
[34] prove that gradient descent on linear classification problem with exponential-tailed loss converges to the direction of the max Lmargin solution of the corresponding hard-margin Support Vector Machine.
[24] extend the results in [34] to the stochastic case, proving that the convergent direction of SGD is the same as GD almost surely.
[29] go beyond the vanilla gradient descent methods and consider the AdaGrad optimizer instead.
They prove that the convergent direction of AdaGrad has a dependency on the optimizing trajectory, which varies according to the initialization.
[15] propose a primal-dual analysis framework for the linear classification models and prove a faster convergent rate of the margin by increasing the learning rate according to the loss.
Based on [15], [12] design another algorithm with an even faster convergent rate of margin by applying the Nesterov’s Acceleration Method on the dual space.
However, the corresponding form of the algorithm on the primal space is no longer a Nesterov’s
Acceleration Method nor GDM, which is significantly different from our settings.
On the other hand, another line of work is trying to extend the linear case result to deep neural networks.
[13, 9] study the deep linear network and [34] study the two-layer neural network with ReLU activation.
[21] propose a framework to analyze the asymptotic direction of GD on homogeneous neural networks, proving that given there exists a time the network achieves 100% training accuracy, GD will converge to some KKT point of the L 2 max-margin problem.
[43] extend the framework of [21] to adaptive optimizers and prove RMSProp and Adam without momentum have the same convergent direction as GD, while AdaGrad does not.
The results [21, 43] indicate that results in the linear model can be extended to deep homogeneous neural networks and suggest that the linear model is an appropriate starting point to study the implicit bias.
Except for the exponential-tailed loss, there are also works on the implicit bias with squared loss.
Interested readers can refer to [30, 19, 1] etc. for details.
Convergence of Momentum-Based Optimization Methods.
For convex optimization problems, the convergence rate of Nesterov’s Acceleration Method [25] has been proved in various approaches (e.g., [25, 35, 44]).
In contrast, although GDM (Polyak’s Heavy-Ball Method) was proposed in [28] before the Nesterov’s Acceleration Method, the convergence of GDM on convex loss with Lipschitz gradient was not solved until [7] provides an ergodic convergent result for GDM, i.e., the convergent result for the running average of the iterates.
[36] provide a non-ergodic analysis when the training loss is coercive (the training loss goes to infinity whenever parameter norm goes to infinity), convex, and globally smooth.
However, all existing results cannot be directly applied to exponential-tailed loss, which is non-coercive.
There are also works on the convergence of SGDM under various settings.
[47] prove SGDM converges to a bounded region assuming both bounded gradient norm and bounded gradient variance.
The bounded gradient norm assumption is further removed by [48, 20].
Nevertheless, a convergingto-stationary-point analysis is required in the implicit regularization analysis.
Thus their results can not be directly applied.
[39] analyze a particular case when the momentum parameter increases over iterations, which, however, does not agree with the practice where the momentum parameter is fixed.
As for (stochastic)
Adam, its convergence analysis is still an open problem, and the current analyses are restricted to specific settings (e.g., bounded gradient, dynamical momentum hyperparameters).
We recommend interested readers to refer to [5, 4, 33, 10] for details.
Preliminaries
This paper focuses on the linear model with the exponential-tailed loss.
We mainly investigate binary classification.
However, the methodology can be easily extended to the multi-class classification problem (please refer to Appendix F.3 for details).
Problem setting.
The dataset used for training is denoted as S =
( x i , y i )
Ni
=1 , where x i ∈ R
d is the i -th input feature, and y i ∈ R is the i -th label ( i = 1 , 2 , · · · , N ).
We will use the linear model to fit the label: for any feature x ∈ R d and parameter w ∈ R d , the prediction is given by ⟨ w , x ⟩ .
For binary classification, given any data z i =
( x i , y i ) ∈ S , the individual loss for parameter w is given as ℓ ( y i ⟨ w , x i ⟩ ) .
As only y i x i is used in the loss, we then ensemble the feature and label together and assume y i = 1 ( ∀ i ∈ { 1 , · · · , N } ) without the loss of generality.
We then drop y i for brevity and redefine S =
( x i ) Ni =1 .
The spectral norm of the data matrix ( x 1 , · · · , x N ) is defined as σ max .
We use ˜ ℓ ( w , x ) ≜ ℓ ( ⟨ w , x ⟩ ) for brevity.
The optimization target is defined as the averaged loss:
L ( w ) =
(cid:80)
Ni =1 ˜ ℓ ( w , x i ) N .
Optimizer.
Here we will introduce the update rules of GDM, SGDM and deterministic Adam.
GDM’s update rule is
[Equation]
SGDM can be viewed as a stochastic version of GDM by randomly choosing a subset of the dataset to update.
Specifically, SGDM changes the update of m ( t ) into
[Equation]
where ξ ( t ) is the gradient noise of the t -th step, independent of the randomness of the previous steps and satisfying E ξ ( t ) ∇L ξ ( t ) ( w ) =
∇L ( w ) , ∀ w .
In this paper, ξ ( t ) comes from the random sampling of a subset B ( t ) of S with size b (either with replacement ( abbreviated as "w/. r" ) or without replacement (i.e., with random shuffling, abbreviated as "w/o. r" )), and L ξ ( t ) is defined as L
[Equation]
We also define F t as the sub-sigma algebra over the gradient noise collecting randomness before the t -th step. Therefore, w ( t ) is adapted with respect to the sigma algebra flow F t .
The Adam optimizer can be viewed as a variant of SGDM in which the preconditioner is adopted, whose form is characterized as follows:
[Equation]
[Equation]
where
1 √ ˆ ν
[Equation]
+ ε 1 d is called the preconditioner.
Assumptions:
The analysis of this paper are based on three common assumptions in existing literature (first proposed by [34]).
They are respectively on the separability of the dataset, the individual loss behavior at the tail, and the smoothness of the individual loss.
We list them as follows:
Assumption 1 (Linearly Separable Dataset) .
There exists one parameter w ∈ R d , such that ⟨ w , x i ⟩ > 0 , ∀ i ∈ [ N ] .
Assumption 2 (Exponential-tailed Loss) .
The individual loss ℓ is exponential-tailed, i
e., • Differentiable and monotonically decreasing to zero, with its derivative converging to zero at positive infinity and to non-zero at negative infinity,
i
e., lim x →∞ ℓ ( x ) =
lim x →∞ ℓ ′ ( x ) = 0 , lim x →−∞ ℓ ′ ( x ) < 0 , and ℓ ′ ( x ) < 0 , ∀ x ∈ R ;
• Close to exponential loss when x is large enough, i.e., there exist positive constants c, a, µ + , µ − , x + , and x − , such that,
[Equation]
Assumption 3 (Smooth Loss) .
Either of the following assumptions holds regarding the case:
(D): (Without Stochasticity)
The individual loss ℓ is locally smooth, i
e., for any s 0 ∈ R , there exists a positive real H s 0 , such that ∀ x, y ≥ s 0 , | ℓ ′ ( x ) − ℓ ′ ( y ) | ≤ H s 0 | x − y | .
(S): (With Stochasticity)
The individual loss ℓ is globally smooth, i.e., there exists a positive real H , such that ∀ x, y ∈ R , | ℓ ′ ( x ) − ℓ ′ ( y ) | ≤ H | x − y | .
We provide explanations of these three assumptions, respectively.
Based on Assumption 1, we can formally define the margin and the maximum margin solution of an optimization problem:
Definition 1.
Let the margin ˆ γ ( w ) of parameter w defined as the lowest score of the prediction of w over the dataset S , i.e., ˆ γ ( w ) =
min x ∈ S ⟨ w , x ⟩ .
We then define the maximum margin solution ˆ w and the L 2 max margin γ of the dataset S as follows:
[Equation]
Since ∥ · ∥ 2 is strongly convex and set { w : ˆ γ ( w ) ≥ 1 } is convex, ˆ w is uniquely defined.
Assumption 2 constraints the loss to be exponential-tailed, which is satisfied by many popular choices of ℓ , including the exponential loss ( ℓ exp ( x ) =
e − x ) and the logistic loss ( ℓ log ( x ) =
log(1 + e − x ) ).
Also, as c and a can be respectively absorbed by resetting the learning rate and data as η = cη and x i = a x i , without loss of generality, in this paper we only analyze the case that c = a = 1 .
The globally smooth assumption (Assumption 3. (S)) is strictly stronger than the locally smooth assumption (Assumption 3.
(D)).
One can easily verify that both the exponential loss and the logistic loss meet Assumption 3.
( D ), and the logistic loss also meets Assumption 3.
( S ).
The implicit regularization of GDM
In this section, we analyze the implicit regularization of GDM with a two-stage framework 1 .
Later, we will use this framework to investigate SGDM further and deterministic Adam.
The formal theorem of the implicit regularization of GDM is as follows:
Theorem 2.
Let Assumptions 1, 2, and 3.
( D ) hold.
Let β ∈
[0 , 1) and η <
[Equation]
( ℓ − 1 is the inverse function of ℓ ).
Then, for almost every data set S , with arbitrary initialization point w (1) , GDM (Eq.
equation 1) satisfies that ∥ w ( t ) ∥ w ( t ) ∥ − ˆ w ∥ ˆ w ∥ ∥ = O (1 / log( t ))
and L ( w ( t ))
= O ( 1 t ) .
Theorem 2 shows that the implicit regularization of GDM agrees with GD in linear classification with exponential-tailed loss (c.f.
[34] for results on GD).
This consistency can be verified by existing and our experiments (c.f. Section 7 for detailed discussions).
Remark 1 (On the hyperparameter setting) .
Firstly, the learning rate upper bound
N σ 2 max H ℓ − 1 ( N L ( w 1 )) agrees with that of GD exactly [34], indicating our analysis is tight.
Secondly, Theorem 2 adopts a constant momentum hyper-parameter, which agrees with the practical use (e.g., β is fixed to be 0 .
Also, Theorem 2 puts no restriction on the range of β , which allows wider choices of hyper-parameter tuning.
We then present a proof sketch of Theorem 2, which is divided into two parts: we first prove that the sum of squared gradients is bounded, which indicates both the loss and the norm of gradient converge to 0 and the parameter diverges to infinity; these properties will then be applied to show the difference between w ( t ) and ln( t ) ˆ w is bounded, and therefore, the direction of ˆ w dominates as t → ∞ .
Stage I: Loss Dynamics.
The goal of this stage is to characterize the dynamics of the loss and prove the convergence of GDM.
The core of this stage is to select a proper potential function ξ ( t ) , which is required to correlate with the training loss L and be non-increasing along the optimization trajectory.
For GD, since L is non-increasing with a properly chosen learning rate, we can pick ξ ( t ) =
L ( t ) .
However, as the update of GDM does not align with the direction of the negative gradient, training loss L ( t ) in GDM is no longer monotonously decreasing, and the potential function requires special construction.
Inspired by [36], we choose the following ξ ( t ) :
Lemma 1.
Let all conditions in Theorem 2 hold.
Define ξ ( t ) △ =
L ( w [Equation]
+
[Equation] ∥ [Equation]
Define C 1 as a positive real with C 1 ≜ σ 2 max H ℓ − 1 ( N L ( [Equation] ) 2 N η .
We then have
[Equation]
Remark 2.
Although this potential function is obtained by [36] by directly examining Taylor’s expansion at w ( t ) , the proof here is non-trivial as we only require the loss to be locally smooth instead of globally smooth in [36].
We need to prove that the smoothness parameter along the trajectory is upper bounded.
We defer the detailed proof to Appendix C.1.1.
By Lemma 1, we have that ξ ( t ) is monotonously decreasing by gap 1 − C 1 η ∥ w ( t + 1) − w ( t ) ∥ 2 .
As ξ (1) =
L ( w (1)) is a finite number, we have (cid:80) ∞ t =1 ∥ w ( t + 1) − w ( t ) ∥ 2 < ∞ .
By that (1 − β ) η ∇L ( w ( t ))
=
( w ( t + 1) − w ( t ))
− β ( w ( t ) − w ( t − 1)) , it immediately follows that (cid:80) [Equation]
∥∇L ( w ( t [Equation]
∥ 2 < ∞ .
Stage II.
Parameter Dynamics.
The goal of this stage is to characterize the dynamics of the parameter and show that GDM asymptotically converges (in direction) to the max-margin solution ˆ w .
To see this, we define a residual term r ( t ) △ = w ( t ) − ln( t ) ˆ w − ˜ w with some constant vector ˜ w (specified in Appendix C.1.2).
If we can show the norm of r ( t ) is bounded over the iterations, we complete the proof as ln( t ) ˆ w will then dominates the dynamics of w ( t ) .
For simplicity, we use the continuous dynamics approximation of GDM [36] to demonstrate why r ( t ) is bounded:
[Equation]
We start by directly examining the evolution of ∥ r ( t ) ∥ , i.e.,
[Equation]
which by integration by part leads to
[Equation]
( s ) d s (cid:29) d s (cid:19) ,
We then check the terms one by one:
• (cid:82) T 1 (cid:10) r
( s ) , −∇L ( w ( s ))
− 1 s ˆ w (cid:11) d s : This term also occurs in the analysis of GD [34], the analysis of which can be generalized as it does not depend on the form of w ( s ) ; • ⟨ r ( T ) , − d w ( T ) d t ⟩ : as shown in Stage I, d w ( T ) d t → 0 (i
e., w ( T ) − w ( T − 1) → 0 in the discrete case) as T → ∞ .
Thus, this term is o ( ∥ r ( T ) ∥ ) ; • (cid:82) T 1 (cid:68) d r ( s ) d s , d w ( s ) d s (cid:69) d s : finite due to d r ( s ) d s = d w ( s ) d s [Equation]
s is finite [Equation] (i.e., (cid:80) [Equation]
∥ w ( t + 1) − w ( t ) ∥ 2 < ∞ in the discrete case) by Stage I, and the mean-value inequality. Please
The derivation of this property is in the same manner as in the discrete case: we choose the potential function as ξ ( t ) = L ( w ( t )) + β 2(1 − β )
∥
d w ( t ) d t ∥ 2 . The derivative of ξ ( t ) is d ξ ( t ) d t = ⟨∇L ( w ( t )) , d w ( t ) d t ⟩ + β 1 − β ⟨ d 2 w ( t ) d t 2 , d w ( t ) d t ⟩ = −∥ d w ( t ) d t ∥ 2 . Therefore, ξ (0) − ξ ( T ) {+= (cid:82) T 
+}
∥ d w ( t ) d t ∥ 2 d t .
Then, taking T → ∞ yields the conclusion.
Putting them together, we show that ∥ r ( T ) ∥ 2 + o ( ∥ r ( T ) ∥ ) is upper bounded over the iterations, which immediately leads to that ∥ r ( T ) ∥ is bounded.
Applying similar methodology to the discrete update rule, we have the following lemma (the proof can be found in Appendix C.1.2).
Lemma 2.
Define potential function g : Z + → R as
[Equation]
g ( t ) is upper bounded, which further indicates ∥ r ( t ) ∥ is upper bounded.
Remark 3.
Our technique for analyzing GDM here is essentially more complex and elaborate than that for GD in [34] due to the historical information of gradients GDM.
The approach in [34] cannot be directly applied.
It is worth mentioning that we provide a more easy-to-check condition for whether r ( t ) is bounded, i.e., "is g ( t ) upper-bounded?".
This condition can be generalized for other momentum-based implicit regularization analyses.
E.g., SGDM and Adam later in this paper.
Tackle the difficulty brought by random sampling
In this section, we analyze the implicit regularization of SGDM.
Parallel to GDM, we establish the following implicit regularization result for SGDM:
Theorem 3.
Let Assumption 1, 2, and 3.
( S ) hold.
Let β ∈
[0 , 1) and η <
[Equation]
Then, with arbitrary initialization point w (1) , SGDM (w/. r) satisfies ∥ w ( t ) ∥
w ( t ) ∥ − ˆ w ∥ ˆ w ∥ ∥ = O (1 / log( t )) and L ( w ( t ))
= O ( 1 t ) , almost surely (a.s.).
Similar to the GDM case, Theorem 3 shows that the implicit regularization of SGDM under this setting is consistent with SGD (c.f.
[24] for the implicit regularization of SGD).
This matches the observations in practice (c.f. Section 7 for details), and is later supported by our experiments (e.g.,
Figure 1).
We add two remarks on the learning rate upper bound and extension to SGDM (w/. r).
Remark 4 (On the learning rate) .
Firstly, our learning rate upper bound 1 / (
[Equation]
exactly matches that of SGD 2 bγ 2 Hσ 4 max
[24] when β
= 0 , and matches that of SGD in terms of the order of σ max , H , and b
when β ̸ = 0 .
This indicates our analysis is tight.
Secondly, as the bound is monotonously increasing with respect to batch size b , Theorem 3 also sheds light on the learning rate tuning, i.e., the larger the batch size is, the larger the learning rate is.
Remark 5.
(On SGDM (w/o. r))
Theorem 3 can be similarly extended to SGDM (w/o. r).
We defer the detailed description of the corresponding theorem together with the proof to Appendix F.2.
Next, we show the proof sketch for Theorem 3.
The proof also contains two stages, where Stage II is similar to that for GDM.
However, we highlight that Stage I for SGDM is not a trivial extension of that for GDM .
The methodology used to construct GDM’s potential function fails for SGDM: roughly speaking, the change of ξ ( t ) (defined in Lemma 1) can be spitted into the "decrease term" and the "error term". The "decrease term" comes from the alignment between directions of the update of GDM and the negative gradient, and contributes to the decrease of ξ ( t ) . However, a gap remains between the directions of the update of GDM and the negative gradient. The "error term" measures this gap and potentially contributes to the increase of ξ ( t ) .
For GDM, both the terms are in the order of ∥∇L ( w ( t ))
∥ 2 while the "decrease term" has a larger coefficient, leading to the decrease of ξ ( t ) . For SGDM, the "decrease term" is in the order of ( E ∥∇L ( w ( t )) ∥ ) 2 , while the "error term" is in the order of E ( ∥∇L ( w ( t )) ∥ 2 ) , bigger than the "decrease term" in an unbounded magnitude.
Therefore, ξ ( t ) may no longer decrease.
We defer a detailed discussion to Appendix C.2.3.
On the other hand, we find the stochastic gradient ∇L B ( t ) ( w ) is special, in the sense that it satisfies the strong growth condition [32] as follows.
Lemma 3.
Let all the assumptions in Theorem 3 hold. Then, for every t , ∇L B ( t ) ( w ) satisfies the strong growth condition, i.e., ∀ w , E B ( t ) ∥∇L B ( t ) ( w ) ∥ 2 ≤ C 2 ∥∇L ( w ) ∥ 2 , where C 2 ≜ Nσ 2 max γ 2 b .
The proof is deferred to Appendix B.2.
SGD has been known to converge faster when the strong growth condition holds compared to the case when the variance of the stochastic gradient norm is bounded [32, 42].
However, to the best of our knowledge, convergence of SGDM under the strong growth condition has not been established. We then bridge this gap by proposing the following theorem. Theorem 4. Consider SGDM defined in Eq.
(2) with general L and ξ ( t ) .
Let ∇L ξ ( t ) ( w ) satisfy the following properties:
• (Affine noise variance
[Equation]
[Equation]
Then, with learning rate η ≤
[Equation] 1 , we have [Equation] To the best of our knowledge, Theorem 4 is the first to establish the convergence of SGDM under the affine noise variance condition [6] , which generalizes the strong growth condition and the bounded gradient variance condition.
Note that the affine noise variance condition with σ 0 = 0 is exact the strong growth condition, in which case Theorem 4 shows that 1 T (cid:80) Tt =1 E ∥∇L ( w ( t )) ∥decays with rate 1 T . Combining Lemma 3 and Theorem 4 completes the proof of Stage I for SGDM.
We then briefly state the proof idea of Theorem 4. Inspired by SGD’s simple update rule, we rearrange the update rule of SGDM such that only the gradient information of the current step is contained, i.e.,
[Equation]
By defining u ( t ) ≜ w ( t ) − β w ( t − 1) 1 − β = w ( t ) + β 1 − β ( w ( t ) − w ( t − 1)) , we have that u ( t ) is close to w ( t ) (differs by order of one-step update w ( t ) − w ( t − 1) ), and the update rule of u ( t ) only contains the current-step-gradient information ∇L ( w ( t )) .
We then select potential function as L ( u ( t )) , and a simple Taylor’s expansion directly leads to:
E [ L ( u ( t + 1))
|F t ] ≈ L ( u ( t )) − η ⟨∇L ( w ( t )) , ∇L ( u ( t )) ⟩ ≈ L ( u ( t )) − η ∥∇L ( w ( t )) ∥ 2 , i.e., L ( u ( t )) is a proper potential function. Summing the above equation across iterations completes the proof.
Analyze the effect of preconditioners 6.1 Implicit regularization of deterministic Adam
This section presents the implicit regularization of deterministic Adam, i.e., Adam without random sampling.
Theorem 5.
Let Assumption 1, 2, and 3.
( D ) hold.
Let 1 > β 2 > β 41 ≥ 0 , and the learning rate η is a small enough constant (The upper bound of learning rate is complex, and we defer it to
Appendix D.1).
Then, with arbitrary initialization point w (1) , deterministic Adam (Eq. 3) satisfies that ∥ w ( t ) ∥ w ( t ) ∥ − ˆ w ∥ ˆ w ∥ ∥ = O (1 / log( t )) and L ( w ( t ))
= O ( 1 t ) .
Remark 6 (On the β 1 and β 2 range) .
Almost all existing literature assume a time-decaying hyperparameter choice of β 1 or β 2 (c.f., [18, 3]).
On the other hand, our result proves that deterministic
Adam converges with constant settings of β 1 and β 2 , which agrees with the practical use.
Remark 7 ((Discussion on the results in [34])) .
[34] observe that, on a synthetic dataset, the direction of the output parameter by Adam still does not converge to the max-margin direction after 2 × 10iterations (but is getting closer).
At the same time, GD seems to converge to the max-margin direction.
However, this example does not contradict Theorem 5, as we study the asymptotic behavior.
In the example, it can be observed that the angle gap keeps decreasing after 10 5 iteration, which stands with our result.
Furthermore, in Figure 3 and 4 of our paper, we reproduce the experiment and find that the angle gap still keeps decreasing after extending the training time. Secondly, this synthetic dataset is ill-posed, which has large singular value and makes the training rather slow, and it can be observed that the training loss of Adam does not vary much until 10 4 iteration.
On the well-posed dataset adopted by (Figure 1, [34]), we observe that Adam will converge to the max-margin solution rapidly (please refer to Figure 1 for details).
We simply introduce the proof idea here and put the full proof in Appendix D. The proof of determin- istic Adam is very similar to that of GDM with minor changes in potential functions.
Specifically, ξ ( t ) in Lemma 1 is changed to ξ ( t ) ≜ L ( w ( t )) + 12 1 −
[Equation]
, and g ( t ) in Lemma 2 is changed to
[Equation]
The rest of the proof then flows similarly to the GDM case.
6.2 What if random sampling is added?
We have obtained the implicit regularization for GDM, SGDM, and deterministic Adam.
One may wonder whether the implicit regularization of stochastic Adam can be obtained.
Unfortunately, the gap can not be closed yet.
This is because an implicit regularization analysis requires the knowledge of the loss dynamics, little of which, however, has been ever known even for stochastic RMSProp (i.e., Adam with β 1 = 0 in Eq.
(3)) with constant learning rates.
Specifically, the main difficulty lies in bounding the change of conditioner
[Equation]
√
[Equation]
t ) across iterations, which is required to make the drift term ⟨∇L ( w ( t )) , E ( w ( t + 1) − w ( t )) ⟩ (derived by Taylor’s expansion of the epoch start from
Kt ) negative to ensure a non-increasing loss.
On the other hand, if we adopt decaying learning rates η t = η 1 √ t , [33] shows β 2 close enough to 1 , the following equation holds for stochastic RMSProp (w/o. r) (recall that K ≜
Nb is the epoch size)
[Equation]
Based on this result, we have the following theorem for stochastic RMSProp (w/o. r):
Theorem 6.
Let Assumptions 1, 2, and 3.
(S) hold.
Let β 2 be close enough to 1 .
Then, with arbitrary initialization point w (1) and decaying learning rate η t = η 1 √ t , stochastic RMSProp satisfies ∥ w ( t ) ∥ w ( t ) ∥ − ˆ w ∥ ˆ w ∥ ∥ = O (1 / log( t )) and L ( w ( t ))
=
O ( 1 √ t ) .
Remark 8 (On the decaying learning rate) .
The decaying learning rate is a "stronger" setting compared to the constant learning rate, both in the sense that GDM, SGDM, and deterministic Adam can be shown to converge to the max-margin solution following the same routine as Theorems 2, 3, and 5, and in the sense that we usually adopt constant learning rate in practice.
The proof is on the grounds of a novel characterization of the loss convergence rate derived from Eq.
(8), and readers can find the details in Appendix E.
[Figure]
1 σ 2 max .
Figure (b) shows (1).
all the optimizers converge to the max margin solution, and (2).
the asymptotic behaviors with & without momentum are similar.
The experimental observation support our theoretical results.
Discussions
Consistency with the Experimental Results.
We conduct experiments to verify our theoretical findings.
Specifically, we (1).
run GD, GDM, SGD, SGDM, and Adam on a synthetic dataset to observe their implicit regularization; (2).
run GD and Adam on ill-posed dataset proposed in [34] to verify Theorem 5; (3).
run SGD and SGDM on neural networks to classify the MNIST dataset and compare their implicit regularization.
The experimental observations stand with our theoretical results.
Furthermore, it is worth mentioning that experimental phenomenons that adding momentum will not change the implicit regularization have also been observed by existing literature [34, 23, 43].
Influence of hyperparameters on convergence rates.
Our results can be further extended to provide a precise characterization of the influence of the hyperparameters η and β on the convergence rate of (S)GDM.
Specifically, in Appendix F.4, we show that the asymptotic convergence rate of (S)GDM is C 1 η 1 t , where C is some constant independent of β and η .
Therefore, increase η can lead to a faster convergence rate (with learning rate requirements in Theorems 2 and 3 satisfied).
However, changing β does not affect the convergence rate, which is also observed in our experiments (e.g., Figure 1).
Furthermore, to the best of our knowledge, there still lacks the theoretical justification of the acceleration effect of the momentum except the strongly convex case.
On the other hand, we still can not provide a precise characterization of the exact convergence behavior of Adam in this case due to the ever-changing preconditioner.
As this work mainly focuses on the linear model, whether momentum and preconditioner can accelerate the training of non-linear model with the non-convex landscape is an exciting future direction and deserves further investigation.
Generalization Behavior of Adam.
Our results characterize the implicit regularization behavior of optimizers when the time is large enough, which can be changed by early stopping, especially for Adam. This is because the proof of Theorem 5 relies on that when the training time is large enough, the gradient is small and the adaptive learning rate [Equation]
is dominated by [Equation] , and then Adam behaves like GDM and thus converges to the max margin solution. This can also be verified by Figure 1, as the angle gap of deterministic Adam and RMSProp only start to decrease late in the training.
Gap Between The Linear Model and Deep Neural Networks.
While our results only hold for the linear classification problem, extending the results to the deep neural networks is possible.
Specifically, existing literature [21, 43] provide a framework for deriving implicit regularization for deep homogeneous neural networks.
However, the approach in [21, 43] can not be trivially applied to the momentum-based optimizers, as their proofs require the specific gradient-based updates to lower bound a smoothed margin (c.f., Theorem 4.1, [21]).
It remains an exciting work to see how our results can be expanded to GDM and Adam for deep neural networks.
Conclusion
This paper studies the implicit regularization of momentum-based optimizers in linear classification with exponential-tailed loss.
Our results indicate that for SGD and the deterministic version of Adam, adding momentum will not influence the implicit regularization, and the direction of the parameter converges to the L 2 max-margin solution.
Our theoretical results stand with existing experimental observations, and developed techniques such as the potential functions may inspire the analyses on other momentum-based optimizers.
Motivated by the results and techniques for linear cases in this paper, it has the potential to extend them to the homogeneous neural network in the future.
Another topic left for future work is to derive the implicit regularization of constant learning rate stochastic
Adam.
As discussed in Section 6.2, this topic is non-trivial, and it requires new techniques and assumptions to be developed.  Acknowledgement
This research is supported in part by CAS Project for Young Scientists in Basic Research under Grant
No. YSBR-034 and Innovation Project of ICT CAS under Grants No. E261090.