D OES MOMENTUM C HANGE THE I MPLICIT B IAS ON SEPARABLE DATA ?
Anonymous authors
Paper under double-blind review
A BSTRACT
The momentum acceleration technique is widely adopted in many optimization algorithms.
However, the theoretical understanding on how the momentum affects the generalization performance of the optimization algorithms is still unknown.
In this paper, we answer this question through analyzing the implicit bias of momentum-based optimization.
We prove that on the linear classiﬁcation problem with separable data, both SGD with momentum and Adam (without stochasticity) converge to the L 2 max-margin solution for exponential-tailed loss, which is the same as vanilla gradient descent.
That means, these optimizers with momentum acceleration still converge to a model with low complexity, which provides guarantees on their generalization.
Technically, to overcome the difﬁculty brought by the error accumulation in analyzing the momentum, we construct new Lyapunov functions as a tool to analyze the gap between the model parameter and the max-margin solution.
I NTRODUCTION
It is widely believed that the optimizers have implicit bias in terms of selecting output parameters among all the local minima on the landscape (Neyshabur et al., 2015; Keskar et al., 2017; Wilson et al., 2017).
It is shown in the analysis of Adaboost that the coordinate descent would converge to the L 1 max-margin solution for the linear classiﬁcation task with exponetial-tailed loss ((Schapire & Freund, 2013; Telgarsky, 2013)). Latter, Soudry et al. (2018) shows that gradient descent would converge to the L 2 max-margin solution under the same setting, which mirrors its good generalization property in practice.
Since then, many efforts have been taken on analyzing the implicit bias of various local-search optimizers, including stochastic gradient descent (Nacson et al., 2019), steepest descent (Gunasekar et al., 2018a), AdaGrad (Qian & Qian, 2019) and optimizers for homogeneous neural networks (Lyu & Li, 2019; Ji & Telgarsky, 2020; Wang et al., 2021).
However, though the momentum acceleration technique is widely adopted in the optimization algorithms in both convex and non-convex learning tasks (Sutskever et al., 2013; Vaswani et al., 2017;
Tan & Le, 2019), the understanding on how the momentum would affect generalization performance of the optimization algorithms is still unclear.
A natural question is:
Can we theoretically analyze the implicit bias of momentum-based optimizers?
In this paper, we take the ﬁrst step to analyze the convergence of momentum based optimizers and unveil their implicit bias.
Speciﬁcally, we study the classiﬁcation problem with linear model and exponential-tailed loss using Stochastic Gradient Descent with Momentum (SGDM) and Adam optimizers.
We consider the optimizers with constant learning rate and constant momentum hyperparameters, which are widely adopted in practice, e.g., the default setting in popular machine learning frameworks (Paszke et al., 2019) and in experiments (Xie et al., 2017).
We note that Gradient Descent with Momentum (GDM) can be viewed as a special case of SGDM, and naturally share the properties for SGDM.
Our main results are summarized in Theorem 1.
Theorem 1 (informal) .
With linear separable dataset S , for SGDM and Adam (without stochasticity, abbreviated as w/s latter), the loss converges to 0 with rate O ( 1 t ) where t is the number of iterations, the parameter norm diverges to inﬁnity, and direction of parameters converges to the direction of the
L 2 max-margin solution.
Theorem 1 states SGDM converges to the L 2 max-margin solution, which is the same as SGD, indicating that momentum does not affect the convergent direction.
The good generalization behavior of the output parameters of SGDM is well validated as the margin of a classiﬁer is positively correlated with its generalization error (Jiang et al., 2019).
This is supported by existing experimental observations (c.f. Figure 1, (Soudry et al., 2018) and Figure 2, (Nacson et al., 2019)).
Similar claims hold for Adam (w/s), which is also well supported by empirical results in Wang et al.
Our contributions are signiﬁcant in terms of the following aspects:
• We establish the implicit bias of the momentum based optimizers, an open problem since the initial work Soudry et al.
The momentum based optimizers are widely used in practice and our theoretical characterization deepens the understanding on their generalization property, which is important by its own.
Technically, we propose a new Lyapunov function to analyze the convergence of SGDM, which helps to bound the sum of squared gradients along the training trajectory.
Compared to the usual one, the new Lyapunov function depends on a middle variable of an alternative update rule of SGDM, which helps to capture the historical dependence in the momentum update.
To our knowledge, such a technique has not been exploited ever before and can be of independent interest for convergence analysis of momentum-based optimizers.
We then construct a new Lyapunov function to bound the difference of learned parameters and the scaled max-margin solution, which ﬁnally leads to the direction convergence.
This Lyapunov function provides a direct way to establish the convergence to the desired direction.
Organization of This Paper.
Section 2 collects further related works on the implicit bias of ﬁrst order optimizers and convergence of momentum-based optimizers.
Section 3 shows basic settings and assumptions which will be used throughout this paper.
Section 4 studies the implicit bias of
GDM as a warm up, while Section 5 and Section 6 explore respectively the implicit bias of SGDM and Adam (w/s).
Discussions of these results are put in Section 7.
F URTHER R ELATED W ORKS
Implicit Bias of First-order Optimization Methods.
Soudry et al.
(2018) prove that gradient descent on linear classiﬁcation problem with exponential-tailed loss converges to the direction of the max L 2 margin solution of the corresponding hard-margin Support Vector Machine.
Nacson et al.
(2019) extend the results in (Soudry et al., 2018) to the stochastic case, proving that the convergent direction of SGD is the same as GD almost surely.
Qian & Qian (2019) go beyond the vanilla gradient descent methods and consider the AdaGrad optimizer instead.
They prove that the convergent direction of AdaGrad has a dependency on the optimizing trajectory, which varies according to the initialization.
Ji & Telgarsky (2021) propose a primal-dual analysis framework for the linear classiﬁcation models, and prove a faster convergent rate of the margin by increasing the learning rate according to the loss.
Based on (Ji & Telgarsky, 2021), (Ji et al., 2021) design another algorithm with an even faster convergent rate of margin by applying the Nesterov’s Acceleration Method on the dual space.
However, the corresponding form of the algorithm on the primal space is no longer a
Nesterov’s Acceleration Method nor GDM, which is signiﬁcantly different from our settings.
On the other hand, there is another line of work trying to extend the result in the linear case to deep neural networks.
Ji & Telgarsky (2018); Gunasekar et al.
(2018b) study the deep linear network and
Soudry et al.
(2018) study the two-layer neural network with ReLU activation.
Lyu & Li (2019) propose a framework to analyze the asymptotic direction of GD on homogeneous neural networks, proving that given there exists a time the network achieves 100% training accuracy, GD will converge to some KKT point of the L 2 max-margin problem.
Wang et al.
(2021) extend the framework of Lyu & Li (2019) to adaptive optimizers, and prove RMSProp and Adam without momentum have the same convergent direction as GD, while AdaGrad doesn’t.
The results (Lyu & Li, 2019; Wang et al., 2021) indicate that results in the linear model can be extended to deep homogeneous neural networks, and suggest that the linear model is a proper start point to study the implicit bias.
There are also works on the implicit bias of regression problems with bounded optimal points and interesting readers can refer to (Rosasco & Villa, 2015; Lin & Rosasco, 2017; Ali et al., 2020) etc. for details.
Convergence of Momentum-Based Optimization Methods.
For convex optimization problems, the convergence rate of Nesterov’s Acceleration Method has been proved in Nesterov (1983).
In contrast, although GDM (Polyak’s Heavy-Ball Method) was proposed in (Polyak, 1964) prior to the
Nesterov’s Acceleration Method, the convergence of GDM on convex loss with Lipschitz gradient was not solved until Ghadimi et al.
(2015) provides an ergodic convergent result for GDM, i.e., the convergent result for the running average of the iterates.
However, the ergodic result is undesired under many learning scenarios, e.g., in classiﬁcation tasks, the optimization algorithms usually output the parameters of the last step.
To the best of our knowledge, there is only two works on the non-ergodic analysis of (S)GDM: Sun et al.
and Tao et al.
Sun et al.
(2019) prove that if the training loss is coercive (the training loss goes to inﬁnity whenever parameter norm goes to inﬁnity), convex, and globally smooth, then, with constant momentum hyper-parameter, the training loss converges to minima with rate O ( t − 1 ) .
Tao et al.
(2021) analyze a case when momentum coefﬁcient increases to 1 and the gradient is bounded all over the parameter space, showing SGDM can improve the convergence rate of SGD by a factor log( t ) .
There are also works on the gradient norm convergence of SGDM under various settings (Yan et al., 2018;
Yu et al., 2019; Liu et al., 2020) and on the investigation of momentum-based method from the view point of dynamics (Sarao Mannelli & Urbani, 2021).
However, there is no existing work on the implicit bias of momentum-based optimizers for the classiﬁcation problem, which is ﬁrst analyzed by this paper.
P RELIMINARIES
In this paper, we focus on the linear model with exponential-tailed loss.
We ﬁrst derive the results for binary classiﬁcation, then we show that the methodology can be easily extended to the multi-class classiﬁcation problem.
Problem setting.
The dataset used for training is deﬁned as S =
( x i , y i )
Ni
=1 , where x i ∈ R
d is the i -th feature, and
y i ∈
R is the i -th label ( i = 1 , 2 , · · · , N ).
We will use the linear model to ﬁt the label: for any feature x ∈ R d and parameter w ∈ R d , the prediction is given by (cid:104) w , x (cid:105) .
For binary classiﬁcation, given any data z i =
( x i , y i ) ∈ S , the individual loss for parameter w is given as (cid:96) ( y i (cid:104) w , x i (cid:105) ) .
A common setting is to ensemble the feature and label together as ˜ x i =
y
i x i (there is a mapping T : ( x , y )
→ yx , and ˜ x
i is T (( x i , y i )) ).
The individual loss can be rewritten as
[Equation]
The optimization target is deﬁned as the averaged loss:
[Equation]
Without loss of generality, we consider the case with normalized data 1 , that is, (cid:107) ˜ x
i (cid:107) ≤ 1 , ∀ i ∈ [ N ] .
Optimizer.
Here we will introduce the update rules of SGDM and Adam (w/s).
SGDM can be viewed as a stochastic version of GDM by randomly choosing a subset of the dataset to update.
Speciﬁcally, the update rule of SGDM is given as
[Equation]
where B ( t ) is a subset of S with size b which is sampled independently and uniformly with replacement, and L B ( t ) is deﬁned as L B ( t ) ( w )
= (cid:80) z ∈ B ( t ) ˜ (cid:96) ( w , z ) b .
We also deﬁne F t as the sub-sigma ﬁeld such that { w ( t ) } ∞ t =1 is adapted with respect to Filtration {F t } ∞ t =1 .
The Adam (w/s) can be viewed as a variant of GDM in which the preconditioner is adopted, whose form is characterized as follows:
[Equation]
where 1 √ ˆ ν ( t − 1)+ ε 1 d ) is called the preconditioner at step t .
Assumptions:
The analysis of this paper are based on three common assumptions in existing literature (ﬁrst proposed by (Soudry et al., 2018)), respectively on the separability of the dataset, the individual loss behaviour at the tail , and the smoothness of the individual loss.
We list them as follows:
Assumption 1 (Linearly Separable Dataset) .
There exists one parameter w ∈ R d , such that
[Equation]
Assumption 2 (Exponential-tailed Loss) .
The individual loss (cid:96) is exponential-tailed, i.e., • Differentiable and monotonically decreasing to zero, with its derivative also converging to zero, i.e., lim x →∞ (cid:96) ( x ) =
lim x →∞ (cid:96) (cid:48) ( x ) = 0 , and (cid:96) (cid:48) ( x ) < 0 ∀ x ; • Close to exponential loss when x is large enough,
i
e., there exist positive constants c, a, K, µ + , µ − , x + , x − and x 0 , such that,
[Equation]
Assumption 3 (Smooth Loss) .
Either of the following assumptions holds regarding the case:
(D): (Deterministic Case)
The individual loss (cid:96) is locally smooth, i e., for any s 0 ∈ R , there exists a positive real H s 0 , such that ∀ x, y ≥ s 0 , | (cid:96) (cid:48)
( x ) − (cid:96) (cid:48) ( y ) | ≤ H s 0 | x − y | .
(S): (Stochastic Case)
The individual loss (cid:96) is globally smooth, i.e., there exists a positive real H , such that ∀ x, y ∈ R , | (cid:96) (cid:48)
( x ) − (cid:96) (cid:48) ( y ) | ≤ H | x − y | .
We provide explanations of these three assumptions respectively.
Based on Assumption 1, we can formally deﬁne the margin and the maximum margin solution of an optimization problem, which is introduced in Deﬁnition
Deﬁnition 1.
Let the margin γ ( w ) of parameter w deﬁned as the lowest score of the prediction of w over the dataset S , i.e.
, γ ( w ) =
min
˜ x i ∈T ( S ) (cid:104) w , ˜ x
i (cid:105) .
By Assumption 1 and the positive homogeneous of γ , γ ( ˆ w γ ( ˆ w ) )
= 1 , and thus we deﬁne the maximum margin solution ˆ w and the L 2 max margin γ of the dataset S as follows:
[Equation]
(cid:107) Since (cid:107) · (cid:107) 2 is strongly convex and set { w : γ ( w ) ≥ 1 } is convex, ˆ w is uniquely deﬁned.
Assumption 2 constraints the loss to be exponential-tailed, which is satisﬁed by many popular choices of (cid:96) , including the ( (cid:96) exp ( x ) =
e − x ) and the logistic loss ( (cid:96) log ( x ) =
log(1 + e − x ) ).
Also, as c and a can be respectively absorbed by resetting the learning rate and data as η = cη and x i = a x i , without loss of generality, in this paper we only analyze the case that c = a = 1 .
The globally smooth assumption (Assumption 3. (S)) is strictly stronger than the locally smooth assumption (Assumption 3.
(D)).
One can easily verify that both the exponential loss and the logistic loss meet Assumption 3.
( D ), and the logistic loss also meets Assumption 3.
( S ).
W ARM UP : I MPLICIT B IAS OF GDM
In this section, we study the implicit bias of GDM as a warm up.
The update rule of GDM is
[Equation]
which is a determined case of SGDM, with batch size b = N and thus without randomness.
Although the analysis of GDM is essentially easier than that of SGDM, it helps us gain a better understanding of this problem, and helps us to demonstrate several techniques which will be applied latter.
The formal theorem of the implicit bias of GDM is as follows:
Theorem 2.
Let Assumptions 1, 2, and 3.
( D ) hold.
Let β ∈
[0 , 1) and η <
[Table]
− β H [Table] [Table] w 1)) .
Then, for almost every data set S 2 , with arbitrary initialization point w (1) , GDM (Eq.
(5) ) satisﬁes that w ( t ) − log( t )
ˆ w is bounded as t → ∞ , and lim t →∞ w ( t ) (cid:107) w ( t ) (cid:107)
[Equation]
Before we move on to the proof, we would like to provide some explanations of the results in Theorem 2.
To begin with, Theorem 2 adopts an constant momentum hyper-parameter, which agrees with the practice use (e.g., β is set to be 0 .
9 (Xie et al., 2017)).
Also, Theorem 2 puts no restriction on the range of β , which allows wider choices of hyper-parameter tuning.
Furthermore, Theorem 2 shows the implicit bias of GDM agrees with GD in linear classiﬁcation with exponential-tailed loss (c.f.
Soudry et al.
(2018) for results on GD), and this consistency can be veriﬁed by existing results (c.f.
Section 7 for detailed discussions).
We then present a proof sketch of Theorem 2, which is divided into two parts: we ﬁrst prove that the sum of squared gradients is bounded, which indicates both the loss and the norm of gradient converge to 0 ; we then show the difference between w ( t ) and log( t ) ˆ w is bounded, and therefore, the direction of ˆ w dominates as t → ∞ .
Stage I: Bound the sum of squared gradients.
The core of Stage I is to select a proper Lyapunov function ξ ( t ) , which is required to correlated with the training loss L and be non-increasing along the optimization trajectory.
For GD, since L itself is non-increasing with properly chosen learning rate, we can just pick ξ ( t ) =
L ( t ) .
However, as the update of GDM doesn’t align with the direction of negative gradient, training loss L ( t ) in GDM is no longer monotonously decreasing, and the
Lyapunov function requires special construction.
A choice of ξ ( t ) is proposed as the follows:
Lemma 1.
Let all conditions in Theorem 2 hold.
Deﬁne
[Equation]
Let C 1 be a positive real, s.t., η =
[Equation]
We then have
[Equation]
Remark 1.
Although this Lyapunov function is obtained by (Sun et al., 2019) by directly examining the Taylor’s expansion at w ( t ) , the proof here is non-trivial as we only requires the loss to be locally smooth instead of globally smooth in (Sun et al., 2019), and the Taylor’s expansion can only be applied to w ( t +1) if it’s ensured that all parameters on the line α w ( t )+(1 − α ) w ( t +1) ( α ∈ (0 , 1) ) have training loss no larger than L ( w (0)) .
To prove Lemma 1, we deﬁne w ( t + α ) =
α w ( t ) + (1 − α ) w ( t + 1) for any t ∈ Z + and α ∈ (0 , 1) , and prove a generalized version of Eq.
[Equation]
The proof idea is that as long as Eq.
(7) holds for all time in [1 , t + α ) , the training loss across [1 , t
+ α ] will be smaller than L ( w (1)) , and the strict inequality in Eq.
(7) holds.
Consequently, there exists some small enough positive real ε , such that Eq.
(7) holds for all time in [1 , t
+ α + ε ) , and we are able to extend the feasible set where Eq.
(7) holds.
The formal description of the generalized lemma and its corresponding proof is deferred to Appendix B.1.1.
By Lemma 1, we have that ξ ( t ) is monotonously decreasing by gap (1 − β )(1 − c )
α 2 η
[Equation]
As ξ (1) =
L ( w (1)) is a ﬁnite number, we have (cid:80) ∞ t =1 (cid:107) w ( t + 1) − w ( t ) (cid:107) 2 < ∞ , which further leads to the following corollary by the negative derivative of the individual loss and separable data:
Corollary 1.
Let all conditions in Theorem 2 hold.
We have, (cid:80) ∞ t =1 (cid:107)∇L ( w ( t )) (cid:107) 2 < ∞ .
The proof of Corollary 1 can be found in Appendix B.1.1.
Resulted from Corollary 1, we have (cid:107)∇L ( w ( t ))
(cid:107) 2 → 0 as t → ∞ , which by the exponential-tailed loss assumption further leads to lim t →∞ L ( w ( t ))
= 0 .
Stage II.
Bound the difference between w ( t ) and log( t ) ˆ w .
It seems natural to directly bound (cid:107) w ( t ) − log( t )
ˆ w (cid:107) across all iterations.
Soudry et al.
(2018) and Nacson et al.
(2019) indeed follow this routine by showing the norm of r ( t ) (cid:52) =
w ( t ) − log( t )
ˆ w − ˜ w is bounded, where ˜ w is some constant vector satisfying e (cid:104) ˜ w , ˜ x (cid:105) recovers the coefﬁcient of support vector ˜ x in ˆ w .
However, their analyses rely on the fact that w ( t + 1) − w ( t ) equals − η ∇L ( w ( t )) , which has a simple form, i.e.,
For GD : (cid:107) r
( t + 1) (cid:107) 2 − (cid:107) r ( t ) (cid:107)
[Equation]
However, it no longer holds for GDM, as w ( t +1) − w ( t ) is a exponentially-decayed sum of gradients at w ( s ) , s ≤ t .
To handle this problem, we propose a novel Lyapunov’s function for the direction convergence, concluded as the following Lemma.
Lemma 2.
Let all conditions in Theorem 2 hold.
Then, (cid:107) r ( t ) (cid:107) is bounded if and only if the function g ( t ) is upper bounded, where g : Z + → R is deﬁned as
[Equation]
Furthermore, we have (cid:80) ∞ t =1 ( g ( t + 1) − g ( t )) is upper bounded.
The ﬁrst claim in Lemma 2 provides an alternative approach to verify that (cid:107) r ( t ) (cid:107) is bounded, while the second claim shows this approach can be fulﬁlled.
The motivation of this lemma is to construct a easy-to-verify criterion (i.e., g ( t ) is upper bounded) of (cid:107) r ( t ) (cid:107) is upper bounded.
To demonstrate how g ( t ) is constructed intuitively, we consider the continuous dynamics approximation of GDM, i.e.,
[Equation]
We then calculate the 12 (cid:107) r ( t ) (cid:107) 2 − 12 (cid:107) r (1) (cid:107) 2 by applying the integral of the dynamics as (cid:90)
[Equation]
where the last equation is due to Integration by part.
We denote ˜ g ( t ) =
[Equation]
( (cid:104) r ( t ) [Equation] d w ( t )
[Equation]
(cid:105) − (cid:82) t 1 (cid:104)
d r ( s ) d s , d w ( s )
d s (cid:105) d s )+ 12 (cid:107) r ( t ) (cid:107) 2 , and it can be veriﬁed the derivative of ˜ g
( t ) is (cid:104) r
( t ) , − η 1 − β ∇L ( w ( t )) − 1 t ˆ w (cid:105) .
By replacing d w ( t ) d t as w ( t ) − w ( t − 1) , and d r ( t ) d t as r ( t ) − r ( t − 1) in ˜ g
( t ) , we obtain g ( t ) , and g ( t + 1) − g ( t ) has the form (cid:104) r ( t ) , − η 1 − β ∇L ( w ( t ))
− log t +1 t ˆ w (cid:105) , which can be analyzed following the similar routine as (Soudry et al., 2018).
The proof of the second claim is completed.
On the other hand, the core of the proof of Lemma 2 is that 12 (cid:107) r ( t ) (cid:107) 2 + β 1 − β (cid:104) r
( t ) , w ( t ) − w ( t − 1) (cid:105) is bounded if and only if 12 (cid:107) r ( t ) (cid:107) 2 is bounded, while (cid:104) r ( τ ) − r ( τ − 1) , w ( τ ) − w ( τ − 1) (cid:105) is an absolute convergent based on (cid:80) ∞ t =1 (cid:107) w ( t + 1)
− w ( t ) (cid:107) 2 < ∞ .
T ACKLE THE DIFFICULTY BROUGHT BY RANDOM SAMPLING
In this section, we analyze the implicit bias of SGDM.
The randomness introduced by random sampling makes the analysis of GDM no longer work for SGDM.
As an example, if we want to follow the same routine of the GDM case to show L ( w ( t )) + β 2 η E (cid:107) w ( t ) − w ( t − 1) (cid:107) 2 is a Lyapunov function of SGDM, we can only have
[Equation]
As the squared ﬁrst momentum is smaller than the second momentum, E L ( w ( t )) + β 2 η E (cid:107) w ( t ) − w ( t − 1) (cid:107)
may no longer be monotonously decreasing for every β ∈ [0 , 1) .
Furthermore, if the margin γ is small, β will be required to be upper bounded by a small number, which contradicts to the relatively large choice of β in practice, e.g. 0 .
9 (We defer a detailed discussion to Appendix B.2.3).
To tackle this problem, we propose a new Lyapunov function, with which we derive the following theorem for the implicit bias of SGDM:
Theorem 3.
Let Assumption 1, 2, and 3.
( S ) hold.
Let β ∈
[0 , 1) and η < min { 1 − β 1+ HNbγ 2 , (1 − β ) 3 γ 4 b 2 H 2 N 3 β 2 } .
Then, for almost every data set S , with arbitrary initialization point w (1) , SGDM (Eq.
(1) ) satisﬁes w ( t ) − log( t ) ˆ w is bounded as t → ∞ and lim t →∞ w ( t ) (cid:107) w ( t [Equation] =
[Equation] almost surely (a.s.).
To the best of our knowledge, this is the ﬁrst convergence analysis of SGDM with no restriction on the gradient norm and with constant learning rates.
Also, as both 1 − β 1+ HNbγ 2 and (1 − β ) 3 γ 4 b 2 H 2 N 3 are monotonously increasing with respect to batch size b , Theorem 3 also sheds light on the learning rate tuning, i.e., the larger the batch size is, the larger the learning rate is.
Furthermore, similar to the GDM case,
Theorem 3 shows the implicit bias of SGDM under this setting is consistent with SGD (c.f.
(Nacson et al., 2019) for the implicit bias of SGD).
This matches the observations in practice (c.f. Sectionfor details).
Remark 2.
In practice, mini-batch SGDM are more widely adopted, whose update rule differs from SGDM only by the way obtaining B ( t ) .
Speciﬁcally, within the T -th epoch E ( T ) = { KT + 1 , · · · , KT + T } ( K is the length of one epoch), { B ( t ) } t ∈ E ( T ) is a randomly and uniformly partition of S .
One may wonder whether the same result hold for mini-batch SGDM.
The answer is "Yes", with the a.s. condition removed in this case.
We defer the detailed description of the corresponding theorem together with the proof to Appendix D.1.
The proof sketch follows the same framework as that for GDM by dividing the proof into two stages.
However, the implementations in both stages differ, among which the proof of Stage I is signiﬁcantly distinctive, as a new Lyapunov function is proposed.
Stage I: Bound the sum of squared gradients.
To obtain the new Lyapunov function, we ﬁrst present a lemma to provide an alternative form of update rule of SGDM:
Lemma 3.
Deﬁne ˜ η
(cid:52) =
η 1 − β , and u (1) (cid:52) =
w (1) .
The update rule of SGDM (Eq.
(1) ) is equivalent to
[Equation]
By a simple rearrangement of the second equation, we have u ( t +1) =
w ( t )+ 1 1 − β ( w ( t +1) − w ( t )) , which differs from w ( t + 1) only by a larger step size updated from w ( t ) .
The following lemma then indicates that E L ( u ( t )) is a proper selection of Lyapunov function.
Lemma 4.
Let all conditions in Theorem 3 hold.
Then, we have
[Equation]
and
[Equation]
E L ( u ( t )) may be not monotonously decreasing.
However, the upper bound of E L ( u ( t )) , i.e.,
L ( u (1)) − (cid:80) ts
[Equation]
‘ E (cid:107)∇L ( w ( s )) (cid:107) 2 , is monotonously decreasing by ˜
η 4 E (cid:107)∇L ( w ( t ))
(cid:107) 2 at step t , and leads to the sum of expected squared gradients being ﬁnite along the trajectory (which further indicates the sum of squared gradients is ﬁnite, a.s.).
The proof idea of Lemma 4 is the expectation of the ﬁrst order Taylor’s expansion of L at w ( t ) has the form
[Equation]
As L is H smooth, we have ∇L ( u ( t ))
=
∇L ( w ( t ))
+ O ( (cid:107) u
( t ) − w ( t ) (cid:107) ) , where u ( t ) − w ( t ) =
β [Equation]
[Equation] = − [Equation] s ∇L B ( s ) ( w ( s )) is proportional to η .
Therefore, when η is small, −(cid:104)∇L ( u ( t )) , ∇L ( w ( t )) (cid:105) is close to −(cid:107)∇L ( w ( t ))
(cid:107) 2 , and the coefﬁcient of −(cid:107)∇L ( w ( t ))
(cid:107) 2 becomes dominant.
Stage II.
Bound the difference between w ( t ) and log( t ) ˆ w .
The Lyapunov function used by Stage
II of SGDM is the same as the g ( t ) in the GDM case.
As we have proved the sum of squared gradient along the trajectory is bounded, one can easily obtain that (cid:80) ∞ t =1 (cid:107) w ( t + 1) − w ( t ) (cid:107) 2 is bounded, and (cid:107) r ( t ) (cid:107) being bounded is still equivalent to that g ( t ) is upper bounded.
However, when it comes to the detailed calculation of analyzing g ( t + 1) − g ( t ) , it differs from the GDM case due to the random subset.
We defer the proof to Appendix B.2.2.
A NALYZE THE EFFECT OF PRECONDITIONERS
When it comes to the analysis of Adam (w/s), the effect of the preconditioner should be taken into consideration when designing the corresponding Lyapunov functions.
To tackle this problem, we incorporate the preconditioner into the Lyapunov function, and obtain the implicit bias of Adam (w/s) as follows:
Theorem 4.
Let Assumption 1, 2, and 3.
( D ) hold.
Let 1 > β 2 > β 41 ≥ 0 , and the learning rate η is a small enough constant (The upper bound of learning rate is complex, and we defer it to Appendix
C.1).
Then, for almost every data set S , with arbitrary initialization point w (1) , Adam (w/s) (Eq.
(2) ) satisﬁes that w ( t ) − log( t )
ˆ w is bounded as t → ∞ , and lim t →∞ w ( t ) (cid:107) w ( t ) (cid:107)
= ˆ w (cid:107) ˆ w (cid:107) .
Remark 3.
Existing literature usually assume a time-decaying hyperparameter choice of β 1 or β(c.f.
, (Kingma & Ba, 2014; Chen et al., 2018)).
To the best of our knowledge, the only work analyzing
Adam with constant β 1 and β 2 is (Reddi et al., 2019), which assumes β 2 > β 21 (stronger than our assumption).
Our result indicates that the ranges of β 1 and β 2 can be broader in hyper-parameter selection.
We still start the proof sketch by proving the sum of squared gradients is ﬁnite in Stage I.
Compared to GDM, the difference is that in Stage I, we will also prove the loss converges to 0 with rate Θ( t − 1 ) .
This property will be used in Stage II to analyze the effect of preconditioner on implicit bias.
Stage I: Bound the sum of squared gradients.
The next lemma can be viewed as an extension of
Lemma 1 by taking the preconditioner into consideration, and characterizes the one-step loss update in Adam (w/s).
Lemma 5.
Let all conditions in Theorem 4 hold.
Then, for any t ≥ 1 ,
[Equation]
The difference between the proof of Lemma 5 and Lemma 1 is that we need to handle the gap between the preconditioners at step t and step t + 1 , this leads to a amplifying factor
[Equation]
2 , which, however, is smaller than the shrinking factor
[Equation]
) introduced by the gap between the coefﬁcients of momentum ˆ m ( t ) and ˆ m ( t − 1) .
As t → ∞ , both β t 1 → 0 and ( cβ 1 ) t → 0 , and we can obtain that ξ ( t ) [Equation]
= L ( w ( t ))
+
[Equation]
function
[Equation]
Based on Lemma 5, we can prove the following asymptotic rate of L .
Lemma 6.
Let all conditions in Theorem 4 hold.
Then,
[Equation]
The proof idea of Lemma 6 is that by the exponential-tailed property, when time is large enough, the training loss can be bounded by the gradient, and further bounded by (cid:107) 4 (cid:112)
ε 1 d + ˆ ν ( t − 1) (cid:12) ( w ( t ) − w ( t − 1))
(cid:107) .
Consequently, ξ ( t + 1) − ξ ( t ) ≤ −O ( ξ ( t )) , and ξ ( t ) =
O ( t − 1 ) .
Stage II.
Bound the difference between w ( t ) and log( t ) ˆ w .
The Lyapunov function for the direction difference is also modiﬁed according to the preconditioner, as introduced in Lemma 7:
Lemma 7.
Let all conditions in Theorem 4 hold.
Then, (cid:107) r ( t ) (cid:107) is bounded if and only if g ( t ) is upper bounded, where g ( t ) is deﬁned as follows.
[Equation]
Furthermore, we have (cid:80) ∞ s =1 ( g ( t + 1) − g ( t )) is upper bounded.
The ﬁrst claim of Lemma 7 is similar to that in GDM and SGDM case.
However, when we come to the second claim, simple calculation of g ( t + 1) − g ( t ) leads to
[Equation]
This is where we need Lemma 6, which bounds the gap between (cid:112) ε 1 d + ˆ ν ( t ) and √ ε by O ( t − 2 ) and makes (cid:80) ∞ t =1 (cid:104) r ( t ) , ( √ ε − (1 − β t 1 ) (cid:112) ε 1 d
+ ˆ ν ( t ))
(cid:12) ( w ( t + 1) − w ( t ))
(cid:105) an absolute continuous sequence.
The second term (cid:104) r ( t ) , −√ ε log( t +1 t )
ˆ w − η 1 − β ∇L ( w ( t )) (cid:105) can be tackled by following the same routine as the GDM/SGDM case.
Consequently, the proof of Lemma 7 is completed.
Till now, we have obtained the implicit bias for GDM, SGDM and Adam (w/s), one may wonder whether the implicit bias of stochastic Adam can be obtained.
Here, we make some discussions here and put its further investigation for future works.
First, our analysis can be extended to prove that the stochastic adaptive heavy-ball algorithm converges to the L 2 max margin solution, which is proposed by (Tao et al., 2021) and has the following form 3 :
[Equation]
where ˆ ν ( t ) is deﬁned as Eq.
(2).The proof is a simple combination of the proof techniques in SGDM and Adam by observing that the stochastic adaptive heavy-ball algorithm has the following equivalent update rule ( u (1) = w (1) )
[Equation]
However, Adam is different from the stochastic adaptive heavy-ball algorithm as it combine the conditioner and momentum as a whole, which makes our constructed Lyapunov function can not be applied.
We will put further investigation on stochastic Adam in future work.
D ISCUSSIONS
Extension to the Multi-Class Classiﬁcation Problem.
As mentioned in the introduction, despite all the previous analyses are aimed at the binary classiﬁcation problem, they can be naturally extended to the analyses multi-class classiﬁcation problem.
Speciﬁcally, in the linear multi-class classiﬁcation problem, for any ( x , y ) ∈
R d X × { 1 , · · · , C } in the sample set S , the (individual) logistic loss with parameter W ∈ R C ×
d X is denoted as
[Equation]
Correspondingly, dataset S is separable if there exists a parameter W , such that ∀ ( x , y ) ∈ S , we have W y , x > W i, x , ∀ i (cid:54) =
y .
The multi-class L 2 max-margin problem is then deﬁned as min (cid:107) W (cid:107) F , subject to : W y , x ≥ W i, x + 1 , ∀ ( x , y ) ∈ S , i (cid:54) =
y , where (cid:107) · (cid:107) F denotes the Frobenius norm.
Denote ˆ W as the L 2 max-margin solution, we have (mini-batch) SGDM and Adam (w/s) still converges to the direction of ˆ W , the proof of which is deferred to the Appendix D.2.
Theorem 5.
For linear multi-class classiﬁcation problem using logistic loss and almost every separable data, with a small enough learning rate, and 1 > β 2 > β 41 ≥ 0 (for Adam (w/s)), (mini-batch) SGDM and Adam (w/s) converge to the multi-class L 2 max-margin solution (a.s. for
SGDM).
Consistency with the Existing Experimental Results.
Our results stand with existing experimental observations.
Speciﬁcally, Soudry et al.
(2018) conduct experiments using GD and GDM on the same linear separable data (c.f., Figure 1 in (Soudry et al., 2018)), and it is observed that the training behaviors of GD and GDM are quite similar in terms of the direction w ( t ) , the training loss, and the margin, which supports our Theorem 2.
Nacson et al.
(2019) extend the experiment to the stochastic setting (c.f. Figure 2 in (Nacson et al., 2019)), and observe the same similarity between SGD and
SGDM, which agrees with our Theorem 3.
Wang et al.
(2021) conduct the experiments of SGD,
SGDM, Adam (without momentum) and Adam on MNIST using homogeneous neural networks (c.f.
Appendix F.1.2 in (Wang et al., 2021)), and observe such similarity for deep neural networks.
Our theorems apply to linear models, which is a special case of homogeneous neural network, and meet their observation.
Gap Between The Linear Model and Deep Neural Networks.
While our results only hold for the linear classiﬁcation problem, extending the results to the deep neural networks is possible.
Speciﬁcally, Lyu & Li (2019) construct a Lyapunov function bounding the rate between loss and parameter norm of GD on the homogeneous deep neural networks, and prove the parameter direction converges to some KKT point of the L 2 max-margin problem, which extends the result of GD on linear model.
Wang et al.
(2021) further show the proof techniques in (Lyu & Li, 2019) can be generalized, and successfully extend the Lyapunov function to AdaGrad, RMSProp and Adam (without momentum) on deep homogeneous neural networks.
It will be interesting to see whether such a Lyapunov function can be constructed for GDM and Adam in homogeneous neural networks.
C ONCLUSION
In this paper, we study the implicit bias of momentum-based optimizers in linear classiﬁcation with exponential-tailed loss.
Our results indicates that for SGD and Adam, adding momentum will not inﬂuence the implicit bias, and the direction of parameter converges to the L 2 max-margin solution.
Our theoretical results stands with existing experimental observations.
A P REPARATIONS
This section collect deﬁnitions and lemmas which will be used throughout the proofs.
A. C HARACTERIZATION OF THE M AX - MARGIN S OLUTION
This section collects several commonly-used characterization of the max-margin solution from
Nacson et al.
(2019) and Soudry et al.
To start with, we deﬁne support vectors and support set, which are two common terms in margin analysis.
Deﬁnition 2 (Support vectors and support set) .
For any i ∈ [ N ] , ˜ x i is called a support vector of the dataset S , if
[Equation]
Correspondingly, ˜ x i is called a non-support vector if (cid:104) ˜
x
i , ˆ w (cid:105) > 1 .
The support set of S is then deﬁned as
[Equation]
The following lemma delivers ˆ w as an linear combination of support vectors.
Lemma 8 (Lemma 12, Soudry et al.
For almost every datasets S , there exists a unique vector v = ( v 1 , · · · , v N ) , such that ˆ w can be represented as
[Equation]
(11) where v satisﬁes v i =
if ˜ x i / ∈ T ( S s ) , and v i > 0 if ˜ x i ∈ T ( S s ) .
Furthermore, the size of ˜ S s is at most d .
By Lemma 8, we further have the following corollary:
Corollary 2.
For almost every datasets S , the unique v given by Lemma 8 further satisﬁes that for any positive constant C 2 , there exists a non-zero vector ˜ w , such that, ˜ x i ∈ T
( S s ) , we have
[Equation]
Proof.
For almost every datasets S , any subsets with size d of S is linearly independent.
Since ˜ S s has size no larger than d (by Lemma 8), and Eq.
(12) is equivalent to linear equations, the proof is completed.
For the stochastic case, we will also need the following lemma when we calculate the form of parameter at time t .
Lemma 9 (Lemma 5, Nacson et al.
Let B ( s ) be the random subset used in SGD (i.e., the one used in SGDM).
Almost surely, there exists a vector ˇ w
[Equation]
where n ( t ) satisﬁes (cid:107) n ( t ) (cid:107) =
o ( t − 0 .
5+ ε ) for any ε > 0 , and (cid:107) n ( t + 1) − n ( t ) (cid:107) =
O ( t − 1 ) .
If the
B ( s ) is the random subset used in SGDM (i.e., the one used in mini-batch SGDM), then the a.s. condition can be removed.
A. P REPARATIONS OF THE OPTIMIZATION ANALYSIS
This section collects technical lemmas which will be used in latter proofs.
We begin with a lemma bounding the smooth constants if the loss is bounded.
Lemma 10.
If loss (cid:96) satisﬁes (D) in Assumption 3, then for any w 0 , if L ( w ) ≤ L ( w 0 ) , then we have
L is H s 0 smooth at point w , where s
=
(cid:96) − 1 ( N L ( w 0 )) .
Furthermore, L is globally H s 0 smooth over the set { w : L ( w ) ≤ L ( w 0 ) } .
Proof.
Since (cid:96) is positive, we have ∀ i ∈
[ N ] ,
[Equation] [Equation]
which leads to ˜ (cid:96) ( w , z i ) < N L ( w 0 )
, and (cid:96) is H s 0 smooth at (cid:104) w , ˜ x
i (cid:105) .
Furthermore, since ∇ w ˜ (cid:96) ( w , z i ) =
[Equation]
w
[Equation]
w , ˜ x
i
[Equation]
˜ x
i , for any two parameters w 1 and w 2 close enough to w ,
[Equation]
w
[Equation]
, ˜ x
i
[Equation]
−
[Equation]
| ≤ H s 0 (cid:107) w 1 − w 2 (cid:107) , and thus,
[Equation]
which completes the proof that L is locally H s 0 smooth at w .
Now if w 1 and w 2 both belong to { w : L ( w ) ≤ L ( w 0 ) } , we have for any ˜ x i ∈ T ( S ) , (cid:104) w 1 , ˜ x i (cid:105) > (cid:96) − 1 ( N L ( w 0 )) , and (cid:104) w 2 , ˜ x i (cid:105) > (cid:96) − 1 ( N L ( w 0 )) .
Following the same routine as the locally smooth proof, we complete the second argument.
The proof is completed.
Based on Assumption 2, we also have the following lemma characterizing the relationship between loss (cid:96) and its derivative (cid:96) (cid:48) when x is large enough.
Lemma 11.
Let loss (cid:96) satisfy Assumption 2.
Then, there exists an large enough x 0 and a positive real K , such that, ∀ x > x 0 , we have
[Equation]
Proof.
By Assumption 2, there exists a large enough x 0 , such that ∀ x > x 0 , we have
[Equation]
On the other hand, as lim t →∞ (cid:96) ( x ) = 0 , we have
[Equation]
which by Eq.
(13) leads to
[Equation]
Therefore, setting K = 4 completes the proof.
The following lemma bridges the second moment of ∇L B ( t ) with its squared ﬁrst moment.
Lemma 12.
Let the dataset S satisﬁes the separable assumption 1.
Let B be a random subset of S with size b sampled independently and uniformly without replacement.
Then, at any point w , we have
[Equation]
Proof.
To start with, notice that
[Equation]
Therefore, the ﬁrst inequality can be directly obtained by Cauchy-Schwartz’s inequality.
To prove the second inequality
[Equation]
we ﬁrst calculate the explicit form of ∇L B ( w ) .
[Equation]
Therefore,
[Equation]
On the other hand,
[Equation]
x
[Equation]
) where Eq.
( (cid:63) ) is due to ∀ z ∈ (cid:104)
˜ x , − ˆ w (cid:105) ≥ 1 and (cid:96) (cid:48) < 0 .
Therefore, (cid:107)∇L
[Equation]
The proof is completed by putting Eqs.
(15) and (16) together.
In the following lemma, we show the updates of GDM, Adam, and SGDM are all non-zero.
Lemma 13.
Regardless of GDM, Adam, or SGDM, the updates of all steps are non-zero, i.e.,
[Equation]
Proof.
We start with the alternative forms of the update rule of GDM, Adam, and SGDM using the gradients along the trajectory respectively.
For GDM, by Eq.
(5), the update rule can be written as
[Equation]
Similarly, the update rule of SGDM can be written as
[Equation]
while the update rule of Adam can be given as
[Equation]
On the other hand, by the deﬁnition of empirical risk L , the gradient of L at point w can be given as
[Equation]
By Eq.
(20) and Eq.
(17), we further have for GDM,
[Equation]
N
[Equation]
By Assumption 1, there exists a non-zero parameter ˆ w , such that, (cid:104) ˆ w , ˜ x i (cid:105) > 0 , ∀ i .
Therefore, by executing inner product between Eq.
(21) and ˆ w , we have
[Equation]
N
[Equation]
where Eq.
( ∗ ) is due to (cid:96) (cid:48) < 0 .
This complete the proof for GDM.
Similarly, for SGDM, we have
[Equation]
b
[Equation]
>
[Equation]
which completes the proof of SGDM.
For Adam, we have
[Equation]
(cid:118)(cid:117)(cid:117)(cid:116)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
N
[Equation]
which completes the proof of Adam.
The proof is completed.
B I MPLICIT B IAS OF GD/SGD WITH MOMENTUM
This section collects the proof of the implicit bias of gradient descent with momentum and stochastic gradient descent with momentum.
B. I MPLICIT B IAS OF GD WITH M OMENTUM
This section collects the proof of Theorem 2.
B.1. P ROOF OF THE SUM OF SQUARED GRADIENTS CONVERGES
To begin with, we will prove the sum of squared norm of gradients along the trajectory is ﬁnite for gradient descent with momentum.
To see this, we ﬁrst deﬁne the continuous-time update rule as
[Equation] [Equation]
We then prove a generalized case of Lemma 1 for any w ( t + α ) .
Lemma 14 (Lemma 1, extended) .
Let all conditions in Theorem 2 hold.
We then have
[Equation]
where C 1 is a positive real such that η = 2 1 − β H s 0 C 1 .
Proof of Lemma 14.
For brevity, we denote s 0 (cid:52) =
(cid:96) − 1 ( N L ( w 1 )) .
We prove this lemma by reduction to absurdity.
Concretely, let t ∗ be the smallest positive integer time such that there exists an α ∈ [0 , 1] , such that
Eq.
(7) doesn’t hold.
Let α ∗ =
inf { α ∈ [0 , 1] : Eq.
(7) doesn (cid:48)
t hold for ( t ∗ , α ) } .
By continuity,
Eq.
(7) holds for ( t ∗ , α ∗ ) .
We further divide the proof into two cases depending on the value of α ∗ .
Case 1: α ∗ =
For any t ∗ > t ≥ 1 , we have Eq.
(7) holds for ( t, 1) .
Speciﬁcally, we have
[Equation]
which further leads to
[Equation]
Since β 2 η (cid:107) w ( t ∗ ) − w ( t ∗ − 1) (cid:107) 2 is non-negative, we have
L
[Equation]
w
[Equation] [Equation]
) ≥ L ( w ( t ∗ )) .
By Lemma 10, we have L is H s 0 smooth at w ( t ∗ ) .
Therefore, by Taylor’s expansion for L at point w ( t ∗ ) , we have for small enough α >
L ( w ( t ∗ + α ))
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Equation]
where Eq.
( ∗ ) is due to a simple rearrangement of the update rule of gradient descent with momentum (Eq.
(5)), i.e.,
[Equation]
Inequality ( ∗∗ ) is due to Cauchy Schwarz’s inequality and arithmetic-geometric average inequality, and Inequality ( (cid:5) ) is due to
[Equation]
Here the inequality is due to that − (1 − α ) β 2 η
[Equation]
2 tend to − β 2 η
[Equation]
w
[Equation]
t
[Equation]
− w
[Equation] [Equation]
as α tend to zero, which is a negative constant by Lemma 13.
Eq.
(22) indicates Eq.
(7) holds at ( t ∗ , α ) for α > 0 is small enough, which contradicts to α ∗ = 0 .
Case 2: α ∗ (cid:54) =
: Same as Case 1 , we have for any 1 ≤ t < t ∗ ,
[Equation]
which further leads to
[Equation] [Equation]
On the other hand, by the deﬁnition of α ∗ , we have for any 0 ≤ α < α ∗ , we have Eq.
(7) holds for ( t ∗ , α ) , which by continuity further leads to Eq.
(7) holds for ( t ∗ , α ∗ ) .
Therefore, α ∗ < 1 , otherwise,
Eq.
(7) holds for ( t ∗ , α ) , ∀ α ∈ [0 , 1] which contradicts the deﬁnition of t ∗ .
Combining Eq.
(7) with ( t ∗ , α ) and Eq.
(24), we further have
[Equation]
Consequently, for any α ∈ [0 , α ∗ ]
L ( w (1)) ≥ L ( w ( t ∗ + α )) , and by Lemma 10, we then have L is H s 0 smooth at w ( t ∗ + α ) , which further by Taylor’s expansion leads to
[Equation]
where Eq.
( ◦ ) follows the same routine as Case 1 , Eq.
( • ) is due to the deﬁnition of η and C 1 , and
Eq.
( ∗ ) is due to α ∗ < 1 , and (cid:107) w ( t ∗ + 1) − w ( t ∗ ) (cid:107) 2 > 0 (given by Lemma 13).
By the continuity of L , for any small enough δ > 0 , Eq.
(7) holds for ( t ∗ , α ∗ + δ ) , which contradicts to the deﬁnition of α ∗ .
The proof is completed.
By Lemma 1, one can easily obtain the sum of the squared norms of the updates across the trajectory converges.
Corollary 3.
Let all conditions in Theorem 2 hold.
We have
[Equation]
Consequentially, we have
[Equation]
Proof.
By Lemma 1, we have
[Equation]
which by summing over t further leads to
L ( w (1)) ≥ L ( w (1)) − (cid:18)
L ( w ( t + 1))
+ β 2 η (cid:107) w ( t + 1) − w ( t ) (cid:107)(cid:19) ≥ (1 − C 1 )(1 − β ) η t (cid:88) s = (cid:107)
w ( s +1)
− w ( s ) (cid:107) 2 .
Taking t → ∞ leads to ∞ (cid:88) s = (cid:107) w ( s + 1) − w ( s ) (cid:107) 2 < ∞ .
By triangle inequality, we further have
[Equation]
where Eq.
( (cid:63) ) is due to Cauchy-Schwartz’s inequality.
The proof is completed.
By the negative derivative of the loss and the separable data
[Equation]
we can ﬁnally prove Corollary 1.
Proof of Corollary 1.
By Eq.
[Equation]
we have
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(26) where Inequality ( ∗ ) is due to Cauchy-Schwartz’s inequality
[Table]
Inequality ( ∗∗ ) is due to (cid:96) (cid:48) ( s ) < 0 , ∀ s ∈ R and (cid:104)
ˆ w , ˜ x i (cid:105) ≥ γ , ∀ i ∈ [ N ] , and Inequality ( • ) is due to (cid:107)
˜ x
i (cid:107) ≤ 1 .
By combining Eq.
(25) and Eq.
(26), we complete the proof.
By the exponential-tailed assumption of the loss (Assumption 2), we further have the following corollary.
Corollary 4.
Let all conditions in Theorem 2 hold.
Then, lim t →∞ (cid:107)∇L ( w ( t )) (cid:107) = 0 , and
[Equation]
Consequently, there exists an large enough time t 0 , such that, ∀ t > t 0 , ∀ i , we have (cid:104) w ( t ) ,
˜ x i (cid:105) > 0 , and
[Equation]
B.1. B OUNDING THE ORTHOGONAL PART
To prove Theorem 2, we only need to show w ( t ) − log( t )
ˆ w ( t ≥ 1) has bounded norm for any iteration t > 0 .
Letting C 2 = η (1 − β ) N in Corollary 2, we obtain an constant vector ˜ w satisfying Eq.
Deﬁne
[Equation]
As ˜ w is a constant vector, that w ( t ) − log( t ) ˆ w ( t ≥ 1) has bounded norm is equivalent to r ( t ) has bounded norm.
Lemma 2 then propose an equivalent proposition of (cid:107) r ( t ) (cid:107) is bounded, and further prove this proposition is fulﬁlled.
As the proof is rather complex, we separate it into two sub-lemmas.
We ﬁrst prove (cid:107) r ( t ) (cid:107) is bounded if and only if function g ( t ) is upper bounded.
Lemma 15 (First argument in Lemma 2) .
Let all conditions in Theorem 2 hold.
Then, (cid:107) r ( t ) (cid:107) is bounded if and only if function g ( t ) is upper bounded.
Proof.
We start the proof by showing that A 1 ( t ) (cid:52) =
(cid:80) tτ
=2 (cid:104) r
[Equation]
) − r (
[Equation]
1) , w ( τ ) − w ( τ − 1) (cid:105) has bounded absolute value.
By the deﬁnition of r ( t ) , we have
[Equation]
which further indicates
[Equation]
w ( τ − 1) (cid:29) .
Therefore, the absolute value of A 1 ( t ) can be bounded as | A
[Equation]
(cid:88)
[Equation]
=
[Equation]
w (
[Equation] [Equation]
w
[Equation]
− 1)
[Equation]
log (cid:18)
[Equation]
(cid:19) ˆ w , w ( τ ) − w ( τ − 1) (cid:29)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ t (cid:88) τ = (cid:12)(cid:12)(cid:12)(cid:12)(cid:28) w ( τ ) − w ( τ − 1) − log (cid:18)
τ τ −(cid:19) ˆ w , w ( τ ) − w ( τ − 1) (cid:29)(cid:12)(cid:12)(cid:12)(cid:12) ≤ t (cid:88) τ
= (cid:107) w ( τ ) − w ( τ − 1) (cid:107) 2 + t (cid:88) τ
= (cid:12)(cid:12)(cid:12)(cid:12)(cid:28) log (cid:18)
τ τ −(cid:19) ˆ w , w ( τ ) − w ( τ − 1) (cid:29)(cid:12)(cid:12)(cid:12)(cid:12) ≤ t (cid:88) τ
= (cid:107) w ( τ ) − w ( τ − 1) (cid:107) 2 + t (cid:88) τ
= (cid:13)(cid:13)(cid:13)(cid:13) log (cid:18)
τ τ −(cid:19)
ˆ w (cid:13)(cid:13)(cid:13)(cid:13) (cid:107) w ( τ ) − w (
[Equation]
< ∞ , where Inequality ( (cid:63) ) is due to the Inequality of arithmetic and geometric means, and Inequality ( ◦ ) is due to Corollary 3 and log ττ −
= O ( 1 τ ) .
Therefore, g ( t ) is upper bounded is then equivalent to 12 (cid:107) r ( t ) (cid:107) 2 + β 1 − β (cid:104) r ( t ) , w ( t ) − w ( t − 1) (cid:105) is upper bounded.
Now if 12 (cid:107) r ( t ) (cid:107)
+
β 1 − β (cid:104) r
( t ) , w ( t ) − w ( t − 1) (cid:105) is upper bounded, we will prove (cid:107) r ( t ) (cid:107) is bounded by reduction to absurdity.
Suppose that (cid:107) r ( t ) (cid:107) has unbounded norm.
By Corollary 3, we have lim t →∞ (cid:107) w ( t ) − w ( t − 1) (cid:107) = 0 , and there exists a large enough time T , such that (cid:107) w ( t ) − w ( t − 1) (cid:107) < 1 for any t ≥ T .
On the other hand, since r ( t ) is unbounded from above, there exists an increasing time sequence k i > T , i ∈
Z + , such that
[Equation]
Therefore, we have lim
i
[Equation]
, which leads to contradictory, and completes the proof of necessity.
On the other hand, if (cid:107) r ( t ) (cid:107) is upper bounded, since (cid:107) w ( t ) − w ( t − 1) (cid:107) is also upper bounded, we have 12 (cid:107) r ( t ) (cid:107) 2 + β 1 − β (cid:104) r ( t ) , w ( t ) − w ( t − 1) (cid:105) is upper bounded, which completes the proof of sufﬁciency.
The proof is completed.
Therefore, the last piece of this puzzle is to prove g ( t ) is upper bounded ∀ t > 0 .
Lemma
(Second argument in Lemma 2) .
Let all conditions in Theorem 2 hold.
Then, we have g ( t ) is upper bounded.
Proof.
We start the proof by calculating g ( t + 1) − g ( t ) .
For any t ≥ 2 , we have
[Equation]
On the other hand
[Equation]
by simply rearranging the update rule Eq.
[Equation]
we have
[Equation]
which further indicates
[Equation]
Denote A 2 ( t ) = (cid:107) r
( t + 1) − r ( t ) (cid:107)
[Equation]
and A 3 ( t ) =
(cid:68) r ( t ) , − log (cid:0) t +1 t (cid:1)
ˆ w − η 1 − β ∇L ( w ( t )) (cid:69) .
We then prove respectively (cid:80) ∞ t =1 A 2 ( t ) and (cid:80) ∞ t =
A 3 ( t ) are upper bounded.
First of all, by deﬁnition of r ( t ) Eq.(27), we have ∞ (cid:88) t =
[Equation]
(29) where Eq.
( • ) is due to Lemma 3 and log (cid:0) t +1 t (cid:1)
=
O ( 1 t ) .
Then we only need to prove (cid:80) ∞ t =
A 3 ( t ) < ∞ .
To begin with, by adding one additional term 1 t ˆ w into A 3 , we have
[Equation]
On the one hand, by Corollary 3, (cid:107) w ( t ) (cid:107) =
O ( √ t ) , which further leads to
[Equation]
By 1 t − log t +1 t = O (cid:0)
[Equation] [Equation]
, we have
[Equation]
On the other hand, by direct calculation of the gradient, we have (cid:28)
[Equation]
=
N
[Equation]
, where Eq.
( (cid:63) ) is due to the deﬁnition of ˜ w (Eq.
(12) with C 2 =
η 1 − β ).
Denote
A
[Equation]
, and
A 5 ( t ) =
(cid:42)
[Equation]
We then analysis these two terms respectively.
As for A 4 ( t ) , due to (cid:96) (cid:48) < 0 , we have
[Equation]
t
[Equation]
t
[Equation]
i (cid:43) .
By Corollary
[Equation]
we further have ∀ t > t
[Equation]
which further indicates
[Equation]
(cid:104)
[Table]
[Table]
≤ η 1 − β (cid:16) max
[Table]
2 e −(cid:104)
[Table]
[Table]
i
[Table]
( ◦ ) ≤ η 1 − β
[Table]
η 1 − β
[Table]
t θ 2 N, where θ in Eq.
( ◦ ) is deﬁned as
[Equation]
As (cid:80) ∞ t =1 1 t θ < ∞ , we have
[Equation]
For each term (cid:68) r ( t ) , − η 1 − β
[Equation]
1 t e −(cid:104) ˜ w , ˜ x
i
[Equation]
+
[Equation]
˜ x
i
[Equation] [Equation]
˜ x
i
[Equation]
( ˜ x i / ∈ T ( S s ) ) in A 5 ( t ) , we divide the analysis into two parts depending on the sign of (cid:104) r ( t ) , x i (cid:105) .
Case 1: (cid:104) r ( t ) , ˜ x i (cid:105) ≥ 0 .
By Corollary 4, we have
[Equation]
where Eq.
( (cid:5) ) is due to the deﬁnition of r ( t )
(Eq.
[Equation]
Since (cid:104) r ( t )
[Equation]
˜ x i (cid:105) ≥
[Equation]
we further have
[Equation]
( t ) , ˜ x
[Equation]
(cid:105) , where Eq.
( (cid:3) ) is due to (cid:104)
ˆ w , ˜ x i (cid:105) = 1 , ∀ ˜ x i ∈ T ( S s ) .
Speciﬁcally,
[Equation]
Therefore, η
[Equation] [Equation]
≤ η 1 − β t e −(cid:104)
[Table]
[Equation]
[Table]
[Equation]
Case to case we have
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Speciﬁcally
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
, where Eq. ( † ) is due to if (cid:104) r ( t ) ,
˜ x i (cid:105) ≥ − t −
[Table]
5 µ − ,
[Equation]
[Table]
[Equation]
[Table]
If − 2 ≤ (cid:104) r ( t ) , ˜ x i (cid:105) <
− t − 0 .
5 µ − , we have η
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
=
η
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
≤ η
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Therefore
[Equation]
when t is large enough
[Equation]
1 − e 2 µ − t µ − e − µ − (cid:104) ˜ w , ˜ x i (cid:105) >
[Equation]
which by e −(cid:104) r ( t ) , ˜ x i (cid:105) ≥ 1 − (cid:104) r ( t ) , ˜ x i (cid:105) leads to
[Equation]
(cid:18)
[Equation]
[Equation]
(cid:19)(cid:19) (cid:104) r
( t ) , ˜ x
i (cid:105)
[Equation] [Equation]
If − 2 > (cid:104) r ( t )
[Equation]
˜ x i (cid:105) ,
[Equation] [Equation]
[Table]
e
[Equation]
[Table]
[Equation] [Equation]
and
[Equation]
[Table]
[Equation]
Therefore
[Equation]
in Case 2.
[Equation]
for large enough t
[Equation]
we have
[Equation]
Combining Case 1.
and Case 2.
[Equation]
we conclude that
[Equation]
which further yields
[Equation]
Combining Eq.
(32) and Eq.
(33), we conclude that (cid:80) ∞ t =
A 3 ( t ) < ∞ , which together with Eq.
(29) yields (cid:80) ∞ t =2 g ( t + 1) − g ( t ) < ∞ , and completes the proof.
We are now ready to prove Theorem 2.
Proof of Theorem 2.
By Lemma 16, we have g ( t ) is upper bounded.
Therefore, by Lemma 2, we have (cid:107) r ( t ) (cid:107) is bounded, which further indicates (cid:107) w ( t ) − log( t ) ˆ w (cid:107) is bounded.
Therefore, the direction of w ( t ) can be calculated as w ( t
[Equation]
The proof is completed.
B. I MPLICIT B IAS OF SGD WITH M OMENTUM
This section collects the proof of Theorem 3.
Following the same framework as Appendix B.1, we will ﬁrst prove that the sum of the squared gradient norms along the trajectory is ﬁnite.
One may expect L ( w ( t )) + β 2 η (cid:107) w ( t ) − w ( t − 1) (cid:107) 2 is a Lyapunov function of SGDM.
However, due to the randomness of the update rule of SGDM, L ( w ( t )) + β 2 η (cid:107) w ( t ) − w ( t − 1) (cid:107) 2 may no longer decrease (we will show this in the end of Appendix B.2, please see Appendix B.2.3 for explanation).
B.2. P ROOF OF THE SUM OF GRADIENTS ALONG THE TRAJECTORY IS FINITE
We ﬁrst provide a proof of Lemma 3.
Proof of Lemma 3.
We denote the parameter of the t -th step in Eq.
(9) as ˜ w , while that in Eq.
(1) as w ( t ) .
With w (1) =
˜ w (1) , we will prove ˜ w ( t ) =
w ( t ) , ∀ t > 1 iteratively.
Speciﬁcally, suppose for any 1 ≤ k ≤ t w ( k ) =
˜ w ( k ) .
Then, we have
[Equation]
Therefore, ˜
[Equation]
, which by iteration further indicates
[Equation]
) − ˜ η (cid:32) t − 1 (cid:88) k
= (1 − β t − k ) ∇L B ( k ) ( w ( k )) (cid:33) , and w ( t − 1) = w (1) − ˜ η
(cid:32) t − 2 (cid:88)
k
= (1 − β t − 1 − k ) ∇L B ( k ) ( w ( k )) (cid:33) .
On the other hand, by Eq.
(1), we have w ( t + 1) =
w ( t ) − η ∇L B ( t ) ( w ( t ))
+ β ( w ( t ) − w ( t − 1))
=
w (1) − ˜ η
(cid:32) t − 1 (cid:88) k
= (1 − β t − k ) ∇L B ( k ) ( w ( k )) (cid:33) − η ∇L B ( t ) ( w ( t ))
+ β ˜ η (cid:32) t − 1 (cid:88)
k
= ( β t − 1 − k − β t − k ) ∇L B ( k ) ( w ( k )) (cid:33) =
w (1) − ˜ η
(cid:32) t (cid:88) k
= (1 − β t +1 − k ) ∇L B ( k ) ( w ( k )) (cid:33) =
˜ w ( t + 1) .
The proof is completed.
Using the alternative form given by Lemma 3, we then prove Lemma 4, which indicates L ( u ( t )) is a proper choice of Lyapunov function.
Proof of Lemma 4.
We start the proof by applying the Taylor’s expansion of L at the point u ( t ) to the point u
( t + 1) .
Concretely, by Assumption 3.
(S), we have
L
[Equation]
which by Eq.
(9) leads to
[Equation] [Equation]
Taking the expectation of Eq.
(34) with respect to w ( t + 1) conditioning on { w ( s ) } ts =1 , we have
[Equation]
where Eq.
( (cid:63) ) is due to that w ( t + 1) is uniquely determined by B ( t ) given { w ( s ) } ts =1 , Eq.
( ◦ ) is due to u ( t ) is uniquely determined by { w ( s ) } ts =1 , and Inequality.
( • ) is due to Lemma 12.
Therefore, we have
[Equation]
By Assumption 3.
(S), (cid:96) is H -smooth, which further leads to (cid:107)∇L
[Table]
[Equation]
[Table]
[Equation]
where Inequality ( (cid:3) ) is due to β ( w ( t ) − w ( t − 1))
= (1 − β )( u ( t ) − w ( t )) by Eq.
(9), Inequality ( (cid:5) ) is due to triangular inequality, and Inequality ( ♣ ) is due to Cauchy-Schwartz Inequality.
we have
[Equation]
which by taking expectation to { w ( s ) } ts =1 leads to
[Equation]
By Lemma 12 and η < min { 1 − β 1+ HNbγ 2 , (1 − β ) 3 γ 4 b 2 H 2 N 3 β 2 } , we further have
[Equation]
Summing Eq.
(37) with t from 1 to T leads to
E [ L ( u ( T + 1))] =
E F T +
[ L ( u ( T + 1))]
≤L ( u (1)) −
T (cid:88) t = ˜
η 2 E F t (cid:107)∇L ( w ( t ))
(cid:107) 2 +
T (cid:88) t = 1 − β ˜ η (cid:32) t − 1 (cid:88) s =
β t − 1 − s E F s (cid:107)∇L ( w ( s ))
(cid:107)(cid:33) =
L ( u (1)) −
T (cid:88) t = ˜
η 2 E F t (cid:107)∇L ( w ( t ))
(cid:107) 2 +
T − 1 (cid:88) s =
1 − β ˜ η (cid:32) T (cid:88) t
= s + β t − 1 − s E F s (cid:107)∇L ( w ( s ))
(cid:107)(cid:33) ≤L ( u (1))
−
T (cid:88) t = ˜
η 2 E F t (cid:107)∇L ( w ( t ))
(cid:107) 2 +
T − 1 (cid:88) s =
4 ˜ η (cid:16) E F s (cid:107)∇L ( w ( s )) (cid:107) 2 (cid:17) ≤L ( u (1))
−
T (cid:88) t = ˜
η 4 E F t (cid:107)∇L ( w ( t ))
(cid:107)= L ( u (1)) −
T (cid:88) t = ˜
η 4 E (cid:107)∇L ( w ( t ))
(cid:107) 2 .
The proof is completed.
As L ( u (1)) is upper bounded, we have the following corollary given by Lemma 4.
Corollary 5.
Let all conditions in Theorem 3 hold.
Then, we have ∞ (cid:88) t =
E (cid:107)∇L ( w ( t ))
(cid:107) 2 < ∞ .
Consequently,
[Equation]
and
[Equation]
hold almost surely.
Proof.
By Lemma 4, we have for any T > 1 ,
[Equation]
which completes the proof of Eq.
The rest of claims follows immediately by Fubini’s Theorem and Assumption 2.
The proof is completed.
B.2. B OUNDING THE ORTHOGONAL PART
Similar to the case of GDM, we deﬁne ˜ w as the solution of Eq.
(12) with C 2 = η (1 − β ) N .
We also let n ( t ) be given by Lemma 9, and deﬁne r ( t ) in this case as
[Equation]
As ˜ w is a constant vector, and (cid:107) n ( t ) (cid:107) → 0 as t → ∞ , we have w ( t ) − log( t ) ˆ w has bounded norm if and only if (cid:107) r ( t ) (cid:107) is upper bounded.
Similar to the GDM case, we have the following equivalent condition of that (cid:107) r ( t ) (cid:107) is bounded.
Lemma 17.
Let all conditions in Theorem 3 hold.
Then, (cid:107) r ( t ) (cid:107) is bounded almost surely if and only if function g ( t ) is upper bounded almost surely, where g : Z + → R is deﬁned as
[Equation]
β
[Equation]
Proof.
To begin with, we prove that almost surely | (cid:80) tτ
=2 (cid:104)
r ( τ ) − r ( τ − 1) , w ( τ ) − w ( τ − 1) (cid:105)| is upper bounded for any t .
By Corollary 5, we have almost surly ∞ (cid:88) t =
[Equation]
w
[Equation]
< ∞ .
On the other hand, for any w , we have
[Equation]
Therefore, we have almost surely,
[Equation]
which further leads to almost surely ∞
[Equation]
By the deﬁnition of r ( t ) (Eq. (39)), we further have (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) t (cid:88) τ =
[Equation]
≤
[Equation] [Equation]
where Inequality ( (cid:63) ) is due to (cid:107) n ( τ ) − n ( τ − 1) (cid:107)
=
O ( 1 τ ) and log τ +
τ = O ( 1 τ ) .
Therefore, g ( t ) is upper bounded almost surely is equivalent to 1 2 (cid:107) r ( t ) (cid:107) 2 + β 1 − β (cid:104) r
( t ) , w ( t ) − w ( t − 1) (cid:105) is upper bounded, which can be shown to be equivalent with (cid:107) r ( t ) (cid:107) is bounded following the same routine as Lemma 2.
The proof is completed.
As the case of GDM, we only need to prove g ( t ) is upper bounded to complete the proof of Theorem 3.
Lemma 18.
Let all conditions in Theorem 3 hold.
We have g ( t ) is upper bounded.
Proof.
Following the same routine as Lemma 2, we have g
[Equation]
where (cid:80) ∞ t =1 (cid:107) r ( t + 1) − r ( t ) (cid:107) 2 is upper bounded.
On the other hand, by the deﬁnition of r ( t ) (Eq. (39)), we have
[Equation]
while by Lemma
[Equation]
Combining the above two equations
[Equation]
we further have
[Equation]
which further indicates
[Equation]
Therefore
[Equation]
we only need to prove
[Equation]
By directly applying the form of ∇L ( w ( t ))
[Equation]
we have
[Equation]
(cid:88) i :˜ x i ∈T ( B ( s ) ∩ S cs ) (cid:104) r ( t ) , − (cid:96) (cid:48)
( (cid:104) w ( t ) ,
˜ x i (cid:105) )
˜ x
i (cid:105) .
Let A 6 ( t ) =
(cid:80) i :˜ x i ∈T ( B ( s ) ∩ S s ) (cid:0) − (cid:96) (cid:48)
( (cid:104) w ( t ) ,
˜ x i (cid:105) ) − 1 t e −(cid:104)
˜ w ( t ) , ˜ x i (cid:105) (cid:1) (cid:104) r ( t ) , ˜ x i (cid:105) , and A 7 ( t ) =
(cid:80) i :˜ x i ∈T ( B ( s ) ∩ S cs ) (cid:104) r ( t ) , − (cid:96) (cid:48)
( (cid:104) w ( t ) ,
˜ x i (cid:105) )
˜ x
i (cid:105) .
We will investigate these two terms respectively.
As (cid:104) w ( t ) , ˜ x (cid:105) → ∞ , ∀ ˜ x ∈ T ( S ) , a.s., we have a.s., there exists a large enough time t 0 , s.t., ∀ t ≥ t 0 , ∀ ˜ x ∈ T ( S ) , − (cid:96) (cid:48) ( (cid:104) w ( t ) ,
˜ x (cid:105) ) ≤ (1 + e − µ + (cid:104) w ( t ) , ˜ x (cid:105) )
e −(cid:104) w ( t ) , ˜ x (cid:105) , − (cid:96) (cid:48)
( (cid:104) w ( t ) ,
˜ x i (cid:105) ) ≥ (1 − e − µ − (cid:104) w ( t ) , ˜ x (cid:105) )
e −(cid:104) w ( t ) , ˜ x (cid:105) , (cid:104) ˜
x , w ( t ) (cid:105) > 0 .
Therefore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where Inequality.
( (cid:63) ) is due the deﬁnition of θ (Eq.
(31)), Inequality.
( † ) is due to e
[Equation] [Equation]
e
[Equation]
1 , and Eq.
( ◦ ) is due to
lim t
[Equation]
e
[Equation]
t
[Equation]
e
[Equation]
˜ x
i
[Equation]
Thus,
[Equation]
A
[Equation]
< ∞ .
On the other hand, A 6 ( t ) can be rewritten as
[Equation]
If (cid:104) r ( t ) , ˜ x i (cid:105) ≥ 0 , we have for ε < 0 .
[Equation]
[Equation] [Equation]
, 1 . 5 − ε } (cid:19) , where Eq.
( • ) is due to n ( t )
=
O ( 1 t 0 .
5 − ε ) , and Eq.
( (cid:5) ) is due to e −(cid:104) r ( t ) ,
˜ x i (cid:105) (cid:104) r ( t ) , ˜ x i (cid:105) ≤ 1 e .
On
[Table]
other hand, if
[Table]
r
[Table]
t ) ,
[Table]
i
[Table] [Table]
have
[Table]
Speciﬁcally, if
[Table]
[Equation]
, where Eq.
( (cid:3) ) is due to that as (cid:104) w ( t ) ,
˜ x i (cid:105) → ∞ and t − 0 .
5min { µ − , 0 . 5 } → 0 as t → ∞ , there exists a large enough time T , s.t., ∀ t > T , under the circumstance 0 > (cid:104) r
( t ) ,
˜ x
i
[Equation]
≥ − t − 0 .
5min { µ − , 0 . 5 } , e −(cid:104) r ( t )+ n ( t ) , ˜ x i (cid:105) < 1 and e − µ − (cid:104) w ( t ) , ˜ x i (cid:105) < 1 .
If − 2 ≤ (cid:104) r ( t ) , ˜ x i (cid:105) <
− t − 0 .
5min { µ − , 0 . 5 } , then, for large enough t , |(cid:104) ˜ x
i , n ( t ) (cid:105)| < 2 , 1 − e
[Equation] [Equation]
t µ − > 0 , and
[Equation]
[Table]
[Equation] [Equation]
=
[Equation]
If − 2 > (cid:104) r ( t ) , ˜ x i (cid:105) , then for large enough time t , e −(cid:104)
[Equation]
+ n
[Equation]
i
[Equation]
, and
[Equation]
Conclusively
[Equation]
if (cid:104) r ( t )
[Equation]
˜ x i (cid:105) <
[Equation]
for large enough t
[Equation]
we have
[Equation]
which further indicates, for large enough t , we have
[Equation]
which indicates ∞ (cid:88) t
[Equation]
[Equation] [Equation]
Therefore,
[Equation]
< ∞ .
The proof is completed.
B.2. E XPLANATION FOR P ROPER L YAPUNOV F UNCTION
Based on the success of applying Lyapunov function L ( w ( t ))
+ β 2 η (cid:107) w ( t ) − w ( t − 1) (cid:107) 2 to analyze gradient descent with momentum, it is natural to try to extend this routine to analyze stochastic gradient descent with momentum.
However, in this section, we will show such Lyapunov function is not proper to analyze SGDM as this will put constraints on the range of the momentum rate β .
Speciﬁcally, at any step t , since the loss L is H smooth at w ( t ) , we can expand the loss L in the same way as the GDM case:
[Equation]
By taking expectation with respect to w ( t + 1) conditioning on { w ( s ) } ts =1 for both sides
[Equation]
we further obtain
[Equation]
= L ( w ( t ))
+ β η (cid:10) ( w ( t ) − w ( t − 1)) , E w ( t +1) (cid:2) w ( t + 1) − w ( t ) (cid:12)(cid:12) { w ( s ) } ts =1 (cid:3)(cid:11)
+ H
[Equation]
≤L (
[Equation] [Equation]
η (cid:107)
[Equation]
(cid:107)
[Equation]
β
[Equation]
η (cid:13) (cid:13) E w ( t +1) (cid:2) w ( t + 1)
[Equation]
w ( t ) (cid:12) (cid:12)
[Equation]
w
[Equation]
) } t s
[Equation]
[Equation] [Equation]
H
[Equation]
[Equation] [Equation]
{ w
[Equation]
) } ts
[Equation] [Equation]
[Equation] [Equation]
w
[Equation]
2 , where Eq.
( (cid:63) ) is becasue E w ( t +1)
[ w ( t + 1) − w ( t ) |{ w ( s ) } ts =1 ] = − η ∇L ( w ( t ))
+ β ( w ( t ) − w ( t − 1)) due to the deﬁnition of SGDM (Eq.
Rearranging the above inequal- ity respect
( s
= leads to
[Equation]
On the other hand
[Equation]
we wish to obtain some positive constant α from Eq.
(41), such that (at least),
[Equation]
which requires to lower bound
[Equation]
by
[Equation]
However, in general cases,
[Equation]
1 (cid:13)(cid:13)
E w ( t +1)
[ w ( t + 1) − w ( t ) |{ w ( s ) } ts =1 ] (cid:13)(cid:13)is only upper bounded by
E { w ( s ) } t +1 s = (cid:107) w ( t + 1) − w ( t ) (cid:107)(Holder’s
Inequality), although in our case, (cid:13)(cid:13)
E w ( t +1)
[ w ( t + 1) − w ( t ) |{ w ( s ) } ts =1 ] (cid:13)(cid:13) 2 can be bounded as (cid:13)(cid:13)
E w ( t +1) (cid:2) w ( t + 1) − w ( t ) (cid:12)(cid:12) { w ( s ) } ts =1 (cid:3)(cid:13)(cid:13)= (cid:107)− η ∇L ( w ( t ))
+ β ( w ( t ) − w ( t − 1))
(cid:107)= (cid:107)− η ∇L ( w ( t ))
(cid:107)
+ (cid:107) β ( w ( t ) − w ( t − 1))
(cid:107)
+ 2 βη (cid:104) w ( t ) − w ( t − 1) , −∇L ( w ( t ))
(cid:105) , while E w ( t +1) (cid:104) (cid:107) w ( t + 1) − w ( t ) (cid:107) 2 |{ w ( s ) } ts =1 (cid:105) can be calculated as
E w ( t +1) (cid:104) (cid:107) w ( t + 1) − w ( t ) (cid:107) 2 (cid:12)(cid:12) { w ( s ) } ts =1 (cid:105) =
E B ( t ) (cid:13)(cid:13) − η ∇L B
t ) ( w ( t ))
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
By Eqs.
(41) and (43), we have that to ensure Eq.
(42), it is required that
[Equation]
which puts constraint on β that
[Equation]
Speciﬁcally, β < 2 bN γ 2 1+ bN γ 2 , which approaches 0 when γ approaches 0 , and constrains β in a small range.
C I MPLICIT B IAS OF A DAM
This section collects the proof of the convergent direction of Adam, i.e., Theorem 4.
The methodology of this section bears great similarity with GDM, although the preconditioner of Adam requires speciﬁc treatment for analysis.
The proof is still divided into two stages: (1).
we ﬁrst prove the sum of squared gradients along the trajectory is ﬁnite.
Additionally, we prove the convergent rate of loss is O ( 1 t ) ; (2).
we prove w ( t ) − log( t ) ˆ w has bounded norm.
Before we present these two stages of proof, we will ﬁrst give the required range of η for which Theorem 3 holds.
C. C HOICE OF LEARNING RATE
Let H s 0 be the smooth parameter over [ s 0 , ∞ ) given by Assumption 3.
(D).
Let β 2 = ( cβ 1 ) 4 ( c > 1 ).
The "sufﬁciently small learning rate" in Theorem 3 means
[Equation]
To ensure η is well-deﬁned, we need to prove
[Equation]
and we introduce the following technical lemma:
Lemma 19.
Deﬁne f t ( x ) = 1 − x t x (1 − x t − 1 ) , ∀ t ∈ Z , t ≥ 2 .
We have f t ( x ) is decreasing with respect to x .
Furthermore, for any x ∈ [0 , 1) , we have
[Equation] [Equation]
Proof.
First of all, by deﬁnition,
[Equation]
is monotonously decreasing as 0 ≤ x < 1 .
Secondly, Eq.
(44) is equivalent to
[Equation]
The left side of the above inequality is no smaller than 1, while the right side is no larger than 1, which completes the proof.
We are now ready to prove η is well-deﬁned.
First of all, for every t , we have
[Equation] [Equation]
where Eq.
( (cid:63) ) is by Lemma 19 and cβ 1 = 4 √ β 2 < 1 .
On the other hand, we have
[Equation]
By Eq.
(45) and Eq.
(46), we obtain 1 − β t 1 1 − β 1 − 1 − β t − 1 1 c
[Equation]
1 − β 1 ) 1 −
[Equation]
cβ 1 ) t 1 − ( cβ 1 ) t − 1 is lower bounded by some positive constant across t , and η is well deﬁned.
C. S UM OF GRADIENTS ALONG THE TRAJECTORY IS BOUNDED
We start with the following lemma, which indicates L ( w ( t ))
+ (cid:107) 4 (cid:112)
[Equation]
[Equation]
+ ˆ ν ( t ) (cid:12) ( w ( t ) − w ( t − 1))
(cid:107)is a proper Lyapunov function for Adam.
Proof of Lemma 5.
We start with the case t = 1 .
To begin with
[Equation]
we have L is H (cid:96) − 1 ( N L ( w (1)))
smooth around w (1) .
By deﬁnition H x is non-increasing with respect to x
[Equation]
and since (cid:96) − 1 is also non-increasing, we have
[Equation]
which further indicates when α is small enough,
[Equation]
where in Eq.
( (cid:63) )
we denote
[Equation]
1))) , and the last inequality is due to 12 η α(cid:13)(cid:13)(cid:13) 4 (cid:112) ε 1 d
+ ˆ ν (1) (cid:12) ( w (2) − w ( t ))
(cid:13)(cid:13)(cid:13)
= o ( α 2 ) , and (cid:28) ∇L ( w (1)) , η 1 √ ε 1 d
+ˆ ν (1) (cid:12) ∇L ( w (1))
(cid:29) is positive.
Now if there exists an α ∈ (0 , 1) , such that Eq.
(10) fails, we denote α ∗ =
inf { α :
Eq.
(10) fails for 1 + α } .
We have α ∗ > 0 , and the equality in Eq.
(10) holds for 1 + α ∗ .
Therefore, we have for any α ∈ (0 , α ∗ ) ,
[Equation]
which by Lemma 10 leads to L is H (cid:96) − 1 ( N L ( w (1)))
smooth (thus L smooth) over the set { w (1 + α ) : α ∈ [0 , α ∗ ] } , and
L ( w (1 + α ∗ )) ≤L ( w (1))
+ α ∗ (cid:104)∇L ( w (1)) , w (2) − w (1) (cid:105)
[Equation] [Equation]
∗ )
[Equation]
(cid:107) w (2) − w (1) (cid:107)=
[Equation] [Equation]
∗
[Equation]
η (cid:112)
[Equation]
(1) (cid:12) ( w (2) − w (
[Equation]
[Table]
[Equation]
where the second-to-last inequality is due to (cid:107) w (2) − w (1) (cid:107) > 0 (by Lemma 13) and α ∗ > ( α ∗ ) 2 , while the last inequality is due to
[Equation]
Eq.
(47) contradicts the fact that the equality in Eq.
(10) holds for 1 + α ∗ , which completes the proof of t = 1 .
If t ≥ 2 , following the similar routine as t = 1 , we also prove Eq.
(10) by reduction to absurdity.
If there exist t and α such that Eq.
(10) fails.
Denote t ∗ as the smallest time such that there exists an α ∈ [0 , 1) such that Eq.
(10) fails for t ∗ and α .
By Lemma 13, (cid:13)(cid:13)(cid:13)
[Equation]
w
[Equation] [Equation]
w
[Equation]
t
[Equation]
(cid:13)(cid:13)(cid:13) 2 is positive, and strict inequality in Eq.
(10) holds for t and α = 0 , which by continuity leads to
[Equation]
Then, for any α ∈ [0 , α ∗ ] , we have
[Equation]
On the other hand, for any time 2 ≤ s ≤ t ∗ − 1 , we have
L (
[Equation]
( s +
[Equation]
−
[Equation] [Equation]
[Equation]
−
[Equation] [Equation]
(cid:13)(cid:13)(cid:13) 4 (cid:112)
[Equation]
( s ) (cid:12) ( w ( s + 1) − w ( s ))
(cid:13)(cid:13)(cid:13)≤L ( w ( s ))
[Equation] [Equation]
−
[Equation]
cβ
[Equation]
(cid:13)(cid:13)(cid:13) 4 (cid:112) ε
[Equation]
+ ˆ ν ( s − 1) (cid:12) ( w ( s ) − w ( s − 1))
(cid:13)(cid:13)(cid:13) 2 .
By Eq.
(46), we have 1 − β s
[Equation]
[Equation]
, which by (1 − β s − 1 1 ) (1 − β s 1 ) further leads to
[Equation]
≥L
[Equation]
4 (cid:112) ε 1 d + ˆ ν ( s ) (cid:12) ( w ( s + 1) − w ( s ))
(cid:13)(cid:13)(cid:13)>
L ( w ( s + 1))
+ 1 − β s2 cη (1 − β 1 ) 1 − ( cβ 1 ) s + 1 −
( cβ 1 ) s 1 − ( cβ 1 ) s 1 − ( cβ 1 ) s +1 (cid:13)(cid:13)(cid:13) 4 (cid:112)
ε 1 d + ˆ ν ( s ) (cid:12) ( w ( s + 1) − w ( s )) (cid:13)(cid:13)(cid:13)> 1 − ( cβ 1 ) s 1 − ( cβ 1 )
s + (cid:18)
L ( w ( s + 1))
+ 1 − β s2 cη (1 − β 1 ) 1 − ( cβ 1 ) s + 1 −
( cβ 1 ) s (cid:13)(cid:13)(cid:13) 4 (cid:112) ε 1 d
+ ˆ ν ( s ) (cid:12) ( w ( s + 1) − w ( s ))
(cid:13)(cid:13)(cid:13)(cid:19) .
On the other hand, for s = 1 , we have
L ( w (1)) ≥L
( w (2))
+ 1 2 1 − β 1 η (1 − β 1 ) (cid:13)(cid:13)(cid:13) 4 (cid:112)
ε 1 d + ˆ ν (1) (cid:12) ( w (2) − w (1))
(cid:13)(cid:13)(cid:13)≥ 1 − ( cβ 1 ) 1 − ( cβ 1 )(cid:18)
L ( w (2))
+ 1 − β2 cη (1 − β 1 ) 1 − ( cβ 1 )1 − ( cβ 1 ) (cid:13)(cid:13)(cid:13)
[Equation]
Combining Eqs.
(48), (49), and (50), we have
[Equation]
Therefore, by Lemma 10, L is H (cid:96) − 1 (
[Equation]
cβ 1 N L ( w (1)))
smooth (thus L smooth) over the set { w ( t ∗ + α ) : α ∈ [0 , α ∗ ] } , which further leads to
L
[Equation] [Equation]
t
[Equation] [Equation]
t
[Equation] [Equation]
=
[Equation] [Equation]
[Equation] [Equation]
t
[Equation]
) (cid:13)(cid:13)(cid:13)
[Equation]
+
[Equation]
α
[Equation] [Equation]
β
[Equation]
[Equation] [Equation]
t
[Equation]
t
[Equation] [Equation]
≤L ( w ( t
[Equation] [Equation]
L
[Equation] [Equation]
w
[Equation]
t
[Equation] [Equation]
t
[Equation] [Equation]
(cid:112)
[Equation]
t
[Equation]
w
[Equation]
t
[Equation] [Equation]
t
[Equation]
)) (cid:13)(cid:13)(cid:13)
[Equation]
+ β 1 ( α ∗ ) 2 (1 − β t ∗ − 1)
[Table] [Table]
[Equation]
where Eq.
( • ) is due to an alternative form of the Adam’s update rule:
[Equation]
Inequality ( (cid:5) ) is due to
[Equation]
1 −
[Equation]
β 2 (1 − β t
[Equation] [Equation]
) 1 d ( all the computings are component - wisely ) , and f ( cβ 1 ) ≥ 4 (cid:112) f
(( cβ 1 ) 4 ) , and Inequality ( (cid:3) ) is due to
L
[Equation]
≤
[Equation]
α ∗ > ( α ∗ ) 2 , and (cid:13)(cid:13)(cid:13) 4 (cid:112) ε 1 d
+ ˆ ν ( t
[Equation]
) (cid:12) ( w ( t ∗
[Equation]
1) − w ( t ∗
[Equation]
) (cid:13)(cid:13)(cid:13) 2 > 0 .
This contradicts to that the equality in Eq.
(10) holds for t ∗ + α ∗ .
The proof is completed.
As lim t →∞ β t 1 = 0 and lim t →∞ ( cβ 1 ) t = 0 , we have the following corollary based on Lemma 1.
Corollary 6.
Let all assumptions in Theorem 4 hold.
Then, for large enough t , we have
[Equation]
Consequently, we have
[Equation]
The proof of Corollary 6 requires the following classical lemma on the equivalence between the convergence of two non-negative sequence.
The proof is omitted here and can be found in Wang et al.
Lemma 20 (c.f. Lemma 27, Wang et al.
Let { a i } ∞ i =1 be a series of non-negative reals, and ε be a positive real.
Then, (cid:80) ∞ i =1 a i < ∞ is equivalent to (cid:80) ∞ i =
[Equation]
<
[Equation]
Proof of Corollary 6.
We have
[Equation]
which completes the proof of Eq.
Rearranging Eq.
(52) leads to
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
+ L
[Equation]
(cid:19) , which by iteration further leads to that for a large enough time T
T 2 (cid:88) t =
[Equation]
[Equation] [Equation]
Consequently, we obtain ∞
[Equation]
On the other hand, for any t , we have
[Equation]
(cid:112)
[Equation] [Equation]
ˆ ν
[Equation]
t ) (cid:12) ˆ w (cid:13)(cid:13)(cid:13) ≥ (cid:68)
[Equation]
(cid:112) ε 1 d + ˆ ν ( t )
[Equation]
w ( t + 1) − w ( t )) , 4 (cid:112) ε 1 d + ˆ ν ( t ) (cid:12) ˆ w (cid:69) =
(cid:68)
(cid:112) ε 1 d + ˆ ν ( t ) (cid:12) ( w ( t + 1) − w ( t )) , ˆ w (cid:69) =
(cid:104)− η ˆ m ( t ) , ˆ w (cid:105) = − η (1 − β 1 ) 1 − β t (cid:42) t (cid:88) s
= β t − s∇L ( w ( s )) , ˆ w (cid:43) = − η (1 − β 1 ) 1 − β t
N (cid:42) t (cid:88) s =
β t − s(cid:88)
˜ x i ∈T ( S ) (cid:96) (cid:48)
( (cid:104) ˜ x i , w ( s ) (cid:105)
[Equation]
which by Eq.
(54) indicates
[Equation]
As lim t →∞ (cid:16)
[Equation]
β t 1 (cid:17) 2 = η 2 (1 − β 1 ) 2 , we then obtain
[Equation]
which by Lemma 20 completes the proof
[Equation]
Based on Corollary
[Equation]
we can further prove Lemma
[Equation]
characterizing the convergent rate of loss L directly.
Proof of Lemma 6.
To begin with, Eq.
(51) indicates
[Equation]
[Table]
[Equation]
[Table]
On the other hand
[Table]
by Corollary 6,
[Equation]
which following the same routine as Corollary 5 leads to
[Equation]
Therefore, by Lemma
[Table]
there exists a large enough time T 1 , such that ∀ t ≥ T 1 ,
[Equation]
which by the separable assumption further leads to
[Equation]
Combining Eq.
(26) and the above inequality, we have
[Equation]
On the other hand
[Equation]
by Eq.
[Equation]
we have
[Equation]
Therefore
[Equation]
there exists large enough time T 2 , such that ∀ t > T 2 ,
[Equation]
1 , and thus,
[Equation] [Equation]
Combining Eq.
(57) and Eq.
(58), there exists a positive real constant C , such that
[Equation]
Rearranging Eq.
(52) leads to
[Equation]
(cid:13) (cid:13)
[Equation]
ε
[Equation]
ν
[Equation]
t
[Equation]
1) (cid:12)
[Equation]
w ( t )
[Equation]
t − 1)) (cid:13) (cid:13) (cid:13)
[Equation]
≤L ( w
[Equation]
t
[Equation] + [Equation] (cid:19) , which further indicates
[Equation]
Denote ξ ( t ) as
[Equation]
We then have ξ ( t
[Equation]
√
[Equation]
+
[Equation] [Equation]
which leads to ξ ( t )
=
O (cid:18)t (cid:19) , i.e., L ( w ( t ))
=
O (cid:18)t (cid:19) , and (cid:13)(cid:13)(cid:13) 4 (cid:112) ε 1 d + ˆ ν ( t − 1) (cid:12) ( w ( t ) − w ( t − 1))
(cid:13)(cid:13)(cid:13)
= O (cid:18) 1 t (cid:19) .
Due to Eq.
(56), we further have (cid:107)∇L ( w ( t ))
(cid:107) =
O ( t − 1 ) , which indicates (cid:107) w ( t ) (cid:107) ≤(cid:107) w (1) (cid:107) + t (cid:88) s = (cid:107)
w
( s + 1) − w ( s ) (cid:107) =
(cid:107) w (1) (cid:107) + η t (cid:88) s
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
ˆ m ( s ) (cid:112)
ˆ ν
( s ) + ε 1 d (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤(cid:107)
w (1) (cid:107) + η √ ε t (cid:88) s = (cid:107)
ˆ m ( s ) (cid:107) =
(cid:107) w (1) (cid:107) + η √ ε t (cid:88) s =  (1 − β s ) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) s (cid:88)
i =
β s − i ∇L ( w ( i ))
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≤(cid:107) w (1) (cid:107) + η √ ε (1 − β ) t (cid:88) s
= s (cid:88)
i =
β s − i (cid:107)∇L ( w ( i ))
(cid:107) ≤(cid:107)
w (1) (cid:107) + η √ ε (1 − β )t (cid:88) s
= (cid:107)∇L ( w ( s ))
(cid:107) =
O (log( t )) .
Therefore, for any ˜ x ∈ T ( S ) , we have (cid:104) w ( t ) , ˜ x
(cid:105)
=
O (log( t )) , which by (cid:96) is exponential-tailed leads to (cid:96) ( (cid:104) w ( t ) , ˜ x (cid:105) )
=
Ω( t − 1 ) , and thus L ( w ( t )) =
Θ( t − 1 ) .
Also, since L ( w ( t )) =
O ( t − 1 ) , we have (cid:104) w ( t ) , ˜ x
(cid:105)
= Ω(log( t )) , which further leads to (cid:107) w ( t ) (cid:107) =
Ω(log( t )) , and thus (cid:107) w ( t ) (cid:107) = Θ(log( t )) .
Finally, we have
[Equation]
[Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
which leads to (cid:107)
m ( t ) (cid:107) =
Θ( t − 1 ) .
Similarly, we have ν ( t ) =
O ( t − 2 ) , component-wisely.
As lim t →∞ β t 1 = 0 and lim t →∞ β t 2 = 0 , we have
[Equation]
The proof is completed.
C. B OUNDING T HE O RTHOGONAL P ART
By Lemma 8, there exists a solution ˜ w as the solution of Eq.
(12) with C 2 = η (1 − β ) √ ε .
Deﬁne r ( t ) as
[Equation]
[Equation]
and we only need to prove (cid:107) r ( t ) (cid:107) is bounded over time.
We then prove Lemma 7, providing an equivalent condition of (cid:107) r ( t ) (cid:107) being bounded.
As the GDM and SGDM case, we separate the proof into two sub-lemmas.
Lemma 21.
Let all conditions in Theorem 4 hold.
Then, (cid:107) r ( t ) (cid:107) is bounded if and only if g ( t ) is upper bounded.
Proof.
Following the same routine as Lemma 2 and Lemma 17, we only need to prove
[Equation]
60) and ∞ (cid:88) τ
= (cid:12)(cid:12)(cid:12)
[Equation]
r
[Equation] [Equation]
[Equation]
1 − β τ − 1) (cid:112) ε 1 d + ˆ ν ( τ − 1) (cid:12) ( w ( τ ) − w ( τ − 1))
(cid:105) (cid:12)(cid:12)(cid:12) < ∞ .
As for Eq.
(60), by Lemma 6, we have (cid:13) (cid:13)(cid:13) (1 − β t − 1) (cid:112) ε 1 d + ˆ ν ( t − 1) (cid:12) ( w ( t ) − w ( t − 1))
(cid:13) (cid:13)(cid:13) =
O ( t − 1 ) =
o (1) .
As for Eq.
(60), we have (cid:12)(cid:12)(cid:12)(cid:68) r ( τ ) − r ( τ − 1) , (1 − β τ − 1) (cid:112)
ε 1 d + ˆ ν ( τ − 1) (cid:12) ( w ( τ ) − w ( τ − 1))
(cid:69)(cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:28) w ( τ ) − w ( τ − 1) − log τ τ − 1 ˆ w , (1 − β τ − 1) (cid:112)
ε 1 d + ˆ ν ( τ − 1) (cid:12) ( w ( τ ) − w ( τ − 1))
(cid:29)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:28) log τ τ − 1 ˆ w , (1 − β τ − 1) (cid:112)
ε 1 d + ˆ ν ( τ − 1) (cid:12) ( w ( τ ) − w ( τ − 1))
(cid:29)(cid:12)(cid:12)(cid:12)(cid:12) + (1 − β τ − 1) (cid:13)
[Table]
ε
[Table]
+
[Table]
[Equation]
=
O ( τ − 2 ) , where Eq.
( (cid:63) ) is due to Lemma 6 and log( ττ − 1 )
=
Θ( τ − 1 ) .
The proof is completed.
We conclude the proof of Theorem 4 by showing g ( t ) is upper bounded.
Lemma 22.
Let all conditions in Theorem 4 hold.
Then, g ( t ) is upper bounded.
Proof.
g ( t ) is upper bounded is equivalent to (cid:80) ∞ t =1 g ( t + 1) − g
( t ) < ∞ .
We then prove this lemma by calculating g ( t + 1) − g ( t ) directly.
[Equation]
where Eq.
( (cid:63) ) is due to a simple rearranging of the update rule of Adam, i.e., β1 − β
[Equation]
On the one hand, as (cid:107) r ( t + 1) − r ( t ) (cid:107) =
(cid:107) w ( t + 1) − w ( t ) − log t +1 t ˆ w (cid:107) =
O ( t − 1 ) , ∞ (cid:88) t = √ ε 2 (cid:107) r ( t +
[Equation]
− r ( t
[Equation]
(cid:107) 2 < ∞ .
On the other hand,
[Equation]
where Eq.
( • ) is due to ˆ ν ( t )
=
O ( t − 2 ) .
Furthermore, following exactly the same routine as Lemma 16, we have
[Equation]
The proof is completed.
D A PPLICATIONS &E XTENSIONS D. A PPLICATION TO THE MINI - BATCH SGDM
This section provides formal description of the implicit bias of mini-batch SGDM and its corresponding proof.
To begin with, we would like to provide a formal deﬁnition of mini-batch SGDM.
Mini-batch SGDM differs from SGDM by applying sampling without replacement to obtain B ( t ) in
Eq.
Speciﬁcally, let K
= Nb .
For any T ≥ 0 , we call time series { KT + 1 , · · · , KT + K } the ( T + 1) -th epoch, and during the T + 1 -th epoch, the dataset S is randomly uniformly divided into K parts { B ( KT + 1) , · · · , B ( KT + K ) } , with (cid:83) KT + K t = KT +1 B ( t ) =
S .
The implicit bias of mini-batch
SGDM is then stated as the following theorem:
Theorem 6.
Let Assumptions 1, 2, and 3.
(S) hold.
Let learning rate η be small enough, and β ∈ [0 , 1)
.Then, for almost every dataset S , mini-batch SGDM satisﬁes w ( t ) − log( t ) ˆ w is bounded as t → ∞ , and lim t →∞ w ( t ) (cid:107) w ( t ) (cid:107) =
[Equation]
The without-replacement sampling method leads to the direction of every trajectory of mini-SGDM converge to the max-margin solution, compared to the same conclusion holds for SGDM a.s..
We prove the theorem following the same framework of GDM, by proceeding with two stages.
Stage I.
The following lemma proves L ( u ( t ))
is an Lyapunov function for mini-batch SGDM and without the a.s. condition.
Lemma 23.
Let all conditions in Theorem 6 hold.
Then, we have
[Equation]
s =
[Equation]
(cid:107)∇L
[Equation]
Proof.
By the Taylor Expansion of L ( u ( t + 1)) at u ( t ) , we have
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) .
On the other hand, for any t ∈ { 2 , · · · , K } , we have w ( KT + t ) − w ( KT + 1) =
η t (cid:88) s = (cid:32) KT + s (cid:88) (cid:96)
=
β KT + s − (cid:96) ∇L B ( (cid:96) )
( w ( (cid:96) )) (cid:33) =
η t (cid:88) s = (cid:32) KT + s (cid:88) (cid:96)
= KT + β KT + s − (cid:96) ∇L B ( (cid:96) )
( w ( (cid:96) ))
(cid:33) + η t (cid:88) s = (cid:32) KT (cid:88) (cid:96) =
β KT + s − (cid:96) ∇L B ( (cid:96) )
( w ( (cid:96) )) (cid:33) =
η t (cid:88) (cid:96) = 1 − β t − (cid:96) + 1 − β ∇L B ( KT + (cid:96) ) ( w ( KT + (cid:96) )) + η β (1 − β t ) 1 − β
KT (cid:88) (cid:96) =
β KT − (cid:96) ∇L B ( (cid:96) )
( w ( (cid:96) ))
=
η t (cid:88) (cid:96) = 1 − β t − (cid:96) + 1 − β ∇L B ( KT + (cid:96) ) ( w ( KT + (cid:96) ))
− η t (cid:88) (cid:96) = 1 − β t − (cid:96) + 1 − β ∇L B ( KT + (cid:96) )
( w ( KT + 1))
+ η β (1 − β t ) 1 − β
KT (cid:88) (cid:96) =
β KT − (cid:96) ∇L B ( (cid:96) )
( w ( (cid:96) ))
+ η t (cid:88) (cid:96) = 1 − β t − (cid:96) + 1 − β ∇L B ( KT + (cid:96) )
( w ( KT + 1)) ,
[Table]
[Equation]
Applying the same analysis to (cid:107) w ( KT + t − 1) − w ( KT + 1) (cid:107) recursively
[Equation]
we ﬁnally obtain (cid:107) w ( KT
[Equation]
− w
[Equation]
Applying Eq.
(63) to the (cid:13)(cid:13) ∇L B ( (cid:96) )
( w ( (cid:96) ))
(cid:13)(cid:13) in Eq.
(63) ( ∀ (cid:96) ∈ [1 , KT ] ) iterative and choosing η to be small enough
[Equation]
we further have
[Equation]
Therefore
[Equation]
Similarly
[Equation]
one can obtain ∇L
[Equation]
) (cid:13)(cid:13)(cid:33)(cid:33) .
Applying Eq.
(64) and Eq.
(65) back to the Taylor Expansion (Eq.
(62)), we have
[Equation]
Summing the above inequality over T and setting η small enough leads to the conclusion.
The proof is completed.
D. E XTENSION TO THE MULTI - CLASS CLASSIFICATION PROBLEM
Here we use several notations and lemmas from (Soudry et al., 2018).
We deﬁne w = vec ( W ) , ˆ w = vec ( ˆ W ) , e i ∈ R C ( i ∈ { 1 , · · · , C } ) satisfying ( e i )
j = δ ij , and A i =
e i ⊗ I d X , where I d X is the identity matrix with dimension d X .
We still consider the normalized data, i.e., (cid:107) x (cid:107) ≤ 1 , ∀ ( x , y ) ∈ S .
Then, the individual loss of sample ( x , y ) can be then represented as
[Equation]
Furthermore, the gradient of training error at W has the form
[Equation]
and the Hessian matrix of L can be represented as
[Equation]
one can then easily verify all absolute value of the eigenvalues of HL ( w ) is no larger than 2 , which indicates L is 2 -globally smooth.
On the other hand, the separable assumption leads to (cid:104) ˆ w , ( A y − A i ) x (cid:105) > 0 , ∀ y (cid:54) =
i , which further indicates (cid:104)∇L ( w ) , ˆ w (cid:105) > 0 .
Let γ = 1 (cid:107) ˆ w (cid:107) , following the similar routine as the binary case, we have for a random subset of S sampled uniformly without replacement with size b , we have
[Equation]
Similarly, we have for any positive real series { a t } t 2 t = t 1 ,
[Equation]
The proofs of Stage I can then be obtained with Lyapunov functions unchanged and by replacing the corresponding lemmas using Eq.
(66) and Eq.
As for the proofs of Stage II, the Lyapunov functions are still the same, while we only need to prove the sum of (cid:104) r ( t ) , − η 1 − β ∇L ( w ( t )) − log t +1 t ˆ w (cid:105) (for GDM, (cid:104) r ( t ) , − η 1 − β ∇L B ( t ) ( w ( t ))
−
Nbt (cid:80) i :˜ x i ∈T ( B ( t ) ∩ S s ) v i ˜ x
i (cid:105) for SGDM, (cid:104) r ( t ) , −√ ε log (cid:0) t +1 t (cid:1)
ˆ w − η 1 − β ∇L ( w ( t )) (cid:105) for Adam).
For the multi-class case using GDM, We present the following lemma from (Soudry et al., 2018), while the other two cases can be proved similarly:
Lemma 24 (Part of the proof of Lemma 20, (Soudry et al., 2018)) .
If (cid:104) w ( t ) , ( A y − A i ) x (cid:105) → ∞ as t → ∞ , ∀ ( x , y ) ∈ S and ∀ i (cid:54) =
y , we have the sum of (cid:104) r ( t ) , − η 1 − β ∇L ( w ( t )) − log t +1 t ˆ w (cid:105) is upper bounded.
The proof of Theorem 5 is then completed.
E E XPERIMENTS
This section collects several experiments supporting our theoretical results.
E. E XPERIMENTS OF LINEAR MODEL E.1. C OMPARING THE TRAINING BEHAVIOR OF GD, GDM, AND A DAM ( W / S )
The experiments in this section is designed to verify Theorem 2, i.e., with proper learning rates, gradient descent with momentum converges to the max margin solution, which is the same as gradient descent.
We use the synthetic dataset as (Figure 1, (Soudry et al., 2018)) and run GD, GDM and Adam (w/s) over it with different learning rates η = 0 .
01 and different random seeds (for random initialization and random samples despite the support sets { ((1 . 5 , 0 . 5) , 1) , ((0 . 5 , 1 . 5) , 1) , (( − 1 .
5 , − 0 .
5) , − 1) , (( − 1 .
5 , − 0 .
5) , − 1) } ).
Both the angle between the output parameter and max margin solution and the training accuracy are plotted in Figure 1.
Speciﬁcally, we plot the results with learning rate (1).
η GD = η GDM =
η
Adam = 0 .
η GD = η GDM =
η
Adam = 0 .
We plot the training accuracy and the angle between the output parameter and the max margin solution for each setting respectively.
The results are shown in Figure 1.
The observations can be summarized as follows:
• When learning rate is small enough ( η = 0 . 1 , 0 . 001 ), both GD, GDM, and Adam converge to the max margin solution, which supports our theoretical results; • (Similarity between GD and GDM)
The training behaviors of GD and GDM are highly similar.
• (The acceleration effect of Adam) Adam achieves smaller angle with the max margin solution under the same number of iterations.
E.1. C OMPARING THE TRAINING BEHAVIOR OF SGD AND SGDM
We also run the stochastic optimizers (SGD and SGDM) on the same synthetic dataset as Figure 1.
The learning rate is same as Figure 1, namely 0 .
001 and 0 .
The results of the experiment are plotted in Figure 2.
It can be observed that when learning rate is small enough, SGD and SGDM have the similar training behavior, both converging to the max margin solution.
E.1. A DAM ON ILL - POSED D ATASET IN (S OUDRY ET AL ., 2018)
In Figure 3 of (Soudry et al., 2018), an ill-posed synthetic dataset is proposed to support the argument "Adam does not converge to max margin solution", which contradicts to the theoretical results of this paper.
We re-conduct the experiment of Figure 3 in (Soudry et al., 2018) with the same ill-posed synthetic dataset with different learning rates and different random seeds as Figure 3.
Figure 3.
(f) is similar to Figure 3 in (Soudry et al., 2018), where with learning rate η = 0 .
1 and random seed 1 , the angle of GD to the max margin solution is smaller than Adam all the time.
{+However, it can be observed from the ampliﬁed ﬁgure that the angle of GD keeps still and above 
+}
all the time, meaning that GD doesn’t converge to the max margin solution under this setting.
However, the angle of Adam to the max margin solution still keeps decreasing and it’s unreasonable to claim "Adam doesn’t converge to the max margin solution" in this case (the same issue exists in Figure 3 in (Soudry et al., 2018)).
Also, as we mentioned at the beginning of this section, this dataset is ill-posed, which is due to the imbalance between the two components of the data (for all data (( x 1 , x 2 ) , y ) in the dataset,
[Figure]
[Figure]
| x 1 | is always smaller than 2 , while | x 2 | is larger than 10 (and even larger than 30 despite two data in the dataset)), which requires smaller learning rate.
To tackle this problem, we need to tune down the learning rate.
By Figure 3.
(b),(d) and Figure 4.
(b), after scaling down the learning rate, both GD’s angle and Adam’s angle keep decreasing.
E. E VIDENCE IN THE DEEP NEURAL NETWORKS
We conduct an experiment on the MNIST dataset using the four layer convolutional networks used in (Lyu & Li, 2019; Wang et al., 2021) (ﬁrst proposed by (Madry et al., 2018)) to verify whether
SGD and SGDM still behave similarly in (homogeneous) deep neural networks.
The learning rates of the optimizers are all set to be the default in Pytorch.
The results can be seen in Figure 5.
It can be observed that (1).
SGDM achieves similar test accuracy compared to SGD while (2).
SGDM converges faster than SGD.
[Figure]
[Figure]
[Figure]