Revisit last-iterate convergence of mSGD under milder requirement on step size
Author(s)
AffiliationAddress
Abstract
Understanding convergence of SGD-based optimization algorithms can help dealwith enormous machine learning problems.
To ensure last-iterate convergence of
SGD and momentum-based SGD (mSGD), the existing studies usually constrainthe step size ϵ n to decay as (cid:80) + ∞ n =1 ϵ 2 n < + ∞ , which however is rather conservativeand may lead to slow convergence in the early stage of the iteration.
In this paper,we relax this requirement by studying an alternate step size for the mSGD.
First,we relax the requirement of the decay on step size to (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ (0 ≤η 0 < 1 / 2) .
This implies that a larger step size, such as ϵ n = 1 √ n can be utilizedfor accelerating the mSGD in the early stage.
Under this new step size and somecommon conditions, we prove that the gradient norm of mSGD for non-convex lossfunctions asymptotically decays to zero.
In addition, we show that this step sizecan indeed help make the convergence into a neighborhood of the stationary pointsquicker in the early stage.
In addition, we establish the convergence of mSGDunder a constant step size ϵ n ≡ ϵ > 0 by removing the common requirement in theliterature on the strong convexity of the loss function.
Some experiments are givento illustrate the developed results.
Introduction
The booming development of machine learning over the past decade relies on the employment ofeffective optimization algorithms.
A large number of such optimization algorithms are based ongradient descent (GD).
The optimization problem in machine learning can be cast as minimizing aloss function g ( θ ) ∈ R over the choice of an N -dimensional real-valued parameter θ , i.e., by solvingthe problem
[Equation]
This problem can be solved with a typical GD algorithm through an iteration of the form
[Equation]
where θ n is the estimate of θ ∗ at step n , ϵ n is a positive step size (learning rate) to be designed, and∇ θ n g ( θ n ) stands for the gradient of g ( θ n ) at step n .
Under certain technical conditions, θ n in (2)can asymptotically (as n → ∞ ) converge to the optimal solution θ ∗ .
However, (2) is not efficientin machine leaning applications especially with enormous data.
To accelerate (2), one attempt isstochastic gradient descent (SGD) originating from [1].
Instead of calculating ∇ θ n g ( θ n ) over allthe data, SGD computes gradient estimate ∇ θ n g ( θ n , ξ n ) by sampling a subset of data (using thedistribution of the sampling noise { ξ n } and then updates the iterate as follows
[Equation]
In addition to the sampling noise, ξ n can be used to model any external noises arising from thegradient computation.
However the price to pay for this data efficiency is that the employment of stochastic gradients (3)displays a relatively slow convergence rate during the learning process.
In order to improve theconvergence rate of SGD, momentum-based stochastic gradient descend (mSGD), which reduces theupdate variance by averaging the past gradients ([2]), has been proposed.
An iteration in mSGD is asfollows ([3, 4]):
[Equation]
where α ∈ [0 , 1) and ϵ n > 0 are momentum coefficient and step size (learning rate), respectively.
Analternative formulation of mSGD, named stochastic heavy ball (SHB) [5, 6], has also been proposedas follows
[Equation]
where β n ∈ (0 , 1) and γ n are momentum coefficient and step size.
It has been shown that mSGDand SHB are essentially equivalent [7].
In recent years, mSGD has been widely employed inmany applications of deep learning such as image classification [8], fault diagnosis [9], statisticalimage reconstruction [10], among others.
Moreover, a number of variants on momentum have beenemerging, see, e.g., synthesized Nesterov variants [11], robust momentum [12], and PID-controlbased methods [13].
The importance of momentum in deep learning has been illustrated in [4]through experiments.
There has been a long line of literature analyzing the convergence of SGD and mSGD algorithms.
Regarding SGD, authors in [14] studied the last-iterate convergence when the step size is chosen todecay as 1 /n .
Authors in [15, 16] established the last-iterate convergence of SGD for strongly convexloss functions without the bounded gradients assumption.
For the normalized mSGD (SHB), Polyak[17, 2] and Kaniovski [18] established the convergence (subsequence convergence and convergence oftime averages) properties for convex loss functions.
Igor Gitman [5] provided convergence results ofmSGD (SHB) for non-convex loss functions under the somewhat restrictive requirement for uniformboundedness of a noise term E
( ∥∇ θ n g ( θ n , ξ n ) −∇ θ n g ( θ n ∥ ) ≤ δ .
It has been pointed out [19, 4] thatthe designs of momentum coefficients in [17, 2, 18, 5] may not be consistent with the requirements ofsome practical applications.
The last-iterate convergence of mSGD for non-convex loss functions hasrecently been established in [20, 7] with the step size condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ , which has beentraditionally utilized in stochastic optimization [1, 21, 16, 15].
Under the strong convexity condition,since this condition requires the step size to decay relatively fast, it may lead to slow convergenceespecially in the early stage of the iteration.
With the requirement that the loss function is stronglyconvex, [22] showed that the last-iterate of mSGD can converge to a neighbor of the stationary pointwhen the step size is a constant.
Problem of interest and contributions
In this paper, we consider the problem of last-iterate convergence of mSGD under relaxed requirements on step size.
Specifically, we seek to relax the condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ [1, 21, 16, 15, 7]that has been required in the literature for proving last-iterate convergence of mSGD.
We considertwo possible step sizes: step size that decay as ϵ n = 1 √ n and constant step size ϵ n ≡ ϵ > 0 .
Usingsuch larger step size is expected to lead to the last iterate of mSGD (i.e., θ n ) converging faster.
Wenote that since the SGD (3) is a special case of mSGD (4), the developed results in this paper formSGD also work for SGD.
The contributions of this paper are summarized as follows:
1) Under the setting of step size decaying as ϵ n = 1 √ n , we prove that the gradient norm of mSGDfor non-convex loss functions asymptotically decays to zero.
This result is more general than meansquare gradient convergence established in [20].
In fact, we generalize the result for any step sizes ofthe form (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ (0 ≤ η 0 < 1 / 2) , which is a superset of the widely required condition 76 (cid:80) + ∞ n =1 ϵ 2 n < + ∞ [1, 21, 16, 15, 7].
Given this condition, one can run mSGD by employing a largerstep size like ϵ n = 1 √ n for last-iterate convergence.
[Equation]
2) Under some mild conditions, we provide an estimate of the convergence rate of mSGD.
Furthermore, we reveal that the new step size can indeed help to improve the convergence speed in the earlystage of the iteration.3) Under the setting of constant step size, first we prove that given any small neighborhood ofstationary points, it is feasible to design a constant step size, such that there is a convergent iteratesubsequence of mSGD staying within the neighborhood almost surely.
In addition, we prove thatthe mean-square gradient can be arbitrary small by tunning the step size.
Comparing with [22], weremove the requirement of strong convexity on loss function.
Regarding the definitions on sequence convergence, the following ones from literature are typical.
For a stochastic variables sequence { ξ n } ∈ R N with 0 as the only limit point, the sequence issaid to satisfy last-iterate convergence if lim n → + ∞ ∥ ξ n ∥ = 0; and time-average (mean-squareconvergence) if lim n → + ∞ 1 n (cid:80) nk =1 E ( ∥ ξ k
∥ 2 ) = 0 .
It can be proved that last-iterate convergenceimplies time-average convergence under the general setting of the considered problem.
Paper outline.
The rest of the paper is organized as follows.
In Section 3, we provide the mainresults of the paper on the convergence of mSGD under decaying step size and constant step sizerespectively.
In Section 4, two experiments are given.
Conclusion is made in Section 5.
The mainproofs are given in Appendix.
Main results
In this section, we provide the main results of this paper on the last-iterate convergence for mSGD in(4) under two possible step sizes: step sizes that decay as ϵ n = 1 √ n and constant step size ϵ n ≡ ϵ > 0 .
3.1 Convergence of mSGD with decaying step size
The following assumptions are needed in this paper.
Assumption 3.1.
Loss function g ( θ ) in (1) satisfies the following conditions:
Noise sequence { ξ n } are mutually independent and independent of θ 1 and v 0 , such thatg ( θ )
=
E ξ n (cid:0) g ( θ, ξ n ) (cid:1) for any θ ∈ R N .2.
g ( θ ) is a non-negative and continuously differentiable function.
The set of its stationarypoints J = { θ |∥∇ θ g ( θ ) ∥
= 0 } is a bounded set which has only finite connected components
J 1 , ..., J n .
In addition, there is ϵ 1 > 0 , such that for any i and 0 <
d ( θ, J i ) <
ϵ 1 , it holdsthat (cid:12)(cid:12) g
( θ ) − g i (cid:12)(cid:12) ̸ = 0 , where g i = { g ( θ ) | θ ∈ J i } is a constant.3.
There are two constants M ′ > 0 and a ′ > 0 such that for any θ ∈ R N and n ∈ N + ,
[Equation]
The gradient ∇ θ g ( θ ) satisfies the Lipschitz condition, i.e., there is a constant c > 0 , suchthat for any x, y ∈ R N ,
[Equation]
Conditions 1 and 4 in Assumption 3.1 are common in the literature [1, 21, 16, 15].
[7] required acondition on strong growth, i.e., E ξ n ∥∇ ξ n g ( θ, ξ n ) ∥ 2 ≤ M 0 ∥∇ θ g ( θ ) ∥ 2 , which however makes itclose to the deterministic case.
In contrast, condition 3 allows more randomness of data sampling.
Condition 2 is a mild condition for the reasons as follows.
In some works, the non-negative conditionmay be replaced by a lower bound condition g ( θ ) >
ˆ l 0 > −∞ .
These two conditions are essentiallyequivalent, since one can construct a new loss function g = g − ˆ l 0 under the lower bound condition,such that the new loss function is non-negative.
The rest of Condition 2 is quite general, since itallows the loss function to have multiple stationary points and to be non-convex.
Assumption 3.2.
In the mSGD (4) , momentum coefficient α ∈ [0 , 1) and the sequence of step size ϵ n ispositive and monotonically decreasing to zero, such that (cid:80) + ∞ n =1 ϵ n = + ∞ and (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ ,where 0 ≤ η 0 < 1 / 2 is a constant.
In Assumption 3.2, the momentum coefficient α ∈ [0 , 1) is a constant.
Compared with the setting in[6, 17, 18, 5], where α ( n ) tends to 1 or 0 , constant momentum coefficient is more general in practice[7, 19, 4].
Regarding the step size condition, it is more general than the one in many existing workson last-iterate convergence of SGD and mSGD [1, 21, 16, 15, 7, 13], i.e., (cid:80) + ∞ n =1 ϵ 2 n < + ∞ , whichis obtained from Assumption 3.2 if η 0 = 0 .
Under the step size condition in Assumption 3.2, onecan choose larger step size like ϵ n = 1 √ n , which however is not feasible in the commonly requiredcondition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ .
Since the new step size can decay more slowly than the existing one,it provides more space for step size fine tinning, such that the algorithm can quickly converge to aneighborhood of the stationary point [23, 24, 22].
Then we attain the first main result in this paper on last-iterate convergence of mSGD with the newstep size satisfying Assumption 3.2.
Theorem 3.1.
Consider the mSGD in (4) with any v 0 ∈ R N and θ 1 ∈ R N .
Under Assumptions 3.1 and 3.2, the gradient norm tends to 0 almost surely, i.e.
[Equation]
This theorem shows that the gradient norm of the last iteration θ n is convergent to 0 almost surely.
Due to page constraint, the complete proof is given Appendix.
Here, we provide a proof outline topresent the main proof ideas.
Proof Sketch of Theorem 3.1 : We aim to prove that ∇ θ n g ( θ n ) → 0 a.s. via the following key steps.
Step 1:
In this step we aim to prove E (cid:0)
ϵ η 0 n g ( θ n ) (cid:1) is bounded.
We present this result as Lemma B.8in Appendix.
Step 2: We prove there exists a subsequence of ∇ θ n g ( θ n ) which is convergent to 0 with probability1 .
It is attained by proving (cid:80) + ∞ n =1 ϵ 1+2 η 0 n ∥∇ θ n g ( θ n ) ∥ 2 < + ∞ a.s. .
We present this result as Lemma
B.10 in Appendix.
Step 3: We aim to extend the subsequence convergence in Step 2 to the asymptotic convergence.
Thebasic idea is to prove that the adjacent terms g ( θ n ) and g ( θ n +1 ) are “close" enough, such that
[Equation]
where ˆ k > 0 is a constant and { Q n } is a sequence, such that (cid:80) + ∞ n =1 Q n convergence a.s.. This resultis presented in Lemma B.11.
Step 4: Finally, we use some techniques to attain the result of asymptotic convergence ∇ θ n g ( θ n ) →
a.s. .
Technically, there are several big challenges when we try to extend the condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞to (cid:80) + ∞ n =1 ϵ 2+ δ 0 n < + ∞ with 0 ≤ η 0 < 1 / 2 .
One of the challenges is: Usually when we try to prove 149 (cid:80) + ∞ n =1 ζ n = (cid:80) + ∞ n =1 ϵ n ∇ θ n g ( θ n ) T ( ∇ θ n g ( θ n , ξ n ) − ∇ θ n g ( θ n )) is convergent a.s., which is the vitalstep to get the last-iterate convergence, we use the Martingale convergence theorem , throughwhich we just need to prove (cid:80) + ∞ n =1 E ( ζ 2 n ) is convergent.
If the condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ holds, wecan use the following inequality to get (cid:80) + ∞ n =1 E ( ζ 2 n ) <
(cid:80) + ∞ n =1 ˆ Kϵ 2 n < + ∞
( ˆ K is a constant) directly.
However, under the new condition (cid:80) + ∞ n =1 ϵ 2+ δ 0 n , we can not use this approach to prove it.
Instead, weintroduce a more technical approach to handle this problem.
Comparing with the corresponding result in [7] (Theorem 1 in [7]), we relax the condition on thedecay of step size.
With this new condition, one can utilize step size with slow decay in the iterationof mSGD.
Let α = 0 in mSGD, then we can get a similar result for SGD
Corollary 3.1.
Consider the SGD in (3) with any θ 1 ∈ R N .
Under Assumptions 3.1 and 3.2 with α = 0 , there is the gradient norm trend to 0 almost surely, i.e.
[Equation]
In the following, we aim to show the last-iterate convergence rate under the new step size condition.
Since ∥∇ θ n g ( θ n ) ∥ 2 is a stochastic variable, it is not favorable to estimate the almost-sure convergence rate of ∇ θ n g ( θ n ) .
Instead, the convergence rate of E (cid:0) ∥∇ θ g ( θ n ) ∥ 2 (cid:1) is quite meaningful.
In general,if we need to estimate the convergence rate of the last iterate, we usually need some extra assumptionsto establish a quantitative relationship between g and ∇ g .
Existing works were established usuallyunder the strong-convex assumption [25, 26, 13], local P-L condition [7] or convex loss functionwith some stability conditions, i.e., E ∥∇ θ n g ( θ n ) ∥ 2 < δ < + ∞ .
These assumptions are all relativelystrong to some extent, especially the stability condition which is very difficult to verify.
In our paper,we need a milder assumption than the above conditions as follow.
Assumption 3.3.
Regarding the loss function g ( θ ) defined in (1) , there exists δ 0 > 0 , ˆ T > 0 , such thatfor any θ ∈ { θ |∥∇ θ g ( θ ) ∥ 2 / ( g ( θ ) − g ∗ ) <
δ 0 } , it holds that g ( θ ) < ˆ T , where g ∗ = inf θ ∈ R N g
( θ ) .
This assumption requires the value of the loss function g ( θ ) to be bounded in the set S
={ θ |∥∇ θ g ( θ )
∥ 2 / ( g ( θ ) − g ∗ ) <
δ 0 } .
The motivation of this assumption is: For a stochastic algorithm, there is always a probability at which the iterates are far away from the true value.
Sincesecond-derivative information will have a more positive effect on convergence (such as stronglyconvex loss functions), this condition restricts iterates which are far away due to extreme noise, sothat it does not have a significant effect on global updates.
This requirement is milder than the strongconvexity condition, which is commonly used in existing works [25, 26, 13] for the analysis of thelast-iterate convergence.
In addition, this assumption allows the loss function to be non-convex,and does not need the local P-L condition used in [7] on the second derivative information nearthe stationary point.
Note that Assumption 3.3 can be satisfied by many loss functions commonlyencountered in machine learning.
First of all, many loss functions usually used are bounded [27–30],and thus satisfy Assumption 3.3.
For unbounded loss functions, there are a number of instancessatisfying Assumption 3.3.
For linear regression functions, since they are strongly convex, it is easyto verify the assumption.
In the Appendix, we explain (not in a strict but an illustrating manner) thisassumption can also be satisfied by a two layer neural network with Relu active function under asquare loss function.
In the following theorem, we provide an estimate of the convergence rate for the mSGD in (4).
Theorem 3.2.
Consider the mSGD in (4) with any v 0 ∈ R N under Assumptions 3.1–3.3.
Then existsa > 0 , for any ∥∇ θ 1 g ( θ 1 ) ∥ 2 , there is
[Equation]
(7) where ˆ c = min { a/ 2 T 0 , 2 δ 0 } .
Comparing with the result of the last-iterate convergence rate in [7], we remove the strong growth condition, i.e., E ξ n ∥∇ θ g ( θ, ξ n ) ∥ 2 ≤ M ∥∇ θ g ( θ ) ∥ 2 , and relax the step size condition to be (cid:80) + ∞ n =1 ϵ 2+ δ 0 n (0 < δ 0 ≤ 1 / 2) .
In the comparison with [20], we do not require the convexity of loss function or E ∥∇ θ n g ( θ n ) ∥ 2 < G .
In addition, we relax the step size condition in [20].
There are quite a few results on the average iterates in the literature, such as [31, 32, 13].
In [31, 32, 13], the convergence rate O ( 1 √ n ) of mSGD is established with the step size 1 √ n .
According to our result, let ϵ n = 1 √ n , if holds that
[Equation]
It can be proved that the convergence rate of average iterates is also O (cid:16) 1 √ n (cid:17) .
From (7), we see that larger step size may not lead to larger convergence rate.
This is reasonable,since the convergence rate reflects how quickly the iterate θ n converges when n → + ∞ and a largerstep size usually makes the iterate converges to a neighborhood of stationary point quicker.
We havethe following theorem on this point.
Theorem 3.3.
Consider the mSGD in (4) with any v 0 ∈ R N .
Under Assumptions 3.1–3.3, given anya > 0 , if ∥∇ θ 1 g ( θ 1 ) ∥ 2 > a , then it holds that
[Equation]
where τ ( a ) = min n> 0 {∥∇ θ n g ( θ n ) ∥ 2 < a } and ˆ c = min { a/ 2 T 0 , 2 δ 0 } .
Proposition 3.1.
Consider the mSGD in (4) with any v 0 ∈ R N .
Under Assumptions 3.1–3.3, givenany a > 0 , if ∥∇ θ 1 g ( θ 1 ) ∥ 2 > a , then there exists ˆ k 0 > 0 , when n is sufficiently large, it holds that
[Equation]
where τ ( a ) = min n> 0 {∥∇ θ n g ( θ n ) ∥ 2 < a } .
Theorem 3.3 and Proposition 3.1 provides a probability description of how fast the iterates of mSGDconverge into the preset neighborhood of stationary points.
It can be illustrated as follows.
It followsfrom Theorem 3.3 that
[Equation]
where ˆ k 1 > 0 is a constant.
From the inequality, as n increases, the event ∥∇ θ n g ( θ n ) ∥ 2 < a occurswith higher and higher probability tending to probability 1.
The influence of momentum coefficient αand step size ϵ n is also quantified.
We note that with the measure in Theorem 3.3, the convergenceto a neighbor of the stationary point is faster than under the traditional condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞by setting the step size with a relatively slow decay speed as (cid:80) + ∞ n =1 ϵ 2+ δ 0 n (0 < δ 0 ≤ 1 / 2) , such asϵ n = 1 √ n and η 0 = 1 / 4 .
Similar results (the large step size can make the iteration quickly convergeto a neighbor of stationary point, but may converge slowly in the late stage of the algorithm) appearin [23, 24, 22] through experiments.
3.2 Convergence of mSGD with constant step size
In the previous subsection, we establish the last-iterate convergence of mSGD under a relaxed decaycondition on step size.
In this subsection, we study the convergence of mSGD under a constant stepsize ϵ n ≡ ϵ > 0 .
We start with the following result on subsequence convergence.
Theorem 3.4.
Consider the mSGD in (4) with any v 0 ∈ R N and θ 1 ∈ R N .
Under Assumption 3.1,for any φ > 0 , there exists µ ( φ )> 0 , such that for any ϵ n ≡ ϵ < µ ( φ ) 0 , it holds that
[Equation]
where { θ ( φ ) k n } is a subsequence { θ n } .
Proof.
Due to page constraint, the complete proof is given Appendix.
Here, we provide a proofoutline to present the main proof ideas.
To study the mSGD with constant step size, we notice the fact that the impact of the noisecannot be totally eliminated through the step size.
But when ∥∇ θ n g ( θ n ) ∥ is relatively large(meaning the distance between θ n and the stationary point θ ∗ is relatively large), it holds that
E ξ n (cid:0) ∥∇ θ n g ( θ n , ξ n ) ∥ 2 (cid:1) ≤ M ′ ∥∇ θ n g ( θ n ) ∥ 2 + a ′ ≤ ( M ′ + ˆ k ) ∥∇ θ n g ( θ n ) ∥ 2 .
This implies that witha proper step size ϵ , the noise can be controlled.
As a result, the algorithm can be stabilized around astationary point.
In the following, several key steps in the proof are provided.
Step 1: We define an event by A ( φ ) n = (cid:8) ∥∇ θ 1 g ( θ 1 ) ∥ 2 > φ, ∥∇ θ 2 g ( θ 2 ) ∥ 2 > φ, ..., ∥∇ θ n g ( θ n ) ∥ 2 >φ (cid:9) and a characteristic function of this event by I ( φ ) n .
Step 2: We prove that the probability of A ( φ ) n is tending to 0 as n → ∞ .
First, we attain a recursionformula:
E (cid:0)
[Equation]
By taking the sum of multiple inequalities as above
[Equation]
we have
[Equation]
This leads to
[Equation]
From the Chebyshev inequality , we attain P ( A ( φ ) n ) → 0 .
Step 3:
Step 2 implies that with probability 1 that there exists at least one instant n at which∥∇ θ n g ( θ n ) ∥ 2 ≤ φ .
We note that for the time instant m at which ∥∇ θ m g ( θ m ) ∥ 2 > φ , then we canuse a new start point.
By repeating the process of Step 2, we attain the conclusion of this theorem.
Theorem 3.4 shows that under the common conditions in the literature, the constant step size canguarantee the subsequence convergence of mSGD (4) in any small neighborhood of stationary point.
In addition, we have a similar result on E (cid:0) ∥∇ θ n g ( θ n ) ∥ 2 (cid:1) as follows.
Proposition 3.2.
Suppose { θ n } is a sequence generated by (4) with any θ 1 ∈ R N and v 0 ∈ 241 R N .
Under Assumptions 3.1 and 3.3, if g ( θ ) has only one optimal point g ∗ = g ( θ ∗ ) andinf d ( θ,θ ∗ ) >δ 0 ∥∇ θ g ( θ ) ∥ 2 > 0 ( ∀ δ > 0) , then for any φ 0 > 0 , there exist scalars µ ( φ 0 )> 0and n 0 > 0 , such that for any ϵ n ≡ ϵ < µ ( φ 0 )and n > n 0 , it holds that
[Equation]
This result is a substantial extension of the existing results on SGD with constant step size to mSGD.
Comparing with the results in [22], we remove the requirement of µ -strong convexity on loss function.
Experiments
Since this paper focuses on convergence analysis on the well-known mSGD, two experiments onregression and classification tasks are given respectively to show the effectiveness of the results.
4.1 Regression task
In this subsection, we study a house price prediction problem with neural networks trained by mSGD.
Network Architectures .
We respectively employ a 3-layer and a 4-layer fully-connected neuralnetwork with ReLu active function and squared loss function.
Implementation .
We implement two neural networks to train on this dataset using Keras.
The firstneural network consists of three fully connected layers with 13 , 8 , and 1 neurons.
The second neuralnetwork consists of four fully connected layers with 13 , 32 , 16 , 1 neurons respectively.
We initializethe weights use glorot uniform algorithm.
We use mSGD with a mini-batch size of 16, and use meansquare error loss function to train the model.
The momentum term coefficient is 0 .
9 and the modelsare trained for up to 1000 epochs, training takes about two hours at a time in 3080GPU.
We do notuse dropout.
Dataset .
The training dataset we use is from Boston House Price Dataset.
This dataset belongs tothe regression prediction dataset, which consists of 506 sets of data with dimension 13.
During thetraining process, we randomly divide 30% as the test set and normalize the data before training.
Results .
By using the mSGD in (4) with three different step sizes respectively, i.e., ϵ n = 1 n , ϵ n = 1 n 0 . 7 ,and ϵ n = 1 n 0 . 5 .
The experiment results are given in Figure 1.
The figure shows that 1) the lossdecays to zero under the three step size setting; 2)
ϵ n = 1 n 0 .
5 can make the loss tend to a smallneighborhood of zero fastest among the three step size settings.
This conforms to the theoreticalanalysis in Theorems 3.1–3.3.
4.2 Classification task
In this subsection, we study a image classification problem with neural networks trained by mSGD.
Network Architectures .
We employ a 20-layer ResNet network.
The convolutional layers mostlyhave 3×3 filters and follow two simple design rules: (i) for the same output feature map size, thelayers have the same number of filters; (ii) if the feature map size is halved, the number of filters
[Figure]
[Figure]
is doubled so as to preserve the time complexity per layer.
We perform downsampling directly byconvolutional layers that have a stride of 2 and padding with valid.
The network ends with an averagepooling layer and a 10-way fully-connected layer with softmax.
Implementation .
We implement the ResNet20 network using Keras.
We initialize the weights useglorot uniform algorithm.
We use mSGD with a mini-batch size of 64, momentum coefficient 0 .
9and use categorical crossentropy loss function to train the model.
The models are trained for up to1000 epochs, taking about two hours at a time in 3080GPU.
We do not use dropout.
Dataset .
We use two different datasets CIFAR-10 and CIFAR-100.
CIFAR-10 consists of 50ktraining images and 10k testing images in 10 classes.
CIFAR-10 is a color image dataset closerto ubiquitous objects, and the image size is 32x32x3.
CIFAR-100 consists of 50k training imagesand 10k testing images in 100 classes.
Each category contains 500 training images and 100 testingimages.
CIFAR-100 is a color image dataset closer to ubiquitous objects,and the image size is 32x32x3.
Normalize the dataset between 0-1 before training.
Results .
By using the mSGD in (4) with three different step sizes respectively, i.e., ϵ n = 1 n , ϵ n = 1 n 0 . 7 ,and ϵ n = 1 n 0 . 5 .
The experiment results are given in Figure 2 and Figure 3.
The figures show that 1)the loss decays to zero and the accuracy goes up to 1 under the three step size setting; 2) ϵ n = 1 n 0 .
5can make the loss tend to a small neighborhood of zero and make the accuracy tend to a smallneighborhood of 1 fastest among the three step size settings, followed by the setting of ϵ n = 1 n 0 .
These results theoretical analysis
Theorems 3.1–3.3.
L o ss
Epochs mSGD
[Figure]
.1/n
[Figure] [Figure]
0.1/n^0.
[Figure]
A cc u r ac y
Epochs
[Figure]
0.1/n
[Figure] [Figure]
0.1/n^0.
[Figure]
Conclusion
In this paper, we studied the last-iterate convergence of momentum-based stochastic gradient descent(mSGD) under relaxed conditions on step sizes.
First, under the relaxed condition (cid:80) + ∞ n =1 ϵ 2+ η 0 n <+ ∞ (0 ≤ η 0 < 1 / 2) , we proved the last-iterate convergence of mSGD for non-convex loss functions.
In addition, we showed that this step size can indeed help to improve the convergence speed in theearly stage of the algorithm by quantifying the influence of the step size and momentum coefficient.
This implies that a larger step size, such as ϵ n = 1 √ n can be utilized for accelerating the mSGD.
Wealso proved that the algorithm with a constant step size (i.e., ϵ n ≡ ϵ > 0 ) can ensure the last-iterateconvergence of mSGD without requiring the strong convexity assumption on loss function.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental
results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e
g., data splits, hyperparameters, how they were chosen)?
[Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experiments
multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e
g., type of GPUs,internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[N/A]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a
[Table] [Table]
[Table]
A An illustration example for Assumption 3.
In the following, we show that Assumption 3.3 can be met in simple neural networks by using a two layer neural 406 network with Relu active function under a square loss function.
First we have that
[Equation]
we can calculate its partial derivatives with respect to θ and W
[Equation]
For illustration, we just consider the situation which only has one date, i.e., N = 1 .
We derive that
[Equation] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
i ≥
∥ 2 ∥ x ∥ 2 = 4 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙ 1 Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1)(cid:12) (cid:12) y − θ T σ ( Wx ) (cid:12) (cid:12) 2 .
As a result, ∥∇ θ g ( θ ) ∥ 2 /g ( θ ) = 4 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙
Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1) .
When 4 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙
Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1) <
δ 0 , we obtain ∥ θ T σ ( Wx ) ∥ 2 ≤ 1 / 2 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙ 1 Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1) ≤ 2 δ 0 .
Thisensures that g ( θ, W ) bounded.
B Useful lemmas
To prove the main results, we need the following lemmas, where Lemmas B.1–B.5 are from the literature and 414 Lemmas B.6–B.11 are first introduced in this paper with proofs given in the next section.
Lemma B.1.
(Lemma 1.2.3 in [33]) Suppose f ( θ ) ∈ C 1 ( x ∈ R N ) with gradient satisfying the following
Lipschitz condition ∥∇ f ( x ) − ∇ f ( y ) ∥
≤ c ∥ x − y ∥ , where c > 0 , then for any x, y ∈ R N , it holds that
[Equation]
f
[Equation]
y )
[Equation]
Lemma B.2.
(Lemma 10, [7]) Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with f ( x ) >
−∞ and its gradient satisfyingthe following Lipschitz condition ∥∇ f ( x ) − ∇ f ( y ) ∥
≤ c ∥ x − y ∥ , where c > 0 , then ∀ x 0 ∈ R N , there is
[Equation]
where f ∗ = inf x ∈ R N f ( x ) .
Lemma B.3.
(Lemma 4, [7]) Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with gradient satisfying the following Lipschitzcondition ∥∇ f
( x ) − ∇ f ( y ) ∥
≤ c ∥ x − y ∥ , where c > 0 , and the set S
= { x |∇ f ( x ) = 0 } is bounded and only 422 has finite connected components { S 1 , S 2 , . . .
, S m } .
Furthermore, assume there exists ϵ ′ 1 > 0 , such that for any 423 i = 1 , 2 , ..., m and x ∈ { x | 0 <
d ( x, S i ) <
ϵ ′ 1 } , it holds that (cid:12)(cid:12) f
( x ) − f i (cid:12)(cid:12) ̸ = 0 , where f i = f ( x ) for x ∈ S i .
Then for any i = 1 , 2 , ..., m , if there is ϵ ′ 0 > 0 satisfying d ( x, S i ) <
ϵ ′ 0 , it follows that
[Equation]
Lemma B.4.
[34] Suppose that { x n } ∈ R N is an L 2 martingale difference sequence, and ( x n , F n ) is an 426 adaptive process.
Then it holds that (cid:80) ∞ k
=0 x k < + ∞ a.s., if∞ (cid:88) n =
E ( ∥ x n ∥ 2 ) < + ∞ , or ∞ (cid:88) n =
[Equation]
Lemma B.5.
(Lemma 6, [7]) Suppose that { x n } ∈ R N is a non-negative sequence of random variables, then it 428 holds that (cid:80) ∞ n =0 x n < + ∞ a.s., if (cid:80) ∞ n =0 E (cid:0) x n (cid:1) <
+ ∞ .
Lemma B.6.
If 0 < µ < 1 and 0 < σ < 1 ( σ ̸ = µ ) are two constants, then for any positive sequence { ψ n } , 430 there is
[Equation]
where κ = max { µ, σ } and ω
= log κ min { µ, σ } .
Lemma B.7.
Let 0 < τ < 1 be a constant, then for any sequence { ϵ n } satisfying ϵ n → 0 , it holds that 433 (cid:80) + ∞ n =1 c ( ϵ ) n = 0 , where c ( ϵ ) n = (cid:80) ni
=1 τ n − i
ϵ i − (1 / (1 − τ ))
ϵ n .
Lemma B.8.
Consider the mSGD in (4) .
If Assumptions 1–3 hold, then for any θ 1 ∈ R N , v 0 ∈ R N andη 0 ≥ η 0 , there is a scalar T ( θ 1 , v 0 ) , such that E (cid:0) ϵ η 0 n g ( θ n ) (cid:1) <
T ( θ 1 , v 0 ) for any n ≥ 1 .
Lemma B.9.
Suppose { v n } is a sequence generated by mSGD in (4) .
Under Assumptions 3.1–3.2, it holdsthat (cid:80) + ∞ n =
E (cid:0) ϵ 2 η 0 n ∥ v n ∥ 2 (cid:1) <
c ( v 0 , θ 1 ) , and (cid:80) + ∞ n =
ϵ 2 η 0 n ∥ v n ∥ 2 < + ∞ a.s. , where c ( v 0 , θ 1 ) is a constant onlyrelated to v
and θ 1 .
Lemma B.10.
Suppose { θ n } is a sequence generated by mSGD in (4) .
Under Assumptions 3.1–3.2, it holds 440 that: for n ≥ 1 ,n
(cid:88) t
=
ϵ 1+2 η 0 t
[Equation]
, n (cid:88)
[Equation]
∞ , where B ( v 0 , θ 1 ) > 0 is a constant only related to v
and θ 1 .
Lemma B.11.
Suppose { θ n } is a sequence generated by mSGD in (4) .
Under Assumptions 3.1–3.2, it holds 443 that: for n ≥ 1 ,
[Equation] [Equation]
where ˆ k > 0 is a constant and { Q n } is a sequence, such that (cid:80) + ∞ n =1 Q n exists and is finite a.s..
B.1 Proof of Lemma B.6
The proof of this lemma needs some identical transformations.
We assume µ > σ (the case µ < σ is the same), 447 and let ω 0 = log µ σ > 1 .
Then we deriven
[Equation]
B.2 Proof of Lemma B.7
We calculate (cid:80) nt =1 c ( ϵ ) t as followsn (cid:88) t = c ( ϵ ) t = n (cid:88) t = (cid:18) t (cid:88) i =
τ t − i ϵ
i (cid:19) − n (cid:88) t =  1 − τ ϵ t = n (cid:88) i = (cid:18) n (cid:88) t
= i τ t − i (cid:19) ϵ
i − n (cid:88) t =  1 − τ ϵ t = n (cid:88) t
= (cid:18) n (cid:88) i =
t τ i − t (cid:19)
ϵ t − n (cid:88) t =  1 − τ ϵ t = n (cid:88) t = (cid:18)
n − t (cid:88) i =
τ i (cid:19)
ϵ t − n (cid:88) t =  1 − τ ϵ t = n (cid:88) t = (cid:18) 1 − τ n − t + 1 − τ (cid:19)
ϵ t − n (cid:88) t =  1 − τ ϵ t = − τ 1 − τ n (cid:88) t =
τ n − t ϵ t .
So we just need to prove lim n → + ∞ (cid:12)(cid:12) (cid:80) nt
=1 τ n − t ϵ t (cid:12)(cid:12) = 0 .
It holds that(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) t
= τ n − t ϵ t (cid:12)(cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32) n − [ √ n ] (cid:88) t = + n (cid:88) t =[ n −√ n ]+ (cid:33) τ n − t ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n − [ √ n ] (cid:88) t = τ n − t ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) t =[ n −√ n ]+ τ n − t ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .
Due to n − √ n → + ∞ , ∀ σ 0 > 0 , ∃ N 0 > 0 , when t > N 0 − [ √ N 0 ] , it holds that | ϵ t | < σ 0 .
So when 452 n > N 0 , it holds that(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) t
= n − [ √ n ]+ τ n − t ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) < n (cid:88) t
= n − [ √ n ]+ τ n − t σ 0 < σ 0 + ∞ (cid:88) t = τ t
= σ 0 1 − τ .
On the other hand(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n − [ √ n ] (cid:88) t =
τ n − t ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) < | ϵ 1 | n − [ √ n ] (cid:88) t
= τ n − t = | ϵ 1 | n − 1 (cid:88) t =[ √ n ] τ t → 0 , so ∀
σ 0 > 0 , ∃ N 1 > 1 , when n > N 1 , it holds that (cid:12)(cid:12) (cid:80) n − [ √ n ] t = τ n − t
[Table] [Table]
[Table]
Due to the arbitrary of σ 0 , we know
[Equation]
Thus, it holds that (cid:80) nt =1 c ( ϵ ) t = 0 .
B.3 Proof of Lemma B.8
Recall the mSGD algorithm in (4)
[Equation]
Equation (9) is equivalent to
[Equation]
Under Assumption 3.1 4), it follows from Lemma B.1 that
[Equation]
In this subsection, we just use the right side of (11).
The left side will be used in the next subsection.
Consider 463 ∇ θ t g ( θ t ) T v t in the following
[Equation]
Recursively applying the above equation yields
[Equation]
[Equation]
Substitute (12) into (11), then we have
[Equation]
From Lemma B.2, it follows that E ξ n (cid:0) ∥∇
[Equation]
M
[Equation]
∥∇
[Equation] [Equation]
g ( θ n ) − g ∗ (cid:1) +a ′ ≤ M ( g ( θ n ) + 1) ( M always exists ) .
Given ∀ η 0 > 0 , we denote Z ( t ) :
= (cid:81) + ∞ k
= t (1 + M 0 ϵ 2+ η 0 k ) , where
[Equation]
in which δ > 0 is a constant and M is defined above.
Here we define M 0 , Z ( t ) and δ to ease the proof.
From
Assumption 3.2, it holds that (cid:80) + ∞ t =1 ϵ 2+ η 0 t < + ∞ , which ensures (cid:80) + ∞ t =1 M 0 ϵ 2+ η 0 t < + ∞ .
From a general 470 inequality ln(1 + θ ) ≤ θ for θ > − 1 , we get
Z ( t ) ≤
[Equation]
(cid:89)
[Equation]
which means that Z ( t ) is uniformly upper bounded.
Then multiplying Z ( t + 1) on both sides of (69) and takingthe mathematical expectation yield
[Equation]
where the first and second inequalities are respectively due to Z
( i + 1) <
Z ( i ) and
[Equation]
Next, we aim to analyze c (cid:80) ti
[Equation]
α
[Equation]
i Z
[Equation]
+
[Equation]
ϵ
[Equation]
i ∥ 2 (cid:1) in (14).
It is proved in Appendix B.4 that
[Equation]
Taking a weighted sum of (15) yields
[Equation]
Derive that
[Equation] [Equation]
[Equation]
Substituting (17)–(19) into (14) yields
[Equation]
Construct a sequence { V n } as follows
[Equation]
By substituting (20) into (21) following the way of (17)–(19), we have
[Equation]
where f ( n − t ) is defined as follows,
[Equation]
Move the last term to the left-hand side of the above inequality, then we have
[Equation]
where
[Equation]
Because of α <
[Equation]
it holds that f ( n − t ) > 0 and e ( n − 1) t > 1 .
It follows from Assumption 3.2 that
[Equation]
24) where
[Equation]
cM
[Equation]
Calculate f ( n − t ) , then we obtainf ( n − t
[Equation]
=
[Equation]
=
[Equation]
(cid:0)
[Equation]
(cid:1)
[Equation]
> 0 .
It holds thatn (cid:88)
[Equation]
=
[Equation]
n
[Equation]
= n
[Equation]
−
[Equation]
− c
[Equation]
Substituting (24) and (25) into (22) yieldsn (cid:88)
[Equation]
=
[Equation]
According to the definition of Z ( t ) , we have
[Equation]
Let
[Equation]
and it follows from (26)
that
[Equation]
where (cid:101) k 0 ( θ 1 , v 0 ) .
Denote p = E (cid:26) (cid:80) ∞
k =1 M 0 ϵ 2+ η 0 k (cid:27) .
By taking the summation of (27), we obtain
[Equation]
Thus, we derive that
[Equation]
where T ( θ 1 , v 0 ) is a constant determined by v
and θ 1 .
B.4 Proof of (15)
We consider
[Equation]
η
[Equation]
where the first inequality is due to ϵ i ≤ ϵ i − 1 in Assumption 3.2, and the last equality is from (10).
Since ξ i and 498 θ
i are independent, taking the mathematical expectation of (28) and noting that
[Equation]
yield
[Equation]
Moreover, it holds that
[Equation]
Combining (29) and (30)
[Equation]
we get
[Equation]
Since ϵ i →
[Equation]
given any δ > 0 , there is an integer i 0 ≥ 0 , such that for i ≥ i 0 , 1 − cϵ i > α 1 − δ ( δ >
[Equation]
Sincei 0 is finite, without loss of generality, we assume i
= 0 for convenience, i.e., 1 − cϵ i > α 1 − δ ( δ > 0)
( i ≥ 1) .
Thus, we have
[Equation]
(32) Multiplying both sides of (32) by Z ( i + 1) , and noticing that Z ( i )
>
Z ( i + 1) , we have
[Equation]
Then (15) is obtained by recursively applying the above inequality.
B.5 Proof of Lemma B.
From (31), we have
[Equation]
It follows from Assumption 3.1 3) and Lemma B.8 that
[Equation]
<
[Equation]
Because of (cid:80) nt =1 ϵ 2+
[Equation]
< + ∞ , there is a scalar ¯ M > 0 such that for ∀ n
[Equation]
Then it follows from (33) that
[Equation]
where K is a positive scalar.
Since ϵ n → 0 when n is large enough, it holds that 15 (1 − α 2 ) < 1 − α 2 − cϵ n .
Without loss of generality, assume 15 (1 − α 2 ) < 1 − α 2 − cϵ 2 n for n ≥ 0 , so (cid:80) nt =1 E (cid:0) ϵ 2 η 0 t ∥ v t ∥ 2 (cid:1) <
10 K 1 − α 2 <+ ∞ .
By Lemma B.5, we obtain (cid:80) nt =1 ϵ 2 η 0 t ∥ v t ∥ 2 < + ∞ .
B.6 Proof of Lemma B.10
Through Taylor expansion, we derive that
[Equation]
) We first focus on the term ϵ
[Equation]
∇
[Equation]
g
[Equation]
v
[Equation]
It holds that
[Equation]
By substituting the above equation into (11) and noting − ( ∇ θ i g
( θ i ) −
[Equation]
−
[Equation]
we obtain
[Equation]
where c ′ is a constant which does not affect the result.
It follows that
[Equation]
where c ′′ is a constant which does not affect the result.
Take the mathematical expectation of (36), and notice
Assumption 3.1 1), then we have
[Equation]
where c ′′′ is a constant which does not affect the result.
From Lemma B.9, it follows that for some positiveconstant Q ,c ′′ n (cid:88) t =
ϵ 2 η
i
E (cid:0)
[Equation]
Q.
Since ( ϵ 1+2 η 0 t − ( ϵ
[Equation]
0 t − ϵ
[Equation]
0 t − 1 ) 2 ) can be controlled by k 0 ϵ
[Equation] [Equation]
t , it holds that
[Equation]
(37) where Q ′ is a constant.
From Lemma B.5, we have (cid:80) nt =1 ϵ 1+2 η 0 t ∥∇ θ t g ( θ t ) ∥ 2 < + ∞ a.s..
B.7 Proof of Lemma B.
It follows from (11) and (12) that
[Equation]
Then in light of (16), we get thatt
[Equation]
where σ 0 and σ 1 are two constants not affecting the analysis.
Then we make some transformations to obtain
[Equation]
where
[Equation]
From Assumption 3.1 3)
[Equation]
it follows that
[Equation]
Then we derive that
[Equation]
It holds that
[Equation]
It follows from some transformations that
[Equation]
where
[Equation]
θ i g
( θ i , ξ i )) .
Next we will prove that (cid:80) + ∞ t =1 Q t exists and is finite a.s. .
First through Lemma B.7, we get+ ∞ (cid:88)
[Equation]
= a ′ (cid:18) t (cid:88) i
= α (1+ δ )( t
[Equation]
i
[Equation] [Equation]
ϵ
[Equation]
t (cid:19) = 0 .
Obviously
[Equation]
∞ .
For the term σ 0 M 0 (cid:80) ti =
1 α (1+ δ )( t − i )
ϵ
i ∥∇ θ
i
g
( θ i ) ∥ 2 , from (B.10) and 0 < η 0 < 1 / 2 , we derive
[Equation]
For the term − σ 1 (cid:80) ti =
1 α (1+ δ )( t − i ) (cid:0)
ϵ
k +1 g
( θ k +1 ) − ϵ k g ( θ k ) (cid:1) , from (36), it follows that ϵ 2 η 0 n g ( θ n ) < ζ ′ <+ ∞ ( ∀ n > 0) .
Furthermore, it holds that ϵ n g ( θ n ) =
ϵ 1 − 2 η 0 n ϵ 2 η 0 n g ( θ n ) → 0 a.s. and+ ∞ (cid:88) t = − σt
[Equation]
α
[Equation]
For the term (cid:80)
[Equation]
=
[Equation]
g
[Equation]
θ i
[Equation]
∇
[Equation]
i g
[Equation]
θ i ) − ∇ θ i g ( θ i , ξ i )) , we first consider 543 (cid:80) + ∞ t =1 ϵ t ∇ θ t g ( θ t ) T ( ∇ θ t g ( θ t ) − ∇ θ t g ( θ t , ξ t )) .
From Lemma B.10, it follows that
[Equation]
≤
[Equation]
.s.
This means (cid:80) + ∞ t =1 ϵ t ∇ θ t g ( θ t ) T ( ∇ θ t g ( θ t ) − ∇ θ t g
θ t , ξ t )) exists and is finite a.s.. From Lemma B.7, it holdsthat (cid:80) ti =
[Equation]
i
[Equation]
∇
C Proofs of Theorems 3.1–3.4 and Proposition 3.2C.1 Proof of Theorem 3.1
We prove this theorem under two situations, namely, (cid:80) + ∞ n =1 ϵ 1+2 η 0 n < + ∞ and (cid:80) + ∞ n =1 ϵ 1+2 η 0 n = + ∞ .
If 550 (cid:80) + ∞ n =1 ϵ 1+2 η 0 n < + ∞ , we can conclude that (cid:80) + ∞ n =1 ϵ 2 n < + ∞ .
Then our condition degenerates to the classicalcondition (cid:80) + ∞ i =1 ϵ n = + ∞ , (cid:80) + ∞ i =1 ϵ 2 n < + ∞ .
The results then follow from Theorem 1 in [7].
Thus, thefollowing proof is all based on the condition that (cid:80) + ∞ n =1 ϵ 1+2 η 0 n = + ∞ .
First we will prove that there exists a 553 subsequence { θ k n } , such that ∥∇ θ kn g
( θ k n ) ∥ → 0 a.s. .
We prove it by contradiction.
We assume ∃ N ( ζ ) > 0 ,when θ n > N ( ζ ) , it holds that ∥∇ θ t g ( θ t ) ∥ 2 > δ 20 > 0 .
Then we have+ ∞ (cid:88) t = N ( ζ ) ϵ 1+2 η 0 t ∥∇ θ t g ( θ t ) ∥ 2 > δ+ ∞ (cid:88)
[Equation]
N ( ζ )
[Equation] [Equation]
(39) which contradicts with (37).
Thus, there exists a subsequence { θ k n } satisfying ∥∇ θ kn g
( θ k n ) ∥ → 0 a.s. .
It iseasy to find that J i is a bounded closed set.
So ∀ ϵ > 0 , we can construct an open cover H ( i )
ϵ
= { U ( θ, ϵ ) } ( θ ∈
J i ) of J i .
Through the Heine ˘ Borel theorem , we can get a finite open subcover { U ( θ k , ϵ )
( k = 0 , 1 , ..., n )from H ( i ) ϵ .
Let U ( i )
ϵ
= (cid:83)
nk =
U ( θ k , ϵ ) , then U ( i ) ϵ is a open set.
Under Assumption 3.1, J = { θ |∇ θ g ( θ ) }has only finite connected components J 1 , J 2 , ..., J m .
So inf i ̸
= j d ( J i , J j ) =
min i ̸ =
j d ( J i , J j ) .
Let δ
= 560 min i ̸
= j d ( J i , J j ) .
It follows from Lemma B.3 that ∃ ϵ 0 > 0 , such that when d ( θ, J i ) <
ϵ 0 , it holds that
[Equation]
where g i denotes g ( θ ) ( θ ∈ J i ) .
Let c =
min
{ ϵ 0 , δ 0 / 4 } and construct U (1) c , U (2) c , ..., U ( m ) c .
It is obvious that∀ U ( i ) c , U ( j ) c
( i ̸ = j ) , d ( U ( i ) c , U ( j ) ) >
δ 0 / 2 , and ∥∇ θ g ( θ ) ∥ 2 ≤ 2 c | g ( θ ) − g i | ( θ ∈ U ( i ) c ) .
Since J is a boundedset, ∃ N > 0 , such that J ⊂ K ( K is the closure of U (0 , N ) ).
Then we construct a set M = K/ (cid:83) mi
=1 U ( i ) c .
Since U ( i ) c is an open set and K is a closed set, we conclude M is a closed set.
Since ∥∇ θ g ( θ ) ∥ is a continuousfunction, ∃ θ 0 ∈ M , ∥∇ θ 0 g ( θ 0 ) ∥
=
min θ ∈ M ∥∇ θ g ( θ ) ∥ .
Let r = ∥∇ θ 0 g ( θ 0 ) ∥ > 0 .
Then we prove that ∀ u > 0 , θ ∈ K , ∃ δ > 0 , if ∥∇ θ g ( θ ) ∥ < δ , then d ( θ, J ) < u holds.
We prove itby contradiction.
Assume ∃ u 0 > 0 , ∀ δ 1 > 0 , ∃ θ δ 1 , such that ∥∇ θ δ 1 g
θ δ 1 ) ∥ < δ 1 and d ( θ, J ) ≥
u 0 .
We choose δ 1 = 1 , 1 / 2 , 1 / 3 ... , and construct a sequence { θ 1 /n } .
It is obvious that ∥∇
θ 1 /n g
( θ 1 /n )
→ 0 ∥ .
Since { θ 1 /n } is bounded, through the Accumulation point theorem , there exists a convergent subsequence{ θ 1 /k n } ⊂ { θ 1 /n } .
Let θ (0) =
lim n → + ∞ θ 1 /k n .
From the continuity of d ( θ, J ) and ∥∇ θ g ( θ ) ∥ , we getd ( θ (0) ) ≥ u 0 and ∥∇ θ (0) g
( θ (0) )
∥
= 0 .
It contradicts with the definition of J .
So ∀ u > 0 , θ ∈ K , ∃ δ > 0 ,∥∇ θ g ( θ ) ∥ < δ , such that d ( θ, J ) <
u .
Furthermore, due to the continuity of g ( θ ) , we can get ∀ ϵ 1 > 0 , 573 ∃ δ ′ > 0 , if d ( θ, J i ) <
δ ′ , it holds that | g ( θ ) − g i | < ϵ 1 .
Combine these two consequences, then we can prove∀ ϵ 1 > 0 , ∃ b > 0 , if θ ∈ U ( i ) c and ∥∇ θ g ( θ ) ∥ < b , it holds that (cid:12)(cid:12) g
( θ ) − g i (cid:12)(cid:12) <
ϵ 1 .
Through (39)
[Equation]
Next we aim to prove lim n → + ∞ ∥∇ θ n g ( θ n ) ∥ 2 = 0 .
It is equivalent to prove that {∥∇ n g ( θ n ) ∥ 2 } has no 576 positive accumulation points, that is to say, ∀ e 0 > 0 , there are only finite values of ∥∇ θ n g ( θ ) ∥ larger than e 0 .
Obviously, we just need to prove ∀ 0 <
e 0 < r , there are only finite values of ∥∇ θ n g ( θ ) ∥
larger than e .
We prove 578 this by contradiction.
We suppose ∃ 0 <
e < a , such that the set S =
{∥∇ θ n g ( θ n ) ∥ > e } is an infinite set.
Thenwe let ϵ 1 =
e/ 8 c
and o = min { b, e/ 4 } .
Due to (41), there exists a subsequence { θ p n } of { θ n } which satisfies 580 ∥∇ θ pn g ( θ p n ) ∥ < o .
We rank S as a subsequence {∥∇ m n g ( θ m n ) ∥ 2 } of {∥∇ n g ( θ n ) ∥ 2 } .
Then there is aninfinite subsequence {∥∇ m in g ( θ m in )
∥ 2 } of {∥∇ m n g ( θ m n ) ∥ 2 } such that ∀ n ∈ N + , ∃ l, n p n ∈ ( m i l , m i l +1 ) .
For convenience, we abbreviate { m i n } as { i n } .
We construct another infinite sequence { q n } as follows
[Equation] [Equation]
∇ θ n g
[Equation]
θ n )
[Equation]
≤ o } , q 2 n = min (cid:8) n : n > q 2 n − 1 , (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) >
e (cid:9) .
Now we prove that ∃ N 0 , when q 2 n > N 0 , it holds that e < (cid:13)(cid:13) ∇ θ q 2 n g ( θ q 2 n ) (cid:13)(cid:13) r left-hand side is obvious definition q 2 n
For
-hand side holds
2 n −
B.7, it follows that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
∥ 2 ≤ c ∥ θ n +1 − θ n ∥ → 0 a.s.,
So ∃ N 0 , when n > N 0 ,it holds that (cid:12)(cid:12) ∥∇ θ n +1 g ( θ n +1 )
∥ 2 − ∥∇ θ n g ( θ n ) ∥ (cid:12)(cid:12) <
r − e .
Then we can get that when q 2 n > N 0 + 1 , it holdsthat (cid:13)(cid:13) ∇ θ q 2 n g ( θ q n ) (cid:13)(cid:13) ≤
[Equation]
r .
This means that θ q 2 n ∈ (cid:83) mi =1 U ( i ) τ .
Then ∃ i 0 , such that θ q 2 n ∈ U ( i 0 )
τ .
Due to ∥∇ θ n g ( θ n ) ∥ ≤ e < r ( n ∈[ q 2 n − 1 , q 2 n )) , it holds that ∀ k ∈ [ q 2 n − 1 , q 2 n ) , ∃ i k , such that θ n ∈ U ( i k ) τ .
Since ∥ θ n +1 − θ n ∥ → 0 a.s. , it 592 holds that i
= i k ( ∀ j ∈ [ q 2 n − 1 , q 2 n )) .
For convenience, we let i
=
i q 2 n − 1 = ...
=
i q 2 n − 1 = i q 2 n .
Then 593 we conclude that
[Equation]
Due to the locally sign-preserving property, it follows that
[Equation]
Since it is the same to study the two cases
[Equation]
we just show how to prove the first case.
We derive that
[Equation]
From (38), we obtain
[Equation]
So it holds that
[Equation]
where
[Equation]
Due to ∥∇ θ q 2 n − 1 g ( θ q 2 n − 1 ) ∥ 2 < o < b , it follows that g ( θ q 2 n − 1 ) − g
i 2 n − 1 < e/ 8 c .
Substitute it into (43),then we get
[Equation]
From Lemma B.11, it follows that (cid:80) + ∞ n =1 Q n is finite almost surely.
Thus, we attain
[Equation]
by the Cauchy ′ s test for convergence .
From ϵ 1+2 η 0 q 2 n − 1 + i → 0 a.s. , it follows that
[Equation]
Thus, it holds that
[Equation]
On the other hand, it holds that ∥∇ θ q 2 n − 1+ i g ( θ q 2 n − 1 + i ) ∥ >
o ( i > 0) .
Together with (37), we get
[Equation]
∞
[Equation] [Equation]
It contradicts with (45), so we get that ∥∇ θ n g ( θ n ) ∥
→ 0 a.s. .
Under Assumption 3.1 1), it is safe to conclude 605 that there exists a connected component J ∗ of J such that lim n
[Equation]
d
[Equation]
θ n , J ∗ ) = 0 .
C.2 Proof of Theorem 3.
First we construct an eventˆ B n = { there exists at least one point n 0 ∈ [0 , n ] , satisfing ∥∇ θ n 0 g
[Equation]
θ
[Equation]
∥
[Equation]
≤ k
[Equation]
n 0 } , where k 0 is an undetermined coefficient Then we define another events
[Equation]
[Equation] [Equation]
and for all n 0 ∈ ( i, j ] ∥∇ θ n g ( θ n ) ∥ 2 > k 0 ϵ 2 n } .
Specially, we denote B n,n = B n Absolutely, there is
[Equation]
Let the characteristic function of B n be ˆ I n , B i,j be ˆ
I
i,j .
Through Theorem 3.1, we know θ n must converge to 610 some stationary point, so g ( θ n ) →
(cid:80) mk
=
I k g k a.s. and g k ( k = 1 , 2 , ..., m ) must be a local minimum.
We 611 denote ˆ g = g − (cid:80) mk
=
I k g k .
Then we get ˆ g → 0 a.s. and lim inf n →
+ ∞ sgn(ˆ g ( θ n )) > 0 a.s. .
There isˆ I i,n +1 ˆ g
[Equation]
n
[Equation]
−
[Equation]
i,
[Equation]
+
[Equation]
where µ > 0 is a constant.
We define ˆ S =
{ g ( θ ) < ˆ T } , ˆ I ( a ) i,
j = { some t ∈ ( i, j ) satisfies ∥∇ θ t g ( θ t ) ∥ 2 >q t ϵ 2 t and ∥∇ θ j g ( θ j ) ∥ 2 > a } .
We denote k 0 = 2 ac ′′ .
Then exists a > 0 , after adjusting { q t } , making
E (ˆ I
i,j ∥∇ θ j g ( θ j ) ∥ 2 ) ≥ E
(ˆ I ( a ) i,j ∥∇ θ j g ( θ j ) ∥ 2 ) and 1 / 2 E (ˆ I i,j )
≤ E (ˆ I ( a ) i,j ) .
Then through Assumption 3.1 (3),i.e., E ξ n (cid:0) (cid:13)(cid:13) ∇ θ g ( θ, ξ n ) (cid:13) (cid:13) 2 (cid:1) ≤ M ′ (cid:13) (cid:13) ∇ θ g ( θ ) (cid:13) (cid:13) 2 + a ′ .
We take the mathematical expectation on above equation toobtain
[Equation]
We construct a variable F i,n +1 as follows
[Equation]
So we can get
[Equation]
Through Assumption 3.3, we know when θ ∈ S
= {∥∇ θ g ( θ ) ∥ 2 / ( g ( θ ) − g ∗ ) <
δ 0 } , there isg ( θ ) <
ˆ T .
We denote k
= a ′ c ′′ /a .
Then when θ j ∈ ˆ S , there is E θ j ∈ ˆ S (cid:0) ˆ
I i,j ∥∇ θ i,j g ( θ i,j ) ∥ 2 (cid:1) =
E
[Equation]
>
[Equation] [Equation]
(cid:0) ˆ
[Equation]
θ
[Equation]
) (cid:1) , and when θ j ∈
R N /S , there is
E
[Equation]
ˆ
[Equation]
E
[Equation]
ˆ
[Equation] [Equation]
We assign ˆ c = min { 2 δ 0 , a/ 2 ˆ T } .
Then wecan get that
[Equation]
where K 0 is a constant.
Regarding the value of F i,i +1 , it holds that
[Equation] [Equation]
As a result,
[Equation]
Then we get
[Equation]
For the adverse events of ˆ B n ,
[Equation]
we use the same function, getting
[Equation]
Finally, we get that
[Equation]
C.3 Proof of Theorem 3.3
Proof.
First we have
[Equation]
Then we calculate that
[Equation]
First, we focus on the term ∇ θ n g ( θ n ) T v n , which satisfies
[Equation]
We multiple I ( a ) n on both sides of (48), leading to−
I
[Equation]
−
[Equation]
For convenience, we denote ˆ u n − 1 = (cid:0) I ( a ) n − 1 − I ( a ) n (cid:1) ∇ θ n − 1 g ( θ n − 1 ) T v n − 1 , and then we iterate (49),
[Equation]
[Equation]
We multiple I ( a ) n on both sides of (48) and note I ( a ) n ≥ I ( a ) n , then it holds that
[Equation]
Then we substitute (50) into (51) to obtain
[Equation]
Then we use the am − gm inequality on the term (cid:80)
[Equation]
to
[Equation] [Equation]
We substitute (53) into (52), note I ( a ) n − I ( a ) n +
=
o
( ϵ n ) a.e.
(due to (cid:80) + ∞ n =1 I ( a ) n − I ( a ) n +1 < + ∞ , (cid:80) + ∞ n =1 ϵ n =+ ∞ ), and make the mathematical expectation, then we derive
[Equation]
On the other hand, we have
[Equation]
) Substitute (55) into (54) yields
[Equation]
(56) where c ′′ and µ are two constants which do not affect results Then we note when ∥∇ θ n g ( θ n ) ∥ 2 > a , there is
[Equation]
where ˆ M is a constant which does not affect results.
So we getn
[Equation]
i =
[Equation] [Equation]
Substitute (57) into (56) and note c ′′
ˆ Mϵ 2 n = o ( ϵ n ) , then it follows that
[Equation]
We denote
[Equation]
For convenience, we let
[Equation]
Through (54), we get
[Equation]
so there is
[Equation]
where ˆ q 0 is a constant.
We focus on ˆ G ( a )
i ˆ F ( a ) i .
Using O ′ stolz theorem yields
[Equation]
Through Assumption 3.3, we know when θ ∈ S
= {∥∇ θ g ( θ ) ∥ 2 < δg ( θ ) } , there is g ( θ ) <
ˆ T .
We defineˆ S
= { g ( θ ) < 2 ˆ T } .
Then when θ i ∈ ˆ S , there is I ( a ) i ∥∇ θ i
g
( θ i )
∥ 2 > ( a/ 2 ˆ T )
I ( a ) i g ( θ i ) , and when θ i ∈ R N /S ,there is I ( a ) i ∥∇ θ i
g
( θ i )
∥ 2 >
δ
I ( a ) i g ( θ i ) .
We assign ˆ c = min { 2 δ 0 , a/ 2 ˆ T } , then it holds that
[Equation]
Thus, it follows that
[Equation] [Equation]
C.4 Proof of Proposition 3.1
We use the inverse inequality
[Equation]
to get the inverse inequation
I
[Equation]
where E ( ζ n ) = 0 and k ′′ is a constant which does not affect the result.
We denote
[Equation]
For convenience, we let
[Equation]
Through (63), we get
[Equation]
which leads to
[Equation]
Then we calculate
[Equation]
Then it holds that
[Equation]
[Equation]
As a result, we have
[Equation]
We focus on ˆ H ( a ) i
ˆ F ( a ) i .
Using O ′ stolz theorem and Lemma B.2 yields
[Equation]
Thus, by recursively using the above inequality and making the mathematical expectation, due to E ( ζ n ) = 0 , it follows that
[Equation]
Then through Theorem 3.3, we obtain that g ( θ n ) is convergent
a.s.. Thus, there exist ˆ k 0 > 0 , ˆ k 1 > 0 , such that
[Equation]
C.5 Proof of Theorem 3.4.
From (31) with ϵ n ≡ ϵ , it follows that
[Equation]
[Equation]
Let the event A ( φ ) n = [Equation] ∥∇ θ 1 g
[Equation]
1 ) ∥ 2 > φ, ∥∇ θ 2 g ( θ 2 ) ∥ 2 > φ, ..., ∥∇ θ n g ( θ n ) ∥ 2 > φ (cid:9) and the characteristicfunction be I ( φ ) n , then we attain
[Equation]
Using Taylor expansion, we derive that
[Equation]
Considering the term ∇ θ t g
( θ t ) T v t , it holds that
[Equation]
( φ ) t − 1 α ∇ θ t − 1 g ( θ t − 1 ) T v t − 1 + αI ( φ ) t − 1 ( ∇ θ t g
θ t ) − ∇ θ t − 1 g ( θ t − 1 )) T v t −+ ϵI ( φ ) t − 1 ∇ θ t g ( θ t ) T ∇ θ t g ( θ t , ξ t )
+ ( I ( φ ) t − I [Equation] φ ) t − 1 ) ∇ θ t g ( θ t ) T v t .
Note [Equation] −
[Equation]
and
[Equation]
a.s. . Then by substituting the above equationinto (11) and noting − ( ∇ [Equation] obtain
[Equation]
where c > 0 is a constant.
It follows that
[Equation]
where c ′′ is a constant which does not affect the result.
Substituting (67) into (70) yields
[Equation]
Then we multiply I ( φ ) n on the both sides of (71) and notice I ( φ ) n ≥ I ( φ ) n +1 to derive that
[Equation]
t
[Equation]
(72) Then we take the expectation on (71) and obtain
[Equation]
Under Assumption 3.1 2)
[Equation]
it holds that
[Equation]
It follows that
[Equation]
Let µ
= 1 /c ′′
ˆ σ 0 (cid:0) M 0 + aφ (cid:1) .
Then when ϵ < µ 0 , it holds that (cid:0) ϵ − c ′′
ˆ σ 0 ϵ 2 (cid:0) M 0 + aφ (cid:1)(cid:1) > 0 .
We derive thatn
[Equation]
In addition, we have
[Equation]
Due to A ( φ )⊃ A ( φ )⊃ ... ⊃ A ( φ ) n , we get
[Equation]
Through the arbitrariness of ϕ , we attain the conclusion.
C.6 Proof of Proposition
From (73), it follows that E (cid:0) I ( φ ) n ∥∇ θ n g ( θ n ) ∥ 2 (cid:1) → 0 .
So for any φ > 0 , there exists n 0 , when n > n 0 , itholds
that
[Equation]
Then we define another event First we construct an event
[Equation]
there exists at least one point n
[Equation]
where k 0 is an undetermined coefficient Then we define another events
[Equation] [Equation]
φ and for all n 0 ∈ ( i, j ] g ( θ n ) − g ∗ > φ } .
Specially, we denote B n,n = B n .
There isˆ B n = n (cid:91) i =
B i,n .
Let the characteristic function of B n be ˆ I n , B i,j be ˆ
I
i,j .
It is obvious that
[Equation]
Then from (71) we can get thatˆ
I i,n +
[Equation]
+
[Equation]
) − ˆ
[Equation]
i,n ˆ
[Equation]
76) where µ > 0 is a constant.
Then we denote ˆ g = g − φ .
We take the mathematical expectation on (76) to obtain
[Equation]
We construct a variable F i,n +1 as follows
[Equation]
i +
[Equation]
(cid:1)
[Equation]
So we can get
[Equation]
where σ 0 and 𭟋 0 are two constants not dependent on ϵ or φ .
We let
[Equation]
Then we get
[Equation]
Then we get Q i,n /F i,n > ϵ 𭟋 0 /φ .
So we can attain
[Equation]
where y
=
max { 1 − ϵφ 𭟋 0 , 1 / 2 − α } .
Then we first prove E ( g ( θ n )) is uniformly bounded.
We have
[Equation]
θ
[Equation]
=
[Equation]
where ˆ m is a constant.
Then we get
E
[Equation]
Now we have proved E (cid:0) g
( θ n ) (cid:1) uniformly bounded.
Then we aim to prove E (cid:0) g
( θ n ) (cid:1) can become arbitrarilysmall if n sufficient large and ϵ sufficient small.
First We have
[Equation]
≤
[Equation]
where ˆ p 0 is a constant.
Then we get that
[Equation]
Since E (cid:0) ˆ g ( θ i ) (cid:1) ≤ 0 , we get E (cid:0)
ˆ g
( θ i +1 ) (cid:1) ≤ ˆ p 1 ϵ/φ .
Finally we substituted it into (77), getting
[Equation]
where ˆ p 2 is a constant.
Then we can get
E ∥
[Equation]
n
[Equation]
≤
[Equation]
where ˆ p ′ 0 is a constant.
Then we get that
[Equation]
where ˆ p ′ 1 is a constant.
Then we can get
[Equation]
where ˆ p ′ 2 is a constant.
Now we can find that provided ϵ small enough, we can get for any small bound φ 0 , 706 E
[Equation]
∇ θ n g ( θ n ) (cid:13)(cid:13) 2 < φ 0 (when n is large enough).