Revisit last-iterate convergence of mSGD under milder requirement on step size
Abstract
Understanding convergence of stochastic gradient descent (SGD) based optimization algorithms can help deal with enormous machine learning problems.
To ensure last-iterate convergence of SGD and momentum-based SGD (mSGD), the existing studies usually constrain the step size ϵ n to decay as (cid:80) + ∞ n =1 ϵ 2 n < + ∞ , which however is rather conservative and may lead to slow convergence in the early stage of the iteration.
In this paper, we relax this requirement by studying an alternate step size for the mSGD.
First, we relax the requirement of the decay on step size to (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ (0 ≤ η 0 < 1 / 2) .
This implies that a larger step size, such as ϵ n = 1 √ n can be utilized for accelerating the mSGD in the early stage.
Under this new step size and some common conditions, we prove that the gradient norm of mSGD for a class of non-convex loss functions asymptotically decays to zero.
In addition, we show that this step size can indeed help make the iterates of mSGD converge into a neighborhood of the stationary points quicker in the early stage.
Finally, we establish the convergence of mSGD under a constant step size ϵ n ≡ ϵ >by removing a common requirement in the literature on strong convexity of the loss functions.
Some experiments are given to illustrate the developed results.
Introduction
The booming development of machine learning over the past decade relies on the employment of effective optimization algorithms for training parameterized machine learning models (e g., neural networks).
A large number of such optimization algorithms are based on gradient descent (GD).
The optimization problem in machine learning can be cast as minimizing a loss function g ( θ ) ∈ R over the choice of an N -dimensional real-valued parameter vector θ , i.e., by solving the problem
[Equation]
This problem can be solved with a typical GD algorithm through an iteration of the form
[Equation]
where θ n is the estimate of θ ∗ at step n , ϵ n is a positive step size (learning rate) to be designed, and ∇ θ n g ( θ n ) stands for the gradient of g ( θ n ) at step n .
Under certain technical conditions, θ n in (2) can asymptotically (as n → ∞ ) converge to the optimal solution θ ∗ .
However, (2) is not efficient in machine leaning applications with enormous training data.
To accelerate (2), one attempt is stochastic gradient descent (SGD) originating from [1].
Instead of calculating ∇ θ n g ( θ n ) over all the training data, SGD computes gradient estimate ∇ θ n g ( θ n , ξ n ) by sampling a subset of data (following the distribution of sampling noise { ξ n } and then updates the iterate as follows
[Equation]
In addition to sampling noise, ξ n can be used to model any external noises arising from the gradient computation.
However the price to pay for this data efficiency is that the employment of stochastic gradients (3) displays a relatively slow convergence rate during the learning process.
In order to improve the convergence rate of SGD, momentum-based stochastic gradient descend (mSGD), which reduces the update variance by averaging the past gradients ([2]), has been proposed.
A typical iteration form of mSGD is as follows ([3, 4]):
[Equation]
where α ∈ [0 , 1) and ϵ n > 0 are momentum coefficient and step size (learning rate), respectively.
An alternative formulation of mSGD, named stochastic heavy ball (SHB) [5, 6], has also been proposed as follows
[Equation]
where β n ∈ (0 , 1) and γ n are momentum coefficient and step size.
It has been shown that mSGD and SHB are essentially equivalent [7].
In recent years, mSGD has been widely employed in many applications of deep learning such as image classification [8], fault diagnosis [9], statistical image reconstruction [10], among others.
Moreover, a number of variants on momentum have been emerging, see, e.g., synthesized Nesterov variants [11], robust momentum [12], and PID-control based methods [13].
The importance of momentum in deep learning has been illustrated in [4] through experiments.
There has been a long line of literature analyzing the convergence of SGD and mSGD algorithms.
Regarding SGD, authors in [14] studied the last-iterate convergence when the step size is chosen to decay as 1 /n .
Authors in [15, 16] established the last-iterate convergence of SGD for strongly convex loss functions without the bounded gradients assumption.
For the normalized mSGD (SHB), Polyak [2, 17] and Kaniovski [18] established the convergence (subsequence convergence and convergence of time averages) properties for convex loss functions.
Igor Gitman [5] provided convergence results of mSGD (SHB) for non-convex loss functions under a somewhat restrictive requirement on uniform boundedness of a noise term E ( ∥∇ θ n g ( θ n , ξ n ) −∇ θ n g ( θ n ) ∥ ) ≤ δ .
It has been pointed out [4, 19] that the designs of momentum coefficients in [2, 5, 17, 18] may not be consistent with the requirements of some practical applications.
The last-iterate convergence of mSGD for non-convex loss functions has recently been established in [7, 20] with the step size condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ , which has been widely utilized in stochastic optimization [1, 15, 16, 21].
However, this condition requires the step size to decay relatively fast, so it may lead to slow convergence especially in the early stage of the iteration.
With the requirement that the loss function is strongly convex, [22] showed that the last-iterate of mSGD can converge to a neighbor of the stationary point when the step size is a constant.
Problem of interest and contributions
In this paper, we consider the problem of last-iterate convergence of mSGD under relaxed requirements on step size.
Specifically, we seek to relax the condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ [1, 7, 15, 16, 21] that has been required in the literature for proving last-iterate convergence of mSGD.
We consider two possible step sizes: step size that decay as ϵ n = 1 √ n and constant step size ϵ n ≡ ϵ >
Using such larger step size is expected to lead to the last iterate of mSGD (i.e., θ n ) converging faster.
We note that since the SGD (3) is a special case of mSGD (4), the developed results in this paper for mSGD also work for SGD.
The contributions of this paper are summarized as follows:
1) Under the setting of step size decaying as ϵ n = 1 √ n , we prove that the gradient norm of mSGD for a class of non-convex loss functions asymptotically decays to zero.
This result is more general than mean-square gradient convergence established in [20].
In fact, this result holds for any step sizes of the form (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ (0 ≤ η 0 < 1 / 2) , which is a superset of the widely required condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ [1, 7, 15, 16, 21].
Given this relaxed step size condition, one can run mSGD by employing a larger step size like ϵ n = 1 √ n for last-iterate convergence.
2) Under some mild conditions, we provide an estimate of the convergence rate of mSGD.
Furthermore, under a probability-based metric, we show that the new step size can help to improve the convergence speed in the early stage of the iteration. 3) Under the setting of constant step size, first we prove that given any small neighborhood of stationary points, it is feasible to design a step size, such that there is a convergent iterate subsequence of mSGD staying within the neighborhood almost surely.
In addition, we prove that the mean-square gradient can be arbitrary small by tunning the step size.
Comparing with [22], we remove the requirement of strong convexity on the loss functions.
Regarding the definitions on sequence convergence, the following ones from literature are typical.
For a stochastic variable sequence { ξ n } ∈ R N with 0 as the unique limit point, the sequence is said to satisfy last-iterate convergence if lim n → + ∞ ∥ ξ n ∥ = 0 , a.s. ; and time-average (mean-square convergence) if lim n → + ∞ 1 n (cid:80) nk =1 E ( ∥ ξ k
∥ 2 ) = 0 .
It can be proved that last-iterate convergence implies time-average convergence under the general setting of the considered problem.
Paper outline.
The rest of the paper is organized as follows.
In Section 3, we provide the main results of the paper on last-iterate convergence of mSGD under decaying step size and constant step size, respectively.
In Section 4, two simulation experiments are given.
Conclusion is made in Section 5.
The main proofs are given in Appendix.
Main results
In this section, we provide the main results of this paper on the last-iterate convergence for mSGD (4) under two possible step sizes: step size that decays as ϵ n = 1 √ n and constant step size ϵ n ≡ ϵ > 0 .
3.1 Convergence of mSGD under decaying step size
The following assumptions are needed in this paper.
Assumption 3.1.
Loss function g ( θ ) in (1) satisfies the following conditions:
Noise sequence { ξ n } are mutually independent and independent of θ 1 and v 0 , such that g ( θ ) =
E ξ n (cid:0) g ( θ, ξ n ) (cid:1) for any θ ∈ R N . 2.
g ( θ ) is a non-negative and continuously differentiable function.
The set of its stationary points J = { θ |∥∇ θ g ( θ ) ∥ = 0 } is a bounded set which has only finite connected components J 1 , ..., J n .
In addition, there is ˜ ϵ 1 > 0 , such that for any i and 0 <
d ( θ, J i ) <
˜ ϵ 1 , it holds that (cid:12)(cid:12) g
( θ ) − g i (cid:12)(cid:12) ̸ = 0 , where g i = { g ( θ ) | θ ∈ J i } is a constant.
There are two constants M ′ > 0 and a ′ > 0 such that for any θ ∈ R N and n ∈ N + ,
[Equation]
The gradient ∇ θ g ( θ ) satisfies the Lipschitz condition, i [Equation] e., there is a constant c > 0 , such that for any x, y ∈ R N ,
[Equation]
Conditions 1 and 4 in Assumption 3.1 are common in the literature [1, 15, 16, 21].
[7] required a condition on strong growth, i.e., E ξ n ∥∇ ξ n g ( θ, ξ n ) ∥ 2 ≤ M 0 ∥∇ θ g ( θ ) ∥ 2 , which however makes it close to the deterministic case.
In contrast, condition 3 allows more randomness of data sampling.
Condition 2 is a mild condition for the reasons as follows.
In some works, the non-negative condition may be replaced by a lower bound condition g ( θ ) >
ˆ l 0 > −∞ .
These two conditions are essentially equivalent, since one can construct a new loss function g = g − ˆ l 0 under the lower bound condition, such that the new loss function is non-negative.
The rest of condition 2 is quite general, since it allows the loss function to have multiple stationary points and to be non-convex.
Assumption 3.2.
In the mSGD (4) , momentum coefficient α ∈ [0 , 1) and the sequence of step size ϵ n is positive and monotonically decreasing to zero, such that (cid:80) + ∞ n =1 ϵ n = + ∞ and (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ , where 0 ≤ η 0 < 1 / 2 is a constant.
In Assumption 3.2, the momentum coefficient α ∈ [0 , 1) is a constant.
Comparing with the setting in [5, 6, 17, 18], where α ( n ) tends to 1 or 0 , constant momentum coefficient is more common in practice [4, 7, 19].
Regarding the step size condition, it is more general than the one in many existing works on last-iterate convergence of SGD and mSGD [1, 7, 15, 13, 16, 21], i.e., (cid:80) + ∞ n =1 ϵ 2 n < + ∞ , which is obtained from Assumption 3.2 if η 0 = 0 .
Under the step size condition in Assumption 3.2, one can choose larger step size like ϵ n = 1 √ n , which however is not feasible in the commonly required condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ .
Since the new step size can decay more slowly than the existing one, it provides more space for step size fine tuning, such that the algorithm can quickly converge to a neighborhood of the stationary point [23, 24, 22].
Then we attain the first main result in this paper on last-iterate convergence of mSGD under the step size condition in Assumption 3.2.
Theorem 3.1.
Consider the mSGD in (4) with any v 0 ∈ R N and θ 1 ∈ R N .
Under Assumptions 3.1 and 3.2, the gradient norm tends to 0 almost surely, i.e.
[Equation]
Due to page constraint, the complete proof is given Appendix.
Here, we provide a proof outline to present the main proof ideas.
Proof Sketch of Theorem 3.1 : We aim to prove that ∇ θ n g ( θ n ) → 0 a.s. via the following key steps.
Step 1:
In this step we aim to prove E (cid:0)
ϵ η 0 n g ( θ n ) (cid:1) is uniformly upper bounded.
We present this result as Lemma B.8 in Appendix.
Step 2: We prove there exists a subsequence of ∇ θ n g ( θ n ) which is convergent to {+
+}
a.s.. It is attained by proving (cid:80) + ∞ n =1 ϵ 1+2 η 0 n ∥∇ θ n g ( θ n ) ∥ 2 < + ∞ a.s. .
We present this result as Lemma B. in Appendix.
Step 3: We aim to extend the subsequence convergence in Step 2 to asymptotic convergence.
The basic idea is to prove that the adjacent terms g ( θ n ) and g ( θ n +1 ) are “close" enough, such that
[Equation]
where ˆ k > 0 is a constant and { Q n } is a sequence, such that (cid:80) + ∞ n =1 Q n converges a.s.. This result is presented in Lemma B.11.
Step 4: Finally, we use some techniques to attain the result of asymptotic convergence[-∇ θ n g ( θ n ) →
-]
[Equation]
In the last-iterate convergence analysis of mSGD, there are several technical challenges when we try to extend the condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ to (cid:80) + ∞ n =1 ϵ 2+ δ 0 n < + ∞ with 0 ≤ η 0 < 1 / 2 .
One of the challenges is: When we try to prove (cid:80) + ∞ n =1 ζ n = (cid:80) + ∞ n =1 ϵ n ∇ θ n g ( θ n ) T ( ∇ θ n g ( θ n , ξ n ) − ∇ θ n g ( θ n )) is convergent a.s., which is the vital step to get the last-iterate convergence, we usually use the Martingale convergence theorem , through which we just need to prove (cid:80) + ∞ n =1 E ( ζ 2 n ) is convergent.
If the condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ holds, we can get (cid:80) + ∞ n =1 E
( ζ 2 n ) < (cid:80) + ∞ n =1 ˆ Kϵ 2 n < + ∞
( ˆ K is a constant) directly.
However, under the new condition (cid:80) + ∞ n [Equation] 1 ϵ 2+ δ 0 n , we can not use this approach.
Instead, we introduce a more technical approach to handle this problem.
Comparing with the corresponding result in [7] (Theorem 1 in [7]), we relax the condition on the decay of step size.
Under this new condition, one can utilize step size with more slow decaying speed in the iteration of mSGD.
Let α = 0 in mSGD, then we can get a similar result for SGD
Corollary 3.1.
Consider the SGD in (3) with any θ 1 ∈ R N .
Under Assumptions 3.1 and 3.2 with α = 0 , the gradient norm tends to 0 almost surely, i.e.
[Equation]
In the following, we aim to estimate the last-iterate convergence rate of ∇ θ n g ( θ n ) .
Instead, the convergence rate of E (cid:0) ∥∇ θ g ( θ n ) ∥ 2 (cid:1) is quite meaningful.
In general, if we need to estimate the convergence rate of the last iterate, we need a probability − retention assumption , i.e., ∀ k 0 > 0 , ∃ a > 0 , making P ( ∥ θ n ∥ > a ) ∼ P ( ∥ θ 1 ∥ > k 0 ϵ 21 , ..., ∥ θ n ∥ > k 0 ϵ 2 n ) , and we usually need some extra assumptions to establish a quantitative relationship between g and ∇ g .
Existing works were established usually under the strongly-convex assumption [25, 26, 13], local P-L condition [7], or convex loss function satisfying some stability conditions, e.g., E ∥∇ θ n g ( θ n ) ∥ 2 < δ < + ∞ .
However, these assumptions are rather relatively strong, especially the stability condition which is very difficult to verify.
In our paper, we need a milder assumption than the above conditions as follow.
Assumption 3.3.
Regarding the loss function g ( θ ) defined in (1) , there exists δ 0 > 0 , T 0 > 0 , such that for any θ ∈ { θ |∥∇ θ g ( θ ) ∥ 2 / ( g ( θ ) − g ∗ ) <
δ 0 } , it holds that g ( θ ) < T 0 , where g ∗ = inf θ ∈ R N g
( θ ) .
This assumption requires the value of the loss function g ( θ ) to be bounded in the set S
= { θ |∥∇ θ g ( θ )
∥ 2 / ( g ( θ ) − g ∗ ) <
δ 0 } .
The motivation of this assumption is: For a stochastic algorithm, there is always a probability at which the iterates are far away from the true value.
Since second-derivative information has a more positive effect on convergence (such as strongly convex loss functions), this condition restricts iterates which are far away due to extreme noise, so that it does not have a significant effect on global updates.
This requirement is milder than the strong convexity condition, which is commonly used in existing works [25, 26, 13] for the analysis of the last-iterate convergence.
In addition, this assumption allows the loss function to be non-convex, and does not need the local P-L condition used in [7] on the second derivative information near stationary points.
Note that Assumption 3.3 can be satisfied by many loss functions commonly encountered in machine learning.
First of all, many common loss functions are bounded [27–30], and thus satisfy Assumption 3.3.
For unbounded loss functions, there are a number of instances satisfying Assumption 3.3.
For linear regression functions, since they are strongly convex, it is easy to verify the assumption.
In the Appendix, we explain (not in a strict but an illustrating manner) this assumption can also be satisfied by a two layer neural network with Relu active function under a square loss function.
In the following theorem, we provide an estimate of the convergence rate for the mSGD in (4).
Theorem 3.2.
Consider the mSGD in (4) with a unique stationary point θ ∗ . Then for any v 0 ∈ R N under Assumptions 3.1–3.3 and probability-retention assumption.
Then exists a > 0 , for any ∥∇ θ 1 g ( θ 1 ) ∥ 2 , there is
[Equation]
(7) where ˆ c = min { a/ 2 T 0 , 2 δ 0 } .
Comparing with [7], we remove the strong growth condition, i.e., E ξ n ∥∇ θ g ( θ, ξ n ) ∥ 2 ≤ M ∥∇ θ g ( θ ) ∥ 2 , and relax the step size condition to be (cid:80) + ∞ n =1 ϵ 2+ δ 0 n (0 < δ 0 ≤ 1 / 2) .
In the comparison with [20], we do not require the convexity of loss function or E ∥∇ [Equation]
In addition, we relax the step size condition in [20].
There are quite a few results on the average iterates in the literature, such as [13, 31, 32].
In [13, 31, 32], the convergence rate O ( 1 √ n ) of mSGD is established with the step size 1 √ n .
According to our result, let ϵ n = 1 √ n , it holds that
[Equation]
It can be proved that the convergence rate of average iterates is also O (cid:16) 1 √ n (cid:17) .
From (7), we see that larger step size may not lead to larger convergence rate.
This is reasonable, since the convergence rate reflects how fast the iterate θ n converges when n → + ∞ and a larger step size usually makes the iterate converges to a neighborhood of stationary point quicker.
We have the following theorem on this point.
Theorem 3.3.
Consider the mSGD in (4) with any v 0 ∈ R N .
Under Assumptions 3.1–3.3, given any ˆ a > 0 , if ∥∇ θ 1 g ( θ 1 ) ∥ 2 > ˆ a , then it holds that
[Equation]
where τ (ˆ a ) = min n> 0 {∥∇ θ n g ( θ n ) ∥ 2 < ˆ a } and ˆ c = min { ˆ a/ 2 T 0 , 2 δ 0 } .
Theorem 3.3 provides a probability description of how fast the iterates of mSGD converge into the preset neighborhood of stationary points.
It can be illustrated as follows.
It follows from Theorem 3. that
[Equation]
where ˆ k 1 > 0 is a constant.
From the inequality, as n increases, the event ∥∇ θ n g ( θ n ) ∥ 2 < a occurs with a higher and higher probability tending to probability 1.
The influence of momentum coefficient α and step size ϵ n is also quantified.
We note that with the measure in Theorem 3.3, the convergence to a neighborhood of the stationary point is faster than under the traditional condition (cid:80) + ∞ n =1 ϵ 2 n < + ∞ by setting the step size with a relatively slow decay speed as (cid:80) + ∞ n =1 ϵ 2+ δ 0 n (0 < δ 0 ≤ 1 / 2) , such as ϵ n = 1 √ n and η 0 = 1 / 4 .
Similar results (the large step size can make the iteration quickly converge to a neighborhood of stationary point, but may converge slowly in the late stage of the algorithm) appear in [23, 24, 22] through experiments.
3.2 Convergence of mSGD under constant step size
In the previous subsection, we establish the last-iterate convergence of mSGD under a relaxed decay condition on step size.
In this subsection, we study the convergence of mSGD under a constant step size ϵ n ≡ ϵ > 0 .
We start with the following result on subsequence convergence.
Theorem 3.4.
Consider the mSGD in (4) with any v 0 ∈ R N and θ 1 ∈ R N .
Under Assumption 3.1, for any φ > 0 , there exists µ ( φ )> 0 , such that for any 0 < ϵ n ≡ ϵ < µ ( φ ) 0 , it holds that
[Equation]
where { θ ( φ ) k n } is a subsequence { θ n } .
Proof.
Due to page constraint, the complete proof is given Appendix.
Here, we provide a proof outline to present the main proof ideas.
To study the mSGD with constant step size, we notice that the impact of the noise cannot be totally eliminated through the step size.
But when ∥∇ θ n g ( θ n ) ∥ is relatively large (meaning the distance between θ n and the stationary point θ ∗ is relatively large), it holds that E ξ n (cid:0) ∥∇ θ n g ( θ n , ξ n ) ∥ 2 (cid:1) ≤ M ′ ∥∇ θ n g ( θ n ) ∥ 2 + a ′ ≤ ( M ′ + ˆ k ) ∥∇ θ n g ( θ n ) ∥ 2 .
This implies that with a proper step size ϵ , the noise can be controlled.
As a result, the algorithm can be stabilized around a stationary point.
In the following, several key steps in the proof are provided.
Step 1: We define an event by A ( φ ) n = (cid:8) ∥∇ θ 1 g ( θ 1 ) ∥ 2 > φ, ∥∇ θ 2 g ( θ 2 ) ∥ 2 > φ, ..., ∥∇ θ n g ( θ n ) ∥ 2 > φ (cid:9) and a characteristic function of this event by I ( φ ) n .
Step 2: We prove that the probability of A ( φ ) n is tending to 0 as n → ∞ .
First, we attain a recursion formula:
[Equation]
∥ 2 (cid:1) .
By taking the sum of multiple inequalities as above, we have
[Equation]
This leads to
[Equation]
From the Chebyshev inequality , we attain P ( A ( φ ) n ) → 0 .
[Figure]
Step 3:
Step 2 implies that with probability 1 there exists at least one instant n at which ∥∇ θ n g ( θ n ) ∥ 2 ≤ φ .
We note that for the time instant m at which ∥∇ θ m g ( θ m ) ∥ 2 > φ , we can use a new start point.
By repeating the process of Step 2, we attain the conclusion of this theorem.
Theorem 3.4 shows that under the common conditions in the literature, the constant step size can guarantee the subsequence convergence of mSGD (4) in any small neighborhood of stationary point.
In addition, we have a similar result on E (cid:0) ∥∇ θ n g ( θ n ) ∥ 2 (cid:1) as follows.
Proposition 3.1.
Suppose { θ n } is a sequence generated by (4) with any θ 1 ∈ R N and v 0 ∈ R N .
Under Assumptions 3.1 and 3.3, if (1) has a unique solution θ ∗ and inf d ( θ,θ ∗ ) > δ 0 ∥∇ θ g ( θ ) ∥ 2 > 0 ( ∀ δ 0 > 0) , then for any φ 0 > 0 , there exist scalars µ ( φ 0 )> 0 and n 0 > 0 , such that for any ϵ n ≡ ϵ < µ ( φ 0 )and n > n 0 , it holds that
[Equation]
This result is a substantial extension of the existing results on SGD with constant step size to mSGD.
Comparing with the results in [22], we remove the requirement of µ -strong convexity on loss function.
Experiments
Since this paper focuses on convergence analysis on the well-known mSGD, two relatively simple experiments on regression and classification tasks are given respectively to show the effectiveness of the results.
4.1 Regression task
In this subsection, we study a house price prediction problem with neural networks trained by mSGD.
Network Architectures .
We respectively employ a 3-layer and a 4-layer fully-connected neural network with ReLu active function and squared loss function.
Implementation .
We implement two neural networks to train on this dataset using Keras.
The first neural network consists of three fully connected layers with 13 , 8 , and 1 neurons.
The second neural network consists of four fully connected layers with 13 , 32 , 16 , 1 neurons, respectively.
We initialize the weights use glorot uniform algorithm.
We use mSGD with a mini-batch size of 16, and use mean square error loss function to train the model.
The momentum term coefficient is 0 .
9 and the model is trained for up to 1000 epochs. The training takes about two hours at a time in 3080GPU.
We do not use dropout.
[Figure]
Dataset .
The training dataset we use is from Boston House Price Dataset 2 .
This dataset is a regression prediction dataset, which consists of 506 sets of data with dimension 13.
During the training process, we randomly divide 30% as the test set and normalize the data before training.
Results .
We use the mSGD in (4) under three different step sizes respectively: ϵ n = 1 n , ϵ n = 1 n 0 . 7 , and ϵ n = 1 n 0 .
The experiment results are given in Figure 1.
The figure shows that 1) the loss decays to zero under the three step size settings; 2) ϵ n = 1 n 0 .
5 can make the loss tend to a small neighborhood of zero fastest among the three step size settings.
This conforms to the theoretical analysis in Theorems 3.1–3.3.
4.2 Classification task
In this subsection, we study an image classification problem with neural networks trained by mSGD.
Network Architectures .
We employ a 20-layer ResNet network.
The convolutional layers mostly have 3×3 filters and follow two simple design rules: (i) for the same output feature map size, the layers have the same number of filters; (ii) if the feature map size is halved, the number of filters is doubled so as to preserve the time complexity per layer.
We perform downsampling directly by convolutional layers that have a stride of 2 and padding with valid.
The network ends with an average pooling layer and a 10-way fully-connected layer with softmax.
Implementation .
We implement the ResNet20 network using Keras.
We initialize the weights use glorot uniform algorithm.
We use mSGD with a mini-batch size of 64, momentum coefficient 0 .and use categorical crossentropy loss function to train the model.
The models are trained for up to 1000 epochs, taking about two hours at a time in 3080GPU.
We do not use dropout.
Dataset .
We use two different datasets CIFAR-10 and CIFAR-100 3 .
CIFAR-10 consists of 50k training images and 10k testing images in 10 classes.
CIFAR-10 is a color image dataset close to ubiquitous objects, and the image size is 32x32x3.
CIFAR-100 consists of 50k training images and 10k testing images in 100 classes.
Each category contains 500 training images and 100 testing images.
CIFAR-100 is a color image dataset closer to ubiquitous objects, and the image size is 32x32x3.
Normalize the dataset between 0-1 before training.
Results .
We use the mSGD in (4) under three different step sizes: ϵ n = 1 n , ϵ n = 1 n 0 . 7 , and ϵ n = 1 n 0 .
The experiment results are given in Figure 2 and Figure 3.
The figures show that 1) the loss decays to zero and the accuracy goes up to 1 under the three step size settings; 2) ϵ n = 1 n 0 .
5 can make the loss tend to a small neighborhood of zero and make the accuracy tend to a small neighborhood of 1 fastest among the three step size settings, followed by the setting of ϵ n = 1 n 0 .
These results conform to the theoretical analysis in Theorems 3.1–3.3.
L o ss
[Figure]
/n^
[Figure]
[Figure]
A cc u r ac y
Epochs
[Figure]
Conclusion
In this paper, we studied the last-iterate convergence of momentum-based stochastic gradient descent (mSGD) under relaxed conditions on step sizes.
First, under the relaxed condition (cid:80) + ∞ n =1 ϵ 2+ η 0 n < + ∞ (0 ≤ η 0 < 1 / 2) , we proved the last-iterate convergence of mSGD for a class of non-convex loss functions.
In addition, we showed that this step size can indeed help to improve the convergence speed in the early stage of the algorithm by quantifying the influence of the step size and momentum coefficient.
This implies that a larger step size, such as ϵ n = 1 √ n can be utilized with guaranteed convergence.
We also proved that the algorithm with a constant step size (i.e., ϵ n ≡ ϵ > 0 ) can ensure the last-iterate convergence of mSGD without requiring the strong convexity assumption on loss functions.
Acknowledgments and Disclosure of Funding
Xingkang He was supported in part by ARO Grant W911NF1910483 and AFOSR Grant FA9550-211-0231.
Vijay Gupta were supported in part by NSF Grants 2020246, 2225978, and 2208794.
We thank the anonymous reviewers for helpful comments and suggestions.
Appendix A An illustration example for Assumption 3.
In the following, we show that Assumption 3.3 can be met in simple neural networks by using a two layer neural network with Relu active function under a square loss function.
First we have that
[Equation]
then we can calculate its partial derivatives with respect to θ and W
[Equation]
For illustration, we just consider the situation which only has one date,
i
[Equation]
e., N = 1 .
We derive that (cid:13) (cid:13)(cid:13) (cid:13)
[Equation]
2 ∥ θ ⊙ 1 Wx i ≥
∥ 2 ∥ x ∥ 2 = 4 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙ 1 Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1)(cid:12)(cid:12) y − θ T σ ( Wx ) (cid:12)(cid:12) 2 .
As a result, ∥∇ θ g ( θ ) ∥ 2 /g ( θ ) = 4 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙
Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1) .
When 4 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ [-⊙
-]{+⊙ 
+}
Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1) <
δ 0 , we obtain ∥ θ T σ ( Wx ) ∥ 2 ≤ 1 / 2 (cid:0) ∥ σ ( Wx ) ∥ 2 + ∥ θ ⊙ 1 Wx i ≥
∥ 2 ∥ x ∥ 2 (cid:1) ≤ 2 δ 0 .
This ensures that g ( θ, W ) bounded.
B Useful lemmas
To prove the main results, we need the following lemmas, where Lemmas B.1–B.5 are from the literature and Lemmas B.6–B.11 are first introduced in this paper with proofs given in the next section.
Lemma B.1.
(Lemma 1.2.3, [33]) Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with gradient satisfying the following
Lipschitz condition ∥∇ f ( x ) − ∇ f ( y ) ∥
≤ c ∥ x − y ∥ , where c > 0 , then for any x, y ∈ R N , it holds that
[Equation]
Lemma B.2.
(Lemma 10, [7]) Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with f ( x ) >
−∞ and its gradient satisfying the following Lipschitz condition ∥∇ f ( x ) − ∇ f ( y ) ∥
≤ c ∥ x − y ∥ , where c > 0 , then ∀ x 0 ∈ R N , there is
[Equation]
where f ∗ = inf x ∈ R N f ( x ) .
Lemma B.3.
(Lemma 4, [7]) Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with gradient satisfying the following Lipschitz condition ∥∇ f ( x ) − ∇ f ( y ) ∥
≤ c ∥ x − y ∥ , where c > 0 , and the set S
= { x |∇ f ( x ) = 0 } is bounded and only has finite connected components { S 1 , S 2 , . . .
, S m } .
Furthermore, assume there exists ϵ ′ 1 > 0 , such that for any i = 1 , 2 , ..., m and x ∈ { x | 0 <
d ( x, S i ) <
ϵ ′ 1 } , it holds that (cid:12) (cid:12) f
( x ) − f i (cid:12)(cid:12) ̸ = 0 , where f i = f ( x ) for x ∈ S i .
Then for any i = 1 , 2 , ..., m , if there is ϵ ′ 0 > 0 satisfying d ( x, S i ) <
ϵ ′ 0 , it follows that
[Equation]
Lemma B.4.
[34] Suppose that { x n } ∈ R N is an L 2 martingale difference sequence, and ( x n , F n ) is an adaptive process.
Then it holds that (cid:80) ∞ k
=0 x k < + ∞ a.s., if ∞ (cid:88) n =
E ( ∥ x n ∥ 2 ) < + ∞ , or ∞ (cid:88) n =
E (cid:0) ∥ x n ∥ 2 (cid:12)(cid:12)
[Equation]
Lemma B.5.
(Lemma 6, [7]) Suppose that { x n } ∈ R N is a non-negative sequence of random variables, then it holds that (cid:80) ∞ n =0 x n < + ∞ a.s., if (cid:80) ∞ n =0 E (cid:0) x n (cid:1) <
+ ∞ .
Lemma B.6.
If 0 < µ < 1 and 0 < σ < 1 ( σ ̸ = µ ) are two constants, then for any positive sequence { ψ n } , there is
[Equation]
where 1 < ˆ R < 1 / (cid:0) 1 − κ ( ω 0 − 1) (cid:1) , and κ = max { µ, σ } and ω
= log κ min { µ, σ } .
Lemma B.7.
Let 0 < τ < 1 be a constant, then for any sequence { ˆ ϵ n } satisfying ˆ ϵ n → 0 , it holds that (cid:80) + ∞ n =1 c (ˆ ϵ ) n = 0 , where c (ˆ ϵ ) n = (cid:80) ni
=1 τ n − i ˆ
ϵ i − (1 / (1 − τ ))
ˆ ϵ n .
Lemma B.8.
Consider the mSGD in (4) .
If Assumptions 3.1–3.2 hold, then for any θ 1 ∈ R N , v 0 ∈ R N , there is a scalar T ( θ 1 , v 0 ) , such that E (cid:0) ϵ η 0 n g ( θ n ) (cid:1) <
T ( θ 1 , v 0 ) for any n ≥ 1 .
Lemma B.9.
Suppose { v n } is a sequence generated by mSGD in (4) .
Under Assumptions 3.1–3.2, it holds that (cid:80) + ∞ n [-=
-]=0 E (cid:0) ϵ 2 η 0 n ∥ v n ∥ 2 (cid:1) <
c ( v 0 , θ 1 ) , and (cid:80) + ∞ n =
ϵ 2 η 0 n ∥ v n ∥ 2 < + ∞ a.s. , where c ( v 0 , θ 1 ) is a constant only related to v
and θ 1 .
Lemma B.10.
Suppose { θ n } is a sequence generated by mSGD in (4) .
Under Assumptions 3.1–3.2, it holds that: for n ≥ 1 , n (cid:88) t
=
ϵ 1+2 η 0 t
[Equation]
where B ( v 0 , θ 1 ) > 0 is a constant only related to v
and θ 1 .
Lemma B.11.
Suppose { θ n } is a sequence generated by mSGD in (4) .
Under Assumptions 3.1–3.2, it holds that: for n ≥ 1 ,
[Equation]
where ˆ k > 0 is a constant and { Q n } is a sequence, such that (cid:80) + ∞ n =1 Q n exists and is finite a.s..
B.1 Proof of Lemma B.
The proof of this lemma needs some identical transformations.
We assume µ > σ (the case µ < σ is the same), and let ω 0 = log µ σ > 1 .
Then we derive n (cid:88) [Equation] =
[Equation]
{+where 1 < ˆ R < 1 / (cid:0) 1 − κ ( ω 0 − 1) (cid:1) , and κ = max { µ, σ } and ω 
+}
= log κ min { µ, σ } .
B.2 Proof of Lemma B.
We calculate (cid:80) nt =1 c (ˆ ϵ ) t as follows n (cid:88) [Equation] i =
τ t − i ˆ
ϵ i (cid:19) − n (cid:88) t =  1 − τ ˆ ϵ t = n (cid:88) i = (cid:18) n (cid:88) t
= i τ t − i (cid:19)
ˆ ϵ i − n (cid:88) t =  1 − τ ˆ ϵ t = n (cid:88) t
= (cid:18) n (cid:88) i =
t τ i − t (cid:19)
ˆ ϵ t − n (cid:88) t =  1 − τ ˆ ϵ t = n (cid:88) t = (cid:18)
n − t (cid:88) i =
τ i (cid:19)
ˆ ϵ t − n (cid:88) t =  1 − τ ˆ ϵ t = n (cid:88) t = (cid:18) 1 − τ n − t + 1 − τ (cid:19)
ˆ ϵ t − n (cid:88) t =  1 − τ ˆ ϵ t = − τ 1 − τ n (cid:88) t =
τ n − t ˆ ϵ t .
So we just need to prove lim n → + ∞ (cid:12)(cid:12) (cid:80) nt
=1 τ n − t ˆ ϵ t (cid:12)(cid:12) = 0 .
It holds that (cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) t = τ n − t ˆ ϵ t (cid:12)(cid:12)(cid:12)(cid:12) =
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32) n − [ √ n ] (cid:88) t = + n (cid:88) t =[ n −√ n ]+ (cid:33) τ n − t ˆ ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≤ (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n − [ √ n ] (cid:88) t = τ n − t ˆ ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) t =[ n −√ n ]+ τ n − t ˆ ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) .
Due to n − √ n → + ∞ , ∀ σ 0 > 0 , ∃ N 0 > 0 , when t > N 0 − [ √ N 0 ] , it holds that | ˆ ϵ t | < σ 0 .
So when n > N 0 , it holds that (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) n (cid:88) t = n − [ √ n ]+ τ n − t ˆ ϵ t (cid:12)(cid:12)(cid:12)(cid:12) (cid:12) < n (cid:88) t
= n − [ √ n ]+ τ n − t σ 0 < σ 0 + ∞ (cid:88) t = τ t
= σ 0 1 − τ .
On the other hand (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n − [ √ n ] (cid:88) t = τ n − t
ˆ ϵ t (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) < ˜ kn − [ √ n ] (cid:88) t = τ n − t =
˜ k 0 n − 1 (cid:88) t =[ √ n ] τ t → 0 , where ˜ k 0 > 0 is a constant. As a result, ∀ σ 0 > 0 , ∃ N 1 > 1 , when n > N 1 , it holds that (cid:12)(cid:12) (cid:80) n − [ √ n ] t = τ n − t ˆ ϵ t (cid:12)(cid:12) < σ 0 .
Then when n > max { N 0 , N 1 } , it holds that (cid:12)(cid:12)(cid:12)(cid:12) n (cid:88) [Equation] = [Equation]
τ [Equation] t [Equation]
[ √ n ] (cid:88)
[Equation]
= [Equation] n [Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) n (cid:88)
[Equation]
=[ n −√ n
[Equation]
[Equation] n [Equation] (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) <
[Equation]
σ 0 .
Due to the arbitrariness of σ 0 , we know n (cid:88) t = τ n − t
ˆ ϵ t → 0 .
Thus, it holds that (cid:80) nt =1 c (ˆ ϵ ) t = 0 .
B.3 Proof of Lemma B.
Recall the mSGD algorithm in (4)
[Equation]
[Equation]
Equation (9) is equivalent to
[Equation]
Under Assumption 3.1 4), it follows from Lemma B.1 that
[Equation]
In this subsection [Equation] we just use the right side of (11).
The left side will be used in the next subsection.
Consider ∇ θ t g ( θ t ) T v t in the following
[Equation]
Recursively applying the above equation yields
[Equation]
Substitute (12) into (11), then we have
[Equation]
From Lemma B.3, it follows that E ξ n (cid:0) ∥∇
[Equation]
[Equation]
n
[Equation]
2 (cid:1) ≤ M ′ ∥∇ θ n g ( θ n ) ∥ 2 + a ′ ≤ 2 cM ′ (cid:0) g
( θ n ) − g ∗ (cid:1) + a ′ ≤ M ( g ( θ n ) + 1) ( M always exists ) .
Given ∀ η 0 > 0 , we denote Z ( t ) :
= (cid:81) + ∞ k
= t (1 + M 0 ϵ 2+ η 0 k ) , where
[Equation]
in which δ > 0 is a constant and M is defined above.
Here we define M 0 , Z ( t ) and δ to ease the proof.
From
Assumption 3.2, it holds that (cid:80) + ∞ t =1 ϵ 2+ η 0 t < + ∞ , which ensures (cid:80) + ∞ t =1 M 0 ϵ 2+ η 0 t < + ∞ .
From a general inequality ln(1 + θ ) ≤ θ for θ > − 1 , we get
Z ( t ) ≤ + ∞ (cid:89) k
[Equation]
[Equation]
which means that Z ( t ) is uniformly upper bounded.
Then multiplying Z ( t + 1) on both sides of (13), taking the mathematical expectation and noting ϵ t ≥ ϵ t +1 yield [Equation]
[Equation]
where the inequality is due to
ϵ t +1 < ϵ t and Z ( i + 1) <
Z ( i ) .
Next, we aim to analyze c (cid:80) ti =1 α
[Equation]
Z
[Equation]
+
[Equation]
E (cid:0)
[Equation]
∥ v i ∥ 2 (cid:1) in (14).
It is proved in Appendix B.4 that
[Equation]
(15) Taking a weighted sum of (15) yields t
[Equation]
Derive that
[Equation]
Substituting (17)–(19) into (14) yields
[Equation]
Construct a sequence { V n } as follows
[Equation]
By substituting (20) into (21) following the way of (17)–(19), we have
[Equation]
where f ( n − t ) is defined as follows,
[Equation]
Move the last term to the left-hand side of the above inequality, then we have n
[Equation]
where
[Equation]
[Equation]
Because of α <
[Equation]
it holds that f ( n − t ) > 0 and e ( n − 1) t > 1 .
It follows from Assumption 3.2 that
[Equation]
≤
[Equation]
where
[Equation]
Calculate f ( n − t ) , then we obtain
[Equation]
It holds that
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table]
[Equation]
Substituting (24) and (25) into (22) yields
[Equation]
According to the definition of Z ( t )
[Equation]
we have Z [Equation] ) =
[Equation]
Let
F n := n (cid:88) t
= (cid:18)2 −
[Equation]
(cid:19)
n − t Z
[Equation]
+
[Equation]
+
[Equation]
θ
[Equation]
+
[Equation]
, and it follows from (26)
that
[Equation]
27) where (cid:101) k 0 ( θ 1 , v 0 ) .
Denote p = E (cid:26) (cid:80) ∞
k =1 M 0 ϵ 2+ η 0 k (cid:27) .
By taking the summation of (27), we obtain
[Equation]
Thus, we derive that
[Equation]
where T ( θ 1 , v 0 ) is a constant determined by v
and θ 1 .
B.4 Proof of (15)
We consider
[Equation]
where the first inequality is due to ϵ i ≤ ϵ i − 1 in Assumption 3.
[Equation]
and the last equality is from (10).
Since ξ i and θ i are independent, taking the mathematical expectation of (28) and noting that
[Equation]
yield
[Equation]
Moreover, it holds that
[Equation]
[Equation]
Combining (29) and (30)
[Equation]
we get
[Equation]
[Equation]
Since ϵ i → 0 , given any δ > 0 , there is an integer i 0 ≥ 0 , such that for i ≥ i 0 , 1 − cϵ i > α 1 − δ ( δ > 0) .
Since i 0 is finite, without loss of generality, we assume i
= 0 for convenience, i.e., 1 − cϵ i > α 1 − δ ( δ > 0)
( i ≥ 1) .
Thus, we have
[Equation]
(32) Multiplying both sides of (32) by Z ( i + 1) , and noticing that Z ( i )
>
Z ( i + 1) , we have
[Equation]
[Table]
[Equation]
Then (15) is obtained by recursively applying the above inequality
[Table]
B.5 Proof of Lemma B.
From (31), we have
[Equation]
It follows from Assumption 3.1 3) and Lemma B.8 that
[Equation]
Because of (cid:80)
[Equation]
< + ∞ , there is a scalar ¯ M > 0 such that for ∀ n
[Equation]
[Equation]
< ¯ M < + ∞ .
Then it follows from (33) that
[Equation]
where K is a positive scalar.
Since ϵ n → 0 when n is large enough, it holds that 15 (1 − α 2 ) < 1 − α 2 − cϵ n .
Without loss of generality, assume 15 (1 − α 2 ) < 1 − α 2 − cϵ 2 n for n ≥ 0 , so (cid:80) nt = [Equation] E (cid:0) ϵ 2 η 0 t ∥ v t ∥ 2 (cid:1) <
10 K 1 − α 2 < + ∞ .
By Lemma B.5, we obtain (cid:80) nt =1 ϵ 2 η 0 t ∥ v t ∥ 2 < + ∞ .
B.6 Proof of Lemma B.
Through Taylor expansion, we derive that
[Equation]
(34) We first focus on the term ϵ 2 η 0 t ∇ θ t g ( θ t ) T v t .
It holds that
[Equation]
By substituting the above equation into (11) and noting − ( ∇
[Equation]
i
[Equation]
where c ′ is a constant which does not affect the result.
It follows that
[Equation]
where c ′′ is a constant which does not affect the result.
Take the mathematical expectation of (36) [Equation] and notice
Assumption 3.1 1) [Equation] then we have [Equation]
[Equation]
E ( ∥
[Equation]
i ∥ 2 ) , where c ′′′ is a constant which does not affect the result.
From Lemma B.9, it follows that for some positive constant Q [-,c ′′ n (cid:88) t = ϵ 2 η 
 i E (cid:0)-],
[Equation]
Since ( ϵ 1+2 η 0 t − ( ϵ 2 η 0 t − ϵ
[Equation]
0 t [Equation]
[Equation]
2 ) can be controlled by k 0 ϵ 1+2 η 0 t , it holds that
[Equation]
) where Q ′ is a constant.
From Lemma B.5, we have (cid:80) nt =1 ϵ 1+2 η 0 t [Equation]
B.7 Proof of Lemma B.
It follows from (11) and (12) that
[Equation]
i
[Equation]
i )
[Equation]
Then in light of (16), we get that
[Equation]
where σ 0 and σ 1 are two constants not affecting the analysis.
Then we make some transformations to obtain σ
[Equation]
where
[Equation]
From Assumption 3.1 3)
[Equation]
it follows that
[Equation]
Then we derive that
[Equation]
It holds that
[Equation]
+
[Equation]
i =
[Equation]
θ i , ξ i )) .
It follows from some transformations that
[Equation]
) where
[Equation]
Next we will prove that (cid:80) + ∞ t =1 Q t exists and is finite a.s. .
First through Lemma B.7, we get
[Equation]
Obviously
[Equation]
For the term σ 0 M 0 (cid:80) ti =
1 α (1+ δ )( t − i )
ϵ
i ∥∇ θ
i
g
( θ i ) ∥ 2 , from (B.10) and 0 < η 0 < 1 / 2 , we derive
[Equation]
For the term − σ 1 (cid:80) ti =1 α
[Equation]
1+ δ )( t − i ) (cid:0)
ϵ
k +1 g
( θ k +1 ) − ϵ k g ( θ k ) (cid:1) , from (36), it follows that ϵ 2 η 0 n g ( θ n ) < ζ ′ < + ∞ ( ∀ n > 0) .
Furthermore, it holds that ϵ n g ( θ n ) =
ϵ 1 − 2 η 0 n ϵ 2 η 0 n g ( θ n ) → 0 a.s. and + ∞ (cid:88) t = − σ
[Equation]
t
[Equation]
i = α
[Equation]
For the term
[Equation]
, we first consider (cid:80) + ∞ t =1 ϵ
[Equation]
∇ θ t g ( θ t ) T ( ∇ θ t g ( θ t ) − ∇ θ t g ( θ t , ξ t )) .
From Lemma B.10, it follows that
[Equation]
E
[Equation]
≤ a [Equation] [Equation]
This means (cid:80) + ∞ t =1 ϵ t ∇ θ t g ( θ t ) T ( ∇ θ t g ( θ t ) − ∇ θ t g
θ t , ξ t )) exists and is finite a.s.. From Lemma B.7, it holds that (cid:80) ti = 2 α t
[Equation]
i
[Equation]
g [Equation] T [Equation]
) exists and is finite a.s..
Thus, (cid:80) + ∞ t =1 Q t exists and is finite a.s..
C Proofs of Theorems 3.1–3.4 and Proposition 3. C.1 Proof of Theorem 3.
We prove this theorem under two situations, namely, (cid:80) + ∞ n =1 ϵ 1+2 η 0 n < + ∞ and (cid:80) + ∞ n =1 ϵ 1+2 η 0 n = + ∞ .
If (cid:80) + ∞ n =1 ϵ 1+2 η 0 n < + ∞ , we can conclude that (cid:80) + ∞ n =1 ϵ 2 n < + ∞ .
Then our condition degenerates to the classical condition (cid:80) + ∞ i =1 ϵ n = + ∞ , (cid:80) + ∞ i =1 ϵ 2 n < + ∞ .
The results then follow from Theorem 1 in [7].
Thus, the following proof is all based on the condition that (cid:80) + ∞ n =1 ϵ 1+2 η 0 n = + ∞ .
First we will prove that there exists a subsequence { θ k n } , such that ∥∇ θ kn g
( θ k n ) ∥ → 0 a.s. .
We prove it by contradiction.
We assume ∃ N ( ζ ) > 0 , when θ n > N ( ζ ) , it holds that ∥∇ θ t g ( θ t ) ∥ 2 > δ 20 > 0 .
Then we have
[Equation]
which contradicts with (37).
Thus, there exists a subsequence { θ k n } satisfying ∥∇ θ kn g
( θ k n ) ∥ → 0 a.s. .
It is easy to find that J i is a bounded closed set.
So ∀ ϵ > 0 , we can construct an open cover H ( i )
ϵ
= { U ( θ, ϵ ) } ( θ ∈
J i ) of J i .
Through the Heine ˘ Borel theorem , we can get a finite open subcover { U ( θ k , ϵ )
( k = 0 , 1 , ..., n ) from H ( i ) ϵ .
Let U ( i )
ϵ
= (cid:83)
nk =
U ( θ k , ϵ ) , then U ( i ) ϵ is a open set.
Under Assumption 3.1, J = { θ |∇ θ g ( θ ) } has only finite connected components J 1 , J 2 , ..., J m .
So inf i ̸
= j d ( J i , J j ) =
min i ̸ =
j d ( J i , J j ) .
Let δ
= min i ̸ =
j d ( J i , J j ) .
It follows from Lemma B.3 that ∃ ϵ 0 > 0 , such that when d ( θ, J i ) <
ϵ 0 , it holds that
[Equation]
where g i denotes g ( θ ) ( θ ∈ J i ) .
Let c =
min
{ ϵ 0 , δ 0 / 4 } and construct U (1) c , U (2) c , ..., U ( m ) c .
It is obvious that ∀ U ( i ) c , U ( j ) c
( i ̸ = j ) , d ( U ( i ) c , U ( j ) ) >
δ 0 / 2 , and ∥∇ θ g ( θ ) ∥ 2 ≤ 2 c | g ( θ ) − g i | ( θ ∈ U ( i ) c ) .
Since J is a bounded set, ∃ N > 0 , such that J ⊂ K ( K is the closure of U (0 , N ) ).
Then we construct a set M = K/ (cid:83) mi
=1 U ( i ) c .
Since U ( i ) c is an open set and K is a closed set, we conclude M is a closed set.
Since ∥∇ θ g ( θ ) ∥ is a continuous function, ∃ θ 0 ∈ M , ∥∇ θ 0 g ( θ 0 ) ∥
=
min θ ∈ M ∥∇ θ g ( θ ) ∥ .
Let r = ∥∇ θ 0 g ( θ 0 ) ∥ > 0 .
Then we prove that ∀ u > 0 , θ ∈ K , ∃ δ > 0 , if ∥∇ θ g ( θ ) ∥ < δ , then d ( θ, J ) < u holds.
We prove it by contradiction.
Assume ∃ u 0 > 0 , ∀ δ 1 > 0 , ∃ θ δ 1 , such that ∥∇ θ δ 1 g
θ δ 1 ) ∥ < δ 1 and d ( θ, J ) ≥
u 0 .
We choose δ 1 = 1 , 1 / 2 , 1 / 3 ... , and construct a sequence { θ 1 /n } .
It is obvious that ∥∇
θ 1 /n g
( θ 1 /n )
→ 0 ∥ .
Since { θ 1 /n } is bounded, through the Accumulation point theorem , there exists a convergent subsequence { θ 1 /k n } ⊂ { θ 1 /n } .
Let θ (0) =
lim n → + ∞ θ 1 /k n .
From the continuity of d ( θ, J ) and ∥∇ θ g ( θ ) ∥ , we get d ( θ (0) ) ≥ u 0 and ∥∇ θ (0) g
( θ (0) )
∥
= 0 .
It contradicts with the definition of J .
So ∀ u > 0 , θ ∈ K , ∃ δ > 0 , ∥∇ θ g ( θ ) ∥ < δ , such that d ( θ, J ) <
u .
Furthermore, due to the continuity of g ( θ ) , we can get ∀ ϵ 1 > 0 , ∃ δ ′ > 0 , if d ( θ, J i ) <
δ ′ , it holds that | g ( θ ) − g i | < ϵ 1 .
Combine these two consequences, then we can prove ∀ ϵ 1 > 0 , ∃ b > 0 , if θ ∈ U ( i ) c and ∥∇ θ g ( θ ) ∥ < b , it holds that (cid:12)(cid:12)
g ( θ ) − g i (cid:12)(cid:12) <
ϵ 1 .
Through (39)
[Equation]
Next we aim to prove lim n → + ∞ ∥∇ θ n g ( θ n ) ∥ 2 = 0 .
It is equivalent to prove that {∥∇ n g ( θ n ) ∥ 2 } has no positive accumulation points, that is to say, ∀ e 0 > 0 , there are only finite values of ∥∇ θ n g ( θ ) ∥ larger than e 0 .
Obviously, we just need to prove ∀ 0 <
e 0 < r , there are only finite values of ∥∇ θ n g ( θ ) ∥
larger than e .
We prove this by contradiction.
We suppose ∃ 0 <
e < a , such that the set S =
{∥∇ θ n g ( θ n ) ∥ > e } is an infinite set.
Then we let ϵ 1 =
e/ 8 c
and o = min { b, e/ 4 } .
Due to (41), there exists a subsequence { θ p n } of { θ n } which satisfies ∥∇ θ pn g ( θ p n ) ∥ < o .
We rank S as a subsequence {∥∇ m n g ( θ m n ) ∥ 2 } of {∥∇ n g ( θ n ) ∥ 2 } .
Then there is an infinite subsequence {∥∇ m in g ( θ m in ) ∥ 2 } of {∥∇ m n g ( θ m n ) ∥ 2 } such that ∀ n ∈ N + , ∃ l, n p n ∈ ( m i l , m i l +1 ) .
For convenience, we abbreviate { m i n } as { i n } .
We construct another infinite sequence { q n } as follows
[Equation] [Equation]
q
[Equation]
max
[Equation]
(cid:13)(cid:13) [Equation] ∇
[Equation]
Now we prove that ∃ N 0 , when q 2 n > N 0 , it holds that e < (cid:13)(cid:13) ∇ θ q 2 n g ( θ q 2 n ) (cid:13)(cid:13) <
r .
The left-hand side is obvious (the definition of q 2 n ). For the right-hand side, it holds that (cid:13)(cid:13) ∇ θ q 2 n − 1 g
( θ q 2 n − 1 ) (cid:13)(cid:13) ≤ e .
From (36) and Lemma B.7, it follows that
[Equation]
a
[Equation]
+
[Equation]
n
[Equation]
a
[Equation]
s.. From Assumption 3.1 2), it holds that (cid:12)(cid:12) ∥∇ θ n +1 g ( θ n +1 ) ∥ 2 − ∥∇ θ n g ( θ n ) ∥ 2 (cid:12)(cid:12) ≤ (cid:12)(cid:12) ∥∇ θ n +1 g ( θ n +1 ) ∥ − ∥∇ θ n g ( θ n ) ∥ (cid:12)(cid:12) 2 ≤ ∥∇ θ n +1 g ( θ n +1 ) − ∇ θ n g ( θ n ) ∥ 2 ≤ c ∥ θ n +1 − θ n ∥ → 0 a.s.,
So ∃ N 0 , when n > N 0 , it holds that (cid:12)(cid:12) ∥∇ θ n +1 g ( θ n +1 )
∥ 2 − ∥∇ θ n g ( θ n ) ∥ (cid:12)(cid:12) <
r − e .
Then we can get that when q 2 n > N 0 + 1 , it holds that (cid:13)(cid:13) ∇ [Table] g ( θ
[Table]
n ) (cid:13)(cid:13) ≤ [Table] − 1 g ( θ q 2 n − 1 ) ∥ (cid:12)(cid:12) ≤ e + r − e = r .
This means that θ q 2 n ∈ (cid:83) mi =1 U ( i ) τ .
Then ∃ i 0 , such that θ q 2 n ∈ U ( i 0 )
τ .
Due to ∥∇ θ n g ( θ n ) ∥ ≤ e < r ( n ∈ [ q 2 n − 1 , q 2 n )) , it holds that ∀ k ∈ [ q 2 n − 1 , q 2 n ) , ∃ i k , such that θ n ∈ U ( i k ) τ .
Since ∥ θ n +1 − θ n ∥ → 0 a.s. , it holds that i
= i k ( ∀ j ∈ [ q 2 n − 1 , q 2 n )) .
For convenience, we let i
=
i q 2 n − 1 = ...
=
i q 2 n − 1 = i q 2 n .
Then we conclude that
[Equation]
Due to the locally sign-preserving property, it follows that
[Equation]
Since it is the same to study the two cases, we just show how to prove the first case.
We derive that e
[Equation]
From (38), we obtain
[Equation]
So it holds that
[Equation]
where
[Equation]
Due to ∥∇ θ q 2 n − 1 g ( θ q 2 n − 1 ) ∥ 2 < o < b , it follows that g ( θ q 2 n − 1 ) − g
i 2 n − 1 < e/ 8 c .
Substitute it into (43), then we get
[Equation]
From Lemma B.11, it follows that (cid:80) + ∞ n =1 Q n is finite almost surely.
Thus, we attain
[Equation]
→
[Equation]
.s.
by the Cauchy ′ s test for convergence .
From ϵ 1+2 η 0 q 2 n − 1 + i → 0 a.s. , it follows that
[Equation]
Thus, it holds that
[Equation]
On the other hand, it holds that ∥∇ θ q 2 n − 1+ i g ( θ q 2 n − 1 + i ) ∥ >
o ( i > 0) .
Together with (37), we get
[Equation]
<
[Equation] [Equation]
It contradicts with (45), so we get that ∥∇ θ n g ( θ n ) ∥
→ 0 a.s. .
Under Assumption 3.1 1), it is safe to conclude that there exists a connected component J ∗ of J such that lim
[Equation]
C.2 Proof of Theorem 3.
First we construct an event ˆ
[Equation]
{ there exists at least one point
[Equation]
where k 0 is an undetermined coefficient Then we define another events
[Equation]
[Equation]
Specially, we denote B n,n = B n Absolutely, there is
[Equation]
n =
[Equation]
(cid:91) i
[Equation]
B
[Equation]
Let the characteristic function of B n be ˆ I n , B i,j be ˆ
I
i,j .
Through Theorem 3.1, we know θ n must converge to some stationary point, so g ( θ n ) →
(cid:80) mk
=
I k g k a.s. and g k ( k = 1 , 2 , ..., m ) must be a local minimum.
We denote ˆ g = g − (cid:80) mk
=
I k g k .
Then we get ˆ g → 0 a.s. and lim inf n →
+ ∞ sgn(ˆ g ( θ n )) > 0 a.s. .
There is ˆ I i,n [Equation] g
[Equation]
n
[Equation]
1 ) − ˆ I [Equation] [Equation]
g
[Equation]
[Equation] [Table] [Equation] [Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
where µ > 0 is a constant.
We define ˆ S =
{ g ( θ ) < ˆ T } , ˆ I ( a ) i,
j = { some t ∈ ( i, j ) satisfies ∥∇ θ t g ( θ t ) ∥ 2 > q t ϵ 2 t and ∥∇ θ j g ( θ j ) ∥ 2 > a } .
We denote k 0 = 2 ac ′′ .
Then exists a > 0 , after adjusting { q t } , making
E (ˆ I
i,j ∥∇ θ j g ( θ j ) ∥ 2 ) ≥ E
(ˆ I ( a ) i,j ∥∇ θ j g ( θ j ) ∥ 2 ) and 1 / 2 E (ˆ I i,j )
≤ E (ˆ I ( a ) i,j ) .
Then through Assumption 3.1 (3), i.e., E [Equation] n (cid:0)(cid:13)(cid:13) ∇ [Equation]
g ( θ, ξ n )
[Equation] ≤ [Equation] ∇ θ g
[Equation]
θ )
[Equation]
2 + a ′ .
We take the mathematical expectation on above equation to obtain
[Equation]
We construct a variable F i,n +1 as follows
[Equation]
So we can get
F i,n +1 − F i,n ≤ σ
[Equation]
α
[Equation]
n − i E (cid:0) ˆ
{+I i,i +
+}
∥∇ θ
i +1 g
( θ i +1 )
∥ 2 (cid:1) − (1 − α )n (cid:88)
j = i + (cid:16) 1 − a ′ c ′′ 2 k 0 a
(cid:17)(cid:18)2 − α (cid:19) n − j ϵ j E (cid:0)
ˆ
I i,j ∥ g ( θ j ) ∥ 2 (cid:1) .
Through Assumption 3.3, we know when θ ∈ S
= {∥∇ θ g ( θ ) ∥ 2 / ( g ( θ ) − g ∗ ) <
δ 0 } , there is g ( θ ) < ˆ T .
We denote k
= a ′ c ′′ /a .
Then when θ j ∈ ˆ S , there is E θ j ∈ ˆ S (cid:0) ˆ
I i,j ∥∇ θ i,j g ( θ i,j ) ∥ 2 (cid:1) =
E θ j ∈ ˆ S (cid:0) ˆ
I ( a ) i,j ∥∇ θ i,j g ( θ i,j ) ∥ 2 (cid:1) > ( a/ 2 ˆ T ) E θ j ∈ ˆ S (cid:0)
ˆ
I i,j g ( θ j ) (cid:1) , and when θ j ∈
R N /S , there is
E θ j ∈ R N /S (cid:0) ˆ
{+I i,j ∥∇ θ j g ( θ j ) ∥ 2 (cid:1) > 2 δ 
+}
E θ j ∈ R N /S (cid:0) ˆ
I i,j g ( θ j ) (cid:1) .
We assign ˆ c = min { 2 δ 0 , a/ 2 ˆ T } .
Then we can get that
[Equation]
where K 0 is a constant.
Regarding the value of F i,i +1 , it holds that
[Equation]
≤ E (ˆ I i (cid:12)(cid:12)
ˆ g
( θ i +1 ) (cid:12)(cid:12) ) ,
ˆ I i
| ˆ g ( θ i +1 ) − ˆ g
( θ i ) |
=
ˆ I i |∇
i g ( θ i ) T v i |
{+= ˆ I i |∇ i g ( θ i ) T ( αv i − 
 + ϵ i ∇ i+}
g ( θ i , ξ i ))
| ≤ O ( ϵ 2 i ) .
As a result,
[Equation]
Then we get n
[Equation]
For the adverse events of ˆ B n , Ω / ˆ B n = { there is no point n 0 ∈ [0 , n ] , satisfying ∥∇
[Equation]
0 } , we use the same function, getting
[Equation]
Finally, we get that
[Equation]
C.3 Proof of Theorem 3.
Proof.
First we have
[Equation]
Then we calculate that
[Equation]
First, we focus on the term ∇ θ n g ( θ n ) T v n , which satisfies
[Equation]
We multiple I (ˆ a ) n on both sides of (48), leading to −
[Equation]
[Equation]
For convenience, we denote ˆ u n − 1 = (cid:0) I (ˆ a ) n − 1 − I (ˆ a ) n (cid:1) ∇ θ n − 1 g ( θ n − 1 ) T v n − 1 , and then we iterate (49), getting
[Equation]
I
[Equation]
We multiple I (ˆ a ) n on both sides of (48) and note I (ˆ a ) n ≥ I (ˆ a ) n , then it holds that
[Equation]
Then we substitute (50) into (51) to obtain I
[Equation]
−
[Equation]
Then we use the am − gm inequality on the term (cid:80) n − 1 t =1 α n − t c t − 1 to attain n − 1 (cid:88) t =
[Equation]
We substitute (53) into (52), note I (ˆ a ) n − I (ˆ a ) n +
=
o
( ϵ n ) a.e.
(due to (cid:80) + ∞ n =1 I (ˆ a ) n − I (ˆ a ) n +1 < + ∞ , (cid:80) + ∞ n =1 ϵ n = + ∞ ), and make the mathematical expectation, then we derive E [Equation] − n (cid:88) t =
( ϵ t
[Equation]
( ϵ t ))
α n − t E ( I (ˆ a ) t ∥∇ θ t g ( θ t ) ∥ 2 ) .
On the other hand, we have
[Equation]
) Substitute (55) into (54) yields
[Equation]
where c ′′ and µ are two constants which do not affect results Then we note when ∥∇ θ n g ( θ n ) ∥ 2 > ˆ a , there is
[Equation]
where ˆ M is a constant which does not affect results.
So we get n
[Equation]
Substitute (57) into (56) and note c ′′
ˆ Mϵ 2 n = o ( ϵ n ) , then it follows that
[Equation]
We denote
[Equation]
[Equation]
For convenience, we let ˆ
[Equation]
Through (54), we get
[Equation]
n , so there is
[Equation]
where ˆ q 0 is a constant.
We focus on ˆ G (ˆ a ) i ˆ F (ˆ a ) i .
Using O ′ stolz theorem yields
[Equation]
Through Assumption 3.3, we know when θ ∈ S
= {∥∇ θ g ( θ ) ∥ 2 < δg ( θ ) } , there is g ( θ ) <
ˆ T .
We define ˆ S =
{ g ( θ ) < 2 ˆ T } .
Then when θ i ∈ ˆ S , there is I (ˆ a ) i ∥∇ θ i g
( θ i )
∥ 2 > (ˆ a/ 2 ˆ T )
I (ˆ a ) i g ( θ i ) , and when θ i ∈ R N /S , there is I (ˆ a ) i ∥∇ θ i
g
( θ i )
∥ 2 >
δ
I (ˆ a ) i g ( θ i ) .
We assign ˆ c = min { 2 δ 0 , ˆ a/ 2 ˆ T } , then it holds that
[Equation]
Thus, it follows that
[Equation]
[Equation]
C.4 Proof of Theorem 3.4.
From (31) with ϵ n ≡ ϵ , it follows that
[Equation]
Let the event
A ( φ ) n = (cid:8) ∥∇ θ 1 g ( θ 1 ) ∥ 2 > φ, ∥∇ θ 2 g ( θ 2 ) ∥ 2 > φ, ..., ∥∇ θ n g ( θ n ) ∥ 2 > φ (cid:9) and the characteristic function be I ( φ ) n , then we attain n (cid:88) [Equation]
[Equation]
Using Taylor expansion, we derive that
[Equation]
Considering the term ∇ θ t g
( θ t ) T v t , it holds that
[Equation]
Note ( I ( φ ) t − I ( φ ) t − 1 )
I ( φ )
[Equation]
− 1 = I ( φ ) t − I ( φ ) t − 1 and I ( φ ) t − I ( φ ) t − 1 → 0 a.s. .
Then by substituting the above equation into (11) and noting − ( ∇ θ ( i ) g (
[Equation]
( i ) )
[Equation]
∥ ≤ c ∥ v t − 1 ∥ 2 , we obtain
[Equation]
where c > 0 is a constant.
It follows that
[Equation]
where c ′′ is a constant which does not affect the result.
Substituting (64) into (67) yields
I ( φ ) t +1 g
( θ n +
[Equation]
Then we multiply I ( φ ) n on the both sides of (68) and notice I ( φ ) n ≥ I ( φ ) n +1 to derive that
[Equation]
Then we take the expectation on (68) and obtain
E
[Equation]
I [Equation] φ ) n +1 g
[Equation]
n +1 )
[Equation]
Under Assumption 3.1 2), it holds that
[Equation]
It follows that
[Equation]
Let µ
= 1 /c ′′
ˆ σ 0 (cid:0) M 0 + aφ (cid:1) .
Then when ϵ < µ 0 , it holds that (cid:0) ϵ − c
[Equation]
ˆ σ 0 ϵ 2 (cid:0) M 0 + aφ (cid:1)(cid:1) > 0 .
We derive that n
[Equation]
In addition, we have
[Equation]
Due to A ( φ )⊃ A ( φ )⊃ ... ⊃ A ( φ ) n , we get P
[Equation]
Through the arbitrariness of ϕ , we attain the conclusion.
C.5 Proof of Proposition
From (70), it follows that E (cid:0) I ( φ ) n ∥∇ θ n g ( θ n ) ∥ 2 (cid:1) → 0 .
So for any φ > 0 , there exists n 0 , when n > n 0 , it holds that
[Equation]
Then we define another event First we construct an event
[Equation]
where k 0 is an undetermined coefficient Then we define another events
[Equation]
B i,j = { g ( θ i ) − g ∗ ≤ φ and for all n 0 ∈ ( i, j ] g ( θ n ) − g ∗ > φ } .
Specially, we denote B n,n = B n .
There is ˆ B n = n (cid:91) i =
B i,n .
Let the characteristic function of B n be ˆ I n , B i,j be ˆ
I
i,j .
It is obvious that
[Equation]
Then from (68) we can get that
[Equation]
73) where µ > 0 is a constant.
Then we denote ˆ g = g − φ .
We take the mathematical expectation on (73) to obtain
[Equation]
We construct a variable F i,n +1 as follows
[Equation]
So we can get F i,n
[Equation]
1 − F i,n ≤
[Equation]
where σ 0 and 𭟋 0 are two constants not dependent on ϵ or φ .
We let
[Equation]
Then we get
[Equation]
Then we get Q i,n /F i,n > ϵ 𭟋 0 /φ .
So we can attain
[Equation]
[Equation]
where y
=
max { 1 − ϵφ 𭟋 0 , 1 / 2 − α } .
Then we first prove E ( g ( θ n )) is uniformly bounded.
We have
E
[Equation]
θ
[Equation]
where ˆ m is a constant.
Then we get
[Equation]
Now we have proved E (cid:0) g
( θ n ) (cid:1) uniformly bounded.
Then we aim to prove E (cid:0) g
( θ n ) (cid:1) can become arbitrarily small if n sufficient large and ϵ sufficient small.
First We have
[Equation]
∥
[Equation]
≤ 1 − α n (cid:88) i = α
[Equation]
where ˆ p 0 is a constant.
Then we get that
[Equation]
Since E (cid:0) ˆ g ( θ i ) (cid:1) ≤ 0 , we get E (cid:0) ˆ g
[Table]
θ
[Table] +1 ) (cid:1) ≤ ˆ p 1 ϵ/φ .
Finally we substituted it into (74), getting
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table] [Equation] [Table] [Equation]
[Table]
[Equation]
[Table]
[Equation]
where ˆ p 2 is a constant. Then we can get [Table]
[Equation]
where ˆ p ′ 0 is a constant.
Then we get that
[Equation]
where ˆ p ′ 1 is a constant.
Then we can get
[Equation]
where ˆ p ′ 2 is a constant.
Now we can find that provided ϵ small enough, we can get for any small bound φ 0 ,
E (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 < φ 0 (when n is large enough).