O N THE C ONVERGENCE OF M SGD AND A DA G RAD FOR S TOCHASTIC O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
As one of the most fundamental stochastic optimization algorithms, stochastic gradient descent (SGD) has been intensively developed and extensively applied in machine learning in the past decade.
There have been some modiﬁed SGD-type algorithms, which outperform the SGD in many competitions and applications in terms of convergence rate and accuracy, such as momentum-based SGD (mSGD) and adaptive gradient algorithm (AdaGrad).
Despite these empirical successes, the theoretical properties of these algorithms have not been well established due to technical difﬁculties.
With this motivation, we focus on convergence analysis of mSGD and AdaGrad for any smooth (possibly non-convex) loss functions in stochastic optimization.
First, we prove that the iterates of mSGD are asymptotically convergent to a connected set of stationary points with probability one, which is more general than existing works on subsequence convergence or convergence of time averages.
Moreover, we prove that the loss function of mSGD decays at a certain rate faster than that of SGD.
In addition, we prove the iterates of AdaGrad are asymptotically convergent to a connected set of stationary points with probability one.
Also, this result extends the results from the literature on subsequence convergence and the convergence of time averages.
Despite the generality of the above convergence results, we have relaxed some assumptions of gradient noises, convexity of loss functions, as well as boundedness of iterates.
I NTRODUCTION
In recent years, the rapid development of machine learning has stimulated a lot of applications of optimization algorithms to employing tremendous data in practical scenarios.
Many optimization algorithms of machine learning are based on gradient descent (GD).
A typical GD algorithm, minimizing a loss function g ( θ ) ∈ R via seeking an N -dimensional real-valued parameter θ ∗ = arg min θ ∈ R N g
( θ ) , writes as follows
[Equation]
where θ n is the estimate of θ ∗ at step n , ε n is a positive step size (learning rate) to be designed, and ∇ θ n g ( θ n ) stands for the gradient of g ( θ n ) at step n .
When certain conditions are satisﬁed, θ n in equation 1 will converge to the optimal solution.
However, since the computation of ∇ θ n g ( θ n ) relies on all data at each step, it is inefﬁcient to apply GD-based algorithms (e.g., equation 1) when the data amount is very large.
Therefore, more and more attention has been given on how to accelerate the GD-based algorithms.
One of the attempts is stochastic gradient descent (SGD) which originated from Robbins & Monro (1951).
Instead of calculating ∇ θ n g ( θ n ) over all data, SGD algorithms use a relatively small proportion of data to estimate the gradient, i.e., ∇ θ n g ( θ n , ξ n ) where ξ n is a random vector introduced to choose a subset of data for each update.
Then the SGD modiﬁed from GD equation 1 is as follows
[Equation]
Besides reducing gradient computation, g ( θ n , ξ n ) can be used as an estimate of the gradient for the scenarios where the accurate gradient is unavailable due to external noises.
In recent years, SGD has shown its prominent advantages in dealing with high dimensional optimization problems such as regularized empirical risk minimization and training deep neural networks (see, e.g. Graves et al.
; Nguyen et al.
; Hinton & Salakhutdinov (2006); Krizhevsky et al.
(2012) and references therein).
In Nguyen et al.
; Bottou (2012), the convergence and the convergence rate of SGD have been analyzed.
Despite outstanding successes of SGD, it usually has a relatively slow convergence rate induced by random gradients and it provides ﬂuctuating iterates in the learning process.
In order to improve the theoretical and empirical performance of SGD, there have been a number of investigations, including three typical algorithms for accelerating the convergence rate: (1) The momentum-based stochastic gradient descend (mSGD), which reduces the update variance by averaging the past gradients (Polyak, 1964); (2) The adaptive gradient algorithm (AdaGrad), which replaces the step size of SGD with adaptive step size (Duchi et al., 2011b); (3) Adaptive momentum gradient algorithm (Adam), which integrates mSGD and AdaGrad (Kingma & Ba, 2015).
However, the convergence results of these algorithms have not been well established, especially for non-convex loss functions.
In this paper, we focus on the investigation of mSGD and AdaGrad.
We believe that the convergence analysis results in this paper are helpful for the future research on the convergence or generalization of Adam and other stochastic optimization algorithms.
The technique of mSGD is originally developed by Polyak (Polyak, 1964) for the acceleration of convergence rate of gradient-based methods.
A typical expression of mSGD is as follows (Zareba et al., 2015; Sutskever et al., 2013)
[Equation]
where α ∈ [ 0 , 1 ) and ε n > 0 are relatively momentum coefﬁcient and step size (learning rate), respectively.
Another type of mSGD with the name of stochastic heavy ball (SHB) has also been studied (Gitman et al., 2019; Gupal & Bazhenov, 1972),
[Equation]
where β n ∈ ( 0 , 1 ) and γ n are relatively momentum coefﬁcient and step size (learning rate).
In recent years, mSGD has been widely employed in the applications of deep learning such as image classiﬁcation (Krizhevsky et al., 2012), fault diagnosis (Tang et al., 2018), statistical image reconstruction (Kim et al., 2014), etc.
Moreover, a number of variants on momentum are emerging, see, e.g., synthesized Nesterov variants (SNV) (Williams & Lovett, 2016), robust momentum (Cyrus et al., 2018), and PID-control based methods (An et al., 2018).
The importance of momentum in deep learning has been illustrated in Sutskever et al.
through experiments.
However, the theoretical analysis for convergence and convergence rate of mSGD needs further investigation, especially for non-convex loss functions which are common in deep learning.
Most existing results are established with guaranteed subsequence convergence or convergence of time averages 1 (see Yang et al.
; Gitman et al.
; Polyak (1977); Kaniovskii (1983), Li & Orabona (2020) and references therein).
Nevertheless, there is a still distance to asymptotic convergence, which is usually more general and useful in practical applications requiring stable iterates for each realization.
Sebbouh et al.
(2020) studied the asymptotic convergence of g ( θ n + 1 ) by using the time-average estimates θ n + 1 , where θ n +
1 = ω n θ n + ( 1 − ω n ) θ n and ω n ∈ ( 0 , 1 ) .
However, the convergence of g ( θ n + 1 ) was not guaranteed..
Convergence analysis for SGD unveils that the convergence rate heavily depends on the choice of step size ε n (Robbins & Monro, 1951; Nguyen et al., 2018), which may consume a huge amount of efforts on ﬁne tune.
In order to deal with this problem, the authors in Duchi et al.
(2011b) proposed AdaGrad with adaptive step size based on the square summation of previous gradients.
In the literature, there are two main forms of AdaGrad. One is based on the norm of gradients as follows (Streeter & Mcmahan, 2010) [Equation] where α 0 > 0 is a constant.
The other form is based on the coordinate-wise gradients (Duchi et al., 2011a; Li & Orabona, 2019; 2020)
[Equation]
where α 0 is a constant.
In our paper we focus on the norm form (equation 5).
In recent years, AdaGrad has shown its effectiveness in the ﬁeld of sparse optimization (Duchi et al., 2013), tensor factorization (Lacroix et al., 2018), and deep learning (Heaton & Jeff, 2018).
Some algorithm variants like RMSProp (Tieleman & Hinton, 2012) and SAdaGrad
(Chen et al., 2018) are also studied.
However, there are few results on the convergence of AdaGrad.
Most of these results only prove subsequence convergence or convergence of time averages (see, e.g. Zou et al.
; Chen et al.
(2019); D´efossez et al.
Ward et al.
Although Li & Orabona (2019) and Gadat & Gavra (2020) studied the asymptotic convergence of a modiﬁed AdaGrad algorithm, the result is not applicable to AdaGrad in equation 5.
Thus, the asymptotic convergence of AdaGrad in equationis still open.
In this theoretical paper, we aim to establish the convergence of mSGD and AdaGrad under mild conditions.
The main contributions of this paper are three-fold:
• We prove that the iterates of mSGD are asymptotically convergent to a connected set of stationary points for possibly non-convex loss function almost surely ( i e., with probability one), which is more general than existing works on subsequence convergence.
• We quantify the convergence rate of mSGD for the loss functions.
Through this convergence rate we can get a theoretical explanation of why mSGD can be seen as an acceleration of SGD.
Moreover, we provide the convergence rate of mean-square gradients and connect it to the convergence of time averages.
• We prove the iterates of AdaGrad are asymptotically convergent to a connected set of stationary points almost surely for possible non-convex loss functions.
The convergence result for the AdaGrad extends the subsequence convergence in the literature.
The remainder of the paper is organized as follows.
In Section 2, we introduce the related work considering the convergence of mSGD and AdaGrad.
The main results of the paper are given in
Section 3, where we study the convergence and convergence rate of mSGD as well as the convergence of AdaGrad.
Section 4 concludes the whole paper.
Sections 5 and 6 are Code of Ethics and
Reproducibility, respectively.
The proofs are given in Appendix.
R ELATED WORK
Convergence of mSGD:
For the normalized mSGD (SHB), Polyak (Polyak, 1977; 1964) and Kaniovski (Kaniovskii, 1983) studied its convergence (subsequence convergence and convergence of time averages) properties for convex loss functions.
Igor Gitman (Gitman et al., 2019) provided some convergence results of mSGD (SHB) for non-convex loss functions, but there is a considerable distance to the asymptotic convergence.
Moreover, there is a requirement for uniform boundedness of a noise term in Gitman et al.
(2019), i.e., E ( (cid:107) ∇ θ n g ( θ n , ξ n ) − ∇ θ n g ( θ n (cid:107) ) ≤ δ , which conﬁnes the application scope of mSGD (SHB).
In addition, the designs of momentum coefﬁcients in Polyak (1977; 1964); Kaniovskii (1983); Gitman et al.
(2019) are not consistent with some practical applications (Smith et al., 2018;
Sutskever et al., 2013).
Therefore, the asymptotic convergence of mSGD for convex and non-convex loss functions needs further investigation.
Convergence rate of mSGD: Despite outstanding empirical successes of mSGD, there are few results on convergence rate of mSGD.
In these results, Mai & Johansson (2020) studied a class of convex loss functions, and obtained a convergence rate of time averages without reﬂecting the role of momentum parameter.
Gitman et al.
and Nicolas Loizou & Richt´arik (2020) respectively investigated the asymptotic convergence rate of mSGD (SHB) by restricting to quadratic loss functions.
Liu et al.
(2020) studied the properties of SHB, where the relation between the loss function of
SHB and the step size in every step was studied under the setting that α n and β n are constants.
The convergence rate of time averages was also studied.
However, since the momentum parameters are not consistent with some applications (Smith et al., 2018; Sutskever et al., 2013) and the standard mSGD in equation 3 is not covered, further studies are needed.
Convergence of AdaGrad: In the original work for AdaGrad (Duchi et al., 2011b), it was proved that AdaGrad can converge faster in the time averages sense if gradients are sparse and the loss function is convex.
Similar results were established by Chen et al.
(2019), and Ward et al.
Zou et al.
and D´efossez et al.
(2020) established convergence results in the subsequence sense.
Asymptotic convergence was obtained in Li & Orabona (2019) for non-convex functions, but the form of the algorithm is no longer standard, as discussed in Duchi et al. (2011b).
Although such a change alleviates the difﬁculty in the proof of asymptotic convergence, it cannot be applied to the study of the AdaGrad in equation 5.
Moreover, they required that a noise term is of point-wise boundedness (i.e., (cid:13)(cid:13) ∇ θ n g ( θ n , ξ n ) − ∇ θ n g ( θ n ) (cid:13)(cid:13) ≤ δ , where δ is a positive constant), which however is relatively restrictive.
M AIN RESULTS
In this section, we provide the main results of this paper, including the analysis of convergence and convergence rate of mSGD in equation 2 and the analysis of convergence of AdaGrad in equation 5.
In the following, R N denotes the N -dimensional Euclidean space and (cid:107)·(cid:107) stands for the 2-norm, i.e., the Euclidean norm.
To proceed, we need some deﬁnitions, consisting of asymptotic convergence, subsequence convergence, mean-square convergence, and convergence of time averages.
Deﬁnition 1 ( Asymptotic convergence )
A sequence { x n } is asymptotically convergent to a set K , if lim n → + ∞ (cid:16) inf x ∈ K (cid:13)(cid:13) x n − x (cid:107) (cid:17) = 0 .
Deﬁnition 2 ( Subsequence convergence )
A sequence { x n } converges in subsequence to a set K , if there exists at least one subsequence { x k n } of { x n } such that lim n → + ∞ (cid:16) inf x ∈ K (cid:13)(cid:13) x k n − x (cid:107) (cid:17) = 0 .
Deﬁnition 3 ( Mean-square convergence )
A stochastic sequence { x n } converges in mean square to a ﬁxed vector x, if lim n → + ∞ E ( (cid:107) x n − x (cid:107) 2 ) = 0 .
Deﬁnition 4 ( Convergence of time averages )
A stochastic sequence { x n } converges in time averages to a ﬁxed vector x, if lim T → + ∞ 1 T ∑ Tn = 1 E ( (cid:107) x n − x (cid:107) 2 ) = 0 .
It is obvious that asymptotic convergence implies subsequence convergence, and that mean-square convergence ensures convergence of time averages, but not vice versa.
3. C ONVERGENCE OF M SGD
In this subsection, with the help of some stochastic approximation techniques (Chen, 2006), we aim to prove that θ n in equation 3 is asymptotically convergent to a connected component J ∗ of the set J : =
{ θ |(cid:107) ∇ θ g ( θ ) (cid:107) = 0 } almost surely (a.s.) under proper conditions.
When this connected component degenerates to a stationary point θ ∗ , it holds that θ n → θ ∗ , a.s..
In contrast to the existing works of subsequence convergence (cf.
Zou et al.
; Chen et al.
(2019); D ´ efossez et al.
Ward et al.
, we aim to prove that θ n of mSGD in equationis able to achieve asymptotic convergence.
Since mSGD in equation 3 is a stochastic algorithm, we aim to establish its a.s. asymptotic convergence.
To proceed, we need some reasonable assumptions with respect to noise sequence { ξ n } and loss function g ( θ ) .
Assumption 1 Noise sequence { ξ n } are mutually independent and independent of θ 1 and v 0 , such that g ( x ) =
E ξ n (cid:0) g ( x , ξ n ) (cid:1) for any x ∈ R N .
Assumption 2 ( Loss function assumption )
Loss function g ( θ ) satisﬁes the following conditions:
1) g ( θ ) is a non-negative and continuously differentiable function.
The set of stationary points of (cid:107) ∇ θ g ( θ ) (cid:107) is not an empty set, that is
[Equation]
= /0 . 3) ∇ θ g ( θ ) satisﬁes the Lipschitz condition,
i
[Equation]
e., there is a scalar c > 0 , such that for any x , y ∈ R N
[Equation]
There is a scalar M > 0 such that for any θ ∈ R N and positive integer n,
[Equation]
[Equation]
Assumption 1 and conditions 1)–3) of Assumption 2 are common in the literature (Gitman et al., 2019).
Assumption 2 does not pose any requirement on the convexity of g ( θ ) .
In other words, we allow any convex or non-convex loss functions g ( θ ) as long as they satisfy this assumption.
Condition 4) corresponds to the condition in Shalev-Shwartz et al.
(2011); Nemirovski et al.
Hazan & Kale (2014); Gitman et al.
; Yang et al.
; Polyak (1977); Kaniovskii (1983) where the following inequality is assumed to hold for any θ ∈ R N and positive integer n ,
[Equation]
where K is a positive scalar.
Note that equation 6 reduces to equation 7 if g ( θ ) is uniformly upper bounded over the space R N .
Since equation 6 does not need this uniform boundedness, it substantially extends equation 7 such that mSGD is also applicable to the scenarios with unbounded loss functions.
Different from deterministic GD-type algorithms with a constant step size, in order to ensure the convergence of mSGD, we need a decreasing step size for counteracting the randomness induced by noise { ξ n } (Gitman et al., 2019; Robbins & Monro, 1951).
Speciﬁcally, we make the following assumption on step size ε n together with a ﬁxed momentum coefﬁcient α .
Assumption 3 Momentum coefﬁcient α ∈ [ 0 , 1 ) and the sequence of step size ε n is positive, monotonically decreasing to zero, such that ∑ + ∞ n = 1 ε n = + ∞ and ∑ + ∞ n =
ε 2 n < + ∞ .
The setting of ε n in Assumption 3 is consistent with stochastic approximation for root seeking of functions (Chen, 2006) as well as some stochastic optimization algorithms (Gupal & Bazhenov, 1972; Polyak, 1977; Kaniovskii, 1983; Gitman et al., 2019).
An explicit example of step size ε n satisfying Assumption 3 is ε n = 1 / n .
Although SHB in equation 4 shares a similar expression as mSGD, the provided conditions of step sizes in Gupal & Bazhenov (1972); Polyak (1977); Kaniovskii (1983); Gitman et al.
(2019) are not applicable to the general cases of mSGD with static α (Smith et al., 2018;
Sutskever et al., 2013).
In fact, SHB in equation 4 has a similar expression as mSGD in equation 3.
Multiplying γ n on both sides of the ﬁrst equation of equation 4 yields
[Equation]
We treat γ n v n as one term, as the role of v n in equation 3.
By comparing coefﬁcients of equationand equation 3, we get α ( n ) =
( γ n / γ n − 1 ) β n and ε n = γ n ( 1 − β n ) .
In the literature (Gupal & Bazhenov, 1972; Polyak, 1977; Kaniovskii, 1983; Gitman et al., 2019), the parameter setting of SHB in equation 4 has the following requirements
[Equation]
According to the above requirements on β n and γ n , the requirements on ε n and α ( n ) are
[Equation]
We can see that the static momentum parameter α , which is widely used in practical applications, does not satisfy these conditions.
Before providing the main theorem for convergence, we need a useful lemma
In the analysis of asymptotic convergence of mSGD, the following lemma plays an important role.
Lemma 1 Consider the mSGD in equation 3.
If Assumptions 1–3 hold, then for ∀ θ 1 ∈ R N and v 0 ∈ R N , there is a scalar T ( θ 1 , v 0 ) ,
such that E (cid:0) g ( θ n ) (cid:1) <
T ( θ 1 , v 0 ) for any n ≥ 1 .
Lemma 1 actually guarantees stability of mSGD.
Let F n = σ ( θ 1 , v 0 , { ξ i } ni = 1 ) be the minimal σalgebra generated by θ 1 , v 0 , { ξ i } ni = 1 .
As a result, θ n is adapted to F n − 1 .
Then for any n ∈ N + , it holds that
[Equation]
where the second equality follows from the independence between ξ n and { θ 1 , v 0 , { ξ i } n − 1 i = 1 } in Assumption 1, and the last two inequalities hold due to 4) in Assumption 2 and Lemma 1, respectively.
Intuitively, the result in equation 9 means that the ﬂuctuation induced by random noise { ξ i } n − 1 i =is well restrained.
Note that the derived result in equation 9 is totally different from equationrequired in the literature (Shalev-Shwartz et al., 2011;
Nemirovski et al., 2009; Hazan & Kale, 2014; Gitman et al., 2019;
Yang et al., 2016; Polyak, 1977; Kaniovskii, 1983), since equationneeds a uniformly upper bound K over the whole space (i.e., θ ∈ R N ) which is difﬁcult to satisfy when loss function g ( θ ) is quadratic or cubic with respect to θ over unbounded parameter space.
In contrast, regardless of the order of g ( θ ) with respect to θ , equation 9 ensures boundedness of
E (cid:16)(cid:13)(cid:13) ∇ θ n g ( θ n ) − ∇ θ n g ( θ n , ξ n ) (cid:13)(cid:13) 2 (cid:17) for learning any ﬁxed true parameter θ ∗ .
This reﬂects a favorable learning process against random noise ξ n for dealing with general loss functions g ( θ ) .
This result paves the way to the following theorem on asymptotic convergence of mSGD.
Theorem 1 Consider the mSGD in equation 3.
If Assumptions 1–3 hold, then for ∀ θ 1 ∈ R N and ∀ v 0 ∈ R N , there exists a connected set J ∗ ⊆
J such that the iterate θ n is convergent to the set J ∗ almost surely, i.e.,
[Equation]
where d ( x , J ∗ ) =
inf y {(cid:107) x − y (cid:107) , y ∈
J ∗ } denotes the distance between point x and set J ∗ .
In Theorem 1, we prove that the iterates of mSGD asymptotically converge to a connected set of stationary points almost surely.
When this connected set degenerates to a stationary point θ ∗ , it holds that θ n → θ ∗ , a.s.. The result enables engineers for the design of proper momentum coefﬁcients and step sizes (like α = 0 .
9, ε n = 0 .
1 / n ) in the related applications of mSGD with mathematic guarantee.
3. C ONVERGENCE RATE OF M SGD
In this subsection, we analyze the convergence rate of mSGD.
Before that, given positive real sequences { a n } and { b n } , we let a n = O ( b n ) if there is a constant c > 0, such that a n / b n < c for any n ≥ 1 .
For quantitative analysis, we need some extra assumptions.
Assumption 4 ( Loss function assumption )
Loss function g ( θ ) satisﬁes the following conditions:
1) g ( θ ) is a non-negative and continuously differentiable function.
The set of its stationary points J = { θ |(cid:107) ∇ θ g ( θ ) (cid:107) = 0 } is a bounded set which has only ﬁnite connected components J 1 ,..., J n .
In addition, there is ε (cid:48) > 0 , such that for any i ∈ { 1 , 2 ,..., n } and 0 <
d ( θ , J i ) <
ε (cid:48) , it holds that (cid:12)(cid:12) g
( θ ) − g i (cid:12)(cid:12) (cid:54) = 0 , where g i = { g ( θ ) | θ ∈ J i } is a constant. 2)
For any i ∈ { 1 , 2 ,..., n } , it holds that liminf d ( θ , J i ) →(cid:107) ∇ θ g ( θ ) (cid:107)g ( θ ) − g
i ≥ s > 0 .
In many problems of machine learning, especially deep learning, because of strong non-linearity mapping from input data to output data and the structure complexity of employed models, loss functions could be non-convex and may have multiple local critical points.
Assumption 4 does not require convexity of the loss function, but guarantees the properties of the loss function around critical points.
Assumption 4 2) can be treated as a local version of Polyak-Lojasiewicz (P-L) condition.
Theorem 2 Consider the mSGD in equation 3.
If Assumptions 1-4 hold, for ∀ v 0 ∈ R N and ∀ θ 1 ∈ R N , it holds that
[Equation]
where p = exp (cid:26) ∑ ∞ k = 1 M ε 2 k (cid:27) and M is deﬁned in condition 4) of Assumption 2.
In Theorem 2, let α = 0, then we can obtain the convergence rate of SGD,
[Equation]
According to the obtained bounds, the convergence rate of mSGD with α ∈ ( 0 , 1 ) is larger than that of SGD, which is the case with α = 0.
Moreover, Theorem 2 provides a stronger characterization of convergence rate than some existing works considering the convergence rate of time averages
T
[Equation]
1 in Liu et al
[Equation]
and l = − 1 / 2 in Mai & Johansson (2020)).
In the following, we will elaborate on this point.
From equation 10, there exists a scalar t 0 > 0 such that ∀ n >
[Equation]
implying
[Equation]
=
[Equation]
(cid:111) <
t 0 .
So it holds that
T
[Equation]
Let ε n = 1 n , then we know that ∑ Tn = 1 ε n =
O ( ln T ) .
Hence,
T
T
[Equation]
e
[Equation]
[Equation] If s p ( 1 − α ) 2 < 1, ∑ Tn = 1 n − s
[Equation]
[Equation] [Equation] O (cid:16) T −
[Equation]
p
[Equation]
) 2 + 1 (cid:17) , so we have T
[Equation]
p
[Equation]
(cid:17) .
If s p ( 1 − α ) 2 = 1, it holds that ∑ T n = 1 n − s p ( 1 − α )
= O ( ln T ) , and hence
T
T ∑ [Equation] e [Equation]
ln T [Equation]
O (cid:16) ln T T (cid:17) .
If s p ( 1 − α ) 2 > 1, then ∑ Tn = 1 n − s
[Equation]
[Equation] ) 2 : =
C α ( T ) → C α as T → + ∞ , where C α is a positive constant, so
[Equation]
Note that the coefﬁcient C α depends on s , p , and α , where s is the constant given in Assumptiononly depending on g ( θ ) , and p is given in Theorem 2 and relies on M in 4) of Assumption 2 and step size ε n .
The above observation indicates that setting α close to one makes mSGD achieve convergence rate of time averages of order O ( 1 / T ) .
As remarked previously, larger α implies smaller coefﬁcient
C α and thus quicker convergence rate.
Interestingly, since C α ( T ) is monotonically increasing, the convergence rate has a relatively small coefﬁcient when T is small.
This could illustrate why mSGD can achieve better performance in the early phase of iteration.
3. C ONVERGENCE OF A DA G RAD
In this subsection, we aim to establish the convergence of AdaGrad.
Compared to the study of mSGD, the design of adaptive step size increases the technical difﬁculties.
To proceed, the required conditions on loss function g ( θ ) are summarized as follows.
Assumption 5 Loss function g ( θ ) in equation 5 satisﬁes the following conditions:
1) g ( θ ) is a non-negative and continuously differentiable function The
set of its stationary points J = { θ |(cid:107) ∇ θ g ( θ ) (cid:107) = 0 } is a bounded set which has only ﬁnite connected components J 1 ,..., J n .
In addition , there is ε 1 > 0 , such that for any i and 0 <
d ( θ , J i ) <
ε 1 , it holds that (cid:12)(cid:12)
g ( θ ) − g i (cid:12)(cid:12) (cid:54) = 0 , where g i = g ( θ ) | θ ∈ J
i } is a constant.
The gradient ∇ θ g ( θ ) satisﬁes the Lipschitz condition,
i
e., for any x , y ∈ R N ,
[Equation]
There are two constants M (cid:48) > 0 and a > 0 such that for any θ ∈ R N and n ∈ N + ,
[Equation]
Condition 2) is the same as in 2) of Assumption 2.
Condition 1) is relatively weak, because it does not require any convexity of the loss function or global conditions as P-L condition.
There are many functions satisfying Assumption 5 1) but not convex, such as y = sin 2 ( x ) , y =
( x − 1 )( x − 2 )( x − 3 )( x − 4 ) , and y =
cos 2 ( x ) .
Similar to condition 4) of Assumption 2, condition 3) is a condition to restrain the noise inﬂuence.
Equation 11 is milder than (cid:107) ∇ θ g ( θ , ξ n ) − ∇ θ g ( θ ) (cid:107)
≤ S a .
s .
(Li & Orabona, 2019; D ´ efossez et al., 2020), which is relatively restrictive in dealing with unbounded noises, e.g., ∇ θ g ( θ , ξ n ) = ∇ θ g ( θ ) + ξ n , where ξ n is independent identically distributed and Gaussian.
Compared to SGD and mSGD, there are more challenges in analyzing the convergence of AdaGrad.
The challenges mainly come from two aspects: (1) since AdaGrad does not have a decreasing step size, the noise inﬂuence on AdaGrad cannot be restrained as mSGD which is with the help of decreasing step size satisfying Assumption 3; (2) adaptive step size of AdaGrad in equation 5 (i.e., α 0 / (cid:112)
∑ ni = 1 (cid:107) ∇ θ i g
( θ i , ξ i ) (cid:107) 2 ) is a random variable conditionally dependent of ξ n given σ { θ 1 , ξ 1 ,..., ξ n − 1 } .
Then when we deal with terms in the proof like
[Equation] [Equation]
we cannot make the conditional expectation to transform equation 12 to
[Equation]
∇ [Equation] In the literature, Li & Orabona (2019) changed the step size to
[Equation] and Gadat & Gavra (2020) changed the step size to γ n + 1 / √ ω n + ε where [Equation]
and γ n , p n , q n are tuned parameters, in order to make it conditionally independent of ξ n given σ { θ 1 , ξ 1 ,..., ξ n − 1 } , which however is no longer the standard AdaGrad as in equation 5 (Streeter &
Mcmahan, 2010; Chen et al., 2019).
Before we provide the main theorem of this subsection, a useful lemma is worth discussing.
Lemma 2 Let { α k } , { β k } being non-negative random variable sequences, such that the following conditions hold almost surely
• ∑ + ∞ k = 1 α k = ∞ ; • ∑ + ∞ k =
α k β k < + ∞ ;
Then there exists a subsequence { β n k } of { β k } , such that β n k k → ∞ −→ 0 almost surely.
Proof 1 We aim to prove liminf k
→
+ ∞ β k = 0 by contradiction.
Suppose liminf k → + ∞ β k = u > 0 , then ∃ k 0 , such that ∀ k > k 0 , β k > u / 2 .
It follows that
[Equation]
Obviously, there is a contradiction between equation 13 and the condition ∑ + ∞ k = 1 α k = + ∞ .
Thus, liminf k → + ∞ β k = 0 .
So we get that there exists a subsequence { β n k } of { β k } holds β n k k → ∞ −→ 0 almost surely.
This lemma, inspired by Proposition 2 in Ya.
I. Alber (Alber et al., 1998), is quite useful in the convergence analysis of AdaGrad.
Then we are ready to provide the convergence result of AdaGrad in the following theorem.
Theorem 3 Consider the AdaGrad in equation 5.
If Assumptions 1 and 5 hold, then for ∀ θ 1 ∈ R N and S 0 = 0 , there exists a connected component of the set J ∗ ⊆
J, such that the estimate θ n is convergent to the set J ∗ almost surely, i.e.,
[Equation]
In Theorem 3, we prove the standard AdaGrad is convergent almost surely, in contrast to the convergence of time averages in Chen et al.
Ward et al.
(2019) which focus on the metric
T ∑ Ti = 1 E ( (cid:107) ∇ θ n g ( θ n ) (cid:107) 2 ) , or the subsequence convergence in Zou et al.
; D ´ efossez et al.
(2020), focusing on min n = 0 , 1 , 2 ,..., T E ( (cid:107) ∇ θ n g ( θ n ) (cid:107) ) .
C ONCLUSION AND F UTURE W ORK
In this paper, we studied the convergence of two algorithms extensively used in machine learning applications, namely, momentum-based stochastic gradient descent (mSGD) and adaptive step stochastic gradient descent (AdaGrad).
By considering general loss functions (either convex or nonconvex), we ﬁrst establish the almost sure asymptotic convergence of mSGD.
Moreover, we ﬁnd the convergence rate of the mSGD and reveal that the mSGD indeed has a faster convergence rate than the SGD.
Furthermore, we prove AdaGrad is convergent almost surely under mild conditions.
Subsequence convergence and convergence of time averages in the literature are substantially extended in this work to asymptotic convergence.
To better understand the AdaGrad, we will study its convergence rate in the future.
C ODE OF E THICS
This is a theoretical paper focusing on the investigation for the convergence of mSGD and AdaGrad optimization algorithms.
The results in this paper do not have issues in fairness, inappropriate potential applications and impact, privacy and security issues, legal compliance, research Integrity
Issues, or responsible research practice (e.g., IRB, documentation, research ethics).
This paper does no involve human subjects, practices to data set releases, potentially harmful insights, methodologies and applications, or pontential conﬂicts of interest and sponsorship.
R EPRODUCIBILITY
This is a theoretical paper focusing on the investigation for the convergence of mSGD and AdaGrad optimization algorithms.
The developed results are provided in the main paper, i.e., Theorems 1-3.
In Appendix, the proofs of these theorems are provided together with some useful lemmas as well as their proofs.
An outline of these proofs is given in Section A in Appendix.
A A PPENDIX O UTLINE
Section B aims to verify the convergence results of mSGD.
Several auxiliary lemmas are ﬁrst provided, followed by a proof outline for the main results of mSGD.
Then the proofs of these lemmas are given.
Theorems 1-2 are proved in Sections B.9 and B.10, respectively.
Section C aims to verify the convergence results of AdaGrad.
Some auxiliary lemmas are given at the beginning, followed by a proof outline of the main results for AdaGrad in Section C.1.
Then we provide the proofs of the lemmas.
Theorem 3 is proved in Section C.8.
B C ONVERGENCE AND CONVERGENCE RATE OF M SGD
The following lemmas are used for the proofs of Theorems 1-2.
Hereafter, N + denotes the set of all positive integers.
A function f is in a (differentiability) class C k if its l th derivatives exist and are continuous, where l ≤ k .
Lemma 3 (Lemma 1.2.3 in Nesterov (2004))
Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with gradient satisfying the following Lipschitz condition
[Equation]
then for any x , y ∈ R N , it holds that
[Equation]
Lemma 4 Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with gradient satisfying the following Lipschitz condition
[Equation]
and the set S
=
{ x | ∇ f ( x ) = 0 } is bounded and only has ﬁnite connected components S 1 , S 2 ,..., S m .
Furthermore, assume there exists ε (cid:48) 1 > 0 , such that for any i = 1 , 2 ,..., m and x ∈ { x | 0 <
d ( x , S i ) < ε (cid:48) 1 } , it holds that (cid:12)(cid:12) f
( x ) − f i (cid:12)(cid:12) (cid:54) = 0 , where f i = f ( x ) for x ∈ S i .
Then for any i = 1 , 2 ,..., m, if there is ε (cid:48) 0 > 0 satisfying d ( x , S i ) < ε (cid:48) 0 , it follows that
[Equation]
Lemma 5 (Wang et al., 2019) Suppose that { X n } ∈ R N is an L 2 martingale difference sequence, and ( X n , F n ) is an adaptive process.
Then it holds that ∑ ∞ k =
X k < + ∞ a .
s .
, if
[Equation]
Lemma 6 Suppose that { X n } ∈ R N is a non-negative sequence of random variables, then it holds that ∑ ∞ n =
X n < + ∞ a .
s .
, if ∑ ∞ n = 0 E (cid:0) X n (cid:1) < + ∞ .
Lemma 7 Suppose { v n } is a sequence generated by mSGD in equation 3.
Under Assumptions 1–3, it holds that ∑ + ∞ n = 0 E (cid:0) (cid:107) v n (cid:107) 2 (cid:1) <
c ( v 0 , θ 1 ) , and ∑ + ∞ n = 0 (cid:107) v n (cid:107) 2 < + ∞ a .
s . , where c ( v 0 , θ 1 ) is a constant only related to v
and θ 1 .
Lemma 8 Suppose { θ n } is a sequence generated by mSGD in equation 3.
Under Assumptions 1–3, it holds that: for n ≥ 1 ,
[Equation]
where B ( v 0 , θ 1 ) > 0 is a constant only related to v
and θ 1 .
Lemma 9 Suppose { θ n } is a sequence generated by mSGD in equation 3 such that { g ( θ n ) } converges a.s..
If Assumptions 1–3 hold, and
[Equation]
where ζ n is a random variable such that lim n → + ∞ ζ n = ζ < ∞ a .
s . , and b is a positive constant, then there exists a connected component J ∗ of J : = { θ | ∇ θ g ( θ ) = 0 } , such that
[Equation]
B. P ROOF O UTLINE OF T HEOREMS 1 AND
The proof is in light of the Lyapunov method.
We aim to prove that g ( θ n ) is convergent a.s., and then to prove ∇ θ n g ( θ n ) → 0 a . s .
With these two results, we are able to get θ n → J ∗ a .
s .
In the following, we provide the proof outline to show how to obtain the provided results of mSGD.
Step 1: We prove mSGD is a stable algorithm, i.e., E ( g ( θ n )) <
K < + ∞ , ∀ n , in Lemma 1.
The idea is to prove that a weighted sum of the loss function value, i.e.,
[Equation] [Equation]
is bounded through a recursion formula (a rough form)
[Equation]
( A , B are two constants ) .
Then we apply Assumption 2 4) to I and then obtain
U n
[Equation]
( A , B are two constants ) .
Combining U n − 1 and R leads to
[Equation]
are two constants ) , where
[Equation]
and
[Equation]
Thus, we are able to obtain E ( g ( θ n )) <
F n − 1 ≤ A ∑ + ∞ t = 1 α t + B ∑ + ∞ t = 1 (cid:0) 1 / 2 − α (cid:1) t < + ∞ .
Step 2:
From Lemma 1 and the condition ∑ + ∞ t = 1 ε 2 n < + ∞ , we are able to prove that ∑ nt = 1 (cid:107) v t (cid:107) 2 and ∑ ε t (cid:107) ∇ θ t g ( θ t ) (cid:107) 2 are convergent a.s. respectively, as stated in Lemma 7 and Lemma 8.
Step 3: We divide g ( θ n ) into three terms
[Equation]
From Lemma 7 and Lemma 8, we are able to prove that ∑ nt = 1 A ( n ) (cid:107) v t (cid:107) 2 + ∑ nt = 1 B n ε t (cid:107) ∇ θ n g ( θ n ) (cid:107) 2 is convergent
a.s.. From the convergence theorem for martingale-difference sum (Lemma 5), we prove that ∑ nt = 1 C Tn ( ∇ θ n g ( θ n , ξ n ) − ∇ θ n g ( θ n )) is convergent a.s.
Then we prove g ( θ n ) is convergent a.s.
Step 4:
By Lemma 9 and the convergence of g ( θ n ) in Step 3, we get θ n → J ∗ a.s..
Step 5:
After the proof of the convergence of mSGD, we analyze the iterates of F n .
Then under a new assumption liminf d ( θ , J i ) → 0 (cid:107) ∇ θ g ( θ ) (cid:107) 2 / (cid:0) g
( θ )
− g i (cid:1) ≥ s > 0, we are able to obtain the convergent rate of mSGD.
B. P ROOF OF L EMMA
First we construct a closed and bounded set S which satisﬁes S ⊃ (cid:83) mi
= 1 S i .
Since (cid:107) ∇ f ( x ) (cid:107) ( x ∈ S ) is a continuous function on a closed set, (cid:107) ∇ f ( x ) (cid:107) ( x ∈ S ) is a uniformly continuous function.
Then ∃ ε (cid:48) 2 > 0, ∀ S i , if d ( x , S i ) < ε (cid:48) 2 , there is (cid:107) ∇ f ( x ) (cid:107) <
c ε (cid:48) 1 / 4.
We assign ε 0 =
min { ε (cid:48) 1 / 4 , ε (cid:48) 2 } .
Let S (cid:48)
i = { x | d ( x , S i ) < ε (cid:48) 1 } , S (cid:48)(cid:48)
i = { x | d ( x , S i ) < ε (cid:48) 0 } .
Since d ( x , S i ) is a continuous function, ∀ x 0 ∈ S (cid:48)(cid:48)
i , we can always ﬁnd a straight line l 0 paralle to ∇ f ( x 0 ) and passing through x 0 , deﬁned as
[Equation]
R ) .
From S (cid:48)(cid:48) i ⊂ S (cid:48)
i , we know x 0 ∈ S (cid:48)
i .
Since S (cid:48)
i is an open set, there exists α 0 < 0 < β 0 , such that x 0 + t ( ∇ f ( x 0 ) / (cid:107) ∇ f ( x 0 ) (cid:107) ) ∈ S (cid:48)
i ( α 0 < t < β 0 ) , and α 0 = sup t <(cid:32) x
[Equation]
t (cid:18) ∇ f ( x 0 ) (cid:107) ∇ f ( x
[Equation]
Deﬁne function
[Equation]
Since x 0 + t ( ∇ f ( x 0 ) / (cid:107) ∇ f ( x 0 ) (cid:107) ) ∈ S (cid:48) i ( α
0 < t < β 0 ) , it holds that
[Equation]
So g ( t ) ∈ C 1 ( t ∈ ( α 0 , β 0 ) ).
Then for ∀ t (cid:48) , t (cid:48)(cid:48) ∈ ( α 0 , β 0 ) , from Newton-Leibniz formula, it follows that
[Equation]
Next we will prove g (cid:48)
( x ) satisﬁes the Lipschitz condition.
According to the deﬁnition of S (cid:48)
i , f ( x 0 + t ∇ f ( x 0 ) / (cid:107) ∇ f ( x 0 ) (cid:107) )
− f
i keeps the same sign over t ∈ ( α 0 , β 0 ) .
Thus, ∀ τ 1 , τ 2 ∈ ( α 0 , β 0 ) , it holds that
[Equation]
From the Lipschitz condition of ∇ f
[Equation]
we have that
[Equation]
( x 0 ) (cid:107) (cid:13)(cid:13)(cid:13)(cid:13) =
c | τ 1 − τ 2 | .
From the above analsis, we obtain the Lipschitz condition of g (cid:48)
( x ) , that is, ∀ τ 1 , τ 2 , there is | g (cid:48)
( τ 1 ) − g (cid:48) ( τ 2 ) | ≤ c | τ 1 − τ 2 | .
By using the absolute value inequality, we get that − c | τ 1 − τ 2 | ≤ g (cid:48) ( τ 1 ) − g (cid:48) ( τ 2 ) ≤ c | τ 1 − τ 2 | .
Then it follows from equation 15 that
[Equation]
Let t (cid:48)
= 0 ,
t
[Equation]
= − g (cid:48) ( 0 ) / c .
So t (cid:48)
= 0 ∈ ( α 0 , β 0 ) .
Next, we will prove t (cid:48)(cid:48) ∈ ( α 0 , β 0 ) .
We separate it into two cases.
First, we assume that g (cid:48) ( 0 ) > 0.
In this case, because
[Equation]
we just need to prove
[Equation] [Equation]
Note that
[Equation]
From x 0 ∈ S (cid:48)(cid:48)
i , we get (cid:107) f
( x 0 ) (cid:107) < c ε (cid:48) 1 / 4 and d ( x 0 , S i ) <
ε (cid:48) 1 .
Then we have d (cid:18) x
0 + tf ( x 0 ) (cid:107) ∇ f ( x 0 ) (cid:107) , S
[Equation]
Thus, equation 16 holds, meaning when g
[Equation]
> 0,
[Equation]
∈ ( α 0 , β 0 ) .
Secondly, when g (cid:48) ( 0 ) <
[Equation]
we can also prove t (cid:48)(cid:48) ∈ ( α 0 , β 0 ) .
It follows that
[Equation]
That is
[Equation]
Because of the arbitrariness of x 0 , we concludes that ∃ ε (cid:48)
= min { ε (cid:48) 1 / 4 , ε (cid:48) 2 }
, ∀ S i
( i = 1 , 2 ,..., m ) , if d ( x , S i ) < ε (cid:48) 0 , there is
[Equation]
B. P ROOF OF L EMMA
For ∀ δ > 0, we have
[Equation]
where the last inequality is due to Markov inequality.
Since ∑ ∞ t = 1 E (cid:0) X t (cid:1) < ∞ , it follows that
[Equation]
By Cauchy’s convergence test, we have ∑ ∞ t = 1 X t <
[Equation]
, a . s .
B. P ROOF OF L EMMA
Recall the mSGD algorithm in equation
[Equation]
Equation equation 18 is equivalent to
[Equation]
Under Assumption 2 2)
[Equation]
it follows from Lemma 3 that
[Equation]
In this subsection
[Equation]
we just use the right side of equation 20.
The left side will be used in the next subsection.
Consider ∇ θ t g
( θ t ) T v t in the following
[Equation]
Recursively applying the above equation yields
[Equation]
By substituting the above equation into equation 20 and noting − ( ∇ θ i g ( θ
[Equation]
) − ∇
[Equation]
g ( θ i −
[Equation] [Equation]
v
[Equation]
≤
[Equation]
∇ θ i g
[Equation]
θ i ) − ∇
[Equation]
i
[Equation]
g
[Equation]
θ i − 1 ) (cid:13)(cid:13) (cid:107) v
i
[Equation]
c (cid:107) v t −
[Equation]
2 , we obtain
[Equation]
Denote Z ( t ) :
= ∏ + ∞ k = t ( 1 + M 0 ε 2 k ) , where
[Equation]
where δ > 0 is a constant and M is introduced in Assumption 2 4).
Here we deﬁne M 0 , Z ( t ) and δ to facilitate the proof equation 33.
From Assumption 3, it holds that ∑ + ∞ t = 1 ε 2 t < + ∞ .
Thus, ∑ + ∞ t = 1 M 0 ε 2 t < + ∞ .
From a general inequality ln ( 1 + x ) ≤ x for x > − 1, we get
[Equation]
which means that Z ( t ) is uniformly upper bounded.
Then multiplying Z ( t + 1 ) on both sides of equation 22 and taking the mathematical expectation yield
[Equation] [Equation]
where the ﬁrst and second inequalities are respectively due to Z
( i + 1 ) <
Z ( i ) and Z ( t + 1 ) ∑ t i
[Equation]
i
[Equation]
i g
[Equation]
θ
[Equation]
2 > 0 .
Next, we aim to analyze c
∑ ti = 1 α t − i Z
( i + 1 ) E (cid:0) (cid:107) v i (cid:107) 2 (cid:1) in equation 23.
It is proved in Appendix B. that
[Equation]
Taking a weighted sum of equation 24 yields
[Equation]
We derive that
[Equation]
Substituting equation 26–equation 28 into equation 23 yields
[Equation]
Construct a sequence { V n } as follows
[Equation]
By substituting equation 29 into equation 30 following the way of equation 26–equation 28, we have
[Equation]
where f ( n − t ) is deﬁned as follows
[Equation]
Move the last term to the left-hand side of the above inequality, then we have n
[Equation]
Because of α <
[Equation]
it holds that f ( n − t ) > 0 and e ( n ) t + 1 > 1.
It follows from Assumption 3 that
[Equation]
) where
[Equation]
Calculate f ( n − t )
[Equation]
then we obtain
[Equation]
It holds that n
[Equation] [Equation]
∑
[Equation] [Equation]
[Equation]
Substituting equation 33 and equation 34 into equation 31 yields
[Equation]
According to the deﬁnition of Z ( t )
[Equation]
we have
[Equation]
Let
[Equation] [Equation]
and it follows from equation 35 that
[Equation]
Denote p = exp (cid:26)
[Equation]
k (cid:27) .
By taking the summation of equation
[Equation]
we obtain
[Equation]
Deﬁne
[Equation]
It follows from the relationship between g ( θ n + 1 ) and F n that
[Equation]
which leads to E (cid:0) g ( θ n ) (cid:1) <
T ( θ 1 , v 0 )
[Equation]
B. P ROOF OF EQUATION
We consider
[Equation] [Equation]
where the ﬁrst inequality is due to ε i ≤ ε i − 1 in Assumption
[Equation]
and the last equality is from equation 19.
Since ξ i and θ i are independent
[Equation]
taking the mathematical expectation of equation 37 and noting that
[Equation]
yield
[Equation]
Moreover, it holds that
[Equation]
Combining equation 38 and equation
[Equation]
we get
[Equation]
Since ε i → 0, given any δ > 0, there is an integer i 0 ≥ 0, such that for i ≥ i 0 , 1 − c ε i > α 1 − δ ( δ > 0 ) .
Since i 0 is ﬁnite, without loss of generality, we assume i
= 0 for convenience, i.e., 1 − c ε i > α 1 − δ ( δ > 0 )
( i ≥ 1 ) .
Thus, we have
[Equation] [Equation]
Multiplying both sides of equation 41 by Z ( i + 1 ) , and noticing that Z ( i )
>
Z ( i + 1 ) , we have
[Equation]
Then equation 24 is obtained by recursively applying the above inequality
[Equation]
B. P ROOF OF L EMMA
From equation 40, we have
[Equation] [Equation]
It follows from Assumption 2 5) and Lemma 1 that
[Equation]
Because of ∑ nt = 1 ε 2 t < + ∞
[Equation]
there is a scalar ¯ M > 0 such that for ∀ n
[Equation]
Then it follows from equation 42 that
[Equation]
where K is a positive scalar.
Since ε n → 0 when n is large enough, it holds that 15 ( 1 − α 2 ) < 1 − α 2 − c ε n .
Without loss of generality, assume 15 ( 1 − α 2 ) < 1 − α 2 − c ε 2 n for n ≥ 0, so
∑ nt = 1 E (cid:0) (cid:107) v t (cid:107) 2 (cid:1) <
10 K 1 − α 2 < + ∞ .
By Lemma 6, we obtain ∑ nt = 1 (cid:107) v t (cid:107) 2 < + ∞ .
B. P ROOF OF L EMMA
Through Taylor expansion, we derive that
[Equation]
where θ ζ t means a point between θ t and θ t + 1 .
Substituting equation 21 into equation 43 yields
[Equation]
It follows that
[Equation]
Take the mathematical expectation of equation
[Equation]
and notice Assumption 2 3)
[Equation]
then we have
[Equation]
From Lemma
[Equation]
it follows that for some positive constant Q ,
[Equation]
Hence,
[Equation]
As a result, n
[Equation]
( θ t ) (cid:107) 2 (cid:1) <
Q + E (cid:0) g ( θ 1 ) (cid:1) + α 1 − α (cid:12)(cid:12)(cid:12)(cid:12) ∇ θ 1 g ( θ 1 ) T v(cid:12) (cid:12)(cid:12)(cid:12) + 1 − α ε 1 E (cid:0) (cid:107) ∇ θ 1 g ( θ 1 ) (cid:107) 2 (cid:1) − E (cid:0) g
( θ n + 1 ) (cid:1) <
+ ∞ .
From Lemma 6, we have ∑ nt = 1 ε t (cid:107) ∇ θ t g ( θ t ) (cid:107) 2 < + ∞ a .
s ..
B. P ROOF OF L EMMA
We divide the proof into three steps.
The ﬁrst step is to prove
[Equation]
Suppose the above conclusion does not hold, i.e.,
[Equation]
where s is a random variable depending on sample paths.
Then it holds that
[Equation]
From equation 14 and lim n → + ∞ ζ n = ζ < ∞ a .
s .
, if (cid:13)(cid:13) ∇
[Equation]
n g ( θ n ) (cid:13)(cid:13) 2 > 14 s 2 , it holds that
[Equation]
It follows that
[Equation]
As a result, lim n → + ∞ g ( θ n ) = − ∞ , meaning a contradiction.
Hence,
[Equation] [Equation]
Step 2 is to prove that the set { θ n } has an accumulation point contained in J with probability one.
From equation 45, we have ∀ ε >
[Equation]
Since (cid:107) ∇ θ g ( θ ) (cid:107) 2 is continuous, ∀ δ > 0, it holds that
[Equation]
In addition, under the given conditions, J is a closed set.
It holds that
[Equation]
For convenience, let θ ∗ n : = argmin θ ∈ J (cid:107) θ n − θ (cid:107) 2 , then
[Equation] [Equation]
It follows that
[Equation]
where S N is the set of all inﬁnite subsequences of N .
Since { θ ∗ m i } ⊂ J , { θ ∗ m i } is bounded.
From the accumulative point principle, the event
[Equation]
is a deterministic event, i.e., being true for every sample path.
Thus
[Equation]
is a deterministic event as well, where S ( i ) N is the set of all inﬁnite subsequences of { m i } .
Due to the arbitrariness of { m i } , we get that
[Equation]
is a deterministic event.
Therefore,
[Equation]
[Table]
[Equation]
[Table]
[Equation]
(cid:19)(cid:19)(cid:33)
Combine (47) and (48), then we get that
P (cid:32)(cid:18) (cid:92) { m ki }∈
[Equation]
N (cid:91) { m
[Equation]
}∈ S ( i
[Equation]
N (cid:92) i , j ≥(cid:18)(cid:13)(cid:13)
θ ∗ m ki − θ ∗ m kj (cid:13)(cid:13) < √ δ (cid:19)(cid:19) (cid:92) (cid:18) (cid:91) { m i
}∈ S N (cid:18)
+ ∞ (cid:92) i =(cid:13)(cid:13)
θ m i − θ ∗ m i (cid:13)(cid:13) < √ δ (cid:19)(cid:19)(cid:33) = 1 .
So
P (cid:32) (cid:91) { m i }∈ S N (cid:92) i , j ≥(cid:18)(cid:13)(cid:13)
θ ∗ m i − θ ∗ m j (cid:13)(cid:13) < √ δ (cid:19) (cid:92) (cid:18)(cid:13)(cid:13) θ m i − θ ∗ m i (cid:13)(cid:13) < √ δ (cid:19) (cid:92) (cid:18)(cid:13)(cid:13) θ m j − θ ∗ m j (cid:13)(cid:13) < √ δ
(cid:19)(cid:33)
= 1 .
Thus it holds that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
This means we can ﬁnd two ”same” convergent subsequences { θ k n } and { θ ∗ k n } with probability one.
Because J is a closed set and { θ ∗ k n } ⊂ J , we know that there exists θ
(cid:48)(cid:48) ∈
J such that
[Equation]
This indicates that we can ﬁnd an accumlative point of { θ n } in J with probability one.
Denote the connected component containing θ (cid:48)(cid:48) by J ∗ .
Step 3 is to prove that there is a connected component J ∗ of J such that θ n → J ∗ a.s.
From equation 50, we have
[Equation]
Since g ( θ ) is continuous, we have
[Equation]
Since (cid:8) g ( θ n ) (cid:9) converges a.s., let M be the limit, i.e., lim n → + ∞ g ( θ n ) =
M a .
s .
Hence g ( θ (cid:48)(cid:48) ) =
M , and g ( θ )
=
M for all θ ∈ J ∗ .
Deﬁne A
= { θ | g ( θ ) =
M } .
From g ( θ n ) → M a.s. and the continuity of g ( θ ) , it holds that θ n → A a.s.
Obviously, J ∗ ⊆
A .
If J ∗ =
A , then the conclusion follows.
Now assume J ∗ (cid:36) A .
From the deﬁnition of J ∗ , it follows that there are two disjoint open sets V and H such that A ⊂ H (cid:83) V , J ∗ ⊂ V , A / J ∗
⊂ H , H (cid:48) (cid:84) V (cid:48)
= /
( H (cid:48) and V (cid:48) are closures of H and V ).
So we just need to prove that θ n → V a.s.
A contradiction argument is to be used.
Suppose that { θ n } has accumulation points in V and H simultaneously with a probability larger than zero.
That is,
[Equation]
Expanding equation 51, we get
[Equation]
Note that the event
[Equation]
can be written as
[Equation]
where { α i }∩{ β i } = /0.
The event
[Equation]
means ∀ t ≥ r , θ t must belong to one of H and V .
So { α i }∪{ β i } must include the set { r , r + 1 , r + 2 ,... } .
Denote { α i } and { β i } as { α ( r ) i } and { β ( r ) i } , and then equation 52 can be written as
[Equation]
So we get
[Equation]
It follows from V (cid:48) (cid:84) H (cid:48)
= /
that
[Equation]
where s is a constant.
By Lemma 7, it holds that ∀ s >
[Equation]
Now we consider { α ( r ) i + 1 } .
In fact, (cid:8) α ( r ) i + 1 (cid:9) and (cid:8) β ( r ) i (cid:9) have a ﬁnite number of identical elements with probability one.
If not, then
[Equation]
By equation 54, it follows that
[Equation]
>
[Equation]
, which however contradicts with equation 55.
Hence (cid:8) α ( r ) i + 1 (cid:9) and (cid:8) β ( r ) i (cid:9) have a ﬁnite number of identical elements with probability one.
From (cid:8) α
( r ) i (cid:9) (cid:83) (cid:8) β
( r ) i (cid:9) ⊃
{ r , r + 1 , r + 2 ,... } , we know that
[Equation]
Hence (cid:8) β ( r ) i (cid:9) is a ﬁnite sequence with probability one, otherwise it contradicts with equation 53.
Thus, the assumption does not hold, and θ n → V a.s.
Furthermore, θ n → J ∗ .
Therefore, there is a connected component J ∗ of J such that
[Equation]
B. P ROOF OF T HEOREM
First of all we aim to prove that g ( θ n + 1 ) is convergent almost surely.
Divide equation 44 into four parts as follows.
[Equation]
Due to α <
[Equation]
α n is tending to zero
[Equation]
which ensures the convergence of part ( A ) .
For ( C )
[Equation]
we consider the absolute value of ∑ mt = n .
It follows from Assumption 2 3) that
[Equation]
≤ m ∑
[Equation]
Through Lemma 7, we get ∑ mt = n (cid:107) v t (cid:107) 2 → 0 a . s . , leading to (cid:12)(cid:12)(cid:12)(cid:12) m
∑ t = n
[Equation]
→
[Equation]
a .
s ..
Through Cauchy (cid:48) s test for convergence , we know that ( C ) is convergent almost surely.
By using the same function, it holds that ( D ) is convergent almost surely.
For ( B ) , we have ( B ) =
n ∑ t =1 − α n − t
+1 − α ε
[Equation]
∇
[Equation]
g (
[Equation]
t , ξ t ) =
n ∑ t =1 − α
[Equation]
t +1 − α
[Equation]
n
[Equation] [Equation]
From Lemma 7, it follows that ( C ) is convergent almost surely.
For ( B ) , it holds that
[Equation]
B
[Equation]
By Lemma
[Equation]
we know
[Equation]
From Lemmas 5 and 8 it follows that
[Equation]
is convergent a
[Equation]
s. Thus ( B ) is convergent a.s.
[Equation]
and g ( θ n + 1 ) is convergent a.s.. Substituting equation 59 into equation 57 leads to
[Equation]
where { ζ (cid:48) n } is deﬁned as follows
[Equation]
we know { ζ (cid:48) n } is convergent a
[Equation]
s. It follows from Lemma 9 that there exists a connected component
J ∗ of J such that lim n → ∞ d ( θ n , J ∗ ) = 0 .
B. P ROOF OF T HEOREM
First of all we can get that
[Equation] [Equation]
From Theorem
[Equation]
it follows that g ( θ n ) is convergent a.s., and it is orbitally convergent to g i ( i = 1 , 2 ,..., N ) a.s
Then it holds that
[Equation]
where
[Equation]
For convenient, we let g ∗ =
∑ ∞ i =
I
i g i .
Then we make some transformation on equation
[Equation]
[Equation]
Then we make some transformations, take absolute values, take the mathematical expectation
[Equation]
and use same techniques in Theorem 1.
It follows that F (cid:48)
n − F (cid:48)
n − 1 ≤ P n − 1 − Q n − 1 , where P n − 1 , Q n − 1 , and
F (cid:48) n are deﬁned as follows
[Equation]
where g ∗ = inf θ ∈ R N g
( θ )
[Equation]
Then we have
[Equation]
Derive Q n / ε n + 1 F (cid:48) n as follows
[Equation]
It follows that
[Equation]
From equation
[Equation]
we have
[Equation]
where C α = 2 c ( 2 − α )
α − δ ( 1 − α δ )( 1 − α ( 2 − α )) .
Substituting equation 65 into equation 64 yields
[Equation]
[Equation]
Then we proceed with the proof under two different cases, namely,
[Equation]
First
[Equation]
if ∑ nt = 1 (cid:0) 1 2 − α (cid:1) − t E (cid:0) (cid:107) ∇ θ t + 1 g ( θ t + 1 ) (cid:107) 2 (cid:1) =
+ ∞ (the proof for this condition is up to equation 76).
It follows from Assumption 4 2) and the uniform convergence and O (cid:48) stolz theorem that liminf
[Equation]
where p = exp (cid:26)
[Equation]
= 1 M ε 2 k
[Equation]
By using O’stolz theorem on Q n / F (cid:48) n , it follows that
[Equation]
So we conclude that ∃ n 0 ∈ N + , such that ∀ n ≥ n 0 ,
[Equation]
From equation
[Equation]
it follows that
[Equation]
Using the inequality ln ( 1 + x ) ≤ x and equation
[Equation]
we get
[Equation]
where k is a constant.
It follows that
[Equation]
From equation 62, we have
[Equation]
where ¯ p = 1 ( 1 − α ) 2 ε 1 E (cid:0) (cid:107) ∇ θ 1 g ( θ 1 ) (cid:107) 2 (cid:1) + Z ( 1 ) L E (cid:0) g ( θ 1 ) (cid:1) and ¯ q = c
α δ + 1 ( 2 − α ) 2 ( 1 − α )( 1 − α δ
) Z ( 1 ) E
[Equation]
v 0 (cid:107) 2 (cid:1) .
Substituting equation 72 into equation 71 and noting equation 68 yield
[Equation]
where c 0 is a positive constant.
It follows that n −
[Equation]
Combine equation 69, equation 70 and equation 74, then we get
[Equation]
In addition
[Equation]
we have
[Equation]
If ∑ nt = 1 (cid:16) 1 2 − α (cid:17) − t E (cid:0) (cid:107) ∇ θ t + 1 g
( θ t + 1 ) (cid:107) 2 (cid:1) <
+ ∞ , it holds that
[Equation]
that is
[Equation]
Under Assumption 4 2), we have limsup n
[Equation]
It follows that
[Equation]
Now we compare (cid:0) 1 2 − α (cid:1) n with e − ∑ ni =s
ε
i p ( 1 − α ) 2 .
It holds that
[Equation]
It follows from Assumption 3 that ε n → 0, indicating ∑ ni =s
ε i p ( 1 − α ) 2 − n ln ( 2 − α ) < 0 when n is sufﬁciently large.
Thus, we have
[Equation]
Then it holds that
[Equation]
Combining equation 77 and equation 76 leads to
[Equation]
if
[Equation]
The above bound holds trivially if ∑ nt = 1 (cid:0) 1 2 − α (cid:1) − t E (cid:0) (cid:107) ∇ θ t + 1 g ( θ t + 1 ) (cid:107) 2 (cid:1) =
+ ∞ .
It follows from Lemma 4 that
[Table]
C C ONVERGENCE OF A DA G RAD
The following lemmas are used for the proof of Theorem 3.
Lemma 10 Suppose f ( x ) ∈ C 1 ( x ∈ R N ) with f ( x ) > − ∞ and its gradient satisfying the following
Lipschitz condition
[Equation]
then ∀ x 0 ∈ R N , there is
[Equation]
, where f ∗ = inf x ∈ R N f
( x )
Lemma 11 Suppose { θ n } is a sequence generated by AdaGrad in equation 5, and Assumptionsand 5 hold.
If (cid:13)(cid:13) ∇ θ n g ( θ ) (cid:13)(cid:13) 2 > a where a is given in Assumption 5 3), then for any n ∈ N + , θ 1 ∈ R N , and ε ∈ ( 0 , 12 ) , it holds that
[Equation]
where
[Equation]
Lemma 12 Suppose { θ n } is a sequence generated by AdaGrad in equation
[Equation]
and Assumptionsand 5 hold.
If (cid:13)(cid:13) ∇ θ n g ( θ ) (cid:13)(cid:13) 2 ≤ a where a is given in Assumption 5 3), then for any n ∈ N + , θ 1 ∈ R N , and ε ∈ ( 0 , 1 2 ) , it holds that
[Equation]
(cid:13)(cid:13)
[Equation]
[Equation]
θ
[Equation] [Equation]
[Equation] [Equation]
S
[Equation]
−
[Equation]
−
S
[Equation] [Equation]
n
[Equation]
where
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
Lemma 13 Suppose { θ n } is a sequence generated by AdaGrad in equation 5, and Assumptionsand 5 hold.
Then ∀ n ∈ N + , ∀ θ 1 ∈ R N ∀ ε ∈ ( 0 , 12 ) , it holds that
[Equation]
Lemma 14 Suppose { θ n } is a sequence generated by AdaGrad in equation 5, and Assumptionsand 5 hold.
Then ∀ n ∈ N + ∀ θ 1 ∈ R N ∀ ε ∈ ( 0 , 12 ) , ∃ ζ < + ∞ , it holds that
[Equation]
which g ∗ = inf θ ∈ R N g
( θ ) .
Lemma 15 Suppose { θ n } is a sequence generated by AdaGrad in equation 5 subject to S n =
∑ nk = 1 (cid:107) ∇ θ k g
( θ k , ξ k ) (cid:107)
=
+ ∞ a .
s . .
Under Assumptions 1 and 5, ∀ n ∈ N + ∀ θ 1 ∈ R N , ∀ ε 0 ∈ ( 0 , 38 ) , it holds that
[Equation]
Lemma 16 Suppose that { X n } ∈ R N is a vector sequence and f ( x ) ∈ C 1 is a monotonically nonincreasing non-negative function with (cid:82) + ∞ a f ( x ) dx < ∞ ( ∀ a > 0 ).
Then ∀ N ∈ N + , it holds that
[Equation]
X 1 (cid:107)f ( x ) dx < N ∑ n =(cid:107)
X n (cid:107) 2 f (cid:18)
n − 1 ∑ k =(cid:107)
X k (cid:107) 2 (cid:19) .
C. P ROOF O UTLINE OF T HEOREM
Like the proof of mSGD, the proof for AdaGrad is also in light of the Lyapunov method.
We aim to prove ∇ g ( θ n ) → 0 a
[Equation]
s . , and then to get θ n → J ∗ a
[Equation]
s .
The key step to prove convergence of AdaGrad is to show
[Equation]
but the learning rate of AdaGrad is a random variable and it is not conditionally independent of ∇ θ n g ( θ n , ξ n ) , meaning that
[Equation]
In the following
[Equation]
we provide the proof outline of Theorem 3.
Step 1: This step is to ensure
[Equation]
We are able to obtain the following equation
[Equation]
[Table]
[Equation]
For ( K ) , due to S n − 1 ≤ S n , it follows that
[Equation]
Note that S n − 1 is conditional independent on ∇ θ n g ( θ n , ξ n ) , thus it holds that
[Equation]
Next we prove ( K ) can be controlled by ( L ) according to Lemmas 11 and 12.
These two lemmas deal with the cases of (cid:107) ∇ θ n g ( θ n ) (cid:107) ≤ a and (cid:107) ∇ θ n g ( θ n ) (cid:107) > a , respectively.
In these two lemmas, we introduce a constant ε for the reason as stated in Step 2 in the following.
Step 2: Through Lemmas 11 and 12, we obtain that
[Equation]
From Lemma 16, we obtain Lemma 13, stating that ∑ + ∞ k = 3 (cid:107) ∇ k g ( θ k , ξ k ) (cid:107) 2 / S 1 + 2 ε k < + ∞ .
Note that if we do not introduce ε , the term ∑ + ∞ k = 3 (cid:107) ∇ k g
( θ k , ξ k ) (cid:107) 2 / S 1 + 2 ε k will become ∑ + ∞ k = 3 (cid:107) ∇ k g
( θ k , ξ k ) (cid:107) 2 / S k =
O ( ln S n ) , and this term may not be bounded.
Step 3: Lemma 13 ensures that ∇ θ n g ( θ n ) has a subsequence satisfying ∇ θ kn g ( θ k n ) → 0 a.s.
By using the recursion formula g ( θ n + 1 ) − g ( θ n ) ≤ 1 / S n + P n , where ∑ + ∞ n = 1 P n < + ∞ a.s., and Lemma 4, we obtain ∇ θ n g ( θ n ) → 0 a.s.
Consequently, θ n → J ∗ .
C. P ROOF OF L EMMA
For ∀ x ∈ R N , we deﬁne function
[Equation]
where x (cid:48) is a constant point
such that x (cid:48) − x is parallel to ∇ f ( x ) .
By taking the derivative, we obtain
[Equation]
Through the Lipschitz condition of ∇ f ( x )
[Equation]
we get ∀ t
[Equation]
t
[Equation]
x (cid:48) −
[Equation]
(cid:107) (cid:19) −
[Equation]
x
[Equation]
t x (cid:48)− x (cid:107) x
(cid:48)− x (cid:107) f
(cid:18)
[Equation]
(cid:48)
[Equation]
x (cid:107) x (cid:48) − x (cid:107) (cid:19)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) x (cid:48)
[Equation]
x (cid:107)
[Equation]
(cid:48)
[Equation]
(cid:107) (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
So g (cid:48) ( t ) satisﬁes the Lipschitz condition, and we have inf t ∈ R g ( t ) ≥ inf x ∈ R N f ( x ) > − ∞ .
Let g ∗ =
inf x ∈ R g ( x ) , then it holds that for ∀ t 0 ∈ R ,
[Equation]
By using the Newton-Leibniz’s formula, we get that
[Equation]
Through the Lipschitz condition of g (cid:48) , we get that
[Equation]
Then we take a special value of t 0 .
Let t 0 = − g (cid:48) ( 0 ) /
c , then we get
[Equation]
Substituting equation 80 into equation 79, we get
[Equation]
Due to g ∗ ≥ f ∗ and (cid:0) g (cid:48) ( 0 ) (cid:1)
= (cid:107) ∇ f ( x ) (cid:107) 2 , it follows that
[Equation]
C. P ROOF OF L EMMA
First of all, it follows from Lemma 3 that
[Equation]
where
[Equation]
Note that
[Equation] [Equation]
where M = M (cid:48) + 2 and M (cid:48) is deﬁned in Assumption 5 3).
Substitute equation 82 into equation
[Equation]
then we get that
[Equation]
(cid:13)(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)
[Equation]
(cid:13)(cid:13)
[Equation]
S
[Equation]
Due to S n ≥ S n − 1 , it follows that
[Equation]
Substitute equation 84 into equation 83, then we have
[Equation]
Notice that α
[Equation]
[Equation]
Substitute equation 86 into equation
[Equation]
and divide both sides of the inequality by S ε n ( ε < 12 ) , then we get
[Equation]
θ n g ( θ n , ξ n ) (cid:1) .
Notice that g ( θ n + 1 ) S ε n > g ( θ n + 1 ) S ε n + 1 , then we obtain
[Equation]
Rearrange the above inequality, then it holds that
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
X
[Equation]
) n is deﬁned as follow
[Equation]
Due to (cid:107) ∇ θ n g ( θ n ) (cid:107) 2 > a , we have
[Equation]
Moreover, using the Taylor formula, we obtain
[Equation]
g ( θ n − 1 ) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13)+ 2 (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13)(cid:13)(cid:13) ∇ θ n g ( θ n ) − ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13) + (cid:13)(cid:13) ∇ θ n g ( θ n ) − ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13) 2 .
Under Assumption 5 3), we get that (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 ≤ (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13)+ 2 (cid:13) (cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13) (cid:13) (cid:13) (cid:13) ∇ θ n g ( θ n ) − ∇ θ n − 1 g
θ n − 1 ) (cid:13) (cid:13) + (cid:13) (cid:13) ∇ θ n g ( θ n ) − ∇ θ n − 1 g ( θ n − 1 ) (cid:13) (cid:13)≤ (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13)
+ 2 α 0 c √ S n − 1 (cid:13)(cid:13) ∇ θ n − 1 g
( θ n − 1 ) (cid:13)(cid:13)(cid:13)(cid:13) ∇ θ n − 1 g
( θ n − 1 , ξ n − 1 ) (cid:13)(cid:13)
+ c 2 α 20 (cid:13)(cid:13) ∇ θ n − 1 g
( θ n − 1 , ξ n − 1 )
[Equation]
S n
[Equation]
From inequality 2 a T b ≤ λ (cid:107) a (cid:107) 2 + 1 λ (cid:107) b (cid:107) 2 ( λ > 0 )
[Equation]
it follows that
[Equation]
By substituting equation 89 into equation
[Equation]
we have
[Equation] [Equation]
Divide both sides of equation 92 by S 12 + ε n − 1 , and notice M − 1 4 M − 3 > 15 from M > 2, then it holds that
[Equation]
93) where
Y (
[Equation]
) n
[Equation]
Making some simple transformations on equation 93 leads to α
[Equation] [Equation]
[Equation]
Substitute equation 94 into equation 88, then we get
[Equation]
It follows that
[Equation] [Equation]
[Equation] [Equation]
[Equation] [Equation]
(cid:13)(cid:13)
[Equation]
[Equation] [Equation]
n −
[Equation]
n
[Equation] [Equation]
ε
[Equation] [Equation]
C. P ROOF OF L EMMA
[Equation]
Due to S n + 1 ≥ S n , it holds that
[Equation]
Then we make some transformations to obtain that
[Equation] [Equation]
where
[Equation] [Equation]
Due to
[Equation]
∇
[Equation]
a , we get
[Equation]
Substitute it into equation 100, then we get g
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We make some transformations on
[Table]
to obtain that
[Table]
[Equation]
Then we use inequality 2 a T b ≤ λ (cid:107) a (cid:107) 2 + 1 λ (cid:107) b (cid:107) 2 ( λ > 0 ) on equation 90 to get
[Equation]
Divide both sides of equation 105 by S 12 + ε n − 1 and notice S n − 2 ≤ S n − 1 ≤ S n , then we have
[Equation]
Then we calculate α 0 2 ( M + 1 ) equation 106 + equation
[Equation]
[Table]
the then we have
[Table]
[Equation]
Substitute equation 107 into equation 103, then we have
[Equation]
C. P ROOF OF L EMMA
First of all, it holds that
[Equation]
I (cid:16)(cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 ≤ a (cid:17) ∈ F n − 1 is the indicator function such that
[Equation]
For convenient, we abbreviate I (cid:16)(cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 ≤ a (cid:17) as I ≤ a n and I (cid:16)(cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 > a (cid:17) as I > a n in the following.
[Table]
[Equation]
Through Lemma
[Equation]
we get
[Equation] [Equation]
Calculate equation 109 + equation 110, then it holds that
[Equation]
( θ n + 1 )
S ε n + 1 − g ( θ n )
S ε n (cid:19)
+ I > a n (cid:18) g
( θ n + 1 )
S ε n + 1 − g ( θ n )
S ε n (cid:19) ≤ α20 ( I ≤ a n + I > a n ) (cid:32)(cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13)
S 12 + ε n −− (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13)
S 12 + ε n −(cid:33) + α2 ( M + 1 )( I ≤ a n + I > a n ) (cid:32)(cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13)
S 12 + ε n −− (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13)
S 12 + ε n (cid:33) − α (cid:0) I ≤ a n + I > a n (cid:1)(cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13)
S 1 2 + ε n −+ 4 M 2 α 30 c 2 I > a n (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) , ξ n − 1 (cid:13)(cid:13)
S 3 2 + ε n −+ c α2 I > a n (cid:13)(cid:13) ∇ θ n g ( θ n , ξ n ) (cid:13)(cid:13)
S 1 + ε n + 5 α 30 c 2 ( M + 1 ) 2 I ≤ a n (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 , ξ n − 1 ) (cid:13)(cid:13)
[Equation]
Notice I ≤ a n ≤ 1, then we get
[Table]
Substitute equation 112 into equation 111, then we get
[Table]
[Equation]
We make a summation of equation 113 to get
[Equation]
S 12 + ε k (cid:33) − α n ∑ k =(cid:13)(cid:13) ∇ θ k g
( θ k ) (cid:13)(cid:13)
S 1 2 + ε k −+ 4 M 2 α 3 0 c 2 n ∑ k =
I > a k (cid:13)(cid:13) ∇ θ k − 1 g ( θ k − 1 ) , ξ k − 1 (cid:13)(cid:13)
S 3 2 + ε k −+ c
α n ∑ k = 3 I > a k (cid:13)(cid:13) ∇ θ k g
( θ k , ξ k ) (cid:13)(cid:13)
S 1 + ε k + 5 α 30 c 2 ( M + 1 ) 2 n ∑ k =
I ≤ a k (cid:13)(cid:13) ∇ θ k − 1 g ( θ k − 1 , ξ k − 1 ) (cid:13)(cid:13)
S 1 + ε k −+ n ∑ k =
I ≤ a k ( M + 1 ) α 30 c2 S 3 2 + ε k −(cid:13)(cid:13) ∇ θ k − 1 g ( θ k − 1 , ξ k − 1 ) (
cid:13)(cid:13)+ α 0 a ( M + 1 ) n ∑ k =(cid:32)
S 1 2 + ε k −−
S 1 2 + ε k (cid:33)
+ c
α n ∑ k = 3 I ≤ a k (cid:13)(cid:13) ∇ θ k g
( θ k , ξ k ) (cid:13)(cid:13)
S 1 + ε k + n ∑ k =(cid:16) I ≤ a k A ( ε ) k
+ I ≤ a k B ( ε ) k
+ I > a k X ( ε ) k
+ I > a k Y ( ε ) k
(cid:17) .
It follows from Lemma 16 that
[Equation]
and
[Equation]
and
[Equation]
Due to I ≤ a k ≤
and I > a k ≤ 1, we get that
[Equation]
Substituting equation 118 into equation 114 leads to
[Equation]
It is obvious that α (cid:32)(cid:13)(cid:13)
[Equation]
It follows that
[Equation]
Note that { I ≤ a k A k } , { I ≤ a k B k } , { I > a k X k } and { I > a k Y k } are all martingale difference sequences, thus it follows that E (cid:18) ∑ n k = 1 (cid:16) I ≤ a k A k
+ I ≤ a k B k
+ I > a k X k
+ I > a k Y k (cid:17)(cid:19) =
Then we calculate mathematical
expectation on equation
[Equation]
That is
[Equation]
From Lemma
[Equation]
it holds that
[Equation]
C. P ROOF OF L EMMA
It follows from equation 119 that
[Equation]
From equation 101, we obtain
[Equation]
2 − ∇ θ n g ( θ n ) T ∇ θ n g ( θ n , ξ n ) (cid:17)(cid:33) .
With inequality ( a + b ) 2 ≤ 2 ( a 2 + b 2 ) , 2 a T b ≤ a 2 + b 2 ( a , b > 0 ) and equation 120, we get
E (cid:32)
I ≤ a k α
S
[Equation] [Equation]
≤ 2 E (cid:32)
[Equation]
∇
[Equation]
≤ 2 ( M
[Equation]
2 ) E (cid:32)
α 20 a
[Equation]
∇
[Equation]
g
[Equation] [Equation]
S
[Equation]
n
[Equation]
(cid:33) ≤ 2 ( M + 2 )
α
[Equation]
a E (cid:32)
α 20 a (cid:13)(cid:13) ∇
[Equation]
n g ( θ n ) (cid:13)(cid:13)
S
[Equation]
ε n −(cid:33) <
40 ( M + 2 )
α 0 a (cid:18)
[Equation]
It follows from Lemma 5 that ∑ nk = 3 I ≤ a k A ( ε ) k is convergent a.s.
Similarly, ∑ nk = 3 I ≤ a k B ( ε ) k , ∑ nk =
I > a k X ( ε ) k
and ∑ nk = 3 I > a k Y ( ε ) k are both convergent a.s.
It follows that
[Equation]
For convenience, let ξ = g ( θ 3 ) S ε 3 + L + ξ (cid:48) .
Thus, it holds that
[Equation]
C. P ROOF OF L EMMA
First of all, ∀ 0 < ε 0 < 38 , there is 0 < 43 ε 0 < 12 .
From Lemma 13, it follows that
[Equation]
It follows from equation 90 that
[Equation]
Divide both sides of equation 126 by S ε 0 n − 1 and notice that S n > S n − 1 > S n − 2 , then we get
[Equation]
∇
[Equation]
Note that 0 < 23 ε 0 < 1 2 , then it follows from Lemma 10 and Lemma 14 that
[Equation]
It follows that
[Equation]
Thus
[Equation]
we have
[Equation]
where
[Equation]
It follows that
[Equation]
By Lemma 16, we have that
[Equation]
Due to S n − 1 → + ∞ a . s .
, it follows that
[Equation]
From Lemma
[Equation]
we get
[Equation]
Combine equation 129 and equation 130, then
[Equation]
is divergent a.s.
From Lemma 13, we get that ∑ + ∞ n = 4 E (cid:0) (cid:107) K n − 1 (cid:107) 2 (cid:1) <
+ ∞ and K n − 1 is a martingale difference sequence.
Thus, ∑ + ∞ n = 2 K n − 1 is convergent a.s.. Combine equation 128, equation 131 and
Lemma 2, then we have
[Equation]
C. T HE PROOF OF T HEOREM
We consider the proof under two conditions, namely, S n < + ∞ a .
s .
or S n = + ∞ a .
s . .
First, if S n < + ∞ a . s . , from Lemma 13, we get that ∀ ε ∈ ( 0 , 12 ) , it holds that
[Equation]
Thus, we conclude that
[Equation]
Due to S n − 1 < + ∞ a .
s .
, we get
[Equation] [Equation]
Second, if S n = + ∞ a
[Equation]
s .
, let ε → 0 on equation 113, then it holds that
[Equation]
where
[Equation]
(cid:1) .
Make some transformations on (cid:13)(cid:13) ∇ θ n g ( θ n ) , ξ n (cid:13)(cid:13) 2 S n to obtain that
[Equation]
where the last inequality is from Assumption 5 5).
Let 0 < ε (cid:48) < 38 .
It follows from Lemma 15 that (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13)
S ε (cid:48) n −< δ < + ∞ .
Then we have M (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 S n −≤ M δ S 1 − ε (cid:48) n − 1 =
M δ S 1 2 + ε 1 n −(cid:0) ε 1 = 12 − ε (cid:48) ∈ ( 0 , 14 ) (cid:1) and (cid:13)(cid:13) ∇ θ n g ( θ n , ξ n ) (cid:13)(cid:13)
S n ≤ M δ + a
S 12 + ε 1 n +
S n −(cid:16)(cid:13)(cid:13) ∇ θ n g ( θ n , ξ n ) (cid:13)(cid:13) 2 − E (cid:16)(cid:13)(cid:13) ∇ θ n g ( θ n ) , ξ n (cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) F n − 1 (cid:17)(cid:17)
+( M δ + a ) (cid:32)
S 1 2 + ε 1 n −−
S 1 2 + ε 1 n (cid:33) .
Similarly, we have
[Equation]
Substitute equation 135 and equation 136 into equation 134, then we get
[Equation]
where
Q n = c
α
S n −
[Equation]
2 − E (cid:16)(cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) , ξ n − 1 (cid:13)(cid:13) 2 (cid:12)(cid:12)(cid:12) F n − 2 (cid:17)(cid:17)
+ I ≤ a n A n + I ≤ a n B n + I > a n X n
+
I > a n Y n .
P n = α 0 20 (cid:32)(cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13) 2 √
S n −− (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 √
S n −(cid:33) + α2 ( M + 1 ) (cid:32)(cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) (cid:13)(cid:13) 2 √
S n −− (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 √
S n (cid:33) + 4 M 2 α 30 c 2 (cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 ) , ξ n − 1 (cid:13)(cid:13)
S 3 2 n −+ ( M + 1 ) α 30 c2 S 3 2 n −(cid:13)(cid:13) ∇ θ n − 1 g ( θ n − 1 , ξ n − 1 ) (
cid:13)(cid:13)+ α 0 a ( M + 1 ) (cid:32) 1 √
S n − 1 − 1 √
S n (cid:33) + α 30 c 2 ( M + 1 ) 2 ( M δ + a ) (cid:32)
S 12 + ε 1 n −−
S 12 + ε 1 n (cid:33)
+ c
α 20 (
[Equation]
a
[Equation] [Equation]
[Equation]
It follows from S n → + ∞ a
[Table]
s .
that + ∞
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
n
[Equation]
=
[Equation] [Equation]
[Equation]
From Lemma 15, we get that lim
[Table]
S
[Table]
≤ lim
[Table]
( θ n ) (cid:13)(cid:13)
S ε 0 n −= 0 , and lim n → + ∞ (cid:13)(cid:13)
[Equation]
θ n
[Equation]
θ n
[Equation]
(cid:13)(cid:13) 2 √
S n
[Equation]
lim n → + ∞ (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13) 2 √
S n −≤ lim n → + ∞ (cid:13)(cid:13) ∇ θ n g ( θ n ) (cid:13)(cid:13)
S ε 0 n −= 0 .
It follows that
[Equation]
By Lemma 16, we have
[Equation]
Thus, ∑ + ∞ n = 2 P n is convergent.
In addition, it holds that
[Equation]
It follows from Lemma 15 that ∃ 0 < ε 0 <
[Equation]
such that
[Equation]
meaning that there is an almost surely bounded random variable
δ
[Equation]
such that
[Equation]
It follows that
[Equation]
Then we derive that +
[Equation]
From Lemma 5, ∑ + ∞ n = 3 I ≤ a n A n is convergent almost surely.
Similarly, we can prove other parts of ∑ + ∞ n = 3 Q n are also convergent almost surely.
Thus, ∑ + ∞ n = 3 Q n is convergent almost surely.
By summary, we get
[Equation]
almost surely
[Equation]
It is easy to ﬁnd that J i is a bounded closed set.
So ∀ ε > 0 we can construct an open cover H ( i )
ε
= { U ( θ , ε ) } ( θ ∈ J i ) of J i .
Through the HeineBorel theorem , we can get a ﬁnite open subcover { U ( θ k , ε ) ( k = 0 , 1 ,..., n ) from H ( i ) ε .
Then we assign U ( i )
ε
= (cid:83)
nk =
U ( θ k , ε ) .
We can get U ( i ) ε is a open set.
Under Assumption 5, J = { θ | ∇ θ g ( θ ) } has only ﬁnite connected components J 1 , J 2 ,..., J m .
So inf i (cid:54) =
j d ( J i , J j ) =
min i (cid:54) =
j d ( J i , J j ) .
Let δ
= min
i (cid:54) =
j d ( J i , J j ) .
It follows from Lemma 4 that ∃ ε 0 > 0, when d ( θ , J i ) <
ε 0 , there is
[Equation]
) where g i denotes g ( θ ) ( θ ∈ J i ) .
Let c = min { ε 0 , 0 / 4 } and construct U ( 1 ) c , U ( 2 ) c ,..., U ( m ) c .
It is obvious that ∀ U ( i ) c , U ( j ) c ( i (cid:54) =
j ) , d ( U ( i ) c , U ( j ) ) > δ 0 / 2, and (cid:107) ∇ θ g ( θ ) (cid:107) 2 ≤ 2 c | g ( θ ) − g
i | ( θ ∈ U ( i ) c ) .
Since J is a bounded set, ∃ N > 0, such that J ⊂ K
( K is the closure of U ( 0 , N ) ).
Then we construct a set M = K / (cid:83) mi
= 1 U ( i ) c .
Since U ( i ) c is a open set and K is a closed set, we conclude M is a closed set.
Since (cid:107) ∇ θ g ( θ ) (cid:107) is a continuous function, ∃ θ 0 ∈ M , (cid:107) ∇ θ 0 g ( θ 0 ) (cid:107) =
min θ ∈ M (cid:107) ∇ θ g ( θ ) (cid:107) .
Let r = (cid:107) ∇ θ 0 g ( θ 0 ) (cid:107) > 0.
Then we prove that ∀ u > 0 , θ ∈ K , ∃ δ > 0, if (cid:107) ∇ θ g ( θ ) (cid:107) <
δ , makes d ( θ , J ) <
u .
We prove it by contradiction.
Assume ∃ u 0 > 0, ∀ δ 1 > 0, ∃ θ δ 1 holds (cid:107) ∇ θ δ 1 g ( θ δ 1 ) (cid:107) <
δ 1 and d ( θ , J ) ≥ u 0 .
We make δ 1 = 1 , 1 / 2 , 1 / 3 ... , and we form a sequence { θ 1 / n } .
It is obvious that (cid:107) ∇ θ 1 / n g ( θ 1 / n ) → 0 (cid:107) .
Since { θ 1 / n } is bounded, through the Accumulation point theorem , there exists a convergent subsequence { θ 1 / k n } ⊂ { θ 1 / n } .
We deﬁned θ ( 0 ) =
lim n → + ∞ θ 1 / k n .
Through the continuity of d ( θ , J ) and (cid:107) ∇ θ g ( θ ) (cid:107) , we get d ( θ ( 0 ) ) ≥ u 0 and (cid:107) ∇ θ ( 0 )
g
( θ ( 0 ) )
(cid:107) = 0.
It is contradiction by the deﬁnition of J .
So ∀ u > 0 , θ ∈ K , ∃ δ > 0, (cid:107) ∇ θ g ( θ ) (cid:107) <
δ , makes d ( θ , J ) <
u ( ∃ ı, makes d ( θ , J ) < u ).
And furthermore, due to the continuity of g ( θ ) , we can get ∀ ε 1 > 0, ∃ δ (cid:48) > 0, if d ( θ , J i ) <
δ (cid:48) , there is | g ( θ ) − g
i | < ε 1 .
So combine these two consequences.
We can prove ∀ ε 1 > 0, ∃ b > 0, if θ ∈ U ( i ) c and (cid:107) ∇ θ g ( θ ) (cid:107) < b , there is (cid:12)(cid:12) g
( θ ) − g i (cid:12)(cid:12) <
ε 1 .
Through equation 121 and Lemma 6 we get there is a subsequence {(cid:107) ∇ θ k n g ( θ k n ) (cid:107) 2 } of {(cid:107) ∇ θ n g ( θ n ) (cid:107) 2 } which satisﬁes that
[Equation]
[Equation]
Next we aim to prove lim n → + ∞ (cid:107) ∇ θ n g ( θ n ) (cid:107) 2 = 0.
It is equivalent to prove that {(cid:107) ∇ n g ( θ n ) (cid:107) 2 } has no positive accumulation points, that is to say, ∀ e 0 > 0, there are only ﬁnite values of (cid:107) ∇ θ n g ( θ ) (cid:107) larger than e 0 .
And obviously, we just need to prove ∀ 0 <
e 0 < r , there are only ﬁnite values of (cid:107) ∇ θ n g ( θ ) (cid:107) larger than e .
We prove this by contradiction.
We suppose ∃ 0 <
e < a , making the set
S = {(cid:107) ∇ θ n g ( θ n ) (cid:107) 2 } be an inﬁnite set.
Then we assign ε 1 = e / 8 c and deﬁne o
= min { b , e / 4 } .
Due to equation 143, we get there exists a subsequence { θ p n } of { θ n } which satisﬁes (cid:107) ∇ θ pn g
( θ p n ) (cid:107) <
o .
We rank S as a subsequence {(cid:107) ∇ m n g ( θ m n ) (cid:107) 2 } of {(cid:107) ∇ n g ( θ n ) (cid:107) 2 } .
Then there is an inﬁnite subsequence {(cid:107) ∇ m in g ( θ m in ) (cid:107) 2 } of {(cid:107) ∇ m n g ( θ m n ) (cid:107) 2 } such that ∀ n ∈ N + , ∃ l , n p n ∈ ( m i l , m i l + 1 ) .
For convenient, we abbreviate { m i n } as { i n } .
And we construct another inﬁnite sequence { q n } as follows
[Equation]
> e (cid:9) .
Now we prove that ∃ N 0 , when q 2 n > N 0 , it has e < (cid:13)(cid:13) ∇ θ q 2 n g ( θ q 2 n ) (cid:13)(cid:13) <
r .
The left side is obvious (the deﬁnition of q 2 n ).
And for the right side, we know (cid:13) (cid:13) ∇ θ q 2 n − 1 g ( θ q 2 n − 1 ) (cid:13) (cid:13) ≤ e .
It follows from equation 5 that
[Equation]
Through previous consequences we can easily ﬁnd that
[Equation]
< + ∞ a .
s ..
Note that α 20 a / S n − 1 → 0 a . s . .
We conclude
[Equation] [Equation]
[Table]
, So
[Table]
N
[Table]
, when n > N 0 , there is (cid:12)(cid:12) (cid:107) ∇ θ n + 1 g ( θ n + 1 ) (cid:107) 2 − (cid:107) ∇ θ n g ( θ n ) (cid:107) (cid:12)(cid:12) <
r − e .
Then we can get that when q 2 n > N 0 + 1, there is
[Table]
∇
[Table]
q
[Table]
g ( θ
[Table]
n ) (cid:13)(cid:13) ≤ (cid:107) ∇ θ q 2 n − 1 g ( θ q
[Table]
1 ) (cid:107) + (cid:12)(cid:12) (cid:107) ∇ θ
[Table]
g
[Table]
θ
[Table] [Table]
−
[Table]
∇
[Table]
1 g ( θ
[Table]
2 n − 1 )
[Table]
≤ e + r − e = r .
That means that θ q 2 n ∈ (cid:83) mi = 1 U ( i ) τ , so we can prove ∃ i 0 , such that θ q 2 n ∈ U
( i 0 )
τ .
And due to (cid:107) ∇
[Table]
g
[Table]
θ n ) (cid:107) ≤ e < r ( n ∈ [ q 2 n − 1 , q 2 n )) , we get ∀ k ∈
[ q 2 n − 1 , q 2 n ) , ∃ i k , such that θ n ∈ U ( i k ) τ .
Due to (cid:107) θ n + 1 − θ n (cid:107) → 0 a .
s .
, we know i
= i k ( ∀ j ∈ [ q 2 n − 1 , q 2 n )) .
For convenient, we sign i
=
i q 2 n − 1 = ...
=
i q 2 n − 1 = i q 2 n .
And then we can conclude that (cid:107) ∇ θ g ( θ n ) (cid:107) 2 ≤ 2 c | g ( θ n ) − g
i q 2 n | ( n ∈ [ q 2 n − 1 , q 2 n ]) .
Due to locally sign-preserving property, we get (cid:107) ∇ θ g ( θ n ) (cid:107) 2 ≤ 2 c ( g ( θ n ) − g i q 2 n ) ( g ( θ n ) ≥
g i q 2 n ) or (cid:107) ∇ θ g ( θ n ) (cid:107) 2 ≤ − 2 c ( g ( θ n ) − g
i q 2 n ) ( g ( θ n ) ≤ g i q 2 n ) ( n ∈ [ q 2 n − 1 , q 2 n ]) .
Ways to dispose these two cases is same, so we just show how to prove the ﬁrst case.
We get e − o < (cid:13)(cid:13) ∇ θ q 2 n g ( θ q 2 n ) (cid:13)(cid:13) 2 − (cid:13)(cid:13) ∇ θ q 2 n − 1 g ( θ q 2 n − 1 ) (cid:13)(cid:13) 2 < 2 c (cid:0) g
( θ q 2 n ) − g i q 2 n (cid:1) − (cid:13)(cid:13) ∇ θ q 2 n − 1 g ( θ q 2 n − 1 ) (cid:13)(cid:13)= (cid:18)
[Equation]
q 2 n − q 2 n − 1 −∑ i =g
[Equation]
From equation 137, we obtain
[Equation]
So there is e − o <
[Equation]
which
[Equation]
Due to (cid:107) ∇ θ q 2 n − 1 g ( θ q 2 n − 1 ) (cid:107) 2 < o < b , so through equation 141, we get that g ( θ q 2 n − 1 ) − g
i 2 n − 1 < e / 8 c .
Substitute it into equation 146.
We get
[Equation]
Through equation 141, we know ∑ + ∞ n = 1 ( P n + Q n ) is convergence almost surely.
So we get that ∑ q
[Equation]
n − q
[Equation]
n − 1 − 1 i =(cid:16) P
[Equation]
+
[Equation]
+
[Equation]
+ i (cid:17) → 0 a .
s .
by the Cauchy (cid:48) s test for convergence .
Combining 1 / S
[Equation]
+ ε 1 q 2 n − 1 + i → 0 a .
s .
, we get
[Equation]
so there is + ∞
[Equation]
But on the other hand, we know (cid:107) ∇ θ q 2 n
[Equation]
1 + i g ( θ q
[Equation]
+ i )
[Equation]
> o ( i > 0 ) .
Together with equation 121, we get
[Equation]
It contradicts with equation 148, so we get that (cid:107) ∇ θ n g ( θ n ) (cid:107) → 0 a .
s . .
Combining equation 133, we get (cid:107) ∇ θ n g ( θ n ) (cid:107) → 0 no matter S n < + ∞ a .
s .
or S n = + ∞ .
Under Assumption 5 1), it is safe to conclude that there exists a connected component J ∗ of J such that lim n → ∞ d ( θ n , J ∗ )
= 0.