O N THE C ONVERGENCE OF A C LASS OF A DAM -T YPE A LGORITHMS FOR N ON -C ONVEX O PTIMIZATION
Anonymous authors
Paper under double-blind review
A BSTRACT
This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients.
This class, which we refer to as the “Adam-type”, includes the popular algorithms such as Adam (Kingma & Ba, 2014) , AMSGrad (Reddi et al ., 2018) , AdaGrad (Duchi et al.,
Despite their popularity in training deep neural networks (DNNs), the convergence of these algorithms for solving non-convex problems remains an open question.
In this paper, we develop an analysis framework and a set of mild sufﬁcient conditions that guarantee the convergence of the Adam-type methods, with a convergence rate of order O (log T/ √ T ) for non-convex stochastic optimization.
Our convergence analysis applies to a new algorithm called AdaFom (AdaGrad with
First Order Momentum).
We show that the conditions are essential, by identifying concrete examples in which violating the conditions makes an algorithm diverge.
Besides providing one of the ﬁrst comprehensive analysis for Adam-type methods in the non-convex setting our results can also help the practitioners to easily monitor the progress of algorithms and determine their convergence behavior.
I NTRODUCTION
First-order optimization has witnessed tremendous progress in the last decade, especially to solve machine learning problems (Bottou et al., 2018).
Almost every ﬁrst-order method obeys the following generic form (Boyd & Vandenberghe, 2004), x t +1 = x
t − α t ∆ t , where x t denotes the solution updated at the t th iteration for t = 1 , 2 , . .
, T , T is the number of iterations, ∆ t is a certain (approximate) descent direction, and α t > 0 is some learning rate.
The most well-known ﬁrst-order algorithms are gradient descent (GD) for deterministic optimization (Nesterov, 2013; Cartis et al., 2010) and stochastic gradient descent (SGD) for stochastic optimization (Zinkevich, 2003; Ghadimi & Lan, 2013), where the former determines ∆ t using the full (batch) gradient of an objective function, and the latter uses a simpler but more computationally-efﬁcient stochastic (unbiased) gradient estimate.
Recent works have proposed a variety of accelerated versions of GD and SGD (Nesterov, 2013).
These achievements fall into three categories: a) momentum methods (Nesterov, 1983; Polyak, 1964;
Ghadimi et al., 2015) which carefully design the descent direction ∆ t ; b) adaptive learning rate methods (Becker et al., 1988; Duchi et al., 2011; Zeiler, 2012; Dauphin et al., 2015) which determine good learning rates α t , and c) adaptive gradient methods that enjoy dual advantages of a) and b).
In particular, Adam (Kingma & Ba, 2014), belonging to the third type of methods, has become extremely popular to solve deep learning problems, e.g., to train deep neural networks.
Despite its superior performance in practice, theoretical investigation of Adam-like methods for non-convex optimization is still missing.
Very recently, the work (Reddi et al., 2018) pointed out the convergence issues of Adam even in the convex setting, and proposed AMSGrad, a corrected version of Adam.
Although AMSGrad has made a positive step towards understanding the theoretical behavior of adaptive gradient methods, the convergence analysis of (Reddi et al., 2018) was still very restrictive because it only works for convex problems, despite the fact that the most successful applications are for non-convex problems.
Apparently, there still exists a large gap between theory and practice.
To the best of our knowledge, the question that whether adaptive gradient methods such as Adam, AMSGrad, AdaGrad converge for non-convex problems is still open in theory.
After the non-convergence issue of Adam has been raised in (Reddi et al., 2018), there have been a few recent works on proposing new variants of Adam-type algorithms.
In the convex setting, reference (Huang et al., 2018) proposed to stabilize the coordinate-wise weighting factor to ensure convergence.
Reference (Chen & Gu, 2018) developed an algorithm that changes the coordinatewise weighting factor to achieve better generalization performance.
Concurrent with this work, several works are trying to understand performance of Adam in non-convex optimization problems.
Reference (Basu et al., 2018) provided convergence rate of original Adam and RMSprop under full-batch (deterministic) setting, and (Ward et al., 2018) proved convergence rate of a modiﬁed version of AdaGrad where coordinate-wise weighting is removed.
Furthermore, the work (Zhou et al., 2018) provided convergence results for AMSGrad that exhibit a tight dependency on problem dimension compared to (Reddi et al., 2018).
The works (Zou & Shen, 2018) and (Li & Orabona, 2018) proved that both AdaGrad and its variant (AdaFom) converge to a stationary point with a high probability.
The aforementioned works are independent of ours.
In particular, our analysis is not only more comprehensive (it covers the analysis of a large family of algorithms in a single framework), but more importantly, it provides insights on how oscillation of stepsizes can affect the convergence rate.
Contributions Our work aims to build the theory to understand the behavior for a generic class of adaptive gradient methods for non-convex optimization.
In particular, we provide mild sufﬁcient conditions that guarantee the convergence for the Adam-type methods.
We summarize our contribution as follows.
• (Generality)
We consider a class of generalized Adam, referred to as the “Adam-type”, and we show for the ﬁrst time that under suitable conditions about the stepsizes and algorithm parameters, this class of algorithms all converge to ﬁrst-order stationary solutions of the non-convex problem, with O (log T/ √ T ) convergence rate.
This class includes the recently proposed AMSGrad (Reddi et al., 2018), AdaGrad (Duchi et al.,
and stochastic
heavy-ball
methods
as
well as two new algorithms explained below
• (AdaFom) Adam adds momentum to both the ﬁrst and the second moment estimate, but this leads to possible divergence (Reddi et al., 2018).
We show that the divergence issue can actually be ﬁxed by a simple variant which adds momentum to only the ﬁrst moment estimate while using the same second moment estimate as that of AdaGrad, which we call AdaFom (AdaGrad with First Order Moment).
• (Constant Momemtum)
Our convergence analysis is applicable to the constant momentum parameter setting for AMSGrad and AdaFom.
The divergence example of Adam given in (Reddi et al., 2018) is for constant momentum parameter, but the convergence analysis of AMSGrad in (Reddi et al., 2018) is for diminishing momentum parameter.
This discrepancy leads to a question whether the convergence of AMSGrad is due to the algorithm form or due to the momentum parameter choice – we show that the constant-momentum version of AMSGrad indeed converges, thus excluding the latter possibility.
• (Practicality) The sufﬁcient conditions we derive are simple and easy to check in practice.
They can be used to either certify the convergence of a given algorithm for a class of problem instances, or to track the progress and behavior of a particular realization of an algorithm.
• (Tightness and Insight)
We show the conditions are essential and “tight”, in the sense that violating them can make an algorithm diverge.
Importantly, our conditions provide insights on how oscillation of a so-called “effective stepsize" (that we deﬁne later) can affect the convergence rate of the class of algorithms.
We also provide interpretations of the convergence conditions to illustrate why under some circumstances, certain Adam-type algorithms can outperform SGD.
Notations We use z = x/y to denote element-wise division if x and y are both vectors of size d ; x (cid:12) y is element-wise product, x 2 is element-wise square if x is a vector, √ x is element-wise square root if x is a vector, ( x ) j denotes j th coordinate of x , (cid:107) x (cid:107) is (cid:107) x (cid:107) 2 if not otherwise speciﬁed.
We use [ N ] to denote the set { 1 , · · · , N } , and use O ( · ) , o ( · ) , Ω( · ) , ω ( · ) as standard asymptotic notations.
P RELIMINARIES AND A DAM -T YPE A LGORITHMS
Stochastic optimization is a popular framework for analyzing algorithms in machine learning due to the popularity of mini-batch gradient evaluation.
We consider the following generic problem where we are minimizing a function f , expressed in the expectation form as follows
[Equation]
where ξ is a certain random variable representing randomly selected data sample or random noise.
In a generic ﬁrst-order optimization algorithm, at a given time t we have access to an unbiased noisy gradient g t of f ( x ) , evaluated at the current iterate x t .
The noisy gradient is assumed to be bounded and the noise on the gradient at different time t is assumed to be independent.
An important assumption that we will make throughout this paper is that the function f ( x ) is continuously differentiable and has Lipschitz continuous gradient, but could otherwise be a non-convex function.
The non-convex assumption represents a major departure from the convexity that has been assumed in recent papers for analyzing Adam-type methods, such as (Kingma & Ba, 2014) and (Reddi et al., 2018).
Our work focuses on the generic form of exponentially weighted stochastic gradient descent method presented in Algorithm 1, for which we name as generalized Adam due to its resemblance to the original Adam algorithm and many of its variants.
Algorithm 1.
Generalized Adam
S0.
Initialize m 0 = 0 and x
For t = 1 , · · · , T , do
[Equation]
End
In Algorithm 1, α t is the step size at time t , β 1 ,t > 0 is a sequence of problem parameters, m t ∈ R d denotes some (exponentially weighted) gradient estimate, and (cid:98) v t
= h t ( g 1 , g 2 , ..., g t )
∈ R
d takes all the past gradients as input and returns a vector of dimension d , which is later used to inversely weight the gradient estimate m t .
And note that m t / √ (cid:98) v t ∈ R d represents element-wise division.
Throughout the paper, we will refer to the vector α t / √ (cid:98) v t as the effective stepsize .
We highlight that Algorithm 1 includes many well-known algorithms as special cases.
We summarize some popular variants of the generalized Adam algorithm in Table
[Table] [Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
We present some interesting ﬁndings for the algorithms presented in Table 1.
• Adam is often regarded as a “momentum version” of AdaGrad, but it is different from AdaFom which is also a momentum version of AdaGrad 1 .
The difference lies in the form of (cid:98) v t .
Intuitively, Adam adds momentum to both the ﬁrst and second order moment estimate, while in AdaFom we only add momentum to the ﬁrst moment estimate and use the same second moment estimate as AdaGrad.
These two methods are related in the following way: if we let β 2 = 1 − 1 /t in the expression of ˆ v t in Adam, we obtain AdaFom.
We can view AdaFom as a variant of Adam with an increasing sequence of β 2 , or view Adam as a variant of AdaFom with exponentially decaying weights of g 2 t .
However, this small change has large impact on the convergence: we prove that AdaFom can always converge under standard assumptions (see Corollary 3.2) , while Adam is shown to possibly diverge (Reddi et al., 2018)
• The convergence of AMSGrad using a fast diminishing β 1 ,t such that β 1 ,t ≤ β 1 ,t − 1 , β 1 ,t −−−→ t →∞ b, b = 0 in convex optimization was studied in (Reddi et al., 2018).
However, the convergence of the version with constant β 1 or strictly positive b and the version for non-convex setting are unexplored before our work. We notice that an independent work (Zhou et al., 2018) has also proved the convergence of AMSGrad with constant β 1 .
It is also worth mentioning that Algorithm 1 can be applied to solve the popular “ﬁnite-sum” problems whose objective is a sum of n individual cost functions.
That is,
[Equation]
where each f i : R d → R is a smooth and possibly non-convex function.
If at each time instance the index i is chosen uniformly randomly, then Algorithm 1 still applies, with g t = ∇
f i ( x t ) .
It can also be extended to a mini-batch case with g t = 1 b (cid:80) i ∈I t ∇ f i ( x t ) , where I t denotes the minibatch of size b at time t .
It is easy to show that g t is an unbiased estimator for ∇ f ( x ) .
In the remainder of this paper, we will analyze Algorithm 1 and provide sufﬁcient conditions under which the algorithm converges to ﬁrst-order stationary solutions with sublinear rate.
We will also discuss how our results can be applied to special cases of generalized Adam.
C ONVERGENCE A NALYSIS FOR G ENERALIZED A DAM
The main technical challenge in analyzing the non-convex version of Adam-type algorithms is that the actually used update directions could no longer be unbiased estimates of the true gradients.
Furthermore, an additional difﬁculty is introduced by the involved form of the adaptive learning rate.
Therefore the biased gradients have to be carefully analyzed together with the use of the inverse of exponential moving average while adjusting the learning rate.
The existing convex analysis (Reddi et al., 2018) does not apply to the non-convex scenario we study for at least two reasons: ﬁrst, non-convex optimization requires a different convergence criterion, given by stationarity rather than the global optimality; second, we consider constant momentum controlling parameter.
In the following, we formalize the assumptions required in our convergence analysis.
Assumptions
A1: f is differentiable and has L -Lipschitz gradient, i.e. ∀ x, y , (cid:107)∇ f ( x ) − ∇ f ( y ) (cid:107) ≤ L (cid:107) x − y (cid:107) .
It is also lower bounded, i.e. f ( x ∗ ) >
−∞ where x ∗ is an optimal solution.
A2: At time t , the algorithm can access a bounded noisy gradient
and the true gradient is bounded
i
e. (cid:107)∇ f ( x t ) (cid:107) ≤ H, (cid:107) g t (cid:107) ≤ H, ∀ t > 1 .
A3:
The noisy gradient is unbiased and the noise is independent, i.e. g t = ∇ f ( x t ) + ζ t , E [ ζ t =and ζ i is independent of ζ j if i (cid:54) =
j .
Reference (Reddi et al., 2018) uses a similar (but slightly different) assumption as A2, i.e., the bounded elements of the gradient (cid:107) g t (cid:107) ∞ ≤ a for some ﬁnite a .
The bounded norm of ∇ f ( x t ) in
A2 is equivalent to Lipschitz continuity of f (when f is differentiable) which is a commonly used condition in convergence analysis.
This assumption is often satisﬁed in practice, for example it holds for the ﬁnite sum problem (2) when each f i has bounded gradient, and g t = ∇ f i ( x t ) where i is sampled randomly.
A3 is also standard in stochastic optimization for analyzing convergence.
Our main result shows that if the coordinate-wise weighting term √ (cid:98) v t in Algorithm 1 is properly chosen, we can ensure the global convergence as well as the sublinear convergence rate of the algorithm (to a ﬁrst-order stationary solution).
First, we characterize how the effective stepsize parameters α t and (cid:98) v t affect convergence of Adam-type algorithms.
Theorem 3.1.
Suppose that Assumptions A1-A3 are satisﬁed, β 1 is chosen such that β 1 ≥ β 1 ,t , β 1 ,t ∈ [0 , 1) is non-increasing, and for some constant G > 0 , (cid:13)(cid:13)
α t m t / √ (cid:98) v t (cid:13)(cid:13) ≤ G, ∀ t. Then
Algorithm 1 yields
[Equation]
where C 1 , C 2 , C 3 are constants independent of d and T , C 4 is a constant independent of T , the expectation is taken with respect to all the randomness corresponding to { g t } .
Further, let γ t :=
min j ∈ [ d ] min { g i } ti =1 α t / ( √ (cid:98) v t )
j denote the minimum possible value of effective stepsize at time t over all possible coordinate and past gradients { g i } ti =1 .
Then the convergence rate of Algorithm 1 is given by
[Equation]
where s 1 ( T ) is deﬁned through the upper bound of RHS of (3) , namely, O ( s 1 ( T )) , and (cid:80) Tt =1 γ t
= Ω( s 2 ( T )) .
Proof : See Appendix 6.2.
[Equation]
In Theorem 3.1, (cid:107) α t m t / √ (cid:98) v t (cid:107) ≤ G is a mild condition.
Roughly speaking, it implies that the change of x t at each each iteration should be ﬁnite.
As will be evident later, with (cid:107) g t (cid:107) ≤ H , the condition (cid:107) α t m t / √ (cid:98) v t (cid:107) ≤ G is automatically satisﬁed for both AdaGrad and AMSGrad.
Besides, instead of bounding the minimum norm of ∇ f in (4), we can also apply a probabilistic output (e.g., select an output x R with probability p ( R = t ) =
[Equation] Tt =1 γ t ) to bound E [ (cid:107)∇ f ( x R ) (cid:107) 2 ] (Ghadimi & Lan, 2013; Lei et al., 2017).
It is worth mentioning that a small number (cid:15) could be added to ˆ v t for ensuring the numerical stability.
In this case, our Theorem 3.1 still holds given the fact the resulting algorithm is still a special case of Algorithm 1.
Accordingly, our convergence results for AMSGrad and AdaFom that will be derived later also hold as (cid:107)
[Equation] G when (cid:15) is added to (cid:98) v t . We will provide a detailed explanation of Theorem 3.1 in Section 3.1.
Theorem 3.1 implies a sufﬁcient condition that guarantees convergence of the Adam-type methods: s 1 ( T ) grows slower than s 2 ( T ) . We will show in Section
[Equation]
2 that the rate s 1 ( T ) can be dominated by different terms in different cases, i [Equation] e. the non-constant quantities Term A and B below
[Equation]
[Equation] where the growth of third term at LHS of (5) can be directly related to growth of Term B via the relationship between (cid:96) 1 and (cid:96) 2 norm or upper boundedness of ( α t / √ (cid:98) v t ) j .
3. E XPLANATION OF CONVERGENCE CONDITIONS
From (4) in Theorem 3. it is evident that s 1 ( T )
=
o ( s 2 ( T )) can ensure proper convergence of the algorithm.
This requirement has some important implications which we discuss below.
• (The Bounds for s 1 ( T ) and s 2 ( T ) )
First, the requirement that s 1 ( T ) =
o ( s 2 ( T )) implies that E [ (cid:80) T t =1 (cid:107) α t g t / √ (cid:98) v t (cid:107) 2 ]
= o ( (cid:80) T t =1 γ t ) .
This is a common condition generalized from SGD.
Term A in (5) is a generalization of the term (cid:80)
Tt [-=
 α 2-]=1 (cid:107) a t g t (cid:107) 2 for SGD (where { α t } is the stepsize sequence for SGD), and it quantiﬁes possible increase in the objective function brought by higher order curvature.
The term (cid:80) Tt =1 γ t is the lower bound on the summation of effective stepsizes, which reduces to (cid:80)
Tt =
1 α t when Algorithm 1 is simpliﬁed to SGD.
• (Oscillation of Effective Stepsizes) Term B in (5) characterizes the oscillation of effective stepsizes α t / √ (cid:98) v t .
In our analysis such an oscillation term upper bounds the expected possible ascent in objective induced by skewed update direction g t / √ (cid:98) v t (“skewed” in the sense that E [ g t / √ (cid:98) v t ] is not parallel with ∇ f ( x t ) ), therefore it cannot be too large.
Bounding this term is critical, and to demonstrate this fact, in Section 3.2.2 we show that large oscillation can result in non-convergence of Adam for even simple unconstrained non-convex problems.
• (Advantage of Adaptive Gradient).
One possible beneﬁt of adaptive gradient methods can be seen from Term A. When this term dominates the convergence speed in Theorem 3.1, it is possible that proper design of (cid:98) v t can help reduce this quantity compared with SGD (An example is provided in Appendix 6.1.1 to further illustrate this fact.)
in certain cases.
Intuitively, adaptive gradient methods like AMSGrad can provide a ﬂexible choice of stepsizes, since (cid:98) v t can have a normalization effect to reduce oscillation and overshoot introduced by large stepsizes.
At the same time, ﬂexibility of stepsizes makes the hyperparameter tuning of an algorithm easier in practice.
3. T IGHTNESS OF THE RATE BOUND (4)
In the next, we show our bound (4) is tight in the sense that there exist problems satisfying Assumption 1 such that certain algorithms belonging to the class of Algorithm 1 can diverge due to the high growth rate of Term A or
Term B.
3.2. N ON - CONVERGENCE OF SGD AND ADAM DUE TO EFFECT OF T ERM A
We demonstrate the importance of Term A in this subsection.
Consider a simple one-dimensional optimization problem min x f ( x ) , with f ( x ) = 100 x 2 if | x | < = b , and f ( x ) = 200 b | x | − 100 b 2 if | x | > b , where b = 10 .
In Figure 1, we show the growth rate of different terms given in Theorem 3.1, where α 0 (cid:44) 0 , α t = 0 .
for t ≥ 1 , and β 1 ,t = 0 , β 2 ,t = 0 .
9 for both Adam and AMSGrad.
We observe that both SGD and Adam are not converging to a stationary solution ( x = 0 ), which is because (cid:80)
Tt =1 (cid:13)(cid:13)
α t g t / √ (cid:98) v t (cid:13)(cid:13) 2 grows with the same rate as accumulation of effective stepsizes as shown in the ﬁgure.
Actually, SGD only converges when α t < 0 .
and our theory provides an perspective of why SGD diverges when α t ≥ 0 .
In the example, Adam is also not converging to 0 due to Term A. From our observation, Adam oscillates for any constant stepsize within [10 − 4 , 0 . 1] for this problem and Term A always ends up growing as fast as accumulation of effective stepsizes, which implies Adam only converges with diminishing stepsizes even in non-stochastic optimization.
In contrast to SGD and Adam, AMSGrad converges in this case since both Term A and Term B grow slower than accumulation of effective stepsizes.
For AMSGrad and Adam, (cid:98) v t has a strong normalization effect and it allows the algorithm to use a larger range of α t .
The practical beneﬁt of this ﬂexible choice of stepsizes is easier hyperparameter tuning, which is consistent with the impression of practitioners about the original Adam.
We present more experimental results in Appendix 6.1.
accompanied with more detailed discussions.
3.2. N ON - CONVERGENCE OF A DAM DUE TO EFFECT OF T ERM B
Next, we use an example to demonstrate the importance of the Term B for the convergence of
Adam-type algorithms.
Consider optimization problem min x f ( x ) =
(cid:80)
i =
f i ( x ) where
[Equation]
and I [1 = 1] = 1 , I [1 (cid:54) = 1] = 0 .
It is easy to verify that the only point with ∇ f ( x ) = 0 is x = 0 .
The problem satisﬁes the assumptions in Theorem 3.1 as the stochastic gradient g t = ∇ f i ( x t ) is sampled uniformly for i ∈ [11] .
We now use the AMSGrad and Adam to optimize x , and the results are given in Figure 2, where we set α t = 1 , β 1 ,t = 0 , and β 2 ,t = 0 .
We observe that (cid:80)
[Equation]
in Term B grows with the same rate as (cid:80)
Tt =
1 α t / √ (cid:98) v t for Adam, where we recall that (cid:80) Tt =
α t / √ (cid:98) v t is an upper bound of (cid:80)
Tt =1 γ t in Theorem 3.1.
As a result, we obtain O ( s 1 ( T ) /s 2 ( T )) (cid:54) =
o (1) in (4), implying the non-convergence of Adam.
Our theoretical analysis matches the empirical results in Figure 2.
In contrast, AMSGrad converges in Figurebecause of its smaller oscillation in effective stepsizes, associated with Term B. We ﬁnally remark that the importance of the quantity (cid:80) Tt =1 (cid:107) α t / √ (cid:98) v t − α t − 1 / (cid:112)(cid:98) v t − 1 (cid:107) 1 is also noticed by (Huang et al., 2018).
However, they did not analyze its effect on convergence, and their theory is only for convex optimization.
3. C ONVERGENCE OF AMSG RAD AND A DA F OM
Theorem 3.1 provides a general approach for the design of the weighting sequence { (cid:98) v t } and the convergence analysis of Adam-type algorithms.
For example, SGD speciﬁed by Table 1 with stepsizes α t = 1 / √ t yields O (log T/ √ T ) convergence speed by Theorem 3.1.
Moreover, the explanation on the non-convergence of Adam in (Reddi et al., 2018) is consistent with our analysis in Section 3.2.
That is, Term B in (5) can grow as fast as s 2 ( T ) so that s 1 ( T ) /s 2 ( T ) becomes a constant.
Further, we notice that Term A in (5) can also make Adam diverge which is unnoticed before.
Aside from checking convergence of an algorithm, Theorem 3.1 can also provide convergence rates of AdaGrad and AMSGrad, which will be given as corollaries later.
Our proposed convergence rate of AMSGrad matches the result in (Reddi et al., 2018) for stochastic convex optimization.
However, the analysis of AMSGrad in (Reddi et al., 2018) is constrained to diminishing momentum controlling parameter β 1 ,t .
Instead, our analysis is applicable to the more popular constant momentum parameter, leading to a more general non-increasing parameter setting.
In Corollary 3.1 and Corollary 3.2, we derive the convergence rates of AMSGrad (Algorithm 3 in
Appendix 6.2.3) and AdaFom (Algorithm 4 in Appendix 6.2.4), respectively.
Note that AdaFom is more general than AdaGrad since when β 1 ,t = 0 , AdaFom becomes AdaGrad.
Corollary 3.1.
For AMSGrad (Algorithm 3 in Appendix 6.2.3) with β 1 ,t ≤ β 1 ∈ [0 , 1) and β 1 ,t is non-increasing, α t = 1 / √ t , we have for any T ,
[Equation]
where Q 1 and Q 2 are two constants independent of T .
Proof : See Appendix 6.2.3.
[Equation]
Corollary 3.2.
For AdaFom (Algorithm 4 in Appendix 6.2.4) with β 1 ,t ≤ β 1 ∈ [0 , 1) and β 1 ,t is non-increasing, α t = 1 / √ t , we have for any T ,
[Equation]
where Q (cid:48) 1 and Q (cid:48) 2 are two constants independent of T .
Proof : See Appendix 6.2.4.
[Equation]
We remark that the derived convergence rate of AMSGrad and AdaFom involves an additional log T factor compared to the fastest rate of ﬁrst order methods ( 1 / √ T ).
However, such a slowdown can be mitigated by choosing an appropriate stepsize.
To be speciﬁc, the log T factor for AMSGrad would be eliminated when we adopt a constant rather than diminishing stepsize, e.g., α t = 1 / √ T .
It is also worth mentioning that our theoretical analysis focuses on the convergence rate of adaptive methods in the worst case for nonconvex optimization.
Thus, a sharper convergence analysis that can quantify the beneﬁts of adaptive methods still remains an open question in theory.
E MPIRICAL PERFORMANCE OF A DAM - TYPE ALGORITHMS ON MNIST
In this section, we compare the empirical performance of Adam-type algorithms, including AMSGrad,
Adam, AdaFom and AdaGrad, on training two convolutional neural networks (CNNs).
In the ﬁrst example, we train a CNN of 3 convolutional layers and 2 fully-connected layers on MNIST.
In the second example, we train a CIFARNET on CIFAR-10.
We refer readers to Appendix 6.1.2 for more details on the network model and the [-mini-batch size. The stepsize is choose to be α t = 0 . 
-]parameter setting.
[Figure]
In Figure 3, we present the training loss and the classiﬁcation accuracy of Adam-type algorithms versus the number of iterations.
As we can see, AMSGrad performs quite similarly to Adam which conﬁrms the result in (Reddi et al., 2018).
The performance of AdaGrad is worse than other algorithms, because of the lack of momentum and/or the signiﬁcantly different choice of (cid:98) v t . We also observe that the performance of AdaFom lies between AMSGrad/Adam and AdaGrad. This is not
[Figure]
surprising, since AdaFom can be regarded as a momentum version of AdaGrad but uses a simpler adaptive learning rate (independent on β 2 ) compared to AMSGrad/Adam.
In Figure 4, we consider to train a larger network (CIFARNET) on CIFAR-10.
As we can see, Adam and AMSGrad perform similarly and yield the best accuracy.
AdaFom outperforms AdaGrad in both training and testing, which agrees with the results obtained in the MNIST experiment.
C ONCLUSION AND D ISCUSSION
We provided some mild conditions to ensure convergence of a class of Adam-type algorithms, which includes Adam, AMSGrad, AdaGrad, AdaFom, SGD, SGD with momentum as special cases.
Apart from providing general convergence guarantees for algorithms, our conditions can also be checked in practice to monitor empirical convergence.
To the best of our knowledge, the convergence of
Adam-type algorithm for non-convex problems was unknown before.
We also provide insights on how oscillation of effective stepsizes can affect convergence rate for the class of algorithms which could be beneﬁcial for the design of future algorithms.
This paper focuses on unconstrained non-convex optimization problems, and one future direction is to study a more general setting of constrained non-convex optimization.
A PPENDIX 6. R ELATED W ORK
Momentum methods take into account the history of ﬁrst-order information (Nesterov, 2013; 1983;
Nemirovskii et al., 1983; Ghadimi & Lan, 2016; Polyak, 1964; Ghadimi et al., 2015; Ochs et al., 2015; Yang et al., 2016; Johnson & Zhang, 2013; Reddi et al., 2016;
Lei et al., 2017).
A well-known method, called Nesterov’s accelerated gradient (NAG) originally designed for convex deterministic optimization (Nesterov, 2013; 1983; Nemirovskii et al., 1983), constructs the descent direction ∆ t using the difference between the current iterate and the previous iterate.
A recent work (Ghadimi & Lan, 2016) studied a generalization of NAG for non-convex stochastic programming.
Similar in spirit to NAG, heavy-ball (HB) methods (Polyak, 1964;
Ghadimi et al., 2015; Ochs et al., 2015;
Yang et al., 2016) form the descent direction vector through a decaying sum of the previous gradient information.
In addition to NAG and HB methods, stochastic variance reduced gradient (SVRG) methods integrate SGD with GD to acquire a hybrid descent direction of reduced variance (Johnson &
Zhang, 2013; Reddi et al., 2016; Lei et al., 2017).
Recently, certain accelerated version of perturbed gradient descent (PAGD) algorithm is also proposed in (Jin et al., 2017), which shows the fastest convergence rate among all Hessian free algorithms.
Adaptive learning rate methods accelerate ordinary SGD by using knowledge of the past gradients or second-order information into the current learning rate α t (Becker et al., 1988; Duchi et al., 2011; Zeiler, 2012; Dauphin et al., 2015).
In (Becker et al., 1988), the diagonal elements of the
Hessian matrix were used to penalize a constant learning rate.
However, acquiring the second-order information is computationally prohibitive.
More recently, an adaptive subgradient method (i.e.,
AdaGrad) penalized the current gradient by dividing the square root of averaging of the squared gradient coordinates in earlier iterations (Duchi et al., 2011).
Although AdaGrad works well when gradients are sparse, its convergence is only analyzed in the convex world.
Other adaptive learning rate methods include Adadelta (Zeiler, 2012) and ESGD
(Dauphin et al., 2015), which lacked theoretical investigation although some convergence improvement was shown in practice.
Adaptive gradient methods update the descent direction and the learning rate simultaneously using knowledge in the past, and thus enjoy dual advantages of momentum and adaptive learning rate methods.
Algorithms of this family include RMSProp (Tieleman & Hinton, 2012), Nadam (Dozat, 2016), and Adam (Kingma & Ba, 2014).
Among these, Adam has become the most widely-used method to train deep neural networks (DNNs).
Speciﬁcally, Adam adopts exponential moving averages (with decaying/forgetting factors) of the past gradients to update the descent direction.
It also uses inverse of exponential moving average of squared past gradients to adjust the learning rate.
The work (Kingma & Ba, 2014) showed Adam converges with at most O (1 / √ T ) rate for convex problems.
However, the recent work (Reddi et al., 2018) pointed out the convergence issues of Adam even in the convex setting, and proposed a modiﬁed version of Adam (i.e., AMSGrad), which utilizes a non-increasing quadratic normalization and avoids the pitfalls of Adam.
Although AMSGrad has made a signiﬁcant progress toward understanding the theoretical behavior of adaptive gradient methods, the convergence analysis of (Reddi et al., 2018) only works for convex problems.
6.1. A DVANTAGES AND D ISADVANTAGES OF A DAPTIVE GRADIENT METHOD
In this section, we provide some additional experiments to demonstrate how speciﬁc Adam-type algorithms can perform better than SGD and how SGD can out perform Adam-type algorithms in different situations.
One possible beneﬁt of adaptive gradient methods is the “sparse noise reduction” effect pointed out in
Bernstein et al.
Below we illustrate another possible practical advantage of adaptive gradient methods when applied to solve non-convex problems , which we refer to as ﬂexibility of stepsizes .
To highlight ideas, let us take AMSGrad as an example, and compare it with SGD.
First, in nonconvex problems there can be multiple valleys with different curvatures.
When using ﬁxed stepsizes (or even a slowly diminishing stepsize), SGD can only converge to local optima in valleys with small curvature while AMSGrad and some other adaptive gradient algorithms can potentially converge to optima in valleys with relative high curvature (this may not be beneﬁcial if one don’t want to converge to a sharp local minimum).
Second, the ﬂexible choice of stepsizes implies less hyperparameter tuning and this coincides with the popular impression about original Adam.
We empirically demonstrate the ﬂexible stepsizes property of AMSGrad using a deterministic quadratic problem.
Consider a toy optimization problem min x f ( x ) , f ( x ) = 100 x 2 , the gradient is given by 200 x .
For SGD (which reduces to gradient descent in this case) to converge, we must have α t < 0 .
01 ; for AMSGrad, (cid:98) v t has a strong normalization effect and it allows the algorithm to use larger α t ’s. We show the growth rate of different terms given in Theorem 3.1 for different stepsizes in Figure A1 to Figure A4 (where we choose β 1 ,t = 0 , β 2 ,t = 0 .
9 for both Adam and AMSGrad).
In
Figure A1, α t = 0 .
1 and SGD diverges due to large α t , AMSGrad converges in this case, Adam is oscillating between two non-zero points.
In Figure A2, stepsizes α t is set to 0.01, SGD and Adam are oscillating, AMSGrad converges to 0.
For Figure A3, SGD converges to 0 and AMSGrad is converging slower than SGD due to its smaller effective stepsizes, Adam is oscillating.
One may wonder how diminishing stepsizes affects performance of the algorithms, this is shown in Figure A where α t = 0 .
1 / √ t , we can see SGD is diverging until stepsizes is small, AMSGrad is converging all the time, Adam appears to get stuck but it is actually converging very slowly due to diminishing stepsizes.
This example shows AMSGrad can converge with a larger range of stepsizes compared with SGD.
From the ﬁgures, we can see that the term (cid:80) Tt =1 (cid:107) α t g t / √ (cid:98) v t (cid:107) 2 is the key quantity that limits the convergence speed of algorithms in this case.
In Figure A1, Figure A2, and early stage of Figure
A4, the quantity is obviously a good sign of convergence speed.
In Figure A3, since the difference of quantity between AMSGrad and SGD is compensated by the larger effective stepsizes of SGD and some problem independent constant, SGD converges faster.
In fact, Figure A3 provides a case where AMSGrad does not perform well.
Note that the normalization factor √ (cid:98) v t can be understood as imitating the largest Lipschitz constant along the way of optimization, so generally speaking dividing by this number makes the algorithm converge easier.
However when the Lipschitz constant becomes smaller locally around a local optimal point, the stepsizes choice of AMSGrad dictates that √ (cid:98) v t does not change, resulting a small effective stepsizes.
This could be mitigated by AdaGrad and its momentum variants which allows (cid:98) v t to decrease when g t keeps decreasing. [Figure] [Figure] [Figure]
[Figure]
D ETAILS OF EXPERIMENTS ON MNIST AND CIFAR-
In the experiment on MNIST, we consider a convolutional neural network (CNN), which includesconvolutional layers and 2 fully-connected layers.
In convolutional layers, we adopt ﬁlters of sizes 6 × 6 × 1 (with stride 1 ), 5 × 5 × 6 (with stride 2 ), and 6 × 6 × 12 (with stride 2 ), respectively. In both AMSGrad 2 and Adam, we set β 1 = 0 . 9 and β 2 = 0 . In AdaFom, we set β 1 = 0 .
{+We choose 50 as the mini-batch size and the stepsize is choose to be α t = 0 . 
+}
+ 0 . 003 e − t/ 2000 . The architecture of the CIFARNET that we are using is as below. The model starts with two convolutional layers with 32 and 64 kernels of size 3 x 3, followed by 2 x 2 max pooling and dropout with keep probability 0.25.
The next layers are two convolutional layers with 128 kernels of size 3 x 3 and 2 x 2, respectively.
Each of the two convolutional layers is followed by a 2 x 2 max pooling layer.
The last layer is a fully connected layer with 1500 nodes.
Dropout with keep probability 0.
is added between the fully connected layer and the convolutional layer.
All convolutional layers use
ReLU activation and stride 1. The learning rate α t of Adam and AMSGrad starts with 0.001 and decrease 10 times every 20 epochs. The learning rate of AdaGrad and AdaFom starts with 0.05 and decreases to 0.001 after 20 epochs and to 0.0001 after 40 epochs. These learning rates are tuned so that each algorithm has its best performance.
6. C ONVERGENCE PROOF FOR G ENERALIZED A DAM (A LGORITHM 1)
In this section, we present the convergence proof of Algorithm 1.
We will ﬁrst give several lemmas prior to proving Theorem 3.1.
6.2. P ROOF OF A UXILIARY L EMMAS
Lemma 6.1.
Let x 0 (cid:44) x 1 in Algorithm 1, consider the sequence
[Equation]
Then the following holds true
[Equation]
Proof.
[Proof of Lemma 6.1]
By the update rules S1-S3 in Algorithm 1, we have when t > 1 , x t
[Equation]
[Equation]
Since x t +1 − x t = (1 − β 1 ,t ) x t +1 + β 1 ,t ( x t +1 − x t ) − (1 − β 1 ,t ) x t , based on (10) we have (1 − β 1 ,t ) x
t +1 + β
[Equation]
t ( x t +1 − x t ) =
[Equation] [Equation]
v t . Divide both sides by 1 − β 1 ,t , we have x
[Equation]
[Equation]
Deﬁne the sequence
[Equation]
Then (11) can be written as
[Equation]
where the second equality is due to x t +1 − x t = − α t m t / √ (cid:98) v t .
For t = 1 , we have z 1 = x 1 (due to x 1 = x 0 ), and z
[Equation]
where the forth equality holds due to (S1) and (S3) of Algorithm 1.
The proof is now complete.
Q.E.D.
Without loss of generality, we initialize Algorithm 1 as below to simplify our analysis in what follows,
[Equation]
Lemma [Equation]
Suppose that the conditions in Theorem 3.1 hold, then
[Equation]
where
[Equation]
[Equation]
Proof.
[Proof of Lemma 6.2] By the Lipschitz smoothness of ∇ f , we obtain
[Equation]
where d t = z t +1 − z t , and Lemma 6.1 together with (12) yield
[Equation]
Based on (20) and (21), we then have
E [ f ( z t +1 )
[Equation]
f ( z 1 )]
= E (cid:34) t (cid:88) i =
f ( z i +1 )
− f ( z i ) (cid:35) ≤ E (cid:34) t (cid:88) i =
(cid:104)∇ f ( z i ) , d i (cid:105) + L 2 (cid:107) d i (cid:107)(cid:35) = − E (cid:34) t (cid:88) i =
(cid:104)∇ f ( z i ) , β 1 ,i 1 − β 1 ,i (cid:32) α i √ (cid:98) v i − α i −(cid:112)(cid:98)
v i −(cid:33) (cid:12) m i − 1 (cid:105) (cid:35) − E (cid:34) t (cid:88) i
= α
i (cid:104)∇ f ( z i ) , g i / (cid:112)(cid:98) v i (cid:105) (cid:35) − E (cid:34) t (cid:88)
i =
(cid:104)∇ f ( z i ) , (cid:18) β 1 ,i + 1 − β 1 ,i +1 − β 1 ,i 1 − β 1 ,i (cid:19)
α
i m i / (cid:112)(cid:98) v i (cid:105) (cid:35)
+ E (cid:20) t (cid:88) i =
L 2 (cid:107) d
i (cid:107)(cid:21) =
T 1 + T 2 + T 3 + + E (cid:20) t (cid:88) i =
L 2 (cid:107) d
i (cid:107)(cid:21) , (22) where { T i } have been deﬁned in (14)-(19).
Further, using inequality (cid:107) a + b + c (cid:107) 2 ≤ 3 (cid:107) a (cid:107) 2 +3 (cid:107)
[Equation]
+ 3 (cid:107) c (cid:107) 2 and (22), we have
[Equation]
Substituting the above inequality into (22), we then obtain (13).
Q.E.D.
The next series of lemmas separately bound the terms on RHS of (13).
Lemma 6.3.
Suppose that the conditions in Theorem 3.1 hold, T 1 in (14) can be bounded as
[Equation]
Proof.
[Proof of Lemma 6.3] Since (cid:107) g t (cid:107) ≤ H , by the update rule of m t , we have (cid:107) m t (cid:107) ≤ H , this can be proved by induction as below.
Recall that m t = β 1 ,t m t − 1 + (1 − β 1 ,t ) g t , suppose (cid:107) m t − 1 (cid:107) ≤ H , we have
[Equation]
(23) then since m 0 = 0 , we have (cid:107) m 0 (cid:107) ≤ H which completes the induction.
Given (cid:107) m t (cid:107) ≤ H , we further have
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12) (cid:32) α
[Equation]
(cid:98) v
[Equation]
− α
[Equation]
−(cid:112)(cid:98) v i −
[Equation]
(cid:33) j (cid:12)(cid:12)(cid:12)(cid:12) ≤ H
[Equation]
[Equation]
E   t (cid:88) i =
d (cid:88)
[Equation]
  where the ﬁrst equality holds due to (12), and the last inequality is due to β 1 ≥ β 1 ,i .
The proof is now complete.
Lemma 6.4.
Suppose the conditions in Theorem 3.1 hold.
For T 3 in (16) , we have
[Equation]
Proof.
[Proof of Lemma 6.4]
[Equation]
i
[Equation]
where the ﬁrst inequality is due to (cid:104) a, b (cid:105) ≤ 12 ( (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ) , the second inequality is using due to upper bound on (cid:107)∇ f ( x t ) (cid:107) ≤ H and (cid:107) α
i m i / √ (cid:98) v i (cid:107) ≤ G given by the assumptions in Theorem 3.1, the third equality is because β 1 ,t ≤ β 1 and β 1 ,t is non-increasing, the last inequality is due to telescope sum.
This completes the proof.
Q.E.D.
Lemma 6.5.
Suppose the assumptions in Theorem 3.1 hold.
For T 4 in (17) , we have
[Equation]
Proof.
[Proof of Lemma 6.5]
The proof is similar to the previous lemma.
[Equation]
β 1 ,t 1 − β 1 ,t (cid:19)(cid:107)
α t m t / √ v t (cid:107)(cid:35) ≤ E (cid:34) t (cid:88) i = (cid:18) β 1 ,t 1 − β 1 ,t − β 1 ,t + 1 − β 1 ,t + (cid:19)
G(cid:35) ≤ (cid:18) β1 − β 1 − β 1 ,t + 1 − β 1 ,t + (cid:19) t (cid:88)
i = (cid:18) β 1 ,t 1 − β 1 ,t − β 1 ,t + 1 − β 1 ,t + (cid:19)
G≤ (cid:18) β1 − β 1 − β 1 ,t + 1 − β 1 ,t + (cid:19)
Gwhere the ﬁrst inequality is due to (cid:107) α t m t / √ v t (cid:107) ≤ G by our assumptions, the second inequality is due to non-decreasing property of β 1 ,t and β 1 ≥ β 1 ,t , the last inequality is due to telescoping sum.
This completes the proof.
Lemma 6.6.
Suppose the assumptions in Theorem 3.1 hold.
For T 5 in (18) , we have
[Equation]
Proof.
[Proof of Lemma 6.6]
[Equation]

 

[Equation]
where the ﬁst inequality is due to β 1 ≥ β 1 ,t and (12), the second inequality is due to (cid:107) m i (cid:107) < H .
This completes the proof.
Q.E.D.
Lemma 6.7.
Suppose the assumptions in Theorem 3.1 hold.
For T 2 in (15) , we have
T
[Equation]
E
[Equation]
i
[Equation] [Equation]
(cid:112)(cid:98)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) j  
[Equation]
j =
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32)
[Equation]
(cid:98)
[Equation]
−
[Equation]
(cid:112)(cid:98)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
Proof.
[Proof of Lemma 6.7] Recall from the deﬁnition (9), we have
[Equation]
Further we have z 1 = x 1 by deﬁnition of z 1 .
We have
[Equation]
The second term of (26) can be bounded as
[Equation] [Equation]
where the ﬁrst inequality is because (cid:104) a, b
[Equation]
+ (cid:107) b
[Equation]
2 (cid:1) and the fact that z 1 = x 1 , the second inequality is because
[Equation]
and T 7 is deﬁned as
[Equation]
=
[Equation]
We next bound the T 7 in (28), by update rule m i = β 1 ,i m i − 1 + (1 − β 1 ,i g i ) , we have m i =
(cid:80) ik =
[( (cid:81) il =
k +1 β 1 ,l )(1 − β 1 ,k ) g k ] .
Based on that, we obtain
T 7 ≤ (cid:18) β1 − β(cid:19)
E   t (cid:88) i =
d (cid:88)
j = (cid:32)
α i − 1 m i −(cid:112)(cid:98)
v i −(cid:33)j 
 = (cid:18) β1 − β(cid:19)
E   t (cid:88) i =
d (cid:88)
j =   i − 1 (cid:88) k
= α
i − 1 (cid:16)(cid:81)
i − 1 l = k +1 β 1 ,l (cid:17) (1 − β 1 ,k ) g k (cid:112)(cid:98) v i −  j   ≤(cid:18) β1 − β(cid:19)
E   t (cid:88) i =
d (cid:88)
j =   i − 1 (cid:88)
[Equation]
where the ﬁrst inequality is due to β 1 ,t ≤ β 1 , the second equality is by substituting expression of m t , the last inequality is because ( a + b ) 2 ≤ 2( (cid:107) a (cid:107) 2 + (cid:107) b (cid:107) 2 ) , and we have introduced T 8 and T 9 for ease of notation.
In (29), we ﬁrst bound T 8 as below
T
[Equation]
(cid:32) α p g p (cid:112)(cid:98) v p (cid:33) j (cid:32)
i − 1 (cid:89) q = p + β 1 ,p (cid:33) (1 − β 1 ,p ) 
 ( i ) ≤ E   t (cid:88) i =
d (cid:88)
j = i − 1 (cid:88) k
= i − 1 (cid:88) p
= (cid:0) β
i − 1 − k(cid:1) (cid:16) β
i − 1 − p(cid:17)  (cid:18)
α k g k √ (cid:98) v k (cid:19)j + (cid:32) α p g p (cid:112)(cid:98) v p (cid:33)j   
 ( ii ) =
E   t (cid:88) i =
d (cid:88)
j = i − 1 (cid:88) k
= (cid:0) β
i − 1 − k(cid:1) (cid:18) α k g k √ (cid:98) v k (cid:19)j i − 1 (cid:88) p
= (cid:16) β
i − 1 − p(cid:17) 
 ( iii ) ≤ 1 − β 1 E   t (cid:88)
i =
d (cid:88)
j = i − 1 (cid:88) k
= (cid:0) β
i − 1 − k(cid:1) (cid:18) α k g k √ (cid:98) v k (cid:19)j   ( iv ) =
1 − β
[Equation]
E   t − 1 (cid:88) k
= d (cid:88)
[Equation]
where ( i ) is due to ab < 12 ( a 2 + b 2 ) and follows from β 1 ,t ≤ β 1 and β 1 ,t ∈ [0 , 1) , ( ii ) is due to symmetry of p and k in the summation, ( iii ) is because of (cid:80) i − 1 p =1 (cid:16) β
i − 1 − p(cid:17) ≤ 1 1 − β 1 , ( iv ) is exchanging order of summation, and the second-last inequality is due to the similar reason as ( iii ) .
For the T 9 in (29), we have
T 9 =
E   t (cid:88)
[Table]
=
[Table]
d
[Table]
=
[Table]
 
[Table]
=
[Table]
(cid:32)
[Table]
=
[Table]
β
[Table] [Table]
(cid:33) (
[Table]
)( g
[Table] [Table]
H
[Table]
E  
[Table]
H
[Table]
E  
[Table]

[Equation]
√
[Equation]
 (31) where the ﬁrst inequality holds due to β 1 ,k < 1 and | ( g k ) j | ≤ H , the second inequality holds due to β 1 ,k ≤ β 1 , and the last inequality applied the triangle inequality.
For RHS of (31), using Lemma 6.
(that will be proved later) with a i = (cid:12)(cid:12)(cid:12)(cid:12)
α i √ (cid:98) v
[Equation]
−
[Equation]
√ (cid:98) v i −(cid:12)(cid:12)(cid:12)(cid:12)
j , we further have
[Equation]

[Equation]
(cid:88)
[Equation]
(cid:88)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) α
[Equation]
√ (cid:98) v
[Equation]
− α
[Equation]
(cid:112)(cid:98) v
[Equation]
−(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) j   2 
 ≤ H 2 (cid:18) 1 − β(cid:19) 2 (cid:18) β1 − β(cid:19)
E   d (cid:88)
j = t − 1 (cid:88) i = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)
[Equation]
l √ (cid:98)
[Equation]
α l
[Equation]
(cid:112)(cid:98)
[Equation]
l
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) j   (32)
Based on (27), (29), (30) and (32), we can then bound the second term of (26) as
[Equation]
E
[Equation]
+
[Equation]
v i (cid:107)(cid:35) .
Let us turn to the ﬁrst term in (26).
Reparameterize g t as g t = ∇ f ( x t ) + δ t with E [ δ t ] = 0 , we have
E (cid:34) t (cid:88) i
= α
i (cid:104)∇ f ( x i ) , g i / (cid:112)(cid:98) v i (cid:105) (cid:35) =
E (cid:34) t (cid:88) i = α
i (cid:104)∇ f ( x i ) , ( ∇ f ( x i ) + δ i ) / (cid:112)(cid:98) v i (cid:105) (cid:35)
=
E (cid:34) t (cid:88) i = α
i (cid:104)∇ f ( x i ) , ∇ f ( x i ) /
(cid:112)(cid:98) v i (cid:105) (cid:35) + E (cid:34) t (cid:88)
i = α
i (cid:104)∇ f ( x i ) , δ i / (cid:112)(cid:98) v i (cid:105) (cid:35) .
It can be seen that the ﬁrst term in RHS of (34) is the desired descent quantity, the second term is a bias term to be bounded.
For the second term in RHS of (34), we have
[Equation]
where the last equation is because given x i ,
[Equation]
) | x i , (cid:98) v
i − 1 (cid:105)
=
and (cid:107) δ
i (cid:107) ≤ 2 H due to (cid:107) g i (cid:107) ≤ H and (cid:107)∇ f ( x i ) (cid:107) ≤ H based on Assumptions A2 and A3.
Further, we have
E (cid:34) t (cid:88) i =
[Equation]
[Equation]
(cid:112)(cid:98)
[Equation]
(cid:35)
=
E   t (cid:88) i =
d (cid:88)
j = ( ∇
[Equation] [Equation]
( δ
[Equation] [Equation]
/ ( (cid:112)(cid:98) v i )
j − α
i − 1 / ( (cid:112)(cid:98) v
i − 1 ) j ) 
 ≥ − E   t (cid:88) i
=
d (cid:88)
j = | ( ∇
[Equation] [Equation]
| | ( δ
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
( (cid:112)(cid:98)
[Equation]
i )
[Equation]
−
[Equation]
i
[Equation]
( (cid:112)(cid:98) v
i
[Equation] [Equation]
) (cid:12)(cid:12)(cid:12) ≥ −
[Equation]
E   t (cid:88)
i
[Equation]
d (cid:88)
[Equation]
(cid:12)(cid:12)(cid:12)
[Equation]
( (cid:112)(cid:98)
[Equation]
/ ( (cid:112)(cid:98)
[Equation]
(cid:12)(cid:12)(cid:12) (36)
Substituting (35) and (36) into (34), we then bound the ﬁrst term of (26) as
[Equation]
We ﬁnally apply (37) and (33) to obtain (24).
The proof is now complete. Q . D .
Lemma 6.8.
For a i ≥ 0 , β ∈ [0 , 1) , and b i = (cid:80) ik =1 β
i − k
(cid:80)
il =
k +1 a l , we have t (cid:88) i
= b 2 i ≤ (cid:18)
1 − β (cid:19) 2 (cid:18) β 1 − β (cid:19) 2 t (cid:88) i = a
i
Proof.
[Proof of Lemma 6.8]
The result is proved by following t (cid:88)
[Equation]
=
[Equation] [Equation]
[Equation]
(cid:88) l
[Equation]
a l (cid:33)(
[Equation]
(cid:88) i = (cid:32)
[Equation]
(cid:88) l = l −
[Equation]
(cid:88)
[Equation]
=
[Equation]
i
[Equation]
k a l (cid:33)
[Equation]
=
[Equation]
(cid:88) i = (cid:32) i (cid:88) l =
[Equation]
i − l +1 a l l − 1 (cid:88) k
= β l − 1 − k (cid:33)( ii
[Equation]
(cid:18)
[Equation]
(cid:19)
[Equation]
(cid:88) i = (cid:32) i (cid:88) l
= β i − l +1 a l (cid:33)= (cid:18)
[Equation]
β (cid:19) 2 t (cid:88) i = (cid:32) i (cid:88)
l = i (cid:88) m
= β i − l +1 a l β i − m +1 a m (cid:33) (
[Equation]
(cid:18)
[Equation]
(cid:19)
[Equation]
(cid:88)
i
[Equation] [Equation]
(cid:88) l
[Equation] [Equation]
(cid:88) m
[Equation] [Equation]
i
[Equation]
l
[Equation] [Equation]
i
[Equation] [Equation]
(cid:0) a
[Equation]
a
[Equation]
(cid:1) ( iv
[Equation]
(cid:18) 1 − β (cid:19)
[Equation]
(cid:88) i
=
i (cid:88) l = i (cid:88) m
=
β
i − l +1 β
i − m +1 a 2 l ( v ) ≤ (cid:18) 1 − β (cid:19)β 1 − β t (cid:88)
l
=
t (cid:88) i
=
l β i − l +1 a 2 l ≤ (cid:18) 1 − β (cid:19) 2 (cid:18) β 1 − β (cid:19) 2 t (cid:88)
l = a 2 l where ( i ) is by changing order of summation, ( ii ) is due to (cid:80) l − 1 k
=1 β l − 1 − k ≤ 1 1 − β , ( iii ) is by the fact that ab ≤ 12 ( a 2 + b 2 ) , ( iv ) is due to symmetry of a l and a m in the summation, ( v ) is because (cid:80) im =
2 β i − m +1 ≤ β 1 − β and the last inequality is for similar reason.
This completes the proof.
Q.E.D.
6.2. P ROOF OF T HEOREM 3.
Proof.
[Proof of Theorem 3.1] We combine Lemma 6.2, Lemma 6.3, Lemma 6.4, Lemma 6.5,
Lemma 6.6, and Lemma 6.7 to bound the overall expected descent of the objective.
First, from
Lemma 6.2, we have
E [ f ( z t +1 ) − f ( z 1 )] ≤ 6 (cid:88) i =
T i = − E (cid:34) t (cid:88) i
=
(cid:104)∇ f ( z i ) , β 1 ,i 1 − β 1 ,i (cid:32) α i √ (cid:98) v i − α i −(cid:112)(cid:98)
v i −(cid:33) (cid:12) m i − 1 (cid:105) (cid:35) − E (cid:34) t (cid:88) i
= α
i (cid:104)∇ f ( z i ) , g i / (cid:112)(cid:98) v i (cid:105) (cid:35) − E (cid:34) t (cid:88)
i =
(cid:104)∇ f ( z i ) , (cid:18) β 1 ,i + 1 − β 1 ,i +1 − β 1 ,i 1 − β 1 ,i (cid:19)
α
i m i / (cid:112)(cid:98) v i (cid:105) (cid:35) + E (cid:34) t (cid:88)
i =  2 L (cid:13)(cid:13) (cid:13) (cid:13) (cid:18) β 1 ,t + 1 − β 1 ,
t +1 − β 1 ,t 1 − β 1 ,t (cid:19)
α t m t / √ v t (cid:13)(cid:13) (cid:13) (cid:13) 2 (cid:35) + E   t (cid:88)
i =  2 L (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) β 1 ,i 1 − β 1 ,i (cid:32)
α t √ (cid:98) v i − α i −(cid:112)(cid:98)
v i −(cid:33) (cid:12) m i − 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + E (cid:34) t (cid:88) i =  2 L (cid:13)(cid:13)(cid:13) α
i g i / (cid:112)(cid:98) v i (cid:13)(cid:13)(cid:13)(cid:35) (38)
Then from above inequality and Lemma 6.3, Lemma 6.4, Lemma 6.5, Lemma 6.6, Lemma 6.7, we get
[Equation]
(cid:98)
[Equation]
(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) +
[Equation]
By merging similar terms in above inequality, we further have
E [ f ( z t +1 ) − f ( z 1 )] ≤ (cid:18) H 2 β 1 1 − β 1 + 2 H 2 (cid:19)
E   t (cid:88) i =
d (cid:88)
j = (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:32) α i √ (cid:98) v i − α i −(cid:112)(cid:98)
v i −(cid:33)
j (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12) + (cid:32)
+ L 2 (cid:18) 1 − β(cid:19)
2 (cid:18) β1 − β(cid:19) 2 (cid:33)
H 2 (cid:18) β1 − β(cid:19)
E  
d (cid:88)
j = t − 1 (cid:88) i = (cid:32) α i √ (cid:98) v
i − α i −(cid:112)(cid:98)
v i −(cid:33)j 
 + (cid:32)2 L + 1 2 + L 2 β 1 1 − β(cid:18) 1 − β(cid:19) 2 (cid:33)
E (cid:34) t (cid:88) i = (cid:13)(cid:13)(cid:13) α
i g i / (cid:112)(cid:98) v i (cid:13)(cid:13)(cid:13)(cid:35) + (cid:18) β1 − β 1 − β 1 ,t + 1 − β 1 ,t + (cid:19) (cid:0) H 2 + G 2 (cid:1) + (cid:18) β 1 1 − β 1 − β 1 ,t + 1 − β
[Equation]
Rearranging (39), we have
E
[Equation]
E
[Equation]
(cid:13)(cid:13)(cid:13)(cid:35)
+ (cid:18) β1 − β 1 − β 1 ,t + 1 − β 1 ,t + (cid:19)
(cid:0) H 2 + G 2 (cid:1) + (cid:18) β1 − β 1 − β 1 ,t + 1 − β 1 ,t + (cid:19)
G+ 2 H
E  
d (cid:88)
j
= ( α 1 / (cid:112)(cid:98) v 1 ) j   + E [ f ( z 1 ) − f ( z t +1 )] ≤ E  
Ct (cid:88) i = (cid:13)(cid:13)(cid:13) α t g t / (cid:112)(cid:98) v t (cid:13)(cid:13)(cid:13) 2 + Ct (cid:88) i =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
α i √ (cid:98) v i − α i −(cid:112)(cid:98)
v i −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ Ct − 1 (cid:88)
i =
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
α i √ (cid:98) v i − α i −(cid:112)(cid:98)
v i −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
  + Cwhere
C 1 (cid:44) (cid:32)2 L + 1 2 + Lβ1 − β(cid:18) 1 − β(cid:19) 2 (cid:33)
[Equation]
and z ∗ is an optimal of f , i.e. z ∗ ∈ arg min
z f ( z ) .
Using the fact that ( α i / √ (cid:98) v i ) j ≥ γ
i , ∀ j by deﬁnition, inequality (4) directly follows.
This completes the proof.
6.2. P ROOF OF C OROLLARY 3.
Proof.
[Proof of Corollary 3.1]
(S0).
Deﬁne m
= 0 , v 0 = 0 , (cid:98) v 0 = 0 ;
For t = 1 , · · · , T , do (S1).
m t
= β 1 ,t m t − 1 + (1 − β 1 ,t ) g t (S2).
v t = β 2 v t − 1 + (1 − β 2 ) g 2 t (S3).
(cid:98) v t = max { (cid:98) v t − 1 , v t } (S4).
x t +1 = x
t − α t m t / √ (cid:98) v t
End
We ﬁrst bound non-constant terms in RHS of (3), which is given by
E  
C
[Equation]
(cid:88) t = (cid:13)(cid:13)(cid:13)
[Equation]
g
[Equation]
/ (cid:112)(cid:98)
v
[Equation]
(cid:13)(cid:13)(cid:13) 2 + C
T (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98) v t −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ C
T − 1 (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98)
v t −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   + C 4 .
For the term with C 1 , assume min j ∈ [ d ] ( √ (cid:98) v 1 )
j ≥ c > 0 (this is natural since if it is 0, division byerror will happen), we have
E (cid:34)
[Equation]
= (cid:13)(cid:13)(cid:13)
α
[Equation]
g
[Equation]
/ (cid:112)(cid:98)
[Equation]
(cid:13)(cid:13)(cid:13)(cid:35) ≤ E (cid:34)
[Equation]
(cid:88) t = (cid:107) α t g t /c
(cid:107)(cid:35) =
E (cid:34) T (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
where the ﬁrst inequality is due to ( (cid:98) v t )
j ≥ ( (cid:98) v t − 1 ) j , and the last inequality is due to (cid:80) Tt =1 1 /t ≤ 1 + log T .
For the term with C 2 , we have
[Equation]
where the ﬁrst equality is due to ( (cid:98) v t )
j ≥ ( (cid:98) v t − 1 ) j and α t ≤ α t − 1 , and the second equality is due to telescope sum.
For the term with C 3 , we have
E  
T − 1 (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98)
v t −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2   ≤ E (cid:34)c
T − 1 (cid:88) t
= (cid:13) (cid:13)(cid:13) (cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98) v t −(cid:13) (cid:13)(cid:13) (cid:13)(cid:13)(cid:35) ≤ d/cwhere the ﬁrst inequality is due to | ( α t / √ (cid:98) v t − α t − 1 / (cid:112)(cid:98) v t − 1 )
j | ≤ 1 /c .
Then we have for AMSGRAD,
E  
C
T (cid:88) t
= (cid:13)(cid:13)(cid:13) α t g t / (cid:112)(cid:98) v t (cid:13)(cid:13)(cid:13) 2 + C
T (cid:88) t
= (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98) v t −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
+ C
T − 1 (cid:88) t = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
t √ (cid:98) v t −
[Equation]
t −(cid:112)(cid:98) v t −(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2  
[Equation]
≤ C 1 H 2 /c 2 (1 + log T )
+ C
[Equation]
d/c
[Equation] [Equation]
/c )
[Equation]
Now we lower bound the effective stepsizes, since (cid:98) v t is exponential moving average of g 2 t and (cid:107) g t (cid:107) ≤ H , we have ( (cid:98) v t )
j ≤ H 2 , we have
[Equation]
And thus
E
[Equation]
Then by (3)
[Equation]
(41) and (42)
[Equation]
we have
[Equation]
which is equivalent to min
[Equation]
One more thing is to verify the assumption (cid:107) α t m t / √ (cid:98) v t (cid:107) ≤ G in Theorem 3.1, since α t +1 / ( (cid:112)(cid:98) v t +1 )
j ≤ α t / ( √ (cid:98) v t )
j and α 1 / ( √ (cid:98) v 1 ) j ≤ 1 /c in the algorithm, we have (cid:107) α t m t / √ (cid:98) v t (cid:107) ≤
[Equation]
m t (cid:107) /c
≤ H/c .
This completes the proof.
6.2. P ROOF OF C OROLLARY 3.
Proof.
[Proof of Corollary 3.2]
Algorithm
(S0).
Deﬁne m
= 0 , (cid:98) v 0 = 0 ;
For t = 1 , · · · , T , do (S1).
m t
= β 1 ,t m t − 1 + (1 − β 1 ,t ) g t (S2).
(cid:98) v t = (1 − 1 /t ) (cid:98) v t − 1 + (1 /t )
g 2 t (S3).
x t +1 = x
t − α t m t / √ (cid:98) v t
End
The proof is similar to proof for Corollary 3.1, ﬁrst let’s bound RHS of (3) which is
E  
C
T (cid:88) t
= (cid:13) (cid:13)(cid:13)
α t g t / (cid:112) (cid:98) v t (cid:13)(cid:13)(cid:13) 2 + C
T (cid:88) t
= (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98) v t −(cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
+ C
T − 1 (cid:88) t
= (cid:13) (cid:13)(cid:13)(cid:13)(cid:13)
α t √ (cid:98) v t − α t −(cid:112)(cid:98) v t −(cid:13) (cid:13)(cid:13)(cid:13)(cid:13) 2   + C
We recall from Table 1 that in AdaGrad, (cid:98) v t = 1 t (cid:80) ti =1 g 2 t .
Thus, when α t = 1 / √ t , we obtain α t / √ (cid:98) v t = 1 / (cid:80) ti =1 g 2 i .
We assume min j ∈ [ d ] | ( g 1 )
j | ≥ c > 0 , which is equivalent to min j ∈ [ d ] ( √ (cid:98) v 1 )
j ≥ c > 0 (a requirement of the AdaGrad).
For C 1 term we have
E
[Equation]
t = (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
g t (cid:113)(cid:80) ti =1 g 2 i (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13) 2 
 =
E  
d (cid:88) j
=
T (cid:88) t = ( g t ) 2 j (cid:80) ti [-=
-]=
[Equation]
( g i ) 2 j   ≤
[Equation]
where the third inequality used Lemma 6.9 and the last inequality used (cid:107) g t (cid:107) ≤ H and min j ∈
[ d ] | ( g 1 ) j | ≥ c > 0 .
[Table]
[Table] [Table]
[Equation]
where the ﬁrst inequality is due to | ( α t / √ (cid:98) v t − α t − 1 / (cid:112)(cid:98) v t
[Equation]
1 ) j | ≤ 1 /c .
Now we lower bound the effective stepsizes α t / ( √ (cid:98) v t ) j ,
[Equation]
where we recall that α t = 1 / √ t and (cid:107) g t (cid:107) ≤ H .
Following the same argument in the proof of
Corollary 3.1 and the previously derived upper bounds, we have √
[Equation]
which yields
[Equation]
The last thing is to verify the assumption (cid:107) α t m t / √ (cid:98) v t (cid:107) ≤ G in Theorem 3.1, since α t +1 / ( (cid:112) (cid:98) v t +1 )
j ≤ α t / ( √ (cid:98) v t )
j and α 1 / ( √ (cid:98) v 1 ) j ≤ 1 /c in the algorithm, we have (cid:107) α t m t / √ (cid:98) v t (cid:107) ≤ (cid:107) m t (cid:107) /c
≤ H/c .
This completes the proof.
Q.E.D.
Lemma 6.9.
For a t ≥ 0 and (cid:80) ti =
a
i
cid:54)
= 0 , we have
[Equation]
Proof.
[Proof of Lemma 6.9] We will prove it by induction.
Suppose
[Equation]
we have
[Equation]
Applying the deﬁnition of concavity to log( x ) , with f ( z ) (cid:44) log( z ) , we have f ( z ) ≤ f ( z 0 ) + f (cid:48)
( z 0 )( z − z 0 )
, then substitute z = x − b, z 0 = x , we have f ( x − b ) ≤ f ( x ) + f (cid:48)
( x )( − b ) which is equivalent to log( x ) ≥ log( x − b ) + b/x for b < x , using x = (cid:80) Ti =
a i , b = a T , we have
[Equation]
and then
T (cid:88)
[Equation]
Now it remains to check ﬁrst iteration.
We have
[Equation]
This completes the proof.
Q.E.D.