This paper studies a class of adaptive gradient based momentum algorithms that update the search directions and learning rates simultaneously using past gradients. This class, which we refer to as the "Adam-type", includes the popular algorithms such as the Adam [1], AMSGrad [2] and AdaGrad [3] . Despite their popularity in training deep neural networks, the convergence of these algorithms for solving nonconvex problems remains an open question. This paper provides a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We prove that under our derived conditions, these methods can achieve the convergence rate of order O(log T / √ T ) for nonconvex stochastic optimization. We show the conditions are essential in the sense that violating them may make the algorithm diverge. Moreover, we propose and analyze a class of (deterministic) incremental adaptive gradient algorithms, which has the same O(log T / √ T ) convergence rate. Our study could also be extended to a broader class of adaptive gradient methods in machine learning and optimization. Recent works have proposed a variety of accelerated versions of GD and SGD [6, [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] 3, [21] [22] [23] [24] 1, 2] . These achievements fall into three categories: a) momentum methods [6, [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] which carefully design the descent direction ∆ t ; b) adaptive learning rate methods [20, 3, 21, 22] which determine good learning rates α t , and c) adaptive gradient methods [23, 24, 1, 2] that enjoy dual advantages of a) and b). In particular, Adam [1], belonging to the third type of methods, has become extremely popular to solve deep learning problems, e.g., to train deep neural networks. Despite its superior performance in practice, theoretical investigation of Adam-like methods for nonconvex optimization is still missing. Very recently, the work [2] pointed out the convergence issues of Adam even in the convex setting, and proposed AMSGrad, a corrected version of Adam. Although AMSGrad has made a positive step towards understanding the theoretical behavior of adaptive gradient methods, the convergence analysis of [2] was still very restrictive because it only works for convex problems, despite the fact that the most successful applications are for nonconvex problems. Apparently, there still exists a large gap between theory and practice. To the best of our knowledge, the question that whether adaptive gradient methods such as Adam, AMSGrad, AdaGrad converge for nonconvex problems is still open in theory. Related work Momentum methods take into account the history of first-order information [6, [10] [11] [12] [13] [14] [15] [16] [17] [18] [19] . A well-known method, called Nesterov's accelerated gradient (NAG) originally designed for convex deterministic optimization [6, 10, 11] , constructs the descent direction ∆ t using the difference between the current iterate and the previous iterate. A recent work [12] studied a generalization of NAG for nonconvex stochastic programming. Similar in spirit to NAG, heavy-ball (HB) methods [13] [14] [15] [16] form the descent direction vector through a decaying sum of the previous gradient information. In addition to NAG and HB methods, stochastic variance reduced gradient (SVRG) methods integrate SGD with GD to acquire a hybrid descent direction of reduced variance [17] [18] [19] . Recently, certain accelerated version of perturbed gradient descent (PAGD) algorithm is also proposed in [25] , which shows the fastest convergence rate among all Hessian free algorithms. Adaptive learning rate methods accelerate ordinary SGD by using knowledge of the past gradients or second-order information into the current learning rate α t [20, 3, 21, 22] . In [20] , the diagonal elements of the Hessian matrix were used to penalize a constant learning rate. However, acquiring the second-order information is computationally prohibitive. More recently, an adaptive subgradient method (i.e., AdaGrad) penalized the current gradient by dividing the square root of averaging of the squared gradient coordinates in earlier iterations [3] . Although AdaGrad works well when gradients are sparse, its convergence is only analyzed in the convex world. Other adaptive learning rate methods include Adadelta [21] and ESGD [22] , which lacked theoretical investigation although some convergence improvement was shown in practice. adaptive gradient methods update the descent direction and the learning rate simultaneously using knowledge in the past, and thus enjoy dual advantages of momentum and adaptive learning rate methods. Algorithms of this family include RMSProp [23], Nadam [24], and Adam [1]. Among these, Adam has become the most widely-used method to train deep neural networks (DNNs). Specifically, Adam adopts exponential moving averages (with decaying/forgetting factors) of the past gradients to update the descent direction. It also uses inverse of exponential moving average of squared past gradients to adjust the learning rate. The work [1] showed Adam converges with at most O(1/ √ T ) rate for convex problems. However, the recent work [2] pointed out the convergence issues of Adam even in the convex setting, and proposed a modified version of Adam (i.e., AMSGrad), which utilizes a non-increasing quadratic normalization and avoids the pitfalls of Adam. Although AMSGrad has made a significant progress toward understanding the theoretical behavior of adaptive gradient methods, the convergence analysis of [2] only works for convex problems. After the non-convergence issue of Adam has been raised in [2] , there are a few recent works on proposing new variants of Adam-type algorithms. In convex setting, reference [26] proposed to stabilize the coordinate-wise weighting factor to ensure convergence. Reference [27] developed an algorithm that changes the coordinate-wise weighting factor to achieve better generalization performance. Concurrent with this work, several works are trying to understand performance of Adam in nonconvex optimization problems. Reference [28] provided convergence rate of original Adam and RMSprop under full-batch (deterministic) setting, and [29] proved convergence rate of a modified version of AdaGrad where coordinate-wise weighting is removed. Contributions Differently from the aforementioned works, our work aims to build the theory to understand convergence issues for a generic class of adaptive gradient methods for nonconvex optimization. In particular, we provide a set of mild sufficient conditions that guarantee the convergence for the Adam-type methods. We summarize our contribution as follows. • We consider both stochastic version and deterministic incremental version of a class of generalized Adam, referred to as the "Adam-type", which includes the recently proposed AMSGrad [2], AdaGrad [3] , and stochastic heavy-ball methods as special cases. We show for the first time that under suitable conditions about the stepsizes and algorithm parameters, the aforementioned algorithms all converge to first-order stationary solutions of the original nonconvex problems, yielding O(log T / √ T ) convergence rate.
The recent paper REF provides a similar kind of result in the stochastic setting and for a general class of adaptive algorithms.
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization
ArXiv
ArXiv
Computer Science Mathematics