Paper:
RMSProp and ADAM continue to be extremely popular algorithms for training neural nets but their theoretical convergence properties have remained unclear. Further, recent work has seemed to suggest that these algorithms have worse generalization properties when compared to carefully tuned stochastic gradient descent or its momentum variants. In this work, we make progress towards a deeper understanding of ADAM and RMSProp in two ways. First, we provide proofs that these adaptive gradient algorithms are guaranteed to reach criticality for smooth non-convex objectives, and we give bounds on the running time.
Next we design experiments to empirically study the convergence and generalization properties of RMSProp and ADAM against Nesterov's Accelerated Gradient method on a variety of common autoencoder setups and on VGG-9 with CIFAR-10. Through these experiments we demonstrate the interesting sensitivity that ADAM has to its momentum parameter β 1 . We show that at very high values of the momentum parameter (β 1 = 0.99) ADAM outperforms a carefully tuned NAG on most of our experiments, in terms of getting lower training and test losses. On the other hand, NAG can sometimes do better when ADAM's β 1 is set to the most commonly used value: β 1 = 0.9, indicating the importance of tuning the hyperparameters of ADAM to get better generalization performance.
We also report experiments on different autoencoders to demonstrate that NAG has better abilities in terms of reducing the gradient norms, and it also produces iterates which exhibit an increasing trend for the minimum eigenvalue of the Hessian of the loss function at the iterates.

Review:
Summary:
This paper present a convergence analysis of the popular methods RMSProp and ADAM in the case of smooth non-convex functions. In particular it was shown that the above adaptive gradient algorithms are guaranteed to reach critical points for smooth non-convex objectives and bounds on the running time are provided. An empirical investigation is also presented with main focus on the comparison of the adaptive gradient methods and the Nesterov accelerated gradient algorithm (NAG).  
Comments:
Although the results are promising, I found the reading (mainly because of the not defined notation) of this paper really hard. 
In terms of presentation, the motivation in introduction is fine, but the following section named "Notations and Pseudocodes" is confusing and has many undefined notations which makes the paper very hard to read. It gives the impression that the section was added the last minute. For example what is fundtion "g" in the definition 1? What is support(v) and the diag(v) in the definition 2. the diag(v) is more obvious to me but then why at page 18 the diag(v)at the top of the page is bold (are these two things different)?
In the presentation of RMSProp what the $g_t^2$ means? Please have a look to last year's ICLR paper [Reddi, Sashank J., Satyen Kale, and Sanjiv Kumar. "On the convergence of adam and beyond." (2018).] for a more appropriate introduction of the notation.
In the introduction the authors refer to NAG as a stochastic variant of the Nesterov's acceleration and they informally present the algorithm in the end of the first paragraph. There the update rule includes stochastic gradients \nable f_i(.) while in the formal presentation in the update rule there is \nabla f(x) which is the full gradient of the objective function of the original problem. I expect this difference is somehow justified from the mentioning in the algorithm of the possibly noisy oracle but this is never mention in the main text.
If the above statements in terms of presentation, are ignored the convergence results and numerical experiments are interesting. 
However, the numerical evaluation does not correspond to the theoretical results. It is a comparison of NAG ,ADAM and RMSPROP with interesting conclusions  that can be beneficial for practitioners that they use these methods.
Some missing references:
On Adam methods:
1) Chen, Xiangyi, et al. "On the convergence of a class of adam-type algorithms for non-convex optimization." arXiv preprint arXiv:1808.02941 (2018).
2) Zhou, Dongruo, et al. "On the convergence of adaptive gradient methods for nonconvex optimization." arXiv preprint arXiv:1808.05671 (2018).
On momentum (heavy ball) methods:
3) Loizou, Nicolas, and Peter Richtárik. "Momentum and stochastic momentum for stochastic gradient, Newton, proximal point and subspace descent methods." arXiv preprint arXiv:1712.09677 (2017).