Paper:
From a simplified analysis of adaptive methods, we derive AvaGrad, a new optimizer which outperforms SGD on vision tasks when its adaptability is properly tuned. We observe that the power of our method is partially explained by a decoupling of learning rate and adaptability, greatly simplifying hyperparameter search. In light of this observation, we demonstrate that, against conventional wisdom, Adam can also outperform SGD on vision tasks, as long as the coupling between its learning rate and adaptability is taken into account. In practice, AvaGrad matches the best results, as measured by generalization accuracy, delivered by any existing optimizer (SGD or adaptive) across image classification (CIFAR, ImageNet) and character-level language modelling (Penn Treebank) tasks. This later observation, alongside of AvaGrad's decoupling of hyperparameters, could make it the preferred optimizer for deep learning, replacing both SGD and Adam.

Review:
In this paper the authors develop variants of Adam which corrects for the relationship of the gradient and adaptive terms that causes convergence issues, naming them Delayed Adam and AvaGrad. They also provide proofs demonstrating they solve the convergence issues of Adam in O(1/sqrt(T)) time. They also introduce a convex problem where Adam fails to converge to a stationary point.
This paper is clearly written and has reasonable experimental support of its claims. However in terms of novelty, AdaShift was published at ICLR last year (https://openreview.net/forum?id=HkgTkhRcKQ) and seems to include a closely related analysis and update rule of your proposed optimizers. In AdaShift instead of correcting for the correlation between the gradient and eta_t, they correct for the relationship between the gradient and the second moment term v_t. Could you further clarify the differences between the two, both in your approach to deriving the new update rule and the algorithms themselves? Additionally, their Theorem 1 could be compared to yours, but noting the differences for these seems less important. If the optimizers are unique enough, including AdaShift in your experiments would be very useful for demonstrating their differences.
Regarding experiments, while it is true that adaptive methods are supposed to be less “sensitive” to hyperparameter choices, the limits of the feasible ranges for each hyperparameter could vary drastically across problems (especially, as previously demonstrated, across different batch sizes.) Thus, not retuning across experiments seems like it could negatively affect performance for any of the transferred hyperparameter settings. Instead of demonstrating hyperparameter insensitivity by carrying over hyperparameter settings, one could instead retune for each problem and show that a higher percent of hyperparameter combinations result in the same/similar best performance (similar to what is done in Fig. 2, but also showing a (1-dimensional) SGD version which would presumably contain fewer high performing settings.)
Some additional comments:
-The contribution of Theorem 1 is a nice addition to the literature.
-Your tuning of epsilon is great! I believe more papers should include epsilon in their hyperparameter sweeps.
-Scaling epsilon with step size makes sense when considering that Adam is similar to a trust region method, where epsilon is inversely proportional to the trust region radius. However, in section 5 implying that epsilon should be as large as possible in the worst case seems like an odd result given that this would always diminish your second moment term as much as possible, defeating the point of the additional Adam-like adaptivity. Can you comment on why this diminished adaptivity would be desirable in the worst case scenario analyzed?
-The synthetic toy problem is much appreciated, more papers should start with a small, interpretable experiment.
-Was SGD with momentum used? If not, this may not be a fair comparison, as I believe it is much more common to use momentum with SGD. If momentum was used, was the momentum hyperparameter tuned? If not, this may be advantageous to the Adam based methods, as they have more versatile adaptability and thus may not need as much care with their selection of momentum values.
-Was a validation set used for CIFAR? You note in appendix C that there are 50k train and 10k test. You mention validation performance in the main text, so this is just double checking.
-The demonstration in figures 2, 3 of decoupling the step size and epsilon in interesting! Given that the best performing values seem to be on the edges of the ranges tested, I would be curious if the trend continues for more extreme values of alpha and epsilon (one could sparsely search along the predicted trendlines.)
Nits:
-“Vanilla SGD is still prevalent, in spite of the development of seemingly more sophisticated adaptive alternatives...” This could use some citations to back up the claim, because as far as I know it is much more common to use SGD with momentum and is actually rare to use vanilla SGD (the DenseNets and Resnets citations use momentum=0.9.)
-It would be nice to highlight in color the diff from vanilla Adam in the Algorithm sections.
-It is not super clear from the text how in eq 26 you get \sum{E[f(w_t)|Z]} = f(w_1)
-I may be misreading something, but I believe the H in the leftmost term in the last line of eq 33 should be an L.
-In section 5, “for a fixed learning rate (as is typically done in practice, except for discrete decays during training)” seems like an overly broad claim, given that authors commonly use polynomial, linear, exponential, cosine, or other learning rate decay/warmups. Granted for some CIFAR and ImageNet benchmarks there are more common discrete learning rate schedules, but that does not seem to be the overwhelmingly prevalent technique.
Overall, while this area of analyzing Adam and proposing modifications is a popular and crowded subject, I believe this paper may contribute to it if my concerns are addressed. While I currently do not recommend acceptance, I am open to changing my score after considering the author comments!