Paper:
There is no consensus yet on the question whether adaptive gradient methods like Adam are easier to use than non-adaptive optimization methods like SGD. In this work, we fill in the important, yet ambiguous concept of 'ease-of-use' by defining an optimizer's tunability: How easy is it to find good hyperparameter configurations using automatic random hyperparameter search? We propose a practical evaluation protocol for optimizer tunability that can form the basis for a fair optimizer benchmark. Evaluating a variety of optimizers on an extensive set of standard datasets and architectures, we find that Adam is the most tunable for the majority of problems, especially with a low budget for hyperparameter tuning.

Review:
This paper introduces a simple measure of tunability that allows to compare optimizers under varying resource constraints. The tunability of the optimizer is a weighted sum of best performance at a given budget. The authors found that in a setting with low budget for hyperparameter tuning, tuning only Adam optimizer’s learning rate is likely to be a very good choice; it doesn’t guarantee the best possible performance, but it is evidently the easiest to find well-performing hyperparameter configurations.
Comments:
The paper is easy to follow. The motivation of defining tunability of optimizer is a very interesting question, however, the study seems to preliminary and the conclusion is not quite convencing due to several reasons:
In section 3.2, to characterize its difficulties of finding best hyperparameters or tunability, the authors seem to try to connect the concept of “sharpness” of a minima in loss surface to the tunability of an optimizer, which is similar to comparing the loss landscape of minimums. However, while the authors made intuitive explanation about the tunability in section 2.2, I did not see the actual plot of the true hyperpaparameter loss surface of each optimizer to verify these intuitions. Can the author be more specific about the x-axis in the illustration 1.a and 1.b? If I understand correctly, they are not the number of trails.
In addition, the proposed stability metric seems not quite related with the above intuitions, as the illustrations (1.a and 1b) define the tunability to be the flatness of hyperparameter space around the best configurations, but the proposed definition is a weighted sum of the incumbents in terms of the HPO budgets. 
The definition of the tuning budgets is not clear, is it the number of trials or the time/computation budgets? The authors seems interchangeably using “runs” and “iterations”, which makes the concept more confusable. 
The authors further proposed three weighting schemes to emphasize the tunability of different stage of HPO. My concern is that is highly dependent  on the order of hyperparameter searched, which could impact the tunability significantly. For instance, in case of grid search HPO and 0.1 is the best learning rate, different search order such as [10, 1, 0.01, 0.1] and [0.1, 0.01, 1, 10] could results in dramatic different CPE and CPL. 
My major concern is the hyperparameter distributions for each optimizer highly requires prior knowledge. A good prior of one optimizer could significantly affect the HPO cost or increase the tunability, i.e., the better understanding the optimizer, the less tuning cost. My major concern is that the authors assume the hyperparameters to be independent (section 3.2), which is not necessarily true. Actually hyperparameters are highly correlated, such as momentum, batch size and learning rate are correlated in terms of effective learning rate [1,2], so as weight decay and learning rate are [3], which means using non-zero momentum is equivalent to using large learning rate as long as the effective learning rate is the same. This could significantly increase the tunability of SGDM. Another concurrent submission [4] verified this equivalence and showed one can also just tune learning rate for SGDM. The assumption of independent hyperparameters might be fine for black box optimization or with the assumption that practitioners have no knowledge of the importance of each hyperparameter, then the tunability of the optimizer could be different based on the prior knowledge of hyperparameter and their correlations. But it is not rigorous enough to make the conclusion that Adam is easier to tune than SGD.
The author states their method to determine the priors by training each task specified in the DEEPOBS with a large number of hyperparameter samplings and retain the hyperparameters which resulted in performance within 20% of the best performance obtained. Could the authors be more specific on the hyperparameters searched? Is this process counted in the tunability measurement?
[1] Smith and Le, A Bayesian Perspective on Generalization and Stochastic Gradient Descent, https://arxiv.org/abs/1710.06451
[2] Smith et al, Don't Decay the Learning Rate, Increase the Batch Size, https://arxiv.org/abs/1711.00489
[3] van Laarhoven et al, L2 Regularization versus Batch and Weight Normalization, https://arxiv.org/abs/1706.05350
[4] Rethinking the Hyperparameters for Fine-tuning
 https://openreview.net/forum?id=B1g8VkHFPH