Paper:
Many automated machine learning methods, such as those for hyperparameter and neural architecture optimization, are computationally expensive because they involve training many different model configurations. In this work, we present a new method that saves computational budget by terminating poor configurations early on in the training. In contrast to existing methods, we consider this task as a ranking and transfer learning problem. We qualitatively show that by optimizing a pairwise ranking loss and leveraging learning curves from other data sets, our model is able to effectively rank learning curves without having to observe many or very long learning curves. We further demonstrate that our method can be used to accelerate a neural architecture search by a factor of up to 100 without a significant performance degradation of the discovered architecture. In further experiments we analyze the quality of ranking, the influence of different model components as well as the predictive behavior of the model.

Review:
The paper proposes a new method to rank learning curves of neural networks which can be used to speed up neural architecture search.
Compared to previous work, the learning curve model not only takes hyperparameter configurations into account, but by training it on offline generated data, it is able to model learning curves across different datasets.
Applied to a simple neural architecture search strategy, the proposed method achieves higher speed-ups on image classification tasks than other methods from the literature.
While the method seems interesting, I don't think the paper is ready for acceptance yet, since it a) misses some important details and b) the empirical evaluation is not sufficient.
More precisely the following points need to be addressed:
 - In section 3, the paper says that all automated methods follow broadly the same principal that the first model is trained to completion. This is not correct, commonly used methods, such as Hyperband (Li et al.) or BOHB (Falkner et al.), use successive halving (Jamieson et al.) which trains a batch of configurations for just a minimum budget and then already discards poorly performing configurations. I also miss a discussion of these methods in the related work section.
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization
Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., and Talwalkar, A.
Journal of Machine Learning Research 2018
BOHB: Robust and efficient hyperparameter optimization at scale
S Falkner, A Klein, F Hutter
Proceedings of the 35th International Conference on Machine Learning
Non-stochastic best arm identification and hyperparameter optimization
K Jamieson, A Talwalkar
Artificial Intelligence and Statistics, 240-248
 - Is the learning curve model updated during the optimization process with the new observed data? If not, is there any way the model can adapt to new data? Also, what happens if learning curves are fundamentally different to the training data, e.g if different learning rate schedules are used than for generating the offline data?
 - A simple baseline that is missing, is to use the last observed value as approximation for the final performance which often works competitively to more powerful learning curve extrapolation methods (e.g see Klein et al.).
 - The method is only applied for a simple random search strategy, however, in practice, one would use more sophisticated methods, such as for example Bayesian optimization. The question is, how effective is the proposed method in this setting, since the distribution of learning curves might be dramatically different and biased towards more well-performing configurations with almost identical learning curves.
- I think the experiments would be more convincing, if the method shows strong performance when deploy in commonly used NAS/HPO methods, such as Hyperband or successive halving. This should be straight forward, since decisions which configuration is promoted to a higher budget can be made based on the model instead of just the last observed value.
- What is delta in the experiments? How sensitive is this hyperparameter in practice?
Minor comments:
- Related to this paper is the work by Gargiani et al. which also models learning curves across datasets on offline generated data
 Probabilistic Rollouts for Learning Curve Extrapolation Across Hyperparameter Settings
 M Gargiani, A Klein, S Falkner, F Hutter
 arXiv preprint arXiv:1910.04522
- Figure 5: visually it seems that all learning curves are almost identical, maybe it would be better if the plot could zoom in at least for the final stage of training.
after the rebuttal
-----------------------
I thank the authors for answering my  questions. In the rebuttal the authors addressed my concerned about the insufficient empirical evaluation and included other relevant baselines. I will increase my score.