Paper:
Neural Architecture Search (NAS) is an exciting new field which promises to be as much as a game-changer as Convolutional Neural Networks were in 2012. Despite many great works leading to substantial improvements on a variety of tasks, comparison between different methods is still very much an open issue. While most algorithms are tested on the same datasets, there is no shared experimental protocol followed by all. As such, and due to the under-use of ablation studies, there is a lack of clarity regarding why certain methods are more effective than others. Our first contribution is a benchmark of 8 NAS methods on 5 datasets. To overcome the hurdle of comparing methods with different search spaces, we propose using a method's relative improvement over the randomly sampled average architecture, which effectively removes advantages arising from expertly engineered search spaces or training protocols. Surprisingly, we find that many NAS techniques struggle to significantly beat the average architecture baseline. We perform further experiments with the commonly used DARTS search space in order to understand the contribution of each component in the NAS pipeline. These experiments highlight that: (i) the use of tricks in the evaluation protocol has a predominant impact on the reported performance of architectures; (ii) the cell-based search space has a very narrow accuracy range, such that the seed has a considerable impact on architecture rankings; (iii) the hand-designed macrostructure (cells) is more important than the searched micro-structure (operations); and (iv) the depth-gap is a real phenomenon, evidenced by the change in rankings between 8 and 20 cell architectures. To conclude, we suggest best practices, that we hope will prove useful for the community and help mitigate current NAS pitfalls, e.g. difficulties in reproducibility and comparison of search methods. We provide the code used for our experiments at link-to-come.

Review:
The paper scrutinizes commonly used evaluation strategies for neural architecture search.
The first contribution is to compare architectures found by 5 different search strategies from the literature against randomly sampled architectures from the same underlying search space.
The paper shows that across different image classification datasets, the most neural architecture search methods are not consistently finding solutions that achieve a better performance than random architectures.
The second major contribution of the paper is to show that the state-of-the-art performance achieved by the many neural architecture search methods can be largely attributed to advanced regularization tricks.
In general the paper is well written and easy to follow.
The paper sheds a rather grim light on the current state of neural architecture search, but I think it could raise awareness of common pitfalls and help to make future work more rigorous.
While poorly designed search spaces is maybe a problem that many people in the community are aware of, this is, to the best of my knowledge, the first paper that systematically shows that for several commonly used search spaces there is not much to be optimized.
Besides that, the paper shows that, maybe not surprisingly, the training protocol seems to be more important than the actual architecture search, which I also found rather worrisome.
It would be nice, if Figure 3 could include another bar that shows the performance of a manually designed architecture, for example a residual network, trained with the same regularization techniques.
A caveat of the paper is that mostly local methods with weight-sharing are considered, instead of more global methods, such as evolutionary algorithms or Bayesian optimization, which also showed strong performance on neural architecture search problems in the past.
Furthermore, the paper doesn't mention some recent work in neural architecture search that present more thoroughly designed search spaces, e.g Ying et al.
It would also be helpful if the paper could elaborate, on how better search spaces could be designed.
Nas-bench-101: Towards reproducible neural architecture search
C Ying, A Klein, E Real, E Christiansen, K Murphy, F Hutter
ICML 2019
Minor comments:
- Section 4.1 How are the hyperparameters of drop path probability, cutout length and auxiliary tower weight chosen?
post rebuttal
------------------
I thank the authors for taking the time to answer my questions and performing the additional experiments. The paper highlights two important issues in the current NAS literature: evaluating methods with different search spaces and non-standardised training protocols. I do think that the paper makes an important contribution which hopefully helps future work to over come these flaws.