  Early methods in the rapidly developing field of neural architecture search
(NAS) required fully training thousands of neural networks. To reduce this
extreme computational cost, dozens of techniques have since been proposed to
predict the final performance of neural architectures. Despite the success of
such performance prediction methods, it is not well-understood how different
families of techniques compare to one another, due to the lack of an
agreed-upon evaluation metric and optimization for different constraints on the
initialization time and query time. In this work, we give the first large-scale
study of performance predictors by analyzing 31 techniques ranging from
learning curve extrapolation, to weight-sharing, to supervised learning, to
"zero-cost" proxies. We test a number of correlation- and rank-based
performance measures in a variety of settings, as well as the ability of each
technique to speed up predictor-based NAS frameworks. Our results act as
recommendations for the best predictors to use in different settings, and we
show that certain families of predictors can be combined to achieve even better
predictive power, opening up promising research directions. Our code, featuring
a library of 31 performance predictors, is available at
https://github.com/automl/naslib.
