Original: NAS-B ENCH -S UITE : NAS E VALUATION IS (N OW ) S URPRISINGLY E ASY
Revised: NAS-B ENCH -S UITE : NAS E VALUATION IS (N OW ) S URPRISINGLY E ASY

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench201, has signiﬁcantly lowered the computational overhead for conducting scientiﬁc research in neural architecture search (NAS).
Revised: The release of tabular benchmarks, such as NAS-Bench-101 and NAS-Bench201, has signiﬁcantly lowered the computational overhead for conducting scientiﬁc research in neural architecture search (NAS).

Original: Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classiﬁcation.
Revised: Although they have been widely adopted and used to tune real-world NAS algorithms, these benchmarks are limited to small search spaces and focus solely on image classiﬁcation.

Original: Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing.
Revised: Recently, several new NAS benchmarks have been introduced that cover significantly larger search spaces over a wide range of tasks, including object detection, speech recognition, and natural language processing.

Original: However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks.
Revised: However, substantial differences among these NAS benchmarks have so far prevented their widespread adoption, limiting researchers to using just a few benchmarks.

Original: In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, ﬁnding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks.
Revised: In this work, we present an in-depth analysis of popular NAS algorithms and performance prediction methods across 25 different combinations of search spaces and datasets, ﬁnding that many conclusions drawn from a few NAS benchmarks do not generalize to other benchmarks.

Original: To help remedy this problem, we introduce
Revised: To help remedy this problem, we introduce

Original: NAS-Bench-Suite , a comprehensive and extensible collection of NAS benchmarks, accessible through a uniﬁed interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research.
Revised: NAS-Bench-Suite , a comprehensive and extensible collection of NAS benchmarks, accessible through a uniﬁed interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research.

Original: Our code is available at https://anonymous.4open.science/r/NAS-Bench-Suite .
Revised: Our code is available at https://anonymous.4open.science/r/NAS-Bench-Suite .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Automated methods for neural network design, referred to as neural architecture search (NAS), have been used to ﬁnd architectures that are more efﬁcient and more accurate than the best manually designed architectures (Zoph et al., 2018; Real et al., 2019;
Revised: Automated methods for neural network design, referred to as neural architecture search (NAS), have been used to ﬁnd architectures that are more efﬁcient and more accurate than the best manually designed architectures (Zoph et al., 2018; Real et al., 2019;

Original: So et al., 2019).
Revised: So et al., 2019).

Original: However, it is notoriously challenging to provide fair comparisons among NAS methods due to potentially high computational complexity (Zoph & Le, 2017; Real et al., 2019) and the use of different training pipelines and search spaces (Li & Talwalkar, 2019; Lindauer & Hutter, 2020), resulting in the conclusion that “NAS evaluation is frustratingly hard”
Revised: However, it is notoriously challenging to provide fair comparisons among NAS methods due to potentially high computational complexity (Zoph & Le, 2017; Real et al., 2019) and the use of different training pipelines and search spaces (Li & Talwalkar, 2019; Lindauer & Hutter, 2020), resulting in the conclusion that “NAS evaluation is frustratingly hard”

Original: (Yang et al., 2020).
Revised: (Yang et al., 2020).

Original: To make fair, statistically sound comparisons of NAS methods more accessible, tabular NAS benchmarks have been released; these exhaustively evaluate all architectures in a given search space, storing the relevant training metrics in a lookup table (Ying et al., 2019; Dong & Yang, 2020;
Revised: To make fair, statistically sound comparisons of NAS methods more accessible, tabular NAS benchmarks have been released; these exhaustively evaluate all architectures in a given search space, storing the relevant training metrics in a lookup table (Ying et al., 2019; Dong & Yang, 2020;

Original: Zela et al., 2020b; Mehrotra et al., 2021).
Revised: Zela et al., 2020b; Mehrotra et al., 2021).

Original: This substantially lowers the computational overhead of NAS experiments, since the performance of an architecture can be found simply by querying these tables, hence allowing for a rigorous comparison of various NAS algorithms with minimal computation.
Revised: This substantially lowers the computational overhead of NAS experiments, since the performance of an architecture can be found simply by querying these tables, hence allowing for a rigorous comparison of various NAS algorithms with minimal computation.

Original: While early tabular NAS benchmarks, such as NAS-Bench-101 (Ying et al., 2019) and NAS-Bench201 (Dong & Yang, 2020), have been widely adopted by the community, they are limited to small search spaces and focus solely on image classiﬁcation.
Revised: While early tabular NAS benchmarks, such as NAS-Bench-101 (Ying et al., 2019) and NAS-Bench201 (Dong & Yang, 2020), have been widely adopted by the community, they are limited to small search spaces and focus solely on image classiﬁcation.

Original: Recently, benchmarks have been introduced for natural language processing (Klyuchnikov et al., 2020), speech recognition (Mehrotra et al., 2021), object detection, and self-supervised tasks (Duan et al., 2021).
Revised: Recently, benchmarks have been introduced for natural language processing (Klyuchnikov et al., 2020), speech recognition (Mehrotra et al., 2021), object detection, and self-supervised tasks (Duan et al., 2021).

Original: Furthermore, the release of surrogate NAS benchmarks (Siems et al., 2020; Yan et al., 2021), which estimate the performance of all architectures in a search space via a surrogate model, has removed the constraint of exhaustively evaluating the entire search space, expanding the scope of possible search space sizes to 10 18 and beyond.
Revised: Furthermore, the release of surrogate NAS benchmarks (Siems et al., 2020; Yan et al., 2021), which estimate the performance of all architectures in a search space via a surrogate model, has removed the constraint of exhaustively evaluating the entire search space, expanding the scope of possible search space sizes to 10 18 and beyond.

Original: However, substantial differences in the abstractions (such as whether a node or an edge denotes an operation), capabilities (such as whether all, or only some, of the architectures can be queried), and implementations (such as incompatible deep learning libraries) have so far prevented [Figure]
Revised: However, substantial differences in the abstractions (such as whether a node or an edge denotes an operation), capabilities (such as whether all, or only some, of the architectures can be queried), and implementations (such as incompatible deep learning libraries) have so far prevented Black-Box Random Search

Original: Reg.
Revised: Reg.

Original: Evolution BANANASLocal
Revised: Evolution BANANASLocal

Original: Search NPENAS
Revised: Search NPENAS

Original: NAS Methods
Revised: NAS Methods

Original: One-Shot
Revised: One-Shot

Original: DARTSGDASDrNAS
Revised: DARTSGDASDrNAS

Original: GPRFBOHAMIANN
Revised: GPRFBOHAMIANN

Original: XGBNAO
Revised: XGBNAO

Original: Other CV tasks
Revised: Other CV tasks

Original: TransNASBenchNASBenchMR
Revised: TransNASBenchNASBenchMR

Original: NASBenchNLP
Revised: NASBenchNLP

Original: NASBenchASR
Revised: NASBenchASR

Original: Img.
Revised: Img.

Original: Cls
Revised: Cls

Original: Speech
Revised: Speech

Original: NLP
Revised: NLP

Original: NASBench101NASBench201DARTS Algorithm
Revised: NASBench101NASBench201DARTS Algorithm

Original: PerformancePredictors
Revised: PerformancePredictors

Original: NAS Benchmarks
Revised: NAS Benchmarks

Original: Evaluate
Revised: Evaluate

Original: BBO Predictors
Revised: BBO Predictors

Original: OneShot
Revised: OneShot

Original: Model-Based
Revised: Model-Based

Original: RF | GP || |
Revised: RF | GP || |

Original: RS | RE DARTS| GDAS | | || | | | | | | | | | || | || nearly all research in NAS from providing results on more than two families of benchmarks.
Revised: RS | RE DARTS| GDAS | | || | | | | | | | | | || | || nearly all research in NAS from providing results on more than two families of benchmarks.

Original: Overall, the lack of consistency in “NAS-Bench” datasets has signiﬁcantly slowed their collective adoption.
Revised: Overall, the lack of consistency in “NAS-Bench” datasets has signiﬁcantly slowed their collective adoption.

Original: In this work, we show that there is a need to adopt newer benchmarks because many conclusions drawn from a small subset of benchmarks do not generalize across diverse datasets and tasks.
Revised: In this work, we show that there is a need to adopt newer benchmarks because many conclusions drawn from a small subset of benchmarks do not generalize across diverse datasets and tasks.

Original: Speciﬁcally, we present an in-depth analysis of popular black-box (Real et al., 2019; White et al., 2021a; Ottelander et al., 2021), one-shot (Liu et al., 2019b; Chen et al., 2021; Dong & Yang, 2019), and performance prediction methods (White et al., 2021c) across 25 different combinations of search spaces and datasets.
Revised: Speciﬁcally, we present an in-depth analysis of popular black-box (Real et al., 2019; White et al., 2021a; Ottelander et al., 2021), one-shot (Liu et al., 2019b; Chen et al., 2021; Dong & Yang, 2019), and performance prediction methods (White et al., 2021c) across 25 different combinations of search spaces and datasets.

Original: To the best of our knowledge, this is by far the largest set of NAS search spaces and datasets on which experiments have been conducted to date.
Revised: To the best of our knowledge, this is by far the largest set of NAS search spaces and datasets on which experiments have been conducted to date.

Original: We show that many implicit assumptions in the NAS community are wrong.
Revised: We show that many implicit assumptions in the NAS community are wrong.

Original: First, conclusions drawn from NAS-Bench-101 and NAS-Benchoften do not generalize to other search spaces.
Revised: First, if a NAS algorithm does well on NAS-Bench101 and NAS-Bench-201, it does not necessarily perform well on other search spaces.

Original: Similarly, there is often no single NAS method that performs well across the majority of search spaces. Finally, tuning the hyperparameters for a NAS method can give a substantial performance boost, but transferring these hyperparameters to other search spaces often make the NAS method perform signiﬁcantly worse.
Revised: Second, NAS algorithms may not have robust default hyperparameters and therefore require tuning. Finally, tuning the hyperparameters of a NAS method on one search space and transferring these hyperparameters to other search spaces often make the NAS method perform signiﬁcantly worse.

Original: In order to help NAS researchers and practitioners avoid these pitfalls, we release the NAS Benchmark
Revised: In order to help NAS researchers and practitioners avoid these pitfalls, we release the NAS Benchmark

Original: Suite ( NAS-Bench-Suite ), a comprehensive and extensible collection of NAS benchmarks, accessible through a uniﬁed interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research.
Revised: Suite ( NAS-Bench-Suite ), a comprehensive and extensible collection of NAS benchmarks, accessible through a uniﬁed interface, created with the aim to facilitate reproducible, generalizable, and rapid NAS research.

Original: Our work eliminates the overhead for NAS research to evaluate on several different datasets and problem types, helping the community to develop NAS methods that generalize to new problem types and unseen datasets.
Revised: Our work eliminates the overhead for NAS research to evaluate on several different datasets and problem types, helping the community to develop NAS methods that generalize to new problem types and unseen datasets.

Original: See Figure 1 for an overview.
Revised: See Figure 1 for an overview.

Original: To ensure reproducibility and other best practices, we release our code and adhere to the NAS best practices checklist (Lindauer & Hutter, 2020, see Section 8 for details).
Revised: To ensure reproducibility and other best practices, we release our code and adhere to the NAS best practices checklist (Lindauer & Hutter, 2020, see Section 8 for details).

Original: Our contributions.
Revised: Our contributions.

Original: We summarize our main contributions below.
Revised: We summarize our main contributions below.

Original: • We conduct a comprehensive study of the generalizability of NAS algorithms and their hyperparameters across 25 settings, showing that it is often not sufﬁcient to tune on just a few benchmarks, and showing that the best hyperparameters depend on the speciﬁc search space.
Revised: • We conduct a comprehensive study of the generalizability of NAS algorithms and their hyperparameters across 25 settings, showing that it is often not sufﬁcient to tune on just a few benchmarks, and showing that the best hyperparameters depend on the speciﬁc search space.

Original: • We introduce a uniﬁed benchmark suite, NAS-Bench-Suite , which implements 25 different combinations of search spaces and datasets.
Revised: • We introduce a uniﬁed benchmark suite, NAS-Bench-Suite , which implements 25 different combinations of search spaces and datasets.

Original: By making it easy to quickly and comprehensively evaluate new NAS algorithms on a broad range of problems, our benchmark suite can improve experimental rigor and generalizability in NAS research.
Revised: By making it easy to quickly and comprehensively evaluate new NAS algorithms on a broad range of problems, our benchmark suite can improve experimental rigor and generalizability in NAS research.

Original:  NAS B ENCHMARKS O VERVIEW
Revised:  NAS B ENCHMARKS O VERVIEW

Original: Preliminaries.
Revised: Preliminaries.

Original: A search space in NAS is the set of all architectures that the NAS algorithm is allowed to select.
Revised: A search space in NAS is the set of all architectures that the NAS algorithm is allowed to select.

Original: Most recent search spaces are deﬁned by a cell-based (micro) structure and a macro structure.
Revised: Most recent search spaces are deﬁned by a cell-based (micro) structure and a macro structure.

Original: A cell is a small set of neural network operations arranged in a directed acyclic graph (DAG), with constraints on the number of nodes, edges, and incoming edges per node.
Revised: A cell is a small set of neural network operations arranged in a directed acyclic graph (DAG), with constraints on the number of nodes, edges, and incoming edges per node.

Original: The macro structure consists of the architecture skeleton and the arrangement of cells, such as how many times each cell is duplicated.
Revised: The macro structure consists of the architecture skeleton and the arrangement of cells, such as how many times each cell is duplicated.

Original: For many popular search spaces, the macro structure is completely ﬁxed, while for other search spaces, the macro structure can have variable length, width, and number of channels for different architectures in the search space.
Revised: For many popular search spaces, the macro structure is completely ﬁxed, while for other search spaces, the macro structure can have variable length, width, and number of channels for different architectures in the search space.

Original: [Table] A NAS benchmark (Lindauer & Hutter, 2020) consists of a dataset (with a ﬁxed train-test split), a search space
Revised: A NAS benchmark (Lindauer & Hutter, 2020) consists of a dataset (with a ﬁxed train-test split), a search space, and a ﬁxed evaluation pipeline with predeﬁned hyperparameters for training the

Original: [Table]
Revised: [Table]

Original: and a ﬁxed evaluation pipeline with predeﬁned hyperparameters for training the architectures.
Revised: architectures.

Original: A tabular NAS benchmark is one that additionally provides precomputed evaluations with that training pipeline for all possible architectures in the search space.
Revised: A tabular NAS benchmark is one that additionally provides precomputed evaluations with that training pipeline for all possible architectures in the search space.

Original: Finally, a surrogate
Revised: Finally, a surrogate

Original: NAS benchmark (Siems et al., 2020; Yan et al., 2021) is a NAS benchmark that provides a surrogate model that can be used to predict the performance of any architecture in the search space.
Revised: NAS benchmark (Siems et al., 2020; Yan et al., 2021) is a NAS benchmark that provides a surrogate model that can be used to predict the performance of any architecture in the search space.

Original: We say that a NAS benchmark is queryable if it is either a tabular or surrogate benchmark.
Revised: We say that a NAS benchmark is queryable if it is either a tabular or surrogate benchmark.

Original: Queryable NAS benchmarks can be used to simulate NAS experiments very cheaply by querying the performance of neural networks (using a table or a surrogate) instead of training the neural networks from scratch.
Revised: Queryable NAS benchmarks can be used to simulate NAS experiments very cheaply by querying the performance of neural networks (using a table or a surrogate) instead of training the neural networks from scratch.

Original: NAS benchmarks.
Revised: NAS benchmarks.

Original: Now we describe the characteristics of many popular NAS benchmarks.
Revised: Now we describe the characteristics of many popular NAS benchmarks.

Original: For a summary, see Table 1, and for a more comprehensive and detailed survey, see Appendix A.
Revised: For a summary, see Table 1, and for a more comprehensive and detailed survey, see Appendix A.

Original: The ﬁrst tabular NAS benchmark to be released was NAS-Bench-101 (Ying et al., 2019).
Revised: The ﬁrst tabular NAS benchmark to be released was NAS-Bench-101 (Ying et al., 2019).

Original: This benchmark consists of 423 624 architectures trained on CIFAR-10.
Revised: This benchmark consists of 423 624 architectures trained on CIFAR-10.

Original: The cell-based search space consists of a directed acyclic graph (DAG) structure in which the nodes can take on operations.
Revised: The cell-based search space consists of a directed acyclic graph (DAG) structure in which the nodes can take on operations.

Original: A follow-up work, NAS-Bench-1Shot

Revised: A follow-up work, NAS-Bench-1Shot

Original: (Zela et al., 2020b), deﬁnes three subsets of NAS-Benchwhich allow one-shot algorithms to be run.
Revised: (Zela et al., 2020b), deﬁnes three subsets of NAS-Benchwhich allow one-shot algorithms to be run.

Original: The largest subset size in NAS-Bench-1Shot1 is 363 648 .
Revised: The largest subset size in NAS-Bench-1Shot1 is 363 648 .

Original: NAS-Bench-201 Dong & Yang (2020) is another popular tabular NAS benchmark.
Revised: NAS-Bench-201 Dong & Yang (2020) is another popular tabular NAS benchmark.

Original: The cell-based search space consists of a DAG where each edge can take on operations (in contrast to NAS-Bench101, in which the nodes are operations).
Revised: The cell-based search space consists of a DAG where each edge can take on operations (in contrast to NAS-Bench101, in which the nodes are operations).

Original: The number of non-isomorphic architectures is 6 466 and all are trained on CIFAR-10, CIFAR-100, and ImageNet-16-120.
Revised: The number of non-isomorphic architectures is 6 466 and all are trained on CIFAR-10, CIFAR-100, and ImageNet-16-120.

Original: NATS-Bench
Revised: NATS-Bench

Original: (Dong et al., 2021) is an extension of NAS-Bench-201 which also varies the macro architecture.
Revised: (Dong et al., 2021) is an extension of NAS-Bench-201 which also varies the macro architecture.

Original: NAS-Bench-NLP (Klyuchnikov et al., 2020) is a NAS benchmark for natural language processing, which is size 10 53 .
Revised: NAS-Bench-NLP (Klyuchnikov et al., 2020) is a NAS benchmark for natural language processing, which is size 10 53 .

Original: However, only 14 322 of the architectures were trained on Penn TreeBank (Mikolov et al., 2010), meaning NAS-Bench-NLP is not queryable.
Revised: However, only 14 322 of the architectures were trained on Penn TreeBank (Mikolov et al., 2010), meaning NAS-Bench-NLP is not queryable.

Original: NAS-Bench-ASR (Mehrotra et al., 2021) is a tabular NAS benchmark for automatic speech recognition.
Revised: NAS-Bench-ASR (Mehrotra et al., 2021) is a tabular NAS benchmark for automatic speech recognition.

Original: The search space consists of 8 242 architectures trained on the TIMIT dataset.
Revised: The search space consists of 8 242 architectures trained on the TIMIT dataset.

Original: TransNAS-Bench
Revised: TransNAS-Bench

Original: (Duan et al., 2021) is a tabular NAS benchmark consisting of two separate search spaces (cell-level and macro-level) and seven tasks including pixel-level prediction, regression, and self-supervised tasks.
Revised: (Duan et al., 2021) is a tabular NAS benchmark consisting of two separate search spaces (cell-level and macro-level) and seven tasks including pixel-level prediction, regression, and self-supervised tasks.

Original: The cell and macro search spaces are size 4 096 and 3 256 , respectively.
Revised: The cell and macro search spaces are size 4 096 and 3 256 , respectively.

Original: NAS-Bench-MR (Ding et al., 2021) is a surrogate NAS benchmark which evaluates across four datasets: ImageNet50-1000, Cityscapes,
Revised: NAS-Bench-MR (Ding et al., 2021) is a surrogate NAS benchmark which evaluates across four datasets: ImageNet50-1000, Cityscapes,

Original: KITTI, and HMDB51.
Revised: KITTI, and HMDB51.

Original: NAS-Bench-MR consists of a single search space of size 10 23 .
Revised: NAS-Bench-MR consists of a single search space of size 10 23 .

Original: The DARTS (Liu et al., 2019b) search space with CIFAR-10, consisting of 10 18 architectures, is arguably the most widely-used NAS benchmark.
Revised: The DARTS (Liu et al., 2019b) search space with CIFAR-10, consisting of 10 18 architectures, is arguably the most widely-used NAS benchmark.

Original: Recently, 60 000 of the architectures were trained and used to create NASBench-301 (Siems et al., 2020), the ﬁrst surrogate NAS benchmark.
Revised: Recently, 60 000 of the architectures were trained and used to create NASBench-301 (Siems et al., 2020), the ﬁrst surrogate NAS benchmark.

Original: More recently, NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP

Revised: More recently, NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP

Original: (Yan et al., 2021) were released as surrogate benchmarks that extend NAS-Bench-101, NAS-Bench-301, and NAS-Bench-NLP by predicting the full learning curve information.
Revised: (Yan et al., 2021) were released as surrogate benchmarks that extend NAS-Bench-101, NAS-Bench-301, and NAS-Bench-NLP by predicting the full learning curve information.

Original:  NAS BENCHMARK STATISTICS
Revised:  NAS BENCHMARK STATISTICS

Original: In this section and in Appendix B [Figure] we use NAS-Bench-Suite to compute a set of aggregate statistics across a large set of NAS benchmarks.
Revised: In this section and in Appendix B, we use NAS-Bench-Suite to compute a set of aggregate statistics across a large set of NAS benchmarks.

Original: There is a high variance with respect to the distribution of accuracies and other statistics across benchmarks due to substantial differences in the tasks performed and layout of the search space.
Revised: There is a high variance with respect to the distribution of accuracies and other statistics across benchmarks due to substantial differences in the tasks performed and layout of the search space.

Original: It is essential to keep this in mind to ensure a fair comparison of the performance of NAS algorithms across these benchmarks.
Revised: It is essential to keep this in mind to ensure a fair comparison of the performance of NAS algorithms across these benchmarks.

Original: To the best of our knowledge, this is the ﬁrst large-scale aggregation of statistics computed on NAS benchmarks
Revised: To the best of our knowledge, this is the ﬁrst large-scale aggregation of statistics computed on NAS benchmarks

Original: [Figure]
Revised: [Figure]

Revised: RWA computes the autocorrelation of accuracies of architectures during a random walk, in which each step perturbs one operation or edge.

Original: Figure 2 shows box plots for the validation accuracy distribution for a representative set of the 25 NAS benchmarks.
Revised: Figure 2 shows box plots for the validation accuracy distribution for a representative set of the 25 NAS benchmarks.

Original: We ﬁnd that TransNAS-Bench (Sem. Segment) and DARTS achieve the highest median and maximum accuracies, yet they also have among the smallest variance in validation accuracy across the search space.
Revised: We ﬁnd that TransNAS-Bench (Sem. Segment) and DARTS achieve the highest median and maximum accuracies, yet they also have among the smallest variance in validation accuracy across the search space.

Original: On the other hand, the search space with the highest interquartile range is TransNAS-Bench Jigsaw.
Revised: On the other hand, the search space with the highest interquartile range is TransNAS- Bench Jigsaw.

Original: In Figure 3, we assess the level of locality in each search space, or the similarity of validation accuracy among neighboring architectures (architectures which differ by a single operation or edge) using the random walk autocorrelation (RWA)
Revised: In Figure 3, we assess the level of locality in each search space, or the similarity of validation accuracy among neighboring architectures (architectures which differ by a single operation or edge) using the random walk autocorrelation (RWA)

Original: (Weinberger, 1990;
Revised: (Weinberger, 1990;

Original: Ying et al., 2019;
Revised: Ying et al., 2019;

Original: White et al., 2021b).
Revised: White et al., 2021b).

Original: RWA computes the autocorrelation of accuracies of architectures during a random walk, in which each step perturbs one operation or edge.
Revised: RWA computes the autocorrelation of accuracies of architectures during a random walk, in which each step perturbs one operation or edge.

Original: We see that NAS-Bench-201 ImageNet16-120 has the highest autocorrelation, while NAS-Bench-101 has the lowest.
Revised: We see that NAS-Bench-201 ImageNet16-120 has the highest autocorrelation, while NAS-Bench-101 has the lowest.

Original: In
Revised: In

Original: Appendix B, we also discuss plots describing the average runtime for training architectures and the average neighborhood size, for each NAS benchmark.
Revised: Appendix B, we also discuss plots describing the average runtime for training architectures and the average neighborhood size, for each NAS benchmark.

Original: Overall, we see substantial differences among the search spaces along the various axes that we tested.
Revised: Overall, we see substantial differences among the search spaces along the various axes that we tested.

Original: Overall, we ﬁnd that the diversity is important to keep into context when comparing across many different NAS benchmarks.
Revised: Overall, we ﬁnd that the diversity is important to keep into context when comparing across many different NAS benchmarks.

Original: For example, it is more impressive if a NAS algorithm discovers an architecture within 0 .
Revised: For example, it is more impressive if a NAS algorithm discovers an architecture within 0 .

Original: 1% of the optimal on NAS-Bench-201 ImageNet16-120, compared to DARTS, because the standard deviation of accuracies for DARTS is much lower.
Revised: 1% of the optimal on NAS-Bench-201 ImageNet16-120, compared to DARTS, because the standard deviation of accuracies for DARTS is much lower.

Original: Additional factors, such as locality and neighborhood size, also affect the difﬁculty of NAS benchmarks for some NAS algorithms more than for others; for example, locality has a large effect on the performance of regularized evolution but not for random search.
Revised: Additional factors, such as locality and neighborhood size, also affect the difﬁculty of NAS benchmarks for some NAS algorithms more than for others; for example, locality has a large effect on the performance of regularized evolution but not for random search.

Original:  O N THE G ENERALIZABILITY OF NAS A LGORITHMS
Revised:  O N THE G ENERALIZABILITY OF NAS A LGORITHMS

Original: In this section, we carry out a large-scale empirical study on the generalizability of NAS algorithms across diverse search spaces and tasks, using ﬁve different black-box algorithms, ﬁve different performance predictors, and three one-shot methods across the largest set of NAS benchmarks to date.
Revised: In this section, we carry out a large-scale empirical study on the generalizability of NAS algorithms across diverse search spaces and tasks, using ﬁve different black-box algorithms, ﬁve different performance predictors, and three one-shot methods across the largest set of NAS benchmarks to date.

Original: Throughout, we empirically assess three assumptions we have witnessed in the NAS community about the generalizability of NAS algorithms across diverse search spaces and tasks:
Revised: Throughout, we empirically assess three assumptions we have witnessed in the NAS community about the generalizability of NAS algorithms across diverse search spaces and tasks:

Original: “If a NAS algorithm does well on the popular NAS benchmarks NAS-Bench-101 and all three datasets of NAS-Bench-201, it surely must generalize to other NAS benchmarks.”
Revised: “If a NAS algorithm does well on the popular NAS benchmarks NAS-Bench-101 and all three datasets of NAS-Bench-201, it surely must generalize to other NAS benchmarks.”

Original: “NAS algorithms tend to have robust default hyperparameters and do not require tuning.”
Revised: “NAS algorithms tend to have robust default hyperparameters and do not require tuning.”

Original: “To improve a NAS algorithm on a new benchmark, we can cheaply optimize its hyperparameters on a tabular benchmark and then transfer the optimized hyperparameters.”
Revised: “To improve a NAS algorithm on a new benchmark, we can cheaply optimize its hyperparameters on a tabular benchmark and then transfer the optimized hyperparameters.”

Original: Experimental details.
Revised: Experimental details.

Original: A black-box NAS algorithm is an algorithm which iteratively chooses architectures to train, and then uses the ﬁnal validation accuracies in the next iteration.
Revised: A black-box NAS algorithm is an algorithm which iteratively chooses architectures to train, and then uses the ﬁnal validation accuracies in the next iteration.

Original: We run experiments for ﬁve popular black-box NAS algorithms: random search (RS) (Li & Talwalkar, 2019;
Revised: We run experiments for ﬁve popular black-box NAS algorithms: random search (RS) (Li & Talwalkar, 2019;

Original: Sciuto et al., 2020), regularized evolution (RE) (Real et al., 2019), local search (LS) (White et al., 2021b; Ottelander et al., 2021), BANANAS (White et al., 2021a), and NPENAS (Wei et al., 2020).
Revised: Sciuto et al., 2020), regularized evolution (RE) (Real et al., 2019), local search (LS) (White et al., 2021b; Ottelander et al., 2021), BANANAS (White et al., 2021a), and NPENAS (Wei et al., 2020).

Original: We run each black-box algorithm for 200 iterations.
Revised: We run each black-box algorithm for 200 iterations.

Original: Recently, model-based performance prediction methods have gained popularity as subroutines to speed up NAS algorithms (Ning et al., 2020).
Revised: Recently, model-based performance prediction methods have gained popularity as subroutines to speed up NAS algorithms (Ning et al., 2020).

Original: These methods work by training a model using a set of already evaluated architectures, and then using the model to predict the performance of untrained architectures.
Revised: These methods work by training a model using a set of already evaluated architectures, and then using the model to predict the performance of untrained architectures.

Original: We compare ﬁve popular performance predictors: BOHAMIANN (Springenberg et al., 2016), Gaussian process (GP) (Rasmussen, 2003), random forest (RF) (Liaw et al., 2002), neural architecture optimization (NAO)
Revised: We compare ﬁve popular performance predictors: BOHAMIANN (Springenberg et al., 2016), Gaussian process (GP) (Rasmussen, 2003), random forest (RF) (Breiman, 2001), neural architecture optimization (NAO)

Original: (Luo et al., 2018), and XGBoost (Chen & Guestrin, 2016).
Revised: (Luo et al., 2018), and XGBoost (Chen & Guestrin, 2016).

Original: We evaluate the performance prediction methods by computing the Spearman rank correlation of the predictions versus ground truth validation accuracy on a held-out test set of 200 architectures.
Revised: We evaluate the performance prediction methods by computing the Spearman rank correlation of the predictions versus ground truth validation accuracy on a held-out test set of 200 architectures.

Original: For each black-box method and each performance predictor, we evaluate the default hyperparameter conﬁguration, as well as 300 randomly sampled hyperparameter conﬁgurations, with each reported performance averaged over 10 seeds.
Revised: For each black-box method and each performance predictor, we evaluate the default hyperparameter conﬁguration, as well as 300 randomly sampled hyperparameter conﬁgurations, with each reported performance averaged over 10 seeds.

Original: We give descriptions, implementation details, and hyperparameter details of each method in Appendix B.
Revised: We give descriptions, implementation details, and hyperparameter details of each method in Appendix B.

Original: 4. T HE B EST NAS M ETHODS
Revised: 4. T HE B EST NAS M ETHODS

Original: [Figure]
Revised: [Figure]

Original: In Figure 4, we plot the scaled (relative) performance of all ﬁve black-box algorithms and performance predictors across a representative set of NAS benchmarks (with the full plot in Appendix B).
Revised: In Figure 4, we plot the scaled (relative) performance of all ﬁve black-box algorithms and performance predictors across a representative set of NAS benchmarks (with the full plot in Appendix B).

Original: In Table 2, we give a summary by computing the average rank of each black-box algorithm or performance prediction method across all 25 NAS benchmarks.
Revised: In Table 2, we give a summary by computing the average rank of each black-box algorithm or performance prediction method across all 25 NAS benchmarks.

Original: Across black-box algorithms with their default hyperparameters, we ﬁnd that no algorithm performs well across all search spaces: no algorithm achieves an average rank close to 1 across all search spaces.
Revised: Across black-box algorithms with their default hyperparameters, we ﬁnd that no algorithm performs well across all search spaces: no algorithm achieves an average rank close to 1 across all search spaces.

Original: RE and LS perform the best on average across the search spaces, with average rankings of 2.36 and 2.66, respectively.
Revised: RE and LS perform the best on average across the search spaces, with average rankings of 2.36 and 2.66, respectively.

Original: We also ﬁnd that although RE performed the best on average, it performs worse than random search in three cases.
Revised: We also ﬁnd that although RE performed the best on average, it performs worse than random search in three cases.

Original: Therefore, there is no “best” black-box algorithm.
Revised: Therefore, there is no “best” black-box algorithm.

Original: When comparing black-box algorithms tuned on each individual benchmark, RE achieves a ranking of 1.96, although we note that since black-box algorithms are expensive to evaluate, it is computationally prohibitive to tune for each individual NAS benchmark.
Revised: When comparing black-box algorithms tuned on each individual benchmark, RE achieves a ranking of 1.96, although we note that since black-box algorithms are expensive to evaluate, it is computationally prohibitive to tune for each individual NAS benchmark.

Original: Across performance predictors, we ﬁnd that the best predictor with default parameters is RF, and the best predictor when tuned on each individual benchmark is XGBoost, with average rankings of 1.57 and 1.23, respectively.
Revised: Across performance predictors, we ﬁnd that the best predictor with default parameters is RF, and the best predictor when tuned on each individual benchmark is XGBoost, with average rankings of 1.57 and 1.23, respectively.

Original: Note that since performance prediction subroutines are often not the bottleneck of NAS, it is common to run hyperparameter tuning during the NAS search.
Revised: Note that since performance prediction subroutines are often not the bottleneck of NAS, it is common to run hyperparameter tuning during the NAS search.

Original: Therefore, we conclude that XGBoost (when tuned) does generalize well across all 25 search spaces we tested.
Revised: Therefore, we conclude that XGBoost (when tuned) does generalize well across all 25 search spaces we tested.

Original: Generalizing beyond NAS-Bench-101 and -201.
Revised: Generalizing beyond NAS-Bench-101 and -201.

Original: Now we test how well NAS methods generalize from NAS-Bench-101 and -201 to the rest of the NAS benchmarks.
Revised: Now we test how well NAS methods generalize from NAS-Bench-101 and -201 to the rest of the NAS benchmarks.

Original: In Table 2, for both black-box and predictor methods, we compare the average rank of each method across two different subsets of benchmarks: NAS-Bench-101 and the three different datasets of NAS-Bench-201, versus the rest of the 21 settings excluding NAS-Bench-101 and NAS-Bench-201.
Revised: In Table 2, for both black-box and predictor methods, we compare the average rank of each method across two different subsets of benchmarks: NAS-Bench-101 and the three different datasets of NAS-Bench-201, versus the rest of the 21 settings excluding NAS-Bench-101 and NAS-Bench-201.

Original: We ﬁnd that for both black-box method and performance predictor methods, the best method substantially changes between these two subsets.
Revised: We ﬁnd that for both black-box method and performance predictor methods, the best method substantially changes between these two subsets.

Original: For example, NAO is the top-performing predictor across NAS-Bench-101 and NASBench-201, yet it achieves very poor performance on the rest of the benchmarks.
Revised: For example, NAO is the top-performing predictor across NAS-Bench-101 and NASBench-201, yet it achieves very poor performance on the rest of the benchmarks.

Original: This suggests that the insights derived from empirical results are highly dependent on the benchmarks used, and that in order to make reliable claims, evaluating on more than a few benchmarks is crucial
Revised: This suggests that the insights derived from empirical results are highly dependent on the benchmarks used, and that in order to make reliable claims, evaluating on more than a few benchmarks is crucial.

Original: [Table]

Original: [Table]
Revised: [Table]

Original: 4. G ENERALIZABILITY OF H YPERPARAMETERS
Revised: 4. G ENERALIZABILITY OF H YPERPARAMETERS

Original: While the previous section assessed the generalizability of NAS methods, now we assess the generalizability of the hyperparameters within NAS methods.
Revised: While the previous section assessed the generalizability of NAS methods, now we assess the generalizability of the hyperparameters within NAS methods.

Original: For a given NAS method, we can tune it on
Revised: For a given NAS method, we can tune it on

Original: NAS benchmark A , and then evaluate the performance of the tuned method on NAS benchmark B , compared to the performance of the best hyperparameters from NAS benchmark B .
Revised: NAS benchmark A , and then evaluate the performance of the tuned method on NAS benchmark B , compared to the performance of the best hyperparameters from NAS benchmark B .

Original: In other words, we compute the “regret” of tuning a method on one NAS benchmark and deploying it on another.
Revised: In other words, we compute the “regret” of tuning a method on one NAS benchmark and deploying it on another.

Original: In Figure 5 (left), we run this experiment for all pairs of search spaces, averaged over all performance predictors, to give a general estimate of the regret across all search spaces.
Revised: In Figure 5 (left) [Table] we run this experiment for all pairs of search spaces, averaged over all performance predictors, to give a general estimate of the regret across all search spaces.

Original: Unsurprisingly, hyperparameters transfer well within a given search space (such as within the three datasets in
Revised: Unsurprisingly, hyperparameters transfer well within a given search space (such as within the three datasets in

Original: NAS-Bench-201 or the seven datasets in TransNAS-Bench-Micro).
Revised: NAS-Bench-201 or the seven datasets in TransNAS-Bench-Micro).

Original: However, we ﬁnd that no search space achieves strong regret across most search spaces.
Revised: However, we ﬁnd that no search space achieves strong regret across most search spaces.

Original: NAS-Bench-101, DARTS, and the four benchmarks in NAS-Bench-MR have particularly bad regret compared to the other benchmarks.
Revised: NAS-Bench-101, DARTS, and the four benchmarks in NAS-Bench-MR have particularly bad regret compared to the other benchmarks.

Original: Next, in Figure 5 (right), we run the same experiment for black-box algorithms.
Revised: Next, in Figure 5 (right), we run the same experiment for black-box algorithms.

Original: We ﬁnd that the transferability of hyperparameters across black-box algorithms is even worse than across predictors: the hyperparameters do not always transfer well even within different tasks of a ﬁxed search space.
Revised: We ﬁnd that the transferability of hyperparameters across black-box algorithms is even worse than across predictors: the hyperparameters do not always transfer well even within different tasks of a ﬁxed search space.

Original: We also see that, interestingly, the matrix is less symmetric than for performance predictors.
Revised: We also see that, interestingly, the matrix is less symmetric than for performance predictors.

Original: For example, it is particularly hard for hyperparameters to transfer to NAS-Bench-MR, but easier to transfer from
Revised: For example, it is particularly hard for hyperparameters to transfer to NAS-Bench-MR, but easier to transfer from

Original: NAS-Bench-MR.
Revised: NAS-Bench-MR.

Original: Overall, our experiments show that it is not sufﬁcient to tune hyperparameters on one NAS benchmark and deploy on other benchmarks, as this can often make the performance worse.
Revised: Overall, our experiments show that it is not sufﬁcient to tune hyperparameters on one NAS benchmark and deploy on other benchmarks, as this can often make the performance worse.

Revised: In Appendix C, we give additional experiments and summary statistics on the transferability of hyperparameters across search spaces, as well as a guide to interpreting the experiments.

Revised: We also present additional experiments that combine our algorithm and statistics experiments to give relationships between properties of the search space and the performance of different algorithms.

Original: 4. O NE - SHOT A LGORITHMS
Revised: 4. O NE - SHOT A LGORITHMS

Original: One-shot NAS algorithms, in which a single supernetwork representing the entire search space is trained, are a popular choice for NAS due to their strong performance and fast runtimes.
Revised: One-shot NAS algorithms, in which a single supernetwork representing the entire search space is trained, are a popular choice for NAS due to their strong performance and fast runtimes.

Original: In this section, we compare the performance of three one-shot algorithms: DARTS (Liu et al., 2019b), GDAS (Dong & Yang, 2019), and DrNAS (Chen et al., 2021), across several different NAS benchmarks.
Revised: In this section, we compare the performance of three one-shot algorithms: DARTS (Liu et al., 2019b), GDAS (Dong & Yang, 2019), and DrNAS (Chen et al., 2021), across several different NAS benchmarks.

Original: Note that since one-shot algorithms must be able to represent the entire search space in the form of a supernetwork, the algorithms can effectively only be run on cell-based search spaces with a complete graph (Zela et al., 2020b), precluding the use of all 25 NAS benchmarks as in the previous section.
Revised: Note that since one-shot algorithms must be able to represent the entire search space in the form of a supernetwork, the algorithms can effectively only be run on cell-based search spaces with a complete graph (Zela et al., 2020b), precluding the use of all 25 NAS benchmarks as in the previous section.

Original: In Figure 6, we plot the scaled (relative) performance of the three one-shot methods run for ﬁve seeds each, across nine NAS benchmarks.
Revised: In Figure 6, we plot the scaled (relative) performance of the three one-shot methods run for ﬁve seeds each, across nine NAS benchmarks.

Original: There is no clear best algorithm:
Revised: There is no clear best algorithm:

Original: DrNAS performed best on ﬁve benchmarks, and GDAS performed best on the remaining four.
Revised: DrNAS performed best on ﬁve benchmarks, and GDAS performed best on the remaining four.

Original: DARTS did not perform as well, which is consistent with prior work (Zela et al., 2020a; Dong & Yang, 2020).
Revised: DARTS did not perform as well, which is consistent with prior work (Zela et al., 2020a; Dong & Yang, 2020).

Original: Throughout this section, we showed that many implicit assumptions in the NAS community regarding
Revised: Throughout this section, we showed that many implicit assumptions in the NAS community regarding

Original: NAS algorithm generalizability are incorrect, and that it is important to consider a large set of NAS benchmarks in order to avoid false conclusions.
Revised: NAS algorithm generalizability are incorrect, and that it is important to consider a large set of NAS benchmarks in order to avoid false conclusions.

Original: In the next section, in order to help NAS researchers and practitioners avoid these pitfalls, we describe our new NAS benchmark suite, which is designed with the aim to help the community develop generalizable NAS methods.
Revised: In the next section, in order to help NAS researchers and practitioners avoid these pitfalls, we describe our new NAS benchmark suite, which is designed with the aim to help the community develop generalizable NAS methods.

Original:  NAS-B ENCH -S UITE : O VERVIEW
Revised:  NAS-B ENCH -S UITE : O VERVIEW

Original: In this section, we give an overview of the NAS-Bench-Suite codebase, which allows researchers to easily compare NAS algorithms across many tasks, as shown in Section 4.
Revised: In this section, we give an overview of the NAS-Bench-Suite codebase, which allows researchers to easily compare NAS algorithms across many tasks, as shown in Section 4.

Original: 5. K EY CHALLENGES IN CREATING A FLEXIBLE BENCHMARK SUITE
Revised: 5. K EY CHALLENGES IN CREATING A FLEXIBLE BENCHMARK SUITE

Original: Before the introduction of tabular NAS benchmarks, it was common practice to release new NAS methods along with custom-designed search spaces to go with the methods (Lindauer & Hutter, 2020; Li & Talwalkar, 2019), leading to many early NAS methods being intertwined with and hard to separate from their original search space.
Revised: Before the introduction of tabular NAS benchmarks, it was common practice to release new NAS methods along with custom-designed search spaces to go with the methods (Lindauer & Hutter, 2020; Li & Talwalkar, 2019), leading to many early NAS methods being intertwined with and hard to separate from their original search space.

Original: This was especially true for weight sharing methods, many of which require speciﬁc constraints on the topology of the NAS structure and speciﬁc sets of operations in order to run properly.
Revised: This was especially true for weight sharing methods, many of which require speciﬁc constraints on the topology of the NAS structure and speciﬁc sets of operations in order to run properly.

Original: The result was that many algorithms needed substantial code changes in order to run on other search spaces.
Revised: The result was that many algorithms needed substantial code changes in order to run on other search spaces.

Original: [Figure]
Revised: [Figure]

Original: Snippet 1: A minimal example on how one can run a NAS algorithm in NAS-Bench-Suite .
Revised: Snippet 1: A minimal example on how one can run a NAS algorithm in NAS-Bench-Suite .

Original: Both the search space and the algorithm can be changed in one line of code.
Revised: Both the search space and the algorithm can be changed in one line of code.

Original: Even after the release of several queryable benchmarks, it is still not common practice to run NAS algorithms on more than a few benchmarks due to the nontrivial differences among each benchmark.
Revised: Even after the release of several queryable benchmarks, it is still not common practice to run NAS algorithms on more than a few benchmarks due to the nontrivial differences among each benchmark.

Original: For example, as described in Section 2, operations on the nodes (Ying et al., 2019) versus on the edges (Liu et al., 2019b; Dong & Yang, 2020) added complexity in adapting one-shot optimizers to many search spaces, and for some search spaces one-shot optimizers could only be run on subsets of the full space (Zela et al., 2020b).
Revised: For example, as described in Section 2, operations on the nodes (Ying et al., 2019) versus on the edges (Liu et al., 2019b) added complexity in adapting one-shot optimizers to many search spaces, and for some search spaces one-shot optimizers could only be run on subsets of the full space (Zela et al., 2020b).

Original: Other differences, such as the presence of hidden nodes (Klyuchnikov et al., 2020) or skip connections (Mehrotra et al., 2021; Liu et al., 2019b), cause NAS components to require different implementations.
Revised: Other differences, such as the presence of hidden nodes (Klyuchnikov et al., 2020) or skip connections (Mehrotra et al., 2021), cause NAS components to require different implementations.

Original: Creating a robust NAS benchmark suite is not as simple as “combining the individual codebases”, because such a solution would require re-implementing each new NAS algorithm on several search spaces.
Revised: Creating a robust NAS benchmark suite is not as simple as “combining the individual codebases”, because such a solution would require re-implementing each new NAS algorithm on several search spaces.

Original: In fact, the difﬁculty in creating a benchmark suite is likely a key reason why there are so few papers that evaluate on more than a few benchmarks.
Revised: In fact, the difﬁculty in creating a benchmark suite is likely a key reason why there are so few papers that evaluate on more than a few benchmarks.

Original: 5. T HE NAS-B ENCH -S UITE CODEBASE
Revised: 5. T HE NAS-B ENCH -S UITE CODEBASE

Original: To overcome these difﬁculties, NAS-Bench-Suite enacts two key principles: ﬂexibility in deﬁning the search space and modularity of individual NAS components.
Revised: To overcome these difﬁculties, NAS-Bench-Suite enacts two key principles: ﬂexibility in deﬁning the search space and modularity of individual NAS components.

Original: We ﬁrst describe how we achieve ﬂexible search space deﬁnitions, and then we detail the modular design of NAS-Bench-Suite .
Revised: We ﬁrst describe how we achieve ﬂexible search space deﬁnitions, and then we detail the modular design of NAS-Bench-Suite .

Original: A search space is deﬁned with a graph object using PyTorch and NetworkX (Hagberg et al., 2008), a package that allows for easy-to-use and ﬂexible graph creation and manipulations.
Revised: A search space is deﬁned with a graph object using PyTorch and NetworkX (Hagberg et al., 2008), a package that allows for easy-to-use and ﬂexible graph creation and manipulations.

Original: This functionality allows for a dynamic search space deﬁnition, where candidate operations can be encoded both as part of a node (Ying et al., 2019;
Revised: This functionality allows for a dynamic search space deﬁnition, where candidate operations can be encoded both as part of a node (Ying et al., 2019;

Original: Klyuchnikov et al., 2020) as well as a edge (Dong & Yang, 2020; Liu et al., 2019b).
Revised: Klyuchnikov et al., 2020) as well as a edge (Dong & Yang, 2020; Liu et al., 2019b).

Original: It also allows the representation of multiple layers of graphs on top of the computational graph, allowing the formation of nested graph-structures that can be used to deﬁne hierarchical spaces (Ru et al., 2020; Liu et al., 2019a).
Revised: It also allows the representation of multiple layers of graphs on top of the computational graph, allowing the formation of nested graph-structures that can be used to deﬁne hierarchical spaces (Ru et al., 2020; Liu et al., 2019a).

Original: The NAS-Bench-Suite is modular in the sense that individual NAS components, such as the search space, NAS algorithm, or performance predictors, are disentangled and deﬁned separately from one another.
Revised: The NAS-Bench-Suite is modular in the sense that individual NAS components, such as the search space, NAS algorithm, or performance predictors, are disentangled and deﬁned separately from one another.

Original: In Snippet 1, we showcase a minimal example that runs a black-box NAS algorithm on a tabular benchmark in NAS-Bench-Suite .
Revised: In Snippet 1, we showcase a minimal example that runs a black-box NAS algorithm on a tabular benchmark in NAS-Bench-Suite .

Original: Other optimizers and benchmarks can be imported and run similarly.
Revised: Other optimizers and benchmarks can be imported and run similarly.

Original: Due to these design principles, NAS-Bench-Suite allows researchers to implement their NAS algorithm in isolation, and then evaluate on all the benchmarks integrated in
Revised: Due to these design principles, NAS-Bench-Suite allows researchers to implement their NAS algorithm in isolation, and then evaluate on all the benchmarks integrated in

Original: NAS-Bench-Suite without writing any additional code.
Revised: NAS-Bench-Suite without writing any additional code.

Original: Since a variety of NAS algorithms, search spaces, and performance predictors have already been integrated into the open-source framework, this allows for the user to build on top of predeﬁned NAS components.
Revised: Since a variety of NAS algorithms, search spaces, and performance predictors have already been integrated into the open-source framework, this allows for the user to build on top of predeﬁned NAS components.

Original: With the entire pipeline in place, along with the possibility of quick evaluations across search spaces and tasks, we believe that
Revised: With the entire pipeline in place, along with the possibility of quick evaluations across search spaces and tasks, we believe that

Original: NAS-Bench-Suite will allow researchers to rapidly prototype and fairly evaluate NAS methods.
Revised: NAS-Bench-Suite will allow researchers to rapidly prototype and fairly evaluate NAS methods.

Original: All the scripts to run the evaluations conducted in this paper come together with the library codebase.
Revised: All the scripts to run the evaluations conducted in this paper come together with the library codebase.

Original: For more details on the API, see Appendix C.
Revised: For more details on the API, see Appendix D.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We describe work that provides experimental surveys, benchmark suites, or uniﬁed codebases within
Revised: We describe work that provides experimental surveys, benchmark suites, or uniﬁed codebases within

Original: NAS.
Revised: NAS.

Original: For detailed surveys on NAS, see (Elsken et al., 2019; Xie et al., 2020).
Revised: For detailed surveys on NAS, see (Elsken et al., 2019; Xie et al., 2020).

Original: NAS experimental surveys.
Revised: NAS experimental surveys.

Original: Multiple papers have found that random search is a competitive NAS baseline (Li & Talwalkar, 2019;
Revised: Multiple papers have found that random search is a competitive NAS baseline (Li & Talwalkar, 2019;

Original: Sciuto et al., 2020), including a recent work that benchmarked eight
Revised: Sciuto et al., 2020), including a recent work that benchmarked eight

Original: NAS methods with ﬁve datasets on the DARTS search space (Yang et al., 2020).
Revised: NAS methods with ﬁve datasets on the DARTS search space (Yang et al., 2020).

Original: Other recent works have shown experimental surveys for NAS performance predictors (Ning et al., 2020; White et al., 2021c), and experimental analyses on weight sharing (Yu et al., 2020).
Revised: Other recent works have shown experimental surveys for NAS performance predictors (Ning et al., 2020; White et al., 2021c), and experimental analyses on weight sharing (Yu et al., 2020).

Original: Benchmark suites.
Revised: Benchmark suites.

Original: NAS-Bench-360 (Tu et al., 2021) is a very recent benchmark suite which presents NAS benchmarks for ten diverse datasets on three search spaces.
Revised: NAS-Bench-360 (Tu et al., 2021) is a very recent benchmark suite which presents NAS benchmarks for ten diverse datasets on three search spaces.

Original: However, a drawback is that evaluating NAS algorithms requires 1 to 100+ GPU-hours (Tu et al., 2021).
Revised: However, a drawback is that evaluating NAS algorithms requires 1 to 100+ GPU-hours (Tu et al., 2021).

Original: This is in contrast to the NAS-Bench-Suite , where NAS algorithms take at most 5 minutes on a CPU due to the use of queryable benchmarks.
Revised: This is in contrast to the NAS-Bench-Suite , where NAS algorithms take at most 5 minutes on a CPU due to the use of queryable benchmarks.

Original: Outside of NAS, multiple hyperparameter tuning benchmark suites have been released (Eggensperger et al., 2021; Arango et al., 2021).
Revised: Outside of NAS, multiple hyperparameter tuning benchmark suites have been released (Eggensperger et al., 2021; Arango et al., 2021).

Original: NAS codebases.
Revised: NAS codebases.

Original: The DeepArchitect library (Negrinho & Gordon, 2017) was the ﬁrst to have a modular design for search spaces and NAS algorithms.
Revised: The DeepArchitect library (Negrinho & Gordon, 2017) was the ﬁrst to have a modular design for search spaces and NAS algorithms.

Original: PyGlove (Peng et al., 2021) is a library for
Revised: PyGlove (Peng et al., 2021) is a library for

Original: NAS featuring dynamically adapting components, however, it is not open-sourced.
Revised: NAS featuring dynamically adapting components, however, it is not open-sourced.

Original: Neural Network
Revised: Neural Network

Original: Intelligence (NNI) (Microsoft, 2017) is a platform for AutoML that implements many algorithms as well as NAS-Bench-101 and NAS-Bench-201.
Revised: Intelligence (NNI) (Microsoft, 2017) is a platform for AutoML that implements many algorithms as well as NAS-Bench-101 and NAS-Bench-201.

Original: Other NAS repositories are actively being built, such as archai (Shah & Dey, 2020) and aw_nas (Ning et al., 2020).
Revised: Other NAS repositories are actively being built, such as archai (Shah & Dey, 2020) and aw_nas (Ning et al., 2020).

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In a large-scale study across 25 NAS benchmark settings, 5 blackbox NAS methods, 5 NAS predictors, and 3 one-shot methods, we showed that many implicit assumptions in the NAS community are wrong.
Revised: In a large-scale study across 25 NAS benchmark settings, 5 blackbox NAS methods, 5 NAS predictors, and 3 one-shot methods, we showed that many implicit assumptions in the NAS community are wrong.

Original: Firstly, there is no single best NAS method: which method performs best very much depends on the benchmark.
Revised: Firstly, there is no single best NAS method: which method performs best very much depends on the benchmark.

Original: Along similar lines, if a NAS method performs well on the popular NAS benchmarks
Revised: Along similar lines, if a NAS method performs well on the popular NAS benchmarks

Original: NAS-Bench-101 and all three datasets of NAS-Bench-201, in contrast to what one might have expected, this does not imply that it will also perform well on other NAS benchmarks.
Revised: NAS-Bench-101 and all three datasets of NAS-Bench-201, in contrast to what one might have expected, this does not imply that it will also perform well on other NAS benchmarks.

Original: Finally, tuning a NAS algorithm’s hyperparameters can make it dramatically better, but transferring such hyperparameters across benchmarks often fails.
Revised: Finally, tuning a NAS algorithm’s hyperparameters can make it dramatically better, but transferring such hyperparameters across benchmarks often fails.

Original: This analysis strongly suggests adapting the empirical standard of the ﬁeld, to stop focusing too much on smaller NAS benchmarks like NAS-Benchand NAS-Bench-201, and rather also embrace larger and novel NAS benchmarks for natural language processing (Klyuchnikov et al., 2020), automatic speech recognition (Mehrotra et al., 2021), and pixel-level prediction, regression, and self-supervised tasks (Duan et al., 2021).
Revised: This analysis strongly suggests adapting the empirical standard of the ﬁeld, to stop focusing too much on smaller NAS benchmarks like NAS-Benchand NAS-Bench-201, and rather also embrace larger and novel NAS benchmarks for natural language processing (Klyuchnikov et al., 2020), automatic speech recognition (Mehrotra et al., 2021), and pixel-level prediction, regression, and self-supervised tasks (Duan et al., 2021).

Original: While substantial differences across NAS search spaces has so far made it very hard to use many NAS benchmarks, we showed a way out of this dilemma by introducing an easy-to-use, uniﬁed benchmark suite that we hope will facilitate reproducible, generalizable, and rapid NAS research.
Revised: While substantial differences across NAS search spaces has so far made it very hard to use many NAS benchmarks, we showed a way out of this dilemma by introducing an easy-to-use, uniﬁed benchmark suite that we hope will facilitate reproducible, generalizable, and rapid NAS research.

Original: For future work, NAS-Bench-Suite can beneﬁt from additional options, such as distributed training.
Revised: For future work, NAS-Bench-Suite can beneﬁt from additional options, such as distributed training and even more diverse tasks.

Original: We also believe that despite the varied set of queryable benchmarks currently implemented,

Original: NAS-Bench-Suite can beneﬁt from even more diverse tasks.

Original: For example, implementing the benchmarks from NAS-Bench-360{+NAS-Bench
+} (Tu et al., 2021) would give novel tasks, such asﬂuid dynamics, DNA sequencing, and surface electromyography.
Revised: For example, implementing the benchmarks from {+NAS-Bench
+} (Tu et al., 2021) would give novel tasks, such as DNA sequencing and surface electromyography.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: For our reproducibility statement, we use the “NAS Best Practices Checklist” (Lindauer & Hutter, 2020), which was recently released to improve reproducibility in neural architecture search.
Revised: For our reproducibility statement, we use the “NAS Best Practices Checklist” (Lindauer & Hutter, 2020), which was recently released to improve reproducibility in neural architecture search.

Original: 1. Best Practices for Releasing Code
Revised: 1. Best Practices for Releasing Code

Original: For all experiments you report:
Revised: For all experiments you report:

Original: (a) Did you release code for the training pipeline used to evaluate the ﬁnal architectures?
Revised: (a) Did you release code for the training pipeline used to evaluate the ﬁnal architectures?

Original: [Yes] Since we used NAS benchmarks for all of our experiments, we did not evaluate the architectures ourselves.
Revised: [Yes] Since we used NAS benchmarks for all of our experiments, we did not evaluate the architectures ourselves.

Original: The code for the training pipelines of these benchmarks is already publicly available.
Revised: The code for the training pipelines of these benchmarks is already publicly available.

Original: (b) Did you release code for the search space [Yes] Again, since we used NAS benchmarks, this code is already publicly available.
Revised: (b) Did you release code for the search space [Yes] Again, since we used NAS benchmarks, this code is already publicly available.

Original: (c) Did you release the hyperparameters used for the ﬁnal evaluation pipeline, as well as random seeds?
Revised: (c) Did you release the hyperparameters used for the ﬁnal evaluation pipeline, as well as random seeds?

Original: [Yes] Since we used NAS benchmarks, the ﬁnal training pipeline was ﬁxed.
Revised: [Yes] Since we used NAS benchmarks, the ﬁnal training pipeline was ﬁxed.

Original: We released our code, which includes the seeds used.
Revised: We released our code, which includes the seeds used.

Original: (d) Did you release code for your NAS method?
Revised: (d) Did you release code for your NAS method?

Original: [Yes] All of our code for benchmarking NAS methods is available at https://anonymous.4open.science/r/ NASLib .
Revised: [Yes] All of our code for benchmarking NAS methods is available at https://anonymous.4open.science/r/ NASLib .

Original: (e) Did you release hyperparameters for your NAS method, as well as random seeds?
Revised: (e) Did you release hyperparameters for your NAS method, as well as random seeds?

Original: [Yes]
Revised: [Yes]

Original: The
Revised: The

Original: hyperparameters we used is also available at https://anonymous.4open.
Revised: hyperparameters we used is also available at https://anonymous.4open.

Original: science/r/NASLib , and
Revised: science/r/NASLib , and

Original: we
Revised: we

Original: discuss these choices in Appendix B.
Revised: discuss these choices in Appendix B.

Original: 2. Best practices for comparing NAS methods
Revised: 2. Best practices for comparing NAS methods

Original: (a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code?
Revised: (a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code?

Original: [Yes] We only used NAS benchmarks, which means the training details are ﬁxed.
Revised: [Yes] We only used NAS benchmarks, which means the training details are ﬁxed.

Original: (b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)?
Revised: (b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)?

Original: [Yes] We only used NAS Benchmarks, which keep these details ﬁxed.
Revised: [Yes] We only used NAS Benchmarks, which keep these details ﬁxed.

Original: (c) Did you run ablation studies?
Revised: (c) Did you run ablation studies?

Original: [N/A]
Revised: [N/A]

Original: This is not applicable, because we did not devise a new NAS method.
Revised: This is not applicable, because we did not devise a new NAS method.

Original: (d) Did you use the same evaluation protocol for the methods being compared?
Revised: (d) Did you use the same evaluation protocol for the methods being compared?

Original: [Yes] We only used NAS Benchmarks, which keep this ﬁxed.
Revised: [Yes] We only used NAS Benchmarks, which keep this ﬁxed.

Original: (e) Did you compare performance over time?
Revised: (e) Did you compare performance over time?

Original: [No] As we explained in Section 4, since our focus was on comparing the relative performance across a large number of benchmarks, our plots focus on the performance across benchmarks while keeping runtime ﬁxed for all methods.
Revised: [No] As we explained in Section 4, since our focus was on comparing the relative performance across a large number of benchmarks, our plots focus on the performance across benchmarks while keeping runtime ﬁxed for all methods.

Original: (f) Did you compare to random search?
Revised: (f) Did you compare to random search?

Original: [Yes] We did include random search in our experiments in Section 4.
Revised: [Yes] We did include random search in our experiments in Section 4.

Original: (g)
Revised: (g)

Original: Did you perform multiple runs of your experiments and report seeds?
Revised: Did you perform multiple runs of your experiments and report seeds?

Original: [Yes] We averaged 20 trials and 100 or more hyperparameter conﬁgurations for each experiment, and the choice of seeds (0-19) and hyperparameter conﬁgurations is available at https://anonymous.4open.science/r/NASLib .
Revised: [Yes] We averaged 20 trials and 100 or more hyperparameter conﬁgurations for each experiment, and the choice of seeds (0-19) and hyperparameter conﬁgurations is available at https://anonymous.4open.science/r/NASLib .

Original: (h)
Revised: (h)

Original: Did you use tabular or surrogate benchmarks for in-depth evaluations?
Revised: Did you use tabular or surrogate benchmarks for in-depth evaluations?

Original: [Yes] This is in fact one of the main purposes of our paper (cid:44) .
Revised: [Yes] This is in fact one of the main purposes of our paper (cid:44) .

Original: 3. Best practices for reporting important details
Revised: 3. Best practices for reporting important details

Original: (a) Did you report how you tuned hyperparameters, and what time and resources this required?
Revised: (a) Did you report how you tuned hyperparameters, and what time and resources this required?

Original: [Yes] We reported this information in Section 4.
Revised: [Yes] We reported this information in Section 4.

Original: (b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)?
Revised: (b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)?

Original: [Yes] Our results use the end-to-end time.
Revised: [Yes] Our results use the end-to-end time.

Original: (c) Did you report all the details of your experimental setup?
Revised: (c) Did you report all the details of your experimental setup?

Original: [Yes] We did include all details of the setup in Section 4 and B.
Revised: [Yes] We did include all details of the setup in Section 4 and B.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our work gives a large scale evaluation of generalizability in NAS and then proposes a new benchmark suite for NAS.
Revised: Our work gives a large scale evaluation of generalizability in NAS and then proposes a new benchmark suite for NAS.

Original: The goal of our work is to make it faster and more accessible for researchers to run generalizable and reproducible NAS experiments.
Revised: The goal of our work is to make it faster and more accessible for researchers to run generalizable and reproducible NAS experiments.

Original: Speciﬁcally, the use of tabular and surrogate NAS benchmarks allow researchers to simulate NAS experiments directly on a CPU, rather than using a GPU, reducing the carbon footprint of NAS research (Patterson et al., 2021;
Revised: Speciﬁcally, the use of tabular and surrogate NAS benchmarks allow researchers to simulate NAS experiments directly on a CPU, rather than using a GPU, reducing the carbon footprint of NAS research (Patterson et al., 2021;

Original: Hao, 2019).
Revised: Hao, 2019).

Original: This is especially important since the development stage of NAS research may be extremely computationally intensive without the use of NAS benchmarks (Zoph & Le, 2017; Real et al., 2019).
Revised: This is especially important since the development stage of NAS research may be extremely computationally intensive without the use of NAS benchmarks (Zoph & Le, 2017; Real et al., 2019).

Original: Our work is a tool for the NAS community which facilitates NAS research that may be used for positive impacts on society (for example, algorithms that reduce CO 2 emissions (Rolnick et al., 2019)) or negative impacts on society (for example, models that discriminate or exclude groups of people).
Revised: Our work is a tool for the NAS community which facilitates NAS research that may be used for positive impacts on society (for example, algorithms that reduce CO 2 emissions (Rolnick et al., 2019)) or negative impacts on society (for example, models that discriminate or exclude groups of people).

Original: Due to the uptick in conversations about ethics and societal impacts in the AI community Hecht et al.
Revised: Due to the uptick in conversations about ethics and societal impacts in the AI community Hecht et al.

Original: (2018), we are hopeful that the applications of our work will have a net positive impact on society.
Revised: (2018), we are hopeful that the applications of our work will have a net positive impact on society.

Original: A D ETAILS FROM S ECTION
Revised: A D ETAILS FROM S ECTION

Original: This section is an extension of the “NAS benchmarks” part of Section 2, with additional details for each NAS benchmark, as well as a more comprehensive list of NAS benchmarks.
Revised: This section is an extension of the “NAS benchmarks” part of Section 2, with additional details for each NAS benchmark, as well as a more comprehensive list of NAS benchmarks.

Original: For an extension of
Revised: For an extension of

Original: Table 1, which contains a more comprehensive list, see Table 3.
Revised: Table 1, which contains a more comprehensive list, see Table [Table]

Original: NAS benchmarks.
Revised: NAS benchmarks.

Original: The ﬁrst tabular NAS benchmark to be released was NAS-Bench-101 (Ying et al., 2019).
Revised: The ﬁrst tabular NAS benchmark to be released was NAS-Bench-101 (Ying et al., 2019).

Original: This benchmark consists of a cell-based search space of 423 624 architectures with a ﬁxed macro structure.
Revised: This benchmark consists of a cell-based search space of 423 624 architectures with a ﬁxed macro structure.

Original: NAS-Bench-101 comes with precomputed validation and test accuracies at epochs 4, 12, 36, and 108 from training on CIFAR-10.
Revised: NAS-Bench-101 comes with precomputed validation and test accuracies at epochs 4, 12, [Table] and 108 from training on CIFAR-10.

Original: The cell-based search space of NAS-Bench101 consists of ﬁve nodes which can take on any DAG structure with at most seven edges.
Revised: The cell-based search space of NAS-Bench101 consists of ﬁve nodes which can take on any DAG structure with at most seven edges.

Original: Each node can take on one of three operations.
Revised: Each node can take on one of three operations.

Original: Since training with stochastic gradient descent is random, all architectures were trained three times with different seeds and therefore have three sets of accuracies.
Revised: Since training with stochastic gradient descent is random, all architectures were trained three times with different seeds and therefore have three sets of accuracies.

Original: Since NAS-Bench-101 architectures contain a variable amount of nodes, it is not possible to evaluate one-shot algorithms.
Revised: Since NAS-Bench-101 architectures contain a variable amount of nodes, it is not possible to evaluate one-shot algorithms.

Original: Therefore, NAS-Bench-1Shot1 (Zela et al., 2020b) deﬁnes three subsets of NASBench-101 which allow one-shot algorithms to be run.
Revised: Therefore, NAS-Bench-1Shot1 (Zela et al., 2020b) deﬁnes three subsets of NASBench-101 which allow one-shot algorithms to be run.

Original: The largest subset size in NAS-Bench-1Shot is 363 648 .
Revised: The largest subset size in NAS-Bench-1Shot is 363 648 .

Original: NAS-Bench-201 is the second tabular NAS benchmark.
Revised: NAS-Bench-201 is the second tabular NAS benchmark.

Original: It consists of a cell which is a complete directed acyclic graph over 4 nodes.
Revised: It consists of a cell which is a complete directed acyclic graph over 4 nodes.

Original: Therefore, there are (cid:0) 42 (cid:1) =
Revised: Therefore, there are (cid:0) 42 (cid:1) =

Original: 6 edges.
Revised: 6 edges.

Original: Each edge can take on one of ﬁve operations (note that this is in contrast to NAS-Bench-101, in which the nodes are operations).
Revised: Each edge can take on one of ﬁve operations (note that this is in contrast to NAS-Bench-101, in which the nodes are operations).

Original: The search space consists of 5 6 = 15 , 625 neural architectures, although due to none and identity operations, the number of non-isomorphic architectures is 6 466 .
Revised: The search space consists of 5 6 = 15 , 625 neural architectures, although due to none and identity operations, the number of non-isomorphic architectures is 6 466 .

Original: Each architecture has precomputed train, validation, and test losses and accuracies for 200 epochs on CIFAR-10,
Revised: Each architecture has precomputed train, validation, and test losses and accuracies for 200 epochs on CIFAR-10,

Original: CIFAR-100, and ImageNet-16-120.

Original: As in NASBench-101, on each dataset, each architecture was trained three times using different random seeds.

Original: NATS-Bench
Revised: NATS-Bench

Original: (Dong et al., 2021) is an extension of NAS-Bench-201 which also varies the macro architecture.
Revised: (Dong et al., 2021) is an extension of NAS-Bench-201 which also varies the macro architecture.

Original: Speciﬁcally, a search space of 32 768 architectures with varying size were trained across three datasets for three seeds.
Revised: Speciﬁcally, a search space of 32 768 architectures with varying size were trained across three datasets for three seeds.

Original: The ﬁrst non computer vision NAS benchmark to be released was NAS-Bench-NLP (Klyuchnikov et al., 2020).
Revised: The ﬁrst non computer vision NAS benchmark to be released was NAS-Bench-NLP (Klyuchnikov et al., 2020).

Original: Its search space consists of a DAG of up to 24 nodes, each of which can take on one of seven operations and can have at most three incoming edges.
Revised: Its search space consists of a DAG of up to 24 nodes, each of which can take on one of seven operations and can have at most three incoming edges.

Original: With a size of at least 10 53 ,
Revised: With a size of at least 10 53 ,

Original: NAS-Bench-NLP is currently the largest NAS benchmark.
Revised: NAS-Bench-NLP is currently the largest NAS benchmark.

Original: 14 322 of the architectures were trained on Penn Tree Bank (Mikolov et al., 2010) for 50 epochs.
Revised: 14 322 of the architectures were trained on Penn Tree Bank (Mikolov et al., 2010) for 50 epochs.

Original: Since only a fraction of architectures were trained, NAS-Bench-NLP is not queryable.
Revised: Since only a fraction of architectures were trained, NAS-Bench-NLP is not queryable.

Original: The DARTS (Liu et al., 2019b) search space with CIFAR-10 is arguably the most popular NAS benchmark.
Revised: The DARTS (Liu et al., 2019b) search space with CIFAR-10 is arguably the most popular NAS benchmark.

Original: The search space contains 10 18 architectures, consisting of two cells, each of which has six nodes.
Revised: The search space contains 10 18 architectures, consisting of two cells, each of which has six nodes.

Original: Each node has exactly two incoming edges, and each edge can take one of eight operations.
Revised: Each node has exactly two incoming edges, and each edge can take one of eight operations.

Original: Recently, 60 000 of the architectures were trained for 100 epochs and used to create
Revised: Recently, 60 000 of the architectures were trained for 100 epochs and used to create

Original: NASBench-301 Siems et al.
Revised: NASBench-301 Siems et al.

Original: (2020), the ﬁrst surrogate NAS benchmark.
Revised: (2020), the ﬁrst surrogate NAS benchmark.

Original: The authors released pretrained surrogates created using XGBoost Chen & Guestrin (2016) and graph isomorphism networks (Xu et al., 2019).
Revised: The authors released pretrained surrogates created using XGBoost Chen & Guestrin (2016) and graph isomorphism networks (Xu et al., 2019).

Original: NAS-Bench-ASR (Mehrotra et al., 2021) is a tabular NAS benchmark for automatic speech recognition.
Revised: NAS-Bench-ASR (Mehrotra et al., 2021) is a tabular NAS benchmark for automatic speech recognition.

Original: The search space consists of 8 242 architectures trained on the TIMIT dataset.
Revised: The search space consists of 8 242 architectures trained on the TIMIT dataset.

Original: The search space consists of four nodes, with three main edges that can take on one of six operations, and six skip connection edges, which can be set to on or off. 
Revised: The search space consists of four nodes, with three main edges that can take on one of six operations, and six skip connection edges, which can be set to on or off. NAS

Original: NAS-Bench-111, NAS-Bench-311, and NAS-Bench-NLP11 (Yan et al., 2021) were recently released as surrogate benchmarks that extend NAS-Bench-101, NAS-Bench-301, and NAS-Bench-NLP by predicting the full learning curve information.
Revised: -Bench-111, NAS-Bench-311, and NAS-Bench- NLP11 (Yan et al., 2021) were recently released as surrogate benchmarks that extend NAS-Bench-101, NAS-Bench-301, and NAS-Bench-NLP by predicting the full learning curve information.

Original: In particular, none of NAS-Bench-101, NAS-Bench301, and NAS-Bench-NLP allow the validation accuracies to be queried at arbitrary epochs, which is necessary for multi-ﬁdelity NAS techniques such as learning curve extrapolation (Baker et al., 2018;
Revised: In particular, none of NAS-Bench-101, NAS-Bench301, and NAS-Bench-NLP allow the validation accuracies to be queried at arbitrary epochs, which is necessary for multi-ﬁdelity NAS techniques such as learning curve extrapolation (Baker et al., 2018;

Original: Klein et al., 2017).
Revised: Klein et al., 2017).

Original: The surrogates used to create NAS-Bench-111, NAS-Bench-311, and
Revised: The surrogates used to create NAS-Bench-111, NAS-Bench-311, and

Original: NAS-Bench-NLP11 include singular value decomposition and noise modeling (Yan et al., 2021).
Revised: NAS-Bench-NLP11 include singular value decomposition and noise modeling (Yan et al., 2021).

Original: TransNAS-Bench (Duan et al., 2021) is a tabular NAS benchmark consisting of two separate search spaces (cell-level and macro-level) and seven tasks including pixel-level prediction, regression, and [Table] [Table] HW
Revised: TransNAS-Bench (Duan et al., 2021) is a tabular NAS benchmark consisting of two separate search spaces (cell-level and macro-level) and seven tasks including pixel-level prediction, regression, and self-supervised tasks.

Original: [Table]

Original: NAS-Bench

Original: [Table]

Original: FBNet) [Table] self-supervised tasks. The-cell level search space of TransNAS-Bench is similar to that of NASBench-201, but with 4 choices of operations per edge, hence 4 096 architectures altogether.
Revised: The-cell level search space of TransNAS-Bench is similar to that of NASBench-201, but with 4 choices of operations per edge, hence 4 096 architectures altogether.

Original: The macro-level search space instead has a ﬂexible macro skeleton with variable number of blocks, locations to down-sample feature maps, and locations to raise the channels, leading to a total of 3architectures.
Revised: The macro-level search space instead has a ﬂexible macro skeleton with variable number of blocks, locations to down-sample feature maps, and locations to raise the channels, leading to a total of 3architectures.

Original: NAS-Bench-MR (Ding et al., 2021) is a surrogate NAS benchmark which evaluates nine settings total, across four datasets: ImageNet50-1000, Cityscapes, KITTI, and HMDB51.
Revised: NAS-Bench-MR (Ding et al., 2021) is a surrogate NAS benchmark which evaluates nine settings total, across four datasets: ImageNet50-1000, Cityscapes, KITTI, and HMDB51.

Original: NAS-Bench-MR consists of a single search space of size 10 23 , and for each of the nine settings, 2 500 architectures were trained, to create nine different surrogates for each of the nine settings.
Revised: NAS-Bench-MR consists of a single search space of size 10 23 , and for each of the nine settings, 2 500 architectures were trained, to create nine different surrogates for each of the nine settings.

Original: NAS-Bench-360 (Tu et al., 2021) is a very recent benchmark suite which gives NAS benchmarks for ten different datasets, including tasks that are novel for NAS such as spherical projection, ﬂuid dynamics, DNA sequencing, medical imaging, surface electromyography, and cosmic ray detection.
Revised: NAS-Bench-360 (Tu et al., 2021) is a very recent benchmark suite which gives NAS benchmarks for ten different datasets, including tasks that are novel for NAS such as spherical projection, ﬂuid dynamics, DNA sequencing, medical imaging, surface electromyography, and cosmic ray detection.

Original: The tasks are carried out on three different search spaces based on Wide ResNet (He et al., 2016),
Revised: The tasks are carried out on three different search spaces based on Wide ResNet (He et al., 2016),

Original: DARTS (Liu et al., 2019b), and DenseNAS (Fang et al., 2020).
Revised: DARTS (Liu et al., 2019b), and DenseNAS (Fang et al., 2020).

Original: However, a drawback of NAS-Bench360 is that none of the NAS benchmarks are queryable.
Revised: However, a drawback of NAS-Bench360 is that none of the NAS benchmarks are queryable.

Original: Therefore, evaluating NAS algorithms on these benchmarks requires 1 to 100+ GPU-hours of runtime (Tu et al., 2021).
Revised: Therefore, evaluating NAS algorithms on these benchmarks requires 1 to 100+ GPU-hours of runtime (Tu et al., 2021).

Original: NAS-Bench-Macro (Su et al., 2021) is a NAS benchmark which focuses on the macro search space.
Revised: NAS-Bench-Macro (Su et al., 2021) is a NAS benchmark which focuses on the macro search space.

Original: It consists of 6561 pretrained architectures on CIFAR-10.
Revised: It consists of 6561 pretrained architectures on CIFAR-10.

Original: The search space consists of 8 layers, each with 3 choices of blocks.
Revised: The search space consists of 8 layers, each with 3 choices of blocks.

Original: HW-NAS-Bench is a NAS benchmark focusing on hardware-aware neural architecture search.
Revised: HW-NAS-Bench is a NAS benchmark focusing on hardware-aware neural architecture search.

Original: It gives the measured/estimated hardware-cost for all architectures in NAS-Bench-201 and FBNet (Wu et al., 2019)  hardware devices including edge devices
Revised: It gives the measured/estimated hardware-cost for all architectures in NAS-Bench-201 and FBNet (Wu et al., 2019) on six hardware devices, including commercial edge, FPGA, and ASIC devices. [Table] [Table] HW-NAS

Original: B D ETAILS FROM S ECTIONB. NAS ALGORITHM IMPLEMENTATION DETAILS
Revised: [Table] FBNet) [Table] HW-NAS-Bench can be used alongside NAS-Bench-

Revised: for the

Revised: full information on hardware cost and model accuracy

Revised: for all architectures in NAS-

Revised: Bench

Revised: B D ETAILS FROM S ECTIONB. NAS ALGORITHM IMPLEMENTATION DETAILS

Original: Here, we give implementation details for all algorithms that we compared in Section 4.
Revised: Here, we give implementation details for all algorithms that we compared in Section 4.

Original: We made an effort to keep the implementations as close as possible to the original implementation.
Revised: We made an effort to keep the implementations as close as possible to the original implementation.

Original: We start with the black-box optimizers.
Revised: We start with the black-box optimizers.

Original: For a list of the default hyperparameters and hyperparameter ranges, see https://anonymous.4open.science/r/NAS-Bench-Suite/ naslib/benchmarks/create_configs.py .
Revised: For a list of the default hyperparameters and hyperparameter ranges, see https://anonymous.4open.science/r/NAS-Bench-Suite/ naslib/benchmarks/create_configs.py .

Original: • Random search.
Revised: • Random search.

Original: Random search is the simplest baseline for NAS Li & Talwalkar (2019); Sciuto et al.
Revised: Random search is the simplest baseline for NAS Li & Talwalkar (2019); Sciuto et al.

Original: It draws architectures at random and then returns the best architecture.
Revised: It draws architectures at random and then returns the best architecture.

Original: • Local search.
Revised: • Local search.

Original: Another baseline, local search has been shown to perform well on multiple NAS benchmarks (White et al., 2021b;
Revised: Another baseline, local search has been shown to perform well on multiple NAS benchmarks (White et al., 2021b;

Original: Ottelander et al., 2021;
Revised: Ottelander et al., 2021;

Original: Siems et al., 2020).
Revised: Siems et al., 2020).

Original: It works by evaluating all architectures in the neighborhood of the current best architecture found so far.
Revised: It works by evaluating all architectures in the neighborhood of the current best architecture found so far.

Original: The neighborhood of an architecture is the set of architectures which differ by one operation or edge.
Revised: The neighborhood of an architecture is the set of architectures which differ by one operation or edge.

Original: We used the implementation from White et al.
Revised: We used the implementation from White et al.

Original: (White et al., 2021b).
Revised: (White et al., 2021b).

Original: • Regularized evolution.
Revised: • Regularized evolution.

Original: This algorithm (Real et al., 2019) consists of iteratively mutating the best architectures drawn from a sample of the most recent architectures evaluated.
Revised: This algorithm (Real et al., 2019) consists of iteratively mutating the best architectures drawn from a sample of the most recent architectures evaluated.

Original: A mutation is deﬁned by randomly changing one operation or edge.
Revised: A mutation is deﬁned by randomly changing one operation or edge.

Original: We used the NAS-Bench101 (Ying et al., 2019) implementation.
Revised: We used the NAS-Bench101 (Ying et al., 2019) implementation.

Original: • BANANAS.
Revised: • BANANAS.

Original: This NAS algorithm (White et al., 2021a) uses Bayesian optimization with an ensemble of three predictors as the surrogate.
Revised: This NAS algorithm (White et al., 2021a) uses Bayesian optimization with an ensemble of three predictors as the surrogate.

Original: We use the code from the original repository, but using PyTorch for the MLPs instead of Tensorﬂow.
Revised: We use the code from the original repository, but using PyTorch for the MLPs instead of Tensorﬂow.

Original: We use the adjacency matrix encoding instead of the path encoding, since the path encoding does not scale to large search spaces such as NAS-Bench-NLP.
Revised: We use the adjacency matrix encoding instead of the path encoding, since the path encoding does not scale to large search spaces such as NAS-Bench-NLP.

Original: We use variational sparse GPs (Titsias, 2009) in the ensemble of predictors, since this was shown in prior work to perform well and have low runtime (White et al., 2021c).
Revised: We use variational sparse GPs (Titsias, 2009) in the ensemble of predictors, since this was shown in prior work to perform well and have low runtime (White et al., 2021c).

Original: • NPENAS.
Revised: • NPENAS.

Original: This algorithm (Wei et al., 2020) is based on predictor-guided evolution.
Revised: This algorithm (Wei et al., 2020) is based on predictor-guided evolution.

Original: It iteratively chooses the next architectures by mutating the most recent architectures in a random sample of the population to create a set of candidate architectures, and then using a predictor to pick the architectures with the highest expected accuracy.
Revised: It iteratively chooses the next architectures by mutating the most recent architectures in a random sample of the population to create a set of candidate architectures, and then using a predictor to pick the architectures with the highest expected accuracy.

Original: Again, we use variational sparse GP (Titsias, 2009) as the predictor.
Revised: Again, we use variational sparse GP (Titsias, 2009) as the predictor.

Original: Now we describe the performance predictors.
Revised: Now we describe the performance predictors.

Original: For each method, we used the adjacency one-hot encoding (White et al., 2020).
Revised: For each method, we used the adjacency one-hot encoding (White et al., 2020).

Original: • BOHAMIANN.
Revised: • BOHAMIANN.

Original: BOHAMIANN (Springenberg et al., 2016) is a Bayesian inference prediction method which uses stochastic gradient Hamiltonian Monte Carlo (SGHMC) in order to sample from a Bayesian Neural Network.
Revised: BOHAMIANN (Springenberg et al 2016) is a Bayesian inference prediction method which uses stochastic gradient Hamiltonian Monte Carlo (SGHMC) in order to sample from a Bayesian Neural Network.

Original: We use the original implementation from the pybnn package.
Revised: We use the original implementation from the pybnn package.

Original: • GP.
Revised: • GP.

Original: Gaussian Process (GP) (Rasmussen, 2003) is a popular surrogate often used with Bayesian optimization (Frazier, 2018; Snoek et al., 2012).
Revised: Gaussian Process (GP) (Rasmussen, 2003) is a popular surrogate often used with Bayesian optimization (Frazier, 2018; Snoek et al., 2012).

Original: Every feature has a joint Gaussian distribution.
Revised: Every feature has a joint Gaussian distribution.

Original: We use the Pyro implementation (Bingham et al., 2019).
Revised: We use the Pyro implementation (Bingham et al., 2019).

Original: • NAO.
Revised: • NAO.

Original: Neural Architecture Optimization is a NAS method that uses an encoder-decoder (Luo et al., 2018).
Revised: Neural Architecture Optimization is a NAS method that uses an encoder-decoder (Luo et al., 2018).

Original: The encoder is a feedfoward neural network, and the decoder is an LSTM and attention mechanism.
Revised: The encoder is a feedfoward neural network, and the decoder is an LSTM and attention mechanism.

Original: We used the implementation from SemiNAS (Luo et al., 2020).
Revised: We used the implementation from SemiNAS (Luo et al., 2020).

Original: • Random Forest.
Revised: • Random Forest.

Original: Random forests (Liaw et al., 2002) are ensembles of decision trees.
Revised: Random forests (Breiman, 2001) are ensembles of decision trees.

Original: Random forests have been used as model-based predictors in NAS (Siems et al., 2020; White et al., 2021c).
Revised: Random forests have been used as model-based predictors in NAS (Siems et al., 2020; White et al., 2021c).

Original: We use the Scikit-learn implementation (Pedregosa et al., 2011).
Revised: We use the Scikit-learn implementation (Pedregosa et al., 2011).

Original: • XGBoost.
Revised: • XGBoost.

Original: eXtreme Gradient Boosting (XGBoost) (Chen & Guestrin, 2016) is a popular gradient-boosted decision tree which has been used in NAS (Siems et al., 2020;
Revised: eXtreme Gradient Boosting (XGBoost) (Chen & Guestrin, 2016) is a popular gradient-boosted decision tree which has been used in NAS (Siems et al., 2020;

Original: White et al., 2021c).
Revised: White et al., 2021c).

Original: We used the original code (Chen & Guestrin, 2016).
Revised: We used the original code (Chen & Guestrin, 2016).

Original: For one-shot methods, we mainly focus on the differentiable architecture search approaches, and select popular algorithms like DARTS (Liu et al., 2019b), GDAS (Dong & Yang, 2019) and DrNAS (Chen et al., 2021) in our paper.N BC I F A R N BC I F A
Revised: For one-shot methods, we mainly focus on the differentiable architecture search approaches, and select popular algorithms like DARTS (Liu et al., 2019b), GDAS (Dong & Yang, 2019) and DrNAS (Chen et al., 2021) in our paper.

Original: R

Original: N BC I F A

Original: R

Original: N BI m a g e N e t

Original: D A R

Original: T S

Original: C

Original: I F

Original: A

Original: R

Original: N B - A

Original: S R

Original: T

Original: I M

Original: I

Original: T

Original: N B - N

Original: L P

Original: T r ee

Original: B a n k

Original: T N B - M i c

Original: r o

Original: O b j e c t

Original: T N B - M i c

Original: r o

Original: S c e n e

Original: T N B - M

Original: i c

Original: r

Original: o J

Original: i g s a w

Original: T N B - M i c

Original: r o

Original: R oo m

Original: T N B - M i c

Original: r o

Original: S e m a n t

Original: i c

Original: T N B - M i c

Original: r o

Original: S

Original: u r f a c e

Original: T N B - M i c

Original: r o

Original: A u t o e n c .

Original: T N B - M a c r o

Original: O b j e c t

Original: T N B - M a c r o

Original: S c e n e

Original: T N B - M a c r o

Original: J i g s a w

Original: T N B - M a c r o

Original: R oo m

Original: T N B - M a c r o

Original: S e m a n t

Original: i c

Original: T N B - M a c r o

Original: S

Original: u r f a c e

Original: T N B - M a c r o

Original: A u t o e n c .

Original: [Figure]

Original: • DARTS.
Revised: • DARTS.

Original: DARTS (Liu et al., 2019b) is the ﬁrst work on differentiable architecture search.
Revised: DARTS (Liu et al., 2019b) is the ﬁrst work on differentiable architecture search.

Original: Compared to normal one-shot method that has a one-hot encoding to select one architecture out of different choices, it uses a vector, a.k.a.
Revised: Compared to normal one-shot method that has a one-hot encoding to select one architecture out of different choices, it uses a vector, a.k.a.

Original: architecture parameters, where each element ranges from 0 to 1 as probability.
Revised: architecture parameters, where each element ranges from 0 to 1 as probability.

Original: During training, it sums all branches as a weight summation.
Revised: During training, it sums all branches as a weight summation.

Original: After the search, the ﬁnal architecture is converted by selecting the branch that with highest weight.
Revised: After the search, the ﬁnal architecture is converted by selecting the branch that with highest weight.

Original: • GDAS.
Revised: • GDAS.

Original: Since differentiable architecture search suffers from unstable training compared to traditional one-shot methods, GDAS (Dong & Yang, 2019) bridges the gap.
Revised: Since differentiable architecture search suffers from unstable training compared to traditional one-shot methods, GDAS (Dong & Yang, 2019) bridges the gap.

Original: Instead of having a weighted summation of all paths, it discretizes the paths during training to select the path with highest probability, while the rest of algorithms remains similar as original DARTS.
Revised: Instead of having a weighted summation of all paths, it discretizes the paths during training to select the path with highest probability, while the rest of algorithms remains similar as original DARTS.

Original: • DrNAS.
Revised: • DrNAS.

Original: Dirichlet architecture search (DrNAS) (Chen et al., 2021) is another attempt to solve the instability issue of differentiable architecture search.
Revised: Dirichlet architecture search (DrNAS) (Chen et al., 2021) is another attempt to solve the instability issue of differentiable architecture search.

Original: This work treats the continuously relaxed architecture weights as a random variable, which is modeled by a Dirichlet distribution.
Revised: This work treats the continuously relaxed architecture weights as a random variable, which is modeled by a Dirichlet distribution.

Original: To this end, the Direchlet parameters can be updated by the traditional differentiable architecture search optimizer easily in the end-to-end manner.
Revised: To this end, the Direchlet parameters can be updated by the traditional differentiable architecture search optimizer easily in the end-to-end manner.

Original: B. A DDITIONAL EXPERIMENTS

Original: Next, we give additional experiments on the statistics of NAS benchmarks, to supplement Section 4.

Original: In Figure 7, we plot the average time to train an architecture for each NAS benchmark.

Original: In Figure 8, we plot the average neighborhood size for each NAS benchmark.

Original: Note that for some NAS benchmarks, the neighborhood size is ﬁxed, and for other NAS benchmarks, the neighborhood size varies.

Original: In

Original: Figure 9, we give the full box plots, extending Figure 2.

Original: Finally, in Figure 10, we give the full performance predictor and black-box results, extending Figure 4.

Original: C D ETAILS FROM S ECTION
Revised: C A DDITIONAL E XPERIMENTS

Original: In this section, we give more details on the API for NAS-Bench-Suite .

Original: When designing NAS-Bench-Suite , we strove for both minimalism and generalism.

Original: In order to add a new benchmark in NAS-Bench-Suite , one has to ﬁrst deﬁne the computational graph using the NetworkX API.

Original: This graph encompasses high-level abstractions such as the add_node and add_edges_from methods.

Original: If we are implementing a tabular or surrogate benchmark as the ones used in Section 4, a get_dataset_api function needs to be implemented, which is used as an interface to the original pre-computed benchmark data.

Original: The PyTorch computational graph is generated via the adapt_search_space method of the NAS algorithms or performance [Figure] [Figure] [Figure]
Revised: In this section, we give additional statistics, algorithm, and insight experiments to augment Sections 3 and 4.

Original: [Figure]

Original: [Figure] [Figure] [Figure] ee
Revised: N BC I F A R

Original: [Figure] TT I [Figure] [Figure]
Revised: N BC I F A R

Original: , the neighborhood size varies.

Original: N BC I F A R
Revised: N BC I F A R

Original: N BC I F A R

Original: N BC I F A R

Original: N B201 I m a g e N e t
Revised: N B201 I m a g e N e t

Original: D A R T
Revised: D A R T

Original: S C
Revised: S C

Original: I F A
Revised: I F A

Original: R
Revised: R

Original: N B - A S
Revised: N B - A S

Original: R T
Revised: R T

Original: I M
Revised: I M

Original: I
Revised: I

Original: T
Revised: T

Original: N B - N L
Revised: N B - N L

Original: P T r ee
Revised: P T r ee

Original: B a n k
Revised: B a n k

Original: N B - M
Revised: N B - M

Original: R K I TT
Revised: R K I TT

Original: I
Revised: I

Original: N B - M
Revised: N B - M

Original: R H M
Revised: R H M

Original: D B
Revised: D B

Original: N B - M
Revised: N B - M

Original: R I m a g e N e t
Revised: R I m a g e N e t

Original: N B - M
Revised: N B - M

Original: R C i t y
Revised: R C i t y

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: O b j e c t
Revised: O b j e c t

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: S c e n e
Revised: S c e n e

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: J i g s a w
Revised: J i g s a w

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: R oo m
Revised: R oo m

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: S e m a n t
Revised: S e m a n t

Original: i c
Revised: i c

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: S
Revised: S

Original: u r f a c e
Revised: u r f a c e

Original: T N BM i c
Revised: T N BM i c

Original: r o
Revised: r o

Original: A u t o e n c .
Revised: A u t o e n c .

Original: T N BM a c r o
Revised: T N BM a c r o

Original: O b j e c t
Revised: O b j e c t

Original: T N BM a c r o
Revised: T N BM a c r o

Original: S c e n e
Revised: S c e n e

Original: T N BM a c r o
Revised: T N BM a c r o

Original: J i g s a w
Revised: J i g s a w

Original: T N BM a c r o
Revised: T N BM a c r o

Original: R oo m
Revised: R oo m

Original: T N BM a c r o
Revised: T N BM a c r o

Original: S e m a n t
Revised: S e m a n t

Original: i c
Revised: i c

Original: T N BM a c r o
Revised: T N BM a c r o

Original: S
Revised: S

Original: u r f a c e
Revised: u r f a c e

Original: T N BM a c r o
Revised: T N BM a c r o

Original: A u t o e n c .
Revised: A u t o e n c .

Original: [Figure]
Revised: [Figure]

Original: In the case of extremely large search spaces such as DARTS and NAS-Bench-NLP, the statistics are computed only with respect to the tens-of-thousands of precomputed architectures. predictors.
Revised: used instead of validation accuracy, and three datasets of TransNAS-Bench do not use accuracy: Surface

Original: For instance, it can determine if operation choices in edges should be combined as a mixed operation (as done in DARTS (Liu et al., 2019b)) or if they should be categorical choices from which the NAS algorithms sample. Afterwards, the graph instance is stored as an attribute of the optimizer instance. The Trainer , which runs the optimization loop, interacts only with the optimizer (see line 16 in Snippet 1). [Table]
Revised: Normal uses SSIM, Autoencoding uses SSIM, and Room Layout uses negative loss.

Original: [Figure]
Revised: These are in accordance with the metrics used in the original work.

Original: [Table] [Figure] [Table] [Figure]
Revised: Finally, in the case of extremely large search spaces such as DARTS and NAS-Bench-NLP, the statistics are computed only with respect to the tens-of-thousands of precomputed architectures.

Original: C I F A
Revised: N BC I F A

Original: R
Revised: R

Original: N B201 C I F A
Revised: N BC I F A

Original: R
Revised: R

Original: N B201 C I F A
Revised: N BC I F A

Original: R
Revised: R

Original: N B201 I m a g e N e t
Revised: N BI m a g e N e t

Original: D A R T S
Revised: D A R T S

Original: C
Revised: C

Original: I F
Revised: I F

Original: A R
Revised: A R

Original: N B - AS
Revised: N B - A

Original:  R T
Revised: S R T

Original: I M I T
Revised: I M I T

Original: N B - NL
Revised: N B - N

Original:  PT r ee B a n k N B - M R K I TT
Revised: L P

Original: I N B - M R H M D B
Revised: T r ee

Original: N B- M R I m a g e N e t N B - M R C i t y
Revised: B a n k

Original: T N B - M i c
Revised: T N B - M i c

Original: r o
Revised: r o

Original: O b j e c t
Revised: O b j e c t

Original: T N B - M i c
Revised: T N B - M i c

Original: r o
Revised: r o

Original: S c e n e
Revised: S c e n e

Original: T N B - M
Revised: T N B - M

Original: i c
Revised: i c

Original: r
Revised: r

Original: o J
Revised: o J

Original: i g s a w
Revised: i g s a w

Original: T N B - M i c
Revised: T N B - M i c

Original: r o
Revised: r o

Original: R oo m
Revised: R oo m

Original: T N B - M i c
Revised: T N B - M i c

Original: r o
Revised: r o

Original: S e m a n t
Revised: S e m a n t

Original: i c
Revised: i c

Original: T N B - M i c
Revised: T N B - M i c

Original: r o
Revised: r o

Original: S
Revised: S

Original: u r f a c e
Revised: u r f a c e

Original: T N B - M i c
Revised: T N B - M i c

Original: r o
Revised: r o

Original: A u t o e n c .
Revised: A u t o e n c .

Original: T N B - M a c r o
Revised: T N B - M a c r o

Original: O b j e c t
Revised: O b j e c t

Original: T N B - M a c r o
Revised: T N B - M a c r o

Original: S c e n e
Revised: S c e n e

Original: T N B - M a c r 
Revised: T N B - M a c r o

Original: o J i g s a w
Revised: J i g s a w

Original: T N B - M a c r o
Revised: T N B - M a c r o

Original: R oo m
Revised: R oo m

Original: T N B - M a c r o
Revised: T N B - M a c r o

Original: S e m a n t
Revised: S e m a n t

Original: i c
Revised: i c

Original: T N B - M a c r o
Revised: T N B - M a c r o

Original: S
Revised: S

Original: u r f a c e
Revised: u r f a c e

Revised: T N B - M a c r o

Revised: A u t o e n c .

Revised: [Figure]

Revised: A v g .

Revised: t r a i n i n g t

Revised: i m e ( s e c )

Revised: C. A DDITIONAL S TATISTICS E XPERIMENTS

Revised: We start by giving additional experiments on the statistics of NAS benchmarks, to supplement Section 3.

Revised: In Figure 7, we give the full box plots, extending Figure 2.

Revised: We also add new statistics: in Figure 8, we plot the average time to train an architecture for each NAS benchmark.

Revised: In Figure 9, we plot the average neighborhood size for each NAS benchmark.

Revised: Note that for some NAS benchmarks, the neighborhood size is ﬁxed, and for other NAS benchmarks, the neighborhood size varies.

Revised: C. A DDITIONAL A LGORITHM E XPERIMENTS

Revised: Next, we give additional experiments from Section 4.

Revised: In Figure 10, we give the full performance predictor and black-box results, extending Figure 4.

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: ee

Revised: [Figure]

Revised: TT I

Revised: [Figure]

Revised: [Figure]

Revised: , the neighborhood size varies.

Revised: Recall that in Section 4, we assessed the transferability of hyperparameters by tuning algorithms on NAS benchmark A , and evaluating the performance of the tuned method on NAS benchmark B , compared to the performance of the best hyperparameters from NAS benchmark B .

Revised: The results were plotted in Figure 5.

Revised: In Tables 5 and 6, to present the results in another format, we give the raw values from these experiments.

Revised: All values are averaged over each search space.

Revised: For example, the three rows for NAS-Bench-201 were averaged into one row, and similarly for the columns.

Revised: Furthermore, in Tables 7 and 8, we give summary statistics from this experiment.

Revised: For each search space, we compute the transferability of hyperparameters on average to or from all other search spaces.

Revised: For performance predictors, we ﬁnd that hyperparameters from NAS-Bench-MR transfer the least well to or from other search spaces.

Revised: NAS-Bench-NLP transfers the best.

Revised: For black-box algorithms, NAS-Bench-MR transfers the worst, and DARTS transfers the best.

Revised: Therefore, it is safest for practitioners to tune their techniques on NAS-Bench-NLP and DARTS before deploying them in a new setting.

Revised: It is not safe to tune techniques on other benchmarks such as NAS-Bench-MR.

Revised: While all of the hyperparameter transfer experiments to this point have focused on the optimal hyperparameters, we now present two more experiments that focus on the hyperparameters on average.

Revised: In Figure 11, for search spaces A and B, we compute the Kendall Tau rank correlation of the ranking lists for all hyperparameters on search spaces A and B.

Revised: Finally, in Table 9, we present “leave one out” experiments for all search spaces.

Revised: For a search space

Revised: A, the best hyperparameter setting on average over all search spaces except A is computed, and compared to the performance of the best hyperparameters for A.

Revised: This is similar to Table 7 in that it provides a summary of the average transferability of the hyperparameters for each search space.

Revised: However, the leave one out experiments are focused more on the performance of hyperparameters on average, while Table 7 is focused on transferability of the optimal hyperparameters.

Revised: C. A GUIDE TO INTERPRETING HYPERPARAMETER TRANSFER EXPERIMENTS

Revised: Throughout Sections 4 and C.2, we presented several different analyses and summaries on the extent to which hyperparameters transfer from one search space to another.

Revised: In this section, we give a guide for interpreting the results.

Revised: First, practitioners interested in the transfer of hyperparameters which are optimally trained on one search space, should focus on Figure 5 and Tables 7 and 8, because these ﬁgures and tables express the regret of hyperparameters tuned on one search space and evaluated on others.

Revised: On the other hand, practitioners interested on how hyperparameters overall transfer from one search space to others (not just optimal), should focus on Figure 11 and Table 9, because these represent the average transferability of all sets of hyperparameters that we tried.

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: I

Revised: [Figure]

Revised: A

Revised: [Figure]

Revised: N

Revised: [Figure]

Revised: - 201 C

Revised: I F A

Revised: R

Revised: N

Revised: [Figure]

Revised: - 201 C

Revised: I F A

Revised: R

Revised: N B201 I m a g e N e t

Revised: D A R T S

Revised: C

Revised: I F

Revised: A R

Revised: N B - A S

Revised: R T

Revised: I M I T

Revised: N B - N L

Revised: P T r ee

Revised: B a n k

Revised: N B - M

Revised: R K I TT

Revised: I

Original: T N B - M a c r o
Revised: N B - M R H M D B N

Revised: [Figure]

Revised: - M

Revised: R I m a g e N e t

Revised: N

Revised: [Figure]

Revised: M

Original: A u t o e n c .
Revised: R C i [Figure] y

Original: [Figure]
Revised: [Figure]

Revised: N

Revised: [Figure]

Revised: M i

Revised: [Figure]

Revised: t

Revised: [Figure]

Revised: i

Revised: [Figure]

Revised: i

Revised: [Figure]

Revised: e

Revised: [Figure]

Revised: [Table]

Revised: Practitioners interested in the speciﬁc transfer from one search space (or setting) to another, should focus on our matrix results, Figures 5 and 11.

Revised: Practitioners interested in the general transferability on average to or from one search space to others, should focus on the summary tables, Tables 7,

Revised: [Table]

Revised: and

Revised: [Table]

Revised: C. I NSIGHT E XPERIMENTS

Revised: Now we present experiments that give new insights into NAS algorithms and search spaces.

Revised: First, we run experiments to test the following hypothesis: larger search spaces have smaller interquartile ranges (IQR), because a larger cell size gives most architectures a chance to have performant operations.

Revised: For example, for a search space of size three, some architectures will have two or three

Revised: [Table]

Revised: [Table]

Revised: convolution operations, and some architectures will have two or three pooling operations, creating a large IQR.

Revised: But for a search space of size 10, the vast majority of architectures will have a good mix of convolution and pooling operations, creating a comparatively lower IQR.

Revised: We run this experiment on NAS-Bench-101 and NAS-Bench-201.

Revised: See Figure 12.

Revised: We ﬁnd that in all benchmarks, there is a strict negative correlation between number of operations and IQR.

Revised: Finally, we compute correlations between the relative ranking of each NAS technique and properties of the search spaces, such as total size and neighborhood size.

Revised: See Figure 10.

Revised: The largest correlations we ﬁnd are as follows:

Revised: • GP performs comparatively much better on search spaces with small neighborhood sizes.

Revised: • When tuned, RF and XGBoost perform comparatively much better on large search spaces and also search spaces with large neighborhood sizes.

Revised: • Surprisingly, BOHAMIANN performs comparatively better for large neighborhood sizes when not tuned, and comparatively better for small neighborhood sizes when tuned.

Revised: • Default regularized evolution performs comparatively much better on search spaces with small neighborhood sizes.

Revised: • Local search performs comparatively better on small search spaces and also search spaces with small neighborhood sizes.

Revised: D D ETAILS FROM S ECTION

Revised: In this section, we give more details on the API for NAS-Bench-Suite .

Revised: When designing NAS-Bench-Suite , we strove for both minimalism and generalism.

Revised: In order to add a new benchmark in NAS-Bench-Suite , one has to ﬁrst deﬁne the computational graph using the NetworkX API.

Revised: This graph encompasses high-level abstractions such as the add_node and add_edges_from methods.

Revised: If we are implementing a tabular or surrogate benchmark as the ones used in Section 4, a get_dataset_api function needs to be implemented, which is

Revised: [Table]

Revised: used as an interface to the original pre-computed benchmark data.

Revised: The PyTorch computational graph is generated via the adapt_search_space method of the NAS algorithms or performance predictors.

Revised: For instance, it can determine if operation choices in edges should be combined as a mixed operation (as done in DARTS (Liu et al., 2019b)) or if they should be categorical choices from which the NAS algorithms sample.

Revised: Afterwards, the graph instance is stored as an attribute of the optimizer instance.

Revised: The Trainer , which runs the optimization loop, interacts only with the optimizer (see line 16 in Snippet 1).

Revised: [Table]

Revised: [Figure]

Revised: Kendall Tau rank correlations are computed between properties of the search spaces (search space size or neighborhood size) and the relative ranking list for predictors or NAS algorithms, with or without HPO.

Revised: Since a ranking list is used, a negative correlation means a positive correlation between the search space property and algorithmic performance.

Revised: [Table]

Revised: Default vs. SS size -

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: HPO vs. SS size .

Revised: [Table]