Original: NAS-B ENCH -360: B ENCHMARKING D IVERSE T ASKS FOR N EURAL A RCHITECTURE S EARCH
Revised: NAS-P ERF -360: B ENCHMARKING D IVERSE T ASKS FOR N EURAL A RCHITECTURE S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Most existing neural architecture search (NAS) benchmarks and algorithms prioritize performance on well-studied tasks, e.g., image classiﬁcation on CIFAR and ImageNet.
Revised: Most existing neural architecture search (NAS) benchmarks and algorithms prioritize performance on well-studied tasks, e.g., image classiﬁcation on CIFAR and ImageNet.

Original: This makes the applicability of NAS approaches in more diverse areas inadequately understood.
Revised: This makes the applicability of NAS approaches in more diverse areas inadequately understood.

Original: In this paper, we present NAS-Bench-360 , a  benchmark suite forevaluating state-of-the-art NAS methods for convolutional neural networks (CNNs).
Revised: In this paper, we present NAS-Perf-360 , a performance benchmark suite for state-of-the-art NAS methods for convolutional neural networks (CNNs).

Original: 1 To construct it, we curate a collection of ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives.
Revised: 1 To construct it, we curate a collection of ten tasks spanning a diverse array of application domains, dataset sizes, problem dimensionalities, and learning objectives.

Original: By carefully selecting tasks that can both interoperate with modern CNN-based search methods but that are also far-aﬁeld from their original development domain, we can use NAS-Bench-360 to investigate the following central question: do existing state-of-the-art NAS methods perform well on diverse tasks?
Revised: By carefully selecting tasks that can both interoperate with modern CNN-based search methods but that are also far-aﬁeld from their original development domain, we can use NAS-Perf-360 to investigate the following central question: do existing state-of-the-art NAS methods perform well on diverse tasks?

Original: Our experiments show that a modern NAS procedure designed for image classiﬁcation can indeed ﬁnd good architectures for tasks with other dimensionalities and learning objectives; however, the same method struggles against more task-speciﬁc methods and performs catastrophically poorly on classiﬁcation in non-vision domains.
Revised: Our experiments show that a modern NAS procedure designed for image classiﬁcation can indeed ﬁnd good architectures for tasks with other dimensionalities and learning objectives; however, the same method struggles against more task-speciﬁc methods and performs catastrophically poorly on classiﬁcation in non-vision domains.

Original: The case for NAS robustness becomes even more dire in a resource-constrained setting, where a recent NAS method provides little-to-no beneﬁt over much simpler baselines.
Revised: The case for NAS robustness becomes even more dire in a resource-constrained setting, where a recent NAS method provides little-to-no beneﬁt over much simpler baselines.

Original: These results demonstrate the need for a  benchmark such as NAS-Bench-360 to help develop NAS approaches that work well on a variety of tasks, a crucial component of a truly robust and automated pipeline.
Revised: These results demonstrate the need for a performance benchmark such as NAS-Perf-360 to help develop NAS approaches that work well on a variety of tasks, a crucial component of a truly robust and automated pipeline.

Original: We conclude with a demonstration of the kind of future research our suite of tasks will enable.
Revised: We conclude with a demonstration of the kind of future research our suite of tasks will enable.

Original: All data and code is made publicly available.
Revised: All data and code is made publicly available.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Neural architecture search (NAS) aims to automate the design of deep neural networks, ensuring performance on par with hand-crafted architectures while reducing human labor devoted to tedious architecture tuning (Elsken et al., 2019).
Revised: Neural architecture search (NAS) aims to automate the design of deep neural networks, ensuring performance on par with hand-crafted architectures while reducing human labor devoted to tedious architecture tuning (Elsken et al., 2019).

Original: With the growing number of application areas of ML, and thus of use-cases for automating it, NAS has experienced an intense amount of study, with signiﬁcant progress in search space design (Zoph et al., 2018; Liu et al., 2019b; Cai et al., 2019), search efﬁciency (Pham et al., 2018), and search algorithms (Xu et al., 2020;
Revised: With the growing number of application areas of ML, and thus of use-cases for automating it, NAS has experienced an intense amount of study, with signiﬁcant progress in search space design (Zoph et al., 2018; Liu et al., 2019b; Cai et al., 2019), search efﬁciency (Pham et al., 2018), and search algorithms (Xu et al., 2020;

Original: Li et al., 2021a; White et al., 2021).
Revised: Li et al., 2021a; White et al., 2021).

Original: While the use of NAS techniques may be especially impactful in under-explored or under-resourced domains where less expert help is available, the ﬁeld has largely been dominated by methods designed for and evaluated on benchmarks in computer vision (Liu et al., 2019b;
Revised: While the use of NAS techniques may be especially impactful in under-explored or under-resourced domains where less expert help is available, the ﬁeld has largely been dominated by methods designed for and evaluated on benchmarks in computer vision (Liu et al., 2019b;

Original: Ying et al., 2019; Dong & Yang, 2020).
Revised: Ying et al., 2019; Dong & Yang, 2020).

Original: There have been a few recent efforts to diversify these benchmarks to settings such as vision-based transfer learning (Duan et al., 2021) and speech and language processing
Revised: There have been a few recent efforts to diversify these benchmarks to settings such as vision-based transfer learning (Duan et al., 2021) and speech and language processing

Original: Mehrotra et al.
Revised: Mehrotra et al.

Original: ; Klyuchnikov et al.
Revised: ; Klyuchnikov et al.

Original: (2020); however, evaluating NAS methods on such wellstudied tasks using traditional CNN search spaces does not give a good indication of their utility on more far-aﬁeld applications, which have often necessitated the design of custom neural operations (Cohen et al., 2018;
Revised: (2020); however, evaluating NAS methods on such wellstudied tasks using traditional CNN search spaces does not give a good indication of their utility on more far-aﬁeld applications, which have often necessitated the design of custom neural operations (Cohen et al., 2018;

Original: Li et al., 2021b).
Revised: Li et al., 2021b).

Original: We make progress towards studying NAS on more diverse tasks by introducing a suite of benchmark datasets drawn from various data domains that we collectively call NAS-Bench-360 .
Revised: We make progress towards studying NAS on more diverse tasks by introducing a suite of benchmark datasets drawn from various data domains that we collectively call NAS-Perf-360 .

Original: This benchmark consists of an organized setup of ten suitable datasets that (a) can be evaluated in a uniﬁed way using existing NAS approaches and (b) represent diverse application domains, dataset sizes, problem dimensionalities, and learning objectives.
Revised: This benchmark consists of an organized setup of ten suitable datasets that (a) can be evaluated in a uniﬁed way using existing NAS approaches and (b) represent diverse application domains, dataset sizes, problem dimensionalities, and learning objectives.

Original: We also include standard image classiﬁcation evaluations as a baseline point of comparison, as many new methods continue to be designed for such tasks.
Revised: We also include standard image classiﬁcation evaluations as a baseline point of comparison, as many new methods continue to be designed for such tasks.

Original: Following our construction of this suite of tasks, we demonstrate both the usefulness of and need for
Revised: Following our construction of this suite of tasks, we demonstrate both the usefulness of and need for

Original: NAS-Bench-360 by using it to investigate whether modern NAS is useful to practitioners faced with diverse tasks, i.e., whether its success in computer vision is indicative of strong performance on the much broader set of problems to which NAS can conceivably be applied.
Revised: NAS-Perf-360 by using it to investigate whether modern NAS is useful to practitioners faced with diverse tasks, i.e., whether its success in computer vision is indicative of strong performance on the much broader set of problems to which NAS can conceivably be applied.

Original: To address this question, we start with the fact that a common ﬁrst approach when applying deep learning to a new domain is to try an off-the-shelf CNN; in our case, this will be the Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016).
Revised: To address this question, we start with the fact that a common ﬁrst approach when applying deep learning to a new domain is to try an off-the-shelf CNN; in our case, this will be the Wide ResNet (WRN) (Zagoruyko & Komodakis, 2016).

Original: We then consider the scenario of two practitioners: one with only the resources to train one
Revised: We then consider the scenario of two practitioners: one with only the resources to train one

Original: WRN using the default settings and another that has enough to tune WRN using an off-the-shelf hyperparameter optimizer (Li et al., 2018).
Revised: WRN using the default settings and another that has enough to tune WRN using an off-the-shelf hyperparameter optimizer (Li et al., 2018).

Original: Both are faced with a decision: should they use these ﬁxed-architecture baselines or try out the best NAS has to offer?
Revised: Both are faced with a decision should they use these ﬁxed-architecture baselines or try out the best NAS has to offer?

Original: Overall, our empirical investigation suggests the following:
Revised: Overall, our empirical investigation suggests the following:

Original: The less-constrained practitioner might usually do better using NAS—20% relative improvement over WRN on the median task—but risks catastrophic results on speciﬁc non-vision applications.
Revised: The less-constrained practitioner might usually do better using NAS—20% relative improvement over WRN on the median task—but risks catastrophic results on speciﬁc non-vision applications.

Original: The robustness of NAS in the constrained case may be worse: the practitioner is likely better-off simply using the simple off-the-shelf WRN, as its median rank across NAS-Bench-360’s ten tasks is the same as that of our candidate NAS method.
Revised: The robustness of NAS in the constrained case may be worse: the practitioner is likely better-off simply using the simple off-the-shelf WRN, as its median rank across NAS-Perf-360’s ten tasks is the same as that of our candidate NAS method.

Original: These results are obtained via experiments using two well-studied modern search spaces: the cellbased DARTS space (Liu et al., 2019b) and the efﬁciency-focused DenseNAS space (Fang et al., 2020).
Revised: These results are obtained via experiments using two well-studied modern search spaces: the cellbased DARTS space (Liu et al., 2019b) and the efﬁciency-focused DenseNAS space (Fang et al., 2020).

Original: Each space is paired with a search method known to ﬁnd well-performing architectures on
Revised: Each space is paired with a search method known to ﬁnd well-performing architectures on

Original: ImageNet, speciﬁcally the state-of-the-art GAEA PC-DARTS (Li et al., 2021a) for the former and the original weight-sharing algorithm used by DenseNAS for the latter.
Revised: ImageNet, speciﬁcally the state-of-the-art GAEA PC-DARTS (Li et al., 2021a) for the former and the original weight-sharing algorithm used by DenseNAS for the latter.

Original: Note that our assessment includes a more holistic comparison using performance proﬁles (c.f. Figure 1) to reinforce these ranking-based comparisons, which are useful but can miss a lack of robustness or exaggerate minor differences between methods.
Revised: Note that our assessment includes a more holistic comparison using performance proﬁles (c.f. Figure 1) to reinforce these ranking-based comparisons, which are useful but can miss a lack of robustness or exaggerate minor differences between methods.

Original: The initial experimental results enabled by NAS-Bench-360 suggest that the robustness of modern search methods to diverse tasks beyond image classiﬁcation is mixed at best.
Revised: The initial experimental results enabled by NAS-Perf-360 suggest that the robustness of modern search methods to diverse tasks beyond image classiﬁcation is mixed at best.

Original: At the same time, our set of tasks can serve as a crucial tool for investigating and rectifying this issue, and it is thus important for moving towards a truly automated pipeline containing NAS.
Revised: At the same time, our set of tasks can serve as a crucial tool for investigating and rectifying this issue, and it is thus important for moving towards a truly automated pipeline containing NAS.

Original: In particular, NAS-Bench-360 will facilitate such progress via a diverse array of tasks for validating NAS methods that are not only challenging, real-life problem settings but also computationally accessible for academic researchers with limited budgets.
Revised: In particular, NAS-Perf-360 will facilitate such progress via a diverse array of tasks for validating NAS methods that are not only challenging, real-life problem settings but also computationally accessible for academic researchers with limited budgets.

Original: We demonstrate this potential via further studies on the comparative importance of search spaces v. search algorithms and the usefulness of more-customized approaches, speciﬁcally by studying a random search (RS) baseline over the DenseNAS space as well as two domain-speciﬁc methods: Auto-DeepLab (Auto-DL)(Liu et al., 2019a) for dense prediction and AMBER (Zhang et al., 2021b) for prediction from 1D data.
Revised: We demonstrate this potential via further studies on the comparative importance of search spaces v. search algorithms and the usefulness of more-customized approaches, speciﬁcally by studying a random search (RS) baseline over the DenseNAS space as well as two domain-speciﬁc methods: Auto-DeepLab (Auto-DL)(Liu et al., 2019a) for dense prediction and AMBER (Zhang et al., 2021b) for prediction from 1D data.

Original: Among other insights, these experiments provide evidence that a more robust NAS may require better search spaces with a wider variety of operations.
Revised: Among other insights, these experiments provide evidence that a more robust NAS may require better search spaces with a wider variety of operations.

Original: The associated datasets and experiment code will remain open-source and accessible at a temporarily anonymized repository https://anonymous.4open.science/r/
Revised: The associated datasets and experiment code will remain open-source and accessible at a temporarily anonymized repository https://anonymous.4open.science/r/

Original: NAS-Bench-360-26D1 .
Revised: NAS-Bench-360-26D1 .

Original: Reproducibility of all experiments is assured from open-sourcing all relevant code for the end-to-end procedure, with Docker containers and random seeds provided.
Revised: Reproducibility of all experiments is assured from open-sourcing all relevant code for the end-to-end procedure, with Docker containers and random seeds provided.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Benchmarks have been critical to the development of NAS in recent years.
Revised: Benchmarks have been critical to the development of NAS in recent years.

Original: This includes standard evaluation datasets and protocols, of which the most popular are the CIFAR-10 and ImageNet routines used by DARTS (Liu et al., 2019b).
Revised: This includes standard evaluation datasets and protocols, of which the most popular are the CIFAR-10 and ImageNet routines used by DARTS (Liu et al., 2019b).

Original: Another important type of benchmark has been tabular benchmarks such as NAS-Bench-101 (Ying et al., 2019), NAS-Bench-201 (Dong & Yang, 2020), and NAS-Bench-1Shot1 (Zela et al., 2020); these benchmarks exhaustively evaluate all architectures in their search spaces, which is made computationally feasible by deﬁning simple searched cells.
Revised: Another important type of benchmark has been tabular benchmarks such as NAS-Bench-101 (Ying et al., 2019), NAS-Bench-201 (Dong & Yang, 2020), and NAS-Bench-1Shot1 (Zela et al., 2020); these benchmarks exhaustively evaluate all architectures in their search spaces, which is made computationally feasible by deﬁning simple searched cells.

Original: Consequently, they are less expressive than the DARTS cell (Liu et al., 2019b), often regarded as the most powerful search space in the cell-based regime.
Revised: Consequently, they are less expressive than the DARTS cell (Liu et al., 2019b), often regarded as the most powerful search space in the cell-based regime.

Original: Notably, our benchmark is not a tabular benchmark, i.e., we do not evaluate every architecture from a ﬁxed search space; instead, the focus is on the organization of a suite of tasks for assessing both NAS algorithms and search spaces, which would necessarily be restricted by ﬁxing a search space for a tabular benchmark.
Revised: Notably, our benchmark is not a tabular benchmark, i.e., we do not evaluate every architecture from a ﬁxed search space; instead, the focus is on the organization of a suite of tasks for assessing both NAS algorithms and search spaces, which would necessarily be restricted by ﬁxing a search space for a tabular benchmark.

Original: Pre-computing on an expansive search space such as DARTS, with 10 18 possible architectures, is computationally intractable.
Revised: Pre-computing on an expansive search space such as DARTS, with 10 18 possible architectures, is computationally intractable.

Original: Architectures found on lesser search spaces are most likely suboptimal: the vanilla WRN outperforms all networks in the NAS-Bench-201 search space on CIFAR-100.
Revised: Architectures found on lesser search spaces are most likely suboptimal: the vanilla WRN outperforms all networks in the NAS-Bench-201 search space on CIFAR-100.

Original: While NAS methods and benchmarks have generally been focused on computer vision, recent work such as AutoML-Zero (Real et al., 2020) and XD-operations (Roberts et al., 2021) has started moving towards a more generically applicable set of tools for AutoML.
Revised: While NAS methods and benchmarks have generally been focused on computer vision, recent work such as AutoML-Zero (Real et al., 2020) and XD-operations (Roberts et al., 2021) has started moving towards a more generically applicable set of tools for AutoML.

Original: However, even more recent benchmarks that do go beyond the most popular vision datasets have continued to focus on well-studied tasks, including vision-based transfer learning (Duan et al., 2021),
Revised: However, even more recent benchmarks that do go beyond the most popular vision datasets have continued to focus on well-studied tasks, including vision-based transfer learning (Duan et al., 2021),

Original: speech recognition
Revised: speech recognition

Original: (Mehrotra et al., 2021), and natural language processing (Klyuchnikov et al., 2020).
Revised: (Mehrotra et al., 2021), and natural language processing (Klyuchnikov et al., 2020).

Original: We aim to go beyond such areas to evaluate the potential of NAS to automate the application of ML in truly under-explored domains.
Revised: We aim to go beyond such areas to evaluate the potential of NAS to automate the application of ML in truly under-explored domains.

Original: One analogous work to ours in the ﬁeld of meta-learning is the Meta-Dataset benchmark of few-shot tasks (Triantaﬁllou et al., 2020), which similarly aimed to establish a wide-ranging set of evaluations for that ﬁeld.
Revised: One analogous work to ours in the ﬁeld of meta-learning is the Meta-Dataset benchmark of few-shot tasks (Triantaﬁllou et al., 2020), which similarly aimed to establish a wide-ranging set of evaluations for that ﬁeld.

Original: For our inclusion of diverse tasks, we title our benchmark NAS-Benchto resemble the idea of a 360-degree camera that covers all possible directions.
Revised: For our inclusion of diverse tasks, we title our benchmark NAS-Perf-360 to resemble the idea of a 360-degree camera that covers all possible directions.

Original:  NAS-B ENCH -360: A S UITE OF D IVERSE AND P RACTICAL T ASKS
Revised:  NAS-P ERF -360: A S UITE OF D IVERSE AND P RACTICAL T ASKS

Original: In this section, we introduce the NAS setting targeted by our benchmark, our motivation for organizing a new set of diverse tasks as a NAS evaluation suite, and our task-selection methodology.
Revised: In this section, we introduce the NAS setting targeted by our benchmark, our motivation for organizing a new set of diverse tasks as a NAS evaluation suite, and our task-selection methodology.

Original: We report evaluations of speciﬁc algorithms on this new benchmark in the next section.
Revised: We report evaluations of speciﬁc algorithms on this new benchmark in the next section.

Original: 3. N EURAL A RCHITECTURE S EARCH : P ROBLEM F ORMULATION AND B ASELINES
Revised: 3. N EURAL A RCHITECTURE S EARCH : P ROBLEM F ORMULATION AND B ASELINES

Original: For completeness and clarity, we ﬁrst formally discuss the architecture search problem itself, starting with the extended hypothesis class formulation Li et al.
Revised: For completeness and clarity, we ﬁrst formally discuss the architecture search problem itself, starting with the extended hypothesis class formulation Li et al.

Original: (2021a).
Revised: (2021a).

Original: Here the goal is to use a dataset of points x 2 X to ﬁnd parameters w 2 W and a 2 A of a parameterized function f w ,a : X 7!
Revised: Here the goal is to use a dataset of points x 2 X to ﬁnd parameters w 2 W and a 2 A of a parameterized function f w ,a : X 7!

Original: R  0 that minimize the expectation E x
Revised: R  0 that minimize the expectation E x

Original: ⇠ D f w ,a ( x ) for some test distribution D over X
Revised: ⇠ D f w ,a ( x ) for some test distribution D over X

Original: ; here X is the input space,
Revised: ; here X is the input space,

Original: W is the space of model weights, and A is the set of architectures.
Revised: W is the space of model weights, and A is the set of architectures.

Original: For generality, we do not require the training points to be drawn from D to allow for domain adaptation, as is the case for one of our tasks, and we do not require the loss to be supervised.
Revised: For generality, we do not require the training points to be drawn from D to allow for domain adaptation, as is the case for one of our tasks, and we do not require the loss to be supervised.

Original: Note also that the goal here does not depend on computational or memory efﬁciency, which we do not focus on in our evaluations; our restriction is only that the entire pipeline can be run on an NVIDIA V100 GPU.
Revised: Note also that the goal here does not depend on computational or memory efﬁciency, which we do not focus on in our evaluations; our restriction is only that the entire pipeline can be run on an NVIDIA V100 GPU.

Original: Notably, this formulation makes no distinction between the model weights w and architectures a , treating both as parameters of a larger model.
Revised: Notably, this formulation makes no distinction between the model weights w and architectures a , treating both as parameters of a larger model.

Original: Indeed, the goal of NAS may be seen as similar to model design, except now we include the design of an (often-discrete) architecture space
Revised: Indeed, the goal of NAS may be seen as similar to model design, except now we include the design of an (often-discrete) architecture space

Original: A such that it is easy to ﬁnd an architecture a 2 A and model weights w 2 W whose test loss E D f w ,a is low using a search algorithm.
Revised: A such that it is easy to ﬁnd an architecture a 2 A and model weights w 2 W whose test loss E D f w ,a is low using a search algorithm.

Original: This can be done in a one-shot manner—simultaneously optimizing a and w —or using the standard approach of ﬁrst ﬁnding an architecture a
Revised: This can be done in a one-shot manner—simultaneously optimizing a and w —or using the standard approach of ﬁrst ﬁnding an architecture a

Original: and then keeping it ﬁxed while training model weights w using a pre-speciﬁed algorithm such as stochastic gradient descent (SGD).
Revised: and then keeping it ﬁxed while training model weights w using a pre-speciﬁed algorithm such as stochastic gradient descent (SGD).

Original: This formulation also includes non-NAS methods by allowing the architecture search space to be a singleton.
Revised: This formulation also includes non-NAS methods by allowing the architecture search space to be a singleton.

Original: When the sole architecture is a standard and common network such as WRN (Zagoruyko & Komodakis, 2016), this yields a natural baseline with an algorithm searching for training hyperparameters, not architectures.
Revised: When the sole architecture is a standard and common network such as WRN (Zagoruyko & Komodakis, 2016), this yields a natural baseline with an algorithm searching for training hyperparameters, not architectures.

Original: On the other hand, any architecture space A allows for non-one-shot methods to search for architectures, such as random search through repeatedly sampling architectures and evaluating them from partial training.
Revised: On the other hand, any architecture space A allows for non-one-shot methods to search for architectures, such as random search through repeatedly sampling architectures and evaluating them from partial training.

Original: We adopt this simple method as our random baseline.
Revised: We adopt this simple method as our random baseline.

Original: For our empirical investigation, we compare the performance of state-of-the-art
Revised: For our empirical investigation, we compare the performance of state-of-the-art

Original: NAS approaches against that of the two baselines.
Revised: NAS approaches against that of the two baselines.

Original: 3. T ASK S ELECTION : M OTIVATION AND M ETHODOLOGY
Revised: 3. T ASK S ELECTION : M OTIVATION AND M ETHODOLOGY

Original: Curating a diverse, practical set of tasks for the study of NAS is our primary motivation behind this work.
Revised: Curating a diverse, practical set of tasks for the study of NAS is our primary motivation behind this work.

Original: We observe that past NAS benchmarks focused on creating larger search spaces and more sophisticated search methods for neural networks.
Revised: We observe that past NAS benchmarks focused on creating larger search spaces and more sophisticated search methods for neural networks.

Original: However, the utility of these search spaces and methods are only evaluated on canonical computer vision datasets.
Revised: However, the utility of these search spaces and methods are only evaluated on canonical computer vision datasets.

Original: On a broader range of problems, whether these new methods can improve upon simple baselines remains an open question.
Revised: On a broader range of problems, whether these new methods can improve upon simple baselines remains an open question.

Original: This calls for the introduction of new datasets lest NAS research overﬁts to the biases of CIFAR-10 and
Revised: This calls for the introduction of new datasets lest NAS research overﬁts to the biases of CIFAR-10 and

Original: ImageNet.
Revised: ImageNet.

Original: By identifying these possible biases, future directions in NAS research can be better primed to suit the needs of practitioners and to increase the deployment of NAS.
Revised: By identifying these possible biases, future directions in NAS research can be better primed to suit the needs of practitioners and to increase the deployment of NAS.

Original: Summarized in Table 1, NAS-Bench-360 is comprised of problems that are conducive to processing by convolutional neural networks, which includes a trove of applications associated with spatial and temporal data, spanning single and multiple dimensions.
Revised: Summarized in Table 1, NAS-Perf-360 consists of problems that are conducive to processing by convolutional neural networks, which includes a trove of applications associated with spatial and temporal data, spanning single and multiple dimensions.

Original: Most current NAS methods are not implemented to search for other types of architectures to process tabular data and graph data.
Revised: Most current NAS methods are not implemented to search for other types of architectures to process tabular data and graph data.

Original: Therefore, we have set this scope for our investigation.
Revised: Therefore, we have set this scope for our investigation.

Original: During the selection of tasks, diversity is our primary consideration.
Revised: During the selection of tasks, diversity is our primary consideration.

Original: We deﬁne the following axes of diversity to govern our task-ﬁltering process: the ﬁrst is problem dimensionality, including both 2D with matrix inputs and 1D with sequence inputs; the second is dataset size, for which our selection spans the scale from 1,000 to 1,000,000; the third is problem type , divisible into tasks requiring a singular prediction (point prediction) and multiple predictions (dense prediction); fourth and ﬁnally, diversity is achieved through selecting tasks from various learning objectives from applications of deep learning, where introducing NAS could improve upon the performance of handcrafted neural networks.
Revised: We deﬁne the following axes of diversity to govern our task-ﬁltering process: the ﬁrst is problem dimensionality, including both 2D with matrix inputs and 1D with sequence inputs; the second is dataset size, for which our selection spans the scale from 1,000 to 1,000,000; the third is problem type , divisible into tasks requiring a singular prediction (point prediction) and multiple predictions (dense prediction); fourth and ﬁnally, diversity is achieved through selecting tasks from various learning objectives from applications of deep learning, where introducing NAS could improve upon the performance of handcrafted neural networks.

Original: In lieu of providing raw data, we perform data pre-processing locally and store the processed data on a public Amazon Web Service’s S3 data bucket with download links available on our website.
Revised: In lieu of providing raw data, we perform data pre-processing locally and store the processed data on a public Amazon Web Service’s S3 data bucket with download links available on our website.

Original: Our data treatment largely follows the procedure deﬁned by the researchers who provided them.
Revised: Our data treatment largely follows the procedure deﬁned by the researchers who provided them.

Original: This enhances reproducibility by ensuring the uniformity of input data for different pipelines.
Revised: This enhances reproducibility by ensuring the uniformity of input data for different pipelines.

Original: Speciﬁc pre-processing and augmentation steps are described below.
Revised: Speciﬁc pre-processing and augmentation steps are described below.

Original: CIFAR-100: Standard image classiﬁcation As a starting point of comparison to existing benchmarks, we include the CIFAR-100 task (Krizhevksy, 2009), which contains RGB images from natural settings to be classiﬁed into 100 ﬁne-grained categories.
Revised: CIFAR-100: Standard image classiﬁcation As a starting point of comparison to existing benchmarks, we include the CIFAR-100 task (Krizhevksy, 2009), which contains RGB images from natural settings to be classiﬁed into 100 ﬁne-grained categories.

Original: CIFAR-100 is preferred over CIFARbecause it is more challenging and suffers less from over-ﬁtting in previous research.
Revised: CIFAR-100 is preferred over CIFARbecause it is more challenging and suffers less from over-ﬁtting in previous research.

Original: Spherical: Classifying spherically projected CIFAR-100 images To test NAS methods applied to natural-image-like data, we consider the task of classifying spherical projections of the CIFARimages, which we call Spherical .
Revised: Spherical: Classifying spherically projected CIFAR-100 images To test NAS methods applied to natural-image-like data, we consider the task of classifying spherical projections of the CIFARimages, which we call Spherical .

Original: In addition to scientiﬁc interest, spherical image data is also present in various applications, such as omnidirectional vision in robotics and weather modeling in meteorology, as sensors usually produce distorted image signals in real-life settings.
Revised: In addition to scientiﬁc interest, spherical image data is also present in various applications, such as omnidirectional vision in robotics and weather modeling in meteorology, as sensors usually produce distorted image signals in real-life settings.

Original: To create
Revised: To create

Original: Spherical CIFAR, we project the planar signals of the CIFAR images to the northern hemisphere and add a random rotation to produce spherical signals for each channel following the procedure speciﬁed in Cohen et al.
Revised: Spherical CIFAR, we project the planar signals of the CIFAR images to the northern hemisphere and add a random rotation to produce spherical signals for each channel following the procedure speciﬁed in Cohen et al.

Original: The resulting images are 60 ⇥ 60 pixels with RGB channels.
Revised: The resulting images are 60 ⇥ 60 pixels with RGB channels.

Original: NinaPro: Classifying electromyography signals NinaPro moves away from the image domain to classify hand gestures indicated by electromyography signals.
Revised: NinaPro: Classifying electromyography signals NinaPro moves away from the image domain to classify hand gestures indicated by electromyography signals.

Original: For this, we use a subset of the
Revised: For this, we use a subset of the

Original: NinaPro DB5 dataset (Atzori et al., 2012) in which two Myo armbands collect EMG signals fromtest individuals who hold 18 different hand gestures to be classiﬁed.
Revised: NinaPro DB5 dataset (Atzori et al., 2012) in which two Myo armbands collect EMG signals fromtest individuals who hold 18 different hand gestures to be classiﬁed.

Original: These armbands leverage data
Revised: These armbands leverage data

Original: [Table]
Revised: [Table]

Original: from muscle movement, which is collected using electrodes in the form of wave signals.
Revised: from muscle movement, which is collected using electrodes in the form of wave signals.

Original: Each wave signal is then sampled using a wavelength and frequency prescribed in Côté-Allard et al.
Revised: Each wave signal is then sampled using a wavelength and frequency prescribed in Côté-Allard et al.

Original: (2019) to produce 2D signals.
Revised: (2019) to produce 2D signals.

Original: FSD50K: Labeling sound events FSD50K (Fonseca et al., 2020) is derived from the larger
Revised: FSD50K: Labeling sound events FSD50K (Fonseca et al., 2020) is derived from the larger

Original: Freesound dataset (Fonseca et al., 2017) of Youtube videos with 51,000 clips totaling more than 100 hours of sound.
Revised: Freesound dataset (Fonseca et al., 2017) of Youtube videos with 51,000 clips totaling more than 100 hours of sound.

Original: These clips are manually labeled and equally distributed in 200 classes from the AudioSet ontology (Gemmeke et al., 2017).
Revised: These clips are manually labeled and equally distributed in 200 classes from the AudioSet ontology (Gemmeke et al., 2017).

Original: Each clip could receive multiple labels.
Revised: Each clip could receive multiple labels.

Original: Unlike
Revised: Unlike

Original: TIMIT (Garofolo, 1993), FSD50K does not focus exclusively on sounds of spoken language but includes sound events from physical sources and production mechanisms.
Revised: TIMIT (Garofolo, 1993), FSD50K does not focus exclusively on sounds of spoken language but includes sound events from physical sources and production mechanisms.

Original: The mean average precision (mAP) is used to evaluate classiﬁcation results.
Revised: The mean average precision (mAP) is used to evaluate classiﬁcation results.

Original: [Table]
Revised: [Table]

Original:  Solving partial differential equations (PDEs)
Revised: : Solving partial differential equations (PDEs)

Original: Our ﬁrst regression task, Darcy
Revised: Our ﬁrst regression task, Darcy

Original: Flow , focuses on learning a map from the initial conditions of a PDE to the solution at a later timestep.
Revised: Flow , focuses on learning a map from the initial conditions of a PDE to the solution at a later timestep.

Original: This application aims to replace traditional solvers with learned neural networks, which can output a result in a single forward pass.
Revised: This application aims to replace traditional solvers with learned neural networks, which can output a result in a single forward pass.

Original: The input is a 2d grid specifying the initial conditions of a ﬂuid, and the output is a 2d grid specifying the ﬂuid state at a later time, with the ground truth being the result computed by a traditional solver.
Revised: The input is a 2d grid specifying the initial conditions of a ﬂuid, and the output is a 2d grid specifying the ﬂuid state at a later time, with the ground truth being the result computed by a traditional solver.

Original: We report the mean square error (MSE or ` 2 ).
Revised: We report the mean square error (MSE or ` 2 ).

Original: PSICOV:
Revised: PSICOV:

Original: Protein distance prediction PSICOV studies the use of neural networks in the protein folding prediction pipeline, which has recently received signiﬁcant attention to the success of methods like AlphaFold (Jumper et al., 2020).
Revised: Protein distance prediction PSICOV studies the use of neural networks in the protein folding prediction pipeline, which has recently received signiﬁcant attention to the success of methods like AlphaFold (Jumper et al., 2020).

Original: While the dataset and method they use are too large-scale for our purposes, we consider a smaller set of protein structures to tackle the speciﬁc problem of inter-residual distance predictions outlined in Adhikari (2020b).
Revised: While the dataset and method they use are too large-scale for our purposes, we consider a smaller set of protein structures to tackle the speciﬁc problem of inter-residual distance predictions outlined in Adhikari (2020b).

Original: 2D large-scale features are extracted from protein sequences, resulting in input feature maps with a massive number of channels.
Revised: 2D large-scale features are extracted from protein sequences, resulting in input feature maps with a massive number of channels.

Original: Correspondingly, the labels are pairwise-distance matrices with the same spatial dimension.
Revised: Correspondingly, the labels are pairwise-distance matrices with the same spatial dimension.

Original: The evaluation metric is mean absolute error (MAE or ` 1 ) computed on distances below 8 Å, referred to as MAE 8 .
Revised: The evaluation metric is mean absolute error (MAE or ` 1 ) computed on distances below 8 Å, referred to as MAE 8 .

Original: Cosmic: Identifying cosmic ray contamination Images from space-based facilities are prone to corruption by charged particles collectively referred to as "cosmic rays.
Revised: Cosmic: Identifying cosmic ray contamination Images from space-based facilities are prone to corruption by charged particles collectively referred to as "cosmic rays.

Original: " Cosmic rays on images should be identiﬁed and masked before the images are used for further analysis (Zhang & Bloom,
Revised: " Cosmic rays on images should be identiﬁed and masked before the images are used for further analysis (Zhang & Bloom,

Original: [Table]
Revised: [Table]

Original: The Cosmic task uses imaging data of local resolved galaxies collected from the Hubble
Revised: The Cosmic task uses imaging data of local resolved galaxies collected from the Hubble

Original: Space Telescope.
Revised: Space Telescope.

Original: Inputs and outputs are same-size 2D matrices, with the output predicting whether each pixel in the input is an artifact of cosmic rays.
Revised: Inputs and outputs are same-size 2D matrices, with the output predicting whether each pixel in the input is an artifact of cosmic rays.

Original: We report the false-negative rate (FNR) of identiﬁcation results.
Revised: We report the false-negative rate (FNR) of identiﬁcation results.

Original: ECG: Detecting heart disease Electrocardiograms are frequently used in medicine to diagnose sinus rhythm irregularities.
Revised: ECG: Detecting heart disease Electrocardiograms are frequently used in medicine to diagnose sinus rhythm irregularities.

Original: The ECG task is based on the 2017 PhysioNet Challenge (Clifford et al., 2017), with 9 to 60-second ECG recordings sampled at 300 Hz and labeled using four classes: normal, disease, other, or noisy rhythms.
Revised: The ECG task is based on the 2017 PhysioNet Challenge (Clifford et al., 2017), with 9 to 60-second ECG recordings sampled at 300 Hz and labeled using four classes: normal, disease, other, or noisy rhythms.

Original: Recordings are processed using a ﬁxed sliding window of 1,000 ms and stride of 500 ms.
Revised: Recordings are processed using a ﬁxed sliding window of 1,000 ms and stride of 500 ms.

Original: We report the F1-score according to the challenge’s guidelines.
Revised: We report the F1-score according to the challenge’s guidelines.

Original: Satellite: Satellite image time series analysis Satellite image time series (SITS) are becoming more widely available in earth monitoring applications.
Revised: Satellite: Satellite image time series analysis Satellite image time series (SITS) are becoming more widely available in earth monitoring applications.

Original: Our dataset comes from Formosat-2 satellite images acquired over Toulouse, France (Petitjean et al., 2012).
Revised: Our dataset comes from Formosat-2 satellite images acquired over Toulouse, France (Petitjean et al., 2012).

Original: Available in multiple channels, SITS track the land cover changes over several years as each pixel in the image represents a geographical region.
Revised: Available in multiple channels, SITS track the land cover changes over several years as each pixel in the image represents a geographical region.

Original: The goal of the Satellite task is to generate land cover maps for geo-surveying.
Revised: The goal of the Satellite task is to generate land cover maps for geo-surveying.

Original: Speciﬁcally, a series of pixels in a given color channel constitute a time series to be classiﬁed into 46 land cover types.
Revised: Speciﬁcally, a series of pixels in a given color channel constitute a time series to be classiﬁed into 46 land cover types.

Original: DeepSEA: Predicting functional effects from genetic sequences Predicting chromatin effects of genetic sequence alterations is a signiﬁcant challenge in the ﬁeld to understand genetic diseases.
Revised: DeepSEA: Predicting functional effects from genetic sequences Predicting chromatin effects of genetic sequence alterations is a signiﬁcant challenge in the ﬁeld to understand genetic diseases.

Original: DeepSEA (Zhou & Troyanskaya, 2015), provides a compendium of genomic proﬁles from the
Revised: DeepSEA (Zhou & Troyanskaya, 2015), provides a compendium of genomic proﬁles from the

Original: Encyclopedia of DNA Elements (ENCODE) project (Consortium et al., 2004) to train a predictive model estimating the behavior of chromatin proteins, divided into 919 categories.
Revised: Encyclopedia of DNA Elements (ENCODE) project (Consortium et al., 2004) to train a predictive model estimating the behavior of chromatin proteins, divided into 919 categories.

Original: Due to computation constraints, we subsample 36 of these categories as per Zhang et al.
Revised: Due to computation constraints, we subsample 36 of these categories as per Zhang et al.

Original: (2021a)
Revised: (2021a)

Original: and further take 5% of the training data for prediction.
Revised: and further take 5% of the training data for prediction.

Original: We report the area under the receiver operating characteristic (AUROC) following the previous work.
Revised: We report the area under the receiver operating characteristic (AUROC) following the previous work.

Original:  E XPERIMENTAL DESIGN
Revised:  E XPERIMENTAL DESIGN

Original: Having detailed our construction of NAS-Bench-360, we now describe a set of experiments to demonstrate its usefulness for evaluating NAS methods and guiding research on diverse tasks.
Revised: Having detailed our construction of NAS-Perf-360, we now describe a set of experiments to demonstrate its usefulness for evaluating NAS methods and guiding research on diverse tasks.

Original: In this section, we ﬁrst specify the different NAS methods and baselines we compare, followed by the experimental and reproducibility setup we follow.
Revised: In this section, we ﬁrst specify the different NAS methods and baselines we compare, followed by the experimental and reproducibility setup we follow.

Original: The resulting evaluations are reported in Table 2, aggregate performance in Table 3, and performance proﬁles in Figure 2.
Revised: The resulting evaluations are reported in Table 2, aggregate performance in Table 3, and performance proﬁles in Figure 2.

Original: 4. B ASELINES AND S EARCH P ROCEDURES
Revised: 4. B ASELINES AND S EARCH P ROCEDURES

Original: As noted in Section 1, our initial experiments focus on two practitioners with different resource settings, one with enough compute to tune a WRN and another who can only train it once with the default hyperparameters.
Revised: As noted in Section 1, our initial experiments focus on two practitioners with different resource settings, one with enough compute to tune a WRN and another who can only train it once with the default hyperparameters.

Original: In matching these settings, we focus on two well-known search paradigms: cell-based NAS (using DARTS (Liu et al., 2019b)) and macro NAS (using DenseNAS (Fang et al., 2020)).
Revised: In matching these settings, we focus on two well-known search paradigms: cell-based NAS (using DARTS (Liu et al., 2019b)) and macro NAS (using DenseNAS (Fang et al., 2020)).

Original: We further compare these approaches to two customized NAS methods: Auto-DeepLab (Liu et al., 2019a) for 2D dense prediction and AMBER (Zhang et al., 2021b) for 1D prediction.
Revised: We further compare these approaches to two customized NAS methods: Auto-DeepLab (Liu et al., 2019a) for 2D dense prediction and AMBER (Zhang et al., 2021b) for 1D prediction.

Original: We detail these approaches below.
Revised: We detail these approaches below.

Original: Wide ResNet with Hyperparameter Tuning Architectures based on ResNet He et al.
Revised: Wide ResNet with Hyperparameter Tuning Architectures based on ResNet He et al.

Original: (2016) are a common ﬁrst choice by practitioners faced with a new domain (Fonseca et al., 2020; Adhikari, 2020b); it is thus a natural source of ﬁxed-architecture baselines for our study.
Revised: (2016) are a common ﬁrst choice by practitioners faced with a new domain (Fonseca et al., 2020; Adhikari, 2020b); it is thus a natural source of ﬁxed-architecture baselines for our study.

Original: We use the Wide
Revised: We use the Wide

Original: ResNet variant (Zagoruyko & Komodakis, 2016) with 16 layers and a widen factor of 4, and apply its original training routine directly for the constrained practitioner.
Revised: ResNet variant (Zagoruyko & Komodakis, 2016) with 16 layers and a widen factor of 4, and apply its original training routine directly for the constrained practitioner.

Original: For the other practitioner, we wrap the training procedure with a hyperparameter tuner, ASHA (Li et al., 2018), an asynchronous version of Hyperband (Li et al., 2017).
Revised: For the other practitioner, we wrap the training procedure with a hyperparameter tuner, ASHA (Li et al., 2018), an asynchronous version of Hyperband (Li et al., 2017).

Original: Given a range for each hyperparameter, ASHA uniformly samples conﬁgurations and uses brackets of elimination: at each round, each conﬁguration is trained for some epochs, before the algorithm selects the best-performing portion based on validation metrics.
Revised: Given a range for each hyperparameter, ASHA uniformly samples conﬁgurations and uses brackets of elimination: at each round, each conﬁguration is trained for some epochs, before the algorithm selects the best-performing portion based on validation metrics.

Original: This procedure is useful for ﬁnding suitable hyperparameters in an easy-to-use, automated fashion.
Revised: This procedure is useful for ﬁnding suitable hyperparameters in an easy-to-use, automated fashion.

Original: Cell-based Search Using DARTS
Revised: Cell-based Search Using DARTS

Original: The ﬁrst NAS paradigm we consider is cell-based NAS.
Revised: The ﬁrst NAS paradigm we consider is cell-based NAS.

Original: These methods ﬁrst search for a genotype, a cell containing neural operations.
Revised: These methods ﬁrst search for a genotype, a cell containing neural operations.

Original: During evaluation, an architecture is constructed by replicating the searched cell and stacking them together.
Revised: During evaluation, an architecture is constructed by replicating the searched cell and stacking them together.

Original: The most popular search space for this approach is DARTS (Liu et al., 2019b), which assigns one of eight operations to six edges in two types of cells: “normal” cells preserve the shape of the input while “reduction” cells downsample it.
Revised: The most popular search space for this approach is DARTS (Liu et al., 2019b), which assigns one of eight operations to six edges in two types of cells: “normal” cells preserve the shape of the input while “reduction” cells downsample it.

Original: For dense tasks, we do not use the reduction cell to prevent introducing a bottleneck.
Revised: For dense tasks, we do not use the reduction cell to prevent introducing a bottleneck.

Original: For 1D tasks, all convolutions in the cell are converted from 2D to 1D.
Revised: For 1D tasks, all convolutions in the cell are converted from 2D to 1D.

Original: Finally, to adhere to standard ML practices, we do not adapt the standard DARTS pipeline from the original paper.
Revised: Finally, to adhere to standard ML practices, we do not adapt the standard DARTS pipeline from the original paper.

Original: As this search space has been heavily studied, we use as a search routine a recent approach, GAEA PC-DARTS (GAEA), that achieves some of the best-known results on CIFARand ImageNet (Li et al., 2021a).
Revised: As this search space has been heavily studied, we use as a search routine a recent approach, GAEA PC-DARTS (GAEA), that achieves some of the best-known results on CIFARand ImageNet (Li et al., 2021a).

Original: This NAS approach, due to its heavy retraining routine, is compared to the tuned WRN baseline of the less-resource-constrained practitioner.
Revised: This NAS approach, due to its heavy retraining routine, is compared to the tuned WRN baseline of the less-resource-constrained practitioner.

Original:  NAS Using DenseNAS
Revised: Macro NAS Using DenseNAS

Original: The second NAS paradigm we consider is macro NAS.
Revised: The second NAS paradigm we consider is macro NAS.

Original: Instead of building from a ﬁxed cell, it requires the speciﬁcation of a supernet with different inter-connected blocks.
Revised: Instead of building from a ﬁxed cell, it requires the speciﬁcation of a supernet with different inter-connected blocks.

Original: These blocks and connections are then pruned to construct an architecture.
Revised: These blocks and connections are then pruned to construct an architecture.

Original: For our benchmark, we choose a recent search space, DenseNAS (Fang et al., 2020), which has near state-of-the-art results on ImageNet.
Revised: For our benchmark, we choose a recent search space, DenseNAS (Fang et al., 2020), which has near state-of-the-art results on ImageNet.

Original: DenseNAS searches for architectures with densely-connected, customizable routing blocks to emulate DenseNet (Huang et al., 2017).
Revised: DenseNAS searches for architectures with densely-connected, customizable routing blocks to emulate DenseNet (Huang et al., 2017).

Original: In our experiments, we use the ResNet-based search space, DenseNAS-R1, which contains all neural operations of the WRN backbone.
Revised: In our experiments, we use the ResNet-based search space, DenseNAS-R1, which contains all neural operations of the WRN backbone.

Original: For 2D tasks, we adapt two super networks from the one used for ImageNet as inputs to the search algorithm.
Revised: For 2D tasks, we adapt two super networks from the one used for ImageNet as inputs to the search algorithm.

Original: The super network for dense tasks maintains the same spatial dimensions without downsampling to avoid bottlenecks, and we lower the learning rate for evaluating architectures to prevent divergence.
Revised: The super network for dense tasks maintains the same spatial dimensions without downsampling to avoid bottlenecks, and we lower the learning rate for evaluating architectures to prevent divergence.

Original: For transferring to 1D tasks, all network operations are switched from 2D to 1D.
Revised: For transferring to 1D tasks, all network operations are switched from 2D to 1D.

Original: Other training and evaluation procedures are identical to those in the original paper and uniform across all tasks.
Revised: Other training and evaluation procedures are identical to those in the original paper and uniform across all tasks.

Original: DenseNAS is quick to search and evaluate, making it comparable to the ﬁxed WRN baseline.
Revised: DenseNAS is quick to search and evaluate, making it comparable to the ﬁxed WRN baseline.

Original: We apply another search method to the ﬁxed DenseNAS space to study the relative importance of search algorithms.
Revised: We apply another search method to the ﬁxed DenseNAS space to study the relative importance of search algorithms.

Original: Random search is implemented through randomly sampling architectures from the
Revised: Random search is implemented through randomly sampling architectures from the

Original: DenseNAS space and validating them after a brief training period of 10 epochs before evaluating the best performer.
Revised: DenseNAS space and validating them after a brief training period of 10 epochs before evaluating the best performer.

Original: To ensure fairness of comparison, we allot equal GPU hours to random search and regular DenseNAS search, additionally applying a soft constraint that random architecture model sizes should not surpass DenseNAS searched architecture sizes signiﬁcantly.
Revised: To ensure fairness of comparison, we allot equal GPU hours to random search and regular DenseNAS search, additionally applying a soft constraint that random architecture model sizes should not surpass DenseNAS searched architecture sizes signiﬁcantly.

Original: [Table]
Revised: [Table]

Original: Domain-speciﬁc NAS Baselines: Auto-DL and AMBER To study the importance of search spaces, we further handpick two domain-speciﬁc NAS approaches applicable only to a subset of the tasks.
Revised: Domain-speciﬁc NAS Baselines: Auto-DL and AMBER To study the importance of search spaces, we further handpick two domain-speciﬁc NAS approaches applicable only to a subset of the tasks.

Original: Using an encoder-decoder architecture, Auto-DeepLab (Auto-DL) (Liu et al., 2019a) is designed for dense prediction, e.g., semantic segmentation.
Revised: Using an encoder-decoder architecture, Auto-DeepLab (Auto-DL) (Liu et al., 2019a) is designed for dense prediction, e.g., semantic segmentation.

Original: While the decoder is ﬁxed, Auto-DL searches for an encoder via ﬁrst-order DARTS.
Revised: While the decoder is ﬁxed, Auto-DL searches for an encoder via ﬁrst-order DARTS.

Original: We evaluate Auto-DL on Darcy Flow, PSICOV, and
Revised: We evaluate Auto-DL on Darcy Flow, PSICOV, and

Original: Cosmic tasks.
Revised: Cosmic tasks.

Original: AMBER (Zhang et al., 2021b) aims to automate neural network design for 1D genomic data.
Revised: AMBER (Zhang et al., 2021b) aims to automate neural network design for 1D genomic data.

Original: This framework establishes a 12-layer network backbone and parametrizes a long-short term memory network (LSTM) as a controller to search for suitable 1D operations and residual connections, following the ENAS (Pham et al., 2018) optimization protocol.
Revised: This framework establishes a 12-layer network backbone and parametrizes a long-short term memory network (LSTM) as a controller to search for suitable 1D operations and residual connections, following the ENAS (Pham et al., 2018) optimization protocol.

Original: At each step, the controller samples architectures to compute reward based on area under the receiver operating characteristics (AUROC) before outputting the highest-reward architecture.
Revised: At each step, the controller samples architectures to compute reward based on area under the receiver operating characteristics (AUROC) before outputting the highest-reward architecture.

Original: We evaluate AMBER on the ECG, Satellite, and
Revised: We evaluate AMBER on the ECG, Satellite, and

Original: DeepSEA tasks.
Revised: DeepSEA tasks.

Original: 4. E XPERIMENTAL S ETUP
Revised: 4. E XPERIMENTAL S ETUP

Original: Below we discuss the main reporting details of our empirical evaluation.
Revised: Below we discuss the main reporting details of our empirical evaluation.

Original: • Hyperparameter tuning: As detailed in the Appendix, we use the same hyperparameter ranges across all tasks to tune WRN.
Revised: • Hyperparameter tuning: As detailed in the Appendix, we use the same hyperparameter ranges across all tasks to tune WRN.

Original: The tuning budget is selected to match that of DARTS (GAEA).
Revised: The tuning budget is selected to match that of DARTS (GAEA).

Original: • Aggregation Metrics: Table 3 contains the median rank and relative improvement over WRN of each method for direct comparison via a single number.
Revised: • Aggregation Metrics: Table 3 contains the median rank and relative improvement over WRN of each method for direct comparison via a single number.

Original: We also employ performance proﬁles (Dolan & Moré, 2002) in Figure 2, an approach that allows an analysis taking into account outliers while not allowing small differences in performance to dominate; as described in Figure 1 these curves denote for each ⌧ the fraction of tasks on which a method is no worse than a ⌧ -factor from the optimal.
Revised: We also employ performance proﬁles (Dolan & Moré, 2002) in Figure 2, an approach that allows an analysis taking into account outliers while not allowing small differences in performance to dominate; as described in Figure 1 these curves denote for each ⌧ the fraction of tasks on which a method is no worse than a ⌧ -factor from the optimal.

Original: • Software and hardware:
Revised: • Software and hardware:

Original: We adopt the free, open-source software Determined 2 for experiment management, hyperparameter tuning, and cloud deployment.
Revised: We adopt the free, open-source software Determined 2 for experiment management, hyperparameter tuning, and cloud deployment.

Original: All experiments are performed on a single p3.2xlarge instance with an NVIDIA V100 GPU.
Revised: All experiments are performed on a single p3.2xlarge instance with an NVIDIA V100 GPU.

Original: Costs in GPU-hours are in the appendix.
Revised: Costs in GPU-hours are in the appendix.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: We conclude our presentation of NAS-Bench-360 via an analysis that (a) reveals new insights about the capabilities and robustness of current NAS methods and (b) demonstrates how our benchmark can enable critical next steps in NAS research.
Revised: We conclude our presentation of NAS-Perf-360 via an analysis that (a) reveals new insights about the capabilities and robustness of current NAS methods and (b) demonstrates how our benchmark can enable critical next steps in NAS research.

Original: In particular, we start by considering our two practitioners faced with a choice of spending their limited compute on a (possibly tuned) ﬁxedarchitecture CNN or trying to ﬁnd a better architecture using NAS.
Revised: In particular, we start by considering our two practitioners faced with a choice of spending their limited compute on a (possibly tuned) ﬁxedarchitecture CNN or trying to ﬁnd a better architecture using NAS.

Original: With this study, we investigate whether modern NAS methods perform well beyond the tasks for which they were designed.
Revised: With this study, we investigate whether modern NAS methods perform well beyond the tasks for which they were designed.

Original: A surface-level analysis suggests that under light resource constraints, modern NAS in the form of DARTS (GAEA) is quite robust to a wide variety of tasks:
Revised: A surface-level analysis suggests that under light resource constraints, modern NAS in the form of DARTS (GAEA) is quite robust to a wide variety of tasks:

Original: Table 3 shows it is the highestranked domain-independent method and attains the most signiﬁcant improvement over the ﬁxed
Revised: Table 3 shows it is the highestranked domain-independent method and attains the most signiﬁcant improvement over the ﬁxed

Original: WRN baseline.
Revised: WRN baseline.

Original: The performance proﬁle in Figure 2 (left) also seems favorable, although it is overtaken by tuned WRN at a higher ⌧ -suboptimality.
Revised: The performance proﬁle in Figure 2 (left) also seems favorable, although it is overtaken by tuned WRN at a higher ⌧ -suboptimality.

Original: However, a closer look at 2D point tasks in Figure 2 (right) reveals that DARTS is quite poor there, despite its design domain being image classiﬁcation; in particular, it performs very poorly on NinaPro and FSD50K.
Revised: However, a closer look at 2D point tasks in Figure 2 (right) reveals that DARTS is quite poor there, despite its design domain being image classiﬁcation; in particular, it performs very poorly on NinaPro and FSD50K.

Original: Furthermore, on tasks where it performs well, it can still lag behind expert architectures; for example, on Darcy
Revised: Furthermore, on tasks where it performs well, it can still lag behind expert architectures; for example, on Darcy

Original: Flow, networks that use FNO (Li et al., 2021b) or XD (Roberts et al., 2021) operations do much better.
Revised: Flow, networks that use FNO (Li et al., 2021b) or XD (Roberts et al., 2021) operations do much better.

Original: Overall, our results suggest that this practitioner can apply NAS and expect to see some improvement, but also risks catastrophically poor performance (e.g., FSD50K) or not getting truly state-of-the-art results (e.g., Darcy Flow). 
Revised: Overall, our results suggest that this practitioner can apply NAS and expect to see some improvement, but also risks catastrophically poor performance (e.g., FSD50K) or not getting truly state-of- the-art results (e.g., Darcy Flow). Under

Original: Under stronger budget constraints, our experiments strongly suggest that a practitioner shouldsimply apply the default WRN to their problem rather than undergo the additional complexity of using DenseNAS, as the latter attains little-to-no improvement over the former in Tableand has a usually-worse performance proﬁles Figure 2.
Revised: stronger budget constraints, our experiments strongly suggest that a practitioner should

Revised: simply

Revised: apply the

Revised: default WRN

Revised: to their

Revised: problem rather

Revised: than

Revised: undergo the additional complexity of using DenseNAS,

Revised: as the

Revised: latter

Revised: attains

Revised: little

Revised: -to-no improvement over the former in Table

Revised: and

Revised: has a

Revised: usually worse performance proﬁles Figure 2.

Original: One bright point, however, is the strong performance of DenseNAS on FSD50K, where it outperforms all methods even while DARTS (GAEA) fails.
Revised: One bright point , however , is the strong performance of DenseNAS on FSD50K, where it outperforms all methods even while DARTS (GAEA) fails

Original: These ﬁrst experiments suggest that the modern NAS methods are not always robust to diverse tasks, especially under resource-constrained settings.
Revised: These ﬁrst experiments suggest that the modern NAS methods are not always robust to diverse tasks, especially under resource-constrained settings.

Original: We believe that NAS-Bench-360’s main roles as a future benchmark include developing an understanding of the multi-domain performance of existing approaches and guiding research into better NAS methods.
Revised: We believe that NAS-Perf-360’s main roles as a future benchmark include developing an understanding of the multi-domain performance of existing approaches and guiding research into better NAS methods.

Original: While the latter is beyond the scope of this paper, our additional experiments demonstrate how NAS-Bench-360 facilitates the former.
Revised: While the latter is beyond the scope of this paper, our additional experiments demonstrate how NAS-Perf-360 facilitates the former.

Original: Notably, several results address the question of the relative importance of search space v. search algorithm.
Revised: Notably, several results address the question of the relative importance of search space v. search algorithm.

Original: For example, Table 3 shows that on DenseNAS, random search is nearly identical to the more sophisticated weight-sharing scheme of the original paper; the two algorithms’ performance proﬁles are also difﬁcult to distinguish in Figure 2.
Revised: For example, Table 3 shows that on DenseNAS, random search is nearly identical to the more sophisticated weight-sharing scheme of the original paper; the two algorithms’ performance proﬁles are also difﬁcult to distinguish in Figure 2.

Original: Furthermore, AMBER—a 1D NAS method whose search space includes larger-kernel convolutions for handling such tasks—does better than
Revised: Furthermore, AMBER—a 1D NAS method whose search space includes larger-kernel convolutions for handling such tasks—does better than

Original: GAEA even though it uses an older search algorithm (ENAS).
Revised: GAEA even though it uses an older search algorithm (ENAS).

Original: These both suggest that search space design, including the use of a wider variety of operations, may be at least as crucial for success as the search algorithm.
Revised: These both suggest that search space design, including the use of a wider variety of operations, may be at least as crucial for success as the search algorithm.

Original: This point is reinforced by example tasks such as Darcy Flow, where architectures with more exotic operations substantially outperform our best results, as discussed earlier.
Revised: This point is reinforced by example tasks such as Darcy Flow, where architectures with more exotic operations substantially outperform our best results, as discussed earlier.

Original: NAS-Bench-360 also reveals failure points of several methods, not just general ones that usually perform quite well such as DARTS (GAEA) but also the objective-speciﬁc approach Auto-DL, which despite being designed for dense prediction tasks, does poorly on all those considered here.
Revised: NAS-Perf-360 also reveals failure points of several methods, not just general ones that usually perform quite well such as DARTS (GAEA) but also the objective-speciﬁc approach Auto-DL, which despite being designed for dense prediction tasks, does poorly on all those considered here.

Original: Understanding when and why these performance drops happen is critical to developing a more robust NAS that is useful not just on average but in more challenging settings.
Revised: Understanding when and why these performance drops happen is critical to developing a more robust NAS that is useful not just on average but in more challenging settings.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: NAS-Bench-360 is a new  benchmark consisting of ten diverse tasks derived from various ﬁelds of research and practice.
Revised: NAS-Perf-360 is a new performance benchmark consisting of ten diverse tasks derived from various ﬁelds of research and practice.

Original: It is designed for reproducible research on an academic budget that will guide the development of NAS methods and other automated approaches towards more robust performance across different domains.
Revised: It is designed for reproducible research on an academic budget that will guide the development of NAS methods and other automated approaches towards more robust performance across different domains.

Original: In initial results, we have demonstrated both the need for such a benchmark and the utility of NAS-Bench-360 speciﬁcally for developing new search spaces and algorithms; we welcome researchers to use its tasks to develop new procedures for automating
Revised: In initial results, we have demonstrated both the need for such a benchmark and the utility of NAS-Perf-360 speciﬁcally for developing new search spaces and algorithms; we welcome researchers to use its tasks to develop new procedures for automating

Original: ML.
Revised: ML.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Within our array of tasks, only NinaPro, ECG, and DeepSEA contain human-derived data.
Revised: Within our array of tasks, only NinaPro, ECG, and DeepSEA contain human-derived data.

Original: Our chosen subset of NinaPro contains only muscle movement data without any exposure of personal information.
Revised: Our chosen subset of NinaPro contains only muscle movement data without any exposure of personal information.

Original: The original experiments to acquire NinaPro data are approved by the ethics commission of the canton of Valais, Switzerland (Atzori et al., 2012).
Revised: The original experiments to acquire NinaPro data are approved by the ethics commission of the canton of Valais, Switzerland (Atzori et al., 2012).

Original: The ECG data derives from an open challenge and is provided by the medical device company AliveCor, under the GPL license allowing it for public use.
Revised: The ECG data derives from an open challenge and is provided by the medical device company AliveCor, under the GPL license allowing it for public use.

Original: The DeepSEA data derived from ENCODE is part of an international collaborative effort, which is overseen and funded by the National Human Genome Research
Revised: The DeepSEA data derived from ENCODE is part of an international collaborative effort, which is overseen and funded by the National Human Genome Research

Original: Institute (NHGRI).
Revised: Institute (NHGRI).

Original: For other datasets, we have listed the data licenses in the appendix.
Revised: For other datasets, we have listed the data licenses in the appendix.

Original: While we do not view the speciﬁc datasets in NAS-Bench-360 as potential candidates for misuse, the broader goal of applying NAS to new domains comes with inherent risks that may require mitigation on an application-by-application basis.
Revised: While we do not view the speciﬁc datasets in NAS-Perf-360 as potential candidates for misuse, the broader goal of applying NAS to new domains comes with inherent risks that may require mitigation on an application-by-application basis.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: The following measures are taken to ensure reproducibility:
Revised: The following measures are taken to ensure reproducibility:

Original: We store the processed data in AWS S3, and data splits are the same for all experiments.
Revised: We store the processed data in AWS S3, and data splits are the same for all experiments.

Original: Code is always executed in a ﬁxed Docker container using a pre-built image on Docker Hub.
Revised: Code is always executed in a ﬁxed Docker container using a pre-built image on Docker Hub.

Original: This guarantees a uniform execution environment and saves users from conﬁguring dependencies.
Revised: This guarantees a uniform execution environment and saves users from conﬁguring dependencies.

Original: Via the speciﬁcation of a random seed, Determined controls several essential sources of randomness during execution, including hyperparameter sampling and training data shufﬂing. 4.
Revised: Via the speciﬁcation of a random seed, Determined controls several essential sources of randomness during execution, including hyperparameter sampling and training data shufﬂing. 4.

Original: During training, we validate on the entire validation set, not on a mini-batch, to limit stochasticity.
Revised: During training, we validate on the entire validation set, not on a mini-batch, to limit stochasticity.

Original: Code and download links of all datasets are available at the anonymized repository: https: //anonymous.4open.science/r/NAS-Bench-360-26D
Revised: Code and download links of all datasets are available at the anonymized repository: https: //anonymous.4open.science/r/NAS-Bench-360-26D

Original: . High accuracy protein structure prediction
Revised: . High accuracy protein structure prediction