Original: BANANAS: B AYESIAN O PTIMIZATION WITH N EURAL N ETWORKS FOR N EURAL A RCHITECTURE S EARCH
Revised: BANANAS: B AYESIAN O PTIMIZATION WITH N EURAL N ETWORKS FOR N EURAL A RCHITECTURE S EARCH

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural Architecture Search (NAS) has seen an explosion of research in the past few years
Revised: Neural Architecture Search (NAS) has seen an explosion of research in the past few years.

Original: A variety of methods have been proposed to perform NAS, including reinforcement learning, Bayesian optimization with a Gaussian process model, evolutionary search, and gradient descent.
Revised: A variety of methods have been proposed to perform NAS, including reinforcement learning, Bayesian optimization with a Gaussian process model, evolutionary search, and gradient descent.

Original: In this work, we design a NAS algorithm that performs Bayesian optimization using a neural network model.
Revised: In this work, we design a NAS algorithm that performs Bayesian optimization using a neural network model.

Original: We develop a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model.
Revised: We develop a path-based encoding scheme to featurize the neural architectures that are used to train the neural network model.

Original: This strategy is particularly effective for encoding architectures in cell-based search spaces.
Revised: This strategy is particularly effective for encoding architectures in cell-based search spaces.

Original: After training on justrandom neural architectures, we are able to predict the validation accuracy of a new architecture to within one percent of its true accuracy on average.
Revised: After training on justrandom neural architectures, we are able to predict the validation accuracy of a new architecture to within one percent of its true accuracy on average, for popular search spaces.

Original: This may be of independent interest beyond Bayesian neural architecture search.
Revised: This may be of independent interest beyond Bayesian neural architecture search.

Original: We test our algorithm on the NASBenchdataset (Ying et al. 2019) and  show that our algorithmsigniﬁcantly outperforms other NAS methods including evolutionary search, reinforcement learning and AlphaX (Wang et al. 2019).
Revised: We test our algorithm on the NASBench (Ying et al. 2019) and DARTS (Liu et al. 2018) search spaces, and we show that our algorithm outperforms other NAS methods including evolutionary search, reinforcement learning, AlphaX (Wang et al. 2019), ASHA, DARTS, and TPE.

Original: Our algorithm is over 100x more efﬁcient than random search, and 3.8x more efﬁcient than the next-best algorithm.
Revised: Our algorithm is over 100x more efﬁcient than random search, and 3.8x more efﬁcient than the next-best algorithm on the

Original: We also test our algorithm on the search space used in DARTS (Liu et al. 2018) and show that our algorithm is competitive with state-of-the-art NAS algorithms on this search space.
Revised: NASBench dataset. As there have been problems with fair and reproducible experimental evauations in the ﬁeld of NAS, we adhere to the recent NAS research checklist (Lindauer and Hutter 2019) to facilitate NAS research.

Revised: In particular, our implementation has been made publicly available, including all details needed to fully reproduce our results.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Since the deep learning revolution in 2012, neural networks have been growing increasingly more specialized and more complex (Krizhevsky et al., 2012; Huang et al., 2017; Szegedy et al., 2017).
Revised: Since the deep learning revolution in 2012, neural networks have been growing increasingly more specialized and more complex (Krizhevsky et al., 2012; Huang et al., 2017; Szegedy et al., 2017).

Original: Developing new state-of-the-art architectures often takes a vast amount of engineering and domain knowledge.
Revised: Developing new state-of-the-art architectures often takes a vast amount of engineering and domain knowledge.

Original: A new area of research, neural architecture search (NAS), seeks to automate this process.
Revised: A new area of research, neural architecture search (NAS), seeks to automate this process.

Original: Since the popular NASNet paper (Zoph & Le, 2017), there has been a ﬂurry of research on neural architecture search (Liu et al., 2018a; Pham et al., 2018; Liu et al., 2018b; Kandasamy et al., 2018;
Revised: Since the popular NASNet paper (Zoph & Le, 2017), there has been a ﬂurry of research on neural architecture search (Liu et al., 2018a; Pham et al., 2018; Liu et al., 2018b; Kandasamy et al., 2018;

Original: Elsken et al., 2018; Jin et al., 2018; Sciuto et al., 2019; Cortes et al., 2017; Baker et al., 2016; Liu et al., 2017).
Revised: Elsken et al., 2018; Jin et al., 2018; Sciuto et al., 2019; Cortes et al., 2017; Baker et al., 2016; Liu et al., 2017).

Original: Many methods have been proposed for NAS, including random search, evolutionary search, reinforcement learning, Bayesian optimization, and gradient descent.
Revised: Many methods have been proposed for NAS, including random search, evolutionary search, reinforcement learning, Bayesian optimization, and gradient descent.

Original: Each technique has its advantages and disadvantages.
Revised: Each technique has its advantages and disadvantages.

Original: For example, reinforcement learning with a controller neural network can learn very complex policies.
Revised: For example, reinforcement learning with a controller neural network can learn very complex policies.

Original: However, as some past work has argued, NAS is an optimization problem – ﬁnd the architecture with the lowest validation error – so there is no explicit need to maintain a state and transition function, and this approach may require evaluating a larger number of architectues to achieve the same result as other methods (Kandasamy et al., 2018; Elsken et al., 2018; Jiang et al., 2017).
Revised: However, as some past work has argued, NAS is an optimization problem – ﬁnd the architecture with the lowest validation error – so there is no explicit need to maintain a state and transition function, and this approach may require evaluating a larger number of architectues to achieve the same result as other methods (Kandasamy et al., 2018; Elsken et al., 2018; Jiang et al., 2017).

Original: Alternatively, Bayesian optimization with a Gaussian process model is a leading method for zeroth order optimization of expensive-to-evaluate functions, which is precisely the setting of deep learning model selection.
Revised: Alternatively, Bayesian optimization with a Gaussian process model is a leading method for zeroth order optimization of expensive-to-evaluate functions, which is precisely the setting of deep learning model selection.

Original: Although Bayesian optimization has seen great success in hyperparameter optimization (Golovin et al., 2017; Li et al., 2016) for deep models, a host of difﬁculties arise when using Bayesian optimization for neural architecture search.
Revised: Although Bayesian optimization has seen great success in hyperparameter optimization (Golovin et al., 2017; Li et al., 2016) for deep models, a host of difﬁculties arise when using Bayesian optimization for neural architecture search.

Original: For example, while there is a substantial body of work on Bayesian optimization over Euclidean spaces, using
Revised: For example, while there is a substantial body of work on Bayesian optimization over Euclidean spaces, using

Original: Bayesian optimization for NAS has so far required specifying a distance function between neural  architectures, in order to deﬁne a surrogate model.
Revised: Bayesian optimization for NAS has so far required specifying a distance function between neural [Figure] architectures, in order to deﬁne a surrogate model.

Original: This is often a cumbersome task involving tuning hyperparameters of the distance function (Kandasamy et al., 2018; Jin et al., 2018).
Revised: This is often a cumbersome task involving tuning hyperparameters of the distance function (Kandasamy et al., 2018; Jin et al., 2018).

Original: Furthermore,
Revised: Furthermore,

Original: Bayesian optimization most commonly uses a Gaussian process (GP) model, and it can be quite challenging to achieve highly accurate prediction performance with GPs, given the potentially high dimensional input architectures (Elsken et al., 2018).
Revised: Bayesian optimization most commonly uses a Gaussian process (GP) model, and it can be quite challenging to achieve highly accurate prediction performance with GPs, given the potentially high dimensional input architectures (Elsken et al., 2018).

Original: In this work, we run Bayesian optimization with a neural network model.
Revised: In this work, we run Bayesian optimization with a neural network model.

Original: That is, in every iteration of Bayesian optimization, we train a meta neural network to predict the accuracy of unseen neural architectures in the search space.
Revised: That is, in every iteration of Bayesian optimization, we train a meta neural network to predict the accuracy of unseen neural architectures in the search space.

Original: This procedure avoids the aforementioned problems with Bayesian optimization NAS: the model is powerful enough to predict neural network accuracies, and there is no need to construct a distance function between neural networks by hand. 
Revised: This procedure avoids the aforementioned problems with Bayesian optimization NAS: the model is powerful enough to predict neural network accuracies, and there is no need to construct a distance function between neural networks by hand. Furthermore,using a neural network model scales far better than a GP model, as it avoids the computationallyintensive matrix inversion operation.

Original: Training a meta neural network to predict the accuracy of neural architectures is a challenging task.
Revised: Training a meta neural network to predict the accuracy of neural architectures is a challenging task.

Original: The majority of popular NAS algorithms are deployed over a directed acyclic graph (DAG) search space – the set of possible architectures is comprised of the set of all DAGs of a certain size, together with all possible combinations of operations on each node (Zoph & Le, 2017; Pham et al., 2018;
Revised: The majority of popular NAS algorithms are deployed over a directed acyclic graph (DAG) search space – the set of possible architectures is comprised of the set of all DAGs of a certain size, together with all possible combinations of operations on each node (Zoph & Le, 2017; Pham et al., 2018;

Original: Liu et al., 2018b; Ying et al., 2019).
Revised: Liu et al., 2018b; Ying et al., 2019).

Original: This poses a roadblock when using a meta neural network to predict the accuracies of neural architectures, since graph structures are difﬁcult for neural networks to understand directly from the adjacency matrix (Zhou et al., 2018).
Revised: This poses a roadblock when using a meta neural network to predict the accuracies of neural architectures, since graph structures are difﬁcult for neural networks to understand directly from the adjacency matrix (Zhou et al., 2018).

Original: We use a path-based encoding scheme to encode a neural architecture, which drastically improves the predictive accuracy of our meta neural network.
Revised: We use a path-based encoding scheme to encode a neural architecture, which drastically improves the predictive accuracy of our meta neural network.

Original: Each binary feature represents a unique path through the DAG from the input layer to the output layer.
Revised: Each binary feature represents a unique path through the DAG from the input layer to the output layer.

Original: This method improves the accuracy of our meta neural network by a factor of 3.
Revised: This method improves the accuracy of our meta neural network by a factor of 3.

Original: With just 200 training points, our meta neural network is able to predict the accuracy of unseen neural networks to within one percent on average on the popular NASBench dataset (Ying et al., 2019).
Revised: With just 200 training points, our meta neural network is able to predict the accuracy of unseen neural networks to within one percent on average on the popular NASBench dataset (Ying et al., 2019).

Original: This may be of interest beyond Bayesian neural architecture search.
Revised: This may be of interest beyond Bayesian neural architecture search.

Original: In our Bayesian optimization NAS algorithm, we train an ensemble of neural networks to predict the mean and variance of candidate neural architectures, from which we compute the upper conﬁdence bound (UCB) acquisition function (Srinivas et al., 2009).
Revised: In our Bayesian optimization NAS algorithm, we train an ensemble of neural networks to predict the mean and variance of candidate neural architectures, from which we compute the upper conﬁdence bound (UCB) acquisition function (Srinivas et al., 2009).

Original: Finally, we use a mutation algorithm to optimize the acquisition function.
Revised: Finally, we use a mutation algorithm to optimize the acquisition function.

Original: We compare our NAS algorithm against a host of black-box optimization NAS algorithms including regularized evolution, reinforcement learning, Bayesian optimization with a GP model,and AlphaX (Wang et al., 2018).
Revised: We compare our NAS algorithm against a host of black-box optimization NAS algorithms including regularized evolution, reinforcement learning, Bayesian optimization with a GP model, AlphaX (Wang et al., 2018), ASHA (Li & Talwalkar, 2019), DARTS (Liu et al., 2018b), and TPE (Bergstra et al., 2011).

Original: On the NASBench dataset, our method sees signiﬁcant improvements compared to all other algorithms we tried across 200 trials.
Revised: On the NASBench dataset, our method sees signiﬁcant improvements compared to all other algorithms we tried across 200 trials.

Original: It takes the next-best algorithm 3.8 times the number of queries to achieve the same mean test accuracy as our method, and it takes random search more than 100 times the number of queries.
Revised: It takes the next-best algorithm 3.8 times the number of queries to achieve the same mean test accuracy as our method, and it takes random search more than 100 times the number of queries.

Original: We also run our algorithm on the search space fromDARTS (Liu et al., 2018b) and compare to the top architectures found by DARTS and ASHA (Li & Talwalkar, 2019), two leading NAS methods. Our algorithm achieves accuracy greater than ASHA and within 0.2% of DARTS, when given a budget ofjust 100 neural networks trained for 50 epochs each.
Revised: On the search space from DARTS, when given a budget of 100 neural architecture queries for 50 epochs each, our algorithm achieves a best of 2.57% and average of 2.64% test error, which beats all NAS algorithms with which we could fairly compare (e.g., same search space used, and same hyperparameters for the ﬁnal training).

Original: Our NAS algorithm has several moving parts, including the meta neural network, the path-based feature encoding of neural ne, the acquisition function, and the acquisition optimization strategy.
Revised: Our NAS algorithm has several moving parts, including the meta neural network, the path-based feature encoding of neural architectures, the acquisition function, and the acquisition optimization strategy.

Original: We run a thorough ablation study by removing each piece of the algorithm separately.
Revised: We run a thorough ablation study by removing each piece of the algorithm separately.

Original: We show all components are necessary to achieve the best performance.
Revised: We show all components are necessary to achieve the best performance.

Original: Finally, to adhere to recent calls for reproducible NAS research (Ying et al., 2019; Li & Talwalkar, 2019; Lindauer & Hutter, 2019), we address all points in the NAS research checklist (Lindauer & Hutter, 2019) and will make our  code  available upon the public release of this paper.
Revised: Finally, to adhere to recent calls for reproducible NAS research (Ying et al., 2019; Li & Talwalkar, 2019; Lindauer & Hutter, 2019), we address all points in the NAS research checklist (Lindauer & Hutter, 2019) and will make our anonymized code is available at https://www.github.com/iclr474/bananas .

Original: Our contributions.
Revised: Our contributions.

Original: We summarize our main contributions below.
Revised: We summarize our main contributions below.

Original: • We propose a novel path-based encoding scheme for neural architectures.
Revised: • We propose a novel path-based encoding scheme for neural architectures.

Original: This is a method for featurizing neural architectures by enumerating paths along the DAG from the input node to the output node.
Revised: This is a method for featurizing neural architectures by enumerating paths along the DAG from the input node to the output node.

Original: When predicting the validation accuracy of neural networks, using this featurization drastically improves the performance compared to standard encodings such as the adjacency matrix encoding, making it feasible to use a meta neural network as a model in Bayesian optimization.
Revised: When predicting the validation accuracy of neural networks, using this featurization drastically improves the performance compared to standard encodings such as the adjacency matrix encoding, making it feasible to use a meta neural network as a model in Bayesian optimization.

Original: • We design a Bayesian optimization algorithm using a meta neural network model to perform NAS over cell-based search spaces.
Revised: • We design a Bayesian optimization algorithm using a meta neural network model to perform NAS over cell-based search spaces.

Original: We use an ensemble of neural networks to output a mean and variance for each candidate architecture and then use the UCB acquisition function to guide the search.
Revised: We use an ensemble of neural networks to output a mean and variance for each candidate architecture and then use the UCB acquisition function to guide the search.

Original: We perform an extensive study of our algorithm and show it performs on par or superior to several state-of-the-art NAS algorithms on two different search spaces.
Revised: We perform an extensive study of our algorithm and show it performs on par or superior to several state-of-the-art NAS algorithms on two different search spaces.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural architecture search have been studied since at least the 1990s (Floreano et al., 2008; Kitano, 1990; Stanley & Miikkulainen, 2002), but the ﬁeld was revitalized when NASNet gained signiﬁcant attention (Zoph & Le, 2017).
Revised: Neural architecture search have been studied since at least the 1990s (Floreano et al., 2008; Kitano, 1990; Stanley & Miikkulainen, 2002), but the ﬁeld was revitalized when NASNet gained signiﬁcant attention (Zoph & Le, 2017).

Original: There has been a ﬂurry of recent work, where the most popular techniques include evolutionary algorithms (Shah et al., 2018), reinforcement learning (Zoph & Le, 2017; Pham et al., 2018;
Revised: There has been a ﬂurry of recent work, where the most popular techniques include evolutionary algorithms (Shah et al., 2018), reinforcement learning (Zoph & Le, 2017; Pham et al., 2018;

Original: Liu et al., 2018a; Tan & Le, 2019), Bayesian optimization (Kandasamy et al., 2018; Jin et al., 2018;
Revised: Liu et al., 2018a; Tan & Le, 2019), Bayesian optimization (Kandasamy et al., 2018; Jin et al., 2018;

Original: Zhou et al., 2019), and gradient descent (Liu et al., 2018b).
Revised: Zhou et al., 2019), and gradient descent (Liu et al., 2018b).

Original: Recent papers have called for fair and reproducible experiments in the future (Li & Talwalkar, 2019; Sciuto et al., 2019;
Revised: Recent papers have called for fair and reproducible experiments in the future (Li & Talwalkar, 2019; Sciuto et al., 2019;

Original: Ying et al., 2019).
Revised: Ying et al., 2019).

Original: In this vein, the NASBench dataset was created, which contains over 400k neural architectures with precomputed training, validation, and test accuracy (Ying et al., 2019).
Revised: In this vein, the NASBench dataset was created, which contains over 400k neural architectures with precomputed training, validation, and test accuracy (Ying et al., 2019).

Original: A recent algorithm, AlphaX, uses a meta neural network to perform NAS (Wang et al., 2018), where the meta neural network is trained to make small changes to the current architecture, given adjacency matrix featurizations of neural architectures.
Revised: A recent algorithm, AlphaX, uses a meta neural network to perform NAS (Wang et al., 2018), where the meta neural network is trained to make small changes to the current architecture, given adjacency matrix featurizations of neural architectures.

Original: See the recent survey on NAS (Elsken et al., 2018) for more information on NAS research.
Revised: See the recent survey on NAS (Elsken et al., 2018) for more information on NAS research.

Original: There are many additional recent NAS papers (Maziarz et al., 2018;
Revised: There are many additional recent NAS papers (Maziarz et al., 2018;

Original: Borsos et al., 2019; Wang et al., 2019).
Revised: Borsos et al., 2019; Wang et al., 2019).

Original: Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive (Rasmussen, 2003; Frazier, 2018), and it has seen great success in hyperparameter optimization for deep learning (Rasmussen, 2003; Golovin et al., 2017; Li et al., 2016).
Revised: Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive (Rasmussen, 2003; Frazier, 2018), and it has seen great success in hyperparameter optimization for deep learning (Rasmussen, 2003; Golovin et al., 2017; Li et al., 2016).

Original: Most work has focused on Euclidean or categorical input domains using a GP model (Rasmussen, 2003; Golovin et al., 2017; Frazier, 2018; Snoek et al., 2012).
Revised: Most work has focused on Euclidean or categorical input domains using a GP model (Rasmussen, 2003; Golovin et al., 2017; Frazier, 2018; Snoek et al., 2012).

Original: There has also been some previous work on using neural network models in Bayesian optimization (Snoek et al., 2015; Springenberg et al., 2016), applied to the task of hyperparameter optimization.
Revised: There has also been some previous work on using neural network models in Bayesian optimization (Snoek et al., 2015; Springenberg et al., 2016; Perrone et al., 2017), applied to the task of hyperparameter optimization.

Original: There are several approaches for predicting the validation accuracy of neural networks using a layer-wise encoding of the neural architectures (Deng et al., 2017; Istrate et al., 2019), or predicting the learning curve of different neural architectures (Klein et al., 2017; Domhan et al., 2015; Baker et al., 2017). 
Revised: There are several approaches for predicting the validation accuracy of neural networks using a layer-wise encoding of the neural architectures (Deng et al., 2017; Istrate et al., 2019), or predicting the learning curve of different neural architectures (Klein et al., 2017; Domhan et al., 2015; Baker et al., 2017). Ensembles of neural networks is a popular approach for uncertainty estimates (Beluch et al., 2018; Lakshminarayanan et al., 2017; Choi et al., 2016).

Original: For more discussion on related work, see Appendix A.
Revised: For more discussion on related work, see Appendix A.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We give background on three key ingredients of NAS algorithms, as well as on Bayesian optimization.
Revised: We give background on three key ingredients of NAS algorithms, as well as on Bayesian optimization.

Original: Search space.
Revised: Search space.

Original: Before deploying a NAS algorithm, we must deﬁne the space of neural networks that the algorithm can search through.
Revised: Before deploying a NAS algorithm, we must deﬁne the space of neural networks that the algorithm can search through.

Original: Perhaps the most common type of search space for NAS is a cell-based search space (Zoph & Le, 2017; Pham et al., 2018;
Revised: Perhaps the most common type of search space for NAS is a cell-based search space (Zoph & Le, 2017; Pham et al., 2018;

Original: Liu et al., 2018b; Li & Talwalkar, 2019;
Revised: Liu et al., 2018b; Li & Talwalkar, 2019;

Original: Sciuto et al., 2019;
Revised: Sciuto et al., 2019;

Original: Ying et al., 2019).
Revised: Ying et al., 2019).

Original: A cell consists of a relatively small section of a neural network, usually 6-12 nodes forming a directed acyclic graph (DAG).
Revised: A cell consists of a relatively small section of a neural network, usually 6-12 nodes forming a directed acyclic graph (DAG).

Original: A neural architecture is then built by repeatedly stacking one or two different cells on top of each other sequentially, possibly separated by specialized layers.
Revised: A neural architecture is then built by repeatedly stacking one or two different cells on top of each other sequentially, possibly separated by specialized layers.

Original: The layout of cells and specialized layers is called a hyper-architecture , and this is ﬁxed, while the NAS algorithm searches for the best cells.
Revised: The layout of cells and specialized layers is called a hyper-architecture , and this is ﬁxed, while the NAS algorithm searches for the best cells.

Original: The search space over cells consists of all possible DAGs of a certain size, where each node can be one of several operations such as 1 ×convolution, 3 × 3 convolution, or 3 × 3 max pooling.
Revised: The search space over cells consists of all possible DAGs of a certain size, where each node can be one of several operations such as 1 ×convolution, 3 × 3 convolution, or 3 × 3 max pooling.

Original: It is also common to set a restriction on the number of total edges or the in-degree of each node (Ying et al., 2019; Liu et al., 2018b).
Revised: It is also common to set a restriction on the number of total edges or the in-degree of each node (Ying et al., 2019; Liu et al., 2018b).

Original: In this work, we focus on NAS over convolutional cell-based search spaces.
Revised: In this work, we focus on NAS over convolutional cell-based search spaces.

Original: Search strategy.
Revised: Search strategy.

Original: The search strategy is the optimization method that the algorithm uses to ﬁnd the optimal or near-optimal neural architecture from the search space.
Revised: The search strategy is the optimization method that the algorithm uses to ﬁnd the optimal or near-optimal neural architecture from the search space.

Original: There are many varied search strategies, such as Bayesian optimization, evolutionary search, reinforcement learning, and gradient descent.
Revised: There are many varied search strategies, such as Bayesian optimization, evolutionary search, reinforcement learning, and gradient descent.

Original: In Section 4, we introduce a novel search strategy based on Bayesian optimization with a neural network model using a path-based encoding.
Revised: In Section 4, we introduce a novel search strategy based on Bayesian optimization with a neural network model using a path-based encoding.

Original: Evaluation method.
Revised: Evaluation method.

Original: Many types of NAS algorithms consist of an iterative framework in which the algorithm chooses a neural network to train, computes its validation error, and uses this result to guide the choice of neural network in the next iteration.
Revised: Many types of NAS algorithms consist of an iterative framework in which the algorithm chooses a neural network to train, computes its validation error, and uses this result to guide the choice of neural network in the next iteration.

Original: The simplest instantiation of this approach is to train each neural network in a ﬁxed way, i.e., the algorithm has black-box access to a function that trains a neural network for x epochs and then returns the validation error (Pham et al., 2018; Liu et al., 2018a; Shah et al., 2018).
Revised: The simplest instantiation of this approach is to train each neural network in a ﬁxed way, i.e., the algorithm has black-box access to a function that trains a neural network for x epochs and then returns the validation error (Pham et al., 2018; Liu et al., 2018a; Shah et al., 2018).

Original: Algorithms with black-box evaluation methods can be compared[Figure] by returning the architecture with the lowest validation error after a certain number of queries to the black-box function.
Revised: Algorithms with black-box evaluation methods can be compared by returning the architecture with the lowest validation error after a certain number of queries to the black-box function.

Original: There are also multi-ﬁdelity methods, for example, when a NAS algorithm chooses the number of training epochs in addition to the architecture.
Revised: There are also multi-ﬁdelity methods, for example, when a NAS algorithm chooses the number of training epochs in addition to the architecture.

Original: In this work, we show our algorithm far outperforms several other NAS algorithms that use black box optimization, and we discuss multi-ﬁdelity approaches in Section 6.
Revised: In this work, we show our algorithm far outperforms several other NAS algorithms that use black box optimization, and we discuss multi-ﬁdelity approaches in Section 6.

Original: Bayesian optimization.
Revised: Bayesian optimization.

Original: In applications of Bayesian optimization for deep learning, the typical goal is to ﬁnd a neural architecture and/or set of hyperparameters that lead to an optimal validation error.
Revised: In applications of Bayesian optimization for deep learning, the typical goal is to ﬁnd a neural architecture and/or set of hyperparameters that lead to an optimal validation error.

Original: Formally, Bayesian optimization seeks to compute a ∗ =
Revised: Formally, Bayesian optimization seeks to compute a ∗ =

Original: arg min a ∈
Revised: arg min a ∈

Original: A f ( a ) , where A is the search space, and f ( a ) denotes the validation error of architecture a after training on a ﬁxed dataset for a ﬁxed number of epochs.
Revised: A f ( a ) , where A is the search space, and f ( a ) denotes the validation error of architecture a after training on a ﬁxed dataset for a ﬁxed number of epochs.

Original: In the standard Bayesian optimization setting, over a sequence of iterations, the results from all previous iterations are used to model the topology of f ( A ) using the posterior distribution of the model (often a Gaussian process).
Revised: In the standard Bayesian optimization setting, over a sequence of iterations, the results from all previous iterations are used to model the topology of f ( A ) using the posterior distribution of the model (often a Gaussian process).

Original: The next architecture is then chosen by optimizing an acquisition function such as expected improvement (EI) (Moˇckus, 1975) or upper conﬁdence bound (UCB) (Srinivas et al., 2009).
Revised: The next architecture is then chosen by optimizing an acquisition function such as expected improvement (EI) (Moˇckus, 1975) or upper conﬁdence bound (UCB) (Srinivas et al., 2009).

Original: These functions balance exploration with exploitation during the iterative search.
Revised: These functions balance exploration with exploitation during the iterative search.

Original: We discuss UCB in the next section, and we discuss other acquisition functions in Appendix B.
Revised: We discuss UCB in the next section, and we discuss other acquisition functions in Appendix B.

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, we discuss all of the components of our NAS algorithm.
Revised: In this section, we discuss all of the components of our NAS algorithm.

Original: Our Bayesian optimization method uses a meta neural network to predict the validation error of neural architectures from the search space.
Revised: Our Bayesian optimization method uses a meta neural network to predict the validation error of neural architectures from the search space.

Original: First we describe the featurization of neural architectures used in our algorithm.
Revised: First we describe the featurization of neural architectures used in our algorithm.

Original: Then we describle how to predict mean and uncertainty estimates using an ensemble of meta neural networks.
Revised: Then we describle how to predict mean and uncertainty estimates using an ensemble of meta neural networks.

Original: Finally, we describe the acquisition function and acquisition optimization strategy we use, and show how these pieces come together to form our ﬁnal NAS algorithm.
Revised: Finally, we describe the acquisition function and acquisition optimization strategy we use, and show how these pieces come together to form our ﬁnal NAS algorithm.

Original: Architecture featurization via path-based encoding.
Revised: Architecture featurization via path-based encoding.

Original: Prior work aiming to encode or featurize neural networks have proposed using a binary encoding of the adjacency matrix and either a categorical or a one-hot encoding for the operations on each node (Wang et al., 2018;
Revised: Prior work aiming to encode or featurize neural networks have proposed using an adjacency matrix-based approach (Wang et al., 2018;

Original: Ying et al., 2019; Deng et al., 2017; Baker et al., 2017).
Revised: Ying et al., 2019; Deng et al., 2017; Baker et al., 2017).

Revised: The adjacency matrix encoding gives an arbitrary ordering to the nodes, and then gives a binary feature for an edge between node i and node j, for all i, j. Then a list of the operations at each node

Revised: [Figure]

Revised: cell cell cell in out 1x 3x 3x3 MP

Revised: MP in out 1x

Original: It is  challenging even for a deep network to learn graph topologies from an adjacency encoding, and we show in the next section that this featurization does not perform well.
Revised: MP in out 1x 3x MP in out 3x 3x MP in out 3x must also be included in the encoding. This is a challenging data structure for a NAS algorithm to interpret, because it relies on an arbitrary indexing of the nodes, and each feature is highly dependent on one another.

Revised: For example, an edge from the input to node 2 is useless if there is no path from node 2 to the output.

Revised: Also, even if there is an edge from node 2 to the output, it matters a lot whether node 2 is a CONV or a POOL operation.

Revised: Ying et al. tested another encoding that is very similar to the adjacency matrix encoding, but the features of each edge are continuous.

Original: We introduce a path-based encoding and show that it substantially increases the performance of our deep predictive uncertainty model.
Revised: We introduce a path-based encoding and show that it substantially increases the performance of our deep predictive uncertainty model.

Original: A path encoding of a cell is created by enumerating all possible paths from the inputnode to the outputnode, in terms of  the operations.
Revised: Each feature of the path encoding corresponds with a directed path from the input to the output of an architecture cell (for example: input → CONV 1x1 → CONV 3x3 → MP → output). To encode an architecture, we simply check which of the possible paths it contains (i.e. which paths are present in the cell), and write this as a binary vector. Therefore, we do not need to arrange the nodes in any particular order. Furthermore, the features are not nearly as dependent on one another as they are in the adjacency matrix encoding.

Original: The total number of paths is (cid:80) ni =

Revised: The total number of paths is (cid:80) ni =

Original: q i where n denotes the number of nodes in the cell, and q denotes the number of operations for each node.
Revised: q i where n denotes the number of nodes in the cell, and q denotes the number of operations for each node.

Original: For example, the search space from NASBench (Ying et al., 2019) has (cid:80) 5 i =

Revised: For example, the search space from NASBench (Ying et al., 2019) has (cid:80) 5 i =

Original: 3 i = 364 possible paths, and the search space from DARTS (Liu et al., 2018b) has 4 · (cid:80) 4 i =0 8 i = 18724 possible paths.
Revised: 3 i = 364 possible paths, and the search space from DARTS (Liu et al., 2018b) has 4 · (cid:80) 4 i =0 8 i = 18724 possible paths.

Original: We create a binary feature for each path.

Original: See Figure 1 (center).
Revised: See Figure 2 (right).

Original: Our featurization performs very well on the most popular search spaces, however it may not scale to large search spaces.
Revised: Our featurization performs very well on the most popular search spaces, however it may not scale to large search spaces.

Original: We discuss this further in Section 6.
Revised: We discuss this further in Section 6.

Original: Acquisition function and optimization.
Revised: Acquisition function and optimization.

Original: Many acquisition functions used in Bayesian optimization can be approximately computed using a mean and uncertainty estimate for each input datapoint.
Revised: Many acquisition functions used in Bayesian optimization can be approximately computed using a mean and uncertainty estimate for each input datapoint.

Original: We train an ensemble of ﬁve meta neural networks in order to predict mean and uncertainty estimates for new neural architectures.
Revised: We train an ensemble of ﬁve meta neural networks in order to predict mean and uncertainty estimates for new neural architectures.

Original: Concretely, we predict the validation error, as well as conﬁdence intervals around the predicted validation error, of neural architectures that we have not yet observed.
Revised: Concretely, we predict the validation error, as well as conﬁdence intervals around the predicted validation error, of neural architectures that we have not yet observed.

Original: Suppose we have an ensemble of M neural networks, { ˆ f m } Mm =1 , where each ˆ f m : A → R .
Revised: Suppose we have an ensemble of M neural networks, { ˆ f m } Mm =1 , where each ˆ f m : A → R .

Original: We compute an estimate of the UCB acquisition function via
Revised: We compute an estimate of the UCB acquisition function via

Original: [Equation]
Revised: [Equation]

Original: where parameters ˆ f = 1 M (cid:80) Mm
Revised: where parameters ˆ f = 1 M (cid:80) Mm

Original: =

Revised: =

Original: ˆ f m ( a ) , and ˆ σ = (cid:113) (cid:80) Mm =
 
Revised: ˆ f m ( a ) , and ˆ σ = (cid:113) (cid:80) Mm =
 ( ˆ f m (

Original: ( ˆ f m ( a ) − ˆ f ) 2 M −.
Revised: [Equation] − ˆ [Equation] M −.

Original: Here we use UCB to denote an estimate of a lower conﬁdence bound (since we are performing minimization) for the predictive uncertainty, and β is a tradeoff parameter to control exploration vs exploitation.
Revised: Here we use UCB to denote an estimate of a lower conﬁdence bound (since we are performing minimization) for the predictive uncertainty, and β is a tradeoff parameter to control exploration vs exploitation.

Original: In each iteration of Bayesian optimization, our goal is to ﬁnd the neural network from the search space which minimizes the acquisition function.
Revised: In each iteration of Bayesian optimization, our goal is to ﬁnd the neural network from the search space which minimizes the acquisition function.

Original: Evaluating the acquisition function for every neural network in the search space is computationally infeasible.
Revised: Evaluating the acquisition function for every neural network in the search space is computationally infeasible.

Original: Instead, we optimize the acquisition function via a mutation procedure, where we randomlyrandomly mutate the best architectures that we have trained so far, many times, and then select the architecture from this set which minimizes the acquisition function.
Revised: Instead, we optimize the acquisition function via a mutation procedure, where we randomly mutate the best architectures that we have trained so far, many times, and then select the architecture from this set which minimizes the acquisition function.

Original: A cell is mutated by either adding an edge, removing an edge, or changing one of the operations with some probability.
Revised: A cell is mutated by either adding an edge, removing an edge, or changing one of the operations with some probability.

Original: BANANAS: Bayesian optimization with neural architectures for NAS.
Revised: BANANAS: Bayesian optimization with neural architectures for NAS.

Original: Now we present our full NAS algorithm.
Revised: Now we present our full NAS algorithm.

Original: At the start, we draw t 0 random architectures from the search space.
Revised: At the start, we draw t 0 random architectures from the search space.

Original: Then we start an iterative process, where in iteration t , we train an ensemble of meta neural networks on architectures a 0 , . .
Revised: Then we start an iterative process, where in iteration t , we train an ensemble of meta neural networks on architectures a 0 , . .

Original: . , a t − 1 .
Revised: . , a t − 1 .

Revised: Each meta neural network in the ensemble has a different random initialization of the weights and a different random ordering of the training set.

Original: We create a candidate set of architectures by mutating the best architectures we have seen so far, and we choose to train the candidate architecture which minimizes the UCB acquisition function.
Revised: We create a candidate set of architectures by mutating the best architectures we have seen so far, and we choose to train the candidate architecture which minimizes the UCB acquisition function.

Original: See Algorithm 1.
Revised: See Algorithm 1.

Original: Algorithm 
Revised: Algorithm 1 BANANAS

Original: Input: Search space A , dataset D , parameters t 0 , T, M, c, k , acquisition function φ
Revised: Input: Search space A , dataset D , parameters t 0 , T, M, c, k , acquisition function φ

Original: Draw t 0 architectures a 0 , . .
Revised: Draw t 0 architectures a 0 , . .

Original: . , a t 0 uniformly at random from A and train them on D
Revised: . , a t 0 uniformly at random from A and train them on D

Original: Denote f ( a ) as the validation error of a after training
Revised: Denote f ( a ) as the validation error of a after training

Original: For t from t 0 to T ,
Revised: For t from t 0 to T ,

Original: Train an ensemble of meta neural networks on { ( a 0 , f( a 0 )) , . .
Revised: Train an ensemble of meta neural networks on { ( a 0 , f

Original: , ( a t , f( a t )) } 2.
Revised: , ( a t , f a t )) } 2.

Original: Generate a set of c candidate neural architectures from A by randomly mutating the k architectures a from { a 0 , . .
Revised: Generate a set of c candidate neural architectures from A by randomly mutating the k architectures a from { a 0 , . .

Original: . , a t } with the lowest value of f ( a ) 3.
Revised: . , a t } with the lowest value of f ( a ) 3.

Original: For each candidate architecture a , evaluate the acquisition functionφ ( a ) 4. Denote a t +1 as the candidate architecture with minimum φ ( a ) , and evaluate f ( a t +1 ) Output: a ∗ = argmin t =0 ,...,T f ( a t )
Revised: the acquisition function

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we discuss our experimental setup and results.
Revised: In this section, we discuss our experimental setup and results.

Original: We give experimental results on the performance of the meta neural network itself, as well as the full NAS algorithm compared to several
Revised: We give experimental results on the performance of the meta neural network itself, as well as the full NAS algorithm compared to several

Original: NAS algorithms.
Revised: NAS algorithms.

Original: We start by explaining the two search spaces/datasets that we used.
Revised: We start by explaining the two search spaces/datasets that we used.

Original: Search space from NASBench.
Revised: Search space from NASBench.

Original: The NASBench dataset is a tabular dataset designed to facilitate
Revised: The NASBench dataset is a tabular dataset designed to facilitate

Original: NAS research and fair comparisons between NAS algorithms (Ying et al., 2019).
Revised: NAS research and fair comparisons between NAS algorithms (Ying et al., 2019).

Original: It consists of over 423,000 unique neural architectures from a cell-based search space, and each architecture comes with precomputed training, validation, and test accuracies for 108 epochs on CIFAR-10.
Revised: It consists of over 423,000 unique neural architectures from a cell-based search space, and each architecture comes with precomputed training, validation, and test accuracies for 108 epochs on CIFAR-10.

Original: Most NAS algorithms take several GPU days to complete one trial, however, the NASBench dataset makes it feasible to run hundreds of trials of several NAS algorithms.
Revised: Most NAS algorithms take several GPU days to complete one trial, however, the NASBench dataset makes it feasible to run hundreds of trials of several NAS algorithms.

Original: We give a description of the NASBench search space.
Revised: We give a description of the NASBench search space.

Original: For the full details, see (Ying et al., 2019).
Revised: For the full details, see (Ying et al., 2019).

Original: The search space consists of a cell with 7 nodes.
Revised: The search space consists of a cell with 7 nodes.

Original: The ﬁrst node is the input, and the last node is the output.
Revised: The ﬁrst node is the input, and the last node is the output.

Original: The remaining ﬁve nodes can be either 1 × 1 convolution, 3 × 3 convolution, or 3 × 3 max pooling.
Revised: The remaining ﬁve nodes can be either 1 × 1 convolution, 3 × 3 convolution, or 3 × 3 max pooling.

Original: The cell can take on any DAG structure from the input to the output with at most 9 edges.
Revised: The cell can take on any DAG structure from the input to the output with at most 9 edges.

Original: The NASBench search space was chosen to contain ResNet-like and Inception-like cells (He et al., 
Revised: The NASBench search space was chosen to contain ResNet-like and Inception-like cells (He et al., 2016; Szegedy et al., 2016).

Revised: The hyper-architecture consists of nine cells stacked sequentially, with each set of three cells separated by downsampling layers.

Revised: The ﬁrst layer before the ﬁrst cell is a convolutional layer, and the hyper-architecture ends with a global average pooling layer and dense layer.

Revised: See Figure 2 (left).

Revised: Search space from DARTS.

Revised: One of the most popular convolutional cell-based search spaces is the one from DARTS (Liu et al., 2018b), used for CIFAR-10.

Original: 8 10 12 14 16 True Error %
Revised: The search space consists of two cells with 6 nodes each: a convolution cell and a reduction cell, and the hyper-architecture stacks the convolutional and reduction cells. For each cell, the ﬁrst two nodes are the inputs from the previous two cells in the hyper-architecture. The next four nodes each contain exactly two edges as input, such that the cell forms a connected DAG. Each edge can take one of seven operations: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, 3 × 3 average pooling, identity, and zero (this is in contrast to NASBench, where the nodes take on operations).

Revised: 5. NASB ENCH E XPERIMENTS

Revised: First we evaluate the performance of the meta neural network on the NASBench dataset.

Revised: The meta neural network consists of a sequential fully-connected neural network.

Revised: The number of layers is set to 10, and each layer has width 20.

Revised: We use the Adam optimizer with a learning rate of 0 . 1 , and use a mean absolute error loss function.

Revised: We train the meta neural network on 200 architectures, and test

Revised: 8 10 12 14 16 True Error %

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: E rr o r % ( w i t
Revised: E rr o r % ( w i t

Original: h U n c e r t a i n t y )
Revised: h U n c e r t a i n t y )

Original: [Figure] Encoding ( [Figure] Set) 
Revised: Matrix Encoding (Train Set) Prediction mean

Original: [Figure]

Original: Prediction uncertainty 8 10 12 14 16 True Error %
Revised: Prediction uncertainty 8 10 12 14 16 True Error %

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: E rr o r % ( w i t
Revised: E rr o r % ( w i t

Original: h U n c e r t a i n t y )
Revised: h U n c e r t a i n t y )

Original: Matrix Encoding (Test Set) 8 10 12 14 

Revised: Matrix Encoding (Test Set) 8 10 12 14

Original: True Error %
Revised: True Error %

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: E rr o r % ( w i t
Revised: E rr o r % ( w i t

Original: h U n c e r t a i n t y )
Revised: h U n c e r t a i n t y )

Original: Path Encoding (Train Set) 

Revised: Path Encoding (Train Set)

Original: True Error %
Revised: True Error %

Original: P r e d i c t e d
Revised: P r e d i c t e d

Original: E rr o r % ( w i t
Revised: E rr o r % ( w i t

Original: h U n c e r t a i n t y )
Revised: h U n c e r t a i n t y )

Original: Path Encoding (Test Set)
Revised: Path Encoding (Test Set)

Original: We train this model on 200 architectures, and test the adjacency matrix encoding and path encoding on a held-out set of architectures as well as on a subset of the training set. [Figure]
Revised: We train this model on 200 architectures, and test the adjacency matrix encoding and path encoding on a held-out set of architectures as well as on a subset of the training set. Training set size

Original: Train
Revised: Train

Original: Test
Revised: Test

Original: Train
Revised: Train

Original: Test
Revised: Test

Original: Train
Revised: Train

Revised: Test

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 2016; Szegedy et al., 2016). The hyper-architecture consists of nine cells stacked sequentially, with each set of three cells separated by downsampling layers. The ﬁrst layer before the ﬁrst cell is a convolutional layer, and the hyper-architecture ends with a global average pooling layer and dense layer.
Revised: [Table]

Original: See Figure 1 (left).
Revised: [Table]

Original: Search space from DARTS. One of the most popular convolutional cell-based search spaces is the one from DARTS (Liu et al., 2018b), used for CIFAR-10. The search space consists of two cells with 6 nodes each: a convolution cell and a reduction cell, and the hyper-architecture stacks the convolutional and reduction cells.
Revised: encoding

Original: For each cell, the ﬁrst two nodes are the inputs from the previous two cells in the hyper-architecture. The next four nodes each contain exactly two edges as input, such that the cell forms a connected DAG. Each edge can take one of seven operations: 3 × 3 and 5 × 5 separable convolutions, 3 × 3 and 5 × 5 dilated separable convolutions, 3 × 3 max pooling, 3 × 3 average pooling, identity, and zero (this is in contrast to NASBench, where the nodes take on operations). 5. NASB ENCH E XPERIMENTS
Revised: [Table]

Original: First we evaluate the performance of the meta neural network on the NASBench dataset. The meta neural network consists of a sequential fully-connected neural network. The number of layers is set to 10, and each layer has width 20. We use the Adam optimizer with a learning rate of 0 . 1 , and use a mean absolute error loss function.
Revised: [Table]

Original: We train the meta neural network on 200 architectures, and test the standard adjacency matrix encoding as well as the path encoding discussed in Section 4.
Revised: [Table] [Table] [Table] the standard adjacency matrix encoding as well as the path encoding discussed in Section 4.

Original: See Figure 2.
Revised: See Figure 3.

Original: The path-based encoding outperforms the adjacency matrix encoding by a factor of 3 with respect to mean absolute error.
Revised: The path-based encoding outperforms the adjacency matrix encoding by a factor of 3 with respect to mean absolute error.

Original: In Appendix B, we plot the meta neural network performance with both featurizations for different training set sizes, and we report the training and test errors in Table 5.1.
Revised: In Appendix B, we plot the meta neural network performance with both featurizations for different training set sizes, and we report the training and test errors in Table 1.

Original: Next, we compare BANANAS to ﬁve other algorithms.
Revised: Next, we compare BANANAS to ﬁve other algorithms.

Original: We give a brief description of each one.
Revised: We give a brief description of each one.

Original: Random search.
Revised: Random search.

Original: The simplest baseline, random search, draws n architectures at random and outputs the architecture with the lowest validation error.
Revised: The simplest baseline, random search, draws n architectures at random and outputs the architecture with the lowest validation error.

Original: Despite its simplicity, multiple papers have concluded that random search is a competitive baseline for NAS algorithms (Li & Talwalkar, 2019; Sciuto et al., 2019).
Revised: Despite its simplicity, multiple papers have concluded that random search is a competitive baseline for NAS algorithms (Li & Talwalkar, 2019; Sciuto et al., 2019).

Original: Regularized evolution.
Revised: Regularized evolution.

Original: This algorithm consists of drawing an initial set of cells at random, and then iteratively mutating the best architecture out of a sample of all architectures evaluated so far.
Revised: This algorithm consists of drawing an initial set of cells at random, and then iteratively mutating the best architecture out of a sample of all architectures evaluated so far.

Original: The architectures with the worst validation error in each iteration are removed from the population.We [Figure] t
Revised: The architectures with the worst validation error in each iteration are removed from the population.

Original: [Figure]

Original: s t e rr

Original: [Figure]

Original: o f b

Original: [Figure] s t n e u r a l n [Figure] used the same hyperparameters as in the NASBench paper, but changing the population size fromto 30 to account for fewer total queries.
Revised: We used the same hyperparameters as in the NASBench paper, but changing the population size fromto 30 to account for fewer total queries.

Original: Reinforcement Learning.
Revised: Reinforcement Learning.

Original: We used the NASBench implementation of reinforcement learning for
Revised: We used the NASBench implementation of reinforcement learning for

Original: NAS (Ying et al., 2019) based on the REINFORCE algorithm (Williams, 1992).
Revised: NAS (Ying et al., 2019) based on the REINFORCE algorithm (Williams, 1992).

Original: We used this algorithm because prior work has shown that a 1-layer LSTM controller trained with PPO is not effective on the NASBench dataset (Ying et al., 2019).
Revised: We used this algorithm because prior work has shown that a 1-layer LSTM controller trained with PPO is not effective on the NASBench dataset (Ying et al., 2019).

Original: AlphaX. AlphaX casts NAS as a reinforcement learning problem, using a neural network to guide the search (Wang et al., 2018).
Revised: AlphaX. AlphaX casts NAS as a reinforcement learning problem, using a neural network to guide the search (Wang et al., 2018).

Original: Each iteration, a neural network is trained to select the best action, such as making a small change to, or growing, the current architecture. 
Revised: Each iteration, a neural network is trained to select the best action, such as making a small change to, or growing, the current architecture. TPE.

Revised: Tree-structured Parzen estimator (TPE) is a hyperparameter optimization algorithm based on adaptive Parzen windows.

Revised: We used the nasbench implementation (Ying et al., 2019).

Original: Bayesian optimization with a GP model.
Revised: Bayesian optimization with a GP model.

Original: We set up Bayesian optimization with a Gaussian process model and UCB acquisition.
Revised: We set up Bayesian optimization with a Gaussian process model and UCB acquisition.

Original: In the Gaussian process, we set the distance function between two neural networks as the sum of the Hamming distances between the adjacency matrices and the list of operations.
Revised: In the Gaussian process, we set the distance function between two neural networks as the sum of the Hamming distances between the adjacency matrices and the list of operations.

Original: Note we also used a path-based distance function in our ablation study.
Revised: Note we also used a path-based distance function in our ablation study.

Original: Experimental setup and results.
Revised: Experimental setup and results.

Original: Each NAS algorithm is given a budget of 150 queries.
Revised: Each NAS algorithm is given a budget of 150 queries.

Original: That is, each algorithm can train and output the validation error of at most 150 architectures.
Revised: That is, each algorithm can train and output the validation error of at most 150 architectures.

Original: Everyiterations, we return the architecture with the best validation error found by each algorithm so far. 
Revised: Everyiterations, we return the architecture with the best validation error found by each algorithm so far. [Figure] s t n [Figure]

Revised: u

Original:  After all NAS algorithms have completed, we output the test error for each returned architecture.
Revised: [Figure] a [Figure] After all NAS algorithms have completed, we output the test error for each returned architecture.

Original: We ran 200 trials for each algorithm.
Revised: We ran 200 trials for each algorithm.

Original: For BANANAS, we use an ensemble of size 5 and UCB with β = 0 .
Revised: For BANANAS, we use an ensemble of size 5 and UCB with β = 0 .

Original: 5 as in Equation 1.
Revised: 5 as in Equation 1.

Original: In each iteration, we chose the top ten candidate architectures to query, to improve runtime and so that the algorithm can later be parallelized.
Revised: In each iteration, we chose the top ten candidate architectures to query, to improve runtime and so that the algorithm can later be parallelized.

Original: Although this strategy may not perform well for continuous search spaces, we found the NASBench search space was discrete enough that this method performs well.
Revised: Although this strategy may not perform well for continuous search spaces, we found the NASBench search space was discrete enough that this method performs well.

Original: We compared against other black-box optimization algorithms, so we report test error with respect to the number of queries.
Revised: We compared against other black-box optimization algorithms, so we report test error with respect to the number of queries.

Original: We note that a plot with respect to wall-clock time would look nearly identical, since all NAS algorithms ﬁnish the set of 150 queries in roughly the same amount of time.
Revised: We note that a plot with respect to wall-clock time would look nearly identical, since all NAS algorithms ﬁnish the set of 150 queries in roughly the same amount of time.

Original: The fastest, random search, takes 46.5 TPU hours, while BANANAS, the slowest, takes 47.0 TPU hours (the average time to train a NASBench architecture is 18.6 minutes on a TPU, reported by (Ying et al., 2019)).
Revised: The fastest, random search, takes 46.5 TPU hours, while BANANAS, the slowest, takes 47.0 TPU hours (the average time to train a NASBench architecture is 18.6 minutes on a TPU, reported by (Ying et al., 2019)).

Original: See Figure 3 (left).
Revised: See Figure 4 (left).

Original: BANANAS signiﬁcantly outperforms all other baselines, and to the best of our knowledge, BANANAS achieves state-of-the-art error on NASBench in the 100-150 queries setting.
Revised: BANANAS signiﬁcantly outperforms all other baselines, and to the best of our knowledge, BANANAS achieves state-of-the-art error on NASBench in the 100-150 queries setting.

Original: The standard deviation among 200 trials for all NAS algorithms were between 0.16 and 0.22.
Revised: The standard deviation among 200 trials for all NAS algorithms were between 0.16 and 0.22.

Original: BANANAS had the lowest standard deviation, and regularized evolution had the highest standard deviation.
Revised: BANANAS had the lowest standard deviation, and regularized evolution had the highest standard deviation.

Original: Ablation study.
Revised: Ablation study.

Original: Our NAS algorithm has several moving parts, including the meta neural network model, the path-based feature encoding of neural architectures, the acquisition function, and the acquisition optimization strategy.
Revised: Our NAS algorithm has several moving parts, including the meta neural network model, the path-based feature encoding of neural architectures, the acquisition function, and the acquisition optimization strategy.

Original: We run a thorough ablation study by removing each piece of the algorithm separately.
Revised: We run a thorough ablation study by removing each piece of the algorithm separately.

Original: In particular, we compare against the following algorithms: (1) BANANAS where the featurization of each neural architecture is not the path-based encoding, but instead the common adjacency matrix and one-hot list of operations encoding;  instead of a neural network model, where the distance function in the GP is computed as the Hamming distance between the path-encoding vectors; (3) BANANAS with a GP model, where the distance function is computed as the sum of the Hamming distances between the adjacency matrices and the list of operations; (4) BANANAS where the acquisition funtion is optimized by drawing 1000 random architectures, instead of a mutation algorithm.
Revised: In particular, we compare against the following algorithms: (1) BANANAS where the featurization of each neural architecture is not the path-based encoding, but instead the common adjacency matrix and one-hot list of operations encoding; (2) BANANAS where the featurization of each neural architecture is the continuous variant of the adjacency matrix encoding from the nasbench paper (Ying et al., 2019); (3) BANANAS with a GP model instead of a neural network model, where the distance function in the GP is computed as the Hamming distance between the path-encoding vectors; (4) BANANAS with a GP model, where the distance function is computed as the sum of the Hamming distances between the adjacency matrices and the list of operations; (5) BANANAS where the acquisition funtion is optimized by drawing 1000 random architectures, instead of a mutation algorithm.

Original: We found that BANANAS signiﬁcantly outperformed all of these variants.
Revised: We found that BANANAS signiﬁcantly outperformed all of these variants.

Original: The path-encoding had the greatest effect on performance, while the acquisition optimization strategy had the smallest effect on performance.
Revised: The path-encoding had the greatest effect on performance, while the acquisition optimization strategy had the smallest effect on performance.

Original: See Figure 3 (right).
Revised: See Figure 4 (right).

Original: In Appendix B, we present several more experiments.
Revised: In Appendix B, we present several more experiments.

Original: We give each NAS algorithm a budget of 500 queries instead of 150, and we show that BANANAS still outperforms all other algorithms.
Revised: We give each NAS algorithm a budget of 500 queries instead of 150, and we show that BANANAS still outperforms all other algorithms.

Original: We discuss a different method to use the NASBench dataset (querying random vs mean validation error) and we present results in this setting, showing that our algorithm still performs the best.
Revised: We discuss a different method to use the NASBench dataset (querying random vs mean validation error) and we present results in this setting, showing that our algorithm still performs the best.

Original: We also test three other common acquisition functions with BANANAS. 
Revised: We also test three other common acquisition functions with BANANAS. [Table]

Original: 5. DARTS SEARCH SPACE EXPERIMENTS .
Revised: (Pham et al., 2018) [Table] (Liu et al., 2018b) [Table]

Revised: [Table]

Revised: ., 2018b

Revised: [Table]

Revised: ASHA (Li & Talwalkar, 2019)

Revised: [Table]

Revised: Random search WS (Li & Talwalkar, 2019)

Revised: [Table]

Revised: 5. DARTS SEARCH SPACE EXPERIMENTS .

Original: We test the BANANAS algorithm on the search space from DARTS.
Revised: We test the BANANAS algorithm on the search space from DARTS.

Original: We give BANANAS a budget of 100 queries, where each query involves training a neural network for 50 epochs and reporting its validation error.
Revised: We give BANANAS a budget of 100 queries, where each query involves training a neural network for 50 epochs and reporting its validation error.

Original: We parallelize the algorithm by choosing 10 neural networks to train in each iteration of Bayesian optimization.
Revised: We parallelize the algorithm by choosing 10 neural networks to train in each iteration of Bayesian optimization.

Original: The total computation is 15.3 GPU days.
Revised: The total computation is 11.8 GPU days.

Original: To ensure a fair comparison by controlling all hyperparameter settings and hardware (Lindauer &
Revised: To ensure a fair comparison by controlling all hyperparameter settings and hardware (Lindauer &

Original: Hutter, 2019), we chose to re-train the architectures from papers which used the DARTS search space and reported the ﬁnal architecture (instead of comparing to published CIFAR-10 accuracies from prior work).
Revised: Hutter, 2019), we re-trained the architectures from papers which used the DARTS search space and reported the ﬁnal architecture. We report the mean test error over ﬁve random seeds of the best architectures found by BANANAS, DARTS, and ASHA (Li & Talwalkar, 2019).

Original: We trained each architecture using the default hyperparameters from  (Li & Talwalkar, 2019).
Revised: We trained each architecture using the default hyperparameters from Li and Talwalkar (Li & Talwalkar, 2019) forepochs.

Original: Using these hyperparameters, the ﬁnal CIFAR-10 error for all NAS methods fell roughly one percent short of the published values.
Revised: The BANANAS architecture achieved an average of 2.64% accuracy, which is state-of-the-art for this search-space and ﬁnal training parameter settings. See Table 2. We also compare to other NAS algorithms using the DARTS search space, however, the comparison is not perfect due to differences in the pytorch version of the ﬁnal training code (which appears to change the ﬁnal percent error by ∼ 0.1%). We cannot fairly compare our method to recent NAS algorithms which use a larger search space than DARTS, or which train the ﬁnal architecture for signiﬁcantly more than 600 epochs (Laube & Zell, 2019;

Original: We report the mean test errorover three random seeds, of our method, the best architecture from DARTS, and the best architecture from ASHA (Li & Talwalkar, 2019). 
Revised: Liang et al., 2019; Zhou et al., 2019). Our algorithm signiﬁcantly beats ASHA, a multi-ﬁdelity zeroth order NAS algorithm, and is on par with DARTS, a ﬁrst-order method.

Original: To chek the variance, we ran BANANAS four times total, and the difference in test error of the trained models were within 0.19. [Table]
Revised: We emphasize that since BANANAS is a zeroth order method, it allows for easy parallelism, integration with optimizing other hyperparameters, easy implementation, and optimization with respect to other non-differentiable objectives.

Original: In the appendix, we run through the NAS research checklist (Lindauer & Hutter, 2019).
Revised: In the appendix, we run through the NAS research checklist (Lindauer & Hutter, 2019).

Original:  C ONCLUSION AND F UTURE W ORK
Revised:  C ONCLUSION AND F UTURE W ORK

Original: In this work, we propose a novel method for neural architecture search.
Revised: In this work, we propose a novel method for neural architecture search.

Original: Our method uses Bayesian optimization with a meta neural network model, and we encode the neural architectures using a path-based encoding scheme.
Revised: Our method uses Bayesian optimization with a meta neural network model, and we encode the neural architectures using a path-based encoding scheme.

Original: This allows the meta neural network to accurately predict the validation error of new neural architectures.
Revised: This allows the meta neural network to accurately predict the validation error of new neural architectures.

Original: Our NAS algorithm signiﬁcantly outperforms all other baselines on the NASBench dataset and competitive results on the search space from DARTS.
Revised: Our NAS algorithm signiﬁcantly outperforms all other baselines on the NASBench dataset and outperforms state-of-the-art algorithms on the search space from DARTS.

Original: One of the most interesting follow-up ideas is to develop a multi-ﬁdelity version of BANANAS.
Revised: One of the most interesting follow-up ideas is to develop a multi-ﬁdelity version of BANANAS.

Original: For example, incorporating a successive-halving approach to BANANAS could result in a signiﬁcant decrease in the runtime without substantially sacriﬁcing accuracy.
Revised: For example, incorporating a successive-halving approach to BANANAS could result in a signiﬁcant decrease in the runtime without substantially sacriﬁcing accuracy.

Original: It would also be interesting to run our algorithm on other search spaces and datasets, such as Penn TreeBank and ImageNet.
Revised: It would also be interesting to run our algorithm on other search spaces and datasets, such as Penn TreeBank and ImageNet.

Original: Finally, while we demonstrated that our path-based featurization performs very well for the most popular cell-based search spaces, our featurization would have trouble scaling to large cell-based search spaces, as its dimensionality is exponential in the number of nodes per cell.
Revised: Finally, while we demonstrated that our path-based featurization performs very well for the most popular cell-based search spaces, our featurization would have trouble scaling to large cell-based search spaces, as its dimensionality is exponential in the number of nodes per cell.

Original: It would be interesting to modify our approach so that it scales better to larger search spaces. 
Revised: It would be interesting to modify our approach so that it scales better to larger search spaces. [Figure]

Original: A R ELATED W ORK C ONTINUED
Revised: A R ELATED W ORK C ONTINUED

Original: Neural architecture search.
Revised: Neural architecture search.

Original: Evolutionary algorithms for neural architecture search have been studied since at least the 1990s (Floreano et al., 2008; Kitano, 1990; Stanley & Miikkulainen, 2002).
Revised: Evolutionary algorithms for neural architecture search have been studied since at least the 1990s (Floreano et al., 2008; Kitano, 1990; Stanley & Miikkulainen, 2002).

Original: This research area was revitalized when NASNet gained signiﬁcant attention (Zoph & Le, 2017).
Revised: This research area was revitalized when NASNet gained signiﬁcant attention (Zoph & Le, 2017).

Original: NASNet is a reinforcement learning algorithm for NAS which achieves state-of-the-art results on
Revised: NASNet is a reinforcement learning algorithm for NAS which achieves state-of-the-art results on

Original: CIFAR-10 and PTB; however, the algorithm requires 3000 GPU days to train.
Revised: CIFAR-10 and PTB; however, the algorithm requires 3000 GPU days to train.

Original: NASNet spurred a number of follow-up work, eventually pushing the required computation for NAS on CIFAR-10 and
Revised: NASNet spurred a number of follow-up work, eventually pushing the required computation for NAS on CIFAR-10 and

Original: PTB to under 10 GPU days (Pham et al., 2018;
Revised: PTB to under 10 GPU days (Pham et al., 2018;

Original: Liu et al., 2018a;b; Ying et al., 2019; Kandasamy et al., 2018; Jin et al., 2018; Shah et al., 2018).
Revised: Liu et al., 2018a;b; Ying et al., 2019; Kandasamy et al., 2018; Jin et al., 2018; Shah et al., 2018).

Original: Some of the most popular techniques for NAS include evolutionary algorithms (Shah et al., 2018), reinforcement learning (Zoph & Le, 2017; Pham et al., 2018;
Revised: Some of the most popular techniques for NAS include evolutionary algorithms (Shah et al., 2018), reinforcement learning (Zoph & Le, 2017; Pham et al., 2018;

Original: Liu et al., 2018a; Tan & Le, 2019), Bayesian optimization (Kandasamy et al., 2018; Jin et al., 2018), and gradient descent (Liu et al., 2018b).
Revised: Liu et al., 2018a; Tan & Le, 2019), Bayesian optimization (Kandasamy et al., 2018; Jin et al., 2018), and gradient descent (Liu et al., 2018b).

Original: Recent papers have called for fair and reproducible experiments in the future (Li & Talwalkar, 2019;
Revised: Recent papers have called for fair and reproducible experiments in the future (Li & Talwalkar, 2019;

Original: Sciuto et al., 2019;
Revised: Sciuto et al., 2019;

Original: Ying et al., 2019).
Revised: Ying et al., 2019).

Original: In this vein, the NASBench dataset was created, which contains over 400k neural architectures with precomputed training, validation, and test accuracy (Ying et al., 2019).
Revised: In this vein, the NASBench dataset was created, which contains over 400k neural architectures with precomputed training, validation, and test accuracy (Ying et al., 2019).

Original: A recent algorithm, AlphaX, uses a meta neural network to perform NAS (Wang et al., 2018).
Revised: A recent algorithm, AlphaX, uses a meta neural network to perform NAS (Wang et al., 2018).

Original: This is a similar high-level idea to our work but with many key differences.
Revised: This is a similar high-level idea to our work but with many key differences.

Original: Their search is progressive, and each iteration makes a small change to the current neural network, rather than choosing a completely new neural network.
Revised: Their search is progressive, and each iteration makes a small change to the current neural network, rather than choosing a completely new neural network.

Original: Therefore, their meta neural network is trained to optimize the selection of new actions as well as to predict the performance of the new progressive network.
Revised: Therefore, their meta neural network is trained to optimize the selection of new actions as well as to predict the performance of the new progressive network.

Original: Furthermore, they use an adjacency matrix featurization for the inputs to the meta neural network, which we ﬁnd to perform worse than our path-based encoding.
Revised: Furthermore, they use an adjacency matrix featurization for the inputs to the meta neural network, which we ﬁnd to perform worse than our path-based encoding.

Original: Bayesian optimization.
Revised: Bayesian optimization.

Original: Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive (Rasmussen, 2003; Frazier, 2018), and it has seen great success in hyperparameter optimization for deep learning (Rasmussen, 2003; Golovin et al., 2017; Li et al., 2016).
Revised: Bayesian optimization is a leading technique for zeroth order optimization when function queries are expensive (Rasmussen, 2003; Frazier, 2018), and it has seen great success in hyperparameter optimization for deep learning (Rasmussen, 2003; Golovin et al., 2017; Li et al., 2016).

Original: The majority of Bayesian optimization literature has focused on Euclidean or categorical input domains, and has used a GP model (Rasmussen, 2003;
Revised: The majority of Bayesian optimization literature has focused on Euclidean or categorical input domains, and has used a GP model (Rasmussen, 2003;

Original: Golovin et al., 2017; Frazier, 2018; Snoek et al., 2012).
Revised: Golovin et al., 2017; Frazier, 2018; Snoek et al., 2012).

Original: There has also been some previous work on using neural network models in Bayesian optimization (Snoek et al., 2015; Springenberg et al., 2016), applied to the task of hyperparameter optimization.
Revised: There has also been some previous work on using neural network models in Bayesian optimization (Snoek et al., 2015; Springenberg et al., 2016), applied to the task of hyperparameter optimization.

Original: Recently, there have been a couple works that use Bayesian optimization with GPs for
Revised: Recently, there have been a couple works that use Bayesian optimization with GPs for

Original: NAS (Kandasamy et al., 2018; Jin et al., 2018).
Revised: NAS (Kandasamy et al., 2018; Jin et al., 2018).

Original: Predicting neural network accuracy.
Revised: Predicting neural network accuracy.

Original: There are several approaches for predicting the validation accuracy of neural networks, such as a layer-wise encoding of neural networks with an LSTM algorithm (Deng et al., 2017), and a layer-wise encoding and dataset features to predict the accuracy for neural network + dataset pairs (Istrate et al., 2019).
Revised: There are several approaches for predicting the validation accuracy of neural networks, such as a layer-wise encoding of neural networks with an LSTM algorithm (Deng et al., 2017), and a layer-wise encoding and dataset features to predict the accuracy for neural network + dataset pairs (Istrate et al., 2019).

Original: There is also work in predicting the learning curve of neural networks for hyperparameter optimization (Klein et al., 2017; Domhan et al., 2015) or NAS (Baker et al., 2017) using Bayesian techniques. 
Revised: There is also work in predicting the learning curve of neural networks for hyperparameter optimization (Klein et al., 2017; Domhan et al., 2015) or NAS (Baker et al., 2017) using Bayesian techniques. Hyperparameter optimization using Bayesian optimization with neural networks. Snoek et al. showed that Bayesian optimization can scale by using a neural network to set up a kernel and a posterior distribution, instead of a Gaussian process (Snoek et al., 2012). Springenberg et al.

Original:  B A DDITIONAL E XPERIMENTS
Revised: give similar results using a stochastic gradient Hamiltonian Monte Carlo approach (Springenberg et al., 2016). Perrone et al. show how to apply warm start to Bayesian optimization with neural networks (Perrone et al., 2017). B A DDITIONAL E XPERIMENTS

Original: In this section, we present more experiments for the meta neural network and for BANANAS.
Revised: In this section, we present more experiments for the meta neural network and for BANANAS.

Original: We also discuss our experimental setup in more detail.
Revised: We also discuss our experimental setup in more detail [Figure]

Original: B. M ETA N EURAL N ETWORK E XPERIMENTS
Revised: B. M ETA N EURAL N ETWORK E XPERIMENTS

Original: We plot the meta neural network performance with both featurizations for training sets of size 20, 100, and 500.
Revised: We plot the meta neural network performance with both featurizations for training sets of size 20, 100, and 500.

Original: We use the same experimental setup as described in Section 5.
Revised: We use the same experimental setup as described in Section 5.

Original: See Figure 4.
Revised: See Figure 5.

Original: We note that a training set of size 20 neural networks is realistic at the start of a NAS algorithm, and sizeis realistic near the middle or end of a NAS algorithm.[Figure] [Figure] r
Revised: We note that a training set of size 20 neural networks is realistic at the start of a NAS algorithm, and sizeis realistic near the middle or end of a NAS algorithm.

Original: [Figure]

Original: s t n

Original: [Figure]

Original: u r

Original: [Figure]

Original: l n e t

Original: [Figure]

Original: B. NASB ENCH E XPERIMENTS
Revised: B. NASB ENCH E XPERIMENTS

Original: We run three more experiments on NASBench: running the algorithms with a budget of 500 queries rather than 150, testing out different acquisition functions with BANANAS, and using a slightly different validation error metric.
Revised: We run three more experiments on NASBench: running the algorithms with a budget of 500 queries rather than 150, testing out different acquisition functions with BANANAS, and using a slightly different validation error metric.

Original: All algorithms were run with 200 trials unless otherwise noted.
Revised: All algorithms were run with 200 trials unless otherwise noted.

Original: NAS algorithms running for 500 queries.
Revised: NAS algorithms running for 500 queries.

Original: In our main set of experiments, we gave each NAS algorithm a budget of 150 queries.
Revised: In our main set of experiments, we gave each NAS algorithm a budget of 150 queries.

Original: That is, each NAS algorithm can only choose 150 architectures to train and evaluate.
Revised: That is, each NAS algorithm can only choose 150 architectures to train and evaluate.

Original: Now we give each NAS algorithm a budget of 500 queries.
Revised: Now we give each NAS algorithm a budget of 500 queries.

Original: See Figure 5 (left).
Revised: See Figure 6 (left).

Original: We largely see the same trends, although the gap between BANANAS and regularized evolution becomes smaller after query 400.
Revised: We largely see the same trends, although the gap between BANANAS and regularized evolution becomes smaller after query 400.

Original: Acquisition functions.
Revised: Acquisition functions.

Original: We tested BANANAS with three other common acquisition function in addition to UCB: expected improvement (EI) (Mo ˇ ckus, 1975), probability of improvement (PI) (Kush- ner, 1964), and Thompson sampling (TS) (Thompson, 1933).
Revised: We tested BANANAS with three other common acquisition function in addition to UCB: expected improvement (EI) (Mo ˇ ckus, 1975), probability of improvement (PI) (Kushner, 1964), and Thompson sampling (TS) (Thompson, 1933).

Original: First we give the formal deﬁnitions of each acquisition function.
Revised: First we give the formal deﬁnitions of each acquisition function.

Original: Suppose we have trained a collection of M predictive models, { f m } Mm =1 , where f m : A → R .
Revised: Suppose we have trained a collection of M predictive models, { f m } Mm =1 , where f m : A → R .

Original: Following previous work (Neiswanger et al., 2019), we use the following acquisition function estimates for an input architecture a ∈ A : 
Revised: Following previous work (Neiswanger et al., 2019), we use the following acquisition function estimates for an input architecture a ∈ A : [Figure] [Equation]

Original: [Equation]
Revised: [Equation]

Original: In these acquisition function deﬁnitions, 1 ( x ) = 1 if x is true and 0 [-otherwise, and we are making a normal approximation for our model’s posterior predictive density, where we estimate parameters ˆ f = 1 M (cid:80) Mm =
 ˆ f m ( x ) , and ˆ σ = (cid:113) (cid:80)-]
Revised: In these acquisition function deﬁnitions, 1 ( x ) = 1 if x is true and 0 [-otherwise, and we are making a normal approximation for our model’s posterior predictive density, where we estimate parameters ˆ f = 1 M (cid:80) Mm =
 ˆ f m ( x ) , and ˆ σ = (cid:113) (cid:80)-]otherwise

Original: [Equation]
Revised: [Equation]

Original: −
Revised: and we are making a normal approximation for our model’s posterior predictive density, where we estimate parameters

Original: [Equation]
Revised: [Equation]

Original: {+and ˆ σ = (cid:113) (cid:80) Mm =
+}
Revised: {+and ˆ σ = (cid:113) (cid:80) Mm =
+}

Original:  In the UCB acquisition function, we use UCB to denote some estimate of a lower conﬁdence bound for the posterior predictive density (note that we use a lower conﬁdence bound rather than an upper conﬁdence bound since we are performing minimization), and β is a tradeoff parameter.
Revised: ( ˆ f m ( x ) − ˆ f ) 2 M −. In the UCB acquisition function, we use UCB to denote some estimate of a lower conﬁdence bound for the posterior predictive density (note that we use a lower conﬁdence bound rather than an upper conﬁdence bound since we are performing minimization), and β is a tradeoff parameter.

Original: In experiments we set β = 0 .
Revised: In experiments we set β = 0 .

Original: See Figure 5 (right).
Revised: See Figure 6 (right).

Original: We see that the acquisition function has little effect on the performance of
Revised: We see that the acquisition function has little effect on the performance of

Original: BANANAS, though UCB performs best.
Revised: BANANAS, though UCB performs best.

Original: Mean vs. random validation error.
Revised: Mean vs. random validation error.

Original: In the NASBench dataset, each architecture was trained to 108 epochs three separate times with different random seeds.
Revised: In the NASBench dataset, each architecture was trained to 108 epochs three separate times with different random seeds.

Original: The NASBench paper conducted experiments by choosing a random validation error when querying each architecture, and then reporting the mean test error at the conclusion of the NAS algorithm.
Revised: The NASBench paper conducted experiments by choosing a random validation error when querying each architecture, and then reporting the mean test error at the conclusion of the NAS algorithm.

Original: We found that the mismatch (choosing random validation error, but mean test error) added extra noise that would not be present during a real-life NAS experiment.
Revised: We found that the mismatch (choosing random validation error, but mean test error) added extra noise that would not be present during a real-life NAS experiment.

Original: Another way to conduct experiments on NASBench is to use mean validation error and mean test error.
Revised: Another way to conduct experiments on NASBench is to use mean validation error and mean test error.

Original: This is the method we used in Section 5 and the previous experiments from this section.
Revised: This is the method we used in Section 5 and the previous experiments from this section.

Original: Perhaps the most realistic experiment would be to use a random validation error and the test error corresponding to that validation error, however, the NASBench dataset does not explicitly give this functionality.
Revised: Perhaps the most realistic experiment would be to use a random validation error and the test error corresponding to that validation error, however, the NASBench dataset does not explicitly give this functionality.

Original: In Figure 6, we give results in the setting of the
Revised: In Figure 7, we give results in the setting of the

Original: NASBench paper, using random validation error and mean test error.
Revised: NASBench paper, using random validation error and mean test error.

Original: We ran each algorithm fortrials.
Revised: We ran each algorithm fortrials.

Original: Note that the test error across the board is higher because the correlation between validation and test error is lower (i.e., validation error is noisier than normal).
Revised: Note that the test error across the board is higher because the correlation between validation and test error is lower (i.e., validation error is noisier than normal).

Original: BANANAS still performs the best out of all algorithms that we tried.
Revised: BANANAS still performs the best out of all algorithms that we tried.

Original: We see the trends are largely unchanged, although Bayesian optimization with a GP model performs better than regularized evolution.
Revised: We see the trends are largely unchanged, although Bayesian optimization with a GP model performs better than regularized evolution.

Original: A possible explanation is that Bayesian optimization with a GP model is better-suited to handle noisy data than regularized evolution.
Revised: A possible explanation is that Bayesian optimization with a GP model is better-suited to handle noisy data than regularized evolution.

Original: C B EST PRACTICES CHECKLIST FOR NAS RESEARCH
Revised: C B EST PRACTICES CHECKLIST FOR NAS RESEARCH

Revised: The area of NAS has seen problems with reproducibility, as well as fair empirical comparisons, even more so than most other areas of machine learning.

Original: Following calls for fair and reproducible NAS  research (Li & Talwalkar, 2019; Sciuto et al., 2019;
Revised: Following calls for fair and reproducible NAS [Figure] research (Li & Talwalkar, 2019; Sciuto et al., 2019;

Original: Ying et al., 2019), a best practices checklist was recently created (Lindauer & Hutter, 2019).
Revised: Ying et al., 2019), a best practices checklist was recently created (Lindauer & Hutter, 2019).

Original: We address the points on the checklist.
Revised: In order to promote fair and reproducible NAS research, we address all points on the checklist, and we encourage future papers to do the same. • Code for the training pipeline used to evaluate the ﬁnal architectures. We used two of the most popular search spaces in NAS research, the NASBench search space, and the DARTS search space.

Original: Note that many points are addressed by virtue of using the NASBench dataset.
Revised: For NASBench, the accuracy of all architectures were precomputed.

Original:  •  We usedtwo of themost popular search spaces in NAS research, the NASBenchsearch space, andthe DARTS searchspace, and both search spaces and associated training pipelines are publicly released.
Revised: For the DARTS search space, we released our fork of the DARTS repo, which is forked from the DARTS repo designed speciﬁcally for reproducible experiments (Li & Talwalkar, 2019), making trivial changes to account for pytorch 1 . 2 . 0 . • Code for the search space. We used the popular and publicly avaliable NASBench and DARTS search spaces with no changes.

Original: • We did not change the hyperparmeters for the ﬁnal evaluation pipeline, and we will release the random seeds. 
Revised: • Hyperparameters used for the ﬁnal evaluation pipeline, as well as random seeds. We left all hyperparameters unchanged.

Original: • We will release the hyperparameters and  random seeds  for our search method.
Revised: We trained the architectures found by BANANAS, ASHA, and DARTS ﬁve times each, using random seeds 0, 1, 2, 3, 4. • For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset, search space, and code for training the architectures and hyperparameters for that code? Yes, we did this by virtue of the NASBench dataset. For the DARTS experiments, we used the reported architectures (found using the same search space and dataset as our method), and then we trained the ﬁnal architectures using the same code, including hyperparameters.

Original: • We compared different NAS methods using exactly the same NAS benchmark.
Revised: We compared different NAS methods using exactly the same NAS benchmark.

Original: [Figure] s t e rr o r
Revised: • Did you control for confounding factors?

Original: [Figure] f b e s t n
Revised: Yes, we used the same setup for all of our NASBench experiments.

Original: [Figure] u r [Figure] l n
Revised: For the DARTS search space, we compared our algorithm to two other algorithms using the same setup (pytorch version, CUDA version, etc).

Original: [Figure] • By re-training the best architecture released from other NAS methods, we were able to control all confounding factors including different hardware.
Revised: Across training over 5 seeds for each algorithm, we used different GPUs, which we found to have no greater effect than using a different random seed. • Did you run ablation studies?

Original: • We ran a thorough ablation study.
Revised: Yes, we ran a thorough ablation study.

Original: • We used the same evaluation protocol for all methods and we tried multiple evaluation protocols.
Revised: • Did you use the same evaluation protocol for the methods being compared? Yes, we used the same evaluation protocol for all methods and we tried multiple evaluation protocols.

Original: • On the NASBench experiments, we compared performance over number of queries, since all of our methods were  black-box optimization algorithms.
Revised: • Did you compare performance over time? In our main NASBench plots, we compared performance to number of queries, since all of our comparisons were to black-box optimization algorithms.

Revised: Note that the number of queries is almost perfectly correlated with runtime.

Original: We discussed the runtime and found all algorithms were between 46.5 and 47.0 TPU hours.
Revised: We computed the run time and found all algorithms were between 46.5 and 47.0 TPU hours.

Original: • We compared to random search.
Revised: • Did you compare to random search?

Original:  •  We ran 200 trials of our NASBench experiments and 4 trials of our experiments on the DARTS search space.
Revised: Yes. • Did you perform multiple runs of your experiments and report seeds? We ran 200 trials of our NASBench experiments.

Revised: Since we ran so many trials, we did not report random seeds.

Original: • We used NASBench, a  tabular dataset.
Revised: We ran four total trials of BANANAS on the DARTS search space. Currently we do not have a fully deterministic version of BANANAS on the DARTS search space (which would be harder to implement as the algorithm runs on 10 GPUs). However, the average ﬁnal error across trials was within 0.1%. • Did you use tabular or surrogate benchmarks for in-depth evaluations

Original:  •  We  found our algorithm to work well without hyperparameter tuning, although we performed light hyperparameter tuning for all algorithms to optimize for the 150 query setting on NASBench.
Revised: Yes, we used NASBench. • Did you report how you tuned hyperparameters, and what time and resources this required? We performed light hyperparameter tuning for the meta neural network to choose the number of layers, layer size, learning rate, and number of epochs. In general, we found our algorithm to work well without hyperparameter tuning.

Original: •  We reported time for the entire end-to-end NAS method.
Revised: • Did you report the time for the entire end-to-end NAS method? We reported time for the entire end-to-end NAS method.

Original: •  We reported all details of our experimental setup.
Revised: • Did you report all details of your experimental setup? We reported all details of our experimental setup.