Original: G RAPH PNAS: L EARNING D ISTRIBUTIONS OF G OOD N EURAL A RCHITECTURES VIA D EEP G RAPH G ENERATIVE M ODELS
Revised: G RAPH PNAS: L EARNING D ISTRIBUTIONS OF G OOD N EURAL A RCHITECTURES VIA D EEP G RAPH G ENERATIVE M ODELS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Neural architectures can be naturally viewed as computational graphs.
Revised: Neural architectures can be naturally viewed as computational graphs.

Original: Motivated by this perspective, we, in this paper, study neural architecture search (NAS) through the lens of learning random graph models.
Revised: Motivated by this perspective, we, in this paper, study neural architecture search (NAS) through the lens of learning random graph models.

Original: In contrast to existing NAS methods which largely focus on searching for a single best architecture, i.e ., point estimation, we propose GraphPNAS , a deep graph generative model that learns a distribution of well-performing architectures.
Revised: In contrast to existing NAS methods which largely focus on searching for a single best architecture, i.e ., point estimation, we propose GraphPNAS , a deep graph generative model that learns a distribution of well-performing architectures.

Original: Relying on graph neural networks (GNNs), our GraphPNAS can better capture topologies of good neural architectures and relations between operators therein.
Revised: Relying on graph neural networks (GNNs), our GraphPNAS can better capture topologies of good neural architectures and relations between operators therein.

Original: Moreover, our graph generator leads to a learnable probabilistic search method that is more flexible and efficient than the commonly used RNN generator and random search methods.
Revised: Moreover, our graph generator leads to a learnable probabilistic search method that is more flexible and efficient than the commonly used RNN generator and random search methods.

Original: Finally, we learn our generator via an efficient reinforcement learning formulation for NAS.
Revised: Finally, we learn our generator via an efficient reinforcement learning formulation for NAS.

Original: To assess the effectiveness of our GraphPNAS, we conduct extensive experiments on three search spaces, including the challenging RandWire on Tiny-ImageNet,
Revised: To assess the effectiveness of our GraphPNAS, we conduct extensive experiments on three search spaces, including the challenging RandWire on Tiny-ImageNet,

Original: ENAS on CIFAR10, and NAS-Bench-101.
Revised: ENAS on CIFAR10, and NAS-Bench-101.

Original: The complexity of RandWire is significantly larger than other search spaces in the literature.
Revised: The complexity of RandWire is significantly larger than other search spaces in the literature.

Original: We show that our proposed graph generator consistently outperforms RNN based one and achieves better or comparable performances than state-of-the-art NAS methods.
Revised: We show that our proposed graph generator consistently outperforms RNN based one and achieves better or comparable performances than state-of-the-art NAS methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, we have witnessed a rapidly growing list of successful neural architectures that underpin deep learning, e.g ., VGG, LeNet, ResNets
Revised: In recent years, we have witnessed a rapidly growing list of successful neural architectures that underpin deep learning, e.g ., VGG, LeNet, ResNets

Original: (He et al., 2016),
Revised: (He et al., 2016),

Original: Transformers (Dosovitskiy et al., 2020).
Revised: Transformers (Dosovitskiy et al., 2020).

Original: Designing these architectures requires researchers to go through time-consuming trial and errors.
Revised: Designing these architectures requires researchers to go through time-consuming trial and errors.

Original: Neural architecture search (NAS) (Zoph & Le, 2016; Elsken et al., 2018b) has emerged as an increasingly popular research area which aims to automatically find state-of-the-art neural architectures without human-in-the-loop.
Revised: Neural architecture search (NAS) (Zoph & Le, 2016; Elsken et al., 2018b) has emerged as an increasingly popular research area which aims to automatically find state-of-the-art neural architectures without human-in-the-loop.

Original: NAS methods typically have two components: a search module and an evaluation module.
Revised: NAS methods typically have two components: a search module and an evaluation module.

Original: The search module is expressed by a machine learning model, such as a deep neural network, designed to operate in a high dimensional search space.
Revised: The search module is expressed by a machine learning model, such as a deep neural network, designed to operate in a high dimensional search space.

Original: The search space, of all admissible architectures, is often designed by hand in advance.
Revised: The search space, of all admissible architectures, is often designed by hand in advance.

Original: The evaluation module takes an architecture as input and outputs the reward, e.g ., performance of this architecture trained and then evaluated with a metric.
Revised: The evaluation module takes an architecture as input and outputs the reward, e.g ., performance of this architecture trained and then evaluated with a metric.

Original: The learning process of NAS methods typically iterates between the following two steps.
Revised: The learning process of NAS methods typically iterates between the following two steps.

Original: The search module produces candidate architectures and sends them to the evaluation module; 2) The evaluation module evaluates these architectures to get the reward and sends the reward back to the search module.
Revised: The search module produces candidate architectures and sends them to the evaluation module; 2) The evaluation module evaluates these architectures to get the reward and sends the reward back to the search module.

Original: Ideally, based on the feedback from the evaluation module, the search module should learn to produce better and better architectures.
Revised: Ideally, based on the feedback from the evaluation module, the search module should learn to produce better and better architectures.

Original: Unsurprisingly, this learning paradigm of
Revised: Unsurprisingly, this learning paradigm of

Original: NAS methods fits well to reinforcement learning (RL).
Revised: NAS methods fits well to reinforcement learning (RL).

Original: Most NAS methods (Liu et al., 2018b; White et al., 2020; Cai et al., 2019) only return a single best architecture ( i.e ., a point estimate) after the learning process.
Revised: Most NAS methods (Liu et al., 2018b; White et al., 2020; Cai et al., 2019) only return a single best architecture ( i.e ., a point estimate) after the learning process.

Original: This point estimate could be very biased as it typically underexplores the search space.
Revised: This point estimate could be very biased as it typically underexplores the search space.

Original: Further, a given search space may contain multiple (equally) good architectures, a feature that a point estimate cannot capture.
Revised: Further, a given search space may contain multiple (equally) good architectures, a feature that a point estimate cannot capture.

Original: Even worse, since the learning problem of NAS is essentially a discrete optimization where multiple local minima exist, many local search style NAS methods (Ottelander et al., 2020) tend to get stuck in local minima.
Revised: Even worse, since the learning problem of NAS is essentially a discrete optimization where multiple local minima exist, many local search style NAS methods (Ottelander et al., 2020) tend to get stuck in local minima.

Original: From the Bayesian perspective, modelling the distribution of architectures is inherently better than point estimation, e.g ., leading to the ability to form ensemble methods that work better in practice.
Revised: From the Bayesian perspective, modelling the distribution of architectures is inherently better than point estimation, e.g ., leading to the ability to form ensemble methods that work better in practice.

Original: Moreover, modelling the distribution of architectures naturally caters to probabilistic search methods which are better suited for avoiding local optima, e.g ., simulated annealing.
Revised: Moreover, modelling the distribution of architectures naturally caters to probabilistic search methods which are better suited for avoiding local optima, e.g ., simulated annealing.

Original: Finally, modeling the distribution of architectures allows to capture complex structural dependencies between operations that characterize good architectures capable of more efficient learning and generalization.
Revised: Finally, modeling the distribution of architectures allows to capture complex structural dependencies between operations that characterize good architectures capable of more efficient learning and generalization.

Original: Motivated by the above observations and the fact that neural architectures can be naturally viewed as attributed graphs, we propose a probabilistic graph generator which models the distribution over good architectures using graph neural networks (GNNs).
Revised: Motivated by the above observations and the fact that neural architectures can be naturally viewed as attributed graphs, we propose a probabilistic graph generator which models the distribution over good architectures using graph neural networks (GNNs).

Original: Our generator excels at generating topologies with complicated structural dependencies between operations.
Revised: Our generator excels at generating topologies with complicated structural dependencies between operations.

Original: From the Bayesian inference perspective, our generator returns a distribution over good architectures, rather than a single point estimate, allowing to capture the multi-modal nature of the posterior distribution of good architectures and to effectively average or ensemble architecture (sample) estimates.
Revised: From the Bayesian inference perspective, our generator returns a distribution over good architectures, rather than a single point estimate, allowing to capture the multi-modal nature of the posterior distribution of good architectures and to effectively average or ensemble architecture (sample) estimates.

Original: Different from the
Revised: Different from the

Original: Bayesian deep learning (Neal, 2012; Blundell et al., 2015; Gal & Ghahramani, 2016) that models distributions of weights/hidden units, we model distributions of neural architectures.
Revised: Bayesian deep learning (Neal, 2012; Blundell et al., 2015; Gal & Ghahramani, 2016) that models distributions of weights/hidden units, we model distributions of neural architectures.

Original: Lastly, our probabilistic generator is less prone to the issue of local minima, since multiple random architectures are generated at each step during learning.
Revised: Lastly, our probabilistic generator is less prone to the issue of local minima, since multiple random architectures are generated at each step during learning.

Original: In summary, our key contributions are as below.
Revised: In summary, our key contributions are as below.

Original: • We propose a GNN-based graph generator for neural architectures which empowers a learnable probabilistic search method.
Revised: • We propose a GNN-based graph generator for neural architectures which empowers a learnable probabilistic search method.

Original: To the best of our knowledge, we are the first to explore learning deep graph generative models as generators in NAS.
Revised: To the best of our knowledge we are the first to explore learning deep graph generative models as generators in NAS.

Original: • We explore a significantly larger search space ( e
Revised: • We explore a significantly larger search space ( e

Original: g ., graphs with 32 operators) than the literature ( e.g ., garphs with up to 12 operators) and propose to evaluate architectures under low-data regime, which altogether boost effectiveness and efficiency of our NAS system.
Revised: g ., graphs with 32 operators) than the literature ( e.g ., garphs with up to 12 operators) and propose to evaluate architectures under low-data regime, which altogether boost effectiveness and efficiency of our NAS system.

Original: • Extensive experiments on three different search spaces show that our method consistently outperforms RNN-based generators and is slightly better or comparable to the state-of-theart NAS methods.
Revised: • Extensive experiments on three different search spaces show that our method consistently outperforms RNN-based generators and is slightly better or comparable to the state-of-theart NAS methods.

Original: Also, it can generalize well across different NAS system setups.
Revised: Also, it can generalize well across different NAS system setups.

Original:  R ELATED W ORKS
Revised:  R ELATED W ORKS

Original: Neural Architecture Search.
Revised: Neural Architecture Search.

Original: The main challenges in NAS are 1) the hardness of discrete optimization, 2) the high cost for evaluating neural networks, and 3) the lack of principles in the search space design.
Revised: The main challenges in NAS are 1) the hardness of discrete optimization, 2) the high cost for evaluating neural networks, and 3) the lack of principles in the search space design.

Original: First, to tackle the discrete optimization, evolution strategies (ES) (Elsken et al., 2019; Real et al., 2019), reinforcement learning (RL) (Baker et al., 2017; Zhong et al., 2018; Pham et al., 2018; Liu et al., 2018a), Bayesian optimization (Bergstra et al., 2013; White et al., 2019) and continuous relaxations (Liu et al., 2018b) have been explored in the literature.
Revised: First, to tackle the discrete optimization, evolution strategies (ES) (Elsken et al., 2019; Real et al., 2019a), reinforcement learning (RL) (Baker et al., 2017; Zhong et al., 2018; Pham et al., 2018; Liu et al., 2018a), Bayesian optimization (Bergstra et al., 2013; White et al., 2019) and continuous relaxations (Liu et al., 2018b) have been explored in the literature.

Original: We follow the RL path as it is principled, flexible in injecting prior knowledge, achieves the state-of-the-art performances (Tan &
Revised: We follow the RL path as it is principled, flexible in injecting prior knowledge, achieves the state-of-the-art performances (Tan &

Original: Le, 2019), and can be naturally applied to our graph generator.
Revised: Le, 2019), and can be naturally applied to our graph generator.

Original: Second, the evaluation requires training individual neural architectures which is notoriously time consuming(Zoph & Le, 2016).
Revised: Second, the evaluation requires training individual neural architectures which is notoriously time consuming(Zoph & Le, 2016).

Original: Pham et al.
Revised: Pham et al.

Original: ; Liu et al.
Revised: ; Liu et al.

Original: (2018b) propose a weight-sharing supernet to reduce the training time.
Revised: (2018b) propose a weight-sharing supernet to reduce the training time.

Original: Baker et al.
Revised: Baker et al.

Original: (2018) use a machine learning model to predict the performance of fully-trained architectures conditioned on early-stage performances.
Revised: (2018) use a machine learning model to predict the performance of fully-trained architectures conditioned on early-stage performances.

Original: Brock et al.
Revised: Brock et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018) directly predict weights from the search architectures via hypernetworks.
Revised: (2018) directly predict weights from the search architectures via hypernetworks.

Original: Since our graph generator do not relies on specific choice of evaluation method, we choose to experiment on both oracle training(training from scratch) and supernet settings for completeness.
Revised: Since our graph generator do not relies on specific choice of evaluation method, we choose to experiment on both oracle training(training from scratch) and supernet settings for completeness.

Original: Third, the search space of NAS largely determines the optimization landscape and bounds the best-possible performance.
Revised: Third, the search space of NAS largely determines the optimization landscape and bounds the best-possible performance.

Original: It is obvious that the larger the search space is, the better the best-possible performance and the higher the search cost would likely be.
Revised: It is obvious that the larger the search space is, the better the best-possible performance and the higher the search cost would likely be.

Original: Besides this trade-off, few principles are known about designing the search space.
Revised: Besides this trade-off, few principles are known about designing the search space.

Original: Previous work (Pham et al., 2018; Liu et al., 2018b; Ying et al., 2019; Li et al., 2020) mostly focuses on cell-based search space.
Revised: Previous work (Pham et al., 2018; Liu et al., 2018b; Ying et al., 2019; Li et al., 2020) mostly focuses on cell-based search space.

Original: A cell is defined as a small ( e.g ., up to 8 operators) computational graph where nodes ( i.e ., operators like 3 × 3 convolution) are connected following some topology.
Revised: A cell is defined as a small ( e.g ., up to 8 operators) computational graph where nodes ( i.e ., operators like 3 × 3 convolution) are connected following some topology.

Original: Once the search is done, one often stacks up multiple cells with the same topology but different weights to build the final neural network.
Revised: Once the search is done, one often stacks up multiple cells with the same topology but different weights to build the final neural network.

Original: Other works (Tan et al., 2019; Cai et al., 2019; Tan & Le, 2019) typically fix the topology, e.g ., a sequential backbone, and search for layer-wise configurations ( e.g ., operator types like 3 × 3 vs. 5 × 5 convolution and number of filters).
Revised: Other works (Tan et al., 2019; Cai et al., 2019; Tan & Le, 2019) typically fix the topology, e.g ., a sequential backbone, and search for layer-wise configurations ( e.g ., operator types like 3 × 3 vs. 5 × 5 convolution and number of filters).

Original: In our method, to demonstrate our graph generator’s ability in exploring large topology search space, we first explore on a challenging large cell space (32 operators), after which we experiment on ENAS Macro (Pham et al., 2018) and Nas-Benchmark-101(Ying et
Revised: In our method, to demonstrate our graph generator’s ability in exploring large topology search space, we first explore on a challenging large cell space (32 operators), after which we experiment on ENAS Macro (Pham et al., 2018) and NAS-Benchmark-101(Ying et

Original: al., 2019) for more comparison with previous methods.
Revised: al., 2019) for more comparison with previous methods.

Revised: Neural Architecture as Graph for NAS.

Revised: Recently, a line of NAS research works propose to view neural architecture as a graph and encode it using graph neural networks (GNN).

Revised: In (Zhang et al., 2020; Luo et al., 2018), graph auto-encoders are used to map neural architectures to and back from a continuous space for gradient-based optimization.

Original:  Closely related to our work, Xie et al.
Revised: Shi et al. (2020) use bayesian optimization method, where GNN is used to get embedding from neural architectures. Despite the extensive use of GNN as an encoder, few works have been focusing on building graph generative models for NAS. Closely related to our work, Xie et al.

Original: (2019) explore different topologies of the similar cell space using non-learnable random graph models and You et al.
Revised: (2019) explore different topologies of the similar cell space using non-learnable random graph models. You et al.

Original: (2020) subsequently investigate the relationship between topologies and performances.
Revised: (2020) subsequently investigate the relationship between topologies and performances.

Revised: Following this, Ru et al.

Revised: (2020) propose a hierarchical search space modeled by random graph generators and optimize hyper-parameters using bayesian optimization.

Original: They are different from our work as we learn the graph generator to automatically explore the cell space.
Revised: They are different from our work as we learn the graph generator to automatically explore the cell space.

Original: Deep Graph Generative Models.
Revised: Deep Graph Generative Models.

Original: Graph generative models date back to the Erd ˝ os–R ´ enyi model (Erd ¨ os & R ´ enyi, 1959), of which the probability of generating individual edges is the same.
Revised: Graph generative models date back to the Erd˝os–R´enyi model (Erd¨os & R´enyi, 1959), of which the probability of generating individual edges is the same.

Original: Other well-known graph generative models include the stochastic block model (Holland et al., 1983), the small-world model (Watts & Strogatz, 1998), and the preferential attachment model (Barab ´ asi &
Revised: Other well-known graph generative models include the stochastic block model (Holland et al., 1983), the small-world model (Watts & Strogatz, 1998), and the preferential attachment model (Barab´asi &

Original: Albert, 1999).
Revised: Albert, 1999).

Original: Recently, deep graph generative models instead parameterize the probability of generating edges and nodes using deep neural networks in, e.g ., the auto-regressive fashion (Li et al., 2018;
Revised: Recently, deep graph generative models instead parameterize the probability of generating edges and nodes using deep neural networks in, e.g ., the auto-regressive fashion (Li et al., 2018;

Original: You et al., 2018; Liao et al., 2019) or variational autoencoder fashion (Kipf & Welling, 2016;
Revised: You et al., 2018; Liao et al., 2019) or variational autoencoder fashion (Kipf & Welling, 2016;

Original: Grover et al., 2018; Liu et al., 2019).
Revised: Grover et al., 2018; Liu et al., 2019).

Original: These models are highly flexible and can model complicated distributions of real-world graphs, e.g ., molecules (Jin et al., 2018), road networks (Chu et al., 2019), and program structures (Brockschmidt et al., 2018).
Revised: These models are highly flexible and can model complicated distributions of real-world graphs, e.g ., molecules (Jin et al., 2018), road networks (Chu et al., 2019), and program structures (Brockschmidt et al., 2018).

Original: Our graph generator builds on top of the stateof-the-art deep graph generative model in (Liao et al., 2019) with several important distinctions.
Revised: Our graph generator builds on top of the stateof-the-art deep graph generative model in (Liao et al., 2019) with several important distinctions.

Original: First, instead of only generating nodes and edges, we also generate node attributes ( e.g ., operator types in neural architectures).
Revised: First, instead of only generating nodes and edges, we also generate node attributes ( e.g ., operator types in neural architectures).

Original: Second, since good neural architectures are actually latent, our learning objective maximizes the expected reward ( e.g ., validation accuracies) rather than the simple log likelihood, thus being more challenging.
Revised: Second, since good neural architectures are actually latent, our learning objective maximizes the expected reward ( e.g ., validation accuracies) rather than the simple log likelihood, thus being more challenging.

Original:  M ETHODS
Revised:  M ETHODS

Original: The architecture of any feedforward neural network can be naturally represented as a directed acyclic graph (DAG), a.k.a., computational graph .
Revised: The architecture of any feedforward neural network can be naturally represented as a directed acyclic graph (DAG), a.k.a., computational graph .

Original: There exist two equivalent ways to define the computational graph.
Revised: There exist two equivalent ways to define the computational graph.

Original: First, we denote operations ( e.g ., convolutions) as nodes and denote operands ( e.g ., tensors) as edges which indicate how the computation flows.
Revised: First, we denote operations ( e.g ., convolutions) as nodes and denote operands ( e.g ., tensors) as edges which indicate how the computation flows.

Original: Second, we denote operands as nodes and denote operators as edges.
Revised: Second, we denote operands as nodes and denote operators as edges.

Original: We adopt the first view.
Revised: We adopt the first view.

Original: In particular, a neural network G with N operations is defined as a tuple ( A, X ) where A ∈ { 0 , 1 } N × N is an N × N adjacent matrix with
Revised: In particular, a neural network G with N operations is defined as a tuple ( A, X ) where A ∈ { 0 , 1 } N × N is an N × N adjacent matrix with

Original: A ij = 1 indicates that the output of the j -th operator is used as the input of the i -th operator.
Revised: A ij = 1 indicates that the output of the j -th operator is used as the input of the i -th operator.

Original: For operator with multiple inputs, inputs are combined together ( e.g ., using sum or average operator) before sending into the operator.
Revised: For operator with multiple inputs, inputs are combined together ( e.g ., using sum or average operator) before sending into the operator.

Original: X is a N -size attribute vector encoding operation types.
Revised: X is a N -size attribute vector encoding operation types.

Original: For any operation i , its operation type X i can only choose from a pre-defined list with length D , e.g ., 1 × 1 , 3 × 3 or 5 × 5 convolutions.
Revised: For any operation i , its operation type X i can only choose from a pre-defined list with length D , e.g ., 1 × 1 , 3 × 3 or 5 × 5 convolutions.

Original: Note that for any valid feedforward architecture, G can not have loops.
Revised: Note that for any valid feedforward architecture, G can not have loops.

Original: One sufficient condition to satisfy the requirement is to constrain A to be a lower triangu- lar matrix with zero diagonal ( i.e ., excluding self-loops).
Revised: One sufficient condition to satisfy the requirement is to constrain A to be a lower triangular matrix with zero diagonal ( i.e ., excluding self-loops).

Original: This formalism creates a search space of
Revised: This formalism creates a search space of

Original: D N 2 N ( N − 1) / 2 possible architectures, which is huge even for moderately large number of operators
Revised: D N 2 N ( N − 1) / 2 possible architectures, which is huge even for moderately large number of operators

Original: N and number of operation types D .
Revised: N and number of operation types D .

Original: The goal of NAS is to find an architecture or a set of architectures within this search space that would perform well.
Revised: The goal of NAS is to find an architecture or a set of architectures within this search space that would perform well.

Original: For practical consideration, we search for cell graphs ( e.g ., N = 32 ) and then replicate this cell several times to build a deep neural architecture.
Revised: For practical consideration, we search for cell graphs ( e.g ., N = 32 ) and then replicate this cell several times to build a deep neural architecture.

Original: We also experiment on the ENAS Macro search space where G defines a entire network.
Revised: We also experiment on the ENAS Macro search space where G defines a entire network.

Original: More details for the corresponding search spaces can be found in Section 4.
Revised: More details for the corresponding search spaces can be found in Section 4.

Original: 3. N EURAL A RCHITECTURE S EARCH S YSTEM
Revised: 3. N EURAL A RCHITECTURE S EARCH S YSTEM

Original: Before delving into details, we first give an overview of our NAS system, which consists of two parts: a generator and an evaluator.
Revised: Before delving into details, we first give an overview of our NAS system, which consists of two parts: a generator and an evaluator.

Original: The system diagram is shown in Fig.
Revised: The system diagram is shown in Fig.

Original: At each step, the probabilistic graph generator samples a set of cell graphs, which are further translated to neural architectures by replicating the cell graph multiple times and stacking them up.
Revised: At each step, the probabilistic graph generator samples a set of cell graphs, which are further translated to neural architectures by replicating the cell graph multiple times and stacking them up.

Original: Then the evaluator evaluates these architectures, obtains rewards, and sends architecture-reward pairs to the replay buffer.
Revised: Then the evaluator evaluates these architectures, obtains rewards, and sends architecture-reward pairs to the replay buffer.

Original: The replay buffer is then used to improve the generator, effectively forming a reinforcement learning loop.
Revised: The replay buffer is then used to improve the generator, effectively forming a reinforcement learning loop.

Original: 3.1. P ROBABILISTIC G ENERATORS FOR N EURAL A RCHITECTURES
Revised: 3.1. P ROBABILISTIC G ENERATORS FOR N EURAL A RCHITECTURES

Original: Now we introduce our probabilistic graph generator which is based on a state-of-the-art deep autoregressive graph generative model in (Liao et al., 2019).
Revised: Now we introduce our probabilistic graph generator which is based on a state-of-the-art deep autoregressive graph generative model in (Liao et al., 2019).

Original: Auto-Regressive Generation.
Revised: Auto-Regressive Generation.

Original: Specifically, we decompose the distribution of a cell graph along with attributes (operation types) in an auto-regressive fashion,
Revised: Specifically, we decompose the distribution of a cell graph along with attributes (operation types) in an auto-regressive fashion,

Original: [Equation]
Revised: [Equation]

Original: where A i, : and X i denote the i -th row of the adjacency matrix A and the i -th operation type respectively.
Revised: where A i, : and X i denote the i -th row of the adjacency matrix A and the i -th operation type respectively.

Original: To ensure the generated graphs are DAGs, we constrain A to be lower triangular by adding a binary mask, i.e ., the i -th node can only be reached from the first i − 1 nodes.
Revised: To ensure the generated graphs are DAGs, we constrain A to be lower triangular by adding a binary mask, i.e ., the i -th node can only be reached from the first i − 1 nodes.

Original: We omit the masks in the equations for better readability.
Revised: We omit the masks in the equations for better readability.

Original: We further model the conditional distributions as follows,
Revised: We further model the conditional distributions as follows,

Original: [Equation]
Revised: [Equation]

Original: where the distributions of the operation type and edges are categorical and K -mixture of Bernoulli respectively.
Revised: where the distributions of the operation type and edges are categorical and K -mixture of Bernoulli respectively.

Original: D is again the number of operation types.
Revised: D is again the number of operation types.

Original: MLP α , MLP β , and MLP θ are different instances of two-layer MLPs with ReLU activations.
Revised: MLP α , MLP β , and MLP θ are different instances of two-layer MLPs with ReLU activations.

Original: Here h Si is the representation of i -th node returned by a GNN which has been executed S steps of message passing at each generation step.
Revised: Here h S i is the representation of i -th node returned by a GNN which has been executed S steps of message passing at each generation step.

Original: This auto-regressive construction breaks down the nice property of permutation invariance for graph generation.
Revised: This auto-regressive construction breaks down the nice property of permutation invariance for graph generation.

Original: However, we do not find it as an issue in practice, partly due to the fact that the graph isomorphism becomes less likely to happen while considering both topology and operation types.
Revised: However, we do not find it as an issue in practice, partly due to the fact that the graph isomorphism becomes less likely to happen while considering both topology and operation types.

Original: Message Passing GNNs.
Revised: Message Passing GNNs.

Original: Each generation step n ≤ N in auto-regressive generation above relies on representations of nodes up to and including n itself (see Eq.
Revised: Each generation step n ≤ N in auto-regressive generation above relies on representations of nodes up to and including n itself (see Eq.

Original: To obtain these node representations { h S i } , we exploit message passing GNNs (Gilmer et al., 2017) with an attention mechanism similar to (Liao et al., 2019).
Revised: To obtain these node representations { h Si } , we exploit message passing GNNs (Gilmer et al., 2017) with an attention mechanism similar to (Liao et al., 2019).

Original: In particular, the s -th message passing step involves executing the following equations successively,
Revised: In particular, the s -th message passing step involves executing the following equations successively,

Original: [Equation]
Revised: [Equation]

Original: where N ( i ) is the set of node i along with its neighboring nodes.
Revised: where N ( i ) is the set of node i along with its neighboring nodes.

Original: m sij is the message sent from node i to j at the s -th message passing step.
Revised: m sij is the message sent from node i to j at the s -th message passing step.

Original: The connectivity for the propagation in GNN is given by
Revised: The connectivity for the propagation in GNN is given by

Original: A 1: i − 1 , 1: i − 1 with the last node (for which A i, : has not been generated yet) being fully connected.
Revised: A 1: i − 1 , 1: i − 1 with the last node (for which A i, : has not been generated yet) being fully connected.

Original: Note that message passing step is different from the generation step and we run multiple message passing steps per generation step in order to capture the structural dependency among nodes and edges.
Revised: Note that message passing step is different from the generation step and we run multiple message passing steps per generation step in order to capture the structural dependency among nodes and edges.

Original: The f and g are two-layer MLPs.
Revised: The f and g are two-layer MLPs.

Original: Since graphs are DAGs in our case rather than undirected ones as in (Liao et al., 2019), we add 1 ij in Eq.
Revised: Since graphs are DAGs in our case rather than undirected ones as in (Liao et al., 2019), we add 1 ij in Eq.

Original: (7), a one-hot vector for indicating the direction of the edge.
Revised: (7), a one-hot vector for indicating the direction of the edge.

Original: We initialize the node representations
Revised: We initialize the node representations

Original: h 

Revised: h

Original: i (for i < n ) as the corresponding one-hot encoded operation type vectors; h 0 n is initialized to a special one-hot vector.
Revised: i (for i < n ) as the corresponding one-hot encoded operation type vectors; h 0 n is initialized to a special one-hot vector.

Original: Here u i is an additional feature vector that helps distinguish i -th node from others.
Revised: Here u i is an additional feature vector that helps distinguish i -th node from others.

Original: We found using one-hot-encoded incoming neighbors of i -th node and a positional encoding of the node index i work well in practice.
Revised: We found using one-hot-encoded incoming neighbors of i -th node and a positional encoding of the node index i work well in practice.

Original: We encourage readers to reference Fig.
Revised: We encourage readers to reference Fig.

Original: 4 for a detailed visualization of graph generation process.
Revised: 4 for a detailed visualization of graph generation process.

Original: Sampling.
Revised: Sampling.

Original: To sample from our generator, we first draw architectures following the standard ancestral sampling where each step involves drawing random samples from a categorical distribution and a mixture of Bernoulli distribution.
Revised: To sample from our generator, we first draw architectures following the standard ancestral sampling where each step involves drawing random samples from a categorical distribution and a mixture of Bernoulli distribution.

Original: At each step, this sampling process adds a new operator with a certain operation type and wire it to previously sampled operators.
Revised: At each step, this sampling process adds a new operator with a certain operation type and wire it to previously sampled operators.

Original: 3.1. E VALUATOR
Revised: 3.1. E VALUATOR

Original: Our design of generator and NAS pipeline do not rely on a specific choice of evaluator.
Revised: Our design of generator and NAS pipeline do not rely on a specific choice of evaluator.

Original: Motivated by (Mnih et al., 2013), we use a replay buffer for storing the evaluated architectures.
Revised: Motivated by (Mnih et al., 2013), we use a replay buffer for storing the evaluated architectures.

Original: In our paper, based on specific datasets, we explore three types of evaluators, namely, oracle evaluator, supernet evaluator and benchmark evaluator, which are briefly introduced as follows.
Revised: In our paper, based on specific datasets, we explore three types of evaluators, namely, oracle evaluator, supernet evaluator and benchmark evaluator, which are briefly introduced as follows.

Original: Oracle evaluator.
Revised: Oracle evaluator.

Original: Given a sample from the generator, an oracle evaluator trains the corresponding network from scratch and tests it to get the validation performances.
Revised: Given a sample from the generator, an oracle evaluator trains the corresponding network from scratch and tests it to get the validation performances.

Original: To reduce computation overhead, a common approach is to use early stopping (training with fewer epochs) as in (Tan et al., 2019; Tan & Le, 2019).
Revised: To reduce computation overhead, a common approach is to use early stopping (training with fewer epochs) as in (Tan et al., 2019; Tan & Le, 2019).

Original: In our experiment, we instead use a low-data evaluator similar to few-shot learning where we keep the same number of classes but use fewer samples per class to train.
Revised: In our experiment, we instead use a low-data evaluator similar to few-shot learning where we keep the same number of classes but use fewer samples per class to train.

Original: SuperNet evaluator.
Revised: SuperNet evaluator.

Original: Aiming at further reducing the amount of compute, this evaluator uses a weight-sharing strategy where each graph is a sub-graph of the supernet.
Revised: Aiming at further reducing the amount of compute, this evaluator uses a weight-sharing strategy where each graph is a sub-graph of the supernet.

Original: We followed the singlepath supernet setup used in (Pham et al., 2018) to compare with previous methods.
Revised: We followed the singlepath supernet setup used in (Pham et al., 2018) to compare with previous methods.

Original: Benchmark evaluator.
Revised: Benchmark evaluator.

Original: NAS benchmarks, e.g ., (Ying et al., 2019), provide accurate evaluation for architectures within the search space, which can be seen as oracle evaluators with full training budgets on target datasets.
Revised: NAS benchmarks, e.g ., (Ying et al., 2019), provide accurate evaluation for architectures within the search space, which can be seen as oracle evaluators with full training budgets on target datasets.

Original: 3. L EARNING M ETHOD
Revised: 3. L EARNING M ETHOD

Original: Since we are dealing with discrete latent variables, i.e ., good architectures in our case, we train our NAS system using REINFORCE (Williams, 1992) algorithm with the control variate (a.k.a. baseline) to reduce the variance.
Revised: Since we are dealing with discrete latent variables, i.e ., good architectures in our case, we train our NAS system using REINFORCE (Williams, 1992) algorithm with the control variate (a.k.a. baseline) to reduce the variance.

Original: In particular, the gradient of the loss or negative expected reward
Revised: In particular, the gradient of the loss or negative expected reward

Original: L w.r.t.
Revised: L w.r.t.

Original: the generator parameters ϕ is,
Revised: the generator parameters ϕ is,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where the reward ¯ R is standardized as ¯ R ( G ) =
Revised: where the reward ¯ R is standardized as ¯ R ( G ) =

Original: ( R ( G ) − C ) /σ .
Revised: ( R ( G ) − C ) /σ .

Original: Here the baseline C is the average reward of architectures in the replay buffer and σ is standard deviation of rewards in the replay buffer.
Revised: Here the baseline C is the average reward of architectures in the replay buffer and σ is standard deviation of rewards in the replay buffer.

Original: The expectation in Eq.
Revised: The expectation in Eq.

Original: (11) is approximated by the Monte Carlo estimation.
Revised: (11) is approximated by the Monte Carlo estimation.

Original: However, the score function ( i.e ., the gradient of log likelihood w.r.t.
Revised: However, the score function ( i.e ., the gradient of log likelihood w.r.t.

Original: parameters) in the above equation may numerically differ a lot for different architectures.
Revised: parameters) in the above equation may numerically differ a lot for different architectures.

Original: For example, if a negative sample, i.e ., an architecture with a reward lower than the baseline, has a low probability P ( G ) , it would highly likely to have an extremely large absolute score function value, thus leading to a negative reward with an extremely large magnitude.
Revised: For example, if a negative sample, i.e ., an architecture with a reward lower than the baseline, has a low probability P ( G ) , it would highly likely to have an extremely large absolute score function value, thus leading to a negative reward with an extremely large magnitude.

Original: Therefore, in order to balance positive and negative rewards, we propose to use the reweighted log likelihood as follows,
Revised: Therefore, in order to balance positive and negative rewards, we propose to use the reweighted log likelihood as follows,

Original: [Equation]
Revised: [Equation]

Original: where β is a hyperparameter that controls the weighting between negative and positive rewards.
Revised: where β is a hyperparameter that controls the weighting between negative and positive rewards.

Original: P ( G ) is the original probability given by our generator.
Revised: P ( G ) is the original probability given by our generator.

Original: Exploration vs. Exploitation Similar to many RL approaches, our NAS system faces the exploration vs. exploitation dilemma.
Revised: Exploration vs. Exploitation Similar to many RL approaches, our NAS system faces the exploration vs. exploitation dilemma.

Original: We found that our NAS system may quickly collapse ( i.e ., overly exploit) to a few good architectures due to the powerful graph generative model, thus losing the diversity and reducing to point estimate.
Revised: We found that our NAS system may quickly collapse ( i.e ., overly exploit) to a few good architectures due to the powerful graph generative model, thus losing the diversity and reducing to point estimate.

Original: Inspired by the epsilon greedy algorithm (Sutton & Barto, 2018) used in multi-armed bandit problems, we design a random explorer to encourage more exploration in the early stage.
Revised: Inspired by the epsilon greedy algorithm (Sutton & Barto, 2018) used in multi-armed bandit problems, we design a random explorer to encourage more exploration in the early stage.

Original: Specifically, at each search step, our generator samples from either itself or a prior graph distribution like the Watts–Strogatz model with a probability ϵ .
Revised: Specifically, at each search step, our generator samples from either itself or a prior graph distribution like the Watts–Strogatz model with a probability ϵ .

Original: As the search goes on, ϵ is gradually annealed to 0 so that the generator gradually has more exploitation over exploration.
Revised: As the search goes on, ϵ is gradually annealed to 0 so that the generator gradually has more exploitation over exploration.

Revised: Whats more, we design our replay buffer to keep a small portion of candidates.

Revised: As training goes on, bad samples will be gradually be replaced by good samples for training our generators, which encourage the model to exploit more.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we extensively investigate our NAS system on three different search spaces to verify its effectiveness.
Revised: In this section, we extensively investigate our NAS system on three different search spaces to verify its effectiveness.

Original: First, we adopt the challenging RandWire search space (Xie et al., 2019) which is significantly larger than common ones.
Revised: First, we adopt the challenging RandWire search space (Xie et al., 2019) which is significantly larger than common ones.

Original: To the best of our knowledge, we are the first to explore learning NAS systems in this space.
Revised: To the best of our knowledge, we are the first to explore learning NAS systems in this space.

Original: Then we search on the ENAS Macro (Pham et al., 2018) and
Revised: Then we search on the ENAS Macro (Pham et al., 2018) and

Original: NAS-Bench-101(Ying et al., 2019) search spaces to further compare with previous literature.
Revised: NAS-Bench-101(Ying et al., 2019) search spaces to further compare with previous literature.

Original: For all experiments, we set the number of mixture Bernoulli K to be 10, the number of message passing steps S to 7, hidden sizes of node representation h s i and message m s ij to 128.
Revised: For all experiments, we set the number of mixture Bernoulli K to be 10, the number of message passing steps S to 7, hidden sizes of node representation h si and message m sij to 128.

Original: For RNN-based baselines, we follow the design in (Zoph et al., 2018) if not other specified.
Revised: For RNN-based baselines, we follow the design in (Zoph et al., 2018) if not other specified.

Original: 4. R AND W IRE S EARCH S PACE ON T INY -I MAGE N ET
Revised: 4. R AND W IRE S EARCH S PACE ON T INY -I MAGE N ET

Original: RandWire Search Space.
Revised: RandWire Search Space.

Original: Originally proposed in (Xie et al., 2019), a randomly wired neural network is a ResNet-like four-stage network with the cell graph G defines the connectivity of N convolution layers within each stage.
Revised: Originally proposed in (Xie et al., 2019), a randomly wired neural network is a ResNet-like four-stage network with the cell graph G defines the connectivity of N convolution layers within each stage.

Original: At the end of each stage, the resolution is downsampled by 3 ×convolution with stride 2 whereas the number of channels is doubled.
Revised: At the end of each stage, the resolution is downsampled by 3 ×convolution with stride 2 whereas the number of channels is doubled.

Original: While following the RandWire small regime in (Xie et al., 2019), we share the cell graph G among last three stages for simplification.
Revised: While following the RandWire small regime in (Xie et al., 2019), we share the cell graph G among last three stages for simplification.

Original: To keep the number of parameters roughly the same, we fix the node type to be separable 3 ×convolution.
Revised: To keep the number of parameters roughly the same, we fix the node type to be separable 3 ×convolution.

Original: The number of nodes N within the cell graph G is set to 32 excluding the input and output nodes.
Revised: The number of nodes N within the cell graph G is set to 32 excluding the input and output nodes.

Original: This yields a search space of 2 . 1 × 10 149 valid adjacency matrices, which is extremely large and renders the neural architecture search challenging.
Revised: This yields a search space of 2 . 1 × 10 149 valid adjacency matrices, which is extremely large and renders the neural architecture search challenging.

Original: More details of the RandWire search space can be found in the Appendix B.
Revised: More details of the RandWire search space can be found in the Appendix E.1.

Original: [Table]

Original: Tiny-ImageNet w. Oracle Evaluator.
Revised: Tiny-ImageNet w. Oracle Evaluator.

Original: To enable search on the RandWire space, we exploit the oracle evaluator on the Tiny-ImageNet dataset (Chrabaszcz et al., 2017).
Revised: To enable search on the RandWire space, we exploit the oracle evaluator on the Tiny-ImageNet dataset (Chrabaszcz et al., 2017).

Original: To save computation, we employ a low-data oracle evaluator where we sample 1 / 10 of Tiny-ImageNet training set for training and use the rest for validation at each search step.
Revised: To save computation, we employ a low-data oracle evaluator where we sample 1 / 10 of Tiny-ImageNet training set for training and use the rest for validation at each search step.

Original: Similar to the few-shot learning, we keep the number of classes unchanged but reduce the number of samples per class.
Revised: Similar to the few-shot learning, we keep the number of classes unchanged but reduce the number of samples per class.

Original: After the search, we retrain our found architectures on the full training set and evaluate it on the original validation set.
Revised: After the search, we retrain our found architectures on the full training set and evaluate it on the original validation set.

Original: Specifically, for each model, the oracle evaluator trains for 300 epochs and uses the average validation accuracy of the last 3 epochs as the reward.
Revised: Specifically, for each model, the oracle evaluator trains for 300 epochs and uses the average validation accuracy of the last 3 epochs as the reward.

Original: Our total search budget is around 16 GPU days, which approximately amounts to 320 model evaluations, e.g ., 40 search steps and 8 samples evaluated per step.
Revised: Our total search budget is around 16 GPU days, which approximately amounts to 320 model evaluations, e.g ., 40 search steps and 8 samples evaluated per step.

Original: For random search baselines, we choose Erd˝os–R´enyi (ER) and Watts–Strogatz (WS) models.
Revised: For random search baselines, we choose Erd˝os–R´enyi (ER) and Watts–Strogatz (WS) models.

Original: Specifically, we first randomly draw hyperparameters from certain ranges, i.e ., 0 .
Revised: Specifically, we first randomly draw hyperparameters from certain ranges, i.e ., 0 .

Original: 1 ≤p ≤ 0 .
Revised: 1 ≤

Original: 5 for ER and (2 , 0 . 2) ≤ ( k, p ) ≤ (6 , 0 . 8) for WS, and then sample G from individual models.
Revised: [Table]

Original: We set the reweight coefficient β to 0.05.
Revised: We set the reweight coefficient β to 0.05.

Original: For the random explorer, we choose WS model with the same hyperparameter range as a prior distribution and set ϵ = 0 .
Revised: For the random explorer, we choose WS model with the same hyperparameter range as a prior distribution and set ϵ = 0 .

Original: 6 in the beginning and decay it by a factor of 0.2 every 10 search steps.
Revised: 6 in the beginning and decay it by a factor of 0.2 every 10 search steps.

Original: We also find that gradually shrinking replay buffer size to keep 30% to 10% of top-performing architectures helps stabilize the training of the generator.
Revised: We also find that gradually shrinking replay buffer size to keep 30% to 10% of top-performing architectures helps stabilize the training of the generator.

Original: At the search time, we reject samples that already appear in the replay buffer to avoid duplications.
Revised: At the search time, we reject samples that already appear in the replay buffer to avoid duplications.

Original: We apply the same setting to the RNN generator for a fair comparison.
Revised: We apply the same setting to the RNN generator for a fair comparison.

Original: Results.
Revised: Results.

Original: As shown in Table 1, we compare our NAS system with other random search methods and learning-to-search methods.
Revised: As shown in Table [Table] we compare our NAS system with other random search methods and learning-to-search methods.

Original: We can see that our method outperforms the RNN-based generator and other random search methods in terms of average validation accuracy on the full dataset.
Revised: We can see that our method outperforms the RNN-based generator and other random search methods in terms of average validation accuracy on the full dataset.

Original: Our generator also has a lower variance compared to the RNN-based one.
Revised: Our generator also has a lower variance compared to the RNN-based one.

Original: Moreover, we observed that
Revised: Moreover, we observed that

Original: RNN based generator sometimes degenerates so that it frequently sample densely-connected graphs.
Revised: RNN based generator sometimes degenerates so that it frequently sample densely-connected graphs.

Original: This is probably due to the fact that RNN based generator does not effectively utilize the topology information.
Revised: This is probably due to the fact that RNN based generator does not effectively utilize the topology information.

Original: We can see that a high search reward ( i.e ., a low-data validation accuracy) do not necessarily lead to better performances in full data training, which indicates a bias of the oracle evaluator within the low-data regime.
Revised: We can see that a high search reward ( i [Table] e ., a low-data validation accuracy) do not necessarily lead to better performances in full data training, which indicates a bias of the oracle evaluator within the low-data regime.

Original: Random search methods are prone to be biased as they select architectures solely based on the search reward.
Revised: Random search methods are prone to be biased as they select architectures solely based on the search reward.

Original: Nevertheless, our generator is less affected by the bias and able to learn a distribution of good architectures that perform well on full data training.
Revised: Nevertheless, our generator is less affected by the bias and able to learn a distribution of good architectures that perform well on full data training

Original: [Table]
Revised: [Table]

Original: We also show results of the best architectures found within 4 samples in Table 2.
Revised: We also show results of the best architectures found within 4 samples in Table 2.

Revised: Here, ER-top-1 and WS-toprefers to the best model found from corresponding random search.

Revised: FC refers to the fully-connected graph, which takes three times longer to train compared to our model.

Original: It is clear that the best model found by our method outperforms those discovered by other methods by a considerable margin.
Revised: It is clear that the best model found by our method outperforms those discovered by other methods by a considerable margin.

Original: Moreover, we scale up the best models (denoted as large) by adding more channels and one more computation stage (more details are in Appendix B.1).
Revised: Moreover, we scale up the best models (denoted as large) by adding more channels and one more computation stage (more details are in Appendix E.1).

Original: We can see that our searched architectures perform favorably against manually designed architectures like ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017).
Revised: We can see that our searched architectures perform favorably against manually designed architectures like ResNet (He et al., 2016) and ResNeXt (Xie et al., 2017).

Original: 4. ENAS M ACRO S EARCH S PACE ON CIFAR 
Revised: 4. ENAS M ACRO S EARCH S PACE ON CIFAR

Original: ENAS Macro Search Space , originally proposed by Pham et al.
Revised: ENAS Macro Search Space , originally proposed by Pham et al.

Original: (2018), is a search space which focuses on the entire network.
Revised: (2018), is a search space which focuses on the entire network.

Original: G here defines the entire network with N = 12 nodes.
Revised: G here defines the entire network with N = 12 nodes.

Original: The operation type ( D =6) 1 per node is also searchable.
Revised: The operation type ( D =6) 1 per node is also searchable.

Original: G is guaranteed to contain a length-11 path, i [Table] e ., ∀ i > 1 ,
Revised: G is guaranteed to contain a length-11 path, i.e ., ∀ i > 1 ,

Original: A i,i − 1 = 1 .
Revised: A i,i − 1 = 1 .

Original: The goal is to search off-diagonal entries, i.e ., skip connections.
Revised: The goal is to search off-diagonal entries, i.e ., skip connections.

Original: This gives a search space of 1 . 6 × 10 29 valid networks in total. 
Revised: This gives a search space of 1 . 6 × 10 29 valid networks in total. [Table] [Table]

Original: SuperNet Evaluator.
Revised: SuperNet Evaluator.

Original: For ENAS Macro search space, we experiment on CIFAR10 (Krizhevsky et al., 2009) dataset.
Revised: For ENAS Macro search space, we experiment on CIFAR10 (Krizhevsky et al., 2009) dataset.

Original: For our generator, we use the ER model with p = 0 .
Revised: For our generator, we use the ER model with p = 0 .

Original: 4 as our explorer, where ϵ decays from 1 to 0 in the first 100 search steps.
Revised: 4 as our explorer, where ϵ decays from 1 to 0 in the first 100 search steps.

Original: For RNN based generator, we follow the setup in (Pham et al., 2018).
Revised: For RNN based generator, we follow the setup in (Pham et al., 2018).

Original: We also adopt the weight-sharing mechanism in (Pham et al., 2018) to obtain a
Revised: We also adopt the weight-sharing mechanism in (Pham et al., 2018) to obtain a

Original: SuperNet evaluator that efficiently evaluates a model’s performance.
Revised: SuperNet evaluator that efficiently evaluates a model’s performance.

Original: We use a budget of 300 search steps with around 100 architectures evaluated per step for all methods.
Revised: We use a budget of 300 search steps with around 100 architectures evaluated per step for all methods.

Original: After the search, we use a short-training of 100 epochs to evaluate the performances of 8 sampled architectures, after which top-4 performing ones are chosen for a 600-epoch full training.
Revised: After the search, we use a short-training of 100 epochs to evaluate the performances of 8 sampled architectures, after which top-4 performing ones are chosen for a 600-epoch full training.

Original: The best validation error rate among these 4 architectures is reported.
Revised: The best validation error rate among these 4 architectures is reported.

Original: For simplicity and a fair comparison, we do not use additional tricks ( e.g ., adding entropy regularizer) in (Pham et al., 2018).
Revised: For simplicity and a fair comparison, we do not use additional tricks ( e.g ., adding entropy regularizer) in (Pham et al., 2018).

Original: More details are provided in Appendix C.
Revised: More details are provided in Appendix F.

Original: In Table 3, we compare the error rates and variances for different NAS methods.
Revised: In Table 3, we compare the error rates and variances for different NAS methods.

Original: Note that this variance reflects the uncertainty of the distribution of architectures as it is computed based on sampled architectures.
Revised: Note that this variance reflects the uncertainty of the distribution of architectures as it is computed based on sampled architectures.

Original: It is clear that our GraphPNAS achieves both lower error rates and lower variances compared to RNN based generator and is on par with the state-of-the-art NAS methods on other search spaces.
Revised: It is clear that our GraphPNAS achieves both lower error rates and lower variances compared to RNN based generator and is on par with the state-of-the-art NAS methods on other search spaces.

Original: We also see that the best architecture performance of our generator outperforms
Revised: We also see that the best architecture performance of our generator outperforms

Original: [Table]

Original: [Table]

Original: RNN based generator by a significant margin.
Revised: RNN based generator by a significant margin.

Original: This verifies that our GraphPNAS is able to learn a distribution of well-performing neural architectures.
Revised: This verifies that our GraphPNAS is able to learn a distribution of well-performing neural architectures.

Original: Given that we only sample 8 architectures, the performances could be further improved with more computational budgets.
Revised: Given that we only sample 8 architectures, the performances could be further improved with more computational budgets.

Original: 4. NAS-B ENCH -
Revised: 4. NAS B ENCHMARKS

Original: NAS-Bench-101 (Ying et al., 2019) is a tabulate benchmark containing 423 K cell graphs, each of which is a DAG with up to 7 nodes and 9 edges including input and output nodes.
Revised: NAS-Bench-101 (Ying et al., 2019) is a tabulate benchmark containing 423 K cell graphs, each of which is a DAG with up to 7 nodes and 9 edges including input and output nodes.

Original: We compare the performances of our GraphPNAS to open-source implementations of random search methods, local search methods, and BANANAS (White et al., 2019).
Revised: We compare the performances of our GraphPNAS to open-source implementations of random search methods, local search methods, and BANANAS (White et al., 2019).

Original: The latter two are the best algorithms found by White et al.
Revised: The latter two are the best algorithms found by White et al.

Original: (2020) on NAS-Bench-101.
Revised: (2020) on NAS-Bench-101.

Original: ForRNN based generator, GCN prediction and evolution methods, we use the score reported in (White et al., 2020).
Revised: For GCN prediction and evolution methods, we use the score reported in (White et al., 2020).

Original: We give each NAS method the same budget of 300 queries and plot the curve of lowest test error as a function of the number of evaluated architectures.
Revised: We give each NAS method the same budget ofqueries and plot the curve of lowest test error as a function of the number of evaluated architectures.

Original: As shown in Fig.
Revised: As shown in Fig.

Original: 2, our GraphPNAS is able to quickly find well-performing architectures.
Revised: 2, our GraphPNAS is able to quickly find well-performing architectures.

Original: We also report the avg error rate over 10 runs in Table 4.
Revised: We also report the avg error rate over 10 runs in Table 4.

Original: Our GraphPNAS again outperforms RNN  based generator by a significant margin and beats strong baselines like local search methods and BANANAS.
Revised: Our GraphPNAS again outperforms RNN [Figure] n t based generator by a significant margin and beats strong baselines like local search methods and BANANAS.

Original: Notably, our GraphPNAS has a much lower variance than other methods, thus being more stable across multiple runs.
Revised: Notably, our GraphPNAS has a much lower variance than other methods, thus being more stable across multiple runs.

Revised: NAS-Bench-201 (Dong & Yang, 2020) is defined on a smaller search space where up to 4 nodes and 6 edges is allowed, results for can be found in section A.

Original:  D ISCUSSION & C ONCLUSION
Revised:  D ISCUSSION & C ONCLUSION

Original: Qualitative Comparisons between RNN and GraphPNAS.
Revised: Qualitative Comparisons between RNN and GraphPNAS.

Original: In (You et al., 2020), the clustering coefficient and the average path length have been used to investigate distributions of graphs.
Revised: In (You et al., 2020), the clustering coefficient and the average path length have been used to investigate distributions of graphs.

Original: Here we adopt the same metrics to visualize architectures (graphs) sampled by RNN based and our generators in RandWire experiments.
Revised: Here we adopt the same metrics to visualize architectures (graphs) sampled by RNN based and our generators in RandWire experiments.

Original: Points in Fig.
Revised: Points in Fig.

Original: 3 refer to architectures sampled from both generators in the last 15 search steps where random explorers are disabled.
Revised: 3 refer to architectures sampled from both generators in the last 15 search steps where random explorers are disabled.

Original: The validation performances are color-coded.
Revised: The validation performances are color-coded.

Original: We can see that our GraphPNAS samples a set of graphs that have better validation accuracies while the ones of RNN generator have large variances in performances.Moreover, the [Figure] C l u s t e r i n g
Revised: We can see that our GraphPNAS samples a set of graphs that have better validation accuracies while the ones of RNN generator have large variances in performances.

Original: C o e ff i c i e n t graphs in our case concentrate to those with smaller clustering coefficients, thus less likely being densely-connected.
Revised: Moreover, the graphs in our case concentrate to those with smaller clustering coefficients, thus less likely being densely-connected.

Original: On the contrary, RNN generator tends to sample graphs that are more likely to be densely-connected.
Revised: On the contrary, RNN generator tends to sample graphs that are more likely to be densely-connected.

Original: While RNN has been widely used for NAS, we show in our experiments that our graph generator consistently outperforms RNN over three search spaces on two different datasets.
Revised: While RNN has been widely used for NAS, we show in our experiments that our graph generator consistently outperforms RNN over three search spaces on two different datasets.

Original: This is likely due to the fact that our graph generator better leverages graph topologies, thus being more expressive in learning the distribution of graphs.
Revised: This is likely due to the fact that our graph generator better leverages graph topologies, thus being more expressive in learning the distribution of graphs.

Original: Bias in Evaluator.
Revised: Bias in Evaluator.

Original: In our experiments, we use SuperNet evaluator, low-data, and full-data Oracle evaluator to efficiently evaluate the model.
Revised: In our experiments, we use SuperNet evaluator, low-data, and full-data Oracle evaluator to efficiently evaluate the model.

Original: From the computational efficiency perspective, one would prefer the SuperNet evaluator.
Revised: From the computational efficiency perspective, one would prefer the SuperNet evaluator.

Original: However, it tends to give high rewards to those architectures used for training SuperNet.
Revised: However, it tends to give high rewards to those architectures used for training SuperNet.

Original: Although the low-data evaluator is more efficient than the full-data one, its reward is biased as discussed in Section 4.1.
Revised: Although the low-data evaluator is more efficient than the full-data one, its reward is biased as discussed in Section 4.1.

Original: This bias is caused by the discrepancy between the data distributions in low-data and full-data regimes.
Revised: This bias is caused by the discrepancy between the data distributions in low-data and full-data regimes.

Original: We also tried to follow (Tan et al., 2019) to use early stopping to reduce the time cost of the full-data evaluator.
Revised: We also tried to follow (Tan et al., 2019) to use early stopping to reduce the time cost of the full-data evaluator.

Original: However, we found that it assigns higher rewards to those shallow networks which converge much faster in the early stage of training.
Revised: However, we found that it assigns higher rewards to those shallow networks which converge much faster in the early stage of training.

Original: We show detailed results in Appendix B.4.
Revised: We show detailed results in Appendix E.4.

Original: Search Space Design.
Revised: Search Space Design.

Original: The design of search space largely affects the performances of NAS methods.
Revised: The design of search space largely affects the performances of NAS methods.

Original: Our GraphPNAS successfully learns good architectures on the challenging RandWire search space.
Revised: Our GraphPNAS successfully learns good architectures on the challenging RandWire search space.

Original: However, the search space is still limited as the cell graph across different stages is shared.
Revised: However, the search space is still limited as the cell graph across different stages is shared.

Original: A promising direction is to learn to generate graphs in a hierarchical fashion.
Revised: A promising direction is to learn to generate graphs in a hierarchical fashion.

Original: For example, one can first generate a macro graph and then generate individual cell graphs (each cell is a node in the macro graph) conditioned on the macro graph.
Revised: For example, one can first generate a macro graph and then generate individual cell graphs (each cell is a node in the macro graph) conditioned on the macro graph.

Original: This will significantly enrich the search space by including the macro graph and untying cell graphs.
Revised: This will significantly enrich the search space by including the macro graph and untying cell graphs.

Original: Conclusion.
Revised: Conclusion.

Original: In this paper, we propose a GNN-based graph generator for NAS, called GraphPNAS.
Revised: In this paper, we propose a GNN-based graph generator for NAS, called GraphPNAS.

Original: Our graph generator naturally captures topologies and dependencies of operations of wellperforming neural architectures.
Revised: Our graph generator naturally captures topologies and dependencies of operations of wellperforming neural architectures.

Original: It can be learned efficiently through reinforcement learning.
Revised: It can be learned efficiently through reinforcement learning.

Original: We extensively study its performances on the challenging RandWire as well as two widely used search spaces.
Revised: We extensively study its performances on the challenging RandWire as well as two widely used search spaces.

Original: Experimental results show that our GraphPNAS consistently outperforms RNN-based generator on all datasets.
Revised: Experimental results show that our GraphPNAS consistently outperforms RNN-based generator on all datasets.

Original: Future works include exploring ensemble methods based on our GraphPNAS and hierarchical graph generation on even larger search spaces.
Revised: Future works include exploring ensemble methods based on our GraphPNAS and hierarchical graph generation on even larger search spaces.

Original: A M ORE DETAILS ON G RAPH G ENERATOR
Revised: A E XPERIMENTS ON THE NAS-B ENCH -201 S EARCH S PACE

Revised: Here we compare our method on NAS-Bench-201(Dong & Yang, 2020) with random search (RANDOM) Bergstra & Bengio (2012), random search with parameter sharing (RSPS) Li & Talwalkar (2020), REA Real et al. (2019b), REINFORCE Williams (1992), ENAS Pham et al. (2018), first order DARTS (DARTS 1st)

Revised: Liu et al.

Revised: (2018b), second order DARTS (DARTS 2nd),

Revised: GDAS Dong & Yang (2019c), SETN Dong & Yang (2019b), TAS Dong & Yang (2019a), FBNet-V2 Wan et al.

Revised: (2020), TuNAS Bender et al.

Revised: (2020), BOHB Falkner et al.

Revised: [Table]

Revised: To fairly compare with scores reported in (Dong et al., 2021)

Revised: [Table]

Revised: we fix a search budget of 20000s for cifar10 and cifar

Revised: [Table]

Revised: and 30000s for ImangeNet-16-120, which is approximately 150, 80,oracle evaluations on cifar10, cifar100 and ImageNet-16-

Revised: [Table]

Revised: with 1 sample being evaluated per step.

Revised: Specifically for NAS-Bench-201, we use random explorer in the first 10 steps and keep the top 15 architectures in the replay buffer.

Revised: We found that our model is able to outperform previous methods on cifar10 and cifar100 datasets and is on par with previous methods on the ImageNet-16120 dataset.

Revised: Interestingly, for ImageNet-16-120, we are able to largely boost the performance with 45.57(+0.57) and 45.79(+0.4) on validation and test set respectively when we extend the number of search budget from 40 to 60 steps.

Revised: This probably indicates that the search process with our model hasn’t converged due to the limited search steps.

Revised: This suggests that a reasonable number of search steps is needed for our model to fully show the potential.

Revised: B C OMPARISON WITH NAGO

Revised: Following Xie et al.

Revised: (2019)’s work on random graph models, Ru et al.

Revised: (2020) propose to learn the parameter for random graph models with bayesian optimization.

Revised: We compare with (Ru et al., 2020)

Revised: the original randwire search space (refers to as RANG in (Ru et al., 2020)).

Revised: Since the original search space in (Xie et al., 2019) do not reuse cell graphs for different stages, we train conditionally independent graph generators for different stages respectively.

Revised: That is 3 conditionally independent generators for conv 3 , conv 4 , and conv 5 stage in table 9.

Revised: We perform a search on the cifar10 dataset, where each model is evaluated for 100 epochs.

Revised: We restrict the search budget to 600 oracle evaluations.

Revised: We align with settings in (Ru et al., 2020) for retraining and report sampled architecture’s test accuracy and standard deviation in the table 6.

Revised: We can see that our method is able to learn a distribution of graphs that outperforms previous methods.

Revised: C E XPERIMENTS ON THE D ARTS S EARCH S PACE

Revised: DARTS is a small cell-graph-based search space (Liu et al., 2018b).

Revised: It is defined on a DAG with up to 6 nodes and 8 possible operators.

Revised: Each node can have no more than two inputs.

Revised: Like NASBench-201, node operations are defined on the edge where nodes represent fuse operation between

Revised: [Table]

Revised: incoming edges.

Revised: Desipte the fact that search on cifar10 dataset with darts is already very saturated, we experiment on it to compare with preivous NAS methods.

Revised: We split the CIFAR-10 dataset into subsets containing 40000 training and 10000 validation samples, and, maintaining consistency with

Revised: DARTS, train candidate architectures with SGD for 50 epochs with an initial learning rate of 0 .(annealed to 0 via cosine decay), momentum of 0 .

Revised: 9 , weight decay of 0 .

Revised: 0003 , as well as auxiliary towers, path, and cutout.

Revised: We allow a budget of 96 oracle evaluations, or roughly 72 hours on

Revised: We follow the evaluation pipeline annealed to decay, momentum decay

Revised: We also perform augmentation

Revised: [Table]

Revised: From table 7 we can see that our method is on par with the state-of-art methods.

Revised: The computation cost is majorly due to the fact that oracle evaluation is used, where future works using a performance predictor or supernet-based evaluator can be easily exploited to reduce the computation overhead.

Revised: Due to constrain on available computation, we reuse the best search hyperparameters found in nasbench-101, we expect further hyperparameter tuning would further boost the performance.

Revised: We also provide search results on CIFAR 100 dataset.

Revised: Due to time constraints, we only compare with a few open-source NAS methods.

Revised: For a fair comparison, we reuse the final training hyperparameters from

Original:  To more clearly illustrate the sampling process of our generator, we detailed probabilistic sampling process of our generator in Fig.
Revised: Chen et al. [Table] D M ORE DETAILS ON G RAPH G ENERATOR To more clearly illustrate the sampling process of our generator, we detailed probabilistic sampling process of our generator in Fig.

Original: B M ORE DETAILS ON THE R ANDWIRE EXPERIMENTS B. D ETAILS OF R AND W IRE SEARCH SPACE
Revised: E M ORE DETAILS ON THE R ANDWIRE EXPERIMENTS E. D ETAILS OF R AND W IRE SEARCH SPACE

Original: Here we provided more details on the RandWire search space shown in Table 5 and Fig. 5.
Revised: Here we provided more details on the RandWire search space shown in Table 9 and Fig. 5.

Original: [Table]
Revised: [Table]

Original: B. D ETAILS FOR R AND W IRE EXPERIMENTS
Revised: E. D ETAILS FOR R AND W IRE EXPERIMENTS

Original: For experiment on Tiny-Imagenet, we resize image to 224 × 224 as showed in Table 5.
Revised: For experiment on Tiny-Imagenet, we resize image to 224 × 224 as showed in Table 9.

Original: We apply the basic data augmentation of horizontal random flip and random cropping with padding size 4.
Revised: We apply the basic data augmentation of horizontal random flip and random cropping with padding size 4.

Original: We provide detailed hyper-parameters for oracle evaluator training and learning for GraphPNAS in
Revised: We provide detailed hyper-parameters for oracle evaluator training and learning for GraphPNAS in

Original: Table
Revised: Table

Original: B. V ISUALIZATION OF ARCHITECTURES FROM OUR GENERATOR
Revised: E. V ISUALIZATION OF ARCHITECTURES FROM OUR GENERATOR

Original: Here we visualize the top candidate architectures in Fig.
Revised: Here we visualize the top candidate architectures in Fig.

Original: [Figure]
Revised: [Figure]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: graph batch size
Revised: batch size

Original: [Table]
Revised: [Table]

Original: generator
Revised: generator

Original: [Table]
Revised: [Table]

Original: rate

Original: [Table]

Original: [Table]
Revised: [Table]

Original: B. B IAS FOR EARLY STOPPING
Revised: E. B IAS FOR EARLY STOPPING

Original: As discussed in Section 5, using early stopping will lead to local minimal where the generator learns to generate shallow cell structure.
Revised: As discussed in Section 5, using early stopping will lead to local minimal where the generator learns to generate shallow cell structure.

Original: We quantify this phenomenon in table 7, where we can see that with early stopping training, the generator will generate more shallow architectures with a shorter path from input to output.
Revised: We quantify this phenomenon in table 11, where we can see that with early stopping training, the generator will generate more shallow architectures with a shorter path from input to output.

Original: The corresponding average final validation accuracy also dropped by a large margin compared to the low data evaluator counter part.
Revised: The corresponding average final validation accuracy also dropped by a large margin compared to the low data evaluator counter part.

Original: [Table]
Revised: [Table]

Original:  average Path and  are computed as the average path length and longest path  from 
Revised: The average Path and Longest path are computed as the average path length and longest path length from from the corresponding generator.

Original: C M ORE DETAILS ON ENAS M ACRO EXPERIMENTS
Revised: F M ORE DETAILS ON ENAS M ACRO EXPERIMENTS

Original: For ENAS Macro search space, we use a pytorch-based open source implementation 2 and follow the detailed parameters provided in (Pham et al., 2018) for RNN generator.
Revised: For ENAS Macro search space, we use a pytorch-based open source implementation 2 and follow the detailed parameters provided in (Pham et al., 2018) for RNN generator.

Original: Specifically, we follow (Pham et al., 2018) to train the SuperNet and update  generator in an iterative style.
Revised: Specifically, we follow (Pham et al., 2018) to train the SuperNet and update the generator in an iterative style.

Original: At each search step, two sets of samples G train and G eval are sampled from the generator.
Revised: At each search step, two sets of samples G train and G eval are sampled from the generator.

Original: G train is used to learn the SuperNet’s weights by back-propagating the training loss.
Revised: G train is used to learn the SuperNet’s weights by back-propagating the training loss.

Original: The updated SuperNet is used for evaluating G eval , which is then used for updating the generator.
Revised: The updated SuperNet is used for evaluating G eval , which is then used for updating the generator.

Original: For our generator, we evaluate 100 architecture per step and update our generator every 5 epochs of SuperNet training.
Revised: For our generator, we evaluate 100 architectures per step and update our generator every 5 epochs of SuperNet training.

Original: Instead of evaluating on a single batch, we reduce the number of models
Revised: Instead of evaluating on a single batch, we reduce the number of models

Original: Correspond to  report in evaluated per step and evaluate on the full test set.
Revised: Correspond to scores report in evaluated per step and evaluate on the full test set.

Original: We found this stables the training of our generator while keeping 
Revised: We found this stables the training of our generator while keeping evaluation costs the same the buffer

Original: For training SuperNet and RNN generator, we follow the same hyper-parameter setting in (Pham et al., 2018) except  learning rate decay policy is changed to cosine learning rate decay.
Revised: For training SuperNet and RNN generator, we follow the same hyper-parameter setting in (Pham et al., 2018) except the learning rate decay policy is changed to cosine learning rate decay.

Original: For training our generator we use the same hyperparameter as in Table 6 with graph batch size changed to 32.
Revised: For training our generator we use the same hyperparameter as in Table 10 with graph batch size changed to 32.

Original: For retraining the found the best architecture, we use a budget of 600 epoch training with a learning rate of 0.1, batch size 256, and weight decay 2e-4.
Revised: For retraining the found the best architecture, we use a budget of 600 epoch training with a learning rate of 0.1, batch size 256, and weight decay 2e-4.

Original: We also apply a cutout with a probability of 0. to all the models when retraining the model.
Revised: We also apply a cutout with a probability of 0.4 to all the models when retraining the model.

Original: C. V ISUALIZATION OF BEST ARCHITECTURE FOUND
Revised: F. V ISUALIZATION OF BEST ARCHITECTURE FOUND

Original: Here we visualize the best architecture found by GraphPNAS and RNN generator for Enas Macro search space in Fig.
Revised: Here we visualize the best architecture found by GraphPNAS and RNN generator for Enas Macro search space in Fig.

Original: D M ORE DETAILS ON NAS-B ENCH - 
Revised: G M ORE DETAILS ON NAS-B ENCH -

Original: For sampling on NAS-Bench-101 (Ying et al., 2019), we first sample a 7-node DAG, then we remove any node that is not connected to the input or the output node.
Revised: For sampling on NAS-Bench-101 (Ying et al., 2019), we first sample a 7-node DAG, then we remove any node that is not connected to the input or the output node.

Original: We reject samples that have more than 9 edges or don’t have a path from input to output.
Revised: We reject samples that have more than 9 edges or don’t have a path from input to output.

Original: To train our generator on Nas-Bench-101, we use Erd˝os–R´enyi with p = 0 .
Revised: To train our generator on Nas-Bench-101, we use Erd˝os–R´enyi with p = 0 .

Original: 25 , ϵ is set to 1 in the beginning and decreased to 0 after 30 search steps.
Revised: 25 , ϵ is set to 1 in the beginning and decreased to 0 after 30 search steps.

Original: For the replay buffer, we keep the toparchitectures.
Revised: For the replay buffer, we keep the toparchitectures.

Original: Our model is updated every 10 model evaluations, where we train 70 epochs on the replay buffer at each update time.
Revised: Our model is updated every 10 model evaluations, where we train 70 epochs on the replay buffer at each update time.

Original: The learning rate is set to 1e-3 for with a batch size of 2 on
Revised: The learning rate is set to 1e-3 for with a batch size of 2 on

Original: Nas-bench-101.
Revised: Nas-bench-101.

Revised: per