Original: R APID N EURAL A RCHITECTURE S EARCH BY L EARNING TO G ENERATE G RAPHS FROM D ATASETS
Revised: R APID N EURAL A RCHITECTURE S EARCH BY L EARNING TO G ENERATE G RAPHS FROM D ATASETS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform humandesigned networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets).
Revised: Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform humandesigned networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets).

Original: Moreover, since such task-speciﬁc methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited.
Revised: Moreover, since such task-speciﬁc methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited.

Original: In this paper, we propose an efﬁcient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search a neural architecture for a novel dataset.
Revised: In this paper, we propose an efﬁcient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search a neural architecture for a novel dataset.

Original: The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning.
Revised: The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning.

Original: Moreover, we also propose a metaperformance predictor to estimate and select the best architecture from those sampled from MetaD2A.
Revised: Moreover, we also propose a metaperformance predictor to estimate and select the best architecture from those sampled from MetaD2A.

Original: The experimental results demonstrate that our model metalearned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple benchmark datasets including CIFAR10 and CIFAR-100, with the search time of less than 30 GPU seconds on CIFAR10.
Revised: The experimental results demonstrate that our model metalearned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple benchmark datasets including CIFAR10 and CIFAR-100, with the search time of less than 30 GPU seconds on CIFAR10.

Original: We believe that the MetaD2A proposes a new research direction for rapid
Revised: We believe that the MetaD2A proposes a new research direction for rapid

Original: NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years.
Revised: NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The rapid progress in the design of neural architectures has largely contributed to the success of deep learning on many applications (Krizhevsky et al., 2012; Cho et al., 2014;
Revised: The rapid progress in the design of neural architectures has largely contributed to the success of deep learning on many applications (Krizhevsky et al., 2012; Cho et al., 2014;

Original: He et al., 2016; Szegedy et al.; Vaswani et al., 2017; Zhang et al., 2018).
Revised: He et al., 2016; Szegedy et al.; Vaswani et al., 2017; Zhang et al., 2018).

Original: However, due to the vast search space, designing a novel neural architecture requires a time-consuming trial-and-error search by human experts.
Revised: However, due to the vast search space, designing a novel neural architecture requires a time-consuming trial-and-error search by human experts.

Original: To tackle such inefﬁciency in the manual architecture design process, researchers have proposed various Neural Architecture Search (NAS) methods that automatically search for optimal architectures, achieving models with impressive performances on various tasks that outperform human-designed counterparts (Baker et al., 2017; Zoph & Le, 2017; Kandasamy et al., 2018; Liu et al., 2018a;
Revised: To tackle such inefﬁciency in the manual architecture design process, researchers have proposed various Neural Architecture Search (NAS) methods that automatically search for optimal architectures, achieving models with impressive performances on various tasks that outperform human-designed counterparts (Baker et al., 2017; Zoph & Le, 2017; Kandasamy et al., 2018; Liu et al., 2018a;

Original: Luo et al., 2018; Pham et al., 2018; Liu et al., 2019; Xu et al., 2020; Chen et al., 2020).
Revised: Luo et al., 2018; Pham et al., 2018; Liu et al., 2019; Xu et al., 2020; Chen et al., 2020).

Original: Recently, large benchmarks for NAS (NAS-101, NAS-201)
Revised: Recently, large benchmarks for NAS (NAS-101, NAS-201)

Original: (Ying et al., 2019; Dong & Yang, 2020) have been introduced, which provide databases of architectures and their performances on benchmark datasets.
Revised: (Ying et al., 2019; Dong & Yang, 2020) have been introduced, which provide databases of architectures and their performances on benchmark datasets.

Original: Yet, most conventional NAS methods cannot beneﬁt from the availability of such databases, due to their task-speciﬁc nature which requires repeatedly training the model from scratch for each new dataset (See Figure 1 (a)).
Revised: Yet, most conventional NAS methods cannot beneﬁt from the availability of such databases, due to their task-speciﬁc nature which requires repeatedly training the model from scratch for each new dataset (See Figure 1 (a)).

Original: Thus, searching for an architecture for a new task (dataset) may require a large number of computations, which may be problematic when the time and monetary budget are limited.
Revised: Thus, searching for an architecture for a new task (dataset) may require a large number of computations, which may be problematic when the time and monetary budget are limited.

Original: How can we then exploit the vast knowledge of neural architectures that have been already trained on a large number of datasets, to better generalize over an unseen task?
Revised: How can we then exploit the vast knowledge of neural architectures that have been already trained on a large number of datasets, to better generalize over an unseen task?

Original: To this end, we propose an efﬁcient NAS framework that is trained once from a database containing datasets and their corresponding neural architectures and then generalizes to multiple datasets for searching neural architectures, by learning to generate a neural architecture from a given dataset.
Revised: To this end, we propose an efﬁcient NAS framework that is trained once from a database containing datasets and their corresponding neural architectures and then generalizes to multiple datasets for searching neural architectures, by learning to generate a neural architecture from a given dataset.

Original: The proposed MetaD2A (Meta Dataset-to-Architecture) framework consists of a set encoder and a graph
Revised: The proposed MetaD2A (Meta Dataset-to-Architecture) framework consists of a set encoder and a graph

Original: [Figure]
Revised: [Figure]

Original: decoder, which are used to learn a cross-modal latent space for datasets and neural architectures via amortized inference.
Revised: decoder, which are used to learn a cross-modal latent space for datasets and neural architectures via amortized inference.

Original: For a new dataset, MetaD2A stochastically generates neural architecture candidates from set-dependent latent representations which are encoded from a new dataset and selects the ﬁnal neural architecture based on their accuracies predicted by a performance predictor, which is also trained with amortized meta-learning (See Figure 1 (c))
Revised: For a new dataset, MetaD2A stochastically generates neural architecture candidates from set-dependent latent representations which are encoded from a new dataset and selects the ﬁnal neural architecture based on their accuracies predicted by a performance predictor, which is also trained with amortized meta-learning (See Figure 1 (c))

Original: [Figure]
Revised: [Figure]

Original: Several Meta-NAS methods (Lian et al., 2019; Elsken et al., 2020; Shaw et al., 2019) have studied the transferability of NAS to unseen datasets based on gradient-based meta-learning framework.
Revised: Several Meta-NAS methods (Lian et al., 2019; Elsken et al., 2020; Shaw et al., 2019) have studied the transferability of NAS to unseen datasets based on gradient-based meta-learning framework.

Original: However, their expensive multiple unrolling gradient steps for one meta-update of each task require large computational cost, which limits the applications of those Meta-NAS methods to small scale tasks such as few-shot classiﬁcation (See Figure 1 (b)).
Revised: However, their expensive multiple unrolling gradient steps for one meta-update of each task require large computational cost, which limits the applications of those Meta-NAS methods to small scale tasks such as few-shot classiﬁcation (See Figure 1 (b)).

Original: On the other hand, the proposed MetaD2A can output a set-dependent neural architecture for a large dataset, through an amortized inference which encodes the dataset as a latent embedding and decodes it as a directed acyclic graph.
Revised: On the other hand, the proposed MetaD2A can output a set-dependent neural architecture for a large dataset, through an amortized inference which encodes the dataset as a latent embedding and decodes it as a directed acyclic graph.

Original: We meta-learn the proposed MetaD2A on MetaImageNet, which is compiled to contain subsets of
Revised: We meta-learn the proposed MetaD2A on MetaImageNet, which is compiled to contain subsets of

Original: ImageNet-1K, and neural architectures from the NAS-Bench201 search space.
Revised: ImageNet-1K, and neural architectures from the NAS-Bench201 search space.

Original: Then we validate it on the architecture search problems for unseen subsets of ImageNet-1K. On this experiment, our model obtains a neural architecture largely outperforming all baselines for a given task within 27.03 GPU seconds on average.
Revised: Then we validate it on the architecture search problems for unseen subsets of ImageNet-1K. On this experiment, our model obtains a neural architecture largely outperforming all baselines for a given task within 27.03 GPU seconds on average.

Original: Further, our model meta-learned on MetaImageNet can generalize to search for neural architectures on multiple unseen datasets such as MNIST, SVHN, CIFAR-10,
Revised: Further, our model meta-learned on MetaImageNet can generalize to search for neural architectures on multiple unseen datasets such as MNIST, SVHN, CIFAR-10,

Original: CIFAR-100, Aircraft, and Oxford-IIIT Pets.
Revised: CIFAR-100, Aircraft, and Oxford-IIIT Pets.

Original: The results show that our model achieves state-of-theart performance on all datasets.
Revised: The results show that our model achieves state-of-theart performance on all datasets.

Original: Moreover, our model can search for a neural architecture within 22.47 GPU seconds on average, on CIFAR10.
Revised: Moreover, our model can search for a neural architecture within 22.47 GPU seconds on average, on CIFAR10.

Original: To summarize, our contribution in this work is threefold:
Revised: To summarize, our contribution in this work is threefold:

Original: • We propose a novel NAS framework, MetaD2A, which rapidly searches for a neural architecture on a new dataset, by sampling architectures from latent embeddings of the given dataset, then selecting the best one based on their predicted performances.
Revised: • We propose a novel NAS framework, MetaD2A, which rapidly searches for a neural architecture on a new dataset, by sampling architectures from latent embeddings of the given dataset, then selecting the best one based on their predicted performances.

Original: • To this end, we propose to learn a cross-modal latent space of datasets and architectures, by performing amortized meta-learning of it using a set encoder and a graph decoder on subsets of ImageNet-1K. •
Revised: • To this end, we propose to learn a cross-modal latent space of datasets and architectures, by performing amortized meta-learning of it using a set encoder and a graph decoder on subsets of ImageNet-1K. •

Original: The meta-learned our model successfully searches for neural architectures on multiple unseen datasets and achieves
Revised: The meta-learned our model successfully searches for neural architectures on multiple unseen datasets and achieves

Original: state-of-the-art performance on them in NAS-Bench201 search space, especially searching for architectures within 22.47 GPU seconds on CIFAR10.
Revised: state-of-the-art performance on them in NAS-Bench201 search space, especially searching for architectures within 22.47 GPU seconds on CIFAR10.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Architecture Search (NAS) NAS is an automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space for optimal neural architectures.
Revised: Neural Architecture Search (NAS) NAS is an automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space for optimal neural architectures.

Original: NAS methods can be roughly categorized into reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018;
Revised: NAS methods can be roughly categorized into reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018;

Original: Pham et al., 2018), evolutionary algorithm-based methods (Real et al., 2019), and gradient-based methods (Liu et al., 2019; Cai et al., 2019; Luo et al., 2018; Dong & Yang, 2019b; Chen et al., 2020; Xu et al., 2020; Fang et al., 2020).
Revised: Pham et al., 2018), evolutionary algorithm-based methods (Real et al., 2019), and gradient-based methods (Liu et al., 2019; Cai et al., 2019; Luo et al., 2018; Dong & Yang, 2019b; Chen et al., 2020; Xu et al., 2020; Fang et al., 2020).

Original: [Figure]
Revised: [Figure]

Original: Among existing approaches, perhaps the most relevant approach to ours is NAO (Luo et al., 2018), which maps DAGs onto a continuous latent embedding space.
Revised: Among existing approaches, perhaps the most relevant approach to ours is NAO (Luo et al., 2018), which maps DAGs onto a continuous latent embedding space.

Original: However, while NAO performs graph reconstruction for a single task, ours generates data-dependent Directed Acyclic Graphs (DAGs) across multiple tasks.
Revised: However, while NAO performs graph reconstruction for a single task, ours generates data-dependent Directed Acyclic Graphs (DAGs) across multiple tasks.

Original: Besides NAO, FNA (Fang et al., 2020) focuses on reducing the adaptation time by using parameter remapping from the pre-trained network, while ours reduces the search time by generating graphs via latent space.
Revised: Besides NAO, FNA (Fang et al., 2020) focuses on reducing the adaptation time by using parameter remapping from the pre-trained network, while ours reduces the search time by generating graphs via latent space.

Original: Another important open problem in NAS is reducing the tremendous computational cost resulting from the large search space (Cai et al., 2019; Liu et al., 2018a; Pham et al., 2018; Liu et al., 2019; Chen et al., 2020).
Revised: Another important open problem in NAS is reducing the tremendous computational cost resulting from the large search space (Cai et al., 2019; Liu et al., 2018a; Pham et al., 2018; Liu et al., 2019; Chen et al., 2020).

Original: GDAS (Dong & Yang, 2019b) tackles this by optimizing sampled sub-graphs of DAG.
Revised: GDAS (Dong & Yang, 2019b) tackles this by optimizing sampled sub-graphs of DAG.

Original: PC-DARTS
Revised: PC-DARTS

Original: (Xu et al., 2020) reduces GPU overhead and search time by partially selecting channel connections.
Revised: (Xu et al., 2020) reduces GPU overhead and search time by partially selecting channel connections.

Original: However, due to the task-speciﬁc nature of those methods, they should train a model from the scratch for each new unseen task repeatably and each will take a few GPU hours.
Revised: However, due to the task-speciﬁc nature of those methods, they should train a model from the scratch for each new unseen task repeatably and each will take a few GPU hours.

Original: Meta-learning Meta-learning (learning to learn) aims to train a model to generalize over a distribution of tasks, such that it can rapidly adapt to a new task (Vinyals et al., 2016;
Revised: Meta-learning Meta-learning (learning to learn) aims to train a model to generalize over a distribution of tasks, such that it can rapidly adapt to a new task (Vinyals et al., 2016;

Original: Snell et al., 2017;
Revised: Snell et al., 2017;

Original: Finn et al., 2017; Nichol et al., 2018;
Revised: Finn et al., 2017; Nichol et al., 2018;

Original: Lee et al., 2019b;
Revised: Lee et al., 2019b;

Original: Hou et al., 2019).
Revised: Hou et al., 2019).

Original: Recently, LEO (Rusu et al., 2019) proposed a scalable meta-learning framework which learns the latent generative representations of model parameters for a given data in low-dimensional space for few-shot classiﬁcation.
Revised: Recently, LEO (Rusu et al., 2019) proposed a scalable meta-learning framework which learns the latent generative representations of model parameters for a given data in low-dimensional space for few-shot classiﬁcation.

Original: Similarly to LEO (Rusu et al., 2019), our method learns a low-dimensional latent embedding space, but we learn a cross-modal space for both datasets and models for task-dependent model generation.
Revised: Similarly to LEO (Rusu et al., 2019), our method learns a low-dimensional latent embedding space, but we learn a cross-modal space for both datasets and models for task-dependent model generation.

Original: Neural Architecture Search with Meta-Learning Recent NAS methods with gradient-based meta-learning (Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) have shown promising results on adapting to different tasks.
Revised: Neural Architecture Search with Meta-Learning Recent NAS methods with gradient-based meta-learning (Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) have shown promising results on adapting to different tasks.

Original: However, they are only applicable on small scale tasks such as few-shot classiﬁcation tasks (Elsken et al., 2020; Lian et al., 2019) and require high-computation time, due to the multiple unrolling gradient steps for one meta-update of each task.
Revised: However, they are only applicable on small scale tasks such as few-shot classiﬁcation tasks (Elsken et al., 2020; Lian et al., 2019) and require high-computation time, due to the multiple unrolling gradient steps for one meta-update of each task.

Original: While some attempt to bypass the bottleneck with a ﬁrst-order approximation (Lian et al., 2019; Shaw et al., 2019) or parallel computations with GPUs (Shaw et al., 2019), but their scalability is intrinsically limited due to gradient updates over a large number of tasks.
Revised: While some attempt to bypass the bottleneck with a ﬁrst-order approximation (Lian et al., 2019; Shaw et al., 2019) or parallel computations with GPUs (Shaw et al., 2019), but their scalability is intrinsically limited due to gradient updates over a large number of tasks.

Original: To tackle such a scalability issue, we perform amortized inference over the multiple tasks by encoding a dataset as the low-dimensional latent vector and exploit fast GNN propagation instead of the expensive gradient update.
Revised: To tackle such a scalability issue, we perform amortized inference over the multiple tasks by encoding a dataset as the low-dimensional latent vector and exploit fast GNN propagation instead of the expensive gradient update.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our goal is to output a high-performing neural architecture for a given dataset rapidly by learning the prior knowledge obtained from the rich database consisting of datasets and its neural architectures.
Revised: Our goal is to output a high-performing neural architecture for a given dataset rapidly by learning the prior knowledge obtained from the rich database consisting of datasets and its neural architectures.

Original: To this end, we propose Meta Dataset-to-Architecture (MetaD2A) framework which learns the cross-modal latent space of datasets and their neural architectures.
Revised: To this end, we propose Meta Dataset-to-Architecture (MetaD2A) framework which learns the cross-modal latent space of datasets and their neural architectures.

Original: Further, we introduce a metaperformance predictor, which predicts accuracies of given architectures on an unseen input dataset to select the best neural architecture for the dataset.
Revised: Further, we introduce a metaperformance predictor, which predicts accuracies of given architectures on an unseen input dataset to select the best neural architecture for the dataset.

Original: Figure 2 illustrates the overview of our framework.
Revised: Figure 2 illustrates the overview of our framework.

Original: To formally deﬁne the problem, let us assume that we have a database of N τ number of tasks, where we randomly obtain tasks τ from task distribution p ( τ ) to train for the model.
Revised: To formally deﬁne the problem, let us assume that we have a database of N τ number of tasks, where we randomly obtain tasks τ from task distribution p ( τ ) to train for the model.

Original: Each task τ = {D , G , s } consists of a dataset D
Revised: Each task τ = {D , G , s } consists of a dataset D

Original: =
Revised: =

Original: { X , Y } , where X =
Revised: { X , Y } , where X =

Original: { X c } Cc =1 and Y =
Revised: { X c } Cc =1 and Y =

Original: { Y c } Cc =1 are the set of instances and target labels of C classes respectively, a neural architecture represented as a
Revised: { Y c } Cc =1 are the set of instances and target labels of C classes respectively, a neural architecture represented as a

Original: Directed Acyclic Graph (DAG)
Revised: Directed Acyclic Graph (DAG)

Original: G =
Revised: G =

Original: {V , E} , and an accuracy s obtained from the neural architecture
Revised: {V , E} , and an accuracy s obtained from the neural architecture

Original: PMA …
Revised: PMA …

Original: S
Revised: S

Original: Intra SetPool/ Inter SetPool …
Revised: Intra SetPool/ Inter SetPool …

Original: Intra SetPool
Revised: Intra SetPool

Original: Intra SetPool Intra SetPool
Revised: Intra SetPool Intra SetPool

Original: Inter SetPool
Revised: Inter SetPool

Original: Hierarchical Set Pooling v 𝐶 vv
Revised: Hierarchical Set Pooling v 𝐶 vv

Original: SAB … …
Revised: SAB … …

Original: SAB … … (a) Hierarchical Set Pooling
Revised: SAB … … (a) Hierarchical Set Pooling

Original: GNN-AMP Graph Decoder 3 × 3 conv.
Revised: GNN-AMP Graph Decoder 3 × 3 conv.

Original: o 𝑣… 𝑣 0 𝑒 𝑣 0 ,𝑣 1 𝑣 0 𝑣𝑣 0 𝑣𝑣 2 𝑣𝑣(b) GNN-AMP Graph Decoder
Revised: o 𝑣… 𝑣 0 𝑒 𝑣 0 ,𝑣 1 𝑣 0 𝑣𝑣 0 𝑣𝑣 2 𝑣𝑣(b) GNN-AMP Graph Decoder

Original: GNN-AMP Graph Encoder … …
Revised: GNN-AMP Graph Encoder … …

Original: [Figure]
Revised: [Figure]

Original: G trained on D .
Revised: G trained on D .

Original: Each node v ∈ V of G then represents a computational operator (e.g. 3-byconvolution, mean-pooling, ...) and each edge ( u, v ) ∈
Revised: Each node v ∈ V of G then represents a computational operator (e.g. 3-byconvolution, mean-pooling, ...) and each edge ( u, v ) ∈

Original: E represents a computation ﬂow from an output of the operator u to input of the operator v .
Revised: E represents a computation ﬂow from an output of the operator u to input of the operator v .

Original: The whole framework consists of a set encoder, a graph decoder, and a meta-performance predictor parameterized by φ , θ , and ω , respectively.
Revised: The whole framework consists of a set encoder, a graph decoder, and a meta-performance predictor parameterized by φ , θ , and ω , respectively.

Original: The dataset D is embedded into a latent space Z as a vector z through the set encoder q φ and then the graph decoder p θ estimates the distribution and generates a new graph ˜ G from z which are sampled from prior the p ( z ) .
Revised: The dataset D is embedded into a latent space Z as a vector z through the set encoder q φ and then the graph decoder p θ estimates the distribution and generates a new graph ˜ G from z which are sampled from prior the p ( z ) .

Original: We learn both components q φ ( z |D ) and p θ ( G| z ) , generalizing over the task distribution p ( τ ) .
Revised: We learn both components q φ ( z |D ) and p θ ( G| z ) , generalizing over the task distribution p ( τ ) .

Original: After training, for an unseen dataset ˆ D , our model with θ and φ can easily generate DAGs { ˆ G i } n s i =1 as candidates, where n s is the number of sampled architectures.
Revised: After training, for an unseen dataset ˆ D , our model with θ and φ can easily generate DAGs { ˆ G i } n s i =1 as candidates, where n s is the number of sampled architectures.

Original: This is done by decoding the latent embeddings { ˆ z i | ˆ z i ∈ Z} n s i =1 sampled from the prior p ( z ) consisting of ˆ µ and ˆ σ generated from the set encoder for given ˆ D .
Revised: This is done by decoding the latent embeddings { ˆ z i | ˆ z i ∈ Z} n s i =1 sampled from the prior p ( z ) consisting of ˆ µ and ˆ σ generated from the set encoder for given ˆ D .

Original: The predictor f ω ( s |D , G ) is also learned from the task distribution p ( τ ) in the meta-training stage and then selects the best performing neural architecture for ˆ D from { ˆ G i } n s i =1 based on their predicted accuracies { ˆ s i } n s i
Revised: The predictor f ω ( s |D , G ) is also learned from the task distribution p ( τ ) in the meta-training stage and then selects the best performing neural architecture for ˆ D from { ˆ G i } n s i =1 based on their predicted accuracies { ˆ s i } n s i

Original: =1 in the meta-test.
Revised: =1 in the meta-test.

Original: 3. L EARNING TO G ENERATE G RAPHS FROM D ATASETS
Revised: 3. L EARNING TO G ENERATE G RAPHS FROM D ATASETS

Original: Dataset Encoding To compress the entire instances from a dataset D into a single latent code z , the set encoder should process input sets of any size and summarize consistent information agnostically to the order of the instances (permutation-invariance).
Revised: Dataset Encoding To compress the entire instances from a dataset D into a single latent code z , the set encoder should process input sets of any size and summarize consistent information agnostically to the order of the instances (permutation-invariance).

Original: While simple functions such as mean/sum-pooling can fulﬁll those requirements, they cannot model high-order interactions between the set elements to accurately generate complex DAGs such as neural architectures.
Revised: While simple functions such as mean/sum-pooling can fulﬁll those requirements, they cannot model high-order interactions between the set elements to accurately generate complex DAGs such as neural architectures.

Original: Further, those non-hierarchical poolings consider all elements equally, and thus cannot accurately model individual classes in the given dataset, which may lead to poor performance (Lee et al., 2020).
Revised: Further, those non-hierarchical poolings consider all elements equally, and thus cannot accurately model individual classes in the given dataset, which may lead to poor performance (Lee et al., 2020).

Original: Therefore, we introduce a novel Hierarchical Set Pooling for dataset encoding which stacks two permutation-invariant encoders with learnable parameters, that can extract information related to generating set-conditioned DAGs.
Revised: Therefore, we introduce a novel Hierarchical Set Pooling for dataset encoding which stacks two permutation-invariant encoders with learnable parameters, that can extract information related to generating set-conditioned DAGs.

Original: The lower-level intra-class encoder captures the class prototypes that reﬂect label information, and the high-level inter-class encoder considers the relationship between class prototypes and aggregates them into a latent vector.
Revised: The lower-level intra-class encoder captures the class prototypes that reﬂect label information, and the high-level inter-class encoder considers the relationship between class prototypes and aggregates them into a latent vector.

Original: Speciﬁcally, for a given dataset D , we construct the subset for class c as { x | x ∈ B c } ∈ R b × d x , where B c ⊂ X c and || B c ||
Revised: Speciﬁcally, for a given dataset D , we construct the subset for class c as { x | x ∈ B c } ∈ R b × d x , where B c ⊂ X c and || B c ||

Original: =
Revised: =

Original: b by random sampling.
Revised: b by random sampling.

Original: We input the sub-dataset into the IntraSetPool, the intra-class encoder, to encode class prototype v c ∈ R 1 × d v c for each class c = 1 , ..., C : v c = IntraSetPool (cid:0) { x | x ∈ B c } (cid:1) .
Revised: We input the sub-dataset into the IntraSetPool, the intra-class encoder, to encode class prototype v c ∈ R 1 × d v c for each class c = 1 , ..., C : v c = IntraSetPool (cid:0) { x | x ∈ B c } (cid:1) .

Original: Then we further feed the class-speciﬁc set representations { v c } Cc =1 into the InterSetPool, the inter-class encoder, to generate the dataset representation h e ∈ R 1 × d h e : h e =
Revised: Then we further feed the class-speciﬁc set representations { v c } Cc =1 into the InterSetPool, the inter-class encoder, to generate the dataset representation h e ∈ R 1 × d h e : h e =

Original: InterSetPool (cid:0) { v c } Cc =1 (cid:1) .
Revised: InterSetPool (cid:0) { v c } Cc =1 (cid:1) .

Original: Both the set encoders are comprised of two building blocks: Set Attention Block (SAB) and Pooling by Multi-head Attention (PMA)
Revised: Both the set encoders are comprised of two building blocks: Set Attention Block (SAB) and Pooling by Multi-head Attention (PMA)

Original: (Lee et al., 2019a), where the former learns the features for each element in the set using self-attention while the latter pools the input features into k representative vectors.
Revised: (Lee et al., 2019a), where the former learns the features for each element in the set using self-attention while the latter pools the input features into k representative vectors.

Original: Please see Section A.1 of the Appendix for more details.
Revised: Please see Section A.1 of the Appendix for more details.

Original: We set k = 1 to produce a single vector, and stack SAB twice followed by PMA for both set encoders as follows (See Figure 3 (a)):
Revised: We set k = 1 to produce a single vector, and stack SAB twice followed by PMA for both set encoders as follows (See Figure 3 (a)):

Original: [Equation]
Revised: [Equation]

Original: Then the latent code z ∈ R 1 × d z can be sampled from a dataset-conditioned Gaussian distribution with diagonal covariance where NN µ , NN σ are single linear layers:
Revised: Then the latent code z ∈ R 1 × d z can be sampled from a dataset-conditioned Gaussian distribution with diagonal covariance where NN µ , NN σ are single linear layers:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Graph Decoding To generate a DAG G describing a neural architecture from the latent code z , we use a Graph Neural Network(GNN)-based decoder p θ ( G| z ) .
Revised: Graph Decoding To generate a DAG G describing a neural architecture from the latent code z , we use a Graph Neural Network(GNN)-based decoder p θ ( G| z ) .

Original: While GNN-based decoders have been actively used to generate graph-structured data (Liu et al., 2018b; Liao et al., 2019; Jin et al., 2020), they are suboptimal for our goal since they generally target undirected graphs and therefore ignore the topological order of DAGs.
Revised: While GNN-based decoders have been actively used to generate graph-structured data (Liu et al., 2018b; Liao et al., 2019; Jin et al., 2020), they are suboptimal for our goal since they generally target undirected graphs and therefore ignore the topological order of DAGs.

Original: Thus, we use a GNN with asynchronous message passing (GNN-AMP) which allows message passing to happen only along the topological order of the
Revised: Thus, we use a GNN with asynchronous message passing (GNN-AMP) which allows message passing to happen only along the topological order of the

Original: DAGs (Zhang et al., 2019).
Revised: DAGs (Zhang et al., 2019).

Original: The graph decoder starts from an initial hidden state h v 0 =
Revised: The graph decoder starts from an initial hidden state h v 0 =

Original: NN init ( z ) , where NN init is an MLP followed by tanh .
Revised: NN init ( z ) , where NN init is an MLP followed by tanh .

Original: For i th node v i according to topological order , we compute the probability of each operation type o v i ∈ R 1 × n o over n o operations, given the current graph state as the last hidden node h G :=
Revised: For i th node v i according to topological order , we compute the probability of each operation type o v i ∈ R 1 × n o over n o operations, given the current graph state as the last hidden node h G :=

Original: h v i .
Revised: h v i .

Original: That is, o v i =
Revised: That is, o v i =

Original: NN node ( h G ) , where NN node is an MLP followed by softmax .
Revised: NN node ( h G ) , where NN node is an MLP followed by softmax .

Original: When the predicted v i type is the end-of-graph, we stop the decoding process and connect all leaf nodes to v i .
Revised: When the predicted v i type is the end-of-graph, we stop the decoding process and connect all leaf nodes to v i .

Original: Otherwise we update hidden state h ( t ) v i at time step t as follows:
Revised: Otherwise we update hidden state h ( t ) v i at time step t as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The function UPDATE is a gated recurrent unit (GRU) (Cho et al., 2014), i is the order of v i , and m ( t ) v
Revised: The function UPDATE is a gated recurrent unit (GRU) (Cho et al., 2014), i is the order of v i , and m ( t ) v

Original: i is the incoming message to v i .
Revised: i is the incoming message to v i .

Original: The function AGGREGATE consists of mapping and gating functions with MLPs, where V inv i is a set of predecessors with incoming edges to v i .
Revised: The function AGGREGATE consists of mapping and gating functions with MLPs, where V inv i is a set of predecessors with incoming edges to v i .

Original: For all previously processed nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) , we decide whether to link an edge from v j to v i by sampling the edge based on edge connection probability e { v j ,v i } =
Revised: For all previously processed nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) , we decide whether to link an edge from v j to v i by sampling the edge based on edge connection probability e { v j ,v i } =

Original: NN edge ( h j , h i ) , where NN edge is a MLP followed by sigmoid .
Revised: NN edge ( h j , h i ) , where NN edge is a MLP followed by sigmoid .

Original: We update h v i by Eq.
Revised: We update h v i by Eq.

Original: (4) whenever a new edge is connected to v i .
Revised: (4) whenever a new edge is connected to v i .

Original: For meta-test, we select the operation with the max probability for each node and edges with e { v j ,v i } > 0 .
Revised: For meta-test, we select the operation with the max probability for each node and edges with e { v j ,v i } > 0 .

Original: Fig 3(b) illustrates the decoder.
Revised: Fig 3(b) illustrates the decoder.

Original: For more details, please see Section A.2.
Revised: For more details, please see Section A.2.

Original: Meta-training Strategy for NAS
Revised: Meta-training Strategy for NAS

Original: We meta-learn the model using set-amortized inference, by maximizing the approximated evidence lower bound (ELBO) for each task τ as follows:
Revised: We meta-learn the model using set-amortized inference, by maximizing the approximated evidence lower bound (ELBO) for each task τ as follows:

Original: [Equation]
Revised: [Equation]

Original: where each dimension of prior p ( z ) factorizes into N (0 , 1) .
Revised: where each dimension of prior p ( z ) factorizes into N (0 , 1) .

Original: L τKL is the KL divergence between two multivariate Gaussian distributions which has a simple closed form (Kingma & Welling, 2014) and λ is the scalar weighting value.
Revised: L τKL is the KL divergence between two multivariate Gaussian distributions which has a simple closed form (Kingma & Welling, 2014) and λ is the scalar weighting value.

Original: Using the reparameterization trick on z , we can optimize the above objective by stochastic gradient variational Bayes (Kingma & Welling, 2014).
Revised: Using the reparameterization trick on z , we can optimize the above objective by stochastic gradient variational Bayes (Kingma & Welling, 2014).

Original: The expectation of the log-likelihood of (5) can be rewritten with negative cross-entropy loss − L τCE for nodes and binary cross-entropy loss − L τBCE for edges, and we slightly modify it using the generated set-dependent graph ˜ G and the ground truth graph G as the input as follows:
Revised: The expectation of the log-likelihood of (5) can be rewritten with negative cross-entropy loss − L τCE for nodes and binary cross-entropy loss − L τBCE for edges, and we slightly modify it using the generated set-dependent graph ˜ G and the ground truth graph G as the input as follows:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: We substitute the log-likelihood term of the equation (5) such as equation (6) and take negative to the equation (5) to convert it to the resultant objective
Revised: We substitute the log-likelihood term of the equation (5) such as equation (6) and take negative to the equation (5) to convert it to the resultant objective

Original: L τ φ , θ ( D , G ) as follows:
Revised: L τ φ , θ ( D , G ) as follows:

Original: [Equation]
Revised: [Equation]

Original: We learn our MetaD2A generator by minimizing the ﬁnal objective to learn φ , θ , which are shared across all tasks.
Revised: We learn our MetaD2A generator by minimizing the ﬁnal objective to learn φ , θ , which are shared across all tasks.

Original: In the meta-test stage, for a unseen dataset ˆ D , we can obtain set-dependent DAGs { ˆ G i } n s i =1 , with the meta-trained generator parameterized by φ ∗ , θ ∗ , by feeding ˆ D as the input.
Revised: In the meta-test stage, for a unseen dataset ˆ D , we can obtain set-dependent DAGs { ˆ G i } n s i =1 , with the meta-trained generator parameterized by φ ∗ , θ ∗ , by feeding ˆ D as the input.

Original: Thus our method can generate neural architecture(s) for the novel dataset via set-level amortized inference.
Revised: Thus our method can generate neural architecture(s) for the novel dataset via set-level amortized inference.

Original: 3. M ETA -P ERFORMANCE P REDICTOR
Revised: 3. M ETA -P ERFORMANCE P REDICTOR

Original: We design a meta-performance predictor f ω to select the best architecture from the architectures generated by MetaD2A generator, based on their predicted accuaracies on the target task.
Revised: We design a meta-performance predictor f ω to select the best architecture from the architectures generated by MetaD2A generator, based on their predicted accuaracies on the target task.

Original: Since our predictor should accurately predict the model performance for any datasets unlike the predictors used in task-speciﬁc NAS, it also meta-learned.
Revised: Since our predictor should accurately predict the model performance for any datasets unlike the predictors used in task-speciﬁc NAS, it also meta-learned.

Original: The meta-performance predictor consists of a dataset encoder and a graph encoder, followed by two linear layers with relu .
Revised: The meta-performance predictor consists of a dataset encoder and a graph encoder, followed by two linear layers with relu .

Original: For dataset encoding, we use the Hierarchical Set Pooling of Section 3.1 which takes D as an input, using only the mean µ from Eq.
Revised: For dataset encoding, we use the Hierarchical Set Pooling of Section 3.1 which takes D as an input, using only the mean µ from Eq.

Original: (3) as an output.
Revised: (3) as an output.

Original: We perform a bi-directional encoding of the given graph following GNNAMP (Zhang et al., 2019), where given G , we sequentially perform message passing for nodes from the predecessors following the topological order of the DAG G and then reverse the node orders and perform an additional encoding process (See Figure 3 (c), and Section A.3 of the appendix for more details).
Revised: We perform a bi-directional encoding of the given graph following GNNAMP (Zhang et al., 2019), where given G , we sequentially perform message passing for nodes from the predecessors following the topological order of the DAG G and then reverse the node orders and perform an additional encoding process (See Figure 3 (c), and Section A.3 of the appendix for more details).

Original: We concatenate the outputs of both the bi-directional graph encoder and the set encoder, and feed them to two linear layers with relu to predict accuracy.
Revised: We concatenate the outputs of both the bi-directional graph encoder and the set encoder, and feed them to two linear layers with relu to predict accuracy.

Original: We train the predictor f ω to minimize the MSE loss between the predicted accuracy and the true accuracy s i of the model on each task:
Revised: We train the predictor f ω to minimize the MSE loss between the predicted accuracy and the true accuracy s i of the model on each task:

Original: [Equation]
Revised: [Equation]

Original: In the meta-test, the predictor f ω ∗ (ˆ s i | ˆ D , ˆ G i ) predicts accuracies { ˆ s i } n s i =1 for a given ˆ D and { ˆ G i } n s i = and then select the neural architecture having the highest predicted accuracy among { ˆ s i } n s i =1 .
Revised: In the meta-test, the predictor f ω ∗ (ˆ s i | ˆ D , ˆ G i ) predicts accuracies { ˆ s i } n s i =1 for a given ˆ D and { ˆ G i } n s i = and then select the neural architecture having the highest predicted accuracy among { ˆ s i } n s i =1 .

Original: To our knowledge, the performance predictors proposed in the existing NAS works need re-training for the new dataset since they have no module to handle multiple datasets but only encoders for graphs.
Revised: To our knowledge, the performance predictors proposed in the existing NAS works need re-training for the new dataset since they have no module to handle multiple datasets but only encoders for graphs.

Original: On the other hand, our proposed meta-performance predictor consists of a dataset encoder and graph encoder to address such a problem with one training.
Revised: On the other hand, our proposed meta-performance predictor consists of a dataset encoder and graph encoder to address such a problem with one training.

Original: Moreover, the proposed metaperformance predictor learns meta-knowledge accumulated from multiple tasks of MetaImageNet.
Revised: Moreover, the proposed metaperformance predictor learns meta-knowledge accumulated from multiple tasks of MetaImageNet.

Original: Thus, the predictor can predict performance for the new unseen dataset.
Revised: Thus, the predictor can predict performance for the new unseen dataset.

Original: As the number of target datasets increases, the time complexity of other predictors linearly increases while ours needs a constant time complexity of O(1).
Revised: As the number of target datasets increases, the time complexity of other predictors linearly increases while ours needs a constant time complexity of O(1).

Original: In addition, none of the existing works in both NAS or metalearning can handle the scale of meta-learning as our MetaD2A does.
Revised: In addition, none of the existing works in both NAS or metalearning can handle the scale of meta-learning as our MetaD2A does.

Original: 3. C OMPARISION WITH C ONVENTIONAL M ETA -NAS
Revised: 3. C OMPARISION WITH C ONVENTIONAL M ETA -NAS

Original: Existing meta-NAS approaches (Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) mainly target few-shot classiﬁcation and cannot scale to such a large task with a large number of samples, since they rely on gradient-based meta-learning with rollout gradient steps.
Revised: Existing meta-NAS approaches (Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) mainly target few-shot classiﬁcation and cannot scale to such a large task with a large number of samples, since they rely on gradient-based meta-learning with rollout gradient steps.

Original: For few-shot learning, taking a few gradient steps is more than enough since there are only a few samples from which we can compute the gradients, but for large-scale learning, taking few gradient steps with minibatch sampling will only consider a very small portion of the samples in the task, which will be largely insufﬁcient for the model to converge.
Revised: For few-shot learning, taking a few gradient steps is more than enough since there are only a few samples from which we can compute the gradients, but for large-scale learning, taking few gradient steps with minibatch sampling will only consider a very small portion of the samples in the task, which will be largely insufﬁcient for the model to converge.

Original: Thus the model needs to take a large number of gradient steps with a large number of samples, however, this is extremely costly and almost infeasible when computing second-order derivatives as in MAML.
Revised: Thus the model needs to take a large number of gradient steps with a large number of samples, however, this is extremely costly and almost infeasible when computing second-order derivatives as in MAML.

Original: On the contrary, MetaD2A can handle the largeway many-shot problem tackled by general classiﬁcation tasks, since it does not require any rollout gradient steps when inferring the architecture.
Revised: On the contrary, MetaD2A can handle the largeway many-shot problem tackled by general classiﬁcation tasks, since it does not require any rollout gradient steps when inferring the architecture.

Original: For conventional few-shot settings (5-way 5-shot), the number of samples per task is 25.
Revised: For conventional few-shot settings (5-way 5-shot), the number of samples per task is 25.

Original: Contrarily, the number of samples of CIFAR-100 used in
Revised: Contrarily, the number of samples of CIFAR-100 used in

Original: MetaD2A is 100 × 20 = 2 , 000 which is 80 times larger.
Revised: MetaD2A is 100 × 20 = 2 , 000 which is 80 times larger.

Original: Further, the number of instances(e.g.
Revised: Further, the number of instances(e.g.

Original: 20) can be set to a much larger number.
Revised: 20) can be set to a much larger number.

Original: Overcoming such a limitation of conventional meta-NAS methods, and meta-learning in general, is one of the main contributions of this work, which we believe is highly novel, in both the perspective of NAS and meta-learning.
Revised: Overcoming such a limitation of conventional meta-NAS methods, and meta-learning in general, is one of the main contributions of this work, which we believe is highly novel, in both the perspective of NAS and meta-learning.

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We conduct extensive experiments to demonstrate the effectiveness and efﬁciency of MetaD2A.
Revised: We conduct extensive experiments to demonstrate the effectiveness and efﬁciency of MetaD2A.

Original: Search Space We explore the same search space of NAS-Bench-201 (Dong & Yang, 2020), cellbased NAS, for all experiments.
Revised: Search Space We explore the same search space of NAS-Bench-201 (Dong & Yang, 2020), cellbased NAS, for all experiments.

Original: (See the Section B of the Appendix for more details).
Revised: (See the Section B of the Appendix for more details).

Original: Dataset We validate our method on multiple datasets as follows.
Revised: Dataset We validate our method on multiple datasets as follows.

Original: 1) MetaImageNet :
Revised: 1) MetaImageNet :

Original: To meta-learn our model on multiple tasks, we compiled a new dataset consisting of multiple subsets of ImageNet1K (Deng et al., 2009) and their neural architectures.
Revised: To meta-learn our model on multiple tasks, we compiled a new dataset consisting of multiple subsets of ImageNet1K (Deng et al., 2009) and their neural architectures.

Original: We downsample the image size to 32 ×and split classes into 600/400 for meta-training/meta-test, respectively.
Revised: We downsample the image size to 32 ×and split classes into 600/400 for meta-training/meta-test, respectively.

Original: For each task of metatraining/meta-test, we make a dataset by sampling 20 classes randomly from meta-training/metatest with an average of 26K images, we search set-speciﬁc architecture on the sampled dataset using
Revised: For each task of metatraining/meta-test, we make a dataset by sampling 20 classes randomly from meta-training/metatest with an average of 26K images, we search set-speciﬁc architecture on the sampled dataset using

Original: GDAS (Dong & Yang, 2019b) or random search, and we obtain its accuracy by training the searched architecture on the dataset.
Revised: GDAS (Dong & Yang, 2019b) or random search, and we obtain its accuracy by training the searched architecture on the dataset.

Original: We collect N τ = 1,679/3,752 meta-training tasks for the MetaD2A generator/predictor and 20/400 meta-test tasks for them, respectively.
Revised: We collect N τ = 1,679/3,752 meta-training tasks for the MetaD2A generator/predictor and 20/400 meta-test tasks for them, respectively.

Original: We apply our model trained from
Revised: We apply our model trained from

Original: MetaImageNet to multiple benchmark datasets such as 2) MNIST (LeCun & Cortes, 2010), 3)
Revised: MetaImageNet to multiple benchmark datasets such as 2) MNIST (LeCun & Cortes, 2010), 3)

Original: SVHN (Netzer et al., 2011), 4) CIFAR-10 (Krizhevsky et al., 2009), 5) CIFAR-100 (Krizhevsky et al., 2009), 6) Aircraft (Maji et al., 2013), and 7) Oxford-IIIT Pets (Parkhi et al., 2012).
Revised: SVHN (Netzer et al., 2011), 4) CIFAR-10 (Krizhevsky et al., 2009), 5) CIFAR-100 (Krizhevsky et al., 2009), 6) Aircraft (Maji et al., 2013), and 7) Oxford-IIIT Pets (Parkhi et al., 2012).

Original: (For detailed descriptions of the baselines, datasets and experimental setup, see Appendix, Section C)
Revised: (For detailed descriptions of the baselines, datasets and experimental setup, see Appendix, Section C)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. S EARCH R ESULTS ON M ETA I MAGE N ET
Revised: 4. S EARCH R ESULTS ON M ETA I MAGE N ET

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we report the performance of baselines and our model trained on the training set of
Revised: we report the performance of baselines and our model trained on the training set of

Original: MetaImageNet, on unseen tasks of MetaImageNet.
Revised: MetaImageNet, on unseen tasks of MetaImageNet.

Original: Note that for the unseen MetaImageNet tasks, all baseline models ∗ require to train the NAS model from scratch on each individual task .
Revised: Note that for the unseen MetaImageNet tasks, all baseline models ∗ require to train the NAS model from scratch on each individual task .

Original: On the contrary, our model can generate set-dependent architectures with the amortized inference through the architecture generator.
Revised: On the contrary, our model can generate set-dependent architectures with the amortized inference through the architecture generator.

Original: In the inference process
Revised: In the inference process

Original: [Table]
Revised: [Table]

Original: we sample 1000 architecture candidates from the MetaD2A generator and select and train one architecture with the highest predicted accuracy computed by the meta-performance predictor.
Revised: we sample 1000 architecture candidates from the MetaD2A generator and select and train one architecture with the highest predicted accuracy computed by the meta-performance predictor.

Original: We report the average accuracy of over 20 unseen tasks with 3 different random seeds for all models.
Revised: We report the average accuracy of over 20 unseen tasks with 3 different random seeds for all models.

Original: For our model, we report both the search time (27.03s) averaged on the 20 tasks, and meta-training time (26964.18s) † .
Revised: For our model, we report both the search time (27.03s) averaged on the 20 tasks, and meta-training time (26964.18s) † .

Original: We calculate the search expense by with the hourly cost of 1.46$/h, of Tesla P100 on Google Cloud.
Revised: We calculate the search expense by with the hourly cost of 1.46$/h, of Tesla P100 on Google Cloud.

Original: All models are run on an Nvidia 2080ti GPU.
Revised: All models are run on an Nvidia 2080ti GPU.

Original: The result shows that the proposed MetaD2A outperforms all baselines in both the accuracy and search time on unseen tasks, with fast search time (27.03s).
Revised: The result shows that the proposed MetaD2A outperforms all baselines in both the accuracy and search time on unseen tasks, with fast search time (27.03s).

Original: We further report the accuracy over the search time in Figure 4 (c).
Revised: We further report the accuracy over the search time in Figure 4 (c).

Original: Once meta-training is done, our model can generate architectures on any new tasks without training the NAS framework again, which can be done at extremely low cost ($0.01 for each dataset), which makes it highly practical.
Revised: Once meta-training is done, our model can generate architectures on any new tasks without training the NAS framework again, which can be done at extremely low cost ($0.01 for each dataset), which makes it highly practical.

Original: 4. S EARCH R ESULTS ON M ULTIPLE D ATASETS
Revised: 4. S EARCH R ESULTS ON M ULTIPLE D ATASETS

Original: Table 2 shows that our model meta-trained on the MetaImageNet dataset can generalize to multiple datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets.
Revised: Table 2 shows that our model meta-trained on the MetaImageNet dataset can generalize to multiple datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets.

Original: Since
Revised: Since

Original: MetaD2A can output set-adaptive architectures on the target datasets, there is no training cost for the unseen tasks.
Revised: MetaD2A can output set-adaptive architectures on the target datasets, there is no training cost for the unseen tasks.

Original: MetaD2A infers fast, as Table 3 shows, such as 22.47(s) for CIFAR

Revised: MetaD2A infers fast, as Table 3 shows, such as 22.47(s) for CIFAR

Original: and it has relatively comparable parameters 1.05M for CIFAR10 .
Revised: and it has relatively comparable parameters 1.05M for CIFAR10 .

Original: On CIFAR10 and CIFAR100 , we report the mean accuracies over 10 runs of the search process by retrieving accuracies of searched architectures from NAS-Bench-201.
Revised: On CIFAR10 and CIFAR100 , we report the mean accuracies over 10 runs of the search process by retrieving accuracies of searched architectures from NAS-Bench-201.

Original: Speciﬁcally, following SETN (Dong & Yang, 2019a), we retrieve the accuracies of N architecture candidates from the NAS-bench-201 and report the highest ﬁnal accu-
Revised: Speciﬁcally, following SETN (Dong & Yang, 2019a), we retrieve the accuracies of N architecture candidates from the NAS-bench-201 and report the highest ﬁnal accu-

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: racy for each run.
Revised: racy for each run.

Original: While N = 1000 in SETN, MetaD2A searches for an optimal architecture using a smaller number of samples ( N = 40 ), reliably returning 94 .
Revised: While N = 1000 in SETN, MetaD2A searches for an optimal architecture using a smaller number of samples ( N = 40 ), reliably returning 94 .

Original: 37 and 73 .
Revised: 37 and 73 .

Original: 51 over 10 runs, in very small conﬁdence interval.
Revised: 51 over 10 runs, in very small conﬁdence interval.

Original: N = 10 is sufﬁcient to obtain the model that outperforms the baselines as shown in Figure 5.
Revised: N = 10 is sufﬁcient to obtain the model that outperforms the baselines as shown in Figure 5.

Original: On MNIST , SVHN , Aircraft , and Oxford-IIIT Pets , we search one architecture by MetaD2A model and report the accuracy averaging over 3 runs with different seeds.
Revised: On MNIST , SVHN , Aircraft , and Oxford-IIIT Pets , we search one architecture by MetaD2A model and report the accuracy averaging over 3 runs with different seeds.

Original: Such rapid search of REA, RS, REINFORCE and BOHB is only possible where all of the accuracies are precomputed like NAS-Bench201 so that it can retrieve instantly on the target dataset, therefore, it is difﬁcult to apply them to other non-benchmark datasets.
Revised: Such rapid search of REA, RS, REINFORCE and BOHB is only possible where all of the accuracies are precomputed like NAS-Bench201 so that it can retrieve instantly on the target dataset, therefore, it is difﬁcult to apply them to other non-benchmark datasets.

Original: These experiments show the transferability of MetaD2A, as obtaining higher accuracy against all baselines on multiple datasets.
Revised: These experiments show the transferability of MetaD2A, as obtaining higher accuracy against all baselines on multiple datasets.

Original: Especially, we observe that MetaD2A which is learned over multiple tasks beneﬁt to search set-dependent neural architectures for ﬁne-grained datasets such as Aircraft and Oxford-IIIT Pets .
Revised: Especially, we observe that MetaD2A which is learned over multiple tasks beneﬁt to search set-dependent neural architectures for ﬁne-grained datasets such as Aircraft and Oxford-IIIT Pets .

Original: Qualitative analysis We further perform several qualitative analysis in Figure 4.
Revised: Qualitative analysis We further perform several qualitative analysis in Figure 4.

Original: We ﬁrst visualize the dataset and architecture embeddings in the latent search space in Figure 4 (a).
Revised: We ﬁrst visualize the dataset and architecture embeddings in the latent search space in Figure 4 (a).

Original: To embed graphs, we combine the graph encoder of Section 3.2 and the graph decoder of Section 3.1 and train it for graph reconstruction on MetaImageNet training tasks.
Revised: To embed graphs, we combine the graph encoder of Section 3.2 and the graph decoder of Section 3.1 and train it for graph reconstruction on MetaImageNet training tasks.

Original: We observe that the architecture whose embeddings are closer to the given dataset achieves better performance on it, compared to architecture embedded far away.
Revised: We observe that the architecture whose embeddings are closer to the given dataset achieves better performance on it, compared to architecture embedded far away.

Original: For example, the ﬁnal accuracy of the closest architecture point for the Aircraft dataset is 50.87% while the farthest architecture only achieves 16.77%.
Revised: For example, the ﬁnal accuracy of the closest architecture point for the Aircraft dataset is 50.87% while the farthest architecture only achieves 16.77%.

Original: In Figure 4 (b), we visualize the latent vectors of the multiple datasets.
Revised: In Figure 4 (b), we visualize the latent vectors of the multiple datasets.

Original: Similar datasets, such as CIFAR-10 and CIFAR-100 are embedded closely in the latent space, while datasets dissimilar to them, such as MNIST and SVHN, are embedded far away.
Revised: Similar datasets, such as CIFAR-10 and CIFAR-100 are embedded closely in the latent space, while datasets dissimilar to them, such as MNIST and SVHN, are embedded far away.

Original: This enables generating architectures that perform better on each dataset.
Revised: This enables generating architectures that perform better on each dataset.

Original: 4. E FFECTIVENESS OF M ETA D2A
Revised: 4. E FFECTIVENESS OF M ETA D2A

Original: Now, we verify the efﬁcacy of each component of MetaD2A with further analysis.
Revised: Now, we verify the efﬁcacy of each component of MetaD2A with further analysis.

Original: [Table]
Revised: [Table]

Original: Generative Abilities as a VAE model We show the basic abilities of VAE where MetaD2A is based.
Revised: Generative Abilities as a VAE model We show the basic abilities of VAE where MetaD2A is based.

Original: We evaluate generated 10,000 neural architecture samples with the measures Validity,
Revised: We evaluate generated 10,000 neural architecture samples with the measures Validity,

Original: Uniqueness, and Novelty.
Revised: Uniqueness, and Novelty.

Original: Each is deﬁned as how often the model can generate valid neural architectures from the prior distribution, the proportion of unique graphs out of the valid generations, and the proportion of valid generations that are not included in the training set, respectively.
Revised: Each is deﬁned as how often the model can generate valid neural architectures from the prior distribution, the proportion of unique graphs out of the valid generations, and the proportion of valid generations that are not included in the training set, respectively.

Original: The result shows MetaD2A can generate 67.31% new graphs that do not belong to the training set and can generate 35.19% various graphs, not picking alwaysperforming-well architecture.
Revised: The result shows MetaD2A can generate 67.31% new graphs that do not belong to the training set and can generate 35.19% various graphs, not picking alwaysperforming-well architecture.

Original: Ablation Study on MetaD2A
Revised: Ablation Study on MetaD2A

Original: We train different variations of our model on the MetaImageNet, and test on CIFAR10, CIFAR100, and Aircraft in Table 5 with the same experimental setup as the main experiments in Table 2.
Revised: We train different variations of our model on the MetaImageNet, and test on CIFAR10, CIFAR100, and Aircraft in Table 5 with the same experimental setup as the main experiments in Table 2.

Original: We observe that combining the performance predictor to the random architecture sampler (Random w/ PP.)
Revised: We observe that combining the performance predictor to the random architecture sampler (Random w/ PP.)

Original: enhances the accuracy of the ﬁnal architecture on all datasets.
Revised: enhances the accuracy of the ﬁnal architecture on all datasets.

Original: The
Revised: The

Original: MetaD2A generator without the performance predictor (MetaD2A w/o PP.) outperforms the simple random architecture sampler (Random), especially by 15 .
Revised: MetaD2A generator without the performance predictor (MetaD2A w/o PP.) outperforms the simple random architecture sampler (Random), especially by 15 .

Original: 3% on Aircraft, which demonstrates the effectiveness of MetaD2A over the random sampler.
Revised: 3% on Aircraft, which demonstrates the effectiveness of MetaD2A over the random sampler.

Original: Finally, MetaD2A combined with the performance predictor outperforms all baselines, especially by 22.16% on Aircraft, suggesting that our
Revised: Finally, MetaD2A combined with the performance predictor outperforms all baselines, especially by 22.16% on Aircraft, suggesting that our

Original: MetaD2A generator can sample architectures that are more relevant to the given task.
Revised: MetaD2A generator can sample architectures that are more relevant to the given task.

Original: We further report the accuracy of the retrieved architecture on CIFAR10 and CIFAR100 with a different number of samples in Figure 5, which shows that our model requires only a small number of samples to obtain a good neural architecture.
Revised: We further report the accuracy of the retrieved architecture on CIFAR10 and CIFAR100 with a different number of samples in Figure 5, which shows that our model requires only a small number of samples to obtain a good neural architecture.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: We further analyze our performance predictor using the
Revised: We further analyze our performance predictor using the

Original: Pearson correlation coefﬁcient on the validation tasks of
Revised: Pearson correlation coefﬁcient on the validation tasks of

Original: MetaImageNet to measure the linear correlation between the actual performance and the predicted performance (higher the better).
Revised: MetaImageNet to measure the linear correlation between the actual performance and the predicted performance (higher the better).

Original: Table 6 shows that using both the dataset and the computational graph of the target architecture as inputs, instead of using one, leads to better performance.
Revised: Table 6 shows that using both the dataset and the computational graph of the target architecture as inputs, instead of using one, leads to better performance.

Original: Moreover, the proposed encoder yields models with a higher correlation than the mean pooling, which suggests that the higher-order interaction between the input instances is useful for graph generation.
Revised: Moreover, the proposed encoder yields models with a higher correlation than the mean pooling, which suggests that the higher-order interaction between the input instances is useful for graph generation.

Original: Finally, using hierarchical set encoding and bi-directional graph encoding both clearly improve the performance of the predictor.
Revised: Finally, using hierarchical set encoding and bi-directional graph encoding both clearly improve the performance of the predictor.

Original: We also further evaluate the predictor performance for both the Pearson correlation coefﬁcient and the accuracy of the searched architecture on CIFAR10 and CIFAR100 under the same experiment condition of the Table 2.
Revised: We also further evaluate the predictor performance for both the Pearson correlation coefﬁcient and the accuracy of the searched architecture on CIFAR10 and CIFAR100 under the same experiment condition of the Table 2.

Original: To separate the effect of the predictor, we use random sampling instead of MetaD2A generator for this experiment.
Revised: To separate the effect of the predictor, we use random sampling instead of MetaD2A generator for this experiment.

Original: Using a larger number of tasks for training improves the performance of the predictor in both measures, suggesting that we can further improve the performance of MetaD2A by collecting more training pairs.
Revised: Using a larger number of tasks for training improves the performance of the predictor in both measures, suggesting that we can further improve the performance of MetaD2A by collecting more training pairs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a novel NAS framework, MetaD2A (Meta Dataset-to-Architecture), that can output a neural architecture for an unseen dataset.
Revised: We proposed a novel NAS framework, MetaD2A (Meta Dataset-to-Architecture), that can output a neural architecture for an unseen dataset.

Original: The MetaD2A generator learns a dataset-to-architecture transformation over a database of datasets and neural architectures by encoding each dataset using a set encoder and generating each neural architecture with a graph decoder.
Revised: The MetaD2A generator learns a dataset-to-architecture transformation over a database of datasets and neural architectures by encoding each dataset using a set encoder and generating each neural architecture with a graph decoder.

Original: While the model can generate a novel architecture given a new dataset in an amortized inference, we further learn a meta-performance predictor to select the best architecture for the dataset among multiple sampled architectures.
Revised: While the model can generate a novel architecture given a new dataset in an amortized inference, we further learn a meta-performance predictor to select the best architecture for the dataset among multiple sampled architectures.

Original: The experimental results show that our method outperforms conventional NAS methods on various datasets with very small search time (less than a minute) as it generalizes well across datasets.
Revised: The experimental results show that our method outperforms conventional NAS methods on various datasets with very small search time (less than a minute) as it generalizes well across datasets.

Original: We believe that our work is a meaningful step towards building a practical NAS system for real-world scenarios, where we need to handle diverse datasets while minimizing the search cost.
Revised: We believe that our work is a meaningful step towards building a practical NAS system for real-world scenarios, where we need to handle diverse datasets while minimizing the search cost.

Original: A M ETA D ATASET -D EPENDENT A RCHITECTURE G ENERATION M ODEL A. B UILDING B LOCKS OF THE S ET E NCODER
Revised: A M ETA D ATASET -D EPENDENT A RCHITECTURE G ENERATION M ODEL A. B UILDING B LOCKS OF THE S ET E NCODER

Original: Set Attention Block (SAB) is an attention-based block, which makes the features of all of the instances in the set reﬂect the relations between itself and others such as:
Revised: Set Attention Block (SAB) is an attention-based block, which makes the features of all of the instances in the set reﬂect the relations between itself and others such as:

Original: [Equation]
Revised: [Equation]

Original: where H = LN ( X + MH ( X , X , X ))
Revised: where H = LN ( X + MH ( X , X , X ))

Original: (9) where LN and MLP is the layer normalization (Ba et al., 2016) and the multilayer perceptron respectively, and H ∈ R n B c × d H is computed with multi-head attention MH ( Q , K , V )
Revised: (9) where LN and MLP is the layer normalization (Ba et al., 2016) and the multilayer perceptron respectively, and H ∈ R n B c × d H is computed with multi-head attention MH ( Q , K , V )

Original: (Vaswani et al., 2017) which queries, keys, and values are elements of input set X .
Revised: (Vaswani et al., 2017) which queries, keys, and values are elements of input set X .

Original: Features encoded from the SAB layers can be pooled by PMA on learnable seed vectors S ∈ R k × d S to produce k vectors by slightly modifying H calculation of (9):
Revised: Features encoded from the SAB layers can be pooled by PMA on learnable seed vectors S ∈ R k × d S to produce k vectors by slightly modifying H calculation of (9):

Original: PMA
Revised: PMA

Original: [Equation]
Revised: [Equation]

Original: MLP ( H )) where H = LN
Revised: MLP ( H )) where H = LN

Original: ( X + MH ( S , MLP ( X ) , MLP ( X ))) (10)
Revised: ( X + MH ( S , MLP ( X ) , MLP ( X ))) (10)

Original: While k can be any size (i.e. k =1,2,10,16), we set k = 1 for generating the single latent vector.
Revised: While k can be any size (i.e. k =1,2,10,16), we set k = 1 for generating the single latent vector.

Original: For extracting consistent information not depending the order and the size of input elements, encoding functions should be constructed by stacking permutation-equivariant layers E, which satisﬁes below condition for any permutation π on a set X (Zaheer et al., 2017):
Revised: For extracting consistent information not depending the order and the size of input elements, encoding functions should be constructed by stacking permutation-equivariant layers E, which satisﬁes below condition for any permutation π on a set X (Zaheer et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: Since all of the components in SAB and PMA are row-wise computation functions, SAB and PMA is permutation equivarint by deﬁnition (11).
Revised: Since all of the components in SAB and PMA are row-wise computation functions, SAB and PMA is permutation equivarint by deﬁnition (11).

Original: A. G RAPH D ECODING
Revised: A. G RAPH D ECODING

Original: We use a GNN-based decoder with asynchronous message passing (GNN-AMP) which allows message passing to happen only along the topological order of the DAGs (Zhang et al., 2019).
Revised: We use a GNN-based decoder with asynchronous message passing (GNN-AMP) which allows message passing to happen only along the topological order of the DAGs (Zhang et al., 2019).

Original: The graph decoder starts from an initial hidden state h v 0 =
Revised: The graph decoder starts from an initial hidden state h v 0 =

Original: NN init ( z ) , where NN init is an MLP with tanh output and it ﬁnally completes building DAGs by adding a node iteratively.
Revised: NN init ( z ) , where NN init is an MLP with tanh output and it ﬁnally completes building DAGs by adding a node iteratively.

Original: We assume that a node without any predecessors/successors is a start/end node, and if multiple nodes meet aforementioned conditions, we create a virtual starts/end node.
Revised: We assume that a node without any predecessors/successors is a start/end node, and if multiple nodes meet aforementioned conditions, we create a virtual starts/end node.

Original: To generate the i th node v i , we compute the operation type o v i ∈ R 1 × n o over n o operations based on the current graph state h G :
Revised: To generate the i th node v i , we compute the operation type o v i ∈ R 1 × n o over n o operations based on the current graph state h G :

Original: =
Revised: =

Original: h v i − 

Revised: h v i −

Original: and then predict whether the edge exists between the node v i and other existing nodes.
Revised: and then predict whether the edge exists between the node v i and other existing nodes.

Original: h v i is updated following equation (4) when a new edge is added.
Revised: h v i is updated following equation (4) when a new edge is added.

Original: During the propagation process of GNN operator, h ( t ) v i is continuously encoded by aggregating the graph-level information.
Revised: During the propagation process of GNN operator, h ( t ) v i is continuously encoded by aggregating the graph-level information.

Original: Following (Zhang et al., 2019), when we compute the edge probability e { v j ,v i } , we consider nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) in the reverse order to reﬂect information from nodes close to v i to the root node when deciding whether edge connection.
Revised: Following (Zhang et al., 2019), when we compute the edge probability e { v j ,v i } , we consider nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) in the reverse order to reﬂect information from nodes close to v i to the root node when deciding whether edge connection.

Original: Note that the proposed process guarantees the generation of directed acyclic graph since directed edge is always created from existing nodes to a new node.
Revised: Note that the proposed process guarantees the generation of directed acyclic graph since directed edge is always created from existing nodes to a new node.

Original: Note that while DVAE (Zhang et al., 2019) demonstrates their performance on the simple task such as graph-to-graph reconstruction, we aim a more challenging problem of generating a DAG from a latent embedding of a dataset.
Revised: Note that while DVAE (Zhang et al., 2019) demonstrates their performance on the simple task such as graph-to-graph reconstruction, we aim a more challenging problem of generating a DAG from a latent embedding of a dataset.

Original: A. B I - DIRECTIONAL G RAPH E NCODING
Revised: A. B I - DIRECTIONAL G RAPH E NCODING

Original: Graph encoding model is based on GNN-AMP .
Revised: Graph encoding model is based on GNN-AMP .

Original: For a given graph candidate G , we sequentially perform message passing for nodes from the predecessors following the topological order of the DAG
Revised: For a given graph candidate G , we sequentially perform message passing for nodes from the predecessors following the topological order of the DAG

Original: G .
Revised: G .

Original: We iteratively update hidden states h ( t ) v i using the equation (12) by feeding in its predecessors’ hidden states { u ∈ V in v i } .
Revised: We iteratively update hidden states h ( t ) v i using the equation (12) by feeding in its predecessors’ hidden states { u ∈ V in v i } .

Original: [Equation]
Revised: [Equation]

Original: i , m ( t ) v i ) , where m ( t ) v i =
Revised: i , m ( t ) v i ) , where m ( t ) v i =

Original: [Equation]
Revised: [Equation]

Original: For starting node v 0 which the set of predecessors is the empty, we output the zero vector as the hidden state of v 0 .
Revised: For starting node v 0 which the set of predecessors is the empty, we output the zero vector as the hidden state of v 0 .

Original: We use the last hidden states of the ending node as the output of the graph encoder h f .
Revised: We use the last hidden states of the ending node as the output of the graph encoder h f .

Original: The graph encoding process with GNN-AMP can be regarded as a mimic of forwarding propagation of computation ﬂows of neural networks.
Revised: The graph encoding process with GNN-AMP can be regarded as a mimic of forwarding propagation of computation ﬂows of neural networks.

Original: Inspired by the forward-backward propagation algorithm of neural network, we exploit Bi-directional encoding (Zhang et al., 2019) which reverses the node orders to perform the encoding process.
Revised: Inspired by the forward-backward propagation algorithm of neural network, we exploit Bi-directional encoding (Zhang et al., 2019) which reverses the node orders to perform the encoding process.

Original: In this case, the ﬁnal node becomes the starting point (See Figure 3 (c)).
Revised: In this case, the ﬁnal node becomes the starting point (See Figure 3 (c)).

Original: Thus, the backward graph encoder outputs h b , which is the last hidden states of the starting node.
Revised: Thus, the backward graph encoder outputs h b , which is the last hidden states of the starting node.

Original: We concatenate the outputs
Revised: We concatenate the outputs

Original: h f of the forward graph encoder and h b of the backward graph encoder as the ﬁnal output of the Bi-directional graph encoding.
Revised: h f of the forward graph encoder and h b of the backward graph encoder as the ﬁnal output of the Bi-directional graph encoding.

Original: B S EARCH S PACE
Revised: B S EARCH S PACE

Original: Following the NAS-Bench-201 (Dong & Yang, 2020), We explore the search space consisting of 15,625 possible cell-based neural architectures for all experiments.
Revised: Following the NAS-Bench-201 (Dong & Yang, 2020), We explore the search space consisting of 15,625 possible cell-based neural architectures for all experiments.

Original: Macro skeleton is stacked with one stem cell, three stages consisting of 5 cells for each, and a residual block (He et al., 2016) between stages.
Revised: Macro skeleton is stacked with one stem cell, three stages consisting of 5 cells for each, and a residual block (He et al., 2016) between stages.

Original: The stem cell consists of 3-by-3 convolution with 16 channels and cells of the ﬁrst, second and third stages have 16, 32 and 64, respectively.
Revised: The stem cell consists of 3-by-3 convolution with 16 channels and cells of the ﬁrst, second and third stages have 16, 32 and 64, respectively.

Original: Residual blocks have convolution layer with the stride 2 for down-sampling.
Revised: Residual blocks have convolution layer with the stride 2 for down-sampling.

Original: A fully connected layer is attached to the macro skeleton for classiﬁcation.
Revised: A fully connected layer is attached to the macro skeleton for classiﬁcation.

Original: Each cell is DAG which consists of  4 nodes with 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.
Revised: Each cell is DAG which consists of the ﬁxed 4 nodes and the ﬁxed 6 edge connections. For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.

Revised: To effectively encode the operation information as the node features, we represent edges of graphs in NAS-Bench201 as nodes, and nodes of them as edges.

Original: We add a starting node and an ending node to the cell during training.
Revised: Additionally, we add a starting node and an ending node to the cell during training.

Original: All nodes which have no predecessors (suceessors) are connected to the starting (ending) node, which we delete after generating the full neural architectures.
Revised: All nodes which have no predecessors (suceessors) are connected to the starting (ending) node, which we delete after generating the full neural architectures.

Original: C E XPERIMENTAL S ETUP C. D ATASET
Revised: C E XPERIMENTAL S ETUP C. D ATASET

Original: 1) MetaImageNet 447 neural architectures out of the 1,679 tasks used for training MetaD2A generator are searched with GDAS (Dong & Yang, 2019b).
Revised: 1) MetaImageNet 447 neural architectures out of the 1,679 tasks used for training MetaD2A generator are searched with GDAS (Dong & Yang, 2019b).

Original: We observed that the MetaD2A generator trained with tasks including neural architectures retrieved from GDA tend to converge to generate similar cell structures even for different tasks.
Revised: We observed that the MetaD2A generator trained with tasks including neural architectures retrieved from GDA tend to converge to generate similar cell structures even for different tasks.

Original: Thus, we use additional 1,232 training tasks to increase the exploration diversity with the random search as follows:
Revised: Thus, we use additional 1,232 training tasks to increase the exploration diversity with the random search as follows:

Original: We ﬁrst randomly sample ﬁve architecture candidates from 1000 architectures, which are in the rank of top-1000 accuracy on the
Revised: We ﬁrst randomly sample ﬁve architecture candidates from 1000 architectures, which are in the rank of top-1000 accuracy on the

Original: ImageNet-16-120 from NAS-Bench-201 (Dong & Yang, 2020).
Revised: ImageNet-16-120 from NAS-Bench-201 (Dong & Yang, 2020).

Original: Then we train them on the dataset of the given task and select the architecture with the highest accuracy among the ﬁve.
Revised: Then we train them on the dataset of the given task and select the architecture with the highest accuracy among the ﬁve.

Original: Although architectures searched in this manner are not optimal for the corresponding tasks, we observe that neural architectures generated from the latent space formed through training on those tasks significantly improve the search efﬁciency compared with the random sampling, as denoted in Table 5.
Revised: Although architectures searched in this manner are not optimal for the corresponding tasks, we observe that neural architectures generated from the latent space formed through training on those tasks significantly improve the search efﬁciency compared with the random sampling, as denoted in Table 5.

Original: Further, considering collectable database in real-world scenarios are usually not optimal, we believe that this is more realistic approach.
Revised: Further, considering collectable database in real-world scenarios are usually not optimal, we believe that this is more realistic approach.

Original: To collect tasks for MetaD2A predictor, we randomly sample a subset from the meta-training set/meta-test set of MetaImageNet for meta-training/meta-test, respectively, and a neural architecture from the search space of NAS-Bench201.
Revised: To collect tasks for MetaD2A predictor, we randomly sample a subset from the meta-training set/meta-test set of MetaImageNet for meta-training/meta-test, respectively, and a neural architecture from the search space of NAS-Bench201.

Original: Then we train the neural architecture on the given subset to get accuracy.
Revised: Then we train the neural architecture on the given subset to get accuracy.

Original: In this way, we simply collect 3,752/400 tasks for meta-training/meta-test, respectively.
Revised: In this way, we simply collect 3,752/400 tasks for meta-training/meta-test, respectively.

Original: MNIST (LeCun & Cortes, 2010):
Revised: MNIST (LeCun & Cortes, 2010):

Original: This is a standard image classiﬁcation dataset which contains 70K 28 × 28 grey colored images that describe 10 digits.
Revised: This is a standard image classiﬁcation dataset which contains 70K 28 × 28 grey colored images that describe 10 digits.

Original: We upsample the images to 32 × 32 pixels to satisfy the minimum required pixel size of the NAS-Bench 201 due to the residual blocks in the macro skeleton.
Revised: We upsample the images to 32 × 32 pixels to satisfy the minimum required pixel size of the NAS-Bench 201 due to the residual blocks in the macro skeleton.

Original: We use the training/test split from the original dataset, where 60K images are used for training and 10K are used for test.
Revised: We use the training/test split from the original dataset, where 60K images are used for training and 10K are used for test.

Original: 3) SVHN (Netzer et al., 2011):
Revised: 3) SVHN (Netzer et al., 2011):

Original: This dataset consists of 32 × 32 color images where each has a digit with a natural scene background.
Revised: This dataset consists of 32 × 32 color images where each has a digit with a natural scene background.

Original: The number of classes is 10 denoting from digit 1 to 10 and the number of training/test images is 73257/26032, respectively.
Revised: The number of classes is 10 denoting from digit 1 to 10 and the number of training/test images is 73257/26032, respectively.

Original: 4) CIFAR-10 (Krizhevsky et al., 2009):
Revised: 4) CIFAR-10 (Krizhevsky et al., 2009):

Original: This dataset is a popular benchmark dataset for NAS, which consists of 32 × 32 colour images from 10 general object classes.
Revised: This dataset is a popular benchmark dataset for NAS, which consists of 32 × 32 colour images from 10 general object classes.

Original: The training set consists of 50K images, 5K for each class, and the test set consists of 10K images, 1K for each class.
Revised: The training set consists of 50K images, 5K for each class, and the test set consists of 10K images, 1K for each class.

Original: 5) CIFAR-100 (Krizhevsky et al., 2009):
Revised: 5) CIFAR-100 (Krizhevsky et al., 2009):

Original: This dataset consists of colored images from 100 ﬁne-grained general object classes.
Revised: This dataset consists of colored images from 100 ﬁne-grained general object classes.

Original: Each class has 500/100 images for training and test, respectively.
Revised: Each class has 500/100 images for training and test, respectively.

Original: 6) Aircraft (Maji et al., 2013)
Revised: 6) Aircraft (Maji et al., 2013)

Original: This is ﬁne-grained classiﬁcation benchmark dataset containing 10K images from 30 different aircraft classes.
Revised: This is ﬁne-grained classiﬁcation benchmark dataset containing 10K images from 30 different aircraft classes.

Original: We resize all images into 32 × 32.
Revised: We resize all images into 32 × 32.

Original: 7) Oxford-IIIT Pets (Parkhi et al., 2012)
Revised: 7) Oxford-IIIT Pets (Parkhi et al., 2012)

Original: This dataset is for ﬁne-grained classiﬁcation which has 37 breeds of pets with roughly 200 instances for each class.
Revised: This dataset is for ﬁne-grained classiﬁcation which has 37 breeds of pets with roughly 200 instances for each class.

Original: There is no split ﬁle provided, so we use the 85% of the dataset for training and the other 15% are as a test set.
Revised: There is no split ﬁle provided, so we use the 85% of the dataset for training and the other 15% are as a test set.

Original: We also resize all im- ages into 32 × 32.
Revised: We also resize all images into 32 × 32.

Original: For CIFAR10 and CIFAR100 , we used the training, validation, and test splits from the NAS-Bench-201, and use random validation/test splits for MNIST , SVHN , Aircraft , and
Revised: For CIFAR10 and CIFAR100 , we used the training, validation, and test splits from the NAS-Bench-201, and use random validation/test splits for MNIST , SVHN , Aircraft , and

Original: Oxford-IIIT Pets by splitting the test set into two subsets of the same size.
Revised: Oxford-IIIT Pets by splitting the test set into two subsets of the same size.

Original: The validation set is used to update the searching algorithms as a supervision signal and the test set is used to evaluate the performance of the searched architectures.
Revised: The validation set is used to update the searching algorithms as a supervision signal and the test set is used to evaluate the performance of the searched architectures.

Original: C. B ASELINES
Revised: C. B ASELINES

Original: We now brieﬂy describe the baseline models and our MetaD2A model.
Revised: We now brieﬂy describe the baseline models and our MetaD2A model.

Original: 1) RSPS (Li & Talwalkar, 2019)
Revised: 1) RSPS (Li & Talwalkar, 2019)

Original: This method is a combination of random search and weight sharing, which trains randomly sampled sub-graphs from weight shared DAG of the search space.
Revised: This method is a combination of random search and weight sharing, which trains randomly sampled sub-graphs from weight shared DAG of the search space.

Original: The method then selects the best performing sub-graph among the sampled ones as the ﬁnal neural architecture.
Revised: The method then selects the best performing sub-graph among the sampled ones as the ﬁnal neural architecture.

Original: 2) ENAS (Pham et al., 2018)
Revised: 2) ENAS (Pham et al., 2018)

Original: This is an RL-based NAS method, which introduces the controller to select a sub-graph from a large computational graph, which allows efﬁcient architecture search by sharing parameters among the child models and providing empirical performance as the reward.
Revised: This is an RL-based NAS method, which introduces the controller to select a sub-graph from a large computational graph, which allows efﬁcient architecture search by sharing parameters among the child models and providing empirical performance as the reward.

Original: 3) DARTS
Revised: 3) DARTS

Original: (Liu et al., 2019)
Revised: (Liu et al., 2019)

Original: This gradient-based NAS method speeds up the search time by introducing differentiable architecture representations with continuous relaxation of the discrete computational graph.
Revised: This gradient-based NAS method speeds up the search time by introducing differentiable architecture representations with continuous relaxation of the discrete computational graph.

Original: We use the ﬁrst-order DARTS in all experiments. 4) SETN (Dong & Yang, 2019a)
Revised: We use the ﬁrst-order DARTS in all experiments. 4) SETN (Dong & Yang, 2019a)

Original: SETN is an one-shot
Revised: SETN is an one-shot

Original: NAS method, which selectively samples competitive child candidates by learning to evaluate the quality of the candidates based on the validation loss.
Revised: NAS method, which selectively samples competitive child candidates by learning to evaluate the quality of the candidates based on the validation loss.

Original: 5) GDAS (Dong & Yang, 2019b)
Revised: 5) GDAS (Dong & Yang, 2019b)

Original: This is a
Revised: This is a

Original: Gumbel-Softmax based differentiable neural architecture sampler, which is trained to minimize the validation loss with the architecture sampled from DAGs. 6) PC-DARTS (Xu et al., 2020)
Revised: Gumbel-Softmax based differentiable neural architecture sampler, which is trained to minimize the validation loss with the architecture sampled from DAGs. 6) PC-DARTS (Xu et al., 2020)

Original: This is a gradient-based NAS which partially samples channels to apply operations, to improve the efﬁciency of NAS in terms of memory usage and search time compared to DARTS.
Revised: This is a gradient-based NAS which partially samples channels to apply operations, to improve the efﬁciency of NAS in terms of memory usage and search time compared to DARTS.

Original: We exploit the code at https://github.com/yuhuixu1993/PC-DARTS .
Revised: We exploit the code at https://github.com/yuhuixu1993/PC-DARTS .

Original: 7) DrNAS (Chen et al., 2020)
Revised: 7) DrNAS (Chen et al., 2020)

Original: This is a NAS approach that introduces Dirichlet distribution to approximate the architecture distribution, to enhance the generalization performance of differentiable architecture search.
Revised: This is a NAS approach that introduces Dirichlet distribution to approximate the architecture distribution, to enhance the generalization performance of differentiable architecture search.

Original: We use the code at https://github.com/xiangning-chen/DrNAS .
Revised: We use the code at https://github.com/xiangning-chen/DrNAS .

Original: We report the results on CIFAR10 and
Revised: We report the results on CIFAR10 and

Original: CIFAR100 in this paper using the provided code from the authors on the split set of NAS-Benchwhile their reported results in the paper of the authors are 94 .
Revised: CIFAR100 in this paper using the provided code from the authors on the split set of NAS-Benchwhile their reported results in the paper of the authors are 94 .

Original: 37 and 73 .
Revised: 37 and 73 .

Original: 51 , respectively on random training/test splits on CIFAR10 and CIFAR100 .
Revised: 51 , respectively on random training/test splits on CIFAR10 and CIFAR100 .

Original: 8) ResNet (He et al., 2016)
Revised: 8) ResNet (He et al., 2016)

Original: This is a convolutional network which connects the output of previous layer as input to the current layer.
Revised: This is a convolutional network which connects the output of previous layer as input to the current layer.

Original: It has achieved impressive performance on many challenging image tasks.
Revised: It has achieved impressive performance on many challenging image tasks.

Original: We use ResNet56 in all experiments. 9)
Revised: We use ResNet56 in all experiments. 9)

Original: REA (Real et al., 2019)
Revised: REA (Real et al., 2019)

Original: This is an evolutional-based search method by using aging based tournament selection, showing evolution can work in NAS. 10) RS (Bergstra & Bengio, 2012)
Revised: This is an evolutional-based search method by using aging based tournament selection, showing evolution can work in NAS. 10) RS (Bergstra & Bengio, 2012)

Original: This is based on random search and we randomly samples architectures until the total time of training and evaluation reaches the budget.
Revised: This is based on random search and we randomly samples architectures until the total time of training and evaluation reaches the budget.

Original: 11) REINFORCE (Williams, 1992)
Revised: 11) REINFORCE (Williams, 1992)

Original: This is a RL-based NAS.
Revised: This is a RL-based NAS.

Original: We reward the model with the validation accuracy after 12 epochs of training.
Revised: We reward the model with the validation accuracy after 12 epochs of training.

Original: 12) BOHB (Falkner et al., 2018)
Revised: 12) BOHB (Falkner et al., 2018)

Original: This combines the strengths of tree-structured parzen estimator based baysian optimization and hyperband, performing better than standard baysian optimization methods. 13) DVAE (Zhang et al., 2019)
Revised: This combines the strengths of tree-structured parzen estimator based baysian optimization and hyperband, performing better than standard baysian optimization methods. 13) DVAE (Zhang et al., 2019)

Original: This uses bayesian optimization to search for the best performing DAGs by using the latent space produced by D-VAE.
Revised: This uses bayesian optimization to search for the best performing DAGs by using the latent space produced by D-VAE.

Original: We use the code at https://github.com/muhanzhang/D-VAE 14)
Revised: We use the code at https://github.com/muhanzhang/D-VAE 14)

Original: FNA (Fang et al., 2020)
Revised: FNA (Fang et al., 2020)

Original: This starts from a pretrained seed network and adapts it to a target dataset using a differentiable NAS.
Revised: This starts from a pretrained seed network and adapts it to a target dataset using a differentiable NAS.

Original: When searching and training the architecture, it uses parameter remapping.
Revised: When searching and training the architecture, it uses parameter remapping.

Original: We make the seed network by retrieving the best-performing architecture from NASBench-201 and train the network on CIFAR-100.
Revised: We make the seed network by retrieving the best-performing architecture from NASBench-201 and train the network on CIFAR-100.

Original: The total cost for pre-training is 6856(s).
Revised: The total cost for pre-training is 6856(s).

Original: Here we use DARTS for architecture search and we use the same procedure of parameter remapping. 15)
Revised: Here we use DARTS for architecture search and we use the same procedure of parameter remapping. 15)

Original: MetaD2A (Ours)
Revised: MetaD2A (Ours)

Original: This is our meta-NAS framework described in section 3, which can stochastically generate task-dependent computational graphs from a given dataset, and use the performance predictor to select the best performing candidates.
Revised: This is our meta-NAS framework described in section 3, which can stochastically generate task-dependent computational graphs from a given dataset, and use the performance predictor to select the best performing candidates.

Original: We follow the same settings of NAS-Bench-201 (Dong & Yang, 2020) for all baselines and use the code at https://github.com/D-X-Y/AutoDL-Projects except for 6), 7) and 13).
Revised: We follow the same settings of NAS-Bench-201 (Dong & Yang, 2020) for all baselines and use the code at https://github.com/D-X-Y/AutoDL-Projects except for 6), 7) and 13).

Original: C. I MPLEMENTATION D ETAILS
Revised: C. I MPLEMENTATION D ETAILS

Original: [Table]
Revised: [Table]

Original: We adopt the teacher forcing training strategy (Jin et al., 2018), which performs the current decoding process after correcting the decoded graph as the true graph until the previous step.
Revised: We adopt the teacher forcing training strategy (Jin et al., 2018), which performs the current decoding process after correcting the decoded graph as the true graph until the previous step.

Original: This strategy is only used during meta-training and we progress subsequent generation based on the currently decoded graph part without the true graph information in the meta-test.
Revised: This strategy is only used during meta-training and we progress subsequent generation based on the currently decoded graph part without the true graph information in the meta-test.

Original: We use mini-batch gradient descent to train the model with Eq.
Revised: We use mini-batch gradient descent to train the model with Eq.

Original: The values of hyperparameters which we used for MetaD2A model in this paper are described in Table 7.
Revised: The values of hyperparameters which we used for MetaD2A model in this paper are described in Table 7.

Original: For unseen datasets of MetaImageNet tasks, MNIST ,
Revised: For unseen datasets of MetaImageNet tasks, MNIST ,

Original: SVHN , Aircraft , and Oxford-IIIT Pets , we sample n s = 1000 neural architecture candidates by
Revised: SVHN , Aircraft , and Oxford-IIIT Pets , we sample n s = 1000 neural architecture candidates by

Original: MetaD2A generator, select the best one neural architecture ( N = 1 ) with the highest predicted accuracy computed by the meta-performance predictor, and report the mean accuracy of trained the search neural architecture run with 3 different random seeds.
Revised: MetaD2A generator, select the best one neural architecture ( N = 1 ) with the highest predicted accuracy computed by the meta-performance predictor, and report the mean accuracy of trained the search neural architecture run with 3 different random seeds.

Original: For CIFAR10 and CIFAR100 , we sample n s = 1000 neural architecture candidates by MetaD2A generator, select N = 40 neural architecture candidates in order of high predicted accuracies computed by the meta-performance predictor, retrieve the real accuracies of N = 40 neural architecture candidates from NAS-Bench 201 (Dong & Yang, 2020), and report the highest real accuracy among real accuracies as the ﬁnal performance of the searched neural architecture by MetaD2A. To train searched neural architectures for all datasets, we follow the hyperparameter setting of NAS-Bench-201 (Dong & Yang, 2020), which is used for training searched neural architectures on CIFAR10 and CIFAR100.
Revised: For CIFAR10 and CIFAR100 , we sample n s = 1000 neural architecture candidates by MetaD2A generator, select N = 40 neural architecture candidates in order of high predicted accuracies computed by the meta-performance predictor, retrieve the real accuracies of N = 40 neural architecture candidates from NAS-Bench 201 (Dong & Yang, 2020), and report the highest real accuracy among real accuracies as the ﬁnal performance of the searched neural architecture by MetaD2A. To train searched neural architectures for all datasets, we follow the hyperparameter setting of NAS-Bench-201 (Dong & Yang, 2020), which is used for training searched neural architectures on CIFAR10 and CIFAR100.

Original: While we report accuracy after training 50 epoch for MNIST, the accuracy of 200 epoch are reported for all datasets except MNIST.
Revised: While we report accuracy after training 50 epoch for MNIST, the accuracy of 200 epoch are reported for all datasets except MNIST.

Original: D M ORE R ESULTS D. T HE EFFECTIVENESS OF SET - DEPENDENT PREDICTOR
Revised: D M ORE R ESULTS D. T HE EFFECTIVENESS OF SET - DEPENDENT PREDICTOR

Original: [Table]
Revised: [Table]

Original: To demonstrate the set-dependent predictor on multiple tasks, we compare our preidctor (MetaD2AGD) with other competitive setindependent predictors such as linaer layer-based, LSTM-based (Hochreiter & Schmidhuber, 1997), GCN-based (Kipf & Welling, 2016) predictors.
Revised: To demonstrate the set-dependent predictor on multiple tasks, we compare our preidctor (MetaD2AGD) with other competitive setindependent predictors such as linaer layer-based, LSTM-based (Hochreiter & Schmidhuber, 1997), GCN-based (Kipf & Welling, 2016) predictors.

Original: Note that the proposed predictor can accumulate meta-knowledge from multiple tasks to handle multiple datasets with once training while none of other set-independent predictors can do.
Revised: Note that the proposed predictor can accumulate meta-knowledge from multiple tasks to handle multiple datasets with once training while none of other set-independent predictors can do.

Original: Also, we add our predictor with graph encoder (MetaD2A-G) which is set-independent.
Revised: Also, we add our predictor with graph encoder (MetaD2A-G) which is set-independent.

Original: We train with them on the multiple tasks of MetaImageNet and then compute Pearson correlation coefﬁcient between the predicted accuracy and the true accuracy on 400 unseen MetaImageNet tasks.
Revised: We train with them on the multiple tasks of MetaImageNet and then compute Pearson correlation coefﬁcient between the predicted accuracy and the true accuracy on 400 unseen MetaImageNet tasks.

Original: The higher score means the predictor estimates better the true accuracy.
Revised: The higher score means the predictor estimates better the true accuracy.

Original: We observe that linear layersbased or LSTM-based predictors fail to be learned on multiple tasks.
Revised: We observe that linear layersbased or LSTM-based predictors fail to be learned on multiple tasks.

Original: We conjecture the reason is that they are too simple and ignore the change in accuracy dependent on data even the same graph.
Revised: We conjecture the reason is that they are too simple and ignore the change in accuracy dependent on data even the same graph.

Original: GCN-based predictor quite successfully works well even it also set-independent.
Revised: GCN-based predictor quite successfully works well even it also set-independent.

Original: However, our
Revised: However, our

Original: MetaD2A-DG, which can consider both input dataset and graph, outperforms the GCN predictor.
Revised: MetaD2A-DG, which can consider both input dataset and graph, outperforms the GCN predictor.

Original: Even MetaD2A-G, which uses graph encoder only performs better than the GCN predictor due to the asynchronous message passing.
Revised: Even MetaD2A-G, which uses graph encoder only performs better than the GCN predictor due to the asynchronous message passing.

Original: Our predictor can search architecture for new dataset instantly by designed for capturing both dataset and graph and trained with amortized learning to accumulate meta-knowledge.
Revised: Our predictor can search architecture for new dataset instantly by designed for capturing both dataset and graph and trained with amortized learning to accumulate meta-knowledge.

Original: D. P ERFORMANCE ON FEW - SHOT CLASSIFICATION TASK
Revised: D. P ERFORMANCE ON FEW - SHOT CLASSIFICATION TASK

Original: [Table]
Revised: [Table]

Original: We further compared our method against Meta-NAS methods (Kim et al., 2018; Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) on few-shot classiﬁcation tasks, which are the main setting existing MetaNAS methods consider.
Revised: We further compared our method against Meta-NAS methods (Kim et al., 2018; Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) on few-shot classiﬁcation tasks, which are the main setting existing MetaNAS methods consider.

Original: Note that we can generate an architecture for each task almost instantly with MetaD2A for the few-shot task.
Revised: Note that we can generate an architecture for each task almost instantly with MetaD2A for the few-shot task.

Original: Following (Elsken et al., 2020; Lian et al., 2019), we adopt bi-level optimization (e.g., MAML framework) to meta-learn initial weight on a meta-training set of mini-imagenet.
Revised: Following (Elsken et al., 2020; Lian et al., 2019), we adopt bi-level optimization (e.g., MAML framework) to meta-learn initial weight on a meta-training set of mini-imagenet.

Original: As shown in the Table 9, the few-shot classiﬁcation results on MiniImageNet further clearly show the
Revised: As shown in the Table 9, the few-shot classiﬁcation results on MiniImageNet further clearly show the

Original: MetaD2A’s effectiveness over existing Meta-NAS methods, as well as conventional meta-learning without NAS.
Revised: MetaD2A’s effectiveness over existing Meta-NAS methods, as well as conventional meta-learning without NAS.

Original: D. TSNE VISUALIZATION OF Z
Revised: D. TSNE VISUALIZATION OF Z

Original: [Figure]
Revised: [Figure]

Original: DATASETS
Revised: DATASETS

Original: [Figure]
Revised: [Figure]

Original: b) .
Revised: b) .

Original: D. TSNE VISUALIZATION OF Z OF DATASETS AND GRAPHS
Revised: D. TSNE VISUALIZATION OF Z OF DATASETS AND GRAPHS

Original: [Figure] [Figure]
Revised: [Figure] [Figure]