Original: R APID N EURAL A RCHITECTURE S EARCH BY L EARNING TO G ENERATE G RAPHS FROM D ATASETS
Revised: R APID N EURAL A RCHITECTURE S EARCH BY L EARNING TO G ENERATE G RAPHS FROM D ATASETS

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform humandesigned networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets).
Revised: Despite the success of recent Neural Architecture Search (NAS) methods on various tasks which have shown to output networks that largely outperform humandesigned networks, conventional NAS methods have mostly tackled the optimization of searching for the network architecture for a single task (dataset), which does not generalize well across multiple tasks (datasets).

Original: Moreover, since such task-speciﬁc methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited.
Revised: Moreover, since such task-speciﬁc methods search for a neural architecture from scratch for every given task, they incur a large computational cost, which is problematic when the time and monetary budget are limited.

Original: In this paper, we propose an efﬁcient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search  a neural architecture for a novel dataset.
Revised: In this paper, we propose an efﬁcient NAS framework that is trained once on a database consisting of datasets and pretrained networks and can rapidly search for a neural architecture for a novel dataset.

Original: The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning.
Revised: The proposed MetaD2A (Meta Dataset-to-Architecture) model can stochastically generate graphs (architectures) from a given set (dataset) via a cross-modal latent space learned with amortized meta-learning.

Original: Moreover, we also propose a metaperformance predictor to estimate and select the best architecture without direct training on target datasets.
Revised: Moreover, we also propose a meta-performance predictor to estimate and select the best architecture without direct training on target datasets.

Original: The experimental results demonstrate that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Benchsearch space successfully generalizes to multiple benchmark datasets including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU seconds.
Revised: The experimental results demonstrate that our model meta-learned on subsets of ImageNet-1K and architectures from NAS-Bench 201 search space successfully generalizes to multiple unseen datasets including CIFAR-10 and CIFAR-100, with an average search time of 33 GPU seconds.

Original: Even under a large search space, MetaD2A is 5.5K times faster than NSGANetV2, a transferable NAS method, with comparable performance.
Revised: Even under MobileNetV3 search space, MetaD2A is 5.5K times faster than NSGANetV2, a transferable NAS method, with comparable performance.

Original: We believe that the MetaD2A proposes a new research direction for rapid NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years.
Revised: We believe that the MetaD2A proposes a new research direction for rapid NAS as well as ways to utilize the knowledge from rich databases of datasets and architectures accumulated over the past years.

Original: Code is available at https://github.com/HayeonLee/MetaD2A.
Revised: Code is available at https://github.com/HayeonLee/MetaD2A.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The rapid progress in the design of neural architectures has largely contributed to the success of deep learning on many applications (Krizhevsky et al., 2012; Cho et al., 2014;
Revised: The rapid progress in the design of neural architectures has largely contributed to the success of deep learning on many applications (Krizhevsky et al., 2012; Cho et al., 2014;

Original: He et al., 2016; Szegedy et al.; Vaswani et al., 2017; Zhang et al., 2018).
Revised: He et al., 2016; Szegedy et al.; Vaswani et al., 2017; Zhang et al., 2018).

Original: However, due to the vast search space, designing a novel neural architecture requires a time-consuming trial-and-error search by human experts.
Revised: However, due to the vast search space, designing a novel neural architecture requires a time-consuming trial-and-error search by human experts.

Original: To tackle such inefﬁciency in the manual architecture design process, researchers have proposed various Neural Architecture Search (NAS) methods that automatically search for optimal architectures, achieving models with impressive performances on various tasks that outperform human-designed counterparts (Baker et al., 2017; Zoph & Le, 2017; Kandasamy et al., 2018; Liu et al., 2018;
Revised: To tackle such inefﬁciency in the manual architecture design process, researchers have proposed various Neural Architecture Search (NAS) methods that automatically search for optimal architectures, achieving models with impressive performances on various tasks that outperform human-designed counterparts (Baker et al., 2017; Zoph & Le, 2017; Kandasamy et al., 2018; Liu et al., 2018;

Original: Luo et al., 2018; Pham et al., 2018; Liu et al., 2019; Xu et al., 2020; Chen et al., 2021).
Revised: Luo et al., 2018; Pham et al., 2018; Liu et al., 2019; Xu et al., 2020; Chen et al., 2021).

Original: Recently, large benchmarks for NAS (NAS-101, NAS-201)
Revised: Recently, large benchmarks for NAS (NAS-101, NAS-201)

Original: (Ying et al., 2019; Dong & Yang, 2020) have been introduced, which provide databases of architectures and their performances on benchmark datasets.
Revised: (Ying et al., 2019; Dong & Yang, 2020) have been introduced, which provide databases of architectures and their performances on benchmark datasets.

Original: Yet, most conventional NAS methods cannot beneﬁt from the availability of such databases, due to their task-speciﬁc nature which requires repeatedly training the model from scratch for each new dataset (See Figure 1 Left).
Revised: Yet, most conventional NAS methods cannot beneﬁt from the availability of such databases, due to their task-speciﬁc nature which requires repeatedly training the model from scratch for each new dataset (See Figure 1 Left).

Original: Thus, searching for an architecture for a new task (dataset) may require a large number of computations, which may be problematic when the time and mon- etary budget are limited.
Revised: Thus, searching for an architecture for a new task (dataset) may require a large number of computations, which may be problematic when the time and monetary budget are limited.

Original: How can we then exploit the vast knowledge of neural architectures that have been already trained on a large number of datasets, to better generalize over an unseen task?
Revised: How can we then exploit the vast knowledge of neural architectures that have been already trained on a large number of datasets, to better generalize over an unseen task?

Original: [Figure]
Revised: [Figure]

Original: Training

Original: Training

Original: [Figure]

Original: [Figure]

Original: [Figure] Dataset [Figure] In this paper, we introduce amortized meta-learning for NAS, where the goal is to learn a NAS model that generalizes well over the task distribution, rather than a single task, to utilize the accumulated meta-knowledge to new target tasks.
Revised: In this paper, we introduce amortized meta-learning for NAS, where the goal is to learn a NAS model that generalizes well over the task distribution, rather than a single task, to utilize the accumulated meta-knowledge to new target tasks.

Original: Speciﬁcally, we propose an efﬁcient NAS framework that is trained once from a database containing datasets and their corresponding neural architectures and then generalizes to multiple datasets for searching neural architectures, by learning to generate a neural architecture from a given dataset.
Revised: Speciﬁcally, we propose an efﬁcient NAS framework that is trained once from a database containing datasets and their corresponding neural architectures and then generalizes to multiple datasets for searching neural architectures, by learning to generate a neural architecture from a given dataset.

Original: The proposed MetaD2A (Meta Dataset-to-Architecture) framework consists of a set encoder and a graph decoder, which are used to learn a cross-modal latent space for datasets and neural architectures via amortized inference.
Revised: The proposed MetaD2A (Meta Dataset-to-Architecture) framework consists of a set encoder and a graph decoder, which are used to learn a cross-modal latent space for datasets and neural architectures via amortized inference.

Original: For a new dataset, MetaD2A stochastically generates neural architecture candidates from set-dependent latent representations, which are encoded from a new dataset, and selects the ﬁnal neural architecture based on their predicted accuracies by a performance predictor, which is also trained with amortized meta-learning.
Revised: For a new dataset, MetaD2A stochastically generates neural architecture candidates from set-dependent latent representations, which are encoded from a new dataset, and selects the ﬁnal neural architecture based on their predicted accuracies by a performance predictor, which is also trained with amortized meta-learning.

Original: The proposed meta-learning framework reduces the search cost from O(N) to O(1) for multiple datasets due to no training on target datasets.
Revised: The proposed meta-learning framework reduces the search cost from O(N) to O(1) for multiple datasets due to no training on target datasets.

Original: After one-time building cost, our model only takes just a few GPU seconds to search for neural architecture on an unseen dataset (See Figure 1).
Revised: After one-time building cost, our model only takes just a few GPU seconds to search for neural architecture on an unseen dataset (See Figure 1).

Original: We meta-learn the proposed MetaD2A on subsets of ImageNet-1K and neural architectures from the NAS-Bench201 search space.
Revised: We meta-learn the proposed MetaD2A on subsets of ImageNet-1K and neural architectures from the NAS-Bench201 search space.

Original: Then we validate it to search for neural architectures on multiple unseen datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets.
Revised: Then we validate it to search for neural architectures on multiple unseen datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets.

Original: In this experiment, our meta-learned model obtains a neural architecture within 33 GPU seconds on average without direct training on a target dataset and largely outperforms all baseline NAS models.
Revised: In this experiment, our meta-learned model obtains a neural architecture within 33 GPU seconds on average without direct training on a target dataset and largely outperforms all baseline NAS models.

Original: Further, we compare our model with representative transferable NAS method (Lu et al., 2020) on MobileNetV3 search space.
Revised: Further, we compare our model with representative transferable NAS method (Lu et al., 2020) on MobileNetV3 search space.

Original: We meta-learn our model on subsets of ImageNet-1K and neural architectures from the MobileNetV3 search space.
Revised: We meta-learn our model on subsets of ImageNet-1K and neural architectures from the MobileNetV3 search space.

Original: The meta-learned our model successfully generalizes, achieving extremely fast search with competitive performance on four unseen datasets such as CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets.
Revised: The meta-learned our model successfully generalizes, achieving extremely fast search with competitive performance on four unseen datasets such as CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets.

Original: To summarize, our contribution in this work is threefold:
Revised: To summarize, our contribution in this work is threefold:

Original: • We propose a novel NAS framework, MetaD2A, which rapidly searches for a neural architecture on a new dataset, by sampling architectures from latent embeddings of the given dataset then selecting the best one based on their predicted performances.
Revised: • We propose a novel NAS framework, MetaD2A, which rapidly searches for a neural architecture on a new dataset, by sampling architectures from latent embeddings of the given dataset then selecting the best one based on their predicted performances.

Original: • To this end, we propose to learn a cross-modal latent space of datasets and architectures, by performing amortized meta-learning, using a set encoder and a graph decoder on subsets of ImageNet-1K. •
Revised: • To this end, we propose to learn a cross-modal latent space of datasets and architectures, by performing amortized meta-learning, using a set encoder and a graph decoder on subsets of ImageNet-1K. •

Original: The meta-learned our model successfully searches for neural architectureson multiple unseen datasets and achieves state-of-the-art performance
Revised: The meta-learned our model successfully searches for neural architectures

Original: on  them in NAS-Bench201 search space, especially searching for architectures within 33 GPU seconds on average.
Revised: on multiple unseen datasets and achieves state-of-the-art performance on them in NAS-Bench201 search space, especially searching for architectures within 33 GPU seconds on average.

Original: [Figure]
Revised: [Figure]

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Architecture Search (NAS) NAS is an automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space.
Revised: Neural Architecture Search (NAS) NAS is an automated architecture search process which aims to overcome the suboptimality of manual architecture designs when exploring the extensive search space.

Original: NAS methods can be roughly categorized into reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018;
Revised: NAS methods can be roughly categorized into reinforcement learning-based methods (Zoph & Le, 2017; Zoph et al., 2018;

Original: Pham et al., 2018), evolutionary algorithm-based methods (Real et al., 2019; Lu et al., 2020), and gradient-based methods (Liu et al., 2019; Cai et al., 2019; Luo et al., 2018; Dong & Yang, 2019b; Chen et al., 2021; Xu et al., 2020; Fang et al., 2020).
Revised: Pham et al., 2018), evolutionary algorithm-based methods (Real et al., 2019; Lu et al., 2020), and gradient-based methods (Liu et al., 2019; Cai et al., 2019; Luo et al., 2018; Dong & Yang, 2019b; Chen et al., 2021; Xu et al., 2020; Fang et al., 2020).

Original: Among existing approaches, perhaps the most relevant approach to ours is NAO (Luo et al., 2018), which maps DAGs onto a continuous latent embedding space.
Revised: Among existing approaches, perhaps the most relevant approach to ours is NAO (Luo et al., 2018), which maps DAGs onto a continuous latent embedding space.

Original: However, while NAO performs graph reconstruction for a single task, ours generates data-dependent Directed Acyclic Graphs (DAGs) across multiple tasks.
Revised: However, while NAO performs graph reconstruction for a single task, ours generates data-dependent Directed Acyclic Graphs (DAGs) across multiple tasks.

Original: Another important open problem in NAS is reducing the tremendous computational cost resulting from the large search space (Cai et al., 2019; Liu et al., 2018; Pham et al., 2018; Liu et al., 2019;
Revised: Another important open problem in NAS is reducing the tremendous computational cost resulting from the large search space (Cai et al., 2019; Liu et al., 2018; Pham et al., 2018; Liu et al., 2019;

Original: Chen et al., 2021).
Revised: Chen et al., 2021).

Original: GDAS (Dong & Yang, 2019b) tackles this by optimizing sampled sub-graphs of
Revised: GDAS (Dong & Yang, 2019b) tackles this by optimizing sampled sub-graphs of

Original: DAG.
Revised: DAG.

Original: PC-DARTS
Revised: PC-DARTS

Original: (Xu et al., 2020) reduces GPU overhead and search time by partially selecting channel connections.
Revised: (Xu et al., 2020) reduces GPU overhead and search time by partially selecting channel connections.

Original: However, due to the task-speciﬁc nature of those methods, they should be retrained from the scratch for each new unseen task repeatedly and each will take a few GPU hours.
Revised: However, due to the task-speciﬁc nature of those methods, they should be retrained from the scratch for each new unseen task repeatedly and each will take a few GPU hours.

Original: The accuracy-predictor-based transferable NAS called NSGANetV2 (Lu et al., 2020) alleviates this issue by adapting the ImageNet-1K pre-trained network to multiple target datasets, however, this method is still expensive due to adapting procedure on each dataset.
Revised: The accuracy-predictor-based transferable NAS called NSGANetV2 (Lu et al., 2020) alleviates this issue by adapting the ImageNet-1K pre-trained network to multiple target datasets, however, this method is still expensive due to adapting procedure on each dataset.

Original: Meta-learning Meta-learning (learning to learn) aims to train a model to generalize over a distribution of tasks, such that it can rapidly adapt to a new task (Vinyals et al., 2016;
Revised: Meta-learning Meta-learning (learning to learn) aims to train a model to generalize over a distribution of tasks, such that it can rapidly adapt to a new task (Vinyals et al., 2016;

Original: Snell et al., 2017;
Revised: Snell et al., 2017;

Original: Finn et al., 2017; Nichol et al., 2018;
Revised: Finn et al., 2017; Nichol et al., 2018;

Original: Lee et al., 2019b;
Revised: Lee et al., 2019b;

Original: Hou et al., 2019).
Revised: Hou et al., 2019).

Original: Recently, LEO (Rusu et al., 2019) proposed a scalable meta-learning framework which learns the latent generative representations of model parameters for a given data in a low-dimensional space for few-shot classiﬁcation.
Revised: Recently, LEO (Rusu et al., 2019) proposed a scalable meta-learning framework which learns the latent generative representations of model parameters for a given data in a low-dimensional space for few-shot classiﬁcation.

Original: Similarly to LEO (Rusu et al., 2019), our method learns a low-dimensional latent embedding space, but we learn a cross-modal space for both datasets and models for task-dependent model generation.
Revised: Similarly to LEO (Rusu et al., 2019), our method learns a low-dimensional latent embedding space, but we learn a cross-modal space for both datasets and models for task-dependent model generation.

Original: Neural Architecture Search with Meta-Learning Recent NAS methods with gradient-based meta-learning (Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) have shown promising results on adapting to different tasks.
Revised: Neural Architecture Search with Meta-Learning Recent NAS methods with gradient-based meta-learning (Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) have shown promising results on adapting to different tasks.

Original: However, they are only applicable on small scale tasks such as few-shot classiﬁcation tasks (Elsken et al., 2020; Lian et al., 2019) and require high-computation time, due to the multiple unrolling gradient steps for one meta-update of each task.
Revised: However, they are only applicable on small scale tasks such as few-shot classiﬁcation tasks (Elsken et al., 2020; Lian et al., 2019) and require high-computation time, due to the multiple unrolling gradient steps for one meta-update of each task.

Original: While some attempt to bypass the bottleneck with a ﬁrst-order approximation (Lian et al., 2019; Shaw et al., 2019) or parallel computations with GPUs (Shaw et al., 2019), but their scalability is intrinsically limited due to gradient updates over a large number of tasks.
Revised: While some attempt to bypass the bottleneck with a ﬁrst-order approximation (Lian et al., 2019; Shaw et al., 2019) or parallel computations with GPUs (Shaw et al., 2019), but their scalability is intrinsically limited due to gradient updates over a large number of tasks.

Original: To tackle such a scalability issue, we perform amortized inference over the multiple tasks by encoding a dataset into the low-dimensional latent vector and exploit fast GNN propagation instead of the expensive gradient update.
Revised: To tackle such a scalability issue, we perform amortized inference over the multiple tasks by encoding a dataset into the low-dimensional latent vector and exploit fast GNN propagation instead of the expensive gradient update.

Original:  M ETHOD
Revised:  M ETHOD

Original: Our goal is to output a high-performing neural architecture for a given dataset rapidly by learning the prior knowledge obtained from the rich database consisting of datasets and their corresponding neural architectures.
Revised: Our goal is to output a high-performing neural architecture for a given dataset rapidly by learning the prior knowledge obtained from the rich database consisting of datasets and their corresponding neural architectures.

Original: To this end, we propose Meta Dataset-to-Architecture (MetaD2A) framework which learns the cross-modal latent space of datasets and their neural architectures.
Revised: To this end, we propose Meta Dataset-to-Architecture (MetaD2A) framework which learns the cross-modal latent space of datasets and their neural architectures.

Original: Further, we introduce a meta-performance predictor, which predicts accuracies of given architectures without training the predictor on an unseen target dataset.
Revised: Further, we introduce a meta-performance predictor, which predicts accuracies of given architectures without training the predictor on an unseen target dataset.

Original: Overview of the proposed approach is illustrated in Figure 1.
Revised: Overview of the proposed approach is illustrated in Figure 1.

Original: 3. M ETA -T RAINING NAS MODEL
Revised: 3. M ETA -T RAINING NAS MODEL

Original: To formally deﬁne the problem, let us assume that we have a source database of N τ number of tasks, where each task τ
Revised: To formally deﬁne the problem, let us assume that we have a source database of N τ number of tasks, where each task τ

Original: = {D , G , s } consists of a dataset D , a neural architecture represented as a
Revised: = {D , G , s } consists of a dataset D , a neural architecture represented as a

Original: Directed Acyclic Graph (DAG) G and an accuracy s obtained from the neural architecture G trained on D .
Revised: Directed Acyclic Graph (DAG) G and an accuracy s obtained from the neural architecture G trained on D .

Original: In the meta-training phase, the both dataset-to-architecture generator and meta-predictor learn to generalize over task distribution p ( τ ) using the source database.
Revised: In the meta-training phase, the both dataset-to-architecture generator and meta-predictor learn to generalize over task distribution p ( τ ) using the source database.

Original: We describe how to empirically construct source database in Section 4.1.1.
Revised: We describe how to empirically construct source database in Section 4.1.1.

Original: 3.1. L EARNING TO GENERATE GRAPHS FROM DATASETS
Revised: 3.1. L EARNING TO GENERATE GRAPHS FROM DATASETS

Original: We propose a dataset-to-architecture generator which takes a dataset and then generates high-quality architecture candidates for the set.
Revised: We propose a dataset-to-architecture generator which takes a dataset and then generates high-quality architecture candidates for the set.

Original: We want the generator to generate even novel architectures, which are not contained in the source database, at meta-test.
Revised: We want the generator to generate even novel architectures, which are not contained in the source database, at meta-test.

Original: Thus, the generator learns the continuous cross-modal latent space Z of datasets and neural architectures from the source database.
Revised: Thus, the generator learns the continuous cross-modal latent space Z of datasets and neural architectures from the source database.

Original: For each task τ , the generator encodes dataset D as a vector z through the set encoder q φ ( z |D ) parameterized by φ and then decodes a new graph ˜ G from z which are sampled from the prior p ( z ) by using the graph decoder p θ ( G| z ) parameterized by θ .
Revised: For each task τ , the generator encodes dataset D as a vector z through the set encoder q φ ( z |D ) parameterized by φ and then decodes a new graph ˜ G from z which are sampled from the prior p ( z ) by using the graph decoder p θ ( G| z ) parameterized by θ .

Original: Then, our goal is that ˜ G generated from D to be the true
Revised: Then, our goal is that ˜ G generated from D to be the true

Original: G which is pair of D .
Revised: G which is pair of D .

Original: We meta-learn the generator using set-amortized inference, by maximizing the approximated evidence lower bound (ELBO) as follows:
Revised: We meta-learn the generator using set-amortized inference, by maximizing the approximated evidence lower bound (ELBO) as follows:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: L
Revised: L

Original: [Equation]
Revised: [Equation]

Original: Each dimension of the prior p ( z ) factorizes into N (0 , 1) .
Revised: Each dimension of the prior p ( z ) factorizes into N (0 , 1) .

Original: L τKL is the KL divergence between two multivariate Gaussian distributions which has a simple closed form (Kingma & Welling, 2014) and λ is the scalar weighting value.
Revised: L τKL is the KL divergence between two multivariate Gaussian distributions which has a simple closed form (Kingma & Welling, 2014) and λ is the scalar weighting value.

Original: Using the reparameterization trick on z , we optimize the above objective by stochastic gradient variational Bayes (Kingma & Welling, 2014).
Revised: Using the reparameterization trick on z , we optimize the above objective by stochastic gradient variational Bayes (Kingma & Welling, 2014).

Original: We use a set encoder described in Section 3.1.3 and we adopt a Graph Neural Network (GNN)-based decoder for directed acyclic graph (DAG)s (Zhang et al., 2019), which allows message passing to happen only along the topological order of the DAGs.
Revised: We use a set encoder described in Section 3.1.3 and we adopt a Graph Neural Network (GNN)-based decoder for directed acyclic graph (DAG)s (Zhang et al., 2019), which allows message passing to happen only along the topological order of the DAGs.

Original: For detailed descriptions for the generator, see Section A of Suppl.
Revised: For detailed descriptions for the generator, see Section A of Suppl.

Original: 3.1. M ETA -P ERFORMANCE P REDICTOR
Revised: 3.1. M ETA -P ERFORMANCE P REDICTOR

Original: While many performance predictor for NAS have been proposed (Luo et al., 2018; Cai et al., 2020;
Revised: While many performance predictor for NAS have been proposed (Luo et al., 2018; Cai et al., 2020;

Original: Lu et al., 2020; Zhou et al., 2020; Tang et al., 2020), those performance predictors repeatedly collect architecture-accuracy database for each new dataset, which results in huge total cost on many datasets.
Revised: Lu et al., 2020; Zhou et al., 2020; Tang et al., 2020), those performance predictors repeatedly collect architecture-accuracy database for each new dataset, which results in huge total cost on many datasets.

Original: Thus, the proposed predictor f ω ( s |D , G ) takes a dataset as well as graph as an input to support multiple datasets, while the existing performance predictor takes a graph only.
Revised: Thus, the proposed predictor f ω ( s |D , G ) takes a dataset as well as graph as an input to support multiple datasets, while the existing performance predictor takes a graph only.

Original: Then, the proposed predictor meta-learns set-dependent performance proxy generalized over the task distribution p ( τ ) in the meta-training stage.
Revised: Then, the proposed predictor meta-learns set-dependent performance proxy generalized over the task distribution p ( τ ) in the meta-training stage.

Original: This allows the meta-learned predictor to accurately predict performance on unseen datasets without additional training.
Revised: This allows the meta-learned predictor to accurately predict performance on unseen datasets without additional training.

Original: The proposed predictor f ω consists of a dataset encoder and a graph encoder, followed by two linear layers with relu .
Revised: The proposed predictor f ω consists of a dataset encoder and a graph encoder, followed by two linear layers with relu .

Original: For dataset encoding, we use the set encoder of Section 3.1.3 which takes D as an input.
Revised: For dataset encoding, we use the set encoder of Section 3.1.3 which takes D as an input.

Original: We adopt direct acyclic graph encoder (Zhang et al., 2019) for DAG G
Revised: We adopt direct acyclic graph encoder (Zhang et al., 2019) for DAG G

Original: (Please refer to Section B of Suppl.).
Revised: (Please refer to Section B of Suppl.).

Original: We concatenate the outputs of both graph encoder and the set encoder, and feed them to two linear layers with relu to predict accuracy.
Revised: We concatenate the outputs of both graph encoder and the set encoder, and feed them to two linear layers with relu to predict accuracy.

Original: We train the predictor f ω to minimize the MSE loss L τ ω
Revised: We train the predictor f ω to minimize the MSE loss L τ ω

Original: ( s, D , G ) between the predicted accuracy and the true accuracy s of the model on each task sampled from the source database:
Revised: ( s, D , G ) between the predicted accuracy and the true accuracy s of the model on each task sampled from the source database:

Original: [Equation]
Revised: [Equation]

Original: 3.1. S ET E NCODER
Revised: 3.1. S ET E NCODER

Original: The efﬁcacy of the proposed framework is dependent on how accurately set encoder captures the distribution of the target dataset and extracts information related with the goal of the generator and the predictor.
Revised: The efﬁcacy of the proposed framework is dependent on how accurately set encoder captures the distribution of the target dataset and extracts information related with the goal of the generator and the predictor.

Original: To compress the entire instances from a dataset D into a single latent code z , the set encoder should process input sets of any size and summarize consistent information agnostically to the order of the instances (permutation-invariance).
Revised: To compress the entire instances from a dataset D into a single latent code z , the set encoder should process input sets of any size and summarize consistent information agnostically to the order of the instances (permutation-invariance).

Original: Existing set encoders such as DeepSet (Zaheer et al., 2017), SetTransformer (Lee et al., 2019a), and StatisticsPooling (Lee et al., 2020) fulﬁll those requirements and might be used.
Revised: Existing set encoders such as DeepSet (Zaheer et al., 2017), SetTransformer (Lee et al., 2019a), and StatisticsPooling (Lee et al., 2020) fulﬁll those requirements and might be used.

Original: However, DeepSet and SetTransformer are non-hierarchical poolings, thus cannot accurately model individual classes in the given dataset.
Revised: However, DeepSet and SetTransformer are non-hierarchical poolings, thus cannot accurately model individual classes in the given dataset.

Original: Moreover, DeepSet and
Revised: Moreover, DeepSet and

Original: StatisticsPooling resort to simple averaging of the instance-wise representations.
Revised: StatisticsPooling resort to simple averaging of the instance-wise representations.

Original: Therefore, we introduce a novel set encoder which stacks two permutation-invariant modules with attention-based learnable parameters.
Revised: Therefore, we introduce a novel set encoder which stacks two permutation-invariant modules with attention-based learnable parameters.

Original: The lower-level intra-class encoder captures the class prototypes that reﬂect label information, and the high-level inter-class encoder considers the relationship between class prototypes and aggregates them into a latent vector.
Revised: The lower-level intra-class encoder captures the class prototypes that reﬂect label information, and the high-level inter-class encoder considers the relationship between class prototypes and aggregates them into a latent vector.

Original: The proposed structure of the set encoder models high-order interactions between the set elements allowing the generator and predictor to effectively extract useful information to achieve each goal.
Revised: The proposed structure of the set encoder models high-order interactions between the set elements allowing the generator and predictor to effectively extract useful information to achieve each goal.

Original: Speciﬁcally, for a given dataset D =
Revised: Speciﬁcally, for a given dataset D =

Original: { X , Y } , where X =
Revised: { X , Y } , where X =

Original: { X c } C c =1 and Y =
Revised: { X c } C c =1 and Y =

Original: { Y c } C c =1 are the set of instances and target labels of C classes respectively.
Revised: { Y c } C c =1 are the set of instances and target labels of C classes respectively.

Original: We randomly sample instances { x | x ∈ B c } ∈ R b c × d x of class c , where x is a d x dimensional feature vector, B c ⊂ X c and || B c ||
Revised: We randomly sample instances { x | x ∈ B c } ∈ R b c × d x of class c , where x is a d x dimensional feature vector, B c ⊂ X c and || B c ||

Original: =
Revised: =

Original: b c .
Revised: b c .

Original: We input the sampled instances into the IntraSetPool, the intra-class encoder, to encode class prototype v c ∈ R 1 × d v c for each class c = 1 , ..., C .
Revised: We input the sampled instances into the IntraSetPool, the intra-class encoder, to encode class prototype v c ∈ R 1 × d v c for each class c = 1 , ..., C .

Original: Then we further feed the class-speciﬁc set representations { v c } Cc =1 into the InterSetPool, the inter-class encoder, to generate the dataset representation h e ∈ R 1 × d h e as follows:
Revised: Then we further feed the class-speciﬁc set representations { v c } Cc =1 into the InterSetPool, the inter-class encoder, to generate the dataset representation h e ∈ R 1 × d h e as follows:

Original: [Equation]
Revised: [Equation]

Original: Both the set poolings are stacked attention-based blocks borrowed from Lee et al. (2019a).
Revised: Both the set poolings are stacked attention-based blocks borrowed from Lee et al. (2019a).

Original: Note that while Lee et al.
Revised: Note that while Lee et al.

Original: (2019a) is an attention-based set encoder, it ignores class label information of given dataset, which may lead to poor performance.
Revised: (2019a) is an attention-based set encoder, it ignores class label information of given dataset, which may lead to poor performance.

Original: Please see Section C of the Suppl.
Revised: Please see Section C of the Suppl.

Original: for more details.
Revised: for more details.

Original: M ETA -T EST (S EARCHING )
Revised: M ETA -T EST (S EARCHING )

Original: In the meta-test stage, for an unseen dataset ˆ D , we can obtain n set-dependent DAGs { ˆ G i } ni =1 , with the meta-trained generator parameterized by φ ∗ and θ ∗ , by feeding ˆ D as an input.
Revised: In the meta-test stage, for an unseen dataset ˆ D , we can obtain n set-dependent DAGs { ˆ G i } ni =1 , with the meta-trained generator parameterized by φ ∗ and θ ∗ , by feeding ˆ D as an input.

Original: Through such setlevel amortized inference, our method can easily generate neural architecture(s) for the novel dataset.
Revised: Through such setlevel amortized inference, our method can easily generate neural architecture(s) for the novel dataset.

Original: The latent code z ∈ R 1 × d z can be sampled from a dataset-conditioned Gaussian distribution with diagonal covariance where NN µ , NN σ are single linear layers:
Revised: The latent code z ∈ R 1 × d z can be sampled from a dataset-conditioned Gaussian distribution with diagonal covariance where NN µ , NN σ are single linear layers:

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: σ
Revised: σ

Original: [Equation]
Revised: [Equation]

Original: NN µ
Revised: NN µ

Original: [Equation]
Revised: [Equation]

Original: In the meta-test, the predictor f ω ∗ (ˆ s i | ˆ D , ˆ G i ) predicts accuracies { ˆ s i } ni =1 for a given unseen dataset ˆ D and each generated architecture of { ˆ G i } ni =1 and then select the neural architecture having the highest predicted accuracy among { ˆ s i } ni =1 .
Revised: In the meta-test, the predictor f ω ∗ (ˆ s i | ˆ D , ˆ G i ) predicts accuracies { ˆ s i } ni =1 for a given unseen dataset ˆ D and each generated architecture of { ˆ G i } ni =1 and then select the neural architecture having the highest predicted accuracy among { ˆ s i } ni =1 .

Original:  E XPERIMENT
Revised:  E XPERIMENT

Original: We conduct extensive experiments to validate MetaD2A framework.
Revised: We conduct extensive experiments to validate MetaD2A framework.

Original: First, we compare our model with conventional NAS methods on NAS-Bench-201 search space in Section 4.1.
Revised: First, we compare our model with conventional NAS methods on NAS-Bench-201 search space in Section 4.1.

Original: Second, we compare our model with transferable NAS method under a large search space in Section 4.2.
Revised: Second, we compare our model with transferable NAS method under a large search space in Section 4.2.

Original: Third, we compare our model with other Meta-NAS approaches on few-shot classiﬁcation tasks in Section 4.3.
Revised: Third, we compare our model with other Meta-NAS approaches on few-shot classiﬁcation tasks in Section 4.3.

Original: Finally, we analyze the effectiveness of our framework in Section 4.4.
Revised: Finally, we analyze the effectiveness of our framework in Section 4.4.

Original: 4. NAS-B ENCH -201 S EARCH S PACE 4.1. E XPERIMENT S ETUP
Revised: 4. NAS-B ENCH -201 S EARCH S PACE 4.1. E XPERIMENT S ETUP

Original: We learn our model on source database consisting of subsets of ImageNet-1K and neural architectures of NAS-Bench-201 (Dong & Yang, 2020) and (meta-)test our model by searching for architectures on 6 benchmark datasets without additional NAS model training.
Revised: We learn our model on source database consisting of subsets of ImageNet-1K and neural architectures of NAS-Bench-201 (Dong & Yang, 2020) and (meta-)test our model by searching for architectures on 6 benchmark datasets without additional NAS model training.

Original: NAS-Bench-201 search space contains cell-based neural architectures, where each cell is represented as directed ayclic graph (DAG) consisting of the 4 nodes and the 6 edge connections.
Revised: NAS-Bench-201 search space contains cell-based neural architectures, where each cell is represented as directed ayclic graph (DAG) consisting of the 4 nodes and the 6 edge connections.

Original: For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.
Revised: For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.

Original: Source Database To meta-learn our model, we practically collect multiple tasks where each task consists of (dataset, architecture, accuracy).
Revised: Source Database To meta-learn our model, we practically collect multiple tasks where each task consists of (dataset, architecture, accuracy).

Original: We compile ImageNet-1K (Deng et al., 2009) as multiple sub-sets by randomly sampling 20 classes with an average of 26K images for each sub-sets and assign them to each task.
Revised: We compile ImageNet-1K (Deng et al., 2009) as multiple sub-sets by randomly sampling 20 classes with an average of 26K images for each sub-sets and assign them to each task.

Original: All images are downsampled by 32 × 32 size.
Revised: All images are downsampled by 32 × 32 size.

Original: We search for the setspeciﬁc architecture of each sampled dataset using random search among high-quality architectures which are included top-5000 performance architecture group on ImageNet-16-120 or GDAS (Dong & Yang, 2019b).
Revised: We search for the setspeciﬁc architecture of each sampled dataset using random search among high-quality architectures which are included top-5000 performance architecture group on ImageNet-16-120 or GDAS (Dong & Yang, 2019b).

Original: For the predictor, we additionally collect 2,920 tasks through random sampling.
Revised: For the predictor, we additionally collect 2,920 tasks through random sampling.

Original: We obtain its accuracy by training the searched architecture on the dataset of each task.
Revised: We obtain its accuracy by training the searched architecture on the dataset of each task.

Original: We collect N τ = 1,310/4,230 meta-training tasks for the MetaD2A generator/predictor and 400/400 metavalidation tasks for them, respectively.
Revised: We collect N τ = 1,310/4,230 meta-training tasks for the MetaD2A generator/predictor and 400/400 metavalidation tasks for them, respectively.

Original: Meta-training time is 12.7/8.4 GPU hours for the generator/the predictor and note that meta-training phase is needed only once for all experiments.
Revised: Meta-training time is 12.7/8.4 GPU hours for the generator/the predictor and note that meta-training phase is needed only once for all experiments.

Original: Meta-Test Datasets
Revised: Meta-Test Datasets

Original: We apply our model trained from source database to 6 benchmark datasets such as 1)
Revised: We apply our model trained from source database to 6 benchmark datasets such as 1)

Original: CIFAR-10 (Krizhevsky et al., 2009), 2) CIFAR-100 (Krizhevsky et al., 2009), 3) MNIST (LeCun & Cortes, 2010), 4) SVHN (Netzer et al., 2011), 5) Aircraft (Maji et al., 2013), and 6) OxfordIIIT Pets (Parkhi et al., 2012).
Revised: CIFAR-10 (Krizhevsky et al., 2009), 2) CIFAR-100 (Krizhevsky et al., 2009), 3) MNIST (LeCun & Cortes, 2010), 4) SVHN (Netzer et al., 2011), 5) Aircraft (Maji et al., 2013), and 6) OxfordIIIT Pets (Parkhi et al., 2012).

Original: On CIFAR10 and CIFAR100, the generator generates 500 neural architectures and we select 30 architectures based on accuracies predicted by the predictor.
Revised: On CIFAR10 and CIFAR100, the generator generates 500 neural architectures and we select 30 architectures based on accuracies predicted by the predictor.

Original: Following SETN (Dong & Yang, 2019a), we retrieve the accuracies of N architecture candidates from the
Revised: Following SETN (Dong & Yang, 2019a), we retrieve the accuracies of N architecture candidates from the

Original: NAS-bench-201 and report the highest ﬁnal accuracy for each run.
Revised: NAS-bench-201 and report the highest ﬁnal accuracy for each run.

Original: While N = 1000 in SETN, we set a smaller number of samples ( N = 30 ) for MetaD2A. We report the mean accuracies overruns of the search process by retrieving accuracies of searched architectures from NAS-Bench-201.
Revised: While N = 1000 in SETN, we set a smaller number of samples ( N = 30 ) for MetaD2A. We report the mean accuracies overruns of the search process by retrieving accuracies of searched architectures from NAS-Bench-201.

Original: On MNIST, SVHN, Aircraft, and Oxford-IIIT Pets, the generator generates 50 architectures and select the best one with the highest predicted accuracy.
Revised: On MNIST, SVHN, Aircraft, and Oxford-IIIT Pets, the generator generates 50 architectures and select the best one with the highest predicted accuracy.

Original: we report the accuracy averaging over 3 runs with different seeds.
Revised: we report the accuracy averaging over 3 runs with different seeds.

Original: For fair comparison, the searched architectures from our model are trained on each target datasets from the scratch.
Revised: For fair comparison, the searched architectures from our model are trained on each target datasets from the scratch.

Original: Note that once trained MetaD2A can be used for more datasets without additional training.
Revised: Note that once trained MetaD2A can be used for more datasets without additional training.

Original: Our model is performed with a single Nvidia 2080ti GPU.
Revised: Our model is performed with a single Nvidia 2080ti GPU.

Original: 4.1. R ESULTS ON U NSEEN D ATASETS
Revised: 4.1. R ESULTS ON U NSEEN D ATASETS

Original: Table 1 shows that our model meta-learned on the source database can successfully generalize to 6 unseen datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets by outperforming all baselines.
Revised: Table 1 shows that our model meta-learned on the source database can successfully generalize to 6 unseen datasets such as MNIST, SVHN, CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets by outperforming all baselines.

Original: Since the meta-learned MetaD2A can output set-specialized architectures on target datasets through inference process with no training cost, the search speed is extremely fast.
Revised: Since the meta-learned MetaD2A can output set-specialized architectures on target datasets through inference process with no training cost, the search speed is extremely fast.

Original: As shown in Table 1, the search time of 6 benchmark datasets is within 33 GPU seconds on average.
Revised: As shown in Table 1, the search time of 6 benchmark datasets is within 33 GPU seconds on average.

Original: This is impressive results in that it is at least 147 × (maximum: 12169 × ) faster than conventional set-speciﬁc NAS approaches which need training NAS models on each target dataset.
Revised: This is impressive results in that it is at least 147 × (maximum: 12169 × ) faster than conventional set-speciﬁc NAS approaches which need training NAS models on each target dataset.

Original: Such rapid search of REA, RS, REINFORCE and BOHB is only possible where all of the accuracies are pre-computed like NAS-Bench201 so that it can retrieve instantly on the target dataset, therefore, it is difﬁcult to apply them to other non-benchmark datasets.
Revised: Such rapid search of REA, RS, REINFORCE and BOHB is only possible where all of the accuracies are pre-computed like NAS-Bench201 so that it can retrieve instantly on the target dataset, therefore, it is difﬁcult to apply them to other non-benchmark datasets.

Original: Especially, we observe that
Revised: Especially, we observe that

Original: MetaD2A which is learned over multiple tasks beneﬁt to search set-dependent neural architectures for ﬁne-grained datasets such as Aircraft and Oxford-IIIT Pets.
Revised: MetaD2A which is learned over multiple tasks beneﬁt to search set-dependent neural architectures for ﬁne-grained datasets such as Aircraft and Oxford-IIIT Pets.

Original: 4. M OBILE N ET V3 S EARCH S PACE 4.2. E XPERIMENT S ETUP
Revised: 4. M OBILE N ET V3 S EARCH S PACE 4.2. E XPERIMENT S ETUP

Original: We apply our meta-trained model on four unseen datasets, comparing with transferable NAS (NSGANetV2 (Lu et al., 2020)) under the same search space of MobileNetV3, where it contains more than 10 19 architectures.
Revised: We apply our meta-trained model on four unseen datasets, comparing with transferable NAS (NSGANetV2 (Lu et al., 2020)) under the same search space of MobileNetV3, where it contains more than 10 19 architectures.

Original: Each CNN architecture consists of ﬁve sequential blocks and the targets of searching are the number of layers, the number of channels, kernel size, and input resolutions.
Revised: Each CNN architecture consists of ﬁve sequential blocks and the targets of searching are the number of layers, the number of channels, kernel size, and input resolutions.

Original: For a fair comparison, we also exploit the supernet for the parameters as NSGANetV2 does.
Revised: For a fair comparison, we also exploit the supernet for the parameters as NSGANetV2 does.

Original: We collect N τ = 3 , 018 / 153 , 408 meta-training tasks for the generator/predictor and 646 / 32 , 872 metavalidation tasks, respectively as a source database from the ImageNet-1K dataset and architectures of MobileNetV3 search space.
Revised: We collect N τ = 3 , 018 / 153 , 408 meta-training tasks for the generator/predictor and 646 / 32 , 872 metavalidation tasks, respectively as a source database from the ImageNet-1K dataset and architectures of MobileNetV3 search space.

Original: Meta-training time is 2 .
Revised: Meta-training time is 2 .

Original: 41 GPU days for the generator/the predictor.
Revised: 41 GPU days for the generator/the predictor.

Original: Note that the meta-training phase is needed only once on the source database.
Revised: Note that the meta-training phase is needed only once on the source database.

Original: [Table]
Revised: [Table]

Original: 352 × 1.
Revised: 352 × 1.

Original: 26.33 ± 3.40 DrNAS (Chen et al., 2021) 1.
Revised: 26.33 ± 3.40 DrNAS (Chen et al., 2021) 1.

Original: 3452 × 13.
Revised: 3452 × 13.

Original: 46.08 ± 7.00 MetaD2A (Ours) (cid:88) 0.
Revised: 46.08 ± 7.00 MetaD2A (Ours) (cid:88) 0.

Original: 1 × 0.
Revised: 1 × 0.

Original: 58.43 ± 1.
Revised: 58.43 ± 1.

Original: Oxford-IIIT Pets
Revised: Oxford-IIIT Pets

Original: ResNet (He et al., 2016) 0.
Revised: ResNet (He et al., 2016) 0.

Original: N/A N/A
Revised: N/A N/A

Original: N/A 25.58 ± 3.43 RSPS (Li & Talwalkar, 2019) 0.
Revised: N/A 25.58 ± 3.43 RSPS (Li & Talwalkar, 2019) 0.

Original: 420 × 1.
Revised: 420 × 1.

Original: 22.91 ± 1.65 SETN (Dong & Yang, 2019a) 0.
Revised: 22.91 ± 1.65 SETN (Dong & Yang, 2019a) 0.

Original: 1078 × 3.
Revised: 1078 × 3.

Original: 25.17 ± 1.68 GDAS (Dong & Yang, 2019b) 0.
Revised: 25.17 ± 1.68 GDAS (Dong & Yang, 2019b) 0.

Original: 870 × 2.
Revised: 870 × 2.

Original: 24.02 ± 2.75 PC-DARTS (Xu et al., 2020) 0.
Revised: 24.02 ± 2.75 PC-DARTS (Xu et al., 2020) 0.

Original: 355 × 1.
Revised: 355 × 1.

Original: 25.31 ± 1.38 DrNAS (Chen et al., 2021) 0.
Revised: 25.31 ± 1.38 DrNAS (Chen et al., 2021) 0.

Original: 752 × 2.
Revised: 752 × 2.

Original: 26.73 ± 2.61 MetaD2A (Ours) (cid:88) 0.
Revised: 26.73 ± 2.61 MetaD2A (Ours) (cid:88) 0.

Original: 1 × 0.
Revised: 1 × 0.

Original: 41.50 ± 4.
Revised: 41.50 ± 4.

Original: 4.2. R ESULTS ON U NSEEN D ATASETS
Revised: 4.2. R ESULTS ON U NSEEN D ATASETS

Original: We search and evaluate the architecture multiple times with both NSGANetV2 and ours on four unseen datasets such as CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets with different random seeds.
Revised: We search and evaluate the architecture multiple times with both NSGANetV2 and ours on four unseen datasets such as CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets with different random seeds.

Original: Search times of MetaD2A for CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets are within 57, 195, 77, and 170 GPU seconds on average with a single Nvidia RTX 2080ti GPU respectively, while NSGANetV2 needs 1 GPU day with 8 1080ti GPUs on each dataset, which is 5,523 times slower than MetaD2A. Besides the huge speed up, Figure 3 shows that our model can search for a comparable architecture to the NSGANetV2 over ﬂops without a performance drop.
Revised: Search times of MetaD2A for CIFAR-10, CIFAR-100, Aircraft, and Oxford-IIIT Pets are within 57, 195, 77, and 170 GPU seconds on average with a single Nvidia RTX 2080ti GPU respectively, while NSGANetV2 needs 1 GPU day with 8 1080ti GPUs on each dataset, which is 5,523 times slower than MetaD2A. Besides the huge speed up, Figure 3 shows that our model can search for a comparable architecture to the NSGANetV2 over ﬂops without a performance drop.

Original: Interestingly, even we use naive ﬂop ﬁltering and NSGANetV2 uses an objective function for ﬂop constraints, MetaD2A performs consistently comparably to NSGANetV2 over the different ﬂops.
Revised: Interestingly, even we use naive ﬂop ﬁltering and NSGANetV2 uses an objective function for ﬂop constraints, MetaD2A performs consistently comparably to NSGANetV2 over the different ﬂops.

Original: Overall, the results demonstrate that our model also can generalize to unseen datasets not only under the NAS-Bench-201 space but also under a larger MobileNetV3 space with its meta-knowledge.
Revised: Overall, the results demonstrate that our model also can generalize to unseen datasets not only under the NAS-Bench-201 space but also under a larger MobileNetV3 space with its meta-knowledge.

Original: 4. C OMPARISON WITH M ETA -NAS A PPROACHES
Revised: 4. C OMPARISON WITH M ETA -NAS A PPROACHES

Original: [Table]
Revised: [Table]

Original: We further compare our method against
Revised: We further compare our method against

Original: Meta-NAS methods (Kim et al., 2018;
Revised: Meta-NAS methods (Kim et al., 2018;

Original: Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) on few-shot classiﬁcation tasks, which are the main setting existing MetaNAS methods have been consider.
Revised: Elsken et al., 2020; Lian et al., 2019; Shaw et al., 2019) on few-shot classiﬁcation tasks, which are the main setting existing MetaNAS methods have been consider.

Original: Following (Elsken et al., 2020; Lian et al., 2019), we adopt bi-level optimization (e.g., MAML framework) to meta-learn initial weights of neural architectures searched by our model on a meta-training set of mini-imagenet.
Revised: Following (Elsken et al., 2020; Lian et al., 2019), we adopt bi-level optimization (e.g., MAML framework) to meta-learn initial weights of neural architectures searched by our model on a meta-training set of mini-imagenet.

Original: As shown in the Table 2, the few-shot classiﬁcation results on MiniImageNet further clearly show the MetaD2A’s effectiveness over existing Meta-NAS methods, as well as the conventional meta-learning methods without NAS (Finn et al., 2017; Antoniou et al., 2018).
Revised: As shown in the Table 2, the few-shot classiﬁcation results on MiniImageNet further clearly show the MetaD2A’s effectiveness over existing Meta-NAS methods, as well as the conventional meta-learning methods without NAS (Finn et al., 2017; Antoniou et al., 2018).

Original: 4. E FFECTIVENESS OF M ETA D2A
Revised: 4. E FFECTIVENESS OF M ETA D2A

Original: Now, we verify the efﬁcacy of each component of MetaD2A with further analysis.
Revised: Now, we verify the efﬁcacy of each component of MetaD2A with further analysis.

Original: [Table]
Revised: [Table]

Original: Ablation Study on MetaD2A
Revised: Ablation Study on MetaD2A

Original: We train different variations of our model on the subsets of ImageNet-1K, and test on CIFAR10, CIFAR100, and Aircraft in Table 3 with the same experimental setup as the main experiments in Table 1.
Revised: We train different variations of our model on the subsets of ImageNet-1K, and test on CIFAR10, CIFAR100, and Aircraft in Table 3 with the same experimental setup as the main experiments in Table 1.

Original: The
Revised: The

Original: MetaD2A generator without the performance predictor (Generator only) outperforms the simple random architecture sampler (Random Sampling), especially by 15 .
Revised: MetaD2A generator without the performance predictor (Generator only) outperforms the simple random architecture sampler (Random Sampling), especially by 15 .

Original: 3% on Aircraft, which demonstrates the effectiveness of MetaD2A over the random sampler.
Revised: 3% on Aircraft, which demonstrates the effectiveness of MetaD2A over the random sampler.

Original: Also, we observe that combining the meta-performance predictor to the random architecture sampler (Predictor only) enhances the accuracy of the ﬁnal architecture on all datasets.
Revised: Also, we observe that combining the meta-performance predictor to the random architecture sampler (Predictor only) enhances the accuracy of the ﬁnal architecture on all datasets.

Original: Finally, MetaD2A combined with the performance predictor (MetaD2A) outperforms all baselines, especially by 20.28% on Aircraft, suggesting that our MetaD2A can output architectures that are more relevant to the given task
Revised: Finally, MetaD2A combined with the performance predictor (MetaD2A) outperforms all baselines, especially by 20.28% on Aircraft, suggesting that our MetaD2A can output architectures that are more relevant to the given task

Original: [Figure]
Revised: [Figure]

Original: Effectiveness of Set-to-Architecture Generator
Revised: Effectiveness of Set-to-Architecture Generator

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: We ﬁrst visualize cross-modal latent embeddings { z } of unseen datasets encoded by the meta- learned generator with T-SNE in Figure 4.
Revised: We ﬁrst visualize cross-modal latent embeddings { z } of unseen datasets encoded by the meta- learned generator with T-SNE in Figure 4.

Original: Each marker indicates { z } of the sampled subsets of each dataset with different seeds.
Revised: Each marker indicates { z } of the sampled subsets of each dataset with different seeds.

Original: We observe that the generator classiﬁes well embeddings { z } by datasets in the latent space while clusters z of the subset of the same dataset.
Revised: We observe that the generator classiﬁes well embeddings { z } by datasets in the latent space while clusters z of the subset of the same dataset.

Original: Furthermore, we investigate the quality of generated architectures from those embeddings { z } .
Revised: Furthermore, we investigate the quality of generated architectures from those embeddings { z } .

Original: In the Figure 5, the generator sample 2000 architecture candidates from the embeddings encoded each target dataset and computes the validate accuracy of those architectures.
Revised: In the Figure 5, the generator sample 2000 architecture candidates from the embeddings encoded each target dataset and computes the validate accuracy of those architectures.

Original: The proposed generator generates more high-performing architectures than the simple random architecture sampler for each target dataset.
Revised: The proposed generator generates more high-performing architectures than the simple random architecture sampler for each target dataset.

Original: These results are consistent with Table 3, where the generator (Generator only) enhances the performance compared with the simple random architecture sampler (Random Sampling) consistently on CIFAR10 and CIFAR100.
Revised: These results are consistent with Table 3, where the generator (Generator only) enhances the performance compared with the simple random architecture sampler (Random Sampling) consistently on CIFAR10 and CIFAR100.

Original: The meta-learned generator allows us to effective and efﬁcient search by excluding the poor-performing architectures of broad search space.
Revised: The meta-learned generator allows us to effective and efﬁcient search by excluding the poor-performing architectures of broad search space.

Original: We believe the generator replaces the random sampling stage of other NAS methods.
Revised: We believe the generator replaces the random sampling stage of other NAS methods.

Original: We leave to valid it as the future work.
Revised: We leave to valid it as the future work.

Original: [Table]
Revised: [Table]

Original: Could the Generator Create Novel Architectures?
Revised: Could the Generator Create Novel Architectures?

Original: Since the generator maps set-architecture pairs in the continuous latent space, it can generate novel architectures in the meta-test, which are not contained in the source database.
Revised: Since the generator maps set-architecture pairs in the continuous latent space, it can generate novel architectures in the meta-test, which are not contained in the source database.

Original: To validate it, we evaluate generated 10,000 neural architecture samples of both search space with the measures Validity , Uniqueness , and Novelty following (Zhang et al., 2019) in Table 4.
Revised: To validate it, we evaluate generated 10,000 neural architecture samples of both search space with the measures Validity , Uniqueness , and Novelty following (Zhang et al., 2019) in Table 4.

Original: Each is deﬁned as how often the model can generate valid neural architectures from the prior distribution, the proportion of unique graphs out of the valid generations, and the proportion of valid generations that are not included in the training set, respectively.
Revised: Each is deﬁned as how often the model can generate valid neural architectures from the prior distribution, the proportion of unique graphs out of the valid generations, and the proportion of valid generations that are not included in the training set, respectively.

Original: For NAS-Benchsearch space and MobileNetV3 search space, respectively, the results show the meta-learned generator can generate 67.31%/100% new graphs that do not belong to the training set and can generate 35.19%/100% various graphs, not picking always-the-same architecture seen of the source database.
Revised: For NAS-Benchsearch space and MobileNetV3 search space, respectively, the results show the meta-learned generator can generate 67.31%/100% new graphs that do not belong to the training set and can generate 35.19%/100% various graphs, not picking always-the-same architecture seen of the source database.

Original: [Table]
Revised: [Table]

Original: Effectiveness of Meta-Predictor
Revised: Effectiveness of Meta-Predictor

Original: We ﬁrst demonstrate the necessity of set encoding to handle multiple datasets with a single predictor.
Revised: We ﬁrst demonstrate the necessity of set encoding to handle multiple datasets with a single predictor.

Original: In Table 5, we metatrain all models on the source database of NAS-Bench-201 search space and measure Pearson correlation coefﬁcient on the validation tasks (400 unseen tasks) of the source database.
Revised: In Table 5, we metatrain all models on the source database of NAS-Bench-201 search space and measure Pearson correlation coefﬁcient on the validation tasks (400 unseen tasks) of the source database.

Original: Pearson correlation coefﬁcient is the linear correlation between the actual performance and the predicted performance (higher the better).
Revised: Pearson correlation coefﬁcient is the linear correlation between the actual performance and the predicted performance (higher the better).

Original: Using both the dataset and the computational graph of the target architecture as inputs, instead of using graphs only (Graph Encoder Only), clearly leads to better performance to support multiple datasets.
Revised: Using both the dataset and the computational graph of the target architecture as inputs, instead of using graphs only (Graph Encoder Only), clearly leads to better performance to support multiple datasets.

Original: Moreover, the predictor with the proposed set encoder clearly shows a higher correlation than other set encoders (DeepSet (Zaheer et al., 2017), SetTransformer (Lee et al., 2019a), and Statistical Pooling (Lee et al., 2020)).
Revised: Moreover, the predictor with the proposed set encoder clearly shows a higher correlation than other set encoders (DeepSet (Zaheer et al., 2017), SetTransformer (Lee et al., 2019a), and Statistical Pooling (Lee et al., 2020)).

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We proposed a novel NAS framework, MetaD2A (Meta Dataset-to-Architecture), that can output a neural architecture for an unseen dataset.
Revised: We proposed a novel NAS framework, MetaD2A (Meta Dataset-to-Architecture), that can output a neural architecture for an unseen dataset.

Original: The MetaD2A generator learns a dataset-to-architecture transformation over a database of datasets and neural architectures by encoding each dataset using a set encoder and generating each neural architecture with a graph decoder.
Revised: The MetaD2A generator learns a dataset-to-architecture transformation over a database of datasets and neural architectures by encoding each dataset using a set encoder and generating each neural architecture with a graph decoder.

Original: While the model can generate a novel architecture given a new dataset in an amortized inference, we further learn a meta-performance predictor to select the best architecture for the dataset among multiple sampled architectures.
Revised: While the model can generate a novel architecture given a new dataset in an amortized inference, we further learn a meta-performance predictor to select the best architecture for the dataset among multiple sampled architectures.

Original: The experimental results show that our method shows competitive performance with conventional NAS methods on various datasets with very small search time as it generalizes well across datasets.
Revised: The experimental results show that our method shows competitive performance with conventional NAS methods on various datasets with very small search time as it generalizes well across datasets.

Original: We believe that our work is a meaningful step towards building a practical NAS system for real-world scenarios, where we need to handle diverse datasets while minimizing the search cost.
Revised: We believe that our work is a meaningful step towards building a practical NAS system for real-world scenarios, where we need to handle diverse datasets while minimizing the search cost.

Original: Acknowledgements This work was supported by Samsung Research Funding Center of Samsung
Revised: Acknowledgements This work was supported by Samsung Research Funding Center of Samsung

Original: Electronics (No.
Revised: Electronics (No.

Original: SRFC-IT1502-51) and Institute of Information & communications Technology
Revised: SRFC-IT1502-51) and Institute of Information & communications Technology

Original: Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)
Revised: Planning & Evaluation (IITP) grant funded by the Korea government(MSIT)

Original: (No.2019-0-00075,
Revised: (No.2019-0-00075,

Original: Artiﬁcial Intelligence Graduate School Program(KAIST)).
Revised: Artiﬁcial Intelligence Graduate School Program(KAIST)).

Original: A D ETAILS OF T HE G ENERATOR A. G RAPH D ECODING
Revised: A D ETAILS OF T HE G ENERATOR A. G RAPH D ECODING

Original: To generate the i th node v i , we compute the operation type o v i ∈ R 1 × n o over n o operations based on the current graph state h G :
Revised: To generate the i th node v i , we compute the operation type o v i ∈ R 1 × n o over n o operations based on the current graph state h G :

Original: =
Revised: =

Original: h v i − 

Revised: h v i −

Original: and then predict whether the edge exists between the node v i and other existing nodes.
Revised: and then predict whether the edge exists between the node v i and other existing nodes.

Original: Following (Zhang et al., 2019), when we compute the edge probability e { v j ,v i } , we consider nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) in the reverse order to reﬂect information from nodes close to v i to the root node when deciding whether edge connection.
Revised: Following (Zhang et al., 2019), when we compute the edge probability e { v j ,v i } , we consider nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) in the reverse order to reﬂect information from nodes close to v i to the root node when deciding whether edge connection.

Original: Note that the proposed process guarantees the generation of directed acyclic graph since directed edge is always created from existing nodes to a new node.
Revised: Note that the proposed process guarantees the generation of directed acyclic graph since directed edge is always created from existing nodes to a new node.

Original: The graph decoder starts from an initial hidden state h v 0 =
Revised: The graph decoder starts from an initial hidden state h v 0 =

Original: NN init ( z ) , where NN init is an MLP followed by tanh .
Revised: NN init ( z ) , where NN init is an MLP followed by tanh .

Original: For i th node v i according to topological order , we compute the probability of each operation type o v i ∈ R 1 × n o over n o operations, given the current graph state as the last hidden node h G :=
Revised: For i th node v i according to topological order , we compute the probability of each operation type o v i ∈ R 1 × n o over n o operations, given the current graph state as the last hidden node h G :=

Original: h v i .
Revised: h v i .

Original: That is, o v i =
Revised: That is, o v i =

Original: NN node ( h G ) , where NN node is an MLP followed by softmax .
Revised: NN node ( h G ) , where NN node is an MLP followed by softmax .

Original: When the predicted v i type is the end-of-graph, we stop the decoding process and connect all leaf nodes to v i .
Revised: When the predicted v i type is the end-of-graph, we stop the decoding process and connect all leaf nodes to v i .

Original: Otherwise we update hidden state h ( t ) v i at time step t as follows:
Revised: Otherwise we update hidden state h ( t ) v i at time step t as follows:

Original: [Equation]
Revised: [Equation]

Original: The function UPDATE is a gated recurrent unit (GRU) (Cho et al., 2014), i is the order of v i , and m ( t ) v
Revised: The function UPDATE is a gated recurrent unit (GRU) (Cho et al., 2014), i is the order of v i , and m ( t ) v

Original: i is the incoming message to v i .
Revised: i is the incoming message to v i .

Original: The function AGGREGATE consists of mapping and gating functions with MLPs, where V inv i is a set of predecessors with incoming edges to v i .
Revised: The function AGGREGATE consists of mapping and gating functions with MLPs, where V inv i is a set of predecessors with incoming edges to v i .

Original: For all previously processed nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) , we decide whether to link an edge from v j to v i by sampling the edge based on edge connection probability e { v j ,v i } =
Revised: For all previously processed nodes (cid:8) v j | j = i − 1 , ..., 1 (cid:9) , we decide whether to link an edge from v j to v i by sampling the edge based on edge connection probability e { v j ,v i } =

Original: NN edge ( h j , h i ) , where NN edge is a MLP followed by sigmoid .
Revised: NN edge ( h j , h i ) , where NN edge is a MLP followed by sigmoid .

Original: We update h v i by Eq.
Revised: We update h v i by Eq.

Original: (6) whenever a new edge is connected to v i .
Revised: (6) whenever a new edge is connected to v i .

Original: For meta-test, we select the operation with the max probability for each node and edges with e { v j ,v i } > 0 .
Revised: For meta-test, we select the operation with the max probability for each node and edges with e { v j ,v i } > 0 .

Original: A. M ETA -T RAINING O BJECTIVE
Revised: A. M ETA -T RAINING O BJECTIVE

Original: We meta-learn the model using Eq.
Revised: We meta-learn the model using Eq.

Original: The expectation of the log-likelihood E z ∼ q φ ( z |D ) (cid:2) log p θ (cid:0)
Revised: The expectation of the log-likelihood E z ∼ q φ ( z |D ) (cid:2) log p θ (cid:0)

Original: G| z (cid:1)(cid:3) of (2) can be rewritten with negative cross-entropy loss − L τCE for nodes and binary cross-entropy loss − L τBCE for edges, and we slightly modify it using the generated set-dependent graph ˜ G and the ground truth graph G as the input as follows:
Revised: G| z (cid:1)(cid:3) of (2) can be rewritten with negative cross-entropy loss − L τCE for nodes and binary cross-entropy loss − L τBCE for edges, and we slightly modify it using the generated set-dependent graph ˜ G and the ground truth graph G as the input as follows:

Original: [Equation]
Revised: [Equation]

Original: We substitute the log-likelihood term of Eq.
Revised: We substitute the log-likelihood term of Eq.

Original: (2) such as Eq.
Revised: (2) such as Eq.

Original: (7) and learn the proposed generator by maximizing the objective (1) to learn φ , θ , which are shared across all tasks.
Revised: (7) and learn the proposed generator by maximizing the objective (1) to learn φ , θ , which are shared across all tasks.

Original: B G RAPH E NCODING OF THE S ET - DEPENDENT P REDICTOR
Revised: B G RAPH E NCODING OF THE S ET - DEPENDENT P REDICTOR

Original: For a given graph candidate G , we sequentially perform message passing for nodes from the predecessors following the topological order of the DAG G .
Revised: For a given graph candidate G , we sequentially perform message passing for nodes from the predecessors following the topological order of the DAG G .

Original: We iteratively update hidden states h ( t ) v i using the Eq.
Revised: We iteratively update hidden states h ( t ) v i using the Eq.

Original: (8) by feeding in its predecessors’ hidden states { u ∈ V inv i } .
Revised: (8) by feeding in its predecessors’ hidden states { u ∈ V inv i } .

Original: [Equation]
Revised: [Equation]

Original: where m ( t ) v i = (cid:88) u ∈V invi
Revised: where m ( t ) v i = (cid:88) u ∈V invi

Original: AGGREGATE
Revised: AGGREGATE

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For starting node v 0 which the set of predecessors is the empty, we output the zero vector as the hidden state of v 0 .
Revised: For starting node v 0 which the set of predecessors is the empty, we output the zero vector as the hidden state of v 0 .

Original: We use the last hidden states of the ending node as the output of the graph encoder h f .
Revised: We use the last hidden states of the ending node as the output of the graph encoder h f .

Original: Additionally, we exploit Bi-directional encoding (Zhang et al., 2019) which reverses the node orders to perform the encoding process.
Revised: Additionally, we exploit Bi-directional encoding (Zhang et al., 2019) which reverses the node orders to perform the encoding process.

Original: In this case, the ﬁnal node becomes the starting point.
Revised: In this case, the ﬁnal node becomes the starting point.

Original: Thus, the backward graph encoder outputs h b , which is the last hidden states of the starting node.
Revised: Thus, the backward graph encoder outputs h b , which is the last hidden states of the starting node.

Original: We concatenate the outputs
Revised: We concatenate the outputs

Original: h f of the forward graph encoder and h b of the backward graph encoder as the ﬁnal output of the Bi-directional graph encoding.
Revised: h f of the forward graph encoder and h b of the backward graph encoder as the ﬁnal output of the Bi-directional graph encoding.

Original: C B UILDING B LOCKS OF THE S ET E NCODER
Revised: C B UILDING B LOCKS OF THE S ET E NCODER

Original: We use Set Attention Block (SAB) and Pooling by Multi-head Attention (PMA)
Revised: We use Set Attention Block (SAB) and Pooling by Multi-head Attention (PMA)

Original: (Lee et al., 2019a), where the former learns the features for each element in the set using self-attention while the latter pools the input features into k representative vectors.
Revised: (Lee et al., 2019a), where the former learns the features for each element in the set using self-attention while the latter pools the input features into k representative vectors.

Original: Set Attention Block (SAB) is an attentionbased block, which makes the features of all of the instances in the set reﬂect the relations between itself and others such as:
Revised: Set Attention Block (SAB) is an attentionbased block, which makes the features of all of the instances in the set reﬂect the relations between itself and others such as:

Original: [Equation]
Revised: [Equation]

Original: where H = LN ( X + MH ( X , X , X ))
Revised: where H = LN ( X + MH ( X , X , X ))

Original: [Equation]
Revised: [Equation]

Original: 9) where LN and MLP is the layer normalization (Ba et al., 2016) and the multilayer perceptron respectively, and H ∈ R n B c × d H is computed with multi-head attention MH ( Q , K , V ) (Vaswani et al., 2017) which queries, keys, and values are elements of input set X .
Revised: 9) where LN and MLP is the layer normalization (Ba et al., 2016) and the multilayer perceptron respectively, and H ∈ R n B c × d H is computed with multi-head attention MH ( Q , K , V ) (Vaswani et al., 2017) which queries, keys, and values are elements of input set X .

Original: Features encoded from the SAB layers can be pooled by PMA on learnable seed vectors S ∈ R k × d S to produce k vectors by slightly modifying H calculation of Eq.
Revised: Features encoded from the SAB layers can be pooled by PMA on learnable seed vectors S ∈ R k × d S to produce k vectors by slightly modifying H calculation of Eq.

Original: [Equation]
Revised: [Equation]

Original: where H = LN ( X + MH
Revised: where H = LN ( X + MH

Original: [Equation]
Revised: [Equation]

Original: S ,
Revised: S ,

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: X
Revised: X

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: While k can be any size (i.e. k =1,2,10,16), we set k = 1 for generating the single latent vector.
Revised: While k can be any size (i.e. k =1,2,10,16), we set k = 1 for generating the single latent vector.

Original: For extracting consistent information not depending the order and the size of input elements, encoding functions should be constructed by stacking permutation-equivariant layers E, which satisﬁes below condition for any permutation π on a set X (Zaheer et al., 2017):
Revised: For extracting consistent information not depending the order and the size of input elements, encoding functions should be constructed by stacking permutation-equivariant layers E, which satisﬁes below condition for any permutation π on a set X (Zaheer et al., 2017):

Original: [Equation]
Revised: [Equation]

Original: Since all of the components in SAB and PMA are row-wise computation functions, SAB and PMA is permutation equivarint by deﬁnition Eq.
Revised: Since all of the components in SAB and PMA are row-wise computation functions, SAB and PMA is permutation equivarint by deﬁnition Eq.

Original: D S EARCH S PACE
Revised: D S EARCH S PACE

Original: Following the NAS-Bench-201 (Dong & Yang, 2020), We explore the search space consisting of 15,625 possible cell-based neural architectures for all experiments.
Revised: Following the NAS-Bench-201 (Dong & Yang, 2020), We explore the search space consisting of 15,625 possible cell-based neural architectures for all experiments.

Original: Macro skeleton is stacked with one stem cell, three stages consisting of 5 cells for each, and a residual block (He et al., 2016) between stages.
Revised: Macro skeleton is stacked with one stem cell, three stages consisting of 5 cells for each, and a residual block (He et al., 2016) between stages.

Original: The stem cell consists of 3-by-3 convolution with 16 channels and cells of the ﬁrst, second and third stages have 16, 32 and 64, respectively.
Revised: The stem cell consists of 3-by-3 convolution with 16 channels and cells of the ﬁrst, second and third stages have 16, 32 and 64, respectively.

Original: Residual blocks have convolution layer with the stride 2 for down-sampling.
Revised: Residual blocks have convolution layer with the stride 2 for down-sampling.

Original: A fully connected layer is attached to the macro skeleton for classiﬁcation.
Revised: A fully connected layer is attached to the macro skeleton for classiﬁcation.

Original: Each cell is DAG which consists of the ﬁxed 4 nodes and the ﬁxed 6 edge connections.
Revised: Each cell is DAG which consists of the ﬁxed 4 nodes and the ﬁxed 6 edge connections.

Original: For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.
Revised: For each edge connection, NAS models select one of 5 operation candidates such as zerorize, skip connection, 1-by-1 convolution, 3-by-3 convolution, and 3-by-3 average pooling.

Original: To effectively encode the operation information as the node features, we represent edges of graphs in NAS-Bench201 as nodes, and nodes of them as edges.
Revised: To effectively encode the operation information as the node features, we represent edges of graphs in NAS-Bench201 as nodes, and nodes of them as edges.

Original: Additionally, we add a starting node and an ending node to the cell during training.
Revised: Additionally, we add a starting node and an ending node to the cell during training.

Original: All nodes which have no predecessors (suceessors) are connected to the starting (ending) node, which we delete after generating the full neural architectures.
Revised: All nodes which have no predecessors (suceessors) are connected to the starting (ending) node, which we delete after generating the full neural architectures.

Original: E E XPERIMENTAL S ETUP E. D ATASET
Revised: E E XPERIMENTAL S ETUP E. D ATASET

Original: 1) CIFAR-10 (Krizhevsky et al., 2009):
Revised: 1) CIFAR-10 (Krizhevsky et al., 2009):

Original: This dataset is a popular benchmark dataset for NAS, which consists of 32 × 32 colour images from 10 general object classes
Revised: This dataset is a popular benchmark dataset for NAS, which consists of 32 × 32 colour images from 10 general object classes

Original: The training set consists of 50K images, 5K for each class, and the test set consists of 10K images, 1K for each class.
Revised: The training set consists of 50K images, 5K for each class, and the test set consists of 10K images, 1K for each class.

Original: 2) CIFAR100 (Krizhevsky et al., 2009):
Revised: 2) CIFAR100 (Krizhevsky et al., 2009):

Original: This dataset consists of colored images from 100 ﬁne-grained general object classes.
Revised: This dataset consists of colored images from 100 ﬁne-grained general object classes.

Original: Each class has 500/100 images for training and test, respectively.
Revised: Each class has 500/100 images for training and test, respectively.

Original: 3) MNIST (LeCun & Cortes, 2010):
Revised: 3) MNIST (LeCun & Cortes, 2010):

Original: This is a standard image classiﬁcation dataset which contains 70K 28 × 28 grey colored images that describe 10 digits.
Revised: This is a standard image classiﬁcation dataset which contains 70K 28 × 28 grey colored images that describe 10 digits.

Original: We upsample the images to 32 × 32 pixels to satisfy the minimum required pixel size of the NAS-Bench 201 due to the residual blocks in the macro skeleton.
Revised: We upsample the images to 32 × 32 pixels to satisfy the minimum required pixel size of the NAS-Bench 201 due to the residual blocks in the macro skeleton.

Original: We use the training/test split from the original dataset, where 60K images are used for training and 10K are used for test.
Revised: We use the training/test split from the original dataset, where 60K images are used for training and 10K are used for test.

Original: 4) SVHN (Netzer et al., 2011):
Revised: 4) SVHN (Netzer et al., 2011):

Original: This dataset consists of 32 × 32 color images where each has a digit with a natural scene background.
Revised: This dataset consists of 32 × 32 color images where each has a digit with a natural scene background.

Original: The number of classes is 10 denoting from digit 1 to 10 and the number of training/test images is 73257/26032, respectively.
Revised: The number of classes is 10 denoting from digit 1 to 10 and the number of training/test images is 73257/26032, respectively.

Original: 5) Aircraft (Maji et al., 2013)
Revised: 5) Aircraft (Maji et al., 2013)

Original: This is ﬁne-grained classiﬁcation benchmark dataset containing 10K images fromdifferent aircraft classes.
Revised: This is ﬁne-grained classiﬁcation benchmark dataset containing 10K images fromdifferent aircraft classes.

Original: We resize all images into 32 × 32. 6) Oxford-IIIT Pets (Parkhi et al., 2012)
Revised: We resize all images into 32 × 32. 6) Oxford-IIIT Pets (Parkhi et al., 2012)

Original: This dataset is for ﬁne-grained classiﬁcation which has 37 breeds of pets with roughly 200 instances for each class.
Revised: This dataset is for ﬁne-grained classiﬁcation which has 37 breeds of pets with roughly 200 instances for each class.

Original: There is no split ﬁle provided, so we use the 85% of the dataset for training and the other 15% are as a test set.
Revised: There is no split ﬁle provided, so we use the 85% of the dataset for training and the other 15% are as a test set.

Original: We also resize all images into 32 × 32.
Revised: We also resize all images into 32 × 32.

Original: For CIFAR10 and CIFAR100 , we used the training, validation, and test splits from the NAS-Bench-201, and use random validation/test splits for MNIST , SVHN , Aircraft , and Oxford-IIIT Pets by splitting the test set into two subsets of the same size.
Revised: For CIFAR10 and CIFAR100 , we used the training, validation, and test splits from the NAS-Bench-201, and use random validation/test splits for MNIST , SVHN , Aircraft , and Oxford-IIIT Pets by splitting the test set into two subsets of the same size.

Original: The validation set is used to update the searching algorithms as a supervision signal and the test set is used to evaluate the performance of the searched architectures.
Revised: The validation set is used to update the searching algorithms as a supervision signal and the test set is used to evaluate the performance of the searched architectures.

Original: E. B ASELINES
Revised: E. B ASELINES

Original: We now brieﬂy describe the baseline models and our MetaD2A model.
Revised: We now brieﬂy describe the baseline models and our MetaD2A model.

Original: 1) ResNet (He et al., 2016)
Revised: 1) ResNet (He et al., 2016)

Original: This is a convolutional network which connects the output of previous layer as input to the current layer.
Revised: This is a convolutional network which connects the output of previous layer as input to the current layer.

Original: It has achieved impressive performance on many challenging image tasks.
Revised: It has achieved impressive performance on many challenging image tasks.

Original: We use ResNet in all experiments.
Revised: We use ResNet in all experiments.

Original: 2) REA (Real et al., 2019)
Revised: 2) REA (Real et al., 2019)

Original: This is an evolutional-based search method by using aging based tournament selection, showing evolution can work in NAS. 3) RS (Bergstra &
Revised: This is an evolutional-based search method by using aging based tournament selection, showing evolution can work in NAS. 3) RS (Bergstra &

Original: Bengio, 2012)
Revised: Bengio, 2012)

Original: This is based on random search and we randomly samples architectures until the total time of training and evaluation reaches the budget.
Revised: This is based on random search and we randomly samples architectures until the total time of training and evaluation reaches the budget.

Original: 4) REINFORCE (Williams, 1992)
Revised: 4) REINFORCE (Williams, 1992)

Original: This is a RL-based NAS.
Revised: This is a RL-based NAS.

Original: We reward the model with the validation accuracy after 12 epochs of training.
Revised: We reward the model with the validation accuracy after 12 epochs of training.

Original: 5) BOHB (Falkner et al., 2018)
Revised: 5) BOHB (Falkner et al., 2018)

Original: This combines the strengths of tree-structured parzen estimator based baysian optimization and hyperband, performing better than standard baysian optimization methods.
Revised: This combines the strengths of tree-structured parzen estimator based baysian optimization and hyperband, performing better than standard baysian optimization methods.

Original: 5) RSPS (Li & Talwalkar, 2019)
Revised: 5) RSPS (Li & Talwalkar, 2019)

Original: This method is a combination of random search and weight sharing, which trains randomly sampled sub-graphs from weight shared DAG of the search space.
Revised: This method is a combination of random search and weight sharing, which trains randomly sampled sub-graphs from weight shared DAG of the search space.

Original: The method then selects the best performing sub-graph among the sampled ones as the ﬁnal neural architecture.
Revised: The method then selects the best performing sub-graph among the sampled ones as the ﬁnal neural architecture.

Original: 6) SETN (Dong & Yang, 2019a)
Revised: 6) SETN (Dong & Yang, 2019a)

Original: SETN is an one-shot NAS method, which selectively samples competitive child candidates by learning to evaluate the quality of the candidates based on the validation loss.
Revised: SETN is an one-shot NAS method, which selectively samples competitive child candidates by learning to evaluate the quality of the candidates based on the validation loss.

Original: 7) GDAS (Dong & Yang, 2019b)
Revised: 7) GDAS (Dong & Yang, 2019b)

Original: This is a Gumbel-Softmax based differentiable neural architecture sampler, which is trained to minimize the validation loss with the architecture sampled from DAGs.
Revised: This is a Gumbel-Softmax based differentiable neural architecture sampler, which is trained to minimize the validation loss with the architecture sampled from DAGs.

Original: 8) PC-DARTS
Revised: 8) PC-DARTS

Original: (Xu et al., 2020)
Revised: (Xu et al., 2020)

Original: This is a gradientbased NAS which partially samples channels to apply operations, to improve the efﬁciency of
Revised: This is a gradientbased NAS which partially samples channels to apply operations, to improve the efﬁciency of

Original: NAS in terms of memory usage and search time compared to DARTS.
Revised: NAS in terms of memory usage and search time compared to DARTS.

Original: We exploit the code at https://github.com/yuhuixu1993/PC-DARTS . 9)
Revised: We exploit the code at https://github.com/yuhuixu1993/PC-DARTS . 9)

Original: DrNAS (Chen et al., 2021)
Revised: DrNAS (Chen et al., 2021)

Original: This is a NAS approach that introduces Dirichlet distribution to approximate the architecture distribution, to enhance the generalization performance of differentiable architecture search.
Revised: This is a NAS approach that introduces Dirichlet distribution to approximate the architecture distribution, to enhance the generalization performance of differentiable architecture search.

Original: We use the code at https://github.com/xiangning-chen/DrNAS .
Revised: We use the code at https://github.com/xiangning-chen/DrNAS .

Original: We report the results on CIFAR10 and
Revised: We report the results on CIFAR10 and

Original: CIFAR100 in this paper using the provided code from the authors on the split set of NAS-Bench 201 while their reported results in the paper of the authors are 94 . 37 and 73 . 51 , respectively on random training/test splits on CIFAR10 and CIFAR100 . 10)
Revised: CIFAR100 in this paper using the provided code from the authors on the split set of NAS-Bench 201 while their reported results in the paper of the authors are 94 . 37 and 73 . 51 , respectively on random training/test splits on CIFAR10 and CIFAR100 . 10)

Original: MetaD2A (Ours)
Revised: MetaD2A (Ours)

Original: This is our meta-NAS framework described in section 3, which can stochastically generate task-dependent computational graphs from a given dataset, and use the performance predictor to select the best performing candidates.
Revised: This is our meta-NAS framework described in section 3, which can stochastically generate task-dependent computational graphs from a given dataset, and use the performance predictor to select the best performing candidates.

Original: We follow the same settings of NAS-Bench-201 (Dong & Yang, 2020) for all baselines and use the code at https://github.com/D-X-Y/AutoDL-Projects except for 8), 9) and 10).
Revised: We follow the same settings of NAS-Bench-201 (Dong & Yang, 2020) for all baselines and use the code at https://github.com/D-X-Y/AutoDL-Projects except for 8), 9) and 10).

Original: E. I MPLEMENTATION D ETAILS
Revised: E. I MPLEMENTATION D ETAILS

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: We use embedding features as inputs of the proposed set encoder instead of raw images, where the embedding features are generated by ResNet18 (He et al., 2016) pretrained with ImageNet-1K (Deng et al., 2009).
Revised: We use embedding features as inputs of the proposed set encoder instead of raw images, where the embedding features are generated by ResNet18 (He et al., 2016) pretrained with ImageNet-1K (Deng et al., 2009).

Original: We adopt the teacher forcing training strategy (Jin et al., 2018), which performs the current decoding process after correcting the decoded graph as the true graph until the previous step.
Revised: We adopt the teacher forcing training strategy (Jin et al., 2018), which performs the current decoding process after correcting the decoded graph as the true graph until the previous step.

Original: This strategy is only used during meta-training and we progress subsequent generation based on the currently decoded graph part without the true graph information in the meta-test.
Revised: This strategy is only used during meta-training and we progress subsequent generation based on the currently decoded graph part without the true graph information in the meta-test.

Original: We use mini-batch gradient descent to train the model with Eq.
Revised: We use mini-batch gradient descent to train the model with Eq.

Original: The values of hyperparameters which we used for both MetaD2A generator and predictor in this paper are described in Table 6.
Revised: The values of hyperparameters which we used for both MetaD2A generator and predictor in this paper are described in Table 6.

Original: To train searched neural architectures for all datasets, we follow the hyperparameter setting of NAS-Bench201 (Dong & Yang, 2020), which is used for training searched neural architectures on CIFAR10 and
Revised: To train searched neural architectures for all datasets, we follow the hyperparameter setting of NAS-Bench201 (Dong & Yang, 2020), which is used for training searched neural architectures on CIFAR10 and

Original: CIFAR100.
Revised: CIFAR100.

Original: While we report accuracy after training 50 epoch for MNIST, the accuracy of 200 epoch are reported for all datasets except MNIST.
Revised: While we report accuracy after training 50 epoch for MNIST, the accuracy of 200 epoch are reported for all datasets except MNIST.