Original: Transfer NAS with Meta-learned Bayesian Surrogates
Revised: Transfer NAS with Meta-learned Bayesian Surrogates

Original: Abstract
Revised: Abstract

Original: While neural architecture search (NAS) is an intensely-researched area, approachestypically still suffer from either (i) high computational costs or (ii) lack of robustness across datasets and experiments.
Revised: While neural architecture search (NAS) is an intensely-researched area, approaches typically still suffer from either (i) high computational costs or (ii) lack of robustness across datasets and experiments.

Original: Furthermore, most methods start searchingfor an optimal architecture from scratch , ignoring prior knowledge.
Revised: Furthermore, most methods start searching for an optimal architecture from scratch , ignoring prior knowledge.

Original: This is incontrast to the manual design process by researchers and engineers that leverageprevious deep learning experiences by, e.g., transferring architectures from previously solved, related problems.
Revised: This is in contrast to the manual design process by researchers and engineers that leverage previous deep learning experiences by, e.g., transferring architectures from previously solved, related problems.

Original: We propose to adopt this human design strategy andintroduce a novel surrogate for NAS, that is meta-learned across prior architectureevaluations across different datasets.
Revised: We propose to adopt this human design strategy and introduce a novel surrogate for NAS, that is meta-learned across prior architecture evaluations across different datasets.

Original: We utilize Bayesian Optimization (BO) withdeep-kernel Gaussian Processes, graph neural networks for the architecture embeddings and a transformer-based set encoder of datasets.
Revised: We utilize Bayesian Optimization (BO) with deep-kernel Gaussian Processes, graph neural networks for the architecture embeddings and a transformer-based set encoder of datasets.

Original: As a result, our methodconsistently achieves state-of-the-art results on six computer vision datasets, whilebeing as fast as one-shot NAS methods.
Revised: As a result, our method consistently achieves state-of-the-art results on six computer vision datasets, while being as fast as one-shot NAS methods.

Original:  Introduction
Revised:  Introduction

Original: While deep learning has removed the need for manual feature engineering, it has shifted this manualwork to the meta-level, introducing the need for manual architecture engineering.
Revised: While deep learning has removed the need for manual feature engineering, it has shifted this manual work to the meta-level introducing the need for manual architecture engineering.

Original: The natural nextstep is to also remove the need to manually define the architecture.
Revised: The natural next step is to also remove the need to manually define the architecture.

Original: This is the problem tackled by thefield of neural architecture search (NAS).
Revised: This is the problem tackled by the field of neural architecture search (NAS).

Original: However, there is no NAS method yet that is both generally robust and efficient.
Revised: However, there is no NAS method yet that is both generally robust and efficient.

Original: Blackbox optimization methods, such as reinforcement learning [1], evolutionary algorithms [2] and Bayesianoptimization [3, 4] reliably minimize the validation error but are slow.
Revised: Blackbox optimization methods, such as reinforcement learning [1], evolutionary algorithms [2] and Bayesian optimization [3, 4] reliably minimize the validation error but are slow.

Original: On the other hand, one-shotmethods [5, 6] often have problems with robustness [7] and the newest trend of zero-cost proxiesoften does not provide more information about an architecture’s performance than simple statistics,such as the architecture’s number of parameters [8].
Revised: On the other hand, one-shot methods [5, 6] often have problems with robustness [7] and the newest trend of zero-cost proxies often does not provide more information about an architecture’s performance than simple statistics, such as the architecture’s number of parameters [8].

Original: An understudied path towards efficiency in NAS is to transfer information across datasets.
Revised: An understudied path towards efficiency in NAS is to transfer information across datasets.

Original: While afew NAS approaches in this direction exist [9–13, 3], an effective approach to obtain efficient NASmethods that has been overlooked in the literature is to exploit the common formulation of NAS as a  optimization (HPO) problem [14–17] and draw on the extensive literature on transfer
Revised: While a few NAS approaches in this direction exist [9–13, 3], an effective approach to obtain efficient NAS methods that has been overlooked in the literature is to exploit the common formulation of NAS as a hyperparameter optimization (HPO) problem [14–17] and draw on the extensive literature on transfer

Original: HPO [18–22] 
Revised: HPO [18–22] 1 .

Original: This approach is naturally motivated by how researchers and engineers tackle new deep learning problems they leverage the knowledge they obtained from previous 
Revised: This approach is naturally motivated by how researchers and engineers tackle new deep learning problems: they leverage the knowledge they obtained from previous experimentation and, e.g., re-use architectures designed for one task and apply or adapt them to a novel task.

Original: In this work, we present an efficient Bayesian Optimization (BO) method with a novel deep-kernelsurrogate that yields a new NAS method which combines the best of both worlds: the reliability ofblackbox optimization at a computational cost in the same order of magnitude as one-shot approaches.
Revised: In this work, we present an efficient Bayesian Optimization (BO) method with a novel deep-kernel surrogate that yields a new NAS method which combines the best of both worlds: the reliability of blackbox optimization at a computational cost in the same order of magnitude as one-shot approaches.

Original: Concretely, we propose a BO method for NAS that leverages dataset-contextualized surrogatesfor transfer learning.
Revised: Concretely, we propose a BO method for NAS that leverages dataset-contextualized surrogates for transfer learning.

Original: Following Lee et al.
Revised: Following Lee et al.

Original: [13], we use a graph encoder [23] to encode neuralarchitectures and an attention-based dataset encoder to obtain context features.
Revised: [13], we use a graph encoder [23] to encode neural architectures and an attention-based dataset encoder to obtain context features.

Original: We then use deepkernel learning [24] to obtain meta-learned kernels for the joint space of architectures and datasets ,allowing us to use the full power of BO for efficient NAS.
Revised: We then use deep kernel learning [24] to obtain meta-learned kernels for the joint space of architectures and datasets , allowing us to use the full power of BO for efficient NAS.

Original: In contrast to standard transfer HPOmethods that meta-learn parametric surrogates from a pool of source datasets [18, 19, 22], in thiswork we explore the direction of meta learning surrogates by contextualizing them on the datasetcharacteristics (a.k.a.
Revised: In contrast to standard transfer HPO methods that meta-learn parametric surrogates from a pool of source datasets [18, 19, 22], in this work we explore the direction of meta learning surrogates by contextualizing them on the dataset characteristics (a.k.a.

Original: meta-features) [25–27].
Revised: meta-features) [25–27].

Original: Exploiting these dataset characteristics is particularlyimportant in NAS to find a strong architecture quickly.
Revised: Exploiting these dataset characteristics is particularly important in NAS to find a strong architecture quickly.

Original: Instead of using manually chosen ad-hocmeta-features, we represent meta-features as the latent embeddings of a set-transformer network forimage datasets [28], that is jointly meta-learned as part of our deep surrogate in an end-to-end manner.
Revised: Instead of using manually chosen ad-hoc meta-features, we represent meta-features as the latent embeddings of a set-transformer network for image datasets [28], that is jointly meta-learned as part of our deep surrogate in an end-to-end manner.

Original: As a result, our surrogates are optimized for efficiently transferring architectures for a new targetdataset based on its meta-features.
Revised: As a result, our surrogates are optimized for efficiently transferring architectures for a new target dataset based on its meta-features.

Original: To sum up, our contributions are as follows:
Revised: To sum up, our contributions are as follows:

Original: • Inspired by manual architecture design, we treat NAS as a transfer or few-shot learning problem.
Revised: • Inspired by manual architecture design, we treat NAS as a transfer or few-shot learning problem.

Original: We leverage ideas from transfer HPO to meta-learn a kernel for Bayesian Optimization, which encodes both architecture and dataset information.
Revised: We leverage ideas from transfer HPO to meta-learn a kernel for Bayesian Optimization, which encodes both architecture and dataset information.

Original: • We are the first to combine deep-kernel Gaussian Processes (GPs) with a graph neuralnetwork encoder, a transformer-based dataset encoder, the first to apply BO with deep GPsto NAS, and the first to do all of this in a transfer NAS setting.
Revised: • We are the first to combine deep-kernel Gaussian Processes (GPs) with a graph neural network encoder, a transformer-based dataset encoder, the first to apply BO with deep GPs to NAS, and the first to do all of this in a transfer NAS setting.

Original: • Our resulting method outperforms both state-of-the-art blackbox NAS methods as well asstate-of-the-art one-shot methods across six computer vision benchmarks.
Revised: • Our resulting method outperforms both state-of-the-art blackbox NAS methods as well as state-of-the-art one-shot methods across six computer vision benchmarks.

Original: To foster reproducibility, we make our code available at https://anonymous.4open.science/r/
Revised: To foster reproducibility, we make our code available at https://github.com/TNAS-DCS/

Original: TNAS-DCS-CC08 .
Revised: TNAS-DCS.git .

Original: We address the points in the “NAS Best Practices Checklist” in Appendix I.
Revised: We address the points in the “NAS Best Practices Checklist” in Appendix H.  Related Work

Revised: NAS is an intensely-researched field, with over 1000 papers published in the last two years alone 2 .

Revised: We therefore limit our discussion of NAS to the most related fields of Bayesian optimization for

Revised: NAS and meta learning approaches for NAS.

Revised: For a full discussion of the NAS literature, we refer the interested readers to a series of recent surveys by Elsken et al.

Original: Proposed Method
Revised: [29], Wistuba et al. [30], Xie et al. [31], Ren et al. [32], Benmeziane et al.

Revised: [33] and Dong et al.

Revised: [34], and for an introduction to BO to

Revised: Shahriari et al.

Revised: [35], Hutter et al.

Revised: Bayesian optimization (BO) for NAS .

Revised: As BO is commonly used in hyperparameter optimization (HPO), one can simply treat architectural choices as categorical hyperparameters and re-use, e.g., tree-based HPO methods that can natively handle categorical choices well [14, 15, 37].

Revised: While

Revised: Gaussian Processes (GPs) are more typically applied to continuous hyperparameters, they can also be used for NAS by creating an appropriate kernel; such kernels for GP-based BO were manually engineered by Swersky et al.

Revised: [38], Kandasamy et al.

Revised: [39] and Ru et al.

Revised: A recent alternative is to exploit (Bayesian) neural networks for BO [40, 41, 4].

Revised: However, while these neural networks are very expressive and parallelize well, they require more data to fit well than GPs and thus are outperformed by GP-based approaches when only a few function evaluations can be afforded.

Revised: In this work, we combine the sample efficiency of GPs and the expressive power of neural networks, by using deep

Revised: GPs combined with a graph neural network encoder.

Revised: Meta learning for NAS.

Revised: To mitigate the computational infeasibility of starting NAS methods from scratch for each new task, several approaches have been proposed along the lines of meta and transfer learning [9–13, 3].

Revised: Most of these warm-start the weights of architectures in a target task [9–12].

Revised: Ru et al.

Revised: [3] extracts architectural motifs that can be reused on other datasets.

Revised: Most related to our work is MetaD2A [13], where the authors propose to generate candidate architectures and rank them conditioned directly on a task, utilizing a meta-feature extractor [28].

Revised: However, there are two key differences in our work: (i) the performance predictor of MetaD2A is not probabilistic and thus can not naturally trade-off exploration vs. exploitation but rather only exploits what has been observed during meta-training.

Revised: (ii)

Revised: During the meta testing phase, MetaD2A simply proposes N architectures for a new task, trains the top- K (as estimated by the performance predictor) out of N architectures and returns the best.

Revised: In particular, the performance of the K evaluated architectures is not used as feedback for MetaD2A and thus the method never adapts to function evaluations on the test task, blindly following what has been observed during meta training.

Revised: This can cause problems for new tasks which are poorly correlated with the meta training data, and indeed, MetaD2A stagnates on several of the datasets in our experimental analysis.

Revised: In contrast, TNAS uses the function evaluations from meta testing to update the surrogate employed within our BO framework, thus allowing to adapt to the meta testing scenario.

Revised: Meta & transfer learning for HPO There are many approaches to achieve meta or transfer learning in HPO, see, e.g., the survey by Vanschoren [42] or the nine different classes of methods discussed by Feurer et al.

Revised: [43, Section 7].

Revised: One particularly promising approach is to employ Deep Kernel

Revised: Learning.

Revised: While existing kernels for BO for NAS are manually defined [38, 39, 3], Deep Kernel

Revised: Learning [24] is an approach that strives to learn the kernel function by using a neural network to transform the input to a latent representation, which is then used in a kernel function.

Revised: Wistuba and Grabocka [22] and Jomaa et al.

Revised: [44] utilized a deep kernel for transfer learning in HPO.

Revised: While numerical hyperparameters can be encoded using an MLP, we adapt this Deep Kernel Learning approach to NAS by using a graph neural network to encode architectures as inputs; we also extend it by encoding datasets into a latent embedding and learning a deep kernel that spans the combined space of architectures and datasets.

Original:  2.1 Bayesian optimization with deep kernel Gaussian processes
Revised:  Proposed Method 3.1 Bayesian optimization with deep kernel Gaussian processes

Original: We start by introducing Gaussian Processes, which represents the surrogate of our method within
Revised: We start by introducing Gaussian Processes, which represents the surrogate of our method within

Original: Bayesian optimization (BO).
Revised: Bayesian optimization (BO).

Original: In a typical hyperparameter optimization (HPO) setup, the inputs x ∈ Xrepresent hyperparameter configurations, and the target y ∈ Y denotes the performance of a machinelearning method when trained with the hyperparameter configuration x .
Revised: In a typical hyperparameter optimization (HPO) setup, the inputs x ∈ X represent hyperparameter configurations, and the target y ∈ Y denotes the performance of a machine learning method when trained with the hyperparameter configuration x .

Original: Consider the training
Revised: Consider the training

Original: D = { ( x i , y i ) } ni =1 and testing D ∗ = { ( x ∗ i , y ∗ i ) } n ∗ i =1 splits of a dataset of evaluated hyperparameters.
Revised: D = { ( x i , y i ) } ni =1 and testing D ∗ = { ( x ∗ i , y ∗ i ) } n ∗ i =1 splits of a dataset of evaluated hyperparameters.

Original: In that context, GPs are non-parametric models that assume a prior over functions, and approximatethe target y ∈ Y ⊆ R + given the features x ∈ X ⊆ R L .
Revised: In that context, GPs are non-parametric models that assume a prior over functions, and approximate the target y ∈ Y ⊆ R + given the features x ∈ X ⊆ R L .

Original: The estimation of the target variable y ∗ forthe test instances x ∗ is also jointly Gaussian as (cid:20) yy ∗ (cid:21) ∼ N (cid:18) 0 , (cid:18)
Revised: The estimation of the target variable y ∗ for the test instances x ∗ is also jointly Gaussian as (cid:20) yy ∗ (cid:21) ∼ N (cid:18) 0 , (cid:18)

Original: K
Revised: K

Original: ( x, x)
Revised: ( x, x

Original:  K( x, x ∗ ) K ( x, x ∗ ) T
Revised: [Equation] K

Original: K ( x ∗ , x ∗ ) (cid:19)(cid:19) .
Revised: [Equation] (cid:19)(cid:19) .

Original: Eachrespective block of the covariance matrix is the result of applying a kernel function k : X × X
Revised: Each respective block of the covariance matrix is the result of applying a kernel function k : X × X

Original: → R +on pairs of instances, e.g., K ( x, x ∗ ) i,j :=
Revised: → R + on pairs of instances, e.g., K ( x, x ∗ ) i,j :=

Original: k ( x i , x ∗ j ) .
Revised: k ( x i , x ∗ j ) .

Original: The estimated posterior mean and the posteriorcovariance of GPs [29] for the target y ∗ of the test instances x ∗ is given as follows:
Revised: The estimated posterior mean and the posterior covariance of GPs [45] for the target y ∗ of the test instances x ∗ is given as follows:

Original: [Equation]
Revised: [Equation]

Original: We refer to, e.g., Murphy [30] for the derivation.
Revised: We refer to, e.g., Murphy [46] for the derivation.

Original: GPs are lazy models that rely on the similarity of thetest instances to the training instances via kernel functions k , such as the Mat´ern kernel.
Revised: GPs are lazy models that rely on the similarity of the test instances to the training instances via kernel functions k , such as the Mat´ern kernel.

Original: Unfortunately,typical kernels used with GPs are designed manually and rely on sub-optimal assumptions [31], whichdeteriorates the GP’s performance.
Revised: Unfortunately, typical kernels used with GPs are designed manually and rely on sub-optimal assumptions [47], which deteriorates the GP’s performance.

Original: A promising direction for designing powerful and efficient kernelfunctions that adapt to a learning task is Deep Kernel Learning [24], where kernels are represented as trainable neural networks.
Revised: A promising direction for designing powerful and efficient kernel functions that adapt to a learning task is Deep Kernel Learning [24], where kernels are represented as trainable neural networks.

Original: A mapping ξ : X → R M projects the features to a latent representation,where similar instances are co-located.
Revised: A mapping ξ : X → R M projects the features to a latent representation, where similar instances are co-located.

Original: The embedding ξ for the deep kernel of our GPs is a fully-connected neural network, that takes asinput the encoding of the architecture ψ as well as the dataset encoding ϕ .
Revised: The embedding ξ for the deep kernel of our GPs is a fully-connected neural network, that takes as input the encoding of the architecture ψ as well as the dataset encoding ϕ .

Original: In detail, the L -dimensionalarchitecture encoding ψ is fused with the K -dimensional dataset encoding ϕ and processed through afully connected neural network ξ : R K + L → R M , where the last layer has M neurons.
Revised: In detail, the L -dimensional architecture encoding ψ is fused with the K -dimensional dataset encoding ϕ and processed through a fully connected neural network ξ : R K + L → R M , where the last layer has M neurons.

Original: The architecture encoding ψ consists of a directed acyclic graph encoder [23] to obtain theencoding for the architectures.
Revised: The architecture encoding ψ consists of a directed acyclic graph encoder [23] to obtain the encoding for the architectures.

Original: By using one GRU cell to traverse the topological order of the DAGin the direction from the input to the output, and another GRU cell to pass through the DAG inthe backward direction, we obtain latent representations of the graph which are then put through afully-connected neural network layer to obtain the encoding for the architecture. 
Revised: By using one GRU cell to traverse the topological order of the DAG in the direction from the input to the output, and another GRU cell to pass through the DAG in the backward direction, we obtain latent representations of the graph which are then put through a fully-connected neural network layer to obtain the encoding for the architecture. [Figure] encoding [Figure]

Original:  The dataset encoding ϕ is based on the encoder proposed by Lee et al.
Revised: GP Bayesian optimization [Figure] The dataset encoding ϕ is based on the encoder proposed by Lee et al.

Original: It consists of two stacked
Revised: It consists of two stacked

Original: Set-Transformer [28] architectures.
Revised: Set-Transformer [28] architectures.

Original: The first Set-Transformer layer captures the interaction betweenrandomly sampled data points of the same class, whereas the second one captures the interactionsbetween the different classes of the dataset.
Revised: The first Set-Transformer layer captures the interaction between randomly sampled data points of the same class, whereas the second one captures the interactions between the different classes of the dataset.

Original: The resulting output of the second Set-Transformer layerrepresents the dataset encoding.
Revised: The resulting output of the second Set-Transformer layer represents the dataset encoding.

Original: We tuned the dimensionality of the embedding of the dataset encoderand graph encoder, as well as the architecture of the feed-forward neural network of our method usingthe multi-fidelity Bayesian optimization method BOHB [32] on the meta-training dataset; pleaserefer to Appendix F for details.
Revised: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder, as well as the architecture of the feed-forward neural network of our method using the multi-fidelity Bayesian optimization method BOHB [ on the -training dataset; please refer to Appendix E for details.

Original: Putting it all together, the kernel/similarity between two architectures, specifically x evaluated ondataset D , and x ′ evaluated on dataset D ′ , is:
Revised: Putting it all together, the kernel/similarity between two architectures, specifically x evaluated on dataset D , and x ′ evaluated on dataset D ′ , is:

Original: [Equation]
Revised: [Equation]

Original: ; w ( ξ ) (cid:17) , ξ (cid:16) (cid:104) ψ ( x ′ ; w ( ψ ) ) , ϕ (cid:16) D ′ ; w ( ϕ ) (cid:17)(cid:105) ; w ( ξ ) (cid:17) ; w ( k ) (cid:33) with w ( ξ ) being the parameters of the neural network ξ , w ( ψ ) the parameters of the architectureencoding ψ , w ( ϕ ) the parameters of the dataset encoding ϕ , and w ( k ) additional parameters ofthe kernel function.
Revised: with w ( ξ ) being the parameters of the neural network ξ , w ( ψ ) the parameters of the architecture encoding ψ , w ( ϕ ) the parameters of the dataset encoding ϕ , and w ( k ) additional parameters of the kernel function.

Original: We denote the cumulative parameters as w := (cid:0) w ( ξ ) , w ( ψ ) , w ( ϕ ) , w ( k ) (cid:1) . 
Revised: We denote the cumulative parameters as w := (cid:0) w ( ξ ) , w ( ψ ) , w ( ϕ ) , w ( k ) (cid:1) . We refer to

Revised: Figure 1 for an overview of our framework.

Original: Allparameters are jointly meta-learned to maximize the marginal likelihood [22], which we discuss indetail in Appendix C.
Revised: All parameters are jointly meta-learned to maximize the marginal likelihood [22], which we discuss in detail in Appendix B.

Original:  Experimental Setup
Revised:  Experimental Setup

Original: We follow the experimental setup and use the same meta datasets as Lee et al.
Revised: We follow the experimental setup and use the same meta datasets as Lee et al.

Original: It consists of4230 meta-training datasets derived from ImageNet.
Revised: It consists of 4230 meta-training datasets derived from ImageNet.

Original: For each of these datasets, the accuracy of one(different) architecture from the NAS-Bench-201 [33] search space is given.
Revised: For each of these datasets, the accuracy of one (different) architecture from the NAS-Bench-201 [48] search space is given.

Original: For the evaluation ofour method and the baselines, we use six popular computer vision datasets: CIFAR-10, CIFAR100, SVHN, Aircraft, Oxford IIT Pets, and MNIST.
Revised: For the evaluation of our method and the baselines, we use six popular computer vision datasets: CIFAR-10, CIFAR100, SVHN, Aircraft, Oxford IIT Pets, and MNIST.

Original: For CIFAR-10 and CIFAR-100, we query theperformances of architectures from the NAS-Bench-201 benchmark, whereas for the other fourdatasets we train the suggested architectures from scratch using the NAS-Bench-201 pipeline (asthese are not available in the benchmark).
Revised: For CIFAR-10 and CIFAR-100, we query the performances of architectures from the NAS-Bench-201 benchmark, whereas for the other four datasets we train the suggested architectures from scratch using the NAS-Bench-201 pipeline (as these are not available in the benchmark).

Original: For all experiments, we use the NAS-Bench-201 searchspace.
Revised: For all experiments, we use the NAS-Bench-201 search space.

Original: We ran three trials for each experiment and report the mean and standard deviations.
Revised: We ran three trials for each experiment and report the mean and standard deviations.

Original:  Research Hypotheses and Experimental Results
Revised:  Research Hypotheses and Experimental Results

Original: Our experiments are designed to validate the following research hypotheses for our approach, dubbed
Revised: Our experiments are designed to validate the following research hypotheses for our approach, dubbed

Original: TNAS :
Revised: TNAS :

Original: C
Revised: C

Original: I F
Revised: I F

Original: A R
Revised: A R

Original: C
Revised: C

Original: I F
Revised: I F

Original: A R
Revised: A R

Original: M N I S T
Revised: M N I S T

Original: A I R C R
Revised: A I R C R

Original: A F T
Revised: A F T

Original: P E T S
Revised: P E T S

Original: S V H N
Revised: S V H N

Original: R a n k
Revised: R a n k

Original: SETN GDAS
Revised: SETN GDAS

Original: PC-Darts DrNAS
Revised: PC [Figure] Darts DrNAS

Original: RS GP
Revised: RS GP

Original: BANANAS NASBOWL
Revised: BANANAS NASBOWL

Original: HEBO MetaD2A
Revised: HEBO MetaD2A

Original: TNAS (a) Ranking of TNAS across benchmarks.
Revised: TNAS ( [Figure] Ranking of TNAS across benchmarks.

Original: 30 # GPU hours 
Revised: [Figure] hours [Figure]

Original: A v e r a g e R a n k
Revised: v e r a g e R a n k

Original: SETN GDAS PC-Darts DrNAS RS GP
Revised: [Figure]

Original: BANANAS NASBOWL
Revised: BANANAS [Figure]

Original: HEBOMetaD2A TNAS (
Revised: HEBO

Original: [Figure]
Revised: [Figure]

Original: Average Ranking over the course of runtime. Hypothesis 1 : TNAS is more efficient than classical HPO methods applied to NAS as well as HPOmethods specifically adapted to NAS and outperforms them in terms of anytime-performance, whileachieving strong final performance.
Revised: Hypothesis 1 : TNAS is more efficient than classical HPO methods applied to NAS as well as HPO methods specifically adapted to NAS and outperforms them in terms of anytime-performance, while achieving strong final performance.

Original: Hypothesis 2 : TNAS is competitive with one-shot approaches in terms of runtime.
Revised: Hypothesis 2 : TNAS is competitive with one-shot approaches in terms of runtime.

Original: Hypothesis 3 : TNAS achieves the consistency of blackbox optimization algorithms (such as classical
Revised: Hypothesis 3 : TNAS achieves the consistency of blackbox optimization algorithms (such as classical

Original: HPO methods) while being as efficient as one-shot methods.
Revised: HPO methods) while being as efficient as one-shot methods.

Original: For all classical HPO methods except for RS, we use the 5 top-performing architectures from themeta-training dataset as a starting point.
Revised: For all classical HPO methods except for RS, we use the 5 top-performing architectures from the meta-training dataset as a starting point.

Original: In that sense, all these baselines are “transfer learning” bybeing initialized with the best architectures on the meta-training dataset.
Revised: In that sense, all these baselines are “transfer learning” by being initialized with the best architectures on the meta-training dataset.

Original: In Figure 1, we show that TNAS consistently achieves strong results, while the existing state-of-the-art
Revised: In Figure 2, we show that TNAS consistently achieves strong results, while the existing state-of-the-art

Original: NAS baselines have much higher variance - their ranking changes across benchmarks.
Revised: NAS baselines have much higher variance - their ranking changes across benchmarks.

Original: Furthermore,the figure shows how the ranking evolves over the course of running of the methods (by means ofruntime in GPU hours).
Revised: Furthermore, the figure shows how the ranking evolves over the course of running of the methods (by means of runtime in GPU hours).

Original: The analysis indicates that TNAS consistently achieves the best performance,for both small and large computational budgets, in particular also when compared to MetaD2A.
Revised: The analysis indicates that TNAS consistently achieves the best performance, for both small and large computational budgets, in particular also when compared to MetaD2A.

Original: We show more detailed results on these six datasets in Appendix E.1.
Revised: We show more detailed results on these six datasets in Appendix D.1.

Original: We show the performancecomparison of TNAS to classical HPO methods in Figure 3, whereas Figure 4 shows the comparisonto one-shot NAS methods.
Revised: We show the performance comparison of TNAS to classical HPO methods in Figure 3, whereas Figure 4 shows the comparison to one-shot NAS methods.

Original: Furthermore, in Appendix E.2 we empirically analyse our design choices,and demonstrate a lift in performance compared to ceteris paribus ablations that do not employ thesedesigns.
Revised: Furthermore, in Appendix D.2 we empirically analyse our design choices, and demonstrate a lift in performance compared to ceteris paribus ablations that do not employ these designs.

Original:  Conclusions and Future Work
Revised:  Conclusions and Future Work

Original: NAS is an intensely researched task and in essence is an instance of the hyperparameter optimization (HPO) problem.
Revised: NAS is an intensely researched task and in essence is an instance of the hyperparameter optimization (HPO) problem.

Original: In this work, we exploited this relationship and, motivated by state-of-the-arttransfer HPO methods, adapted deep Gaussian Process (GP) surrogates to capture architecture representations.
Revised: In this work, we exploited this relationship and, motivated by state-of-the-art transfer HPO methods, adapted deep Gaussian Process (GP) surrogates to capture architecture representations.

Original: Furthermore, we proposed a novel conditioning of the deep GP on dataset meta-features toenable transferring well-performing architectures from source datasets with similar meta-features.
Revised: Furthermore, we proposed a novel conditioning of the deep GP on dataset meta-features to enable transferring well-performing architectures from source datasets with similar meta-features.

Original: Inaddition, we showed that our novel deep GPs with dataset meta-features and architecture encodingsachieve the highest accuracy on six computer vision datasets compared to a broad range of HPOmethods, BO methods for NAS, and one-shot NAS methods.
Revised: In addition, we showed that our novel deep GPs with dataset meta-features and architecture encodings achieve the highest accuracy on six computer vision datasets compared to a broad range of HPO methods, BO methods for NAS, and one-shot NAS methods.

Original: Lastly, we demonstrated that proxyarchitecture evaluations allow our method to discover more accurate architectures within the samesearch time one-shot NAS methods require.
Revised: Lastly, we demonstrated that proxy architecture evaluations allow our method to discover more accurate architectures within the same search time one-shot NAS methods require.

Original: B Method Preliminaries
Revised: A Method Preliminaries

Original: [Figure] We consider the following problem: given a history of Q datasets, where for each dataset D ( q )we have already evaluated a set of neural network architectures x ( q ) 1 , . . .
Revised: We consider the following problem: given a history of Q datasets, where for each dataset D ( q ) we have already evaluated a set of neural network architectures x ( q ) 1 , . . .

Original: , x ( q ) n with correspondingperformance (e.g., accuracy) y ( q ) 1 , . ..
Revised: , x ( q ) n with corresponding performance (e.g., accuracy) y ( q ) 1 , . .

Original: , y ( q ) n .
Revised: , y ( q ) n .

Original: For a new dataset D ( new ) , we want to quickly discoveran optimal architecture by leveraging information from the history of datasets.
Revised: For a new dataset D ( new ) , we want to quickly discover an optimal architecture by leveraging information from the history of datasets.

Original: We build upon thestate-of-the-art few-shot Bayesian optimization (BO) framework by Wistuba and Grabocka [22],which was proposed to address a similar problem: transferring optimal hyperparameter configurationsacross datasets.
Revised: We build upon the state-of-the-art few-shot Bayesian optimization (BO) framework by Wistuba and Grabocka [22], which was proposed to address a similar problem: transferring optimal hyperparameter configurations across datasets.

Original: The authors propose to learn a deep kernel across tasks, which is then used for a
Revised: The authors propose to learn a deep kernel across tasks, which is then used for a

Original: Gaussian process (GP) surrogate in the typical BO setup.
Revised: Gaussian process (GP) surrogate in the typical BO setup.

Original: However, while hyperparameter configurations can typically be presented by an N-dimensional vector,it is less clear how to represent neural network architectures.
Revised: However, while hyperparameter configurations can typically be presented by an N-dimensional vector, it is less clear how to represent neural network architectures.

Original: Simply representing architectures asvectors and plugging them into an off-the-shelf GP kernel is likely sub-optimal.
Revised: Simply representing architectures as vectors and plugging them into an off-the-shelf GP kernel is likely sub-optimal.

Original: In fact, White et al.[49] have shown that the type of architecture representation substantially impacts the performance ofa downstream NAS algorithm.
Revised: In fact, White et al. [49] have shown that the type of architecture representation substantially impacts the performance of a downstream NAS algorithm.

Original: To address this issue, we employ graph neural networks (GNNs) toobtain a learnable representation of neural networks.
Revised: To address this issue, we employ graph neural networks (GNNs) to obtain a learnable representation of neural networks.

Original: GNNs are a common choice in NAS as neuralnetworks architectures can be naturally represented as graphs [50, 49, 51, 52].
Revised: GNNs are a common choice in NAS as neural networks architectures can be naturally represented as graphs [50, 49, 51, 52].

Original: Furthermore, while we could directly feed this architecture encoding into the GP’s kernel, we arguethat the kernel should also be conditioned on the characteristics of a dataset in order to meaningfullyasses (dis-)similarities between architectures.
Revised: Furthermore, while we could directly feed this architecture encoding into the GP’s kernel, we argue that the kernel should also be conditioned on the characteristics of a dataset in order to meaningfully asses (dis-)similarities between architectures.

Original: To motivate this, consider the question “What makestwo architectures similar?”.
Revised: To motivate this, consider the question “What makes two architectures similar?”.

Original: We argue that two architectures are not similar only because they sharesome similar sub-graph components (which will be represented by the GNN encoding), but alsobecause they achieve similar performance on the target dataset.
Revised: We argue that two architectures are not similar only because they share some similar sub-graph components (which will be represented by the GNN encoding), but also because they achieve similar performance on the target dataset.

Original: Following this line of reasoning, wecondition the deep kernel on the characteristics (meta-features) of a dataset [25].
Revised: Following this line of reasoning, we condition the deep kernel on the characteristics (meta-features) of a dataset [25].

Original: In a similar fashionas for the architecture encoding, we again use a learnable representation of datasets via employing aset transformer [28].
Revised: In a similar fashion as for the architecture encoding, we again use a learnable representation of datasets via employing a set transformer [28].

Original: The architecture and dataset encoding are then processed by a fully-connected neural network, whoseoutput serves as the input for an off-the-shelf kernel function, e.g., a Mat´ern kernel, which is finallyused to compute the distance of two (architecture, dataset) datapoints.
Revised: The architecture and dataset encoding are then processed by a fully-connected neural network, whose output serves as the input for an off-the-shelf kernel function, e.g., a Mat´ern kernel, which is finally used to compute the distance of two (architecture, dataset) datapoints.

Original: This results in an end-to-endlearnable encoding of the problem, and the parameters of the GNN, transformer and fully-connectedneural network are meta-learned in a similar fashion as in Wistuba and Grabocka [22].We refer to Figure 2 for an overview of our framework.
Revised: This results in an end-to-end learnable encoding of the problem, and the parameters of the GNN, transformer and fully-connected neural network are meta-learned in a similar fashion as in Wistuba and Grabocka [22].

Original: C Meta-learning deep-kernel GP Surrogates
Revised: B Meta-learning deep-kernel GP Surrogates

Original: Recall that we assume we are given a set of Q datasets, where on each dataset D q we have N q ∈ N +evaluated architectures.
Revised: Recall that we assume we are given a set of Q datasets, where on each dataset D q we have N q ∈ N + evaluated architectures.

Original: We denote the n -th architecture evaluated on the q -th dataset as x q,n and itsvalidation accuracy as y q,n .
Revised: We denote the n -th architecture evaluated on the q -th dataset as x q,n and its validation accuracy as y q,n .

Original: Overall, the meta-dataset of all the evaluations on all the datasets is definedas M :
Revised: Overall, the meta-dataset of all the evaluations on all the datasets is defined as M :

Original: = (cid:83) Qq =

Revised: = (cid:83) Qq =

Original: (cid:83) N q n =1 { ( x q,n , y q,n , D q ) } .
Revised: (cid:83) N q n =1 { ( x q,n , y q,n , D q ) } .

Original: By x := ( x 1 , 1 , . . .
Revised: By x := ( x 1 , 1 , . . .

Original: , x Q,N q ) , y :
Revised: , x Q,N q ) , y :

Original: =
Revised: =

Original: ( y 1 , 1 , . . .
Revised: ( y 1 , 1 , . . .

Original: , y Q,N q ) and
Revised: , y Q,N q ) and

Original: D :=
Revised: D :=

Original: ( D 1 , . . .
Revised: ( D 1 , . . .

Original: , D Q ) we denote the vectors containing all the architectures, accuracies and datasets,respectively.
Revised: , D Q ) we denote the vectors containing all the architectures, accuracies and datasets, respectively.

Original: The parameters w of the deep kernel are optimized jointly by maximizing the log marginal likelihoodof the GP surrogate on the meta training dataset:
Revised: The parameters w of the deep kernel are optimized jointly by maximizing the log marginal likelihood of the GP surrogate on the meta training dataset:

Original: [Equation]
Revised: [Equation]

Original: 1: Require: meta-dataset M ; learning rates η SGD , η REP ; inner update steps v . 2: while not converged do 3: Sample mini M : x = [ x 1 , . . .
Revised: [Equation]

Original: , x k ] , y =
Revised: , x k ] , y =

Original: [ y 1 , . . .
Revised: [ y 1 , . . .

Original: , y k ] , D =
Revised: , y k ] , D =

Original: [ D 1 , . . .
Revised: [ D 1 , . . .

Original: , D k ] 4: L ( w ) =
Revised: , D k ] 4: L ( w ) =

Original: y T K − 1 ( x , D ; w ) y + log | K ( x , D ; w ) | 5: w ′ ← w 6: for j = 1 to v do 

Revised: y T K − 1 ( x , D ; w ) y + log | K ( x , D ; w ) | 5: w ′ ← w 6: for j = 1 to v do

Original: : w ′ ← w ′ − η SGD ∇ w ′ L ( w ′ ) 8: Update w ← w − η REP ( w − w ′ )
Revised: : w ′ ← w ′ − η SGD ∇ w ′ L ( w ′ ) 8: Update w ← w − η REP ( w − w ′ )

Original: In practice, we resort to samplingmini-batches and employ stochastic gradient descent, following established practices on Deep Kernel
Revised: In practice, we resort to sampling mini-batches and employ stochastic gradient descent, following established practices on Deep Kernel

Original: Learning for GPs [24, 22, 53].
Revised: Learning for GPs [24, 22, 53].

Original: As allcomponents of our method are differentiable, our approach is end-to-enddifferentiable.
Revised: As all components of our method are differentiable, our approach is end-to-end differentiable.

Original: For updating the meta-parameters w ,we use the meta-learning algorithm
Revised: For updating the meta-parameters w , we use the meta-learning algorithm

Original: REPTILE [54], due to its simplicityand faster convergence time comparedto, e.g., MAML [55].
Revised: REPTILE [54], due to its simplicity and faster convergence time compared to, e.g., MAML [55].

Original: Our method’s pseudocode is given in Algorithm 1.
Revised: Our method’s pseudocode is given in Algorithm 1.

Original: The procedure samples amini-batch of (architecture, dataset) pairs with corresponding validation accuracy (line 3).
Revised: The procedure samples a mini-batch of (architecture, dataset) pairs with corresponding validation accuracy (line 3).

Original: We simplysample datasets and architectures uniformly at random from the meta-training dataset.
Revised: We simply sample datasets and architectures uniformly at random from the meta-training dataset.

Original: Then we fit thesurrogate to estimate the accuracies y of architectures x on a data set D using SGD by minimizing Equation 3 (lines 4-7).
Revised: Then we fit the surrogate to estimate the accuracies y of architectures x on a data set D using SGD by minimizing Equation 3 (lines 4-7).

Original: Finally, we use REPTILE [54] to update w (line 8).
Revised: Finally, we use REPTILE [54] to update w (line 8).

Original: Once the optimal ware found, we plug in the meta-learned kernel for a GP, with the posterior and use vanilla Bayesian Optimization (BO) to quickly identify the optimal configuration in the new response surface.
Revised: Once the optimal w are found, we plug in the meta-learned kernel for a GP, with the posterior and use vanilla Bayesian Optimization (BO) to quickly identify the optimal configuration in the new response surface.

Original: D Baselines
Revised: C Baselines

Original: Classic HPO.
Revised: Classic HPO.

Original: A simple and standard baseline is Random Search (RS) [56].
Revised: A simple and standard baseline is Random Search (RS) [56].

Original: RS samples architecturesuniformly at random from the search space and returns the top preforming one.
Revised: RS samples architectures uniformly at random from the search space and returns the top preforming one.

Original: Another simple yetpowerful baseline is Bayesian Optimization with a vanilla GP surrogate [57].
Revised: Another simple yet powerful baseline is Bayesian Optimization with a vanilla GP surrogate [57].

Original: We use the Mat´ern 5 / 2kernel and rely on the GPytorch [58] implementation.
Revised: We use the Mat´ern 5 /kernel and rely on the GPytorch [58] implementation.

Original: We tried both Expected Improvement (EI)and Upper Confidence Bound (UCB) as acquisition functions, with UCB performing better in ourexperiments.
Revised: We tried both Expected Improvement (EI) and Upper Confidence Bound (UCB) as acquisition functions, with UCB performing better in our experiments.

Original: Additionally, we compare to HEBO [31], a black-box HPO method that performs inputand output warping to mitigate the effects of heteroscedasticity and non-stationarity on HPO problems.
Revised: Additionally, we compare to HEBO [47], a black-box HPO method that performs input and output warping to mitigate the effects of heteroscedasticity and non-stationarity on HPO problems.

Original: HEBO won the 2020 NeurIPS blackbox optimization challenge [59].
Revised: HEBO won the 2020 NeurIPS blackbox optimization challenge [59].

Original: We use the implementationprovided by the authors.
Revised: We use the implementation provided by the authors.

Original: HPO for NAS.
Revised: HPO for NAS.

Original: White et al.
Revised: White et al.

Original: [4] proposed a BO method for NAS that uses an ensemble of fullyconnected neural networks as a surrogate, named BANANAS.
Revised: [4] proposed a BO method for NAS that uses an ensemble of fullyconnected neural networks as a surrogate, named BANANAS.

Original: Moreover, BANANAS uses a pathencoding for the neural architectures, which serves as an input to the ensemble.
Revised: Moreover, BANANAS uses a path encoding for the neural architectures, which serves as an input to the ensemble.

Original: When appliedto a new test task, BANANAS starts the neural architecture search from scratch.
Revised: When applied to a new test task, BANANAS starts the neural architecture search from scratch.

Original: NASBOWL [3]is a GP-based BO method for NAS and utilizes the Weisfeiler-Lehman kernel [60].
Revised: NASBOWL [3] is a GP-based BO method for NAS and utilizes the Weisfeiler-Lehman kernel [60].

Original: We use theimplementations provided by the authors for BANANAS and NASBOWL.[-A cc u r a c y CIFAR      
 CIFAR      99.-]
Revised: We use the implementations provided by the authors for BANANAS and NASBOWL.[-A cc u r a c y CIFAR      
 CIFAR      99.-]

Original: MNIST    20 #

Original: [Figure]

Original: hours

Original: A cc u r a c y

Original: AIRCRAFT    20 # GPU hours

Original: PETS     25 # GPU hours 96.

Original: SVHN

Original: [Figure]

Original: TNAS

Original: State-of-the-art in NAS.
Revised: State-of-the-art in NAS.

Original: One-shot methods have recently shown strong empirical performance for
Revised: One-shot methods have recently shown strong empirical performance for

Original: NAS.
Revised: NAS.

Original: We compare to GDAS [6], SETN [61], PC-DARTS [62] and DrNAS [63].
Revised: We compare to GDAS [6], SETN [61], PC-DARTS [62] and DrNAS [63].

Original: For these methods,we compare to published results from the literature for six computer vision datasets.
Revised: For these methods, we compare to published results from the literature for six computer vision datasets.

Original: Transfer NAS.
Revised: Transfer NAS.

Original: We compare to the most related transfer NAS method, MetaD2A [13], whichis a transfer learning NAS method with a dataset-contextualized neural network generator andperformance predictor.
Revised: We compare to the most related transfer NAS method, MetaD2A [13], which is a transfer learning NAS method with a dataset-contextualized neural network generator and performance predictor.

Original: The neural network generator and performance predictor are meta-trainedon the same source datasets that we also use.
Revised: The neural network generator and performance predictor are meta-trained on the same source datasets that we also use.

Original: When applied to a test task, MetaD2A generates 500candidate architectures conditioned on the test dataset and then selects the top architectures based onits performance predictor.
Revised: When applied to a test task, MetaD2A generatescandidate architectures conditioned on the test dataset and then selects the top architectures based on its performance predictor.

Original: We use the implementation provided by MetaD2A’s authors.
Revised: We use the implementation provided by MetaD2A’s authors.

Original: E Additional Experimental Results 
Revised: D Additional Experimental Results D.1 Comparison of TNAS to baselines

Original: E.1 Comparison of TNAS to baselines

Original: Results for Hypothesis 1.
Revised: Results for Hypothesis 1.

Original: In Figure 3, we compare the performance of TNAS, with several HPObaselines.
Revised: In Figure 3, we compare the performance of TNAS, with several HPO baselines.

Original: For all methods, we use the 5 top-performing architectures from the meta-training datasetas a starting point.
Revised: For all methods, we use the 5 top-performing architectures from the meta-training dataset as a starting point.

Original: In that sense, all these baselines are “transfer learning” by being initialized with thebest architectures on the meta-training dataset.
Revised: In that sense, all these baselines are “transfer learning” by being initialized with the best architectures on the meta-training dataset.

Original: On all of the datasets except CIFAR100, TNAS findstop-performing architectures faster than all the baselines and achieves stronger anytime performance.
Revised: On all of the datasets except CIFAR100, TNAS finds top-performing architectures faster than all the baselines and achieves stronger anytime performance.

Original: Furthermore, on all benchmarks, TNAS eventually performs best.
Revised: Furthermore, on all benchmarks, TNAS eventually performs best.

Original: Results for Hypothesis 2.
Revised: Results for Hypothesis 2.

Original: We demonstrate that with our meta-learned deep kernel within Bayesian
Revised: We demonstrate that with our meta-learned deep kernel within Bayesian

Original: Optimization, the search time can be significantly reduced, to the same order as one-shot approaches.
Revised: Optimization, the search time can be significantly reduced, to the same order as one-shot approaches.

Original: Figure 4 shows the performance of both TNAS and MetaD2A compared to state-of-the-art NASmethods.
Revised: Figure 4 shows the performance of both TNAS and MetaD2A compared to state-of-the-art NAS methods.

Original: Except for CIFAR10 and CIFAR100, TNAS clearly outperforms the NAS baselines by thetime the one-shot approaches finish the search.
Revised: Except for CIFAR10 and CIFAR100, TNAS clearly outperforms the NAS baselines by the time the one-shot approaches finish the search.

Original: On CIFAR10, TNAS achieves similar performance asthe baselines, while it is slightly inferior on CIFAR100 for some baselines and only overtakes themgiven more time.
Revised: On CIFAR10, TNAS achieves similar performance as [Figure] [Figure] the baselines, while it is slightly inferior on CIFAR100 for some baselines and only overtakes them given more time.

Original: We refer to Table 1 in the appendix for concrete numbers.
Revised: We refer to Table 1 in the appendix for concrete numbers.

Original: We can furthermoreobserve the drawbacks of MetaD2A discussed earlier: (i) MetaD2A does not trade-off exploration vs.exploitation but rather only exploits what has been observed during meta-training and (ii) MetaD2A [Figure] [Figure]
Revised: We can furthermore observe the drawbacks of MetaD2A discussed earlier: (i) MetaD2A does not trade-off exploration vs. exploitation but rather only exploits what has been observed during meta-training and (ii) MetaD2A does not use evaluations on the new target dataset as feedback and never adapts.

Original: does not use evaluations on the new target dataset as feedback and never adapts.

Original: As a result, MetaD2Astagnates on several of the datasets.
Revised: As a result, MetaD2A stagnates on several of the datasets.

Original: E.2 Ablating our Design Choices
Revised: D.2 Ablating our Design Choices

Original: We empirically analyse our design choices, namely the graph and dataset encoder and demonstrate alift in performance compared to ceteris paribus ablations that do not employ these designs. 
Revised: We empirically analyse our design choices, namely the graph and dataset encoder and demonstrate a lift in performance compared to ceteris paribus ablations that do not employ these designs. [Table] [Figure]

Original: Concretely, we ablate our method’s components and test them on CIFAR-10 and CIFAR-100,reporting the result in Figure 5.
Revised: Concretely [Figure] we ablate our method’s components and test them on CIFAR-10 and CIFAR-100, reporting the result in Figure 5.

Original: TNAS (no Set Encoder) shows the performance of our method usinga graph encoder, but no set-encoder (i.e., without using any dataset meta-features).
Revised: TNAS (no Set Encoder) shows the performance of our method using a graph encoder [Figure] but no set-encoder (i [Figure]

Revised: e., without using any dataset meta-features).

Original: TNAS (no Graph
Revised: TNAS (no Graph

Original: Encoder) shows the performance of our method using the dataset encoding in combination with amatrix encoding for the architectures (i.e., no graph encoder).
Revised: Encoder) shows the performance of our method using the dataset encoding in combination with a matrix encoding for the architectures (i [Figure] e., no graph encoder).

Original: TNAS outperforms the other variations;thus, we conclude that using both the learnable dataset meta-features and a graph neural networkencoding is the most robust surrogate design.
Revised: TNAS outperforms the other variations; thus, we conclude that using both the learnable dataset meta-features and a graph neural network encoding is the most robust surrogate design.

Original: This finding validates the design choices of our method.
Revised: This finding validates the design choices of our method

Original:  We also empirically evaluated (i) whether there is actually a benefit in meta-learning the surrogate,and (ii) our method’s performance with different initial architectures.
Revised: [Figure] We also empirically evaluated (i) whether there is actually a benefit in meta-learning the surrogate, and (ii) our method’s performance with different initial architectures.

Original: Remember that as an initialdesign for our method, we used the top-5 performing architectures from the meta-training dataset.
Revised: Remember that as an initial design for our method, we used the top-5 performing architectures from the meta-training dataset.

Original: As alternatives, we consider values other than 5 and also start from randomly sampled architectures.
Revised: As alternatives, we consider values other than 5 and also start from randomly sampled architectures.

Original: We also turn-off meta-learning.
Revised: We also turn-off meta-learning.

Original: The plots for these experiments are shown in Figure 6, evaluated on
Revised: The plots for these experiments are shown in Figure 6, evaluated on

Original: CIFAR-10 and CIFAR-100.
Revised: CIFAR-10 and CIFAR-100.

Original: The results suggest that both design choices are beneficial.
Revised: The results suggest that both design choices are beneficial.

Original: [Table]

Original: F Hyperparameter optimization for GNN and set transformer architectures
Revised: E Hyperparameter optimization for GNN and set transformer architectures

Original: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder ( Embeddingdims. ), the architecture of the feed-forward neural network of our method ( Num. layers , Num. unitsin layer 1 , Num. units in layer 2 , Num. units in layer 3 , Num. units in layer 4 ), and the learning rateof the joint meta-training using BOHB [32] on the meta-training dataset.
Revised: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder ( Embedding dims. ), the architecture of the feed-forward neural network of our method ( Num. layers , Num. units in layer 1 , Num. units in layer 2 , Num. units in layer 3 , Num. units in layer 4 ), and the learning rate of the joint meta-training using BOHB [37] on the meta-training dataset.

Original: G Societal Implications
Revised: F Societal Implications

Original: Better optimization of neural network architectures allows the enhancement of the performance ofmany crucial applications of deep learning, such as climate science, medicine, and recommendersystems.
Revised: Better optimization of neural network architectures allows the enhancement of the performance of many crucial applications of deep learning, such as climate science, medicine, and recommender systems.

Original: Furthermore, through the use of transfer learning to speed up the optimization of neuralarchitectures, we reduce the time costs, and therefore the energy consumption of our method.
Revised: Furthermore, through the use of transfer learning to speed up the optimization of neural architectures, we reduce the time costs, and therefore the energy consumption of our method.

Original: However,there are also potential disadvantages in the neural network components of our method because they
Revised: However, there are also potential disadvantages in the neural network components of our method because they

Original: [Figure]

Original: [Table]
Revised: [Table]

Original: are more difficult to interpret compared to simpler, traditional ML methods; we therefore encourageresearch on interpretable deep learning.
Revised: are more difficult to interpret compared to simpler, traditional ML methods; we therefore encourage research on interpretable deep learning.

Original: H Future work
Revised: G Future work

Original: A deep surrogate in principle allows us to capture the interaction between architectures and hyperparameter configurations.
Revised: A deep surrogate in principle allows us to capture the interaction between architectures and hyperparameter configurations.

Original: The NAS literature at the moment underexplores the impact of hyperparameters on the performance of an architecture.
Revised: The NAS literature at the moment underexplores the impact of hyperparameters on the performance of an architecture.

Original: In fact, it is commonly known that the same architecturewould perform differently if the training pipeline is altered, for example by changing the learningrate, the number of epochs, or the degree of regularization.
Revised: In fact, it is commonly known that the same architecture would perform differently if the training pipeline is altered, for example by changing the learning rate, the number of epochs, or the degree of regularization.

Original: Our previously-defined surrogate can betrivially extended to model the interaction of architecture embeddings, the dataset meta-features andhyperparameter configurations.
Revised: Our previously-defined surrogate can be trivially extended to model the interaction of architecture embeddings, the dataset meta-features and hyperparameter configurations.

Original: We have not explored this direction empirically due to the lack ofavailable NAS meta-datasets that vary both architectures and hyperparameters but would like to do inthe future.
Revised: We have not explored this direction empirically due to the lack of available NAS meta-datasets that vary both architectures and hyperparameters but would like to do in the future.

Original: I NAS Best Practice Checklist
Revised: H NAS Best Practice Checklist

Original: We now describe how we addressed the individual points of the NAS best practice checklist [64].
Revised: We now describe how we addressed the individual points of the NAS best practice checklist [64].

Original: 1. Best Practices for Releasing Code
Revised: 1. Best Practices for Releasing Code

Original: For all experiments you report:
Revised: For all experiments you report:

Original: (a) Did you release code for the training pipeline used to evaluate the final architectures?
Revised: (a) Did you release code for the training pipeline used to evaluate the final architectures?

Original: The code for the training pipeline for the architectures can be found in the repo we provide.
Revised: The code for the training pipeline for the architectures can be found in the repo we provide.

Original: (b) Did you release code for the search space?
Revised: (b) Did you release code for the search space?

Original: We used the NAS-Bench-201 search space in our experiments, the description and code for which is publicly available.
Revised: We used the NAS-Bench-201 search space in our experiments, the description and code for which is publicly available.

Original: (c) Did you release the hyperparameters used for the final evaluation pipeline, as well as random seeds?
Revised: (c) Did you release the hyperparameters used for the final evaluation pipeline, as well as random seeds?

Original: We the NAS-Bench-201 pipeline and hyperparameters as our final evaluation pipeline.
Revised: We the NAS-Bench-201 pipeline and hyperparameters as our final evaluation pipeline.

Original: We release it as well as the random seeds in the repo we provide.
Revised: We release it as well as the random seeds in the repo we provide.

Original: (d) Did you release code for your NAS method?
Revised: (d) Did you release code for your NAS method?

Original: The code for our NAS method can be found in https://anonymous.4open.science/r/TNAS-DCS-CC08 .
Revised: The code for our NAS method can be found in https://anonymous.4open.science/r/TNAS-DCS-CC08 .

Original: (e) Did you release hyperparameters for your NAS method, as well as random seeds?
Revised: (e) Did you release hyperparameters for your NAS method, as well as random seeds?

Original: The

Original:  hyperparameters for our NAS method, as well as random seeds for the experiments can be found in the repo we provide.
Revised: The hyperparameters for our NAS method, as well as random seeds for the experiments can be found in the repo we provide.

Original: 2. Best practices for comparing NAS methods
Revised: 2. Best practices for comparing NAS methods

Original: (a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code??
Revised: (a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code??

Original: Yes, for fair comparison we made sure to use the same evaluation pipeline for all NAS methods we compare.
Revised: Yes, for fair comparison we made sure to use the same evaluation pipeline for all NAS methods we compare.

Original: (b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)?
Revised: (b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)?

Original: We ran all the methods on the same hardware and the same environment to control for confounding factors.
Revised: We ran all the methods on the same hardware and the same environment to control for confounding factors.

Original: (c) Did you run ablation studies?
Revised: (c) Did you run ablation studies?

Original: We ran extensive ablation studies on the components of our method.
Revised: We ran extensive ablation studies on the components of our method.

Original: (d) Did you use the same evaluation protocol for the methods being compared?
Revised: (d) Did you use the same evaluation protocol for the methods being compared?

Original: Yes.
Revised: Yes.

Original: (e) Did you compare performance over time?
Revised: (e) Did you compare performance over time?

Original: Yes.
Revised: Yes.

Original: (f) Did you compare to random search?
Revised: (f) Did you compare to random search?

Original: Performance comparison to random search as well as other baselines can be found in Figure 3.
Revised: Performance comparison to random search as well as other baselines can be found in Figure 3.

Original: (g) Did you perform multiple runs of your experiments and report seeds?
Revised: (g) Did you perform multiple runs of your experiments and report seeds?

Original: For each of the experiments we performed three runs with different seeds (333 , 444 , 555) .
Revised: For each of the experiments we performed three runs with different seeds (333 , 444 , 555) .

Original: (h) Did you use tabular or surrogate benchmarks for in-depth evaluations?
Revised: (h) Did you use tabular or surrogate benchmarks for in-depth evaluations?

Original: We used NAS-Bench-201 as a tabular benchmark.
Revised: We used NAS-Bench-201 as a tabular benchmark.

Original: 3. Best practices for reporting important details
Revised: 3. Best practices for reporting important details

Original: (a) Did you report how you tuned hyperparameters, and what time and resources this required?
Revised: (a) Did you report how you tuned hyperparameters, and what time and resources this required?

Original: We use BOHB[32] to tune the hyperparameters of our method.
Revised: We use BOHB[37] to tune the hyperparameters of our method.

Original: We ran BOHB with three different random seeds for 24 hours.
Revised: We ran BOHB with three different random seeds for 24 hours.

Original: (b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)?
Revised: (b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)?

Original: Yes.
Revised: Yes.

Original: (c) Did you report all the details of your experimental setup?
Revised: (c) Did you report all the details of your experimental setup?

Original: The details of our experimen- tal setup can be found in Section 3.
Revised: The details of our experimental setup can be found in Section 4.