Original: T RANSFER NAS WITH M ETA - LEARNED B AYESIAN S URROGATES
Revised: T RANSFER NAS WITH M ETA - LEARNED B AYESIAN S URROGATES

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: While neural architecture search (NAS) is an intensely-researched area, approaches typically still suffer from either (i) high computational costs or (ii) lack of robustness across datasets and experiments.
Revised: While neural architecture search (NAS) is an intensely-researched area, approaches typically still suffer from either (i) high computational costs or (ii) lack of robustness across datasets and experiments.

Original: Furthermore, most methods start searching for an optimal architecture from scratch , ignoring prior knowledge.
Revised: Furthermore, most methods start searching for an optimal architecture from scratch , ignoring prior knowledge.

Original: This is in contrast to the manual design process by researchers and engineers that leverage previous deep learning experiences by, e.g., transferring architectures from previously solved, related problems.
Revised: This is in contrast to the manual design process by researchers and engineers that leverage previous deep learning experiences by, e.g., transferring architectures from previously solved, related problems.

Original: We propose to adopt this human design strategy and introduce a novel surrogate for NAS, that is meta-learned across prior architecture evaluations across different datasets.
Revised: We propose to adopt this human design strategy and introduce a novel surrogate for NAS, that is meta-learned across prior architecture evaluations across different datasets.

Original: We utilize Bayesian Optimization (BO) with deep-kernel Gaussian Processes, graph neural networks for the architecture embeddings and a transformer-based set encoder of datasets.
Revised: We utilize Bayesian Optimization (BO) with deep-kernel Gaussian Processes, graph neural networks for obtaining architecture embeddings and a transformer-based dataset encoder.

Original: As a result, our method consistently achieves state-of-the-art results on six computer vision datasets, while being as fast as one-shot NAS methods.
Revised: As a result, our method consistently achieves state-of-the-art results on six computer vision datasets, while being as fast as one-shot NAS methods.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: While deep learning has removed the need for manual feature engineering, it has shifted this manual work to the meta-level, introducing the need for manual architecture engineering.
Revised: While deep learning has removed the need for manual feature engineering, it has shifted this manual work to the meta-level, introducing the need for manual architecture engineering.

Original: The natural next step is to also remove the need to manually define the architecture.
Revised: The natural next step is to also remove the need to manually define the architecture.

Original: This is the problem tackled by the field of neural architecture search (NAS).
Revised: This is the problem tackled by the field of neural architecture search (NAS).

Original: However, there is  no NAS methodyet that is both generally robust and efficient.
Revised: Even though NAS is an intensely-researched area, there is still no NAS method that is both generally robust and efficient.

Original: Blackbox optimization methods, such as reinforcement learning (Zoph & Le, 2017), evolutionary algorithms (Real et al., 2019), and Bayesian optimization (Ru et al., 2021;
Revised: Blackbox optimization methods, such as reinforcement learning (Zoph & Le, 2017), evolutionary algorithms (Real et al., 2019), and Bayesian optimization (Ru et al., 2021;

Original: White et al., 2021)  reliablyminimize the validation error but are slow.
Revised: White et al., 2021) work reliably but are slow.

Original: On the other hand, one-shot methods (Liu et al., 2019; Dong & Yang, 2019b) often have problems with robustness (Zela et al., 2020), and the newest trend of zero-cost proxies often does not provide more information about an architecture’s performance than simple statistics, such as the architecture’s number of parameters (White et al., 2022).
Revised: On the other hand, one-shot methods (Liu et al., 2019; Dong & Yang, 2019b) often have problems with robustness (Zela et al., 2020), and the newest trend of zero-cost proxies often does not provide more information about an architecture’s performance than simple statistics, such as the architecture’s number of parameters (White et al., 2022).

Original: An understudied path towards efficiency in NAS is to transfer information across datasets.
Revised: An understudied path towards efficiency in NAS is to transfer information across datasets.

Revised: This idea is naturally motivated by how researchers and engineers tackle new deep learning problems: they leverage the knowledge they obtained from previous experimentation and, e.g., re-use architectures designed for one task and apply or adapt them to a novel task.

Original: While a few NAS approaches in this direction exist (Wong et al., 2018; Lian et al., 2020; Elsken et al., 2020; Wistuba, 2021; Lee et al., 2021; Ru et al., 2021), an effective approach to obtain efficient NAS methods that has been overlooked in the literature  is to exploit the common formulation of NAS as a hyperparameter optimization (HPO) problem (Bergstra et al., 2013;Domhan et al., 2015;
Revised: While a few NAS approaches in this direction exist (Wong et al., 2018; Lian et al., 2020; Elsken et al., 2020; Wistuba, 2021; Lee et al., 2021; Ru et al., 2021), they typically come with one or more of the following limitations: (i) they are only applicable to settings with little data, (ii) they only explore a fairly limited search space or even can just choose from a handful of pre-selected architecture, or (iii) they can not adapt to data seen at test-time. One approach to obtain efficient NAS methods that has been overlooked in the literature so far is to exploit the common formulation of NAS as a hyperparameter optimization (HPO) problem (Bergstra et al., 2013;

Original: Real et al., 2017; Awad et al., 2021) and draw on the extensive literature on transfer HPO (Wistuba et al., 2016; Feurer et al., 2018a; Perrone & Shen, 2019; Salinas et al., 2020; Wistuba & Grabocka, 2021).
Revised: Domhan et al., 2015; Awad et al., 2021) and draw on the extensive literature on transfer HPO (Wistuba et al., 2016; Feurer et al., 2018a; Perrone & Shen, 2019; Salinas et al., 2020; Wistuba & Grabocka, 2021).

Original: This approach is naturally motivated by how researchers and engineers tackle new deep learning problems: they leverage the knowledge they obtained from previous experimentation and, e.g., re-use architectures designed for one task and apply or adapt them to  a novel task.
Revised: In contrast to standard transfer HPO methods that meta-learn parametric surrogates from a pool of source datasets (Wistuba et al., 2016; Feurer et al., 2018a; Wistuba & Grabocka, 2021), in this work we explore the direction of meta learning surrogates by contextualizing them on the dataset characteristics (a.k.a. meta-features) (Vanschoren, 2018; Jomaa et al., 2021a; Rivolli et al., 2022).

Original: In this work, we present an efficient Bayesian Optimization (BO) method with a novel deep-kernel surrogate that yields a new NAS method which combines the best of both worlds: the reliability of blackbox optimization at a computational cost in the same order of magnitude as one-shot approaches.
Revised: In this work, we present an efficient Bayesian Optimization (BO) method with a novel deep-kernel surrogate that yields a new NAS method which combines the best of both worlds: the reliability of blackbox optimization at a computational cost in the same order of magnitude as one-shot approaches.

Original: Concretely, we propose a BO method for NAS that leverages dataset-contextualized surrogates for transfer learning.
Revised: Concretely, we propose a BO method for NAS that leverages dataset-contextualized surrogates for transfer learning.

Original: Following Lee et al.
Revised: Following Lee et al.

Original: (2021), we use a graph encoder (Zhang et al., 2019) to encode neural architectures and an attention-based dataset encoder  to obtain context features.
Revised: (2021), we use a graph encoder (Zhang et al., 2019) to encode neural architectures and an attention-based dataset encoder (Lee et al., 2019) to obtain context features.

Original: We then use deep kernel learning (Wilson et al., 2016) to obtain meta-learned kernels for the joint space of architectures and datasets , allowing us to use the full power of BO for efficient NAS.
Revised: We then use deep kernel learning (Wilson et al., 2016) to obtain meta-learned kernels for the joint space of architectures and datasets , allowing us to use the full power of BO for efficient NAS.

Original: In contrast to standard transfer HPO methods that meta-learn parametric surrogates from a pool of source datasets (Wistuba et al., 2016; Feurer et al., 2018a; Wistuba & Grabocka, 2021), in this work we explore the direction of meta learning surrogates by contextualizing them on the dataset characteristics (a.k.a. meta-features)
Revised: This approach solves two key issues of Lee et al.

Original: (Vanschoren, 2018; Jomaa et al., 2021a; Rivolli et al., 2022). Exploiting these dataset characteristics is particularly important in NAS to find a strong architecture quickly. Instead of using manually chosen ad-hoc meta-features, we represent meta-features as the latent embeddings of  a set-transformer network for image datasets (Lee et al., 2019), that is jointly meta-learned as part of our deep surrogate in an end-to-end manner.
Revised: (2021), which is closest to our work: (i) a lack of trading-off exploration vs. exploitation and (ii) the lack of exploiting new function evaluations on a test task, blindly following what has been observed during meta training.

Original: As a result, our surrogates are optimized for efficiently transferring architectures for a new target dataset based on its meta-features.
Revised: As a result, our surrogates are optimized for efficiently transferring architectures for a new target dataset based on its meta-features.

Original: To sum up, our contributions are as follows:
Revised: To sum up, our contributions are as follows:

Original: • Inspired by manual architecture design, we treat NAS as a transfer or few-shot learning problem.
Revised: • Inspired by manual architecture design, we treat NAS as a transfer or few-shot learning problem.

Original: We leverage ideas from transfer HPO to meta-learn a kernel for Bayesian Optimization, which encodes both architecture and dataset information.
Revised: We leverage ideas from transfer HPO to meta-learn a kernel for Bayesian Optimization, which encodes both architecture and dataset information.

Original: • We are the first to combine deep-kernel Gaussian Processes (GPs) with a graph neural network encoder, a transformer-based dataset encoder, the first to apply BO with deep GPs to NAS, and the first to do all of this in a transfer NAS setting.
Revised: • We are the first to combine deep-kernel Gaussian Processes (GPs) with a graph neural network encoder, a transformer-based dataset encoder, the first to apply BO with deep GPs to NAS, and the first to do all of this in a transfer NAS setting.

Original: • Our resulting method outperforms both state-of-the-art blackbox NAS methods as well as state-of-the-art one-shot methods across six computer vision benchmarks.
Revised: • Our resulting method outperforms both state-of-the-art blackbox NAS methods as well as state-of-the-art one-shot methods across six computer vision benchmarks.

Original: To foster reproducibility, we make our code available at https anonymous .4open.
Revised: To foster reproducibility, we make our code available at https://anonymous.4open.

Original: science/r/TNAS-DCS-CC08 .
Revised: science/r/TNAS-DCS-CC08 .

Original: We address the points in the “NAS Best Practices Checklist” in Appendix D.
Revised: We address the points in the “NAS Best Practices Checklist” in Appendix F.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: NAS is an intensely-researched field, with over 1000 papers published in the last two years alone 1 .
Revised: NAS is an intensely-researched field, with over 1000 papers published in the last two years alone 1 .

Original: We therefore limit our discussion of NAS to the most related fields of Bayesian optimization for
Revised: We therefore limit our discussion of NAS to the most related fields of Bayesian optimization for

Original: NAS and meta learning approaches for NAS.
Revised: NAS and meta learning approaches for NAS.

Original: For a full discussion of the NAS literature, we refer the interested readers to a series ofrecent surveys by Elsken et al.
Revised: For a full discussion of the NAS literature, we refer the interested readers to a series of surveys by Elsken et al.

Original: , Wistuba et al.
Revised: , Wistuba et al.

Original: (2019), Xie et al. (2020), Ren et al. (2020), Benmeziane et al. (2021) and Dong et al.
Revised: and Ren et al.

Original: (2021), and for an introduction to BO to Shahriari et al.
Revised: (2020), and for an introduction to BO to Shahriari et al.

Original: ; Hutter et al.
Revised: ; Hutter et al.

Original: Bayesian optimization (BO) for NAS .
Revised: Bayesian optimization (BO) for NAS .

Original: As BO is commonly used in hyperparameter optimization (HPO), one can simply treat architectural choices as categorical hyperparameters and re-use, e.g., tree-based HPO methods that can natively handle categorical choices well (Bergstra et al., 2013;
Revised: As BO is commonly used in hyperparameter optimization (HPO), one can simply treat architectural choices as categorical hyperparameters and re-use, e.g., tree-based HPO methods that can natively handle categorical choices well (Bergstra et al., 2013;

Original: Domhan et al., 2015; Falkner et al., 2018).
Revised: Domhan et al., 2015; Falkner et al., 2018).

Original: While Gaussian Processes (GPs) are more typically applied to continuous hyperparameters, they can also be used for NAS by creating an appropriate kernel; such kernels for GP-based BO were manually engineered by Swersky et al. (2013), Kandasamy et al. (2018) and Ru et al.
Revised: While Gaussian Processes (GPs) are more typically applied to continuous hyperparameters, they can also be used for NAS by creating an appropriate kernel; such kernels for GP-based BO can be manually engineered (Swersky et al., 2013; Kandasamy et al., 2018; Ru et al., 2021).

Original: A recent alternative is to exploit (Bayesian) neural networks for BO (Snoek et al., 2015; Springenberg et al., 2016; White et al., 2021).
Revised: A recent alternative is to exploit (Bayesian) neural networks for BO (Snoek et al., 2015; Springenberg et al., 2016; White et al., 2021).

Original: However, while these neural networks are very expressive and parallelize well, they require more data to fit well than GPs and thus are outperformed by GP-based approaches when only a few function evaluations can be afforded.
Revised: However, while these neural networks are very expressive, they require more data to fit well than GPs and thus are outperformed by GP-based approaches when only a few function evaluations can be afforded.

Original: In this work, we combine the sample efficiency of GPs and the expressive power of neural networks, by using deep GPs combined with a graph neural network encoder.
Revised: In this work, we combine the sample efficiency of GPs and the expressive power of neural networks, by using deep GPs combined with a graph neural network encoder.

Original: Meta learning for NAS.
Revised: Meta learning for NAS.

Original: To mitigate the computational infeasibility of starting NAS methods from scratch for each new task, several approaches have been proposed along the lines of meta and transfer learning (Wong et al., 2018; Lian et al., 2020; Elsken et al., 2020; Wistuba, 2021; Lee et al., 2021; Ru et al., 2021).
Revised: To mitigate the computational infeasibility of starting NAS methods from scratch for each new task, several approaches have been proposed along the lines of meta and transfer learning.

Original: Most of these warm-start the weights of architectures in a target task Wong et al.
Revised: Most of these warm-start the weights of architectures in a target task Wong et al.

Original: (2018); Lian et al.
Revised: Lian et al.

Original: Elsken et al.
Revised: Elsken et al.

Original: (2020); Wistuba (2021).
Revised: (2020); Wistuba (2021).

Original: Ru et al.
Revised: Ru et al.

Original: (2021) extracts architectural motifs that can be reused on other datasets.
Revised: (2021) extracts architectural motifs that can be reused on other datasets.

Original: Most related to our work is MetaD2A (Lee et al., 2021), where the authors propose to generate candidate architectures and rank them conditioned directly on a task, utilizing a meta-feature extractor (Lee et al., 2019).
Revised: Most related to our work is MetaD2A (Lee et al., 2021), where the authors propose to generate candidate architectures and rank them conditioned directly on a task, utilizing a meta-feature extractor (Lee et al., 2019).

Original: However, there are two key differences in our work: (i) the performance predictor of MetaD2A is not probabilistic and thus can not naturally trade-off exploration vs. exploitation but rather only exploits what has been observed during meta-training. 
Revised: However, there are two key differences in our work: (i) the performance predictor of MetaD2A is not probabilistic and thus can not naturally tradeoff exploration vs. exploitation but rather only exploits what has been observed during meta-training. [Figure]

Original: (ii)
Revised: (ii)

Original: During the meta testing phase, MetaD2A simply proposes N architectures[Figure] for a new task, trains the top- K (as estimated by the performance predictor) out of N architectures and returns the best.
Revised: During the meta testing phase, MetaD2A simply proposes N architectures for a new task, trains the top- K (as estimated by the performance predictor) out of N architectures and returns the best.

Original: In particular, the performance of the K evaluated architectures is not used as feedback for MetaD2A and thus the method never adapts to function evaluations on the test task, blindly following what has been observed during meta training.
Revised: In particular, the performance of the K evaluated architectures is not used as feedback for MetaD2A and thus the method never adapts to function evaluations on the test task, blindly following what has been observed during meta training.

Original: This can cause problems for new tasks which are poorly correlated with the meta training data, and indeed, MetaD2A stagnates on several of the datasets in our experimental analysis.
Revised: This can cause problems for new tasks which are poorly correlated with the meta training data. And, indeed, MetaD2A stagnates on several of the datasets in our experimental analysis.

Original: In contrast,  TNAS  uses the function evaluations frommeta testing to update the surrogate employed within our BO framework, thus allowing to adapt to the testing scenario.
Revised: In contrast, our approach, dubbed TNAS (transferrable NAS), uses the function evaluations from testing to update the surrogate employed within our BO framework, thus allowing to adapt to the testing scenario.

Original: Meta & transfer learning for HPO There are many approaches to achieve meta or transfer learning in HPO, see, e.g., the survey by Vanschoren (2019) orthe nine different classes of methods discussed by Feurer et al.
Revised: Meta & transfer learning for HPO There are many approaches to achieve meta or transfer learning in HPO, see, e.g., the survey by Vanschoren (2019) or Feurer et al.

Original: (2018b, Section 7).
Revised: (2018b, Section 7).

Original: One particularly promising approach is to employ Deep Kernel

Original: Learning.

Original: While existing kernels for BO for NAS are manually defined (Swersky et al., 2013;

Original: Kandasamy et al., 2018; Ru et al., 2021),

Original:  Deep Kernel Learning (Wilson et al., 2016) is an approach that strives to learn the kernel function by using a neural network to transform the input to a latent representation, which is then used in a kernel function.
Revised: One particularly promising approach is to employ Deep Kernel Learning (Wilson et al., 2016) which strives to learn the kernel function by using a neural network to transform the input to a latent representation, which is then used in a kernel function.

Original: Wistuba & Grabocka (2021) and Jomaa et al.
Revised: Wistuba & Grabocka (2021) and Jomaa et al.

Original: (2021b) utilized a deep kernel for transfer learning in HPO.
Revised: (2021b) utilized a deep kernel for transfer learning in HPO.

Original: While numerical hyperparameters can be encoded using an MLP, we adapt thisDeep Kernel Learning approach to NAS by using a graph neural network to encode architectures as inputs; we also extend it by encoding datasets into a latent embedding and learning a deep kernel that spans the combined space of architectures and datasets.
Revised: While numerical hyperparameters can be encoded using an MLP, we adapt this approach to NAS by using a graph neural network to encode architectures as inputs; we also extend it by encoding datasets into a latent embedding and learning a deep kernel that spans the combined space of architectures and datasets.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: We consider the following problem: given a history of Q datasets, where for each dataset D ( q ) we have already evaluated a set of neural network architectures x ( q ) 1 , . . .
Revised: We consider the following problem: given a history of Q datasets, where for each dataset D ( q ) we have already evaluated a set of neural network architectures x ( q ) 1 , . . .

Original: , x ( q ) n with corresponding performance (e.g., accuracy) y
Revised: , x ( q ) n with corresponding performance (e.g., accuracy) y

Original: ( q ) 1 , . .
Revised: ( q ) 1 , . .

Original: , y ( q ) n .
Revised: , y ( q ) n .

Original: For a new dataset D ( new ) , we want to quickly discover an optimal architecture by leveraging information from the history of datasets.
Revised: For a new dataset D ( new ) , we want to quickly discover an optimal architecture by leveraging information from the history of datasets.

Original: We build upon the state-of-the-art few-shot Bayesian optimization (BO) framework by Wistuba & Grabocka (2021), which was proposed to address a similar problem: transferring optimal hyperparameter configurations across datasets.
Revised: We build upon the state-of-the-art few-shot Bayesian optimization (BO) framework by Wistuba & Grabocka (2021), which was proposed to address a similar problem: transferring optimal hyperparameter configurations across datasets.

Original: The authors propose to learn a deep kernel across tasks, which is then used for a
Revised: The authors propose to learn a deep kernel across tasks, which is then used for a

Original: Gaussian process (GP) surrogate in the typical BO setup.
Revised: Gaussian process (GP) surrogate in the typical BO setup.

Original: However, while hyperparameter configurations can typically be presented by an N-dimensional vector, it is less clear how to represent neural network architectures.
Revised: However, while hyperparameter configurations can typically be presented by an N-dimensional vector, it is less clear how to represent neural network architectures.

Original: Simply representing architectures as vectors and plugging them into an off-the-shelf GP kernel is likely sub-optimal.
Revised: Simply representing architectures as vectors and plugging them into an off-the-shelf GP kernel is likely sub-optimal.

Original: In fact, White et al.
Revised: In fact, White et al.

Original: (2020) have shown that the type of architecture representation substantially impacts the performance of a downstream NAS algorithm.
Revised: (2020) have shown that the type of architecture representation substantially impacts the performance of a downstream NAS algorithm.

Original: To address this issue, we employ graph neural networks (GNNs) to obtain a learnable representation of neural networks.
Revised: To address this issue, we employ graph neural networks (GNNs) to obtain a learnable representation of neural networks.

Original: GNNs are a common choice in NAS as neural networks architectures can be naturally represented as graphs (Siems et al., 2020; White et al., 2020;
Revised: GNNs are a common choice in NAS as neural networks architectures can be naturally represented as graphs (Siems et al., 2020; White et al., 2020;

Original: Wen et al., 2020; Dudziak et al., 2020).
Revised: Wen et al., 2020; Dudziak et al., 2020).

Original: Furthermore, while we could directly feed this architecture encoding into the GP’s kernel, we argue that the kernel should also be conditioned on the characteristics of a dataset in order to meaningfully asses (dis-)similarities between architectures.
Revised: Furthermore, while we could directly feed this architecture encoding into the GP’s kernel, we argue that the kernel should also be conditioned on the characteristics of a dataset in order to meaningfully asses (dis-)similarities between architectures.

Original: To motivate this, consider the question “What makes two architectures similar?”.
Revised: To motivate this, consider the question “What makes two architectures similar?”.

Original: We argue that two architectures are not similar only because they share some similar sub-graph components (which will be represented by the GNN encoding), but also because they achieve similar performance on the target dataset.
Revised: We argue that two architectures are not similar only because they share some similar sub-graph components (which will be represented by the GNN encoding), but also because they achieve similar performance on the target dataset.

Original: Following this line of reasoning, we condition the deep kernel on the characteristics (meta-features) of a dataset (Vanschoren, 2018).
Revised: Following this line of reasoning, we condition the deep kernel on the characteristics (meta-features) of a dataset (Vanschoren, 2018).

Original: In a similar fashion as for the architecture encoding, we again use a learnable representation of datasets via employing a set transformer (Lee et al., 2019).
Revised: In a similar fashion as for the architecture encoding, we again use a learnable representation of datasets via employing a set transformer, as also done by (Lee et al., 2019).

Original: The architecture and dataset encoding are then processed by a fully-connected neural network, whose output serves as the input for an off-the-shelf kernel function, e.g., a Matérn kernel, which is finally used to compute the distance of two (architecture, dataset) datapoints.
Revised: The architecture and dataset encoding are then processed by a fully-connected neural network, whose output serves as the input for an off-the-shelf kernel function, e.g., a Matérn kernel, which is finally used to compute the distance of two (architecture, dataset) datapoints.

Original: This results in an end-to-end learnable encoding of the problem, and the parameters of the GNN, transformer and fully-connected neural network are meta-learned in a similar fashion as in Wistuba & Grabocka (2021).
Revised: This results in an end-to-end learnable encoding of the problem, and the parameters of the GNN, transformer and fully-connected neural network are meta-learned in a similar fashion as in Wistuba & Grabocka (2021).

Original: We refer to Figure 1 for an overview of our framework.
Revised: We refer to Figure 1 for an overview of our framework.

Original: In the following subsection, we discuss the details of the different components.
Revised: In the following subsection, we discuss the details of the different components.

Original: 3. B AYESIAN OPTIMIZATION WITH DEEP KERNEL G AUSSIAN PROCESSES
Revised: 3. B AYESIAN OPTIMIZATION WITH DEEP KERNEL G AUSSIAN PROCESSES

Original: We start by introducing Gaussian Processes, which represents the surrogate of our method within
Revised: We start by introducing Gaussian Processes, which represents the surrogate of our method within

Original: Bayesian optimization (BO).
Revised: Bayesian optimization (BO).

Original: In a typical hyperparameter optimization (HPO) setup, the inputs x ∈ X represent hyperparameter configurations, and the target y ∈ Y denotes the performance of a machine learning method when trained with the hyperparameter configuration x .
Revised: In a typical hyperparameter optimization (HPO) setup, the inputs x ∈ X represent hyperparameter configurations, and the target y ∈ Y denotes the performance of a machine learning method when trained with the hyperparameter configuration x .

Original: Consider the training
Revised: Consider the training

Original: D = { ( x i , y i ) } ni =1 and testing D ∗ = { ( x ∗ i , y ∗ i ) } n ∗ i =1 splits of a dataset of evaluated hyperparameters.
Revised: D = { ( x i , y i ) } ni =1 and testing D ∗ = { ( x ∗ i , y ∗ i ) } n ∗ i =1 splits of a dataset of evaluated hyperparameters.

Original: In that context, GPs are non-parametric models that assume a prior over functions, and approximate the target y ∈ Y ⊆ R + given the features x ∈ X ⊆ R L .
Revised: In that context, GPs are non-parametric models that assume a prior over functions, and approximate the target y ∈ Y ⊆ R + given the features x ∈ X ⊆ R L .

Original: The estimation of the target variable y ∗ for the test instances x ∗ is also jointly Gaussian as (cid:20) yy ∗
Revised: The estimation of the target variable y ∗ for the test instances x ∗ is also jointly Gaussian as (cid:20) yy ∗

Original: [Equation]
Revised: [Equation]

Original: N
Revised: N

Revised: [Equation]

Revised: [Equation]

Revised: K

Original: [Equation]
Revised: [Equation]

Original: (cid:19)(cid:19) .
Revised: (cid:19)(cid:19) .

Original: Each respective block of the covariance matrix is the result of applying a kernel function k : X × X
Revised: Each respective block of the covariance matrix is the result of applying a kernel function k : X × X

Original: → R + on pairs of instances, e.g., K ( x, x ∗ ) i,j :=
Revised: → R + on pairs of instances, e.g., K ( x, x ∗ ) i,j :=

Original: k ( x i , x ∗ j ) .
Revised: k ( x i , x ∗ j ) .

Original: The estimated posterior mean andthe posterior covariance of GPs (Rasmussen & Williams, 2006) for the target y ∗ of the test instances x ∗ is given as follows:
Revised: The estimated posterior mean and covariance of GPs (Rasmussen & Williams, 2006) for the target y ∗ of the test instances x ∗ is given as follows:

Original: [Equation]
Revised: [Equation]

Original: We refer to, e.g., Murphy (2012) for the derivation.
Revised: We refer to, e.g., Murphy (2012) for the derivation.

Original: GPs are lazy models that rely on the similarity of the test instances to the training instances via kernel functions k , such as the Matérn kernel.
Revised: GPs are lazy models that rely on the similarity of the test instances to the training instances via kernel functions k , such as the Matérn kernel.

Original: Unfortunately, typical kernels used with GPs are designed manually and rely on sub-optimal assumptions (Cowen-Rivers et al., 2020), which deteriorates the GP’s performance.
Revised: Unfortunately, typical kernels used with GPs are designed manually and rely on sub-optimal assumptions (Cowen-Rivers et al., 2020), which deteriorates the GP’s performance.

Original: A promising direction for designing powerful and efficient kernel functions that adapt to a learning task is Deep Kernel
Revised: A promising direction for designing powerful and efficient kernel functions that adapt to a learning task is Deep Kernel

Original: Learning (Wilson et al., 2016), where kernels are represented as trainable neural networks.
Revised: Learning (Wilson et al., 2016), where kernels are represented as trainable neural networks.

Original: A mapping ξ : X → R L projects the features to a latent representation, where similar instances are co-located.
Revised: A mapping ξ : X → R L projects the features to a latent representation, where similar instances are co-located.

Original: The embedding ξ for the deep kernel of our GPs is a fully-connected neural network, that takes as input the encoding of the architecture ψ as well as the dataset encoding ϕ .
Revised: The embedding ξ for the deep kernel of our GPs is a fully-connected neural network, that takes as input the encoding of the architecture ψ as well as the dataset encoding ϕ .

Original: In detail, the L -dimensional architecture encoding ψ is fused with the K -dimensional dataset encoding ϕ and processed through a fully connected neural network ξ : R K + L → R M , where the last layer has M neurons. 
Revised: In detail, the L -dimensional architecture encoding ψ is fused with the K -dimensional dataset encoding ϕ and processed through a fully connected neural network ξ : R K + L → R M , where the last layer has M neurons. We re-use both the GNN-based architecture encoding and the transformer-based dataset encoding from Lee et al. (2021), thus we only briefly describe it below and refer to Lee et al. (2021) for details.

Original: The architecture encoding ψ consists of a directed acyclic graph encoder (Zhang et al., 2019) to obtain the encoding for the architectures.
Revised: The architecture encoding ψ consists of a directed acyclic graph encoder (Zhang et al., 2019) to obtain the encoding for the architectures.

Original: By using one GRU cell to traverse the topological order of the DAG in the direction from the input to the output, and another GRU cell to pass through the DAG in the backward direction, we obtain latent representations of the graph which are then put through a fully-connected neural network layer to obtain the encoding for the architecture.
Revised: By using one GRU cell to traverse the topological order of the DAG in the direction from the input to the output, and another GRU cell to pass through the DAG in the backward direction, we obtain latent representations of the graph which are then put through a fully-connected neural network layer to obtain the encoding for the architecture.

Original: The dataset encoding ϕis based on the encoder proposed by Lee et al. It consists of two stacked Set-Transformer (Lee et al., 2019) architectures.
Revised: The dataset encoding ϕ consists of two stacked Set-Transformer (Lee et al., 2019) architectures.

Original: The first Set-Transformer layer captures the interaction between randomly sampled data points of the same class, whereas the second one captures the interactions between the different classes of the dataset.
Revised: The first Set-Transformer layer captures the interaction between randomly sampled data points of the same class, whereas the second one captures the interactions between the different classes of the dataset.

Original: The resulting output of the second Set-Transformer layer represents the dataset encoding.
Revised: The resulting output of the second Set-Transformer layer represents the dataset encoding.

Original: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder, as well as the architecture of the feed-forward neural network of our method using the multi-fidelity Bayesian optimization method BOHB (Falkner et al., 2018) on the meta-training dataset; please refer to Appendix A for details.
Revised: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder, as well as the architecture of the feed-forward neural network of our method using the multi-fidelity Bayesian optimization method BOHB (Falkner et al., 2018) on the meta-training dataset; please refer to Appendix A for details.

Original: Putting it all together, the kernel/similarity between two architectures, specifically x evaluated on dataset D , and x ′ evaluated on dataset D ′ , is:
Revised: Putting it all together, the kernel/similarity between two architectures, specifically x evaluated on dataset D , and x ′ evaluated on dataset D ′ , is:

Original: [Equation]
Revised: [Equation]

Original: with w ( ξ ) being the parameters of the neural network ξ , w ( ψ ) the parameters of the architecture encoding ψ , w ( ϕ ) the parameters of the dataset encoding ϕ , and w ( k ) additional parameters of the kernel function.
Revised: with w ( ξ ) being the parameters of the neural network ξ , w ( ψ ) the parameters of the architecture encoding ψ , w ( ϕ ) the parameters of the dataset encoding ϕ , and w ( k ) additional parameters of the kernel function.

Original: We denote the cumulative parameters as w := (cid:0) w ( ξ ) , w ( ψ ) , w ( ϕ ) , w ( k ) (cid:1) .
Revised: We denote the cumulative parameters as w := (cid:0) w ( ξ ) , w ( ψ ) , w ( ϕ ) , w ( k ) (cid:1) .

Original: All parameters are jointly meta-learned to maximize the marginal likelihood (Wistuba & Grabocka, 2021), as will be discussed in the next section.
Revised: All parameters are jointly meta-learned to maximize the marginal likelihood (Wistuba & Grabocka, 2021), as will be discussed in the next section.

Original: 3. M ETA - LEARNING DEEP - KERNEL GP S URROGATES
Revised: 3. M ETA - LEARNING DEEP - KERNEL GP S URROGATES

Original: Recall that we assume we are given a set of Q datasets, where on each dataset D q we have N q ∈ N + evaluated architectures.
Revised: Recall that we assume we are given a set of Q datasets, where on each dataset D q we have N q ∈ N + evaluated architectures.

Original: We denote the n -th architecture evaluated on the q -th dataset as x q,n and its validation accuracy as y q,n .
Revised: We denote the n -th architecture evaluated on the q -th dataset as x q,n and its validation accuracy as y q,n .

Original: Overall, the meta-dataset of all the evaluations on all the datasets is defined as M :
Revised: The meta-dataset of all the evaluations on all the datasets is defined as M :

Original: = (cid:83) Qq =

Revised: = (cid:83) Qq =

Original: (cid:83) N q n =1 { ( x q,n , y q,n , D q ) } .
Revised: (cid:83) N q n =1 { ( x q,n , y q,n , D q ) } .

Original: By x := ( x 1 , 1 , . . .
Revised: By x := ( x 1 , 1 , . . .

Original: , x Q,N q ) , y :
Revised: , x Q,N q ) , y :

Original: =
Revised: =

Original: ( y 1 , 1 , . . .
Revised: ( y 1 , 1 , . . .

Original: , y Q,N q ) and
Revised: , y Q,N q ) and

Original: D :=
Revised: D :=

Original: ( D 1 , . . .
Revised: ( D 1 , . . .

Original: , D Q ) we denotethe vectors containing all the architectures, accuracies and datasets, respectively.
Revised: , D Q ) we denote vectors containing all the architectures, accuracies and datasets.

Original: The parameters w of the deep kernel are optimized jointly by maximizing the log marginal likelihood of the GP surrogate on the meta training dataset:
Revised: The parameters w of the deep kernel are optimized jointly by maximizing the log marginal likelihood of the GP surrogate on the meta training dataset:

Original: [Equation]
Revised: [Equation]

Original: In practice, we resort to sampling mini-batches and employ stochastic gradient descent, following established practiceson Deep Kernel Learning for GPs (Wilson et al., 2016;
Revised: In practice, we resort to sampling mini-batches and employ stochastic gradient descent, following established practices (Wilson et al., 2016;

Original: Wistuba & Grabocka, 2021; Patacchiola et al., 2020).
Revised: Wistuba & Grabocka, 2021; Patacchiola et al., 2020).

Original: As all components of our method are differentiable, our approach is end-to-end differentiable.
Revised: As all components of our method are differentiable, our approach is end-to-end differentiable.

Original: For updating the meta-parameters w , we use the meta-learning algorithm
Revised: For updating the meta-parameters w , we use the meta-learning algorithm

Original: REPTILE (Nichol et al., 2018), due to its simplicityand faster convergence time compared to, e.g., MAML (Finn et al., 2017).
Revised: REPTILE (Nichol et al., 2018), due to its simplicity compared to, e.g., MAML (Finn et al., 2017).

Original: Our method’s pseudocode is given in Algorithm 1.
Revised: Our method’s pseudocode is shown in Algorithm 1.

Original: The procedure samples a mini-batch of (architecture, dataset) pairs with corresponding validation accuracy (line 3).
Revised: The procedure samples a mini-batch of (architecture, dataset) pairs with corresponding validation accuracy (line 3).

Original: Wesimply sample datasets and architectures uniformly at random from the metatraining dataset.
Revised: We sample datasets and architectures uniformly at random from the meta-training dataset.

Original: Then we fit the surrogate to estimate the accuracies y of architectures x on a data set D using SGD by minimizing Equation 3 (lines 4-7).
Revised: Then we fit the surrogate to estimate the accuracies y of architectures x on a dataset D using SGD by minimizing Equation 3 (lines 4-7).

Original: Finally, we use REPTILE (Nichol et al., 2018) to update w (line 8).
Revised: Finally, we use REPTILE (Nichol et al., 2018) to update w (line 8).

Original: Once the optimal w are found, we plug in the meta-learned kernel for a GP, with the posterior and use vanilla Bayesian Optimization (BO) to quickly identify the optimal configuration in the new response surface.
Revised: Once the optimal w are found, we plug in the meta-learned kernel for a GP, with the posterior and use vanilla Bayesian Optimization (BO) to quickly identify the optimal configuration in the new response surface.

Original: [Figure]
Revised: Note that by employing BO in the meta-testing phase, TNAS adapts to the test task by updating the posterior of the GP surrogate with function evaluations from the test task (remember equations 1, 2). We offer a more detailed description of the BO loop in Appendix C.

Original:  cc [Figure] a c y
Revised: A cc u r a c y

Original: CIFAR      

Revised: CIFAR

Original: CIFAR      99.
Revised: CIFAR      99.

Original: MNIST
Revised: MNIST

Original: [Figure] 
Revised: [Figure] [Figure]

Revised: hours

Revised: A cc

Original: [Figure]
Revised: [Figure]

Revised: r a c y

Revised: AIRCRAFT

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: hours

Original: PETS
Revised: PETS

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original:  E XPERIMENTAL S ETUP
Revised:  E XPERIMENTAL S ETUP

Original: We follow the experimental setup  and  use the same meta datasets as Lee et al.
Revised: We follow the experimental setup as Lee et al. (2021) for the NAS-Bench-201 (Dong & Yang, 2020) and MobileNetV3 search spaces. On the NAS-Bench-201 search space, we also use the same meta datasets as Lee et al.

Original: It consists of 4230 meta-training datasets derived from ImageNet.
Revised: It consists of 4230 meta-training datasets derived from ImageNet.

Original: For each of these datasets, the accuracy of one (different) architecture from the NAS-Bench-201 (Dong & Yang, 2020) search space is given.
Revised: For each of these datasets, the accuracy of one (different) architecture from the NAS-Benchsearch space is given.

Original: For the evaluation of our method and the baselines, we use six popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN, Aircraft, Oxford IIT Pets, and MNIST.
Revised: For the evaluation of our method and the baselines, we use six popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN, Aircraft, Oxford IIT Pets, and MNIST.

Revised: For

Original: For CIFAR-10 and CIFAR-100, we query the performances of architectures from the NAS-Bench-201 benchmark, whereas for the other four datasets we train the suggested architectures from scratch using the NAS-Bench-201 pipeline (as these are not available in the benchmark).For all experiments, we use the NAS-Bench-201 search space.
Revised: CIFAR-10 and CIFAR-100, we query the performances of architectures from the NAS-Benchbenchmark, whereas for the other four datasets we train the suggested architectures from scratch using the NAS-Bench-201 pipeline (as these are not available in the benchmark).

Original: We ran three trials for each experiment and report the mean and standard deviations.
Revised: We ran three trials for each experiment and report the mean and standard deviations.

Original: 4. B ASELINES
Revised: 4. B ASELINES

Original: Classic HPO.
Revised: Classic HPO.

Original: A simpleand standard baseline is Random Search (RS) (Bergstra & Bengio, 2012).
Revised: A simple baseline is Random Search (RS) (Bergstra & Bengio, 2012).

Original: RS samples architectures uniformly at random from the search space and returns the top preforming one.
Revised: RS samples architectures uniformly at random from the search space and returns the top preforming one.

Original: Another simple yet powerful baseline is Bayesian Optimization with a vanilla GP surrogate (Snoek et al., 2012).
Revised: Another simple yet powerful baseline is Bayesian Optimization with a vanilla GP surrogate (Snoek et al., 2012).

Original: We use the Matérn 5 / 2 kernel and rely on the GPytorch (Gardner et al., 2018) implementation.
Revised: We use the Matérn 5 / 2 kernel and rely on the GPytorch (Gardner et al., 2018) implementation.

Original: We tried both Expected Improvement (EI) and Upper Confidence Bound (UCB) as acquisition functions, with UCB performing better in our experiments.
Revised: We tried both Expected Improvement (EI) and Upper Confidence Bound (UCB) as acquisition functions, with UCB performing better in our experiments.

Original: Additionally, we compare to HEBO (Cowen-Rivers et al., 2020), a black-box HPO method that performs input and output warping to mitigate the effects of heteroscedasticity and non-stationarity on HPO problems.
Revised: We also compare to HEBO (Cowen-Rivers et al., 2020), a black-box HPO method that performs input and output warping to mitigate the effects of heteroscedasticity and non-stationarity on HPO problems.

Original: HEBO won the 2020 NeurIPS blackbox optimization challenge (Turner et al., 2021).
Revised: HEBO won the 2020 NeurIPS blackbox optimization challenge (Turner et al., 2021).

Original: We use the implementation provided by the authors.
Revised: We use the implementation provided by the authors.

Original: HPO for NAS.
Revised: HPO for NAS.

Original: White et al.
Revised: White et al.

Original: (2021) proposed a BO method for NAS that uses an ensemble of fullyconnected neural networks as a surrogate, named BANANAS.
Revised: (2021) proposed a BO method for NAS that uses an ensemble of fullyconnected neural networks as a surrogate, named BANANAS.

Original: Moreover, BANANAS uses a path encoding for the neural architectures, which serves as an input to the ensemble.
Revised: Moreover, BANANAS uses a path encoding for the neural architectures, which serves as an input to the ensemble.

Original: When applied to a new test task, BANANAS starts the neural architecture search from scratch.
Revised: When applied to a new test task, BANANAS starts the neural architecture search from scratch.

Original: NASBOWL (Ru et al., 2021) is a GP-based BO method for NAS and utilizes the Weisfeiler-Lehman kernel (Shervashidze et al., 2011).
Revised: NASBOWL (Ru et al., 2021) is a GP-based BO method for NAS and utilizes the Weisfeiler-Lehman kernel (Shervashidze et al., 2011).

Original: We use the implementations provided by the authors for BANANAS and NASBOWL.
Revised: We use the implementations provided by the authors for BANANAS and NASBOWL.

Original: State-of-the-art in NAS.
Revised: State-of-the-art in NAS.

Original: One-shot methods have recently shown strong empirical performance for
Revised: One-shot methods have recently shown strong empirical performance for

Original: NAS.
Revised: NAS.

Original: We compare to GDAS (Dong & Yang, 2019b), SETN (Dong & Yang, 2019a), PC-DARTS
Revised: We compare to GDAS (Dong & Yang, 2019b), SETN (Dong & Yang, 2019a), PC-DARTS

Original: (Xu
Revised: (Xu

Original: [Figure]
Revised: [Figure]

Original: et al., 2020) and DrNAS (Chen et al., 2021).
Revised: et al., 2020) and DrNAS (Chen et al., 2021).

Original: For these methods, we compare to published results from the literature for six computer vision datasets.
Revised: For these methods, we compare to published results from the literature for six computer vision datasets.

Original: Transfer NAS.
Revised: Transfer NAS.

Original: We compare to the most related transfer NAS method, MetaD2A (Lee et al., 2021), which is a transfer learning NAS method with a dataset-contextualized neural network generator and performance predictor.
Revised: We compare to the most related transfer NAS method, MetaD2A (Lee et al., 2021), which is a transfer learning NAS method with a dataset-contextualized neural network generator and performance predictor.

Original: The neural network generator and performance predictor are meta-trained on the same source datasets that we also use.
Revised: The neural network generator and performance predictor are meta-trained on the same source datasets that we also use.

Original: When applied to a test task, MetaD2A generatescandidate architectures conditioned on the test dataset and then selects the top architectures based on its performance predictor.
Revised: When applied to a test task, MetaD2A generatescandidate architectures conditioned on the test dataset and then selects the top architectures based on its performance predictor.

Original: We use the implementation provided by MetaD2A’s authors.
Revised: We use the implementation provided by MetaD2A’s authors.

Original:  R ESEARCH H YPOTHESES AND E XPERIMENTAL R ESULTS
Revised:  R ESEARCH H YPOTHESES AND E XPERIMENTAL R ESULTS

Original: Our experiments are designed to validate the following research hypotheses for our approach, dubbed
Revised: Our experiments are designed to validate the following research hypotheses for our approach, dubbed

Original: TNAS :
Revised: TNAS :

Original: Hypothesis 1 : TNAS is more efficient than classical HPO methods applied to NAS as well as HPO methods specifically adapted to NAS and outperforms them in terms of anytime-performance, while achieving strong final performance.
Revised: Hypothesis 1 : TNAS is more efficient than classical HPO methods applied to NAS as well as HPO methods specifically adapted to NAS and outperforms them in terms of anytime-performance, while achieving strong final performance.

Original: Hypothesis 2 : TNAS is competitive with one-shot approaches in terms of runtime.
Revised: Hypothesis 2 : TNAS is competitive with one-shot approaches in terms of runtime.

Original: In summary, we validate our claim from the introduction:
Revised: In summary, we validate our claim from the introduction:

Original: Hypothesis 3 : TNAS achieves the consistency of blackbox optimization algorithms (such as classical
Revised: Hypothesis 3 : TNAS achieves the consistency of blackbox optimization algorithms (such as classical

Original: HPO methods) while being as efficient as one-shot methods.
Revised: HPO methods) while being as efficient as one-shot methods.

Original: Results for Hypothesis 1.
Revised: Results for Hypothesis 1.

Original: In Figure 2, we compare the performance of TNAS, with several HPO baselines.
Revised: In Figure 2, we compare the performance of TNAS, with several HPO baselines.

Original: For all methods, we use the 5 top-performing architectures from the meta-training dataset as a starting point.
Revised: For all methods, we use the 5 top-performing architectures from the meta-training dataset as a starting point.

Original: In that sense, all these baselines are “transfer learning” by being initialized with the best architectures on the meta-training dataset.
Revised: In that sense, all these baselines are “transfer learning” by being initialized with the best architectures on the meta-training dataset.

Original: On all of the datasets except CIFAR100, TNAS finds top-performing architectures faster than all the baselines and achieves stronger anytime performance.
Revised: On all of the datasets except CIFAR100, TNAS finds top-performing architectures faster than all the baselines and achieves stronger anytime performance.

Original: Furthermore, on all benchmarks, TNAS eventually performs best.
Revised: Furthermore, on all benchmarks, TNAS eventually performs best.

Original: Results for Hypothesis 2.
Revised: Results for Hypothesis 2.

Original: We demonstrate that with our meta-learned deep kernel within Bayesian
Revised: We demonstrate that with our meta-learned deep kernel within Bayesian

Original: Optimization, the search time can be significantly reduced, to the same order as one-shot approaches.
Revised: Optimization, the search time can be significantly reduced, to the same order as one-shot approaches.

Original: Figure 3 shows the performance of both TNAS and MetaD2A compared to state-of-the-art NAS
Revised: Figure 3 shows the performance of both TNAS and MetaD2A compared to state-of-the-art NAS

Original: [Figure]
Revised: [Figure]

Original: methods.
Revised: methods.

Original: Except for CIFAR10 and CIFAR100, TNAS clearly outperforms the NAS baselines by the time the one-shot approaches finish the search.
Revised: Except for CIFAR10 and CIFAR100, TNAS clearly outperforms the NAS baselines by the time the one-shot approaches finish the search.

Original: On CIFAR10, TNAS achieves similar performance as the baselines, while it is slightly inferior on CIFAR100 for some baselines and only overtakes them given more time.
Revised: On CIFAR10, TNAS achieves similar performance as the baselines, while it is slightly inferior on CIFAR100 for some baselines and only overtakes them given more time.

Original: We refer to Table 2 in the appendix for concrete numbers.
Revised: We refer to Table 3 in the appendix for concrete numbers.

Original: We can furthermore observe the drawbacks of MetaD2A discussed earlier: (i) MetaD2A does not trade-off exploration vs. exploitation but rather only exploits what has been observed during meta-training and (ii) MetaD2A does not use evaluations on the new target dataset as feedback and never adapts.
Revised: We can furthermore observe the drawbacks of MetaD2A discussed earlier: (i) MetaD2A does not trade-off exploration vs. exploitation but rather only exploits what has been observed during meta-training and (ii) MetaD2A does not use evaluations on the new target dataset as feedback and never adapts.

Original: As a result, MetaD2A stagnates on several of the datasets.
Revised: As a result, MetaD2A stagnates on several of the datasets.

Original: Results for Hypothesis 3.
Revised: Results for Hypothesis 3.

Original: In Figure 4, we show that TNAS consistently achieves strong results, while the existing state-of-the-art NAS baselines have much higher variance - their ranking changes across benchmarks.
Revised: In Figure 4, we show that TNAS consistently achieves strong results, while the existing state-of-the-art NAS baselines have much higher variance - their ranking changes across benchmarks.

Original: Furthermore, the figure shows how the ranking evolves over the course of running of the methods (by means of runtime in GPU hours).
Revised: Furthermore, the figure shows how the ranking evolves over the course of running of the methods (by means of runtime in GPU hours).

Original: The analysis indicates that TNAS consistently achieves the best performance, for both small and large computational budgets, in particular also when compared to MetaD2A
Revised: The analysis indicates that TNAS consistently achieves the best performance, for both small and large computational budgets, in particular also when compared to MetaD2A

Original: . 5. A BLATING OUR D ESIGN
Revised: . 5. A BLATING OUR D ESIGN

Original: [Figure]
Revised: [Figure]

Original: We empirically analyse our design choices, namely the graph and dataset encoder and demonstrate a lift in performance compared to ceteris paribus ablations that do not employ these designs.
Revised: We empirically analyse our design choices, namely the graph and dataset encoder and demonstrate a lift in performance compared to ceteris paribus ablations that do not employ these designs.

Original: Concretely, we ablate our method’s components and test them on CIFAR-10 and CIFAR-100, reporting the result in Figure 5.
Revised: Concretely, we ablate our method’s components and test them on CIFAR-10 and CIFAR-100, reporting the result in Figure 5.

Original: TNAS (no Set Encoder) shows the performance of our method using a graph encoder, but no set-encoder (i.e., without using any dataset meta-features).
Revised: TNAS (no Set Encoder) shows the performance of our method using a graph encoder, but no set-encoder (i.e., without using any dataset meta-features).

Original: TNAS (no Graph
Revised: TNAS (no Graph

Original: Encoder) shows the performance of our method using the dataset encoding in combination with a matrix encoding for the architectures (i.e., no graph encoder).
Revised: Encoder) shows the performance of our method using the dataset encoding in combination with a matrix encoding for the architectures (i.e., no graph encoder).

Original: TNAS outperforms the other variations; thus, we conclude that using both the learnable dataset meta-features and a graph neural network encoding is the most robust surrogate design.
Revised: TNAS outperforms the other variations; thus, we conclude that using both the learnable dataset meta-features and a graph neural network encoding is the most robust surrogate design.

Original: This finding validates the design choices of our method.
Revised: This finding validates the design choices of our method.

Original: [Figure]
Revised: [Figure]

Original: We also empirically evaluated (i) whether there is actually a benefit in meta-learning the surrogate, and (ii) our method’s performance with different initial architectures.
Revised: We also empirically evaluated (i) whether there is actually a benefit in meta-learning the surrogate, and (ii) our method’s performance with different initial architectures.

Original: Remember that as an initial design for our method, we used the top-5 performing architectures from the meta-training dataset.
Revised: Remember that as an initial design for our method, we used the top-5 performing architectures from the meta-training dataset.

Original: As alternatives, we consider values other than 5 and also start from randomly sampled architectures.
Revised: As alternatives, we consider values other than 5 and also start from randomly sampled architectures.

Original: We also turn-off meta-learning.
Revised: We also turn-off meta-learning.

Original: The plots for these experiments are shown in Figure 6, evaluated on
Revised: The plots for these experiments are shown in Figure 6, evaluated on

Original: CIFAR-10 and CIFAR-100.
Revised: CIFAR-10 and CIFAR-100.

Original: The results suggest that both design choices are beneficial.
Revised: The results suggest that both design choices are beneficial.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: NAS is an intensely researched task and in essence is an instance of the hyperparameter optimization (HPO) problem.
Revised: NAS is an intensely researched task and in essence is an instance of the hyperparameter optimization (HPO) problem.

Original: In this work, we exploited this relationship and, motivated by state-of-the-art transfer HPO methods, adapted deep Gaussian Process (GP) surrogates to capture architecture representations.
Revised: In this work, we exploited this relationship and, motivated by state-of-the-art transfer HPO methods, adapted deep Gaussian Process (GP) surrogates to capture architecture representations.

Original: Furthermore, we proposed a novel conditioning of the deep GP on dataset meta-features to enable transferring well-performing architectures from source datasets with similar meta-features.
Revised: Furthermore, we proposed a novel conditioning of the deep GP on dataset meta-features to enable transferring well-performing architectures from source datasets with similar meta-features.

Original: We empirically motivated the impact of each component of our proposed method through extensive ablations.
Revised: We empirically motivated the impact of each component of our proposed method through extensive ablations.

Original: In addition, we showed that our novel deep GPs with dataset meta-features and architecture encodings achieve the highest accuracy on six computer vision datasets compared to a broad range of HPO methods, BO methods for NAS, and one-shot NAS methods.
Revised: In addition, we showed that our novel deep GPs with dataset meta-features and architecture encodings achieve the highest accuracy on six computer vision datasets compared to a broad range of HPO methods, BO methods for NAS, and one-shot NAS methods.

Original: Lastly, we demonstrated that proxy architecture evaluations allow our method to discover more accurate architectures within the same search time one-shot NAS methods require.
Revised: Lastly, we demonstrated that proxy architecture evaluations allow our method to discover more accurate architectures within the same search time one-shot NAS methods require.

Original: Future work.
Revised: Future work.

Original: A deep surrogate in principle allows us to capture the interaction between architectures and hyperparameter configurations.
Revised: A deep surrogate in principle allows us to capture the interaction between architectures and hyperparameter configurations.

Original: The NAS literature at the moment underexplores the impact of hyperparameters on the performance of an architecture.
Revised: The NAS literature at the moment underexplores the impact of hyperparameters on the performance of an architecture.

Original: In fact, it is commonly known that the same architecture would perform differently if the training pipeline is altered, for example by changing the learning rate, the number of epochs, or the degree of regularization.
Revised: In fact, it is commonly known that the same architecture would perform differently if the training pipeline is altered, for example by changing the learning rate, the number of epochs, or the degree of regularization.

Original: Our previously-defined surrogate can be trivially extended to model the interaction of architecture embeddings, the dataset meta-features and hyperparameter configurations.
Revised: Our previously-defined surrogate can be trivially extended to model the interaction of architecture embeddings, the dataset meta-features and hyperparameter configurations.

Original: We have not explored this direction empirically due to the lack of available NAS meta-datasets that vary both architectures and hyperparameters but would like to do in the future.
Revised: We have not explored this direction empirically due to the lack of available NAS meta-datasets that vary both architectures and hyperparameters but would like to do in the future.

Original: Reproducibility Statement.
Revised: Reproducibility Statement.

Original: To foster reproducibility, we make our code available at https: //anonymous.4open.science/r/TNAS-DCS-CC08 .
Revised: To foster reproducibility, we make our code available at https: //anonymous.4open.science/r/TNAS-DCS-CC08 .

Original: We give details on our experimental protocol in the “NAS Best Practices Checklist” in Appendix D.
Revised: We give details on our experimental protocol in the “NAS Best Practices Checklist” in Appendix F.

Original: A H YPERPARAMETER OPTIMIZATION FOR GNN AND SET TRANSFORMER ARCHITECTURES
Revised: A H YPERPARAMETER OPTIMIZATION FOR GNN AND SET TRANSFORMER ARCHITECTURES

Original: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder ( Embedding dims. ), the architecture of the feed-forward neural network of our method ( Num. layers , Num. units in layer 1 , Num. units in layer 2 , Num. units in layer 3 , Num. units in layer 4 ), and the learning rate of the joint meta-training using BOHB (Falkner et al., 2018) on the meta-training dataset. 
Revised: We tuned the dimensionality of the embedding of the dataset encoder and graph encoder ( Embedding dims. ), the architecture of the feed-forward neural network of our method ( Num. layers , Num. units in layer 1 , Num. units in layer 2 , Num. units in layer 3 , Num. units in layer 4 ), and the learning rate of the joint meta-training using BOHB (Falkner et al., 2018) on the meta-training dataset. [Table] dims.

Original: [Table]
Revised: [Table]

Original: B S OCIETAL I MPLICATIONS
Revised: B A RCHITECTURE ENCODING IN TNAS

Original: Better optimization of neural network architectures allows the enhancement of the performance of many crucial applications of deep learning, such as climate science, medicine
Revised: To encode the architectures, we use a directed acyclic graph encoder Zhang et al.

Original: [Table]
Revised: It takes architectures represented as a graph as input, and provides as output vector representations for those architectures. Following Zhang et al. (2019), the graph encoder consists of two GRU cells and a fully-connected neural network layer.

Original:  and recommender systems.
Revised: Each GRU cell traverses the graph representation of the architecture in opposite directions to the other. The output of both cells is then concatenated and input through the fully-connected layer, which outputs the vector representation that will be used as an encoding for the architecture. We meta-train the graph encoder, as well as the dataset encoder, jointly with the embedding of the deep kernel during meta-training to maximize the log marginal likelihood of the GP surrogate. C B AYESIAN O PTIMIZATION WITH TNAS

Original: Furthermore, through the  use of transfer learning to speed up the optimization of neural architectures, we reduce thetime costs
Revised: Once we meta-train our surrogate on the meta-training set using Algorithm 1, we use it at meta-testing time in a Bayesian optimization (BO) loop. First, we evaluate the top-5 architectures from the meta-training set on the test dataset. Once the GP is fit to these architectures and their respective accuracies, we use EI as the acquisition function to find the next architecture to evaluate. In the

Original: [Table]
Revised: NASBench201 search space, we repeat this BO loop for a total of 100 evaluations for CIFAR10 and CIFAR100, whereas for SVHN, Aircraft, Pets, and MNIST, we do 40 evaluations. In the MobilenetV search space we do 50 evaluations for each of the datasets. For maximizing the acquisition function in order to find the next candidate architecture, we either exhaustively evaluate the entire search space (in the case of small search spaces such as NAS-Bench201), or simply sample K architectures randomly and pick the best (in the case of the MobileNet V space, K = 10 4 ).

Original: and therefore the energy consumption of  our method.
Revised: We refer to Table 2 for a comparison of the key ingredients of Lee et al. (2021) and our approach.

Original: However, there are also potential disadvantages in the neural network components of our method because they are more difficult to interpret compared to simpler, traditional ML methods; we therefore encourage research on interpretable deep learning
Revised: predictor candidate architecture generation test-time adaptation Lee et al. (2021) (dataset, architecture) encoding, GNN-based architecture generator, no MLP (deterministic prediction) select top-K based on predictor

Original: [Table]
Revised: Ours (dataset, architecture) encoding, BO, maximize yes, adapt GP predictor MLP, GP (probabilistic prediction) acquisition function w.r.t. new candidate evaluations

Original: C C OMPARISON OF TNAS TO SOTA NAS METHODS 
Revised: E C OMPARISON OF TNAS TO SOTA NAS METHODS AND M OBILE N ET V SPACE

Original: [Table]
Revised: [Table]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Original: D NAS B EST P RACTICE C HECKLIST
Revised: F NAS B EST P RACTICE C HECKLIST

Original: We now describe how we addressed the individual points of the NAS best practice checklist (Lindauer & Hutter, 2020).
Revised: We now describe how we addressed the individual points of the NAS best practice checklist (Lindauer & Hutter, 2020).

Original: 1. Best Practices for Releasing Code
Revised: 1. Best Practices for Releasing Code

Original: For all experiments you report:
Revised: For all experiments you report:

Original: (a) Did you release code for the training pipeline used to evaluate the final architectures?
Revised: (a) Did you release code for the training pipeline used to evaluate the final architectures?

Original: The code for the training pipeline for the architectures can be found in the repo we provide.
Revised: The code for the training pipeline for the architectures can be found in the repo we provide.

Original: (b) Did you release code for the search space?
Revised: (b) Did you release code for the search space?

Original: We used the NAS-Bench-201 search space in our experiments, the description and code for which is publicly available.
Revised: We used the NAS-Bench-201 search space in our experiments, the description and code for which is publicly available.

Original: (c) Did you release the hyperparameters used for the final evaluation pipeline, as well as random seeds?
Revised: (c) Did you release the hyperparameters used for the final evaluation pipeline, as well as random seeds?

Original: We the NAS-Bench-201 pipeline and hyperparameters as our final evaluation pipeline.
Revised: We the NAS-Bench-201 pipeline and hyperparameters as our final evaluation pipeline.

Original: We release it as well as the random seeds in the repo we provide.
Revised: We release it as well as the random seeds in the repo we provide.

Original: (d) Did you release code for your NAS method?
Revised: (d) Did you release code for your NAS method?

Original: The code for our NAS method can be found in https://anonymous.4open.science/r/TNAS-DCS-CC08 .
Revised: The code for our NAS method can be found in https://anonymous.4open.science/r/TNAS-DCS-CC08 .

Original: (e) Did you release hyperparameters for your NAS method, as well as random seeds?
Revised: (e) Did you release hyperparameters for your NAS method, as well as random seeds?

Original: The hyperparameters for our NAS method, as well as random seeds for the experiments can be found in the repo we provide.
Revised: The hyperparameters for our NAS method, as well as random seeds for the experiments can be found in the repo we provide.

Original: 2. Best practices for comparing NAS methods
Revised: 2. Best practices for comparing NAS methods

Original: (a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code??
Revised: (a) For all NAS methods you compare, did you use exactly the same NAS benchmark, including the same dataset (with the same training-test split), search space and code for training the architectures and hyperparameters for that code??

Original: Yes, for fair comparison we made sure to use the same evaluation pipeline for all NAS methods we compare.
Revised: Yes, for fair comparison we made sure to use the same evaluation pipeline for all NAS methods we compare.

Original: (b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)?
Revised: (b) Did you control for confounding factors (different hardware, versions of DL libraries, different runtimes for the different methods)?

Original: We ran all the methods on the same hardware and the same environment to control for confounding factors.
Revised: We ran all the methods on the same hardware and the same environment to control for confounding factors.

Original: (c) Did you run ablation studies?
Revised: (c) Did you run ablation studies?

Original: We ran extensive ablation studies on the components of our method.
Revised: We ran extensive ablation studies on the components of our method.

Original: (d) Did you use the same evaluation protocol for the methods being compared?
Revised: (d) Did you use the same evaluation protocol for the methods being compared?

Original: Yes.
Revised: Yes.

Original: (e) Did you compare performance over time?
Revised: (e) Did you compare performance over time?

Original: Yes.
Revised: Yes.

Original: (f) Did you compare to random search?
Revised: (f) Did you compare to random search?

Original: Performance comparison to random search as well as other baselines can be found in Figure 2.
Revised: Performance comparison to random search as well as other baselines can be found in Figure 2.

Original: (g)
Revised: (g)

Original: Did you perform multiple runs of your experiments and report seeds?
Revised: Did you perform multiple runs of your experiments and report seeds?

Original: For each of the experiments we performed three runs with different seeds (333 , 444 , 555) .
Revised: For each of the experiments we performed three runs with different seeds (333 , 444 , 555) .

Original: (h)
Revised: (h)

Original: Did you use tabular or surrogate benchmarks for in-depth evaluations?
Revised: Did you use tabular or surrogate benchmarks for in-depth evaluations?

Original: We used NAS-Bench-201 as a tabular benchmark.
Revised: We used NAS-Bench-201 as a tabular benchmark.

Original: 3. Best practices for reporting important details
Revised: 3. Best practices for reporting important details

Original: (a) Did you report how you tuned hyperparameters, and what time and resources this required?
Revised: (a) Did you report how you tuned hyperparameters, and what time and resources this required?

Original: We use BOHBFalkner et al.
Revised: We use BOHBFalkner et al.

Original: (2018) to tune the hyperparameters of our method.
Revised: (2018) to tune the hyperparameters of our method.

Original: We ran BOHB with three different random seeds for 24 hours.
Revised: We ran BOHB with three different random seeds for 24 hours.

Original: (b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)?
Revised: (b) Did you report the time for the entire end-to-end NAS method (rather than, e.g., only for the search phase)?

Original: Yes.
Revised: Yes.

Original: (c) Did you report all the details of your experimental setup?
Revised: (c) Did you report all the details of your experimental setup?

Original: The details of our experimental setup can be found in Section 4.
Revised: The details of our experimental setup can be found in Section 4.