Original: E FFICIENT A UTOMATIC M ACHINE L EARNING VIA D ESIGN G RAPHS
Revised: E FFICIENT A UTOMATIC G RAPH L EARNING VIA D ESIGN R ELATIONS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite the success of automated machine learning (AutoML), which aims to find the best design, including the architecture of deep networks and hyper-parameters, conventional AutoML methods are computationally expensive and hardly provide insights into the relations of different model design choices.
Revised: Despite the success of automated machine learning (AutoML), which aims to find the best design, including the architecture of neural networks and hyper-parameters, conventional AutoML methods are computationally expensive and hardly provide insights into the relations of different model design choices.

Revised: This work focus on the scope of AutoML on graph tasks.

Original: To tackle the challenges, we propose F ALCON , an efficient sample-based method to search for the optimal model design.
Revised: To tackle the challenges, we propose F ALCON , an efficient sample-based method to search for the optimal model design on graph tasks.

Original: Our key insight is to model the design space of possible model designs as a design graph , where the nodes represent design choices, and the edges denote design similarities.
Revised: Our key insight is to model the design space of possible model designs as a design graph , where the nodes represent design choices, and the edges denote design similarities.

Original: F ALCON features 1) a task-agnostic module, which performs message passing on the design graph via a Graph Neural Network (GNN), and 2) a task-specific module, which conducts label propagation of the known model performance information on the design graph.
Revised: F ALCON features 1) a task-agnostic module, which performs message passing on the design graph via a Graph Neural Network (GNN), and 2) a task-specific module, which conducts label propagation of the known model performance information on the design graph.

Original: Both modules are combined to predict the design performances in the design space, navigating the search direction.
Revised: Both modules are combined to predict the design performances in the design space, navigating the search direction.

Original: We conduct extensive experiments on 27 node and graph classification tasks from various application domains, and an image classification task on the CIFAR-10 dataset.
Revised: We conduct extensive experiments on 27 node and graph classification tasks from various application domains.

Original: We empirically show that F ALCON can efficiently obtain the well-performing designs for each task using only 30 explored nodes.
Revised: We empirically show that F ALCON can efficiently obtain the well-performing designs for each task using only 30 explored nodes.

Original: Specifically, F ALCON has a comparable time cost with the one-shot approaches while achieving an average improvement of 3.3% compared with the best baselines.
Revised: Specifically, F ALCON has a comparable time cost with the one-shot approaches while achieving an average improvement of 3.3% compared with the best baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Automated machine learning (AutoML) (Liu et al., 2019; Pham et al., 2018; Bender et al., 2018;
Revised: Automated machine learning (AutoML) (Liu et al., 2019; Pham et al., 2018; Bender et al., 2018;

Original: Real et al., 2019; Zoph & Le, 2017;
Revised: Real et al., 2019; Zoph & Le, 2017;

Original: Cai et al., 2019; 2021; Gao et al., 2019; You et al., 2020b; Zhang et al., 2021) has demonstrated great success in various domains including computer vision (Chu et al., 2020;
Revised: Cai et al., 2019; 2021; Gao et al., 2019; You et al., 2020b; Zhang et al., 2021) has demonstrated great success in various domains including computer vision (Chu et al., 2020;

Original: Ghiasi et al., 2019; Chen et al., 2019), language modeling (Zoph & Le, 2017; So et al., 2019), and recommender systems (Chen et al., 2022).
Revised: Ghiasi et al., 2019; Chen et al., 2019), language modeling (Zoph & Le, 2017; So et al., 2019), and recommender systems (Chen et al., 2022).

Original: It is an essential component for the state-of-the-art deep learning models (Liu et al., 2018; Baker et al., 2017; Xu et al., 2020; Chen et al.).
Revised: It is an essential component for the state-of-the-art deep learning models (Liu et al., 2018; Baker et al., 2017; Xu et al., 2020; Chen et al., 2021).

Original: Given a machine learning task, e.g., a node/graph classification task on graphs or an image classification task, our goal of AutoML is to search for a model architecture and hyper-parameter setting from a design space that results in the best test performance on the task.
Revised: Given a graph learning task, e.g., a node/graph classification task on graphs, our goal of AutoML is to search for a model architecture and hyper-parameter setting from a design space that results in the best test performance on the task.

Original: Following previous works (You et al., 2020b), we define design as a set of architecture and hyper-parameter choices ( e.g., 3 layer, 64 embedding dimensions, batch normalization, skip connection between consecutive layers), and define design space as the space of all possible designs for a given task.
Revised: Following previous works (You et al., 2020b), we define design as a set of architecture and hyper-parameter choices ( e.g., 3 layer, 64 embedding dimensions, batch normalization, skip connection between consecutive layers), and define design space as the space of all possible designs for a given task.

Original: However, AutoML is very computationally intensive.
Revised: However, AutoML is very computationally intensive.

Original: The design space of interest often involves millions of possible designs (Elsken et al.
Revised: The design space of interest often involves millions of possible designs (Elsken et al.

Original: ; You et al., 2020a).
Revised: ; You et al., 2020a).

Original: Sample-based AutoML (Zoph &
Revised: Sample-based AutoML (Zoph &

Original: Le, 2017; Gao et al., 2019; Bergstra et al., 2011; Liu et al., 2017; Luo et al., 2018) has been used to perform search via sampling candidate designs from the design space to explore.
Revised: Le, 2017; Gao et al., 2019; Bergstra et al., 2011; Liu et al., 2017; Luo et al., 2018) has been used to perform search via sampling candidate designs from the design space to explore.

Original: One central challenge of existing sample-based AutoML solutions is its sample efficiency: it needs to train as few models as possible to identify the best-performing model in the vast design space.
Revised: One central challenge of existing sample-based AutoML solutions is its sample efficiency: it needs to train as few models as possible to identify the best-performing model in the vast design space.

Original: To improve the efficiency, existing research focuses on developing good search algorithms to navigate in the design space (White et al., 2021;
Revised: To improve the efficiency, existing research focuses on developing good search algorithms to navigate in the design space (White et al., 2021;

Original: Shi et al., 2020; Ma et al., 2019).
Revised: Shi et al., 2020; Ma et al., 2019).

Original: However, these methods do not consider modeling the effect of model design choices, which provides strong inductive biases in searching for the best-performing model.
Revised: However, these methods do not consider modeling the effect of model design choices, which provides strong inductive biases in searching for the best-performing model.

Original: For example, human often relies on two types of intuitions when designing architectures and hyper-parameters: (1) boundedness:
Revised: By “inductive bias”, we refer to the patterns of multiple variables interacting together, which can happen in multiple parts of the

Original: [Figure]
Revised: [Figure]

Original: Locally, F ALCON extends the design subgraph via asearch strategy detailed in Section
Revised: Locally, F ALCON extends the design subgraph via a strategy detailed in Section

Original: Globally,
Revised: Globally,

Original: F ALCON approaches the optimal design navigated by the inductive bias of the design relations.
Revised: F ALCON approaches the optimal design navigated by the inductive bias of the design relations.

Original: “higher embedding dimensions result in higher performance until reaching a threshold”; (2) dependency: “using a large number of layers without batch normalization degrades the performance” to find the best design.
Revised: design space.

Original: Thus, an efficient search strategy should rapidly rule out a large subset of the design space  leveraging such learned inductive bias.
Revised: Thus, an efficient search strategy should rapidly rule out a large subset of the design space with potentially bad performance leveraging such learned inductive bias.

Original: Proposed approach .
Revised: Proposed approach .

Original: To overcome the limitations, we propose F ALCON , an AutoML framework  that achieves state-of-the-art sample efficiency and performance by leveraging model design insights.
Revised: To overcome the limitations, we propose F ALCON , an AutoML framework on graph tasks that achieves state-of-the-art sample efficiency and performance by leveraging model design insights.

Original: Our key insight is to build a design graph over the design space of architecture and hyper-parameter choices.
Revised: Our key insight is to build a design graph over the design space of architecture and hyper-parameter choices.

Original: F ALCON extracts model design insights by learning a meta-model that captures the relation between the design graph and model performance and uses it to inform a sample-efficient search strategy.
Revised: F ALCON extracts model design insights by learning a meta-model that captures the relation between the design graph and model performance and uses it to inform a sample-efficient search strategy.

Original: F ALCON consists of the following two novel components.
Revised: F ALCON consists of the following two novel components.

Original: Design space as a graph .
Revised: Design space as a graph .

Original: Previous works view the model design space as a high-dimensional space with isolated design choices (You et al., 2020b), which offer few insights regarding the relations between different design choices.
Revised: Previous works view the model design space as a high-dimensional space with isolated design choices (You et al., 2020b), which offer few insights regarding the relations between different design choices.

Original: For example, through trial runs if we find the models with more than 3 layers do not work well without batch normalization, this knowledge can help us reduce the search space by excluding all model designs of more than 3 layers with batch normalization set to false.
Revised: For example, through trial runs if we find the models with more than 3 layers do not work well without batch normalization, this knowledge can help us reduce the search space by excluding all model designs of more than 3 layers with batch normalization set to false.

Original: While such insights are hardly obtained with existing AutoML algorithms (Liu et al., 2019;
Revised: While such insights are hardly obtained with existing AutoML algorithms (Liu et al., 2019;

Original: Pham et al., 2018; Gao et al., 2019; Zoph & Le, 2017;
Revised: Pham et al., 2018; Gao et al., 2019; Zoph & Le, 2017;

Original: Cai et al., 2019), F ALCON achieves it via constructing a graph representation, design graph, among all the design choices.
Revised: Cai et al., 2019), F ALCON achieves it via constructing a graph representation, design graph, among all the design choices.

Original: Figure 1(a) shows a visualization of a design graph, where each node represents a candidate design, and edges denote the similarity between the designs.
Revised: Figure 1(a) shows a visualization of a design graph, where each node represents a candidate design, and edges denote the similarity between the designs.

Original: See Section 3.1 for details on the similarity and graph construction.
Revised: See Section 3.1 for details on the similarity and graph construction.

Original: Search by navigating on the design graph .
Revised: Search by navigating on the design graph .

Original: Given the design graph, F ALCON deploys a Graph Neural
Revised: Given the design graph, F ALCON deploys a Graph Neural

Original: Network predictor, short for meta-GNN , which is supervised by the explored nodes’ performances and learns to predict the performance of a specific design given the corresponding node in the design graph.
Revised: Network predictor, short for meta-GNN , which is supervised by the explored nodes’ performances and learns to predict the performance of a specific design given the corresponding node in the design graph.

Original: The meta-GNN is designed with 1) a task-agnostic module, which performs message passing on the design graph, and 2) a task-specific module, which conducts label propagation of the known model performance information on the design graph.
Revised: The meta-GNN is designed with 1) a task-agnostic module, which performs message passing on the design graph, and 2) a task-specific module, which conducts label propagation of the known model performance information on the design graph.

Original: Furthermore, we propose a search strategy that uses meta-GNN predictions to navigate the search in the design graph efficiently.
Revised: Furthermore, we propose a search strategy that uses meta-GNN predictions to navigate the search in the design graph efficiently.

Original: Experiments .
Revised: Experiments .

Original: Without loss of generality, we mainly focus on AutoML for graph representation learning in this paper.

Original: We conduct extensive experiments on 27 graph datasets, covering node- and graph-level tasks with distinct distributions.
Revised: We conduct extensive experiments on 27 graph datasets, covering node- and graphlevel tasks with distinct distributions.

Original: Moreover,  F ALCON can work well on other datasets such as the CIFAR-10 image dataset.
Revised: Moreover, we demonstrate F ALCON ’ potential applicability on image datasets by conducting experiments on the CIFAR-10 image dataset.

Original: Our code is available at https://anonymous.4open. science/r/Falcon .
Revised: Our code is available at https://anonymous.4open.science/r/Falcon .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Automatic Machine Learning (AutoML) is the cornerstone of discovering state-of-the-art model designs without costing massive human efforts.
Revised: Automatic Machine Learning (AutoML) is the cornerstone of discovering state-of-the-art model designs without costing massive human efforts.

Original: We introduce four types of related works below.
Revised: We introduce four types of related works below.

Original: Sample-based AutoML methods .
Revised: Sample-based AutoML methods .

Original: Existing sample-based approaches explore the search space via sampling candidate designs, which includes heuristic search algorithms, e.g., Simulated Annealing,
Revised: Existing sample-based approaches explore the search space via sampling candidate designs, which includes heuristic search algorithms, e.g., Simulated Annealing,

Original: Bayesian Optimization approaches (Bergstra et al., 2011; White et al., 2021; Ma et al., 2019), evolutionary- (Xie & Yuille, 2017; Real et al., 2017) and reinforcement-based methods (Zoph &Le, 2017;
Revised: Bayesian Optimization approaches (Bergstra et al., 2011; White et al., 2021; Ma et al., 2019), evolutionary- (Xie & Yuille, 2017; Real et al., 2017) and reinforcement-based methods (Zoph &

Original:  Zhou et al., 2019; Gao et al., 2019).
Revised: Le, 2017; Zhou et al., 2019; Gao et al., 2019).

Original: However, they tend to train thousands of models from scratch, which results in the low sample efficiency.
Revised: However, they tend to train thousands of models from scratch, which results in the low sample efficiency.

Original: For example, (Zoph & Le, 2017; Gao et al., 2019) usually involve training hundreds of GPUs for several days, hindering the development of AutoML in real-world applications (Bender et al., 2018).
Revised: For example, (Zoph & Le, 2017; Gao et al., 2019) usually involve training hundreds of GPUs for several days, hindering the development of AutoML in real-world applications (Bender et al., 2018).

Revised: Some hyper-parameter search methods aim to reduce the computational cost.

Revised: For example, Successive Halving (Karnin et al., 2013) allocates the training resources to more potentially valuable models based on the early-stage training information.

Revised: Li et al.

Revised: (2017) further extend it using different budgets to find the best configurations to avoid the trade-off between selecting the configuration number and allocating the budget.

Original:  One-shot AutoML methods .
Revised: Jaderberg et al. (2017) combine parallel search and sequential optimisation methods to conduct fast search. However, their selective mechanisms are only based on the model performance and lack of deep knowledge, which draws less insight into the relation of design variables and limits the sample efficiency. One-shot AutoML methods .

Original: The one-shot approaches (Liu et al., 2019; Pham et al., 2018; Xie et al., 2019; Bender et al., 2018; Qin et al., 2021) have been popular for the high search efficiency.
Revised: The one-shot approaches (Liu et al., 2019; Pham et al., 2018; Xie et al., 2019; Bender et al., 2018; Qin et al., 2021) have been popular for the high search efficiency.

Original: Specifically, they involve training a super-net representing the design space, i.e., containing every candidate design, and shares the weights for the same computational cell.
Revised: Specifically, they involve training a super-net representing the design space, i.e., containing every candidate design, and shares the weights for the same computational cell.

Original: Nevertheless, weight sharing degrades the reliability of design ranking, as it fails to reflect the true performance of the candidate designs (Yu et al., 2020).
Revised: Nevertheless, weight sharing degrades the reliability of design ranking, as it fails to reflect the true performance of the candidate designs (Yu et al., 2020).

Original: Graph-based AutoML methods .
Revised: Graph-based AutoML methods .

Original: The key insight of our work is to construct the design space as a design graph, where nodes are candidate designs and edges denote design similarities, and deploy a Graph Neural Network, i.e., meta-GNN, to predict the design performance.
Revised: The key insight of our work is to construct the design space as a design graph, where nodes are candidate designs and edges denote design similarities, and deploy a Graph Neural Network, i.e., meta-GNN, to predict the design performance.

Original: Graph HyperNetwork (Zhang et al., 2019a) directly generates weights for each node in a computation graph representation.
Revised: Graph HyperNetwork (Zhang et al., 2019a) directly generates weights for each node in a computation graph representation.

Original: You et al.
Revised: You et al.

Original: (2020a) study network generators that output relational graphs and analyze the link between their predictive performance and the graph structure.
Revised: (2020a) study network generators that output relational graphs and analyze the link between their predictive performance and the graph structure.

Original: Recently, Zhao et al.
Revised: Recently, Zhao et al.

Original: (2020) considers both the micro- ( i.e., a single block) and macro-architecture ( i.e., block connections) of each design in graph domain.
Revised: (2020) considers both the micro- ( i.e., a single block) and macro-architecture ( i.e., block connections) of each design in graph domain.

Revised: AutoGML (Park et al., 2022) designs a meta-graph to capture the relations among models and graphs and take a meta-learning approach to estimate the relevance of models to different graphs.

Original: Notably, none of these works model the search space as a design graph.
Revised: Notably, none of these works model the search space as a design graph.

Original: Design performance predictor .
Revised: Design performance predictor .

Original: Previous works predict the performance of a design using the learning curves (Baker et al., 2018), layer-wise features (Deng et al., 2017), computational graph structure (Zhang et al., 2019a;
Revised: Previous works predict the performance of a design using the learning curves (Baker et al., 2018), layer-wise features (Deng et al., 2017), computational graph structure (Zhang et al., 2019a;

Original: White et al., 2021; Shi et al., 2019; Ma et al., 2019; Zhang et al., 2019b;
Revised: White et al., 2021; Shi et al., 2019; Ma et al., 2019; Zhang et al., 2019b;

Original: Lee et al., 2021a), or combining dataset information (Lee et al., 2021a) via a dataset encoder.
Revised: Lee et al., 2021a), or combining dataset information (Lee et al., 2021a) via a dataset encoder.

Original: To highlight, F ALCON explicitly models the relations among model designs.
Revised: To highlight, F ALCON explicitly models the relations among model designs.

Original: Moreover, it leverages the performance information on training instances to provide task-specific information besides the design features, which is differently motivated compared with Lee et al.
Revised: Moreover, it leverages the performance information on training instances to provide task-specific information besides the design features, which is differently motivated compared with Lee et al.

Original: (2021b) that employs meta-learning techniques and incorporate hardware features to rapidly adapt to unseen devices.
Revised: (2021b) that employs meta-learning techniques and incorporate hardware features to rapidly adapt to unseen devices.

Original: Besides, meta-GNN is applicable for both images and graphs, compared with Lee et al.
Revised: Besides, meta-GNN is applicable for both images and graphs, compared with Lee et al.

Original: (2021a).
Revised: (2021a).

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: This section introduces our proposed approach F ALCON for sample-based AutoML.
Revised: This section introduces our proposed approach F ALCON for sample-based AutoML.

Original: In Section 3.1, we introduce the construction of design graph, and formulate the AutoML goal as a search on the design graph for the node with the best task performance.
Revised: In Section 3.1, we introduce the construction of design graph, and formulate the AutoML goal as a search on the design graph for the node with the best task performance.

Original: In Section 3.2, we introduce our novel neural predictor consisting of a task-agnostic module and a task-specific module, which predicts the performances of unknown designs.
Revised: In Section 3.2, we introduce our novel neural predictor consisting of a task-agnostic module and a task-specific module, which predicts the performances of unknown designs.

Original: Finally, we detail our search strategy in Section 3.3.
Revised: Finally, we detail our search strategy in Section 3.3.

Original: We refer the reader to Figure 1 (b) for a high-level overview of F ALCON .
Revised: We refer the reader to Figure 1 (b) for a high-level overview of F ALCON .

Original: 3. D ESIGN S PACE AS A G RAPH
Revised: 3. D ESIGN S PACE AS A G RAPH

Original: Motivation .
Revised: Motivation .

Original: Previous works generally consider each design choice as isolated from other designs.
Revised: Previous works generally consider each design choice as isolated from other designs.

Original: However, it is often observed that some designs that share the same design features, e.g., graph neural networks (GNNs) that are more than 3 layers and have batch normalization layers, may have similar performances.
Revised: However, it is often observed that some designs that share the same design features, e.g., graph neural networks (GNNs) that are more than 3 layers and have batch normalization layers, may have similar performances.

Original: Moreover, the inductive bias of the relations between design choices can provide valuable information for navigating the design space for the best design.
Revised: Moreover, the inductive bias of the relations between design choices can provide valuable information for navigating the design space for the best design.

Original: For example, suppose we find that setting batch normalization of a 3-layer GCN (Kipf & Welling, 2017) and a 4-layer GIN (Xu et al., 2019) to false both degrade the performance.
Revised: For example, suppose we find that setting batch normalization of a 3-layer GCN (Kipf & Welling, 2017) and a 4-layer GIN (Xu et al., 2019) to false both degrade the performance.

Original: Then we can reasonably infer that a 3-layer
Revised: Then we can reasonably infer that a 3-layer

Original: GraphSAGE (Hamilton et al., 2017) with batch normalization outperforms the one without.
Revised: GraphSAGE (Hamilton et al., 2017) with batch normalization outperforms the one without.

Original: We could leverage such knowledge and only search for the designs that are more likely to improve the task performance.
Revised: We could leverage such knowledge and only search for the designs that are more likely to improve the task performance.

Original: To the best of our knowledge, F ALCON is the first method to explicitly consider such relational information among model designs.
Revised: To the best of our knowledge, F ALCON is the first method to explicitly consider such relational information among model designs.

Original: Design graph .
Revised: Design graph .

Original: We denote the design graph as G ( N , E ) , where the nodes N include the candidate designs, and edges E denote the similarities between the candidate designs.
Revised: We denote the design graph as G ( N , E ) , where the nodes N include the candidate designs, and edges E denote the similarities between the candidate designs.

Original: Specifically, we use the notion of design distance to decide the graph connectivity, and we elaborate on them below.
Revised: Specifically, we use the notion of design distance to decide the graph connectivity, and we elaborate on them below.

Original: Design distance .
Revised: Design distance .

Original: For each numerical design dimension, two design choices have a distance 1 if they are adjacent in the ordered list of design choices.
Revised: For each numerical design dimension, two design choices have a distance 1 if they are adjacent in the ordered list of design choices.

Original: For example, if the hidden dimension size can take values [16 , 32 , 64 , 128] , then the distance between 16 and 32 is 1 , and the distance between 32 and 128 is 2 .
Revised: For example, if the hidden dimension size can take values [16 , 32 , 64 , 128] , then the distance between 16 and 32 is 1 , and the distance between 32 and 128 is 2 .

Original: For each categorical design dimension, any two distinct design choices have a distance 1 .
Revised: For each categorical design dimension, any two distinct design choices have a distance 1 .

Original: We then define the connectivity of the design graph in terms of the design distance:
Revised: We then define the connectivity of the design graph in terms of the design distance:

Original: Definition 1 (Design Graph Connectivity)
Revised: Definition 1 (Design Graph Connectivity)

Original: The design graph can be expressed as G ( N , E ) , where the nodes N = { d 1 , . . .
Revised: The design graph can be expressed as G ( N , E ) , where the nodes N = { d 1 , . . .

Original: , d n } are model designs, and ( d i , d
Revised: , d n } are model designs, and ( d i , d

Original: j ) ∈
Revised: j ) ∈

Original: E iff the design distance between d i and d j is 1 .
Revised: E iff the design distance between d i and d j is 1 .

Original: Structure of the design graph .
Revised: Structure of the design graph .

Original: The definition of edges implies that the design graph is highly structured, with the following properties: (1) All designs that are the same except for one categorical design dimension form a clique subgraph.
Revised: The definition of edges implies that the design graph is highly structured, with the following properties: (1) All designs that are the same except for one categorical design dimension form a clique subgraph.

Original: (2) All designs that are the same except k numerical design dimensions form a grid graph structure.
Revised: (2) All designs that are the same except k numerical design dimensions form a grid graph structure.

Original: Moreover, we use a special calculation for the design distance with a combination of design dimensions that have dependencies.
Revised: Moreover, we use a special calculation for the design distance with a combination of design dimensions that have dependencies.

Original: For example, the design dimensions of pooling operations, pooling layers, and the number of layers can depend on each other, thus the design graph structure becomes more complex.
Revised: For example, the design dimensions of pooling operations, pooling layers, and the number of layers can depend on each other, thus the design graph structure becomes more complex.

Original: See the details in Appendix A.2.
Revised: See the details in Appendix A.2.

Original: Design subgraph .
Revised: Design subgraph .

Original: The design graph may contain millions of nodes.
Revised: The design graph may contain millions of nodes.

Original: Therefore, directly applying the meta-model to the design graph is computationally intensive.
Revised: Therefore, directly applying the meta-model to the design graph is computationally intensive.

Original: Moreover, a reliable performance estimation for an unknown node depends on its similarity between the nodes already explored by the search algorithm.
Revised: Moreover, a reliable performance estimation for an unknown node depends on its similarity between the nodes already explored by the search algorithm.

Original: Therefore, we focus on using a meta-model to predict performance for a dynamic subgraph, i.e., design subgraph, containing the explored nodes in the current search stage and the candidate nodes to be sampled in the next step.
Revised: Therefore, we focus on using a meta-model to predict performance for a dynamic subgraph, i.e., design subgraph, containing the explored nodes in the current search stage and the candidate nodes to be sampled in the next step.

Original: The candidate set can be constructed by selecting the multi-hop neighbors of explored nodes on the design graph.
Revised: The candidate set can be constructed by selecting the multi-hop neighbors of explored nodes on the design graph.

Original: The design subgraph is defined as:
Revised: The design subgraph is defined as:

Original: Definition 2 (Design Subgraph)
Revised: Definition 2 (Design Subgraph)

Original: During a search, suppose the explored node set is N e and the candidate set is N c .
Revised: During a search, suppose the explored node set is N e and the candidate set is N c .

Original: The design subgraph is formulated as G s ( N s , E s ) , where N s = N e ∪ N c are the nodes and E s
Revised: The design subgraph is formulated as G s ( N s , E s ) , where N s = N e ∪ N c are the nodes and E s

Original: = { ( u, v ) | u ∈ N s , v ∈ N s , ( u, v ) ∈ N} are the edges.
Revised: = { ( u, v ) | u ∈ N s , v ∈ N s , ( u, v ) ∈ N} are the edges.

Original: Given the design subgraph, we formulate the AutoML problem as searching for the node, i.e., design choice, with the best task performance.
Revised: Given the design subgraph, we formulate the AutoML problem as searching for the node, i.e., design choice, with the best task performance.

Original: 3. M ETA -GNN FOR P ERFORMANCE P REDICTION
Revised: 3. M ETA -GNN FOR P ERFORMANCE P REDICTION

Original: Here we introduce a meta-model, named meta-GNN, to predict the performance of model designs, i.e., nodes of the design subgraph.
Revised: Here we introduce a meta-model, named meta-GNN, to predict the performance of model designs, i.e., nodes of the design subgraph.

Original: The goal of meta-GNN is learning the inductive bias of design relations, which is used to navigate the search path on the design graph.
Revised: The goal of meta-GNN is learning the inductive bias of design relations, which is used to navigate the search path on the design graph.

Original: As is illustrated in Figure 2, the meta-GNN comprises a task-agnostic module and a task-specific module, used to capture the knowledge of model design and task performance, respectively.
Revised: As is illustrated in Figure 2, the meta-GNN comprises a task-agnostic module and a task-specific module, used to capture the knowledge of model design and task performance, respectively.

Original: Task-agnostic module .
Revised: Task-agnostic module .

Original: The task-agnostic module uses a design encoder to encode the design features on nodes of the design subgraph, and a relation encoder to capture the design similarities and differences on edges of the design subgraph.
Revised: The task-agnostic module uses a design encoder to encode the design features on nodes of the design subgraph, and a relation encoder to capture the design similarities and differences on edges of the design subgraph.

Original: After that, it performs message passing on the design subgraph.
Revised: After that, it performs message passing on the design subgraph.

Original: We introduce each component below:
Revised: We introduce each component below:

Original: • Design encoder : it computes the node features of design subgraph by the concatenation of the feature encoding of each design dimension.
Revised: • Design encoder : it computes the node features of design subgraph by the concatenation of the feature encoding of each design dimension.

Original: For numerical design dimensions,we conduct min-max normalization on their values as the node features.
Revised: For numerical design dimensions,we conduct min-max normalization on their values as the node features.

Original: For categorical design dimensions such as aggregation operator which takes one of (S UM , M AX , M EAN ), we encode it as a one-hot feature.
Revised: For categorical design dimensions such as aggregation operator which takes one of (S UM , M AX , M EAN ), we encode it as a one-hot feature.

Original: • Relation encoder : it captures the similarity relationships between the connecting designs.
Revised: • Relation encoder : it captures the similarity relationships between the connecting designs.

Original: For each ( d i , d j ) ∈ E , we encode the design dimension where d i and d j differ by a one-hot encoding.
Revised: For each ( d i , d j ) ∈ E , we encode the design dimension where d i and d j differ by a one-hot encoding.

Original: [Figure]
Revised: [Figure]

Original: Training
Revised: Training

Original: [Figure]
Revised: [Figure]

Original: 𝟏𝒏 𝒃 𝒄
Revised: 𝟏𝒏 𝒃 𝒄

Original: Entropy
Revised: Entropy

Original: [Figure]
Revised: [Figure]

Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: D es i gn s 

Revised: D es i gn s

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]

Original: [Figure]
Revised: [Figure]

Original: 𝒃
Revised: 𝒃

Original: [Figure]
Revised: [Figure]

Original: • Message passing module : a GNN model is used to take the design subgraph and the processed features to perform message passing and output node representations.
Revised: • Message passing module : a GNN model is used to take the design subgraph and the processed features to perform message passing and output node representations.

Original: This information will be combined with the task-specific module to predict the design’s performance.
Revised: This information will be combined with the task-specific module to predict the design’s performance.

Original: Task-specific module .
Revised: Task-specific module .

Original: The task-specific module takes into account the information of design performance on selected training instances and thus is specific for one dataset.
Revised: The task-specific module takes into account the information of design performance on selected training instances and thus is specific for one dataset.

Original: The challenge of including such instance-wise performance is that it is only available on a very limited set of explored nodes.
Revised: The challenge of including such task-specific performance is that it is only available on a very limited set of explored nodes.

Original: To overcome the challenge, we use label propagation to propagate the performance information of explored nodes to the unexplored nodes.
Revised: To overcome the challenge, we use label propagation to propagate the performance information of explored nodes to the unexplored nodes.

Original: This is based on our observation that models trained with similar designs typically make similar predictions on instances.
Revised: This is based on our observation that models trained with similar designs typically make similar predictions on instances.

Original: We provide an example in Figure 2 to illustrate the task-specific module.
Revised: We provide an example in Figure 2 to illustrate the task-specific module.

Original: • Identifying critical instances : The first step is to identify critical training instances that result in different performances across different designs.
Revised: • Identifying critical instances : The first step is to identify critical training instances that result in different performances across different designs.

Original: Here we use a set of explored designs (anchors) to provide the instance-wise performances and compute the entropy of each training instance’s performance over the anchors.
Revised: Here we use a set of explored designs (anchors) to provide the instance-wise performances. Specifically, the ( i, j ) element of the top left matrix of Figure 2 represents whether the i − th design can correctly predict the label of j − th instance. Then, we compute the entropy of each training instance’s performance over the anchors.

Original: Then we obtain the instance-wise probability via Softmax on the entropy vector, from which we sample instances that result in high variation across designs.
Revised: Then we obtain the instance-wise probability via Softmax on the entropy vector, from which we sample instances that result in high variation across designs.

Original: The high variation implies that these instances can distinguish good designs from bad ones in the design subgraph, which are informative.
Revised: The high variation implies that these instances can distinguish good designs from bad ones in the design subgraph, which are informative.

Original: • Label propagation and linear projection : Based on the inductive bias of smoothness, we perform label propagation to make the task-specific information available to all candidate designs.
Revised: • Label propagation and linear projection : Based on the inductive bias of smoothness, we perform label propagation to make the task-specific information available to all candidate designs.

Original: Concretely, label propagation can be written as
Revised: Concretely, label propagation can be written as

Original: [Equation]
Revised: [Equation]

Original: where each row of Y is the performance vector of design i (if explored) or a zero vector (for the unexplored designs).
Revised: where each row of Y is the performance vector of design i (if explored) or a zero vector (for the unexplored designs).

Original: D ∈ R |N s |×|N s | is the diagonal matrix of node degree, A ∈ R
Revised: D ∈ R |N s |×|N s | is the diagonal matrix of node degree, A ∈ R

Original: |N
Revised: |N

Original: s |×|N s | is the adjacent matrix of the design subgraph, and α is a hyper-parameter.
Revised: s |×|N s | is the adjacent matrix of the design subgraph, and α is a hyper-parameter.

Original: After label propagation, we use a linear layer to project the performance information to another high-dimensional space.
Revised: After label propagation, we use a linear layer to project the performance information to another high-dimensional space.

Original: Finally, as shown in Figure 2, we concatenate the output embeddings of the task-specific and taskagnostic modules and use an MLP to produce the performance predictions.
Revised: Finally, as shown in Figure 2, we concatenate the output embeddings of the task-specific and taskagnostic modules and use an MLP to produce the performance predictions.

Original: Objective for Meta-GNN .
Revised: Objective for Meta-GNN .

Original: The training of neural performance predictor is commonly formulated as a regression using mean square error (MSE) in order to predict how good the candidate designs are for the current task.
Revised: The training of neural performance predictor is commonly formulated as a regression using mean square error (MSE), in order to predict how good the candidate designs are for the current task.

Original: However, the number of explored designs is usually small for sample-efficient
Revised: However, the number of explored designs is usually small for sample-efficient

Original: AutoML, especially during the early stage of the search process.
Revised: AutoML, especially during the early stage of the search process.

Original: Thus, the deep predictor tends to overfit, degrading the reliability of performance prediction.
Revised: Thus, the deep predictor tends to overfit, degrading the reliability of performance prediction.

Original: To solve this problem, we incorporate a pair-wise rank loss (Burges et al., 2005;
Revised: To solve this problem, we incorporate a pair-wise rank loss (Burges et al., 2005;

Original: Hu et al., 2021) with the MSE objective, resulting ina quadratic number of training pairs, thus reducing overfitting.
Revised: Hu et al., 2021) with the MSE objective, resulting in

Original: Furthermore, predicting relative Require: S : Design space. K : Exploration size.
Revised: Ω

Original: h θ : Meta-GNN. V/V ′ :
Revised: 2: Γ ← M EIGHBORS

Original: Warm-up / Full epoch. η : Learning rate.
Revised: Construct candidate hop neighbors 3: Y Ω =

Original: C :
Revised: G ET

Original: Number of start nodes.

Original: 1: Ω ← S AMPLE -N
Revised: V ALIDATION

Original: ODES ( S, C ) //
Revised: P ERFORMANCE

Original: Initialize the exploration set 2: Γ ← M ULTI - HOP -N EIGHBORS (Ω) // Multi-hop neighbors as the candidate set 3: Y Ω = G ET -V ALIDATION -P ERFORMANCE
Revised: Explore the initial nodes

Original: (Ω , V ) 4: while t =
Revised: V epochs) 4: while t =

Original: | Ω | < K do 5:
Revised: | Ω | < K do 5:

Original: G( t ) s ←( N v =
Revised: G s ←

Original: Ω ∪ Γ , E = S IMILARITY ( N v ) // (1) Design subgraph update 6: whilenot converge do 7:θ ← θ − η · ∂ L ( ˆ Y Ω , h θ ( G ( t ) s ) Ω ) /∂θ //
Revised: 6: while converge do 7:

Original: (2) Meta-GNN optimization 8: end while 9:  d( t ) =
Revised: ← conduct optimization 8: end while 9: // Sample a candidate node probability the meta GNN’s prediction 10: d

Original: S AMPLE - WITH -L OGITS (Γ , h θ ( G ( t ) s ) Γ ) //
Revised: AMPLE P ROBABILITY

Original: (3) Node exploration 10:
Revised: , Softmax

Original: Yt = 
Revised: Y = ET

Original: G ET -V ALIDATION -P ERFORMANCE( d ( t ) , V ) 11:
Revised: V ALIDATION P ERFORMANCE

Original:  Ω ← Ω ∪ { d ( t ) } , Γ ← Γ ∪ M ULTI - HOP -N EIGHBORS ( { d ( t ) } ) 12: end while 13: D = S ELECT -T OPK( { Ω i : Y i } Ki =1 , size =
Revised: Explore the current selected node 12: Ω ← 13: end while 14: D = S ELECT -T OPK

Original: M IN ( ⌈ 10% · K ⌉ , 5)) //
Revised: ( { i : Y i i = size

Original: Models to be fully trained 14: Y ′ =
Revised: M fully trained 15: Y ′ =

Original: G ET -V ALIDATION -P ERFORMANCE ( D, V ′ ) 15: I =
Revised: G ET -V ALIDATION -P ERFORMANCE ( D, V final performance 16: I =

Original: A RGMAX( Y ′ ) 16: return D
Revised: A RGMAX Y ′ ) best model 17: return D

Original: I , Y ′ 
Revised: I , Y ′ I a quadratic number of training pairs, thus reducing overfitting.

Original: I performance is more robust across datasets than predicting absolute performance.
Revised: Furthermore, predicting relative performance is more robust across datasets than predicting absolute performance.

Original: Overall, the objective is formulated as follows:
Revised: Overall, the objective is formulated as follows:

Original: [Equation]
Revised: [Equation]

Original: , Y ) , where [Equation] where λ is the trade-off hyper-parameter, τ is the temperature controlling the minimal performance gap that will be highly penalized, and σ is the Sigmoid function.
Revised: where λ is the trade-off hyper-parameter, τ is the temperature controlling the minimal performance gap that will be highly penalized, and σ is the Sigmoid function.

Original: Thus, the meta-GNN is trained to predict the node performance on the design subgraph supervised by the explored node performance.
Revised: Thus, the meta-GNN is trained to predict the node performance on the design subgraph supervised by the explored node performance.

Original: 3. S EARCH S TRATEGY
Revised: 3. S EARCH S TRATEGY

Original: Equipped with the meta-GNN, we propose a sequential search strategy to search for the best design in the design graph.
Revised: Equipped with the meta-GNN, we propose a sequential search strategy to search for the best design in the design graph.

Original: The core idea is to leverage meta-GNN to perform fast inference on the dynamic design subgraph, and decide what would be the next node to explore, thus navigating the search.
Revised: The core idea is to leverage meta-GNN to perform fast inference on the dynamic design subgraph, and decide what would be the next node to explore, thus navigating the search.

Original: We summarize our search strategy in Algorithm 1.
Revised: We summarize our search strategy in Algorithm 1.

Original: Concretely, our search strategy consists of the following three steps:
Revised: Concretely, our search strategy consists of the following three steps:

Original: • Initialization : As shown in Figure 1 (b), F ALCON randomly samples multiple nodes on the design graph.
Revised: • Initialization : As shown in Figure 1 (b), F ALCON randomly samples multiple nodes on the design graph.

Original: The motivation of sampling multiple nodes in the initial step is to enlarge the receptive field on the design graph to avoid the local optima and bad performance, which is empirically verified in Appendix D.
Revised: The motivation of sampling multiple nodes in the initial step is to enlarge the receptive field on the design graph to avoid the local optima and bad performance, which is empirically verified in Appendix D.

Original: Then, F ALCON explore the initialized nodes by training designs on the tasks and construct the instance mask for the task-specific module.
Revised: Then, F ALCON explore the initialized nodes by training designs on the tasks and construct the instance mask for the task-specific module.

Original: • Meta-GNN training : Following Figure 2, meta-GNN predicts the performance of the explored nodes.
Revised: • Meta-GNN training : Following Figure 2, meta-GNN predicts the performance of the explored nodes.

Original: The loss is computed via Equation 2 and back-propagated to optimize the meta-GNN.
Revised: The loss is computed via Equation 2 and back-propagated to optimize the meta-GNN.

Original: • Exploration via inference : Meta-GNN is then used to make predictions for the performances of all candidate nodes.
Revised: • Exploration via inference : Meta-GNN is then used to make predictions for the performances of all candidate nodes.

Revised: Then we apply Softmax on the predictions

Original:  F ALCON samples the best candidate nodebased on the predicted performance and updates the design subgraph in the next step.
Revised: as the probability distribution of candidate designs, from which F ALCON samples a new node and updates the design subgraph .

Original: At every iteration, F ALCON extends the design subgraph through the last two steps.
Revised: At every iteration, F ALCON extends the design subgraph through the last two steps.

Original: After several iterations, it selects and retrains a few designs in the search trajectory with top performances.
Revised: After several iterations, it selects and retrains a few designs in the search trajectory with top performances.

Original: Overall, F ALCON approaches the optimal design navigated by the relational inductive bias learned by metaGNN, as shown in Figure 1 (b).[Table]
Revised: Overall, F ALCON approaches the optimal design navigated by the relational inductive bias learned by metaGNN, as shown in Figure 1 (b).

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We conduct extensive experiments on 27 graph datasets and an image dataset.
Revised: We conduct extensive experiments on 27 graph datasets and an image dataset.

Original: The goal is twofold: (1) to show F ALCON ’s sample efficiency over the existing AutoML methods ( cf.
Revised: The goal is twofold: (1) to show F ALCON ’s sample efficiency over the existing AutoML methods ( cf.

Original: Section 4.2) and (2) to provide insights into how the inductive bias of design relartions navigate the search on design graph ( cf.
Revised: Section 4.2) and (2) to provide insights into how the inductive bias of design relartions navigate the search on design graph ( cf.

Original: Section 4.3).
Revised: Section 4.3).

Original: 4. E XPERIMENTAL S ETTINGS
Revised: 4. E XPERIMENTAL S ETTINGS

Original: We consider the following tasks in our evaluation and we leave the details including dataset split, evaluation metrics, and hyper-parameters in Appendix A.
Revised: We consider the following tasks in our evaluation and we leave the details including dataset split, evaluation metrics, and hyper-parameters in Appendix A.

Original: Node classification .
Revised: Node classification .

Original: We use 6 benchmarks ranging from citation networks to product or social networks: Cora, CiteSeer, PubMed (Sen et al., 2008), ogbn-arxiv (Hu et al., 2020),
Revised: We use 6 benchmarks ranging from citation networks to product or social networks: Cora, CiteSeer, PubMed (Sen et al., 2008), ogbn-arxiv (Hu et al., 2020),

Original: AmazonComputers (Shchur et al., 2018), and Reddit (Zeng et al., 2020).
Revised: AmazonComputers (Shchur et al., 2018), and Reddit (Zeng et al., 2020).

Original: Graph classification .
Revised: Graph classification .

Original: We use 21 benchmark binary classification tasks in TUDataset (Morris et al., 2020), which are to predict certain properties for molecule datasets with various distribution.
Revised: We use 21 benchmark binary classification tasks in TUDataset (Morris et al., 2020), which are to predict certain properties for molecule datasets with various distribution.

Original: Image classification .
Revised: Image classification .

Original: We use CIFAR-10 (Krizhevsky, 2009).
Revised: We use CIFAR-10 (Krizhevsky, 2009).

Original: See details in Appendix C.
Revised: See details in Appendix C.

Original: Baselines .
Revised: Baselines .

Original: We compare F ALCON with three types of baselines:
Revised: We compare F ALCON with three types of baselines:

Original: • Simple search strategies: Random, Simulated Annealing (SA), Bayesian Optimization (BO) (Bergstra et al., 2011).
Revised: • Simple search strategies: Random, Simulated Annealing (SA), Bayesian Optimization (BO) (Bergstra et al., 2011).

Original: • AutoML approaches: DARTS (Liu et al., 2019), ENAS (Pham et al., 2018), GraphNAS (Gao et al., 2019), AutoAttend (Guan et al., 2021), GASSO (Qin et al., 2021), where the last three methods are specifically designed for graph tasks.
Revised: • AutoML approaches: DARTS (Liu et al., 2019), ENAS (Pham et al., 2018), GraphNAS (Gao et al., 2019), AutoAttend (Guan et al., 2021), GASSO (Qin et al., 2021), where the last three methods are specifically designed for graph tasks.

Original: • Ablation models: F ALCON -G and F ALCON -LP, where F ALCON -G discards the design graph and predicts the design performance using an MLP, and F ALCON -LP removes the task-specific module and predicts design performance using only the task-agnostic module.
Revised: • Ablation models: F ALCON -G and F ALCON -LP, where F ALCON -G discards the design graph and predicts the design performance using an MLP, and F ALCON -LP removes the task-specific module and predicts design performance using only the task-agnostic module.

Original: We also include a naive method, B RUTEFORCE , which trains 5% designs from scratch and returns the best design among them.
Revised: We also include a naive method, B RUTEFORCE , which trains 5% designs from scratch and returns the best design among them.

Original: The result of B RUTEFORCE is regarded as the approximated ground truth performance.
Revised: The result of B RUTEFORCE is regarded as the approximated ground truth performance.

Original: We compare F ALCON and the simple search baselines under sample size controlled search , where we limit the number of explored designs.
Revised: We compare F ALCON and the simple search baselines under sample size controlled search , where we limit the number of explored designs.

Original: We set the exploration size as 30 by default.
Revised: We set the exploration size as 30 by default.

Original: Design Space .
Revised: Design Space .

Original: We use different design spaces on node- and graph-level tasks.
Revised: We use different design spaces on node- and graph-level tasks.

Original: Specifically, The design variables include common hyper-parameters, e.g., dropout ratio, and architecture choices, e.g., layer connectivity and batch normalization.
Revised: Specifically, The design variables include common hyper-parameters, e.g., dropout ratio, and architecture choices, e.g., layer connectivity and batch normalization.

Original: Moreover, we consider node pooling choices for the graph classification datasets, which is less studied in the previous works (Cai et al., 2021;
Revised: Moreover, we consider node pooling choices for the graph classification datasets, which is less studied in the previous works (Cai et al., 2021;

Original: Gao et al., 2019; Zhou et al., 2019).
Revised: Gao et al., 2019; Zhou et al., 2019).

Original: Besides, we follow You et al.
Revised: Besides, we follow You et al.

Original: (2020b)
Revised: (2020b)

Original: and control the number of parameters for all the candidate designs to ensure a fair comparison.
Revised: and control the number of parameters for all the candidate designs to ensure a fair comparison.

Original: See Appendix A.2 for the details.
Revised: See Appendix A.2 for the details.

Original: [Figure]
Revised: 4. M AIN R ESULTS

Original: Node classification tasks .
Revised: Node classification tasks .

Original: Table 1 summarizes the performance of F ALCON and the baselines.
Revised: Table 1 summarizes the performance of F ALCON and the baselines.

Original: Notably, F ALCON takes comparable search cost as the oneshot methods and is 15x less expensive than GraphNAS.
Revised: Notably, F ALCON takes comparable search cost as the oneshot methods and is 15x less expensive than GraphNAS.

Original: Moreover, F ALCON achieves the best performances over the baselines with sufficient margins in the most datasets, using only 30 explored designs.
Revised: Moreover, F ALCON achieves the best performances over the baselines with sufficient margins in the most datasets, using only 30 explored designs.

Original: For example, F ALCON outperforms
Revised: For example, F ALCON outperforms

Original: ENAS by 1.8% in CiteSeer and GASSO by 1.6% in AmazonComputers.
Revised: ENAS by 1.8% in CiteSeer and GASSO by 1.6% in AmazonComputers.

Original: Also, the removal of the design graph and task-specific module decreases the performance constantly, which validates their effectiveness.
Revised: Also, the removal of the design graph and task-specific module decreases the performance constantly, which validates their effectiveness.

Original: It is worth mentioning that
Revised: It is worth mentioning that

Original: F ALCON is competitive with B RUTEFORCE , demonstrating the excellence of F ALCON in searching for globally bestperforming designs.
Revised: F ALCON is competitive with B RUTEFORCE , demonstrating the excellence of F ALCON in searching for globally bestperforming designs.

Revised: [Table]

Original: We further investigate the speed-performance trade-off of F ALCON and other sample-based approaches in ogbn-arxiv.
Revised: We further investigate the speed-performance trade-off of F ALCON and other sample-based approaches in ogbn-arxiv.

Original: We run several search trials under different sample sizes.
Revised: We run several search trials under different sample sizes.

Original: As shown in Figure 3,
Revised: As shown in Figure 3,

Original: F ALCON reaches the approximated ground truth result with very few explored nodes.
Revised: F ALCON reaches the approximated ground truth result with very few explored nodes.

Original: In contrast, SA and Random require more samples to converge, while BO performs bad even with a large number of explored nodes, potentially due to its inability in dealing with high-dimensional design features.
Revised: In contrast, SA and Random require more samples to converge, while BO performs bad even with a large number of explored nodes, potentially due to its inability in dealing with high-dimensional design features.

Original: Graph classification tasks .
Revised: Graph classification tasks .

Original: The graph classification datasets cover a wide range of graph distributions.
Revised: The graph classification datasets cover a wide range of graph distributions.

Original: In Table 2, we report the selected performance results for graph classification tasks and leave other results including the search costs in Appendix B.
Revised: In Table 2, we report the selected performance results for graph classification tasks and leave other results including the search costs in Appendix B.

Original: We highlight three observations:
Revised: We highlight three observations:

Original: • On average, the state-of-the-art AutoML baselines algorithms perform close to the simple search methods, indicating the potentially unreliable search, as similarly concluded by Yu et al.
Revised: • On average, the state-of-the-art AutoML baselines algorithms perform close to the simple search methods, indicating the potentially unreliable search, as similarly concluded by Yu et al.

Original: • F ALCON surpasses the best AutoML baselines with an average improvement of 3.3%.
Revised: • F ALCON surpasses the best AutoML baselines with an average improvement of 3.3%.

Original: The sufficient and consistent improvement greatly validates our sample efficiency under a controlled sample size.
Revised: The sufficient and consistent improvement greatly validates our sample efficiency under a controlled sample size.

Original: where F ALCON can explore the designs that are more likely to perform well through the relational inference based on the relations of previously explored designs and their performances.
Revised: where F ALCON can explore the designs that are more likely to perform well through the relational inference based on the relations of previously explored designs and their performances.

Original: • In the second block, we attribute the high sample efficiency of F ALCON to the exhibition of design relations and the performance information from the training instances.
Revised: • In the second block, we attribute the high sample efficiency of F ALCON to the exhibition of design relations and the performance information from the training instances.

Original: Specifically, F ALCON outperforms F ALCON -LP by 4.87% on average, indicating that the task-specific module provides more task information that aids the representation learning of model designs, enabling a fast adaption on a certain task.
Revised: Specifically, F ALCON outperforms F ALCON -LP by 4.87% on average, indicating that the task-specific module provides more task information that aids the representation learning of model designs, enabling a fast adaption on a certain task.

Original: Moreover, F ALCON gains an average improvement of 6.43% compared to F ALCON -G, which justifies our motivation that the design relations promote the learning of relational inductive bias and guide the search on the design graph.
Revised: Moreover, F ALCON gains an average improvement of 6.43% compared to F ALCON -G, which justifies our motivation that the design relations promote the learning of relational inductive bias and guide the search on the design graph.

Original: We also conduct experiments similar to Figure 3 to investigate how F ALCON converges with the increasing sample size ( cf.
Revised: We also conduct experiments similar to Figure 3 to investigate how F ALCON converges with the increasing sample size ( cf.

Original: Appendix B.1)
Revised: Appendix B.1)

Original: and report the best designs found by F ALCON for each dataset ( cf.
Revised: and report the best designs found by F ALCON for each dataset ( cf.

Original: Appendix B.2).
Revised: Appendix B.2).

Original: Besides, we provide sensitivity analysis on F ALCON ’s hyper-parameters, e.g., number of random start nodes C ( cf.
Revised: Besides, we provide sensitivity analysis on F ALCON ’s hyper-parameters, e.g., number of random start nodes C ( cf.

Original: Appendix D).
Revised: Appendix D).

Original: Image classification task .
Revised: Image classification task .

Original:  Due to space limitation, we leave the results of CIFAR-10 to Appendix C and discuss the conclusion here. We found F ALCON can search for both hyper-parameters and architectures that are best-performing, compared with the baselines.
Revised: We demonstrate the potential of F ALCON in image domain. Due to space limitation, we leave the results of CIFAR-10 to Appendix C. We found F ALCON can search for designs that are best-performing, compared with the baselines.

Original: Specifically, it gains average improvements of 1.4% over the simple search baselines and 0.3% over the one-shot baselines on the architecture design space. Moreover, the search costof F ALCON is comparable to the one-shot based baselines, using approximately 0.8 GPU · days.
Revised: Specifically, it gains average improvements of 1.4% over the simple search baselines and 0.3% over the one-shot baselines on the architecture design space, with search cost comparable to the one-shot based baselines.

Original: 4. C ASE S TUDIES OF F ALCON
Revised: 4. C ASE S TUDIES OF F ALCON

Original: We study F ALCON in two dimensions: (1) Search process: we probe F ALCON ’s inference process through the explanations of meta-GNN on a design graph, and (2) Design representations: we visualize the node representations output by the meta-GNN to examine the effect of design choices.
Revised: We study F ALCON in two dimensions: (1) Search process: we probe F ALCON ’s inference process through the explanations of meta-GNN on a design graph, and (2) Design representations: we visualize the node representations output by the meta-GNN to examine the effect of design choices.

Original: Search process .
Revised: Search process .

Original: We use GNNExplainer (Ying et al., 2019) to explain the node prediction of metaGNN and shed light on the working mechanism of F ALCON .
Revised: We use GNNExplainer (Ying et al., 2019) to explain the node prediction of metaGNN and shed light on the working mechanism of F ALCON .

Original: Here we consider the importance of  each design dimension for each node’s prediction.
Revised: Here we consider the importance of [Table] each design dimension for each node’s prediction.

Original: We demonstrate on a real case when searching on
Revised: We demonstrate on a real case when searching on

Original: [Table]

Original: CIFAR-10 ( cf.
Revised: CIFAR-10 ( cf.

Original: Table 12 for the design space).
Revised: Table 12 for the design space).

Original: For conciseness, we focus on two design dimensions: (Weight Decay, Batch Size).
Revised: For conciseness [Table] we focus on two design dimensions: (Weight Decay, Batch Size).

Original: Then, given a node of interest n ′ =
Revised: Then, given a node of interest n ′ =

Original: (0 . 9 , 128) , we observe the change in its predictions and dimension importance during the search process.
Revised: [Table] we observe the change in its predictions and dimension importance during the search process.

Original: Explored node n t : ...
Revised: Explored node n t : ...

Original: Performance of n t : ...
Revised: n t : ...

Original: ++ − ...
Revised: ++ − ...

Original: + ...
Revised: + ...

Original: Prediction on n ′ : ...
Revised: Prediction on n ′ : ...

Original: Dimension importance: ...
Revised: Dimension importance: ...

Original: Where + and − indicate the relative performance of the explored nodes, t is the current search step.
Revised: Where + and − indicate the relative performance of the explored nodes, t is the current search step.

Original: Interestingly, we see that the prediction on n ′ and the dimension importance evolve with the explored designs and their relations.
Revised: Interestingly, we see that the prediction on n ′ and the dimension importance evolve with the explored designs and their relations.

Original: For example, when weight decay changes from 0 . 99 to 0 .
Revised: For example, when weight decay changes from 0 . 99 to 0 .

Original: 9 , there is a huge drop in the node performance, which affects the prediction of n ′ and increases the importance of Weight Decay as design performance seems to be sensitive to this dimension.
Revised: 9 , there is a huge drop in the node performance, which affects the prediction of n ′ and increases the importance of Weight Decay as design performance seems to be sensitive to this dimension.

Original: Design representations .
Revised: Design representations .

Original: In Figure 4, we visualize the high-dimensional design representations via
Revised: In Figure 4, we visualize the high-dimensional design representations via

Original: T-SNE (van der Maaten & Hinton, 2008) after training the meta-GNN on the Cora dataset.
Revised: T-SNE (van der Maaten & Hinton, 2008) after training the meta-GNN on the Cora dataset.

Original: In the left figure, the better the design performance, the darker the color.
Revised: In the left figure, the better the design performance, the darker the color.

Original: Generally, the points with small distance have similar colors or performances, indicating that meta-GNN can distinguish “good” nodes from “bad” nodes.
Revised: Generally, the points with small distance have similar colors or performances, indicating that meta-GNN can distinguish “good” nodes from “bad” nodes.

Original: For the right figure, different colors represent different dropout ratios.
Revised: For the right figure, different colors represent different dropout ratios.

Original: The high discrimination indicates that the dropout ratio is an influential variable for learning the design representation, which further affects design performance.
Revised: The high discrimination indicates that the dropout ratio is an influential variable for learning the design representation, which further affects design performance.

Original: This evidence validates the meta-GNN’s expressiveness and capacity to learn the relational inductive bias between the design choices.
Revised: This evidence validates the meta-GNN’s expressiveness and capacity to learn the relational inductive bias between the design choices.

Original:  C ONCLUSION , L IMITATION , AND F UTURE W ORK
Revised:  C ONCLUSION , L IMITATION , AND F UTURE W ORK

Original: This work introduces F ALCON , an efficient sample-based AutoML framework.
Revised: This work introduces F ALCON , an efficient sample-based AutoML framework.

Original: We propose the concept of design graph that explicitly models and encodes relational information among model designs.
Revised: We propose the concept of design graph that explicitly models and encodes relational information among model designs.

Original: The design graph is a basis for incorporating inductive biases of designs, which are essential for design space pruning and search navigation.

Original: On top of the design graph, we develop a sampleefficient strategy to navigate the search on the design graph with a novel meta-model.
Revised: On top of the design graph, we develop a sample-efficient strategy to navigate the search on the design graph with a novel meta-model.

Original: One future direction is to better tackle the high average node degree on the design graph, especially when the design variables include many categorical variables.Since conducting message passing or label propagation on a dense graph structure could result in over-smoothing of the meta-GNN.
Revised: One future direction is to better tackle the high average node degree on the design graphwhich could cause over-smoothing, especially when the design variables include many categorical variables.

Original: A simple solution is to use edge dropout to randomly remove a portion of edges at each training epoch.
Revised: And a simple solution is to use edge dropout to randomly remove a portion of edges at each training epoch.

Original: Further, we may develop a local filtering mechanism by edge attention and filter edges for sparsity.
Revised: Another future direction is to better adapt FALCON on continuous design variables via developing a dynamic design graph that enable a more fine-grained search between the discretized values.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: All of the datasets used in this work are public.
Revised: All of the datasets used in this work are public.

Original: For experimental setup, we state the detailed settings in Appendix A and Appendix C, including the graph pre-processing, dataset splits, hyper-parameters.
Revised: For experimental setup, we state the detailed settings in Appendix A and Appendix C, including the graph pre-processing, dataset splits, hyper-parameters.

Original: Moreover, we include our code in an anonymous link for public access.
Revised: Moreover, we include our code in an anonymous link for public access.

Original: For the results, we report the best models found by our algorithm as well as their corresponding performances.
Revised: For the results, we report the best models found by our algorithm as well as their corresponding performances.

Original: Overall, we believe we have made great efforts to ensure reproducibility in this paper.
Revised: Overall, we believe we have made great efforts to ensure reproducibility in this paper.

Original: E THICS S TATEMENT
Revised: E THICS S TATEMENT

Original: In this work we propose a novel algorithm to search for the best model designs where no human subject is related.
Revised: In this work we propose a novel algorithm to search for the best model designs where no human subject is related.

Original: This work could promote the discovery of more powerful and expressive models and provide insights into design relations.
Revised: This work could promote the discovery of more powerful and expressive models and provide insights into design relations.

Original: However, while best-performing models may be “experts” in fulfilling given tasks, they are not necessarily fair towards different user or entity groups.
Revised: However, while best-performing models may be “experts” in fulfilling given tasks, they are not necessarily fair towards different user or entity groups.

Original: We believe this is a general issue in the AutoML area and should be well addressed to ensure the ethics of models in real-world applications.
Revised: We believe this is a general issue in the AutoML area and should be well addressed to ensure the ethics of models in real-world applications.

Original: A E XPERIMENT D ETAILS A. S ETTINGS
Revised: A E XPERIMENT D ETAILS A. S ETTINGS

Original: Graph classification datasets .
Revised: Graph classification datasets .

Original: The graph classification datasets used in this work are summarized in Table 3.
Revised: The graph classification datasets used in this work are summarized in Table 3.

Original: And the detailed dataset statics can be referred from https://chrsmrrs.github.
Revised: And the detailed dataset statics can be referred from https://chrsmrrs.github.

Original: io/datasets/docs/datasets/ .
Revised: io/datasets/docs/datasets/ .

Original: [Table]
Revised: [Table]

Original: Specifically, all datasets are binary classification tasks that predict certain properties for small molecules.
Revised: Specifically, all datasets are binary classification tasks that predict certain properties for small molecules.

Original: For example, the labels in Tox21-AhR represent toxicity/non-toxicity, while the graphs in
Revised: For example, the labels in Tox21-AhR represent toxicity/non-toxicity, while the graphs in

Original: Mutagenicity are classified into two classes based on their mutagenic effect on a bacterium (Morris et al., 2020).
Revised: Mutagenicity are classified into two classes based on their mutagenic effect on a bacterium (Morris et al., 2020).

Original: Consequently, we use atom types as the node features and bond types as edge features.
Revised: Consequently, we use atom types as the node features and bond types as edge features.

Original: Evaluation metrics .
Revised: Evaluation metrics .

Original: For Reddit, we use F1 score (micro) as the evaluation metric following the previous work (Zeng et al., 2020).
Revised: For Reddit, we use F1 score (micro) as the evaluation metric following the previous work (Zeng et al., 2020).

Original: For other node classification tasks and image dataset, we use classification accuracy as the evaluation metric.
Revised: For other node classification tasks and image dataset, we use classification accuracy as the evaluation metric.

Original: For the graph classification tasks, we use ROC-AUC as the evaluation metric.
Revised: For the graph classification tasks, we use ROC-AUC as the evaluation metric.

Original: Dataset splits .
Revised: Dataset splits .

Original: For ogbn-arxiv and Reddit, we use the standardized dataset split.
Revised: For ogbn-arxiv and Reddit, we use the standardized dataset split.

Original: For other node classification datasets, we split the nodes in each graph into 70%, 10%, 20% in training, validation, and test sets, respectively.
Revised: For other node classification datasets, we split the nodes in each graph into 70%, 10%, 20% in training, validation, and test sets, respectively.

Original: For graph classification tasks, we split the graphs into 80%, 10%, 10% for training, validation, and test sets, respectively.
Revised: For graph classification tasks, we split the graphs into 80%, 10%, 10% for training, validation, and test sets, respectively.

Original: Hyper-Parameters .
Revised: Hyper-Parameters .

Original: We tuned the hyper-parameters of the baselines based on the default setting in their public codes.
Revised: We tuned the hyper-parameters of the baselines based on the default setting in their public codes.

Original: For F ALCON , we construct the candidate set as the 3-hop neighbors of the explored nodes and set the number of start nodes as min ( ⌈ 10% · K ⌉ , 10) , where K denotes the exploration size.
Revised: For F ALCON , we construct the candidate set as the 3-hop neighbors of the explored nodes and set the number of start nodes as min ( ⌈ 10% · K ⌉ , 10) , where K denotes the exploration size.

Original: The meta-GNN is constitute of 3 message-passing layers and 3 label propagation layers.
Revised: The meta-GNN is constitute of 3 message-passing layers and 3 label propagation layers.

Original: All the experiments are repeated at least 3 times.
Revised: All the experiments are repeated at least 3 times.

Original: A. D ESIGN S PACES A.2. D ESIGN S PACES FOR THE S AMPLE - BASED M ETHODS
Revised: A. D ESIGN S PACES A.2. D ESIGN S PACES FOR THE S AMPLE - BASED M ETHODS

Original: In this work, we use different design spaces for the datasets depending on the task types, i.e., node or graph level.
Revised: In this work, we use different design spaces for the datasets depending on the task types, i.e., node or graph level.

Original: We summarize the design variables and choices in Table 4 and Table 5.
Revised: We summarize the design variables and choices in Table 4 and Table 5.

Original: For the design space of Reddit, we replace ”Aggregation” in Table 4 with ”Convolutional layer type”, which takes values from { GCNConv (Kipf & Welling, 2017), SAGEConv (Hamilton et al., 2017),
Revised: For the design space of Reddit, we replace ”Aggregation” in Table 4 with ”Convolutional layer type”, which takes values from { GCNConv (Kipf & Welling, 2017), SAGEConv (Hamilton et al., 2017),

Original: GraphConv (Morris et al., 2019), GINConv (Xu et al., 2019), ARMAConv (Bianchi et al., 2019),
Revised: GraphConv (Morris et al., 2019), GINConv (Xu et al., 2019), ARMAConv (Bianchi et al., 2019),

Original: TAGConv (Du et al.) } .
Revised: TAGConv (Du et al.) } .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Specifically, The STACK design choice means directly stacking multiple GNN layers,
Revised: Specifically, The STACK design choice means directly stacking multiple GNN layers,

Original: i
Revised: i

Original: [Table]
Revised: [Table]

Original: e., without skip-connections.
Revised: e., without skip-connections.

Original: We also support node pooling operations for graph classification tasks, where the pooling loop stands for the number of message passing layers between each pooling operation.
Revised: We also support node pooling operations for graph classification tasks, where the pooling loop stands for the number of message passing layers between each pooling operation.

Original: If the number of message passing layers is m and the node pooling loop is l , there will be a node pooling layer after the i th message passing layer in the design model (hierarchical pooling), where i ∈ { 1 + k · l | k = 0 , . .
Revised: If the number of message passing layers is m and the node pooling loop is l , there will be a node pooling layer after the i th message passing layer in the design model (hierarchical pooling), where i ∈ { 1 + k · l | k = 0 , . .

Original: , ⌈ ( m − 1) /l
Revised: , ⌈ ( m − 1) /l

Original: ⌉ − 1 } .
Revised: ⌉ − 1 } .

Original: Moreover, to avoid duplicated and invalid designs, some design variables are required to satisfy specific dependency rules, and
Revised: Moreover, to avoid duplicated and invalid designs, some design variables are required to satisfy specific dependency rules, and

Original: we take
Revised: we take

Original: two examples
Revised: two examples

Original: to
Revised: to

Original: elaborate on this point
Revised: elaborate on this point

Original: • If the node pooling flag of a design is False, then the design does not have any value on node pooling type and node pooling loop, and vice versa.
Revised: • If the node pooling flag of a design is False, then the design does not have any value on node pooling type and node pooling loop, and vice versa.

Original: For example, we denote node pooling flag as f , node pooling type as t , and ∗ as any design choice, then( f =False, t = ∗ ) or ( f =False, l =*) will both be invalid.
Revised: For example, we denote node pooling flag as f , node pooling type as t , and ∗ as any design choice, then( f =False, t = ∗ ) or ( f =False, l =*) will both be invalid.

Original: The node pooling loop should not exceed the number of message passing layers.
Revised: The node pooling loop should not exceed the number of message passing layers.

Original: For
Revised: For

Original: example
Revised: example

Original: , design A ( m =4, l =4) and design B ( m =4, l =6) that take the same values on other design variables are duplicated
Revised: , design A ( m =4, l =4) and design B ( m =4, l =6) that take the same values on other design variables are duplicated

Original: Thus, the design graph constructed under dependency rules is more complex.
Revised: Thus, the design graph constructed under dependency rules is more complex.

Original: Without loss of generality, we define that the distance of ( f =False) and ( f =True, l =M IN ( { i ∈ L } )) as 1, where L represents the design choice of node pooling loop.
Revised: Without loss of generality, we define that the distance of ( f =False) and ( f =True, l =M IN ( { i ∈ L } )) as 1, where L represents the design choice of node pooling loop.

Original: Thus, the design graph is a connected graph that enables the exploration of any node with random initialization.
Revised: Thus, the design graph is a connected graph that enables the exploration of any node with random initialization.

Original: It is also worth mentioning that the search strategy of F ALCON is modularized given the design graph.
Revised: It is also worth mentioning that the search strategy of F ALCON is modularized given the design graph.

Original: In contrast, the dependency rules constrain the action space of reinforcement learning methods, e.g., ( f =Ture → False) is inapplicable, which requires special operation inside the controller.
Revised: In contrast, the dependency rules constrain the action space of reinforcement learning methods, e.g., ( f =Ture → False) is inapplicable, which requires special operation inside the controller.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: We further summarize the statistics and construction time of the design graphs in Table 6, where
Revised: We further summarize the statistics and construction time of the design graphs in Table 6, where

Original: DG-1 and DG-2 denote the design graphs for node-level and graph-level tasks, respectively.
Revised: DG-1 and DG-2 denote the design graphs for node-level and graph-level tasks, respectively.

Original: We use multi-processing programing on 50 CPUs (Intel Xeon Gold 5118 CPU @ 2.30GHz) to conduct the graph construction.
Revised: We use multi-processing programing on 50 CPUs (Intel Xeon Gold 5118 CPU @ 2.30GHz) to conduct the graph construction.

Original: Note that we don’t have to construct the entire design graph in the pre-processing step, since we only extend the small portion of the design graph, i.e., , the design subgraph, during the search process.
Revised: Note that we don’t have to construct the entire design graph in the pre-processing step, since we only extend the small portion of the design graph, i.e., , the design subgraph, during the search process.

Original: Thus, the total time cost of constructing the design subgraph will be O ( E ′ )
Revised: Thus, the total time cost of constructing the design subgraph will be O ( E ′ )

Original: where
Revised: where

Original: E ′ is the number of edges in the design subgraph, which largely lowers the time costs.
Revised: E ′ is the number of edges in the design subgraph, which largely lowers the time costs.

Original: A.2. D ESIGN S PACES FOR THE O NE - SHOT B ASELINES
Revised: A.2. D ESIGN S PACES FOR THE O NE - SHOT B ASELINES

Original: The one-shot models (Liu et al., 2019; Pham et al., 2018) is built upon a super-model that is required to contain all of the architecture choices.
Revised: The one-shot models (Liu et al., 2019; Pham et al., 2018) is built upon a super-model that is required to contain all of the architecture choices.

Original: We build the macro search space over entire models for both node classification and graph classification datasets with constraints.
Revised: We build the macro search space over entire models for both node classification and graph classification datasets with constraints.

Original: Firstly, we do not consider
Revised: Firstly, we do not consider

Original: CAT (skip-concatenate) a layer connectivity choice, and we also remove design variables for node pooling.
Revised: CAT (skip-concatenate) a layer connectivity choice, and we also remove design variables for node pooling.

Original: The reason is that CAT and node pooling operations change the input shape and make the output embeddings inapplicable for the subsequent weight-sharing modules in our settings.
Revised: The reason is that CAT and node pooling operations change the input shape and make the output embeddings inapplicable for the subsequent weight-sharing modules in our settings.

Original: Secondly, the layer connectivity is customized for each layer following the previous works (Liu et al., 2019;
Revised: Secondly, the layer connectivity is customized for each layer following the previous works (Liu et al., 2019;

Original: Pham et al., 2018), instead of setting as a global value for every layer.
Revised: Pham et al., 2018), instead of setting as a global value for every layer.

Original: Overall, we summarize the design space in Table 7.
Revised: Overall, we summarize the design space in Table 7.

Original: To enable a fair comparison, we fine-tune the hyper-parameters and report the best results of the architectures found by the one-shot methods according to their performance in the validation sets.
Revised: To enable a fair comparison, we fine-tune the hyper-parameters and report the best results of the architectures found by the one-shot methods according to their performance in the validation sets.

Original: B M ORE E XPERIMENTAL R ESULTS ON G RAPH T ASKS B. G RAPH C LASSIFICATION T ASKS
Revised: B M ORE E XPERIMENTAL R ESULTS ON G RAPH T ASKS B. G RAPH C LASSIFICATION T ASKS

Original: Task performance .
Revised: Task performance .

Original: Here we provide more results of task performance on the graph classification dataset.
Revised: Here we provide more results of task performance on the graph classification dataset.

Original: We repeat each experiment at least 3 times and report the average performances and the standard errors.
Revised: We repeat each experiment at least 3 times and report the average performances and the standard errors.

Original: The results are summarized in Table 8.
Revised: The results are summarized in Table 8.

Original: The results well demonstrate the preeminence of F ALCON in searching for good designs under different data distributions.
Revised: The results well demonstrate the preeminence of F ALCON in searching for good designs under different data distributions.

Original: Search cost .
Revised: Search cost .

Original: As shown in Figure 5, we report the search cost of Random, DARTS, ENAS, GraphNAS, and F ALCON on the selected datasets.
Revised: As shown in Figure 5, we report the search cost of Random, DARTS, ENAS, GraphNAS, and F ALCON on the selected datasets.

Original: The time measurements are conducted on a single NVIDIA
Revised: The time measurements are conducted on a single NVIDIA

Original: GeForce 3070 GPU (24G).
Revised: GeForce 3070 GPU (24G).

Original: We see F ALCON has a comparable time cost with Random and DARTS, which empirically proves the efficiency of F ALCON .
Revised: We see F ALCON has a comparable time cost with Random and DARTS, which empirically proves the efficiency of F ALCON .

Original: However, as F ALCON still needs to sample designs and train them from scratch ( i.e., the search cost of F ALCON is bounded by the search cost of Random), the computational cost is relatively high in
Revised: However, as F ALCON still needs to sample designs and train them from scratch ( i.e., the search cost of F ALCON is bounded by the search cost of Random), the computational cost is relatively high in

Original: [Table]
Revised: [Table]

Original: large datasets, e.g., OVCAR-8 and MCF-7.
Revised: large datasets, e.g., OVCAR-8 and MCF-7.

Original: We can potentially alleviate this limitation via integrating dataset sampling to reduce time costs.
Revised: We can potentially alleviate this limitation via integrating dataset sampling to reduce time costs.

Original: ROC-AUC v.s. exploration size .
Revised: ROC-AUC v.s. exploration size .

Original: Here we report the change in task performance on graph classification datasets with the number of explored nodes.
Revised: Here we report the change in task performance on graph classification datasets with the number of explored nodes.

Original: In Figure 6, we visualize the results on two graph classification datasets.
Revised: In Figure 6, we visualize the results on two graph classification datasets.

Original: We see that F ALCON can approach the best-performing designs quickly as the explored size grows.
Revised: We see that F ALCON can approach the best-performing designs quickly as the explored size grows.

Original: B. B EST D ESIGNS
Revised: B. B EST D ESIGNS

Original: In Table 9 and Table 10, we summarize the best designs found by F ALCON and B RUTEFORCE in each dataset, where the average number of parameters is 137.5k for all the graph classification datasets.
Revised: In Table 9 and Table 10, we summarize the best designs found by F ALCON and B RUTEFORCE in each dataset, where the average number of parameters is 137.5k for all the graph classification datasets.

Original: Note that we select the best designs according to their performance on validation sets; thus, there are cases where F ALCON surpasses B RUTEFORCE .
Revised: Note that we select the best designs according to their performance on validation sets; thus, there are cases where F ALCON surpasses B RUTEFORCE .

Original: We highlight the design variables that are different between F ALCON and B RUTEFORCE for comparison.
Revised: We highlight the design variables that are different between F ALCON and B RUTEFORCE for comparison.

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: B RUTEFORCE ’ S CONFIDENCE INTERVAL AND VARIANT
Revised: B RUTEFORCE ’ S CONFIDENCE INTERVAL AND VARIANT

Original: To estimate the uncertainty of Bruteforce
Revised: To estimate the uncertainty of Bruteforce

Original: [Table]
Revised: [Table]

Original: we compute the 95% confidence interval of Bruteforce using bootstrapping.
Revised: we compute the 95% confidence interval of Bruteforce using bootstrapping.

Original: Moreover, we consider a variant of Bruteforce baseline to compare with Bruteforce.
Revised: Moreover, we consider a variant of Bruteforce baseline to compare with Bruteforce.

Original: Specifically, we train all the designs in the design space for 30 epochs, select the top 10% design, and resume the training until 50 epochs.
Revised: Specifically, we train all the designs in the design space for 30 epochs, select the top 10% design, and resume the training until 50 epochs.

Original: After that, we choose the top 50% designs to be fully trained and return the best fully trained design based on the validation performance.
Revised: After that, we choose the top 50% designs to be fully trained and return the best fully trained design based on the validation performance.

Original: We run Bruteforce-bootstrap on four datasets as demonstrations.
Revised: We run Bruteforce-bootstrap on four datasets as demonstrations.

Original: We summarize the results in Table 11.
Revised: We summarize the results in Table 11.

Original: [Table]
Revised: [Table]

Original: Surprisingly, we found that the performance of Bruteforce and Bruteforce-bootstrap are very close.
Revised: Surprisingly, we found that the performance of Bruteforce and Bruteforce-bootstrap are very close.

Original: This indicates that Bruteforce (fully trained 5% design) is a good surrogate of Bruteforcebootstrapping (fully trained 5% design, but using bootstrapping selection), and could also well approximate the ground truth performance of the best design.
Revised: This indicates that Bruteforce (fully trained 5% design) is a good surrogate of Bruteforcebootstrapping (fully trained 5% design, but using bootstrapping selection), and could also well approximate the ground truth performance of the best design.

Original: [Table]
Revised: [Table]

Original: C E XPERIMENTAL R ESULTS ON THE I MAGE T ASK C. D ATASET PRE - PROCESSING
Revised: C E XPERIMENTAL R ESULTS ON THE I MAGE T ASK C. D ATASET PRE - PROCESSING

Original: We use the CIFAR-10 (Krizhevsky, 2009) image dataset to show F ALCON can work well on other machine learning domains.
Revised: We use the CIFAR-10 (Krizhevsky, 2009) image dataset to show F ALCON can work well on other machine learning domains.

Original: This dataset consists of 50,000 training images and 10,000 test images.
Revised: This dataset consists of 50,000 training images and 10,000 test images.

Original: We randomly crop them to size 32 × 32, and conduct random flipping and normalization.
Revised: We randomly crop them to size 32 × 32, and conduct random flipping and normalization.

Original: C. D ESIGN S PACES
Revised: C. D ESIGN S PACES

Original: Here we use two different design spaces to demonstrate F ALCON ’s ability in searching for both hyper-parameters and architectures on image dataset.
Revised: Here we use two different design spaces to demonstrate F ALCON ’s ability in searching for both hyper-parameters and architectures on image dataset.

Original: Hyper-parameter Design Space .
Revised: Hyper-parameter Design Space .

Original: We consider a broad space of hyper-parameter search, including common hyper-parameters like Batch Size.
Revised: We consider a broad space of hyper-parameter search, including common hyper-parameters like Batch Size.

Original: We train each design using a SGD optimizer, which requires weight decay and momentum as hyper-parameters.
Revised: We train each design using a SGD optimizer, which requires weight decay and momentum as hyper-parameters.

Original: We also use a learning rate (LR) scheduler, which reduces the learning rate when validation performance has stopped improving.
Revised: We also use a learning rate (LR) scheduler, which reduces the learning rate when validation performance has stopped improving.

Original: Specifically, the scheduler will reduce the learning rate by a factor if no improvement is seen for a ‘patience’ number of epochs.
Revised: Specifically, the scheduler will reduce the learning rate by a factor if no improvement is seen for a ‘patience’ number of epochs.

Original: It is also worth mentioning that F ALCON is flexible for other sets of hyper-parameter choices determined by the user end.
Revised: It is also worth mentioning that F ALCON is flexible for other sets of hyper-parameter choices determined by the user end.

Original: [Table]
Revised: [Table]

Original: Architecture Design Space .
Revised: Architecture Design Space .

Original: We construct micro design space for the computational cells.
Revised: We construct micro design space for the computational cells.

Original: Each cell constitutes of two branches, where we enable five selections: separable convolution with kernel size× 3 and 5 × 5, average pooling and max pooling with kernel size 3×3, and identity.
Revised: Each cell constitutes of two branches, where we enable five selections: separable convolution with kernel size× 3 and 5 × 5, average pooling and max pooling with kernel size 3×3, and identity.

Original: In each branch, we have one dropout layer, where the dropout ratio is one of { 0 . 0 , 0 . 3 , 0 . 6 } , and one batch normalization layer.
Revised: In each branch, we have one dropout layer, where the dropout ratio is one of { 0 . 0 , 0 . 3 , 0 . 6 } , and one batch normalization layer.

Original: We also use one of identity and skip-sum as skip-connection within each branch.
Revised: We also use one of identity and skip-sum as skip-connection within each branch.

Original: After the input data is separately computed in each branch, the outputs are added as the final cell output, which is different with the original ENAS paper that searches the computational DAG on the defined nodes.
Revised: After the input data is separately computed in each branch, the outputs are added as the final cell output, which is different with the original ENAS paper that searches the computational DAG on the defined nodes.

Original: C. E XPERIMENTAL R ESULTS
Revised: C. E XPERIMENTAL R ESULTS

Original: Here we set the exploration size as 20 for all the sample-based methods.
Revised: Here we set the exploration size as 20 for all the sample-based methods.

Original: For hyper-parameter design space, we compare F ALCON with the baselines that are available for hyper-parameter tuning.
Revised: For hyper-parameter design space, we compare F ALCON with the baselines that are available for hyper-parameter tuning.

Original: Moreover, to accelerate the search process, F ALCON explores an unknown design by fine-tuning a pretrained model for several epochs based on the selected hyper-parameters, instead of training each candidate design from scratch.
Revised: Moreover, to accelerate the search process, F ALCON explores an unknown design by fine-tuning a pretrained model for several epochs based on the selected hyper-parameters, instead of training each candidate design from scratch.

Original: The results are summarized in Table 13.
Revised: The results are summarized in Table 13.

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: For the architecture design space.
Revised: For the architecture design space.

Original: For ENAS and DARTS, the learning rate is 0.01, and the maximum training epoch is 300.
Revised: For ENAS and DARTS, the learning rate is 0.01, and the maximum training epoch is 300.

Original: We repeat each experiment three times and summarize the results in Table 14.
Revised: We repeat each experiment three times and summarize the results in Table 14.

Original: [Table]
Revised: [Table]

Original: D S ENSITIVITY A NALYSIS
Revised: D S ENSITIVITY A NALYSIS

Original: We analyze the sensitivity of the hyper-parameters in the search strategy of F ALCON , using a node classification task, CiteSeer, and a graph classification task, Tox21-AhR. Specifically, we study the influence of the number of random starting nodes and the candidate scale resulting from an explored node ( i.e., how many hop neighbors of an explored node are to be included in the candidate set).
Revised: We analyze the sensitivity of the hyper-parameters in the search strategy of F ALCON , using a node classification task, CiteSeer, and a graph classification task, Tox21-AhR. Specifically, we study the influence of the number of random starting nodes and the candidate scale resulting from an explored node ( i.e., how many hop neighbors of an explored node are to be included in the candidate set).

Original: As shown in Figure 7, we find that F ALCON outperforms the best AutoML baselines with a large range of hyper-parameters.
Revised: As shown in Figure 7, we find that F ALCON outperforms the best AutoML baselines with a large range of hyper-parameters.

Original: Specifically, 73% and 97% hyper-parameter combinations of F ALCON rank best among the baselines in CiteSeer and Tox21-AhR, respectively.
Revised: Specifically, 73% and 97% hyper-parameter combinations of F ALCON rank best among the baselines in CiteSeer and Tox21-AhR, respectively.

Original: Moreover, we discover an interesting insight about the size of receptive field , i.e., the number of design candidates, during the search process of F ALCON .
Revised: Moreover, we discover an interesting insight about the size of receptive field , i.e., the number of design candidates, during the search process of F ALCON .

Original: According to the construction of design subgraph, the receptive field size is O ( rh d ) , where r is the number of random start nodes, h is the number of neighbor hops, and d is the average node degree.
Revised: According to the construction of design subgraph, the receptive field size is O ( rh d ) , where r is the number of random start nodes, h is the number of neighbor hops, and d is the average node degree.

Original: We find that the performance of design searched by F ALCON increases with the receptive field’s size until it reaches a certain scale.
Revised: We find that the performance of design searched by F ALCON increases with the receptive field’s size until it reaches a certain scale.

Original: Such patterns have been widely observed in multiple datasets.
Revised: Such patterns have been widely observed in multiple datasets.

Original: While the receptive field on the design subgraph should contain sufficient candidates for sampling good ones, it should also prune inferior design space, which doesn’t provide insights on navigating the best-performing designs.
Revised: While the receptive field on the design subgraph should contain sufficient candidates for sampling good ones, it should also prune inferior design space, which doesn’t provide insights on navigating the best-performing designs.

Original: Thus, the size of the receptive field may be a crucial factor influencing the search quality of F ALCON .
Revised: Thus, the size of the receptive field may be a crucial factor influencing the search quality of F ALCON .