E FFICIENT G RAPH N EURAL A RCHITECTURE S EARCH
Anonymous authors
Paper under double-blind review
A BSTRACT
Recently, graph neural networks (GNN) have been demonstrated effective in various graph-based tasks.
To obtain state-of-the-art (SOAT) data-speciﬁc GNN architectures, researchers turn to the neural architecture search (NAS) methods.
However, it remains to be a challenging problem to conduct efﬁcient architecture search for GNN.
In this work, we present a novel framework for Efﬁcient GrAph Neural architecture search (EGAN).
By designing a novel and expressive search space, an efﬁcient one-shot NAS method based on stochastic relaxation and natural gradient is proposed.
Further, to enable architecture search in large graphs, a transfer learning paradigm is designed.
Extensive experiments, including node-level and graph-level tasks, are conducted. The results show that the proposed EGAN can obtain SOTA data-speciﬁc architectures, and reduce the search cost by two orders of magnitude compared to existing NAS baselines.
I NTRODUCTION
Recent years have witnessed the success of graph neural networks (GNN) (Gori et al., 2005;
Battaglia et al., 2018) in various graph-based tasks, e.g., recommendation (Ying et al., 2018a), fraud detection (Liu et al., 2019b), chemistry (Gilmer et al., 2017).
To adapt to different graph-based tasks, various GNN models, e.g., GCN(Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2018), or
GIN (Xu et al., 2019), have been designed in the past ﬁve years.
Most existing GNN models follow a neighborhood aggregation (or message passing ) schema (Gilmer et al., 2017), as shown in the left part of Figure 1, which is that the representation of a node in a graph is learned by iteratively aggregating the features of its neighbors.
Despite the broad applications of GNN models, researchers have to take tedious efforts to design proper GNN architectures given different tasks by imposing different relational biases (Battaglia et al., 2018).
As pointed out by Battaglia et al.
(2018), the GNN architectures can support one form of combinatorial generalization given different tasks, i.e., graphs.
Then a natural and interesting question can be asked: Can we automatically design state-ofthe-art (SOTA) GNN architectures for graph-based tasks?
A straightforward solution is to adopt the neural architecture search (NAS) approaches, which have shown promising results in automatically designing architectures for convolutional neural networks (CNN) (Zoph & Le, 2017; Pham et al., 2018;
Liu et al., 2019a; Tan & Le, 2019; You et al., 2020).
However, it is nontrivial to adopt NAS to GNN.
The ﬁrst challenge is to deﬁne the search space.
A dummy search space can include all model parameters, e.g., aggregation functions, hidden embedding size, etc., related to the message passing framework (Eq.
and it leads to a combinatorial optimization problem in a large discrete space, which is challenging for any search algorithm.
The second challenge is to design an effective and efﬁcient search algorithm.
In the literature, reinforcement learning (RL) based and evolutionary based algorithms have been explored for GNN architecture search (Gao et al., 2020;
Zhou et al., 2019; Lai et al., 2020; Nunes & Pappa, 2020).
However, they are inherently computationally expensive due to the stand-alone training manner.
In the NAS literature, by adopting the weight sharing strategy, one-shot methods NAS methods are in orders more efﬁcient than RL based ones (Pham et al., 2018;
Liu et al., 2019a; Xie et al., 2019;
Guo et al., 2019).
However, the one-shot methods cannot be directly applied to the aforementioned dummy search space, since it remains unknown how to search for some model parameters like hidden embedding size by weight sharing strategy.
Therefore, it is a challenging problem to conduct effective and efﬁcient architecture search for GNN.
In this work, we propose a novel framework, called EGAN (Efﬁcient GrAph Neural architecture search), to automatically design SOTA GNN architectures.
Motivated by two well-established works (Xu et al., 2019; Garg et al., 2020) that the expressive capabilities of GNN models highly rely on the properties of the aggregation functions, a novel search space consisting of node and layer aggregators is designed, which can emulate many popular GNN models.
Then by representing the search space as a directed acyclic graph (DAG) (Figure 1(c)), we design a one-shot framework by using the stochastic relaxation and natural gradient method which can optimize the architecture selection and model parameters in a differentiable manner.
To enable architecture search in large graphs, we further design a transfer learning paradigm, which ﬁrstly constructs a proxy graph out of the large graph by keeping the properties and then searches for GNN architectures in the proxy graph, ﬁnally transfers the searched architecture to the large graph.
To demonstrate the effectiveness and efﬁciency of the proposed framework, we apply EGAN to various tasks, from node-level to graph-level ones.
The experimental results on ten different datasets show that EGAN can obtain SOTA data-speciﬁc architectures for different tasks, and at the same time, reduce the search cost by two orders of magnitude. Moreover, the transfer learning paradigm, for the best of our knowledge, is the ﬁrst framework to conduct architecture search in large graphs.
Notations.
Let G =
( V , E ) be a simple graph with node features X ∈ R N × d , where V and E represent the node and edge sets, respectively.
N represents the number of nodes and d is the dimension of node features.
We use N ( v ) to represent the ﬁrst-order neighbors of a node v in G , i.e., N ( v ) = { u ∈ V| ( v, u ) ∈ E} .
In the literature, we also create a new set (cid:101) N ( v ) is the neighbor set including itself, i.e., (cid:101) N ( v ) = { v } ∪ { u ∈ V| ( v, u ) ∈ E} .
R ELATED W ORKS
GNN was ﬁrst proposed in (Gori et al., 2005) and in the past ﬁve years many different variants (Kipf & Welling, 2016; Hamilton et al., 2017; Veliˇckovi´c et al., 2018; Gao et al., 2018; Battaglia et al., 2018; Xu et al., 2019; 2018;
Liu et al., 2019b; Abu-El-Haija et al., 2019; Wu et al., 2019; Zeng et al., 2020; Zhao & Akoglu, 2020; Rong et al., 2020) have been designed, and they rely on a neighborhood aggregation (or message passing ) schema (Gilmer et al., 2017), which learns the representation of a given node by iteratively aggregating the hidden features (“message”) of its neighbors. By stacking GNN layers, multi-hop, the so-called higher-order , neighbors, can be utilized to enhance the performance.
Besides, in Xu et al.
; Chen et al.
, the design of residual networks (He et al., 2016a;b) are incorporated into existing message passing GNN models.
Battaglia et al. (2018) pointed out that the GNN architectures can provide one form of combinatorial generalization for graph-based tasks, and Xu et al.
; Garg et al.
(2020) further show that the expressive capability of existing GNN models is upper bounded by the well-known Weisfeiler-Lehman (WL) test. It will be an interesting question to explore GNN architectures with better combinatorial generalization, thus neural architecture search (NAS) can be a worthwhile approach for this consideration. NAS (Baker et al., 2017; Zoph & Le, 2017; Elsken et al., 2018) aims to automatically ﬁnd SOTA architectures beyond human-designed ones, which have shown promising results in architecture design for CNN and Recurrent Neural Network (RNN) (Liu et al., 2019a; Zoph et al., 2018; Tan & Le, 2019).
In the literature, representative NAS approaches can be roughly categorized into two groups according to the training paradigms (Bender et al., 2018): i.e. the stand-alone and one-shot ones. The former ones tend to obtain the SOTA architecture from training thousands of architectures from scratch, including reinforcement learning (RL) based (Baker et al., 2017; Zoph & Le, 2017) and evolutionary-based ones (Real et al., 2019), while the latter ones tend to train a supernet containing thousands of architectures based on the weight sharing strategy, and then extract a submodel as the
SOTA architecture at the end of the search phase (Pham et al., 2018; Bender et al., 2018;
Liu et al., 2019a; Xie et al., 2019).
The difference of training paradigms leads to that one-shot NAS methods tend to be more efﬁcient in orders of magnitude than the RL based ones.
Recently, there are several works on architecture search for GNN, e.g., RL based ones (Gao et al., 2020;
Zhou et al., 2019; Lai et al., 2020), and evolutionary-based ones (Nunes & Pappa, 2020; Jiang & Balaprakash, 2020), thus all existing works are computational expensive. Besides, Franceschi et al.
(2019) proposes to jointly learn the edge probabilities and the parameters of GCN given a graph, thus orthogonal to our work. In this work, for the best of our knowledge, we are the ﬁrst to design a one-shot NAS method for GNN architecture search, which is thus, in nature, more efﬁcient than existing NAS methods for GNN.
T HE P ROPOSED F RAMEWORK 3. T HE DESIGN OF SEARCH SPACE
As introduced in Section 2, most existing GNN architectures rely on a message passing framework (Gilmer et al., 2017), which constitutes the backbone of the designed search space in this work.
To be speciﬁc, a K -layer GNN can be written as follows: the l -th layer ( l = 1 , · · · , K ) updates h v for each node v by aggregating its neighborhood as
[Equation]
where h ( l ) v ∈ R d l represents the hidden features of a node v learned by the l -th layer, and d l is the corresponding dimension.
W ( l ) is a trainable weight matrix shared by all nodes in the graph, and σ is a non-linear activation function, e.g., a sigmoid or ReLU.
Φ n is the key component, i.e., a pre-deﬁned aggregation function, which varies across different GNN models.
Thus a dummy search space is to include all model parameters in Eq.
However, it leads to a very large search space, making the search process very expensive.
In this work, motivated by two wellestablished works (Xu et al., 2019; Garg et al., 2020), which show that the expressive capabilities of GNN models highly rely on the properties of aggregation functions, we propose to search for different aggregation functions by simplifying the dummy search space.
For other parameters, we do simple tuning in the re-training stage, which is also a standard practice in existing NAS methods (Liu et al., 2019a; Xie et al., 2019).
Then the ﬁrst component of the proposed search space is the node aggregators , which consists of existing GNN models.
To improve the expressive capability, we add the other component, layer aggregators , to combine the outputs of node aggregator in all layers, which have been demonstrated effectiveness in JK-Network (Xu et al., 2018).
Then we introduce the proposed search space, as shown in Figure 1(c), in the following:
• Node aggregators :
We choose 12 node aggregators based on popular GNN models, and they are presented in Table 7 in Appendix A.1.
The node aggregator set is denoted by O n .
• Layer aggregators :
We choose 3 layer aggregators as shown in Table 7 in Appendix A.1.
Besides, we have two more operations, IDENTITY and ZERO , related to skip-connections.
Instead of requiring skip-connections between all intermediate layers and the ﬁnal layer in JK-Network, in this work, we generalize this option by proposing to search for the existence of skip-connections between each intermediate layer and the last layer.
To connect, we choose IDENTITY , and ZERO otherwise.
The layer aggregator set is denoted by O l and the skip operation set by O s .
The node aggregators focus on how to aggregate the neighborhood features, while the layer aggregators focus on the range of neighborhood to use. Then we can see that the designed search space can emulate extensive human-designed GNN architectures.
To further inject relational bias from existing GNN architectures, when searching for the skip-connections for each GNN layer, we add one more constraint that the output of the node aggregator in the last layer should always be used as the input of the layer aggregator,, thus for a K -layer GNN architecture, we need to search K −IDENTITY or ZERO for the skip-connection options.
3. D IFFERENTIABLE ARCHITECTURE SEARCH
Following existing NAS works (Liu et al., 2019a; Xie et al., 2019), we represent the search space by a directed acyclic graph (DAG), as shown in Figure 1(c), where nodes represent embeddings, and edges represent operations between the two end nodes.
Then the intermediate nodes are
[Equation]
(2) where ˜ O i,j is the selected operation at edge ( i, j ) .
In our work, each edge corresponds to an operation in the search space, and we represent it with a distribution p α ( Z ) , which generates the one-hot random variable Z i,j multiplied by the operation edge O i,j in the DAG. Then the intermediate nodes in each child graph are represented by
[Equation]
Note that in our work, not all previous nodes are used when computing for each intermediate nodes, and for each node aggregator, the input is from the previous one, and for the layer aggregators, the input are from outputs of all node aggregators.
Following the setting in Zoph & Le (2017) and Gao et al.
(2020), the objective of the framework is
[Equation]
where R ( Z ) represents the reward, which is deﬁned by training loss L W ( Z ) in our framework.
W represents the model parameters.
In the GNN literature, node-level or graph-level classiﬁcation tasks are commonly used, thus the cross-entropy loss is chosen.
In Zoph & Le (2017) and Gao et al.
(2020), Eq.
(4) could be optimized by policy gradient methods (Williams, 1992), however, in this work, to make use of the differentiable nature of L θ ( Z ) , and design a differentiable search method to optimize Eq.
To be speciﬁc, we use the Gumbel softmax (Maddison et al., 2017; Xie et al., 2019; Noy et al., 2020) to relax the discrete architecture distribution to be continuous and differentiable with reparameterization trick: [Equation]
[Equation]
where Z i,j is the softened one-hot random variable for operation selection at edge ( i, j ) , and G ki,j = − log( − log( U ki,j )) is the k th Gumbel random variable, U ki,j is a uniform random variable.
α i,j is the architecture parameter.
λ is the temperature of the softmax, which is steadily annealed to be close to zero (Xie et al., 2019; Noy et al., 2020).
In Maddison et al.
(2017), it is proved that p (lim λ → 0 Z ki,j = 1) =
α ki,j / ( (cid:80) nl =
α li,j ) , which means that this relaxation is unbiased once converged.
Then we can use gradient descent methods to optimize the operation parameters and model parameters together in an end-to-end manner.
The gradients are given in Appendix A.2.
To improve the search efﬁciency, we further design an adaptive stochastic natural gradient method to update the architecture parameters in an end-to-end manner following Akimoto et al.
To be speciﬁc, the update of α at the m th iteration is given as:
[Equation]
where ρ is the step size.
H is the Fisher matrix, which can be computed as H =
E p
α m
[¯ p α ( Z )¯ p α ( Z ) T ] with ¯ p
α
( Z ) := ∇ log p
α
( Z ) .
After the searching process terminates, we derive the ﬁnal architecture by retaining the edge with the largest weight, which is the same as existing DARTS (Liu et al., 2019a) and SNAS (Xie et al., 2019).
To robust the ﬁnal results, the search process is executed 5 times with different random seeds, thus 5 architectures are obtained at the end of the search phase.
Then the 5 architectures are re-trained from scratch with some hyper-parameters tunning on the validation set, and the one with the best validation accuracy is returned as the ﬁnal architecture.
3. T RANSFER LEARNING PARADIGM
For large graphs, since NAS approaches can be extremely expensive in terms of search time and GPU memory due to the fact that K -hop neighbors have to be accessed in a K -layer GNN, whose numbers exponentially increase.
Therefore, it tends to be infeasible to directly search for architectures in large graphs.
Motivated by transferring searched blocks and cells in CNN architectures from small to large data sets in the NAS literature (Zoph et al., 2018; Tan & Le, 2019), we propose to address the above problem by transfer learning (Pan & Yang, 2009).
The core idea of the transferable architecture search is to ﬁnd a small proxy graph G proxy (the source), then search in the proxy graph, ﬁnally tune the searched architecture { α n , α s , α l } in the large graph G (the target).
However, in order to make the architecture transfer feasible, we need to make the proxy graph sharing the same property distribution with the original graph (Pan & Yang, 2009).
Since the properties vary across different graphs, it is not suitable to transfer across different datasets, like that from CIFAR-10 to ImageNet for image classiﬁcation (Zoph et al., 2018).
Thus, we propose to sample a smaller graph from the original one, and then apply the transfer paradigm.
Many distribution-preserving sampling schema have been proposed in the literature, e.g., random sampling by node or edge, or sampling by PageRank, proposed in an established work (Leskovec & Faloutsos, 2006).
In this work, we adopt the Random PageRank Node (RPN) sampling method in (Leskovec & Faloutsos, 2006), which is empirically demonstrated to be able to preserve the properties with a sample size not less than 15% of the original graph. [Table]
3. C OMPARISONS WITH EXISTING NAS METHODS FOR GNN
In this section
[Table]
we compare EGAN and existing NAS methods for GNN regarding search space, search algorithm
[Table]
and running in large graphs
[Table]
as shown in Table 1. We emphasize the advantages of EGAN in the following: • In terms of the search space, by designing the layer aggregators, EGAN can emulate more GNN models than existing methods, thus being more expressive.
Moreover, by only focusing on the “aggregation function”, the search space is smaller than previous methods, which include other model parameters into the search space. In this sense, it can accelerate the search process. • In terms of the search algorithm, the one-shot nature of EGAN makes it much more efﬁcient than stand-alone methods.
• The transfer paradigm of EGAN alleviates the problem of exponential increasing of multi-hop neighbors when stacking GNN layers in large graphs, thus making it feasible to enable architecture search in large graphs. Therefore, the advantages of EGAN over existing NAS methods are evident, especially efﬁciency. In the experiments (Section 4.2.2 and 4.3), we can see that EGAN can reduce the search cost in two orders of magnitude compared to NAS baselines, which demonstrates the superiority of the proposed framework.  E XPERIMENTS
In this section, we conduct extensive experiments to demonstrate the effectiveness and efﬁciency of the proposed EGAN, including node-level and graph-level tasks.
4. S ETUP
Datasets.
For node-level tasks, we have three settings: transductive, inductive, and transfer.
The task is node classiﬁcation on 8 datasets, which are given in Appendix A.3.1.
For graph-level tasks, the task is whole graph classiﬁcation on 2 datasets, which are given in Appendix A.4.
Baselines In general, we have two types of baselines: human-designed GNN models, and NAS methods.
Details of baselines are given in Appendix A.3.2.
Note that for NAS baselines in Table 1, we only use GraphNAS (Gao et al., 2020) and its variant using weight sharing (GraphNAS-WS).
The search spaces of Auto-GNN (Zhou et al., 2019) and Nunes & Pappa (2020) are the same as GraphNAS, while their codes are not available.
Jiang & Balaprakash (2020) is a concurrent work without public codes when we are preparing for this submission, and focusing more on global pooling methods for whole graph representation learning, which we leave it for future work.
For Policy-GNN (Lai et al., 2020), they work on searching for different numbers of layers per node in a selected GNN base model, i.e., GCN or GAT, thus it can be an orthogonal work to EGAN.
al., 2020), where the data split ratio is the same to our experiments
[Table]
4. P ERFORMANCE COMPARISON 4.2. P ERFORMANCE COMPARISONS IN T RANSDUCTIVE AND I NDUCTIVE T ASKS
From Table 2, we can see that EGAN consistently obtain better or close performance compared to all baselines, which demonstrates the effectiveness of the proposed framework.
In other words, with EGAN, we can obtain SOTA data-speciﬁc GNN architectures.
When comparing EGAN with GraphNAS methods, the performance gain is evident.
We attribute this to the superiority of the expressive search space and the search algorithm.
We visualize the searched architectures in Figure 2, and we can see that the searched architectures vary per dataset. More ﬁgures can be checked in Figure 6 and 7 in Appendix A.5.3.
[Table]
4.2. P ERFORMANCE COMPARISONS IN T RANSFER T ASKS
In terms of baselines
[Table]
we only report the results of those, which are able to, or reported to, run in the two large graphs, i
[Table]
e., Reddit and Arxiv. Detailed explanations are given in Appendix A.3.
[Table]
The results are shown in Table 3, from which we can see that the improvement of efﬁciency is in two orders of magnitude, from 3 GPU days 1 to less than a minute, which demonstrates the superiority of the proposed transfer learning paradigm for architecture search in large graphs. Moreover, the performance of the transferred architectures is better or close to those of the baselines, which demonstrates the effectiveness of the transfer learning paradigm
[Table]
4.2. G RAPH - LEVEL TASKS
The results of the graph-level task are shown in Table 4, and we can see that the performance trending is similar to node-level tasks.
The searched architectures are shown in Figure 7 in Appendix A.5.3, which also shows that they are data speciﬁc. Note that the global pooling method, or the readout function, in the whole graph learning can also be incorporated into the search space of EGAN, thus it can also be learned.
We leave this for future work.
Taking into consideration the results of all tasks
[Table]
the effectiveness of the proposed EGAN can be demonstrated.
tasks.
We use 10-fold cross-validation accuracy as the evaluation metrics and show the mean test accuracy (with standard deviation) on these
[Table]
4. S EARCH EFFICIENCY
In this section, we conduct some experiments to show the superiority of efﬁciency of EGAN over NAS baselines. And for simplicity, we only use the four commonly used datasets in node-level tasks, i.e., Cora, CiteSeer, PubMed, and PPI. First,, we record the search time of run each method once, for which the epochs of EGAN and NAS baselines are all set to 200.
The results are given in Table 5, from which we can see that the search time cost of EGAN is two orders of magnitude smaller than those of NAS baselines.
Second, we show the trending of the test accuracy w.r.t.
the running time of different methods, and the result of Cora is shown in Figure 3, from which we can observe that EGAN can obtain architectures with better performance more quickly than NAS baselines during the search phase.
More ﬁgures are shown in Figure 5 in Appendix A.5.2.
Taking these results into consideration, the efﬁciency of EGAN is signiﬁcant, which is mainly attributed to the one-shot training paradigm as introduced in Section 3.
[Table]
4. A BLATION STUDY
In this section, we present two ablation studies on EGAN. First, we show the importance of the layer aggregators in the designed search space by running
EGAN in the search space without the layer aggregators, and the results are shown in Table 6, from which we can see that the performance consistently drops on all datasets except Computer when removing the layer aggregators.
This observation aligns with the results in JK-Network(Xu et al., 2018) that the performance of GNN models can be improved by adding an extra layer. Second, we show the inﬂuence of K , i.e., the number of layers of GNN, in the search space, for which we conduct experiments with EGAN by varying layer K ∈ { 1 , 2 , 3 , 4 , 5 , 6 } and show the test accuracy in Figure 4.
The results suggest that along with the increment of layers, the test accuracy may decrease.
Considering the computational resources, 3-layer architecture is a good choice for these datasets.
[Table]
C ONCLUSION AND F UTURE WORK
In this paper, we propose EGAN, an effective and efﬁcient framework for graph neural architecture search. BY designing a novel search space, which can emulate various human-designed GNN models, we propose a one-shot NAS framework by stochastic relaxation and natural gradient.
Further, to enable architecture search in large graphs, we design a transfer learning paradigm, which ﬁrstly runs EGAN in a proxy graph and then transfers the searched architecture to the large one.
We conduct extensive experiments, including node-level and graph-level tasks, which demonstrates the effectiveness and efﬁciency of the proposed EGAN compared to various baselines.
Based on this work, we show that NAS approaches can obtain data-speciﬁc GNN architectures, which supports one form of combinatorial generalization for GNN models.
For future work, we will explore more aspects regarding the combinatorial generalization of GNN models beyond the aggregation functions, like the construction of the graph, or the number of layers as done in PolicyGNN (Lai et al., 2020), as introduced in Battaglia et al.
A A PPENDIX A. D ETAILS OF N ODE A GGREGATORS
As introduced in Section 3.
[Table]
we have 12 types of node aggregators, which are based on wellknown existing GNN models: GCN Kipf & Welling (2016), GraphSAGE Hamilton et al.
GAT Veliˇckovi´c et al.
GIN Xu et al.
(2019), and GeniePath Liu et al.
(2019b).
Here we give key explanations to these node aggregators in Table 8.
For more details, we refer readers to the original papers.
[Table]
[Table]
A. G RADIENTS OF L
Here we give the gradients of L w ( Z ) w.r.t.
the hidden embeddings, model parameters, and the architecture parameters, which is in the following:
[Equation]
For full derivation, we refer readers to the appendix of Xie et al.
A. E XPERIMENT S ETUP OF NODE - LEVEL TASKS A.3. D ATASETS
Transductive Setting Only a subset of nodes in one graph are used as training data, and other nodes are used as validation and test data.
For this setting, we use three benchmark [Table]
PubMed of
Besides the three benchmark datasets, we use two more datasets: Coauthor CS and Amazon Computers, provided by (Shchur et al., 2018).
Coauthor CS is a co-authorship graph where nodes are authors which are connected by an edge if they co-author a paper.
Given paper keywords for each author’s paper as node features, the task is to map each author to its most active ﬁeld of study.
Amazon Computers is segments of the Amazon co-purchase graph where nodes represent goods which are linked by an edge if these goods are frequently bought together.
Node features encode product reviews as bag-of-word feature vectors, and class labels are given by product category.
For all 5 datasets, We split the nodes in all graphs into 60%, 20%, 20% for training, validation, and test.
For the transductive task, we use the classiﬁcation accuracy as the evaluation metric.
Inductive Setting In this task, we use a number of graphs as training data, and other completely unseen graphs as validation/test data.
For this setting, we use the PPI dataset, provided by (Hamilton et al., 2017), on which the task is to classify protein functions.
PPI consists of 24 graphs, with each corresponding to a human tissue.
Each node has positional gene sets, motif gene sets and immunological signatures as features and gene ontology sets as labels.
20 graphs are used for training, 2 graphs are used for validation and the rest for testing, respectively.
For the inductive task, we use Micro-F1 as the evaluation metric.
Transfer Setting In this task, we use two datasets, Reddit and Arxiv, which are two orders of magnitude larger than Cora and CiterSeer in number of nodes, as shown in Table 9.
The Reddit dataset is provided by Hamilton et al.
and the task is to predict the community to which different
Reddit posts belong.
Reddit is an online discussion forum where users comment on different topics.
Each node represents a post, and each edge represents a link between two posts, when they are commented by the same user.
The dataset contains word vectors as node features.
The graph is constructed from Reddit posts made in the month of September 2014, and we follow the same settings in the original paper (Hamilton et al., 2017), which uses the ﬁrst 20 days for training and the remaining days for test (with 30% used for validation).
The Arxiv dataset is constructed based on the citation network between all papers, and we use the speciﬁc version of ogbn-arxiv , provided by a recent open graph benchmark (OGB) project (Hu et al., 2020), where the task is to predict the 40 subject areas of Arxiv CS papers, e.g., cs.
AI, cs.LG.
Each node (paper) has a 128-dimensional feature vector obtains by averaging the embedding of words in its title and abstract.
And all papers are also associated with the year that the corresponding paper was published.
The dataset is split by time.
To be speciﬁc, papers published before 2017 are used as training data, while those in 2018 and 2019 are used respectively, as validation and test set.
For more details, we refer readers to Hu et al.
A.3. C OMPARED M ETHODS
We compare EGAN with two groups of state-of-the-art methods: human-designed GNN architectures and NAS methods for GNN.
Human-designed GNNs.
We use the following popular GNN architectures:
• GCN (Kipf & Welling, 2016) proposes a sum aggregator normalized by the degrees of nodes.
• GraphSAGE (Hamilton et al., 2017) proposes scalable graph neural network with different aggregators:
Mean, Sum, Max-Pool, LSTM.
• GAT (Veliˇckovi´c et al., 2018) proposes the attention aggregators, and it has different variants according to the attention functions: GAT, GAT-SYS, GAT-LINEAR, GAT-COS, GATGENERALIZED-LINEAR.
The detail of these attention functions are given in (Gao et al., 2020).
GIN (Xu et al., 2019) proposes to use Multi-layern Perceptron (MLP) as aggregators.
• LGCN (Gao et al., 2018) proposes to automatically select topK neighbors for each node, and use the 1-D regular convolutional operation as the aggregator.
• GeniePath
(Liu et al., 2019b) proposes a composition of attention and LSTM-style aggregators, which can learn adaptive neighborhood for each node.
• Geom-GCN
(Pei et al., 2020) propose a geometric bi-level aggregation schema over structureaware neighbors in an continuous space and neighbors by adjacency matrix.
For models with variants, like different aggregators in GraphSAGE or different attention functions in GAT, we report the best performance across the variants.
Besides, we extend the idea of JKNetwork (Xu et al., 2018) in all models except for LGCN, and obtain 5 more variants: GCN-JK,
GraphSAGE-JK, GAT-JK, GIN-JK, GeniePath-JK, which add an extra layer.
In the experiments, we only report the best performance of these 5 models and their JK versions, which are denoted by their original names, as shown in Table 2, and 4, respectively.
For LGCN, we use the code released by the authors 2 .
For other baselines, we use the popular open source library Pytorch Geometric (PyG) (Fey & Lenssen, 2019) 3 (Version: 1.6.0), which implements various GNN models.
For all baselines, we train it from scratch with the obtained best hyper-parameters on validation datasets, and get the test performance.
We repeat this process fortimes, and report the ﬁnal mean accuracy with standard deviation.
NAS methods for GNN.
We consider the following methods:
• Random search (denoted as “Random”) is a simple baseline in NAS, which uniformly randomly samples architectures from the search space, and keeps track of the optimal architecture during the search process.
• Bayesian optimization 4 (denoted as “Bayesian”)
(Bergstra et al., 2011) is a popular sequential model-based global optimization method for hyper-parameter optimization, which uses treestructured Parzen estimator as the metric for expected improvement.
• GraphNAS 5 (Gao et al., 2020), a NAS method for searching GNN architecture, which is based on reinforcement learning (Zoph & Le, 2017).
Random and Bayesian are searching on the designed search space of EGAN, where a GNN architecture is sampled from the search space, and trained till convergence to obtain the validation performance.
200 models are sampled in total and the architecture with the best validation performance is trained from scratch, and do some hyper-parameters tuning on the validation dataset
and obtain the test performance.
For GraphNAS, we set the epoch of training the RL-based controller to 200, and in each epoch, a GNN architecture is sampled, and trained for enough epochs ( 600 ∼depending on datasets), update the parameters of RL-based controller.
In the end, we samplearchitectures and collect the top 5 architectures that achieve the best validation accuracy.
Then the best architecture is trained from scratch.
Again, we do some hyper-parameters tuning based on the validation dataset and report the best test performance.
Note that we repeat the re-training of the architecture for ﬁve times, and report the ﬁnal mean accuracy with standard deviation.
Note that for human-designed GNN models and NAS methods, for fair comparison and good balance between efﬁciency and performance, we choose set the number of GNN layers to be 3, which is an empirically good choice in the literature (Veliˇckovi´c et al., 2018; Liu et al., 2019b).
A.3. I MPLEMENTATION DETAILS OF EGAN
Our experiments are running with Pytorch (version 1.6.0) on a GPU 2080Ti (memory: 12GB, cuda version: 10.2).
We implement EGAN on top of the building codes provided by PyG (version 1.6.0) and SNAS 6 .
For all tasks, we run the search process for 5 times with different random seeds, and retrieve top-1 architecture each time. By collecting the best architecture out of the 5 toparchitectures on validation datasets, we repeat 5 times the process in re-training the best one, ﬁnetuning hyper-parameters on validation data, and reporting the test performance.
Again, the ﬁnal mean accuracy with standard deviations are reported. In training stage, we set the search epoch to 400, and the learning rate to 0.005, L2 norm to 0.0005, dropout rate to 0.5. In ﬁnetuning stage, each architecture is trained from scratch with 600 epochs.
A.3. B ASELINES IN T RANSFER LEARNING
For the transfer task, we use two datasets: Reddit and Arxiv.
For Reddit, we follow the same settings of GraphSAGE in the original paper (Hamilton et al., 2017), except that we use a 3-layer
GNN as backbone to search, while GraphSAGE use 2-layer.
Since the performance of Reddit is only reported in GraphSAGE (Hamilton et al., 2017) and JK-Network (Xu et al., 2018), we use them as human-designed architectures.
Note that the authors of JK-Network did not release their implementation, thus we use the implementation by the PyG framework. To keep it consistent, we also use the implementation for GraphSAGE by the PyG framework. For Arxiv, we follow the same setting in OGB project(Hu et al., 2020), where only two human-designed architectures, GCN and GraphSAGE, are tested, thus we use these two as human-designed architectures.
For NAS baselines, we use the same NAS approaches in transductive and inductive tasks: Random, Bayesian, GraphNAS, and GraphNAS-WS, and run them directly in the original graphs, i.e., Reddit and Arxiv.
However, since the code of GraphNAS and GraphNAS-WS crashed due to out of memory error, we only report the performance of Random and Bayesian. Besides, we report the search cost in terms of GPU hours to compare the efﬁciency of different methods. A. E XPERIMENT S ETUP OF G RAPH - LEVEL TASKS A.4. D ATASETS
[Table]
PROTEINS
[Table]
In this part, we evaluate EGAN on graph classiﬁcation task two datasets: D&D and PROTEINS datasets, provided in Dobson & Doig (2003). These two datasets are both the protein graphs. In
D&D dataset, nodes represent the amino acids and two nodes are connected iff the distance is less than 6 ˙ A . In PROTEINS dataset, nodes are secondary structure elements and edges represent nodes are in an amino acid or in a close 3D space.
More information are shown in Table 10.
Besides the GNN baselines in node-level tasks, we use three more methods, which use hierarchy pooling to learn whole graph representation given the embeddings of all nodes.
DiffPool (Ying et al., 2018b), SAGPool (Lee et al., 2019) and ASAP (Ranjan et al., 2020) are latest methods based on hierarchical pooling schema, which learn node embeddings with node aggregators and coarsen graphs with pooling aggregators.
The ﬁnal graph embeddings are generated by a readout operation based on the ﬁnal coarsend graph. DiffPool learns a soft assignment matrix for each node with any GNN methods, combine with entropy regularization and link prediction objective, so that the coarsened graph can preserve as much information as possible. SAGPool learns node weights with attention mechanism and keeps the top-k nodes in pooling layer. In ASAP, it learns a soft cluster assignment matrix for each node with self-attention mechanism, and calculates the ﬁtness score for each cluster and select top-k clusters.
For other methods, including GNN models in node-level tasks and NAS methods, to obtain the representation of a whole graph, we use the global sum pooling method at the end of retraining the derived ﬁnal architecture. In this part, we use 10-fold cross-validation accuracy as evaluate metric, and the implementation details are presented in A.3.3.
After ﬁnding the best architecture and ﬁnetuning hyper-parameters, we report the mean accuracy and standard deviations on 10 folds data. In the search stage, we set the search epoch to 100, and the learning rate to 0.01, L 2 norm to 0.0005, dropout rate to 0.5. In the re-training stage, each architecture is trained from scratch withepochs.
[Table] .5. T HE ADVANTAGE OF THE PROPOSED SEARCH SPACE In Section 3.1, we discuss the advantages of search space between EGAN and GraphNAS/AutoGNN.
In this part, we conduct experiments to further show the advantages.
To be speciﬁc, we run GraphNAS over its own and EGAN’s search space, given the same time budget (20 hours), and compare the ﬁnal test accuracy of the searched architectures in Table 11. From Table 11, we can see that despite the simplicity of the search space, EGAN can obtain better or at least close accuracy compared to GraphNAS, which means better architectures can be obtained given the same time budget, thus demonstrating the efﬁcacy of the designed search space.
A.5. T EST ACCURACY DURING THE SEARCH PHASE
In this part, we compare the efﬁciency of EGAN and NAS baselines by showing the test accuracy w.r.t the running time, as shown in Figure 5, from which we can observe that the efﬁciency improvements are in orders of magnitude, which aligns with the experiments in previous one-shot NAS methods, like DARTS (Liu et al., 2019a).
[Figure]
A.5. M ORE S EARCHED ARCHITECTURES