Original: AutoCoG: A Unified Data-Model Co-Search Framework for Graph Neural Networks
Revised: AutoCoG: A Unified Data-Model Co-Search Framework for Graph Neural Networks

Original: Anonymous 1 Anonymous Institution Abstract Neural architecture search (NAS) has demonstrated success in discovering promising ar5 chitectures for vision or language modeling tasks, and it has recently been introduced to6 searching for graph neural networks (GNNs) as well.
Revised: Abstract Neural architecture search (NAS) has demonstrated success in discovering promising architectures for vision or language modeling tasks, and it has recently been introduced to searching for graph neural networks (GNNs) as well.

Original: Despite the preliminary success, GNNs7 struggle in dealing with heterophily or low-homophily graphs where connected nodes may8 have different class labels and dissimilar features.
Revised: Despite the preliminary success, GNNs struggle in dealing with heterophily or low-homophily graphs where connected nodes may have different class labels and dissimilar features.

Original: To this end, we propose co-optimizing9 both the input graph topology and the model’s architecture topology simultaneously.
Revised: To this end, we propose co-optimizing both the input graph topology and the model’s architecture topology simultaneously.

Original: That10 yields AutoCoG , the first unified data-model co-search NAS framework for GNNs.
Revised: That yields AutoCoG , the first unified data-model co-search NAS framework for GNNs.

Original: By11 defining a highly flexible data-model co-search space, AutoCoG is gracefully formulated as12 a principled bi-level optimization that can be end-to-end solved by the differentiable search13 methods.
Revised: By defining a highly flexible data-model co-search space, AutoCoG is gracefully formulated as a principled bi-level optimization that can be end-to-end solved by the differentiable search methods.

Original: Experiments show  AutoCoG achieves gains of up to 4% for Actor, 7.3% on average 14 for Web datasets, 0.17% for CoAuthor-CS, and finally 5.4% for Wikipedia-Photo benchmarks. 15 All codes will be released upon paper acceptance.
Revised: Experiments show that AutoCoG achieves an average performance gain across all datasets of 3.18% over the following best approach and ranks best against all other state-of-the-art methods with an average ranking of 2.5. Code is available at https://github.com/VITA-Group/AutoCoG .

Original: 1 Introduction
Revised: 1 Introduction

Original: Graph neural networks (GNNs) have emerged as promising tools to analyze networked data in18 various real-world scenarios, such as social media Grover and Leskovec (2016) and biochemical19 graph analytics Zitnik and Leskovec (2017).
Revised: Graph neural networks (GNNs) have emerged as promising tools to analyze networked data in various real-world scenarios, such as social media (Grover and Leskovec, 2016) and biochemical graph analytics (Zitnik and Leskovec, 2017).

Original: Specifically, GNNs apply recursive message passing20 to learn the embedding representation of each node via aggregating the representations of its21 neighbors and itself.
Revised: Specifically, GNNs apply recursive message passing to learn the embedding representation of each node via aggregating the representations of its neighbors and itself.

Original: Motivated by the significant success of node embedding learning, plenty22 of GNN variants have been explored for the diverse downstream graph analysis tasks, including23 GCN Kipf and Welling (2016a), GraphSAGE Hamilton et al.
Revised: Motivated by the significant success of node embedding learning, plenty of GNN variants have been explored for the diverse downstream graph analysis tasks, including

Original: (2018), and GCNII Chen et al. (2020a).
Revised: GCN (Kipf and Welling, 2016a), GraphSAGE (Hamilton et al., 2018) and GCNII (Chen et al., 2020a).

Original: 24 However, training GNNs is notoriously challenging, more so when they are train under het25 erophily or noisy graphs, not to mention deep GNNs Chen et al. (2020a).
Revised: However, training GNNs is notoriously challenging, more so when they are train under heterophily or disassociative graphs, not to mention deep GNNs (Chen et al., 2020a).

Original: First, since graphs abstract26 diverse data sources and present tremendous heterogeneity, the success of GNNs is often accompa27 nied by extensive tuning of model architectural hyperparameters to characterize specific graph28 data.
Revised: First, since graphs abstract diverse data sources and present tremendous heterogeneity, the success of GNNs is often accompanied by extensive tuning of model architectural hyperparameters to characterize specific graph data.

Original: For example, it was reported that graph attention networks GAT Veličković et al. (2018) are29 sensitive to the number of attention heads, which has to be carefully searched for the citation30 networks and the protein-protein interaction data, respectively.
Revised: For example, it was reported that graph attention networks (GATs) (Veličković et al., 2018) are sensitive to the number of attention heads, which has to be carefully searched for the citation networks and the protein-protein interaction data, respectively.

Original: Second, in the real world graphs31 often opposites attract which inevitably lead to noisy setting where GNNs tend to suffer from32 overfitting and generalize poorly to the unseen testing data.
Revised: Second, in the real world, graphs often opposites attract which inevitably lead to noisy setting where GNNs tend to suffer from overfitting and generalize poorly to the unseen testing data.

Original: Third, despite the potential of deep33 GNNs in learning the informative high-order neighborhood, the training of deep GNNs is widely34 known to be limited by the issues of over-smoothing, gradient vanishing, and over-squashing Chen 35 et al. (2020a).
Revised: Third, despite the potential of deep GNNs in learning the informative high-order neighborhood, the training of deep GNNs is widely known to be limited by the issues of over-smoothing, gradient vanishing, and over-squashing (Chen et al., 2020a).

Original: Recently, the automated graph neural architecture search (NAS), graph augmentation tricks, and37 deeper architectures have been independently proposed to tackle the above GNN training challenges38 partially.
Revised: Recently, the automated graph neural architecture search (NAS), graph augmentation tricks, and deeper architectures have been independently proposed to tackle the above GNN training challenges partially.

Original: Expressly, most of the existing automated efforts are limited to neural architecture tuning,39 while graph augmentation is often overlooked and untouched despite often being effective to gain40 performance Li and King (2020); Zhou et al. (2019a).
Revised: Expressly, most of the existing automated efforts are limited to neural architecture tuning, while graph augmentation is often overlooked and untouched despite often being effective to gain performance (Li and King, 2020; Zhou et al., 2019a).

Original: This is primarily because changes to the41 existing graph structure can have a cascading effect on the process of information aggregation,42 which adds a new layer of complexity above the already complex architecture tuning problem.
Revised: This is primarily because changes to the existing graph structure can have a cascading effect on the process of information aggregation, which adds a new layer of complexity above the already complex architecture tuning problem.

Original: Additionally, existing GNN NAS works are known to scale poorly in deeper architectures.
Revised: Additionally, existing GNN NAS works are known to scale poorly in deeper architectures.

Original: This44 is primarily due the exploding search space which makes training unstable.
Revised: This is primarily due the exploding search space which makes training unstable.

Original: Previous efforts45 have limited themselves in searching the shallow GNNs with less than 3 layers.
Revised: Previous efforts have limited themselves in searching the shallow GNNs with less than 3 layers.

Original: Finally, Figure(1)46 illustrates circumstances where the aggregation mechanism fails due to unfavorable graph topology[Figure] thus, it remains a daunting task to optimize the design philosophy for GNNs comprehensively.[Figure]
Revised: Finally, Figure(1) illustrates circumstances where the aggregation mechanism fails due to unfavorable graph topology thus, it remains a daunting task to optimize the design philosophy for GNNs comprehensively.

Original: [Figure]

Original: To bridge the gaps, we propose AutoCoG ,

Original: [Figure]
Revised: [Figure]

Original: the first NAS framework towards unified

Original: data-

Original: [Figure]

Original: model co -search for GNNs to specifically tame[Figure] the problem training under heterophily con-
Revised: To bridge the gaps, we propose AutoCoG , the first NAS framework towards unified datamodel co-search for GNNs to specifically tame the problem training under heterophily condition.

Original: [Figure]

Original: dition.

Original: Besides automatically optimizing the
Revised: Besides automatically optimizing the

Original: [Figure] GNN model architecture, we propose to simul- [Figure] taneously optimize the input graph topology, 
Revised: GNN model architecture, we propose to simultaneously optimize the input graph topology, via progressively growing and pruning using a separate GNN model to learn to attune the graph to the proposed architecture.

Original: [Figure]

Original: via progressively growing and pruning using

Original: [Figure]

Original: a separate GNN model to learn to attune the

Original: [Figure]

Original: graph to the proposed architecture.

Original: Addition-

Original: [Figure] ally, by defining the highly flexible data-model[Figure] co-search space, AutoCoG is formulated as a 
Revised: Additionally, by defining the highly flexible data-model co-search space, AutoCoG is formulated as a principled bi-level optimization that can be endto-end solved by the differentiable search methods.

Original: [Figure]

Original: principled bi-level optimization that can be end-

Original: [Figure]

Original: to-end solved by the differentiable search meth-

Original: [Figure]

Original: ods.

Original: To scale up our core framework to search-

Original: [Figure]

Original: ing deep GNN architectures, we curb an explo-

Original: [Figure]

Original: sive search space as the number of layers increased by performing multiple searching stages with[Figure] increasing depth, as inspired by Chen et al. (2019c).
Revised: To scale up our core framework to searching deep GNN architectures, we curb an explosive search space as the number of layers increased by performing multiple searching stages with increasing depth, as inspired by Chen et al. (2019c).

Original: Additionally, for each search stages, we67 evolve the graph by growing/pruning it at the same time.
Revised: Additionally, for each search stages, we evolve the graph by growing/pruning it at the same time.

Original: To stabilize the search landscape from[Figure] the shifting topologies of graph and model, we further utilize Chen et al. (2020a) to combat the69 over-smoothing/over-squashing issues.
Revised: To stabilize the search landscape from the shifting topologies of graph and model, we further utilize (Chen et al., 2020a) to combat the over-smoothing/over-squashing issues.

Original: 70 Together, our framework ensures a reliable way to discover powerful architectures, a stable71 model training environment, and state-of-the-art results to train graph different degree of homophily.
Revised: Together, our framework ensures a reliable way to discover powerful architectures, a stable model training environment, and state-of-the-art results to train graph different degree of homophily.

Original: AutoCoG searches for and trains on deep or shallow graph neural networks to successfully achieve73 state-of-the-art results in Web datasets, Actor, Coauthor, and Wikipedia benchmarks.
Revised: AutoCoG searches for and trains on deep or shallow graph neural networks to successfully achieve state-of-the-art results in Web datasets, Actor, Coauthor, and Wikipedia benchmarks.

Original: In summary, our three contributing novelties are:
Revised: In summary, our three contributing novelties are:

Original: • We propose AutoCoG , the first NAS framework towards unified data-model co-search for GNNs. 
Revised: • We propose AutoCoG , the first NAS framework towards unified data-model co-search for GNNs. Our

Original: Our novel bi-level optimization formulation uniquely enables the end-to-end discovery of state- of-the-art GNN model and graph altogether.
Revised: novel bi-level optimization formulation uniquely enables the end-to-end discovery of stateof-the-art GNN model and graph altogether.

Original: • We perform extensive analysis the resulting learned graph-structures for each benchmarks.
Revised: • We perform extensive analysis the resulting learned graph-structures for each benchmarks.

Original: To strengthen the co-search framework, we organically integrate several techniques to directly combat issues of searching unreliability, training instability, and scalability, that have previously81 plagued NAS approaches for searching deeper GNNs.
Revised: To strengthen the co-search framework, we organically integrate several techniques to directly combat issues of searching unreliability, training instability, and scalability, that have previously plagued NAS approaches for searching deeper GNNs.

Original: • Experiments show  AutoCoG achieves gains of up to 4% for Actor, 7.3% on average for Web datasets, 0.17% for CoAuthor-CS, and finally 5.4% for
Revised: • Experiments show that AutoCoG achieves an average performance gain across all datasets of 3.18% over the following best approach and ranks best against all other state-of-the-art methods with an average ranking of 2.5.

Original: 2 Related works
Revised: 2 Related works

Original: Graph neural networks.
Revised: Graph neural networks.

Original: Motivated by the state-of-the-art results of GNNs in graph analytics,86 there have been numerous GNN variants Bruna et al. Hamilton et al. ; Xu et al.
Revised: Motivated by the state-of-the-art results of GNNs in graph analytics, there have been numerous GNN variants (Bruna et al., 2013; Hamilton et al., 2017; Xu et al., 2019;

Original: (2019); 87 Chen et al. (2020a) ; Wu et al. (2019a).
Revised: Chen et al., 2020a; Wu et al., 2019a).

Original: Most of these existing approaches fit within the category of88 spatial GNNs.
Revised: Most of these existing approaches fit within the category of spatial GNNs.

Original: Namely, following the spatial message passing strategy, the core idea of GNNs is89 to learn the embedding representation of a node by aggregating the embeddings of its neighbors and node itself recursively.
Revised: Namely, following the spatial message passing strategy, the core idea of GNNs is to learn the embedding representation of a node by aggregating the embeddings of its neighbors and node itself recursively.

Original: The previous empirical studies show that GNNs often achieve the91 best performance with less than 3 layers Kipf and Welling (2016a); Veličković et al.
Revised: The previous empirical studies show that GNNs often achieve the best performance with less than 3 layers (Kipf and Welling, 2016a; Veličković et al., 2018).

Original: Key92 limitations of GNNs are their performances decrease significantly with the increasing of model93 depth and the degree of graph homophily they operate on.
Revised: Key limitations of GNNs are their performances decrease significantly with the increasing of model depth and the degree of graph homophily they operate on.

Original: As the graph convolutional layer94 increases, the node representations will converge to indistinguishable vectors due to the recursive95 neighborhood aggregation and non-linear activation Li et al.
Revised: As the graph convolutional layer increases, the node representations will converge to indistinguishable vectors due to the recursive neighborhood aggregation and non-linear activation (Li et al., 2018;

Original: (2018); Oono and Suzuki (2020), which96 is well recognized as  over-smoothing issue NT and Maehara (2019); Chen et al. (2019a)
Revised: Oono and Suzuki, 2020), which is well known as the over-smoothing issue (NT and Maehara, 2019; Chen et al., 2019a; Alon and Yahav, 2020;

Original: ; Alon and 97 Yahav (2020); Chien et al. ; Huang et al.
Revised: Chien et al., 2021; Huang et al., 2020).

Original: Graph augmentation.
Revised: Graph augmentation.

Original: Data augmentation methods has been widely applied to improve the gen99 eralization performances of deep neural networks, such as convolutional and recurrent neural100 networks Shorten and Khoshgoftaar (2019); Antoniou et al.
Revised: Data augmentation methods has been widely applied to improve the generalization performances of deep neural networks, such as convolutional and recurrent neural networks (Shorten and Khoshgoftaar, 2019; Antoniou et al., 2017;

Original: ; Feng et al.
Revised: Feng et al., 2021).

Original: They aim to101 craft the out-of-distribution training data to avoid overfitting with the customized augmentation102 policies.
Revised: They aim to craft the out-of-distribution training data to avoid overfitting with the customized augmentation policies.

Original: In the graph analytics, GNNs are prone to overfit the naturally noisy training graphs,103 which may miss the ground-truth nodes/edges or contain the erroneous information Zügner et al.(2018).
Revised: In the graph analytics, GNNs are prone to overfit the naturally noisy training graphs, which may miss the ground-truth nodes/edges or contain the erroneous information (Zügner et al., 2018).

Original: Different from the grid-like image data, the graph augmentation is often operated on the105 adjacency structure or node features.
Revised: Different from the grid-like image data, the graph augmentation is often operated on the adjacency structure or node features (You et al., 2020b, 2021, 2022).

Original: The existing graph augmentations could be catagorized into106 the following two classes.
Revised: Existing graph augmentations could be categorized into the following two classes.

Original: (i)
Revised: (i)

Original: The random augmentation either drops/adds edges to modify the107 graph, or masks parts of the node features Rong et al.
Revised: The random augmentation either drops/adds edges to modify the graph, or masks parts of the node features (Rong et al., 2020a;

Original: (2020a) ; You et al. (2020b) ; Feng et al.
Revised: You et al., 2020b; Feng et al., 2020).

Original: (2020).(ii)
Revised: (ii)

Original: The differentiable augmentation learns to optimize the adjacency affinity matrix by minimizing109 the concerned task loss.
Revised: The differentiable augmentation learns to optimize the adjacency affinity matrix by minimizing the concerned task loss.

Original: Based upon the computed affinity matrix, the differentiable augmentation110 either continuously combines it into the original adjacency matrix Zhao et al.
Revised: Based upon the computed affinity matrix, the differentiable augmentation either continuously combines it into the original adjacency matrix (Zhao et al., 2020b;

Original: (2020b) ; Chen et al.(2020b), or samples the discrete edges to formulate new graph Chen et al. (2019b).
Revised: Chen et al., 2020b), or samples the discrete edges to formulate new graph (Chen et al., 2019b).

Original: Neural architecture search.
Revised: Neural architecture search.

Original: Targeting at alleviating the laborious hyperparameter tuning, NAS113 automates the designing of good neural architectures for any a given application.
Revised: Targeting at alleviating the laborious hyperparameter tuning, NAS automates the designing of good neural architectures for any a given application.

Original: It is shockingly114 reported that the searched neural architectures could outperform the human-designed ones in115 many real-world scenarios, such as image classification Zoph and Le (2016); Zoph et al. (2018) and116 generation Wang and Huan (2019);
Revised: It is shockingly reported that the searched neural architectures could outperform the human-designed ones in many real-world scenarios, such as image classification (Zoph and Le, 2016; Zoph et al., 2018) and generation (Wang and Huan, 2019;

Original: Gong et al.
Revised: Gong et al., 2019).

Original: Most of NAS frameworks apply one of the117 following search algorithms: reinforcement learning (RL) Pham et al. ; Baker et al. [Figure] evolution algorithm (EA) Liu et al.
Revised: Most of NAS frameworks apply one of the following search algorithms: reinforcement learning (RL) (Pham et al., 2018; Baker et al., 2016), evolution algorithm (EA) (Liu et al., 2017; Miikkulainen et al., 2019; Xie and Yuille, 2017), and one-shot differentiable search (Liu et al., 2018; Zela et al., 2020).

Original: ; Miikkulainen et al. ; Xie and Yuille (2017), and 119 one-shot differentiable search Liu et al. ; Zela et al. There are several recent efforts to120 conjoin the researches of GNNs and NAS Gao et al.
Revised: There are several recent efforts to conjoin the researches of GNNs and NAS (Gao et al., 2019;

Original: ; Zhou et al. (2019b) ; You et al. (2020a);
Revised: Zhou et al., 2019b; You et al., 2020a;

Original: [Figure] Ding et al. Zhao et al. (2020a).
Revised: Ding et al., 2020; Zhao et al., 2020a).

Original: However, all of them are limited in exploring the shallow
Revised: However, all of them are limited in exploring the shallow

Original: [Figure] GNNs, and fail to denoise the underlying graph to further ameliorate the model performance.In
Revised: GNNs, and fail to denoise the underlying graph to further ameliorate the model performance.

Original: [Figure] this work, we aim to simultaneously search the deep GNN models and graph structure to optimize 
Revised: In this work, we aim to simultaneously search the deep GNN models and graph structure to optimize the downstream graph analytics.

Original: [Figure]
Revised: Co-Adaptive Search between graph’s structure and model’s architecture. GASSO (Qin et al., 2021) is a recent work that similarly proposes the idea of model-graph co-search. Yet two differentiation factors uniquely defined our two works. Firstly, GASSO is a technique that learns attention coefficients G ∈ [0, 1] only for existing edges E, which is mathematically equivalent to graph attention neural networks (Qin et al., 2021).

Original:  the downstream graph analytics.
Revised: In contrast, Auto-CoG directly modifies the graph’s structure via pruning poorly attended edges and adding new un-seen edges. Thus the derived graph structure is unique from the original underlying graph. Finally, GASSO employs a coarse macro-level search space with only eight operators and two layers.

Original: [Figure]
Revised: Its design decision space is shallow (small), consisting of only 256 unique combinations. Theoretically, by searching the optimal attention function in our Auto-Cog , we could approximate the "attentional structure learning" in GASSO.

Original: 3 Methodology
Revised: 3 Methodology

Original: [Figure]

Original: Preliminary.
Revised: Preliminary.

Original: We briefly review the basic of a message-passing based graph convolution network[Figure] (GCN) and the definition of homophily.[Figure]
Revised: We briefly review the basic of a message-passing based graph convolution network (GCN) and the definition of homophily.

Original: First, the homophily or edge-homophily ratio, of a graph measures the ratio between intra-node[Figure] pairs ( 𝑣,𝑤 ) overall all edges 𝐸 and is given as:[Figure]
Revised: First, the homophily or edge-homophily ratio, of a graph measures the ratio between intra-node pairs ( 𝑣,𝑤 ) overall all edges 𝐸 and is given as:

Original: [Equation]
Revised: [Equation]

Original: Second, given a GCN, its 𝑘 -th layer could be written generally as:
Revised: Second, given a GCN, its 𝑘 -th layer could be written generally as:

Original: [Equation]
Revised: [Equation]

Original: l a y e rl a y e rl a y e rl a y e rl a y e rl a y e r- … A tt en t i on A gg r ega t e
Revised: 𝑥

Original: C o m b i ne A c t i v a t i on S k i p
Revised: [Equation]

Original: Graph search denotes the node embedding of element 𝑖 at 𝑘 -th layer.
Revised: 𝑖 denotes the node embedding of element 𝑖 at 𝑘 -th layer.

Original: 𝑊 ( 𝑘 ) ∈ R 𝐷 × 𝐷 represents the learnable132 layer-wise weights for all { 𝑥 𝑖 : 𝑖 ∈ | 𝑉 |} , where | 𝑉 | is our total number of nodes and 𝐷 our number of hidden features.
Revised: 𝑊 ( 𝑘 ) ∈ R 𝐷 × 𝐷 represents the learnable layer-wise weights for all { 𝑥 𝑖 : 𝑖 ∈ | 𝑉 |} , where | 𝑉 | is our total number of nodes and 𝐷 our number of hidden features.

Original: 𝑎 ( 𝑘 ) 𝑖𝑗 dictates the attention coefficient between 𝑖 and 𝑗 derived from some Attention function.
Revised: 𝑎 ( 𝑘 ) 𝑖𝑗 dictates the attention coefficient between 𝑖 and 𝑗 derived from some Attention function.

Original: N ( 𝑖 ) denotes the neighboring nodes of node 𝑖 from a graph 𝐺 .
Revised: N ( 𝑖 ) denotes the neighboring nodes of node 𝑖 from a graph 𝐺 .

Original: ℎ ( 𝑘 ) 𝑖 is the resulting135 embeddings after applying an AGGR function to aggregate a set of neighboring embeddings from136 the previous 𝑘 − 1 layers.
Revised: ℎ ( 𝑘 ) 𝑖 is the resulting embeddings after applying an AGGR function to aggregate a set of neighboring embeddings from the previous 𝑘 − 1 layers.

Original: In addition, function COMB incorporates information from itself with its neighboring embeddings ℎ ( 𝑘 ) 𝑖 , and 𝜎 provides the nonlinear activation.
Revised: In addition, function COMB incorporates information from itself with its neighboring embeddings ℎ ( 𝑘 ) 𝑖 , and 𝜎 provides the nonlinear activation.

Original: 3.1 Unified Data-Model Co-Search Space 
Revised: 3.1 Unified Data-Model Co-Search Space 3.1.1 Model Search Space: Attention, Activation, and Skip Connection.

Original: 3.1.1 Model Search Space: Attention, Activation, and Skip Connection.

Original: [Table]
Revised: [Table]

Original: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 
Revised: ) 𝑥 ( 𝑘 − 1 ) 𝑗 )

Original: 𝑖 ||

Original: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 ) LINEAR tanh ( ¯ 𝑎 𝑙 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖
Revised: LINEAR tanh ( ¯ 𝑎 𝑙 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖

Original: || 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 )
Revised: || 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 )

Original: GERE-LINEAR 𝑊𝐺 tanh ( 𝑊 ( 𝑘 )
Revised: [Table] 𝑊

Original: 𝑥 ( 𝑘  − 1 ) 𝑖
Revised: [Table] tanh ( 𝑊 ( 𝑘 ) 𝑥 𝑘 − 1 ) 𝑖

Original: + 𝑊 ( 𝑘 ) 𝑥
Revised: + 𝑊 ( 𝑘 ) 𝑥

Original: ( 𝑘 − 1 ) 𝑖 )
Revised: ( 𝑘 − 1 ) 𝑖 )

Revised: [Table]

Original: GAT LeakyReLU ( ¯ 𝑎 ( 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖 ||
Revised: LeakyReLU ( ¯ 𝑎 ( 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖 ||

Original: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 ))
Revised: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 ))

Original: GAT-SUM 𝑎 ( 𝑘 ) 𝑖𝑗 + 𝑎 ( 𝑘 ) 𝑖𝑗 based on GAT
Revised: [Table] SUM 𝑎 ( 𝑘 ) 𝑖𝑗 + 𝑎 ( 𝑘 ) [Table]

Original: CONST
Revised: CONST

Revised: [Table]

Original: Defining the Model Search Space.
Revised: Defining the Model Search Space.

Original: The design141 of model search space should achieve a balanced142 trade-off between the diversity and efficiency.
Revised: The design of model search space should achieve a balanced trade-off between the diversity and efficiency.

Original: 143 Although a large search space subsumes the di144 verse GNN architectures to adapt to the differ145 ent graph analysis tasks, it would be extremely146 time-consuming to explore the optimal design.
Revised: Although a large search space subsumes the diverse GNN architectures to adapt to the different graph analysis tasks, it would be extremely time-consuming to explore the optimal design.

Original: In the existing search spaces of148 GNNs (Gao et al., 2019; Zhou et al., 2019a;149 You et al., 2020a), they often contain the archi150 tecture components of hidden units, attention,151 aggregation, combination, and activation func152 tions, as well as the skip connections.
Revised: In the existing search spaces of GNNs (Gao et al., 2019; Zhou et al., 2019a; You et al., 2020a), they often contain the architecture components of hidden units, attention, aggregation, combination, and activation functions, as well as the skip connections.

Original: To effi153 ciently search the outperforming shallow and deep GNNs, we compare the effectiveness of each154 component, and greatly shrink down the search space to focus on three key components: the155 Activation function, the Attention module, and the skip connections.
Revised: To efficiently search the outperforming shallow and deep GNNs, we compare the effectiveness of each component, and greatly shrink down the search space to focus on three key components: the Activation function, the Attention module, and the skip connections.

Original: They are generally believed156 to impact GNN’s expressive capability and depth scalability (Chen et al., 2021b).
Revised: They are generally believed to impact GNN’s expressive capability and depth scalability (Chen et al., 2021b).

Original: We fix the Ag- greation function and Combination function to be simple summation, and treat the hidden units as158 hyperparameter.
Revised: We fix the Aggregation function and Combination function to be simple summation, and treat the hidden units as hyperparameter.

Original: Below we lay out our searchable design for them one-by-one:
Revised: Below we lay out our searchable design for them one-by-one:

Original: • Attention Search Space :
Revised: • Attention Search Space :

Original: Attention mechanism has

Original: been

Original:  shown by (Veličković et al., 2018) to effectively stabilize training by placing proper neighborhood scaling with attention coefficient 
Revised: Attention mechanism has been shown by (Veličković et al., 2018) to effectively stabilize training by placing proper neighborhood scaling with attention coefficient 𝑎 𝑖𝑗 .

Original: 𝑎 𝑖𝑗 .

Original: We list our attention choices in Table 1.
Revised: We list our attention choices in Table 1.

Original: • Activation Search Space : for basic activation functions, we search among these operations { ReLU , Sigmoid , Tanh , Linear , SoftPlus , LeakyReLU , ReLU6 , ELU } .
Revised: • Activation Search Space : for basic activation functions, we search among these operations { ReLU , Sigmoid , Tanh , Linear , SoftPlus , LeakyReLU , ReLU , ELU } .

Original: • Skip Connection Search Space : For an 𝐿 -Layer GCN, various skip connections can be applied to165 overcome the effect of over-smoothing.
Revised: • Skip Connection Search Space : For an 𝐿 -Layer GCN, various skip connections can be applied to overcome the effect of over-smoothing.

Original: Previous deep GCN works (Chen et al., 2020a; Zhang et al., 2020; Chen et al., 2021b) illustrated a significant correlation between the type of skipconnections to the performance.
Revised: Previous deep GCN works (Chen et al., 2020a; Zhang et al., 2020; Chen et al., 2021b) illustrated a significant correlation between the type of skipconnections to the performance.

Original: We include three skip-connection types — i)
Revised: We include three skip-connection types — i)

Original: Initial Connection,
Revised: Initial Connection,

Original: ii) Jumping-Knowledge aggregation.
Revised: ii) Jumping-Knowledge aggregation

Revised: l a y e rl a y e rl a y e rl a y e rl a y e rl a y e r- …

Revised: A tt en t i on

Revised: A gg r ega t e

Revised: C o m b

Revised: i ne

Original: 3.1.2 Graph Augmentation.
Revised: A c t i v a t i on S k i p [Figure] .2 Graph Augmentation.

Original: We often expect a clustering of liked nodes when operating on graph170 data in a "like attracts like" world.
Revised: We often expect a clustering of liked nodes when operating on graph data in a "like attracts like" world.

Original: However, in reality, when modeling complex relationships, we171 often observe the opposite, where node identities are often best described by contrasting with172 their neighbors in "different attract" relationships.
Revised: However, in reality, when modeling complex relationships, we often observe the opposite, where node identities are often best described by contrasting with their neighbors in "different attract" relationships.

Original: Under such heterophily circumstance, GNNs’173 performances degrade (Pei et al., 2020a; Zhu et al., 2020; Battaglia et al., 2018), which makes sense174 intuitively, since relating unrelated nodes can lead to class obscurity (Zhu et al., 2020), i.e oversmoothing.
Revised: Under such heterophily circumstance, GNNs’ performances degrade (Pei et al., 2020a; Zhu et al., 2020a; Battaglia et al., 2018), which makes sense intuitively, since relating unrelated nodes can lead to class obscurity (Zhu et al., 2020a), i.e over-smoothing.

Original: Thus overcoming heterophily with model architecture alone is difficult and often176 requires complicated, and exotic works flow (Abu-El-Haija et al., 2019; Pei et al., 2020b; Lim et al.,177 2021).
Revised: Thus overcoming heterophily with model architecture alone is difficult and often requires complicated, and exotic works flow (Abu-El-Haija et al., 2019; Pei et al., 2020b; Lim et al., 2021).

Original: Lately, a number of works (Srivastava et al., 2014; Zou et al., 2019;
Revised: Lately, a number of works (Srivastava et al., 2014; Zou et al., 2019;

Original: Rong et al., 2020b; Chen178 et al., 2021a; Huang et al., 2021) have found that direct graph augmentations with stochastic policies179 — drop/add edges — can decelerate both the over-fitting and over-smoothing issues in training deep180 GNNs.
Revised: Rong et al., 2020b; Chen et al., 2021a; Huang et al., 2021) have found that direct graph augmentations with stochastic policies — drop/add edges — can decelerate both the over-fitting and over-smoothing issues in training deep GNNs.

Original: By learning the graph’s topology and the model’s architecture, we naturally adapt our data181 structure around the model’s strength and co-optimize data flow around the message-aggregation182 mechanism.
Revised: By learning the graph’s topology and the model’s architecture, we naturally adapt our data structure around the model’s strength and co-optimize data flow around the message-aggregation mechanism.

Original: The scoring function.
Revised: The scoring function.

Original: Given graph 𝐺 ( 𝑉, 𝐸 ) can be expressed in the form of an adjacent matrix184 𝐴 ∈ R | 𝑉 |×| 𝑉 | , where 𝑉 is the set of vertices and 𝐸 is the set of edges.
Revised: Given graph 𝐺 ( 𝑉, 𝐸 ) can be expressed in the form of an adjacent matrix 𝐴 ∈ R | 𝑉 |×| 𝑉 | , where 𝑉 is the set of vertices and 𝐸 is the set of edges.

Original: We learn an edge score matrix185 𝑆 ∈ R | 𝑉 |×| 𝑉 | such that we rewrite the aggregation step in Eqn (2) as:
Revised: We learn an edge score matrix 𝑆 ∈ R | 𝑉 |×| 𝑉 | such that we rewrite the aggregation step in Eqn (2) as:

Original: [Equation]
Revised: [Equation]

Original: where 𝐻 ( 𝑘 ) =
Revised: where 𝐻 ( 𝑘 ) =

Original: { ℎ ( 𝑘 ) 𝑖 : 0 ≤ 𝑖 ≤ | 𝑉 |} and 𝑋 ( 𝑘 − 1 ) =
Revised: { ℎ ( 𝑘 ) 𝑖 : 0 ≤ 𝑖 ≤ | 𝑉 |} and 𝑋 ( 𝑘 − 1 ) =

Original: { 𝑥 ( 𝑘 − 1 ) 𝑖 : 0 ≤ 𝑖 ≤ | 𝑉 |} .
Revised: { 𝑥 ( 𝑘 − 1 ) 𝑖 : 0 ≤ 𝑖 ≤ | 𝑉 |} .

Original: We formally denote187 𝑆 ∈ [ 0 , 1 ] as:
Revised: We formally denote 𝑆 ∈ [ 0 , 1 ] as:

Original: [Equation]
Revised: [Equation]

Original: Where 𝑓 ( . ; 𝑊 𝑠 ) is simply the classic VGAE model by Kipf and Welling (2016b), 𝑍 𝑉 𝑠𝑟𝑐 / 𝑍 𝑉 𝑡𝑔𝑡 are the189 source and target nodes available from 𝐺 ( 𝑉, 𝐸 ) , || denotes the concatenation function, and 𝜎 is the190 sigmoid function.
Revised: Where 𝑓 ( . ; 𝑊 𝑠 ) is simply the classic VGAE model by (Kipf and Welling, 2016b), 𝑍 𝑉 𝑠𝑟𝑐 / 𝑍 𝑉 𝑡𝑔𝑡 are the source and target nodes available from 𝐺 ( 𝑉, 𝐸 ) , || denotes the concatenation function, and 𝜎 is the sigmoid function.

Original: 𝑆 is therefore a sparse matrix with only | 𝐸 | number of scores.
Revised: 𝑆 is therefore a sparse matrix with only | 𝐸 | number of scores.

Original: Edge growing and pruning.
Revised: Edge growing and pruning.

Original: Taking advantage of the Progressive NAS workflow Chen et al. (2019c), 192 at each searching stage, we prune the bottom 𝑝 -percentile from 𝑆 , and at the same-time we grow193 our graph by appending 𝑘 new edges for each nodes via embedding similarity.
Revised: Taking advantage of the Progressive NAS workflow (Chen et al., 2019c), at each searching stage, we prune the bottom 𝑝 -percentile from 𝑆 , and at the same-time we grow our graph by appending 𝑘 new edges for each nodes via embedding similarity.

Original: This embedding194 similarity function is best defined as:
Revised: This embedding similarity function is best defined as:

Original: [Equation]
Revised: [Equation]

Original: 𝑖𝑗 (5) Where 𝐶𝑜𝑠𝑖𝑛𝑒 ( . ) is the cosine-similarity between two nodes’ scoring embeddings, while 𝐷 is the196 shortest distance between them.
Revised: Where 𝐶𝑜𝑠𝑖𝑛𝑒 ( . ) is the cosine-similarity between two nodes’ scoring embeddings, while 𝐷 is the shortest distance between them.

Original: We illustrate visually in Figure(2) and in code via Algorithm(1) in197 the our appendix.
Revised: We illustrate visually in Figure(2) and in code via Algorithm(1) in the our appendix.

Original: 3.2 Optimization Formulation and Algorithm
Revised: 3.2 Optimization Formulation and Algorithm

Original: 3.2.1 A Principled Bi-Level Optimization Formulation.
Revised: 3.2.1 A Principled Bi-Level Optimization Formulation.

Original: For the sake of conciseness, we use 𝛼 as the model space architecture parameters , and denote L obj as the objective loss function given 𝛼 .
Revised: For the sake of conciseness, we use 𝛼 as the model space architecture parameters , and denote L obj as the objective loss function given 𝛼 .

Original: With 𝛼 defined, we further denote ˆ 𝑊 =
Revised: With 𝛼 defined, we further denote ˆ 𝑊 =

Original: 𝑊 (cid:199) 𝑚
Revised: 𝑊 (cid:199) 𝑚

Original: 𝛼 as the pruned sub-model from the supernet derived from202 𝛼 description, where ˆ 𝑊 ,𝑚 𝛼 ∈ R 𝐿 × 𝐷 × 𝐷 , D denotes the size of hidden embeddings.
Revised: 𝛼 as the pruned sub-model from the supernet derived from 𝛼 description, where ˆ 𝑊 ,𝑚 𝛼 ∈ R 𝐿 × 𝐷 × 𝐷 , D denotes the size of hidden embeddings.

Original: Additionally, we203 can write our augmented graph ˆ 𝐺 as ˆ 𝐴 =
Revised: Additionally, we can write our augmented graph ˆ 𝐺 as ˆ 𝐴 =

Original: 𝐴 (cid:199) 𝑆 , where 𝑆 is defined as our learned scoring matrix.
Revised: 𝐴 (cid:199) 𝑆 , where 𝑆 is defined as our learned scoring matrix.

Original: Then, let 𝑍 represents our output vector for a hypothetical 2-layer AutoCoG :
Revised: Then, let 𝑍 represents our output vector for a hypothetical 2-layer AutoCoG :

Original: [Equation]
Revised: [Equation]

Original: Thus the objective loss function L obj for a transductive semi-supervised node classification tasks is206 formally denoted as:
Revised: Thus the objective loss function L obj for a transductive semi-supervised node classification tasks is formally denoted as:

Original: [Equation]
Revised: [Equation]

Original: [Equation] Extending from (Dong and Yang, 2019), we formulate our data-model co-search as a joint bi-level optimization , to solve 𝛼, 𝑆 concurrently with the weights 𝑊 and data space parameters:
Revised: Extending from (Dong and Yang, 2019), we formulate our data-model co-search as a joint bi-level optimization , to solve 𝛼, 𝑆 concurrently with the weights 𝑊 and data space parameters:

Original: [Equation]
Revised: [Equation]

Original: Note that 𝛼 are optimized using the objective loss function on the validation set, while 𝑊 , 𝑆 are210 optimized under training set.
Revised: Note that 𝛼 are optimized using the objective loss function on the validation set, while 𝑊 , 𝑆 are optimized under training set.

Original: Additionally, ˆ 𝐺 also consists of modified edges ¯ 𝐸 , not-shown explicitly211 in Equation (8), but is illustrated in our Algorithm(1).
Revised: Additionally, ˆ 𝐺 also consists of modified edges ¯ 𝐸 , not-shown explicitly in Equation (8), but is illustrated in our Algorithm(1).

Original: We adopt the same hard-
Revised: We adopt the same hard-

Original: Gumbel-softmax212 trick (Jang et al., 2017) to differentially optimize architectural variables during search.
Revised: Gumbel-softmax trick (Jang et al., 2017) to differentially optimize architectural variables during search.

Original: 3.2.2 Scaling and Stabilizing the Search.
Revised: 3.2.2 Scaling and Stabilizing the Search

Revised: Thus,

Original: Thus, the bi-level optimization (8) can be solved by differential
Revised: the bi -level optimization (

Original:  search methods, and we adopt the GDAS approach in (Dong and Yang, 2019) by default.
Revised: can be solved by differential search methods and we adopt the GDAS approach in (Dong and Yang, 2019) by default.

Original: However, 215 when exploring GCN deep architectures and larger graphs, the data/model search spaces grow216 exponentially with the layer depth/graph size, and they can be entangled to cause even more
Revised: However when exploring GCN deep architectures and larger graphs the data/model search spaces grow

Original:  serious scalability challenge. 
Revised: exponentially with the layer depth/graph size, and they can be entangled to cause even more serious scalability challenge. That is

Original: That is further amplified by the training difficulty and instability of deep GNNs (Chen et al., 2021b).
Revised: further amplified by the training difficulty and instability of deep GNNs (Chen et al., 2021b).

Original: Indeed, we observe that naively applying GDAS is prone to over-smoothening and search collapse, only yielding very poor architectures when searching for more than three layers.
Revised: Indeed, we observe that naively applying GDAS is prone to over smoothening and search collapse, only yielding very poor architectures when searching for more than three layers.

Original: Besides, it is not uncommon for the derived graph and model to have 
Revised: Besides it is not uncommon for the derived graph and model to have considerable

Original: considerable performance variations across repeated experiments, due the stochastic initialization and training.
Revised: performance variations across repeated experiments, due the stochastic initialization and training.

Original: 223 We investigate and incorporate the following two “tricks" into our differential search process.

Original: They are found to contribute remarkably to the scalability, stability and consistency of AutoCoG .

Original: Progressive search space.
Revised: Progressive search space.

Original: We follow the idea proposed by (Chen et al., 2019c)
Revised: We follow the idea proposed by (Chen et al., 2019c)

Original: (also illustrated226 in Algorithm 1), to divide search into 𝑁 progressive stages, with each consecutive stage having a227 larger or equal number of layers than those previously.
Revised: (also illustrated in Algorithm 1), to divide search into 𝑁 progressive stages, with each consecutive stage having a larger or equal number of layers than those previously.

Original: At each stage, we greedily remove the least228 selected options (by taking the mean of Soft-Max across 𝐿 layers and removing the option with229 the smallest value) from the data’s 𝑝 parameters or model space, and pass on the shrunk co-search230 space to the next stage.
Revised: At each stage, we greedily remove the least selected options (by taking the mean of Soft-Max across 𝐿 layers and removing the option with the smallest value) from the data’s 𝑝 parameters or model space, and pass on the shrunk co-search space to the next stage.

Original: Note that we do not shrink the number of augmentation policies.
Revised: Note that we do not shrink the number of augmentation policies.

Original: Identity mapping..

Original: To relieve the over-smoothing and over-fitting issues, the Identity Mapping 232 was first proposed by (Chen et al., 2020a) to fashion deep GCN.

Original: It is formally denoted as:

Original: [Equation]

Original: Where 𝐵 𝑘 = log ( 𝛾𝑘 + 1 ) and 𝛾 is a positive hype-parameter.

Original: 4 Experiments 
Revised: 4 Experiments 4.1 Experimental settings.

Original: 4.1 Datasets and experimental settings. Datasets.

Original: We evaluate AutoCoG on several popular semi-node classification datasets including (i) 237 the WebKB datasets —Cornell, Texas, Wisconsin— (Craven et al., 1998), (ii) Actor dataset (Tang 238 et al., 2009), (iii) Co-author datasets —CS, Physics— (Shchur et al., 2019), and (iv) Amazon datasets 239 — Photo, Computers — (Shchur et al., 2019).

Original: We select these datasets to represent a wide range of 240 graphs under different degrees of homophily, which will serve to demonstrate Auto-CoG robustness 241 in comparison to the different SOTA methods.

Original: The specifics on each datasets, in sorted order under 242 homophily, is recorded in Table(2).

Original: [Table]

Original: Physics 34, 495, 8, 0. Experimental settings. We will list our default 244 training hype-parameters common across all245 datasets,  then we will note each dataset’s spe246 cific particularity, if any.
Revised: We first list our shared training hype-parameters for all datasets, and then list each dataset’s specific particularity.

Original: By default, we employ247 the Adam-optimizer (Kingma and Ba, 2017) to248 learn edges’ scores, model’s architecture and249 model’s weights with equal learning rate of250 0 . 005, and a 𝐿 2 regularization of 0 . 0005.
Revised: By default, we employ the Adam-optimizer (Kingma and Ba, 2017) to learn edges’ scores, model’s architecture and model’s weights with equal learning rate of 0 . 005, and a 𝐿 2 regularization of 0 . 0005.

Original: We251 set the hidden-dimension 𝐷 to be 256 with a252 dropout rate of 0.6.
Revised: We set the hidden-dimension 𝐷 to be 256 with a dropout rate of 0.6.

Original: As for P-DARTS, for every253 stage we prune the bottom 10% of edges, and254 add one new edge per node, and the number255 of stages are set to be 4, starting from 2 layers,256 and with a 2 layers increment.
Revised: As for P-DARTS, for every stage we prune the bottom 10% of edges, and add one new edge per node, and the number of stages are set to be 4, starting from 2 layers, and with a 2 layers increment.

Original: Furthermore, for Identity-Mapping, 𝛾 is set to be 0.5.
Revised: Furthermore, for Identity-Mapping, 𝛾 is set to be 0.5.

Original: We search/train257 for 1000 epochs, while setting our rate of patient to be 400 and 200 respectively.
Revised: We search/train for 1000 epochs, while setting our rate of patient to be 400 and 200 respectively.

Original: To get our final258 results, we train the network 10 times to get the average and standard deviation.
Revised: To get our final results, we train the networktimes to get the average and standard deviation.

Original: The only notable259 exception to the default settings are the Co-author datasets, where we set the dropout rate to be260 0.8 and 0 for CS and Physics respectively.
Revised: The only notable exception to the default settings are the Co-author datasets, where we set the dropout rate to be 0.8 and 0 for CS and Physics respectively.

Original: We typically only search between two and eight layers.
Revised: We typically only search between two and eight layers.

Original: Finally, for all datasets, we average their accuracy over 10 runs, with random seed between 0 to 9.
Revised: Finally, for all datasets, we average their accuracy over 10 runs, with random seed between 0 to 9.

Original: 4.2 Ablation studies
Revised: 4.2 Ablation studies

Original: [Table]
Revised: [Table]

Original: Model-Graph codependancy.
Revised: Model-Graph codependancy.

Original: We justify the need for model-graph co-search by performing three264 experiments, namely — co-search, data-search, and model-search — to illustrate the respective265 effectiveness of the individual components which constitutes our framework.
Revised: We justify the need for model-graph co-search by performing three experiments, namely — co-search, data-search, and model-search — to illustrate the respective effectiveness of the individual components which constitutes our framework.

Original: We collected these results from several datasets at a fixed depth of 8 while maintaining identical searching settings for267 all experiments.
Revised: We collected these results from several datasets at a fixed depth of 8 while maintaining identical searching settings for all experiments.

Original: Note that for data-search, we substitute our model with the vanilla GCN (Kipf268 and Welling, 2016a).
Revised: Note that for data-search, we substitute our model with the vanilla GCN (Kipf and Welling, 2016a).

Original: The results are collected in Table(3).
Revised: The results are collected in Table(2).

Original: Herein our results speak for themselves;269 we observe a significant improvement in performance, especially for graphs under heterophily,270 utilizing co-search over model-only and data-only search.4-layers 8-layers 16-layers 1. [Figure] R e l a t i v e p e r f o r m a n
Revised: Herein our results speak for themselves; we observe a significant improvement in performance, especially for graphs under heterophily, utilizing co-search over model-only and data-only search.

Original: [Figure]

Original: e g a i n i n %

Original: [Figure]

Original: Correlation between depth and performance.
Revised: Correlation between depth and performance.

Original: 272 We observe that an increasing depth does not273 always positively correlate to performance gain.
Revised: We observe that an increasing depth does not always positively correlate to performance gain.

Original: 274 Homophily negatively correlates to our perfor275 mance at depth.
Revised: Homophily negatively correlates to our performance at depth.

Original: To explain this phenomenon,276 we offer this hypothesis: since the number277 of layers in a model correlates to the num278 ber of k-hop neighbors observed, graphs un279 der heterophily need to observe a much larger280 sub-graph to aggregate meaning information281 against the inherent noisy neighbors. 
Revised: To explain this phenomenon, we offer this hypothesis: since the number of layers in a model correlates to the number of k-hop neighbors observed, graphs under heterophily need to observe a much larger sub-graph to aggregate meaning information against the inherent noisy neighbors. 4-layers 8-layers 16-layers 1. R e l a t i v e p e r f o r m a n c e g a i n i n % Depth to Performance correlation

Revised: ActorTexasWisconsinCornell

Original: In con282 trast, with an increasing degree of homophily,283 more layers may induce over-smoothing sooner,284 contributing to an overall degradation in perfor285 mance.
Revised: In contrast, with an increasing degree of homophily, more layers may induce oversmoothing sooner, contributing to an overall degradation in performance.

Original: We investigate the relationship between286 the model’s depth and performance.
Revised: We investigate the relationship between the model’s depth and performance.

Original: As depth is287 a hyper-parameter, we perform a search on 2, 4, 8, and 16 layers configurations — while maintaining288 identical searching parameters — on several datasets at an increasing rate of homophily.
Revised: As depth is a hyper-parameter, we perform a search on 2, 4, 8, and 16 layers configurations — while maintaining identical searching parameters — on several datasets at an increasing rate of homophily.

Original: We then289 normalize our final accuracy results for each graph against the result of our 2-layer configuration290 to obtain relative performance gain in percentage.
Revised: We then normalize our final accuracy results for each graph against the result of our 2-layer configuration to obtain relative performance gain in percentage.

Original: We illustrate our results in Figure(3).
Revised: We illustrate our results in Figure(3)

Original: [Table]
Revised: [Table]

Original: Analyzing the augmented graphs.

Original: We charac-

Original: [Figure]
Revised: [Table]

Original: terize thenewly augmented graphs to under- [Figure] stand better how they affect the overall perfor- [Figure]
Revised: Analyzing the augmented graphs.

Original: mance by analyzing those searched under the[Figure] eight-layers configuration.We first calculate
Revised: We characterize the newly augmented graphs to understand better how they affect the overall performance by analyzing those searched under the eight-layers configuration.

Original: [Figure]

Original:  the difference of the homophily rate between[Figure] the new and old graphs.Next, we count the
Revised: We first calculate the difference of the homophily rate between the new and old graphs.

Original: [Figure]

Original:  difference of total informative edges, i.e., edges[Figure] between nodes of similar classes.Finally, we
Revised: Next, we count the difference of total informative edges, i.e., edges between nodes of similar classes.

Original: [Figure] calculate the Intersection between edges of the[Figure] original to the augmented graph.
Revised: Finally, we calculate the Intersection between edges of the original to the augmented graph.

Original: Table(4) sum- [Figure] marizes our findings.
Revised: Table(3) summarizes our findings.

Original: Herein we observe that

Original: [Figure] our improved graphs do not exhibit a stronger[Figure] rate of homophily — in comparison to the original graphs — as initially assumed.
Revised: Herein we observe that our improved graphs do not exhibit a stronger rate of homophily — in comparison to the original graphs — as initially assumed.

Original: However, we

Original: [Figure]

Original:  also observe that, despite the increasing heterophily, the searched graphs include more informative[Figure] edges while retaining most of the original edges, indicating we are learning new and relevant [Figure]
Revised: However, we also observe that, despite the increasing heterophily, the searched graphs include more informative edges while retaining most of the original edges, indicating we are learning new and relevant unseen relationships.

Original: unseen relationships. Nevertheless, this observation challenges the current research assumption[Figure] on the correlation between heterophily and performance.
Revised: Nevertheless, this observation challenges the current research assumption on the correlation between heterophily and performance.

Original: We show that performance can still be[Figure] achieved under low homophily given that enough informative edges are added to the graph and a 
Revised: We show that performance can still be achieved under low homophily given that enough informative edges are added to the graph and a deeper architecture.

Original: [Figure]
Revised: Analyzing the method’s effectiveness and efficiency.

Original: deeper architecture. 4.3 Results We compare Auto-CoG to several notable state-of-the-methods, which include: GCN (Kipf and 313 Welling, 2016a), SGC (Wu et al., 2019b), GAT (Veličković et al., 2018), GCNII Chen et al.
Revised: To evaluate the efficiency of our design, we compare its memory usage and total run time to other NAS-based approaches such as GraphNAS (Gao et al., 2019) and SANE (Zhao et al., 2021). Table(4) summarizes our findings.

Original: (2020a),
Revised: We could observe: that AutoCoG maintains relatively low memory utilization for each of the datasets tested, and AutoCoG is also the fastest model to complete both its full-search and training stages.

Original: [Table]
Revised: [Table]

Original: JKNet (Xu et al., 2018) and APPNP (Klicpera et al., 2018).
Revised: Analyzing the effectiveness of progressive search. To test the benefit of progressive search, we perform three ablation studies on Texas, Wisconsin, and Cornell at various depths. As shown in

Original: To test our framework robustness, we 315 experiment on graphs under different degrees of homophily ranging between 0.3 to 0.9.
Revised: Table(5), note that, with progressive search, accuracy is positively correlated to a model’s depth, while the opposite is true when searching without it. This is due to the search instability from the resulting search space size, and a deeper network only further exacerbates the problem. Indeed, we further observe that simply applying GDAS, as in the case of searching without progressive search, only yields poor architectures. The results align with our reasoning in section 3.2.2.

Original: Table(5)
Revised: [Table]

Original: 316 summarizes our results.
Revised: Analyzing graph-model co-search improvement on model’s robustness. Table (6) summarizes our improved robustness on noisy graphs.

Original: Herein, we make the following observations; First, concerning heterophily 317 graphs, in this case, those under 0.6 such as Actor and the WebKB datasets, Auto-CoG performs 318 exceptionally well. Across the board, we observe a difference between our performance to the 319 next best to be 4 .
Revised: We validate our robustness by performing node classification on graphs with added edges. The number of new edges corresponds to a percentage relative to the total edges. From the table, we make the following observations: First, noisy graph data leads to poor model performance from a lack of robustness.

Original: 04% for Actor, 9 .
Revised: Second,

Original: 19% for Texas, 10% for Wisconsin, and 3 . 7% for Cornell. These 320 are significant improvements in performance that validate our approach’s robustness  in handling 321 noisy graphs.
Revised: Graph-search can rectify and improve the model’s robustness by removing artificial noise, leading to better model performance. Third, random edges can help improve some sparse graphs base performance, as in the case of Cornell and Wisconsin, by enabling skip-connections between distance neighborhoods.

Original: On the other hand, for graphs under high-degree of homophily, Auto-CoG performs 322 convincingly better for two out of four datasets tested, namely CS and Photos.
Revised: [Table] 4.3 Results [Table] We compare Auto-CoG to several notable state-of-the-methods inferencing on graphs with increasing degrees of associativity from 0.3 and 0.9.

Revised: Additionally, we also include average improvement and average rank for quick performance comparison at a glance.

Revised: Average improvement is the average accuracy difference between Auto-CoG and another model across all datasets, so a higher score indicates a better result.

Revised: Average rank is a model’s average performance rank for all datasets, so lower is better.

Revised: For comparison, we include:

Original: Nevertheless, We 323 fail to achieve SOTA performance in Physics and Computer, where we  are narrowly beaten by 324 0 .
Revised: • NAS based graph models: for this category, we include GraphNAS(Gao et al., 2019), SANE(Zhao et al., 2021) and GASSO(Qin et al., 2021). • Handcrafted graph models: we compare against traditional designs such as GCN (Kipf and Welling, 2016a), SGC (Wu et al., 2019b), GAT (Veličković et al., 2018), GCNII (Chen et al., 2020a), JKNet (Xu et al., 2018) and APPNP (Klicpera et al., 2018). Additionally, we also compare against designs that are crafted specifically for disassortative graphs such as Geom-GCN (Pei et al., 2020c) and H2GCN (Zhu et al., 2020b). We summarizes our finding in Table(7).

Revised: From the results, we make the following observations:

Revised: • Highlighting the challenge of heterophily, we observe the lack of a dominant approach that can outperform all datasets.

Original: 47% and 2 .
Revised: However, when we compare their average ranking overall, we do find Auto-CoG ranks highly at 2.5 and able to improve against all other approaches on average. This showcase our method’s robustness in dealing with graphs under different homophily settings • In comparison to other NAS approaches, Auto-CoG reliably outperforms all of them when it comes to disassociative datasets since typical GNNs tend to over-smooth on noisy graph data — an inherent problem for message-passing. Auto-CoG directly modifies its graph data and network’s architecture to overcome this weakness. GASSO (Qin et al., 2021) also performs graph structure search, but it is limited to only learning existing edges attention coefficients and therefore is still susceptible to some degree of over-smoothing.

Original: 71% respectively.
Revised: • In comparison to handcrafted baselines, Auto-CoG comfortably outperforms Geom-GCN (Pei et al., 2020c ) on Actor and WebKB datasets. Our graph-structured learning process provides a similar function as the "structural neighborhood" concept, which Geom-GCN utilizes for bi-level aggregation. On the other hand, H2GCN (Pei et al., 2020c) shows impressive performance on small Webkb datasets, outperforming Auto-CoG in both Texas and Cornell.

Revised: However, the model’s ’ego-embeddings’ concept does not scale well on larger datasets such as CoAuthor and Amazon, where it repeatedly fails to produce competitive results.

Original: 5 Conclusion and Limitations
Revised: 5 Conclusion and Limitations

Original: In this paper, we present AutoCoG the first NAS framework towards unified data-model co-search for GNNs .
Revised: In this paper, we present AutoCoG the first NAS framework towards unified data-model co-search for GNNs .

Original: We design a highly flexible data-model co-search space aiming to formulated as a 328 principled bi-level optimization that can be end-to-end solved by the differentiable search methods.

Original: Our results convincingly demonstrate the benefit of data-graph co-search for both deep and shallow330 graph neural networks.
Revised: Our results convincingly demonstrate the benefit of data-graph co-search for both deep and shallow graph neural networks.

Original: Our ablation study shows that controlled variances in graph heterophily can331 result in a better, more generalized model and the necessity for graph-augmentation to be modelaware.
Revised: Our ablation study shows that controlled variances in graph heterophily can result in a better, more generalized model and the necessity for graph-augmentation to be model-aware.

Original: We confidently demonstrate AutoCoG to be a reliable way to discover robust architectures,333 a stable model training environment, and state-of-the-art results.
Revised: We confidently demonstrate AutoCoG to be a reliable way to discover robust architectures, a stable model training environment, and state-of-the-art results.

Original: Additionally, we show that the334 localized disturbance of graph structure motivates node position learning, allowing for greater335 generalizability of the model.
Revised: Additionally, we show that the localized disturbance of graph structure motivates node position learning, allowing for greater generalizability of the model.

Original: 336 However, there are still limitations that need to be addressed: large graph scalability and337 understanding heterophily’s relationship to performance.
Revised: However, there are still limitations that need to be addressed: large graph scalability and understanding heterophily’s relationship to performance.

Original: To address this, we first plan to follow338 up by learning meaningful model/graph using Auto-CoG via graph-batching.
Revised: To address this, we first plan to follow up by learning meaningful model/graph using Auto-CoG via graph-batching.

Original: Secondly, we want339 to conduct a study to understand better the phenomenon between heterophily and performance340 observed in our ablation.
Revised: Secondly, we want to conduct a study to understand better the phenomenon between heterophily and performance observed in our ablation.

Original: There is no negative societal impact to our best knowledge, except that341 the NASsearch process is resource consuming - but even that excessive cost can be amortized by342 the re-usability of the searched model, which can achieve superior accuracy-resource trade-off.6 Reproducibility Checklist For all authors...
Revised: There is no negative societal impact to our best knowledge, except that the NAS process is resource-consuming - but even that excessive cost can be amortized by the re-usability of the searched model, which can achieve superior accuracy-resource trade-off.

Original: (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s

Original: contributions and scope?

Original: [Yes] We claim the benefit of model-graph co-search and prove it

Original: with our superior performance in most datasets tested.

Original: (b) Did you describe the limitations of your work?

Original: [Yes] We describe our two limitations in

Original: section 5.

Original: (c) Did you discuss any potential negative societal impacts of your work?

Original: [Yes] We briefly

Original: discuss it in section 5.

Original: (d) Have you read the ethics author’s and review guidelines and ensured that your paper

Original: conforms to them?

Original: https://automl.cc/ethics-accessibility/

Original: [Yes] If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results? [N/A]
Revised: A Appendix A.1 Datasets

Original: (b) Did you include complete proofs of all theoretical results?

Original: [N/A]

Original: If you ran experiments...

Original: (a) Did you include the code, data, and instructions needed to reproduce the main experimen-

Original: tal results, including all requirements (e.g., requirements.txt with explicit version), an instructive README with installation, and execution commands (either in the supplemental material or as a url)? [Yes] We provide pseudo-code, experimental settings, and model’s
Revised: We evaluate AutoCoG on several popular semi-node classification datasets including (i) the WebKB datasets —Cornell, Texas, Wisconsin— (Craven et al., 1998), (ii) Actor dataset (Tang et al., 2009), (iii) Co-author datasets —CS, Physics— (Shchur et al., 2019), and (iv) Amazon datasets — Photo,

Original: architecture in our supplementary.

Original: Full code will be release upon acceptance.

Original: (b) Did you include the raw results of running the given instructions on the given code and

Original: data? [Yes] (c) Did you include scripts and commands that can be used to generate the figures and tables in your paper based on the raw results of the code, data, and instructions given?
Revised: Computers — (Shchur et al., 2019).

Original: [Yes]

Original: (d) Did you ensure sufficient code quality such that your code can be safely executed and the

Original: code is properly documented?

Original: [Yes]

Original: (e) Did you specify all the training details (e

Original: g., data splits, pre-processing, search spaces,

Original: fixed hyperparameter settings, and how they were chosen)?

Original: [Yes] We detailed our hyper-

Original: parameters in section 4.

Original: (f) Did you ensure that you compared different methods (including your own) exactly on the same benchmarks, including the same datasets, search space, code for training and 374 hyperparameters for that code?
Revised: We select these datasets to represent a wide range of graphs under different degrees of homophily, which will serve to demonstrate Auto-CoG robustness in comparison to the different SOTA methods.

Original: [Yes]

Original: (g) Did you run ablation studies to assess the impact of different components of your approach?

Original: [Yes] see section 4.

Original: (h) Did you use the same evaluation protocol for the methods being compared?

Original: [Yes]

Original: (i) Did you compare performance over time?

Original: [No]

Original: (j) Did you perform multiple runs of your experiments and report random seeds?

Original: [Yes] See

Original: section

Original: (k) Did you report error bars (e

Original: g., with respect to the random seed after running experiments

Original: multiple times)?

Original: [Yes] All our results are averaged over 10 times for seed between 0 and

Original: (l) Did you use tabular or surrogate benchmarks for in-depth evaluations?

Original: [No] This is not

Original: within-the-scope of our research.

Original: (m) Did you include the total amount of compute and the type of resources used (e

Original: g., type of

Original: gpus, internal cluster, or cloud provider)?

Original: [No]

Original: (n) Did you report how you tuned hyperparameters, and what time and resources this required

Original: (if they were not automatically tuned by your AutoML method, e.g. in a nas approach; and

Original: also hyperparameters of your own method)?

Original: [Yes] we went with default hyper-parameters.

Original: If you are using existing assets (e

Original: g., code, data, models) or curating/releasing new assets...

Original: (a) If your work uses existing assets, did you cite the creators?

Original: [Yes] We cited datasets used in section 4. (b) Did you mention the license of the assets? [No]
Revised: The specifics on each datasets, in sorted order under homophily, is recorded in Table(8).

Original: (c) Did you include any new assets either in the supplemental material or as a url?

Original: [No]

Original: (d) Did you discuss whether and how consent was obtained from people whose data you’re

Original: using/curating?

Original: [No]

Original: (e) Did you discuss whether the data you are using/curating contains personally identifiable

Original: information or offensive content?

Original: [No] the benchmarks we used contained no personally

Original: identifiable information or offensive content

Original: If you used crowdsourcing or conducted research with human subjects.

Original: (a) Did you include the full text of instructions given to participants and screenshots, if appli-

Original: cable? [N/A]
Revised: [Table]

Original: (b) Did you describe any potential participant risks, with links to Institutional Review Board

Original: (irb) approvals, if applicable?

Original: [N/A]

Original: (c) Did you include the estimated hourly wage paid to participants and the total amount spent

Original: on participant compensation?

Original: [N/A]

Original: A Appendix

Original:  Input: 𝑊 𝑠 , 𝑋 , 𝐺 ( 𝑉, 𝐸 ) , searchSpace, epochs, startNumLayer, endNumLayer, stages ;Output: 𝛼 , 𝐺 ( 𝑉, ¯
Revised: Photos [Table] Input: 𝑊 𝑠 , 𝑋 , 𝐺 ( 𝑉, 𝐸 ) , searchSpace, epochs, startNumLayer, endNumLayer, stages ;

Original:  𝐸 ) , 𝑆 ; ¯ 𝐸 ← 𝐸 ; for s = 0 to stages-1 do #Initialize new model and architecture parameters 𝛼 ← OnesInitParameters (searchSpace); 𝑊 ← initModel ( min (startNumLayer+s, endNumLayer)); for e=0 to epochs-1 do 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ; ¯ 𝑎 ← Sample ( 𝛼 );
Revised: Output: 𝛼 , 𝐺 ( 𝑉, ¯ 𝐸 ) , 𝑆 ; ¯ 𝐸 ← 𝐸 ; for s = 0 to stages-1 do #Initialize new model and architecture parameters 𝛼 ← OnesInitParameters (searchSpace); 𝑊 ← initModel ( min (startNumLayer+s, endNumLayer)); for e=0 to epochs-1 do 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ; ¯ 𝑎 ← Sample ( 𝛼 );

Original: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑡𝑟𝑎𝑖𝑛 ,𝐺 ( 𝑉, ¯ 𝐸 ))
Revised: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑡𝑟𝑎𝑖𝑛 ,𝐺 ( 𝑉, ¯ 𝐸 ))

Original: → 𝑊 ,𝑊 𝑠 ;
Revised: → 𝑊 ,𝑊 𝑠 ;

Original: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑣𝑎𝑙𝑖𝑑 ,𝐺 ( 𝑉, ¯ 𝐸 ))
Revised: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑣𝑎𝑙𝑖𝑑 ,𝐺 ( 𝑉, ¯ 𝐸 ))

Original: → 𝛼 ; end #Reduce search space and augment edges ¯ 𝐸 ← PruneAndGrow
Revised: → 𝛼 ; end #Reduce search space and augment edges ¯ 𝐸 ← PruneAndGrow

Original: ( 𝑓 ( . ; 𝑊 𝑠 ))
Revised: ( 𝑓 ( . ; 𝑊 𝑠 ))

Original: ; searchSpace ← ReduceSearchSpace (searchSpace, 𝛼 ); end 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ;[Table]
Revised: ; searchSpace ← ReduceSearchSpace (searchSpace, 𝛼 ); end 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ;

Original: A.2 Best Model’s architectures

Original: [Table]

Original: For each dataset

Original: [Table]

Original: we report its best architecture found.

Original: [Table]

Original: – layer-wise attention: [’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’tanh’]

Original: – layer-wise skip-connection: [True, True, True, True, True, True, True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • Texas

Original: – layer-wise attention: [’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish’, ’swish’, ’tanh’, ’swish’, ’swish’, ’swish’, ’swish’]

Original: – layer-wise skip-connection: [True, True, True, True, True, True, True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • Wisconsin

Original: – layer-wise attention: [’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’leakyRelu’]

Original: – layer-wise skip-connection: [True, True, True, True, True, True, True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • Cornell

Original: – layer-wise attention: [’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’swish’, ’elu’, ’swish’]

Original: – layer-wise skip-connection: [True, True, True, True, True, True, True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • Computer

Original: – layer-wise attention: [’gcn’, ’generalizedLinear’, ’generalizedLinear’, ’linear’, ’linear’, ’linear’,

Original: ’linear’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish’, ’swish’, ’swish’, ’tanh’, ’swish’, ’tanh’, ’tanh’]

Original: – layer-wise skip-connection: [True, True, True, True, True, True, True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • CS

Original: – layer-wise attention: [’gcn’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish]

Original: – layer-wise skip-connection: [True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • Photos

Original: – layer-wise attention: [’gcn’, ’gcn’]

Original: – layer-wise activation: [’prelu’, ’elu’]

Original: – layer-wise skip-connection: [True, True]

Original: – Jumping-Knowledge aggregation: mean

Original: • Physics

Original: – layer-wise attention: [’gcn’, ’gcn’, ’cos’, ’gcn’]

Original: – layer-wise activation: [’swish’, ’swish’, ’leakyRelu’, ’relu’]

Original: – layer-wise skip-connection: [True, True, True, True]

Original: – Jumping-Knowledge aggregation: mean