Original: AutoCoG: A Unified Data-Model Co-Search Framework for Graph Neural Networks
Revised: AutoCoG: A Unified Data-Model Co-Search Framework for Graph Neural Networks

Original: Abstract Neural architecture search (NAS) has demonstrated success in discovering promising architectures for vision or language modeling tasks, and it has recently been introduced to searching for graph neural networks (GNNs) as well.
Revised: Abstract Neural architecture search (NAS) has demonstrated success in discovering promising architectures for vision or language modeling tasks, and it has recently been introduced to searching for graph neural networks (GNNs) as well.

Original: Despite the preliminary success,
Revised: Despite the preliminary success,

Original: GNNs struggle in dealing with heterophily or low-homophily graphs where connected nodes may have different class labels and dissimilar features.
Revised: GNNs struggle in dealing with heterophily or low-homophily graphs where connected nodes may have different class labels and dissimilar features.

Original: To this end, we propose co-optimizing both the input graph topology and the model’s architecture topology simultaneously.
Revised: To this end, we propose co-optimizing both the input graph topology and the model’s architecture topology simultaneously.

Original: That yields AutoCoG , the first unified data-model co-search NAS framework for GNNs.
Revised: That yields AutoCoG , the first unified data-model co-search NAS framework for GNNs.

Original: By defining a highly flexible data-model co-search space, AutoCoG is gracefully formulated as a principled bi-level optimization that can be end-to-end solved by the differentiable search methods.
Revised: By defining a highly flexible data-model co-search space, AutoCoG is gracefully formulated as a principled bi-level optimization that can be end-to-end solved by the differentiable search methods.

Original: Experiments show that AutoCoG achieves an average performance gain across all datasets of 3.18% over the following best approach and ranks best against all other state-of-the-art methods with an average ranking of 2.5.
Revised: Experiments show that AutoCoG achieves an average performance gain across all datasets of 3.18% over the following best approach, and ranks best against all other state-of-the-art methods with an average ranking of 2.5.

Original: Code is available at https://github.com/VITA-Group/AutoCoG .
Revised: Code is available at https://github.com/VITA-Group/AutoCoG .

Original: 1 Introduction
Revised: 1 Introduction

Original: Graph neural networks (GNNs) have emerged as promising tools to analyze networked data in various real-world scenarios, such as social media (Grover and Leskovec, 2016) and biochemical graph analytics (Zitnik and Leskovec, 2017).
Revised: Graph neural networks (GNNs) have emerged as promising tools to analyze networked data in various real-world scenarios, such as social media (Grover and Leskovec, 2016) and biochemical graph analytics (Zitnik and Leskovec, 2017).

Original: Specifically, GNNs apply recursive message passing to learn the embedding representation of each node via aggregating the representations of its neighbors and itself.
Revised: Specifically, GNNs apply recursive message passing to learn the embedding representation of each node via aggregating the representations of its neighbors and itself.

Original: Motivated by the significant success of node embedding learning, plenty of GNN variants have been explored for the diverse downstream graph analysis tasks, including
Revised: Motivated by the significant success of node embedding learning, plenty of GNN variants have been explored for the diverse downstream graph analysis tasks, including

Original: GCN (Kipf and Welling, 2016a), GraphSAGE (Hamilton et al., 2018) and GCNII (Chen et al., 2020a).
Revised: GCN (Kipf and Welling, 2016a), GraphSAGE (Hamilton et al., 2018) and GCNII (Chen et al., 2020a).

Original: However, training GNNs is notoriously challenging, more so when they are train under heterophily or disassociative graphs, not to mention deep GNNs (Chen et al., 2020a).
Revised: However, training GNNs is notoriously challenging, more so when they are trained under heterophily or disassociative graphs, not to mention deep GNNs (Chen et al., 2020a; Zhou et al., 2020).

Original: First, since graphs abstract diverse data sources and present tremendous heterogeneity, the success of GNNs is often accompanied by extensive tuning of model architectural hyperparameters to characterize specific graph data.
Revised: First, since graphs abstract diverse data sources and present tremendous heterogeneity, the success of GNNs is often accompanied by extensive tuning of model architectural hyperparameters to characterize specific graph data.

Original: For example, it was reported that graph attention networks (GATs) (Veličković et al., 2018) are sensitive to the number of attention heads, which has to be carefully searched for the citation networks and the protein-protein interaction data, respectively.
Revised: For example, it was reported that graph attention networks (GAT) (Veličković et al., 2018) are sensitive to the number of attention heads, which has to be carefully searched for the citation networks and the protein-protein interaction data, respectively.

Original: Second, in the real world, graphs often opposites attract which inevitably lead to noisy setting where GNNs tend to suffer from overfitting and generalize poorly to the unseen testing data.
Revised: Second, in the real world, graphs often opposites attract which inevitably lead to noisy setting, where GNNs tend to suffer from overfitting and generalize poorly to the unseen testing data.

Original: Third, despite the potential of deep GNNs in learning the informative high-order neighborhood, the training of deep GNNs is widely known to be limited by the issues of over-smoothing, gradient vanishing, and over-squashing (Chen et al., 2020a).
Revised: Third, despite the potential of deep GNNs in learning the informative high-order neighborhood, the training of deep GNNs is widely known to be limited by the issues of over-smoothing, gradient vanishing, and over-squashing (Chen et al., 2020a).

Original: Recently, the automated graph neural architecture search (NAS), graph augmentation tricks, and deeper architectures have been independently proposed to tackle the above GNN training challenges partially.
Revised: Recently, the automated graph neural architecture search (NAS), graph augmentation tricks, and deeper architectures have been independently proposed to tackle the above GNN training challenges partially.

Original: Expressly, most of the existing automated efforts are limited to neural architecture tuning, while graph augmentation is often overlookedand untouched despite often being effective to gain performance (Li and King, 2020; Zhou et al., 2019a).
Revised: Expressly, most of the existing automated efforts are limited to neural architecture tuning, while graph augmentation is often overlooked despite often being effective to gain performance (Li and King, 2020; Zhou et al., 2019a).

Original: This is primarily because changes to the existing graph structure can have a cascading effect on the process of information aggregation, which adds a new layer of complexity above the already complex architecture tuning problem.
Revised: This is primarily because changes to the existing graph structure could have a cascading effect on the process of information aggregation, but adds a new layer of complexity to the above complex architecture tuning problem.

Original: Additionally, existing GNN NAS works are known to scale poorly in deeper architectures.
Revised: Additionally, existing GNN NAS works are known to scale poorly in deeper architectures.

Original: This is primarily due  the exploding search space which makes training unstable.
Revised: This is primarily due to the exploding search space which makes searching unstable.

Original: Previous efforts have limited themselves in searching the shallow GNNs with less than 3 layers.
Revised: Previous efforts have limited themselves in searching the shallow GNNs with less than 3 layers.

Original: Finally, Figure(1) illustrates circumstances where the aggregation mechanism fails due to unfavorable graph topology thus, it remains a daunting task to optimize the design philosophy for GNNs comprehensively.
Revised: Finally, Figure(1) illustrates circumstances where the aggregation mechanism fails due to unfavorable graph topology thus, it remains a daunting task to optimize the design philosophy for GNNs comprehensively.

Original: [Figure]
Revised: [Figure]

Original: To bridge the gaps, we propose AutoCoG , the first NAS framework towards unified datamodel co-search for GNNs to specifically tame the problem training under heterophily condition.
Revised: To bridge the gaps, we propose AutoCoG , the first NAS framework towards unified datamodel co-search for GNNs to specifically tame the problem training under heterophily condition.

Original: Besides automatically optimizing the
Revised: Besides automatically optimizing the

Original: GNN model architecture, we propose to simultaneously optimize the input graph topology, via progressively growing and pruning using a separate GNN model to learn to attune the graph to the proposed architecture.
Revised: GNNs’ neural architecture, we propose to simultaneously optimize the input graph topology, via progressively growing and pruning using a separate GNN model to learn to attune the graph to the proposed architecture.

Original: Additionally, by defining the highly flexible data-model co-search space, AutoCoG is formulated as a principled bi-level optimization that can be endto-end solved by the differentiable search methods.
Revised: Additionally, by defining the highly flexible datamodel co-search space, AutoCoG is formulated as a principled bi-level optimization that can be end-to-end solved by the differentiable search methods.

Original: To scale up our core framework to searching deep GNN architectures, we curb an explosive search space as the number of layers increased by performing multiple searching stages with increasing depth, as inspired by Chen et al. (2019c).
Revised: To scale up our core framework to searching deep GNN architectures, we curb an explosive search space as the number of layers increased by performing multiple searching stages with increasing depth, as inspired by Chen et al. (2019c).

Original: Additionally, for each search stages, we evolve the graph by growing/pruning it at the same time.
Revised: Additionally, for each search stage, we evolve the graph by growing/pruning it at the same time.

Original: To stabilize the search landscape from the shifting topologies of graph and model, we further utilize (Chen et al., 2020a) to combat the over-smoothing/over-squashing issues.
Revised: To stabilize the search landscape from the shifting topologies of graph and model, we further utilize (Chen et al., 2020a) to combat the over-smoothing/over-squashing issues.

Original: Together, our framework ensures a reliable way to discover  powerful architectures, a stable model training environment, and state-of-the-art results to train graph different degree of homophily.
Revised: Together, our framework ensures a reliable way to discover the powerful architectures, a stable model training environment, and state-of-the-art results to train graphs of different degrees of homophily.

Original: AutoCoG searches for and trains on deep or shallow graph neural networks to successfully achieve state-of-the-art results in Web datasets, Actor, Coauthor, and Wikipedia benchmarks.
Revised: AutoCoG searches for and trains on deep or shallow graph neural networks to successfully deliver superior results in Web datasets, Actor, Coauthor, and Wikipedia benchmarks.

Original: In summary, our three contributing novelties are:
Revised: In summary, our three contributing novelties are:

Original: • We propose AutoCoG , the first NAS framework towards unified data-model co-search for GNNs.Our
Revised: • We propose AutoCoG , the first NAS framework towards unified data-model co-search for GNNs.

Original: novel bi-level optimization formulation uniquely enables the end-to-end discovery of stateof-the-art GNN model and graph  altogether.
Revised: Our bi-level optimization formulation uniquely enables the end-to-end discovery of GNNs’ neural architecture and graph topology altogether.

Original: • We perform extensive analysis the resulting learned graph-structures for each benchmarks.
Revised: • We perform extensive analysis the resulting learned graph-structures for each benchmarks.

Original: To strengthen the co-search framework, we organically integrate several techniques to directly combat issues of searching unreliability, training instability, and scalability, that have previously plagued NAS approaches for searching deeper GNNs.
Revised: To strengthen the co-search framework, we organically integrate several techniques to directly combat issues of searching unreliability, training instability, and scalability, that have previously plagued NAS approaches for searching deeper GNNs.

Original: • Experiments show that AutoCoG achieves an average performance gain across all datasets of 3.18% over the following best approach and ranks best against all other state-of-the-art methods with an average ranking of 2.5.
Revised: • Experiments show that AutoCoG achieves an average performance gain across all datasets of 3.18% over the following best approach, and ranks best against all other state-of-the-art methods with an average ranking of 2.5.

Original: 2 Related works
Revised: 2 Related works

Original: Graph neural networks.
Revised: Graph neural networks.

Original: Motivated by the state-of-the-art results of GNNs in graph analytics, there have been numerous GNN variants (Bruna et al., 2013; Hamilton et al., 2017; Xu et al., 2019;
Revised: Motivated by the state-of-the-art results of GNNs in graph analytics, there have been numerous GNN variants (Bruna et al., 2013; Hamilton et al., 2017; Xu et al., 2019;

Original: Chen et al., 2020a; Wu et al., 2019a).
Revised: Chen et al., 2020a; Wu et al., 2019a; Zhou et al., 2021b).

Original: Most of these existing approaches fit within the category of spatial GNNs.
Revised: Most of these existing approaches fit within the category of spatial GNNs.

Original: Namely, following the spatial message passing strategy, the core idea of GNNs is to learn the embedding representation of a node by aggregating the embeddings of its neighbors and node itself recursively.
Revised: Namely, following the spatial message passing strategy, the core idea of GNNs is to learn the embedding representation of a node by aggregating the embeddings of its neighbors and node itself recursively.

Original: The previous empirical studies show that GNNs often achieve the best performance with less than 3 layers (Kipf and Welling, 2016a; Veličković et al., 2018).
Revised: The previous empirical studies show that GNNs often achieve the best performance with less than 3 layers (Kipf and Welling, 2016a; Veličković et al., 2018).

Original: Key limitations of GNNs are their performances decrease significantly with the increasing of model depth and the degree of graph homophily they operate on.
Revised: Key limitations of GNNs are their performances decrease significantly with the increasing of model depth and the degree of graph homophily they operate on.

Original: As the graph convolutional layer increases, the node representations will converge to indistinguishable vectors due to the recursive neighborhood aggregation and non-linear activation (Li et al., 2018;
Revised: As the graph convolutional layer increases, the node representations will converge to indistinguishable vectors due to the recursive neighborhood aggregation and non-linear activation (Li et al., 2018;

Original: Oono and Suzuki, 2020), which is well known as the over-smoothing issue (NT and Maehara, 2019; Chen et al., 2019a; Alon and Yahav, 2020; Chien et al., 2021; Huang et al., 2020).
Revised: Oono and Suzuki, 2020; Zhou et al., 2021a; Guo et al., 2021), which is well known as the over-smoothing issue (NT and Maehara, 2019; Chen et al., 2019a; Alon and Yahav, 2020; Chien et al., 2021; Huang et al., 2020).

Original: Graph augmentation.
Revised: Graph augmentation.

Original: Data augmentation methods has been widely applied to improve the generalization performances of deep neural networks, such as convolutional and recurrent neural networks (Shorten and Khoshgoftaar, 2019; Antoniou et al., 2017;
Revised: Data augmentation methods has been widely applied to improve the generalization performances of deep neural networks, such as convolutional and recurrent neural networks (Shorten and Khoshgoftaar, 2019; Antoniou et al., 2017;

Original: Feng et al., 2021).
Revised: Feng et al., 2021).

Original: They aim to craft the out-of-distribution training data to avoid overfitting with the customized augmentation policies.
Revised: They aim to craft the out-of-distribution training data to avoid overfitting with the customized augmentation policies.

Original: In the graph analytics, GNNs are prone to overfit the naturally noisy training graphs, which may miss the ground-truth nodes/edges or contain the erroneous information (Zügner et al., 2018).
Revised: In the graph analytics, GNNs are prone to overfit the naturally noisy training graphs, which may miss the ground-truth nodes/edges or contain the erroneous information (Zügner et al., 2018).

Original: Different from the grid-like image data, the graph augmentation is often operated on the adjacency structure or node features (You et al., 2020b, 2021, 2022).
Revised: Different from the grid-like image data, the graph augmentation is often operated on the adjacency structure or node features (You et al., 2020b, 2021, 2022; Lai et al., 2020).

Original: Existing graph augmentations could be categorized into the following two classes.
Revised: Existing graph augmentations could be categorized into the following two classes.

Original: (i)
Revised: i)

Original: The random augmentation either drops/adds edges to modify the graph, or masks parts of the node features (Rong et al., 2020a;
Revised: The random augmentation either drops/adds edges to modify the graph, or masks parts of the node features (Rong et al., 2020a;

Original: You et al., 2020b; Feng et al., 2020).
Revised: You et al., 2020b; Feng et al., 2020).

Original: (ii)
Revised: (ii)

Original: The differentiable augmentation learns to optimize the adjacency affinity matrix by minimizing the concerned task loss.
Revised: The differentiable augmentation learns to optimize the adjacency affinity matrix by minimizing the concerned task loss.

Original: Based upon the computed affinity matrix, the differentiable augmentation either continuously combines it into the original adjacency matrix (Zhao et al., 2020b; Chen et al., 2020b), or samples the discrete edges to formulate new graph (Chen et al., 2019b).
Revised: Based upon the computed affinity matrix, the differentiable augmentation either continuously combines it into the original adjacency matrix (Zhao et al., 2020b; Chen et al., 2020b), or samples the discrete edges to formulate new graph (Chen et al., 2019b).

Original: Neural architecture search.
Revised: Neural architecture search.

Original: Targeting at alleviating the laborious hyperparameter tuning, NAS automates the designing of good neural architectures for any a given application.
Revised: Targeting at alleviating the laborious hyperparameter tuning, NAS automates the designing of good neural architectures for any a given application.

Original: It is shockingly reported that the searched neural architectures could outperform the human-designed ones in many real-world scenarios, such as image classification (Zoph and Le, 2016; Zoph et al., 2018) and generation (Wang and Huan, 2019;
Revised: It is shockingly reported that the searched neural architectures could outperform the human-designed ones in many real-world scenarios, such as image classification (Zoph and Le, 2016; Zoph et al., 2018) and generation (Wang and Huan, 2019;

Original: Gong et al., 2019).
Revised: Gong et al., 2019).

Original: Most of NAS frameworks apply one of the following search algorithms: reinforcement learning (RL) (Pham et al., 2018; Baker et al., 2016), evolution algorithm (EA) (Liu et al., 2017; Miikkulainen et al., 2019; Xie and Yuille, 2017), and one-shot differentiable search (Liu et al., 2018; Zela et al., 2020).
Revised: Most of NAS frameworks apply one of the following search algorithms: reinforcement learning (RL) (Pham et al., 2018; Baker et al., 2016), evolution algorithm (EA) (Liu et al., 2017; Miikkulainen et al., 2019; Xie and Yuille, 2017), and one-shot differentiable search (Liu et al., 2018; Zela et al., 2020).

Original: There are several recent efforts to conjoin the researches of GNNs and NAS (Gao et al., 2019;
Revised: There are several recent efforts to conjoin the researches of GNNs and NAS (Gao et al., 2019;

Original: Zhou et al., 2019b; You et al., 2020a;
Revised: Zhou et al., 2019a; You et al., 2020a;

Original: Ding et al., 2020; Zhao et al., 2020a).
Revised: Ding et al., 2020; Zhao et al., 2020a).

Original: However, all of them are limited in exploring the shallow
Revised: However, all of them are limited in exploring the shallow

Original: GNNs, and fail to denoise the underlying graph to further ameliorate the model performance.
Revised: GNNs, and fail to denoise the underlying graph to further ameliorate the model performance.

Original: In this work, we aim to simultaneously search the deep GNN models and graph structure to optimize the downstream graph analytics.
Revised: In this work, we aim to simultaneously search the deep GNN models and graph structure to optimize the downstream graph analytics.

Original: Co-Adaptive Search between graph’s structure and model’s architecture.
Revised: Co-adaptive search between graph’s structure and model’s architecture.

Original: GASSO (Qin et al., 2021) is a recent work that similarly proposes the idea of model-graph co-search.
Revised: GASSO (Qin et al., 2021) is a recent work that similarly proposes the idea of model-graph co-search.

Original: Yet two differentiation factors uniquely defined our two works.
Revised: Yet two differentiation factors uniquely defined our two works.

Original: Firstly, GASSO is a technique that learns attention coefficients G ∈ [0, 1] only for existing edges E, which is mathematically equivalent to graph attention neural networks (Qin et al., 2021).
Revised: Firstly, GASSO is a technique that learns attention coefficients G ∈ [0, 1] only for existing edges E, which is mathematically equivalent to graph attention neural networks (Qin et al., 2021).

Original: In contrast, Auto-CoG directly modifies the graph’s structure via pruning poorly attended edges and adding new un-seen edges.
Revised: In contrast, Auto-CoG directly modifies the graph’s structure via pruning poorly attended edges and adding new un-seen edges.

Original: Thus the derived graph structure is unique from the original underlying graph.
Revised: Thus the derived graph structure is unique from the original underlying graph.

Original: Finally, GASSO employs a coarse macro-level search space with only eight operators and two layers.
Revised: Finally, GASSO employs a coarse macro-level search space with only eight operators and two layers.

Original: Its design decision space is shallow (small), consisting of only 256 unique combinations.
Revised: Its design decision space is shallow (small), consisting of only 256 unique combinations.

Original: Theoretically, by searching the optimal attention function in our Auto-Cog , we could approximate the "attentional structure learning" in GASSO.
Revised: Theoretically, by searching the optimal attention function in our Auto-Cog , we could approximate the "attentional structure learning" in GASSO.

Original: 3 Methodology
Revised: 3 Methodology

Original: Preliminary.
Revised: Preliminary.

Original: We briefly review the basic of a message-passing based graph convolution network (GCN) and the definition of homophily.
Revised: We briefly review the basic of the definition of homophily and the message-passing based graph convolution networks (GCN).

Original: First, the homophily or edge-homophily ratio, of a graph measures the ratio between intra-node pairs ( 𝑣,𝑤 ) overall all edges 𝐸 and is given as:
Revised: The homophily or edge-homophily ratio of a graph measures the ratio between intra-node pairs ( 𝑖, 𝑗 ) overall all edges E and is given as:

Original: [Equation]
Revised: [Equation]

Revised: where E denotes the set of edges in the graph, and | E | denotes the cardinality of edges.

Original: Second, given a GCN, its 𝑘 -th layer  could bewritten generally  as:[Equation] 𝑥
Revised: Being defined as the message passing along the edges of graph, the 𝑘 -th layer of GNNs could be generally written as:

Original: [Equation]
Revised: [Equation]

Original: 𝑖 denotes the node embedding of element 𝑖 at  𝑘 -th layer.
Revised: denotes the node embedding of node 𝑖 at the 𝑘 -th layer.

Original: 𝑊 ( 𝑘 ) ∈ R 𝐷 × 𝐷 represents the learnable layer-wise weights for all { 𝑥 𝑖 : 𝑖 ∈ | 𝑉 |} , where| 𝑉 | is our total number of nodes and 𝐷 our number of hidden features.
Revised: 𝑊 ( 𝑘 ) ∈ R 𝐷 × 𝐷 represents the learnable layer-wise weights shared by all the nodes, where 𝐷 denotes the dimension of hidden units.

Original: 𝑎 ( 𝑘 ) 𝑖𝑗 dictates the attention coefficient between  𝑖 and 𝑗 derived from some Attention function.
Revised: 𝑎 ( 𝑘 ) 𝑖𝑗 dictates the attention coefficient between nodes 𝑖 and 𝑗 derived from some Attention functions.

Original: N ( 𝑖 ) denotes the  neighboring nodes of node 𝑖from a graph 𝐺 .
Revised: N ( 𝑖 ) denotes the set of neighboring nodes of node 𝑖 .

Original: ℎ ( 𝑘 ) 𝑖 is the resulting embeddings after applying an AGGR function to aggregate a set of neighboring embeddings from the previous 𝑘 − 1 layers.
Revised: ℎ ( 𝑘 ) 𝑖 is the resulting embeddings after applying an AGGR function to aggregate the set of neighboring embeddings from the previous layer.

Original: In addition, function COMB incorporates information from  itself with its neighboring embeddings ℎ ( 𝑘 ) 𝑖 , and 𝜎 provides the nonlinear activation.
Revised: In addition, function COMB incorporates information from the node itself with its neighboring embeddings ℎ ( 𝑘 ) 𝑖 , and 𝜎 provides the nonlinear activation.

Original: 3.1 Unified Data-Model Co-Search Space3.1.1 Model Search Space: Attention, Activation, and Skip Connection.
Revised: 3.1 Unified Data-Model Co-Search Space

Revised: where || denotes the concatenation operation, ¯ 𝑎, ¯ 𝑎 𝑖 , ¯ 𝑎 𝑗 denote learnable vectors, 𝑊 𝐺

Original: [Table]
Revised: [Table]

Revised: 1 √ | N ( 𝑖 )|| N ( 𝑗 )|

Revised: COS ¯ 𝑎

Original:  ) 𝑥 ( 𝑘 − 1 )𝑗 )
Revised: ( 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 )

Revised: 𝑖 ||

Original:  LINEAR tanh ( ¯ 𝑎 𝑙 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖
Revised: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 ) LINEAR tanh ( ¯ 𝑎 𝑙 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖

Original: || 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 )
Revised: || 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 )

Revised: GERE

Original: [Table]
Revised: [Table]

Original:  𝑊 [Table] tanh ( 𝑊 ( 𝑘 ) 𝑥  𝑘 − 1 ) 𝑖
Revised: LINEAR 𝑊 𝐺 tanh ( 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖

Original: + 𝑊 ( 𝑘 ) 𝑥
Revised: + 𝑊 ( 𝑘 ) 𝑥

Original: ( 𝑘 − 1 ) 𝑖 )
Revised: ( 𝑘 − 1 ) 𝑖 )

Original: [Table]
Revised: [Table]

Original: LeakyReLU
Revised: LeakyReLU

Original: ( ¯ 𝑎 ( 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖 ||
Revised: ( ¯ 𝑎 ( 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑖 ||

Original: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 ))
Revised: 𝑊 ( 𝑘 ) 𝑥 ( 𝑘 − 1 ) 𝑗 ))

Original: [Table]
Revised: [Table]

Original: SUM 𝑎 ( 𝑘 ) 𝑖𝑗 + 𝑎 ( 𝑘 )
Revised: ( 𝑘 )

Original: [Table]
Revised: [Table]

Original: CONST
Revised: 𝑘 )

Original: [Table]
Revised: [Table]

Original: Defining the Model Search Space. 
Revised: 3.1.1 Model Search Space. The

Original: The design of model search space should achieve a balanced trade-off between the diversity and efficiency.
Revised: design of model search space should achieve a balanced tradeoff between the diversity and efficiency Zhou et al.

Original: Although a large search space subsumes the diverse GNN architectures to adapt to the different graph analysis tasks, it would be extremely time-consuming to explore the optimal design.
Revised: Although a large search space subsumes the diverse GNN architectures to adapt to the different graph analysis tasks, it would be extremely time-consuming to explore the optimal design.

Original: In the existing search spaces of
Revised: In the existing search spaces of

Original: GNNs (Gao et al., 2019; Zhou et al., 2019a; You et al., 2020a), they often contain the architecture components of hidden units, attention, aggregation, combination, and activation functions, as well as the skip connections.
Revised: GNNs (Gao et al., 2019; Zhou et al., 2019b; You et al., 2020a), they often contain the architecture components of hidden units, attention, aggregation, combination, and activation functions, as well as the skip connections.

Original: To efficiently search the outperforming shallow and deep GNNs, we compare the effectiveness of each component, and greatly shrink down the search space to focus on three key components: the Activation function, the Attention module, and the skip connections.
Revised: To efficiently search the outperforming shallow and deep GNNs, we compare the effectiveness of each component, and greatly shrink down the search space to focus on three key components: the activation function, the attention module, and the skip connections.

Original: They are generally believed to impact GNN’s expressive capability and depth scalability (Chen et al., 2021b).
Revised: They are generally believed to impact GNN’s expressive capability and depth scalability (Chen et al., 2021b).

Original: We fix the Aggregation function and Combination function to be simple summation, and treat the hidden units as hyperparameter.
Revised: We fix the aggregation function and combination function to be simple summation, and treat the hidden units as hyperparameter.

Original: Below we lay out our searchable design for them one-by-one:
Revised: Below we lay out our searchable design for them one-by-one:

Original: • Attention Search Space : Attention mechanism has been shown by (Veličković et al., 2018) to effectively stabilize training by placing proper neighborhood scaling with attention coefficient 𝑎 𝑖𝑗 .
Revised: • Attention search space : Attention mechanism has been shown by (Veličković et al., 2018) to effectively stabilize training by placing proper neighborhood scaling with attention coefficient 𝑎 𝑖𝑗 .

Original: We list our attention choices in Table 1.
Revised: We list our attention choices in Table 1.

Original: • Activation Search Space : for basic activation functions, we search among these operations { ReLU , Sigmoid , Tanh , Linear , SoftPlus , LeakyReLU , ReLU , ELU } .
Revised: • Activation search space : For the activation functions, we search among these operations: { ReLU , Sigmoid , Tanh , Linear , SoftPlus , LeakyReLU , ReLU6 , ELU } .

Original: • Skip Connection Search Space : For an 𝐿 -Layer GCN, various skip connections can be applied to overcome the effect of over-smoothing.
Revised: • Skip connection search space : For a 𝐿 -Layer GNN, various skip connections can be applied to overcome the effect of over-smoothing.

Original: Previous deep GCN works (Chen et al., 2020a; Zhang et al., 2020; Chen et al., 2021b) illustrated a significant correlation between the type of skipconnections to the  performance.
Revised: Previous deep GNN works (Chen et al., 2020a; Zhang et al., 2020; Chen et al., 2021b) illustrate a significant correlation between the type of skip connections and the model performance.

Original: We include three skip-connection types — i) Initial Connection, ii) Jumping-Knowledge aggregation
Revised: We include two categories of skip connections: (i) Initial Connection, and (ii) Jumping-Knowledge aggregation.

Original: l a y e rl a y e rl a y e rl a y e rl a y e rl a y e r- …
Revised: l a y e rl a y e rl a y e rl a y e rl a y e rl a y e r- …

Original: A tt en t i on
Revised: A tt en t i on

Original: A gg r ega t e
Revised: A gg r ega t e

Original: C o m b
Revised: C o m b

Original: i ne
Revised: i ne

Original: A c t i v a t i on
Revised: A c t i v a t i on

Original: S k ip
Revised: S k i

Original: [Figure]
Revised: [Figure]

Original: .2 Graph Augmentation.
Revised: Graph Augmentation.

Original: We often expect a clustering of liked nodes when operating on graph data in a "like attracts like" world.
Revised: We often expect a clustering of liked nodes when operating on graph data in a “like attracts like" world.

Original: However, in reality, when modeling complex relationships, we often observe the opposite, where node identities are often best described by contrasting with their neighbors in "different attract" relationships.
Revised: However, in reality, when modeling complex relationships, we may observe the opposite, where node identities are often best described by contrasting with their neighbors in "different attract" relationships.

Original: Under such heterophily circumstance, GNNs’ performances degrade (Pei et al., 2020a; Zhu et al., 2020a; Battaglia et al., 2018), which makes sense intuitively, since relating unrelated nodes can lead to class obscurity (Zhu et al., 2020a), i.e over-smoothing.
Revised: Under such heterophily circumstance, GNNs’ performances degrade (Pei et al., 2020a; Zhu et al., 2020a; Battaglia et al., 2018), which makes sense intuitively, since aggregating the unrelated neighbors can lead to class obscurity (Zhu et al., 2020a), i.e over-smoothing.

Original: Thus overcoming heterophily with model architecture alone is difficult and often requires complicated, and exotic works flow (Abu-El-Haija et al., 2019;
Revised: Thus overcoming heterophily with model architecture alone is difficult and often requires complicated, and exotic works flow (Abu-El-Haija et al., 2019;

Original: Pei et al., 2020b; Lim et al., 2021).
Revised: Pei et al., 2020b; Lim et al., 2021).

Original: Lately, a number of works (Srivastava et al., 2014; Zou et al., 2019;
Revised: Lately, a number of works (Srivastava et al., 2014; Zou et al., 2019;

Original: Rong et al., 2020b; Chen et al., 2021a; Huang et al., 2021) have found that direct graph augmentations with stochastic policies — drop/add edges — can decelerate both the over-fitting and over-smoothing issues in training deep GNNs.
Revised: Rong et al., 2020b; Chen et al., 2021a; Huang et al., 2021) have found that direct graph augmentations with stochastic policies — drop/add edges — can decelerate both the over-fitting and over-smoothing issues in training deep GNNs.

Original: By learning the graph’s topology and the model’s architecture, we naturally adapt our data structure around the model’s strength and co-optimize data flow around the message-aggregation mechanism.
Revised: By learning the graph’s topology and the model’s architecture, we naturally adapt our data structure around the model’s strength, and co-optimize data flow around the message passing mechanism.

Original: The scoring function.
Revised: The scoring function.

Original: Given graph 𝐺 ( 𝑉, 𝐸 ) can be expressed in the form of an adjacent matrix 𝐴 ∈ R | 𝑉 |×| 𝑉 | , where 𝑉 is the set of vertices and 𝐸 is the set of edges.
Revised: Graph 𝐺 ( V , E ) can be expressed in the form of an adjacent matrix 𝐴 ∈ R | V |×| V | , where V is the set of vertices and E is the set of edges.

Original: We learn an edge score matrix 𝑆 ∈ R | 𝑉 |×| 𝑉 | such that we rewrite the aggregation step in Eqn (2) as:
Revised: We learn an edge score matrix 𝑆 ∈ R | V |×| V | such that we rewrite the aggregation step in Eqn (2) as:

Original: [Equation]
Revised: [Equation]

Original: where 𝐻 ( 𝑘 ) = { ℎ ( 𝑘 ) 𝑖 : 0 ≤ 𝑖 ≤ | 𝑉 |} and 𝑋 ( 𝑘 − 1 )=
Revised: where 𝐻 ( 𝑘 ) = { ℎ ( 𝑘 ) 𝑖 : 0 ≤ 𝑖 ≤ | V |} and 𝑋 ( 𝑘 − 1 )

Original:  { 𝑥 ( 𝑘 − 1 ) 𝑖 : 0 ≤ 𝑖 ≤ | 𝑉 |} .
Revised: = { 𝑥 ( 𝑘 − 1 ) 𝑖 : 0 ≤ 𝑖 ≤ | V |} .

Original: We formally denote 𝑆 ∈ [ 0 , 1 ] as:
Revised: We formally define 𝑆 ∈ [ 0 , 1 ] as:

Original: [Equation]
Revised: [Equation]

Original: Where 𝑓 ( . ; 𝑊 𝑠 ) is simply the classic VGAE model by (Kipf and Welling, 2016b), 𝑍 𝑉 𝑠𝑟𝑐 / 𝑍 𝑉 𝑡𝑔𝑡 are the source and target nodes available from 𝐺 ( 𝑉, 𝐸 ) , || denotes the concatenation function, and 𝜎 is the sigmoid function.
Revised: Where 𝑓 ( . ; 𝑊 𝑠 ) is simply the classic VGAE model by (Kipf and Welling, 2016b), 𝑍 𝑉 src / 𝑍 𝑉 src are the source and target nodes available from 𝐺 ( 𝑉, E ) , || denotes the concatenation function, and 𝜎 is the sigmoid function.

Original: 𝑆 is therefore a sparse matrix with only | 𝐸 | number of scores.
Revised: 𝑆 is therefore a sparse matrix with only | E | number of scores.

Original: Edge growing and pruning.
Revised: Edge growing and pruning.

Original: Taking  advantage ofthe Progressive NAS workflow (Chen et al., 2019c), at each searching stage, we prune the bottom 𝑝 -percentile from 𝑆 , and at the same-time we grow our graph by appending 𝑘 new edges for each nodes via embedding similarity.
Revised: Taking the advantage of Progressive NAS workflow (Chen et al., 2019c), at each searching stage, we prune the bottom 𝑝 -percentile from 𝑆 , and at the same-time we grow our graph by appending 𝑘 new edges for each node via embedding similarity.

Original: This embedding similarity function isbest defined as:
Revised: This embedding similarity function is defined as:

Original: [Equation]
Revised: [Equation]

Original: Where 𝐶𝑜𝑠𝑖𝑛𝑒 ( . ) is the cosine-similarity between two nodes’ scoring embeddings, while 𝐷  is the shortest distance between them.
Revised: Where Cosine (·) is the cosine similarity between two nodes’ scoring embeddings, while 𝐷 𝑖𝑗 is the shortest distance between them.

Original: We illustrate  visually in Figure(2) and in  code via Algorithm(1) in theour appendix.
Revised: We illustrate the process of graph topology modification visually in Figure 2 and in the pseudo code of Algorithm 1 in the appendix.

Original: 3.2 Optimization Formulation and Algorithm
Revised: 3.2 Optimization Formulation and Algorithm

Original: 3.2.1 A Principled Bi-Level Optimization Formulation.
Revised: 3.2.1 A Principled Bi-Level Optimization Formulation.

Original: For the sake of conciseness, we use 𝛼 as the model space architecture parameters , and denote L obj as the objective loss function given 𝛼 .
Revised: For the sake of conciseness, we use 𝛼 as the model space architecture parameters , and denote L obj as the objective loss function given 𝛼 .

Original: With 𝛼 defined, we further denote ˆ 𝑊 =
Revised: With 𝛼 defined, we further denote ˆ 𝑊 =

Original: 𝑊 (cid:199) 𝑚
Revised: 𝑊 (cid:199) 𝑚

Original: 𝛼 as the pruned sub-model from the supernet derived from 𝛼 description, where ˆ 𝑊 ,𝑚 𝛼 ∈ R 𝐿 × 𝐷 × 𝐷 , D denotes the size of hidden embeddings.
Revised: 𝛼 as the pruned sub-model from the supernet derived according to 𝛼 description, where ˆ 𝑊 ,𝑚 𝛼 ∈ R 𝐿 × 𝐷 × 𝐷 .

Original: Additionally, we can write our augmented graph ˆ 𝐺 as ˆ 𝐴 =
Revised: Additionally, we can write our augmented graph ˆ 𝐺 as ˆ 𝐴 =

Original: 𝐴 (cid:199) 𝑆 , where 𝑆 is defined as our learned scoring matrix.
Revised: 𝐴 (cid:199) 𝑆 , where 𝑆 is defined as our learned scoring matrix.

Original: Then, let 𝑍 represents our output vector for a hypothetical 2-layer AutoCoG :
Revised: Let 𝑍 represents the node embedding matrix for a hypothetical 2-layer AutoCoG :

Original: [Equation]
Revised: [Equation]

Original: Thus the objective loss function L obj for a transductive semi-supervised node classification tasks is formally denoted as:
Revised: Thus the objective loss function L obj for a transductive semi-supervised node classification tasks is formally denoted as:

Original: [Equation]
Revised: [Equation]

Original: Extending from (Dong and Yang, 2019), we formulate our data-model co-search as a joint bi-level optimization , to solve 𝛼, 𝑆 concurrently withthe weights 𝑊 and data space parameters:
Revised: Extending from (Dong and Yang, 2019), we formulate our data-model co-search as a joint bi-level optimization , to solve 𝛼, 𝑆 concurrently with weights 𝑊 and data space parameters:

Original: [Equation]
Revised: [Equation]

Original: Note that 𝛼 are optimized using the objective loss function on the validation set, while 𝑊 , 𝑆 are optimized under training set.
Revised: Note that 𝛼 are optimized using the objective loss function on the validation set, while 𝑊 , 𝑆 are optimized under training set.

Original: Additionally, ˆ 𝐺also consists of modified edges ¯ 𝐸 , not-shown explicitly in Equation (8), but is illustrated in our Algorithm(1).
Revised: Additionally, ˆ 𝐺 consists of modified edges, not-shown explicitly in Equation (8), but is illustrated in our Algorithm 1.

Original: We adopt the same hard-
Revised: We adopt the same hard-

Original: Gumbel-softmax trick (Jang et al., 2017) to differentially optimize architectural variables during search.
Revised: Gumbel-softmax trick (Jang et al., 2017) to differentially optimize architectural variables during search.

Original: 3.2.2 Scaling and Stabilizing the Search Thus, the bi
Revised: 3.2.2 Scaling And Stabilizing Search.

Original: -level

Original: optimization (

Original:  can be solved by differential search methods and we adopt the GDAS approach in (Dong and Yang, 2019) by default.
Revised: The bi-level optimization problem (8) can be solved by differential search methods, and we adopt the GDAS approach in (Dong and Yang, 2019) by default.

Original: However when exploringGCN deep  architectures and larger graphs the data/model search spaces grow exponentially 
Revised: However, when exploring deep GNN architectures and larger graphs, the model/data search spaces grow exponentially with the layer depth/graph size.

Original: with the layer depth/graph size, and they can be entangled to cause even more serious scalability challenge.
Revised: They can be entangled to cause even more serious scalability challenge.

Original: That is further amplified by the training difficulty and instability of deep GNNs (Chen et al., 2021b).
Revised: That is further amplified by the training difficulty and instability of deep GNNs (Chen et al., 2021b).

Original: Indeed, we observe that naively applying GDAS is prone to over smoothening and search collapse, only yielding very poor architectures when searching for more than three layers.
Revised: Indeed, we observe that naively applying GDAS is prone to over-smoothing and search collapse, only yielding very poor architectures when searching for more than three layers.

Original: Besides it is not uncommon for the derived graph and model to have considerable performance variations across repeated experiments, due the stochastic initialization and training.
Revised: Besides, it is not common for the derived graph and model to have considerable performance variations across repeated experiments, due the stochastic initialization and training.

Original: Progressive search space.
Revised: Progressive search space.

Original: We follow the idea proposed by (Chen et al., 2019c)
Revised: We follow the idea proposed by (Chen et al., 2019c)

Original: (also illustrated in Algorithm 1), to divide search into 𝑁 progressive stages, with each consecutive stage having a larger or equal number of layers than those previously.
Revised: (also illustrated in Algorithm 1), to divide search into 𝑁 progressive stages, with each consecutive stage having a larger or equal number of layers than those previously.

Original: At each stage, we greedily remove the least selected options (by taking the mean of Soft-Max across 𝐿 layers and removing the option with the smallest value) from the data’s 𝑝 parameters or model space, and pass on the shrunk co-search space to the next stage.
Revised: At each stage, we greedily remove the least selected options (by taking the mean of Soft-Max across 𝐿 layers and removing the option with the smallest value) from the data’s 𝑝 parameters or model space, and pass on the shrunk co-search space to the next stage.

Original: Note that we do not shrink the number of augmentation policies.
Revised: Note that we do not shrink the number of augmentation policies.

Original: 4 Experiments 4.1 Experimental settings.
Revised: 4 Experiments 4.1 Experimental Settings.

Original: We first list our shared training hype-parameters for all datasets, and then list each dataset’s specific particularity.
Revised: We first list our shared training hype-parameters for all datasets, and then list each dataset’s specific particularity.

Original: By default, we employ the Adam-optimizer (Kingma and Ba, 2017) to learn edges’ scores, model’s architecture and model’s weights with equal learning rate of 0 . 005, and a 𝐿 2 regularization of 0 . 0005.
Revised: By default, we employ the Adam-optimizer (Kingma and Ba, 2017) to learn edges’ scores, model’s architecture and model’s weights with equal learning rate of 0 . 005, and a 𝐿 2 regularization of 0 . 0005.

Original: We set the hidden-dimension 𝐷 to be 256 with a dropout rate of 0.6.
Revised: We set the hidden-dimension 𝐷 to be 256 with a dropout rate of 0.6.

Original: As for P-DARTS, for every stage we prune the bottom 10% of edges, and add one new edge per node, and the number of stages are set to be 4, starting from 2 layers, and with a 2 layers increment.
Revised: As for P-DARTS, for every stage we prune the bottom 10% of edges, and add one new edge per node, and the number of stages are set to be 4, starting from 2 layers, and with a 2 layers increment.

Original: Furthermore, for Identity-Mapping, 𝛾 is set to be 0.5.
Revised: Furthermore, for Identity-Mapping, 𝛾 is set to be 0.5.

Original: We search/train for 1000 epochs, while setting our rate of patient to be 400 and 200 respectively.
Revised: We search/train for 1000 epochs, while setting our rate of patient to be 400 and 200 respectively.

Original: To get our final results, we train the networktimes to get the average and standard deviation.
Revised: To get our final results, we train the networktimes to get the average and standard deviation.

Original: The only notable exception to the default settings are the Co-author datasets, where we set the dropout rate to be 0.8 and 0 for CS and Physics respectively.
Revised: The only notable exception to the default settings are the Co-author datasets, where we set the dropout rate to be 0.8 and 0 for CS and Physics respectively.

Original: We typically only search between two and eight layers.
Revised: We typically only search between two and eight layers.

Original: Finally, for all datasets, we average their accuracy over 10 runs, with random seed between 0 to 9.
Revised: Finally, for all datasets, we average their accuracy over 10 runs, with random seed between 0 to 9.

Original: 4.2 Ablation studies
Revised: 4.2 Ablation Studies

Original: [Table]
Revised: [Table]

Original: Model-Graph codependancy.
Revised: Model-graph codependancy.

Original: We justify the need for model-graph co-search by performing three experiments, namely — co-search, data-search, and model-search — to illustrate the respective effectiveness of the individual components which constitutes our framework.
Revised: We justify the need for model-graph co-search by performing three experiments, namely — co-search, data-search [Table] and model-search — to illustrate the respective effectiveness of the individual components which constitutes our framework.

Original: We collected these results from several datasets at a fixed depth of 8 while maintaining identical searching settings for all experiments.
Revised: We collected these results from several datasets at a fixed depth of 8 while maintaining identical searching settings for all experiments.

Original: Note that for data-search, we substitute our model with the vanilla GCN (Kipf and Welling, 2016a).
Revised: Note that for data-search, we substitute our model with the vanilla GCN (Kipf and Welling, 2016a).

Original: The results are collected in Table(2).
Revised: The results are collected in Table(2).

Original: Herein our results speak for themselves; we observe a significant improvement in performance, especially for graphs under heterophily, utilizing co-search over model-only and data-only search.
Revised: Herein our results speak for themselves; we observe a significant improvement in performance, especially for graphs under heterophily, utilizing co-search over model-only and data-only search.

Original: Correlation between depth and performance.
Revised: Correlation between depth and performance.

Original: We observe that an increasing depth does not always positively correlate to performance gain.
Revised: We observe that an increasing depth does not always positively correlate to performance gain.

Original: Homophily negatively correlates to our performance at depth.
Revised: Homophily negatively correlates to our performance at depth.

Original: To explain this phenomenon, we offer this hypothesis: since the number of layers in a model correlates to the number of k-hop neighbors observed, graphs under heterophily need to observe a much larger sub-graph to aggregate meaning information against the inherent noisy neighbors.
Revised: To explain this phenomenon, we offer this hypothesis: since the number of layers in a model correlates to the number of k-hop neighbors observed, graphs under heterophily need to observe a much larger sub-graph to aggregate meaning information against the inherent noisy neighbors.

Original: 4-layers 8-layers 16-layers 1.
Revised: 4-layers 8-layers 16-layers 1.

Original: R e l a t i v e p e r f o r m a n c e g a i n i n %
Revised: R e l a t i v e p e r f o r m a n c e g a i n i n %

Original: Depth to Performance correlation
Revised: [Figure]

Original: ActorTexasWisconsinCornell
Revised: ActorTexasWisconsinCornell

Original: In contrast, with an increasing degree of homophily, more layers may induce oversmoothing sooner, contributing to an overall degradation in performance.
Revised: In contrast, with an increasing degree of homophily, more layers may induce oversmoothing sooner, contributing to an overall degradation in performance.

Original: We investigate the relationship between the model’s depth and performance.
Revised: We investigate the relationship between the model’s depth and performance.

Original: As depth is a hyper-parameter, we perform a search on 2, 4, 8, and 16 layers configurations — while maintaining identical searching parameters — on several datasets at an increasing rate of homophily.
Revised: As depth is a hyper-parameter, we perform a search on 2, 4, 8, and 16 layers configurations — while maintaining identical searching parameters — on several datasets at an increasing rate of homophily.

Original: We then normalize our final accuracy results for each graph against the result of our 2-layer configuration to obtain relative performance gain in percentage.
Revised: We then normalize our final accuracy results for each graph against the result of our 2-layer configuration to obtain relative performance gain in percentage.

Original: We illustrate our results in Figure(3) [Table] [Table]
Revised: We illustrate our results in Figure(3).

Original: Analyzing the augmented graphs.
Revised: Analyzing the augmented graphs.

Original: We characterize the newly augmented graphs to understand better how they affect the overall performance by analyzing those searched under the eight-layers configuration.
Revised: We characterize the newly augmented graphs to understand better how they affect the overall performance by analyzing those searched under the eight-layers configuration.

Original: We first calculate the difference of the homophily rate between the new and old graphs.
Revised: We first calculate the difference of the homophily rate between the new and old graphs.

Original: Next, we count the difference of total informative edges, i.e., edges between nodes of similar classes.
Revised: Next, we count the difference of total informative edges, i.e., edges between nodes of similar classes.

Original: Finally, we calculate the Intersection between edges of the original to the augmented graph.
Revised: Finally, we calculate the Intersection between edges of the original to the augmented graph.

Original: Table(3) summarizes our findings.
Revised: Table(3) summarizes our findings.

Original: Herein we observe that our improved graphs do not exhibit a stronger rate of homophily — in comparison to the original graphs — as initially assumed.
Revised: Herein we observe that our improved graphs do not exhibit a stronger rate of homophily — in comparison to the original graphs — as initially assumed.

Original: However, we also observe that, despite the increasing heterophily, the searched graphs include more informative edges while retaining most of the original edges, indicating we are learning new and relevant unseen relationships.
Revised: However, we also observe that, despite the increasing heterophily, the searched graphs include more informative edges while retaining most of the original edges, indicating we are learning new and relevant unseen relationships.

Original: Nevertheless, this observation challenges the current research assumption on the correlation between heterophily and performance.
Revised: Nevertheless, this observation challenges the current research assumption on the correlation between heterophily and performance.

Original: We show that performance can still be achieved under low homophily given that enough informative edges are added to the graph and a deeper architecture. 
Revised: We show that performance can still be achieved under low homophily given that enough informative edges are added to the graph and a deeper architecture. [Table]

Original: Analyzing the method’s effectiveness and efficiency.
Revised: Analyzing the method’s effectiveness and efficiency.

Original: To evaluate the efficiency of our design, we compare its memory usage and total run time to other NAS-based approaches such as GraphNAS (Gao et al., 2019) and SANE (Zhao et al., 2021).
Revised: To evaluate the efficiency of our design, we compare its memory usage and total run time to other NAS-based approaches such as GraphNAS (Gao et al., 2019) and SANE (Zhao et al., 2021).

Original: Table(4) summarizes our findings.
Revised: Table(4) summarizes our findings.

Original: We could observe: that AutoCoG maintains relatively low memory utilization for each of the datasets tested, and AutoCoG is also the fastest model to complete both its full-search and training stages.[Table]
Revised: We could observe: that AutoCoG maintains relatively low memory utilization for each of the datasets tested, and AutoCoG is also the fastest model to complete both its full-search and training stages.

Original: Analyzing the effectiveness of progressive search.
Revised: Analyzing the effectiveness of progressive search.

Original: To test the benefit of progressive search, we perform three ablation studies on Texas, Wisconsin, and Cornell at various depths.
Revised: To test the benefit of progressive search, we perform three ablation studies on Texas, Wisconsin, and Cornell at various depths.

Original: As shown in
Revised: As shown in

Original: Table(5), note that, with progressive search, accuracy is positively correlated to a model’s depth,  while the opposite is true when searching without it.
Revised: Table(5), note that, with progressive search, accuracy is positively correlated to a model’s depth, [Table] while the opposite is true when searching without it.

Original: This is due to the search instability from the resulting search space size, and a deeper network only further exacerbates the problem.
Revised: This is due to the search instability from the resulting search space size [Table] and a deeper network only further exacerbates the problem.

Original: Indeed, we further observe that simply applying GDAS, as in the case of searching without progressive search, only yields poor architectures.
Revised: Indeed, we further observe that simply applying GDAS, as in the case of searching without progressive search, only yields poor architectures.

Original: The results align with our reasoning in section 3.2.2.
Revised: The results align with our reasoning in section 3.2.2.

Original: [Table]
Revised: [Table]

Original: Analyzing graph-model co-search improvement on model’s robustness.
Revised: Analyzing graph-model co-search improvement on model’s robustness.

Original: Table (6) summarizes our improved robustness on noisy graphs.
Revised: Table (6) summarizes our improved robustness on noisy graphs.

Original: We validate our robustness by performing node classification on graphs with added edges.
Revised: We validate our robustness by performing node classification on graphs with added edges.

Original: The number of new edges corresponds to a percentage relative to the total edges.
Revised: The number of new edges corresponds to a percentage relative to the total edges.

Original: From the table, we make the following observations: First, noisy graph data leads to poor model performance from a lack of robustness.
Revised: From the table, we make the following observations: First, noisy graph data leads to poor model performance from a lack of robustness.

Original: Second,
Revised: Second,

Original: Graph-search can rectify and improve the model’s robustness by removing artificial noise, leading to better model performance.
Revised: Graph-search can rectify and improve the model’s robustness by removing artificial noise, leading to better model performance.

Original: Third, random edges can help improve some sparse graphs base performance, as in the case of Cornell and Wisconsin, by enabling skip-connections between distance neighborhoods.
Revised: Third, random edges can help improve some sparse graphs base performance, as in the case of Cornell and Wisconsin, by enabling skip-connections between distance neighborhoods.

Original: [Table]
Revised: [Table]

Original: 4.3 Results
Revised: 4.3 Results

Original: [Table] We compare Auto-CoG to several notable state-of-the-methods inferencing on graphs with increasing degrees of associativity from 0.3 and 0.9.
Revised: We compare Auto-CoG to several notable state-of-the-methods inferencing on graphs with increasing degrees of associativity from 0.3 and 0.9.

Original: Additionally, we also include average improvement and average rank for quick performance comparison at a glance.
Revised: Additionally, we also include average improvement and average rank for quick performance comparison at a glance.

Original: Average improvement is the average accuracy difference between Auto-CoG and another model across all datasets, so a higher score indicates a better result.
Revised: Average improvement is the average accuracy difference between Auto-CoG and another model across all datasets, so a higher score indicates a better result.

Original: Average rank is a model’s average performance rank for all datasets, so lower is better.
Revised: Average rank is a model’s average performance rank for all datasets, so lower is better.

Original: For comparison, we include:
Revised: For comparison, we include:

Original: • NAS based graph models: for this category, we include GraphNAS(Gao et al., 2019), SANE(Zhao et al., 2021) and GASSO(Qin et al., 2021).
Revised: • NAS based graph models: for this category, we include GraphNAS(Gao et al., 2019), SANE(Zhao et al., 2021) and GASSO(Qin et al., 2021).

Original: • Handcrafted graph models: we compare against traditional designs such as GCN (Kipf and Welling, 2016a), SGC (Wu et al., 2019b), GAT (Veličković et al., 2018), GCNII (Chen et al., 2020a),  JKNet (Xu et al., 2018) and APPNP (Klicpera et al., 2018). 
Revised: • Handcrafted graph models: we compare against traditional designs such as GCN (Kipf and Welling, 2016a), SGC (Wu et al., 2019b), GAT (Veličković et al., 2018), GCNII (Chen et al., 2020a), [Table] JKNet (Xu et al., 2018) and APPNP (Klicpera et al., 2018). Additionally

Original: Additionally, we also compare against designs that are crafted specifically for disassortative graphs such as Geom-GCN (Pei et al., 2020c) and H2GCN (Zhu et al., 2020b).
Revised: [Table] we also compare against designs that are crafted specifically for disassortative graphs such as Geom-GCN (Pei et al., 2020c) and H2GCN (Zhu et al., 2020b).

Original: We summarizes our finding in Table(7).
Revised: We summarizes our finding in Table(7).

Original: From the results, we make the following observations:
Revised: From the results, we make the following observations:

Original: • Highlighting the challenge of heterophily, we observe the lack of a dominant approach that can outperform all datasets.
Revised: • Highlighting the challenge of heterophily, we observe the lack of a dominant approach that can outperform all datasets.

Original: However, when we compare their average ranking overall, we do find Auto-CoG ranks highly at 2.5 and able to improve against all other approaches on average.
Revised: However, when we compare their average ranking overall, we do find Auto-CoG ranks highly at 2.5 and able to improve against all other approaches on average.

Original: This showcase our method’s robustness in dealing with graphs under different homophily settings • In comparison to other NAS approaches, Auto-CoG reliably outperforms all of them when it comes to disassociative datasets since typical GNNs tend to over-smooth on noisy graph data — an inherent problem for message-passing.
Revised: This showcase our method’s robustness in dealing with graphs under different homophily settings • In comparison to other NAS approaches, Auto-CoG reliably outperforms all of them when it comes to disassociative datasets since typical GNNs tend to over-smooth on noisy graph data — an inherent problem for message-passing.

Original: Auto-CoG directly modifies its graph data and network’s architecture to overcome this weakness.
Revised: Auto-CoG directly modifies its graph data and network’s architecture to overcome this weakness.

Original: GASSO (Qin et al., 2021) also performs graph structure search, but it is limited to only learning existing edges attention coefficients and therefore is still susceptible to some degree of over-smoothing.
Revised: GASSO (Qin et al., 2021) also performs graph structure search, but it is limited to only learning existing edges attention coefficients and therefore is still susceptible to some degree of over-smoothing.

Original: • In comparison to handcrafted baselines, Auto-CoG comfortably outperforms Geom-GCN (Pei et al., 2020c ) on Actor and WebKB datasets.
Revised: • In comparison to handcrafted baselines, Auto-CoG comfortably outperforms Geom-GCN (Pei et al., 2020c) on Actor and WebKB datasets.

Original: Our graph-structured learning process provides a similar function as the "structural neighborhood" concept, which Geom-GCN utilizes for bi-level aggregation.
Revised: Our graph-structured learning process provides a similar function as the "structural neighborhood" concept, which Geom-GCN utilizes for bi-level aggregation.

Original: On the other hand, H2GCN (Pei et al., 2020c) shows impressive performance on small Webkb datasets, outperforming Auto-CoG in both Texas and Cornell.
Revised: On the other hand, H2GCN (Pei et al., 2020c) shows impressive performance on small Webkb datasets, outperforming Auto-CoG in both Texas and Cornell.

Original: However, the model’s ’ego-embeddings’ concept does not scale well on larger datasets such as CoAuthor and Amazon, where it repeatedly fails to produce competitive results.
Revised: However, the model’s ’ego-embeddings’ concept does not scale well on larger datasets such as CoAuthor and Amazon, where it repeatedly fails to produce competitive results.

Original: 5 Conclusion and Limitations
Revised: 5 Conclusion and Limitations

Original: In this paper, we present AutoCoG the first NAS framework towards unified data-model co-search for GNNs .
Revised: In this paper we present AutoCoG the first NAS framework towards unified data-model co-search for GNNs .

Original: Our results convincingly demonstrate the benefit of data-graph co-search for both deep and shallow graph neural networks.
Revised: Our results convincingly demonstrate the benefit of data-graph co-search for both deep and shallow graph neural networks.

Original: Our ablation study shows that controlled variances in graph heterophily can result in a better, more generalized model and the necessity for graph-augmentation to be model-aware.
Revised: Our ablation study shows that controlled variances in graph heterophily can result in a better, more generalized model and the necessity for graph-augmentation to be model-aware.

Original: We confidently demonstrate AutoCoG to be a reliable way to discover robust architectures, a stable model training environment, and state-of-the-art results.
Revised: We confidently demonstrate AutoCoG to be a reliable way to discover robust architectures, a stable model training environment, and state-of-the-art results.

Original: Additionally, we show that the localized disturbance of graph structure motivates node position learning, allowing for greater generalizability of the model.
Revised: Additionally, we show that the localized disturbance of graph structure motivates node position learning, allowing for greater generalizability of the model.

Original: However, there are still limitations that need to be addressed: large graph scalability and understanding heterophily’s relationship to performance.
Revised: However, there are still limitations that need to be addressed: large graph scalability and understanding heterophily’s relationship to performance.

Original: To address this, we first plan to follow up by learning meaningful model/graph using Auto-CoG via graph-batching.
Revised: To address this, we first plan to follow up by learning meaningful model/graph using Auto-CoG via graph-batching.

Original: Secondly, we want to conduct a study to understand better the phenomenon between heterophily and performance observed in our ablation.
Revised: Secondly, we want to conduct a study to understand better the phenomenon between heterophily and performance observed in our ablation.

Original: There is no negative societal impact to our best knowledge, except that the NAS process is resource-consuming - but even that excessive cost can be amortized by the re-usability of the searched model, which can achieve superior accuracy-resource trade-off.
Revised: There is no negative societal impact to our best knowledge, except that the NAS process is resource-consuming - but even that excessive cost can be amortized by the re-usability of the searched model, which can achieve superior accuracy-resource trade-off.

Original: A Appendix A.1 Datasets
Revised: A Appendix A.1 Datasets

Original: We evaluate AutoCoG on several popular semi-node classification datasets including (i) the WebKB datasets —Cornell, Texas, Wisconsin— (Craven et al., 1998), (ii) Actor dataset (Tang et al., 2009), (iii) Co-author datasets —CS, Physics— (Shchur et al., 2019), and (iv) Amazon datasets — Photo,
Revised: We evaluate AutoCoG on several popular semi-node classification datasets including (i) the WebKB datasets —Cornell, Texas, Wisconsin— (Craven et al., 1998), (ii) Actor dataset (Tang et al., 2009), (iii) Co-author datasets —CS, Physics— (Shchur et al., 2019) and (iv) Amazon datasets — Photo,

Original: Computers — (Shchur et al., 2019).
Revised: Computers — (Shchur et al., 2019).

Original: We select these datasets to represent a wide range of graphs under different degrees of homophily, which will serve to demonstrate Auto-CoG robustness in comparison to the different SOTA methods.
Revised: We select these datasets to represent a wide range of graphs under different degrees of homophily, which will serve to demonstrate Auto-CoG robustness in comparison to the different SOTA methods.

Original: The specifics on each datasets, in sorted order under homophily, is recorded in Table(8).
Revised: The specifics on each datasets, in sorted order under homophily, is recorded in Table(8)

Original: [Table]
Revised: [Table]

Original: Photos [Table]
Revised: Algorithm

Original: Input: 𝑊 𝑠 , 𝑋 , 𝐺 ( 𝑉, 𝐸 ) , searchSpace, epochs, startNumLayer, endNumLayer, stages ;
Revised: Input: 𝑊 𝑠 , 𝑋 , 𝐺 ( 𝑉, 𝐸 ) , searchSpace, epochs, startNumLayer, endNumLayer, stages ;

Original: Output: 𝛼 , 𝐺 ( 𝑉, ¯ 𝐸 ) , 𝑆 ; ¯ 𝐸 ← 𝐸 ; for s = 0 to stages-1 do #Initialize new model and architecture parameters 𝛼 ← OnesInitParameters (searchSpace); 𝑊 ← initModel ( min (startNumLayer+s, endNumLayer)); for e=0 to epochs-1 do 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ; ¯ 𝑎 ← Sample ( 𝛼 );
Revised: Output: 𝛼 , 𝐺 ( 𝑉, ¯ 𝐸 ) , 𝑆 ; ¯ 𝐸 ← 𝐸 ; for s = 0 to stages-1 do #Initialize new model and architecture parameters 𝛼 ← OnesInitParameters (searchSpace); 𝑊 ← initModel ( min (startNumLayer+s, endNumLayer)); for e=0 to epochs-1 do 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ; ¯ 𝑎 ← Sample ( 𝛼 );

Original: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑡𝑟𝑎𝑖𝑛 ,𝐺 ( 𝑉, ¯ 𝐸 ))
Revised: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑡𝑟𝑎𝑖𝑛 ,𝐺 ( 𝑉, ¯ 𝐸 ))

Original: → 𝑊 ,𝑊 𝑠 ;
Revised: → 𝑊 ,𝑊 𝑠 ;

Original: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑣𝑎𝑙𝑖𝑑 ,𝐺 ( 𝑉, ¯ 𝐸 ))
Revised: BackPropgate 𝐿 𝑜𝑏𝑗 ( ¯ 𝛼,𝑊 , 𝑆, 𝑋 𝑣𝑎𝑙𝑖𝑑 ,𝐺 ( 𝑉, ¯ 𝐸 ))

Original: → 𝛼 ; end #Reduce search space and augment edges ¯ 𝐸 ← PruneAndGrow
Revised: → 𝛼 ; end #Reduce search space and augment edges ¯ 𝐸 ← PruneAndGrow

Original: ( 𝑓 ( . ; 𝑊 𝑠 ))
Revised: ( 𝑓 ( . ; 𝑊 𝑠 ))

Original: ; searchSpace ← ReduceSearchSpace (searchSpace, 𝛼 ); end 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ;
Revised: ; searchSpace ← ReduceSearchSpace (searchSpace, 𝛼 ); end 𝑆 ← 𝜎 ( MLP ( 𝑓 ( 𝑋,𝐺 ( 𝑉, ¯ 𝐸 ) ; 𝑊 𝑠 )) ;