Original: D EEPER -GXX:
Revised: D EEPER -GXX:

Original: D EEPENING A RBITRARY GNN S
Revised: D EEPENING A RBITRARY GNN S

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Recently, motivated by real applications, a major research direction in graph neural networks (GNNs) is to explore deeper structures.
Revised: Recently, motivated by real applications, a major research direction in graph neural networks (GNNs) is to explore deeper structures.

Original: For instance, the graph connectivity is not always consistent with the label distribution (e.g., the closest neighbors of some nodes are not from the same category).
Revised: For instance, the graph connectivity is not always consistent with the label distribution (e.g., the closest neighbors of some nodes are not from the same category).

Original: In this case, GNNs need to stack more layers, in order to find the same categorical neighbors in a longer path for capturing the class-discriminative information.
Revised: In this case, GNNs need to stack more layers, in order to find the same categorical neighbors in a longer path for capturing the class-discriminative information.

Original: However, two major problems hinder the deeper GNNs to obtain satisfactory performance, i.e., vanishing gradient and over-smoothing .
Revised: However, two major problems hinder the deeper GNNs to obtain satisfactory performance, i.e., vanishing gradient and over-smoothing .

Original: On one hand, stacking layers makes the neural network hard to train as the gradients of the first few layers vanish.
Revised: On one hand, stacking layers makes the neural network hard to train as the gradients of the first few layers vanish.

Original: Moreover, when simply addressing vanishing gradient in GNNs, we discover the shading neighbors effect (i.e., stacking layers inappropriately distorts the non-IID information of graphs and degrade the performance of GNNs).
Revised: Moreover, when simply addressing vanishing gradient in GNNs, we discover the shading neighbors effect (i.e., stacking layers inappropriately distorts the non-IID information of graphs and degrade the performance of GNNs).

Original: On the other hand, deeper GNNs aggregate much more information from common neighbors such that individual node representations share more overlapping features, which makes the final output representations not discriminative (i.e., overly smoothed).
Revised: On the other hand, deeper GNNs aggregate much more information from common neighbors such that individual node representations share more overlapping features, which makes the final output representations not discriminative (i.e., overly smoothed).

Original: In this paper, for the first time, we address both problems to enable deeper GNNs, and propose DeeperGXX , which consists of the Weight-Decaying Graph Residual Connection module (WDG-ResNet) and Topology-Guided Graph Contrastive Loss (TGCL).
Revised: In this paper, for the first time, we address both problems to enable deeper GNNs, and propose DeeperGXX , which consists of the Weight-Decaying Graph Residual Connection module (WDG-ResNet) and Topology-Guided Graph Contrastive Loss (TGCL).

Original: Extensive experiments on real-world data sets demonstrate that Deeper-GXX outperforms state-of-the-art deeper baselines.
Revised: Extensive experiments on real-world data sets demonstrate that Deeper-GXX outperforms state-of-the-art deeper baselines.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph neural networks (GNNs) have been proven successful at modeling graph data by extracting node hidden representations that are effective for many downstream tasks.
Revised: Graph neural networks (GNNs) have been proven successful at modeling graph data by extracting node hidden representations that are effective for many downstream tasks.

Original: In general, they are realized by the message passing schema and aggregate neighbor features to obtain node hidden representations (Kipf & Welling, 2017; Hamilton et al., 2017a; Velickovic et al., 2018;
Revised: In general, they are realized by the message passing schema and aggregate neighbor features to obtain node hidden representations (Kipf & Welling, 2017; Hamilton et al., 2017a; Velickovic et al., 2018;

Original: Xu et al., 2019).
Revised: Xu et al., 2019).

Original: Recently, the surge of big data makes graphs’ structural and attribute information much more complex and uncertain, which urges the researchers to make GNNs deeper (i.e., stacking more graph neural layers), in order to capture more meaningful information for better performance.
Revised: Recently, the surge of big data makes graphs’ structural and attribute information much more complex and uncertain, which urges the researchers to make GNNs deeper (i.e., stacking more graph neural layers), in order to capture more meaningful information for better performance.

Original: For example, in social media, people from different categories (e.g., occupation, interests, etc.) are often connected (e.g., become friends), and one user’s immediate neighbor information may not reflect his/her categorical information.
Revised: For example, in social media, people from different categories (e.g., occupation, interests, etc.) are often connected (e.g., become friends), and users’ immediate neighbor information may not reflect their categorical information.

Original: Thus, deepening GNNs is necessary to identify the neighbors from the same category in a longer path (e.g., k -hop neighbors), and to aggregate their features to obtain the class-discriminative node representations.
Revised: Thus, deepening GNNs is necessary to identify the neighbors from the same category in a longer path (e.g., k -hop neighbors), and to aggregate their features to obtain the class-discriminative node representations.

Original: To demonstrate the benefit of deeper GNNs, we conduct a case study shown in Figure 1 (See the detailed experimental setup in Appendix A.1).
Revised: To demonstrate the benefit of deeper GNNs, we conduct a case study shown in Figure 1 (See the detailed experimental setup in Appendix A.1).

Original: In Figure 1a, we observe that the query node (the diamond in the black dashed circle) cannot rely on its closest labeled neighbor (the red star in the circle) to correctly predict its label (the blue).
Revised: In Figure 1a, we observe that the query node (the diamond in the black dashed circle) cannot rely on its closest labeled neighbor (the red star in the circle) to correctly predict its label (the blue).

Original: Only by exploring longer paths consisting of more similar neighbors are we able to predict its label as blue.
Revised: Only by exploring longer paths consisting of more similar neighbors are we able to predict its label as blue.

Original: Figure 1b compares the classification accuracy of shallow GNNs and deeper GNNs.
Revised: Figure 1b compares the classification accuracy of shallow GNNs and deeper GNNs.

Original: We can see that deeper
Revised: We can see that deeper

Original: GNNs significantly outperform shallow ones by more than 11%, due to their abilities to explore longer paths on the graph.
Revised: GNNs significantly outperform shallow ones by more than 11%, due to their abilities to explore longer paths on the graph.

Original: Similar observations of the benefits of deeper GNNs are also found in the missing feature scenario presented in Section 3.3.
Revised: Similar observations of the benefits of deeper GNNs are also found in the missing feature scenario presented in Section 3.3.

Original: However, simply stacking layers of GNNs can be problematic, due to vanishing gradient and oversmoothing issues.
Revised: However, simply stacking layers of GNNs can be problematic, due to vanishing gradient and oversmoothing issues.

Original: On one hand, increasing the number of neural layers can induce the hard-to-train model, where both the training error and test error are higher than shallow ones.
Revised: On one hand, increasing the number of neural layers can induce the hard-to-train model, where both the training error and test error are higher than shallow ones.

Original: This is mainly caused by the vanishing gradient issue (He et al., 2016), where the gradient of the first few layers vanish
Revised: This is mainly caused by the vanishing gradient issue (He et al., 2016), where the gradient of the first few layers vanish

Original: [Figure]
Revised: [Figure]

Original: such that the training loss could not be successfully propagated through deeper models.
Revised: such that the training loss could not be successfully propagated through deeper models.

Original: ResNet (He et al., 2016) has been proposed to address this issue.
Revised: ResNet (He et al., 2016) has been proposed to address this issue.

Original: However, we discover that simply combining
Revised: However, we discover that simply combining

Original: ResNet with GNNs still leads to the sub-optimal solution: as ResNet stacks layers, the importance of close neighbors’ features gradually decreases during the GNN information aggregation process, and the faraway neighbor information becomes dominant.
Revised: ResNet with GNNs still leads to the sub-optimal solution: as ResNet stacks layers, the importance of close neighbors’ features gradually decreases during the GNN information aggregation process, and the faraway neighbor information becomes dominant.

Original: We call this effect as shading neighbors .
Revised: We call this effect as shading neighbors .

Original: On the other hand, GNN utilizes the message passing schema to aggregate neighbor features, in order to get class-discriminative node representations.
Revised: On the other hand, GNN utilizes the message passing schema to aggregate neighbor features, in order to get class-discriminative node representations.

Original: However, by stacking more layers, each node begins to share more and more overlapping neighbor information during the aggregation process and the node representations gradually become indistinguishable (Li et al., 2018; Oono & Suzuki, 2020).
Revised: However, by stacking more layers, each node begins to share more and more overlapping neighbor information during the aggregation process and the node representations gradually become indistinguishable (Li et al., 2018; Oono & Suzuki, 2020).

Original: This has been referred to as the over-smoothing issue, and it significantly affects the performance of downstream tasks such as node classification and link prediction.
Revised: This has been referred to as the over-smoothing issue, and it significantly affects the performance of downstream tasks such as node classification and link prediction.

Original: In this paper, we study how to effectively stack GNN layers by addressing shading neighbors and oversmoothing at the same time.
Revised: In this paper, we study how to effectively stack GNN layers by addressing shading neighbors and oversmoothing at the same time.

Original: First, to address the shading neighbors caused by the direct application of ResNet on GNNs, we propose Weight-Decaying Graph Residual Connection (WDG-ResNet), which learns the weight of each residual connection layer (instead of setting it as 1 in ResNet), and further introduces a decaying factor to refine the weight of each layer.
Revised: First, to address the shading neighbors caused by the direct application of ResNet on GNNs, we propose Weight-Decaying Graph Residual Connection (WDG-ResNet), which learns the weight of each residual connection layer (instead of setting it as 1 in ResNet), and further introduces a decaying factor to refine the weight of each layer.

Original: Interestingly, we find that the hyperparameter λ of the weight decaying factor actually controls the number of effective layers in deeper GNNs based on the input graph inherent property, which is verified in Appendix A.4.
Revised: Interestingly, we find that the hyperparameter λ of the weight decaying factor actually controls the number of effective layers in deeper GNNs based on the input graph inherent property, which is verified in Appendix A.5.

Original: Second, for addressing over-smoothing , we propose Topology-Guided Graph Contrastive Loss (TGCL) in the contrastive learning manner (van den Oord et al., 2018), where the hidden representations of the positive pairs should be closer, and those of the negative pairs should be pushed apart.
Revised: Second, for addressing over-smoothing , we propose Topology-Guided Graph Contrastive Loss (TGCL) in the contrastive learning manner (van den Oord et al., 2018), where the hidden representations of the positive pairs should be closer, and those of the negative pairs should be pushed apart.

Original: Through theoretical and empirical analysis, we find that TGCL can be effectively and efficiently realized by only considering 1-hop neighbors as the positive pair and all the rest as negative pairs.
Revised: Through theoretical and empirical analysis, we find that TGCL can be effectively and efficiently realized by only considering 1-hop neighbors as the positive pair and all the rest as negative pairs.

Original: Combining the proposed WDG-ResNet and TGCL, we propose an end-to-end model called Deeper-GXX to help arbitrary GNNs go deeper.
Revised: Combining the proposed WDG-ResNet and TGCL, we propose an end-to-end model called Deeper-GXX to help arbitrary GNNs go deeper.

Original: Our contributions can be summarized as follows.
Revised: Our contributions can be summarized as follows.

Original: • We propose Weight-Decaying Graph Residual Connection (WDG-ResNet) to address the shading neighbors effect caused by vanilla ResNet in dealing with the vanishing gradient of GNNs.
Revised: • We propose Weight-Decaying Graph Residual Connection (WDG-ResNet) to address the shading neighbors effect caused by vanilla ResNet in dealing with the vanishing gradient of GNNs.

Original: • We propose Topology-Guided Graph Contrastive Loss (TGCL) to address the over-smoothing problem by encoding the graph topological information to the discriminative node representations.
Revised: • We propose Topology-Guided Graph Contrastive Loss (TGCL) to address the over-smoothing problem by encoding the graph topological information to the discriminative node representations.

Original: • We combine the proposed WDG-ResNet and TGCL into an end-to-end model called Deeper-GXX , which is model-agnostic and can help arbitrary GNNs go deeper.
Revised: • We combine the proposed WDG-ResNet and TGCL into an end-to-end model called Deeper-GXX , which is model-agnostic and can help arbitrary GNNs go deeper.

Original: • Extensive experiments show that Deeper-GXX outperforms state-of-the-art deeper baselines.
Revised: • Extensive experiments show that Deeper-GXX outperforms state-of-the-art deeper baselines.

Original:  P ROPOSED M ETHOD
Revised:  P ROPOSED M ETHOD

Original: In this section, we begin with the overview of Deeper-GXX.
Revised: In this section, we begin with the overview of Deeper-GXX.

Original: Then, we provide the details of the proposed Weight-Decaying Graph Residual Connection (WDG-ResNet) and Topology-Guided
Revised: Then, we provide the details of the proposed Weight-Decaying Graph Residual Connection (WDG-ResNet) and Topology-Guided

Original: Graph Contrastive Loss (TGCL), which address shading neighbors and over-smoothing problems, respectively.
Revised: Graph Contrastive Loss (TGCL), which address shading neighbors and over-smoothing problems, respectively.

Original: We formalize the graph embedding problem in the context of undirected graph G =
Revised: We formalize the graph embedding problem in the context of undirected graph G =

Original: ( V, E, X ) , where V consists of n vertices, E consists of m edges, X ∈ R n × d denotes the feature matrix and d is the feature dimension.
Revised: ( V, E, X ) , where V consists of n vertices, E consists of m edges, X ∈ R n × d denotes the feature matrix and d is the feature dimension.

Original: We let A ∈ R n × n denote the adjacency matrix and denote
Revised: We let A ∈ R n × n denote the adjacency matrix and denote

Original: A i ∈ R n as the adjacency vector for node v i .
Revised: A i ∈ R n as the adjacency vector for node v i .

Original: H i
Revised: H i

Original: ∈
Revised: ∈

Original: R h is the hidden representation vector of v i .
Revised: R h is the hidden representation vector of v i .

Original: 2. O VERVIEW OF D EEPER -GXX
Revised: 2. O VERVIEW OF D EEPER -GXX

Original: The overview of our proposed Deeper-GXX is shown in Figure 2.
Revised: The overview of our proposed Deeper-GXX is shown in Figure 2.

Original: The green dash line stands for
Revised: The green dash line stands for

Original: Weight-Decaying Graph Residual Connection (WDG-ResNet).
Revised: Weight-Decaying Graph Residual Connection (WDG-ResNet).

Original: In WDG-ResNet, after the current hidden representation H ( l ) is generated by the l -th layer of arbitrary GNNs, H ( l ) will be adjusted by its second last layer H ( l − 2) and the first layer H (1) with proper weights.
Revised: In WDG-ResNet, after the current hidden representation H ( l ) is generated by the l -th layer of arbitrary GNNs, H ( l ) will be adjusted by its second last layer H ( l − 2) and the first layer H (1) with proper weights.

Original: The red dash line stands for
Revised: The red dash line stands for

Original: Topology-Guided Graph Contrastive Loss (TGCL).
Revised: Topology-Guided Graph Contrastive Loss (TGCL).

Original: In TGCL, we first need to sample positive node pairs and negative node pairs based on the input graph topology such that the hidden representations of positive node pairs get closer and negative ones are pushed farther apart.
Revised: In TGCL, we first need to sample positive node pairs and negative node pairs based on the input graph topology such that the hidden representations of positive node pairs get closer and negative ones are pushed farther apart.

Original: After introducing the
Revised: After introducing the

Original: TGCL loss to GNNs, the overall loss function L overall of Deeper-GXX is expressed as follows.
Revised: TGCL loss to GNNs, the overall loss function L overall of Deeper-GXX is expressed as follows.

Original: [Equation]
Revised: [Equation]

Original: where L GNN denotes the loss of the downstream task using an arbitrary GNN model (e.g., node classification in GCN), L TGCL is the TGCL loss, and α is a constant hyperparameter.
Revised: where L GNN denotes the loss of the downstream task using an arbitrary GNN model (e.g., node classification in GCN), L TGCL is the TGCL loss, and α is a constant hyperparameter.

Original: Deeper-GXX combines WDG-ResNet and TGCL to address shading neighbors and over-smoothing problems.
Revised: Deeper-GXX combines WDG-ResNet and TGCL to address shading neighbors and over-smoothing problems.

Original: The design of WDG-ResNet and TGCL are introduced in the following subsections.
Revised: The design of WDG-ResNet and TGCL are introduced in the following subsections.

Original: 2. W EIGHT -D ECAYING G RAPH R ESIDUAL C ONNECTION (WDG-R ES N ET )
Revised: 2. W EIGHT -D ECAYING G RAPH R ESIDUAL C ONNECTION (WDG-R ES N ET )

Original: As we increase the number of layers, one unavoidable problem brought by neural networks is the vanishing gradient , which means that the first several layers of the deeper neural network become hard to optimize as their gradients vanish during the training process (He et al., 2016).
Revised: As we increase the number of layers, one unavoidable problem brought by neural networks is the vanishing gradient , which means that the first several layers of the deeper neural network become hard to optimize as their gradients vanish during the training process (He et al., 2016).

Original: Currently, nascent deeper GNN methods (Zhao & Akoglu, 2020; Rong et al., 2020; Li et al., 2019) solve this problem by adding ResNet (He et al., 2016) on graph neural networks.
Revised: Currently, nascent deeper GNN methods (Zhao & Akoglu, 2020; Rong et al., 2020; Li et al., 2019) solve this problem by adding ResNet (He et al., 2016) on graph neural networks.

Original: Taking GCN (Kipf & Welling, 2017) as an example, the graph residual connection is expressed as follows.
Revised: Taking GCN (Kipf & Welling, 2017) as an example, the graph residual connection is expressed as follows.

Original: [Equation]
Revised: [Equation]

Original: where l ( ≥ 2) denotes the index of layers, H ( l − 1) and H ( l − 2) are the hidden representations, σ ( · ) is the activation function, W ( l − 1) is the learnable weight matrix, and ˆA is the re-normalized selflooped adjacency matrix with ˆA = ˜D − 12 ˜A ˜D − 12 and
Revised: where l ( ≥ 2) denotes the index of layers, H ( l − 1) and H ( l − 2) are the hidden representations, σ ( · ) is the activation function, W ( l − 1) is the learnable weight matrix, and ˆA is the re-normalized selflooped adjacency matrix with ˆA = ˜D − 12 ˜A ˜D − 12 and

Original: ˜A =
Revised: ˜A =

Original: A + I .
Revised: A + I .

Original: In ResNet, the residual connection connects the current layer and its second last layer.
Revised: In ResNet, the residual connection connects the current layer and its second last layer.

Original: Without loss of generality, we assume the last layer of GNNs is stacked by ResNet, i.e., l is divisible by 2.
Revised: Without loss of generality, we assume the last layer of GNNs is stacked by ResNet, i.e., l is divisible by 2.

Original: Then, by extending H ( l − 2) iteratively (i.e., substituting it with its previous residual blocks), the above Eq.
Revised: Then, by extending H ( l − 2) iteratively (i.e., substituting it with its previous residual blocks), the above Eq.

Original: 2 could be rewritten as follows.
Revised: 2 could be rewritten as follows.

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: According to the GNN theoretical analysis, stacking l layers and getting H ( l ) in GNNs can be interpreted as aggregating l -hop neighbors’ feature information for the node hidden representations (Xu et al., 2019).
Revised: According to the GNN theoretical analysis, stacking l layers and getting H ( l ) in GNNs can be interpreted as aggregating l -hop neighbors’ feature information for the node hidden representations (Xu et al., 2019).

Original: As shown in Eq.
Revised: As shown in Eq.

Original: 3, when we stack more layers in GNNs, the information collected from faraway neighbors become dominant (as there are more terms regarding the information from faraway neighbors), compared with the information collected from the nearest neighbors (e.g., 1-hop or 2-hop neighbors).
Revised: 3, when we stack more layers in GNNs, the information collected from faraway neighbors becomes dominant (as there are more terms regarding the information from faraway neighbors), compared with the information collected from the nearest neighbors (e.g., 1-hop or 2-hop neighbors).

Original: This phenomenon contradicts the general intuition that the close neighbors of a node carry the most important information, and the importance degrades with faraway neighbors.
Revised: This phenomenon contradicts the general intuition that the close neighbors of a node carry the most important information, and the importance degrades with faraway neighbors.

Original: Formally, we describe this phenomenon as shading neighbors effect when stacking graph neural layers, as the importance of the nearest neighbors is diminishing.
Revised: Formally, we describe this phenomenon as shading neighbors effect when stacking graph neural layers, as the importance of the nearest neighbors is diminishing.

Original: We show that shading neighbors effect downgrades the GNNs performance in downstream tasks in Section 3.4.
Revised: We show that shading neighbors effect downgrades the GNNs performance in downstream tasks in Section 3.4.

Original: To address the shading neighbors effect, we propose Weight-Decaying Graph Residual Connection (WDG-ResNet).
Revised: To address the shading neighbors effect, we propose Weight-Decaying Graph Residual Connection (WDG-ResNet).

Original: Here, we first introduce the formulation and then provide the insights regarding why it can address the problem.
Revised: Here, we first introduce the formulation and then provide the insights regarding why it can address the problem.

Original: Specifically, WDG-ResNet introduces the layer similarity and weight decaying factor as follows.
Revised: Specifically, WDG-ResNet introduces the layer similarity and weight decaying factor as follows.

Original: [Equation]
Revised: [Equation]

Original: where cos ( H (1) , ˜ H ( l ) )
Revised: where cos ( H (1) , ˜ H ( l ) )

Original: =
Revised: =

Original: 1 n (cid:80) i H (1) i ( ˜ H ( l ) i )T
Revised: 1 n (cid:80) i H (1) i ( ˜ H ( l ) i )

Original:  ∥ H (1) i ∥∥ ˜ H ( l ) i ∥ measures the similarity between the l -th layer and the 1 -st layer, and we use the exponential function to map the cosine similarity ranging from {− 1 , 1 } to { e − 1 , e 1 }, to avoid the negative similarity weights.
Revised: ⊤ ∥ H (1) i ∥∥ ˜ H ( l ) i ∥ measures the similarity between the l -th layer and the 1 -st layer, and we use the exponential function to map the cosine similarity ranging from {− 1 , 1 } to { e − 1 , e 1 }, to avoid the negative similarity weights.

Original: The term e − l/λ is the decaying factor to further adjust the similarity weight of ˜ H ( l ) , where λ is a constant hyperparameter.
Revised: The term e − l/λ is the decaying factor to further adjust the similarity weight of ˜ H ( l ) , where λ is a constant hyperparameter.

Original: Different from ResNet (He et al., 2016), we add the learnable similarity sim
Revised: Different from ResNet (He et al., 2016), we add the learnable similarity sim

Original: ( H (1) , ˜ H ( l ) ) to essentially expand the hypothesis space of deeper GNNs.
Revised: ( H (1) , ˜ H ( l ) ) to essentially expand the hypothesis space of deeper GNNs.

Original: Additional to that, simply adding vanilla ResNet on GNNs will cause the shading neighbors effect.
Revised: Additional to that, simply adding vanilla ResNet on GNNs will cause the shading neighbors effect.

Original: The introduced decaying factor e − l/λ can alleviate this negative effect, because it brings the layer-wise dependency to stacking operations, in order to preserve the graph hierarchical information when GNNs go deeper.
Revised: The introduced decaying factor e − l/λ can alleviate this negative effect, because it brings the layer-wise dependency to stacking operations, in order to preserve the graph hierarchical information when GNNs go deeper.

Original: Since λ is a constant, the value of e − l/λ is decreasing while l increases.
Revised: Since λ is a constant, the value of e − l/λ is decreasing while l increases.

Original: Thus, the later stacked layer is always less important than previously stacked ones by the decaying weight, which addresses the shading neighbors effect.
Revised: Thus, the later stacked layer is always less important than previously stacked ones by the decaying weight, which addresses the shading neighbors effect.

Original: Without the decaying factor, the layer-wise weights are independent, and the shading neighbor effect still exists.
Revised: Without the decaying factor, the layer-wise weights are independent, and the shading neighbor effect still exists.

Original: Moreover, we visualize the layer-wise weight distribution of different residual connection methods (including our WDG-ResNet) and their effectiveness in addressing shading neighbors effect in Appendix A.3.
Revised: Moreover, we visualize the layer-wise weight distribution of different residual connection methods (including our WDG-ResNet) and their effectiveness in addressing shading neighbors effect in Appendix A.4.

Original: From another perspective, the hyperparameter λ of the decaying factor actually controls the number of effective neural layers in deeper GNNs.
Revised: From another perspective, the hyperparameter λ of the decaying factor actually controls the number of effective neural layers in deeper GNNs.

Original: For example, when λ = 10 , the decaying factor for the 10-th layer is 0.3679 (i.e., e − 1 ); but for the 30-th layer, it is 0.0049 (i.e., e − 3 ).
Revised: For example, when λ = 10 , the decaying factor for the 10-th layer is 0.3679 (i.e., e − 1 ); but for the 30-th layer, it is 0.0049 (i.e., e − 3 ).

Original: Thisradioactive decay limits the effective information aggregation scope of deeper GNNs, because the later stacked layers will become significantly less important.
Revised: This decay limits the effective information aggregation scope of deeper GNNs, because the later stacked layers will become significantly less important.

Original: Based on this controlling property of λ , a natural follow-up question is whether its value depends on the property of input graphs.
Revised: Based on this controlling property of λ , a natural follow-up question is whether its value depends on the property of input graphs.

Original: Interestingly, through our experiments, we find that the optimal λ is very close to the diameter of input graphs (if it is connected) or the largest component (if it does not have many separate components) .
Revised: Interestingly, through our experiments, we find that the optimal λ is very close to the diameter of input graphs (if it is connected) or the largest component (if it does not have many separate components) .

Original: This observation verifies our conjecture regarding the property of λ (i.e., it controls the number of effective layers or number of hops during the message passing aggregation schema of GNNs).
Revised: This observation verifies our conjecture regarding the property of λ (i.e., it controls the number of effective layers or number of hops during the message passing aggregation schema of GNNs).

Original: Hence, the value of λ can be searched around the diameter of the input graph, and we discuss the details in the Appendix A.4.
Revised: Hence, the value of λ can be searched around the diameter of the input graph, and we discuss the details in the Appendix A.5.

Original: Simplified WDG-ResNet .
Revised: Simplified WDG-ResNet .

Original: In the experiments, we observe that if the number of layers of GNNs is too large ( e.g. , 50 layers or more), the computational cost of adding the similarity function sim ( H (1) , ˜ H ( l ) ) at each graph residual connection layer can be expensive.
Revised: In the experiments, we observe that if the number of layers of GNNs is too large ( e.g. , 50 layers or more), the computational cost of adding the similarity function sim ( H (1) , ˜ H ( l ) ) at each graph residual connection layer can be expensive.

Original: To accelerate the computation, we formulate a simpler version of Eq. 4 by fixing the sim ( H (1) , ˜ H ( l ) ) but keeping the decaying factor.
Revised: To accelerate the computation, we formulate a simpler version of Eq. 4 by removing the sim ( H (1) , ˜ H ( l ) ) but keeping the decaying factor.

Original: The simplified version of WDG-ResNet is expressed as follows.
Revised: The simplified version of WDG-ResNet is expressed as follows.

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Compared with Eq. 4, Eq. 5 gets rid of the exponential cosine similarity measurement, which greatly reduces the computational cost.
Revised: Compared with Eq. 4, Eq. 5 gets rid of the exponential cosine similarity measurement, which greatly reduces the computational cost.

Original: However, the simplified WDG-ResNet still keeps the decaying factor for the layer-wise dependency, such that the shading neighbors effect can still be alleviated.
Revised: However, the simplified WDG-ResNet still keeps the decaying factor for the layer-wise dependency, such that the shading neighbors effect can still be alleviated.

Original: 2. T OPOLOGY -G UIDED G RAPH C ONTRASTIVE L OSS (TGCL)
Revised: 2. T OPOLOGY -G UIDED G RAPH C ONTRASTIVE L OSS (TGCL)

Original: To address over-smoothing , current deeper GNN solutions depend on certain hard-to-acquire prior knowledge (e.g., important hyperparameters or sampling randomness) to get the discriminative node representations (Zhao & Akoglu, 2020;
Revised: To address over-smoothing , current deeper GNN solutions depend on certain hard-to-acquire prior knowledge (e.g., important hyperparameters or sampling randomness) to get the discriminative node representations (Zhao & Akoglu, 2020;

Original: Rong et al., 2020;
Revised: Rong et al., 2020;

Original: Zhou et al., 2020).
Revised: Zhou et al., 2020).

Original: Inspired by these solutions, we are seeking for an effective discriminative indicator that can be easily obtained from the input graph without any prior knowledge.
Revised: Inspired by these solutions, we are seeking for an effective discriminative indicator that can be easily obtained from the input graph without any prior knowledge.

Original: Thus, we propose a novel contrastive regularization term, named Topology-Guided Graph Contrastive Loss (TGCL), to transfer this hard-to-acquire knowledge into the topology information of the input graph as follows.
Revised: Thus, we propose a novel contrastive regularization term, named Topology-Guided Graph Contrastive Loss (TGCL), to transfer this hard-to-acquire knowledge into the topology information of the input graph as follows.

Original: [Equation]
Revised: [Equation]

Original: where z i =
Revised: where z i =

Original: g
Revised: g

Original: ( H ( l ) i ) , g ( · ) is an encoder mapping H ( l ) i to another latent space, f ( · ) is a similarity function (e.g., f ( a, b ) =
Revised: ( H ( l ) i ) , g ( · ) is an encoder mapping H ( l ) i to another latent space, f ( · ) is a similarity function ( e.g., f ( a, b ) =

Original: exp( ab T || a |||| b || ) ), dist ( · ) is a distance measurement function (e.g., hamming distance(Norouzi et al., 2012)), N i is the set containing one-hop neighbors of node v i , and ¯ N i is the complement of the set N i .
Revised: exp( ab ⊤ || a |||| b || ) ), dist ( · ) is a distance measurement function (e.g., hamming distance (Norouzi et al., 2012)), N i is the set containing one-hop neighbors of node v i , and ¯ N i is the complement of the set N i .

Original: In Eq. 6, directly connected nodes ( v i , v j ) form the positive pair, while not directly connected nodes ( v i , v k ) form the negative pair.
Revised: In Eq. 6, directly connected nodes ( v i , v j ) form the positive pair, while not directly connected nodes ( v i , v k ) form the negative pair.

Original: The intuition of this equation is to maximize the similarity of the representations of the positive pairs, and to minimize the similarity of the representations of the negative pairs, such that the node representations become discriminative.
Revised: The intuition of this equation is to maximize the similarity of the representations of the positive pairs, and to minimize the similarity of the representations of the negative pairs, such that the node representations become discriminative.

Original: In the graph contrastive learning setting, designing the positive and negative pairs is very important, because it needs to effectively address the over-smoothing issue, and it should also be efficiently obtained given the topology of the input graph.
Revised: In the graph contrastive learning setting, designing the positive and negative pairs is very important, because it needs to effectively address the over-smoothing issue, and it should also be efficiently obtained given the topology of the input graph.

Original: We provide further discussion and theoretical analysis to demonstrate the effectiveness of our positive/negative pair sampling strategy for Eq.
Revised: We provide further discussion and theoretical analysis to demonstrate the effectiveness of our positive/negative pair sampling strategy for Eq.

Original: 6 in Appendix B.2.
Revised: 6 in Appendix B.2.

Original: With Eq.
Revised: With Eq.

Original: 6, we can analyze the proposed TGCL loss bound in terms of mutual information, as stated in Proposition 1.
Revised: 6, we can analyze the proposed TGCL loss bound in terms of mutual information, as stated in Proposition 1.

Original: In particular, Proposition 1 demonstrates that the TGCL loss for the graph is the lower bound of the mutual information between two representations of a neighbor node pair.
Revised: In particular, Proposition 1 demonstrates that the TGCL loss for the graph is the lower bound of the mutual information between two representations of a neighbor node pair.

Original: Therefore, minimizing TGCL is equivalent to maximizing the mutual information of connected nodes by taking graph topology information into consideration.
Revised: Therefore, minimizing TGCL is equivalent to maximizing the mutual information of connected nodes by taking graph topology information into consideration.

Original: Proposition 1 Given a neighbor node pair sampled from the graph G =
Revised: Proposition 1 Given a neighbor node pair sampled from the graph G =

Original: ( V, E, X ) , i.e. , nodes v i and v j , we have I ( z i , z j ) ≥ −L TGCL
Revised: ( V, E, X ) , i.e. , nodes v i and v j , we have I ( z i , z j ) ≥ −L TGCL

Original: + E v i ∼ V log( | ¯ N i | ) , where I ( z i , z j ) is the mutual information between two representations of the node pair v i and v j , and L TGCL is the topology-guided contrastive loss weighted by hamming distance measurement.
Revised: + E v i ∼ V log( | ¯ N i | ) , where I ( z i , z j ) is the mutual information between two representations of the node pair v i and v j , and L TGCL is the topology-guided contrastive loss weighted by hamming distance measurement.

Original: The proof of this proposition can be found in Appendix B.1.
Revised: The proof of this proposition can be found in Appendix B.1.

Original:  E XPERIMENT
Revised:  E XPERIMENTS

Original: In this section, we demonstrate the performance of our proposed Deeper-GXX in terms of effectiveness by comparing it with state-of-the-art deeper GNN methods and self-ablations.
Revised: In this section, we demonstrate the performance of our proposed Deeper-GXX in terms of effectiveness by comparing it with state-of-the-art deeper GNN methods and self-ablations.

Original: In addition, we conduct a case study to show how the increasing number of layers influences the performance of deeper GNNs when the input graph has missing features.
Revised: In addition, we conduct a case study to show how the increasing number of layers influences the performance of deeper GNNs when the input graph has missing features.

Original: 3. E XPERIMENT S ETUP
Revised: 3. E XPERIMENT S ETUP

Original: Data sets: The Cora (Lu & Getoor, 2003) data set is a citation network consisting of 5,429 edges and 2,708 scientific publications from 7 classes.
Revised: Data sets: The Cora (Lu & Getoor, 2003) data set is a citation network consisting of 5,429 edges and 2,708 scientific publications from 7 classes.

Original: The edge in the graph represents the citation of one paper by another.
Revised: The edge in the graph represents the citation of one paper by another.

Original: CiteSeer (Lu & Getoor, 2003) data set consists of 3,327 scientific publications which could be categorized into 6 classes, and this citation network has 9,228 edges.
Revised: CiteSeer (Lu & Getoor, 2003) data set consists of 3,327 scientific publications which could be categorized into 6 classes, and this citation network has 9,228 edges.

Original: PubMed (Namata et al., 2012) is a citation network consisting of 88,651 edges and 19,717 scientific publications fromclasses.
Revised: PubMed (Namata et al., 2012) is a citation network consisting of 88,651 edges and 19,717 scientific publications fromclasses.

Original: The Reddit (Hamilton et al., 2017b) data set is extracted from Reddit posts, which consists of 4,584 nodes and 19,460 edges.
Revised: The Reddit (Hamilton et al., 2017b) data set is extracted from Reddit posts, which consists of 4,584 nodes and 19,460 edges.

Original: OGB-Arxiv (Wang et al., 2020) is a citation network, which consists of 1,166,243 edges and 169,343 nodes from 40 classes.
Revised: OGB-Arxiv (Wang et al., 2020) is a citation network, which consists of 1,166,243 edges and 169,343 nodes from 40 classes.

Original: Baselines: We compare the performance of our method with the following  deeper baselines: (1) GCN (Kipf & Welling, 2017): the vanilla graph convolutional network; (2) GCNII (Chen et al., 2020b): an extension of GCN with skip connections and additional identity matrices; (3) DGN (Zhou et al., 2020): the differentiable group normalization for GNNs to normalize nodes within the same group and separate nodes among different groups; (4) PairNorm (Zhao & Akoglu, 2020): a GNN normalization layer designed to prevent node representations from becoming too similar; (5) DropEdge (Rong et al., 2020): a GNN-agnostic framework that randomly removes a certain number of edges from the input graph at each training epoch to reduce the speed of over-fitting and prevent the oversmoothing issue; (6) Deeper-GXX-S: using the simplified WDG-ResNet in Deeper-GXX.
Revised: Baselines: We compare the performance of our method with the following baselines including one vanilla GNN model and four state-of-the-art deeper GNN models: (1) GCN (Kipf & Welling, 2017): the vanilla graph convolutional network; (2) GCNII (Chen et al., 2020b): an extension of GCN with skip connections and additional identity matrices; (3) DGN (Zhou et al., 2020): the differentiable group normalization for GNNs to normalize nodes within the same group and separate nodes among different groups; (4) PairNorm (Zhao & Akoglu, 2020): a GNN normalization layer designed to prevent node representations from becoming too similar; (5) DropEdge (Rong et al., 2020): a GNN-agnostic framework that randomly removes a certain number of edges from the input graph at each training epoch; (6) Deeper-GXX-S: using the simplified WDG-ResNet in Deeper-GXX.

Original: Configurations : In the experiments, we follow the splitting strategy used in (Zhao & Akoglu, 2020) by randomly sampling 3% of the nodes as the training samples, 10% of the nodes as the validation samples, and the remaining 87% as the test samples.
Revised: Configurations : In the experiments, we follow the splitting strategy used in (Zhao & Akoglu, 2020) by randomly sampling 3% of the nodes as the training samples, 10% of the nodes as the validation samples, and the remaining 87% as the test samples.

Original: We set the learning rate to be 0.001 and the optimizer is RMSProp, which is one variant of ADAGRAD (Duchi et al., 2011).
Revised: We set the learning rate to be 0.001 and the optimizer is RMSProp, which is one variant of ADAGRAD (Duchi et al., 2011).

Original: For fair comparison, we set the feature dimension of the hidden layer to be 50, the dropout rate to be 0.5, the weight decay rate to be 0.0005, and the total number of iterations to be 1500 for all methods.
Revised: For fair comparison, we set the feature dimension of the hidden layer to be 50, the dropout rate to be 0.5, the weight decay rate to be 0.0005, and the total number of iterations to be 1500 for all methods.

Original: For Deeper-GXX and
Revised: For Deeper-GXX and

Original: Deeper-GXX-S, we sample 10 instances and 5 neighbors for each class from the training set, dist ( · ) is the hamming distance, and f ( · ) is the cosine similarity measurement.
Revised: Deeper-GXX-S, we sample 10 instances and 5 neighbors for each class from the training set, dist ( · ) is the hamming distance, and f ( · ) is the cosine similarity measurement.

Original: The experiments are repeated 10 times if not otherwise specified.
Revised: The experiments are repeated 10 times if not otherwise specified.

Original: The code of our algorithm could be found in an anonymous link * .All of the real-world data sets are publicly available.
Revised: The code of our algorithm can be found in an anonymous link * . All of the real-world data sets are publicly available.

Original: The experiments are performed on a Windows machine with a 16GB RTX 5000 GPU.
Revised: The experiments are performed on a Windows machine with a 16GB RTX 5000 GPU.

Original: The detailed hyperparameters (e.g., λ and α ) setting for the experimental results in each table could be found in Appendix A.2.
Revised: The detailed hyperparameters (e.g., λ and α ) setting for the experimental results in each table can be found in Appendix A.2.

Original: Hyperparameter study and efficiency analysis could also be found in Appendix A.4 and A.5, respectively.
Revised: Hyperparameter study and efficiency analysis could also be found in Appendix A.5 and A.6, respectively.

Original: 3. E XPERIMENTAL A NALYSIS
Revised: 3. E XPERIMENTAL A NALYSIS

Original: In this subsection, we evaluate the effectiveness of the proposed method on four benchmark data sets by comparing it with state-of-the-art methods.
Revised: In this subsection, we evaluate the effectiveness of the proposed method on four benchmark data sets by comparing it with state-of-the-art methods.

Original: The base model for all methods we used in these experiments is graph convolutional neural network (GCN)
Revised: The base model for all methods we used in these experiments is graph convolutional neural network (GCN)

Original: (Kipf & Welling, 2017).
Revised: (Kipf & Welling, 2017).

Original: For fair comparison, we set the numbers of hidden layers to be 60 for all methods and the dimension of the hidden layer to be 50.
Revised: For fair comparison, we set the numbers of hidden layers to be 60 for all methods and the dimension of the hidden layer to be 50.

Original: The experiments are repeated 5 times and we record the mean accuracy as well as the standard deviation in Table 1.
Revised: The experiments are repeated 5 times and we record the mean accuracy as well as the standard deviation in Table 1.

Original: [Table]
Revised: [Table]

Original: In the first five rows of Table
Revised: In the first five rows of Table

Original: [Table]
Revised: [Table]

Original: we observe that when we set 60 hidden layers as the reference
Revised: we observe that when we set 60 hidden layers as the reference

Original: [Table]
Revised: [Table]

Original: DropEdge has almost identical performance as the vanilla GCN.
Revised: DropEdge has almost identical performance as the vanilla GCN.

Original: PairNorm, GCNII
Revised: PairNorm, GCNII

Original: [Table]
Revised: [Table]

Original: and DGN increase the performance by more than 30% in four data sets compared with GCN.
Revised: and DGN increase the performance by more than 30% in four data sets compared with GCN.

Original: The latent reason is that the over-smoothing problem is alleviated to some extent, since these three are deliberately proposed to deal with over-smoothing problem in deeper GNNs.
Revised: The latent reason is that the over-smoothing problem is alleviated to some extent, since these three are deliberately proposed to deal with over-smoothing problem in deeper GNNs.

Original: Besides, our proposed method (i.e., DeeperGXX) and its simpler version (i.e., Deeper-GXX-S) outperform all of these baselines over four data sets.
Revised: Besides, our proposed method (i.e., DeeperGXX) and its simpler version (i.e., Deeper-GXX-S) outperform all of these baselines over four data sets.

Original: Addition to addressing the over-smoothing problem, another part of our outperformance can be credited to our proposed residual connection.
Revised: Addition to addressing the over-smoothing problem, another part of our outperformance can be credited to our proposed residual connection.

Original: To verify this conjecture, we further incorporate
Revised: To verify this conjecture, we further incorporate

Original: ResNet into PairNorm, DropEdge, DGN, GCNII, and DropEdge.
Revised: ResNet into PairNorm, DropEdge, DGN, GCNII, and DropEdge.

Original: Then, in the sixth to the eighth rows of Table 1, we can observe (1) some baselines like PairNorm and GCNII suffer from the vanishing gradient (e.g., the performance of PairNorm increases by 6% on Cora and 7% on CiteSeer, while the performance of GCNII rises to 75.38% on Reddit and 70.24% on Cora).
Revised: Then, in the sixth to the eighth rows of Table 1, we can observe (1) some baselines like PairNorm and GCNII suffer from the vanishing gradient (e.g., the performance of PairNorm increases by 6% on Cora and 7% on CiteSeer, while the performance of GCNII rises to 75.38% on Reddit and 70.24% on Cora).

Original: Also, although GCNII designs a ResNet-like architecture, adding ResNet to GCNII can still boost its performance on three data sets; (2) Compared with their "+ResNet" versions, our Deeper-GXX and Deeper-GXXS still outperform, which implies that our designed residual connection indeed contributes to the outperformance, and we do the ablation study to quantify each component’s contribution of DeeperGXX in Section 3.4; (3) Not all deeper baselines need ResNet.
Revised: Also, although GCNII designs a ResNet-like architecture, adding ResNet to GCNII can still boost its performance on three data sets; (2) Compared with their "+ResNet" versions, our Deeper-GXX and Deeper-GXXS still outperform, which implies that our designed residual connection indeed contributes to the outperformance, and we do the ablation study to quantify each component’s contribution of DeeperGXX in Section 3.4; (3) Not all deeper baselines need ResNet.

Original: For example, DropEdge+ResNet almost maintains the performance, and DGN+ResNet drops the performance.
Revised: For example, DropEdge+ResNet almost maintains the performance, and DGN+ResNet drops the performance.

Original: In addition to four small data sets, we also examine the the node classification performance of
Revised: In addition to four small data sets, we also examine the the node classification performance of

Original: Deeper-GXX on a large-scale graph called OGB-Arxiv shown in Figure 3a and Figure 3b.
Revised: Deeper-GXX on a large-scale graph called OGB-Arxiv shown in Figure 3a and Figure 3b.

Original: In this experiment, we fix the feature dimension of the hidden layer as 100, the total iteration is set asand GCN is chosen as the base model.
Revised: In this experiment, we fix the feature dimension of the hidden layer as 100, the total iteration is set asand GCN is chosen as the base model.

Original: Due to the memory limitation, we set the number of layers as 10 for all baselines methods in Figure 3b for fair comparison.
Revised: Due to the memory limitation, we set the number of layers as 10 for all baselines methods in Figure 3b for fair comparison.

Original: By observation, we find that (1) in
Revised: By observation, we find that (1) in

Original: Figure 3a, the performance of Deeper-GXX increases as we increase the number of layers, which verifies our conjuncture that increasing the number of layers indeed leads to better performance in large graphs due to more information aggregated from neighbors; (2) comparing with PairNorm,
Revised: Figure 3a, the performance of Deeper-GXX increases as we increase the number of layers, which verifies our conjuncture that increasing the number of layers indeed leads to better performance in large graphs due to more information aggregated from neighbors; (2) comparing with PairNorm,

Original: Deeper-GXX further boosts the performance by more than 5.6% on OGB-Arxiv data set in Figure 3b.
Revised: Deeper-GXX further boosts the performance by more than 5.6% on OGB-Arxiv data set in Figure 3b.

Original: [Figure]
Revised: [Figure]

Original: In Figure 3c, we show the performance of our proposed method with different base models ( e.g. ,
Revised: In Figure 3c, we show the performance of our proposed method with different base models ( e.g. ,

Original: GAT (Velickovic et al., 2018) and SGC (Wu et al., 2019)).
Revised: GAT (Velickovic et al., 2018) and SGC (Wu et al., 2019)).

Original: In the experiment, we set the numbers of the hidden layers as 60 for all methods and the dimension of the hidden layer as 50.
Revised: In the experiment, we set the numbers of the hidden layers as 60 for all methods and the dimension of the hidden layer as 50.

Original: The total number of training iteration is 1500.
Revised: The total number of training iteration is 1500.

Original: By observation, we find that both GAT and SGC suffer from vanishing gradient and over-smoothing when the architecture becomes deeper, and our proposed method DeeperGXX greatly alleviates them and boosts the performance by 40%-60% on average over four data sets.
Revised: By observation, we find that both GAT and SGC suffer from vanishing gradient and over-smoothing when the architecture becomes deeper, and our proposed method DeeperGXX greatly alleviates them and boosts the performance by 40%-60% on average over four data sets.

Original: Specifically, compared with the vanilla SGC, our Deeper-GXX boosts its performance by 43% on the CiteSeer data set and more than 67% on the Reddit data set.
Revised: Specifically, compared with the vanilla SGC, our Deeper-GXX boosts its performance by 43% on the CiteSeer data set and more than 67% on the Reddit data set.

Original: 3. C ASE S TUDY : M ISSING FEATURE SCENARIO
Revised: 3. C ASE S TUDY : M ISSING FEATURE SCENARIO

Original: Why do we need to stack more layers of GNNs?
Revised: Why do we need to stack more layers of GNNs?

Original: To answer this question, let us first imagine a scenario where some values of attributes are missing in the input graph.
Revised: To answer this question, let us first imagine a scenario where some values of attributes are missing in the input graph.

Original: In this scenario, the shallow GNNs may not work well because GNNs could not collect useful information from the neighbors due to the massive missing values.
Revised: In this scenario, the shallow GNNs may not work well because GNNs could not collect useful information from the neighbors due to the massive missing values.

Original: However, if we increase the number of layers, GNNs are able to gather more information from the k -hop neighbors and capture latent knowledge to compensate for missing features.
Revised: However, if we increase the number of layers, GNNs are able to gather more information from the k -hop neighbors and capture latent knowledge to compensate for missing features.

Original: To verify this, we conduct the following experiment: we randomly mask p % attributes in Cora and CiteSeer data sets (i.e., setting the masked attributes to be 0), gradually increase the number of layers, and record the accuracy for each setting.
Revised: To verify this, we conduct the following experiment: we randomly mask p % attributes in Cora and CiteSeer data sets (i.e., setting the masked attributes to be 0), gradually increase the number of layers, and record the accuracy for each setting.

Original: In this case study, the number of layers
Revised: In this case study, the number of layers

Original: [Table] [Table]
Revised: [Table] [Table]

Original: is selected from { 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 15 , 20 , 25 , 30 , 40 , 50 , 60 } , and the base model is GCN.
Revised: is selected from { 2 , 3 , 4 , 5 , 6 , 7 , 8 , 9 , 10 , 15 , 20 , 25 , 30 , 40 , 50 , 60 } , and the base model is GCN.

Original: For fair comparison, we add ResNet (He et al., 2016) if it can boost the baseline model’s performance by avoiding the vanishing gradient issue.
Revised: For fair comparison, we add ResNet (He et al., 2016) if it can boost the baseline model’s performance by avoiding the vanishing gradient issue.

Original: We repeat the experiments five times and record the mean accuracy and standard deviation.
Revised: We repeat the experiments five times and record the mean accuracy and standard deviation.

Original: Table 2 shows the performance of Deeper-GXX and various baselines with the optimal number of layers denoted as #L, i.e., when the model achieves the best performance.
Revised: Table 2 shows the performance of Deeper-GXX and various baselines with the optimal number of layers denoted as #L, i.e., when the model achieves the best performance.

Original: By observation, we find that when the missing rate is 25%, shallow GCN with ResNet has enough capability to achieve the best performance on both CiteSeer and Cora data sets.
Revised: By observation, we find that when the missing rate is 25%, shallow GCN with ResNet has enough capacity to achieve the best performance on both CiteSeer and Cora data sets.

Original: Compared with GCN, our proposed method further improves the performance by more than 3.83% on the CiterSeer data set and 4.08% on the Cora data set by stacking more layers.
Revised: Compared with GCN, our proposed method further improves the performance by more than 3.83% on the CiterSeer data set and 4.08% on the Cora data set by stacking more layers.

Original: However, when we increase the missing rate to 50% and 75%, we observe that most methods tend to achieve the best performance by stacking more layers.
Revised: However, when we increase the missing rate to 50% and 75%, we observe that most methods tend to achieve the best performance by stacking more layers.

Original: Specifically, PairNorm achieves the best performance at 10 layers when 25% features are missing, while it has the best performance at 40 layers when 75% features are missing.
Revised: Specifically, PairNorm achieves the best performance at 10 layers when 25% features are missing, while it has the best performance at 40 layers when 75% features are missing.

Original: A similar observation could also be found with GCNII on the Cora data set, DropEdge on CiteSeer data set as well as our proposed methods in both data sets.
Revised: A similar observation could also be found with GCNII on the Cora data set, DropEdge on CiteSeer data set as well as our proposed methods in both data sets.

Original: Overall, the experimental results verify that the more features a data set is missing, the more layers GNNs need to be stacked to achieve better performance.
Revised: Overall, the experimental results verify that the more features a data set is missing, the more layers GNNs need to be stacked to achieve better performance.

Original: One possible explanation  is that, when more features are missing, we need more  neighborsaggregated for collecting their partial features to compensate for the missing such that stacking GNN layers works.
Revised: Our explanation for this observation is that if the number of layers increases, more information will be collected from the k -hop neighbors to recover the missing information of its 1-hop and 2-hop neighbors.

Original: 3. A BLATION S TUDY
Revised: 3. A BLATION S TUDY

Original: [Table]
Revised: [Table]

Original: In this subsection, we conduct the ablation study on Cora to show the effectiveness of WDGResNet and TGCL in Table 3.
Revised: In this subsection, we conduct an ablation study on Cora to show the effectiveness of WDGResNet and TGCL in Table 3.

Original: In this experiment, we fix the feature dimension of the hidden layer as 50, the total iteration is set as 3000, the number of layers is set as 60, the sampling batch size for Deeper-GXX is 10, and GCN is chosen as the base model.
Revised: In this experiment, we fix the feature dimension of the hidden layer as 50, the total iteration is set as 3000, the number of layers is set as 60, the sampling batch size for Deeper-GXX is 10, and GCN is chosen as the base model.

Original: In Table 3, Deeper-GXX-T removes TGCL loss, Deeper-GXX-D removes the weight decaying factor in WDG-ResNet and
Revised: In Table 3, Deeper-GXX-T removes TGCL loss, Deeper-GXX-D removes the weight decaying factor in WDG-ResNet and

Original: Deeper-GXX-S achieves the simplified WDG-ResNet in Deeper-GXX, which removes the similarity measure in WDG-ResNet.
Revised: Deeper-GXX-S achieves the simplified WDG-ResNet in Deeper-GXX, which removes the similarity measure in WDG-ResNet.

Original: In Table 3, we have the following observations (1) comparing DeeperGXX with Deeper-GXX-T, we find that Deeper-GXX boosts the performance by 1.84% after adding
Revised: In Table 3, we have the following observations (1) comparing DeeperGXX with Deeper-GXX-T, we find that Deeper-GXX boosts the performance by 1.84% after adding

Original: TGCL loss, which demonstrates the effectiveness of TGCL to address over-smoothing issue; (2)
Revised: TGCL loss, which demonstrates the effectiveness of TGCL to address over-smoothing issue; (2)

Original: Deeper-GXX outperforms Deeper-GXX-D by 5.61%, which shows that Deeper-GXX could address the shading neighbors effect by adding the weight decaying factor; (3) comparing Deeper-GXX with
Revised: Deeper-GXX outperforms Deeper-GXX-D by 5.61%, which shows that Deeper-GXX could address the shading neighbors effect by adding the weight decaying factor; (3) comparing Deeper-GXX with

Original: Deeper-GXX-S, we verify that adding exponential cosine similarity measure e cos ( H (1) ,
Revised: Deeper-GXX-S, we verify that adding exponential cosine similarity measure e cos ( H (1) ,

Original: ˜ H ( l ) ) could further improve the performance by extending the hypothesis space of neural networks.
Revised: ˜ H ( l ) ) could further improve the performance by extending the hypothesis space of neural networks.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Contrastive Learning on Graphs .
Revised: Contrastive Learning on Graphs .

Original: Recently, contrastive learning attracts researchers’ great attention due to its outstanding performance by leveraging the rich unsupervised data.
Revised: Recently, contrastive learning attracts researchers’ great attention due to its outstanding performance by leveraging the rich unsupervised data.

Original: (van den Oord et al., 2018) is one of the earliest works, which proposes a Contrastive Predictive Coding framework to extract useful information from high dimensional data with a theoretical guarantee.
Revised: (van den Oord et al., 2018) is one of the earliest works, which proposes a Contrastive Predictive Coding framework to extract useful information from high dimensional data with a theoretical guarantee.

Original: Based on this work, recent studies (Song & Ermon, 2020; Chuang et al., 2020; Khosla et al., 2020; Tian et al., 2020;
Revised: Based on this work, recent studies (Song & Ermon, 2020; Chuang et al., 2020; Khosla et al., 2020; Tian et al., 2020;

Original: Chen et al., 2020c; Zheng et al., 2022) reveal a surge of research interest in contrastive learning.
Revised: Chen et al., 2020c; Zheng et al., 2022) reveal a surge of research interest in contrastive learning.

Original: (You et al., 2020) proposes a graph contrastive learning (GraphCL) framework that utilizes different types of augmentation methods to incorporate various priors and to learn unsupervised representations of graph data.
Revised: (You et al., 2020) proposes a graph contrastive learning (GraphCL) framework that utilizes different types of augmentation methods to incorporate various priors and to learn unsupervised representations of graph data.

Original: (Qiu et al., 2020) proposes a Graph Contrastive pre-training model named GCC to capture the graph topological properties across multiple networks by utilizing contrastive learning to learn the intrinsic and transferable structural representations.
Revised: (Qiu et al., 2020) proposes a Graph Contrastive pre-training model named GCC to capture the graph topological properties across multiple networks by utilizing contrastive learning to learn the intrinsic and transferable structural representations.

Original: (Zheng et al., 2021) introduced a weakly supervised contrastive learning framework to tackle the class collision problem by first generating a weak label for similar samples and then pulling the similar samples closer with contrastive regularization.
Revised: (Zheng et al., 2021) introduced a weakly supervised contrastive learning framework to tackle the class collision problem by first generating a weak label for similar samples and then pulling the similar samples closer with contrastive regularization.

Original: Authors Hassani & Ahmadi (2020) aims to learn node and graph level representations by contrasting structural views of graphs.
Revised: Authors Hassani & Ahmadi (2020) aims to learn node and graph level representations by contrasting structural views of graphs.

Original: In this paper, we leverage the contrastive learning manner and design positive and negative node pairs such that we can discriminate node representations based on the input graph inherent information, which paves the way for us to design effective deeper GNNs.
Revised: In this paper, we leverage the contrastive learning manner and design positive and negative node pairs such that we can discriminate node representations based on the input graph inherent information, which paves the way for us to design effective deeper GNNs.

Original: Deeper Graph Neural Networks .
Revised: Deeper Graph Neural Networks .

Original: To effectively make GNNs deeper, many research works focus on addressing the over-smoothing problem.
Revised: To effectively make GNNs deeper, many research works focus on addressing the over-smoothing problem.

Original: Over-smoothing problem of GNNs is formally described by (Li et al., 2018) to demonstrate that the final output node representations become indiscriminative after stacking many layers in GNN models.
Revised: Over-smoothing problem of GNNs is formally described by (Li et al., 2018) to demonstrate that the final output node representations become indiscriminative after stacking many layers in GNN models.

Original: This problem is also analyzed by (Oono & Suzuki, 2020) showing how over-smoothing hurts the node classification performance.
Revised: This problem is also analyzed by (Oono & Suzuki, 2020) showing how over-smoothing hurts the node classification performance.

Original: To quantify the degree of over-smoothing, different measurements are proposed (Chen et al., 2020a; Zhao & Akoglu, 2020;
Revised: To quantify the degree of over-smoothing, different measurements are proposed (Chen et al., 2020a; Zhao & Akoglu, 2020;

Original: Liu et al., 2020; Zhou et al., 2020).
Revised: Liu et al., 2020; Zhou et al., 2020).

Original: For example, Mean Average Distance (Chen et al., 2020a) is proposed by calculating the divergences between learned node representations.
Revised: For example, Mean Average Distance (Chen et al., 2020a) is proposed by calculating the divergences between learned node representations.

Original: To make GNNs deeper and maintain performance, some nascent research works are proposed (Klicpera et al., 2019;
Revised: To make GNNs deeper and maintain performance, some nascent research works are proposed (Klicpera et al., 2019;

Original: Chen et al., 2020a; Zhao & Akoglu, 2020; Rong et al., 2020; Chen et al., 2020b; Liu et al., 2020;
Revised: Chen et al., 2020a; Zhao & Akoglu, 2020; Rong et al., 2020; Chen et al., 2020b; Liu et al., 2020;

Original: Zhou et al., 2020).
Revised: Zhou et al., 2020).

Original: Among those, some of them share the same logic of keeping the divergence between node representations but differ in specific methodologies.
Revised: Among those, some of them share the same logic of keeping the divergence between node representations but differ in specific methodologies.

Original: For example, PairNorm (Zhao &
Revised: For example, PairNorm (Zhao &

Original: Akoglu, 2020) introduces a normalization layer to keep the divergence of node representation from the original input node feature.
Revised: Akoglu, 2020) introduces a normalization layer to keep the divergence of node representation from the original input node feature.

Original: In DGN (Zhou et al., 2020), node representation by deep GNNs is regularized by group-based mutual information.
Revised: In DGN (Zhou et al., 2020), node representation by deep GNNs is regularized by group-based mutual information.

Original: However, the hyperparameters selection is important and data-driven, which means many efforts need to be paid for the hyperparameter searching.
Revised: However, the hyperparameters selection is important and data-driven, which means many efforts need to be paid for the hyperparameter searching.

Original: Some methods focus on changing the information aggregation scheme to make GNNs deeper, such as
Revised: Some methods focus on changing the information aggregation scheme to make GNNs deeper, such as

Original: APPNP (Klicpera et al., 2019), GCNII (Chen et al., 2020b), DropEdge (Rong et al., 2020) and etc.
Revised: APPNP (Klicpera et al., 2019), GCNII (Chen et al., 2020b), DropEdge (Rong et al., 2020) and etc.

Original: To the best of our knowledge, in dealing with over-smoothing problem, our Deeper-GXX is the first attempt to transfer hard-to-acquire discriminative knowledge into the topology information of the input graph by comparing adjacency vectors of nodes.
Revised: To the best of our knowledge, in dealing with over-smoothing problem, our Deeper-GXX is the first attempt to transfer hard-to-acquire discriminative knowledge into the topology information of the input graph by comparing adjacency vectors of nodes.

Original: Also, we analyze another problem (i.e., vanishing gradient) that hinders GNNs to be effectively deeper and discover the shading neighbors effect caused by simply applying ResNet on GNNs, and propose a GNN-based residual connection to avoid this issue and improve the performance of deeper GNNs.
Revised: Also, we analyze another problem (i.e., vanishing gradient) that hinders GNNs to be effectively deeper and discover the shading neighbors effect caused by simply applying ResNet on GNNs, and propose a GNN-based residual connection to avoid this issue and improve the performance of deeper GNNs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we focus on building deeper graph neural networks to effectively model graph data.
Revised: In this paper, we focus on building deeper graph neural networks to effectively model graph data.

Original: To this end, we first provide insights regarding why ResNet is not best suited for many deeper GNN solutions, i.e., the shading neighbors effect.
Revised: To this end, we first provide insights regarding why ResNet is not best suited for many deeper GNN solutions, i.e., the shading neighbors effect.

Original: Then we propose a new residual architecture, WeightDecaying Graph Residual Connection (WDG-ResNet) to address this effect.
Revised: Then we propose a new residual architecture, WeightDecaying Graph Residual Connection (WDG-ResNet) to address this effect.

Original: In addition, we propose a Topology-guided Graph Contrastive Loss (TGCL) to address the problem of over-smoothing, where we utilize graph topological information, pull the representations of connected node pairs closer, and push remote node pairs farther apart via contrastive learning regularization.
Revised: In addition, we propose a Topology-guided Graph Contrastive Loss (TGCL) to address the problem of over-smoothing, where we utilize graph topological information, pull the representations of connected node pairs closer, and push remote node pairs farther apart via contrastive learning regularization.

Original: Combining WDGResNet with TGCL, an end-to-end model named Deeper-GXX is proposed towards deeper GNNs.
Revised: Combining WDGResNet with TGCL, an end-to-end model named Deeper-GXX is proposed towards deeper GNNs.

Original: We provide the theoretical analysis of our proposed method, and demonstrate the effectiveness of
Revised: We provide the theoretical analysis of our proposed method, and demonstrate the effectiveness of

Original: Deeper-GXX by extensive experiment comparing with state-of-the-art de-oversmoothing algorithms.
Revised: Deeper-GXX by extensive experiment comparing with state-of-the-art de-oversmoothing algorithms.

Original: A case study regarding the missing feature scenario demonstrates the necessity to deepen the GNNs.
Revised: A case study regarding the missing feature scenario demonstrates the necessity to deepen the GNNs.

Original: A E XPERIMENT A. C ASE S TUDY : A TOY EXAMPLE
Revised: A E XPERIMENT A. C ASE S TUDY : A TOY EXAMPLE

Original: Why do we need to stack more layers of GNNs?
Revised: Why do we need to stack more layers of GNNs?

Original: To answer this question, we conduct the experiment in a toy example.
Revised: To answer this question, we conduct the experiment in a toy example.

Original: We first use the existing package (i.e., draw circle function in the Scikit-learn package) to generate a synthetic data set by setting the number of data points to be 1,000 and the noise level to be 0.01.
Revised: We first use the existing package (i.e., draw circle function in the Scikit-learn package) to generate a synthetic data set by setting the number of data points to be 1,000 and the noise level to be 0.01.

Original: Then, we measure the euclidean distance between each pair of data points, and if the the distance is less than a threshold, then this two data points are connected in a graph.
Revised: Then, we measure the euclidean distance between each pair of data points, and if the the distance is less than a threshold, then this two data points are connected in a graph.

Original: In this way, the adjacency matrix is derived after adding the self loop.
Revised: In this way, the adjacency matrix is derived after adding the self loop.

Original: Next, we sample 1% data points as the training set, 9% data points as the validation set, and 90% data points as the test set.
Revised: Next, we sample 1% data points as the training set, 9% data points as the validation set, and 90% data points as the test set.

Original: These data points are visualized in Figure 1a and the experimental results are shown in Figure 1b.
Revised: These data points are visualized in Figure 1a and the experimental results are shown in Figure 1b.

Original: In
Revised: In

Original: Figure 1a, we observe that the query node (the diamond in the red dashed circle) cannot rely on its closest labeled neighbor (the red star in the circle) to correctly predict its label (red or blue).
Revised: Figure 1a, we observe that the query node (the diamond in the red dashed circle) cannot rely on its closest labeled neighbor (the red star in the circle) to correctly predict its label (red or blue).

Original: Only by exploring longer paths consisting of more similar neighbors are we able to predict its label as blue.
Revised: Only by exploring longer paths consisting of more similar neighbors are we able to predict its label as blue.

Original: Figure 1b compares the classification accuracy of shallow GNNs and deeper GNNs.
Revised: Figure 1b compares the classification accuracy of shallow GNNs and deeper GNNs.

Original: We can see that deeper GNNs significantly outperform shallow ones by more than 11%, due to their abilities to explore longer paths on the graph.
Revised: We can see that deeper GNNs significantly outperform shallow ones by more than 11%, due to their abilities to explore longer paths on the graph.

Original: A. E XPERIMENTAL SETTING
Revised: A. E XPERIMENTAL SETTING

Original: In this subsection, we provide the detailed experimental setting for each experiment shown in Table 4.
Revised: In this subsection, we provide the detailed experimental setting for each experiment shown in Table 4.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: CiteSeer
Revised: A. M ORE RESULTS IN ABLATION STUDY

Original: [Table] [Table] = [Table]
Revised: In addition to the ablation study in Section 3.4, here in Table 5, we conduct more ablation experiments on CiteSeer, PubMed, Reddit, and OGB-Arxiv data sets, following the same experimental setting in

Original: PubMed [Table] [Table] .α
Revised: Table 3, to demonstrate the effectiveness of each component in Deeper-GXX by comparing it with three variants.

Original: [Table]
Revised: [Table]

Original: Reddit λ
Revised: A. V ISUALIZATION OF WEIGHT OF EACH LAYER WITH DIFFERENT WEIGHTING FUNCTIONS

Original: = 20 , [Table] α =
Revised: In this subsection

Original: [Table]
Revised: [Table]

Original: A. V ISUALIZATION

Original: In this subsection, we visualize the weight of each layer with different weighting functions on the
Revised: we visualize the weight of each layer with different weighting functions on the

Original: Cora data set.
Revised: Cora data set.

Original: In this experiment, we fix the feature dimension of the hidden layer to be 50; the total iteration is set to be 3000; the number of layers is set to be 60; the sampling batch size for
Revised: In this experiment, we fix the feature dimension of the hidden layer to be 50; the total iteration is set to be 3000; the number of layers is set to be 60; the sampling batch size for

Original: Deeper-GXX is 10; GCN is chosen as the base model; λ is set to be 20.
Revised: Deeper-GXX is 10; GCN is chosen as the base model; λ is set to be 20.

Original: In Figure 4, The x-axis is the index of each layer and the y-axis is the weight for each layer.
Revised: In Figure 4, The x-axis is the index of each layer and the y-axis is the weight for each layer.

Original: Deeper-GXX-D removes the decaying weight factor and only keeps the exponential cosine similarity e cos ( H (1) , ˜ H ( l ) ) to measure the weight for each layer.
Revised: Deeper-GXX-D removes the decaying weight factor and only keeps the exponential cosine similarity e cos ( H (1) , ˜ H ( l ) ) to measure the weight for each layer.

Original: Deeper-GXX-S achieves the simplified WDG-ResNet in Deeper-GXX, which removes the exponential cosine similarity e cos ( H (1) ,
Revised: Deeper-GXX-S achieves the simplified WDG-ResNet in Deeper-GXX, which removes the exponential cosine similarity e cos ( H (1) ,

Original: ˜ H ( l ) ) in Deeper-GXX.
Revised: ˜ H ( l ) ) in Deeper-GXX.

Original: By observation, we find that (1) ResNet sets the weight of each layer to be 1, which easily leads to shading neighbors effect when stacking more layers, because the faraway neighbor information becomes more dominant in the GCN information aggregation; (2) without weight decaying factor, the weight for each layer in Deeper-GXX-D fluctuates because they are randomly independent.
Revised: By observation, we find that (1) ResNet sets the weight of each layer to be 1, which easily leads to shading neighbors effect when stacking more layers, because the faraway neighbor information becomes more dominant in the GCN information aggregation; (2) without weight decaying factor, the weight for each layer in Deeper-GXX-D fluctuates because they are randomly independent.

Original: More specially, the weights for the last several layers (e.g., L=58 or L=60) are larger than the weights for the first several layers, which contradicts the intuition that the first several layers should be important than the last several layers; (3) the weights for each layer in both Deeper-GXX and Deeper-GXX-S reduce as the number of layers increase, which suggests that both of them could address the shading neighbors effect to some extents; (4) combining the results from Table 3, Deeper-GXX achieves better performance than Deeper-GXX-S, as it imposes larger weights to the first several layers, which verifies that the learnable similarity sim ( H (1) ,
Revised: More specially, the weights for the last several layers (e.g., L=58 or L=60) are larger than the weights for the first several layers, which contradicts the intuition that the first several layers should be important than the last several layers; (3) the weights for each layer in both Deeper-GXX and Deeper-GXX-S reduce as the number of layers increase, which suggests that both of them could address the shading neighbors effect to some extents; (4) combining the results from Table 3, Deeper-GXX achieves better performance than Deeper-GXX-S, as it imposes larger weights to the first several layers, which verifies that the learnable similarity sim ( H (1) ,

Original: ˜ H ( l ) ) achieves better performance with the enlarged hypothesis space for neural networks.
Revised: ˜ H ( l ) ) achieves better performance with the enlarged hypothesis space for neural networks.

Original: A. H YPERARAMETER A NALYSIS
Revised: A. H YPERARAMETER A NALYSIS

Original: In this subsection, we conduct the hyperparameter analysis of Deeper-GXX, regarding λ in the weight decaying function of Eq.
Revised: In this subsection, we conduct the hyperparameter analysis of Deeper-GXX, regarding λ in the weight decaying function of Eq.

Original: 4 and α in the overall loss function of Eq 1.
Revised: 4 and α in the overall loss function of Eq 1.

Original: [Figure]
Revised: [Figure]

Original: To analyze the hyperparameter λ , we fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 3000, the number of layers is set to be 60, the sampling batch size for
Revised: To analyze the hyperparameter λ , we fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 3000, the number of layers is set to be 60, the sampling batch size for

Original: Deeper-GXX is 10, and GCN is chosen as the base model.
Revised: Deeper-GXX is 10, and GCN is chosen as the base model.

Original: The experiment is repeated five times for each configuration.
Revised: The experiment is repeated five times for each configuration.

Original: In each sub-figure of Figure 5, the x-axis is the value of λ , and the y-axis is the accuracy of 60-layer GCN in the above setting.
Revised: In each sub-figure of Figure 5, the x-axis is the value of λ , and the y-axis is the accuracy of 60-layer GCN in the above setting.

Original: First, we can observe that it’s not true that
Revised: First, we can observe that it’s not true that

Original: Deeper-GXX achieves the best performance with a larger λ .
Revised: Deeper-GXX achieves the best performance with a larger λ .

Original: Specifically, we find that the optimal λ = 20 on Cora data set, the optimal λ = 10 on CiteSeer data set, the optimal λ = 18 on PubMed data set, and the optimal λ = 20 on Reddit data set.
Revised: Specifically, we find that the optimal λ = 20 on Cora data set, the optimal λ = 10 on CiteSeer data set, the optimal λ = 18 on PubMed data set, and the optimal λ = 20 on Reddit data set.

Original: Then, natural questions to ask are (1) what determines the optimal value of λ in different data sets?
Revised: Then, natural questions to ask are (1) what determines the optimal value of λ in different data sets?

Original: (2) can we gather some heuristics to narrow down the hyperparameter search space to efficiently establish effective GNNs ?
Revised: (2) can we gather some heuristics to narrow down the hyperparameter search space to efficiently establish effective GNNs ?

Original: Here, we provide our discovery.
Revised: Here, we provide our discovery.

Original: In the main text of the paper, we have analyzed that the decaying factor λ in Eq.
Revised: In the main text of the paper, we have analyzed that the decaying factor λ in Eq.

Original: 4 controls the number of effective layers in deeper GNNs by introducing the layer-wise dependency.
Revised: 4 controls the number of effective layers in deeper GNNs by introducing the layer-wise dependency.

Original: It means that larger λ slows down the weight decay and gives considerable large weights to more layers such that they can be effective, and the information aggregation scope of GNN extends as more multi-hop neighbors features are collected and aggregated.
Revised: It means that larger λ slows down the weight decay and gives considerable large weights to more layers such that they can be effective, and the information aggregation scope of GNN extends as more multi-hop neighbors features are collected and aggregated.

Original: In graph theory, diameter represents the scope of the graph, which is the largest value of the shortest path between any node pairs in the graph.
Revised: In graph theory, diameter represents the scope of the graph, which is the largest value of the shortest path between any node pairs in the graph.

Original: Therefore, the optimal λ should be restricted by the input graph, i.e., being close to the input graph diameter.
Revised: Therefore, the optimal λ should be restricted by the input graph, i.e., being close to the input graph diameter.

Original: Interestingly, our experiments reflect this observation.
Revised: Interestingly, our experiments reflect this observation.

Original: Combining the optimal λ in Figure 5 and the diameter in Table 5, for connected graphs PubMed and Reddit, the optimal λ is very close to the graph diameter.
Revised: Combining the optimal λ in Figure 5 and the diameter in Table 6, for connected graphs PubMed and Reddit, the optimal λ is very close to the graph diameter.

Original: This also happens to Cora (even though Cora is not connected), because the number of components is not large.
Revised: This also happens to Cora (even though Cora is not connected), because the number of components is not large.

Original: As for CiteSeer, the optimal λ is less than the diameter of its largest component.
Revised: As for CiteSeer, the optimal λ is less than the diameter of its largest component.

Original: A possible reason is that CiteSeer has many (i.e., 438) small components, which shrinks the information propagation scope, such that we do not need to stack many layers and we do not need to enlarge λ to the largest diameter (i.e., 28).
Revised: A possible reason is that CiteSeer has many (i.e., 438) small components, which shrinks the information propagation scope, such that we do not need to stack many layers and we do not need to enlarge λ to the largest diameter (i.e., 28).

Original: In general, based on the above analysis, we find the optimal value of λ can be searched around the diameter of the input graph.
Revised: In general, based on the above analysis, we find the optimal value of λ can be searched around the diameter of the input graph.

Original: [Table]
Revised: [Table]

Original: Diameter of the graph or the largest component
Revised: PubMed Reddit Number of nodes 2, 3, 19, 4, Connected graph or not

Original: [Table]
Revised: No No Yes

Original:  To analyze the hyperparameter α in DeeperGXX, we fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 3000, the number of layers is set to be 60, the sampling batch size for Deeper-GXX is 10, GCN is chosen as the base model, and the data set is Cora.
Revised: Yes Number of components Diameter of the graph or the largest component To analyze the hyperparameter α in DeeperGXX, we fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 3000, the number of layers is set to be 60, the sampling batch size for Deeper-GXX is 10, GCN is chosen as the base model, and the data set is Cora.

Original: We gradually increase the value of α and record the accuracy.
Revised: We gradually increase the value of α and record the accuracy.

Original: The experiment is repeated five times in each setting.
Revised: The experiment is repeated five times in each setting.

Original: In Figure 6, the x-axis is α and the y-axis is the accuracy score.
Revised: In Figure 6, the x-axis is α and the y-axis is the accuracy score.

Original: By observation, when α = 1 , the performance is worst and the performance begins to increase by decreasing the value of α .
Revised: By observation, when α = 1 , the performance is worst and the performance begins to increase by decreasing the value of α .

Original: It achieves the best accuracy when α = 0 .
Revised: It achieves the best accuracy when α = 0 .

Original: The performance starts to decrease again if we further decrease the value of α .
Revised: The performance starts to decrease again if we further decrease the value of α .

Original: Our conjecture is that when α is large, it will dominate the overall objective function, thus jeopardizing the classification performance.
Revised: Our conjecture is that when α is large, it will dominate the overall objective function, thus jeopardizing the classification performance.

Original: Besides, if we set the value of α to be a small number (i.e., α = 0 . 001 ), the performance also decreases.
Revised: Besides, if we set the value of α to be a small number (i.e., α = 0 . 001 ), the performance also decreases.

Original: In addition, comparing with the performance without using TGCL regularization (i.e., α = 0 ), our proposed method with α = 0 .
Revised: In addition, comparing with the performance without using TGCL regularization (i.e., α = 0 ), our proposed method with α = 0 .

Original: 03 can boost the performance by more than 1.8%, which demonstrates that our proposed TGCL alleviates the issue of oversmoothing to some extent.
Revised: 03 can boost the performance by more than 1.8%, which demonstrates that our proposed TGCL alleviates the issue of oversmoothing to some extent.

Original: A. E FFICIENCY A NALYSIS
Revised: A. E FFICIENCY A NALYSIS

Original: In this subsection, we conduct an efficiency analysis regarding our proposed method in the Cora data set.
Revised: In this subsection, we conduct an efficiency analysis regarding our proposed method in the Cora data set.

Original: We fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 1500, the sampling batch size for Deeper-GXX and Deeper-GXX-S is 10, and GCN is chosen as the base model.
Revised: We fix the feature dimension of the hidden layer to be 50, the total iteration is set to be 1500, the sampling batch size for Deeper-GXX and Deeper-GXX-S is 10, and GCN is chosen as the base model.

Original: We gradually increase the number of layers and record the running time.
Revised: We gradually increase the number of layers and record the running time.

Original: In Figure 7, the x-axis is the number of layers and the y-axis is the running time in second.
Revised: In Figure 7, the x-axis is the number of layers and the y-axis is the running time in second.

Original: We observe that the running time of both Deeper-GXX and Deeper-GXX-S is linearly proportional to the number of layers.
Revised: We observe that the running time of both Deeper-GXX and Deeper-GXX-S is linearly proportional to the number of layers.

Original: Comparing the running time of Deeper-GXX, the running time of DeeperGXX-S is further reduced after the weighting function in Deeper-GXX
Revised: Comparing the running time of Deeper-GXX, the running time of Deeper-GXX-S is further reduced after the weighting function in Deeper-GXX

Original: (e.g., sim ( · ) ) is replaced by a constant.
Revised: (e.g., sim ( · ) ) is replaced by a constant.

Original: B A NALYSIS B. P ROOF OF P ROPOSITION
Revised: B A NALYSIS B. P ROOF OF P ROPOSITION

Original: Proof 1 Following the theoretical analysis in (van den Oord et al., 2018) Section 2.3, the optimal value of f ( z i , z j ) is given by P ( z j | z i ) P ( z j ) .
Revised: Proof 1 Following the theoretical analysis in (van den Oord et al., 2018) Section 2.3, the optimal value of f ( z i , z j ) is given by P ( z j | z i ) P ( z j ) .

Original: Thus, the weighted supervised contrastive loss could be rewritten as follows:
Revised: Thus, the weighted supervised contrastive loss could be rewritten as follows:

Original: [Equation]
Revised: [Equation]

Original: Since ( v i , v k ) is defined as a remote (i.e., negative) node pair, it means that node v i and node v k are not connected in the graph, i.e., A ( i, k )
Revised: Since ( v i , v k ) is defined as a remote (i.e., negative) node pair, it means that node v i and node v k are not connected in the graph, i.e., A i,k

Original: =
Revised: =

Original: A ( k, i ) = 0 .
Revised: A k,i = 0 .

Original: Therefore, we have γ ik ∈ (1 , 2] for all remote nodes v k and σ ij ∈ (0 , 1] for all neighbor nodes v j with hamming distance measurement, which leads to 1 σ ij · P ( z j ) P ( z j | z i ) ≥ P ( z j ) P ( z j | z i ) and γ ik P ( z k | z i ) P ( z k ) ≥ P ( z k | z i ) P ( z k ) .
Revised: Therefore, we have γ ik ∈ (1 , 2] for all remote nodes v k and σ ij ∈ (0 , 1] for all neighbor nodes v j with hamming distance measurement, which leads to 1 σ ij · P ( z j ) P ( z j | z i ) ≥ P ( z j ) P ( z j | z i ) and γ ik P ( z k | z i ) P ( z k ) ≥ P ( z k | z i ) P ( z k ) .

Original: Thus, we have
Revised: Thus, we have

Original: [Equation]
Revised: [Equation]

Original: Finally, we have I ( z i , z j ) ≥ −L TGCL
Revised: Finally, we have I ( z i , z j ) ≥ −L TGCL

Original: + E v i ∼ V log( | ¯ N i | ) , which completes the proof.
Revised: + E v i ∼ V log( | ¯ N i | ) , which completes the proof.

Original: B. S AMPLING M ETHOD FOR TGCL
Revised: B. S AMPLING M ETHOD FOR TGCL

Original: To realize TGCL loss function expressed in Eq.
Revised: To realize TGCL loss function expressed in Eq.

Original: 6, we need to get the positive nodes v j and negative nodes v k towards the selected central node v i .
Revised: 6, we need to get the positive nodes v j and negative nodes v k towards the selected central node v i .

Original: To avoid iterating over all existing nodes or randomly sampling several nodes, we propose to sample positive nodes v j and negative nodes v k from the star subgraph S i of the central node v i .
Revised: To avoid iterating over all existing nodes or randomly sampling several nodes, we propose to sample positive nodes v j and negative nodes v k from the star subgraph S i of the central node v i .

Original: Moreover, to make the sampling be scalable and to reduce the search space of negative nodes, we propose a batch sampling method.
Revised: Moreover, to make the sampling be scalable and to reduce the search space of negative nodes, we propose a batch sampling method.

Original: As shown in Figure 8, the batch size is controlled by the number of central nodes (i.e., star nodes in the figure).
Revised: As shown in Figure 8, the batch size is controlled by the number of central nodes (i.e., star nodes in the figure).

Original: For each central node, the positive nodes are those 1-hop neighbors, and the negative nodes consist of unreachable nodes.
Revised: For each central node, the positive nodes are those 1-hop neighbors, and the negative nodes consist of unreachable nodes.

Original: In our batch sampling, we strictly constrain that the positive nodes are only from the 1-hop neighborhood for the following three reasons: (1) they are efficient to be accessed; (2) considering all k -hop neighbors as positive will enlarge the scope of positive nodes and further decrease the intimacy of the directly connected nodes; (3) 1-hop positive nodes in the star subgraph can preserve enough useful information, compared with the positive nodes from the whole graph.
Revised: In our batch sampling, we strictly constrain that the positive nodes are only from the 1-hop neighborhood for the following three reasons: (1) they are efficient to be accessed; (2) considering all k -hop neighbors as positive will enlarge the scope of positive nodes and further decrease the intimacy of the directly connected nodes; (3) 1-hop positive nodes in the star subgraph can preserve enough useful information, compared with the positive nodes from the whole graph.

Original: For the third point, we prove it through the graph influence loss (Huang & Zitnik, 2020) in Proposition 2, and the formal definition of graph influence loss is given in the following paragraph after Proposition 2.
Revised: For the third point, we prove it through the graph influence loss (Huang & Zitnik, 2020) in Proposition 2, and the formal definition of graph influence loss is given in the following paragraph after Proposition 2.

Original: Proposition 2 (Bounded Graph Influence Loss for Sampling Positive Pairs Locally) Taking
Revised: Proposition 2 (Bounded Graph Influence Loss for Sampling Positive Pairs Locally) Taking

Original: GCN as an example of GNN, the graph influence loss R ( v c ) on node v c w.r.t positive nodes from the whole graph against positive nodes from the 1-hop neighborhood star subgraph is bounded by
Revised: GCN as an example of GNN, the graph influence loss R ( v c ) on node v c w.r.t positive nodes from the whole graph against positive nodes from the 1-hop neighborhood star subgraph is bounded by

Original: R ( v c ) ≤ ( n − d c ) 
Revised: R ( v c ) ≤ ( n − d c ) µ ( D

Original: [Equation]
Revised: [Equation]

Original:  |  , where n is the number of nodes, d c is the degree of node v c including the self-loop, µ is a constant, ¯ P ∗ is the path from center node v c to a 1-hop outside node v s which has the maximal node influence I v c ,v s , and | ¯ P ∗ | denotes the number of nodes in path ¯ P ∗ .
Revised: ) | ¯ P∗| , where n is the number of nodes, d c is the degree of node v c including the self-loop, µ is a constant, ¯ P ∗ is the path from center node v c to a 1-hop outside node v s which has the maximal node influence I v c ,v s , and | ¯ P ∗ | denotes the number of nodes in path ¯ P ∗ .

Original: Proof 

Revised: Proof

Original: According to the assumption of (Wang & Leskovec, 2020), σ ( · ) can be identity function and
Revised: According to the assumption of (Wang & Leskovec, 2020), σ ( · ) can be identity function and

Original: W ( · ) can be identity matrix.
Revised: W ( · ) can be identity matrix.

Original: Then, the hidden node representation (of node v c ) in the last layer of
Revised: Then, the hidden node representation (of node v c ) in the last layer of

Original: GCN can be written as follows.
Revised: GCN can be written as follows.

Original: [Equation]
Revised: [Equation]

Original: Then, based on the above equation, we can iteratively replace h ( ∞ ) i with its neighbors until the representation h ( ∞ ) s of node v s is included.
Revised: Then, based on the above equation, we can iteratively replace h ( ∞ ) i with its neighbors until the representation h ( ∞ ) s of node v s is included.

Original: The extension procedure is written as follows.
Revised: The extension procedure is written as follows.

Original: [Equation]
Revised: [Equation]

Original: The above equation suggests that the influence from the positive node v s to the center node v c is through the path P
Revised: The above equation suggests that the influence from the positive node v s to the center node v c is through the path P

Original: = ( v c , v i , v j , . .
Revised: = ( v c , v i , v j , . .

Original: . , v k , v s ) .
Revised: . , v k , v s ) .

Original: Following the above path formation and assume the edge weight A ( i, j ) as the positive constant, according to (Huang & Zitnik, 2020), we can obtain the node influence I v c ,v s of v s on v c as follows.
Revised: Following the above path formation and assume the edge weight A ( i, j ) as the positive constant, according to (Huang & Zitnik, 2020), we can obtain the node influence I v c ,v s of v s on v c as follows.

Original: [Equation]
Revised: [Equation]

Original: where µ is a constant, D ¯ P GM is the geometric mean of degree of nodes sitting in path ¯ P , and ¯ P is the path from the positive node v s to the center node v c that could generate the maximal multiplication of normalized edge weight, | ¯ P| denotes the number of nodes in path ¯ P .
Revised: where µ is a constant, D ¯ P GM is the geometric mean of degree of nodes sitting in path ¯ P , and ¯ P is the path from the positive node v s to the center node v c that could generate the maximal multiplication of normalized edge weight, | ¯ P| denotes the number of nodes in path ¯ P .

Original: The above analysis suggests that the node influence of long-distance positive nodes is decaying.
Revised: The above analysis suggests that the node influence of long-distance positive nodes is decaying.

Original: Hence, the graph influence loss about learning node v c from the whole graph positive nodes versus from the 1-hop localized positive nodes can be expressed expressed as follows.
Revised: Hence, the graph influence loss about learning node v c from the whole graph positive nodes versus from the 1-hop localized positive nodes can be expressed expressed as follows.

Original: [Equation]
Revised: [Equation]

Original: where I G ( v c ) denotes global influence, I L ( v c ) is the influence for star subgraph, d c is the degree of node v c (including self-loop), and
Revised: where I G ( v c ) denotes global influence, I L ( v c ) is the influence for star subgraph, d c is the degree of node v c (including self-loop), and

Original: [Equation]
Revised: [Equation]

Original:  is the maximal among all
Revised: | is the maximal among all

Original: [Equation]
Revised: [Equation]

Original: Specifically, the graph influence loss (Huang & Zitnik, 2020) R ( v c ) can be expressed as R ( v c ) =
Revised: Specifically, the graph influence loss (Huang & Zitnik, 2020) R ( v c ) can be expressed as R ( v c ) =

Original: I G ( v c ) − I L ( v c ) , which is determined by the global graph influence on v c (i.e., I G ( v c ) ) and the star subgraph influence on v c (i.e., I L ( v c ) ).
Revised: I G ( v c ) − I L ( v c ) , which is determined by the global graph influence on v c (i.e., I G ( v c ) ) and the star subgraph influence on v c (i.e., I L ( v c ) ).

Original: Then, to compute the graph influence I G ( v c ) , we need to compute node influence of each node v j to node v c , where node v j is reachable from node v c .
Revised: Then, to compute the graph influence I G ( v c ) , we need to compute node influence of each node v j to node v c , where node v j is reachable from node v c .

Original: Based on the final output node representation vectors, the node influence is expressed as I v c ,v j = ∥ 
Revised: Based on the final output node representation vectors, the node influence is expressed as I v c ,v j = ∥ ∂ h ( ∞ ) c [Equation]

Original: ∂ h ( ∞ ) c /∂ h ( ∞ ) j ∥ , and the norm can be any subordinate norm (Wang & Leskovec, 2020).
Revised: ∞ [Equation] ∥ , and the norm can be any subordinate norm (Wang & Leskovec, 2020).

Original: Then, I G ( v c ) is computed by the L 1 -norm of the following vector, i.e., I G ( v c ) =
Revised: Then, I G ( v c ) is computed by the L 1 -norm of the following vector, i.e., I G ( v c ) =

Original: ∥
Revised: ∥

Original: [ I v c ,v 1 , I v c ,v 2 , . .
Revised: [ I v c ,v 1 , I v c ,v 2 , . .

Original: , I v c ,v n ] ∥ 1 .
Revised: , I v c ,v n ] ∥ 1 .

Original: Similarly, we can compute the star subgraph influence I L ( v c ) on node v c .
Revised: Similarly, we can compute the star subgraph influence I L ( v c ) on node v c .

Original: The only difference is that we collect each reachable node v j in the star subgraph L (i.e., 1-hop neighbours of v c ).
Revised: The only difference is that we collect each reachable node v j in the star subgraph L (i.e., 1-hop neighbours of v c ).

Original: Overall, in Proposition 2, we show why positive pairs can be locally sampled with the support from graph influence loss of a node representation vector output by the GCN final layer.
Revised: Overall, in Proposition 2, we show why positive pairs can be locally sampled with the support from graph influence loss of a node representation vector output by the GCN final layer.