Original: T WO S IDES OF THE S AME C OIN : H ETEROPHILY AND O VERSMOOTHING IN G RAPH C ONVOLUTIONAL N EURAL N ETWORKS
Revised: T WO S IDES OF THE S AME C OIN : H ETEROPHILY AND O VERSMOOTHING IN G RAPH C ONVOLUTIONAL N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: In node classiﬁcation tasks, heterophily and oversmoothing are two problems that can hurt the performance of graph convolutional neural networks (GCNs).
Revised: In node classiﬁcation tasks, heterophily and oversmoothing are two problems that can hurt the performance of graph convolutional neural networks (GCNs).

Original: The heterophily problem refers to the model’s inability to handle heterophilous graphs where neighboring nodes belong to different classes; the oversmoothing problem refers to the model’s degenerated performance with increasing number of layers.
Revised: The heterophily problem refers to the model’s inability to handle heterophilous graphs where neighboring nodes belong to different classes; the oversmoothing problem refers to the model’s degenerated performance with increasing number of layers.

Original: These two seemingly unrelated problems have been studied mostly independently, but there is recent empirical evidence that solving one problem may beneﬁt the other.
Revised: These two seemingly unrelated problems have been studied mostly independently, but there is recent empirical evidence that solving one problem may beneﬁt the other.

Original: In this work, beyond empirical observations, we aim to: (1) analyze the heterophily and oversmoothing problems from a uniﬁed theoretical perspective, (2) identify the common causes of the two problems based on our theories, and (3) propose simple yet effective strategies to address the common causes.
Revised: In this work, beyond empirical observations, we aim to: (1) analyze the heterophily and oversmoothing problems from a uniﬁed theoretical perspective, (2) identify the common causes of the two problems based on our theories, and (3) propose simple yet effective strategies to address the common causes.

Original: In our theoretical analysis, we show that the common causes of the heterophily and oversmoothing problems—namely, the relative degree of a node (compared to its neighbors) and its heterophily level—trigger the node representations in consecutive layers to "move" closer to the original decision boundary, which increases the misclassiﬁcation rate of node labels under certain constraints.
Revised: In our theoretical analysis, we show that the common causes of the heterophily and oversmoothing problems—namely, the relative degree of a node (compared to its neighbors) and its heterophily level—trigger the node representations in consecutive layers to "move" closer to the original decision boundary, which increases the misclassiﬁcation rate of node labels under certain constraints.

Original: We theoretically show that: (1) Nodes with high heterophily have a higher misclassiﬁcation rate.
Revised: We theoretically show that: (1) Nodes with high heterophily have a higher misclassiﬁcation rate.

Original: (2) Even with low heterophily, degree disparity in a node’s neighborhood can inﬂuence the movements of node representations and result in a "pseudo-heterophily" situation, which helps to explain oversmoothing.
Revised: (2) Even with low heterophily, degree disparity in a node’s neighborhood can inﬂuence the movements of node representations and result in a "pseudo-heterophily" situation, which helps to explain oversmoothing.

Original: (3) Allowing not only positive, but also negative messages during message passing can help counteract the common causes of the two problems.
Revised: (3) Allowing not only positive, but also negative messages during message passing can help counteract the common causes of the two problems.

Original: Based on our theoretical insights, we propose simple modiﬁcations to the GCN architecture (i.e., learned degree corrections and signed messages), and we
Revised: Based on our theoretical insights, we propose simple modiﬁcations to the GCN architecture (i.e., learned degree corrections and signed messages), and we

Original: show that they alleviate the heteorophily and oversmoothing problems with extensive experiments on nine real networks.
Revised: show that they alleviate the heteorophily and oversmoothing problems with extensive experiments on nine real networks.

Original: Compared to other approaches, which tend to work well in either heterophily or oversmoothing, our modiﬁed GCN model performs well in both problems.
Revised: Compared to other approaches, which tend to work well in either heterophily or oversmoothing, our modiﬁed GCN model performs well in both problems.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In recent years, GCNs (Defferrard et al., 2016; Kipf & Welling, 2016; Veli ˇ ckovi ´ c et al., 2017) have been widely used in applications ranging from social science (Li & Goldwasser, 2019) and biology (Yan et al., 2019) to program understanding (Allamanis et al., 2018; Shi et al., 2019).
Revised: In recent years, GCNs (Defferrard et al., 2016; Kipf & Welling, 2016; Veli ˇ ckovi ´ c et al., 2017) have been widely used in applications ranging from social science (Li & Goldwasser, 2019) and biology (Yan et al., 2019) to program understanding (Allamanis et al., 2018; Shi et al., 2019).

Original: A typical GCN architecture (Gilmer et al., 2017) for the node classiﬁcation task can be decomposed into two main components: propagation/aggregation, and combination.
Revised: A typical GCN architecture (Gilmer et al., 2017) for the node classiﬁcation task can be decomposed into two main components: propagation/aggregation, and combination.

Original: Messages are ﬁrst exchanged between neighboring nodes, then aggregated.
Revised: Messages are ﬁrst exchanged between neighboring nodes, then aggregated.

Original: Afterwards, the messages are combined with the self-representations (a.k.a., the current node representations) to update the node representations.
Revised: Afterwards, the messages are combined with the self-representations (a.k.a., the current node representations) to update the node representations.

Original: Though GCNs are effective, they have some key limitations.
Revised: Though GCNs are effective, they have some key limitations.

Original: The ﬁrst limitation is the “oversmoothing" problem (Li et al., 2018): the performance of GCNs degrade when stacking many layers.
Revised: The ﬁrst limitation is the “oversmoothing" problem (Li et al., 2018): the performance of GCNs degrade when stacking many layers.

Original: Recent work has found that oversmoothing could be caused by
Revised: Recent work has found that oversmoothing could be caused by

Original: GCNs exponentially losing expressive power in the node classiﬁcation task (Oono & Suzuki, 2019) and that the node representations converge to a stationary state which is decided by the degree of the nodes and the input features (Chen et al., 2020; Wang et al., 2019;
Revised: GCNs exponentially losing expressive power in the node classiﬁcation task (Oono & Suzuki, 2019) and that the node representations converge to a stationary state which is decided by the degree of the nodes and the input features (Chen et al., 2020; Wang et al., 2019;

Original: Rong et al., 2019; Rossi et al., 2020).
Revised: Rong et al., 2019; Rossi et al., 2020).

Original: These works focus on the analysis of the steady state in the limit of inﬁnite layers, but they do not explore the dynamics on how oversmoothing is triggered, or which nodes tend to cause it.
Revised: These works focus on the analysis of the steady state in the limit of inﬁnite layers, but they do not explore the dynamics on how oversmoothing is triggered, or which nodes tend to cause it.

Original: Other works like (Xu et al., 2018; Wang et al., 2019; Chen et al., 2020) focus on heuristic-based model designs to alleviate the oversmoothing problem.
Revised: Other works like (Xu et al., 2018; Wang et al., 2019; Chen et al., 2020) focus on heuristic-based model designs to alleviate the oversmoothing problem.

Original: To ﬁll this gap, we seek to theoretically analyze the dynamics around oversmoothing.
Revised: To ﬁll this gap, we seek to theoretically analyze the dynamics around oversmoothing.

Original: In our empirical analysis, by dividing the nodes in real graphs into several degree groups and tracking their average accuracy while increasing the layers of GCN (Kipf & Welling, 2016), we observe two phases (Fig. 1): the initial stage when higher-degree nodes have higher accuracy and the developing stage when the accuracy of higher-degree nodes decreases more sharply (cf. § 5.4 for details).
Revised: In our empirical analysis, by dividing the nodes in real graphs into several degree groups and tracking their average accuracy while increasing the layers of GCN (Kipf & Welling, 2016), we observe two phases (Fig. 1): the initial stage when higher-degree nodes have higher accuracy and the developing stage when the accuracy of higher-degree nodes decreases more sharply (cf. § 5.4 for details).

Original: As we will show theoretically and empirically, the low-degree nodes are the ones that trigger oversmoothing.
Revised: As we will show theoretically and empirically, the low-degree nodes are the ones that trigger oversmoothing.

Original: Though Chen et al.
Revised: Though Chen et al.

Original: (2020) provide some explanations for the developing stage by analyzing the convergence rate, their theory fails to characterize the initial stage.
Revised: (2020) provide some explanations for the developing stage by analyzing the convergence rate, their theory fails to characterize the initial stage.

Original: In fact, we ﬁnd that the analysis of initial stage provides valuable insights about oversmoothing.
Revised: In fact, we ﬁnd that the analysis of initial stage provides valuable insights about oversmoothing.

Original: The second limitation of GCNs is their poor performance on heterophilous graphs (Pei et al., 2019; Lim et al., 2021), which—unlike homophilous graphs—have many neighboring nodes that belong to different classes (Newman, 2002).
Revised: The second limitation of GCNs is their poor performance on heterophilous graphs (Pei et al., 2019; Lim et al., 2021), which—unlike homophilous graphs—have many neighboring nodes that belong to different classes (Newman, 2002).

Original: For instance, in protein networks, amino acids of different types tend to form links (Zhu et al., 2020), and in transaction networks, fraudsters are more likely to connect to accomplices than to other fraudsters (Pandit et al., 2007).
Revised: For instance, in protein networks, amino acids of different types tend to form links (Zhu et al., 2020), and in transaction networks, fraudsters are more likely to connect to accomplices than to other fraudsters (Pandit et al., 2007).

Original: Most
Revised: Most

Original: GCNs (Kipf & Welling, 2016;
Revised: GCNs (Kipf & Welling, 2016;

Original: Veli ˇ ckovi ´ c et al., 2017) fail to effectively capture heterophily, so emerging works have proposed some effective designs (Pei et al., 2019; Zhu et al., 2020), including signed messages (positive and negative interactions between nodes) proposed by concurrent works (Chien et al., 2021;
Revised: Veli ˇ ckovi ´ c et al., 2017) fail to effectively capture heterophily, so emerging works have proposed some effective designs (Pei et al., 2019; Zhu et al., 2020), including signed messages (positive and negative interactions between nodes) proposed by concurrent works (Chien et al., 2021;

Original: Bo et al., 2021).
Revised: Bo et al., 2021).

Original: Though these works justify signed messages from a spectral perspective, they focus on asymptotic results, requiring an inﬁnite number of ﬁlters.
Revised: Though these works justify signed messages from a spectral perspective, they focus on asymptotic results, requiring an inﬁnite number of ﬁlters.

Original: Our work does not make this assumption, and our theory works in more practical scenarios (§ 3.4).
Revised: Our work does not make this assumption, and our theory works in more practical scenarios (§ 3.4).

Original: These two limitations have mostly been studied independently, but recent work on oversmoothing (Chen et al., 2020) was shown empirically to address heterophily, and vice versa (Chien et al., 2021).
Revised: These two limitations have mostly been studied independently, but recent work on oversmoothing (Chen et al., 2020) was shown empirically to address heterophily, and vice versa (Chien et al., 2021).

Original: Motivated by this empirical observation, we set out to understand the theoretical connection between the oversmoothing and heterophily problems by studying the change in node representations during message passing.
Revised: Motivated by this empirical observation, we set out to understand the theoretical connection between the oversmoothing and heterophily problems by studying the change in node representations during message passing.

Original: Speciﬁcally, we make the following contributions:
Revised: Speciﬁcally, we make the following contributions:

Original: • Theory: We introduce the ﬁrst theory that explains the connections between heterophily and oversmoothing which opens up the possibility of jointly studying the two problems.
Revised: • Theory: We introduce the ﬁrst theory that explains the connections between heterophily and oversmoothing which opens up the possibility of jointly studying the two problems.

Original: • Insights: We ﬁnd that under conditions, the relative degree of a node (compared to its neighbors) and its heterophily level affect the misclassiﬁcation rate of node labels in the oversmoothing and heterophily problems.
Revised: • Insights: We ﬁnd that under conditions, the relative degree of a node (compared to its neighbors) and its heterophily level affect the misclassiﬁcation rate of node labels in the oversmoothing and heterophily problems.

Original: We further prove that signed messages can mitigate their negative impact.
Revised: We further prove that signed messages can mitigate their negative impact.

Original: • Improved model & empirical analysis: Based on these insights, we design a generalized GCN model, GGCN, that allows negative interactions between nodes and compensates for the effect of low-degree nodes through a learned rescaling scheme.
Revised: • Improved model & empirical analysis: Based on these insights, we design a generalized GCN model, GGCN, that allows negative interactions between nodes and compensates for the effect of low-degree nodes through a learned rescaling scheme.

Original: Our empirical results show that our model is robust to oversmoothing, achieves state-of-the-art performance on datasets with high levels of heterophily, and achieves competitive performance on homophilous datasets.
Revised: Our empirical results show that our model is robust to oversmoothing, achieves state-of-the-art performance on datasets with high levels of heterophily, and achieves competitive performance on homophilous datasets.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: We ﬁrst provide the notations & deﬁnitions that we use in the paper, and a brief background on GCNs.
Revised: We ﬁrst provide the notations & deﬁnitions that we use in the paper, and a brief background on GCNs.

Original: Notation.
Revised: Notation.

Original: We denote an unweighted and self-loop-free graph as G ( V , E ) and its adjacency matrix as
Revised: We denote an unweighted and self-loop-free graph as G ( V , E ) and its adjacency matrix as

Original: A .
Revised: A .

Original: We represent the degree of node v i ∈ V by d i , and the degree matrix—which is a diagonal matrix whose elements are node degrees—by D .
Revised: We represent the degree of node v i ∈ V by d i , and the degree matrix—which is a diagonal matrix whose elements are node degrees—by D .

Original: Let N i be the set of nodes directly connected to v i , i.e., its neighbors.
Revised: Let N i be the set of nodes directly connected to v i , i.e., its neighbors.

Original: I is the identity matrix.
Revised: I is the identity matrix.

Original: We denote the node representations at l -th layer as F ( l ) , and the i -th row of F is f ( l ) i , which is the representation of node v i .
Revised: We denote the node representations at l -th layer as F ( l ) , and the i -th row of F is f ( l ) i , which is the representation of node v i .

Original: The input node features are given by
Revised: The input node features are given by

Original: F (0) .
Revised: F (0) .

Original: The weight matrix and bias vector at the l -th layer are denoted as W ( l ) and b ( l ) , respectively.
Revised: The weight matrix and bias vector at the l -th layer are denoted as W ( l ) and b ( l ) , respectively.

Original: Homophily and Heterophily.
Revised: Homophily and Heterophily.

Original: Given a set of node labels/classes, homophily captures the tendency of a node to have the same class as its neighbors.
Revised: Given a set of node labels/classes, homophily captures the tendency of a node to have the same class as its neighbors.

Original: Speciﬁcally, the homophily of node v i is deﬁned as h i ≡ E (cid:16) |N si | |N i | (cid:17) , where N si is the set of neighboring nodes with the same label as v i , | · | is the cardinality operator, and the expectation is taken over the randomness of the node labels.
Revised: Speciﬁcally, the homophily of node v i is deﬁned as h i ≡ E (cid:16) |N si | |N i | (cid:17) , where N si is the set of neighboring nodes with the same label as v i , | · | is the cardinality operator, and the expectation is taken over the randomness of the node labels.

Original: High homophily corresponds to low heterophily, and vice versa, so we use these terms interchangeably.
Revised: High homophily corresponds to low heterophily, and vice versa, so we use these terms interchangeably.

Original: Supervised Node Classiﬁcation Task.
Revised: Supervised Node Classiﬁcation Task.

Original: We focus on node classiﬁcation: given a subset of labeled nodes (from a label set L ), the goal is to learn a mapping F : f (0) i (cid:55)→
Revised: We focus on node classiﬁcation: given a subset of labeled nodes (from a label set L ), the goal is to learn a mapping F : f (0) i (cid:55)→

Original: y i between each node v i and its ground truth label y i ∈ L .
Revised: y i between each node v i and its ground truth label y i ∈ L .

Original: We use ˆ y i to represent the predicted label of v i and deﬁne the misclassiﬁcation rate of a ﬁxed learnt model over a node set V as: E (cid:16) |{ v i |
Revised: We use ˆ y i to represent the predicted label of v i and deﬁne the misclassiﬁcation rate of a ﬁxed learnt model over a node set V as: E (cid:16) |{ v i |

Original: ˆ y i (cid:54)
Revised: ˆ y i (cid:54)

Original: =
Revised: =

Original: y i ,v i ∈V}| |V| (cid:17) .
Revised: y i ,v i ∈V}| |V| (cid:17) .

Original: The expectation is taken over the randomness of graph structures and input node features.
Revised: The expectation is taken over the randomness of graph structures and input node features.

Original: Graph Convolutional Neural Networks.
Revised: Graph Convolutional Neural Networks.

Original: In node classiﬁcation tasks, an L -layer GCN contains two components (Gilmer et al., 2017): (1) neighborhood propagation and aggregation: (cid:99) f ( l ) i = AGGREGATE ( f ( l ) j , v j ∈ N i ), and (2) combination: f ( l +1) i = COMBINE ( (cid:99) f
Revised: In node classiﬁcation tasks, an L -layer GCN contains two components (Gilmer et al., 2017): (1) neighborhood propagation and aggregation: (cid:99) f ( l ) i = AGGREGATE ( f ( l ) j , v j ∈ N i ), and (2) combination: f ( l +1) i = COMBINE ( (cid:99) f

Original: ( l ) i , f ( l ) i ), where AGGREGATE and COMBINE are learnable functions.
Revised: ( l ) i , f ( l ) i ), where AGGREGATE and COMBINE are learnable functions.

Original: The loss is given by
Revised: The loss is given by

Original: [Equation]
Revised: [Equation]

Original: The vanilla GCN suggests a renormalization trick on the adjacency
Revised: The vanilla GCN suggests a renormalization trick on the adjacency

Original: A to prevent gradient explosion (Kipf & Welling, 2016).
Revised: A to prevent gradient explosion (Kipf & Welling, 2016).

Original: The ( l + 1) -th output is given by: F ( l +1)
Revised: The ( l + 1) -th output is given by: F ( l +1)

Original: =
Revised: =

Original: σ ( ˜ AF ( l ) W ( l ) ) ,
Revised: σ ( ˜ AF ( l ) W ( l ) ) ,

Original: where ˜ A = ˜ D − 1 / 2 ( I + A ) ˜ D − 1 / 2 , ˜ D is the degree matrix of I + A , and σ is ReLU.
Revised: where ˜ A = ˜ D − 1 / 2 ( I + A ) ˜ D − 1 / 2 , ˜ D is the degree matrix of I + A , and σ is ReLU.

Original: When the non-linearities in the vanilla GCN are removed, it reduces to a linear model called SGC (Wu et al., 2019), which has competitive performance and is widely used in theoretical analyses (Oono & Suzuki, 2019;
Revised: When the non-linearities in the vanilla GCN are removed, it reduces to a linear model called SGC (Wu et al., 2019), which has competitive performance and is widely used in theoretical analyses (Oono & Suzuki, 2019;

Original: Chen et al., 2020).
Revised: Chen et al., 2020).

Original: For SGC, the l -th layer representations are given by: F ( l ) =
Revised: For SGC, the l -th layer representations are given by: F ( l ) =

Original: ˜
Revised: ˜

Original: A l F (0) and the last layer is a logistic-regression layer:
Revised: A l F (0) and the last layer is a logistic-regression layer:

Original: ˆ y i =
Revised: ˆ y i =

Original: Softmax ( F ( L ) W ( L )
Revised: Softmax ( F ( L ) W ( L )

Original: + b ( L ) ) .
Revised: + b ( L ) ) .

Original: We note that only one weight matrix W ( L ) is learned; it is equivalent to the products of all weight matrices in a linear GCN.
Revised: We note that only one weight matrix W ( L ) is learned; it is equivalent to the products of all weight matrices in a linear GCN.

Original: Find more related works in § 6.
Revised: Find more related works in § 6.

Original:  T HEORETICAL A NALYSIS
Revised:  T HEORETICAL A NALYSIS

Original: In this section, we formalize the connection between the oversmoothing and heterophily problems in two steps: (1) We show how the movements of node representations relate to the misclassiﬁcation rate of node labels; and (2) we identify the factors that affect the movements of representations.
Revised: In this section, we formalize the connection between the oversmoothing and heterophily problems in two steps: (1) We show how the movements of node representations relate to the misclassiﬁcation rate of node labels; and (2) we identify the factors that affect the movements of representations.

Original: We use the SGC model to analyze binary node classiﬁcation with nodes in class 1 denoted as set V 1 and nodes in class 2 denoted as set V 2 .
Revised: We use the SGC model to analyze binary node classiﬁcation with nodes in class 1 denoted as set V 1 and nodes in class 2 denoted as set V 2 .

Original: We show empirically (§ 5) that the insights obtained in this section are effective for other non-linear models in multi-class classiﬁcation.
Revised: We show empirically (§ 5) that the insights obtained in this section are effective for other non-linear models in multi-class classiﬁcation.

Original: 3. A SSUMPTIONS
Revised: 3. A SSUMPTIONS

Original: Next, we use “i.d.” to represent random variables/vectors that follow the same marginal distribution and their joint probability density function (PDF) is a permutation-invariant function p ( x 1 , . . .
Revised: Next, we use “i.d.” to represent random variables/vectors that follow the same marginal distribution and their joint probability density function (PDF) is a permutation-invariant function p ( x 1 , . . .

Original: , x n ) =
Revised: , x n ) =

Original: p ( P ( x 1 , . . . , x n )) , where P ( · ) means permutation.
Revised: p ( P ( x 1 , . . . , x n )) , where P ( · ) means permutation.

Original: (1) Random graph : Node degrees { d i } are i.d. random variables.
Revised: (1) Random graph : Node degrees { d i } are i.d. random variables.

Original: { ( · ) i } represents a set with i = 1 , . .
Revised: { ( · ) i } represents a set with i = 1 , . .

Original: |V| .
Revised: |V| .

Original: (2) Inputs : (2.1) Node labels { y i } are i.d.
Revised: (2) Inputs : (2.1) Node labels { y i } are i.d.

Original: Bernoulli random variables given by the ratio ρ : ρ ≡ P ( y i =1) P ( y i =2) , ∀ i .
Revised: Bernoulli random variables given by the ratio ρ : ρ ≡ P ( y i =1) P ( y i =2) , ∀ i .

Original: (2.2) Initial input node features { f (0) i } are i.d. random vectors given by (PDF)
Revised: (2.2) Initial input node features { f (0) i } are i.d. random vectors given by (PDF)

Original: f ( x ) , which is expressed as: f ( x ) = (cid:26) f 1 ( x ) ,
Revised: f ( x ) , which is expressed as: f ( x ) = (cid:26) f 1 ( x ) ,

Original: when y i = 1 . f
Revised: when y i = 1 . f

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: when y i = 2 .
Revised: when y i = 2 .

Original: E ( f (0) i | y i )
Revised: E ( f (0) i | y i )

Original: = (cid:26) µ , when y i
Revised: = (cid:26) µ , when y i

Original: = 1 .
Revised: = 1 .

Original: − ρ µ , when y i = 2 .
Revised: − ρ µ , when y i = 2 .

Original: so E ( f (0) i ) = 0 .
Revised: so E ( f (0) i ) = 0 .

Original: (3) Independence : { d i } are independent of { y i } and { f (0) i } .
Revised: (3) Independence : { d i } are independent of { y i } and { f (0) i } .

Original: 3. M ISCLASSIFICATION R ATE & M OVEMENTS OF N ODE R EPRESENTATIONS
Revised: 3. M ISCLASSIFICATION R ATE & M OVEMENTS OF N ODE R EPRESENTATIONS

Original: The following lemmas illustrate why the movements of node representations are a good indicator of SGC’s performance.
Revised: The following lemmas illustrate why the movements of node representations are a good indicator of SGC’s performance.

Original: We view the changes of node representations across the layers as their movements; i.e., f ( l +1)
Revised: We view the changes of node representations across the layers as their movements; i.e., f ( l +1)

Original: i is moved from f ( l ) i , ∀ i .
Revised: i is moved from f ( l ) i , ∀ i .

Original: The misclassiﬁcation rate of an L -layer SGC is determined by { f ( L ) i } , which is the input to the last regression layer, and can be studied through the decision boundary of f ( L ) i .
Revised: The misclassiﬁcation rate of an L -layer SGC is determined by { f ( L ) i } , which is the input to the last regression layer, and can be studied through the decision boundary of f ( L ) i .

Original: Lemma 1 In SGC, the decision boundaries w.r.t.
Revised: Lemma 1 In SGC, the decision boundaries w.r.t.

Original: { f ( L ) i } in multi-class classiﬁcation are linear hyperplanes.
Revised: { f ( L ) i } in multi-class classiﬁcation are linear hyperplanes.

Original: In particular, the decision boundary is a single hyperplane for binary classiﬁcation.
Revised: In particular, the decision boundary is a single hyperplane for binary classiﬁcation.

Original: Proof.
Revised: Proof.

Original: We provide the proof in App. A.1.
Revised: We provide the proof in App. A.1.

Original: The decision boundary of an ( L + 1) -layer SGC can be viewed as a perturbation to the decision boundary of an L -layer SGC due to the movements from { f ( L ) i } to { f ( L +1) i } .
Revised: The decision boundary of an ( L + 1) -layer SGC can be viewed as a perturbation to the decision boundary of an L -layer SGC due to the movements from { f ( L ) i } to { f ( L +1) i } .

Original: Due to Lemma 1, we can use the hyperplane xw + b = 0 to represent the decision boundary of { f ( L ) i } , where w is a unit vector, b is a scalar, and the { f ( L ) i } are classiﬁed to class 

Revised: Due to Lemma 1, we can use the hyperplane xw + b = 0 to represent the decision boundary of { f ( L ) i } , where w is a unit vector, b is a scalar, and the { f ( L ) i } are classiﬁed to class

Original: if f ( L ) i w + b > 0 . Suppose when w = w ∗ and b = b ∗ , we ﬁnd an optimal hyperplane xw ∗ + b ∗ = 

Revised: if f ( L ) i w + b > 0 . Suppose when w = w ∗ and b = b ∗ , we ﬁnd an optimal hyperplane xw ∗ + b ∗ =

Original: that achieves the lowest total misclassiﬁcation rate (§ 2) for { f ( L ) i } .
Revised: that achieves the lowest total misclassiﬁcation rate (§ 2) for { f ( L ) i } .

Original: Due to the movements of node representations, the new optimal hyperplane is decided by: xw ∗(cid:48) + b ∗(cid:48)
Revised: Due to the movements of node representations, the new optimal hyperplane is decided by: xw ∗(cid:48) + b ∗(cid:48)

Original: = 0 .
Revised: = 0 .

Original: In Lemma 2, we show that, under constraints, moving towards the original decision boundary with a non-zero step results in a non-decreasing total misclassiﬁcation rate under the new decision boundary.
Revised: In Lemma 2, we show that, under constraints, moving towards the original decision boundary with a non-zero step results in a non-decreasing total misclassiﬁcation rate under the new decision boundary.

Original: Lemma 2 Moving representations { f ( L ) i } from class 1 by adding − t w ∗ T , t > 0 , s.t.
Revised: Lemma 2 Moving representations { f ( L ) i } from class 1 by adding − t w ∗ T , t > 0 , s.t.

Original: w ∗ T w ∗(cid:48) > 0 , the new total misclassiﬁcation rate is no less than the misclassiﬁcation rate before the movements.
Revised: w ∗ T w ∗(cid:48) > 0 , the new total misclassiﬁcation rate is no less than the misclassiﬁcation rate before the movements.

Original: Proof.
Revised: Proof.

Original: We provide the proof in App.
Revised: We provide the proof in App.

Original: A.2.
Revised: A.2.

Original: Note that the special case where the representations of the two classes swap positions (e.g, bipartite graphs) violates the condition w ∗ T w ∗ (cid:48) > 0 and leads to different conclusions.
Revised: Note that the special case where the representations of the two classes swap positions (e.g, bipartite graphs) violates the condition w ∗ T w ∗ (cid:48) > 0 and leads to different conclusions.

Original: We refer to the condition w ∗ T w ∗(cid:48) > 0 as "non-swapping condition" and throughout the paper, we analyze SGC under that.
Revised: We refer to the condition w ∗ T w ∗(cid:48) > 0 as "non-swapping condition" and throughout the paper, we analyze SGC under that.

Original: Lemma 2 shows that, under the "non-swapping condition", if { f ( l ) i } move towards the original decision boundary (or the other class), SGC tends to perform worse.
Revised: Lemma 2 shows that, under the "non-swapping condition", if { f ( l ) i } move towards the original decision boundary (or the other class), SGC tends to perform worse.

Original: 3. M OVEMENTS OF N ODE R EPRESENTATIONS IN S HALLOW AND D EEPER L AYERS
Revised: 3. M OVEMENTS OF N ODE R EPRESENTATIONS IN S HALLOW AND D EEPER L AYERS

Original: In the following theorems, we will show how relative degree and homophily level will affect the movement of node representations in both shallow and deeper layers.
Revised: In the following theorems, we will show how relative degree and homophily level will affect the movement of node representations in both shallow and deeper layers.

Original: We show results given the degree, d i , of node v i and assume that v i is in class 1.
Revised: We show results given the degree, d i , of node v i and assume that v i is in class 1.

Original: The results in class 2 can be derived similarly.
Revised: The results in class 2 can be derived similarly.

Original: [Equation]
Revised: [Equation]

Original: S HALLOW L AYERS
Revised: S HALLOW L AYERS

Original: Theorem 1.
Revised: Theorem 1.

Original: [Initial Stage] Under the "non-swapping condition", during message passing in shallow layers, the nodes with low homophily
Revised: [Initial Stage] Under the "non-swapping condition", during message passing in shallow layers, the nodes with low homophily

Original: h i ≤ ρ 1+ ρ and the nodes with high homophily
Revised: h i ≤ ρ 1+ ρ and the nodes with high homophily

Original: h i >
Revised: h i >

Original: ρ 1+ ρ but small relative degree are prone to be misclassiﬁed.
Revised: ρ 1+ ρ but small relative degree are prone to be misclassiﬁed.

Original: Deﬁning r ij ≡ (cid:113) d
Revised: Deﬁning r ij ≡ (cid:113) d

Original: i +

Revised: i +

Original: d j +

Revised: d j +

Original: and the relative degree r i ≡ E A
Revised: and the relative degree r i ≡ E A

Original: |
Revised: |

Original: d i ,v i ∈V 1 ( 1 d i (cid:80) j ∈N i r ij ) , the conditional expectation of representation f (1) i is given by:
Revised: d i ,v i ∈V 1 ( 1 d i (cid:80) j ∈N i r ij ) , the conditional expectation of representation f (1) i is given by:

Original: [Equation]
Revised: [Equation]

Original: where the multiplicative factor γ 1 i is:
Revised: where the multiplicative factor γ 1 i is:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We provide the proof in App.
Revised: We provide the proof in App.

Original: A.3.
Revised: A.3.

Original: From Thm.
Revised: From Thm.

Original: 1, we identify three types of movements of node representations, which are characterized by relative degree r i and homophily level h i .
Revised: 1, we identify three types of movements of node representations, which are characterized by relative degree r i and homophily level h i .

Original: For illustration purposes, in Fig. 2, we illustrate the three cases when we apply our theorem to 1D node representations.
Revised: For illustration purposes, in Fig. 2, we illustrate the three cases when we apply our theorem to 1D node representations.

Original: The bars reﬂect the value change of v i ’s node representation.
Revised: The bars reﬂect the value change of v i ’s node representation.

Original: Intuitively, under heterophily (case 1) or under high homophily but low degrees (case 2), the expected representations will move heavily towards the opposing class.
Revised: Intuitively, under heterophily (case 1) or under high homophily but low degrees (case 2), the expected representations will move heavily towards the opposing class.

Original: Otherwise, they will move away from the opposing class (case 3).
Revised: Otherwise, they will move away from the opposing class (case 3).

Original: Among the three cases, only nodes with high homophily and high relative degree beneﬁt from propagation and aggregation.
Revised: Among the three cases, only nodes with high homophily and high relative degree beneﬁt from propagation and aggregation.

Original: Explanation of the heterophily and oversmoothing problems
Revised: Explanation of the heterophily and oversmoothing problems

Original: : • In heterophilous graphs, node representations move heavily towards the opposite class.
Revised: : • In heterophilous graphs, node representations move heavily towards the opposite class.

Original: According to Lemma 2, under the "non-swapping" condition, it indicates higher misclassiﬁcation rate (i.e., poorer performance) for SGC.
Revised: According to Lemma 2, under the "non-swapping" condition, it indicates higher misclassiﬁcation rate (i.e., poorer performance) for SGC.

Original: • In homophilous graphs, high-degree nodes may initially beneﬁt (case 3), leading to performance gain in shallow layers.
Revised: • In homophilous graphs, high-degree nodes may initially beneﬁt (case 3), leading to performance gain in shallow layers.

Original: However, the representations of low-degree nodes tend to move towards the opposing class (case 2) and eventually ﬂip the labels.
Revised: However, the representations of low-degree nodes tend to move towards the opposing class (case 2) and eventually ﬂip the labels.

Original: The misclassiﬁcation of low-degree nodes causes a pseudo-heterophily situation (case 1) later.
Revised: The misclassiﬁcation of low-degree nodes causes a pseudo-heterophily situation (case 1) later.

Original: 3.3. D EVELOPING S TAGE : D EEPER L AYERS
Revised: 3.3. D EVELOPING S TAGE : D EEPER L AYERS

Original: The following theorem analyzes the pseudo-heterophily scenario in deeper layers.
Revised: The following theorem analyzes the pseudo-heterophily scenario in deeper layers.

Original: Based on Thm. 1, message passing will result in scaling the node representations ( γ 1 i ) and some nodes may cross the origin.
Revised: Based on Thm. 1, message passing will result in scaling the node representations ( γ 1 i ) and some nodes may cross the origin.

Original: To account for those, let ξ li be an accumulated discount factor at the l -th layer and
Revised: To account for those, let ξ li be an accumulated discount factor at the l -th layer and

Original: E A , { y i } , { f (0) i }| d i (cid:18) f
Revised: E A , { y i } , { f (0) i }| d i (cid:18) f

Original: ( l ) i
Revised: ( l ) i

Original: d
Revised: d

Original: i +1 |
Revised: i +1 |

Original: d i (cid:19) =
Revised: d i (cid:19) =

Original: ξ li µ .
Revised: ξ li µ .

Original: The nodes contributing positively are deﬁned as a random set: ˆ N si  ≡ { v j | E A , { y i } , { f (0) i }| d i , f ( l ) i ( f ( l ) i |
Revised: The nodes contributing positively are deﬁned as a random set: ˆ N si ( A , ξ li ) ≡ { v j | v [Equation]

Original: d i , f ( l ) i ) · EA , { y i } , { f (0) i }| d i , f ( l ) i ( f ( l ) j T | d i , f ( l ) i ) > 0 , v j ∈ N i } .
Revised: ∈ N i and ( ξ li µ ) · E { y i } , { f [Equation] [Equation]

Original: For ∀ v j ∈ N i , we denote:E A , { y i } , { f (0) i }|
Revised: ξ li [Equation] [Equation] We denote:

Original: d i , f ( l )
Revised: [Equation]

Original: i
Revised: [Equation] [Equation] [Equation]

Original: [Equation]
Revised: [Equation]

Original: and ρ li characterize the neighborhood property of v i .
Revised: and ρ li characterize the neighborhood property of v i .

Original: We further deﬁne the effective homophily of node i as ˆ h li =
Revised: We further deﬁne the effective homophily of node i as ˆ h li =

Original: P ( v j ∈ ˆ N si | d i , f ( l ) i  ) .
Revised: P ( v j ∈ ˆ N si | v j ∈ N i , d i , ξ li ) .

Original: Intiuitively, even if the original graph is not heterophilous, from the perspective of message passing under this pseudo-heterophily scenario, the deeper layers will act as if the graph is more heterophilous than it truly is, which may increase the misclassiﬁcation rate.
Revised: Intiuitively, even if the original graph is not heterophilous, from the perspective of message passing under this pseudo-heterophily scenario, the deeper layers will act as if the graph is more heterophilous than it truly is, which may increase the misclassiﬁcation rate.

Original: Theorem 2.
Revised: Theorem 2.

Original: [Developing Stage] Under the “non-swapping condition”, during message passing in deeper layers, when the effective homophily (cid:98) h li is sufﬁciently low ( (cid:98) h li ≤ ρ li 1+ ρ li ) , the nodes with higher effective relative degree are prone to misclassiﬁcation.
Revised: [Developing Stage] Under the “non-swapping condition”, during message passing in deeper layers, when the effective homophily (cid:98) h li is sufﬁciently low ( (cid:98) h li ≤ ρ li 1+ ρ li ) , the nodes with higher effective relative degree are prone to misclassiﬁcation.

Original: Speciﬁcally, by deﬁning the effective relative degree as ¯ r il ≡ ξ li (cid:48) ξ li , the conditional expectation of f ( l +1)
Revised: Speciﬁcally, by deﬁning the effective relative degree as ¯ r il ≡ ξ li (cid:48) ξ li , the conditional expectation of f ( l +1)

Original: i is:
Revised: i is:

Original: [Equation]
Revised: [Equation]

Revised: i

Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We provide the proof in App.
Revised: We provide the proof in App.

Original: A.4.
Revised: A.4.

Original: Other nodes can be derived similarly.
Revised: Other nodes can be derived similarly.

Original: We note that Eq.
Revised: We note that Eq.

Original: (3) holds for layers l ≥ 2 in general.
Revised: (3) holds for layers l ≥ 2 in general.

Original: We regard layers for which (cid:98) h li > ρ li 1+ ρ li holds for most nodes i as part of the initial stage, as they exhibit three cases similar to Eq.
Revised: We regard layers for which (cid:98) h li > ρ li 1+ ρ li holds for most nodes i as part of the initial stage, as they exhibit three cases similar to Eq.

Original: 3. S IGNED M ESSAGES & M OVEMENTS OF R EPRESENTATIONS
Revised: 3. S IGNED M ESSAGES & M OVEMENTS OF R EPRESENTATIONS

Original: In this section, we provide theory to show that signed messages can help enhance the performance in heterophilous graphs and also alleviate oversmoothing.
Revised: In this section, we provide theory to show that signed messages can help enhance the performance in heterophilous graphs and also alleviate oversmoothing.

Original: Due to limited space, we only show the effect of signed messages in the initial stage; similar results can be derived in the developing stage.
Revised: Due to limited space, we only show the effect of signed messages in the initial stage; similar results can be derived in the developing stage.

Original: Setup.
Revised: Setup.

Original: Signed messages consist of the negated messages sent by neighbors of the opposing class (i.e., after multiplying initial messages by -1), and the unchanged messages ("positive messages") sent by neighbors of the same class.
Revised: Signed messages consist of the negated messages sent by neighbors of the opposing class (i.e., after multiplying initial messages by -1), and the unchanged messages ("positive messages") sent by neighbors of the same class.

Original: In reality, ground-truth labels are not accessible for every node, so we use approximations, which introduces errors.
Revised: In reality, ground-truth labels are not accessible for every node, so we use approximations, which introduces errors.

Original: For node v i , we deﬁne m li as the ratio of neighbors that send incorrect messages at the l -th layer (i.e., different-class neighbors that send non-negated messages and same-class neighbors that send negated messages).
Revised: For node v i , we deﬁne m li as the ratio of neighbors that send incorrect messages at the l -th layer (i.e., different-class neighbors that send non-negated messages and same-class neighbors that send negated messages).

Original: We deﬁne the l -th layer error rate as e li =
Revised: We deﬁne the l -th layer error rate as e li =

Original: E ( m li ) , where the expectation is over the randomness of the neighbors that send incorrect messages.
Revised: E ( m li ) , where the expectation is over the randomness of the neighbors that send incorrect messages.

Original: We make the following assumption: m li is independent of { d i } , { y i } and { f 0 i } .
Revised: We make the following assumption: m li is independent of { d i } , { y i } and { f 0 i } .

Original: Theorem 3.
Revised: Theorem 3.

Original: [Signed Messages]
Revised: [Signed Messages]

Original: With the independence assumptions and under the “non-swapping condition”, by allowing the messages to be optionally multiplied by a negative sign, in the initial stage, the movements of representations will be less affected by the initial homophily level h i , and will be dependent on the error rate e li .
Revised: With the independence assumptions and under the “non-swapping condition”, by allowing the messages to be optionally multiplied by a negative sign, in the initial stage, the movements of representations will be less affected by the initial homophily level h i , and will be dependent on the error rate e li .

Original: Speciﬁcally, the multiplicative factor γ 1 i at the ﬁrst layer is:
Revised: Speciﬁcally, the multiplicative factor γ 1 i at the ﬁrst layer is:

Original: [Equation]
Revised: [Equation]

Original: where γ 1 i ∈
Revised: where γ 1 i ∈

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: otherwise
Revised: if

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is provided in App.
Revised: The proof is provided in App.

Original: A.5.
Revised: A.5.

Original: From Eq.
Revised: From Eq.

Original: 5, we see that when using signed messages, to beneﬁt from case 3 ( γ 1 i > 1 ), the minimum relative degree satisﬁes: r i > 1 (1 − 2 e 0 i )( ρ +(1 − ρ ) h i ) .
Revised: 5, we see that when using signed messages, to beneﬁt from case 3 ( γ 1 i > 1 ), the minimum relative degree satisﬁes: r i > [Equation] [Equation] [Equation]

Revised: 0 i )( ρ +(1 − ρ ) h i ) .

Original: Given h i ≤ 1 , if the error rate is low ( e 0 i (cid:28) 0 . 5 ), we get:
Revised: Given h i ≤ 1 , if the error rate is low ( e 0 i (cid:28) 0 . 5 ), we get:

Original: [Equation]
Revised: [Equation]

Original: ≤ 1 (1+ ρ ) h i − ρ , and 1 (1+ ρ ) h i − ρ is the minimum relative degree required when not using signed messages.
Revised: (1+ ρ ) h i − ρ is the minimum relative degree required when not using signed messages.

Original: This implies that more nodes can beneﬁt from using signed messages.
Revised: This implies that more nodes can beneﬁt from using signed messages.

Original: We note that if low error rate cannot be guaranteed, signed messages may hurt the performance.
Revised: We note that if low error rate cannot be guaranteed, signed messages may hurt the performance.

Original:  M ODEL D ESIGN
Revised:  M ODEL D ESIGN

Original: Based on our theoretical analysis, we propose two new, simple mechanisms to address both the heterophily and oversmoothing problems: signed messages and degree corrections.
Revised: Based on our theoretical analysis, we propose two new, simple mechanisms to address both the heterophily and oversmoothing problems: signed messages and degree corrections.

Original: We integrate these mechanisms, along with a decaying combination of the current and previous node representations (Chen et al., 2020), into a generalized GCN model, GGCN.
Revised: We integrate these mechanisms, along with a decaying combination of the current and previous node representations (Chen et al., 2020), into a generalized GCN model, GGCN.

Original: In § 5, we empirically show its effectiveness in addressing the two closely-related problems.
Revised: In § 5, we empirically show its effectiveness in addressing the two closely-related problems.

Original: 4. S IGNED M ESSAGES
Revised: 4. S IGNED M ESSAGES

Original: Thm 3 points out the importance of signed messages in tackling the heterophily and oversmoothing problems.
Revised: Thm 3 points out the importance of signed messages in tackling the heterophily and oversmoothing problems.

Original: Our ﬁrst proposed mechanism uses cosine similarity to send signed messages.
Revised: Our ﬁrst proposed mechanism uses cosine similarity to send signed messages.

Original: For expressiveness, as in GCN (Kipf & Welling, 2016), we ﬁrst perform a learnable linear transformation of each node’s representation at the l -th layer: (cid:100) F ( l ) = F ( l ) W ( l ) + b ( l ) Then, we deﬁne a sign function to be multiplied with the messages exchanged between neighbors.
Revised: For expressiveness, as in GCN (Kipf & Welling, 2016), we ﬁrst perform a learnable linear transformation of each node’s representation at the l -th layer: (cid:100) F ( l ) = F ( l ) W ( l ) + b ( l ) Then, we deﬁne a sign function to be multiplied with the messages exchanged between neighbors.

Original: To allow for backpropagation of the gradient information, we approximate the sign function with cosine similarity.
Revised: To allow for backpropagation of the gradient information, we approximate the sign function with cosine similarity.

Original: Denote S l as the matrix which stores the sign information about the edges, deﬁned as: S ( l ) [ i, j ] =
Revised: Denote S l as the matrix which stores the sign information about the edges, deﬁned as: S ( l ) [ i, j ] =

Original: Cosine ( f ( l ) i , f ( l ) j ) if ( i (cid:54) =
Revised: Cosine ( f ( l ) i , f ( l ) j ) if ( i (cid:54) =

Original: j ) & ( v j ∈ N i ); 0 otherwise.
Revised: j ) & ( v j ∈ N i ); 0 otherwise.

Original: In order to separate the contribution of similar neighbors (likely in the same class) from that of dissimilar neighbors (unlikely to be in the same class), we split S ( l ) into a positive matrix S ( l ) pos and a negative matrix S ( l ) neg .
Revised: In order to separate the contribution of similar neighbors (likely in the same class) from that of dissimilar neighbors (unlikely to be in the same class), we split S ( l ) into a positive matrix S ( l ) pos and a negative matrix S ( l ) neg .

Original: Thus, our proposed GGCN model learns a weighted combination of the self-representations, the positive messages, and the negative messages:
Revised: Thus, our proposed GGCN model learns a weighted combination of the self-representations, the positive messages, and the negative messages:

Original: [Equation]
Revised: [Equation]

Original: where ˆ β l 0 , ˆ β l 1 and ˆ β l 2 are scalars obtained by applying softmax to the learned scalars β l 0 , β l 1 and β l 2 ; the non-negative scaling factor ˆ α l = softplus ( α l ) is derived from the learned scalar α l ; (cid:12) is element-wise multiplication; and σ is the nonlinear function Elu.
Revised: where ˆ β l 0 , ˆ β l 1 and ˆ β l 2 are scalars obtained by applying softmax to the learned scalars β l 0 , β l 1 and β l 2 ; the non-negative scaling factor ˆ α l = softplus ( α l ) is derived from the learned scalar α l ; (cid:12) is element-wise multiplication; and σ is the nonlinear function Elu.

Original: We note that we learn different α and β parameters per layer for ﬂexibility.
Revised: We note that we learn different α and β parameters per layer for ﬂexibility.

Original: We also require the combined weights, ˆ α l ˆ β lx , to be non-negative so that they do not negate the intended effect of the signed information.
Revised: We also require the combined weights, ˆ α l ˆ β lx , to be non-negative so that they do not negate the intended effect of the signed information.

Original: 4. D EGREE C ORRECTIONS
Revised: 4. D EGREE C ORRECTIONS

Original: Our analysis in § 3.3 and § 3.4 highlights that, when the error rate and homophily level are high, oversmoothing is initially triggered by low-degree nodes.
Revised: Our analysis in § 3.3 and § 3.4 highlights that, when the error rate and homophily level are high, oversmoothing is initially triggered by low-degree nodes.

Original: Thus, our second proposed mechanism aims to compensate for low degrees via degree corrections.
Revised: Thus, our second proposed mechanism aims to compensate for low degrees via degree corrections.

Original: Based on Eq.
Revised: Based on Eq.

Original: (5), and given that most well-trained graph models have a relatively low error rate at the shallow layers (i.e, e li (cid:28) 0 .
Revised: (5), and given that most well-trained graph models have a relatively low error rate at the shallow layers (i.e, e li (cid:28) 0 .

Original: 5 for small l ), we require that the node degrees satisfy r i >(1 − 2 e 0 i )
Revised: 5 for small l ), we require that the node degrees satisfy r i >(1 − 2 e 0 i )

Original: ( π +(1 − π ) h i ) to prevent oversmoothing.
Revised: ( π +(1 − π ) h i ) to prevent oversmoothing.

Original: Since the node degrees cannot be modiﬁed, our strategy is to rescale them.
Revised: Since the node degrees cannot be modiﬁed, our strategy is to rescale them.

Original: Speciﬁcally, we correct the degrees by multiplying with scalars τ lij and change the original formulation as follows:
Revised: Speciﬁcally, we correct the degrees by multiplying with scalars τ lij and change the original formulation as follows:

Original: [Equation]
Revised: [Equation]

Original: This multiplication is equivalent to changing the ratio r ij in Thm.
Revised: This multiplication is equivalent to changing the ratio r ij in Thm.

Original: 1 to
Revised: 1 to

Original: [Equation]
Revised: [Equation]

Original: That is, a larger τ lij increases the effective r i at layer l .
Revised: That is, a larger τ lij increases the effective r i at layer l .

Original: Training independent τ li,j is not practical because it would require O ( |V| 2 ) additional parameters per layer, which can lead to overﬁtting.
Revised: Training independent τ li,j is not practical because it would require O ( |V| 2 ) additional parameters per layer, which can lead to overﬁtting.

Original: Moreover, low-rank parameterizations suffer from unstable training dynamics.
Revised: Moreover, low-rank parameterizations suffer from unstable training dynamics.

Original: Intuitively, when r ij is small, we would like to compensate for it via a larger τ li,j .
Revised: Intuitively, when r ij is small, we would like to compensate for it via a larger τ li,j .

Original: Thus, we set τ li,j to be a function of r ij as follows:
Revised: Thus, we set τ li,j to be a function of r ij as follows:

Original: [Equation]
Revised: [Equation]

Original: where λ l 0 and λ l 1 are learnable parameters.
Revised: where λ l 0 and λ l 1 are learnable parameters.

Original: We subtract 1 so that when r ij = 1 (i.e., d i = d j ), then τ lij = softplus ( λ l 1 ) is a constant bias.
Revised: We subtract 1 so that when r ij = 1 (i.e., d i = d j ), then τ lij = softplus ( λ l 1 ) is a constant bias.

Original: In our proposed GGCN model, we combine the signed messages in Eq.
Revised: In our proposed GGCN model, we combine the signed messages in Eq.

Original: (6) and our degree correction mechanism to obtain the representations at layer l + 1 :
Revised: (6) and our degree correction mechanism to obtain the representations at layer l + 1 :

Original: [Equation]
Revised: [Equation]

Original: where TTT ( l ) is a matrix with elements τ lij .
Revised: where TTT ( l ) is a matrix with elements τ lij .

Original: 4. D ECAYING A GGREGATION
Revised: 4. D ECAYING A GGREGATION

Original: In addition to our two proposed mechanisms that are theoretically grounded in our analysis (§ 3), we also incorporate into GGCN an existing design—decaying aggregation of messages—that empirically increases performance.
Revised: In addition to our two proposed mechanisms that are theoretically grounded in our analysis (§ 3), we also incorporate into GGCN an existing design—decaying aggregation of messages—that empirically increases performance.

Original: However, we note that, even without this design, our GCN architecture still performs well under heterophily and is robust to oversmoothing (App. §B.1).
Revised: However, we note that, even without this design, our GCN architecture still performs well under heterophily and is robust to oversmoothing (App. §B.1).

Original: Decaying aggregation was introduced in (Chen et al., 2020) as a way to slow down the convergence rate of node representations.
Revised: Decaying aggregation was introduced in (Chen et al., 2020) as a way to slow down the convergence rate of node representations.

Original: Inspired by this work, we modify the decaying function, ˆ η , and integrate it to our GGCN model:
Revised: Inspired by this work, we modify the decaying function, ˆ η , and integrate it to our GGCN model:

Original: [Equation]
Revised: [Equation]

Original: In practice, we found that the following decaying function works well: ˆ η
Revised: In practice, we found that the following decaying function works well: ˆ η

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 , otherwise.
Revised: 1 , otherwise.

Original: The hyperparameters k , l 0 , η are tuned on the validation set.
Revised: The hyperparameters k , l 0 , η are tuned on the validation set.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Due to the space limit, in the main paper, we focus on the following three questions: ( Q1 )
Revised: Due to the space limit, in the main paper, we focus on the following three questions: ( Q1 )

Original: Compared to the baselines, how does GGCN perform on homophilous and heterophilous graphs? ( Q2 )
Revised: Compared to the baselines, how does GGCN perform on homophilous and heterophilous graphs? ( Q2 )

Original: How robust is it against oversmoothing under homophily and heterophily?
Revised: How robust is it against oversmoothing under homophily and heterophily?

Original: ( Q3 )
Revised: ( Q3 )

Original: How can we verify the correctness of our theorems on real datasets?
Revised: How can we verify the correctness of our theorems on real datasets?

Original: We provide the ablation study of signed messages and degree correction in App.
Revised: We provide the ablation study of signed messages and degree correction in App.

Original: § B.1 and the study of different normalization strategies in App.
Revised: § B.1 and the study of different normalization strategies in App.

Original: § B.2.
Revised: § B.2.

Original: 5. E XPERIMENTAL
Revised: 5. E XPERIMENTAL

Original: S
Revised: S

Original: ETUP
Revised: ETUP

Original: Datasets.
Revised: Datasets.

Original: We evaluate the performance of our GGCN model and existing GNNs in node classiﬁcation on various real-world datasets (Tang et al., 2009;
Revised: We evaluate the performance of our GGCN model and existing GNNs in node classiﬁcation on various real-world datasets (Tang et al., 2009;

Original: Rozemberczki et al., 2019; Sen et al., 2008; Namata et al., 2012; Bojchevski & Günnemann, 2018; Shchur et al., 2018).
Revised: Rozemberczki et al., 2019; Sen et al., 2008; Namata et al., 2012; Bojchevski & Günnemann, 2018; Shchur et al., 2018).

Original: We provide their summary statistics in Table 1, where we compute the homophily level h of a graph as the average of h i of all nodes v i ∈ V .
Revised: We provide their summary statistics in Table 1, where we compute the homophily level h of a graph as the average of h i of all nodes v i ∈ V .

Original: For all benchmarks, we use the feature vectors, class labels, and 10 random splits (48%/32%/20% of nodes per class for train/validation/test 1 ) from (Pei et al., 2019).
Revised: For all benchmarks, we use the feature vectors, class labels, and 10 random splits (48%/32%/20% of nodes per class for train/validation/test 1 ) from (Pei et al., 2019).

Original: Baselines.
Revised: Baselines.

Original: For baselines we use (1) classic GNN models for node classiﬁcation: vanilla GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´ c et al., 2017) and GraphSage (Hamilton et al., 2017); (2) recent models tackling heterophily: Geom-GCN (Pei et al., 2019), H2GCN (Zhu et al., 2020) and
Revised: For baselines we use (1) classic GNN models for node classiﬁcation: vanilla GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´ c et al., 2017) and GraphSage (Hamilton et al., 2017); (2) recent models tackling heterophily: Geom-GCN (Pei et al., 2019), H2GCN (Zhu et al., 2020) and

Original: GPRGNN (Chien et al., 2021); (3) models tackling oversmoothing: PairNorm (Zhao & Akoglu, 2019) and GCNII (Chen et al., 2020) (state-of-the-art); and (4) 2-layer MLP (with dropout and Elu non-linearity).
Revised: GPRGNN (Chien et al., 2021); (3) models tackling oversmoothing: PairNorm (Zhao & Akoglu, 2019) and GCNII (Chen et al., 2020) (state-of-the-art); and (4) 2-layer MLP (with dropout and Elu non-linearity).

Original: For GCN, PairNorm, Geom-GCN, GCNII, H2GCN and GPRGNN, we use the original codes provided by the authors.
Revised: For GCN, PairNorm, Geom-GCN, GCNII, H2GCN and GPRGNN, we use the original codes provided by the authors.

Original: For GAT, we use the code from a well-accepted Github repository 2 .
Revised: For GAT, we use the code from a well-accepted Github repository 2 .

Original: For GraphSage, we report the results from (Zhu et al., 2020), which uses the same data and splits.
Revised: For GraphSage, we report the results from (Zhu et al., 2020), which uses the same data and splits.

Original: For the baselines that have multiple variants (Geom-GCN, GCNII, H2GCN), we choose the best variant for each dataset and denote them as [model]*.
Revised: For the baselines that have multiple variants (Geom-GCN, GCNII, H2GCN), we choose the best variant for each dataset and denote them as [model]*.

Original: For each dataset and baseline, if applicable, we use the best hyperparameters provided by the authors.
Revised: For each dataset and baseline, if applicable, we use the best hyperparameters provided by the authors.

Original: Otherwise, we perform parameter search to set the best hyperparameters for each baseline.
Revised: Otherwise, we perform parameter search to set the best hyperparameters for each baseline.

Original: The setting and analysis of hyperparameters are in App.
Revised: The setting and analysis of hyperparameters are in App.

Original: § C.
Revised: § C.

Original: Machine.
Revised: Machine.

Original: We ran our experiments on Nvidia V100 GPU.
Revised: We ran our experiments on Nvidia V100 GPU.

Original: [Table]
Revised: [Table]

Original: (Pei et al.,
Revised: (Pei et al.,

Original: [Table]
Revised: [Table]

Original: Q1.
Revised: Q1.

Original: P ERFORMANCE U NDER H OMOPHILY & H ETEROPHILY
Revised: P ERFORMANCE U NDER H OMOPHILY & H ETEROPHILY

Original: Table 1 provides the test accuracy of different GNNs on the supervised node classiﬁcation task over datasets with varying homophily levels (arranged from low homophily to high homophily).
Revised: Table 1 provides the test accuracy of different GNNs on the supervised node classiﬁcation task over datasets with varying homophily levels (arranged from low homophily to high homophily).

Original: A graph’s homophily level is the average of nodes’ homophily levels.
Revised: A graph’s homophily level is the average of nodes’ homophily levels.

Original: We report the best performance of each model across different layers.
Revised: We report the best performance of each model across different layers.

Original: In App.
Revised: In App.

Original: § B.4, we provide the analysis based on the nodes’ homophily levels (§ 3), which is a better metric to predict GCN’s performance and is aligned with our theory.
Revised: § B.4, we provide the analysis based on the nodes’ homophily levels (§ 3), which is a better metric to predict GCN’s performance and is aligned with our theory.

Original: GGCN performs the best in terms of average rank (1.78) across all datasets, which suggests its strong adaptability to graphs of various homophily levels.
Revised: GGCN performs the best in terms of average rank (1.78) across all datasets, which suggests its strong adaptability to graphs of various homophily levels.

Original: In particular, GGCN achieves the highest accuracy in 5 out of 6 heterophilous graphs ( h is low).
Revised: In particular, GGCN achieves the highest accuracy in 5 out of 6 heterophilous graphs ( h is low).

Original: For datasets like Chameleon and Cornell, GGCN enhances accuracy by around 6% and 3% compared to the second-best model.
Revised: For datasets like Chameleon and Cornell, GGCN enhances accuracy by around 6% and 3% compared to the second-best model.

Original: On homophily datasets (Citeseer, Pubmed, Cora), the accuracy of GGCN is within a 1% difference of the best model.
Revised: On homophily datasets (Citeseer, Pubmed, Cora), the accuracy of GGCN is within a 1% difference of the best model.

Original: Our experiments highlight that MLP is a good baseline in heterophilous datasets.
Revised: Our experiments highlight that MLP is a good baseline in heterophilous datasets.

Original: In heterophilous graphs, the models that are not speciﬁcally designed for heterophily usually perform worse than an
Revised: In heterophilous graphs, the models that are not speciﬁcally designed for heterophily usually perform worse than an

Original: MLP.
Revised: MLP.

Original: Though H2GCN* is the second best model in heterophilous datasets, we can still see that in the
Revised: Though H2GCN* is the second best model in heterophilous datasets, we can still see that in the

Original: Actor dataset, MLP performs better.
Revised: Actor dataset, MLP performs better.

Original: GPRGNN and Geom-GCN*, which are speciﬁcally designed for heterophily, achieve better performance than classic GNNs (GCN and GAT) in heterophilous datasets, but do not show clear advantage over MLP.
Revised: GPRGNN and Geom-GCN*, which are speciﬁcally designed for heterophily, achieve better performance than classic GNNs (GCN and GAT) in heterophilous datasets, but do not show clear advantage over MLP.

Original: Our GGCN model is the only model that performs better than MLP across all the datasets.
Revised: Our GGCN model is the only model that performs better than MLP across all the datasets.

Original: In general, GNN models perform well in homophilous datasets.
Revised: In general, GNN models perform well in homophilous datasets.

Original: GCNII* performs the best, and
Revised: GCNII* performs the best, and

Original: GGCN, H2GCN*, GPRGNN and Geom-GCN* also achieve high performance.
Revised: GGCN, H2GCN*, GPRGNN and Geom-GCN* also achieve high performance.

Original: 5. Q2. O VERSMOOTHING
Revised: 5. Q2. O VERSMOOTHING

Original: We also test how robust the models are to oversmoothing.
Revised: We also test how robust the models are to oversmoothing.

Original: To this end, we measure the supervised node classiﬁcation accuracy for 2 to 64 layers.
Revised: To this end, we measure the supervised node classiﬁcation accuracy for 2 to 64 layers.

Original: Table 2 presents the results for two homophilous datasets (top) and two heterophilous datasets (bottom).
Revised: Table 2 presents the results for two homophilous datasets (top) and two heterophilous datasets (bottom).

Original: Per model, we also report the layer at which the best performance is achieved (column ‘Best’).
Revised: Per model, we also report the layer at which the best performance is achieved (column ‘Best’).

Original: We provides Table 2 in larger font in App.
Revised: We provides Table 2 in larger font in App.

Original: § D.
Revised: § D.

Original: According to Table 2, GGCN and GCNII* achieve increase in accuracy when stacking more layers in four datasets, while GPRGNN and PairNorm exhibit robustness against oversmoothing.
Revised: According to Table 2, GGCN and GCNII* achieve increase in accuracy when stacking more layers in four datasets, while GPRGNN and PairNorm exhibit robustness against oversmoothing.

Original: Models that are not designed for oversmoothing have various issues.
Revised: Models that are not designed for oversmoothing have various issues.

Original: The performance of GCN and Geom-GCN* drops rapidly as the number of layers grows; H2GCN* requires concatenating all the intermediate outputs and quickly reaches memory capacity; GAT’s attention mechanism also has high memory requirements.
Revised: The performance of GCN and Geom-GCN* drops rapidly as the number of layers grows; H2GCN* requires concatenating all the intermediate outputs and quickly reaches memory capacity; GAT’s attention mechanism also has high memory requirements.

Original: We also ﬁnd that GAT needs careful initialization when stacking many layers as it may suffer from numerical instability in sparse tensor operations.
Revised: We also ﬁnd that GAT needs careful initialization when stacking many layers as it may suffer from numerical instability in sparse tensor operations.

Original: In general, models like GGCN, GCNII*, and GPRGNN that perform well under heterophily usually exhibit higher resilience against oversmoothing.
Revised: In general, models like GGCN, GCNII*, and GPRGNN that perform well under heterophily usually exhibit higher resilience against oversmoothing.

Original: One exception is Geom-GCN*, which suffers more than GCN.
Revised: One exception is Geom-GCN*, which suffers more than GCN.

Original: This model incorporates structurally similar nodes into each node’s neighborhood; this design may beneﬁt Geom-GCN* in the shallow layers as the node degrees increase.
Revised: This model incorporates structurally similar nodes into each node’s neighborhood; this design may beneﬁt Geom-GCN* in the shallow layers as the node degrees increase.

Original: However, as we point out in Thm. 2, when the effective homophily is low, higher degrees are harmful.
Revised: However, as we point out in Thm. 2, when the effective homophily is low, higher degrees are harmful.

Original: If the structurally similar nodes introduce lower homophily levels, their performance will rapidly degrade once the effective homophily is lower than ρ li 1+ ρ li .
Revised: If the structurally similar nodes introduce lower homophily levels, their performance will rapidly degrade once the effective homophily is lower than ρ li 1+ ρ li .

Original: On the other hand, GGCN virtually changes the degrees thanks to the degree correction mechanism (§ 4.2), and, in practice, this design has positive impact on its robustness to oversmoothing.
Revised: On the other hand, GGCN virtually changes the degrees thanks to the degree correction mechanism (§ 4.2), and, in practice, this design has positive impact on its robustness to oversmoothing.

Original: 5. Q3. E MPIRICAL V ERIFICATION OF THE I NITIAL & D EVELOPING S TAGES
Revised: 5. Q3. E MPIRICAL V ERIFICATION OF THE I NITIAL & D EVELOPING S TAGES

Original: Using the vanilla GCN model (Kipf & Welling, 2016), we validate our theorems by measuring the test accuracy and effective homophily for different node degrees (binned logarithmically) on real datasets.
Revised: Using the vanilla GCN model (Kipf & Welling, 2016), we validate our theorems by measuring the test accuracy and effective homophily for different node degrees (binned logarithmically) on real datasets.

Original: We estimate the effective homophily as the portion of the same-class neighbors that are correctly classiﬁed before the last propagation.
Revised: We estimate the effective homophily as the portion of the same-class neighbors that are correctly classiﬁed before the last propagation.

Original: Figure 1 shows the results for Citeseer.
Revised: Figure 1 shows the results for Citeseer.

Original: In the initial stage ( (cid:98) h li is high), the accuracy increases with the degree, but in the developing stage, the trend changes, with high-degree nodes being impacted the most, as predicted by our theorems.
Revised: In the initial stage ( (cid:98) h li is high), the accuracy increases with the degree, but in the developing stage, the trend changes, with high-degree nodes being impacted the most, as predicted by our theorems.

Original: In App.
Revised: In App.

Original: § B.3, we provide more details for this analysis and we further verify our conclusion on Cora.
Revised: § B.3, we provide more details for this analysis and we further verify our conclusion on Cora.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Graph Convolutional Neural Networks.
Revised: Graph Convolutional Neural Networks.

Original: Early on, (Defferrard et al., 2016) proposed a GCN model that combines spectral ﬁltering of graph signals and non-linearity for supervised node classiﬁcation.
Revised: Early on, (Defferrard et al., 2016) proposed a GCN model that combines spectral ﬁltering of graph signals and non-linearity for supervised node classiﬁcation.

Original: The scalability and numerical stability of GCNs was later improved with a localized ﬁrst-order approximation of spectral graph convolutions proposed in (Kipf & Welling, 2016).
Revised: The scalability and numerical stability of GCNs was later improved with a localized ﬁrst-order approximation of spectral graph convolutions proposed in (Kipf & Welling, 2016).

Original: (Veliˇckovi´c et al., 2017) proposes the ﬁrst graph attention network to improve neighborhood aggregation.
Revised: (Veliˇckovi´c et al., 2017) proposes the ﬁrst graph attention network to improve neighborhood aggregation.

Original: Many more
Revised: Many more

Original: GCN variants have been proposed for different applications such as: computer vision (Satorras &
Revised: GCN variants have been proposed for different applications such as: computer vision (Satorras &

Original: Estrach, 2018), social science (Li & Goldwasser, 2019), biology (Yan et al., 2019), algorithmic tasks (Veliˇckovi´c et al., 2020; Yan et al., 2020), and inductive classiﬁcation (Hamilton et al., 2017).
Revised: Estrach, 2018), social science (Li & Goldwasser, 2019), biology (Yan et al., 2019), algorithmic tasks (Veliˇckovi´c et al., 2020; Yan et al., 2020), and inductive classiﬁcation (Hamilton et al., 2017).

Original: Concurrent work (Baranwal et al., 2021) provides theoretic analysis on the linear separability of graph convolution but does not provide effective strategies to increase the separability.
Revised: Concurrent work (Baranwal et al., 2021) provides theoretic analysis on the linear separability of graph convolution but does not provide effective strategies to increase the separability.

Original: Oversmoothing.
Revised: Oversmoothing.

Original: The oversmoothing problem was ﬁrst discussed in (Li et al., 2018), which proved that by repeatedly applying Laplacian smoothing, the representations of nodes within each connected component of the graph converge to the same value.
Revised: The oversmoothing problem was ﬁrst discussed in (Li et al., 2018), which proved that by repeatedly applying Laplacian smoothing, the representations of nodes within each connected component of the graph converge to the same value.

Original: Since then, various empirical solutions have been proposed: residual connections and dilated convolutions (Li et al., 2019); skip links (Xu et al., 2018); new normalization strategies (Zhao & Akoglu, 2019); edge dropout (Rong et al., 2019); and a new model that even increases performance as more layers are stacked (Chen et al., 2020).
Revised: Since then, various empirical solutions have been proposed: residual connections and dilated convolutions (Li et al., 2019); skip links (Xu et al., 2018); new normalization strategies (Zhao & Akoglu, 2019); edge dropout (Rong et al., 2019); and a new model that even increases performance as more layers are stacked (Chen et al., 2020).

Original: Some recent works provide theoretical analyses: (Oono & Suzuki, 2019) showed that a k -layer renormalized graph convolution with a residual link simulates a lazy random walk and (Chen et al., 2020) proved that the convergence rate is related to the spectral gap of the graph.
Revised: Some recent works provide theoretical analyses: (Oono & Suzuki, 2019) showed that a k -layer renormalized graph convolution with a residual link simulates a lazy random walk and (Chen et al., 2020) proved that the convergence rate is related to the spectral gap of the graph.

Original: Heterophily & GCNs.
Revised: Heterophily & GCNs.

Original: Heterophily has recently been recognized as an important issue for GCNs.
Revised: Heterophily has recently been recognized as an important issue for GCNs.

Original: It is ﬁrst outlined in the context of GCNs in (Pei et al., 2019).
Revised: It is ﬁrst outlined in the context of GCNs in (Pei et al., 2019).

Original: (Zhu et al., 2020) identiﬁed a set of effective designs that allow GCNs to generalize to challenging heterophilous settings, and (Zhu et al., 2021) introduced a new GCN model that leverages ideas from belief propagation (Gatterbauer et al., 2015).
Revised: (Zhu et al., 2020) identiﬁed a set of effective designs that allow GCNs to generalize to challenging heterophilous settings, and (Zhu et al., 2021) introduced a new GCN model that leverages ideas from belief propagation (Gatterbauer et al., 2015).

Original: Though recent work (Chen et al., 2020) focused on solving the oversmoothing problem, it also empirically showed improvement on heterophilous datasets; these empirical observations formed the basis of our work.
Revised: Though recent work (Chen et al., 2020) focused on solving the oversmoothing problem, it also empirically showed improvement on heterophilous datasets; these empirical observations formed the basis of our work.

Original: Finally, (Chien et al., 2021) recently proposed a PageRank-based model that performs well under heterophily and alleviates the oversmoothing problem.
Revised: Finally, (Chien et al., 2021) recently proposed a PageRank-based model that performs well under heterophily and alleviates the oversmoothing problem.

Original: However, they view the two problems independently and analyze their model via an asymptotic spectral perspective.
Revised: However, they view the two problems independently and analyze their model via an asymptotic spectral perspective.

Original: Our work studies the representation dynamics and unveils the connections between the oversmoothing and heterophily problems theoretically and empirically.
Revised: Our work studies the representation dynamics and unveils the connections between the oversmoothing and heterophily problems theoretically and empirically.

Original: As we demonstrate with GGCN, addressing both issues in a principled manner provides superior performance across a variety of datasets.
Revised: As we demonstrate with GGCN, addressing both issues in a principled manner provides superior performance across a variety of datasets.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work provides the ﬁrst theoretical and empirical analysis that unveils the connections between the oversmoothing and heterophily problems.
Revised: Our work provides the ﬁrst theoretical and empirical analysis that unveils the connections between the oversmoothing and heterophily problems.

Original: By analyzing the statistical change of the node representations after the graph convolution, we identiﬁed two causes, i.e., the relative degree of a node compared to its neighbors and the level of heterophily in its neighborhood, which inﬂuence the movements of node representations and lead to a higher misclassiﬁcation rate.
Revised: By analyzing the statistical change of the node representations after the graph convolution, we identiﬁed two causes, i.e., the relative degree of a node compared to its neighbors and the level of heterophily in its neighborhood, which inﬂuence the movements of node representations and lead to a higher misclassiﬁcation rate.

Original: Based on our new, uniﬁed theoretical perspective, we obtained three important insights: (1) Nodes with high heterophily tend to be misclassiﬁed after graph convolution; (2) Even with low heterophily, low-degree nodes can trigger a pseudo-heterophily situation that explains oversmoothing.
Revised: Based on our new, uniﬁed theoretical perspective, we obtained three important insights: (1) Nodes with high heterophily tend to be misclassiﬁed after graph convolution; (2) Even with low heterophily, low-degree nodes can trigger a pseudo-heterophily situation that explains oversmoothing.

Original: (3) Allowing signed messages (instead of only positive messages) helps alleviate the heterophily and oversmoothing problems.
Revised: (3) Allowing signed messages (instead of only positive messages) helps alleviate the heterophily and oversmoothing problems.

Original: Based on these insights, we designed a generalized model, GGCN, that addresses the identiﬁed causes using signed messages and degree corrections.
Revised: Based on these insights, we designed a generalized model, GGCN, that addresses the identiﬁed causes using signed messages and degree corrections.

Original: Though other designs may also address these two problems, our work points out two effective directions that are theoretically grounded (§ 3).
Revised: Though other designs may also address these two problems, our work points out two effective directions that are theoretically grounded (§ 3).

Original: In summary, our research suggests it is beneﬁcial to study the oversmoothing and heterophily problems jointly; this leads to architectural insights that can improve the learned representations of graph neural network models across a variety of domains.
Revised: In summary, our research suggests it is beneﬁcial to study the oversmoothing and heterophily problems jointly; this leads to architectural insights that can improve the learned representations of graph neural network models across a variety of domains.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: To reproduce our experimental results we provide the references to the datasets and baselines in § 5.
Revised: To reproduce our experimental results we provide the references to the datasets and baselines in § 5.

Original: For the baselines, we use code provided by the authors and set the hyperparameters as the authors suggest.
Revised: For the baselines, we use code provided by the authors and set the hyperparameters as the authors suggest.

Original: In App.
Revised: In App.

Original: § C, we provide detailed hyperparameter setting for GGCN.
Revised: § C, we provide detailed hyperparameter setting for GGCN.

Original: We also provide our code in the supplementary material and we will share the link to our code upon acceptance
Revised: We also provide our code in the supplementary material and we will share the link to our code upon acceptance

Original: A D ETAILED P ROOFS OF T HEOREMS IN §A. P ROOF OF L EMMA
Revised: A D ETAILED P ROOFS OF T HEOREMS IN §A. P ROOF OF L EMMA

Original: Proof.
Revised: Proof.

Original: The loss function for SGC is: CrossEntropy ( Softmax ( F ( L ) W ( L ) + b ( L ) ), y i ).
Revised: The loss function for SGC is: CrossEntropy ( Softmax ( F ( L ) W ( L ) + b ( L ) ), y i ).

Original: Rewrite
Revised: Rewrite

Original: W ( L ) into an array of column vectors: W ( L ) =
Revised: W ( L ) into an array of column vectors: W ( L ) =

Original: [ w ( L ) 0 , w ( L ) 1 , . . .
Revised: [ w ( L ) 0 , w ( L ) 1 , . . .

Original: w ( L ) |L| ] , b ( L ) into an array of scalars: b ( L ) =
Revised: w ( L ) |L| ] , b ( L ) into an array of scalars: b ( L ) =

Original: [ b ( L ) 0 , b ( L ) 1 , . . .
Revised: [ b ( L ) 0 , b ( L ) 1 , . . .

Original: b ( L ) |L| ] , and F ( L ) into an array of row vectors: F ( L ) =
Revised: b ( L ) |L| ] , and F ( L ) into an array of row vectors: F ( L ) =

Original: [ f ( L ) 0 ; f ( L ) 1 ; . . .
Revised: [ f ( L ) 0 ; f ( L ) 1 ; . . .

Original: f ( L ) |V| ] ( [ · ; · ] means stacking vertically).
Revised: f ( L ) |V| ] ( [ · ; · ] means stacking vertically).

Original: Let p i,c ≡
Revised: Let p i,c ≡

Original: [Equation]
Revised: [Equation]

Original: and let y i,c ≡ (cid:26) 1 , iff y i
Revised: and let y i,c ≡ (cid:26) 1 , iff y i

Original: =
Revised: =

Original: c 0 , otherwise then we rewrite the loss function as:
Revised: c 0 , otherwise then we rewrite the loss function as:

Original: [Equation]
Revised: [Equation]

Original: (11) n is the number of nodes used for training.
Revised: (11) n is the number of nodes used for training.

Original: To predict that the node v i belongs to class c , we require: ∀ c
Revised: To predict that the node v i belongs to class c , we require: ∀ c

Original: [Equation]
Revised: [Equation]

Original: (cid:54) =
Revised: (cid:54) =

Original: c , p i,c > p i,c (cid:48) .
Revised: c , p i,c > p i,c (cid:48) .

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: It is equivalent to:
Revised: It is equivalent to:

Original: [Equation]
Revised: [Equation]

Original: It shows that the decision region of class c is decided by a set of hyper-planes:
Revised: It shows that the decision region of class c is decided by a set of hyper-planes:

Original: [Equation]
Revised: [Equation]

Original: x is the vector variable that describes the plane.
Revised: x is the vector variable that describes the plane.

Original: Any vectors that satisfy the equation are on this plane.
Revised: Any vectors that satisfy the equation are on this plane.

Original: We note that for binary classiﬁcation, there is one hyper-plane.
Revised: We note that for binary classiﬁcation, there is one hyper-plane.

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: Before giving the proof, we ﬁrst explain the meaning of the Lemma.
Revised: Before giving the proof, we ﬁrst explain the meaning of the Lemma.

Original: As we point out in § 3, the misclassiﬁcation rate of an L -layer SGC can be studied through its last logistic-regression layer, the input to which is { f ( L ) i } .
Revised: As we point out in § 3, the misclassiﬁcation rate of an L -layer SGC can be studied through its last logistic-regression layer, the input to which is { f ( L ) i } .

Original: To study the misclassiﬁcation rate of an ( L + 1) -layer SGC, we can view the input to the last layer { f ( L +1) i } as being moved from { f ( L ) i } .
Revised: To study the misclassiﬁcation rate of an ( L + 1) -layer SGC, we can view the input to the last layer { f ( L +1) i } as being moved from { f ( L ) i } .

Original: The misclassiﬁcation rate is closely related to the decision boundary and will be our tool for studying the change of the misclassiﬁcation rate.
Revised: The misclassiﬁcation rate is closely related to the decision boundary and will be our tool for studying the change of the misclassiﬁcation rate.

Original: This Lemma studies the movements that bring the node representations closer to the original decision boundary; we will prove that moving towards the original decision boundary by a non-zero step is not beneﬁcial (i.e., harmful) to the SGC’s performance.
Revised: This Lemma studies the movements that bring the node representations closer to the original decision boundary; we will prove that moving towards the original decision boundary by a non-zero step is not beneﬁcial (i.e., harmful) to the SGC’s performance.

Original: Proof.
Revised: Proof.

Original: We prove it by contradiction and suppose that after the movements, the total misclassiﬁcation rate is lowered.
Revised: We prove it by contradiction and suppose that after the movements, the total misclassiﬁcation rate is lowered.

Original: We denote the conditional distribution of { f ( L ) i } as f L 1 ( x ) conditioned on that they are from class 1 and f L 2 ( x ) conditioned on that they are from class 1 or class 2, respectively.
Revised: We denote the conditional distribution of { f ( L ) i } as f L 1 ( x ) conditioned on that they are from class 1 and f L 2 ( x ) conditioned on that they are from class 1 or class 2, respectively.

Original: To note that, the distributions of different layers are different.
Revised: To note that, the distributions of different layers are different.

Original: After the movements, the representations from class 1 become: { f ( L )
Revised: After the movements, the representations from class 1 become: { f ( L )

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: t w ∗ T represents moving towards the original decision boundary along the norm direction by a non-zero step.
Revised: t w ∗ T represents moving towards the original decision boundary along the norm direction by a non-zero step.

Original: This causes a corresponding change in the conditional PDF given that they are from class 1: f L 1 ( x + t w ∗ T ) .
Revised: This causes a corresponding change in the conditional PDF given that they are from class 1: f L 1 ( x + t w ∗ T ) .

Original: Recall that in § 3, the parameters for the original optimal hyperplane w ∗ and b ∗ , and the parameters for the later optimal hyperplane w ∗(cid:48) and b ∗(cid:48) are normalized such that when xw ∗ + b ∗ > 0 and xw ∗(cid:48) + b ∗(cid:48) > 0 , we predict class 1.
Revised: Recall that in § 3, the parameters for the original optimal hyperplane w ∗ and b ∗ , and the parameters for the later optimal hyperplane w ∗(cid:48) and b ∗(cid:48) are normalized such that when xw ∗ + b ∗ > 0 and xw ∗(cid:48) + b ∗(cid:48) > 0 , we predict class 1.

Original: The new misclassiﬁcation rate conditioned on class 1 is:
Revised: The new misclassiﬁcation rate conditioned on class 1 is:

Original: [Equation]
Revised: [Equation]

Original: The new misclassiﬁcation rate conditioned on class 2 is:
Revised: The new misclassiﬁcation rate conditioned on class 2 is:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The new total misclassiﬁcation rate is:
Revised: The new total misclassiﬁcation rate is:

Original: [Equation]
Revised: [Equation]

Original: Next, we will prove that if the total misclassiﬁcation is lowered, xw ∗ + b ∗ = 0 is not the optimal hyper-plane before the movements which should achieve the lowest total misclassiﬁcation rate.
Revised: Next, we will prove that if the total misclassiﬁcation is lowered, xw ∗ + b ∗ = 0 is not the optimal hyper-plane before the movements which should achieve the lowest total misclassiﬁcation rate.

Original: Consider a hyper-plane xw ∗(cid:48)
Revised: Consider a hyper-plane xw ∗(cid:48)

Original: + b ∗(cid:48) − t w ∗ T w ∗(cid:48)= 0 .
Revised: + b ∗(cid:48) − t w ∗ T w ∗(cid:48)= 0 .

Original: Given that w ∗ T w ∗(cid:48) > 0 and t > 0 , for ∀ x , s.t.
Revised: Given that w ∗ T w ∗(cid:48) > 0 and t > 0 , for ∀ x , s.t.

Original: [Equation]
Revised: [Equation]

Original: < 0 , we have:
Revised: < 0 , we have:

Original: [Equation]
Revised: [Equation]

Original: t w
Revised: t w

Original: [Equation]
Revised: [Equation]

Original: It means:
Revised: It means:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Because f L 1 ( x ) is a PDF which is a nonnegative function,
Revised: Because f L 1 ( x ) is a PDF which is a nonnegative function,

Original: [Equation]
Revised: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we can obtain
Revised: we can obtain

Original: [Equation]
Revised: [Equation]

Original: Thus,
Revised: Thus,

Original: [Equation]
Revised: [Equation]

Original: Deﬁne
Revised: Deﬁne

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The quantity M f represents the total misclassiﬁcation rate before the movements if the decision boundary is xw ∗(cid:48)
Revised: The quantity M f represents the total misclassiﬁcation rate before the movements if the decision boundary is xw ∗(cid:48)

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: = 0 .
Revised: = 0 .

Original: Given Eq.
Revised: Given Eq.

Original: 17, 19 and 21, we have:
Revised: 17, 19 and 21, we have:

Original: [Equation]
Revised: [Equation]

Original: Let M denotes the total classiﬁcation rate before the movements.
Revised: Let M denotes the total classiﬁcation rate before the movements.

Original: Based on the assumption that the total misclassiﬁcation rate is lowered after the movements ( M > M (cid:48) ), we have:
Revised: Based on the assumption that the total misclassiﬁcation rate is lowered after the movements ( M > M (cid:48) ), we have:

Original: [Equation]
Revised: [Equation]

Original: Eq.
Revised: Eq.

Original: 24 indicates that we ﬁnd a hyper-plane xw
Revised: 24 indicates that we ﬁnd a hyper-plane xw

Original: [Equation]
Revised: [Equation]

Original: = 0 , which yields smaller total misclassiﬁcation rate than xw ∗ + b ∗
Revised: = 0 , which yields smaller total misclassiﬁcation rate than xw ∗ + b ∗

Original: =
Revised: =

Original: before the movements.
Revised: before the movements.

Original: This contradicts to the fact that xw ∗ + b ∗ = 0 is the optimal hyper-plane before the movements.
Revised: This contradicts to the fact that xw ∗ + b ∗ = 0 is the optimal hyper-plane before the movements.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: The ﬁrst layer node representations are given by:
Revised: The ﬁrst layer node representations are given by:

Original: [Equation]
Revised: [Equation]

Original: Without loss of generality, we assume node v i is in the ﬁrst class V 1 .
Revised: Without loss of generality, we assume node v i is in the ﬁrst class V 1 .

Original: Then, we can express the conditional expectation of f (1) i as:
Revised: Then, we can express the conditional expectation of f (1) i as:

Original: [Equation]
Revised: [Equation]

Original: Recall that A is the graph adjacency matrix; the ﬁrst expectation is taken over the randomness of ground truth labels and initial input features
Revised: Recall that A is the graph adjacency matrix; the ﬁrst expectation is taken over the randomness of ground truth labels and initial input features

Original: [Equation]
Revised: [Equation]

Original: and the second expectation is taken over the randomness of graph structure ( A ) given the degree of node v i and its label.
Revised: and the second expectation is taken over the randomness of graph structure ( A ) given the degree of node v i and its label.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: Let 1 {·} represent the indicator function: it equals 1 if and only if the event in the bracket holds.
Revised: Let 1 {·} represent the indicator function: it equals 1 if and only if the event in the bracket holds.

Original: [Equation]
Revised: [Equation]

Original: Due to our assumptions in § 3, though the labels of nodes are dependent, they share the same marginal distribution and their joint distribution is a permutation-invariant function.
Revised: Due to our assumptions in § 3, though the labels of nodes are dependent, they share the same marginal distribution and their joint distribution is a permutation-invariant function.

Original: Thus, given v i ∈ V 1 ( y i = 1) , the conditional joint distribution of { y t | t (cid:54) = i } is also permutation invariant— i.e., ∀ j, j (cid:48) ∈ N i , P ( v j ∈ V 1 | v i ∈ V 1 ) =
Revised: Thus, given v i ∈ V 1 ( y i = 1) , the conditional joint distribution of { y t | t (cid:54) = i } is also permutation invariant— i.e., ∀ j, j (cid:48) ∈ N i , P ( v j ∈ V 1 | v i ∈ V 1 ) =

Original: P ( v (cid:48)
Revised: P ( v (cid:48)

Original: j ∈ V 1 | v i ∈ V 1 ) .
Revised: j ∈ V 1 | v i ∈ V 1 ) .

Original: Then for ∀ j ∈ N i , we have: d i P ( v j ∈ V 1 | v i ∈ V 1 ) =
Revised: Then for ∀ j ∈ N i , we have: d i P ( v j ∈ V 1 | v i ∈ V 1 ) =

Original: d i k i and P ( v j ∈ V 1 | v i ∈ V 1 )
Revised: d i k i and P ( v j ∈ V 1 | v i ∈ V 1 )

Original: =
Revised: =

Original: k i .
Revised: k i .

Original: Due to the independence of A and { y i } , P ( v j ∈ V 1 | A , v i ∈ V 1 ) =
Revised: Due to the independence of A and { y i } , P ( v j ∈ V 1 | A , v i ∈ V 1 ) =

Original: k i .
Revised: k i .

Original: [Equation]
Revised: [Equation]

Original: f (1)
Revised: f (1)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: v i
Revised: v i

Original: [Equation]
Revised: [Equation]

Original: µ
Revised: µ

Original: [Equation]
Revised: [Equation]

Original: Combining Equation (26) and Equation (29) and given the independence of r ij and k i , we have:
Revised: Combining Equation (26) and Equation (29) and given the independence of r ij and k i , we have:

Original: E A , { y i } , { f (0) i }|
Revised: E A , { y i } , { f (0) i }|

Original: d
Revised: d

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: v i ∈V
Revised: v i ∈V

Original: [Equation]
Revised: [Equation]

Original: f (
Revised: f (

Original: [Equation]
Revised: [Equation]

Original: i | v i
Revised: i | v i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: d
Revised: d

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Deﬁne (cid:15) ≡ (1 + ρ ) h i − ρ , and we consider three cases: (1) h i ≤ ρ 1+ ρ , (2) h i > ρ 1+ ρ & r i ≤ 1 (cid:15) , and (3) h i > ρ 1+ ρ & r i > 1 (cid:15) .
Revised: Deﬁne (cid:15) ≡ (1 + ρ ) h i − ρ , and we consider three cases: (1) h i ≤ ρ 1+ ρ , (2) h i > ρ 1+ ρ & r i ≤ 1 (cid:15) , and (3) h i > ρ 1+ ρ & r i > 1 (cid:15) .

Original: • CASE 1: h i ≤ ρ 1+ ρ h i ≤ ρ 1+ ρ
Revised: • CASE 1: h i ≤ ρ 1+ ρ h i ≤ ρ 1+ ρ

Original: [Equation]
Revised: [Equation]

Original: ρ • Upper Bound
Revised: ρ • Upper Bound

Original: We have:
Revised: We have:

Original: [Equation]
Revised: [Equation]

Original: • Lower Bound
Revised: • Lower Bound

Original: To see if there exists a lower bound, we ﬁrst show that when h i ≤ ρ 1+ ρ , 1+((1+ ρ ) h i − ρ )
Revised: To see if there exists a lower bound, we ﬁrst show that when h i ≤ ρ 1+ ρ , 1+((1+ ρ ) h i − ρ )

Original: d i r i
Revised: d i r i

Original: d
Revised: d

Original: i + is a decreasing function of d i given that d i ≥ 1 .
Revised: i + is a decreasing function of d i given that d i ≥ 1 .

Original: When h i ≤ ρ 1+ ρ , we have:
Revised: When h i ≤ ρ 1+ ρ , we have:

Original: ((1 + ρ ) h i − ρ ) ≤2.
Revised: ((1 + ρ ) h i − ρ ) ≤2.

Original: d i d i +1 is an increasing non-negative function of d i 3.
Revised: d i d i +1 is an increasing non-negative function of d i 3.

Original: r i is an increasing non-negative function of d i 4.
Revised: r i is an increasing non-negative function of d i 4.

Original: 1 d i +1 is a decreasing function of d i
Revised: 1 d i +1 is a decreasing function of d i

Original: Thus 1+((1+ ρ ) h i − ρ )
Revised: Thus 1+((1+ ρ ) h i − ρ )

Original: d i r i
Revised: d i r i

Original: d
Revised: d

Original: i + is a decreasing function of d i
Revised: i + is a decreasing function of d i

Original: When h i = ρ 1+ ρ , 1+((

Revised: When h i = ρ 1+ ρ , 1+((

Original: + ρ ) h
Revised: + ρ ) h

Original: i − ρ )
Revised: i − ρ )

Original: d i r i
Revised: d i r i

Original: d i + = 1 d
Revised: d i + = 1 d

Original: i +1 and 0 < 1 d i +1 ≤ 12 .
Revised: i +1 and 0 < 1 d i +1 ≤ 12 .

Original: When h i < ρ 1+ ρ ,
Revised: When h i < ρ 1+ ρ ,

Original: [Equation]
Revised: [Equation]

Original: And we know that when h i < ρ 

Revised: And we know that when h i < ρ

Original: + ρ :
Revised: + ρ :

Original: [Equation]
Revised: [Equation]

Original: Thus lim r
Revised: Thus lim r

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: • CASE 2: h i > ρ 1+ ρ & r i ≤ 1 (cid:15) h i > ρ 1+ ρ & r i ≤ 1 (cid:15) h i > ρ 1+ ρ & r i ≤ 1 (cid:15)
Revised: • CASE 2: h i > ρ 1+ ρ & r i ≤ 1 (cid:15) h i > ρ 1+ ρ & r i ≤ 1 (cid:15) h i > ρ 1+ ρ & r i ≤ 1 (cid:15)

Original: If h i > ρ 1+ ρ , 0 < (cid:15) ≤ 1 ; if r i ≤ 1 (cid:15) , 0 < (cid:15)r i ≤ 1 .
Revised: If h i > ρ 1+ ρ , 0 < (cid:15) ≤ 1 ; if r i ≤ 1 (cid:15) , 0 < (cid:15)r i ≤ 1 .

Original: Given
Revised: Given

Original: [Equation]
Revised: [Equation]

Original: we have 0 < 1 d i +1 < γ 1 i ≤ 1 .
Revised: we have 0 < 1 d i +1 < γ 1 i ≤ 1 .

Original: • CASE 3: h i > ρ 1+ ρ & r i > 1 (cid:15) h
Revised: • CASE 3: h i > ρ 1+ ρ & r i > 1 (cid:15) h

Original: i > ρ 1+ ρ & r i > 1 (cid:15) h i >
Revised: i > ρ 1+ ρ & r i > 1 (cid:15) h i >

Original: ρ 1+ ρ & r i > 1 (cid:15)
Revised: ρ 1+ ρ & r i > 1 (cid:15)

Original: In this case, Equation (35) still holds, since h i >
Revised: In this case, Equation (35) still holds, since h i >

Original: ρ 

Revised: ρ

Original: + ρ .
Revised: + ρ .

Original: • Lower Bound
Revised: • Lower Bound

Original: If r i > 1 (cid:15) , then (cid:15) r i > 1 and therefore γ 1 i > 1 .
Revised: If r i > 1 (cid:15) , then (cid:15) r i > 1 and therefore γ 1 i > 1 .

Original: • Upper Bound
Revised: • Upper Bound

Original: When (cid:15) > 0 ,
Revised: When (cid:15) > 0 ,

Revised: [Equation]

Revised: i

Original: [Equation]
Revised: [Equation]

Original: Because lim
Revised: Because lim

Original: [Equation]
Revised: [Equation]

Original: ) we have:
Revised: 37) we have:

Original: [Equation]
Revised: [Equation]

Original: To sum it up,
Revised: To sum it up,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Remarks: Special cases
Revised: Remarks: Special cases

Original: For different types of graphs and different nodes in the graph, d i r i might be different.
Revised: For different types of graphs and different nodes in the graph, d i r i might be different.

Original: For a d-regular graph whose nodes have constant degree or a graph whose adjacency matrix is row-normalized , we will have:
Revised: For a d-regular graph whose nodes have constant degree or a graph whose adjacency matrix is row-normalized , we will have:

Original: [Equation]
Revised: [Equation]

Original: Equality is achieved if and only if h i = 1 .
Revised: Equality is achieved if and only if h i = 1 .

Original: To note that, h i = 1 is not achievable for every node as long as there are more than one class.
Revised: To note that, h i = 1 is not achievable for every node as long as there are more than one class.

Original: Boundary nodes will suffer most.
Revised: Boundary nodes will suffer most.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: After propagating over multiple layers, we use ξ li to account for the accumulated scaling effects ( γ li ).
Revised: After propagating over multiple layers, we use ξ li to account for the accumulated scaling effects ( γ li ).

Original: That is, E A , { y i } , { f (0) i }| d i (cid:18)
Revised: That is, E A , { y i } , { f (0) i }| d i (cid:18)

Original: f
Revised: f

Original: ( l ) i
Revised: ( l ) i

Original: d
Revised: d

Original: i +1 |
Revised: i +1 |

Original: d i (cid:19) =
Revised: d i (cid:19) =

Original: ξ li µ .
Revised: ξ li µ .

Original: Further,  the group ˆ N si is deﬁned as:
Revised: Further, conditioned on A and ξ li , the set of nodes contributing positively to node v i is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: 0) i }| d i , f ( l ) i ( f ( l ) i | d i , f ( l ) i ) · EA , { y i } , { f (0) i }| d i , f ( l ) i ( f ( l ) j T | d i , f ( l ) i ) > 0 , v j ∈ N i } .
Revised: li ) ≡ { v j | v j ∈ N i and ( ξ li µ ) · E { y i } , { f (0) i }| ξ li , A

Revised: [Equation]

Original: E A , { y i } , { f (0) i }|
Revised: E A , { y i } , { f (0) i }|

Original: d
Revised: d

Original: i , f ( l ) i (cid:32) f ( l ) j r ij | d i , f ( l ) i (cid:33) = (cid:40) ξ li µ v j ∈ ˆ N si − ρ li ξ li µ v j (cid:54)∈ ˆ N si The representations at ( l + 1) -th layer: f ( l +1) i= f
Revised: i ,ξ li ,v j ∈N i

Original: ( l ) i d i + 1 + (cid:88) j ∈N i f ( l ) i √ d i + 1 · (cid:112)
Revised: [Equation]

Original: d j +(41) E A , { y i } , { f (0) i }|
Revised: In the following proof, we use ˆ N si to refer to ˆ N si ( A , ξ li ) for conciseness.

Original: d i , f ( l ) i (cid:32) f ( l +1)
Revised: The representations at ( l + 1) -th layer:

Original: i | d i , f ( l ) i (cid:33) (42) = E A , { y i } , { f (0) i }| d
Revised: [Equation]

Original: i , f ( l ) i (cid:32) f ( l ) i d i + 1 | d i , f ( l ) i (cid:33) +
Revised: [Equation]

Original: E A , { y i } , { f (0) i } | d i , f ( l ) i   (cid:88) j ∈N i f ( l ) j √ d i + 1 · (cid:112) d j + 1 | d i , f ( l ) i   (43) =
Revised: In § 3, we assume that { d j } follow the same distribution and their joint distribution function is permutation-invariant, and { y j } and { f lj } also have this property.

Original: ξ li µ d i + 1 + d i+ 1 E A , { y i } , { f (0) i }| d i ,  f ( l )i (cid:32) (cid:88) j ∈N i f ( l ) j √ d  i +(cid:112)
Revised: Thus, conditioned on d i , ξ li and N i , the distribution of f ( l ) j √ d i +1 √ d j +1 is the same for ∀ j ∈ N i (neighbors are indistinguishable) and we obtain:

Original: [Equation]
Revised: [Equation]

Original: In § 3, we assume that  { d j } follow thesame distribution for every node and their joint distributionfunction is permutation-invariant, which is also the property of { y j } and  { f lj } .
Revised: Given that the joint distribution of { d j } , the joint distribution of { y j } and the joint distribution of { f lj } are permutation-invariant functions, for any

Original: Thus, the distribution of f ( l ) j √ d i +1 √ d j +1 conditioned on d i and f ( l ) i is the same for ∀ j ∈ N i (neighbors are indistinguishable) and we obtain:
Revised: N i ,

Original: [Equation]
Revised: [Equation]

Revised: is the same.

Revised: That is:

Revised: [Equation]

Revised: We note that v j in Equation 49 can be any node except v i due to the equivalence of those nodes.

Revised: Combining Equation 44 and Equation 49, we can obtain:

Original:  Deﬁne ¯ r il ≡ ξ li (cid:48) ξ li .
Revised: [Equation] Deﬁne ¯ r il ≡ ξ li (cid:48) ξ li .

Original: If we know the degree, the l -th layer representation of v i ,  and that a neighbor v j is in the group N si , ¯ r il actually represents the ratio of expected f lj r i,j to  f li .
Revised: If we know the degree d i , ξ li , and that a neighbor v j is in the group N si , ¯ r il actually represents the ratio of expected f lj r i,j to expected f li .

Original: We regard it as the effective related degree at l -th layer.
Revised: We regard it as the effective related degree at l -th layer.

Original: Recall that the related degree r i at layer 1 is the expected average of r ij in the neighborhood.
Revised: Recall that the related degree r i at layer 1 is the expected average of r ij in the neighborhood.

Original: Moreover, let ˆ h li =
Revised: Moreover, let ˆ h li =

Original: P ( v j ∈ ˆ N si | d i , f ( l ) i  ) represent the probability of a neighbor whose representation has a positive contribution in expectation.
Revised: P ( v j ∈ ˆ N si | v j ∈ N i , d i , ξ li ) represent the probability of a neighbor whose representation has a positive contribution in expectation.

Original: This naturally extends the meaning of homophily in deeper layers.
Revised: This naturally extends the meaning of homophily in deeper layers.

Original: Thus, we regard it as effective homophily of node v i at l -th layer.
Revised: Thus, we regard it as effective homophily of node v i at l -th layer.

Original: Given the degree, the l -th layer representation of v i , ρ li represents the ratio of the probability that a neighbor will have a positive rather than a negative contribution to v i .
Revised: Given the degree, the l -th layer representation of v i , ρ li represents the ratio of the probability that a neighbor will have a positive rather than a negative contribution to v i .

Original: This naturally extends the meaning of ρ .
Revised: This naturally extends the meaning of ρ .

Original: We further write Equation 51 as:
Revised: We further write Equation 56 as:

Original: [Equation]
Revised: [Equation]

Original: and it will have three cases similar to Thm. 1.
Revised: and it will have three cases similar to Thm. 1.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: Proof.
Revised: Proof.

Original: Similar to Equation
Revised: Similar to Equation

Original: [Equation]
Revised: [Equation]

Original: we can express the conditional expectation of f (1) i as:
Revised: we can express the conditional expectation of f (1) i as:

Original: [Equation]
Revised: [Equation]

Original: Next, we will show how to compute the conditional expectation and conditional probability in the summand.
Revised: Next, we will show how to compute the conditional expectation and conditional probability in the summand.

Original: [Equation]
Revised: [Equation]

Original: wrongly send information (cid:33) · P
Revised: wrongly send information (cid:33) · P

Original: [Equation]
Revised: [Equation]

Original: v j wrongly send information |
Revised: v j wrongly send information |

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: m 

Revised: m

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: v i ∈ V
Revised: v i ∈ V

Original: [Equation]
Revised: [Equation]

Original: v j ∈ V 1 )
Revised: v j ∈ V 1 )

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: V

Original: [Equation] V [Equation] v j correctly send information (cid:33) · P ( v j correctly send information | A , m 0 i , v i ∈ V 1 , v j ∈ V 1 ) .
Revised: v j correctly send information (cid:33) · P ( v j correctly send information | A , m 0 i , v i ∈ V 1 , v j ∈ V 1 ) .

Original: Combined with the independence assumption, we have:
Revised: Combined with the independence assumption, we have:

Original: [Equation]
Revised: [Equation]

Original: wrongly
Revised: wrongly

Original: [Equation]
Revised: [Equation]

Original: Similarly, we obtain:
Revised: Similarly, we obtain:

Original: P [Equation] v j correctly send information | A , m 0 i , v i ∈ V 1 , v j ∈ V 1 ) = 1 − m 0 i .
Revised: P ( v j correctly send information | A , m 0 i , v i ∈ V 1 , v j ∈ V 1 ) = 1 − m 0 i .

Original: Then Equation (53) can be rewritten as:
Revised: Then Equation (58) can be rewritten as:

Original: [Equation]
Revised: [Equation]

Original: wrongly send information
Revised: wrongly send information

Original: [Equation]
Revised: [Equation]

Original: v
Revised: v

Original: [Equation]
Revised: [Equation]

Original: correctly send information
Revised: correctly send information

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Similarly
Revised: Similarly

Original: [Equation]
Revised: [Equation]

Original: we have:
Revised: we have:

Original: [Equation]
Revised: [Equation]

Original: Consider the independence between m 

Revised: Consider the independence between m

Original: i and node class & degrees
Revised: i and node class & degrees

Original: [Equation]
Revised: [Equation]

Original: and insert Equation (56) and
Revised: and insert Equation (61) and

Original: Equation (57) into Equation (52), we will have:
Revised: Equation (62) into Equation (57), we will have:

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: When h i ≤ 1 , ρ + (1 − ρ ) h
Revised: When h i ≤ 1 , ρ + (1 − ρ ) h

Original: i =
Revised: i =

Original: (1 − h i )
Revised: (1 − h i )

Original: ρ
Revised: ρ

Original: + h i > 0 .
Revised: + h i > 0 .

Original: Deﬁne (cid:15) (cid:48) ≡
Revised: Deﬁne (cid:15) (cid:48) ≡

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: ( ρ + (1 − ρ ) h i ) .
Revised: ( ρ [Equation] (1 − ρ ) h i ) .

Original: To obtain the ranges of γ 1 i , when e 

Revised: To obtain the ranges of γ 1 i , when e

Original: i ≥ 0 .
Revised: i ≥ 0 .

Original: 5 , (cid:15) (cid:48) ≤ 0 , it resembles the derivations of CASE 1 in Proof A.3; when e 0 i < 0 .
Revised: 5 , (cid:15) (cid:48) ≤ 0 , it resembles the derivations of CASE 1 in Proof A.3; when e 0 i < 0 .

Original: 5 , 0 < (cid:15)r i ≤ 1 , it resembles the derivations of CASE 2; when e 0 i < 0 .
Revised: 5 , 0 < (cid:15)r i ≤ 1 , it resembles the derivations of CASE 2; when e 0 i < 0 .

Original: 5 , (cid:15)r i > 1 , it resembles the derivations of CASE 3.
Revised: 5 , (cid:15)r i > 1 , it resembles the derivations of CASE 3.

Original: B A DDITIONAL E XPERIMENTS B. A BLATION STUDY
Revised: B A DDITIONAL E XPERIMENTS B. A BLATION STUDY

Original: [Table]
Revised: [Table]

Original: We now study the impact of two proposed mechanisms (signed messages and degree correction, § 4).
Revised: We now study the impact of two proposed mechanisms (signed messages and degree correction, § 4).

Original: To better show their effects, we add each design choice to a base model and track the changes in the node classiﬁcation performance.
Revised: To better show their effects, we add each design choice to a base model and track the changes in the node classiﬁcation performance.

Original: As the base model, we choose a GCN variant that uses the message passing mechanism (Kipf & Welling, 2016) with weight bias, a residual connection (but not decaying aggregation ) for robustness to oversmoothing, and Elu non-linearity σ .
Revised: As the base model, we choose a GCN variant that uses the message passing mechanism (Kipf & Welling, 2016) with weight bias, a residual connection (but not decaying aggregation ) for robustness to oversmoothing, and Elu non-linearity σ .

Original: We denote the model that replaces the message passing with our signed messages mechanism as +sign , and the model that incorporates the degree correction as +deg .
Revised: We denote the model that replaces the message passing with our signed messages mechanism as +sign , and the model that incorporates the degree correction as +deg .

Original: The model that uses both designs is denoted as +sign,deg .
Revised: The model that uses both designs is denoted as +sign,deg .

Original: Table B.1 gives the accuracy of the models in the supervised node classiﬁcation task for different layers.
Revised: Table B.1 gives the accuracy of the models in the supervised node classiﬁcation task for different layers.

Original: We observe that both mechanisms alleviate the oversmoothing problem.
Revised: We observe that both mechanisms alleviate the oversmoothing problem.

Original: Speciﬁcally, the base model has a sharp performance decrease after 32 layers, while the other models have signiﬁcantly higher performance.
Revised: Speciﬁcally, the base model has a sharp performance decrease after 32 layers, while the other models have signiﬁcantly higher performance.

Original: In general, the +deg model is better than +sign in alleviating oversmoothing, and has a consistent performance gain across different data.
Revised: In general, the +deg model is better than +sign in alleviating oversmoothing, and has a consistent performance gain across different data.

Original: For Chameleon, we observe increase in accuracy as we stack more layers; the large performance gain of GGCN results from the degree correction.
Revised: For Chameleon, we observe increase in accuracy as we stack more layers; the large performance gain of GGCN results from the degree correction.

Original: In
Revised: In

Original: Cora and Cornell, +sign,deg consistently performs better across different layers than the model with only one design, demonstrating the constructive effect from our two proposed mechanisms.
Revised: Cora and Cornell, +sign,deg consistently performs better across different layers than the model with only one design, demonstrating the constructive effect from our two proposed mechanisms.

Original: The signed message design has an advantage in heterophilous datasets.
Revised: The signed message design has an advantage in heterophilous datasets.

Original: In the Cornell dataset, using signed information rather than plain message passing results in over 20% gain, which explains the strong performance of GGCN.
Revised: In the Cornell dataset, using signed information rather than plain message passing results in over 20% gain, which explains the strong performance of GGCN.

Original: However, in homophilous datasets, the beneﬁt from signed messages is limited, as these datasets have few different-class neighbors.
Revised: However, in homophilous datasets, the beneﬁt from signed messages is limited, as these datasets have few different-class neighbors.

Original: This is because when the effective homophily (cid:98) h li is high and error rate e 

Revised: This is because when the effective homophily (cid:98) h li is high and error rate e

Original: i is low, lim (cid:98)
Revised: i is low, lim (cid:98)

Original: h li
Revised: h li

Original: → 1 ,e 0 i →
Revised: → 1 ,e 0 i →

Original: [Equation]
Revised: [Equation]

Original: = 1 , so using signed messages is less beneﬁcial in homophilous datasets.
Revised: = 1 , so using signed messages is less beneﬁcial in homophilous datasets.

Original: B. B ATCH NORM & L AYER NORM
Revised: B. B ATCH NORM & L AYER NORM

Original: In § 4, we use decaying aggregation instead of other normalizing mechanisms.
Revised: In § 4, we use decaying aggregation instead of other normalizing mechanisms.

Original: Other mechanisms, such as batch or layer norm, may be seen as solutions to the heterophily and oversmoothing problems.
Revised: Other mechanisms, such as batch or layer norm, may be seen as solutions to the heterophily and oversmoothing problems.

Original: However, batch norm cannot compensate for the dispersion of the mean vectors (§ 3) due to different degrees and homophily levels of the nodes.
Revised: However, batch norm cannot compensate for the dispersion of the mean vectors (§ 3) due to different degrees and homophily levels of the nodes.

Original: Although, to some extent, it reduces the speed by which the representations of the susceptible nodes (case 1 & 2) move towards the other class (good for oversmoothing), it also prevents the representations of the nodes that could beneﬁt from the propagation (case 3) from increasing the distances (drop in accuracy).
Revised: Although, to some extent, it reduces the speed by which the representations of the susceptible nodes (case 1 & 2) move towards the other class (good for oversmoothing), it also prevents the representations of the nodes that could beneﬁt from the propagation (case 3) from increasing the distances (drop in accuracy).

Original: Layer norm is better at overcoming the dispersion effect but may lead to a signiﬁcant accuracy drop in some datsets when a subset of features are more important than the others.
Revised: Layer norm is better at overcoming the dispersion effect but may lead to a signiﬁcant accuracy drop in some datsets when a subset of features are more important than the others.

Original: Thus, we do not use any of these normalizations.
Revised: Thus, we do not use any of these normalizations.

Original: Next, we provide experiments to show the effects of batch norm and layer norm.
Revised: Next, we provide experiments to show the effects of batch norm and layer norm.

Original: We use the following base model (the same model used in §B.1): a GCN Kipf & Welling (2016) with weight bias, Elu non-linearity and residual connection.
Revised: We use the following base model (the same model used in §B.1): a GCN Kipf & Welling (2016) with weight bias, Elu non-linearity and residual connection.

Original: We do not include any of our designs so as to exclude any other factors that can affect the performance.
Revised: We do not include any of our designs so as to exclude any other factors that can affect the performance.

Original: The models we compare against are +BN and +LN , which represent the models that add batch norm and layer norm right before the non-linear activation, respectively.
Revised: The models we compare against are +BN and +LN , which represent the models that add batch norm and layer norm right before the non-linear activation, respectively.

Original: [Table]
Revised: [Table]

Original: Table B.2 shows that both batch norm and layer norm can help with oversmoothing.
Revised: Table B.2 shows that both batch norm and layer norm can help with oversmoothing.

Original: Moreover, adding layer norm is in general better than adding batch norm.
Revised: Moreover, adding layer norm is in general better than adding batch norm.

Original: This is expected because the scaling effect caused by the propagation can be alleviated by normalizing across the node representations.
Revised: This is expected because the scaling effect caused by the propagation can be alleviated by normalizing across the node representations.

Original: Thus, the dispersion of the expected representations can be mitigated.
Revised: Thus, the dispersion of the expected representations can be mitigated.

Original: On the other hand, batch norm normalizes across all the nodes, so it requires sacriﬁcing the nodes that beneﬁt (case 3) to compensate for the nodes that are prone to moving towards the other classes (case 1 & case 2).
Revised: On the other hand, batch norm normalizes across all the nodes, so it requires sacriﬁcing the nodes that beneﬁt (case 3) to compensate for the nodes that are prone to moving towards the other classes (case 1 & case 2).

Original: As a result, batch norm is less effective in mitigating oversmoothing and leads to a bigger decrease in accuracy.
Revised: As a result, batch norm is less effective in mitigating oversmoothing and leads to a bigger decrease in accuracy.

Original: Another ﬁnding is that both layer norm and batch norm lead to a signiﬁcant accuracy decrease (2%-3%) in the heterophilous datasets.
Revised: Another ﬁnding is that both layer norm and batch norm lead to a signiﬁcant accuracy decrease (2%-3%) in the heterophilous datasets.

Original: +BN has a clear accuracy drop even in the homophilous datasets.
Revised: +BN has a clear accuracy drop even in the homophilous datasets.

Original: As Thm.
Revised: As Thm.

Original: 1 points out: higher heterophily level may result in sign ﬂip.
Revised: 1 points out: higher heterophily level may result in sign ﬂip.

Original: If the representations ﬂip the sign, using batch norm or layer norm will not revert the sign, but they may instead encourage the representations to move towards the other class more.
Revised: If the representations ﬂip the sign, using batch norm or layer norm will not revert the sign, but they may instead encourage the representations to move towards the other class more.

Original: Given the limitations shown above, we do not use either batch norm or layer norm in our proposed model, GGCN.
Revised: Given the limitations shown above, we do not use either batch norm or layer norm in our proposed model, GGCN.

Original: B. M ORE ON THE I NITIAL & D EVELOPING S TAGES
Revised: B. M ORE ON THE I NITIAL & D EVELOPING S TAGES

Original: Section 5.4 shows how the node classiﬁcation accuracy changes for nodes of different degrees with the number of layers on Citeseer.
Revised: Section 5.4 shows how the node classiﬁcation accuracy changes for nodes of different degrees with the number of layers on Citeseer.

Original: Here, we provide more details of this experiment and give the results on another dataset, Cora.
Revised: Here, we provide more details of this experiment and give the results on another dataset, Cora.

Original: Datasets.
Revised: Datasets.

Original: According to Thm. 1 and 2, in order to see both the initial and the developing stage, we need to use homophilous datasets.
Revised: According to Thm. 1 and 2, in order to see both the initial and the developing stage, we need to use homophilous datasets.

Original: In heterophilous datasets, most nodes satisfy case 1, so the initial stage does not exist.
Revised: In heterophilous datasets, most nodes satisfy case 1, so the initial stage does not exist.

Original: Measurement of effective homophily (cid:98) h li (cid:98) h li (cid:98) h li .
Revised: Measurement of effective homophily (cid:98) h li (cid:98) h li (cid:98) h li .

Original: To estimate the effective homophily (cid:98) h li , we measure the portion of neighbors that have the same ground truth label as v i and are correctly classiﬁed.
Revised: To estimate the effective homophily (cid:98) h li , we measure the portion of neighbors that have the same ground truth label as v i and are correctly classiﬁed.

Original: Following our theory, we estimate (cid:98) h li
Revised: Following our theory, we estimate (cid:98) h li

Original: before the last propagation takes place and analyze its impact on the accuracy of the ﬁnal layer.
Revised: before the last propagation takes place and analyze its impact on the accuracy of the ﬁnal layer.

Original: In more detail, we obtain the node representations before the last propagation from a trained vanilla GCN Kipf & Welling (2016) and then perform a linear transformation using the weight matrix and bias vector from the last layer.
Revised: In more detail, we obtain the node representations before the last propagation from a trained vanilla GCN Kipf & Welling (2016) and then perform a linear transformation using the weight matrix and bias vector from the last layer.

Original: Then, we use the transformed representations to classify the neighbors of node v i and compute (cid:98) h li .
Revised: Then, we use the transformed representations to classify the neighbors of node v i and compute (cid:98) h li .

Original: We note that we leverage the intermediate representations only for the estimation of (cid:98) h li ; the accuracy of the ﬁnal layer is still measured using the outputs from the ﬁnal layer.
Revised: We note that we leverage the intermediate representations only for the estimation of (cid:98) h li ; the accuracy of the ﬁnal layer is still measured using the outputs from the ﬁnal layer.

Original: [Table]
Revised: [Table]

Original: Degree intervals.
Revised: Degree intervals.

Original: To investigate the change in GCN accuracy with different layers for nodes with different degrees, we categorize the nodes in n degree intervals.
Revised: To investigate the change in GCN accuracy with different layers for nodes with different degrees, we categorize the nodes in n degree intervals.

Original: For the degree intervals, we use logarithmic binning (base 2).
Revised: For the degree intervals, we use logarithmic binning (base 2).

Original: In detail, we denote the highest and lowest degree by d max and d min , respectively, and let Ω ≡ log 2 d max − log 2 d min n .
Revised: In detail, we denote the highest and lowest degree by d max and d min , respectively, and let Ω ≡ log 2 d max − log 2 d min n .

Original: Then, we divide the nodes into n intervals, where the j -th interval is deﬁned as: [ d min · 2 ( j − 1)Ω , d min · 2 j Ω ) .
Revised: Then, we divide the nodes into n intervals, where the j -th interval is deﬁned as: [ d min · 2 ( j − 1)Ω , d min · 2 j Ω ) .

Original: Dataset: Citeseer.
Revised: Dataset: Citeseer.

Original: Figure 1 and Table B.3 show how the accuracy changes with the number of layers for different node degree intervals.
Revised: Figure 1 and Table B.3 show how the accuracy changes with the number of layers for different node degree intervals.

Original: We observe that in the initial stage, the accuracy increases as the degree and ¯ (cid:98) h li increase.
Revised: We observe that in the initial stage, the accuracy increases as the degree and ¯ (cid:98) h li increase.

Original: However, in the developing stage, the accuracy of high-degree nodes drops more sharply than that of low-degree nodes.
Revised: However, in the developing stage, the accuracy of high-degree nodes drops more sharply than that of low-degree nodes.

Original: Dataset: Cora.
Revised: Dataset: Cora.

Original: The results for Cora are shown in Figure B.1 and Table B.4.
Revised: The results for Cora are shown in Figure B.1 and Table B.4.

Original: In the initial stage, the nodes with lower
Revised: In the initial stage, the nodes with lower

Original: [Table]
Revised: [Table]

Original: degrees usually have lower accuracy.
Revised: degrees usually have lower accuracy.

Original: One exception is the nodes with degrees in the range [3 , 7] .
Revised: One exception is the nodes with degrees in the range [3 , 7] .

Original: These nodes have higher accuracy because the average effective homophily ¯ (cid:98) h li of that degree group is the second highest.
Revised: These nodes have higher accuracy because the average effective homophily ¯ (cid:98) h li of that degree group is the second highest.

Original: In the developing stage, the accuracy of the high-degree nodes drops more than the accuracy of the remaining node groups.
Revised: In the developing stage, the accuracy of the high-degree nodes drops more than the accuracy of the remaining node groups.

Original: GCN’s behavior on both Citesser and Cora datasets veriﬁes our conjecture based on our theorems in § 3.3.
Revised: GCN’s behavior on both Citesser and Cora datasets veriﬁes our conjecture based on our theorems in § 3.3.

Original: B. H OMOPHILY M ETRIC
Revised: B. H OMOPHILY M ETRIC

Original: In Table
Revised: In Table

Original: [Table]
Revised: [Table]

Original: we provide the graph homophily level h
Revised: we provide the graph homophily level h

Original: [Table]
Revised: [Table]

Original: which is deﬁned as the average of the nodes’ homophily levels.
Revised: which is deﬁned as the average of the nodes’ homophily levels.

Original: In our theoretical analysis in §
Revised: In our theoretical analysis in §

Original: [Table]
Revised: [Table]

Original: we use a node’s homophily level and relative degree to characterize three cases (Fig. 2 and Thm. 1).
Revised: we use a node’s homophily level and relative degree to characterize three cases (Fig. 2 and Thm. 1).

Original: To better align our empirical results with our theoretical analysis, we provide below the proportion of nodes in each case.
Revised: To better align our empirical results with our theoretical analysis, we provide below the proportion of nodes in each case.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: We observe that a low graph homophily level indicates that case 1 is the dominant case, while a high graph homophily level indicates that case 3 is the dominant case.
Revised: We observe that a low graph homophily level indicates that case 1 is the dominant case, while a high graph homophily level indicates that case 3 is the dominant case.

Original: As we stated in Thm 1, under "non-swapping" condition, case 3 is the only case to have a beneﬁt.
Revised: As we stated in Thm 1, under "non-swapping" condition, case 3 is the only case to have a beneﬁt.

Original: Thus, in the commonly-used homophilous datasets (Citeseer, Pubmed and Cora), case 3 dominates and enjoys the beneﬁts from graph convolution.
Revised: Thus, in the commonly-used homophilous datasets (Citeseer, Pubmed and Cora), case 3 dominates and enjoys the beneﬁts from graph convolution.

Original: In the Texas, Wisconsin and Actor datasets, only case 1 dominates and casehurts the performance most, this explains why GCN performs worse than most methods in Table 1.
Revised: In the Texas, Wisconsin and Actor datasets, only case 1 dominates and casehurts the performance most, this explains why GCN performs worse than most methods in Table 1.

Original: In the Squirrel and Chameleon datasets, both case 1 and case 2 dominate, GCN yields acceptable results.
Revised: In the Squirrel and Chameleon datasets, both case 1 and case 2 dominate, GCN yields acceptable results.

Original: Using portions of the three cases to analyze the datasets is a better metric, because it aligns more with the GCN’s performance.
Revised: Using portions of the three cases to analyze the datasets is a better metric, because it aligns more with the GCN’s performance.

Original: As shown by Table B.5, Wisconsin, Actor, Squirrel and Chameleon have very close graph level homophily, but GCN’s performance are quite different as shown in Table 1.
Revised: As shown by Table B.5, Wisconsin, Actor, Squirrel and Chameleon have very close graph level homophily, but GCN’s performance are quite different as shown in Table 1.

Original: Our way to deﬁne homophily level does predict that GCN should perform decently in Squirrel and
Revised: Our way to deﬁne homophily level does predict that GCN should perform decently in Squirrel and

Original: Chameleon datasets.
Revised: Chameleon datasets.

Original: B. C OMPLEXITY ANALYSIS
Revised: B. C OMPLEXITY ANALYSIS

Original: We ﬁrst analyze the time complexity of the forward path GGCN.
Revised: We ﬁrst analyze the time complexity of the forward path GGCN.

Original: For degree correction, we need to compute the r ij .
Revised: For degree correction, we need to compute the r ij .

Original: However, we do not need to compute all of them, and we only need to compute r ij for node pairs who are linked.
Revised: However, we do not need to compute all of them, and we only need to compute r ij for node pairs who are linked.

Original: The time complexity is O ( |E| ) , but it is a one-time computation, the results of which can be saved.
Revised: The time complexity is O ( |E| ) , but it is a one-time computation, the results of which can be saved.

Original: We compute τ lij based on the learned weights λ l 0 , λ l 1 .
Revised: We compute τ lij based on the learned weights λ l 0 , λ l 1 .

Original: The time complexity is O ( |E|· L ) , where L is the number of layers.
Revised: The time complexity is O ( |E|· L ) , where L is the number of layers.

Original: The time complexity to compute the signed function is O ( |E| · H · L ) , where H is the hidden dimension of representations, and we only compute the cosine similarity between the nodes that are linked.
Revised: The time complexity to compute the signed function is O ( |E| · H · L ) , where H is the hidden dimension of representations, and we only compute the cosine similarity between the nodes that are linked.

Original: We also need to compute the multiplication of the propagation matrix and representation matrices.
Revised: We also need to compute the multiplication of the propagation matrix and representation matrices.

Original: The time complexity is O ( |V| 2 · H · L ) , .
Revised: The time complexity is O ( |V| 2 · H · L ) , .

Original: Similarly, the time complexity of the multiplication of representation matrices and the weight matrices is O ( |V| · H 2 · L ) .
Revised: Similarly, the time complexity of the multiplication of representation matrices and the weight matrices is O ( |V| · H 2 · L ) .

Original: The total time complexity would be O ( max ( |V| 2 , |E| ) · H · L ) .
Revised: The total time complexity would be O ( max ( |V| 2 , |E| ) · H · L ) .

Original: The complexity of
Revised: The complexity of

Original: GGCN resembles the complexity of attention-based model.
Revised: GGCN resembles the complexity of attention-based model.

Original: Thus we compare with GAT (Veliˇckovi´c et al., 2017)
Revised: Thus we compare with GAT (Veliˇckovi´c et al., 2017)

Original: the total time to train and test on 4 larger homophilous and heterophilous datasets.
Revised: the total time to train and test on 4 larger homophilous and heterophilous datasets.

Original: We run the training and testing for 10 times.
Revised: We run the training and testing for 10 times.

Original: [Table]
Revised: [Table]

Original: In general, GGCN runs faster than GAT because degree correction and signed messages in GGCN learn fewer parameters.
Revised: In general, GGCN runs faster than GAT because degree correction and signed messages in GGCN learn fewer parameters.

Original: [Equation]
Revised: [Equation]

Original: AND P ARAMETERS C. H YPERPARAMETER SETTINGS
Revised: AND P ARAMETERS C. H YPERPARAMETER SETTINGS

Original: Experiments for Table 1 & Table
Revised: Experiments for Table 1 & Table

Original: For the baselines, we set the same hyperparameters that are provided by the original papers or the authors’ github repositories, and we match the results they reported in their respective papers.
Revised: For the baselines, we set the same hyperparameters that are provided by the original papers or the authors’ github repositories, and we match the results they reported in their respective papers.

Original: In our experiments, we ﬁnd that the original hyperparameters set by the authors are already well-tuned.
Revised: In our experiments, we ﬁnd that the original hyperparameters set by the authors are already well-tuned.

Original: All the models use Adam as the optimizer.
Revised: All the models use Adam as the optimizer.

Original: GAT sets the initial learning rate as 0.005 and Geom-GCN uses a custom learning scheduler.
Revised: GAT sets the initial learning rate as 0.005 and Geom-GCN uses a custom learning scheduler.

Original: All the other models (include GGCN) use the initial learning rate 0.01.
Revised: All the other models (include GGCN) use the initial learning rate 0.01.

Original: For GGCN, we use the following hyperparameters:
Revised: For GGCN, we use the following hyperparameters:

Original: • k in the decaying aggregation:
Revised: • k in the decaying aggregation:

Original: We tune the parameters in the following ranges:
Revised: We tune the parameters in the following ranges:

Original: • Dropout rate: [0.0, 0.7] • Weight decay: [1e-7, 1e-2] • Hidden units: {8, 16, 32, 64, 80} • Decay rate η : [0.0, 1.5]
Revised: • Dropout rate: [0.0, 0.7] • Weight decay: [1e-7, 1e-2] • Hidden units: {8, 16, 32, 64, 80} • Decay rate η : [0.0, 1.5]

Original: Experiments for Table B.1 and Table B.
Revised: Experiments for Table B.1 and Table B.

Original: The hyperparameters that are used in all the models (Base, +deg , +sign , +deg,sign , +BN , +LN ) are set to be the same and they are tuned for every dataset.
Revised: The hyperparameters that are used in all the models (Base, +deg , +sign , +deg,sign , +BN , +LN ) are set to be the same and they are tuned for every dataset.

Original: Those common hyperparameters are:
Revised: Those common hyperparameters are:

Original: • Dropout rate: [0.0, 0.7] • Weight decay: [1e-7, 1e-2] • Hidden units: {8, 16, 32, 64, 80}
Revised: • Dropout rate: [0.0, 0.7] • Weight decay: [1e-7, 1e-2] • Hidden units: {8, 16, 32, 64, 80}

Original: C. I NITIALIZATION OF PARAMETERS Initialization
Revised: C. I NITIALIZATION OF PARAMETERS Initialization

Original: For GGCN, we adopt the following parameter initialization in the experiments for Table 1 & Table
Revised: For GGCN, we adopt the following parameter initialization in the experiments for Table 1 & Table

Original: • Initialization of λ l 0 and λ l 1 : 0.5 and 0, respectively • Initialization of α l , β l 0 , β l 1 and β l 2 : 2, 0, 0, 0, respectively.
Revised: • Initialization of λ l 0 and λ l 1 : 0.5 and 0, respectively • Initialization of α l , β l 0 , β l 1 and β l 2 : 2, 0, 0, 0, respectively.

Original: We initialize β l { 0 , 1 , 2 } = 0 in Eq.
Revised: We initialize β l { 0 , 1 , 2 } = 0 in Eq.

Original: (6) because, after applying softmax, ˆ β l { 0 , 1 , 2 } = 1 / 3 ; this ensures equal contributions from positive and negative neighbors and themselves (and sum=1).
Revised: (6) because, after applying softmax, ˆ β l { 0 , 1 , 2 } = 1 / 3 ; this ensures equal contributions from positive and negative neighbors and themselves (and sum=1).

Original: We initialize λ l 1 =in Eq.
Revised: We initialize λ l 1 =in Eq.

Original: (9) following the common practice for initializing the bias.
Revised: (9) following the common practice for initializing the bias.

Original: We set α l = 2 and λ l 0 = 0 .
Revised: We set α l = 2 and λ l 0 = 0 .

Original: 5 in
Revised: 5 in

Original: Eq.
Revised: Eq.

Original: (6) and Eq.
Revised: (6) and Eq.

Original: (9), because when r ij → + ∞ , the degree correction (including global scaling) is: ˆ α l τ lij = softplus (2) · softplus (0 . 5 · (0 − 1) + 0)
Revised: (9), because when r ij → + ∞ , the degree correction (including global scaling) is: ˆ α l τ lij = softplus (2) · softplus (0 . 5 · (0 − 1) + 0)

Original: ≈ 1 .
Revised: ≈ 1 .

Original: As we mention in Thm.
Revised: As we mention in Thm.

Original: 1, when the homophily level is high, nodes with large r i may beneﬁt (case 3), thus we do not want to compensate for these nodes and would like to keep ˆ α l τ lij close to 1.
Revised: 1, when the homophily level is high, nodes with large r i may beneﬁt (case 3), thus we do not want to compensate for these nodes and would like to keep ˆ α l τ lij close to 1.

Original: C. P ARAMETERS AFTER TRAINING
Revised: C. P ARAMETERS AFTER TRAINING

Original: Figure C.1 shows the original and corrected r ij (avg r ij ).
Revised: Figure C.1 shows the original and corrected r ij (avg r ij ).

Original: Corrected r ij are given by ˆ α l τ lij r ij , which are a combination of global scaling and local degree correction.
Revised: Corrected r ij are given by ˆ α l τ lij r ij , which are a combination of global scaling and local degree correction.

Original: Because β l { 0 , 1 , 2 } sum to 1 and do not change global scaling, they are not considered in the corrected r ij .
Revised: Because β l { 0 , 1 , 2 } sum to 1 and do not change global scaling, they are not considered in the corrected r ij .

Original: As can be seen in Fig.
Revised: As can be seen in Fig.

Original: C.1, after the training, GGCN learns to increase r ij , which satisﬁes our theorems.
Revised: C.1, after the training, GGCN learns to increase r ij , which satisﬁes our theorems.

Original: D T ABLE 2 WITH LARGER FONTS
Revised: D T ABLE 2 WITH LARGER FONTS

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]