I S H OMOPHILY A N ECESSITY FOR G RAPH N EURAL N ETWORKS ?
A BSTRACT
Graph neural networks (GNNs) have shown great prowess in learning representations suitable for numerous graph-based machine learning tasks.
When applied to semi-supervised node classiﬁcation, GNNs are widely believed to work well due to the homophily assumption (“like attracts like”), and fail to generalize to heterophilous graphs where dissimilar nodes connect.
Recent works have designed new architectures to overcome such heterophily-related limitations.
However, we empirically ﬁnd that standard graph convolutional networks (GCNs) can actually achieve strong performance on some commonly used heterophilous graphs.
This motivates us to reconsider whether homophily is truly necessary for good GNN performance.
We ﬁnd that this claim is not quite accurate, and certain types of “good” heterophily exist, under which GCNs can achieve strong performance.
Our work carefully characterizes the implications of different heterophily conditions, and provides supporting theoretical understanding and empirical observations.
Finally, we examine existing heterophilous graphs benchmarks and reconcile how the GCN (under)performs on them based on this understanding
I NTRODUCTION
Graph neural networks (GNNs) are a prominent approach for learning representations for graph structured data.
Thanks to their great capacity in jointly leveraging attribute and graph structure information, they have been widely adopted to promote improvements for numerous graph-related learning tasks (Kipf and
Welling, 2016; Hamilton et al., 2017;
Ying et al., 2018; Fan et al., 2019; Zitnik et al., 2018), especially centered around node representation learning and semi-supervised node classiﬁcation (SSNC).
GNNs learn node representations by a recursive neighborhood aggregation process, where each node aggregates and transforms features from its neighbors.
The node representations can then be utilized for downstream node classiﬁcation or regression tasks.
Due to this neighborhood aggregation mechanism, several existing works posit that many
GNNs implicitly assume strong homophily and homophily is critical for GNNs to achieve strong performance on SSNC
(Zhu et al., 2020b;a; Chien et al., 2021;
Maurya et al., 2021; Halcrow et al., 2020; Lim et al., 2021).
In general, homophily describes the phenomenon that nodes tend to connect with “similar” or “alike” others.
Homophily is observed in a wide range of real-world graphs including friendship networks (McPherson et al., 2001), political networks (Gerber et al., 2013; Newman, 2018), citation networks (Ciotti et al., 2016) and more.
Under the homophily assumption, through the aggregation process, a node’s representation is “smoothed” via its neighbors’ representations, since each node is able to receive additional information from neighboring nodes, which are likely to share the same label.
Several recent works (Zhu et al.
, 2020b;a) claim that GNNs are implicitly (or explicitly) designed with homophily in mind, are not suitable for graphs exhibiting heterophily , where connected nodes are prone to have different properties or labels, e.g dating networks or molecular networks (Zhu et al., 2020b).
Such works accordingly design and modify new architectures and demonstrate outperformance over other GNN models on several heterophilous graphs.
Present work.
In our work, we empirically ﬁnd that the graph convolutional network (GCN) Kipf and Welling (2016), a fundamental, representative GNN model (which we focus on in this work) is actually able to outperform such heterophily-speciﬁc models on some heterophilous graphs after careful hyperparameter tuning.
This motivates us to reconsider the popular notion in the literature that GNNs exhibit a homophilous inductive bias, and more speciﬁcally that strong homophily is crucial to strong GNN performance.
Counter to this idea, we ﬁnd that GCN model has the potential to work well for heterophilous graphs under suitable conditions.
We demonstrate intuition with the following toy example: Consider the perfectly heterophilous graph (with all inter-class edges) shown in
Figure 1, where the color indicates the node label.
Blue-labeled and orange-labeled nodes are associated with the scalar feature 0 and 1 , respectively.
If we consider a single-layer GCN by performing an averaging feature aggregation over all neighboring nodes, it is clear that all blue nodes will have a representation of 1 , while the orange nodes will have that of 0 .
Additional layers/aggregations will continue to alternate the features between the two types of nodes.
Regardless of the number of layers, the two classes can still be perfectly separated.
In this toy example, each blue (orange) node only connects orange (blue) nodes, and all blue (orange) nodes share similar neighborhood patterns in terms of their neighbors’ label/feature distributions.
Our work elucidates this intuition and extends it to a more general case: put simply, given a (homophilous or heterophilous) graph, GCN has the potential to achieve good performance if nodes with the same label share similar neighborhood patterns.
We theoretically support this argument by investigating the learned node embeddings from the GCN model.
We ﬁnd that homophilous graphs always satisfy such assumptions, which explains why GCN typically works well for them.
On other hand, there exist both “good” and “bad” heterophily, and GCNs can actually achieve strong performance for “good” heterophily settings while they usually fail on “bad” heterophily settings.
Our work characterizes these settings, and provides a new perspective and solid step towards deeper understanding for heterophilous graphs.
In short:
Our contributions.
(1) We reveal that strong homophily is not a necessary assumption for the GCN model.
The GCN model can perform well over some heterophilous graphs under certain conditions.
(2) We carefully characterize these conditions and provide theoretical understandings on how
GCNs can achieve good SSNC performance under these conditions by investigating their embedding learning process.
(3) We carefully investigate commonly used homophilous and heterophilous benchmarks and reason about GCN’s performs on them utilizing our theoretical understanding.
P RELIMINARIES
Let G =
{V , E} denote a graph, where V and E are the sets of nodes and edges, respectively.
The graph connection information can also be represented as an adjacency matrix A ∈ { 0 , 1 } |V|×|V| , where |V| is the number of nodes in the graph.
The i, j -th element of the adjacency matrix
A
[ i, j ] is equal to 1 if and only if nodes i and j are adjacent to each other, otherwise
A [ i, j ] = 0 .
Each node i is associated with a l -dimensional vector of node features x i ∈ R l ; the features for all nodes can be summarized as a matrix X ∈ R |V|× l .
Furthermore, each node i is associated with a label y i ∈ C , where C denotes the set of labels.
We also denote the set of nodes with a given label c ∈ C as V c .
We assume that labels are only given for a subset of nodes V label ⊂ V .
The goal of semi-supervised node classiﬁcation (SSNC) is to learn a mapping f : V → C utilizing the graph G , the node features
X and the labels for nodes in V label .
2. H OMOPHILY IN G RAPHS
In this work, we focus on investigating performance in the context of graph homophily and heterophily properties.
Homophily in graphs is typically deﬁned based on similarity between connected node pairs, where two nodes are considered similar if they share the same node label.
The homophily ratio is deﬁned based on this intuition following Zhu et al.
(2020b).
Deﬁnition 1 (Homophily) .
Given a graph G = {V , E} and node label vector y , the edge homophily ratio is deﬁned as the fraction of edges that connect nodes with the same labels.
Formally, we have:
[Equation]
where |E| is the number of edges in the graph and 1 ( · ) is the indicator function.
A graph is typically considered to be highly homophilous when h ( · ) is large (typically, 0 .
5 ≤ h ( · ) ≤ 1 ), given suitable label context.
On the other hand, a graph with a low edge homophily ratio is considered to be heterophilous.
In future discourse, we write h ( · ) as h when discussing given a ﬁxed graph and label context.
2. G RAPH N EURAL N ETWORKS
Graph neural networks learn node representations by aggregating and transforming information over the graph structure.
There are different designs and architectures for the aggregation and transformation, which leads to different graph neural network models (Scarselli et al., 2008; Kipf and Welling, 2016; Hamilton et al., 2017; Veliˇckovi´c et al., 2017; Gilmer et al., 2017; Zhou et al., 2020).
One of the most popular and widely adopted GNN models is the graph convolutional network (GCN).
A single GCN operation takes the following form H (cid:48)
=
D − 1 AHW , where H and H (cid:48) denote the input and output features of layer,
W ( k ) ∈ R l × l is a parameter matrix to transform the features, and D is a diagonal matrix and D
[ i, i ]
= deg ( i ) with deg ( i ) denoting the degree of node i .
From a local perspective for node i , the process can be written as a feature averaging process h i = 1 deg ( i ) (cid:80) j ∈N ( i )
Wx j , where N ( i ) denotes the neighbors of node i .
The neighborhood N ( i ) may contain the node i itself.
Usually, when building GCN model upon GCN operations, nonlinear activation functions are added between consecutive GCN operations.
G RAPH C ONVOLUTIONAL N ETWORKS UNDER H ETEROPHILY
Considerable prior literature posits that graph neural networks (such as GCN) work by assuming and exploiting homophily assumptions in the underlying graph (Maurya et al., 2021; Halcrow et al., 2020; Wu et al., 2018).
To this end, researchers have determined that such models are considered to be ill-suited for heterophilous graphs, where the homophily ratio is low (Zhu et al., 2020b;a;
Chien et al., 2021).
To deal with this limitation, researchers proposed several methods including
H2GNN (Zhu et al., 2020b), CPGNN (Zhu et al., 2020a) and GPRGNN (Chien et al., 2021), which are explicitly designed to handle heterophilous graphs via architectural choices (e.g. adding skipconnections, carefully choosing aggregators, etc.)
[Table]
In this section, we revisit the claim that GCNs have fundamental homophily assumptions and are not suited for heterophilous graphs.
To this end, we ﬁrst observe empirically that the GCN model achieves fairly good performance on some of the commonly used heterophilous graphs; speciﬁcally, we present SSNC performance on two commonly used heterophilous graph datasets, Chameleon and Squirrel in Table 1 (see
Appendix D for further details about the datasets and models).
Both Chameleon and Squirrel are highly heterophilous ( h ≈ 0 . 2 ).
We ﬁnd that with some hyperparameter tuning, GCN can outperform alternative methods uniquely designed to operate on some certain heterophilous graphs .
This observation suggests that GCN does not always “underperform” on heterophilous graphs, and it leads us to reconsider the prevalent assumption in literature.
Hence, we next examine how GCNs learn representations, and how this information is used in downstream SSNC tasks.
3. W HEN DOES GCN LEARN SIMILAR EMBEDDINGS FOR NODES WITH THE SAME LABEL ? GCN
is considered to be unable to tackle heterophilous graphs due to its feature averaging process (Zhu et al., 2020b; Chien et al., 2021).
Namely, a node’s newly aggregated features are considered “corrupted” by those neighbors that do not share the same label, leading to the intuition that GCN embeddings are noisy and un-ideal for SSNC.
However, we ﬁnd that crucially, for some heterophilous graphs, the features of nodes with the same label are “corrupted in the same way.”
Hence, the obtained embeddings still contain informative characteristics and thus facilitate SSNC.
We next illustrate when GCN learns similar embeddings for nodes with the same label, beginning with a toy example and generalizing to more practical cases.
a b
GCNs have been shown to be able to capture the local graph topological and structural information (Xu et al., 2019; Morris et al., 2019).
Speciﬁcally, the aggregation step in the GCN model is able to capture and discriminate neighborhood distribution information, e.g. the mean of the neighborhood features (Xu et al., 2019).
Let us consider the two nodes a and b shown in Figure 2, where we use color to indicate the label of each node.
If we further assume that all nodes sharing the same label are associated with exactly the same features, then clearly, after 1 -step aggregation, the GCN operation will output exactly the same embedding for nodes a and b .
Accordingly, Xu et al.
(2019) reasons that the GCN model lacks expressiveness due to its inability to differentiate the two nodes in the embedding space.
However, in the SSNC task, mapping a and b to the same location in the embedding space is explicitly desirable.
Intuitively, if all nodes with the same label are mapped to the same embedding and embeddings for different labels are distinct, SSNC is effortless (Zhao et al., 2020).
Such assumptions are hard to meet in practice.
Thus, to consider a more practical scenario, we assume that both features and neighborhood patterns for nodes with a certain label are sampled from some ﬁxed distributions.
Under these conditions, same-label nodes may not share ﬁxed embeddings, but we can aim to characterize their closeness.
Intuitively, if the learned embeddings for same-label nodes are close and embeddings for other-label nodes are far, we expect strong SSNC performance to be good, given class separability (low intra-class variance and high inter-class variance)
(Fisher, 1936).
We prove that, for graphs meeting suitable conditions the distance between GCN-learned embeddings of any same-label node pair is bounded by a small quantity with high probability.
Assumptions on Graphs.
We consider a graph G , where each node i has features x i ∈ R l and label y
i .
We assume that (1) The features of node i are sampled from feature distribution F y i , i.e, x i ∼ F y i , with µ ( F y i ) denoting its mean; (2) Dimensions of x i are independent to each other; (3)
The features in X are bounded by a positive scalar B , i.e, max i,j | X [ i, j ] | ≤ B ; (4) For node i , its neighbor’s labels are independently sampled from neighbor distribution D y i .
The sampling is repeated for deg ( i ) times to sample the labels for deg ( i ) neighbors.
We denote a graph following these assumptions (1)-(4) as G =
{V , E , {F c , c ∈ C} , {D c , c ∈ C}} .
Note that we use the subscripts in F y i and D y i to indicate that these two distributions are shared by all nodes with the same label as node i .
Next, we analyze the embeddings obtained after a GCN operation.
Following previous works (Li et al., 2018; Chen et al., 2020; Baranwal et al., 2021), we drop the non-linearity in the analysis.
Theorem 1.
Consider a graph G = {V , E , {F c , c ∈ C} , {D c , c ∈ C}} , which follows Assump- tions (1)-(4).
For any node i ∈ V , the expectation of the pre-activation output of a single GCN operation is given by
[Equation]
and for any t > 0 , the probability that the distance between the observation h i and its expectation is larger than t is bounded by
[Equation]
where l denotes the feature dimensionality and ρ ( W ) denotes the largest singular value of W .
The detailed proof can be found in Appendix A. Theorem 1 demonstrates two key ideas.
First, in expectation, all nodes with the same label have the same embedding (Eq.
Second, the distance between the output embedding of a node and its expectation is small with a high probability.
Specifically, this probability is related to the node degree and higher degree nodes have higher probability to be close to the expectation.
Together, these results show that the GCN model is able to map nodes with the same label to an area centered around the expectation in the embedding space under given assumptions.
Then, the downstream classiﬁer in the GCN model is able to assign these nodes to the same class with high probability.
To ensure that the classiﬁer achieves strong performance, the centers (or the expectations) of different classes must be distant from each other; if we assume that µ ( F y i ) are distinct from each other (as is common), then the neighbor distributions {D c , c ∈ C} must be distinguishable to ensure good SSNC performance.
Based on these understandings and discussions, we have the following key (informal) observations on GCN’s performance for graphs with homophily and heterophily.
Observation 1 (GCN under Homophily) .
In homophilous graphs, the neighborhood distribution of nodes with the same label (w.l.o.g c ) can be approximately regarded as a highly skewed discrete D c , with most of the mass concentrated on the category c .
Thus, different labels clearly have distinct distributions.
Hence, the GCN model typically in SSNC on such graph, with high degree nodes beneﬁting more, which is consistent with previous work (Tang et al., 2020b).
Observation 2 (GCN under Heterophily) .
In heterophilous graphs, if the neighborhood distribution of nodes with the same label (w.l.o.g. c ) is (approximately) sampled from a ﬁxed distribution D c , and different labels have distinguishable distributions, then GCN can excel at SSNC, especially when node degrees are large.
Otherwise, GCNs may fail for heterophilous graphs.
Notably, our ﬁndings illustrate that disruptions of certain conditions inhibit GCN performance on heterophilous graphs, but heterophily is not a sufﬁcient condition for poor GCN performance .
GCNs are able to achieve reasonable performance for both homophilous and heterophilous graphs if they follow certain assumptions as discussed in the two observations.
In Section 3.2, we theoretically demonstrate these observations for graphs sampled from the Contextual Stochastic Block Model (CSBM)
(Deshpande et al., 2018) with two classes, whose distinguishablilty of neighborhood distributions can be explicitly characterized.
Furthermore, in Section 3.3, we empirically demonstrate these observations on graphs with multiple classes.
We note that although our derivations are for
GCN, a similar line of analysis can be used for more general message-passing neural networks.
3. A NALYSIS BASED ON CSBM MODEL WITH TWO CLASSES
The CSBM model.
To clearly control assumptions, we study the contextual stochastic block model(CSBM), a generative model for random graphs; such models have been previously adopted for benchmarking graph clustering (Fortunato and Hric, 2016) and GNNs (Tsitsulin et al., 2021).
Speciﬁcally, we consider a CSBM model consisting of two classes c 1 and c 2 .
In this case, the nodes in the generated graphs consist of two disjoint sets C 1 and C 2 corresponding to the two classes, respectively.
Edges are generated according to an intra-class probability p and an inter-class probability q .
Speciﬁcally, any two nodes in the graph, are connected by an edge with probability p , if they are from the same class, otherwise, the probability is q .
For each node i , its initial features x i ∈ R l are sampled from a Gaussian distribution x i ∼ N ( µ , I ) , where µ = µ k ∈ R l for i ∈
C k with k ∈ { 1 , 2 } and µ 1 (cid:54) = µ 2 .
We denote a graph generated from such an CSBM model as
G ∼ CSBM ( µ 1 , µ 2 , p, q ) .
We denote the features for node i obtained after a GCN operation as h i .
Linear separability under GCN.
To better evaluate the effectiveness of GCN operation, we study the linear classiﬁers with the largest margin based on { x i , i ∈ V} and { h i , i ∈ V} and compare their performance.
Since the analysis is based on linear classiﬁers, we do not consider the linear transformation in the GCN operation as it can be absorbed in the linear model, i.e, we only consider the process h
i = 1 deg ( i ) (cid:80) j ∈N ( i ) x j .
For a graph G ∼ CSBM ( µ 1 , µ 2 , p, q ) , we can approximately regard that for each node i , its neighbor’s labels are independently sampled from a neighborhood distribution D y i , where y i denotes the label of node i .
Speciﬁcally, the neighborhood distributions corresponding to c 1 and c 2 are D c 1 =
[ pp + q , qp + q ] and D c 2 =
[ qp + q , pp + q ] , respectively.
Based on the neighborhood distributions, the features obtained from GCN operation follow Gaussian distributions:
[Equation]
Based on the properties of Gaussian distributions, it is easy to see that Theorem 1 holds.
We denote the expectation of the original features for nodes in the two classes as E c 1 [ x i ] and E c 2 [ x i ] .
Similarly, we denote the expectation of the features obtained from GCN operation as E c 1 [ h i ] and E c 2 [ h i ] .
The following proposition describes their relations.
Proposition 1.
( E c 1 [ x i ] , E c 2 [ x i ]) and ( E c 1 [ h i ] , E c 2 [ h i ]) share the same middle point.
E c 1 [ x i ] − E c 2 [ x i ] and E c 1 [ h i ] − E c 2 [ h i ] share the same direction.
Speciﬁcally, the middle point m and the shared direction w are as follows: m = ( µ 1 + µ 2 ) / 2 , and w = ( µ 1 − µ 2 ) / [Table]
This proposition follows from direct calculations.
Given that the feature distributions of these two classes are systematic to each other (for both x i and h i ), the hyperplane that is orthogonal to w and goes through m deﬁnes the decision boundary of the optimal linear classiﬁer for both types of features.
We denote this decision boundary as P = { x | w (cid:62) x − w (cid:62) ( µ 1 + µ 2 ) / 2 } .
Next, to evaluate how GCN operation affects the classiﬁcation performance, we compare the probability that this linear classiﬁer misclassiﬁes a certain node based on the features before and after the GCN operation.
We summarize the results in the following theorem.
Theorem 2.
Consider a graph G ∼ CSBM ( µ 1 , µ 2 , p, q ) .
For any node i in this graph, the linear classiﬁer deﬁned by the decision boundary P has a lower probability to misclassify h i than x i when deg ( i ) >
( p + q ) 2 / ( p − q ) 2 .
The detailed proof can be found in Appendix B. Note that the Euclidean distance between the two discrete neighborhood distributions D c 0 and D c 1 is √ 2 | p − q | ( p + q ) .
Hence, Theorem 2 demonstrates that the node degree deg ( i ) and the distinguishability (measured by the Euclidean distance) of the neighborhood distributions both affect GCN’s performance.
Speciﬁcally, we can make the following conclusions: (1) When p and q are ﬁxed, the GCN operation is more likely to improve the linear separability of the high-degree nodes than low-degree nodes, which is consistent with observations in (Tang et al., 2020b).
(2) The more distinguishable the neighborhood distributions are (or the larger the Euclidean distance is), the more nodes can be beneﬁted from the GCN operation. For
[Figure] example, when p = 9 q or 9 p = q , ( p + q ) 2 / ( p − q ) 2 ≈ 1 . 23 , thus nodes with degree larger thancan beneﬁt from the GCN operation.
These two cases correspond to extremely homophily ( h = 0 . 9 ) and extremely heterophily ( h = 0 . 1 ), respectively.
However, GCN model behaves similarly on these two cases and is able to improve the performance for most nodes.
This clearly demonstrates that heterophily is not a sufﬁcient condition for poor GCN performance.
Likewise, when p ≈ q , the two neighborhood distributions are hardly distinguishable, and only nodes with extremely large degrees can beneﬁt.
In the extreme case, where p = q , the GCN operation cannot help any nodes at all.
Note that Theorem 2 and the followed analysis can be extended to a multi-class CSBM scenario as well – see Appendix F for an intuitive explanation and proof sketch.
3. E MPIRICAL INVESTIGATIONS ON GRAPHS WITH MULTIPLE CLASSES Alg. 1: Hetero. Edge Addition
We conduct experiments to substantiate our claims in Observations 1 and 2 in graphs with multiple classes.
We evaluate how SSNC performance changes as we make a homophilous graph more and more heterophilous under two settings: (1) different labels have distinct distributions, and (2) different labels’ distributions are muddled
. 3.3. T ARGETED H ETEROPHILOUS E DGE A DDITION
Graph generation strategy.
We start with common, realworld benchmark graphs, and modify their topology by adding synthetic, cross-label edges that connect nodes with different labels.
Following our discussion in Observation 2, we construct synthetic graphs that have similar neighborhood distributions for same-label nodes.
Specifically, given a real-world graph G , we ﬁrst deﬁne a discrete neighborhood target distribution D c for each label c ∈ C .
We then follow these target distributions to add cross-label edges.
The process of generating new graphs by adding edges to G is shown in Algorithm 1.
Speciﬁcally, we add a total K edges to the given graph G : to add each edge, we ﬁrst uniformly sample a node i from V with label
y i , then we sample a label c from C according to D y i , and ﬁnally, we uniformly sample a node j from
V c and add the edge ( i, j ) to the graph.
We generate synthetic graphs based on several real-world graphs.
We present the results based on Cora and Citeseer (Sen et al., 2008).
The results for other datasets can be found in Appendix C. Both Cora and Citeseer exhibit strong homophily.
For both datasets, we ﬁx D c for all labels.
Although many suitable D c could be speciﬁed in line with Observation 2, we ﬁx one set for illustration and brevity.
For both datasets, we vary K over 11 values and thus generate 11 graphs.
Notably, as K increases, the homophily h decreases.
More detailed information about the {D c , c ∈ C} and K for both datasets is included in Appendix C.
Observed results.
Figure 3(a-b) show SSNC results (accuracy) on graphs generated based on Cora and Citeseer , respectively.
The black line in both ﬁgures shows results for the presented setting (we introduce γ in next subsection).
Without loss of generality, we use Cora (a) to discuss our ﬁndings, since observations are similar over these datasets.
Each point on the black line in Figure 3(a) represents the performance of GCN model on a certain generated graph and the corresponding value in x -axis denotes the homophily ratio of this graph.
The point with homophily ratio h = 0 .
81 denotes the original Cora graph, i.e, K = 0 .
We observe that as K increases, h decreases, and while the classiﬁcation performance ﬁrst decreases, it eventually begins to increase , showing a V -shape pattern.
For instance, when h = 0 .
25 (a rather heterophilous graph), the GCN model achieves an impressive 86% accuracy, even higher than that achieved on the original Cora graph.
We note that [Table] (c) γ = 0 . 8 , Acc: 44% 0 1 2 3 4 5       0.8 0.82 0.78 0.7 0.77 0.81 0. (d) γ = 1 , Acc: 41% performance continues to increase as K increases further to the right (we censor due to space limitations; see Appendix C for details).
This clearly demonstrates that the GCN model can work well on heterophilous graphs under certain conditions.
Intuitively, the V -shape arises due to a “phase transition”, where the initial topology is overridden by added edges according to the associated D c target neighbor distributions.
In the original graph, the homophily ratio is quite high ( h = 0 . 81 ), and classiﬁcation behavior is akin to that discussed in Observation 1, where same-label nodes have similar neighborhood patterns.
As we add edges to the graph, the originally evident neighborhood patterns are perturbed by added edges and gradually become less informative, which leads to the performance decrease in the decreasing segment of the V -shape in Figure 3(a).
Then, as we keep adding more edges, the neighborhood pattern gradually approaches D c for all c , corresponding to the increasing segment of the V -shape.
3.3. I NTRODUCING NOISE TO NEIGHBORHOOD DISTRIBUTIONS
Graph generation strategy.
In Section 3.3.1, we showed that the GCN model can achieve reasonable performance on heterophilous graphs constructed following distinct, pre-deﬁned neighborhood patterns.
As per Observation 2, our theoretical understanding suggests that performance should degrade under heterophily if the distributions of different labels get more and more indistinguishable.
Hence, we next demonstrate this empirically by introducing controllable noise levels into our edge addition strategy.
We adopt a strategy similar to that described in Algorithm 1, but with the key difference being that we introduce an additional parameter γ , which controls the probability that we add cross-label edges randomly rather than following the pre-deﬁned distributions.
A detailed description of this approach is demonstrated in Algorithm 2 in Appendix C. For nodes of a given class c (w.l.o.g), compared to the edges added according to D c , the randomly added edges can be regarded as noise.
Speciﬁcally, by increasing the noise parameter γ , we increase the similarity between D c , D c (cid:48) for any pair of labels c, c (cid:48) .
If γ = 1 , then all neighborhood distributions will be indistinguishable (they will all be approximately Uniform ( |C| ) .
By ﬁxing K and varying γ , we can generate graph variants with the same homophily ratio but different similarities between D c and
D c (cid:48) .
As in Section 3.3.1, we create graphs by adding edges at various K , but also vary γ ∈ [0 , 1] in increments of 0 .
2 on both Cora and Citeseer .
Observed results.
We report the SSNC performance on these graphs in Figure 3.
Firstly, we observe that noise affects the performance signiﬁcantly when the homophily ratio is low.
For example, observing Figure 3(a) vertically at homophily ratio h = 0 .
25 , higher γ clearly results in worse performance.
This indicates that not only the ﬁxed-ness of the neighborhood distributions, but their similarities are important for the SSNC task (aligned with Observation 2.
It also indicates that there are “good” and “bad” kinds of heterophily.
On the other hand, high γ does not too-negatively impact when K is small, since noise is minimal and the original graph topology is yet largely homophilous.
At this stage, both “good” (ﬁxed and disparate patterns) and “bad” (randomly added edges) heterophilous edges introduce noise to the dominant homophilous patterns.
When the noise level γ is not too large, we can still observe the V -shape: e.g. γ = 0 .
4 in Figure 3(a) and γ = 0 .in
Figure 3 (b)
; this is because the designed pattern is not totally dominated by the noise.
However, when γ is too high, adding edges will constantly decrease the performance, as nodes of different classes have indistinguishably similar neighborhoods.
To further demonstrate how γ affects the neighborhood distributions in the generated graph, we examine the cross-class neighborhood similarity, which we deﬁne as follows:
Deﬁnition 2 (Cross-Class Neighborhood Similarity (CCNS)) .
Given graph G and labels y for all nodes, the CCNS between classes c, c (cid:48) ∈ C is s ( c, c (cid:48) ) =
[Equation]
[Equation]
∈V c (cid:48)
cos ( d ( i ) , d ( j )) where
V c indicates the set of nodes in class c and d ( i ) denotes the empirical histogram (over |C| classes) of node i ’s neighbors’ labels, and the function cos( · , · ) measures the cosine similarity.
When c = c (cid:48) , s ( c, c (cid:48) ) calculates the intra-class similarity, otherwise, it calculates the inter-class similarity from a neighborhood label distribution perspective.
Intuitively, if nodes with the same label share the same neighborhood distributions, the intra-class similarity should be high.
Likewise, to ensure that the neighborhood patterns for nodes with different labels are distinguishable, the interclass similarity should be low.
To illustrate how various γ values affect the neighborhood patterns, we illustrate the intra-class and inter-class similarities in Figure 4 for γ = 0 , 0 . 4 , 0 . 8 , 1 on graphs generated from Cora with homophily ratio h = 0 . 25 .
The diagonal cells in each heatmap indicate the intra-class similarity while off-diagonal cells indicate inter-class similarity.
Clearly, when γ is small, the intra-class similarity is high while the inter-class similarity is low, which demonstrates the existence of strongly discriminative neighborhood patterns in the graph.
As γ increases, the intra-class and inter-class similarity get closer, becoming more and more indistinguishable, leading to bad performance due to indistinguishable distributions as referenced in Observation 2.
R EVISITING GCN’ S P ERFORMANCE ON R EAL - WORLD G RAPHS
In this section, we ﬁrst give more details on the experiments we run to compare GCN and MLP.
We next investigate why the GCN model does or does not work well on certain datasets utilizing the understanding developed in earlier sections.
4. Q UANTITATIVE A NALYSIS
Following previous work (Pei et al., 2020; Zhu et al., 2020b), we evaluate the performance of the GCN model on several real-world graphs with different levels of homophily.
We include the citation networks Cora , Citeseer and Pubmed (Kipf and Welling, 2016), which are highly homophilous.
We also adopt several heterophilous benchmark datasets including Chameleon , Squirrel , Actor , Cornell , Wisconsin and Texas (Rozemberczki et al., 2021; Pei et al., 2020).
Appendix D.1 gives descriptions and summary statistics of these datasets.
For all datasets, we follow the experimental setting provided in (Pei et al., 2020), which consists of 10 random splits with proportions 48 / 32 / 20 % corresponding to training/validation/test for each graph.
For each split, we use 10 random seeds, and report the average performance and standard deviation across 100 runs.
We compare the GCN model with the MLP model, which does not utilize the graph structure.
With this comparison, we aim to check whether the GCN model always fails for for heterophilous graphs (perform even worse than MLP).
We also compare GCN with state-of-the-art methods and their descriptions and performance are included in the Appendix D.
The node classiﬁcation performance (accuracy) of these models is reported in Table 2.
Notably, GCN achieves better performance than MLP on graphs with high homophily ( Cora , Citeseer , and Pubmed ), as expected.
For the heterophilous graphs, the results are comparatively mixed.
The GCN model outperforms MLP on Squirrel and Chameleon (it even outperforms methods speciﬁcally designed for heterophilous graphs as shown in Appendix D.5.), while underperforming on the other datasets ( Actor , Cornell , Wisconsin , and Texas ).
In the next section, we provide explanations for GCN’s distinct behaviors on these graphs based on the understanding developed in earlier sections.
4. Q UALITATIVE A NALYSIS
Our work so far illustrates that the popular notion of GCNs not being suitable for heterophily, or homophily being a mandate for good GCN performance is not accurate.
In this subsection, we aim to use the understanding we developed in Section 3 to explain why GCN does (not) work well on real-world graphs.
As in Section 3.3.2, we inspect cross-class neighborhood similarity (Deﬁnition 2) for each dataset; due to the space limit, we only include representative ones here ( Cora , Chameleon , Actor and Cornell ); see Figure 5).
Heatmaps for the other datasets can be found in Appendix E. From Figure 5(a), it is clear that the intra-class similarity is much higher than the inter-similarity ones, hence Cora contains distinct neighborhood patterns, consistent with Observation 1.
In Figure 5(b), we can observe that in Chameleon , intra-class similarity is generally higher than inter-class similarity, though not as strong as in Figure 5(a).
Additionally, there is an apparent gap between labels 0 , 1 and 2 , 3 , 4 , which contributes to separating nodes of the former 2 from the latter 3 classes, but potentially increasing misclassiﬁcation within each of the two groupings.
These observations also help substantiate why GCN can achieve reasonable performance (much higher than MLP) on Chameleon .
The GCN model underperforms MLP in Actor
and we suspect that the graph does not provide useful information.
The heatmap for Actor in Figureshows that the intra-class and inter-class similarities are almost equivalent, making the neighborhood [Table] (b) Chameleon          0.51 0.51 0.51 0.5 0. (c) Actor          0.6 0.69 0.53 0.57 0.
(d) Cornell distributions for different classes hard to distinguish and leading to bad GCN performance.
Similar observations are made for Cornell .
Note that Cornell only consists of 183 nodes andedges, hence, the similarities shown in Figure 5 are impacted signiﬁcantly (e.g. there is a single node with label 1 , leading to perfect intra-class similarity for label 1 ).
R ELATED W ORK
Graph neural networks (GNNs) are powerful models for graph representation learning.
They have been widely adopted to tackle numerous applications from various domains (Kipf and Welling, 2016;
Fan et al., 2019; Bastings et al., 2017; Shi et al., 2019).
(Scarselli et al., 2008) proposed the ﬁrst GNN model, to tackle both node and graph level tasks.
Subsequently, Bruna et al.
and Defferrard et al.
(2016) generalized convolutional neural networks to graphs from the graph spectral perspective.
Kipf and Welling (2016) simpliﬁed the spectral GNN model and proposed graph convolutional networks (GCNs).
Since then, numerous GNN variants, which follow speciﬁc forms of feature transformation (linear layers) and aggregation have been proposed (Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017; Gilmer et al., 2017; Klicpera et al., 2019).
The aggregation process can be usually understood as feature smoothing (Li et al., 2018;
Ma et al., 2020; Jia and Benson, 2021;
Zhu et al., 2021).
Hence, several recent works claim (Zhu et al., 2020b;a; Chien et al., 2021), assume (Halcrow et al., 2020; Wu et al., 2018;
Zhao et al., 2020) or remark upon (Abu-El-Haija et al., 2019;
Maurya et al., 2021;
Hou et al., 2020)
GNN models homophily-reliance or unsuitability in capturing heterophily.
Several recent works speciﬁcally develop GNN models choices to tackle heterophilous graphs by carefully designing or modifying model architectures such as Geom-GCN (Pei et al., 2020), H2GCN (Zhu et al., 2020b), GPR-GNN (Chien et al., 2021), and CPGNN (Zhu et al., 2020a).
Some other works aim to modify/construct graphs to be more homophilous (Suresh et al., 2021).
There is concurrent work (Luan et al., 2021) also pointing out that GCN can potentially achieve strong performance on heterophilous graphs.
A major focus of this paper is to propose a new model to handle heterophilous graphs.
However, our work aims to empirically and theoretically understand whether homophily is a necessity for GNNs and how GNNs work for heterophilous graphs.
C ONCLUSION
It is widely believed that GNN models inherently assume strong homophily and hence fail to generalize to graphs with heterophily.
In this paper, we revisit this popular notion and show it is not quite accurate.
We investigate one representative model, GCN, and show empirically that it can achieve good performance on some heterophilous graphs under certain conditions.
We analyze theoretically the conditions required for GCNs to learn similar embeddings for same-label nodes, facilitating the
SSNC task; put simply, when nodes with the same label share similar neighborhood patterns, and different classes have distinguishable patterns, GCN can achieve strong class separation, regardless of homophily or heterophily properties.
Empirical analysis supports our theoretical ﬁndings.
Finally, we revisit several existing homophilous and heterophilous SSNC benchmark graphs, and investigate GCN’s empirical performance in light of our understanding.
Note that while there exist graphs with “good heterophily”, “bad heterophily” still poses challenges to GNN models, which calls for dedicated efforts.
We discuss the limitation of the current work in Appendix I.
A P ROOF OF T HEOREM
To prove Theorem 1, we ﬁrst introduce the celebrated Hoeffding inequality below.
Lemma 1 (Hoeffding’s Inequality) .
Let Z 1 , . . .
, Z n be independent bounded random variables with
Z i ∈ [ a, b ] for all i , where −∞ < a ≤ b < ∞ .
Then
[Equation]
and
[Equation]
for all t ≥ 0 .
Theorem 1.
Consider a graph G = {V , E , {F c , c ∈ C} , {D c , c ∈ C}} , which follows Assumptions (1)-(4).
For any node i ∈ V , the expectation of the pre-activation output of a single GCN operation is given by
[Equation]
and for any t > 0 , the probability that the distance between the observation h i and its expectation is larger than t is bounded by
[Equation]
where l is the feature dimensionality and ρ ( W ) denotes the largest singular value of W .
Proof.
The expectation of h i can be derived as follows.
[Equation]
We utilize Hoeffding’s Inequality to prove the bound in Eq.
Let x i [ k ] , k = 1 , . . .
, l denote the i -th element of x .
Then, for any dimension k , { x j [ k ] , j ∈ N ( i ) } is a set of independent bounded random variables.
Hence, directly applying Hoeffding’s inequality, for any t 1 ≥ 0 , we have the following bound:
[Equation]
If (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) (cid:80) j ∈N ( i ) ( x j − E [ x j ]) (cid:13)(cid:13)(cid:13)(cid:13)(cid:13) ≥ √ lt 1 , then at least for one k ∈ { 1 , . . .
, l } , the inequality (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) (cid:80) j ∈N ( i ) ( x j [ k
[Equation]
− E
[Equation]
x j [ k ]]) (cid:12)(cid:12)(cid:12)(cid:12)(cid:12) ≥ t 1 holds.
Hence, we have
P 
[Equation]
Let t 1 = t 2 √ l , then we have
[Equation]
Furthermore, we have
[Equation]
where (cid:107) W (cid:107) 2 is the matrix 2 -norm of W .
Note that the last line uses the identity (cid:107) W (cid:107) 2 = ρ ( W ) .
Then, for any t > 0 , we have
[Equation]
which completes the proof.
B P ROOF OF T HEOREM
Theorem 2.
Consider a graph G ∼ CSBM ( µ 1 , µ 2 , p, q ) .
For any node i in this graph, the linear classiﬁer deﬁned by the decision boundary P has a lower probability to mis-classify h i than x i when deg ( i ) >
( p + q ) 2 / ( p − q ) 2 .
Proof.
We only prove for nodes from classes c 0 since the case for nodes from classes c 1 is symmetric and the proof is exactly the same.
For a node i ∈ C 0 , we have the follows
[Equation]
8) where w and b = − w (cid:62) ( µ 1 + µ 1 ) / 2 is the parameters of the decision boundary P .
we have that
[Equation]
We denote the scaled version of h i as h (cid:48)
i = (cid:112) deg ( i ) h
i .
Then, h (cid:48)
i follows
[Equation]
Because of the scale in Eq.
(9), the decision boundary for h (cid:48)
i is correspondingly moved to w (cid:62) h (cid:48) + (cid:112) deg ( i )
b
= 0 .
Now, since x i and h (cid:48)
i share the same variance, to compare the mis-classiﬁcation probabilities, we only need to compare the distance from their expected value to their corresponding decision boundary.
Speciﬁcally, the two distances are as follows:
[Equation]
The larger the distance is the smaller the mis-classiﬁcation probability is.
Hence, when dis h (cid:48)
i < dis x
i , h (cid:48)
i has a lower probability to be mis-classiﬁed than x i .
Comparing the two distances, we conclude that when deg ( i ) > (cid:16) p + q p − q (cid:17) 2 , h (cid:48)
i has a lower probability to be mis-classiﬁed than x i .
Together with Eq.
9, we have that
[Equation]
which completes the proof.
C A DDITIONAL D ETAILS AND R ESULTS FOR S ECTION 3.3. C. D ETAILS ON THE GENERATED GRAPHS
In this subsection, we present the details of the graphs that we generate in Section 3.3.1.
Speciﬁcally, we detail the distributions {D c , c ∈ C} used in the examples, the number of added edges K , and the homophily ratio h .
We provide the details for Cora and Citeseer in the following subsections.
Note that the choices of distributions shown here are for illustrative purposes, to coincide with
Observations 1 and 2.
We adapted circulant matrix-like designs due to their simplicity.
C.1. C ORA
There are 7 labels, which we denote as { 0 , 1 , 2 , 3 , 4 , 5 , 6 } .
The distributions {D c , c ∈ C} are listed as follows.
The values of K and the homophily ratio of their corresponding generated graphs are shown in Table 3.
D 0 : Categorical ([0 , 0 .
5]) , D 1 : Categorical ([0 . 5 , 0 , 0 .
5 , 0 , 0 , 0 , 0]) , D 2 : Categorical ([0 , 0 .
5 , 0 , 0 , 0]) , D 3 : Categorical ([0 , 0 , 0 .
5 , 0 , 0]) , D 4 : Categorical ([0 , 0 , 0 , 0 .
5 , 0]) , D 5 : Categorical ([0 , 0 , 0 , 0 , 0 .
5]) , D 6 : Categorical ([0 . 5 , 0 , 0 , 0 , 0 , 0 .
[Table]
C.1. C ITESEER
There are 6 labels, which we denote as { 0 , 1 , 2 , 3 , 4 , 5 } .
The distributions {D c , c ∈ C} are listed as follows.
The values of K and the homophily ratio of their corresponding generated graphs are shown in Table 4.
D 0 : Categorical ([0 , 0 .
5]) , D 1 : Categorical ([0 . 5 , 0 , 0 .
5 , 0 , 0 , 0]) , D 2 : Categorical ([0 , 0 .
5 , 0 , 0]) , D 3 : Categorical ([0 , 0 , 0 .
5 , 0]) , D 4 : Categorical ([0 , 0 , 0 , 0 .
5]) , D 5 : Categorical ([0 . 5 , 0 , 0 , 0 , 0 .
[Table]
R ESULTS ON MORE DATASETS : C HAMELEON AND S QUIRREL
We conduct similar experiments as those in Section 3.3.1 based on Chameleon and Squirrel .
Note that both Squirrel and Chameleon have 5 labels, which we denote as { 0 , 1 , 2 , 3 , 4 } .
We pre-deﬁne the same distributions for them as listed as follows.
The values of K and the homophily ratio of their corresponding generated graphs based on Squirrel and Chameleon are shown in
Table 5 and Table
[Table]
respectively.
[Equation]
[Equation] [Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation] [Equation] [Equation] [Equation]
[Equation] [Equation]
[Table]
Chameleon .
K
[Figure]
The performance of the GCN model on these two sets of graphs (generated from Squirrel and Chameleon ) is shown in Figure 6.
The observations are similar to what we found for those generated graphs based on Cora and Citeseer in Section 3.3.1.
Note that the original Squirrel and Chameleon graphs already have very low homophily, but we still observe a V -shape from the ﬁgures.
This is because there are some neighborhood patterns in the original graphs, which are distinct from those that we designed for addition.
Hence, when we add edges in the early stage, the performance decreases.
As we add more edges, the designed pattern starts to mask the original patterns and the performance starts to increase.
C. GCN’ S P ERFORMANCE IN THE L IMIT ( AS K → ∞ )
In this subsection, we illustrate that as K → ∞ , the accuracy of the GCN model approaches 100% .
Speciﬁcally, we set K to a set of larger numbers as listed in Table 7.
Ideally, when K → ∞ , the homophily ratio will approach 0 and the model performance will approach 100% (for diverseenough D c ).
The performance of the GCN model on the graphs described in Table 3 and Tableare shown in Figure 7.
Clearly, the performance of the GCN model approaches the maximum as we continue to increase K .
[Table]
[Figure]
C. D ETAILS OF A LGORITHM
The pseudo code to describe the process to generate graphs with a noise level γ is shown in Algorithm 2.
The only difference from Algorithm 1 is in Line 6-7, we randomly add edges if the generated random number r is smaller than the pre-deﬁned γ (with probability γ ).
Alg. 2: Heterophilous Edge Addition with Noise input : G = {V , E} , K , {D c } |C|− 1 c =0 and {V c } |C|− 1 c
= output: G (cid:48)
= {V , E (cid:48) }
Initialize G (cid:48)
= {V , E} , k = 1 ; while 1 ≤ k ≤ K do
Sample node i ∼ Uniform ( V ) ; Obtain the label, y i of node i ;
Sample a number r ∼ Uniform (0,1) ; // Uniform (0,1) denotes the continuous standard uniform distribution if r ≤ γ then Sample a label c ∼ Uniform ( C \ { y i } ); else Sample a label c ∼ D y i ;
Sample node j ∼ Uniform ( V c ) ;
Update edge set E (cid:48) =
E (cid:48) ∪ { ( i, j ) } ; k ← k + 1 ; return G (cid:48)
= {V , E (cid:48) }
D E XPERIMENTAL D ETAILS : D ATASETS , M ODELS , AND R ESULTS
We compare the standard GCN model (Kipf and Welling, 2016) with several recently proposed methods speciﬁcally designed for heterophilous graphs including H2GCN (Zhu et al., 2020b), GPRGNN (Chien et al., 2021), and CPGNN (Zhu et al., 2020a).
A brief introduction of these methods can be found in Appendix D.2.
D. D ATASETS
We give the number of nodes, edges, homophily ratios and distinct classes of datasets we used in this paper in Table 8.
[Table]
D. M ODELS
• H2GCN (Zhu et al., 2020b) speciﬁcally designed several architectures to deal with heterophilous graphs, which include ego- and neighbor-embedding separation (skip connection), aggregation from higher-order neighborhoods, and combination of intermediate representations.
We include two variants H2GCN-1 and H2GCN-2 with 1 or 2 steps of aggregations, respectively.
We adopt the code published by the authors at https://github.com/GemsLab/H2GCN .
• GPR-GNN (Chien et al., 2021) performs feature aggregation for multiple steps and then linearly combines the features aggregated with different steps.
The weights of the linear combination are learned during the model training.
Note that it also includes the original features before aggregation in the combination.
We adopt the code published by the authors at https://github.com/jianhao2016/GPRGNN .
• CPGNN (Zhu et al., 2020a) incorporates the label compatibility matrix to capture the connection information between classes.
We adopted two variants of CPGNN that utilize MLP and ChebyNet (Defferrard et al., 2016) as base models to pre-calculate the compatibility matrix, respectively.
We use two aggregation layers for both variants.
We adopt the code published by the authors at https://github.com/GemsLab/CPGNN .
D. MLP+GCN
We implement a simple method to linearly combine the learned features from the GCN model and an MLP model.
Let H (2) GCN ∈ R |V|×|C| denote the output features from a 2 -layer GCN model, where |V| and |C| denote the number of nodes and the number of classes, respectively.
Similarly, we use
H (2) MLP ∈ R |V|×|C| to denote the features output from a 2 -layer MLP model.
We then combine them for classiﬁcation.
The process can be described as follows.
[Equation]
(13) where α is a hyperparameter balancing the two components.
We then apply a row-wise softmax to each row of H to perform the classiﬁcation.
D. P ARAMETER T UNING AND R ESOURCES U SED
We tune parameters for GCN, GPR-GCN, CPGNN, and MLP+GCN from the following options:
• learning rate: { 0 . 002 , 0 .
} • weight decay { 5 e − 04 , 5 e − 05 , 5 e − 06 , 5 e − 07 , 5 e − 08 , 1 e − 05 , 0 } • dropout rate: { 0 , 0 . 2 , 0 . 5 , 0 . 8 } .
All experiments are run on a cluster equipped with Intel(R) Xeon(R)
CPU E5-2680 v4 @ 2.40GHz
CPUs and NVIDIA Tesla K80 GPUs
[Table]
D. M ORE R ESULTS
[Table]
E H EATMAPS FOR O THER B ENCHMARKS
We provide the heatmaps for Citeseer and Pubmed in Figure 8 and those for Squirrel , Texas , and Wisconsin in Figure 9.
For the Citeseer and Pubmed , which have high homophily, the observations are similar to those of Cora as we described in Section 4.2.
For Squirrel , there are some patterns; the intra-class similarity is generally higher than inter-class similarities.
However, these patterns are not very strong, i.e, the differences between them are not very large, which means that the neighborhood patterns of different labels are not very distinguishable from each other.
This substantiates the middling performance of GCN on Squirrel .
Both Texas and Wisconsin are very small, with 183 nodes, 295 edges and 251 nodes, 466 edges, respectively.
The average degree is extremely small ( < 2 ).
Hence, the similarities presented in the heatmap may present strong bias.
Especially, in Texas , there is only 1 node with label 1 .
In Wisconsin , there are only 10 nodes with label 0 .
[Figure]
F E XTENDING T HEOREM 2 TO M ULTIPLE C LASSES
Below, we provide a proof sketch and illustration for an extension to Theorem 2’s main results in a multi-class context.
Speciﬁcally, we consider a special case for more tractable analysis.
Consider a K -class CSBM.
The nodes in the generated graphs consist of K disjoint sets of the same size C 1 , . . .
, C K corresponding to the K classes, respectively.
Edges are generated according to an intra-class probability p and an inter-class probability q .
Speciﬁcally, for any two nodes in the graph, if they are from the same class, then an edge is generated to connect them with probability p , otherwise, the probability is q .
For each node i , its initial associated features x i ∈ R l are sampled from a Gaussian distribution x i ∼ N ( µ , I ) , where µ = µ k ∈ R l for i ∈
C k with k ∈ { 1 , . . .
, K } and µ z (cid:54) =
µ w ∀ z, w ∈ { 1 , . .
. , K } .
We further assume that the distance between the mean of distributions corresponding to any two classes is equivalent, i.e, (cid:107)
u
z − u w (cid:107) 2 =
D, ∀ z, w ∈ { 1 , . .
, K } , where D is a positive constant.
We illustrate the 3 -classes case in Figure 10, where we use the dashed circles to demonstrate the standard deviation.
Note that the K classes of the CSBM model are symmetric to each other.
Hence, the optimal decision boundary of the K classes are deﬁned by a set
[Figure]
Squirrel          0.65 0.03 0.61 0.17 0.
(b) Texas          0.75 0.25 0.27 0.36 0. (c
[Figure] Wisconsin of (cid:0) K 2 (cid:1) hyperplanes (see Figure 10 for an example), where each hyperplane equivalently separates two classes.
Similar to the analysis in the binary case (see the description below Proposition 1), for any given two classes
c z and c w , the hyperplane is P = { x | w (cid:62) x − w (cid:62) ( µ z + µ w ) / 2 } with w = ( u z − u w ) / (cid:107) u
z − u w ) (cid:107) 2 , which is orthogonal to ( µ z − µ w ) and going through ( µ z + µ w ) / 2 .
For example, in Figure 10, the decision boundaries separate the entire space to 3 areas corresponding to the 3 classes.
Each decision boundary is deﬁned by a hyperplane equally separating two classes.
For example, the descision bounadry between c 1 and c 2 is orthogonal to ( µ 1 − µ 2 ) and going through the middle point ( µ 1 + µ 2 ) / 2 .
Clearly, the linear separability is dependent on the distance D between the classes and also the standard deviations of each class’ node features.
More speciﬁcally, linear separability is favored by a larger distance D and smaller standard deviation.
Next, we discuss how the GCN operation affects the linear separability.
Overall, we want to demonstrate the following: (i) after the GCN operation, the classes are still symmetric to each other; (ii) the GCN operation will reduce the distance between the classes (in terms of the expectation of the output embeddings) and reduce the standard deviation of the classes; and (iii) reducing the distance impairs the separability while reducing the standard deviation improves the separability.
Hence, we analyze the two effect and provide a threshold.
We describe these three items in a more detailed way as follows.
• Preservation of Symmetry.
For any class c z , z ∈ { 1 , . . .
, K } , its neighborhood label
distribution
D c z can be described by a vector where only the z -th element equals
p p +(
K
− q
and all other elements equal to
[Equation]
q
We consider the aggression process h
i =
[Equation]
x j Then
for a node i with label c z , its features obtained after this process follow the following Gaussian distribution
[Equation]
Speciﬁcally, we denote the expectation of class c z after the GCN operation as E c z
[ h ] =
[Equation]
We next show that the distance between any two classes is the same.
Speciﬁcally, for
any two classes c z
and
c w
, after the GCN operation the distance between their expectation is as follows
[Equation]
(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
[Equation]
Note that we have (cid:107) µ z − µ w (cid:107) 2 =
D for any pair of classes
c z and c w .
Hence, after the GCN operation, the distance between any two classes is still the same.
Thus, the classes are symmetric two each other.
• Reducing inter-class distance and intra-class standard deviation.
According to Eq (15), after the graph convolution operation, the distance between any two classes is reduced by a factor of | p − q | p +( K − 1) q .
On the other hand, according to Eq.
(14), the standard deviation depends on the degree of nodes.
Speciﬁcally, for node i with degree deg ( i ) , its standard deviation is reduced by a factor of (cid:112) deg ( i ) .
• Implications for separability.
For a node i , the probability of being mis-classiﬁed is the probability of its embedding falling out of its corresponding decision area.
This probability depends on both the inter-class distance between classes (in terms of means) and the intra-class standard deviation.
To compare the linear separability before and after the graph convolution operation, we scale the distance between classes before and after the graph convolution operation to be the same.
Speciﬁcally, we scale h i as h (cid:48)
i = p
+( K − 1) q | p − q | h i .
Note that classifying h i is equivalent to classifying h (cid:48) i .
Based
on h (cid:48) , the distance between any two classes c z , c w is scaled to (cid:107) µ z − µ w (cid:107) [-
-]2 = D , which is equivalent to the class distance before the graph convolution operation.
Correspondingly, the standard deviation for h (cid:48)
i equals to p +( K − 1) q | p − q | ·
I √ deg ( i ) .
Now, to compare the mis-classiﬁcation probability before and after the graph convolution, we only need to compare the standard deviations as the distances between classes in these two scenarios has been scaled to the same.
More speciﬁcally, when p +( K − 1) q | p − q | ·
I √ deg ( i ) < 1 , the mis-classiﬁcation probability for node i is reduced after the GCN model, otherwise, the mis-classiﬁcation probability is increased after the
GCN model.
In other words, for nodes with degree larger than ( p +( K − 1) q ) 2 ( p − q ), the mis-classiﬁcation rate can be reduced after the graph convolution operation.
This thereold is similar to the one we developed in Theorem 2 and similar analysis/discussions as those for Theorem 2 follows for the this multiple-class case.
G O THER M ETRICS T HAN C OSINE S IMILARITY
The choice of similarity measure would not affect the results and conclusions signiﬁcantly.
We empirically demonstrate this argument by investigating two other metrics: Euclidean distance and
Hellinger distance.
The heatmaps based on these two metrics for Chamelon dataset is shown in
Figure 11 in Appendix G.
The patterns demonstrated in these two heatmaps are similar to those observed in Figure 5(b), where cosine similarity is adopted.
Note that larger distance means lower similarity.
Hence, the numbers in Figure 11 should be interpreted in the opposite way as those in
Figure 5.
[Figure]
Euclidean Distance          1.
[Figure]
H A DDITIONAL E XPERIMENTS WITH D IFFERENT S ETTINGS
In this section, we discuss additional experiments for Section 3.3.1.
H. G ENERATING G RAPHS WITH O THER N EIGHBORHOOD D ISTRIBUTIONS
In this section, we extend the experiments in Section 3.3.1 by including more patterns for neighborhood distributions.
We aim to illustrate if the neighborhood distribution for different classes (labels) are sufﬁciently distinguishable from each other, we should generally observe similar V -shape curves as in Figure 3 in Section 3.3.1.
However, it is impractical to enumerate all possible neighborhood distributions.
Hence, in this section, we include two additional neighborhood distribution patterns in Section H.1.1 and two extreme patterns in Section H.1.2.
H.1. A DDITIONAL P ATTERNS
Here, for both Cora and Citeseer , we adopt two additional sets of neighborhood distributions for adding new edges.
For convenience, for Cora , we name the two neighborhood distribution patterns as Cora Neighborhood Distribution Pattern 1 and Cora Neighborhood Distribution Pattern 2 .
Similarly, for Citeseer , we name the two neighborhood distribution patterns as Citeseer
Neighborhood Distribution Pattern 1 and Citeseer Neighborhood Distribution Pattern 2 .
We follow
Algorithm 1 to generate graphs while utilizing these neighborhood distributions as the {D c } | C |− 1 c = for Algorithm 1.
The two neighborhood distribution patterns for Cora are listed as bellow.
Figure 12 and Figureshow GCN’s performance on graphs generated from Cora following Cora Neighborhood Distribution Pattern 1 and Cora Neighborhood Distribution Pattern 2 , respectively.
Cora Neighborhood Distribution Pattern 1 :
[Figure]
[Figure]
[Figure]
[Figure]
Clearly, the results demonstrated in Figure 12 and Figure 13 are similar to the black curve ( γ =0) in 3a.
More speciﬁcally, they present V -shape curves.
The two neighborhood distribution patterns for Citeseer are listed as bellow.
Figure 14 and Figure 15 show GCN’s performance on graphs generated from Citeseer following Citeseer Neighborhood Distribution Pattern 1 and Citeseer Neighborhood Distribution Pattern 2 , respectively.
Citeseer Neighborhood Distribution Pattern 1 :
D 0 : Categorical ([0 , 1 2 , 1 2 , 0 , 0 , 0]) , D 1 : Categorical ([1 3 , 0 , 0 , 0 , 1 3 , 1 3 ]) , D 2 : Categorical ([12 , 0 , 0 , 0 , 0 , 1 2]) , D 3 : Categorical ([0 , 0 , 0 , 0 , 1 , 0) , D 4 : Categorical ([0 , 1 3 , 0 , 1 3 , 0 , 1 3]) , D 5 : Categorical ([0 , 1 3 , 1 3 , 0 , 1 3 , 0]) ,
[Figure]
Graphs are generated following Citeseer Neighborhood Distribution Pattern
Citeseer Neighborhood Distribution Pattern 2 :
[Figure]
Categorical ([
[Figure]
[Figure]
[Figure]
[Figure]
D
[Figure]
: Categorical ([12 , 0 ,
[Figure]
D
[Figure]
: Categorical ([0 , 0 ,
[Figure]
D
[Figure]
{+: Categorical ([15 , 
+}
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
: Categorical ([0 , 0 ,
[Figure]
[Figure] [Figure] [Figure] : Categorical ([
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] Clearly, the results demonstrated in Figure 14 and Figure 15 are similar to the black curve ( γ =0) in 3b.
More speciﬁcally, they present V -shape curves.
H.1. E XTREME N EIGHBORHOOD D ISTRIBUTION P ATTERNS
In this subsection, we further extend the experiments in Section 3.3.1 by investigating “extreme” neighborhood distribution patterns suggested by WXGg.
More speciﬁcally, these two patterns are “a given label are connected to a single different label” and “a given label are connected to all other labels other excluding its own label”.
For convenience, we denote these two types of neighborhood distributions as single and all , respectively.
We utilize these neighborhood distributions as {D c } |C|− 1 c =0 for generating synthetic graphs.
Next, we ﬁrst present the results for Cora with analysis for both the single and all neighborhood distribution patterns.
Then, we present the results for Citeseer but omit analysis and discussion since the observations are similar to those we make for Cora .
Note that to ensure “a label is only connected to a single different label”, we need to group the labels into pairs and “connect them”.
Since there are different ways to group labels into pairs, there exists various ways to formulate the single neighborhood distributions.
We demonstrate one of the single neighborhood distributions as all the other possible are symmetric to each other.
For Cora , the single neighborhood distribution we adopted is as follows.
Note that there are 7 labels in Cora , and we could not pair all the labels.
Thus, we leave one of labels (label 4 in our setting) untouched, i.e, it is not connect to other labels during the edge addition process. [Equation]
D [Equation] Categorical
[Equation]
Categorical
[Equation]
The GCN’s performance on these graphs generated from Cora following the single neighborhood distribution pattern is shown in Figure 16a.
It clearly presents a V -shape curve.
Almost perfect performance can be achieved when the homophily ratio gets close to 0 .
This is because the single neighborhood distributions for different labels are clearly distinguishable from each other.
We further demonstrate the heatmap of cross-class similarity for the generated graph with homophily ratio 0 .
(the right most point in Figure 16a) in Figure 16b.
Clearly, this graph has very high intra-class similarity and very low inter-class similarity, which explains the good performance.
For Cora , the all neighborhood distribution patterns can be described as follows.
D 0 : Categorical ([0 , 1 6 , 1 6 , 1 6 , 1 6 , 1 6 , 1 6]) [Equation] D 1 : Categorical ([ [Table]
[Figure]
[Figure]
[Figure] [Figure] [Figure] Categorical ([16 , 1 6 , 1 6 , 1 6 , 0 , 1 6 , 1 6])
[Figure] : Categorical ([16 , 1 6 , 1 6 , 1 6 , 1 6 , 0 , 1 6]) , D [Figure] : Categorical ([16 , 1 6 , 1 6 , 1 6 , 1 6 , 1 6 , 0]) .
The GCN’s performance on these graphs generated from Cora following the all neighborhood distribution pattern is shown in Figure 17a.
Again, it clearly presents a V -shape curve.
However, the performance is not perfectly good even when we add extremely large number of edges.
For example, for the right most point in Figure 17a, we add almost as 50 times many as edges into the graph and the homophily ratio is reduced to 0 .
03 while GCN’s performance for it is only around 70% .
This is because the all neighborhood distributions for different labels are not easily distinguishable from each other.
More speciﬁcally, any two neighborhood distributions for different labels in all are very similar each other (they share “ 4 labels”).
We further empirically demonstrate this by providing the heatmap of cross-class similarity for the generated graph with homophily ratio 0 .
03 (the right most point in Figure 17a) in Figure 17b.
As per our discussion in Observation 2, this graph is with not such “good” heterophily and GCNs cannot produce perfect performance for it.
This observation further demonstrates our key argument that the distinguishability of the distributions for different labels are important for performance.
For Citeseer , the single neighborhood distribution we adopted is as follows.
The GCN’s perfor- mance on these graphs generated from Citeseer following the single neighborhood distribution pattern is shown in Figure 18a.
The heatmap of cross-class similarity for the generated graph with homophily ratio 0 .
06 (the right most point in Figure 18a) in Figure 18b.
D 0 : Categorical ([0 , 1 , 0 , 0 , 0 , 0]) , D 1 : Categorical ([1 , 0 , 0 , 0 , 0 , 0]) , D 2 : Categorical ([0 , 0 , 0 , 1 , 0 , 0]) , D 3 : Categorical ([0 , 0 , 1 , 0 , 0 , 0]) , D 4 : Categorical ([0 , 0 , 0 , 0 , 0 , 1]) , D 5 : Categorical ([0 , 0 , 0 , 0 , 1 , 0]) .
For Citeseer , the all neighborhood distribution patterns can be described as follows.
The GCN’s performance on these graphs generated from Citeseer following the all neighborhood distribution pattern is shown in Figure 19a.
The heatmap of cross-class similarity for the generated graph with homophily ratio 0 .
01 (the right most point in Figure 19a) in Figure 19b.
[Equation]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Equation]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure] [Figure] [Figure] [Figure]
[Equation]
[Figure] [Figure] [Figure] [Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
Similar observations as those we made for Cora can be made for Citeseer , hence we do not repeat the analysis here
[Figure]
I L IMITATION
Though we provide new perspectives and understandings of GCN’s performance on heterophilous graphs, our work has some limitations. To make the theoretical analysis more feasible, we make a few assumptions. We dropped the non-linearity in the analysis since the main focus of this paper is the aggregation part of GCN.
While the experiment results empirically demonstrate that our analysis seems to hold with non-linearity, more formal investigation for GCN with non-linearity is valuable.
Our analysis in Theorem 2 assumes the independence between features, which limits the generality of the analysis and we would like to conduct further instigation to more general case.
We provide theoretical understanding on GCN’s performance based on CSBM, which stands for a type of graphs
[Figure]
T e s t
A cc u r a c y
[Figure]
[Figure]
attracting increasing attention in the research community.
However, it is not ideal for modeling sparse graphs, which are commonly observed in the real-world.
Hence, it is important to devote more efforts to analyzing more general graphs.
We believe our results established a solid initial study for further investigation.
Finally, our current theoretical analysis majorly focuses on the GCN model; we hope to extend this analysis in the future to more general message-passing neural networks.
J B ROADER I MPACT
Graph neural networks (GNNs) are a prominent architecture for modeling and understanding graphstructured data in a variety of practical applications.
Most GNNs have a natural inductive bias towards leveraging graph neighborhood information to make inferences, which can exacerbate unfair or biased outcomes during inference, especially when such neighborhoods are formed according to inherently biased upstream processes, e.g. rich-get-richer phenomena and other disparities in the opportunities to “connect” to other nodes: For example, older papers garner more citations than newer ones, and are hence likely to have a higher in-degree in citation networks and hence beneﬁt more from neighborhood information; similar analogs can be drawn for more established webpages attracting more attention in search results.
Professional networking (“ability to connect”) may be easier for those individuals (nodes) who are at top-tier, well-funded universities compared to those who are not.
Such factors inﬂuence network formation, sparsity, and thus GNN inference quality simply due to network topology
(Tang et al., 2020b).
Given these acknowledged issues,
GNNs are still used in applications including ranking (Sankar et al., 2021), recommendation(Jain and Molino), engagement prediction (Tang et al., 2020a), trafﬁc modeling(Jiang and Luo, 2021), search and discovery (Ying et al., 2018) and more, and when unchecked, suffer traditional machine learning unfairness issues (Dai and Wang, 2021).
Despite these practical impacts, the prominent notion in prior literature in this space has been that such methods are inapplicable or perform poorly on heterophilous graphs, and this may have mitigated practitioners’ interests in applying such methods for ML problems in those domains conventionally considered heterophily-dominant (e.g. dating networks).
Our work shows that this notion is misleading, and that heterophily and homophily are not themselves responsible for good or bad inference performance.
We anticipate this ﬁnding to be helpful in furthering research into the capacity of GNN models to work in diverse data settings, and emphasize that our work provides an understanding, rather than a new methodology or approach, and thus do not anticipate negative broader impacts from our ﬁndings. A CKNOWLEDGEMENTS This research is supported by the National Science Foundation (NSF) under grant numbers IIS1714741, CNS1815636, IIS1845081, IIS1907704, IIS1928278, IIS1955285, IOS2107215, and
IOS2035472, the Army Research Ofﬁce (ARO) under grant number W911NF-21-1-0198, the Home
Depot, Cisco Systems Inc and Snap Inc.