Revisiting Heterophily For Graph Neural Networks
Anonymous Author(s)
AfﬁliationAddress email
Abstract
Graph Neural Networks (GNNs) extend basic Neural Networks (NNs) by usinggraph structures based on the relational inductive bias (homophily assumption).
While GNNs have been commonly believed to outperform NNs in real-world tasks,recent work has identiﬁed a non-trivial set of datasets where their performancecompared to NNs is not satisfactory.
Heterophily has been considered the maincause of this empirical observation and numerous works have been put forward toaddress it.
In this paper, we ﬁrst revisit the widely used homophily metrics andpoint out that their consideration of only graph-label consistency is a shortcoming.
Then, we study heterophily from the perspective of post-aggregation node similarityand deﬁne new homophily metrics, which are potentially advantageous comparedto existing ones.
Based on this investigation, we prove that some harmful cases ofheterophily can be effectively addressed by local diversiﬁcation operation.
Then,we propose the Adaptive Channel Mixing (ACM), a framework to adaptivelyexploit aggregation, diversiﬁcation and identity channels to extract richer localizedinformation in each baseline GNN layer.
ACM is more powerful than the commonlyused uni-channel framework for node classiﬁcation tasks on heterophilic graphs.
When evaluated on 10 benchmark node classiﬁcation tasks, ACM-augmentedbaselines consistently achieve signiﬁcant performance gain, exceeding state-of-theart GNNs on most tasks without incurring signiﬁcant computational burden.
Introduction
Deep Neural Networks (NNs) [22] have revolutionized many machine learning areas, includingimage recognition [21], speech recognition [13] and natural language processing [2], due to theireffectiveness in learning latent representations from Euclidean data.
Recent research has shifted focuson non-Euclidean data [6], e.g., relational data or graphs.
Combining graph signal processing andconvolutional neural networks [23], numerous Graph Neural Networks (GNNs) architectures havebeen proposed [38, 10, 15, 40, 19, 29], which empirically outperform traditional NNs on graph-basedmachine learning tasks such as node classiﬁcation, graph classiﬁcation, link prediction and graphgeneration, etc.
GNNs are built on the homophily assumption [34]: connected nodes tend to sharesimilar attributes with each other [14], which offers additional information besides node features.
This relational inductive bias [3] is believed to be a key factor leading to GNNs’ superior performanceover NNs’ in many tasks.
However, growing empirical evidence suggests that GNNs are not always advantageous compared totraditional NNs.
In some cases, even simple Multi-Layer Perceptrons (MLPs) can outperform GNNsby a large margin on relational data [45, 27, 31, 8].
An important reason for this is believed to be theheterophily problem: the homophily assumption does not always hold, so connected nodes may infact have different attributes.
Heterophily has received lots of attention recently and an increasingnumber of models have been put forward to address this problem [45, 27, 8, 44, 43, 32, 16].
In this paper, we ﬁrst show that by only considering graph-label consistency, existing homophily metrics are not able to describe the effect of some cases of heterophily on aggregation-based GNNs.
Wepropose a post-aggregation node similarity matrix, and based on it, we derive new homophily metrics,whose advantages are illustrated on synthetic graphs (Sec. 3).
Then, we prove that diversiﬁcationoperation can help to address some harmful cases of heterophily (Sec. 4).
Based on this, we proposean Adaptive Channel Mixing (ACM)
GNN framework which generalizes the concept of ﬁlterbanksand augments uni-channel baseline GNNs, allowing them to exploit aggregation, diversiﬁcation and identity channels adaptively, nodewisely and locally in each layer.
ACM signiﬁcantly boosts theperformance of 3 uni-channel baseline GNNs by 2.04% ∼ 27.5% for node classiﬁcation tasks on 7widely used benchmark heterophilic graphs, exceeding SOTA models (Sec. 6) on 6 of them.
For 3homophilic graphs, ACM-augmented GNNs perform at least as well as the uni-channel baselines.
Contributions 1.
To our knowledge, we are the ﬁrst to analyze heterophily from post-aggregationnode similarity perspective.
The proposed ACM framework is highly different from adaptiveﬁlterbank and existing GNNs for heterophily: 1) the traditional adaptive ﬁlterbank [39] uses a scalarweight for each ﬁlter and this weight is shared by all nodes.
In contrast, we suggest a method such thatdifferent nodes can learn different weights to utilizes information from different channels to accountfor heterophily; 2) Unlike existing methods that leverage the high-order ﬁlters and global propertyof high-frequency signals [45, 27, 8, 16], ACM successfully addresses heterophily by consideringonly the nodewise local information adaptively .
Unlike existing methods that try to facilitatelearning ﬁlters with high expressive power [45, 44, 8, 16], the goal of ACM is that, when given aﬁlter with certain expressive power, we can extract richer information from additional channels in acertain way to address heterophily.
This makes ACM more ﬂexible and easier to implement.
Preliminaries
In this section, we introduce notation and background knowledge.
We use bold font for vectors ( e.g.,v ).
Suppose we have an undirected connected graph G =
( V , E , A ) , where V is the node set with|V| = N ; E is the edge set without self-loops; A ∈ R N × N is the symmetric adjacency matrix with
A i,j = 1 iff e ij ∈ E .
Let D denote the diagonal degree matrix of G , i.e., D i,i =
d i = (cid:80)
j A i,j .
Let N i denote the neighborhood set of node i , i.e., N i =
{ j : e ij ∈ E} .
A graph signal is a vectorx ∈ R N deﬁned on V , where x i is associated with node i .
We also have a feature matrix X ∈ R N × F ,whose columns are graph signals and whose i -th row
X i, : is a feature vector of node i .
We use
Z ∈ R N × C to denote the label encoding matrix, whose i -th row Z i, : is the one-hot encoding of thelabel of node i .
2.1 Graph Laplacian, Afﬁnity Matrix and Variants
The (combinatorial) graph Laplacian is deﬁned as L = D − A , which is Symmetric Positive
Semi-Deﬁnite (SPSD) [9].
Its eigendecomposition is L = U Λ U T , where the columns u i of
U ∈ R N × N are orthonormal eigenvectors, namely the graph Fourier basis , Λ = diag( λ 1 , . .
. , λ N )with λ 1 ≤ · · · ≤ λ N .
These eigenvalues are also called frequencies .
In additional to L , some variants are also commonly used, e.g., the symmetric normalized Laplacian
L sym =
D − 1 / 2 LD − 1 / 2 =
I − D − 1 / 2 AD − 1 / 2 and the random walk normalized Laplacian L
rw =
D − 1 L =
I − D − 1 A .
The graph Laplacian and its variants can be considered as high-pass ﬁltersfor graph signals.
The afﬁnity (transition) matrices can be derived from the Laplacians, e.g., A rw =
I − L rw =
D − 1 A , A sym
=
I − L sym
=
D − 1 / 2 AD − 1 / 2 and are considered to be low-pass ﬁlters[33].
Their eigenvalues satisfy λ
i ( A rw )
=
λ i ( A sym )
=
1 − λ i ( L sym )
= 1 − λ i ( L rw ) ∈ ( − 1 , 1] .
Applying the renormalization trick [19] to afﬁnity and Laplacian matrices respectively leads toˆ A sym =
˜ D − 1 / 2 ˜
A ˜ D − 1 / 2 and ˆ L sym
=
I − ˆ A sym , where ˜
A ≡ A + I and ˜ D ≡ D + I .
The renormalizedafﬁnity matrix essentially adds a self-loop to each node in the graph, and is widely used in Graph
Convolutional Network (GCN) [19] as follows,
[Equation]
where W 0 ∈ R F × F 1 and W 1 ∈ R F 1 × O are learnable parameter matrices.
GCNs can be trained byminimizing the following cross entropy loss
[Equation]
where log( · ) is a component-wise logarithm operation.
The random walk renormalized matrixˆ A rw =
˜ D − 1 ˜ A , which shares the same eigenvalues as ˆ A sym , can also be applied in GCN.
Thecorresponding Laplacian is deﬁned as ˆ L rw =
I − ˆ A rw .
The matrix ˆ
A rw is essentially a random walkmatrix and behaves as a mean aggregator that is applied in spatial-based GNNs [15, 14].
To bridgespectral and spatial methods, we use ˆ A rw .
2.2 Metrics of Homophily
The homophily metrics are deﬁned by considering different relations between node labels and graphstructures deﬁned by their adjacency matrix.
There are three commonly used homophily metrics: edge homophily [1, 45], node homophily [35], and class homophily [25] 1 , deﬁned as follows:
[Equation]
v ∈V
[Equation]
u ∈ N
[Equation]
Z u
[Equation]
[Equation]
Z v
[Equation]
d v
[Equation]
[Equation]
N (cid:35) + with h k = (cid:80) v ∈V (cid:12)(cid:12) { u | Z v,k
= 1 , u ∈ N v , Z u, : =
Z v, : } (cid:12)(cid:12) (cid:80) v ∈{ v | Z v,k =1 } d v (3) where
[ a ] + = max( a, 0) ; h k is the class-wise homophily metric [25].
All metrics are in the range of[0 , 1] ; a value close to 1 corresponds to strong homophily, while a value close to 0 indicates strongheterophily.
H edge ( G ) measures the proportion of edges that connect two nodes in the same class;
H node ( G ) evaluates the average proportion of edge-label consistency of all nodes; H class ( G ) tries toavoid sensitivity to imbalanced classes, which can cause H edge ( G ) to be misleadingly large.
The abovedeﬁnitions are all based on the linear feature-independent graph-label consistency .
Inconsistencyis implied to have a negative effect to the performance of GNNs.
With this in mind, in the followingsection, we will show an example to illustrate the shortcomings of the above metrics and we proposenew feature-independent metrics that are deﬁned from a post-aggregation perspective.
Analysis of Heterophily
[Figure]
3.1 Motivation and Aggregation Homophily
[Figure]
After
Aggregation
Metrics: H !
[Figure] [Figure]
G)
[Figure]
0 H
[Figure]
Figure harmless heterophily
Heterophily is widely believed to be harmfulfor message-passing based GNNs [45, 35, 8] because intuitively features of nodes in differentclasses will be falsely mixed, leading nodes tobe indistinguishable [45].
Nevertheless, it is notalways the case, e.g., the bipartite graph 2 shownin Figure 1 is highly heterophilic according tothe existing homophily metrics in equation 3,but after mean aggregation, the nodes in classes1 and 2 only exchange colors and are still distinguishable 3 .
This example tells us that, besidesgraph-label consistency, we need to study therelation between nodes after aggregation.
Tothis end, we ﬁrst deﬁne the post-aggregation node similarity matrix as follows:
[Equation]
where ˆ A ∈ R N × N denotes a general aggregation operator.
S ( ˆ A, X ) is essentially the gram matrixthat measures the similarity between each pair of aggregated node features.
Relationship Between S ( ˆ A, X ) and Gradient of SGC SGC [41] is one of the most simple butrepresentative GNN models and its output can be written as:
[Equation]
With the loss function in equation 2, after each gradient descent step, we have ∆ W = γ d L dW , where γis the learning rate.
The update of Y (cid:48) is (see Appendix C for derivation):
[Equation]
where Z − Y is the prediction error matrix.
The update direction of the prediction for node i isessentially a weighted sum of the prediction error, i.e., ∆( Y (cid:48) )
i, : = (cid:80) j ∈V (cid:104) S
( ˆ A, X ) (cid:105) i,j ( Z − Y )
j, :and (cid:104) S
( ˆ A, X ) (cid:105) i,j can be considered as the weights.
Intuitively, a high similarity value means node i tends to be updated to the same class as node j .
This indicates that S ( ˆ A, X ) is closely related to asingle layer GNN model.
Based on the above deﬁnition and observation, we deﬁne the aggregation similarity score as follows.
Deﬁnition 1.
The aggregation similarity score is:
[Equation] [Equation]
where Mean u ( {·} ) takes the average over u of a given multiset of values or variables.
S agg ( S ( ˆ A, X )) measures the proportion of nodes v ∈ V for which the average of weights on theset of nodes in the same class as v (including v ) is larger than that in other classes.
In practice, weobserve that in most datasets, we will have S agg ( S ( ˆ A, X )) ≥ 0 .
To make the metric range in [0,1],like existing metrics, we rescale Eq.(equation 7) to the following modiﬁed aggregation similarity,
[Equation]
In order to measure the consistency between labels and graph structures without considering nodefeatures and to make a fair comparison with the existing homophily metrics in equation 3, we deﬁnethe graph ( G ) aggregation ( ˆ A ) homophily and its modiﬁed version as:
[Equation]
In practice, we will only check H agg ( G )
when H M agg ( G ) = 0 .
For the example in Figure 1, whenˆ A
= ˆ
A rw , it is easy to see that H agg ( G ) =
H M agg ( G ) = 1 and other metrics are 0.
Thus, this newmetric reﬂects the fact that nodes in classes 1 and 2 are still highly distinguishable after aggregation,while other metrics mentioned before fail to capture such information and misleadingly give value0.
This shows the advantage of H agg ( G ) and H M agg ( G ) , which additionally exploit information fromaggregation operator ˆ A and the similarity matrix.
To comprehensively compare H M agg ( G ) with the existing metrics on their ability to elucidate theinﬂuence of graph structure on GNN performance, we generate synthetic graphs with differenthomophily levels and evaluate SGC [41] and GCN [19] on them in the next subsection.
[Figure]
3.2 Empirical Evaluation and Comparison on Synthetic Graphs
In this subsection, we conduct experiments on synthetic graphs generated with different levels of
H M edge ( G ) to assess the output of H M agg ( G ) in comparison with existing metrics.
Data Generation & Experimental Setup We ﬁrst generated 10 graphs for each of 28 edge homophily levels, from 0.005 to 0.95, for a total of 280 graphs.
For every generated graph, we had5 classes, with 400 nodes in each class.
For nodes in each class, we randomly generated 800intra-class edges and [ 800 H edge ( G ) − 800 ] inter-class edges.
The features of nodes in each class aresampled from node features in the corresponding class of 6 base datasets ( Cora, CiteSeer, PubMed,
Chameleon, Squirrel, Film ).
Nodes were randomly split into train/validation/test sets, in proportionof 60%/20%/20%.
We trained 1-hop SGC ( sgc-1 ) [41] and GCN [19] on the synthetic graphs 5 .
Foreach value of H edge ( G ) , we take the average test accuracy and standard deviation of runs over the 10generated graphs with that value.
For each generated graph, we also calculate H node ( G ) , H class ( G )and H M agg ( G ) .
Model performance with respect to different homophily values is shown in Figure 2.
[Figure] [Figure]
[Figure] [Figure]
[Figure] [Figure]
[Figure]
Homophily Metrics
The performance of SGC-1 and GCN is expected to be monotonically increasing if the homophily metric is informative.
However, Figure 2(a)(b)(c) show that the performance curves under H edge ( G ) , H node ( G ) and H class ( G ) are U -shaped 6 , while Figure2(d) reveals a nearly monotonic curve with a little numerical perturbation around 1.
This indicates that
H M agg ( G ) provides a better indication of the way in which the graph structure affects the performanceof SGC-1 and GCN than existing metrics.
(See more discussion on aggregation homophily andtheoretical results for regular graphs in Appendix B.)
Adaptive Channel Mixing (ACM)
In prior work [31, 8, 4], it has been shown that high-frequency graph signals, which can be extractedby a high-pass ﬁlter (HP), is empirically useful for addressing heterophily.
In this section, based onthe similarity matrix in equation 6, we theoretically prove that a diversiﬁcation operation, i.e., HPﬁlter, can address some cases of harmful heterophily locally.
Besides, a node-wise analysis shows thatdifferent nodes may need different ﬁlters to process their information.
Based on the above analysis, in
Sec.
4.2 we propose Adaptive Channel Mixing (ACM), a 3-channel architecture which can adaptivelyexploit local and node-wise information from aggregation, diversiﬁcation and identity channels.
4.1 Diversiﬁcation Helps with Harmful Heterophily
We ﬁrst consider the example shown in Figure 3.
From S ( ˆ A, X ) , we can see that nodes { 1 , 3 } assignrelatively large positive weights to nodes in class 2 after aggregation, which will make nodes { 1 , 3 }hard to be distinguished from nodes in class 2.
However, we can still distinguish nodes { 1 , 3 } and{ 4 , 5 , 6 , 7 } by considering their neighborhood differences: nodes { 1 , 3 } are different from most oftheir neighbors while nodes { 4 , 5 , 6 , 7 } are similar to most of their neighbors.
This indicates thatalthough some nodes become similar after aggregation, they are still distinguishable through theirlocal surrounding dissimilarities.
This observation leads us to introduce the diversiﬁcation operation , i.e., HP ﬁlter I − ˆ A [11] toextract information regarding neighborhood differences, thereby addressing harmful heterophily.
As
S ( I − ˆ A, X ) in Fig.
3 shows, nodes { 1 , 3 } will assign negative weights to nodes { 4 , 5 , 6 , 7 } afterthe diversiﬁcation operation, i.e., nodes 1,3 treat nodes 4,5,6,7 as negative samples and will moveaway from them during backpropagation.
This example reveals that there are cases in which thediversiﬁcation operation is helpful to handle heterophily, while the aggregation operation is not.
Based on this observation, we ﬁrst deﬁne the diversiﬁcation distinguishability of a node and thegraph diversiﬁcation distinguishability value, which measures the proportion of nodes for which thediversiﬁcation operation is potentially helpful.
Deﬁnition 2.
Diversiﬁcation Distinguishability (DD) based on S ( I − ˆ A, X ) .
Given S ( I − ˆ A, X ) , a node v is diversiﬁcation distinguishable if the following two conditions aresatisﬁed at the same time,
[Equation]
Then, graph diversiﬁcation distinguishability value is deﬁned as
[Equation]
We can see that DD ˆ A,X ( G ) ∈ [0 , 1] .
Based on Def. 2 , the effectiveness of diversiﬁcation in addressingheterophily can be proved under certain conditions:
Theorem 1.
(See Appendix E for proof).
For C = 2 , suppose X
=
Z, ˆ A
= ˆ
A rw .
Then for any
I − ˆ A rw , all nodes are diversiﬁcation distinguishable and DD ˆ A,Z ( G ) = 1 .
We will now introduce the concept of ﬁlterbank which uses both LP and HP ﬁlters, combiningdiversiﬁcation and aggregation, to extract information from graph signals.
We then develop ACMframework in the following subsection.
4.2 Filterbank and Adaptive Channel Mixing (ACM) Framework
Filterbank For the graph signal x deﬁned on G , a 2-channel linear (analysis) ﬁlterbank [11] 7includes a pair of ﬁlters H LP , H HP , which retain the low-frequency and high-frequency content of x ,respectively.
Most existing GNNs use a uni-channel ﬁltering architecture [19, 40, 15] with either the
H LP or the H HP channel, which only partially preserves the input information.
Unlike the uni-channelarchitecture, ﬁlterbanks with H LP + H HP
=
I do not lose any information from the input signal,which is called the perfect reconstruction property [11].
Generally, the Laplacian matrices ( L sym , L rw ,ˆ L sym , ˆ L rw ) can be regarded as HP ﬁlters [11] and afﬁnity matrices ( A sym , A rw , ˆ A sym , ˆ A rw ) can betreated as LP ﬁlters [33, 14].
Moreover, we extend the concept of ﬁlterbank and view MLPs as usingthe identity ﬁlterbank with H LP
=
I and H HP = 0 , which also satisﬁes H LP + H HP
= I +
= I .
Node-wise Channel Mixing Figure 3 presents an example also showing that different nodes mayneed the local information extracted from different channels, e.g., nodes { 1 , 3 } demand informationfrom the HP channel while node 2 only needs information from the LP channel.
In order to adaptivelyleverage the LP, HP and identity channels in GNNs, we will now describe our proposed Adaptive
Channel Mixing (ACM) framework.
Adaptive Channel Mixing (ACM)
We will use GCN as an example to introduce the ACM framework in matrix form, but the framework can be combined in a similar manner to many different
GNNs.
The ACM framework includes the following steps:
Step 1.
Feature Extraction for Each Channel:
Option
[Equation]
(cid:1) ;
Option 2: H lL =
H LP ReLU (cid:0) H l − 1 W l − 1 L (cid:1) , H lH
[Equation]
H HP ReLU (cid:0) H l −
[Equation]
W
[Equation]
H
[Equation]
H lI =
I ReLU
[Equation]
H
[Equation]
−
[Equation]
W
[Equation]
− 1 I (cid:1)
[Equation]
H
[Equation]
X ∈ R N × F 0 , W l − 1 L , W l − 1 H , W l − 1 I ∈
R F l − 1 × F l
[Equation]
Step 2.
Row-wise Feature-based Weight Learning ˜
[Equation]
−
[Equation]
H
[Equation]
Step 3.
Node-wise Adaptive Channel Mixing:
[Equation]
We will refer to the instantiation which uses option 1 in step 1 as ACM and to the one using option 2in step 1 as ACMII.
In step 1, ACM-GCN and ACMII-GCN implement different feature extractionfor 3 channels using a set of ﬁlterbanks.
Three ﬁltered components, H lL , H lH , H lI , are obtained.
To adaptively exploit information from each channel, ACM-GCN and ACMII-GCN ﬁrst extractnonlinear information from the ﬁltered signals, then use W l Mix to learn which channel is important foreach node, leading to the row-wise weight vectors α lL , α lH , α lI ∈ R N × 1 whose i -th elements are theweights for node i .
These three vectors are then used as weights in deﬁning the updated H l in step 3.
Complexity
The number of learnable parameters in layer l of ACM-GCN and ACMII-GCN is3 F l − 1 ( F l +1)+9 , compared to F l − 1 F l in GCN.
The computation of steps 1-3 takes NF l (8+6 F l − 1 )
+2 F l ( nnz ( H LP ) + nnz
( H HP ))
+ 18 N ﬂops, while the GCN layer takes 2 NF l − 1 F l + 2 F l ( nnz ( H LP ))ﬂops, where nnz ( · ) is the number of non-zero elements.
An ablation study and a detailed comparisonon running time is conducted in Sec. 6.1.
[Figure]
Limitations of Diversiﬁcation Like any other method, the diversiﬁcation operation may not workwell in all cases of harmful heterophily.
For example, suppose we have an imbalanced dataset whereseveral small clusters with distinctive labels are densely connected to a large cluster.
In this case,the surrounding differences of nodes in small clusters are similar, i.e., the neighborhood differencesmainly come from their connections to the same large cluster, and this can lead to the diversiﬁcationoperation failing to discriminate them.
A more detailed discussion is included in Appendix G.
Related Work
We now discuss relevant work on addressing heterophily in GNNs.
[1] acknowledges the difﬁculty oflearning on graphs with weak homophily and propose MixHop to extract features from multi-hopneighborhoods to get more information.
[17] propose measurements based on feature smoothnessand label smoothness that are potentially helpful to guide GNNs when dealing with heterophilicgraphs.
Geom-GCN [35] precomputes unsupervised node embeddings and uses the graph structuredeﬁned by geometric relationships in the embedding space to deﬁne the bi-level aggregation processto handle heterophily.
H 2 GCN [45] combines 3 key designs to address heterophily: (1) ego- andneighbor-embedding separation; (2) higher-order neighborhoods; (3) combination of intermediaterepresentations.
CPGNN [44] models label correlations through a compatibility matrix, which isbeneﬁcial for heterophilic graphs, and propagates a prior belief estimation into the GNN by usingthe compatibility matrix.
FAGCN [4] learns edge-level aggregation weights as GAT [40] but allowsthe weights to be negative, which enables the network to capture high-frequency components in thegraph signals.
GPRGNN [8] uses learnable weights that can be both positive and negative for featurepropagation.
This allows GPRGNN to adapt to heterophilic graphs and to handle both high- andlow-frequency parts of the graph signals (See Appendix J for a more comprehensive comparisonbetween ACM-GNNs, ACMII-GNNs and FAGCN, GPRGNN).
BernNet [16] designs a scheme tolearn arbitrary graph spectral ﬁlters with Bernstein polynomial to address heterophily.
[32] points outthat homophily is not necessary for GNNs and characterizes conditions that GNNs can perform wellon heterophilic graphs.
Empirical Evaluation
In this section, we evaluate the two versions of the proposed framework, ACM and ACMII, onreal-world datasets (see Appendix F for a performance comparison with basline models on syntheticdatasets).
We ﬁrst conduct ablation studies in Sec.
6.1 to validate the effectiveness and efﬁciency ofdifferent components of ACM and ACMII.
Then, we compare with state-of-the-art (SOTA) models in
Sec. 6.2.
The hyperparameter search range and computing resources for all experiments are describedin
Appendix A.
6.1 Ablation Study & Efﬁciency
We will now investigate the effectiveness and efﬁciency of adding HP, identity channels and theadaptive mixing mechanism in the proposed framework by performing an ablation study.
Speciﬁcally,we apply the components of ACM to SGC-1 [41] 8 and the components of ACM and ACMII to
[Table]
GCN [19] separately.
We run 10 times on each of the 9 benchmark datatsets, Cornell , Wisconsin ,
Texas , Film , Chameleon , Squirrel , Cora , Citeseer and Pubmed used in [36, 35], with the same60%/20%/20% random splits for train/validation/test used in [8] and report the average test accuracyas well as the standard deviation.
We also record the average running time per epoch (in milliseconds)to compare the computational efﬁciency.
We set the temperature T in equation 4.2 to be 3 , which isthe number of channels.
The results in Table 1 show that on most datasets, the additional HP and identity channels are helpful,even for strong homophily datasets such as Cora, CiteSeer and PubMed .
The adaptive mixingmechanism also has an advantage over directly adding the three channels together.
This illustratesthe necessity of learning to customize the channel usage adaptively for different nodes.
The t-SNEvisualization in Figure 4 can demonstrate the effectiveness of each components in ACM-GCN andshow the advantages over GCN.
The running time is approximately doubled in the ACM and ACMIIframework compared to the original models.
6.2 Comparison with Baseline and SOTA Models
Datasets & Experimental Setup
In this section, we evaluate SGC [41] with 1 hop and 2 hops(SGC-1, SGC-2), GCNII [7], GCNII ∗ [7], GCN [19] and snowball networks [29] with 2 and 3 layers(snowball-2, snowball-3) and combine them with the ACM or ACMII framework 9 .
We use ˆ A rw as the
LP ﬁlter and the corresponding HP ﬁlter is I − ˆ A rw .
Both ﬁlters are deterministic.
We compare theseapproaches with several baselines and SOTA GNN models: MLP with 2 layers (MLP-2), GAT [40],
APPNP [20], GPRGNN [8], H 2 GCN [45], MixHop [1], GCN+JK [19, 42, 25], GAT+JK [40, 42, 25],
FAGCN [4] GraphSAGE [15] and Geom-GCN [35].In addition to the 9 benchmark datasets used insection 6.1, we further test the above models on a new benchmark dataset, Deezer-Europe [37].
On each dataset used in [36, 35], we test the models 10 times following the same early stoppingstrategy, the same 60%/20%/20% random data split 10 and Adam [18] optimizer as used in GPRGNN
[Figure]
For Deezer-Europe , we test the above models 5 times with the same early stopping strategy, thesame ﬁxed splits and AdamW [28] used in [25].
To visualize the performance, in Fig.
5, we plot the bar charts of the test accuracy of SOTA models,three selected baselines (GCN, snowball-2, snowball-3) and their ACM and ACMII augmentedmodels on the 6 most commonly used benchmark heterophily datasets (See Table 4 in Appendix
A.3 for the full results, comparison and ranking).
From Fig.
5, we can see that (1) after beingcombined with the ACM or ACMII framework, the performance of the three baseline models issigniﬁcantly boosted, by (2.04 % ∼ 27.50 %) on all the 6 tasks.
The ACM and ACMII in factachieve SOTA performance.
(2) On Cornell, Wisconsin, Texas and Squirrel , the augmented baselinemodels signiﬁcantly outperform the current SOTA models .
Overall, these results suggest that theproposed approach can help GNNs to generalize better on node classiﬁcation tasks on heterophilicgraphs, without adding too much computational cost.
Conclusions and Limitations
We have presented an analysis of existing homophily metrics and proposed a new metric which ismore informative in terms of correlating with GNN performance.
To our knowledge, this is the ﬁrstwork analyzing heterophily from the perspective of post-aggregation node similarity.
The similaritymatrix and the new metrics we deﬁned mainly capture linear feature-independent relationships of eachnode.
This might be insufﬁcient when nonlinearity and feature-dependent information is importantfor classiﬁcation.
In the future, it would be useful to investigate if a similarity matrix could be deﬁnedwhich is capable of capturing nonlinear and feature-dependent relations between aggregated node.
We have also proposed a 3-way ﬁlterbank which leverages the intuitions gained in the ﬁrst partof the paper and can be combined with different GNN architectures, enabling adaptive ﬁltering(high-pass, low-pass or identity) at different nodes.
Empirically, this approach shows very promisingresults, improving the performance of the base GNNs with which it is combined and achieving SOTAresults on several datasets, at the cost of a reasonable increase in computation time.
As discussed in
Sec. 4.2, however, the ﬁlterbank method cannot properly handle all cases of harmful heterophily, andalternative ideas should be explored as well in the future.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reﬂect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[Yes]
(b) Did you include complete proofs of all theoretical results?
[Yes]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes]
(c) Did you report error bars (e
g., with respect to the random seed after running experiments multiple times)?
[Yes]
(d) Did you include the total amount of compute and the type of resources used (e g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
If you are using existing assets (e
g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes]
(b) Did you mention the license of the assets?
[N/A]
(c) Did you include any new assets either in the supplemental material or as a URL?
[No]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[No]
(e) Did you discuss whether the data you are using/curating contains personally identiﬁable
information or offensive content?
[No]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Hyperparameters & Details of The Experiments
A.1 Hyperparameter Searching Range for Synthetic Experiments
Hyperparameter Searching Range for Synthetic Experiments
[Table] [Table]
A.2 Hyperparameter Searching Range for Ablation Study
[Table]
A.3 Full Results of The Comparison with SOTA Models
main results of the full sets of experiments 11 with statistics of datasets are summarized in Table4, where we report the mean accuracy and standard deviation.
We can see that after applied in ACMor ACMII framework, the performance of baseline models are boosted on almost all tasks and achieve
SOTA performance on 8 out of 10 datasets.
Especially, ACMII-GCN performs the best in terms ofaverage rank (3.40) across all datasets.
Overall, It suggests that ACM or ACMII framework cansigniﬁcantly increase the performance of GNNs on node classiﬁcation tasks on heterophilic graphs.
A.4 Hyperparameter Searching Range for GNNs on Real-world Datasets
See table 5 for the hyperparameter seaching range of baseline GNNs, ACM-GNNs, ACMII-GNNsand several SOTA models.
A.5 Optimal Hyperparameters for Baselines and ACM(II)-GNNs on Real-world Tasks
See the reported optimal hyperparameters for baseline GNNs in table 6 and for ACM-GNNs and
ACMII-GNNs in table 7.
A.6 Details of the Implementation of ACM and ACMII Frameworks
Unlike some baseline GNN models, in ACM(II) framework, we ﬁrst use dropout operation over theinput data.
The implementation of ACM(II)-GCN and ACM(II)-snowball is straightforward, but
[Table]
SGC 1, SGC-2, GCNII and GCNII* are not able to be applied under ACM(II) framework and wewill make an explanation as follows.
• SGC-1 and SGC-2: SGC does not contain nonlinearity, so the option 1 and option 2 in step
1 of equation 4.2 is the same for ACM-SGC and ACMII-SGC.
Thus, we only implement
ACM-SGC.
• GCNII and GCNII*:
[Equation]
[Equation]
) (cid:17)(cid:17)
GCNII*: H
[Equation]
−
[Equation]
AH
[Equation]
(cid:96)
[Equation]
(cid:16)
[Equation]
1 −
[Equation]
(cid:96)
[Equation]
I n
[Equation]
(cid:96) W ( (cid:96) ) 1 (cid:17) + + α (cid:96) H (0) (cid:16) (1 − β (cid:96) )
I n + β (cid:96) W ( (cid:96) ) 2 (cid:17)(cid:17)
From the above architecture of GCNII and GCNII ∗ we cam see that, without major modiﬁcation, GCNII and GCNII* are hard to be put into ACMII framework.
In ACMII framework,before apply ˆ A , we ﬁrst implement a nonlinear feature extractor σ ( H (cid:96) W ( (cid:96) ) ) .
But in GCNIIand GCNII*, before multiplying W (cid:96) ( or W (cid:96) 1 , W (cid:96) 2 ) to extract features, we need to add anotherterm including H (0) , which are not ﬁltered by ˆ A .
This makes the order of aggregator ˆ A andnonlinear extractor unexchangable and thus, incompatible with ACMII framework.
So wedid not implement GCNII and GCNII* in ACMII framework.
Besides the features extracted by different ﬁlters, some recent SOTA models use additional graphstructure information explicitly, i.e., MLP θ ( A ) , to address heterophily problem, e.g., LINKX [25]and GloGNN [24] and is found effective on some datasets, e.g., Chameleon, Squirrel .
The explicit
[Table]
Searching Range for Real-world Datasets structure information can be directly incorporated into ACM and ACMII framework, and we have
ACM(II)-GCN+ and ACM(II)-GCN++ as follows.
• ACM-GCN+ and ACMII-GCN+ have an option to include structure information channel(the 4-th channel) in each layer and their differences from ACM-GCN and ACMII-GCN arehighlighted in red) as
follows Step
Feature Extraction for LP, HP, Identity and Structure Information
Channel: H lA = ReLU (cid:0) AW lA (cid:1) , W lA [Equation] R N × F l , get H lL , H lH , H lI with the same step as ACM-GCN and ACMII-GCN.
Step 2.
Row-wise Feature-based Weight Learning with Layer Normalization (LN ) ˜ [Equation] lI )
[Equation]
H lA
[Equation]
LN
[Equation] H lA ) [Equation] ˜ α lL =
Sigmoid (cid:16) ˜ H lL ˜ W lL (cid:17)
[Equation]
˜
α lH
[Equation]
Sigmoid (cid:16) ˜ H lH
[Equation]
W lH (cid:17)
[Equation]
˜
[Equation]
lI
[Equation]
Sigmoid (cid:16)
[Equation]
H
[Equation]
W
[Equation]
(cid:17)
[Equation]
˜ α lA
[Equation]
Sigmoid (cid:16) ˜
H lA ˜ W lA (cid:17)
[Equation]
˜ W l − 1 L
[Equation]
˜ W l − 1 H , ˜ W l − 1 I
[Equation]
˜ W lA ∈ R F l ×
Step Node-wise Adaptive Channel Mixing
Option 1: without structure information
[Equation]
H
[Equation]
diag [Equation] α lL ) H lL + diag
[Equation]
α lH ) H lH + diag
α lI ) H lI (cid:1) Option 2: with structure information (cid:2)
[Equation]
= Softmax
[Equation]
lH
[Equation]
˜ α lA
[Equation]
T ) W l Mix
[Equation]
∈ R N × 4 , T = 4 temperature , W l Mix ∈ R 4 ×
[Equation]
H l = ReLU (cid:0) diag ( α lL )
H lL + diag ( α lH ) H lH + diag ( α lI ) H lI + diag ( α lA ) H lA [Equation] • ACM-GCN++ and ACMII-GCN++ have an option to include structure information channel
(the 4-th channel) in each layer and residual connection and their differences from ACM-
GCN+ and ACMII-GCN+ are highlighted in red) as follows,
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
Cora
[Figure]
[Table] [Table]
[Table]
[Table]
[Table]
Squirrel
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
GNNs and ACMII-GNNs
Step 1.
Feature Extraction for LP, HP, Identity and Structure Information Channel, Get H X :
[Equation]
lA
[Equation] [Equation]
lA
[Equation]
N
[Equation]
F
[Equation]
get H lL , H lH , H lI with the same step as ACM-GCN and ACMII-GCN.
Step 2.
Row-wise Feature-based Weight Learning with Layer Normalization (LN)
[Equation]
Step 3.
Mixing:
Option 1: without structure information
[Equation]
Option 2: with structure information
[Equation]
, ˜ α lA (cid:3) /T )
W l Mix (cid:1) ∈
R N × 4 , T = 4 temperature , W l Mix ∈ R 4 × 4 ;
H l = ReLU (cid:0) diag ( α l L ) H lL + diag ( α l H )
H lH + diag ( α l I ) H lI + diag ( α l A ) H lA (cid:1) + H X
The results of ACM-GCN+, ACMII-GCN+, ACM-GCN++ and ACMII-GCN++ trained on ﬁxedsplits are reported in table 11 in Appendix I.The open source code is attached in the supplementarymaterial.
A.7 Comparison of Random Walk and Symmetric Renormalized FiltersDatasets/Models
[Table]
[Table]
75 ± 3.
95.9 ± 1.
94.92 ± 2.
94.1 ±
[Table]
[Table]
[Table]
75 ±
[Table]
96.62 ± 2.
95.63 ± 2.
96.25 ± 2.
[Table]
[Table]
[Table]
95.08 ± 2.
94.75 ± 2.
94.59 ± 2.
[Table]
[Table]
[Table]
[Table]
41.84 ± 1.
41.58 ± 1.
41.65 ± 0.
[Table]
[Table]
04 ±
[Table]
[Table]
38 ±
[Table]
67.9 ± 2.
68.03 ± 1.
[Table]
[Table]
02 ± 1.
54.53 ± 2.
[Table]
18 ±
[Table]
53.68 ± 1.
[Table]
[Table]
[Table]
[Table]
[Table]
00 ±
[Table]
[Table]
65 ± 1.
88.19 ±
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
The deﬁnitions of the similarity matrix, (modiﬁed) aggregation similarity score and diversiﬁcationdistinguishability value can be extended to symmetric normalized Laplacian or other aggregationoperations.
Yet unfortunately, we cannot extend Theorem 1 at this moment, because we need acondition that the row sum of ˆ A is not greater than 1 in the proof.
This condition is guaranteed forrandom walk normalized Laplacian but not for symmetric normalized Laplacian.
We will investigatethis more to follow up on this line of research.
While in practice, the answer is probably yes.
We evaluate our models with symmetric ﬁlters andcompare them with random walk ﬁlters.
From table 8 we can see that, there are no big differencesbetween these two ﬁlters.
A.8 Ablation Study of W mix
From table 9 we can see that ACM with W mix shows superiority in most datasets, although it is notstatistically signiﬁcant on some of them.
One possible explanation of the advantage is that W mix could help alleviate the dominance and biasto majority: Suppose in a dataset, most of the nodes need more information from LP channel than HP
Datasets/Models
[Table]
and identity channels, then W L , W H , W I tend to learn larger α L than α H and α I .
For the minoritynodes that need more information from HP or identity channels, they are hard to get large α H or α Ivalues because W L , W H , W I are biased to the majority.
And W mix can help us to learn more diversealpha values when W L , W H , W I are biased.
Attention with more complicated design can be found for the node-wise adaptive channel mixingmechanism, but we do not explore this direction deeper in this paper because investigating attentionfunction is not the main contribution of our paper.
A.9 Computing Resources
For all experiments on synthetic datasets and real-world datasets, we use NVidia V100 GPUs with16/32GB GPU memory, 8-core CPU, 16G Memory.
The software implementation is based on
PyTorch and PyTorch Geometric [12].
B Experimental Setup and Further Discussion on Synthetic Graphs
B.1 Detailed Description of Data Generation Process
• For each node v , we ﬁrst randomly generate its degree d v .
• Given d v , for any h , we sample hd v intra-class edges and (1 − h )
d v inter-class edges.
More speciﬁcally in our synthetic
experiments, for a given
h
• we generate node degree d v for nodes in each class from multinomial distribution with numpy.random.multinomial(800/h, numpy.ones(400)/400, size=1)[0] .
• For a sampled d v , we generate intra-class edges from (does not include self-loop)
numpy.random.multinomial(h d v , numpy.ones(399)/399, size=1)[0] and inter- class edges from numpy.random.multinomial((1-h)
d v , numpy.ones(1600)/1600,size=1)[0] .
For each generated graph, we calculate their H node , H class , H M agg .
Then, we reorder the value of themetrics in ascend order for x-axis and plot the corresponding test accuracy.
Here is a simpliﬁed example.
Suppose we generate 3 graphs with H edge = 0 .
9 , the test accuracy of GCN on these 3 synthetic graphs are 0 . 8 , 0 .
For the generated graphs, we calculate their H M agg , and suppose we get H M agg = 0 . Then we will draw the performance of GCN under
H M agg with ascend
B.2 Further Discussion of Aggregation Homophily on Regular Graphs
We notice that in Figure 2(a), the performance of SGC-1 and GCN both have a turning point, i.e.,when H edge ( G ) is smaller than a certain value, the performance will get better instead of getting worse.
With some extra restriction on node degree in data generation process, we ﬁnd that this interestingphenomenon can be theoretically explained by the following proposition 1 based on our proposedsimilarity matrix which can verify the usefulness of H M agg ( G ) .
We ﬁrst generate regular graphs , i.e.,each node has the same degree, as follows,
Generate Synthetic Regular Graphs We ﬁrst generate 180 graphs in total with 18 edge homophilylevels varied from 0.05 to 0.9, each corresponding to 10 graphs.
For every generated graph, we have5 classes with 400 nodes in each class.
For each node, we randomly generate 10 intra-class edgesand [ 10 H edge ( G ) − 10 ] inter-class edges.
The features of nodes in each class are sampled from nodefeatures in the corresponding class of the base dataset.
Nodes are randomly split into 60%/20%/20%for train/validation/test.
We train 1-hop SGC ( sgc-1 ) [41] and GCN [19] on synthetic data (see
Appendix A.1 for hyperparameter searching range).
For each value of H edge ( G ) , we take the averagetest accuracy and standard deviation of runs over 10 generated graphs.
We plot the performancecurves in Figure 6.
From Figure 6 we can see that the turning point is a bit less than 0.2.
We derive the followingproposition for d -regular graph to explain and predict it.
Proposition 1.
(See Appendix D for proof).
Suppose there are C classes in the graph G and G is ad -regular graph (each node has d neighbors).
Given d , edges for each node are i.i.d.
generated, suchthat each edge of any node has probability h to connect with nodes in the same class and probability1 − h to connect with nodes in different classes.
Let the aggregation operator ˆ
A
=
ˆ
A rw .
Then, fornodes v , u 1 and u 2 , where Z u 1 , : =
Z v, : and Z u 2 , : (cid:54) =
Z v, : , we haveg (
[Equation]
and the minimum of g ( h ) is reached at
[Equation]
where d intra = dh , which is the expected number of neighbors of a node that have the same label asthe node.
The value of g ( h ) in equation 12 is the expected differences of the similarity values between nodesin the same class as v and nodes in other classes.
g ( h ) is strongly related to the deﬁnition ofaggregation homophily and its minimum potentially implies the turning point of performance curves.
In the synthetic experiments, we have d intra = 10 , C = 5 and the minimum of g ( h ) is reached ath = 5 / 27 ≈ 0 .
1852 , which corresponds to the lowest point in the performance curve in Figure 6.
Inother words, the H edge ( G ) where SGC-1 and GCN perform worst is where g ( h ) gets the smallestvalue, instead of the point with the smallest edge homophily value H edge ( G ) = 0 .
This reveals theadvantage of H agg ( G ) over H edge ( G ) by taking use of the similarity matrix.
C Details of Gradient Calculation in equation 6C.1 Derivation in Matrix Form
This derivation is similar to [30].
In output layer, we have
[Equation]
L = − trace( Z T log Y ) where 1 C ∈ R C × 1 , ( · ) − 1 is point-wise inverse function and each element of Y is positive.
Then
[Equation]
T
[Equation]
Note that
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]
[Equation]
Then,
[Equation]
(cid:19)(cid:21)(cid:19)(cid:33)
[Equation]
where the 4-th equation holds due to (cid:16) Z (cid:12) (cid:0)
exp( Y [Equation]
[Equation]
= (cid:0) exp ( Y (cid:48) )
[Equation]
1 TC (cid:1) − 1 .
Thus,we have
[Equation]
For Y (cid:48) and W , we have
[Equation]
= ˆ AXdW and d L = trace
[Equation]
To get d L dW we have,
[Equation]
Denote ˜ X
=
XW .
We rewrite L as follows:
L = − trace (cid:0) Z T log (cid:0) (exp( Y (cid:48) ) 1 C 1 TC ) − 1 (cid:12) exp( Y (cid:48) ) (cid:1)(cid:1) = − trace (cid:0) Z T (cid:0) − log(exp ( Y (cid:48) )
1 C 1 TC )
+ Y (cid:48)
(cid:1)(cid:1)
= − trace (cid:0) Z T Y (cid:48) (cid:1) + trace (cid:0) Z T log (cid:0) exp( Y (cid:48) )
1 C 1 TC (cid:1)(cid:1)
= − trace (cid:16) Z T
ˆ AXW (cid:17) + trace (cid:0) Z T log (cid:0) exp
( Y (cid:48) )
1 C 1 TC (cid:1)(cid:1)
= − trace (cid:16) Z T
ˆ AXW (cid:17) + trace (cid:0) 1 TC log (exp( Y (cid:48) ) 1 C ) (cid:1)
=
−
N (cid:88) i = (cid:88)
j ∈N i
ˆ
A
i,j Z i, : ˜ X Tj : +
N (cid:88) i = log  
C (cid:88) c
= exp( (cid:88)
j ∈N i
ˆ
A
i,j ˜ X j,c ) 
 = −
N (cid:88) i = log   exp  
C (cid:88) c
= (cid:88)
j ∈N i
ˆ
A
i,j Z i,c ˜ X j,c   
 +
N (cid:88) i = log  
C (cid:88) c
= exp  
[Equation]
Note that C (cid:80) c
=1 Z j,c = 1 for any j .
Consider the derivation of L over ˜ X j (cid:48) ,c (cid:48) :
[Equation]
[Table]
[Equation]
[Table]
[Equation]
[Table]

[Table]
[Equation]

[Equation]

[Equation]
Writing the above in matrix form, we have
[Equation]
D Proof of Proposition
Proof.
According to the given assumptions, for node v , we have ˆ A v,k =
d
+1 , the expected number ofintra-class edges is dh (here the self-loop edge introduced by ˆ A is not counted based on the deﬁnitionof edge homophily and data generation process) and inter-class edges is (1 − h ) d .
Suppose there are
C ≥ 2 classes.
Consider matrix ˆ AZ ,
Then, we have E (cid:104) ( ˆ AZ ) v,c (cid:105) =
E (cid:20) (cid:80)
[Equation]
d
[Equation]
, where 1 is the indicator function.
When v is in class c , we have (cid:80) k ∈V
[Equation]
[Equation]
eTc
[Equation]
d + = hd
[Equation]
1 d
[Equation]
( hd + [Equation] =
hd intra-class edges + 1 self-loopintroduced by ˆ A ).
When v is not in class c , we have (cid:80)
[Equation] [Equation]
1 − h ) d inter-class edgesuniformly distributed in the other C − 1 classes).
For nodes v, u , we have ( ˆ AZ ) v, : , ( ˆ AZ ) u, : ∈ R C and since elements in ˆ A v,k and ˆ A u,k (cid:48) are independently generated for all k, k (cid:48) ∈ V , we have
[Equation]
Thus,
E (cid:104) S
( ˆ A, Z ) v,u (cid:105) =
E (cid:104) <
( ˆ AZ ) v, : , ( ˆ AZ ) u, : > (cid:105) =
[Equation]
u, v are in the same class
[Equation] [Equation]
u
[Equation]
v are in different classes
For nodes u 1 , u 2 , and v , where Z u 1 , : = Z v, : and Z u 2 , : (cid:54) =
Z v, : ,g
[Equation]
[Equation]
Setting g
( h )
= 0 , we obtain the optimal h :
[Equation]
Cd
[Equation]
For the data generation process in the synthetic experiments, we ﬁx d intra , then d = d intra /h , which isa function of h .
We change d in equation 16 to d intra /h , leading to
[Equation]
It is easy to observe that h satisfying equation 17 still makes g ( h ) = 0 , when d in g ( h ) is replaced by d intra /h .
From equation 17 we obtain the optimal h in terms of d intra : h = d intra
Cd
[Equation]
D.1 An Extension of Proposition 1
Base on the deﬁnition of aggregation similarity [Equation] we have
[Equation]
|V|
Then,
[Equation]
Consider the random variable
[Equation]
Since RV is symmetrically distributed and under the conditions in proposition 1, its expectation is 627 E [ RV ] = g ( h ) as showed in equation 15.
Since the minimum of g ( h ) is 0 and RV is symmetricallydistributed, we have P ( RV ≥ 0) ≥ 0 .
5 and this can explain why H agg ( G ) is always greater than 0.5in many real-world tasks.
E Proof of Theorem 1
Proof.
Deﬁne W cv =
( ˆ AZ ) v,c .
Then,
[Equation]
Note that
[Equation]
For any node v , let the class v belongs to be denoted by c v .
For two nodes v, u , if Z v, : (cid:54) =
Z u, : , we 634 have
[Equation]
Then, from equation 18 it follows that
[Equation]
When C = 2 ,
[Equation]
If Z v, : = Z u, : , i.e., c v = c u , we have
[Equation]
Then, from equation 18 it follows that
[Equation]
c =
W cv
[Equation]
c v u =
C (cid:88) c =1 ,c (cid:54) =
c v
W cv W cu + 1 + W c v v W c v u − W c v v − W c v u =
C (cid:88) c
=1 ,c (cid:54)
[Equation]
c v
W
[Equation]
c
[Equation]
c v
[Equation]
Thus, if C = 2 , for any v ∈ V , if Z u, : (cid:54) =
Z v, : , we have S ( I − ˆ A, Z ) v,u ≤ 0 ; if Z u, : =
Z v, : , wehave S ( I − ˆ A, Z ) v,u ≥ 0 .
Apparently, the two conditions in equation 10 are satisﬁed.
Thus v isdiversiﬁcation distinguishable and DD ˆ A,X ( G ) = 1 .
The theorem is proved.
F Model Comparison on Synthetic Graphs
[Figure]
[Figure]
In order to separate the effects of nonlinearity and graph structure, we compare sgc with 1 hop(sgc-1) with MLP-1 (linear model).
For GCN which includes nonlinearity, we use MLP-2 as itscorresponding graph-agnostic baseline model.
We train the above GNN models, graph-agnosticbaseline models and ACM-GNN models on all synthetic datasets and plot the mean test accuracywith standard deviation on each dataset.
From Figure 7 and Figure 8, we can see that on each H M agg ( G )level, ACM-GNNs will not underperform baseline GNNs and the graph-agnostic models.
But when
H M agg ( G ) is small, baseline GNNs will be outperformed by graph-agnostic models by a large margin.
This demonstrate the advantage of the ACM framework.
The curves will almost be the same for
ACMII framework.
G Discussion of the Limitations of Diversiﬁcation Operation
[Figure]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Table] [Table]
[Equation]
[Table]
[Figure]
From the black box area of S ( I − ˆ A, X ) in the example in Figure 9 we can see that nodes in class 1 and 4 assign non-negative weights to each other although there is no edge between them; nodesin class 2 and 3 assign non-negative weights to each other as well.
This is because the surroundingdifferences of class 1 are similar as class 4, so are class 2 and 3.
In real-world applications, when nodes in several small clusters connect to a large cluster, the surrounding differences of the nodes inthe small clusters will become similar.
In such case, HP ﬁlter are not able to distinguish the nodesfrom different small clusters.
H The Similarity, Homophily and DD ˆ A,X ( G ) Metrics and Their Estimations
Firstly, we would like to clarify that, for each curve in the synthetic experiments, the node featuresare ﬁxed and we only change the homophily values. But in real-world tasks, different datasets havedifferent features and aggregated features.
Thus, to get more instructive information for differentdatasets and compare them, we need to consider more metrics, e.g. feature-label consistency andaggregated-feature-label consistency.
With the similarity score of the features S agg ( S ( I, X )) andaggregated features S agg (cid:16) S ( ˆ A, X ) (cid:17) listed in Table 10, our methods open up a new perspective onanalyzing and comparing the performance of graph-agnostic models and graph-aware models inreal-world tasks.
Here are 2 examples.
Example 1: People observe that GCN (graph-aware model) underperforms MLP-2 (graph-agnosticmodel) on Cornell, Wisconsin, Texas, Film and people commonly believe that the bad graph structure(low H edge , H node , H class values) is the reason for performance degradation.
But based on the highaggregation homophily values, the graph structure inconsistency is not the main cause of the performance degradation.
And from Table 10 we can see that the S agg (cid:16) S ( ˆ A, X ) (cid:17) for those 4 datasets arelower than their corresponding S agg ( S ( I, X )) , which implies that it is the aggregated-feature-labelinconsistency that causes the performance degradation, i.e. the aggregation step actually decrease thequality of node features rather than making them more distinguishable.
For the rest 5 datasets Chameleon, Squirrel, Cora, Citeseer, PubMed , we all have S agg (cid:16) S ( ˆ A, X ) (cid:17)larger than S agg ( S ( I, X )) except PubMed , which means the aggregated features have higher qualitythan raw features. We can see that the proposed metrics are much more instructive than the existingones.
Example 2:
According to H edge , H node , H class , the value for Chameleon , and Squirrel are extremelylow indicating graph structure are bad for GNNs.
But on contrary, GCN outperforms MLP-2 onthose 2 datasets.
Traditional homophily metrics fail to explain such phenomenon but our methodcan give an explanation from different angles: For Chameleon, its modiﬁed aggregation homophilyis not low and its S agg (cid:16) S ( ˆ A, X ) (cid:17) is higher than its S agg ( S ( I, X )) , which means its graph-labelconsistency together with aggregated-feature-label consistency help the graph-aware model obtainthe performance gain; for Squirrel, its modiﬁed aggregation homophily is low but its S agg (cid:16) S ( ˆ A, X ) (cid:17)is higher than its S agg ( S ( I, X )) , which means although its graph-label consistency is bad, theaggregated-feature-label consistency is the key factor to help the graph-aware model perform better.
We also need to point out that (modiﬁed) aggregation similarity score, S agg (cid:16) S ( ˆ A, X ) (cid:17) and
S agg ( S ( I, X )) are not deciding values because they do not consider the nonlinear structure inthe features.
In practice, a low score does not tell us the GNN models will deﬁnitely perform bad. Cornell Wisconsin Texas
Film
Chameleon
Squirrel Cora CiteSeer PubMed
H agg ( G ) 0. [Table] Furthermore, in most real-world applications, not all labels are available to calculate the datasetstatistics.
In this section, We randomly split the data into 60%/20%/20% for training/validation/test,and only use the training labels for the estimation of the statistics.
We repeat each estimation for 10times and report the mean with standard deviation.
The results are shown in table 10.
Analysis From the reported results we can see that the estimations are accurate and the errors arewithin the acceptable range, which means the proposed metrics and similarity scores can be accuratelyestimated with a subset of labels and this is important for real-world applications.
I Experiments on Fixed Splits Provided by Geom-GCN [35]I.1 Comparison with SOTA Models
[Table]
See table 11 for the results and table 12 the optimal searched hyperparameters.
[Table]
I.2 Distribution of α L , α H , α I in the Output Layer Trained on Different Datasets
See table
[Table]
Alpha values
D e n s i t y
Alpha values
L
H
I (a) Cornell 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (b) Wisconsin 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (c) Texas 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (d) Chameleon 0.25 0.50 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (e) Squirrel 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (f) Film 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (g) Cora 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (h) Citeseer 0.
Alpha values
D e n s i t y
Alpha values
L
H
I (i) Pubmed
J A Detailed Explanation of the Differences Between ACM(II)-GNNs andGPRGNN, FAGCN
with GPRGNN [8]: • GPRGNN does not feed distinct node-wise feature transformation to
We ﬁrst rewrite GPRGNN as Z =
K (cid:88) k
= γ k H ( k ) =
K (cid:88) k
[-=
-]= γ k I H ( k ) =
K (cid:88) k
= diag ( γ k , γ k , . . .
, γ k ) H ( k ) , where H ( k ) = ˆ A sym H ( k − 1) , H (0) i : = f θ ( X i : ) .
From the above equation we can see that Z
[Equation] [Equation] ., thenode-wise feature transformation in GPRGNN is only learned by the same θ for allthe "multi-scale channels". But in the ACM framework, different channels extract distinctinformation with different parameters separately.
• GPRGNN does not have node-wise mixing mechanism. There is no node-wise mixing in GPRGNN. The mixing mechanism in GPRGNN is Z = K (cid:80) k
=0 diag ( γ k , γ k , . . .
, γ k ) H ( k ) , i.e. for each "multi-scale channel k ", all nodes share the
same mixing parameter γ k . But in the ACM framework, the node-wise channel mixing canbe written as Z = K (cid:80) k =0 diag ( γ 1 k , γ 2 k , . . γ Nk ) H ( k ) where K is the number of channels, N is
the number of nodes and γ ik , i = 1 , . .
, N are the mixing weights that are learned by node ito mix channel k . ACM and ACMII allow GNNs to learn more diverse mixing parameters
in diagonal than GPRGNN and thus, have stronger expressive power than GPRGNN.
Differences with FAGCN [4]: • The targets of node-wise operations in ACM (channel mixing) and FAGCN (negative message passing) are different. Instead of using a ﬁxed low-pass ﬁlter ˆ A , FAGCN tries to learn a more powerful aggregator ˆ A (cid:48) based on ˆ A by allowing negative message passing.
Thenode-wise operation in FAGCN is similar to GAT [3] which is trying to mod- ify the node-wise ﬁltering
(message passing) process , i.
e
for each node i , it assigns different weights α ij ∈ [ − 1 , 1] to different neighborhood nodes (equation 7 in FAGCN paper). The goal of this node-wise operation in FAGCN isto learn a new ﬁlter during the ﬁltering process node-wisely . But in ACM, the nodewise operation is to mix the ﬁltered information from each channel which is processedby different ﬁxed ﬁlters. The targets of two the node-wise operations are
actually different
things. • FAGCN does not learn distinct information from different "channels". FAGCN only uses simple addition to mix information instead of node-wise channel mixing mechanism
The learned ﬁlter ˆ A (cid:48) can be decomposed as ˆ A (cid:48)
=
ˆ A (cid:48) {+
 +(+} − ˆ A (cid:48) 2 ) , where ˆ A (cid:48) 1 and − ˆ A (cid:48) 2 represent positive and negative edge (propagation) information, respectively.
But FAGCN does not feed distinct information to ˆ A (cid:48) 1 and − ˆ A (cid:48) 2 . Moreover, the aggregated ˆ A (cid:48) 1 X and "diversiﬁed"information ( − ˆ A (cid:48) 2 ) X are simply added together instead of using any node-wise mixing mechanism.
In ACM, we learn distinct information separately in each channel with different parameters and add them adaptively and node-wisely instead of just adding them together. In section 6.1, the ablation study has empirically shown that node-wise adaptive channel mixing is better than simple addition.