Simplified Graph Convolution with Heterophily
Abstract
Recent work has shown that a simple, fast method called Simple Graph Convolution (SGC) (Wu et al., 2019), which eschews deep learning, is competitive with deep methods like graph convolutional networks (GCNs)
(Kipf & Welling, 2017) in common graph machine learning benchmarks.
The use of graph data in SGC implicitly assumes the common but not universal graph characteristic of homophily, wherein nodes link to nodes which are similar.
Here we confirm that SGC is indeed ineffective for heterophilous (i.e., non-homophilous) graphs via experiments on synthetic and real-world datasets.
We propose Adaptive Simple Graph Convolution (ASGC), which we show can adapt to both homophilous and heterophilous graph structure.
Like SGC, ASGC is not a deep model, and hence is fast, scalable, and interpretable; further, we can prove performance guarantees on natural synthetic data models.
Empirically, ASGC is often competitive with recent deep models at node classification on a benchmark of real-world datasets.
The SGC paper questioned whether the complexity of graph neural networks is warranted for common graph problems involving homophilous networks; our results similarly suggest that, while deep learning often achieves the highest performance, heterophilous structure alone does not necessitate these more involved methods.
Introduction
Data involving relationships between entities arises in biology, sociology, and many other fields, and such data is often best expressed as a graph.
Therefore, models of graph data that yield algorithms for common graph machine learning tasks, like node classification, have wide-reaching impact.
Deep learning (LeCun et al., 2015) has enjoyed great success in modeling image and text data, and graph convolutional networks (GCNs)
(Kipf & Welling, 2017) attempt to extend this success to graph data.
Various deep models branch off from GCNs, offering additional speed, accuracy, or other features.
However, like other deep models, these algorithms involve repeated nonlinear transformations of inputs and are therefore time and memory intensive to train.
Recent work has shown that a much simpler model, simple graph convolution (SGC) (Wu et al., 2019), is competitive with GCNs in common graph machine learning benchmarks.
SGC is much faster than GCNs, partly because the role of the graph in SGC is restricted to a feature extraction step in which node features are smoothed across the graph; by contrast, GCNs include graph convolution steps as part of an end-to-end model, resulting in expensive backpropagation calculations.
However, the feature smoothing operation in SGC implicity assumes the common but not universal graph characteristic of homophily, wherein nodes mostly link to similar nodes; indeed, recent work (NT & Maehara, 2019) suggests that many GCNs may assume such structure.
We ask whether a feature extraction approach, that, like SGC, is free of deep learning, can tackle heterophilous (i.e., non-homophilous) graph structure.
We see this work as extending the following broader research question introduced by the SGC paper to a wider range of graphs:
Are nonlinearities, end-to-end backpropagation, and other characteristics of deep learning essential to effective learning on graphs?
Contributions We confirm that SGC, which uses a fixed smoothing filter, can indeed be ineffective for heterophilous features via experiments on synthetic and real-world datasets.
We propose adaptive simple graph convolution (ASGC), which fits a different filter for each feature.
These filters can be smoothing or non-smoothing, and thus can adapt to both homophilous and heterophilous structures.
Like SGC, ASGC is not a deep model, instead being based on linear least squares, and hence is fast, scalable, and interpretable.
We propose a natural synthetic model for networks with a node feature,
Featured Stochastic Block Models (FSBMs), and prove that ASGC denoises the network feature regardless of whether the model is set to produce homophilous or heterophilous networks, in contrast to SGC, which we show is inappropriate for the heterophilous setting.
Finally, we show that the performance of ASGC is superior to that of SGC at node classification on real-world heterophilous networks, and generally competitive with recent deep methods on a benchmark of both heterophilous and homophilous networks.
The SGC paper suggested that deep learning is not necessarily required for good performance on common graph learning tasks and benchmarks involving homophilous networks; our results suggest that simple methods can also be competitive for heterophilous networks.
Background
Preliminaries We first establish common notation for working with graphs.
An undirected, possibly weighted graph on n nodes can be represented by its symmetric adjacency matrix A ∈ R n × n ≥ 0 .
Letting 1 denote an all-ones column vector, d
= A 1 is the vector of degrees of the nodes; the degree matrix
D is the diagonal matrix with d along the diagonal.
The normalized adjacency matrix is given by
S = D − 1 / 2 AD − 1 / 2 , while the symmetric normalized graph Laplacian is L =
I − S , where I is an identity matrix.
Note that the eigenvectors of L are exactly the eigenvectors of S .
It is a well known fact that the eigenvalues of S are contained within [ − 1 , +1] .
It follows that the eigenvalues of L are within [0 , 2] , so L is positive
semidefinite
Consider a feature x ∈ R n on the graph, that is, a real-valued vector where each entry is associated with a node.
The quadratic form over L is known to have the following equivalency:
[Equation]
Up to a reweighting based on the nodes’ degrees, this expression is the sum of squared differences of the feature’s values between adjacent nodes.
Hence, the quadratic form has a low value, near 0 , if the feature x is ‘smooth’ across the graph, that is, if adjacent nodes have generally similar values of the feature.
Similarly, the quadratic has a high value if the feature is ‘rough’ across the graph, if adjacent nodes have generally differing values of the feature.
When x is an eigenvector of L , these ‘smooth’ and ‘rough’ cases correspond to the eigenvalue being low or high, respectively.
(In terms of eigenvalues of S , the opposite is true, with positive eigenvalues being smooth and negative eigenvalues being rough.)
More generally, decomposition of an arbitrary feature vector x as a linear combination of the eigenvectors of L separates x into components ranging from ‘smooth’ to ‘rough’ across the graph.
In the graph signal processing literature (Ortega et al., 2018;
Huang et al., 2016;
Stankovi´c et al., 2019), these smooth and rough components are also called low and high frequency ‘modes,’ respectively, based on the eigenvalues of L .
Simple graph convolution Our work is primarily inspired by the simple graph convolution (SGC) algorithm (Wu et al., 2019).
SGC comprises logistic regression after the following feature extraction step, which can be interpreted as smoothing the nodes’ features across the graph’s edges:
[Equation]
This equation shows how a single raw feature x is filtered to produce the smoothed feature x SGC .
Here ˜ S ∈ R n × n is the normalized adjacency matrix after addition of a self-loop to each node (that is, addition of the identity matrix to the adjacency matrix), and
K ∈ Z + is a hyperparameter; K determines the radius of the filter, that is, the maximum number of hops between two nodes whose features can directly influence others’ features in the filtering process.
Wu et al.
(2019) show that the addition of the self-loop increases the minimum eigenvalue of S ; intuitively, the self-loop limits the extent to which a feature can be ‘rough’ across the graph.
This results in the highest magnitude eigenvalues of the normalized adjacency ˜ S tending to be positive (smooth).
Because the eigenvalues of ˜ S all have magnitude at most 1 , powering up ˜ S results in a filter which generally attenuates the feature x , but does so least along these high magnitude, smooth eigenvectors.
Hence the SGC filter smooths out the feature locally along the edges.
Since it attenuates the high-frequency modes more than the low-frequency ones, SGC is described as a ‘low-pass’ filter.
Heterophily If node features are used for node classification or regression, smoothing the features of nodes along edges encourages similar predictions along locally connected nodes.
This seems sensible when locally connected nodes should be generally similar in terms of features and labels; if the variance in features and labels between connected nodes is generally attributable to noise, then this smoothing procedure acts as a useful denoising step.
Graphs in which connected nodes tend to be similar are called homophilous or assortative.
An example would be a citation network of papers on various topics: papers concerning the same topic tend to cite each other.
Much of the existing work on graph models has an underlying assumption of network homophily, and there has been significant recent interest (discussed further in Section 6) on the limitations of graph models at addressing network heterophily/disassortativity, wherein connections tend to be between dissimilar nodes.
An example would be a network based on adjacencies of words in text, where the labels are based on the part of speech: adjacent words tend to be of different parts of speech.
For disassortative networks, smoothing a feature across connections as in SGC may not be sensible, since encouraging predictions of connected nodes to be similar is contrary to disassortativity.
Methodology
Adaptive SGC In our method, we replace the fixed feature propagation step of SGC (Equation 1) with an adaptive one, which may or may not be smoothing based on the feature and graph.
We produce a filtered version x ASGC of the raw feature x by multiplying x with a learned polynomial of the normalized adjacency matrix S ; this polynomial is set so that x ASGC ≈ x :
[Equation]
where K is a hyperparameter which, as in SGC, controls the radius of feature propagation, and the coefficients β ∈ R K +1 are learned by minimizing the approximation error in a least squares sense.
Note that, unlike SGC, we do not add self-loops to S .
This approximation error would be trivially minimized with β 0 = 1 and all other coefficients set to zero, resulting in x ASGC =
S 0 x = x , so we regularize the magnitude of β 0 .
More concretely, let T ∈ R n × ( K +1) denote the Krylov matrix generated by multiplying a feature vector x ∈ R n by the normalized adjacency matrix S up to some K ∈ Z > 0 times:
[Equation]
Here the leftmost column of T is just the raw feature x , and each column represents the feature generated by propagating the feature vector to its left across the graph, i.e., by multiplying once by S .
We produce a filtered version x ASGC of the raw feature x by linear combination of the columns of
T .
That is, x ASGC =
T β , and we set the combination coefficients β ∈ R k +1 by minimizing a loss function as follows:
[Equation]
The term on the right is L 2 regularization applied to the zeroth combination coefficient β 0 (which multiplies the raw feature S 0 x ), and R ∈ R ≥
if a hyperparameter which controls the strength of this regularization.
Equation 4 is solved for the optimal coefficents β
using linear least squares.
Intuition As noted above, if we set the regularization R = 0 , or more generally in the limit as
R → 0 , the approximation error in Equation 4 is trivially minimized and results in x ASGC
= x ; that is, the learned filter just ignores the graph structure and maps the input feature through unchanged.
Nonzero regularization forces the least squares reconstruction to use the graph structure as it approximates the raw, unpropagated feature.
Ideally, this results in a denoising effect that is able to extend beyond SGC’s fixed smoothing along edges.
For example, when a graph S is homophilous with
Input: undirected graph A ∈ { 0 , 1 } n × n , node feature x ∈ R n , number of hops K ∈ Z > 0 , regularization strength R ∈ R ≥
Output: ASGC-filtered feature x ASGC
D ← diag ( A 1 ) {degree matrix}
S ← D − 1 / 2 AD − 1 / 2 {normalized adjacency matrix}
T ∈ R n × ( K +1) ← 0 { k -step propagated features}
T 0 ← x for k = 1 to K do
T k ← ST k − 1 end for
R ∈ R K +1 ← ( R, 0 , 0 , . .
. , 0) β ← least squares solution of (cid:16) TR (cid:17) β ≈ (cid:16) x 0 (cid:17)
Return: T β respect to a node feature x , in that neighbors tend to have similar feature values, the raw feature is correlated positively with the propagated version Sx of the feature.
By contrast, when a graph is heterophilous with respect to a feature, the correlation is negative.
The least squares in ASGC is able to adapt to both cases and exploit this correlation, as well as correlations that occur when repeatedly propagating features (i.e., correlations of x with S k x for k > 1 ).
Further remarks In theory, as K is raised to higher values, T will provide a sufficiently high-rank basis that x ASGC will be arbitrarily close to x , even if R is very high.
Then ASGC would have essentially the same performance as using the raw feature.
While this issue could be resolved by introducing a smaller regularization term for the remaining coefficients, we find that this is generally not a problem over reasonable values of K on real-world graphs; hence, for simplicity, we do not introduce this further regularization.
Pseudocode to filter a single feature is given in Algorithm 1.
This algorithm is applied independently to all features; note that this is trivially parallelizable across features.
After this, as in SGC, the filtered features are passed as input to a logistic regression classifier for node classification.
The core computations in Algorithm 1 are 1) creation of the matrix T by multiplying x by S up to K times, for which the time complexity is O ( mK ) , where m is the number of edges; and 2) linear least squares with a matrix of dimensionality ( n + 1) ×
( K + 1) , for which the complexity is O ( nK 2 ) .
Spectral Interpretation of ASGC ASGC admits an interesting alternate interpretation based on a spectral view of Equation 4.
Let S =
Q diag( λ ) Q ⊤ be an eigendecomposition of S , and let γ = Q − 1 x , that is, γ is the graph Fourier transform of the feature x .
The central objective in ASGC is the norm of the residual of the least squares in Equation 4.
As in Parseval’s Theorem, due to the orthogonality of Q , this norm is invariant under the graph Fourier transform:
[Equation]
Recall that each column of T is of the form S
i x for some nonnegative power i .
Then
[Equation]
and the minimization objective can be written as
[Equation]
where, letting superscript ◦ i denote the entrywise i th power of a vector,
[Equation]
is the Vandermonde matrix of powers 0 to K of the eigenvalues of S .
Note that multiplying V λ by the vector β yields the values of the polynomial with coefficients β , evaluated at the eigenvalues λ .
Hence ASGC can be interpreted as fitting a K -degree polynomial over the graph’s eigenvalues, with the target being all-ones.
The value the polynomial takes over each eigenvalue represents how the learned filter scales the component of the feature x which is along the direction of the corresponding eigenvector of S ; the all-ones target corresponds to a do-nothing filter.
The least squares loss is weighed proportionately to the magnitude of this component at each eigenvalue.
That K is small, and the use of regularization on the zeroth coefficient, precludes the learned filter actually being the do-nothing filter, and instead results in a simple polynomial which adapts to the feature.
Motivating Example
We now use a synthetic network to demonstrate the capability of ASGC, and the potential deficiencies of SGC, at denoising a single heterophilous feature.
We propose Featured SBMs, which augment stochastic block models (SBMs) (Holland et al., 1983) with a single feature; we note that our FSBMs can be seen as a simplified variant of recently studied Contextual SBMs (Deshpande et al., 2018).
Definition 1 (Featured SBM) .
An SBM graph G has n nodes partitioned into r communities
C 1 , C 2 , . .
, C r , with intra- and inter- community edge probabilities p and q .
Let c 1 , c 2 , . .
. , c r ∈ { 0 , 1 } n be indicator vectors for membership in each community, i.e., the j th entry of c i is 1 if the j th node is in C i and 0 otherwise.
A Featured SBM (FSBM) is such a graph model G , plus a feature vector x = f + η ∈ R n , where η ∼ N (0 , σ 2 I ) is zero-centered, isotropic Gaussian noise and f = (cid:80) i µ
i c i for some µ 1 , µ 2 , . .
. , µ r ∈ R , which are the expected feature values of each community.
We consider FSBMs with n = 1000 , 2 equally-sized communities C + and C − , feature means µ + = +1 , and µ − = − 1 , and noise variance σ = 1 .
Thus, there are 500 nodes in each community; calling these communities ‘plus’ and ‘minus,’ the feature mean is +1 for nodes in the former and − 1 for the latter, to which standard normal noise is added.
We generate different graphs by setting the expected degree of all nodes to 10 (that is, 12 ( p + q ) · n = 10 ) , then varying the intra- and inter-community edge probabilities p and q from p ≫ q (highly homophilous, in that ‘plus’ nodes are much more likely to connect to other ‘plus’ nodes than to ‘minus’ nodes) to q ≫ p (highly heterophilous, in that ‘plus’ nodes tend to connect to ‘minus’ nodes).
See Figure 1 for illustration.
[Figure]
We seek to denoise the feature by exploiting the graph, which should result in the feature values moving towards the means of the respective communities.
We employ SGC and our ASGC, both with number of hops K = 2 .
Figure 2 (left) shows the deviation from the feature means after denoising.
It also shows the proportion of nodes whose filtered feature differs from the community mean in sign, that is, the error when classifying nodes into C + and C − .
By both metrics, ASGC outperforms
SGC on heterophilous graphs, while SGC outperforms ASGC on homophilous graphs.
Both methods can lose accuracy relative to just using the raw feature when the graph is neither homophilous nor heterophilous, that is, when the graph is not informative about the communities.
However, the performance of ASGC increases similarly as the graph becomes either more heterophilous or homophilous, whereas SGC’s performance improves significantly less in the heterophilous direction.
Finally, the performance gap between the two is smaller on homophilous graphs, particularly on sign accuracy, suggesting that ASGC can better adapt to varying degrees of homophily/heterophily.
We examine the highly heterophilous case in more detail in Figure 2 (right), which shows the distributions of the feature before and after filtering.
The fixed propagation of SGC tends towards merging the two communities’ feature distributions; by contrast, ASGC is able to keep them separated, preserves the feature means, and pulls the feature distributions towards the respective community means.
M e a n
A b s .
E rr o r -1.0 (Heterophilous) -0.
+0.
+1.0 (Homophilous) p q p + q 0.
S
i g n E rr o r
Raw SGC
ASGC
Raw
SGC   1
ASGC
Theoretical Guarantees
To support our empirical investigation in Section 4, we now theoretically verify the limitations and capabilities of SGC and ASGC at denoising FSBM networks.
For simplicity, we analyze SGC without the addition of a self-loop (that is, using S in Equation 1 rather than ˜ S ); the distinction between the two in the analysis vanishes as the number of intra-community edges grows, i.e., if n · p is high.
Further, we assume that the regularization hyperparameter R for ASGC is high, in which case the coefficient β 0 is fixed to zero, or equivalently, the column S 0 x is removed from the Krylov matrix
T in Equations 3 and 4.
Finally, we analyze using expected adjacency matrices from the model, though we conjecture that via concentration bounds one could extend the following results to the sampled setting (Spielman, 2012).
Here, we analyze FSBMs with 2 equally-sized communities.
In
Appendix 9.1, we prove that these results extend to an arbitrary number of communities.
Unless otherwise specified, we use the same notation as Definition 1.
Theorem 1 (Effect of SGC on FSBM Networks) .
Consider FSBMs having 2 equally-sized communities with indicator vectors c u and c v , expected adjacency matrix A , and feature vector x = f + η .
Let x SGC be the feature vector returned by applying SGC, with number of hops K , to A and x .
Further, let ¯ µ = 12 ( µ u + µ v ) be the average of the feature means.
Then, x SGC = f ′ + θ u c u
+ θ v c v , where f ′ =
λ K 2 f
+ (1 − λ K 2 )(¯ µ 1 ) , and θ u and θ v are both distributed according to N (cid:0) 0 , 1 n (1 + λ 2 K 2 ) σ 2 (cid:1) .
Proof.
In expectation, an entry A ij of the adjacency matrix of the graph is p if both i, j ∈ C u or both i, j ∈ C v , and it is q otherwise.
The eigendecomposition Q diag( λ ) Q ⊤ of the associated normalized adjacency matrix S has two nonzero eigenvalues: λ 1 = 1 , with eigenvector q 1 = ( 1 / √ n ) 1 = ( 1 / √ n )( c u + c v ) , and λ 2 =
p − q p + q , with q 2 = ( 1 / √ n )( c u − c v ) .
In the following analysis, we use the fact that zero-centered, isotropic Gaussian distributions are invariant to rotation, meaning
Q ⊤ η = η ′ ∼ N (0 , σ
I ) for any orthonormal matrix Q .
[Equation]
where θ ± = 1 √ n ( η ′ 1 ± λ K 2 η ′ 2 ) , which has the specified distribution.
Note that λ 2 =
p − q p + q ∈ [ − 1 , +1] , with negative values indicating heterophily ( p < q ) and positive values indicating homophily ( p > q ).
SGC only preserves the feature means in certain limiting cases.
In particular, this occurs as λ 2 → +1 , or as λ 2 → − 1 if K is even; then λ K 2 → 1 , so the expected filtered feature vector f ′
→ f .
On the other hand, if λ 2 → 0 , then λ K 2 → 0 and f ′ → ¯ µ 1 , that is, the feature value means are averaged between the communities.
Finally, if λ 2 → − 1 and K is odd, then λ K 2 → − 1 and f ′ =
¯ µ 1 + (¯ µ 1 − f ) = µ v c u + µ u c v : the feature value means are entirely exchanged across the communities.
By contrast, ASGC preserves the means, while similarly reducing noise by an O ( n ) factor, with much looser restrictions on λ 2 and K :
Theorem 2 (Effect of ASGC on FSBM Networks) .
Consider FSBMs with p ̸
= q having 2 equallysized communities with community indicator vectors c u and c v , expected adjacency matrix A , and feature vector x = f + η .
Let x ASGC be the feature vector returned by applying ASGC, with number of hops K ≥ 2 , to A and x .
Then x ASGC =
f + θ ′ + c u + θ ′ − c v , where θ ′ + and θ ′ − are both distributed according to N (cid:0) 0 , 2 n σ 2 (cid:1) .
Proof.
The least squares in ASGC is equivalent to projecting the feature x onto the column span of the Krylov matrix T
= (cid:0) S 1 x ; S 2 x ; . . .
; S K x (cid:1) .
Observe that the column span of T is contained in the column span of S .
Further, S is rank- 2 (by the assumption that p ̸ = q ), so with probability 1 over the distribution of η , as long as K ≥ 2 , the column span of T equals that of S .
Thus ASGC projects x onto the column span of S , i.e., the span of q 1 , q 2 , the eigenvectors of S .
The following analysis proceeds exactly like the one for SGC, just without the terms for the eigenvalue λ 2 , so we use the same notation and abbreviate the steps:
[Equation]
where θ ′± = 1 √ n ( η ′ 1 ± η ′ 2 ) , which has the specified distribution.
Observe that in the sampled setting, standard matrix concentration results can be used to show that, while S will be full rank with high probability, it will have two outlying eigenvalues, corresponding to eigenvectors close to q 1 and q 2 (Spielman, 2012).
It is well known that the span of the Krylov matrix T will align well with these outlying eigendirections (Saad, 2011).
Thus, we expect the projection QQ ⊤ x = QQ ⊤ f
+ QQ ⊤ η to still approximately preserve f .
At the same time, QQ T η is the projection of a random Gaussian vector η onto a fixed K -dimensional subspace.
Thus, we will have (cid:13)(cid:13) QQ ⊤ η
(cid:13)(cid:13)
≈
Kn ∥
η ∥ 22 , so ASGC will still perform significant denoising when K is small.
Related Work
Deep graph models As discussed previously, the SGC algorithm is a drastic simplification of the graph convolutional network (GCN) model (Kipf & Welling, 2017).
GCNs learn a sequence of node representations that evolve via repeated propagation through the graph and nonlinear transformations.
The starting node representations H (0) are set to the input feature matrix X ∈ R n × f , where f is the number of features.
The k th -step representations are H ( k )
=
σ (cid:0) SH ( k − 1) Θ
( k ) (cid:1) , where Θ ( k ) is the learned linear transformation matrix for the k th layer and σ is a nonlinearity like ReLU.
After K such steps, the representations are used to classify the nodes via a softmax layer, and the whole model is trained end-to-end via gradient descent.
Wu et al.
(2019) observe that if the nonlinearities are ignored, all of the linear transformations collapse into a single one, while the repeated multiplications by S collapse into a single one by S K ; this yields their algorithm of the SGC filter (Equation 1) followed by logistic regression.
GCNs have spawned streamlined versions like FastGCN (Chen et al., 2018), as well as more complicated variants like graph isomorphism networks (GINs)
(Xu et al., 2019) and graph attention networks (GATs) (Veliˇckovi´c et al., 2018); despite being much simpler and faster than these competitors, SGC manages similar performance on common benchmarks, though, based on the analysis of NT & Maehara (2019), this may be due in part to the simplicity of the benchmark datasets in that they mainly exhibit homophily/assortativity.
[Table]
Addressing heterophily Like our work, some other recent methods attempt to address node heterophily.
Gatterbauer (2014) and Zhu et al.
(2020a) augment classical feature propagation and
GCNs, respectively, to accommodate heterophily by modifying feature propagation based on node classes.
Zhu et al.
(2020b) and Yan et al.
(2021) analyze common structures in heterophilous graphs and the failure points of GCNs, then propose GCN variants based on their analyses.
The Geom-GCN paper of Pei et al.
(2020) introduces several of the real-world heterophilous networks which are commonly used in related papers, including this one.
Their method allows for long-range feature propagation based on similarity of pre-trained node embeddings.
The preceding is just a sample of recent works in this area, which has seen a surge of activity (Liu et al., 2020;
Luan et al., 2021;
Suresh et al., 2021).
We note that, like the GNNs of Kipf & Welling (2017) but unlike SGC and our ASGC, almost all of these methods are based on deep learning and are trained via backpropagation through repeated feature propagation and linear transformation steps, and hence incur an associated speed and memory requirement.
Understanding and implementing these methods is also more complicated relative to our method, which just constitutes a learned feature filter and logistic regression.
We mainly compare our results with the deep method which is most similar to ours, Generalized PageRank
GNN (GPR-GNN) (Chien et al., 2021).
Like ASGC, GPR-GNN produces node representations by linear combination of propagated versions of node features; unlike ASGC, the raw features are first transformed by a neural network, and parameters for this network, as well as the linear combination coefficients, are learned by backpropagation using the known node labels.
To our knowledge, our work is the first to show that heterophily can be handled using just feature pre-processing.
Empirical Performance
We test the performance of ASGC for the node classification task on a benchmark of real-world datasets given in Table 1, and compare with SGC and several deep methods.
Real-world datasets We experiment on 10 commonly-used datasets, the same collection of datasets as Chien et al.
C ORA , C ITESEER , and P UBMED are citation networks which are common benchmarks for node classification (Sen et al., 2008; Namata et al., 2012); these have been used for evaluation on the GCN (Kipf & Welling, 2017) and
GAT
(Veli ˇ ckovi ´ c et al., 2018) papers, in addition to SGC itself.
The features are bag-of-words representations of papers, and the node labels give the topics of the paper.
C OMPUTERS and P HOTO are segments of the Amazon copurchase graph (McAuley et al., 2015; Shchur et al., 2018); features are derived from product reviews, and labels are product categories.
These first 5 datasets are considered assortative/homophilous; the remaining 5 datasets, which are disassortative/heterophilous, come from the Geom-GCN paper (Pei et al., 2020), which also introduces the following measure of of a network’s homophily:
[Equation]
[Table]
∈ [0 , 1] .
We include this statistic in Table 1.
The latter 5 datasets have much lower values of H ( G ) .
C HAMELEON and S QUIRREL are hyperlink networks of pages in Wikipedia which concern the two topics (Rozemberczki et al., 2021).
Features derive from text in the pages, and labels correspond to the amount of web traffic to the page, split into 5 categories.
A CTOR is the actor-only induced subgraph of the film-director-actor-writer network of Tang & Liu (2009).
Nodes and edges represent actors and co-occurrence on a Wikipedia page.
Features are based on keywords on the webpage, and labels derive from the number of words on the page, split into 5 categories.
Finally, T EXAS and C ORNELL are hyperlink networks from university websites (Craven et al., 2000); features derive from webpage text, and the labels represent the type of page: student, project, course, staff, or faculty.
Implementation The SGC and ASGC algorithms are implemented in Python using NumPy (Harris et al., 2020) for least squares regression and other linear algebraic computations.
We use scikit-
[Figure]
learn (Pedregosa et al., 2011) for logistic regression with 1,000 maximum iterations and otherwise default settings.
For our implementations of SGC and ASGC, we treat each network as undirected, in that if edge ( i, j ) appears, we also include edge ( j, i ) .
Like Chien et al.
(2021), we use random 60%/20%/20% splits as training/validation/test data for the 5 heterophilous datasets, as in Pei et al.
(2020), and use random 2.5%/2.5%/95% splits for the homophilous datasets, which is closer to the original setting from Kipf & Welling (2017) and Shchur et al.
We release code in the form of a Jupyter notebook (Pérez & Granger, 2007) demo which is available at github.com/schariya/adaptive-simple-convolution.
Hyperparameter settings We tune the number of hops over K ∈ { 1 , 2 , 4 , 8 } , roughly covering the range analyzed in Wu et al.
(2019), and the regularization strength R = √ n · R ′ over log 10 ( R ′ ) ∈ {− 4 , − 3 , − 2 , − 1 , 0 } .
This dependency on the number of nodes n allows the regularization loss to scale with the least squares loss, which generally grows linearly with n .
In Appendix 9.2, we report results for some additional experiments investigating the effect of fixing these hyperparameters.
Classification results We apply our implementations of SGC and ASGC to these datasets and report the mean test accuracy across 10 random splits of the data.
As a baseline, we also report the accuracy of logistic regression on the ‘raw,’ unfiltered features, ignoring the graph.
We compare to results from Chien et al.
(2021) for 9 deep methods applied to these datasets.
These methods are 1) a multi-layer perceptron which ignores the graph; 2) GCN; 3) GAT; 4) SAGE (Hamilton et al., 2017); 5)
JKNet (Xu et al., 2018); 6) GCN-Cheby (Defferrard et al., 2016); 7) Geom-GCN; 8)
APPNP (Klicpera et al., 2019); and 9) GPR-GNN.
Full results are given in Table 2 in Appendix 9.3.
We plot accuracies for selected methods in Figure 3.
In addition to SGC and ASGC, we include 3 deep methods: ‘vanilla’ GCN; Geom-GCN, which originated the heterophilous datasets; and
GPR-GNN, a recent method claiming state-of-the-art performance.
We find that SGC is generally competitive with the deep methods on the homophilous datasets, but not so on the heterophilous ones, whereas ASGC is generally competitive throughout.
Interestingly, the datasets on which GPRGNN significantly outperforms ASGC (P UB M ED , A CTOR , T EXAS , C ORNELL ) are exactly those on which a multi-layer perceptron significantly outperforms logistic regression; note that the latter two methods both ignore the graph.
This suggests that the some nonlinear processing of the node features may be key to performance on these networks, separate from how the graph is exploited.
To compactly compare all 12 of the methods across these 10 datasets, we aggregate the performance across the datasets as follows.
For each dataset, we calculate the accuracy of each method as a proportion of the accuracy of the best method.
We plot the mean and the minimum across the 10 datasets of each method’s proportional accuracies.
See Figure 4.
GPRGNN and ASGC achieve the highest mean
[Figure]
Conclusion, Limitations, and Broader Impact
Building on SGC, we propose a feature filtering technique, ASGC, based on feature propagation and least-squares regression.
We propose a natural class of synthetic featured networks, FSBMs, and show both empirically and with theoretical guarantees that ASGC can denoise both homophilous and heterophilous FSBMs, whereas SGC is inappropriate for the latter.
Further, we find that ASGC is generally competitive with recent deep learning-based methods on a benchmark of real-world datasets, covering both homophilous and heterophilous networks.
Our results suggest that deep learning is not strictly necessary for handling heterophily, and that even a simple feature pre-processing method can be competitive.
We hope that, like SGC, ASGC can serve as a good first method to try, especially for node classification on heterophilous networks, and a baseline for future works.
We note certain limitations of our work.
First, our theoretical guarantees are in the fairly idealized setting of FSBM networks; we provide no general guarantee of performance.
Second, our empirical evaluations on real-world datasets do not include any truly large-scale datasets ( n ∼ 10 6 or higher), though our collection of 10 datasets is a fairly broad selection amongst those which are commonly used in related papers about heterophily.
Finally, as we note in Section 6, there are many more recent works and methods for handling heterophily to which we could compare our results.
We limit ourselves to a subset due to the amount of activity in this area; further, we are not asserting absolute state-of-the-art results in this area, merely that a simple method like ours can be competitive.
We do not anticipate direct negative societal impacts of our work, but improved methods for network analysis, including those for node classification, have social consequences.
For example, improved graph-based recommendations may be associated with negative impacts to well-being due to increased social media use (Verduyn et al., 2017; Kelly et al., 2018).
They might also contribute to concerns of filter bubbles and polarization (Nguyen et al., 2014; Lee et al., 2014; Musco et al., 2018).
Acknowledgments and Disclosure of Funding
This project was partially supported by an Adobe Research grant, a Google Research Scholar
Award, and NSF Grants 2046235 and 1763618.
We also thank Raj Kumar Maity and Konstantinos
Sotiropoulos for helpful conversations.
Checklist
For all authors... (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes] (b) Did you describe the limitations of your work?
[Yes] Section 8 describes limitations.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] Sectiondiscusses potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes] 2.
If you are including theoretical results... (a) Did you state the full set of assumptions of all theoretical results?
[Yes] Sectiondefines the FSBM networks and Section 5 describes any further assumptions for theoretical results.
(b) Did you include complete proofs of all theoretical results?
[Yes] Section 5 includes proofs.
If you ran experiments... (a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] The code is a straightforward Jupyter demo, and it is included in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] The implementation subsection of Section 7 specifies the data splits and hyperparameters and where they came from.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] Both the main results plot and the full table of results include confidence intervals.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[No] As should be clear from the time complexity discussion in Section 3 and the dataset statistics in Table 1, our method is lightweight enough to run the benchmarks within a few hours on a laptop without a GPU, so compute is not a significant concern.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators?
[Yes] We only use standard node classification benchmarks and credit the original papers.
(b) Did you mention the license of the assets?
[N/A] (c) Did you include any new assets either in the supplemental material or as a URL?
[N/A] We do not produce new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[N/A] (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[N/A] 5.
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A] (b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[N/A] (c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
Appendix 9.1 Multi-Community FSBM Proofs
We now prove generalizations of the theorems from Section 5 for FSBMs that may have more thancommunities, i.e., with r ≥ 2 from Definition 1, and otherwise the same assumptions.
The theorem statements and their implications are essentially the same.
The proofs are also similar in concept, just using a different projection matrix Q , though the proof for the generalized Theorem 1 is significantly more involved.
Theorem 1 (Effect of SGC on FSBM Networks) .
Consider FSBMs having r equally-sized communities with indicator vectors c 1 , c 2 , . .
c k , expected adjacency matrix A , and feature vector x = f + η .
Let x SGC be the feature vector returned by applying SGC, with number of hops K , to A and x .
Further, let ¯ µ = 1 r (cid:80) i µ i be the average of the feature means.
Then, x SGC = f ′ + (cid:80)
i
θ
i
c i , where f ′ =
λ K 2 f
+ (1 − λ K 2 )(¯ µ 1 ) , and each θ i is distributed according to N (cid:0) 0 , 1 n (cid:0) 1 + λ 2 K 2 ( r − 1) (cid:1) σ 2 (cid:1) .
Proof.
Let ˆ 1 = ( 1 / √ n ) 1 , where 1 is the n -length all-ones vector.
Let C
=
( c 1 ; c 2 ; . . .
; c r ) and
Q = (cid:0)(cid:112) r / n (cid:1) C .
Note that ˆ 1 has norm 1 , and the columns of Q are orthonormal.
Finally, let λ
=
p − q p +( r − 1) q .
We we will not make use of this fact, but, as in the 2-community case from Section 5, this is still the second largest eigenvalue of S , and it now has multiplicity r − 1 .
The expected adjacency matrix of the graph is
[Equation]
[Equation]
1 ⊤ , and the expected degree vector is
[Equation]
yielding the expected normalized adjacency matrix
[Equation]
Note that (cid:0) QQ ⊤ (cid:1) 2 = QQ ⊤ and (cid:0) ˆ 1 ˆ 1 ⊤ (cid:1)
=
ˆ 1 ˆ 1 ⊤ since these are projection matrices.
We show that
[Equation]
by induction as follows:
[Equation]
Using this result and the fact that (cid:0) QQ ⊤ (cid:1) (cid:0) ˆ 1 ˆ 1 ⊤ (cid:1) = (cid:0) ˆ 1 ˆ 1 ⊤ (cid:1) (cid:0) QQ ⊤ (cid:1)
=
ˆ 1 ˆ 1 ⊤ , we have
[Equation]
Now, as in the 2-community case, Q ⊤ η = η ′ ∼ N (0 , σ 2 I ) , yielding
[Equation]
Finally
[Equation]
combining these equations with the expression for S K
[Equation]
we have
[Equation]
so the expectation f ′ of the filtered feature is as desired.
Further
[Equation]
letting the noise term summands be θ
i
c i
[Equation]
we have
[Equation]
which is normally distributed with mean 0 and variance
[Equation]
so the noise variance is also as desired.
Theorem 2 (Effect of ASGC on FSBM Networks) .
Consider FSBMs with p ̸
=
q having r equallysized communities with indicator vectors c 1 , c 2 , . .
c k , expected adjacency matrix A , and feature vector x = f + η .
Let x ASGC be the feature vector returned by applying ASGC, with number of hops K ≥ r , to A and x .
Then, x ASGC =
f
+ (cid:80) i θ ′
i c i , where each θ ′
i is distributed according to
N (cid:0) 0 , rn σ 2 (cid:1) .
Proof.
Following the same argument from Section 5, the least squares in ASGC is equivalent to projecting the feature x onto the column span of the Krylov matrix T = (cid:0) S 1 x ; S 2 x ; . . .
; S K x (cid:1) .
The column span of T is contained in the column span of S , and since S is rank- r (by the assumption that p ̸ = q ), with probability 1 over the distribution of η , as long as K ≥ r , the column span of T equals that of S ; by Equation 5 for S , this span is exactly that of the community indicator matrix
Q .
Thus, ASGC is equivalent to multiplication of the feature x by the projection matrix QQ ⊤ , for which we use Equation 6 as follows:
[Equation]
where θ ′ i
= (cid:0)(cid:112) r / n (cid:1)
η ′ i , which has the specified distribution.
9.2 Hyperparameter Study
We investigate the effect of two hyperparameter choices on validation accuracy: number of hops K and regularization strength R ′ .
We plot results for each hyperparameter in Figures 5 and 6, respectively.
Trends vary significantly across datasets, indicating that tuning these hyperparameters is essential to optimal performance.
This is perhaps to be expected for R ′ in particular, since this hyperparameter controls, in a fairly direct way, the influence of the raw feature on the nodes’ representations, whereas in an end-to-end method, the influence can be determined more indirectly based on information from the class labels
[Figure]
′ with highest mean validation accuracy
[Figure]
9.3 Full Node Classification Results
Table 2 reports the full node classification performance results, which were deferred in Section 7.
We provide results for our implementations of logistic regression on raw features, SGC, and ASGC; we also include results from Chien et al.
(2021) for 9 deep methods, including their GPR-GNN.
As the authors note, the results for Geom-GCN (Pei et al., 2020) on datasets not originally tested in that paper (in particular, the co-purchasing networks C OMPUTERS and P HOTO ) cannot be included due to a pre-processing subroutine that is not publicly available.
[Table]