GLINKX: A Scalable Unified Framework for Homophilous and Heterophilous Graphs
Abstract
In graph learning, there have been two main inductive biases regarding graphinspired architectures:
On the one hand, higher-order interactions and message passing work well on homophilous graphs and are leveraged by GCNs and GATs.
Such architectures, however, cannot easily scale to large real-world graphs.
On the other hand, shallow (or node-level) models using ego features and adjacency embeddings work well in heterophilous graphs.
In this work, we propose a novel scalable shallow method – GLINKX – that can work both on homophilous and heterophilous graphs.
To achieve scaling in large graphs, GLINKX leverages (i) novel monophilous label propagations (ii) ego/node features, (iii) knowledge graph embeddings as positional embeddings, (iv) node-level training, and (v) lowdimensional message passing.
We show the effectiveness of GLINKX on several homophilous and heterophilous datasets.
Introduction
In recent years, graph learning methods have emerged with a strong performance for various ML tasks.
Graph ML methods leverage the topology of graphs underlying the data [4] to improve their performance.
Two very important design options for proposing graph ML based architectures in the context of node classification are related to whether the data is homophilous or heterophilous .
For homophilous data – where neighboring nodes share similar labels [26, 3] – Graph Neural Network (GNN)-based methods are able to achieve high accuracy.
Specifically, a broad subclass sucessfull GNNs are Graph Convolutional Networks (GCNs) (e.g., GCN, GAT, etc.) In the
GCN paradigm, message passing and higher-order interactions help node classification tasks in the homophilous setting since such inductive biases tend to bring the learned representations of linked nodes close to each other.
However, GCN-based architectures suffer from scalability issues .
Performing (higher-order) propagations during the training stage are hard to scale in large graphs because the number of nodes grows exponentially with the increase of the filter receptive field.
Thus, for practical purposes, GCN-based methods require node sampling , substantially increasing their training time.
For this reason, architectures [16, 41, 36, 25, 30] that leverage propagations outside of the training loop (as a preprocessing step) have shown promising results in terms of scaling to large graphs.
In heterophilous datasets [29], the nodes that are connected tend to have different labels.
Many works that address heterophily so far can be classified into two main categories concerning scale.
On the one hand, recent successful architectures (in terms of accuracy) [17, 10, 43, 23, 8] that address heterophily resemble GCNs in terms of design and thus suffer from the same scalability issues.
On the other hand, shallow or node-level models (see, e.g., [22, 44]), i.e., models that are treating graph data as tabular data and do not involve propagations during training, have shown a lot of promise for large heterophilous graphs.
In [22], it is shown that combining ego embeddings (aka.
node features) and adjacency embeddings works in the heterophilous setting.
However, their design is still impractical in real-world data since the method (LINKX) is not inductive, and embedding the adjacency matrix directly requires many parameters in a model.
In LINKX, the adjacency embedding of a node can alternatively be thought of as a positional embedding (PE) of the node in the graph, and recent developments [18, 12, 22] have elevated the importance of PEs in both homophilous and heterophilous settings.
However, most of these works suggest PE parametrizations that are difficult to compute in large-scale settings.
For this reason, more scalable ways of computing PEs via knowledge graph embeddings [13, 20, 6, 39] are useful in practical settings.
Goal & Contribution
In this work, we develop a scalable method for node classification that: (i) works both on homophilous and heterophilous graphs (ii) is simpler and faster than conventional message passing networks (by avoiding the neighbor sampling and message passing overhead), and (iii) can work in both a transductive and an inductive 2 setting.
For a method to be scalable, we argue that it should: (i) run models on node-scale (thus leveraging i.i.d.
minibatching), (ii) avoid doing message passing during training and do it a constant number of times before training, and (iii) transmit small messages along the edges.
Our proposed method – GLINKX (see Section 3) – combines all the above desiderata.
GLINKX has three components: (i) ego embeddings 3 , (ii) PEs inspired by architectures suited for heterophilous settings, and (iii) scalable 2nd -hop-neighborhood propagations inspired by architectures suited for monophilous settings.
Finally, we evaluate GLINKX’s empirical effectiveness on several homophilous and heterophilous datasets (Section 4).
Preliminaries 2.1 Notation
We represent scalars with lower-case, vectors with bold lower-case letters, and matrices with bold upper-case letters.
We consider a directed graph G =
G ( V, E ) with vertex set V with | V | = n nodes, and edge set E with | E | = m edges.
Let X ∈ R n × d X represent the d X dimensional node feature matrix and P ∈ R n × d P represent the d P dimensional node positional embedding matrix.
A node i has a feature vector x i ∈ R d X and a positional embedding p i ∈ R d P and belongs to a class y i ∈ { 1 , . . .
, c } .
The training set is denoted by V train , validation set by V valid , and test set by V test . I {·} denotes the indicator function.
2.2 Homophily, Heterophily & Monophily
Homophily and Heterophily: There are various measures of homophily in the GNN literature like node homophily and edge homophily [22]. Intuitively, homophily in a graph implies that nodes with similar labels are connected. GNN-based approaches like GCN, GAT, etc., leverage this property to improve the node classification performance. Alternatively, if a graph has low homophily – namely, nodes that connect tend to have different labels – it is said to be heterophilous .
In other words, a graph is heterophilous if neighboring nodes do not share similar labels.
Monophily: Generally, we define a graph to be monophilous if the label of a node is similar to that of its neighbors’ neighbors 4 . Etymologically, the word “monophily” is derived from the Greek words “monos” (unique) and “philos” (friend), which in our context means that a node – regardless of its label – has neighbors of primarily one label. In the context of a directed graph, monophily can be thought of as a structure that resembles Fig. 1(b) where similar nodes (in this case, three green nodes connected to a yellow node) are connected to a node with a different label. We argue that encoding monophily into a model can be helpful for both heterophilous and homophilous graphs (see Figs. 3(b) and 3(c)), which is one of the main motivators behind our work .
In homophilous graphs, monophily will fundamentally encode the 2nd-hop neighbor’s label information, and since in such graphs, neighboring nodes have similar labels, it can provide a helpful signal for node classification.
In heterophily, neighboring nodes have different labels, but the 2nd-hop neighbors may share the same label, providing helpful information for node classification.
Monophily is effective for heterophilous graphs [22].
Therefore, an approach encoding monophily has an advantage over methods designed specifically for homophilous and heterophilous graphs, especially when varying levels of homophily can exist between different sub-regions in the same graph (see Section 3.3).
Our Method: GLINKX 3.1 Components & Motivation
The desiderata we laid down on Section 1 can be realized by three components: (i) PEs, (ii) ego embeddings, and (iii) label propagations that encode monophily.
More specifically, ego embeddings and PEs are used as primary features, which have been shown to work for both homophilous and heterophilous graphs for the models we end up training. Finally, the propagation step is used to encode monophily to provide additional information to our final prediction. GLINKX is described in Alg. 1 and consists of three main components: (i) PEs, (ii) ego embeddings, and (iii) propagations.
Fig.
1 shows Alg.
1 on a toy graph.
Positional Embeddings: We use PEs to provide our model information about the position of each node and hypothesize that PEs are an important piece of information in the context of large-scale node classification.
PEs have been used to help discriminate isomorphic graph (sub)-structures [18, 12, 35].
This is useful for both homophily [18, 12] and heterophily [22] because isomorphic (sub)-structures can exist in both the settings.
In the homophilous case, adding positional information can help distinguish nodes that have the same neighborhood, but distinct position [12, 27, 38], circumventing the to do higher-order propagations [12, 21, 7] which are prone to over-squashing [2].
In heterophily, structural similarity among nodes is important for classification, as in the case of LINKX – where adjacency embedding can be considered a PE.
However, in large graphs, using adjacency embeddings or Laplacian eigenvectors (as methods such as [18] suggest) can be a computational bottleneck and may be infeasible. [Figure]
In this work, we leverage knowledge graph embeddings (KGEs) to encode positional information about the nodes.
Using KGEs has two benefits: Firstly, KGEs can be trained quickly for large graphs.
This is because KGEs compress the adjacency matrix into a fixed-sized embedding, and adjacency matrices have been shown to be effective in heterophilous cases.
Further, KGEs are low-dimensional compared to the adjacency matrix (e.g., d P ∼ 10 2 ), which allows for faster training and inference times.
Secondly, KGEs can be pre-trained efficiently on such graphs [20] and can be used off-the-shelf for other downstream tasks, including node classification (see, e.g., [13]).
So, in the 1st Stage of our methods in Alg. 1, we train the KGEs model on the available graph structure.
Here, we fix this positional encoding once they are pre-trained for downstream usage.
One can fine-tune these along with learning [12] in the downstream task, but we leave this for future work.
Finally, we note that this step is transductive but we can easily make it inductive (see e.g. [13, 1]).
Ego Embeddings: We obtain ego embeddings from the node features.
Such embeddings have been used in homophilous and heterophilous settings (e.g., [22, 46]).
Node embeddings are useful for tasks where the graph structure provides little/no information about the task.
Monophilous Label Propagations: We now propose a novel monophily-inspired label propagation (see Section 2.2) which we refer to as Monophilous Label Propagation (MLaP). MLaP has the advantage that we can use it both for homophilous and heterophilous graphs or in a scenario with varying levels of graph homophily (see Section 3.3) as it encodes monophily (Section 2.2). To understand how MLaP encodes monophily, we consider the example in Fig.
In this example, we have three green nodes connected to a yellow node and two nodes of different colors connected to the yellow node. Then, one way to encode monophily in Fig. 1(b) while predicting label for j ℓ , ℓ ∈ [5] , is to get a distribution of labels of nodes connected to node i thus encoding its neighbors’ distribution.
The fact that there are more nodes with green color than other colors can be used by the model to make a prediction. But this information may only sometimes be present, or there may be few labeled nodes around node i .
Consequently, we propose to use a model that predicts the label distribution of nodes connected to i . We use the node features ( x i ) and PE ( p i ) of node i to build this model since nodes that are connected to node i share similar labels, and thus, the features of node i must be predictive of its neighbors.
So, in Fig. 1(b), we train a model to predict a distribution of i ’s neighbors. Next, we provide j ℓ the learned distribution of i ’s neighbors by propagating the learned distribution from i back to j ℓ . Eqs.
(1) to (3) correspond to MLaP. We train a final model that leverages this information together with node features and PEs (Fig. 1(c)). 3.2 GLINKX
We put the components discussed in Section 3.1 together into three stages. In the first stage, we pre-train the PEs by using KGEs. Next, encode monophily into our model by training a model that
[Figure]
predicts a node’s neighbors’ distribution and by propagating the soft labels from the fitted model.
Finally, we combine the propagated information, node features, and PEs to train a final model. GLINKX is described in Alg. 1 and consists of three main components detailed as block diagrams in Fig. Fig.
1 shows the GLINKX stages from Alg. 1 on a toy graph: 1st Stage (KGEs):
We train KGEs as PEs by using Pytorch-Biggraph and the DistMult method [39]. 2nd Stage (MLaP): First (2nd Stage in Alg. 1), for a node we want to learn the distribution of its neighbors and then propagate this information back to its neighbors. To achieve this, we propagate the labels from a node’s neighbors, i.e. calculate [Equation]
We train a model that predicts the distribution of neighbors, which we denote with ˜ y
i using the ego features { x i } i ∈ V train and the PEs { p i } i ∈ V train and maximize the cross-entropy with treating { ˆ y i } i ∈ V train as ground truth labels, namely we maximize [Equation] where ˜ y i
=
f 1 ( x i , p i ; θ 1 ) and θ 1 ∈ Θ 1 is a learnable parameter vector.
Although in this paper we assume to be in the transductive setting , this step allows us to be inductive (see App. A.1).
In ?? we give a theoretical justification of this step, namely “why is it good to use a parametric model to predict the distribution of neighbors?” .
Finally, we propagate the predicted soft-labels ˜
y i back to the original nodes, i.e. calculate [Equation] where the soft labels { ˜ y i } i ∈ V train have been computed with the parameter θ ∗ 1 of the epoch with the best validation accuracy from the model f 1 ( ·| θ 1 ) .
3rd Stage (Final Model): As a final stage (3rd Stage in Alg. 1), we make the final predictions y final, i = f 2 ( x i , p i , y ′
i ; θ 2 ) by combining the ego embeddings, PEs, and the (back)-propagated soft labels ( θ 2 is a learnable parameter vector). We use the soft-labels ˜ y i instead of the actual labels one-hot ( y i ) in order to avoid label leakage, which hurts performance (see also [33] for a different way to combat label leakage). Finally, we maximize the cross-entropy with respect to a node’s own labels,
[Equation]
Overall, Stage 2 corresponds to learning the neighbor distributions and propagating this information back to them and Stage 3 uses these distributions to train a new model which predicts a node’s own labels. Label Sparsity: In the case that the graph is very sparsely labeled, we expect that the performance from the MLaP step to go down, however, other components (PEs, node features) can be more predictive of the node’s label. Also, we note that other standard methods, such as GCNs, would have decreased performance due to label sparsity.
Graphs with monophily experience homophily, heterophily, or both. For instance, in the yelp-chi dataset – where we classify a review as spam/non-spam (see Fig. 3) – we observe a case of monophily together with varying homophily.
Specifically in this dataset, spam reviews are linked to non-spam reviews, and non-spam reviews usually connect to other non-spam reviews, which makes the node homophily distribution bimodal. Here the 2nd-order similarity makes both the MLaP mechanism particularly effective and PEs since we can use the PEs to distinguish nodes that have similar features but are located in different regions of the graph (homophilous/heterophilous area). 3.4 Scalability
GLINKX is highly scalable as it performs message passing a constant number of times by paying an O ( mc ) cost, where the dimensionality of classes c is usually small (compared to d X that GCNs rely on). In both Stages 2 and 3 of Alg. 1, we train node-level MLPs, which allow us to leverage i.i.d. (row-wise) mini batching, like tabular data, and thus our complexity is similar to other shallow methods (LINKX, FSGNN) This, combined with the propagations outside the training loops, circumvent the scalability issues of GCNs.
For more details, refer App. A.2.
Different components of GLINKX provide a complementary signal to components proposed in the GNN literature [25, 41, 30].
One can combine GLINKX with existing architectures (e.g. feature propagations [25, 30], label propagations [41]) for potential metric gains. For example, SIGN computes a series of r ∈ N feature propagations [ X, Φ X, Φ 2 X, . . .
, Φ r X ] where Φ is a matrix (e.g., normalized adjacency or normalized Laplacian) as a preprocessing step. We can include this complementary signal, namely, embed each of the propagated features and combine them in the 3rd Stage to GLINKX. Overall, although in this paper, we want to keep GLINKX simple to highlight its main components, we conjecture that adding more components to GLINKX would improve its performance on datasets with highly variable homophily.
Experiments 4.1 Comparisons
We experiment with homophilous and heterophilous datasets (see Tab. 1 and App. B.3).
We train KGEs with Pytorch-Biggraph [20, 39].
For homophilous datasets we compare with vanilla GCN and GAT, FSGNN and Label Propagation (LP).
For a fair comparison, we compare with one-layer GCN/GAT/FSGNN/LP since our method is one-hop.
We also compare with higher-order (h.o.)
GCN/GAT/FSGNN/LP with 2 and 3 layers.
In the heterophilous case, we compare with LINKX results F r e q u e n c y Node Homophily Distribution 0. 0.04 Class Homophily
C l a ss
Class Homophily (Lim et al., 2021) yelp-chi
[Figure] because it is scalable and is shown to work better than other baselines (e.g. H2GCN), as well as with FSGNN. Note that we do not compare GLINKX with other more complex methods because GLINKX is complementary to methods (see Section 3) and design principles from these methods can be incorporated into GLINKX.
We use a ResNet module to combine the components from Stages 2 and 3 of our algorithm.
Details about the hyperparameters we use are in App.
D.
In the heterophilous datasets, GLINKX outperforms LINKX (except arxiv-year where we are within the confidence interval).
Moreover, the performance gap between using KGEs and adjacency embeddings shrinks as the dataset grows.
In the homophilous datasets GLINKX outperforms 1-layer
GCN/GAT/LP/FSGNN and LINKX.
In PubMed, GLINKX outperforms h.o. GCN/GAT and in arxiv-year GLINKX is very close to the performance of GCN/GAT.
Finally, we note that our method produces consistent results across regime shifts .
In detail, in the heterophilous regime our method performs on par with LINKX, however when we shift to the homophilous regime, LINKX’s performance drops, whereas our method’s performance continues to be high.
Similarly, while FSGNN performs similar to GLINKX on the homophilous datasets, we observe a big performance drop on the heterophilous datasets (see arxiv-year).
We ablate each component of Alg.
1 to see each component’s performance contribution.
We use the hyperparameters of the best model from Tab. 1.
We perform two types of ablations: (i) we remove each of the components from all stages of the training, and (ii) we remove the corresponding components only from the 3rd stage of Alg. 1.
Except for removing the PEs from the 3rd stage only on ogbn-arxiv, all components contribute to increasing performance on both datasets (where adding PEs on the 1st stage improves performance).
[Table]
[Table]
Conclusion & Future Work
We present GLINKX, a scalable method for node classification in homophilous and heterophilous graphs that combines three components: (i) ego embeddings, (ii) PEs, and (iii) monophilous propagations.
Our method is complementary to what other methods propose since we can incorporate extra components such as feature propagations, label propagations, and attention to GLINKX.
As future work, (i) GLINKX can be extended in heterogeneous graphs, (ii) use more expressive methods such as attention or Wasserstein barycenters [9] for averaging the low-dimensional messages, and (iii) add complementary signals.
Acknowledgements
The authors would like to thank Maria Gorinova, Fabrizio Frasca, Sophie Hilgard, Ben Chamberlain, and Katarzyna Janocha from Twitter Cortex Applied Research for the useful discussions and comments on our work.
MP would also like to thank Felix Hohne for his help regarding questions about the LINKX codebase.
A Model Addendum A.1 Inductive Setting
In the paper we focus on the transductive setting.
Here we show how we can extend our framework to the inductive setting.
In the inductive setting, during training we only have access to the training graph
G train ( V train , E train ) .
During test, the whole graph G ⊆ G train is revealed and we make predictions for the test set.
The stages of GLINKX in the inductive setting are as follows: 1st
Stage (KGEs). For the KGE pretraining stage, we train KGEs on G train .
Then we can use existing methods in the literature such as [13, 1] to infer the KGEs of the test nodes. 2nd Stage (MLaP).
We train the model that predicts the distribution of the neighbors on G train as in Alg. 1.
Then for the test nodes, we know their ego features x i and we can also infer their PEs (see above) from the pre-trained PEs on the training set.
We use these and the pre-trained shallow model in order to predict ˜
y
i for the test nodes. From the soft-labels ˜ y i we calculate y ′ i 3rd Stage.
First, we train the second shallow model by using the propagated soft-predictions only on
G train .
Then, we infer the soft-labels ˜
y i from the 2nd stage and propagate them back to the original nodes.
We then use the ego features, PEs and the propagated information y
i to make predictions on the test set.
Test set inference is illustrated by Alg.[-
-]
Algorithm
Input: Graph G ( V, E ) with train set V train ⊆
V and test set V test ⊆
V , node features X , labels Y
Output: Predictions for all nodes i ∈ V test
Pre-training.
Call Alg.
with input graph G train ( V train , E train ) and compute PEs { p i } i ∈ V train , and pre-trained models f 1 (from 2nd stage) and f 2 (from 3rd stage) 1st Stage (KGEs).
Create PEs { p i } i ∈ V test for the nodes of the test set (see e.g. [13, 1]) 2nd Stage (MLaP).
Predict the distribution of neighbors for the nodes of the test set as ˜ y
i =
f 1 ( x i , p i ) for all i ∈ V test .
Propagate labels backwards and train model to predict a node’s own labels Calculate y ′ i
= (cid:80) j ∈ V : j → i ˜ y
j |{ j ∈ V : i → j }| for all i ∈ V test .
3rd Stage (Final Model).
Compute y final ,
i = f 2 ( x i , p
i , y ′ i ) for all i ∈ V test .
Return { y final ,i } i ∈ V test
A.2 Scalability
The propagations have to be done constant number (ideally once) of times by paying an O ( mc ) cost, where the dimensionality of classes c is usually small (compared to d X that GCNs rely on).
In both stages 2 and 3 of Alg. 1 we train node-level MLPs which allow us to leverage i.i.d. (row-wise) minibatching, like tabular data, and thus our complexity is similar to other shallow methods (LINKX, FSGNN) [22, 25]; meeting our requirements.
This, combined with the propagations that happen outside of the training loops, circumvent the issues of GCNs in large-scale graphs.
Cost of 1st Stage.
For the first stage, we are using Pytorch-Biggraph to train PEs.
Please refer to [20] for more information.
Cost of 2st Stage.
For the second stage we pay once O ( m train c ) ( m train =
| E train | is the number of edges in the graph induced by the train nodes) to run MLaP Forward and then the MLP that takes x i and p i with L X layers for the features x i and L P layers for the PEs p i , and L agg layers for the ResNet and hidden dimension h , has forward pass complexity O (cid:0) n ( d X h
+ h 2 L X
+ d P h
+ h 2 L P + h 2 L agg ) (cid:1) .
Finally, we pay again once O ( m train c ) for MLaP Backward.
Cost of 3rd Stage.
For the 3rd Stage, we pay O (cid:18)
[Equation]
h 2 L agg ) (cid:19) for a forward pass of the MLP, where c is the class dimensionality and L prop are the layers for the MLP handling the propagated labels y
i .
Comparison with LINKX.
Compared to LINKX we pay an O ( m train c ) extra cost once.
Regarding embedding the adjacency matrix LINKX pays a O ( mh + nh 2 L P ) cost for generating the h -dimensional adjacency embeddings whereas we pay O ( nd P h + nh 2 L P ) which is better for d P
= O ( m/n ) .
This holds in most real-world large networks since d P ∼ 10 2 whereas n ∼ 1B and m ∼ 100B .
Also note that using KGEs as PEs has an additional benefit, since KGEs can be trained only once and can be used off-the-shelf for other downstream tasks.
Inference Complexity (Alg. 2).
On the inductive setting, given pretrained models f 1 , f 2 from stages 2 and 3, the runtime Alg. 2 without the pretraining stage is
O (cid:0) mc + n test ( d X h
+ h 2 L X
+ d P h
+ h 2 L P + h 2 L agg ) (cid:1) to generate predictions for the test nodes, where n test =
| V test | .
A.3 Monophilous Label Propagation (MLaP): Motivation
In monophily, a node shares labels with its neighbors’ neighbors.
One way to encode monophily in Fig.
1(a) while predicting the label for j i is to get a distribution of labels of nodes connected to node i , thus encoding its neighbors’ distribution.
The fact that there are more nodes with green color than other colors can be used by the model to make a prediction.
But this information may only sometimes be present, or there may be few labeled nodes around node i .
For this reason, we propose to use a model that predicts the label distribution of nodes connected to node i .
And we use the node features ( x i ) and PE ( p i ) of node i to build this model because nodes that are connected to node i share similar labels and hence, the features of node i must be predictive of its neighbors.
So, in Stage 2, for every node, we train a model to predict the distribution of its neighbors.
We start with the objective to provide node j i with the neighbors’ label distribution of node i .
Then, we propagate this information to j i , thus encoding monophily. Finally, in Stage 3, we train a model that uses the node features, the PEs, and the propagated labels.
B Experiments Addendum B.1 Implementation and Environment
GLINKX is implemented in Pytorch-Geometric.
For the knowledge graph embeddings we use the official Pytorch-Biggraph implementation.
We use the official implementation of LINKX.
For hardware we used Vertex AI notebooks with 8 NVIDIA Tesla V100 with 16GB of memory and[-
-]
CPUs.
B.2 Experimental Protocol
Error Bars.
For the PubMed dataset and the heterophilous datasets provided by [28, 22, 40] we used the fixed splits provided with a fixed seed.
For the OGB datasets, where there exists only one officialy released split for each dataset [15] we generate the error bars by runnning the respective experiments with 10 different seeds (0-9).
B.3 Datasets
For our experiments we use the following datasets (see Tab. 1):
• PubMed [40, 32].
The PubMed dataset consists of scientific publications from PubMed database pertaining to diabetes classified into one of three classes.
Each node is described by a TF-IDF weighted word vector from a dictionary which consists of 500 unique words.
• ogbn-arxiv [15, 5, 34].
The ogbn-arxiv dataset is a directed graph, representing the citation network between all CS papers on arxiv mined from the Microsoft Academic Graph (MAG). [Table]
[Table]
[Figure]
• squirrel [31].
The data represents page-to-page networks on squirrels mined from Wikipedia between October 2017 and November 2018.
Node represent articles and edges are mutual links between the pages.
The features of each node are binary and represent the existence of informative nouns that appear on the corresponding Wikipedia pages.
• arxiv-year, yelp-chi
The arXiv-year dataset is derived from the ogbn-arxiv dataset where the labels have been changed in order to convert the dataset from homophilous to heterophilous.
Instead of predicting each paper’s subcategories, the new task focuses on predicting the year that a paper is posted, where batches of years have been converted to labels.
The yelp-chi dataset includes hotel and restaurant reviews filtered (spam) and recommended (legitimate) by Yelp.
The graph structure comes from [11] and the features from [32].
The task is to predict whether a review is a spam or not.
C Further Related Work
Homophilous Methods.
The backbone of graph learning is based on homophily.
A graph is homophilous if connected nodes have similar labels (and usually similar features).
In such settings, propagating node embeddings and then combining these embeddings with a permutation-invariant function and perhaps an edge-wise attention mechanism yields decent results [19, 37].
Recently, much attention has been given to large-scale settings where conventional GNNs – such as GCN and
GAT – cannot scale.
Such methods are based on multiple propagations of the features over layers, and back-propagating through the weights is an infeasible task for large-scale networks.
For this reason, several methods have been proposed to tackle this computational bottleneck which can scale to networks with hundreds of millions of nodes.
Conceptually, the most characteristic large-scale method is SIGN [30], where multiple stages of non-trainable filters are applied to the initial features to create node embeddings – which can be stored for later use. Then a shallow model (MLP) is applied on top of the final embeddings to produce the final predictions.
Various follow-up methods tweak the SIGN architecture to add attention (SAGN [36]), softmax regularization, soft-selection, and hop-normalization (FSGNN [25]) and improve upon its performance.
A related method is also GAMLP [41], where the authors combine label propagations and feature propagations and combine them with an attention mechanism.
However, their method does not take into account heterophilous inductive biases.
Another important method for large-scale homophilous node classification is Correct-And-Smooth (C&S) [16].
C&S lifts the assumption that GNNs are essentially necessary for good performance on graph ML tasks.
They propose to train a shallow classifier over the nodes and produce a set of soft labels from its predictions.
Afterward, the authors leverage the graph structure and the label correlations in two steps: an error correction step that spreads residual errors in training data to correct the errors present in the test data and a prediction correlation that smooths the predictions on the test data.
Heterophilous Methods.
Recently, there has been a surge in research regarding node classification in heterophilous graphs, i.e., graphs where connected nodes may have different class labels (and maybe different features).
One of the first proposed methods is H2GCN [46], where the authors introduce a set of design principles for heterophilous model design.
They propose the following design principles: ego and neighbor embedding separation, higher-order neighborhoods, and a combination of intermediate representations.
Accordingly, they build a model – H2GCN – based on such principles and evaluate it on various real-world and synthetic datasets.
Following up this work, the authors of [22] propose a significantly simpler architecture – LINKX – which combines embeddings of the ego features and embeddings of the corresponding rows of the adjacency matrix (inspired by the LINK method [42]) with a ResNet block [14].
The LINKX method is very lightweight since nodes are processed on an i.i.d.
(row-wise) basis – i.e., there are no graph dependencies between the nodes – and mini-batching can be performed efficiently (contrary to propagation-based methods).
LINKX is shown to outperform H2GCN on a variety of datasets.
However, LINKX falls short in node classification tasks that regard homophilous datasets (such as Cora, CiteSeer, and PubMed [40]) where standard methods such as GCN and GAT yield better results.
The recent framework of [23] argues that not all cases of heterophily are harmful to GNN-based architectures and propose a set of metrics based on the construction of a similarity matrix that measures the influence of the graph and the input features on GNNs.
Based on their observations, the authors design a model based on a filterbank which consists of a high-pass and a low-pass filter to address harmful heterophily achieving state-of-the-art results.
Methods for Homophily and Heterophily.
Recently, there have been lots of methods that have been adapted to work both in homophily and heterophily regimes.
H2GCN [46] (see above) was one of the first methods shown to work in both datasets.
RAW-GNN [17] is a random-walk-based GCN that exploits both homophily and heterophily by doing random walks and aggregations in two ways: breadth-first for homophily and depth-first for heterophily.
CPGNN [45] is a GCN-based architecture that uses a compatibility matrix for modeling the heterophily or homophily level in the graph, which can be learned in an end-to-end fashion, enabling it to go beyond the assumption of strong homophily.
GPR-GNN [8] addresses feature over-smoothing and homophily/heterophily by combining GNNs with Generalized PageRank techniques, where each step of feature propagation is associated with a learnable weight.
The amplitudes of the weights trade off the degree of smoothing of node features and the aggregation power of topological features.
However, all the above methods suffer from the same scalability issues that GCNs have.
Regarding scalable methods for both homophily and heterophily, the FSGNN [25] method is also closely related to our work and relies on using propagations which are separate from the training (similarly to SIGN) and also uses separate (higher-order) feature propagations for homophily and heterophily.
On the contrary, our method, in its current form, is one-hop and relies on different components to address homophily and heterophily.
Besides, their components can be added to our method, together with the ego features, the PEs, and the propagations, and thus, from another viewpoint, their work can be seen as complementary to ours.
CLP [44] combines a shallow (base) predictor model with a modified Label Propagation that works both in homophily and heterophily, where they use a class compatibility matrix (similarly to CPGNN) for the Label Propagation step.
Our method is fundamentally different than theirs; however, incorporating a compatibility matrix for the propagation stage constitutes interesting future work.
Positional Embeddings.
Following up on the recent success of LINKX in classifying nodes in heterophilous settings based partially on the position of each node (adjacency embedding), a series of methods have been suggested for incorporating positional embeddings on graph methods [18, 12].
More specifically, [12] proposes the MLPGNN-LSPE architecture, which is able to simultaneously learn both structural and positional embeddings for nodes, whereas [18] proposes TokenGT, which treats all nodes and edges as independent tokens, augments them with positional embeddings – i.e., eigenvectors of the normalized Laplacian of the graph ignoring edge directions – and then feed them to a Transformer model.
They prove that their framework is at least as expressive as a 2-IGN [24] composed of equivariant linear layers, which is more expressive than all message-passing GNNs.
However, creating positional embeddings that require the factorization of, e.g., the Laplacian matrix to compute a set of orthonormal eigenvectors is infeasible for large datasets.
For this reason, various methods have been proposed for embedding nodes in a graph in a scalable manner, such as TransE [6],
DistMult [39] 6 .
The recent development of Pytorch-Biggraph [20] allows training such embeddings on billion-scale heterogeneous graphs such as the Twitter graph [13].
D Hyperparameters
For each of the methods we train for 200 epochs and report the test set accuracy on the epoch with the best validation accuracy.
D.1 KGEs
We use the following hyperparameters for the KGEs using Pytorch-Biggraph:
• dimension =•
50 epochs • negative samples =• batch size =• dot comparator, softmax loss [39] • learning rate = [-0.
-]0.
D.2 GLINKX w/ Adjacency D.2.1 Sweeps
We perform the following sweeps:
• glinkx_init_layers_X ∈ { 1 , 2 } • glinkx_init_layers_A" ∈ { 1 , 2 } • glinkx_init_layers_agg ∈ { 1 , 2 } • glinkx_inner_dropout : ∈ { 0 . 5 } • lr ∈ { 0 . 1 , 0 .
001 } • optimize:r: AdamW
D.3 GLINKX w/ KGEs D.3.1 Sweeps
We perform the following sweeps for all datasets
[Table]
• glinkx_init_layers_X ∈ { 1 , 2 } • glinkx_init_layers_A ∈ { 1 , 2 } • glinkx_init_layers_agg ∈ { 1 , 2 } • glinkx_inner_dropout : ∈ { 0 . 5 } • lr ∈ { 0 . 1 , 0 . 001 } • optimize:r: AdamW • biggraph_vector_length:
[Table]
D.4 GAT D.4.1 Sweeps
• gat_num_layers ∈ { 1 } • gat_hidden_channels ∈ { 4 , 8 , 16 , 32 } • gat_num_heads ∈ { 2 , 4 } • lr ∈ { 0 . 01 , 0 .
001 } name
[Table] [Table]
D.5 GCN D.5.1 Sweeps
We perform the following sweeps:
• gcn_num_layers ∈ { 1 } • gcn_hidden_channels ∈ { 4 , 8 , 16 , 32 , 64 } • lr ∈ { 0 . 01 , 0 . 001 }
[Table]
D.6 FSGNN
We run the following sweeps for FSGNN
• layers = 1 for the 1-layer case and layers ∈ { 2 , 3 } for the higher-order case • hidden channels ∈ { 32 , 64 , 128 } • learning rate ∈ { 0 . 01 , 0 .
001 } • layer normalization ∈ { true , false }
D.7 Label Propagation
We run the following sweeps using the implementation of Label Propagation from [22]:
• α ∈ { 0 . 01 , 0 . 1 , 0 .
75 , 0 . 99 } • hops ∈ { 1 , 2 }
[Figure] [Figure]