E FFICIENT R EPRESENTATION L EARNING OF S UBGRAPHS BY S UBGRAPH -T O -N ODE T RANSLATION
A BSTRACT
A subgraph is a data structure that can represent various real-world problems.
We propose Subgraph-To-Node (S2N) translation, which is a novel formulation to efficiently learn representations of subgraphs.
Specifically, given a set of subgraphs in the global graph, we construct a new graph by coarsely transforming subgraphs into nodes.
We perform subgraph-level tasks as node-level tasks through this translation.
By doing so, we can significantly reduce the memory and computational costs in both training and inference.
We conduct experiments on four real-world datasets to evaluate performance and efficiency.
Our experiments demonstrate that models with S2N translation are more efficient than state-of-theart models without substantial performance decrease.
I NTRODUCTION
Graph neural networks (GNNs) have been developed to learn representations of nodes, edges and graphs (Bronstein et al., 2017; Battaglia et al., 2018;
Zhou et al., 2020).
Recently, Alsentzer et al.
(2020) has proposed SubGNN, a specialized architecture for learning representations of subgraphs.
This architecture outperforms prior models; however, it requires a lot of memory and computations to learn the non-trivial structure and various attributes in subgraphs.
In this paper we propose ‘Subgraph-To-Node (S2N)’ translation, a novel method to create data structures to solve subgraph-level prediction tasks efficiently.
The S2N translation constructs a new graph where its nodes are original subgraphs and its edges are relations between subgraphs.
The
GNN models can encode the node representations in the translated graph.
Then, we can get the results of the subgraph-level tasks by performing node-level tasks from these node representations.
For example, in a knowledge graph where subgraphs are diseases, nodes are symptoms, and edges are relations between symptoms based on knowledge in the medical domain, the goal of the diagnosis task is to predict the type of a disease (i.e., the class of a subgraph).
Using S2N translation, we can make a new graph of diseases, nodes of which are diseases and edges of which are relations between them (e.g., whether two diseases share any symptoms).
The S2N translation enables efficient subgraph representation learning for the following two reasons.
First, it provides a small and coarse graph in which the number of nodes is reduced to the number of original subgraphs.
We can load large batches of subgraphs on the GPU and parallelize the training and inference.
Second, there is a wider range of models to choose from in encoding translated graphs.
We confirm that even a simple pipeline of DeepSets (Zaheer et al., 2017) and GCN (Kipf &
Welling, 2017) can outperform state-of-the-art models.
We conduct experiments with four real-world datasets to evaluate the performance and efficiency of S2N translation.
We measure the number of parameters, throughput (samples per second), and latency (seconds per forward pass) for efficiency (Dehghani et al., 2021).
We demonstrate that models with S2N translation are more efficient than the existing approach without a significant performance drop.
Even some models perform better than baselines in three of the four datasets.
[Equation] [Figure] [Figure] [Figure]
[Figure] [Figure] [Figure] [Figure]
[Figure] [Figure]
[Figure]
[Figure]
S UBGRAPH -T O -N ODE T RANSLATION
We introduce the Subgraph-To-Node (S2N) translation and our specific design choices.
We also suggest model families for the subgraph prediction task using S2N translated graphs.
Notations We first summarize the notations in the subgraph representation learning, particularly in the subgraph classification task.
Let G =
( V , A , X ) be a global graph where V is a set of nodes ( | V | = N ), A ∈ { 0 , 1 } N × N is an adjacency matrix, and X ∈ R N × F 0 is a node feature matrix.
A subgraph S =
( V sub , A sub ) is a graph formed by subsets of nodes and edges in the global graph G .
For the subgraph classification task, there is a set of M subgraphs S
= {S 1 , S 2 , ..., S M } , and for
S i
=
( V sub i , A sub i ) , the goal is to learn its representation and the logit vector
y i ∈ R C where C is the number of classes.
Overview of S2N Translation
The S2N translation reduces the memory and computational costs in the model training and inference by constructing a new coarse graph that summarizes the original subgraph into a node.
As illustrated in Figure 1a, for each subgraph S i ∈ S in the global graph G , we create a node ˆ v i
= T v ( S i ) in the translated graph ˆ G ; for all pairs ( S i , S j ) of two close subgraphs in G , we make an edge ˆ e ij =
T e ( S i , S j ) between corresponding nodes in ˆ G .
Here, T v and T e are translation functions for nodes and edges in ˆ G , respectively.
Formally, the S2N translated graph ˆ G =
(ˆ V , ˆ A ) where | ˆ V | =
M and ˆ A ∈ { 0 , 1 } M × M is defined by
[Equation]
We can choose any function for T v and T e .
They can be simple heuristics or modeled with neural networks to learn the graph structure (Franceschi et al., 2019;
Kim & Oh, 2021; Fatemi et al., 2021).
Detailed Design of S2N Translation In this paper, we choose straightforward designs of T v and
T e with negligible translation costs.
For T v , we use a function that ignores the internal structure
A sub i of the subgraph S i
=
( V sub i , A sub i ) and treats the node as a set (i.e., V sub i ).
For T e , we make an edge if at least one common node between two subgraphs S i and S j .
They are defined as follows:
[Equation]
In some cases, this particular translation provides a more intuitive description for real-world problems than a form of subgraphs.
For a fitness social network ( EM-User ) from Alsentzer et al.
(2020) (subgraphs: users, nodes: workouts, edges: whether multiple users complete workouts), it will be translated into a network of users connected if they complete the same workouts.
This graph directly expresses the relation between users and follows the conventional approach to express social networks where nodes are users.
Models for S2N Translated Graphs We propose simple but strong model pipelines for S2N translated graphs.
Since the node ˆ v i is a set of original nodes in S i , we first use a set encoder
E set : ˆ V → R F (Wagstaff et al., 2021) where F is a dimension of the representation.
It takes a set of [Table] node features in ˆ v i as an input and generates the representation ˆ h i ∈ R F of ˆ v i , that is,
[Equation]
Then, given the node representation ˆ h i , we apply a graph encoder E graph : R M × F × [Equation] } M × M →
R M × C to get the logit vector ˆ y i ∈ R C .
For the input and output of E graph , we use matrices ˆ H ∈
R M × F and ˆ Y ∈ R M × C where the i th rows are ˆ h
i and ˆ y i , respectively.
[Equation]
For E graph , we can take any GNNs that perform message-passing between nodes.
This node-level message-passing on translated graphs is analogous to message-passing at the subgraph level in SubGNN (Alsentzer et al., 2020).
E XPERIMENTS
This section describes the experimental setup, including datasets, training, evaluation, and models.
Datasets We use four real-world datasets, PPI-BP , HPO-Neuro , HPO-Metab , and EM-User , introduced in Alsentzer et al.
The task is subgraph classification where nodes V , edges A , and subgraphs S ∈ S are given in datasets.
There are two input node features X pretrained with GIN or
GraphSAINT from the same paper.
Detailed description and statistics are in Appendix B.
Training and Evaluation
In the original setting from the SubGNN paper, evaluation (i.e., validation and test) samples cannot be seen during the training stage.
Following this protocol, we create different S2N graphs for each stage using train and evaluation sets of subgraphs ( S train and S eval ).
For the S2N translation, we use S train only in the training stage, and use both S train ∪ S eval in the evaluation stage.
That is, we predict unseen nodes based on structures translated from S train ∪ S eval in the evaluation stage.
In this respect, node classification on S2N translated graphs is inductive.
Models for S2N Translated Graphs We use two- or four-layer DeepSets (Zaheer et al., 2017) with sum or max operations as E set for all S2N models.
For E graph , we use well-known graph neural networks: GCN (Kipf & Welling, 2017) and GAT (Veliˇckovi´c et al., 2018).
In addition, LINKX (Lim et al., 2021) and FAGCN (Bo et al., 2021), models that perform well in non-homophilous graphs are employed.
All GNNs are one- or two-layer models.
See Appendix C.1 for their hyperparameters.
Since LINKX is designed for the transductive setting, we make a small change in LINKX to work in the inductive setting.
We call this variant LINKX-I. See Appendix C.2 for this modification.
Baselines We use current state-of-the-art models for subgraph classification as baselines:
Sub2Vec (Adhikari et al., 2018), Graph-level GIN (Xu et al., 2019), and SubGNN (Alsentzer et al., 2020).
We report the best performance among three variants for Sub2Vec (N, S, and NS) and two results by different pretrained embeddings for SubGNN.
All baselines results are reprinted from
Alsentzer et al.
R ESULTS
In this section, we analyze the characteristics of S2N translated graphs and compare our models and baselines on classification performance and efficiency.
We mark with daggers ( † ) the reprinted from et
[Table] Analysis of S2N Translated Graphs Table 1 summarizes dataset statistics before and after S2N translation, including node (Pei et al., 2020) and edge homophily (Zhu et al., 2020).
Except for
HPO-Neuro , translated graphs have a smaller number of nodes ( × 0 . 006 – × 0 . 03 ) and edges ( × 0 .
– × 0 . 78 ) than original graphs.
For HPO-Neuro , it has twice as many edges as the original graph, but has × 0 .
27 fewer nodes.
Since the number of edges decreased less than nodes, translated graphs are denser than originals ( × 9 . 7 – × 297 ).
We also find that they are non-homophilous (low homophily), which means there are many connected nodes of different classes.
Note that we propose multi-label node and edge homophily for multi-label datasets ( HPO-Neuro ):
[Equation]
where L v is a set of labels of v , N ( v ) is a set of neighbors of v , and A = { ( u, v ) |
A [ u, v ] = 1 } .
They generalize the existing multi-class homophily and we discuss more in Appendix D.
Performance In Table 2, we report the mean and standard deviation of micro-F1 score over ten runs of our models and baselines.
LINKX-I and FAGCN, which are known to work well in nonhomophilous graphs, perform on par with or better than the best baseline in 12 of 16 cases.
Here, ‘performance on par with the baseline’ implies no significant difference from the t -test at a level of 0.05 ( ∼ : p -value > . 05 ), which does not mean that our model is superior.
For PPI-BP and HPO-Metab , some models even outperform SubGNN with statistical significance ( ⋆ : p -value < . 05 ).
Notably, all S2N models outperform SubGNN in the PPI-BP , which has relatively high homophily.
GCN and GAT underperform LINKX-I and FAGCN for most experiments.
Efficiency In Figure 2, we show the number of parameters, throughput (subgraphs per second), and latency (seconds per forward pass) of S2N models and SubGNN on HPO-Neuro , HPO-Metab , and EM-User .
We cannot experiment with PPI-BP since it takes more than 48 hours in pre-computation.
We make three observations in this figure.
First, S2N models use fewer parameters and process many samples faster (i.e., higher throughput and lower latency) than SubGNN.
In particular, for throughput, S2N models can process 8 to 300 times more samples than SubGNN for the same amount of time.
Second, the training throughput is higher than the inference throughput in S2N models.
Generally, as in SubGNN, throughput increases in the inference step, which does not require gradient calculation.
This is because the S2N models use message-passing between training and inference samples (See §3).
Thus, they compute both training and inference samples, requiring more computation for the inference stage.
Lastly, as one exception to general trends, the training latency of GAT on HPO-Metab is higher than that of SubGNN.
Note that latency ignores the parallelism from large batch sizes (Dehghani et al., 2021).
Our model can show relatively high latency since it requires full batch computation.
See Appendix E for the experimental setup. [Figure] [Figure] [Figure]
[Figure]
[Figure]
C ONCLUSION AND F UTURE R ESEARCH
We propose Subgraph-To-Node (S2N) translation, a novel way to learn representations of subgraphs efficiently.
Using S2N, we create a new graph where nodes are original subgraphs, edges are relations between subgraphs, and perform subgraph-level tasks as node-level tasks.
S2N translation significantly reduces memory and computation costs without performance degradation.
There are limitations in this research.
First, we used simple translate functions and did not explore them deeply.
How do we define aggregated features and structures in translated graphs?
Second, we do not yet know the properties of subgraphs that affect the performance of the S2N translation.
What properties of subgraphs can be learned after translation?
We leave these as future directions.
A CKNOWLEDGMENTS
This research was supported by the Engineering Research Center Program through the National Research Foundation of Korea (NRF) funded by the Korean Government MSIT (NRF2018R1A5A1059921).
A R ELATED W ORK
Subgraph Representation Learning There have been various approaches to use subgraphs for expressiveness (Niepert et al., 2016; Morris et al., 2019;
Bouritsas et al., 2020), scalability (Hamilton et al., 2017; Chiang et al., 2019; Zeng et al., 2020)
[Table]
and augmentation (Qiu et al., 2020; You et al., 2020).
However, only a few studies deal with learning representations of subgraphs.
The Subgraph Pattern Neural Network (Meng et al., 2018) learns subgraph evolution patterns but does not generalize to subgraphs with varying sizes.
The Subgraph Neural Network (SubGNN) (Alsentzer et al., 2020) is the first approach of subgraph representation learning using topology, positions, and connectivity.
However, SubGNN requires large memory and computation costs to encode the mentioned information for prediction.
Our method allows efficient learning of subgraph representations without a complex model design.
Graph Coarsening Our S2N translation summarizes subgraphs into nodes, and in that sense, it is related to graph coarsening methods (Loukas & Vandergheynst, 2018; Loukas, 2019; Bravo Hermsdorff & Gunderson, 2019; Jin et al., 2020; Deng et al., 2020;
Cai et al., 2021; Huang et al., 2021).
They focus on creating coarse graphs while preserving specific properties in a given graph, such as spectral similarity or distance.
The difference between them and ours is whether the node boundaries in coarse graphs (or super-nodes) are given or not. Super-nodes are unknown in existing works of graph coarsening; thus, algorithms to decide on super-nodes are required.
In S2N translation, we treat subgraphs as super-nodes and can create coarse graphs with simple heuristics.
B D ATASETS
Subgraph datasets PPI-BP , HPO-Neuro , HPO-Metab , and EM-User are proposed in Alsentzer et al.
(2020), and can be downloaded from the GitHub repository 1 .
In Table 3, we summarize statistics of datasets in original forms without S2N translation.
We describe their nodes, edges, subgraphs, tasks, and references in the following paragraphs.
PPI-BP The global graph of PPI-BP (Zitnik et al., 2018;
Subramanian et al., 2005; Consortium, 2019; Ashburner et al., 2000) is a human protein-protein interaction (PPI) network; nodes are proteins, and edges are whether there is a physical interaction between proteins.
Subgraphs are sets of proteins in the same biological process (e.g., alcohol bio-synthetic process).
The task is to classify processes into six categories.
HPO-Neuro and HPO-Metab
These two HPO (Human Phenotype Ontology) datasets (Hartley et al., 2020; K¨ohler et al., 2019; Mordaunt et al., 2020) are knowledge graphs of phenotypes (i.e., symptoms) of rare neurological and metabolic diseases.
Each subgraph is a collection of symptoms associated with a monogenic disorder.
The task is to diagnose the rare disease: classifying the disease type among subcategories (ten for HPO-Neuro and six for HPO-Metab ).
EM-User EM-User (Users in EndoMondo) dataset is a social fitness network from Endomondo (Ni et al., 2019).
Here, subgraphs are users, nodes are workouts, and edges exist between workouts completed by multiple users.
Each subgraph represents the workout history of a user.
The task is to profile a user’s gender.
C M ODELS
This section describes the model details we used: hyperparameter tuning and LINKX-I design.
All models are implemented with PyTorch (Paszke et al., 2019), PyTorch Geometric (Fey & Lenssen, 2019), and PyTorch Lightning (Falcon & The PyTorch Lightning team, 2019).
C. H YPERPARAMETERS
We tune seven hyperparameters using TPE (Tree-structured Parzen Estimator) algorithm in Optuna (Akiba et al., 2019) by 30 trials: weight decay ( 10 − 9 – 10 − 6 ), the number of layers in E set ( or 4), the number of layers in E graph (1 or 2), the pooling operating in E set (sum or max), dropout of channels and edges ( { 0 . 0 , 0 .
1 , ..., 0 . 5 } ), and gradient clipping ( { 0 . 0 , 0 .
We use batch normalization (Ioffe & Szegedy, 2015) for all S2N models except LINKX-I.
C. I NDUCTIVE LINKX (LINKX-I)
Given node features X ∈ R N × F 0 and an adjacent matrix A ∈ R N × N , LINKX (Lim et al., 2021) model computes the logit matrix Y ∈ R N × C by following equations,
[Equation]
[Equation]
W f ∈ R F × 2 F (7)
The computation of the first single layer in MLP A =
Linear A ◦ ReLU ◦
Linear A ◦ ... is as follows
[Equation]
In our inductive setting, we have ˆ A train and ˆ A train + eval , the shapes of which are
[Equation]
If we train MLP A on ˆ A train , we cannot process ˆ A train + eval , because shapes of matrix multiplication do not match (i.e., M train + M
eval ̸ = M train ).
Thus, in LINKX-I, we use the modified matrix multiplication ⊛ in MLP A to aggregate parameters corresponding training nodes only.
Formally, for the matrix ˆ A M ∗ ∈ R M ∗ × M ∗ of the arbitrary shape,
[Equation]
The remaining parts are the same as LINKX.
D G ENERALIZATION OF H OMOPHILY TO M ULTI - LABEL C LASSIFICATION
Node (Pei et al., 2020) and edge homophily (Zhu et al., 2020) are defined by,
[Equation]
[Equation]
where y v is the label of the node v .
In the main paper, we define multi-label node and edge homophily by,
[Equation]
If we compute r = | L u ∩ L v | | L u ∪ L v | for single-label multi-class graphs, r = 11 = 1 for nodes of same classes, and r = 02 = 0 for nodes of different classes.
That makes h edge, ml = h edge and h node, ml
=
h node for single-label graphs.
E D ETAILS ON E FFICIENCY E XPERIMENTS
We compute throughput (subgraphs per second) and latency (seconds per forward pass) by following equations.
[Equation]
[Equation]
We use the best hyperparameters (including batch sizes) for each model and take the mean wallclock time over 50 epochs.
Our computation device is Intel(R)
Xeon(R) CPU E5-2640 v4 and single GeForce GTX 1080 Ti.