GLASS: GNN WITH L ABELING T RICKS FOR S UBGRAPH R EPRESENTATION L EARNING
Anonymous authors
Paper under double-blind review
A BSTRACT
Despite the remarkable achievements of Graph Neural Networks (GNNs) on graph representation learning, few works have tried to use them to predict properties of subgraphs in the whole graph.
The existing state-of-the-art method SubGNN introduces an overly complicated subgraph-level GNN model which synthesizes three artificial channels each of which has two carefully designed subgraph-level message passing modules, yet only slightly outperforms a plain GNN which performs node-level message passing and then pools node embeddings within the subgraph.
By analyzing SubGNN and plain GNNs, we find that the key for subgraph representation learning might be to distinguish nodes inside and outside the subgraph.
With this insight, we propose an expressive and scalable labeling trick, namely max-zero-one, to enhance plain GNNs for subgraph tasks.
The resulting model is called GLASS (GNN with LAbeling trickS for Subgraph).
We theoretically characterize GLASS’s expressive power.
Compared with SubGNN, GLASS is more expressive, more scalable, and easier to implement.
Experiments on eight benchmark datasets show that GLASS outperforms the strongest baseline by 13% on average.
And ablation analysis shows that our max-zero-one labeling trick can boost the performance of a plain GNN by up to 18% .
Furthermore, training a
GLASS model only takes 26% time needed for a SubGNN on average.
I NTRODUCTION
Graph is a natural tool for modeling objects with complex internal relationships, which is widely used in fields such as natural language processing (Yao et al., 2019), biology (Fout et al., 2017), and social network (Chen et al., 2018).
Among the various graph representation learning methods, GNN has achieved state-of-the-art performance on almost all sorts of tasks.
Existing GNNs are mainly designed for node (Dabhi & Parmar, 2020; Chen et al., 2020), edge (Singh et al., 2021; Galkin et al., 2021) and whole graph (Ying et al., 2021;
Yang et al., 2020) property prediction tasks.
An ordinary
GNN produces embeddings of a node by aggregating the features from the (multi-hop) neighbors of the node, which is equivalent to encoding a breadth-first-search (BFS) tree rooted in the node (Xu et al., 2019).
Such embeddings can be used to predict node properties directly.
As for edge and graph tasks, pooling the embeddings of nodes related to the structure is a prevailing method.
Though node, edge, and graph tasks are the three most common graph representation learning tasks, sometimes properties of subgraphs are also worth predicting.
Take company structure network as an example.
The nodes are employees, and the edges between them represent cooperation relations.
We want to predict the performance of a department, in other words, a subgraph in the network.
Obviously, on the one hand, we need to consider the internal organization (such as the collaboration within the department and the competence of individual employees).
If the structure is disorganized or the employees are incompetent, we can expect the department to perform poorly.
On the other hand, the external information of the department also deserves attention.
A department is less likely to be productive if the company as a whole is facing bankruptcy.
In contrast, close cooperation with other remarkable departments can be a sign of good performance.
As illustrated by this example, a subgraph task is to predict the property of subgraphs in the whole graph.
It needs to consider the topology both inside and outside the subgraph.
It may also need to combine multi-level information of nodes, edges, higher-order substructures, and even the whole graph.
Thus, a very general model is needed, and a natural idea is to extend ordinary GNNs to subgraph tasks.
Figure 1 left shows a typical subgraph to predict.
The target subgraph S is embedded in the whole graph and may have
[Figure]
multiple connected components, and our task is to produce a subgraph representation, which can be used to predict properties of S .
However, in experiments, we find that SubGNN (Alsentzer et al., 2020), the current state-of-the-art method for subgraph tasks, only slightly outperforms plain GNNs which directly pool node embeddings within the subgraph as the subgraph representation.
SubGNN replaces the message passing between nodes with a subgraph-level message passing framework and designs three channels.
Each channel is further divided into an internal and a border module to aggregate subgraph features separately.
Despite its performance, SubGNN is both space and time-consuming due to its lengthy precomputation.
Furthermore, the units of message passing are subgraph patches sampled from the whole graph randomly, and there is no guarantee of the optimality of the samples, leading to high variance in performance and dubious robustness.
Last but not least, the framework of SubGNN is overly complicated—many of its designs seem ad hoc or suboptimal, resulting in poor compatibility with recent advances in GNN research.
Nevertheless, by comparing SubGNN with plain GNNs, we find that differentiating internal and external topology is crucial for subgraph tasks.
And inspired by this insight, we introduce a kind of node labeling trick, which explicitly marks whether a node is within a subgraph or not, to augment GNNs and show that plain GNNs with this node labeling trick is superior to SubGNN.
Here we give a brief introduction to node labeling tricks.
A node labeling trick assigns a label to each node and combines the node features and the labels as the new input node features to GNNs.
Zhang et al.
(2020) show that producing expressive representations for high-order structures needs to capture the interaction among the different nodes within the structure.
They theoretically show that directly aggregating node representations to represent high-order structures is not expressive enough, and labeling tricks can aid this problem.
Node labeling tricks have achieved great successes on graph representation learning in previous works.
For example, the state-of-the-art link prediction method
SEAL (Zhang & Chen, 2018) gains better performance with carefully designed node labeling tricks.
IDGNN (You et al., 2021) differentiates one center node from others, and Distance Encoding (Li et al., 2020) uses the distance to the target nodes to label other nodes, both of which gain improved performance on node, edge, and graph tasks.
In this paper, we for the first time introduce node labeling tricks to subgraph problems, and design an expressive and scalable labeling trick called max-zero-one.
Max-zero-one is the first labeling trick that enables jointly predicting a batch of structures within the same graph.
Present work We propose GLASS (GNN with LAbeling trickS for Subgraph), a novel and simple graph neural network for subgraph tasks.
To the best of our knowledge, GLASS is the first subgraph representation learning method using the ordinary message passing framework and a node labeling trick.
GLASS is more scalable, more expressive, and easier to implement and extend than the existing state-of-the-art method.
Theoretically, we prove that GLASS is more expressive than plain
GNNs, and can capture a range of important subgraph properties defined in SubGNN like density, cut ratio, border, positions, etc.
Experiments on eight datasets show that GLASS achieves new state-of-the-art performance.
On synthetic datasets, GLASS with a single message passing layer beats SubGNN with three carefully designed channels by up to 50 .
9% , illustrating the expressive power of node-level message passing augmented by labeling trick for subgraph tasks.
On real-world datasets, GLASS also outperforms the strongest baseline SubGNN by up to 10 .
Moreover, training a GLASS model on average only takes 26% time needed to train a SubGNN.
R ELATED W ORK
Subgraph Representation Learning.
Though some works have utilized subgraphs to perform other graph representation tasks (Sun et al., 2021; Wang et al., 2021; Huang & Zitnik, 2020) or studied some specific tasks involving subgraphs (Bordes et al., 2014; Meng et al.,
Ying et al., 2020), few works have studied the general subgraph representation learning problem.
Alsentzer et al.
(2020) introduced the problem formally and proposed SubGNN (Alsentzer et al., 2020), the current state-of-the-art method, which samples patches from the whole graph and aggregates their features to produce subgraph representations.
Before that, Sub2Vec (Adhikari et al., 2018), designed for graph classification and community detection, samples random walks in subgraphs and feeds them to the language model Paragraph2vec (Le & Mikolov, 2014) to generate embeddings for subgraphs.
Node Labeling Tricks.
SEAL (Zhang & Chen, 2018) first introduces node labeling tricks to graph representation learning and applies them to link prediction.
IDGNN (You et al., 2021) uses different message passing parameters for a target node and the other nodes, which is essentially a node labeling trick assigning different labels to the target node and others.
Distance Encoding (Li et al., 2020) uses distances to target nodes as node labels.
Zhang et al.
(2020) give a theoretical analysis of labeling tricks and prove that they can produce the most expressive representations for substructures with a GNN expressive enough.
These previous methods ignore subgraph tasks and have poor scalability due to the relabeling for every target substructure to predict.
Besides these deterministic node labeling methods, rGIN (Sato et al., 2021) assigns a random vector to each nodes as its label in each forward process.
Similarly, GNN-RNI (Abboud et al., 2021) randomly initializes node embeddings and can approximate any functions mapping graphs to real numbers.
Despite the theoretical power, random labels suffer from slow convergence and subpar performance.
P RELIMINARIES
Let G =
( V , E , X ) denote a graph with a finite node set V = { 1 , 2 , ..., n } , an edge set E ⊆ V × V and node feature matrix X , whose i th row
X i is the feature of node i .
N ( v ) refers to the set of nodes adjacent to node v .
S =
( V S , E S , X S ) is a subgraph of G if V S ⊆ V and E S ⊆
( V S × V S )
∩ E and
X S is the stack of the rows of X corresponding to nodes in V S .
In this paper, we focus on induced subgraphs, whose edge set E S
=
( V S × V S )
∩ E .
Let S ⊆ G donate that S is a subgraph in G .
Problem (Subgraph Representation and Property Prediction) .
Given the whole graph G , its subgraphs S =
{S 1 , S 2 , ..., S n } and their target properties T = { t S 1 , t S 2 , ..., t S n } , the goal is to learn a representation vector h S i that can be used to predict t S i of S i .
A Plain GNN for Subgraph Tasks.
Message passing neural network (MPNN) (Gilmer et al., 2017) is a common framework of GNNs.
A message passing layer aggregates embeddings from neighbors to update the representation of a node.
The k th message passing layer can be formulated as follows.
[Equation]
where h ( k ) v is the embedding of node v at the k th layer and h (0) v
= X v .
The embeddings at the last layer can be used to predict node properties.
As for edge or graph tasks, pooling the multiset of embeddings of nodes belonging to the edge or within the graph is a widely used method.
Naturally, we can extend it to subgraphs, and the representation of a subgraph S can be learned by pooling the embeddings of nodes within the subgraph as follows, which we call a plain GNN .
[Equation]
[Table] [Table]
A C OMPARISON BETWEEN S UB GNN AND P LAIN GNN S
In this section
[Table]
we first illustrate the failure of plain GNNs on some simple subgraph tasks.
Then, to find the reason for the deficiency, we dive into the state-of-the-art subgraph representation learning method, SubGNN
[Table]
and compare it with plain GNNs.
We find that the key advantage of SubGNN comes from dealing with internal and border topology separately, which motivates our GLASS.
Though numerous works have proven plain GNNs’ graph representation learning ability, we find that they can fail in some simple subgraph representation situations.
For example, the representations of two different subgraphs S and S ′ in Figure 2 must be the same as all nodes in the graph have identical neighborhood structures and thus equal embeddings from plain GNNs.
The failure to differentiate
S and S ′ suggests the limitation of using plain GNNs for subgraph representation learning.
Unlike plain GNNs, the existing state-of-the-art method, SubGNN, designs a special message passing architecture.
SubGNN introduces three channels, namely position , neighborhood , and structure (P, N, S for short), and learns each channel’s internal and border (I, B for short) properties separately.
Table 1 defines the six properties.
They claim that these six properties are key for learning powerful subgraph representations.
To capture channel i for a target subgraph S , SubGNN samples anchor patches A i
= {A (1) i , ..., A ( n A ) i } and then learns representations of each connected component in S by propagating messages from the anchor patches to components of S .
See the top half of
Figure 1.
The k th subgraph-level message passing layer for channel i can be formulated as follows.
[Equation]
where S ( c ) is
the c th component of the target subgraph S , γ i ( S ( c ) , A i ) is a similarity function between S ( c ) and an anchor patch A
i , g A i is a (pretrained) representation of patch
A i , h ( k ) i, S ( c ) is the representation of the component S ( c ) in channel i at the k th layer, and σ is an activation function and W i is a weight matrix.
The implementation of each channel in SubGNN can be found in Table 2.
The representation of the whole subgraph is the sum of the representations of all components.
We call the above formulation subgraph-level message passing to distinguish it from the node-level message passing defined in Equations (1) and (2).
In general, SubGNN pretrains a plain GNN to produce node embeddings, pools them to produce patch embeddings and then smoothes the embeddings of the target component with the embeddings of patches close to or structurally similar to the target component.
Through the carefully designed subgraph-level message passing layer for each of the six properties separately, SubGNN can capture
𝕊 𝕊 ′          7 13       2  7 1  3 5
[Figure]
the internal structure, border connectivity, and position relative to the rest of the graph.
Looking back on plain GNNs, we wonder why they are inferior to SubGNN.
By encoding the topology of
BFS trees, plain GNNs can easily capture (multi-hop) neighborhood and the position of nodes in the whole graph.
However, from the example in Figure 2 we can see that plain GNNs cannot represent internal structure and border connectivity well, as they cannot tell whether a neighbor of a node is in the subgraph or in the rest of the graph .
Therefore, the missing part of plain GNNs from
SubGNN is that SubGNN passes internal and border messages separately, while plain GNNs cannot distinguish nodes in and outside the target subgraph when passing messages between nodes.
GLASS: G NNS WITH LA BELING TRICK S FOR S UBGRAPH
Inspired by SubGNN, we aim to differentiate nodes in the target subgraphs from the rest of the graph and see how such differentiation can improve plain GNNs.
Still using S and S ′ in Figure 2 as an example, the initially identical rooted trees become different when we differentiate internal and external nodes for each subgraph (Figure 2 right), leading to different node embeddings between the two subgraphs.
Therefore, GNNs can differentiate these two subgraphs after the pooling.
Inspired by this insight, we propose GLASS, which uses node labeling tricks to enhance plain GNNs.
5. T HE Z ERO -O NE L ABELING T RICK FOR S UBGRAPHS
We first introduce the zero-one labeling trick for subgraphs.
Definition 1.
Given a graph G and a subgraph S in G , the zero-one label of node v is
[Equation]
GLASS is a plain GNN augmented by the zero-one labeling trick.
The k th layer of GLASS is:
[Equation]
Briefly speaking, GLASS assigns nodes in and outside the subgraph different labels and enhances the hidden representation of a node in each layer with its zero-one label l ( S ) v .
Finally, the representation of S with a K -layer GLASS is
[Equation]
The bottom half of Figure 1 illustrates the GLASS framework.
The following proposition shows that GLASS is more expressive than plain GNNs.
Proposition 1.
Given a graph G and subgraphs S and S ′ in G , if plain GNNs can distinguish them, GLASS can also produce different representations of S and S ′ .
However, there exist pairs of subgraphs that plain GNNs cannot differentiate while GLASS can.
In addition, we prove that GLASS can precisely predict two critical metrics used in SubGNN to evaluate a model’s expressive power for subgraph representation learning, namely cut ratio and density, while plain GNNs cannot.
Theorem 1.
Given any graph G , there exists a GLASS model that can precisely predict the density and cut ratio of any subgraph in G .
Now we have seen that GLASS is a more powerful model than plain GNNs for subgraph representation learning.
We next compare GLASS with the state-of-the-art model, SubGNN, and analyze their expressive power differences.
We show that GLASS can represent all the channels of SubGNN.
Proposition 2.
GLASS can cover the six properties of SubGNN defined in Table 1.
The concrete analysis and proofs of Proposition 2 can be found in Appendix A.3.
In summary,
GLASS can learn all properties designed by SubGNN while using node-level message passing only.
In Appendix A.4, we further characterize the expressive power of GLASS using the structural representation theory (Zhang et al., 2020; Srinivasan & Ribeiro, 2020).
We prove that (1) With a GNN expressive enough, GLASS can learn the most expressive structural representations for subgraphs, thus enabling learning any functions over subgraphs (in theory can solve any tasks over subgraphs).
(2) However, the expressive power of practical GNNs is usually bounded by the 1-dimensional
Weisfeiler-Lehman (1-WL) test (Weisfeiler & Leman, 1968).
We prove that the zero-one labeling trick can also boost those GNNs as powerful as 1-WL for subgraph tasks
. 5. T HE M AX -Z ERO -O NE L ABELING T RICK
Though the zero-one labeling trick can boost GNNs, it makes batch training hard, as the input node feature vectors change with different target subgraphs.
Time-consuming message passing needs to be done separately for each target subgraph, resulting in higher time complexity than plain GNNs.
To alleviate this issue, we can combine the labels of different target subgraphs and produce their representations in a single forward process.
More specifically, we can jointly label a batch of subgraphs within the graph, perform message passing on the whole graph, and then pool individual node embeddings to produce multiple subgraph representations at the same time.
This enables learning a batch of subgraph representations from the same labeled graph instead of learning one subgraph representation from one labeled graph.
However, node labels might conflict with each other for different subgraphs, resulting in inconsistencies with the canonical way.
Nevertheless, we argue that such processing has a controllable effect on the final results.
On the one hand, as ordinary k -layer GNNs only encode neighbors within k hops, the representation will not change if the node labels of neighbors within k hops stay the same.
We can expect the errors to be negligible if the change in node labels is distant from the target subgraph.
This can be achieved if the subgraphs are sparsely located in the graph.
On the other hand, such inconsistencies provide a regularization effect, which prevents GNNs from overfitting the node labels of individual subgraphs but promotes learning general node features useful for multiple subgraphs.
How can we combine the node labels for different target subgraphs?
One solution is to concatenate them, which seems to have no loss of information.
However, this method leads to variable lengths of node labels, and depending on which other subgraphs attend this batch, it leads to possibly different node labels for the same target subgraph.
The sum of all labels has similar problems.
However, using the max of all labels avoids such a problem, which can help the model converge faster and facilitate processing samples in varied batch sizes.
Taking the zero-one labeling trick as an example, we will label a node by 1 if at least one subgraph in the batch contains this node.
Otherwise, we will label a node by 0.
We call such a labeling trick the max-zero-one labeling trick .
In our experiments, we uniformly used the max-zero-one labeling trick for GLASS and verified its fast training speed as well as excellent empirical performance.
Previous works have explored other ways to enable batch training.
For example, SEAL (Zhang &
Chen, 2018) segregates small enclosing subgraphs from the whole graph to perform link prediction tasks.
However, subgraphs usually contain much more nodes than links, resulting in huge enclosing subgraphs to be segregated.
Moreover, in some datasets, subgraphs have multiple components.
To capture the interaction among components, the depth of the enclosing subgraphs has to be large, leading to exponentially increasing subgraph sizes.
IDGNN (You et al., 2021) uses fabricated additional features instead of labels, which is equivalent to using a linear GNN to aggregate node labels, leading to loss of expressive power.
Moreover, in Appendix A.5, we show that no labeling trick can avoid the inconsistencies of node labels within a batch.
Therefore, it is impossible to solve this problem by designing a fancy labeling trick, and approximation is a must .
5. S ELF -S UPERVISED L EARNING
In practice, subgraph datasets are often not large, and target subgraphs are often sparse in the whole graph.
Thus, training GLASS only from the subgraph property prediction signals tends to overfit.
Therefore, we introduce three levels of self-supervised learning (SSL) tasks to assist the GLASS training.
1) Node level: We train a GNN to produce node embeddings to predict A k X , where A is the adjacency matrix of the whole graph.
Intuitively, it helps GNNs differentiate different BFS trees (Yehudai et al., 2021), thus preserving better local subtree information.
Edge level: We perform a link prediction task on the whole graph.
Intuitively, it pushes nearby nodes to have similar embeddings, thus encoding the node distance information into the node embeddings.
3) Subgraph level: We let the final subgraph representations predict some subgraph structural properties, including density, cut ratio, coreness, and the number of connected components.
It helps the model better capture some higher-order substructure information.
We take a pretraining strategy for node and edge-level SSL tasks, whose output node embeddings are used as initial features of the GLASS.
Then, we jointly train the subgraph-level SSL tasks and the main subgraph property prediction task.
Donate the loss for the main task, node-level SSL, edgelevel SSL, and subgraph-level SSL as L main , L node , L edge , L subg .
The pretraining loss is given by
L pre = αL node + βL edge , and the joint training loss is given by L joint =
L main + γL subg , where α, β, γ are hyperparameters.
Pretrained GNN embeddings can also help GLASS capture distant neighbors.
If the pretrained GNN has l layers and the GNN used in GLASS has k layers, it can capture ( l + k ) -hop neighbors, while a k -layer MPNN can only capture neighbors within k hops.
E XPERIMENT
In this section, we compare GLASS with state-of-the-art subgraph representation learning methods, especially SubGNN, on both synthetic and real-world datasets to demonstrate that GLASS is a model with superior performance and scalability, despite being much simpler than SubGNN.
6. D ATASETS AND B ASELINES
Datasets .
We use four synthetic datasets: density , cut ratio , coreness , component , and four real-world subgraph datasets, namely ppi-bp , em-user , hpo-metab , hpo-neuro .
The four synthetic datasets are introduced by Alsentzer et al.
(2020) to test a model’s ability to learn the six properties: density tests the ability to learn internal structure; cut ratio tests border structure; coreness tests border structure and position; and component tests internal and external position.
The four real-world datasets are also provided by Alsentzer et al.
The base graph of the ppi-bp dataset is a human protein-protein interaction network.
Each subgraph is induced by proteins in a biological process, whose label is its cellular function.
The hpo-metab and hpo-neuro datasets are knowledge graphs containing phenotype and genotype information about rare diseases.
Each subgraph represents a rare monogenic disease.
The em-user dataset
[Table] [Table]
contains the workout history of users, where each subgraph makes up a user’s workout history
[Table]
and the label is the gender of the user.
Detailed information on these datasets are in Appendix A.7.
Baseline Models .
We consider four baseline methods.
(1) SubGNN (Alsentzer et al., 2020) uses subgraph-level message passing with six artificial channels.
We use the numbers provided by the original paper.
(2) Sub2Vec (Adhikari et al., 2018)
samples random walks in subgraphs which are fed to Paragraph2Vec to train subgraph embeddings.
We train Sub2Vec using the official implementation to produce subgraph representations and then feed them to an MLP.
Like SubGNN, Sub2Vec can also capture neighborhood and structure without passing messages between nodes.
However, it uses node ID and degree ratio in the target subgraph to represent nodes, rather than pretraining GNNs.
Moreover, it only learns the internal topology by sampling random walks in the target subgraph.
We compare these two existing subgraph representation methods to show the superior performance of GLASS.
(3) GNN-seg is an ordinary MPNN performing graph classification task on subgraphs segregated from the whole graph.
We use it to illustrate that subgraph tasks need external topology.
(4) GNN-plain is GLASS without node labeling tricks.
Its performance can both illustrate the effect of the max-zero-one labeling trick and the expressive power of node-level message passing on subgraph tasks.
We detail our GLASS architecture in Appendix A.6.
6. R ESULTS
Synthetic Datasets .
To evaluate the expressive power of GLASS, we test GLASS and other baselines on four synthetic datasets.
Results are shown in Table 3.
GLASS significantly outperforms all baselines and outperforms the previous state-of-the-art model SubGNN by 20 .
7% on average and 50 .
9% in maximum.
The results show that GLASS can capture density and cut ratio information very well as proved.
Moreover, they illustrate the expressive power of ordinary node-level MPNNs for subgraph representation learning—we do not need to use subgraph-level message passing with complicated artificial channels, but a plain GNN with a simple labeling trick is enough.
As for GNNseg, its failure to predict cut ratio is expected as this property requires topology outside the subgraph.
However, the lower scores GNN-seg gains on coreness and density are hard to comprehend, as these two properties are closely correlated with internal structure.
We suspect that segregating subgraphs from the whole graph makes it difficult to leverage the correlation between different subgraphs, and the limited number of training subgraphs in the synthetic datasets makes this completely inductive setting much harder to train.
As for Sub2Vec, it cannot capture position information and external topology, thus its failure on cut ratio and component is reasonable.
Moreover, as it uses transductive features like node IDs to represent nodes, Sub2Vec has worse generalization ability.
Real-world datasets Table 4 shows the results.
GLASS gains a 6 . 0% increase on average and 10 .
5% increase in maximum compared with SubGNN.
On hpo-metab and hpo-neuro datasets, GNNseg beats other baselines very easily, which might be because the subgraphs in these two datasets are dense and localized, making the topology outside the subgraph less important.
In contrast, on ppi-bp and em-user , the density of subgraphs is lower, and some subgraphs are even composed of single nodes.
Thus GNN-seg can learn little from the internal topology as shown by the worse performance.
Sub2Vec is less expressive than SubGNN as expected.
However, on ppi-bp and em-user datasets, it has an edge over GNN-seg.
We suspect that the node ID can leak some exter-
[Table] [Table]
nal topology information to Sub2Vec, as the same nodes can appear in different subgraphs.
We find that GLASS performs especially well on em-user .
As the target subgraphs in this dataset contains hundreds of components and each component only has less than five nodes, directly pooling node embeddings rather than pooling component embeddings might avoid loss of information without introducing too much noise.
Ablation Analysis To demonstrate the power of the max-zero-one labeling trick, we do an ablation study by removing the max-zero-one labels from GLASS, which is called GNN-plain.
A comparison between GLASS and GNN-plain is shown in Table 5 (synthetic datasets) and 6 (real-world datasets)
[Table]
Our node labeling trick improves the performance by 33 . 1% on synthetic datasets and 3 . 9% on real-world datasets on average.
The relatively lower improvement on real-world datasets might be because improving the expressive power is not the most critical factor for real-world datasets, but rather, filtering noise and smoothing features is more effective.
In other words, the synthetic datasets require a much stronger structure learning ability than the real-world datasets to perform well.
This can also be seen from the comparison between SubGNN and GNN-seg in Table 4, where SubGNN only marginally outperforms GNN-seg on real-world datasets.
In Appendix A.8, we analyze the effect of batch size for GLASS on some datasets.
In general, the performance of GLASS drops as the batch size increases.
However, the max-zero-one labeling trick can still boost plain GNNs at a large batch size.
Computation Time Comparison
The precomputation of SubGNN is quite time-consuming.
After precomputation, the training of SubGNN is also slower than GLASS.
On the 7 used datasets (except ppi-bp ), SubGNN on average takes 3 .
8 × more time than GLASS to converge.
Moreover, the ratio jumps to 17 .
2 when we count in precomputation.
The results can be found in Figure 3.
[Figure]
C ONCLUSION
We have proposed GLASS, a simple yet powerful model for subgraph representation learning.
We illustrate the failure of plain GNNs on some simplest subgraph learning tasks, and demonstrate the power of node labeling tricks to enhance plain GNNs both theoretically and empirically.
GLASS outperforms the previous state-of-the-art method SubGNN by 20 .
7% on synthetic datasets and 6 . 0% on real-world datasets, verifying the power of node-level message passing for subgraph representation learning.
We argue that plain GNNs + labeling tricks are enough for learning higher-order substructures, and the complicated subgraph-level message passing is not necessary.
R EPRODUCIBILITY S TATEMENT
Our main theoretical contribution is in Appendix A.4, and complete proofs are also shown there.
Moreover, our proposed model GLASS consists of our max-zero-one labeling trick and a plain
GNN.
We describe the labeling trick in Section 5.2 and the architecture of GLASS in Appendix A. in detail.
All datasets used in our experiments are public, and both the processed versions of data and code are provided in the supplementary materials.
We will make them public on Github in the future.
A A PPENDIX A. P ROOF OF T HEOREM
Proof.
Consider the following GNN, which ignores node features.
[Equation]
Density d and cut ratio c can be predicted with GLASS in a transductive setting.
[Equation]
A. P ROOF OF P ROPOSITION
Proof.
First, we need to prove that given any plain GNN m 1 , there exists a GLASS m 2 producing the same output for any target subgraph S .
Assume that the AGGREGATE function of m 1 is f ( k ) 1 , the COMBINE function of m 1 is g ( k ) 1 at the k th layer and the READOUT function of m 1 is ϕ 1 .
We can define a function θ mapping labeled node features to the initial node features, θ ( CONCATENATE ( h ( k − 1) u , l ( S ) ))
=
h ( k − 1) u , which is a well defined function.
And we can design a GLASS as follows.
[Equation]
We can use ϕ 1 as the READOUT function, therefore m 2 will produce the same embeddings as mfor the target subgraph.
Second, GLASS can differentiate at least one pair of subgraphs which plain GNNs can not distinguish.
Figure 2 gives us an example.
A. A NALYSIS OF H OW GLASS C AN C OVER THE SIX PROPERTIES IN S UB GNN
The channels in SubGNN are non-deterministic and thus are not well-defined functions on subgraphs.
Therefore our goal is to prove that GLASS can represent the six channels by definition.
As the SubGNN uses embeddings produced by plain GNNs as node features, we first show that
GLASS can also produce such node embeddings.
Proposition 3.
Given a graph G and node embeddings H produced by plain GNNs, GLASS can also produce the same embeddings.
Proof.
We have prove that GLASS is more expressive than plain GNNs in 1.
Therefore, for any GLASS model, we can use some bottom layers to produce node embedding, and the rest of the model is still a GLASS.
Therefore, we can assume the input node features of our
GLASS are the same as SubGNN.
We also need to show the performance of plain GNNs.
Xu et al.
(2019) prove that plain GNNs can map BFS trees to the embedding space injectively.
In this section, we consider isomorphic trees as the same tree.
The internal N property encodes the set of nodes in the target subgraph, and the border N property encodes the set of neighbors within k hops of nodes in the target subgraph.
Therefore, to show that
GLASS can cover the N channel in SubGNN, we only need to prove that GLASS can differentiate any pair of neighbor feature sets.
See Proposition 4.
Proposition 4.
Given a target subgraph S in a graph G and node embeddings H produced by plain GNNs, GLASS can produce different embeddings for any pair of border neighbor feature sets { H u | d ( u, v ) ≤ k, v ∈ V S } or internal neighbor feature sets { H v | v ∈ V S } .
Proof.
For border neighbor feature sets, we can use a k -layer GLASS to encodes BFS trees thus capture the nodes within k hops.
Assuming the k-layer BFS tree set is T
=
{T 1 , T 2 , ..., T | V | } , where T u means the tree rooted in node u.
The MPNN in GLASS is a function mapping T to R d injectively.
If the READOUT is an injective multiset function like DeepSet (Zaheer et al., 2017), GLASS will be an injective function ψ : { T S |S ⊆ G} → R d , where T S =
{T u | u ∈ V S } .
And we can define a function f : { T S |S ⊆ G} →
{ V S |S ⊆ G} , where f ( T S )
= { H u | u ∈ V T , T ∈ T S } .
Therefore, ψ − 1 ◦ f is a well-defined function mapping embeddings to border neighbor set, and its inverse, may be multivalued, can map a border neighbor feature set to a set of embeddings.
These embedding sets do not overlap with each other.
As for the internal neighbor feature set, we can use a glass layer whose COMBINE function maps all nodes labeled zero to a constant embedding vector h 0 , which is different from any existing node embeddings, and maintain the input embeddings of nodes labeled one and pass it to a GLASS used in last paragraph.
We can get a function ϕ , which maps subgraph embedding to a set of node embeddings.
And we can design a function θ : {{ h ′ u | u ∈ V S }|S ⊆ G} → {{ h ′ u | u ∈ V S }|S ⊆ G} , θ ( U )
=
U − { v 0 } .
Therefore, ϕ ◦ θ maps embeddings to internal neighbor feature sets, and its inverse can map different internal neighbor sets to different embeddings.
For the P channel, the internal position is defined as the distance (shortest path length) between nodes in the target subgraph, and the border position is the distance between nodes in the target subgraph and nodes in the whole graph.
Proposition 5.
Given a target subgraph S in graph G and node embeddings H produced by plain
GNNs, if embeddings of nodes are different from each other, GLASS can produce different embeddings for subgraphs with different position information.
Proof.
We have proved that k -layer MPNN can capture the set of embeddings of neighbor nodes within k hops from the target node.
As the node embeddings are different, the set of embeddings of nodes can be mapped to the set of nodes bijectively.
If we use JK connection, we can capture neighbor set within 1 , ..., k hops respectively, and if u appear in l -hop neighbor set but not in h -hop neighbor set, we can infer that the distance between the target node and node u is h .
Moreover, using the DeepSet function as READOUT, we can capture the distance between nodes in the target subgraph and any other nodes.
As for the S channel, it is defined as the internal feature connectivity (the multiset { ( H u , H v ) | ( u, v )
∈
E S } ), and the border feature connectivity (the multiset { ( H u , H v ) | u ∈ V S , v / ∈ V
S , ( u, v ) ∈ E } ).
Proposition 6.
GLASS can represent the internal and border structure.
In other words, there exists a GLASS model mapping the multiset { ( g u , g v ) | ( u, v ) ∈
E S } and { ( g u , g v ) | u ∈ S and v / ∈ S} to the embedding space injectively, where g is the pretrained GNN embeddings.
Proof.
WLGNN can encode the multiset of feature pairs {{ ( g u , g v ) | v ∈ N ( u ) }| u ∈ V S } with the following encoder.
[Equation]
where f 1 and f 2 are some functions whose existence Xu et al.
(2019) have proved.
In other words, we can encode BFS trees whose depth=1.
If we concatenate the node labels and g (namely g ′ ), and design
[Equation]
[Equation]
22) (cid:80) u ∈ V S ( c ) f [-(cid:48) 
-]′ 1 ( (cid:80) v ∈ N ( u ) f 2 ( g v )) can represent { ( g u , g v ) | ( u, v )
∈ E S ( c ) } .
And (cid:80) u ∈ V S ( c ) f ′′ 1 ( (cid:80) v ∈ N ( u ) f 2 ( g v )) can represent { ( g u , g v ) | u ∈ S ( c ) and v / ∈ S ( c ) } .
Note that though we assume that the node embeddings are different from each other in Proposition 4, we can let the input feature be learnable in implementation.
Thus the coincidence of node embeddings can be avoided.
Moreover, with such technique, we can build a bijective function from the embedding space to the node set.
The theoretical expressive power of GLASS will be stronger and no longer be limited to node embedding generated by plain GNNs.
A. T HE E XPRESSIVE P OWER OF GLASS UNDER S TRUCTURAL R EPRESENTATION T HEORY
Zhang et al.
(2020) introduce a theoretical framework to analyze the expressive power of GNNs.
Under this framework, we theoretically characterize the expressive power of GLASS.
We first define some basic concepts.
A permutation π is a bijective mapping from { 1 , 2 , ..., n } to { 1 , 2 , ..., n } , where n ∈ N + .
All possible permutations constitute the permutation group Π n .
For node set U ⊆ V , π ( U ) = { π ( i ) | i ∈
U } , where i is the node index in a graph G , and π ( i ) can be node index in G or any other graph.
For edge set F ⊆ E , π ( F ) =
{ ( π ( i ) , π ( j )) | ( i, j ) ∈ F } .
For node attribute matrix X , π ( X ) π ( i ) =
X i .
And for subgraph S and graph G , π ( S ) =
( π ( V S ) , π ( E S ) , π ( X S )) and π ( G ) =
( π ( V ) , π ( E ) , π ( X )) .
The definition of node labeling tricks is shown as follows.
Definition 2.
Given a subgraph S in a graph G , we design a label matrix L ( S ) ∈ R | V |× d , whose u th is the label of node u. L satisfies: for any two subgraphs S and S ′ in G (1) L ( S ) =
π ( L ( S ′ ) )
⇒
V S =
π ( V S ′ ) (2) S =
π ( S ′ ) , G = π ( G ) ⇒ L ( S ) =
π ( L ( S ′ ) ) .
Moreover, we can prove that the zero-one labeling trick is a valid node labeling trick.
Theorem 2.
Zero-one is a node labeling trick.
Proof.
Obviously v ∈ V S ⇔ l ( S ) v = 1 .
(1) Given l (
[Equation] [Equation]
(2) Given S =
π ( S ′ ) , G = π ( G ′ ) ,
[Equation]
Now we analyze the power of node labeling tricks.
Ideally, GNN is more expressive if it can distinguish more non-isomorphic graphs.
We first introduce the concept of subgraph isomorphism .
Definition 3.
Given two n -node graphs G =
( V , E , X ) , G ′
=
( V ′ , E ′ , X ′ ) , and one subgraph
S of G and one subgraph S ′ of G ′ , S and S ′ are isomorphic (denoted by S ≃ S ′ )
iff ∃ π ∈ Π n , π ( S ) =
π ( S ′ ) and π ( G ) =
π ( G ′ ) .
Isomorphic subgraphs should be mapped to the same embeddings, and the most expressive representations should differentiate all non-isomorphic subgraphs.
Therefore we introduce the concept of structural subgraph embeddings to illustrate such perfect representations.
Definition 4.
Γ( S ) is a structural subgraph embedding for subgraph S if ∀S , S ′ , Γ( S ) =
Γ( S ′ ) ⇔ S ≃ S ′
Numerous works have been exploring the expressive power of GNNs on node tasks.
However, we focus on how labeling tricks can help GNNs on subgraph representation learning.
Therefore, we introduce an ideal GNN model for node tasks.
Definition 5.
If a GNN can map a node u to the embedding vector h u so that given any two graphs
G and G ′ , i ∈ V , j ∈ V ′ , h i = h
j ⇔ ∃ π ∈ Π n , i =
π ( j ) and G =
π ( G ′ ) , we call it a node-mostexpressive GNN.
Though GNN is powerful on various tasks, Zhang et al.
(2020) show that node embeddings generated by GNNs are not enough to produce structural subgraph embedding even if we use a node-mostexpressive GNN.
However, node labeling tricks can solve this problem.
Zhang et al.
(2020) prove that with node labeling trick and injective READOUT function, node-most-expressive GNNs can produce structural subgraph embeddings.
Theorem 3.
There exists function Γ , where Γ( S ) is structural subgraph embedding for subgraph
S , can be formulated as Γ( S ) =
READOUT ( { H v | v ∈ V S } ) , where H is the node embedding matrix of a node-most-expressive GNN with node label as input.
Proof.
We need to show H S
= H S ′ ⇔ S ≃ S ′ .
(1) To prove the ⇐ , we notice that with an injective READOUT
[Equation]
[Equation] [Equation] [Equation]
[Equation]
(2) As for the ⇒ , Given S =
π
( S
G = π ( G ′ )
⇒ L ( S ) =
[Equation] ( S [Equation]
[Equation]
Note that H S
= H S ′ ⇒ S ≃ S ′ only demands l ( S ) =
π ( l ( S [Equation]
) ⇒ S =
π ( S ′ ) while the reverse only demands S
= π ( S ′ ) , G = π ( G [Equation]
⇒
[Equation]
[Equation]
π
[Equation]
Assuming the embedding space is countable, DeepSet function (Zaheer et al., 2017) can be an injective and order-invariant READOUT function.
Therefore, structural subgraph embeddings can be produced with a node-most-expressive GNN and the zero-one labeling trick.
In other words, we have made the best in expressive power.
All subgraph tasks can be solved if the prediction target is a function of the equivalence class of isomorphic subgraphs.
However, up to now, no scalable node-most-expressive GNNs have been realized.
However, with less expressive GNNs, node labeling tricks can still make a difference.
Figure 2 provides an example.
Here we prove that node labeling tricks can boost WLGNNs, a kind of GNNs whose expressive power is the same as 1-WL.
Theorem 4.
For any ϵ ∈ R + , given any graph G with n nodes, whose node degrees range fromto O (log (1 − ϵ ) / 2 n ) , and having an empty feature, there exists w (2 n n 2 ϵ − 1 ) pairs of non-isomorphic subgraphs such that any h-layer WLGNN produces the same representation, while with node labeling tricks WLGNN can distinguish them.
Proof.
We prove that there are w ( n 2 ϵ ) nodes that WLGNNs cannot differentiate in (1), and get the bound in (2).
(1)The bound of the number of nodes in a n-hop BFS tree is
[Equation]
We can add virtual nodes to make the trees become complete k-ary trees.
And by enumerating the position of the virtual node we can get a bound 2 C 2 K
= O ( n 1 − ϵ ) for the number of non-isomorphic trees.
According to the pigeonhole principle, there exist n/ O ( n 1 − ϵ )
= n ϵ nodes which cannot be differentiated with each other by WLGNNs.
They form a set V iso .
(2) Let us partition V iso =
(cid:83) qi
=1 V i , nodes in each V i share the same one-hop neighbor.
Consider a node u ∈ V i , v ∈ V j , i ̸
= j .
There exists a node w ∈ N ( u ) , w / ∈ N ( v ) .
Let ˜ V v donate
V − { u, v, w } − N ( w ) .
| V v | ≥ w ( n − log 1 − ϵ 2 h n ) .
Consider arbitrary subset ˜ V v ⊆ V .
Let S 1 donate the subgraph induced by ˜ V v (cid:83) { u, w } , S 2 donate the subgraph induced by ˜ V v (cid:83) { v, w } .
The density of S 1 is higher than S 2 .
And GLASS can fit density perfectly, so GLASS can distinguish S 1 and S 2 , while WLGNNs cannot.
And the number of pairs ( u, v, w ) is
[Equation]
[Equation]
As nodes in | V i | share the same neighbor, | V i | ≤ O (log 1 − ϵ 2 h n ) .
So the expression above = w ( n 2 ϵ )
And the bound for the number of these pairs of subgraphs is
[Equation]
A. T HEORETICAL A NALYSIS OF M AX -Z ERO -O NE L ABELING T RICK
Take a closer look at the definition of node labeling tricks.
Node label should be (1) Expressive.
Labels of nodes in S must be different from those out of S .
(2) Invariant.
Isomorphic nodes (in S and out of S separately) must be mapped to the same label.
We can illustrate the function of the conditions with the following theorem.
Theorem 5.
For any S , S ′ in given graphs G , G ′ respectively, there exists a node-most-expressive
GNN with labeling tricks satisfying: (1) h S = h S ′ ⇒ S ≃ S ′ if l ( S ) =
π ( l ( S ′ ) )
⇒ V S
=
π ( V S ′ ) ; (2) h S = h S ′
⇐ S ≃ S ′ if S
= π ( S ′ ) , G = π ( G ) ⇒ l ( S ) =
π ( l ( S ′ ) ) , where h is GNN embedding of subgraph.
This theorem has been proven in the proof of Theorem 3.
Proposition 7.
For all node labeling tricks, there exists a case that given two different subgraphs
S , S ′ in graph G , node label l ( S ) ̸
= l ( S ′ ) .
Proof.
Consider three isomorphic nodes, namely u, v, w , in a graph.
Let S 1 donate subgraph induced by { u, v } , S 2 donate subgraph induced by { u, w } .
Therefore, L ( S 1 ) u
= L ( S 1 ) v ̸ = L ( S 1 ) w and L ( S 2 ) u
= L ( S 2 ) w ̸
= L ( S 2 ) v . L ( S 1 ) ̸ = L ( S 2 ) .
Therefore, developing less theoretically powerful tricks is a must.
Here, we introduce the max-zeroone labeling trick.
Looking back on the definition of node labeling tricks, it is invariant but not expressive enough.
Definition 6.
Given subgraphs in a batch {S 1 , ..., S n } .
The max-zero-one label of a node v in G is
[Equation]
With Theorem 5, we can prove the invariance of the max-zero-one node labels.
Corollary A.1.
With a node-most-expressive GNN, injective READOUT and the max-zero-one node labels, in a single batch, S ≃ S ′ ⇒ H S
= H S
Though the max-zero-one results in loss of expressive power compared with zero-one, experiments show that the model can fit cut ratio and density very well.
These two properties need a clear division of nodes in and outside the target subgraph.
A. GLASS A RCHITECTURE
Figure 4 shows the architecture of GLASS.
To accelerate convergence, we utilize GraphNorm (Cai et al., 2021).
As for the pooling layer, though the DeepSet function is most expressive in theory, the number of nodes in subgraphs varies from less than ten to hundreds, producing representations in different magnitudes.
Thus we also try some less expressive pooling methods.
JK connection (Xu et al., 2018) can both accelerate convergence and capture local features in the embeddings of lower layers and global features in higher layers.
For the same sake, we introduce node labels at each layer.
A. I MPLEMENTATION D ETAILS
Computing infrastructure.
We leverage Pytorch Geometric and Pytorch for model development.
Models were trained on single Nvidia V100 GPUs on a Linux server.
× 𝒏 pooling pooling
Prediction Prediction
Task
SSL
M PN
N 𝓧 123  45 node label 4𝓗 𝒏𝒐𝒅𝒆
MLP
MLP
G N
N pooling
MLP
Prediction
Pretrained SSL 𝕊
[Table]
Input node feature for GNN.
We use an all-one vector for GNN-plain, GNN-seg, and GLASS on synthetic datasets to avoid overfitting.
Moreover, on real-world datasets, GLASS and GNN-plain use SSL, which reduces the number of message passing layers and also helps avoid overfitting.
GNN-seg uses node degree, as the SSL may leak the information of topology outside the target subgraph.
Pretraining SSL.
Node embeddings were pretrained using a l -layer GNN with node id as the input node feature.
Fixed hyperparameters were batch size = 35536 , learning rate = 1 e − 4 and α = 1 . 0 .
The hidden dimension ranges from 8 to 64, l ranges from 1 to 8, and β are selected from [0 . 0 , 2 . 0] .
Datasets.
We use the code offered by the author of SubGNN to produce synthetic datasets and use the real-world datasets provided by SubGNN directly.
The statistics of these datasets can be found in Table 7.
As for dataset division, the real-world datasets take an 80:10:10 split, and the synthetic datasets follow a 50:25:25 split, following (Alsentzer et al., 2020).
Baselines.
We directly use the results reported.
Sub2Vec is designed for connected subgraphs, so we sample random walk at each component separately.
We use all the channels of Sub2Vec.
Model hyperparameter tuning.
We use optuna to perform random search.
Hyperparameters were selected to optimize micro F1 scores on validation datasets.
The best hyperparameters selected for each model can be found in our code in the supplement materials.
We select the learning rate from 5e-5 to 5e-3; number of layers, [1 , 3] ; hidden dimension, { 8 , 16 , 24 , 32 , 40 , 48 , 56 , 64 } ; γ , [0 . 0 , 2 . 0] ; aggregation, [ mean , sum , gcn ] ; pool, [ mean , sum , max , size ] ; batch size, [ ns/ 16 , ns/ 8 , ns/ 4 , ns/ 2 , ns ] , where ns is the size of test dataset.
The size pooling is to sum the node representation and divide it by the square root of number of nodes.
A. T HE D ROP OF P ERFORMANCE VERSUS B ATCH S IZE
On coreness, cut ratio, and density datasets, we test the performance of GLASS at different training batch sizes.
The results are in Figure 5.
We also measure the training time of GLASS.
The results are in Figure 6.
In general, the performance drops as the batch size gets larger.
However, the max-zero-one labeling trick is still effective at a large batch size as on all datasets, as the difference of F1 score keeps [Figure] [Figure] positive.
However, we find that when we set the test and valid batch size to one, the performance of
GLASS drops faster, which shows that max-zero-one is not just an approximation of zero-one.
With max-zero-one, GLASS is trained to predict the properties of the target subgraph with the existence of other subgraphs.
As for other datasets [Figure] even plain GNNs can predict component precisely.
Furthermore, the targets in real-world datasets are less closely connected to subgraph topologies.
Last but not least, real-world datasets are bigger, and we also use bigger models for them, leading to large variance in performance and high time complexity.