M OTIF -D RIVEN C ONTRASTIVE L EARNING OF G RAPH R EPRESENTATIONS
Anonymous authors
Paper under double-blind review
A BSTRACT
Graph motifs are signiﬁcant subgraph patterns occurring frequently in graphs, and they play important roles in representing the whole graph characteristics.
For example, in chemical domain, functional groups are motifs that can determine molecule properties.
Mining and utilizing motifs, however, is a non-trivial task for large graph datasets.
Traditional motif discovery approaches mostly rely on exact counting or statistical estimation, which are hard to scale for a large number of graphs with continuous and high-dimension features.
In light of the signiﬁcance and challenges of motif mining, we propose MICRO-Graph : a framework for M otIf-driven Contrastive leaRning Of Graph representations to: 1) pre-train
Graph Neural Networks (GNNs) in a self-supervised manner to automatically extract graph motifs from large graph datasets; 2) leverage learned motifs to guide the contrastive learning of graph representations, which further beneﬁt various graph downstream tasks.
Speciﬁcally, given a graph dataset, a motif learner cluster similar and signiﬁcant subgraphs into corresponding motif slots.
Based on the learned motifs, a motif-guided subgraph segmenter is trained to generate more informative subgraphs, which are used to conduct graph-to-subgraph contrastive learning of GNNs.
By pre-training on ogbg-molhiv molecule dataset with our proposed MICRO-Graph , the pre-trained GNN model can enhance various chemical property prediction downstream tasks with scarce label by 2.8 % , and signiﬁcantly higher than other state-of-the-art self-supervised learning baselines.
I NTRODUCTION
Graph-structured data, such as molecules and social networks, is ubiquitous in many scientiﬁc research areas and real-world applications.
To represent graph characteristics, graph motifs were proposed in Milo et al.
as signiﬁcant subgraph patterns occurring frequently in graphs and uncovering graph structural principles
Functional groups are important motifs that can determine molecule properties.
For example, for chemical compounds, hydroxide (– OH ) usually implies higher water solubility, and for proteins, Zif268 can mediate protein-protein interactions in sequence-speciﬁc DNA-binding proteins.
(Pabo et al., 2001).
Researchers have been studying graph motifs for years.
On one hand, utilizing meaningful motifs can beneﬁt many important applications like quantum chemistry and drug discovery (Ramsundar et al., 2019).
On the other hand, extracting motifs from large graph datasets is still a challenging question.
Traditional motif discovery approaches (Milo et al., 2002; Kashtan et al., 2004; Schreiber & Schw¨obbermeyer, 2005; Chen et al., 2006; Wernicke, 2006) mostly rely on discrete counting or statistical estimation, and thus are hard to generalize to large-scale graph datasets with continuous and high-dimension features, as often the case in real-world applications.
Recently, Graph Neural Networks (GNNs) have shown great expressive power for learning graph representations without explicit structural feature engineering (Kipf & Welling, 2016; Hamilton et al., 2017; Veliˇckovi´c et al., 2017; Xu et al., 2018).
In addition, GNNs can be trained in a selfsupervised manner without human annotations to capture important graph structural and semantic properties (Veliˇckovi´c et al., 2018;
Hu et al., 2020c; Qiu et al., 2020; Bai et al., 2019; Navarin et al., 2018; Wang et al., 2020; Sun et al., 2019; Hu et al., 2020b).
This motivates us to rethink about motifs as more general representations than exact structure matches and ask the following research questions:
• Can we use GNNs to automatically extract graph motifs from large graph datasets?
• Can we leverage the learned graph motif to beneﬁt self-supervised GNN learning?
In this paper, we propose MICRO-Graph : a framework for MotIf-driven Contrastive leaRning Of
Graph representations.
The key idea of this framework is to learn graph motifs as prototypical cluster centers of subgraph embeddings encoded by GNNs.
In this way, the discrete counting problem is transfered to a fully-differentiable framework that can generalize to large-scale graph datasets with continuous and high-dimensional features.
In addition, the learned motif can help generate more informative subgraph for graph-to-subgraph contrastive learning.
The motif learning and contrastive learning are mutually reinforced, and eventually help pre-train a more generalizable GNN encoder.
For motif learning, given a graph dataset, a motif-guided subgraph segmenter generates subgraph samples for each graph, and the GNN encoder turns these subgraphs into d -dimensional vector representations.
We learn graph motifs also as d -dimensional vectors through clustering, where K prototypical cluster centers are kept in an embedding table as representations of motifs.
Similar and signiﬁcant subgraphs are assigned to the same motif, and thus moved to be closer to the corresponding motif representation.
We train our model in an Expectation-Maximization (EM) fashion to update both the motif assignment of each subgraph and the representations of motifs.
For leveraging learned motifs, we propose a novel graph-to-subgraph contrastive learning framework for GNN pre-training.
One of the key components for a successful contrastive learning is to generate semantically meaningful and diverse views of each data instance.
For example, a continuous span within a sentence (Joshi et al., 2020) or a random crop of a image (Chen et al., 2020).
Previous approaches, such as Deep Graph Infomax and InfoGraph leverage node-to-graph views for contrastive prediction, which is not sufﬁcient enough to capture the subgraph information.
To address this issue, we propose to learn graph-to-subgraph contrastive.
As motifs can represent the key graph property by its nature, we propose to leverage the learned motifs to generate more informative subgraphs, and beneﬁt GNN contrastive learning.
For example, alpha helix and beta sheet can come together as a simple ββα fold to form a zinc ﬁnger protein, and have unique properties.
By learning such subgraph co-occurrence via contrastive learning, the pre-trained GNN should capture more subtle structural semantics of graph.
After pre-training on ogbg-molhiv dataset using our proposed MICRO-Graph , the framework can successfully learn meaningful motifs, including Benzene ring, Nitro and Acetate, etc.
Meanwhile, the pre-trained GNN encoder can be used to ﬁne-tune on seven chemical property prediction benchmarks, yielding 2.8% average improvement over non-pretrained GNNs and outperforming other self-supervised pre-training baselines.
Extensive ablation studies further show that the learned motifs are indeed signiﬁcant for the contrastive learning framework.
R ELATED W ORK
The goal of self-supervised learning is to train a neural network model to capture signiﬁcant characteristics of data without human annotations.
This paper studies whether we can use such framework to automatically extract graph motifs, one of the signiﬁcant structural patterns in graph, and leverage the learned motifs to beneﬁt self-supervised learning.
In the following, we ﬁrst review some preliminaries about graph motifs and the challenge for motif mining, and then review approaches of pre-training graph neural networks in a self-supervised manner.
Graph motifs are building blocks of complex graphs.
They reveal the interconnections of graphs and represent graph characteristics.
Mining motifs can beneﬁt different tasks, from exploratory analysis to graph transfer learning (Henderson et al., 2012).
Motif discovery has been studied for many years, and various algorithms have been proposed.
There are generally two categories for automatically extracting motifs from graph data, either exact counting as in Milo et al.
; Kashtan et al.
(2004); Schreiber & Schw¨obbermeyer (2005)
; Chen et al.
(2006), or sampling and statistical estimation as in Wernicke (2006).
However, both approaches can not scale to large graph datasets with continuous and high-dimension features, which is often the case in real-world applications.
In this paper, we proposes to turn the discrete motif mining problem into differentiable cluster learning, and adopt a self-supervised GNNs learning framework to automatically extract important graph motifs.
Such method can be generalized to large-scale graph datasets.
Contrastive learning is one of the state-of-the-art self-supervised representation learning algorithms that achieves great results for visual representation learning (Chen et al., 2020;
He et al., 2019).
Contrastive learning force that embeddings of views generated from the same data (e.g. a image crops) instance to be closer to each other, while those from different instances apart.
One of the key component in contrastive learning is to generate informative and diverse views from each data instance.
In computer vision, researchers adopt various image augmentation techniques, including crop, color distortion, Gaussian blurs, to generate such views.
However, when in comes to graph domain, constructing informative view of graph, e.g., subgraph, is a challenging task.
In our framework, we utilize the learned motifs, which are signiﬁcant subgraph patterns, to guide constructing informative views from each graph, and conduct graph-to-subgraph contrastive learning.
Self-supervised learning for GNNs also draws many attention recently.
For graphs, representations can be at different levels, e.g. node level and (sub)graph level.
Veliˇckovi´c et al.
; Hu et al.
(2020c)
; Qiu et al.
(2020) mainly focus on node-level representation learning in a single large graph, as opposed to the focus of this paper, representations of the whole graphs.
Hu et al.
(2020b) provides a systematic analysis of pre-training strategies on graphs for both node-level and graph-level representations.
However, only the node-level learning is self-supervised, and annotated labels are utilized to perform supervised learning at the graph level.
For graph level self-supervised representation learning, Sun et al.
(2019) proposed a contrastive framework InfoGraph to maximize the mutual information between the representations of the whole graphs and node representations.
The views they utilize are each node in the graph with the whole graph, which capture more subtle subgraph semantics.
To alleviate this issue, we propose a novel graph-to-subgraph view construction, and the subgraph generation is guided by learned motifs.
M ETHODOLOGY
The goal of this paper is to train a GNN encoder that can automatically extract graph motifs, i.e. signiﬁcant subgraph patterns.
Motif discovery on discrete graph structures is a combinatorial problem, and it is hard to generalize to large datasets with continuous features.
We thus propose to formalize this problem as a differentiable clustering learning problem and solve it via self-supervised GNN learning.
In this section, we formalize the problem and introduce the overall framework of MICROGraph in Section 3.1, and then describe each module in details in the following sections.
3. T HE OVERALL F RAMEWORK OF MICRO-Graph
Given a dataset with M graphs G = {G 1 , ..., G M } , the differentiable clustering learning problem is meant to learn two things.
One is a GNN-based graph encoder E ( · ) that maps input (sub)graphs into a D-dimensional embedding space.
The other is a K-slot embedding table { m 1 , ..., m K } , where each slot is a D-dimensional motif vector m corresponding to a cluster center of embeddings of frequently occurred subgraphs.
To tackle this problem, we introduce the MICRO-Graph framework which consists three modules: 1) a Motif-guided Segmenter to extract important subgraphs; 2) a
Motif-Learner to cluster sampled subgraphs and identify motifs; 3) a constrastive learning module for graph-subgraph contrastive learning.
The overall framework is shown in Figure1.
We now describe details of each module in the following sections.
We ﬁrst introduce the Motif Learner in section 3.2, and then get to the Motif-guided
Segmenter and the constrastive learning module in section 3.3 and section 3.4 respectively.
3. M OTIF LEARNER VIA EM CLUSTERING
The Motif Learner learns motifs by applying an Expectation-Maximization (EM) style clustering algorithm on sampled subgraphs.
The Motif-guided Subgraph Segmenter extracts N subgraphs { g j } Nj =1 from the input whole graphs.
For each subgraph g j , we generate its embedding e j = E ( g j ) and calculate the cosine similarity between e j and each of the K motif vectors { m 1 , ..., m K } .
We denote the similarity between e j and the kth motif vector as S k,j = φ ( m k ) T φ ( e j ) 1 .
In vector notation, the K-dimensional vector of similarities between g j and all K motif vectors is denoted as s j , and the K-by-N-dimensional motif-to-subgraph similarity matrix is denoted as S , where the j -th column of S is s j and the entry (k, j) of S is S k,j .
E-Step.
The goal of the E-step is to come up with motif-based cluster assignments for subgraph embeddings { e j } Nj =1 .
The assignments can be represented by a K-by-N-dimensional matrix
Q =
[ q 1 , ..., q N ] , where the j -th column q j contains the probabilities of assigning the j -th subgraph to each of the K motifs.
When considering hard cluster assignments, each q j will be a one-hot vector.
When considering soft cluster assignments, each q j will be a probability vector with all the entries sum up to one.
This vanilla version clustering problem boils down to maximizing the objective Tr ( Q T S ) , which corresponds to a cluster assignment Q that maximizes similarities between embeddings and its assigned motif.
This objective works ﬁne for a traditional EM clustering algorithm when embeddings are ﬁxed.
However, as we are doing representation learning, and the embeddings themselves will also change during learning, using this vanilla objective in the E-step can lead to a degenerate solution, i.e. all embeddings collapse to a single cluster center.
To avoid this issue, we follow YM.
et al.
(2020) to introduce an entropy term and an equal-size constraint on
Q to make all the clusters to have similar sizes.
Our ﬁnal objective is:
[Equation]
where H ( Q ) = − (cid:80) i,j Q i,j log Q
i,j is the entropy function, and the constraint set Q requires the marginal projection of Q onto its columns and rows to be uniform.
[Equation]
where 1 N and 1 K are all one vectors.
This speciﬁc constraint optimization problem turns out to be an optimal transportation problem with a closed-form solution as below and can be solved efﬁciently using a fast Sinkhorn-Knopp algorithm.
[Equation]
Here u and v are normalization vectors.
The derivations can be found in Cuturi (2013).
M-Step.
The goal of the M-step is to maximize the log-likelihood of our data given the cluster assignment matrix Q estimated in the E-step.
We update parameters in the GNN encoder and the motif embedding table through the M-step.
This step is equivalent to a supervised K-class classiﬁcation problem with labels Q and prediction scores S .
Thus, we ﬁrst apply a columnwise softmax normalization with temperature τ g to S to convert all entries of S to probabilities, i.e.
˜ S k,j = softmax k (cid:0) S k,j /τ g (cid:1) .
Then we use the negative likelihood as the loss function.
[Equation]
3. M OTIF - GUIDED S UBGRAPH S EGMENTER
Sampling informative subgraphs is crucial for both the Motif-Learner and the contrastive learning module .
Traditional heuristic approaches such as random walk and k-hop neighbour sampling cannot guarantee to generate semantically reasonable and informative subgraphs.
For example, heuristically sampled molecule subgraphs are likely to be a chain of carbons, which doesn’t contain much information about the original molecule, or it can be a fragment of a meaningful chemical structure, which loses the original chemical property.
Since motifs are by nature informative subgraph patterns, we propose to leverage the learned graph motifs to design the Motif-guided segmenter , which will learn to segment a given graph into several subgraphs that are close to some discovered motifs.
Subgraph Sampling via Segmentation.
To generate subgraphs from a graph G i via segmentation, we ﬁrst generate a node afﬁnity matrix A ( i ) , and then do segmentation based on A ( i ) .
Speciﬁcally, given the graph G i with n nodes, we use the GNN encoder E ( · ) to generate the D-dimensional node embeddings { n 1 , ..., n n } of all n nodes in G i .
After that, we compute the n-by-n-dimensional
node
afﬁnity
matrix A ( i ) by ﬁrst computing pairwise cosine similarities between node embeddings, and then applying row-wise softmax normalization with temperature τ n to the cosine similarity matrix.
This normalization step is important because it transforms all the afﬁnity scores to be in the range (0,1) and make the afﬁnity scores of node pairs with high cosine similarities further stand out.
The formula for computing the afﬁnity score between node s and node t in graph G i , i.e. the entry ( s , t ) of A ( i ) is the following.
[Equation]
Afterwards, we treat this afﬁnity matrix
A ( i ) as a complete graph with n nodes and afﬁnity scores as edge weights.
Applying spectral clustering on this complete graph segments nodes into different groups.
Within these groups, the connected components that have more than three nodes are collected as our sampled subgraphs.
Multiple subgraphs may be sampled from the whole graph
G i .
For a particular subgraph g j , its embedding e j will be generated by indexing and aggregating the node embeddings generated when computing the afﬁnity matrix.
For example, if I is the indices of the nodes forming the subgraph g j selected by the Motif-guided segmenter , then e j = Aggregate ( { n 1 , ..., n n } [ I ]) .
The aggregate operation can be any order-invariant operation over a set of vectors, e.g. mean, sum, or elementwise max.
In our experiment, we follow the startof-the-art result from previous works and use the mean.
Collecting subgraphs from all M whole graphs result in the total set of subgraphs { g j } Nj =1 mentioned above.
Motif-guided Training.
To train the segmenter to produce subgraphs close to some motifs, we use the motif-to-subgraph similarity matrix S mention above as the guidance.
Consider a subgraph g j sampled from a whole graph G i whose node afﬁnity matrix is A ( i ) .
If the similarity between g j and any motif is higher than a threshold, we would like the afﬁnity value among all the nodes within this subgraph g j to increase, and the afﬁnity value between these nodes and other nodes not in g j to decrease.
The loss function is as below.
[Equation]
Here η k is the threshold used to decide whether a subgraph is similar enough to the learned moﬁt k, and it is dynamically computed.
In each iteration, we set η k to select the top 10% most similar subgraphs to motif k.
The intuition is that if the subgraph g j is considered similar to a motif, then we update the embeddings of its nodes to become similar.
By optimizing this loss, during the next sampling round, nodes produced motif-like subgraphs are more likely to be segmented together, which leads to more subgraph samples align with the motifs.
3. C ONTRASTIVE L EARNING BETWEEN GRAPHS AND SUBGRAPHS
An expressive GNN encoder E ( · ) is essential for capturing graph properties and moreover accurately identifying motifs.
We thus introduce a constrastive learning module to help guide the GNN learning.
This module and the Motif Learner will mutually enhance each other to train the GNN encoder to better capture the graph properties.
Contrastive learning is one of the state-of-the-art self-supervised learning methods.
One key component in contrastive learning is to generate informative and diverse views of data instances.
Previous contrastive methods on graphs utilized either nodes or whole graphs as views, which do not very well capture the micro-structure of graphs.
To alleviate this issue in our constrastive learning module, we use the subgraph generated by our Motif-guided segmenter as one view of the graph, and the whole graph as another view.
Similarly to how we generate the subgraph embeddings, the whole graph embedding h i of a graph G i is generated by aggregating node embeddings output by the GNN encoder E ( · ) , h i =
Aggregate ( { n 1 , ..., n n } ) .
Then we construct the M-by-N dimensional graph-to-subgraph similarity matrix W , which is cosine similarity between each graph-subgraph pair followed by a row-wise softmax normalization with temperature τ g .
[Equation]
For the whole graph G i , subgraphs sampled from it are considered as positive pairs to it, and subgraphs sampled from other graphs are considered as negative pairs to it.
Then the contrastive objective function is the following:
[Equation]
Our method considers subgraphs as high-level views of each graph to conduct graph-to-subgraph contrastive learning, and thus the GNN can capture higher-order and global structure information.
We show the main difference for generating views of the original graph between our framework and other existing methods in Figure 2.
Most existing contrastive learning methods are conducted on node-node views or node-graph views.
Researchers in computer vision domain have found that conducting contrastive learning on more ﬂexible views, e.g. an arbitrary-sized image crop, leads to better representations, especially much better than views at the pixel level (Chen et al., 2020).
In our framework, the contrastive learning on graph-subgraph views is similar to this intuition from computer vision.
It can capture higher-level structural patterns the node views cannot capture, and thus produce more meaningful representations.
Note that the subgraphs we utilize is generated by our Motif-guided Segmenter .
In section 4, we systematically study the inﬂuence of subgraph sampling methods.
The experiment results show that simple heuristic sampling approaches lead to really bad generalization performance.
This shows the signiﬁcance of leveraging the learned graph motifs to generate more informative views of graphstructured data.
3. J OINT TRAINING
The overall objective of MICRO-Graph is a weighted sum of all three loss terms described above.
[Equation]
With the proper constraints introduced to the E-step of the Motif-Learner (equation (1) and (2)), the whole framework can be trained end-to-end without worrying about degenerate solutions.
We show the the pseudocode of our approach in Algorithm 1. We ﬁrst initialize the motif vectors and GNN encoder (line 2 - 3). Then for each batch of graphs G , our segmenter will calculate the node-node afﬁnity matrices A , and extract the subgraphs based on the afﬁnity scores (line 5 - 7). Then we apply GNN message passing on whole graphs to get the node embeddings, and aggregate them for both graph and subgraph embeddings (line 9 - 11).
The next step is to compute the motifto-subgraph similarity matrix S , based on that we can compute both the threshold η and cluster assignment matrix Q (line 15 - 17).
With all these values, our three loss terms the ﬁnal joint loss can be computed accordingly (line 18 - 26).
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Figure]
[Table]
[Table]
[Figure]
[Table]
[Figure]
[Table] [Table] MICRO-Graph can simultaneously train a expressive GNN encoder and learn the underlying motifs of the given graph dataset. Moreover, the motif learning and contrastive learning are mutually reinforced: a better GNN encoder can help get more accurate subgraph embeddings and thus help motif mining, while a better motif can help generate more informative graph-to-subgraph views and help contrastive learning.
E XPERIMENTS
We evaluate the effectiveness of MICRO-Graph from two perspective: 1) Whether the selfsupervised framework can learn better GNNs to generalize well on graph classiﬁcation tasks; 2)
Whether the learned motifs are reasonable and can truly beneﬁt contrastive learning.
4. E XPERIMENT S ETUP
We mainly focus on chemical property prediction task.
Speciﬁcally, we pre-train GNNs using
MICRO-Graph on the ogbg-molhiv dataset from Open Graph Benchmark (OGB) (Hu et al., 2020a).
This dataset contains 40K chemical compounds with average graph size 25.
We test our pretrained model on other smaller molecule-related graph classiﬁcation datasets from OGB.
For more details of the datasets, please see Appendix E.
4.1. E VALUATION P ROTOCOLS
After we pre-train a GNN encoder on ogbg-molhiv dataset, we evaluate its effectiveness using the following two evaluation protocols:
Transfer Learning Setting : we ﬁne-tune the pre-trained GNN model with a small portion of labels of the downstream task.
We adopt the same train-test and model selection procedure as in Yanardag & Vishwanathan (2015); Zhang et al.
Xu et al.
(2018), where we perform 10-fold crossvalidation and select the epoch with the best cross-validation accuracy averaged over the 10 folds.
We follow the OGB ofﬁcial setting to use the ROC-AUC as our metric.
Feature Extraction Setting : we ﬁx the pre-trained GNN, use it as feature extractor to get graph representations of all the data in downstream tasks, and apply linear classiﬁers to train on top of them.
We mainly follow the experimental settings in Sun et al.
For datasets with multiple tasks, we report the averaged classiﬁcation accuracy over all tasks.
4.1. B ASELINES
For transfer learning setting, we adopt non-pretrain and direct supervised learning as the baseline.
We also use three state-of-the-art GNN Self-Supervised Learning (SSL) methods as baselines:
InfoGraph (Sun et al., 2019) maximizes the mutual information between the representations of the whole graphs and the representations of its substructures at different granularity.
Context prediction (Hu et al., 2020b) uses subgraphs to predict their surrounding graph structures, so that nodes appearing in similar structural contexts will be mapped to nearby representations.
GPT-GNN (Hu et al., 2020c) predicts masked edges and masked node attributes.
The edge prediction makes node representations to be close when there are edges between them.
The attribute prediction captures how node attributes are distributed over all graphs.
4.1. M ODEL C ONFIGURATION AND H YPERPARAMETERS
We use the Deeper Graph Convolutional Networks (DeeperGCNs) proposed in Li et al.
(2020), the state-of-the-art GNN architectures, as our base GNN encoder.
For fair comparison, we use the same model hyperparameters (number of layers is 5 and number of hidden dimension is 300) recommended in Li et al.
(2020) for our method and all baselines.
We pre-train our model using
Adam optimizer for 100 epochs, with batch size (number of graphs per batch) as 512.
During ﬁnetuning, we train the model for 100 epochs with batch size as 32, and select the output model with highest validation performance.
4. E VALUATION R ESULT
The evaluation results under transfer learning setting and feature extraction setting is illustrated in
Table 1 and Table 2.
For both setting, the average performance of our proposed MICRO-Graph is higher than all the self-supervised learning baselines, and it achieves the highest results on most datasets.
For transfer learning setting, we can gain about 2.8 % performance enhancement against non-pretrain baseline.
This shows the effectiveness of our self-supervised learning framework for pre-training GNNs, especially when the label data is scarce.
4. A BLATION S TUDY
In the following, we conduct a series of ablation studies to systematically analyze how the motif learning can beneﬁt the contrastive learning.
4.3. W HETHER M OTIF IS HELPFUL FOR SUBGRAPH SAMPLING ?
As previously discussed, the main difference of our contrastive framework with existing works is that we leverage the subgraph-to-graph view as contrastive prediction task.
As there are multiple ways to generate subgraphs, we thus study whether our proposed motif-guided subgraph segmenter can indeed help contrastive learning. [Table] [Table]
We implement three widely adopted heuristic subgraph sampling baseline: random walk (RW) and
K-hop neighbours sampling (K-hop).
We replace our Motif-guided Subgraph Segmenter (MSS) component with these two heuristic sampling algorithm, and conduct experiments under the Transfer
Learning setting.
All the other settings are the same.
As is shown in Table [Table] replacing our Motif-guided Subgraph Segmenter (MSS) with heurstic subgraph sampler will signiﬁcantly inﬂuence the performance.
With random walk or k-hop sampling, subgraph-to-graph contrastive learning can only bring in 0.3-0.5 % average performance enhancement against non-pretrain, which is far less than the one with MSS.
This shows that the key of our overall performance enhancement of is not only based on utilizing the subgraph-to-graph view [Table] but also highly depends on the informative subgraphs generated with the help of motifs.
For the details of sample strategies and the subgraphs sampled by each strategies, please refer to Appendix D.
4.3. W HETHER M OTIF NUMBER WILL INFLUENCE CONTRASTIVE L EARNING ?
Number of motifs is an important hyperparameter in our motif mining framework.
We thus conduct ablation study with three different number of motifs, 5, [Table] and 100.
As is illustrated in Table 4, with different number of motifs, our MICRO-Graph framework can consistently enhance the transfer performance by a large margin.
Among the three numbers used, the middle one (20) gives the best result on average. [Table] [Table]
4. V ISUALIZATION OF THE LEARNED M OTIFS
We further shows some learned motifs by collecting the closest subgraph to them.
As is illustrated in
Figure 3, the MICRO-Graph automatically learns some motifs that are similar to important function groups in molecule domain, such as Benzene ring and Acetate.
This shows that our framework can indeed learn reasonable and meaningful motifs.
For a complete list of all the learned motifs, please refer to Appendix C.
C ONCLUSION
In this paper, we propose MICRO-Graph to pre-train GNN in a self-supervised manner that can automatically extract graph motifs from large-scale graph datasets.
In addition, the learned motifs can guide to generate more informative subgraphs, and help conduct graph-to-subgraph contrastive learning.
The motif learning and contrastive learning are mutually reinforced, and eventually help pre-train a more generalizable GNN encoder.
By pre-training on ogbn-molhiv molecule dataset with our proposed MICRO-Graph , we can learn meaningful motifs that ﬁt existing molecular functional groups.
Meanwhile, the pre-trained GNN can be used to ﬁne-tune on seven chemical property prediction benchmarks, yielding 2.8% average improvement over non-pretrained GNNs and outperforming other self-supervised pre-training baselines.
A P SEUDOCODE OF THE MAIN ALGORITHM
[Table]
temperature
[Table]
[Table]
[Table]
[Table]
[Table]
contains all
[Table]
[Table]
.motifs
[Table]
for pretraining 67 for
[Table]
in loader:
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
no
[Table]
grad
[Table]
[Table]
[Table] ) > 0 # 1 x N # compute the sampler lossloss_s = sum_A[S_mask] # normalize S S_tilde =
torch.softmax(S / tau_g, dim=0)
# K x N # compute motif learning lossloss_m = - (Q * S_tilde.log()).sum(dim=0).mean() # compute
[Table]
similarities between each [Table] [Table] [Table]
h
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
60 def segmenter
[Table]
[Table]
):with torch.no
[Table]
[Table]
[Table]
[Table] [Table] [Table] to get node
[Table]
n
[Table] [Table] _i) # compute the
[Table] [Table]
affinity matrix A_
[Table]
A_
[Table]
torch.softmax(pairwise_cosine_sim(n, n) / tau_n, dim=1) # apply spectral
[Table]
find connected [Table]
# to
[Table]
, I_i is a list I_i = find_connected_components(spectral
[Table]
clustering
[Table]
_i)) # the
[Table]
indices of these
[Table]
I +=
[Table]
[Figure]
B N OTATION SUMMARY
Below, we summary the important notations and symbols used paper in the order they appeared.
Graph
[Table]
Node afﬁnity matrix, n-by-n dimensional for a whole graph with n nodes
[Table]
Node afﬁnity between node s and node t in a graph
I
Indices of nodes forming a subgraph selected by the Motif-guided Segmenter Motifs
[Table]
vectors [Table] Similarities between all K motifs and all N subgraphs, K-by-N dimensional
[Table]
Similarities between all K motifs and the subgraph j, K-by-1 dimensional
[Table]
Similarity between the motif k and the subgraph j, scalar ˜
[Table]
j Normalized similarity between the motif k and the subgraph j
Q
Motif-based cluster assignment matrix, K-by-N dimensional q [Table] Motif-based cluster assignment of subgraph j, 1-by-N dimensional [Table]
Threshold for deciding whether a subgraph is similar enough to motif k
Contrastive
W
Normalized similarities between M graphs and N subgraphs, M-by-N dimensional [Table]
i [Table]
Normalized similarities between graph i and subgraph j Others
[Table] Temperature for the softmax normalization of node-node similarity τ g Temperature for the softmax of motif-subgraph and graph-subgraph similarity E
[Table]
[Table]
GNN encoder used to generate node embeddings and (sub)graph embeddings
D
Dimension of node embeddings, (sub)graph embeddings, and motif vectors
C T OP K CLOSESET SUBGRAPHS TO LEARNED MOTIFS
Examples of the ﬁrst 10 learned motifs of the ogbg-molhiv dataset is shown in Figure 4 and 5.
D S AMPLING S TRATEGIES AND S AMPLED S UBGRAPHS
Here we describe the details of our heuristic sampling strategies.
[Table]
, we use walk length uniform in [10, 40].
, we randomly select neighborhood
For K -hop neighbors, we pick hop number k
to be 1 or 2 with equal probability.
Starting randomly selected seed node, we collect all the neighbors within k hop as the sampled subgraph.
We also shows some subgraph examples generated by these two heuristic strategies and our proposed motif-guided subgraph segmenter in Figure 6.
From the sampled subgraphs, we can see that random walk is more likely to generate chains, while k-hop sampling is more likely to generate half part of a Benzene ring.
Neither of these two heuristic approaches can successfully generate a complete and clean functional group, and the generated subgraphs are not that meaningful.
On the contrary, our motif-guided sampler can succssfully generate a complete benzene ring and two other molecule substructures.
This intuitively explains why the graph-to-subgraph contrastive learning can only work with our proposed subgraph segmenter.
E C HEMICAL P ROPERTY P REDICTION B ENCHMARKS
In our experiments, we evaluated model performance on seven Open Graph Benchmark (OGB) molecule property prediction datasets.
We provide a synopsis of each downstream task dataset from
Hu et al.
(2020b) below:
• bace: Qualitative binding results for a set of inhibitors of human β -secretase 1. • bbbp: Blood-brain barrier penetration (membrane permeability).
• clintox: Qualitative data classifying drugs approved by the FDA and those that have failed clinical trials for toxicity reasons.
• hiv:
Experimentally measured abilities to inhibit HIV replication.
• sider: Database of marketed drugs and adverse drug reactions (ADR), grouped intosystem organ classes.
• tox21: Toxicity data on 12 biological targets, including nuclear receptors and stress response pathways.
• toxcast: Toxicology measurements based on over 600 in vitro high-throughput screenings.
Table 5 summarizes important statistics of the OGB molecule datasets related to the number of graphs, the size of graphs, and number of properties that require prediction for each molecule.
For further information on the OGB datasets (e.g. node and edge features), please refer to Hu et al.
(2020b) and Hu et al.
(2020a).
F L INEAR E VALUATION IN A CCURACY G A S YNTHETIC DATASET TO STUDY MOTIF LEARNING
In addition to studying the pre-training in chemical domain, we also construct a synthetic dataset to that align with our assumptions to verify the effectiveness of the propose method.
We assume there exist K graph motifs, and each whole graph can be representated by certain combinations of these
[Table]
motifs. Following this rule, we ﬁrst select some graph structures as motifs, and randomly sample some combinations, and generate graphs (as is illustrated in Figure 7) by combining the motifs, with some randomly added or deleted nodes and edges.
Each graph will also be assigned a corresponding one-hot vector label, whose dimension is the total number of combinations we generated for the dataset.
Eight subgraph templates and examples of generated whole graphs are shown below.
As is illustrated in Figure 8, our MICRO-Graph can successfully learn the underlying graph motifs and templates without any annotations.
The advantage of constructing such synthetic dataset is that we can know the underlying groundtruth of graph motifs and combination rules.
We believe on top of this toy dataset, more complex syntax and semantic knowledge can be incorporated for automatic motif and even rule mining communities.