Graph Self-supervised Learning with Accurate Discrepancy Learning
Abstract
Self-supervised learning of graph neural networks (GNNs) aims to learn an accuraterepresentation of the graphs in an unsupervised manner, to obtain transferablerepresentations of them for diverse downstream tasks.
Predictive learning andcontrastive learning are the two most prevalent approaches for graph self-supervisedlearning.
However, they have their own drawbacks.
While the predictive learningmethods can learn the contextual relationships between neighboring nodes andedges, they cannot learn global graph-level similarities.
Contrastive learning, whileit can learn global graph-level similarities, its objective to maximize the similaritybetween two differently perturbed graphs may result in representations that cannotdiscriminate two similar graphs with different properties.
To tackle such limitations,we propose a framework that aims to learn the exact discrepancy between theoriginal and the perturbed graphs, coined as Discrepancy-based Self-supervised
LeArning (D-SLA).
Specifically, we create multiple perturbations of the givengraph with varying degrees of similarity, and train the model to predict whethereach graph is the original graph or the perturbed one.
Moreover, we further aim toaccurately capture the amount of discrepancy for each perturbed graph using thegraph edit distance.
We validate our D-SLA on various graph-related downstreamtasks, including molecular property prediction, protein function prediction, andlink prediction tasks, on which ours largely outperforms relevant baselines.
Introduction
A graph, consisting of nodes and edges, is a data structure that defines a relationship among objects.
Recently, graph neural networks (GNNs) [15, 8, 30, 37], which aim to represent this structurewith neural networks, have achieved great successes in modeling real-world graphs such as socialnetworks [5], knowledge graphs [1], biological networks [20] and molecular graphs [35].
However,it is extremely costly and time-consuming to annotate every label of the graphs.
For example, labelingthe properties of molecular graphs requires time-consuming laboratory experiments [11].
Recently, various self-supervised learning methods for GNNs [11, 44, 12, 27] have been studied toovercome this issue of the lack of labeled graph data.
The existing self-supervised learning methods can be classified into two categories: predictive learning and contrastive learning.
Predictive learning methods [11, 22, 14] aim to learn representations by predicting contexts of a graph (Figure 1, (a-1)).
However, predictive learning schemes are limited as they consider only the subgraphical semantics
Contrastive learning [31, 26, 41, 45, 42, 36], which is a popular self-supervised learning approach inthe computer vision field [24, 34, 4, 9], can learn the global semantics by maximizing the similarity between instances (images) perturbed by semantic-invariant perturbations (e.g. color jittering orflipping).
Graph contrastive learning methods also use a similar strategy, generating positive examples
[Figure]
𝑑 =
Global Graph Prediction 𝑑 =
Original
D i sc r
i m i
na t o r
Perturbed Perturbed
[Figure]
with edge perturbation, attribute masking, and subgraph sampling, and then maximizing the similaritybetween two differently perturbed graphs from the original graph (See Figure 1, (b-1)).
However, is it always correct to assume that the perturbed graphs by such perturbations will preservethe semantics of the original graph?
In computer vision, there exists a set of well-defined perturbations,that does not change the human perception of the given image.
However, in contrast to images thatreside in a continuous domain, graphs are discrete data structures by nature, and thus their propertiescould become completely different even with slight perturbations.
For example, two molecules in
Figure 1 (d) and (e) show that, although they have a highly correlated structure, one molecular graphcould be actively bound to a set of inhibitors of human β -secretase 1 [33], whereas the other is not.
In this case, existing contrastive learning with perturbations may lead the representations for twosemantically dissimilar graphs to collapse into the one representation (See Figure 1 (b-2)).
To tackle this problem, we propose a novel self-supervised learning method which aims to learn thediscrepancy between graphs, which we refer to as D iscrepancy-based S
elf-supervised L e A rning (DSLA).
Specifically, we first perturb the given graph similarly as with contrastive learning schemes, butinstead of maximizing the similarities across the perturbed graphs as done with contrastive learning,we aim to learn their discrepancies.
To this end, we first introduce a discriminator which learns todiscriminate real graphs from the perturbed ones (See Discriminator in Figure 1, (c-1)).
This allowsthe model to learn small differences that may largely impact the global property of the graph, asillustrated by the example in Figure 1 (d) and (e).
However, simply knowing that two graphs aredifferent is insufficient, and we would want to learn the exact amount of discrepancy between them.
Notably, with our perturbation scheme, the exact discrepancy can be trivially obtained since the graphedit distance [23] is automatically derived as the number of edges we perturb for a given graph.
Then,we enforce the embedding-level difference between original and perturbed graphs to be proportionalto the graph edit distance (See dotted circles in Figure 1 (c-1)).
Finally, to discriminate the originaland perturbed graphs from the completely different graphs, we enforce the relative distance of thelatter to be larger than the former, by a large margin (See blue arrows in Figure 1, (c-1)).
This will enable our D-SLA to capture both small and large topological differences (See Figure 1(c-2)).
Thus, our framework can enjoy the best of both worlds for graph self-supervised learning:predictive and contrastive learning.
The experimental results show that our model significantlyoutperforms baselines, especially on the datasets where graphs with different properties have highlycorrelated structures, demonstrating the effectiveness of ours.
Our main contributions are as follows:
• We propose a novel graph self-supervised learning framework with a completely opposite objective
from contrastive learning, which aims to learn to differentiate a graph and its perturbed ones using a
discriminator, as even slight perturbations could lead to completely different properties for graphs.• Utilizing the graph edit distance that is obtained for perturbed graphs at no cost, we propose a novel
objective to preserve the exact amount of discrepancy between graphs in the representation space.
• We validate our D-SLA by pre-training and fine-tuning it on various benchmarks of chemical,
biology
and social domains, on which it significantly outperforms baselines.
Related Work
We now briefly review the existing works on graph neural networks (GNNs), and self-supervisedlearning methods for GNNs including predictive and contrastive learning.
Graph Neural Networks Most existing graph neural networks (GNNs) could be formulated underthe message passing scheme [7], which represents each node by firstly aggregating the featuresfrom its neighbors, and then combining the aggregated message with its own node representation.
Different variants of update and aggregation functions have been studied.
To mention a few, Graph
Convolutional Network (GCN) [15] generalizes the convolution operation in a spectral domain ofgraphs, approximated by the mean aggregation.
GraphSAGE [8] concatenates the representationsof neighbors with its own representation when updating the node representation.
Graph Attention
Network (GAT) [30] considers the relative importance among neighboring nodes for neighborhoodaggregation, which helps identify relevant neighbors for the given task.
Graph Isomorphism Network(GIN)
[37] uses the sum aggregation, allowing the model to distinguish two different graphs aspowerfully as the Weisfeiler-Lehman (WL) test [17].
While GNNs have achieved impressive resultson various graph-related tasks, the trained representations from one dataset are usually not transferableto different downstream tasks.
Moreover, labels of such task-oriented data are often scarce, especiallyin scientific domains (e.g., chemistry and biology)
Thus, in this work, we aim at obtainingtransferable representations of graph-structured data with self-supervised learning, without using anylabels.
Self-supervised learning for GNNs can be broadly classified into two categories: predictivelearning and contrastive learning, which we will briefly introduce in the following paragraphs.
Predictive Learning for Graph Self-supervised Learning Predictive learning aims to learncontextual relationships by predicting subgraphical features, for example, nodes, edges, and subgraphs.
Specifically, Hu et al.
[11] propose to predict the attributes of masked nodes.
Also, Hwang et al.
[13]and Kim and Oh [14] propose to predict the presence of an edge or a path with the link predictionscheme.
Furthermore, Hu et al.
[12] and Rong et al.
[22] propose to predict the generative sequence,contextual properties, and motifs of the given graphs.
However, the predictive learning methods arelimited in that they may not well capture the global structures and/or semantics of graphs.
Contrastive Learning for Graph Self-supervised Learning The limitations of predictive learningfor graph self-supervised learning gave rise to contrastive learning methods, which aim to captureglobal graph-level information.
Early contrastive learning methods for GNNs aim to learn thesimilarity between the entire graph and its substructure, to obtain the representations for them withoutusing any perturbations [31, 26].
Despite their successes, to further expand the embedding space ofa model, contrastive learning methods have been actively studied with perturbation methods suchas attribute masking, edge perturbation, and subgraph sampling [41, 45, 32, 42].
Recently proposedadversarial methods further generate positive examples either by adaptively removing the edges [28]or by adjusting the attributes [39].
Further, Xu et al.
[38] aim to cluster the graph representations viacontrastive learning, and Thakoor et al.
[29] propose a memory-efficient method for large-scale graphrepresentation learning.
However, while contrastive learning allows learning the global semanticswith perturbed graphs, there exists a potential risk where the model can consider two dissimilar graphsas the same (See Figure 1 (b), (d), and (e)), due to the discrete nature of the graphs.
Contrarily, ournovel pretext task of predicting the correct graph from multiple choices, which includes both slightlyperturbed graphs and strongly perturbed graphs, allows the model to learn the correct relationshipsamong the local elements, but also learn the global graph-level differences.
Method
In this section, we introduce our novel graph self-supervised learning framework, D iscrepancy-based
S elf-supervised L e A rning ( D-SLA ), which is illustrated in Figure 2.
Specifically, we first recount thenotion of graph neural networks and existing graph self-supervised learning in section 3.1.
Then, weintroduce each component of our D-SLA in Section 3.2, 3.3, and 3.4, respectively.
After that, wefinally describe the overall framework that combines all three components in Section 3.5.
3.1 Preliminaries
Graph Neural Networks A graph G can be defined by four components: G =
( V , E , X V , X E ) ,where V is the set of nodes
, E is the set of edges,
X V ∈ R |V|× d is the matrix of node features, and
𝐺 0 (Original Graph)
Edge Addition
Edge Deletion 𝐺 1 :𝑑 =𝐺 3 :𝑑 =2 𝐺 2 :𝑑 =
N ode
A tt r i bu t e M
a sk
i ng
G r aph
E n c od
i ng
Node Masking
Other graphs in the same batch
Original
Graph Discriminator
Perturbed Perturbed Perturbed 𝑮′𝑮𝑮𝑮 𝟑 𝑮𝑮′ 𝟐 𝒅= 𝒅=𝟐 Distance-based Learning 𝒅>𝜶+ 𝟐 𝒅>𝜶+
X E ∈ R |E|× d ′ is the matrix of edge features, where d and d ′ are the dimensionalities of node andedge attributes, respectively.
Let h v be a representation of a node v , and h G be a representation of agraph G .
Then, the goal of graph neural networks (GNNs) is to represent nodes of the given graphby leveraging its topological structure.
To be specific, GNNs learn node representations (i.e., h v )by iteratively aggregating messages from their neighbors in a layer-wise manner, which is usuallyreferred to as message-passing [7] and formally defined as follows:
[Equation]
where f ( l ) U denotes an update function that updates a representation of the given node along with itsneighbors’ representations, f ( l ) A denotes an aggregate function that aggregates messages from thenode’s neighbors, N ( v ) denotes a set of neighbors of the node v , and l denotes a l -th layer of GNNs.
To further obtain the representation for an entire graph, we usually summarize all node representationsfrom Equation 1 into a single embedding vector with a permutation-invariant function f R as follows:
[Equation]
While a natural choice of f R is to use simple operations, such as mean or sum, various graphrepresentation learning methods have been recently studied to accurately capture the entire graphinformation, including node clustering based [40, 2], and node drop based methods [6, 16].
Contrastive Learning for Self-supervised Learning of GNNs We now describe why contrastivelearning fails to distinguish two topologically similar graphs yet having completely different properties.
In particular, contrastive learning aims to increase the similarity between positive pairs of graphswhile increasing the dissimilarity between negative pairs [41, 45, 42].
Thereat, to define positive pairs,they perturb the original graph by manipulating edges, masking attributes, and sampling subgraphs,and then consider the perturbed graphs as the same as the original graph.
On the other hand, theyconsider other graphs in the same batch as dissimilar.
Formally, for the perturbed graphs G i and G jfrom the original graph G 0 , the objective of contrastive learning is defined as follows:
[Equation]
where G ′ is the other graph in the same batch with the graph G 0 , which is also referred to as a negativegraph .
Therefore, G i and G j are a positive pair, whereas, G i and G ′ are a negative pair.
f sim denotes asimilarity function between two graphs, for example, L 2 distance or cosine similarity.
By minimizingthe objective in Equation 3, existing contrastive learning methods closely embed two perturbed graphs
G i and G j : G i ∼ G j .
However, as the perturbations may not preserve the properties of the given graphdue to the discrete nature of graph-structured data, they indeed should not be similar: G i ̸∼ G j (See
Figure 1 (d) and (e)).
To this end, we propose a new objective that can differentiate between theoriginal graph and its perturbations, which we describe in the next subsection.
3.2 Discrepancy Learning with Graph Discrimination
We now introduce our novel graph self-supervised learning objective to preserve the discrepancybetween two graphs in the learned representation space, for which we then describe the perturbationscheme to generate augmented graphs from the original graph.
Discriminating Original Graphs from Perturbed Graphs Contrarily to contrastive learning thattrains perturbed graphs to be similar, our goal is to distinguish the original graph from perturbedgraphs, by predicting the original graph among original and perturbed graphs, which we term as theoriginal graph discrimination (See Graph Discriminator in Figure 2).
Let G 0 be an original graph,and G i be a perturbed graph with i ≥ 1 .
Then objective of our discrimination scheme is as follows:
[Equation]
where S k is a score of the graph G k , obtained from a learnable score network f S of the discriminator,i.e., f S : h G (cid:55)→ S ∈ R .
By training to discriminate the original graph from the perturbed graphs,the model is enforced to embed the perturbed graphs apart from the original graph (Figure 3 (b)).
Therefore, the model learned by our discrimination scheme will be capable of distinguishing even theslight differences between the original graph and its perturbations, as well as be capable of capturingthe correct distribution of the graphs, since perturbed graphs could be semantically incorrect.
Perturbation The remaining question is how to perturb the graph to generate negative examples(i.e.
, G i ).
We aim at adding or deleting the subset of edges in the given graph, and thus such perturbedgraphs are slightly different from the original graph but also could be semantically incorrect.
Again, inthis way, since our graph discrimination objective in Equation 4 discriminates the original graph fromslightly perturbed graphs, the graph-level representations from our pre-trained GNNs can capturesubtle difference that may have a large impact on the properties of graphs, for downstream tasks.
In particular, our perturbation scheme consists of the following two steps: 1) removing and adding asmall number of edges, and 2) masking node attributes (Figure 2).
To be specific, given an originalgraph G
=
( V , E , X V , X E ) , we aim to perturb it n times, to obtain {G 1 , ..., G n } .
To do so, we firstmanipulate the edge set E by removing existing edges as well as adding new edges on it and thenadjust its corresponding edge matrix X E .
After that, we further randomly mask the node attributeson X V for both original and perturbed graphs, to make it more difficult to distinguish between them.
Formally, the original and perturbed graphs from our node/edge perturbations are obtained as follows:
[Equation]
where M and P are the node masking and edge perturbation functions, respectively.
The scores oforiginal and perturbed graphs are then computed with the score function f S :
[ S 0 , S 1 , ..., S n ] =[ f S ( G 0 ) , f S ( G 1 ) , ..., f S ( G n )] , which are then used for our previous learning objective in Equation 4.
3.3 Learning Discrepancy with Edit Distance
While the learning to discriminate original and perturbed graphs described in Section 3.2 allowsthe model to learn the discrepancy between original and perturbed graphs by embedding them ontodifferent points, it cannot learn how dissimilar the perturbed graph is from the original one as allperturbed graphs are considered as equal (Figure 3 (b)).
Thus, to further learn the accurate discrepancyamong different graphs, here we introduce a method for efficiently computing the graph discrepancybetween original and perturbed graphs leveraging the graph edit distance [23].
Then we describe howto preserve the calculated discrepancy in the learned graph representation space.
[Table]
Graph Edit Distance To preserve the exact amount ofdiscrepancies between original and perturbed graphs in thelearned representation space, we first need to measure thegraph distance.
Graph edit distance, which is widely usedto measure the dissimilarity between two graphs [23], isdefined by the number of insertion, deletion, and substitution operations for nodes and edges, to transform one graph into the other graph.
Although computingthe graph edit distance is the NP-hard problem [43], for our case, it is trivially obtained without anycost.
This is because, for each perturbed graph, we know the exact number of edge addition anddeletion steps as we have generated it using the same process.
In other words, the number of addedand deleted edges is simply the edit distance between original and perturbed graphs.
This bringsus significant computational advantages, as we generally require a significant amount of costs for
[Figure]
calculating or estimating the distances between two graphs, as shown in Table 1.
Note that althoughseveral existing works [18, 3] leverage graph edit distance, they are completely different from ourwork since they are neither self-supervised learning methods nor leverage the graph perturbation.
Distance-based Discrepancy Learning Based upon the graph edit distance, we design the regularization term to learn the exact amount of differences between original and perturbed graphs over theembedding space.
To be specific, we regularize the model to learn that the embedding-level differencebetween original and perturbed graphs is proportional to their actual graph edit distance (i.e., if theedit distance between two graphs is large, then they are far away in the embedding space, whereas, ifit is small, then they are close to each other) as shown in Distance-based Learning of Figure 2.
We first let an original graph G 0 be an anchor graph, then a graph edit distance and an embedding-leveldistance between the anchor graph G 0 and the perturbed graph G i are defined as e i and d i , respectively.
With such notations, to achieve our objective of learning an exact amount of discrepancy acrossoriginal and perturbed graphs, we formalize our edit distance-based regularization loss as follows:
[Equation]
where f diff measures the embedding-level differences between graphs with L 2 -norm.
We findthat Equation 6 can capture the exact amount of differences as shown in Figure 3 (c).
Note that
Equation 6 alone does not work as shown in Table 5, since the trivial solution of Equation 6 is toset all the embedding-level distance between original and perturbed graphs as zero (i.e., d i = 0 ∀ i ) .
However, learning the accurate amount of discrepancy without the trivial solution is feasible by jointlytraining with the objective of original graph discrimination in Equation 4 where the model shoulddifferently embed G
and G i to discriminate them.
Furthermore, as we can generate different levelsof perturbations according to the graph edit distance, our graph embedding space can interpolatebetween two different graphs, for example, weakly and strongly perturbed graphs in terms of theiractual distances, unlike previous predictive and contrastive learning methods.
3.4 Relative Discrepancy Learning with Other Graphs
Thus far, we aimed toward distinguishing the perturbed graphs from the original graph in Section 3.2,with their actual edit distance-based discrepancy learning in Section 3.3.
However, such schemesalone will not allow the model to learn the discrepancy between two completely different graphs(Figure 3 (c)).
While one might consider learning the discrepancy between original and its negativegraphs (other graphs in the same batch) with their graph edit distance, it is impractical as computingthe edit distance between completely different graphs is NP-hard [43].
Thus, we instead propose tolearn a relative distance, exploiting the assumption that the distance between original and negativegraphs in the same batch is larger than the distance between original and perturbed graphs with someamount of margin (See Figure 2, Distance-based Learning).
The usage of margin is highly beneficialfor our discrepancy-based learning framework, since, if the negative graphs are far apart than theamount of margin plus the distance between original and perturbed graphs, the model does not attractthe perturbed graphs to the original graph, therefore not losing the discrepancy learned in Equation 6.
Formally, we realize the relative discrepancy learning with the triplet margin loss as follows:
[Equation]
where d i is the distance between original and its perturbed graphs: d i = f diff ( h G 0 , h G i ) , d ′ j is thedistance between original and its negative graphs: d ′ j = f diff ( h G 0 , h G ′ j ) with G ′ as in-batch negativegraphs, and α > 0 as a margin hyperparameter.
In this way, we can allow the model to learn that, forthe original graph, negative graphs are more dissimilar than the perturbed graphs, while perturbedgraphs are also marginally dissimilar to it with the amount of edit distances as shown in Figure 3 (d).
3.5 Overall Framework
To sum up, our D-SLA framework aims to learn a graph representation space which preserves thediscrepancy among graphs.
Our graph discriminator described in Section 3.2 helps learn a space thatcan discriminate even the slightly perturbed graph from its original graph (Figure 3 (b)).
Then, thediscrepancy learning with graph edit distance described in Section 3.3 (Figure 3 (c)) enforces theembedding space to preserve the exact discrepancy among graphs.
Finally, the margin-based tripletconstraints between the perturbed and completely different graphs in Section 3.4 allow the learnedembedding space to capture the relative distance between graphs (Figure 3 (d)).
Our overall learningobjective, dubbed as Discrepancy-based Self-supervised LeArning (D-SLA), is given as follows:
[Equation]
where hyperparameters λ 1 and λ 2 are scaling weights to each loss.
Note that, unlike predictivelearning [11] which aims to capture local semantics of graphs by masking and predicting localcomponents, such as nodes and edges, our D-SLA learns the graph-level representations where thesubtle differences on corner regions of different graphs could be captured by distinguishing originaland perturbed graphs.
Moreover, our D-SLA can discriminate differently perturbed graphs havingdifferent properties, unlike contrastive learning [41, 42] that considers them as similar.
Experiments
In this section, we experimentally validate the proposed Discrepancy-based Self-supervised LeArning(D-SLA) on both graph classification and link prediction tasks, for which capturing the global andlocal semantics of graphs is important, respectively.
4.1 Graph Classification
Accurately capturing the global semantics of given graphs is crucial for graph classification tasks, onwhich we validate the performance of our D-SLA against existing baselines.
Experimental Setup We evaluate our D-SLA on two different domains: the molecular propertyprediction task from chemical domain [33, 10] and the protein function prediction task from biologicaldomain [11].
For the chemical domain, we follow the experimental setup from You et al.
[42], whosegoal is to predict the molecules’ biochemical activities.
For the biological domain, we follow thesetup from You et al.
[42], for which the goal is to predict the proteins’ biological functions.
We usethe ROC-AUC value as an evaluation metric, and report the average performance over five differentruns following Xu et al.
See Section A.2 in the supplementary file for dataset details.
Models We compare our D-SLA against predictive learning baselines:
EdgePred [8], AttrMasking [11] and ContextPred [11], and contrastive learning baselines: Infomax [31], GraphCL [41],
JOAO [42], GraphLoG [38], and BGRL [29].
Detailed explanations of models including baselinesand ours are provided in Section A.1 of the supplementary file.
Implementation Details We follow the conventional experimental setup of graph self-supervisedlearning from Hu et al.
[11], where we use the GIN [37] as the base network.
For pre-training of ourmodel, we perturb the original graph three times.
To be specific, to obtain the perturbed graphs fromthe original graph, we first randomly select a subgraph of it, and then add or remove edges in the rangeof { 20% , 40% , 60% } over the sampled subgraph.
For fine-tuning, we follow the hyperparametersfrom You et al.
We provide additional details in Section A.2 of the supplementary file.
Results Table 2 shows that our D-SLA achieves the best average performance against existingpredictive and contrastive learning baselines on tasks from both chemical and biological domains,demonstrating the effectiveness of our discrepancy-based framework.
To better see what aspects of
D-SLA contribute to the performance improvements, we perform in-depth analyses of each dataset,and how our model can effectively handle the task on it, in the next paragraph.
Analysis To analyze whether the structural similarity of molecules is correlated to their biochemicalactivities, we measure the inherent discrepancy of graphs with the Tanimoto similarity over Morganfingerprints [21].
To be specific, we first iteratively sample an anchor molecule among activemolecules in the dataset, and then measure the average Tanimoto similarities of the five most similaractive/inactive molecules.
In other words, the high similarity values of Inact.
Sim.
in Table 3 suggests
[Table]
[Figure] [Figure]
that the molecules have highly overlapped structures regardless of their biochemical activities.
Forexample, as shown in Table 3, the molecules in the BACE dataset are highly correlated, although theiractivities are different.
Also, we further observe that, as shown in Figure 4 that visualizes the mostsimilar active/inactive molecules with respect to the certain anchor molecule in the BACE dataset,the structures between active and inactive molecules are highly similar.
From the above observations, we suggest that, due to the discrete nature of graphs, two slightlydifferent graphs can have completely different properties, which may be the reason for the performancedegeneration of contrastive learning methods in Table 2 – which consider perturbed graphs as similarwhile they might not be the semantically same – on such particular datasets (e.g., BACE, MUV, Tox21,
ToxCast) having the high Inact.
Sim. scores in Table 3.
However, our D-SLA largely outperformscontrastive learning baselines on them 1 , because ours not only discriminates an original graph fromits perturbations but also can learn their exact discrepancy via the graph edit distance.
We note thatour method shows the competitive performance on ClinTox and SIDER in Table 2, since they have thelowest structural similarities across different biochemical properties, for which contrastive learningcould be effective.
We provide more analysis in Section B.1 of the supplementary file.
4.2 Link Prediction
Accurately capturing the local semantics of a graph is an important requisite for solving node/edgelevel tasks.
Thus, we further validate our D-SLA on the link prediction task.
Experimental Setup We conduct link prediction experiments on social network datasets – COLLAB, IMDB-B, and IMDB-M – from the TU benchmarks [19].
We separate the dataset into fourparts: pretraining, training, validation, and test sets in the ratio of 5:1:1:3.
We use the accuracy – theratio of correctly predicted edges among all edges – as an evaluation metric, and report the resultsover five different runs.
We provide more details in Section A.3 of the supplementary file.
Implementation Details For GNNs, we use the GCN [15] following You et al.
For perturbation, we add or delete only a tiny amount of edges (e.g., 1 or 2 edges) while increasing the magnitudeof perturbation to obtain three perturbed graphs.
Note that we do not use the margin triplet loss in
Section 3.4, since for local prediction tasks, the graph-level discrepancy learning between completelydifferent graphs is not much helpful in capturing local semantics.
For fine-tuning, we train the GNNsto predict whether there is an edge between nodes, and evaluate the GNNs for predicting the existenceof 10 edges.
For more implementation details, please see Section A.3 of the supplementary file.
Results As shown in Table 4, a predictive learning baseline – AttrMasking – outperforms otherbaselines, since it learns to predict local node/edge attributes.
However, our D-SLA largely outper- [Table] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table] [Figure] [Table] [Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
forms all baselines, while it aims to discriminate graph-level representations though.
We suggest thatthis is because ours can capture subtle differences of graphs by leveraging the graph edit distance,demonstrating that accurate discrepancy learning is obviously useful for the local link prediction task.
4.3 Analysis
In this section, we further analyze the efficacy of our D-SLA.
Specific experimental details areprovided in Section A.4 of the supplementary file.
Rank Correlation Coefficient To see whether learned representations capture the exact amountof discrepancy, we compare the ranks of the calculated vs actual similarities between original andperturbed graphs with Spearman’s rank correlation coefficient.
Note that the results of the WLalgorithm [17] are merely a performance indicator of discriminative power since it cannot obtainrepresentations that generalize to downstream tasks.
As shown in Figure 5, unlike baselines thatmostly fail to discriminate different graphs, our D-SLA has the discriminative power that is on parwith the powerful WL algorithm, while ours can generalize to downstream tasks as shown in Table 2.
[Table]
Ablation Study To see how much each component contributesto the performance gain, we conduct an ablation study.
As shownin Table 5, we observe that our graph discrimination task ( L GD )significantly improves the performance on down-stream tasks againstno-pretraining.
However, leveraging the exact edit distance alone( L edit ) does not learn meaningful representations, since the modeltrivially sets all the distances between original and its perturbationsas zero, as discussed in Section 3.3.
Also, we further observe thateach component of distance-based learning in Section 3.3 and 3.4 helps to improve the performance,verifying that accurate discrepancy learning with edit and relative distances is important for modelinggraphs.
We further discuss the ablation study in Section B.2 of the supplementary file.
Embedding Visualization We visualize the embedding space from various pre-training methods:predictive, contrastive, and ours, in Figure 1 (a,b,c-2).
We observe that predictive learning cannotcapture the global graph-level similarity well, as it aims to predict subgraphical semantics of graphsduring the pretext task.
Also, while contrastive learning can closely embed the highly similar graphsto the original graph, it cannot accurately capture the exact amount of discrepancies among perturbedgraphs.
Contrarily, our D-SLA can accurately distinguish between the original, perturbed, andnegative graphs, while accurately capturing the exact amount of discrepancy for the perturbed graphs.
Conclusion
In this work, we focused on the limitations of existing self-supervised learning for GNNs: predictivelearning does not capture the graph-level similarities; contrastive learning might treat two semanticallydifferent graphs from perturbations as similar.
To overcome such limitations, we proposed a novelframework (D-SLA) that can learn the graph-level differences among different graphs while also canlearn the slight edge-wise differences, by discriminating the original from perturbed graphs.
Further,the model is trained to differentiate the target graph from its perturbations and other graphs, whilepreserving the accurate graph edit distance, allowing the model to discriminate between not only twostructurally different graphs but also similar graphs with slight differences.
We validated our D-SLAon 12 benchmark datasets, achieving the best average performance.
Further analysis shows that itlearns a discriminative space of graphs, reflecting the graph edit distances between them.
We providethe limitations and potential societal impacts of our work in Section C of the supplementary file.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] We discuss the limitation in
Section C of the supplementary file.
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] We discuss
the potential societal impacts in Section C of the supplementary file.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes] We provide thecode, data, and instructions in the supplementary materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] We provide the detailed experimental setups in section A of the
supplementary file.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[Yes] All of our main results contain variances or standarddeviations of multiple runs.
[Table]
(d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] We provide the type of GPUs inSection A of the supplementary file.
[Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] We uses the publiccode and data and we provide the citation and URLs of them in Section A in the
[Table]
supplementary file.
[Table]
(b) Did you mention the license of the assets?
[N/A]
[Table]
(c) Did you include any new assets either in the supplemental material or as a URL?
[N/A]
[Table]
(d) Did you discuss whether and how consent was obtained from people whose data you’re
[Table]
using/curating?
[N/A]
[Table]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
[Table]
information or offensive content?
[N/A]
[Table]
If you used crowdsourcing or conducted research with human subjects...
[Table]
(a) Did you include the full text of instructions given to participants and screenshots, if
[Table]
applicable?
[N/A]
[Table]
(b) Did you describe any potential participant risks, with links to Institutional Review
[Table]
Board (IRB) approvals, if applicable?
[N/A]
[Table]
(c) Did you include the estimated hourly wage paid to participants and the total amount
[Table]
spent on participant compensation?
[N/A] [Table]