Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination
Abstract
Graph contrastive learning (GCL) alleviates the heavy reliance on label informationfor graph representation learning (GRL) via self-supervised learning schemes.
Thecore idea is to learn by maximising mutual information for similar instances, whichrequires similarity computation between two node instances.
However, this operation can be computationally expensive.
For example, the time complexity of twocommonly adopted contrastive loss functions (i.e., InfoNCE and JSD estimator)for a node are O ( ND ) and O ( D ) , respectively, where N is the number of nodes,and D is the embedding dimension.
Additionally, GCL normally requires a largenumber of training epochs to be well-trained on large-scale datasets.
Inspired byan observation of a technical defect (i.e., inappropriate usage of Sigmoid function)commonly used in two representative GCL works, DGI and MVGRL, we revisit
GCL and introduce a new learning paradigm for self-supervised GRL, namely,
Group Discrimination (GD), and propose a novel GD-based method called Graph
G roup Discrimination ( GGD ).
Instead of similarity computation, GGD directly discriminates two groups of node samples with a very simple binary cross-entropyloss.
In addition, GGD requires much fewer training epochs to obtain competitiveperformance compared with GCL methods on large-scale datasets.
These twoadvantages endow GGD with very efficient property.
Extensive experiments showthat GGD outperforms state-of-the-art self-supervised methods on eight datasets.
In particular, GGD can be trained in 0.18 seconds (6.44 seconds including datapreprocessing) on ogbn-arxiv, which is orders of magnitude (10,000+) fasterthan GCL baselines while consuming much less memory.
Trained with 9 hourson ogbn-papers100M with billion edges, GGD outperforms its GCL counterparts in both accuracy and efficiency.
Introduction
Graph Neural Networks (GNNs) have been widely-adopted in learning representations for graphstructured data.
With message-passing to utilise the topology of a graph, GNNs can learn effectivelow-dimensional node embeddings, which can be used for a variety of downstream tasks such as nodeclassification [1], link prediction [2], and graph classification [3].
GNNs have been further applied indiverse domains, e.g., social networks [4], recommendation system [4], and anomaly detection [5, 6].
However, many GNNs adopt a supervised learning manner to train models with label information,which is expensive and labour-intensive to collect in real-world.
To address this issue, a few studies(e.g.
, DGI [7], MVGRL [3], GMI [8], GRACE [9], and Subg-Con [10]) borrow the idea of contrastivelearning from computer vision (CV), and introduce Graph contrastive learning (GCL) methodsfor self-supervised GRL.
The core idea of these methods is to maximise the mutual information
(MI) between an anchor node and its positive counterparts, sharing similar semantic informationwhile doing the opposite for negative counterparts as shown in Figure 1(a).
Nonetheless, sucha scheme relies on expensive similarity calculation in contrastive loss computation.
Taking twocommonly-adopted contrastive loss functions, InfoNCE and JSD estimator as examples, the timecomplexity of loss computation for a node is O ( ND ) and O ( D ) , respectively.
Here, N representsthe number of nodes, and D is the embedding dimension.
Additionally, GCL normally requires alarge number of training epochs to be well-trained on large-scale datasets.
Thus, when the size of thedataset is large, these methods require a significant amount of time and resources to be well-trained.
[Figure]
Though a few GCL works attempt toalleviate the training load of InfoNCEwith specially designed schemes, e.g.,
BGRL [12] and GBT [13], their proposed contrastive learning losses stillhave high complexity in training.
Inspired by BYOL [14], BGRL [12]adopts a bootstrapping scheme and remove negative node pairs.
It only contrasts a node from the online network(i.e.
, updated with gradient) to its corresponding embedding from the target network (i.e., updated momentumly withstop gradient).
This contrastiveness requires O ( D ) for a node.
Based on
Barlow-Twins [11], GBT [13] utilises a cross-correlation-based loss to get rid of negative samples.
However, it still requires O ( ND ) for the loss computation of an embedding dimension.
[Table]
To boost training efficiency of self-supervised
GRL, inspired by an observation of a technicaldefect (i e., inappropriate application of Sigmoidfunction) in two representative GCL studies weintroduce a novel learning paradigm, namely,
Group Discrimination (GD).
Remarkably, with
GD, the time complexity of loss computationfor a node is only O (1) , endowing the schemewith the extremely efficient property.
Insteadof similarity computation, GD directly discriminates a group of positive nodes from a group ofnegative nodes, as shown in Figure 1(b).
Specifically, GD defines node samples generated withoriginal graph as the positive group, while nodesamples obtained with corrupted topology areregarded as the negative group.
Then, GD trainsthe model by classifying these node samplesinto the correct group with a very simple binarycross-entropy loss.
By doing so, the model can extract valuable self-supervised signals from learning the edge distribution of a graph.
Compared with GCL, GD enjoys numerous merits includingextremely fast training, fast convergence (e.g., 1 epoch to be well-trained on large-scale datasets),and high scalability while achieving SOTA or on par performance with existing GCL approaches.
Using GD as backbone, we design a new self-supervised GRL model with the Siamese structurecalled Graph Group Discrimination (GGD).
Firstly, we can optionally augment a given graph withaugmentation techniques, e.g., feature and edge dropout.
Then, the augmented graph is fed into a
GNN encoder and a projector to obtain embeddings for the positive group.
After that, the augmentedfeature is corrupted with node shuffling (i.e., disarranging the order of nodes in the feature matrix)to disrupt the topology of a graph and input to the same network for obtaining embeddings of theopposing group.
Finally, the model is trained by discriminating the these two groups of node samples.
The contributions of this paper are three-fold: 1) We introduce a novel and efficient self-supervised
GRL paradigm, namely, Group Discrimination (GD).
Notably, with GD, the time complexity of loss computation for a node is only O (1) .
Based on GD, we propose a new self-supervised GRLmodel, GGD, which is fast in training and convergence, and possess high scalability.
We conductextensive experiments on eight datasets, including an extremely large dataset, ogbn-papers100Mwith billion edges.
The experiment results show that our proposed method reaches state-of-the-artperformance while consuming much less time and memory than baselines, e.g., 10783 × fasterthan the most efficient GCL baseline with its best selected epochs number [13], as shown in Table 1.
Rethinking Representative GCL Methods
[Figure]
In this section, we analyse a technical defectobserved in two representative GCL methods,
DGI [7] and MVGRL [3].
Based on the technical defect, we show the learning method behindthese two approaches is not contributed to Contrastive Learning, but a new paradigm, Group
Discrimination.
Finally, from the analysis, weprovide the definition of this new concept.
2.1 Rethinking GCL Methods
DGI [7] is the first work introducing Contrastive
Learning into GRL.
However, due to a technical defect observed in their official open-sourcecode, we found it is essentially not working as the authors thought (i.e., learning via MI interaction).
The backbone that truly makes it work is a new paradigm, Group Discrimination.
Constant Summary Vector.
As shown in Figure 2, the original idea of DGI is to maximise the MI(i.e.
, the red line) between a node a and the summary vector s , which is obtained by averaging allnode embeddings in a graph G .
Also, to regularise the model training, DGI corrupts G by shuffling thenode order of the input feature matrix to get ˜ G .
Then, generated embeddings of ˜ G serve as negativesamples, which are pulled apart from the summary vector s via MI minimisation.
[Table]
Nonetheless, in the implementation of
DGI, a Sigmoid function is inappropriately applied on the summary vectorgenerated from a GNN whose weightis initialised with Xavier initialisation.
As a result, elements in the summaryvector are very close to the same value.
We have validated this finding on threedatasets, Cora, CiteSeer and PubMed.
The experiment result is shown in Table 2, which shows that summary vectors in all datasets areapproximately a constant vector ϵI , where ϵ is a scalar and I
∈
R D is an all-ones vector (i.e., ϵ =0.50with ReLU/LReLU/PReLU and ϵ =0.62 with Sigmoid as non-linear activation in these datasets).
[Table]
To theoretically explain this phenomenon, we presentthe theorem below:
Theorem 1 Given G =
( X ∈ R N × D , A ∈ R N × N ) ,and a GCN encoder g () initialised with Xavier initialisation, we can obtain its embedding H =
g ( G ) .
Then, when D increases, the value in H converges to0.5 with ReLU/LReLU/PReLU or 0.62 with Sigmoid as non-linear activation for g () .
Based on this theorem, we can see if the feature dimension D is large, these summary vectors canlose variance and become a constant vector.
The proof for the theorem is presented in Appendix A.1.
To evaluate the effect of ϵ to constant summary vector, we vary the scalar ϵ (from 0 to 1 incrementby 0.2) to change the constant summary vector and report the model performance (i.e., averagedaccuracy on five runs) in Table 3.
From this table, we can see, except for 0, the model performance is trivially affected by ϵ for constantsummary vector.
When the summary vector is set to 0, the model performance plummets becausenode embeddings become all 0 when multiplying with such vector and the model converges to thetrivial solution.
As the summary vector only has a trivial effect on model training, the hypothesis of
DGI [7] on learning via contrastiveness between anchor nodes and the summary instance does nothold, which raises a question to be investigated: What truly leads to the success of DGI?
Simplifying DGI.
To answer the question, we predigest the loss proposed in DGI by using an all-onesvector as the summary vector s (i.e., setting s = ϵI
= I ) and simplifying the discriminator D (i.e.,removing the learnable weight vector).
Then, we rewrite the loss to the following form:
[Equation]
where · is the vector multiplication operation, N is the number of nodes in a graph, z i ∈ R 1 × D and˜ z i ∈ R 1 × D are the original and corrupted embedding for node i , sum () is the summation function,and D is a discriminator for bilinear transformation, which can be formulated as follows:
[Equation]
proved on top of DGI .
[Table]
where w is a learnable weight vector.
Specifically, asshown in Equation 2, by removing the weight vectorw , z i is directly multiplied with s .
As s is a vectorcontaining only one, the multiplication of z i and s isequivalent to summing z i itself directly.
From thisform, we can see that the multiplication of z i and thesummary vector only serves as an aggregation function (i.e., summation aggregation) to aggregate z i .
Toexplore the effect of other aggregation functions, wereplace the summation function in Equation 1 withother aggregation methods such as mean-, minimum-,and maximum- pooling, and present the experimentresult in Appendix A.3.
Based on Equation 1, we can rewrite it to a very simple binary cross entropy loss if we also includecorrupted nodes as data samples and setting h i =
agg ( z i ) , where agg () stands for aggregation:
[Equation]
[Equation]
where y i ∈ R 1 × 1 means the indicator for node i (i.e., if node i is corrupted, y i is 0, otherwise it is
1),and h i ∈
R 1 × 1 represents the prediction for a node sample i .
As we include corrupted nodes as datasamples, the size of nodes to be processed is doubled to 2 N (i.e., the number of corrupted nodes isequal to the number of original nodes).
From the equation above, we can easily observe that what
DGI truly does is discriminate between a group of nodes generated with correct topology and nodesgenerated with corrupted topology, as shown in Figure 1.
We name this self-supervised learningparadigm " Group Discrimination ".
To validate the effectiveness of this paradigm, we replace theoriginal DGI loss with Equation 3, namely, DGI BCE and compare it with DGI on three datasetsin terms of training time, memory efficiency and model performance as shown in Table 4.
Here,
DGI BCE adopts the same parameter setting as DGI.
From this table, we can observe DGI BCEdramatically improves DGI in both memory- and time-efficiency while it slightly enhances the modelperformance of DGI.
This is contributed to the removal of multiplication operations between nodepairs, which eases the burden of computation and memory consumption in both forward and backwardpropagation.
In section A.2, we further explore GD and compare it with existing contrastive losses.
Similar to DGI, the same technical defect is observed in MVGRL [3], which makes it become a
GD-based method.
Extended on DGI, MVGRL [3] incorporates diffusion augmentation to injectadditional global information into model training, which enhances the model performance.
Thedetailed analysis for MVGRL is presented in Appendix A.4.
2.2 Definition of Group Discrimination
As mentioned above, Group Discrimination is a self-supervised GRL paradigm, which learns by discriminating different groups of node samples.
Specifically, the paradigm assigns different indicatorsto different groups of node samples.
For example, for binary group discrimination, one group isconsidered as the positive group with class 1 as its indicator, whereas the other group is the negativegroup, having its indicator assigned as 0.
Given a graph G , the positive group usually includes nodesamples generated with the original graph G or its augmented views (i.e., similar graph instances of Gcreated by augmentation).
In contrast, the opposing group contains negative samples obtained bycorrupting G , e.g., changing its topology structure.
Methodology
[Figure]
We first define unsupervised node representation learning and then present the architecture of GGD ,which extends DGI BCE with additional augmentation, the projector and embedding reinforcementto reach better model performance.
Given a graph G with attributes X ∈ R N × D , where N is thenumber of nodes in G , and D is the number of dimensions of X , our aim is to train a GNN encoderwithout the reliance on labelling information.
With the trained encoder, taking G and X as input, itcan output learned representations H ∈ R N × ˆ D , where ˆ D is the predefined output dimension.
H canthen be used in many downstream tasks such as node classification.
3.1 Graph Group Discrimination
Based on the proposed self-supervised GRL paradigm, Group Discrimination, we have designed anovel method, namely GGD , to learn node representations using a Siamese network structure and a
BCE loss.
The architecture of the GGD is presented in Figure 3.
The framework mainly consists offour components: augmentation, corruption, a Siamese GNN network, and Group Discrimination.
Augmentation.
With a given graph G and feature matrix X , optionally, we can augment it withaugmentation techniques such as edge and feature dropout to create ˆ G and ˆ X .
In practice, we follow theaugmentation proposed in GraphCL [15].
Specifically, edge dropout removes a predefined fraction ofedges, while we use node dropout to mask a predefined proportion of feature dimension, i.e., assigning0 to replace values in randomly selected dimensions.
This step is optional in implementation.
Notably, the motivation of using augmentation in our framework is distinct from contrastive learningmethods.
In our study, augmentation is used to increase the difficulty of the self-supervised trainingtasks.
With augmentation, ˆ G and ˆ X change in every training iteration, which forces the model to lessenthe dependence on the fixed pattern (i.e., unchanged edge and feature distribution) in a monotonousgraph.
However, in Contrastive Learning, augmentation creates augmented views sharing similarsemantic information for building contrastiveness.
Corruption.
ˆ G and ˆ X are then corrupted to build ˜ G and ˜ X for the generation of node embeddings inthe negative group.
We adopt the same corruption technique used in DGI [7] and MVGRL [3] (asshown in Figure 5 in Appendix A.2.
The corruption technique devastates the topology structure of ˆ Gby randomly changing the order of nodes in ˆ X .
The corrupted ˜ X and ˜ G can be used for producingnode representations with incorrect network connections.
The Siamese GNN.
We have designed a Siamese GNN network to output node representations givena graph and its attribute.
The Siamese GNN network is made up of two components, which are a
GNN encoder and a projector.
The backbone GNN encoder is replaceable with a variety of choicesof GNNs, e.g., GCN [16] and GAT [17].
In our work, we adopt GCN as the backbone.
The projectoris a multi-layer perceptron network, whose number of layers can be adjusted.
When generating nodeembeddings of the positive group, the Siamese network takes ˆ G and ˆ X as input.
Using the sameencoder and projector, the Siamese network output the negative group with ˜ G and ˜ X .
These twogroups of node embeddings are considered as a collection of data samples with a size of 2 N fordiscrimination.
Before conducting group discrimination, in the “aggregation” phase, all data samplesare aggregated with the same aggregation technique, e.g., sum-, mean-, and linear aggregation.
Group Discrimination.
In the group discrimination process, we adopt a very simple binary crossentropy (BCE) loss to discriminate two groups of node samples as shown in Equation 3.
In ourimplementation, y i is 0 and 1 for node embeddings in negative and positive groups.
During modeltraining, the model is optimised by categorising node embeddings in the collection of data samplesinto their corresponding class correctly.
The loss is computed by comparing the prediction of a nodei , i.e., a scalar, with its indicator y i .
With the ease of BCE loss computation, the training process of
GGD is very fast and memory efficient.
3.2 Model Inference
During training, the model is optimised via loss minimisation with Equation 3.
The time complexityanalysis of GGD is provided in Appendix A.5.
In the inference phase, we freeze the trained GNNencoder g θ and obtain node embeddings H θ with the input G .
Inspired by MVGRL [3], which strengthens the output embeddings by including additional globalinformation, we adopt a conceptually similar embedding reinforcement approach.
Specifically, theyobtain the final embeddings by summing up embeddings from two views: the original view comprisinglocal information and the diffused view with global information.
This operation reinforces the finalembeddings and leads to model performance improvement.
Nonetheless, graph diffusion impairs thescalability of a model [18] and hence cannot be directly applied in our embedding generation.
Toavoid the diffusion computation, we have come up with a workaround in the virtue of the power of agraph to extract global information.
The power of a graph can extend the message passing scope of
H θ to n-hop neighbourhood, which encodes global information from distant neighbours.
It can beformulated as follows:
[Equation]
where H globalθ is the global embedding, and A is the adjacency matrix of the graph G .
It is notable thatthis operation can be easily decomposed with the associative property of matrix multiplication and iseasy to compute.
To show the easiness of such computation, we conduct an experiment showing itstime consumption on various datasets in Appendix A.6.
Finally, the final embedding can be achievedby H =
H globalθ + H θ , which can be used for downstream tasks.
In our experiment, we conduct nodeclassification tasks.
Following the common practice of GCL methods [7, 9, 12, 8, 11], these tasks areperformed by using the final embeddings H to train and test a simple logistic regression classifier.
Related Work
Graph Neural Networks (GNNs) are generalised deep neural networks for graph-structured data.
GNNs mainly have two categories, spectral-based GNNs and spatial-based GNNs.
Spectral GNNsattempt to use eigen-decomposition to obtain the spectral-based representation of graphs, whereasspatial GNNs focus on using spatial neighbours of nodes for message passing.
Extending spectralbased methods to the spatial domain, GCN [16] utilises first-order Chebyshev polynomial filters toapproximate spectral-based graph convolution.
Taking the weight of spatial neighbours in consideration, GAT [17], improves GCN by introducing the attention module in message passing.
To decouplemessage passing from neural networks, SGC [19] simplifies GCN by removing the non-linearity andweight matrices in graph convolution layers.
However, these studies cannot handle datasets withlimited or no labels.
Graph contrastive learning has been recently exploited to address this issue.
Graph Contrastive Learning (GCL) aims to alleviate the reliance on labelling information inmodel training based on the concept of mutual information (MI).
Specifically, GCL approachesmaximise MI between instances with similar semantic information, and minimise the MI betweendissimilar instances.
For example, DGI [7] builds contrastiveness between node embeddings and asummary vector (i.e., a graph level embedding obtained by averaging all node embeddings) with a
JSD estimator.
To improve DGI, MVGRL [3] and GMI [8] extends the idea of DGI by introducingmulti-view contrastiveness with diffusion augmentation, and focusing on a local scope with thefirst-order neighbourhood, respectively.
Adopting InfoNCE loss, GRACE [9] applies augmentationtechniques to create two augmented views and inject contrastiveness between them.
Though these
GCL methods have successfully outperformed some supervised baselines in benchmark datasets,these methods suffer from significant limitations, including time-consuming loss computation, a largenumber of training epochs and poor scalability.
For example, InfoNCE and JSD estimator require
O ( ND ) and O ( D ) in loss computation of a node, respectively.
In contrast, our method GGD reducesthe time complexity of loss computation for a node to only O (1) and converge fast.
Scalable GNNs.
Efficiency is a bottleneck for most existing GNNs to handle large graphs.
Toaddress this challenge, there are mainly three categories of approaches: layer-wise sampling (e.g.,
GraphSage [20]), graph sampling methods such as Cluster-GCN [21] and GraphSAINT [22], andlinear models, e.g., SGC [19] and PPRGo [23].
GraphSage [20] introduces a neighbour-samplingapproach, which creates fixed-size subgraphs for each node.
Underpinned by graph sampling,
Cluster-GCN [21] decomposes a large-scale graph into multiple subgraphs based on clustering, while
GraphSAINT [22] utilises light-weight graph samplers along with a normalisation technique forbiases elimination in mini-batches.
Linear models, SGC [19] and PPRGo [23], decouple graphconvolution from embedding transformation (i.e., matrix multiplication with weight matrices), andleverage Personalised PageRank to encode multi-hop neighbourhood, respectively.
However, all thesemethods only focus on supervised learning on graphs.
For unsupervised/self-supervised learningsettings where no labelled supervision signal is available, these frameworks are not applicable.
Theclosest works to ours to handle large scale graph datasets under self-supervised settings are BGRL [12]and GBT [11].
They try to reduce the time complexity of the contrastive losses by removing negativesamples.
However, BGRL [12] still requires O ( D ) to calculate the loss for a node, while GBT [11]requires O ( ND ) for a embedding dimension.
In comparison, GGD only requires O (1) .
Experiments
each dataset is in bold .
Comp and Photo refer to
Amazon Computers and Amazon Photos.
[Table]
We evaluate the effectiveness of our model using eight benchmark datasets of different sizes.
These datasets include five small- and mediumscale datasets: Cora, CiteSeer, PubMed [24],
Amazon Computers, and Amazon Photos [25],as well as large-scale datasets ogbn-arxiv, ogbnproducts and ogbn-papers100M.
Notably, ogbnpapers100M is the largest dataset provided by
Open Graph Benchmark[26] for node propertyprediction tasks.
It has over 110 million nodesand 1 billion edges.
The statistics of thesedatasets are summarised in Appendix A.7.
To ensure reproducibility, the detailed experiment settings and computing infrastructure are summarised in Appendix A.8.
5.1 Evaluating
on Small
- and Medium-
scale
Datasets
[Table]
We compare GGD with ten baselines including four supervised GNNs (i.e., GCN [16],
GAT [17], SGC [19], and CG3 [27]) andsix GCL methods (i.e., DGI [7], GMI [8],
MVGRL [3], GRACE [9], BGRL [12] and
GBT [11]) on five small- and medium scalebenchmark datasets.
In the experiment, wefollow the same data splits as [28] for Cora,
CiteSeer and PubMed.
For Amazon Computers and Photos, we adopt the semi-supervisedexperiment setting, which includes 30 randomlyselected nodes per class in the training and validation set.
The remaining nodes are used as thetest set.
The model performance is measured using the averaged classification accuracy with fiveresults along with standard deviations and reported in Table 5.
[Table]
Accuracy.
From Table 5, we can observe that
GGD generally outperforms all baselines inall datasets.
The only exception is on CiteSeer dataset, where the semi-supervised method,
CG3[27], slightly outperforms GGD , which stillprovides the 2nd best performance.
In this experiment, we use the officially released code of
GraphCL [15], BGRL [12] and GBT [11] to reproduce the result, while the other results aresourced from previous studies [27, 1].
Efficiency and Memory Consumption.
GGD is substantially more efficient than other self-supervisedbaselines in time and memory consumption as shown in Table 6 and Table 7.
Remarkably, GGD is19.2 times faster in Amazon Photos for training time per epoch, and consumes 64.5% less memoryin
Amazon Computers for memory consumption than the most efficient baseline (i.e., GBT [11]).
The dramatic boost of time and memory efficiency of GGD is contributed to the exclusion of similaritycomputation, which enables model training without multiplication of node embeddings.
5.2 Evaluating on Large-scale datasets
To evaluate the scalability of GGD , we choose three large-scale datasets from Open Graph Benchmark [26], which are ogbn-arxiv, ogbn-products, and ogbn-papers100M. ogbn-papers100M is themost challenging large-scale graph available in Open Graph Benchmark for node property predictionwith over 1 billion edges and 110 million nodes.
Extending to extremely large graphs (i.e., ogbnproducts and ogbn-papers100M), we adopt a Neighbourhood Sampling strategy, which is describedin Appendix A.8.
‘Total’ is total training time (Number of epochs memory
[Table]
ogbn-arxiv & ogbn-products.
For ogbnarxiv, we compare GGD against four selfsupervised baselines (i.e., DGI [7], GRACE [9],
BGRL [12]and GBT [11]), whereas BGRL [12]and GBT [11] are selected to be compared forogbn-products.
In addition, we include the performance of MLP, Node2vec [29], and supervised GCN [16] sourced from [26] in Table 8and Table 9.
For memory and training timecomparison, we only compare GGD with the twomost efficient baselines (i.e., BGRL and GBTaccording to Tables 6 and 7).
In ogbn-arxiv, we reproduce BGRL [12] and found it fails to process ogbn-arxiv in full batch.
Thus, we only compare
GGD and GBT in this dataset, which can successfully train in full-graph processing mode.
[Table]
From Table 8 and Table 9, we can see GGD remarkably achieves the state-of-the-art performance using only one epoch to train.
As a result, GGD is 10,783 times faster than the mostefficient baseline, i.e., GBT [11], on total training time to reach the desirable performance inogbn-arxiv.
Please be noted that the number ofepoches in our experiment is consistent with theoptimal choice of this hyperparameter specifiedin GBT [11].
For ogbn-products, we are 381 × faster than GBT [11] on total training time.
Notably,our performance is significantly higher than GCL baselines using 100 epochs (i.e., 6% and 5.2%improvement on GBT [11] in validation and test set) with only one epoch training in this dataset.
Inaddition, we compare the convergence speed among GGD , BGRL [12] and GBT [11] on ogbn-arxivand ogbn-products, which are shown in Figure 4.
For ogbn-arxiv, BGRL [11] is running using batchedprocessing with neighbour sampling.
This figure shows the preeminence of our GGD in convergencespeed as GGD can be well-trained with only one epoch (i.e., reaching the peak model performance inthe first epoch and staying stable with increased epochs).
In contrast, the other two baselines requirecomparatively much more epochs to gradually improve their performance.
Compared with GCLbaselines, GGD achieves much faster convergence via Group Discrimination.
We conjecture this isbecause GD-based method focuses on the general edge distribution of graphs instead of node-specificinformation.
Inversely, GCL methods can suffer from convergence inefficiency as they may be easilydistracted from too-detailed node-specific information during training.
[Figure]
ogbn-papers100M. We further compare GGDwith BGRL [12] and GBT [11] on ogbnpapers100M, the largest OGB dataset with billion scale edges.
Other self-supervised learningalgorithms such as DGI [7] and GMI [8] fail toscale to such a large graph with a reasonablebatch size (i.e., 256).
We only report the performance of each algorithm after a single epochof training in Table 10 due to the extreme scaleof the dataset and the limitation of our available resources.
From the table, we can observethat GGD outperforms the two GCL counterparts,
BGRL [12] and GBT [11] in both accuracy and efficiency.
Specifically, GGD achieves 60.2 in accuracywhile BGRL and GBT reach 59.3 and 58.9 in test set, respectively.
With only one epoch, these twoalgorithms may not be well trained.
However, training each epoch of these two requires over 1 dayand if we would like to train them for 100 epochs, then we will need 100+ GPU days, which isprohibitively impractical for general practitioners.
In contrast, GGD can be trained in about 9 hours toachieve a good result for this dataset, which is more appealing in practice.
Future Work
[Table]
In this paper, we have introduced a new selfsupervised GRL paradigm: Group Discrimination, which achieves the same level of performance as GCL methods with much less resourceconsumption (i.e., training time and memory).
Some limitations of this work are we still havenot explored some questions for GD.
For example, can we extend the current binary Group Discrimination scheme (i.e., classifying nodes generated with different topology) to discrimination amongmultiple groups?
Are there any other corruption technique to create a more difficult negative groupfor discrimination?
More importantly, with the extremely efficient property, GD has the potential tobe deployed to various real-world applications, e.g., recommendation systems, which have limitedlabelling information and desire fast computation with limited resources.
Checklist
For all authors...(
a) Do the main claims made in the abstract and introduction accurately reflect the paper’scontributions and scope?
[Yes] See Abstract and section 1.
[Table]
(b) Did you describe the limitations of your work?
[Yes] See section
[Table]
(c) Did you discuss any potential negative societal impacts of your work?
[N/A]
[Table]
(d) Have you read the ethics review guidelines and ensured that your paper conforms tothem?
[Yes]
[Table]
If you are including theoretical results...
[Table]
(a) Did you state the full set of assumptions of all theoretical results?
[Yes] Please refer tosection 2 and Appendix A. [Table]
(b) Did you include complete proofs of all theoretical results?
[Yes] Please refer to section2 and Appendix A.13.
If you ran experiments...
[Table] (a) Did you include the code, data, and instructions needed to reproduce the main ex- [Table]
perimental results (either in the supplemental material or as a URL)?
[Yes] See Appendix A.8 and the source code in supplemental material. [Table]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See section 5 and Appendix A.8.(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
[Table] (d) Did you include the total amount of compute and the type of resources used (e.g., typeof GPUs, internal cluster, or cloud provider)?
[Yes] See section 5 and Appendix A.8. [Table]
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
[Table]
(a) If your work uses existing assets, did you cite the creators?
[Yes] See section 5 and [Table] Appendix A.8. [Table]
(b) Did you mention the license of the assets?
[Yes] See section 5 and Appendix A.8.
[Table]
(c) Did you include any new assets
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[N/A]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[N/A]
A Appendix A
A.1 Proof of Theorem
Proof.
To prove Theorem 1, given a graph G =
( X , A ) , where X ∈ R N × D and a GNN encoder g () ,for simplicity, we consider g () as a one-layer GCN and conduct the following normalisation to X tomake its value to be within the range [0 , 1] :
[Equation]
where z ij ∈ Z is the normalised x ij ∈ X , i denotes the row-entry of the matrix, j indicates thecolumn-entry of the matrix, min () is the function to get the minimum value, and max () is thefunction for getting the maximum value.
Then, we input Z to g whose weight matrix W is initialised with Xavier initialisation:
[Equation]
(6) where H is the output embedding, σ is non-linear activation function, ˆ A
=
A + I , ˆ D is the degreematrix for ˆ A , W is the learnable weight matrix.
As multiplying the normalised adjacency matrix willnot change the output data range, the output of ˆ D 12 ˆ A
ˆ D 12 Z is still within the range [0 , 1] .
Then, as theelement in Xavier initialised W is in between − 1 √ D and 1 √ D , we can easily derive that the output datarange of the matrix multiplication of ˆ D 12 ˆ A ˆ D 12 Z and W still stay in the same range [ − 1 √ D , 1 √ D ] as W .
After that, we need to apply σ to the multiplication output.
Here, we analyse four commonly adoptednon-linear activation functions: Sigmoid, ReLu, LeakyReLu and PReLu.
We first consider the
Sigmoid function σ ( x ) = 1 1+ e − x , as it is a monotonic increasing function, we can derive that the datarange of H ∈ [
[Equation]
[Equation] D ] . Then, we apply the Sigmoid function S () again as DGI[7] did to
H and obtain:
[Equation]
Similarily, as the Sigmoid function is a monotonic increasing function, we can easily derive that thedata range for S ( H ) is in [
[Equation]
Then, when D → ∞ , for the upper bound
e −
[Equation]
− 1 √ D +
[Equation]
, we can obtain:
[Equation]
Also, for the upper bound
[Equation]
, when D → ∞
[Equation]
we can obtain:
[Equation]
Finally, we can easily observe that lim D →∞ ( e − e 1 √ D +1 + ) =
lim D →∞ ( e − e − 1 √ D +1 + ) ≈ 0 .
62and thus prove Theorem 1 when σ is Sigmoid function.
For the other three non-linear activationfunctions ReLu σ ( x ) = max (0 , x ) , Leaky ReLu σ ( x ) = max (0 .
01 ∗ x, x ) and PReLu σ ( x ) =max (0 , x ) + a ∗ min (0 , x ) , where a ≤ 1 is a learnable parameter, we can also derive the data rangeof these functions for H are [0 , 1 √ D ] , [ − 0 .
01 √ D , 1 √ D ] and [ − a √ D , 1 √ D ] , respectively.
Here, we can seethese functions share the same upper bound 1 √ D .
The only difference for their lower bound is thecoefficient c ∈ [0 , 1] for − 1 √ D , i.e., 0 for ReLu, 0.01 for Leaky ReLu and a for PReLu.
By inputting Hwith these activation functions to the Sigmoid function, we can derive the output data range of thefunction is S ( H ) ∈ [ 1+ e
[Equation]
√ D , 1+ e − 1 √ D ] .
For the lower bound 1+
[Equation]
D , we can obtain:
[Equation]
For the lower bound 1+ e − 1 √ D , we can obtain:
[Equation]
Finally, we can easily observe that lim D →∞
[Equation]
and thus prove Theorem 1when σ is ReLu, LeakyReLu or PReLu.
A.2 Exploring Group Discrimination
[Figure]
Firstly, we explore the corruption techniqueused in DGI [7] and MVGRL [3], which isshown in Figure 5.
These two studies corrupt the topology of a given graph G by shuffling the feature matrix X .
This is becauseby changing the node order of X , the neighbouring structure of G is completely changed,e.g., neighbours of node a become node bneighbours.
With the corruption technique, negative samples in the negative group are generated with incorrectedges.
Thus, by discriminating the positive group (i.e., nodes generated with ground truth edges) andthe negative group, we conjecture the model can distil valuable signals by learning how to identifynodes generated with correct topology and output effective node embeddings.
A.2.1 Theoretical Analysis of Group Discrimination
In this section, we first present the theoretical analysis and then provide an intuitive explanation forgroup discrimination.
Theorem 2 Given a graph G , a corrupted graph ˜ G , and a encoding network f θ () , we consider thedistribution of positive embeddings f θ ( G ) as P pos and negative embeddings f θ (˜ G ) as P neg .
Byoptimising the group discrimination loss, P pos and P neg tends to separated.
Proof.
The following proof is inspired by the theoretical proof in GAN [30]. During training, we canrewrite the group discrimination loss in Equation 3 into the following form:
[Equation]
where agg () is the aggregation function to turn z into the 1 × 1 prediction, P pos are the distributionof positive embeddings, P neg are the distribution of negative embeddings.
As our objective hereis to maximise L , and P pos ( z ) > 0; P neg ( z ) > 0 , we can obtain the optimal solution for agg ( z )is
P pos ( z ) P pos ( z )+ P neg ( z ) .
This is because for any ( a, b ) ∈ R 2 { 0 , 0 } , the maximum of a function y =alog ( x )+ blog (1 − x ) is achieved at aa + b [30].
By replacing agg ( z ) with P pos ( z ) P pos ( z )+ P neg ( z ) in Equation12, we can obtain: [Equation] [Equation]
From the equation above, we can see it looks similar to the Jensen-Shannon divergence between thedistribution of positive embeddings and negative embeddings: [Equation] Thus, we can rewrite Equation 13 as:
[Equation] where we can see maximising L is the same as [Figure]
maximising JS ( P pos ∥ P neg ) . Thus, by opti- [Figure] mising L , P pos and P neg tend to be separated.
[Figure] Based on the proof above, we can see group discrimination intuitively is learning to avoid making ‘mistakes’ (i.e., bias the encoder towardsavoiding mistaken samples) as shown in Figure6. This is because optimising the group discrim- [Figure]
ination loss can separate P pos and P neg . As [Figure] P optimal is disjoint with P neg , if the generated
[Figure]
embeddings can avoid being similar to out-of-
[Figure]
distribution samples, i.e., negative samples, itcan be more close to P optimal .
Therefore, thetrained model can improve the quality of generated node embeddings for node samples.
To explore the effect of other aggregation functions, we replace the summation function in Equation 1 with other aggregation methods, includingmean-, minimum-, maximum- pooling and linear aggregation.
We report the experiment results (i.e.,averaged accuracy on five runs) in Table 11.
The table shows that replacing the summation with otheraggregation methods still works, while summation and linear aggregation achieve comparativelybetter performance.
[Figure]
[Table]
Extending the architecture of DGI, MVGRL resortsto multi-view contrastiveness via additional augmentation.
Specifically, as shown in Figure 7, it first usesthe diffusion augmentation to create G aug .
Then, itcorrupts G and G aug to generate negative samples ˜
Gand ˜ G aug .
To build contrastiveness, MVGRL alsogenerates two summary vectors S aug and S by averaging all embeddings in G aug and G , respectively.
Based on the design of MVGRL, the model trainingis driven by mutual information maximisation between an anchor node embedding and its correspond- [Table] ing augmented summary vector.
However, MVGRL has the same technical error in their official JSD-based implementation as DGI, which makes it also becomes a group-discrimination-basedapproach.
Similar to Equation 3 of DGI, the proposed loss in MVGRL can also be rewritten as a binary crossentropy loss: [Equation]
[Equation]
here the number of nodes is increased to 4 N as we include nodes in G aug and ˜
G aug as data samples.
The indicator y i for G and G aug are 1, while ˜ G and ˜ G aug are considered as negative samples (i.e., theindicator y
i for them are 0).
To explore why MVGRL can achieve a better performance than DGI,we replace the original MVGRL loss with Equation 16 and conduct ablation study of MV GRL bceby removing different set of data samples in Equation 16, and report the experiment result in Table12.
From the table, the performance of MV GRL bce is on par with MV GRL , which reconfirms theeffectiveness of using the BCE loss.
Also, we can observe that including G aug and ˜ G aug is the key of
MVGRL surpassing DGI.
With G aug and ˜ G aug , the model performance of MV GRL bce w/o G aug and˜ G aug is improved from 82.2 to 83.1.
We conjecture this is because, with the diffusion augmentation,
MVGRL is trained with the additional global information provided by the diffused view G aug .
However, the diffusion augmentation involves expensive matrix inversion computation and significantlydensifies the given graph, which requires much more memory and time to store and process than theoriginal view.
This can hinder the model from extending to large-scale datasets [18].
A.5 Complexity Analysis
The time complexity of our method consists of two components: the siamese GNN and the losscomputation .
Existing self-supervised baselines share similar time complexity for the first component.
In GGD , Given a graph G = ( X ∈ R N × D , A ∈ R N × N ) in the sparse format, taking a L -layer
GCN [16] encoder as an example, the time complexity of it is O ( LMD + LND 2 ) .
As we need toprocess both the augmented graph ˆ G and the corrupted graph ˜ G , GGD requires the encoder computationtwice.
Then, the projector network (i.e., MLP) with K linear layers will be applied to the encoderoutput, which takes O ( ND 2 ) for each layer in computation.
Here D is a parameter defining thehidden size.
Before group discrimination, we aggregate the generated embedding with aggregationtechniques.
Here we take the simple summation, which consumes O ( ND ) as example.
For the loss computation, we use the BCE loss, i.e., Equation 3, to category summarised nodeembeddings, i.e., scalars.
The time complexity of this final step is O (2 N ) (i.e., processing all datasamples from the positive and negative groups).
Ignoring the computation cost of the augmentation,the overall time complexity of GGD for computing a graph G is O (2( LMD + LND 2 + KND 2 +
ND + N ))
→ O ( LMD + LND 2 + KND 2 + ND + N ) , where we can see the time complexity ismainly contributed by the siamese GNN.
More importantly, we compress the self-supervised learningloss computation to O ( N ) for a whole graph.
A.6 Graph Power
To show the easiness of graph power computation, we conduct an experiment to evaluate the timeconsumption for graph power computation on eight datasets, whose statistics are shown in Appendix
A.7.
Specifically, we set the hidden size of H θ to 256, and n is fixed to 10 for all datasets.
Theexperiment results are shown as below:
[Table]
This table shows that the computation of graph power is very trivial on small and medium sizegraphs, e.g., ogbn-arxiv, which has million of edges, consuming only 0.22 seconds.
Extending to anextremely large graph, ogbn-papers100M, which has over 1 billion edges and 11 million nodes, thecomputation only requires 209 seconds (i.e., around three minutes), which is acceptable consideringthe sheer size of the dataset.
A.7 Dataset Statistics
The following table presents the statistics of eight benchmark datasets including five small to medium-scale datasets and three large-scale datasets from OGB Graph Benchmark[26].
[Table]
10 Amazon Photo 7, 119,  8 ogbn-arxiv 169, 1,166,  40 ogbn-products 2,449, 61,859,  47 ogbn-papers-100M 111,059,956 1,615,685,
A.8 Experiment Settings & Computing Infrastructure
Extending to Extremely Large Datasets.
Extending to extremely large graphs (i.e., ogbn-productsand ogbn-papers100M), we adopt a simple Neighbourhood Sampling strategy introduced in GraphSage [20] to decouple model training from the sheer size of graphs.
Specifically, we create a fixedsize subgraph for each node, which is created by sampling a predefined number of neighbours in eachconvolution layer for sampled nodes.
The same approach is employed in the testing phase to obtainfinal embeddings.
General Parameter Settings.
In our experiment, we mainly tune four parameters for GGD ,which arelearning rate, hidden size, number of convolution layers in the GNN encoder, and number of linearlayers in the projector.
For simplicity, we set the n-th graph power for final embedding generationfixed to 5 for all datasets.
The parameter setting for each dataset is shown below:
[Table]
Large-scale Datasets Parameter Settings.
To decouple model training from the scale of graphs,we adopt the neighbouring sampling technique, which has three parameters: batch size, sample size,and number of hops to be sampled.
Batch size refers to the number of nodes to be processed inone parameter optimisation step.
Sample size means the number of nodes to be sampled in eachconvolution layer, and the number of hops determines the scope of the neighbourhood for sampling.
In GGD implementation, the batch size, sample size, and the number of hops are fixed to 2048, 12 and3, respectively.
Memory and Training Time Comparison.
As memory and training time are very sensitive to hyperparameters related to the structure of GNNs, including hidden size, number of convolution layers,and batch processing for large-scale datasets, e.g., batch size and number of neighbours sampled ineach layer.
Thus, in memory and training comparison, to be fair, we set all these parameters to be thesame for all baselines and GGD .
The specific parameter setting for each dataset is shown below:
[Table]
Computing Infrastructure.
For experiments in section 3, 5 and 6.1, they are conducted using
Nvidia GRID T4 (16GB memory) and Intel Xeon Platinum 8260 with 8 core.
For experiments onlarge-scale datasets (i.e.,ogbn-arxiv, ogbn-products and ogbn-papers100M), we use NVIDIA A40(48GB memory) and Intel Xeon Gold 5320 with 13 cores.
A.9 Algorithm
We have summarised the overall procedure of GGD in Algorithm 1 as follows:
Specifically, we first conduct augmentation on G to obtain ˆ G .
Then, ˆ G is corrupted to generate thecorrupted graph ˜ G .
In the encoding phase, we feed ˆ G and ˜ G to GNN encoder g θ and projector f θ togenerate embeddings ˆ H θ for positive samples and ˜ H θ for negative samples.
After that, we obtainthe binary classification result by aggregating the concatenation of ˆ H θ and ˜ H θ .
The result is a 2 Ndimension vector, which can be used for calculating the loss with a BCE loss.
Finally, based on thecalculated loss, trainable parameters in g θ and f θ can be updated.
This training process will continueiteratively until we reach the predefined number of epochs E .
When the training process is completed,we freeze the g θ and f θ and feed G to these two encoding components to obtain the local embedding,
H θ .
Then, we obtain global embedding H globalθ with a global information injection operation.
Bysumming H θ and H globalθ , we can get the final embeddings H .
A.10 Ablation Study
In this section, we conduct an ablation study to evaluate the effectiveness of different components in GGD .
Specifically, we evaluate three variants of GGD , including GGD w/o aug , GGD w/o proj ,
GGD w/o power , which represent GGD without augmentation, the projector and the global informationinjection process respectively.
The experiment results on five small to medium size datasets arepresented in Table 17.
From the table, we can see without any mentioned component, the performanceof GGD degrades, which validates the effectiveness of these components.
It is worth noting that evenwithout the global information injection in the inference phase, GGD w/o power still achieves thehighest performance in 4 out of 5 datasets compared with six self-supervised baselines.
This indicatesthat even without global information injection, GGD is still effective.
[Table]
[Table]