Rethinking and Scaling Up Graph Contrastive Learning: An Extremely Efficient Approach with Group Discrimination
Abstract
Graph contrastive learning (GCL) alleviates the heavy reliance on label information for graph representation learning (GRL) via self-supervised learning schemes.
The core idea is to learn by maximising mutual information for similar instances, which requires similarity computation between two node instances.
However,
GCL is inefficient in both time and memory consumption.
In addition, GCL normally requires a large number of training epochs to be well-trained on largescale datasets.
Inspired by an observation of a technical defect (i.e., inappropriate usage of Sigmoid function) commonly used in two representative GCL works,
DGI and MVGRL, we revisit GCL and introduce a new learning paradigm for self-supervised graph representation learning, namely, Group Discrimination (GD), and propose a novel GD-based method called Graph Group Discrimination ( GGD ).
Instead of similarity computation, GGD directly discriminates two groups of node samples with a very simple binary cross-entropy loss.
In addition, GGD requires much fewer training epochs to obtain competitive performance compared with
GCL methods on large-scale datasets.
These two advantages endow GGD with very efficient property.
Extensive experiments show that GGD outperforms state-of-theart self-supervised methods on eight datasets.
In particular, GGD can be trained in 0.18 seconds (6.44 seconds including data preprocessing) on ogbn-arxiv, which is orders of magnitude (10,000+) faster than GCL baselines while consuming much less memory.
Trained with 9 hours on ogbn-papers100M with billion edges,
GGD outperforms its GCL counterparts in both accuracy and efficiency.
Introduction
Graph Neural Networks (GNNs) have been widely-adopted in learning representations for graphstructured data.
By utilising message-passing over the topology of a graph, GNNs can learn effective low-dimensional node embeddings, which can be used for a variety of downstream tasks such as node classification [1].
GNNs have been further applied in diverse domains, e.g., federated learning [2, 3], trustworthy systems [4, 5], dynamic graphs [6, 7] and anomaly detection [8, 9].
However, many GNNs adopt a supervised learning manner to train models with label information, which is expensive and labour-intensive to collect in real-world.
To address this issue, a few studies (e.g., DGI [10], MVGRL [11], GMI [12], and GRACE [13]) borrow the idea of contrastive learning from computer vision (CV), and introduce graph contrastive learning (GCL) methods for selfsupervised GRL.
The core idea of these methods is to maximise the mutual information (MI) between an anchor node and its positive counterparts, sharing similar semantic information while doing the opposite for negative counterparts as shown in Figure 1(a).
Nonetheless, such a scheme relies on similarity calculation in contrastive loss computation.
Additionally, GCL normally requires a large number of training epochs to be well-trained on large-scale datasets.
Thus, when the size of the dataset is large, these methods require a significant amount of time and resources to be well-trained.
[Figure]
Though a few GCL works attempt to improve graph contrastive learning with specially designed schemes, e.g.,
BGRL [15] and GBT [14], they are still inefficient and require high time consumption for model training.
Inspired by BYOL [16], BGRL [15] adopts a bootstrapping scheme and remove negative node pairs.
It only contrasts a node from the online network (i.e., updated with gradient) to its corresponding embedding from the target network (i.e., updated momentumly with stop gradient).
Based on Barlow-Twins [17],
GBT [14] borrows the idea of redundancy-reduction principle and utilises a cross-correlation-based loss to build contrastiveness between embedding dimensions.
including preprocessing), which equals to ( Pre+Epo × Tr ) and total training time, which is ( Epo ×
Tr )
‘Imp(E)’ and ‘Imp many times GGD improve
[Table]
To boost training efficiency of self-supervised
GRL, inspired by an observation of a technical defect (i.e., inappropriate application of Sigmoid function) in two representative GCL studies, we introduce a novel learning paradigm, namely,
Group Discrimination (GD).
Instead of similarity computation, GD directly discriminates a group of positive nodes from a group of negative nodes, as shown in Figure 1(b).
Specifically, GD defines node samples generated with original graph as the positive group, while node samples obtained with corrupted topology are regarded as the negative group.
Then, GD trains the model by classifying these node samples into the correct group with a very simple binary cross-entropy loss.
By doing so, the model can extract valuable self-supervised signals from learning the edge distribution of a graph.
Compared with GCL, GD enjoys numerous merits including extremely fast training, fast convergence (e.g., 1 epoch to be well-trained on large-scale datasets), and high scalability while achieving SOTA performance with existing GCL approaches.
Using GD as backbone, we design a new self-supervised GRL model with the Siamese structure called Graph Group Discrimination (GGD).
Firstly, we can optionally augment a given graph with augmentation techniques, e.g., feature and edge dropout.
Then, the augmented graph is fed into a
GNN encoder and a projector to obtain embeddings for the positive group.
After that, the augmented feature is corrupted with node shuffling (i.e., disarranging the order of nodes in the feature matrix) to disrupt the topology of a graph and input to the same network for obtaining embeddings of the opposing group.
Finally, the model is trained by discriminating these two groups of node samples.
The contributions of this paper are three-fold: 1) We re-examine existing GCL approaches (e.g.,
DGI [10] and MVGRL [11]), and we introduce a novel and efficient self-supervised GRL paradigm, namely, Group Discrimination (GD). 2)
Based on GD, we propose a new self-supervised GRL model,
GGD, which is fast in training and convergence, and possess high scalability.
We conduct extensive experiments on eight datasets, including an extremely large dataset, ogbn-papers100M with billion edges.
The experiment results show that our proposed method reaches state-of-the-art performance while consuming much less time and memory than baselines, e.g., 10783 × faster than the most efficient GCL baseline with its best selected epochs number [14], as shown in Table 1.
Rethinking Representative GCL Methods
[Figure]
In this section, we analyse a technical defect observed in two representative GCL methods,
DGI [10] and MVGRL [11].
Based on the technical defect, we show that mutual information maximisation behind these two approaches is not the contributed factor to contrastive learning, but a new paradigm, group discrimination.
Finally, from the analysis, we provide the definition of this new concept.
2.1 Rethinking GCL Methods
DGI [10] is the first work introducing contrastive learning into GRL.
However, due to a technical defect observed in their official opensource code, we found it is essentially not working as the authors thought (i.e., learning via MI interaction).
Constant Summary Vector.
As shown in Figure 2, the original idea of DGI is to maximise the MI (i.e., the red line) between a node a and the summary vector s , which is obtained by averaging all node embeddings in a graph G .
Also, to regularise the model training, DGI corrupts G by shuffling the node order of the input feature matrix to get ˜ G .
Then, generated embeddings of ˜ G serve as negative samples, which are pulled apart from the summary vector s via MI minimisation.
[Table]
Nonetheless, in the implementation of
DGI, a Sigmoid function is inappropriately applied on the summary vector generated from a GNN whose weight is initialised with Xavier initialisation.
As a result, elements in the summary vector are very close to the same value.
We have validated this finding on three datasets, Cora, CiteSeer and PubMed.
The experiment result is shown in Table 2, which shows that summary vectors in all datasets are approximately a constant vector
ϵ I , where ϵ is a scalar and I is an all-ones vector (i.e., ϵ =0.50 with ReLU/LReLU/PReLU and ϵ =0.62 with Sigmoid as non-linear activation in these datasets).
[Table]
To theoretically explain this phenomenon, we present the proposition below:
Proposition 1 Given G = { X ∈ R N × D , A ∈ R N × N } , and a GCN encoder g ( · ) initialised with Xavier initialisation, we can obtain its embedding H =
σ ( g ( G )) , where σ ( · ) is a non-linear activation function.
By applying the sigmoid function σ sig ( · ) to the summary vector s (i.e., the average row vector of H ), values in σ sig ( s ) approximately become 0.5 with ReLU/LReLU/PReLU or 0.62 with Sigmoid as non-linear activation of g ( · ) at the initialisation stage.
Based on this proposition, we can see these summary vectors can lose variance and become a constant vector at the initialisation stage.
Based on Table 2, we can see the constant in the summary vector remain unchanged, and the information loss still occurs even if the GNN encoder is trained.
Thus, we conjecture the training process won’t affect the constant value much in the summary vector of DGI.
The proof for the proposition is presented in Appendix A.1.
To evaluate the effect of ϵ to constant summary vector, we vary the scalar ϵ (from 0 to 1 increment by 0.2) to change the constant summary vector and report the model performance (i.e., averaged accuracy on five runs) in Table 3.
From this table, we can see, except for 0, the model performance is trivially affected by ϵ for constant summary vector.
When the summary vector is set to 0, the model performance plummets because node embeddings become all 0 when multiplying with such vector and the model converges to the trivial solution.
As the summary vector only has a trivial effect on model training, the hypothesis of
DGI [10] on learning via contrastiveness between anchor nodes and the summary instance does not hold, which raises a question to be investigated: What truly leads to the success of DGI?
Simplifying DGI.
To answer the question, we predigest the objective function proposed in DGI (i.e., maximising the MI between h i and the summary vector s ) by using an all-ones vector as the summary vector s (i.e., setting s =
ϵ I = I )
and simplifying the discriminator D ( · ) (i.e., removing the learnable weight matrix).
Then, we rewrite the objective function to the following form:
[Equation]
where · is the vector multiplication operation, N is the number of nodes in a graph, h i ∈ R 1 × D and ˜ h
i ∈ R 1 × D are the original and corrupted embedding for node i , sum ( · ) is the summation function, and D ( · ) is a discriminator for bilinear transformation, which can be formulated as follows:
[Equation]
[Table]
where W is a learnable weight matrix and σ sig ( · ) is the sigmoid function.
Specifically, as shown in
Equation 2, by removing the weight matrix W , h i is directly multiplied with s .
As s is a vector containing only one, the multiplication of h i and s is equivalent to summing h i itself directly.
From this form, we can see that the multiplication of h i and the summary vector only serves as an aggregation function (i.e., summation aggregation) to aggregate h i .
To explore the effect of other aggregation functions, we replace the summation function in Equation 1 with other aggregation methods such as mean-, minimum-, and maximum- pooling, and present the experiment result in Appendix A.3.
Based on Equation 1, we can rewrite it to a very simple binary cross entropy loss if we also include corrupted nodes as data samples and setting ˆ y i
=
agg ( h i ) , where agg ( · ) stands for aggregation:
[Equation]
where y i ∈ R 1 × 1 means the indicator for node i (i.e., if node i is corrupted, y i is 0, otherwise it is 1), and ˆ y i ∈ R 1 × 1 represents the prediction for a node sample i .
As we include corrupted nodes as data samples, the size of nodes to be processed is doubled to 2 N (i.e., the number of corrupted nodes is equal to the number of original nodes).
From the equation above, we can easily observe that what DGI truly does is discriminate between a group of nodes generated with correct topology and nodes generated with corrupted topology, as shown in Figure 1.
We name this self-supervised learning paradigm " Group Discrimination ".
To validate the effectiveness of this paradigm, we replace the original DGI loss with Equation 3, namely, DGI BCE and compare it with DGI on three datasets in terms of training time, memory efficiency and model performance as shown in Table 4.
Here, DGI BCE adopts the same parameter setting as DGI.
From this table, we can observe DGI BCE dramatically improves DGI in both memory and time efficiency while it slightly enhances the model performance of DGI.
This may be contributed to the removal of multiplication operations between node pairs, which eases the burden of computation and memory consumption.
Similar to DGI, the same technical defect is observed in MVGRL [11], which makes it become a
GD-based method.
Extended on DGI, MVGRL [11] incorporates diffusion augmentation to inject additional global information into model training, which enhances the model performance.
The detailed analysis for MVGRL is presented in Appendix A.4.
2.2 Definition of Group Discrimination
As mentioned above, Group Discrimination is a self-supervised GRL paradigm, which learns by discriminating different groups of node samples.
Specifically, the paradigm assigns different indicators to different groups of node samples.
For example, for binary group discrimination, one group is considered as the positive group with class 1 as its indicator, whereas the other group is the negative group, having its indicator assigned as 0.
Given a graph G , the positive group usually includes node samples generated with the original graph G or its augmented views (i.e., similar graph instances of G created by augmentation).
In contrast, the opposing group contains negative samples obtained by corrupting G , e.g., changing its topology structure.
Based on our theoretical analysis, group discrimination is learning to avoid making ‘mistakes’ (i.e., bias the encoder towards avoiding mistaken samples), thus improving the quality of generated embeddings.
The analysis and an intuitive explanation are presented in Section 6.1.1 and A.2.1.
Methodology
We first define unsupervised node representation learning and then present the architecture of GGD , which extends DGI BCE with additional augmentation, the projector and embedding reinforcement to reach better model performance.
Given a graph G with attributes X ∈ R N × D , where N is the number of nodes in G , and D is the number of dimensions of X , our aim is to train a GNN encoder without the reliance on labelling information.
With the trained encoder, taking G and X as input, it can output learned representations H ∈ R N × D ′ , where D ′ is the predefined hidden dimension.
H can then be used in many downstream tasks such as node classification.
[Figure]
on them to generate ˆ G and ˆ X .
Then, we corrupt ˆ X
and ˆ G to obtain ˜ X and ˜ G .
Taking ˆ X and ˆ G as input to the encoder and the projector, i.e., a multilayer perceptron, positive node samples can be obtained.
Similarly, ˜ X and ˜ G are fed to the same encoder and projector to generate negative samples.
The generated embeddings are aggregated to get predictions for the group discrimination task.
This process will be iteratively conducted until reaching the predefined training epochs.
3.1 Graph Group Discrimination
Based on the proposed self-supervised GRL paradigm, group discrimination, we have designed a novel method, namely GGD , to learn node representations using a siamese network structure and a
BCE loss.
The architecture of GGD is presented in Figure 3.
The framework mainly consists of four components: augmentation, corruption, a siamese GNN network, and group discrimination.
Augmentation.
With a given graph G and feature matrix X , optionally, we can augment it with augmentation techniques such as edge and feature dropout to create ˆ G and ˆ X .
In practice, we follow the augmentation proposed in GraphCL [18].
Specifically, edge dropout removes a predefined fraction of edges, while we use node dropout to mask a predefined proportion of feature dimension, i.e., assigning 0 to replace values in randomly selected dimensions.
This step is optional in implementation.
Notably, the motivation of using augmentation in our framework is distinct from contrastive learning methods.
In our study, augmentation is used to increase the difficulty of the self-supervised training tasks.
With augmentation, ˆ G and ˆ X change in every training iteration, which forces the model to lessen the dependence on the fixed pattern (i.e., unchanged edge and feature distribution) in a monotonous graph.
However, in contrastive learning, augmentation creates augmented views sharing similar semantic information for building contrastiveness.
Corruption.
ˆ G and ˆ X are then corrupted to build ˜ G and ˜ X for the generation of node embeddings in the negative group.
We adopt the same corruption technique used in DGI [10] and MVGRL [11] (as shown in Figure 5).
The corruption technique devastates the topology structure of ˆ G by randomly changing the order of nodes in ˆ X .
The corrupted ˜ X and ˜ G can be used for producing node representations with incorrect network connections.
The Siamese GNN.
We have designed a siamese GNN network to output node representations given a graph and its attribute.
The siamese GNN network is made up of two components, which are a
GNN encoder and a projector.
The backbone GNN encoder is replaceable with a variety of choices of GNNs, e.g., GCN [19] and GAT [20].
In our work, we adopt GCN as the backbone.
The projector is a multi-layer perceptron network, whose number of layers can be adjusted.
When generating node embeddings of the positive group, the Siamese network takes ˆ G and ˆ X as input.
Using the same encoder and projector, the Siamese network output the negative group with ˜ G and ˜ X .
These two groups of node embeddings are considered as a collection of data samples with a size of 2 N for discrimination.
Before conducting group discrimination, in the “aggregation” phase, all data samples are aggregated with the same aggregation technique, e.g., sum-, mean-, and linear aggregation.
Group Discrimination.
In the group discrimination process, we adopt a very simple binary cross entropy (BCE) loss to discriminate two groups of node samples as shown in Equation 3.
In our implementation, y i is 0 and 1 for node embeddings in negative and positive groups.
During model training, the model is optimised by categorising node embeddings in the collection of data samples into their corresponding class correctly.
The loss is computed by comparing the prediction of a node i , i.e., a scalar, with its indicator y i .
With the ease of BCE loss computation, the training process of
GGD is very fast and memory efficient.
3.2 Model Inference
During training, the model is optimised via loss minimisation with Equation 3.
The time complexity analysis of GGD is provided in Appendix A.5.
In the inference phase, we freeze the trained GNN encoder g θ and obtain node embeddings H θ with the input G .
Inspired by MVGRL [11], which strengthens the output embeddings by including additional global information, we adopt a conceptually similar embedding reinforcement approach.
Specifically, they obtain the final embeddings by summing up embeddings from two views: the original view comprising local information and the diffused view with global information.
This operation reinforces the final embeddings and leads to model performance improvement.
Nonetheless, graph diffusion impairs the scalability of a model [21] and hence cannot be directly applied in our embedding generation process.
To avoid the diffusion computation, we have come up with a workaround in the virtue of the power of a graph to extract global information.
The power of a graph can extend the message passing scope of H θ to n-hop neighbourhood, which encodes global information from distant neighbours.
It can be formulated as follows:
[Equation]
where H globalθ is the global embedding, and A is the adjacency matrix of the graph G .
It is notable that this operation can be easily decomposed with the associative property of matrix multiplication and is easy to compute.
To show the easiness of such computation, we conduct an experiment showing its time consumption on various datasets in Appendix A.6.
Finally, the final embedding can be achieved by H
=
H globalθ + H θ , which can be used for downstream tasks.
In our experiment, we conduct node classification tasks.
Following the common practice of GCL methods [10, 13, 15, 12, 17], these tasks are performed by using the final embeddings H to train and test a simple logistic regression classifier.
Related Work
Graph Neural Networks (GNNs).
are generalised deep neural networks for graph-structured data.
GNNs mainly have two categories, spectral-based GNNs and spatial-based GNNs.
Spectral
GNNs attempt to use eigen-decomposition to obtain the spectral-based representation of graphs, whereas spatial GNNs focus on using spatial neighbours of nodes for message passing.
Extending spectral-based methods to the spatial domain, GCN [19] utilises first-order Chebyshev polynomial filters to approximate spectral-based graph convolution.
Taking the weight of spatial neighbours in consideration, GAT [20], improves GCN by introducing attention module in message passing.
To decouple message passing from neural networks, SGC [22] simplifies GCN by removing non-linearity and weight matrices in graph convolution layers.
However, these studies cannot handle datasets with limited or no labels.
Graph contrastive learning has been recently exploited to address this issue.
Graph Contrastive Learning (GCL).
aims to alleviate the reliance on labelling information in model training based on the concept of mutual information (MI).
Specifically, GCL approaches maximise MI between instances with similar semantic information, and minimise MI between dissimilar instances.
For example, DGI [10] builds contrastiveness between node embeddings and a summary vector (i.e., a graph level embedding obtained by averaging all node embeddings) with a JSD estimator.
To improve DGI, MVGRL [11] and GMI [12] extends the idea of DGI by introducing multi-view contrastiveness with diffusion augmentation, and focusing on a local scope with the first-order neighbourhood, respectively.
Adopting InfoNCE loss, GRACE [13] applies augmentation techniques to create two augmented views and inject contrastiveness between them.
Though these GCL methods have successfully outperformed some supervised baselines in benchmark datasets, these methods suffer from significant limitations, including time-consuming training, memory inefficiency, and poor scalability.
In contrast, GGD requires much less time in training and posses high scalability.
Scalable GNNs.
Efficiency is a bottleneck for most existing GNNs to handle large graphs.
To address this challenge, there are mainly three categories of approaches: layer-wise sampling (e.g.,
GraphSage [23]), graph sampling methods such as Cluster-GCN [24] and GraphSAINT [25], and linear models, e.g., SGC [22] and PPRGo [26].
GraphSage [23] introduces a neighbour-sampling approach, which creates fixed-size subgraphs for each node.
Underpinned by graph sampling,
Cluster-GCN [24] decomposes a large-scale graph into multiple subgraphs based on clustering, while
GraphSAINT [25] utilises light-weight graph samplers along with a normalisation technique for biases elimination in mini-batches.
Linear models, SGC [22] and PPRGo [26], decouple graph convolution from embedding transformation (i.e., matrix multiplication with weight matrices), and leverage Personalised PageRank to encode multi-hop neighbourhood, respectively.
However, all these methods only focus on supervised learning on graphs.
For unsupervised/self-supervised learning settings where no labelled supervision signal is available, these frameworks are not applicable.
The closest works to ours to handle large scale graph datasets under self-supervised settings are
BGRL [15] and GBT [14].
They try to improve the contrastive losses by removing negative samples.
However, BGRL [15] and GBT [14] still require much more time in training compared with GGD .
Experiments
[Table]
We evaluate the effectiveness of our model using eight benchmark datasets of different sizes.
These datasets include five small- and mediumscale datasets: Cora, CiteSeer, PubMed [27],
Amazon Computers
[Table]
and Amazon Photos [28], as well as large-scale datasets ogbn-arxiv, ogbnproducts and ogbn-papers100M. Notably, ogbnpapers100M is the largest dataset provided by
Open Graph Benchmark[29] for node property prediction tasks.
It has over 110 million nodes and 1 billion edges.
The statistics of these datasets are summarised in Appendix A.7.
To ensure reproducibility, the detailed experiment settings and computing infrastructure are summarised in Appendix A.8.
The source code is already open sourced 2 .
[Table]
We compare GGD with ten baselines including four supervised GNNs (i.e., GCN [19],
GAT [20], SGC [22], and CG3 [30]) and six GCL methods (i.e., DGI [10], GMI [12],
MVGRL [11], GRACE [13], BGRL [15] and
GBT [14]) on five small- and medium scale benchmark datasets.
In the experiment, we follow the same data splits as [31] for Cora, CiteSeer and PubMed.
For Amazon Computers and
Photos, we use a random split setting, which randomly allocates 10/10/80% of data to training/validation/test set, respectively.
The model performance is measured using the averaged classification accuracy with five results along with standard deviations and reported in Table 5.
[Table]
Accuracy.
From Table 5, we can observe that
GGD generally outperforms all baselines in all datasets.
The only exception is on CiteSeer dataset, where the semi-supervised method,
CG3[30], slightly outperforms GGD , which still provides the 2nd best performance.
In this experiment, we use the officially released code of
GraphCL [18], BGRL [15] and GBT [14] to reproduce the result, while the other results are sourced from previous studies [30, 1].
Efficiency and Memory Consumption.
GGD is substantially more efficient than other self-supervised baselines in time and memory consumption as shown in Table 6 and Table 7.
Remarkably, GGD is 19.2 times faster in Amazon Photos for training time per epoch, and consumes 64.5% less memory in Amazon Computers for memory consumption than the most efficient baseline (i
[Table]
e., GBT [14]).
The dramatic boost of time and memory efficiency of
GGD is contributed to the exclusion of similarity computation, which enables model training without multiplication of node embeddings.
5.2 Evaluating on Large-scale datasets
To evaluate the scalability of GGD
[Table]
we choose three large-scale datasets from Open Graph Benchmark [29], which are ogbn-arxiv, ogbn-products
[Table]
and ogbn-papers100M. ogbn-papers100M is the most challenging large-scale graph available in Open Graph Benchmark for node property prediction with over 1 billion edges and 110 million nodes.
Extending to extremely large graphs (i
[Table]
e., ogbnproducts and ogbn-papers100M), we adopt a Neighbourhood Sampling strategy, which is described in Appendix A.
[Table]
[Table]
ogbn-arxiv & ogbn-products.
For ogbn-arxiv, we compare GGD against four self-supervised baselines (i.e., DGI [10],
GRACE [13], BGRL [15]and GBT [14]), whereas BGRL [15] and GBT [14] are selected to be compared for ogbn-products.
In addition, we include the performance of MLP,
Node2vec [32], and supervised GCN [19] sourced from [29] in Table 8 and Table 9.
For memory and training time comparison, we only compare GGD with the two most efficient baselines (i.e., BGRL and GBT according to
Tables 6 and 7).
In ogbn-arxiv, we reproduce
BGRL [15] and found it fails to process ogbn-arxiv in full batch.
Thus, we only compare GGD and GBT in this dataset, which can successfully train in full-graph processing mode.
[Table]
From Table 8 and Table 9, we can see GGD remarkably achieves the state-of-the-art performance using only one epoch to train.
As a result,
GGD is 10,783 times faster than the most efficient baseline, i.e., GBT [14], on total training time to reach the desirable performance in ogbn-arxiv.
Please note that the number of epoches in our experiment is consistent with the optimal choice of this hyperparameter specified in GBT [14].
For ogbn-products, we are 381 × faster than GBT [14] on total training time.
Notably, our performance is significantly higher than GCL baselines using 100 epochs (i.e., 6% and 5.2% improvement on GBT [14] in validation and test set, respectively) with only one epoch training in this dataset.
In addition, we compare the convergence speed among
GGD , BGRL [15] and GBT [14] on ogbn-arxiv and ogbn-products, which are shown in Figure 4.
For ogbn-arxiv, BGRL [15] is running using batched processing with neighbour sampling.
This figure shows the preeminence of our GGD in convergence speed as GGD can be well-trained with only one epoch (i.e., reaching the peak model performance in the first epoch and staying stable with increased epochs).
In contrast, the other two baselines require comparatively much more epochs to gradually improve their performance.
Compared with GCL baselines, GGD achieves much faster convergence via Group Discrimination.
We conjecture this is because GD-based method focuses on the general edge distribution of graphs instead of node-specific information.
Inversely, GCL methods can suffer from convergence inefficiency as they may be easily distracted from too-detailed node-specific information during training.
[Figure]
ogbn-papers100M.
We further compare GGD with BGRL [15] and
GBT [14] on ogbn-papers100M, the largest OGB dataset with billion scale edges.
Other self-supervised learning algorithms such as DGI [10] and
GMI [12] fail to scale to such a large graph with a reasonable batch size (i.e., 256).
We only report the performance of each algorithm after a single epoch of training in Tabledue to the extreme scale of the dataset and the limitation of our available resources.
From the table, we can observe that GGD outperforms the two GCL counterparts, BGRL [15] and GBT [14] in both accuracy and efficiency.
Specifically, GGD achieves 60.2 in accuracy while
BGRL and GBT reach 59.3 and 58.9 in test set, respectively.
With only one epoch, these two algorithms may not be well trained.
However, training each epoch of these two requires over 1 day and if we would like to train them for 100 epochs, then we will need 100+ GPU days, which is prohibitively impractical for general practitioners.
In contrast, GGD can be trained in about 9 hours to achieve a good result for this dataset, which is more appealing in practice.
Explore Group Discrimination
In this section
[Table]
we explore the corruption technique in GGD and provide the theoretical analysis of group discrimination.
6.1 Exploring Corruption
[Table]
Firstly, we explore the corruption technique used in DGI [10] and MVGRL [11]
[Table]
which is shown in Figure 5.
These two studies corrupt the topology of a given graph G by shuffling the feature matrix X .
This is because by changing the node order of X , the neighbouring structure of G is completely changed, e
[Table]
g., neighbours of node a become node b neighbours.
With the corruption technique, negative samples in the negative group are generated with incorrect edges.
Thus, by discriminating the positive group (i.e., nodes generated with ground truth edges) and the negative group, we conjecture the model can distil valuable signals by learning how to identify nodes generated with correct topology and output effective node embeddings.
To provide explanation to this, we present the theoretical analysis of group discrimination in the following section.
6.1.1 Theoretical Analysis of Group Discrimination
Group discrimination is learning to avoid making ‘mistakes’ (i.e., bias the encoder towards avoiding mistaken samples).
To explain this point, we first present Theoremand then provide an intuitive explanation for group discrimination.
Theorem 1 Given a graph G , a corrupted graph ˜ G , and a encoding network g ( · ) , we consider the distribution of positive embeddings g ( G ) as
P pos and negative embeddings g (˜ G ) as P neg .
Optimising the group discrimination loss is equivalent to maximising the Jensen-Shannon divergence between P pos and P neg .
The proof for Theorem 1 is presented in Appendix A.2.
From the theorem above, we can see maximising the group discrimination loss L is the same as maximising JS ( P pos ∥ P neg ) , where
JS represents the Jenson-Shannon divergence.
Thus, by optimising the loss L , P pos and P neg tend to be separated.
As a result, group discrimination is intuitively learning to avoid making ‘mistakes’ (i.e., bias the encoder towards avoiding mistaken samples) as shown in Figure 6.
This is because by separating P pos and P neg , P pos can gradually become similar to P optimal , the optimal distribution for node embeddings.
As P optimal , is disjoint with P neg , if the generated embeddings can avoid being similar to out-of-distribution samples, i.e., negative samples, it can be ideally closer to P optimal .
Therefore, the trained model can improve the quality of generated node embeddings for node samples.
Future Work
Become is the optimal distribution for node embeddings, P pos is the distribution of positive samples, P neg is the distribution of negative samples, blue nodes represent negative samples, and red nodes are samples in the optimal distribution.
At the beginning, P pos is overlapped with P neg .
Then, P pos is gradually separated from P neg and ideally become closer to P optimal .
In this paper, we have introduced a new self-supervised
GRL paradigm: Group Discrimination, which achieves the same level of performance as GCL methods with much less resource consumption (i.e., training time and memory).
Some limitations of this work are we still have not explored some questions for
GD.
For example, can we extend the current binary Group
Discrimination scheme (i.e., classifying nodes generated with different topology) to discrimination among multiple groups?
Are there any other corruption technique to create a more difficult negative group for discrimination?
More importantly, with the extremely efficient property, GD has the potential to be deployed to various real-world applications, e.g., recommendation systems, which have limited labelling information and desire fast computation with limited resources.
Acknowledgments and Disclosure of Funding
This research was partially supported by an Australian Research Council (ARC)
Future Fellowship (FT210100097).
This work is supported in part by NSF under grants III-1763325, III-1909323, III- and
[-Lemma 
-]
[-The parameter setting for each dataset is shown below: [Table] 1e
 PubMed 1e
-]SaTC-
[-Amazon Computers 1e
-]