B OOTSTRAPPED R EPRESENTATION L EARNING ON G RAPHS
Shantanu Thakoor DeepMind thakoor@google.com Corentin Tallec DeepMind corentint@google.com
Mohammad Gheshlaghi Azar
DeepMind mazar@google.com
R´emi Munos DeepMind munos@google.com Petar Veliˇckovi´c DeepMind petarv@google.com
Michal Valko
DeepMind valkom@google.com
A BSTRACT
Current state-of-the-art self-supervised learning methods for graph neural networks are based on contrastive learning.
As such, they heavily depend on the construction of augmentations and negative examples.
Increasing the number of negative pairs improves performance, thereby requiring quadratic computation and memory cost to achieve peak performance.
Inspired by BYOL , a recently introduced method for self-supervised learning that does not require negative pairs, we present Bootstrapped Graph Latents, BGRL , a self-supervised graph representation method that gets rid of this potentially quadratic bottleneck.
BGRL outperforms or matches the previous unsupervised state-of-the-art results on several established benchmarks.
Moreover, it enables the effective usage of graph attentional (GAT) encoders, allowing us to further improve the state of the art, in particular achieving 70.49%
Micro-F1 on the PPI dataset using the linear evaluation protocol.
I NTRODUCTION
Self-supervised learning is a promising path towards eliminating the need for costly label information in representation learning on many domains, including images, video, speech and text.
This is especially relevant in the graph domain, where unsupervised data is abundant, but label information is scarce.
Most of the best performing self-supervised learning methods are contrastive (Hjelm et al., 2019;
Oord et al., 2018).
Speciﬁcally, contrastive methods build representations by pulling together representations of related objects and pushing apart representations of unrelated pairs.
They have displayed performance that matches or improves over equivalent methods trained with labeled data (Tian et al., 2020; Bachman et al., 2019;
Mitrovic et al., 2021; Caron et al., 2020; Xu et al., 2020).
Inspired by the success of contrastive methods in vision and elsewhere, contrastive learning methods were adapted to graphs (Veliˇckovi´c et al., 2019; Peng et al., 2020; Hassani & Khasahmadi, 2020;
Zhu et al., 2020b).
The ﬁrst such method, DGI (Veliˇckovi´c et al., 2019), is closely aligned with
Deep InfoMax (Hjelm et al., 2019).
More recently, GRACE (Zhu et al., 2020b) adapts the SimCLR (Chen et al., 2020a;b) method to graphs and achieves state-of-the-art performance.
GRACE learns node representations by creating two augmented versions of a graph, pulling together representations of the same node in the two graphs, and pushing apart representations of every other node.
Appendix A contains further discussion of self-supervised learning methods on graphs.
However, the practical efﬁciency of contrastive methods relies on the ability to compare each object to a large number of negative examples (He et al., 2020).
This is especially prohibitive for large graphs as this requires a worst-case time and space complexity quadratic in the number of nodes.
In general, relying on negative examples seems undesirable, particularly for graphs, where negative examples are difﬁcult to deﬁne in a principled way.
Introduced in the vision domain, BYOL (Grill et al., 2020) is a method competitive with the best contrastive approaches while avoiding the need for negative examples (Richemond et al., 2020).
We adapt BYOL to graphs, and propose Bootstrapped Graph Latents ( BGRL ).
BGRL maintains two
(cid:126)x ( X , A ) (cid:126) (cid:101) x
( (cid:101) X 1 , (cid:101)
A 1 ) (cid:126) (cid:101) x
( (cid:101) X 2 , (cid:101)
A 2 )
A
A(cid:126) (cid:101) h
( (cid:101) H 1 , (cid:101)
A 1 ) (cid:126) (cid:101) h
( (cid:101) H 2 , (cid:101)
A 2 )
E θ
E φ (cid:126) (cid:101) z
( (cid:101) Z 1 , (cid:101)
A 1 ) p θ
EMA (cid:96) ( θ,φ ) (cid:107) distinct graph encoders, and learns a node representation by training an online encoder to predict the representation of a target one. By removing the need to contrast different node representations, BGRL relieves self-supervised learning in graphs from its reliance on negative examples.
B OOTSTRAPPED G RAPH L ATENTS 2. BGRL C OMPONENTS
We consider a graph G =
( X , A ) , with node features X ∈ R N × F and adjacency matrix A ∈ R N × N .
Here N represents the number of nodes and F the number of features.
BGRL maintains two graph encoders, an online encoder E θ and a target encoder E φ , where θ and φ denote distinct parameters.
BGRL ﬁrst produces two alternate views of G : G 1 = ( (cid:101) X 1 , (cid:101)
A 1 ) and G 2 = ( (cid:101) X 2 , (cid:101)
A 2 ) , by applying stochastic graph augmentation functions A 1 and A 2 respectively.
We consider simple augmentations used previously (You et al., 2020; Zhu et al., 2020b), stochastic node feature masking and edge masking .
These augmentations are graph-level: they do not operate on each node independently, and leverage graph topology information.
Further details are available in Appendix B.
The online encoder produces an online representation from the ﬁrst augmented graph, (cid:101) H 1 :=
E θ ( (cid:101) X 1 , (cid:101)
A 1 ) ; similarly the target encoder produces a target representation of the second augmented graph, (cid:101)
H 2 :
= E φ ( (cid:101) X 2 , (cid:101)
A 2 ) .
The online representation is fed into a predictor p θ that outputs a prediction of the target representation, (cid:101)
Z 1 :
= p θ ( (cid:101) H 1 , (cid:101)
A 1 ) .
Unless otherwise speciﬁed, the predictor works at the node level, without using graph information (ie.
operating over (cid:101) H 1 only, and not (cid:101)
A 1 ).
Note that a contrastive approach would instead encourage (cid:101) H (1 ,i ) and (cid:101) H (2 ,j ) to be far apart for node pairs ( i, j ) that are dissimilar.
However, in the absence of a principled way of choosing negative examples, the na ¨ ıve approach of simply contrasting all pairs { ( i, j ) | i (cid:54) =
j } (as done by GRACE ), scales quadratically causing memory issues, and sampling negatives randomly worsens performance.
BGRL ’s computation scales linearly and does not require arbitrary choices on sampling nodes.
2. BGRL UPDATE STEP
The online parameters θ (and not φ ), are updated to make the predictions (cid:101)
Z 1 closer to the true targets (cid:101)
H 2 for each node, following the gradient of the cosine similarity w.r.t.
θ , i.e.,
[Equation]
where η is the learning rate and the ﬁnal updates are computed from the gradients of the objective with respect to θ only .
In practice, we symmetrize the training, by also predicting the target representation of the ﬁrst view using the online representation of the second.
The target parameters φ are updated as an exponential moving average of the online parameters θ , i.e.
[Equation]
[Table]
where τ is a decay rate controlling how close φ remains to θ .
Note that although the objective (cid:96) ( θ, φ ) has undesirable or trivial solutions, the BGRL update as a whole does not optimize this loss.
Only the online parameters θ are updated to reduce this loss, while the target parameters φ follow a different objective.
Empirically, similarly to BYOL , BGRL does not collapse to trivial solutions, and (cid:96) ( θ, φ ) does not converge to 0 (see Appendix F).
2. D ISCUSSION OF COMPUTATIONAL COSTS
The BGRL step takes time and space only linear in the size of the input graph, as opposed to contrastive methods such as GRACE , which are quadratic. Consider a graph with N nodes and M edges, and simple graph models that compute embeddings in time and space O ( N + M ) . BGRL does 4 encoder computations per update step (2 for target/online encoders, and 2 for each augmentation) plus a prediction step; GRACE does 2 (one for each augmentation), plus a projection step. Both methods have the same cost for computing the augmentations, which we ignore in this comparison.
Both methods backpropagate the learning signal 2 times (once for each augmentation), and we assume the backward pass to be approximately as costly as a forward pass.
Thus the total time complexity per update step for BGRL is 6 C encoder ( M + N ) + 4 C prediction N + C BGRL N , and for GRACE is 4 C
[Equation]
M
[Equation]
N )+4 C
[Equation]
N + C GRACE N 2 , where C · are constants depending on architecture of the different components.
A similar analysis applies to the memory complexities.
We further provide an empirical analysis and comparison of runtime and memory requirements in
Section 3.3.
E XPERIMENTS
We evaluate representation learning methods following the standard linear evaluation protocol (Veli ˇ ckovi ´ c et al., 2019): we ﬁrst learn node representations in a fully unsupervised manner, and a linear model is then trained on top of these frozen embeddings without ﬂowing any gradients back to the graph encoder network.
Our training and evaluation process is implemented in JAX (Babuschkin et al., 2020) and Scikit-Learn (Pedregosa et al., 2011) .
Details about the datasets used, our GNN models, and training and evaluation setup are provided in Appendix C, D, and E respectively.
3. E XPERIMENTS ON TRANSDUCTIVE TASKS
WikiCS, Amazon Computers/Photos, Coauthor CS/Physics We
ﬁrst evaluate our method on a set of 5 recent real-world datasets (Mernyei & Cangea, 2020; Shchur et al., 2018) for evaluating node classiﬁcation in the transductive setting.
In our experiments, we primarily compare BGRL against
GRACE (Zhu et al., 2020b), the current state-of-the-art contrastive representation learning approach that relies on negative samples.
For fair comparison, we also report results of running GRACE using our training setup, improving over previously published results on 3 datasets.
Where available, we also report performances of other methods from previously published results.
Finally, we report the performance of Random-Init (Veliˇckovi´c et al., 2019), a randomly initialized encoder with an identical architecture, showing that this simple baseline can lead to very strong embeddings in particular outperforming all previously reported baselines on the WikiCS dataset.
Table 1 shows that BGRL performs competitively both with our unsupervised and fully supervised baselines using comparable architectures
[Table]
ogbn-arXiv We consider a much larger dataset from the OGB benchmark (Hu et al., 2020a)
[Table]
and show in Table 2 that BGRL still performs well.
We implement and compare against two representative contrastive learning approaches, DGI and GRACE .
We also report results for node2vec (Grover &
Leskovec, 2016) and a supervised learning baseline.
For GRACE , since taking negative examples from the entire graph consumes O ( N 2 ) memory and is impractical for a dataset of this size, we subsample k nodes as negative examples per gradient step, seeing that GRACE requires high k to be competitive.
3. E XPERIMENTS ON AN INDUCTIVE TASK WITH MULTIPLE GRAPHS
Finally, we examine the PPI (Protein-Protein Interaction) inductive task, and show in Table 3 that our proposed changes to the training process improve the performance of both BGRL and GRACE to set a new state-of-the-art while maintaining training stability for thousands of epochs.
Effect of subsampling on GRACE The high performance of GRACE is due in part to providing a rich contrastive learning signal that considers every pair of nodes in a graph.
In Figure 2 we more closely analyze the effect of subsampling fewer negative examples per gradient step, to study how this contrastive loss behaves in the absence of being able to provide a rich enough signal.
In our experiments, we sample k random nodes to use as negative examples, and measure how performance is affected by varying k from 1 to 4096.
Note that since the average number of nodes in each graph is 2372, the higher values of k we consider are very close to a quadratic computation.
GRACE requires high k to be competitive, making BGRL promising to apply to larger graphs with memory constraints.
GNN-based BGRL predictor Next, we examine ways to improve the performance of BGRL by strengthening the predictor p θ used in the BGRL loss.
Unlike the case of applying BYOL to images, where the predictor must learn to predict the projection of each image in the minibatch independently, here our graph augmentations are applied to all the nodes in the graph in a coherent way.
Thus, it is intuitive that a model that makes use of all the node embeddings in one view, would be better able to predict the embeddings of nodes in the second view.
Thus, we compare using an MLP to predict each target independently, or a GNN that uses all the node embeddings at once.
Table 3 shows this more powerful graph-based predictor further increases BGRL ’s performance.
GAT encoder models GAT models are known to outperform MeanPooling encoders on PPI task in supervised learning, but have thus far not been trained to a similarly higher performance through unsupervised techniques.
Table 3 shows that BGRL can more effectively train the more complex GAT model to achieve a new state-of-the-art on this dataset.
We also report the performance of a ConstGAT model (i.e. using constant attention weights) trained with BGRL , showing that the non-contrastive loss is able to provide enough signal to allow nodes to aggregate over their neighbors in a non-uniform way.
Interestingly, the more complex GRACE contrastive loss is unable to improve performance of a
GAT model over the standard, smaller MeanPooling encoder.
The all-vs-all loss provided by GRACE is less guided than the targeted bootstrapping objective of BGRL , meaning that it is also less suited to [Table] predictor [Table]
encoder
[Table] ConstGAT [Table] encoder
[Table]
Supervised MeanPooling
[Table]
Supervised
[Table]
[Figure]
guiding the (often brittle) attentional coefﬁcients.
This aligns with recent observations that carefully chosen auxiliary losses are often paramount for stability of GAT models (Kim & Oh, 2021;
Wang et al., 2019).
We provide further analyses of unsupervised training of GAT models in Appendix G.
3. E MPIRICAL C OMPUTATION C OST A NALYSIS
We provide empirical comparisons of BGRL and GRACE on the datasets from Section 3.1 in Table 4. As noted in Section 2.3, theoretical analysis of the BGRL update step shows a linear dependence on graph size, whereas GRACE is quadratic. In practice, runtime is affected by factors such as different operations parallelizing, or implementations trading off memory requirements for higher runtime costs. We show that an advantage of BGRL is reducing memory requirements sufﬁciently to allow scaling to larger datasets easily.
This is particularly signiﬁcant for graph-based applications in practice, where the bottleneck is often memory rather than speed.
[Table]
C ONCLUSION
We have introduced BGRL , a new method for self-supervised graph representation learning.
Through a wide range of experiments, we have shown that our method is competitive with state-of-the-art approaches, in spite of not requiring negative examples and substantially reducing computational requirements.
Moreover, our approach can be naturally extended to learn graph-level embeddings, where deﬁning negative examples is challenging, and an all-vs-all objective does not scale.
Prior to using graph neural networks (GNNs) as graph encoders, dominant methods in the area relied on random-walk objectives such as DeepWalk (Perozzi et al., 2014) and node2vec (Grover &
Leskovec, 2016).
Even though GNNs have an inductive bias that aligns with these objectives, composing GNNs and random-walks does not work very well – randomly-initialised GNNs (Veliˇckovi´c et al., 2019; Kipf & Welling, 2017) and nonparametric GNNs (Wu et al., 2019) are both competitive with
DeepWalk , with no training necessary.
Moreover, training GNN encoders with such loss functions (as done by Hamilton et al., 2017) can even degrade performance relatively to an untrained encoder.
Earlier combinations of GNNs and self-supervised learning involve Embedding Propagation (Garc´ıa-Dur´an & Niepert, 2017), Variational Graph Autoencoders (Kipf & Welling, 2016) and Graph2Gauss (Bojchevski & G¨unnemann, 2018).
While all of these are effectively variations of a random-walk style objective, they are enhanced by relevant additions that are still applicable for current methods, such as edge-wise corruption or explicitly incorporating embedding uncertainty.
Yet another but tangential direction for training encoders for representation is re-using inspiration from
BERT-style (Devlin et al., 2019) losses in graph-structured inputs, as leveraged by Hu et al.
(2020b).
In particular, the strategies of Hu et al.
(2020b) assume that the input graph is attributed in a way that would make feature masking objectives viable.
Recently, contrastive methods effective on images have also been adapted to graphs using GNNs.
This includes DGI (Veli ˇ ckovi ´ c et al., 2019), inspired by Deep InfoMax (Hjelm et al., 2019), which contrasts node-local patches against global graph representations.
Next, InfoGraph (Sun et al., 2020) provided modiﬁcations to DGI ’s pipeline to make the global embedding useful for graph classiﬁcation tasks.
DGI was also generalized to spatiotemporal graphs by ST-DGI (Opolka et al., 2019), and multiplex networks by DMGI (Park et al., 2020).
GMI (Peng et al., 2020) directly maximizes a notion of graphical mutual information inspired by MINE (Belghazi et al., 2018), allowing for a more ﬁne-grained contrastive loss than DGI ’s. Furthermore, the SimCLR method of Chen et al.
(2020a) has been specialized for graphs by GRACE and variants such as GCA (Zhu et al., 2020b;a).
GraphCL (You et al., 2020) adapts GRACE to learn graph-level embeddings using a contrastive objective.
Additionally, MVGRL method (Hassani & Khasahmadi, 2020) generalizes CMC (Tian et al., 2020) to graphs.
GRACE and MVGRL have emerged as state-of-the-art methods, contrasting nodes across various graph views.
Finally, concurrently to our work, Che et al.
(2020) have explored the possibility of using bootstrapping for self-supervised learning in graphs.
However, they only consider small citations datasets with ﬁxed train/test splits, known to be saturated and unreliable to evaluate GNN methods (Shchur et al., 2018).
B G RAPH A UGMENTATIONS
In this work, we consider the standard graph augmentation pipeline that has been used in previous works on representation learning (You et al., 2020; Zhu et al., 2020b).
We use the term “augmentation” as opposed to “corruption” as has been used before (Veliˇckovi´c et al., 2019), as our intention is to produce two views which are semantically similar.
This differs from, e.g. DGI , where semantically dissimilar views are constructed and used to contrast against the original one.
We consider two simple graph augmentation functions — node feature masking and edge masking .
These augmentations are graph-wise: they do not operate on each node independently, and leverage graph topology information through edge masking.
This contrasts with transformations used in BYOL , which operate on each image independently.
First, we generate a single random binary mask of size F , each element of which follows a Bernoulli distribution B (1 − p f ) , and use it to mask the features of all nodes in the graph (i.e., all nodes have the same features masked).
In addition to this node-level attribute transformation, we then also compute a binary mask of size E (where E is the number of edges in the original graph), each element of which follows a Bernoulli distribution B (1 − p e ) , and use it to mask edges in the augmented graph.
To compute our ﬁnal augmented graphs, we make use of both augmentation functions with different hyperparameters for each graph, i.e. p f 1 and p e 1 for the ﬁrst view, and p f 2 and p e 2 for the second view.
The exact values of these hyperparameters follow very closely those reported in prior works and are summarized in Table 6.
Some prior works have also investigated adaptive augmentations (Zhu et al., 2020a;
Che et al., 2020), using heuristics such as node centrality or PageRank centrality (Page et al., 1999) to mask different edges with different probabilities.
This improves the quality of the augmented graphs by helping these transformations preserve semantic similarity (e.g. by making it less likely to mask critical edges that connect otherwise disjoint parts of the graph).
We consider only simple, standard augmentations in order to isolate and study the effect of BGRL as a representation learning method, as it is known that stronger augmentations can have a large impact on the quality of representations learned (Grill et al., 2020).
However, as we show in Table 1, our method is competitive with baselines that use adaptive augmentations.
C D ATASET D ETAILS
[Table]
24 graphs)
[Table]
WikiCS
This graph is constructed from Wikipedia references, with nodes representing articles about Computer Science and edges representing links between them.
Articles are classiﬁed intoclasses based on their subﬁeld, and node features are the average of GloVE (Pennington et al., 2014) embeddings of all words in the article.
This dataset comes with 20 canonical train/valid/test splits, which we use directly.
Amazon Computers, Amazon Photos These graphs are from the Amazon co-purchase graph (McAuley et al., 2015) with nodes representing products and edges being between pairs of goods frequently purchased together.
Products are classiﬁed into 10 (for Computers) and 8 (for Photos) classes based on product category, and node features are a bag-of-words representation of a product’s reviews.
We use a random split of the nodes into (10/10/80%) train/validation/test nodes respectively as these datasets do not come with a standard dataset split.
Coauthor CS, Coauthor Physics These graphs are from the Microsoft Academic Graph (Sinha et al., 2015), with nodes representing authors and edges between authors who have co-authored a paper.
Authors are classiﬁed into 15 (for CS) and 5 (for Physics) classes based on the author’s research ﬁeld, and node features are a bag-of-words representation of the keywords of an author’s papers.
We again use a random (10/10/80%) split for these datasets.
ogbn-arXiv This is another citation network, where nodes represent CS papers on arXiv indexed by the Microsoft Academic Graph (Sinha et al., 2015).
In our experiments, we symmetrize this graph and thus there is an edge between any pair of nodes if one paper has cited the other.
Papers are classiﬁed into 40 classes based on arXiv subject area.
The node features are computed as the average word-embedding of all words in the paper, where the embeddings are computed using a skip-gram model (Mikolov et al., 2013) over the entire corpus.
PPI is a protein-protein interaction network (Zitnik & Leskovec, 2017; Hamilton et al., 2017), comprised of multiple ( 24 ) graphs each corresponding to different human tissues.
We use the standard dataset split as 20 graphs for training, 2 for validation, and 2 for testing.
Each node has 50 features computed from various biological properties.
This is a multilabel classiﬁcation task, where each node can possess up to 121 labels.
D GNN M ODELS
GCN for transductive experiments We use a GCN model (Kipf & Welling, 2017) as our graph encoder E in these experiments.
Formally, the GCN propagation rule for a single layer is as follows,
[Equation]
where (cid:98)
A =
A
+ I is the adjacency matrix with self-loops, (cid:98) D is the degree matrix, σ is a non-linearity such as ReLU, and W i is a learned weight matrix for the i ’th layer.
MeanPooling encoder for inductive experiments Here we use a simple mean-pooling propagation rule from the GraphSAGE-GCN model (Hamilton et al., 2017):
[Equation]
As proposed by Veli ˇ ckovi ´ c et al.
(2019), our exact encoder E is a 3 -layer mean-pooling network with skip connections.
We use a layer size of 512 and PReLU (He et al., 2015) activation.
Thus, we compute:
[Equation]
Graph Attention Networks We also consider GAT (Veli ˇ ckovi ´
c et al., 2018) models where each node aggregates features from its neighbors non-uniformly using a learned attention weight.
The GAT layer consists of a learned matrix W that transforms each node features.
We then use self-attention to compute attention coefﬁcient for a pair of nodes i and j as e ij = a ( h i , h j ) .
The attention function a is computed as LeakyReLU ( a [ Wh i ||
Wh j ]) , where a is a learned matrix transforming a pair of concatenated attention queries into a single scalar attention logit.
The weight of the edge between nodes i and j is computed as α ij = softmax j ( e ij ) .
E I MPLEMENTATION DETAILS
We use GCN (Kipf & Welling, 2017) encoders in our experiments on the transductive tasks, while on the inductive task of PPI we use MeanPooling encoders with residual connections.
The BGRL predictor p θ is implemented as a mutilayer perceptron (MLP).
We also used stabilization techniques like batch normalization (Ioffe & Szegedy, 2015), layer normalization (Ba et al., 2016), and weight standardization (Qiao et al., 2019), as suggested in prior works (Grill et al., 2020; Richemond et al., 2020).
The decay rate use for statistics in the batch normalization is ﬁxed to 0.99.
We use PReLU activation (He et al., 2015) in all experiments except those using a GAT encoder, where we use the
ELU activation (Clevert et al., 2016).
In all our models, at each layer including the ﬁnal layer, we apply ﬁrst the batch/layer normalization as applicable, and then the activation function.
Tabledescribes hyperparameter and architectural details for most of our experimental setups with BGRL .
In addition to these standard settings, we perform two additional experiments on the PPI dataset— using a GNN-based predictor p θ , and using a GAT (Veliˇckovi´c et al., 2018) model as the encoder.
When experimenting on PPI with graph-based predictors, we deﬁne p θ to be an MLP with one hidden layer of size 512, plus a stacked MeanPooling model with the same structure as the encoder E .
We found this combination of the node-based MLP and graph-based GNN to provide the best results.
When using the GAT encoder on PPI, we use 3 attention layers — the ﬁrst two with 4 attention heads of size 256 each, and the ﬁnal with 6 attention heads of size 512, following a very similar model proposed by Veliˇckovi´c et al.
We concatenate the attention head outputs for the ﬁrst 2 layers, and use the mean for the ﬁnal output.
We also use the ELU activation (Clevert et al., 2016), and skip connections in the intermediate attention layers, as suggested by Veli
ˇ ckovi ´ c et al.
[Table]
Optimization settings We perform full-graph training at each gradient step on all experiments, with the exception of experiments using GAT encoders on the PPI dataset.
Here, due to memory constraints, we perform training with a batch size of 1 graph.
Since the PPI dataset consists of multiple smaller, disjoint subgraphs, we do not have to perform any graph subsampling at training time.
All experiments use Glorot initialization (Glorot & Bengio, 2010) the AdamW optimizer (Kingma & Ba, 2015; Gugger & Howard, 2018) with a base learning rate η base and weight decay set to 10 − 5 .
The learning rate is annealed using a cosine schedule over the course of learning of n total total steps with an initial warmup period of n warmup steps.
Hence, the learning rate at step i is computed as
[Equation]
We ﬁx n total to be 10,000 total steps and n warmup to 1,000 warmup steps, with the exception of experiments on the GAT encoder that requires using a batch size of 1 graph on the PPI dataset.
In this case, we increase the number of total steps to 20,000 and warmup to 2,000 steps.
The target network parameters φ are initialized randomly from the same distribution of the online parameters θ but with a different random seed.
The decay parameter τ is also updated using a cosine schedule starting from an initial value of τ base = 0 .
and is computed as
[Equation]
These annealing schedules for both η and τ follow the procedure used by Grill et al.
Evaluation of embeddings The ﬁnal evaluation is done by ﬁtting a linear classiﬁer on top of the frozen learned embeddings without ﬂowing any gradients back to the encoder.
For the smaller datasets of WikiCS, Amazon Computers/Photos, and Coauthor CS/Physics, we use an (cid:96) 2 -regularized LogisticRegression classiﬁer from Scikit-Learn (Pedregosa et al., 2011) using the ‘liblinear’ solver.
We do a hyperparameter search over the regularization strength to be between { 2 − 10 , 2 − 9 , . . .
For larger PPI and ogbn-arXiv datasets, where the liblinear solver takes too long to converge, we instead perform 100 steps of gradient descent using AdamW with learning rate 0.01, with a smaller hyperparameter search on the weight decay between { 2 − 10 , 2 − 8 , 2 − 6 , . . .
F BGRL DOES NOT CONVERGE TO TRIVIAL SOLUTIONS
In Figure 3 we show the BGRL loss curve throughout training for all the datasets considered.
As we see, the loss does not converge to zero, indicating that the training does not result in a trivial solution.
In Figure 4 we plot the spread of the node embeddings, i.e., the standard deviation of the representations learned across all nodes, divided by the average norm.
As we see, the embeddings learned across all datasets have a standard deviation that is a similar order of magnitude as the norms of the embeddings themselves, further indicating that the training dynamics do not converge to a constant solution.
Further, Figure 5 shows that the embeddings do not collapse to zero or blow up as training progresses
[Figure] [Figure]
[Figure]
G A NALYZING LEARNED GRAPH ATTENTION MODELS
[Figure] [Figure]
In this section, we analyze the training of GAT encoders on the PPI dataset more closely.
For this purpose, we additionally evaluate the effect of subsampling negative examples when training with
GRACE .
All other experimental settings are unchanged.
Figure 6 shows that, similar to the MeanPooling models, GAT models are negatively affected by subsampling of negative examples in the GRACE contrastive loss.
We see a steady increase in performance when more negative examples are used, with the peak reached by the version that does not do any subsampling.
Next, we study the entropy of the attention weights learned.
Speciﬁcally, for each node in the PPI training graphs, we compute the average entropy of its attention weights across all GAT layers and attention heads.
Since this entropy depends on the number of neighbors this node has, we subtract as a baseline the entropy of a uniform distribution at the node (i.e., taking the ConstGAT model as a reference point for studying the GAT model).
In Figure 7, we observe that there is a signiﬁcant correlation between model performance and GAT attention entropies.
We see that GAT models learned using GRACE , particularly when subsampling few negative examples, tend to have very low attention entropies.
This provides further evidence that training GAT models can be difﬁcult.
On the other hand, BGRL is able to train the model to have meaningful attention weights, striking a balance between the low-entropy models learned through
GRACE and the maximum-entropy ConstGAT model, thus achieving the best performance.