Original: L ARGE -S CALE R EPRESENTATION L EARNING ON G RAPHS VIA B OOTSTRAPPING
Revised: L ARGE -S CALE R EPRESENTATION L EARNING ON G RAPHS VIA B OOTSTRAPPING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Self-supervised learning provides a promising path towards eliminating the need for costly label information in representation learning on graphs.
Revised: Self-supervised learning provides a promising path towards eliminating the need for costly label information in representation learning on graphs

Original: However, to achieve state-of-the-art performance, methods often need large numbers of negative examples and rely on complex augmentations.
Revised: However, to achieve state -of-the- art performance, methods often need large numbers of neg - ative examples and rely on complex augmentations.

Original: This can be prohibitively expensive, especially for large graphs.
Revised: This can be prohibitively expensive, especially for large graphs

Original: To address these challenges, we introduce
Revised: To address these challenges, we introduce

Original: Bootstrapped Graph Latents ( BGRL ) - a graph representation learning method that learns by predicting alternative augmentations of the input.
Revised: Bootstrapped Graph Latents ( BGRL ) - a graph representation learning method that learns by predicting alternative augmentations of the input

Original: BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and is thus scalable by design.
Revised: BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and is thus scalable by design

Original: BGRL outperforms or matches prior methods on several established benchmarks, while achieving a 2-10x reduction in memory costs.
Revised: BGRL outperforms or matches prior methods on sev - eral established benchmarks, while achieving a 2-10x reduction in memory costs.

Original: Furthermore, we show that BGRL can be scaled up to extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving state-ofthe-art performance and improving over supervised baselines where representations are shaped only through label information. 
Revised: Furthermore we show that BGRL can be scaled up to extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving state-ofthe-art performance and improving over supervised baselines where representations are shaped only through label information. In particular, our solution centered on BGRL constituted one of the winning entries to the Open Graph Benchmark Large Scale

Original: In particular, our solution centered on BGRL constituted one of the winning entries to the Open Graph BenchmarkLarge Scale Challenge at KDD Cup 2021 , on a graph orders of magnitudes larger than all previously available benchmarks, thus demonstrating the scalability and effectiveness of our approach.
Revised: Challenge at KDD Cup 2021 , on a graph orders of magnitudes larger than all previously available benchmarks, thus demonstrating the scalability and effectiveness of our approach

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graphs provide a powerful abstraction for complex datasets that arise in a variety of applications such as social networks, transportation networks, and biological sciences (Hamilton et al., 2017;
Revised: Graphs provide a powerful abstraction for complex datasets that arise in a variety of applications such as social networks, transportation networks, and biological sciences (Hamilton et al., 2017;

Original: DerrowPinion et al., 2021; Zitnik & Leskovec, 2017; Chanussot et al., 2021).
Revised: DerrowPinion et al., 2021; Zitnik & Leskovec, 2017; Chanussot et al., 2021).

Original: Despite recent advances in graph neural networks (GNNs), when trained with supervised data alone, these networks can easily overﬁt and may fail to generalize (Rong et al., 2019).
Revised: Despite recent advances in graph neural networks (GNNs), when trained with supervised data alone, these networks can easily overﬁt and may fail to generalize (Rong et al., 2019).

Original: Thus, ﬁnding ways to form simpliﬁed representations of graph-structured data without labels is an important yet unsolved challenge.
Revised: Thus, ﬁnding ways to form simpliﬁed representations of graph-structured data without labels is an important yet unsolved challenge.

Original: Current state-of-the-art methods for unsupervised representation learning on graphs (Veliˇckovi´c et al., 2019; Peng et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al.
Revised: Current state-of-the-art methods for unsupervised representation learning on graphs (Veliˇckovi´c et al., 2019; Peng et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al.

Original: , 2020b;a; You et al., 2020) are contrastive .
Revised: , 2020b;a; You et al., 2020) are contrastive .

Original: These methods work by pulling together representations of related objects and pushing apart representations of unrelated ones.
Revised: These methods work by pulling together representations of related objects and pushing apart representations of unrelated ones.

Original: For example, current best methods Zhu et al.
Revised: For example, current best methods Zhu et al.

Original: (2020b)
Revised: (2020b)

Original: and
Revised: and

Original: Zhu et al.
Revised: Zhu et al.

Original: (2020a)
Revised: (2020a)

Original: learn node representations by creating two augmented versions of a graph, pulling together the representation of the same node in the two graphs, while pushing apart every other node pair .
Revised: learn node representations by creating two augmented versions of a graph, pulling together the representation of the same node in the two graphs, while pushing apart every other node pair .

Original: As such, they inherently rely on the ability to compare each object to a large number of negatives .
Revised: As such, they inherently rely on the ability to compare each object to a large number of negatives .

Original: In the absence of a principled way of choosing these negatives, this can require computation and memory quadratic in the number of nodes.
Revised: In the absence of a principled way of choosing these negatives, this can require computation and memory quadratic in the number of nodes.

Original: In many cases, the generation of a large number of negatives poses a prohibitive cost, especially for large graphs.
Revised: In many cases, the generation of a large number of negatives poses a prohibitive cost, especially for large graphs.

Original: In this paper, we introduce a scalable approach for self-supervised representation learning on graphs called Bootstrapped Graph Latents ( BGRL ).

Original: Inspired by recent advances in self-supervised learning in vision (Grill et al., 2020) , BGRL learns node representations by encoding two augmented versions of a graph using two distinct graph encoders: an online encoder, and a target encoder.

Original: The online encoder is trained through predicting the representation of the target encoder, while the target encoder is updated as an exponential moving average of the online network.

Original: Critically, BGRL does not require contrasting negative examples, and thus can scale easily to very large graphs.

Original: (cid:126)x ( X , A ) (cid:126) (cid:101) x
Revised: (cid:126)x ( X , A ) (cid:126) (cid:101) x

Original: ( (cid:101) X 1 , (cid:101)
Revised: ( (cid:101) X 1 , (cid:101)

Original: A 1 ) (cid:126) (cid:101) x
Revised: A 1 ) (cid:126) (cid:101) x

Original: ( (cid:101) X 2 , (cid:101)
Revised: ( (cid:101) X 2 , (cid:101)

Original: A 2 )
Revised: A 2 )

Original: T
Revised: T

Original: T(cid:126) (cid:101) h
Revised: T(cid:126) (cid:101) h

Original: ( (cid:101) H 1 , (cid:101)
Revised: ( (cid:101) H 1 , (cid:101)

Original: A 1 ) (cid:126) (cid:101) h
Revised: A 1 ) (cid:126) (cid:101) h

Original: ( (cid:101) H 2 , (cid:101)
Revised: ( (cid:101) H 2 , (cid:101)

Original: A 2 )
Revised: A 2 )

Original: E θ
Revised: E θ

Original: E φ (cid:126) (cid:101) z
Revised: E φ (cid:126) (cid:101) z

Original: ( (cid:101) Z 1 , (cid:101)
Revised: ( (cid:101) Z 1 , (cid:101)

Original: A 1 ) p θ
Revised: A 1 ) p θ

Original: EMA − 2 N N − 1 (cid:80) i =0 (cid:101) Z (1 ,i ) (cid:101) H (cid:62) (2 ,i ) (cid:107) (cid:101)
Revised: EMA − 2 N N − 1 (cid:80) i =0 (cid:101) Z (1 ,i ) (cid:101) H (cid:62) (2 ,i ) (cid:107) (cid:101)

Original: Z (1 ,i ) (cid:107)(cid:107) (cid:101) H (2 ,i ) (cid:107) (cid:107)
Revised: Z (1 ,i ) (cid:107)(cid:107) (cid:101) H (2 ,i ) (cid:107) (cid:107)

Revised: In this paper we introduce a scalable approach for self-supervised representation learning on graphs called Bootstrapped Graph Latents ( BGRL ).

Revised: Inspired by recent advances in self-supervised learning in vision (Grill et al., 2020) , BGRL learns node representations by encoding two augmented versions of a graph using two distinct graph encoders: an online encoder, and a target encoder.

Revised: The online encoder is trained through predicting the representation of the target encoder, while the target encoder is updated as an exponential moving average of the online network.

Revised: Critically, BGRL does not require contrasting negative examples, and thus can scale easily to very large graphs.

Original: Our main contributions are:
Revised: Our main contributions are:

Original: • We introduce Bootstrapped Graph Latents ( BGRL ), a graph self-supervised learning method that effectively scales to extremely large graphs and outperforms existing methods, while using only simple graph augmentations and not requiring negative examples (Section 2).
Revised: • We introduce Bootstrapped Graph Latents ( BGRL ), a graph self-supervised learning method that effectively scales to extremely large graphs and outperforms existing methods, while using only simple graph augmentations and not requiring negative examples (Section 2).

Original: • We show that contrastive methods face a trade-off between peak performance and memory constraints, due to their reliance on negative examples (Section 4.2).
Revised: • We show that contrastive methods face a trade-off between peak performance and memory constraints, due to their reliance on negative examples (Section 4.2).

Original: Due to having time and space complexity scaling only linearly in the size of the input, BGRL avoids the performancememory trade-off inherent to contrastive methods altogether.
Revised: Due to having time and space complexity scaling only linearly in the size of the input, BGRL avoids the performancememory trade-off inherent to contrastive methods altogether.

Original: BGRL provides performance competitive with the best contrastive methods, while using 2-10x less memory on standard benchmarks (Section 3).
Revised: BGRL provides performance competitive with the best contrastive methods, while using 2-10x less memory on standard benchmarks (Section 3).

Original: • We show that leveraging the scalability of BGRL allows making full use of the vast amounts of unlabeled data present in large graphs via semi-supervised learning.
Revised: • We show that leveraging the scalability of BGRL allows making full use of the vast amounts of unlabeled data present in large graphs via semi-supervised learning.

Original: In particular, we ﬁnd that efﬁcient use of unlabeled data for representation learning prevents representations from overﬁtting to the classiﬁcation task, and achieves signiﬁcantly higher, state-of-the-art performance.
Revised: In particular, we ﬁnd that efﬁcient use of unlabeled data for representation learning prevents representations from overﬁtting to the classiﬁcation task, and achieves signiﬁcantly higher, state-of-the-art performance.

Original: This was critical to the success of our solution at KDD Cup 2021 in which our BGRL -based solution was awarded one of the winners , on the largest publicly available graph dataset, of size 360GB consisting of 240 million nodes and 1.7 billion edges (Section 4.3)
Revised: This was critical to the success of our solution at KDD Cup 2021 in which our BGRL -based solution was awarded one of the winners , on the largest publicly available graph dataset, of size 360GB consisting of 240 million nodes and 1.7 billion edges (Section 4.3).

Original:  B OOTSTRAPPED G RAPH L ATENTS 2. BGRL C OMPONENTS
Revised:  B OOTSTRAPPED G RAPH L ATENTS 2. BGRL C OMPONENTS

Original: BGRL builds representations through the use of two graph encoders, an online encoder E θ and a target encoder E φ , where θ and φ denote two distinct sets of parameters.
Revised: BGRL builds representations through the use of two graph encoders, an online encoder E θ and a target encoder E φ , where θ and φ denote two distinct sets of parameters.

Original: We consider a graph G =
Revised: We consider a graph G =

Original: ( X , A ) , with node features X ∈ R N × F and adjacency matrix A ∈ R N × N .
Revised: ( X , A ) , with node features X ∈ R N × F and adjacency matrix A ∈ R N × N .

Original: BGRL ﬁrst produces two alternate views of G : G 1 = ( (cid:101) X 1 , (cid:101)
Revised: BGRL ﬁrst produces two alternate views of G : G 1 = ( (cid:101) X 1 , (cid:101)

Original: A 1 ) and G 2 = ( (cid:101) X 2 , (cid:101)
Revised: A 1 ) and G 2 = ( (cid:101) X 2 , (cid:101)

Original: A 2 ) , by applying stochastic graph augmentation functions T 1 and T 2 respectively.
Revised: A 2 ) , by applying stochastic graph augmentation functions T 1 and T 2 respectively.

Original: The online encoder produces an online representation from the ﬁrst augmented graph, (cid:101) H 1 :=
Revised: The online encoder produces an online representation from the ﬁrst augmented graph, (cid:101) H 1 :=

Original: E θ ( (cid:101) X 1 , (cid:101)
Revised: E θ ( (cid:101) X 1 , (cid:101)

Original: A 1 ) ; similarly the target encoder produces a target representation of the second augmented graph, (cid:101)
Revised: A 1 ) ; similarly the target encoder produces a target representation of the second augmented graph, (cid:101)

Original: H 2 :
Revised: H 2 :

Original: = E φ ( (cid:101) X 2 , (cid:101)
Revised: = E φ ( (cid:101) X 2 , (cid:101)

Original: A 2 ) .
Revised: A 2 ) .

Original: The online representation is fed into a node-level predictor p θ that outputs a prediction of the target representation, (cid:101)
Revised: The online representation is fed into a node-level predictor p θ that outputs a prediction of the target representation, (cid:101)

Original: Z 1 :
Revised: Z 1 :

Original: = p θ ( (cid:101) H 1 ) .
Revised: = p θ ( (cid:101) H 1 ) .

Original: BGRL differs from prior bootstrapping approaches such as BYOL (Grill et al., 2020) in that it does not use a projector network .
Revised: BGRL differs from prior bootstrapping approaches such as BYOL (Grill et al., 2020) in that it does not use a projector network .

Original: Unlike vision tasks, in which a projection step is used by BYOL for dimensionality reduction, common embedding sizes are quite small for graph tasks and so this is not a concern in our case.
Revised: Unlike vision tasks, in which a projection step is used by BYOL for dimensionality reduction, common embedding sizes are quite small for graph tasks and so this is not a concern in our case.

Original: In fact, we observe that this step can be eliminated altogether without loss in performance (Appendix B).
Revised: In fact, we observe that this step can be eliminated altogether without loss in performance (Appendix B).

Original: The augmentation functions T 1 and T 2 used are simple, standard graph perturbations previously explored (You et al., 2020; Zhu et al., 2020b).
Revised: The augmentation functions T 1 and T 2 used are simple, standard graph perturbations previously explored (You et al., 2020; Zhu et al., 2020b).

Original: We use a combination of random node feature masking and edge masking with ﬁxed masking probabilities p f and p e respectively.
Revised: We use a combination of random node feature masking and edge masking with ﬁxed masking probabilities p f and p e respectively.

Original: More details and background on graph augmentations is provided in Appendix D.
Revised: More details and background on graph augmentations is provided in Appendix D.

Original: 2. BGRL UPDATE STEP
Revised: 2. BGRL UPDATE STEP

Original: Updating the online encoder E θ : The online parameters θ (and not φ ), are updated to make the predicted target representations (cid:101)
Revised: Updating the online encoder E θ : The online parameters θ (and not φ ), are updated to make the predicted target representations (cid:101)

Original: Z 1 closer to the true target representations (cid:101)
Revised: Z 1 closer to the true target representations (cid:101)

Original: H 2 for each node, by following the gradient of the cosine similarity w.r.t.
Revised: H 2 for each node, by following the gradient of the cosine similarity w.r.t.

Original: θ , i.e.,
Revised: θ , i.e.,

Original: [Equation]
Revised: [Equation]

Original: where η is the learning rate and the ﬁnal updates are computed from the gradients of the objective with respect to θ only , using an optimization method such as SGD or Adam (Kingma & Ba, 2015).
Revised: where η is the learning rate and the ﬁnal updates are computed from the gradients of the objective with respect to θ only , using an optimization method such as SGD or Adam (Kingma & Ba, 2015).

Original: In practice, we symmetrize this loss, by also predicting the target representation of the ﬁrst view using the online representation of the second.
Revised: In practice, we symmetrize this loss, by also predicting the target representation of the ﬁrst view using the online representation of the second.

Original: Updating the target encoder E φ : The target parameters φ are updated as an exponential moving average of the online parameters θ , using a decay rate τ , i.e.,
Revised: Updating the target encoder E φ : The target parameters φ are updated as an exponential moving average of the online parameters θ , using a decay rate τ , i.e.,

Original: [Equation]
Revised: [Equation]

Original: Figure 1 visually summarizes BGRL ’s architecture.
Revised: Figure 1 visually summarizes BGRL ’s architecture.

Original: Note that although the objective (cid:96) ( θ, φ ) has undesirable or trivial solutions, BGRL does not actually optimize this loss.
Revised: Note that although the objective (cid:96) ( θ, φ ) has undesirable or trivial solutions, BGRL does not actually optimize this loss.

Original: Only the online parameters θ are updated to reduce this loss, while the target parameters φ follow a different objective.
Revised: Only the online parameters θ are updated to reduce this loss, while the target parameters φ follow a different objective.

Original: This non-collapsing behavior even without relying on negatives has been studied further (Tian et al., 2021).
Revised: This non-collapsing behavior even without relying on negatives has been studied further (Tian et al., 2021).

Original: We provide an empirical analysis of this behavior in Appendix A, showing that in practice BGRL does not collapse to trivial solutions and (cid:96) ( θ, φ ) does not converge to 0 .
Revised: We provide an empirical analysis of this behavior in Appendix A, showing that in practice BGRL does not collapse to trivial solutions and (cid:96) ( θ, φ ) does not converge to 0 .

Original: Scalable non-contrastive objective: Here we note that a contrastive approach would instead encourage (cid:101) Z (1 ,i ) and (cid:101) H (2 ,j ) to be far apart for node pairs ( i, j ) that are dissimilar.
Revised: Scalable non-contrastive objective: Here we note that a contrastive approach would instead encourage (cid:101) Z (1 ,i ) and (cid:101) H (2 ,j ) to be far apart for node pairs ( i, j ) that are dissimilar.

Original: In the absence of a principled way of choosing such dissimilar pairs, the naïve approach of simply contrasting all pairs { ( i, j ) | i (cid:54) =
Revised: In the absence of a principled way of choosing such dissimilar pairs, the naïve approach of simply contrasting all pairs { ( i, j ) | i (cid:54) =

Original: j } , scales quadratically in the size of the input.
Revised: j } , scales quadratically in the size of the input.

Original: As BGRL does not rely on this contrastive step, BGRL scales linearly in the size of the graph, and thus is scalable by design.
Revised: As BGRL does not rely on this contrastive step, BGRL scales linearly in the size of the graph, and thus is scalable by design.

Original:  C OMPUTATIONAL C OMPLEXITY A NALYSIS
Revised:  C OMPUTATIONAL C OMPLEXITY A NALYSIS

Original: We provide a brief description of the time and space complexities of the BGRL update step, and illustrate its advantages compared to previous strong contrastive methods such as GRACE (Zhu et al., 2020b), which perform a quadratic all-pairs contrastive computation at each update step.
Revised: We provide a brief description of the time and space complexities of the BGRL update step, and illustrate its advantages compared to previous strong contrastive methods such as GRACE (Zhu et al., 2020b), which perform a quadratic all-pairs contrastive computation at each update step.

Original: The same analysis applies to variations of the GRACE method such as GCA (Zhu et al., 2020a).
Revised: The same analysis applies to variations of the GRACE method such as GCA (Zhu et al., 2020a).

Original: Consider a graph with N nodes and M edges, and simple encoders E that compute embeddings in time and space O
Revised: Consider a graph with N nodes and M edges, and simple encoders E that compute embeddings in time and space O

Original: ( N + M ) .
Revised: ( N + M ) .

Original: This property is satisﬁed by most popular GNN architectures such as convolutional (Kipf & Welling, 2017), attentional (Veliˇckovi´c et al., 2018), or message-passing (Gilmer et al., 2017) networks.
Revised: This property is satisﬁed by most popular GNN architectures such as convolutional (Kipf & Welling, 2017), attentional (Veliˇckovi´c et al., 2018), or message-passing (Gilmer et al., 2017) networks.

Original: BGRL performs four encoder computations per update step (twice for the target and online encoders, and twice for each augmentation) plus a node-level prediction step; GRACE performs two encoder computations (once for each augmentation), plus a node-level projection step.
Revised: BGRL performs four encoder computations per update step (twice for the target and online encoders, and twice for each augmentation) plus a node-level prediction step; GRACE performs two encoder computations (once for each augmentation), plus a node-level projection step.

Original: Both methods backpropagate the learning signal twice (once for each augmentation), and we assume the backward pass to be approximately as costly as a forward pass.
Revised: Both methods backpropagate the learning signal twice (once for each augmentation), and we assume the backward pass to be approximately as costly as a forward pass.

Original: We ignore the cost for computation of the augmentations in this analysis.
Revised: We ignore the cost for computation of the augmentations in this analysis.

Original: Thus the total time and space complexity per update step for BGRL is 6 C encoder ( M + N ) + 4 C prediction N + C BGRL N , compared to 4 C encoder ( M + N ) + 4 C projection N + C GRACE N 2 for GRACE , where C · are constants depending on architecture of the different components.
Revised: Thus the total time and space complexity per update step for BGRL is 6 C encoder ( M + N ) + 4 C prediction N + C BGRL N , compared to 4 C encoder ( M + N ) + 4 C projection N + C GRACE N 2 for GRACE , where C · are constants depending on architecture of the different components.

Original: Table 1 shows an empirical comparison of BGRL and
Revised: Table 1 shows an empirical comparison of BGRL and

Original: GRACE ’s computational requirements on a set of benchmark tasks. [Table]
Revised: GRACE ’s computational requirements on a set of benchmark tasks, with further details in Appendix I.

Original: [Table]
Revised: [Table]

Original:  E XPERIMENTAL A NALYSIS
Revised:  E XPERIMENTAL A NALYSIS

Original: We present an extensive empirical study of performance and scalability, showing that BGRL is effective across a wide range of settings from frozen linear evaluation to semi-supervised learning, and both when performing full-graph training and training on subsampled node neighborhoods.
Revised: We present an extensive empirical study of performance and scalability, showing that BGRL is effective across a wide range of settings from frozen linear evaluation to semi-supervised learning, and both when performing full-graph training and training on subsampled node neighborhoods.

Original: We give results across a range of dataset scales and encoder architectures including convolutional, attentional, and message-passing neural networks.
Revised: We give results across a range of dataset scales and encoder architectures including convolutional, attentional, and message-passing neural networks.

Original: We analyze the performance of BGRL on a set of 7 standard transductive and inductive benchmark tasks, as well as in the very high-data regime by evaluating on the MAG240M dataset (Hu et al., 2021).
Revised: We analyze the performance of BGRL on a set of 7 standard transductive and inductive benchmark tasks, as well as in the very high-data regime by evaluating on the MAG240M dataset (Hu et al., 2021).

Original: We present results on medium-sized datasets where contrastive objectives can be computed on the entire graph (Section 4.1), on larger datasets where this objective must be approximated (Section 4.2), and ﬁnally on the much larger MAG

Revised: We present results on medium-sized datasets where contrastive objectives can be computed on the entire graph (Section 4.1), on larger datasets where this objective must be approximated (Section 4.2), and ﬁnally on the much larger MAG

Original: M dataset designed to test scalability limits (Section 4.3), showing that BGRL improves performance across all scales of datasets.
Revised: M dataset designed to test scalability limits (Section 4.3), showing that BGRL improves performance across all scales of datasets.

Original: In Appendix C, we show that BGRL achieves state-of-the-art performance even in the low-data regime on a set ofsmall-scale datasets.
Revised: In Appendix C, we show that BGRL achieves state-of-the-art performance even in the low-data regime on a set ofsmall-scale datasets.

Original: Dataset sizes are summarized in Table 2 and described further in Appendix E.
Revised: Dataset sizes are summarized in Table 2 and described further in Appendix E.

Original: Evaluation protocol:
Revised: Evaluation protocol:

Original: In most tasks, we follow the standard linear-evaluation protocol on graphs (Veli ˇ ckovi ´ c et al., 2019).
Revised: In most tasks, we follow the standard linear-evaluation protocol on graphs (Veli ˇ ckovi ´ c et al., 2019).

Original: This involves ﬁrst training each graph encoder in a fully unsupervised manner and computing embeddings for each node; a simple linear model is then trained on top of these frozen embeddings through a logistic regression loss with (cid:96) 2 regularization, without ﬂowing any gradients back to the graph encoder network.
Revised: This involves ﬁrst training each graph encoder in a fully unsupervised manner and computing embeddings for each node; a simple linear model is then trained on top of these frozen embeddings through a logistic regression loss with (cid:96) 2 regularization, without ﬂowing any gradients back to the graph encoder network.

Original: In the more challenging MAG240M task, we extend BGRL to the semi-supervised setting by combining our self-supervised representation learning loss together with a supervised loss.
Revised: In the more challenging MAG240M task, we extend BGRL to the semi-supervised setting by combining our self-supervised representation learning loss together with a supervised loss.

Original: We show that BGRL ’s bootstrapping objective obtains state-of-the-art performance in this hybrid setting, and even improves further with the added use of unlabeled data for representation learning - properties which have not been previously demonstrated by prior works on self-supervised representation learning on graphs.
Revised: We show that BGRL ’s bootstrapping objective obtains state-of-the-art performance in this hybrid setting, and even improves further with the added use of unlabeled data for representation learning - properties which have not been previously demonstrated by prior works on self-supervised representation learning on graphs.

Original: Implementation details including model architectures and hyperparameters are provided in Appendix F. Algorithm implementation and experiment code for most tasks have been provided in supplementary material, while code for our solution on MAG240M has been open-sourced1 as part of the KDD Cup 2021 .
Revised: Implementation details including model architectures and hyperparameters are provided in Appendix F. Algorithm implementation and experiment code for most tasks can be found at https://github.com/nerdslab/bgrl while code for our solution on MAG240M has been open-sourced as part of the KDD Cup 2021 (Addanki et al., 2021) at https://github.com/deepmind/deepmindresearch/tree/master/ogb_lsc/mag.

Original: 4. P ERFORMANCE AND E FFICIENCY GAINS WHEN SCALABILITY IS NOT A BOTTLENECK
Revised: 4. P ERFORMANCE AND E FFICIENCY GAINS WHEN SCALABILITY IS NOT A BOTTLENECK

Original: We ﬁrst evaluate our method on a set of 5 recent real-world datasets — WikiCS, Amazon-Computers,
Revised: We ﬁrst evaluate our method on a set of 5 recent real-world datasets — WikiCS, Amazon-Computers,

Original: Amazon-Photos, Coauthor-CS, Coauthor-Physics — in the transductive setting.
Revised: Amazon-Photos, Coauthor-CS, Coauthor-Physics — in the transductive setting.

Original: Note that these are challenging medium-scale datasets speciﬁcally proposed for rigorous evaluation of semi-supervised node classiﬁcation methods (Mernyei & Cangea, 2020; Shchur et al., 2018), but are almost all small enough that constrastive approaches such as GRACE (Zhu et al., 2020b) can compute their quadratic objective exactly.
Revised: Note that these are challenging medium-scale datasets speciﬁcally proposed for rigorous evaluation of semi-supervised node classiﬁcation methods (Mernyei & Cangea, 2020; Shchur et al., 2018), but are almost all small enough that constrastive approaches such as GRACE (Zhu et al., 2020b) can compute their quadratic objective exactly.

Original: Thus, these experiments present a comparison of BGRL with prior methods in the  idealized case where scalability is not a bottleneck.
Revised: Thus, these experiments present a comparison of BGRL with prior methods in the [Table] idealized case where scalability is not a bottleneck.

Original: We show that even in this steelmanned setting, our method outperforms or matches prior methods while requiring a fraction of the memory costs.
Revised: We show that even in this steelmanned setting, our method outperforms or matches prior methods while requiring a fraction of the memory costs.

Original: We primarily compare BGRL against GRACE , a recent strong contrastive representation learning method on graphs.
Revised: We primarily compare BGRL against GRACE , a recent strong contrastive representation learning method on graphs.

Original: We also report performances for other commonly used self-supervised graph methods from previously published results (Perozzi et al., 2014; Veliˇckovi´c et al., 2019; Peng et al., 2020;
Revised: We also report performances for other commonly used self-supervised graph methods from previously published results (Perozzi et al., 2014; Veliˇckovi´c et al., 2019; Peng et al., 2020;

Original: Hassani & Khasahmadi, 2020; Zhu et al., 2020a), as well as Random-Init (Veliˇckovi´c et al., 2019), a baseline using embeddings from a randomly initialized encoder, thus measuring the quality of the inductive biases present in the encoder model.
Revised: Hassani & Khasahmadi, 2020; Zhu et al., 2020a) [Table] as well as Random-Init (Veliˇckovi´c et al., 2019), a baseline using embeddings from a randomly initialized encoder, thus measuring the quality of the inductive biases present in the encoder model.

Original: We use a 2-layer GCN model (Kipf & Welling, 2017) as our graph encoder E , and closely follow models, architectures, and graph-augmentation settings used in prior works (Zhu et al., 2020a; Veliˇckovi´c et al., 2019; Zhu et al., 2020b).
Revised: We use a 2-layer GCN model (Kipf & Welling, 2017) as our graph encoder E [Table] and closely follow models, architectures

Original: [Table]
Revised: [Table]

Original: In Table
Revised: and graph-augmentation settings used in prior works (Zhu et al., 2020a; Veliˇckovi´c et al., 2019; Zhu et al., 2020b).

Original: [Table]
Revised: [Table]

Original:  we report results of our experiments on these standard benchmark tasks.
Revised: In Table 3, we report results of our experiments on these standard benchmark tasks.

Original: We see that even when scalability does not prevent the use of contrastive objectives, BGRL performs competitively both with our unsupervised and fully supervised baselines, achieving state-of-the-art performance inof the 5 datasets.
Revised: We see that even when scalability does not prevent the use of contrastive objectives, BGRL performs competitively both with our unsupervised and fully supervised baselines, achieving state-of-the-art performance inof the 5 datasets.

Original: Further, as noted in Table 1, BGRL achieves this despite using 2-10x less memory.
Revised: Further, as noted in Table 1, BGRL achieves this despite using 2-10x less memory.

Original: BGRL provides this improvement in memory-efﬁciency at no cost in performance, demonstrating a useful practical advantage over prior methods such as GRACE .
Revised: BGRL provides this improvement in memory-efﬁciency at no cost in performance, demonstrating a useful practical advantage over prior methods such as GRACE .

Original: Effect of more complex augmentations:
Revised: Effect of more complex augmentations:

Original: In addition to the original GRACE method, we also highlight GCA , a variant of it that has the same learning objective but trades off more expressive but expensive graph augmentations for better performance.
Revised: In addition to the original GRACE method, we also highlight GCA , a variant of it that has the same learning objective but trades off more expressive but expensive graph augmentations for better performance.

Original: However, these augmentations often take time cubic in the size of the graph [Table] or are otherwise cumbersome to implement on large graphs.
Revised: However, these augmentations often take time cubic in the size of the graph, or are otherwise cumbersome to implement on large graphs.

Original: As we focus on scalability to the high-data regime, we primarily restrict our comparisons to the base method
Revised: As we focus on scalability to the high-data regime, we primarily restrict our comparisons to the base method

Original: GRACE , which uses the same simple, easily scalable augmentations as BGRL .
Revised: GRACE , which uses the same simple, easily scalable augmentations as BGRL .

Original: Nevertheless, for the sake of completeness, in Table 4 we investigate the effect of these complex augmentations with BGRL [Table] [Table]
Revised: Nevertheless, for the sake of completeness, in Table 4 we investigate the effect of these complex augmentations with BGRL .

Original: We see that BGRL obtains equivalent performance with both simple and complex augmentations, while
Revised: We see that BGRL obtains equivalent performance with both simple and complex augmentations, while

Original: GCA requires more expensive augmentations for peak performance.
Revised: GCA requires more expensive augmentations for peak performance.

Original: This indicates that BGRL can safely rely on simple augmentations when scaling to larger graphs without sacriﬁcing performance. 
Revised: This indicates that BGRL can safely rely on simple augmentations when scaling to larger graphs without sacriﬁcing performance. [Table] [Table]

Original: 4. S CALABILITY - PERFORMANCE TRADE - OFFS FOR LARGE GRAPHS
Revised: 4. S CALABILITY - PERFORMANCE TRADE - OFFS FOR LARGE GRAPHS

Original: When scaling up to large graphs, it may not be possible to compare each node’s representation to all others.
Revised: When scaling up to large graphs, it may not be possible to compare each node’s representation to all others.

Original: In this case, a natural way to reduce memory is to compare each node with only a subset of nodes in the rest of the graph.
Revised: In this case, a natural way to reduce memory is to compare each node with only a subset of nodes in the rest of the graph.

Original: To study how the number of negatives impacts performance in this case, we propose an approximation of GRACE ’s objective called GRACE-SUBSAMPLING , where instead of contrasting every pair of nodes in the graph, we subsample k nodes randomly across the graph to use as negative examples for each node at every gradient step.
Revised: To study how the number of negatives impacts performance in this case [Table] we propose an approximation of GRACE ’s objective called GRACE-SUBSAMPLING , where instead of contrasting every pair of nodes in the graph, we subsample k nodes randomly across the graph to use as negative examples for each node at every gradient step.

Original: Note that k
Revised: Note that k

Original: = 2 is the asymptotic equivalent of BGRL in terms of memory costs, as BGRL always only compares each node with itself across both views, i.e., BGRL faces no such computational difﬁculty or design choice in scaling up.
Revised: = 2 is the asymptotic equivalent of BGRL in terms of memory costs, as BGRL always only compares each node with itself across both views, i.e., BGRL faces no such computational difﬁculty or design choice in scaling up.

Original: E VALUATING ON OGBN - AR X IV D ATASET
Revised: E VALUATING ON OGBN - AR X IV D ATASET

Original: To study the tradeoff between performance and complexity we consider a node classiﬁcation task on a much larger dataset, from the OGB benchmark (Hu et al., 2020a), ogbn-arXiv.
Revised: To study the tradeoff between performance and complexity we consider a node classiﬁcation task on a much larger dataset, from the OGB benchmark (Hu et al., 2020a), ogbn-arXiv.

Original: In this case, GRACE cannot run without subsampling (on a GPU with 16GB of memory).
Revised: In this case, GRACE cannot run without subsampling (on a GPU with 16GB of memory).

Original: Considering the increased difﬁculty of this task, we slightly expand our model to use 3 GCN layers, following the baseline model provided by Hu et al.
Revised: Considering the increased difﬁculty of this task, we slightly expand our model to use 3 GCN layers, following the baseline model provided by Hu et al.

Original: (2020a).
Revised: (2020a).

Original: As there has not been prior work on applying GNN-based unsupervised approaches to the ogbn-arXiv task, we implement and compare against two representative contrastivelearning approaches, DGI and GRACE .
Revised: As there has not been prior work on applying GNN-based unsupervised approaches to the ogbn-arXiv task, we implement and compare against two representative contrastivelearning approaches, DGI and GRACE .

Original: In addition, we report results from Hu et al.
Revised: In addition, we report results from Hu et al.

Original: (2020a) for node2vec (Grover & Leskovec, 2016) and a supervised-learning baseline.
Revised: (2020a) for node2vec (Grover & Leskovec, 2016) and a supervised-learning baseline.

Original: We report results on both validation and test sets, as is convention for this task since the dataset is split based on a chronological ordering.
Revised: We report results on both validation and test sets, as is convention for this task since the dataset is split based on a chronological ordering.

Original: Our results, summarized in Table 5, show that BGRL is competitive with the supervised learning baseline.
Revised: Our results, summarized in Table 5, show that BGRL is competitive with the supervised learning baseline.

Original: Further, we note that the performance of GRACE-SUBSAMPLING is very sensitive to the parameter k —requiring a large number of negatives to match the performance of BGRL .
Revised: Further, we note that the performance of GRACE-SUBSAMPLING is very sensitive to the parameter k —requiring a large number of negatives to match the performance of BGRL .

Original: Note that
Revised: Note that

Original: BGRL far exceeds the performance of GRACE-SUBSAMPLING with k = 2 , its asymptotic equivalent in terms of memory; and that larger values of k lead to out-of-memory errors on a 16GB GPU.
Revised: BGRL far exceeds the performance of GRACE-SUBSAMPLING with k = 2 , its asymptotic equivalent in terms of memory; and that larger values of k lead to out-of-memory errors on a 16GB GPU.

Original: These results suggest that the performance of contrastive methods such as GRACE may suffer due to approximations to their objectives that must be made when scaling up. 
Revised: These results suggest that the performance of contrastive methods such as GRACE may suffer due to approximations to their objectives that must be made when scaling up. [Table]

Original: E VALUATING ON P ROTEIN -P ROTEIN I NTERACTION D ATASET
Revised: E VALUATING ON P ROTEIN -P ROTEIN I NTERACTION D ATASET

Original: Next, we consider the Protein-Protein Interaction (PPI) task—a more challenging inductive task on multiple graphs where the gap between the best self-supervised methods and fully supervised methods is (still) signiﬁcantly large, due to 40% of the nodes missing feature information.
Revised: Next, we consider the Protein-Protein Interaction (PPI) task—a more challenging inductive task on multiple graphs where the gap between the best self-supervised methods and fully supervised methods is (still) signiﬁcantly large, due to 40% of the nodes missing feature information.

Original: In addition to simple mean-pooling propagation rules from GraphSage-GCN (Hamilton et al., 2017), we also consider
Revised: In addition to simple mean-pooling propagation rules from GraphSage-GCN (Hamilton et al., 2017), we also consider

Original: Graph Attention Networks (GAT, Veli
Revised: Graph Attention Networks (GAT, Veli

Original: ˇ ckovi ´ c et al., 2018) where each node aggregates features from[Table] its neighbors non-uniformly using a learned attention weight.
Revised: ˇ ckovi ´ c et al., 2018) where each node aggregates features from its neighbors non-uniformly using a learned attention weight.

Original: It has been shown (Veliˇckovi´c et al., 2018) that GAT improves over non-attentional models on this dataset when trained in supervised settings, but these models have thus far not been able to be trained to a higher performance than non-attentional models through contrastive techniques.
Revised: It has been shown (Veliˇckovi´c et al., 2018) that GAT improves over non-attentional models on this dataset when trained in supervised settings, but these models have thus far not been able to be trained to a higher performance than non-attentional models through contrastive techniques.

Original: We report our results in Table 6, showing that BGRL is competitive with GRACE when using the simpler MeanPooling networks.
Revised: We report our results in Table 6, showing that BGRL is competitive with GRACE when using the simpler MeanPooling networks.

Original: Applying BGRL to a GAT model, results in a new state-of-the-art performance, improving over the MeanPooling network.
Revised: Applying BGRL to a GAT model, results in a new state-of-the-art performance, improving over the MeanPooling network.

Original: On the other hand, the GRACE contrastive loss is unable to improve performance of a GAT model over the non-attentional MeanPooling encoder.
Revised: On the other hand, the GRACE contrastive loss is unable to improve performance of a GAT model over the non-attentional MeanPooling encoder.

Original: We observe that the approximation of the contrastive objective results not only in lower accuracies
Revised: We observe that the approximation of the contrastive objective results not only in lower accuracies

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: (Figure 2), but also qualitatively different behaviors in the GAT models being trained.
Revised: (Figure 2), but also qualitatively different behaviors in the GAT models being trained.

Original: In Figure 3, we examine the internals of GAT models trained through both BGRL and GRACE by analyzing the entropy of the attention weights learned.
Revised: In Figure 3, we examine the internals of GAT models trained through both BGRL and GRACE by analyzing the entropy of the attention weights learned.

Original: For each training node, we compute the average entropy of its attention weights across all GAT layers and attention heads, minus the entropy of a uniform attention distribution as a baseline.
Revised: For each training node, we compute the average entropy of its attention weights across all GAT layers and attention heads, minus the entropy of a uniform attention distribution as a baseline.

Original: We see that GAT models learned using GRACE , particularly when subsampling few negative examples, tend to have very low attention entropies and perform poorly.
Revised: We see that GAT models learned using GRACE , particularly when subsampling few negative examples, tend to have very low attention entropies and perform poorly.

Original: On the other hand, BGRL is able to train the model to have meaningful attention weights, striking a balance between the low-entropy models learned through GRACE and the maximum-entropy uniform-attention distribution.
Revised: On the other hand, BGRL is able to train the model to have meaningful attention weights, striking a balance between the low-entropy models learned through GRACE and the maximum-entropy uniform-attention distribution.

Original: This aligns with recent observations (Kim & Oh, 2021; Wang et al., 2019) that auxiliary losses must be chosen carefully for the stability of GAT attention weights.
Revised: This aligns with recent observations (Kim & Oh, 2021; Wang et al., 2019) that auxiliary losses must be chosen carefully for the stability of GAT attention weights.

Original: 4. S CALING TO EXTREMELY LARGE GRAPHS
Revised: 4. S CALING TO EXTREMELY LARGE GRAPHS

Original: To further test the scalability and evaluate the performance of BGRL in the very high-data regime, we consider the MAG240M node classiﬁcation task (Hu et al., 2021).
Revised: To further test the scalability and evaluate the performance of BGRL in the very high-data regime, we consider the MAG240M node classiﬁcation task (Hu et al., 2021).

Original: As a single connected graph of 360GB with over 240 million nodes (of which 1.4 million are labeled) and 1.7 billion edges, this dataset is orders of magnitude larger than previously available public datasets, and poses a signiﬁcant scaling challenge.
Revised: As a single connected graph of 360GB with over 240 million nodes (of which 1.4 million are labeled) and 1.7 billion edges, this dataset is orders of magnitude larger than previously available public datasets, and poses a signiﬁcant scaling challenge.

Original: Since the test labels for this dataset are (still) hidden, we report performance based on validation accuracies in our experiments.
Revised: Since the test labels for this dataset are (still) hidden, we report performance based on validation accuracies in our experiments.

Original: Implementation and experiment details are in
Revised: Implementation and experiment details are in

Original: Appendix G.
Revised: Appendix G.

Original: To account for the increased scale and difﬁculty of the classiﬁcation task on this dataset, we make a number of changes to our learning setup.
Revised: To account for the increased scale and difﬁculty of the classiﬁcation task on this dataset, we make a number of changes to our learning setup.

Original: First, since we can no longer perform full-graph training due to the sheer size of the graph, we thus adopt the Neighborhood Sampling strategy proposed by Hamilton et al.
Revised: First, since we can no longer perform full-graph training due to the sheer size of the graph, we thus adopt the Neighborhood Sampling strategy proposed by Hamilton et al.

Original: (2017) to sample a small number of central nodes at which our loss is to be applied, and sampling a ﬁxed size neighborhood around them.
Revised: (2017) to sample a small number of central nodes at which our loss is to be applied, and sampling a ﬁxed size neighborhood around them.

Original: Second, we use more expressive
Revised: Second, we use more expressive

Original: Message Passing Neural Networks (Gilmer et al., 2017) as our graph encoders.
Revised: Message Passing Neural Networks (Gilmer et al., 2017) as our graph encoders.

Original: Finally, as we are interested in pushing performance on this competition dataset, we make use of the available labels for representation learning and shift from evaluating on top of a frozen representation, to semi-supervised training by combining both supervised and self-supervised signals at each update step.
Revised: Finally, as we are interested in pushing performance on this competition dataset, we make use of the available labels for representation learning and shift from evaluating on top of a frozen representation, to semi-supervised training by combining both supervised and self-supervised signals at each update step.

Original: We emphasize that these are signiﬁcant changes from the standard small-scale evaluation setup for graph representation learning methods studied previously, and more closely resemble real-world conditions in which these algorithms would be employed.
Revised: We emphasize that these are signiﬁcant changes from the standard small-scale evaluation setup for graph representation learning methods studied previously, and more closely resemble real-world conditions in which these algorithms would be employed.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: In Figure 4, we see that BGRL used as an auxiliary signal is able to learn faster and signiﬁcantly improve ﬁnal performance over fully supervised learning on this challenging task.
Revised: In Figure [Figure] we see that BGRL used as an auxiliary signal is able to learn faster and signiﬁcantly improve ﬁnal performance over fully supervised learning on this challenging task.

Original: Considering the difﬁculty of this task and the small gap in ﬁnal performance between winning entries in the KDD Cup 2021 contest, this is a signiﬁcant improvement.
Revised: Considering the difﬁculty of this task and the small gap in ﬁnal performance between winning entries in the KDD Cup 2021 contest, this is a signiﬁcant improvement.

Original: On the other hand, GRACE-SUBSAMPLING provides much lower beneﬁts over fully supervised learning, possibly due to no longer being able to sample sufﬁciently many negatives over the whole graph.
Revised: On the other hand, GRACE-SUBSAMPLING provides much lower beneﬁts over fully supervised learning, possibly due to no longer being able to sample sufﬁciently many negatives over the whole graph.

Original: Here we used k = 256 negatives, the largest value we were able to run without running out of memory.
Revised: Here we used k = 256 negatives, the largest value we were able to run without running out of memory.

Original: We further show that we can leverage the high scalability of BGRL to make use of the vast amounts of unlabeled data present in the dataset.
Revised: We further show that we can leverage the high scalability of BGRL to make use of the vast amounts of unlabeled data present in the dataset.

Original: Since labeled nodes form only 0.5% of the graph, unlabeled data offers a rich self-supervised signal for learning better representations and ultimately improving performance on the supervised task.
Revised: Since labeled nodes form only 0.5% of the graph, unlabeled data offers a rich self-supervised signal for learning better representations and ultimately improving performance on the supervised task.

Original: In Figure 5, we consider adding some number of unlabeled nodes to each minibatch of data, and examine the effect on performance as this ratio of unlabeled to labeled data in each batch increases.
Revised: In Figure 5, we consider adding some number of unlabeled nodes to each minibatch of data, and examine the effect on performance as this ratio of unlabeled to labeled data in each batch increases.

Original: Thus at each step, we apply the supervised loss on only the labeled nodes in the batch, and BGRL on all nodes.
Revised: Thus at each step, we apply the supervised loss on only the labeled nodes in the batch, and BGRL on all nodes.

Original: Note that a ratio of 0 corresponds to the case where we apply BGRL as an auxiliary loss only to the training nodes, already examined in Figure 4.
Revised: Note that a ratio of 0 corresponds to the case where we apply BGRL as an auxiliary loss only to the training nodes, already examined in Figure 4.

Original: We observe a dramatic increase in both stability and peak performance as we increase this ratio, showing that
Revised: We observe a dramatic increase in both stability and peak performance as we increase this ratio, showing that

Original: BGRL can utilize the unlabeled nodes effectively to shape higher quality representations and prevent early overﬁtting to the supervised signal.
Revised: BGRL can utilize the unlabeled nodes effectively to shape higher quality representations and prevent early overﬁtting to the supervised signal.

Original: This effect shows steady improvement as we increase this ratio from 1x to 10x unlabeled data, where we stop due to resource constraints on running ablations on this large-scale graph - however, this trend may continue to even higher ratios, as the true ratio of unlabeled to labeled nodes present in the graph is 99x.
Revised: This effect shows steady improvement as we increase this ratio from 1x to 10x unlabeled data, where we stop due to resource constraints on running ablations on this large-scale graph - however, this trend may continue to even higher ratios, as the true ratio of unlabeled to labeled nodes present in the graph is 99x.

Original: This result of 73.89% is state-of-the-art for this dataset for the highest single-model performance (i.e., without ensembling) - the OGB baselines report a score of 70.02% (Hu et al., 2021) while the
Revised: This result of 73.89% is state-of-the-art for this dataset for the highest single-model performance (i.e., without ensembling) - the OGB baselines report a score of 70.02% (Hu et al., 2021) while the

Original: KDD Cup 2021 contest ﬁrst place solution reported a score of 73.71% before ensembling.
Revised: KDD Cup 2021 contest ﬁrst place solution reported a score of 73.71% before ensembling.

Original: KDD Cup 2021:
Revised: KDD Cup 2021:

Original: Our solution using BGRL to shape representations, utilizing unlabeled data in conjunction with a supervised signal for semi-supervised learning, was awarded as one of the winners of the MAG240M track at OGB-LSC .
Revised: Our solution using BGRL to shape representations, utilizing unlabeled data in conjunction with a supervised signal for semi-supervised learning, was awarded as one of the winners of the MAG240M track at OGB-LSC(Addanki et al., 2021).

Original: It achieved a ﬁnal position of second overall, achieving 75.19% accuracy on the test set.
Revised: It achieved a ﬁnal position of second overall, achieving 75.19% accuracy on the test set.

Original: The ﬁrst and third place solutions achieved 75.49% and 74.60% respectively.
Revised: The ﬁrst and third place solutions achieved 75.49% and 74.60% respectively.

Original: Although differences in many other factors such as model architectures, feature engineering, ensembling strategies, etc. prevent a direct comparison 3 between these solutions, these results serve as a strong empirical evidence for the effectiveness of BGRL for learning representations on extremely large scale datasets.
Revised: Although differences in many other factors such as model architectures, feature engineering, ensembling strategies, etc. prevent a direct comparison 2 between these solutions, these results serve as a strong empirical evidence for the effectiveness of BGRL for learning representations on extremely large scale datasets.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Early methods in the area relied on random-walk objectives such as DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016).
Revised: Early methods in the area relied on random-walk objectives such as DeepWalk (Perozzi et al., 2014) and node2vec (Grover & Leskovec, 2016).

Original: Even though the graph neural networks (GNNs) inductive bias aligns with these objectives (Wu et al., 2019; Veliˇckovi´c et al., 2019; Kipf & Welling, 2017), composing GNNs and random-walks does not work very well and can even degrade performance (Hamilton et al., 2017).
Revised: Even though the graph neural networks (GNNs) inductive bias aligns with these objectives (Wu et al., 2019; Veli ˇ ckovi ´ c et al., 2019; Kipf & Welling, 2017), composing GNNs and random-walks does not work very well and can even degrade performance (Hamilton et al., 2017).

Original: Earlier combinations of GNNs and self-supervised learning involve Embedding
Revised: Earlier combinations of GNNs and self-supervised learning involve Embedding

Original: Propagation (García-Durán & Niepert, 2017), Variational Graph Autoencoders (Kipf &
Revised: Propagation (García-Durán & Niepert, 2017), Variational Graph Autoencoders (Kipf &

Original: Welling, 2016) and Graph2Gauss (Bojchevski & Günnemann, 2018).
Revised: Welling, 2016) and Graph2Gauss (Bojchevski & Günnemann, 2018).

Original: Hu et al.
Revised: Hu et al.

Original: (2020b) leverages
Revised: (2020b) leverages

Original: BERT (Devlin et al., 2019) for representation learning in graph-structured inputs.
Revised: BERT (Devlin et al., 2019) for representation learning in graph-structured inputs.

Original: Hu et al.
Revised: Hu et al.

Original: (2020b) assumes speciﬁc graph structures and uses feature masking objectives to shape representations.
Revised: (2020b) assumes speciﬁc graph structures and uses feature masking objectives to shape representations.

Original: Recently, contrastive methods effective on images have also been adapted to graphs using GNNs.
Revised: Recently, contrastive methods effective on images have also been adapted to graphs using GNNs.

Original: This includes DGI (Veliˇckovi´c et al., 2019), inspired by Deep InfoMax Hjelm et al.
Revised: This includes DGI (Veli ˇ ckovi ´ c et al., 2019), inspired by Deep InfoMax Hjelm et al.

Original: (2019), which contrasts node-local patches against global graph representations.
Revised: (2019), which contrasts node-local patches against global graph representations.

Original: Next, InfoGraph (Sun et al., 2020)
Revised: Next, InfoGraph (Sun et al., 2020)

Original: modiﬁed DGI ’s pipeline for graph classiﬁcation tasks.
Revised: modiﬁed DGI ’s pipeline for graph classiﬁcation tasks.

Original: GMI Peng et al.
Revised: GMI Peng et al.

Original: (2020) maximizes a notion of graphical mutual information inspired by MINE (Belghazi et al., 2018), allowing for a more ﬁne-grained contrastive loss than DGI ’s. The SimCLR method of Chen et al.
Revised: (2020) maximizes a notion of graphical mutual information inspired by MINE (Belghazi et al., 2018), allowing for a more ﬁnegrained contrastive loss than DGI ’s. The SimCLR method of Chen et al.

Original: (2020a;b) has been specialized for graphs by GRACE and variants such as GCA (Zhu et al.
Revised: (2020a;b) has been specialized for graphs by GRACE and variants such as GCA (Zhu et al.

Original: , 2020b;a) that rely on more complex data-adaptive augmentations.
Revised: , 2020b;a) that rely on more complex dataadaptive augmentations.

Original: GraphCL (You et al., 2020) adapts SimCLR to learn graph-level embeddings using a contrastive objective.
Revised: GraphCL (You et al., 2020) adapts SimCLR to learn graph-level embeddings using a contrastive objective.

Original: MVGRL (Hassani & Khasahmadi, 2020) generalizes CMC (Tian et al., 2020) to graphs.
Revised: MVGRL (Hassani & Khasahmadi, 2020) generalizes CMC (Tian et al., 2020) to graphs.

Original: Graph Barlow Twins (Bielak et al., 2021) presents a method to learn representations by minimizing correlation between different representation dimensions.
Revised: Graph Barlow Twins (Bielak et al., 2021) presents a method to learn representations by minimizing correlation between different representation dimensions.

Original: Concurrent works4 DGB (Che et al., 2020) and SelfGNN (Kefato & Girdzijauskas, 2021), like BGRL , adapt BYOL (Grill et al., 2020) for graph representation learning.
Revised: Concurrent works DGB (Che et al., 2020) and SelfGNN (Kefato & Girdzijauskas, 2021), like BGRL , adapt BYOL (Grill et al., 2020) for graph representation learning.

Original: However, BGRL differs from these works in the following ways:
Revised: However, BGRL differs from these works in the following ways:

Original: • We show that BGRL scales to and attains state-of-the-art results on the very high-data regime on the360GB MAG240M graph with 240 million nodes.
Revised: • We show that BGRL scales to and attains state-of-the-art results on the very high-data regime on the MAG240M dataset.

Original: Theselarge-scale results are unprecedented in the graph self-supervised learning literature and demonstrate a high degree of scalability.
Revised: These results are unprecedented in the graph self-supervised learning literature and demonstrate a high degree of scalability.

Original: • We show that BGRL is effective even when training on sampled subgraphsinstead of the full graph, whereas DGB and SelfGNN only consider full-graph training in small-to-medium scale datasets.
Revised: • We show that BGRL is effective even when trained on sampled subgraphs and not full graph.

Original: • We provide an extensive analysis of the performance-computation trade-off of BGRL versus contrastive methods, showing that BGRL can be more efﬁcient in terms of computation and memory usage as it requires no negative examples, whereas SelfGNN and DGB only consider the metric of task performance.
Revised: • We provide an extensive analysis of the performance-computation trade-off of BGRL versus contrastive methods, showing that BGRL can be more efﬁcient in terms of computation and memory usage as it requires no negative examples.

Original: • We show that BGRL is effective when performing semi-supervised training, providing further gains when leveraging both labeled data and unlabeled data.
Revised: • We show that BGRL is effective when performing semi-supervised training, providing further gains when leveraging both labeled data and unlabeled data.

Original: This is a signiﬁcant result that has not been demonstrated in graph representation learning methods using neural networks prior to our work.
Revised: This is a signiﬁcant result that has not been demonstrated in graph representation learning methods using neural networks prior to our work.

Original: ICLR E THICS S TATEMENT
Revised: ICLR E THICS S TATEMENT

Original: Our contributions are in developing and evaluating a general method for self-supervised representation learning in graphs.
Revised: Our contributions are in developing and evaluating a general method for self-supervised representation learning in graphs.

Original: As such, they may be helpful in applications where obtaining labels can be challenging or expensive, thus enabling newer applications potentially in the direction of positive social good.
Revised: As such, they may be helpful in applications where obtaining labels can be challenging or expensive, thus enabling newer applications potentially in the direction of positive social good.

Original: On the other hand, as an unsupervised pretraining method, there is a risk of practitioners using it for downstream tasks without carefully considering how these embeddings were originally trained, potentially leading to stereotyping or unfair biases.
Revised: On the other hand, as an unsupervised pretraining method, there is a risk of practitioners using it for downstream tasks without carefully considering how these embeddings were originally trained, potentially leading to stereotyping or unfair biases.

Original: Further, since the bootstrapping dynamics of
Revised: Further, since the bootstrapping dynamics of

Original: BGRL are not yet fully understood, there is a higher chance of it being used as a blackbox machine learning method and harmful downstream effects being difﬁcult to diagnose and resolve.
Revised: BGRL are not yet fully understood, there is a higher chance of it being used as a blackbox machine learning method and harmful downstream effects being difﬁcult to diagnose and resolve.

Original: ICLR R EPRODUCIBILITY S TATEMENT
Revised: ICLR R EPRODUCIBILITY S TATEMENT

Original: We believe that the results we report in this paper are reproducible and strengthen our empirical contributions.
Revised: We believe that the results we report in this paper are reproducible and strengthen our empirical contributions.

Original: We have submitted our algorithm implementation and experimental setup, conﬁg, and code for almost all of our experiments as supplementary material.
Revised: We have submitted our algorithm implementation and experimental setup, conﬁg, and code for almost all of our experiments as supplementary material.

Original: Most experiments ﬁnish within 30 minutes on a single V100 GPU, and thus are easy to verify with few resources.
Revised: Most experiments ﬁnish within 30 minutes on a single V100 GPU, and thus are easy to verify with few resources.

Original: In addition, we are providing trained model weights/checkpoints for directly loading and verifying performance without training.
Revised: In addition, we are providing trained model weights/checkpoints for directly loading and verifying performance without training.

Original: Experiments for which code has not been provided are described in detail in the appendices (Appendix E and Appendix F) and should allow for reproduction.
Revised: Experiments for which code has not been provided are described in detail in the appendices (Appendix E and Appendix F) and should allow for reproduction.

Original: Besides this, code for our large-scale MAG240M solution has been open-sourced as part of the KDD
Revised: Besides this, code for our large-scale MAG240M solution has been open-sourced as part of the KDD

Original: Cup 2021 and has been veriﬁed independently by the OGB-LSC contest organizers.
Revised: Cup 2021 and has been veriﬁed independently by the OGB-LSC contest organizers.

Original: A BGRL DOES NOT CONVERGE TO TRIVIAL SOLUTIONS
Revised: A BGRL DOES NOT CONVERGE TO TRIVIAL SOLUTIONS

Original: In Figure 6 we show the BGRL loss curve throughout training for all the datasets considered.
Revised: In Figure 6 we show the BGRL loss curve throughout training for all the datasets considered.

Original: As we see, the loss does not converge to zero, indicating that the training does not result in a trivial solution.
Revised: As we see, the loss does not converge to zero, indicating that the training does not result in a trivial solution.

Original: In Figure 7 we plot the spread of the node embeddings,
Revised: In Figure 7 we plot the spread of the node embeddings,

Original: i
Revised: i

Original: [Figure]
Revised: [Figure]

Original: e., the standard deviation of the representations learned across all nodes, divided by the average norm.
Revised: e., the standard deviation of the representations learned across all nodes, divided by the average norm.

Original: As we see, the embeddings learned across all datasets have a standard deviation that is a similar order of magnitude as the norms of the embeddings themselves, further indicating that the training dynamics do not converge to a constant solution.
Revised: As we see, the embeddings learned across all datasets have a standard deviation that is a similar order of magnitude as the norms of the embeddings themselves, further indicating that the training dynamics do not converge to a constant solution.

Original: Further, Figure 8 shows that the embeddings do not collapse to zero or blow up as training progresses
Revised: Further, Figure 8 shows that the embeddings do not collapse to zero or blow up as training progresses

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: B A BLATIONS ON P ROJECTOR N ETWORK
Revised: B A BLATIONS ON P ROJECTOR N ETWORK

Original: As noted in Section
Revised: As noted in Section

Original: [Figure]
Revised: [Figure]

Original: BGRL does not use a projector network, unlike both BYOL and GRACE .
Revised: BGRL does not use a projector network, unlike both BYOL and GRACE .

Original: Prior works such as GRACE use a projector network to prevent the embeddings from becoming completely invariant to the augmentations used - however in BGRL , the predictor network can serve the same purpose.
Revised: Prior works such as GRACE use a projector network to prevent the embeddings from becoming completely invariant to the augmentations used - however in BGRL , the predictor network can serve the same purpose.

Original: On the other hand, BYOL relies on this for dimensionality reduction, to simplify the task of the predictor p θ , as it is challenging to directly predict very high-dimensional embeddings.
Revised: On the other hand, BYOL relies on this for dimensionality reduction, to simplify the task of the predictor p θ , as it is challenging to directly predict very high-dimensional embeddings.

Original: required for large-scale vision tasks like ImageNet (Deng et al., 2009).
Revised: required for large-scale vision tasks like ImageNet (Deng et al., 2009).

Original: Here we empirically verify that even in our most challenging, large-scale task of MAG240M, the projector network is not needed and only slows down learning.
Revised: Here we empirically verify that even in our most challenging, large-scale task of MAG240M, the projector network is not needed and only slows down learning.

Original: In Figure 9 we can see that adding the projector network leads to both slower learning and a lower ﬁnal performance.
Revised: In Figure 9 we can see that adding the projector network leads to both slower learning and a lower ﬁnal performance.

Original: [Figure]
Revised: [Figure]

Original: C C OMPARISON ON SMALL DATASETS
Revised: C C OMPARISON ON SMALL DATASETS

Original: We perform additional experiments on 4 commonly used small datasets, (Cora, CiteSeer, PubMed, and DBLP) (Sen et al., 2008; Bojchevski & Günnemann, 2017) and show that BGRL ’s bootstrapping mechanism still performs well in the low-data regime, even attaining a new state of the art performance on two of the datasets.
Revised: We perform additional experiments on 4 commonly used small datasets, (Cora, CiteSeer, PubMed, and DBLP) (Sen et al., 2008; Bojchevski & Günnemann, 2017) and show that BGRL ’s bootstrapping mechanism still performs well in the low-data regime, even attaining a new state of the art performance on two of the datasets.

Original: Note that Table 7 reports results averaged over 20 random dataset splits, as has been followed in Zhu et al.
Revised: Note that Table 7 reports results averaged over 20 random dataset splits, as has been followed in Zhu et al.

Original: (2020b), instead of using the standard ﬁxed splits for these datasets which are known to be easy to unreliable for evaluating GNN methods (Shchur et al., 2018).
Revised: (2020b), instead of using the standard ﬁxed splits for these datasets which are known to be easy to unreliable for evaluating GNN methods (Shchur et al., 2018).

Original: [Table]
Revised: [Table]

Original: D G RAPH AUGMENTATION FUNCTIONS
Revised: D G RAPH AUGMENTATION FUNCTIONS

Original: Generating meaningful augmentations is a much less explored problem in graphs than in other domains such as vision.
Revised: Generating meaningful augmentations is a much less explored problem in graphs than in other domains such as vision.

Original: Further, since we work over entire graphs, complex augmentations can be very expensive to compute and will impact all nodes at once.
Revised: Further, since we work over entire graphs, complex augmentations can be very expensive to compute and will impact all nodes at once.

Original: Our contributions are orthogonal to this problem, and we primarily consider only the standard graph augmentation pipeline that has been used in previous works on representation learning (You et al., 2020; Zhu et al., 2020b).
Revised: Our contributions are orthogonal to this problem, and we primarily consider only the standard graph augmentation pipeline that has been used in previous works on representation learning (You et al., 2020; Zhu et al., 2020b).

Original: In particular, we consider two simple graph augmentation functions — node feature masking and edge masking .
Revised: In particular, we consider two simple graph augmentation functions — node feature masking and edge masking .

Original: These augmentations are graph-wise: they do not operate on each node independently, and instead leverage graph topology information through edge masking.
Revised: These augmentations are graph-wise: they do not operate on each node independently, and instead leverage graph topology information through edge masking.

Original: This contrasts with transformations used in BYOL , which operate on each image independently.
Revised: This contrasts with transformations used in BYOL , which operate on each image independently.

Original: First, we generate a single random binary mask of size F , each element of which follows a Bernoulli distribution B (1 − p f ) , and use it to mask features of all nodes in the graph (i.e., all nodes have the same features masked).
Revised: First, we generate a single random binary mask of size F , each element of which follows a Bernoulli distribution B (1 − p f ) , and use it to mask features of all nodes in the graph (i.e., all nodes have the same features masked).

Original: Empirically, we found that performance is similar for using different random masks per node or sharing them, and so we use a single mask for simplicity.
Revised: Empirically, we found that performance is similar for using different random masks per node or sharing them, and so we use a single mask for simplicity.

Original: In addition to this node-level attribute transformation, we also compute a binary mask of size E (where E is the number of edges in the original graph), each element of which follows a Bernoulli distribution B (1 − p e ) , and use it to mask edges in the augmented graph.
Revised: In addition to this node-level attribute transformation, we also compute a binary mask of size E (where E is the number of edges in the original graph), each element of which follows a Bernoulli distribution B (1 − p e ) , and use it to mask edges in the augmented graph.

Original: To compute our ﬁnal augmented graphs, we make use of both augmentation functions with different hyperparameters for each graph, i.e. p f 1 and p e 1 for the ﬁrst view, and p f 2 and p e 2 for the second view.
Revised: To compute our ﬁnal augmented graphs, we make use of both augmentation functions with different hyperparameters for each graph, i.e. p f 1 and p e 1 for the ﬁrst view, and p f 2 and p e 2 for the second view.

Original: Beyond these standard augmentations, in Section 4.1 we also consider more complex adaptive augmentations proposed by prior works (Zhu et al., 2020a) which use various heuristics to mask different features or edges with different probabilities.
Revised: Beyond these standard augmentations, in Section 4.1 we also consider more complex adaptive augmentations proposed by prior works (Zhu et al., 2020a) which use various heuristics to mask different features or edges with different probabilities.

Original: E D ATASET DETAILS WikiCS
Revised: E D ATASET DETAILS WikiCS

Original: This graph is constructed from Wikipedia references, with nodes representing articles about Computer Science and edges representing links between them.
Revised: This graph is constructed from Wikipedia references, with nodes representing articles about Computer Science and edges representing links between them.

Original: Articles are classiﬁed intoclasses based on their subﬁeld, and node features are the average of GloVE (Pennington et al., 2014) embeddings of all words in the article.
Revised: Articles are classiﬁed intoclasses based on their subﬁeld, and node features are the average of GloVE (Pennington et al., 2014) embeddings of all words in the article.

Original: This dataset comes with 20 canonical train/valid/test splits, which we use directly.
Revised: This dataset comes with 20 canonical train/valid/test splits, which we use directly.

Original: Amazon Computers, Amazon Photos
Revised: Amazon Computers, Amazon Photos

Original: These graphs are from the Amazon co-purchase graph (McAuley et al., 2015) with nodes representing products and edges being between pairs of goods frequently purchased together.
Revised: These graphs are from the Amazon co-purchase graph (McAuley et al., 2015) with nodes representing products and edges being between pairs of goods frequently purchased together.

Original: Products are classiﬁed into 10 (for Computers) and 8 (for Photos) classes based on product category, and node features are a bag-of-words representation of a product’s reviews.
Revised: Products are classiﬁed into 10 (for Computers) and 8 (for Photos) classes based on product category, and node features are a bag-of-words representation of a product’s reviews.

Original: We use a random split of the nodes into (10/10/80%) train/validation/test nodes respectively as these datasets do not come with a standard dataset split.
Revised: We use a random split of the nodes into (10/10/80%) train/validation/test nodes respectively as these datasets do not come with a standard dataset split.

Original: Coauthor CS, Coauthor Physics
Revised: Coauthor CS, Coauthor Physics

Original: These graphs are from the Microsoft Academic Graph (Sinha et al., 2015), with nodes representing authors and edges between authors who have co-authored a paper.
Revised: These graphs are from the Microsoft Academic Graph (Sinha et al., 2015), with nodes representing authors and edges between authors who have co-authored a paper.

Original: Authors are classiﬁed into 15 (for CS) and 5 (for Physics) classes based on the author’s research ﬁeld, and node features are a bag-of-words representation of the keywords of an author’s papers.
Revised: Authors are classiﬁed into 15 (for CS) and 5 (for Physics) classes based on the author’s research ﬁeld, and node features are a bag-of-words representation of the keywords of an author’s papers.

Original: We again use a random (10/10/80%) split for these datasets.
Revised: We again use a random (10/10/80%) split for these datasets.

Original: ogbn-arXiv: This is another citation network, where nodes represent CS papers on arXiv indexed by the Microsoft Academic Graph (Sinha et al., 2015).
Revised: ogbn-arXiv: This is another citation network, where nodes represent CS papers on arXiv indexed by the Microsoft Academic Graph (Sinha et al., 2015).

Original: In our experiments, we symmetrize this graph and thus there is an edge between any pair of nodes if one paper has cited the other.
Revised: In our experiments, we symmetrize this graph and thus there is an edge between any pair of nodes if one paper has cited the other.

Original: Papers are classiﬁed into 40 classes based on arXiv subject area.
Revised: Papers are classiﬁed into 40 classes based on arXiv subject area.

Original: The node features are computed as the average word-embedding of all words in the paper, where the embeddings are computed using a skip-gram model (Mikolov et al., 2013) over the entire corpus.
Revised: The node features are computed as the average word-embedding of all words in the paper, where the embeddings are computed using a skip-gram model (Mikolov et al., 2013) over the entire corpus.

Original: PPI
Revised: PPI

Original: is a protein-protein interaction network (Zitnik & Leskovec, 2017; Hamilton et al., 2017), comprised of multiple ( 24 ) graphs each corresponding to different human tissues.
Revised: is a protein-protein interaction network (Zitnik & Leskovec, 2017; Hamilton et al., 2017), comprised of multiple ( 24 ) graphs each corresponding to different human tissues.

Original: We use the standard dataset split as 20 graphs for training, 2 for validation, and 2 for testing.
Revised: We use the standard dataset split as 20 graphs for training, 2 for validation, and 2 for testing.

Original: Each node has 50 features computed from various biological properties.
Revised: Each node has 50 features computed from various biological properties.

Original: This is a multilabel classiﬁcation task, where each node can possess up to 121 labels.
Revised: This is a multilabel classiﬁcation task, where each node can possess up to 121 labels.

Original: F I MPLEMENTATION DETAILS
Revised: F I MPLEMENTATION DETAILS

Original: In all our experiments, we use the AdamW optimizer (Kingma & Ba, 2015; Gugger & Howard, 2018) with weight decay set to 10 − 5 , and all models initialized using Glorot initialization (Glorot & Bengio, 2010).
Revised: In all our experiments, we use the AdamW optimizer (Kingma & Ba, 2015; Gugger & Howard, 2018) with weight decay set to 10 − 5 , and all models initialized using Glorot initialization (Glorot & Bengio, 2010).

Original: The BGRL predictor p θ used to predict the embedding of nodes across views is ﬁxed to be a
Revised: The BGRL predictor p θ used to predict the embedding of nodes across views is ﬁxed to be a

Original: Multilayer Perceptron (MLP) with a single hidden layer.
Revised: Multilayer Perceptron (MLP) with a single hidden layer.

Original: The decay rate τ controlling the rate of updates of the BGRL target parameters φ is initialized to 0 .
Revised: The decay rate τ controlling the rate of updates of the BGRL target parameters φ is initialized to 0 .

Original: 99 and gradually increased to 1 . 

Revised: 99 and gradually increased to 1 .

Original: over the course of training following a cosine schedule.
Revised: over the course of training following a cosine schedule.

Original: Other model architecture and training details vary per dataset and are described further below.
Revised: Other model architecture and training details vary per dataset and are described further below.

Original: The augmentation hyperparameters p f 1 , 2 and p e 1 , 2 are reported below.
Revised: The augmentation hyperparameters p f 1 , 2 and p e 1 , 2 are reported below.

Original: Graph Convolutional Networks Formally, the GCN propagation rule (Kipf & Welling, 2017) for a single layer is as follows,
Revised: Graph Convolutional Networks Formally, the GCN propagation rule (Kipf & Welling, 2017) for a single layer is as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:98)
Revised: where (cid:98)

Original: A =
Revised: A =

Original: A
Revised: A

Original: + I is the adjacency matrix with self-loops, (cid:98) D is the degree matrix, σ is a non-linearity such as ReLU, and W i is a learned weight matrix for the i ’th layer.
Revised: + I is the adjacency matrix with self-loops, (cid:98) D is the degree matrix, σ is a non-linearity such as ReLU, and W i is a learned weight matrix for the i ’th layer.

Original: Mean Pooling Rule Formally, the Mean Pooling (Hamilton et al., 2017) rule for a single layer is given by:
Revised: Mean Pooling Rule Formally, the Mean Pooling (Hamilton et al., 2017) rule for a single layer is given by:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: As proposed by Veliˇckovi´c et al.
Revised: As proposed by Veliˇckovi´c et al.

Original: (2019), our exact encoder in inductive experiments E is a 3 -layer mean-pooling network with skip connections.
Revised: (2019), our exact encoder in inductive experiments E is a 3 -layer mean-pooling network with skip connections.

Original: We use a layer size of 512 and PReLU (He et al., 2015) activation.
Revised: We use a layer size of 512 and PReLU (He et al., 2015) activation.

Original: Thus, we compute:
Revised: Thus, we compute:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Graph Attention Networks The GAT layer (Veliˇckovi´c et al., 2018) consists of a learned matrix
Revised: Graph Attention Networks The GAT layer (Veliˇckovi´c et al., 2018) consists of a learned matrix

Original: W that transforms each node features.
Revised: W that transforms each node features.

Original: We then use self-attention to compute attention coefﬁcient for a pair of nodes i and j as e ij = a ( h i , h j ) .
Revised: We then use self-attention to compute attention coefﬁcient for a pair of nodes i and j as e ij = a ( h i , h j ) .

Original: The attention function a is computed as
Revised: The attention function a is computed as

Original: LeakyReLU ( a [ Wh i ||
Revised: LeakyReLU ( a [ Wh i ||

Original: Wh j ]) , where a is a learned matrix transforming a pair of concatenated attention queries into a single scalar attention logit.
Revised: Wh j ]) , where a is a learned matrix transforming a pair of concatenated attention queries into a single scalar attention logit.

Original: The weight of the edge between nodes i and j is computed as α ij = softmax j ( e ij ) .
Revised: The weight of the edge between nodes i and j is computed as α ij = softmax j ( e ij ) .

Original: We follow the architecture proposed by Veliˇckovi´c et al.
Revised: We follow the architecture proposed by Veliˇckovi´c et al.

Original: (2018), including a 3 -layer GAT model (with the ﬁrst 2 layers consisting of 4 heads of size 256 each and the ﬁnal layer size 512 with 6 output heads), ELU activation (Clevert et al., 2016), and skip-connections in intermediate layers.
Revised: (2018), including a 3 -layer GAT model (with the ﬁrst 2 layers consisting of 4 heads of size 256 each and the ﬁnal layer size 512 with 6 output heads), ELU activation (Clevert et al., 2016), and skip-connections in intermediate layers.

Original: Model architectures As described in Section 4, we use GCN (Kipf & Welling, 2017) encoders in our experiments on the smaller transductive tasks, while on the inductive task of PPI we use
Revised: Model architectures As described in Section 4, we use GCN (Kipf & Welling, 2017) encoders in our experiments on the smaller transductive tasks, while on the inductive task of PPI we use

Original: MeanPooling encoders with residual connections.
Revised: MeanPooling encoders with residual connections.

Original: The BGRL predictor p θ is implemented as a mutilayer perceptron (MLP).
Revised: The BGRL predictor p θ is implemented as a mutilayer perceptron (MLP).

Original: We also used stabilization techniques like batch normalization (Ioffe & Szegedy, 2015), layer normalization (Ba et al., 2016), and weight standardization (Qiao et al., 2019).
Revised: We also used stabilization techniques like batch normalization (Ioffe & Szegedy, 2015), layer normalization (Ba et al., 2016), and weight standardization (Qiao et al., 2019).

Original: The decay rate use for statistics in the batch normalization is ﬁxed to 0.99.
Revised: The decay rate use for statistics in the batch normalization is ﬁxed to 0.99.

Original: We use PReLU activation (He et al., 2015) in all experiments except those using a GAT encoder, where we use the
Revised: We use PReLU activation (He et al., 2015) in all experiments except those using a GAT encoder, where we use the

Original: ELU activation (Clevert et al., 2016).
Revised: ELU activation (Clevert et al., 2016).

Original: In all our models, at each layer including the ﬁnal layer, we apply ﬁrst the batch/layer normalization as applicable, and then the activation function.
Revised: In all our models, at each layer including the ﬁnal layer, we apply ﬁrst the batch/layer normalization as applicable, and then the activation function.

Original: Tabledescribes hyperparameter and architectural details for most of our experimental setups with BGRL .
Revised: Tabledescribes hyperparameter and architectural details for most of our experimental setups with BGRL .

Original: In addition to these standard settings, we perform additional experiments on the PPI dataset using a GAT (Veliˇckovi´c et al., 2018) model as the encoder.
Revised: In addition to these standard settings, we perform additional experiments on the PPI dataset using a GAT (Veliˇckovi´c et al., 2018) model as the encoder.

Original: When using the GAT encoder on PPI, we use 3 attention layers — the ﬁrst two with 4 attention heads of size 256 each, and the ﬁnal withattention heads of size 512, following a very similar model proposed by Veli
Revised: When using the GAT encoder on PPI, we use 3 attention layers — the ﬁrst two with 4 attention heads of size 256 each, and the ﬁnal withattention heads of size 512, following a very similar model proposed by Veli

Original: ˇ ckovi ´ c et al.
Revised: ˇ ckovi ´ c et al.

Original: We concatenate the attention head outputs for the ﬁrst 2 layers, and use the mean for the ﬁnal output.
Revised: We concatenate the attention head outputs for the ﬁrst 2 layers, and use the mean for the ﬁnal output.

Original: We also use the ELU activation (Clevert et al., 2016), and skip connections in the intermediate attention layers, as suggested by Veli
Revised: We also use the ELU activation (Clevert et al., 2016), and skip connections in the intermediate attention layers, as suggested by Veli

Original: ˇ ckovi ´ c et al.
Revised: ˇ ckovi ´ c et al.

Original: [Table]
Revised: [Table]

Original: Augmentation parameters The hyperparameter settings for graph augmentations, as well as the sizes of the embeddings and hidden layers, very closely follow previous work (Zhu et al.
Revised: Augmentation parameters The hyperparameter settings for graph augmentations, as well as the sizes of the embeddings and hidden layers, very closely follow previous work (Zhu et al.

Original: , 2020b;a) on all datasets with the exception of ogbn-arXiv.
Revised: , 2020b;a) on all datasets with the exception of ogbn-arXiv.

Original: On this dataset, since there has not been prior work on applying self-supervised graph learning methods, we provide the hyperparameters we found through a small grid search.
Revised: On this dataset, since there has not been prior work on applying self-supervised graph learning methods, we provide the hyperparameters we found through a small grid search.

Original: Optimization settings We perform full-graph training at each gradient step on all small-scale experiments, with the exception of experiments using GAT encoders on the PPI dataset.
Revised: Optimization settings We perform full-graph training at each gradient step on all small-scale experiments, with the exception of experiments using GAT encoders on the PPI dataset.

Original: Here, due to memory constraints, we perform training with a batch size of 1 graph.
Revised: Here, due to memory constraints, we perform training with a batch size of 1 graph.

Original: Since the PPI dataset consists of multiple smaller, disjoint subgraphs, we do not have to perform any node subsampling at training time.
Revised: Since the PPI dataset consists of multiple smaller, disjoint subgraphs, we do not have to perform any node subsampling at training time.

Original: We use Glorot initialization (Glorot & Bengio, 2010) the AdamW optimizer (Kingma & Ba, 2015;
Revised: We use Glorot initialization (Glorot & Bengio, 2010) the AdamW optimizer (Kingma & Ba, 2015;

Original: Gugger & Howard, 2018) with a base learning rate η base and weight decay set to 10 − 5 .
Revised: Gugger & Howard, 2018) with a base learning rate η base and weight decay set to 10 − 5 .

Original: The learning rate is annealed using a cosine schedule over the course of learning of n total total steps with an initial warmup period of n warmup steps.
Revised: The learning rate is annealed using a cosine schedule over the course of learning of n total total steps with an initial warmup period of n warmup steps.

Original: Hence, the learning rate at step i is computed as
Revised: Hence, the learning rate at step i is computed as

Original: [Equation]
Revised: [Equation]

Original: We ﬁx n total to be 10,000 total steps and n warmup to 1,000 warmup steps, with the exception of experiments on the GAT encoder that requires using a batch size of 1 graph on the PPI dataset.
Revised: We ﬁx n total to be 10,000 total steps and n warmup to 1,000 warmup steps, with the exception of experiments on the GAT encoder that requires using a batch size of 1 graph on the PPI dataset.

Original: In this case, we increase the number of total steps to 20,000 and warmup to 2,000 steps.
Revised: In this case, we increase the number of total steps to 20,000 and warmup to 2,000 steps.

Original: The target network parameters φ are initialized randomly from the same distribution of the online parameters θ but with a different random seed.
Revised: The target network parameters φ are initialized randomly from the same distribution of the online parameters θ but with a different random seed.

Original: The decay parameter τ is also updated using a cosine schedule starting from an initial value of τ base = 0 .
Revised: The decay parameter τ is also updated using a cosine schedule starting from an initial value of τ base = 0 .

Original: and is computed as
Revised: and is computed as

Original: [Equation]
Revised: [Equation]

Original: These annealing schedules for both η and τ follow the procedure used by Grill et al.
Revised: These annealing schedules for both η and τ follow the procedure used by Grill et al.

Original: Frozen linear evaluation of embeddings In the linear evaluation protocol, the ﬁnal evaluation is done by ﬁtting a linear classiﬁer on top of the frozen learned embeddings without ﬂowing any gradients back to the encoder.
Revised: Frozen linear evaluation of embeddings In the linear evaluation protocol, the ﬁnal evaluation is done by ﬁtting a linear classiﬁer on top of the frozen learned embeddings without ﬂowing any gradients back to the encoder.

Original: For the smaller datasets of WikiCS, Amazon Computers/Photos, and Coauthor CS/Physics, we use an (cid:96) 2 -regularized LogisticRegression classiﬁer from Scikit-Learn (Pedregosa et al., 2011) using the ‘liblinear’ solver.
Revised: For the smaller datasets of WikiCS, Amazon Computers/Photos, and Coauthor CS/Physics, we use an (cid:96) 2 -regularized LogisticRegression classiﬁer from Scikit-Learn (Pedregosa et al., 2011) using the ‘liblinear’ solver.

Original: We do a hyperparameter search over the regularization strength to be between { 2 − 10 , 2 − 9 , . . .
Revised: We do a hyperparameter search over the regularization strength to be between { 2 − 10 , 2 − 9 , . . .

Original: For larger PPI and ogbn-arXiv datasets, where the liblinear solver takes too long to converge, we instead perform 100 steps of gradient descent using AdamW with learning rate 0.01, with a smaller hyperparameter search on the weight decay between { 2 − 10 , 2 − 8 , 2 − 6 , . . .
Revised: For larger PPI and ogbn-arXiv datasets, where the liblinear solver takes too long to converge, we instead perform 100 steps of gradient descent using AdamW with learning rate 0.01, with a smaller hyperparameter search on the weight decay between { 2 − 10 , 2 − 8 , 2 − 6 , . . .

Original: In all cases, we (cid:96) 2 -normalize the frozen learned embeddings over the entire graph before ﬁtting the classiﬁer on top.
Revised: In all cases, we (cid:96) 2 -normalize the frozen learned embeddings over the entire graph before ﬁtting the classiﬁer on top.

Original: G MAG240M E XPERIMENT D ETAILS
Revised: G MAG240M E XPERIMENT D ETAILS

Original: Full implementation and experiment code has been open-sourced as part of the KDD Cup 2021 9 .
Revised: Full implementation and experiment code has been open-sourced as part of the KDD Cup 2021.

Original: Key implementation details and hyperparameter descriptions are reproduced below.
Revised: Key implementation details and hyperparameter descriptions are reproduced below.

Original: OGB-LSC MAG240M Dataset: This is a heterogeneous graph introduced for the KDD Cup(Hu et al., 2021), comprised of 121 million academic papers, 122 million authors, and 26 thousand institutions.
Revised: OGB-LSC MAG240M Dataset: This is a heterogeneous graph introduced for the KDD Cup(Hu et al., 2021), comprised of 121 million academic papers, 122 million authors, and 26 thousand institutions.

Original: Papers are represented by 768-dimensional BERT embeddings (Devlin et al., 2019), and the task is to classify arXiv papers into one of 153 categories, where 1% of the paper nodes are from arXiv.
Revised: Papers are represented by 768-dimensional BERT embeddings (Devlin et al., 2019), and the task is to classify arXiv papers into one of 153 categories, where 1% of the paper nodes are from arXiv.

Original: Message Passing Neural Networks encoders: We use a bi-directional version of the standard
Revised: Message Passing Neural Networks encoders: We use a bi-directional version of the standard

Original: MPNN (Gilmer et al., 2017) architectures with 4 message passing steps, a hidden size of 256 at each layer, with node and edge update functions represented by Multilayer Perceptrons (MLPs) withhidden layers of size 512 each.
Revised: MPNN (Gilmer et al., 2017) architectures with 4 message passing steps, a hidden size of 256 at each layer, with node and edge update functions represented by Multilayer Perceptrons (MLPs) withhidden layers of size 512 each.

Original: Node Neighborhood Sampling: Since we can no longer perform full-graph training, we sample a batch size of 1024 central nodes split across 8 devices, and subsample a ﬁxed-size neighborhood for each.
Revised: Node Neighborhood Sampling: Since we can no longer perform full-graph training, we sample a batch size of 1024 central nodes split across 8 devices, and subsample a ﬁxed-size neighborhood for each.

Original: Speciﬁcally, we sample a depth-2 neighborhood with different numbers of neighbors sampled per layer depending on the type (paper, author, institution) of each neighbor.
Revised: Speciﬁcally, we sample a depth-2 neighborhood with different numbers of neighbors sampled per layer depending on the type (paper, author, institution) of each neighbor.

Original: We sample up topapers and 20 authors for each paper; and 40 papers and 10 institutions per author.
Revised: We sample up topapers and 20 authors for each paper; and 40 papers and 10 institutions per author.

Original: Other hyperparamters: We use edge masking probability p e of 0.2 and feature masking probability p f of 0.4 for each augmentation.
Revised: Other hyperparamters: We use edge masking probability p e of 0.2 and feature masking probability p f of 0.4 for each augmentation.

Original: We use a higher decay rate τ , starting at 0.999 and decayed to 1.0 with a cosine schedule.
Revised: We use a higher decay rate τ , starting at 0.999 and decayed to 1.0 with a cosine schedule.

Original: We use AdamW optimizer with a weight decay of 10 − 5 , and a learning rate starting at 0.01 and annealed to 0 over the course of learning with a warmup step equal to 10% the period of learning.
Revised: We use AdamW optimizer with a weight decay of 10 − 5 , and a learning rate starting at 0.01 and annealed to 0 over the course of learning with a warmup step equal to 10% the period of learning.

Original: H M ODEL A BLATIONS ON MAG240M E XPERIMENTS
Revised: H M ODEL A BLATIONS ON MAG240M E XPERIMENTS

Original: In our main experiments on the OGB-LSC MAG240M dataset, we focus on MPNN encoders to (i) achieve high accuracy on this challenging dataset, and (ii) to evaluate the stability of training these more complex models with the BGRL approach.
Revised: In our main experiments on the OGB-LSC MAG240M dataset, we focus on MPNN encoders to (i) achieve high accuracy on this challenging dataset, and (ii) to evaluate the stability of training these more complex models with the BGRL approach.

Original: In this section, we further experiment with simpler
Revised: In this section, we further experiment with simpler

Original: GCN encoders and evaluate the beneﬁts of applying BGRL even on top of these weaker encoders.
Revised: GCN encoders and evaluate the beneﬁts of applying BGRL even on top of these weaker encoders.

Original: We use a 2-layer GCN encoder, with an embedding size of 128 .
Revised: We use a 2-layer GCN encoder, with an embedding size of 128 .

Original: All other settings such as learning rate schedules, augmentation parameters, etc. are unchanged.
Revised: All other settings such as learning rate schedules, augmentation parameters, etc. are unchanged.

Original: In Figure 10, we see that both BGRL and GRACE improve over the performance of a fully supervised approach, with BGRL learning faster and more stably.
Revised: In Figure 10, we see that both BGRL and GRACE improve over the performance of a fully supervised approach, with BGRL learning faster and more stably.

Original: Thus the effectiveness of BGRL even with weaker encoder architectures makes it more applicable in practice.
Revised: Thus the effectiveness of BGRL even with weaker encoder architectures makes it more applicable in practice

Original: [Figure]
Revised: [Figure]

Original: I V ISUALIZATION OF S CALING B EHAVIOR
Revised: I V ISUALIZATION OF S CALING B EHAVIOR

Original: [Figure]
Revised: [Figure]

Original: In this section, we provide the information contained in Table 1 as a scatterplot, to more easily visualize the different scaling properties of BGRL and GRACE .
Revised: In this section, we provide the information contained in Table 1 as a scatterplot, to more easily visualize the different scaling properties of BGRL and GRACE .

Original: We see in Figure 11 that the empirical scaling behavior matches the theoretical predictions in Section 3.
Revised: We see in Figure 11 that the empirical scaling behavior matches the theoretical predictions in Section 3.

Original: Note that we do not show the memory usage of GRACE for the largest dataset, as it runs out of memory here, and we only visualize as a function of the number of nodes in the graph and ignore the number of edges for the purposes of this visualization.
Revised: Note that we do not show the memory usage of GRACE for the largest dataset, as it runs out of memory here, and we only visualize as a function of the number of nodes in the graph and ignore the number of edges for the purposes of this visualization.

Revised: J F ROZEN L INEAR E VALUATION ON MAG240M

Revised: We brieﬂy run experiments evaluating BGRL and GRACE under the frozen evaluation protocol using an

Revised: MLP classiﬁcation layer on the MAG240M dataset.

Revised: We see that although GRACE outperforms BGRL in this setting, both methods perform poorly.

Revised: In particular, they underperform LabelProp (Zhu &

Revised: Ghahramani, 2002; Hu et al., 2021) a simple parameterless, graph-agnostic baseline.

Revised: This, combined with our goal of pushing performance on the competition dataset, motivates our consideration of the semi-supervised learning setting.

Revised: [Figure]