Original: N ODE F EATURE E XTRACTION BY S ELF -S UPERVISED M ULTI - SCALE N EIGHBORHOOD P REDICTION
Revised: N ODE F EATURE E XTRACTION BY S ELF -S UPERVISED M ULTI - SCALE N EIGHBORHOOD P REDICTION

Original: A BSTRACT
Revised: A BSTRACT

Original: Learning on graphs has attracted signiﬁcant attention in the learning community due to numerous real-world applications.
Revised: Learning on graphs has attracted signiﬁcant attention in the learning community due to numerous real-world applications.

Original: In particular, graph neural networks (GNNs), which take numerical node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on various graph-related learning tasks.
Revised: In particular, graph neural networks (GNNs), which take numerical node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on various graph-related learning tasks.

Original: Recent works exploring the correlation between numerical node features and graph structure via self-supervised learning have paved the way for further performance improvements of GNNs.
Revised: Recent works exploring the correlation between numerical node features and graph structure via self-supervised learning have paved the way for further performance improvements of GNNs.

Original: However, methods used for extracting numerical node features from raw data are still graph-agnostic within standard
Revised: However, methods used for extracting numerical node features from raw data are still graph-agnostic within standard

Original: GNN pipelines.
Revised: GNN pipelines.

Original: This practice is sub-optimal as it prevents one from fully utilizing potential correlations between graph topology and node attributes.
Revised: This practice is sub-optimal as it prevents one from fully utilizing potential correlations between graph topology and node attributes.

Original: To mitigate this issue, we propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT).
Revised: To mitigate this issue, we propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT).

Original: GIANT makes use of the eXtreme
Revised: GIANT makes use of the eXtreme

Original: Multi-label Classiﬁcation (XMC) formalism, which is crucial for ﬁne-tuning the language model based on graph information, and scales to large datasets.
Revised: Multi-label Classiﬁcation (XMC) formalism, which is crucial for ﬁne-tuning the language model based on graph information, and scales to large datasets.

Original: We also provide a theoretical analysis that justiﬁes the use of XMC over link prediction and motivates integrating XR-Transformers, a powerful method for solving XMC problems, into the GIANT framework.
Revised: We also provide a theoretical analysis that justiﬁes the use of XMC over link prediction and motivates integrating XR-Transformers, a powerful method for solving XMC problems, into the GIANT framework.

Original: We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets: For
Revised: We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets: For

Original: example, we improve the accuracy of the top-ranked method GAMLP from 68 .
Revised: example, we improve the accuracy of the top-ranked method GAMLP from 68 .

Original: 25% to 69 .
Revised: 25% to 69 .

Original: 67% , SGC from 63 .
Revised: 67% , SGC from 63 .

Original: 29% to 66 .
Revised: 29% to 66 .

Original: 10% and MLP from 47 .
Revised: 10% and MLP from 47 .

Original: 24% to 61 .
Revised: 24% to 61 .

Original: 10% on the ogbn-papers100M dataset by leveraging GIANT.
Revised: 10% on the ogbn-papers100M dataset by leveraging GIANT.

Original: Our implementation is public available 1 .
Revised: Our implementation is public available 1 .

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: The ubiquity of graph-structured data and its importance in solving various real-world problems such as node and graph classiﬁcation have made graph-centered machine learning an important research area (L¨u & Zhou, 2011; Shervashidze et al., 2011; Zhu, 2005).
Revised: The ubiquity of graph-structured data and its importance in solving various real-world problems such as node and graph classiﬁcation have made graph-centered machine learning an important research area (L¨u & Zhou, 2011; Shervashidze et al., 2011; Zhu, 2005).

Original: Graph neural networks (GNNs) offer state-of-the-art performance on many graph learning tasks and have by now become a standard methodology in the ﬁeld (Kipf & Welling, 2017; Hamilton et al., 2017; Velickovic et al., 2018; Chien et al., 2020).
Revised: Graph neural networks (GNNs) offer state-of-the-art performance on many graph learning tasks and have by now become a standard methodology in the ﬁeld (Kipf & Welling, 2017; Hamilton et al., 2017; Velickovic et al., 2018; Chien et al., 2020).

Original: In most such studies, GNNs take graphs with numerical node attributes as inputs and train them with task-speciﬁc labels.
Revised: In most such studies, GNNs take graphs with numerical node attributes as inputs and train them with task-speciﬁc labels.

Original: Recent research has shown that self-supervised learning (SSL) leads to performance improvements in many applications, including graph learning, natural language processing and computer vision.
Revised: Recent research has shown that self-supervised learning (SSL) leads to performance improvements in many applications, including graph learning, natural language processing and computer vision.

Original: [Figure]
Revised: [Figure]

Original: Several SSL approaches have also been successfully used with GNNs (Hu et al., 2020b;
Revised: Several SSL approaches have also been successfully used with GNNs (Hu et al., 2020b;

Original: You et al., 2018; 2020; Hu et al., 2020c; Velickovic et al., 2019; Kipf & Welling, 2016; Deng et al., 2020).
Revised: You et al., 2018; 2020; Hu et al., 2020c; Velickovic et al., 2019; Kipf & Welling, 2016; Deng et al., 2020).

Original: The common idea behind these works is to explore the correlated information provided by the numerical node features and graph topology, which can lead to improved node representations and GNN initialization.
Revised: The common idea behind these works is to explore the correlated information provided by the numerical node features and graph topology, which can lead to improved node representations and GNN initialization.

Original: However, one critical yet neglected issue in the current graph learning literature is how to actually obtain the numerical node features from raw data such as text, images and audio signals.
Revised: However, one critical yet neglected issue in the current graph learning literature is how to actually obtain the numerical node features from raw data such as text, images and audio signals.

Original: As an example, when dealing with raw text features, the standard approach is to apply graph-agnostic methods such as bag-of-words, word2vec (Mikolov et al., 2013) or pre-trained BERT (Devlin et al., 2019)
Revised: As an example, when dealing with raw text features, the standard approach is to apply graph-agnostic methods such as bag-of-words, word2vec (Mikolov et al., 2013) or pre-trained BERT (Devlin et al., 2019)

Original: (As a further example, raw texts of product descriptions are used to construct node features via the bag-of-words model for benchmarking GNNs on the ogbn-products dataset (Hu et al., 2020a;
Revised: (As a further example, raw texts of product descriptions are used to construct node features via the bag-of-words model for benchmarking GNNs on the ogbn-products dataset (Hu et al., 2020a;

Original: Chiang et al., 2019)).
Revised: Chiang et al., 2019)).

Original: The pre-trained BERT language model, as well as convolutional neural networks (CNNs) (Goyal et al., 2019; Kolesnikov et al., 2019), produce numerical features that can signiﬁcantly improve the performance of various downstream learners (Devlin et al., 2019).
Revised: The pre-trained BERT language model, as well as convolutional neural networks (CNNs) (Goyal et al., 2019; Kolesnikov et al., 2019), produce numerical features that can signiﬁcantly improve the performance of various downstream learners (Devlin et al., 2019).

Original: Still, none of these works leverage graph information for actual self-supervision.
Revised: Still, none of these works leverage graph information for actual self-supervision.

Original: Clearly, using graphagnostic methods to extract numerical features is sub-optimal, as correlations between the graph topology and raw features are ignored.
Revised: Clearly, using graphagnostic methods to extract numerical features is sub-optimal, as correlations between the graph topology and raw features are ignored.

Original: Motivated by the recent success of SSL approaches for GNNs, we propose GIANT, an SSL framework that resolves the aforementioned issue of graph-agnostic feature extraction in the standard
Revised: Motivated by the recent success of SSL approaches for GNNs, we propose GIANT, an SSL framework that resolves the aforementioned issue of graph-agnostic feature extraction in the standard

Original: GNN learning pipeline.
Revised: GNN learning pipeline.

Original: Our framework takes raw node attributes and generates numerical node features with graph-structured self-supervision.
Revised: Our framework takes raw node attributes and generates numerical node features with graph-structured self-supervision.

Original: To integrate the graph topology information into language models such as BERT, we also propose a novel SSL task termed neighborhood prediction , which works for both homophilous and heterophilous graphs, and establish connections between neighborhood prediction and the eXtreme Multi-label Classiﬁcation (XMC) problem (Shen et al., 2020; Yu et al., 2022; Chang et al., 2020b).
Revised: To integrate the graph topology information into language models such as BERT, we also propose a novel SSL task termed neighborhood prediction , which works for both homophilous and heterophilous graphs, and establish connections between neighborhood prediction and the eXtreme Multi-label Classiﬁcation (XMC) problem (Shen et al., 2020; Yu et al., 2022; Chang et al., 2020b).

Original: Roughly speaking, the neighborhood of each node can be encoded using binary multi-labels (indicating whether a node is a neighbor or not) and the
Revised: Roughly speaking, the neighborhood of each node can be encoded using binary multi-labels (indicating whether a node is a neighbor or not) and the

Original: BERT model is ﬁne-tuned by successively improving the predicted neighborhoods.
Revised: BERT model is ﬁne-tuned by successively improving the predicted neighborhoods.

Original: This approach allows us to not only leverage the advanced solvers for the XMC problem and address the issue of graph-agnostic feature extraction, but also to perform a theoretical study of the XMC problem and determine its importance in the context of graph-guided SSL.
Revised: This approach allows us to not only leverage the advanced solvers for the XMC problem and address the issue of graph-agnostic feature extraction, but also to perform a theoretical study of the XMC problem and determine its importance in the context of graph-guided SSL.

Original: Throughout the work, we focus on raw texts as these are the most common data used for largescale graph benchmarking.
Revised: Throughout the work, we focus on raw texts as these are the most common data used for largescale graph benchmarking.

Original: Examples include titles/abstracts in citation networks and product descriptions in co-purchase networks.
Revised: Examples include titles/abstracts in citation networks and product descriptions in co-purchase networks.

Original: To solve our proposed self-supervised XMC task, we adopt the state-of-the-art XR-Transformer method (Zhang et al., 2021a).
Revised: To solve our proposed self-supervised XMC task, we adopt the state-of-the-art XR-Transformer method (Zhang et al., 2021a).

Original: By using the encoder from the
Revised: By using the encoder from the

Original: XR-Transformer pre-trained with GIANT, we obtain informative numerical node features which consistently boost the performance of GNNs on downstream tasks.
Revised: XR-Transformer pre-trained with GIANT, we obtain informative numerical node features which consistently boost the performance of GNNs on downstream tasks.

Original: Notably, GIANT signiﬁcantly improves state-of-the-art methods for node classiﬁcation tasks described on the Open Graph Benchmark (OGB)
Revised: Notably, GIANT signiﬁcantly improves state-of-the-art methods for node classiﬁcation tasks described on the Open Graph Benchmark (OGB)

Original: (Hu et al., 2020a)
Revised: (Hu et al., 2020a)

Original: leaderboard on three large-scale graph datasets, with absolute improvements in accuracy roughly 1 . 5% for the ﬁrst-ranked methods, 3% for standard GNNs and 14% for multilayer perceptron (MLP).
Revised: leaderboard on three large-scale graph datasets, with absolute improvements in accuracy roughly 1 . 5% for the ﬁrst-ranked methods, 3% for standard GNNs and 14% for multilayer perceptron (MLP).

Original: GIANT coupled with XRTransformer is also highly scalable and can be combined with other downstream learning methods.
Revised: GIANT coupled with XRTransformer is also highly scalable and can be combined with other downstream learning methods.

Original: Our contributions may be summarized as follows.
Revised: Our contributions may be summarized as follows.

Original: We identify the issue of graph-agnostic feature extraction in standard GNN pipelines and propose a new GIANT self-supervised framework as a solution to the problem.
Revised: We identify the issue of graph-agnostic feature extraction in standard GNN pipelines and propose a new GIANT self-supervised framework as a solution to the problem.

Original: We introduce a new approach to  numerical feature extraction supervised by graph information based on the idea of neighborhood prediction.
Revised: We introduce a new approach to extract numerical features by graph information based on the idea of neighborhood prediction.

Original: The gist of the approach is to use neighborhood prediction within a language model such as BERT to guide the process of ﬁne-tuning the features.
Revised: The gist of the approach is to use neighborhood prediction within a language model such as BERT to guide the process of ﬁne-tuning the features.

Original: Unlike link-prediction, neighborhood prediction resolves problems associated with heterophilic graphs.
Revised: Unlike linkprediction, neighborhood prediction resolves problems associated with heterophilic graphs.

Original: We establish pertinent connections between neighborhood prediction and the XMC problem by noting that neighborhoods of individual nodes can be encoded by binary vectors which may be interpreted as multi-labels.
Revised: We establish pertinent connections between neighborhood prediction and the XMC problem by noting that neighborhoods of individual nodes can be encoded by binary vectors which may be interpreted as multi-labels.

Original: This allows for performing neighborhood prediction via XR-Transformers, especially designed to solve XMC problems at scale.
Revised: This allows for performing neighborhood prediction via XR-Transformers, especially designed to solve XMC problems at scale.

Original: We demonstrate through extensive experiments that GIANT consistently improves the performance of tested GNNs on downstream tasks by large margins.
Revised: We demonstrate through extensive experiments that GIANT consistently improves the performance of tested GNNs on downstream tasks by large margins.

Original: We also report new state-of-the-art results on the OGB leaderboard, including absolute improvements in accuracy roughly 1 . 5% compared to the top-ranked method, 3% for standard GNNs and 14% for multilayer perceptron (MLP).
Revised: We also report new state-of-the-art results on the OGB leaderboard, including absolute improvements in accuracy roughly 1 . 5% compared to the top-ranked method, 3% for standard GNNs and 14% for multilayer perceptron (MLP).

Original: More precisely, we improve the accuracy of the top-ranked method GAMLP (Zhang et al., 2021b) from 68 .
Revised: More precisely, we improve the accuracy of the top-ranked method GAMLP (Zhang et al ., 2021b) from 68 .

Original: 25% to 69 .
Revised: 25% to 69 .

Original: 67% , SGC (Wu et al., 2019) from 63 .
Revised: 67% , SGC (Wu et al., 2019) from 63 .

Original: 29% to 66 .
Revised: 29% to 66 .

Original: 10% and MLP from 47 .
Revised: 10% and MLP from 47 .

Original: 24% to 61 .
Revised: 24% to 61 .

Original: 10% on the ogbn-papers100M dataset
Revised: 10% on the ogbn-papers100M dataset.

Original: We present a new theoretical analysis that veriﬁes the beneﬁts of key components in XRTransformers on our neighborhood prediction task.
Revised: We present a new theoretical analysis that veriﬁes the beneﬁts of key components in XRTransformers on our neighborhood prediction task.

Original: This analysis also further improves our understanding of XR-Transformers and the XMC problem.
Revised: This analysis also further improves our understanding of XR-Transformers and the XMC problem.

Original: Due to the space limitation, all proofs are deferred to the Appendix.
Revised: Due to the space limitation, all proofs are deferred to the Appendix.

Original:  B ACKGROUND AND RELATED WORK
Revised:  B ACKGROUND AND RELATED WORK

Original: General notation.
Revised: General notation.

Original: Throughout the paper, we use bold capital letters such as A to denote matrices.
Revised: Throughout the paper, we use bold capital letters such as A to denote matrices.

Original: We use A i for the i -th row of the matrix and A ij for its entry in row i and column j .
Revised: We use A i for the i -th row of the matrix and A ij for its entry in row i and column j .

Original: We reserve bold lowercase letters such as a for vectors.
Revised: We reserve bold lowercase letters such as a for vectors.

Original: The symbol I denotes the identity matrix while 1 denotes the all-ones vector.
Revised: The symbol I denotes the identity matrix while 1 denotes the all-ones vector.

Original: We use o ( · ) , O ( · ) , ω ( · ) , Θ( · ) in the standard manner.
Revised: We use o ( · ) , O ( · ) , ω ( · ) , Θ( · ) in the standard manner.

Original: SSL in GNNs.
Revised: SSL in GNNs.

Original: SSL is a topic of substantial interest due to its potential for improving the performance of GNNs on various tasks.
Revised: SSL is a topic of substantial interest due to its potential for improving the performance of GNNs on various tasks.

Original: Exploiting the correlation between node features and the graph structure is known to lead to better node representations or GNN initialization (Hu et al., 2020b;
Revised: Exploiting the correlation between node features and the graph structure is known to lead to better node representations or GNN initialization (Hu et al., 2020b;

Original: You et al., 2018; 2020; Hu et al., 2020c).
Revised: You et al., 2018; 2020; Hu et al., 2020c).

Original: Several methods have been proposed for improving node representations, including (variational) graph autoencoders (Kipf & Welling, 2016), Deep Graph
Revised: Several methods have been proposed for improving node representations, including (variational) graph autoencoders (Kipf & Welling, 2016), Deep Graph

Original: Infomax (Velickovic et al., 2019) and GraphZoom (Deng et al., 2020).
Revised: Infomax (Velickovic et al., 2019) and GraphZoom (Deng et al., 2020).

Original: For more information, the interested reader is referred to a survey of SSL GNNs (Xie et al., 2021).
Revised: For more information, the interested reader is referred to a survey of SSL GNNs (Xie et al., 2021).

Original: While these methods can be used as SSL modules in GNNs (Figure 1), it is clear that they do not solve the described graph agnostics issue in the standard GNN pipeline.
Revised: While these methods can be used as SSL modules in GNNs (Figure 1), it is clear that they do not solve the described graph agnostics issue in the standard GNN pipeline.

Original: Furthermore, as the above described SSL GNNs modules and other pre-processing and post-processing methods for GNNs such as C&S (Huang et al., 2021) and FLAG (Kong et al., 2020) in general improve graph learners, it is worth pointing out that they can be naturally be integrated into the GIANT framework.
Revised: Furthermore, as the above described SSL GNNs modules and other pre-processing and post-processing methods for GNNs such as C&S (Huang et al., 2021) and FLAG (Kong et al., 2020) in general improve graph learners, it is worth pointing out that they can be naturally be integrated into the GIANT framework.

Original: This topic is left as a future work.
Revised: This topic is left as a future work.

Original: The XMC problem, PECOS and XR-Transformer.
Revised: The XMC problem, PECOS and XR-Transformer.

Original: The XMC problem can be succinctly formulated as follows:
Revised: The XMC problem can be succinctly formulated as follows:

Original: We are given a training set { T i , y i } n i =1 , where T i ∈
Revised: We are given a training set { T i , y i } n i =1 , where T i ∈

Original: D is the i th input text instance and y i ∈ { 0 , 1 } L is the target multi-label from an extremely large collection of labels.
Revised: D is the i th input text instance and y i ∈ { 0 , 1 } L is the target multi-label from an extremely large collection of labels.

Original: The goal is to learn a function f : D×
Revised: The goal is to learn a function f : D×

Original: [ L ] (cid:55)→ R , where f ( T, l ) captures the relevance between the input text T and the label l .
Revised: [ L ] (cid:55)→ R , where f ( T, l ) captures the relevance between the input text T and the label l .

Original: The XMC problem is of importance in many real-world applications (Jiang et al., 2021;
Revised: The XMC problem is of importance in many real-world applications (Jiang et al., 2021;

Original: Ye et al., 2020):
Revised: Ye et al., 2020):

Original: For example, in E-commerce dynamic search advertising, XMC arises when trying to ﬁnd a “good” mapping from items to bid queries on the market (Prabhu et al., 2018;
Revised: For example, in E-commerce dynamic search advertising, XMC arises when trying to ﬁnd a “good” mapping from items to bid queries on the market (Prabhu et al., 2018;

Original: Prabhu &
Revised: Prabhu &

Original: Varma, 2014).
Revised: Varma, 2014).

Original: In open-domain question answering, XMC problems arise when trying to map ques- tions to “evidence” passages containing the answers (Chang et al., 2020a; Lee et al., 2019).
Revised: In open-domain question answering, XMC problems arise when trying to map ques- tions to “evidence” passages containing the answers (Chang et al., 2020a; Lee et al., 2019).

Original: Many methods for the XMC problem leverage hierarchical clustering approaches for labels (Prabhu et al., 2018; You et al., 2019).
Revised: Many methods for the XMC problem leverage hierarchical clustering approaches for labels (Prabhu et al., 2018; You et al., 2019).

Original: This organizational structure allows one to handle potentially enormous numbers of labels, such as used by PECOS (Yu et al., 2022).
Revised: This organizational structure allows one to handle potentially enormous numbers of labels, such as used by PECOS (Yu et al., 2022).

Original: The key is to take advantage of the correlations among labels within the hierarchical clustering.
Revised: The key is to take advantage of the correlations among labels within the hierarchical clustering.

Original: In our approach, we observe that the multi-labels correspond to neighborhoods of nodes in the given graph.
Revised: In our approach, we observe that the multi-labels correspond to neighborhoods of nodes in the given graph.

Original: Neighborhoods have to be predicted using the textual information in order to best match the a priori given graph topology.
Revised: Neighborhoods have to be predicted using the textual information in order to best match the a priori given graph topology.

Original: We use the state-of-the-art XR-Transformer (Zhang et al., 2021a) method for solving the XMC problem to achieve this goal.
Revised: We use the state-of-the-art XR-Transformer (Zhang et al., 2021a) method for solving the XMC problem to achieve this goal.

Original: The high-level idea is to ﬁrst cluster the output labels, and then learn the instance-to-cluster “matchers” (please refer to Figure 2).
Revised: The high-level idea is to ﬁrst cluster the output labels, and then learn the instance-to-cluster “matchers” (please refer to Figure 2).

Original: Note that many other methods have used
Revised: Note that many other methods have used

Original: PECOS (including XR-Transformers) for solving large-scale real-world learning problems (Etter et al., 2022; Liu et al., 2021; Chang et al., 2020b; Baharav et al., 2021; Chang et al., 2021;
Revised: PECOS (including XR-Transformers) for solving large-scale real-world learning problems (Etter et al., 2022; Liu et al., 2021; Chang et al., 2020b; Baharav et al., 2021; Chang et al., 2021;

Original: Yadav et al., 2021; Sen et al., 2021), but not in the context of self-supervised numerical feature extraction as done in our work.
Revised: Yadav et al., 2021; Sen et al., 2021), but not in the context of self-supervised numerical feature extraction as done in our work.

Original: GNNs with raw text data.
Revised: GNNs with raw text data.

Original: It is conceptually possible to jointly train BERT and GNNs in an end-to-end fashion, which could potentially resolve the issue of being graph agnostic in the standard pipeline.
Revised: It is conceptually possible to jointly train BERT and GNNs in an end-to-end fashion, which could potentially resolve the issue of being graph agnostic in the standard pipeline.

Original: However, the excessive model complexity of BERT makes such a combination practically prohibitive due to GPU memory limitations.
Revised: However, the excessive model complexity of BERT makes such a combination practically prohibitive due to GPU memory limitations.

Original: Furthermore, it is nontrivial to train this combination of methods with arbitrary mini-batch sizes (Chiang et al., 2019; Zeng et al., 2020).
Revised: Furthermore, it is nontrivial to train this combination of methods with arbitrary mini-batch sizes (Chiang et al., 2019; Zeng et al., 2020).

Original: In contrast, the XRTransformer architecture naturally supports mini-batch training and scales well (Jiang et al., 2021).
Revised: In contrast, the XRTransformer architecture naturally supports mini-batch training and scales well (Jiang et al., 2021).

Original: Hence, our GIANT method uses XR-Transformers instead of combinations of BERT and GNNs.
Revised: Hence, our GIANT method uses XR-Transformers instead of combinations of BERT and GNNs.

Original: To the best of our knowledge, we are aware of only one prior work that uses raw text inputs for node classiﬁcation problem (Zhang et al., 2020), but it still follows the standard pipeline described in
Revised: To the best of our knowledge, we are aware of only one prior work that uses raw text inputs for node classiﬁcation problem (Zhang et al., 2020), but it still follows the standard pipeline described in

Original: Figure 1.
Revised: Figure 1.

Original: Some other works apply GNNs on texts and for document classiﬁcation, where the actual graphs are constructed based on the raw text.
Revised: Some other works apply GNNs on texts and for document classiﬁcation, where the actual graphs are constructed based on the raw text.

Original: This is clearly not the focus of this work (Yao et al., 2019; Huang et al., 2019; Zhang & Zhang, 2020;
Revised: This is clearly not the focus of this work (Yao et al., 2019; Huang et al., 2019; Zhang & Zhang, 2020;

Original: Liu et al., 2020).
Revised: Liu et al., 2020).

Original:  M ETHODS
Revised:  M ETHODS

Original: Our goal is to resolve the issue of graph-agnostic numerical feature extraction for standard GNN learning pipelines.
Revised: Our goal is to resolve the issue of graph-agnostic numerical feature extraction for standard GNN learning pipelines.

Original: Although our interest lies in raw text data, as already pointed out, the proposed methodology can be easily extended to account for other types of raw data and corresponding feature extraction methods.
Revised: Although our interest lies in raw text data, as already pointed out, the proposed methodology can be easily extended to account for other types of raw data and corresponding feature extraction methods.

Original: To this end, consider a large-scale graph G with node set V = { 1 , 2 , . .
Revised: To this end, consider a large-scale graph G with node set V = { 1 , 2 , . .

Original: . , n } and adjacency matrix
Revised: . , n } and adjacency matrix

Original: A ∈ { 0 , 1 } n × n .
Revised: A ∈ { 0 , 1 } n × n .

Original: Each node i is associated with some raw text, which we denote by T i .
Revised: Each node i is associated with some raw text, which we denote by T i .

Original: The language model is treated as an encoder Φ that maps the raw text T i to numerical node feature
Revised: The language model is treated as an encoder Φ that maps the raw text T i to numerical node feature

Original: X i ∈ R d .
Revised: X i ∈ R d .

Original: Key to our SSL approach is the task of neighborhood prediction , which aims to determine the neighborhood
Revised: Key to our SSL approach is the task of neighborhood prediction , which aims to determine the neighborhood

Original: A i from T i .
Revised: A i from T i .

Original: The neighborhood vector A i can be viewed as a target multi-label y i for node i , where we have L = n labels.
Revised: The neighborhood vector A i can be viewed as a target multi-label y i for node i , where we have L = n labels.

Original: Hence, neighborhood prediction represents an instance of the XMC problem, which we solve by leveraging XR-Transformers.
Revised: Hence, neighborhood prediction represents an instance of the XMC problem, which we solve by leveraging XR-Transformers.

Original: The trained encoder in a
Revised: The trained encoder in an

Original: XR-Transformer generates informative numerical node features, which can then be used further in downstream tasks, the SSL GNNs module and for GNN pre-training.
Revised: XR-Transformer generates informative numerical node features, which can then be used further in downstream tasks, the SSL GNNs module and for GNN pre-training.

Original: Detailed description regarding the use of XR-Transformers for Neighborhood Prediction.
Revised: Detailed description regarding the use of XR-Transformers for Neighborhood Prediction.

Original: The most straightforward instance of the XMC problem is the one-versus-all (OVA) model, which can be formalized as f ( T, l ) =
Revised: The most straightforward instance of the XMC problem is the one-versus-all (OVA) model, which can be formalized as f ( T, l ) =

Original: w Tl Φ( T ); l ∈ [ L ] , where W =
Revised: w Tl Φ( T ); l ∈ [ L ] , where W =

Original: [ w 1 , . .
Revised: [ w 1 , . .

Original: . , w L ] ∈ R d × L are weight vectors and Φ : D (cid:55)→ R d is the encoder that maps T to a d -dimensional feature vector.
Revised: . , w L ] ∈ R d × L are weight vectors and Φ : D (cid:55)→ R d is the encoder that maps T to a d -dimensional feature vector.

Original: OVA can be a deterministic model such as bag-of-words, the Term Frequency-Inverse Document Frequency (TFIDF) model or some other model with learnable parameters, such as XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019b).
Revised: OVA can be a deterministic model such as bag-of-words, the Term Frequency-Inverse Document Frequency (TFIDF) model or some other model with learnable parameters, such as XLNet (Yang et al., 2019) and RoBERTa (Liu et al., 2019b).

Original: We choose to work with pre-trained BERT (Devlin et al., 2019).
Revised: We choose to work with pre-trained BERT (Devlin et al., 2019).

Original: Also, one can change Φ according to the type of input data format (i.e., CNNs for images).
Revised: Also, one can change Φ according to the type of input data format (i.e., CNNs for images).

Original: Despite their simple formulation, it is known Chang et al.
Revised: Despite their simple formulation, it is known Chang et al.

Original: (2020b) that ﬁne-tuning transformer models directly on large output spaces can be prohibitively complex.
Revised: (2020b) that ﬁne-tuning transformer models directly on large output spaces can be prohibitively complex.

Original: For neighborhood prediction, L = n , and the graphs encountered may have millions of nodes.
Revised: For neighborhood prediction, L = n , and the graphs encountered may have millions of nodes.

Original: Hence, we need a more scalable approach to training Transformers.
Revised: Hence, we need a more scalable approach to training Transformers.

Original: As part of an XR-Transformer, one builds a hierarchical label clustering tree based on the label features Z ∈ R L × d ; Z is based on Positive Instance Feature Aggregation
Revised: As part of an XR-Transformer, one builds a hierarchical label clustering tree based on the label features Z ∈ R L × d ; Z is based on Positive Instance Feature Aggregation

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Note that for neighborhood prediction, the above expression represents exactly one step of a graph convolution with node features Ψ( T i ) , followed by a norm normalization; here, Ψ( · ) denotes some text vectorizer such as bag-of-words or TFIDF.
Revised: Note that for neighborhood prediction, the above expression represents exactly one step of a graph convolution with node features Ψ( T i ) , followed by a norm normalization; here, Ψ( · ) denotes some text vectorizer such as bag-of-words or TFIDF.

Original: In the next step, XR-Transformer uses balanced kmeans to recursively partition label sets and generate the hierarchical label cluster tree in a top-down fashion.
Revised: In the next step, XR-Transformer uses balanced kmeans to recursively partition label sets and generate the hierarchical label cluster tree in a top-down fashion.

Original: This step corresponds to Step 1 in Figure 2.
Revised: This step corresponds to Step 1 in Figure 2.

Original: Note that at each intermediate level, it learns a matcher to ﬁnd the most relevant clusters, as illustrated in Step 2 of Figure 2.
Revised: Note that at each intermediate level, it learns a matcher to ﬁnd the most relevant clusters, as illustrated in Step 2 of Figure 2.

Original: By leveraging the label hierarchy deﬁned by the cluster tree, the XR-Transformer can train the model on multi-resolution objectives.
Revised: By leveraging the label hierarchy deﬁned by the cluster tree, the XR-Transformer can train the model on multi-resolution objectives.

Original: Multi-resolution learning has been used in many different contexts, including computer vision (Lai et al., 2017; Karras et al., 2018; 2019;
Revised: Multi-resolution learning has been used in many different contexts, including computer vision (Lai et al., 2017; Karras et al., 2018; 2019;

Original: Pedersoli et al., 2015), meta-learning (Liu et al., 2019a), but has only recently been applied to the XMC problem as part of PECOS and XRTransformers.
Revised: Pedersoli et al., 2015), meta-learning (Liu et al., 2019a), but has only recently been applied to the XMC problem as part of PECOS and XRTransformers.

Original: For neighborhood prediction, multi-resolution amounts to generating a hierarchy of coarse-to-ﬁne views of neighborhoods.
Revised: For neighborhood prediction, multi-resolution amounts to generating a hierarchy of coarse-to-ﬁne views of neighborhoods.

Original: The only line of work in self-supervised graph learning that somewhat resembles this approach is GraphZoom (Deng et al., 2020), in so far that it applies SSL on coarsened graphs.
Revised: The only line of work in self-supervised graph learning that somewhat resembles this approach is GraphZoom (Deng et al., 2020), in so far that it applies SSL on coarsened graphs.

Original: Nevertheless, the way in which we perform coarsening is substantially different; furthermore, GraphZoom still falls into the standard GNN pipeline category depicted in Figure 1.
Revised: Nevertheless, the way in which we perform coarsening is substantially different; furthermore, GraphZoom still falls into the standard GNN pipeline category depicted in Figure 1.

Original:  T HEORETICAL ANALYSIS
Revised:  T HEORETICAL ANALYSIS

Original: We also provide theoretical evidence in support of using each component of our proposed learning framework.
Revised: We also provide theoretical evidence in support of using each component of our proposed learning framework.

Original: First, we show that self-supervised neighborhood prediction is better suited to the task at hand than standard link prediction.
Revised: First, we show that self-supervised neighborhood prediction is better suited to the task at hand than standard link prediction.

Original: More speciﬁcally, we show that the standard design criteria in self-supervised link prediction tasks are biased towards graph homophily assumptions
Revised: More speciﬁcally, we show that the standard design criteria in self-supervised link prediction tasks are biased towards graph homophily assumptions

Original: McPherson et al., 2001; Klicpera et al., 2018).
Revised: McPherson et al., 2001; Klicpera et al., 2018).

Original: In contrast, our self-supervised neighborhood prediction model works for both homophilic and heterophilic graphs.
Revised: In contrast, our self-supervised neighborhood prediction model works for both homophilic and heterophilic graphs.

Original: This universality property is crucial for the robustness of graph learning methods, especially in relationship to GNNs (Chien et al., 2020).
Revised: This universality property is crucial for the robustness of graph learning methods, especially in relationship to GNNs (Chien et al., 2020).

Original: Second, we demonstrate the beneﬁts of using PIFA embeddings and clustering in XR-Transformers for graph-guided numerical feature extraction.
Revised: Second, we demonstrate the beneﬁts of using PIFA embeddings and clustering in XR-Transformers for graph-guided numerical feature extraction.

Original: Our analysis is based on the contextual stochastic block model (cSBM)
Revised: Our analysis is based on the contextual stochastic block model (cSBM)

Original: (Deshpande et al., 2018), which was also used in Chien et al.
Revised: (Deshpande et al., 2018), which was also used in Chien et al.

Original: (2020) for testing the GPR-GNN framework and in Baranwal et al.
Revised: (2020) for testing the GPR-GNN framework and in Baranwal et al.

Original: (2021) for establishing the utility of graph convolutions for node classiﬁcation.
Revised: (2021) for establishing the utility of graph convolutions for node classiﬁcation.

Original: Link versus neighborhood prediction.
Revised: Link versus neighborhood prediction.

Original: One standard SSL task on graphs is link prediction, which aims to ﬁnd an entry in the adjacency matrix according to
Revised: One standard SSL task on graphs is link prediction, which aims to ﬁnd an entry in the adjacency matrix according to

Original: [Equation]
Revised: [Equation]

Original: Here, the function Similarity ( x , y ) is a measure of similarity of two vectors, x and y .
Revised: Here, the function Similarity ( x , y ) is a measure of similarity of two vectors, x and y .

Original: The most frequently used choice for the function is the inner product of two input vectors followed by a sigmoid function.
Revised: The most frequently used choice for the function is the inner product of two input vectors followed by a sigmoid function.

Original: However, this type of design implicitly relies on the homophily assumption:
Revised: However, this type of design implicitly relies on the homophily assumption:

Original: Nodes with similar node representations are more likely to have links.
Revised: Nodes with similar node representations are more likely to have links.

Original: It has been shown in Pei et al.
Revised: It has been shown in Pei et al.

Original: Chien et al.
Revised: Chien et al.

Original: Zhu et al.
Revised: Zhu et al.

Original: ; Lim et al.
Revised: ; Lim et al.

Original: (2021) that there are real-world graph datasets that violate the homophily assumption and on which many GNN architectures fail.
Revised: (2021) that there are real-world graph datasets that violate the homophily assumption and on which many GNN architectures fail.

Original: A simple example that shows how SSL link prediction may fail is presented in Figure 3.
Revised: A simple example that shows how SSL link prediction may fail is presented in Figure 3.

Original: Nodes of the same color share the same features (these are for simplicity represented as numerical values).
Revised: Nodes of the same color share the same features (these are for simplicity represented as numerical values).

Original: Clearly, no matter what encoder Φ we have, the similarity of node features for nodes of the same color is the highest.
Revised: Clearly, no matter what encoder Φ we have, the similarity of node features for nodes of the same color is the highest.

Original: However, there is no edge between nodes of the same color, hence the standard methodology of link prediction based on homophily assumption fails to work for this simple heterophilous graph.
Revised: However, there is no edge between nodes of the same color, hence the standard methodology of link prediction based on homophily assumption fails to work for this simple heterophilous graph.

Original: In order to ﬁx this issue, we use a different modeling assumption, stated below.
Revised: In order to ﬁx this issue, we use a different modeling assumption, stated below.

Original: Assumption 4.1.
Revised: Assumption 4.1.

Original: Nodes with similar node features have similar “structural roles” in the graph.
Revised: Nodes with similar node features have similar “structural roles” in the graph.

Original: In our study, we equate “structure” with the 1-hop neighborhood of a node (i.e., the row of the adjacency matrix indexed by the underlying node).
Revised: In our study, we equate “structure” with the 1-hop neighborhood of a node (i.e., the row of the adjacency matrix indexed by the underlying node).

Original: The above assumption is in alignment with our XMC problem assumptions, where nodes with a small perturbation in their raw text should be mapped to a similar multi-label.
Revised: The above assumption is in alignment with our XMC problem assumptions, where nodes with a small perturbation in their raw text should be mapped to a similar multi-label.

Original: Our assumption is more general then the standard homophily assumption; it is also clear that there exists a perfect mapping from node features to their neighborhoods for the example in Figure 3.
Revised: Our assumption is more general then the standard homophily assumption; it is also clear that there exists a perfect mapping from node features to their neighborhoods for the example in Figure 3.

Original: Hence, neighborhood prediction appears to be a more suitable SSL approach than SSL link prediction for graph-guided feature extraction.
Revised: Hence, neighborhood prediction appears to be a more suitable SSL approach than SSL link prediction for graph-guided feature extraction.

Original: Analysis of key components in XR-Transformers.
Revised: Analysis of key components in XR-Transformers.

Original: In the original XR-Transformer work (Zhang et al., 2021a), the authors argued that one needs to perform clustering of the multi-label space in order to resolve scarce training instances in XMC.
Revised: In the original XR-Transformer work (Zhang et al., 2021a), the authors argued that one needs to perform clustering of the multi-label space in order to resolve scarce training instances in XMC.

Original: They also empirically showed that directly ﬁnetuning language models on extremely large output spaces is prohibitive.
Revised: They also empirically showed that directly ﬁnetuning language models on extremely large output spaces is prohibitive.

Original: Furthermore, they empirically established that constructing clusters based on PIFA embedding with TFIDF features gives the best performance.
Revised: Furthermore, they empirically established that constructing clusters based on PIFA embedding with TFIDF features gives the best performance.

Original: However, no theoretical evidence was given in support of this approach to solving the XMC problem.
Revised: However, no theoretical evidence was given in support of this approach to solving the XMC problem.

Original: We next leverage recent advances in graph learning to analytically characterize the beneﬁts of using XR-Transformers.
Revised: We next leverage recent advances in graph learning to analytically characterize the beneﬁts of using XR-Transformers.

Original: Description of the cSBM.
Revised: Description of the cSBM.

Original: Using our Assumption 4.1, we analyze the case where the graph and node features are generated according to a cSBM (Deshpande et al., 2018)
Revised: Using our Assumption 4.1, we analyze the case where the graph and node features are generated according to a cSBM (Deshpande et al., 2018)

Original: (see Figure 4).
Revised: (see Figure 4).

Original: For simplicity, we use the most straightforward two-cluster cSBM.
Revised: For simplicity, we use the most straightforward two-cluster cSBM.

Original: Let { y i } ni =1 ∈ { 0 , 1 } be the labels of nodes in a graph.
Revised: Let { y i } ni =1 ∈ { 0 , 1 } be the labels of nodes in a graph.

Original: We denote the size of class j ∈ { 0 , 1 } by C j = |{ i : y i
Revised: We denote the size of class j ∈ { 0 , 1 } by C j = |{ i : y i

Original: = j, ∀ i ∈ [ n ] }| .
Revised: = j, ∀ i ∈ [ n ] }| .

Original: We also assume that the classes are balanced, i.e., C 0 = C 1 = n 2 .
Revised: We also assume that the classes are balanced, i.e., C 0 = C 1 = n 2 .

Original: The node features { X i } ni
Revised: The node features { X i } ni

Original: =1 ∈ R
Revised: =1 ∈ R

Original: d are independent d -dimensional
Revised: d are independent d -dimensional

Original: Gaussian random vectors, such that X i ∼ N ( r √ d 1 , σ 2 d I )
Revised: Gaussian random vectors, such that X i ∼ N ( r √ d 1 , σ 2 d I )

Original: if y i
Revised: if y i

Original: =
Revised: =

Original: and X i ∼ N ( − r √ d 1 , σ 2 d I ) if y i = 1 .
Revised: and X i ∼ N ( − r √ d 1 , σ 2 d I ) if y i = 1 .

Original: The adjacency matrix of the cSBM is denoted by A , and is clearly symmetric.
Revised: The adjacency matrix of the cSBM is denoted by A , and is clearly symmetric.

Original: All edges are drawn according to independent Bernoulli random variables, so that A ij ∼ Ber ( p ) if y i =
Revised: All edges are drawn according to independent Bernoulli random variables, so that A ij ∼ Ber ( p ) if y i =

Original: y j and A ij ∼ Ber
Revised: y j and A ij ∼ Ber

Original: ( q ) if y i (cid:54) =
Revised: ( q ) if y i (cid:54) =

Original: y j .
Revised: y j .

Original: Our analysis is inspired by Baranwal et al.
Revised: Our analysis is inspired by Baranwal et al.

Original: (2021) and Li et al.
Revised: (2021) and Li et al.

Original: (2019), albeit their deﬁnitions of graph convolutions and random walks differ from those in PIFA.
Revised: (2019), albeit their deﬁnitions of graph convolutions and random walks differ from those in PIFA.

Original: For our subsequent analysis, we also make use of the following standard assumption and deﬁne the notion of effect size .
Revised: For our subsequent analysis, we also make use of the following standard assumption and deﬁne the notion of effect size .

Original: Assumption 4.2.
Revised: Assumption 4.2.

Original: p, q = ω ( (cid:113) log nn ) .
Revised: p, q = ω ( (cid:113) log nn ) .

Original: | p − q | p + q = Θ(1) .
Revised: | p − q | p + q = Θ(1) .

Original: d = o ( n ) . 0 < r, σ = Θ(1) .
Revised: d = o ( n ) . 0 < r, σ = Θ(1) .

Original: Note that Baranwal et al.
Revised: Note that Baranwal et al.

Original: (2021) also poses constraints on p, q, | p − q | p + q and d .
Revised: (2021) also poses constraints on p, q, | p − q | p + q and d .

Original: In contrast, we do not require p − q > 0 to hold (Baranwal et al., 2021;
Revised: In contrast, we do not require p − q > 0 to hold (Baranwal et al., 2021;

Original: Li et al., 2019) so that we can address graph structures that are either homophilic or heterophilic.
Revised: Li et al., 2019) so that we can address graph structures that are either homophilic or heterophilic.

Original: Due to the difference between PIFA and standard graph convolution, we require p, q to be larger compared to the corresponding values used in Baranwal et al.
Revised: Due to the difference between PIFA and standard graph convolution, we require p, q to be larger compared to the corresponding values used in Baranwal et al.

Original: [Table]
Revised: [Table]

Original: Deﬁnition 4.3.
Revised: Deﬁnition 4.3.

Original: For cSBM, the effect size of the two centroids of the node features X of the two different classes is deﬁned as
Revised: For cSBM, the effect size of the two centroids of the node features X of the two different classes is deﬁned as

Original: [Equation]
Revised: [Equation]

Original: y i (cid:54) =
Revised: y i (cid:54) =

Original: y j .
Revised: y j .

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In the standard deﬁnition of effect size, the mean difference is divided by the standard deviation of a class, as the latter is assumed to be the same for both classes.
Revised: In the standard deﬁnition of effect size, the mean difference is divided by the standard deviation of a class, as the latter is assumed to be the same for both classes.

Original: We use the sum of both standard deviations to prevent any ambiguity in our deﬁnition.
Revised: We use the sum of both standard deviations to prevent any ambiguity in our deﬁnition.

Original: Note that for the case of isotropic Gaussian distributions, the larger the effect size the larger the separation of the two classes.
Revised: Note that for the case of isotropic Gaussian distributions, the larger the effect size the larger the separation of the two classes.

Original: Theoretical results.
Revised: Theoretical results.

Original: We are now ready to state our main theoretical result, which asserts that the effect size of centroids for PIFA embeddings Z is asymptotically larger than that obtained from the original node features.
Revised: We are now ready to state our main theoretical result, which asserts that the effect size of centroids for PIFA embeddings Z is asymptotically larger than that obtained from the original node features.

Original: Our Theorem 4.4 provides strong evidence that using PIFA in XRTransformers offers improved clustering results and consequently, better feature quality.
Revised: Our Theorem 4.4 provides strong evidence that using PIFA in XRTransformers offers improved clustering results and consequently, better feature quality.

Original: Theorem 4.4.
Revised: Theorem 4.4.

Original: For the cSBM and under Assumption 4.2, the effect size of the two centroids of the node features X of the two different classes is rσ = Θ(1) .
Revised: For the cSBM and under Assumption 4.2, the effect size of the two centroids of the node features X of the two different classes is rσ = Θ(1) .

Original: Moreover, the effect size of the two centroids of the PIFA embedding Z of the two different classes, conditioned on an event of probability at least 1 − O ( 1 n c ) for some constant c > 0 , is ω (1) .
Revised: Moreover, the effect size of the two centroids of the PIFA embedding Z of the two different classes, conditioned on an event of probability at least 1 − O ( 1 n c ) for some constant c > 0 , is ω (1) .

Original: We see that although two nodes i, j from the same class have the same neighborhood vectors in expectation, E A i =
Revised: We see that although two nodes i, j from the same class have the same neighborhood vectors in expectation, E A i =

Original: E A j , their Hamming distance can be large in practice.
Revised: E A j , their Hamming distance can be large in practice.

Original: This ﬁnding is formally characterized in Proposition 4.5.
Revised: This ﬁnding is formally characterized in Proposition 4.5.

Original: Proposition 4.5.
Revised: Proposition 4.5.

Original: For the cSBM and under Assumption 4.2, the Hamming distance between A i and
Revised: For the cSBM and under Assumption 4.2, the Hamming distance between A i and

Original: A j with y i =
Revised: A j with y i =

Original: y j is ω ( √ n log n ) with probability at least 1 − O ( 1 n c ) , for some c > 0 .
Revised: y j is ω ( √ n log n ) with probability at least 1 − O ( 1 n c ) , for some c > 0 .

Original: Hence, directly using neighborhood vectors for self-supervision is not advisable.
Revised: Hence, directly using neighborhood vectors for self-supervision is not advisable.

Original: Our result also agrees with ﬁndings from the XMC literature (Chang et al., 2020b).
Revised: Our result also agrees with ﬁndings from the XMC literature (Chang et al., 2020b).

Original: It is also intuitively clear that averaging neighborhood vectors from the same class can reduce the variance, which is approximately performed by clustering based on node representations (in our case, via a PIFA embedding).
Revised: It is also intuitively clear that averaging neighborhood vectors from the same class can reduce the variance, which is approximately performed by clustering based on node representations (in our case, via a PIFA embedding).

Original: This result establishes the importance of clustering within the XR-Transformer approach and for the
Revised: This result establishes the importance of clustering within the XR-Transformer approach and for the

Original: SSL neighborhood prediction task.
Revised: SSL neighborhood prediction task.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: Evaluation Datasets.
Revised: Evaluation Datasets.

Original: We consider node classiﬁcation as our downstream task and evaluate GIANT on three large-scale OGB datasets (Hu et al., 2020a) with available raw text: ogbn-arxiv, ogbn-products, and
Revised: We consider node classiﬁcation as our downstream task and evaluate GIANT on three large-scale OGB datasets (Hu et al., 2020a) with available raw text: ogbn-arxiv, ogbn-products, and

Original: ogbn-papers100M.
Revised: ogbn-papers100M.

Original: The parameters of these datasets are given in Table 1 and detailed descriptions are available in the Appendix E.1.
Revised: The parameters of these datasets are given in Table 1 and detailed descriptions are available in the Appendix E.1.

Original: Following the OGB benchmarking protocol, we report the average test accuracy and the corresponding standard deviation by repeating 3 runs of each downstream GNN model.
Revised: Following the OGB benchmarking protocol, we report the average test accuracy and the corresponding standard deviation by repeating 3 runs of each downstream GNN model.

Original: Evaluation Protocol.
Revised: Evaluation Protocol.

Original: We refer to our actual implementation as GIANT-XRT since the multi-scale neighborhood prediction task in the proposed GIANT framework is solved by an XR-Transformer.
Revised: We refer to our actual implementation as GIANT-XRT since the multi-scale neighborhood prediction task in the proposed GIANT framework is solved by an XR-Transformer.

Original: In the pre-training stage, GIANT-XRT learns a raw text encoder by optimizing the self-supervised neighborhood prediction objective, and generates a ﬁne-tuned node embedding for later stages.
Revised: In the pre-training stage, GIANT-XRT learns a raw text encoder by optimizing the self-supervised neighborhood prediction objective, and generates a ﬁne-tuned node embedding for later stages.

Original: For the node classiﬁcation downstream tasks, we input the node embeddings from GIANT-XRT into several different GNN models.
Revised: For the node classiﬁcation downstream tasks, we input the node embeddings from GIANT-XRT into several different GNN models.

Original: One is the multi-layer perceptron (MLP), which does not use graph information.
Revised: One is the multi-layer perceptron (MLP), which does not use graph information.

Original: Two other methods are GraphSAGE (Hamilton et al., 2017), which we applied to ogbn-arxiv, and GraphSAINT (Zeng et al., 2020), which we used for ogbn-products as it allows for mini-batch training.
Revised: Two other methods are GraphSAGE (Hamilton et al., 2017), which we applied to ogbn-arxiv, and GraphSAINT (Zeng et al., 2020), which we used for ogbn-products as it allows for mini-batch training.

Original: Due to scalability issues, we used Simple Graph Convolution (SGC) (Wu et al., 2019) for ogbn-papers100M.
Revised: Due to scalability issues, we used Simple Graph Convolution (SGC) (Wu et al., 2019) for ogbn-papers100M.

Original: We also tested the state-of-the-art GNN for each dataset.
Revised: We also tested the state-of-the-art GNN for each dataset.

Original: [Table]
Revised: [Table]

Original: At the time we conducted the main experiments (07/01/2021), the top-ranked model for ogbn-arxiv was RevGAT 2 (Li et al., 2021) and the top-ranked model for ogbn-products was SAGN 3 (Sun & Wu, 2021).
Revised: At the time we conducted the main experiments (07/01/2021), the top-ranked model for ogbn-arxiv was RevGAT 2 (Li et al., 2021) and the top-ranked model for ogbn-products was SAGN 3 (Sun & Wu, 2021).

Original: When we conducted the experiment on ogbn-papers100M (09/10/2021), the topranked model for ogbn-papers100M was GAMLP 4 (Zhang et al., 2021b)
Revised: When we conducted the experiment on ogbn-papers100M (09/10/2021), the topranked model for ogbn-papers100M was GAMLP 4 (Zhang et al., 2021b)

Original: (Since then, the highest reported accuracy was improved by 0 .
Revised: (Since then, the highest reported accuracy was improved by 0 .

Original: 05% for ogbn-arxiv and 0 .
Revised: 05% for ogbn-arxiv and 0 .

Original: 31% for ogbn-products; both of these improvements fall short compared to those offered by GIANT).
Revised: 31% for ogbn-products; both of these improvements fall short compared to those offered by GIANT).

Original: For all evaluations, we use publicly available implementations of the GNNs.
Revised: For all evaluations, we use publicly available implementations of the GNNs.

Original: For RevGAT, we report the performance of the model with and without self knowledge distillation; the former setting is henceforth referred to as +SelfKD.
Revised: For RevGAT, we report the performance of the model with and without self knowledge distillation; the former setting is henceforth referred to as +SelfKD.

Original: For SAGN, we report results with the self-label-enhanced (SLE) feature, and denote them by SAGN+SLE.
Revised: For SAGN, we report results with the self-label-enhanced (SLE) feature, and denote them by SAGN+SLE.

Original: For GAMLP, we report results with and without Reliable Label Utilization (RLU); the former is denoted as GAMLP+RLU.
Revised: For GAMLP, we report results with and without Reliable Label Utilization (RLU); the former is denoted as GAMLP+RLU.

Original: SSL GNN Competing Methods.
Revised: SSL GNN Competing Methods.

Original: We compare GIANT-XRT to methods that rely on graphagnostic feature inputs and use node embeddings generated by various SSL GNNs modules.
Revised: We compare GIANT-XRT to methods that rely on graphagnostic feature inputs and use node embeddings generated by various SSL GNNs modules.

Original: The graph-agnostic features are either default features available from the OGB datasets (denoted by
Revised: The graph-agnostic features are either default features available from the OGB datasets (denoted by

Original: OGB-feat) or obtained from plain BERT embeddings (without ﬁne-tuning) generated from raw text (denoted by BERT (cid:63) ).
Revised: OGB-feat) or obtained from plain BERT embeddings (without ﬁne-tuning) generated from raw text (denoted by BERT (cid:63) ).

Original: For OGB-feat combined with downstream GNN methods, we report the results from the OGB leaderboard (and denote them by † ).
Revised: For OGB-feat combined with downstream GNN methods, we report the results from the OGB leaderboard (and denote them by † ).

Original: For the SSL GNNs modules, we test three frequently-used methods: (Variantional) Graph AutoEncoders (Kipf & Welling, 2016) (denoted by (V)GAE); Deep Graph Infomax (Velickovic et al., 2019) (denoted by DGI); and GraphZoom (Deng et al., 2020) (denoted by GZ).
Revised: For the SSL GNNs modules, we test three frequently-used methods: (Variantional) Graph AutoEncoders (Kipf & Welling, 2016) (denoted by (V)GAE); Deep Graph Infomax (Velickovic et al., 2019) (denoted by DGI); and GraphZoom (Deng et al., 2020) (denoted by GZ).

Original: The hyper-parameters of SSL GNNs modules are given in the Appendix E.2.
Revised: The hyper-parameters of SSL GNNs modules are given in the Appendix E.2.

Original: For all reported results, we use X plain , X SSLGNN and X GIANT (c.f. Figure 1) to denote which framework the method belongs to.
Revised: For all reported results, we use X plain , X SSLGNN and X GIANT (c.f. Figure 1) to denote which framework the method belongs to.

Original: Note that X GIANT refers to our approach.
Revised: Note that X GIANT refers to our approach.

Original: The implementation details and hyper-parameters of GIANT-XRT can be founded in the Appendix E.3.
Revised: The implementation details and hyper-parameters of GIANT-XRT can be founded in the Appendix E.3.

Original: 5. M AIN R ESULTS
Revised: 5. M AIN R ESULTS

Original: The results for the ogbn-arxiv and ogbn-products datasets are listed in Table 2.
Revised: The results for the ogbn-arxiv and ogbn-products datasets are listed in Table 2.

Original: Our GIANT-XRT approach gives the best results for both datasets and all downstream models.
Revised: Our GIANT-XRT approach gives the best results for both datasets and all downstream models.

Original: It improves the accuracy of the top-ranked OGB leaderboard models by a large margin: 1 . 86% on ogbn-arxiv and 1 .
Revised: It improves the accuracy of the top-ranked OGB leaderboard models by a large margin: 1 . 86% on ogbn-arxiv and 1 .

Original: 19% on ogbn-products.
Revised: 19% on ogbn-products.

Original: Using graph-agnostic BERT embeddings does not necessarily lead to good results (see the ﬁrst two rows in Table 2).
Revised: Using graph-agnostic BERT embeddings does not necessarily lead to good results (see the ﬁrst two rows in Table 2).

Original: This shows that the improvement of our method is not merely
Revised: This shows that the improvement of our method is not merely

Original: [Table]
Revised: [Table]

Original: due to the use of a more powerful language model, and establishes the need for self-supervision governed by graph information.
Revised: due to the use of a more powerful language model, and establishes the need for self-supervision governed by graph information.

Original: Another observation is that among possible combinations involving a standard GNN pipeline with a SSL module, BERT+(V)GAE offers the best performance.
Revised: Another observation is that among possible combinations involving a standard GNN pipeline with a SSL module, BERT+(V)GAE offers the best performance.

Original: This can be attributed to exploiting the correlation between numerical node features and the graph structure, albeit in a two-stage approach within the standard GNN pipeline.
Revised: This can be attributed to exploiting the correlation between numerical node features and the graph structure, albeit in a two-stage approach within the standard GNN pipeline.

Original: The most important ﬁnding is that using node features generated by GIANT-XRT leads to consistent and signiﬁcant improvements in the accuracy of all tested methods, when compared to the standard GNN pipeline:
Revised: The most important ﬁnding is that using node features generated by GIANT-XRT leads to consistent and signiﬁcant improvements in the accuracy of all tested methods, when compared to the standard GNN pipeline:

Original: In particular, on ogbn-arxiv, the improvement equals 17 .
Revised: In particular, on ogbn-arxiv, the improvement equals 17 .

Original: 58% for MLP and 3 .
Revised: 58% for MLP and 3 .

Original: 1% for GraphSAGE; on ogbn-products, the improvement equals 18 .
Revised: 1% for GraphSAGE; on ogbn-products, the improvement equals 18 .

Original: 76% for MLP and 5 .
Revised: 76% for MLP and 5 .

Original: 32% for GraphSAINT.
Revised: 32% for GraphSAINT.

Original: Figure 5 in the Appendix E. further illustrate the gain obtained by our GIANT-XRT over SOTA methods on OGB leaderboard.
Revised: Figure 5 in the Appendix E. further illustrate the gain obtained by our GIANT-XRT over SOTA methods on OGB leaderboard.

Original: Another important observation is that GIANT-XRT is highly scalable, which can be clearly observed on the example of the ogbn-papers100M dataset, for which the results are shown in Table 3.
Revised: Another important observation is that GIANT-XRT is highly scalable, which can be clearly observed on the example of the ogbn-papers100M dataset, for which the results are shown in Table 3.

Original: In particular, GIANT-XRT improves the accuracy of the top-ranked model, GAMLP-RLU, by a margin of 1 . 42% .
Revised: In particular, GIANT-XRT improves the accuracy of the top-ranked model, GAMLP-RLU, by a margin of 1 . 42% .

Original: Furthermore, GIANT-XRT again consistently improves all tested downstream methods on the ogbn-papers100M dataset.
Revised: Furthermore, GIANT-XRT again consistently improves all tested downstream methods on the ogbn-papers100M dataset.

Original: As a ﬁnal remark, we surprisingly ﬁnd that combining
Revised: As a ﬁnal remark, we surprisingly ﬁnd that combining

Original: MLP with GIANT-XRT greatly improves the performance of the former learner on all datasets.
Revised: MLP with GIANT-XRT greatly improves the performance of the former learner on all datasets.

Original: It becomes just slightly worse then GIANT-XRT+GNNs and can even outperform the GraphSAGE and GraphSAINT methods with default OGB features on ogbn-arxiv and ogbn-products datasets.
Revised: It becomes just slightly worse then GIANT-XRT+GNNs and can even outperform the GraphSAGE and GraphSAINT methods with default OGB features on ogbn-arxiv and ogbn-products datasets.

Original: This is yet another positive property of GIANT, since MLPs are low-complexity and more easily implementable than other GNNs.
Revised: This is yet another positive property of GIANT, since MLPs are low-complexity and more easily implementable than other GNNs.

Original: 5. A BLATION STUDY
Revised: 5. A BLATION STUDY

Original: We also conduct an ablation study of the GIANT framework to determine the relevance of each module involved.
Revised: We also conduct an ablation study of the GIANT framework to determine the relevance of each module involved.

Original: The ﬁrst step is to consider alternatives to the proposed multi-scale neighborhood prediction task:
Revised: The ﬁrst step is to consider alternatives to the proposed multi-scale neighborhood prediction task:

Original: In this case, we ﬁne-tune BERT with a SSL link prediction approach, which we for simplicity refer to as BERT+LP.
Revised: In this case, we ﬁne-tune BERT with a SSL link prediction approach, which we for simplicity refer to as BERT+LP.

Original: In addition, we examine how the PIFA embedding affects the performance of GIANT-XRT and how more informative node features (TFIDF) can improve the clustering steps.
Revised: In addition, we examine how the PIFA embedding affects the performance of GIANT-XRT and how more informative node features (TFIDF) can improve the clustering steps.

Original: First, recall that in GIANT-XRT, we use TFIDF features from raw text to construct PIFA embeddings.
Revised: First, recall that in GIANT-XRT, we use TFIDF features from raw text to construct PIFA embeddings.

Original: We subsequently use the term “NO TFIDF” to indicate that we replaced the TFIDF feature matrix by an identity matrix, which contain no raw text information.
Revised: We subsequently use the term “NO TFIDF” to indicate that we replaced the TFIDF feature matrix by an identity matrix, which contain no raw text information.

Original: The term “TFIDF+NO PIFA” is used to refer to the setting where only raw text information (node attributes) is used to perform hierarchical clustering.
Revised: The term “TFIDF+NO PIFA” is used to refer to the setting where only raw text information (node attributes) is used to perform hierarchical clustering.

Original: Similarly, “NO TFIDF+PIFA” indicates that we only use normalized neighborhood vectors (graph structure) to construct the hierarchical clustering.
Revised: Similarly, “NO TFIDF+PIFA” indicates that we only use normalized neighborhood vectors (graph structure) to construct the hierarchical clustering.

Original: If both node attributes and graph structure are ignore, the result is a random clustering.
Revised: If both node attributes and graph structure are ignore, the result is a random clustering.

Original: Nevertheless, we keep the same sizes of clusters at each level in the hierarchical clustering.
Revised: Nevertheless, we keep the same sizes of clusters at each level in the hierarchical clustering.

Original: The results of the ablation study are listed under rows indexed by X GIANT in Table 2 for ogbn-arxiv and ogbn-products datasets.
Revised: The results of the ablation study are listed under rows indexed by X GIANT in Table 2 for ogbn-arxiv and ogbn-products datasets.

Original: They once again conﬁrm that GIANT-XRT consistently outperforms other tested methods.
Revised: They once again conﬁrm that GIANT-XRT consistently outperforms other tested methods.

Original: For BERT+LP, we ﬁnd that it has better performance on ogbn-arixv compared to that of the standard GNN pipeline but a worse performance on ogbn-products.
Revised: For BERT+LP, we ﬁnd that it has better performance on ogbn-arixv compared to that of the standard GNN pipeline but a worse performance on ogbn-products.

Original: This shows that using link prediction to ﬁne-tune BERT in a self-supervised manner is not robust in general, and further strengthens the case for using neighborhood instead of link prediction.
Revised: This shows that using link prediction to ﬁne-tune BERT in a self-supervised manner is not robust in general, and further strengthens the case for using neighborhood instead of link prediction.

Original: With respect to the ablation study of GIANT-XRT, we see that NO TFIDF+NO PIFA indeed gives the worst results.
Revised: With respect to the ablation study of GIANT-XRT, we see that NO TFIDF+NO PIFA indeed gives the worst results.

Original: Using node attributes (TFIDF features) or graph information (PIFA) to construct the hierarchical clustering in
Revised: Using node attributes (TFIDF features) or graph information (PIFA) to construct the hierarchical clustering in

Original: GIANT-XRT leads to performance improvements that can be seen from Table 2.
Revised: GIANT-XRT leads to performance improvements that can be seen from Table 2.

Original: Nevertheless, combining both as done in GIANT-XRT gives the best results.
Revised: Nevertheless, combining both as done in GIANT-XRT gives the best results.

Original: Moreover, one can observe that using
Revised: Moreover, one can observe that using

Original: PIFA always gives better results compared to the case when PIFA is not used.
Revised: PIFA always gives better results compared to the case when PIFA is not used.

Original: It aligns with our theoretical analysis, which shows that PIFA embeddings lead to better hierarchical clusterings.
Revised: It aligns with our theoretical analysis, which shows that PIFA embeddings lead to better hierarchical clusterings.

Original: A CKNOWLEDGEMENT
Revised: A CKNOWLEDGEMENT

Original: The authors thank the support from Amazon and the Amazon Post Internship Fellowship.
Revised: The authors thank the support from Amazon and the Amazon Post Internship Fellowship.

Original: Cho-Jui
Revised: Cho-Jui

Original: Hsieh is partially supported by NSF under IIS-2008173 and IIS-2048280.
Revised: Hsieh is partially supported by NSF under IIS-2008173 and IIS-2048280.

Original: This work was funded in part by the NSF grant 1956384.
Revised: This work was funded in part by the NSF grant 1956384.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: We are not aware of any potential ethical issues regarding our work.
Revised: We are not aware of any potential ethical issues regarding our work.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: We provide our code in the supplementary material along with an easy-to-follow description and package dependency for reproducibility.
Revised: We provide our code in the supplementary material along with an easy-to-follow description and package dependency for reproducibility.

Original: Our experimental setting is stated in Section 5 and details pertaining to hyperparameters and computational environment are described in the Appendix.
Revised: Our experimental setting is stated in Section 5 and details pertaining to hyperparameters and computational environment are described in the Appendix.

Original: All tested methods are integrated in our code: https://github.com/amzn/pecos/tree/ mainline/examples/giant-xrt .
Revised: All tested methods are integrated in our code: https://github.com/amzn/pecos/tree/ mainline/examples/giant-xrt .

Original: A PPENDIX A C ONCLUSIONS
Revised: A PPENDIX A C ONCLUSIONS

Original: We introduced a novel self-supervised learning framework for graph-guided numerical node feature extraction from raw data, and evaluated it within multiple GNN pipelines.
Revised: We introduced a novel self-supervised learning framework for graph-guided numerical node feature extraction from raw data, and evaluated it within multiple GNN pipelines.

Original: Our method, termed
Revised: Our method, termed

Original: GIANT, for the ﬁrst time successfully resolved the issue of graph-agnostic numerical feature extraction.
Revised: GIANT, for the ﬁrst time successfully resolved the issue of graph-agnostic numerical feature extraction.

Original: We also described a new SSL task, neighborhood prediction, established a connection between the task and the XMC problem, and solved it using XR-Transformers.
Revised: We also described a new SSL task, neighborhood prediction, established a connection between the task and the XMC problem, and solved it using XR-Transformers.

Original: We also examined the theoretical properties of GIANT in order to evaluate the advantages of neighborhood prediction over standard link prediction, and to assess the beneﬁts of using XR-Transformers.
Revised: We also examined the theoretical properties of GIANT in order to evaluate the advantages of neighborhood prediction over standard link prediction, and to assess the beneﬁts of using XR-Transformers.

Original: Our extensive numerical experiments, which showed that GIANT consistently improves state-of-the-art GNN models, were supplemented with an ablation study that aligns with our theoretical analysis.
Revised: Our extensive numerical experiments, which showed that GIANT consistently improves state-of-the-art GNN models, were supplemented with an ablation study that aligns with our theoretical analysis.

Original: B P ROOF OF T HEOREM 4. 
Revised: B P ROOF OF T HEOREM 4.

Original: Throughout the proof, we use µ = rd 1 to denote the mean of node feature from class 0 and ν = − rdfor class 1 .
Revised: Throughout the proof, we use µ = rd 1 to denote the mean of node feature from class 0 and ν = − rdfor class 1 .

Original: We choose to keep this notation to demonstrate that our setting on mean can be easily generalized.
Revised: We choose to keep this notation to demonstrate that our setting on mean can be easily generalized.

Original: The choice of the high probability events will be clear in the proof.
Revised: The choice of the high probability events will be clear in the proof.

Original: The proof for the effect size of centroid for node feature X is quite straightforward from the Deﬁnition 4.3.
Revised: The proof for the effect size of centroid for node feature X is quite straightforward from the Deﬁnition 4.3.

Original: By directly plugging in the mean and standard deviation, we have
Revised: By directly plugging in the mean and standard deviation, we have

Original: [Equation]
Revised: [Equation]

Original: The last equality is due to our assumption that both r, σ > 0 are both constants.
Revised: The last equality is due to our assumption that both r, σ > 0 are both constants.

Original: To prove the effect size of centroid for PIFA embedding Z , we need to ﬁrst introduce some standard concentration results for sum of Bernoulli and Gaussian random variables.
Revised: To prove the effect size of centroid for PIFA embedding Z , we need to ﬁrst introduce some standard concentration results for sum of Bernoulli and Gaussian random variables.

Original: Lemma B.1 (Hoefﬁding’s inequality (Hoeffding, 1994)) .
Revised: Lemma B.1 (Hoefﬁding’s inequality (Hoeffding, 1994)) .

Original: Let S n = (cid:80) ni
Revised: Let S n = (cid:80) ni

Original: =1 X n , where X i are i.i.d.
Revised: =1 X n , where X i are i.i.d.

Original: Bernoulli random variable with parameter p .
Revised: Bernoulli random variable with parameter p .

Original: Then for any t > 0 , we have
Revised: Then for any t > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Lemma B.2 (Concentration of sum of Gaussian) .
Revised: Lemma B.2 (Concentration of sum of Gaussian) .

Original: Let S n = (cid:80) ni
Revised: Let S n = (cid:80) ni

Original: =1 X n , where X i are i.i.d.
Revised: =1 X n , where X i are i.i.d.

Original: Gaussian random variable with zero mean and standard deviation σ .
Revised: Gaussian random variable with zero mean and standard deviation σ .

Original: Then for some constant c > 0 , we have
Revised: Then for some constant c > 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Now we are ready to prove our claim.
Revised: Now we are ready to prove our claim.

Original: Recall that the deﬁnition of PIFA embedding Z is as follows:
Revised: Recall that the deﬁnition of PIFA embedding Z is as follows:

Original: [Equation]
Revised: [Equation]

Original: , where
Revised: , where

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst focus on analyzing the vector v i .
Revised: We ﬁrst focus on analyzing the vector v i .

Original: We denote N iy =
Revised: We denote N iy =

Original: |{ j : y j
Revised: |{ j : y j

Original: =
Revised: =

Original: y
Revised: y

Original: ∧ A ij = 1 , j ∈ [ n ] }| and
Revised: ∧ A ij = 1 , j ∈ [ n ] }| and

Original: N i
Revised: N i

Original: = N
Revised: = N

Original: i 

Revised: i

Original: + N i 1 .
Revised: + N i 1 .

Original: Without loss of generality, we assume y i = 0 .
Revised: Without loss of generality, we assume y i = 0 .

Original: The conditional expectation of it is as follows:
Revised: The conditional expectation of it is as follows:

Original: [Equation]
Revised: [Equation]

Original: Next, by leveraging Lemma B.1, we have
Revised: Next, by leveraging Lemma B.1, we have

Original: [Equation]
Revised: [Equation]

Original: By choosing t = c 1 √ n log n for some constant c 1 > 1 , we know that with probability at least 1 − O ( 1 n c 1 ) , N i 1 ∈ [ nq 2 − c 1 √ n log n, nq 2 + c 1 √ n log n ] .
Revised: By choosing t = c 1 √ n log n for some constant c 1 > 1 , we know that with probability at least 1 − O ( 1 n c 1 ) , N i 1 ∈ [ nq 2 − c 1 √ n log n, nq 2 + c 1 √ n log n ] .

Original: Finally, by our Assumption 4.2, we know that nq = ω ( √ n log n ) .
Revised: Finally, by our Assumption 4.2, we know that nq = ω ( √ n log n ) .

Original: Thus, we arrive to the conclusion that with probability at least 1 − O ( 1 n c 1 ) ,
Revised: Thus, we arrive to the conclusion that with probability at least 1 − O ( 1 n c 1 ) ,

Original: N i 1 = nq 2 ±
Revised: N i 1 = nq 2 ±

Original: o (1) .
Revised: o (1) .

Original: Following the same analysis, we can prove that with probability at least 1 − O ( 1 n c 1 ) , N i 

Revised: Following the same analysis, we can prove that with probability at least 1 − O ( 1 n c 1 ) , N i

Original: = np 2 ± o (1) .
Revised: = np 2 ± o (1) .

Original: The only difference is that we are dealing with n 2 − 1 random variables in this case as there’s no self-loop.
Revised: The only difference is that we are dealing with n 2 − 1 random variables in this case as there’s no self-loop.

Original: Nevertheless, ( n 2 − 1) =
Revised: Nevertheless, ( n 2 − 1) =

Original: n 2 (1 − o (1))
Revised: n 2 (1 − o (1))

Original: so the result is the same.
Revised: so the result is the same.

Original: Note that we need to apply union bound over 2 n error events ( ∀ i ∈ [ n ] and 2 cases for N i 0 and N i 1 respectively).
Revised: Note that we need to apply union bound over 2 n error events ( ∀ i ∈ [ n ] and 2 cases for N i 0 and N i 1 respectively).

Original: Together we know that the error probability is upper bounded by O ( 1 n c 2 ) for some new constant c
Revised: Together we know that the error probability is upper bounded by O ( 1 n c 2 ) for some new constant c

Original: 2 = c 1 − 1 > 0 .
Revised: 2 = c 1 − 1 > 0 .

Original: Hence, we characterize the mean of v i on a high probability event B 1 .
Revised: Hence, we characterize the mean of v i on a high probability event B 1 .

Original: Next we need to analyze its norm.
Revised: Next we need to analyze its norm.

Original: By direct analysis and condition on A , we have
Revised: By direct analysis and condition on A , we have

Original: [Equation]
Revised: [Equation]

Original: where G jk are i.i.d.
Revised: where G jk are i.i.d.

Original: Gaussian random variables with zero mean, σ standard deviation and d = stands for equal in distribution.
Revised: Gaussian random variables with zero mean, σ standard deviation and d = stands for equal in distribution.

Original: Then by Lemma B.2 we know that with probability at least 1 − O ( 1 N c 3 i ) for some constant c 3 > 2 + c| n (cid:88)
Revised: Then by Lemma B.2 we know that with probability at least 1 − O ( 1 N c 3 i ) for some constant c 3 > 2 + c| n (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This is because condition on A , we are summing over N i Gaussian random variables.
Revised: This is because condition on A , we are summing over N i Gaussian random variables.

Original: Recall that condition on our high probability event B 1 , N i = n 2 ( p + q )(1 + o (1))
Revised: Recall that condition on our high probability event B 1 , N i = n 2 ( p + q )(1 + o (1))

Original: = ω ( √ n log n ) ≤ n .
Revised: = ω ( √ n log n ) ≤ n .

Original: Thus, we know that for some c 2 > 0 , with probability at least 1 − O ( 1 n c 2 + 1 n c 3 ) = 1 − O ( 1 n c 2 )
Revised: Thus, we know that for some c 2 > 0 , with probability at least 1 − O ( 1 n c 2 + 1 n c 3 ) = 1 − O ( 1 n c 2 )

Original: we have |
Revised: we have |

Original: [Equation]
Revised: [Equation]

Original: σ √ d √ n log n ) .
Revised: σ √ d √ n log n ) .

Original: Again, recall that both N i 0 , N i 1 = ω ( √ n log n ) , thus together we have
Revised: Again, recall that both N i 0 , N i 1 = ω ( √ n log n ) , thus together we have

Original: [Equation]
Revised: [Equation]

Original: Again, we need to apply union bound over nd error events, which result in the error probability
Revised: Again, we need to apply union bound over nd error events, which result in the error probability

Original: O ( 1 n c 2 ) since c 3 > 2 + c 2 and d = o ( n ) in our Assumption 4.2.
Revised: O ( 1 n c 2 ) since c 3 > 2 + c 2 and d = o ( n ) in our Assumption 4.2.

Original: We denote the corresponding high probability event to be B 2 .
Revised: We denote the corresponding high probability event to be B 2 .

Original: Note that the same analysis can be applied to the case y i = 1 , where the result for the norm is the same and the result for v i would be just swapping p and q .
Revised: Note that the same analysis can be applied to the case y i = 1 , where the result for the norm is the same and the result for v i would be just swapping p and q .

Original: Combine all the current result, we know that with probability at least 1 − O ( 1 n c 2 ) for some c 2 > 0 , the PIFA embedding Z i equals to the following
Revised: Combine all the current result, we know that with probability at least 1 − O ( 1 n c 2 ) for some c 2 > 0 , the PIFA embedding Z i equals to the following

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Hence, the centroid distance would be
Revised: Hence, the centroid distance would be

Original: [Equation]
Revised: [Equation]

Original: Now we turn to the standard deviation part.
Revised: Now we turn to the standard deviation part.

Original: Speciﬁcally, we will characterize the following quantity (again, recall that we assume y i = 0 w.l.o.g.).
Revised: Speciﬁcally, we will characterize the following quantity (again, recall that we assume y i = 0 w.l.o.g.).

Original: [Equation]
Revised: [Equation]

Original: Recall that the latter part is the centroid for nodes with label 0 .
Revised: Recall that the latter part is the centroid for nodes with label 0 .

Original: Hence, by characterize this quantity we can understand the deviation of PIFA embedding around its centroid.
Revised: Hence, by characterize this quantity we can understand the deviation of PIFA embedding around its centroid.

Original: From the analysis above, we know that given A , we have
Revised: From the analysis above, we know that given A , we have

Original: [Equation]
Revised: [Equation]

Original: For the terms (cid:107) v i (cid:107) , N i 0 , N i 1 and (cid:107) (cid:80) nj =

Revised: For the terms (cid:107) v i (cid:107) , N i 0 , N i 1 and (cid:107) (cid:80) nj =

Original: A ij Z j (cid:107) , we already derive their concentration results above.
Revised: A ij Z j (cid:107) , we already derive their concentration results above.

Original: Plug in those results, the ﬁrst term becomes
Revised: Plug in those results, the ﬁrst term becomes

Original: [Equation]
Revised: [Equation]

Original: The second term becomes
Revised: The second term becomes

Original: [Equation]
Revised: [Equation]

Original: where the last equality is from our Assumption 4.2 that r, σ are constants.
Revised: where the last equality is from our Assumption 4.2 that r, σ are constants.

Original: Together we show that the deviation of nodes from their centroid is of scale o (1) .
Revised: Together we show that the deviation of nodes from their centroid is of scale o (1) .

Original: The similar result holds for the case y i = 1 .
Revised: The similar result holds for the case y i = 1 .

Original: Together we have shown that the standard deviation of Z i is o (1) on the high probability event B 1 ∩ B 2 .
Revised: Together we have shown that the standard deviation of Z i is o (1) on the high probability event B 1 ∩ B 2 .

Original: Hence, the effect size for the PIFA embedding is ω (1) with probability at least 1 − O ( 1 n c 2 ) for some constant c 2 > 0 , which implies that PIFA embedding gives a better clustered node representation.
Revised: Hence, the effect size for the PIFA embedding is ω (1) with probability at least 1 − O ( 1 n c 2 ) for some constant c 2 > 0 , which implies that PIFA embedding gives a better clustered node representation.

Original: Thus, it is preferable to use PIFA embedding and we complete the proof.
Revised: Thus, it is preferable to use PIFA embedding and we complete the proof.

Original: C P ROOF OF P ROPOSITION 4. 
Revised: C P ROOF OF P ROPOSITION 4.

Original: Note that under the setting of cSBM and the Assumption 4.2, the Hamming distance of A i , A j for y i =
Revised: Note that under the setting of cSBM and the Assumption 4.2, the Hamming distance of A i , A j for y i =

Original: y j is a Poisson-Binomial random variable.
Revised: y j is a Poisson-Binomial random variable.

Original: More precisely, note that
Revised: More precisely, note that

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where they are all independent.
Revised: where they are all independent.

Original: Hence, we have
Revised: Hence, we have

Original: [Equation]
Revised: [Equation]

Original: where Hamming ( A i , A j ) denotes the Hamming distance of A
Revised: where Hamming ( A i , A j ) denotes the Hamming distance of A

Original: i , A j and Bin ( a, b ) stands for the Binomial random variable with a trials and the success probability is b .
Revised: i , A j and Bin ( a, b ) stands for the Binomial random variable with a trials and the success probability is b .

Original: By leveraging the Lemma B.1, we know that for a random variable X ∼ Bin ( n 2 , 2 q (1 − q )) , we have
Revised: By leveraging the Lemma B.1, we know that for a random variable X ∼ Bin ( n 2 , 2 q (1 − q )) , we have

Original: [Equation]
Revised: [Equation]

Original: Note that the function q (1 − q ) is monotonic increasing for q ∈ [0 , 12 ] and has maximum at q = 12 .
Revised: Note that the function q (1 − q ) is monotonic increasing for q ∈ [0 , 12 ] and has maximum at q = 12 .

Original: Combine with Assumption 4.2 we know that nq (1 − q ) = ω ( √ n log n ) .
Revised: Combine with Assumption 4.2 we know that nq (1 − q ) = ω ( √ n log n ) .

Original: Hence, by choosing t = √ cn log nfor some constant c > 0 , with probability at least 1 − O ( 1 n c ) we have
Revised: Hence, by choosing t = √ cn log nfor some constant c > 0 , with probability at least 1 − O ( 1 n c ) we have

Original: [Equation]
Revised: [Equation]

Original: Finally, by noting the fact that with probability 1 we have Bin ( n 2 − 2 , 2 p (1 − p )) ≥ 0 .
Revised: Finally, by noting the fact that with probability 1 we have Bin ( n 2 − 2 , 2 p (1 − p )) ≥ 0 .

Original: Hence, by showing X ∼ Bin ( n 2 , 2 q (1 − q )) is of order ω ( √ n log n ) with probability at least 1 − O ( 1 n c ) implies that the Hamming distance of A i , A j is of order ω ( √ n log n ) with at least the same probability.
Revised: Hence, by showing X ∼ Bin ( n 2 , 2 q (1 − q )) is of order ω ( √ n log n ) with probability at least 1 − O ( 1 n c ) implies that the Hamming distance of A i , A j is of order ω ( √ n log n ) with at least the same probability.

Original: Together we complete the proof.
Revised: Together we complete the proof.

Original: D P ROOF OF L EMMA B. 
Revised: D P ROOF OF L EMMA B.

Original: By Chernoff bound, we have
Revised: By Chernoff bound, we have

Original: [Equation]
Revised: [Equation]

Original: By the i.i.d assumption, we have
Revised: By the i.i.d assumption, we have

Original: [Equation]
Revised: [Equation]

Original: Note that the moment generating function (MGF) of a zero-mean, σ standard deviation Gaussian random variable is exp( σ 2 t 2 2 ) .
Revised: Note that the moment generating function (MGF) of a zero-mean, σ standard deviation Gaussian random variable is exp( σ 2 t 2 2 ) .

Original: Hence we have
Revised: Hence we have

Original: [Equation]
Revised: [Equation]

Original: t
Revised: t

Original: [Equation]
Revised: [Equation]

Original: By minimizing the upper bound with respect to t , we can choose t = anσ 2 .
Revised: By minimizing the upper bound with respect to t , we can choose t = anσ 2 .

Original: Plug in this choice of t we have
Revised: Plug in this choice of t we have

Original: [Equation]
Revised: [Equation]

Original: Finally, by choosing a = cσ √ n log n for some c > 0 , applying the same bound for the other side and the union bound, we complete the proof.
Revised: Finally, by choosing a = cσ √ n log n for some c > 0 , applying the same bound for the other side and the union bound, we complete the proof.

Original: E E XPERIMENTAL DETAIL E. D ATASETS
Revised: E E XPERIMENTAL DETAIL E. D ATASETS

Original: In this work, we choose node classiﬁcation as our downstream task to focus.
Revised: In this work, we choose node classiﬁcation as our downstream task to focus.

Original: We conduct experiments on three large-scale datasets, ogbn-arxiv, ogbn-products and ogbn-papers100M as these are the only three datasets with raw text available in OGB.
Revised: We conduct experiments on three large-scale datasets, ogbn-arxiv, ogbn-products and ogbn-papers100M as these are the only three datasets with raw text available in OGB.

Original: Ogbn-arxiv and ogbn-papers100M (Wang et al., 2020; Hu et al., 2020a) are citation networks where each node represents a paper.
Revised: Ogbn-arxiv and ogbn-papers100M (Wang et al., 2020; Hu et al., 2020a) are citation networks where each node represents a paper.

Original: The corresponding input raw text consists of titles and abstracts and the node labels are the primary categories of the papers.
Revised: The corresponding input raw text consists of titles and abstracts and the node labels are the primary categories of the papers.

Original: Ogbn-products (Chiang et al., 2019; Hu et al., 2020a) is an Amazon co-purchase network where each node represents a product.
Revised: Ogbn-products (Chiang et al., 2019; Hu et al., 2020a) is an Amazon co-purchase network where each node represents a product.

Original: The corresponding input raw text consists of titles and descriptions of products.
Revised: The corresponding input raw text consists of titles and descriptions of products.

Original: The node labels are categories of products
Revised: The node labels are categories of products

Original: . E. H YPER - PARAMETERS OF SSL GNN M ODULES
Revised: . E. H YPER - PARAMETERS OF SSL GNN M ODULES

Original: The implementation of (V)GAE and DGI are taken from Pytorch Geometric Library (Fey &
Revised: The implementation of (V)GAE and DGI are taken from Pytorch Geometric Library (Fey &

Original: Lenssen, 2019).
Revised: Lenssen, 2019).

Original: Note that due to the GPU memory constraint, we adopt GraphSAINT (Zeng et al., 2020) to (V)GAE and DGI for ogbn-products.
Revised: Note that due to the GPU memory constraint, we adopt GraphSAINT (Zeng et al., 2020) to (V)GAE and DGI for ogbn-products.

Original: GraphZoom is directly taken from the ofﬁcial repository 5 .
Revised: GraphZoom is directly taken from the ofﬁcial repository 5 .

Original: For all downstream GNNs in the experiment, we average the results over three independent runs.
Revised: For all downstream GNNs in the experiment, we average the results over three independent runs.

Original: The only exception is OGB-feat + downstream GNNs, where we directly take the results from
Revised: The only exception is OGB-feat + downstream GNNs, where we directly take the results from

Original: OGB leaderboard.
Revised: OGB leaderboard.

Original: Note that we also try to repeat the experiment of OGB-feat + downstream GNNs, where the accuracy is similar to the one reported on the leaderboard.
Revised: Note that we also try to repeat the experiment of OGB-feat + downstream GNNs, where the accuracy is similar to the one reported on the leaderboard.

Original: To prevent confusion we decide to take the results from OGB leaderboard for comparison.
Revised: To prevent confusion we decide to take the results from OGB leaderboard for comparison.

Original: For the BERT model used throughout the paper, we use “bert-base-uncased” downloaded from HuggingFace 6 .
Revised: For the BERT model used throughout the paper, we use “bert-base-uncased” downloaded from HuggingFace 6 .

Original: For the methods used in the
Revised: For the methods used in the

Original: SSL GNNs module, we try our best to follow the default setting.
Revised: SSL GNNs module, we try our best to follow the default setting.

Original: We slightly optimize some hyperparameters (such as learning rate, max epochs...etc) to ensure the training process converge.
Revised: We slightly optimize some hyperparameters (such as learning rate, max epochs...etc) to ensure the training process converge.

Original: To ensure the fair comparison, we ﬁx the output dimension for all SSL GNNs as 768 which is the same as bert-base-uncased and XR-Transformer.
Revised: To ensure the fair comparison, we ﬁx the output dimension for all SSL GNNs as 768 which is the same as bert-base-uncased and XR-Transformer.

Original: E. H YPER - PARAMETERS OF GIANT-XRT AND BERT+LP
Revised: E. H YPER - PARAMETERS OF GIANT-XRT AND BERT+LP

Original: Pre-training of GIANT-XRT.
Revised: Pre-training of GIANT-XRT.

Original: In Table 4, we outline the pre-training hyper-parameter of
Revised: In Table 4, we outline the pre-training hyper-parameter of

Original: GIANT-XRT for all three OGB benchmark datasets.
Revised: GIANT-XRT for all three OGB benchmark datasets.

Original: We mostly follow the convention of XRTransformer (Zhang et al., 2021a) to set the hyper-parameters.
Revised: We mostly follow the convention of XRTransformer (Zhang et al., 2021a) to set the hyper-parameters.

Original: For ogbn-arxiv and ogbn-products 5 https://github.com/cornell-zhang/GraphZoomhttps://huggingface.co/bert-base-uncased datasets, we use the full graph adjacency matrix as the XMC instance-to-label matrix Y ∈ { 0 , 1 } n × n , where n is number of nodes in the graph.
Revised: For ogbn-arxiv and ogbn-products 5 https://github.com/cornell-zhang/GraphZoomhttps://huggingface.co/bert-base-uncased datasets, we use the full graph adjacency matrix as the XMC instance-to-label matrix Y ∈ { 0 , 1 } n × n , where n is number of nodes in the graph.

Original: For ogbn-papers100M, we sub-sample 50 M (out of 111M) most important nodes based on page rank scores of the bipartite graph (He et al., 2016).
Revised: For ogbn-papers100M, we sub-sample 50 M (out of 111M) most important nodes based on page rank scores of the bipartite graph (He et al., 2016).

Original: The resulting XMC instance-to-label matrix Y has 50 .
Revised: The resulting XMC instance-to-label matrix Y has 50 .

Original: M of rows, 49 .
Revised: M of rows, 49 .

Original: 9 M of columns, and 2 .
Revised: 9 M of columns, and 2 .

Original: 5 B of edges.
Revised: 5 B of edges.

Original: The PIFA node embedding for hierarchical clustering is constructed by aggregating its neighborhood nodes’ TF-IDF features.
Revised: The PIFA node embedding for hierarchical clustering is constructed by aggregating its neighborhood nodes’ TF-IDF features.

Original: Speciﬁcally, the PIFA node embedding is a 4.2M highdimensional sparse vector, consisting of 1M of word unigram, 3M of word bigram, and 200K of character triagram.
Revised: Speciﬁcally, the PIFA node embedding is a 4.2M highdimensional sparse vector, consisting of 1M of word unigram, 3M of word bigram, and 200K of character triagram.

Original: Finally, for ogbn-arxiv and ogbn-products, we use TFN + MAN negative sampling to pre-train XR-Transformer where the model-aware negatives (MAN) are selected from top 20 model’s predictions.
Revised: Finally, for ogbn-arxiv and ogbn-products, we use TFN + MAN negative sampling to pre-train XR-Transformer where the model-aware negatives (MAN) are selected from top 20 model’s predictions.

Original: Because of the extreme scale of ogbn-papers100M, we consider TFN only to avoid excessive CPU memory consumption on the GPU machine.
Revised: Because of the extreme scale of ogbn-papers100M, we consider TFN only to avoid excessive CPU memory consumption on the GPU machine.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Pre-training on BERT+LP.
Revised: Pre-training on BERT+LP.

Original: To verify the effectiveness of multi-scale neighborhood prediction loss, we consider learning a Siamese BERT encoder with the alternative Link Prediction loss for pretraining, hence the name BERT+LP.
Revised: To verify the effectiveness of multi-scale neighborhood prediction loss, we consider learning a Siamese BERT encoder with the alternative Link Prediction loss for pretraining, hence the name BERT+LP.

Original: We implement BERT+LP baseline with the triplet loss (Balntas et al., 2016) as we empirically observed it has better performance than other loss functions for the link prediction task.
Revised: We implement BERT+LP baseline with the triplet loss (Balntas et al., 2016) as we empirically observed it has better performance than other loss functions for the link prediction task.

Original: We sample one positive pair and one negative pair for each node as training samples for each epoch, and train the model until the loss is converged.
Revised: We sample one positive pair and one negative pair for each node as training samples for each epoch, and train the model until the loss is converged.

Original: E. H YPER - PARAMETERS OF DOWNSTREAM METHODS
Revised: E. H YPER - PARAMETERS OF DOWNSTREAM METHODS

Original: For the downstream models, we optimize the learning rate within { 0 . 01 , 0 . 001 } for all models.
Revised: For the downstream models, we optimize the learning rate within { 0 . 01 , 0 . 001 } for all models.

Original: For
Revised: For

Original: MLP, GraphSAGE and GraphSAINT, we optimize the number of layers within { 1 , 3 } .
Revised: MLP, GraphSAGE and GraphSAINT, we optimize the number of layers within { 1 , 3 } .

Original: For RevGAT, we keep the hyperparameter choice the same as default.
Revised: For RevGAT, we keep the hyperparameter choice the same as default.

Original: For SAGN, we also optimize the number of layers within { 1 , 3 } .
Revised: For SAGN, we also optimize the number of layers within { 1 , 3 } .

Original: For GAMLP, we directly adopt the setting from the ofﬁcial implementation.
Revised: For GAMLP, we directly adopt the setting from the ofﬁcial implementation.

Original: Note that all hyperparameter tuning applies for all pre-trained node features ( X plain , X SSLGNN and
Revised: Note that all hyperparameter tuning applies for all pre-trained node features ( X plain , X SSLGNN and

Original: X GIANT ).
Revised: X GIANT ).

Original: E. C OMPUTATIONAL E NVIRONMENT
Revised: E. C OMPUTATIONAL E NVIRONMENT

Original: All experiments are conducted on the AWS p3dn.24xlarge instance, consisting of 96 Intel Xeon
Revised: All experiments are conducted on the AWS p3dn.24xlarge instance, consisting of 96 Intel Xeon

Original: CPUs with 768 GB of RAM and 8 Nvidia V100 GPUs with 32 GB of memory each.
Revised: CPUs with 768 GB of RAM and 8 Nvidia V100 GPUs with 32 GB of memory each.

Original: E. I LLUSTRATION ON THE IMPROVEMENT OF GIANT-XRT
Revised: E. I LLUSTRATION ON THE IMPROVEMENT OF GIANT-XRT

Original: See Figure 5.
Revised: See Figure 5.