Original: C AN S INGLE -P ASS C ONTRASTIVE L EARNING W ORK FOR B OTH H OMOPHILIC AND H ETEROPHILIC G RAPH ?
Revised: C AN S INGLE -P ASS C ONTRASTIVE L EARNING W ORK FOR B OTH H OMOPHILIC AND H ETEROPHILIC G RAPH ?

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing graph contrastive learning (GCL) typically requires two forward pass for a single instance to construct the contrastive loss.
Revised: Existing graph contrastive learning (GCL) typically requires two forward pass for a single instance to construct the contrastive loss.

Original: Despite its remarkable success, it is unclear whether such a dual-pass design is (theoretically) necessary.
Revised: Despite its remarkable success, it is unclear whether such a dual-pass design is (theoretically) necessary.

Original: Besides, the empirical results are hitherto limited to the homophilic graph benchmarks.
Revised: Besides, the empirical results are hitherto limited to the homophilic graph benchmarks.

Original: Then a natural question arises: Can we design a method that works for both homophilic and heterophilic graphs with a performance guarantee?
Revised: Then a natural question arises: Can we design a method that works for both homophilic and heterophilic graphs with a performance guarantee?

Original: To answer this, we theoretically analyze the concentration property of features obtained by neighborhood aggregation on both homophilic and heterophilic graphs, introduce the single-pass graph contrastive learning loss based on the property, and provide performance guarantees of the minimizer of the loss on downstream tasks.
Revised: To answer this, we theoretically analyze the concentration property of features obtained by neighborhood aggregation on both homophilic and heterophilic graphs, introduce the single-pass graph contrastive learning loss based on the property, and provide performance guarantees of the minimizer of the loss on downstream tasks.

Original: As a direct consequence of our theory, we introduce the S ingle- P ass G raph C ontrastive
Revised: As a direct consequence of our theory, we implement the S ingle- P ass G raph C ontrastive

Original: L earning method (SP-GCL).
Revised: L earning method (SP-GCL).

Original: Empirically, on 14 benchmark datasets with varying degrees of heterophily, the features learned by the SP-GCL can match or outperform existing strong baselines with significantly less computational overhead, which support ourtheoretical findings in real-world cases.
Revised: Empirically, on 14 benchmark datasets with varying degrees of heterophily, the features learned by the SP-GCL can match or outperform existing strong baselines with significantly less computational overhead, which verifies the usefulness of our findings in real-world cases.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) (Kipf & Welling, 2016a; Xu et al., 2018;
Revised: Graph Neural Networks (GNNs) (Kipf & Welling, 2016a; Xu et al., 2018;

Original: Veli ˇ ckovi ´ c et al., 2017;
Revised: Veli ˇ ckovi ´ c et al., 2017;

Original: Hamilton et al., 2017) have demonstrated great power in various graph-related tasks, especially the problems centered around node representation learning, such as node classification (Kipf & Welling, 2016a), edge prediction (Kipf & Welling, 2016b), graph classification (Xu et al., 2018), etc.
Revised: Hamilton et al., 2017) have demonstrated great power in various graph-related tasks, especially the problems centered around node representation learning, such as node classification (Kipf & Welling, 2016a), edge prediction (Kipf & Welling, 2016b), graph classification (Xu et al., 2018), etc.

Original: Prior studies posit that the good performance of GNNs largely attribute to the homophily nature of the graph data (Pei et al., 2020; Lim et al., 2021b; Zhu et al., 2020b; Abu-El-Haija et al., 2019; Chien et al., 2020; Li et al., 2021; Bo et al., 2021), i.e. , the linked nodes are likely from the same class in homophilic graphs, e.g. social network and citation networks (McPherson et al., 2001).
Revised: Prior studies posit that the good performance of GNNs largely attribute to the homophily nature of the graph data (Pei et al., 2020; Lim et al., 2021b; Zhu et al., 2020b; Abu-El-Haija et al., 2019; Chien et al., 2020; Li et al., 2021; Bo et al., 2021), i.e. , the linked nodes are likely from the same class in homophilic graphs, e.g. social network and citation networks (McPherson et al., 2001).

Original: In contrast, for heterophilic graphs, on which existing GNNs might suffer from performance drop (Pei et al., 2020; Chien et al., 2020; Zhu et al., 2020b), similar nodes are often far apart (e.g., the majority of people tend to connect with people of the opposite gender (Zhu et al., 2020b) in dating networks).
Revised: In contrast, for heterophilic graphs, on which existing GNNs might suffer from performance drop (Pei et al., 2020; Chien et al., 2020; Zhu et al., 2020b), similar nodes are often far apart (e.g., the majority of people tend to connect with people of the opposite gender (Zhu et al., 2020b) in dating networks).

Original: As a remedy, researchers have attempted to design new GNNs able to generalize well on heterophilic graph data (Pei et al., 2020; Abu-El-Haija et al., 2019; Zhu et al., 2020a;
Revised: As a remedy, researchers have attempted to design new GNNs able to generalize well on heterophilic graph data (Pei et al., 2020; Abu-El-Haija et al., 2019; Zhu et al., 2020a;

Original: Chien et al., 2020; Li et al., 2021; Bo et al., 2021).
Revised: Chien et al., 2020; Li et al., 2021; Bo et al., 2021).

Original: For both homophilic and heterophilic graphs, GNNs, like other modern deep learning approaches, require a sufficient amount of labels for training to enjoy a decent performance, while the recent trend of the Graph Contrastive Learning (GCL) (Xie et al., 2021), as an approach for learning better representation without the demand of manual annotations, has attracted great attention.
Revised: For both homophilic and heterophilic graphs, GNNs, like other modern deep learning approaches, require a sufficient amount of labels for training to enjoy a decent performance, while the recent trend of the Graph Contrastive Learning (GCL) (Xie et al., 2021), as an approach for learning better representation without the demand of manual annotations, has attracted great attention.

Original: Existing work of GCL could be roughly divided into two categories according to whether or not a graph augmentation is employed.
Revised: Existing work of GCL could be roughly divided into two categories according to whether or not a graph augmentation is employed.

Original: First, the augmentation-based GCL (You et al., 2020; Peng et al., 2020;
Revised: First, the augmentation-based GCL (You et al., 2020; Peng et al., 2020;

Original: Hassani & Khasahmadi, 2020; Zhu et al., 2021a;b; 2020d;c; Thakoor et al., 2021) follows the initial exploration of contrastive learning in the visual domain (Chen et al., 2020;
Revised: Hassani & Khasahmadi, 2020; Zhu et al., 2021a;b; 2020d;c; Thakoor et al., 2021) follows the initial exploration of contrastive learning in the visual domain (Chen et al., 2020;

Original: He et al., 2020) and involves pre-specified graph augmentations (Zhu et al., 2021a); specifically, these methods encourage representations of the same node encoded from two augmentation views to contain as less information about the way the inputs are transformed as possible during training, i.e. , to be invariant to a set of manually specified transformations.
Revised: He et al., 2020) and involves pre-specified graph augmentations (Zhu et al., 2021a); specifically, these methods encourage representations of the same node encoded from two augmentation views to contain as less information about the way the inputs are transformed as possible during training, i.e. , to be invariant to a set of manually specified transformations.

Original: Secondly, augmentation-free GCL (Lee et al., 2021; Xia et al., 2022) follows the recent bootsrapped framework (Grill et al., 2020) and constructs different views through two encoders of different updating strategies and pushes together the representations of the same node/class.
Revised: Secondly, augmentation-free GCL (Lee et al., 2021; Xia et al., 2022) follows the recent bootsrapped framework (Grill et al., 2020) and constructs different views through two encoders of different updating strategies and pushes together the representations of the same node/class.

Original: In both categories, existing GCL methods typically require two graph forward-pass, i.e. , one forward-pass for each augmented graph in the augmentation-based GCL or one for each encoder in augmentation-free GCL.
Revised: In both categories, existing GCL methods typically require two graph forward-pass, i.e. , one forward-pass for each augmented graph in the augmentation-based GCL or one for each encoder in augmentation-free GCL.

Original: Unfortunately, despite the remarkable empirical success, theoretical understanding of the necessity of such dual-pass design is missing and the empirical results are limited to the homophilic graph benchmarks.
Revised: Unfortunately, despite the remarkable empirical success, theoretical understanding of the necessity of such dual-pass design is missing and the empirical results are limited to the homophilic graph benchmarks.

Original: Therefore, in this paper, we ask the following question:
Revised: Therefore, in this paper, we ask the following question:

Original: Can one design a simple single-pass graph contrastive learning method effective on both homophilic and heterophilic graphs?
Revised: Can one design a simple single-pass graph contrastive learning method effective on both homophilic and heterophilic graphs?

Original: We provide an affirmative answer to this question both theoretically and empirically.
Revised: We provide an affirmative answer to this question both theoretically and empirically.

Original: First, we theoretically analyze the neighborhood aggregation mechanism on a homophilic/heterophilic graph and present the concentration property of the obtained features.
Revised: First, we theoretically analyze the neighborhood aggregation mechanism on a homophilic/heterophilic graph and present the concentration property of the obtained features.

Original: By exploiting such property, we introduce the single-pass graph contrastive loss and show its minimizer is equivalent to that of Matrix
Revised: By exploiting such property, we introduce the single-pass graph contrastive loss and show its minimizer is equivalent to that of Matrix

Original: Factorization (MF) over the transformed graph where the edges are constructed based on the aggregated features, which in turn allows us to derive the theoretical guarantee of the performance of the learned representations in the down-streaming node classification task.
Revised: Factorization (MF) over the transformed graph where the edges are constructed based on the aggregated features, which in turn allows us to derive the theoretical guarantee of the performance of the learned representations in the down-streaming node classification task.

Original: To verify our theoretical findings, we introduce a direct implementation of our analysis, S ingleP ass G raph C ontrastive L earning (SP-GCL).
Revised: To verify our theoretical findings, we introduce a direct implementation of our analysis, S ingleP ass G raph C ontrastive L earning (SP-GCL).

Original: Experimental results show that SP-GCL achieves competitive performance on 8 homophilic graph benchmarks and outperforms state-of-the-art GCL algorithms on all 6 heterophilic graph benchmarks with a nontrivial margin.
Revised: Experimental results show that SP-GCL achieves competitive performance on 8 homophilic graph benchmarks and outperforms state-of-the-art GCL algorithms on all 6 heterophilic graph benchmarks with a nontrivial margin.

Original: Besides, we analyze the computational complexity of SP-GCL and empirically demonstrate a significant reduction of computational overhead brought by SP-GCL.
Revised: Besides, we analyze the computational complexity of SP-GCL and empirically demonstrate a significant reduction of computational overhead brought by SP-GCL.

Original: Coupling with extensive ablation studies, we verify that the conclusions derived from our theoretical analysis are feasible forthe real-world cases.
Revised: Coupling with extensive ablation studies, we verify that the conclusions derived from our theoretical analysis are feasible for real-world cases.

Original: Our contribution could be summarized as:
Revised: Our contribution could be summarized as:

Original: • We theoretically show the concentration property of representations obtained by the neighborhood feature aggregation, which in turn inspires our novel single-pass graph contrastive learning loss.
Revised: • We theoretically show the concentration property of representations obtained by the neighborhood feature aggregation, which in turn inspires our novel single-pass graph contrastive learning loss.

Original: • We provide the theoretical guarantee for the node embedding obtained by optimizing graph contrastive learning loss in the down-streaming node classification task.
Revised: • We provide the theoretical guarantee for the node embedding obtained by optimizing graph contrastive learning loss in the down-streaming node classification task.

Original: • Experimental results show that without complex designs, compared with SOTA GCL methods, SP-GCL achieves competitive or better performance on 8 homophilic graph benchmarks andheterophilic graph benchmarks, with significantly less computational overhead.
Revised: • Experimental results show that without complex designs, compared with SOTA GCL methods, SP-GCL achieves competitive or better performance on 8 homophilic graph benchmarks andheterophilic graph benchmarks, with significantly less computational overhead.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Graph neural network on heterophilic graph.
Revised: Graph neural network on heterophilic graph.

Original: Recently, the heterophily has been recognized as an important issue for graph neural networks, which is outlined by Pei et al.
Revised: Recently, the heterophily has been recognized as an important issue for graph neural networks, which is outlined by Pei et al.

Original: (2020) firstly.
Revised: (2020) firstly.

Original: To make graph neural networks able to generalize well on the heterophilic graph, several efforts have been done from both the spatial and spectral perspectives (Pei et al., 2020; Abu-El-Haija et al., 2019; Zhu et al., 2020a;
Revised: To make graph neural networks able to generalize well on the heterophilic graph, several efforts have been done from both the spatial and spectral perspectives (Pei et al., 2020; Abu-El-Haija et al., 2019; Zhu et al., 2020a;

Original: Chien et al., 2020; Li et al., 2021;
Revised: Chien et al., 2020; Li et al., 2021;

Original: Bo et al., 2021).
Revised: Bo et al., 2021).

Original: Firstly, Chien et al.
Revised: Firstly, Chien et al.

Original: (2020) and Bo et al.
Revised: (2020) and Bo et al.

Original: (2021) analyze the necessary frequency component for GNNs to achieve good performance on heterophilic graph and propose methods which is able to utilize high-frequency information.
Revised: (2021) analyze the necessary frequency component for GNNs to achieve good performance on heterophilic graphs and propose methods that are able to utilize high-frequency information.

Original: From the spatial perspective, several graph neural networks are designed to capture important dependencies between distant nodes (Pei et al., 2020; Abu-El-Haija et al., 2019; Bo et al., 2021; Zhu et al., 2020a).
Revised: From the spatial perspective, several graph neural networks are designed to capture important dependencies between distant nodes (Pei et al., 2020; Abu-El-Haija et al., 2019; Bo et al., 2021; Zhu et al., 2020a).

Original: Although those methods have shown their effectiveness on heterophilic graphs,the human annotations are required to guide the learning of neural networks.
Revised: Although those methods have shown their effectiveness on heterophilic graphs, human annotations are required to guide the learning of neural networks.

Original: Graph contrastive learning.
Revised: Graph contrastive learning.

Original: Existing graph contrastive learning methods can be categorized intothe augmentation-based and augmentation-free methods, according to whether or not the graph augmentation techniques are employed during training.
Revised: Existing graph contrastive learning methods can be categorized into augmentation-based and augmentation-free methods, according to whether or not the graph augmentation techniques are employed during training.

Original: The augmentation-based methods (You et al., 2020; Peng et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al., 2021a;b; 2020d; Thakoor et al., 2021; Zhu et al., 2020c) encourage the target graph encoder to be invariant to the manually specified graph transformations.
Revised: The augmentation-based methods (You et al., 2020; Peng et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al., 2021a;b; 2020d; Thakoor et al., 2021; Zhu et al., 2020c) encourage the target graph encoder to be invariant to the manually specified graph transformations.

Original: Therefore, the design of graph augmentation is critical to the success of augmentation-based GCL.
Revised: Therefore, the design of graph augmentation is critical to the success of augmentation-based GCL.

Original: We summarized the augmentation methods commonly used by recent works in Table 7 of Appendix C. Other works (Lee et al., 2021; Xia et al., 2022) try to get rid of the manual design of augmentation strategies, following the bootstrapped framework (Grill et al.,
Revised: We summarized the augmentation methods commonly used by recent works in Table 7 of Appendix C. Other works (Lee et al., 2021; Xia et al., 2022) try to get rid of the manual design of augmentation strategies, following the bootstrapped framework (Grill et al.,

Original: They construct different views through two graph encoders updated with different strategies and push together the representations of the same node/class from different views.
Revised: They construct different views through two graph encoders updated with different strategies and push together the representations of the same node/class from different views.

Original: In both categories, those existing GCL methods require two graph forward-pass.
Revised: In both categories, those existing GCL methods require two graph forward-pass.

Original: Specifically, two augmented views of the same graph will be encoded separately by the same or two graph encoders for augmentation-based GCLs and the same graph will be encoded with two different graph encoders for augmentation-free GCLs, which is prohibitively expensive for large graphs.
Revised: Specifically, two augmented views of the same graph will be encoded separately by the same or two graph encoders for augmentation-based GCLs and the same graph will be encoded with two different graph encoders for augmentation-free GCLs, which is prohibitively expensive for large graphs.

Original: Besides, the theoretical analysis for the performance of GCL in the downstreaming tasks is still lacking.
Revised: Besides, the theoretical analysis for the performance of GCL in the downstream tasks is still lacking.

Original: Although several efforts have been made in  visual domain (Arora et al., 2019; Lee et al., 2020; Tosh et al., 2021; HaoChen et al., 2021), the analysis for image classification cannot be trivially extended to graph setting, since the non-Euclidean graph structure is far more complex.
Revised: Although several efforts have been made in the visual domain (Arora et al., 2019; Lee et al., 2020; Tosh et al., 2021; HaoChen et al., 2021), the analysis for image classification cannot be trivially extended to graph setting, since the non-Euclidean graph structure is far more complex.

Original:  P RELIMINARY
Revised:  P RELIMINARY

Original: Notation.
Revised: Notation.

Original: Let G =
Revised: Let G =

Original: ( V , E ) denote an undirected graph, where
Revised: ( V , E ) denote an undirected graph, where

Original: V = { v i } i ∈ [ N ] and E ⊆
Revised: V = { v i } i ∈ [ N ] and E ⊆

Original: V × V denote the node set and the edge set respectively.
Revised: V × V denote the node set and the edge set respectively.

Original: We denote the number of nodes and edges as N and E , and the label of nodes as y ∈ R N , in which y i ∈ [1 , c ] , c ≥ 2 is the number of classes.
Revised: We denote the number of nodes and edges as N and E , and the label of nodes as y ∈ R N , in which y i ∈ [1 , c ] , c ≥ 2 is the number of classes.

Original: The associated node feature matrix denotes as X ∈ R N × F , where x i ∈ R F is the feature of node v i ∈ V and F is the input feature dimension.
Revised: The associated node feature matrix denotes as X ∈ R N × F , where x i ∈ R F is the feature of node v i ∈ V and F is the input feature dimension.

Original: We denote the adjacent matrix as A ∈ { 0 , 1 } N × N , where A ij = 1 if ( v i , v j ) ∈ E ; and the corresponding degree matrix as D = diag (cid:0) d 1 , . . .
Revised: We denote the adjacent matrix as A ∈ { 0 , 1 } N × N , where A ij = 1 if ( v i , v j ) ∈ E ; and the corresponding degree matrix as D = diag (cid:0) d 1 , . . .

Original: , d N (cid:1) , d i = (cid:80)
Revised: , d N (cid:1) , d i = (cid:80)

Original: j A i,j .
Revised: j A i,j .

Original: Our objective is to unsupervisedly learn a GNN encoder f
Revised: Our objective is to unsupervisedly learn a GNN encoder f

Original: θ : X , A → R N × K receiving the node features and graph structure as input, that produces node representations in low dimensionality, i.e., K ≪ F .
Revised: θ : X , A → R N × K receiving the node features and graph structure as input, that produces node representations in low dimensionality, i.e., K ≪ F .

Original: The representations can benefit the downstream supervised or semisupervised tasks, e.g., node classification.
Revised: The representations can benefit the downstream supervised or semisupervised tasks, e.g., node classification.

Original: Homophilic and heterophilic graph.
Revised: Homophilic and heterophilic graph.

Original: Various metrics have been proposed to measure the homophily degree of a graph.
Revised: Various metrics have been proposed to measure the homophily degree of a graph.

Original: Here we adopt two representative metrics, namely, node homophily and edge homophily.
Revised: Here we adopt two representative metrics, namely, node homophily and edge homophily.

Original: The edge homophily (Zhu et al., 2020b) is the proportion of edges that connect two nodes of the same class: h edge =
Revised: The edge homophily (Zhu et al., 2020b) is the proportion of edges that connect two nodes of the same class: h edge =

Original: |{ ( v i ,v j ):( v i ,v j ) ∈E∧ y i =
Revised: |{ ( v i ,v j ):( v i ,v j ) ∈E∧ y i =

Original: y j }| E , And the node homophily (Pei et al., 2020) is defined as, h node = 1 N (cid:80) v i ∈V |{ v j :( v i ,v j ) ∈E∧ y i =
Revised: y j }| E , And the node homophily (Pei et al., 2020) is defined as, h node = 1 N (cid:80) v i ∈V |{ v j :( v i ,v j ) ∈E∧ y i =

Original: y j }| |{ v j :( v i ,v j ) ∈E}| , which evaluates the average proportion of edge-label consistency of all nodes.
Revised: y j }| |{ v j :( v i ,v j ) ∈E}| , which evaluates the average proportion of edge-label consistency of all nodes.

Original: They are all in the range of [0 , 1] and a value close to 1 corresponds to strong homophily while a value close to 0 indicates strong heterophily.
Revised: They are all in the range of [0 , 1] and a value close to 1 corresponds to strong homophily while a value close to 0 indicates strong heterophily.

Original: As conventional, we refer the graph with high homophily degree as homophilic graph, and the graph with  low homophily degree as heterophilic graph.
Revised: As conventional, we refer the graph with high homophily degree as homophilic graph, and the graph with a low homophily degree as heterophilic graph.

Original: And we provided the homophily degree of graph considered in this work in Table 5 of Appendix A.1.
Revised: And we provided the homophily degree of graph considered in this work in Table 5 of Appendix A.1.

Original:  T HEORETICAL ANALYSES
Revised:  T HEORETICAL ANALYSES

Original: In this section, we firstly analyze the property of node representations obtained through the neighbor aggregation (Lemma 1).
Revised: In this section, we firstly analyze the property of node representations obtained through the neighbor aggregation (Lemma 1).

Original: Then, based on the property, we introduce the single-pass graph contrastive loss (Equation (5)), which aims to learn an embedding space in which nodes from  same class are encoded close to each other while nodes from different classes are far apart.
Revised: Then, based on the property, we introduce the single-pass graph contrastive loss (Equation (5)), which aims to learn an embedding space in which nodes from the same class are encoded close to each other while nodes from different classes are far apart.

Original: After that, we bridge the contrastive loss and the Matrix Factorization (Lemma 2).
Revised: After that, we bridge the contrastive loss and the Matrix Factorization (Lemma 2).

Original: Leveraging the analysis for matrix factorization, we obtain the performance guarantee for the embedding learned with SP-GCL in the downstream  task (Theorem 2).
Revised: Leveraging the analysis for matrix factorization, we obtain the performance guarantee for the embedding learned with SP-GCL in the downstream node classification task (Theorem 2).

Original: 4. A NALYSIS OF AGGREGATED FEATURES
Revised: 4. A NALYSIS OF AGGREGATED FEATURES

Original: Assumptions on graph data.
Revised: Assumptions on graph data.

Original: To obtain analytic and conceptual insights of the aggregated features, we firstly describe the graph data we considered.
Revised: To obtain analytic and conceptual insights of the aggregated features, we firstly describe the graph data we considered.

Original: We assume that the node feature follows the
Revised: We assume that the node feature follows the

Original: Gaussian mixture model (Reynolds, 2009).
Revised: Gaussian mixture model (Reynolds, 2009).

Original: For simplicity, we focus on the binary classification problem.
Revised: For simplicity, we focus on the binary classification problem.

Original: Conditional on the (binary-) label y and a latent vector µ ∼ N ( 0 , I F /F ) where the identity matrix I F ∈ R F × F , the features are governed by:
Revised: Conditional on the (binary-) label y and a latent vector µ ∼ N ( 0 , I F /F ) where the identity matrix I F ∈ R F × F , the features are governed by:

Original: [Equation]
Revised: [Equation]

Original: where random variable q i ∈ R F has independent standard normal entries and y i ∈ {− 1 , 1 } representing latent classes with abuse of notation.
Revised: where random variable q i ∈ R F has independent standard normal entries and y i ∈ {− 1 , 1 } representing latent classes with abuse of notation.

Original: Then, the features of nodes with class y i follow the same distribution depending on y i , i.e. , x i ∼ P y
Revised: Then, the features of nodes with class y i follow the same distribution depending on y i , i.e. , x i ∼ P y

Original: i
Revised: i

Original: ( x ) .
Revised: ( x ) .

Original: Furthermore, we make an assumption on the neighborhood patterns, For node i , its neighbor’s labels are independently sampled from a distribution P ( y i ) .
Revised: Furthermore, we make an assumption about the neighborhood patterns, For node i , its neighbor’s labels are independently sampled from a distribution P ( y i ) .

Original: Remark.
Revised: Remark.

Original: The above assumption implies that the feature of a node depends  its label and the neighbor’s label is generated froma distribution only dependent on the label of the central node, which contains both cases of homophily and heterophily .
Revised: The above assumption implies that the feature of a node depends on its label and the neighbor’s label is generated from distribution only dependent on the label of the central node, which contains both cases of homophily and heterophily .

Original: With this assumption, we present the following Lemma 1, where we denote the learned embedding through the neighbor aggregation and a linear projection by Z , and Z i is the learned embedding with respect to input x i , and the W denotes the linear weight.
Revised: With this assumption, we present the following Lemma 1, where we denote the learned embedding through the neighbor aggregation and a linear projection by Z , and Z i is the learned embedding with respect to input x i , and the W denotes the linear weight.

Original: Lemma 1 (Concentration Property of Aggregated Features) Consider a graph G following the graph data assumption and Eq.
Revised: Lemma 1 (Concentration Property of Aggregated Features) Consider a graph G following the graph data assumption and Eq.

Original: (1), then the expectation of embedding is given by
Revised: (1), then the expectation of embedding is given by

Original: [Equation]
Revised: [Equation]

Original: Furthermore, with  probability at least 1 − δ over the distribution for  graph, we have:
Revised: Furthermore, with a probability at least 1 − δ over the distribution for the graph, we have:

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: where D ii is the degree of node v i , D ≡ min i D ii , and the sub-gaussian norms ∥ x ∥ ψ 2 ≡ min i ∥ x i,d
Revised: where D ii is the degree of node v i , D ≡ min i D ii , and the sub-gaussian norms ∥ x ∥ ψ 2 ≡ min i ∥ x i,d

Original: ∥ ψ 2 , sub-exponential norms ∥ x 2 ∥ ψ 1 ≡ min i ∥ x 2 i,d ∥ ψ 1 for d ∈ [1 , F ] .
Revised: ∥ ψ 2 , sub-exponential norms ∥ x 2 ∥ ψ 1 ≡ min i ∥ x 2 i,d ∥ ψ 1 for d ∈ [1 , F ] .

Original: Besides, σ 2max ( W ) is the largest singular value of W .
Revised: Besides, σ 2max ( W ) is the largest singular value of W .

Original: Remark.
Revised: Remark.

Original: Extending from Theorem 1 in (Ma et al., 2021), the above lemma indicates that, for any graph following the graph data assumption, (i) in expectation, nodes with the same label have the same embedding, Equation (2); (ii) the embeddings of nodes with  same label tends to concentrate onto a certain area in the embedding space, which can be regarded as the inductive bias of the neighborhood aggregation mechanism with the graph data assumption, Equation (3); (iii) the inner product of hidden representations approximates to its expectation with a high probability, Equation (4); (iv) with commonly used initialization, e.g. Kaiming initialization (He et al., 2015) and
Revised: Extending from Theorem 1 in (Ma et al., 2021), the above lemma indicates that, for any graph following the graph data assumption, (i) in expectation, nodes with the same label have the same embedding, Equation (2); (ii) the embeddings of nodes with the same label tends to concentrate onto a certain area in the embedding space, which can be regarded as the inductive bias of the neighborhood aggregation mechanism with the graph data assumption, Equation (3); (iii) the inner product of hidden representations approximates to its expectation with a high probability, Equation (4); (iv) with commonly used initialization, e.g. Kaiming initialization (He et al., 2015) and

Original: Lecun initialization (LeCun et al., 2012), the σ 2max ( W ⊤ W ) is bounded and the concentration is relatively tight.
Revised: Lecun initialization (LeCun et al., 2012), the σ 2max ( W ⊤ W ) is bounded and the concentration is relatively tight.

Original: We leave the proof in Appendix B.1 and the discussion about theexistence of a poor initialization in Appendix B.5.
Revised: We leave the proof in Appendix B.1 and the discussion about the initialization in Appendix B.5.

Original: Although the Gaussian mixture modeling on the feature, the neighborhood patterns modeling on the graph structure and the linearization on the graph neural network are commonly adopted in several recent works for the theoretical analysis of GNNs (Deshpande et al., 2018; Baranwal et al., 2021; Ma et al., 2021), and their high-level conclusions still hold empirically for a wide range of real-world cases, we empirically verify the derived Concentration Property (Lemma 1) on the multi-class realworld graph data and the non-linear graph neural network ( Section 6.3 and Table 4b) and provide a discussion about the empirical observations of the concentration property in previous works (Wang et al., 2022; Trivedi et al., 2022)
Revised: Although the Gaussian mixture modeling on the feature, the neighborhood patterns modeling on the graph structure and the linearization on the graph neural network are commonly adopted in several recent works for the theoretical analysis of GNNs (Deshpande et al., 2018; Baranwal et al., 2021; Ma et al., 2021), and their high-level conclusions still hold empirically for a wide range of real-world cases, we empirically verify the derived Concentration Property (Lemma 1) on the multiclass real-world graph data and the non-linear graph neural network ( Section 6.3 and Table 4b) and provide a discussion about the empirical observations of the concentration property in other recent works (Wang et al., 2022; Trivedi et al., 2022)

Original: (Appendix D.1).
Revised: (Appendix D.1).

Original: 4. S INGLE - PASS GRAPH CONTRASTIVE LOSS
Revised: 4. S INGLE - PASS GRAPH CONTRASTIVE LOSS

Original: In order to learn a more compact and linearly separable embedding space, we introduce the singlepass graph contrastive loss based on the property of aggregated features.
Revised: In order to learn a more compact and linearly separable embedding space, we introduce the singlepass graph contrastive loss based on the property of aggregated features.

Original: Exploiting the concentration property explicitly, we regard nodes with small distance in the embedding space as positive pairs, and nodes with large distance as negative pairs.
Revised: Exploiting the concentration property explicitly, we regard nodes with small distance in the embedding space as positive pairs, and nodes with large distance as negative pairs.

Original: We formally define the positive and negative pairs to introduce the loss.
Revised: We formally define the positive and negative pairs to introduce the loss.

Original: We draw a node v i uniformly from the node set V , v i ∼ Uni ( V ) , and draw the node v i + uniformly from the set S i , where the set S i is consisted by the K pos nodes closest to node v i .
Revised: We draw a node v i uniformly from the node set V , v i ∼ Uni ( V ) , and draw the node v i + uniformly from the set S i , where the set S i is consisted by the K pos nodes closest to node v i .

Original: Concretely, S i
Revised: Concretely, S i

Original: = { v 1 i , v 2 i , . .
Revised: = { v 1 i , v 2 i , . .

Original: . , v K pos i } = 
Revised: , v K pos i } = arg max

Original: [Equation]
Revised: [Equation]

Original: Z
Revised: j

Original: [Equation]
Revised: [Equation]

Original:  Z
Revised: (cid:0) Z

Original: [Equation]
Revised: [Equation]

Original: [Equation] (cid:1) , where
Revised: Z j , K pos (cid:1) , where

Original: K pos ∈
Revised: K pos ∈

Original: Z + and arg max( · , K pos ) denotes the operator for the top- K pos selection.
Revised: Z + and arg max( · , K pos ) denotes the operator for the top- K pos selection.

Original: The two sampled nodes form a positive pair ( v i , v i + ) .
Revised: The two sampled nodes form a positive pair ( v i , v i + ) .

Original: Two nodes v j and v k , sampled independently from the node set, can be regarded as a negative pair ( v j , v k ) (Arora et al., 2019).
Revised: Two nodes v j and v k , sampled independently from the node set, can be regarded as a negative pair ( v j , v k ) (Arora et al., 2019).

Original: Following the insight of Contrastive
Revised: Following the insight of Contrastive

Original: Learning (Wang & Isola, 2020), similar sample pairs stay close to each other while dissimilar ones are far apart, the Single-Pass Graph Contrastive Loss is defined as,
Revised: Learning (Wang & Isola, 2020), similar sample pairs stay close to each other while dissimilar ones are far apart, the Single-Pass Graph Contrastive Loss is defined as,

Original: [Equation]
Revised: [Equation]

Original: 4. P ERFORMANCE G UARANTEE FOR L EARNING L INEAR C LASSIFIER
Revised: 4. P ERFORMANCE G UARANTEE FOR L EARNING L INEAR C LASSIFIER

Original: Firstly, we  introduce the concept of transformed graph as follows, which is constructed based on the original graph and the selected positive pairs.
Revised: For the convenience of analysis, we firstly introduce the concept of transformed graph as follows, which is constructed based on the original graph and the selected positive pairs.

Original: Definition 1 (Transformed Graph)
Revised: Definition 1 (Transformed Graph)

Original: Given the original graph G and its node set V , the transformed graph, (cid:98) G , has the same node set V but with the selected positive pairs as the edge set, (cid:98) 
Revised: Given the original graph G and its node set V , the transformed graph, (cid:98) G , has the same node set V but with the selected positive pairs as the edge set, (cid:98) E = (cid:83) i { ( v i , v ki )

Original: [Equation]
Revised: | Kk =1 } . Note, the transformed graph is formed by positive pairs selected based on aggregated features. Coupling with the Concentration Property of Aggregated Features (Lemma 1), the transformed graph tends to have a larger homophily degree than the original graph. We provide more empirical verifications in Section 6.3.

Original: The transformed graph is illustrated in Figure 1.
Revised: The transformed graph is illustrated in Figure 1.

Original: We denote the adjacency matrix of transformed graph as (cid:98)
Revised: We denote the adjacency matrix of transformed graph as (cid:98)

Original: A ∈ { 0 , 1 } N × N , the number of edges as ˆ E , and the symmetric normalized matrix as (cid:98)
Revised: A ∈ { 0 , 1 } N × N , the number of edges as ˆ E , and the symmetric normalized matrix as (cid:98)

Original: A sym = (cid:98) D − 1 / 2 (cid:98) A (cid:98) D − 1 / 2 , where (cid:98) D
Revised: A sym = (cid:98) D − 1 / 2 (cid:98) A (cid:98) D − 1 / 2 , where (cid:98) D

Original: = diag (cid:0) ˆ d 1 , . . .
Revised: = diag (cid:0) ˆ d 1 , . . .

Original: , ˆ d N (cid:1) , ˆ d i = (cid:80) j (cid:98)
Revised: , ˆ d N (cid:1) , ˆ d i = (cid:80) j (cid:98)

Original: A i,j .
Revised: A i,j .

Original: Correspondingly, we denote the symmetric normalized Laplacian as (cid:98) L sym =
Revised: Correspondingly, we denote the symmetric normalized Laplacian as (cid:98) L sym =

Original: I − (cid:98)
Revised: I − (cid:98)

Original: A sym =
Revised: A sym =

Original: UΛU ⊤ .
Revised: UΛU ⊤ .

Original: Here U ∈ R N × N = [ u 1 , . . .
Revised: Here U ∈ R N × N = [ u 1 , . . .

Original: , u N ] , where u i ∈
Revised: , u N ] , where u i ∈

Original: R N denotes the i -th eigenvector of (cid:98) L sym and Λ = diag ( λ 1 , . . .
Revised: R N denotes the i -th eigenvector of (cid:98) L sym and Λ = diag ( λ 1 , . . .

Original: , λ N ) is the corresponding eigenvalue matrix.
Revised: , λ N ) is the corresponding eigenvalue matrix.

Original: λ 1 and λ N be the smallest and largest eigenvalue respectively.
Revised: λ 1 and λ N be the smallest and largest eigenvalue respectively.

Original: Then we show that optimizing a model with the contrastive loss (Equation (5)) is equivalent to the matrix factorization over the transformed graph, as stated in the following lemma:
Revised: Then we show that optimizing a model with the contrastive loss (Equation (5)) is equivalent to the matrix factorization over the transformed graph, as stated in the following lemma:

Original: Lemma 

Revised: Lemma

Original: Denote the learnable embedding for matrix factorization as F ∈ R N × K .
Revised: Denote the learnable embedding for matrix factorization as F ∈ R N × K .

Original: Let F i =
Revised: Let F i =

Original: F ψ ( v i ) .
Revised: F ψ ( v i ) .

Original: Then, the matrix factorization loss function L mf is equivalent to the contrastive loss, Equation (5) , up to an additive constant:
Revised: Then, the matrix factorization loss function L mf is equivalent to the contrastive loss, Equation (5) , up to an additive constant:

Original: [Equation]
Revised: [Equation]

Original: The above lemma bridges the graph contrastive learning and the graph matrix factorization and therefore allows us to provide the performance guarantee of SP-GCL by leveraging the power of matrix factorization.
Revised: The above lemma bridges the graph contrastive learning and the graph matrix factorization and therefore allows us to provide the performance guarantee of SP-GCL by leveraging the power of matrix factorization.

Original: We leave the derivation in Appendix B.2.
Revised: We leave the derivation in Appendix B.2.

Original: With the previous lemmas, we arrive at a conclusion about the expected value of  inner product of embedding:
Revised: With the previous lemmas, we arrive at a conclusion about the expected value of the inner product of embeddings:

Original: Theorem 1 For a graph G following the graph data assumption, then when the optimal of the contrastive loss is achieved, i.e., L mf ( F ∗ ) = 0 , we have,
Revised: Theorem 1 For a graph G following the graph data assumption, then when the optimal of the contrastive loss is achieved, i.e., L mf ( F ∗ ) = 0 , we have,

Original: [Equation]
Revised: [Equation]

Original: where ¯ ϕ =
Revised: where ¯ ϕ =

Original: E v i ,v j ∼ Uni ( V ) (cid:0) (cid:98)
Revised: E v i ,v j ∼ Uni ( V ) (cid:0) (cid:98)

Original: A i,j · 1 [ y i ̸
Revised: A i,j · 1 [ y i ̸

Original: = y j ] (cid:1) .
Revised: = y j ] (cid:1) .

Original: Remark.
Revised: Remark.

Original: The theorem shows that the embedding inner product of nodes from the same class is larger than the inner product of nodes from different classes.
Revised: The theorem shows that the embedding inner product of nodes from the same class is larger than the inner product of nodes from different classes.

Original: Besides, the 1 − ¯ ϕ , indicating the probability of an edge connecting two nodes from the same class, can be regarded as the edge homophily of the transformed graph.
Revised: Besides, the 1 − ¯ ϕ , indicating the probability of an edge connecting two nodes from the same class, can be regarded as the edge homophily of the transformed graph.

Original: Therefore, the theorem implies that if the edge homophily of the transformed graph is larger, embeddings of nodes from  same class will be more compact in the high dimensional space.
Revised: Therefore, the theorem implies that if the edge homophily of the transformed graph is larger, embeddings of nodes from the same class will be more compact in the high dimensional space.

Original: Finally, we can provide a performance guarantee for node embeddings learned by SP-GCL with a linear classifier in the downstream task:
Revised: Finally, we can provide a performance guarantee for node embeddings learned by SP-GCL with a linear classifier in the downstream task:

Original: Theorem 2 Let f ∗ SP - GCL ∈ arg min f : X→ R K be a minimizer of the contrastive loss, L SP - GCL .
Revised: Theorem 2 Let f ∗ SP - GCL ∈ arg min f : X→ R K be a minimizer of the contrastive loss, L SP - GCL .

Original: Then there exists
Revised: Then there exists

Original: a linear classifier B ∗ ∈ R c × K with norm ∥ B ∗ ∥ F ≤ 1 / (1 − λ K ) such that, with a probability at least 1 − δ ,
Revised: a linear classifier B ∗ ∈ R c × K with norm ∥ B ∗ ∥ F ≤ 1 / (1 − λ K ) such that, with a probability at least 1 − δ ,

Original: [Equation]
Revised: [Equation]

Original: ˆ λ i are the i smallest eigenvalues of the symmetrically normalized Laplacian matrix of the transformed graph.
Revised: ˆ λ i are the i smallest eigenvalues of the symmetrically normalized Laplacian matrix of the transformed graph.

Original: Remark.

Original: The above theorem implies that if the transformed graph has a larger homophily degree (smaller ¯ ϕ ), the bound of the prediction error will be lower.

Original: In other words, in positive pairs, if the percentage of two nodes belonging to the same class is higher, the obtained node embedding tends to have better performance in the downstream classification task.

Original: Besides, the theorem reveals a trend that with the increase of hidden representation dimension K , a lower bound will be obtained.

Original:  S INGLE - PASS GRAPH CONTRASTIVE LEARNING (SP-GCL)
Revised:  S INGLE - PASS GRAPH CONTRASTIVE LEARNING (SP-GCL)

Original: A direct consequence of our theory is a new method.
Revised: As a direct consequence of our theory, we introduce the Single-Pass Graph Contrastive Learning (SP-GCL) to verify our findings.

Original: Instead of relying on the graph augmentation function or the exponential moving average, we design a new learning frameworkwhich only forward single time and the contrastive pairs are constructed based on the aggregated features.
Revised: Instead of relying on the graph augmentation function or the exponential moving average, our new learning framework only forwards single time and the contrastive pairs are constructed based on the aggregated features.

Original: As we shall see, this exceedingly simple, theory motivated method also yields better performance in practice compared to dual-pass graph contrastive learning methods on both homophilic and heterophilic graph benchmarks.
Revised: As we shall see, this exceedingly simple, theory motivated method also yields better performance in practice compared to dual-pass graph contrastive learning methods on both homophilic and heterophilic graph benchmarks.

Original: The above theoretical analysis reveals that, for each class, the embedding obtained from neighbor aggregation will concentrate toward the expectation of embedding belonging to the class.
Revised: The above theoretical analysis reveals that, for each class, the embedding obtained from neighbor aggregation will concentrate toward the expectation of embedding belonging to the class.

Original: Inspired by this, we design the self-supervision signal based on the obtained embedding and propose a novel single-pass graph contrastive learning framework, SP-GCL, which selects similar nodes as positive node pairs.
Revised: Inspired by this, we design the self-supervision signal based on the obtained embedding and propose a novel single-pass graph contrastive learning framework, SP-GCL, which selects similar nodes as positive node pairs.

Original: In each iteration, the proposed framework firstly encodes the graph with a graph encoder f
Revised: In each iteration, the proposed framework firstly encodes the graph with a graph encoder f

Original: θ denoted by H = f θ ( X , A ) .
Revised: θ denoted by H = f θ ( X , A ) .

Original: Then, a projection head with L2 -normalization, g ω , is employed to project the node embedding into the hidden representation Z = g ω
Revised: Then, a projection head with L2 -normalization, g ω , is employed to project the node embedding into the hidden representation Z = g ω

Original: ( H ) .
Revised: ( H ) .

Original: At each iteration, b nodes are sampled as the seed node set S ; and their surrounding T -hop neighbors consist the node pool, P .
Revised: At each iteration, b nodes are sampled as the seed node set S ; and their surrounding T -hop neighbors consist the node pool, P .

Original: For each seed node v i ∈ S , the top- K pos nodes with highest similarity from the node pool are selected as positive set for it which denote as S ipos = arg max v j ∈ P (cid:0) Z ⊤ i Z j , K pos (cid:1)  .
Revised: For each seed node v i ∈ S , the top- K pos nodes with highest similarity from the node pool are selected as positive set for it which denote as S ipos = arg max v j ∈ P (cid:0) Z ⊤ i Z j , K pos (cid:1) , and K neg nodes are sampled from V to form the negative set S ineg , S ineg ⊆ V .

Original: Concretely, the framework is optimized with the following objective:
Revised: Concretely, the framework is optimized with the following objective:

Original: [Equation]
Revised: [Equation]

Original: 9) where S ineg is consisted by K neg nodes sampled from V , S ineg ⊆ V .

Original: Notably, the empirical contrastive loss is an unbiased estimation of the Equation (5).
Revised: Notably, the empirical contrastive loss is an unbiased estimation of the Equation (5).

Original: Overall, the training algorithm SP-GCL is summarized in Algorithm 1.
Revised: Overall, the training algorithm SP-GCL is summarized in Algorithm 1.

Original: As shown in the previous theoretical analysis, the performance of SP-GCL is independent with the homophily or heterophily assumption on the input graph, therefore SP-GCL is expected to generalize well on both homophilic and heterophilic graphs. And the single-pass design makes
Revised: Although we provide a theoretical discussion about the manner of self-selection for positive pairs and the proposed method, whether the method is effective and whether the self-selected manner is feasible for real-world cases are still not answered.

Original: SP-GCL get rid of the commonly adapted dual-branch design (Zhu et al., 2021a) and significantly reduce the computational overhead. We empirically verify the theoretical analysis in the Section 6.
Revised: In the following section, we empirically verify the effectiveness of the method and usefulness of our findings over a wide range of graph datasets.

Original: Input: Graph neural network f θ , MLP projection head g ω , input adjacency matrix A , node features X , batch size b , number of hops T , number of positive nodes K pos .
Revised: Input: Graph neural network f θ , MLP projection head g ω , input adjacency matrix A , node features X , batch size b , number of hops T , number of positive nodes K pos .

Original: for epoch ← 1, 2, · · · do
Revised: for epoch ← 1, 2, · · · do

Original: 1.Obtain the node embedding, H = f θ ( X , A ) .
Revised: Obtain the node embedding, H = f θ ( X , A ) .

Original: 2.Obtain the hiddenrepresentation, Z = g ω
Revised: Obtain hidden

Original:  ( H ) .
Revised: Z = g ω ( H ) .

Original: 3.Sample b nodesas seed node set S . 
Revised: Sample nodes seed node set S . 4.

Original: 4.Construct the node pool P with the T -hop neighbors of each node in the node set S .
Revised: Construct the node pool P with the T -hop neighbors of each node in the node set S .

Original: 5.Select top- K pos similar nodes for every v i ∈ S to form the positive node set S ipos .
Revised: Select top- K pos similar nodes for every v i ∈ S to form the positive node set S ipos .

Original: 6.Compute the contrastive objective with Eq.
Revised: Compute the contrastive objective with Eq.

Original: and update parameters by applying stochastic gradient.
Revised: and update parameters by applying stochastic gradient.

Original: end for return Final model f θ .

Original:  E XPERIMENTS 6. P ERFORMANCE ON HOMOPHILIC AND HETEROPHILIC GRAPH
Revised:  E XPERIMENTS 6. P ERFORMANCE ON HOMOPHILIC AND HETEROPHILIC GRAPH

Original: The homophilic graph benchmarks have been studied by several previous works (Velickovic et al., 2019;
Revised: The homophilic graph benchmarks have been studied by several previous works (Velickovic et al., 2019;

Original: Peng et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al., 2020d; Thakoor et al., 2021; Lee et al., 2021).
Revised: Peng et al., 2020; Hassani & Khasahmadi, 2020; Zhu et al., 2020d; Thakoor et al., 2021; Lee et al., 2021).

Original: We re-use their configuration and compare SP-GCL with those methods and leave the detailed description about the experiment setting in Appendix A.
Revised: We re-use their configuration and compare SP-GCL with those methods and leave the detailed description about the experiment setting in Appendix A.

Original: And we leave the implementation details and the hyperparameter selection in Appendix A.4.
Revised: And we leave the implementation details and the hyperparameter selection in Appendix A.4.

Original: The result is summarized in Table 1, in which the best performance achieved by self-supervised methods are marked in boldface.
Revised: The result is summarized in Table 1, in which the best performance achieved by self-supervised methods is marked in boldface.

Original: Compared with augmentation-based and augmentation-free GCL methods, SP-GCL outperforms previous methods on 2 datasets and achieves competitive performance on the others, which shows the effectiveness of the single-pass contrastive loss on homophilic graphs.
Revised: Compared with augmentation-based and augmentation-free GCL methods, SP-GCL outperforms previous methods on 2 datasets and achieves competitive performance on the others, which shows the effectiveness of the single-pass contrastive loss on homophilic graphs.

Original: We further assess the model performance on heterophilic graph benchmarks that introduced by Pei et al.
Revised: We further assess the model performance on heterophilic graph benchmarks that introduced by Pei et al.

Original: (Pei et al., 2020) and Lim et al.
Revised: (Pei et al., 2020) and Lim et al.

Original: (Lim et al., 2021a).
Revised: (Lim et al., 2021a).

Original: As shown in Table 2, SP-GCL achieves the best performance onof 6 heterophilic graphs by an evident margin.
Revised: As shown in Table 2, SP-GCL achieves the best performance on 6 of 6 heterophilic graphs by an evident margin.

Original: The above result indicates that, instead of relying on the augmentations which are sensitive to the graph type, SP-GCL is able to work well over a wide range of real-world graphs (described in Appendix A.1) with different homophily degree .
Revised: The above result indicates that, instead of relying on the augmentations which are sensitive to the graph type, SP-GCL is able to work well over a wide range of real-world graphs (described in Appendix A.1) with different homophily degree .

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 6. C OMPUTATIONAL COMPLEXITY ANALYSIS
Revised: 6. C OMPUTATIONAL COMPLEXITY ANALYSIS

Original: In order to illustrate the advantages of SP-GCL
Revised: In order to illustrate the advantages of SP-GCL

Original: [Table]
Revised: [Table]

Original: we provide a brief comparison of the time and space complexities between SP-GCL, the previous strong contrastive method, GCA (Zhu et al., 2021b), and the memory-efficient contrastive method, BGRL (Thakoor et al., 2021).
Revised: we provide a brief comparison of the time and space complexities between SP-GCL, the previous strong contrastive method, GCA (Zhu et al., 2021b), and the memory-efficient contrastive method, BGRL (Thakoor et al., 2021).

Original: Consider a graph with
Revised: Consider a graph with

Original: N nodes and E edges, and a graph neural network (GNN), f , that compute node embeddings in time and space O ( N + E ) .
Revised: N nodes and E edges, and a graph neural network (GNN), f , that compute node embeddings in time and space O ( N + E ) .

Original: BGRL performs four GNN computations per update step, in which twice for
Revised: BGRL performs four GNN computations per update step, in which twice for

Original: [Table]
Revised: [Table]

Original: [Table] [Table] 
Revised: [Table] [Table] [Table]

Original: [Table]
Revised: [Table]

Original: the target and online encoders, and twice for each augmentation, and a node-level projection; GCA performs two GNN computations (once for each augmentation), plus a node-level projection.
Revised: the target and online encoders, and twice for each augmentation, and a node-level projection; GCA performs two GNN computations (once for each augmentation), plus a node-level projection.

Original: Both methods backpropagate the learning signal twice (once for each augmentation), and we assume the backward pass to be approximately as costly as a forward pass.
Revised: Both methods backpropagate the learning signal twice (once for each augmentation), and we assume the backward pass to be approximately as costly as a forward pass.

Original: Both of them will compute the augmented graphs by feature masking and edge masking on the fly, the cost for augmentation computation is nearly the same.
Revised: Both of them will compute the augmented graphs by feature masking and edge masking on the fly, the cost for augmentation computation is nearly the same.

Original: Thus the total time and space complexity per update step for BGRL is 6 C encoder ( E + N )
Revised: Thus the total time and space complexity per update step for BGRL is 6 C encoder ( E + N )

Original: +4 C proj N + C prod N + C aug and 4 C encoder ( E + N )
Revised: +4 C proj N + C prod N + C aug and 4 C encoder ( E + N )

Original: +4 C proj N + C prod N 

Revised: +4 C proj N + C prod N

Original: +
Revised: +

Original: C aug for GCA.
Revised: C aug for GCA.

Original: The C prod depends on the dimension of node embedding and we assume the node embeddings of all the methods with the same size.
Revised: The C prod depends on the dimension of node embedding and we assume the node embeddings of all the methods with the same size.

Original: For our proposed method, only one GNN encoder is employed and we compute the inner product of b nodes to construct positive samples and K pos and K neg inner product for the loss computation.
Revised: For our proposed method, only one GNN encoder is employed and we compute the inner product of b nodes to construct positive samples and K pos and K neg inner product for the loss computation.

Original: Then for SP-GCL, we have: 2 C encoder ( E +
Revised: Then for SP-GCL, we have: 2 C encoder ( E +

Original: N ) + 2 C proj N + C prod
Revised: N ) + 2 C proj N + C prod

Original: ( K pos + K neg ) 2 .
Revised: ( K pos + K neg ) 2 .

Original: We empirically measure the peak of GPU memory usage of SP-GCL, GCA and BGRL.
Revised: We empirically measure the peak of GPU memory usage of SP-GCL, GCA and BGRL.

Original: As a fair comparison [Table] we set the embedding size as 128 for all those methods on the four datasets and keep the other hyper-parameters of the three methods the same as the main experiments.
Revised: As a fair comparison, we set the embedding size as 128 for all those methods on the four datasets and keep the other hyper-parameters of the three methods the same as the main experiments.

Original: As shown by Table 3a, the computational overhead of SP-GCL is much less than the previous methods .
Revised: As shown by Table 3a, the computational overhead of SP-GCL is much less than the previous methods .

Original: [Table]
Revised: [Table]

Original: 6. E MPIRICAL VERIFICATION FOR THE THEORETICAL ANALYSIS
Revised: 6. E MPIRICAL VERIFICATION FOR THE THEORETICAL ANALYSIS

Original: Effect of embedding dimension ( Scaling Behavior ).
Revised: Effect of embedding dimension ( Scaling Behavior ).

Original: We observe performance gains by scaling up models, as shown in Table 3b.
Revised: We observe performance gains by scaling up models, as shown in Table 3b.

Original: These observations are aligned with our analysis (Theorem 2) [Table] a larger hidden dimension leads to better performance.
Revised: These observations are aligned with our analysis (Theorem 2), a larger hidden dimension, K , leads to better performance (larger ˆ λ K +1 leads to a lower bound).

Original: Coupling with the efficiency property (Section 6.2) [Table] our scalable approach allows for learning high-capacity models that generalize well under the same computational requirement [Table]
Revised: Coupling with the efficiency property (Section 6.2), our scalable approach allows for learning high-capacity models that generalize well under the same computational requirement.

Original: Homophily of transformed graph ( Initial Stage ).
Revised: Homophily of transformed graph ( Initial Stage ).

Original: The positive sampling depends largely on how the hidden representation is obtained.
Revised: The positive sampling depends largely on how the hidden representation is obtained.

Original: In other words, if starting from a “poor” initialization, the
Revised: In other words, if starting from a “poor” initialization [Table]

Original:  GNN encoder could yield false positive samples since the inner product is not correctly evaluated in the beginning.
Revised: the GNN encoder could yield false positive samples since the inner product is not correctly evaluated in the beginning.

Original: Although, our analysis shows that the concentration property (Lemma 1) is relatively tight with commonly used initialization methods ( σ max ( W ⊤ W ) of the Equation (4) is bounded,
Revised: Although, our analysis shows that the concentration property (Lemma 1) is relatively tight with commonly used initialization methods ( σ max ( W ⊤ W ) of the Equation (4) is bounded,

Original: Appendix B.5)
Revised: Appendix B.5)

Original: and indicates that the transformed graph formed by selected positive pairs will have large edge homophily degree, we empirically measure the edge homophily of the transformed graph at the beginning of the training stage with 20 runs.
Revised: and indicates that the transformed graph formed by selected positive pairs will have large edge homophily degree, we empirically measure the edge homophily of the transformed graph at the beginning of the training stage with 20 runs.

Original: The mean and standard deviation are reported in Table 4b.
Revised: The mean and standard deviation are reported in Table 4b.

Original: The edge homophily of all transformed graphs are larger than the original one with a small standard deviation, except the CiteSeer in which a relatively high edge homophily (0.691) can still be achieved.
Revised: The edge homophily of all transformed graphs are larger than the original one with a small standard deviation, except the CiteSeer in which a relatively high edge homophily (0.691) can still be achieved.

Original: The result indicates that the initialization is “good” enough at the beginning stage to form a useful transformed graph and, in turn, support the feasibility of the Lemma 1 over the real-world data .
Revised: The result indicates that the initialization is “good” enough at the beginning stage to form a useful transformed graph and, in turn, support the feasibility of the Lemma 1 over the real-world data .

Original: Distance to class center and Node Coverage ( Learning Process ).
Revised: Distance to class center and Node Coverage ( Learning Process ).

Original: We measure change of the average cosine distance ( 1 − CosineSimilarity ) between the node embeddings and the class-center embeddings during training.
Revised: We measure change of the average cosine distance ( 1 − CosineSimilarity ) between the node embeddings and the class-center embeddings during training.

Original: Specifically, the class-center embedding is the average of the node embeddings of the same class.
Revised: Specifically, the class-center embedding is the average of the node embeddings of the same class.

Original: As shown in Figure 2a, we found that the node embeddings will concentrate on their corresponding class centers during training, which implies that the learned embedding space becomes more compact and the learning process is stable .
Revised: As shown in Figure 2a, we found that the node embeddings will concentrate on their corresponding class centers during training, which implies that the learned embedding space becomes more compact and the learning process is stable .

Original: Intuitively, coupling with the “good” initialization as discussed above, the SP-GCL iteratively leverage the inductive bias in the stable learning process and refine the embedding space.
Revised: Intuitively, coupling with the “good” initialization as discussed above, the SP-GCL iteratively leverage the inductive bias in the stable learning process and refine the embedding space.

Original: Furthermore, to study which nodes are benefited during the learning process, we measure the Node Cover Ratio and Overlapped
Revised: Furthermore, to study which nodes are benefited during the learning process, we measure the Node Cover Ratio and Overlapped

Original: Selection Ratio on four graph datasets.
Revised: Selection Ratio on four graph datasets.

Original: We denote the set of edges forming by the K pos positive selection at epoch t as e t and define the Overlapped Selection Ratio as e t +1 ∩ e t | e t +1 | .
Revised: We denote the set of edges forming by the K pos positive selection at epoch t as e t and define the Overlapped Selection Ratio as e t +1 ∩ e t | e t +1 | .

Original: Besides, we denote the set of positive nodes at at epoch t as v t .
Revised: Besides, we denote the set of positive nodes at at epoch t as v t .

Original: Then the Node Cover Ratio at epoch t is defined as:
Revised: Then the Node Cover Ratio at epoch t is defined as:

Original: [Equation]
Revised: [Equation]

Original: Following the hyperparameters described in the previous section, we measure the Node
Revised: Following the hyperparameters described in the previous section, we measure the Node

Original: Cover Ratio and Overlapped Selection Ratio during training.
Revised: Cover Ratio and Overlapped Selection Ratio during training.

Original: The results are shown in Figure 2b, which shows that all nodes are benefited from the optimization procedure .
Revised: The results are shown in Figure 2b, which shows that all nodes are benefited from the optimization procedure .

Original: Besides, we attribute the relatively small and non-increasing overlapped selection ratio to the batch training.
Revised: Besides, we attribute the relatively small and non-increasing overlapped selection ratio to the batch training.

Original: Selected positive pairs ( End of Training ).
Revised: Selected positive pairs ( End of Training ).

Original: To further study the effect of the proposed method, we provide the true positive ratio (TPR) of the selected pairs at the end of training with 20 runs.
Revised: To further study the effect of the proposed method, we provide the true positive ratio (TPR) of the selected pairs at the end of training with 20 runs.

Original: As shown in Table 4a, the relatively large TPR and low variance suggests that the quality of the selected positive pairs is good and the learning process is stable .
Revised: As shown in Table 4a, the relatively large TPR and low variance suggests that the quality of the selected positive pairs is good and the learning process is stable .

Original: (a) Average cosine distance between node embeddings and their corresponding class centers.
Revised: (a) Average cosine distance between node embeddings and their corresponding class centers.

Original: (b) The Node Cover Ratio and Overlapped Selection Ratio.
Revised: (b) The Node Cover Ratio and Overlapped Selection Ratio.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we firstly analyze the concentration property of embedding obtained through neighborhood aggregation which holds for both homophilic and heterophilic graph.
Revised: In this work, we firstly analyze the concentration property of embedding obtained through neighborhood aggregation which holds for both homophilic and heterophilic graphs.

Original: Then, exploiting the concentration property, we introduce the single-pass graph contrastive loss.
Revised: Then, exploiting the concentration property, we introduce the single-pass graph contrastive loss.

Original: We theoretically show that the equivalence between the contrastive objective and the matrix factorization.
Revised: We theoretically show that the equivalence between the contrastive objective and the matrix factorization.

Original: Further, leveraging the analysis of the matrix factorization, we provide a theoretical guarantee for the node embedding, obtained through minimizing the contrastive loss, in the downstream classification task.
Revised: Further, leveraging the analysis of the matrix factorization, we provide a theoretical guarantee for the node embedding, obtained through minimizing the contrastive loss, in the downstream classification task.

Original: As a direct consequence of the theory, we propose the Single-Pass Graph Contrastive Learning framework (SP-GCL) which only forwards single time in each iteration and the contrastive pairs are constructed based on the aggregated features.
Revised: To verify the usefulness of our findings in real-world datasets, we implement the Single-Pass Graph Contrastive Learning framework (SP-GCL).

Original: Empirically, we show that SP-GCL can outperform or be competitive with SOTA methods on 8 homophilic graph benchmarks and 6 heterophilic graph benchmarks with significantly less computational overhead.
Revised: Empirically, we show that SP-GCL can outperform or be competitive with SOTA methods on 8 homophilic graph benchmarks and 6 heterophilic graph benchmarks with significantly less computational overhead.

Revised: The empirical results verify the feasibility and effectiveness of our analysis in real-world cases.

Original: We leave the discussion about the connection with existing observation, limitation and future work inthe Appendix D.
Revised: We leave the discussion about the connection with existing observation, limitation and future work in Appendix D.

Revised:  R EPRODUCIBILITY S TATEMENT

Revised: To ensure the results and conclusions of our paper are reproducible, we make the following efforts:

Revised: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 4 and

Revised: Appendix B.

Revised: Experimentally, we provide our code, and instructions needed to reproduce the main experimental results.

Original:  A A PPENDIX : E XPERIMENT S ETTING A. D ATASET INFORMATION
Revised: And we specify all the training and implementation details in Section 6 and Appendix A. Besides, we independently run experiments and report the mean and standard deviation. A A PPENDIX : E XPERIMENT S ETTING A. D ATASET INFORMATION

Original: We analyze the quality of representations learned by SP-GCL on transductive node classification benchmarks.
Revised: We analyze the quality of representations learned by SP-GCL on transductive node classification benchmarks.

Original: Specifically, we evaluate the performance of using the pretraining representations on 8 benchmark homophilic graph datasets, namely, Cora, Citeseer, Pubmed (Kipf & Welling, 2016a) and Wiki-CS, Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics (Shchur et al., 2018), as well as 6 heterophilic graph datasets, namely, Chameleon, Squirrel, Actor (Pei et al., 2020), Twitch-DE, Twitch-gamers, and Genius (Lim et al., 2021b).
Revised: Specifically, we evaluate the performance of using the pretraining representations on 8 benchmark homophilic graph datasets, namely, Cora, Citeseer, Pubmed (Kipf & Welling, 2016a) and Wiki-CS, Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics (Shchur et al., 2018), as well as 6 heterophilic graph datasets, namely, Chameleon, Squirrel, Actor (Pei et al., 2020), Twitch-DE, Twitch-gamers, and Genius (Lim et al., 2021b).

Original: The datasets are collected from real-world networks from different domains; their detailed statistics are summarized in Table 5.
Revised: The datasets are collected from real-world networks from different domains; their detailed statistics are summarized in Table 5.

Original: For the 8 homophilic graph data, we use the processed version provided by PyTorch Geometric (Fey & Lenssen, 2019).
Revised: For the 8 homophilic graph data, we use the processed version provided by PyTorch Geometric (Fey & Lenssen, 2019).

Original: Besides, for the 6 heterophilic graph data, 3 of them, e.g., Chameleon, Squirrel and Actor are provided by PyTorch Geometric library.
Revised: Besides, for the 6 heterophilic graph data, 3 of them, e.g., Chameleon, Squirrel and Actor are provided by PyTorch Geometric library.

Original: The other three dataset, genius, twitch-DE and twitch-gamers can be obtained from the official github repository 1 , in which the standard splits for all the 6 heterophilic graph datasets can also be obtained.
Revised: The other three dataset, genius, twitch-DE and twitch-gamers can be obtained from the official github repository 1 , in which the standard splits for all the 6 heterophilic graph datasets can also be obtained.

Original: Those graph datasets follow the MIT license, and the personal identifiers are not included.
Revised: Those graph datasets follow the MIT license, and the personal identifiers are not included.

Original: We do not foresee any form of privacy issues.
Revised: We do not foresee any form of privacy issues.

Original: [Table]
Revised: [Table]

Original: A. B ASELINES
Revised: A. B ASELINES

Original: We consider representative baseline methods belonging to the following three categories (1) Traditional unsupervised graph embedding methods, including DeepWalk (Perozzi et al., 2014) and
Revised: We consider representative baseline methods belonging to the following three categories (1) Traditional unsupervised graph embedding methods, including DeepWalk (Perozzi et al., 2014) and

Original: Node2Vec (Grover & Leskovec, 2016) , (2) Self-supervised learning algorithms with graph neural networks including Graph Autoencoders (GAE, VGAE)
Revised: Node2Vec (Grover & Leskovec, 2016) , (2) Self-supervised learning algorithms with graph neural networks including Graph Autoencoders (GAE, VGAE)

Original: (Kipf & Welling, 2016b) , Deep Graph Infomax (DGI) (Velickovic et al., 2019) , Graphical Mutual Information Maximization (GMI) (Peng et al., 2020), and Multi-View Graph Representation Learning (MVGRL)
Revised: (Kipf & Welling, 2016b) , Deep Graph Infomax (DGI) (Velickovic et al., 2019) , Graphical Mutual Information Maximization (GMI) (Peng et al., 2020), and Multi-View Graph Representation Learning (MVGRL)

Original: (Hassani & Khasahmadi, 2020), graph contrastive representation learning (GRACE) (Zhu et al., 2020d)
Revised: (Hassani & Khasahmadi, 2020), graph contrastive representation learning (GRACE) (Zhu et al., 2020d)

Original: Graph Contrastive learning with Adaptive augmentation (GCA) (Zhu et al., 2021b), Bootstrapped Graph Latents (BGRL) (Thakoor et al., 2021), Augmentation-Free Graph Representation Learning (AFGRL)
Revised: Graph Contrastive learning with Adaptive augmentation (GCA) (Zhu et al., 2021b), Bootstrapped Graph Latents (BGRL) (Thakoor et al., 2021), Augmentation-Free Graph Representation Learning (AFGRL)

Original: (Lee et al., 2021), (3) Supervised learning and Semi-supervised learning, e.g., Multilayer Perceptron (MLP) and Graph Convolutional Networks (GCN) (Kipf & Welling, 2016a), where they are trained in an end-to-end fashion.
Revised: (Lee et al., 2021), (3) Supervised learning and Semi-supervised learning, e.g., Multilayer Perceptron (MLP) and Graph Convolutional Networks (GCN) (Kipf & Welling, 2016a), where they are trained in an end-to-end fashion.

Original: A. E VALUATION P ROTOCOL
Revised: A. E VALUATION P ROTOCOL

Original: We follow the evaluation protocol of previous state-of-the-art graph contrastive learning approaches.
Revised: We follow the evaluation protocol of previous state-of-the-art graph contrastive learning approaches.

Original: Specifically, for every experiment, we employ the linear evaluation scheme as introduced in (Velickovic et al., 2019), where each model is firstly trained in an unsupervised manner; then, the pretrained representations are used to train and test via a simple linear classifier.
Revised: Specifically, for every experiment, we employ the linear evaluation scheme as introduced in (Velickovic et al., 2019), where each model is firstly trained in an unsupervised manner; then, the pretrained representations are used to train and test via a simple linear classifier.

Original: For the datasets that came with standard train/valid/test splits, we evaluate the models on the public splits.
Revised: For the datasets that came with standard train/valid/test splits, we evaluate the models on the public splits.

Original: For datasets without standard split, e.g., Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics, we randomly split the datasets, where 10%/10%/80% of nodes are selected for the training, validation, and test set, respectively.
Revised: For datasets without standard split, e.g., Amazon-Computers, Amazon-Photo, Coauthor-CS, Coauthor-Physics, we randomly split the datasets, where 10%/10%/80% of nodes are selected for the training, validation, and test set, respectively.

Original: For most datasets, we report the averaged test accuracy and standard deviation over 10 runs of classification.
Revised: For most datasets, we report the averaged test accuracy and standard deviation over 10 runs of classification.

Original: While, following the previous works (Lim et al., 2021b;a), we report the test ROC AUC on genius and Twitch-DE datasets.
Revised: While, following the previous works (Lim et al., 2021b;a), we report the test ROC AUC on genius and Twitch-DE datasets.

Original: A. I MPLEMENTATION DETAILS
Revised: A. I MPLEMENTATION DETAILS

Original: Model architecture and hyperparamters.
Revised: Model architecture and hyperparamters.

Original: We employ a two-layer GCN (Kipf & Welling, 2016a) as the encoder for all methods.
Revised: We employ a two-layer GCN (Kipf & Welling, 2016a) as the encoder for all methods.

Original: The propagation for a single layer GCN is given by,
Revised: The propagation for a single layer GCN is given by,

Original: [Equation]
Revised: [Equation]

Original: where ¯ A =
Revised: where ¯ A =

Original: A
Revised: A

Original: + I is the adjacency matrix with self-loops, ¯ D is the degree matrix, σ is a nonlinear activation function, such as ReLU, and W i is the learnable weight matrix for the i ’th layer.
Revised: + I is the adjacency matrix with self-loops, ¯ D is the degree matrix, σ is a nonlinear activation function, such as ReLU, and W i is the learnable weight matrix for the i ’th layer.

Original: Besides, following the previous works (Lim et al., 2021a;b), we use batch normalization within the graph encoder for the heterophilic graphs.
Revised: Besides, following the previous works (Lim et al., 2021a;b), we use batch normalization within the graph encoder for the heterophilic graphs.

Original: The hyperparameters setting for all experiments are summarized in Table 6.
Revised: The hyperparameters setting for all experiments are summarized in Table 6.

Original: We would like to release our code after acceptance.
Revised: We would like to release our code after acceptance.

Original: Linear evaluation of embeddings.
Revised: Linear evaluation of embeddings.

Original: In the linear evaluation protocol, the final evaluation is over representations obtained from pretrained model.
Revised: In the linear evaluation protocol, the final evaluation is over representations obtained from pretrained model.

Original: When fitting the linear classifier on top of the frozen learned embeddings, the gradient will not flow back to the encoder.
Revised: When fitting the linear classifier on top of the frozen learned embeddings, the gradient will not flow back to the encoder.

Original: We optimize the one layer linear classifier 1000 epochs using Adam with learning rate 0.0005.
Revised: We optimize the one layer linear classifier 1000 epochs using Adam with learning rate 0.0005.

Original: Hardware and software infrastructures.
Revised: Hardware and software infrastructures.

Original: Our model are implemented with PyTorch Geometric 2.0.3 (Fey & Lenssen, 2019), PyTorch 1.9.0 (Paszke et al., 2017).
Revised: Our model are implemented with PyTorch Geometric 2.0.3 (Fey & Lenssen, 2019), PyTorch 1.9.0 (Paszke et al., 2017).

Original: We conduct experiments on a computer server with four NVIDIA Tesla V100 SXM2 GPUs (with 32GB memory each) and twelve
Revised: We conduct experiments on a computer server with four NVIDIA Tesla V100 SXM2 GPUs (with 32GB memory each) and twelve

Original: Intel Xeon Gold 6240R 2.40GHz CPUs.
Revised: Intel Xeon Gold 6240R 2.40GHz CPUs.

Original: [Table]
Revised: [Table]

Original: B A PPENDIX : D ETAILED PROOFS B. P ROOF OF LEMMA
Revised: B A PPENDIX : D ETAILED PROOFS B. P ROOF OF LEMMA

Original: Proof.
Revised: Proof.

Original: We first calculate the expectation of aggregated embedding:
Revised: We first calculate the expectation of aggregated embedding:

Original: [Equation]
Revised: [Equation]

Original: This equation is based on the graph data assumption such that x j ∼ P y
Revised: This equation is based on the graph data assumption such that x j ∼ P y

Original: i
Revised: i

Original: ( x ) for every j .
Revised: ( x ) for every j .

Original: Now we provide a concentration analysis.
Revised: Now we provide a concentration analysis.

Original: Because each feature x i is a sub-Gaussian variable, then by
Revised: Because each feature x i is a sub-Gaussian variable, then by

Original: Hoeffding’s inequality, with probability at least 1 − δ ′ for each d ∈ [1 , F ] , we have,
Revised: Hoeffding’s inequality, with probability at least 1 − δ ′ for each d ∈ [1 , F ] , we have,

Original: [Equation]
Revised: [Equation]

Original: where ∥ x j,d ∥ ψ 2 is sub-Gaussian norm of x j,d .
Revised: where ∥ x j,d ∥ ψ 2 is sub-Gaussian norm of x j,d .

Original: Furthermore, because each dimension of x j is i.i.d. , thus we have ∥ x j ∥ ψ 2 = ∥ x j,d ∥ ψ 2 , for d ∈ [1 , F ] .
Revised: Furthermore, because each dimension of x j is i.i.d. , thus we have ∥ x j ∥ ψ 2 = ∥ x j,d ∥ ψ 2 , for d ∈ [1 , F ] .

Original: Then we apply a union bound by setting δ ′ =
Revised: Then we apply a union bound by setting δ ′ =

Original: Fδ on the feature dimension k .
Revised: Fδ on the feature dimension k .

Original: Then with probability at least 1 − δ we have
Revised: Then with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: Next, we use the matrix perturbation theory,
Revised: Next, we use the matrix perturbation theory,

Original: [Equation]
Revised: [Equation]

Original: Finally, plug the weight matrix into the inequality,
Revised: Finally, plug the weight matrix into the inequality,

Original: [Equation]
Revised: [Equation]

Original: where σ max is the largest singular value of weight matrix.
Revised: where σ max is the largest singular value of weight matrix.

Original: Next, we perform a concentration analysis for the inner product.
Revised: Next, we perform a concentration analysis for the inner product.

Original: We first write down the detailed expression for each pair of i, j ,
Revised: We first write down the detailed expression for each pair of i, j ,

Original: [Equation]
Revised: [Equation]

Original: We first bound x ⊤ i x j .
Revised: We first bound x ⊤ i x j .

Original: Because x i and x j are independently sampled from an identical distribution,
Revised: Because x i and x j are independently sampled from an identical distribution,

Original: then the product x ⊤ i x j is sub-exponential.
Revised: then the product x ⊤ i x j is sub-exponential.

Original: This can been seen from Orilicz norms relation that ∥ x 2 ∥ ψ 1 = ( ∥ x 2 ∥ ψ 2 ) 2 , where ∥ x ∥ ψ 2 is sub-exponential norm of x 2 .
Revised: This can been seen from Orilicz norms relation that ∥ x 2 ∥ ψ 1 = ( ∥ x 2 ∥ ψ 2 ) 2 , where ∥ x ∥ ψ 2 is sub-exponential norm of x 2 .

Original: Then by the Hoeffding’s inequality for sub-exponential variable, with a probability at least 1 − δ , we have
Revised: Then by the Hoeffding’s inequality for sub-exponential variable, with a probability at least 1 − δ , we have

Original: [Equation]
Revised: [Equation]

Original: Because that the aggregated feature is normalized by the degree of corresponding node, we have, for each pair of i, j
Revised: Because that the aggregated feature is normalized by the degree of corresponding node, we have, for each pair of i, j

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: where D = min i D ii for i ∈ [1 , N ] .
Revised: where D = min i D ii for i ∈ [1 , N ] .

Original: Finally we apply a union bound over a pair of i, j .
Revised: Finally we apply a union bound over a pair of i, j .

Original: Then with probability at least 1 − δ we have
Revised: Then with probability at least 1 − δ we have

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF LEMMA
Revised: B. P ROOF OF LEMMA

Original: To prove this lemma, we first introduce the concept of the probability adjacency matrix.
Revised: To prove this lemma, we first introduce the concept of the probability adjacency matrix.

Original: For the transformed graph (cid:98) G , we denote its probability adjacency matrix as (cid:99) W , in which ˆ w ij = 1 (cid:98) E · (cid:98)
Revised: For the transformed graph (cid:98) G , we denote its probability adjacency matrix as (cid:99) W , in which ˆ w ij = 1 (cid:98) E · (cid:98)

Original: A ij .
Revised: A ij .

Original: ˆ w ij can be understood as the probability that two nodes have an edge and the weights sum to 1 because the total probability mass is 1: (cid:80) i,j ˆ w i,j ′
Revised: ˆ w ij can be understood as the probability that two nodes have an edge and the weights sum to 1 because the total probability mass is 1: (cid:80) i,j ˆ w i,j ′

Original: = 1 , for v i , v j ∈ V .
Revised: = 1 , for v i , v j ∈ V .

Original: The corresponding symmetric normalized matrix is (cid:99) W sym
Revised: The corresponding symmetric normalized matrix is (cid:99) W sym

Original: = (cid:98) D − 1 / 2 w (cid:99) W (cid:98) D − 1 / 2 w , and the (cid:98) D w = diag (cid:0) [ ˆ w 1 , . . .
Revised: = (cid:98) D − 1 / 2 w (cid:99) W (cid:98) D − 1 / 2 w , and the (cid:98) D w = diag (cid:0) [ ˆ w 1 , . . .

Original: , ˆ w N ] (cid:1) , where ˆ w i = (cid:80)
Revised: , ˆ w N ] (cid:1) , where ˆ w i = (cid:80)

Original: j ˆ w ij .
Revised: j ˆ w ij .

Original: We then introduce the Matrix Factorization Loss which is defined as:
Revised: We then introduce the Matrix Factorization Loss which is defined as:

Original: [Equation]
Revised: [Equation]

Original: By the classical theory on low-rank approximation, Eckart-Young-Mirsky theorem (Eckart & Young, 1936), any minimizer (cid:98) F of L mf ( F ) contains scaling of the smallest eigenvectors of L sym (also, the largest eigenvectors of (cid:98) A sym ) up to a right transformation for some orthonormal matrix
Revised: By the classical theory on low-rank approximation, Eckart-Young-Mirsky theorem (Eckart & Young, 1936), any minimizer (cid:98) F of L mf ( F ) contains scaling of the smallest eigenvectors of L sym (also, the largest eigenvectors of (cid:98) A sym ) up to a right transformation for some orthonormal matrix

Original: R ∈
Revised: R ∈

Original: R
Revised: R

Original: K
Revised: K

Original: ×
Revised: ×

Original: K .
Revised: K .

Original: We have (cid:98) F = F ∗ .
Revised: We have (cid:98) F = F ∗ .

Original: diag (cid:0)(cid:2) √ 1 − λ 1 , . . . , √ 1 − λ K (cid:3)(cid:1) R , where F ∗ =
Revised: diag (cid:0)(cid:2) √ 1 − λ 1 , . . . , √ 1 − λ K (cid:3)(cid:1) R , where F ∗ =

Original: [ u 1 , u 2 , · · · , u K ] ∈ R N × K .
Revised: [ u 1 , u 2 , · · · , u K ] ∈ R N × K .

Original: To proof the Lemma 2, we first present the Lemma 3.
Revised: To proof the Lemma 2, we first present the Lemma 3.

Original: Lemma 3 For transformed graph, its probability adjacency matrix (cid:99) W , and adjacency matrix (cid:98) A are equal after the symmetric normalization, (cid:99) W sym =
Revised: Lemma 3 For transformed graph, its probability adjacency matrix (cid:99) W , and adjacency matrix (cid:98) A are equal after the symmetric normalization, (cid:99) W sym =

Original: (cid:98)
Revised: (cid:98)

Original: A sym .
Revised: A sym .

Original: Proof.
Revised: Proof.

Original: For any two nodes v i , v j ∈ V and i ̸
Revised: For any two nodes v i , v j ∈ V and i ̸

Original: = j , we denote the the element in i -th row and j -th column of matrix (cid:99) W sym as (cid:99) W ijsym .
Revised: = j , we denote the the element in i -th row and j -th column of matrix (cid:99) W sym as (cid:99) W ijsym .

Original: [Equation]
Revised: [Equation]

Original: Leveraging the Lemma 3, we present the proof of Lemma 2.
Revised: Leveraging the Lemma 3, we present the proof of Lemma 2.

Original: Proof of Lemma 2 We start from the matrix factorization loss over (cid:98)
Revised: Proof of Lemma 2 We start from the matrix factorization loss over (cid:98)

Original: A sym to show the equivalence.
Revised: A sym to show the equivalence.

Original: [Equation]
Revised: [Equation]

Original: where f mf ( v i ) is the i -th row of the embedding matrix F .
Revised: where f mf ( v i ) is the i -th row of the embedding matrix F .

Original: The ˆ w i which can be understood as the node selection probability which is proportional to the node degree.
Revised: The ˆ w i which can be understood as the node selection probability which is proportional to the node degree.

Original: Then, we can define the corresponding sampling distribution as P deg .
Revised: Then, we can define the corresponding sampling distribution as P deg .

Original: If and only if √ w i · F ψ ( v i ) = f mf ( v i ) =
Revised: If and only if √ w i · F ψ ( v i ) = f mf ( v i ) =

Original: F i , the we have:
Revised: F i , the we have:

Original: [Equation]
Revised: [Equation]

Original: where N ( v i ) denotes the neighbor set of node v i and Uni ( · ) is the uniform distribution over the given set.
Revised: where N ( v i ) denotes the neighbor set of node v i and Uni ( · ) is the uniform distribution over the given set.

Original: Because we constructed the transformed graph by selecting the top- K pos nodes for each node, then all nodes have the same degree.
Revised: Because we constructed the transformed graph by selecting the top- K pos nodes for each node, then all nodes have the same degree.

Original: We can further simplify the objective as:
Revised: We can further simplify the objective as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Due to the node selection procedure, the factor √ w i is a constant and can be absorbed by the neural network, F ψ .
Revised: Due to the node selection procedure, the factor √ w i is a constant and can be absorbed by the neural network, F ψ .

Original: Then, because Z i = F ψ ( v i ) , we can have the Equation 23.
Revised: Then, because Z i = F ψ ( v i ) , we can have the Equation 23.

Original: Therefore, the minimizer of matrix factorization loss is equivalent with the minimizer of the contrastive loss.
Revised: Therefore, the minimizer of matrix factorization loss is equivalent with the minimizer of the contrastive loss.

Original: B. P ROOF OF THEOREM
Revised: B. P ROOF OF THEOREM

Original: Proof.
Revised: Proof.

Original: Now we provide the proof for the inner product of embedding.
Revised: Now we provide the proof for the inner product of embedding.

Original: In particular, when we can achieve near-zero contrastive learning loss, i.e. , L mf ( F ) = 0 , the minimizer F ∗ of L mf ( F ) contains scaling of the smallest eigenvectors of (cid:98) L sym (also, the largest eigenvectors of (cid:98) A sym ),
Revised: In particular, when we can achieve near-zero contrastive learning loss, i.e. , L mf ( F ) = 0 , the minimizer F ∗ of L mf ( F ) contains scaling of the smallest eigenvectors of (cid:98) L sym (also, the largest eigenvectors of (cid:98) A sym ),

Original: F ∗ =
Revised: F ∗ =

Original: [ u 1 , u 2 , · · · , u K ] ∈ R N × K , according to the Eckart-Young-Mirsky theorem (Eckart &
Revised: [ u 1 , u 2 , · · · , u K ] ∈ R N × K , according to the Eckart-Young-Mirsky theorem (Eckart &

Original: Young, 1936).
Revised: Young, 1936).

Original: Recall that y = { 1 , − 1 } N ∈ R N × 1 is label of all nodes.
Revised: Recall that y = { 1 , − 1 } N ∈ R N × 1 is label of all nodes.

Original: Then we show there is a constraint on the quadratic form with respect to the optimal classifier.
Revised: Then we show there is a constraint on the quadratic form with respect to the optimal classifier.

Original: According to graph spectral theory (Chung & Graham, 1997), the quadratic form
Revised: According to graph spectral theory (Chung & Graham, 1997), the quadratic form

Original: [Equation]
Revised: [Equation]

Original: which captures the amount of edges connecting different labels.
Revised: which captures the amount of edges connecting different labels.

Original: Furthermore, suppose that the expected homophily over distribution of graph feature and label, i.e., y ∼ P ( y i ) , x ∼ P y ( x ) , through similarity selection satisfies E [ h edge ( ˆ G )]
Revised: Furthermore, suppose that the expected homophily over distribution of graph feature and label, i.e., y ∼ P ( y i ) , x ∼ P y ( x ) , through similarity selection satisfies E [ h edge ( ˆ G )]

Original: = 1 − ¯ ϕ .
Revised: = 1 − ¯ ϕ .

Original: Here ¯ ϕ
Revised: Here ¯ ϕ

Original: = E v i ,v j ∼ Uni ( V ) (cid:0) (cid:98)
Revised: = E v i ,v j ∼ Uni ( V ) (cid:0) (cid:98)

Original: A i,j · 1 [ y i ̸
Revised: A i,j · 1 [ y i ̸

Original: = y j ] (cid:1) .
Revised: = y j ] (cid:1) .

Original: Since we have defined that ¯ ϕ as the density of edges that connects different labels, we can directly show that y ⊤ (cid:98) L sym y ≤ ¯ ϕN .
Revised: Since we have defined that ¯ ϕ as the density of edges that connects different labels, we can directly show that y ⊤ (cid:98) L sym y ≤ ¯ ϕN .

Original: Then we expand the above expression accoding to (cid:98) L sym =
Revised: Then we expand the above expression accoding to (cid:98) L sym =

Original: I − (cid:98)
Revised: I − (cid:98)

Original: A sym ,
Revised: A sym ,

Original: [Equation]
Revised: [Equation]

Original: Next we consider the situation that we achieve the optimal solution of the contrastive loss, Z = arg min L SP - GCL .
Revised: Next we consider the situation that we achieve the optimal solution of the contrastive loss, Z = arg min L SP - GCL .

Original: Then, we have L mf ( F ∗ ) = 0 , which implies that (cid:98) A sym =
Revised: Then, we have L mf ( F ∗ ) = 0 , which implies that (cid:98) A sym =

Original: ( F ∗ )
Revised: ( F ∗ )

Original: ⊤ F ∗ .
Revised: ⊤ F ∗ .

Original: As we analyzed in Section B.2, we have Z = F ∗ in this case.
Revised: As we analyzed in Section B.2, we have Z = F ∗ in this case.

Original: Furthermore, we have,
Revised: Furthermore, we have,

Original: [Equation]
Revised: [Equation]

Original: This leads to,
Revised: This leads to,

Original: [Equation]
Revised: [Equation]

Original: B. P ROOF OF THEOREM
Revised: B. P ROOF OF THEOREM

Original: Recently, HaoChen et al.
Revised: Recently, HaoChen et al.

Original: (2021) presented the following theoretical guarantee for the model learned with the matrix factorization loss.
Revised: (2021) presented the following theoretical guarantee for the model learned with the matrix factorization loss.

Original: Lemma 

Revised: Lemma

Original: For a graph G , let f ∗ mf ∈ arg min f mf : V→ R K be a minimizer of the matrix factorization loss, L mf ( F ) ,
Revised: For a graph G , let f ∗ mf ∈ arg min f mf : V→ R K be a minimizer of the matrix factorization loss, L mf ( F ) ,

Original: where F i = f mf ( v i ) .
Revised: where F i = f mf ( v i ) .

Original: Then, for any label y , there exists
Revised: Then, for any label y , there exists

Original: a linear classifier B ∗ ∈ R c × K with norm ∥ B ∗ ∥ F ≤ 1 / (1 − λ K ) such that
Revised: a linear classifier B ∗ ∈ R c × K with norm ∥ B ∗ ∥ F ≤ 1 / (1 − λ K ) such that

Original: [Equation]
Revised: [Equation]

Original: 26) where ⃗y i is the one-hot embedding of the label of node v i .
Revised: 26) where ⃗y i is the one-hot embedding of the label of node v i .

Original: The difference between labels of connected data points is measured by ϕ y , ϕ y :
Revised: The difference between labels of connected data points is measured by ϕ y , ϕ y :

Original: = 1 E (cid:80) v i ,v j ∈V A ij · 

Revised: = 1 E (cid:80) v i ,v j ∈V A ij ·

Original: [ y i ̸
Revised: [ y i ̸

Original: = y j ] .
Revised: = y j ] .

Original: Proof of Theorem 2.
Revised: Proof of Theorem 2.

Original: This proof is a direct summary on the established lemmas in previous section.
Revised: This proof is a direct summary on the established lemmas in previous section.

Original: By Lemma 2 and Lemma 4, we have,
Revised: By Lemma 2 and Lemma 4, we have,

Original: [Equation]
Revised: [Equation]

Original: (27) where ˆ λ i is the i -th smallest eigenvalue of the Laplacian matrix (cid:98) L sym
Revised: (27) where ˆ λ i is the i -th smallest eigenvalue of the Laplacian matrix (cid:98) L sym

Original: =
Revised: =

Original: I − (cid:98)
Revised: I − (cid:98)

Original: A sym .
Revised: A sym .

Original: Note that ϕ y in
Revised: Note that ϕ y in

Original: Lemma 4 equals 1 − h edge .
Revised: Lemma 4 equals 1 − h edge .

Original: Then we apply Theorem 1 and Lemma 1 for h edge to conclude the proof:
Revised: Then we apply Theorem 1 and Lemma 1 for h edge to conclude the proof:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: . I NFLUENCE OF RANDOM INITIALIZATION
Revised: . I NFLUENCE OF RANDOM INITIALIZATION

Original: Recall that the concentration inequality for inner product of embedding in Lemma 1 is:
Revised: Recall that the concentration inequality for inner product of embedding in Lemma 1 is:

Original: [Equation]
Revised: [Equation]

Original: As shown by Lemma 1, the accuracy of using the graph encoder to obtain the similarity between two nodes is determined by the maximum singular value of the weight of the graph encoder.
Revised: As shown by Lemma 1, the accuracy of using the graph encoder to obtain the similarity between two nodes is determined by the maximum singular value of the weight of the graph encoder.

Original: In other words, the quality of the transformed graph is guaranteed if we can be sure that the maximum singular value of the weight (of the graph encoder) is bounded.
Revised: In other words, the quality of the transformed graph is guaranteed if we can be sure that the maximum singular value of the weight (of the graph encoder) is bounded.

Original: When initializing the network, we generally use Gaussian initialization, such as, Kaiming and Lecun initialization (He et al., 2015;
Revised: When initializing the network, we generally use Gaussian initialization, such as, Kaiming and Lecun initialization (He et al., 2015;

Original: LeCun et al., 2012).
Revised: LeCun et al., 2012).

Original: Then W ⊤ W will be a Wishart matrix, and the limiting spectral density can be shown to be the Marchenko-Pastur distribution (Wigner, 1993), whose largest singular value is bounded when K/F is bounded, where W ∈ R F × K .
Revised: Then W ⊤ W will be a Wishart matrix, and the limiting spectral density can be shown to be the Marchenko-Pastur distribution (Wigner, 1993), whose largest singular value is bounded when K/F is bounded, where W ∈ R F × K .

Original: As a result, the quality of the transformed graph can be guaranteed with a high probability under condition that K/F is bounded.
Revised: As a result, the quality of the transformed graph can be guaranteed with a high probability under condition that K/F is bounded.

Original: On the other hand, we assume that for node i , its neighbor’s labels are independently sampled from a distribution P ( y i ) .
Revised: On the other hand, we assume that for node i , its neighbor’s labels are independently sampled from a distribution P ( y i ) .

Original: This means that the probability of an edge between two nodes can be fixed given labels.
Revised: This means that the probability of an edge between two nodes can be fixed given labels.

Original: In this case, the degree scales linearly with number of nodes N .
Revised: In this case, the degree scales linearly with number of nodes N .

Original: In the large graph limit, D will tends to infinity, thus our bound can be tighter.
Revised: In the large graph limit, D will tends to infinity, thus our bound can be tighter.

Original: C A UGMENTATIONS IN EXISTING WORKS
Revised: C A UGMENTATIONS IN EXISTING WORKS

Original: [Table]
Revised: [Table]

Original: (Zhu et al., 2020d)
Revised: 2020d

Original: [Table]
Revised: [Table]

Original: GCA (Zhu et al
Revised: (Grill et al

Original: [Table]

Original: , 2021b)

Original: [Table]
Revised: [Table]

Original: BGRL (Grill et al., 2020)

Original: [Table]
Revised: [Table]

Original: D D ISCUSSION D. C ONNECTION WITH EXISTING OBSERVATIONS
Revised: D D ISCUSSION D. C ONNECTION WITH EXISTING OBSERVATIONS

Original: In Lemma 1, we theoretically analyze the concentration property of aggregated features of graphs following the Graph Assumption (Section 4.1).
Revised: In Lemma 1, we theoretically analyze the concentration property of aggregated features of graphs following the Graph Assumption (Section 4.1).

Original: The concentration property has also been empirically observed in some other recent works (Wang et al., 2022; Trivedi et al., 2022).
Revised: The concentration property has also been empirically observed in some other recent works (Wang et al., 2022; Trivedi et al., 2022).

Original: Specifically, in the
Revised: Specifically, in the

Original: Figure 1 of Wang et al.
Revised: Figure 1 of Wang et al.

Original: (2022), a t-SNE visualization of node representations of the Amazon-Photo dataset shows that the representations obtained by a randomly initialized untrained SGC (Wu et al., 2019) model will concentrate to a certain area if they are from the same class.
Revised: (2022), a t-SNE visualization of node representations of the Amazon-Photo dataset shows that the representations obtained by a randomly initialized untrained SGC (Wu et al., 2019) model will concentrate to a certain area if they are from the same class.

Original: This observation also provides an explanation for the salient performance achieved by SP-GCL on the Amazon-Photo, as shown in Table 1.
Revised: This observation also provides an explanation for the salient performance achieved by SP-GCL on the Amazon-Photo, as shown in Table 1.

Original: D. L IMITATION AND FUTURE WORK
Revised: D. L IMITATION AND FUTURE WORK

Original: Our work is only the first step in understanding the possibility of single-pass graph contrastive learning and the corresponding performance guarantee on both the homophilic graph and heterophilic graph.
Revised: Our work is only the first step in understanding the possibility of single-pass graph contrastive learning and the corresponding performance guarantee on both the homophilic graph and heterophilic graph.

Original: There is still a lot of future work to be done.
Revised: There is still a lot of future work to be done.

Original: Below we indicate two questions that need to be addressed.
Revised: Below we indicate three questions that need to be addressed.

Original: • Our theoretical framework is built upon the graph assumption (Section 4.1) in which we assume the neighbor pattern.
Revised: • Our theoretical framework is built upon the graph assumption (Section 4.1) in which we assume the neighbor pattern.

Original: The graph assumption includes the homophilic graph and the “benign” heterophilic graph.
Revised: The graph assumption includes the homophilic graph and the “benign” heterophilic graph.

Original: However, there could exist graphs in which the neighbor pattern is messy or arbitrarily distributed.
Revised: However, there could exist graphs in which the neighbor pattern is messy or arbitrarily distributed.

Original: It is still an open question to understand the graph contrastive learning on those graphs.
Revised: It is still an open question to understand the graph contrastive learning on those graphs.

Original: • Recently, several Graph neural networks (Zhu et al., 2020b; Chien et al., 2020; Luan et al., 2021; Li et al., 2022) have been proposed to work on heterophilic graphs.
Revised: • Recently, several Graph neural networks (Zhu et al., 2020b; Chien et al., 2020; Luan et al., 2021; Li et al., 2022) have been proposed to work on heterophilic graphs.

Original: Whether further improvement can be achieved through combining with those GNNs is still an open question.
Revised: Whether further improvement can be achieved through combining with those GNNs is still an open question.

Revised: • To verify the effectiveness of our theoretical findings, we keep the implementation to be simple.

Revised: Whether a better performance can be achieved by involving more complex techniques still need to be explored.

Revised: [Figure]