Original: Spectrum Guided Topology Augmentation for Graph Contrastive Learning
Revised: Spectrum Guided Topology Augmentation for Graph Contrastive Learning

Original: Abstract
Revised: Abstract

Original: Graph contrastive learning (GCL) is a major self-supervised graph learning technique that aims to capture invariant properties of graphs via instance discrimination.
Revised: Graph contrastive learning (GCL) is a major self-supervised graph learning technique that aims to capture invariant properties of graphs via instance discrimination.

Original: Its performance heavily relies on the construction of multiple graph views yet itstill remains unclear about what makes effective topology augmentations .
Revised: Its performance heavily relies on the construction of multiple graph views yet it still remains unclear about what makes effective topology augmentations .

Original: Recentstudies mainly perform topology augmentations in a uniformly random mannerwithout considering graph properties.
Revised: Recent studies mainly perform topology augmentations in a uniformly random manner without considering graph properties.

Original: In this work, we aim to find principledways for topology augmentations by exploring the invariance of graphs from thegraph spectral perspective.
Revised: In this work, we aim to find principled ways for topology augmentations by exploring the invariance of graphs from the graph spectral perspective.

Original: Specifically, we propose a novel topology augmentationmethod guided by spectral change.
Revised: Specifically, we propose a novel topology augmentation method guided by spectral change.

Original: Extensive experiments on both graph and nodeclassification tasks demonstrate the effectiveness of our method in capturing thestructural essence of graphs for self-supervised learning.
Revised: Extensive experiments on both graph and node classification tasks demonstrate the effectiveness of our method in capturing the structural essence of graphs for self-supervised learning.

Original: The proposed method alsobrings promising performance in transfer learning and adversarial attack settings.
Revised: The proposed method also brings promising performance in transfer learning and adversarial attack settings.

Original: We envision this work to provide a principled way for graph augmentation.
Revised: We envision this work to provide a principled way for graph augmentation.

Original:  Introduction
Revised:  Introduction

Original: Graph neural networks (GNNs) [26, 52, 58] have advanced graph representation learning in a (semi-)supervised manner, yet it requires supervised labels and may fail to generalize due to overfitting [42].
Revised: Graph neural networks (GNNs) [26, 54, 60] have advanced graph representation learning in a (semi-) supervised manner, yet it requires supervised labels and may fail to generalize due to overfitting [43].

Original: To obtain more generalizable, transferable, and robust representations, the self-supervised learning(SSL) paradigm has recently emerged which enables GNNs to learn from pretext tasks constructedon unlabeled graph-structured data [21, 20, 61, 23].
Revised: To obtain more generalizable, transferable, and robust representations, the self-supervised learning (SSL) paradigm has recently emerged which enables GNNs to learn from pretext tasks constructed on unlabeled graph-structured data [21, 20, 63, 23].

Original: As the current state-of-the-art SSL technique,graph contrastive learning (GCL) has attracted the most attention due to its simplicity and remarkableempirical performance [53, 65, 17, 59, 47, 48].
Revised: As the current state-of-the-art SSL technique, graph contrastive learning (GCL) has attracted the most attention due to its simplicity and remarkable empirical performance [55, 67, 17, 61, 48, 49].

Original: A typical GCL method works by creating augmented views of the input graph and learning node(or graph) representations by contrasting related graph objects against unrelated ones.
Revised: A typical GCL method works by creating augmented views of the input graph and learning node (or graph) representations by contrasting related graph objects against unrelated ones.

Original: Differentcontrastive objects are studied, such as node-node [65, 66, 36], node-(sub)graph [54, 17,46] andgraph-graph [1, 48, 47]  contrastive pairs.
Revised: Different contrastive objects are studied, such as node-node [67, 68, 37], node-(sub)graph [56, 17, 47] and graph-graph [1, 49, 48] contrastive pairs.

Original: The goal of GCL is to maximize the congruence between therepresentations of graph objects in augmented views, following the mutual information maximization(InfoMax) principle [18].
Revised: The goal of GCL is to maximize the congruence between the representations of graph objects in augmented views, following the mutual information maximization (InfoMax) principle [18].

Original: This makes graph augmentation one of the most critical designs in GCL, asit determines the effectiveness of the contrastive objective.
Revised: This makes graph augmentation one of the most critical designs in GCL, as it determines the effectiveness of the contrastive objective.

Original: However, despite various GCL methods have been proposed, it remains a mystery about what makes the most effective graph augmentations.
Revised: However, despite various GCL methods have been proposed, it remains a mystery about what makes the most effective graph augmentations.

Original: Unlike images, which can be augmented by rotation or cropping to naturally highlight the mainsubject from the background, it is less intuitive and more challenging to augment graphs due to the complicated topology structure of diverse nature (e.g., citation networks [44], social networks [33],chemical and biomedical molecules [29, 20]).
Revised: Unlike images, which can be augmented by rotation or cropping to naturally highlight the main subject from the background, it is less intuitive and more challenging to augment graphs due to the complicated topology structure of diverse nature (e.g., citation networks [45], social networks [34], chemical and biomedical molecules [29, 20]).

Original: Most existing works perform topology augmentations in a uniformly random manner [60, 65, 48, 1].
Revised: Most existing works perform topology augmentations in a uniformly random manner [62, 67, 49, 1].

Original: Although such a strategy indeed achieves a certain levelof empirical success, it is far from optimal: recent studies show that perturbations on different edgespost unequal influence on the graph spectrum [8, 3] while the uniformly random edge perturbationadopted in many GCL methods treats all edges equally and ignores such differences.
Revised: Although such a strategy indeed achieves a certain level of empirical success, it is far from optimal: recent studies show that perturbations on different edges post unequal influence on the graph spectrum [8, 3] while the uniformly random edge perturbation adopted in many GCL methods treats all edges equally and ignores such differences.

Original: Given theimportance of graph spectrum for spectral filters in GNNs [6], such a discrepancy between uniformedge perturbations and their non-uniform influence on the graph spectrum urges us to rethink afundamental but not yet clearly answered question:
Revised: Given the importance of graph spectrum for spectral filters in GNNs [6], such a discrepancy between uniform edge perturbations and their non-uniform influence on the graph spectrum urges us to rethink a fundamental but not yet clearly answered question:

Original: Will graph spectrum based topology augmentations be more effective for GCL?
Revised: Will graph spectrum based topology augmentations be more effective for GCL?

Original: In this paper, we answer this question affirmatively.
Revised: In this paper, we answer this question affirmatively.

Original: By studying the influence of different edgeperturbation strategies on graph spectrum, we observe a clear positive relationship between the overallspectral change on augmented graphs and the resulting GCL performance.
Revised: By studying the influence of different edge perturbation strategies on graph spectrum, we observe a clear positive relationship between the overall spectral change on augmented graphs and the resulting GCL performance.

Original: This empirical findingfurther motivates us to propose a principled Graph Contrastive Learning scheme with Topology
Revised: This empirical finding further motivates us to propose a principled Graph Contrastive Learning scheme with Topology

Original: Augmentation guided by the Graph Spectrum, termed GCL-TAGS .
Revised: Augmentation guided by the Graph Spectrum, termed GCL-TAGS .

Original: Specifically, instead of perturbingedges uniformly at random, we search for graph augmentations that mostly change the graph spectrumof the input graph.
Revised: Specifically, instead of perturbing edges uniformly at random, we search for graph augmentations that mostly change the graph spectrum of the input graph.

Original: By identifying sensitive edges where the graph spectrum is largely affected, GCLTAGS allows the GNN encoder to focus on robust components (which can be hardly affected bysmall edge perturbations) in the spectral filters and to reduce its dependency on relatively vulnerablecomponents (which can be easily affected).
Revised: By identifying sensitive edges where the graph spectrum is largely affected, GCLTAGS allows the GNN encoder to focus on robust components (which can be hardly affected by small edge perturbations) in the spectral filters and to reduce its dependency on relatively vulnerable components (which can be easily affected).

Original: Therefore, the learned encoder captures the minimallysufficient information about the graph [50, 49] for the downstream tasks.
Revised: Therefore, the learned encoder captures the minimally sufficient information about the graph [51, 50] for the downstream tasks.

Original: We extensively evaluate GCL-TAGS on various benchmark datasets consisting of social networksand molecules, which cover commonly seen graph learning tasks such as node classification, graphclassification and regression.
Revised: We extensively evaluate GCL-TAGS on various benchmark datasets consisting of social networks and molecules, which cover commonly seen graph learning tasks such as node classification, graph classification and regression.

Original: We also consider various settings to test the applicability of ourproposed method including unsupervised learning, transfer learning and adversarial learning setting.
Revised: We also consider various settings to test the applicability of our proposed method including unsupervised learning, transfer learning and adversarial learning setting.

Original: In general, GCL-TAGS achieves remarkable performance gains compared to the state-of-the-artbaselines.
Revised: In general, GCL-TAGS achieves remarkable performance gains compared to the state-of-the-art baselines.

Original: Meanwhile, GCL-TAGS is easy to plug and play with different GCL paradigms as it onlyrequires a one-time pre-computation of the edge perturbation probability for topology augmentation.
Revised: Meanwhile, GCL-TAGS is easy to plug and play with different GCL paradigms as it only requires a one-time pre-computation of the edge perturbation probability for topology augmentation.

Original:  Related Works
Revised:  Related Works

Original: Existing graph SSL methods focus on self-prediction [14, 37, 13, 61, 23] and contrastive learning .
Revised: Existing graph SSL methods focus on self-prediction [14, 38, 13, 63, 23] and contrastive learning .

Original: We focus on contrastive learning, and will mainly discuss existing designs of topology augmentation.
Revised: We focus on contrastive learning, and will mainly discuss existing designs of topology augmentation.

Original: Graph Contrastive Learning (GCL) leverages the InfoMax principle [18] to maximize the correspondence between related objects on the graph such that invariant property across objects is captured.
Revised: Graph Contrastive Learning (GCL) leverages the InfoMax principle [18] to maximize the correspondence between related objects on the graph such that invariant property across objects is captured.

Original: Depending on how the positive objects are defined, one line of work treats different parts of a graphas positive pairs, while constructing negative examples from a corrupted graph [20, 22, 54, 36, 46].
Revised: Depending on how the positive objects are defined, one line of work treats different parts of a graph as positive pairs, while constructing negative examples from a corrupted graph [20, 22, 56, 37, 47].

Original: Insuch works, contrastive pairs are defined as nodes [54] or substructures [46] v.s. the entire graph, andthe input graph v.s. reconstructed graph [36].
Revised: In such works, contrastive pairs are defined as nodes [56] or substructures [47] v.s. the entire graph, and the input graph v.s. reconstructed graph [37].

Original: The other line of works exploits graph augmentation togenerate multiple views, which enable more flexible contrastive pairs [48, 1, 60, 17, 39, 47, 59, 9].
Revised: The other line of works exploits graph augmentation to generate multiple views, which enable more flexible contrastive pairs [49, 1, 62, 17, 40, 48, 61, 9].

Original: Bygenerating augmented views, the GNN model is encouraged to encode crucial graph information thatis invariant to different views.
Revised: By generating augmented views, the GNN model is encouraged to encode crucial graph information that is invariant to different views.

Original: While both topology and feature augmentations are explored in prior
Revised: While both topology and feature augmentations are explored in prior

Original: GCL works, we focus on topology augmentation strategies.
Revised: GCL works, we focus on topology augmentation strategies.

Original: As a parallel effort in self-supervisedlearning, augmentation-free techniques [28, 56] recently arise, which avoid augmentation but requiresspecial treatments (e.g., kNN search or clustering) to obtain positive and negative pairs.
Revised: As a parallel effort in self-supervised learning, augmentation-free techniques [28, 58] recently arise, which avoid augmentation but requires special treatments (e.g., kNN search or clustering) to obtain positive and negative pairs.

Original: Graph Topology Augmentation.
Revised: Graph Topology Augmentation.

Original: The most widely adopted topology augmentation is the edgeperturbation following uniform distribution [65, 48, 1, 60].
Revised: The most widely adopted topology augmentation is the edge perturbation following uniform distribution [67, 49, 1, 62].

Original: The underlying assumption is thateach edge is equally important to the property of the input graph.
Revised: The underlying assumption is that each edge is equally important to the property of the input graph.

Original: However, a recent study showsthat edge perturbations do not post equal influence to the graph spectrum [3] which summarizesa graph’s structural property.
Revised: However, a recent study shows that edge perturbations do not post equal influence to the graph spectrum [3] which summarizes a graph’s structural property.

Original: To better preserve graph property that has been ignored by uniformperturbations, domain knowledge from network science is leveraged by considering the importanceof edges measured via node centrality [66], the global diffusion matrix [17], and the random-walkbased context graph [39].
Revised: To better preserve graph property that has been ignored by uniform perturbations, domain knowledge from network science is leveraged by considering the importance of edges measured via node centrality [68], the global diffusion matrix [17], and the random-walk based context graph [40].

Original: While these works consider ad-hoc heuristics, our method targets thegraph spectrum, which comprehensively summarizes global graph properties and plays a crucial role in the spectral filter of GNNs.
Revised: While these works consider ad-hoc heuristics, our method targets the graph spectrum, which comprehensively summarizes global graph properties and plays a crucial role in the spectral filter of GNNs.

Original: To capture minimally sufficient information from the graph andremove redundancy that could compromise downstream performance, adversarial training strategyis paired with GCL for graph augmentation [47, 59, 9], following the information bottleneck (IB)[50] and InfoMin principle [49].
Revised: To capture minimally sufficient information from the graph and remove redundancy that could compromise downstream performance, adversarial training strategy is paired with GCL for graph augmentation [48, 61, 9], following the information bottleneck (IB) [51] and InfoMin principle [50].

Original: While the adversarial augmentation method requires frequentback-propagation during training, our method realizes a similar principle with a simpler but effectiveaugmentation by maximizing the spectral change of graph with only one-time pre-computation.
Revised: While the adversarial augmentation method requires frequent back-propagation during training, our method realizes a similar principle with a simpler but effective augmentation by maximizing the spectral change of graph with only one-time pre-computation.

Revised: Recent analyses also find some data-centric properties which are essential for GCL when using generic graph augmentations, yielding insights about the necessity of inducing task-relevant variance

Revised: [52] and maintaining the low-frequency information [31].

Revised: The finding of maximizing the spectral difference between views contributes a new dimension of such properties to investigate GCL.

Original:  Preliminaries
Revised:  Preliminaries

Original: Notations.
Revised: Notations.

Original: We focus on connected undirected graphs G =
Revised: We focus on connected undirected graphs G =

Original: ( X , A ) with n nodes and m edges, where
Revised: ( X , A ) with n nodes and m edges, where

Original: X ∈ R n × d describes node features, and A ∈ R n × n denotes its adjacency matrix such that A ij = 1 ifan edge exists between node i and j , otherwise A ij = 0 .
Revised: X ∈ R n × d describes node features, and A ∈ R n × n denotes its adjacency matrix such that A ij = 1 if an edge exists between node i and j , otherwise A ij = 0 .

Original: The unnormalized Laplacian matrix of thegraph is defined as L u = D − A , where D = diag ( A1 n ) is the diagonal degree matrix with entry
Revised: The unnormalized Laplacian matrix of the graph is defined as L u = D − A , where D = diag ( A1 n ) is the diagonal degree matrix with entry

Original: D ii = (cid:80) ni =

Revised: D ii = (cid:80) ni =

Original: A ij and 1 n being an all-one vector with dimension n .
Revised: A ij and 1 n being an all-one vector with dimension n .

Original: The normalized Laplacian matrixis further defined as L norm =
Revised: The normalized Laplacian matrix is further defined as L norm =

Original: Lap ( A ) =
Revised: Lap ( A ) =

Original: I n − D − 1 / 2 AD − 1 / 2 , where I n is an n × n identity matrix.
Revised: I n − D − 1 / 2 AD − 1 / 2 , where I n is an n × n identity matrix.

Original: Graph Spectrum.
Revised: Graph Spectrum.

Original: By treating node features as signals, one can apply graph signal processing(GSP) techniques to conduct graph filtering for representation learning.
Revised: By treating node features as signals, one can apply graph signal processing (GSP) techniques to conduct graph filtering for representation learning.

Original: The most essential component of GSP is the graph shift operator (GSO), which commonly adopts the normalized Laplacian matrix L norm and admits an eigendecomposition as L norm =
Revised: The most essential component of GSP is the graph shift operator (GSO), which commonly adopts the normalized Laplacian matrix L norm and admits an eigendecomposition as L norm =

Original: UΛU ⊤ .
Revised: UΛU ⊤ .

Original: The diagonal matrixΛ = eig ( L norm ) =
Revised: The diagonal matrix Λ = eig ( L norm ) =

Original: diag ( λ 1 , . . . , λ n ) consists of the real eigenvalues which are known as graphspectrum , and the corresponding U =
Revised: diag ( λ 1 , . . . , λ n ) consists of the real eigenvalues which are known as graph spectrum , and the corresponding U =

Original: [ u 1 , . . . , u n ] ∈ R n × n collecting the orthonormal eigenvectorsare the spectral bases .
Revised: [ u 1 , . . . , u n ] ∈ R n × n collecting the orthonormal eigenvectors are the spectral bases .

Original: Graph spectrum plays a significant role in analyzing and modeling graphs, asdiscussed in Appendix A.
Revised: Graph spectrum plays a significant role in analyzing and modeling graphs, as discussed in Appendix A.

Original: On one hand, it comprehensively summarizes important graph structuralproperties, including connectivity [5], clusterability [27] and diffusion distance [15].
Revised: On one hand, it comprehensively summarizes important graph structural properties, including connectivity [5], clusterability [27] and diffusion distance [15].

Original: On the otherhand, at the essence of many graph models, including GNNs, is the spectral filter which is definedon the graph spectrum, and different filters can manipulate graph signals in various ways, such assmoothing and denoising [43], anomaly detection [32] and clustering [55].
Revised: On the other hand, at the essence of many graph models, including GNNs, is the spectral filter which is defined on the graph spectrum, and different filters can manipulate graph signals in various ways, such as smoothing and denoising [44], anomaly detection [33] and clustering [57].

Original: Graph Representation Learning.
Revised: Graph Representation Learning.

Original: Given a graph G ∈ G , the goal of node representation learningis to train an encoder f θ : G → R n × d ′ , such that f θ ( G ) produces a low-dimensional vector foreach node in G which can be served in downstream tasks, such as node classification.
Revised: Given a graph G ∈ G , the goal of node representation learning is to train an encoder f θ : G → R n × d ′ , such that f θ ( G ) produces a low-dimensional vector for each node in G which can be served in downstream tasks, such as node classification.

Original: One canfurther obtain a graph representation by pooling the set of node representations via a readout functiong ϕ : R n × d ′ → R d ′ , such that g ϕ ( f θ ( G )) outputs a low-dimensional vector for graph G which can beused in graph-level tasks such as graph classification or regression task.
Revised: One can further obtain a graph representation by pooling the set of node representations via a readout function g ϕ : R n × d ′ → R d ′ , such that g ϕ ( f θ ( G )) outputs a low-dimensional vector for graph G which can be used in graph-level tasks such as graph classification or regression task.

Original: Graph Contrastive Learning by Topology Augmentation.
Revised: Graph Contrastive Learning by Topology Augmentation.

Original: GCL methods generally apply graphaugmentation to perturb the input graph and decrease the amount of information inherited from theoriginal graph; then they leverage the InfoMax principle [18] over the perturbed graph views such thatan encoder is trained to capture the remaining information [47].
Revised: GCL methods generally apply graph augmentation to perturb the input graph and decrease the amount of information inherited from the original graph; then they leverage the InfoMax principle [18] over the perturbed graph views such that an encoder is trained to capture the remaining information [48].

Original: Given a graph G ∈ G with adjacencymatrix A , we denote a topology augmentation scheme as T ( A ) and a sampled augmented view ast ( A ) ∼ T ( A ) .
Revised: Given a graph G ∈ G with adjacency matrix A , we denote a topology augmentation scheme as T ( A ) and a sampled augmented view as t ( A ) ∼ T ( A ) .

Original: GCL with two-branch augmentation can be formulated as the following problem:
Revised: GCL with two-branch augmentation can be formulated as the following problem:

Original: [Equation]
Revised: [Equation]

Original: where the contrastive loss L GCL measures the disagreement between representations from contrastivepositive pairs, which can be defined among different levels of representations, such as node-node[65], graph-graph [47], and node-graph [54, 17] representations.
Revised: where the contrastive loss L GCL measures the disagreement between representations from contrastive positive pairs, which can be defined among different levels of representations, such as node-node [67], graph-graph [48], and node-graph [56, 17] representations.

Original: The topology augmentation schemedetermines a distribution from which perturbed graphs are sampled in augmented views, and itsdetailed formulation will be presented in Section 4.2.
Revised: The topology augmentation scheme determines a distribution from which perturbed graphs are sampled in augmented views, and its detailed formulation will be presented in Section 4.2.

Original:  Graph Contrastive Learning Guided by Graph Spectrum
Revised:  Graph Contrastive Learning Guided by Graph Spectrum

Original: In this section, we introduce our graph contrastive learning framework with topology augmentationguided by the change of graph spectrum (GCL-TAGS).
Revised: In this section, we introduce our graph contrastive learning framework with topology augmentation guided by the change of graph spectrum (GCL-TAGS).

Original: We start with some empirical evidence on therelationship between the spectral change and GCL performance, then propose a new augmentationprinciple to maximize the spectral change when perturbing the graph.
Revised: We start with some empirical evidence on the relationship between the spectral change and GCL performance, then propose a new augmentation principle to maximize the spectral change when perturbing the graph.

Original: 4.1 Behavior of Edge Perturbation on Graph Spectrum
Revised: 4.1 Behavior of Edge Perturbation on Graph Spectrum

Original: Given that graph spectrum is a comprehensive manifestation of graph structural properties [5, 27, 15],to further understand its role in GCL, we study how the behavior of edge perturbation on graphspectrum correlates with GCL performance.
Revised: Given that graph spectrum is a comprehensive manifestation of graph structural properties [5, 27, 15], to further understand its role in GCL, we study how the behavior of edge perturbation on graph spectrum correlates with GCL performance.

Original: [Figure]
Revised: [Figure]

Original: Pre-analysis Setup.
Revised: Pre-analysis Setup.

Original: We take node representationlearning on Cora dataset as an example, and adopt thesame contrastive learning setup as in GRACE [65].
Revised: We take node representation learning on Cora dataset as an example, and adopt the same contrastive learning setup as in GRACE [67].

Original: We consider two topology augmentation heuristics:1) uniform : the original augmentation from GRACE[65] which removes edges with uniformly randomprobability; 2) clustered : a cluster-based strategywhich removes edges between different clusters witha larger probability.
Revised: We consider two topology augmentation heuristics: 1) uniform : the original augmentation from GRACE [67] which removes edges with uniformly random probability; 2) clustered : a cluster-based strategy which removes edges between different clusters with a larger probability.

Original: The cluster-based strategy explicitly modifies the connectivity between clusters whichmay result in large change on the graph spectrum assuggested by recent studies [4, 30], thus it stands insharp contrast to the uniform perturbations.
Revised: The cluster-based strategy explicitly modifies the connectivity between clusters which may result in large change on the graph spectrum as suggested by recent studies [4, 30], thus it stands in sharp contrast to the uniform perturbations.

Original: For thetwo augmentation branches, one is fixed with the uniform version, and we compare the strategieswhen the other augmentation branch adopts the uniform or the clustered perturbation.
Revised: For the two augmentation branches, one is fixed with the uniform version, and we compare the strategies when the other augmentation branch adopts the uniform or the clustered perturbation.

Original: Figure 1 showstheir comparison with respect to spectral change (measured by the L 2 distance of graph spectrumbetween the original and the augmented graphs), and F1 score (measuring the downstream nodeclassification performance).
Revised: Figure 1 shows their comparison with respect to spectral change (measured by the L 2 distance of graph spectrum between the original and the augmented graphs), and F1 score (measuring the downstream node classification performance).

Original: More experiment details are left in Appendix C.1.
Revised: More experiment details are left in Appendix C.1.

Original: Remarks.
Revised: Remarks.

Original: From Figure 1, we can clearly observe that under the same perturbation budget indicatedby x-axis, the cluster-based strategy leads to a larger change on graph spectrum, while achieving betterperformance on the downstream task.
Revised: From Figure 1, we can clearly observe that under the same perturbation budget indicated by x-axis, the cluster-based strategy leads to a larger change on graph spectrum, while achieving better performance on the downstream task.

Original: This analysis suggests that an effective edge augmentationshould pay more attention to sensitive edges that introduce large disturbance to graph spectrum.
Revised: This analysis suggests that an effective edge augmentation should pay more attention to sensitive edges that introduce large disturbance to graph spectrum.

Original: Theperformance gap between these two simple strategies suggests a distinct possibility to improve overthe uniformly random augmentation.
Revised: The performance gap between these two simple strategies suggests a distinct possibility to improve over the uniformly random augmentation.

Original: Unlike the proof-of-concept cluster-based heuristic, we aim ondesigning a principled topology augmentation by directly maximizing the spectral change.
Revised: Unlike the proof-of-concept cluster-based heuristic, we aim on designing a principled topology augmentation by directly maximizing the spectral change.

Original: 4.2 Augmentation Scheme via Spectral Change Maximization
Revised: 4.2 Augmentation Scheme via Spectral Change Maximization

Original: We now introduce our augmentation scheme guided by graph spectrum.
Revised: We now introduce our augmentation scheme guided by graph spectrum.

Original: We first define the edgeperturbation based topology augmentation scheme determined by a Bernoulli probability matrix .
Revised: We first define the edge perturbation based topology augmentation scheme determined by a Bernoulli probability matrix .

Original: Based on that, we formulate our augmentation principle as a spectral change maximization problem.
Revised: Based on that, we formulate our augmentation principle as a spectral change maximization problem.

Original: Edge Perturbation Based Augmentation Scheme.
Revised: Edge Perturbation Based Augmentation Scheme.

Original: We focus on topology augmentation using edgeperturbation.
Revised: We focus on topology augmentation using edge perturbation.

Original: Following the GCL formulation in Eq.
Revised: Following the GCL formulation in Eq.

Original: (1), we define topology augmentation T ( A ) as a
Revised: (1), we define topology augmentation T ( A ) as a

Original: Bernoulli distribution B (∆ ij ) for each entry A ij .
Revised: Bernoulli distribution B (∆ ij ) for each entry A ij .

Original: All Bernoulli parameters for all entries constitutea probability matrix ∆ ∈ [0 , 1] n × n .
Revised: All Bernoulli parameters for all entries constitute a probability matrix ∆ ∈ [0 , 1] n × n .

Original: We can sample an edge perturbation matrix E ∈ { 0 , 1 } n × n ,where E ij ∼ B (∆ ij ) indicates whether to flip the edge between node i and j , and the edge is flippedif E ij = 1 otherwise remaining unchanged.
Revised: We can sample an edge perturbation matrix E ∈ { 0 , 1 } n × n , where E ij ∼ B (∆ ij ) indicates whether to flip the edge between node i and j , and the edge is flipped if E ij = 1 otherwise remaining unchanged.

Original: A sampled augmented graph is then obtained via:
Revised: A sampled augmented graph is then obtained via:

Original: [Equation]
Revised: [Equation]

Original: where ¯A is the complement matrix of the adjacency matrix A , calculated by ¯A = 1 n 1 ⊤ n − I n − A ,with ( 1 n 1 ⊤ n − I n ) denoting the fully-connected graph without self-loops.
Revised: where ¯A is the complement matrix of the adjacency matrix A , calculated by ¯A = 1 n 1 ⊤ n − I n − A , with ( 1 n 1 ⊤ n − I n ) denoting the fully-connected graph without self-loops.

Original: Therefore, C =
Revised: Therefore, C =

Original: ¯ A − A ∈{− 1 , 1 } n × n denotes legitimate edge adding or removing operations for each node pair: edge addingbetween node i and j is allowed if C ij = 1 , and edge removing is allowed if C ij = − 1 .
Revised: ¯ A − A ∈ {− 1 , 1 } n × n denotes legitimate edge adding or removing operations for each node pair: edge adding between node i and j is allowed if C ij = 1 , and edge removing is allowed if C ij = − 1 .

Original: Taking the
Revised: Taking the

Original: Hadamard product C ◦ E finally gives valid edge perturbations to the graph.
Revised: Hadamard product C ◦ E finally gives valid edge perturbations to the graph.

Original: Since E is a matrix of random variables following Bernoulli distributions, we can easily obtain theexpectation of sampled augmented graphs in Eq.
Revised: Since E is a matrix of random variables following Bernoulli distributions, we can easily obtain the expectation of sampled augmented graphs in Eq.

Original: (2) as E [ t ( A )]
Revised: (2) as E [ t ( A )]

Original: =
Revised: =

Original: A + C ◦ ∆ .
Revised: A + C ◦ ∆ .

Original: Therefore, the designof ∆ determines the topology augmentation scheme.
Revised: Therefore, the design of ∆ determines the topology augmentation scheme.

Original: Taking uniformly random edge removal as anexample, the entry ∆ ij is set as a fixed dropout ratio if C ij = − 1 ; and 0 otherwise.
Revised: Taking uniformly random edge removal as an example, the entry ∆ ij is set as a fixed dropout ratio if C ij = − 1 ; and 0 otherwise.

Original: Spectral Change Maximization.
Revised: Spectral Change Maximization.

Original: The default uniform edge perturbation adopted in many GCLmethods is far from satisfactory.
Revised: The default uniform edge perturbation adopted in many GCL methods is far from satisfactory.

Original: Motivated by our observation in Section 4.1, instead of settingfixed values for ∆ , we propose to optimize it guided by graph spectrum.
Revised: Motivated by our observation in Section 4.1, instead of setting fixed values for ∆ , we propose to optimize it guided by graph spectrum.

Original: Specifically, we aim tosearch for ∆ that in expectation maximizes the spectral difference between the original graph and theaugmented graph.
Revised: Specifically, we aim to search for ∆ that in expectation maximizes the spectral difference between the original graph and the augmented graph.

Original: Recall that we denote the normalized Laplacian matrix of A as Lap ( A ) , and thegraph spectrum vector can be calculated by Λ = eig
Revised: Recall that we denote the normalized Laplacian matrix of A as Lap ( A ) , and the graph spectrum vector can be calculated by Λ = eig

Original: ( Lap ( A )) .
Revised: ( Lap ( A )) .

Original: We formulate the following problemto search for the desired perturbation matrix ∆ in a single augmentation branch:
Revised: We formulate the following problem to search for the desired perturbation matrix ∆ in a single augmentation branch:

Original: [Equation]
Revised: [Equation]

Original: where S = { s | s ∈ [0 , 1] n × n , ∥ s ∥ 1 ≤ ϵ } and ϵ controls the strength of graph perturbation.
Revised: where S = { s | s ∈ [0 , 1] n × n , ∥ s ∥ 1 ≤ ϵ } and ϵ controls the strength of graph perturbation.

Original: Bysolving Eq.
Revised: By solving Eq.

Original: (3), we obtain the optimal Bernoulli probability matrix ∆ ∗ , from which we can sampleaugmented views that in expectation differ the most from the original graph in graph spectrum.
Revised: (3), we obtain the optimal Bernoulli probability matrix ∆ ∗ , from which we can sample augmented views that in expectation differ the most from the original graph in graph spectrum.

Original: Notethat Eq.
Revised: Note that Eq.

Original: (3) only provides one augmented view, to further introduce flexibility for a two-branchaugmentation framework and enlarge the spectral difference between the resulting two views, weextend Eq.
Revised: (3) only provides one augmented view, to further introduce flexibility for a two-branch augmentation framework and enlarge the spectral difference between the resulting two views, we extend Eq.

Original: (3) as follows:
Revised: (3) as follows:

Original: [Equation]
Revised: [Equation]

Original: where ∆ i is the Bernoulli probability matrix for augmentation branch
Revised: where ∆ i is the Bernoulli probability matrix for augmentation branch

Original: i ’s scheme T
Revised: i ’s scheme T

Original: i ( A ) in Eq.
Revised: i ( A ) in Eq.

Original: Note that Eq.
Revised: Note that Eq.

Original: (3) is a special case of Eq.
Revised: (3) is a special case of Eq.

Original: (4) when setting ∆ 2 = 0 .
Revised: (4) when setting ∆ 2 = 0 .

Original: Eq.
Revised: Eq.

Original: (4) gives better flexibilityyet also makes the nonconvex optimization problem harder to solve, thus we further simplify it bypushing two branches towards opposite directions: maximizing the spectral norm in one branch,while minimizing it in the other, which leads to the final objective for our augmentation scheme:Opposite-direction scheme: max
Revised: (4) gives better flexibility yet also makes the nonconvex optimization problem harder to solve, thus we further simplify it by pushing two branches towards opposite directions: maximizing the spectral norm in one branch, while minimizing it in the other, which leads to the final objective for our augmentation scheme:

Original: [Equation]
Revised: [Equation]

Original: where L GS ( ∆ ) =
Revised: where L GS ( ∆ ) =

Original: ∥ eig ( Lap ( A + C ◦ ∆ ))
Revised: ∥ eig ( Lap ( A + C ◦ ∆ ))

Original: ∥ 22 measures the Graph Spectral norm under augmentationscheme with ∆ .
Revised: ∥ 22 measures the Graph Spectral norm under augmentation scheme with ∆ .

Original: For scheme T 1 ( A ) , ∆ 1 produces views that overall have larger spectral norm thanthe original graph, while for T 2 ( A ) , ∆ 2 produces views with smaller spectrum.
Revised: For scheme T 1 ( A ) , ∆ 1 produces views that overall have larger spectral norm than the original graph, while for T 2 ( A ) , ∆ 2 produces views with smaller spectrum.

Original: We can understandthem as setting a spectral boundary for the input graph such that the encoder is trained to captureinformation that is essential and robust regarding perturbations within this region.
Revised: We can understand them as setting a spectral boundary for the input graph such that the encoder is trained to capture information that is essential and robust regarding perturbations within this region.

Original: Optimizing ∆ 1 and ∆ 2 .
Revised: Optimizing ∆ 1 and ∆ 2 .

Original: Eq.
Revised: Eq.

Original: (5) can be solved via projected gradient descent (for ∆ 2 ) or ascent(for ∆ 1 ).
Revised: (5) can be solved via projected gradient descent (for ∆ 2 ) or ascent (for ∆ 1 ).

Original: Taking ∆ 2 as an example, its update works as follows:
Revised: Taking ∆ 2 as an example, its update works as follows:

Original: [Equation]
Revised: [Equation]

Original: where t is the iteration step, η t > 0 is the learning rate for step t , and P S ( a ) =
Revised: where t is the iteration step, η t > 0 is the learning rate for step t , and P S ( a ) =

Original: argmin s ∈S ∥ s − a ∥ 22is the projection operator at a over the constraint set S .
Revised: argmin s ∈S ∥ s − a ∥is the projection operator at a over the constraint set S .

Original: The calculation of gradient ∇L GS ( ∆ ( t − 1))is done via chain rule.
Revised: The calculation of gradient ∇L GS ( ∆ ( t − 1)) is done via chain rule.

Original: We now explain how to obtain a closed-form gradient over eigenvalues as itlooks less straightforward.
Revised: We now explain how to obtain a closed-form gradient over eigenvalues as it looks less straightforward.

Original: For a real and symmetric matrix L , one can obtain the derivatives of itsk -th eigenvalue λ k by:
Revised: For a real and symmetric matrix L , one can obtain the derivatives of its k -th eigenvalue λ k by:

Original: ∂λ k /∂ L = u k u ⊤ k [41], where u k is the corresponding eigenvector.
Revised: ∂λ k /∂ L = u k u ⊤ k [42], where u k is the corresponding eigenvector.

Original: Note thatthe derivative calculation requires distinct eigenvalues, which does not hold for graphs satisfyingautomorphism [10].
Revised: Note that the derivative calculation requires distinct eigenvalues, which does not hold for graphs satisfying automorphism [10].

Original: To avoid such cases, we add a small noise term to the adjacency matrix 1 , e.g.,
Revised: To avoid such cases, we add a small noise term to the adjacency matrix 1 , e.g.,

Original: A + C ◦ ∆ + ε × ( N + N ⊤ ) / 2 , where each entry in N is sampled from a uniform distribution
Revised: A + C ◦ ∆ + ε × ( N + N ⊤ ) / 2 , where each entry in N is sampled from a uniform distribution

Original: U (0 , 1) and ε is a very small constant.
Revised: U (0 , 1) and ε is a very small constant.

Original: Such a noise addition will almost surely break the graphautomorphism, thus enabling a valid gradient calculation of eigenvalues.
Revised: Such a noise addition will almost surely break the graph automorphism, thus enabling a valid gradient calculation of eigenvalues.

Original: For T iterations, the time complexity of optimizing such a scheme is O ( Tn 3 ) due to the eigendecomposition eig ( · ) in L GS , which is prohibitively expensive for large graphs.
Revised: For T iterations, the time complexity of optimizing such a scheme is O ( Tn 3 ) due to the eigendecomposition eig ( · ) in L GS , which is prohibitively expensive for large graphs.

Original: To reduce thecomputational cost, instead of measuring the spectral change over all eigenvalues, we only maintainthe K lowest- and highest-eigenvalues which are the most informative, as suggested by the spectralgraph theory.
Revised: To reduce the computational cost, instead of measuring the spectral change over all eigenvalues, we only maintain the K lowest- and highest-eigenvalues which are the most informative, as suggested by the spectral graph theory.

Original: The performance with different choices of K is studied in Appendix D.1.
Revised: The performance with different choices of K is studied in Appendix D.1.

Original: Usingselective eigen-decomposition via the Lanczos Algorithm [35], the time complexity of augmentationscheme optimization is recuded to O ( TKn 2 ) 2 .
Revised: Using selective eigen-decomposition via the Lanczos Algorithm [36], the time complexity of augmentation scheme optimization is recuded to O ( TKn 2 ) 2 .

Original: The scalability can be further improved by deployingthe practical treatments in modeling large-scale graphs [39] (e.g., ego-nets sampling, batch training),which is left as our future work.
Revised: The scalability can be further improved by deploying the practical treatments in modeling large-scale graphs [40] (e.g., ego-nets sampling, batch training), which is left as our future work.

Revised: 4.3 Formulation and Framework of GCL-TAGS

Revised: Figure 2 illustrates our GCL framework equipped with the spectrum-guided augmentation.

Original: A max Δ ℒ GS ( Δ ) min Δ ℒ GS ( Δ ) 
Revised: A detailed algorithm can be found in Appendix B. We first pre-compute the probability matrices ∆ 1 and ∆ 2 via Eq. (5) to set up the augmentation scheme. For each iteration of contrastive learning, we sample two augmented graphs for the input graph t 1 ( A ) ∼ T ( A | ∆ 1 ) and t 2 ( A ) ∼ T ( A | ∆ 2 ) .

Original: Augmentation Scheme
Revised: The augmented graphs are then fed into a GNN encoder f θ , which outputs two sets of node representations H (1) , H (2) ∈ R n × d ′ . A readout pooling function g ϕ is further applied to aggregate and transform the node representations and obtain graph representations z (1) , z (2) ∈ R d ′ . Finally, the GNN encoder and

Revised: A max Δ ℒ GS ( Δ ) min Δ ℒ GS ( Δ )

Original: Input Graph
Revised: Input Graph

Original: T ( A | Δ 1 ) : 0 . 2 0 .

Original: Graph View t 1 ( A ) ∼ T
Revised: Graph View t 1 ( A ) ∼ T

Original: ( A | Δ 1 )
Revised: ( A | Δ 1 )

Original: GNN Encoder f θ
Revised: GNN Encoder f θ

Original: Graph-level Representation z (1)

Original: Shared Shared
Revised: Shared Shared

Original: Node-level Representation
Revised: Node-level Representation

Original: H (1)
Revised: H (1)

Original: Readout Function g ϕ
Revised: Readout Function g ϕ

Revised: Graph-level Representation z (1)

Revised: GNN Encoder f θ

Revised: Node-level Representation

Original:  Graph View t 2 ( A ) ∼ T
Revised: H (2) Readout Function g ϕ Graph-level Representation z (2) ℒ GCL Graph View t 2 ( A ) ∼ T

Original: ( A | Δ 2 )
Revised: ( A | Δ 2 )

Original: Augmentation Scheme
Revised: Augmentation Scheme

Original: T ( A | Δ 2 ) : 
Revised: T ( A | Δ 1 ) : 0 .0 .

Original: GNN Encoder f
Revised: Augmentation Scheme

Original: [Figure]
Revised: T ( A | Δ 2 ) :

Original: The opposite-direction augmentation scheme guided by graph spectrum is pre-computed following
Revised: The opposite-direction augmentation scheme guided by graph spectrum is pre-computed following

Original: Eq.
Revised: Eq.

Original: The contrastive objective is to maximize the mutual information between node representations from one view and the graph representation from another view, and vice versa.
Revised: The contrastive objective is to maximize the mutual information between node representations from one view and the graph representation from another view, and vice versa.

Original: 4.3 Formulation and Framework of GCL-TAGS

Original: Figure 2 illustrates our GCL framework equipped with the spectrum-guided augmentation.

Original: A detailedalgorithm can be found in Appendix B. We first pre-compute the probability matrices ∆ 1 and∆ 2 via Eq.

Original: (5) to set up the augmentation scheme.

Original: For each iteration of contrastive learning, wesample two augmented graphs for the input graph t 1 ( A ) ∼ T ( A | ∆ 1 ) and t 2 ( A ) ∼ T ( A | ∆ 2 ) .

Original: Theaugmented graphs are then fed into a GNN encoder f θ , which outputs two sets of node representations H (1) , H (2) ∈ R n × d ′ . A readout pooling function g ϕ is further applied to aggregate and transform thenode representations and obtain graph representations z (1) , z (2) ∈ R d ′ . Finally, theGNN encoder andthe readout function are trained by a contrastive objective L GCL that maximizes the correspondencebetween local node representations of one view and the global graph representation of the other view.
Revised: the readout function are trained by a contrastive objective L GCL that maximizes the correspondence between local node representations of one view and the global graph representation of the other view.

Original: Given training graphs G , we formulate GCL-TAGS as the following optimization problem:
Revised: Given training graphs G , we formulate GCL-TAGS as the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: where I ( X 1 , X 2 ) calculates the mutual information between variables X 1 and X 2 , and we adopt
Revised: where I ( X 1 , X 2 ) calculates the mutual information between variables X 1 and X 2 , and we adopt

Original: InfoNCE as its estimator which is proven to be a lower bound of mutual information [51, 38].
Revised: InfoNCE as its estimator which is proven to be a lower bound of mutual information [53, 39].

Original: Specifically, denoting cosine similarity as sim ( · , · ) , we estimate the mutual information as follows:
Revised: Specifically, denoting cosine similarity as sim ( · , · ) , we estimate the mutual information as follows:

Original: [Equation]
Revised: [Equation]

Original: ] + where a and b index the augmented views, and (cid:101)
Revised: where a and b index the augmented views, and (cid:101)

Original: H is the noderepresentations for a corrupted graph by randomly shuffling the features of the input graph [54, 17]to serve as negative examples.
Revised: H is the node representations for a corrupted graph by randomly shuffling the features of the input graph [56, 17] to serve as negative examples.

Original: Note that the augmentation scheme is optimized prior to contrastivelearning, which is a one-time computation thus does not introduce any extra complexity to thecontrastive learning process.
Revised: Note that the augmentation scheme is optimized prior to contrastive learning, which is a one-time computation thus does not introduce any extra complexity to the contrastive learning process.

Original:  Experiments
Revised:  Experiments

Original: Experiment Setup: An extensive set of evaluations is performed for node classification, graphclassification and regression tasks under unsupervised learning, transfer learning and adversarialattack settings.
Revised: Experiment Setup: An extensive set of evaluations is performed for node classification, graph classification and regression tasks under unsupervised learning, transfer learning and adversarial attack settings.

Original: Our evaluations include various graph datasets ranging from citation networks, socialnetworks to chemical molecules.
Revised: Our evaluations include various graph datasets ranging from citation networks, social networks to chemical molecules.

Original: We use GCN (for node prediction tasks) and GIN (for graphprediction tasks) as the base encoder for all methods to demonstrate the performance gain fromcontrastive learning.
Revised: We use GCN (for node prediction tasks) and GIN (for graph prediction tasks) as the base encoder for all methods to demonstrate the performance gain from contrastive learning.

Original: We adopt the following linear evaluation protocol for downstream tasks [47]: [Table] based on the representations given by the encoder, we train and evaluate a Logistic classifier or a
Revised: We adopt the following linear evaluation protocol for downstream tasks [48]: based on the representations given by the encoder, we train and evaluate a Logistic classifier or a

Original: Ridge regressor.
Revised: Ridge regressor.

Original: We repeat our experiments for 10 times and report the mean and standard derivationof the evaluation metrics.
Revised: We repeat our experiments for 10 times and report the mean and standard derivation of the evaluation metrics.

Original: We summarize the experimental details about datasets, baselines andconfigurations in Appendix C.
Revised: We summarize the experimental details about datasets, baselines and configurations in Appendix C.

Original: 5.1 Unsupervised Learning
Revised: 5.1 Unsupervised Learning

Original: To evaluate the quality of learned representations, a linear model is trained for the downstream tasksusing these learned representations as features and the resulting prediction performance is reported.
Revised: To evaluate the quality of learned representations, a linear model is trained for the downstream tasks using these learned representations as features and the resulting prediction performance is reported.

Original: The effectiveness of GCL-TAGS is evaluated on both node- and graph-level prediction tasks.
Revised: The effectiveness of GCL-TAGS is evaluated on both node- and graph-level prediction tasks.

Original: Node Classification Task.
Revised: Node Classification Task.

Original: The datasets include Cora, Citeseer, PubMed citation networks [44], WikiCS hyperlink network [31], Amazon-Computer/Photo co-purchase network [45], and Coauthor-CSnetwork [45].
Revised: The datasets include Cora, Citeseer, PubMed citation networks [45], WikiCS hyperlink network [32], Amazon-Computer/Photo co-purchase network [46], and Coauthor-CS network [46].

Original: We compare GCL-TAGS against GCL methods that augment topology with uniformlyrandom edge perturbation (e.g., GRACE [65], BGRL [48], GBT [1]), centrality (GCA [66]), diffusionmatrix (MVGRL [17]) and the original graph (e.g., GMI [36] and DGI [54]).
Revised: We compare GCL-TAGS against GCL methods that augment topology with uniformly random edge perturbation (e.g., GRACE [67], BGRL [49], GBT [1]), centrality (GCA [68]), diffusion [Table] [Table] matrix (MVGRL [17]) and the original graph (e.g., GMI [37] and DGI [56]).

Original: We also consider a fullysemi-supervised GCN ( S-GCN ), a randomly initialized untrained GCN ( R-GCN ) and using the rawnode features as node representations ( Raw-X ).
Revised: We also consider a fully semi-supervised GCN ( S-GCN ), a randomly initialized untrained GCN ( R-GCN ) and using the raw node features as node representations ( Raw-X ).

Original: All the methods exploit a 2 -layer GCN encoder and adownstream linear classifier with the same hyper-parameters for a fair comparison.
Revised: All the methods exploit a 2 -layer GCN encoder and a downstream linear classifier with the same hyper-parameters for a fair comparison.

Original: We adopt randomfeature masking in GCL-TAGS, following the setup in SOTA works [66, 1].
Revised: We adopt random feature masking in GCL-TAGS, following the setup in SOTA works [68, 1].

Original: Table 1 shows that on the node classification task, GCL-TAGS achieves state-of-the-art performancein 6 out of the 7 datasets, 5 of which are significantly better than others.
Revised: Table 1 shows that on the node classification task, GCL-TAGS achieves state-of-the-art performance in 6 out of the 7 datasets, 5 of which are significantly better than others.

Original: Specifically, comparing
Revised: Specifically, comparing

Original: GCL-TAGS with MVGRL and GCA which use domain knowledge of the graph (e.g., node centralityor graph diffusion), the performance gain suggests the advantage of the spectrum based augmentationover previous domain-knowledge based heuristics.
Revised: GCL-TAGS with MVGRL and GCA which use domain knowledge of the graph (e.g., node centrality or graph diffusion), the performance gain suggests the advantage of the spectrum based augmentation over previous domain-knowledge based heuristics.

Original: Meanwhile, GCL-TAGS is shown to be moreeffective than GRACE, BGRL and GBT which adopt uniformly random augmentation 3 and use anode-level contrastive objective.
Revised: Meanwhile, GCL-TAGS is shown to be more effective than GRACE, BGRL and GBT which adopt uniformly random augmentation 3 and use a node-level contrastive objective.

Original: It is noteworthy that the representations learned by GCL methodsachieve better performance than the base encoder R-GCN with semi-supervision, which suggests theeffectiveness of self-supervised learning when label information is limited.
Revised: It is noteworthy that the representations learned by GCL methods achieve better performance than the base encoder R-GCN with semi-supervision, which suggests the effectiveness of self-supervised learning when label information is limited.

Original: Graph Prediction Task.
Revised: Graph Prediction Task.

Original: We test on TU biochemical and social networks [33], Open Graph Benchmark (OGB) [19] and ZINC [20, 11] chemical molecules, and Protein-Protein Interaction (PPI)biological networks [20, 67] for graph prediction.
Revised: We test on TU biochemical and social networks [34], Open Graph Benchmark (OGB) [19] and ZINC [20, 11] chemical molecules, and Protein-Protein Interaction (PPI) biological networks [20, 69] for graph prediction.

Original: We compare GCL-TAGS with five GCL methodsincluding InfoGraph [46], GraphCL [60], MVGRL [17], AD-GCL (with fixed regularization weight)[47] and JOAO (v2) [59].
Revised: We compare GCL-TAGS with five GCL methods including InfoGraph [47], GraphCL [62], MVGRL [17], AD-GCL (with fixed regularization weight) [48] and JOAO (v2) [61].

Original: We use a 5 -layer GIN encoder for all methods, including a semi-supervised
Revised: We use a 5 -layer GIN encoder for all methods, including a semi-supervised

Original: S-GIN and a randomly initialized R-GIN.
Revised: S-GIN and a randomly initialized R-GIN.

Original: A readout function with a graph pooling layer and a 2 -layer
Revised: A readout function with a graph pooling layer and a 2 -layer

Original: MLP is applied to generate graph representations.
Revised: MLP is applied to generate graph representations.

Original: Table 2 summarizes the graph prediction performance.
Revised: Table 2 summarizes the graph prediction performance.

Original: GCL-TAGS gives the best results on 13out of 17 datasets, of which 10 are significantly better than others.
Revised: GCL-TAGS gives the best results onout of 17 datasets, of which 10 are significantly better than others.

Original: Compared with GraphCL and
Revised: Compared with GraphCL and

Original: [Table] JOAO which select the best combination of augmentations for each dataset from a pool of methodsincluding edge perturbation, node dropping and subgraph sampling, GCL-TAGS using only edgeperturbation based augmentation still outperforms them.
Revised: JOAO which select the best combination of augmentations for each dataset from a pool of methods including edge perturbation, node dropping and subgraph sampling, GCL-TAGS using only edge perturbation based augmentation still outperforms them.

Original: This suggests the effectiveness of graphspectrum in guiding topology augmentation.
Revised: This suggests the effectiveness of graph spectrum in guiding topology augmentation.

Original: Compared with MVGRL, our performance gain mainlycomes from the augmentation scheme, as these two methods share similar contrastive objectives,and our augmentation guided by graph spectrum is clearly more effective than the widely adopteduniformly random augmentation.
Revised: Compared with MVGRL, our performance gain mainly comes from the augmentation scheme, as these two methods share similar contrastive objectives, and our augmentation guided by graph spectrum is clearly more effective than the widely adopted uniformly random augmentation.

Original: While AD-GCL and GCL-TAGS follow a similar principle toremove edges that carry non-important and redundant information, GCL-TAGS is more flexible sincethe augmentation scheme is optimized in an independent pre-computation step without interferingwith the contrastive learning procedure.
Revised: While AD-GCL and GCL-TAGS follow a similar principle to remove edges that carry non-important and redundant information, GCL-TAGS is more flexible since the augmentation scheme is optimized in an independent pre-computation step without interfering with the contrastive learning procedure.

Original: 5.2 Transfer Learning
Revised: 5.2 Transfer Learning

Original: This experiment evaluates the generalizability of the GNN models, which are pre-trained on somedatasets and re-purposed on different but related datasets.
Revised: This experiment evaluates the generalizability of the GNN models, which are pre-trained on some datasets and re-purposed on different but related datasets.

Original: Table 3 reports the performance onchemical and biological graph classification datasets from [20].
Revised: Table 3 reports the performance on chemical and biological graph classification datasets from [20].

Original: Appendix C.5 further studies an evenmore challenging setting [39] where the encoder is pre-trained on social networks and transferredto multiple out-of-domain tasks.
Revised: Appendix C.5 further studies an even more challenging setting [40] where the encoder is pre-trained on social networks and transferred to multiple out-of-domain tasks.

Original: A reference model without pre-training ( No-Pre-Train-GIN ) is[Table] [Table] compared to demonstrate the gain of pre-training.
Revised: A reference model without pre-training ( No-Pre-Train-GIN ) is compared to demonstrate the gain of pre-training.

Original: GCL-TAGS is shown to be more effective inlearning generalizable encoders.
Revised: GCL-TAGS is shown to be more effective in learning generalizable encoders.

Original: This supports our augmentation principle: by perturbing edges thatcause large spectral changes, the encoder is pre-trained to ignore unreliable structural information,such that the relationship between such information and downstream labels can be removed tomitigate the overfitting issue.
Revised: This supports our augmentation principle: by perturbing edges that cause large spectral changes, the encoder is pre-trained to ignore unreliable structural information, such that the relationship between such information and downstream labels can be removed to mitigate the overfitting issue.

Original: The generalizability of the GNN encoder on molecule classificationdepends on the structural fingerprints such as subgraphs [7].
Revised: The generalizability of the GNN encoder on molecule classification depends on the structural fingerprints such as subgraphs [7].

Original: JOAO and GraphCL using subgraphsampling augmentation is outperformed by GCL-TAGS, which suggests that the graph spectrumcould be another important fingerprint to study chemical and biological molecular properties.
Revised: JOAO and GraphCL using subgraph sampling augmentation is outperformed by GCL-TAGS, which suggests that the graph spectrum could be another important fingerprint to study chemical and biological molecular properties.

Original: 5.3 Adversarial Attack Setting
Revised: 5.3 Adversarial Attack Setting

Original: This setting demonstrates the robustness property of the proposed augmentation when the inputgraph is adversarially poisoned.
Revised: This setting demonstrates the robustness property of the proposed augmentation when the input graph is adversarially poisoned.

Original: The representations are learned from graphs poisoned by differentstructural attack strategies, including Random (which randomly flips edges), DICE (which deletesedges internally and connects nodes externally across classes), GF-Attack (which maximizes a lowrank matrix approximation loss) and Mettack (which maximizes the training loss via meta-gradients).
Revised: The representations are learned from graphs poisoned by different structural attack strategies, including Random (which randomly flips edges), DICE (which deletes edges internally and connects nodes externally across classes), GF-Attack (which maximizes a lowrank matrix approximation loss) and Mettack (which maximizes the training loss via meta-gradients).

Original: We test the perturbation ratios σ ∈ { 0 . 05 , 0 . 2 } : σ × m edges are flipped for a graph with m edges.
Revised: We test the perturbation ratios σ ∈ { 0 . 05 , 0 . 2 } : σ × m edges are flipped for a graph with m edges.

Original: Table 4 reports the node classification performance under adversarial attack.
Revised: Table 4 reports the node classification performance under adversarial attack.

Original: The encoders learnedby GCL methods with graph augmentations are generally more robust to perturbed graph structurecompared with S-GCN.
Revised: The encoders learned by GCL methods with graph augmentations are generally more robust to perturbed graph structure compared with S-GCN.

Original: GCL-TAGS outperforms baselines with a clear margin, even under the strong 
Revised: GCL-TAGS outperforms baselines with a clear margin, even under the strong [Table] [Table]

Original: Mettack which solicits the downstream label information.
Revised: Mettack which solicits the downstream label information.

Original: This demonstrates a good property of ourproposed augmentation scheme: even though it is not explicitly designed for adversarial robustness,it achieves an effect that the encoder can stay invariant to the adversarially perturbed graph if itsspectrum falls into the range captured by the opposite-direction augmentation scheme.
Revised: This demonstrates a good property of our proposed augmentation scheme: even though it is not explicitly designed for adversarial robustness, it achieves an effect that the encoder can stay invariant to the adversarially perturbed graph if its spectrum falls into the range captured by the opposite-direction augmentation scheme.

Original: Comparedwith the parallel efforts in designing robust GNNs [57, 8, 64, 24], the proposed augmentation providesa new insight using graph spectrum as a tool to study graph invariants and perturbations.
Revised: Compared with the parallel efforts in designing robust GNNs [59, 8, 66, 24], the proposed augmentation provides a new insight using graph spectrum as a tool to study graph invariants and perturbations.

Revised: 5.4 Behavior of Spectrum Guided Augmentation

Revised: To intuitively show the spatial change caused by perturbing the graph spectrum, we visualize a case study on a random geometric graph in Figure 3 where3a draws the original graph, 3b and 3c show the perturbation probability obtained by maximizing and minimizing L GS ( ∆ ) , respectively.

Revised: We can observe that maximizing the spectral norm assigns larger probability to remove edges bridging clusters such that the clustering effect becomes more obvious, while minimizing the spectral norm tends to add edges connecting clusters such that the clustering effect is blurred.

Revised: Intuitively, such an augmentation perturbs these spurious edges that can easily affects the structural property (e.g. clustering) to preserve structural invariant, and the information about these edges are disentangled and minimized in the learned representations by contrastive learning.

Revised: Such an interplay of spectral change and spatial change is also justified theoretically in Appendix D.3.

Original:   Conclusion
Revised: [Figure]  Conclusion

Original: In this work, we proposed a principled topology augmentation scheme guided by graph spectrum.
Revised: In this work, we proposed a principled topology augmentation scheme guided by graph spectrum.

Original: Ourprinciple is that a well-behaving GNN encoder should stay invariant to sensitive structures that causelarge changes on graph spectrum.
Revised: Our principle is that a well-behaving GNN encoder should stay invariant to sensitive structures that cause large changes on graph spectrum.

Original: To achieve this goal, we searched for the augmentation schemethat would mostly change the graph spectrum of the input graph, which leads to an opposite-directionaugmentation scheme that changes the graph spectrum towards opposite directions.
Revised: To achieve this goal, we searched for the augmentation scheme that would mostly change the graph spectrum of the input graph, which leads to an opposite-direction augmentation scheme that changes the graph spectrum towards opposite directions.

Original: Based on theaugmentation scheme, we developed a practical GCL framework GCL-TAGS, which has shownremarkable advantage in a variety of settings.
Revised: Based on the augmentation scheme, we developed a practical GCL framework GCL-TAGS, which has shown remarkable advantage in a variety of settings.

Original: Currently we only focus on topology augmentation,and ignore its interplay with node features, which is another important dimension in GCL.
Revised: Currently we only focus on topology augmentation, and ignore its interplay with node features, which could be another important dimension in GCL.

Original: Recentstudies show the relationship between graph homophily/heterophily and GNNs, which suggests afuture effort to explore the alignment between graph topology and node features in GCL.

Original: A A Review of Graph Spectrum
Revised: A A Review of Graph Spectrum

Original: Graph spectrum plays a significant role in analyzing graph property (e.g., connectivity, clusterstructure, diameter etc.) and is the foundation of spectral filters in GNNs.
Revised: Graph spectrum plays a significant role in analyzing graph property (e.g., connectivity, cluster structure, diameter etc.) and is the foundation of spectral filters in GNNs.

Original: This motivates us to guideour proposed topology augmentation method using graph spectrum.
Revised: This motivates us to guide our proposed topology augmentation method using graph spectrum.

Original: Graph Spectrum and Graph Property.
Revised: Graph Spectrum and Graph Property.

Original: The graph spectrum summarizes important propertiesrelated to a graph’s global structure, which has been studied in graph spectral theory.
Revised: The graph spectrum summarizes important properties related to a graph’s global structure, which has been studied in graph spectral theory.

Original: We list somewidely discussed properties revealed by graph spectrum to support our design: graph spectrum canbe used as a comprehensive proxy for capturing graph properties in GCL.
Revised: We list some widely discussed properties revealed by graph spectrum to support our design: graph spectrum can be used as a comprehensive proxy for capturing graph properties in GCL.

Original: • Algebraic connectivity [5], also known as Fiedler eigenvalue, of a graph is the second-smallesteigenvalue (counting multiple eigenvalues separately) of the Laplacian matrix.
Revised: • Algebraic connectivity [5], also known as Fiedler eigenvalue, of a graph is the second-smallest eigenvalue (counting multiple eigenvalues separately) of the Laplacian matrix.

Original: This eigenvalue isgreater than 0 if and only if the graph is a connected graph.
Revised: This eigenvalue is greater than 0 if and only if the graph is a connected graph.

Original: A corollary is that the number of times0 appears as an eigenvalue in the Laplacian is the number of connected components in the graph.
Revised: A corollary is that the number of times 0 appears as an eigenvalue in the Laplacian is the number of connected components in the graph.

Original: The magnitude of this value reflects how well connected the overall graph is.
Revised: The magnitude of this value reflects how well connected the overall graph is.

Original: • Diameter of a graph can be upper and lower bounded from its spectrum [5].
Revised: • Diameter of a graph can be upper and lower bounded from its spectrum [5].

Original: If the graph has rdistinct eigenvalues, its diameter d is at most r − 1 .
Revised: If the graph has r distinct eigenvalues, its diameter d is at most r − 1 .

Original: Meanwhile, if the graph has m edges andn nodes, we can bound the diameter by the first and second smallest non-zero eigenvalues as1 / 2 mλ 1 ≥ d ≥ 4 /nλ 2 .
Revised: Meanwhile, if the graph has m edges and n nodes, we can bound the diameter by the first and second smallest non-zero eigenvalues as 1 / 2 mλ 1 ≥ d ≥ 4 /nλ 2 .

Original: For all k ≥ 2 , we also have d ≤ k log n/λ k .•
Revised: For all k ≥ 2 , we also have d ≤ k log n/λ k .

Original:  Clusterability of a graph reveals how easy it is to partition the graph, which can be captured bythe differences between the smallest successive eigenvalues of connected graphs.
Revised: • Clusterability of a graph reveals how easy it is to partition the graph, which can be captured by the differences between the smallest successive eigenvalues of connected graphs.

Original: The differencebetween the first two eigenvalues, often referred to as the spectral gap, upper and lower bounds thegraph expansion and conductance by the Cheeger inequality [25].
Revised: The difference between the first two eigenvalues, often referred to as the spectral gap, upper and lower bounds the graph expansion and conductance by the Cheeger inequality [25].

Original: Nevertheless, analogous results also hold for higher-order eigenvalues [27].• Diffusion distance [15] between two nodes v i and v j can be defined as D ( v i , v j ) =
Revised: Nevertheless, analogous results also hold for higher-order eigenvalues [27]. • Diffusion distance [15] between two nodes v i and v j can be defined as D ( v i , v j ) =

Original: ∥ [ ϕ ( L )] i, : −[ ϕ ( L )] j, : ∥ 

Revised: ∥ [ ϕ ( L )] i, : − [ ϕ ( L )] j, : ∥

Original: = (cid:80) nl =1 ϕ ( λ l ) 2 ( u l [ i ] − u l [ j ]) 2 , where ϕ ( L ) =
Revised: = (cid:80) nl =1 ϕ ( λ l ) 2 ( u l [ i ] − u l [ j ]) 2 , where ϕ ( L ) =

Original: U ϕ ( Λ ) U ⊤ and ϕ ( x ) is a decreasingkernel function such as ϕ ( x ) =
Revised: U ϕ ( Λ ) U ⊤ and ϕ ( x ) is a decreasing kernel function such as ϕ ( x ) =

Original: e − tx .
Revised: e − tx .

Original: Therefore, a map that separates nodes with a specificdiffusion distance is obtained when embedding graph nodes using eigenvectors.
Revised: Therefore, a map that separates nodes with a specific diffusion distance is obtained when embedding graph nodes using eigenvectors.

Original: Graph Spectrum and Spectral Filter.
Revised: Graph Spectrum and Spectral Filter.

Original: By viewing graph embedding models from a signal processingperspective, the normalized Laplacian L norm serves as a graph shift operator and defines the frequencydomain of a graph.
Revised: By viewing graph embedding models from a signal processing perspective, the normalized Laplacian L norm serves as a graph shift operator and defines the frequency domain of a graph.

Original: As a result, its eigenvectors U can be considered as the graph Fourier bases,and the eigenvalues Λ (a.k.a., graph spectrum) correspond to the frequency components.
Revised: As a result, its eigenvectors U can be considered as the graph Fourier bases, and the eigenvalues Λ (a.k.a., graph spectrum) correspond to the frequency components.

Original: Take onecolumn of node features X as an example of graph signal, which can be compactly represented asx ∈ R n .
Revised: Take one column of node features X as an example of graph signal, which can be compactly represented as x ∈ R n .

Original: The graph Fourier transform of graph signal x is given by ˆ x = U ⊤ x and the inverse graph
Revised: The graph Fourier transform of graph signal x is given by ˆ x = U ⊤ x and the inverse graph

Original: Fourier transform is then x
Revised: Fourier transform is then x

Original: = U ˆ x .
Revised: = U ˆ x .

Original: The graph signals in the Fourier domain are filtered by amplifyingor attenuating the frequency components Λ .
Revised: The graph signals in the Fourier domain are filtered by amplifying or attenuating the frequency components Λ .

Original: At the essence of GNNs is the spectral convolution , which can be defined as the multiplication of asignal vector x with a spectral filter g ϕ in the Fourier domain [6]:
Revised: At the essence of GNNs is the spectral convolution , which can be defined as the multiplication of a signal vector x with a spectral filter g ϕ in the Fourier domain [6]:

Original: [Equation]
Revised: [Equation]

Original: The filter g ϕ defines a smooth transformation function of the graph spectrum.
Revised: The filter g ϕ defines a smooth transformation function of the graph spectrum.

Original: One can apply a spectralfilter and graph Fourier transformation to manipulate graph signals in various way, such as smoothingand denoising [43], abnormally detection [32] and clustering [55].
Revised: One can apply a spectral filter and graph Fourier transformation to manipulate graph signals in various way, such as smoothing and denoising [44], abnormally detection [33] and clustering [57].

Original: Here we show how the spectralconvlution is defined in two popular GNNS used in our experiments: GCN [26] and GIN [58].
Revised: Here we show how the spectral convlution is defined in two popular GNNS used in our experiments: GCN [26] and GIN [60].

Original: The vanilla GCN [26] is a first-order approximation of Chebyshev polynomial filter [16] withg ϕ ( Λ ) =
Revised: The vanilla GCN [26] is a first-order approximation of Chebyshev polynomial filter [16] with g ϕ ( Λ ) =

Original: (2 − Λ ) ϕ , and the corresponding convolution for d -dimensional signal X is:
Revised: (2 − Λ ) ϕ , and the corresponding convolution for d -dimensional signal X is:

Original: [Equation]
Revised: [Equation]

Original: ϕ [Equation] 10) where Φ ∈ R d × d ′ is the matrix of spectral filter parameters, and a renormalization trick I n +
Revised: where Φ ∈ R d × d ′ is the matrix of spectral filter parameters, and a renormalization trick I n +

Original: D − 1 / 2 AD − 1 / 2 → ˜ D − 1 / 2 ˜ A ˜ D − 1 / 2 is applied by adding self-loop ˜
Revised: D − 1 / 2 AD − 1 / 2 → ˜ D − 1 / 2 ˜ A ˜ D − 1 / 2 is applied by adding self-loop ˜

Original: A =
Revised: A =

Original: A + I n .
Revised: A + I n .

Original: GIN [58] withequal discriminative power as WL test designs spectral convolution as: 
Revised: GIN [60] with equal discriminative power as WL test designs spectral convolution as: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where ϵ can be a learnable parameter or a fixed scalar.
Revised: where ϵ can be a learnable parameter or a fixed scalar.

Original: Since the spectral filters g ϕ ( Λ ) are the key ingraph convolutions to encode graph signals that are transformed in the Fourier domain.
Revised: Since the spectral filters g ϕ ( Λ ) are the key in graph convolutions to encode graph signals that are transformed in the Fourier domain.

Original: The output ofthe spectral filters is then transformed back to the spatial domain to generate node representations.
Revised: The output of the spectral filters is then transformed back to the spatial domain to generate node representations.

Original: Therefore, we aim to augment graphs to influence the graph spectrum and the filtered graph signals,such that the encoder with altered spectral filters is encouraged to stay invariant to such perturbationsthrough GCL.
Revised: Therefore, we aim to augment graphs to influence the graph spectrum and the filtered graph signals, such that the encoder with altered spectral filters is encouraged to stay invariant to such perturbations through GCL.

Original: B Algorithm of GCL-TAGS
Revised: B Algorithm of GCL-TAGS

Original: Algorithm 1 illustrates the detailed steps of GCL-TAGS.
Revised: Algorithm 1 illustrates the detailed steps of GCL-TAGS.

Original: Note that only one training graph is included(e.g., L = 1 ) for node representations learning, and multiple graphs are used (e.g., L ≥ 2 ) for graphrepresentation learning.
Revised: Note that only one training graph is included (e.g., L = 1 ) for node representations learning, and multiple graphs are used (e.g., L ≥ 2 ) for graph representation learning.

Original: Algorithm 
Revised: Algorithm 1: GCL-TAGS

Original: Input : Data [Equation] L } ; GNN encoder f θ ; Readout function g ϕ .
Revised: Input : Data { G l = ( X l , A l ) ∼ G| l = 1 · · · , L } ; GNN encoder f θ ; Readout function g ϕ .

Original: Params : Augmentation optimization step M and learning rate η ;
Revised: Params : Augmentation optimization step M and learning rate η ;

Original: Contrastive learning step N and learning rate β .
Revised: Contrastive learning step N and learning rate β .

Original: Output : Trained encoder  and 
Revised: Output : Trained encoder f θ ∗ and readout function f ϕ ∗ .

Original: 1 /* Lower in Eq.
Revised: 1 /* Lower-level optimization for augmentation scheme in Eq.

Original: (7) */  do
Revised: (7) */ 2 for each graph l ← 1 to L do

Original: Initialize Bernoulli 
Revised: Initialize Bernoulli parameters for each graph: ∆ (0) l, 1 ∈ [0 , 1] n × n , ∆ (0) l, 2 ∈ [0 , 1] n × n ; for t ← 1 to M do /*

Original: M do /* Optimize one direction of scheme: [Equation]
Revised: Optimize one direction of scheme: max ∆ ∈S L GS ( ∆ ) */

Original: [Equation]
Revised: L GS ( ∆ ( t − 1) l,) = ∥ eig ( Lap ( A + C ◦ ∆ ( t − 1) l,))

Original:  /* Optimize the other direction of scheme min ∆ ∈S L GS ( ∆ ) */∥
Revised: ∥ 22 ; ∆ ( t ) l, 1 ← P S [ ∆ ( t − 1) l,+ η ∇L GS ( ∆ ( t − 1) l,)] ; /* Optimize the other direction of scheme: min ∆ ∈S L GS ( ∆ ) */

Original: A C ◦ 1) l, l
Revised: L GS ( ∆ ( t − 1) l,) =

Original:  A  ◦ l l A
Revised: ∥ eig ( Lap ( A + C ◦ ∆ ( t − 1) l,)) ∥ 22 ; ∆ ( t ) l, 2 ← P

Original: X l
Revised: L L

Original: X l applicable; Two graph views are generated as G l, 1 = ( X l, 1 , A l, 1 ) , G l, 2 = ( X l, 2 , A l, 2 ) end / objective min θ,ϕ L GCL */ Define L ( G l, 1 , G l, 2 , θ, ϕ ) = 1 n (cid:80) ni =1 (cid:16) I ( H (1) i , z (2) )
Revised: − t −

Original: + I ( H (2) i , z (1) ) (cid:17) for G l ; Calculate
Revised: Ql

Original:  L GCL ( θ ( t − 1) , ϕ
Revised: 1 L G l G t − t

Original: C Experiment Setup Details
Revised: C Experiment Setup Details

Original: This section includes the detailed setup for all experiments, including the procedure of conductingpre-analysis, datasets, baselines, and hyper-parameter settings.
Revised: This section includes the detailed setup for all experiments, including the procedure of conducting pre-analysis, datasets, baselines, and hyper-parameter settings.

Original: The experiments were performed on
Revised: The experiments were performed on

Original: Nvidia GeForce RTX 2080Ti (12GB)
Revised: Nvidia GeForce RTX 2080Ti (12GB)

Original: GPU cards for most datasets, and RTX A6000 (48GB) cards for
Revised: GPU cards for most datasets, and RTX A6000 (48GB) cards for

Original: PubMed and Coauthor-CS datasets.
Revised: PubMed and Coauthor-CS datasets.

Original: Optimizing memory use for large graphs will be our future work.
Revised: Optimizing memory use for large graphs will be our future work.

Original: C.1 Pre-analysis Experiment of Figure1
Revised: C.1 Pre-analysis Experiment of Figure

Original: We now introduce the detailed information to reproduce Figure 1 on Cora.
Revised: We now introduce the detailed information to reproduce Figure 1 on Cora.

Original: This experiment is to showthat uniformly random edge perturbation adopted in many GCL methods is not effective enoughto reveal essential graph properties, described by graph spectrum.
Revised: This experiment is to show that uniformly random edge perturbation adopted in many GCL methods is not effective enough to reveal essential graph properties, described by graph spectrum.

Original: Since graph spectrum is closelyrelated to graph properties such as clusterability (as discussed in Appendix A), in contrast to theuniform edge perturbation, we created a node cluster based strategy as follows: We first groupedthe edges among nodes by whether the end nodes belong to the same cluster, treating nodes’ classlabels as their clusters.
Revised: Since graph spectrum is closely related to graph properties such as clusterability (as discussed in Appendix A), in contrast to the uniform edge perturbation, we created a node cluster based strategy as follows: We first grouped the edges among nodes by whether the end nodes belong to the same cluster, treating nodes’ class labels as their clusters.

Original: For inter-cluster edges, we assign a larger removing probability, while forintra-cluster edges we assign a smaller removing probability.
Revised: For inter-cluster edges, we assign a larger removing probability, while for intra-cluster edges we assign a smaller removing probability.

Original: Note that in expectation, we removethe same amount of edges as the uniformly random strategy, but allocate different probabilities tothese two groups of edges.
Revised: Note that in expectation, we remove the same amount of edges as the uniformly random strategy, but allocate different probabilities to these two groups of edges.

Original: We should note the purpose of this experiment is only to demonstrate theimpact of graph spectrum for GCL, and we used the class label of nodes as a proxy about the graphspectrum (due to the relation between clusterability and graph spectrum).
Revised: We should note the purpose of this experiment is only to demonstrate the impact of graph spectrum for GCL, and we used the class label of nodes as a proxy about the graph spectrum (due to the relation between clusterability and graph spectrum).

Original: Our proposed solution
Revised: Our proposed solution

Original: GCL-TAGS is fully unsupervised, i.e., it does not depend on the node labels at all.
Revised: GCL-TAGS is fully unsupervised, i.e., it does not depend on the node labels at all.

Original: Specifically, an edge removing ratio σ indicated by the x-axis of Figure 1 represents the augmentationstrength: for an input graph with m edges, we remove σ · m edges to generate an augmented graph.
Revised: Specifically, an edge removing ratio σ indicated by the x-axis of Figure 1 represents the augmentation strength: for an input graph with m edges, we remove σ · m edges to generate an augmented graph.

Original: For the uniformly random augmentation method ( Uniform ), each edge is assigned an equal removingprobability as σ ; for the cluster-based augmentation heuristic ( Clustered ), given m inter inter-clusteredges and m intra = m − m inter intra-cluster edges, we increase the removing probability of each intercluster edge as σ inter = min { 1 . 2 σ, σ · m/m inter } , and the removing probability of each intra-clusteredge is decreased to σ intra = ( σ · m − σ inter · m inter ) /m intra to make sure that in expectation σ · medges are removed as in the uniform strategy.
Revised: For the uniformly random augmentation method ( Uniform ), each edge is assigned an equal removing probability as σ ; for the cluster-based augmentation heuristic ( Clustered ), given m inter inter-cluster edges and m intra = m − m inter intra-cluster edges, we increase the removing probability of each intercluster edge as σ inter = min { 1 . 2 σ, σ · m/m inter } , and the removing probability of each intra-cluster edge is decreased to σ intra = ( σ · m − σ inter · m inter ) /m intra to make sure that in expectation σ · m edges are removed as in the uniform strategy.

Original: When conducting the contrastive learning procedure following GRACE [65], one augmentationbranch used the uniformly random edge removing strategy, and the other branch adopted either theuniform or the clustered strategy.
Revised: When conducting the contrastive learning procedure following GRACE [67], one augmentation branch used the uniformly random edge removing strategy, and the other branch adopted either the uniform or the clustered strategy.

Original: Both branches included a random feature masking with ratio 0 . 3 .
Revised: Both branches included a random feature masking with ratio 0 . 3 .

Original: For these two GCL methods based on different augmentation strategies, the experiment setup isas follows: both methods used a GCN encoder with the same architecture and hyper-parameters(e.g., 2 convolutional layers with the embedding dimension of 32 ).
Revised: For these two GCL methods based on different augmentation strategies, the experiment setup is as follows: both methods used a GCN encoder with the same architecture and hyper-parameters (e.g., 2 convolutional layers with the embedding dimension of 32 ).

Original: Both performed 1000 trainingiterations to obtain node representations, whose quality was evaluated by using them as features for adownstream node classification task.
Revised: Both performed 1000 training iterations to obtain node representations, whose quality was evaluated by using them as features for a downstream node classification task.

Original: We compared the clustered augmentation based GCL with the uniform augmentation based GCL fromtwo aspects: their performance in the downstream task and the spectral change on the augmentedgraphs.
Revised: We compared the clustered augmentation based GCL with the uniform augmentation based GCL from two aspects: their performance in the downstream task and the spectral change on the augmented graphs.

Original: The right-side of Figure 1 reports the mean and standard derivation of F1 score for 10 runswith different random seeds, which measures the downstream task performance.
Revised: The right-side of Figure 1 reports the mean and standard derivation of F1 score for 10 runs with different random seeds, which measures the downstream task performance.

Original: Meanwhile, wecalculated the eigenvalues of the normalized Laplacian matrix of the input graph ( Λ ), the augmentedgraphs with uniform strategy ( Λ ′ uniform ) and the augmented graphs with clustered strategy ( Λ ′ clustered ).
Revised: Meanwhile, we calculated the eigenvalues of the normalized Laplacian matrix of the input graph ( Λ ), the augmented graphs with uniform strategy ( Λ ′ uniform ) and the augmented graphs with clustered strategy ( Λ ′ clustered ).

Original: The left-side of Figure 1 reports the L 2 distance of eigenvalues between the input and augmentedgraphs (e.g., ∥ Λ − Λ ′ uniform ∥ 2 and ∥ Λ − Λ ′ clustered ∥ 2 ) to measure the spectral change.
Revised: The left-side of Figure 1 reports the L 2 distance of eigenvalues between the input and augmented graphs (e.g., ∥ Λ − Λ ′ uniform ∥ 2 and ∥ Λ − Λ ′ clustered ∥ 2 ) to measure the spectral change.

Original: C.2 Summary of Datasets
Revised: C.2 Summary of Datasets

Original: The proposed GCL-TAGS is evaluated on 25 graph datasets.
Revised: The proposed GCL-TAGS is evaluated on 25 graph datasets.

Original: Specifically, for the node classificationtask , we included Cora, Citeseer, PubMed citation networks [44], Wiki-CS hyperlink network [31],
Revised: Specifically, for the node classification task , we included Cora, Citeseer, PubMed citation networks [45], Wiki-CS hyperlink network [32],

Original: Amazon-Computer and Amazon-Photo co-purchase network [45], and Coauthor-CS network [45].
Revised: Amazon-Computer and Amazon-Photo co-purchase network [46], and Coauthor-CS network [46].

Original: For the graph classification and regression tasks , we employed TU biochemical and social networks [Table] [Table] [33], Open Graph Benchmark (OGB) [19] and ZINC [20, 11] chemical molecules
Revised: For the graph classification and regression tasks , we employed TU biochemical and social networks [34], Open Graph Benchmark (OGB) [19] and ZINC [20, 11] chemical molecules, and ProteinProtein Interaction (PPI) biological networks [20, 69].

Revised: We summarize the statistics of these datasets and briefly introduce the experiment settings on them.

Original: [Table]
Revised: [Table]

Original: and ProteinProtein Interaction (PPI) biological networks [20, 67]. We summarize the statistics of these datasetsand briefly introduce the experiment settings on them.
Revised: [Table]

Original: • A collection of datasets were used to evaluate node classification performance in both unsupervisedlearning and adversarial attack settings, and Table 5 summarizes the statistics of these datasets.
Revised: • A collection of datasets were used to evaluate node classification performance in both unsupervised learning and adversarial attack settings, and Table 5 summarizes the statistics of these datasets.

Original: Cora, Citeseer, PubMed citation networks [44] contain nodes representing documents and edgesdenoting citation links.
Revised: Cora, Citeseer, PubMed citation networks [45] contain nodes representing documents and edges denoting citation links.

Original: The task is to predict the research topic of a document given its bag-of-wordrepresentation.
Revised: The task is to predict the research topic of a document given its bag-of-word representation.

Original: Wiki-CS hyperlink network [31] consists of nodes corresponding to ComputerScience articles, with edges based on hyperlinks.
Revised: Wiki-CS hyperlink network [32] consists of nodes corresponding to Computer Science articles, with edges based on hyperlinks.

Original: The task is to predict the branch of the fieldabout the article using its 300 -dimension pretrained GloVe word embeddings.
Revised: The task is to predict the branch of the field about the article using its 300 -dimension pretrained GloVe word embeddings.

Original: Amazon-Computer,Amazon-Photo co-purchase networks [45] have nodes being items and edges representing that twoitems are frequently bought together.
Revised: Amazon-Computer, Amazon-Photo co-purchase networks [46] have nodes being items and edges representing that two items are frequently bought together.

Original: Given item reviews as bag-of-word node features, the task isto map items to their respective item category.
Revised: Given item reviews as bag-of-word node features, the task is to map items to their respective item category.

Original: Coauthor-CS network [45] contains node to beauthors and edges to be co-author relationship.
Revised: Coauthor-CS network [46] contains node to be authors and edges to be co-author relationship.

Original: Given keywords of each author’s papers, the taskis to map authors to their respective field of study.
Revised: Given keywords of each author’s papers, the task is to map authors to their respective field of study.

Original: All of these datasets are included in the PyG(PyTorch Geometric) library 4 .•
Revised: All of these datasets are included in the PyG (PyTorch Geometric) library 4 .

Original:  Two sets of datasets were used to evaluate graph prediction tasks under the unsupervised learningsetting.
Revised: • Two sets of datasets were used to evaluate graph prediction tasks under the unsupervised learning setting.

Original: TU Datasets [33] provides a collection of benchmark datasets, and we used severalbiochemical molecules and social networks for graph classification as summarized in Table 6.
Revised: TU Datasets [34] provides a collection of benchmark datasets, and we used several biochemical molecules and social networks for graph classification as summarized in Table 6.

Original: Thedata collection is also included in the PyG library following a 10-fold evaluation data split.
Revised: The data collection is also included in the PyG library following a 10-fold evaluation data split.

Original: We usedthese datasets for evaluation of the graph classification task in unsupervised learning setting.
Revised: We used these datasets for evaluation of the graph classification task in unsupervised learning setting.

Original: OpenGraph Benchmark (OGB) [19] contains datasets for chemical molecular property classificationand regression tasks, which are summarized in Table 7.
Revised: Open Graph Benchmark (OGB) [19] contains datasets for chemical molecular property classification and regression tasks, which are summarized in Table 7.

Original: This data collection can be load via theOGB platform 5 , and we used its processed format available in PyG library.• A set of biological and chemical datasets were used to evaluate graph classification task under thetransfer learning setting, summarized in Table 8.
Revised: This data collection can be load via the OGB platform 5 , and we used its processed format available in PyG library. • A set of biological and chemical datasets were used to evaluate graph classification task under the transfer learning setting, summarized in Table 8.

Original: Following the transfer learning pipeline in [20], 
Revised: Following the transfer learning pipeline in [20], an encoder was first pre-trained on a large biological Protein-Protein Interaction ( PPI ) network or ZINC chemical molecule dataset, and then was evaluated on small datasets from the same domains.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: an encoder was first pre-trained on a large biological Protein-Protein Interaction ( PPI ) network or ZINC chemical molecule dataset, and then was evaluated on small datasets from the same domains.
Revised: [Table]

Original: C.3 Summary of GCL Baselines
Revised: C.3 Summary of GCL Baselines

Original: We compared GCL-TAGS against seven self-supervised learning baselines for node representationlearning , including GRACE [65], its extension GCA [66], BGRL [48], GBT [1], MVGRL [17],
Revised: We compared GCL-TAGS against seven self-supervised learning baselines for node representation learning , including GRACE [67], its extension GCA [68], BGRL [49], GBT [1], MVGRL [17],

Original: GMI [36] and DGI [54].
Revised: GMI [37] and DGI [56].

Original: Meanwhile, five baselines designed for graph representation learningwere compared, including InfoGraph [46], GraphCL [60], MVGRL [17], AD-GCL (with fixedregularization weight) [47] and JOAO (v2) [59].
Revised: Meanwhile, five baselines designed for graph representation learning were compared, including InfoGraph [47], GraphCL [62], MVGRL [17], AD-GCL (with fixed regularization weight) [48] and JOAO (v2) [61].

Original: In the contrastive objective design of these methods,different contrastive modes are adopted to compare node-level or graph-level representations.
Revised: In the contrastive objective design of these methods, different contrastive modes are adopted to compare node-level or graph-level representations.

Original: Wesummarize them based on their contrastive modes as follows:
Revised: We summarize them based on their contrastive modes as follows:

Original: • Node v.s. node mode specifies the contrastive examples as node pairs in a local perspective, whichfocuses on node-level representation learning to serve node prediction tasks.
Revised: • Node v.s. node mode specifies the contrastive examples as node pairs in a local perspective, which focuses on node-level representation learning to serve node prediction tasks.

Original: In particular, GRACE[65] employs uniformly random edge removing to generate two views, and treats the same nodefrom two views as positive pairs, and all the other nodes as negatives.
Revised: In particular, GRACE [67] employs uniformly random edge removing to generate two views, and treats the same node from two views as positive pairs, and all the other nodes as negatives.

Original: GCA [66] extends GRACE[65] with an adaptive augmentation considering the node centrality.
Revised: GCA [68] extends GRACE [67] with an adaptive augmentation considering the node centrality.

Original: BGRL [48] adopts uniformlyrandom edge removing augmentation and applies a bootstrapping [12] framework to avoid collapsewithout negative sampling.
Revised: BGRL [49] adopts uniformly random edge removing augmentation and applies a bootstrapping [12] framework to avoid collapse without negative sampling.

Original: GBT [1] uses uniformly random edge removing as graph augmentation and a Barlow-twins [62] objective to avoid collapse without requiring negative sampling.
Revised: GBT [1] uses uniformly random edge removing as graph augmentation and a Barlow-twins [64] objective to avoid collapse without requiring negative sampling.

Original: GMI[36] maximizes a general form of graphical mutual information defined on both features and edges between nodes in input graph and reconstructed output graph.
Revised: GMI [37] maximizes a general form of graphical mutual information defined on both features and edges between nodes in input graph and reconstructed output graph.

Original: • Graph v.s. node mode takes graph and node pairs as contrastive examples to decide whether they are from the same graph, which obtains both node- and graph-level representations.In
Revised: • Graph v.s. node mode takes graph and node pairs as contrastive examples to decide whether they are from the same graph, which obtains both node- and graph-level representations.

Original:  particular, MVGRL [ ] maximizes the mutual information between the local Laplacian matrix and a global diffusion matrix, which obtains both  node 
Revised: In particular, MVGRL [17] maximizes the mutual information between the local Laplacian matrix and a global diffusion matrix, which obtains both node-level and graph-level representations that can serve for both node and graph prediction tasks.

Original: -level and

Original: graph

Original: -level

Original: representations

Original: that can serve for both node and graph prediction tasks. DGI [54] proposes to maximize the mutual information between representations of local nodes and the entire graph, in contrast with a corrupted 
Revised: DGI [56] proposes to maximize the mutual information between representations of local nodes and the entire graph, in contrast with a corrupted graph by node shuffling.

Original: graph by node shuffling.

Original: InfoGraph [46] aims to maximize the mutual information between therepresentations of entire graphs and substructures (e
Revised: InfoGraph [47] aims to maximize the mutual information between the

Original: g., nodes, edges and triangles) with different granularity, and it is evaluated on graph-level prediction tasks.
Revised: representations of entire graphs and substructures (e.g., nodes, edges and triangles) with different granularity, and it is evaluated on graph-level prediction tasks.

Original: • Graph v.s. graph mode treats contrastive examples as graph pairs from a global perspective, whichmainly targets on graph-level representation learning for graph prediction tasks.
Revised: • Graph v.s. graph mode treats contrastive examples as graph pairs from a global perspective, which mainly targets on graph-level representation learning for graph prediction tasks.

Original: Specifically,AD-GCL [47] aims to avoid capturing redundant information during the training by optimizingadversarial graph augmentation strategies in GCL, and designs a trainable non-i.i
Revised: Specifically, AD-GCL [48] aims to avoid capturing redundant information during the training by optimizing adversarial graph augmentation strategies in GCL, and designs a trainable non-i

Original: d. edge-droppinggraph augmentation.
Revised: i.d. edge-dropping graph augmentation.

Original: JOAO [59] adopts a bi-level optimization framework to search the optimalstrategy among multiple types of augmentations such as uniform edge or node dropping, subgraphsampling.
Revised: JOAO [61] adopts a bi-level optimization framework to search the optimal strategy among multiple types of augmentations such as uniform edge or node dropping, subgraph sampling.

Original: GraphCL [60] extensively studies graph structure augmentations including randomedge removing, node dropping and subgraph sampling.
Revised: GraphCL [62] extensively studies graph structure augmentations including random edge removing, node dropping and subgraph sampling.

Original: C.4 Hyper-parameter Setting
Revised: C.4 Hyper-parameter Setting

Original: Training Configuration.
Revised: Training Configuration.

Original: We summarize the configuration of our GCL framework, including the GNNencoder and training parameters.
Revised: We summarize the configuration of our GCL framework, including the GNN encoder and training parameters.

Original: For node representation learning, we used GCN [26] encoder, and setthe number of GCN layers to 2 , the size of hidden dimension for each layer to 512 .
Revised: For node representation learning, we used GCN [26] encoder, and set the number of GCN layers to 2 , the size of hidden dimension for each layer to 512 .

Original: The training epochis 1000 .
Revised: The training epoch is 1000 .

Original: For graph representation learning, we adopted GIN [58] encoder with 5 layers, which wasconcatenated by a readout function that adds node representations for pooling.
Revised: For graph representation learning, we adopted GIN [60] encoder with 5 layers, which was concatenated by a readout function that adds node representations for pooling.

Original: The embedding sizewas set to 32 for TU dataset and 300 for OBG dataset.
Revised: The embedding size was set to 32 for TU dataset and 300 for OBG dataset.

Original: We used 100 training epochs with batch size 32 .
Revised: We used 100 training epochs with batch size 32 .

Original: In all the experiments, we used the Adam optimizer with learning rate 0 .
Revised: In all the experiments, we used the Adam optimizer with learning rate 0 .

Original: 001 and weight decay 10 − 5 .
Revised: 001 and weight decay 10 − 5 .

Original: For data augmentation, we adopted both edge perturbation and feature masking, whose perturbationratio σ e and σ f were tuned by grid search among { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } based onthe validation set.
Revised: For data augmentation, we adopted both edge perturbation and feature masking, whose perturbation ratio σ e and σ f were tuned by grid search among { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 } based on the validation set.

Original: Note that in our formulation Eq.
Revised: Note that in our formulation Eq.

Original: (7), the augmentation strength ϵ =
Revised: (7), the augmentation strength ϵ =

Original: σ e · m wherem is the number of edges in the input graph.
Revised: σ e · m where m is the number of edges in the input graph.

Original: Evaluation Protocol.
Revised: Evaluation Protocol.

Original: When evaluating the quality of learned representations on downstream tasks inan unsupervised setting, we adopted the evaluation protocol proposed in [47].
Revised: When evaluating the quality of learned representations on downstream tasks in an unsupervised setting, we adopted the evaluation protocol proposed in [48].

Original: Specifically, based onthe representations given by the encoder, we trained and evaluated a Logistic classifier or a Ridgeregressor with L 2 regularization, whose weight was tuned among { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 , 100 . 0 }on the validation set for each dataset.
Revised: Specifically, based on the representations given by the encoder, we trained and evaluated a Logistic classifier or a Ridge regressor with L 2 regularization, whose weight was tuned among { 0 . 001 , 0 . 01 , 0 . 1 , 1 . 0 , 10 . 0 , 100 . 0 } on the validation set for each dataset.

Original: C.5 Out-of-Domain Transfer Learning
Revised: C.5 Out-of-Domain Transfer Learning

Original: A more challenging transfer learning setting is proposed in [39], which supports model training onmultiple graphs from academic and social networks and transferring to different downstream tasks forother datasets.
Revised: A more challenging transfer learning setting is proposed in [40], which supports model training on multiple graphs from academic and social networks and transferring to different downstream tasks for other datasets.

Original: We further conducted a transfer learning experiment under such setting to demonstratethe out-of-domain transferability of our proposed solution GCL-TAGS.
Revised: We further conducted a transfer learning experiment under such setting to demonstrate the out-of-domain transferability of our proposed solution GCL-TAGS.

Original: Specifically, we pre-trainthe encoder on the Yelp dataset [63] which contains 716,847 nodes and 13,954,819 edges; and thenthe encoder is fine-tuned and evaluated on the US-airport dataset [40] for node classification taskand three TU social networks [33] for graph classification task via 10-fold CV.
Revised: Specifically, we pre-train the encoder on the Yelp dataset [65] which contains 716,847 nodes and 13,954,819 edges; and then the encoder is fine-tuned and evaluated on the US-airport dataset [41] for node classification task and three TU social networks [34] for graph classification task via 10-fold CV.

Original: We sample 80,0002-hop ego-nets from the Yelp dataset for pre-training, and use node degree as the node feature for alldatasets.
Revised: We sample 80, 2-hop ego-nets from the Yelp dataset for pre-training, and use node degree as the node feature for all datasets.

Original: Table 9 summarizes the results of downstream classification accuracy.
Revised: Table 9 summarizes the results of downstream classification accuracy.

Original: Overall, we can observesatisfactory performance gain using contrastive learning to pre-train the GIN encoder on Yelp datasetunder such a cross-domain setting.
Revised: Overall, we can observe satisfactory performance gain using contrastive learning to pre-train the GIN encoder on Yelp dataset under such a cross-domain setting.

Original: Meanwhile, our proposed method GCL-TAGS outperforms othercontrastive learning methods on three out of four downstream datasets.[Table]
Revised: Meanwhile, our proposed method GCL-TAGS outperforms other contrastive learning methods on three out of four downstream datasets.

Original: D Model AnalysisD.1 Influence of Choosing Eigenvalues
Revised: D Model Analysis D.1 Influence of Choosing Eigenvalues

Original: To reduce the time complexity of eigen-decomposition when calculating the spectrum norm L GS ( ∆ ) [Table] we can approximate the norm by only using the K lowest- and highest-eigenvalues.
Revised: To reduce the time complexity of eigen-decomposition when calculating the spectrum norm L GS ( ∆ ) , we can approximate the norm by only using the K lowest- and highest-eigenvalues.

Original: The timecomplexity of optimizing the augmentation scheme in Eq.
Revised: The time complexity of optimizing the augmentation scheme in Eq.

Original: (5) with T iterations is O ( TKn 2 ) .
Revised: (5) with T iterations is O ( TKn 2 ) .

Original: Thisexperiment shows the influence of K to the resulting GCL performance.
Revised: This experiment shows the influence of K to the resulting GCL performance.

Original: Since the graphs encounteredin the node prediction tasks are much larger than those in graph prediction tasks [Table] we used the nodeclassification datasets in Table 5 to conduct this experiment.
Revised: Since the graphs encountered in the node prediction tasks are much larger than those in graph prediction tasks, we used the node classification datasets in Table 5 to conduct this experiment.

Original: Specifically, we test influence of Kon four large graphs representing different types of networks:
Revised: Specifically, we test influence of K on four large graphs representing different types of networks:

Original: PubMed citation network, Wiki-CShyperlink network, Amazon-Computers co-purchase network and Coauthor-CS network.
Revised: PubMed citation network, Wiki-CS [Table] hyperlink network, Amazon-Computers co-purchase network and Coauthor-CS network.

Original: We tuned
Revised: We tuned

Original: K among { 50 , 100 , 200 , 500 , 1000 , 5000 } for each of the datasets containing n ≥ 10 , 000 nodes.
Revised: K among { 50 , 100 , 200 , 500 , 1000 , 5000 } for each of the datasets containing n ≥ 10 , 000 nodes.

Original: The other components of GCL maintained the same, except the resulting augmentation scheme usingspectrum norm with different K .
Revised: The other components of GCL maintained the same, except the resulting augmentation scheme using spectrum norm with different K .

Original: [Figure]
Revised: [Figure]

Original: Figure 3 demonstrates the performance of GCL-TAGS on the node classification task when different
Revised: Figure 4 demonstrates the performance of GCL-TAGS on the node classification task when different

Original: K was used to generate the augmentation scheme.
Revised: K was used to generate the augmentation scheme.

Original: The x-axis denotes the value of K with “all”indicating that all the eigenvalues were used.
Revised: The x-axis denotes the value of K with “all” indicating that all the eigenvalues were used.

Original: The performance decreases marginally when we useda smaller K , and generally when K = 1000 we can still achieve a comparable performance.
Revised: The performance decreases marginally when we used a smaller K , and generally when K = 1000 we can still achieve a comparable performance.

Original: Thissuggests that low and high eigenvalues are already quite informative in capturing graph structuralproperties.
Revised: This suggests that low and high eigenvalues are already quite informative in capturing graph structural properties.

Original: Similar phenomenon is also discussed in previous works [30]: small eigenvalues carrysmoothly varying signals (e.g., similar neighbor nodes within the same cluster), while high eigenvaluescarry sharply varying signals (e.g., dissimilar nodes from different clusters).
Revised: Similar phenomenon is also discussed in previous works [30]: small eigenvalues carry smoothly varying signals (e.g., similar neighbor nodes within the same cluster), while high eigenvalues carry sharply varying signals (e.g., dissimilar nodes from different clusters).

Original: D.2 Gain of Spectrum Guided Augmentation on Other GCL Frameworks
Revised: D.2 Gain of Spectrum Guided Augmentation on Other GCL Frameworks

Original: In this experiment, we use an ablation study to evaluate the effectiveness of the graph spectrumguided topology augmentation scheme when applied to different contrastive learning frameworks.
Revised: In this experiment, we use an ablation study to evaluate the effectiveness of the graph spectrum guided topology augmentation scheme when applied to different contrastive learning frameworks.

Original: Wefocus on GCL for node-level representation learning, as this line of work adopts distinct contrastiveobjectives (e.g., bootstrapping in BGRL, and Barlow twins in GBT) and contrastive modes (e.g., nodev.s. node in GRACE, and node v.s. graph in MVGRL), such that we can comprehensively demonstratethe effectiveness of our proposed augmentation in covering a variety of GCL frameworks.[Table]
Revised: We focus on GCL for node-level representation learning, as this line of work adopts distinct contrastive objectives (e.g., bootstrapping in BGRL, and Barlow twins in GBT) and contrastive modes (e.g., node v.s. node in GRACE, and node v.s. graph in MVGRL), such that we can comprehensively demonstrate the effectiveness of our proposed augmentation in covering a variety of GCL frameworks.

Original: Specifically, we replace the original uniformly random edge removing augmentation in GRACE,
Revised: Specifically, we replace the original uniformly random edge removing augmentation in GRACE,

Original: BGRL, GBT, and the diffusion matrix based augmentation in MVGRL with the proposed spectrumbased augmentation scheme, and use -TAGS as suffix to denote them.
Revised: BGRL, GBT, and the diffusion matrix based augmentation in MVGRL with the proposed spectrum based augmentation scheme, and use -TAGS as suffix to denote them.

Original: Note that MVGRL-TAGS isbasically GCL-TAGS since it uses the same contrastive objective as in MVGRL such that both nodeand graph-level representations are obtained to serve a broader range of downstream tasks.
Revised: Note that MVGRL-TAGS is basically GCL-TAGS since it uses the same contrastive objective as in MVGRL such that both nodeand graph-level representations are obtained to serve a broader range of downstream tasks.

Original: Table 10 shows the results of plugging our augmentation scheme on four types of GCL frameworks.
Revised: Table 10 shows the results of plugging our augmentation scheme on four types of GCL frameworks.

Original: We can observe that our augmentation scheme does not depend on a particular contrastive objective,but brings a clear performance gain across different GCL frameworks.
Revised: We can observe that our augmentation scheme does not depend on a particular contrastive objective, but brings a clear performance gain across different GCL frameworks.

Original: Intuitively, our augmentationcaptures the essential structural properties by perturbing edges that cause large spectral change.
Revised: Intuitively, our augmentation captures the essential structural properties by perturbing edges that cause large spectral change.

Original: Therefore, no matter what contrastive objective or mode is used, maximizing the correspondence oftwo views encourages the encoder to ignore the information carried by such sensitive edges.
Revised: Therefore, no matter what contrastive objective or mode is used, maximizing the correspondence of two views encourages the encoder to ignore the information carried by such sensitive edges.

Original: Thisdemonstrates the importance of studying graph spectral properties for graph augmentation.
Revised: This demonstrates the importance of studying graph spectral properties for graph augmentation.

Original: D.3 Spatial Behavior of Spectrum Guided Augmentation

Original: To intuitively show the spatial change caused by perturbing the graph spectrum, we visualize a casestudy on a random geometric graph in Figure 4.

Original: Figure 4a draws the original graph.

Original: Figure 4b showsthe perturbation probability obtained by maximizing L GS ( ∆ ) .

Original: Figure 4c illustrates the perturbationprobability obtained by minimizing L GS ( ∆ ) .

Original: [Equation]
Revised: [Table]

Original: graph 0. [Equation] probability ∆ 1 by max ∆ 1 L GS ( ∆ 1 ) 0. (c)
Revised: D.3 Spatial Behavior of Spectrum Guided Augmentation

Original: [Figure]

Original: We can observe that maximizing the spectral norm assigns larger probability to remove edges bridgingclusters such that the clustering effect becomes more obvious, while minimizing the spectral normtends to add edges connecting clusters such that the clustering effect is blurred.

Original: Intuitively, suchan augmentation perturbs these spurious edges that can easily affects the structural property (e.g.clustering) to preserve structural invariant, and the information about these edges are disentangledand minimized in the learned representations by contrastive learning.

Original: Wealso provide a theoretical justification which reveals the interplay of spectral change and spatialchange.
Revised: We provide a theoretical justification which reveals the interplay of spectral change and spatial change.

Original: As derived in Theorem 2 of [2], given an edge flip ∆ w ij =
Revised: As derived in Theorem 2 of [2], given an edge flip ∆ w ij =

Original: (1 − 2 A ij ) ∈ {− 1 , 1 } betweennode i and j (e.g. if ∆ w ij = 1 , adding edge ( i, j ) ; otherwise removing edge ( i, j ) ), the k -theigenvalue is changed as λ ′ k
Revised: (1 − 2 A ij ) ∈ {− 1 , 1 } between node i and j (e.g. if ∆ w ij = 1 , adding edge ( i, j ) ; otherwise removing edge ( i, j ) ), the k -th eigenvalue is changed as λ ′ k

Original: = λ k + ∆ λ k .
Revised: = λ k + ∆ λ k .

Original: ∆ λ k can be approximated by:
Revised: ∆ λ k can be approximated by:

Original: [Equation]
Revised: [Equation]

Original: 12) where u k is the k -th eigenvector corresponding to the eigenvalue λ k .
Revised: (12) where u k is the k -th eigenvector corresponding to the eigenvalue λ k .

Original: If we only focus on themagnitude of eigenvalue change, we can obtain:
Revised: If we only focus on the magnitude of eigenvalue change, we can obtain:

Original: [Equation]
Revised: [Equation]

Original: Remarks .
Revised: Remarks .

Original: Since the eigenvectors are normalized, we can treat ( u 2 ki + u 2 kj ) as a constant as it is arelatively stable value.
Revised: Since the eigenvectors are normalized, we can treat ( u 2 ki + u 2 kj ) as a constant as it is a relatively stable value.

Original: Based on the theory in spectral clustering [34], if the eigenvectors of node iand node j have a larger difference (i.e., ∥ u .,i − u .,j
Revised: Based on the theory in spectral clustering [35], if the eigenvectors of node i and node j have a larger difference (i.e., ∥ u .,i − u .,j

Original: ∥ 2 is large), these two nodes should belong todifferent clusters.
Revised: ∥ 2 is large), these two nodes should belong to different clusters.

Original: The first term in Eq.
Revised: The first term in Eq.

Original: (13) suggests a larger eigenvalue change, if u ki and u kj havea larger difference.
Revised: (13) suggests a larger eigenvalue change, if u ki and u kj have a larger difference.

Original: Therefore, flipping the edge between nodes from different clusters (thus with alarger ( u ki − u kj ) 2 ) results in a larger spectral change.
Revised: Therefore, flipping the edge between nodes from different clusters (thus with a larger ( u ki − u kj ) 2 ) results in a larger spectral change.

Original: The second term suggests that such an effectbecomes more obvious for eigenvalues close to 0 or 2.
Revised: The second term suggests that such an effect becomes more obvious for eigenvalues close to 0 or 2.

Original: D.4 The Convergence of Optimizing the Spectrum Guided Augmentations
Revised: D.4 The Convergence of Optimizing the Spectrum Guided Augmentations

Original: In this section, we show how the graph spectrum norm changes as the augmentation optimizationproceeds following Eq.
Revised: In this section, we show how the graph spectrum norm changes as the augmentation optimization proceeds following Eq.

Original: To better show the relative change of graph spectrum compared withthe original graph, we calculate L GS ( ∆ ) normalized by the spectrum norm of the original graph,that is, L GS ( ∆ ) / L GS ( 0 ) =
Revised: To better show the relative change of graph spectrum compared with the original graph, we calculate L GS ( ∆ ) normalized by the spectrum norm of the original graph, that is, L GS ( ∆ ) / L GS ( 0 ) =

Original: ∥ eig ( Lap ( A + C ◦ ∆ ))
Revised: ∥ eig ( Lap ( A + C ◦ ∆ ))

Original: ∥ 22 / ∥ eig ( Lap ( A )) ∥ 22 .
Revised: ∥ 22 / ∥ eig ( Lap ( A )) ∥ 22 .

Original: The value of normalized
Revised: The value of normalized

Original: L GS ( ∆ ) / L GS ( 0 ) is reported in Figure 5.
Revised: L GS ( ∆ ) / L GS ( 0 ) is reported in Figure 5.

Original: [Figure]
Revised: [Figure]

Original: Based on the graph spectral theory, the eigenvalues are bounded within [0 , 2] , thus the L2-norm of thegraph spectrum is also bounded by the total number of nodes, For example, the values of the originalspectrum norm L GS ( 0 ) for Cora, Amazon-Photo, Wiki-CS and PubMed are 24 . 88 , 22 . 13 , 31 . 79 ,and 65 . 26 respectively.
Revised: Based on the graph spectral theory, the eigenvalues are bounded within [0 , 2] , thus the L2-norm of the graph spectrum is also bounded by the total number of nodes, For example, the values of the original spectrum norm L GS ( 0 ) for Cora, Amazon-Photo, Wiki-CS and PubMed are 24 . 88 , 22 . 13 , 31 . 79 , and 65 . 26 respectively.

Original: From Figure 5, we can observe that maximizing L GS ( ∆ ) indeed resultsin an a larger spectrum norm compared with the original graph (i.e. L GS ( ∆ ) /
Revised: From Figure 5, we can observe that maximizing L GS ( ∆ ) indeed results in an a larger spectrum norm compared with the original graph (i.e. L GS ( ∆ ) /

Original: L GS ( 0 ) > 1 ), whileminimizing it achieves a smaller spectrum norm (i.e. L GS ( ∆ ) /
Revised: L GS ( 0 ) > 1 ), while minimizing it achieves a smaller spectrum norm (i.e. L GS ( ∆ ) /

Original: L GS ( 0 ) < 1 ).
Revised: L GS ( 0 ) < 1 ).