Original: GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks
Revised: GARNET: Reduced-Rank Topology Learning for Robust and Scalable Graph Neural Networks

Original: Abstract
Revised: Abstract

Original: Graph neural networks (GNNs) have been increasingly deployed in various applications that involve learning on non-Euclidean data.
Revised: Graph neural networks (GNNs) have been increasingly deployed in various applications that involve learning on non-Euclidean data.

Original: However, recent studies show that GNNs are vulnerable to graph adversarial attacks.
Revised: However, recent studies show that GNNs are vulnerable to graph adversarial attacks.

Original: Although there are several defense methods to improve GNN robustness by eliminating adversarial components, they may also impair the underlying clean graph structure that contributes to GNN training.
Revised: Although there are several defense methods to improve GNN robustness by eliminating adversarial components, they may also impair the underlying clean graph structure that contributes to GNN training.

Original: In addition, few of those defense models can scale to large graphs due to their high computational complexity and memory usage.
Revised: In addition, few of those defense models can scale to large graphs due to their high computational complexity and memory usage.

Original: In this paper, we propose GARNET 1 , a scalable spectral method to boost the adversarial robustness of GNN models.
Revised: In this paper, we propose GARNET 1 , a scalable spectral method to boost the adversarial robustness of GNN models.

Original: GARNET first leverages weighted spectral embedding to construct a base graph, which is not only resistant to adversarial attacks but also contains critical (clean) graph structure for GNN training.
Revised: GARNET first leverages weighted spectral embedding to construct a base graph, which is not only resistant to adversarial attacks but also contains critical (clean) graph structure for GNN training.

Original: Next, GARNET further refines the base graph by pruning additional uncritical edges based on probabilistic graphical model.
Revised: Next, GARNET further refines the base graph by pruning additional uncritical edges based on probabilistic graphical model.

Original: GARNET has been evaluated on various datasets, including a large graph with millions of nodes.
Revised: GARNET has been evaluated on various datasets, including a large graph with millions of nodes.

Original: Our extensive experiment results show that
Revised: Our extensive experiment results show that

Original: GARNET achieves adversarial accuracy improvement and runtime speedup over state-of-the-art GNN (defense) models by up to 10 .
Revised: GARNET achieves adversarial accuracy improvement and runtime speedup over state-of-the-art GNN (defense) models by up to 10 .

Original: 23% and 14 .
Revised: 23% and 14 .

Original: 7 × , respectively.
Revised: 7 × , respectively.

Original:  Introduction
Revised:  Introduction

Original: Recent years have witnessed a surge of interest in graph neural networks (GNNs), which incorporate both graph structure and node attributes to produce low-dimensional embedding vectors that maximally preserve graph structural information [1].
Revised: Recent years have witnessed a surge of interest in graph neural networks (GNNs), which incorporate both graph structure and node attributes to produce low-dimensional embedding vectors that maximally preserve graph structural information [1].

Original: GNNs have achieved promising results in various real-world applications, such as recommendation systems [2], self-driving car [3] and chip placements [4].
Revised: GNNs have achieved promising results in various real-world applications, such as recommendation systems [2], self-driving car [3] and chip placements [4].

Original: However, recent studies have shown that adversarial attacks on graph structure accomplished by inserting, deleting or rewiring edges in an unnoticeable way, can easily fool the GNN models and drastically degrade their accuracy in downstream tasks (e.g., node classification) [5, 6].
Revised: However, recent studies have shown that adversarial attacks on graph structure accomplished by inserting, deleting or rewiring edges in an unnoticeable way, can easily fool the GNN models and drastically degrade their accuracy in downstream tasks (e.g., node classification) [5, 6].

Original: In literature, one of the most effective ways to defend GNNs is to purify the graph by removing adversarial graph structures.
Revised: In literature, one of the most effective ways to defend GNNs is to purify the graph by removing adversarial graph structures.

Original: Entezari et al.
Revised: Entezari et al.

Original: [7] observe that adversarial attacks mainly affect highrank graph properties; thus they propose to first construct a low-rank graph by performing truncated singular value decomposition (TSVD) on the graph adjacency matrix, which can then be exploited for training a robust GNN model.
Revised: [7] observe that adversarial attacks mainly affect highrank graph properties; thus they propose to first construct a low-rank graph by performing truncated singular value decomposition (TSVD) on the graph adjacency matrix, which can then be exploited for training a robust GNN model.

Original: Later, Jin et al.
Revised: Later, Jin et al.

Original: [8] propose Pro-GNN to jointly learn a new graph and a robust GNN model with the low-rank constraints imposed by the graph structure.
Revised: [8] propose Pro-GNN to jointly learn a new graph and a robust GNN model with the low-rank constraints imposed by the graph structure.

Original: While prior methods using low-rank approximation largely eliminate adversarial components in the graph spectrum, they involve dense adjacency matrices during GNN training, leading to a much higher time/space complexity and prohibiting their applications in large-scale graph learning tasks.
Revised: While prior methods using low-rank approximation largely eliminate adversarial components in the graph spectrum, they involve dense adjacency matrices during GNN training, leading to a much higher time/space complexity and prohibiting their applications in large-scale graph learning tasks.

Original: [Figure]
Revised: [Figure]

Original: In addition, due to the high computational cost of TSVD, existing low-rank based methods can only preserve top r singular components (e
Revised: In addition, due to the high computational cost of TSVD, existing low-rank based methods can only preserve top r singular components (e

Original: [Figure]
Revised: [Figure]

Original: g., r = 50 ).
Revised: g., r = 50 ).

Original: Consequently, as shown in Figure 1(a)
Revised: Consequently, as shown in Figure 1(a)

Original: [Figure]
Revised: [Figure]

Original: these methods may lose a wide range of clean graph spectrum that corresponds to important structures of the clean graph in the spatial domain.
Revised: these methods may lose a wide range of clean graph spectrum that corresponds to important structures of the clean graph in the spatial domain.

Original: This is confirmed in Figure 1(c)
Revised: This is confirmed in Figure 1(c)

Original: [Figure]
Revised: [Figure]

Original: where the clean accuracy of the
Revised: where the clean accuracy of the

Original: TSVD-based method largely increases when preserving more spectral information via increasing the graph rank r .
Revised: TSVD-based method largely increases when preserving more spectral information via increasing the graph rank r .

Original: In other words, prior low-rank approximation methods eliminate high-rank adversarial components at the cost of inevitably impairing the important (clean) graph structure, which degrades the overall quality of the reconstructed graph and therefore limits the performance of GNN training.
Revised: In other words, prior low-rank approximation methods eliminate high-rank adversarial components at the cost of inevitably impairing the important (clean) graph structure, which degrades the overall quality of the reconstructed graph and therefore limits the performance of GNN training.

Original: In this work
Revised: In this work

Original: [Figure]
Revised: [Figure]

Original: we propose GARNET, a novel spectral approach to learning the underlying clean graph topology of an adversarial graph via combining spectral embedding with probabilistic graphical model (PGM), where the learned graph structure encodes the conditional dependence among lowdimensional node representations (spectral embedding vectors)
Revised: we propose GARNET, a novel spectral approach to learning the underlying clean graph topology of an adversarial graph via combining spectral embedding with probabilistic graphical model (PGM), where the learned graph structure encodes the conditional dependence among lowdimensional node representations (spectral embedding vectors)

Original: More concretely, given an adversarial graph, GARNET first constructs a base graph topology by leveraging weighted spectral embeddings that are resistant to adversarial attacks, which is followed by an effective and efficient graph refinement scheme for pruning noncritical edges in the base graph by exploiting PGM.
Revised: More concretely, given an adversarial graph, GARNET first constructs a base graph topology by leveraging weighted spectral embeddings that are resistant to adversarial attacks, which is followed by an effective and efficient graph refinement scheme for pruning noncritical edges in the base graph by exploiting PGM.

Original: By recovering the clean graph structure, Figures 1(a) and 1(b) show that the adversarial graph purified by GARNET largely restores the rank of the underlying clean graph.
Revised: By recovering the clean graph structure, Figures 1(a) and 1(b) show that the adversarial graph purified by GARNET largely restores the rank of the underlying clean graph.

Original: Thus, GARNET can be viewed as a reduced-rank topology learning approach that slightly reduces the rank of the input adversarial graph, which is fundamentally different from the prior low-rank based defense methods (e.g., TSVD and ProGNN).
Revised: Thus, GARNET can be viewed as a reduced-rank topology learning approach that slightly reduces the rank of the input adversarial graph, which is fundamentally different from the prior low-rank based defense methods (e.g., TSVD and ProGNN).

Original: Moreover, GARNET scales comfortably to large graphs due to its nearly-linear algorithm complexity, and produces a sparse yet high-quality graph that improves GNN robustness without involving any dense adjacency matrices during GNN training.
Revised: Moreover, GARNET scales comfortably to large graphs due to its nearly-linear algorithm complexity, and produces a sparse yet high-quality graph that improves GNN robustness without involving any dense adjacency matrices during GNN training.

Original: As a byproduct, unlike existing defense methods (e.g., ProGNN) that assume graphs to be homophilic, i.e, adjacent nodes in a graph tend to have similar attributes [10], GARNET does not have such an assumption and thus can protect
Revised: As a byproduct, unlike existing defense methods (e.g., ProGNN) that assume graphs to be homophilic, i.e, adjacent nodes in a graph tend to have similar attributes [10], GARNET does not have such an assumption and thus can protect

Original: GNNs against adversarial attacks on both homophilic and heterophilic graphs.
Revised: GNNs against adversarial attacks on both homophilic and heterophilic graphs.

Original: We evaluate GARNET on both homophilic and heterophilic datasets under strong graph adversarial attacks such as Nettack [5] and Metattack [6].
Revised: We evaluate GARNET on both homophilic and heterophilic datasets under strong graph adversarial attacks such as Nettack [5] and Metattack [6].

Original: Moreover, we further show the nearly-linear scalability of our approach on the ogbn-products dataset that consists of millions of nodes [11].
Revised: Moreover, we further show the nearly-linear scalability of our approach on the ogbn-products dataset that consists of millions of nodes [11].

Original: Our experimental results indicate that GARNET largely improves both clean and adversarial accuracy over baselines in most cases.
Revised: Our experimental results indicate that GARNET largely improves both clean and adversarial accuracy over baselines in most cases.

Original: Our main technical contributions are summarized as follows:
Revised: Our main technical contributions are summarized as follows:

Original: • To our knowledge, we are the first to exploit spectral graph embedding and probabilistic graphical model for improving robustness of GNN models, which is achieved by learning a reduced-rank graph topology for recovering the underlying clean graph structure from the input adversarial graph.
Revised: • To our knowledge, we are the first to exploit spectral graph embedding and probabilistic graphical model for improving robustness of GNN models, which is achieved by learning a reduced-rank graph topology for recovering the underlying clean graph structure from the input adversarial graph.

Original: • By recovering the critical edges that contribute to maximum likelihood estimation in PGM while ignoring adversarial components, GARNET produces a high-quality graph on which existing GNN models can be trained to achieve high accuracy.
Revised: • By recovering the critical edges that contribute to maximum likelihood estimation in PGM while ignoring adversarial components, GARNET produces a high-quality graph on which existing GNN models can be trained to achieve high accuracy.

Original: Our experimental results show that GARNET gains up to 10 .
Revised: Our experimental results show that GARNET gains up to 10 .

Original: 23% adversarial accuracy improvement over state-of-the-art defense baselines.
Revised: 23% adversarial accuracy improvement over state-of-the-art defense baselines.

Original: • Our proposed reduced-rank topology learning method has a nearly-linear complexity in time/space and produces a sparse graph structure for scalable GNN training.
Revised: • Our proposed reduced-rank topology learning method has a nearly-linear complexity in time/space and produces a sparse graph structure for scalable GNN training.

Original: This allows GARNET to run up to 14 .
Revised: This allows GARNET to run up to 14 .

Original: 7 × faster than prior defense methods on popular data sets such as Cora and Squirrel.
Revised: 7 × faster than prior defense methods on popular data sets such as Cora and Squirrel.

Original: In addition, GARNET scales comfortably to very large graph data sets with millions of nodes, while prior defense methods run out of memory even on a graph with 20 k nodes.
Revised: In addition, GARNET scales comfortably to very large graph data sets with millions of nodes, while prior defense methods run out of memory even on a graph with 20 k nodes.

Original:  Background 2.1 Undirected Probabilistic Graphical Models
Revised:  Background 2.1 Undirected Probabilistic Graphical Models

Original: Consider an n -dimensional random vector x that follows a multivariate Gaussian distribution x ∼
Revised: Consider an n -dimensional random vector x that follows a multivariate Gaussian distribution x ∼

Original: N (0 , Σ) , where Σ = E [ xx ⊤ ] ≻ 0 represents the covariance matrix, and Θ = Σ − 1 represents the precision matrix (inverse covariance matrix).
Revised: N (0 , Σ) , where Σ = E [ xx ⊤ ] ≻ 0 represents the covariance matrix, and Θ = Σ − 1 represents the precision matrix (inverse covariance matrix).

Original: Given a data matrix X ∈ R n × d that includes d i.i.d (independent and identically distributed) samples X
Revised: Given a data matrix X ∈ R n × d that includes d i.i.d (independent and identically distributed) samples X

Original: =
Revised: =

Original: [ x 1 , ..., x d ] , where x i ∼ N (0 , Σ) has an n -dimensional Gaussian distribution with zero mean, the goal of probabilistic graphical models (PGM) is to learn a precision matrix Θ that corresponds to an undirected graph structure G for encoding the conditional dependence between variables of the observations on columns of X [12, 13].
Revised: [ x 1 , ..., x d ] , where x i ∼ N (0 , Σ) has an n -dimensional Gaussian distribution with zero mean, the goal of probabilistic graphical models (PGM) is to learn a precision matrix Θ that corresponds to an undirected graph structure G for encoding the conditional dependence between variables of the observations on columns of X [12, 13].

Original: Specifically, the classical graphical Lasso method aims at estimating a sparse Θ through maximum likelihood estimation (MLE) of f ( x ) leveraging convex optimization [13].
Revised: Specifically, the classical graphical Lasso method aims at estimating a sparse Θ through maximum likelihood estimation (MLE) of f ( x ) leveraging convex optimization [13].

Original: In this work, we focus on one increasingly popular type of Gaussian graphical models, which is also known as attractive Gaussian Markov random fields (GMRFs).
Revised: In this work, we focus on one increasingly popular type of Gaussian graphical models, which is also known as attractive Gaussian Markov random fields (GMRFs).

Original: Attractive GMRFs restrict the precision matrix to be a Laplacian-like matrix Θ =
Revised: Attractive GMRFs restrict the precision matrix to be a Laplacian-like matrix Θ =

Original: L + Iσ 2 , where
Revised: L + Iσ 2 , where

Original: L = D −
Revised: L = D −

Original: A denotes the set of valid graph Laplacian matrices with D and A representing the diagonal degree matrix and adjacency matrix of the underlying undirected graph, respectively, I denotes the identity matrix, and σ 2 is a constant denoting prior data variance.
Revised: A denotes the set of valid graph Laplacian matrices with D and A representing the diagonal degree matrix and adjacency matrix of the underlying undirected graph, respectively, I denotes the identity matrix, and σ 2 is a constant denoting prior data variance.

Original: Similar to the graphical Lasso method [13], recent methods for estimating attractive GMRFs leverage emerging graph signal processing (GSP) techniques to solve the following convex problem [9, 14–17]:
Revised: Similar to the graphical Lasso method [13], recent methods for estimating attractive GMRFs leverage emerging graph signal processing (GSP) techniques to solve the following convex problem [9, 14–17]:

Original: [Equation]
Revised: [Equation]

Original: (1) where det ( · ) and tr ( · ) denote the determinant and trace operators, respectively, α is a hyperparameter to control the regularization term.
Revised: (1) where det ( · ) and tr ( · ) denote the determinant and trace operators, respectively, α is a hyperparameter to control the regularization term.

Original: The first two terms together can be interpreted as log-likelihood under a GMRF.
Revised: The first two terms together can be interpreted as log-likelihood under a GMRF.

Original: The last ℓ 1 regularization term is to enforce Θ (and the corresponding graph) to be sparse.
Revised: The last ℓ 1 regularization term is to enforce Θ (and the corresponding graph) to be sparse.

Original: If X is non-Gaussian, Equation 1 can be regarded as Laplacian estimation based on minimizing the Bregman divergence between positive definite matrices induced by the function Θ (cid:55)→ − log det(Θ) [18].
Revised: If X is non-Gaussian, Equation 1 can be regarded as Laplacian estimation based on minimizing the Bregman divergence between positive definite matrices induced by the function Θ (cid:55)→ − log det(Θ) [18].

Original: 2.2 Graph Adversarial Attacks
Revised: 2.2 Graph Adversarial Attacks

Original: Most existing graph adversarial attacks aim at degrading the accuracy of GNN models by inserting/deleting edges in an unnoticeable way (e.g., maintaining node degree distribution) [19].
Revised: Most existing graph adversarial attacks aim at degrading the accuracy of GNN models by inserting/deleting edges in an unnoticeable way (e.g., maintaining node degree distribution) [19].

Original: The most popular graph adversarial attacks fall into the following two categories: (1) targeted attack, (2) non-targeted attack.
Revised: The most popular graph adversarial attacks fall into the following two categories: (1) targeted attack, (2) non-targeted attack.

Original: The targeted attacks attempt to mislead a GNN model to produce a wrong prediction on a target sample (e.g., node), while the non-targeted attacks strive to degrade the overall accuracy of a GNN model for the whole graph data set.
Revised: The targeted attacks attempt to mislead a GNN model to produce a wrong prediction on a target sample (e.g., node), while the non-targeted attacks strive to degrade the overall accuracy of a GNN model for the whole graph data set.

Original: Dai et al.
Revised: Dai et al.

Original: [20] first formulate the targeted attack as a combinatorial optimization problem and leverages reinforcement learning to insert/delete edges such that the target node is misclassified.
Revised: [20] first formulate the targeted attack as a combinatorial optimization problem and leverages reinforcement learning to insert/delete edges such that the target node is misclassified.

Original: Zügner et al.
Revised: Zügner et al.

Original: [5] propose another targeted attack called Nettack, which produces an adversarial graph by maximizing the training loss of GNNs.
Revised: [5] propose another targeted attack called Nettack, which produces an adversarial graph by maximizing the training loss of GNNs.

Original: Zügner and Günnemann [6] further introduce Metattack, a non-targeted attack that treats the graph as a hyperparameter and uses meta-gradients to perturb the graph structure.
Revised: Zügner and Günnemann [6] further introduce Metattack, a non-targeted attack that treats the graph as a hyperparameter and uses meta-gradients to perturb the graph structure.

Original: It is worth noting that graph adversarial attacks have two different settings: poison (perturb a graph prior to GNN training) and evasion (perturb a graph after GNN training).
Revised: It is worth noting that graph adversarial attacks have two different settings: poison (perturb a graph prior to GNN training) and evasion (perturb a graph after GNN training).

Original: As shown by Zhu et al.
Revised: As shown by Zhu et al.

Original: [21], the poison setting is typically more challenging to defend, as it changes the graph structure that fools GNN training.
Revised: [21], the poison setting is typically more challenging to defend, as it changes the graph structure that fools GNN training.

Original: Thus, we aim to improve model robustness against attacks under the poison setting.
Revised: Thus, we aim to improve model robustness against attacks under the poison setting.

Original: 2.3 Graph Adversarial Defenses
Revised: 2.3 Graph Adversarial Defenses

Original: To defend GNN against adversarial attacks, Entezari et al.
Revised: To defend GNN against adversarial attacks, Entezari et al.

Original: [7] first observe that Nettack, a strong targeted attack, only changes the high-rank information of the adjacency matrix.
Revised: [7] first observe that Nettack, a strong targeted attack, only changes the high-rank information of the adjacency matrix.

Original: Thus, they propose to construct a low-rank graph by performing truncated SVD to undermine the effects of adversarial attacks.
Revised: Thus, they propose to construct a low-rank graph by performing truncated SVD to undermine the effects of adversarial attacks.

Original: Later, Jin et al.
Revised: Later, Jin et al.

Original: [8] propose Pro-GNN that adopts a similar idea yet encourages nodes with similar attributes to be connected when jointly learning the low-rank graph and GNN model.
Revised: [8] propose Pro-GNN that adopts a similar idea yet encourages nodes with similar attributes to be connected when jointly learning the low-rank graph and GNN model.

Original: Although those low-rank approximation based methods achieve state-of-the-art results on several datasets, they produce dense adjacency matrices that correspond to complete graphs, which would limit their applications for large graphs.
Revised: Although those low-rank approximation based methods achieve state-of-the-art results on several datasets, they produce dense adjacency matrices that correspond to complete graphs, which would limit their applications for large graphs.

Original: Moreover, they only preserve a small region of the graph spectrum and thus may lose too much important information corresponding to the clean graph structure in the spatial domain, which limits the performance of GNN training.
Revised: Moreover, they only preserve a small region of the graph spectrum and thus may lose too much important information corresponding to the clean graph structure in the spatial domain, which limits the performance of GNN training.

Original: Recently, Chang et al.
Revised: Recently, Chang et al.

Original: [22] exploit Laplacian eigenpairs to guide GNN training, which produces a robust model with quadratic time complexity and is thus not scalable to large graphs.
Revised: [22] exploit Laplacian eigenpairs to guide GNN training, which produces a robust model with quadratic time complexity and is thus not scalable to large graphs.

Original: In addition to the aforementioned spectral-based defense methods, GCNJaccard [23] and RS-GNN [24] purify the adversarial graph by connecting nodes with similar attributes or same labels.
Revised: In addition to the aforementioned spectral-based defense methods, GCNJaccard [23] and RS-GNN [24] purify the adversarial graph by connecting nodes with similar attributes or same labels.

Original: However, those defense methods explicitly (or implicitly) assume the underlying graph to be homophilic
Revised: However, those defense methods explicitly (or implicitly) assume the underlying graph to be homophilic

Original: [Figure]
Revised: [Figure]

Original: which results in rather poor performance when defending
Revised: which results in rather poor performance when defending

Original: GNN models on heterophilic graphs.
Revised: GNN models on heterophilic graphs.

Original: In contrast to the prior arts, GARNET achieves highly robust yet scalable performance on both homophilic and heterophilic graphs under adversarial attacks by leveraging a novel graph purification scheme based on spectral embedding and graphical model
Revised: In contrast to the prior arts, GARNET achieves highly robust yet scalable performance on both homophilic and heterophilic graphs under adversarial attacks by leveraging a novel graph purification scheme based on spectral embedding and graphical model

Original: [Figure]
Revised: [Figure]

Original:  The GARNET Approach
Revised:  The GARNET Approach

Original: [Figure]
Revised: [Figure]

Original: Recently, Entezari et al.
Revised: Recently, Entezari et al.

Original: [7] and Jin et al.
Revised: [7] and Jin et al.

Original: [8] have shown that the well-known graph adversarial attacks (e.g., Nettack and Metattack) are essentially high-rank attacks, which increase graph rank by enlarging the smallest singular values of adjacency matrix when perturbing the graph structure, while rest of the graph spectrum remains almost the same.
Revised: [8] have shown that the well-known graph adversarial attacks (e.g., Nettack and Metattack) are essentially high-rank attacks, which increase graph rank by enlarging the smallest singular values of adjacency matrix when perturbing the graph structure, while rest of the graph spectrum remains almost the same.

Original: Consequently, a natural way for improving GNN robustness is to find the low-rank approximation of the adversarial adjacency matrix.
Revised: Consequently, a natural way for improving GNN robustness is to find the low-rank approximation of the adversarial adjacency matrix.

Original: Low-rank topology learning (prior work).
Revised: Low-rank topology learning (prior work).

Original: Given an adversarial adjacency matrix A adv ∈ R n × n ,
Revised: Given an adversarial adjacency matrix A adv ∈ R n × n ,

Original: Entezari et al.
Revised: Entezari et al.

Original: [7] propose to reconstruct a low-rank approximated adjacency matrix via performing
Revised: [7] propose to reconstruct a low-rank approximated adjacency matrix via performing

Original: TSVD: ˆ A = U Σ V T , where Σ ∈ R r × r is a diagonal matrix consisting of r largest singular values of A adv .
Revised: TSVD: ˆ A = U Σ V T , where Σ ∈ R r × r is a diagonal matrix consisting of r largest singular values of A adv .

Original: U ∈ R n × r and V ∈ R n × r contain the corresponding left and right singular vectors, respectively.
Revised: U ∈ R n × r and V ∈ R n × r contain the corresponding left and right singular vectors, respectively.

Original: As the largest singular values are hardly affected by graph adversarial attacks, the reconstructed low-rank adjacency matrix ˆ A is resistant to adversarial attacks.
Revised: As the largest singular values are hardly affected by graph adversarial attacks, the reconstructed low-rank adjacency matrix ˆ A is resistant to adversarial attacks.

Original: However, due to the high computational cost of TSVD, ˆ A is typically computed by only using top r largest singular values and their corresponding singular vectors, where r is a relatively small number (e.g., r = 50 ).
Revised: However, due to the high computational cost of TSVD, ˆ A is typically computed by only using top r largest singular values and their corresponding singular vectors, where r is a relatively small number (e.g., r = 50 ).

Original: Consequently, the rank of ˆ A is only r = 50 , which is two orders of magnitude smaller than the rank of the clean graph, as shown in Figure 1(a).
Revised: Consequently, the rank of ˆ A is only r = 50 , which is two orders of magnitude smaller than the rank of the clean graph, as shown in Figure 1(a).

Original: Since these low-rank methods are overly aggressive in reducing the graph rank, ˆ A may lose too much important spectral information corresponding to the clean graph structure.
Revised: Since these low-rank methods are overly aggressive in reducing the graph rank, ˆ A may lose too much important spectral information corresponding to the clean graph structure.

Original: As shown in Figure 1(c), the clean accuracy of the
Revised: As shown in Figure 1(c), the clean accuracy of the

Original: TSVD-based method is largely improved by increasing the graph rank r , which indicates the low-rank graph obtained with a small r loses the key graph structure contributing to GNN training.
Revised: TSVD-based method is largely improved by increasing the graph rank r , which indicates the low-rank graph obtained with a small r loses the key graph structure contributing to GNN training.

Original: Note that the adversarial and clean graphs share most of the graph structure, as adversarial attacks perturb the clean graph in an unnoticeable way.
Revised: Note that the adversarial and clean graphs share most of the graph structure, as adversarial attacks perturb the clean graph in an unnoticeable way.

Original: Consequently, losing those important clean graph structures will also limit
Revised: Consequently, losing those important clean graph structures will also limit

Original: Reduced-rank topology learning (this work).
Revised: Reduced-rank topology learning (this work).

Original: Given the adversarial graph G adv and its adjacency matrix A adv , our goal is to learn a reduced-rank graph, which slightly reduces the rank of G adv to mitigate the effects of adversarial attacks, while retaining most of the important graph spectrum corresponding to the clean graph structure.
Revised: Given the adversarial graph G adv and its adjacency matrix A adv , our goal is to learn a reduced-rank graph, which slightly reduces the rank of G adv to mitigate the effects of adversarial attacks, while retaining most of the important graph spectrum corresponding to the clean graph structure.

Original: As adversarial attacks mainly affect the least dominant singular components of A adv [7], one straightforward way for constructing such a reduced-rank graph is to utilize all the singular components except those least dominant ones via TSVD.
Revised: As adversarial attacks mainly affect the least dominant singular components of A adv [7], one straightforward way for constructing such a reduced-rank graph is to utilize all the singular components except those least dominant ones via TSVD.

Original: Nonetheless, computing such a large number of singular components is computationally expensive [25], and is thus not scalable to large graphs.
Revised: Nonetheless, computing such a large number of singular components is computationally expensive [25], and is thus not scalable to large graphs.

Original: To learn the reduced-rank graph in a scalable way, in this work, we leverage only the top few (e.g., 50) dominant singular components of A adv to restore its important graph spectrum, via recovering the corresponding clean graph structure with the aid of PGM.
Revised: To learn the reduced-rank graph in a scalable way, in this work, we leverage only the top few (e.g., 50) dominant singular components of A adv to restore its important graph spectrum, via recovering the corresponding clean graph structure with the aid of PGM.

Original: Figure 2 gives an overview of our proposed approach, GARNET, which consists of three major phases.
Revised: Figure 2 gives an overview of our proposed approach, GARNET, which consists of three major phases.

Original: The first phase constructs a base graph by exploiting spectral embedding and a scalable nearest-neighbor graph algorithm.
Revised: The first phase constructs a base graph by exploiting spectral embedding and a scalable nearest-neighbor graph algorithm.

Original: The second phase further refines the base graph by pruning noncritical edges based on PGM.
Revised: The second phase further refines the base graph by pruning noncritical edges based on PGM.

Original: The last phase trains existing GNN models on the refined base graph to improve their robustness.
Revised: The last phase trains existing GNN models on the refined base graph to improve their robustness.

Original: Next, we will first describe our notion of clean graph recovery via PGM as well as the scalability issue of prior
Revised: Next, we will first describe our notion of clean graph recovery via PGM as well as the scalability issue of prior

Original: PGM-based work in Section 3.1, which motivates us to develop scalable GARNET kernels described in Sections 3.2 and 3.3.
Revised: PGM-based work in Section 3.1, which motivates us to develop scalable GARNET kernels described in Sections 3.2 and 3.3.

Original: We further provide the overall complexity of GARNET in Section 3.4.
Revised: We further provide the overall complexity of GARNET in Section 3.4.

Original: 3.1 Graph Recovery via Graphical Model
Revised: 3.1 Graph Recovery via Graphical Model

Original: A general philosophy behind PGM is that there exists an underlying graph G , whose structure determines the joint probability distribution of the observations on the data entities, i.e., columns of a data matrix X ∈ R n × d , where n is the number of data points, d the dimension per data point.
Revised: A general philosophy behind PGM is that there exists an underlying graph G , whose structure determines the joint probability distribution of the observations on the data entities, i.e., columns of a data matrix X ∈ R n × d , where n is the number of data points, d the dimension per data point.

Original: To recover the underlying graph structure from the data matrix X , one common way is to leverage MLE by solving Equation 1 in Section 2.1.
Revised: To recover the underlying graph structure from the data matrix X , one common way is to leverage MLE by solving Equation 1 in Section 2.1.

Original: As the top few dominant singular components of the adjacency matrix capture the corresponding graph structure, we can naturally construct the data matrix X based on those dominant singular components, and then adopt PGM to recover an underlying graph via
Revised: As the top few dominant singular components of the adjacency matrix capture the corresponding graph structure, we can naturally construct the data matrix X based on those dominant singular components, and then adopt PGM to recover an underlying graph via

Original: MLE.
Revised: MLE.

Original: To this end, we define a weighted spectral embedding matrix as follows:
Revised: To this end, we define a weighted spectral embedding matrix as follows:

Original: Definition 3.1.
Revised: Definition 3.1.

Original: Given the top r smallest eigenvalues λ 1 , λ 2 , ..., λ r and their corresponding eigenvectors v 1 , v 2 , ..., v r of normalized graph Laplacian matrix L norm =
Revised: Given the top r smallest eigenvalues λ 1 , λ 2 , ..., λ r and their corresponding eigenvectors v 1 , v 2 , ..., v r of normalized graph Laplacian matrix L norm =

Original: I − D − 12 AD − 12 , where
Revised: I − D − 12 AD − 12 , where

Original: I and A are the identity matrix and graph adjacency matrix, respectively, and D is a diagonal matrix of node degrees, the weighted spectral embedding matrix is defined as V def =
Revised: I and A are the identity matrix and graph adjacency matrix, respectively, and D is a diagonal matrix of node degrees, the weighted spectral embedding matrix is defined as V def =

Original: (cid:104)(cid:112) | 1 − λ 1 | v 1 , ..., (cid:112) | 1 − λ r | v r (cid:105) , whose i -th row V i, : is the weighted spectral embedding of the corresponding i -th node in the graph.
Revised: (cid:104)(cid:112) | 1 − λ 1 | v 1 , ..., (cid:112) | 1 − λ r | v r (cid:105) , whose i -th row V i, : is the weighted spectral embedding of the corresponding i -th node in the graph.

Original: Proposition 3.2.
Revised: Proposition 3.2.

Original: Given a normalized graph adjacency matrix A norm =
Revised: Given a normalized graph adjacency matrix A norm =

Original: D − 12 AD − 12 and weighted spectral embedding matrix V of an undirected graph, let ˆ A be the rank- r approximation of A norm via TSVD.
Revised: D − 12 AD − 12 and weighted spectral embedding matrix V of an undirected graph, let ˆ A be the rank- r approximation of A norm via TSVD.

Original: If the top r dominant eigenvalues of A norm are non-negative, then we have ˆ A = V V T .
Revised: If the top r dominant eigenvalues of A norm are non-negative, then we have ˆ A = V V T .

Original: Our proof for Proposition 3.2 is available in Appendix A. Proposition 3.2 shows the connection between weighted spectral embedding and the low-rank adjacency matrix ˆ A obtained by TSVD.
Revised: Our proof for Proposition 3.2 is available in Appendix A. Proposition 3.2 shows the connection between weighted spectral embedding and the low-rank adjacency matrix ˆ A obtained by TSVD.

Original: Specifically, the weighted spectral embedding matrix V can be viewed as an eigensubspace matrix consisting of a few dominant singular components of the corresponding adjacency matrix.
Revised: Specifically, the weighted spectral embedding matrix V can be viewed as an eigensubspace matrix consisting of a few dominant singular components of the corresponding adjacency matrix.

Original: Thus, we can use V to recover the underlying clean graph via PGM.
Revised: Thus, we can use V to recover the underlying clean graph via PGM.

Original: However, obtaining V requires the knowledge of the clean graph structure, which seems to create a chicken and egg problem.
Revised: However, obtaining V requires the knowledge of the clean graph structure, which seems to create a chicken and egg problem.

Original: Fortunately, since the dominant singular components are hardly affected by adversarial attacks [7], the weighted spectral embedding is therefore also resistant to adversarial attacks, indicating that the underlying clean graph G clean and its corresponding adversarial graph G adv share almost the same weighted spectral embeddings.
Revised: Fortunately, since the dominant singular components are hardly affected by adversarial attacks [7], the weighted spectral embedding is therefore also resistant to adversarial attacks, indicating that the underlying clean graph G clean and its corresponding adversarial graph G adv share almost the same weighted spectral embeddings.

Original: As a result, we can exploit the weighted spectral embedding matrix V of G adv to represent that of G clean .
Revised: As a result, we can exploit the weighted spectral embedding matrix V of G adv to represent that of G clean .

Original: By replacing the data matrix X with V in Equation 1, we have the following objective function
Revised: By replacing the data matrix X with V in Equation 1, we have the following objective function

Original: [Equation]
Revised: [Equation]

Original: More discussions on Equation 2 are available in Appendix Q. By finding the optimizer Θ ∗ , we can recover the underlying graph that maximizes the likelihood given the observation on the weighted spectral embedding V .
Revised: More discussions on Equation 2 are available in Appendix Q. By finding the optimizer Θ ∗ , we can recover the underlying graph that maximizes the likelihood given the observation on the weighted spectral embedding V .

Original: However, solving Equation 2 requires at least O ( n 2 ) time/space complexity per iteration with the most efficient algorithms, which thus cannot scale to large graphs [13, 26, 27].
Revised: However, solving Equation 2 requires at least O ( n 2 ) time/space complexity per iteration with the most efficient algorithms, which thus cannot scale to large graphs [13, 26, 27].

Original: As Θ is constrained to be a Laplacian-like matrix, finding the optimizer Θ ∗ in Equation 2 is equivalent to searching for critical edges from a complete graph, which would involve all possible (i.e., O ( n 2 ) ) edges.
Revised: As Θ is constrained to be a Laplacian-like matrix, finding the optimizer Θ ∗ in Equation 2 is equivalent to searching for critical edges from a complete graph, which would involve all possible (i.e., O ( n 2 ) ) edges.

Original: Here we say an edge is critical (noncritical) if including it to the graph significantly increases (decreases) F in Equation 2.
Revised: Here we say an edge is critical (noncritical) if including it to the graph significantly increases (decreases) F in Equation 2.

Original: Hence we can recover the underlying graph by pruning noncritical edges from the complete graph.
Revised: Hence we can recover the underlying graph by pruning noncritical edges from the complete graph.

Original: However, storing a complete graph is still expensive.
Revised: However, storing a complete graph is still expensive.

Original: To have a near-linear algorithm for clean graph recovery, instead of searching in the complete graph, we limit our search within an initial base graph G base that is much sparser but containing sufficient information for identifying the candidate edges critical to recover the clean graph.
Revised: To have a near-linear algorithm for clean graph recovery, instead of searching in the complete graph, we limit our search within an initial base graph G base that is much sparser but containing sufficient information for identifying the candidate edges critical to recover the clean graph.

Original: Subsequently, the final graphical model (graph Laplacian) can be obtained by further pruning noncritical edges from G base .
Revised: Subsequently, the final graphical model (graph Laplacian) can be obtained by further pruning noncritical edges from G base .

Original: 3.2 Base Graph Construction
Revised: 3.2 Base Graph Construction

Original: During the first phase of GARNET (shown in Figure 2), our goal is to build a base graph G base , which greatly reduces the search space by not constructing a complete graph while preserving the critical candidate edges that are key to clean graph recovery.
Revised: During the first phase of GARNET (shown in Figure 2), our goal is to build a base graph G base , which greatly reduces the search space by not constructing a complete graph while preserving the critical candidate edges that are key to clean graph recovery.

Original: To this end, we give the following theorem:
Revised: To this end, we give the following theorem:

Original: Theorem 3.3.
Revised: Theorem 3.3.

Original: Given a graph G =
Revised: Given a graph G =

Original: ( V , E ) and its normalized Laplacian matrix L G , let V i denote the weighted spectral embedding of node i by using top r eigenpairs of L G .
Revised: ( V , E ) and its normalized Laplacian matrix L G , let V i denote the weighted spectral embedding of node i by using top r eigenpairs of L G .

Original: Suppose a relatively small r is picked such that λ r ≤ 1 , where λ r is the r -th smallest eigenvalue of L G , then we have (cid:80) ( i,j
Revised: Suppose a relatively small r is picked such that λ r ≤ 1 , where λ r is the r -th smallest eigenvalue of L G , then we have (cid:80) ( i,j

Original: ) ∈E ∥
Revised: ) ∈E ∥

Original: V i − V j ∥
Revised: V i − V j ∥

Original: 22 ≤
Revised: 22 ≤

Original: 25 r .
Revised: 25 r .

Original: Our proof for Theorem 3.3 is available in Appendix B. Note that r is a small constant, which is independent of the graph size.
Revised: Our proof for Theorem 3.3 is available in Appendix B. Note that r is a small constant, which is independent of the graph size.

Original: Thus, Theorem 3.3 indicates that, if an edge connects nodes i and j in the clean graph, then the Euclidean distance between the weighted spectral embeddings of these two nodes will be small, which motivates us to build a k-nearest neighbor (kNN) graph as G base to incorporate those clean edges.
Revised: Thus, Theorem 3.3 indicates that, if an edge connects nodes i and j in the clean graph, then the Euclidean distance between the weighted spectral embeddings of these two nodes will be small, which motivates us to build a k-nearest neighbor (kNN) graph as G base to incorporate those clean edges.

Original: Concretely, we first obtain the weighted spectral embedding matrix V of the input adversarial graph
Revised: Concretely, we first obtain the weighted spectral embedding matrix V of the input adversarial graph

Original: G adv to represent that of the underlying clean graph G clean , as V consists of dominant singular components that are shared by G adv and G clean [7].
Revised: G adv to represent that of the underlying clean graph G clean , as V consists of dominant singular components that are shared by G adv and G clean [7].

Original: We then leverage V to construct a kNN graph, where each node is connected to its k most similar nodes based on the Euclidean distance between their spectral embeddings.
Revised: We then leverage V to construct a kNN graph, where each node is connected to its k most similar nodes based on the Euclidean distance between their spectral embeddings.

Original: Note that V can be further concatenated with node feature matrix for constructing the kNN graph.
Revised: Note that V can be further concatenated with node feature matrix for constructing the kNN graph.

Original: A thorough discussion on incorporating node feature information is available in Appendix L.
Revised: A thorough discussion on incorporating node feature information is available in Appendix L.

Original: In this work, we exploit an approximate kNN algorithm for constructing the graph, which has O ( |V| log |V| ) complexity and thus can scale to very large graphs [28].
Revised: In this work, we exploit an approximate kNN algorithm for constructing the graph, which has O ( |V| log |V| ) complexity and thus can scale to very large graphs [28].

Original: By choosing a proper k (e.g., k = 50 ), G base is likely to cover edges in the underlying clean graph.
Revised: By choosing a proper k (e.g., k = 50 ), G base is likely to cover edges in the underlying clean graph.

Original: Thus, G base can serve as a reasonable search space for identifying critical edges in the next step.
Revised: Thus, G base can serve as a reasonable search space for identifying critical edges in the next step.

Original: 3.3 Graph Refinement via Edge Pruning
Revised: 3.3 Graph Refinement via Edge Pruning

Original: For the second phase of GARNET shown in Figure 2, we refine G base by aggressively pruning noncritical edges from G base , such that the refined graph only preserves the most important edges that contribute most to the log-likelihood F in Equation 2.
Revised: For the second phase of GARNET shown in Figure 2, we refine G base by aggressively pruning noncritical edges from G base , such that the refined graph only preserves the most important edges that contribute most to the log-likelihood F in Equation 2.

Original: To identify critical (noncritical) edges that can most effectively increase (decrease) F , we exploit the update of Θ based on gradient ascent: Θ ← Θ + η ∂F∂ Θ , where η is the step size.
Revised: To identify critical (noncritical) edges that can most effectively increase (decrease) F , we exploit the update of Θ based on gradient ascent: Θ ← Θ + η ∂F∂ Θ , where η is the step size.

Original: As mentioned in
Revised: As mentioned in

Original: Section 2.1, Θ is constrained to be L + Iσ 2 , which means the off-diagonal elements in Θ correspond to negative of edge weights in the underlying graph, i.e., Θ i,j = − w i,j .
Revised: Section 2.1, Θ is constrained to be L + Iσ 2 , which means the off-diagonal elements in Θ correspond to negative of edge weights in the underlying graph, i.e., Θ i,j = − w i,j .

Original: Thus, the update of Θ i,j during gradient ascent can be viewed as:
Revised: Thus, the update of Θ i,j during gradient ascent can be viewed as:

Original: [Equation]
Revised: [Equation]

Original: Equation 3 means that, if ∂F∂w i,j is large and positive, Θ i,j will become more negative, which corresponds to increasing the edge weight in the underlying graph.
Revised: Equation 3 means that, if ∂F∂w i,j is large and positive, Θ i,j will become more negative, which corresponds to increasing the edge weight in the underlying graph.

Original: Similarly, if ∂F∂w i,j is small and negative, Θ i,j will be less negative, corresponding to decreasing the edge weight.
Revised: Similarly, if ∂F∂w i,j is small and negative, Θ i,j will be less negative, corresponding to decreasing the edge weight.

Original: In other words, the edge weight w i,j with a large (small) ∂F∂w
Revised: In other words, the edge weight w i,j with a large (small) ∂F∂w

Original: i,j should be increased (decreased) to maximize the log-likelihood F , meaning the corresponding edge is critical (noncritical).
Revised: i,j should be increased (decreased) to maximize the log-likelihood F , meaning the corresponding edge is critical (noncritical).

Original: Thus, we can identify the critical edges once we know
Revised: Thus, we can identify the critical edges once we know

Original: ∂F∂w i,j .
Revised: ∂F∂w i,j .

Original: By setting α = 0 in Equation 2 (as GARNET naturally produces a sparse graph) and taking the partial derivative with respect to an edge weight w i,j , we have:
Revised: By setting α = 0 in Equation 2 (as GARNET naturally produces a sparse graph) and taking the partial derivative with respect to an edge weight w i,j , we have:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: where λ k , ∀ k
Revised: where λ k , ∀ k

Original: = 1 , 2 , ..., n are the Laplacian eigenvalues of G base (the initial graph for edge pruning), e i,j = e i − e j , and e i denotes the vector with all zero entries except for the i -th entry being 1 .
Revised: = 1 , 2 , ..., n are the Laplacian eigenvalues of G base (the initial graph for edge pruning), e i,j = e i − e j , and e i denotes the vector with all zero entries except for the i -th entry being 1 .

Original: Theorem 3.4 (Feng [17]) .
Revised: Theorem 3.4 (Feng [17]) .

Original: Let λ k and u k be the k -th eigenvalue and the corresponding eigenvector of the Laplacian matrix, respectively.
Revised: Let λ k and u k be the k -th eigenvalue and the corresponding eigenvector of the Laplacian matrix, respectively.

Original: The spectral perturbation δλ k due to the increase of an edge weight w i,j can be estimated by δλ k = δw i,j ( u Tk e i,j ) 2 .
Revised: The spectral perturbation δλ k due to the increase of an edge weight w i,j can be estimated by δλ k = δw i,j ( u Tk e i,j ) 2 .

Original: The proof for Theorem 3.4 is available in Feng [17].
Revised: The proof for Theorem 3.4 is available in Feng [17].

Original: According to Theorem 3.4 and Equation 4, we can estimate ∂F∂w i,j ≈ ∥
Revised: According to Theorem 3.4 and Equation 4, we can estimate ∂F∂w i,j ≈ ∥

Original: U T e i,j ∥ 22 − 1 r ∥ V T e i,j ∥ 22 , where U =
Revised: U T e i,j ∥ 22 − 1 r ∥ V T e i,j ∥ 22 , where U =

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1 /σ 2 ] , λ i is the i -th smallest Laplacian eigenvalue of G base , and u i is the corresponding eigenvector.
Revised: 1 /σ 2 ] , λ i is the i -th smallest Laplacian eigenvalue of G base , and u i is the corresponding eigenvector.

Original: Consequently, an edge ( i, j ) is critical if ∥ U T e i,j ∥ 

Revised: Consequently, an edge ( i, j ) is critical if ∥ U T e i,j ∥

Original: ≫ 1 r ∥ V T e i,j ∥ 22 .
Revised: ≫ 1 r ∥ V T e i,j ∥ 22 .

Original: As V and U are the spectral embeddings on the input adversarial graph and the base graph, respectively, we define the spectral embedding distortion s i,j = ∥
Revised: As V and U are the spectral embeddings on the input adversarial graph and the base graph, respectively, we define the spectral embedding distortion s i,j = ∥

Original: U T e i,j ∥ 22 ∥ V T e i,j ∥ 22 to measure the edge importance.
Revised: U T e i,j ∥ 22 ∥ V T e i,j ∥ 22 to measure the edge importance.

Original: Consequently, we prune edges in the base graph
Revised: Consequently, we prune edges in the base graph

Original: G base that have small spectral embedding distortion, i.e., s i,j < γ , where γ is a hyperparameter to control the sparsity of the refined graph.
Revised: G base that have small spectral embedding distortion, i.e., s i,j < γ , where γ is a hyperparameter to control the sparsity of the refined graph.

Original: We further provide a strategy to simplify the distortion metric for edge pruning in Appendix S. Hence, the refined base graph G ′ base largely recovers the underlying clean graph structure from the input adversarial graph.
Revised: We further provide a strategy to simplify the distortion metric for edge pruning in Appendix S. Hence, the refined base graph G ′ base largely recovers the underlying clean graph structure from the input adversarial graph.

Original: Since G ′ base is constructed by only leveraging the top few dominant singular components of G adv , it ignores the high-rank adversarial components and thus robust to adversarial attacks.
Revised: Since G ′ base is constructed by only leveraging the top few dominant singular components of G adv , it ignores the high-rank adversarial components and thus robust to adversarial attacks.

Original: As a result, we can train a given GNN model on
Revised: As a result, we can train a given GNN model on

Original: G ′ base to improve its robustness, which is the last phase of GARNET.
Revised: G ′ base to improve its robustness, which is the last phase of GARNET.

Original: 3.4 Complexity of GARNET
Revised: 3.4 Complexity of GARNET

Original: The first phase of GARNET requires O ( r |E| ) time for computing top r Laplacian eigenpairs [25], and O ( |V| log |V| ) time for kNN graph construction [28].
Revised: The first phase of GARNET requires O ( r |E| ) time for computing top r Laplacian eigenpairs [25], and O ( |V| log |V| ) time for kNN graph construction [28].

Original: The second phase involves O ( rk |V| ) time for computing spectral embeddings and edge pruning on the kNN graph.
Revised: The second phase involves O ( rk |V| ) time for computing spectral embeddings and edge pruning on the kNN graph.

Original: Thus, the overall time complexity for graph purification is O ( r ( |E| + k |V| ) + |V| log |V| ) , where |V| ( |E| ) denotes the number of nodes (edges) in the adversarial graph, and k is the averaged node degree in the kNN graph.
Revised: Thus, the overall time complexity for graph purification is O ( r ( |E| + k |V| ) + |V| log |V| ) , where |V| ( |E| ) denotes the number of nodes (edges) in the adversarial graph, and k is the averaged node degree in the kNN graph.

Original: Our systematic approach of choosing r and the space complexity analysis are in Appendix F.
Revised: Our systematic approach of choosing r and the space complexity analysis are in Appendix F.

Original:  Experiments
Revised:  Experiments

Original: We have conducted comparative evaluation of GARNET against state-of-the-art defense GNN models under targeted attack (Nettack) [5] and non-targeted attack (Metattack) [6] on both homophilic and heterophilic datasets.
Revised: We have conducted comparative evaluation of GARNET against state-of-the-art defense GNN models under targeted attack (Nettack) [5] and non-targeted attack (Metattack) [6] on both homophilic and heterophilic datasets.

Original: Besides, we also evaluate GARNET robustness against adaptive attacks.
Revised: Besides, we also evaluate GARNET robustness against adaptive attacks.

Original: In addition, we further show the scalability of GARNET by comparing its run time with prior defense methods and evaluating GARNET on ogbn-products, which consists of more than 2 million nodes [11].
Revised: In addition, we further show the scalability of GARNET by comparing its run time with prior defense methods and evaluating GARNET on ogbn-products, which consists of more than 2 million nodes [11].

Original: Finally, we conduct ablation studies to understand the effectiveness of GARNET kernels.
Revised: Finally, we conduct ablation studies to understand the effectiveness of GARNET kernels.

Original: Experimental Setup.
Revised: Experimental Setup.

Original: Table 1 shows the statistics of the datasets used in our experiments.
Revised: Table 1 shows the statistics of the datasets used in our experiments.

Original: We follow Zhu et al.
Revised: We follow Zhu et al.

Original: [10] to compute the homophily score per dataset (lower score means more heterophilic).
Revised: [10] to compute the homophily score per dataset (lower score means more heterophilic).

Original: More details of datasets are available in Appendix C. We choose as baselines two state-of-the-art defense methods based on graph purification: TSVD [7] and Pro-GNN [8].
Revised: More details of datasets are available in Appendix C. We choose as baselines two state-of-the-art defense methods based on graph purification: TSVD [7] and Pro-GNN [8].

Original: Besides, we evaluate training based defense methods GCN-LFR [22] and GNNGuard [29] on homophilic and heterophilic graphs, respectively.
Revised: Besides, we evaluate training based defense methods GCN-LFR [22] and GNNGuard [29] on homophilic and heterophilic graphs, respectively.

Original: Moreover, we use GCN [30] and GPRGNN [31] as the backbone GNN models for defense on homophilic datasets (i.e., Cora and Pubmed).
Revised: Moreover, we use GCN [30] and GPRGNN [31] as the backbone GNN models for defense on homophilic datasets (i.e., Cora and Pubmed).

Original: As GCN performs poorly on heterophilic datasets [10, 32], we choose GPRGNN as the backbone model on Chameleon and Squirrel datasets.
Revised: As GCN performs poorly on heterophilic datasets [10, 32], we choose GPRGNN as the backbone model on Chameleon and Squirrel datasets.

Original: Due to the space limit, we provide defense results with H2GCN [10] as the backbone model in
Revised: Due to the space limit, we provide defense results with H2GCN [10] as the backbone model in

Original: Appendix J. For all baselines, we tune their hyperparameters against adversarial attacks with a
Revised: Appendix J. For all baselines, we tune their hyperparameters against adversarial attacks with a

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: small perturbation, and keep the same hyperparameters for larger adversarial perturbations.
Revised: small perturbation, and keep the same hyperparameters for larger adversarial perturbations.

Original: Detailed hyperparameter settings of baselines and GARNET are available in Appendix D. Our hardware information is provided in Appendix E.
Revised: Detailed hyperparameter settings of baselines and GARNET are available in Appendix D. Our hardware information is provided in Appendix E.

Original: 4.1 Robustness of GARNET
Revised: 4.1 Robustness of GARNET

Original: Defense on homophilic graphs .
Revised: Defense on homophilic graphs .

Original: We first evaluate the model robustness on homophilic graphs against the targeted attack (Nettack) and the non-targeted attack (Metattack).
Revised: We first evaluate the model robustness on homophilic graphs against the targeted attack (Nettack) and the non-targeted attack (Metattack).

Original: Specifically, Nettack aims to fool a GNN model to misclassify some target nodes with a few structure (edge) perturbations.
Revised: Specifically, Nettack aims to fool a GNN model to misclassify some target nodes with a few structure (edge) perturbations.

Original: The goal of Metattack is to drop the overall accuracy of the whole test set with a given perturbation ratio budget (i.e., the number of adversarial edges over the number of total edges).
Revised: The goal of Metattack is to drop the overall accuracy of the whole test set with a given perturbation ratio budget (i.e., the number of adversarial edges over the number of total edges).

Original: Due to the space limit, we only show defense results under Nettack and Metattack with 5 perturbed edges per target node and 20% perturbation ratio, respectively.
Revised: Due to the space limit, we only show defense results under Nettack and Metattack with 5 perturbed edges per target node and 20% perturbation ratio, respectively.

Original: Results with other perturbation budgets are in Appendix I.
Revised: Results with other perturbation budgets are in Appendix I.

Original: Table 2 reports the average accuracy over 10 runs on Cora and Pubmed.
Revised: Table 2 reports the average accuracy over 10 runs on Cora and Pubmed.

Original: It shows that GARNET, with either a backbone GNN model (GCN or GPRGNN), outperforms defense baselines in terms of both clean and adversarial accuracy in most cases.
Revised: It shows that GARNET, with either a backbone GNN model (GCN or GPRGNN), outperforms defense baselines in terms of both clean and adversarial accuracy in most cases.

Original: We attribute the large accuracy improvement to
Revised: We attribute the large accuracy improvement to

Original: GARNET’s strengths in recovering key structures of the clean graph while ignoring the high-rank adversarial components during graph purification.
Revised: GARNET’s strengths in recovering key structures of the clean graph while ignoring the high-rank adversarial components during graph purification.

Original: Moreover, as both TSVD and ProGNN involve dense matrices during GNN training, they run out of GPU memory even on Pubmed, a graph with only 20k nodes.
Revised: Moreover, as both TSVD and ProGNN involve dense matrices during GNN training, they run out of GPU memory even on Pubmed, a graph with only 20k nodes.

Original: In contrast, GARNET is not only robust to adversarial attacks, but also scalable to large graphs, as empirically shown in Section 4.2.
Revised: In contrast, GARNET is not only robust to adversarial attacks, but also scalable to large graphs, as empirically shown in Section 4.2.

Original: Defense on heterophilic graphs .
Revised: Defense on heterophilic graphs .

Original: We report the averaged accuracy over 10 runs on heterophilic graphs in Table 3, which shows that all defense baselines fail to defend GPRGNN on heterophilic graphs and even degrade the accuracy of the vanilla GPRGNN by a large margin.
Revised: We report the averaged accuracy over 10 runs on heterophilic graphs in Table 3, which shows that all defense baselines fail to defend GPRGNN on heterophilic graphs and even degrade the accuracy of the vanilla GPRGNN by a large margin.

Original: The reason why
Revised: The reason why

Original: ProGNN performs poorly is that it follows the graph homophily assumption for improving GNN robustness, which contradicts the property of heterophilic graphs.
Revised: ProGNN performs poorly is that it follows the graph homophily assumption for improving GNN robustness, which contradicts the property of heterophilic graphs.

Original: For the TSVD-based defense method, the low-rank graph generated by TSVD contains negative edge weights, which degrade the performance of GPRGNN for adapting its graph filter on heterophilic graphs [31].
Revised: For the TSVD-based defense method, the low-rank graph generated by TSVD contains negative edge weights, which degrade the performance of GPRGNN for adapting its graph filter on heterophilic graphs [31].

Original: Although Zhang and Zitnik [29] have shown GNNGuard can improve model robustness on synthetic heterophillic graphs, our results indicate that it fails to defend GNN models on realistic heterohilic graphs.
Revised: Although Zhang and Zitnik [29] have shown GNNGuard can improve model robustness on synthetic heterophillic graphs, our results indicate that it fails to defend GNN models on realistic heterohilic graphs.

Original: We attribute it to that the quality of graphlet degree vectors used in GNNGuard is degraded by structural perturbations induced via adversarial attacks.
Revised: We attribute it to that the quality of graphlet degree vectors used in GNNGuard is degraded by structural perturbations induced via adversarial attacks.

Original: In contrast, GARNET largely recovers the clean graph structure based on Theorem 3.3 without the assumption on whether adjacent nodes have similar
Revised: In contrast, GARNET largely recovers the clean graph structure based on Theorem 3.3 without the assumption on whether adjacent nodes have similar

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 1616.1x attributes.
Revised: 1616.1x attributes.

Original: In other words, GARNET will produce a heterophilic graph if the underlying clean graph is heterophilic, which is further confirmed in Appendix O. Consequently, GARNET improves accuracy over defense baselines by up to 10 .
Revised: In other words, GARNET will produce a heterophilic graph if the underlying clean graph is heterophilic, which is further confirmed in Appendix O. Consequently, GARNET improves accuracy over defense baselines by up to 10 .

Original: 23% (i.e., 43 .
Revised: 23% (i.e., 43 .

Original: 64% − 33 .
Revised: 64% − 33 .

Original: 41% on Squirrel under Nettack) on heterophilic graphs.
Revised: 41% on Squirrel under Nettack) on heterophilic graphs.

Original: Defense against adaptive attacks .
Revised: Defense against adaptive attacks .

Original: As GARNET is non-differentiable during kNN graph construction, it is difficult to optimize a specific loss function for adaptive attack.
Revised: As GARNET is non-differentiable during kNN graph construction, it is difficult to optimize a specific loss function for adaptive attack.

Original: Instead, we adopt an attack called LowBlow from [7], which deliberately perturbs low-rank singular components in the graph spectrum, yet violates the unnoticeable condition (i.e., preserving node degree distribution after attacking).
Revised: Instead, we adopt an attack called LowBlow from [7] (based on Metattack), which deliberately perturbs low-rank singular components in the graph spectrum, yet violates the unnoticeable condition (i.e., preserving node degree distribution after attacking).

Original: Since LowBlow has cubic complexity for computing the full set of adjacency eigenpairs, we only show results on the small graph Cora in Table 4, which indicates GARNET still achieves the highest adversarial accuracy under LowBlow, while all low-rank defense baselines perform even worse than vanilla GPRGNN model.
Revised: Since LowBlow has cubic complexity for computing the full set of adjacency eigenpairs, we only show results on the small graph Cora in Table 4, which indicates GARNET still achieves the highest adversarial accuracy under LowBlow, while all low-rank defense baselines perform even worse than vanilla GPRGNN model.

Original: The reason lies in that the kNN graph (with a relatively large k ) in GARNET is less vulnerable to the perturbations of weighted spectral embeddings (i
Revised: The reason lies in that the kNN graph (with a relatively large k ) in GARNET is less vulnerable to the perturbations of weighted spectral embeddings (i

Original: [Table]
Revised: [Table]

Original: e., low-rank components of the clean graph) [33], compared to prior low-rank defense methods.
Revised: e., low-rank singular components) [33], compared to prior low-rank defense methods.

Original: 4.2 Scalability of GARNET
Revised: 4.2 Scalability of GARNET

Original: [Table]
Revised: [Table]

Original: To demonstrate the scalability of GARNET
Revised: To demonstrate the scalability of GARNET

Original: [Table]
Revised: [Table]

Original: we first compare the run time of GARNET with prior low-rank defense methods with GPRGNN as the backbone GNN model.
Revised: we first compare the run time of GARNET with prior low-rank defense methods with GPRGNN as the backbone GNN model.

Original: As shown in Figure 3, the
Revised: As shown in Figure [Table] the

Original: TSVD defense method is slower than GARNET since it produces a dense adjacency matrix that slows down the GNN training.
Revised: TSVD defense method is slower than GARNET since it produces a dense adjacency matrix that slows down the GNN training.

Original: Moreover, ProGNN is extremely slow as it jointly learns the low-rank graph structure and the robust GNN model, which requires performing TSVD for every epoch.
Revised: Moreover, ProGNN is extremely slow as it jointly learns the low-rank graph structure and the robust GNN model, which requires performing TSVD for every epoch.

Original: In contrast, GARNET can efficiently produce a sparse graph for downstream GNN training, leading to end-to-end runtime speedup over prior methods by up to 14 .
Revised: In contrast, GARNET can efficiently produce a sparse graph for downstream GNN training, leading to end-to-end runtime speedup over prior methods by up to 14 .

Original: 7 × .
Revised: 7 × .

Original: In addition, we further evaluate the robustness of GARNET on two large datasets: ogbn-arxiv and ogbn-products, under powerful and scalable attacks proposed by [34].
Revised: In addition, we further evaluate the robustness of GARNET on two large datasets: ogbn-arxiv and ogbn-products, under powerful and scalable attacks proposed by [34].

Original: As we run out of GPU memory when performing the PR-BCD attack, we choose the more scalable version GR-BCD that has less memory usage.
Revised: As we run out of GPU memory when performing the PR-BCD attack, we choose the more scalable version GR-BCD that has less memory usage.

Original: We use GCN as the backbone model since it outperforms GPRGNN on large graphs.
Revised: We use GCN as the backbone model since it outperforms GPRGNN on large graphs.

Original: As TSVD and ProGNN run out of memory on these two datasets, we choose GNNGuard,
Revised: As TSVD and ProGNN run out of memory on these two datasets, we choose GNNGuard,

Original: GCNJaccard [23], and Soft Median GDC [34] as baselines.
Revised: GCNJaccard [23], and Soft Median GDC [34] as baselines.

Original: Table 5 shows GARNET achieves comparable clean accuracy compared to GCN, and drastically improves the adversarial accuracy over defense baselines by up to 16 .
Revised: Table 5 shows GARNET achieves comparable clean accuracy compared to GCN, and drastically improves the adversarial accuracy over defense baselines by up to 16 .

Original: Moreover, we also evaluate the run time of GARNET on the large graphs.
Revised: Moreover, we also evaluate the run time of GARNET on the large graphs.

Original: Concretely, the end-to-end run time of GARNET is 40 mins and 4 hours on ogbn-arxiv and ogbn-products, respectively, which is 3 × faster than the most competitive baseline GNNGuard that takes more than 2 hours on ogbn-arxiv and 11 hours on ogbn-products.
Revised: Concretely, the end-to-end run time of GARNET is 40 mins and 4 hours on ogbn-arxiv and ogbn-products, respectively, which is 3 × faster than the most competitive baseline GNNGuard that takes more than 2 hours on ogbn-arxiv and 11 hours on ogbn-products.

Original: We provide potential ways to further accelerate GARNET in Appendix N.
Revised: We provide potential ways to further accelerate GARNET in Appendix N.

Original: 4.3 Ablation Analysis of GARNET
Revised: 4.3 Ablation Analysis of GARNET

Original: [Figure]
Revised: [Figure]

Original: Figure 4 shows the comparison of GARNET results with and without graph refinement.
Revised: Figure 4 shows the comparison of GARNET results with and without graph refinement.

Original: When only constructing the base graph, GARNET achieves better adversarial accuracy than the vanilla GNN model, which confirms our Theorem 3.3 that the base graph construction can successfully recover clean graph edges.
Revised: When only constructing the base graph, GARNET achieves better adversarial accuracy than the vanilla GNN model, which confirms our Theorem 3.3 that the base graph construction can successfully recover clean graph edges.

Original: The graph refinement step further improves GARNET accuracy ( ∼ 2% increase) since some noncritical or even harmful edges are removed based on PGM.
Revised: The graph refinement step further improves GARNET accuracy ( ∼ 2% increase) since some noncritical or even harmful edges are removed based on PGM.

Original: Due to the space limitation, the ablation studies of GARNET on the kNN graph and edge pruning are available in Appendix G.
Revised: Due to the space limitation, the ablation studies of GARNET on the kNN graph and edge pruning are available in Appendix G.

Original: 4.4 Visualization
Revised: 4.4 Visualization

Original: We visualize the local structure (within 2-hop neighbors) of a target node (randomly picked) on Cora in Figure 5.
Revised: We visualize the local structure (within 2-hop neighbors) of a target node (randomly picked) on Cora in Figure 5.

Original: By comparing Figures 5(b) and 5(c), it is clear that GARNET effectively removes most of the adversarial edges induced by Nettack that connect nodes with different labels [8].
Revised: By comparing Figures 5(b) and 5(c), it is clear that GARNET effectively removes most of the adversarial edges induced by Nettack that connect nodes with different labels [8].

Original: As a result, it is trivial for the backbone GNN model to correctly predict the target node since the surrounding nodes share the same label as the target node in GARNET graph.
Revised: As a result, it is trivial for the backbone GNN model to correctly predict the target node since the surrounding nodes share the same label as the target node in GARNET graph.

Original: This explains why GARNET substantially improves the adversarial accuracy of GNN models.
Revised: This explains why GARNET substantially improves the adversarial accuracy of GNN models.

Original: More visualizations are available in
Revised: More visualizations are available in

Original: Appendix M.
Revised: Appendix M.

Original: [Figure]
Revised: [Figure]

Original:  Conclusions
Revised:  Conclusions

Original: This work introduces GARNET, a spectral approach to robust and scalable graph neural networks by combining spectral embedding and the probabilistic graphical model.
Revised: This work introduces GARNET, a spectral approach to robust and scalable graph neural networks by combining spectral embedding and the probabilistic graphical model.

Original: GARNET first uses weighted spectral embedding to construct a base graph, which is then refined by pruning uncritical edges based on the graphical model.
Revised: GARNET first uses weighted spectral embedding to construct a base graph, which is then refined by pruning uncritical edges based on the graphical model.

Original: Results show that GARNET not only outperforms state-of-the-art defense models, but also scales to large graphs with millions of nodes.
Revised: Results show that GARNET not only outperforms state-of-the-art defense models, but also scales to large graphs with millions of nodes.

Original: An interesting direction for future work is to incorporate high-order structural information (e.g., motifs) to further boost model robustness.
Revised: An interesting direction for future work is to incorporate high-order structural information (e.g., motifs) to further boost model robustness.

Original: Acknowledgements
Revised: Acknowledgements

Original: This work is supported in part by NSF grants #2021309, #2205572, #2212370, and #2212371, and a
Revised: This work is supported in part by NSF grants #2021309, #2205572, #2212370, and #2212371, and a

Original: Facebook Research Award.
Revised: Facebook Research Award.

Original: A Proof for Proposition 3. 
Revised: A Proof for Proposition 3.

Original: Proof.
Revised: Proof.

Original: As the graph is undirected, we can perform eigendecomposition on both A norm and L norm to obtain their real eigenvalues and the corresponding eigenvectors.
Revised: As the graph is undirected, we can perform eigendecomposition on both A norm and L norm to obtain their real eigenvalues and the corresponding eigenvectors.

Original: Let λ
Revised: Let λ

Original: i , ˆ λ i , and σ i , i = 1 , 2 , ..., r denote the r smallest eigenvalues of L norm , r largest eigenvalues of A norm , and r largest singular values of A norm , respectively.
Revised: i , ˆ λ i , and σ i , i = 1 , 2 , ..., r denote the r smallest eigenvalues of L norm , r largest eigenvalues of A norm , and r largest singular values of A norm , respectively.

Original: Since A norm =
Revised: Since A norm =

Original: I − L norm , A norm and L norm share the same set of eigenvectors while their eigenvalues satisfy:
Revised: I − L norm , A norm and L norm share the same set of eigenvectors while their eigenvalues satisfy:

Original: ˆ λ i = 1 − λ i , i = 1 , 2 , ..., r .
Revised: ˆ λ i = 1 − λ i , i = 1 , 2 , ..., r .

Original: Moreover, since we assume that the r largest magnitude eigenvalues of A norm are non-negative, we have σ i =
Revised: Moreover, since we assume that the r largest magnitude eigenvalues of A norm are non-negative, we have σ i =

Original: (cid:12)(cid:12)(cid:12) ˆ
Revised: (cid:12)(cid:12)(cid:12) ˆ

Original: λ i (cid:12)(cid:12)(cid:12) =
Revised: λ i (cid:12)(cid:12)(cid:12) =

Original: ˆ λ i , i = 1 , 2 , ..., r .
Revised: ˆ λ i , i = 1 , 2 , ..., r .

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: B Proof for Theorem 3. 
Revised: B Proof for Theorem 3.

Original: Proof.
Revised: Proof.

Original: Since the weighted embedding matrix V is defined as V def = (cid:104)(cid:112) | 1 − λ 1 | v
Revised: Since the weighted embedding matrix V is defined as V def = (cid:104)(cid:112) | 1 − λ 1 | v

Original: [Equation]
Revised: [Equation]

Original: , ..., (cid:112) |
Revised: , ..., (cid:112) |

Original: [Equation]
Revised: [Equation]

Original: λ r | v r (cid:105) , where λ 1 , λ 2 , ..., λ r and v 1 , v 2 , ..., v r are the top r smallest eigenvalues and the corresponding eigenvectors of normalized graph Laplacian matrix L norm =
Revised: λ r | v r (cid:105) , where λ 1 , λ 2 , ..., λ r and v 1 , v 2 , ..., v r are the top r smallest eigenvalues and the corresponding eigenvectors of normalized graph Laplacian matrix L norm =

Original: I − D − 12 AD − 12 , we have:
Revised: I − D − 12 AD − 12 , we have:

Original: [Equation]
Revised: [Equation]

Original: The fourth equation above is based on Courant-Fischer Theorem [35] with the assumption that λ r ≤and the Laplacian eigenvectors are normalized (i.e., ∥ v k ∥ 2 = 1 , ∀ k = 1 , ..., r ).
Revised: The fourth equation above is based on Courant-Fischer Theorem [35] with the assumption that λ r ≤and the Laplacian eigenvectors are normalized (i.e., ∥ v k ∥ 2 = 1 , ∀ k = 1 , ..., r ).

Original: The inequality is derived by arithmetic mean-geometric mean (AM-GM) inequality.
Revised: The inequality is derived by arithmetic mean-geometric mean (AM-GM) inequality.

Original: C Dataset Details
Revised: C Dataset Details

Original: As in Jin et al.
Revised: As in Jin et al.

Original: [8], we extract the largest connected components of the original Cora and Pubmed datasets [36] for the adversarial evaluation, with the same train/validation/test split.
Revised: [8], we extract the largest connected components of the original Cora and Pubmed datasets [36] for the adversarial evaluation, with the same train/validation/test split.

Original: For Chameleon and Squirrel [37], we keep the same split setting as Chien et al.
Revised: For Chameleon and Squirrel [37], we keep the same split setting as Chien et al.

Original: Finally, we follow the split setting of Open Graph Benchmark (OGB) [11] on ogbn-arxiv and ogbn-products.
Revised: Finally, we follow the split setting of Open Graph Benchmark (OGB) [11] on ogbn-arxiv and ogbn-products.

Original: Note that all data used in our experiments do not contain personally identifiable information or offensive content.
Revised: Note that all data used in our experiments do not contain personally identifiable information or offensive content.

Original: In addition, we follow Jin et al.
Revised: In addition, we follow Jin et al.

Original: [8] for the selection of target nodes on Cora and Pubmed under
Revised: [8] for the selection of target nodes on Cora and Pubmed under

Original: Nettack.
Revised: Nettack.

Original: For the Chameleon and Squirrel datasets under Nettack, we choose target nodes that have degrees within the range of [20 , 50] and [20 , 140] , respectively.
Revised: For the Chameleon and Squirrel datasets under Nettack, we choose target nodes that have degrees within the range of [20 , 50] and [20 , 140] , respectively.

Original: In regard to non-targeted attacks (i.e.,
Revised: In regard to non-targeted attacks (i.e.,

Original: Metattack), we choose nodes in the test set as target nodes for all datasets.
Revised: Metattack), we choose nodes in the test set as target nodes for all datasets.

Original: We implement all the adversarial attacks based on the DeepRobust library [38].
Revised: We implement all the adversarial attacks based on the DeepRobust library [38].

Original: D Hyperparameters Settings D.1 Backbone GNN Models
Revised: D Hyperparameters Settings D.1 Backbone GNN Models

Original: GCN.
Revised: GCN.

Original: We choose the GCN hyperparameters based on the DeepRobust library [38].
Revised: We choose the GCN hyperparameters based on the DeepRobust library [38].

Original: GPRGNN.
Revised: GPRGNN.

Original: We follow the hyperparameter settings provided at github.com/jianhao2016/GPRGNN with slightly different dropout rates (chosen from 0 . 3 , 0 . 5 , 0 . 7 ) and learning rates (chosen from 0 .
Revised: We follow the hyperparameter settings provided at github.com/jianhao2016/GPRGNN with slightly different dropout rates (chosen from 0 . 3 , 0 . 5 , 0 . 7 ) and learning rates (chosen from 0 .

Original: Specifically, we provide the complete choices of dropout rates and learning rates across all datasets and attack settings below:
Revised: Specifically, we provide the complete choices of dropout rates and learning rates across all datasets and attack settings below:

Original: • Cora-Nettack: dropout of 0 .
Revised: • Cora-Nettack: dropout of 0 .

Original: 5 and learning rate of 0 . 01 .
Revised: 5 and learning rate of 0 . 01 .

Original: • Cora-Metattack: dropout of 0 .
Revised: • Cora-Metattack: dropout of 0 .

Original: 5 and learning rate of 0 . 01 .
Revised: 5 and learning rate of 0 . 01 .

Original: • Pubmed-Nettack: dropout of 0 .
Revised: • Pubmed-Nettack: dropout of 0 .

Original: 5 and learning rate of 0 . 01 .
Revised: 5 and learning rate of 0 . 01 .

Original: • Pubmed-Metattack: dropout of 0 .
Revised: • Pubmed-Metattack: dropout of 0 .

Original: 5 and learning rate of 0 . 01 .
Revised: 5 and learning rate of 0 . 01 .

Original: • Chameleon-Nettack: dropout of 0 .
Revised: • Chameleon-Nettack: dropout of 0 .

Original: 5 and learning rate of 0 .
Revised: 5 and learning rate of 0 .

Original: • Chameleon-Metattack: dropout of 0 .
Revised: • Chameleon-Metattack: dropout of 0 .

Original: 3 and learning rate of 0 . 05 .
Revised: 3 and learning rate of 0 . 05 .

Original: • Squirrel-Nettack: dropout of 0 .
Revised: • Squirrel-Nettack: dropout of 0 .

Original: 5 and learning rate of 0 . 1 . • Squirrel-Metattack: dropout of 0 .
Revised: 5 and learning rate of 0 . 1 . • Squirrel-Metattack: dropout of 0 .

Original: 5 and learning rate of 0 . 1 .
Revised: 5 and learning rate of 0 . 1 .

Original: H2GCN.
Revised: H2GCN.

Original: We train a three-layer model in full batch, with a learning rate of 0 . 01 , dropout of 0 .
Revised: We train a three-layer model in full batch, with a learning rate of 0 . 01 , dropout of 0 .

Original: 5 , hidden dimension of 64 , and 300 epochs for both Chameleon and Squirrel datasets.
Revised: 5 , hidden dimension of 64 , and 300 epochs for both Chameleon and Squirrel datasets.

Original: D.2 Defense Baselines
Revised: D.2 Defense Baselines

Original: TSVD.
Revised: TSVD.

Original: We use the same r eigenvectors in TSVD as those used in GARNET, which is shown in
Revised: We use the same r eigenvectors in TSVD as those used in GARNET, which is shown in

Original: Table 6.
Revised: Table 6.

Original: GCNJaccard.
Revised: GCNJaccard.

Original: We choose the GCNJaccard hyperparameters based on the DeepRobust library [38].
Revised: We choose the GCNJaccard hyperparameters based on the DeepRobust library [38].

Original: GNNGuard.
Revised: GNNGuard.

Original: We set edge pruning threshold (the only hyperparameter in GNNGuard) to be P 0 = 0 . 1 .
Revised: We set edge pruning threshold (the only hyperparameter in GNNGuard) to be P 0 = 0 . 1 .

Original: Soft Median GDC.
Revised: Soft Median GDC.

Original: We strictly follow the hyperparameter setting suggested by Geisler et al.
Revised: We strictly follow the hyperparameter setting suggested by Geisler et al.

Original: In particular, we choose temperature T of 5 . 0 for soft median, α of 0.1 (0.15) and k of 64 (32) for GDC on ogbn-arxiv (ogbn-products).
Revised: In particular, we choose temperature T of 5 . 0 for soft median, α of 0.1 (0.15) and k of 64 (32) for GDC on ogbn-arxiv (ogbn-products).

Original: ProGNN.
Revised: ProGNN.

Original: We find out its performance is very sensitive to hyperparameters.
Revised: We find out its performance is very sensitive to hyperparameters.

Original: Thus we strictly follow the tuned hyperparameters available at github.com/ChandlerBang/Pro-GNN/scripts.
Revised: Thus we strictly follow the tuned hyperparameters available at github.com/ChandlerBang/Pro-GNN/scripts.

Original: As GCN-ProGNN training is very slow on Pubmed (estimated time is 30 days for 10 runs), we follow the suggestion from ProGNN authors to replace “svd” with “truncated svd” in the ProGNN implementation.
Revised: As GCN-ProGNN training is very slow on Pubmed (estimated time is 30 days for 10 runs), we follow the suggestion from ProGNN authors to replace “svd” with “truncated svd” in the ProGNN implementation.

Original: D.3 GARNET
Revised: D.3 GARNET

Original: We run all GNN training with a full batch way and show the hyperparameters of GARNET on different datasets under Nettack ( 1 perturbation per node), Metattack ( 10% perturbation ratio), and GR-BCD ( 25% perturbation ratio) in Table 6.
Revised: We run all GNN training with a full batch way and show the hyperparameters of GARNET on different datasets under Nettack ( 1 perturbation per node), Metattack ( 10% perturbation ratio), and GR-BCD ( 25% perturbation ratio) in Table 6.

Original: Note that we provide our strategy of choosing r in Appendix F,
Revised: Note that we provide our strategy of choosing r in Appendix F,

Original: [Table]
Revised: [Table]

Original: which avoids conducting hyperparameter tuning on r per dataset.
Revised: which avoids conducting hyperparameter tuning on r per dataset.

Original: Besides, we set the prior data variance σ 2 to be positive infinity for all graphs (i.e., ignore self-loops of Θ in Equation 2).
Revised: Besides, we set the prior data variance σ 2 to be positive infinity for all graphs (i.e., ignore self-loops of Θ in Equation 2).

Original: Last but not least, we refer readers to our official configuration files at github.com/cornell-zhang/GARNET/configs for the detailed choices of the threshold γ for edge pruning.
Revised: Last but not least, we refer readers to our official configuration files at github.com/cornell-zhang/GARNET/configs for the detailed choices of the threshold γ for edge pruning.

Original: E Hardware Information
Revised: E Hardware Information

Original: We conduct all experiments on a Linux machine with an Intel Xeon Gold 5218 CPU ( 8 cores @ 2 . 30 GHz) CPU, 8 NVIDIA RTX 2080 Ti GPU ( 11 GB memory per GPU), and 1 RTX A 6000 GPU (48 GB memory).
Revised: We conduct all experiments on a Linux machine with an Intel Xeon Gold 5218 CPU ( 8 cores @ 2 . 30 GHz), 8 NVIDIA RTX 2080 Ti GPU ( 11 GB memory per GPU), and 1 RTX A 6000 GPU ( GB memory).

Original: F Complexity Analysis of GARNET F.1 Time Complexity – Choice of r
Revised: F Complexity Analysis of GARNET F.1 Time Complexity – Choice of r

Original: We choose r based on the number of classes per dataset, which depends on the downstream task rather than number of nodes in the graph.
Revised: We choose r based on the number of classes per dataset, which depends on the downstream task rather than number of nodes in the graph.

Original: Specifically, suppose λ r is the r -th largest eigenvalue, an appropriate r is chosen if there is a large gap between λ r and λ r +1 (i.e., a large eigengap) in the graph spectrum.
Revised: Specifically, suppose λ r is the r -th largest eigenvalue, an appropriate r is chosen if there is a large gap between λ r and λ r +1 (i.e., a large eigengap) in the graph spectrum.

Original: According to [39], the eigengap is highly related to the number of clusters in the graph.
Revised: According to [39], the eigengap is highly related to the number of clusters in the graph.

Original: In this work, we approximate r by r ≈ 10 c to cover the large eigengap, where c denotes the number of classes/clusters.
Revised: In this work, we approximate r by r ≈ 10 c to cover the large eigengap, where c denotes the number of classes/clusters.

Original: As shown in Tables 1 and 6, the number of classes in small (large) graphs is around 5 ( 50 ), so we use r = 50 ( r = 500 ) in experiments.
Revised: As shown in Tables 1 and 6, the number of classes in small (large) graphs is around 5 ( 50 ), so we use r = 50 ( r = 500 ) in experiments.

Original: As a result, GARNET has the near-linear time complexity O
Revised: As a result, GARNET has the near-linear time complexity O

Original: ( r ( | E |
Revised: ( r ( | E |

Original: + k | V | )
Revised: + k | V | )

Original: + | V | log | V | )
Revised: + | V | log | V | )

Original: =
Revised: =

Original: O ( c ( | E | + k | V | )
Revised: O ( c ( | E | + k | V | )

Original: + | V | log | V | ) .
Revised: + | V | log | V | ) .

Original: F.2 Space Complexity
Revised: F.2 Space Complexity

Original: GARNET involves forming a sparse kNN graph by building hierarchical navigable small world (HNSW) graphs [28] that contain O ( | V | log | V | ) nodes in total and each node connects to a fixed number of neighbors.
Revised: GARNET involves forming a sparse kNN graph by building hierarchical navigable small world (HNSW) graphs [28] that contain O ( | V | log | V | ) nodes in total and each node connects to a fixed number of neighbors.

Original: Thus, the space complexity of storing the HNSW graphs is O ( | V | log | V | ) .
Revised: Thus, the space complexity of storing the HNSW graphs is O ( | V | log | V | ) .

Original: In addition, GARNET also needs to store the input adversarial graph and the produced kNN graph.
Revised: In addition, GARNET also needs to store the input adversarial graph and the produced kNN graph.

Original: As a result, the total space complexity of GARNET is O ( | V | (log | V | + k ) + |E| ) , where |V| and |E| denote the number of nodes and edges in the adversarial graph, respectively, and k is the averaged node degree in the kNN graph.
Revised: As a result, the total space complexity of GARNET is O ( | V | (log | V | + k ) + |E| ) , where |V| and |E| denote the number of nodes and edges in the adversarial graph, respectively, and k is the averaged node degree in the kNN graph.

Original: Apart from the complexity analysis, we further provide the algorithms of GARNET and TSVD below for comparison.
Revised: Apart from the complexity analysis, we further provide the algorithms of GARNET and TSVD below for comparison.

Original: For GARNET algorithm, the embedding matrix V at line 3 can be further concatenated with the node feature matrix, which may improve the quality of G base as discussed in
Revised: For GARNET algorithm, the embedding matrix V at line 3 can be further concatenated with the node feature matrix, which may improve the quality of G base as discussed in

Original: Appendix L. Moreover, lines 5 and 6 are optional as illustrated in Appendix S.
Revised: Appendix L. Moreover, lines 5 and 6 are optional as illustrated in Appendix S.

Original: G Ablation Study
Revised: G Ablation Study

Original: G.1 Choice of k for kNN Graph Construction
Revised: G.1 Choice of k for kNN Graph Construction

Original: [Figure]
Revised: [Figure]

Original: k
Revised: k

Original: A d v e r s a r i
Revised: A d v e r s a r i

Original: a l A cc u r a c y
Revised: a l A cc u r a c y

Original: Metattack w/ 10% Perturbation
Revised: Metattack w/ 10% Perturbation

Original: [Figure]
Revised: [Figure]

Original: k
Revised: k

Original: [Figure]
Revised: [Figure]

Original: A d v e r s a r i a
Revised: A d v e r s a r i a

Original: [Figure]
Revised: [Figure]

Original: cc u
Revised: cc u

Original: [Equation]
Revised: [Equation]

Original: c y
Revised: c y

Original: Mettack w/ 20% Perturbation
Revised: Mettack w/ 20% Perturbation

Original: [Figure]
Revised: [Figure]

Original: To evaluate the sensitivity of GARNET to k nearest-neighbor (kNN) graph construction, we evaluate the adversarial accuracy of GARNET with different k values for constructing kNN graphs.
Revised: To evaluate the sensitivity of GARNET to k nearest-neighbor (kNN) graph construction, we evaluate the adversarial accuracy of GARNET with different k values for constructing kNN graphs.

Original: Figure
Revised: Figure

Original: 6 shows that the accuracy of GARNET does not change too much when varying k value within the range of [10 , 100] , indicating a relatively large k (e.g., k ≥ 10 ) can enable the kNN graph to incorporate most of edges in the underlying clean graph.
Revised: 6 shows that the accuracy of GARNET does not change too much when varying k value within the range of [10 , 100] , indicating a relatively large k (e.g., k ≥ 10 ) can enable the kNN graph to incorporate most of edges in the underlying clean graph.

Original: Consequently, the performance of GARNET is relatively robust to the choice of k for kNN graph construction.
Revised: Consequently, the performance of GARNET is relatively robust to the choice of k for kNN graph construction.

Original: As the peak performance is typically achieved in [30 , 80] , we recommend choosing k = 30 ∼ 80 for building the kNN graph in practice.
Revised: As the peak performance is typically achieved in [30 , 80] , we recommend choosing k = 30 ∼ 80 for building the kNN graph in practice.

Original: G.2 Choice of γ for edge pruning
Revised: G.2 Choice of γ for edge pruning

Original: [Figure]
Revised: [Figure]

Original: Apart from the choice of k for kNN graph construction, another critical hyperparameter of GARNET is the threshold γ that determines whether an edge should be pruned in the base graph.
Revised: Apart from the choice of k for kNN graph construction, another critical hyperparameter of GARNET is the threshold γ that determines whether an edge should be pruned in the base graph.

Original: Thus, we further evaluate the effect of γ on the performance of GARNET.
Revised: Thus, we further evaluate the effect of γ on the performance of GARNET.

Original: Specifically, we pick γ in the set of { 0 . 001 , 0 . 005 , 0 . 01 , 0 .
Revised: Specifically, we pick γ in the set of { 0 . 001 , 0 . 005 , 0 . 01 , 0 .

Original: and evaluate the corresponding adversarial accuracy of GARNET under Nettack with 1 perturbation per target node.
Revised: and evaluate the corresponding adversarial accuracy of GARNET under Nettack with 1 perturbation per target node.

Original: As shown in Figure 7, the improper choice of γ may degrade the adversarial accuracy of GARNET by 3% .
Revised: As shown in Figure 7, the improper choice of γ may degrade the adversarial accuracy of GARNET by 3% .

Original: It is worth noting that the proper value of γ depends on the specific method adopted for measuring spectral embedding distortion.
Revised: It is worth noting that the proper value of γ depends on the specific method adopted for measuring spectral embedding distortion.

Original: In other words, the optimal γ will be different if we use the simplified version of spectral embedding distortion discussed in Appendix S.
Revised: In other words, the optimal γ will be different if we use the simplified version of spectral embedding distortion discussed in Appendix S.

Original: Thus, we refer readers to our official configuration files at github.com/cornell-zhang/GARNET/configs for the proper choices of γ based on the simplified distortion metric.
Revised: Thus, we refer readers to our official configuration files at github.com/cornell-zhang/GARNET/configs for the proper choices of γ based on the simplified distortion metric.

Original: H Backbone GNN Models for Defense
Revised: H Backbone GNN Models for Defense

Original: As GARNET can be integrated with any existing GNN models to improve their adversarial accuracy, we choose two popular GNN models as the backbone model in our experiments: GCN and
Revised: As GARNET can be integrated with any existing GNN models to improve their adversarial accuracy, we choose two popular GNN models as the backbone model in our experiments: GCN and

Original: GPRGNN [30, 31].
Revised: GPRGNN [30, 31].

Original: As the GCN model implicitly assumes the underlying graph is homophilic, it performs poorly on heterophilic graphs [10].
Revised: As the GCN model implicitly assumes the underlying graph is homophilic, it performs poorly on heterophilic graphs [10].

Original: In contrast, GPRGNN can work on both homophilic and heterophilic datasets, due to its learned graph filter that can adapt to the homophily/heterophily property of the underlying graph.
Revised: In contrast, GPRGNN can work on both homophilic and heterophilic datasets, due to its learned graph filter that can adapt to the homophily/heterophily property of the underlying graph.

Original: Thus, we choose GPRGNN as the backbone model for evaluation on heterophilic datasets.
Revised: Thus, we choose GPRGNN as the backbone model for evaluation on heterophilic datasets.

Original: In addition, we also show the defense results with the H2GCN [10] as backbone model in Appendix J.
Revised: In addition, we also show the defense results with the H2GCN [10] as backbone model in Appendix J.

Original: I Defense Results with Various Perturbation Budgets
Revised: I Defense Results with Various Perturbation Budgets

Original: We provide additional defense results under Nettack and Metattack with various perturbations in
Revised: We provide additional defense results under Nettack and Metattack with various perturbations in

Original: Tables 7 and 8 respectively.
Revised: Tables 7 and 8 respectively.

Original: The results indicate that GARNET outperforms prior defense methods in most cases.
Revised: The results indicate that GARNET outperforms prior defense methods in most cases.

Original: J Defense on H2GCN
Revised: J Defense on H2GCN

Original: We provide the results of combining GARNET with H2GCN [10] on heterophilic graphs in Table 9, which shows that GARNET achieves the highest accuracy in most cases and improves the accuracy on all perturbed graphs by a large margin compared to the vanilla H2GCN as well as H2GCN-TSVD.
Revised: We provide the results of combining GARNET with H2GCN [10] on heterophilic graphs in Table 9, which shows that GARNET achieves the highest accuracy in most cases and improves the accuracy on all perturbed graphs by a large margin compared to the vanilla H2GCN as well as H2GCN-TSVD.

Original: The results further confirm that GARNET is able to improve robustness of different backbone GNN models.
Revised: The results further confirm that GARNET is able to improve robustness of different backbone GNN models.

Original: [Table]
Revised: [Table]

Original: As GCN is not designed for heterophilic graphs, we only show of defense methods with
Revised: As GCN is not designed for heterophilic graphs, we only show of defense methods with

Original: GPRGNN as the backbone model on Chameleon and Squirrel.
Revised: GPRGNN as the backbone model on Chameleon and Squirrel.

Original: We and highest backbone
Revised: We and highest backbone

Original: OOM out of
Revised: OOM out of

Original: [Table]
Revised: [Table]

Original: highest accuracy, respectively.
Revised: highest accuracy, respectively.

Original: The backbone GNN model is H2GCN
Revised: The backbone GNN model is H2GCN

Original: [Table]
Revised: [Table]

Original: K Broader Impact
Revised: K Broader Impact

Original: Zügner et al.
Revised: Zügner et al.

Original: [5] have shown that graph adversarial attacks can drastically degrade the performance of GNN models for downstream applications.
Revised: [5] have shown that graph adversarial attacks can drastically degrade the performance of GNN models for downstream applications.

Original: For instance, an attacker can attack a GNN-based recommender system on Facebook social network or Amazon co-purchasing network, via creating a fake account and make some connections to other users or items.
Revised: For instance, an attacker can attack a GNN-based recommender system on Facebook social network or Amazon co-purchasing network, via creating a fake account and make some connections to other users or items.

Original: Those connections can be viewed as adversarial edges in the graph.
Revised: Those connections can be viewed as adversarial edges in the graph.

Original: As a result
Revised: As a result

Original: [Table]
Revised: [Table]

Original: the attacker can deliberately enforce a GNN model to recommend some irrelevant or even harmful contents to other users.
Revised: the attacker can deliberately enforce a GNN model to recommend some irrelevant or even harmful contents to other users.

Original: Thus, improving adversarial robustness of GNN models has the potential for positive societal benefit.
Revised: Thus, improving adversarial robustness of GNN models has the potential for positive societal benefit.

Original: We hope that this paper provides insight on the robustness and scalablity limitations of prior defense methods.
Revised: We hope that this paper provides insight on the robustness and scalablity limitations of prior defense methods.

Original: Moreover, we believe that the proposed GARNET can largely overcome these two limitations and produce a robust GNN model against adversarial attacks on large-scale graph datasets.
Revised: Moreover, we believe that the proposed GARNET can largely overcome these two limitations and produce a robust GNN model against adversarial attacks on large-scale graph datasets.

Original: Nevertheless, we have to admit that GARNET may potentially provide the attacker with some hints about developing an even more powerful and scalable adversarial attack than all existing attacks, which is a possible negative consequence
Revised: Nevertheless, we have to admit that GARNET may potentially provide the attacker with some hints about developing an even more powerful and scalable adversarial attack than all existing attacks, which is a possible negative consequence

Original: [Table]
Revised: [Table]

Original: L Discussion on Node Features L.1 Graph Construction with Node Features
Revised: L Discussion on Node Features L.1 Graph Construction with Node Features

Original: [Table]
Revised: [Table]

Original: As GARNET purifies the adversarial graph by building a kNN graph based on dominant singular components, a natural question is whether the kNN graph constructed from node features can also achieve similar performance.
Revised: As GARNET purifies the adversarial graph by building a kNN graph based on dominant singular components, a natural question is whether the kNN graph constructed from node features can also achieve similar performance.

Original: We answer this question by comparing the results of GARNET graph and the node feature graph in Table 10.
Revised: We answer this question by comparing the results of GARNET graph and the node feature graph in Table 10.

Original: Note that the clean and adversarial accuracy are the same on the graph constructed from node features, since node features are unchanged after graph adversarial attack.
Revised: Note that the clean and adversarial accuracy are the same on the graph constructed from node features, since node features are unchanged after graph adversarial attack.

Original: Besides, we only show results on homophilic graphs as the kNN graph constructed from node features naturally falls into this category.
Revised: Besides, we only show results on homophilic graphs as the kNN graph constructed from node features naturally falls into this category.

Original: Table 10 shows that the node feature graph performs much worse than GARNET graph.
Revised: Table 10 shows that the node feature graph performs much worse than GARNET graph.

Original: This further confirms that the method proposed in this work is critical to improve the robustness of GNN models.
Revised: This further confirms that the method proposed in this work is critical to improve the robustness of GNN models.

Original: Apart from constructing the kNN graph purely from node features, we can also concatenate node features with dominant singular components for kNN graph construction, which may further improve the accuracy of GARNET.
Revised: Apart from constructing the kNN graph purely from node features, we can also concatenate node features with dominant singular components for kNN graph construction, which may further improve the accuracy of GARNET.

Original: Note that we only adopt this notion for homophilic graphs, as this approach implicitly assumes that the graph is homophilic (nodes with similar features are adjacent in the kNN graph).
Revised: Note that we only adopt this notion for homophilic graphs, as this approach implicitly assumes that the graph is homophilic (nodes with similar features are adjacent in the kNN graph).

Original: The results in Table 10 indicate that augmenting GARNET with node features can further improve the accuracy in several cases.
Revised: The results in Table 10 indicate that augmenting GARNET with node features can further improve the accuracy in several cases.

Original: L.2 Defense Against Node Feature Attack
Revised: L.2 Defense Against Node Feature Attack

Original: GARNET can be extended to handle node feature attack, although this paper mainly focuses on defending against graph structure attack, which we believe is more challenging than defending node feature attack due to the discrete nature.
Revised: GARNET can be extended to handle node feature attack, although this paper mainly focuses on defending against graph structure attack, which we believe is more challenging than defending node feature attack due to the discrete nature.

Original: Specifically, we can perform TSVD to obtain the low-rank approximation of the node feature matrix, which can remove high-rank adversarial components in node features [7].
Revised: Specifically, we can perform TSVD to obtain the low-rank approximation of the node feature matrix, which can remove high-rank adversarial components in node features [7].

Original: The low-rank feature matrix is then concatenated to the weighted spectral embeddings to produce the kNN base graph.
Revised: The low-rank feature matrix is then concatenated to the weighted spectral embeddings to produce the kNN base graph.

Original: In this way, the downstream GNN model will be able to aggregate neighbors whose features are less perturbed during message passing.
Revised: In this way, the downstream GNN model will be able to aggregate neighbors whose features are less perturbed during message passing.

Original: M Additional Visualization Results
Revised: M Additional Visualization Results

Original: [Figure]
Revised: [Figure]

Original: We visualize more target nodes and their local structures in Figure 8, which reveals that GARNET consistently improves the quality of adversarial graph by removing adversarial edges that connect nodes with different labels.
Revised: We visualize more target nodes and their local structures in Figure 8, which reveals that GARNET consistently improves the quality of adversarial graph by removing adversarial edges that connect nodes with different labels.

Original: As a result, the adversarial accuracy of backbone GNN models can be largely improved once they are trained on the GARNET graph.
Revised: As a result, the adversarial accuracy of backbone GNN models can be largely improved once they are trained on the GARNET graph.

Original: N Acceleration of GARNET on Large Graphs
Revised: N Acceleration of GARNET on Large Graphs

Original: There are two major kernels in GARNET: (1) weighted spectral embedding (i.e., computing the top r singular components or Laplacian eigenpairs), (2) kNN graph construction.
Revised: There are two major kernels in GARNET: (1) weighted spectral embedding (i.e., computing the top r singular components or Laplacian eigenpairs), (2) kNN graph construction.

Original: For accelerating weighted spectral embedding, we can leverage the notion of multi-level graph coarsening [40, 41] so that we only need to perform TSVD on the coarsest graph.
Revised: For accelerating weighted spectral embedding, we can leverage the notion of multi-level graph coarsening [40, 41] so that we only need to perform TSVD on the coarsest graph.

Original: To speedup the process of kNN graph construction, we can exploit Faiss to enable performing kNN on GPU [42].
Revised: To speedup the process of kNN graph construction, we can exploit Faiss to enable performing kNN on GPU [42].

Original: O Homophily Score of GARNET Graph
Revised: O Homophily Score of GARNET Graph

Original: [Table]
Revised: [Table]

Original: We follow Zhu et al.
Revised: We follow Zhu et al.

Original: [10] to compute the homophily score per dataset (lower score means more heterophilic).
Revised: [10] to compute the homophily score per dataset (lower score means more heterophilic).

Original: As shown in Table 11, the GARNET graph is homophilic (heterophilic) if the corresponding clean graph is homophilic (heterophilic), which further confirms Theorem 3.3 that our approach can effectively recover the clean graph structure.
Revised: As shown in Table 11, the GARNET graph is homophilic (heterophilic) if the corresponding clean graph is homophilic (heterophilic), which further confirms Theorem 3.3 that our approach can effectively recover the clean graph structure.

Original: As a result, GARNET supports both homophilic and heterophilic graphs.
Revised: As a result, GARNET supports both homophilic and heterophilic graphs.

Original: P Accuracy of Clean Graph Recovery
Revised: P Accuracy of Clean Graph Recovery

Original: [Table]
Revised: [Table]

Original: Apart from visualizing GARNET graph in Figures 5 and 8, we further quantify how well GARNET recovers the clean graph structure.
Revised: Apart from visualizing GARNET graph in Figures 5 and 8, we further quantify how well GARNET recovers the clean graph structure.

Original: Concretely, given a target node, we first extract nodes within its 2-hop neighbors in the clean graph and GARNET graph (under Metattack with 20% perturbation ratio), respectively.
Revised: Concretely, given a target node, we first extract nodes within its 2-hop neighbors in the clean graph and GARNET graph (under Metattack with 20% perturbation ratio), respectively.

Original: By denoting the extracted nodes by N clean for clean graph and N garnet for
Revised: By denoting the extracted nodes by N clean for clean graph and N garnet for

Original: GARNET graph, we define the recall score and precision score as follows:
Revised: GARNET graph, we define the recall score and precision score as follows:

Original: [Equation]
Revised: [Equation]

Original: Table 12 shows the averaged recall and precision over 5 nodes on Cora and Chameleon graphs.
Revised: Table 12 shows the averaged recall and precision over 5 nodes on Cora and Chameleon graphs.

Original: The results show that the recall scores are very high for both graphs, which indicates GARNET is able to accurately recover clean graph structure.
Revised: The results show that the recall scores are very high for both graphs, which indicates GARNET is able to accurately recover clean graph structure.

Original: The relatively low precision scores indicate that GARNET also introduces new edges to the graph (i.e., |
Revised: The relatively low precision scores indicate that GARNET also introduces new edges to the graph (i.e., |

Original: N garnet | > | N clean | ).
Revised: N garnet | > | N clean | ).

Original: We argue that those new edges are likely to connect spectrally similar nodes that are far away in the original clean graph, which enables GARNET to also incorporate global structural information.
Revised: We argue that those new edges are likely to connect spectrally similar nodes that are far away in the original clean graph, which enables GARNET to also incorporate global structural information.

Original: This explains why GARNET can sometimes outperform vanilla GNN models on clean heterophilic graphs (shown in Table 3), where global structural information is very critical for node prediction.
Revised: This explains why GARNET can sometimes outperform vanilla GNN models on clean heterophilic graphs (shown in Table 3), where global structural information is very critical for node prediction.

Original: Q Further Discussion on Graph Recovery with PGM
Revised: Q Further Discussion on Graph Recovery with PGM

Original: Intuitively, if we use more (clean) Laplacian eigenpairs (i.e., a larger r ) for constructing the embedding matrix V based on Definition 3.1, the optimal solution for Equation 2 (i.e., Θ ∗ ) will be closer to the actual clean graph.
Revised: Intuitively, if we use more (clean) Laplacian eigenpairs (i.e., a larger r ) for constructing the embedding matrix V based on Definition 3.1, the optimal solution for Equation 2 (i.e., Θ ∗ ) will be closer to the actual clean graph.

Original: In this section, we confirm this intuition based on graph resistance distances between node pairs.
Revised: In this section, we confirm this intuition based on graph resistance distances between node pairs.

Original: Specifically, consider the following expression for calculating effective-resistance distances between nodes p and q using all Laplacian eigenvalues/eigenvectors except λ 1 = 0 :
Revised: Specifically, consider the following expression for calculating effective-resistance distances between nodes p and q using all Laplacian eigenvalues/eigenvectors except λ 1 = 0 :

Original: | V | (cid:88)
Revised: | V | (cid:88)

Original: i =
Revised: i =

Original: ( u Ti e p,q )λ
Revised: ( u Ti e p,q )λ

Original: i
Revised: i

Original: Feng [17] has shown that the effective-resistance distance between any node pair on the learned graph Θ ∗ (when σ approaches infinity) will fully match the Euclidean distance between the corresponding data samples (i.e., rows in weighted spectral embedding matrix V in our case).
Revised: Feng [17] has shown that the effective-resistance distance between any node pair on the learned graph Θ ∗ (when σ approaches infinity) will fully match the Euclidean distance between the corresponding data samples (i.e., rows in weighted spectral embedding matrix V in our case).

Original: Moreover, it can be shown that the Euclidean distance between the data samples in our case will match the effectiveresistance distance on the original graph when r = |V| (with proper normalization on Laplacian eigenvectors).
Revised: Moreover, it can be shown that the Euclidean distance between the data samples in our case will match the effectiveresistance distance on the original graph when r = |V| (with proper normalization on Laplacian eigenvectors).

Original: As a result, the resistance distances on the learned graph Θ ∗ will be the same as the ones on the original graph when r = |V| .
Revised: As a result, the resistance distances on the learned graph Θ ∗ will be the same as the ones on the original graph when r = |V| .

Original: Moreover, using a larger r value will lead to a more accurate estimation of the learned (clean) graph.
Revised: Moreover, using a larger r value will lead to a more accurate estimation of the learned (clean) graph.

Original: In practice, if r satisfies that λ r ≪ λ r +1 , dropping the terms with much larger eigenvalues (i.e., λ r +1 , λ r +2 , ..., λ |V| ) will not significantly impact the approximation accuracy.
Revised: In practice, if r satisfies that λ r ≪ λ r +1 , dropping the terms with much larger eigenvalues (i.e., λ r +1 , λ r +2 , ..., λ |V| ) will not significantly impact the approximation accuracy.

Original: A proper r can be effectively determined based on the strategy proposed in Appendix F.
Revised: A proper r can be effectively determined based on the strategy proposed in Appendix F.

Original: We leave the theoretical guarantee of other metrics for graph comparison to our future work.
Revised: We leave the theoretical guarantee of other metrics for graph comparison to our future work.

Original: R Connection Between kNN Graph and TSVD Graph
Revised: R Connection Between kNN Graph and TSVD Graph

Original: Apart from the motivation of constructing a kNN graph as G base based on Theorem 3.3, we further motivate the kNN graph construction from the perspective of improving the scalability of TSVDbased defense methods.
Revised: Apart from the motivation of constructing a kNN graph as G base based on Theorem 3.3, we further motivate the kNN graph construction from the perspective of improving the scalability of TSVDbased defense methods.

Original: Concretely, as previous TSVD-based methods produce a dense (low-rank) adjacency matrix ˆ A , they involve dense matrices during GNN training, which has quadratic time/space complexity and thus cannot scale to large graphs.
Revised: Concretely, as previous TSVD-based methods produce a dense (low-rank) adjacency matrix ˆ A , they involve dense matrices during GNN training, which has quadratic time/space complexity and thus cannot scale to large graphs.

Original: A potential solution is to sparsify ˆ A by preserving the top k largest elements per row.
Revised: A potential solution is to sparsify ˆ A by preserving the top k largest elements per row.

Original: However, naïvely selecting the largest elements of each row in ˆ A requires forming/storing ˆ A first, which still has quadratic time/space complexity.
Revised: However, naïvely selecting the largest elements of each row in ˆ A requires forming/storing ˆ A first, which still has quadratic time/space complexity.

Original: In contrast, we leverage the (approximate) kNN algorithm to construct the sparsified ˆ A by taking as input the weighted spectral embedding V (note that ˆ A = V V T based on Proposition 3.2).
Revised: In contrast, we leverage the (approximate) kNN algorithm to construct the sparsified ˆ A by taking as input the weighted spectral embedding V (note that ˆ A = V V T based on Proposition 3.2).

Original: Consequently, our kNN graph construction step can also be viewed as a scalable way of sparsifying the dense adjacency matrix ˆ A generated by TSVD.
Revised: Consequently, our kNN graph construction step can also be viewed as a scalable way of sparsifying the dense adjacency matrix ˆ A generated by TSVD.

Original: Moreover, Theorem 3.3 theoretically guarantees that the sparsified graph serves as a reasonable G base for edge pruning.
Revised: Moreover, Theorem 3.3 theoretically guarantees that the sparsified graph serves as a reasonable G base for edge pruning.

Original: S Simplified Version of Spectral Embedding Distortion
Revised: S Simplified Version of Spectral Embedding Distortion

Original: To obtain s i,j = ∥
Revised: To obtain s i,j = ∥

Original: U T e i,j ∥ 22 ∥ V T e i,j ∥ 22 , we have to compute both the top r Laplacian eigenpairs of G base (for constructing U ) and those of G adv (for constructing V ).
Revised: U T e i,j ∥ 22 ∥ V T e i,j ∥ 22 , we have to compute both the top r Laplacian eigenpairs of G base (for constructing U ) and those of G adv (for constructing V ).

Original: However, as G base is relatively dense (a large k used for kNN graph construction), computing its Laplacian eigenpairs is time-consuming, especially on large graphs.
Revised: However, as G base is relatively dense (a large k used for kNN graph construction), computing its Laplacian eigenpairs is time-consuming, especially on large graphs.

Original: Fortunately, Von Luxburg [39] has shown that the top r Laplacian eigenvectors (corresponding to smallest eigenvalues) only vary a little across nodes in a dense graph (i.e., they are smooth over the graph), which means ∥ U T e
Revised: Fortunately, Von Luxburg [39] has shown that the top r Laplacian eigenvectors (corresponding to smallest eigenvalues) only vary a little across nodes in a dense graph (i.e., they are smooth over the graph), which means ∥ U T e

Original: i,j ∥ 2 is very similar across different edges in
Revised: i,j ∥ 2 is very similar across different edges in

Original: G base .
Revised: G base .

Original: Consequently, the term ∥ V T e i,j ∥ 2 becomes the dominant factor in s i,j to identify (non)critical edges: a large value of ∥ V T e i,j ∥ 2 means a small value of s i,j , indicating the corresponding edge is noncritical.
Revised: Consequently, the term ∥ V T e i,j ∥ 2 becomes the dominant factor in s i,j to identify (non)critical edges: a large value of ∥ V T e i,j ∥ 2 means a small value of s i,j , indicating the corresponding edge is noncritical.

Original: As we empirically find out that exploiting ∥
Revised: As we empirically find out that exploiting ∥

Original: V T e i,j ∥ 2 to prune noncritical edges does not degrade the accuracy of GARNET, we adopt this simplified version of spectral embedding distortion in our implementation.
Revised: V T e i,j ∥ 2 to prune noncritical edges does not degrade the accuracy of GARNET, we adopt this simplified version of spectral embedding distortion in our implementation.