Original: H OW P OWERFUL IS I MPLICIT D ENOISING IN G RAPH N EURAL N ETWORKS
Revised: H OW P OWERFUL IS I MPLICIT D ENOISING IN G RAPH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Neural Networks (GNNs), which aggregate features from neighbors, are widely used for graph-structured data processing due to their powerful representation learning capabilities.
Revised: Graph Neural Networks (GNNs), which aggregate features from neighbors, are widely used for graph-structured data processing due to their powerful representation learning capabilities.

Original: It is generally believed that GNNs can implicitly remove the non-predictive noises.
Revised: It is generally believed that GNNs can implicitly remove the non-predictive noises.

Original: However, the analysis of implicit denoising effect in graph neural networks remains open.
Revised: However, the analysis of implicit denoising effect in graph neural networks remains open.

Original: In this work, we conduct a comprehensive theoretical study and analyze when and why the implicit denoising happens in
Revised: In this work, we conduct a comprehensive theoretical study and analyze when and why the implicit denoising happens in

Original: GNNs.
Revised: GNNs.

Original: Specifically, we study the convergence properties of noise matrix.
Revised: Specifically, we study the convergence properties of noise matrix.

Original: Our theoretical analysis suggests that the implicit denoising largely depends on the connectivity, the graph size, and GNN architectures.
Revised: Our theoretical analysis suggests that the implicit denoising largely depends on the connectivity, the graph size, and GNN architectures.

Original: Moreover, we formally define and propose the adversarial graph signal denoising (AGSD) problem by extending graph signal denoising problem.
Revised: Moreover, we formally define and propose the adversarial graph signal denoising (AGSD) problem by extending graph signal denoising problem.

Original: By solving such a problem, we derive a robust graph convolution, where the smoothness of the node representations and the implicit denoising effect can be enhanced.
Revised: By solving such a problem, we derive a robust graph convolution, where the smoothness of the node representations and the implicit denoising effect can be enhanced.

Original: Extensive empirical evaluations verify our theoretical analyses and the effectiveness of our proposed model.
Revised: Extensive empirical evaluations verify our theoretical analyses and the effectiveness of our proposed model.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) (Kipf & Welling, 2017; Veli ˇ ckovi ´ c et al., 2018; Hamilton et al., 2017) have been widely used in graph learning and achieved remarkable performance on graphbased tasks, such as traffic prediction (Guo et al., 2019), drug discovery (Dai et al., 2019), and recommendation system (Ying et al., 2018).
Revised: Graph Neural Networks (GNNs) (Kipf & Welling, 2017; Veli ˇ ckovi ´ c et al., 2018; Hamilton et al., 2017) have been widely used in graph learning and achieved remarkable performance on graphbased tasks, such as traffic prediction (Guo et al., 2019), drug discovery (Dai et al., 2019), and recommendation system (Ying et al., 2018).

Original: A general principle behind Graph Neural Networks (GNNs) (Kipf & Welling, 2017;
Revised: A general principle behind Graph Neural Networks (GNNs) (Kipf & Welling, 2017;

Original: Veli ˇ ckovi ´ c et al., 2018; Hamilton et al., 2017) is to perform a message passing operation that aggregates node features over neighborhoods, such that the smoothness of learned node representations on the graph is enhanced.
Revised: Veli ˇ ckovi ´ c et al., 2018; Hamilton et al., 2017) is to perform a message passing operation that aggregates node features over neighborhoods, such that the smoothness of learned node representations on the graph is enhanced.

Original: By promoting graph smoothness, the message passing and aggregation mechanism naturally leads to GNN models whose predictions are not only dependent on the feature of one specific node, but also the features from a set of neighboring nodes.
Revised: By promoting graph smoothness, the message passing and aggregation mechanism naturally leads to GNN models whose predictions are not only dependent on the feature of one specific node, but also the features from a set of neighboring nodes.

Original: Therefore, this mechanism can, to a certain extent, protect GNN models from noises: real-world graphs are usually noisy, e.g., Gaussian white noise exists on node features (Zhou et al., 2021), however, the influence of feature noises on the model’s output could be counteracted by the feature aggregation operation in GNNs.
Revised: Therefore, this mechanism can, to a certain extent, protect GNN models from noises: real-world graphs are usually noisy, e.g., Gaussian white noise exists on node features (Zhou et al., 2021), however, the influence of feature noises on the model’s output could be counteracted by the feature aggregation operation in GNNs.

Original: We term this effect as implicit denoising .
Revised: We term this effect as implicit denoising .

Original: While many works have been conducted in the empirical exploration of GNNs, relatively fewer advances have been achieved in theoretically studying this denoising effect.
Revised: While many works have been conducted in the empirical exploration of GNNs, relatively fewer advances have been achieved in theoretically studying this denoising effect.

Original: Early GNN models, such as the vanilla GCN (Kipf & Welling, 2017), GAT (Veliˇckovi´c et al., 2018) and GraphSAGE (Hamilton et al., 2017), propose different designs of aggregation functions, but the denoising effect is not discussed in these works.
Revised: Early GNN models, such as the vanilla GCN (Kipf & Welling, 2017), GAT (Veliˇckovi´c et al., 2018) and GraphSAGE (Hamilton et al., 2017), propose different designs of aggregation functions, but the denoising effect is not discussed in these works.

Original: Some recent attempts (Ma et al., 2021) are made to mathematically establish the connection between a variety of GNNs and the graph signal denoising problem (GSD)
Revised: Some recent attempts (Ma et al., 2021b) are made to mathematically establish the connection between a variety of GNNs and the graph signal denoising problem (GSD)

Original: (Chen et al., 2014):
Revised: (Chen et al., 2014):

Original: [Equation]
Revised: [Equation]

Original: where X = X ∗ + η is the observed noisy feature matrix, η ∈ R n × d is the noise matrix, X ∗ is the clean feature matrix, and (cid:101) L is the graph Laplacian.
Revised: where X = X ∗ + η is the observed noisy feature matrix, η ∈ R n × d is the noise matrix, X ∗ is the clean feature matrix, and (cid:101) L is the graph Laplacian.

Original: The second term encourages the smoothness of the filtered feature matrix F over the graph.
Revised: The second term encourages the smoothness of the filtered feature matrix F over the graph.

Original: , i.e., nearby vertices should have similar vertex features.
Revised: , i.e., nearby vertices should have similar vertex features.

Original: By regarding the feature aggregation process in GNNs as solving a GSD problem, more advanced GNNs are proposed, such as GLP (Li et al., 2019), S 2 GC (Zhu & Koniusz, 2021), and IRLS (Yang et al.,
Revised: By regarding the feature aggregation process in GNNs as solving a GSD problem, more advanced GNNs are proposed, such as GLP (Li et al., 2019), S 2 GC (Zhu & Koniusz, 2021), and IRLS (Yang et al.,

Original: Despite these prior attempts, little efforts have been made to rigorously study the denoising effect of message passing and aggregation operation.
Revised: Despite these prior attempts, little efforts have been made to rigorously study the denoising effect of message passing and aggregation operation.

Original: This urges us to think about a fundamental but not clearly answered question:
Revised: This urges us to think about a fundamental but not clearly answered question:

Original: Why and when implicit denoising happens in GNNs?
Revised: Why and when implicit denoising happens in GNNs?

Original: In this work, we focus on the non-predictive stochasticity of noise in GNNs’ aggregated features and analyze its properties.
Revised: In this work, we focus on the non-predictive stochasticity of noise in GNNs’ aggregated features and analyze its properties.

Original: We prove that with the increase in graph size and graph connectivity factor, the stochasticity tends to diminish, which is called the “denoising effect” in our work.
Revised: We prove that with the increase in graph size and graph connectivity factor, the stochasticity tends to diminish, which is called the “denoising effect” in our work.

Original: We will address this question using the tools from concentration inequalities and matrix theories, which are concerned with the study of the convergence of noise matrix.
Revised: We will address this question using the tools from concentration inequalities and matrix theories, which are concerned with the study of the convergence of noise matrix.

Original: It offers a new framework to study the properties of graphs and GNNs in terms of the denoising effect.
Revised: It offers a new framework to study the properties of graphs and GNNs in terms of the denoising effect.

Original: In order to facilitate our theoretical analysis, we derive Neumann Graph Convolution (NGC) from GSD.
Revised: In order to facilitate our theoretical analysis, we derive Neumann Graph Convolution (NGC) from GSD.

Original: Specifically, to study the convergence rate, we introduce an insightful measurement on the convolution operator, termed high-order graph connectivity factor , which reveals how uniform the nodes are distributed in the neighborhood and reflects the strength of information diluted on a single neighboring node during the feature aggregation step.
Revised: Specifically, to study the convergence rate, we introduce an insightful measurement on the convolution operator, termed high-order graph connectivity factor , which reveals how uniform the nodes are distributed in the neighborhood and reflects the strength of information diluted on a single neighboring node during the feature aggregation step.

Original: Intuitively, as the General Hoeffding Inequality (Hoeffding, 1994) (Lemma.
Revised: Intuitively, as the General Hoeffding Inequality (Hoeffding, 1994) (Lemma.

Original: D.1) suggests, a larger high-order graph connectivity factor, i.e., nodes are more uniformly distributed in the neighborhood, accelerates the convergence of the noise matrix and a larger graph size leads to faster convergence.
Revised: D.1) suggests, a larger high-order graph connectivity factor, i.e., nodes are more uniformly distributed in the neighborhood, accelerates the convergence of the noise matrix and a larger graph size leads to faster convergence.

Original: Besides, GNN architectures also affect the convergence rate.
Revised: Besides, GNN architectures also affect the convergence rate.

Original: Deeper
Revised: Deeper

Original: GNNs can have a faster convergence rate.
Revised: GNNs can have a faster convergence rate.

Original: To further strengthen the denoising effect, inspired by the adversarial training method (Madry et al., 2018), we propose the adversarial graph signal denoising problem (AGSD).
Revised: To further strengthen the denoising effect, inspired by the adversarial training method (Madry et al., 2018), we propose the adversarial graph signal denoising problem (AGSD).

Original: By solving such a problem, we derive a robust graph convolution model based on the correlation of node feature and graph structure to increase the high-order graph connectivity factor, which helps us improve the denoising performance.
Revised: By solving such a problem, we derive a robust graph convolution model based on the correlation of node feature and graph structure to increase the high-order graph connectivity factor, which helps us improve the denoising performance.

Original: Extensive experimental results on standard graph learning tasks verify our theoretical analyses and the effectiveness of our derived robust graph convolution model.
Revised: Extensive experimental results on standard graph learning tasks verify our theoretical analyses and the effectiveness of our derived robust graph convolution model.

Original: Notations.
Revised: Notations.

Original: Let G =
Revised: Let G =

Original: ( V , E ) represent a undirected graph, where V is the set of vertices { v 1 , · · · , v n } with |V| = n and E is the set of edges.
Revised: ( V , E ) represent a undirected graph, where V is the set of vertices { v 1 , · · · , v n } with |V| = n and E is the set of edges.

Original: The adjacency matrix is defined as A ∈ { 0 , 1 } n × n , and
Revised: The adjacency matrix is defined as A ∈ { 0 , 1 } n × n , and

Original: A i,j = 1 if and only if ( v i , v j ) ∈ E .
Revised: A i,j = 1 if and only if ( v i , v j ) ∈ E .

Original: Let N i
Revised: Let N i

Original: = { v j | A
Revised: = { v j | A

Original: i,j = 1 } denote the neighborhood of node v i and D denote the diagonal degree matrix, where D i,i = (cid:80) nj
Revised: i,j = 1 } denote the neighborhood of node v i and D denote the diagonal degree matrix, where D i,i = (cid:80) nj

Original: =1 A i,j .
Revised: =1 A i,j .

Original: The feature matrix is denoted as
Revised: The feature matrix is denoted as

Original: X ∈ R n × d where each node v i is associated with a d -dimensional feature vector X i .
Revised: X ∈ R n × d where each node v i is associated with a d -dimensional feature vector X i .

Original: Y ∈ { 0 , 1 } n × c denotes the matrix, where Y i ∈ { 0 , 1 } c is a one-hot vector and (cid:80) cj
Revised: Y ∈ { 0 , 1 } n × c denotes the matrix, where Y i ∈ { 0 , 1 } c is a one-hot vector and (cid:80) cj

Original: =1 Y i,j = 1 for any v i ∈ V .
Revised: =1 Y i,j = 1 for any v i ∈ V .

Original:  A S IMPLE U NIFYING F RAMEWORK : N EUMANN G RAPH C ONVOLUTION
Revised:  A S IMPLE U NIFYING F RAMEWORK : N EUMANN G RAPH C ONVOLUTION

Original: A General Framework.
Revised: A General Framework.

Original: In this section, we discuss a simple yet general framework for solving graph signal denoising problem, namely Neumann Graph Convolution (NGC).
Revised: In this section, we discuss a simple yet general framework for solving graph signal denoising problem, namely Neumann Graph Convolution (NGC).

Original: Note that NGC is not a new GNN architecture.
Revised: Note that NGC is not a new GNN architecture.

Original: There also exist similar GNN architectures, such as GLP (Li et al., 2019), S 2 GC (Zhu & Koniusz, 2021), and GaussianMRF (Jia & Benson, 2022).
Revised: There also exist similar GNN architectures, such as GLP (Li et al., 2019), S 2 GC (Zhu & Koniusz, 2021), and GaussianMRF (Jia & Benson, 2022).

Original: We focus on the theoretical analysis of the denoising effect in GNNs in this work.
Revised: We focus on the theoretical analysis of the denoising effect in GNNs in this work.

Original: NGC can facilitate our theoretical analysis.
Revised: NGC can facilitate our theoretical analysis.

Original: By taking the derivative ∇ q ( F ) = 2 (cid:101) LF + 2( F − X ) to zero, we obtain the solution of
Revised: By taking the derivative ∇ q ( F ) = 2 (cid:101) LF + 2( F − X ) to zero, we obtain the solution of

Original: GSD optimization problem as follows:
Revised: GSD optimization problem as follows:

Original: [Equation]
Revised: [Equation]

Original: To avoid the expensive computation of the inverse matrix, we can use Neumann series (Stewart, 1998) expansion to approximate Eq.
Revised: To avoid the expensive computation of the inverse matrix, we can use Neumann series (Stewart, 1998) expansion to approximate Eq.

Original: (2) up to up to S -th order:
Revised: (2) up to up to S -th order:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101)
Revised: (cid:101)

Revised: [Equation]

Original:  A can take the form of (cid:101)
Revised: (cid:19) [Equation] (3) where (cid:101) A can take the form of (cid:101)

Original: A = (cid:101) D − 12 (cid:101)
Revised: A = (cid:101) D − 12 (cid:101)

Original: A (cid:101)
Revised: A (cid:101)

Original: D − 12 or (cid:101)
Revised: D − 12 or (cid:101)

Original: A = (cid:101) D − 1 (cid:101) A , and the proof can be found in
Revised: A = (cid:101) D − 1 (cid:101) A , and the proof can be found in

Original: Appendix B. Based on the Neumann series expansion of the solution of GSD, we introduce a general graph convolution model – Neumann Graph Convolution defined as the following expansion:
Revised: Appendix B. Based on the Neumann series expansion of the solution of GSD, we introduce a general graph convolution model – Neumann Graph Convolution defined as the following expansion:

Original: [Equation]
Revised: [Equation]

Original: where (cid:101) A S = {+
+}
Revised: where (cid:101) A S = {+
+}

Original: [Equation]
Revised: λ +1 (cid:80)

Original:  s and W is the weight matrix.
Revised: Ss =0 (cid:16) λλ +1 (cid:101) A (cid:17) s and W is the weight matrix.

Original: Our spectral convolution (cid:101) A S X on graphs is a multi-scale graph convolution (Abu-El-Haija et al., 2020; Liao et al., 2019), which covers the single-scale graph convolution models such as SGC (Wu et al., 2019a) since the graph convolution of SGC is (cid:101)
Revised: Our spectral convolution (cid:101) A S X on graphs is a multi-scale graph convolution (Abu-El-Haija et al., 2020; Liao et al., 2019), which covers the single-scale graph convolution models such as SGC (Wu et al., 2019a) since the graph convolution of SGC is (cid:101)

Original: A 2 X
Revised: A 2 X

Original: and (cid:101)
Revised: and (cid:101)

Original: A 2 is the third term of (cid:101) A S .
Revised: A 2 is the third term of (cid:101) A S .

Revised: Specifically, the N-GCN (Abu-El-Haija et al., 2020) extends the feature aggregation module in GCN by concatenating neighboring feature information using (cid:101)

Revised: A k at each layer, while for NGC, we incorporate such multi-scale information by summing up (cid:16) λλ +1 (cid:101)

Revised: A (cid:17) s as can be seen from Eq.

Original: Besides, if we remove the non-linear functions in GCN (Kipf & Welling, 2017), it also can be covered by our model.
Revised: Besides, if we remove the non-linear functions in GCN (Kipf & Welling, 2017), it also can be covered by our model.

Original: Therefore, we can draw the conclusion that our proposed NGC is a general framework.
Revised: Therefore, we can draw the conclusion that our proposed NGC is a general framework.

Original: High-order Graph Connectivity Factor.
Revised: High-order Graph Connectivity Factor.

Original: Based on NGC, we obtain the filtered graph signal via
Revised: Based on NGC, we obtain the filtered graph signal via

Original: F = (cid:101)
Revised: F = (cid:101)

Original: A S X .
Revised: A S X .

Original: Intuitively, (cid:101)
Revised: Intuitively, (cid:101)

Original: A S captures not only the connectivity of the graph structure (represented by (cid:101) A ), but also the higher order connectivity (represented by (cid:101)
Revised: A S captures not only the connectivity of the graph structure (represented by (cid:101) A ), but also the higher order connectivity (represented by (cid:101)

Original: A 2 , (cid:101)
Revised: A 2 , (cid:101)

Original: A 3 , . .
Revised: A 3 , . .

Original: , (cid:101) A S ).
Revised: , (cid:101) A S ).

Original: As will be discussed in Sec. 3, larger high order graph connectivity can accelerate the convergence of the noise feature matrix.
Revised: As will be discussed in Sec. 3, larger high order graph connectivity can accelerate the convergence of the noise feature matrix.

Original: To formally quantify the high order graph connectivity, we give the following definition:
Revised: To formally quantify the high order graph connectivity, we give the following definition:

Original: Definition 1 (High-order Graph Connectivity Factor) .
Revised: Definition 1 (High-order Graph Connectivity Factor) .

Original: We define the high-order graph connectivity factor τ as τ [Equation]
Revised: We define the high-order graph connectivity factor τ as τ = max

Original: i τ i , where τ i = n
Revised: i τ i , where τ i = n

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: Here we give some intuitions about why Eq.
Revised: Here we give some intuitions about why Eq.

Original: (5) represents high-order graph connectivity.
Revised: (5) represents high-order graph connectivity.

Original: Note that each element in (cid:101)
Revised: Note that each element in (cid:101)

Original: A S is non-negative and each row sum satisfies
Revised: A S is non-negative and each row sum satisfies

Original: [Equation]
Revised: [Equation]

Original: Based on Eq.
Revised: Based on Eq.

Original: (6), the sum of squares of elements in each row satisfy:
Revised: (6), the sum of squares of elements in each row satisfy:

Original: [Equation]
Revised: [Equation]

Original: When the high-order graph has a high connectivity, i.e., the elements in row i of (cid:101)
Revised: When the high-order graph has a high connectivity, i.e., the elements in row i of (cid:101)

Original: A S are more uniformly distributed, Eq.
Revised: A S are more uniformly distributed, Eq.

Original: (7) reaches its lower bound.
Revised: (7) reaches its lower bound.

Original: Meanwhile, if the graph is not connected and there is only one element whose value is larger than 0 in row i , Eq.
Revised: Meanwhile, if the graph is not connected and there is only one element whose value is larger than 0 in row i , Eq.

Original: (7) reaches its upper bound.
Revised: (7) reaches its upper bound.

Original: Therefore, the value of τ ∈ [1 , n ] is determined as follows: when the high-order graph connectivity is high, τ → 1 and when the graph is less connected, τ → n .
Revised: Therefore, the value of τ ∈ [1 , n ] is determined as follows: when the high-order graph connectivity is high, τ → 1 and when the graph is less connected, τ → n .

Original:  M AIN T HEORY
Revised:  M AIN T HEORY

Original: In this section, we analyze the denoising effect of NGC.
Revised: In this section, we analyze the denoising effect of NGC.

Original: Before we present our main theory, we first present our aggregation on noisy feature matrix and formulate four assumptions, which are necessary to construct our theory.
Revised: Before we present our main theory, we first present our aggregation on noisy feature matrix and formulate four assumptions, which are necessary to construct our theory.

Original: For the convenience of theoretical analysis, we adopt MSE loss 2 for our main theory.
Revised: For the convenience of theoretical analysis, we adopt MSE loss 2 for our main theory.

Original: Consider (cid:101)
Revised: Consider (cid:101)

Original: A S as our aggregation scheme, the NGC training based on Eq.
Revised: A S as our aggregation scheme, the NGC training based on Eq.

Original: (4) can be formulated as
Revised: (4) can be formulated as

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where X ∗ is the clean feature matrix, η denotes the noise added on X ∗ , and X = X ∗ + η is the observed data matrix.
Revised: where X ∗ is the clean feature matrix, η denotes the noise added on X ∗ , and X = X ∗ + η is the observed data matrix.

Original: Intuitively, if (cid:101)
Revised: Intuitively, if (cid:101)

Original: A S η is small enough, the added noise will not change the optimization direction on which the parameter is updated under the clean feature matrix X ∗ .
Revised: A S η is small enough, the added noise will not change the optimization direction on which the parameter is updated under the clean feature matrix X ∗ .

Original: Before we present our main theory, we give four assumptions about noise η , (cid:101) A S , and parameters W .
Revised: Before we present our main theory, we give four assumptions about noise η , (cid:101) A S , and parameters W .

Original: Assumption 1.
Revised: Assumption 1.

Original: Each entry of the noise matrix η , i.e., [ η ] ij is i.i.d sub-Gaussian random variable with variance σ and mean µ = 0 , i.e.,
Revised: Each entry of the noise matrix η , i.e., [ η ] ij is i.i.d sub-Gaussian random variable with variance σ and mean µ = 0 , i.e.,

Original: [Equation]
Revised: [Equation]

Revised: for all λ ∈ R .

Original:  Note that it is common to assume that the noise follows Gaussian distribution (Zhou et al., 2021;
Revised: [Equation] Note that it is common to assume that the noise follows Gaussian distribution (Zhou et al., 2021;

Original: Chen et al., 2021; Zhang et al., 2022), which is also covered by our sub-Gaussian assumption.
Revised: Chen et al., 2021; Zhang et al., 2022), which is also covered by our sub-Gaussian assumption.

Original: Assumption 2.
Revised: Assumption 2.

Original: The high-order graph connectivity factor τ is O ( n ) , i.e., lim n →∞ τn = 0 .
Revised: The high-order graph connectivity factor τ is O ( n ) , i.e., lim n →∞ τn = 0 .

Original: As we have discussed in Sec. 2, τ depends on the graph structure.
Revised: As we have discussed in Sec. 2, τ depends on the graph structure.

Original: In a well-connected graph, τ is usually relatively small compared with n .
Revised: In a well-connected graph, τ is usually relatively small compared with n .

Original: Only if all nodes of a graph are isolated, τ reaches its upper bound n .
Revised: Only if all nodes of a graph are isolated, τ reaches its upper bound n .

Original: Assumption 3.
Revised: Assumption 3.

Original: The Frobenius norm of the parameter matrix W is bounded by a constant.
Revised: The Frobenius norm of the parameter matrix W is bounded by a constant.

Original: There exists C > 0 such that ∥ W ∥ F ≤ C, which is unrelated to n .
Revised: There exists C > 0 such that ∥ W ∥ F ≤ C, which is unrelated to n .

Original: We assume that the Frobenius norm of W is bounded by a constant.
Revised: We assume that the Frobenius norm of W is bounded by a constant.

Original: This is reasonable since recent advances in Neural Tangent Kernel (Jacot et al., 2018) indicate that over-parameterized network weights lie in the neighborhood of the small random initialization, which justifies Assumption 3.
Revised: This is reasonable since recent advances in Neural Tangent Kernel (Jacot et al., 2018) indicate that over-parameterized network weights lie in the neighborhood of the small random initialization, which justifies Assumption 3.

Original: Assumption 4.
Revised: Assumption 4.

Original: The loss function in Eq.
Revised: The loss function in Eq.

Original: (8) is L -smooth,
Revised: (8) is L -smooth,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The L -smoothness of f depends on the largest singular value of (cid:101) A S X .
Revised: The L -smoothness of f depends on the largest singular value of (cid:101) A S X .

Original: For conciseness, we start with the smooth case.
Revised: For conciseness, we start with the smooth case.

Original: As the core part of our proof, we first derive the upper bound of the Frobenius norm of (cid:101)
Revised: As the core part of our proof, we first derive the upper bound of the Frobenius norm of (cid:101)

Original: A S η .
Revised: A S η .

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose we choose t = 2 τ
Revised: Suppose we choose t = 2 τ

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: S

Original: [Equation] [Equation] 4 log n + log 2 d ) /n .
Revised: S (cid:19) 2 (4 log n + log 2 d ) /n .

Original: Then under Assumptions 1 and 2, with a high probability 1 − 1 /d , we have
Revised: Then under Assumptions 1 and 2, with a high probability 1 − 1 /d , we have

Original: [Equation]
Revised: [Equation]

Original: where the proof can be found in Appendix D.
Revised: where the proof can be found in Appendix D.

Original: Lemma 1 implies that the norm of the aggregated noise matrix (cid:101)
Revised: Lemma 1 implies that the norm of the aggregated noise matrix (cid:101)

Original: A S η is bounded by three terms: the number of nodes of a graph n , the expansion order S , the high-order graph connectivity factor τ .
Revised: A S η is bounded by three terms: the number of nodes of a graph n , the expansion order S , the high-order graph connectivity factor τ .

Original: Intuitively, as the concentration bounds suggest, if we extract enough samples from the same sub-Gaussian variable, the average of these samples will converge to zero with a high probability.
Revised: Intuitively, as the concentration bounds suggest, if we extract enough samples from the same sub-Gaussian variable, the average of these samples will converge to zero with a high probability.

Original: This requires our graph to be large enough and the sum of squares of the elements in the row of (cid:101)
Revised: This requires our graph to be large enough and the sum of squares of the elements in the row of (cid:101)

Original: A S to be small enough, which depends on the graph structure.
Revised: A S to be small enough, which depends on the graph structure.

Original: Now we start to present our main theorem for graph denoising.
Revised: Now we start to present our main theorem for graph denoising.

Original: In order to demonstrate the effect of graph denoising, we further consider another loss function g ( · ) with the clean feature matrix:
Revised: In order to demonstrate the effect of graph denoising, we further consider another loss function g ( · ) with the clean feature matrix:

Original: [Equation]
Revised: [Equation]

Original: Let W ∗ g
Revised: Let W ∗ g

Original: = arg
Revised: = arg

Original: min W g ( W ) be the minimizer of clean loss g , we aim to demonstrate that the learned model (from gradient descent on the noisy data X ) has essentially the same performance as W ∗ g which is the optimal solution for the clean loss g .
Revised: min W g ( W ) be the minimizer of clean loss g , we aim to demonstrate that the learned model (from gradient descent on the noisy data X ) has essentially the same performance as W ∗ g which is the optimal solution for the clean loss g .

Original: Theorem 1.
Revised: Theorem 1.

Original: Under Assumptions 1, 2, 3, 4 and Lemma 1, let W ( k ) f denote the k -th step gradient descent solution for min W f ( W ) with step size α ≤ 1 /L , with probability 1 − 1 /d
Revised: Under Assumptions 1, 2, 3, 4 and Lemma 1, let W ( k ) f denote the k -th step gradient descent solution for min W f ( W ) with step size α ≤ 1 /L , with probability 1 − 1 /d

Original: we have 
Revised: we have g (cid:16)

Original: [Equation]
Revised: [Equation]

Original: where W ∗ g = arg
Revised: where W ∗ g = arg

Original: min W g ( W ) is the optimal solution of the clean loss function g ( W ) , τ is the high-order graph connectivity factor, and n is the number of nodes of a graph.
Revised: min W g ( W ) is the optimal solution of the clean loss function g ( W ) , τ is the high-order graph connectivity factor, and n is the number of nodes of a graph.

Original: 𝑣 ! 𝑣 " 𝑣 # 𝑣 $
Revised: 𝑣 ! 𝑣 " 𝑣 # 𝑣 $

Original: Graph
Revised: Graph

Original: Metric 𝒢 !
Revised: Metric 𝒢 !

Original: 𝒢 $ 𝒢 " 𝒢 # #Nodes    𝜏 1.
Revised: 𝒢 $ 𝒢 " 𝒢 # #Nodes    𝜏 1.

Original: 𝜏 log𝑛 /𝑛 1.
Revised: 𝜏 log𝑛 /𝑛 1.

Original: The proof of Theorem 1 can be found in Appendix E.
Revised: The proof of Theorem 1 can be found in Appendix E.

Original: Remark 2.
Revised: Remark 2.

Original: Denoising Effect.
Revised: Denoising Effect.

Original: Theorem 1 suggests that the k -th step gradient descent solution W ( k ) f which is trained using the noisy feature matrix X enjoys a similar performance as the actual clean loss minimizer W ∗ g with large enough k and n .
Revised: Theorem 1 suggests that the k -th step gradient descent solution W ( k ) f which is trained using the noisy feature matrix X enjoys a similar performance as the actual clean loss minimizer W ∗ g with large enough k and n .

Original: This implies the denoising effect of our proposed solution in Eq.
Revised: This implies the denoising effect of our proposed solution in Eq.

Original: Remark 3.
Revised: Remark 3.

Original: Effect of graph structure on denoising.
Revised: Effect of graph structure on denoising.

Original: Note that the second term in Eq.
Revised: Note that the second term in Eq.

Original: (13) suggests that the denoising effect is linear with respect to τ , which is directly related to the dataset graph structure.
Revised: (13) suggests that the denoising effect is linear with respect to τ , which is directly related to the dataset graph structure.

Original: Specifically, as will be shown in Sec. 3,refer to Sec. 3 on how graph structure affects the value of τ , and thus the denoising effect.
Revised: Specifically, as will be shown in Sec. 3, graph structure affects the value of τ , and thus the denoising effect.

Original: A large well-connected graph tend to have a better denoising performance since n is large and τ is close to 1.
Revised: A large well-connected graph tend to have a better denoising performance since n is large and τ is close to 1.

Original: Case Study: the Influence of Graph Structure on Implicit Denoising.
Revised: Case Study: the Influence of Graph Structure on Implicit Denoising.

Original: Remark 3 suggests that τ plays an important role on the implicit denoising effect.

Original: In this case study, we give four illustration samples in Figure 1.
Revised: In this case study, we give four illustration samples in Figure 1.

Original: G 1 , G 2 , and G 3 have the same number of nodes.
Revised: G 1 , G 2 , and G 3 have the same number of nodes.

Original: But the nodes on G 1 are isolated.
Revised: But the nodes on G 1 are isolated.

Original: G 2 has only one connected component and has a center node v 1 on the graph.
Revised: G 2 has only one connected component and has a center node v 1 on the graph.

Original: G 3 is a complete graph such that there is an edge between any two nodes.
Revised: Gis a complete graph such that there is an edge between any two nodes.

Original: In addition, we give a larger illustration graph G 4 to understand the influence of graph size.
Revised: In addition, we give a larger illustration graph G 4 to understand the influence of graph size.

Original: From Figure 1, we can extract the following insights: 1) There is no denoising effect (the value of τ log n/n is quite large) on G 1 since the nodes are isolated.
Revised: From Figure 1, we can extract the following insights: 1) There is no denoising effect (the value of τ log n/n is quite large) on G 1 since the nodes are isolated.

Original: And GNNs will degrade to MLP under such graph structure, leading to no aggregation.

Original: The complete graph G 3 has the best denoising effect among the graphs of the same size if we only consider the influence of graph structure. Since the values of elements in each row are distributed uniformly, leading to the lower bound of τ . 3)
Revised: The complete graph G 3 has the best denoising effect among the graphs of the same size since the values of elements in each row are distributed uniformly, leading to the lower bound of τ . 3)

Original: Although G 2 has only one connected component, there is a center node v 1 on the graph.
Revised: Although G 2 has only one connected component, there is a center node v 1 on the graph.

Original: The existence of the center node makes the value of elements in each row imbalanced, which means that τ tends to have a larger value compared with G 3 . 4)
Revised: The existence of the center node makes the value of elements in each row imbalanced, which means that τ tends to have a larger value compared with G 3 . 4)

Original: The decentralized graph like G 4 also can get a smaller τ . 
Revised: The decentralized graph like Galso can get a smaller τ . 5)

Original: In terms of graph size, the graph with a larger size has a better denoising effect.
Revised: In terms of graph size, the graph with a larger size has a better denoising effect.

Revised: Combining our Theorem 1 and case study, we conclude that the denoising effect is influenced by graph size n and the high-order graph connectivity factor τ , which reflects the graph structure.

Original:  R OBUST N EUMANN G RAPH C ONVOLUTION
Revised:  R OBUST N EUMANN G RAPH C ONVOLUTION

Original: In this section, we propose a new graph signal denoising problem - adversarial graph signal denoising (AGSD) problem to improve the denoising performance by deriving a robust graph convolution model.
Revised: In this section, we propose a new graph signal denoising problem - adversarial graph signal denoising ( AGSD ) problem to improve the denoising performance by deriving a robust graph convolution model.

Original: 4. A DVERSARIAL G RAPH S IGNAL D ENOISING P ROBLEM
Revised: 4. A DVERSARIAL G RAPH S IGNAL D ENOISING P ROBLEM

Original: Note that the second term in the GSD problem (Eq.
Revised: Note that the second term in the GSD problem (Eq.

Original: (1)) which controls the smoothness of the feature matrix over graphs, is related to both the graph Laplacian and the node features.
Revised: (1)) which controls the smoothness of the feature matrix over graphs, is related to both the graph Laplacian and the node features.

Original: Therefore, the slight changes in the graph Laplacian matrix could lead to an unstable denoising effect.
Revised: Therefore, the slight changes in the graph Laplacian matrix could lead to an unstable denoising effect.

Original: Inspired by the recent studies in adversarial training (Madry et al., 2018), we formulate the adversarial graph signal denoising problem as a min-max optimization problem: 
Revised: Inspired by the recent studies in adversarial training (Madry et al., 2018), we formulate the adversarial graph signal denoising problem as a min-max optimization problem: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Intuitively, the inner maximization on the Laplacian L ′ generates perturbations on the graph structure 3 , and enlarges the distance between the node representations of connected neighbors.
Revised: Intuitively, the inner maximization on the Laplacian L ′ generates perturbations on the graph structure 3 , and enlarges the distance between the node representations of connected neighbors.

Original: Such maximization finds the worst case perturbations on the graph Laplacian that hinders the global smoothness of F over the graph.
Revised: Such maximization finds the worst case perturbations on the graph Laplacian that hinders the global smoothness of F over the graph.

Original: Therefore, by training on those worse case Laplacian perturbations, one could obtain a robust graph signal denoising solution.
Revised: Therefore, by training on those worse case Laplacian perturbations, one could obtain a robust graph signal denoising solution.

Original: Ideally, through solving Eq.
Revised: Ideally, through solving Eq.

Original: (14), the smoothness of the node representations as well as the implicit denoising effect can be enhanced.
Revised: (14), the smoothness of the node representations as well as the implicit denoising effect can be enhanced.

Original: 4. M INIMIZATION OF THE O PTIMIZATION P ROBLEM
Revised: 4. M INIMIZATION OF THE O PTIMIZATION P ROBLEM

Original: The min-max formulation in Eq.
Revised: The min-max formulation in Eq.

Original: (14) also makes the adversarial graph signal denoising problem much harder to solve.
Revised: (14) also makes the adversarial graph signal denoising problem much harder to solve.

Original: Fortunately, unlike adversarial training (Madry et al., 2017) where we need to first adopt PGD to solve the inner maximization problem before we solve the outer minimization problem, here inner maximization problem is simple and has a closed form solution.
Revised: Fortunately, unlike adversarial training (Madry et al., 2017) where we need to first adopt PGD to solve the inner maximization problem before we solve the outer minimization problem, here inner maximization problem is simple and has a closed form solution.

Original: In other words, we do not need to add random perturbations on the graph structure at each training epoch and can find the largest perturbation which maximizes the inner adversarial loss function.
Revised: In other words, we do not need to add random perturbations on the graph structure at each training epoch and can find the largest perturbation which maximizes the inner adversarial loss function.

Original: Denote the perturbation as δ , and
Revised: Denote the perturbation as δ , and

Original: L ′ =
Revised: L ′ =

Original: (cid:101) L + δ .
Revised: (cid:101) L + δ .

Original: Directly solving 4 the inner maximization problem, we get δ = ε ∇ h
Revised: Directly solving 4 the inner maximization problem, we get δ = ε ∇ h

Original: ( δ ) =
Revised: ( δ ) =

Original: ε FF ⊤ ∥ FF ⊤ ∥ F .
Revised: ε FF ⊤ ∥ FF ⊤ ∥ F .

Original: Plugging this solution into Eq.
Revised: Plugging this solution into Eq.

Original: (14), we can rewrite the outer optimization problem as follows:
Revised: (14), we can rewrite the outer optimization problem as follows:

Original: [Equation]
Revised: [Equation]

Original: Taking the gradient of ρ ( F ) to zero, we get the solution of the outer optimization problem as follows:
Revised: Taking the gradient of ρ ( F ) to zero, we get the solution of the outer optimization problem as follows:

Original: [Equation]
Revised: [Equation]

Original: Both sides of Eq.
Revised: Both sides of Eq.

Original: (16) contains F , directly computing the solution is difficult.
Revised: (16) contains F , directly computing the solution is difficult.

Original: Note that in Eq.
Revised: Note that in Eq.

Original: (14) we also require F to be close to X , we can approximate Eq.
Revised: (14) we also require F to be close to X , we can approximate Eq.

Original: (16) by replacing the F with X in the inverse matrix on the right hand side.
Revised: (16) by replacing the F with X in the inverse matrix on the right hand side.

Original: With the Neumann series expansion of the inverse matrix, we get the final approximate solution as
Revised: With the Neumann series expansion of the inverse matrix, we get the final approximate solution as

Original: [Equation]
Revised: [Equation]

Original: The difference between Eq.
Revised: The difference between Eq.

Original: (17) and Eq.
Revised: (17) and Eq.

Original: (4) is that there is one more term in Eq.
Revised: (4) is that there is one more term in Eq.

Original: (17) derived from solving the inner optimization problem of Eq.
Revised: (17) derived from solving the inner optimization problem of Eq.

Original: Based on this, we proposed our robust Neumann graph convolution (RNGC).
Revised: Based on this, we proposed our robust Neumann graph convolution (RNGC).

Original: Scalability.
Revised: Scalability.

Original: Although RNGC introduces extra computational burdens for large graphs due to the
Revised: Although RNGC introduces extra computational burdens for large graphs due to the

Original: XX ⊤ term, if the feature matrix is sparse, the extra computational effort is minimal as the XX ⊤ term can also be sparse.
Revised: XX ⊤ term, if the feature matrix is sparse, the extra computational effort is minimal as the XX ⊤ term can also be sparse.

Original: For the scalability of RNGC on large graphs with dense feature matrix, we only compute the inner product of feature vectors ( X i , X j | j ∈N i ) between adjacent neighbors like masked attention in GAT.
Revised: For the scalability of RNGC on large graphs with dense feature matrix, we only compute the inner product of feature vectors ( X i , X j | j ∈N i ) between adjacent neighbors like masked attention in GAT.

Original: Compared with NGC, the additional computation cost is O ( |E| ) .
Revised: Compared with NGC, the additional computation cost is O ( |E| ) .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct a comprehensive empirical study to understand the influence of different factors on the denoising effect of various models.
Revised: In this section, we conduct a comprehensive empirical study to understand the influence of different factors on the denoising effect of various models.

Original: To quantify the denoising effect, we test the model accuracy on noisy data on various GNN architectures and MLP for standard node classification tasks, where the noisy data is synthesized by mixing Gaussian noise with the original feature matrix.
Revised: To quantify the denoising effect, we test the model accuracy on noisy data on various GNN architectures and MLP for standard node classification tasks, where the noisy data is synthesized by mixing Gaussian noise with the original feature matrix.

Original: We also synthesize noisy data by flipping individual feature with a small Bernoulli probability on three citation datasets with binary features.
Revised: We also synthesize noisy data by flipping individual feature with a small Bernoulli probability on three citation datasets with binary features.

Original: 5. D ENOISING E FFECTIVENESS C OMPARISON OF V ARIOUS GNN M ODELS
Revised: 5. D ENOISING E FFECTIVENESS C OMPARISON OF V ARIOUS GNN M ODELS

Original: In this section, we compare the denoising effectiveness of different GNN models through their test accuracy by training on the noisy feature matrix with Gaussian noise.
Revised: In this section, we compare the denoising effectiveness of different GNN models through their test accuracy by training on the noisy feature matrix with Gaussian noise.

Original: [Figure]
Revised: [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: [Table]
Revised: [Table]

Original: Datasets.
Revised: Datasets.

Original: In our experiments, we utilize three public citation network datasets Cora, Citeseer, and Pubmed (Sen et al., 2008) which are homophily graphs for semi-supervised node classification.
Revised: In our experiments, we utilize three public citation network datasets Cora, Citeseer, and Pubmed (Sen et al., 2008) which are homophily graphs for semi-supervised node classification.

Original: For the semi-supervised learning experimental setup
Revised: For the semi-supervised learning experimental setup

Original: [Table]
Revised: [Table]

Original: we follow the standard fixed splits employed in (Yang et al., 2016)
Revised: we follow the standard fixed splits employed in (Yang et al., 2016)

Original: [Table]
Revised: [Table]

Original: with 20 nodes per class for training, 500 nodes for validation
Revised: with 20 nodes per class for training, 500 nodes for validation

Original: [Table]
Revised: [Table]

Original: and 1,000 nodes for testing.
Revised: and 1,000 nodes for testing.

Original: We also use four datasets: Cornell, Texas, Wisconsin, and Actor which are heterophily graphs for full-supervised node classification.
Revised: We also use four datasets: Cornell, Texas, Wisconsin, and Actor which are heterophily graphs for full-supervised node classification.

Original: For each dataset, we randomly split nodes into 60%, 20%, and 20% for training, validation, and testing as suggested in (Pei et al., 2020).
Revised: For each dataset, we randomly split nodes into 60%, 20%, and 20% for training, validation, and testing as suggested in (Pei et al., 2020).

Original: Moreover, we utilize three large-scale graph datasets: Coauthor-CS, Coauthor-Phy (Shchur et al., 2018), and ogbn-products (Hu et al., 2020) for evaluation.
Revised: Moreover, we utilize three large-scale graph datasets: Coauthor-CS, Coauthor-Phy (Shchur et al., 2018), and ogbn-products (Hu et al., 2020) for evaluation.

Original: For Coauthor datasets, we split nodes into 60%, 20%, and 20% for training, validation, and testing.
Revised: For Coauthor datasets, we split nodes into 60%, 20%, and 20% for training, validation, and testing.

Original: For ogbn-products dataset, we follow the dataset split in OGB (Hu et al., 2020).
Revised: For ogbn-products dataset, we follow the dataset split in OGB (Hu et al., 2020).

Original: Baselines.
Revised: Baselines.

Original: For the baselines, we consider graph neural networks derived from graph signal denoising, including GLP (Li et al., 2019), S 2 GC (Zhu & Koniusz, 2021), and IRLS (Yang et al., 2021); popular GNN architectures, such as GCN (Kipf & Welling, 2017) and GAT (Veliˇckovi´c et al., 2018); and MLP which has no aggregation operation.
Revised: For the baselines, we consider graph neural networks derived from graph signal denoising, including GLP (Li et al., 2019), S 2 GC (Zhu & Koniusz, 2021), and IRLS (Yang et al., 2021); popular GNN architectures, such as GCN (Kipf & Welling, 2017) and GAT (Veliˇckovi´c et al., 2018); and MLP which has no aggregation operation.

Original: Experimental Setup and Implementations.
Revised: Experimental Setup and Implementations.

Original: We assume that the original feature matrix is clean and do not have noise and we synthesize the noise from the standard Gaussian distribution and add them on the original feature matrix.
Revised: We assume that the original feature matrix is clean and do not have noise and we synthesize the noise from the standard Gaussian distribution and add them on the original feature matrix.

Original: By default, we apply row normalization for data after adding the Gaussian noise 5 , and train all the models based on these noisy feature matrix.
Revised: By default, we apply row normalization for data after adding the Gaussian noise 5 , and train all the models based on these noisy feature matrix.

Original: For the hyper-parameters of each model, we follow the setting that reported in their original papers.
Revised: For the hyper-parameters of each model, we follow the setting that reported in their original papers.

Original: To eliminate the effect of randomness, we repeat such experiment for 100 or 10 times and report the mean accuracy.
Revised: To eliminate the effect of randomness, we repeat such experiment for 100 or 10 times and report the mean accuracy.

Original: Note that in each repeated run, we add different Gaussian noises.
Revised: Note that in each repeated run, we add different Gaussian noises.

Original: While for the same run, we apply the same noisy feature matrix for training all the models.
Revised: While for the same run, we apply the same noisy feature matrix for training all the models.

Original: For our NGC and RNGC model, the hyper-parameter details can be found in Appendix H.2.
Revised: For our NGC and RNGC model, the hyper-parameter details can be found in Appendix H.2.

Original: Results on Supervised Node Classification.
Revised: Results on Supervised Node Classification.

Original: Figure 2 illustrates the comparison of classification accuracy against the various noise levels for semi-supervised node classification tasks.
Revised: Figure 2 illustrates the comparison of classification accuracy against the various noise levels for semi-supervised node classification tasks.

Original: The noise level ξ
Revised: The noise level ξ

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: controls the magnitude of the Gaussian noise we add to the feature matrix: X + ξ η where η is sampled from standard i.i.d., Gaussian distribution.
Revised: controls the magnitude of the Gaussian noise we add to the feature matrix: X + ξ η where η is sampled from standard i.i.d., Gaussian distribution.

Original: For Cora and Citeseer, we test ξ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } and for Pubmed, we test ξ ∈ { 0 . 01 , 0 . 02 , 0 .
Revised: For Cora and Citeseer, we test ξ ∈ { 0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 } and for Pubmed, we test ξ ∈ { 0 . 01 , 0 . 02 , 0 .

Original: From Figure 2, we can observe that the test accuracy of MLP is close to randomly guessing (RG) when the noise level is relatively large.
Revised: From Figure 2, we can observe that the test accuracy of MLP is close to randomly guessing (RG) when the noise level is relatively large.

Original: This implies the weak denoising effect of MLP models.
Revised: This implies the weak denoising effect of MLP models.

Original: For shallow GNN models, such as GCN and
Revised: For shallow GNN models, such as GCN and

Original: GAT (which usually contain 2 layers), their denoising performance is limited especially on Pubmed since they do not aggregate information (features and noise) from higher-order neighbors.
Revised: GAT (which usually contain 2 layers), their denoising performance is limited especially on Pubmed since they do not aggregate information (features and noise) from higher-order neighbors.

Original: For models with deep layers 6 , such as IRLS ( ≥ 8 layers), the denoising performance is much better compared to shallow models.
Revised: For models with deep layers 6 , such as IRLS ( ≥ 8 layers), the denoising performance is much better compared to shallow models.

Original: Lastly, our NGC and RNGC model with 16 layers ( S = 16 ) achieve significantly better denoising performance compared with other baseline methods, which backup our theoretical analyses.
Revised: Lastly, our NGC and RNGC model with 16 layers ( S = 16 ) achieve significantly better denoising performance compared with other baseline methods, which backup our theoretical analyses.

Original: In most cases, NGC and RNGC achieve very similar denoising performance but in general,
Revised: In most cases, NGC and RNGC achieve very similar denoising performance but in general,

Original: RNGC still slightly outperforms NGC, suggesting that we indeed gain more benefits by solving the adversarial graph denoising problem.
Revised: RNGC still slightly outperforms NGC, suggesting that we indeed gain more benefits by solving the adversarial graph denoising problem.

Original: Table 1 reports the comparison of classification accuracy against the various noise levels for fullsupervised node classification tasks on heterophily graphs.
Revised: Table 1 reports the comparison of classification accuracy against the various noise levels for fullsupervised node classification tasks on heterophily graphs.

Original: The first- and second-highest accuracies are highlighted in bold.
Revised: The first- and second-highest accuracies are highlighted in bold.

Original: For these datasets, we test ξ ∈ { 0 . 01 , 1 } .
Revised: For these datasets, we test ξ ∈ { 0 . 01 , 1 } .

Original: From Table 1, we can observe that MLP is better than most GNN models in most cases due to the heterophily properties of these graphs.
Revised: From Table 1, we can observe that MLP is better than most GNN models in most cases due to the heterophily properties of these graphs.

Original: However, our proposed RNGC achieves significantly better or matches denoising performance compared with other baseline methods, which demonstrates the superiority of our RNGC.
Revised: However, our proposed RNGC achieves significantly better or matches denoising performance compared with other baseline methods, which demonstrates the superiority of our RNGC.

Original: For ogbn-products, we only choose MLP, GCN, and S 2 GC as baselines, since the results are sensitive concerning model size and various tricks from the OGB leaderboard.
Revised: For ogbn-products, we only choose MLP, GCN, and S 2 GC as baselines, since the results are sensitive concerning model size and various tricks from the OGB leaderboard.

Original: For fair comparison, the size of parameters for these baselines and RNGC is the same.
Revised: For fair comparison, the size of parameters for these baselines and RNGC is the same.

Original: We also use full-batch training for the baselines and our model.
Revised: We also use full-batch training for the baselines and our model.

Original: Table 2 and 3 report the comparison of classification accuracy against the various noise levels for full-supervised node classification tasks on large-scale graphs.
Revised: Table 2 and 3 report the comparison of classification accuracy against the various noise levels for full-supervised node classification tasks on large-scale graphs.

Original: The first- and second-highest accuracies are highlighted in bold.
Revised: The first- and second-highest accuracies are highlighted in bold.

Original: For these datasets, we test ξ ∈ { 0 . 1 , 1 } .
Revised: For these datasets, we test ξ ∈ { 0 . 1 , 1 } .

Original: Compared with the above small datasets, the node degree on these three datasets is larger, which means they have better connectivity.
Revised: Compared with the above small datasets, the node degree on these three datasets is larger, which means they have better connectivity.

Original: From Table 2 and 3, we can observe that the test accuracy of MLP is far lower than GCN and RNGC.
Revised: From Table 2 and 3, we can observe that the test accuracy of MLP is far lower than GCN and RNGC.

Original: This implies the weak denoising effect of MLP.
Revised: This implies the weak denoising effect of MLP.

Original: The test accuracy of
Revised: The test accuracy of

Original: GCN is slightly smaller than RNGC on these datasets since they are well-connected and have a large graph size and we can achieve a good denoising performance with shallow-layer GNN models.
Revised: GCN is slightly smaller than RNGC on these datasets since they are well-connected and have a large graph size and we can achieve a good denoising performance with shallow-layer GNN models.

Original: For the scalability of RNGC on large graphs such as ogbn-products, we use the acceleration method mentioned in Sec.
Revised: For the scalability of RNGC on large graphs such as ogbn-products, we use the acceleration method mentioned in Sec.

Original: 5. D ENOISING P ERFORMANCE ON F EATURE F LIPPING P ERTURBATION
Revised: 5. D ENOISING P ERFORMANCE ON F EATURE F LIPPING P ERTURBATION

Original: In this section, we compare the denoising effectiveness of different models through their test accuracy by training on the noisy feature matrix which is perturbated through flipping the individual feature with a small Bernoulli probability on three citation datasets.
Revised: In this section, we compare the denoising effectiveness of different models through their test accuracy by training on the noisy feature matrix which is perturbated through flipping the individual feature with a small Bernoulli probability on three citation datasets.

Original: Setting and Results.
Revised: Setting and Results.

Original: We flip the individual feature on three citation datasets: Cora, Citeseer, and
Revised: We flip the individual feature on three citation datasets: Cora, Citeseer, and

Original: Pubmed as the noise.
Revised: Pubmed as the noise.

Original: And we compare the denoising performance of RNGC with MLP and GCN.
Revised: And we compare the denoising performance of RNGC with MLP and GCN.

Original: From Table 4, we can observe that the denoisingperformance of RNGC is much better than baselines
Revised: From Table 4 (the std information can be found in Appendix I.2), we can observe that the denoising

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  when the flip probability is 0.4.
Revised: performance of RNGC is much better than baselines when the flip probability is 0.4.

Original: In fact, the added perturbations by flipping the individual feature approximately follow a Bernoulli distribution [Table] which is also a Sub-Gaussian distribution.
Revised: In fact, the added perturbations by flipping the individual feature approximately follow a Bernoulli distribution, which is also a Sub-Gaussian distribution.

Original: The results verify our theoretical analysis further.
Revised: The results verify our theoretical analysis further

Original:  5. D EFENSE P ERFORMANCE OF RNGC AGAINST G RAPH S TRUCTURE A TTACK
Revised: . 5. D EFENSE P ERFORMANCE OF RNGC AGAINST G RAPH S TRUCTURE A TTACK

Original: Although we do not perform actual graph structure perturbations as in graph adversarial attacks (Zügner et al., 2018; Zügner & Günnemann, 2019a) but a virtual perturbation in the Laplacian.
Revised: Although we do not perform actual graph structure perturbations as in graph adversarial attacks (Zügner et al., 2018; Zügner & Günnemann, 2019a) but a virtual perturbation in the Laplacian.

Original: Therefore, it’s not clear how much perturbations on the Laplacian correspond to the actual perturbations on graph structure.
Revised: Therefore, it’s not clear how much perturbations on the Laplacian correspond to the actual perturbations on graph structure.

Original: Nevertheless, we still conduct the experiments of RNGC against graph structure meta-attack where the ptb rate is 25%.
Revised: Nevertheless, we still conduct the experiments of RNGC against graph structure meta-attack where the ptb rate is 25%.

Original: As shown in the Table 5, our RNGC model  outperforms than GCN, GAT, RobustGCN (Zügner & Günnemann, 2019b), GCN-Jaccard (Wu et al., 2019b), GCN-SVD (Entezari et al., 2020), and S 2 GC on Cora, Citeseer, and Pubmed.
Revised: As shown in the Table 5 (the std information can be found in Appendix I.2), our RNGC model still outperforms than GCN, GAT, RobustGCN (Zügner & Günnemann, 2019b), GCN-Jaccard (Wu et al., 2019b), GCN-SVD (Entezari et al., 2020), and S 2 GC on Cora, Citeseer, and Pubmed.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Implicit Denoisingin GNNs. Existing graph denoising works are mainly based on the graph smoothing technique (Chen et al., 2014; Zhou et al., 2021).
Revised: Graph Signal Denoising Existing graph denoising works are mainly based on the graph smoothing technique (Chen et al., 2014; Zhou et al., 2021).

Original: It is well known that GNNs can increase the smoothness of node features through aggregating information from neighbors, thus the influence from noisy features can be counteracted in GNN’s output.
Revised: It is well known that GNNs can increase the smoothness of node features through aggregating information from neighbors, thus the influence from noisy features can be counteracted in GNN’s output.

Original: Somerecent GNN models are derived from the perspective of signal denoising, such as S 2 GC (Zhu & Koniusz, 2021), GLP (Li et al., 2019), and IRLS (Yang et al., 2021).
Revised: Some GNN models are derived from the perspective of signal denoising, such as S 2 GC (Zhu & Koniusz, 2021), GLP (Li et al., 2019), and IRLS (Yang et al., 2021).

Original: Moreover, Ma et al.
Revised: Moreover, Ma et al.

Original: (Ma et al., 2021) builds the connection between signal denoising and existingpopular GNNs by formulating message passing as a process of solving the GSD problem.
Revised: (2021b) builds the connection between signal denoising and existing GNNs by formulating message passing as a process of solving the GSD problem.

Original: The relationship between GSD and GCN can be briefly illustrated as follows (Ma et al., 2021).

Original: This suggests a possibility for us to understand the behavior of GNNs through the lens of signal denoising.
Revised: This suggests a possibility for us to understand the behavior of GNNs through the lens of signal denoising.

Original: To our best knowledge, we are the first to offer a theoretical analysis to understand the denoising effect of GNNs. Besides, there is a recent work (Zhang et al., 2022)  to conduct  the empirical study of the denoising effect in GNNs.
Revised: Besides, some works (Liu et al., 2021a; Fan et al., 2022; Ma et al., 2021a; Jin et al., 2021; Liu et al., 2021b; Jin et al., 2020; Zhang et al., 2022) have been proposed to conduct statistical analysis on the graph noise from the empirical perspective.

Original: In this work, we perform an extensive analysis to understand the denoising effect of GNNs from both theoretical and experimental perspectives.
Revised: In this work, we perform an extensive analysis to understand the denoising effect of GNNs from both theoretical and experimental perspectives.

Original: Smoothing and Over-smoothing.
Revised: Smoothing and Over-smoothing.

Original: One key principle of GNNs is to improve the smoothness of node representations.
Revised: One key principle of GNNs is to improve the smoothness of node representations.

Original: But stacking graph layers can lead to over-smoothing (Li et al., 2018), where the node representations can not be distinguishable.
Revised: But stacking graph layers can lead to over-smoothing (Li et al., 2018), where the node representations can not be distinguishable.

Original: There are some recent works that have been proposed to address over-smoothing such as JKnet (Xu et al., 2018), GCNII (Chen et al., 2020), and
Revised: There are some recent works that have been proposed to address over-smoothing such as JKnet (Xu et al., 2018), GCNII (Chen et al., 2020), and

Original: RevGNN-Deep (Li et al., 2021).
Revised: RevGNN-Deep (Li et al., 2021).

Original: They add the output of shallow layers to the final layers with a residual-style design.
Revised: They add the output of shallow layers to the final layers with a residual-style design.

Original: In this work, we will show smoothing can help the denoising effect of GNNs.
Revised: In this work, we will show smoothing can help the denoising effect of GNNs.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Our work conducts a comprehensive study on the implicit denoising effect of graph neural networks.
Revised: Our work conducts a comprehensive study on the implicit denoising effect of graph neural networks.

Original: We theoretical show that the denoising effect of GNNs are largely influenced by the connectivity and the size of the graph structure, as well as the GNN architectures.
Revised: We theoretical show that the denoising effect of GNNs are largely influenced by the connectivity and the size of the graph structure, as well as the GNN architectures.

Original: Motivated by our analysis, we also propose a robust graph convolution model by solving the robust graph signal denoising problem which enhances the smoothness of node representations and the implicit denoising effect.
Revised: Motivated by our analysis, we also propose a robust graph convolution model by solving the robust graph signal denoising problem which enhances the smoothness of node representations and the implicit denoising effect.

Original: A T HE DETAILS ON HOW TO SOLVE THE INNER MAXIMIZATION PROBLEM IN S EC . 4. 
Revised: A T HE DETAILS ON HOW TO SOLVE THE INNER MAXIMIZATION PROBLEM IN S EC . 4.

Original: Different from the non-concave inner maximization problem in the adversarial attack, our inner maximization problem is indeed a convex optimization problem.
Revised: Different from the non-concave inner maximization problem in the adversarial attack, our inner maximization problem is indeed a convex optimization problem.

Original: Hence, we do not need to add random perturbations on the graph structure at each training epoch and can find the largest perturbation which maximizes the inner adversarial loss function.
Revised: Hence, we do not need to add random perturbations on the graph structure at each training epoch and can find the largest perturbation which maximizes the inner adversarial loss function.

Original: Denote the perturbation as δ , and L ′ = (cid:101)
Revised: Denote the perturbation as δ , and L ′ = (cid:101)

Original: L + δ .
Revised: L + δ .

Original: We can rewrite the inner maximization problem as
Revised: We can rewrite the inner maximization problem as

Original: [Equation]
Revised: [Equation]

Original: We denote h ( δ ) = ⟨ δ , F ⊤ F ⟩ .
Revised: We denote h ( δ ) = ⟨ δ , F ⊤ F ⟩ .

Original: Obviously, h ( δ ) reaches the largest value when δ has the same direction with the gradient of h ( δ ) , e.g. δ = ε ∇ h ( δ ) =[Equation]
Revised: Obviously, h ( δ ) reaches the largest value when δ has the same direction with the gradient of h ( δ ) , e.g. δ = ε ∇ h ( δ ) =

Original:  FF ⊤ ∥ FF ⊤ ∥ F , which is illustrated in Fig.
Revised: ε FF ⊤ ∥ FF ⊤ ∥ F , which is illustrated in Fig.

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]

Original: [Figure]

Original: [Equation]
Revised: [Equation]

Original: B A DDITIONAL D ETAILS ON THE N EUMANN S ERIES
Revised: B A DDITIONAL D ETAILS ON THE N EUMANN S ERIES

Original: We provide additional details and derivations on how to obtain the Neumann Series which leads to our Neumann Graph Convolution (NGC) method.
Revised: We provide additional details and derivations on how to obtain the Neumann Series which leads to our Neumann Graph Convolution (NGC) method.

Original: Before we derive the Neumann Series, we first introduce the following lemmas which are crucial to the derivation of the Neumann Series.
Revised: Before we derive the Neumann Series, we first introduce the following lemmas which are crucial to the derivation of the Neumann Series.

Original: Lemma B.1.
Revised: Lemma B.1.

Original: (Gelfand formula) (Bhatia, 2013)
Revised: (Gelfand formula) (Bhatia, 2013)

Original: Given any matrix norm ∥| · |∥ , then ρ ( A ) =
Revised: Given any matrix norm ∥| · |∥ , then ρ ( A ) =

Original: lim k

Original: [Equation]
Revised: [Equation]

Original: ∥| A k |∥ 1 /k = inf k ≥ 1 ∥|

Original: A k |∥ 1 /k ≤ ∥| A |∥ .

Original: Lemma B.1 describes the relationship between the spectral radius of a matrix and its matrix norm, i.e. ρ ( A ) =
Revised: Lemma B.1 describes the relationship between the spectral radius of a matrix and its matrix norm, i.e. ρ ( A ) =

Original: lim k →∞ ∥| A k |∥ 1 /k .
Revised: lim k →∞ ∥| A k |∥ 1 /k .

Original: Lemma B.2.
Revised: Lemma B.2.

Original: Let A ∈ C n × n , the spectral radius ρ ( A ) =
Revised: Let A ∈ C n × n , the spectral radius ρ ( A ) =

Original: max(abs(spec( A ))) , if ρ ( A ) < 1 , then (cid:80) ∞ k =

Revised: max(abs(spec( A ))) , if ρ ( A ) < 1 , then (cid:80) ∞ k =

Original: A k converges to ( I − A ) − 1 .
Revised: A k converges to ( I − A ) − 1 .

Original: Proof.
Revised: Proof.

Original: We first prove that ( I − A ) − 1 exists as follows:
Revised: We first prove that ( I − A ) − 1 exists as follows:

Original: Based on the definition of eigenvalues of
Revised: Based on the definition of eigenvalues of

Original: A , we have | λ
Revised: A , we have | λ

Original: I − A | = 0 and the solution is the eigenvalue of A .
Revised: I − A | = 0 and the solution is the eigenvalue of A .

Original: Since ρ ( A ) < 1 , if λ ≥ 1 , then | λ
Revised: Since ρ ( A ) < 1 , if λ ≥ 1 , then | λ

Original: I − A | ̸ = 0 , so | I − A | ̸ = 0 , which means ( I − A ) − 1 exists.
Revised: I − A | ̸ = 0 , so | I − A | ̸ = 0 , which means ( I − A ) − 1 exists.

Original: Since ρ ( A ) < 1 and by Lemma B.1, we have lim k →∞ ∥| A k |∥ = ρ ( A ) k = 0 .
Revised: Since ρ ( A ) < 1 and by Lemma B.1, we have lim k →∞ ∥| A k |∥ = ρ ( A ) k = 0 .

Original: Let S k
Revised: Let S k

Original: =
Revised: =

Original: A 0 + A 1 + · · · + A k , then we have
Revised: A 0 + A 1 + · · · + A k , then we have

Original: [Equation]
Revised: [Equation]

Original: Since ( I − A ) − 1 exists, so we have ( I − A )
Revised: Since ( I − A ) − 1 exists, so we have ( I − A )

Original: lim k →∞ S k
Revised: lim k →∞ S k

Original: = I , and lim k →∞ S k
Revised: = I , and lim k →∞ S k

Original: = ( I − A ) − 1 , which finishes the proof.
Revised: = ( I − A ) − 1 , which finishes the proof.

Original: Lemma B.2 describes the convergence of Neumann Series and the condition to get the convergence.
Revised: Lemma B.2 describes the convergence of Neumann Series and the condition to get the convergence.

Original: Lemma B.3.
Revised: Lemma B.3.

Original: (Gerschgorin Disc)
Revised: (Gerschgorin Disc)

Original: (Bhatia, 2013) Let A ∈ C n × n , with entries a ij .
Revised: (Bhatia, 2013) Let A ∈ C n × n , with entries a ij .

Original: For any eigenvalue λ , there exits i and the corresponding Gerschgorin disc D ( a ii , R i ) ⊆
Revised: For any eigenvalue λ , there exits i and the corresponding Gerschgorin disc D ( a ii , R i ) ⊆

Original: C such that λ lies in this disc, i.e.
Revised: C such that λ lies in this disc, i.e.

Original: [Equation]
Revised: | λ − a ii | ≤ n (cid:88) j ̸ = i | a ij | .

Original: Lemma B.3 describes the estimated range of eigenvalues.
Revised: Lemma B.3 describes the estimated range of eigenvalues.

Original: Now we start to derive the Neumann Series expansion of the solution of GSD as follows.
Revised: Now we start to derive the Neumann Series expansion of the solution of GSD as follows.

Original: Lemma B.4.
Revised: Lemma B.4.

Original: Let A ∈ { 0 , 1 } n × n be the adjacency matrix of a graph and (cid:101)
Revised: Let A ∈ { 0 , 1 } n × n be the adjacency matrix of a graph and (cid:101)

Original: A = (cid:101) D − 12 (cid:101)
Revised: A = (cid:101) D − 12 (cid:101)

Original: A (cid:101)
Revised: A (cid:101)

Original: D − 12 or (cid:101)
Revised: D − 12 or (cid:101)

Original: A = (cid:101) D − 1 (cid:101) A , then 
Revised: A = (cid:101) D − 1 (cid:101) A , then ( I −

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We first prove that ρ ( (cid:101) A ) ≤ 1 where (cid:101)
Revised: We first prove that ρ ( (cid:101) A ) ≤ 1 where (cid:101)

Original: A = (cid:101) D − 12 (cid:101)
Revised: A = (cid:101) D − 1 2 (cid:101)

Original: A (cid:101) D − 12 .
Revised: A (cid:101) D − 1 2 .

Original: Let λ be the eigenvalue of (cid:101) A , and v be the corresponding eigenvector.
Revised: Let λ be the eigenvalue of (cid:101) A , and v be the corresponding eigenvector.

Original: Then we have
Revised: Then we have

Revised: [Equation]

Revised: =

Original: [Equation]
Revised: [Equation]

Original: , which means ( λ, (cid:101) D − 12 v ) is the eigen-pair of (cid:101) D − 1 A .
Revised: which means ( λ, (cid:101) D − 12 v ) is the eigen-pair of (cid:101) D − 1 A .

Original: By Lemma B.3, there exists i , such that
Revised: By Lemma B.3, there exists i , such that

Original: [Equation]
Revised: [Equation]

Original: = 
Revised: = ⇒ [Equation] j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16)

Original: [Equation]
Revised: [Equation]

Original:  Since (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij > 0 and (cid:80) j (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) =
Revised: D [Equation] (cid:12)(cid:12)(cid:12)(cid:12) . Since (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij > 0 and (cid:80) j (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) =

Original: (cid:80) j (cid:16) (cid:101)
Revised: (cid:80) j (cid:16) (cid:101)

Original: D − 1 (cid:101) A (cid:17) ij = 1 , obviously − 1 < (cid:16) (cid:101) D − 1 (cid:101)
Revised: D − 1 (cid:101) A (cid:17) ij = 1 , obviously − 1 < (cid:16) (cid:101) D − 1 (cid:101)

Original: A (cid:17) ii − (cid:88)
Revised: A (cid:17) ii − (cid:88)

Original: j ̸ = i [Table]
Revised: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) ≤ λ ≤ (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ii + (cid:88)

Original:  = 1 .
Revised: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) = 1 .

Original: So if (cid:101)
Revised: So if (cid:101)

Original: A = (cid:101) D − 1 2 (cid:101)
Revised: A = (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 1 2 , we have ρ ( (cid:101) A ) ≤ 1 .
Revised: A (cid:101) D − 12 , we have ρ ( (cid:101) A ) ≤ 1 .

Original: When (cid:101)
Revised: When (cid:101)

Original: A = (cid:101) D − 1 (cid:101)
Revised: A = (cid:101) D − 1 (cid:101)

Original: A , we denote ( λ, v ) as the eigen-pair of (cid:101) D − 1 A .
Revised: A , we denote ( λ, v ) as the eigen-pair of (cid:101) D − 1 A .

Original: Similarly, by Lemma B.3, there exists i , such that
Revised: Similarly, by Lemma B.3, there exists i , such that

Original: [Equation]
Revised: [Equation]

Original: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16)  (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) =
Revised: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) [Equation] (cid:101) [Equation]

Revised: (cid:12)(cid:12)(cid:12)(cid:12)

Revised: =

Original: ⇒ (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ii − (cid:88)
Revised: ⇒ (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ii − (cid:88)

Original: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) ≤ λ ≤ (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ii + (cid:88)
Revised: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) ≤ λ ≤ (cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ii + (cid:88)

Original: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) .
Revised: j ̸ = i (cid:12)(cid:12)(cid:12)(cid:12)(cid:16) (cid:101) D − 1 (cid:101) A (cid:17) ij (cid:12)(cid:12)(cid:12)(cid:12) .

Original: Obviously, we can get the same conclusion for (cid:101)
Revised: Obviously, we can get the same conclusion for (cid:101)

Original: A = (cid:101) D − 1 (cid:101) A .
Revised: A = (cid:101) D − 1 (cid:101) A .

Original: So it is true for ρ (cid:16) λλ +1 (cid:101)
Revised: So it is true for ρ (cid:16) λλ +1 (cid:101)

Original: A (cid:17) ≤ λλ +1 <
Revised: A (cid:17) ≤ λλ +1 <

Original: By Lemma B.2, we get the result ( I − λ λ +1 (cid:101){+(cid:
 )+} A ) − 1 = (cid:80) ∞k
Revised: By Lemma B.2, we get the result ( I − λλ +1 {+(cid:
 )+} A ) − 1 = (cid:80) ∞

Original:  =0 (cid:16) λ λ +1 (cid:101) A (cid:17) k , which finishes the proof.
Revised: k =0 (cid:16) λλ +1 (cid:101) A (cid:17) k , which finishes the proof.

Original: By Lemma B.4, we approximate the inverse matrix ( I + λ ˜ L ) − 1 up to S -th order with (cid:16) I + λ (cid:101) L (cid:17) − 1 = λ
Revised: By Lemma B.4, we approximate the inverse matrix ( I + λ ˜ L ) − 1 up to S -th order with (cid:16) I + λ (cid:101) L (cid:17) − 1 = λ

Original: +(cid:18)
Revised: +(cid:18)

Original: I − λ λ
Revised: I − λ λ

Original: + 1 (cid:101) A (cid:19) −≈ λ
Revised: + 1 (cid:101) A (cid:19) −≈ λ

Original: +
Revised: +

Original: S (cid:88) s
Revised: S (cid:88) s

Original: = (cid:18) λ λ
Revised: = (cid:18) λ λ

Original: + 1 (cid:101)
Revised: + 1 (cid:101)

Original: A (cid:19) s .
Revised: A (cid:19) s .

Original: C T HE R OW S UMMATION OF THE N EUMANN S ERIES
Revised: C T HE R OW S UMMATION OF THE N EUMANN S ERIES

Original: We provide the derivations of the row sum of (cid:101)
Revised: We provide the derivations of the row sum of (cid:101)

Original: A S in this section.
Revised: A S in this section.

Original: Before we derive the row summation of (cid:101)
Revised: Before we derive the row summation of (cid:101)

Original: A S , we first derive the row summation of (cid:101)
Revised: A S , we first derive the row summation of (cid:101)

Original: A k .
Revised: A k .

Original: Lemma C.1.
Revised: Lemma C.1.

Original: Consider a probability matrix P ∈ R n × n , where P ij ≥ 0 .
Revised: Consider a probability matrix P ∈ R n × n , where P ij ≥ 0 .

Original: Besides, for all i , we have (cid:80) nj =1 P ij = 1 .
Revised: Besides, for all i , we have (cid:80) nj =1 P ij = 1 .

Original: Then for any s ∈ Z + , we have (cid:80) nj =1 P sij = 1 ,
Revised: Then for any s ∈ Z + , we have (cid:80) nj =1 P sij = 1 ,

Original: Proof.
Revised: Proof.

Original: We give a proof by induction on k .
Revised: We give a proof by induction on k .

Original: Base case: When k = 1 , the case is true.
Revised: Base case: When k = 1 , the case is true.

Original: Inductive step: Assume the induction hypothesis that for a particular k , the single case n = k holds, meaning P k is true: ∀ i, n (cid:88)
Revised: Inductive step: Assume the induction hypothesis that for a particular k , the single case n = k holds, meaning P k is true: ∀ i, n (cid:88)

Original: j =
Revised: j =

Original: P kij = 1 .
Revised: P kij = 1 .

Original: As P k +1 =
Revised: As P k +1 =

Original: P k P , so we have n (cid:88)
Revised: P k P , so we have n (cid:88)

Original: j =
Revised: j =

Original: P k +1 ij = n (cid:88)
Revised: P k +1 ij = n (cid:88)

Original: j = n (cid:88) k
Revised: j = n (cid:88) k

Original: =
Revised: =

Original: P kik P kj = n (cid:88) k
Revised: P kik P kj = n (cid:88) k

Original: = n (cid:88) j =
Revised: = n (cid:88) j =

Original: P kik P kj = n (cid:88) k
Revised: P kik P kj = n (cid:88) k

Original: =
Revised: =

Original: P kik   n (cid:88)
Revised: P kik   n (cid:88)

Original: j =
Revised: j =

Original: P kj 
Revised: P kj 

Original:  = n (cid:88) k
Revised:  = n (cid:88) k

Original: =
Revised: =

Original: P kik = 1 , which finishes the proof.
Revised: P kik = 1 , which finishes the proof.

Original: Lemma C.1 describes the row summation of (cid:101)
Revised: Lemma C.1 describes the row summation of (cid:101)

Original: A k is 1.
Revised: A k is 1.

Original: Now we can obtain the row summation for (cid:101) A S .
Revised: Now we can obtain the row summation for (cid:101) A S .

Original: Then for any i , we have n (cid:88)
Revised: Then for any i , we have n (cid:88)

Original: j = (cid:104) (cid:101)
Revised: j = (cid:104) (cid:101)

Original: A S (cid:105) ij = λ +
Revised: A S (cid:105) ij = λ +

Original: S (cid:88) s
Revised: S (cid:88) s

Original: = (cid:18)
Revised: = (cid:18)

Original: λ λ +(cid:104) (cid:101)
Revised: λ λ +(cid:104) (cid:101)

Original: [Equation]
Revised: [Equation]

Original: D P ROOF OF L EMMA
Revised: D P ROOF OF L EMMA

Original: We provide the details of proof of Lemma 1.
Revised: We provide the details of proof of Lemma 1.

Original: We first introduce the General Hoeffding Inequality (Hoeffding, 1994), which is essential for bounding (cid:13) (cid:13)(cid:13) (cid:101)
Revised: We first introduce the General Hoeffding Inequality (Hoeffding, 1994), which is essential for bounding (cid:13) (cid:13)(cid:13) (cid:101)

Original: A S η (cid:13) (cid:13)(cid:13) 2 F .
Revised: A S η (cid:13) (cid:13)(cid:13) 2 F .

Original: Lemma D.1.
Revised: Lemma D.1.

Original: (General Hoeffding Inequality (Hoeffding, 1994)) Suppose that the variables
Revised: (General Hoeffding Inequality (Hoeffding, 1994)) Suppose that the variables

Original: X 1 , · · · , X n are independent, and X i has mean µ i and sub-Gaussian parameter σ i .
Revised: X 1 , · · · , X n are independent, and X i has mean µ i and sub-Gaussian parameter σ i .

Original: Then for all t ≥ 0 , we have
Revised: Then for all t ≥ 0 , we have

Original: [Equation]
Revised: [Equation]

Original: Now let’s prove Lemma 1.
Revised: Now let’s prove Lemma 1.

Original: Proof of Lemma 1.
Revised: Proof of Lemma 1.

Original: For any entry (cid:104) (cid:101)
Revised: For any entry (cid:104) (cid:101)

Original: A S η (cid:105) ij = (cid:80)
Revised: A S η (cid:105) ij = (cid:80)

Original: np =1 (cid:16) (cid:101)
Revised: np =1 (cid:16) (cid:101)

Original: A S (cid:17) ip η pj , where η pj is a sub-Gaussian variable with parameter σ 2 .
Revised: A S (cid:17) ip η pj , where η pj is a sub-Gaussian variable with parameter σ 2 .

Original: By the General Hoeffding inequality D.1, we have
Revised: By the General Hoeffding inequality D.1, we have

Original: P   (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:34) λ +
Revised: P   (cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:12)(cid:34) λ +

Original: S (cid:88)
Revised: S (cid:88)

Original: [Equation]
Revised: [Equation]

Original: (cid:18)
Revised: (cid:18)

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: η (cid:35) ij
Revised: η (cid:35) ij

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: where τ = max i τ i and τ i = n (cid:80) nj =1 (cid:104)
Revised: where τ = max i τ i and τ i = n (cid:80) nj =1 (cid:104)

Original: [Equation]
Revised: [Equation]

Original: ij
Revised: ij

Original: [Equation]
Revised: [Equation]

Original: Applying union bound (Vershynin, 2010) to all possible pairs of i ∈ [ n ] , j ∈ [ n ] , we get
Revised: Applying union bound (Vershynin, 2010) to all possible pairs of i ∈ [ n ] , j ∈ [ n ] , we get

Original: [Equation]
Revised: [Equation]

Original: (cid:101)
Revised: (cid:101)

Original: [Equation]
Revised: [Equation]

Original: (cid:19)
Revised: (cid:19)

Original: [Equation]
Revised: [Equation]

Original: Applying union bound again, we have
Revised: Applying union bound again, we have

Original: [Equation]
Revised: [Equation]

Original: Choose t = 2 τ
Revised: Choose t = 2 τ

Original: [Equation]
Revised: [Equation]

Original: 2 (4 log n + log 2 d ) /n
Revised: 2 (4 log n + log 2 d ) /n

Original: and with probability 1 − 1 /d , we have
Revised: and with probability 1 − 1 /d , we have

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: which finishes the proof.
Revised: which finishes the proof.

Original: E P ROOF OF THE M AIN T HEOREM
Revised: E P ROOF OF THE M AIN T HEOREM

Original: We provide the details of proof of main theorem 1.
Revised: We provide the details of proof of main theorem 1.

Original: [Restatement of Theorem 1] Under Assumptions 1,2,3,4, let W ( k ) f denote the k -th step gradient descent solution for min W f ( W ) with step size α ≤ 1 /L , with probability 1 − 1 /d
Revised: [Restatement of Theorem 1] Under Assumptions 1,2,3,4, let W ( k ) f denote the k -th step gradient descent solution for min W f ( W ) with step size α ≤ 1 /L , with probability 1 − 1 /d

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: where W ∗ g = arg
Revised: where W ∗ g = arg

Original: min W g ( W ) is the optimal solution of the clean loss function g ( W ) , τ is the high-order graph connectivity factor, and n is the number of nodes of a graph.
Revised: min W g ( W ) is the optimal solution of the clean loss function g ( W ) , τ is the high-order graph connectivity factor [Equation] and n is the number of nodes of a graph.

Original: Proof.
Revised: Proof.

Original: By the definition of L -smooth, we can obtain the following inequality:
Revised: By the definition of L -smooth [Equation] we can obtain the following inequality:

Original: [Equation]
Revised: [Equation]

Original: Let’s use the gradient descent algorithm with W ′ f
Revised: Let’s use the gradient descent algorithm with W ′ f

Original: = W + f = W f − α ∇ f ( W f ) .
Revised: = W + f = W f − α ∇ f ( W f ) .

Original: We then get:
Revised: We then get:

Original: [Equation]
Revised: [Equation]

Original: With the fixed step size α ≤ 1 /L
Revised: With the fixed step size α ≤ 1 /L

Original: [Equation]
Revised: [Equation]

Original: we know that − (1 − 12 Lα )
Revised: we know that − (1 − 12 Lα )

Original: = 12 Lα − 1 ≤ 12 L (1 /L )
Revised: = 12 Lα − 1 ≤ 12 L (1 /L )

Original: − 1 = 12 − 1 = − 12 .
Revised: − 1 = 12 − 1 = − 12 .

Original: Plugging this into Eq.
Revised: Plugging this into Eq.

Original: (27), we have the following inequality:
Revised: (27), we have the following inequality:

Original: [Equation]
Revised: [Equation]

Original: If we choose t to be small enough such that t ≤ 1 /L
Revised: If we choose t to be small enough such that t ≤ 1 /L

Original: [Equation]
Revised: [Equation]

Original: this inequality implies that the loss function value strictly decreases under each iteration of gradient descent since ∥∇ f ( W f ) ∥ is positive unless ∇ f ( W f ) = 

Revised: this inequality implies that the loss function value strictly decreases under each iteration of gradient descent since ∥∇ f ( W f ) ∥ is positive unless ∇ f ( W f ) =

Original: e.g. W f = W ∗ f , where W f reaches W ∗ f .
Revised: e.g. W f = W ∗ f , where W f reaches W ∗ f .

Original: Now, let’s bound the loss function value f ( W + f ) .
Revised: Now, let’s bound the loss function value f ( W + f ) .

Original: Since f is convex
Revised: Since f is convex

Original: [Equation]
Revised: [Equation]

Original: we can write
Revised: we can write

Original: [Equation]
Revised: [Equation]

Original: Introducing this inequality into Eq.
Revised: Introducing this inequality into Eq.

Original: [Equation]
Revised: [Equation]

Original: we can obtain the following:
Revised: we can obtain the following:

Original: [Equation]
Revised: [Equation]

Original: Notice that by the definition of gradient descent update
Revised: Notice that by the definition of gradient descent update

Original: [Equation]
Revised: [Equation]

Original: we have W + f =
Revised: we have W + f =

Original: W f − α ∇ f ( W f ) .
Revised: W f − α ∇ f ( W f ) .

Original: Plugging this into the final inequality of Eq.
Revised: Plugging this into the final inequality of Eq.

Original: [Equation]
Revised: [Equation]

Original: we can get:
Revised: we can get:

Original: [Equation]
Revised: [Equation]

Original: This inequality holds for W + f on every iteration of gradient descent.
Revised: This inequality holds for W + f on every iteration of gradient descent.

Original: Summing over iterations, we get:
Revised: Summing over iterations, we get:

Original: [Equation]
Revised: [Equation]

Original: With the inequality of Eq.
Revised: With the inequality of Eq.

Original: [Equation]
Revised: [Equation]

Original: we know that f ( W f ) strictly decreases over each iteration.
Revised: we know that f ( W f ) strictly decreases over each iteration.

Original: So we have following:
Revised: So we have following:

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: we have
Revised: we have

Original: [Table]
Revised: [Table]

Original: function
Revised: function

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: which finishes the proof.
Revised: which finishes the proof.

Original: F M ORE D ETAILS ON E QUATION (1).
Revised: F M ORE D ETAILS ON E QUATION (1).

Original: We provide more details on how to obtain Equation (1).
Revised: We provide more details on how to obtain Equation (1).

Original: Note that if we set (cid:101)
Revised: Note that if we set (cid:101)

Original: L = I − (cid:101) D − 12 (cid:101)
Revised: L = I − (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 12 (cid:101)
Revised: A (cid:101) D − 12 , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 ) F (cid:17) = tr (cid:0) F ⊤ F (cid:1) − tr (cid:16) F ⊤ (cid:101) D − 12 (cid:101)
Revised: A (cid:101) D − 12 ) F (cid:17) = tr (cid:0) F ⊤ F (cid:1) − tr (cid:16) F ⊤ (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 F (cid:17) = tr (cid:0) FF ⊤ (cid:1) − tr (cid:16) (cid:101)
Revised: A (cid:101) D − 12 F (cid:17) = tr (cid:0) FF ⊤ (cid:1) − tr (cid:16) (cid:101)

Original: D − 12 (cid:101)
Revised: D − 12 (cid:101)

Original: A (cid:101) D − 12 FF ⊤ (cid:17) .
Revised: A (cid:101) D − 12 FF ⊤ (cid:17) .

Original: On the other hand, if we set (cid:101) L = I − (cid:101) D − 1 (cid:101)
Revised: On the other hand, if we set (cid:101) L = I − (cid:101) D − 1 (cid:101)

Original: A , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 1 (cid:101) A ) F (cid:17) = tr (cid:0) F ⊤ F (cid:1) − tr (cid:16) F ⊤ (cid:101) D − 1 (cid:101) AF (cid:17) = tr (cid:0) FF ⊤ (cid:1) − tr (cid:16) (cid:101) D − 1 (cid:101) AFF ⊤ (cid:17) .
Revised: A , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 1 (cid:101) A ) F (cid:17) = tr (cid:0) F ⊤ F (cid:1) − tr (cid:16) F ⊤ (cid:101) D − 1 (cid:101) AF (cid:17) = tr (cid:0) FF ⊤ (cid:1) − tr (cid:16) (cid:101) D − 1 (cid:101) AFF ⊤ (cid:17) .

Original: We denote F =  
Revised: We denote F =  

Original: F 1 .
Revised: F 1 .

Original: F n   and F ⊤ = (cid:2) F ⊤ 1 · · · F ⊤ n (cid:3) , where F i =
Revised: F n   and F ⊤ = (cid:2) F ⊤ 1 · · · F ⊤ n (cid:3) , where F i =

Original: [ F i 1 · · · F id ] , then we have tr (cid:0) FF ⊤ (cid:1) =
Revised: [ F i 1 · · · F id ] , then we have tr (cid:0) FF ⊤ (cid:1) =

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 F
Revised: =1 F

Original: i F ⊤ i .
Revised: i F ⊤ i .

Original: When (cid:101) L
Revised: When (cid:101) L

Original: = I − (cid:101) D − 12 (cid:101)
Revised: = I − (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 , we have tr (cid:16) (cid:101) D − 12 (cid:101)
Revised: A (cid:101) D − 12 , we have tr (cid:16) (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 FF ⊤ (cid:17) =
Revised: A (cid:101) D − 12 FF ⊤ (cid:17) =

Original: tr        
Revised: tr        

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: A n
Revised: A n

Original: [Table]
Revised: [Table]

Original: A
Revised: A

Original: [Table]
Revised: [Table]

Original: 
Revised: 

Original:       
Revised:       

Original: F 1 F ⊤
Revised: F 1 F ⊤

Original: F 1 F ⊤· · ·
Revised: F 1 F ⊤· · ·

Original: F 1 F ⊤ n
Revised: F 1 F ⊤ n

Original: F 2 F ⊤
Revised: F 2 F ⊤

Original: F 2 F ⊤· · ·
Revised: F 2 F ⊤· · ·

Original: F 2 F ⊤ n .
Revised: F 2 F ⊤ n .

Original: F n F ⊤
Revised: F n F ⊤

Original: F n F ⊤· · ·
Revised: F n F ⊤· · ·

Original: F n F ⊤ n         = n (cid:88) i
Revised: F n F ⊤ n         = n (cid:88) i

Original: = n (cid:88)
Revised: = n (cid:88)

Original: j =
Revised: j =

Original: A ij √ d
Revised: A ij √ d

Original: i + 1 (cid:112) d j + 1 F j F ⊤ i .
Revised: i + 1 (cid:112) d j + 1 F j F ⊤ i .

Original: On the other hand, when (cid:101) L = I − (cid:101) D − 1 (cid:101)
Revised: On the other hand, when (cid:101) L = I − (cid:101) D − 1 (cid:101)

Original: A , we have tr (cid:16) (cid:101)
Revised: A , we have tr (cid:16) (cid:101)

Original: D − 1 (cid:101)
Revised: D − 1 (cid:101)

Original: AFF ⊤ (cid:17) =
Revised: AFF ⊤ (cid:17) =

Original: tr    
Revised: tr    

Original: A 11 d 1 +
Revised: A 11 d 1 +

Original: A 12 d 1 + · · ·
Revised: A 12 d 1 + · · ·

Original: A 1 n d 1 +1 Ad 2 +
Revised: A 1 n d 1 +1 Ad 2 +

Original: A 22 d 2 + · · ·
Revised: A 22 d 2 + · · ·

Original: A 2 n d 2 +1 .
Revised: A 2 n d 2 +1 .

Original: A n 1 d n +
Revised: A n 1 d n +

Original: A n 2 d n + · · ·
Revised: A n 2 d n + · · ·

Original: A nn d n +    
Revised: A nn d n +    

Original: F 1 F ⊤
Revised: F 1 F ⊤

Original: F 1 F ⊤· · ·
Revised: F 1 F ⊤· · ·

Original: F 1 F ⊤ n
Revised: F 1 F ⊤ n

Original: F 2 F ⊤
Revised: F 2 F ⊤

Original: F 2 F ⊤· · ·
Revised: F 2 F ⊤· · ·

Original: F 2 F ⊤ n .
Revised: F 2 F ⊤ n .

Original: F n F ⊤
Revised: F n F ⊤

Original: F n F ⊤· · ·
Revised: F n F ⊤· · ·

Original: F n F ⊤ n     = n (cid:88) i = n (cid:88) j
Revised: F n F ⊤ n     = n (cid:88) i = n (cid:88) j

Original: =
Revised: =

Original: A ij d i + 1 F j F ⊤ i .
Revised: A ij d i + 1 F j F ⊤ i .

Original: So when (cid:101)
Revised: So when (cid:101)

Original: L = I − (cid:101) D − 12 (cid:101)
Revised: L = I − (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) (cid:16)(cid:101)
Revised: A (cid:101) D − 12 , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) (cid:16)(cid:101)

Original: L = I − (cid:101) D − 12 (cid:101)
Revised: L = I − (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 12 (cid:101)
Revised: A (cid:101) D − 12 (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 12 (cid:101)

Original: A (cid:101) D − 12 ) F (cid:17) = tr (cid:0) FF ⊤ (cid:1)
Revised: A (cid:101) D − 12 ) F (cid:17) = tr (cid:0) FF ⊤ (cid:1)

Original: [Equation]
Revised: [Equation]

Original: (cid:16) (cid:101) D − 12 (cid:101)
Revised: (cid:16) (cid:101) D − 12 (cid:101)

Original: A (cid:101) D
Revised: A (cid:101) D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:17) =
Revised: (cid:17) =

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i =
Revised: (cid:88) i =

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: (cid:88) i = n (cid:88)
Revised: (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: = n (cid:88)
Revised: = n (cid:88)

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: =  
Revised: =  

Original: [Equation]
Revised: [Equation]

Original: (cid:88)
Revised: (cid:88)

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: A
Revised: A

Original: [Equation]
Revised: [Equation]

Original: √
Revised: √

Original: [Equation]
Revised: [Equation]

Original: F
Revised: F

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original:  =   n (cid:88) i = n (cid:88)
Revised:  =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij F i F
Revised: A ij F i F

Original: ⊤
Revised: ⊤

Original: i
Revised: i

Original: d i ++ n (cid:88) i = n (cid:88)
Revised: d i ++ n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij F j F ⊤ j d j +−n (cid:88) i = n (cid:88)
Revised: A ij F j F ⊤ j d j +−n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij √ d
Revised: A ij √ d

Original: i + 1 (cid:112) d j + 1 F j F ⊤ i 
Revised: i + 1 (cid:112) d j + 1 F j F ⊤ i 

Original:  undirected graph =   n (cid:88) i = n (cid:88)
Revised:  undirected graph =   n (cid:88) i = n (cid:88)

Original: j = (cid:32)
Revised: j = (cid:32)

Original: A ij F i F
Revised: A ij F i F

Original: ⊤
Revised: ⊤

Original: i
Revised: i

Original: d i ++ A ij F j F ⊤ j d j +−
Revised: d i ++ A ij F j F ⊤ j d j +−

Original: A ij √ d
Revised: A ij √ d

Original: i + 1 (cid:112) d j + 1 F j F ⊤ i −
Revised: i + 1 (cid:112) d j + 1 F j F ⊤ i −

Original: A ij √ d
Revised: A ij √ d

Original: i + 1 (cid:112) d j + 1 F i F ⊤ j (cid:33)  =   n (cid:88) i = n (cid:88)
Revised: i + 1 (cid:112) d j + 1 F i F ⊤ j (cid:33)  =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij (cid:32) F i F ⊤ i
Revised: A ij (cid:32) F i F ⊤ i

Original: d i + 1 +
Revised: d i + 1 +

Original: F j F ⊤ j d j + 1 −
Revised: F j F ⊤ j d j + 1 −

Original: F j F ⊤ i √
Revised: F j F ⊤ i √

Original: d i + 1 (cid:112) d j + 1 −
Revised: d i + 1 (cid:112) d j + 1 −

Original: F i F ⊤ j √ d
Revised: F i F ⊤ j √ d

Original: i + 1 (cid:112) d j +(cid:33)  =   n (cid:88) i = n (cid:88)
Revised: i + 1 (cid:112) d j +(cid:33)  =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij (cid:32)
Revised: A ij (cid:32)

Original: F i √
Revised: F i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F j (cid:112) d j +(cid:33) (cid:32)
Revised: F j (cid:112) d j +(cid:33) (cid:32)

Original: F ⊤ i √
Revised: F ⊤ i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F ⊤ j (cid:112) d j +(cid:33)  =   n (cid:88) i = n (cid:88)
Revised: F ⊤ j (cid:112) d j +(cid:33)  =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: A ij (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: F i √
Revised: F i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F j (cid:112) d j + 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  
Revised: F j (cid:112) d j + 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  

Original:  =
Revised:  =

Original: (cid:88)
Revised: (cid:88)

Original: ( i,j ) ∈E
Revised: ( i,j ) ∈E

Original: A ij (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)
Revised: A ij (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)

Original: F i √
Revised: F i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F j (cid:112) d j + 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  .
Revised: F j (cid:112) d j + 1 (cid:13)(cid:13)(cid:13)(cid:13)(cid:13)  .

Original: On the other hand, when (cid:101) L = I − (cid:101) D − 1 (cid:101)
Revised: On the other hand, when (cid:101) L = I − (cid:101) D − 1 (cid:101)

Original: A , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) (cid:16)(cid:101) L
Revised: A , we have tr (cid:16) F ⊤ (cid:101) LF (cid:17) (cid:16)(cid:101) L

Original: = I − (cid:101) D − 1 (cid:101) A (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 1 (cid:101) A ) F (cid:17) = n (cid:88) i =
Revised: = I − (cid:101) D − 1 (cid:101) A (cid:17) = tr (cid:16) F ⊤ ( I − (cid:101) D − 1 (cid:101) A ) F (cid:17) = n (cid:88) i =

Original: F i F ⊤ i − n (cid:88) i = n (cid:88) j
Revised: F i F ⊤ i − n (cid:88) i = n (cid:88) j

Original: =
Revised: =

Original: A ij d i + 1 F j F ⊤ i = n (cid:88) i =
Revised: A ij d i + 1 F j F ⊤ i = n (cid:88) i =

Original: F i F ⊤ i + 1n (cid:88)
Revised: F i F ⊤ i + 1n (cid:88)

Original: j =
Revised: j =

Original: F j F ⊤ j − n (cid:88) i = n (cid:88)
Revised: F j F ⊤ j − n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij d i + 1 F j F ⊤ i =   n (cid:88) i = n (cid:88)
Revised: A ij d i + 1 F j F ⊤ i =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij F i F
Revised: A ij F i F

Original: ⊤
Revised: ⊤

Original: i
Revised: i

Original: d i ++ n (cid:88) i = n (cid:88)
Revised: d i ++ n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij F j F ⊤ j d i +−n (cid:88) i = n (cid:88)
Revised: A ij F j F ⊤ j d i +−n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij √ d
Revised: A ij √ d

Original: i + 1 √ d i + 1 F j F ⊤ i 
Revised: i + 1 √ d i + 1 F j F ⊤ i 

Original:  undirected graph =   n (cid:88) i = n (cid:88)
Revised:  undirected graph =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij (cid:18)
Revised: A ij (cid:18)

Original: F i √
Revised: F i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F j √ d i +(cid:19) (cid:32)
Revised: F j √ d i +(cid:19) (cid:32)

Original: F ⊤ i √
Revised: F ⊤ i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F ⊤ j √ d
Revised: F ⊤ j √ d

Original: i +(cid:33)  =   n (cid:88) i = n (cid:88)
Revised: i +(cid:33)  =   n (cid:88) i = n (cid:88)

Original: j =
Revised: j =

Original: A ij (cid:13)(cid:13)(cid:13)(cid:13)
Revised: A ij (cid:13)(cid:13)(cid:13)(cid:13)

Original: F i √
Revised: F i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F j √ d i + 1 (cid:13)(cid:13)(cid:13)(cid:13)  
Revised: F j √ d i + 1 (cid:13)(cid:13)(cid:13)(cid:13)  

Original:  =
Revised:  =

Original: (cid:88)
Revised: (cid:88)

Original: ( i,j ) ∈E
Revised: ( i,j ) ∈E

Original: A ij (cid:13)(cid:13)(cid:13)(cid:13)
Revised: A ij (cid:13)(cid:13)(cid:13)(cid:13)

Original: F i √
Revised: F i √

Original: d i + 1 −
Revised: d i + 1 −

Original: F j √ d i + 1 (cid:13)(cid:13)(cid:13)(cid:13)  .
Revised: F j √ d i + 1 (cid:13)(cid:13)(cid:13)(cid:13)  .

Original: G D ATASETS D ETAILS
Revised: G D ATASETS D ETAILS

Original: Cora, Citeseer, and Pubmed are standard citation network benchmark datasets (Sen et al., 2008).
Revised: Cora, Citeseer, and Pubmed are standard citation network benchmark datasets (Sen et al., 2008).

Original: Coauthor-CS and Coauthor-Phy are extracted from Microsoft Academic Graph (Shchur et al., 2018).
Revised: Coauthor-CS and Coauthor-Phy are extracted from Microsoft Academic Graph (Shchur et al., 2018).

Original: Cornell, Texas, Wisconsin, and Actor are constructed by Pei et al.
Revised: Cornell, Texas, Wisconsin, and Actor are constructed by Pei et al.

Original: ogbn-products is a large-scale product, constructed by Hu et al.
Revised: ogbn-products is a large-scale product, constructed by Hu et al.

Original: [Table]
Revised: [Table]

Original: H R EPRODUCIBILITY H. I MPLEMENTATION D ETAILS
Revised: H R EPRODUCIBILITY H. I MPLEMENTATION D ETAILS

Original: We use Pytorch (Paszke et al., 2019) and PyG (Fey & Lenssen, 2019) to implement NGC and RNGC.
Revised: We use Pytorch (Paszke et al., 2019) and PyG (Fey & Lenssen, 2019) to implement NGC and RNGC.

Original: The codes of baselines are implemented referring to the implementation of MLP 78 , GCN 910 , GAT 11 ,
Revised: The codes of baselines are implemented referring to the implementation of MLP 78 , GCN 910 , GAT 11 ,

Original: GLP 12 , S 2 GC 13 , and IRLS 14 .
Revised: GLP 12 , S 2 GC 13 , and IRLS 14 .

Original: All the experiments in this work are conducted on a single NVIDIA
Revised: All the experiments in this work are conducted on a single NVIDIA

Original: Tesla A100 with 80GB memory size.
Revised: Tesla A100 with 80GB memory size.

Original: The software that we use for experiments are Python 3.6.8, pytorch 1.9.0, pytorch-scatter 2.0.9, pytorch-sparse 0.6.12, pyg 2.0.3, ogb 1.3.4, numpy 1.19.5, torchvision 0.10.0, and CUDA 11.1.
Revised: The software that we use for experiments are Python 3.6.8, pytorch 1.9.0, pytorch-scatter 2.0.9, pytorch-sparse 0.6.12, pyg 2.0.3, ogb 1.3.4, numpy 1.19.5, torchvision 0.10.0, and CUDA 11.1.

Original: H. H
Revised: H. H

Original: YPERPARAMETER
Revised: YPERPARAMETER

Original: D ETAILS
Revised: D ETAILS

Original: We provide details about hyparatemeters of NGC and RNGC in Table 7, 8, 9, 10, and 11.
Revised: We provide details about hyparatemeters of NGC and RNGC in Table 7, 8, 9, 10, and 11.

Original: I A DDITIONAL E XPERIMENTS I. A NALYSIS ON R OW N ORMALIZATION
Revised: I A DDITIONAL E XPERIMENTS I. A NALYSIS ON R OW N ORMALIZATION

Original: In this section, we analyze the influence of row normalization on denoising performance.
Revised: In this section, we analyze the influence of row normalization on denoising performance.

Original: The noise level ξ controls the magnitude of the Gaussian noise we add to the feature matrix: X + ξ η where η is sampled from standard i.i.d., Gaussian distribution.
Revised: The noise level ξ controls the magnitude of the Gaussian noise we add to the feature matrix: X + ξ η where η is sampled from standard i.i.d., Gaussian distribution.

Original: For Cora, Citeseer, and Pubmed, we test ξ ∈ { 1 , 10 , 100 } .
Revised: For Cora, Citeseer, and Pubmed, we test ξ ∈ { 1 , 10 , 100 } .

Original: From Table 12, we can observe that the denoising performance of w/ row normalization is better than w/o row normalization.
Revised: From Table 12, we can observe that the denoising performance of w/ row normalization is better than w/o row normalization.

Original: Since row normalization can shrink the value
Revised: Since row normalization can shrink the value

Original: [Table] [Table]
Revised: [Table] [Table]

Original: of elements in η , thus reducing the variance σ .
Revised: of elements in η , thus reducing the variance σ .

Original: In other words, row normalization
Revised: In other words, row normalization

Original: [Table]
Revised: [Table]

Original: converge to zero faster.
Revised: converge to zero faster.

Original: I. A NALYSIS ON THE D EPTH OF NGC AND RNGC
Revised: I. A NALYSIS ON THE D EPTH OF NGC AND RNGC

Original: In this section
Revised: In this section

Original: [Table]
Revised: [Table]

Original: we analyze the influence of the depth of NGC and RNGC model on denoising performance by testing the classification accuracy on semi-supervised node classification tasks.
Revised: we analyze the influence of the depth of NGC and RNGC model on denoising performance by testing the classification accuracy on semi-supervised node classification tasks.

Original: We conduct two sets of experiments: with/without noise in feature matrix.
Revised: We conduct two sets of experiments: with/without noise in feature matrix.

Original: For experiment with feature noise
Revised: For experiment with feature noise

Original: [Table]
Revised: [Table]

Original: we simple fix the noise level ξ = 1 .
Revised: we simple fix the noise level ξ = 1 .

Original: In each set of experiments
Revised: In each set of experiments

Original: [Table]
Revised: [Table]

Original: we evaluate the test accuracy with respect to NGC and RNGC model depth, which corresponding to the value of S in (cid:101) A S .
Revised: we evaluate the test accuracy with respect to NGC and RNGC model depth, which corresponding to the value of S in (cid:101) A S .

Original: From
Revised: From

Original: Figure 4 and 5, we can observe that the test accuracy barely changes with depth if the model is trained on the clean features on Cora and Pubmed but changes greatly if the model is trained on the clean feature on Citeseer.
Revised: Figure 4 and 5, we can observe that the test accuracy barely changes with depth if the model is trained on the clean features on Cora and Pubmed but changes greatly if the model is trained on the clean feature on Citeseer.

Original: In this regard, the over-smoothing issue exists in RNGC model on citeseer.
Revised: In this regard, the over-smoothing issue exists in RNGC model on citeseer.

Original: However, the denoising performance of shallow RNGC is not good as deeper RNGC models, especially on the large graph like Pubmed.
Revised: However, the denoising performance of shallow RNGC is not good as deeper RNGC models, especially on the large graph like Pubmed.

Original: This suggests that we do need to increase the depth of
Revised: This suggests that we do need to increase the depth of

Original: GNN model to include more higher-order neighbors for better denoising performances.
Revised: GNN model to include more higher-order neighbors for better denoising performances.

Revised: I. M ORE DETAILS ABOUT THE STD INFORMATION OF THE RESULTS IN T ABLE 4 AND

Original: [Table]
Revised: [Table]

Original: [Table] 
Revised: [Table] [Table] [Table] [Figure]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original:  [Table]
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Figure] [Table]

Original: [Figure] 
Revised: [Figure] [Table] [Table]