Original: A NON - ASYMPTOTIC ANALYSIS OF OVERSMOOTHING IN G RAPH N EURAL N ETWORKS
Revised: A NON - ASYMPTOTIC ANALYSIS OF OVERSMOOTHING IN G RAPH N EURAL N ETWORKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: A central challenge of building more powerful Graph Neural Networks (GNNs) is the oversmoothing phenomenon, where increasing the network depth leads to homogeneous node representations and thus worse classification performance.
Revised: A central challenge of building more powerful Graph Neural Networks (GNNs) is the oversmoothing phenomenon, where increasing the network depth leads to homogeneous node representations and thus worse classification performance.

Original: While previous works have only demonstrated that oversmoothing is inevitable when the number of graph convolutions tends to infinity, in this paper, we precisely characterize the mechanism behind the phenomenon via a non-asymptotic analysis.
Revised: While previous works have only demonstrated that oversmoothing is inevitable when the number of graph convolutions tends to infinity, in this paper, we precisely characterize the mechanism behind the phenomenon via a non-asymptotic analysis.

Original: Specifically, we distinguish between two different effects when applying graph convolutions—an undesirable mixing effect that homogenizes node representations in different classes, and a desirable denoising effect that homogenizes node representations in the same class.
Revised: Specifically, we distinguish between two different effects when applying graph convolutions—an undesirable mixing effect that homogenizes node representations in different classes, and a desirable denoising effect that homogenizes node representations in the same class.

Original: By quantifying these two effects on random graphs sampled from the Contextual Stochastic Block Model (CSBM), we show that oversmoothing happens once the mixing effect starts to dominate the denoising effect, and the number of layers required for this transition is O (log N/ log(log N )) for sufficiently dense graphs with N nodes.
Revised: By quantifying these two effects on random graphs sampled from the Contextual Stochastic Block Model (CSBM), we show that oversmoothing happens once the mixing effect starts to dominate the denoising effect, and the number of layers required for this transition is O (log N/ log(log N )) for sufficiently dense graphs with N nodes.

Original: We also extend our analysis to study the effects of Personalized PageRank (PPR) on oversmoothing.
Revised: We also extend our analysis to study the effects of Personalized PageRank (PPR) on oversmoothing.

Original: Our results suggest that while PPR mitigates oversmoothing at deeper layers, PPR-based architectures still achieve their best performance at a shallow depth and are outperformed by the graph convolution approach on certain graphs.
Revised: Our results suggest that while PPR mitigates oversmoothing at deeper layers, PPR-based architectures still achieve their best performance at a shallow depth and are outperformed by the graph convolution approach on certain graphs.

Original: Finally, we support our theoretical results with numerical experiments, which further suggest that the oversmoothing phenomenon observed in practice may be exacerbated by the difficulty of optimizing deep GNN models.
Revised: Finally, we support our theoretical results with numerical experiments, which further suggest that the oversmoothing phenomenon observed in practice may be exacerbated by the difficulty of optimizing deep GNN models.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Graph Neural Networks (GNNs) are a powerful framework for learning with graph-structured data and have shown great promise in diverse domains such as molecular chemistry, physics, and social network analysis (Gori et al., 2005;
Revised: Graph Neural Networks (GNNs) are a powerful framework for learning with graph-structured data and have shown great promise in diverse domains such as molecular chemistry, physics, and social network analysis (Gori et al., 2005;

Original: Scarselli et al., 2009; Bruna et al., 2014; Duvenaud et al., 2015;
Revised: Scarselli et al., 2009; Bruna et al., 2014; Duvenaud et al., 2015;

Original: Defferrard et al., 2016; Battaglia et al., 2016;
Revised: Defferrard et al., 2016; Battaglia et al., 2016;

Original: Li et al., 2016).
Revised: Li et al., 2016).

Original: Most GNN models are built by stacking graph convolutions or message-passing layers (Gilmer et al., 2017), where the representation of each node is computed by recursively aggregating and transforming the representations of its neighboring nodes.
Revised: Most GNN models are built by stacking graph convolutions or message-passing layers (Gilmer et al., 2017), where the representation of each node is computed by recursively aggregating and transforming the representations of its neighboring nodes.

Original: The most representative and popular example is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017), which has demonstrated success in node classification , a primary graph task which asks for node labels and identifies community structures in real graphs.
Revised: The most representative and popular example is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017), which has demonstrated success in node classification , a primary graph task which asks for node labels and identifies community structures in real graphs.

Original: Despite these achievements, the choice of depth for these GNN models remains an intriguing question.
Revised: Despite these achievements, the choice of depth for these GNN models remains an intriguing question.

Original: GNNs often achieve optimal classification performance when networks are shallow.
Revised: GNNs often achieve optimal classification performance when networks are shallow.

Original: For example, many widely used GNNs such as the GCN are no deeper than 4 layers (Kipf & Welling, 2017;
Revised: For example, many widely used GNNs such as the GCN are no deeper than 4 layers (Kipf & Welling, 2017;

Original: Wu et al., 2019).
Revised: Wu et al., 2019).

Original: It has been observed that for deeper GNNs, repeated message-passing makes node representations in different classes indistinguishable and thus leads to lower node classification accuracy, a phenomenon known as oversmoothing (Kipf & Welling, 2017;
Revised: It has been observed that for deeper GNNs, repeated message-passing makes node representations in different classes indistinguishable and thus leads to lower node classification accuracy, a phenomenon known as oversmoothing (Kipf & Welling, 2017;

Original: Li et al., 2018; Klicpera et al., 2019; Wu et al., 2019; Oono & Suzuki, 2020;
Revised: Li et al., 2018; Klicpera et al., 2019; Wu et al., 2019; Oono & Suzuki, 2020;

Original: Chen et al., 2020a;b; Keriven, 2022).
Revised: Chen et al., 2020a;b; Keriven, 2022).

Original: Through the insight that graph convolutions can be regarded as low-pass filters on graph signals, prior studies have established that oversmoothing is inevitable when the number of layers in a GNN increases to infinity (Li et al., 2018; Oono & Suzuki, 2020).
Revised: Through the insight that graph convolutions can be regarded as low-pass filters on graph signals, prior studies have established that oversmoothing is inevitable when the number of layers in a GNN increases to infinity (Li et al., 2018; Oono & Suzuki, 2020).

Original: However, these asymptotic analyses do not fully explain the rapid occurrence of oversmoothing when we increase the network depth, let alone the fact that for some datasets, having no graph convolution is even optimal (Liu et al., 2021).
Revised: However, these asymptotic analyses do not fully explain the rapid occurrence of oversmoothing when we increase the network depth, let alone the fact that for some datasets, having no graph convolution is even optimal (Liu et al., 2021).

Original: These observations motivate the following key questions about oversmoothing in GNNs:
Revised: These observations motivate the following key questions about oversmoothing in GNNs:

Original: Why
Revised: Why

Original: does oversmoothing happen at a relatively shallow depth?
Revised: does oversmoothing happen at a relatively shallow depth?

Original: Can we quantitatively model the effect of applying a finite number of graph convolutions and theoretically predict the “sweet spot” for the choice of depth?
Revised: Can we quantitatively model the effect of applying a finite number of graph convolutions and theoretically predict the “sweet spot” for the choice of depth?

Original: [Figure]
Revised: [Figure]

Original: In this paper, we propose a non-asymptotic analysis framework using a random graph model with planted community structure called the Contextual Stochastic Block Model (CSBM)
Revised: In this paper, we propose a non-asymptotic analysis framework using a random graph model with planted community structure called the Contextual Stochastic Block Model (CSBM)

Original: (Deshpande et al., 2018).
Revised: (Deshpande et al., 2018).

Original: The CSBM mimics the community structure of real graphs and enables us to evaluate the performance of linear GNNs through the probabilistic model with ground truth community labels.
Revised: The CSBM mimics the community structure of real graphs and enables us to evaluate the performance of linear GNNs through the probabilistic model with ground truth community labels.

Original: More importantly, as a generative model, the CSBM gives us full control over the graph structure and allows us to analyze the effect of graph convolutions non-asymptotically.
Revised: More importantly, as a generative model, the CSBM gives us full control over the graph structure and allows us to analyze the effect of graph convolutions non-asymptotically.

Original: Our theoretical analysis provides deeper insights into the effects of message-passing and the problem of oversmoothing, closes the gap between finite-depth GNNs and the previous asymptotic analyses, and provides an explanation for why the optimal number of GNN layers for node classification is relatively small.
Revised: Our theoretical analysis provides deeper insights into the effects of message-passing and the problem of oversmoothing, closes the gap between finite-depth GNNs and the previous asymptotic analyses, and provides an explanation for why the optimal number of GNN layers for node classification is relatively small.

Original: In particular, we distinguish between two counteracting effects of applying graph convolutions:
Revised: In particular, we distinguish between two counteracting effects of applying graph convolutions:

Original: • mixing effect (undesirable) : homogenizing node representations in different classes; • denoising effect (desirable) : homogenizing node representations in the same class.
Revised: • mixing effect (undesirable) : homogenizing node representations in different classes; • denoising effect (desirable) : homogenizing node representations in the same class.

Original: Adding graph convolutions will increase both the mixing and denoising effects.
Revised: Adding graph convolutions will increase both the mixing and denoising effects.

Original: As a result, oversmoothing happens not just because the mixing effect keeps accumulating as the depth increases, on which the asymptotic analyses are based (Li et al., 2018;
Revised: As a result, oversmoothing happens not just because the mixing effect keeps accumulating as the depth increases, on which the asymptotic analyses are based (Li et al., 2018;

Original: Oono & Suzuki, 2020), but rather because the mixing effect starts to dominate the denoising effect (see Figure 1 for a schematic illustration).
Revised: Oono & Suzuki, 2020), but rather because the mixing effect starts to dominate the denoising effect (see Figure 1 for a schematic illustration).

Original: By quantifying both effects as a function of the model depth, we show that the turning point of the tradeoff between the two effects is O (log N/ log(log N )) for graphs with N nodes sampled from the CSBM in sufficiently dense regimes.
Revised: By quantifying both effects as a function of the model depth, we show that the turning point of the tradeoff between the two effects is O (log N/ log(log N )) for graphs with N nodes sampled from the CSBM in sufficiently dense regimes.

Original: Besides new theory, this paper also presents numerical results directly comparing theoretical predictions and empirical results.
Revised: Besides new theory, this paper also presents numerical results directly comparing theoretical predictions and empirical results.

Original: This comparison leads to new insights highlighting the fact that the oversmoothing phenomenon observed in practice is often a mixture of pure oversmoothing and difficulty of optimizing weights in deep GNN models.
Revised: This comparison leads to new insights highlighting the fact that the oversmoothing phenomenon observed in practice is often a mixture of pure oversmoothing and difficulty of optimizing weights in deep GNN models.

Original: In addition, we apply our framework to analyze the effects of Personalized PageRank (PPR) on oversmoothing.
Revised: In addition, we apply our framework to analyze the effects of Personalized PageRank (PPR) on oversmoothing.

Original: Personalized propagation of neural predictions (PPNP) and its approximate variant (APPNP) make use of PPR and its approximate variant, respectively, and were proposed as a solution to mitigate oversmoothing while retaining the ability to aggregate information from larger neighborhoods in the graph (Klicpera et al., 2019).
Revised: Personalized propagation of neural predictions (PPNP) and its approximate variant (APPNP) make use of PPR and its approximate variant, respectively, and were proposed as a solution to mitigate oversmoothing while retaining the ability to aggregate information from larger neighborhoods in the graph (Klicpera et al., 2019).

Original: We show mathematically that PPR makes the model performance more robust to increasing number of layers by reducing the mixing effect at each layer, while it nonetheless reduces the desirable denoising effect at the same time.
Revised: We show mathematically that PPR makes the model performance more robust to increasing number of layers by reducing the mixing effect at each layer, while it nonetheless reduces the desirable denoising effect at the same time.

Original: For graphs with a large size or strong community structure, the reduction of the denoising effect would be greater than the reduction of the mixing effect and thus PPNP and APPNP would perform worse than the baseline GNN on those graphs.
Revised: For graphs with a large size or strong community structure, the reduction of the denoising effect would be greater than the reduction of the mixing effect and thus PPNP and APPNP would perform worse than the baseline GNN on those graphs.

Original: Our contributions are summarized as follows:
Revised: Our contributions are summarized as follows:

Original: • We show that adding graph convolutions strengthens the denoising effect while exacerbates the mixing effect.
Revised: • We show that adding graph convolutions strengthens the denoising effect while exacerbates the mixing effect.

Original: Oversmoothing happens because the mixing effect dominates the denoising effect beyond a certain depth.
Revised: Oversmoothing happens because the mixing effect dominates the denoising effect beyond a certain depth.

Original: For sufficiently dense CSBM graphs with N nodes, the required number of layers for this to happen is O (log N/ log(log N )) .
Revised: For sufficiently dense CSBM graphs with N nodes, the required number of layers for this to happen is O (log N/ log(log N )) .

Original: • We apply our framework to rigorously characterize the effects of PPR on oversmoothing.
Revised: • We apply our framework to rigorously characterize the effects of PPR on oversmoothing.

Original: We show that PPR reduces both the mixing effect and the denoising effect of message-passing and thus does not necessarily improve node classification performance.
Revised: We show that PPR reduces both the mixing effect and the denoising effect of message-passing and thus does not necessarily improve node classification performance.

Original: • We verify our theoretical results in experiments.
Revised: • We verify our theoretical results in experiments.

Original: Through comparison between theory and experiments, we find that the difficulty of optimizing weights in deep GNN architectures often aggravates oversmoothing.
Revised: Through comparison between theory and experiments, we find that the difficulty of optimizing weights in deep GNN architectures often aggravates oversmoothing.

Original:  A DDITIONAL R ELATED W ORK
Revised:  A DDITIONAL R ELATED W ORK

Original: Oversmoothing problem in GNNs Oversmoothing is a well-known issue in deep GNNs, and many techniques have been proposed to relieve it practically (Xu et al., 2018;
Revised: Oversmoothing problem in GNNs Oversmoothing is a well-known issue in deep GNNs, and many techniques have been proposed to relieve it practically (Xu et al., 2018;

Original: Li et al., 2019; Chen et al., 2020b; Huang et al., 2020; Zhao & Akoglu, 2020).
Revised: Li et al., 2019; Chen et al., 2020b; Huang et al., 2020; Zhao & Akoglu, 2020).

Original: On the theory side, by viewing GNN layers as a form of Laplacian filter, prior works have shown that as the number of layers goes to infinity, the node representations within each connected component of the graph will converge to the same values (Li et al., 2018; Oono & Suzuki, 2020).
Revised: On the theory side, by viewing GNN layers as a form of Laplacian filter, prior works have shown that as the number of layers goes to infinity, the node representations within each connected component of the graph will converge to the same values (Li et al., 2018; Oono & Suzuki, 2020).

Original: However, oversmoothing can be observed in GNNs with as few as 2 − 4 layers (Kipf & Welling, 2017; Wu et al., 2019).
Revised: However, oversmoothing can be observed in GNNs with as few as 2 − 4 layers (Kipf & Welling, 2017; Wu et al., 2019).

Original: The early onset of oversmoothing renders it an important concern in practice, and it has not been satisfyingly explained by the previous asymptotic studies.
Revised: The early onset of oversmoothing renders it an important concern in practice, and it has not been satisfyingly explained by the previous asymptotic studies.

Original: Our work addresses this gap by quantifying the effects of graph convolutions as a function of model depth and justifying why oversmoothing happens in shallow
Revised: Our work addresses this gap by quantifying the effects of graph convolutions as a function of model depth and justifying why oversmoothing happens in shallow

Original: GNNs.
Revised: GNNs.

Original: A recent study shared a similar insight of distinguishing between two competing effects of message-passing and showed the existence of an optimal number of layers for node prediction tasks on a latent space random graph model.
Revised: A recent study shared a similar insight of distinguishing between two competing effects of message-passing and showed the existence of an optimal number of layers for node prediction tasks on a latent space random graph model.

Original: But the result had no further quantification and hence the oversmoothing phenomemon was still only characterized asymptotically (Keriven, 2022).
Revised: But the result had no further quantification and hence the oversmoothing phenomemon was still only characterized asymptotically (Keriven, 2022).

Original: Analysis of GNNs on CSBMs Stochastic block models (SBMs) and their contextual counterparts have been widely used to study node classification problems (Abbe, 2018; Chen et al., 2019).
Revised: Analysis of GNNs on CSBMs Stochastic block models (SBMs) and their contextual counterparts have been widely used to study node classification problems (Abbe, 2018; Chen et al., 2019).

Original: Recently there have been several works proposing to use CSBMs to theoretically analyze GNNs for the node classification task.
Revised: Recently there have been several works proposing to use CSBMs to theoretically analyze GNNs for the node classification task.

Original: Wei et al.
Revised: Wei et al.

Original: (2022) used CSBMs to study the function of nonlinearity on the node classification performance, while Fountoulakis et al.
Revised: (2022) used CSBMs to study the function of nonlinearity on the node classification performance, while Fountoulakis et al.

Original: (2022) used CSBMs to study the attention-based GNNs.
Revised: (2022) used CSBMs to study the attention-based GNNs.

Original: More relevantly, Baranwal et al.
Revised: More relevantly, Baranwal et al.

Original: (2021; 2022) showed the advantage of applying graph convolutions up to three times for node classification on CSBM graphs.
Revised: (2021; 2022) showed the advantage of applying graph convolutions up to three times for node classification on CSBM graphs.

Original: Nonetheless, they only focused on the desirable denoising effect of graph convolution instead of its tradeoff with the undesirable mixing effect, and therefore did not explain the occurance of oversmoothing.
Revised: Nonetheless, they only focused on the desirable denoising effect of graph convolution instead of its tradeoff with the undesirable mixing effect, and therefore did not explain the occurance of oversmoothing.

Original:  P ROBLEM S ETTING AND M AIN R ESULTS
Revised:  P ROBLEM S ETTING AND M AIN R ESULTS

Original: We first introduce our theoretical analysis setup using the Contextual Stochastic Block Model, a random graph model with planted community structure (Deshpande et al., 2018;
Revised: We first introduce our theoretical analysis setup using the Contextual Stochastic Block Model, a random graph model with planted community structure (Deshpande et al., 2018;

Original: Baranwal et al., 2021; 2022; Ma et al., 2022; Wei et al., 2022; Fountoulakis et al., 2022).
Revised: Baranwal et al., 2021; 2022; Ma et al., 2022; Wei et al., 2022; Fountoulakis et al., 2022).

Original: We choose the CSBM to study GNNs for the node classification task because the main goal of node classification is to discover node communities from the data.
Revised: We choose the CSBM to study GNNs for the node classification task because the main goal of node classification is to discover node communities from the data.

Original: The CSBM imitates the community structure of real graphs and provides a clear ground truth for us to evaluate the model performance.
Revised: The CSBM imitates the community structure of real graphs and provides a clear ground truth for us to evaluate the model performance.

Original: Moreover, it is a generative model which gives us full control of the data to perform a non-asymptotic analysis.
Revised: Moreover, it is a generative model which gives us full control of the data to perform a non-asymptotic analysis.

Original: We then present a set of theoretical results establishing bounds for the representation power of GNNs in terms of the best-case node classification accuracy.
Revised: We then present a set of theoretical results establishing bounds for the representation power of GNNs in terms of the best-case node classification accuracy.

Original: The proofs of all the theorems and additional claims will be provided in the Appendix.
Revised: The proofs of all the theorems and additional claims will be provided in the Appendix.

Original: 3. N OTATIONS
Revised: 3. N OTATIONS

Original: We represent an undirected graph with N nodes by G =
Revised: We represent an undirected graph with N nodes by G =

Original: ( A, X ) , where A ∈ { 0 , 1 } N × N is the adjacency matrix and X ∈ R N is the node feature vector .
Revised: ( A, X ) , where A ∈ { 0 , 1 } N × N is the adjacency matrix and X ∈ R N is the node feature vector .

Original: For nodes u, v ∈ [ N ] , A uv = 1 if and only if u and v are connected with an edge in G , and X u ∈ R represents the node feature of u .
Revised: For nodes u, v ∈ [ N ] , A uv = 1 if and only if u and v are connected with an edge in G , and X u ∈ R represents the node feature of u .

Original: We let 1 N denote the all-one vector of length N and D = diag
Revised: We let 1 N denote the all-one vector of length N and D = diag

Original: ( A 1 N ) be the degree matrix of G .
Revised: ( A 1 N ) be the degree matrix of G .

Original: 3. T HEORETICAL A NALYSIS F RAMEWORK
Revised: 3. T HEORETICAL A NALYSIS F RAMEWORK

Original: Contextual Stochastic Block Models We will focus on the case where the CSBM consists of two classes C 1 and C 2 of nodes of equal size, in total with N nodes.
Revised: Contextual Stochastic Block Models We will focus on the case where the CSBM consists of two classes C 1 and C 2 of nodes of equal size, in total with N nodes.

Original: For any two nodes in the graph, if they are from the same class, they are connected by an edge independently with probability p , or if they are from different classes, the probability is q .
Revised: For any two nodes in the graph, if they are from the same class, they are connected by an edge independently with probability p , or if they are from different classes, the probability is q .

Original: For each node v ∈ C i , i ∈ { 1 , 2 } , the initial feature
Revised: For each node v ∈ C i , i ∈ { 1 , 2 } , the initial feature

Original: X v is sampled independently from a Gaussian distribution N ( µ i , σ 2 ) , where µ i ∈ R , σ ∈ (0 , ∞ ) .
Revised: X v is sampled independently from a Gaussian distribution N ( µ i , σ 2 ) , where µ i ∈ R , σ ∈ (0 , ∞ ) .

Original: Without loss of generality, we assume that µ 1 < µ 2 .
Revised: Without loss of generality, we assume that µ 1 < µ 2 .

Original: We denote a graph generated from such a
Revised: We denote a graph generated from such a

Original: CSBM as G ( A, X ) ∼ CSBM ( N, p, q, µ 1 , µ 2 , σ 2 ) .
Revised: CSBM as G ( A, X ) ∼ CSBM ( N, p, q, µ 1 , µ 2 , σ 2 ) .

Original: We further impose the following assumption on the CSBM used in our analysis.
Revised: We further impose the following assumption on the CSBM used in our analysis.

Original: Assumption 1.
Revised: Assumption 1.

Original: p, q = ω (log N/N ) and p > q > 0 .
Revised: p, q = ω (log N/N ) and p > q > 0 .

Original: The choice p, q = ω (log N/N ) ensures that the generated graph G is connected almost surely (Abbe, 2018) while being slightly more general than the p, q = ω (log 2 N/N ) regime considered in some concurrent works (Baranwal et al., 2021; Wei et al., 2022).
Revised: The choice p, q = ω (log N/N ) ensures that the generated graph G is connected almost surely (Abbe, 2018) while being slightly more general than the p, q = ω (log 2 N/N ) regime considered in some concurrent works (Baranwal et al., 2021; Wei et al., 2022).

Original: In addition, this regime also guarantees that G has a small diameter.
Revised: In addition, this regime also guarantees that G has a small diameter.

Original: Real-world graphs are known to exhibit the “smallworld” phenomenon—even if the number of nodes N is very large, the diameter of graph remains small (Girvan & Newman, 2002; Chung, 2010).
Revised: Real-world graphs are known to exhibit the “smallworld” phenomenon—even if the number of nodes N is very large, the diameter of graph remains small (Girvan & Newman, 2002; Chung, 2010).

Original: We will see in the theoretical analysis (Section 3.3) how this small-diameter characteristic contributes to the occurrence of oversmoothing in shallow
Revised: We will see in the theoretical analysis (Section 3.3) how this small-diameter characteristic contributes to the occurrence of oversmoothing in shallow

Original: GNNs.
Revised: GNNs.

Original: We remark that our results in fact hold for the more general choice of p, q = Ω(log N/N ) , for which only the concentration bound in Theorem 1 needs to be modified in the threshold log N/N case where all the constants need a more careful treatment.
Revised: We remark that our results in fact hold for the more general choice of p, q = Ω(log N/N ) , for which only the concentration bound in Theorem 1 needs to be modified in the threshold log N/N case where all the constants need a more careful treatment.

Original: Further, the choice p > q ensures that the graph structure has homophily , meaning that nodes from the same class are more likely to be connected than nodes from different classes.
Revised: Further, the choice p > q ensures that the graph structure has homophily , meaning that nodes from the same class are more likely to be connected than nodes from different classes.

Original: This characteristic is observed in a wide range of real-world graphs (Easley & Kleinberg, 2010; Ma et al., 2022).
Revised: This characteristic is observed in a wide range of real-world graphs (Easley & Kleinberg, 2010; Ma et al., 2022).

Original: We note that this homophily assumption ( p > q ) is not essential to our analysis, though we add it for simplicity since the discussion of homophily versus heterophily ( p < q ) is not the focus of our paper.
Revised: We note that this homophily assumption ( p > q ) is not essential to our analysis, though we add it for simplicity since the discussion of homophily versus heterophily ( p < q ) is not the focus of our paper.

Original: Graph convolution and linear GNN In this paper, our theoretical analysis focuses on the simplified linear GNN model defined as follows: a graph convolution using the (left-)normalized adjacency matrix takes the operation h ′
Revised: Graph convolution and linear GNN In this paper, our theoretical analysis focuses on the simplified linear GNN model defined as follows: a graph convolution using the (left-)normalized adjacency matrix takes the operation h ′

Original: =
Revised: =

Original: ( D − 1 A ) h , where h and h ′ are the input and output node representations, respectively.
Revised: ( D − 1 A ) h , where h and h ′ are the input and output node representations, respectively.

Original: A linear GNN layer can then be defined as h ′ =
Revised: A linear GNN layer can then be defined as h ′ =

Original: ( D − 1 A ) hW , where
Revised: ( D − 1 A ) hW , where

Original: W is a learnable weight matrix.
Revised: W is a learnable weight matrix.

Original: As a result, the output of n linear GNN layers can be written as h ( n ) (cid:81) nk =1 W ( k ) , where h ( n ) =
Revised: As a result, the output of n linear GNN layers can be written as h ( n ) (cid:81) nk =1 W ( k ) , where h ( n ) =

Original: ( D − 1 A ) n X is the output of n graph convolutions, and W ( k ) is the weight matrix of the k th layer.
Revised: ( D − 1 A ) n X is the output of n graph convolutions, and W ( k ) is the weight matrix of the k th layer.

Original: Since this is linear in h ( n ) , it follows that n -layer linear GNNs have the equivalent representation power as linear classifiers applied to h ( n ) .
Revised: Since this is linear in h ( n ) , it follows that n -layer linear GNNs have the equivalent representation power as linear classifiers applied to h ( n ) .

Original: In practice, when building GNN models, nonlinear activation functions can be added between consecutive linear GNN layers.
Revised: In practice, when building GNN models, nonlinear activation functions can be added between consecutive linear GNN layers.

Original: We will see that adding certain nonlinearity would not improve the classification performance (Proposition 1).
Revised: We will see that adding certain nonlinearity would not improve the classification performance (Proposition 1).

Original: Bayes error rate and z-score Thanks to the linearity of the model, we see that the representation of node v ∈ C
Revised: Bayes error rate and z-score Thanks to the linearity of the model, we see that the representation of node v ∈ C

Original: i after n graph convolutions is distributed as N ( µ ( n ) i , ( σ ( n ) ) 2 ) , where the variance ( σ ( n ) ) 2 is shared between classes.
Revised: i after n graph convolutions is distributed as N ( µ ( n ) i , ( σ ( n ) ) 2 ) , where the variance ( σ ( n ) ) 2 is shared between classes.

Revised: See Appendix for the detailed proof of this claim.

Original: The optimal node-wise classifier in this case is the Bayes optimal classifier , given by the following lemma.
Revised: The optimal node-wise classifier in this case is the Bayes optimal classifier , given by the following lemma.

Original: Lemma 1.
Revised: Lemma 1.

Original: Suppose the label y is drawn uniformly from { 1 , 2 } , and given y , x ∼ N ( µ ( n ) y , ( σ ( n ) ) 2 ) .
Revised: Suppose the label y is drawn uniformly from { 1 , 2 } , and given y , x ∼ N ( µ ( n ) y , ( σ ( n ) ) 2 ) .

Original: Then the Bayes optimal classifier, which minimizes the probability of misclassification among all classifiers, has decision boundary D
Revised: Then the Bayes optimal classifier, which minimizes the probability of misclassification among all classifiers, has decision boundary D

Original: = ( µ 1 + µ 2 ) / 2 , and predicts y = 1 , if x ≤ D or y = 2 , if x >
Revised: = ( µ 1 + µ 2 ) / 2 , and predicts y = 1 , if x ≤ D or y = 2 , if x >

Original: D .
Revised: D .

Original: The associated Bayes error rate is 1 − Φ( z ( n ) ) , where Φ denotes the cumulative distribution function of the standard Gaussian distribution and z ( n ) = 12 ( µ ( n ) 2 − µ ( n ) 1 ) /σ ( n ) is the z-score of D with respect to N ( µ ( n ) 1 , ( σ ( n ) ) 2 ) .
Revised: The associated Bayes error rate is 1 − Φ( z ( n ) ) , where Φ denotes the cumulative distribution function of the standard Gaussian distribution and z ( n ) = 12 ( µ ( n ) 2 − µ ( n ) 1 ) /σ ( n ) is the z-score of D with respect to N ( µ ( n ) 1 , ( σ ( n ) ) 2 ) .

Original: Lemma 1 states that we can estimate the optimal performance of an n -layer linear GNN through the z-score z ( n )
Revised: Lemma 1 states that we can estimate the optimal performance of an n -layer linear GNN through the z-score z ( n )

Original: = 12 ( µ ( n )− µ ( n ) 1 ) /σ
Revised: = 12 ( µ ( n )− µ ( n ) 1 ) /σ

Original: ( n ) .
Revised: ( n ) .

Original: A higher z-score indicates a smaller Bayes error rate, and hence a better expected performance of node classification.
Revised: A higher z-score indicates a smaller Bayes error rate, and hence a better expected performance of node classification.

Original: The z-score serves as a basis for our quantitative analysis of oversmoothing.
Revised: The z-score serves as a basis for our quantitative analysis of oversmoothing.

Original: In the following section, by estimating µ ( n )− µ ( n )and ( σ ( n ) ) 2 , we quantify the two counteracting effects of graph convolutions and obtain bounds on the z-score z ( n ) as a function of n , which allows us to characterize oversmoothing quantitatively.
Revised: In the following section, by estimating µ ( n )− µ ( n )and ( σ ( n ) ) 2 , we quantify the two counteracting effects of graph convolutions and obtain bounds on the z-score z ( n ) as a function of n , which allows us to characterize oversmoothing quantitatively.

Original: Specifically, there are two potential interpretations of oversmoothing based on the z-score:
Revised: Specifically, there are two potential interpretations of oversmoothing based on the z-score:

Original: (1) z ( n ) <
Revised: (1) z ( n ) <

Original: z ( n ⋆ ) , where n ⋆ = arg max n ′ z ( n ′ ) ; and (2) z ( n ) <
Revised: z ( n ⋆ ) , where n ⋆ = arg max n ′ z ( n ′ ) ; and (2) z ( n ) <

Original: z (0) .
Revised: z (0) .

Original: They correspond to the cases (1) n > n ⋆ ; and (2) n > n 0 , where n 0 ≥ 0 corresponds to the number of layers that yield a z-score on par with z (0) .
Revised: They correspond to the cases (1) n > n ⋆ ; and (2) n > n 0 , where n 0 ≥ 0 corresponds to the number of layers that yield a z-score on par with z (0) .

Original: The bounds on the z-score z ( n ) , z ( n ) lower and z ( n ) upper , enable us to estimate n ⋆ and nunder different scenarios and provide insights into the optimal choice of depth.
Revised: The bounds on the z-score z ( n ) , z ( n ) lower and z ( n ) upper , enable us to estimate n ⋆ and nunder different scenarios and provide insights into the optimal choice of depth.

Original: 3. M AIN R ESULTS
Revised: 3. M AIN R ESULTS

Original: We first estimate the gap between the means µ ( n )− µ ( n )with respect to the number of layers n .
Revised: We first estimate the gap between the means µ ( n )− µ ( n )with respect to the number of layers n .

Original: µ ( n ) 2 − µ ( n )measures how much node representations in different classes have homogenized after n
Revised: µ ( n ) 2 − µ ( n )measures how much node representations in different classes have homogenized after n

Original: GNN layers, which is the undesirable mixing effect.
Revised: GNN layers, which is the undesirable mixing effect.

Original: Lemma 2.
Revised: Lemma 2.

Original: For n ∈ N ∪ { 0 } , assuming D − 1 A ≈ E
Revised: For n ∈ N ∪ { 0 } , assuming D − 1 A ≈ E

Original: [ D ] − 1 E [ A ] ,
Revised: [ D ] − 1 E [ A ] ,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 states that the means µ ( n )and µ ( n )get closer exponentially fast and as n → ∞ , both µ ( n )and µ ( n )will converge to the same value (in this case ( µ ( n )+ µ ( n ) 2 ) / 2 ).
Revised: Lemma 2 states that the means µ ( n )and µ ( n )get closer exponentially fast and as n → ∞ , both µ ( n )and µ ( n )will converge to the same value (in this case ( µ ( n )+ µ ( n ) 2 ) / 2 ).

Original: The rate of change
Revised: The rate of change

Original: ( p − q ) / ( p + q ) is determined by the intra-community edge density p and the inter-community edge density q .
Revised: ( p − q ) / ( p + q ) is determined by the intra-community edge density p and the inter-community edge density q .

Original: Lemma 2 suggests that graphs with higher inter-community density ( q ) or lower intra-community density ( p ) are expected to suffer from a higher mixing effect when we perform message-passing.
Revised: Lemma 2 suggests that graphs with higher inter-community density ( q ) or lower intra-community density ( p ) are expected to suffer from a higher mixing effect when we perform message-passing.

Original: We provide the following concentration bound for our estimate of µ ( n )− µ ( n ) 1 , which states that the estimate concentrates at a rate of O (1 / (cid:112) N ( p + q )) .
Revised: We provide the following concentration bound for our estimate of µ ( n )− µ ( n ) 1 , which states that the estimate concentrates at a rate of O (1 / (cid:112) N ( p + q )) .

Original: Theorem 1.
Revised: Theorem 1.

Original: Fix K ∈ N and r > 0 .
Revised: Fix K ∈ N and r > 0 .

Original: There exists a constant C ( r, K ) such that with probability at least 1 − O (1 /N r ) , it holds for all 1 ≤ k ≤ K that
Revised: There exists a constant C ( r, K ) such that with probability at least 1 − O (1 /N r ) , it holds for all 1 ≤ k ≤ K that

Original: [Equation]
Revised: [Equation]

Original: We then study the variance ( σ ( n ) ) 2 with respect to the number of layers n .
Revised: We then study the variance ( σ ( n ) ) 2 with respect to the number of layers n .

Original: The variance ( σ ( n ) )measures how much the node representations in the same class have homogenized, which is the desirable denoising effect.
Revised: The variance ( σ ( n ) )measures how much the node representations in the same class have homogenized, which is the desirable denoising effect.

Original: We first state that no matter how many layers are applied, there is a nontrivial fixed lower bound for ( σ ( n ) ) 2 for a graph with N nodes.
Revised: We first state that no matter how many layers are applied, there is a nontrivial fixed lower bound for ( σ ( n ) ) 2 for a graph with N nodes.

Original: Lemma 3.
Revised: Lemma 3.

Original: For all n ∈ N ∪ { 0 } , 1 N σ 2 ≤ ( σ ( n ) ) 2 ≤ σ 2 .
Revised: For all n ∈ N ∪ { 0 } , 1 N σ 2 ≤ ( σ ( n ) ) 2 ≤ σ 2 .

Original: Lemma 3 implies that for a given graph, even as the number of layers n goes to infinity, the variance ( σ ( n ) ) 2 does not converge to zero, meaning that there is a fixed lower bound for the denoising effect.
Revised: Lemma 3 implies that for a given graph, even as the number of layers n goes to infinity, the variance ( σ ( n ) ) 2 does not converge to zero, meaning that there is a fixed lower bound for the denoising effect.

Original: See Appendix L.1 for the exact theoretical limit for the variance ( σ ( n ) ) 2 as n goes to infinity.
Revised: See Appendix L.1 for the exact theoretical limit for the variance ( σ ( n ) ) 2 as n goes to infinity.

Original: We now establish a set of more precise upper and lower bounds for the variance ( σ ( n ) ) 2 with respect to the number of layers n .
Revised: We now establish a set of more precise upper and lower bounds for the variance ( σ ( n ) ) 2 with respect to the number of layers n .

Original: Theorem 2.
Revised: Theorem 2.

Original: Let a = Np/ log N .
Revised: Let a = Np/ log N .

Original: With probability at least 1 − O (1 /N ) , it holds for all 1 ≤ n ≤ N that
Revised: With probability at least 1 − O (1 /N ) , it holds for all 1 ≤ n ≤ N that

Original: [Equation]
Revised: [Equation]

Original: Theorem 2 holds for all 1 ≤ n ≤ N .
Revised: Theorem 2 holds for all 1 ≤ n ≤ N .

Original: To clarify the upper bound, we provide the following corollary when n is bounded by a constant K .
Revised: To clarify the upper bound, we provide the following corollary when n is bounded by a constant K .

Original: Corollary 1.
Revised: Corollary 1.

Original: Let a = Np/ log N .
Revised: Let a = Np/ log N .

Original: Fix K ∈ N .
Revised: Fix K ∈ N .

Original: There exists a constant C ( K ) such that with probability at least 1 − O (1 /N ) , it holds for all 1 ≤ n ≤ K that
Revised: There exists a constant C ( K ) such that with probability at least 1 − O (1 /N ) , it holds for all 1 ≤ n ≤ K that

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 states that the variance ( σ ( n ) ) 2 for each Gaussian distribution decreases more for larger graphs or denser graphs.
Revised: Corollary 1 states that the variance ( σ ( n ) ) 2 for each Gaussian distribution decreases more for larger graphs or denser graphs.

Original: Moreover, the upper bound implies that the variance ( σ ( n ) ) 2 will initially go down at least at a rate exponential in O (1 / log N ) before reaching the fixed lower bound σ 2 /N suggested by Lemma 3.
Revised: Moreover, the upper bound implies that the variance ( σ ( n ) ) 2 will initially go down at least at a rate exponential in O (1 / log N ) before reaching the fixed lower bound σ 2 /N suggested by Lemma 3.

Original: This means that after O (log N/ log(log N )) layers, the desirable denoising effect homogenizing node representations in the same class will saturate and the undesirable mixing effect will start to dominate.
Revised: This means that after O (log N/ log(log N )) layers, the desirable denoising effect homogenizing node representations in the same class will saturate and the undesirable mixing effect will start to dominate.

Original: Why does oversmoothing happen at a shallow depth?
Revised: Why does oversmoothing happen at a shallow depth?

Original: For each node, message-passing with different-class nodes homogenizes their representations exponentially.
Revised: For each node, message-passing with different-class nodes homogenizes their representations exponentially.

Original: The exponential rate depends on the fraction of different-class neighbors among all neighbors (Lemma 2, mixing effect).
Revised: The exponential rate depends on the fraction of different-class neighbors among all neighbors (Lemma 2, mixing effect).

Original: Meanwhile, message-passing with nodes that have not been encountered before causes the denoising effect, and the magnitude depends on the absolute number of newly encountered neighbors.
Revised: Meanwhile, message-passing with nodes that have not been encountered before causes the denoising effect, and the magnitude depends on the absolute number of newly encountered neighbors.

Original: The diameter of the graph is approximately log N/ log( Np ) in the p, q = Ω(log N/N ) regime (Graham & Lu, 2001), and thus is at most log N/ log(log N ) in our case.
Revised: The diameter of the graph is approximately log N/ log( Np ) in the p, q = Ω(log N/N ) regime (Graham & Lu, 2001), and thus is at most log N/ log(log N ) in our case.

Original: After the number of layers surpasses the diameter, for each node, there will be no nodes that have not been encountered before in message-passing and hence the denoising effect will almost vanish (Theorem 2, denoising effect).
Revised: After the number of layers surpasses the diameter, for each node, there will be no nodes that have not been encountered before in message-passing and hence the denoising effect will almost vanish (Theorem 2, denoising effect).

Original: log N/ log(log N ) grows very slowly with N ; for example, when N = 10 6 , log N/ log(log N ) ≈ 8 .
Revised: log N/ log(log N ) grows very slowly with N ; for example, when N = 10 6 , log N/ log(log N ) ≈ 8 .

Original: This is why even in a large graph, the mixing effect will quickly dominate the denoising effect when we increase the number of layers, and so oversmoothing is expected to happen at a shallow depth.
Revised: This is why even in a large graph, the mixing effect will quickly dominate the denoising effect when we increase the number of layers, and so oversmoothing is expected to happen at a shallow depth.

Original: Our theory suggests that the optimal number of layers, n ⋆ , is at most O (log N/ log(log N )) .
Revised: Our theory suggests that the optimal number of layers, n ⋆ , is at most O (log N/ log(log N )) .

Original: For a more quantitative estimate, we can use Lemma 2 and Theorem 2 to compute bounds z ( n ) lower and z ( n ) upper for z = 12 ( µ ( n )− µ ( n ) 1 ) /σ
Revised: For a more quantitative estimate, we can use Lemma 2 and Theorem 2 to compute bounds z ( n ) lower and z ( n ) upper for z = 12 ( µ ( n )− µ ( n ) 1 ) /σ

Original: ( n ) and use them to infer n ⋆ and n 0 , as defined in Section 3.2.
Revised: ( n ) and use them to infer n ⋆ and n 0 , as defined in Section 3.2.

Original: See
Revised: See

Original: Appendix I for detailed discussion.
Revised: Appendix I for detailed discussion.

Original: In the previous analysis, we consider the case of a simplified linear GNN.
Revised: In the previous analysis, we consider the case of a simplified linear GNN.

Original: What would happen if we add nonlinearity after linear graph convolutions?
Revised: What would happen if we add nonlinearity after linear graph convolutions?

Original: Here, we consider the case of a GNN with a ReLU activation function added after n linear graph convolutions, i.e. h ( n ) ReLU =
Revised: Here, we consider the case of a GNN with a ReLU activation function added after n linear graph convolutions, i.e. h ( n ) ReLU =

Original: ReLU (( D − 1 A ) n X ) .
Revised: ReLU (( D − 1 A ) n X ) .

Original: We show that adding such nonlinearity does not improve the classification performance.
Revised: We show that adding such nonlinearity does not improve the classification performance.

Original: Proposition 1.
Revised: Proposition 1.

Original: Applying a ReLU activation function after n linear graph convolutions does not decrease the Bayes error rate, i.e. Bayes error rate based on h ( n ) ReLU ≥ Bayes error rate based on h ( n ) , and equality holds if µ 1 ≥ − µ 2 .
Revised: Applying a ReLU activation function after n linear graph convolutions does not decrease the Bayes error rate, i.e. Bayes error rate based on h ( n ) ReLU ≥ Bayes error rate based on h ( n ) , and equality holds if µ 1 ≥ − µ 2 .

Original: Next, we investigate the effect of increasing the dimension of the node features X .
Revised: Next, we investigate the effect of increasing the dimension of the node features X .

Original: So far, we have only considered the case with one-dimensional node features.
Revised: So far, we have only considered the case with one-dimensional node features.

Original: The following proposition states that if features in each dimension are independent, increasing input feature dimension decreases the
Revised: The following proposition states that if features in each dimension are independent, increasing input feature dimension decreases the

Original: Bayes error rate for a fixed n .
Revised: Bayes error rate for a fixed n .

Original: The intuition is that when node features provide more evidence for classification, it is easier to classify nodes correctly.
Revised: The intuition is that when node features provide more evidence for classification, it is easier to classify nodes correctly.

Original: Proposition 2.
Revised: Proposition 2.

Original: Let the input feature dimension be d , X ∈ R N × d .
Revised: Let the input feature dimension be d , X ∈ R N × d .

Original: Without loss of generality, suppose for node v in C i , h ( n ) v ∼ N ([0] d , I d ) , if i = 1 or h ( n ) v ∼ N ([1] d ,  I d ) , if i = 2 .
Revised: Without loss of generality, suppose for node v in C i , initial node feature X v ∼ N ([ µ i ] d , σ 2 I d ) independently.

Original: Then the Bayes error rate is 1 − Φ( √ d/ 2) , where Φ denotes the cumulative distribution function of the standard Gaussian distribution.
Revised: Then the Bayes error rate is 1 − Φ [Equation] , where Φ denotes the cumulative distribution function of the standard Gaussian distribution.

Original: Hence the Bayes error rate is decreasing in d , and as d → ∞ , it converges to 0 .
Revised: Hence the Bayes error rate is decreasing in d , and as d → ∞ , it converges to 0 .

Original:  T HE EFFECTS OF P ERSONALIZED P AGE R ANK ON OVERSMOOTHING
Revised:  T HE EFFECTS OF P ERSONALIZED P AGE R ANK ON OVERSMOOTHING

Original: Our analysis framework in Section 3.3 can also be applied to GNNs with other message-passing schemes.
Revised: Our analysis framework in Section 3.3 can also be applied to GNNs with other message-passing schemes.

Original: Specifically, we can analyze the performance of Personalized Propagation of Neural
Revised: Specifically, we can analyze the performance of Personalized Propagation of Neural

Original: Predictions (PPNP) and its approximate variant, Approximate PPNP (APPNP), which were proposed for alleviating oversmoothing while still making use of multi-hop information in the graph.
Revised: Predictions (PPNP) and its approximate variant, Approximate PPNP (APPNP), which were proposed for alleviating oversmoothing while still making use of multi-hop information in the graph.

Original: The main idea is to use Personalized PageRank (PPR) or the approximate Personalized PageRank (APPR) in place of graph convolutions (Klicpera et al., 2019).
Revised: The main idea is to use Personalized PageRank (PPR) or the approximate Personalized PageRank (APPR) in place of graph convolutions (Klicpera et al., 2019).

Original: Mathematically, the output of PPNP can be written as h PPNP =
Revised: Mathematically, the output of PPNP can be written as h PPNP =

Original: α ( I N − (1 − α )
Revised: α ( I N − (1 − α )

Original: ( D − 1 A ))
Revised: ( D − 1 A ))

Original: − 1 X , while APPNP computes h APPNP ( n +1)
Revised: − 1 X , while APPNP computes h APPNP ( n +1)

Original: = (1 − α )
Revised: = (1 − α )

Original: ( D − 1 A ) h APPNP ( n ) + αX iteratively in n , where I N is the identity matrix of size N and in both cases α is the teleportation probability.
Revised: ( D − 1 A ) h APPNP ( n ) + αX iteratively in n , where I N is the identity matrix of size N and in both cases α is the teleportation probability.

Original: Then for nodes in C i , i ∈ { 1 , 2 } , the node representations follow a Gaussian distribution N (cid:16) µ PPNP i , ( σ PPNP ) 2 (cid:17) after applying PPNP, or a Gaussian distribution N (cid:16) µ APPNP ( n ) i , ( σ APPNP ( n ) ) 2 (cid:17) after applying n APPNP layers.
Revised: Then for nodes in C i , i ∈ { 1 , 2 } , the node representations follow a Gaussian distribution N (cid:16) µ PPNP i , ( σ PPNP ) 2 (cid:17) after applying PPNP, or a Gaussian distribution N (cid:16) µ APPNP ( n ) i , ( σ APPNP ( n ) ) 2 (cid:17) after applying n APPNP layers.

Original: We quantify the effects on the means and variances for PPNP and APPNP in the CSBM case.
Revised: We quantify the effects on the means and variances for PPNP and APPNP in the CSBM case.

Original: We can similarly use them to calculate the z-score of ( µ 1 + µ 2 ) / 2 and compare it to the one derived for the baseline GNN in Section 3.
Revised: We can similarly use them to calculate the z-score of ( µ 1 + µ 2 ) / 2 and compare it to the one derived for the baseline GNN in Section 3.

Original: The key idea is that the PPR propagation can be written as a weighted average of the standard message-passing, i.e. α ( I N − (1 − α )
Revised: The key idea is that the PPR propagation can be written as a weighted average of the standard message-passing, i.e. α ( I N − (1 − α )

Original: ( D − 1 A ))
Revised: ( D − 1 A ))

Original: − 1 = (cid:80) ∞
Revised: − 1 = (cid:80) ∞

Original: k =

Revised: k =

Original: (1 − α ) k
Revised: (1 − α ) k

Original: ( D − 1 A ) k
Revised: ( D − 1 A ) k

Original: (Andersen et al., 2006).
Revised: (Andersen et al., 2006).

Original: We first state the resulting mixing effect measured by the difference between the two means.
Revised: We first state the resulting mixing effect measured by the difference between the two means.

Original: Proposition 3.
Revised: Proposition 3.

Original: Fix r > 0 , K ∈ N .
Revised: Fix r > 0 , K ∈ N .

Original: For PPNP, with probability at least 1 − O (1 /N r ) , there exists a constant C ( α, r, K ) such that
Revised: For PPNP, with probability at least 1 − O (1 /N r ) , there exists a constant C ( α, r, K ) such that

Original: [Equation]
Revised: [Equation]

Original: where the error term | ϵ | ≤ C/ (cid:112) N ( p + q [Equation]
Revised: µ 1 )

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: 1 − α ) K + [Equation]
Revised: where the error term | ϵ | ≤ C/ (cid:112) N ( p + q ) + (1 − α ) K +1 .

Original: Proposition 4.
Revised: Proposition 4.

Original: Let r > 0 .
Revised: Let r > 0 .

Original: For APPNP, with probability at least 1 − O (1 /N r ) , 
Revised: For APPNP, with probability at least 1 − O (1 /N r ) , µ APPNP

Original: [Equation]
Revised: [Equation]

Original: where the error term ϵ is the same as the one defined in Theorem 1 for the case of K = n .
Revised: where the error term ϵ is the same as the one defined in Theorem 1 for the case of K = n .

Original: Bothp
Revised: Both

Original: [Equation]
Revised: [Equation]

Original:  and 
Revised: αα q and (2 − 2 α ) q

Original: [Equation]
Revised: [Equation]

Original: p
Revised: +(

Original: [Equation]
Revised: [Equation]

Original:  are monotone increasing in α .
Revised: ) q (1 − α ) (cid:16) p − q p + q (cid:17) are monotone increasing in α .

Original: Hence from Propositionand 4, we see that with larger α , meaning a higher probability of teleportation back to the root node at each step of message-passing, PPNP and APPNP will indeed make the difference between the means of the two classes larger: while the difference in means for the baseline GNN decays as (cid:0) p − q p + q (cid:1) n , the difference for PPNP/APPNP is lower bounded by a constant.
Revised: Hence from Propositionand 4, we see that with larger α , meaning a higher probability of teleportation back to the root node at each step of message-passing, PPNP and APPNP will indeed make the difference between the means of the two classes larger: while the difference in means for the baseline GNN decays as (cid:0) p − q p + q (cid:1) n , the difference for PPNP/APPNP is lower bounded by a constant.

Original: This validates the original intuition behind PPNP and APPNP that compared to the baseline GNN, they reduce the mixing effect of message-passing, as staying closer to the root node means aggregating less information from nodes of different classes.
Revised: This validates the original intuition behind PPNP and APPNP that compared to the baseline GNN, they reduce the mixing effect of message-passing, as staying closer to the root node means aggregating less information from nodes of different classes.

Original: This advantage becomes more prominent when n is larger, where the model performance is dominated by the mixing effect: as n tends to infinity, while the means converge to the same value for the baseline GNN, their separation is lower-bounded for PPNP/APPNP.
Revised: This advantage becomes more prominent when n is larger, where the model performance is dominated by the mixing effect: as n tends to infinity, while the means converge to the same value for the baseline GNN, their separation is lower-bounded for PPNP/APPNP.

Original: However, the problem with the previous intuition is that PPNP and APPNP will also reduce the denoising effect at each layer, as staying closer to the root node also means aggregating less information from new nodes that have not been encountered before.
Revised: However, the problem with the previous intuition is that PPNP and APPNP will also reduce the denoising effect at each layer, as staying closer to the root node also means aggregating less information from new nodes that have not been encountered before.

Original: Hence, for an arbitrary graph, the result of the tradeoff after the reduction of both effects is not trivial to analyze.
Revised: Hence, for an arbitrary graph, the result of the tradeoff after the reduction of both effects is not trivial to analyze.

Original: Here, we quantify the resulting denoising effect for CSBM graphs measured by the variances.
Revised: Here, we quantify the resulting denoising effect for CSBM graphs measured by the variances.

Original: We denote ( σ ( n ) )
Revised: We denote ( σ ( n ) )

Original: 2 upper as the variance upper bound for depth n in Theorem 2.
Revised: 2 upper as the variance upper bound for depth n in Theorem 2.

Original: Proposition 5.
Revised: Proposition 5.

Original: For PPNP, with probability at least 1 − O (1 /N ) , it holds for all 1 ≤ K ≤ N that
Revised: For PPNP, with probability at least 1 − O (1 /N ) , it holds for all 1 ≤ K ≤ N that

Original: [Equation]
Revised: [Equation]

Original: Proposition 6.
Revised: Proposition 6.

Original: For APPNP, with probability at least 1 − O (1 /N ) , it holds for all 1 ≤ n ≤ N that
Revised: For APPNP, with probability at least 1 − O (1 /N ) , it holds for all 1 ≤ n ≤ N that

Original: [Equation]
Revised: [Equation]

Original: , σ   .
Revised: [Table]

Original: By comparing the lower bounds in Proposition 5 and 6 with that in Theorem 2, we see that PPR reduces the beneficial denoising effect of message-passing: for large or dense graphs, while the variances for the baseline GNN decay as 1 / ( Np ) n , the variances for PPNP/APPNP are lower bounded by the constant α 2 min { a, 2 } / 10 .
Revised: By comparing the lower bounds in Proposition 5 and 6 with that in Theorem 2, we see that PPR reduces the beneficial denoising effect of message-passing: for large or dense graphs, while the variances for the baseline GNN decay as 1 / ( Np ) n , the variances for PPNP/APPNP are lower bounded by the constant α 2 min { a, 2 } / 10 .

Original: In total, the mixing effect is reduced by a factor of (cid:0) p − q p + q (cid:1) n , while the denoising effect is reduced by a factor of 1 / ( Np ) n .
Revised: In total, the mixing effect is reduced by a factor of (cid:0) p − q p + q (cid:1) n , while the denoising effect is reduced by a factor of 1 / ( Np ) n .

Original: Hence PPR would cause greater reduction in the denoising effect than the improvement in the mixing effect for graphs where N and p are large.
Revised: Hence PPR would cause greater reduction in the denoising effect than the improvement in the mixing effect for graphs where N and p are large.

Original: This drawback would be especially notable at a shallow depth, where the denoising effect is supposed to dominate the mixing effect.
Revised: This drawback would be especially notable at a shallow depth, where the denoising effect is supposed to dominate the mixing effect.

Original: PPNP/APPNP would perform worse than the baseline GNN on these graphs in terms of the optimal classification performance.
Revised: APPNP would perform worse than the baseline GNN on these graphs in terms of the optimal classification performance.

Original: We remark that in each APPNP layer, another way to interpret the term αX is to regard it as a residual connection to the initial representation X (Chen et al., 2020b).
Revised: We remark that in each APPNP layer, another way to interpret the term αX is to regard it as a residual connection to the initial representation X (Chen et al., 2020b).

Original: Thus, our theory also validates the empirical observation that adding initial residual connections allows us to build very deep models without catastrophic oversmoothing.
Revised: Thus, our theory also validates the empirical observation that adding initial residual connections allows us to build very deep models without catastrophic oversmoothing.

Original: However, our results suggest that initial residual connections do not guarantee an improvement in model performance by themselves.
Revised: However, our results suggest that initial residual connections do not guarantee an improvement in model performance by themselves.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we first demonstrate our theoretical results in previous sections on synthetic CSBM data.
Revised: In this section, we first demonstrate our theoretical results in previous sections on synthetic CSBM data.

Original: Then we discuss the role of optimizing weights W ( k ) in GNN layers in the occurrence of oversmoothing through both synthetic data and the three widely used benchmarks: Cora, CiteSeer and
Revised: Then we discuss the role of optimizing weights W ( k ) in GNN layers in the occurrence of oversmoothing through both synthetic data and the three widely used benchmarks: Cora, CiteSeer and

Original: PubMed (Yang et al., 2016).
Revised: PubMed (Yang et al., 2016).

Original: Our results highlight the fact that the oversmoothing phenomenon observed in practice may be exacerbated by the difficulty of optimizing weights in deep GNN models.
Revised: Our results highlight the fact that the oversmoothing phenomenon observed in practice may be exacerbated by the difficulty of optimizing weights in deep GNN models.

Original: More details about the experiments are provided in Appendix K.
Revised: More details about the experiments are provided in Appendix K.

Original: 5. T HE EFFECT OF GRAPH TOPOLOGY ON OVERSMOOTHING
Revised: 5. T HE EFFECT OF GRAPH TOPOLOGY ON OVERSMOOTHING

Original: We first show how graph topology affects the occurrence of oversmoothing and the effects of PPR.
Revised: We first show how graph topology affects the occurrence of oversmoothing and the effects of PPR.

Original: We randomly generated synthetic graph data from CSBM( N = 2000 , p , q = 0 .
Revised: We randomly generated synthetic graph data from CSBM( N = 2000 , p , q = 0 .

Original: 0038 , µ 1 = 1 , µ 2 = 1 . 5 , σ 2 = 1 ).
Revised: 0038 , µ 1 = 1 , µ 2 = 1 . 5 , σ 2 = 1 ).

Original: We used 60% / 20% / 20% random splits and ran GNN and APPNP with α = 0 .
Revised: We used 60% / 20% / 20% random splits and ran GNN and APPNP with α = 0 .

Original: For results in Figure 2, we report averages over 5 graphs and for results in Figure 3, we report averages over 5 runs.
Revised: For results in Figure 2, we report averages over 5 graphs and for results in Figure 3, we report averages over 5 runs.

Original: In Figure 2, we study how the  community structure affects oversmoothing.
Revised: In Figure 2, we study how the strength of community structure affects oversmoothing.

Original: We can see that when graphs have a stronger community structure in terms of a higher intra-community edge density p , they would benefit more from repeated message-passing.
Revised: We can see that when graphs have a stronger community structure in terms of a higher intra-community edge density p , they would benefit more from repeated message-passing.

Original: As a result,  oversmoothing would happen later and a classifier could achieve better classification performance.
Revised: As a result, given the same set of node features, oversmoothing would happen later and a classifier could achieve better classification performance.

Original: A similar trend can also be observed in Figure 4A. Our theory predicts n ⋆ and n 0 , as defined in Section 3.2, with high accuracy.
Revised: A similar trend can also be observed in Figure 4A. Our theory predicts n ⋆ and n 0 , as defined in Section 3.2, with high accuracy.

Original: In Figure 3, we compare APPNP and GNN under different graph topologies.
Revised: In Figure 3, we compare APPNP and GNN under different graph topologies.

Original: In all three cases,
Revised: In all three cases,

Original: APPNP manifests its advantage of reducing the mixing effect compared to GNN when the number of layers is large, i.e. when the undesirable mixing effect is dominant.
Revised: APPNP manifests its advantage of reducing the mixing effect compared to GNN when the number of layers is large, i.e. when the undesirable mixing effect is dominant.

Original: However, as Figure 3B,C show, when we have large graphs or graphs with strong community structure, APPNP’s disadvantage of concurrently reducing the denoising effect is more severe, particularly when the number of layers is small.
Revised: However, as Figure 3B,C show, when we have large graphs or graphs with strong community structure, APPNP’s disadvantage of concurrently reducing the denoising effect is more severe, particularly when the number of layers is small.

Original: As a result, APPNP’s optimal performance is worse than the baseline GNN.
Revised: As a result, APPNP’s optimal performance is worse than the baseline GNN.

Original: These observations accord well with our theoretical discussions in Section 4.
Revised: These observations accord well with our theoretical discussions in Section 4.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: The performance of APPNP is more robust when we increase the model depth.
Revised: The performance of APPNP is more robust when we increase the model depth.

Original: However, compared to the base case ( A ), APPNP tends to have worse optimal performance than GNN on graphs with larger size ( B ) or stronger community structure ( C ), as predicted by the theory.
Revised: However, compared to the base case ( A ), APPNP tends to have worse optimal performance than GNN on graphs with larger size ( B ) or stronger community structure ( C ), as predicted by the theory.

Original: 5. T HE EFFECT OF OPTIMIZING WEIGHTS ON OVERSMOOTHING
Revised: 5. T HE EFFECT OF OPTIMIZING WEIGHTS ON OVERSMOOTHING

Original: We investigate how adding learnable weights W ( k ) in each GNN layers affects the node classification performance in practice.
Revised: We investigate how adding learnable weights W ( k ) in each GNN layers affects the node classification performance in practice.

Original: Consider the case when all the GNN layers used have width one, meaning that the learnable weight matrix W ( k ) in each layer is a scalar.
Revised: Consider the case when all the GNN layers used have width one, meaning that the learnable weight matrix W ( k ) in each layer is a scalar.

Original: In theory, the effects of adding such weights on the means and the variances would cancel each other and therefore they would not affect the z-score of our interest and the classification performance.
Revised: In theory, the effects of adding such weights on the means and the variances would cancel each other and therefore they would not affect the z-score of our interest and the classification performance.

Original: Figure 4A shows the the value of n 0 predicted by the z-score, the actual n 0 without learnable weights according to the test accuracy and the actual n 0 with learnable weights according to the test accuracy.
Revised: Figure 4A shows the the value of n 0 predicted by the z-score, the actual n 0 without learnable weights according to the test accuracy and the actual n 0 with learnable weights according to the test accuracy.

Original: The results are averages over 5 graphs for each case.
Revised: The results are averages over 5 graphs for each case.

Original: We empirically observe that GNNs with weights are much harder to train, and the difficulty increases as we increases the number of layers.
Revised: We empirically observe that GNNs with weights are much harder to train, and the difficulty increases as we increase the number of layers.

Original: As a result, n 0 is smaller for the model with weights and the gap is larger when n 0 is supposed to be larger, possibly due to greater difficulty in optimizing deeper architectures (Shamir, 2019).
Revised: As a result, n 0 is smaller for the model with weights and the gap is larger when n 0 is supposed to be larger, possibly due to greater difficulty in optimizing deeper architectures (Shamir, 2019).

Original: To relieve this potential optimization problem, we increase the width of each GNN layer (Du &
Revised: To relieve this potential optimization problem, we increase the width of each GNN layer (Du &

Original: Hu, 2019).
Revised: Hu, 2019).

Original: Figure 4B,C presents the training and testing accuracies of GNNs with increasing width with respect to the number of layers on a specific synthetic example.
Revised: Figure 4B,C presents the training and testing accuracies of GNNs with increasing width with respect to the number of layers on a specific synthetic example.

Original: The results are averages over 5 runs.
Revised: The results are averages over 5 runs.

Original: We observe that increasing the width of the network mitigates the difficulty of optimizing weights, and the performance after adding weights is able to gradually match the performance without weights.
Revised: We observe that increasing the width of the network mitigates the difficulty of optimizing weights, and the performance after adding weights is able to gradually match the performance without weights.

Original: This empirically validates our claim in Section 3.2 that adding learnable weights should not affect the representation power of GNN in terms of node classification accuracy on CSBM graphs, besides empirical optimization issues.
Revised: This empirically validates our claim in Section 3.2 that adding learnable weights should not affect the representation power of GNN in terms of node classification accuracy on CSBM graphs, besides empirical optimization issues.

Original: In practice, as we build deeper GNNs for more complicated tasks on real graph data, the difficulty of optimizing weights in deep GNN models persists.
Revised: In practice, as we build deeper GNNs for more complicated tasks on real graph data, the difficulty of optimizing weights in deep GNN models persists.

Original: We revisit the multi-class node classification task on the three widely used benchmark datasets: Cora, CiteSeer and PubMed (Yang et al., 2016).
Revised: We revisit the multi-class node classification task on the three widely used benchmark datasets: Cora, CiteSeer and PubMed (Yang et al., 2016).

Original: We compare the performance of GNN without weights against the performance of GNN with weights in terms of test accuracy.
Revised: We compare the performance of GNN without weights against the performance of GNN with weights in terms of test accuracy.

Original: We used 60% / 20% / 20% random splits, as in Wang & Leskovec (2020)  and Huang et al.
Revised: We used 60% / 20% / 20% random splits, as in Wang & Leskovec (2020) [Figure] and Huang et al.

Original: and report averages over 5 runs.
Revised: and report averages over 5 runs.

Original: Figure 5 shows the same kind of difficulty[Figure] in optimizing deeper models with learnable weights in each GNN layer as we have seen for the synthetic data.
Revised: Figure 5 shows the same kind of difficulty in optimizing deeper models with learnable weights in each GNN layer as we have seen for the synthetic data.

Original: Increasing the width of each GNN layer still mitigates the problem for shallower models, but it becomes much more difficult to tackle beyond 10 layers to the point that simply increasing the width could not solve it.
Revised: Increasing the width of each GNN layer still mitigates the problem for shallower models, but it becomes much more difficult to tackle beyond 10 layers to the point that simply increasing the width could not solve it.

Original: As a result, although GNNs with and without weights are on par with each other when both are shallow, the former has much worse performance when the number of layers goes beyond 10 .
Revised: As a result, although GNNs with and without weights are on par with each other when both are shallow, the former has much worse performance when the number of layers goes beyond 10 .

Original: These results suggest that the oversmoothing phenomenon observed in practice is aggravated by the difficulty of optimizing deep GNN models.
Revised: These results suggest that the oversmoothing phenomenon observed in practice is aggravated by the difficulty of optimizing deep GNN models.

Original:  D ISCUSSION
Revised:  D ISCUSSION

Original: Designing more powerful GNNs requires deeper understanding of current GNNs—how they work and why they fail.
Revised: Designing more powerful GNNs requires deeper understanding of current GNNs—how they work and why they fail.

Original: In this paper, we precisely characterize the mechanism of overmoothing via a non-asymptotic analysis and justify why oversmoothing happens at a shallow depth.
Revised: In this paper, we precisely characterize the mechanism of overmoothing via a non-asymptotic analysis and justify why oversmoothing happens at a shallow depth.

Original: Our analysis suggests that oversmoothing happens once the undesirable mixing effect homogenizing node representations in different classes starts to dominate the desirable denoising effect homogenizing node representations in the same class.
Revised: Our analysis suggests that oversmoothing happens once the undesirable mixing effect homogenizing node representations in different classes starts to dominate the desirable denoising effect homogenizing node representations in the same class.

Original: Due to the small diameter characteristic of real graphs, the turning point of the tradeoff will occur after only a few rounds of message-passing, resulting in oversmoothing in shallow GNNs.
Revised: Due to the small diameter characteristic of real graphs, the turning point of the tradeoff will occur after only a few rounds of message-passing, resulting in oversmoothing in shallow GNNs.

Original: It is worth noting that oversmoothing becomes an important problem in the literature partly because typical Convolutional Neural Networks (CNNs) used for image processing are much deeper than
Revised: It is worth noting that oversmoothing becomes an important problem in the literature partly because typical Convolutional Neural Networks (CNNs) used for image processing are much deeper than

Original: GNNs (He et al., 2016).
Revised: GNNs (He et al., 2016).

Original: As such, researchers have been trying to use methods that have previously worked for CNNs to make current GNNs deeper (Li et al., 2019; Chen et al., 2020b).
Revised: As such, researchers have been trying to use methods that have previously worked for CNNs to make current GNNs deeper (Li et al., 2019; Chen et al., 2020b).

Original: However, if we regard images as grids, images and real-world graphs have fundamentally different characteristics.
Revised: However, if we regard images as grids, images and real-world graphs have fundamentally different characteristics.

Original: In particular, images are giant grids, meaning that aggregating information between faraway pixels often requires a large number of layers.
Revised: In particular, images are giant grids, meaning that aggregating information between faraway pixels often requires a large number of layers.

Original: This contrasts with real-world graphs, which often have small diameters.
Revised: This contrasts with real-world graphs, which often have small diameters.

Original: Hence we believe that building more powerful GNNs will require us to think beyond CNNs and images and take advantage of the structure in real graphs.
Revised: Hence we believe that building more powerful GNNs will require us to think beyond CNNs and images and take advantage of the structure in real graphs.

Original: Our work has several limitations and possible directions for improvement.
Revised: Our work has several limitations and possible directions for improvement.

Original: First, while CSBMs mimic real graphs in some aspects and help us develop critical intuition into the function of GNNs, they cannot capture every characteristic of real graphs, such as degree heterogeneity.
Revised: First, while CSBMs mimic real graphs in some aspects and help us develop critical intuition into the function of GNNs, they cannot capture every characteristic of real graphs, such as degree heterogeneity.

Original: In addition, our analysis is based on the oracle Bayes classifier; in practice, one often trains a classifier in a semisupervised setting.
Revised: In addition, our analysis is based on the oracle Bayes classifier; in practice, one often trains a classifier in a semisupervised setting.

Original: Further research should aim to study the problem from a learning perspective.
Revised: Further research should aim to study the problem from a learning perspective.

Original: A P ROOF OF L EMMA
Revised: A P ROOF OF L EMMA

Original: Following the definition of the Bayes optimal classifier (Devroye et al., 1996),
Revised: Following the definition of the Bayes optimal classifier (Devroye et al., 1996),

Original: [Equation]
Revised: [Equation]

Original: we get that the Bayes optimal classifier has a linear decision boundary D = ( µ 1 + µ 2 ) / 2 such that the decision rule is
Revised: we get that the Bayes optimal classifier has a linear decision boundary D = ( µ 1 + µ 2 ) / 2 such that the decision rule is

Original: [Equation]
Revised: [Equation]

Original: Probability of misclassification could be written as
Revised: Probability of misclassification could be written as

Original: [Equation]
Revised: [Equation]

Original: When D =
Revised: When D =

Original: ( µ 1 + µ 2 ) / the expression is called the Bayes error rate, which is the minimal probability of misclassification among all classifiers.
Revised: ( µ 1 + µ 2 ) / the expression is called the Bayes error rate, which is the minimal probability of misclassification among all classifiers.

Original: Geometrically, it is easy to see that the Bayes error rate equals 12 S , where S is the overlapping area between the two Gaussian distributions
Revised: Geometrically, it is easy to see that the Bayes error rate equals 12 S , where S is the overlapping area between the two Gaussian distributions

Original: N (cid:16) µ ( n ) 1 , ( σ ( n ) ) 2 (cid:17) and N (cid:16) µ ( n ) 2 , ( σ ( n ) ) 2 (cid:17) .
Revised: N (cid:16) µ ( n ) 1 , ( σ ( n ) ) 2 (cid:17) and N (cid:16) µ ( n ) 2 , ( σ ( n ) ) 2 (cid:17) .

Original: Hence one can use the z-score of ( µ 1 + µ 2 ) / 2 with respect to either of the two Gaussian distributions to directly calculate the Bayes error rate.
Revised: Hence one can use the z-score of ( µ 1 + µ 2 ) / 2 with respect to either of the two Gaussian distributions to directly calculate the Bayes error rate.

Original: B P ROOF OF L EMMA
Revised: B P ROOF OF L EMMA

Revised: Under the heuristic assumption D − 1 A ≈ E

Revised: [ D ] − 1 E [ A ] , we can write

Revised: [Equation]

Revised: all

Revised: [Equation]

Original:  Fix n and let the element in the i th row and j th column of ( D − 1 A ) n be p ( n ) ij .
Revised: Writing recursively, we get that [Equation] C P ROOF OF L EMMA Fix n and let the element in the i th row and j th column of ( D − 1 A ) n be p ( n ) ij .

Original: Consider a fixed node i .
Revised: Consider a fixed node i .

Original: The variance of the feature for node i after n layers of convolutions is ( (cid:80) j ( p ( n ) ij ) 2 ) σ 2 , by the basic property of variance of sum.
Revised: The variance of the feature for node i after n layers of convolutions is ( (cid:80) j ( p ( n ) ij ) 2 ) σ 2 , by the basic property of variance of sum.

Original: Since (cid:80) j | p ( n ) ij | = 1 , it follows that (cid:80) j ( p ( n ) ij ) 2 ≤ 1 , which is the second inequality.
Revised: Since (cid:80) j | p ( n ) ij | = 1 , it follows that (cid:80) j ( p ( n ) ij ) 2 ≤ 1 , which is the second inequality.

Original: To show the first inequality, consider the following optimization problem:
Revised: To show the first inequality, consider the following optimization problem:

Original: [Equation]
Revised: [Equation]

Original: This part of proof goes by contradiction.
Revised: This part of proof goes by contradiction.

Original: Suppose ∃ k, l such that p ( n ) ik ̸ = ∃ p ( n ) il .
Revised: Suppose ∃ k, l such that p ( n ) ik ̸ = ∃ p ( n ) il .

Original: Fixing all other p ( n ) ij , j ̸ = k, l , if we average p ( n ) ik and p ( n ) il , their sum of squares will strictly decrease while not breaking the constraints:
Revised: Fixing all other p ( n ) ij , j ̸ = k, l , if we average p ( n ) ik and p ( n ) il , their sum of squares will strictly decrease while not breaking the constraints:

Original: [Equation]
Revised: [Equation]

Original: So we obtain a contradiction.
Revised: So we obtain a contradiction.

Original: Thus to minimize (cid:80) j ( p ( n ) ij ) 2 , p ( n ) ij = 1 N , 1 ≤ j ≤ N , and the mimimum is 1 /N .
Revised: Thus to minimize (cid:80) j ( p ( n ) ij ) 2 , p ( n ) ij = 1 N , 1 ≤ j ≤ N , and the mimimum is 1 /N .

Original: C P ROOF OF T HEOREM
Revised: D P ROOF OF T HEOREM

Original: We use ∥ · ∥ 2 to denote the spectral norm, ∥
Revised: We use ∥ · ∥ 2 to denote the spectral norm, ∥

Original: A ∥ 2 = max x : ∥ x ∥ =1 ∥
Revised: A ∥ 2 = max x : ∥ x ∥ =1 ∥

Original: Ax ∥ 2 .
Revised: Ax ∥ 2 .

Original: We denote ¯ A = E
Revised: We denote ¯ A = E

Original: [ A ] , ¯
Revised: [ A ] , ¯

Original: D = E
Revised: D = E

Original: [ D ] , d = A 1 N and ¯ d = E
Revised: [ D ] , d = A 1 N and ¯ d = E

Original: [ d ] i .
Revised: [ d ] i .

Original: We further define the following relevant vectors:
Revised: We further define the following relevant vectors:

Original: [Equation]
Revised: [Equation]

Original: The quantity of interest is µ ( k ) 2 − µ ( k ) 1 = 1 N/ 2 w ⊤ 

Revised: The quantity of interest is µ ( k ) 2 − µ ( k ) 1 = 1 N/ 2 w ⊤

Original: ( D − 1 A ) k µ .
Revised: ( D − 1 A ) k µ .

Original: C. A UXILIARY RESULTS
Revised: D. A UXILIARY RESULTS

Original: We record some properties of the adjacency matrices:
Revised: We record some properties of the adjacency matrices:

Original: D − 1 A and ¯ D − 1 ¯ A have an eigenvalue of 1 , corresponding to the right ) eigenvector 
Revised: D − 1 A and ¯ D − 1 ¯ A have an eigenvalue of 1 , corresponding to the (right) eigenvector w 1 . 2.

Original: w If J n = 1 n 1 ⊤ n , where 1 n is all-one vector of length n , then
Revised: If J n = 1 n 1 ⊤ n , where 1 n is all-one vector of length n , then

Original: [Equation]
Revised: [Equation]

Original: ¯ D =
Revised: ¯ D =

Original: N 2 ( p + q ) I N . 4.
Revised: N [Equation] N . 4.

Original: µ = αw 1 + βw 2 , where α = µ 1 + µ 2and β = µ 1 − µ 2.
Revised: µ = αw 1 + βw 2 , where α = µ 1 + µ 2and β = µ 1 − µ 2.

Original: To control the degree matrix D − 1 , we will use the following standard Chernoff bound Chung & Lu (2006):
Revised: To control the degree matrix D − 1 , we will use the following standard Chernoff bound Chung & Lu (2006):

Original: Lemma 4 (Chernoff Bound) .
Revised: Lemma 4 (Chernoff Bound) .

Original: Let X 1 , ..., X n be independent, S :
Revised: Let X 1 , ..., X n be independent, S :

Original: = (cid:80) ni
Revised: = (cid:80) ni

Original: =

Revised: =

Original: X i , and ¯ S = E
Revised: X i , and ¯ S = E

Original: [ S ] .
Revised: [ S ] .

Original: Then for all ε > 0 ,
Revised: Then for all ε > 0 ,

Original: [Equation]
Revised: [Equation]

Original: We can thus derive a uniform lower bound on the degree of every vertex:
Revised: We can thus derive a uniform lower bound on the degree of every vertex:

Original: Corollary 2.
Revised: Corollary 2.

Original: For every r > 0 , there is a constant C ( r ) such that whenever ¯ d ≥ C log N , with probability at least 1 − N − r ,
Revised: For every r > 0 , there is a constant C ( r ) such that whenever ¯ d ≥ C log N , with probability at least 1 − N − r ,

Original: [Equation]
Revised: [Equation]

Original: for all 1 ≤ i ≤ N. Consequently, with probability at least 1 − N − r , ∥ D − 1 − ¯ D − 1 ∥ 2 ≤ C/ ¯ d for some C .
Revised: Consequently, with probability at least 1 − N − r , ∥ D − 1 − ¯ D − 1 ∥ 2 ≤ C/ ¯ d for some C .

Original: Proof.
Revised: Proof.

Original: By applying Lemma 1 and a union bound, all degrees are within 1 / 2 ¯ d of their expectations, with probability at least 1 − e − ¯ d/ 8+log N .
Revised: By applying Lemma 1 and a union bound, all degrees are within 1 / 2 ¯ d of their expectations, with probability at least 1 − e − ¯ d/ 8+log N .

Original: Taking C = 8 r + 8 yields the desired lower bound.
Revised: Taking C = 8 r + 8 yields the desired lower bound.

Original: An analogous proof works for the upper bound.
Revised: An analogous proof works for the upper bound.

Original: To show the latter part, write
Revised: To show the latter part, write

Original: [Equation]
Revised: [Equation]

Original: Using the above bounds, the numerator for each i is at most 1 / 2 ¯ d and the denominator for each i is at least 1 / 2 ¯ d 2 , with probability at least 1 − N − r .
Revised: Using the above bounds, the numerator for each i is at most 1 / 2 ¯ d and the denominator for each i is at least 1 / 2 ¯ d 2 , with probability at least 1 − N − r .

Original: Combining the bounds yields the claim.
Revised: Combining the bounds yields the claim.

Original: We will also need a result on concentration of random adjacency matrices, which is a corollary of the sharp bounds derived in Bandeira & Van Handel (2016)
Revised: We will also need a result on concentration of random adjacency matrices, which is a corollary of the sharp bounds derived in Bandeira & Van Handel (2016)

Original: Lemma 5 (Concentration of Adjacency Matrix) .
Revised: Lemma 5 (Concentration of Adjacency Matrix) .

Original: For every r > 0 , there is a constant C ( r ) such that whenever ¯ d ≥ log N , with probability at least 1 − N − r ,
Revised: For every r > 0 , there is a constant C ( r ) such that whenever ¯ d ≥ log N , with probability at least 1 − N − r ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By corollary 3.12 from Bandeira & Van Handel (2016), there is a constant κ such that
Revised: By corollary 3.12 from Bandeira & Van Handel (2016), there is a constant κ such that

Original: [Equation]
Revised: [Equation]

Original: Setting t = (cid:112) (1 + r ) ¯ d , C = 3 + (cid:112) (1 + r ) κ suffices to achieve the desired bound.
Revised: Setting t = (cid:112) (1 + r ) ¯ d , C = 3 + (cid:112) (1 + r ) κ suffices to achieve the desired bound.

Original: C. S HARP CONCENTRATION OF THE RANDOM WALK OPERATOR D − 1 A
Revised: D. S HARP CONCENTRATION OF THE RANDOM WALK OPERATOR D − 1 A

Original: In this section, we aim to show the following concentration result for the random walk operator
Revised: In this section, we aim to show the following concentration result for the random walk operator

Original: D − 1 A :
Revised: D − 1 A :

Original: Theorem 3.
Revised: Theorem 3.

Original: Suppose the edge probabilities are ω (cid:16) log NN (cid:17) , and let ¯ d be the average degree.
Revised: Suppose the edge probabilities are ω (cid:16) log NN (cid:17) , and let ¯ d be the average degree.

Original: For any r , there exists a constant C such that for sufficiently large N , with probability at least 1 − O ( N − r ) ,
Revised: For any r , there exists a constant C such that for sufficiently large N , with probability at least 1 − O ( N − r ) ,

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We decompose the error
Revised: We decompose the error

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: We bound the two terms separately.
Revised: We bound the two terms separately.

Original: Bounding T 1 :
Revised: Bounding T 1 :

Original: By Corollary 2, ∥ D − 1 ∥ 2 = max i 1 /d i ≤ 2 / ¯ d with probability 1 − N − r .
Revised: By Corollary 2, ∥ D − 1 ∥ 2 = max i 1 /d i ≤ 2 / ¯ d with probability 1 − N − r .

Original: Combining this with Lemma 5, we see that with probability at least 1 − 2 N − r ,
Revised: Combining this with Lemma 5, we see that with probability at least 1 − 2 N − r ,

Original: [Equation]
Revised: [Equation]

Original: for some C depending only on r .
Revised: for some C depending only on r .

Original: Bounding T 2 : Similar to Lu & Peng (2013), we bound T 2 by exploiting the low-rank structure of the expected adjacency matrix, ¯ A .
Revised: Bounding T 2 : Similar to Lu & Peng (2013), we bound T 2 by exploiting the low-rank structure of the expected adjacency matrix, ¯ A .

Original: Recall that ¯ A has a special block form.
Revised: Recall that ¯ A has a special block form.

Original: The eigendecomposition of ¯ A is thus
Revised: The eigendecomposition of ¯ A is thus

Original: [Equation]
Revised: [Equation]

Original: where w 
Revised: where w (1) = 1 √ N 1 N , λ 1 = N ( p + q )

Original: [Equation]
Revised: [Equation]

Original: Using the definition of the spectral norm, we can bound ∥ T 2 ∥ 2 as
Revised: Using the definition of the spectral norm, we can bound ∥ T 2 ∥ 2 as

Original: [Equation]
Revised: [Equation]

Original: Note that when ∥ α ∥ 2 = 1 ,
Revised: Note that when ∥ α ∥ 2 = 1 ,

Original: [Equation]
Revised: [Equation]

Original: using Cauchy-Schwarz.
Revised: using Cauchy-Schwarz.

Original: Since | w ( j ) i | ≤ 1 √ N for all i, j , the second summation can be bounded by 1 N (cid:80) 2 j =1 λ 2 j .
Revised: Since | w ( j ) i | ≤ 1 √ N for all i, j , the second summation can be bounded by 1 N (cid:80) 2 j =1 λ 2 j .

Original: Overall, the upper bound is now
Revised: Overall, the upper bound is now

Original: [Equation]
Revised: [Equation]

Revised: λ

Revised: [Equation]

Original: Under the event of Corollary 2, d i ≥ C ¯ d for some C < 1 .
Revised: Under the event of Corollary 2, d i ≥ C ¯ d for some C < 1 .

Original: Under our setup, we also have λ 21 =
Revised: Under our setup, we also have λ 21 =

Original: ¯
Revised: ¯

Original: d 2 , λ 22 ≤ ¯ d 2 .
Revised: d 2 , λ 22 ≤ ¯ d 2 .

Original: This means that the upper bound is
Revised: This means that the upper bound is

Original: [Equation]
Revised: [Equation]

Original: where d is the vector of node degrees.
Revised: where d is the vector of node degrees.

Original: It remains to show that 1 N ∥ d − ¯ d 1 N ∥ 

Revised: It remains to show that 1 N ∥ d − ¯ d 1 N ∥

Original: =
Revised: =

Original: O ( ¯ d ) .
Revised: O ( ¯ d ) .

Original: To do this, we use a form of Talagrand’s concentration inequality, given in Boucheron et al.
Revised: To do this, we use a form of Talagrand’s concentration inequality, given in Boucheron et al.

Original: Since the function 1 √ N ∥ d − ¯ d 1 N ∥ 2 = 1 √ N ∥ ( A − ¯ dI N ) 1 N ∥ 2 is a convex, 1 -Lipschitz function of A , Theorem 6.10 from Boucheron et al.
Revised: Since the function 1 √ N ∥ d − ¯ d 1 N ∥ 2 = 1 √ N ∥ ( A − ¯ dI N ) 1 N ∥ 2 is a convex, 1 -Lipschitz function of A , Theorem 6.10 from Boucheron et al.

Original: (2013) guarantees that for any t > 0 ,
Revised: (2013) guarantees that for any t > 0 ,

Original: [Equation]
Revised: [Equation]

Original: Using Jensen’s inequality,
Revised: Using Jensen’s inequality,

Original: [Equation]
Revised: [Equation]

Original: If ¯ d = ω (log N ) , we can guarantee that
Revised: If ¯ d = ω (log N ) , we can guarantee that

Original: [Equation]
Revised: [Equation]

Original: with probability at least 1 − e − ( C − 1) 2 ¯ d/ 2 = 1 − O ( N − r ) for an appropriate constant C .
Revised: with probability at least 1 − e − ( C − 1) 2 ¯ d/ 2 = 1 − O ( N − r ) for an appropriate constant C .

Original: Thus we have shown that with high probability, T 2 =
Revised: Thus we have shown that with high probability, T 2 =

Original: O (1 / √ ¯ d ) , which proves the claim.
Revised: O (1 / √ ¯ d ) , which proves the claim.

Original: C. P ROOF OF T HEOREM
Revised: D. P ROOF OF T HEOREM

Original: Fix r and K .
Revised: Fix r and K .

Original: We desire to bound
Revised: We desire to bound

Original: [Equation]
Revised: [Equation]

Original: By the first property of adjacency matrices in auxiliary results, it suffices to bound
Revised: By the first property of adjacency matrices in auxiliary results, it suffices to bound

Original: [Equation]
Revised: [Equation]

Original: where β = µ 1 − µ 2.
Revised: where β = µ [Equation]

Original: We will show inductively that there is a C such that for every k = 1 , ..., K ,
Revised: We will show inductively that there is a C such that for every k = 1 , ..., K ,

Original: [Equation]
Revised: [Equation]

Original: If this is true, then Cauchy-Schwarz gives
Revised: If this is true, then Cauchy-Schwarz gives

Original: [Equation]
Revised: [Equation]

Original: By Theorem 3, we have that with probability at least 1 − O ( N − r ) ,
Revised: By Theorem 3, we have that with probability at least 1 − O ( N − r ) ,

Original: [Equation]
Revised: [Equation]

Original: So D − 1 A = ¯ D − 1 ¯ A
Revised: So D − 1 A = ¯ D − 1 ¯ A

Original: + J where ∥ J ∥ ≤ C/ √ ¯ d .
Revised: + J where ∥ J ∥ ≤ C/ √ ¯ d .

Original: Iterating, we have
Revised: Iterating, we have

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: Inductively, ( D − 1 A ) k − 1 = ( ¯ D − 1 ¯ A ) k − 1 + H where ∥ H ∥ 2 ≤ C/ √ ¯ d .
Revised: Inductively, ( D − 1 A ) k − 1 = ( ¯ D − 1 ¯ A ) k − 1 + H where ∥ H ∥ 2 ≤ C/ √ ¯ d .

Original: Plugging this in (1), we have 
Revised: Plugging this in (1), we have ∥

Original: [Equation]
Revised: [Equation]

Revised: A )

Original: Of these terms, ( ¯ D − 1 ¯ A  ) k − 1 J has norm at most ∥ J ∥ 2 , H
Revised: [Equation] ¯ A ) [Equation] − 1 + H )( ¯ D − 1 ¯ A + J ) − ( ¯ D − 1 ¯ A ) k

Revised: ∥ 2 .

Revised: Of these terms, ( ¯ D − 1 ¯ A ) k − 1 J has norm at most ∥ J ∥ 2 , H

Original: ( ¯ D − 1 ¯ A ) has norm at most ∥ H ∥ 2 , and HJ has norm at most C/ ¯ d . 

Revised: ( ¯ D − 1 ¯ A ) has norm at most ∥ H ∥ 2 , and HJ has norm at most C/ ¯ d .

Original: Hence the induction step is complete.
Revised: Hence the induction step is complete.

Original: We have thus shown that there is a constant C ( r, K ) such that with probability at least 1 − N − r ,
Revised: We have thus shown that there is a constant C ( r, K ) such that with probability at least 1 − N − r ,

Original: [Equation]
Revised: [Equation]

Original: which proves the claim.
Revised: which proves the claim.

Original: By simulation one can verify that indeed 1 N/ 2 w ⊤ 2 ( ¯ D − 1 ¯ A ) k µ ≈ (cid:16) p − q p + q (cid:17) k
Revised: By simulation one can verify that indeed 1 N/ 2 w ⊤ 2 ( ¯ D − 1 ¯ A ) k µ ≈ (cid:16) p − q p + q (cid:17) k

Original: ( µ 2 − µ 1 ) .
Revised: ( µ 2 − µ 1 ) .

Original: Figure 6 presents µ ( n ) 1 , µ ( n ) 2 calculated from simulation against predicted values from our theoretical results.
Revised: Figure 6 presents µ ( n ) 1 , µ ( n ) 2 calculated from simulation against predicted values from our theoretical results.

Original: The simulation results are averaged over 20 instances generated from CSBM( N = 2000 , p = 0 .
Revised: The simulation results are averaged over 20 instances generated from CSBM( N = 2000 , p = 0 .

Original: 0114 , q = 0 .
Revised: 0114 , q = 0 .

Original: 0038 , µ 1 = 1 , µ 2 = 1 . 5 ,
Revised: 0038 , µ 1 = 1 , µ 2 = 1 . 5 ,

Original: σ 2 = 1 ).
Revised: σ 2 = 1 ).

Original: D P ROOF OF L EMMA

Original: Under the heuristic assumption D − 1 A ≈ E

Original: [ D ] − 1 E [ A ] , we can write

Original: [Equation]

Original: for all k ∈ N .

Original: Writing recursively, we get that

Original: [Equation]

Original: E P ROOF OF T HEOREM
Revised: E P ROOF OF T HEOREM

Original: The proof relies on the following definition of neighborhood size: in a graph G , we denote by Γ k
Revised: The proof relies on the following definition of neighborhood size: in a graph G , we denote by Γ k

Original: ( x ) the set of vertices in G at distance k from a vertex x :
Revised: ( x ) the set of vertices in G at distance k from a vertex x :

Original: [Equation]
Revised: [Equation]

Original: we define N k ( x ) to be the set of vertices within distance k of x:
Revised: we define N k ( x ) to be the set of vertices within distance k of x:

Original: [Equation]
Revised: [Equation]

Original: To prove the lower bound, we first show an intermediate step that
Revised: To prove the lower bound, we first show an intermediate step that

Original: [Equation]
Revised: [Equation]

Original: The proof is the same as the one for the first inequality in Lemma 3, except we add in another constraint that for a fixed i , the row p i · is | N n ( i ) | -sparse.
Revised: The proof is the same as the one for the first inequality in Lemma 3, except we add in another constraint that for a fixed i , the row p i · is | N n ( i ) | -sparse.

Original: This implies that the minimum of (cid:80) j
Revised: This implies that the minimum of (cid:80) j

Original: ( p ( n ) ij )becomes 1 / | N n ( i ) | .
Revised: ( p ( n ) ij )becomes 1 / | N n ( i ) | .

Original: The we apply the result on upper bound of neighborhood sizes in Erd˝os-R´enyi graph G ( N, p ) (Lemma 2 Graham & Lu (2001)), as it also serves as upper bound of neighborhood sizes in SBM( N , p , q ).
Revised: The we apply the result on upper bound of neighborhood sizes in Erd ˝ os-R ´ enyi graph G ( N, p ) (Lemma 2 Graham & Lu (2001)), as it also serves as upper bound of neighborhood sizes in SBM( N , p , q ).

Original: The result implies that with probability at least 1 − O (1 /N ) , we have 
Revised: The result implies that with probability at least 1 − O (1 /N ) , we have [Equation] N

Original: [Equation]
Revised: [Equation]

Original: We ignore i for N n because of all nodes are identical in CSBM, so the bound applies for every nodes in the graph.
Revised: We ignore i for N n because of all nodes are identical in CSBM, so the bound applies for every nodes in the graph.

Original: The proof of upper bound is combinatorial.
Revised: The proof of upper bound is combinatorial.

Original: Corollary 2 states that when N is large, the degree of node i is approximately the expected degree in G , namely, E [ degree ] = N 2 ( p + q ) .
Revised: Corollary 2 states that when N is large, the degree of node i is approximately the expected degree in G , namely, E [ degree ] = N 2 ( p + q ) .

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: using the approximation of degrees, we get that
Revised: using the approximation of degrees, we get that

Original: [Equation]
Revised: [Equation]

Original:  i and j) . Then we use a tree approximation to calculate the number of paths P of length n between i and j by regarding i as the root.
Revised: paths P of length n between i and j) . Then we use a tree approximation to calculate the number of paths P of length n between i and j by regarding i as the root.

Original: Note that
Revised: Note that

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: and for j ∈ Γ n − 2 k , a deterministic path P ′ of length n − 2 k is needed in order to reach j from i .
Revised: and for j ∈ Γ n − 2 k , a deterministic path P ′ of length n − 2 k is needed in order to reach j from i .

Original: This implies that there are only k steps deviating from P ′ .
Revised: This implies that there are only k steps deviating from P ′ .

Original: There are ( n − 2 k + 1) k ways of choosing when to deviate.
Revised: There are ( n − 2 k + 1) k ways of choosing when to deviate.

Original: For each specific way of when to deviate, there are approximately E
Revised: For each specific way of when to deviate, there are approximately E

Original: [ degree ] k ways of choosing the destinations for deviation.
Revised: [ degree ] k ways of choosing the destinations for deviation.

Original: Hence in total, for j ∈ Γ n − 2 k , there are ( n − 2 k 
Revised: Hence in total, for j ∈ Γ n − 2 k , there are ( n − 2 k + 1) k E

Original: [Equation] 1) k E [ degree ] k path of length n between i and j .
Revised: [ degree ] k path of length n between i and j .

Original: Thus
Revised: Thus

Original: [Equation]
Revised: [Equation]

Original: Plug in (5) into (4), we get that
Revised: Plug in (5) into (4), we get that

Original: [Equation]
Revised: [Equation]

Original: Again, (7) follows from using the upper bound on | Γ n − 2 k | Graham & Lu (2001) such that with probability at least 1 − O (1 /N ) ,
Revised: Again, (7) follows from using the upper bound on | Γ n − 2 k | Graham & Lu (2001) such that with probability at least 1 − O (1 /N ) ,

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma 3, we obtain the final result.
Revised: Combining with Lemma 3, we obtain the final result.

Revised: Figure 7 presents variance calculated from simulation against predicted upper and lower bounds from our theoretical results.

Revised: The simulation results are averaged over 1000 instances generated from

Revised: CSBM( N = 2000 , p = 0 . 0114 , q = 0 . 0038 , µ 1 = 1 , µ 2 = 1 . 5 , σ 2 = 1 ).

Original: F P ROOF OF C OROLLARY
Revised: F P ROOF OF C OROLLARY

Original: When we fix K ∈ N , only the upper bound in Theorem 2 will change.
Revised: When we fix K ∈ N , only the upper bound in Theorem 2 will change.

Original: Note that now the upper bound in (7) can be written as 
Revised: Note that now the upper bound in (7) can be written as ⌊

Original: [Equation]
Revised: [Equation]

Revised: q

Original: (cid:18)
Revised: [Equation] N [Equation] [Equation]

Revised: ≤

Revised: C

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: q 2 p(cid:19)
Revised: q [Equation] p

Original: [Equation]
Revised: [Equation]

Original:  (cid:18) 2 p p + q  (cid:19)n (cid:18)
Revised: (cid:33) (cid:18) N ( p + q ) (cid:19)

Original: N ( p + q ) (cid:19) n ≤ C min { a, 2 } (cid:32) C (cid:88) k = (cid:18) p + q 2 p (cid:19) 2 k (cid:33) (cid:18)
Revised: [Equation]

Original: N ( p + q ) (cid:19) n ≤
Revised: ≤

Original: C min { a, 2 } (cid:18)
Revised: C min { a [Equation] 2 } (cid:18)

Original: N ( p + q ) (cid:19) n .
Revised: N ( p [Equation] q ) (cid:19) n .

Original: G P ROOF OF P ROPOSITION
Revised: G P ROOF OF P ROPOSITION

Original: If is known that if x follows a Gaussian distribution, then ReLU ( x ) follows a Rectified Gaussian distribution.
Revised: If is known that if x follows a Gaussian distribution, then ReLU ( x ) follows a Rectified Gaussian distribution.

Original: Following the definition of the Bayes optimal classifier, we present a geometric proof in Figure 7, where the dark blue bar denotes the location of 0 and the red bar denotes the decision boundary D of the Bayes optimal classifier, and the light blue area denotes the overlapping area S , which is twice the Bayes error rate.
Revised: Following the definition of the Bayes optimal classifier, we present a geometric proof in Figure 8 (see next page, top), where the dark blue bar denotes the location of 0 and the red bar denotes the decision boundary D of the Bayes optimal classifier, and the light blue area denotes the overlapping area S , which is twice the Bayes error rate.

Original: H P ROOF OF P ROPOSITION
Revised: H P ROOF OF P ROPOSITION

Revised: Let the node representation vector of node v after n graph convolutions be h ( n ) v .

Original: The Bayes error rate could be written as 12 ( P [ h ( n ) v > D| v ∈ C 1 ] + P [ h ( n ) v ≤ D| v ∈ C 2 ]) .
Revised: The Bayes error rate could be written as 12 ( P [ h ( n ) v > D| v ∈ C 1 ] + P [ h ( n ) v ≤ D| v ∈ C 2 ]) .

Original: For d ∈ N , due to the symmetry of our setup,  the optimal linear decision boundary is the hyperplane (cid:80) dj 
Revised: For d ∈ N , due to the symmetry of our setup, one can easily see that the optimal linear decision boundary is the hyperplane (cid:80) dj = [Equation]

Original: =
Revised: =

Original:  1 x j = d/ 2  .
Revised: [Equation] 2 ( µ 1 + µ 2 ) .

Original: Then for v ∈ C 1 , (cid:80) d j =1 ( h ( n ) v ) j ∼ N (0 , d  ) and for v ∈ C 2 , (cid:80) d j =1 ( h ( n ) v ) j ∼ N ( d, d ) .
Revised: Then for v ∈ C 1 , (cid:80) dj =1 ( h ( n ) v ) j ∼ N ( dµ ( n ) 1 , d ( σ ( n ) ) 2 ) and for v ∈ C 2 , (cid:80) dj

Original: Thus 2( P [ d (cid:88) j = ( h ( n ) v ) j > D| v ∈ C 1 ] + P [ d (cid:88)
Revised: =1 ( h ( n ) v ) j ∼ N ( dµ ( n ) 2 , d ( σ ( n ) ) 2 ) . Thus [Equation]

Original: j = ( h ( n )
Revised: The last equality follows from the fact that

Original: [Equation]
Revised: [Equation]

Original: I H OW TO USE THE Z - SCORE TO CHOOSE THE NUMBER OF LAYERS
Revised: I H OW TO USE THE Z - SCORE TO CHOOSE THE NUMBER OF LAYERS

Original: The bounds of the z-score with respect to the number of layers, z ( n ) lower and z ( n ) upper allow us to calculate bounds for n ⋆ and n 0 under different scenarios.
Revised: The bounds of the z-score with respect to the number of layers, z ( n ) lower and z ( n ) upper allow us to calculate bounds for n ⋆ and n 0 under different scenarios.

Original: Specifically,
Revised: Specifically,

Original: ∀ n ∈ N , z ( n ) upper <
Revised: ∀ n ∈ N , z ( n ) upper <

Original: z (0) =
Revised: z (0) =

Original: ( µ 2 − µ 1 ) /σ , then n ⋆ = n 0 = 0 , meaning that no graph convolution should be applied.
Revised: ( µ 2 − µ 1 ) /σ , then n ⋆ = n 0 = 0 , meaning that no graph convolution should be applied.

Original: |{ n ∈ N : z ( n ) upper ≥ z (0) }| > 0 , and (a) ∀ n ∈ N , z ( n ) lower <
Revised: |{ n ∈ N : z ( n ) upper ≥ z (0) }| > 0 , and (a) ∀ n ∈ N , z ( n ) lower <

Original: z
Revised: z

Original: (0) , then
Revised: (0) , then

Original: [Equation]
Revised: [Equation]

Original: which means that the number of graph convolutions should not exceed the upper bound of n 0 , or otherwise one gets worse performance than having no graph convolution.
Revised: which means that the number of graph convolutions should not exceed the upper bound of n 0 , or otherwise one gets worse performance than having no graph convolution.

Original: Note that in this case, since n ⋆ ≤ n 0 , we can only conclude that
Revised: Note that in this case, since n ⋆ ≤ n 0 , we can only conclude that

Original: [Equation]
Revised: [Equation]

Original: (b) |{ n ∈ N : z ( n ) [Equation] ≥ z (0) }| > 0 , then
Revised: (b) |{ n ∈ N : z ( n ) lower ≥ z (0) }| > 0 , then

Original: [Equation]
Revised: [Equation]

Original: and let arg max n
Revised: and let arg max n

Original: [Equation]
Revised: [Equation]

Original: meaning that the number of layers one should apply for optimal node classification performance is more than the lower bound of n ⋆ , and less than the upper bound of n ⋆ .
Revised: meaning that the number of layers one should apply for optimal node classification performance is more than the lower bound of n ⋆ , and less than the upper bound of n ⋆ .

Original: J P ROOFS OF P ROPOSITION 3J. P ROOF OF P ROPOSITION
Revised: J P ROOFS OF P ROPOSITION 3J. P ROOF OF P ROPOSITION

Original: Since the spectral radius of D − 1 A is 1,
Revised: Since the spectral radius of D − 1 A is 1,

Original: [Equation]
Revised: [Equation]

Original: Apply Lemma 2, we get that µ  PPNP
Revised: Apply Lemma 2, we get that µ PPNP− µ PPNP

Original: [Equation]
Revised: [Equation]

Original: To bound the approximation error, similar to the proof of the concentration bound in Theorem 1, it suffices to bound
Revised: To bound the approximation error, similar to the proof of the concentration bound in Theorem 1, it suffices to bound

Original: [Equation]
Revised: [Equation]

Original: where T K =
Revised: where T K =

Original: (cid:80)
Revised: (cid:80)

Original: Kk =

Revised: Kk =

Original: α (1 − α ) k
Revised: α (1 − α ) k

Original: (( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k ) , T K +1 , ∞ =
Revised: (( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k ) , T K +1 , ∞ =

Original: (cid:80) ∞
Revised: (cid:80) ∞

Original: k =
Revised: k =

Original: K +1 α
Revised: K +1 α

Original: (1 − α ) k
Revised: (1 − α ) k

Original: (( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k ) , and K ∈ N up to our own choice.
Revised: (( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k ) , and K ∈ N up to our own choice.

Original: Bounding T K : Apply Theorem 1, fix r > 0 , there exists a constant C ( r, K, α ) such that with probability 1 − O ( N − r ) ,
Revised: Bounding T K : Apply Theorem 1, fix r > 0 , there exists a constant C ( r, K, α ) such that with probability 1 − O ( N − r ) ,

Original: [Equation]
Revised: [Equation]

Original: Bounding T K +1 , ∞ : We will show upper bound for ( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k that applies for all k ∈ N .
Revised: Bounding T K +1 , ∞ : We will show upper bound for ( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k that applies for all k ∈ N .

Original: Note that for every k ∈ N ,
Revised: Note that for every k ∈ N ,

Original: [Equation]
Revised: [Equation]

Original: where D − 1 / 2 AD − 1 / 2 = V Λ V ⊤ is the eigenvalue decomposition.
Revised: where D − 1 / 2 AD − 1 / 2 = V Λ V ⊤ is the eigenvalue decomposition.

Original: Then
Revised: Then

Original: [Equation]
Revised: [Equation]

Original: Since ∥ ( D − 1 / 2 AD − 1 / 2 ) k ∥ 2 = 1 and by Corollary 2, with probability at least 1 − N − r , [Equation]
Revised: Since ∥ ( D − 1 / 2 AD − 1 / 2 ) k ∥ 2 = 1 and by Corollary 2, with probability at least 1 − N − r , ∥ D 1 / 2 ∥ 2 ≤ (cid:113) 3 ¯ d/ 2 ,

Original:  the previous inequality becomes ∥ ( D − 1 A ) k − ( ¯ D − 1 ¯ A [Equation] k 
Revised: ∥ D − 1 / 2 ∥ 2 ≤ (cid:113) 2 / ¯ d , the previous inequality becomes ∥ ( D − 1 A ) k − ( ¯ D − 1 ¯ A ) k ) ∥ 2 ≤ √ 3 + 1 .

Original: [Equation] ∥ 2 ≤ √ 3 + 1 .Hence [Equation]
Revised: Hence ∥ T K +1 , ∞ ∥ 2 ≤ (1 − α ) K +1 .

Original: Combining the two results, we prove the claim.
Revised: Combining the two results, we prove the claim.

Original: J. P ROOF OF P ROPOSITION
Revised: J. P ROOF OF P ROPOSITION

Original: The claim is a direct corollary of Theorem 1.
Revised: The claim is a direct corollary of Theorem 1.

Original: J. P ROOF OF P ROPOSITION
Revised: J. P ROOF OF P ROPOSITION

Original: The covariance matrix Σ PPNP of h PPNP could be written as
Revised: The covariance matrix Σ PPNP of h PPNP could be written as

Original: [Equation]
Revised: [Equation]

Original: Note that the variance of node i equals α 2 (cid:80) ∞ k, [Equation] − α ) k[Equation]
Revised: Note that the variance of node i equals α 2 (cid:80) ∞ k,l =0 (1 − α ) k

Original:  1 A [Equation] ki · (( D − 1 A ) l ) ⊤ i · , where i · refers row i of a matrix.
Revised: + l ( D − 1 A ) ki · (( D − 1 A ) l ) ⊤ i · , where i · refers row i of a matrix.

Original: Then by Cauchy-Schwarz Theorem,
Revised: Then by Cauchy-Schwarz Theorem,

Original: [Equation]
Revised: [Equation]

Revised: N

Revised: [Equation]

Original: Moreover, by Lemma 3, ( σ ( k ) ) 2 ≤ σ 2 .
Revised: Moreover, by Lemma 3, ( σ ( k ) ) 2 ≤ σ 2 .

Original: Due to the identity of each node i , we get that with probability 1 − O (1 /N ) , for all 1 ≤ K ≤ N ,
Revised: Due to the identity of each node i , we get that with probability 1 − O (1 /N ) , for all 1 ≤ K ≤ N ,

Original: [Equation]
Revised: [Equation]

Original: For the lower bound, note that with probability 1 − O (1 /N ) ,
Revised: For the lower bound, note that with probability 1 − O (1 /N [Equation]

Original: [Equation]
Revised: [Equation]

Original: where N k is the size of k -hop neighborhood.
Revised: where N k is the size of k -hop neighborhood.

Original: Then
Revised: Then

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: It is easy to see that Lemma 3 applies to any message-passing scheme which could be regarded as a random walk on the graph.
Revised: It is easy to see that Lemma 3 applies to any message-passing scheme which could be regarded as a random walk on the graph.

Original: Combining with Lemma 3, we get the final result.
Revised: Combining with Lemma 3, we get the final result.

Original: J. P ROOF OF P ROPOSITION
Revised: J. P ROOF OF P ROPOSITION

Original: Since
Revised: Since

Original: [Equation]
Revised: [Equation]

Original: Through the same calculation as for the upper bound in the proof of Proposition 3, we get that with probability 1 − O (1 /N ) ,
Revised: Through the same calculation as for the upper bound in the proof of Proposition 3, we get that with probability 1 − O (1 /N ) ,

Original: [Equation]
Revised: [Equation]

Original: For the lower bound, through the same calculation as for the upper bound in the proof of Proposition 3, we get that with probability 1 − O (1 /N ) ,
Revised: For the lower bound, through the same calculation as for the upper bound in the proof of Proposition 3, we get that with probability 1 − O (1 /N ) ,

Original: [Equation]
Revised: [Equation]

Original: Combining with Lemma [Equation] we get the final result.
Revised: Combining with Lemma 3, we get the final result.

Original: K E XPERIMENTS
Revised: K E XPERIMENTS

Original: Here we provide some more details on the models that we use in Section 5.
Revised: Here we provide some more details on the models that we use in Section 5.

Original: In all cases we use the
Revised: In all cases we use the

Original: Adam optimizer and tune some hyperparameters for better performance.
Revised: Adam optimizer and tune some hyperparameters for better performance.

Original: The hyperparameters used are summarized as follows.
Revised: The hyperparameters used are summarized as follows.

Original: [Figure]
Revised: [Figure]

Original: We empirically find that after adding in weights, it takes much longer to train the model for one iteration, and the time increases when the depth or the width increases (Figure 8).
Revised: We empirically find that after adding in weights, it takes much longer to train the model for one iteration, and the time increases when the depth or the width increases (Figure 9).

Original: Since for some combinations, it takes more than 200,000 iterations for the validation accuracy to finally increase, for each case, we only train for a reasonable amount of iterations.
Revised: Since for some combinations, it takes more than 200,000 iterations for the validation accuracy to finally increase, for each case, we only train for a reasonable amount of iterations.

Original: All models were implemented with PyTorch (Paszke et al., 2019) and PyTorch Geometric (Fey &
Revised: All models were implemented with PyTorch (Paszke et al., 2019) and PyTorch Geometric (Fey &

Original: Lenssen, 2019).
Revised: Lenssen, 2019).

Original: L A DDITIONAL R ESULTS L. E XACT LIMIT OF VARIANCE ( σ ( n ) ) 2 AS n → ∞
Revised: L A DDITIONAL R ESULTS L. E XACT LIMIT OF VARIANCE ( σ ( n ) ) 2 AS n → ∞

Original: Proposition 7.
Revised: Proposition 7.

Original: Given a graph G with adjacency matrix A , let its degree vector be d =
Revised: Given a graph G with adjacency matrix A , let its degree vector be d =

Original: A 1 N , where
Revised: A 1 N , where

Original: N is the all-one vector of length N .
Revised: N is the all-one vector of length N .

Original: If G is connected and non-bipartite, the variance of each node i , denoted as ( σ i ( n ) ) 2 , converges asymptotically to ∥ d ∥ 22 ∥ d ∥ 21 , i.e.[Equation] σ [Equation]
Revised: If G is connected and non-bipartite, the variance of each node i , denoted as ( σ i ( n ) ) 2 , converges asymptotically to ∥ d ∥ 22 ∥ d ∥ 21 , i.e.

Original: ) 2 n →∞ −−−−→

Original: [Equation]
Revised: [Equation]

Original: Then ∥ d ∥ 2 ∥ d ∥ 21 ≥ 1 N , and the equality holds if and only if G is regular.
Revised: Then ∥ d ∥ 2 ∥ d ∥ 2 1 ≥ 1 N , and the equality holds if and only if G is regular.

Original: Proof.
Revised: Proof.

Original: Let e i denotes the standard basis unit vector with the i th entry equals 1 , and all other entries equal 0 .
Revised: Let e i denotes the standard basis unit vector with the i th entry equals 1 , and all other entries equal 0 .

Original: Since G is connected and non-bipartite, the random walk represented by P
Revised: Since G is connected and non-bipartite, the random walk represented by P

Original: = D − 1 A is ergodic, meaning that
Revised: = D − 1 A is ergodic, meaning that

Original: [Equation]
Revised: [Equation]

Original: where π is the stationary distribution of this random walk with π i
Revised: where π is the stationary distribution of this random walk with π i

Original: = d i ∥ d ∥ 1 .
Revised: = d i ∥ d ∥ 1 .

Original: Then since norms are continuous functions, we conclude that
Revised: Then since norms are continuous functions, we conclude that

Original: [Equation]
Revised: [Equation]

Original: By Lemma 3, it follows that ∥ d ∥ 22 ∥ d ∥ 21 ≥ 1 N .
Revised: By Lemma 3, it follows that ∥ d ∥ 22 ∥ d ∥ 21 ≥ 1 N .

Original: The unique minimizer of ∥ π ∥ 22 subject to ∥ π ∥ 1 = 1 is π = 1 N 1 N .
Revised: The unique minimizer of ∥ π ∥ 22 subject to ∥ π ∥ 1 = 1 is π = 1 N 1 N .

Original: This means that G must be regular to achieve the lower bound asymptotically.
Revised: This means that G must be regular to achieve the lower bound asymptotically.

Original: Under Assumption 1, the graph generated by our CSBM is almost surely connected.
Revised: Under Assumption 1, the graph generated by our CSBM is almost surely connected.

Original: Here, we remain to show that with high probability, the graph will also be non-bipartite.
Revised: Here, we remain to show that with high probability, the graph will also be non-bipartite.

Original: Proposition 8.
Revised: Proposition 8.

Original: With probability at least 1 − O (1 / ( Np ) 3 ) , a graph G generated from CSBM( N , p , q , µ 1 , µ 2 , σ 2 ) contains a triangle, which implies that it is non-bipartite.
Revised: With probability at least 1 − O (1 / ( Np ) 3 ) , a graph G generated from CSBM( N , p , q , µ 1 , µ 2 , σ 2 ) contains a triangle, which implies that it is non-bipartite.

Original: Proof.
Revised: Proof.

Original: The proof goes by the classic probabilistic method.
Revised: The proof goes by the classic probabilistic method.

Original: Let T ∆ =
Revised: Let T ∆ =

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: i 1 τ i denotes the number of triangles in G , where 1 τ i equals 1 if potential triangle τ
Revised: i 1 τ i denotes the number of triangles in G , where 1 τ i equals 1 if potential triangle τ

Original: i exists and 0 otherwise.
Revised: i exists and 0 otherwise.

Original: Then by second moment method,
Revised: Then by second moment method,

Original: [Equation]
Revised: [Equation]

Original: Since E [ T ∆ ] = O ( Np ) , (cid:80) i ̸ =
Revised: Since E [ T ∆ ] = O ( Np ) , (cid:80) i ̸ =

Original: j E [ 1 τ i 

Revised: j E [ 1 τ i

Original: τ j ] = (1 + O (1 /N ))
Revised: τ j ] = (1 + O (1 /N ))

Original: ( E [ T ∆ ]) 2 , we get that
Revised: ( E [ T ∆ ]) 2 , we get that

Revised: P [ T ∆ =

Revised: [Equation]

Revised: ] ≤ O (1 / ( Np )

Original: [Equation]
Revised: [Equation]

Revised: + O (1 /N ) ≤

Revised: O (1 / ( Np ) 3 ) .

Original: Hence P [ G is non-bipartite ] ≥ P
Revised: Hence P [ G is non-bipartite ] ≥ P

Original: [ T ∆ ≥ 1] ≥ 1 − O (1 / ( Np ) 3 ) .
Revised: [ T ∆ ≥ 1] ≥ 1 − O (1 / ( Np ) 3 ) .

Original: L. S YMMETRIC G RAPH C ONVOLUTION D − 1 / 2 AD − 1 /
Revised: L. [Equation]

Original: Proposition 9.
Revised: Proposition 9.

Original: When using symmetric message-passing convolution D − 1 / 2 AD − 1 / 2 instead, the variance ( σ ( n ) ) 2 is non-increasing with respect to the number of convolutional layers n .
Revised: When using symmetric message-passing convolution D − 1 / 2 AD − 1 / 2 instead, the variance ( σ ( n ) ) 2 is non-increasing with respect to the number of convolutional layers n .

Original: i.e.
Revised: i.e.

Revised: [Equation]

Revised: n

Original: [Equation]
Revised: [Equation]

Revised: n

Revised: [Equation]

Revised: n ∈ N ∪ { 0 } .

Original: Proof.
Revised: Proof.

Original: We want to calculate the diagonal entries of the covariance matrix Σ ( n ) of ( D − 1 / 2 AD − 1 / 2 ) n X , where the covariance matrix of X is σ 2 I N .
Revised: We want to calculate the diagonal entries of the covariance matrix Σ ( n ) of ( D − 1 / 2 AD − 1 / 2 ) n X , where the covariance matrix of X is σ 2 I N .

Original: Hence
Revised: Hence

Original: [Equation]
Revised: [Equation]

Original: Since D − 1 / 2 AD − 1 / 2 is symmetric, let its eigendecomposition be V Λ V ⊤ and we could rewriteΣ ( n )
Revised: Since D − 1 / 2 AD − 1 / 2 is symmetric, let its eigendecomposition be V Λ V ⊤ and we could rewrite

Original: = ( V Λ n V ⊤ )( V Λ n V ⊤ ) = V Λ 2 n V ⊤ .
Revised: [Equation]

Original: Notice that the closed form of the diagonal entries is
Revised: Notice that the closed form of the diagonal entries is

Original: [Equation]
Revised: [Equation]

Original: Since for all 1 ≤ i ≤ N , | λ i | ≤ 1 , we obtain monotonicity of each entry of diag(Σ ( n ) ) , i.e. variance of each node.
Revised: Since for all 1 ≤ i ≤ N , | λ i | ≤ 1 , we obtain monotonicity of each entry of diag(Σ ( n ) ) , i.e. variance of each node.

Original: Although the proposition does not always hold for random walk message-passing convolution
Revised: Although the proposition does not always hold for random walk message-passing convolution

Original: D − 1 A as one can construct specific counterexamples (Appendix L.3), in practice, variances are observed to be decreasing with respect with the number of layers.
Revised: D − 1 A as one can construct specific counterexamples (Appendix L.3), in practice, variances are observed to be decreasing with respect with the number of layers.

Original: Moreover, we empirically observe that variance goes down more than the variance using symmetric message-passing convolutions.
Revised: Moreover, we empirically observe that variance goes down more than the variance using symmetric message-passing convolutions.

Original: Figure 9 presents visualization of node representations comparing the change of variance with respect to the number of layers using random walk convolution and symmetric message-passing convolution.
Revised: Figure 10 presents visualization of node representations comparing the change of variance with respect to the number of layers using random walk convolution and symmetric message-passing convolution.

Original: The data is generated from CSBM( N = 2000 , p = 0 .
Revised: The data is generated from CSBM( N = 2000 , p = 0 .

Original: 0114 , q = 0 .
Revised: 0114 , q = 0 .

Original: 0038 , µ 1 = 1 , µ 2 = 1 . 5 ,
Revised: 0038 , µ 1 = 1 , µ 2 = 1 . 5 ,

Original: σ 2 = 1 ).
Revised: σ 2 = 1 ).

Original: L. C OUNTEREXAMPLES
Revised: L. C OUNTEREXAMPLES

Original: Here, we construct a specific example where the variance ( σ ( n ) ) 2 is not non-increasing with respect to the number of layers n (Figure 10A).
Revised: Here, we construct a specific example where the variance ( σ ( n ) ) 2 is not non-increasing with respect to the number of layers n (Figure 11A).

Original: We remark that such a non-monotone nature of change in variance is not caused by by bipartiteness of the graph, as a cycle graph with even number of nodes is also bipartite, but does not exhibit such phenomenon (Figure 10B).
Revised: We remark that such a non-monotone nature of change in variance is not caused by by bipartiteness of the graph, as a cycle graph with even number of nodes is also bipartite, but does not exhibit such phenomenon (Figure 11B).

Original: We conjecture the increase in variance is rather caused by the tree-like structure.
Revised: We conjecture the increase in variance is rather caused by the tree-like structure.

Original: [Figure]
Revised: [Figure]

Revised: L. T HE MIXING AND DENOISING EFFECTS IN PRACTICE

Revised: In this section, we measure the mixing and denoising effects of graph convolutions identified by our theoretical results in practice, and show that the same tradeoff between the two counteracting effects exists for real-world graphs.

Revised: For the mixing effect, we measure the pairwise L 2 distances between the means of different classes, and for the denoising effect, we measure the within-class variances, both respect to the number of layers.

Revised: Figure 12 gives a visualization of both metrics for all classes on Cora, CiteSeer and PubMed.

Revised: We observe that similar to the synthetic CSBM data, adding graph convolutions increases both the mixing effect (homogenizing node representations in different classes, measured by the inter-class distances) and the denoising effect homogenizing node representations in the same class homogenize, measured by the within-class distances).

Revised: In addition, there is clearly a saturation in the beneficial denoising effect just after a small number of layers, as predicted by our theory.