Original: R EPRESENTATION P OWER OF G RAPH C ONVOLUTIONS : N EURAL T ANGENT K ERNEL A NALYSIS
Revised: R EPRESENTATION P OWER OF G RAPH C ONVOLUTIONS : N EURAL T ANGENT K ERNEL A NALYSIS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: The fundamental principle of Graph Neural Networks (GNNs) is to exploit the structural information of the data by aggregating the neighboring nodes using a ‘graph convolution’.
Revised: The fundamental principle of Graph Neural Networks (GNNs) is to exploit the structural information of the data by aggregating the neighboring nodes using a ‘graph convolution’.

Original: Therefore, understanding its influence on the network performance is crucial.
Revised: Therefore, understanding its influence on the network performance is crucial.

Original: Convolutions based on graph Laplacian have emerged as the dominant choice with the symmetric normalization of the adjacency matrix A , defined as D − 12 AD − 12 , being the most widely adopted one, where D is the degree matrix.
Revised: Convolutions based on graph Laplacian have emerged as the dominant choice with the symmetric normalization of the adjacency matrix A , defined as D − 12 AD − 12 , being the most widely adopted one, where D is the degree matrix.

Original: However, some empirical studies show that row normalization D − 1 A outperforms it in node classification.
Revised: However, some empirical studies show that row normalization D − 1 A outperforms it in node classification.

Original: Despite the widespread use of GNNs, there is no rigorous theoretical study on the representation power of these convolution operators, that could explain this behavior.
Revised: Despite the widespread use of GNNs, there is no rigorous theoretical study on the representation power of these convolution operators, that could explain this behavior.

Original: In this work, we analyze the influence of the graph convolutions theoretically using Graph Neural Tangent Kernel in a semi-supervised node classification setting.
Revised: In this work, we analyze the influence of the graph convolutions theoretically using Graph Neural Tangent Kernel in a semi-supervised node classification setting.

Original: Under a Degree Corrected Stochastic
Revised: Under a Degree Corrected Stochastic

Original: Block Model, we  prove that: (i) row normalization preserves the underlying class structure better than other convolutions; (ii) performance degrades with network depth due to over-smoothing, but the loss in class information is the slowest in row normalization; (iii) skip connections retain the class information even at infinite depth, thereby eliminating over-smoothing.
Revised: Block Model we analyze different graphs that have homophilic, heterophilic and core-periphery structures, and prove that: (i) row normalization preserves the underlying class structure better than other convolutions; (ii) performance degrades with network depth due to over-smoothing, but the loss in class information is the slowest in row normalization; (iii) skip connections retain the class information even at infinite depth, thereby eliminating over-smoothing.

Original: We finally validate our theoretical findings  on real datasets.
Revised: We finally validate our theoretical findings numerically and on real datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: With the advent of Graph Neural Networks (GNNs), there has been a tremendous progress in the development of computationally efficient state-of-the-art methods in various graph based tasks, including drug discovery, community detection and recommendation systems (Wieder et al., 2020;
Revised: With the advent of Graph Neural Networks (GNNs), there has been a tremendous progress in the development of computationally efficient state-of-the-art methods in various graph based tasks, including drug discovery, community detection and recommendation systems (Wieder et al., 2020;

Original: Fortunato & Hric, 2016; van den Berg et al., 2017).
Revised: Fortunato & Hric, 2016; van den Berg et al., 2017).

Original: Many of these problems depend on the structural information of the entities along with the features for effective learning.
Revised: Many of these problems depend on the structural information of the entities along with the features for effective learning.

Original: Because GNNs exploit this topological information encoded in the graph, it can learn better representation of the nodes or the entire graph than traditional deep learning techniques, thereby achieving state-of-the-art performances.
Revised: Because GNNs exploit this topological information encoded in the graph, it can learn better representation of the nodes or the entire graph than traditional deep learning techniques, thereby achieving state-of-the-art performances.

Original: In order to accomplish this, GNNs apply aggregation function to each node in a graph that combines the features of the neighboring nodes, and its variants differ principally in the methods of aggregation.
Revised: In order to accomplish this, GNNs apply aggregation function to each node in a graph that combines the features of the neighboring nodes, and its variants differ principally in the methods of aggregation.

Original: For instance, graph convolution networks use mean neighborhood aggregation through spectral approaches (Bruna et al., 2014; Defferrard et al., 2016; Kipf & Welling, 2017) or spatial approaches (Hamilton et al., 2017; Duvenaud et al., 2015; Xu et al., 2019), graph attention networks apply multi-head attention based aggregation (Velickovic et al., 2018) and graph recurrent networks employ complex computational module (Scarselli et al., 2008; Li et al., 2016).
Revised: For instance, graph convolution networks use mean neighborhood aggregation through spectral approaches (Bruna et al., 2014; Defferrard et al., 2016; Kipf & Welling, 2017) or spatial approaches (Hamilton et al., 2017; Duvenaud et al., 2015; Xu et al., 2019), graph attention networks apply multi-head attention based aggregation (Velickovic et al., 2018) and graph recurrent networks employ complex computational module (Scarselli et al., 2008; Li et al., 2016).

Original: Of all the aggregation policies, the spectral approach based on graph Laplacian is most widely used in practice, specifically the one proposed by Kipf & Welling (2017) owing to its simplicity and empirical success.
Revised: Of all the aggregation policies, the spectral approach based on graph Laplacian is most widely used in practice, specifically the one proposed by Kipf & Welling (2017) owing to its simplicity and empirical success.

Original: In this work, we focus on such graph Laplacian based aggregations in Graph Convolution Networks (GCNs), which we refer to as graph convolutions or diffusion operators .
Revised: In this work, we focus on such graph Laplacian based aggregations in Graph Convolution Networks (GCNs), which we refer to as graph convolutions or diffusion operators .

Original: Kipf & Welling (2017) propose a GCN for node classification, a semi-supervised task, where the goal is to predict the label of a node using its feature and neighboring node information.
Revised: Kipf & Welling (2017) propose a GCN for node classification, a semi-supervised task, where the goal is to predict the label of a node using its feature and neighboring node information.

Original: This work suggests symmetric normalization S sym
Revised: This work suggests symmetric normalization S sym

Original: = D − 12 AD − 12 as the graph convolution.
Revised: = D − 12 AD − 12 as the graph convolution.

Original: Ever since its introduction, S sym remains the popular choice.
Revised: Ever since its introduction, S sym remains the popular choice.

Original: However, subsequent works (Wang et al., 2018; Wang & Leskovec, 2020; Ragesh et al., 2021) explore row normalization S row =
Revised: However, subsequent works (Wang et al., 2018; Wang & Leskovec, 2020; Ragesh et al., 2021) explore row normalization S row =

Original: D − 1 A and  Wang et al.
Revised: D − 1 A and particu- larly, Wang et al.

Original: (2018) observesempirically that S row outperforms S sym for two-layered GCN.
Revised: (2018) observes that S row outperforms S sym for two-layered GCN empirically.

Original: Intrigued by this observation, and as both S sym and S row are simply degree normalized adjacency matrices, we study the behavior over depth and observe that S row performs better than S sym in this case as well, as illustrated in Figure 1 (Details of the experiment in Appendix B.1).8 Depth of GCN A cc u r a c y o f c l a ss p r
Revised: Intrigued by this observation, and as both S sym and S row are simply degree normalized adjacency matrices, we study the behavior over depth and observe that S row performs better than S sym in this case as well, as illustrated in Figure 1 (Details of the experiment in Appendix B.1).

Original: e

Original: d i c t i o n ( % )

Original: S row S sym

Original: [Figure]
Revised: [Figure]

Original: S sym Furthermore, another striking observation from Figure 1 is that the performance of GCN without skip connections decreases considerably with depth for both S sym and S row .
Revised: Furthermore, another striking observation from Figure 1 is that the performance of GCN without skip connections decreases considerably with depth for both S sym and S row .

Original: This contradicts the conventional wisdom about  neural networks which exhibit improvement in the performance as depth increases.
Revised: This contradicts the conventional wisdom about standard neural networks which exhibit improvement in the performance as depth increases.

Original: Several works (Kipf & Welling, 2017; Chen et al., 2018; Wu et al., 2019) observe this behavior empirically and attribute it to the over-smoothing effect from the repeated application of the diffusion operator, resulting in averaging out of the feature information to a degree where it becomes uninformative (Li et al., 2018; Oono & Suzuki, 2019; Esser et al., 2021).
Revised: Several works (Kipf & Welling, 2017; Chen et al., 2018; Wu et al., 2019) observe this behavior empirically and attribute it to the over-smoothing effect from the repeated application of the diffusion operator, resulting in averaging out of the feature information to a degree where it becomes uninformative (Li et al., 2018; Oono & Suzuki, 2019; Esser et al., 2021).

Original: As a solution to this problem, Chen et al.
Revised: As a solution to this problem, Chen et al.

Original: and Kipf & Welling (2017) propose different forms of skip connections that overcome the smoothing effect and thus outperform the vanilla GCN.
Revised: and Kipf & Welling (2017) propose different forms of skip connections that overcome the smoothing effect and thus outperform the vanilla GCN.

Original: Extending it to the comparison of graph convolutions, our experiment shows S row is preferable to S sym over depth even in GCNs with skip connections (Figure 1).
Revised: Extending it to the comparison of graph convolutions, our experiment shows S row is preferable to S sym over depth even in GCNs with skip connections (Figure 1).

Original: Naturally, the following question arises: what characteristics of S row enable better representation learning than S sym in GCNs?
Revised: Naturally, we ask: what characteristics of S row enable better representation learning than S sym in GCNs?

Original: Rigorous theoretical analysis is particularly challenging in GCNs compared to the standard neural networks because of the graph convolution.
Revised: Rigorous theoretical analysis is particularly challenging in GCNs compared to the standard neural networks because of the graph convolution.

Original: Adding skip connections further increase the complexity of the analysis.
Revised: Adding skip connections further increase the complexity of the analysis.

Original: To overcome these difficulties, we consider GCN in infinite width limit wherein the Neural Tangent Kernel (NTK) captures the network characteristics very well (Jacot et al., 2018).
Revised: To overcome these difficulties, we consider GCN in infinite width limit wherein the Neural Tangent Kernel (NTK) captures the network characteristics very well (Jacot et al., 2018).

Original: Although the assumption on network width may appear restrictive at first, it is advantageous in our case as the role of graph convolutions remain unchanged with network width.
Revised: The infinite width assumption is not restrictive for graph convolution analysis as the convolution operates on the graph and not the network directly, thus showing same observations as trained GCN (Figure 5).

Original: Moreover, NTK enables the analysis to be parameter-free and hence eliminating additional complexity induced for example by optimization.
Revised: Moreover, NTK enables the analysis to be parameter-free and hence eliminating additional complexity induced for example by optimization.

Original: Through the lens of NTK, we study the impact of different graph convolutions under a specific data distributional assumption — Degree Corrected Stochastic Block Model (DC-SBM) (Karrer & Newman, 2011), a sparse random graph model.
Revised: Through the lens of NTK, we study the impact of different graph convolutions under a specific data distributional assumption — Degree Corrected Stochastic Block Model (DC-SBM) (Karrer & Newman, 2011), a sparse random graph model.

Original: The node degree heterogeneity induced in DC-SBM allows us to analyze the effect of different types of normalization of the adjacency matrix, thus revealing the characteristic difference between S sym and S row .
Revised: The node degree heterogeneity induced in DC-SBM allows us to analyze the effect of different types of normalization of the adjacency matrix, thus revealing the characteristic difference between S sym and S row .

Revised: Additionally, this model enables analysis of graphs that have homophilic, heterophilic and core-periphery structures.

Original: In this paper, we present a formal approach to analyze GCNs and, specifically, the representation power of different graph convolutions, the influence of depth and the role of skip connections.
Revised: In this paper, we present a formal approach to analyze GCNs and, specifically, the representation power of different graph convolutions, the influence of depth and the role of skip connections.

Original: This is a significant step toward understanding GCNs as it facilitates for more informed network design choices like thegraph convolution and depth, as well asthe development of more competitive methods based on grounded theoretical reasoning rather than heuristics.
Revised: This is a significant step toward understanding GCNs as it facilitates for more informed network design choices like the convolution and depth, as well as development of more competitive methods based on grounded theoretical reasoning rather than heuristics.

Original: Contributions.
Revised: Contributions.

Original: This paper provides rigorous theoretical analysis of the discussed empirical observations in GCN under DC-SBM distribution using graph NTK, leading to the following contributions.
Revised: This paper provides rigorous theoretical analysis of the discussed empirical observations in GCN under DC-SBM distribution using graph NTK, leading to the following contributions.

Original: (i)
Revised: (i)

Original: In Section 2, we derive the NTK for GCN in infinite width limit considering node classification setting.
Revised: In Section 2, we derive the NTK for GCN in infinite width limit considering node classification setting.

Original: Using thederived NTK for linear GCN and under DC-SBM data distribution, we show in Section 3 that S row preserves class information by computing the population NTK for different graph convolutions.
Revised: Using the NTK for linear GCN and under DC-SBM data distribution, we show in Sectionthat S row preserves class information by computing the population NTK for different graph convolutions.

Original: We also present numerical validation of the result.
Revised: We also present numerical validation of the result in homophilic and heterophilic graphs.

Original: ii)
Revised: (ii)

Original: We prove the convolution operator specific 
Revised: We prove the convolution operator specific over-smoothing effect in vanilla

Original: over-smoothing effect in vanilla GCN by showing the degradation in class separability with depth in Section 3.1, and also illustrate it experimentally.
Revised: GCN by showing the degradation in class separability with depth in Section 3.1, and also illustrate it experimentally.

Original: (iii)
Revised: (iii)

Original: In Section 4, we leverage the power of NTK to analyze two different skip connections (
Revised: In Section 4, we leverage the power of NTK to analyze two different skip connections (Kipf & Welling, 2017; Chen et al., 2020).

Original: Kipf

Original: Welling, 2017; Chen et al., We derive the corresponding NTKs and show that skip connections retain class information even at infinite depth along with numerical validation.
Revised: We derive the corresponding NTKs and show that skip connections retain class information even at infinite depth along with numerical validation.

Original: (iv) We perform empirical analysis of  the theoretical results on the most commonly used dataset Cora in Section 5 where we observe good agreement with the theory on graph convolutions, depth andskip connections. Empirical results on Citeseeris discussed in Appendix B.4.
Revised: (iv) We show that S sym maybe preferred over S row in absence of class structure in Section 5, and validate the theoretical results on real datasets Cora in Section 6 and Citeseer in Appendix B.5.

Original: We finally conclude in Section 6 with the discussion on the impact of the result and further possibilities, and provide all the proofs for the theorems with additional experiments in the appendix.
Revised: We finally conclude in Section 7 with the discussion on the impact of the result and further possibilities, and provide all the proofs, experimental details and additional experiments in the appendix.

Original: Related Work.
Revised: Related Work.

Original: While GNNs are extensively used in practice, their understanding is limited, and the analysis is mostly restricted to empirical approaches (Bojchevski et al., 2018; Zhang et al., 2018;
Revised: While GNNs are extensively used in practice, their understanding is limited, and the analysis is mostly restricted to empirical approaches (Bojchevski et al., 2018; Zhang et al., 2018;

Original: Ying et al., 2018; Wu et al., 2020).
Revised: Ying et al., 2018; Wu et al., 2020).

Original: Beyond empirical methods, rigorous theoretical analysis using learning theoretical bounds such as VC Dimension, Scarselli et al.
Revised: Beyond empirical methods, rigorous theoretical analysis using learning theoretical bounds such as VC Dimension, Scarselli et al.

Original: (2018) or PAC-Bayes Liao et al.
Revised: (2018) or PAC-Bayes Liao et al.

Original: (2021) are propounded.
Revised: (2021) are propounded.

Original: Rademacher Complexity bounds (Garg et al., 2020; Esser et al., 2021) show that normalized graph convolution is beneficial, but those works do not provide insight on the different normalizations, and their influence on the GCN performance.
Revised: Rademacher Complexity bounds (Garg et al., 2020; Esser et al., 2021) show that normalized graph convolution is beneficial, but those works do not provide insight on the different normalizations, and their influence on the GCN performance.

Original: Another possible tool is the NTK using which interesting theoretical insights in deep neural networks are derived (e.g.
Revised: Another possible tool is the NTK using which interesting theoretical insights in deep neural networks are derived (e.g.

Original: (Du et al., 2019a)).
Revised: (Du et al., 2019a)).

Original: In the context of GNNs, Du et al.
Revised: In the context of GNNs, Du et al.

Original: (2019b) derives the NTK in the supervised setting (each graph is a data instance to be classified) and empirically studies the NTK performance, however does not extend it to a theoretical analysis.
Revised: (2019b) derives the NTK in the supervised setting (each graph is a data instance to be classified) and empirically studies the NTK performance, however does not extend it to a theoretical analysis.

Original: In contrast, we derive the NTK in the semisupervised setting for GCN with and without skip connections, and use it to further theoretically analyze the influence of different convolutions with respect to over-smoothing.
Revised: In contrast, we derive the NTK in the semisupervised setting for GCN with and without skip connections, and use it to further theoretically analyze the influence of different convolutions with respect to over-smoothing.

Original: Theoretical studies (Oono & Suzuki, 2019; Cai & Wang, 2020) show that over-smoothing causes the expressive power of GNNs to decrease exponentially with depth, while Keriven (2022) proves that in linear GNNs a finite number of convolutions improves learning before over-smoothing kicks in.
Revised: Theoretical studies (Oono & Suzuki, 2019; Cai & Wang, 2020) show that over-smoothing causes the expressive power of GNNs to decrease exponentially with depth, while Keriven (2022) proves that in linear GNNs a finite number of convolutions improves learning before over-smoothing kicks in.

Original: While oversmoothing and role of skip connections in GNNs are theoretically analyzed in some works (Esser et al., 2021), the influence of different convolutions that causes over-smoothing and their interplay with skip connections is not studied.
Revised: While oversmoothing and role of skip connections in GNNs are theoretically analyzed in some works (Esser et al., 2021), the influence of different convolutions that causes over-smoothing and their interplay with skip connections is not studied.

Original: For a comprehensive theory survey see Jegelka (2022).
Revised: For a comprehensive theory survey see Jegelka (2022).

Original: Notations.
Revised: Notations.

Original: We represent matrix and vector by bold faced uppercase and lowercase letters, respectively, the matrix Hadamard (entry-wise) product by ⊙ and the scalar product by ⟨ ., .
Revised: We represent matrix and vector by bold faced uppercase and lowercase letters, respectively, the matrix Hadamard (entry-wise) product by ⊙ and the scalar product by ⟨ ., .

Original: We use M ⊙ k to denote Hadamard product of matrix M with itself repeated k times.
Revised: We use M ⊙ k to denote Hadamard product of matrix M with itself repeated k times.

Original: Let N ( µ, Σ ) be Gaussian distribution with mean µ and co-variance Σ .
Revised: Let N ( µ, Σ ) be Gaussian distribution with mean µ and co-variance Σ .

Original: We use ˙ σ ( . ) to represent derivative of function σ ( . ) , 1 n × n for the n × n matrix of ones, I n for identity matrix of size n × n , 1 [ . ] for indicator function, E [ . ] for expectation, and [ d ] = { 1 , 2 , . . .
Revised: We use ˙ σ ( . ) to represent derivative of function σ ( . ) , 1 n × n for the n × n matrix of ones, I n for identity matrix of size n × n , 1 [ . ] for indicator function, E [ . ] for expectation, and [ d ] = { 1 , 2 , . . .

Original: , d } .
Revised: , d } .

Original:  N EURAL T ANGENT K ERNEL FOR G RAPH C ONVOLUTIONAL N ETWORK
Revised:  N EURAL T ANGENT K ERNEL FOR G RAPH C ONVOLUTIONAL N ETWORK

Original: Before going into a detailed analysis of graph convolutions we provide a brief background on Neural Tangent Kernel (NTK) and derive its formulation in the context of node level prediction using infinitely-wide GCNs.
Revised: Before going into a detailed analysis of graph convolutions we provide a brief background on Neural Tangent Kernel (NTK) and derive its formulation in the context of node level prediction using infinitely-wide GCNs.

Original: Jacot et al.
Revised: Jacot et al.

Original: Arora et al.
Revised: Arora et al.

Original: ; Yang (2019) show that the behavior and generalization properties of randomly initialized wide neural networks trained by gradient descent with infinitesimally small learning rate is equivalent to a kernel machine.
Revised: ; Yang (2019) show that the behavior and generalization properties of randomly initialized wide neural networks trained by gradient descent with infinitesimally small learning rate is equivalent to a kernel machine.

Original: Furthermore, Jacot et al.
Revised: Furthermore, Jacot et al.

Original: (2018) also show that the change in the kernel during training decreases as the network width increases, and hence, asymptotically, one can represent an infinitely wide neural network by a deterministic NTK, which is defined by the gradient of the network with respect to its parameters as
Revised: (2018) also show that the change in the kernel during training decreases as the network width increases, and hence, asymptotically, one can represent an infinitely wide neural network by a deterministic NTK, which is defined by the gradient of the network with respect to its parameters as

Original: [Equation]
Revised: [Equation]

Original: Here F ( W , x ) represents the output of the network at data point x parameterized by W and the expectation is with respect to W , where all the parameters of the network are randomly sampled from
Revised: Here F ( W , x ) represents the output of the network at data point x parameterized by W and the expectation is with respect to W , where all the parameters of the network are randomly sampled from

Original: Gaussian distribution.
Revised: Gaussian distribution.

Original: Although the ‘infinite width’ assumption is too strong to model real (finite width) neural networks, and the absolute performance may not exactly match, the empirical trends of NTK match the corresponding network counterpart, allowing us to draw insightful conclusions.
Revised: Although the ‘infinite width’ assumption is too strong to model real (finite width) neural networks, and the absolute performance may not exactly match, the empirical trends of NTK match the corresponding network counterpart, allowing us to draw insightful conclusions.

Original: This trade-off is worth considering as this allows the analysis of over-parameterised neural networks without having to consider hyper-parameter tuning and training.
Revised: This trade-off is worth considering as this allows the analysis of over-parameterised neural networks without having to consider hyper-parameter tuning and training.

Original: Formal GCN Setup and Graph NTK.
Revised: Formal GCN Setup and Graph NTK.

Original: We present the formal setup of GCN and derive the corresponding NTK, using which we analyze different graph convolutions.
Revised: We present the formal setup of GCN and derive the corresponding NTK, using which we analyze different graph convolutions.

Original: Given a graph with n nodes and a set of node features { x i } ni =1 ⊂ R f , we may assume without loss of generality that the set of observed labels { y i } mi =1 correspond to first m nodes.
Revised: Given a graph with n nodes and a set of node features { x i } ni =1 ⊂ R f , we may assume without loss of generality that the set of observed labels { y i } mi =1 correspond to first m nodes.

Original: We consider K classes, thus y i ∈ { 0 , 1 } K and the goal is to predict the n − m unknown labels { y i } n i
Revised: We consider K classes, thus y i ∈ { 0 , 1 } K and the goal is to predict the n − m unknown labels { y i } n i

Original: = m
Revised: = m

Original: +1 .
Revised: +1 .

Original: We represent the observed labels of m nodes as Y ∈ { 0 , 1 } m × K , and the node features as X ∈ R n × f with the assumption that entire
Revised: We represent the observed labels of m nodes as Y ∈ { 0 , 1 } m × K , and the node features as X ∈ R n × f with the assumption that entire

Original: X is available during training.
Revised: X is available during training.

Original: We define S ∈ R n × n to be the graph convolution operator as an expression of the adjacency matrix A and the degree matrix D .
Revised: We define S ∈ R n × n to be the graph convolution operator as an expression of the adjacency matrix A and the degree matrix D .

Original: The GCN of depth d is given by
Revised: The GCN of depth d is given by

Original: [Equation]
Revised: [Equation]

Original: where W := { W i ∈
Revised: where W := { W i ∈

Original: R h i − 1 × h i } d
Revised: R h i − 1 × h i } d

Original: +1 i =1 is the set of learnable weight matrices with h 0 = f and h d +1 =
Revised: +1 i =1 is the set of learnable weight matrices with h 0 = f and h d +1 =

Original: K , h
Revised: K , h

Original: i is the size of layer i ∈
Revised: i is the size of layer i ∈

Original: [ d ] and σ : R → R is the point-wise activation function.
Revised: [ d ] and σ : R → R is the point-wise activation function.

Original: We initialize all the weights to be i.i.d N (0 , 1) and optimize it using gradient descent.
Revised: We initialize all the weights to be i.i.d N (0 , 1) and optimize it using gradient descent.

Original: We derive the
Revised: We derive the

Original: NTK for the GCN in infinite width setting, that is, h 1 , . .
Revised: NTK for the GCN in infinite width setting, that is, h 1 , . .

Original: , h d → ∞ .
Revised: , h d → ∞ .

Original: While this setup is similar to Kipf & Welling (2017), it is important to note that we consider linear output layer so that NTK remains constant during training (Liu et al., 2020) and additionally add a normalization (cid:112) c σ /h
Revised: While this setup is similar to Kipf & Welling (2017), it is important to note that we consider linear output layer so that NTK remains constant during training (Liu et al., 2020) and additionally add a normalization (cid:112) c σ /h

Original: i for layer i to ensure that the input norm is approximately preserved and c − 1 σ
Revised: i for layer i to ensure that the input norm is approximately preserved and c − 1 σ

Original: [Equation]
Revised: [Equation]

Original: (similar to Du et al. (2019a)).
Revised: (similar to Du et al. (2019a)).

Original: The following theorem states the NTK between every pair of nodes, as a n × n matrix that can be computed at once, as shown below.
Revised: The following theorem states the NTK between every pair of nodes, as a n × n matrix that can be computed at once, as shown below.

Original: Theorem 1 (NTK for Vanilla GCN)
Revised: Theorem 1 (NTK for Vanilla GCN)

Original: For the vanilla GCNas defined in (2) , the NTK Θ at depth d isgiven by
Revised: For the vanilla GCN defined in (2) , the NTK Θ at depth d is

Original: [Equation]
Revised: [Equation]

Original: Here Σ k ∈ R n × n is the co-variance between nodes of layer k , and is given by Σ 1 = SXX T S T ,
Revised: Here Σ k ∈ R n × n is the co-variance between nodes of layer k , and is given by Σ 1 = SXX T S T ,

Original: Σ k = SE k − 1 S T with E k =
Revised: Σ k = SE k − 1 S T with E k =

Original: c σ
Revised: c σ

Original: E F ∼N ( 0 , Σ k ) (cid:2) σ ( F ) σ ( F ) T (cid:3) and ˙ E k =
Revised: E F ∼N ( 0 , Σ k ) (cid:2) σ ( F ) σ ( F ) T (cid:3) and ˙ E k =

Original: c σ
Revised: c σ

Revised: E

Original: [Equation]
Revised: [Equation]

Original: ˙ σ ( F ) ˙ 
Revised: F [Equation] ˙ σ

Revised: [Equation]

Original: σ ( F ) T (cid:3) .
Revised: F [Equation] T (cid:3) .

Original: Comparison to Du et al. (2019b).
Revised: Comparison to Du et al. (2019b).

Original: The derived NTK  is similar to the graph NTK in Du et al.
Revised: While the NTK in (3) is similar to the graph NTK in Du et al.

Original: (2019b) with the primary difference being that the kernel in our case is computed for all pairs of nodes in asingle input graph as we focus on semi-supervised node classification, whereas Du et al.
Revised: (2019b), the main difference is that NTK in our case is computed for all pairs of nodes in a graph as we focus on semi-supervised node classification, whereas Du et al.

Original: (2019b) considers supervised graph classification where multiple graphsare inputs and so the kernel is evaluated for all pairs ofthe graphs.
Revised: (2019b) considers supervised graph classification where input is many graphs and so the NTK is evaluated for all pairs of graphs.

Original:  C ONVOLUTION O PERATOR S row P RESERVES C LASS I NFORMATION
Revised:  C ONVOLUTION O PERATOR S row P RESERVES C LASS I NFORMATION

Original: We use the derived NTK in Theorem 1 to analyze different graph convolutions for S defined in
Revised: We use the derived NTK in Theorem 1 to analyze different graph convolutions for S defined in

Original: Definition 1 by making the following assumption on the network.
Revised: Definition 1 by making the following assumption on the network.

Original: Assumption 1 (Linear GCN with orthonormal features) GCN in (2) is said to be linear with orthonormal features if the activation function σ ( x ) = x and XX T
Revised: Assumption 1 (Linear GCN with orthonormal features) GCN in (2) is said to be linear with orthonormal features if the activation function σ ( x ) = x and XX T

Original: =
Revised: =

Original: I n .
Revised: I n .

Original: Remark on Assumption 1.
Revised: Remark on Assumption 1.

Original: The linear activation does not impact the performance of a GCN significantly as Wu et al.
Revised: The linear activation does not impact the performance of a GCN significantly as Wu et al.

Original: (2019) empirically demonstrates that the linearized GCN performance is at par with the non-linear models with much reduced complexity.
Revised: (2019) empirically demonstrates that the linearized GCN performance is at par with the non-linear models with much reduced complexity.

Original: Additional orthonormal features assumption eliminates the influence of the features and facilitates identification of the influence of different convolution operators.
Revised: Additional orthonormal features assumption eliminates the influence of the features and facilitates identification of the influence of different convolution operators.

Original: Besides, the evaluation of our theoretical results without this assumption on real datasets is presented in Section 5 and Appendix B.4 that substantiate our findings.
Revised: Besides, the evaluation of our theoretical results without this assumption on real datasets is presented in Section 6 and Appendix B.5 that substantiate our findings.

Original: Therefore, the NTK for linear GCN with orthonormal features of depth d is,
Revised: Therefore, the NTK for linear GCN with orthonormal features of depth d is,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: We consider the following graph convolutions for the analysis.

Original: Definition 1 Symmetric degree normalized S sym
Revised: Definition 1 Symmetric degree normalized S sym

Original: = D − 12 AD − 12 , row normalized S row =
Revised: = D − 12 AD − 12 , row normalized S row =

Original: D − 1 A , column normalized S col
Revised: D − 1 A , column normalized S col

Original: =
Revised: =

Original: AD − 1 and unnormalized S adj = 1 n A convolutions.
Revised: AD − 1 and unnormalized S adj = 1 n A convolutions.

Original: While the NTK in (4) gives a precise characterization of the infinitely wide GCN, we can not directly draw conclusions about the convolution operators without further assumptions on the input graph.
Revised: While the NTK in (4) gives a precise characterization of the infinitely wide GCN, we can not directly draw conclusions about the convolution operators without further assumptions on the input graph.

Original: Therefore, we consider a planted graph model as described below, that helps in establishing the exact representation power of each operator.
Revised: Therefore, we consider a planted graph model as described below, that helps in establishing the exact representation power of each operator.

Original: Random Graph Model.
Revised: Random Graph Model.

Original: We consider that the underlying graph is from the Degree Corrected
Revised: We consider that the underlying graph is from the Degree Corrected

Original: Stochastic Block Model (DC-SBM) (Karrer & Newman, 2011) since it enables us to distinguish between S sym , S row and S col  by allowing non-uniform degree distribution on the nodes.
Revised: Stochastic Block Model (DC-SBM) (Karrer & Newman, 2011) since it enables us to distinguish between S sym , S row , S col and S adj by allowing non-uniform degree distribution on the nodes.

Original: The model is defined as follows: Consider a set of n nodes divided into K latent classes (or communities), C i ∈ [1 , K ] .
Revised: The model is defined as follows: Consider a set of n nodes divided into K latent classes (or communities), C i ∈ [1 , K ] .

Original: The DC-SBM model is characterized by the parameters 0 ≤ q < p ≤ 1 — governing the edge probabilitiesoutside and inside  classes—and the degree correction vector π
Revised: The DC-SBM model is characterized by the parameters p, q ∈ [0 , 1] — governing the edge probabilities inside and outside classes—and the degree correction vector π

Original: =
Revised: =

Original: ( π 1 , . . .
Revised: ( π 1 , . . .

Original: , π n ) ∈ [0 , 1] n with (cid:80)
Revised: , π n ) ∈ [0 , 1] n with (cid:80)

Original: i π i
Revised: i π i

Original: = 1 .
Revised: = 1 .

Original: A random graph on n nodes, generated from DCSBM, has mutually independent edges with edge probabilities specified by the population adjacency matrix M = E
Revised: A random graph on n nodes, generated from DCSBM, has mutually independent edges with edge probabilities specified by the population adjacency matrix M = E

Original: [ A ] ∈ R n × n , where
Revised: [ A ] ∈ R n × n , where

Original: [Equation]
Revised: [Equation]

Revised: This allows us to model different graph types:

Revised: Homophilic graphs: 0 ≤ q < p ≤ 1 , Heterophilic graphs: 0 ≤ p < q ≤ 1 and Core-Periphery graphs: p = q (no assumption on class structure) and π encode core and periphery.

Original: It is evident that the NTK is a complex quantity and computing its expectation is challenging given the dependency of terms from the degree normalization in S , its powers S i and SS T .
Revised: It is evident that the NTK is a complex quantity and computing its expectation is challenging given the dependency of terms from the degree normalization in S , its powers S i and SS T .

Original: To simplify our analysis, we make the following assumption on DC-SBM,
Revised: To simplify our analysis, we make the following assumption on DC-SBM,

Original: Assumption 2 (Population DC-SBM)
Revised: Assumption 2 (Population DC-SBM)

Original: The graph has a weighted adjacencyA = M , and π is chosen such that (cid:80) ni
Revised: The graph has a weighted adjacency

Original: =1 π
Revised: A = M .

Original: i 1 [ C i = k ] = 1 K and (cid:80) ni =1 π 2 i 1 [ C i = k ] = γ ∀ k , where γ is a constant.

Original: Remark on Assumption 2.
Revised: Remark on Assumption 2.

Original: Assuming A
Revised: Assuming A

Original: = M is equivalent to analyzing DC-SBM in expected setting and it further enables the computation of analytic expression for the population NTK instead of the expected NTK.
Revised: = M is equivalent to analyzing DC-SBM in expected setting and it further enables the computation of analytic expression for the population NTK instead of the expected NTK.

Original: Moreover, we observe empirically that this analysis hold for random DC-SBM setting as well.
Revised: Moreover, we observe empirically that this analysis hold for random DC-SBM setting as well.

Original: In addition, this consideration also implies addition of self loop with a probability.
Revised: In addition, this consideration also implies addition of self loop with a probability p .

Original: The other two assumptions on π are only to express the kernel in a simplified, easy to comprehend format.

Original: It is derived without the assumption on γ (Appendix A.2.2). Furthermore, the numerical validation of our result is without both the assumptions on π and γ (Section 3.2). In the following theorem, we state the population NTK fordifferent graph convolutions S  for K =with Assumption 1 and 2, but the result extends to K > 2 as discussed in the appendix.
Revised: In the following theorem, we state the population NTK for graph convolutions S sym , S row , S col and S adj for K = 2 with Assumption 1 and 2. The result extends to K > 2 as discussed in the appendix.

Original: Theorem 2 (Population NTKs ˜Θ for the four graph convolutions S ) Let Assumption 1 andhold
Revised: Theorem 2 (Population NTKs ˜Θ for the four graph convolutions S ) Let Assumption 1 andhold, K = 2 and r =

Original: [Equation]{+p − q p + q , δ ij = ( − 1) 
 [ C i ̸ = C j ] . Furthermore, π is chosen such that (cid:80) ni+}
Revised: {+p − q p + q , δ ij = ( − 1) 
 [ C i ̸ = C j ] . Furthermore, π is chosen such that (cid:80) ni+}

Original: and K = 2 , r = p − q p + q , δ ij = ( − 1) 1 [ C i ̸
Revised: =1 π i 1 [ C i = k ] = 1 K and (cid:80) ni =1 π 2 i 1 [ C i = k ] = γ ∀ k , where γ is a constant.

Original: = C j ] , then ∀ i and j , population NTKs ˜Θ sym , ˜Θ row ,˜Θ col and ˜Θ adj of depth d for S = S sym , S row , S col and S adj, respectively, are, 
Revised: Then ∀ i, j , population NTKs ˜Θ sym , ˜Θ row , of depth d for S = S sym , S row , S col and S adj respectively, are, [Equation]

Revised: ) sym

Original: [Equation]
Revised: [Equation]

Revised: ) row

Revised: [Equation]

Revised: 1 −

Revised: [Equation]

Revised: Note that the two assumptions on π are only to express the kernel in a simplified, easy to comprehend format.

Revised: It is derived without the assumptions on π in Appendix A.2.2.

Revised: Furthermore, the numerical validation of our result is without both these assumptions (Section 3.2).

Original: Comparison of graph convolutions.
Revised: Comparison of graph convolutions.

Original: The population NTKs ˜Θ ( d ) of depth d in Theorem 2 describes the information that the kernel has after d convolutions with S .
Revised: The population NTKs ˜Θ ( d ) of depth d in Theorem 2 describes the information that the kernel has after d convolutions with S .

Original: To classify the nodes perfectly, the kernel should ideally have a block structure that aligns with the DC-SBM  , showing class separability , that is, gap between in-class and out-of-class blocks proportional to p − q .
Revised: To classify the nodes perfectly, the kernel should ideally have a block structure that aligns with the DC-SBM ( p and q blocks) unaffected by degree correction π , showing class separability , that is, gap between in-class and out-of-class blocks proportional to p − q .

Original: On this basis, only ˜ Θ row exhibits a block structure unaffected by the degree correction π , and the gap is determined by r 2 and d , making S row preferable over S sym , S adj and S col .
Revised: On this basis, only ˜ Θ row exhibits a block structure unaffected by the degree correction π , and the gap is determined by r 2 and d , making S row preferable over S sym , S adj and S col .

Original: On the other hand, ˜Θ sym , ˜Θ col and ˜Θ adj are influenced by the degree correction which obscures the class information especially with depth.
Revised: On the other hand, ˜Θ sym , ˜Θ col and ˜Θ adj are influenced by the degree correction which obscures the class information especially with depth.

Original: Although ˜Θ sym and ˜Θ col seem similar, ˜Θ col is additionally influenced by the number of nodes n in the graph, making it undesirable over S sym .
Revised: Although ˜Θ sym and ˜Θ col seem similar, ˜Θ col is additionally influenced by the number of nodes n in the graph, making it undesirable over S sym .

Original: As a result, the preference order from the theory is ˜Θ row > ˜Θ sym > ˜Θ col > ˜Θ adj .
Revised: As a result, the preference order from the theory is ˜Θ row ≻ ˜Θ sym ≻ ˜Θ col ≻ ˜Θ adj .

Original: 3. I MPACT OF D EPTH IN V ANILLA GCN
Revised: 3. I MPACT OF D EPTH IN V ANILLA GCN

Original: Given that r = p − q p + q < 1 , Theorem 2 shows that the difference between in-class and out-of-class blocks decreases with depth monotonically which in turn leads to decrease in performance with depth, therefore explaining theempirical observation in Figure 1.
Revised: Given that r = p − q p + q < 1 , Theorem 2 shows that the difference between in-class and out-of-class blocks decreases with depth monotonically which in turn leads to decrease in performance with depth, therefore explaining the observation in Figure 1.

Original: Moreover, the kernel converges to an almost constant kernelas d → ∞ in all the four cases as shown in the following corollary.
Revised: The kernel in the limit is stated below.

Original: Corollary 1 (Population NTK ˜Θ ( ∞ ) as d → ∞ ) From Theorem 2, (cid:16) ˜Θ ( ∞ ) adj (cid:17) ij = 0 and ∀ i, j for conv ∈ { sym, row, col } , (cid:16) ˜Θ ( ∞ ) conv (cid:17) ij = ν ij 1 − ν ij (1 + δ ij r 2 ) where ν
Revised: Corollary 1 (Population NTK ˜Θ ( ∞ ) as d → ∞ ) From Theorem 2, (cid:16) ˜Θ ( ∞ ) adj (cid:17) ij = 0 and ∀ i, j for conv ∈ { sym, row, col } , (cid:16) ˜Θ ( ∞ ) conv (cid:17) ij = ν ij 1 − ν ij (1 + δ ij r 2 ) where ν

Original: ij = √ π i π j for sym , ν ij = 2 γ for row and ν ij =
Revised: ij = √ π i π j for sym , ν ij = 2 γ for row and ν ij =

Original: nπ i π j for col .
Revised: nπ i π j for col .

Original: From the corollary, we infer that the class separability at infinite depth is 0 for S adj , and O ( r 2 ) for
Revised: From the corollary, we infer that the class separability at infinite depth is 0 for S adj , and O ( r 2 ) for

Original: S sym , S row and S col showing that the large depth GCN has very little to zero class information.
Revised: S sym , S row and S col showing that the large depth GCN has very little to zero class information.

Original: To further understand the impact of depth, we plot the average in-class and out-of-class block difference  using the theoretically derived population NTK ˜Θ ( d ) for depths [1 , 10] in a well separatedunderlying DC-SBM with p = 0 .
Revised: To further understand this, we plot the average in-class and out-of-class block difference for homophilic and heterophilic graphs using the theoretically derived population NTK ˜Θ ( d ) for depths [1 , 10] in a well separated DC-SBM (Column 4 of Figure 2).

Original: 8 and q = 0 . The line plot in column 1 of Figure 2 shows theresult which demonstrates the rapid degradation of class separability with depth and the gap converges to 0 for large depths in all the four convolutions.
Revised: It clearly shows the rapid degradation of class separability with depth and the gap goes to 0 for large depths in all the four convolutions.

Original: Additionally, the gap in ˜Θ ( d ) row is the highest showing that the class information is better preserved, illustrating the strong representation power of S row .
Revised: Additionally, the gap in ˜Θ ( d ) row is the highest showing that the class information is better preserved, illustrating the strong representation power of S row .

Original: Consequently, large depth is undesirable for all analyzed graph convolutions in vanilla GCN and the theory suggests S row as the best choice for shallow GCN.
Revised: Consequently, large depth is undesirable for all the four graph convolutions in vanilla GCN and the theory suggests S row as the best choice for shallow GCN.

Original: 3. N UMERICAL V ALIDATION FOR 
Revised: 3. N UMERICAL V ALIDATION FOR R ANDOM G RAPHS

Original: Theorem 2 and Corollary 1 show that S row has better representation power under Assumptionand 2, that is, for the linear GCN with orthonormal features and population DC-SBM.
Revised: Theorem 2 and Corollary 1 show that S row has better representation power under Assumption 1 and 2, that is, for the linear GCN with orthonormal features and population DC-SBM.

Original: In Figure 2, the heatmaps show that the same holds for a random graph generated from DC-SBM with p = 0 .
Revised: We validate this on homophilous and heterophilous random graphs generated from DC-SBM shown in column 1 of

Original: 8 , q = 0 .
Revised: Figure 2.

Original: 1 and π i ∼ Unif(0 , 1) using ReLU GCN. A graph of n = 1000 nodes  is sampled from 
Revised: A graph of n = 1000 nodes with equal sized classes is sampled from each DC-SBM, respectively.

Original: DC-SBM which is shown in the first plot of column 1 in Figure 2, such that first n 2 nodes belong to class 1 and the rest to class 2 for clear visualization. We make the following observations on the exact NTKs of S sym and S row shown in column 2 and 3 , respectively, thatvalidates the results derived from population NTK, (i) for depth= 1 , the class information for all  nodes is well preserved in S row as there is a clear block structure than S sym in which each node is diffused unequally due to the degree correction, (ii) as depth increases to 8 , S row still has block structure although the gap between in-class and out-of-class blocks reduced in comparison to depth= 1 , whereas the influence of degree correction takes over the class information in S sym and becomes uninformative of the classes.
Revised: The heatmaps for depth= 2 in the case of both homophily and heterophily graphs show that the class information for all the nodes is well preserved in S row as there is a clear block structure than S sym in which each node is diffused unequally due to the degree correction. Thus validating the results derived from population NTK.

Original: Appendix B.2 presents the results for S adj and S col where both are uninformative and behave as derived theoretically.
Revised: Appendix B.3 presents the results for S adj and S col where both are uninformative and behave as derived theoretically.

Original:  S KIP C ONNECTIONS R ETAIN I NFORMATION E VEN AT I NFINITE D EPTH
Revised:  S KIP C ONNECTIONS R ETAIN I NFORMATION E VEN AT I NFINITE D EPTH

Original: Skip connection is the most common way to overcome the performance degradation with depth in
Revised: Skip connection is the most common way to overcome the performance degradation with depth in

Original: GCNs, but little is known about the effectiveness of differentforms of available skip connections and their interplay with the convolutional operators.
Revised: GCNs, but little is known about the effectiveness of different skip connections and their interplay with the convolutions.

Original: While there is a number of different formulations of skip connections, we consider the following two variants: Skip-PC (pre-convolution), where the skipconnection is added to the features before applying convolution (Kipf & Welling, 2017); and Skip- α , which gives importance to the features by adding it to each layer without convolving with S (Chen et al., 2020).
Revised: While our focus is to understand the interplay with convolutions, we also include the impact of convolving with and without the feature information. Hence, we consider the following two variants: Skip-PC (pre-convolution), where the skip is added to the features before applying convolution (Kipf & Welling, 2017); and Skip- α , which gives importance to the features by adding it to each layer without convolving with S (Chen et al., 2020).

Original: To facilitate skip connections, we need to enforce constant layer size, that is, h i
Revised: To facilitate skip connections, we need to enforce constant layer size, that is, h i

Original: = h
Revised: = h

Original: i − 1 .
Revised: i − 1 .

Original: Therefore, we transform the input layer using a random matrix W to H 0 =
Revised: Therefore, we transform the input layer using a random matrix W to H 0 =

Original: XW of size n × h where W ij ∼ N (0 , 1) and h is the hidden layer size.
Revised: XW of size n × h where W ij ∼ N (0 , 1) and h is the hidden layer size.

Original: Let H i be the output of layer i using which we formally define the skip connections as follows.
Revised: Let H i be the output of layer i .

Original: Definition 2 (Skip-PC)
Revised: Definition 2 (Skip-PC)

Original: In a Skip-PC (pre-convolution) network, the transformed input His added to the hidden layers before applying the graph convolution S , that is, H i :=
Revised: In a Skip-PC (pre-convolution) network, the transformed input His added to the hidden layers before applying the graph convolution S , that is, H i :=

Original: (cid:112) c σ h S ( H i − 1 + σ s ( H 0 ))
Revised: (cid:112) c σ h S ( H i − 1 + σ s ( H 0 ))

Original: W i ∀ i ∈
Revised: W i ∀ i ∈

Original: [ d ] , where σ s ( . ) can be linear or ReLU.
Revised: [ d ] , where σ s ( . ) can be linear or ReLU.

Original: The above definition deviates from Kipf & Welling (2017) in the fact that we skip to the input layer instead of the previous layer.
Revised: The above definition deviates from Kipf & Welling (2017) in the fact that we skip to the input layer instead of the previous layer.

Original: The following defines the skip connection similar to Chen et al.
Revised: The following defines the skip connection similar to Chen et al.

Original: Definition 3 (Skip- α )
Revised: Definition 3 (Skip- α )

Original: Given an interpolation coefficient α ∈ (0 , 1) , a Skip- α network is defined such that the transformed input H 0 and the hidden layer are interpolated linearly, that is, H i := [Equation] σ [Equation]
Revised: Given an interpolation coefficient α ∈ (0 , 1) , a Skip- α network is defined such that the transformed input H 0 and the hidden layer are interpolated linearly, that is, H i := (cid:112) c σ h ((1 − α )

Original: W [Equation]
Revised: SH i − 1 + ασ s ( H 0 ))

Original:  ∀ i ∈
Revised: W i ∀ i ∈

Original: [ d ] , where σ s ( . ) can be linear or ReLU.
Revised: [ d ] , where σ s ( . ) can be linear or ReLU.

Original: 4. NTK FOR GCN WITH S KIP C ONNECTIONS
Revised: 4. NTK FOR GCN WITH S KIP C ONNECTIONS

Original: We derive NTKs for the skip connections – Skip-PC and Skip- α by considering the hidden layers width h → ∞ .
Revised: We derive NTKs for the skip connections – Skip-PC and Skip- α by considering the hidden layers width h → ∞ .

Original: Both the NTKs maintain the form presented in Theorem 1 with the following changes to the co-variance matrices.
Revised: Both the NTKs maintain the form presented in Theorem 1 with the following changes to the co-variance matrices.

Original: Let ˜E 0 =
Revised: Let ˜E 0 =

Original: [Equation]
Revised: [Equation]

Original: Corollary 2 (NTK for Skip-PC)
Revised: Corollary 2 (NTK for Skip-PC)

Original: The NTK for an infinitely wide Skip-PC network is as presented in Theorem 1 where E k is defined as in the theorem, but Σ k is defined as
Revised: The NTK for an infinitely wide Skip-PC network is as presented in Theorem 1 where E k is defined as in the theorem, but Σ k is defined as

Original: [Equation]
Revised: [Equation]

Original: and 
Revised: and Σ

Revised: [Equation]

Revised: Σ

Original: [Equation]
Revised: [Equation]

Original: Corollary 3 (NTK for Skip- α )
Revised: Corollary 3 (NTK for Skip- α )

Original: The NTK for an infinitely wide
Revised: The NTK for an infinitely wide

Original: Skip- α network is as presented in
Revised: Skip- α network is as presented in

Original: Theorem 1 where E k is defined as in the theorem, but Σ k is defined with Σ 0 =
Revised: Theorem 1 where E k is defined as in the theorem, but Σ k is defined with Σ 0 =

Original: XX T , 
Revised: XX T , Σ

Original: [Equation]
Revised: [Equation]

Original:  +α 2 E0 and Σ k
Revised: SE [Equation] + E

Original:  = (1 − α ) 2 SE k − 1 S T + α 2 ˜ E 0 .
Revised: [Equation] S T (cid:1) + α 2 E 0 and Σ k = (1 − α ) 2 SE k − 1 S T + α 2 ˜E 0 .

Original: 4. I MPACT OF D EPTH IN GCN S WITH S KIP C ONNECTION
Revised: 4. I MPACT OF D EPTH IN GCN S WITH S KIP C ONNECTION

Original: Similar to the previous section we use the NTK for Skip-PC and Skip- α (Corollary 2 and 3) and analyze the graph convolutions S sym and S adj under the same considerations detailed in Section 3.
Revised: Similar to the previous section we use the NTK for Skip-PC and Skip- α (Corollary 2 and 3) and analyze the graph convolutions S sym and S adj under the same considerations detailed in Section 3.

Original: Since, S adj and S col are theoretically worse and not popular in practice, we do not consider them for the skip connection analysis.
Revised: Since, S adj and S col are theoretically worse and not popular in practice, we do not consider them for the skip connection analysis.

Original: The linear orthonormal feature NTK, Θ ( d ) , for depth d is same as (4) with changes to Σ k as follows,
Revised: The linear orthonormal feature NTK, Θ ( d ) , for depth d is same as (4) with changes to Σ k as follows,

Original: [Equation]
Revised: [Equation]

Original: We derive the population NTK ˜Θ ( d ) and, for convenience, only state the result as d → ∞ in the following theorems.
Revised: We derive the population NTK ˜Θ ( d ) and, for convenience, only state the result as d → ∞ in the following theorems.

Original: Theorem 3 (Population NTK for Skip-PC ˜Θ ( ∞ ) PC ) Let Assumption 1 and 2 hold, and K = 2 , r = p − q p + q , δ ij = ( − 1) 1 [ C i ̸ = C j ] , then ∀ i, j ,
Revised: Theorem 3 (Population NTK for Skip-PC ˜Θ ( ∞ ) PC ) Under the assumptions of Theorem 2,

Original: [Equation]
Revised: [Equation]

Original: and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Theorem 4 (Population NTK for Skip-
Revised: Theorem 4 (Population NTK for Skip-

Original: α ˜Θ ( ∞ ) α ) Let Assumptions 1 and 2 hold
Revised: α ˜Θ ( ∞ ) α ) Under the assumptions of Theorem 2,

Original: [Equation]
Revised: [Equation]

Original: and K = 2 , r = p − q p + q , δ ij = ( − 1) 1 [ C i ̸

Original: = C j ] ,

Original: [Equation]

Original: , and
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Similar to Theorem 2, assumptions on πand γ in above theorems are to simplify the results and
Revised: Similar to Theorem 2, assumptions on π in above theorems is to simplify the results.

Original: Appendix A.5 discusses it without the assumption on γ , likewise the numerical validation in Section 4.3.

Original: To understand the role of skip connections, we plot the gap between in-class and out-ofclass blocks at ∞ depth for different values of true class separability r , for vanilla GCN, Skip-PC and Skip-α using Corollary 1, Theorems 3 and 4, respectively.
Revised: To understand the role of skip connections, we plot the gap between in-class and out-of-class blocks at infinite depth for different values of true class separability r , for vanilla GCN, Skip-PC and Skip-

Original:  Theleft plotof Figure 3 shows this and clearly illustrates that the gap is away from 0 for both the skip connections given a reasonable true separation, unlike vanilla GCN.
Revised: α using Corollary 1, Theorems 3–4, respectively (Figure 3). The plot clearly shows that the gap is away from 0 for both the skip connections given a reasonable true separation, unlike vanilla GCN.

Original: This implies theprevalence of class information by skip connections even at infinite depth.
Revised: This implies the class information is retained in skip connections even at infinite depth.

Original: 4. N UMERICAL V ALIDATION FOR R ANDOM G RAPHS
Revised: 4. N UMERICAL V ALIDATION FOR R ANDOM G RAPHS

Original: We validate our theoretical result using the same setup detailed in Section 3.2 without the assumptions, and compute the exact NTK for Skip-PC and Skip- αGCNs for both S sym and S row .
Revised: We validate our theoretical result using the same setup detailed in Section 3.2 without the assumptions, and compute the exact NTKs for Skip-PC and Skip- α for both S sym and S row .

Revised: We show the result on homophilic graphs but they equally extend to the heterophilic case.

Original: Right plot of Figure 3 illustrates the result for  Skip-PC where we  observethat thegap between in-class and out-of-class blocks decreases but does not vanish with depth, thus retaining the class information for large depths as well.
Revised: While S sym has no class information for depth= 8 in vanilla GCN (Figure 3 middle), it is retained well in Skip-PC (right plot). In the case of S row , we clearly observe the blocks in both cases with more prevalent gap in Skip-PC illustrating our theoretical results.

Original: Similar observation is made for Skip-
Revised: Similar observation is made for Skip-

Original: α despite considering XX T =
Revised: α despite considering XX T =

Original: I n and the model interpolates with the feature, and is discussed in Appendix B.2.
Revised: I n as the model interpolates with the feature, and is discussed in Appendix B.3.

Original: Thus, the experiment numerically validates our theory even in  the setting without the assumptions.
Revised: While both S sym and S row retain the class information in larger depths, we observe that the degree correction plays a significant role in S sym as elucidated in our theoretical analysis.  S sym M AYBE P REFERRED O VER S row IN A BSENCE OF C LASS S TRUCTURE While we showed that the graph convolution S row preserves the underlying class structure, it is natural to wonder about the random graphs that have no communities ( p = q ). One such case is graphs with core-periphery structure where the graph has core nodes that are highly interconnected and periphery nodes that are sparsely connected to the core and other periphery nodes.

Original: While both S sym and S row retain the class information in larger depths, we observe experimentally that the degree correction plays a significant role in S sym compared to S row as elucidated in our theoretical analysis.
Revised: Such a graph can be modeled using only the degree correction π such that π j ≪ π i ∀ j ∈ periphery, i ∈ core (similar to Jia & Benson (2019)). Extending Theorem 2, we derive the following Corollary 4 and show that the convolution S sym contains the graph information while S row is a constant kernel.

Original: E MPIRICAL A NALYSIS ON R EAL D ATA
Revised: Corollary 4 (Population NTKs ˜ Θ for p = q ) Let Assumption 1 and 2 hold, K = 2 and p = q . Furthermore, π is chosen such that (cid:80) i ∈ core π 2 i = λ and (cid:80) i ∈ periphery π 2 i = µ .

Revised: Then ∀ i and j , the population NTKs ˜Θ sym and ˜Θ row of depth d for S

Revised: = S sym and S row , respectively, are,

Revised: [Equation]

Revised: and

Revised: [Equation]

Revised: 1 − ( λ + µ ) .

Revised: From Corollary 4, it is evident that the S sym has the graph information and hence could be preferred when there is no community structure.

Revised: Furthermore, interestingly even skip connections prove to be of no use for S row and it remains a constant kernel in this case as well.

Revised: We validate it experimentally and discuss the results in Appendix B.3 (Figure 10).

Revised: While S row results in a constant kernel for coreperiphery without community structure, it is important to note that when there exists a community structure and each community has core-periphery nodes, then S row is still preferable over S sym as it is simply a special case of homophilic networks.

Revised: This is demonstrated in Appendix B.3 (Figure 11).

Revised:  E MPIRICAL A NALYSIS ON R EAL D ATA

Original: In this section, we explore how well the theoretical results translate to real dataset Cora without
Revised: In this section, we explore how well the theoretical results translate to real dataset Cora with features, that is, XX T ̸

Original: Assumption 1 and 2.
Revised: = I n and A ̸ = M .

Original: We also provide  additional experiments on Citeseer in Appendix B.4.
Revised: We also provide experimental details and additional experiments on Citeseer in Appendix B.5.

Original: We consider multi-class node classification  ( K = 7 for Cora) usingGCN with ReLU activations and relax the orthonormal feature condition, so XX T ̸ = I n .
Revised: We consider multi-class node classification for Cora ( K = 7 ) using

Original:  The NTKs for vanilla GCN, GCN with Skip-PC and Skip- α( α = 0 . for depths d = { 1 , 2 , 4 , 8 , 16 } are computed and Figure 4 illustrates the results.
Revised: GCN with linear activations and provide the results for ReLU activations in Appendix B.4 The NTKs for vanilla GCN, GCN with Skip-PC and Skip- α are illustrated in Figure 4.

Original: We make the following observations from the experiments that validate the theory even in a much relaxed setting, (i) clear block structures show up in both GCN with and without skip connections for S row , thus illustrating that the class information is well retained by S row than S sym ; (ii)in the case of vanilla GCN, we observe from the line plot in Figure 4 that the average in-class and out-of-class block difference degrades with depth for each class in Cora, showing the negative impact of depth which aligns well with the theoretical result; (iii) while we cannot compare the skip connections, it is still evident that S row is better than S sym for both Skip-PC and
Revised: We make the following observations from the experiments that validate the theory even in a much relaxed setting, (i) clear block structures show up in both GCN with and without skip connections for S row , thus illustrating that the class information is well retained by S row than S sym ; (ii) while we cannot compare the skip connections, it is still evident that S row is better than S sym for both Skip-PC and

Original: Skip- α as block structures emerge even in the case of large depth.
Revised: Skip- α as block structures emerge even in the case of large depth.

Original: Thus, although the theoretical result is based on
Revised: Thus, although the theoretical result is based on

Original: DC-SBM with mild assumptions, the conclusions hold well in real settings on real datasets as well.
Revised: DC-SBM with mild assumptions, the conclusions hold well in real settings on real datasets as well.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: Graph convolution operators significantly influence the performance of GCNs, but existing learning theoretic bounds for GCNs do not provide insight into the representation power of the operators.
Revised: Graph convolution operators significantly influence the performance of GCNs, but existing learning theoretic bounds for GCNs do not provide insight into the representation power of the operators.

Original: We present a NTK based analysis that characterizes different convolutions, thereby proving the strong representation power of S row  and explaining why S row , and to some extent S sym , are preferred in practice (Theorem 2).
Revised: We present a NTK based analysis that characterizes different convolutions, thereby proving the strong representation power of S row in community detection and explaining why S row , and to some extent S sym , are preferred in practice (Theorem 2).

Original: In contrast to applying spectral analysis of the convolutions to explain oversmoothing, our explicit characterization of the network provides more exact quantification of the impact of over-smoothing in deep GCNs (Corollary 1, see Figure 2).
Revised: In contrast to applying spectral analysis of the convolutions to explain over-smoothing, our explicit characterization of the network provides more exact quantification of the impact of over-smoothing in deep GCNs (Corollary 1, see Figure 2).

Original: In addition, the NTKs for GCNs with skip connections enable precise understanding of the role of skip connections in countering the over-smoothing effect (Theorems 3–4).
Revised: In addition, the NTKs for GCNs with skip connections enable precise understanding of the role of skip connections in countering the over-smoothing effect (Theorems 3–4).

Original: While theassumption of DC-SBM  may seem restrictive, experiments on Cora and Citeseer demonstrate that our theoretical findings hold beyond DC-SBM, although formally characterizing such behavior could be difficult without model assumptions.
Revised: While the DC-SBM assumption may seem restrictive, experiments on Cora and Citeseer show that our theoretical results hold beyond DC-SBM, although formally characterizing such behavior could be difficult without model assumptions.

Original: We note that our analysis could be extended by considering feature information ( XX T ̸
Revised: We note that our analysis could be extended by considering feature information ( XX T ̸

Original: = I n ) or random samples from DC-SBM, which would require more involved analysis but could provide further insights into GCNs, such as interplay between graph and feature information.
Revised: = I n ) or random samples from DC-SBM, which would require more involved analysis but could provide further insights into GCNs, such as interplay between graph and feature information.

Revised: The present NTK based setup allows for the analysis of different graphs having homophilic, heterophilic and core-periphery structures, and can be extended to other graph generating processes.

Original: The present NTK based analysis lays thefoundation for further rigorous understanding of GCNs, for instance, deriving the statistical consistency or information theoretic limits of GCNs. The general formulation of NTK for vanilla GCNs (Theorem 1) and with skip connections (Corollaries 2–3) can befurther used for  theoretical analysis of other graph learning problems, such as link prediction.
Revised: Furthermore, the general formulation of NTK for vanilla GCNs (Theorem 1) and with skip connections (Corollaries 2–3) can be used for analyzing any new convolutions like topological structure preserving convolutions, for obtaining a rigorous understanding of GCNs by deriving statistical consistency results or information theoretic limits, as well as for theoretical analysis of other graph learning problems, such as link prediction.

Original:  E THICS S TATEMENT
Revised:  E THICS S TATEMENT

Original: Our work focuses on understanding some characteristics of the graph neural network theoretically and hence it doesn’t have direct implications on the ethical and fairness aspects.
Revised: Our work focuses on understanding some characteristics of the graph neural network theoretically and hence it doesn’t have direct implications on the ethical and fairness aspects.

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: The assumptions for the theory are stated clearly in Assumptions 1–2, and all the theoretical results,
Revised: The assumptions for the theory are stated clearly in Assumptions 1–2, and all the theoretical results,

Original: Theorems 1–4, Corollaries 1–3, are proved in detail in Appendix A.
Revised: Theorems 1–4, Corollaries 1–3, are proved in detail in Appendix A.

Original: The implementation of GCN and NTK for GCNs with and without skip connections are provided in ntk gcn conv.zip as a supplementary material.
Revised: The implementation of GCN and NTK for GCNs with and without skip connections are provided in ntk gcn conv.zip as a supplementary material.

Original: Datasets used in the experiments are publicly available and also provided in data folder available in the zip.
Revised: Datasets used in the experiments are publicly available and also provided in data folder available in the zip.

Original: The experimental results can be reproduced by following the instructions in readme.md .
Revised: The experimental results can be reproduced by following the instructions in readme.md .

Original: A M ATHEMATICAL DERIVATIONS AND PROOFS
Revised: A M ATHEMATICAL DERIVATIONS AND PROOFS

Original: We first derive the NTK (Theorem 1) for GCN defined in (2) and prove Theorem 2 by considering linear GCN and computing the population NTK ˜Θ ( d ) for different graph convolutions.
Revised: We first derive the NTK (Theorem 1) for GCN defined in (2) and prove Theorem 2 by considering linear GCN and computing the population NTK ˜Θ ( d ) for different graph convolutions.

Original: We use 1 n to denote a vector of n dimension with all 1 s and ˆ 1 n for a vector of n dimension with − 1 as first nentries and +1 as the remaining n 2 entries.
Revised: We use 1 n to denote a vector of n dimension with all 1 s and ˆ 1 n for a vector of n dimension with − 1 as first nentries and +1 as the remaining n 2 entries.

Original: A. T HEOREM 1: NTK FOR V ANILLA GCN
Revised: A. T HEOREM 1: NTK FOR V ANILLA GCN

Original: We rewrite the GCN F W ( X , S ) defined in (2) using the following recursive definitions:
Revised: We rewrite the GCN F W ( X , S ) defined in (2) using the following recursive definitions:

Original: [Equation]
Revised: [Equation]

Original: Thus, F W ( X , S ) =
Revised: Thus, F W ( X , S ) =

Original: F d +1 and using the definitions in (8), the gradient with respect to W i is 
Revised: F d +1 and using the definitions in (7), the gradient with respect to W i is ∂F

Original: [Equation]
Revised: [Equation]

Original: with B d +1 = 1 n ,
Revised: with B d +1 = 1 n ,

Original: B i
Revised: B i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We derive the NTK, as defined in (1), using the recursive definition of F W ( X , S ) in (8) and its derivative in (9).
Revised: We derive the NTK, as defined in (1), using the recursive definition of F W ( X , S ) in (7) and its derivative in (8).

Original: Co-variance between Nodes.
Revised: Co-variance between Nodes.

Original: We will first derive the co-variance matrix of size n × n for each layer comprising of co-variance between any two nodes u and v .
Revised: We will first derive the co-variance matrix of size n × n for each layer comprising of co-variance between any two nodes u and v .

Original: The co-variance between u and v in F 1 and F i are derived below.
Revised: The co-variance between u and v in F 1 and F i are derived below.

Original: We denote u -th row of matrix Z as Z u. throughout our proofs.
Revised: We denote u -th row of matrix Z as Z u. throughout our proofs.

Original: [Equation]
Revised: [Equation]

Original: Evaluating (10) and (11) in terms of the graph in the following,
Revised: Evaluating (9) and (10) in terms of the graph in the following,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: ; law of large numbers
Revised: ; law of large numbers

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ( a ) : using E [( F i − 1 ) rk ( F i − 1 ) sk ]
Revised: ( a ) : using E [( F i − 1 ) rk ( F i − 1 ) sk ]

Original: = ( Σ i − 1 ) rs and the definition of E i − 1 in Theorem 1.
Revised: = ( Σ i − 1 ) rs and the definition of E i − 1 in Theorem 1.

Original: NTK for Vanilla GCN.
Revised: NTK for Vanilla GCN.

Original: Let us first evaluate the tangent kernel component from W i respective to nodes u and v .
Revised: Let us first evaluate the tangent kernel component from W i respective to nodes u and v .

Original: The following two results are needed to derive it.
Revised: The following two results are needed to derive it.

Original: Result 1 (Inner Product of Matrices).
Revised: Result 1 (Inner Product of Matrices).

Original: Let a and b be vectors of size d 1 × 1 and d 2 × 1 , then
Revised: Let a and b be vectors of size d 1 × 1 and d 2 × 1 , then

Original: [Equation]
Revised: [Equation]

Original: Result 2 ⟨ ( B r ) u. , ( B r ) v. ⟩ .
Revised: Result 2 ⟨ ( B r ) u. , ( B r ) v. ⟩ .

Original: We evaluate ⟨ ( B r ) u. , ( B r ) v. ⟩ = (cid:0) B r B Tr (cid:1) uv appearing in the gradient.
Revised: We evaluate ⟨ ( B r ) u. , ( B r ) v. ⟩ = (cid:0) B r B Tr (cid:1) uv appearing in the gradient.

Original: [Equation]
Revised: [Equation]

Original: h
Revised: h

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Revised: [Table]

Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: Now
Revised: Now

Original: [Table] [Equation]
Revised: [Table] [Table]

Original: [Equation]
Revised: [Equation]

Original: Extending (16) to all n nodes which will result in n × n matrix,
Revised: Extending (15) to all n nodes which will result in n × n matrix,

Original: [Equation]
Revised: [Equation]

Original: Finally, NTK Θ is,
Revised: Finally, NTK Θ is,

Original: [Equation]
Revised: [Equation]

Original: with definition of Σ k and ˙ E k mentioned in the theorem.
Revised: with definition of Σ k and ˙ E k mentioned in the theorem.

Original: □
Revised: □

Original: A. T HEOREM 2 AND C OROLLARY 1: P OPULATION NTK ˜Θ FOR D IFFERENT S
Revised: A. T HEOREM 2 AND C OROLLARY 1: P OPULATION NTK ˜Θ FOR D IFFERENT S

Original: We consider Assumption 1, that is, linear GCN with orthonormal features and Assumption 2 without assumption on γ .
Revised: We consider Assumption 1, that is, linear GCN with orthonormal features and Assumption 2 without assumption on γ .

Original: We first prove it for K = 2 and then extend it to K classes.
Revised: We first prove it for K = 2 and then extend it to K classes.

Original: We consider that all nodes are sorted per class for ease of analysis which implies A is a n × n matrix with pπ i π j entries in [1 , n 2 ][1 , n 2 ] and [ n 2 +1 , n ][ n 2 +1 , n ] blocks and qπ i π j entries in [1 , n 2 ][ n 2 +1 , n ] and [ n 2 +1 , n ][1 , n 2 ] blocks.
Revised: We consider that all nodes are sorted per class for ease of analysis which implies A is a n × n matrix with pπ i π j entries in [1 , n 2 ][1 , n 2 ] and [ n 2 +1 , n ][ n 2 +1 , n ] blocks and qπ i π j entries in [1 , n 2 ][ n 2 +1 , n ] and [ n 2 +1 , n ][1 , n 2 ] blocks.

Original: Therefore,
Revised: Therefore,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the entries of ˆ π are − π i ∀ i ∈ [1 , n 2 ] and
Revised: where the entries of ˆ π are − π i ∀ i ∈ [1 , n 2 ] and

Original: + π i ∀ i ∈ [ n 2 + 1 , n ] .
Revised: + π i ∀ i ∈ [ n 2 + 1 , n ] .

Original: D be the degree matrix of A and D = p + q 2 diag ( π ) .
Revised: D be the degree matrix of A and D = p + q 2 diag ( π ) .

Original: A.2. S YMMETRIC D EGREE N ORMALIZED A DJACENCY S sym
Revised: A.2. S YMMETRIC D EGREE N ORMALIZED A DJACENCY S sym

Original: Now, lets compute S sym using A (19) and its degree matrix D .
Revised: Now, lets compute S sym using A (18) and its degree matrix D .

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Note that π T π
Revised: Note that π T π

Original: = ˆ
Revised: = ˆ

Original: π T ˆ π = 1 , π T
Revised: π T ˆ π = 1 , π T

Original: ˆ π = 

Revised: ˆ π =

Original: and U T U
Revised: and U T U

Original: =
Revised: =

Original: I 2 , thus (20) is the singular value decomposition of S sym .
Revised: I 2 , thus (19) is the singular value decomposition of S sym .

Original: To compute the population NTK ˜Θ ( d ) sym in (4)
Revised: To compute the population NTK ˜Θ ( d ) sym in (4)

Original: [Equation]
Revised: [Equation]

Original: we need S ksym S kTsym .
Revised: we need S ksym S kTsym .

Original: Using (20),
Revised: Using (19),

Original: [Equation]
Revised: [Equation]

Original: Consequently
Revised: Consequently

Original: [Equation]
Revised: [Equation]

Original: population NTK ˜Θ ( d ) sym for nodes i and j using (21) is as follows,
Revised: population NTK ˜Θ ( d ) sym for nodes i and j using (20) is as follows,

Original: [Equation]
Revised: [Equation]

Original: Since we consider (cid:80) i ∈C k
Revised: Since we consider (cid:80) i ∈C k

Original: π
Revised: π

Original: i = 1 /K
Revised: i = 1 /K

Original: [Equation]
Revised: [Equation]

Original: the maximum of √ π i π j < 1 / 4 for K = 2 .
Revised: the maximum of √ π i π j < 1 / 4 for K = 2 .

Original: This implies √ π i π j (cid:0) 1 + r 2 (cid:1) < 1 .
Revised: This implies √ π i π j (cid:0) 1 + r 2 (cid:1) < 1 .

Original: Therefore, NTK at d → ∞ is
Revised: Therefore, NTK at d → ∞ is

Original: [Equation]
Revised: [Equation]

Original: Equations (22) and (23) prove the population NTK ˜Θ ( d ) sym and ˜Θ ( ∞ ) sym in Theorem 2 and Corollary 1, respectively.
Revised: Equations (21) and (22) prove the population NTK ˜Θ ( d ) sym and ˜Θ ( ∞ ) sym in Theorem 2 and Corollary 1, respectively.

Original: □
Revised: □

Original: [Equation]
Revised: [Equation]

Original: OW D EGREE N ORMALIZED A DJACENCY Srow
Revised: OW D EGREE N ORMALIZED A DJACENCY S

Original:  The assumption on γ in Assumption 2 is only to simplify the expression of population NTK for
Revised: [Equation] The assumption on γ in Assumption 2 is only to simplify the expression of population NTK for

Original: S row .
Revised: S row .

Original: We derive it without this assumption in the following.
Revised: We derive it without this assumption in the following.

Original: We first derive S krow S kTrow .
Revised: We first derive S krow S kTrow .

Original: [Equation]
Revised: [Equation]

Original: − 12 D += D − 12 UΛU T D +
Revised: − 12 D += D − 12 UΛU T D +

Original: S krow =
Revised: S krow =

Original: D − 12 UΛ k UT D +
Revised: D − 12 UΛ [Equation] U

Revised: [Equation]

Revised: D +

Original: S k row S kT row =
Revised: S k row S kT row =

Original: D − 12 UΛ k U T D + 12 D + 12 UΛ k U T D −= D − 12 UΛ k U T DUΛ k U T D −= (cid:16) D − 12 UΛ k U T D − 12 (cid:17) D + 12 DD + 12 (cid:16) D − 12 UΛ k U T D − 12 (cid:17) = (cid:16) (cid:98) UΛ k (cid:98) U T (cid:17) D 2 (cid:16) (cid:98) UΛ k (cid:98) U T (cid:17) ; (cid:98) U
Revised: D − 12 UΛ k U T D + 12 D + 12 UΛ k U T D −= D [Equation] 12 UΛ k U T DUΛ k U T D −= (cid:16) D − 12 UΛ k U T D − 12 (cid:17) D + 12 DD + 12 (cid:16) D − 12 UΛ k U T D − 12 (cid:17) = (cid:16) (cid:98) UΛ k (cid:98) U T (cid:17) D 2 (cid:16) (cid:98) UΛ k (cid:98) U T (cid:17) ; (cid:98) U

Original: = D − 12 U =
Revised: = D − 12 U =

Original: (cid:114)p + q (cid:20) 

Revised: (cid:114)p + q (cid:20)

Original: Tn ˆ 1 T n (cid:21) n ×
Revised: Tn ˆ 1 T n (cid:21) n ×

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: Note that each block is a constant and independent of individual π i .
Revised: Note that each block is a constant and independent of individual π i .

Original: Using (24), NTK in (4) for i and j belonging to class 1 is,
Revised: Using (23), NTK in (4) for i and j belonging to class 1 is,

Original: [Equation]
Revised: [Equation]

Original: Similarly for i and j in class 2 ,
Revised: Similarly for i and j in class 2 ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: When i and j are in different classes,
Revised: When i and j are in different classes,

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: As λ and µ < 14 , (1 + r ) 2 λ + (
Revised: As λ and µ < 14 , (1 + r ) 2 λ + (

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: µ < 2 (cid:0) 1 + r 2 (cid:1) 14 < 1 , population NTK ˜Θ row at d → ∞ is
Revised: µ < 2 (cid:0) 1 + r 2 (cid:1) 14 < 1 , population NTK ˜Θ row at d → ∞ is

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]

Original: When the assumption on γ is introduced, ∃ γs.t.
Revised: When the assumption on γ is introduced, ∃ γs.t.

Original: (cid:80) ni
Revised: (cid:80) ni

Original: =1 π 2 i 1 [ C i = k ] = γ ∀ k , λ + µ = 2 γ and λ − µ = 0 .
Revised: =1 π 2 i 1 [ C i = k ] = γ ∀ k , λ + µ = 2 γ and λ − µ = 0 .

Original: Hence, equations (25), (26) and (27) of the population NTK ˜Θ ( d ) row and (28) of ˜Θ ( ∞ ) row reduce to the expressions in Theorem 2 and Corollary 1, respectively.
Revised: Hence, equations (24), (25) and (26) of the population NTK ˜Θ ( d ) row and (27) of ˜Θ ( ∞ ) row reduce to the expressions in Theorem 2 and Corollary 1, respectively.

Original: □
Revised: □

Original: A.2. C OLUMN N ORMALIZED A DJACENCY S col
Revised: A.2. C OLUMN N ORMALIZED A DJACENCY S col

Original: In this section we derive the population NTK ˜Θ ( d ) col .
Revised: In this section we derive the population NTK ˜Θ ( d ) col .

Original: [Equation]
Revised: [Equation]

Original: Therefore, ˜Θ ( d ) col is
Revised: Therefore, ˜Θ ( d ) col is

Original: [Equation]
Revised: [Equation]

Original: k =
Revised: k =

Original: [Equation]
Revised: [Equation]

Original: Since (cid:80) ni π i = 1 , π i = O ( 1 n ) .
Revised: Since (cid:80) ni π i = 1 , π i = O ( 1 n ) .

Original: So, nπ i π j (cid:0) 1 + r 2 (cid:1) < 1 .
Revised: So, nπ i π j (cid:0) 1 + r 2 (cid:1) < 1 .

Original: Therefore, using (30),
Revised: Therefore, using (29),

Original: [Equation]
Revised: [Equation]

Original: Hence, equations (30) and (31) prove the population NTK ˜Θ ( d ) col and ˜Θ ( ∞ ) col in Theorem 2 and Corollary 1, respectively.
Revised: Hence, equations (29) and (30) prove the population NTK ˜Θ ( d ) col and ˜Θ ( ∞ ) col in Theorem 2 and Corollary 1, respectively.

Original: □
Revised: □

Original: A.2. U NNORMALIZED A DJACENCY S adj
Revised: A.2. U NNORMALIZED A DJACENCY S adj

Original: We can rewrite A as follows,
Revised: We can rewrite A as follows,

Original: [Equation]
Revised: [Equation]

Original: We consider γ assumption for the analysis of unnormalised adjacency to simplify the computation.
Revised: We consider γ assumption for the analysis of unnormalised adjacency to simplify the computation.

Original: But the result holds without this assumption.
Revised: But the result holds without this assumption.

Original: [Equation]
Revised: [Equation]

Original: γ(cid:0) 4 p 3 q + 4 pq 3 (cid:1) γ(cid:0) 4 p 3 q + 4 pq 3 (cid:1) γ(cid:0) p 4 + q 4 + 6 p 2 q 2 (cid:1) γ    π. . .
Revised: γ(cid:0) 4 p 3 q + 4 pq 3 (cid:1) γ(cid:0) 4 p 3 q + 4 pq 3 (cid:1) γ(cid:0) p 4 + q 4 + 6 p 2 q 2 (cid:1) γ    π. . .

Original: π n 
Revised: π n 

Original: 
Revised: 

Original: Note that in the above shown A 2 k it is the even powers of binomial expansion of ( p + q ) 2 k for i, j in same class whereas it is the odd powers for i, j not in the same class.
Revised: Note that in the above shown A 2 k it is the even powers of binomial expansion of ( p + q ) 2 k for i, j in same class whereas it is the odd powers for i, j not in the same class.

Original: We compute the filter S adj using this fact.
Revised: We compute the filter S adj using this fact.

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: if i and j ∈ same class
Revised: if i and j ∈ same class

Original: [Equation]
Revised: [Equation]

Original: if i and j ∈ different class
Revised: if i and j ∈ different class

Original: [Equation]
Revised: [Equation]

Original: 
Revised: 

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: if i and j ∈ same class
Revised: if i and j ∈ same class

Original: [Equation]
Revised: [Equation]

Original: 1 q 2 l +  if i and j ∈ different class
Revised: [Table] l + [Table] if i and j ∈ different class

Original: The above form is not simplified as it is not an interesting case where the gap between the two blocks disappears rapidly and (cid:16) ˜Θ ( ∞ ) adj (cid:17) ij = 0 .
Revised: The above form is not simplified as it is not an interesting case where the gap between the two blocks disappears rapidly and (cid:16) ˜Θ ( ∞ ) adj (cid:17) ij = 0 .

Original: There is no information in the kernel proving both
Revised: There is no information in the kernel proving both

Original: Theorem 2 and Corollary 1.
Revised: Theorem 2 and Corollary 1.

Original: □
Revised: □

Original: A.2. N UMBER OF C LASSES K >
Revised: A.2. N UMBER OF C LASSES K >

Original: From the above derivation for K = 2 , it can be seen that once S ksym S kTsym is computed, the population NTK for all the graph convolutions can be derived using it.
Revised: From the above derivation for K = 2 , it can be seen that once S ksym S kTsym is computed, the population NTK for all the graph convolutions can be derived using it.

Original: Therefore, we derive it for K >
Revised: Therefore, we derive it for K >

Original: and it suffices to show the conclusions of Theorem 2 and Corollary 1.
Revised: and it suffices to show the conclusions of Theorem 2 and Corollary 1.

Original: We denote the vector ˆ π 1 k with − π i ∀ i ∈ (cid:2)
Revised: We denote the vector ˆ π 1 k with − π i ∀ i ∈ (cid:2)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: , +
Revised: , +

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: ) K , nkK (cid:105) and 0 for the rest.
Revised: ) K , nkK (cid:105) and 0 for the rest.

Original: With this definition, A is
Revised: With this definition, A is

Original: [Equation]
Revised: [Equation]

Original: D for K classes is p +( K − 1) K diag ( π ) from (33).
Revised: D for K classes is p +( K − 1) K diag ( π ) from (32).

Original: We can compute S sym using A and D as follows,
Revised: We can compute S sym using A and D as follows,

Original: [Equation]
Revised: [Equation]

Original: sym
Revised: sym

Original: [Equation]
Revised: [Equation]

Original: It is noted that the equation (34) is very much similar to (21) for K = 2 .
Revised: It is noted that the equation (33) is very much similar to (20) for K = 2 .

Original: The further derivations of the population NTKs ˜Θ for all the convolutions are similar and the theoretical results extend without any issues.
Revised: The further derivations of the population NTKs ˜Θ for all the convolutions are similar and the theoretical results extend without any issues.

Original: A. NTK FOR GCN WITH S KIP C ONNECTIONS (C OROLLARY 2 AND 3)
Revised: A. NTK FOR GCN WITH S KIP C ONNECTIONS (C OROLLARY 2 AND 3)

Original: We observe that the definitions of G i ∀ i ∈ [1 , d +1] are different for GCN with skip connections from the vanilla GCN.
Revised: We observe that the definitions of G i ∀ i ∈ [1 , d +1] are different for GCN with skip connections from the vanilla GCN.

Original: Despite the difference, the definition of gradient with respect to W i in (9) does not change as G i in the gradient accounts for the change and moreover, there is no new learnable parameter since the input transformation H 0 = XW 0 where ( W 0 ) ij is sampled from N (0 , 1) is not learnable in our setting.
Revised: Despite the difference, the definition of gradient with respect to W i in (8) does not change as G i in the gradient accounts for the change and moreover, there is no new learnable parameter since the input transformation H 0 = XW 0 where ( W 0 ) ij is sampled from N (0 , 1) is not learnable in our setting.

Original: Given the fact that the gradient definition holds for GCN with skip connection, the NTK will retain the form from NTK for vanilla GCN as evident from the derivation of NTK for vanilla GCN in Section A.1.
Revised: Given the fact that the gradient definition holds for GCN with skip connection, the NTK will retain the form from NTK for vanilla GCN as evident from the derivation of NTK for vanilla GCN in Section A.1.

Original: The change in G i will only affect the co-variance between nodes.
Revised: The change in G i will only affect the co-variance between nodes.

Original: Hence, we will derive the co-variance matrix for Skip-PC and Skip- α in the following.
Revised: Hence, we will derive the co-variance matrix for Skip-PC and Skip- α in the following.

Original: Skip-PC: Co-variance between nodes.
Revised: Skip-PC: Co-variance between nodes.

Original: The co-variance between nodes u and v in F
Revised: The co-variance between nodes u and v in F

Original: and F i are derived below.
Revised: and F i are derived below.

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original:  law of large numbers
Revised: ; law of large numbers

Original: [Equation]
Revised: [Equation]

Original:  law of large numbers
Revised: ; law of large numbers

Original: [Equation]
Revised: [Equation]

Original: E [( S σ ( F i − 1 )) uk ( S σ s ( XW 0 ))
Revised: E [( S σ ( F i − 1 )) uk ( S σ s ( XW 0 ))

Original: vk ] and E [( S σ s ( XW 0 ))
Revised: vk ] and E [( S σ s ( XW 0 ))

Original: uk ( S σ ( F i − 1 )) vk ] evaluate toby conditioning on W 0 first and rewriting the expectation based on this conditioning.
Revised: uk ( S σ ( F i − 1 )) vk ] evaluate toby conditioning on W 0 first and rewriting the expectation based on this conditioning.

Original: The terms within expectation are independent when conditioned on W 0 , and hence it is
Revised: The terms within expectation are independent when conditioned on W 0 , and hence it is

Original: E W
Revised: E W

Original: [Equation]
Revised: [Equation]

Original: by taking h in W 0 going to infinity first.
Revised: by taking h in W 0 going to infinity first.

Original: Here,
Revised: Here,

Original: [Equation]
Revised: [Equation]

Original: We get the co-variance matrix for all pairs of nodes Σ 1 = S ˜ E 0 S T and Σ i = SE i − 1 S T + Σ 1 from (35) and (36).
Revised: We get the co-variance matrix for all pairs of nodes Σ 1 = S ˜ E 0 S T and Σ i = SE i − 1 S T + Σ 1 from (34) and (35).

Original: Skip- α : Co-variance between nodes.
Revised: Skip- α : Co-variance between nodes.

Original: Let u and v be two nodes and the co-variance between u and v in F 1 and F i are derived below.
Revised: Let u and v be two nodes and the co-variance between u and v in F 1 and F i are derived below.

Original: [Equation]
Revised: [Equation]

Original: [Equation] [Equation] 
Revised: [Equation] [Equation] [Equation]

Original: argument as ( f
Revised: argument as ( f

Original: [Equation]
Revised: [Equation]

Original: in derivation of Σ i in Skip-PC
Revised: in derivation of Σ i in Skip-PC.

Original: [Equation] We get the co-variance matrix for all pairs of nodes Σ 1 = (1 − α ) 2 S ˜ E 0 S T + α (1 − α ) (cid:16) S ˜ E 

Revised: We get the co-variance matrix for all pairs of nodes Σ 1 = (1 − α ) 2 S ˜ E 0 S T + α (1 − α ) (cid:16) S ˜ E

Original: + ˜ E 0 S T (cid:17)
Revised: + ˜ E 0 S T (cid:17)

Original: + α 2 ˜ E 0 and Σ i
Revised: + α 2 ˜ E 0 and Σ i

Original: = (1 − α ) 2 SE i − 1 S T + α 2 ˜ E 0 from (37) and (38).
Revised: = (1 − α ) 2 SE i − 1 S T + α 2 ˜ E 0 from (36) and (37).

Original: A. T HEOREM 3: P OPULATION NTK ˜Θ FOR S KIP -PC
Revised: A. T HEOREM 3: P OPULATION NTK ˜Θ FOR S KIP -PC

Original: NTK at depth d , Θ ( d ) PC for Skip-PC with linear activations is
Revised: NTK at depth d , Θ ( d ) PC for Skip-PC with linear activations is

Original: [Equation]
Revised: [Equation]

Original: In (39)
Revised: In (38)

Original: [Equation]
Revised: [Equation]

Original: I is NTK without skip connection and II is computed for S row and S sym as follows.
Revised: I is NTK without skip connection and II is computed for S row and S sym as follows.

Original: Computing II for population NTK ˜Θ ( d ) for S sym : for nodes i and j ,
Revised: Computing II for population NTK ˜Θ ( d ) for S sym : for nodes i and j ,

Original: [Equation]
Revised: [Equation]

Original:  It converges to (40) as d → ∞ since √ π i π j (cid:0) 1 + δ ij r 2 (cid:1) < 1 according to our setup.
Revised: [Equation] It converges to (39) as d → ∞ since √ π i π j (cid:0) 1 + δ ij r 2 (cid:1) < 1 according to our setup.

Original: Therefore, using (40) and (23) we get the population NTK ˜Θ ( ∞ ) PC,sym for Skip-PC at d → ∞ ,
Revised: Therefore, using (39) and (22) we get the population NTK ˜Θ ( ∞ ) PC,sym for Skip-PC at d → ∞ ,

Original: [Equation]
Revised: [Equation]

Original: hence deriving Theorem 3.
Revised: hence deriving Theorem 3.

Original: □
Revised: □

Original: Similarly, computing II for S row without assumption on γ , i and j in class 1 , d +1 (cid:88)
Revised: Similarly, computing II for S row without assumption on γ , i and j in class 1 , d +1 (cid:88)

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: For i and j in class 2 ,
Revised: For i and j in class 2 ,

Original: [Equation]
Revised: [Equation]

Original: For i and j in different class,
Revised: For i and j in different class,

Original: [Equation]
Revised: [Equation]

Original: Therefore, the population NTK ˜Θ ( ∞ ) α,row with γ assumption is obtained by substituting λ + µ = 2 γ and λ − µ = 0 in (41), (42) and (43) .
Revised: Therefore, the population NTK ˜Θ ( ∞ ) α,row with γ assumption is obtained by substituting λ + µ = 2 γ and λ − µ = 0 in (40), (41) and (42) .

Original: ˜Θ ( ∞ ) α,row
Revised: ˜Θ ( ∞ ) α,row

Original: [Equation]
Revised: [Equation]

Original: hence deriving Theorem□ .
Revised: hence deriving Theorem□ .

Original: A. 
Revised: A.

Original: T HEOREM
Revised: T HEOREM

Original: : P OPULATION NTK ˜Θ FOR S KIP - 
Revised: : P OPULATION NTK ˜Θ FOR S KIP - α

Original: α

Original: We expand Σ 1 and Σ k of Skip- α first to derive the population NTK.
Revised: We expand Σ 1 and Σ k of Skip- α first to derive the population NTK.

Original: [Equation]
Revised: [Equation]

Original: Exact NTK of depth d for Skip- α is expanded using the above as follows.
Revised: Exact NTK of depth d for Skip- α is expanded using the above as follows.

Original: [Equation]
Revised: [Equation]

Original: We compute I , II and III of (45) for  using S  focusing on 
Revised: We compute I , II and III of (44) for population NTK using S sym focusing on d

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: [Table]

Original: [Equation]
Revised: [Equation]

Original: Therefore the population NTK as d → ∞ is obtained
Revised: Therefore the population NTK as d → ∞ is obtained

Original: [Equation]
Revised: [Equation]

Original: proving Theorem 4.
Revised: proving Theorem 4.

Original: □
Revised: □

Original: We now compute I , II and III for population NTK ˜Θ ( ∞ )
Revised: We now compute I , II and III for population NTK ˜Θ ( ∞ )

Original: α using S row .
Revised: α using S row .

Original: For nodes i and j in class 1 :
Revised: For nodes i and j in class 1 :

Original: [Equation]
Revised: [Equation]

Original: Similarly for nodes i and j in class 2 and different classes
Revised: Similarly for nodes i and j in class 2 and different classes

Original: I ij = 0 as d → ∞ .
Revised: I ij = 0 as d → ∞ .

Original: Likewise, II ij =as d → ∞ for any i and j .
Revised: Likewise, II ij =as d → ∞ for any i and j .

Original: This is similar to S sym .
Revised: This is similar to S sym .

Original: For i and j in class 1 ,
Revised: For i and j in class 1 ,

Original: [Equation]
Revised: [Equation]

Original: Trow (cid:1) ⊙ d
Revised: Trow (cid:1) ⊙ d

Original: +1 − k
Revised: +1 − k

Original: III ij = αd +1 (cid:88) k
Revised: III ij = αd +1 (cid:88) k

Original: = (cid:34) k − 1 (cid:88) l
Revised: = (cid:34) k − 1 (cid:88) l

Original: = (1 − α ) 2 l (cid:0) ( λ + µ ) (cid:0) 1 + r 2 l (cid:1) + ( λ − µ ) 2 r l (cid:1)(cid:35) (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d +1 − k
Revised: = (1 − α ) 2 l (cid:0) ( λ + µ ) (cid:0) 1 + r 2 l (cid:1) + ( λ − µ ) 2 r l (cid:1)(cid:35) (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d +1 − k

Original: = αd +1 (cid:88) k
Revised: = αd +1 (cid:88) k

Original: = (cid:34) ( λ + µ ) (cid:32) 1 − (1 − α ) 2 k 1 − (1 − α ) 2 + 1 − (1 − α ) 

Revised: = (cid:34) ( λ + µ ) (cid:32) 1 − (1 − α ) 2 k 1 − (1 − α ) 2 + 1 − (1 − α )

Original: k r 2 k 1 − (1 − α ) 2 r(cid:33) + ( λ − µ ) 1 − (1 − α ) 

Revised: k r 2 k 1 − (1 − α ) 2 r(cid:33) + ( λ − µ ) 1 − (1 − α )

Original: k r k 1 − (1 − α ) 2 r (cid:35) × (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d +1 − k = α(cid:32) λ + µ 1 −
Revised: k r k 1 − (1 − α ) 2 r (cid:35) × (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d +1 − k = α(cid:32) λ + µ 1 −

Original: (1 − α ) 2 + λ + µ 1 − (1 − α )
Revised: (1 − α ) 2 + λ + µ 1 − (1 − α )

Original: 2 r 2 + λ − µ 1 − (1 − α ) 2 r (cid:33) d +1 (cid:88) k
Revised: 2 r 2 + λ − µ 1 − (1 − α ) 2 r (cid:33) d +1 (cid:88) k

Original: = (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d
Revised: = (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d

Original: +1 − k − α(cid:32) λ + µ 1 − (1 − α )(cid:33)
Revised: +1 − k − α(cid:32) λ + µ 1 − (1 − α )(cid:33)

Original: d +1 (cid:88) k
Revised: d +1 (cid:88) k

Original: = (1 − α ) 2 k (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d
Revised: = (1 − α ) 2 k (cid:16) (1 + r ) 2 λ + (1 − r ) 2 µ (cid:17) d

Original: +1 − k − α(cid:32) λ + µ 1 − (1 − α ) 2 r(cid:33)
Revised: +1 − k − α(cid:32) λ + µ 1 − (1 − α ) 2 r(cid:33)

Original: d +1 (cid:88) k
Revised: d +1 (cid:88) k

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: Similarly for i and j in class 2 ,
Revised: Similarly for i and j in class 2 ,

Original: [Equation]
Revised: [Equation]

Original: For i and j in different classes,
Revised: For i and j in different classes,

Original: [Equation]
Revised: [Equation]

Original: Thus, applying γ assumption to (50), (51) and (52) the population NTK ˜Θ ( ∞ ) α,row as d → ∞ is, 
Revised: Thus, applying γ assumption to (49), (50) and (51) the population NTK ˜Θ ( ∞ ) α,row as d → ∞ is, [Equation] 1 − [Equation]

Revised: [Equation]

Revised: ij

Original: [Equation]
Revised: [Equation]

Revised: 2 ) (cid:32)

Revised: [Equation]

Original: 1 − 2 γ (1 + δ ij r 2 ) (cid:32) 1 − (1 − α ) 2 + δ ij 1 − (1 − α ) 2 r(cid:33) , hence proving Theorem 4.
Revised: [Equation] ) 2 + δ ij 1 − ( [Equation] ) 2 r(cid:33) , hence proving Theorem 4.

Original: □
Revised: □

Original: B E MPIRICAL A NALYSIS B. E XPERIMENTAL D ETAILS OF F IGURE
Revised: B E MPIRICAL A NALYSIS B. E XPERIMENTAL D ETAILS OF F IGURE

Original: We use the code for GCN without skip connections from github1(Kipf & Welling, 2017) and skip connection from github2(Chen et al., 2020).
Revised: We use the code for GCN without skip connections from github1(Kipf & Welling, 2017) and skip connection from github2(Chen et al., 2020).

Original: The following hyperparameters are used for GCN without skip connections: learning rate is 0 .
Revised: The following hyperparameters are used for GCN without skip connections: learning rate is 0 .

Original: 01 , weight decay is 5 e − 4 , hidden layer width is 64 and epochs is 500 , 1500 , 2000 for depths 2 , 4 , 8 respectively.
Revised: 01 , weight decay is 5 e − 4 , hidden layer width is 64 and epochs is 500 , 1500 , 2000 for depths 2 , 4 , 8 respectively.

Original: For the skip connections, we used GCNII model, same parameters as vanilla GCN with α = 0 .
Revised: For the skip connections, we used GCNII model, same parameters as vanilla GCN with α = 0 .

Original: The performance is averaged over 5 runs.
Revised: The performance is averaged over 5 runs.

Revised: B. C OMPARISON OF GCN AND NTK

Revised: Although it is theoretically clear that the infinite width assumption should not affect the observations made on performance of GCN with S sym and S row in Figure 1, we illustrate the same using graph NTK.

Revised: Figure 5 shows that the observation is seen in graph NTK as well, thus supporting our theoretical argument.

Original: B. N UMERICAL V ALIDATION FOR DC-SBM FOR V ANILLA GCN AND S KIP - α
Revised: B. N UMERICAL V ALIDATION FOR DC-SBM FOR V ANILLA GCN AND S KIP - α

Revised: Experimental Details .

Original: We extend the experiments on numerical validation for random graphsusing vanilla GCN described in Section 3.2 to column normalized adjacency S col and unnormalized adjacency S adj here.
Revised: For the experiments, we fix the size of the sampled graphs to n = 1000 , p = 0 . 8 and q = 0 . 1 for homophily DC-SBM, p = 0 . 1 and q = 0 .

Revised: 8 for heterophily DC-SBM and p = q = 1 for core-periphery DC-SBM.

Revised: π is sampled uniformly [0 , 1] for homophily and heterophily, and π i ∼ Unif (0 . 5 , 1) ∀ i ∈ core and π i ∼ Unif (0 , 0 . 5) ∀ i ∈ periphery for coreperiphery DC-SBM.

Revised: Illustration of impact of depth in Vanilla GCN using Homophily DC-SBM .

Revised: We show the impact of depth in Vanilla GCN using homophily DC-SBM in Figure 6.

Revised: The DC-SBM is shown in the first column and columns 2 and 3 show the exact NTK for depth= 1 and 8 for symmetric and row normalization, respectively.

Revised: The plots clearly illustrate the complete loss of class information in symmetric normalization with depth (column 2).

Revised: While the prevalence of block difference has decresed in row normalization over depth (column 3), the block/community structure is still retained.

Revised: Thus showing the strong representation power of S row .

Revised: Illustration of S col and S adj in Vanilla GCN using Homophily DC-SBM .

Revised: We extend the experiments on numerical validation for random graphs using vanilla GCN described in Section 3.2 to column normalized adjacency S col and unnormalized adjacency S adj here.

Original: We use the same setup described in Section 3.2 and Figure 5 illustrates the results.
Revised: We use the same setup described in Section 3.2 and Figure 7 illustrates the results.

Original: We observe that even for depth 1 both the convolutions are influenced by the degree correction and there is no class information in the kernels for higher depth.
Revised: We observe that even for depth 1 both the convolutions are influenced by the degree correction and there is no class information in the kernels for higher depth.

Original: Thus, this validates the theoretical result in Theorem 2.
Revised: Thus, this validates the theoretical result in Theorem 2.

Original: In the case of  Skip- α , we present the complementary result to Section 4.3 here.
Revised: Illustration of impact of depth in Skip-PC and Skip- α using Homophily DC-SBM . We present a complementary result to Section 4.3 here.

Original: We use the same setting as described in Section 4.3 and use α = 0 . 1 to obtain the result illustrated in Figure 6.
Revised: We use the same setting as described in Section 4.3 and plot the exact NTKs of depths 1 and 8 for symmetric and row normalization.

Original: Similar conclusions are derived from the experiment. We observe that the gap between in-class and out-of-class blocks decreases for both S row and S sym with depth, but the class information is still retained for larger depth and the gap doesn’t vanish.
Revised: Figure 8 shows the results for Skip-PC and we observe that the gap between in-class and out-of-class blocks decreases for both S row and S sym with depth, but the class information is still retained for larger depth and the gap doesn’t vanish.

Original: Between S row and S sym , the heatmaps show that S row retains the block structure better than S sym and is devoid of the influence of the degree corrections. 
Revised: Between S row and S sym , the heatmaps show that S row retains the block structure better than S sym and is devoid of the influence of the degree corrections. In the case of Skip- α ,we use α = 0 . 1 to obtain the result illustrated in Figure 9. Similar conclusions are derived from the experiment.

Original: Although we consider XX T
Revised: Although we consider XX T

Original: = I n for Skip- α which fundamentally relies on the feature information to interpolate, the results are still meaningful and demonstrate the theoretical findings.
Revised: = I n for Skip- α which fundamentally relies on the feature information to interpolate, the results are still meaningful and demonstrate the theoretical findings.

Revised: Numerical Validation of Core-Periphery DC-SBM .

Revised: In this section, we validate the two scenarios discussed in Section 5 - core-periphery without community structure and core-periphery with community structure.

Revised: For the firsr case, we consider core-periphery DC-SBM with n/ 4 nodes as core and the rest as periphery as shown in the first heatmap of Figure 10.

Revised: We plot the exact NTKs of depth 2 for symmetric and row normalization using Vanilla GCN as shown in the second and third heatmaps of Figure 10.

Revised: This clearly demonstrates the theoretical result presented in Corollary 4 where the symmetric normalization exhibits the graph structure and the row normalization is a constant kernel.

Original:  B. E XPERIMENTS ON R EAL D ATASET : C ORA
Revised: In the second setting, we consider two communities of equal size n/ 2 with core-periphery in each, and the link probabilities between cores of the communities is higher than core-periphery or periphery-periphery of the two communities as shown in the first heatmap of Figure 11. The exact NTKs of symmetric and row normalization are illustrated in the second and third heatmaps of Figure 11 where we see that row normalization retains the community structure again. B. E XPERIMENTS ON R EAL D ATASET : C ORA

Revised: Orthonormal Feature XX T = I n Assumption .

Original: In this section, we present additional experiments on Cora.
Revised: In this section, we present additional experiments on Cora.

Original: Since our theory assumed orthonormal features XX T = I n , we validate it experimentally in similar setup described in Section 5.
Revised: Since our theory assumed orthonormal features XX T = I n , we validate it experimentally in similar setup described in Section 6.

Original: Figureshows the result for S sym and S row for depth 1 and 8 .
Revised: Figure 12 shows the result for S sym and S row for depthand 8 .

Original: The conclusions derived from real setting hold here as well and shows S row preserves the class information better than S sym . 
Revised: The conclusions derived from real setting hold here as well and shows S row preserves the class information better than S sym . ReLU GCN.

Revised: We present the result for ReLU GCN in this section.

Revised: Figure 13 shows the result where the conclusions derived in Section 6 holds very well.

Revised: Additionally, we plot the average in-class and out-of-class block difference in the case of vanilla GCN (line plots in first row of Figure 13), we observe that the average in-class and out-of-class block difference degrades with depth for each class in Cora, showing the negative impact of depth which aligns well with the theoretical result.

Original: [Figure]
Revised: [Figure]

Original: Another experimental study is to understand how easy it is to learn the classes that showed good in-class and out-of-class gap preservation from the experiment in Section 5.
Revised: Another experimental study is to understand how easy it is to learn the classes that showed good in-class and out-of-class gap preservation from the above experiment.

Original: The line plot in Figure 4 shows class C 2 and C 5 are well represented by both S sym and S row .
Revised: The line plot in Figure 13 shows class C 2 and C 5 are well represented by both S sym and S row .

Original: To study how well this holds in the trained GCN, we considered depth 4 vanilla GCN with ReLU activations and used the same hyperparameters mentioned in
Revised: To study how well this holds in the trained GCN, we considered depth 4 vanilla GCN with ReLU activations and used the same hyperparameters mentioned in

Original: Section B.1.
Revised: Section B.1.

Original: The results are shown in Figure 8 where we observe that C 2 and C 5 are well learnt.
Revised: The results are shown in Figure 14 where we observe that C 2 and C 5 are well learnt.

Original: On the other hand, other classes that showed small gap are also well learnt by the trained GCN.
Revised: On the other hand, other classes that showed small gap are also well learnt by the trained GCN.

Original: This needs further investigation as it has to do with the data split and some classes are more represented in the training data and some for instance C 6 is poorly represented.
Revised: This needs further investigation as it has to do with the data split and some classes are poorly represented in the training data, for instance C 6 .

Original: Thus, we leave it for further analysis.
Revised: Thus, we leave it for further analysis.

Original:  We present the result for linear GCN with the same setup as described in Section 5 to check the goodness of our theory.
Revised: Linear GCN. We present the result for linear GCN with the same setup as described in Sectionto check the goodness of our theory.

Original: The results are illustrated in Figure 9 where we observe that the theory holds very well for linear GCN than ReLU GCN.
Revised: The results are illustrated in Figure 15 where we observe that the theory holds very well for linear GCN than ReLU GCN.

Original: The class information is better preserved in S row than S sym especially for higher depth in the case of both GCN with and without skip connections.
Revised: The class information is better preserved in S row than S sym especially for higher depth in the case of both GCN with and without skip connections.

Original: All the conclusions derived in the main section hold here as well.
Revised: All the conclusions derived in the main section hold here as well.

Original: B. E XPERIMENTS ON R EAL D ATASET : C ITESEER
Revised: [Figure]

Original: In this section, we validate our theoretical findings on Citeseer without much of the assumptions.
Revised: In this section, we validate our theoretical findings on Citeseer without much of the assumptions.

Original: We consider multi-class node classification ( K = 6 ) using GCN with linear activations and relax the orthonormal feature condition, so XX T ̸
Revised: We consider multi-class node classification ( K = 6 ) using GCN with linear activations and relax the orthonormal feature condition, so XX T ̸

Original: = I n .
Revised: = I n .

Original: The NTKs for vanilla GCN, GCN with Skip-PC and Skip- α for depths d = 1 , 2 , 4 , 8 , 16 are computed and Figure 10 illustrates the results.
Revised: The NTKs for vanilla GCN, GCN with Skip-PC and Skip- α for depths d = 1 , 2 , 4 , 8 , 16 are computed and Figure 16 illustrates the results.

Original: All the observations made in Section 5 hold here as well and clear blocks emerge for S row making it the preferable choice as suggested in the theory.
Revised: All the observations made in Section 6 hold here as well and clear blocks emerge for S row making it the preferable choice as suggested in the theory.