Original: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE
Revised: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.
Revised: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.

Original: Nevertheless, it is well known that deep GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.
Revised: Nevertheless, it is well known that deep GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.

Original: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.
Revised: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.

Original: Compared to expressivity, trainability attempts to address a more fundamental question:
Revised: Compared to expressivity, trainability attempts to address a more fundamental question:

Original: Given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution via gradient descent-based optimizers?
Revised: Given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution via gradient descent-based optimizers?

Original: This work ﬁlls this gap by exploiting the Graph Neural
Revised: This work ﬁlls this gap by exploiting the Graph Neural

Original: Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.
Revised: Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.

Original: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.
Revised: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.

Original: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be merely able to mitigate the exponential decay of trainability mildly.
Revised: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be merely able to mitigate the exponential decay of trainability mildly.

Original: Inspired by our theoretical insights on trainability, we propose
Revised: Inspired by our theoretical insights on trainability, we propose

Original: Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, to alleviate the exponential decay problem more fundamentally.
Revised: Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, to alleviate the exponential decay problem more fundamentally.

Original: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.
Revised: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, Graph Neural Networks (GNNs) have shown incredible abilities to learn node or graph representations and achieved superior performance on various downstream tasks, such as node classiﬁcation (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Hamilton et al., 2017), graph classiﬁcation (Xu et al., 2019; Lee et al., 2019b; Yuan & Ji, 2020), and link prediction (Kipf & Welling, 2016), etc.
Revised: Recently, Graph Neural Networks (GNNs) have shown incredible abilities to learn node or graph representations and achieved superior performance on various downstream tasks, such as node classiﬁcation (Kipf & Welling, 2017; Veliˇckovi´c et al., 2018; Hamilton et al., 2017), graph classiﬁcation (Xu et al., 2019; Lee et al., 2019b; Yuan & Ji, 2020), and link prediction (Kipf & Welling, 2016), etc.

Original: However, most GNNs (e.g., GCNs) achieve their best only with a shallow depth, e.g.,or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.
Revised: However, most GNNs (e.g., GCNs) achieve their best only with a shallow depth, e.g.,or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.

Original: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.
Revised: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.

Original: Li et al.
Revised: Li et al.

Original: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.
Revised: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.

Original: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.
Revised: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.

Original: Oono & Suzuki (2020) investigated the expressive power of GNNs using the asymptotic behaviors as the layer goes to inﬁnity.
Revised: Oono & Suzuki (2020) investigated the expressive power of GNNs using the asymptotic behaviors as the layer goes to inﬁnity.

Original: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graph inherent in the graph spectra.
Revised: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graph inherent in the graph spectra.

Original: Nevertheless, it remains elusive how to theoretically understand why deep GCNs fail to optimize.
Revised: Nevertheless, it remains elusive how to theoretically understand why deep GCNs fail to optimize.

Original: Existing theoretical investigation (Oono & Suzuki, 2020; Xu et al., 2019) on GNNs focus mainly on expressivity, which measures the complexity of functions that can be represented by a neural network.
Revised: Existing theoretical investigation (Oono & Suzuki, 2020; Xu et al., 2019) on GNNs focus mainly on expressivity, which measures the complexity of functions that can be represented by a neural network.

Original: Exploring expressivity is theoretically convenient, but the corresponding conditions may be violated during the gradient training process, thereby leading to inconsistencies between theoretical conclusions and empirical results of trained networks (G¨uhring et al., 2020).
Revised: Exploring expressivity is theoretically convenient, but the corresponding conditions may be violated during the gradient training process, thereby leading to inconsistencies between theoretical conclusions and empirical results of trained networks (G¨uhring et al., 2020).

Original: Compared to expressivity, trainability addresses a more difﬁcult but fundamental perspective of neural networks:
Revised: Compared to expressivity, trainability addresses a more difﬁcult but fundamental perspective of neural networks:

Original: How effectively a neural network can be optimized via gradient descent.
Revised: How effectively a neural network can be optimized via gradient descent.

Original: The advantage of investigating trainability is that we can directly determine whether GNNs can be successfully trained under certain conditions, and to what extent.
Revised: The advantage of investigating trainability is that we can directly determine whether GNNs can be successfully trained under certain conditions, and to what extent.

Original: We are therefore inspired to raise two important questions:
Revised: We are therefore inspired to raise two important questions:

Original: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?
Revised: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?

Original: • Can we further design an algorithm to facilitate deeper GCNs, beneﬁting from our theoretical investigation?
Revised: • Can we further design an algorithm to facilitate deeper GCNs, beneﬁting from our theoretical investigation?

Original: Our answers are yes to both questions.
Revised: Our answers are yes to both questions.

Original: We resort to the inﬁnitely-wide multi-layer GCN to derive our solution.
Revised: We resort to the inﬁnitely-wide multi-layer GCN to derive our solution.

Original: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed that single hidden layer networks with random weights at initialization (without training) are Gaussian Processes (GPs) in the inﬁnite width limit.
Revised: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed that single hidden layer networks with random weights at initialization (without training) are Gaussian Processes (GPs) in the inﬁnite width limit.

Original: Later, the connection between GPs and multi-layer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2018; de G. Matthews et al., 2018) and orthogonal weights (Huang et al., 2021) was reported.
Revised: Later, the connection between GPs and multi-layer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2018; de G. Matthews et al., 2018) and orthogonal weights (Huang et al., 2021) was reported.

Original: Recent trends in Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.
Revised: Recent trends in Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.

Original: In particular, Jacot et al.
Revised: In particular, Jacot et al.

Original: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an
Revised: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an

Original: NTK.
Revised: NTK.

Original: Du et al.
Revised: Du et al.

Original: (2019b) formulated Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNNs.
Revised: (2019b) formulated Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNNs.

Original: Prior to the discovery of GNTK, there was little understanding of the non-convexity of GNNs, which is analytically intractable.
Revised: Prior to the discovery of GNTK, there was little understanding of the non-convexity of GNNs, which is analytically intractable.

Original: In the learning regime of GCN governed by GNTK, the optimization becomes an almost convex problem, making
Revised: In the learning regime of GCN governed by GNTK, the optimization becomes an almost convex problem, making

Original: GNTK a promising perspective to study the trainability of deep GCNs.
Revised: GNTK a promising perspective to study the trainability of deep GCNs.

Original: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.
Revised: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.

Original: In particular, we formulate the large-depth asymptotic behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b;
Revised: In particular, we formulate the large-depth asymptotic behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b;

Original: Xiao et al., 2020), through which we can analyze the optimization properties of deep GCNs.
Revised: Xiao et al., 2020), through which we can analyze the optimization properties of deep GCNs.

Original: Specifically, we make the following contributions:
Revised: Specifically, we make the following contributions:

Original: • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.
Revised: • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.

Original: We prove that all entries of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.
Revised: We prove that all entries of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.

Original: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.
Revised: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.

Original: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.
Revised: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.

Original: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate of trainability, but lack the ability to fundamentally solve the problem.
Revised: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate of trainability, but lack the ability to fundamentally solve the problem.

Original: This result enables to better understand why and to what extent recent residual connection-based methods work.
Revised: This result enables to better understand why and to what extent recent residual connection-based methods work.

Original: • Our theoretical framework provides insights to guide the development of deep GCNs.
Revised: • Our theoretical framework provides insights to guide the development of deep GCNs.

Original: We further propose an edge-based sampling method, named Critical DropEdge, to effectively mitigate the exponential decay of trainability.
Revised: We further propose an edge-based sampling method, named Critical DropEdge, to effectively mitigate the exponential decay of trainability.

Original: This graph-adaptive and connectivity-aware method is easy to
Revised: This graph-adaptive and connectivity-aware method is easy to

Original: implement in both ﬁnitely-wide and inﬁnitely-wide GNNs.
Revised: implement in both ﬁnitely-wide and inﬁnitely-wide GNNs.

Original: Our experiments show using the proposed method can outperform competitors in the large depth.
Revised: Our experiments show using the proposed method can outperform competitors in the large depth.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.
Revised: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.

Original: We then review NTK, making a connection to trainability.
Revised: We then review NTK, making a connection to trainability.

Original: Finally, we introduce GCNs along with our setup and notation.
Revised: Finally, we introduce GCNs along with our setup and notation.

Original: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION
Revised: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION

Original: We begin by considering a fully-connected network of depth L with width m l in each layer.
Revised: We begin by considering a fully-connected network of depth L with width m l in each layer.

Original: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .
Revised: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .

Original: Letting the pre-activations be given by h ( l ) i , information propagation in this network is governed by,
Revised: Letting the pre-activations be given by h ( l ) i , information propagation in this network is governed by,

Original: [Equation]
Revised: [Equation]

Original: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases, respectively.
Revised: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases, respectively.

Original: Given the parameterization that weights and biases are randomly generated by i.i.d.
Revised: Given the parameterization that weights and biases are randomly generated by i.i.d.

Original: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , . . .
Revised: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , . . .

Original: , m l − 1 → ∞ .
Revised: , m l − 1 → ∞ .

Original: This results from the central limit theorem (CLT).
Revised: This results from the central limit theorem (CLT).

Original: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )
Revised: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .
Revised: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .

Original: According to the signal propagation (1), the covariance matrix or GPK with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )
Revised: According to the signal propagation (1), the covariance matrix or GPK with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: σ 2 w E h ∼N (0 , Σ ( l − 1) )
Revised: σ 2 w E h ∼N (0 , Σ ( l − 1) )

Original: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]
Revised: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]

Original: + σ 2 b .
Revised: + σ 2 b .

Original: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2017).
Revised: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2017).

Original: In particular, expressivity describes to what extent two different inputs can be distinguished.
Revised: In particular, expressivity describes to what extent two different inputs can be distinguished.

Original: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )
Revised: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )

Original: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .
Revised: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .

Original: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;
Revised: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;

Original: Schoenholz et al., 2017).
Revised: Schoenholz et al., 2017).

Original: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.
Revised: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.

Original: Given this reason, we focus on the edge of chaos in this work.
Revised: Given this reason, we focus on the edge of chaos in this work.

Original: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z
Revised: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z

Original: [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal entries in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal entries in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .

Original: 2. N EURAL T ANGENT K ERNEL AND T RAINABILITY
Revised: 2. N EURAL T ANGENT K ERNEL AND T RAINABILITY

Original: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2017) have focused solely on initialization without training.
Revised: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2017) have focused solely on initialization without training.

Original: Jacot et al.
Revised: Jacot et al.

Original: (2018) took a step further by considering inﬁnitely-wide networks trained with gradient descent.
Revised: (2018) took a step further by considering inﬁnitely-wide networks trained with gradient descent.

Original: Let η be the learning rate, and L be the loss function.
Revised: Let η be the learning rate, and L be the loss function.

Original: The dynamics of gradient ﬂow for parameters
Revised: The dynamics of gradient ﬂow for parameters

Original: [Equation]
Revised: [Equation]

Original: ) × 1 , the vector of all parameters, is given by,
Revised: ) × 1 , the vector of all parameters, is given by,

Original: [Equation]
Revised: [Equation]

Original: Then, the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X
Revised: Then, the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X

Original: ) ∈ R
Revised: ) ∈ R

Original: nm L × 1 follow,
Revised: nm L × 1 follow,

Original: [Equation]
Revised: [Equation]

Original: where the NTK at time t is deﬁned as,
Revised: where the NTK at time t is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: In a general case, the NTK varies with the training time, thus providing no substantial insights into the convergence property of neural networks.
Revised: In a general case, the NTK varies with the training time, thus providing no substantial insights into the convergence property of neural networks.

Original: Interestingly, as shown by Jacot et al.
Revised: Interestingly, as shown by Jacot et al.

Original: (2018), the
Revised: (2018), the

Original: NTK converges to an explicit limiting kernel and does not change during training in the inﬁnitewidth limit.
Revised: NTK converges to an explicit limiting kernel and does not change during training in the inﬁnitewidth limit.

Original: This leads to a simple but profound result in the case of mean squared error (MSE) loss,
Revised: This leads to a simple but profound result in the case of mean squared error (MSE) loss,

Original: L = 12 (cid:107) f t ( X ) − Y
Revised: L = 12 (cid:107) f t ( X ) − Y

Original: (cid:107) 22 , where Y is the label associated with the input X ,
Revised: (cid:107) 22 , where Y is the label associated with the input X ,

Original: [Equation]
Revised: [Equation]

Original: (5) where Θ ∞ is the limiting kernel.
Revised: (5) where Θ ∞ is the limiting kernel.

Original: This is the solution to an ordinary differential equation.
Revised: This is the solution to an ordinary differential equation.

Original: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =
Revised: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =

Original: Y .
Revised: Y .

Original: As proved by Lemma 1 in Hayou et al.
Revised: As proved by Lemma 1 in Hayou et al.

Original: (2019b), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.
Revised: (2019b), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.

Original: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.
Revised: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.

Original: 2. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: 2. G RAPH C ONVOLUTIONAL N ETWORKS

Original: We deﬁne an undirected graph as G =
Revised: We deﬁne an undirected graph as G =

Original: ( V , E ) , where V is a set of nodes and E is a set of edges.
Revised: ( V , E ) , where V is a set of nodes and E is a set of edges.

Original: We denote the number of nodes in graph G by n = |V| .
Revised: We denote the number of nodes in graph G by n = |V| .

Original: The node features are denoted as h ( v ) ∈
Revised: The node features are denoted as h ( v ) ∈

Original: R d for each v ∈ V , with d being the dimension of node features.
Revised: R d for each v ∈ V , with d being the dimension of node features.

Original: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.
Revised: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.

Original: GCNs iteratively update node features through aggregating and transforming the representations of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representations of their neighbors.

Original: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.

Original: We deﬁne a propagation unit to be the combination of a R -layer multi-layer perceptron (MLP) and one aggregation operation.
Revised: We deﬁne a propagation unit to be the combination of a R -layer multi-layer perceptron (MLP) and one aggregation operation.

Original: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.
Revised: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.

Original: L is the total number of propagation units.
Revised: L is the total number of propagation units.

Original: To be speciﬁc, the node representation propagation in GCNs through an MLP follows the expression,
Revised: To be speciﬁc, the node representation propagation in GCNs through an MLP follows the expression,

Original: [Equation]
Revised: [Equation]

Original: (7) where W ( l ) ( r ) ∈
Revised: (7) where W ( l ) ( r ) ∈

Original: R m l × m l − 1 and b ( l ) ( r ) ∈
Revised: R m l × m l − 1 and b ( l ) ( r ) ∈

Original: R m l are the learnable weights and biases, respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node u and its neighbors.
Revised: R m l are the learnable weights and biases, respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node u and its neighbors.

Original: Equation (6) reveals the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).
Revised: Equation (6) reveals the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).

Original: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.
Revised: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.

Original: For the activation function, we focus on both ReLU and Tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =
Revised: For the activation function, we focus on both ReLU and Tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =

Original: tanh( x ) , respectively.
Revised: tanh( x ) , respectively.

Original: Without loss of generality, our theoretical framework can handle other common activation functions, whereas the GNTK work (Du et al., 2019b) only adopted ReLU.
Revised: Without loss of generality, our theoretical framework can handle other common activation functions, whereas the GNTK work (Du et al., 2019b) only adopted ReLU.

Original:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION
Revised:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION

Original: Based on the deﬁnition of NTK (4), we recursively formulate the propagation of GNTK in the inﬁnite-width limit.
Revised: Based on the deﬁnition of NTK (4), we recursively formulate the propagation of GNTK in the inﬁnite-width limit.

Original: As information propagation in a GCN is built on two operations: aggregation (6) and non-linear transformation (7), the corresponding formulas of GNTK are expressed as follows,
Revised: As information propagation in a GCN is built on two operations: aggregation (6) and non-linear transformation (7), the corresponding formulas of GNTK are expressed as follows,

Original: [Equation]
Revised: [Equation]

Original: The two equations above correspond to the aggregation operation and MLP transformation, respectively.
Revised: The two equations above correspond to the aggregation operation and MLP transformation, respectively.

Original: To compute the GNTK with respect to the depth
Revised: To compute the GNTK with respect to the depth

Original: [Equation]
Revised: [Equation]

Original: the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )
Revised: the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )

Original: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .
Revised: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .

Original: According to the CLT
Revised: According to the CLT

Original: [Equation]
Revised: [Equation]

Original: node representation h ( l ) ( r ) ( u ) is a
Revised: node representation h ( l ) ( r ) ( u ) is a

Original: Gaussian distribution in the inﬁnite-width limit.
Revised: Gaussian distribution in the inﬁnite-width limit.

Original: Applying this result to equations (6) and (7), the resultant covariance matrix is composed of two parts,
Revised: Applying this result to equations (6) and (7), the resultant covariance matrix is composed of two parts,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equation (10) results from the aggregation operation.
Revised: The ﬁrst equation (10) results from the aggregation operation.

Original: Meanwhile, the second equation (11) corresponds to the R -times non-linear transformations, where E z 1 ,z 2 takes the expectation with respect to a centered Gaussian process of covariance ˜Σ ( l ) ( r − 1)
Revised: Meanwhile, the second equation (11) corresponds to the R -times non-linear transformations, where E z 1 ,z 2 takes the expectation with respect to a centered Gaussian process of covariance ˜Σ ( l ) ( r − 1)

Original: ∈ R 2 × 2 for previous MLP layer across u, u (cid:48) , and ˙ φ denotes the derivative of φ .
Revised: ∈ R 2 × 2 for previous MLP layer across u, u (cid:48) , and ˙ φ denotes the derivative of φ .

Original: 3. T RAINABILITY IN THE L ARGE D EPTH
Revised: 3. T RAINABILITY IN THE L ARGE D EPTH

Original: We aim to characterize the behavior of GNTK matrix Θ ( l ) ( r ) ( G )
Revised: We aim to characterize the behavior of GNTK matrix Θ ( l ) ( r ) ( G )

Original: ∈ R n × n , as the depth tends to inﬁnity.
Revised: ∈ R n × n , as the depth tends to inﬁnity.

Original: From the GNTK formulation, both aggregation (8) and transformation (9) contribute simultaneously to the ﬁnal limiting result.
Revised: From the GNTK formulation, both aggregation (8) and transformation (9) contribute simultaneously to the ﬁnal limiting result.

Original: We derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is given as follows:
Revised: We derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is given as follows:

Original: Theorem 1 (Convergence rate of GNTK) .
Revised: Theorem 1 (Convergence rate of GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G )
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G )

Original: =
Revised: =

Original: A ( G )
Revised: A ( G )

Original: (cid:126) Θ ( l − 1) ( R ) ( G ) and
Revised: (cid:126) Θ ( l − 1) ( R ) ( G ) and

Original: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then, there exist constants 0 < α < 1 and
Revised: Then, there exist constants 0 < α < 1 and

Original: C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12)
Revised: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: The proof sketch follows a divide-and-conquer manner.
Revised: The proof sketch follows a divide-and-conquer manner.

Original: In particular, we ﬁrst analyze the network with only aggregation and prove that A ( G ) is a Markov transition matrix.
Revised: In particular, we ﬁrst analyze the network with only aggregation and prove that A ( G ) is a Markov transition matrix.

Original: Then, we formulate the behavior of MLP in the large depth based on Hayou et al.
Revised: Then, we formulate the behavior of MLP in the large depth based on Hayou et al.

Original: (2019b).
Revised: (2019b).

Original: Finally, we derive the ﬁnal result by considering the two operations.
Revised: Finally, we derive the ﬁnal result by considering the two operations.

Original: We leave the complete proof in Appendix B.
Revised: We leave the complete proof in Appendix B.

Original: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.
Revised: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.

Original: As the depth goes to inﬁnity, all entries in the GNTK converge to a unique quantity at an exponential rate.
Revised: As the depth goes to inﬁnity, all entries in the GNTK converge to a unique quantity at an exponential rate.

Original: We thus have Θ ( l ) ( r ) ( G ) ≈ (cid:126)π ( G ) T ( Rl(cid:126)v ) 1 n × n as l → ∞ , where 1 n × n is an ( n × n )- dimensional matrix whose entries are one.
Revised: We thus have Θ ( l ) ( r ) ( G ) ≈ (cid:126)π ( G ) T ( Rl(cid:126)v ) 1 n × n as l → ∞ , where 1 n × n is an ( n × n )- dimensional matrix whose entries are one.

Original: The exponential convergence rate of Θ ( l ) ( r ) ( G ) implies that the trainability of inﬁnitely-wide GCNs degenerates dramatically, as stated below.
Revised: The exponential convergence rate of Θ ( l ) ( r ) ( G ) implies that the trainability of inﬁnitely-wide GCNs degenerates dramatically, as stated below.

Original: Corollary 1 (Trainability of ultra-wide GCNs) .
Revised: Corollary 1 (Trainability of ultra-wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.
Revised: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.

Original: Then, the output function follows, f t ( X ) =
Revised: Then, the output function follows, f t ( X ) =

Original: e
Revised: e

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: ( l ) ( r ) ( G ) t ) Y , where X and Y are node features and labels from training set.
Revised: ( l ) ( r ) ( G ) t ) Y , where X and Y are node features and labels from training set.

Original: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .
Revised: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .

Original: Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( X ) − Y
Revised: Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( X ) − Y

Original: (cid:107) > C .
Revised: (cid:107) > C .

Original: We leave proof in Appendix C.
Revised: We leave proof in Appendix C.

Original: According to the above corollary, as l → ∞ , the GNTK matrix would become a singular matrix.
Revised: According to the above corollary, as l → ∞ , the GNTK matrix would become a singular matrix.

Original: This would lead to a discrepancy between output f t ( X ) and label
Revised: This would lead to a discrepancy between output f t ( X ) and label

Original: Y , which means GNTK loses the ability to ﬁt the label.
Revised: Y , which means GNTK loses the ability to ﬁt the label.

Original: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.
Revised: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.

Original:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION
Revised:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION

Original: We have so far characterized the trainability of vanilla GCNs through the GNTK and showed that the trainability of ultra-wide GCNs drops at an exponential rate.
Revised: We have so far characterized the trainability of vanilla GCNs through the GNTK and showed that the trainability of ultra-wide GCNs drops at an exponential rate.

Original: Recently, considerable efforts have been made to deepen GCNs, among which residual connection-based techniques are widely applied to resolve the over-smoothing problem (Li et al., 2019).
Revised: Recently, considerable efforts have been made to deepen GCNs, among which residual connection-based techniques are widely applied to resolve the over-smoothing problem (Li et al., 2019).

Original: We now apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.
Revised: We now apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.

Original: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,
Revised: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,

Original: [Equation]
Revised: [Equation]

Original: where 0 < δ < 1 .
Revised: where 0 < δ < 1 .

Original: Taking equation (12) as a new aggregation process, then (cid:126) Θ ( l ) ( G )
Revised: Taking equation (12) as a new aggregation process, then (cid:126) Θ ( l ) ( G )

Original: = ˜
Revised: = ˜

Original: A ( G ) (cid:126) Θ ( l − 1) ( G ) ,
Revised: A ( G ) (cid:126) Θ ( l − 1) ( G ) ,

Original: where ˜ A ( G ) =
Revised: where ˜ A ( G ) =

Original: (1 − δ )
Revised: (1 − δ )

Original: A ( G ) + δI .
Revised: A ( G ) + δI .

Original: We prove that ˜
Revised: We prove that ˜

Original: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix
Revised: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix

Original: A ( G ) .
Revised: A ( G ) .

Original: Theorem 2 (Convergence rate of residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate of residual connection in aggregation) .

Original: Consider a GNTK of nonlinear transformation (9) and residual connection (12).
Revised: Consider a GNTK of nonlinear transformation (9) and residual connection (12).

Original: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <
Revised: A ( G ) , there exist constants 0 <

Original: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on
Revised: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: The detailed proof of Theorem 2 and the relationship between convergence rate and the second largest eigvenvalue can be found in Appendix D.1.
Revised: The detailed proof of Theorem 2 and the relationship between convergence rate and the second largest eigvenvalue can be found in Appendix D.1.

Original: This theorem implies that a residual connection in aggregation can slow down the convergence rate, which is consistent with empirical observations that residual connection can help deepen GCNs.
Revised: This theorem implies that a residual connection in aggregation can slow down the convergence rate, which is consistent with empirical observations that residual connection can help deepen GCNs.

Original: Then, we consider residual connection only applied on non-linear transformation (MLP).
Revised: Then, we consider residual connection only applied on non-linear transformation (MLP).

Original: In this case, the recursive equation for the corresponding GNTK can be expressed as,
Revised: In this case, the recursive equation for the corresponding GNTK can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: This formula is similar to the vanilla GNTK in the inﬁnite-width limit.
Revised: This formula is similar to the vanilla GNTK in the inﬁnite-width limit.

Original: Only an additional residual term appears according to residual connection.
Revised: Only an additional residual term appears according to residual connection.

Original: It turns out that this term may not help slow down the convergence rate for non-linear transformation.
Revised: It turns out that this term may not help slow down the convergence rate for non-linear transformation.

Original: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .

Original: Consider a GNTK of the form (8) and (13).
Revised: Consider a GNTK of the form (8) and (13).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on R , such that, (cid:12)(cid:12)
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on R , such that, (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Note that α in Theorem 3 is the same as in Theorem 1.
Revised: Note that α in Theorem 3 is the same as in Theorem 1.

Original: The proof of Theorem 3 can be found in Appendix D.2.
Revised: The proof of Theorem 3 can be found in Appendix D.2.

Original: Theorem 3 demonstrates that adding residual connection in MLP can not even reduce the convergence rate of trainability.
Revised: Theorem 3 demonstrates that adding residual connection in MLP can not even reduce the convergence rate of trainability.

Original: Finally, we consider residual connection applied to both aggregation and non-linear transformation simultaneously:
Revised: Finally, we consider residual connection applied to both aggregation and non-linear transformation simultaneously:

Original: Corollary 2 (Convergence rate of GNTK with residual connection in aggregation and transformation) .
Revised: Corollary 2 (Convergence rate of GNTK with residual connection in aggregation and transformation) .

Original: Consider a GNTK of the form (12) and (13).
Revised: Consider a GNTK of the form (12) and (13).

Original: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 < ˜
Revised: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 < ˜

Original: α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u
Revised: ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u

Original: [Equation]
Revised: [Equation]

Original: ) −
Revised: ) −

Original: [Equation]
Revised: [Equation]

Original: G ) T (cid:0) Rl
Revised: G ) T (cid:0) Rl

Original: [Equation]
Revised: [Equation]

Original: 1 +
Revised: 1 +

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: (cid:1)(cid:12)(cid:12) ≤ C ˜ α l
Revised: (cid:1)(cid:12)(cid:12) ≤ C ˜ α l

Original: [Equation]
Revised: [Equation]

Original: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE
Revised: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE

Original: Residual connection is designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay of trainability.
Revised: Residual connection is designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay of trainability.

Original: To better resolve this problem, we need to look deeper into the root cause of the problem – the transition matrix corresponding to the aggregation operation.
Revised: To better resolve this problem, we need to look deeper into the root cause of the problem – the transition matrix corresponding to the aggregation operation.

Original: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected.
Revised: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected.

Original: Thus, breaking the connectivity condition is a promising way of better solving the exponential decay problem.
Revised: Thus, breaking the connectivity condition is a promising way of better solving the exponential decay problem.

Original: One such method is to perform edge sampling guided by the critical percolation theory (Huang et al., 2018; Erd˝os & R´enyi, 1961) in random graphs.
Revised: One such method is to perform edge sampling guided by the critical percolation theory (Huang et al., 2018; Erd˝os & R´enyi, 1961) in random graphs.

Original: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.
Revised: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.

Original: A random graph ˆ G is achieved by randomly and uniformly preserving some edges from the complete graph with an edge probability as p = | E | E t , where | E | is the number of edges preserved in the random graph.
Revised: A random graph ˆ G is achieved by randomly and uniformly preserving some edges from the complete graph with an edge probability as p = | E | E t , where | E | is the number of edges preserved in the random graph.

Original: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c
Revised: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c

Original: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).
Revised: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).

Original: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there exists a path from a ﬁxed point to another point within a certain distance decreases polynomially.
Revised: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there exists a path from a ﬁxed point to another point within a certain distance decreases polynomially.

Original: Proposition 1 (Critical connectivity in random graph (Erd˝os & R´enyi, 1961)) .
Revised: Proposition 1 (Critical connectivity in random graph (Erd˝os & R´enyi, 1961)) .

Original: Suppose a random graph ˆ G has n nodes with a constant edge probability p .
Revised: Suppose a random graph ˆ G has n nodes with a constant edge probability p .

Original: (1) If p < p c , then almost every random graph is such that its largest component 1 is of size O (log n ) ; (2) If p > p c , the random graph has a giant component of size (1 − α p
Revised: (1) If p < p c , then almost every random graph is such that its largest component 1 is of size O (log n ) ; (2) If p > p c , the random graph has a giant component of size (1 − α p

Original: + o (1)) n , where α p < 1 ; (3) If p = p c , then the maximal size of a component of almost every graph has order n 2 / 3 .
Revised: + o (1)) n , where α p < 1 ; (3) If p = p c , then the maximal size of a component of almost every graph has order n 2 / 3 .

Original: [Figure]
Revised: [Figure]

Original: The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate.
Revised: The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate.

Original: This inspires us to solve the problem of exponentially dropping trainability through designing a graph-dependent and connectivity-aware sampling algorithm called Critical DropEdge.
Revised: This inspires us to solve the problem of exponentially dropping trainability through designing a graph-dependent and connectivity-aware sampling algorithm called Critical DropEdge.

Original: In particular, given a graph G , we randomly drop some edges and preserve the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.
Revised: In particular, given a graph G , we randomly drop some edges and preserve the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.

Original: Unlike
Revised: Unlike

Original: DropEdge (Rong et al., 2019) that randomly removes a certain number of edges from the input graph, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | .
Revised: DropEdge (Rong et al., 2019) that randomly removes a certain number of edges from the input graph, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | .

Original: It is worth noting that DropEdge may choose the edge probability as p < p c where information can only be passed to a distance of O (log n ) in the graph, or p > p c where the exponential decay of trainability may occur.
Revised: It is worth noting that DropEdge may choose the edge probability as p < p c where information can only be passed to a distance of O (log n ) in the graph, or p > p c where the exponential decay of trainability may occur.

Original: A further discussion of the relationship between Proposition 1 and the trainability of the corresponding GNTK can be found in Appendix E.
Revised: A further discussion of the relationship between Proposition 1 and the trainability of the corresponding GNTK can be found in Appendix E.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically verify our theoretical results and validate the proposed Critical
Revised: In this section, we empirically verify our theoretical results and validate the proposed Critical

Original: DropEdge method on node classiﬁcation tasks.
Revised: DropEdge method on node classiﬁcation tasks.

Original: Details of four real-world graph datasets used for node classiﬁcation are summarized in Table 3 in Appendix F.1.
Revised: Details of four real-world graph datasets used for node classiﬁcation are summarized in Table 3 in Appendix F.1.

Original: 5. C ONVERGENCE R ESULTS OF GNTK S
Revised: 5. C ONVERGENCE R ESULTS OF GNTK S

Original: Theorems 1-3 provide theoretical convergence rates for (residual) GNTK.
Revised: Theorems 1-3 provide theoretical convergence rates for (residual) GNTK.

Original: We show the corresponding numerical veriﬁcation in Figure 1.
Revised: We show the corresponding numerical veriﬁcation in Figure 1.

Original: We select a graph randomly from a bioinformatics dataset (i.e., MUTAG), which consists of 18 nodes and 21 edges.
Revised: We select a graph randomly from a bioinformatics dataset (i.e., MUTAG), which consists of 18 nodes and 21 edges.

Original: We generate a GNTK of the graph using the implementation of Du et al.
Revised: We generate a GNTK of the graph using the implementation of Du et al.

Original: (2019b), with ReLU activation, R = 3 , and L = 300 .
Revised: (2019b), with ReLU activation, R = 3 , and L = 300 .

Original: Figure 1(a)- (c) show that all entries of normalized GNTKs converge to an identical value as the depth goes larger.
Revised: Figure 1(a)- (c) show that all entries of normalized GNTKs converge to an identical value as the depth goes larger.

Original: Figure 1(d)-(f) further indicates that the convergence rate of GNTK is exponential, as reﬂected by our theorems.
Revised: Figure 1(d)-(f) further indicates that the convergence rate of GNTK is exponential, as reﬂected by our theorems.

Original: By comparing convergence rates, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with Theorem 2.
Revised: By comparing convergence rates, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with Theorem 2.

Original: 5. T RAINABILITY OF W IDE GCN S
Revised: 5. T RAINABILITY OF W IDE GCN S

Original: We further examine whether ultra-wide GCNs can be trained successfully for node classiﬁcation.
Revised: We further examine whether ultra-wide GCNs can be trained successfully for node classiﬁcation.

Original: We conduct experiments on a GCN (Kipf & Welling, 2017), where we apply a width of 1 , 000 at each hidden layer and the depth ranging from 2 to 29 .
Revised: We conduct experiments on a GCN (Kipf & Welling, 2017), where we apply a width of 1 , 000 at each hidden layer and the depth ranging from 2 to 29 .

Original: Figure 2 shows the training and test accuracy on Cora, Citesser and Pubmed after 300 training epochs.
Revised: Figure 2 shows the training and test accuracy on Cora, Citesser and Pubmed after 300 training epochs.

Original: These results show a dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation, as revealed by Corollary 1.
Revised: These results show a dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation, as revealed by Corollary 1.

Original: 5. P ERFORMANCE OF C
Revised: 5. P ERFORMANCE OF C

Original: [Figure]
Revised: [Figure]

Original: DGE
Revised: DGE

Original: [Figure]
Revised: [Figure]

Original: We apply Critical DropEdge (referred to as CDropEdge) to ﬁnitely-wide and inﬁnitely-wide
Revised: We apply Critical DropEdge (referred to as CDropEdge) to ﬁnitely-wide and inﬁnitely-wide

Original: GNNs on semi-supervised node classiﬁcation.
Revised: GNNs on semi-supervised node classiﬁcation.

Original: The implementation details are given in Appendix F.2.
Revised: The implementation details are given in Appendix F.2.

Original: For ﬁnitely-wide GNNs, we consider three backbones:
Revised: For ﬁnitely-wide GNNs, we consider three backbones:

Original: GCN, JKNet, and IncepGCN (Rong et al., 2019).
Revised: GCN, JKNet, and IncepGCN (Rong et al., 2019).

Original: For DropEdge, we use the hyperparameters reported in Rong et al.
Revised: For DropEdge, we use the hyperparameters reported in Rong et al.

Original: (2019) to obtain the results.
Revised: (2019) to obtain the results.

Original: For C-DropEdge, we perform a random hyper-parameter search and ﬁx the edge preserving rate as ρ ( G ) =
Revised: For C-DropEdge, we perform a random hyper-parameter search and ﬁx the edge preserving rate as ρ ( G ) =

Original: | V | 2 | E | .
Revised: | V | 2 | E | .

Original: In addition, we compare with
Revised: In addition, we compare with

Original: DGN (Zhou et al., 2020), a normalization-based baseline, which uses GCN (Kipf & Welling, 2017) and GAT (Veliˇckovi´c et al., 2018) as backbones.
Revised: DGN (Zhou et al., 2020), a normalization-based baseline, which uses GCN (Kipf & Welling, 2017) and GAT (Veliˇckovi´c et al., 2018) as backbones.

Original: Table 1 summarizes node classiﬁcation performance of ﬁnitely-wide GNNs with 4/8/16/32 layers on three citation networks (Cora, Citesser, Pubmed) and one co-author network (Physics).
Revised: Table 1 summarizes node classiﬁcation performance of ﬁnitely-wide GNNs with 4/8/16/32 layers on three citation networks (Cora, Citesser, Pubmed) and one co-author network (Physics).

Original: In particular, we report the best performance across different backbones for DropEdge, C-DropEdge and
Revised: In particular, we report the best performance across different backbones for DropEdge, C-DropEdge and

Original: DGN, and leave separate results with different backbones in Appendix G.2.
Revised: DGN, and leave separate results with different backbones in Appendix G.2.

Original: The reported results are the mean and standard deviation over 10 times.
Revised: The reported results are the mean and standard deviation over 10 times.

Original: As can be seen, C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.
Revised: As can be seen, C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.

Original: Besides, C-DropEdge can achieve smaller error variances, demonstrating stronger robustness than DropEdge.
Revised: Besides, C-DropEdge can achieve smaller error variances, demonstrating stronger robustness than DropEdge.

Original: For inﬁnitely-wide GCNs, we consider two backbones: GCN (Kipf & Welling, 2017) and
Revised: For inﬁnitely-wide GCNs, we consider two backbones: GCN (Kipf & Welling, 2017) and

Original: JKNet (Xu et al., 2018).
Revised: JKNet (Xu et al., 2018).

Original: The corresponding results can be found in Appendix G.1.
Revised: The corresponding results can be found in Appendix G.1.

Original: Comparison to DropEdge.
Revised: Comparison to DropEdge.

Original: DropEdge and C-DropEdge differ largely in their hyper-parameter search space.
Revised: DropEdge and C-DropEdge differ largely in their hyper-parameter search space.

Original: To ensure good performance, DropEdge needs to exhaustively search for the most appropriate edge preserving percentage – one of the most signiﬁcant hyper-parameters – that has a crucial inﬂuence on the ﬁnal performance.
Revised: To ensure good performance, DropEdge needs to exhaustively search for the most appropriate edge preserving percentage – one of the most signiﬁcant hyper-parameters – that has a crucial inﬂuence on the ﬁnal performance.

Original: In contrast, C-DropEdge has a ﬁxed and graph-dependent edge preserving rate, which implies its hyper-parameter space is much smaller than that of DropEdge.
Revised: In contrast, C-DropEdge has a ﬁxed and graph-dependent edge preserving rate, which implies its hyper-parameter space is much smaller than that of DropEdge.

Original: To verify if C-DropEdge can achieve the results close to optimal, we conduct experiments with various edge preserving rates, and present the results in Table 2.
Revised: To verify if C-DropEdge can achieve the results close to optimal, we conduct experiments with various edge preserving rates, and present the results in Table 2.

Original: We conclude that, from both theoretical and empirical perspectives, the edge preserving rate set by C-DropEdge is reasonable and effective, achieving the results close to optimal.
Revised: We conclude that, from both theoretical and empirical perspectives, the edge preserving percentage set by C-DropEdge is reasonable and effective, achieving the results close to optimal.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Tangent Kernel.
Revised: Neural Tangent Kernel.

Original: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.
Revised: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.

Original: In the inﬁnite-width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018; Lee et al., 2019a; Allen-Zhu et al., 2019; Du et al., 2019a; Zou et al., 2018).
Revised: In the inﬁnite-width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018; Lee et al., 2019a; Allen-Zhu et al., 2019; Du et al., 2019a; Zou et al., 2018).

Original: Besides, NTK has been applied to various architectures and brought a wealth of results, such as orthogonal initialization (Huang et al., 2021), convolutions (Arora et al., 2019), SVM (Chen et al., 2021), attention (Hron et al., 2020).
Revised: Besides, NTK has been applied to various architectures and brought a wealth of results, such as orthogonal initialization (Huang et al., 2021), convolutions (Arora et al., 2019), SVM (Chen et al., 2021), attention (Hron et al., 2020).

Original: As for graph networks, GNTK helps us understand how GNNs learn a class of smooth functions on graphs (Du et al., 2019b) and how they extrapolate differently from multi-layer perceptron (Xu et al., 2020).
Revised: As for graph networks, GNTK helps us understand how GNNs learn a class of smooth functions on graphs (Du et al., 2019b) and how they extrapolate differently from multi-layer perceptron (Xu et al., 2020).

Original: Deep Graph Neural Networks.
Revised: Deep Graph Neural Networks.

Original: Since deep GNNs suffer from the over-smoothing problem, a large and growing body of literature has made efforts in deepening GNNs.
Revised: Since deep GNNs suffer from the over-smoothing problem, a large and growing body of literature has made efforts in deepening GNNs.

Original: There is a line of methods that resort to residual connection to retain their feature expressivity in deep layers.
Revised: There is a line of methods that resort to residual connection to retain their feature expressivity in deep layers.

Original: Xu et al.
Revised: Xu et al.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: use skip connection along with node representations from different neighborhood ranges to preserve the locality of node representations.
Revised: use skip connection along with node representations from different neighborhood ranges to preserve the locality of node representations.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019) derive a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.
Revised: (2019) derive a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.

Original: Other similar works using residual connection can also be seen in (Li et al., 2019;
Revised: Other similar works using residual connection can also be seen in (Li et al., 2019;

Original: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).
Revised: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).

Original: Another line of approaches tackle the issue via regularization mechanisms, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2020), and group normalization (Zhou et al., 2020).
Revised: Another line of approaches tackle the issue via regularization mechanisms, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2020), and group normalization (Zhou et al., 2020).

Original: Recently, a series of latest works (Loukas, 2020; Zeng et al., 2020; Li et al., 2020;
Revised: Recently, a series of latest works (Loukas, 2020; Zeng et al., 2020; Li et al., 2020;

Original: Cong et al., 2021; Huang et al., 2020) explore the underlying reasons for performance degradation towards mitigation solutions.
Revised: Cong et al., 2021; Huang et al., 2020) explore the underlying reasons for performance degradation towards mitigation solutions.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.
Revised: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.

Original: We prove that the trainability drops at an exponential rate due to the aggregation operation.
Revised: We prove that the trainability drops at an exponential rate due to the aggregation operation.

Original: Furthermore, we apply our theoretical framework to investigate to what extent residual connection-based techniques help deepen GCNs.
Revised: Furthermore, we apply our theoretical framework to investigate to what extent residual connection-based techniques help deepen GCNs.

Original: We demonstrate that these techniques can merely slow down the decay rate, but are unable to solve the exponential decay problem in essence.
Revised: We demonstrate that these techniques can merely slow down the decay rate, but are unable to solve the exponential decay problem in essence.

Original: To overcome the trainability loss problem, we further propose Critical DropEdge illuminated by our theoretical framework.
Revised: To overcome the trainability loss problem, we further propose Critical DropEdge illuminated by our theoretical framework.

Original: The experimental results conﬁrm that our method can mitigate the trainability problem of deep GCNs.
Revised: The experimental results conﬁrm that our method can mitigate the trainability problem of deep GCNs.

Original: Future research directions include designing a critical node-centric method so as to make better use of node information.
Revised: Future research directions include designing a critical node-centric method so as to make better use of node information.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This work was partially supported by a Collaborative Research Project grant between The University of Sydney and Data61, Australia.
Revised: This work was partially supported by a Collaborative Research Project grant between The University of Sydney and Data61, Australia.

Original: We thank the anonymous reviewers for useful suggestions to improve the paper.
Revised: We thank the anonymous reviewers for useful suggestions to improve the paper.

Original: We also thank Ye Su for helpful discussions.
Revised: We also thank Ye Su for helpful discussions.

Original: This work was also in collaboration with Digital Research Centre Denmark – DIREC, supported by Innovation Fund Denmark
Revised: This work was also in collaboration with Digital Research Centre Denmark – DIREC, supported by Innovation Fund Denmark

Original: A APPENDIX: A G ENERAL GCN A RCHITECTURE
Revised: A APPENDIX: A G ENERAL GCN A RCHITECTURE

Original: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.

Original: Figure 3 illustrates an overview of information propagation in a general GCN considered in this work.
Revised: Figure 3 illustrates an overview of information propagation in a general GCN considered in this work.

Original: MLP Aggregation 𝑟 = 𝑅 𝑙 =𝑟
Revised: MLP Aggregation 𝑟 = 𝑅 𝑙 =𝑟

Original: =𝑙 = L
Revised: =𝑙 = L

Original: Propagation Unit
Revised: Propagation Unit

Original: MLP Aggregation 𝑟 =
Revised: MLP Aggregation 𝑟 =

Original: Propagation Unit 𝑟 =
Revised: Propagation Unit 𝑟 =

Original: 𝑅
Revised: 𝑅

Original: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK
Revised: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK

Original: In Theorem 1 we demonstrate that matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.
Revised: In Theorem 1 we demonstrate that matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.

Original: To this end, we block the non-linear transformation by setting R = 0 .
Revised: To this end, we block the non-linear transformation by setting R = 0 .

Original: Then, the propagation of
Revised: Then, the propagation of

Original: GNTK can be expressed as,
Revised: GNTK can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate the calculation, we rewrite Equation (14) as the following format,
Revised: In order to facilitate the calculation, we rewrite Equation (14) as the following format,

Original: [Equation]
Revised: [Equation]

Original: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.
Revised: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.

Original: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( G ) in the following lemma,
Revised: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( G ) in the following lemma,

Original: Lemma 1 (Convergence of aggregation) .
Revised: Lemma 1 (Convergence of aggregation) .

Original: Assume R = 0 , then
Revised: Assume R = 0 , then

Original: [Equation]
Revised: [Equation]

Original: ) where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) =
Revised: ) where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) =

Original: (cid:126)π ( G ) .
Revised: (cid:126)π ( G ) .

Original: Proof.
Revised: Proof.

Original: When R = 0 , Equation (14) reduces to,
Revised: When R = 0 , Equation (14) reduces to,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite the above equation in the format of matrix,
Revised: In order to facilitate calculation, we rewrite the above equation in the format of matrix,

Original: [Equation]
Revised: [Equation]

Original: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.
Revised: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.

Original: Thus, the matrix operation A ( l ) ( G ) ∈
Revised: Thus, the matrix operation A ( l ) ( G ) ∈

Original: R n 2 × n 2 .
Revised: R n 2 × n 2 .

Original: It is obvious that,
Revised: It is obvious that,

Original: [Equation]
Revised: [Equation]

Original: This implies that the aggregation operation is the same for each layer.
Revised: This implies that the aggregation operation is the same for each layer.

Original: The next step is to prove
Revised: The next step is to prove

Original: A ( G ) is a stochastic matrix (transition matrix):
Revised: A ( G ) is a stochastic matrix (transition matrix):

Original: [Equation]
Revised: [Equation]

Original: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrices,
Revised: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrices,

Original: [Equation]
Revised: [Equation]

Original: where B ( G ) , C ( G ) ∈ R n × n .
Revised: where B ( G ) , C ( G ) ∈ R n × n .

Original: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor
Revised: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor

Original: [Equation]
Revised: [Equation]

Original:  (2)
Revised:  (2)

Original: The element of matrix C ( G ) is determined by whether there exists an edge between two vertices,
Revised: The element of matrix C ( G ) is determined by whether there exists an edge between two vertices,

Original: [Equation]
Revised: [Equation]

Original: where ˜ δ ij = 1 if i ==
Revised: where ˜ δ ij = 1 if i ==

Original: j or there is an edge between vertex i and j , otherwise ˜ δ ij = 0 .
Revised: j or there is an edge between vertex i and j , otherwise ˜ δ ij = 0 .

Original: We then use the property of matrix B and C before Kronecker product,
Revised: We then use the property of matrix B and C before Kronecker product,

Original: [Equation]
Revised: [Equation]

Original: According to the deﬁnition of Kronecker product, we have,
Revised: According to the deﬁnition of Kronecker product, we have,

Original: [Equation]
Revised: [Equation]

Original: where i = a + ( c − 1) n , and j = b + ( d − 1) n .
Revised: where i = a + ( c − 1) n , and j = b + ( d − 1) n .

Original: So far, we have proved that A ( G ) is a stochastic matrix.
Revised: So far, we have proved that A ( G ) is a stochastic matrix.

Original: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, which does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution, which is also an eigenvector of the probability matrix, associated with eigenvalue 1:
Revised: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, which does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution, which is also an eigenvector of the probability matrix, associated with eigenvalue 1:

Original: [Equation]
Revised: [Equation]

Original: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.
Revised: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.

Original: Thus, the convergence rate is governed by the second largest eigenvalue.
Revised: Thus, the convergence rate is governed by the second largest eigenvalue.

Original: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,
Revised: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,

Original: [Equation]
Revised: [Equation]

Original: where Π( G ) =  
Revised: where Π( G ) =  

Original: [Equation]
Revised: [Equation]

Original: (cid:126)π
Revised: (cid:126)π

Original: [Equation]
Revised: [Equation]

Original: T ...
Revised: T ...

Original: (cid:126)π ( G
Revised: (cid:126)π ( G

Original: [Equation]
Revised: [Equation]

Original: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.
Revised: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.

Original: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,
Revised: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈
Revised: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈

Original: R nn (cid:48)
Revised: R nn (cid:48)

Original: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertices in graph G (cid:48) .
Revised: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertices in graph G (cid:48) .

Original: B. C ONVERGENCE OF MLP GPK
Revised: B. C ONVERGENCE OF MLP GPK

Original: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.
Revised: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.

Original: By doing so, we consider a network with only non-linear transformation, known as a pure MLP.
Revised: By doing so, we consider a network with only non-linear transformation, known as a pure MLP.

Original: This leads to Σ ( l ) ( r ) ( u, u (cid:48) )
Revised: This leads to Σ ( l ) ( r ) ( u, u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.
Revised: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.

Original: And we rewrite the propagation function for GPK as follows,
Revised: And we rewrite the propagation function for GPK as follows,

Original: [Equation]
Revised: [Equation]

Original: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,
Revised: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,

Original: [Equation]
Revised: [Equation]

Original: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.
Revised: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.

Original: In particular, we set the value of hyper-parameters to satisfy,
Revised: In particular, we set the value of hyper-parameters to satisfy,

Original: [Equation]
Revised: [Equation]

Original: (18) where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: (18) where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, equation (18) requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, equation (18) requires σ 2 w = 2 and σ 2 b = 0 .

Original: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,
Revised: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 (Proposition 1 and 3 in Hayou et al.
Revised: Lemma 2 (Proposition 1 and 3 in Hayou et al.

Original: (2019a)) .
Revised: (2019a)) .

Original: Assume a network without aggregation,
Revised: Assume a network without aggregation,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., L = 0 , with a Lipschitz nonlinearity φ , then,
Revised: e., L = 0 , with a Lipschitz nonlinearity φ , then,

Original: • φ ( x ) =
Revised: • φ ( x ) =

Original: ( x ) + ,
Revised: ( x ) + ,

Original: [Equation]
Revised: [Equation]

Original: ( u,
Revised: ( u,

Original: [Equation]
Revised: [Equation]

Original: ∞ • φ ( x ) =
Revised: ∞ • φ ( x ) =

Original: tanh( x ) , 1 − C
Revised: tanh( x ) , 1 − C

Original: [Equation]
Revised: [Equation]

Original: ( u,
Revised: ( u,

Original: [Equation]
Revised: [Equation]

Original: as r
Revised: as r

Original: [Equation]
Revised: [Equation]

Original: ∞ where β
Revised: ∞ where β

Original: = 2 (cid:82)
Revised: = 2 (cid:82)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, diagonal elements and non-diagonal elements:
Revised: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, diagonal elements and non-diagonal elements:

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: , and
Revised: , and

Original: [Equation]
Revised: [Equation]

Original: , with C r −
Revised: , with C r −

Original: [Equation]
Revised: [Equation]

Original: u,
Revised: u,

Original: [Equation]
Revised: [Equation]

Original: ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .
Revised: ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .

Original: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )
Revised: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u (cid:48) , u (cid:48) ) .
Revised: Σ ( r ) ( u (cid:48) , u (cid:48) ) .

Original: We then proceed with the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.
Revised: We then proceed with the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.

Original: ReLU activation, φ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: The recursive equation (16) for q r ( u ) reduces to,
Revised: The recursive equation (16) for q r ( u ) reduces to,

Original: [Equation]
Revised: [Equation]

Original: The edge of chaos condition σ 2 w (cid:82) D z [ φ
Revised: The edge of chaos condition σ 2 w (cid:82) D z [ φ

Original: [Equation]
Revised: [Equation]

Original: ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,
Revised: ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,

Original: [Equation]
Revised: [Equation]

Original: Then the second integration for C r ( u, u (cid:48) ) becomes,
Revised: Then the second integration for C r ( u, u (cid:48) ) becomes,

Original: [Equation]
Revised: [Equation]

Original: To investigate the propagation of C r ( u, u (cid:48) )
Revised: To investigate the propagation of C r ( u, u (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: we set q r ( u ) =
Revised: we set q r ( u ) =

Original: q r ( u (cid:48) )
Revised: q r ( u (cid:48) )

Original: =
Revised: =

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: and deﬁne,
Revised: and deﬁne,

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: Let x ∈ [
Revised: Let x ∈ [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: the derivative of f ( x ) satisﬁes,
Revised: the derivative of f ( x ) satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: This can be seen from a simple derivation,
Revised: This can be seen from a simple derivation,

Original: [Equation]
Revised: [Equation]

Original: Using an identity for Gaussian integration (cid:82) D z zg ( z ) =
Revised: Using an identity for Gaussian integration (cid:82) D z zg ( z ) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) yields,
Revised: ) yields,

Original: [Equation]
Revised: [Equation]

Original: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48)
Revised: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: [Equation]
Revised: [Equation]

Original: So using the equation above and the condition f (cid:48) (0)
Revised: So using the equation above and the condition f (cid:48) (0)

Original: =
Revised: =

Original: 12 , we can obtain another form of the derivative of f ( x ) ,
Revised: 12 , we can obtain another form of the derivative of f ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,
Revised: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Substituting f ( x ) =
Revised: Substituting f ( x ) =

Original: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression above here, we have,
Revised: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression above here, we have,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Now we study the behavior of C r ( u, u (cid:48) )
Revised: Now we study the behavior of C r ( u, u (cid:48) )

Original: as r tends to inﬁnity.
Revised: as r tends to inﬁnity.

Original: Using Taylor expansion, we have,
Revised: Using Taylor expansion, we have,

Original: [Equation]
Revised: [Equation]

Original: By induction analysis, the sequence C r ( u, u (cid:48) ) increases to the ﬁxed point 1.
Revised: By induction analysis, the sequence C r ( u, u (cid:48) ) increases to the ﬁxed point 1.

Original: Besides, we can replace x with C r ( u, u (cid:48) ) ,
Revised: Besides, we can replace x with C r ( u, u (cid:48) ) ,

Original: [Equation]
Revised: [Equation]

Original: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,
Revised: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: so that, γ
Revised: so that, γ

Original: [Equation]
Revised: [Equation]

Original: As r tends to inﬁnity, we have,
Revised: As r tends to inﬁnity, we have,

Original: [Equation]
Revised: [Equation]

Original: It means,
Revised: It means,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have,
Revised: Therefore, we have,

Original: [Equation]
Revised: [Equation]

Original: Tanh activation, φ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: For the argument of ﬁxed point, we ask readers to refer to
Revised: For the argument of ﬁxed point, we ask readers to refer to

Original: Hayou et al.
Revised: Hayou et al.

Original: (2019a).
Revised: (2019a).

Original: Here, we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to 1 as the depth tends to inﬁnity lim r →∞ C r ( u, u (cid:48) )
Revised: Here, we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to 1 as the depth tends to inﬁnity lim r →∞ C r ( u, u (cid:48) )

Original: = 1 .
Revised: = 1 .

Original: For the function f ( x ) , a Taylor expansion near 1 yields,
Revised: For the function f ( x ) , a Taylor expansion near 1 yields,

Original: [Equation]
Revised: [Equation]

Original: where f (cid:48) (1)
Revised: where f (cid:48) (1)

Original: =
Revised: =

Original: σ 2 w (cid:82) D z [ φ (cid:48) ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =
Revised: σ 2 w (cid:82) D z [ φ (cid:48) ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =

Original: σ 2 w q (cid:82) D
Revised: σ 2 w q (cid:82) D

Original: z
Revised: z

Original: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .
Revised: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .

Original: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,
Revised: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = 2 (cid:82) D z [ φ (cid:48)
Revised: = 2 (cid:82) D z [ φ (cid:48)

Original: [Equation]
Revised: [Equation]

Original: 2 ] q (cid:82)
Revised: 2 ] q (cid:82)

Original: [Equation]
Revised: [Equation]

Original: ( √ q ∗ z ) 2 ] .
Revised: ( √ q ∗ z ) 2 ] .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Thus, we have,
Revised: Thus, we have,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 shows that the covariance matrix converges to a constant matrix at a polynomial rate of 1 /r 2 for ReLU and of 1 /r for tanh activation on the edge of chaos.
Revised: Lemma 2 shows that the covariance matrix converges to a constant matrix at a polynomial rate of 1 /r 2 for ReLU and of 1 /r for tanh activation on the edge of chaos.

Original: This implies that a network without aggregation could retain its expressivity at a large depth.
Revised: This implies that a network without aggregation could retain its expressivity at a large depth.

Original: However, for general cases, due to aggregation, the rate of convergence would change from polynomial to exponential.
Revised: However, for general cases, due to aggregation, the rate of convergence would change from polynomial to exponential.

Original: This would cause the trainability of deep GNNs to be problematic, as shown in the following section.
Revised: This would cause the trainability of deep GNNs to be problematic, as shown in the following section.

Original: B. C ONVERGENCE OF GPK FOR GCN S
Revised: B. C ONVERGENCE OF GPK FOR GCN S

Original: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:
Revised: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:

Original: Lemma 3.
Revised: Lemma 3.

Original: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that
Revised: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove the result via an induction method.
Revised: We prove the result via an induction method.

Original: For l = 0 , according to the CauchyBuniakowsky-Schwarz Inequality
Revised: For l = 0 , according to the CauchyBuniakowsky-Schwarz Inequality

Original: [Equation]
Revised: [Equation]

Original: Thus, there is a constant C , depending on G ( V, E ) , and the number of MLP operations R , over feature initialization, such that,
Revised: Thus, there is a constant C , depending on G ( V, E ) , and the number of MLP operations R , over feature initialization, such that,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,
Revised: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where C 0 is a constant satisfying 0 < C 0 < C .
Revised: where C 0 is a constant satisfying 0 < C 0 < C .

Original: Now we consider the distance between Σ ( l +1)
Revised: Now we consider the distance between Σ ( l +1)

Original: ( r ) ( u, u (cid:48) ) and Cα l +1 .
Revised: ( r ) ( u, u (cid:48) ) and Cα l +1 .

Original: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.
Revised: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.

Original: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .
Revised: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .

Original: It is not hard to ﬁnd that steps 1 and 3 correspond to non-linear transformation while step 2 corresponds to aggregation operation, we then characterize them one by one,
Revised: It is not hard to ﬁnd that steps 1 and 3 correspond to non-linear transformation while step 2 corresponds to aggregation operation, we then characterize them one by one,

Original: MLP Propagation
Revised: MLP Propagation

Original: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:126)v (cid:12)(cid:12) ≤ C 

Revised: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:126)v (cid:12)(cid:12) ≤ C

Original: α l implicitly implies that
Revised: α l implicitly implies that

Original: C r ( u, u (cid:48) ) is close to one.
Revised: C r ( u, u (cid:48) ) is close to one.

Original: Because C r ( u, u (cid:48) )
Revised: Because C r ( u, u (cid:48) )

Original: =
Revised: =

Original: Σ
Revised: Σ

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: ,u
Revised: ,u

Original: [Equation]
Revised: [Equation]

Original: ( l )
Revised: ( l )

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: (cid:48) ) , then,
Revised: (cid:48) ) , then,

Original: [Equation]
Revised: [Equation]

Original: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )
Revised: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )

Original: =
Revised: =

Original: 1+ O ( α l ) .
Revised: 1+ O ( α l ) .

Original: Recall the property of MLP propagation function f ( x ) for x =
Revised: Recall the property of MLP propagation function f ( x ) for x =

Original: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:
Revised: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:

Original: [Equation]
Revised: [Equation]

Original: This implies,
Revised: This implies,

Original: [Equation]
Revised: [Equation]

Original: With this result, we can further obtain,
Revised: With this result, we can further obtain,

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a positive constant.
Revised: where C 1 is a positive constant.

Original: Repeat the proof process
Revised: Repeat the proof process

Original: [Equation]
Revised: [Equation]

Original: we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1,
Revised: we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1,

Original: [Equation]
Revised: [Equation]

Original: Aggregation Propagation We go through an aggregation operation
Revised: Aggregation Propagation We go through an aggregation operation

Original: A ( G ) .
Revised: A ( G ) .

Original: In this case, we use a matrix form, and take equation (20) into aggregation function, yielding the result as follows,
Revised: In this case, we use a matrix form, and take equation (20) into aggregation function, yielding the result as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .
Revised: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .

Original: According to Theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .
Revised: According to Theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .

Original: When l is ﬁnite, the error is of exponential decay.
Revised: When l is ﬁnite, the error is of exponential decay.

Original: Here, we use 0 < α < 1 to denote the corresponding base number.
Revised: Here, we use 0 < α < 1 to denote the corresponding base number.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Finally, by repeating the result in step 1, we have,
Revised: Finally, by repeating the result in step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.
Revised: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.

Original: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) , which has a smaller order compared to O ( α l ) .
Revised: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) , which has a smaller order compared to O ( α l ) .

Original: B. C ONVERGENCE OF GNTK
Revised: B. C ONVERGENCE OF GNTK

Original: Finally, we arrive at our main theorem:
Revised: Finally, we arrive at our main theorem:

Original: Theorem 1 (Convergence rate of GNTK) .
Revised: Theorem 1 (Convergence rate of GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G )
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G )

Original: =
Revised: =

Original: A ( G )
Revised: A ( G )

Original: (cid:126) Θ ( l − 1) ( R ) ( G ) and
Revised: (cid:126) Θ ( l − 1) ( R ) ( G ) and

Original: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then there exist constants 0 < α < 1 and
Revised: Then there exist constants 0 < α < 1 and

Original: C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12)
Revised: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: This proof has the same strategy to that of Lemma 3.
Revised: This proof has the same strategy to that of Lemma 3.

Original: The ﬁrst step is to understand Equation (9) in the large-depth limit.
Revised: The ﬁrst step is to understand Equation (9) in the large-depth limit.

Original: [Equation]
Revised: [Equation]

Original: According to the result of Lemma 3, we have already known,
Revised: According to the result of Lemma 3, we have already known,

Original: [Equation]
Revised: [Equation]

Original: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.
Revised: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.

Original: ReLU activation, φ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: Recall that we deﬁne C r
Revised: Recall that we deﬁne C r

Original: +

Revised: +

Original: =
Revised: =

Original: f
Revised: f

Original: ( C r ) , and we have,
Revised: ( C r ) , and we have,

Original: [Equation]
Revised: [Equation]

Original: Then, at the edge of chaos,
Revised: Then, at the edge of chaos,

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: arcsin
Revised: arcsin

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Tanh activation, φ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: At the edge of chaos,
Revised: At the edge of chaos,

Original: [Equation]
Revised: [Equation]

Original: Now we prove the result via an induction method.
Revised: Now we prove the result via an induction method.

Original: For l = 0 , we directly have,
Revised: For l = 0 , we directly have,

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operations R , over feature initialization,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operations R , over feature initialization,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,
Revised: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: Now we consider the distance between Θ ( l +1)
Revised: Now we consider the distance between Θ ( l +1)

Original: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R
Revised: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R

Original: [Equation]
Revised: [Equation]

Original: l +1) (cid:126)v + (cid:126)v (cid:48) ) .
Revised: l +1) (cid:126)v + (cid:126)v (cid:48) ) .

Original: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.
Revised: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.

Original: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.
Revised: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.

Original: Θ ( l +1) (0)
Revised: Θ ( l +1) (0)

Original: [Equation]
Revised: [Equation]

Original: For step 1, we have,
Revised: For step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the process
Revised: Repeat the process

Original: [Equation]
Revised: [Equation]

Original: we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,
Revised: we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,

Original: [Equation]
Revised: [Equation]

Original: Secondly, we go through an aggregation operation.
Revised: Secondly, we go through an aggregation operation.

Original: Because it is a Markov chain step
Revised: Because it is a Markov chain step

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the result in step 1, we ﬁnally have,
Revised: Repeat the result in step 1, we ﬁnally have,

Original: [Equation]
Revised: [Equation]

Original: C T RAINABILITY OF U LTRA -W IDE GCN S
Revised: C T RAINABILITY OF U LTRA -W IDE GCN S

Original: Corollary 1 (Trainability of ultra-wide GCNs) .
Revised: Corollary 1 (Trainability of ultra-wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.
Revised: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.

Original: Then, the output function follows, f t ( X ) =
Revised: Then, the output function follows, f t ( X ) =

Original: e − η Θ ( l ) ( r ) ( G ) t f 0 ( X )
Revised: e − η Θ ( l ) ( r ) ( G ) t f 0 ( X )

Original: +( I − e − η Θ ( l ) ( r ) ( G ) t Y ) , where X and Y are node features and labels from training set.
Revised: +( I − e − η Θ ( l ) ( r ) ( G ) t Y ) , where X and Y are node features and labels from training set.

Original: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .
Revised: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .

Original: Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( X ) − Y
Revised: Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( X ) − Y

Original: (cid:107) > C .
Revised: (cid:107) > C .

Original: Proof.
Revised: Proof.

Original: According to the result from Jacot et al.
Revised: According to the result from Jacot et al.

Original: (2018), GNTK Θ ( l ) ( r ) ( G ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite-width limit.
Revised: (2018), GNTK Θ ( l ) ( r ) ( G ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite-width limit.

Original: We omit the proof procedure for this result, since it is now a standard conclusion in the NTK study.
Revised: We omit the proof procedure for this result, since it is now a standard conclusion in the NTK study.

Original: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,
Revised: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,

Original: [Equation]
Revised: [Equation]

Original: where ω t = θ t − θ 0 .
Revised: where ω t = θ t − θ 0 .

Original: We call it a linearized model because only zero and ﬁrst order term of Taylor expansion are kept.
Revised: We call it a linearized model because only zero and ﬁrst order term of Taylor expansion are kept.

Original: Since we know the dynamics of gradient ﬂow using this linearized function are governed by,
Revised: Since we know the dynamics of gradient ﬂow using this linearized function are governed by,

Original: [Equation]
Revised: [Equation]

Original: where L is an MSE loss, then the above equations have closed form solutions
Revised: where L is an MSE loss, then the above equations have closed form solutions

Original: [Equation]
Revised: [Equation]

Original: Since Lee et al.
Revised: Since Lee et al.

Original: Lee et al. (2019a) showed that f lin t ( X ) =
Revised: Lee et al. (2019a) showed that f lin t ( X ) =

Original: f t ( X ) in the inﬁnite width limit, thus we have,
Revised: f t ( X ) in the inﬁnite width limit, thus we have,

Original: [Equation]
Revised: [Equation]

Original: In Theorem 1, we have shown Θ ( l ) ( r ) ( G ) converges to a constant matrix at an exponential rate, thus being singular in the large depth limit.
Revised: In Theorem 1, we have shown Θ ( l ) ( r ) ( G ) converges to a constant matrix at an exponential rate, thus being singular in the large depth limit.

Original: According to Equation (21), we know that,
Revised: According to Equation (21), we know that,

Original: [Equation]
Revised: [Equation]

Original: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( G ) =
Revised: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( G ) =

Original: Q T DQ be the decomposition of the
Revised: Q T DQ be the decomposition of the

Original: GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.
Revised: GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.

Original: Because Θ ( ∞ ) ( r ) ( G ) is singular,
Revised: Because Θ ( ∞ ) ( r ) ( G ) is singular,

Original: D has at least one zero value d j = 0 , then
Revised: D has at least one zero value d j = 0 , then

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: In the proof we assume the loss is MSE.
Revised: In the proof we assume the loss is MSE.

Original: Nevertheless, the conclusion regarding trainability can be applied to other common losses such as cross-entropy.
Revised: Nevertheless, the conclusion regarding trainability can be applied to other common losses such as cross-entropy.

Original: For cross-entropy loss, even though we cannot derive an analytical expression for the solution, we can prove that the GNTK still governs the trainability and the law of GNTK is not affected by the loss.
Revised: For cross-entropy loss, even though we cannot derive an analytical expression for the solution, we can prove that the GNTK still governs the trainability and the law of GNTK is not affected by the loss.

Original: Thus, the results for trainability still hold in the cross-entropy case.
Revised: Thus, the results for trainability still hold in the cross-entropy case.

Original: D C ONVERGENCE OF R ESIDUAL GNTK D. R ESIDUAL CONNECTION IN AGGREGATION
Revised: D C ONVERGENCE OF R ESIDUAL GNTK D. R ESIDUAL CONNECTION IN AGGREGATION

Original: Theorem 2 (Convergence rate of residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate of residual connection in aggregation) .

Original: Consider a GNTK of nonlinear transformation (9) and residual connection (12).
Revised: Consider a GNTK of nonlinear transformation (9) and residual connection (12).

Original: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <
Revised: A ( G ) , there exist constants 0 <

Original: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on
Revised: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: Proof.
Revised: Proof.

Original: According to the aggregation function for covariance matrix, we have
Revised: According to the aggregation function for covariance matrix, we have

Original: [Equation]
Revised: [Equation]

Original: The new aggregation matrix ˜
Revised: The new aggregation matrix ˜

Original: A ( G ) is a stochastic transition matrix as well, which can be seen from,
Revised: A ( G ) is a stochastic transition matrix as well, which can be seen from,

Original: [Equation]
Revised: [Equation]

Original: Then we compare the second largest eigenvalue between two transition matrices.
Revised: Then we compare the second largest eigenvalue between two transition matrices.

Original: Suppose the eigenvalues of the original matrix
Revised: Suppose the eigenvalues of the original matrix

Original: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .
Revised: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .

Original: We already know that the maximum eigenvalue is λ 

Revised: We already know that the maximum eigenvalue is λ

Original: = 1 ,
Revised: = 1 ,

Original: and the converge rate is governed by the second largest eigenvalue λ 2 .
Revised: and the converge rate is governed by the second largest eigenvalue λ 2 .

Original: Now we consider the second largest eigenvalue ˜ λ
Revised: Now we consider the second largest eigenvalue ˜ λ

Original: 2 of matrix ˜
Revised: 2 of matrix ˜

Original: A :
Revised: A :

Original: [Equation]
Revised: [Equation]

Original: Remark 4.
Revised: Remark 4.

Original: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 .
Revised: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 .

Original: Here, we discuss more about the relationship between α and ˜ α .
Revised: Here, we discuss more about the relationship between α and ˜ α .

Original: According to (Rosenthal, 1995), the relationship between convergence rate α and the second largest eigenvalue λ 2 of transition matrix
Revised: According to (Rosenthal, 1995), the relationship between convergence rate α and the second largest eigenvalue λ 2 of transition matrix

Original: A ( G ) can be expressed as,
Revised: A ( G ) can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .
Revised: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .

Original: From the above inequality, we can conclude that the second largest eigenvalue almost governs the convergence rate.
Revised: From the above inequality, we can conclude that the second largest eigenvalue almost governs the convergence rate.

Original: However, to maintain rigor, we cannot directly obtain α < ˜ α .
Revised: However, to maintain rigor, we cannot directly obtain α < ˜ α .

Original: Instead, from the result that ˜ λ 2 > λ 2 , we say that ˜ α is roughly larger than α .
Revised: Instead, from the result that ˜ λ 2 > λ 2 , we say that ˜ α is roughly larger than α .

Original: D. R ESIDUAL CONNECTION IN MLP
Revised: D. R ESIDUAL CONNECTION IN MLP

Original: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .

Original: Consider a GNTK of the form (8) and (12).
Revised: Consider a GNTK of the form (8) and (12).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v (cid:48) depending on R , such that, | Θ ( l
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v (cid:48) depending on R , such that, | Θ ( l

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation]
Revised: [Equation]

Original: u, u
Revised: u, u

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For residual connection in MLP, we have,
Revised: For residual connection in MLP, we have,

Original: [Equation]
Revised: [Equation]

Original: Since σ 2 w > 0 , q r ( u ) grows at an exponential rate.
Revised: Since σ 2 w > 0 , q r ( u ) grows at an exponential rate.

Original: Now, we turn to compute the correlation term C r ( u, u (cid:48) ) .
Revised: Now, we turn to compute the correlation term C r ( u, u (cid:48) ) .

Original: For convenience, we suppose q r ( u ) =
Revised: For convenience, we suppose q r ( u ) =

Original: q r ( u (cid:48) ) .
Revised: q r ( u (cid:48) ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Using Taylor expansion of f near 1, as have been done in the proof of Lemma 2,
Revised: Using Taylor expansion of f near 1, as have been done in the proof of Lemma 2,

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Note that it is similar to the case of MLP without residual connection:
Revised: Note that it is similar to the case of MLP without residual connection:

Original: [Equation]
Revised: [Equation]

Original: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,
Revised: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,

Original: [Equation]
Revised: [Equation]

Original: E APPENDIX: D ISCUSSION ON THE TRAINABILITY OF GNTK WITH C RTICAL D ROP E DGE
Revised: E APPENDIX: D ISCUSSION ON THE TRAINABILITY OF GNTK WITH C RTICAL D ROP E DGE

Original: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd˝os & R´enyi, 1961; Erd¨os & R´enyi, 2011), where n is the number of nodes in a graph.
Revised: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd˝os & R´enyi, 1961; Erd¨os & R´enyi, 2011), where n is the number of nodes in a graph.

Original: This implies that there exists a (connected) graph of size O ( n ) when we use the critical dropping rate.
Revised: This implies that there exists a (connected) graph of size O ( n ) when we use the critical dropping rate.

Original: To preserve information, edges would be resampled at every iteration.
Revised: To preserve information, edges would be resampled at every iteration.

Original: This implies that at each iteration during training, the GNN takes a random and large graph.
Revised: This implies that at each iteration during training, the GNN takes a random and large graph.

Original: For the whole training process, we can think that we have used all the information conveyed by the graph.
Revised: For the whole training process, we can think that we have used all the information conveyed by the graph.

Original: It would be interesting to consider deriving some theoretical results for the limiting behavior of the
Revised: It would be interesting to consider deriving some theoretical results for the limiting behavior of the

Original: GNTK in the large depth with Critical DropEdge.
Revised: GNTK in the large depth with Critical DropEdge.

Original: Since Critical DropEdge is originated from the random graph theory, a promising approach is to analyze spectral distributions of adjacency matrix and Laplacian matrix with a random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).
Revised: Since Critical DropEdge is originated from the random graph theory, a promising approach is to analyze spectral distributions of adjacency matrix and Laplacian matrix with a random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).

Original: F APPENDIX: D ATASETS AND I MPLEMENTATION D ETAILS F. D ATASETS
Revised: F APPENDIX: D ATASETS AND I MPLEMENTATION D ETAILS F. D ATASETS

Original: For node classiﬁcation tasks, Cora, Citeseer and Pubmed (Kipf & Welling, 2017)) are three commonly used citation networks, and Physics (Shchur et al., 2018) is a co-author network.
Revised: For node classiﬁcation tasks, Cora, Citeseer and Pubmed (Kipf & Welling, 2017)) are three commonly used citation networks, and Physics (Shchur et al., 2018) is a co-author network.

Original: Detailed information of the four benchmark datasets is listed as follows and summarized in Table 3.
Revised: Detailed information of the four benchmark datasets is listed as follows and summarized in Table 3.

Original: • The Cora dataset consists of 2,708 scientiﬁc publications classiﬁed into one of seven classes, and 5,429 links.
Revised: • The Cora dataset consists of 2,708 scientiﬁc publications classiﬁed into one of seven classes, and 5,429 links.

Original: Each publication is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 1,433 unique words.
Revised: The dictionary consists of 1,433 unique words.

Original: • The Citeseer dataset consists of 3,312 scientiﬁc publications classiﬁed into one of six classes, and 4,732 links.
Revised: • The Citeseer dataset consists of 3,312 scientiﬁc publications classiﬁed into one of six classes, and 4,732 links.

Original: Each publication is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 3,703 unique words.
Revised: The dictionary consists of 3,703 unique words.

Original: • The Pubmed Diabetes dataset consists of 19,717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.
Revised: • The Pubmed Diabetes dataset consists of 19,717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.

Original: The citation network consists of 44,338 links.
Revised: The citation network consists of 44,338 links.

Original: Each publication is described by a TF-IDF weighted word vector from a dictionary comprised of 500 unique words.
Revised: Each publication is described by a TF-IDF weighted word vector from a dictionary comprised of 500 unique words.

Original: • The Physics dataset consists of 34,493 authors as nodes, and edges indicate whether two authors have co-authored a paper.
Revised: • The Physics dataset consists of 34,493 authors as nodes, and edges indicate whether two authors have co-authored a paper.

Original: Node features are paper keywords from the author’s papers.
Revised: Node features are paper keywords from the author’s papers.

Original: Following Kim & Oh (2020), we reduce the original dimension (6,805 and 8,415) to 500 using PCA.
Revised: Following Kim & Oh (2020), we reduce the original dimension (6,805 and 8,415) to 500 using PCA.

Original: The split is the 20-per-class/30-per-
Revised: The split is the 20-per-class/30-per-

Original: class/rest.
Revised: class/rest.

Original: The goal of this task is to classify each author’s respective ﬁeld of study.
Revised: The goal of this task is to classify each author’s respective ﬁeld of study.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F. E XPERIMENTAL I MPLEMENTATION
Revised: E XPERIMENTAL I MPLEMENTATION

Original: We use the PyTorch implementation to simulate both inﬁnitely-wide and ﬁnitely-wide GCNs:
Revised: We use the PyTorch implementation to simulate both inﬁnitely-wide and ﬁnitely-wide GCNs:

Original: • The inﬁnitely-wide GCNs use part of code from Du et al.
Revised: • The inﬁnitely-wide GCNs use part of code from Du et al.

Original: (2019b), which is originally adopted for graph classiﬁcation.
Revised: (2019b), which is originally adopted for graph classiﬁcation.

Original: We redesign the calculation method of GNTK (Du et al., 2019b) 2 according to the formula in Section 3.1 and use it to process node classiﬁcation tasks.
Revised: We redesign the calculation method of GNTK (Du et al., 2019b) 2 according to the formula in Section 3.1 and use it to process node classiﬁcation tasks.

Original: • For ﬁnitely-wide GNNs with DropEdge (Rong et al., 2019) 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.
Revised: • For ﬁnitely-wide GNNs with DropEdge (Rong et al., 2019) 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.

Original: The difference is that, in each experiment, we ﬁx the edge sampling percentage as ρ = | V | 2 | E | , which listed in the last column of Table 3.
Revised: The difference is that, in each experiment, we ﬁx the edge sampling percentage as ρ = | V | 2 | E | , which listed in the last column of Table 3.

Original: All codes mentioned above use the MIT license.
Revised: All codes mentioned above use the MIT license.

Original: All experiments are conducted on two Nvidia
Revised: All experiments are conducted on two Nvidia

Original: Quadro RTX 6000 GPUs.
Revised: Quadro RTX 6000 GPUs.

Original: G APPENDIX: A DDITIONAL E XPERIMENTS AND D ISCUSSION G. I NFINITELY -W IDE B ACKBONES
Revised: G APPENDIX: A DDITIONAL E XPERIMENTS AND D ISCUSSION G. I NFINITELY -W IDE B ACKBONES

Original: In the main paper, we reported the performance of GCNs with ﬁnite-width, here we display more results with respect to inﬁnitely-wide GCNs, as shown in Table 4.
Revised: In the main paper, we reported the performance of GCNs with ﬁnite-width, here we show more results with respect to inﬁnitely-wide GCNs, as shown in Table 4.

Original: We apply Critical DropEdge to inﬁnitely-wide backbones on semi-supervised node classiﬁcation.
Revised: We apply Critical DropEdge to inﬁnitely-wide backbones on semi-supervised node classiﬁcation.

Original: We consider two backbones, which are GCN (Kipf & Welling, 2017) and
Revised: We consider two backbones, which are GCN (Kipf & Welling, 2017) and

Original: JKNet (Xu et al., 2018), and the edge preserving percentageis ρ ( G ) which shown in Table 3.
Revised: JKNet (Xu et al., 2018), and the edge preserving percentage ρ ( G ) is shown in Table 3.

Original: First we calculate a GNTK with respect to the graph data and GNN.
Revised: First, we calculate a GNTK with respect to the graph data and GNN.

Original: Then we apply kernel regression with calculated GNTK to complete node classiﬁcation task.
Revised: Then, we apply kernel regression with calculated GNTK to complete node classiﬁcation task.

Original: Table 4 summaries the performance on three citation networks.
Revised: Table 4 summaries the performance on three citation networks.

Original: In this table, we report the performance of GCNs with 4/8/32/64 layers.
Revised: In this table, we report the performance of GCNs with 4/8/32/64 layers.

Original: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original GNNs in the inﬁnitely-wide case, especially when the model is deep.
Revised: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original GNNs in the inﬁnitely-wide case, especially when the model is deep.

Original: [Table]
Revised: [Table]

Original: G. P ERFORMANCE OF U SING D IFFERENT B ACKBONES
Revised: G. P ERFORMANCE OF U SING D IFFERENT B ACKBONES

Original: The results presented in Table 1 are the best across different backbones.
Revised: The results presented in Table 1 are the best across different backbones.

Original: Table 5 further compares the results of vanilla GNN, DropEdge and C-DropEdge for three different backbones (GCN, JKNet and IncepGCN).
Revised: Table 5 further compares the results of vanilla GNN, DropEdge and C-DropEdge for three different backbones (GCN, JKNet and IncepGCN).

Original: From the table we can conclude that Critical DropEdge consistently outperforms
Revised: From the table we can conclude that Critical DropEdge consistently outperforms

Original: DropEdge and vanilla GNNs.
Revised: DropEdge and vanilla GNNs.

Original: G. T RAINABILITY OF GNTK WITH C RITICAL D ROP E DGE
Revised: G. T RAINABILITY OF GNTK WITH C RITICAL D ROP E DGE

Original: We implement deep GCNs with/without DropEdge and C-DropEdge to compare the training performance.
Revised: We implement deep GCNs with/without DropEdge and C-DropEdge to compare the training performance.

Original: The results are shown in Figure 4.
Revised: The results are shown in Figure 4.

Original: The training loss of GCN and GCN with DropEdge has a slower convergence rate and converges to a larger error compared to GCN with C-DropEdge.
Revised: The training loss of GCN and GCN with DropEdge has a slower convergence rate and converges to a larger error compared to GCN with C-DropEdge.

Original: This indicates that C-DropEdge can achieve better trainability compared to GCN and GCN with
Revised: This indicates that C-DropEdge can achieve better trainability compared to GCN and GCN with

Original: DropEdge.
Revised: DropEdge.

Original: Besides, the convergence results of GNTKs with C-DropEdge are shown in Figure 5.
Revised: Besides, the convergence results of GNTKs with C-DropEdge are shown in Figure 5.

Original: Compared to residual connection, we ﬁnd that the normalized GNTK elements would not converge to a single value
Revised: Compared to residual connection, we ﬁnd that the normalized GNTK elements would not converge to a single value

Original: [Table]
Revised: [Table]

Original: and the convergence rate curve does not depend on the depth.
Revised: and the convergence rate curve does not depend on the depth.

Original: This implies that residual connection mitigates the trainability loss by slowing down the exponential convergence rate, while C-DropEdge directly changes the convergence results.
Revised: This implies that residual connection mitigates the trainability loss by slowing down the exponential convergence rate, while C-DropEdge directly changes the convergence results.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]