Original: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE
Revised: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.
Revised: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data

Original: Nevertheless, it is well known that deep GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.
Revised: Nevertheless , it is well known that deep GCNs suffer from the over -smoothing problem where node representations tend to be indistinguishable as more layers are stacked up.

Original: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.
Revised: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.

Original: Compared to expressivity, trainability attempts to address a more fundamental question:
Revised: Compared to expressivity, trainability attempts to address a more fundamental question:

Original: Given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution via gradient descent-based optimizers?
Revised: Given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution via gradient descent-based optimizers?

Original: This work ﬁlls this gap by exploiting the Graph Neural
Revised: This work ﬁlls this gap by exploiting the Graph Neural

Original: Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.
Revised: Tangent Kernel (GNTK) which governs the optimization trajectory under gradient descent for wide GCNs.

Original: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.
Revised: We formulate the asymptotic behaviors of GNTK in the large depth which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.

Original: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be merely able to mitigate the exponential decay of trainability mildly.
Revised: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be merely able to mitigate the exponential decay of trainability mildly.

Original: Inspired by our theoretical insights on trainability, we propose
Revised: Inspired by our theoretical insights on trainability, we propose

Original: Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, to alleviate the exponential decay problem more fundamentally.
Revised: Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, to alleviate the exponential decay problem more fundamentally.

Original: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.
Revised: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, Graph Neural Networks (GNNs) have shown incredible abilities to learn node or graph representations and achieved superior performance on various downstream tasks, such as node classiﬁcation (Kipf & Welling, 2016a; Veliˇckovi´c et al., 2017; Hamilton et al., 2017), graph classiﬁcation (Xu et al., 2018a; Gilmer et al., 2017; Lee et al., 2019b; Yuan & Ji, 2020), and link prediction (Kipf & Welling, 2016b), etc.
Revised: Recently, Graph Neural Networks (GNNs) have shown incredible abilities to learn node or graph representations and achieved superior performance on various downstream tasks, such as node classiﬁcation (Kipf & Welling, 2016a; Veliˇckovi´c et al., 2017; Hamilton et al., 2017), graph classiﬁcation (Xu et al., 2018a; Gilmer et al., 2017; Lee et al., 2019b; Yuan & Ji, 2020), and link prediction (Kipf & Welling, 2016b), etc.

Original: However, most GNNs (e.g., GCNs) achieve their best only with a shallow depth, e.g., 2 or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.
Revised: However, most GNNs (e.g., GCNs) achieve their best only with a shallow depth, e.g., 2 or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.

Original: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.
Revised: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.

Original: Li et al.
Revised: Li et al.

Original: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.
Revised: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.

Original: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.
Revised: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.

Original: Oono & Suzuki (2019) investigated the expressive power of GNNs using the asymptotic behaviors as the ∗ Equal Contribution.
Revised: Oono & Suzuki (2019) investigated the expressive power of GNNs using the asymptotic behaviors as the layer goes to inﬁnity.

Original: layer goes to inﬁnity.

Original: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graph inherent in the graph spectra.
Revised: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graph inherent in the graph spectra.

Original: Nevertheless, it remains elusive how to theoretically understand why deep GCNs fail to optimize.
Revised: Nevertheless, it remains elusive how to theoretically understand why deep GCNs fail to optimize.

Original: Existing theoretical investigation (Xu et al., 2018a; Oono & Suzuki, 2019) on GNNs focus mainly on expressivity, which measures the complexity of functions that can be represented by a neural network.
Revised: Existing theoretical investigation (Xu et al., 2018a; Oono & Suzuki, 2019) on GNNs focus mainly on expressivity, which measures the complexity of functions that can be represented by a neural network.

Original: Exploring expressivity is theoretically convenient, but the corresponding conditions may be violated during the gradient training process, thereby leading to inconsistencies between theoretical conclusions and empirical results of trained networks (G¨uhring et al., 2020).
Revised: Exploring expressivity is theoretically convenient, but the corresponding conditions may be violated during the gradient training process, thereby leading to inconsistencies between theoretical conclusions and empirical results of trained networks (G¨uhring et al., 2020).

Original: Compared to expressivity, trainability addresses a more difﬁcult but fundamental perspective of neural networks:
Revised: Compared to expressivity, trainability addresses a more difﬁcult but fundamental perspective of neural networks:

Original: How effectively a neural network can be optimized via gradient descent.
Revised: How effectively a neural network can be optimized via gradient descent.

Original: The advantage of investigating trainability is that we can directly determine whether GNNs can be successfully trained under certain conditions, and to what extent.
Revised: The advantage of investigating trainability is that we can directly determine whether GNNs can be successfully trained under certain conditions, and to what extent.

Original: We are therefore inspired to raise two important questions:
Revised: We are therefore inspired to raise two important questions:

Original: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?
Revised: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?

Original: • Can we further design an algorithm to facilitate deeper GCNs, beneﬁting from our theoretical investigation?
Revised: • Can we further design an algorithm to facilitate deeper GCNs, beneﬁting from our theoretical investigation?

Original: Our answers are yes to both questions.
Revised: Our answers are yes to both questions.

Original: We resort to the inﬁnitely-wide multi-layer GCN to derive our solution.
Revised: We resort to the inﬁnitely-wide multi-layer GCN to derive our solution.

Original: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed that single hidden layer networks with random weights at initialization (without training) are Gaussian Processes (GPs) in the inﬁnite width limit.
Revised: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed that single hidden layer networks with random weights at initialization (without training) are Gaussian Processes (GPs) in the inﬁnite width limit.

Original: Later, the connection between GPs and multi-layer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2017;
Revised: Later, the connection between GPs and multi-layer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2017;

Original: Matthews et al., 2018) and orthogonal weights (Huang et al., 2020a) was reported.
Revised: Matthews et al., 2018) and orthogonal weights (Huang et al., 2020a) was reported.

Original: Recent trends in
Revised: Recent trends in

Original: Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.
Revised: Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.

Original: In particular, Jacot et al.
Revised: In particular, Jacot et al.

Original: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an NTK.
Revised: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an NTK.

Original: Du et al.
Revised: Du et al.

Original: (2019) formulated Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNNs.
Revised: (2019) formulated Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNNs.

Original: Prior to the discovery of GNTK, there was little understanding of the non-convexity of GNNs, which is analytically intractable.
Revised: Prior to the discovery of GNTK, there was little understanding of the non-convexity of GNNs, which is analytically intractable.

Original: In the learning regime of
Revised: In the learning regime of

Original: GCN governed by GNTK, the optimization becomes an almost convex problem, making GNTK a promising perspective to study the trainability of deep GCNs.
Revised: GCN governed by GNTK, the optimization becomes an almost convex problem, making GNTK a promising perspective to study the trainability of deep GCNs.

Original: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.
Revised: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.

Original: In particular, we formulate the large-depth asymptotic behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b;
Revised: In particular, we formulate the large-depth asymptotic behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b;

Original: Xiao et al., 2019), through which we can analyze the optimization properties.
Revised: Xiao et al., 2019), through which we can analyze the optimization properties.

Original: Speciﬁcally, we make the following contributions
Revised: Speciﬁcally, we make the following contributions

Original: : • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.
Revised: : • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.

Original: We prove that all entries of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.
Revised: We prove that all entries of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.

Original: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.
Revised: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.

Original: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.
Revised: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.

Original: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate of trainability, but lack the ability to fundamentally solve the problem.
Revised: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate of trainability, but lack the ability to fundamentally solve the problem.

Original: This result enables to better understand why and to what extent recent residual connection-based methods work.
Revised: This result enables to better understand why and to what extent recent residual connection-based methods work.

Original: • Our theoretical framework provides insights to guide the development of deep GCNs.
Revised: • Our theoretical framework provides insights to guide the development of deep GCNs.

Original: We further propose an edge-based sampling method, named Critical DropEdge, to effectively mitigate the exponential decay of trainability.
Revised: We further propose an edge-based sampling method, named Critical DropEdge, to effectively mitigate the exponential decay of trainability.

Original: This graph-adaptive and connectivity-aware method is easy to implement in both ﬁnite-width and inﬁnite-width GNNs.
Revised: This graph-adaptive and connectivity-aware method is easy to implement in both ﬁnite-width and inﬁnite-width GNNs.

Original: Our experiments show using the proposed method can outperform competitors in the large depth.
Revised: Our experiments show using the proposed method can outperform competitors in the large depth.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.
Revised: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.

Original: We then review NTK, making a connection to trainability.
Revised: We then review NTK, making a connection to trainability.

Original: Finally, we introduce GCNs along with our setup and notation.
Revised: Finally, we introduce GCNs along with our setup and notation.

Original: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION
Revised: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION

Original: We begin by considering a fully-connected network of depth L with width m l in each layer.
Revised: We begin by considering a fully-connected network of depth L with width m l in each layer.

Original: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .
Revised: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .

Original: Letting the pre-activations be given by h ( l ) i , information propagation in this network is governed by,
Revised: Letting the pre-activations be given by h ( l ) i , information propagation in this network is governed by,

Original: [Equation]
Revised: [Equation]

Original: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases, respectively.
Revised: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases, respectively.

Original: Given the parameterization that weights and biases are randomly generated by i.i.d.
Revised: Given the parameterization that weights and biases are randomly generated by i.i.d.

Original: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , . . .
Revised: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , . . .

Original: , m l − 1 → ∞ .
Revised: , m l − 1 → ∞ .

Original: This results from the central limit theorem (CLT).
Revised: This results from the central limit theorem (CLT).

Original: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )
Revised: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .
Revised: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .

Original: According to the signal propagation (1), the covariance matrix or GPK with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )
Revised: According to the signal propagation (1), the covariance matrix or GPK with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: σ 2 w E h ∼N (0 , Σ ( l − 1) )
Revised: σ 2 w E h ∼N (0 , Σ ( l − 1) )

Original: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]
Revised: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]

Original: + σ 2 b .
Revised: + σ 2 b .

Original: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2016).
Revised: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2016).

Original: In particular, expressivity describes to what extent two different inputs can be distinguished.
Revised: In particular, expressivity describes to what extent two different inputs can be distinguished.

Original: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )
Revised: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )

Original: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .
Revised: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .

Original: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;
Revised: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;

Original: Schoenholz et al., 2016; Yang & Schoenholz, 2017).
Revised: Schoenholz et al., 2016; Yang & Schoenholz, 2017).

Original: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.
Revised: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.

Original: Given this reason, we focus on the edge of chaos in this work.
Revised: Given this reason, we focus on the edge of chaos in this work.

Original: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z
Revised: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z

Original: [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal entries in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .

Original:  N EURAL T ANGENT K ERNEL AND T RAINABILITY
Revised: 2. N EURAL T ANGENT K ERNEL AND T RAINABILITY

Original: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2016) have focused solely on initialization without training.
Revised: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2016) have focused solely on initialization without training.

Original: Jacot et al.
Revised: Jacot et al.

Original: (2018) took a step further by considering inﬁnitely-wide networks trained with gradient descent.
Revised: (2018) took a step further by considering inﬁnitely-wide networks trained with gradient descent.

Original: Let η be the learning rate, and L be the loss function.
Revised: Let η be the learning rate, and L be the loss function.

Original: The dynamics of gradient ﬂow for parameters
Revised: The dynamics of gradient ﬂow for parameters

Original: [Equation]
Revised: [Equation]

Revised: l

Revised: [Equation]

Original:  , the vector of all parameters, is given by,
Revised: m [Equation] 1 +1)) × 1 , the vector of all parameters, is given by,

Original: [Equation]
Revised: [Equation]

Original: Then, the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X
Revised: Then, the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X

Original: ) ∈ R
Revised: ) ∈ R

Original: nm L × 1 follow,
Revised: nm L × 1 follow,

Original: [Equation]
Revised: [Equation]

Original: where the NTK at time t is deﬁned as,
Revised: where the NTK at time t is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: In a general case, the NTK varies with the training time, thus providing no substantial insights into the convergence property of neural networks.
Revised: In a general case, the NTK varies with the training time, thus providing no substantial insights into the convergence property of neural networks.

Original: Interestingly, as shown by Jacot et al.
Revised: Interestingly, as shown by Jacot et al.

Original: (2018), the NTK converges to an explicit limiting kernel and does not change during training in the inﬁnite width limit.
Revised: (2018), the NTK converges to an explicit limiting kernel and does not change during training in the inﬁnite width limit.

Original: This leads to a simple but profound result in the case of an MSE loss, L = 12 (cid:107) f t ( X ) − Y
Revised: This leads to a simple but profound result in the case of an MSE loss, L = 12 (cid:107) f t ( X ) − Y

Original: (cid:107) 22 , where Y is the label associated with the input X ,
Revised: (cid:107) 22 , where Y is the label associated with the input X ,

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original:  where Θ ∞ is the limiting kernel.
Revised: (5) where Θ ∞ is the limiting kernel.

Original: This is the solution to an ordinary differential equation.
Revised: This is the solution to an ordinary differential equation.

Original: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =
Revised: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =

Original: Y .
Revised: Y .

Original: As proved by  Hayou et al. (2019b) (Lemma 1), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.
Revised: As proved by Lemma 1 in Hayou et al. (2019b), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.

Original: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.
Revised: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.

Original: 2. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: 2. G RAPH C ONVOLUTIONAL N ETWORKS

Original: We deﬁne an undirected graph as G =
Revised: We deﬁne an undirected graph as G =

Original: ( V, E ) , where V is a set of nodes and E is a set of edges.
Revised: ( V, E ) , where V is a set of nodes and E is a set of edges.

Original: We denote the number of nodes in graph G by n = | V | .
Revised: We denote the number of nodes in graph G by n = | V | .

Original: The node features are denoted as h v ∈ R d for each v ∈ V , with d being the dimension of node features.
Revised: The node features are denoted as h v ∈ R d for each v ∈ V , with d being the dimension of node features.

Original: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.
Revised: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.

Original: GCNs iteratively update node features through aggregating and transforming the representations of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representations of their neighbors.

Original: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.

Original: We deﬁne a propagation unit to be the combination of a R -layer multi-layer perceptron (MLP) and one aggregation operation.
Revised: We deﬁne a propagation unit to be the combination of a R -layer multi-layer perceptron (MLP) and one aggregation operation.

Original: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.
Revised: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.

Original: L is the total number of propagation units.
Revised: L is the total number of propagation units.

Original: To be speciﬁc, the node representation propagation in GCNs through an MLP follows the expression,
Revised: To be speciﬁc, the node representation propagation in GCNs through an MLP follows the expression,

Original: [Equation]
Revised: [Equation]

Original: (7) where W ( l ) ( r ) ∈
Revised: [Equation] where W ( l ) ( r ) ∈

Original: R m l × m l − 1 and b ( l ) ( r ) ∈
Revised: R m l × m l − 1 and b ( l ) ( r ) ∈

Original: R m l are the learnable weights and biases, respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node u and its neighbors.
Revised: R m l are the learnable weights and biases, respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node u and its neighbors.

Original: Equation (6) reveals the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).
Revised: Equation (6) reveals the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).

Original: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.
Revised: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.

Original: For the activation function, we focus on both ReLU and Tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =
Revised: For the activation function, we focus on both ReLU and Tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =

Original: tanh( x ) , respectively.
Revised: tanh( x ) , respectively.

Original: Without loss of generality, our theoretical framework can handle other common activation functions, whereas the GNTK work (Du et al., 2019) only adopted ReLU.
Revised: Without loss of generality, our theoretical framework can handle other common activation functions, whereas the GNTK work (Du et al., 2019) only adopted ReLU.

Original:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION
Revised:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION

Original: Based on the deﬁnition of NTK (4), we recursively formulate the propagation of GNTK in the inﬁnite-width limit.
Revised: Based on the deﬁnition of NTK (4), we recursively formulate the propagation of GNTK in the inﬁnite-width limit.

Original: As information propagation in a GCN is built on two operations: aggregation (6) and non-linear transformation (7), the corresponding formulas of GNTK are expressed as follows,
Revised: As information propagation in a GCN is built on two operations: aggregation (6) and non-linear transformation (7), the corresponding formulas of GNTK are expressed as follows,

Original: [Equation]
Revised: [Equation]

Original: The two equations above correspond to the aggregation operation and MLP transformation, respectively.
Revised: The two equations above correspond to the aggregation operation and MLP transformation, respectively.

Original: To compute the GNTK with respect to the depth
Revised: To compute the GNTK with respect to the depth

Original: [Equation]
Revised: [Equation]

Original: the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )
Revised: the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )

Original: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .
Revised: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .

Original: According to the CLT
Revised: According to the CLT

Original: [Equation]
Revised: [Equation]

Original: the node representation h ( l ) ( r ) ( u ) is a Gaussian distributed in the inﬁnite width limit.
Revised: the node representation h ( l ) ( r ) ( u ) is a Gaussian distributed in the inﬁnite width limit.

Original: Applying this result to equations (6) and (7)
Revised: Applying this result to equations (6) and (7)

Original: [Equation]
Revised: [Equation]

Original: the resultant covariance matrix is composed of two parts,
Revised: the resultant covariance matrix is composed of two parts,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: The ﬁrst equation (10) results from the aggregation operation.
Revised: The ﬁrst equation (10) results from the aggregation operation.

Original: Meanwhile, the second equation (11) corresponds to the R -times non-linear transformations, where E z 1 ,z 2 takes the expectation with respect to a centered Gaussian process of covariance ˜Σ ( l ) ( r − 1)
Revised: Meanwhile, the second equation (11) corresponds to the R -times non-linear transformations, where E z 1 ,z 2 takes the expectation with respect to a centered Gaussian process of covariance ˜Σ ( l ) ( r − 1)

Original: ∈ R 2 × 2 for previous MLP layer across u, u (cid:48) , and ˙ φ denotes the derivative of φ .
Revised: ∈ R 2 × 2 for previous MLP layer across u, u (cid:48) , and ˙ φ denotes the derivative of φ .

Original: 3. T RAINABILITY IN THE L ARGE D EPTH
Revised: 3. T RAINABILITY IN THE L ARGE D EPTH

Original: We aim to characterize the behavior of Θ ( l ) ( r ) ( G ) ≡ Θ ( l ) ( r ) ( V , V ) ∈ R n × n , where we denote
Revised: We aim to characterize the behavior of Θ ( l ) ( r ) ( G ) ≡ Θ ( l ) ( r ) ( V , V ) ∈ R n × n , where we denote

Original: V =
Revised: V =

Original: [ v 1 , v 2 , . .
Revised: [ v 1 , v 2 , . .

Original: . , v n ] , as the depth tends to inﬁnity l → ∞ .
Revised: . , v n ] , as the depth tends to inﬁnity l → ∞ .

Original: From the GNTK formulation, both aggregation (8) and transformation (9) contribute simultaneously to the ﬁnal limiting result.
Revised: From the GNTK formulation, both aggregation (8) and transformation (9) contribute simultaneously to the ﬁnal limiting result.

Original: Taking them into consideration, we derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is given as follows:
Revised: We derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is given as follows:

Original: Theorem 1 (Convergence Rate of GNTK) .
Revised: Theorem 1 (Convergence Rate of GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G )
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G )

Original: =
Revised: =

Original: A ( G )
Revised: A ( G )

Original: (cid:126) Θ ( l − 1) ( R ) ( G ) and
Revised: (cid:126) Θ ( l − 1) ( R ) ( G ) and

Original: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then, there exist constants 0 < α < 1 and
Revised: Then, there exist constants 0 < α < 1 and

Original: C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12)
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: The proof sketch follows a divide-and-conquer manner.
Revised: The proof sketch follows a divide-and-conquer manner.

Original: In particular, we ﬁrst analyze the network with only aggregation and prove that A ( G ) is a Markov transition matrix.
Revised: In particular, we ﬁrst analyze the network with only aggregation and prove that A ( G ) is a Markov transition matrix.

Original: Then, we formulate the behavior of MLP in the large depth based on Hayou et al.
Revised: Then, we formulate the behavior of MLP in the large depth based on Hayou et al.

Original: (2019b).
Revised: (2019b).

Original: Finally, we derive the ﬁnal result by considering the two operations.
Revised: Finally, we derive the ﬁnal result by considering the two operations.

Original: We leave the complete proof in Appendix B.
Revised: We leave the complete proof in Appendix B.

Original: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.
Revised: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.

Original: As the depth goes to inﬁnity, all entries in the GNTK converge to a unique quantity at an exponential rate.
Revised: As the depth goes to inﬁnity, all entries in the GNTK converge to a unique quantity at an exponential rate.

Original: We thus have Θ ( l ) ( r ) ( V , V ) ≈
Revised: We thus have Θ ( l ) ( r ) ( V , V ) ≈

Original: (cid:126)π ( G ) T ( Rl(cid:126)v ) 1 n × n as l → ∞ , where 1 n × n is an ( n × n )-dimensional matrix whose entries are one.
Revised: (cid:126)π ( G ) T ( Rl(cid:126)v ) 1 n × n as l → ∞ , where 1 n × n is an ( n × n )-dimensional matrix whose entries are one.

Original: The exponential convergence rate of Θ ( l ) ( r ) ( G ) implies that the trainability of inﬁnitely-wide GCNs degenerates dramatically, as stated below.
Revised: The exponential convergence rate of Θ ( l ) ( r ) ( G ) implies that the trainability of inﬁnitely-wide GCNs degenerates dramatically, as stated below.

Original: Corollary 1 (Trainability of Ultra-Wide GCNs) .
Revised: Corollary 1 (Trainability of Ultra-Wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.
Revised: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.

Original: Then, the output function follows,
Revised: Then, the output function follows,

Original: [Equation]
Revised: [Equation]

Original: where Y is the label with respect to node V .
Revised: where Y is the label with respect to node V .

Original: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .
Revised: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .

Original: Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .
Revised: Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .

Original: We leave proof in Appendix C.
Revised: We leave proof in Appendix C.

Original: According to the above corollary, as l → ∞ , the GNTK matrix would become a singular matrix.
Revised: According to the above corollary, as l → ∞ , the GNTK matrix would become a singular matrix.

Original: This would lead to a discrepancy between output f t ( V ) and label
Revised: This would lead to a discrepancy between output f t ( V ) and label

Original: Y , which means GNTK loses the ability to ﬁt the label.
Revised: Y , which means GNTK loses the ability to ﬁt the label.

Original: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.
Revised: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.

Original:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION
Revised:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION

Original: We have so far characterized the trainability of vanilla GCNs through the GNTK and showed that the trainability of ultra-wide GCNs drops at an exponential rate.
Revised: We have so far characterized the trainability of vanilla GCNs through the GNTK and showed that the trainability of ultra-wide GCNs drops at an exponential rate.

Original: Recently, considerable efforts have been made to deepen GCNs, among which residual connection-based techniques are widely applied to resolve the over-smoothing problem (Li et al., 2019).
Revised: Recently, considerable efforts have been made to deepen GCNs, among which residual connection-based techniques are widely applied to resolve the over-smoothing problem (Li et al., 2019).

Original: We now apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.
Revised: We now apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.

Original: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,
Revised: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,

Original: [Equation]
Revised: [Equation]

Original: where 0 < δ < 1 .
Revised: where 0 < δ < 1 .

Original: Taking equation (12) as a new aggregation process, then (cid:126) Θ ( l ) ( G )
Revised: Taking equation (12) as a new aggregation process, then (cid:126) Θ ( l ) ( G )

Original: = ˜
Revised: = ˜

Original: A ( G ) (cid:126) Θ ( l − 1) ( G ) ,
Revised: A ( G ) (cid:126) Θ ( l − 1) ( G ) ,

Original: where ˜ A ( G ) =
Revised: where ˜ A ( G ) =

Original: (1 − δ )
Revised: (1 − δ )

Original: A ( G ) + δI .
Revised: A ( G ) + δI .

Original: We prove that ˜
Revised: We prove that ˜

Original: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix
Revised: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix

Original: A ( G ) .
Revised: A ( G ) .

Original: Theorem 2 (Convergence rate of residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate of residual connection in aggregation) .

Original: Consider a GNTK of nonlinear transformation (9) and residual connection (12).
Revised: Consider a GNTK of nonlinear transformation (9) and residual connection (12).

Original: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <
Revised: A ( G ) , there exist constants 0 <

Original: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on
Revised: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: The detailed proof of Theorem 2 and the relationship between convergence rate and second largest eigvenvalue can be found in Appendix D.1.
Revised: The detailed proof of Theorem 2 and the relationship between convergence rate and second largest eigvenvalue can be found in Appendix D.1.

Original: This theorem implies that a residual connection in aggregation can slow down the convergence rate, which is consistent with empirical observations that residual connection can help deepen GCNs.
Revised: This theorem implies that a residual connection in aggregation can slow down the convergence rate, which is consistent with empirical observations that residual connection can help deepen GCNs.

Original: Then, we consider residual connection only applied on non-linear transformation (MLP).
Revised: Then, we consider residual connection only applied on non-linear transformation (MLP).

Original: In this case, the recursive equation for the corresponding GNTK can be expressed as,
Revised: In this case, the recursive equation for the corresponding GNTK can be expressed as,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: This formula is similar to the vanilla GNTK in the inﬁnitely-wide limit.
Revised: This formula is similar to the vanilla GNTK in the inﬁnitely-wide limit.

Original: Only an additional residual term appears according to residual connection.
Revised: Only an additional residual term appears according to residual connection.

Original: It turns out that this term may not help slow down the convergence rate for non-linear transformation.
Revised: It turns out that this term may not help slow down the convergence rate for non-linear transformation.

Original: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .

Original: Consider a GNTK of the form (8) and (13).
Revised: Consider a GNTK of the form (8) and (13).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on R , such that
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on R , such that

Original: , (cid:12)(cid:12) Θ
Revised: , (cid:12)(cid:12) Θ

Original: [Equation]
Revised: [Equation]

Original: Note that α in Theorem 3 is the same as in Theorem 1.
Revised: Note that α in Theorem 3 is the same as in Theorem 1.

Original: The proof of Theorem 3 can be found in Appendix D.2.
Revised: The proof of Theorem 3 can be found in Appendix D.2.

Original: Theorem 3 demonstrates that adding residual connection in MLP can not even reduce the convergence rate of trainability.
Revised: Theorem 3 demonstrates that adding residual connection in MLP can not even reduce the convergence rate of trainability.

Original: Finally, we consider residual connection applied to both aggregation and non-linear transformation simultaneously:
Revised: Finally, we consider residual connection applied to both aggregation and non-linear transformation simultaneously:

Original: Corollary 2 (Convergence rate of GNTK with residual connection in aggregation and transformation) .
Revised: Corollary 2 (Convergence rate of GNTK with residual connection in aggregation and transformation) .

Original: Consider a GNTK of the form (12) and (13).
Revised: Consider a GNTK of the form (12) and (13).

Original: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 < ˜
Revised: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 < ˜

Original: α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) −
Revised: ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) −

Original: [Table]
Revised: [Table]

Original: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE
Revised: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE

Original: Residual connection is a technique designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay problem.
Revised: Residual connection is a technique designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay problem.

Original: To better resolve the problem of trainability loss, we need to look deeper into the root cause of the problem – the transition matrix corresponding to the aggregation operation.
Revised: To better resolve the problem of trainability loss, we need to look deeper into the root cause of the problem – the transition matrix corresponding to the aggregation operation.

Original: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected, which is a typical trait for graphs used on both node and graph classiﬁcation tasks.
Revised: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected, which is a typical trait for graphs used on both node and graph classiﬁcation tasks.

Original: Thus, breaking the connectivity condition is a promising direction.
Revised: Thus, breaking the connectivity condition is a promising direction.

Original: Interestingly, the critical percolation theory (Huang et al., 2018; Stauffer & Aharony, 2018) provides useful insights to better solve the exponential decay problem.
Revised: Interestingly, the critical percolation theory (Huang et al., 2018; Stauffer & Aharony, 2018) provides useful insights to better solve the exponential decay problem.

Original: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.
Revised: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.

Original: A random graph ˆ G is achieved by randomly and uniformly retaining some edges from the complete graph with an edge probability as p = | E | E t , where | E | is the number of edges retained in the random graph.
Revised: A random graph ˆ G is achieved by randomly and uniformly retaining some edges from the complete graph with an edge probability as p = | E | E t , where | E | is the number of edges retained in the random graph.

Original: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c
Revised: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c

Original: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).
Revised: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).

Original: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there exists an path from some ﬁxed point to a distance decreases polynomially:
Revised: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there exists an path from some ﬁxed point to a distance decreases polynomially:

Original: Proposition 1 (Critical Connectivity in Random Graph (Erd˝os & R´enyi, 1961)) .
Revised: Proposition 1 (Critical Connectivity in Random Graph (Erd˝os & R´enyi, 1961)) .

Original: Suppose a random graph ˆ G has n nodes with a constant edge probability p .
Revised: Suppose a random graph ˆ G has n nodes with a constant edge probability p .

Original: (1) If p < p c , then almost every random
Revised: (1) If p < p c , then almost every random

Original: [Figure]
Revised: [Figure]

Original: graph is such that its largest component 1 has O (log n ) ; (2) If p > p c , the random graph has a giant component of (1 − α p + o (1)) n , where α p <  If p = p c , then the maximal size of a component 
Revised: graph is such that its largest component 1 has O (log n ) ; (2) If p > p c , the random graph has a giant component of (1 − α p + o (1)) n , where α p < 1 ; (3) If p = p c , then the maximal size of a component of almost every graph has order n 2 / 3 .

Original: of almost every graph has order

Original: n The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate.
Revised: The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate.

Original: This inspires us to solve the problem of exponentially dropping trainability through designing a graph-dependent and connectivity-aware sampling algorithm called Critical DropEdge.
Revised: This inspires us to solve the problem of exponentially dropping trainability through designing a graph-dependent and connectivity-aware sampling algorithm called Critical DropEdge.

Original: In particular, given a graph G , we randomly drop some edges and retain the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.
Revised: In particular, given a graph G , we randomly drop some edges and retain the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.

Original: Unlike DropEdge (Rong et al., 2019) that randomly removes a certain number of edges from the input graph, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | .
Revised: Unlike DropEdge (Rong et al., 2019) that randomly removes a certain number of edges from the input graph, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | .

Original: It is worth noting that
Revised: It is worth noting that

Original: DropEdge may choose the edge probability as p < p c where information can only be passed within a distance of O (log n ) in the graph, or p > p c where the exponential dropping trainability problem may occur.
Revised: DropEdge may choose the edge probability as p < p c where information can only be passed within a distance of O (log n ) in the graph, or p > p c where the exponential dropping trainability problem may occur.

Original: A further discussion of the relationship between Proposition 1 and the trainability of the corresponding GNTK can be found in Appendix E.
Revised: A further discussion of the relationship between Proposition 1 and the trainability of the corresponding GNTK can be found in Appendix E.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we empirically verify our theoretical results and validate the proposed Critical
Revised: In this section, we empirically verify our theoretical results and validate the proposed Critical

Original: DropEdge method.
Revised: DropEdge method.

Original: Details of four real-world graph datasets used are summarized in Table 3 in
Revised: Details of four real-world graph datasets used are summarized in Table 3 in

Original: Appendix F.1.
Revised: Appendix F.1.

Original: 5. C ONVERGENCE R ESULTS OF GNTK S
Revised: 5. C ONVERGENCE R ESULTS OF GNTK S

Original: Theorems 1-3 provide theoretical convergence rates for the (residual) GNTK.
Revised: Theorems 1-3 provide theoretical convergence rates for the (residual) GNTK.

Original: We show the corresponding numerical veriﬁcation in Figure 1.
Revised: We show the corresponding numerical veriﬁcation in Figure 1.

Original: We select a graph randomly from the MUTAG dataset and generate the GNTK using the implementation of Du et al.
Revised: We select a graph randomly from the MUTAG dataset and generate the GNTK using the implementation of Du et al.

Original: (2019), with ReLU activation, R = 3 , and L = 300 .
Revised: (2019), with ReLU activation, R = 3 , and L = 300 .

Original: Figure 1 (a)-(c) show that all elements of normalized GNTKs converge to an identical value as the depth goes larger.
Revised: Figure 1 (a)-(c) show that all entries of normalized GNTKs converge to an identical value as the depth goes larger.

Original: Figure 1 (d)-(f) further illustrate the convergence rate of GNTK is exponential, as reﬂected by our theorems.
Revised: Figure 1 (d)-(f) further illustrate the convergence rate of GNTK is exponential, as reﬂected by our theorems.

Original: By comparing convergence rates, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with
Revised: By comparing convergence rates, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with

Original: Theorem 2.
Revised: Theorem 2.

Original: 5. T RAINABILITY OF W IDE GCN S
Revised: 5. T RAINABILITY OF W IDE GCN S

Original: We further examine whether ultra-wide GCNs can be trained successfully for node classiﬁcation.
Revised: We further examine whether ultra-wide GCNs can be trained successfully for node classiﬁcation.

Original: We conduct experiments on a GCN (Kipf & Welling, 2016a), where we apply a width of 1000 at each hidden layer and the depth ranging from 2 to 29 .
Revised: We conduct experiments on a GCN (Kipf & Welling, 2016a), where we apply a width of 1000 at each hidden layer and the depth ranging from 2 to 29 .

Original: Figure 2 shows the training and test accuracy on Cora, Citesser and Pubmed after 300 training epochs.
Revised: Figure 2 shows the training and test accuracy on Cora, Citesser and Pubmed after 300 training epochs.

Original: These results show a dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation, as revealed by Corollary 1.
Revised: These results show a dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation, as revealed by Corollary 1.

Original: 5. P ERFORMANCE OF C RITICAL D ROP E DGE
Revised: 5. P ERFORMANCE OF C RITICAL D ROP E DGE

Original: [Figure]
Revised: [Figure]

Original: We apply Critical DropEdge (referred to as CDropEdge) to both inﬁnitely-wide and ﬁnitelywide GCNs on node classiﬁcation under a semisupervised setting.
Revised: We apply Critical DropEdge (referred to as CDropEdge) to both inﬁnitely-wide and ﬁnitelywide GCNs on node classiﬁcation under a semisupervised setting.

Original: For inﬁnitely-wide GCNs, we consider two backbones: GCN (Kipf & Welling, 2016a) and
Revised: For inﬁnitely-wide GCNs, we consider two backbones: GCN (Kipf & Welling, 2016a) and

Original: JKNet (Xu et al., 2018b), while we use three backbones (GCN, JKNet, IncepGCN (Rong et al., 2019)) for ﬁnitely-wide GCNs.
Revised: JKNet (Xu et al., 2018b), while we use three backbones (GCN, JKNet, IncepGCN (Rong et al., 2019)) for ﬁnitely-wide GCNs.

Original: The results for inﬁnitely-wide GCNs can be found in Appendix G.1.
Revised: The results for inﬁnitely-wide GCNs can be found in Appendix G.1.

Original: As for ﬁnitely-wide GCNs, we use the hyper-parameters reported in Rong et al.
Revised: As for ﬁnitely-wide GCNs, we use the hyper-parameters reported in Rong et al.

Original: (2019) to obtain the results for DropEdge.
Revised: (2019) to obtain the results for DropEdge.

Original: For C-DropEdge, we perform a random hyper-parameter search and ﬁx the edge preserving rate as ρ ( G ) =
Revised: For C-DropEdge, we perform a random hyper-parameter search and ﬁx the edge preserving rate as ρ ( G ) =

Original: | V | 2 | E | .
Revised: | V | 2 | E | .

Original: In addition, we compare with DGN (Zhou et al., 2020), a normalization-based baseline.
Revised: In addition, we compare with DGN (Zhou et al., 2020), a normalization-based baseline.

Original: The backbone of
Revised: The backbone of

Original: DGN includes GCN (Kipf & Welling, 2016a) and GAT (Veli ˇ ckovi ´ c et al., 2017), and we report best performance across backbones.
Revised: DGN includes GCN (Kipf & Welling, 2016a) and GAT (Veli ˇ ckovi ´ c et al., 2017), and we report best performance across backbones.

Original: Table 1 summarizes node classiﬁcation performance of ﬁnitely-wide
Revised: Table 1 summarizes node classiﬁcation performance of ﬁnitely-wide

Original: GCNs on three citation networks (Cora, Citesser, Pubmed) and one co-author network (Physics).
Revised: GCNs on three citation networks (Cora, Citesser, Pubmed) and one co-author network (Physics).

Original: In this table, we report the performance of GNNs with 4/8/16/32 layers and implementation details can be found in Appendix F.2.
Revised: In this table, we report the performance of GNNs with 4/8/16/32 layers and implementation details can be found in Appendix F.2.

Original: In particular, we report the best performance across different backbones for DropEdge, C-DropEdge and DGN, and leave separate results with different backbones in Appendix G.2.
Revised: In particular, we report the best performance across different backbones for DropEdge, C-DropEdge and DGN, and leave separate results with different backbones in Appendix G.2.

Original: For each hyper-parameter, we run 10 times and report the mean and standard variance.
Revised: For each hyper-parameter, we run 10 times and report the mean and standard variance.

Original: It is shown that C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.
Revised: It is shown that C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.

Original: Besides, our C-DropEdge method can achieve smaller error variances, indicating that it is more robust than DropEdge.
Revised: Besides, our C-DropEdge method can achieve smaller error variances, indicating that it is more robust than DropEdge.

Original: Finally, we perform an empirical study of whether the proposed edge dropping rate is indeed optimal or close to optimal to strengthen our results.
Revised: Finally, we perform an empirical study of whether the proposed edge dropping rate is indeed optimal or close to optimal to strengthen our results.

Original: To this end, we conduct experiments on dropping edges at various rates (sampling percent) and veriﬁed that the rate of Critical DropEdge highlighted in bold can achieve the results close to optimal, as shown in Table 2.
Revised: To this end, we conduct experiments on dropping edges at various rates (sampling percent) and veriﬁed that the rate of Critical DropEdge highlighted in bold can achieve the results close to optimal, as shown in Table 2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Tangent Kernel.
Revised: Neural Tangent Kernel.

Original: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.
Revised: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.

Original: In the inﬁnite-width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018;
Revised: In the inﬁnite-width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018;

Original: Lee et al., 2019a; Arora et al., 2019; Yang, 2019; Huang & Yau, 2019; Allen-Zhu et al., 2019; Du et al., 2018; Zou et al., 2018).
Revised: Lee et al., 2019a; Arora et al., 2019; Yang, 2019; Huang & Yau, 2019; Allen-Zhu et al., 2019; Du et al., 2018; Zou et al., 2018).

Original: Besides, NTK has been applied to various architectures and brought a wealth of results, such as orthogonal initialization (Huang et al., 2020a), convolutions (Arora et al., 2019), graph neural networks (Du et al., 2019), attention (Hron et al., 2020), and refer to (Yang, 2020) for a summary.
Revised: Besides, NTK has been applied to various architectures and brought a wealth of results, such as orthogonal initialization (Huang et al., 2020a), convolutions (Arora et al., 2019), graph neural networks (Du et al., 2019), attention (Hron et al., 2020), and refer to (Yang, 2020) for a summary.

Original: As for graph networks, GNTK helps us understand how GNNs learn a class of smooth functions on graphs (Du et al., 2019) and how they extrapolate differently from multi-layer perceptron (Xu et al., 2020).
Revised: As for graph networks, GNTK helps us understand how GNNs learn a class of smooth functions on graphs (Du et al., 2019) and how they extrapolate differently from multi-layer perceptron (Xu et al., 2020).

Original: Deep Graph Neural Networks.
Revised: Deep Graph Neural Networks.

Original: Since deep GNNs suffer from the over-smoothing problem, a large and growing body of literature has made efforts in deepening GNNs.
Revised: Since deep GNNs suffer from the over-smoothing problem, a large and growing body of literature has made efforts in deepening GNNs.

Original: There is a line of methods that resort to residual connection to retain their feature expressivity in deep layers.
Revised: There is a line of methods that resort to residual connection to retain their feature expressivity in deep layers.

Original: Xu et al.
Revised: Xu et al.

Original: (2018b)
Revised: (2018b)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: use skip connection along with node representations from different neighborhood ranges to preserve the locality of node representations.
Revised: use skip connection along with node representations from different neighborhood ranges to preserve the locality of node representations.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2018) derive a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.
Revised: (2018) derive a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.

Original: Other similar works using residual connection can also be seen in (Li et al., 2019;
Revised: Other similar works using residual connection can also be seen in (Li et al., 2019;

Original: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).
Revised: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).

Original: Another line of approaches tackle the issue via regularization mechanisms, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2019), and group normalization (Zhou et al., 2020).
Revised: Another line of approaches tackle the issue via regularization mechanisms, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2019), and group normalization (Zhou et al., 2020).

Original: Recently, a series of latest works (Loukas, 2019; Zeng et al., 2020; Li et al., 2020;
Revised: Recently, a series of latest works (Loukas, 2019; Zeng et al., 2020; Li et al., 2020;

Original: Cong et al., 2021; Huang et al., 2020b) explore the underlying reasons for performance degradation towards mitigation solutions.
Revised: Cong et al., 2021; Huang et al., 2020b) explore the underlying reasons for performance degradation towards mitigation solutions.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.
Revised: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.

Original: We prove that the trainability drops at an exponential rate due to the aggregation operation.
Revised: We prove that the trainability drops at an exponential rate due to the aggregation operation.

Original: Furthermore, we apply our theoretical framework to investigate to what extent residual connection-based techniques help deepen GCNs.
Revised: Furthermore, we apply our theoretical framework to investigate to what extent residual connection-based techniques help deepen GCNs.

Original: We demonstrate that these techniques can only slow down the decay rate, but are unable to solve the exponential decay problem in essence.
Revised: We demonstrate that these techniques can only slow down the decay rate, but are unable to solve the exponential decay problem in essence.

Original: To overcome the trainability loss problem, we further propose Critical DropEdge illuminated by our theoretical framework.
Revised: To overcome the trainability loss problem, we further propose Critical DropEdge illuminated by our theoretical framework.

Original: The experimental results conﬁrm that our method can mitigate the trainability problems of deep GCNs.
Revised: The experimental results conﬁrm that our method can mitigate the trainability problems of deep GCNs.

Original: Future research directions include designing a critical node-centric method so as to make better use of node information.
Revised: Future research directions include designing a critical node-centric method so as to make better use of node information.

Original: R EPRODUCIBILITY S TATEMENT

Original: To ensure the results and conclusions of our paper are reproducible we make the following efforts:

Original: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 3 and 4; Appendix B and D.

Original: Experimentally, we provide our code, and instructions needed to reproduce the main experimental results.

Original: And we specify all the training and implementation details in Section 5 and Appendix F and

Original: G. Besides, we independently run experiments 10 times and report the mean and standard deviation

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: We thank the anonymous reviewers for useful suggestions to improve the paper.
Revised: We thank the anonymous reviewers for useful suggestions to improve the paper.

Revised: We thank Ye Su for helpful discussions.

Original: This work was carried out in collaboration with Digital Research Centre Denmark – DIREC, supported by Innovation Fund Denmark.
Revised: This work was carried out in collaboration with Digital Research Centre Denmark – DIREC, supported by Innovation Fund Denmark.

Original: A APPENDIX: A G ENERAL GCN CONSIDERED IN THIS WORK
Revised: A APPENDIX: A G ENERAL GCN CONSIDERED IN THIS WORK

Original: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.

Original: Figure 3 illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 illustrates an overview of the information propagation in a general GCN.

Original: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK
Revised: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK

Original: In theorem 1 we demonstrate that the matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.
Revised: In theorem 1 we demonstrate that the matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.

Original: To this end, we block the non-linear transformation by setting R = 0 .
Revised: To this end, we block the non-linear transformation by setting R = 0 .

Original: Then and the propagation of GNTK can be expressed as,
Revised: Then and the propagation of GNTK can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite Equation (14) as the following format,
Revised: In order to facilitate calculation, we rewrite Equation (14) as the following format,

Original: [Equation]
Revised: [Equation]

Original: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and the matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.
Revised: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and the matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.

Original: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( V , V ) in the following lemma,
Revised: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( V , V ) in the following lemma,

Original: Lemma 1 (Convergence of Aggregation) .
Revised: Lemma 1 (Convergence of Aggregation) .

Original: Assume R = 0 , then
Revised: Assume R = 0 , then

Original: [Equation]
Revised: [Equation]

Original: where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) = (cid:126)π ( G ) .
Revised: where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) = (cid:126)π ( G ) .

Original: Proof.
Revised: Proof.

Original: When R = 0 , Equation (14) reduces to,
Revised: When R = 0 , Equation (14) reduces to,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite the above equation in the format of matrix,
Revised: In order to facilitate calculation, we rewrite the above equation in the format of matrix,

Original: [Equation]
Revised: [Equation]

Original: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.
Revised: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.

Original: Thus the matrix operation A ( l ) ( G ) ∈
Revised: Thus the matrix operation A ( l ) ( G ) ∈

Original: R n 2 × n 2 .
Revised: R n 2 × n 2 .

Original: It is obvious that,
Revised: It is obvious that,

Original: [Equation]
Revised: [Equation]

Original: This implies that the aggregate operation is the same for each layer.
Revised: This implies that the aggregate operation is the same for each layer.

Original: The next step is to prove A ( G ) is a stochastic matrix (transition matrix):
Revised: The next step is to prove A ( G ) is a stochastic matrix (transition matrix):

Original: [Equation]
Revised: [Equation]

Original: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrix,
Revised: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrix,

Original: [Equation]
Revised: [Equation]

Original: where B ( G ) , C ( G ) ∈ R n × n .
Revised: where B ( G ) , C ( G ) ∈ R n × n .

Original: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor
Revised: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor

Original: [Equation]
Revised: [Equation]

Original: (2) The element of matrix C ( G ) is determined by whether there is a edge between two vertexes,
Revised: (2) The element of matrix C ( G ) is determined by whether there is a edge between two vertexes,

Original: [Equation]
Revised: [Equation]

Original: where ˜ δ ij = 1 if i ==
Revised: where ˜ δ ij = 1 if i ==

Original: j or there is edge between vertex i and j , else ˜ δ ij = 0 .
Revised: j or there is edge between vertex i and j , else ˜ δ ij = 0 .

Original: We then use the property of matrix B and C before Kronecker product,
Revised: We then use the property of matrix B and C before Kronecker product,

Original: [Equation]
Revised: [Equation]

Original: According to the deﬁnition of Kronecker product, we have,
Revised: According to the deﬁnition of Kronecker product, we have,

Original: [Equation]
Revised: [Equation]

Original: where i = a + ( c − 1) n , and j = b + ( d − 1) n .
Revised: where i = a + ( c − 1) n , and j = b + ( d − 1) n .

Original: So far, we have proved that A ( G ) is a stochastic matrix.
Revised: So far, we have proved that A ( G ) is a stochastic matrix.

Original: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, that does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution which is also a eigenvector of the probability matrix, associated with eigenvalue 1:
Revised: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, that does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution which is also a eigenvector of the probability matrix, associated with eigenvalue 1:

Original: [Equation]
Revised: [Equation]

Original: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.
Revised: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.

Original: Thus the convergence rate is governed by the second largest eigenvalue.
Revised: Thus the convergence rate is governed by the second largest eigenvalue.

Original: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,
Revised: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,

Original: [Equation]
Revised: [Equation]

Original: where Π( G ) = 
Revised: where Π( G ) = 

Original: [Equation]
Revised: [Equation]

Original: (cid:126)π
Revised: (cid:126)π

Original: [Equation]
Revised: [Equation]

Original: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.
Revised: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.

Original: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,
Revised: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈
Revised: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈

Original: R nn (cid:48)
Revised: R nn (cid:48)

Original: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertexes in graph G (cid:48) .
Revised: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertexes in graph G (cid:48) .

Original: B. C ONVERGENCE OF MLP GPK
Revised: B. C ONVERGENCE OF MLP GPK

Original: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.
Revised: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.

Original: By doing so, we consider a network with only non-linear transformation, which is known as a pure MLP.
Revised: By doing so, we consider a network with only non-linear transformation, which is known as a pure MLP.

Original: This leads to Σ ( l ) ( r ) ( u, u (cid:48) )
Revised: This leads to Σ ( l ) ( r ) ( u, u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.
Revised: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.

Original: And we rewrite the propagation function for GPK as follows,
Revised: And we rewrite the propagation function for GPK as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,
Revised: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,

Original: [Equation]
Revised: [Equation]

Original: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.
Revised: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.

Original: In particular, we set the value of hyper-parameters to satisfy,
Revised: In particular, we set the value of hyper-parameters to satisfy,

Original: [Equation]
Revised: [Equation]

Original: where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, Equation (18) requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, Equation (18) requires σ 2 w = 2 and σ 2 b = 0 .

Original: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,
Revised: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 ( Proposition 1 and 3 in Hayou et al.
Revised: Lemma 2 ( Proposition 1 and 3 in Hayou et al.

Original: (2019a)) .
Revised: (2019a)) .

Original: Assume a network without aggregation,
Revised: Assume a network without aggregation,

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: e., L = 0 , with a Lipschitz nonlinearity φ , then,
Revised: e., L = 0 , with a Lipschitz nonlinearity φ , then,

Original: • φ ( x ) =
Revised: • φ ( x ) =

Original: ( x ) + ,
Revised: ( x ) + ,

Original: [Equation]
Revised: [Equation]

Original: r ( u, u (cid:48)
Revised: r ( u, u (cid:48)

Original: [Equation]
Revised: [Equation]

Original: ∞ • φ ( x ) =
Revised: ∞ • φ ( x ) =

Original: tanh( x ) , 1 − C
Revised: tanh( x ) , 1 − C

Original: [Equation]
Revised: [Equation]

Original: ( u, u
Revised: ( u, u

Original: [Equation]
Revised: [Equation]

Original: as
Revised: as

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, one is diagonal element and the other is non-diagonal element:
Revised: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, one is diagonal element and the other is non-diagonal element:

Original: [Equation]
Revised: [Equation]

Original: where u=
Revised: where u=

Original: [Equation]
Revised: [Equation]

Original: Σ ( r − 1) ( u, u ) z 1 , and u
Revised: Σ ( r − 1) ( u, u ) z 1 , and u

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: −
Revised: −

Original: [Equation]
Revised: [Equation]

Original: u,
Revised: u,

Original: [Equation]
Revised: [Equation]

Original: , with C r − 1 ( u, u
Revised: , with C r − 1 ( u, u

Original: [Equation]
Revised: [Equation]

Original: u,
Revised: u,

Original: [Equation]
Revised: [Equation]

Original: ) / (cid:112) Σ ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .
Revised: ) / (cid:112) Σ ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .

Original: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )
Revised: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u (cid:48) , u (cid:48) ) .
Revised: Σ ( r ) ( u (cid:48) , u (cid:48) ) .

Original: We then process the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.
Revised: We then process the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.

Original: ReLU activation, φ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: The recursive equation (16) for q r ( u ) reduces to,
Revised: The recursive equation (16) for q r ( u ) reduces to,

Original: [Equation]
Revised: [Equation]

Original: The edge of chaos condition σ
Revised: The edge of chaos condition σ

Original: [Equation]
Revised: [Equation]

Original: z [
Revised: z [

Original: [Equation]
Revised: [Equation]

Original: ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,
Revised: ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,

Original: [Equation]
Revised: [Equation]

Original: Then the second integration for C r ( u, u (cid:48) ) becomes,
Revised: Then the second integration for C r ( u, u (cid:48) ) becomes,

Original: [Equation]
Revised: [Equation]

Original: To investigate the propagation of C r ( u, u (cid:48) )
Revised: To investigate the propagation of C r ( u, u (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: we set q r ( u ) =
Revised: we set q r ( u ) =

Original: q r ( u (cid:48) )
Revised: q r ( u (cid:48) )

Original: =
Revised: =

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: and deﬁne,
Revised: and deﬁne,

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: Let x ∈ [
Revised: Let x ∈ [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: the derivative of f ( x ) satisﬁes,
Revised: the derivative of f ( x ) satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: This can be seen from a simple derivation,
Revised: This can be seen from a simple derivation,

Original: [Equation]
Revised: [Equation]

Original: Using an identity for Gaussian integration (cid:82) D z zg ( z ) =
Revised: Using an identity for Gaussian integration (cid:82) D z zg ( z ) =

Original: [Equation]
Revised: [Equation]

Original: ) yields,
Revised: ) yields,

Original: [Equation]
Revised: [Equation]

Original: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48)
Revised: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: [Equation]
Revised: [Equation]

Original: So using the equation above and the condition f (cid:48) (0)
Revised: So using the equation above and the condition f (cid:48) (0)

Original: =
Revised: =

Original: 12 , we can get another formation of the derivative of f ( x ) ,
Revised: 12 , we can get another formation of the derivative of f ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,
Revised: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Substituting f ( x ) =
Revised: Substituting f ( x ) =

Original: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression up here, we have,
Revised: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression up here, we have,

Original: [Equation]
Revised: [Equation]

Original: Now we study the behavior of C r ( u, u (cid:48) )
Revised: Now we study the behavior of C r ( u, u (cid:48) )

Original: as r tends to inﬁnity.
Revised: as r tends to inﬁnity.

Original: Using Taylor expansion, we have,
Revised: Using Taylor expansion, we have,

Original: [Equation]
Revised: [Equation]

Original: By induction analysis, the sequence C r ( u, u (cid:48) ) is increasing to the ﬁxed point 1.
Revised: By induction analysis, the sequence C r ( u, u (cid:48) ) is increasing to the ﬁxed point 1.

Original: Besides, we can replace x with C r ( u, u (cid:48) ) ,
Revised: Besides, we can replace x with C r ( u, u (cid:48) ) ,

Original: [Equation]
Revised: [Equation]

Original: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,
Revised: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: so that,
Revised: so that,

Original: [Equation]
Revised: [Equation]

Original: As r tends to inﬁnity, we have,
Revised: As r tends to inﬁnity, we have,

Original: [Equation]
Revised: [Equation]

Original: It means,
Revised: It means,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have,
Revised: Therefore, we have,

Original: [Equation]
Revised: [Equation]

Original: Tanh activation, φ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: For the argument of ﬁxed point, we ask readers to refer to
Revised: For the argument of ﬁxed point, we ask readers to refer to

Original: Hayou et al.
Revised: Hayou et al.

Original: (2019a).
Revised: (2019a).

Original: Here we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to be 1 as the depth tends to be inﬁnity lim r →∞ C r ( u, u (cid:48) )
Revised: Here we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to be 1 as the depth tends to be inﬁnity lim r →∞ C r ( u, u (cid:48) )

Original: = 1 .
Revised: = 1 .

Original: For the function f ( x ) , a Taylor expansion near 1 yields,
Revised: For the function f ( x ) , a Taylor expansion near 1 yields,

Original: [Equation]
Revised: [Equation]

Original: where f (cid:48) (1)
Revised: where f (cid:48) (1)

Original: =
Revised: =

Original: σ 2 w (cid:82) D z [ φ (cid:48) ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =
Revised: σ 2 w (cid:82) D z [ φ (cid:48) ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =

Original: σ 2 w q (cid:82) D
Revised: σ 2 w q (cid:82) D

Original: z
Revised: z

Original: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .
Revised: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .

Original: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,
Revised: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = 2 (cid:82)
Revised: = 2 (cid:82)

Original: [Equation]
Revised: [Equation]

Original: ( √ q ∗ z ) 2 ] .
Revised: ( √ q ∗ z ) 2 ] .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Thus, we have,
Revised: Thus, we have,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 shows the covariance matrix converges to a constant matrix at a polynomial rate of 1 /rfor ReLU and of 1 /r for tanh activation on the edge of chaos.
Revised: Lemma 2 shows the covariance matrix converges to a constant matrix at a polynomial rate of 1 /rfor ReLU and of 1 /r for tanh activation on the edge of chaos.

Original: This implies that a network without aggregation could retain its expressivity at a large depth.
Revised: This implies that a network without aggregation could retain its expressivity at a large depth.

Original: However, for general case, due to the aggregation, the rate of convergence will change from polynomial to exponential.
Revised: However, for general case, due to the aggregation, the rate of convergence will change from polynomial to exponential.

Original: This will make the trainability of the deep graph network problematic, which will be shown in the following section.
Revised: This will make the trainability of the deep graph network problematic, which will be shown in the following section.

Original: B. C ONVERGENCE OF GPK FOR GCN S
Revised: B. C ONVERGENCE OF GPK FOR GCN S

Original: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:
Revised: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:

Original: Lemma 3.
Revised: Lemma 3.

Original: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that
Revised: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove the result by induction method.
Revised: We prove the result by induction method.

Original: For l = 0 , according to the Cauchy-BuniakowskySchwarz Inequality
Revised: For l = 0 , according to the Cauchy-BuniakowskySchwarz Inequality

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization, such that,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization, such that,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,
Revised: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where C 0 is a constant satisfying 0 < C 0 < C .
Revised: where C 0 is a constant satisfying 0 < C 0 < C .

Original: Now we consider the distance between Σ ( l +1)
Revised: Now we consider the distance between Σ ( l +1)

Original: ( r ) ( u, u (cid:48) ) and Cα l +1 .
Revised: ( r ) ( u, u (cid:48) ) and Cα l +1 .

Original: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.
Revised: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.

Original: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .
Revised: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .

Original: It is not hard to ﬁnd that steps 1,3 correspond to non-linear transformation while step 2 corresponds with aggregation operation, we then characterize them one by one,
Revised: It is not hard to ﬁnd that steps 1,3 correspond to non-linear transformation while step 2 corresponds with aggregation operation, we then characterize them one by one,

Original: MLP Propagation
Revised: MLP Propagation

Original: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π
Revised: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π

Original: [Equation]
Revised: [Equation]

Original: G ) T (cid:126)v (cid:12)(cid:12) ≤ C 

Revised: G ) T (cid:126)v (cid:12)(cid:12) ≤ C

Original: α l implicitly implies that
Revised: α l implicitly implies that

Original: C r ( u, u (cid:48) ) is close to one.
Revised: C r ( u, u (cid:48) ) is close to one.

Original: Because C r ( u, u (cid:48) ) =
Revised: Because C r ( u, u (cid:48) ) =

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: r
Revised: r

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (cid:48) ) , then,
Revised: (cid:48) ) , then,

Original: [Equation]
Revised: [Equation]

Original: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )
Revised: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )

Original: =
Revised: =

Original: 1+ O ( α l ) .
Revised: 1+ O ( α l ) .

Original: Recall the property of MLP propagation function f ( x ) for x =
Revised: Recall the property of MLP propagation function f ( x ) for x =

Original: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:
Revised: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:

Original: [Equation]
Revised: [Equation]

Original: This implies,
Revised: This implies,

Original: [Equation]
Revised: [Equation]

Original: With this result, we can further obtain,
Revised: With this result, we can further obtain,

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a positive constant.
Revised: where C 1 is a positive constant.

Original: Repeat the proof process, we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1,
Revised: Repeat the proof process, we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1,

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: Aggregation Propagation We go through an aggregation operation
Revised: Aggregation Propagation We go through an aggregation operation

Original: A ( G ) .
Revised: A ( G ) .

Original: In this case, we use a formation of matrix, and take Equation (20) into aggregation function, yields the result as follows,
Revised: In this case, we use a formation of matrix, and take Equation (20) into aggregation function, yields the result as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .
Revised: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .

Original: According to the theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .
Revised: According to the theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .

Original: When l is ﬁnite, the error is of exponential decay.
Revised: When l is ﬁnite, the error is of exponential decay.

Original: Here we use 0 < α < 1 to denote the corresponding base number.
Revised: Here we use 0 < α < 1 to denote the corresponding base number.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Finally, Repeat the result in step 1, we have,
Revised: Finally, Repeat the result in step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.
Revised: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.

Original: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) which is a smaller order compared to O ( α l ) .
Revised: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) which is a smaller order compared to O ( α l ) .

Original: B. C ONVERGENCE OF THE GNTK
Revised: B. C ONVERGENCE OF THE GNTK

Original: Finally, we arrive at our main theorem:
Revised: Finally, we arrive at our main theorem:

Original: Theorem 1 (Convergence Rate of the GNTK) .
Revised: Theorem 1 (Convergence Rate of the GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) =
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) =

Original: A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and
Revised: A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and

Original: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: Then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ Cα l .
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ Cα l .

Original: Proof.
Revised: Proof.

Original: This proof has the same strategy to that of Lemma 3.
Revised: This proof has the same strategy to that of Lemma 3.

Original: The ﬁrst step is to understand the
Revised: The ﬁrst step is to understand the

Original: Equation (9) in the large-depth limit.
Revised: Equation (9) in the large-depth limit.

Original: [Equation]
Revised: [Equation]

Original: According to the result of Lemma 3, we have already known,
Revised: According to the result of Lemma 3, we have already known,

Original: [Equation]
Revised: [Equation]

Original: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.
Revised: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.

Original: ReLU activation, φ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: Recall that we deﬁne C r
Revised: Recall that we deﬁne C r

Original: +

Revised: +

Original: =
Revised: =

Original: f
Revised: f

Original: ( C r ) , and we have,
Revised: ( C r ) , and we have,

Original: [Equation]
Revised: [Equation]

Original: Then, at the edge of chaos,
Revised: Then, at the edge of chaos,

Original: [Equation]
Revised: [Equation]

Original: arcsin
Revised: arcsin

Original: [Equation]
Revised: [Equation]

Original: +
Revised: +

Original: [Equation]
Revised: [Equation]

Original: Tanh activation, φ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: At the edge of chaos,
Revised: At the edge of chaos,

Original: [Equation]
Revised: [Equation]

Original: Now we prove the result by induction method.
Revised: Now we prove the result by induction method.

Original: For l = 0 , we directly have,
Revised: For l = 0 , we directly have,

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,
Revised: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: Now we consider the distance between Θ ( l +1)
Revised: Now we consider the distance between Θ ( l +1)

Original: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R ( l +1) (cid:126)v + (cid:126)v (cid:48) ) .
Revised: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R ( l +1) (cid:126)v + (cid:126)v (cid:48) ) .

Original: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.
Revised: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.

Original: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.
Revised: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.

Original: Θ ( l +1) (0) →
Revised: Θ ( l +1) (0) →

Original: [Equation]
Revised: [Equation]

Original: For step 1, we have,
Revised: For step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the process, we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,
Revised: Repeat the process, we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,

Original: [Equation]
Revised: [Equation]

Original: Secondly
Revised: Secondly

Original: [Equation]
Revised: [Equation]

Original: we go through an aggregation operation.
Revised: we go through an aggregation operation.

Original: Because it is a Markov chain step
Revised: Because it is a Markov chain step

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the result in step 1, we ﬁnally have,
Revised: Repeat the result in step 1, we ﬁnally have,

Original: [Equation]
Revised: [Equation]

Original: C T RAINABILITY OF U LTRA -W IDE GCN S
Revised: C T RAINABILITY OF U LTRA -W IDE GCN S

Original: Corollary 1 (Trainability of Ultra-Wide GCNs) .
Revised: Corollary 1 (Trainability of Ultra-Wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth l , non-linear transformations number r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.
Revised: Consider a GCN of the form (6) and (7), with depth l , non-linear transformations number r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.

Original: Then the output function follows,
Revised: Then the output function follows,

Original: [Equation]
Revised: [Equation]

Original: where Y is the label with respect to node V .
Revised: where Y is the label with respect to node V .

Original: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .
Revised: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ .

Original: Moreover, there exist a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .
Revised: Moreover, there exist a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .

Original: Proof.
Revised: Proof.

Original: According to the result from Jacot et al.
Revised: According to the result from Jacot et al.

Original: (2018), GNTK Θ ( l ) ( r ) ( G ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite width limit.
Revised: (2018), GNTK Θ ( l ) ( r ) ( G ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite width limit.

Original: We omit proof procedure for this result, since it is now a standard conclusion in the NTK study.
Revised: We omit proof procedure for this result, since it is now a standard conclusion in the NTK study.

Original: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,
Revised: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,

Original: [Equation]
Revised: [Equation]

Original: where ω t = θ t − θ 0 .
Revised: where ω t = θ t − θ 0 .

Original: We call it linearized model because only zero and ﬁrst order term of Taylor expansion are kept.
Revised: We call it linearized model because only zero and ﬁrst order term of Taylor expansion are kept.

Original: Since we know dynamics of gradient ﬂow using this linearized function are governed by,
Revised: Since we know dynamics of gradient ﬂow using this linearized function are governed by,

Original: [Equation]
Revised: [Equation]

Original: where L is an MSE loss, then the above equations have closed form solutions
Revised: where L is an MSE loss, then the above equations have closed form solutions

Original: [Equation]
Revised: [Equation]

Original: Since Lee et al.
Revised: Since Lee et al.

Original: Lee et al. (2019a) showed that f lin t ( V ) =
Revised: Lee et al. (2019a) showed that f lin t ( V ) =

Original: f t in the inﬁnite width limit, thus we have,
Revised: f t in the inﬁnite width limit, thus we have,

Original: [Equation]
Revised: [Equation]

Original: In theorem 1, we have shown Θ ( l ) ( r ) ( G ) converges a constant matrix at an exponential rate, thus being singular in the large depth limit.
Revised: In theorem 1, we have shown Θ ( l ) ( r ) ( G ) converges a constant matrix at an exponential rate, thus being singular in the large depth limit.

Original: According to Equation (21), we know that,
Revised: According to Equation (21), we know that,

Original: [Equation]
Revised: [Equation]

Original: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( G ) =
Revised: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( G ) =

Original: Q T DQ be the decomposition of the
Revised: Q T DQ be the decomposition of the

Original: GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.
Revised: GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.

Original: Because Θ ( ∞ ) ( r ) ( G ) is singular,
Revised: Because Θ ( ∞ ) ( r ) ( G ) is singular,

Original: D has at least one zero value d j = 0 , then
Revised: D has at least one zero value d j = 0 , then

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: In the proof we assume the loss is MSE.
Revised: In the proof we assume the loss is MSE.

Original: Nevertheless, the conclude regarding trainability can be applied to other common loss such as cross-entropy.
Revised: Nevertheless, the conclude regarding trainability can be applied to other common loss such as cross-entropy.

Original: For cross-entropy loss, even though we cannot write down an analytical expression for the solution, we prove the GNTK still governs the trainability and the law of GNTK is not affected by the loss.
Revised: For cross-entropy loss, even though we cannot write down an analytical expression for the solution, we prove the GNTK still governs the trainability and the law of GNTK is not affected by the loss.

Original: Thus the results for trainability hold on the cross-entropy case.
Revised: Thus the results for trainability hold on the cross-entropy case.

Original: D C ONVERGENCE OF R ESIDUAL GNTK D. R ESIDUAL CONNECTION IN AGGREGATION
Revised: D C ONVERGENCE OF R ESIDUAL GNTK D. R ESIDUAL CONNECTION IN AGGREGATION

Original: Theorem 2 (Convergence rate for residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate for residual connection in aggregation) .

Original: Consider a GNTK of nonlinear transformation (9) and residual connection (12).
Revised: Consider a GNTK of nonlinear transformation (9) and residual connection (12).

Original: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <
Revised: A ( G ) , there exist constants 0 <

Original: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on
Revised: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: Proof.
Revised: Proof.

Original: According to the aggregation function for covariance matrix, we have
Revised: According to the aggregation function for covariance matrix, we have

Original: [Equation]
Revised: [Equation]

Original: The new aggregation matrix ˜
Revised: The new aggregation matrix ˜

Original: A ( G ) is a stochastic transition matrix as well, which can be seen from,
Revised: A ( G ) is a stochastic transition matrix as well, which can be seen from,

Original: [Equation]
Revised: [Equation]

Original: Then we compare the second largest eigenvalue between two transition matrix.
Revised: Then we compare the second largest eigenvalue between two transition matrix.

Original: Suppose the eigenvalues of original matrix
Revised: Suppose the eigenvalues of original matrix

Original: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .
Revised: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .

Original: We already know that the maximum eigenvalue is λ 

Revised: We already know that the maximum eigenvalue is λ

Original: = 1 ,
Revised: = 1 ,

Original: and the converge speed is governed by the second largest eigenvalue λ 2 .
Revised: and the converge speed is governed by the second largest eigenvalue λ 2 .

Original: Now we consider the second largest eigenvalue ˜ λ
Revised: Now we consider the second largest eigenvalue ˜ λ

Original: 2 of matrix ˜
Revised: 2 of matrix ˜

Original: A :
Revised: A :

Original: [Equation]
Revised: [Equation]

Original: Remark 4.
Revised: Remark 4.

Original: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 .
Revised: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 .

Original: Here we discuss more about the relationship between α and ˜ α .
Revised: Here we discuss more about the relationship between α and ˜ α .

Original: According to (Rosenthal, 1995), the relationship between convergence rate α and second largest eigenvalue λ 2 of transition matrix
Revised: According to (Rosenthal, 1995), the relationship between convergence rate α and second largest eigenvalue λ 2 of transition matrix

Original: A ( G ) can be expressed as,
Revised: A ( G ) can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .
Revised: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .

Original: From the above inequality, we can conclude that the second largest eigenvalue almost govern the convergence rate.
Revised: From the above inequality, we can conclude that the second largest eigenvalue almost govern the convergence rate.

Original: However, to maintain rigor, we cannot directly obtain α < ˜ α .
Revised: However, to maintain rigor, we cannot directly obtain α < ˜ α .

Original: Instead, from the result that ˜ λ 2 > λ 2 , we say that ˜ α is roughly larger than α .
Revised: Instead, from the result that ˜ λ 2 > λ 2 , we say that ˜ α is roughly larger than α .

Original: D. R ESIDUAL CONNECTION IN MLP
Revised: D. R ESIDUAL CONNECTION IN MLP

Original: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .

Original: Consider a GNTK the form (8) and (12).
Revised: Consider a GNTK the form (8) and (12).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G, G (cid:48) ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v (cid:48) depending on R , such that, | Θ ( l ) ( r )
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G, G (cid:48) ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v (cid:48) depending on R , such that, | Θ ( l ) ( r )

Original: [Equation]
Revised: [Equation]

Original: u, u
Revised: u, u

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For residual connection in MLP, we have,
Revised: For residual connection in MLP, we have,

Original: [Equation]
Revised: [Equation]

Original: Since σ 2 w > 0 , q r ( G ) grows at an exponential rate.
Revised: Since σ 2 w > 0 , q r ( G ) grows at an exponential rate.

Original: Now, we turn to compute the correlation term C r ( u, u (cid:48) ) .
Revised: Now, we turn to compute the correlation term C r ( u, u (cid:48) ) .

Original: For convenience, we suppose q r ( u ) =
Revised: For convenience, we suppose q r ( u ) =

Original: q r ( u (cid:48) ) .
Revised: q r ( u (cid:48) ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Using Taylor expansion of f near 1, as have been done in the proof of Lemma 2,
Revised: Using Taylor expansion of f near 1, as have been done in the proof of Lemma 2,

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Note that it is similar to the case of MLP without residual connection:
Revised: Note that it is similar to the case of MLP without residual connection:

Original: [Equation]
Revised: [Equation]

Original: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,
Revised: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,

Original: [Equation]
Revised: [Equation]

Original: E APPENDIX: D ISCUSSION ON THE TRAINABILITY OF GNTK WITH C RTICAL D ROP E DGE
Revised: E APPENDIX: D ISCUSSION ON THE TRAINABILITY OF GNTK WITH C RTICAL D ROP E DGE

Original: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd ˝ os & R ´ enyi, 1961;
Revised: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd ˝ os & R ´ enyi, 1961;

Original: Erd ¨ os & R ´ enyi, 2011), here n is the number of nodes in a graph.
Revised: Erd ¨ os & R ´ enyi, 2011), here n is the number of nodes in a graph.

Original: This implies that there exist a (connected) graph that of size O ( n ) when we use the critical dropping rate.
Revised: This implies that there exist a (connected) graph that of size O ( n ) when we use the critical dropping rate.

Original: To preserve information, edges will be resampled at every iteration.
Revised: To preserve information, edges will be resampled at every iteration.

Original: This implies that at each iteration during training, the GNN takes a random and large graph.
Revised: This implies that at each iteration during training, the GNN takes a random and large graph.

Original: For the whole training process, we can think that we have used all the information on the graph.
Revised: For the whole training process, we can think that we have used all the information on the graph.

Original: It would be interesting to consider achieving some theoretical results for the limiting behavior of the
Revised: It would be interesting to consider achieving some theoretical results for the limiting behavior of the

Original: GNTK in the large depth with Critical DropEdge.
Revised: GNTK in the large depth with Critical DropEdge.

Original: Since the Critical DropEdge is originated from random graph, a promising way is to analyze the spectral distributions of adjacency and Laplacian matrices with random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).
Revised: Since the Critical DropEdge is originated from random graph, a promising way is to analyze the spectral distributions of adjacency and Laplacian matrices with random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).

Original: F APPENDIX: D ATASET AND E XPERIMENTAL I MPLEMENTATION F. D ATASET
Revised: F APPENDIX: D ATASET AND E XPERIMENTAL I MPLEMENTATION F. D ATASET

Original: MUTAG (Rossi & Ahmed, 2015) is a bioinformatics dataset; Cora, Citeseer, Pubmed (Kipf &
Revised: MUTAG (Rossi & Ahmed, 2015) is a bioinformatics dataset; Cora, Citeseer, Pubmed (Kipf &

Original: Welling, 2016a) are commonly used citation network data for the node classiﬁcation.
Revised: Welling, 2016a) are commonly used citation network data for the node classiﬁcation.

Original: Besides,
Revised: Besides,

Original: Physics (Shchur et al., 2018) is a co-author network dataset.
Revised: Physics (Shchur et al., 2018) is a co-author network dataset.

Original: We illustrate the detailed information for the four node classiﬁcation datasets as follows and summary can be found in Table 3.
Revised: We illustrate the detailed information for the four node classiﬁcation datasets as follows and summary can be found in Table 3.

Original: • The Cora dataset consists of 2708 scientiﬁc publications classiﬁed into one of seven classes.
Revised: • The Cora dataset consists of 2708 scientiﬁc publications classiﬁed into one of seven classes.

Original: The citation network consists of 5429 links.
Revised: The citation network consists of 5429 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 1433 unique words.
Revised: The dictionary consists of 1433 unique words.

Original: • The Citeseer dataset consists of 3312 scientiﬁc publications classiﬁed into one of six classes.
Revised: • The Citeseer dataset consists of 3312 scientiﬁc publications classiﬁed into one of six classes.

Original: The citation network consists of 4732 links.
Revised: The citation network consists of 4732 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 3703 unique words.
Revised: The dictionary consists of 3703 unique words.

Original: The Pubmed Diabetes dataset consists of 19717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.
Revised: The Pubmed Diabetes dataset consists of 19717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.

Original: The citation network consists of 44338 links.
Revised: The citation network consists of 44338 links.

Original: Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words.
Revised: Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words.

Original: • The Physics dataset consists of 34493 authors, and edges mean whether two authors coauthored a paper.
Revised: • The Physics dataset consists of 34493 authors, and edges mean whether two authors coauthored a paper.

Original: Node features are paper keywords from the author’s papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA following Kim &
Revised: Node features are paper keywords from the author’s papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA following Kim &

Original: Oh (2020).
Revised: Oh (2020).

Original: The split is the 20-per-class/30-per-class/rest.
Revised: The split is the 20-per-class/30-per-class/rest.

Original: The goal of this task is to classify each author’s respective ﬁeld of study.
Revised: The goal of this task is to classify each author’s respective ﬁeld of study.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: F. E XPERIMENTAL I MPLEMENTATION
Revised: F. E XPERIMENTAL I MPLEMENTATION

Original: We use the PyTorch implementation to simulate both inﬁnite-width and ﬁnite-width GCNs:
Revised: We use the PyTorch implementation to simulate both inﬁnite-width and ﬁnite-width GCNs:

Original: • The inﬁnite-width GCNs use part of code from Du et al.
Revised: • The inﬁnite-width GCNs use part of code from Du et al.

Original: (2019) which is originally adopted for graph classiﬁcation.
Revised: (2019) which is originally adopted for graph classiﬁcation.

Original: We redesigned the calculation method of GNTK 2 according to the formula in section 3.1 and used it to process node classiﬁcation tasks.
Revised: We redesigned the calculation method of GNTK 2 according to the formula in section 3.1 and used it to process node classiﬁcation tasks.

Original: • For ﬁnite-width GNNs with DropEdge 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.
Revised: • For ﬁnite-width GNNs with DropEdge 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.

Original: The difference is that, in each experiment, we ﬁx the edge sampling percent as ρ = | V | 2 | E | , which listed in the last column of Table 3.
Revised: The difference is that, in each experiment, we ﬁx the edge sampling percent as ρ = | V | 2 | E | , which listed in the last column of Table 3.

Original: All the codes mentioned above use the MIT license.
Revised: All the codes mentioned above use the MIT license.

Original: All experiments are conducted on with two
Revised: All experiments are conducted on with two

Original: Nvidia Quadro RTX 6000 GPUs.
Revised: Nvidia Quadro RTX 6000 GPUs.

Original: G APPENDIX: A DDITIONAL E XPERIMENT AND D ISCUSSION G. I NFINITELY -W IDE B ACKBONES
Revised: G APPENDIX: A DDITIONAL E XPERIMENT AND D ISCUSSION G. I NFINITELY -W IDE B ACKBONES

Original: In the main paper, we reported the performance of GCNs with ﬁnite-width, here we display more results with respect to inﬁnitely-wide GCNs, as shown in Table 4.
Revised: In the main paper, we reported the performance of GCNs with ﬁnite-width, here we display more results with respect to inﬁnitely-wide GCNs, as shown in Table 4.

Original: We apply Critical DropEdge to inﬁnitely-wide backbones on semi-supervised node classiﬁcation.
Revised: We apply Critical DropEdge to inﬁnitely-wide backbones on semi-supervised node classiﬁcation.

Original: We consider two backbones:
Revised: We consider two backbones:

Original: GCN (Kipf & Welling, 2016a) and JKNet (Xu et al., 2018b).
Revised: GCN (Kipf & Welling, 2016a) and JKNet (Xu et al., 2018b).

Original: For the edge rate, we use the edge preserving percent as ρ ( G ) which shown in Table 3.
Revised: For the edge rate, we use the edge preserving percent as ρ ( G ) which shown in Table 3.

Original: Table 4 summaries the performance on three citation networks.
Revised: Table 4 summaries the performance on three citation networks.

Original: In this table, we report the performance of GCNs with 4/8/32/64 layers.
Revised: In this table, we report the performance of GCNs with 4/8/32/64 layers.

Original: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original deep GNNs in the inﬁnitely-wide case, especially when the model is deep.
Revised: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original deep GNNs in the inﬁnitely-wide case, especially when the model is deep.

Original: [Table]
Revised: [Table]

Original: G. P ERFORMANCE OF S EPARATE B ACKBONES
Revised: G. P ERFORMANCE OF S EPARATE B ACKBONES

Original: The performance results presented in Table 1 is the best performance across different backbones.
Revised: The performance results presented in Table 1 is the best performance across different backbones.

Original: In
Revised: In

Original: Table 5, we show the results of vanilla GNNs, DropEdge and C-DropEdge with different backbones.
Revised: Table 5, we show the results of vanilla GNNs, DropEdge and C-DropEdge with different backbones.

Original: From the table we can conclude the Critical DropEdge can consistently outperform DropEdge and vanilla GNNs.
Revised: From the table we can conclude the Critical DropEdge can consistently outperform DropEdge and vanilla GNNs.

Original: G. C OMPARISON TO D ROP E DGE
Revised: G. C OMPARISON TO D ROP E DGE

Original: While our proposed Critical DropEdge shares the same program with the DropEdge method (Rong et al., 2019)
Revised: While our proposed Critical DropEdge shares the same program with the DropEdge method (Rong et al., 2019)

Original: [Table]
Revised: [Table]

Original: we point out the main difference which can result in a corresponding advantage for our approach concerning the hyper-parameter search space.
Revised: we point out the main difference which can result in a corresponding advantage for our approach concerning the hyper-parameter search space.

Original: C-DropEdge has a ﬁxed and graphdependent edge preserving percent, which implies our hyper-parameter space is smaller than that of
Revised: C-DropEdge has a ﬁxed and graphdependent edge preserving percent, which implies our hyper-parameter space is smaller than that of

Original: DropEdge.
Revised: DropEdge.

Original: The edge preserving percent is one of the most signiﬁcant hyper-parameter, which has a crucial inﬂuence on the ﬁnal performance of the algorithm.
Revised: The edge preserving percent is one of the most signiﬁcant hyper-parameter, which has a crucial inﬂuence on the ﬁnal performance of the algorithm.

Original: In order to obtain good performance,
Revised: In order to obtain good performance,

Original: DropEdge needs to spend a lot of effort to search on the space in terms of this hyper-parameter.
Revised: DropEdge needs to spend a lot of effort to search on the space in terms of this hyper-parameter.

Original: In contrast, our method naturally determines the parameters, thus saving plenty of time.
Revised: In contrast, our method naturally determines the parameters, thus saving plenty of time.

Original: Whether from a theoretical perspective or from the ﬁnal performance, the edge preserving percent we set is reasonable and effective.
Revised: Whether from a theoretical perspective or from the ﬁnal performance, the edge preserving percent we set is reasonable and effective.

Original: G. T RAINABILITY OF GNTK WITH C RITICAL D ROP E DGE
Revised: G. T RAINABILITY OF GNTK WITH C RITICAL D ROP E DGE

Original: We implement deep GCN without/without DropEdge and C-DropEdge to compare the training performance.
Revised: We implement deep GCN without/without DropEdge and C-DropEdge to compare the training performance.

Original: The result is shown in Figure 4.
Revised: The result is shown in Figure 4.

Original: The training of GCN and DropEdge with GCN converge
Revised: The training of GCN and DropEdge with GCN converge

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: slower and converge to a higher training error compared to C-DropEdge with GCN, indicating that
Revised: slower and converge to a higher training error compared to C-DropEdge with GCN, indicating that

Original: C-DropEdge can achive better trainablity compared to GCN and DropEdge with GCN.
Revised: C-DropEdge can achive better trainablity compared to GCN and DropEdge with GCN.

Original: Besides, the convergence results of GNTKs with C-DropEdge is shown in Figure 5.
Revised: Besides, the convergence results of GNTKs with C-DropEdge is shown in Figure 5.

Original: Compared to residual connection, we ﬁnd that the normalized GNTK elements will not converge to a single value, and the convergence rate curve does not depend on the depth.
Revised: Compared to residual connection, we ﬁnd that the normalized GNTK elements will not converge to a single value, and the convergence rate curve does not depend on the depth.

Original: This implies that residual connection mitigates the trainability loss by slow down exponential convergence rate, while C-DropEdge directly change the convergence results.
Revised: This implies that residual connection mitigates the trainability loss by slow down exponential convergence rate, while C-DropEdge directly change the convergence results.