Original: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE
Revised: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.
Revised: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.

Original: However, it is well known that deep
Revised: However, it is well known that deep

Original: GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.
Revised: GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.

Original: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.
Revised: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.

Original: Compared to expressivity, trainability attempts to address a more fundamental question: given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution by gradient descentbased optimizer?
Revised: Compared to expressivity, trainability attempts to address a more fundamental question: given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution by gradient descentbased optimizer?

Original: This work ﬁlls this gap by exploiting the Graph Neural Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.
Revised: This work ﬁlls this gap by exploiting the Graph Neural Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.

Original: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.
Revised: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.

Original: Additionally, we extend our theoretical framework to analyze residual connection-resemble techniques, which are found to be only able to mildly mitigate the exponential decay of trainability.
Revised: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be only able to mildly mitigate the exponential decay of trainability.

Original: To overcome the exponential decay problem more fundamentally, we propose Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, inspired by our theoretical insights on trainability.
Revised: To overcome the exponential decay problem more fundamentally, we propose Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, inspired by our theoretical insights on trainability.

Original: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.
Revised: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, Graph Convolutional Networks (GCNs) have shown incredible abilities to learn node or graph representations and achieved superior performances for various downstream tasks, such as node classiﬁcations (Kipf & Welling, 2016a;
Revised: Recently, Graph Convolutional Networks (GCNs) have shown incredible abilities to learn node or graph representations and achieved superior performances for various downstream tasks, such as node classiﬁcations (Kipf & Welling, 2016a;

Original: Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017), graph classiﬁcations (Xu et al., 2018a; Gilmer et al., 2017; Lee et al., 2019b; Yuan & Ji, 2020), and link predictions (Kipf & Welling, 2016b), etc.
Revised: Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017), graph classiﬁcations (Xu et al., 2018a; Gilmer et al., 2017; Lee et al., 2019b; Yuan & Ji, 2020), and link predictions (Kipf & Welling, 2016b), etc.

Original: However, most GCNs achieve their best with a shallow depth, e.g., 2 or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.
Revised: However, most GCNs achieve their best with a shallow depth, e.g., 2 or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.

Original: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.
Revised: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.

Original: Li et al.
Revised: Li et al.

Original: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.
Revised: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.

Original: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.
Revised: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.

Original: Oono & Suzuki (2019) investigated the expressive power of GNNs using the asymptotic behaviors as the layer goes to inﬁnity.
Revised: Oono & Suzuki (2019) investigated the expressive power of GNNs using the asymptotic behaviors as the layer goes to inﬁnity.

Original: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graphs inherent in the graph spectra.
Revised: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graphs inherent in the graph spectra.

Original: However, it remains elusive how to theoretically understand why deepen GCNs fail to optimize.
Revised: However, it remains elusive how to theoretically understand why deepen GCNs fail to optimize.

Original: Existing theoretical investigations (Xu et al., 2018a; Oono & Suzuki, 2019) on GNNs mainly focus on expressivity which measures how complex the function a network can represent.
Revised: Existing theoretical investigations (Xu et al., 2018a; Oono & Suzuki, 2019) on GNNs mainly focus on expressivity which measures how complex the function a network can represent.

Original: Exploring expressiveness is theoretically convenient, but the corresponding conditions may be broken during the gradient training process, thus leading to inconsistencies between theoretical conclusions and results of trained networks (G¨uhring et al., 2020).
Revised: Exploring expressiveness is theoretically convenient, but the corresponding conditions may be broken during the gradient training process, thus leading to inconsistencies between theoretical conclusions and results of trained networks (G¨uhring et al., 2020).

Original: Compared to the expressivity, trainability attempts to address a more difﬁcult but fundamental perspective of neural networks: how effective a network can be optimized via gradient descent.
Revised: Compared to the expressivity, trainability attempts to address a more difﬁcult but fundamental perspective of neural networks: how effective a network can be optimized via gradient descent.

Original: The advantage of studying trainability is that we can directly know whether the graph neural network can be successfully trained under certain conditions, and to what extent.
Revised: The advantage of studying trainability is that we can directly know whether the graph neural network can be successfully trained under certain conditions, and to what extent.

Original: We are therefore inspired to ask:
Revised: We are therefore inspired to ask:

Original: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?
Revised: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?

Original: • Can we further design an algorithm to deepen GCNs beneﬁt from our theoretical investigation?
Revised: • Can we further design an algorithm to deepen GCNs beneﬁt from our theoretical investigation?

Original: Our answers are yes to both questions.
Revised: Our answers are yes to both questions.

Original: We resort to the inﬁnitely wide multi-layer GCN.
Revised: We resort to the inﬁnitely wide multi-layer GCN.

Original: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed the one hidden layer networks with random weights at initialization (without training) are
Revised: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed the one hidden layer networks with random weights at initialization (without training) are

Original: Gaussian Processes (GPs) in the inﬁnite width limit.
Revised: Gaussian Processes (GPs) in the inﬁnite width limit.

Original: Later, the connection between GPs and multilayer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2017; Matthews et al., 2018) and orthogonal weights (Huang et al., 2020) was reported.
Revised: Later, the connection between GPs and multilayer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2017; Matthews et al., 2018) and orthogonal weights (Huang et al., 2020a) was reported.

Original: Recent trends in Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.
Revised: Recent trends in Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.

Original: In particular, Jacot et al.
Revised: In particular, Jacot et al.

Original: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an NTK.
Revised: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an NTK.

Original: Du et al.
Revised: Du et al.

Original: (2019) formulated
Revised: (2019) formulated

Original: Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNN.
Revised: Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNN.

Original: Prior to the discovery of GNTK, there was little understanding of the non-convexity of graph neural networks, which is analytically intractable.
Revised: Prior to the discovery of GNTK, there was little understanding of the non-convexity of graph neural networks, which is analytically intractable.

Original: In the learning regime of GCN governed by GNTK, the optimization becomes an almost convex problem, making GNTK a promising method to study the trainability of deep GCNs.
Revised: In the learning regime of GCN governed by GNTK, the optimization becomes an almost convex problem, making GNTK a promising method to study the trainability of deep GCNs.

Original: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.
Revised: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.

Original: In particular, we formulate the large-depth asymptotics behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b; Xiao et al., 2019), through which we can analyze the optimization properties.
Revised: In particular, we formulate the large-depth asymptotics behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b; Xiao et al., 2019), through which we can analyze the optimization properties.

Original: We further extend our theoretical framework to residual connection-like techniques and show that these techniques can only help slow down the exponential decay of trainability to some extent, but lack the ability to overcome the problem thoroughly.
Revised: We further extend our theoretical framework to residual connection-like techniques and show that these techniques can only help slow down the exponential decay of trainability to some extent, but lack the ability to overcome the problem thoroughly.

Original: Our contributions are as follows:
Revised: Our contributions are as follows:

Original: • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.
Revised: • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.

Original: We prove that all elements of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.
Revised: We prove that all elements of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.

Original: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.
Revised: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.

Original: • We apply our theoretical analysis to the residual connection-resemble techniques for GCNs.
Revised: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.

Original: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate, but can not fundamentally solve the exponential decay problem.
Revised: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate, but can not fundamentally solve the exponential decay problem.

Original: This result can be used to explain to what extent the recent residual connection-resemble method works.
Revised: This result can be used to explain to what extent the recent residual connection-based method works.

Original: • Our theoretical framework provides comprehensive guidance for the development of deep GCNs.
Revised: • Our theoretical framework provides comprehensive guidance for the development of deep GCNs.

Original: We propose an edge-based sampling method, named Critical DropEdge, to overcome the exponential decay of trainability.
Revised: We propose an edge-based sampling method, named Critical DropEdge, to overcome the exponential decay of trainability.

Original: This graph-adaptive and connectivity-aware method is easy to implement in both ﬁnite-width and inﬁnite-width GNNs.
Revised: This graph-adaptive and connectivity-aware method is easy to implement in both ﬁnite-width and inﬁnite-width GNNs.

Original: Our experiments show using the proposed method can outperform counterparts in the large depth.
Revised: Our experiments show using the proposed method can outperform counterparts in the large depth.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.
Revised: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.

Original: Then we introduce the
Revised: Then we introduce the

Original: Neural Tangent Kernel, making a connection to trainability.
Revised: Neural Tangent Kernel, making a connection to trainability.

Original: Finally, we introduce Graph Convolutional Networks along with our setup and notation.
Revised: Finally, we introduce Graph Convolutional Networks along with our setup and notation.

Original: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION
Revised: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION

Original: We begin by considering a fully-connected network of depth L of widths m l in each layer.
Revised: We begin by considering a fully-connected network of depth L of widths m l in each layer.

Original: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .
Revised: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .

Original: Letting the pre-activations be given by h ( l ) i .
Revised: Letting the pre-activations be given by h ( l ) i .

Original: Then the information propagation in this network is governed by,
Revised: Then the information propagation in this network is governed by,

Original: [Equation]
Revised: [Equation]

Original: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases respectively.
Revised: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases respectively.

Original: Given the parameterization that weights and biases are randomly generated by i.i.d.
Revised: Given the parameterization that weights and biases are randomly generated by i.i.d.

Original: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , · · · , m L − 1 → ∞ .
Revised: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , · · · , m l − 1 → ∞ .

Original: This results from the central limit theorem (CLT).
Revised: This results from the central limit theorem (CLT).

Original: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )
Revised: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .
Revised: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .

Original: According to the signal propagation Equation (1), the covariance matrix or Gaussian Process kernel with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )
Revised: According to the signal propagation Equation (1), the covariance matrix or Gaussian Process kernel with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: σ 2 w E h ∼N (0 , Σ ( l − 1) )
Revised: σ 2 w E h ∼N (0 , Σ ( l − 1) )

Original: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]
Revised: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]

Original: + σ 2 b .
Revised: + σ 2 b .

Original: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2016).
Revised: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2016).

Original: In particular, expressivity describes to what extent two different inputs can be distinguished.
Revised: In particular, expressivity describes to what extent two different inputs can be distinguished.

Original: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )
Revised: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )

Original: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .
Revised: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .

Original: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;
Revised: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;

Original: Schoenholz et al., 2016; Yang & Schoenholz, 2017).
Revised: Schoenholz et al., 2016; Yang & Schoenholz, 2017).

Original: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.
Revised: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.

Original: Given this reason, we focus on the edge of chaos in this work.
Revised: Given this reason, we focus on the edge of chaos in this work.

Original: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z
Revised: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z

Original: [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .

Original: 2. N EURAL T ANGENT K ERNEL AND T RAINABILITY
Revised: 2. N EURAL T ANGENT K ERNEL AND T RAINABILITY

Original: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2016) have focused solely on initialization without training.
Revised: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2016) have focused solely on initialization without training.

Original: Jacot et al.
Revised: Jacot et al.

Original: (2018) took a step further by considering the inﬁnitely-wide network trained with gradient descent.
Revised: (2018) took a step further by considering the inﬁnitely-wide network trained with gradient descent.

Original: Let η be the learning rate, and L be the loss function.
Revised: Let η be the learning rate, and L be the loss function.

Original: The dynamics of gradient ﬂow for parameters θ = vec( { W ( l ) ij , b ( l ) i } )
Revised: The dynamics of gradient ﬂow for parameters θ = vec( { W ( l ) ij , b ( l ) i } )

Original: ∈ R ( (cid:80) l
Revised: ∈ R ( (cid:80) l

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1)) × 1 , the vector of all parameters, is given by,
Revised: 1)) × 1 , the vector of all parameters, is given by,

Original: [Equation]
Revised: [Equation]

Original: Then the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X
Revised: Then the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X

Original: ) ∈ R
Revised: ) ∈ R

Original: nm L × 1 follow,
Revised: nm L × 1 follow,

Original: [Equation]
Revised: [Equation]

Original: (3) where the NTK at time t is deﬁned as,
Revised: (3) where the NTK at time t is deﬁned as,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: In a general case, the NTK varies with the training time, thus providing no substantial insight into the convergence property of neural networks.
Revised: In a general case, the NTK varies with the training time, thus providing no substantial insight into the convergence property of neural networks.

Original: Interestingly, as shown by Jacot et al.
Revised: Interestingly, as shown by Jacot et al.

Original: (2018), the NTK converges to an explicit limiting kernel and does not change during training in the inﬁnite width limit.
Revised: (2018), the NTK converges to an explicit limiting kernel and does not change during training in the inﬁnite width limit.

Original: This leads to a simple but profound result in the case of an MSE loss, L = 12 (cid:107)
Revised: This leads to a simple but profound result in the case of an MSE loss, L = 12 (cid:107)

Original: [Equation]
Revised: [Equation]

Original: F , where Y is the label regarding the input X ,
Revised: F , where Y is the label regarding the input X ,

Original: [Equation]
Revised: [Equation]

Original: 5) where Θ ∞ is the limiting kernel.
Revised: 5) where Θ ∞ is the limiting kernel.

Original: This is the solution of an ordinary differential equation.
Revised: This is the solution of an ordinary differential equation.

Original: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =
Revised: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =

Original: Y .
Revised: Y .

Original: As proved by Hayou et al. (2019b) (Lemma 1), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.
Revised: As proved by Hayou et al. (2019b) (Lemma 1), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.

Original: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.
Revised: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.

Original: 2. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: 2. G RAPH C ONVOLUTIONAL N ETWORKS

Original: We deﬁne an undirected graph as G =
Revised: We deﬁne an undirected graph as G =

Original: ( V, E ) , where V is a collection of nodes while E is a set of edges.
Revised: ( V, E ) , where V is a collection of nodes while E is a set of edges.

Original: We denote the number of nodes in graph G by n = | V | .
Revised: We denote the number of nodes in graph G by n = | V | .

Original: The node features are denoted as h v ∈ R d for each v ∈ V , with d being the dimension of node features.
Revised: The node features are denoted as h v ∈ R d for each v ∈ V , with d being the dimension of node features.

Original: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.
Revised: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.

Original: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.

Original: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.

Original: We deﬁne a propagation unit to be the combination of a R -layer MLP and one aggregation operation.
Revised: We deﬁne a propagation unit to be the combination of a R -layer MLP and one aggregation operation.

Original: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and use superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.
Revised: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and use superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.

Original: L is the total number of propagation units.
Revised: L is the total number of propagation units.

Original: Speciﬁcally, the node representation propagation in GCNs through a multi-layer perceptron (MLP) follows the expression,
Revised: Speciﬁcally, the node representation propagation in GCNs through a multi-layer perceptron (MLP) follows the expression,

Original: [Equation]
Revised: [Equation]

Original: where W ( l ) ( r ) ∈
Revised: where W ( l ) ( r ) ∈

Original: R m l × m l − 1 and b ( l ) ( r ) ∈
Revised: R m l × m l − 1 and b ( l ) ( r ) ∈

Original: R m l are the learnable weights and biases respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node v and its neighbors.
Revised: R m l are the learnable weights and biases respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node v and its neighbors.

Original: Equation (6) describes the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).
Revised: Equation (6) describes the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).

Original: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.
Revised: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.

Original: With regard to the activation function, we focus on both ReLU and tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =
Revised: With regard to the activation function, we focus on both ReLU and tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =

Original: tanh( x ) respectively.
Revised: tanh( x ) respectively.

Original: Without loss of generality, our theoretical framework can handle other common activations.
Revised: Without loss of generality, our theoretical framework can handle other common activations.

Original: By comparison, the GNTK work (Du et al., 2019) only adopted ReLU activation.
Revised: By comparison, the GNTK work (Du et al., 2019) only adopted ReLU activation.

Original:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION
Revised:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION

Original: On the basis of the deﬁnition of NTK (Equation 4), we recursively formulate the propagation of
Revised: On the basis of the deﬁnition of NTK (Equation 4), we recursively formulate the propagation of

Original: GNTK in the inﬁnite-width limit.
Revised: GNTK in the inﬁnite-width limit.

Original: Since the information propagation in a GCN is built on two operations: aggregation (Equation 6) and non-linear transformation (Equation 7), the corresponding formulas of GNTK are expressed as follows,
Revised: Since the information propagation in a GCN is built on two operations: aggregation (Equation 6) and non-linear transformation (Equation 7), the corresponding formulas of GNTK are expressed as follows,

Original: [Equation]
Revised: [Equation]

Original: The two equations above correspond to the aggregation operation and MLP transformation respectively.
Revised: The two equations above correspond to the aggregation operation and MLP transformation respectively.

Original: To compute the GNTK with respect the to depth, the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )
Revised: To compute the GNTK with respect the to depth, the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )

Original: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .
Revised: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .

Original: According to the CLT, the node representation h ( l ) ( r ) ( u ) is Gaussian distributed in the inﬁnite width limit.
Revised: According to the CLT, the node representation h ( l ) ( r ) ( u ) is Gaussian distributed in the inﬁnite width limit.

Original: Applying this result to Equation (6) and (7) the resulting covariance matrix is composed of two parts,
Revised: Applying this result to Equation (6) and (7) the resulting covariance matrix is composed of two parts,

Original: [Equation]
Revised: [Equation]

Original: The ﬁrst equation results from the aggregation operation.
Revised: The ﬁrst equation results from the aggregation operation.

Original: Meanwhile, the second equation corresponds to the R -times non-linear transformations, where z 1 , z 2 are drawn from a 2-dimensional
Revised: Meanwhile, the second equation corresponds to the R -times non-linear transformations, where z 1 , z 2 are drawn from a 2-dimensional

Original: Gaussian distribution of the previous MLP layer, and the variance ˜Σ ( l ) ( r − 1)
Revised: Gaussian distribution of the previous MLP layer, and the variance ˜Σ ( l ) ( r − 1)

Original: ∈ R 2 × 2 is across u, u (cid:48) .
Revised: ∈ R 2 × 2 is across u, u (cid:48) .

Original: 3. T RAINABILITY IN THE L ARGE D EPTH
Revised: 3. T RAINABILITY IN THE L ARGE D EPTH

Original: We aim to characterize the behavior of Θ ( l ) ( r ) ( G ) =
Revised: We aim to characterize the behavior of Θ ( l ) ( r ) ( G ) =

Original: Θ ( l ) ( r ) ( V , V ) ∈ R n × n , where we denote
Revised: Θ ( l ) ( r ) ( V , V ) ∈ R n × n , where we denote

Original: V =
Revised: V =

Original: [ v 1 , v 2 , . .
Revised: [ v 1 , v 2 , . .

Original: . , v n ] , as the depth tends to inﬁnity l → ∞ .
Revised: . , v n ] , as the depth tends to inﬁnity l → ∞ .

Original: From the GNTK formulation, both aggregation (Equation 8) and transformation (Equation 9) contribute simultaneously to the ﬁnal limiting result.
Revised: From the GNTK formulation, both aggregation (Equation 8) and transformation (Equation 9) contribute simultaneously to the ﬁnal limiting result.

Original: Taking them into consideration, we derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is as follows:
Revised: Taking them into consideration, we derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is as follows:

Original: Theorem 1 (Convergence Rate of the GNTK) .
Revised: Theorem 1 (Convergence Rate of the GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) =
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) =

Original: A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and
Revised: A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and

Original: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v,(cid:126)v (cid:48) ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G) T (cid:0) Rl(cid:126)v
Revised: Then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48) ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) [Equation] (cid:126)π ( G

Revised: [Equation]

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ Cα l .
Revised: (cid:0) [Equation] (cid:126)v [Equation]

Original: The proof sketch follows a divide-and-conquer manner.
Revised: The proof sketch follows a divide-and-conquer manner.

Original: In particular, we ﬁrst analyze the network with the only aggregation and prove that A ( G ) is a Markov transition matrix.
Revised: In particular, we ﬁrst analyze the network with the only aggregation and prove that A ( G ) is a Markov transition matrix.

Original: Then we formulate the behavior of MLP in the large depth based on Hayou et al.
Revised: Then we formulate the behavior of MLP in the large depth based on Hayou et al.

Original: (2019b).
Revised: (2019b).

Original: Finally, we arrive at the ﬁnal result by considering the two operations.
Revised: Finally, we arrive at the ﬁnal result by considering the two operations.

Original: We leave the complete proof in Appendix B.
Revised: We leave the complete proof in Appendix B.

Original: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.
Revised: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.

Original: As depth goes to inﬁnity, all elements in the GNTK converge to a unique quantity at an exponential rate.
Revised: As depth goes to inﬁnity, all elements in the GNTK converge to a unique quantity at an exponential rate.

Original: We thus have Θ ( L ) ( R ) ( V , V ) ≈
Revised: We thus have Θ ( L ) ( R ) ( V , V ) ≈

Original: (cid:126)π ( G ) T ( RL(cid:126)v ) 1 n × n as L → ∞ , where 1 n × n is the ( n × n )- dimensional matrix whose elements are one.
Revised: (cid:126)π ( G ) T ( RL(cid:126)v ) 1 n × n as L → ∞ , where 1 n × n is the ( n × n )- dimensional matrix whose elements are one.

Original: The exponential convergence rate of Θ ( L ) ( R ) ( V , V ) implies that the trainability of inﬁnitely wide GCNs degenerate dramatically, as shown in following corollary.
Revised: The exponential convergence rate of Θ ( L ) ( R ) ( V , V ) implies that the trainability of inﬁnitely wide GCNs degenerate dramatically, as shown in following corollary.

Original: Corollary 1 (Trainability of Ultra-Wide GCNs) .
Revised: Corollary 1 (Trainability of Ultra-Wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth
Revised: Consider a GCN of the form (6) and (7), with depth

Original: L , non-linear transformations number R , an MSE loss, and a Lipchitz activation φ ( x ) , trained with gradient descent on a node classiﬁcation task.
Revised: L , non-linear transformations number R , an MSE loss, and a Lipchitz activation φ ( x ) , trained with gradient descent on a node classiﬁcation task.

Original: Then the output function follows,
Revised: Then the output function follows,

Original: [Equation]
Revised: [Equation]

Original: where Y is the label with respect to node V .
Revised: where Y is the label with respect to node V .

Original: Then,there exists a positive integer L 0 such that, Θ ( L ) ( R ) ( V , V ) is singular when L > L 0 .
Revised: Then, Θ ( L ) ( R ) ( V , V ) is singular when L → ∞ .

Original: Moreover, there exist a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .
Revised: Moreover, there exist a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .

Original: According to the above corollary, as L → ∞ , the GNTK matrix will become a singular matrix.
Revised: According to the above corollary, as L → ∞ , the GNTK matrix will become a singular matrix.

Original: This would lead to a discrepancy between output f t ( V ) and label Y , which means GNTK loses the ability to ﬁt the label.
Revised: This would lead to a discrepancy between output f t ( V ) and label Y , which means GNTK loses the ability to ﬁt the label.

Original: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.
Revised: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.

Original:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION
Revised:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION

Original: We have characterized the trainability of vanilla GCNs through GNTKs and show that the trainability of ultra-wide GCNs drops at an exponential rate.
Revised: We have characterized the trainability of vanilla GCNs through GNTKs and show that the trainability of ultra-wide GCNs drops at an exponential rate.

Original: Recently, considerable efforts have been made to deepen GCNs, of which residual connection-resemble techniques are widely applied to resolve the over-smoothing problem (Li et al., 2019).
Revised: Recently, considerable efforts have been made to deepen GCNs, of which residual connection-based techniques are widely applied to resolve the oversmoothing problem (Li et al., 2019).

Original: We apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.
Revised: We apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.

Original: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,
Revised: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,

Original: [Equation]
Revised: [Equation]

Original:  where 0 < δ < 1 , and C > 0 .
Revised: (12) where 0 < δ < 1 , and C > 0 .

Original: Taking Equation (12) as a new aggregation operation matrix, (cid:126) Θ ( l ) ( G ) =
Revised: Taking Equation (12) as a new aggregation operation matrix, (cid:126) Θ ( l ) ( G ) =

Original: ˜
Revised: ˜

Original: A ( G ) (cid:126) Θ ( l − 1) ( G ) , where ˜ A ( G )
Revised: A ( G ) (cid:126) Θ ( l − 1) ( G ) , where ˜ A ( G )

Original: =
Revised: =

Original: (1 − δ )
Revised: (1 − δ )

Original: A ( G ) + δI .
Revised: A ( G ) + δI .

Original: We prove that ˜
Revised: We prove that ˜

Original: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix
Revised: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix

Original: A ( G ) .
Revised: A ( G ) .

Original: Theorem 2 (Convergence rate for residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate for residual connection in aggregation) .

Original: Consider a GNTK of the form (12) with non-linear transformation of the form (7).
Revised: Consider a GNTK of the form (12) with non-linear transformation of the form (7).

Original: Then with a stationary vector ˜ π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <  α < 1 and C > 0 , and constant matrix (cid:126)v and (cid:126)v (cid:48) depending on
Revised: A ( G ) , there exist constants 0 < ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Revised: Here ˜ α is roughly larger than α appeared in Theorem 1, because the convergence rate depends on the second largest eigenvalue of transition matrix.

Original: The  proof of theorem 2  can be found in Appendix C.1.
Revised: The detailed the proof of theorem 2 and the relationship between convergence rate and second largest eigvenvalue can be found in Appendix C.1.

Original: This theorem implies a residual connection in aggregation can slow down the convergence rate, which is consistent with the empirical observations that residual connection-resemble methods can deepen GCNs.
Revised: This theorem implies a residual connection in aggregation can slow down the convergence rate, which is consistent with the empirical observations that residual connection-based methods can deepen GCNs.

Original: Then we consider residual connection only applied on non-linear transformations (MLP).
Revised: Then we consider residual connection only applied on non-linear transformations (MLP).

Original: In this case, the recursive equation for the corresponding GNTK can be expressed as, 
Revised: In this case, the recursive equation for the corresponding GNTK can be expressed as, [Equation]

Original: [Equation]
Revised: [Equation]

Original: This formula is similar to the vanilla GNTK in the inﬁnitely wide limit.
Revised: This formula is similar to the vanilla GNTK in the inﬁnitely wide limit.

Original: Only an additional residual term appears according to the residual connection.
Revised: Only an additional residual term appears according to the residual connection.

Original: It turns out that this term may not help slow down the convergence rate for non-linear transformation.
Revised: It turns out that this term may not help slow down the convergence rate for non-linear transformation.

Original: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .

Original: Consider a GNTK the form (8) and (13).
Revised: Consider a GNTK the form (8) and (13).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant matrix (cid:126)v and (cid:126)v (cid:48) depending on R , such that, (cid:12)(cid:12)
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on R , such that, (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Revised: Note that α in this theorem is the same that in Theorem 1.

Original: The proof of the above theorem can be found in Appendix C.2.
Revised: The proof of the above theorem can be found in Appendix C.2.

Original: Theorem 3 demonstrates that adding residual connection in MLP cannot even reduce the convergence speed of trainability.
Revised: Theorem 3 demonstrates that adding residual connection in MLP cannot even reduce the convergence speed of trainability.

Original: Finally, we consider the residual connection applied to both aggregation and non-linear transformation simultaneously:
Revised: Finally, we consider the residual connection applied to both aggregation and non-linear transformation simultaneously:

Original: Corollary 2 (Convergence rate for GNTK with residual connection in aggregation and transformation) .
Revised: Corollary 2 (Convergence rate for GNTK with residual connection in aggregation and transformation) .

Original: Consider a GNTK of the form (12) and (13).
Revised: Consider a GNTK of the form (12) and (13).

Original: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 <  α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)
Revised: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 < ˜ α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Original: ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12)
Revised: ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12)

Original: [Equation]
Revised: [Equation]

Original: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE
Revised: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE

Original: Residual connection is a technique designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay problem.
Revised: Residual connection is a technique designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay problem.

Original: To better solve the problem of trainability loss, we need to look deeper into the cause of the problem, which is the transition matrix corresponding to the aggregation operation.
Revised: To better solve the problem of trainability loss, we need to look deeper into the cause of the problem, which is the transition matrix corresponding to the aggregation operation.

Original: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected, which is a typical attribute for graphs used on both node and graph classiﬁcation tasks.
Revised: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected, which is a typical attribute for graphs used on both node and graph classiﬁcation tasks.

Original: Therefore, breaking the connectivity condition is a promising direction.
Revised: Therefore, breaking the connectivity condition is a promising direction.

Original: Interestingly, the critical percolation theory (Huang et al., 2018; Stauffer & Aharony, 2018) provides useful insights to better solve the exponential decay problem.
Revised: Interestingly, the critical percolation theory (Huang et al., 2018; Stauffer & Aharony, 2018) provides useful insights to better solve the exponential decay problem.

Original: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.
Revised: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.

Original: A random graph ˆ G is achieved by randomly and uniformly dropping some edges from the complete graph with the edge probability as p = | E | E t , where | E | is the number of edges retained in the random graph.
Revised: A random graph ˆ G is achieved by randomly and uniformly retaining some edges from the complete graph with the edge probability as p = | E | E t , where | E | is the number of edges retained in the random graph.

Original: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c
Revised: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c

Original: = 1 / ( n − 1) (Erd¨os & R´enyi, 2011).
Revised: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).

Original: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there is anopen path from some ﬁxed point(say the origin) to a distance decreases polynomially:
Revised: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there is an path from some ﬁxed point to a distance decreases polynomially:

Original: Proposition 1 (Critical Connectivity in Random Graph (Erd¨os & R´enyi, 2011)) .
Revised: Proposition 1 (Critical Connectivity in Random Graph (Erd˝os & R´enyi, 1961)) .

Original: Suppose a random graph ˆ G has n nodes with a constant edge probability p .
Revised: Suppose a random graph ˆ G has n nodes with a constant edge probability p .

Original: (1) If p < p c , then almost every random graph is such its largest component 1 has O (log n ) ; (2) If p > p c , the random graph has a giant component of (1 − α p + o (1)) n , where α p < 1 ; (3) If p = p c , then the maximal size of a component of almost every graph has order n 2 / 3 .
Revised: (1) If p < p c , then almost every random graph is such its largest component 1 has O (log n ) ; (2) If p > p c , the random graph has a giant component of (1 − α p + o (1)) n , where α p < 1 ; (3) If p = p c , then the maximal size of a component of almost every graph has order n 2 / 3 .

Original: [Figure]
Revised: [Figure]

Original: The lemma above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate, thus can solve the problem of exponentially dropping trainability.
Revised: The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate, thus can solve the problem of exponentially dropping trainability.

Original: Inspired by this result, we design a graph-dependent and connectivity-aware sampling algorithm.
Revised: Inspired by this result, we design a graph-dependent and connectivity-aware sampling algorithm.

Original: In particular, given a graph G , we randomly drop some edges and retain the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.
Revised: In particular, given a graph G , we randomly drop some edges and retain the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.

Original: Note that a related work called
Revised: Note that a related work called

Original: DropEdge (Rong et al., 2019), which randomly removes a certain number of edges from the input graph.
Revised: DropEdge (Rong et al., 2019), which randomly removes a certain number of edges from the input graph.

Original: Different from them, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | .
Revised: Different from them, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | .

Original: On the contrary, DropEdge method may choose the edge probability as p < p c where information can only be passed to a distance of O (log n ) in the graph, or p > p c in which the exponential trainability problem just exists.
Revised: On the contrary, DropEdge method may choose the edge probability as p < p c where information can only be passed to a distance of O (log n ) in the graph, or p > p c in which the exponential trainability problem just exists.

Original: Our method is inspired by the critical percolation theory and thus it is named as Critical DropEdge.
Revised: Our method is inspired by the critical percolation theory and thus it is named as Critical DropEdge.

Revised: A further discussion of the relationship between Proposition 1 and the trainability of corresponding GNTK can be found in the Appendix D.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: [Figure]
Revised: [Figure]

Original: In this section, we illustrate empirically the theoretical results obtained in Section 3 and present the performance of the proposed Critical DropEdge on node classiﬁcation.
Revised: In this section, we illustrate empirically the theoretical results obtained in Section 3 and present the performance of the proposed Critical DropEdge on node classiﬁcation.

Original: Details of these datasets are summarized in Table 3 in Appendix D.1.
Revised: Details of these datasets are summarized in Table 3 in Appendix E.1.

Original: We use a bioinformatics dataset (i.e., MUTAG) to verify the theoretical results for GNTK.
Revised: We use a bioinformatics dataset (i.e., MUTAG) to verify the theoretical results for GNTK.

Original: The remaining three citation network datasets (i.e., Cora, Citeseer, and
Revised: The remaining three citation network datasets (i.e., Cora, Citeseer, and

Original: Pubmed) are used to verify the trainability of wide
Revised: Pubmed) are used to verify the trainability of wide

Original: GCNs and the performance of the proposed Critical Dropdege on node classiﬁcation tasks.
Revised: GCNs and the performance of the proposed Critical Dropdege on node classiﬁcation tasks.

Original: 5. C ONVERGENCE R ESULTS OF GNTK S
Revised: 5. C ONVERGENCE R ESULTS OF GNTK S

Original: Theorems 1, 2, and 3 provide a theoretical convergence rate for the (residual) GNTK.
Revised: Theorems 1, 2, and 3 provide a theoretical convergence rate for the (residual) GNTK.

Original: We demonstrate the corresponding numerical veriﬁcation in
Revised: We demonstrate the corresponding numerical veriﬁcation in

Original: Figure 1.
Revised: Figure 1.

Original: We select a graph from the MUTAG dataset and generate the GNTK with ReLU activation, with R = 3 , and L = 300 .
Revised: We select a graph from the MUTAG dataset and generate the GNTK with ReLU activa- tion, with R = 3 , and L = 300 .

Original: Figure 1 (a,b,c) show all elements of normalized GNTKs converge to an identical value as the depth goes larger.
Revised: Figure 1 (a,b,c) show all elements of normalized GNTKs converge to an identical value as the depth goes larger.

Original: Figure 1 (d,e,f) further illustrate the convergence rate of the GNTK is exponential, as reﬂected by theorems.
Revised: Figure 1 (d,e,f) further illustrate the convergence rate of the GNTK is exponential, as reﬂected by theorems.

Original: By comparing the convergence rate, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with Theorem 2.
Revised: By comparing the convergence rate, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with Theorem 2.

Original: 5. T RAINABILITY OF W IDE GCN S
Revised: 5. T RAINABILITY OF W IDE GCN S

Original: We further examine whether ultra-wide GCNs can be trained successfully on node classiﬁcation.
Revised: We further examine whether ultra-wide GCNs can be trained successfully on node classiﬁcation.

Original: We conduct experiments on a GCN (Kipf & Welling, 2016a), where we apply a width of 1000 at each hidden layer and the depth ranging from 2 to 29 .
Revised: We conduct experiments on a GCN (Kipf & Welling, 2016a), where we apply a width of 1000 at each hidden layer and the depth ranging from 2 to 29 .

Original: Figure 2 shows the training and test accuracy on various datasets after 300 training epochs.
Revised: Figure 2 shows the training and test accuracy on various datasets after 300 training epochs.

Original: These results show the dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation as revealed by Corollary 1.
Revised: These results show the dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation as revealed by Corollary 1.

Original: 5. E VALUATION OF C RITICAL D ROP E DGE
Revised: 5. E VALUATION OF C RITICAL D ROP E DGE

Original: We apply Critical DropEdge to both inﬁnite-width and ﬁnite-width GCNs on node classiﬁcation with a semi-supervised setting.
Revised: We apply Critical DropEdge to both inﬁnite-width and ﬁnite-width GCNs on node classiﬁcation with a semi-supervised setting.

Original: For inﬁnite-width GCNs, we consider two backbones: GCN (Kipf & Welling, 2016a) and
Revised: For inﬁnite-width GCNs, we consider two backbones: GCN (Kipf & Welling, 2016a) and

Original: JKNet (Xu et al., 2018b), while we use three backbones (GCN, JKNet, IncepGCN (Rong et al., 2019)) for ﬁnite-width GCNs.
Revised: JKNet (Xu et al., 2018b), while we use three backbones (GCN, JKNet, IncepGCN (Rong et al., 2019)) for ﬁnite-width GCNs.

Original: The corresponding results for the inﬁnite-width
Revised: The corresponding results for the inﬁnite-width

Original: GCNs can be found in Appendix E.1.
Revised: GCNs can be found in Appendix F.1.

Original: As for ﬁnite-width GCNs we use the hyper-parameter reported in Rong et al.
Revised: As for ﬁnite-width GCNs, we use the hyper-parameter reported in Rong et al.

Original: (2019) to obtain the results for DropEdge.
Revised: (2019) to obtain the results for DropEdge.

Original: On the other hand, we perform a random hyper-parameter search and ﬁx the edge preserving percent as ρ ( G ) =
Revised: On the other hand, we perform a random hyper-parameter search and ﬁx the edge preserving percent as ρ ( G ) =

Original: | V | 2 | E | for C-DropEdge.
Revised: | V | 2 | E | for C-DropEdge.

Revised: In addition, we introduce a baseline method name DGN (Zhou et al., 2020) which is a normalized-based approach.

Revised: The backbone of DGN includes GCN (Kipf & Welling, 2016a) and GAT (Veli ˇ ckovi

Original:  Table 1 summarizes the performance on three citation network datasets  with ﬁnite-width GCNs.
Revised: ´ c et al., 2017). Table 1 summarizes the performance on three citation network datasets and one co-author network dataset with ﬁnite-width GCNs.

Revised: All the experiments are conducted in the semi-supervised setting, and more details regarding dataset can be found in Appendix E.1.

Original: In this table, we report the performance of GCNs with 4/8/16/32 layers and implementation details can be found in Appendix D.2.
Revised: In this table, we report the performance of GNNs with 4/8/16/32 layers and implementation details can be found in Appendix E.2.

Revised: In particular, we report the best performance across different backbones for DropEdge,

Revised: C-DropEdge and DGN, and leave separate results with different backbones in Appendix F.2.

Original: For each hyperparameter, we run 10 times and report the mean and standard variance.
Revised: For each hyperparameter, we run 10 times and report the mean and standard variance.

Original: It is shown that C-DropEdge consistently outperforms vanilla and DropEdge counterparts, with both inﬁnite-width and ﬁnite-width, especially when the model is deep.
Revised: It is shown that C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.

Original: Besides, our C-DropEdge method can achieve smaller error bars, indicating that our method is more robust than DropEdge.
Revised: Besides, our C-DropEdge method can achieve smaller error bars, indicating that our method is more robust than DropEdge.

Original: Finally, we perform an empirical study of whether the proposed drop rate value is indeed optimal or close to optimal to strengthen our results.
Revised: Finally, we perform an empirical study of whether the proposed drop rate value is indeed optimal or close to optimal to strengthen our results.

Original: To this end, we conducted experiments on dropping edges at various rates (sampling percent) and veriﬁed that the rate of Critical DropEdge can achieve the results closed to optimal, as shown in Table 2.
Revised: To this end, we conducted experiments on dropping edges at various rates (sampling percent) and veriﬁed that the rate of Critical DropEdge can achieve the results closed to optimal, as shown in Table 2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Tangent Kernel.
Revised: Neural Tangent Kernel.

Original: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.
Revised: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.

Original: In the inﬁnite width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018;
Revised: In the inﬁnite width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018;

Original: Lee et al., 2019a; Arora et al., 2019; Yang, 2019; Huang & Yau, 2019; Allen-Zhu et al., 2019; Du et al., 2018; Zou et al., 2018).
Revised: Lee et al., 2019a; Arora et al., 2019; Yang, 2019; Huang & Yau, 2019; Allen-Zhu et al., 2019; Du et al., 2018; Zou et al., 2018).

Original: Besides, NTK has been applied to various architecture and brought a wealth of results, such as orthogonal initialization (Huang et al., 2020), convolutions (Arora et al., 2019), graph neural networks (Du et al., 2019), attention (Hron et al., 2020), and see (Yang, 2020) for a summary.
Revised: Besides, NTK has been applied to various architecture and brought a wealth of results, such as orthogonal initialization (Huang et al., 2020a), convolutions (Arora et al., 2019), graph neural networks (Du et al., 2019), attention (Hron et al., 2020), and see (Yang, 2020) for a summary.

Original: As for graph networks, GNTK helps us understand how GNN learn a class of smooth functions on graphs (Du et al., 2019) and how GNN extrapolate differently from multi-layer perceptron (Xu et al., 2020).
Revised: As for graph networks, GNTK helps us understand how GNN learn a class of smooth functions on graphs (Du et al., 2019) and how GNN extrapolate differently from multi-layer perceptron (Xu et al., 2020).

Original: Deep Graph Neural Networks.
Revised: Deep Graph Neural Networks.

Original: Since GNNs suffer from over-smoothing problem, a large and growing body of literature has made efforts deepening graph neural networks.
Revised: Since GNNs suffer from over-smoothing problem, a large and growing body of literature has made efforts deepening graph neural networks.

Original: There is a line of methods that resort to residual connection to keep their feature expressivity in deep layers.
Revised: There is a line of methods that resort to residual connection to keep their feature expressivity in deep layers.

Original: Xu et al.
Revised: Xu et al.

Original:  (2018b) used a skip connection method that combined with node representations from different neighborhood ranges to preserve the locality of the node representations.
Revised: [Table] [Table] (2018b) used a skip connection method that combined with node representations from different neighborhood ranges to preserve the locality of the node representations.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2018) derived a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.
Revised: (2018) derived a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.

Original: Other similar works using residual connection method could also be seen in (Li et al., 2019;
Revised: Other similar works using residual connection method could also be seen in (Li et al., 2019;

Original: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).
Revised: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).

Revised: Another line of approaches make attempt to tackle the issue by regularization mechanism, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2019), and group normalization (Zhou et al., 2020).

Original: More recently, a series of latest works (Zeng et al., 2020;
Revised: Recently, a series of latest works (Loukas, 2019; Zeng et al., 2020; Li et al., 2020;

Original: Zhou et al., 2020; Li et al., 2020) explored the underlying reasons for performance degra- [Table] [Table] dation towards mitigation solutions, but the trainability of deep GNNs has not been theoretically investigated.
Revised: Cong et al., 2021; Huang et al., 2020b) explored the underlying reasons for performance degradation towards mitigation solutions.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.
Revised: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.

Original: We prove that the trainability drops at an exponential rate due to the aggregation operation.
Revised: We prove that the trainability drops at an exponential rate due to the aggregation operation.

Original: Furthermore, we apply our theoretical framework to investigate the extent to which residual connection-resemble techniques could deepen GCNs.
Revised: Furthermore, we apply our theoretical framework to investigate the extent to which residual connection -based techniques could deepen GCNs.

Original: We demonstrate that these techniques can only slow down the decay speed, but are unable to solve the exponential decay problem in essence.
Revised: We demonstrate that these techniques can only slow down the decay speed but are unable to solve the exponential decay problem in essence.

Original: To overcome the dropping trainability problem we further propose Critical
Revised: To overcome the dropping trainability problem we further propose Critical

Original: DropEdge illuminated by our theoretical framework.
Revised: DropEdge illuminated by our theoretical framework.

Original: The experimental results conﬁrm that our method can address the trainability problems of deep GCNs.
Revised: The experimental results conﬁrm that our method can address the trainability problems of deep GCNs.

Original: Future research directions include designing a critical node-centered method, so as to make better use of node information.
Revised: Future research directions include designing a critical node-centered method so as to make better use of node information

Original:  R EPRODUCIBILITY S TATEMENT
Revised:  R EPRODUCIBILITY S TATEMENT

Original: To ensure the results and conclusions of our paper are reproducible, we make the following efforts:
Revised: To ensure the results and conclusions of our paper are reproducible, we make the following efforts:

Original: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 3 and 4; Appendix B and C.
Revised: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 3 and 4; Appendix B and C.

Original: Experimentally, we provide our code, and instructions needed to reproduce the main experimental results.
Revised: Experimentally, we provide our code, and instructions needed to reproduce the main experimental results.

Original: And we specify all the training and implementation details in Section 5 and Appendix D and
Revised: And we specify all the training and implementation details in Section 5 and Appendix E and

Original: E. Besides, we independently run experiments 10 times and report the mean and standard deviation
Revised: F. Besides, we independently run experiments 10 times and report the mean and standard deviation

Original: A APPENDIX: A G ENERAL GCN CONSIDERED IN THIS WORK
Revised: A APPENDIX: A G ENERAL GCN CONSIDERED IN THIS WORK

Original: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.

Original: Figure 3 illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 illustrates an overview of the information propagation in a general GCN.

Original: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK
Revised: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK

Original: In theorem 1 we demonstrate that the matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.
Revised: In theorem 1 we demonstrate that the matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.

Original: To this end, we block the non-linear transformation by setting R = 0 .
Revised: To this end, we block the non-linear transformation by setting R = 0 .

Original: Then and the propagation of GNTK can be expressed as,
Revised: Then and the propagation of GNTK can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite Equation (14) as the following format,
Revised: In order to facilitate calculation, we rewrite Equation (14) as the following format,

Original: [Equation]
Revised: [Equation]

Original: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and the matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.
Revised: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and the matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.

Original: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( V , V ) in the following lemma,
Revised: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( V , V ) in the following lemma,

Original: Lemma 1 (Convergence of Aggregation) .
Revised: Lemma 1 (Convergence of Aggregation) .

Original: Assume R = 0 , then
Revised: Assume R = 0 , then

Original: [Equation]
Revised: [Equation]

Original: where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) = (cid:126)π ( G ) .
Revised: where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) = (cid:126)π ( G ) .

Original: Proof.
Revised: Proof.

Original: When R = 0 , Equation (14) reduces to,
Revised: When R = 0 , Equation (14) reduces to,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite the above equation in the format of matrix,
Revised: In order to facilitate calculation, we rewrite the above equation in the format of matrix,

Original: [Equation]
Revised: [Equation]

Original: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.
Revised: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.

Original: Thus the matrix operation A ( l ) ( G ) ∈
Revised: Thus the matrix operation A ( l ) ( G ) ∈

Original: R n 2 × n 2 .
Revised: R n 2 × n 2 .

Original: It is obvious that,
Revised: It is obvious that,

Original: [Equation]
Revised: [Equation]

Original: This implies that the aggregate operation is the same for each layer.
Revised: This implies that the aggregate operation is the same for each layer.

Original: The next step is to prove A ( G ) is a stochastic matrix (transition matrix):
Revised: The next step is to prove A ( G ) is a stochastic matrix (transition matrix):

Original: [Equation]
Revised: [Equation]

Original: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrix,
Revised: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrix,

Original: [Equation]
Revised: [Equation]

Original: where B ( G ) , C ( G ) ∈ R n × n .
Revised: where B ( G ) , C ( G ) ∈ R n × n .

Original: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor
Revised: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor

Original: [Equation]
Revised: [Equation]

Original: (2) The element of matrix C ( G ) is determined by whether there is a edge between two vertexes,
Revised: (2) The element of matrix C ( G ) is determined by whether there is a edge between two vertexes,

Original: [Equation]
Revised: [Equation]

Original: where ˜ δ ij = 1 if i ==
Revised: where ˜ δ ij = 1 if i ==

Original: j or there is edge between vertex i and j , else ˜ δ ij = 0 .
Revised: j or there is edge between vertex i and j , else ˜ δ ij = 0 .

Original: We then use the property of matrix B and C before Kronecker product,
Revised: We then use the property of matrix B and C before Kronecker product,

Original: [Equation]
Revised: [Equation]

Original: According to the deﬁnition of Kronecker product, we have,
Revised: According to the deﬁnition of Kronecker product, we have,

Original: [Equation]
Revised: [Equation]

Original: where i = a + ( c − 1) n , and j = b + ( d − 1) n .
Revised: where i = a + ( c − 1) n , and j = b + ( d − 1) n .

Original: So far, we have proved that A ( G ) is a stochastic matrix.
Revised: So far, we have proved that A ( G ) is a stochastic matrix.

Original: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, that does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution which is also a eigenvector of the probability matrix, associated with eigenvalue 1:
Revised: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, that does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution which is also a eigenvector of the probability matrix, associated with eigenvalue 1:

Original: [Equation]
Revised: [Equation]

Original: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.
Revised: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.

Original: Thus the convergence rate is governed by the second largest eigenvalue.
Revised: Thus the convergence rate is governed by the second largest eigenvalue.

Original: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,
Revised: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,

Original: [Equation]
Revised: [Equation]

Original: where Π( G ) = 
Revised: where Π( G ) = 

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.
Revised: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.

Original: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,
Revised: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈
Revised: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈

Original: R nn (cid:48)
Revised: R nn (cid:48)

Original: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertexes in graph G (cid:48) .
Revised: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertexes in graph G (cid:48) .

Original: B. C ONVERGENCE OF MLP GPK
Revised: B. C ONVERGENCE OF MLP GPK

Original: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.
Revised: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.

Original: By doing so, we consider a network with only non-linear transformation, which is known as a pure MLP.
Revised: By doing so, we consider a network with only non-linear transformation, which is known as a pure MLP.

Original: This leads to Σ ( l ) ( r ) ( u, u (cid:48) )
Revised: This leads to Σ ( l ) ( r ) ( u, u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.
Revised: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.

Original: And we rewrite the propagation function for GPK as follows,
Revised: And we rewrite the propagation function for GPK as follows,

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,
Revised: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,

Original: [Equation]
Revised: [Equation]

Original: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.
Revised: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.

Original: In particular, we set the value of hyper-parameters to satisfy,
Revised: In particular, we set the value of hyper-parameters to satisfy,

Original: [Equation]
Revised: [Equation]

Original: ) where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: ) where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, Equation (18) requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, Equation (18) requires σ 2 w = 2 and σ 2 b = 0 .

Original: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,
Revised: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 ( Proposition 1 and 3 in Hayou et al.
Revised: Lemma 2 ( Proposition 1 and 3 in Hayou et al.

Original: (2019a)) .
Revised: (2019a)) .

Original: Assume a network without aggregation, i.e., L = 0 , with a Lipschitz nonlinearity φ , then,
Revised: Assume a network without aggregation, i.e., L = 0 , with a Lipschitz nonlinearity φ , then,

Original: • φ ( x ) =
Revised: • φ ( x ) =

Original: ( x ) + , 1 −
Revised: ( x ) + , 1 −

Original: [Equation]
Revised: [Equation]

Original: ( u, u (cid:48)
Revised: ( u, u (cid:48)

Original: [Equation]
Revised: [Equation]

Original: → ∞ • φ ( x ) = tanh( x ) , 1 − C r ( u, u (cid:48) ) ∼ βr as r → ∞ where β
Revised: → ∞ • φ ( x ) = tanh( x ) , 1 − C r ( u, u (cid:48) ) ∼ βr as r → ∞ where β

Original: = 2 (cid:82)
Revised: = 2 (cid:82)

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, one is diagonal element and the other is non-diagonal element:
Revised: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, one is diagonal element and the other is non-diagonal element:

Original: [Equation]
Revised: [Equation]

Original: where u=
Revised: where u=

Original: [Equation]
Revised: [Equation]

Original: Σ ( r − 1) ( u, u ) z 1 , and u
Revised: Σ ( r − 1) ( u, u ) z 1 , and u

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: − C
Revised: − C

Original: [Equation]
Revised: [Equation]

Original: u, u
Revised: u, u

Original: [Equation]
Revised: [Equation]

Original: z(cid:19) , with C r − 1 ( u, u (cid:48) )
Revised: z(cid:19) , with C r − 1 ( u, u (cid:48) )

Original: =
Revised: =

Original: Σ ( r − 1) ( u, u
Revised: Σ ( r − 1) ( u, u

Original: [Equation]
Revised: [Equation]

Original: ) / (cid:112) Σ ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .
Revised: ) / (cid:112) Σ ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .

Original: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )
Revised: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u (cid:48) , u (cid:48) ) .
Revised: Σ ( r ) ( u (cid:48) , u (cid:48) ) .

Original: We then process the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.
Revised: We then process the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.

Original: ReLU activation, φ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: The recursive equation (16) for q r ( u ) reduces to,
Revised: The recursive equation (16) for q r ( u ) reduces to,

Original: [Equation]
Revised: [Equation]

Original: The edge of chaos condition σ 2 w (cid:82) D z [ φ (cid:48) ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,
Revised: The edge of chaos condition σ 2 w (cid:82) D z [ φ (cid:48) ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,

Original: [Equation]
Revised: [Equation]

Original: Then the second integration for C r ( u, u (cid:48) ) becomes,
Revised: Then the second integration for C r ( u, u (cid:48) ) becomes,

Original: [Equation]
Revised: [Equation]

Original: To investigate the propagation of C r ( u, u (cid:48) )
Revised: To investigate the propagation of C r ( u, u (cid:48) )

Original: [Equation]
Revised: [Equation]

Original: we set q r ( u ) =
Revised: we set q r ( u ) =

Original: q r ( u (cid:48) )
Revised: q r ( u (cid:48) )

Original: =
Revised: =

Original: q
Revised: q

Original: [Equation]
Revised: [Equation]

Original: and deﬁne,
Revised: and deﬁne,

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: Let x ∈ [
Revised: Let x ∈ [

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: the derivative of f ( x ) satisﬁes,
Revised: the derivative of f ( x ) satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: This can be seen from a simple derivation,
Revised: This can be seen from a simple derivation,

Original: [Equation]
Revised: [Equation]

Original: Using an identity for Gaussian integration (cid:82) D z zg (
Revised: Using an identity for Gaussian integration (cid:82) D z zg (

Original: [Equation]
Revised: [Equation]

Original: ) =
Revised: ) =

Original: [Equation]
Revised: [Equation]

Original: ) yields,
Revised: ) yields,

Original: [Equation]
Revised: [Equation]

Original: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48)
Revised: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48)

Original: ( x ) =
Revised: ( x ) =

Original: [Equation]
Revised: [Equation]

Original: So using the equation above and the condition f (cid:48) (0)
Revised: So using the equation above and the condition f (cid:48) (0)

Original: = 12 , we can get another formation of the derivative of f ( x ) ,
Revised: = 12 , we can get another formation of the derivative of f ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,
Revised: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Substituting f ( x ) =
Revised: Substituting f ( x ) =

Original: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression up here, we have,
Revised: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression up here, we have,

Original: [Equation]
Revised: [Equation]

Original: Now we study the behavior of C r ( u, u (cid:48) )
Revised: Now we study the behavior of C r ( u, u (cid:48) )

Original: as r tends to inﬁnity.
Revised: as r tends to inﬁnity.

Original: Using Taylor expansion, we have,
Revised: Using Taylor expansion, we have,

Original: [Equation]
Revised: [Equation]

Original: By induction analysis, the sequence C r ( u, u (cid:48) ) is increasing to the ﬁxed point 1.
Revised: By induction analysis, the sequence C r ( u, u (cid:48) ) is increasing to the ﬁxed point 1.

Original: Besides, we can replace x with C r ( u, u (cid:48) ) ,
Revised: Besides, we can replace x with C r ( u, u (cid:48) ) ,

Original: [Equation]
Revised: [Equation]

Original: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,
Revised: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: so that,
Revised: so that,

Original: [Equation]
Revised: [Equation]

Original: As r tends to inﬁnity, we have,
Revised: As r tends to inﬁnity, we have,

Original: [Equation]
Revised: [Equation]

Original: It means,
Revised: It means,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have,
Revised: Therefore, we have,

Original: [Equation]
Revised: [Equation]

Original: Tanh activation, φ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: For the argument of ﬁxed point, we ask readers to refer to
Revised: For the argument of ﬁxed point, we ask readers to refer to

Original: Hayou et al.
Revised: Hayou et al.

Original: (2019a).
Revised: (2019a).

Original: Here we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to be 1 as the depth tends to be inﬁnity lim r →∞ C r ( u, u (cid:48) )
Revised: Here we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to be 1 as the depth tends to be inﬁnity lim r →∞ C r ( u, u (cid:48) )

Original: = 1 .
Revised: = 1 .

Original: For the function f ( x ) , a Taylor expansion near 1 yields,
Revised: For the function f ( x ) , a Taylor expansion near 1 yields,

Original: [Equation]
Revised: [Equation]

Original: where f (cid:48) (1)
Revised: where f (cid:48) (1)

Original: =
Revised: =

Original: σ 2 w (cid:82) D z [ φ
Revised: σ 2 w (cid:82) D z [ φ

Original: [Equation]
Revised: [Equation]

Original: ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =
Revised: ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =

Original: σ 2 w q (cid:82) D
Revised: σ 2 w q (cid:82) D

Original: z
Revised: z

Original: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .
Revised: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .

Original: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,
Revised: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = 2 (cid:82)
Revised: = 2 (cid:82)

Original: [Equation]
Revised: [Equation]

Original: ( √ q ∗ z ) 2 ] .
Revised: ( √ q ∗ z ) 2 ] .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Thus, we have,
Revised: Thus, we have,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 shows the covariance matrix converges to a constant matrix at a polynomial rate of 1 /rfor ReLU and of 1 /r for tanh activation on the edge of chaos.
Revised: Lemma 2 shows the covariance matrix converges to a constant matrix at a polynomial rate of 1 /rfor ReLU and of 1 /r for tanh activation on the edge of chaos.

Original: This implies that a network without aggregation could retain its expressivity at a large depth.
Revised: This implies that a network without aggregation could retain its expressivity at a large depth.

Original: However, for general case, due to the aggregation, the rate of convergence will change from polynomial to exponential.
Revised: However, for general case, due to the aggregation, the rate of convergence will change from polynomial to exponential.

Original: This will make the trainability of the deep graph network problematic, which will be shown in the following section.
Revised: This will make the trainability of the deep graph network problematic, which will be shown in the following section.

Original: B. C ONVERGENCE OF GPK FOR GCN S
Revised: B. C ONVERGENCE OF GPK FOR GCN S

Original: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:
Revised: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:

Original: Lemma 3.
Revised: Lemma 3.

Original: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that
Revised: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove the result by induction method.
Revised: We prove the result by induction method.

Original: For l = 0 , according to the Cauchy-BuniakowskySchwarz Inequality
Revised: For l = 0 , according to the Cauchy-BuniakowskySchwarz Inequality

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization, such that,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization, such that,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,
Revised: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where C 0 is a constant satisfying 0 < C 0 < C .
Revised: where C 0 is a constant satisfying 0 < C 0 < C .

Original: Now we consider the distance between Σ ( l +1)
Revised: Now we consider the distance between Σ ( l +1)

Original: ( r ) ( u, u (cid:48) ) and Cα l +1 .
Revised: ( r ) ( u, u (cid:48) ) and Cα l +1 .

Original: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.
Revised: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.

Original: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .
Revised: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .

Original: It is not hard to ﬁnd that steps 1,3 correspond to non-linear transformation while step 2 corresponds with aggregation operation, we then characterize them one by one,
Revised: It is not hard to ﬁnd that steps 1,3 correspond to non-linear transformation while step 2 corresponds with aggregation operation, we then characterize them one by one,

Original: MLP Propagation
Revised: MLP Propagation

Original: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:126)v (cid:12)(cid:12) ≤ C 

Revised: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:126)v (cid:12)(cid:12) ≤ C

Original: α l implicitly implies that
Revised: α l implicitly implies that

Original: C r ( u, u (cid:48) ) is close to one.
Revised: C r ( u, u (cid:48) ) is close to one.

Original: Because C r ( u, u (cid:48) )
Revised: Because C r ( u, u (cid:48) )

Original: =
Revised: =

Original: Σ
Revised: Σ

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) , then,
Revised: ) , then,

Original: [Equation]
Revised: [Equation]

Original: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )
Revised: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )

Original: =
Revised: =

Original: 1+ O ( α l ) .
Revised: 1+ O ( α l ) .

Original: Recall the property of MLP propagation function f ( x ) for x =
Revised: Recall the property of MLP propagation function f ( x ) for x =

Original: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:
Revised: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:

Original: [Equation]
Revised: [Equation]

Original: This implies,
Revised: This implies,

Original: [Equation]
Revised: [Equation]

Original: With this result, we can further obtain,
Revised: With this result, we can further obtain,

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a positive constant.
Revised: where C 1 is a positive constant.

Original: Repeat the proof process, we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1, 
Revised: Repeat the proof process, we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1, [Equation]

Original: [Equation]
Revised: [Equation]

Original: Aggregation Propagation We go through an aggregation operation
Revised: Aggregation Propagation We go through an aggregation operation

Original: A ( G ) .
Revised: A ( G ) .

Original: In this case, we use a formation of matrix, and take Equation (20) into aggregation function, yields the result as follows,
Revised: In this case, we use a formation of matrix, and take Equation (20) into aggregation function, yields the result as follows,

Original: [Equation]
Revised: [Equation]

Original: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .
Revised: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .

Original: According to the theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .
Revised: According to the theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .

Original: When l is ﬁnite, the error is of exponential decay.
Revised: When l is ﬁnite, the error is of exponential decay.

Original: Here we use 0 < α < 1 to denote the corresponding base number.
Revised: Here we use 0 < α < 1 to denote the corresponding base number.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Finally, Repeat the result in step 1, we have,
Revised: Finally, Repeat the result in step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.
Revised: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.

Original: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) which is a smaller order compared to O ( α l ) .
Revised: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) which is a smaller order compared to O ( α l ) .

Original: B. C ONVERGENCE OF THE GNTK
Revised: B. C ONVERGENCE OF THE GNTK

Original: Finally, we arrive at our main theorem:
Revised: Finally, we arrive at our main theorem:

Original: Theorem 1 (Convergence Rate of the GNTK) .
Revised: Theorem 1 (Convergence Rate of the GNTK) .

Original: If  A ( G )  is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G )  , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v,(cid:126)v (cid:48)∈ R n 2 × 1 depending on the number of MLP iterations R , such that
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) = A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized. Then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48)

Revised: ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: [Equation]
Revised: [Equation]

Revised: (cid:48) (cid:1)(cid:12)(cid:12) ≤ Cα l .

Original: Proof.
Revised: Proof.

Original: This proof has the same strategy to that of Lemma 3.
Revised: This proof has the same strategy to that of Lemma 3.

Original: The ﬁrst step is to understand the
Revised: The ﬁrst step is to understand the

Original: Equation (9) in the large-depth limit.
Revised: Equation (9) in the large-depth limit.

Original: [Equation]
Revised: [Equation]

Original: According to the result of Lemma 3, we have already known,
Revised: According to the result of Lemma 3, we have already known,

Original: [Equation]
Revised: [Equation]

Original: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.
Revised: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.

Original: ReLU activation, φ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: Recall that we deﬁne C r
Revised: Recall that we deﬁne C r

Original: +

Revised: +

Original: =
Revised: =

Original: f
Revised: f

Original: ( C r ) , and we have,
Revised: ( C r ) , and we have,

Original: [Equation]
Revised: [Equation]

Original: Then, at the edge of chaos,
Revised: Then, at the edge of chaos,

Original: [Equation]
Revised: [Equation]

Revised: r

Revised: [Equation]

Revised: =

Original:  Tanh activation, φ ( x ) = tanh( x ) .
Revised: [Equation] + [Equation] Tanh activation, φ ( x ) = tanh( x ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: At the edge of chaos,
Revised: At the edge of chaos,

Original: [Equation]
Revised: [Equation]

Original: Now we prove the result by induction method.
Revised: Now we prove the result by induction method.

Original: For l = 0 , we directly have,
Revised: For l = 0 , we directly have,

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,
Revised: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: Now we consider the distance between Θ ( l +1)
Revised: Now we consider the distance between Θ ( l +1)

Original: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R [Equation] l +1) (cid:126)v + (cid:126)v (cid:48) ) .
Revised: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R ( l +1) (cid:126)v + (cid:126)v (cid:48) ) .

Original: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.
Revised: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.

Original: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.
Revised: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.

Original: Θ ( l +1) (0)
Revised: Θ ( l +1) (0)

Original: [Equation]
Revised: [Equation]

Original: For step 1, we have,
Revised: For step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the process
Revised: Repeat the process

Original: [Equation]
Revised: [Equation]

Original: we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,
Revised: we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,

Original: [Equation]
Revised: [Equation]

Original: Secondly, we go through an aggregation operation.
Revised: Secondly [Equation] we go through an aggregation operation.

Original: Because it is a Markov chain step
Revised: Because it is a Markov chain step

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the result in step 1, we ﬁnally have,
Revised: Repeat the result in step 1, we ﬁnally have,

Original: [Equation]
Revised: [Equation]

Original: Corollary 1 (Trainability of Ultra-Wide GCNs) .
Revised: Corollary 1 (Trainability of Ultra-Wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth
Revised: Consider a GCN of the form (6) and (7), with depth

Original: L , non-linear transformations number R , an MSE loss, and a Lipchitz activation φ ( x ) , trained with gradient descent on a node classiﬁcation task.
Revised: L , non-linear transformations number R , an MSE loss, and a Lipchitz activation φ ( x ) , trained with gradient descent on a node classiﬁcation task.

Original: Then the output function follows,
Revised: Then the output function follows,

Original: [Equation]
Revised: [Equation]

Original: where Y is the label with respect to node V .
Revised: where Y is the label with respect to node V .

Original: Then,there exists a positive integer L 0 such that, Θ ( L ) ( R ) ( V , V ) is singular when L > L 0 .
Revised: Then, Θ ( L ) ( R ) ( V , V ) is singular when L → ∞ .

Original: Moreover, there exist a constant C > 0 such that for all t > 0 , [Equation]
Revised: Moreover, there exist a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .

Original: Proof.
Revised: Proof.

Original: According to the result from Jacot et al.
Revised: According to the result from Jacot et al.

Original: (2018), GNTK Θ ( l ) ( r ) ( V , V ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite width limit.
Revised: (2018), GNTK Θ ( l ) ( r ) ( V , V ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite width limit.

Original: We omit proof procedure for this result, since it is now a standard conclusion in the NTK study.
Revised: We omit proof procedure for this result, since it is now a standard conclusion in the NTK study.

Original: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,
Revised: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,

Original: [Equation]
Revised: [Equation]

Original: where ω t = θ t − θ 0 .
Revised: where ω t = θ t − θ 0 .

Original: We call it linearized model because only zero and ﬁrst order term of Taylor expansion are kept.
Revised: We call it linearized model because only zero and ﬁrst order term of Taylor expansion are kept.

Original: Since we know dynamics of gradient ﬂow using this linearized function are governed by,
Revised: Since we know dynamics of gradient ﬂow using this linearized function are governed by,

Original: [Equation]
Revised: [Equation]

Original: where L is an MSE loss, then the above equations have closed form solutions
Revised: where L is an MSE loss, then the above equations have closed form solutions

Original: [Equation]
Revised: [Equation]

Original: Since Lee et al.
Revised: Since Lee et al.

Original: Lee et al. (2019a) showed that f lin t ( V ) =
Revised: Lee et al. (2019a) showed that f lin t ( V ) =

Original: f t in the inﬁnite width limit, thus we have,
Revised: f t in the inﬁnite width limit, thus we have,

Original: [Equation]
Revised: [Equation]

Original: In theorem 1, we have shown Θ ( l ) ( r ) ( V , V ) converges a constant matrix at an exponential rate.
Revised: In theorem 1, we have shown Θ ( l ) ( r ) ( V , V ) converges a constant matrix at an exponential rate, thus being singular in the large depth limit.

Original: According to Equation (21), we know that,
Revised: According to Equation (21), we know that,

Original: [Equation]

Original: According to Theorem 1, there exists a l 0 such that, for any l > l 0 , the GNTK converges to a constant matrix,

Original: [Equation]
Revised: [Equation]

Original: Then the GNTK Θ ( l ) ( r ) ( V , V ) is singular. Let Θ ( l ) ( r ) ( V , V ) =
Revised: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( V , V ) =

Original: Q T DQ be the decomposition of the GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.
Revised: Q T DQ be the decomposition of the GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.

Original: Because Θ ( l ) ( r ) ( V , V ) is singular, D has at least one zero value d j = 0 , then
Revised: Because Θ ( ∞ ) ( r ) ( V , V ) is singular, D has at least one zero value d j = 0 , then

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: In the proof we assume the loss is MSE.
Revised: In the proof we assume the loss is MSE.

Original: Nevertheless, the conclude regarding trainability can be applied to other common loss such as cross-entropy.
Revised: Nevertheless, the conclude regarding trainability can be applied to other common loss such as cross-entropy.

Original: For cross-entropy loss, even though we cannot write down an analytical expression for the solution, we prove the GNTK still governs the trainability and the law of GNTK is not affected by the loss.
Revised: For cross-entropy loss, even though we cannot write down an analytical expression for the solution, we prove the GNTK still governs the trainability and the law of GNTK is not affected by the loss.

Original: Thus the results for trainability hold on the cross-entropy case.
Revised: Thus the results for trainability hold on the cross-entropy case.

Original: C C ONVERGENCE OF R ESIDUAL GNTK C. R ESIDUAL CONNECTION IN AGGREGATION
Revised: C C ONVERGENCE OF R ESIDUAL GNTK C. R ESIDUAL CONNECTION IN AGGREGATION

Original: Theorem 2 (Convergence rate for residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate for residual connection in aggregation) .

Original: Consider a GNTK of the form (12) with non-linear transformation of the form (7).
Revised: Consider a GNTK of the form (12) with non-linear transformation of the form (7).

Original: Then with a stationary vector ˜ π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <  α < 1 and C > 0 , and constant matrix (cid:126)v and (cid:126)v (cid:48) depending on
Revised: A ( G ) , there exist constants 0 < ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ2 and ˜ λ 2 ,respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ ˜ λ 2 ,

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: Proof.
Revised: Proof.

Original: According to the aggregation function for covariance matrix, we have
Revised: According to the aggregation function for covariance matrix, we have

Original: [Equation]
Revised: [Equation]

Original: The new aggregation matrix ˜
Revised: The new aggregation matrix ˜

Original: A ( G ) is a stochastic transition matrix as well, which can be seen from,(cid:88)
Revised: A ( G ) is a stochastic transition matrix as well, which can be seen from,

Original: [Equation]
Revised: [Equation]

Original: The new aggregation can slow down the convergence rate can seen from considering their eigenvalues.
Revised: The new aggregation can slow down the convergence rate can seen from considering their eigenvalues.

Original: Suppose the eigenvalues of original matrix
Revised: Suppose the eigenvalues of original matrix

Original: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .
Revised: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .

Original: We already know that the maximum eigenvalue is λ 

Revised: We already know that the maximum eigenvalue is λ

Original: = 1 ,
Revised: = 1 ,

Original: and the converge speed is governed by the second largest eigenvalue λ 2 .
Revised: and the converge speed is governed by the second largest eigenvalue λ 2 .

Original: Now we consider the second largest eigenvalue ˜ λ
Revised: Now we consider the second largest eigenvalue ˜ λ

Original: 2 of matrix ˜
Revised: 2 of matrix ˜

Original: A :
Revised: A :

Original: [Equation]
Revised: [Equation]

Original: Since λ 2 < ˜ λ 2 < 1 .
Revised: Since λ 2 < ˜ λ 2 < 1 .

Original: Thus the convergence speed of residual GNTK will be slow down, through it is still exponential convergent.
Revised: Thus the convergence speed of residual GNTK will be slow down, through it is still exponential convergent.

Original: For the limit behavior of Θ ( l ) ( r ) ( u, u (cid:48) )
Revised: For the limit behavior of Θ ( l ) ( r ) ( u, u (cid:48) )

Original: as l tends to inﬁnity, we can directly use the proof strategy from Theorem 1.
Revised: as l tends to inﬁnity, we can directly use the proof strategy from Theorem 1.

Original: C. R ESIDUAL CONNECTION IN MLP
Revised: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 . Here we discuss more about the relationship between α and ˜ α . According to (Rosenthal, 1995), the relationship between convergence rate α and second largest eigenvalue λ 2 of transition matrix A ( G ) can be expressed as,

Revised: [Equation]

Revised: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .

Revised: From the above inequality, we can conclude that the second largest eigenvalue almost govern the convergence rate.

Revised: However, to maintain rigor, we cannot directly obtain α < ˜ α .

Revised: Instead, from the result that ˜ λ 2 > λ 2 , we say that ˜ α is roughly larger than α .

Revised: C. R ESIDUAL CONNECTION IN MLP

Original: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .

Original: Consider a GNTK the form (8) and (12).
Revised: Consider a GNTK the form (8) and (12).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G, G (cid:48) ) , then there exist constants 0 < α < 1 and C > 0 , and constant matrix v and v (cid:48) depending on R , such that, |Θ ( l ) ( r ) [Equation] u, u
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G, G (cid:48) ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v (cid:48) depending on R , such that, |

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: For residual connection in MLP
Revised: For residual connection in MLP, we have,

Original: [Equation]
Revised: [Equation]

Original: we have,

Original: [Equation] Since σ 2 w > [Equation] q r ( G ) grows at an exponential rate.
Revised: Since σ 2 w > 0 , q r ( G ) grows at an exponential rate.

Original: Now [Equation] we turn to compute the correlation term C r ( u, u (cid:48) ) .
Revised: Now, we turn to compute the correlation term C r ( u, u (cid:48) ) .

Original: For convenience [Equation] we suppose q r ( u ) =
Revised: For convenience, we suppose q r ( u ) =

Original: q r ( u (cid:48) ) .
Revised: q r ( u (cid:48) ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Using Taylor expansion of f near
Revised: Using Taylor expansion of f near

Original: [Equation]
Revised: [Equation]

Original: as have been done in the proof of Lemma
Revised: as have been done in the proof of Lemma

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Note that it is similar to the case of MLP without residual connection:
Revised: Note that it is similar to the case of MLP without residual connection:

Original: [Equation]
Revised: [Equation]

Original: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,
Revised: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,

Original: [Equation]
Revised: [Equation]

Original: [Table]
Revised: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd˝os & R´enyi, 1961; Erd¨os & R´enyi, 2011), here n is the number of nodes in a graph, when we use the critical drop rate. To preserve information, edges will be resampled at every iteration. This implies that at each iteration during training, the GNN takes a random and large graph.

Original: [Equation] [Table]
Revised: For the whole training process, we can think that we have used all the information on the graph.

Original: [Equation]
Revised: It would be interesting to consider achieving some theoretical results for the limiting behavior of the GNTK in the large depth with Critical DropEdge. Since the Critical DropEdge is originated from random graph, a promising way is to analyze the spectral distributions of adjacency and Laplacian matrices with random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).

Original: D APPENDIX: D ATASET AND E XPERIMENTAL I MPLEMENTATION D. D ATASET
Revised: E APPENDIX: D ATASET AND E XPERIMENTAL I MPLEMENTATION E. D ATASET

Original: MUTAG (Rossi & Ahmed, 2015) is a bioinformatics dataset; Cora, Citeseer and Pubmed (Kipf &
Revised: MUTAG (Rossi & Ahmed, 2015) is a bioinformatics dataset; Cora, Citeseer, Pubmed (Kipf &

Original: Welling, 2016a) are commonly used citation network data for the node classiﬁcation.
Revised: Welling, 2016a) are commonly used citation network data for the node classiﬁcation.

Revised: Besides,

Revised: Physics (Shchur et al., 2018) is a co-author network dataset.

Original: We illustrate the detailed information for the three datasets as follows and summary can be found in Table 3.
Revised: We illustrate the detailed information for the four node classiﬁcation datasets as follows and summary can be found in Table 3.

Original: • The Cora dataset consists of 2708 scientiﬁc publications classiﬁed into one of seven classes.
Revised: • The Cora dataset consists of 2708 scientiﬁc publications classiﬁed into one of seven classes.

Original: The citation network consists of 5429 links.
Revised: The citation network consists of 5429 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 1433 unique words.
Revised: The dictionary consists of 1433 unique words.

Original: • The Citeseer dataset consists of 3312 scientiﬁc publications classiﬁed into one of six classes.
Revised: • The Citeseer dataset consists of 3312 scientiﬁc publications classiﬁed into one of six classes.

Original: The citation network consists of 4732 links.
Revised: The citation network consists of 4732 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 3703 unique words.
Revised: The dictionary consists of 3703 unique words.

Original: • The Pubmed Diabetes dataset consists of 19717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.
Revised: The Pubmed Diabetes dataset consists of 19717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.

Original: The citation network consists of 44338 links.
Revised: The citation network consists of 44338 links.

Original: Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words.
Revised: Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words.

Revised: • The Physics dataset consists of 34493 authors, and edges mean whether two authors coauthored a paper.

Revised: Node features are paper keywords from the author’s papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA following Kim & Oh (2020).

Revised: The split is the 20-per-class/30-per-class

Revised: /rest.

Revised: The goal of this task is to classify each author’s respective ﬁeld of study.

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: D. E XPERIMENTAL I MPLEMENTATION
Revised: E. E XPERIMENTAL I MPLEMENTATION

Original: We use the PyTorch implementation to simulate both inﬁnite-width and ﬁnite-width GCNs:
Revised: We use the PyTorch implementation to simulate both inﬁnite-width and ﬁnite-width GCNs:

Original: • The inﬁnite-width GCNs use part of code from Du et al.
Revised: • The inﬁnite-width GCNs use part of code from Du et al.

Original: (2019) which is originally adopted for graph classiﬁcation.
Revised: (2019) which is originally adopted for graph classiﬁcation.

Original: We redesigned the calculation method of GNTK 2 according to the formula in section 3.1 and used it to process node classiﬁcation tasks.
Revised: We redesigned the calculation method of GNTK 2 according to the formula in section 3.1 and used it to process node classiﬁcation tasks.

Original: • For ﬁnite-width GCNs 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.
Revised: • For ﬁnite-width GNNs with DropEdge 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.

Original: The difference is that, in each experiment, we ﬁx the edge sampling percent as ρ = | V | 2 | E | , which listed in the last column of Table 3.
Revised: The difference is that, in each experiment, we ﬁx the edge sampling percent as ρ = | V | 2 | E | , which listed in the last column of Table 3.

Original: All the codes mentioned above use the MIT license.
Revised: All the codes mentioned above use the MIT license.

Original: All experiments are conducted on with two
Revised: All experiments are conducted on with two

Original: Nvidia Quadro RTX 6000 GPUs.
Revised: Nvidia Quadro RTX 6000 GPUs.

Original: [Table]

Original: E APPENDIX: A DDITIONAL E XPERIMENT AND D ISCUSSION E. I NFINITE -W IDTH B ACKBONE
Revised: F APPENDIX: A DDITIONAL E XPERIMENT AND D ISCUSSION F. I NFINITE -W IDTH B ACKBONE

Original: In the main text

Original: [Table] we reported the performance of GCNs with ﬁnite-width [Table] here we display more results with respect to inﬁnite-width GCNs, as shown in Table 4.
Revised: In the main text, we reported the performance of GCNs with ﬁnite-width, here we display more results with respect to inﬁnite-width GCNs, as shown in Table 4.

Original: We apply Critical DropEdge to inﬁnite-width backbone on node classiﬁcation with a semi-supervised setting.
Revised: We apply Critical DropEdge to inﬁnite-width backbone on node classiﬁcation with a semi-supervised setting.

Original: We consider two backbones: GCN (Kipf & Welling, 2016a) and
Revised: We consider two backbones: GCN (Kipf & Welling, 2016a) and

Original: JKNet (Xu et al., 2018b).
Revised: JKNet (Xu et al., 2018b).

Original: For the edge rate, we use the edge preserving percent as ρ ( G ) which shown in Table 3.
Revised: For the edge rate, we use the edge preserving percent as ρ ( G ) which shown in Table 3.

Original: Table 4 summaries the performance on three citation network datasets.
Revised: Table 4 summaries the performance on three citation network datasets.

Original: In this table, we report the performance of GCNs with 4/8/32/ layers.
Revised: In this table, we report the performance of GCNs with 4/8/32/ layers.

Original: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original deep GNNs in the inﬁnite-width case, especially when the model is deep. 
Revised: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original deep GNNs in the inﬁnite-width case, especially when the model is deep. F. P ERFORMANCE OF S EPARATE BACKBONES [Table]

Revised: The performance results presented in Table 1 is the best performance across different backbones.

Original: E. C OMPARISON TO D ROP E DGE
Revised: In Table 5, we show the results of vanilla GNNs, DropEdge and C-DropEdge with different backbones. From the table we can conclude the Critical DropEdge can consistently outperform DropEdge and vanilla GNNs. F. C OMPARISON TO D ROP E DGE

Original: While our proposed Critical DropEdge shares the same program with the DropEdge method (Rong et al., 2019), we point out the main difference which can result in a corresponding advantage for our approach concerning the hyper-parameter search space.
Revised: While our proposed Critical DropEdge shares the same program with the DropEdge method (Rong et al., 2019), we point out the main difference which can result in a corresponding advantage for our approach concerning the hyper-parameter search space.

Original: C-DropEdge has a ﬁxed and graphdependent edge preserving percent, which implies our hyper-parameter space is smaller than that of
Revised: C-DropEdge has a ﬁxed and graphdependent edge preserving percent, which implies our hyper-parameter space is smaller than that of

Original: DropEdge.
Revised: DropEdge.

Original: The edge preserving percent is one of the most signiﬁcant hyper-parameter, which has a crucial inﬂuence on the ﬁnal performance of the algorithm.
Revised: The edge preserving percent is one of the most signiﬁcant hyper-parameter, which has a crucial inﬂuence on the ﬁnal performance of the algorithm.

Original: In order to obtain good performance,
Revised: In order to obtain good performance,

Original: DropEdge needs to spend a lot of effort to search on the space in terms of this hyper-parameter.
Revised: DropEdge needs to spend a lot of effort to search on the space in terms of this hyper-parameter.

Original: In contrast, our method naturally determines the parameters, thus saving plenty of time.
Revised: In contrast, our method naturally determines the parameters, thus saving plenty of time.

Original: Whether from a theoretical perspective or from the ﬁnal performance, the edge preserving percent we set is reasonable and effective.
Revised: Whether from a theoretical perspective or from the ﬁnal performance, the edge preserving percent we set is reasonable and effective.

Revised: F. T RAINABILITY OF GNTK

Revised: We implement deep GCN without/without DropEdge and C-DropEdge to compare the training performance.

Revised: The result is shown in Figure 4.

Revised: The training of GCN and DropEdge with GCN converge slower and converge to a higher training error compared to C-DropEdge with GCN, indicating that

Revised: C-DropEdge can achive better trainablity compared to GCN and DropEdge with GCN.

Revised: Besides, the

Revised: [Figure]

Revised: [Figure]

Revised: convergence results of GNTKs with C-DropEdge is shown in Figure 5.

Revised: Compared to residual connection, we ﬁnd that the normalized GNTK elements will not converge to a single value, and the convergence rate curve does not depend on the depth.

Revised: This implies that residual connection mitigates the trainability loss by slow down exponential convergence rate, while C-DropEdge directly change the convergence results.