Original: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE
Revised: T OWARDS D EEPENING G RAPH N EURAL N ETWORKS : A GNTK- BASED O PTIMIZATION P ERSPECTIVE

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.
Revised: Graph convolutional networks (GCNs) and their variants have achieved great success in dealing with graph-structured data.

Original: However, it is well known that deep GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.
Revised: Nevertheless, it is well known that deep GCNs suffer from the over-smoothing problem, where node representations tend to be indistinguishable as more layers are stacked up.

Original: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.
Revised: The theoretical research to date on deep GCNs has focused primarily on expressive power rather than trainability, an optimization perspective.

Original: Compared to expressivity, trainability attempts to address a more fundamental question: given a sufficiently expressive space of models, can we successfully find a good solution by gradient descentbased optimizer?
Revised: Compared to expressivity, trainability attempts to address a more fundamental question: Given a sufﬁciently expressive space of models, can we successfully ﬁnd a good solution via gradient descent-based optimizers?

Original: This work fills this gap by exploiting the Graph Neural Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.
Revised: This work ﬁlls this gap by exploiting the Graph Neural Tangent Kernel (GNTK), which governs the optimization trajectory under gradient descent for wide GCNs.

Original: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.
Revised: We formulate the asymptotic behaviors of GNTK in the large depth, which enables us to reveal the dropping trainability of wide and deep GCNs at an exponential rate in the optimization process.

Original: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be only able tomildly mitigate the exponential decay of trainability.
Revised: Additionally, we extend our theoretical framework to analyze residual connection-based techniques, which are found to be merely able to mitigate the exponential decay of trainability mildly.

Original: To overcome the exponential decay problem more fundamentally, we propose Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, inspired by our theoretical insights on trainability.
Revised: Inspired by our theoretical insights on trainability, we propose Critical DropEdge, a connectivity-aware and graph-adaptive sampling method, to alleviate the exponential decay problem more fundamentally.

Original: Experimental evaluation consistently confirms using our proposed method can achieve better results compared to relevant counterparts with both infinite-width and finite-width.
Revised: Experimental evaluation consistently conﬁrms using our proposed method can achieve better results compared to relevant counterparts with both inﬁnite-width and ﬁnite-width.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Recently, Graph Convolutional Networks (GCNs) have shown incredible abilities to learn node or graph representations and achieved superior performances for various downstream tasks, such as node classifications (Kipf & Welling, 2016a; Veli ˇ ckovi ´ c et al., 2017; Hamilton et al., 2017), graph classifications (Xu et al., 2018a; Gilmer et al., 2017; Lee et al., 2019b; Yuan & Ji, 2020), and link predictions (Kipf & Welling, 2016b), etc.
Revised: Recently, Graph Neural Networks (GNNs) have shown incredible abilities to learn node or graph representations and achieved superior performance on various downstream tasks, such as node classiﬁcation (Kipf & Welling, 2016a; Veliˇckovi´c et al., 2017; Hamilton et al., 2017), graph classiﬁcation (Xu et al., 2018a; Gilmer et al., 2017; Lee et al., 2019b; Yuan & Ji, 2020), and link prediction (Kipf & Welling, 2016b), etc.

Original: However, most GCNs achieve their best  with a shallow depth, e.g., 2 or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.
Revised: However, most GNNs (e.g., GCNs) achieve their best only with a shallow depth, e.g., 2 or 3 layers, and their performance on those tasks would promptly degrade as the number of layers grows.

Original: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.
Revised: Towards this phenomenon, research attempts have been made to deepen understanding of current GNN architectures and their expressive power.

Original: Li et al.
Revised: Li et al.

Original: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.
Revised: (2018) showed that GCN is a special form of Laplacian smoothing, which mixes node representations with nearby neighbors.

Original: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.
Revised: This mechanism potentially poses the risk of over-smoothing as more layers are stacked together, where node representations tend to be indistinguishable from each other.

Original: Oono & Suzuki (2019) investigated the expressive power of GNNs using the asymptotic behaviors as the layer goes to infinity.
Revised: Oono & Suzuki (2019) investigated the expressive power of GNNs using the asymptotic behaviors as the ∗ Equal Contribution.

Revised: layer goes to inﬁnity.

Original: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graphs inherent in the graph spectra.
Revised: They proved that under certain conditions, the expressive power of GCN is determined by the topological information of the underlying graph inherent in the graph spectra.

Original: However, it remains elusive how to theoretically understand why deepen GCNs fail to optimize.
Revised: Nevertheless, it remains elusive how to theoretically understand why deep GCNs fail to optimize.

Original: Existing theoretical investigations (Xu et al., 2018a; Oono & Suzuki, 2019) on GNNsmainly focus  on expressivity which measureshow complex the function a network can represent.
Revised: Existing theoretical investigation (Xu et al., 2018a; Oono & Suzuki, 2019) on GNNs focus mainly on expressivity, which measures the complexity of functions that can be represented by a neural network.

Original: Exploring expressiveness is theoretically convenient, but the corresponding conditions may be broken during the gradient training process, thus leading to inconsistencies between theoretical conclusions and  results of trained networks (G¨uhring et al., 2020).
Revised: Exploring expressivity is theoretically convenient, but the corresponding conditions may be violated during the gradient training process, thereby leading to inconsistencies between theoretical conclusions and empirical results of trained networks (G¨uhring et al., 2020).

Original: Compared tothe expressivity, trainability attempts to address a more difficult but fundamental perspective of neural networks: how effective a  network can be optimized via gradient descent.
Revised: Compared to expressivity, trainability addresses a more difﬁcult but fundamental perspective of neural networks: How effectively a neural network can be optimized via gradient descent.

Original: The advantage of studying trainability is that we can directly know whether the graph neural network can be successfully trained under certain conditions, and to what extent.
Revised: The advantage of investigating trainability is that we can directly determine whether GNNs can be successfully trained under certain conditions, and to what extent.

Original: We are therefore inspired to ask:
Revised: We are therefore inspired to raise two important questions:

Original: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?
Revised: • Can we theoretically characterize the trainability of graph neural networks with respect to depth, thus understanding why deep GCNs fail to generalize?

Original: • Can we further design an algorithm to deepen GCNs benefit from our theoretical investigation?
Revised: • Can we further design an algorithm to facilitate deeper GCNs, beneﬁting from our theoretical investigation?

Original: Our answers are yes to both questions.
Revised: Our answers are yes to both questions.

Original: We resort to the infinitely wide multi-layer GCN.
Revised: We resort to the inﬁnitely-wide multi-layer GCN to derive our solution.

Original: The research on infinitely-wide networks can be traced back to the seminal work of Neal (1996), which showed the one hidden layer networks with random weights at initialization (without training) are Gaussian Processes (GPs) in the infinite width limit.
Revised: The research on inﬁnitely-wide networks can be traced back to the seminal work of Neal (1996), which showed that single hidden layer networks with random weights at initialization (without training) are Gaussian Processes (GPs) in the inﬁnite width limit.

Original: Later, the connection between GPs and multilayer infinitely-wide networks with Gaussian initialization (Lee et al., 2017; Matthews et al., 2018) and orthogonal weights (Huang et al., 2020a) was reported.
Revised: Later, the connection between GPs and multi-layer inﬁnitely-wide networks with Gaussian initialization (Lee et al., 2017; Matthews et al., 2018) and orthogonal weights (Huang et al., 2020a) was reported.

Original: Recent trends in Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of infinitely (ultra)-wide networks.
Revised: Recent trends in Neural Tangent Kernel (NTK) have led to a proliferation of studies on the optimization and generalization of inﬁnitely (ultra)-wide networks.

Original: In particular, Jacot et al.
Revised: In particular, Jacot et al.

Original: (2018) made a groundbreaking discovery that gradient descent training in the infinite width limit can be captured by an NTK.
Revised: (2018) made a groundbreaking discovery that gradient descent training in the inﬁnite width limit can be captured by an NTK.

Original: Du et al.
Revised: Du et al.

Original: (2019) formulated Graph Neural Tangent Kernel (GNTK) for infinitely-wide GNNs and shed light on theoretical guarantees for GNN.
Revised: (2019) formulated Graph Neural Tangent Kernel (GNTK) for inﬁnitely-wide GNNs and shed light on theoretical guarantees for GNNs.

Original: Prior to the discovery of GNTK, there was little understanding of the non-convexity of graph neural networks, which is analytically intractable.
Revised: Prior to the discovery of GNTK, there was little understanding of the non-convexity of GNNs, which is analytically intractable.

Original: In the learning regime of GCN governed by GNTK, the optimization becomes an almost convex problem, making GNTK a promising method to study the trainability of deep GCNs.
Revised: In the learning regime of GCN governed by GNTK, the optimization becomes an almost convex problem, making GNTK a promising perspective to study the trainability of deep GCNs.

Original: In this work, we leverage the GNTK techniques of infinitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.
Revised: In this work, we leverage the GNTK techniques of inﬁnitely-wide networks to investigate whether ultra-wide GCNs are trainable in the large depth.

Original: In particular, we formulate the large-depth asymptotics behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b; Xiao et al., 2019), through which we can analyze the optimization properties.
Revised: In particular, we formulate the large-depth asymptotic behavior of the GNTK, illuminated by innovative works on deep networks (Hayou et al., 2019b; Xiao et al., 2019), through which we can analyze the optimization properties.

Original: We further extend our theoretical framework to residual connection-like techniques and show that these techniques can only help slow down the exponential decay of trainability to some extent, but lack the ability to overcome the problem thoroughly.

Original: Our contributionsare as follows:
Revised: Speciﬁcally, we make the following contributions

Original:  • To our best knowledge, we are the first to investigate the trainability of deep GCNs through GNTK.
Revised: : • To our best knowledge, we are the ﬁrst to investigate the trainability of deep GCNs through GNTK.

Original: We prove that all elements of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.
Revised: We prove that all entries of a GNTK matrix regarding a pair of graphs converge exponentially to the same value, making the GNTK matrix singular in the large depth.

Original: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classification tasks.
Revised: We thus establish a corollary that the trainability of ultra-wide GCNs exponentially collapses on node classiﬁcation tasks.

Original: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.
Revised: • We apply our theoretical analysis to the residual connection-based techniques for GCNs.

Original: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate, but can not fundamentally solve theexponential decay problem.
Revised: Our theory shows that residual connection can, to some extent, slow down the exponential decay rate of trainability, but lack the ability to fundamentally solve the problem.

Original: This result can be used to explain to what extentthe recent residual connection-based method works.
Revised: This result enables to better understand why and to what extent recent residual connection-based methods work.

Original: • Our theoretical framework provides comprehensive guidance for the development of deep GCNs.
Revised: • Our theoretical framework provides insights to guide the development of deep GCNs.

Original: We  propose an edge-based sampling method, named Critical DropEdge, to overcome the exponential decay of trainability.
Revised: We further propose an edge-based sampling method, named Critical DropEdge, to effectively mitigate the exponential decay of trainability.

Original: This graph-adaptive and connectivity-aware method is easy to implement in both finite-width and infinite-width GNNs.
Revised: This graph-adaptive and connectivity-aware method is easy to implement in both ﬁnite-width and inﬁnite-width GNNs.

Original: Our experiments show using the proposed method can outperform counterparts in the large depth.
Revised: Our experiments show using the proposed method can outperform competitors in the large depth.

Original:  B ACKGROUND
Revised:  B ACKGROUND

Original: We first review the results of infinitely-wide neural networks at initialization.
Revised: We ﬁrst review the results of inﬁnitely-wide neural networks at initialization.

Original: Then we introduce the Neural Tangent Kernel, making a connection to trainability.
Revised: We then review NTK, making a connection to trainability.

Original: Finally, we introduce Graph Convolutional Networks along with our setup and notation.
Revised: Finally, we introduce GCNs along with our setup and notation.

Original: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION
Revised: 2. I NFINITELY - WIDE N ETWORKS AT I NITIALIZATION

Original: We begin by considering a fully-connected network of depth L of widths m l in each layer.
Revised: We begin by considering a fully-connected network of depth L with width m l in each layer.

Original: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .
Revised: The weight and bias in the l -th layer are denoted by W ( l ) ∈ R m l × m l − 1 and b ( l ) ∈ R m l .

Original: Letting the pre-activations be given by h ( l ) i . Then the information propagation in this network is governed by,
Revised: Letting the pre-activations be given by h ( l ) i , information propagation in this network is governed by,

Original: [Equation]
Revised: [Equation]

Original: where ϕ : R → R is the activation function, σ w and σ b define the variance scale of the weights and biases respectively.
Revised: where φ : R → R is the activation function, σ w and σ b deﬁne the variance scale of the weights and biases, respectively.

Original: Given the parameterization that weights and biases are randomly generated by i.i.d.
Revised: Given the parameterization that weights and biases are randomly generated by i.i.d.

Original: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the infinite width limit as m 1 , m 2 , · · · , m l − 1 → ∞ .
Revised: normal distribution, i.e., W ( l ) ij , b ( l ) i ∼ N (0 , 1) , the pre-activations are Gaussian distributed in the inﬁnite width limit as m 1 , m 2 , . . . , m l − 1 → ∞ .

Original: This results from the central limit theorem (CLT).
Revised: This results from the central limit theorem (CLT).

Original: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding infinitely-wide network is defined by Σ ( l ) ( x, x ′ )
Revised: Consider a dataset X ∈ R n × m 0 of size n = | X | , the covariance matrix of Gaussian process kernel (GPK) regarding inﬁnitely-wide network is deﬁned by Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: E [ h ( l ) i ( x ) h ( l ) i ( x ′ )] .
Revised: E [ h ( l ) i ( x ) h ( l ) i ( x (cid:48) )] .

Original: According to the signal propagationEquation (1), the covariance matrix or Gaussian Process kernel with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x ′ )
Revised: According to the signal propagation (1), the covariance matrix or GPK with respect to layer can be described by a recursion relation, Σ ( l ) ( x, x (cid:48) )

Original: =
Revised: =

Original: σ 2 w E h ∼N (0 , Σ ( l − 1) )
Revised: σ 2 w E h ∼N (0 , Σ ( l − 1) )

Original: [ ϕ ( h ( x )) ϕ ( h ( x ′ ))]
Revised: [ φ ( h ( x )) φ ( h ( x (cid:48) ))]

Original: + σ 2 b .
Revised: + σ 2 b .

Original: The mean-field theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2016).
Revised: The mean-ﬁeld theory is a paradigm that studies the limiting behavior of GPK, which is a measure of expressivity for networks (Poole et al., 2016; Schoenholz et al., 2016).

Original: In particular, expressivity describes to what extent two different inputs can be distinguished.
Revised: In particular, expressivity describes to what extent two different inputs can be distinguished.

Original: The property of evolution for expressivity Σ ( l ) ( x, x ′ ) is determined by how fast it converges to its fixed point Σ ∗ ( x, x ′ )
Revised: The property of evolution for expressivity Σ ( l ) ( x, x (cid:48) ) is determined by how fast it converges to its ﬁxed point Σ ∗ ( x, x (cid:48) )

Original: ≡ lim l →∞ Σ ( l ) ( x, x ′ ) .
Revised: ≡ lim l →∞ Σ ( l ) ( x, x (cid:48) ) .

Original: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;
Revised: It is shown that in almost the entire parameter space spanned by hyper-parameters σ w and σ b , the evolution exhibits a dramatic convergence rate formulated by an exponential function except for a critical line known as the edge of chaos (Poole et al., 2016;

Original: Schoenholz et al., 2016; Yang & Schoenholz, 2017).
Revised: Schoenholz et al., 2016; Yang & Schoenholz, 2017).

Original: Consequently, an infinitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.
Revised: Consequently, an inﬁnitely-wide network loses its expressivity exponentially in most cases while retaining the expressivity at the edge of chaos.

Original: Given this reason, we focus on the edge of chaos in this work.
Revised: Given this reason, we focus on the edge of chaos in this work.

Original: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z [ ϕ ′ ( √ q ∗ z )] 2 = 1 , where q ∗ is the fixed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: In particular, we set the value of hyper-parameters to satisfy, σ 2 w (cid:82) D z [ φ (cid:48) ( √ q ∗ z )] 2 = 1 , where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, edge of chaos requires σ 2 w = 2 and σ 2 b = 0 .

Original: 2. N EURAL T ANGENT K ERNEL AND T RAINABILITY
Revised: N EURAL T ANGENT K ERNEL AND T RAINABILITY

Original: Most studies on infinitely-wide networks through mean-field theory (Poole et al., 2016; Schoenholz et al., 2016) have focused solely on initialization without training.
Revised: Most studies on inﬁnitely-wide networks through mean-ﬁeld theory (Poole et al., 2016; Schoenholz et al., 2016) have focused solely on initialization without training.

Original: Jacot et al.
Revised: Jacot et al.

Original: (2018) took a step further by considering the infinitely-wide network trained with gradient descent.
Revised: (2018) took a step further by considering inﬁnitely-wide networks trained with gradient descent.

Original: Let η be the learning rate, and L be the loss function.
Revised: Let η be the learning rate, and L be the loss function.

Original: The dynamics of gradient flow for parametersθ = vec
Revised: The dynamics of gradient ﬂow for parameters

Original: [Equation]

Original: [Equation]
Revised: [Equation]

Original: 1 , the vector of all parameters, is given by,
Revised: , the vector of all parameters, is given by,

Original: [Equation]
Revised: [Equation]

Original: [Equation]

Original: Then the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X
Revised: Then, the dynamics of output functions f ( X ) = vec( f ( x ) x ∈ X

Original: ) ∈ R
Revised: ) ∈ R

Original: nm L × 1 follow,
Revised: nm L × 1 follow,

Original: [Equation]
Revised: [Equation]

Original: (3) where the NTK at time t is defined as,
Revised: where the NTK at time t is deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: In a general case, the NTK varies with the training time, thus providing no substantial insight into the convergence property of neural networks.
Revised: In a general case, the NTK varies with the training time, thus providing no substantial insights into the convergence property of neural networks.

Original: Interestingly, as shown by Jacot et al.
Revised: Interestingly, as shown by Jacot et al.

Original: (2018), the NTK converges to an explicit limiting kernel and does not change during training in the infinite width limit.
Revised: (2018), the NTK converges to an explicit limiting kernel and does not change during training in the inﬁnite width limit.

Original: This leads to a simple but profound result in the case of an MSE loss, L = 12 ∥ f t ( X ) − Y ∥ 2 F , where Y is the label regarding the input X , 
Revised: This leads to a simple but profound result in the case of an MSE loss, L = 12 (cid:107) f t ( X ) − Y (cid:107) 22 , where Y is the label associated with the input X , [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Θ ∞ is the limiting kernel.
Revised: where Θ ∞ is the limiting kernel.

Original: This is the solution of an ordinary differential equation.
Revised: This is the solution to an ordinary differential equation.

Original: As the training time t tends to infinity, the output function fits the label very well, i.e., f ∞ ( X ) =
Revised: As the training time t tends to inﬁnity, the output function ﬁts the label very well, i.e., f ∞ ( X ) =

Original: Y .
Revised: Y .

Original: As proved by Hayou et al. (2019b) (Lemma 1), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.
Revised: As proved by Hayou et al. (2019b) (Lemma 1), the network is trainable only if Θ ∞ ( X, X ) is nonsingular.

Original: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as confirmed by Xiao et al.
Revised: Quantitatively, the condition number κ ≡ λ max /λ min can be a measure of trainability as conﬁrmed by Xiao et al.

Original: 2. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: 2. G RAPH C ONVOLUTIONAL N ETWORKS

Original: We define an undirected graph as G =
Revised: We deﬁne an undirected graph as G =

Original: ( V, E ) , where V is a collection of nodes while E is a set of edges.
Revised: ( V, E ) , where V is a set of nodes and E is a set of edges.

Original: We denote the number of nodes in graph G by n = | V | .
Revised: We denote the number of nodes in graph G by n = | V | .

Original: The node features are denoted as h v ∈ R d for each v ∈ V , with d being the dimension of node features.
Revised: The node features are denoted as h v ∈ R d for each v ∈ V , with d being the dimension of node features.

Original: In this work, we develop our theory towards understanding the trainability of GCNs on node classification tasks.
Revised: In this work, we develop our theory towards understanding the trainability of GCNs on node classiﬁcation tasks.

Original: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representations of their neighbors.

Original: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 in Appendix A illustrates an overview of the information propagation in a general GCN.

Original: We define a propagation unit to be the combination of a R -layer MLP and one aggregation operation.
Revised: We deﬁne a propagation unit to be the combination of a R -layer multi-layer perceptron (MLP) and one aggregation operation.

Original: We use subscript ( r ) to denote the layer index of MLP in each propagation unit anduse superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.
Revised: We use subscript ( r ) to denote the layer index of MLP in each propagation unit and superscript ( l ) to indicate the index of aggregation operation, which is also the index of the propagation unit.

Original: L is the total number of propagation units.
Revised: L is the total number of propagation units.

Original: Specifically, the node representation propagation in GCNs through a multi-layer perceptron (MLP) follows the expression,
Revised: To be speciﬁc, the node representation propagation in GCNs through an MLP follows the expression,

Original: [Equation]
Revised: [Equation]

Original:  where W ( l ) ( r ) ∈
Revised: (7) where W ( l ) ( r ) ∈

Original: R m l × m l − 1 and b ( l ) ( r ) ∈
Revised: R m l × m l − 1 and b ( l ) ( r ) ∈

Original: R m l are the learnable weights and biases respectively, ϕ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node v and its neighbors.
Revised: R m l are the learnable weights and biases, respectively, φ is the activation function, N ( u ) is the neighborhood of node u , and N ( u ) ∪ u is the union of node u and its neighbors.

Original: Equation (6) describes the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).
Revised: Equation (6) reveals the node feature aggregation operation among its neighborhood according to a GCN variant (Hamilton et al., 2017).

Original: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b define the variance scale of the weights and biases.
Revised: Equation (7) is a standard nonlinear transformation with NTK-parameterization (Jacot et al., 2018), where m is the width, i.e., number of neurons in each layer, σ w and σ b deﬁne the variance scale of the weights and biases.

Original: With regard to the activation function, we focus on both ReLU and tanh, which are denoted as ϕ ( x ) = max { 0 , x } and ϕ ( x ) 
Revised: For the activation function, we focus on both ReLU and Tanh, which are denoted as φ ( x ) = max { 0 , x } and φ ( x ) =

Original: = tanh( x )  respectively.
Revised: tanh( x ) , respectively.

Original: Without loss of generality, our theoretical framework can handle other common activations. By comparison, the GNTK work (Du et al., 2019) only adopted ReLU activation.
Revised: Without loss of generality, our theoretical framework can handle other common activation functions, whereas the GNTK work (Du et al., 2019) only adopted ReLU.

Original:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION
Revised:  A GGREGATION P ROVABLY L EADS TO E XPONENTIAL T RAINABILITY L OSS 3. GNTK F ORMULATION

Original: On the basis of the definition of NTK (Equation 4), we recursively formulate the propagation of GNTK in the infinite-width limit.
Revised: Based on the deﬁnition of NTK (4), we recursively formulate the propagation of GNTK in the inﬁnite-width limit.

Original: Since the information propagation in a GCN is built on two operations: aggregation (Equation 6) and non-linear transformation (Equation 7), the corresponding formulas of GNTK are expressed as follows,
Revised: As information propagation in a GCN is built on two operations: aggregation (6) and non-linear transformation (7), the corresponding formulas of GNTK are expressed as follows,

Original: [Equation]
Revised: [Equation]

Original: The two equations above correspond to the aggregation operation and MLP transformation respectively.
Revised: The two equations above correspond to the aggregation operation and MLP transformation, respectively.

Original: To compute the GNTK with respectthe to depth, the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u ′ )
Revised: To compute the GNTK with respect to the depth [Equation] the key step is to obtain the covariance matrix Σ ( l ) ( r ) ( u, u (cid:48) )

Original: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u ′ )] .
Revised: ≡ E [ h ( l ) ( r ) ( u ) h ( l ) ( r ) ( u (cid:48) )] .

Original: According to the CLT, the node representation h ( l ) ( r ) ( u ) is  Gaussian distributed in the infinite width limit.
Revised: According to the CLT [Equation] the node representation h ( l ) ( r ) ( u ) is a Gaussian distributed in the inﬁnite width limit.

Original: Applying this result to Equation (6) and (7)  the resulting covariance matrix is composed of two parts, 
Revised: Applying this result to equations (6) and (7) [Equation] the resultant covariance matrix is composed of two parts, [Equation]

Original: [Equation]
Revised: [Equation]

Original: The first equation  results from the aggregation operation.
Revised: The ﬁrst equation (10) results from the aggregation operation.

Original: Meanwhile, the second equation  corresponds to the R -times non-linear transformations, where  z 1 , z 2 are drawn from a 2-dimensional
Revised: Meanwhile, the second equation (11) corresponds to the R -times non-linear transformations, where E z 1 ,z 2 takes the expectation with respect to a centered Gaussian process of covariance ˜Σ ( l ) ( r − 1)

Original: Gaussian distribution of the previous MLP layer, and the variance ˜Σ ( l ) ( r − 1) ∈ R 2 × 2 is across u, u ′ .
Revised: ∈ R 2 × 2 for previous MLP layer across u, u (cid:48) , and ˙ φ denotes the derivative of φ .

Original: 3. T RAINABILITY IN THE L ARGE D EPTH
Revised: 3. T RAINABILITY IN THE L ARGE D EPTH

Original: We aim to characterize the behavior of Θ ( l ) ( r ) ( G ) = Θ ( l ) ( r ) ( V , V ) ∈ R n × n , where we denote
Revised: We aim to characterize the behavior of Θ ( l ) ( r ) ( G ) ≡ Θ ( l ) ( r ) ( V , V ) ∈ R n × n , where we denote

Original: V =
Revised: V =

Original: [ v 1 , v 2 , . .
Revised: [ v 1 , v 2 , . .

Original: . , v n ] , as the depth tends to infinity l → ∞ .
Revised: . , v n ] , as the depth tends to inﬁnity l → ∞ .

Original: From the GNTK formulation, both aggregation (Equation 8) and transformation (Equation 9) contribute simultaneously to the final limiting result.
Revised: From the GNTK formulation, both aggregation (8) and transformation (9) contribute simultaneously to the ﬁnal limiting result.

Original: Taking them into consideration, we derive our theorem on the asymptotic behavior of infinitely-wide GCN in the large depth limit, which is  as follows:
Revised: Taking them into consideration, we derive our theorem on the asymptotic behavior of inﬁnitely-wide GCN in the large depth limit, which is given as follows:

Original: Theorem 1 (Convergence Rate ofthe GNTK) .
Revised: Theorem 1 (Convergence Rate of GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector ⃗π ( G ) ∈ R n 2 × 1 , where ⃗ Θ ( l ) (0) ( G ) =
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) =

Original: A ( G ) ⃗ Θ ( l − 1) ( R ) ( G ) and ⃗ Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then there exist constants 0 < α < 1 and C > 0 , and constant vectors ⃗v,⃗v ′ ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u ′ ) 
Revised: Then, there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48) ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: − ⃗π ( G ) T (cid:0) Rl⃗v + ⃗v ′ (cid:1)(cid:12)(cid:12) ≤ Cα l .
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) [Equation]

Original: The proof sketch follows a divide-and-conquer manner.
Revised: The proof sketch follows a divide-and-conquer manner.

Original: In particular, we first analyze the network withthe only aggregation and prove that A ( G ) is a Markov transition matrix.
Revised: In particular, we ﬁrst analyze the network with only aggregation and prove that A ( G ) is a Markov transition matrix.

Original: Then we formulate the behavior of MLP in the large depth based on Hayou et al.
Revised: Then, we formulate the behavior of MLP in the large depth based on Hayou et al.

Original: (2019b).
Revised: (2019b).

Original: Finally, we arrive at the final result by considering the two operations.
Revised: Finally, we derive the ﬁnal result by considering the two operations.

Original: We leave the complete proof in Appendix B.
Revised: We leave the complete proof in Appendix B.

Original: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.
Revised: In Theorem 1, we rigorously characterize the convergence properties of GNTK in the large depth limit.

Original: As  depth goes to infinity, all elements in the GNTK converge to a unique quantity at an exponential rate.
Revised: As the depth goes to inﬁnity, all entries in the GNTK converge to a unique quantity at an exponential rate.

Original: We thus have Θ ( L ) ( R ) ( V , V ) 
Revised: We thus have Θ ( l ) ( r ) ( V , V ) ≈

Original: ≈ ⃗π ( G ) T ( RL⃗v ) 1 n × n as L → ∞ , where 1 n × n is the ( n × n )- dimensional matrix whose elements are one.
Revised: (cid:126)π ( G ) T ( Rl(cid:126)v ) 1 n × n as l → ∞ , where 1 n × n is an ( n × n )-dimensional matrix whose entries are one.

Original: The exponential convergence rate of Θ ( L ) ( R ) ( V , V ) implies that the trainability of infinitely wide GCNs degenerate dramatically, as shown in following corollary.
Revised: The exponential convergence rate of Θ ( l ) ( r ) ( G ) implies that the trainability of inﬁnitely-wide GCNs degenerates dramatically, as stated below.

Original: Corollary 1 (Trainability of Ultra-Wide GCNs) .
Revised: Corollary 1 (Trainability of Ultra-Wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth L ,  non-linear transformations number R , an MSE loss, and a Lipchitz activation ϕ ( x ) , trained with gradient descent on a node classification task.
Revised: Consider a GCN of the form (6) and (7), with depth l , number of non-linear transformations r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.

Original: Then the output function follows,
Revised: Then, the output function follows,

Original: [Equation]
Revised: [Equation]

Original: where Y is the label with respect to node V .
Revised: where Y is the label with respect to node V .

Original: Then,when L → ∞ , Θ ( L ) ( R ) ( V , V ) is singular and there exist a constant C > 0 such that for all t > 0 , ∥ f t ( V ) − Y∥ > C .
Revised: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ . Moreover, there exists a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .

Revised: We leave proof in Appendix C.

Original: According to the above corollary, as L → ∞ , the GNTK matrix will become a singular matrix.
Revised: According to the above corollary, as l → ∞ , the GNTK matrix would become a singular matrix.

Original: This would lead to a discrepancy between output f t ( V ) and label Y , which means GNTK loses the ability to fit the label.
Revised: This would lead to a discrepancy between output f t ( V ) and label Y , which means GNTK loses the ability to ﬁt the label.

Original: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classification tasks.
Revised: Therefore, an ultra-wide GCN with a large depth cannot be trained successfully on node classiﬁcation tasks.

Original:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION
Revised:  T OWARDS D EEPENING G RAPH N EURAL N ETWORKS 4. T HEORETICAL A NALYSIS ON R ESIDUAL C ONNECTION

Original: We have  characterized the trainability of vanilla GCNs through GNTKs and show that the trainability of ultra-wide GCNs drops at an exponential rate.
Revised: We have so far characterized the trainability of vanilla GCNs through the GNTK and showed that the trainability of ultra-wide GCNs drops at an exponential rate.

Original: Recently, considerable efforts have been made to deepen GCNs, of which residual connection-based techniques are widely applied to resolve the oversmoothing problem (Li et al., 2019).
Revised: Recently, considerable efforts have been made to deepen GCNs, among which residual connection-based techniques are widely applied to resolve the over-smoothing problem (Li et al., 2019).

Original: We  apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.
Revised: We now apply our theoretical framework to analyze to what extent residual connection techniques could alleviate the trainability loss problem.

Original: We first consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,
Revised: We ﬁrst consider residual connection in aggregation, in which the propagation of the GNTK can be formulated as,

Original: [Equation]
Revised: [Equation]

Original: where 0 < δ < 1, and C > 0 .
Revised: where 0 < δ < 1 .

Original: Taking Equation (12) as a new aggregation operation matrix, ⃗ Θ ( l ) ( G )=
Revised: Taking equation (12) as a new aggregation process, then (cid:126) Θ ( l ) ( G )

Original:  ˜
Revised: = ˜

Original: A ( G ) ⃗ Θ ( l − 1) ( G ) ,
Revised: A ( G ) (cid:126) Θ ( l − 1) ( G ) ,

Original: where ˜ A ( G ) =
Revised: where ˜ A ( G ) =

Original: (1 − δ )
Revised: (1 − δ )

Original: A ( G ) + δI .
Revised: A ( G ) + δI .

Original: We prove that ˜
Revised: We prove that ˜

Original: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix
Revised: A ( G ) is also a transition matrix with a greater second largest eigenvalue compared to the original matrix

Original: A ( G ) .
Revised: A ( G ) .

Original: Theorem 2 (Convergence rate for residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate of residual connection in aggregation) .

Original: Consider a GNTK of the form (12) with non-linear transformation of the form (7).
Revised: Consider a GNTK of nonlinear transformation (9) and residual connection (12).

Original: Then with a stationary vector ˜ ⃗π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <
Revised: A ( G ) , there exist constants 0 <

Original: ˜ α < 1 and C > 0 , and constant vectors ⃗v and ⃗v ′ depending on
Revised: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u ′ ) − ˜ ⃗π ( G ) T (cid:0) Rl⃗v + ⃗v ′ (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: According to (Rosenthal, 1995), we have α ∈ [ λ 2 , 1) and ˜ α ∈ [˜ λ 2 , 1) .

Original: The result of ˜ λ 2 > λin Theorem 2 causes ˜ α has a smaller value range, thus ˜ α tends to be greater than α .

Original: The detailedthe proof of theorem 2 and the relationship between α and λ 2 can be found in Appendix C.1.
Revised: The detailed proof of Theorem 2 and the relationship between convergence rate and second largest eigvenvalue can be found in Appendix D.1.

Original: This theorem implies  a residual connection in aggregation tends to slow down the convergence rate. Later, we show that ˜ α > α empirically for the GNTK.
Revised: This theorem implies that a residual connection in aggregation can slow down the convergence rate, which is consistent with empirical observations that residual connection can help deepen GCNs.

Original: Then we consider residual connection only applied on non-linear transformations (MLP).
Revised: Then, we consider residual connection only applied on non-linear transformation (MLP).

Original: In this case, the recursive equation for the corresponding GNTK can be expressed as,
Revised: In this case, the recursive equation for the corresponding GNTK can be expressed as,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: This formula is similar to the vanilla GNTK in the infinitely wide limit.
Revised: This formula is similar to the vanilla GNTK in the inﬁnitely-wide limit.

Original: Only an additional residual term appears according tothe residual connection.
Revised: Only an additional residual term appears according to residual connection.

Original: It turns out that this term may not help slow down the convergence rate for non-linear transformation.
Revised: It turns out that this term may not help slow down the convergence rate for non-linear transformation.

Original: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate of GNTK with residual connection between transformations) .

Original: Consider a GNTK  the form (8) and (13).
Revised: Consider a GNTK of the form (8) and (13).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution ⃗π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors ⃗v and ⃗v ′ depending on R , such that,
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on R , such that , (cid:12)(cid:12) Θ

Original: [Equation]
Revised: [Equation]

Original: Note that α in this theorem is the same that in Theorem 1.
Revised: Note that α in Theorem 3 is the same as in Theorem 1.

Original: The proof of the above theorem can be found in Appendix C.2.
Revised: The proof of Theorem 3 can be found in Appendix D.2.

Original: Theorem 3 demonstrates that adding residual connection in MLP cannot even reduce the convergence speed of trainability.
Revised: Theorem 3 demonstrates that adding residual connection in MLP can not even reduce the convergence rate of trainability.

Original: Finally, we considerthe residual connection applied to both aggregation and non-linear transformation simultaneously:
Revised: Finally, we consider residual connection applied to both aggregation and non-linear transformation simultaneously:

Original: Corollary 2 (Convergence rate for GNTK with residual connection in aggregation and transformation) .
Revised: Corollary 2 (Convergence rate of GNTK with residual connection in aggregation and transformation) .

Original: Consider a GNTK of the form (12) and (13).
Revised: Consider a GNTK of the form (12) and (13).

Original: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ ⃗π ( G ) , there exist constants 0 < ˜
Revised: If ˜ A ( G ) is irreducible and aperiodic, with a stationary distribution ˜ (cid:126)π ( G ) , there exist constants 0 < ˜

Original: α < 1 and C > 0 , and constant vectors ⃗v,⃗v ′ ∈ R n 2 × 1 depending on R , such that 
Revised: α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48) ∈ R n 2 × 1 depending on R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) −

Original: [Equation]
Revised: [Table]

Original: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE
Revised: 4. A N EW S AMPLING M ETHOD : C RITICAL D ROP E DGE

Original: Residual connection is a technique designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay problem.
Revised: Residual connection is a technique designed from a layer-wise perspective, but it has limited abilities to mitigate the exponential decay problem.

Original: To better solve the problem of trainability loss, we need to look deeper into the  cause of the problem, which is the transition matrix corresponding to the aggregation operation.
Revised: To better resolve the problem of trainability loss, we need to look deeper into the root cause of the problem – the transition matrix corresponding to the aggregation operation.

Original: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected, which is a typical attribute for graphs used on both node and graph classification tasks.
Revised: A necessary condition for matrix A ( G ) to be a probability transition matrix is that graph G is connected, which is a typical trait for graphs used on both node and graph classiﬁcation tasks.

Original: Therefore, breaking the connectivity condition is a promising direction.
Revised: Thus, breaking the connectivity condition is a promising direction.

Original: Interestingly, the critical percolation theory (Huang et al., 2018; Stauffer & Aharony, 2018) provides useful insights to better solve the exponential decay problem.
Revised: Interestingly, the critical percolation theory (Huang et al., 2018; Stauffer & Aharony, 2018) provides useful insights to better solve the exponential decay problem.

Original: On a finite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.
Revised: On a ﬁnite complete graph of n nodes, there exist E t = n ( n − 1) / 2 edges between all pairs of nodes.

Original: A random graph ˆ G is achieved by randomly and uniformly retaining some edges from the complete graph with the edge probability as p = | E | E t , where | E | is the number of edges retained in the random graph.
Revised: A random graph ˆ G is achieved by randomly and uniformly retaining some edges from the complete graph with an edge probability as p = | E | E t , where | E | is the number of edges retained in the random graph.

Original: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c
Revised: In this way, the critical percolation can be realized in the random graph with a critical edge probability p c

Original: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).
Revised: = 1 / ( n − 1) (Erd˝os & R´enyi, 1961).

Original: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there is an path from some fixed point to a distance decreases polynomially:
Revised: In the thermodynamic limit of n → ∞ , the critical random graph exhibits critical connectivity: the probability that there exists an path from some ﬁxed point to a distance decreases polynomially:

Original: Proposition 1 (Critical Connectivity in Random Graph (Erd˝os & R´enyi, 1961)) .
Revised: Proposition 1 (Critical Connectivity in Random Graph (Erd˝os & R´enyi, 1961)) .

Original: Suppose a random graph ˆ G has n nodes with a constant edge probability p .
Revised: Suppose a random graph ˆ G has n nodes with a constant edge probability p .

Original: (1) If p < p c , then almost every random  graph is such  its largest component 1 has O (log n ) ; (2) If p > p c , the random graph has a giant component of (1 − α p + o (1)) n , where α p <1 ; (3) If p = p c , thenthe maximal size of a component of almost every graph has order n 2 / 3 .
Revised: (1) If p < p c , then almost every random [Figure] graph is such that its largest component 1 has O (log n ) ; (2) If p > p c , the random graph has a giant component of (1 − α p + o (1)) n , where α p < If p = p c , then

Original: [Figure]
Revised: the

Original:  The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate, thus can solve the problem of exponentially dropping trainability.
Revised: maximal size of a component of almost every graph has order n The proposition above implies that the information transforms in the critical random graph at a polynomial rate rather than an exponential rate.

Original: Inspired by this result, we design a graph-dependent and connectivity-aware sampling algorithm.
Revised: This inspires us to solve the problem of exponentially dropping trainability through designing a graph-dependent and connectivity-aware sampling algorithm called Critical DropEdge.

Original: In particular, given a graph G , we randomly drop some edges and retain the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.
Revised: In particular, given a graph G , we randomly drop some edges and retain the number of edges as E r = E t · p c = n/ 2 , to approximate the critical graph.

Original: Note that a related work called

Original:  DropEdge (Rong et al., 2019), which randomly removes a certain number of edges from the input graph. Different from them, our method fixes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | . 
Revised: Unlike DropEdge (Rong et al., 2019) that randomly removes a certain number of edges from the input graph, our method ﬁxes the edge preserving percentage as ρ = E r / | E | = | V | 2 | E | . It is worth noting that

Original: On the contrary, DropEdgemethod may choose the edge probability as p < p c where information can only be passed to a distance of O (log n ) in the graph, or p > p c in which the exponential  trainability problem just exists.
Revised: DropEdge may choose the edge probability as p < p c where information can only be passed within a distance of O (log n ) in the graph, or p > p c where the exponential dropping trainability problem may occur.

Original: Our method is inspired by the critical percolation theory and thus it is named as Critical DropEdge.

Original: A further discussion of the relationship between Proposition 1 and  trainability of  corresponding GNTK can be found inthe Appendix D.
Revised: A further discussion of the relationship between Proposition 1 and the trainability of the corresponding GNTK can be found in Appendix E.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: [Figure] In this section, weillustrate empirically the theoretical resultsobtained in Section 3 and present the performance of the proposed Critical DropEdge on node classification.
Revised: In this section, we empirically verify our theoretical results and validate the proposed Critical DropEdge method.

Original: Details of these datasets  are summarized in Table 3 inAppendix E.1.
Revised: Details of four real-world graph datasets used are summarized in Table 3 in

Original: We use a bioinformatics dataset (i.e., MUTAG) to verify the theoretical results for GNTK. The remaining three citation network datasets (i.e., Cora, Citeseer, and Pubmed) are used to verify the trainability of wide GCNs and the performance of the proposed Critical Dropdege on node classification tasks.
Revised: Appendix F.1.

Original: 5. C ONVERGENCE R ESULTS OF GNTK S
Revised: 5. C ONVERGENCE R ESULTS OF GNTK S

Original: Theorems 1, 2, and 3 providea theoretical convergence rate for the (residual) GNTK.
Revised: Theorems 1-3 provide theoretical convergence rates for the (residual) GNTK.

Original: We demonstrate the corresponding numerical verification in Figure 1.
Revised: We show the corresponding numerical veriﬁcation in Figure 1.

Original: We select a graph  from the MUTAG dataset and generate the GNTK  with ReLU activa- tion, with R = 3 , and L = 300 .
Revised: We select a graph randomly from the MUTAG dataset and generate the GNTK using the implementation of Du et al. (2019), with ReLU activation, R = 3 , and L = 300 .

Original: Figure 1 (a,b,c) show  all elements of normalized GNTKs converge to an identical value as the depth goes larger.
Revised: Figure 1 (a)-(c) show that all elements of normalized GNTKs converge to an identical value as the depth goes larger.

Original: Figure 1 (d,e,f) further illustrate the convergence rate ofthe GNTK is exponential, as reflected by  theorems.
Revised: Figure 1 (d)-(f) further illustrate the convergence rate of GNTK is exponential, as reﬂected by our theorems.

Original: By comparingthe convergence rate, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with Theorem 2.
Revised: By comparing convergence rates, we conclude that the residual connection in aggregation can slow down the convergence rate, which is consistent with Theorem 2.

Original: 5. T RAINABILITY OF W IDE GCN S
Revised: 5. T RAINABILITY OF W IDE GCN S

Original: We further examine whether ultra-wide GCNs can be trained successfully on node classification.
Revised: We further examine whether ultra-wide GCNs can be trained successfully for node classiﬁcation.

Original: We conduct experiments on a GCN (Kipf & Welling, 2016a), where we apply a width of 1000 at each hidden layer and the depth ranging from 2 to 29 .
Revised: We conduct experiments on a GCN (Kipf & Welling, 2016a), where we apply a width of 1000 at each hidden layer and the depth ranging from 2 to 29 .

Original: Figure 2 shows the training and test accuracy on various datasets after 300 training epochs.
Revised: Figure 2 shows the training and test accuracy on Cora, Citesser and Pubmed after 300 training epochs.

Original: These results show the dramatic drop in both training and test accuracy as the depth grows, confirming that wide GCNs lose trainability significantly in the large depth on node classification as revealed by Corollary 1.
Revised: These results show a dramatic drop in both training and test accuracy as the depth grows, conﬁrming that wide GCNs lose trainability signiﬁcantly in the large depth on node classiﬁcation, as revealed by Corollary 1.

Original: 5. E VALUATION OF C RITICAL D ROP E DGE
Revised: 5. P ERFORMANCE OF C RITICAL D ROP E DGE

Original:  We apply Critical DropEdge  to both infinite-width and finite-width GCNs on node classification with a semi-supervised setting.
Revised: [Figure] We apply Critical DropEdge (referred to as CDropEdge) to both inﬁnitely-wide and ﬁnitelywide GCNs on node classiﬁcation under a semisupervised setting.

Original: For infinite-width GCNs, we consider two backbones: GCN (Kipf & Welling, 2016a) and
Revised: For inﬁnitely-wide GCNs, we consider two backbones: GCN (Kipf & Welling, 2016a) and

Original: JKNet (Xu et al., 2018b), while we use three backbones (GCN, JKNet, IncepGCN (Rong et al., 2019)) for finite-width GCNs.
Revised: JKNet (Xu et al., 2018b), while we use three backbones (GCN, JKNet, IncepGCN (Rong et al., 2019)) for ﬁnitely-wide GCNs.

Original: Thecorresponding results for the infinite-width GCNs can be found in Appendix F.1.
Revised: The results for inﬁnitely-wide GCNs can be found in Appendix G.1.

Original: As for finite-width GCNs, we use the hyper-parameter reported in Rong et al.
Revised: As for ﬁnitely-wide GCNs, we use the hyper-parameters reported in Rong et al.

Original: (2019) to obtain the results for DropEdge.
Revised: (2019) to obtain the results for DropEdge.

Original: On the other hand, we perform a random hyper-parameter search and fix the edge preserving percent as ρ ( G ) =
Revised: For C-DropEdge, we perform a random hyper-parameter search and ﬁx the edge preserving rate as ρ ( G ) =

Original: | V | 2 | E | for C-DropEdge.
Revised: | V | 2 | E | .

Original: In addition, we introduce a baseline method named DGN (Zhou et al., 2020) which is a normalized-based approach.
Revised: In addition, we compare with DGN (Zhou et al., 2020), a normalization-based baseline.

Original: The backbone of DGN includes GCN (Kipf & Welling, 2016a) and GAT (Veli ˇ ckovi ´ 
Revised: The backbone of DGN includes GCN (Kipf & Welling, 2016a) and GAT (Veli ˇ ckovi ´ c et al., 2017), and we report best performance across backbones.

Original: c et al., 2017). Table 1 summarizes the performance  on three citation network datasets and one co-author network dataset with finite-width GCNs.
Revised: Table 1 summarizes node classiﬁcation performance of ﬁnitely-wide GCNs on three citation networks (Cora, Citesser, Pubmed) and one co-author network (Physics).

Original: All the experiments are conducted in the semi-supervised setting, and more details regarding dataset can be found in Appendix E.1.

Original: In this table, we report the performance of GNNs with 4/8/16/32 layers and implementation details can be found in Appendix E.2.
Revised: In this table, we report the performance of GNNs with 4/8/16/32 layers and implementation details can be found in Appendix F.2.

Original: In particular, we report the best performance across different backbones for DropEdge, C-DropEdge and DGN, and leave separate results with different backbones in Appendix F.2.
Revised: In particular, we report the best performance across different backbones for DropEdge, C-DropEdge and DGN, and leave separate results with different backbones in Appendix G.2.

Original: For each hyperparameter, we run 10 times and report the mean and standard variance.
Revised: For each hyper-parameter, we run 10 times and report the mean and standard variance.

Original: It is shown that C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.
Revised: It is shown that C-DropEdge consistently outperforms GCN, DGN, and DropEdge, especially when the model is deep.

Original: Besides, our C-DropEdge method can achieve smaller error bars, indicating that our method is more robust than DropEdge.
Revised: Besides, our C-DropEdge method can achieve smaller error variances, indicating that it is more robust than DropEdge.

Original: Finally, we perform an empirical study of whether the proposed drop ratevalue is indeed optimal or close to optimal to strengthen our results.
Revised: Finally, we perform an empirical study of whether the proposed edge dropping rate is indeed optimal or close to optimal to strengthen our results.

Original: To this end, we conducted experiments on dropping edges at various rates (sampling percent) and verified that the rate of Critical DropEdge  can achieve the results closed to optimal, as shown in Table 2.
Revised: To this end, we conduct experiments on dropping edges at various rates (sampling percent) and veriﬁed that the rate of Critical DropEdge highlighted in bold can achieve the results close to optimal, as shown in Table 2.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Neural Tangent Kernel.
Revised: Neural Tangent Kernel.

Original: NTKs are used to describe the dynamics of infinitely-wide networks during gradient descent training.
Revised: NTKs are used to describe the dynamics of inﬁnitely-wide networks during gradient descent training.

Original: In the infinite width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018;
Revised: In the inﬁnite-width limit, NTK converges to an explicit limiting kernel; besides, it stays constant during training, providing a convergence guarantee for overparameterized networks (Jacot et al., 2018;

Original: Lee et al., 2019a; Arora et al., 2019; Yang, 2019; Huang & Yau, 2019; Allen-Zhu et al., 2019; Du et al., 2018; Zou et al., 2018).
Revised: Lee et al., 2019a; Arora et al., 2019; Yang, 2019; Huang & Yau, 2019; Allen-Zhu et al., 2019; Du et al., 2018; Zou et al., 2018).

Original: Besides, NTK has been applied to various architecture and brought a wealth of results, such as orthogonal initialization (Huang et al., 2020a), convolutions (Arora et al., 2019), graph neural networks (Du et al., 2019), attention (Hron et al., 2020), and see (Yang, 2020) for a summary.
Revised: Besides, NTK has been applied to various architectures and brought a wealth of results, such as orthogonal initialization (Huang et al., 2020a), convolutions (Arora et al., 2019), graph neural networks (Du et al., 2019), attention (Hron et al., 2020), and refer to (Yang, 2020) for a summary.

Original: As for graph networks, GNTK helps us understand how GNN learn a class of smooth functions on graphs (Du et al., 2019) and how GNN extrapolate differently from multi-layer perceptron (Xu et al., 2020).
Revised: As for graph networks, GNTK helps us understand how GNNs learn a class of smooth functions on graphs (Du et al., 2019) and how they extrapolate differently from multi-layer perceptron (Xu et al., 2020).

Original: Deep Graph Neural Networks.
Revised: Deep Graph Neural Networks.

Original: Since  GNNs suffer from  over-smoothing problem, a large and growing body of literature has made efforts  deepening graph neural networks.
Revised: Since deep GNNs suffer from the over-smoothing problem, a large and growing body of literature has made efforts in deepening GNNs.

Original: There is a line of methods that resort to residual connection to keep their feature expressivity in deep layers.
Revised: There is a line of methods that resort to residual connection to retain their feature expressivity in deep layers.

Original: Xu et al.
Revised: Xu et al.

Revised: (2018b)

Original: [Table] [Table]
Revised: [Table] [Table]

Original: (2018b) used a skip connection method that combined with node representations from different neighborhood ranges to preserve the locality ofthe node representations.
Revised: use skip connection along with node representations from different neighborhood ranges to preserve the locality of node representations.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2018) derived a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.
Revised: (2018) derive a personalized propagation of neural predictions (PPNP) based on personalized Pagerank.

Original: Other similar works using residual connection method could also be seen in (Li et al., 2019;
Revised: Other similar works using residual connection can also be seen in (Li et al., 2019;

Original: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).
Revised: Gong et al., 2020; Chen et al., 2020; Liu et al., 2020).

Original: Another line of approachesmake attempt to tackle the issue by regularization mechanism, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2019), and group normalization (Zhou et al., 2020).
Revised: Another line of approaches tackle the issue via regularization mechanisms, such as node/edge dropping (Hou et al., 2019; Rong et al., 2019), batch normalization (Dwivedi et al., 2020), pair normalization (Zhao & Akoglu, 2019), and group normalization (Zhou et al., 2020).

Original: Recently, a series of latest works (Loukas, 2019; Zeng et al., 2020; Li et al., 2020;
Revised: Recently, a series of latest works (Loukas, 2019; Zeng et al., 2020; Li et al., 2020;

Original: Cong et al., 2021; Huang et al., 2020b) explored the underlying reasons for performance degradation towards mitigation solutions.
Revised: Cong et al., 2021; Huang et al., 2020b) explore the underlying reasons for performance degradation towards mitigation solutions.

Original:  C ONCLUSION AND D ISCUSSION
Revised:  C ONCLUSION AND D ISCUSSION

Original: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.
Revised: In this work, we have characterized the asymptotic behavior of GNTK to measure the trainability of wide GCNs in the large depth.

Original: We prove that the trainability drops at an exponential rate due to the aggregation operation.
Revised: We prove that the trainability drops at an exponential rate due to the aggregation operation.

Original: Furthermore, we apply our theoretical framework to investigatethe extent to which residual connection -based techniques could deepen GCNs.
Revised: Furthermore, we apply our theoretical framework to investigate to what extent residual connection-based techniques help deepen GCNs.

Original: We demonstrate that these techniques can only slow down the decay speed, but are unable to solve the exponential decay problem in essence.
Revised: We demonstrate that these techniques can only slow down the decay rate, but are unable to solve the exponential decay problem in essence.

Original: To overcome thedropping trainability problem we further propose Critical DropEdge illuminated by our theoretical framework.
Revised: To overcome the trainability loss problem, we further propose Critical DropEdge illuminated by our theoretical framework.

Original: The experimental results confirm that our method can address the trainability problems of deep GCNs.
Revised: The experimental results conﬁrm that our method can mitigate the trainability problems of deep GCNs.

Original: Future research directions include designing a critical node-centered method, so as to make better use of node information.
Revised: Future research directions include designing a critical node-centric method so as to make better use of node information.

Original: R EPRODUCIBILITY S TATEMENT
Revised: R EPRODUCIBILITY S TATEMENT

Original: To ensure the results and conclusions of our paper are reproducible, we make the following efforts:
Revised: To ensure the results and conclusions of our paper are reproducible we make the following efforts:

Original: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 3 and 4; Appendix B and C.
Revised: Theoretically, we state the full set of assumptions and include complete proofs of our theoretical results in Section 3 and 4; Appendix B and D.

Original: Experimentally, we provide our code, and instructions needed to reproduce the main experimental results.
Revised: Experimentally, we provide our code, and instructions needed to reproduce the main experimental results.

Original: And we specify all the training and implementation details in Section 5 and Appendix E and
Revised: And we specify all the training and implementation details in Section 5 and Appendix F and

Original: F. Besides, we independently run experiments 10 times and report the mean and standard deviation
Revised: G. Besides, we independently run experiments 10 times and report the mean and standard deviation

Revised: A CKNOWLEDGMENTS

Revised: We thank the anonymous reviewers for useful suggestions to improve the paper.

Revised: This work was carried out in collaboration with Digital Research Centre Denmark – DIREC, supported by Innovation

Revised: Fund Denmark.

Original: A APPENDIX: A G ENERAL GCN CONSIDERED IN THIS WORK
Revised: A APPENDIX: A G ENERAL GCN CONSIDERED IN THIS WORK

Original: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.
Revised: GCNs iteratively update node features through aggregating and transforming the representation of their neighbors.

Original: Figure 3 illustrates an overview of the information propagation in a general GCN.
Revised: Figure 3 illustrates an overview of the information propagation in a general GCN.

Original: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK
Revised: B APPENDIX: P ROOFS FOR T HEOREMB. C ONVERGENCE OF A GGREGATION GNTK

Original: In theorem 1 we demonstrate that the matrix A ( G ) is a transition matrix, we first prove this proposition.
Revised: In theorem 1 we demonstrate that the matrix A ( G ) is a transition matrix, we ﬁrst prove this proposition.

Original: To this end, we block the non-linear transformation by setting R = 0 .
Revised: To this end, we block the non-linear transformation by setting R = 0 .

Original: Then and the propagation of GNTK can be expressed as,
Revised: Then and the propagation of GNTK can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite Equation (14) as the following format,
Revised: In order to facilitate calculation, we rewrite Equation (14) as the following format,

Original: [Equation]
Revised: [Equation]

Original: (15) where ⃗ Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and the matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.
Revised: (15) where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 is the result of being vectorized, and the matrix A ( G ) ∈ R n 2 × n 2 is a square matrix.

Original: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( V , V ) in the following lemma,
Revised: We show the key result that A ( G ) is a probability transition matrix, and the limiting behavior of Θ ( l ) ( V , V ) in the following lemma,

Original: Lemma 1 (Convergence of Aggregation) .
Revised: Lemma 1 (Convergence of Aggregation) .

Original: Assume R = 0 , then
Revised: Assume R = 0 , then

Original: [Equation]
Revised: [Equation]

Original: where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) ⃗π ( G ) = ⃗π ( G ) .
Revised: where π ( G ) ∈ R n 2 × 1 , satisfying A ( G ) (cid:126)π ( G ) = (cid:126)π ( G ) .

Original: Proof.
Revised: Proof.

Original: When R = 0 , Equation (14) reduces to,
Revised: When R = 0 , Equation (14) reduces to,

Original: [Equation]
Revised: [Equation]

Original: In order to facilitate calculation, we rewrite the above equation in the format of matrix,
Revised: In order to facilitate calculation, we rewrite the above equation in the format of matrix,

Original: [Equation]
Revised: [Equation]

Original: where ⃗ Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.
Revised: where (cid:126) Θ ( l ) ( G ) ∈ R n 2 × 1 , is the result of being vectorized.

Original: Thus the matrix operation A ( l ) ( G ) ∈
Revised: Thus the matrix operation A ( l ) ( G ) ∈

Original: R n 2 × n 2 .
Revised: R n 2 × n 2 .

Original: It is obvious that,
Revised: It is obvious that,

Original: [Equation]
Revised: [Equation]

Original: This implies that the aggregate operation is the same for each layer.
Revised: This implies that the aggregate operation is the same for each layer.

Original: The next step is to prove A ( G ) is a stochastic matrix (transition matrix):(cid:88)
Revised: The next step is to prove A ( G ) is a stochastic matrix (transition matrix):

Original: [Equation]
Revised: [Equation]

Original: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrix,
Revised: According to Equation (14), A ( G ) can be expressed as a Kronecker product of two matrix,

Original: [Equation]
Revised: [Equation]

Original: where B ( G ) , C ( G ) ∈ R n × n .
Revised: where B ( G ) , C ( G ) ∈ R n × n .

Original: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor
Revised: We then analyse the two matrices separately: (1) B ( G ) is a diagonal matrix, which corresponds to the factor

Original: [Equation]
Revised: [Equation]

Original: (2) The element of matrix C ( G ) is determined by whether there is a edge between two vertexes,
Revised: (2) The element of matrix C ( G ) is determined by whether there is a edge between two vertexes,

Original: [Equation]
Revised: [Equation]

Original: where ˜ δ ij = 1 if i ==
Revised: where ˜ δ ij = 1 if i ==

Original: j or there is edge between vertex i and j , else ˜ δ ij = 0 .
Revised: j or there is edge between vertex i and j , else ˜ δ ij = 0 .

Original: We then use the property of matrix B and C before Kronecker product,
Revised: We then use the property of matrix B and C before Kronecker product,

Original: [Equation]
Revised: [Equation]

Original: According to the definition of Kronecker product, we have,
Revised: According to the deﬁnition of Kronecker product, we have,

Original: [Equation]
Revised: [Equation]

Original: where i = a + ( c − 1) n , and j = b + ( d − 1) n .
Revised: where i = a + ( c − 1) n , and j = b + ( d − 1) n .

Original: So far, we have proved that A ( G ) is a stochastic matrix.
Revised: So far, we have proved that A ( G ) is a stochastic matrix.

Original: According to the Perron-Frobenius Theory, a stationary probability vector ⃗π ( G ) ∈ R n 2 × 1 is defined as a distribution, that does not change under application of the transition matrix; that is, it is defined as a probability distribution which is also a eigenvector of the probability matrix, associated with eigenvalue 1:
Revised: According to the Perron-Frobenius Theory, a stationary probability vector (cid:126)π ( G ) ∈ R n 2 × 1 is deﬁned as a distribution, that does not change under application of the transition matrix; that is, it is deﬁned as a probability distribution which is also a eigenvector of the probability matrix, associated with eigenvalue 1:

Original: [Equation]
Revised: [Equation]

Original: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.
Revised: Note that the spectral radius of every stochastic matrix is at most 1 by Gershgorin circle theorem.

Original: Thus the convergence rate is governed by the second largest eigenvalue.
Revised: Thus the convergence rate is governed by the second largest eigenvalue.

Original: Since lim l →∞ A ( l ) ij ( G ) = ⃗π j ( G ) , we have,
Revised: Since lim l →∞ A ( l ) ij ( G ) = (cid:126)π j ( G ) , we have,

Original: [Equation]
Revised: [Equation]

Original: where Π( G ) =  
Revised: where Π( G ) =  [Equation]

Revised: (cid:126)π

Original: [Equation]
Revised: [Equation]

Original: ⃗π ( G ) T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.
Revised: T   is the n 2 × n 2 matrix all of whose rows are the stationary distribution.

Original: Then, we can see that every element in ⃗ Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,
Revised: Then, we can see that every element in (cid:126) Θ ( l ) ( G ) converges exponentially to an identical value, depending on the stationary distribution and initial state,

Original: [Equation]
Revised: [Equation]

Original: Remark 1.
Revised: Remark 1.

Original: This lemma can be extended to the multi-graph setting, where matrix A ( G, G ′ ) ∈
Revised: This lemma can be extended to the multi-graph setting, where matrix A ( G, G (cid:48) ) ∈

Revised: R nn (cid:48)

Original: R nn ′ × nn ′ with respect to a pair of graphs G, G ′ is a transition matrix as well, and n ′ is the number of vertexes in graph G ′ .
Revised: × nn (cid:48) with respect to a pair of graphs G, G (cid:48) is a transition matrix as well, and n (cid:48) is the number of vertexes in graph G (cid:48) .

Original: B. C ONVERGENCE OF MLP GPK
Revised: B. C ONVERGENCE OF MLP GPK

Original: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.
Revised: Before we prove Theorem 1, we introduce the result for the Gaussian Process kernel Σ ( l ) ( r ) ( G ) of a pure MLP.

Original: By doing so, we consider a network with only non-linear transformation, which is known as a pure MLP.
Revised: By doing so, we consider a network with only non-linear transformation, which is known as a pure MLP.

Original: This leads to Σ ( l ) ( r ) ( u, u ′ ) =
Revised: This leads to Σ ( l ) ( r ) ( u, u (cid:48) ) =

Original: Σ ( r ) ( u, u ′ ) , where we use subscript ( r ) to denote the layer index.
Revised: Σ ( r ) ( u, u (cid:48) ) , where we use subscript ( r ) to denote the layer index.

Original: And we rewrite the propagation function for GPK as follows,
Revised: And we rewrite the propagation function for GPK as follows,

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,
Revised: and the variance ˜Σ ( r ) ∈ R 2 × 2 is,

Original: [Equation]
Revised: [Equation]

Original: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.
Revised: The large depth behavior has been well studied in Hayou et al. (2019a), and we introduce the result when the edge of chaos is realized.

Original: In particular, we set the value of hyper-parameters to satisfy,
Revised: In particular, we set the value of hyper-parameters to satisfy,

Original: [Equation]
Revised: [Equation]

Original: where q ∗ is the fixed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.
Revised: where q ∗ is the ﬁxed point of diagonal elements in the covariance matrix, and (cid:82) D z = 1 √ 2 π (cid:82) dze − 12 z 2 is the measure for a normal distribution.

Original: For the ReLU activation, Equation (18) requires σ 2 w = 2 and σ 2 b = 0 .
Revised: For the ReLU activation, Equation (18) requires σ 2 w = 2 and σ 2 b = 0 .

Original: The key idea is to study the asymptotic behavior of the normalized correlation defined as,
Revised: The key idea is to study the asymptotic behavior of the normalized correlation deﬁned as,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 ( Proposition 1 and 3 in Hayou et al.
Revised: Lemma 2 ( Proposition 1 and 3 in Hayou et al.

Original: (2019a)) .
Revised: (2019a)) .

Original: Assume a network without aggregation, i.e., L = 0 , with a Lipschitz nonlinearity ϕ , then,
Revised: Assume a network without aggregation, i [Equation] e., L = 0 , with a Lipschitz nonlinearity φ , then,

Original: • ϕ ( x ) =
Revised: • φ ( x ) =

Original: ( x ) + , 1 − C r ( u, u ′ )
Revised: ( x ) + , [Equation] r ( u, u (cid:48) [Equation]

Original: ∼ 9 π 2 2 r 2 as r → ∞ • ϕ ( x ) = tanh( x ) , 1 − C r ( u, u′ ) ∼ βr as r → ∞ where β
Revised: ∞ • φ ( x ) = tanh( x ) , 1 − C [Equation] ( u, u

Original: = 2 (cid:82) D z
Revised: [Equation]

Original: [ ϕ ′ ( √ q ∗ z ) 2 ] q (cid:82) D z [ ϕ ′′ ( √ q ∗ z ) 2 ] .
Revised: as [Equation] where [Equation]

Original: Proof.
Revised: Proof.

Original: We first decompose the integral calculation in the Equation (16) into two parts, one is diagonal element and the other is non-diagonal element:
Revised: We ﬁrst decompose the integral calculation in the Equation (16) into two parts, one is diagonal element and the other is non-diagonal element:

Original: [Equation]
Revised: [Equation]

Original: where u=
Revised: where u=

Original: [Equation]
Revised: [Equation]

Original:  u
Revised: Σ ( r − 1) ( u, u ) z 1 , and u

Original: [Equation]
Revised: [Equation]

Original: u
Revised: =

Original: [Equation]
Revised: [Equation]

Original: z
Revised: −

Original: [Equation]
Revised: [Equation]

Original: and u
Revised: u,

Original: [Equation]
Revised: [Equation]

Original: =

Original: [Equation]

Original: , with C r − 1 ( u, u
Revised: , with C r − 1 ( u, u

Original: [Equation]
Revised: [Equation]

Original: u
Revised: u,

Original: [Equation]
Revised: [Equation]

Revised: ) / (cid:112) Σ ( r − 1) ( u, u )Σ ( r − 1) ( u (cid:48) , u (cid:48) ) .

Original: For simplicity, we define q r ( u ) = Σ ( r ) ( u, u ) , q r ( u ′ )
Revised: For simplicity, we deﬁne q r ( u ) = Σ ( r ) ( u, u ) , q r ( u (cid:48) )

Original: =
Revised: =

Original: Σ ( r ) ( u ′ , u ′ ) .
Revised: Σ ( r ) ( u (cid:48) , u (cid:48) ) .

Original: We then process the proof by dividing the activation ϕ ( x ) into two classes, namely, ReLU and Tanh.
Revised: We then process the proof by dividing the activation φ ( x ) into two classes, namely, ReLU and Tanh.

Original: ReLU activation, ϕ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: The recursive equation (16) for q r ( u ) reduces to,
Revised: The recursive equation (16) for q r ( u ) reduces to,

Original: [Equation]
Revised: [Equation]

Original: The edge of chaos condition σ2 w (cid:82) D z
Revised: The edge of chaos condition σ

Original:  [ ϕ ′ ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,
Revised: [Equation] z [ [Equation] ( √ q ∗ z )] 2 = 1 requires σ 2 w = 2 and σ 2 b = 0 for ReLU activation, which leads to,

Original: [Equation]
Revised: [Equation]

Original: Then the second integration for C r ( u, u ′ ) becomes,
Revised: Then the second integration for C r ( u, u (cid:48) ) becomes,

Original: [Equation]
Revised: [Equation]

Original: To investigate the propagation of C r ( u, u ′ ) , we set q r ( u ) = 
Revised: To investigate the propagation of C r ( u, u (cid:48) ) [Equation] we set q r ( u ) = q r ( u (cid:48) )

Original: q r ( u ′ ) = 
Revised: = q [Equation]

Original: q , and define,
Revised: and deﬁne,

Original: [Equation]
Revised: [Equation]

Original: q
Revised: q

Original: Let x ∈ [0 , 1] , the derivative of f ( x ) satisfies,
Revised: Let x ∈ [ [Equation]

Original: [Equation]
Revised: [Equation]

Original: This can be seen from a simple derivation, f′ ( x ) = (cid:90) D z(cid:90)
Revised: the derivative of f ( x ) satisﬁes,

Original: [Equation]
Revised: [Equation]

Original: z
Revised: This can be seen from a simple derivation,

Original: [Equation]
Revised: [Equation]

Original: Using an identity for Gaussian integration (cid:82) D z zg ( z ) = (cid:82) D z g ′ ( z ) yields,
Revised: Using an identity for Gaussian integration (cid:82) D z zg ( z ) = [Equation] ) yields,

Original: [Equation]
Revised: [Equation]

Original: Then the second derivative of f ( x ) becomes, f ′′ ( x ) =
Revised: Then the second derivative of f ( x ) becomes, f (cid:48)(cid:48) ( x ) =

Original: 1 π √ 1 − x 2 .
Revised: [Equation]

Original: So using the equation above and the condition f ′ (0) =
Revised: So using the equation above and the condition f (cid:48) (0) =

Original: 12 , we can get another formation of the derivative of f ( x ) ,
Revised: 12 , we can get another formation of the derivative of f ( x ) ,

Original: [Equation]
Revised: [Equation]

Original: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,
Revised: Because (cid:82) arcsin = x arcsin + √ 1 − x 2 and f (1) = 1 , then for x ∈ [0 , 1] ,

Original: [Equation]
Revised: [Equation]

Original: Substituting f ( x ) =
Revised: Substituting f ( x ) =

Original: C r ( u, u ′ ) and x = C r − 1 ( u, u ′ ) , into expression up here, we have,
Revised: C r ( u, u (cid:48) ) and x = C r − 1 ( u, u (cid:48) ) , into expression up here, we have,

Original: [Equation]
Revised: [Equation]

Original: Now we study the behavior of C r ( u, u ′ ) as r tends to infinity.
Revised: Now we study the behavior of C r ( u, u (cid:48) ) as r tends to inﬁnity.

Original: Using Taylor expansion, we have,
Revised: Using Taylor expansion, we have,

Original: [Equation]
Revised: [Equation]

Original: By induction analysis, the sequence C r ( u, u ′ ) is increasing to the fixed point 1.
Revised: By induction analysis, the sequence C r ( u, u (cid:48) ) is increasing to the ﬁxed point 1.

Original: Besides, we can replace x with C r ( u, u ′ ) ,
Revised: Besides, we can replace x with C r ( u, u (cid:48) ) ,

Original: [Equation]
Revised: [Equation]

Original: Let γ r = 1 − C r ( u, u ′ ) , then we have,
Revised: Let γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: so that,
Revised: so that,

Original: [Equation]
Revised: [Equation]

Original: As r tends to infinity, we have,
Revised: As r tends to inﬁnity, we have,

Original: [Equation]
Revised: [Equation]

Original: It means,
Revised: It means,

Original: [Equation]
Revised: [Equation]

Original: Therefore, we have,
Revised: Therefore, we have,

Original: [Equation]
Revised: [Equation]

Original: Tanh activation, ϕ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: For the argument of fixed point, we ask readers to refer to
Revised: For the argument of ﬁxed point, we ask readers to refer to

Original: Hayou et al.
Revised: Hayou et al.

Original: (2019a).
Revised: (2019a).

Original: Here we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to be 1 as the depth tends to be infinity lim r →∞ C r ( u, u ′ )
Revised: Here we only discuss the convergence rate of GPK, which means we directly assume that f ( x ) tends to be 1 as the depth tends to be inﬁnity lim r →∞ C r ( u, u (cid:48) )

Original: = 1 .
Revised: = 1 .

Original: For the function f ( x ) , a Taylor expansion near 1 yields,
Revised: For the function f ( x ) , a Taylor expansion near 1 yields,

Original: [Equation]
Revised: [Equation]

Original: where f ′ (1) =
Revised: where f (cid:48) (1) =

Original: σ 2 w (cid:82) D z [ ϕ ′ ( √ qz ) 2 ] , and f ′′ (1) =
Revised: σ 2 w (cid:82) D z [ φ (cid:48) ( √ qz ) 2 ] , and f (cid:48)(cid:48) (1) =

Original: σ 2 w q (cid:82) D
Revised: σ 2 w q (cid:82) D

Original: z
Revised: z

Original: [ ϕ ′′ ( √ qz ) 2 ] .
Revised: [ φ (cid:48)(cid:48) ( √ qz ) 2 ] .

Original: Denote γ r = 1 − C r ( u, u ′ ) , then we have,
Revised: Denote γ r = 1 − C r ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: where β
Revised: where β

Original: = 2 (cid:82)D z [ ϕ ′ ( √ q ∗ z ) 2 ] q (cid:82) D z [ ϕ
Revised: = 2 (cid:82)

Original: [Equation]
Revised: [Equation]

Original: ( √ q ∗ z ) 2 ] .
Revised: ( √ q ∗ z ) 2 ] .

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Thus, we have,
Revised: Thus, we have,

Original: [Equation]
Revised: [Equation]

Original: Lemma 2 shows the covariance matrix converges to a constant matrix at a polynomial rate of 1 /rfor ReLU and of 1 /r for tanh activation on the edge of chaos.
Revised: Lemma 2 shows the covariance matrix converges to a constant matrix at a polynomial rate of 1 /rfor ReLU and of 1 /r for tanh activation on the edge of chaos.

Original: This implies that a network without aggregation could retain its expressivity at a large depth.
Revised: This implies that a network without aggregation could retain its expressivity at a large depth.

Original: However, for general case, due to the aggregation, the rate of convergence will change from polynomial to exponential.
Revised: However, for general case, due to the aggregation, the rate of convergence will change from polynomial to exponential.

Original: This will make the trainability of the deep graph network problematic, which will be shown in the following section.
Revised: This will make the trainability of the deep graph network problematic, which will be shown in the following section.

Original: B. C ONVERGENCE OF GPK FOR GCN S
Revised: B. C ONVERGENCE OF GPK FOR GCN S

Original: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u ′ ) of a GCN in the infinite-width limit:
Revised: Then we formally characterize the convergence of Gaussian Process kernel Σ ( l ) ( r ) ( u, u (cid:48) ) of a GCN in the inﬁnite-width limit:

Original: Lemma 3.
Revised: Lemma 3.

Original: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector ⃗π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector ⃗v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that
Revised: If A ( G ) is ireducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) , then there exist constants 0 < α < 1 and C > 0 , and constant vector (cid:126)v ∈ R n 2 × 1 depending on the number of MLP iterations R , such that

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: We prove the result by induction method.
Revised: We prove the result by induction method.

Original: For l = 0 , according to the Cauchy-BuniakowskySchwarz Inequality
Revised: For l = 0 , according to the Cauchy-BuniakowskySchwarz Inequality

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization, such that,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization, such that,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Σ ( l ) ( r ) ( u, u ′ ) , then we have,[Equation] ( l ) ( r ) [Equation]
Revised: Assume the result is valid for Σ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: u

Original: [Equation]
Revised: [Equation]

Original: where C 0 is a constant satisfying 0 < C 0 < C .
Revised: where C 0 is a constant satisfying 0 < C 0 < C .

Original: Now we consider the distance between Σ ( l +1)
Revised: Now we consider the distance between Σ ( l +1)

Original: ( r ) ( u, u ′ ) and Cα l +1 .
Revised: ( r ) ( u, u (cid:48) ) and Cα l +1 .

Original: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To compute this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.
Revised: Σ ( l ) ( r ) → Σ ( l ) ( r +1) → · · · → Σ ( l ) ( R ) 2.

Original: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .
Revised: Σ ( l ) ( R ) → Σ ( l +1) (0) 3. Σ ( l +1) (0) → Σ ( l +1) (1) → · · · → Σ ( l +1) ( r ) .

Original: It is not hard to find that steps 1,3 correspond to non-linear transformation while step 2 corresponds with aggregation operation, we then characterize them one by one,
Revised: It is not hard to ﬁnd that steps 1,3 correspond to non-linear transformation while step 2 corresponds with aggregation operation, we then characterize them one by one,

Original: MLP Propagation
Revised: MLP Propagation

Original: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u ′ ) 
Revised: The assumption (cid:12)(cid:12) Σ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π

Revised: [Equation]

Original: − ⃗π ( G ) T ⃗v (cid:12)(cid:12) ≤ C 

Revised: G ) T (cid:126)v (cid:12)(cid:12) ≤ C

Original: α l implicitly implies that
Revised: α l implicitly implies that

Original: C r ( u, u ′ ) is close to one.
Revised: C r ( u, u (cid:48) ) is close to one.

Original: Because C r ( u, u ′ ) = 
Revised: Because C r ( u, u (cid:48) ) = [Equation]

Original: [Equation]
Revised: [Equation]

Original: u,
Revised: l

Original: [Equation]
Revised: [Equation]

Original: Σ ( l ) ( r ) ( u,u ) Σ ( l ) ( r ) ( u ′ ,u ′ ) , then,
Revised: r [Equation] [Equation] (cid:48) ) , then,

Original: [Equation]
Revised: [Equation]

Original: Because C 0 α l ≪ ⃗π ( G ) T ⃗v , we have C r ( u, u ′ )
Revised: Because C 0 α l (cid:28) (cid:126)π ( G ) T (cid:126)v , we have C r ( u, u (cid:48) )

Original: =
Revised: =

Original: 1+ O ( α l ) .
Revised: 1+ O ( α l ) .

Original: Recall the property of MLP propagation function f ( x ) for x =
Revised: Recall the property of MLP propagation function f ( x ) for x =

Original: C r ( u, u ′ ) , when C r ( u, u ′ ) is close to 1:
Revised: C r ( u, u (cid:48) ) , when C r ( u, u (cid:48) ) is close to 1:

Original: [Equation]
Revised: [Equation]

Original: This implies,
Revised: This implies,

Original: [Equation]
Revised: [Equation]

Original: With this result, we can further obtain,
Revised: With this result, we can further obtain,

Original: [Equation]
Revised: [Equation]

Original: where C 1 is a positive constant.
Revised: where C 1 is a positive constant.

Original: Repeat the proof process, we have a relation for Σ ( l ) ( R ) ( u, u ′ ) at the last step of 1, 
Revised: Repeat the proof process, we have a relation for Σ ( l ) ( R ) ( u, u (cid:48) ) at the last step of 1, [Equation]

Original: [Equation]
Revised: [Equation]

Original: Aggregation Propagation We go through an aggregation operation
Revised: Aggregation Propagation We go through an aggregation operation

Original: A ( G ) .
Revised: A ( G ) .

Original: In this case, we use a formation of matrix, and take Equation (20) into aggregation function, yields the result as follows,
Revised: In this case, we use a formation of matrix, and take Equation (20) into aggregation function, yields the result as follows,

Original: [Equation]
Revised: [Equation]

Original: where ⃗O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .
Revised: where (cid:126)O ( α l ) ∈ R n 2 × 1 denotes a vector in which every element is bounded by α l .

Original: According to the theorem 4.9 in Freedman (2017), we have A l Π( G ) ⃗v = Π( G ) ⃗v as l → ∞ .
Revised: According to the theorem 4.9 in Freedman (2017), we have A l Π( G ) (cid:126)v = Π( G ) (cid:126)v as l → ∞ .

Original: When l is finite, the error is of exponential decay.
Revised: When l is ﬁnite, the error is of exponential decay.

Original: Here we use 0 < α < 1 to denote the corresponding base number.
Revised: Here we use 0 < α < 1 to denote the corresponding base number.

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Finally, Repeat the result in step 1, we have,
Revised: Finally, Repeat the result in step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Remark 2.
Revised: Remark 2.

Original: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to infinity.
Revised: In the proof, there is a RC 1 term in each propagation of l → l + 1 , which may lead to explosion when l tends to inﬁnity.

Original: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) which is a smaller order compared to O ( α l ) .
Revised: Basically, this problem can be solved by a careful analysis, because the constant C 1 is associated with O ( α 32 l ) which is a smaller order compared to O ( α l ) .

Original: B. C ONVERGENCE OF THE GNTK
Revised: B. C ONVERGENCE OF THE GNTK

Original: Finally, we arrive at our main theorem:
Revised: Finally, we arrive at our main theorem:

Original: Theorem 1 (Convergence Rate of the GNTK) .
Revised: Theorem 1 (Convergence Rate of the GNTK) .

Original: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector ⃗π ( G ) ∈ R n 2 × 1 , where ⃗ Θ ( l ) (0) ( G ) =
Revised: If transition matrix A ( G ) ∈ R n 2 × n 2 is irreducible and aperiodic, with a stationary distribution vector (cid:126)π ( G ) ∈ R n 2 × 1 , where (cid:126) Θ ( l ) (0) ( G ) =

Original: A ( G ) ⃗ Θ ( l − 1) ( R ) ( G ) and ⃗ Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.
Revised: A ( G ) (cid:126) Θ ( l − 1) ( R ) ( G ) and (cid:126) Θ ( l ) ( r ) ( G ) ∈ R n 2 × 1 is the result of being vectorized.

Original: Then there exist constants 0 < α < 1 and C > 0 , and constant vectors ⃗v,⃗v ′ ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u ′ ) 
Revised: Then there exist constants 0 < α < 1 and C > 0 , and constant vectors (cid:126)v,(cid:126)v (cid:48) ∈ R n 2 × 1 depending on the number of MLP iterations R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − (cid:126)π ( G ) T (cid:0) Rl(cid:126)v

Original: − ⃗π ( G ) T (cid:0) Rl⃗v + ⃗v [Equation] ≤ Cα l .
Revised: + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ Cα l .

Original: Proof.
Revised: Proof.

Original: This proof has the same strategy to that of Lemma 3.
Revised: This proof has the same strategy to that of Lemma 3.

Original: The first step is to understand the
Revised: The ﬁrst step is to understand the

Original: Equation (9) in the large-depth limit.
Revised: Equation (9) in the large-depth limit.

Original: [Equation]
Revised: [Equation]

Original: According to the result of Lemma 3, we have already known,
Revised: According to the result of Lemma 3, we have already known,

Original: [Equation]
Revised: [Equation]

Original: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u ′ ) in the large depth.
Revised: To proceed the proof, we need to work out the behavior of ˙Σ ( l ) ( r ) ( u, u (cid:48) ) in the large depth.

Original: ReLU activation, ϕ ( x ) = max { 0 , x } .
Revised: ReLU activation, φ ( x ) = max { 0 , x } .

Original: Recall that we define C r +

Revised: Recall that we deﬁne C r +

Original: = f
Revised: = f

Original: ( C r ) , and we have,
Revised: ( C r ) , and we have,

Original: [Equation]
Revised: [Equation]

Original: Then, at the edge of chaos,
Revised: Then, at the edge of chaos,

Original: [Equation]
Revised: [Equation]

Revised: arcsin

Revised: [Equation]

Revised: +

Revised: [Equation]

Original: Tanh activation, ϕ ( x ) = tanh( x ) .
Revised: Tanh activation, φ ( x ) = tanh( x ) .

Original: We have
Revised: We have

Original: [Equation]
Revised: [Equation]

Original: At the edge of chaos,
Revised: At the edge of chaos,

Original: [Equation]
Revised: [Equation]

Original: Now we prove the result by induction method.
Revised: Now we prove the result by induction method.

Original: For l = 0 , we directly have,
Revised: For l = 0 , we directly have,

Original: [Equation]
Revised: [Equation]

Original: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization,
Revised: Thus there is a constant C , depending on G ( V, E ) , and the number of MLP operation R , over feature initialization,

Original: [Equation]
Revised: [Equation]

Original: Assume the result is valid for Θ ( l ) ( r ) ( u, u ′ ) , then we have,
Revised: Assume the result is valid for Θ ( l ) ( r ) ( u, u (cid:48) ) , then we have,

Original: [Equation]
Revised: [Equation]

Original: Now we consider the distance between Θ ( l +1)
Revised: Now we consider the distance between Θ ( l +1)

Original: ( r ) ( u, u ′ ) and ⃗π ( G ) T ( R ( l +1) ⃗v + ⃗v ′ ) .
Revised: ( r ) ( u, u (cid:48) ) and (cid:126)π ( G ) T ( R ( l +1) (cid:126)v + (cid:126)v (cid:48) ) .

Original: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:
Revised: To prove this, we need to divide the propagation from l layer to l + 1 layer into three steps:

Original: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.
Revised: Θ ( l ) ( r ) → Θ ( l ) ( r +1) → · · · → Θ ( l ) ( R ) 2.

Original: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.
Revised: Θ ( l ) ( R ) → Θ ( l +1) (0) 3.

Original: Θ ( l +1) (0) → Θ ( l +1) (1) → · · · → Θ ( l +1) ( r ) .
Revised: Θ ( l +1) (0) → [Equation]

Original: For step 1, we have,
Revised: For step 1, we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the process, we have a relation for Θ ( l ) ( R ) ( u, u ′ ) at the last transformation in step 1,
Revised: Repeat the process, we have a relation for Θ ( l ) ( R ) ( u, u (cid:48) ) at the last transformation in step 1,

Original: [Equation]
Revised: [Equation]

Original: Secondly
Revised: Secondly

Original: [Equation]
Revised: [Equation]

Original: we go through an aggregation operation.
Revised: we go through an aggregation operation.

Original: Because it is a Markov chain step
Revised: Because it is a Markov chain step

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Repeat the result in step 1, we finally have,
Revised: Repeat the result in step 1, we ﬁnally have,

Original: [Equation]
Revised: [Equation]

Revised: C T RAINABILITY OF U LTRA -W IDE GCN S

Original: Corollary 1 (Trainability of Ultra-Wide GCNs) .
Revised: Corollary 1 (Trainability of Ultra-Wide GCNs) .

Original: Consider a GCN of the form (6) and (7), with depth L , non-linear transformations number R , an MSE loss [Equation] and a Lipchitz activation ϕ ( x ) , trained with gradient descent on a node classification task.
Revised: Consider a GCN of the form (6) and (7), with depth l , non-linear transformations number r , an MSE loss, and a Lipchitz activation, trained with gradient descent on a node classiﬁcation task.

Original: Then the output function follows,
Revised: Then the output function follows,

Original: [Equation]
Revised: [Equation]

Original: where Y is the label with respect to node V .
Revised: where Y is the label with respect to node V .

Original: Then,when L → ∞ , Θ ( L ) ( R ) ( V , V ) is singular and there exist a constant C > 0 such that for all t > 0 , ∥ f t ( V ) − Y∥ > C .
Revised: Then, Θ ( l ) ( r ) ( G ) is singular when l → ∞ . Moreover, there exist a constant C > 0 such that for all t > 0 , (cid:107) f t ( V ) − Y(cid:107) > C .

Original: Proof.
Revised: Proof.

Original: According to the result from Jacot et al.
Revised: According to the result from Jacot et al.

Original: (2018), GNTK Θ ( l ) ( r ) ( V , V ) converges to a deterministic kernel and remains constant during gradient descent in the infinite width limit.
Revised: (2018), GNTK Θ ( l ) ( r ) ( G ) converges to a deterministic kernel and remains constant during gradient descent in the inﬁnite width limit.

Original: We omit proof procedure for this result, since it is now a standard conclusion in the NTK study.
Revised: We omit proof procedure for this result, since it is now a standard conclusion in the NTK study.

Original: Based on the conclusion above, Lee et al. (2019a) proved that the infinitely-wide neural network is equivalent to its linearized mode,
Revised: Based on the conclusion above, Lee et al. (2019a) proved that the inﬁnitely-wide neural network is equivalent to its linearized mode,

Original: [Equation]
Revised: [Equation]

Original: where ω t = θ t − θ 0 .
Revised: where ω t = θ t − θ 0 .

Original: We call it linearized model because only zero and first order term of Taylor expansion are kept.
Revised: We call it linearized model because only zero and ﬁrst order term of Taylor expansion are kept.

Original: Since we know dynamics of gradient flow using this linearized function are governed by,
Revised: Since we know dynamics of gradient ﬂow using this linearized function are governed by,

Original: [Equation]
Revised: [Equation]

Original: where L is an MSE loss, then the above equations have closed form solutions
Revised: where L is an MSE loss, then the above equations have closed form solutions

Original: [Equation]
Revised: [Equation]

Original: Since Lee et al.
Revised: Since Lee et al.

Original: Lee et al. (2019a) showed that f lin t ( V ) =
Revised: Lee et al. (2019a) showed that f lin t ( V ) =

Original: f t in the infinite width limit, thus we have,
Revised: f t in the inﬁnite width limit, thus we have,

Original: [Equation]
Revised: [Equation]

Original: In theorem 1, we have shown Θ ( l ) ( r ) ( V , V ) converges a constant matrix at an exponential rate, thus being singular in the large depth limit.
Revised: In theorem 1, we have shown Θ ( l ) ( r ) ( G ) converges a constant matrix at an exponential rate, thus being singular in the large depth limit.

Original: According to Equation (21), we know that,
Revised: According to Equation (21), we know that,

Original: [Equation]
Revised: [Equation]

Original: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( V , V ) 
Revised: Let Θ ( ∞ ) ( r ) be the GNTK in the large depth limit, Θ ( ∞ ) ( r ) ( G ) =

Original: =

Original: Q T DQ be the decomposition of the GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.
Revised: Q T DQ be the decomposition of the GNTK, where Q is an orthogonal matrix and D is a diagonal matrix.

Original: Because Θ ( ∞ ) ( r ) ( V , V ) is singular, D has at least one zero value d j = 0 , then
Revised: Because Θ ( ∞ ) ( r ) ( G ) is singular, D has at least one zero value d j = 0 , then

Original: [Equation]
Revised: [Equation]

Original: Remark 3.
Revised: Remark 3.

Original: In the proof we assume the loss is MSE.
Revised: In the proof we assume the loss is MSE.

Original: Nevertheless, the conclude regarding trainability can be applied to other common loss such as cross-entropy.
Revised: Nevertheless, the conclude regarding trainability can be applied to other common loss such as cross-entropy.

Original: For cross-entropy loss, even though we cannot write down an analytical expression for the solution, we prove the GNTK still governs the trainability and the law of GNTK is not affected by the loss.
Revised: For cross-entropy loss, even though we cannot write down an analytical expression for the solution, we prove the GNTK still governs the trainability and the law of GNTK is not affected by the loss.

Original: Thus the results for trainability hold on the cross-entropy case.
Revised: Thus the results for trainability hold on the cross-entropy case.

Original: C C ONVERGENCE OF R ESIDUAL GNTK C. R ESIDUAL CONNECTION IN AGGREGATION
Revised: D C ONVERGENCE OF R ESIDUAL GNTK D. R ESIDUAL CONNECTION IN AGGREGATION

Original: Theorem 2 (Convergence rate for residual connection in aggregation) .
Revised: Theorem 2 (Convergence rate for residual connection in aggregation) .

Original: Consider a GNTK of the form (12) with non-linear transformation of the form (7).
Revised: Consider a GNTK of nonlinear transformation (9) and residual connection (12).

Original: Then with a stationary vector ˜ ⃗π ( G ) for ˜
Revised: Then with a stationary vector ˜ (cid:126)π ( G ) for ˜

Original: A ( G ) , there exist constants 0 <
Revised: A ( G ) , there exist constants 0 <

Original: ˜ α < 1 and C > 0 , and constant vectors ⃗v and ⃗v ′ depending on
Revised: ˜ α < 1 and C > 0 , and constant vectors (cid:126)v and (cid:126)v (cid:48) depending on

Original: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u ′ ) − ˜ ⃗π ( G ) T (cid:0) Rl⃗v + ⃗v ′ (cid:1)(cid:12)(cid:12) ≤ C ˜
Revised: R , such that (cid:12)(cid:12) Θ ( l ) ( r ) ( u, u (cid:48) ) − ˜ (cid:126)π ( G ) T (cid:0) Rl(cid:126)v + (cid:126)v (cid:48) (cid:1)(cid:12)(cid:12) ≤ C ˜

Original: α l .
Revised: α l .

Original: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.
Revised: Furthermore, we denote the second largest eigenvalue of A ( G ) and ˜ A ( G ) as λ 2 and ˜ λ 2 , respectively.

Original: Then, ˜ λ 2 > λ 2 .
Revised: Then, ˜ λ 2 > λ 2 .

Original: Proof.
Revised: Proof.

Original: According to the aggregation function for covariance matrix, we have
Revised: According to the aggregation function for covariance matrix, we have

Original: [Equation]
Revised: [Equation]

Original: The new aggregation matrix ˜
Revised: The new aggregation matrix ˜

Original: A ( G ) is a stochastic transition matrix as well, which can be seen from,
Revised: A ( G ) is a stochastic transition matrix as well, which can be seen from,

Original: [Equation]
Revised: [Equation]

Original: The new aggregation can slow down the convergence rate can seen from considering their eigenvalues.
Revised: Then we compare the second largest eigenvalue between two transition matrix.

Original: Suppose the eigenvalues of original matrix
Revised: Suppose the eigenvalues of original matrix

Original: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .
Revised: A ( G ) are { λ 1 > λ 2 > · · · > λ n 2 } .

Original: We already know that the maximum eigenvalue is λ 

Revised: We already know that the maximum eigenvalue is λ

Original: = 1 ,
Revised: = 1 ,

Original: and the converge speed is governed by the second largest eigenvalue λ 2 .
Revised: and the converge speed is governed by the second largest eigenvalue λ 2 .

Original: Now we consider the second largest eigenvalue ˜ λ
Revised: Now we consider the second largest eigenvalue ˜ λ

Original: 2 of matrix ˜
Revised: 2 of matrix ˜

Original: A :
Revised: A :

Original: [Equation]
Revised: [Equation]

Original: Since λ 2 < ˜ λ 2 < 1 . Thus the convergence speed of residual GNTK will be slow down, through it is still exponential convergent. For the limit behavior of Θ ( l ) ( r ) ( u, u ′ ) as l tends to infinity, we can directly use the proof strategy from Theorem 1.
Revised: Remark 4.

Original: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 .
Revised: The theorem above gives us a certain conclusion about the relationship between λ 2 and ˜ λ 2 .

Original: Here we discuss more about the relationship between α and ˜ α .
Revised: Here we discuss more about the relationship between α and ˜ α .

Original: According to (Rosenthal, 1995), the relationship between convergence rate α and second largest eigenvalue λ 2 of transition matrix
Revised: According to (Rosenthal, 1995), the relationship between convergence rate α and second largest eigenvalue λ 2 of transition matrix

Original: A ( G ) can be expressed as,
Revised: A ( G ) can be expressed as,

Original: [Equation]
Revised: [Equation]

Original: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .
Revised: where α > λ 2 , and J > 1 is the size of the largest Jordan block of A ( G ) .

Original: From the above inequality, we can conclude that the second largest eigenvalue almost govern the convergence rate.
Revised: From the above inequality, we can conclude that the second largest eigenvalue almost govern the convergence rate.

Original: In a summary, we have α ∈ [ λ 2 , 1) and ˜ α∈ [˜ λ 2 , 1) .The result of ˜ λ
Revised: However, to maintain rigor, we cannot directly obtain α < ˜ α .

Original:  2 > λ 2 in Theorem 2 causes ˜ α has a smaller value range, thus ˜ α tends to be greater than α .
Revised: Instead, from the result that ˜ λ 2 > λ 2 , we say that ˜ α is roughly larger than α .

Original: C. R ESIDUAL CONNECTION IN MLP
Revised: D. R ESIDUAL CONNECTION IN MLP

Original: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .
Revised: Theorem 3 (Convergence rate for GNTK with residual connection between transformations) .

Original: Consider a GNTK the form (8) and (12).
Revised: Consider a GNTK the form (8) and (12).

Original: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G, G ′ ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v ′ depending on R , such that, | Θ ( l ) ( r )( u, u ′ )
Revised: If A ( G ) is irreducible and aperiodic, with a stationary distribution π ( G, G (cid:48) ) , then there exist constants 0 < α < 1 and C > 0 , and constant vectors v and v (cid:48) depending on R , such that, | Θ ( l ) ( r )

Original: − ⃗π ( G ) T · (cid:0) Rl (1 + σ 2 w 2 )
Revised: [Equation]

Original: Rl ⃗v + ⃗v ′ (cid:1) | ≤ Cα l .
Revised: u, u [Equation]

Original: Proof.
Revised: Proof.

Original: For residual connection in MLP, we have,
Revised: For residual connection in MLP, we have,

Original: [Equation]
Revised: [Equation]

Original: Since σ 2 w > 0 , q r ( G ) grows at an exponential rate.
Revised: Since σ 2 w > 0 , q r ( G ) grows at an exponential rate.

Original: Now, we turn to compute the correlation term C r ( u, u ′ ) .
Revised: Now, we turn to compute the correlation term C r ( u, u (cid:48) ) .

Original: For convenience, we suppose q r ( u ) =
Revised: For convenience, we suppose q r ( u ) =

Original: q r ( u ′ ) .
Revised: q r ( u (cid:48) ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Using Taylor expansion of f near 1, as have been done in the proof of Lemma 2,
Revised: Using Taylor expansion of f near 1, as have been done in the proof of Lemma 2,

Original: [Equation]
Revised: [Equation]

Original: we have,
Revised: we have,

Original: [Equation]
Revised: [Equation]

Original: Note that it is similar to the case of MLP without residual connection:
Revised: Note that it is similar to the case of MLP without residual connection:

Original: [Equation]
Revised: [Equation]

Original: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u ′ ) and Θ ( l ) ( r ) ( u, u ′ ) in the large depth limit,
Revised: Following the proof diagram in Lemma 3 and Theorem 1, we can obtain the behavior of Σ ( l ) ( r ) ( u, u (cid:48) ) and Θ ( l ) ( r ) ( u, u (cid:48) ) in the large depth limit,

Original: [Equation]
Revised: [Equation]

Original: D APPENDIX: D ISCUSSION ON THE TRAINABILITY OF GNTK WITH C RTICAL D ROP E DGE
Revised: E APPENDIX: D ISCUSSION ON THE TRAINABILITY OF GNTK WITH C RTICAL D ROP E DGE

Original: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd˝os & R´enyi, 1961; Erd¨os & R´enyi, 2011), here n is the number of nodes in a graph, when we use the critical drop rate.
Revised: According to the critical percolation theory, there exists a large connected component of order O ( n ) (Erd ˝ os & R ´ enyi, 1961; Erd ¨ os & R ´ enyi, 2011), here n is the number of nodes in a graph. This implies that there exist a (connected) graph that of size O ( n ) when we use the critical dropping rate.

Original: To preserve information, edges will be resampled at every iteration.
Revised: To preserve information, edges will be resampled at every iteration.

Original: This implies that at each iteration during training, the GNN takes a random and large graph.
Revised: This implies that at each iteration during training, the GNN takes a random and large graph.

Original: For the whole training process, we can think that we have used all the information on the graph.
Revised: For the whole training process, we can think that we have used all the information on the graph.

Original: It would be interesting to consider achieving some theoretical results for the limiting behavior of the
Revised: It would be interesting to consider achieving some theoretical results for the limiting behavior of the

Original: GNTK in the large depth with Critical DropEdge.
Revised: GNTK in the large depth with Critical DropEdge.

Original: Since the Critical DropEdge is originated from random graph, a promising way is to analyze the spectral distributions of adjacency and Laplacian matrices with random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).
Revised: Since the Critical DropEdge is originated from random graph, a promising way is to analyze the spectral distributions of adjacency and Laplacian matrices with random graph model, like existing studies (Ding & Jiang, 2010; Chung & Radcliffe, 2011).

Original: E APPENDIX: D ATASET AND E XPERIMENTAL I MPLEMENTATION E. D ATASET
Revised: F APPENDIX: D ATASET AND E XPERIMENTAL I MPLEMENTATION F. D ATASET

Original: MUTAG (Rossi & Ahmed, 2015) is a bioinformatics dataset; Cora, Citeseer, Pubmed (Kipf &
Revised: MUTAG (Rossi & Ahmed, 2015) is a bioinformatics dataset; Cora, Citeseer, Pubmed (Kipf &

Original: Welling, 2016a) are commonly used citation network data for the node classification.
Revised: Welling, 2016a) are commonly used citation network data for the node classiﬁcation.

Original: Besides,
Revised: Besides,

Original: Physics (Shchur et al., 2018) is a co-author network dataset.
Revised: Physics (Shchur et al., 2018) is a co-author network dataset.

Original: We illustrate the detailed information for the four node classification datasets as follows and summary can be found in Table 
Revised: We illustrate the detailed information for the four node classiﬁcation datasets as follows and summary can be found in Table 3.

Original: • The Cora dataset consists of 2708 scientific publications classified into one of seven classes.
Revised: • The Cora dataset consists of 2708 scientiﬁc publications classiﬁed into one of seven classes.

Original: The citation network consists of 5429 links.
Revised: The citation network consists of 5429 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 1433 unique words.
Revised: The dictionary consists of 1433 unique words.

Original: • The Citeseer dataset consists of 3312 scientific publications classified into one of six classes.
Revised: • The Citeseer dataset consists of 3312 scientiﬁc publications classiﬁed into one of six classes.

Original: The citation network consists of 4732 links.
Revised: The citation network consists of 4732 links.

Original: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.
Revised: Each publication in the dataset is described by a 0/1-valued word vector indicating the absence/presence of the corresponding word from the dictionary.

Original: The dictionary consists of 3703 unique words.
Revised: The dictionary consists of 3703 unique words.

Original: • The Pubmed Diabetes dataset consists of 19717 scientific publications from PubMed database pertaining to diabetes classified into one of three classes.
Revised: The Pubmed Diabetes dataset consists of 19717 scientiﬁc publications from PubMed database pertaining to diabetes classiﬁed into one of three classes.

Original: The citation network consists of 44338 links.
Revised: The citation network consists of 44338 links.

Original: Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words.
Revised: Each publication in the dataset is described by a TF/IDF weighted word vector from a dictionary which consists of 500 unique words.

Original: • The Physics dataset consists of 34493 authors, and edges mean whether two authors coauthored a paper.
Revised: • The Physics dataset consists of 34493 authors, and edges mean whether two authors coauthored a paper.

Original: Node features are paper keywords from the author’s papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA following Kim & Oh (2020).
Revised: Node features are paper keywords from the author’s papers, and we reduce the original dimension (6805 and 8415) to 500 using PCA following Kim & Oh (2020).

Original: The split is the 20-per-class/30-per-class/rest.
Revised: The split is the 20-per-class/30-per-class/rest.

Original: The goal of this task is to classify each author’s respective field of study.
Revised: The goal of this task is to classify each author’s respective ﬁeld of study.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: E. E XPERIMENTAL I MPLEMENTATION
Revised: F. E XPERIMENTAL I MPLEMENTATION

Original: We use the PyTorch implementation to simulate both infinite-width and finite-width GCNs:
Revised: We use the PyTorch implementation to simulate both inﬁnite-width and ﬁnite-width GCNs:

Original: • The infinite-width GCNs use part of code from Du et al.
Revised: • The inﬁnite-width GCNs use part of code from Du et al.

Original: (2019) which is originally adopted for graph classification.
Revised: (2019) which is originally adopted for graph classiﬁcation.

Original: We redesigned the calculation method of GNTK 2 according to the formula in section 3.1 and used it to process node classification tasks.
Revised: We redesigned the calculation method of GNTK 2 according to the formula in section 3.1 and used it to process node classiﬁcation tasks.

Original: • For finite-width GNNs with DropEdge 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.
Revised: • For ﬁnite-width GNNs with DropEdge 3 , we use the code from (Rong et al., 2019), we perform random hyper-parameter search for each model, and report the case giving the best accuracy on validation set of each benchmark, following the same strategy as Rong et al.

Original: The difference is that, in each experiment, we fix the edge sampling percent as ρ = | V | 2 | E | , which listed in the last column of Table 3.
Revised: The difference is that, in each experiment, we ﬁx the edge sampling percent as ρ = | V | 2 | E | , which listed in the last column of Table 3.

Original: All the codes mentioned above use the MIT license.
Revised: All the codes mentioned above use the MIT license.

Original: All experiments are conducted on with two
Revised: All experiments are conducted on with two

Original: Nvidia Quadro RTX 6000 GPUs.
Revised: Nvidia Quadro RTX 6000 GPUs.

Original: F APPENDIX: A DDITIONAL E XPERIMENT AND D ISCUSSION F. I NFINITE -W IDTH B ACKBONE
Revised: G APPENDIX: A DDITIONAL E XPERIMENT AND D ISCUSSION G. I NFINITELY -W IDE B ACKBONES

Original: In the main text, we reported the performance of GCNs with finite-width, here we display more results with respect to infinite-width GCNs, as shown in Table 4.
Revised: In the main paper, we reported the performance of GCNs with ﬁnite-width, here we display more results with respect to inﬁnitely-wide GCNs, as shown in Table 4.

Original: We apply Critical DropEdge to infinite-width backbone onnode classification with a semi-supervised setting.
Revised: We apply Critical DropEdge to inﬁnitely-wide backbones on semi-supervised node classiﬁcation.

Original: We consider two backbones: GCN (Kipf & Welling, 2016a) and JKNet (Xu et al., 2018b).
Revised: We consider two backbones: GCN (Kipf & Welling, 2016a) and JKNet (Xu et al., 2018b).

Original: For the edge rate, we use the edge preserving percent as ρ ( G ) which shown in Table 3.
Revised: For the edge rate, we use the edge preserving percent as ρ ( G ) which shown in Table 3.

Original: Table 4 summaries the performance on three citation network datasets.
Revised: Table 4 summaries the performance on three citation networks.

Original: In this table, we report the performance of GCNs with 4/8/32/ layers.
Revised: In this table, we report the performance of GCNs with 4/8/32/64 layers.

Original: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original deep GNNs in the infinite-width case, especially when the model is deep
Revised: It is shown that, in most cases, using Critical DropEdge (C-DropEdge) can achieve better results than original deep GNNs in the inﬁnitely-wide case, especially when the model is deep.

Original: . F. P ERFORMANCE OF S EPARATE BACKBONES

Original: [Table]
Revised: [Table]

Revised: G. P ERFORMANCE OF S EPARATE B ACKBONES

Original: The performance results presented in Table 1 is the best performance across different backbones.
Revised: The performance results presented in Table 1 is the best performance across different backbones.

Original: In
Revised: In

Original: Table 5, we show the results of vanilla GNNs, DropEdge and C-DropEdge with different backbones.
Revised: Table 5, we show the results of vanilla GNNs, DropEdge and C-DropEdge with different backbones.

Original: From the table we can conclude the Critical DropEdge can consistently outperform DropEdge and vanilla GNNs.
Revised: From the table we can conclude the Critical DropEdge can consistently outperform DropEdge and vanilla GNNs.

Original: F. C OMPARISON TO D ROP E DGE
Revised: G. C OMPARISON TO D ROP E DGE

Original: While our proposed Critical DropEdge shares the same program with the DropEdge method (Rong et al., 2019), we point out the main difference which can result in a corresponding advantage for our approach concerning the hyper-parameter search space.
Revised: While our proposed Critical DropEdge shares the same program with the DropEdge method (Rong et al., 2019) [Table] we point out the main difference which can result in a corresponding advantage for our approach concerning the hyper-parameter search space.

Original: C-DropEdge has a fixed and graphdependent edge preserving percent, which implies our hyper-parameter space is smaller than that of
Revised: C-DropEdge has a ﬁxed and graphdependent edge preserving percent, which implies our hyper-parameter space is smaller than that of

Original: DropEdge.
Revised: DropEdge.

Original: The edge preserving percent is one of the most significant hyper-parameter, which has a crucial influence on the final performance of the algorithm.
Revised: The edge preserving percent is one of the most signiﬁcant hyper-parameter, which has a crucial inﬂuence on the ﬁnal performance of the algorithm.

Original: In order to obtain good performance,
Revised: In order to obtain good performance,

Original: DropEdge needs to spend a lot of effort to search on the space in terms of this hyper-parameter.
Revised: DropEdge needs to spend a lot of effort to search on the space in terms of this hyper-parameter.

Original: In contrast, our method naturally determines the parameters, thus saving plenty of time.
Revised: In contrast, our method naturally determines the parameters, thus saving plenty of time.

Original: Whether from a theoretical perspective or from the final performance, the edge preserving percent we set is reasonable and effective.
Revised: Whether from a theoretical perspective or from the ﬁnal performance, the edge preserving percent we set is reasonable and effective.

Original: F. T RAINABILITY OF GNTK WITH C RITICAL D ROP E DGE
Revised: G. T RAINABILITY OF GNTK WITH C RITICAL D ROP E DGE

Original: We implement deep GCN without/without DropEdge and C-DropEdge to compare the training performance.
Revised: We implement deep GCN without/without DropEdge and C-DropEdge to compare the training performance.

Original: The result is shown in Figure 4.
Revised: The result is shown in Figure 4.

Original: The training of GCN and DropEdge with GCN converge slower and converge to a higher training error compared to C-DropEdge with GCN, indicating that

Original: C-DropEdge can achive better trainablity compared to GCN and DropEdge with GCN. Besides, the
Revised: The training of GCN and DropEdge with GCN converge

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Revised: slower and converge to a higher training error compared to C-DropEdge with GCN, indicating that

Revised: C-DropEdge can achive better trainablity compared to GCN and DropEdge with GCN.

Original:  convergence results of GNTKs with C-DropEdge is shown in Figure 5.
Revised: Besides, the convergence results of GNTKs with C-DropEdge is shown in Figure 5.

Original: Compared to residual connection, we find that the normalized GNTK elements will not converge to a single value, and the convergence rate curve does not depend on the depth.
Revised: Compared to residual connection, we ﬁnd that the normalized GNTK elements will not converge to a single value, and the convergence rate curve does not depend on the depth.

Original: This implies that residual connection mitigates the trainability loss by slow down exponential convergence rate, while C-DropEdge directly change the convergence results.
Revised: This implies that residual connection mitigates the trainability loss by slow down exponential convergence rate, while C-DropEdge directly change the convergence results.