Original: G RAPH N EURAL N ETWORK -I NSPIRED K ERNELS FOR G AUSSIAN P ROCESSES IN S EMI -S UPERVISED L EARNING
Revised: G RAPH N EURAL N ETWORK -I NSPIRED K ERNELS FOR G AUSSIAN P ROCESSES IN S EMI -S UPERVISED L EARNING

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Gaussian processes (GPs) are an attractive class of machine learning models because of their simplicity and ﬂexibility as building blocks of more complex
Revised: Gaussian processes (GPs) are an attractive class of machine learning models because of their simplicity and ﬂexibility as building blocks of more complex

Original: Bayesian models.
Revised: Bayesian models.

Original: Meanwhile, graph neural networks (GNNs) emerged recently as a promising class of models for graph-structured data in semi-supervised learning and beyond.
Revised: Meanwhile, graph neural networks (GNNs) emerged recently as a promising class of models for graph-structured data in semi-supervised learning and beyond.

Original: Their competitive performance is often attributed to a proper capturing of the graph inductive bias.
Revised: Their competitive performance is often attributed to a proper capturing of the graph inductive bias.

Original: In this work, we introduce this inductive bias into GPs to improve their predictive performance for graph-structured data.
Revised: In this work, we introduce this inductive bias into GPs to improve their predictive performance for graph-structured data.

Original: We show that a prominent example of GNNs, the graph convolutional network, is equivalent to some GP when its layers are inﬁnitely wide; and we analyze the kernel universality and the limiting behavior in depth.
Revised: We show that a prominent example of GNNs, the graph convolutional network, is equivalent to some GP when its layers are inﬁnitely wide; and we analyze the kernel universality and the limiting behavior in depth.

Original: We further present a programmable procedure to compose covariance kernels inspired by this equivalence and derive example kernels corresponding to several interesting members of the GNN family.
Revised: We further present a programmable procedure to compose covariance kernels inspired by this equivalence and derive example kernels corresponding to several interesting members of the GNN family.

Original: We also propose a computationally efﬁcient approximation of the covariance matrix for scalable posterior inference with large-scale data.
Revised: We also propose a computationally efﬁcient approximation of the covariance matrix for scalable posterior inference with large-scale data.

Original: We demonstrate that these graph-based kernels lead to competitive classiﬁcation and regression performance, as well as advantages in computation time, compared with the respective GNNs.
Revised: We demonstrate that these graph-based kernels lead to competitive classiﬁcation and regression performance, as well as advantages in computation time, compared with the respective GNNs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gaussian processes (GPs) (Rasmussen & Williams, 2006) are widely used in machine learning, uncertainty quantiﬁcation, and global optimization.
Revised: Gaussian processes (GPs) (Rasmussen & Williams, 2006) are widely used in machine learning, uncertainty quantiﬁcation, and global optimization.

Original: In the Bayesian setting, a GP serves as a prior probability distribution over functions, characterized by a mean (often treated as zero for simplicity) and a covariance.
Revised: In the Bayesian setting, a GP serves as a prior probability distribution over functions, characterized by a mean (often treated as zero for simplicity) and a covariance.

Original: Conditioned on observed data with a Gaussian likelihood, the random function admits a posterior distribution that is also Gaussian, whose mean is used for prediction and the variance serves as an uncertainty measure.
Revised: Conditioned on observed data with a Gaussian likelihood, the random function admits a posterior distribution that is also Gaussian, whose mean is used for prediction and the variance serves as an uncertainty measure.

Original: The closed-form posterior allows for exact Bayesian inference, resulting in great attractiveness and wide usage of GPs.
Revised: The closed-form posterior allows for exact Bayesian inference, resulting in great attractiveness and wide usage of GPs.

Original: The success of GPs in practice depends on two factors: the observations (training data) and the covariance kernel.
Revised: The success of GPs in practice depends on two factors: the observations (training data) and the covariance kernel.

Original: We are interested in semi-supervised learning, where only a small amount of data is labeled while a large amount of unlabeled data can be used together for training (Zhu, 2008).
Revised: We are interested in semi-supervised learning, where only a small amount of data is labeled while a large amount of unlabeled data can be used together for training (Zhu, 2008).

Original: In recent years, graph neural networks (GNNs)
Revised: In recent years, graph neural networks (GNNs)

Original: (Zhou et al., 2020; Wu et al., 2021) emerged as a promising class of models for this problem, when the labeled and unlabeled data are connected by a graph.
Revised: (Zhou et al., 2020; Wu et al., 2021) emerged as a promising class of models for this problem, when the labeled and unlabeled data are connected by a graph.

Original: The graph structure becomes an important inductive bias that leads to the success of GNNs.
Revised: The graph structure becomes an important inductive bias that leads to the success of GNNs.

Original: This inductive bias inspires us to design a GP model under limited observations, by building the graph structure into the covariance kernel.
Revised: This inductive bias inspires us to design a GP model under limited observations, by building the graph structure into the covariance kernel.

Original: An intimate relationship between neural networks and GPs is known: a neural network with fully connected layers, equipped with a prior probability distribution on the weights and biases, converges to a GP when each of its layers is inﬁnitely wide (Lee et al., 2018; de G. Matthews et al., 2018).
Revised: An intimate relationship between neural networks and GPs is known: a neural network with fully connected layers, equipped with a prior probability distribution on the weights and biases, converges to a GP when each of its layers is inﬁnitely wide (Lee et al., 2018; de G. Matthews et al., 2018).

Original: Such a result is owing to the central limit theorem (Neal, 1994; Williams, 1996) and the GP covariance can be recursively computed if the weights (and biases) in each layer are iid Gaussian.
Revised: Such a result is owing to the central limit theorem (Neal, 1994; Williams, 1996) and the GP covariance can be recursively computed if the weights (and biases) in each layer are iid Gaussian.

Original: Similar results for other architectures, such as convolution layers and residual connections, were subsequently established in the literature (Novak et al., 2019; Garriga-Alonso et al., 2019).
Revised: Similar results for other architectures, such as convolution layers and residual connections, were subsequently established in the literature (Novak et al., 2019; Garriga-Alonso et al., 2019).

Original: One focus of this work is to establish a similar relationship between GNNs and the limiting GPs.
Revised: One focus of this work is to establish a similar relationship between GNNs and the limiting GPs.

Original: We will derive the covariance kernel that incorporates the graph inductive bias as GNNs do.
Revised: We will derive the covariance kernel that incorporates the graph inductive bias as GNNs do.

Original: We start with one of the most widely studied GNNs, the graph convolutional network (GCN) (Kipf &
Revised: We start with one of the most widely studied GNNs, the graph convolutional network (GCN) (Kipf &

Original: Welling, 2017), and analyze the kernel universality as well as the limiting behavior when the depth also tends to inﬁnity.
Revised: Welling, 2017), and analyze the kernel universality as well as the limiting behavior when the depth also tends to inﬁnity.

Original: We then derive covariance kernels from other GNNs by using a programmable procedure that corresponds every building block of a neural network to a kernel operation.
Revised: We then derive covariance kernels from other GNNs by using a programmable procedure that corresponds every building block of a neural network to a kernel operation.

Original: Meanwhile, we design efﬁcient computational procedures for posterior inference (i.e., regression and classiﬁcation).
Revised: Meanwhile, we design efﬁcient computational procedures for posterior inference (i.e., regression and classiﬁcation).

Original: GPs are notoriously difﬁcult to scale because of the cubic complexity with respect to the number of training data.
Revised: GPs are notoriously difﬁcult to scale because of the cubic complexity with respect to the number of training data.

Original: Benchmark graph datasets used by the GNN literature may contain thousands or even millions of labeled nodes (Hu et al., 2020b).
Revised: Benchmark graph datasets used by the GNN literature may contain thousands or even millions of labeled nodes (Hu et al., 2020b).

Original: The semi-supervised setting worsens the scenario, as the covariance matrix needs to be (recursively) evaluated in full because of the graph convolution operation.
Revised: The semi-supervised setting worsens the scenario, as the covariance matrix needs to be (recursively) evaluated in full because of the graph convolution operation.

Original: We propose a Nystr¨om-like scheme to perform low-rank approximations and apply the approximation recursively on each layer, to yield a low-rank kernel matrix.
Revised: We propose a Nystr¨om-like scheme to perform low-rank approximations and apply the approximation recursively on each layer, to yield a low-rank kernel matrix.

Original: Such a matrix can be computed scalably.
Revised: Such a matrix can be computed scalably.

Original: We demonstrate through numerical experiments that
Revised: We demonstrate through numerical experiments that

Original: the
Revised: the

Original: GP posterior inference is much faster than training a GNN and subsequently performing predictions on the test set.
Revised: GP posterior inference is much faster than training a GNN and subsequently performing predictions on the test set.

Original: We summarize the contributions of this work as follows:
Revised: We summarize the contributions of this work as follows:

Original: We derive the GP as a limit of the GCN when the layer widths tend to inﬁnity and study the kernel universality and the limiting behavior in depth.
Revised: We derive the GP as a limit of the GCN when the layer widths tend to inﬁnity and study the kernel universality and the limiting behavior in depth.

Original: We propose a computational procedure to compute a low-rank approximation of the covariance matrix for practical and scalable posterior inference.
Revised: We propose a computational procedure to compute a low-rank approximation of the covariance matrix for practical and scalable posterior inference.

Original: We present a programmable procedure to compose covariance kernels and their approximations and show examples corresponding to several interesting members of the GNN family.
Revised: We present a programmable procedure to compose covariance kernels and their approximations and show examples corresponding to several interesting members of the GNN family.

Original: We conduct comprehensive experiments to demonstrate that the GP model performs favorably compared to GNNs in prediction accuracy while being signiﬁcantly faster in computation.
Revised: We conduct comprehensive experiments to demonstrate that the GP model performs favorably compared to GNNs in prediction accuracy while being signiﬁcantly faster in computation.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: It has long been observed that GPs are limits of standard neural networks with one hidden layer when the layer width tends to inﬁnity (Neal, 1994; Williams, 1996).
Revised: It has long been observed that GPs are limits of standard neural networks with one hidden layer when the layer width tends to inﬁnity (Neal, 1994; Williams, 1996).

Original: Recently, renewed interests in the equivalence between GPs and neural networks were extended to deep neural networks (Lee et al., 2018; de G. Matthews et al., 2018) as well as modern neural network architectures, such as convolution layers (Novak et al., 2019), recurrent networks (Yang, 2019), and residual connections (Garriga-Alonso et al., 2019).
Revised: Recently, renewed interests in the equivalence between GPs and neural networks were extended to deep neural networks (Lee et al., 2018; de G. Matthews et al., 2018) as well as modern neural network architectures, such as convolution layers (Novak et al., 2019), recurrent networks (Yang, 2019), and residual connections (Garriga-Alonso et al., 2019).

Original: The term NNGP (neural network Gaussian process) henceforth emerged under the context of Bayesian deep learning.
Revised: The term NNGP (neural network Gaussian process) henceforth emerged under the context of Bayesian deep learning.

Original: Besides the fact that an inﬁnite neural network deﬁnes a kernel, the training of a neural network by using gradient descent also deﬁnes a kernel—the neural tangent kernel (NTK)—that describes the evolution of the network (Jacot et al., 2018; Lee et al., 2019).
Revised: Besides the fact that an inﬁnite neural network deﬁnes a kernel, the training of a neural network by using gradient descent also deﬁnes a kernel—the neural tangent kernel (NTK)—that describes the evolution of the network (Jacot et al., 2018; Lee et al., 2019).

Original: Library supports in Python were developed to automatically construct the NNGP and
Revised: Library supports in Python were developed to automatically construct the NNGP and

Original: NTK kernels based on programming the corresponding neural networks (Novak et al., 2020).
Revised: NTK kernels based on programming the corresponding neural networks (Novak et al., 2020).

Original: GNNs are neural networks that handle graph-structured data (Zhou et al., 2020; Wu et al., 2021).
Revised: GNNs are neural networks that handle graph-structured data (Zhou et al., 2020; Wu et al., 2021).

Original: They are a promising class of models for semi-supervised learning.
Revised: They are a promising class of models for semi-supervised learning.

Original: Many GNNs use the messagepassing scheme (Gilmer et al., 2017), where neighborhood information is aggregated to update the representation of the center node.
Revised: Many GNNs use the messagepassing scheme (Gilmer et al., 2017), where neighborhood information is aggregated to update the representation of the center node.

Original: Representative examples include GCN (Kipf & Welling, 2017),
Revised: Representative examples include GCN (Kipf & Welling, 2017),

Original: GraphSAGE (Hamilton et al., 2017), GAT (Veliˇckovi´c et al., 2018), and GIN (Xu et al., 2019).
Revised: GraphSAGE (Hamilton et al., 2017), GAT (Veliˇckovi´c et al., 2018), and GIN (Xu et al., 2019).

Original: It is found that the performance of GNNs degrades as they become deep; one approach to mitigating the problem is to insert residual/skip connections, as done by JumpingKnowledge (Xu et al., 2018),
Revised: It is found that the performance of GNNs degrades as they become deep; one approach to mitigating the problem is to insert residual/skip connections, as done by JumpingKnowledge (Xu et al., 2018),

Original: APPNP (Gasteiger et al., 2019), and GCNII (Chen et al., 2020).
Revised: APPNP (Gasteiger et al., 2019), and GCNII (Chen et al., 2020).

Original: GP inference is too costly, because it requires the inverse of the N × N dense kernel matrix.
Revised: GP inference is too costly, because it requires the inverse of the N × N dense kernel matrix.

Original: Scalable approaches include low-rank methods, such as Nystr ¨ om approximation (Drineas & Mahoney, 2005), random features (Rahimi & Recht, 2007), and KISS-GP (Wilson & Nickisch, 2015); as well as multi-resolution (Katzfuss, 2017) and hierarchical methods (Chen et al., 2017; Chen & Stein, 2021).
Revised: Scalable approaches include low-rank methods, such as Nystr¨om approximation (Drineas & Mahoney, 2005), random features (Rahimi & Recht, 2007), and KISS-GP (Wilson & Nickisch, 2015); as well as multi-resolution (Katzfuss, 2017) and hierarchical methods (Chen et al., 2017; Chen & Stein, 2021).

Original: Prior efforts on integrating graphs into GPs exist.
Revised: Prior efforts on integrating graphs into GPs exist.

Original: Ng et al.
Revised: Ng et al.

Original: (2018) deﬁne a GP kernel by combing a base kernel with the adjacency matrix; it is related to a special case of our kernels where the network has only one layer and the output admits a robust-max likelihood for classiﬁcation.
Revised: (2018) deﬁne a GP kernel by combing a base kernel with the adjacency matrix; it is related to a special case of our kernels where the network has only one layer and the output admits a robust-max likelihood for classiﬁcation.

Original: Hu et al.
Revised: Hu et al.

Original: (2020a) explore a similar route to us, by taking the limit of a GCN, but its exploration is less comprehensive because it does not generalize to other GNNs and does not tackle the scalability challenge.
Revised: (2020a) explore a similar route to us, by taking the limit of a GCN, but its exploration is less comprehensive because it does not generalize to other GNNs and does not tackle the scalability challenge.

Original:  G RAPH C ONVOLUTIONAL N ETWORK AS A G AUSSIAN P ROCESS
Revised:  G RAPH C ONVOLUTIONAL N ETWORK AS A G AUSSIAN P ROCESS

Original: We start with a few notations used throughout this paper.
Revised: We start with a few notations used throughout this paper.

Original: Let an undirected graph be denoted by
Revised: Let an undirected graph be denoted by

Original: G =
Revised: G =

Original: ( V , E ) with N = |V| nodes and M = |E| edges.
Revised: ( V , E ) with N = |V| nodes and M = |E| edges.

Original: For notational simplicity, we use A ∈ R N × N to denote the original graph adjacency matrix or any modiﬁed/normalized version of it.
Revised: For notational simplicity, we use A ∈ R N × N to denote the original graph adjacency matrix or any modiﬁed/normalized version of it.

Original: Using d l to denote the width of the l -th layer, the layer architecture of GCN reads
Revised: Using d l to denote the width of the l -th layer, the layer architecture of GCN reads

Original: [Equation]
Revised: [Equation]

Original: where X ( l − 1) ∈ R N × d l − 1 and X ( l ) ∈
Revised: where X ( l − 1) ∈ R N × d l − 1 and X ( l ) ∈

Original: R N × d l are layer inputs and outputs, respectively; W ( l ) ∈
Revised: R N × d l are layer inputs and outputs, respectively; W ( l ) ∈

Original: R d l − 1 × d l and b ( l ) ∈ R 1 × d l are the weights and the biases, respectively; and φ is the ReLU activation function.
Revised: R d l − 1 × d l and b ( l ) ∈ R 1 × d l are the weights and the biases, respectively; and φ is the ReLU activation function.

Original: The graph convolutional operator A is a symmetric normalization of the graph adjacency matrix with self-loops added (Kipf & Welling, 2017).
Revised: The graph convolutional operator A is a symmetric normalization of the graph adjacency matrix with self-loops added (Kipf & Welling, 2017).

Original: For ease of exposition, it will be useful to rewrite the matrix notation (1) in element-sums and products.
Revised: For ease of exposition, it will be useful to rewrite the matrix notation (1) in element-sums and products.

Original: To this end, for a node x , let z ( l ) i ( x ) and x ( l ) i ( x ) denote the pre- and post-activation value at the i -th coordinate in the l -th layer, respectively.
Revised: To this end, for a node x , let z ( l ) i ( x ) and x ( l ) i ( x ) denote the pre- and post-activation value at the i -th coordinate in the l -th layer, respectively.

Original: Particularly, in an L -layer GCN, x (0) ( x ) is the input feature vector and z ( L ) ( x ) is the output vector.
Revised: Particularly, in an L -layer GCN, x (0) ( x ) is the input feature vector and z ( L ) ( x ) is the output vector.

Original: The layer architecture of GCN reads 
Revised: The layer architecture of GCN reads y

Original: [Equation]
Revised: [Equation]

Original: 3. L IMIT IN THE W IDTH
Revised: 3. L IMIT IN THE W IDTH

Original: The following theorem states that when the weights and biases in each layer are iid zero-mean
Revised: The following theorem states that when the weights and biases in each layer are iid zero-mean

Original: Gaussians, in the limit on the layer width, the GCN output z ( L ) ( x ) is a multi-output GP over the index x .
Revised: Gaussians, in the limit on the layer width, the GCN output z ( L ) ( x ) is a multi-output GP over the index x .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume d 1 , . . .
Revised: Assume d 1 , . . .

Original: , d L − 1 to be inﬁnite in succession and let the bias and weight terms be independent with distributions
Revised: , d L − 1 to be inﬁnite in succession and let the bias and weight terms be independent with distributions

Original: [Equation]
Revised: [Equation]

Original: Then, for each i , the collection { z ( l ) i ( x ) } over all graph nodes x follows the normal distribution
Revised: Then, for each i , the collection { z ( l ) i ( x ) } over all graph nodes x follows the normal distribution

Original: N (0 , K ( l ) ) , where the covariance matrix K ( l ) can be computed recursively by
Revised: N (0 , K ( l ) ) , where the covariance matrix K ( l ) can be computed recursively by

Original: [Equation]
Revised: [Equation]

Original: All proofs of this paper are given in the appendix.
Revised: All proofs of this paper are given in the appendix.

Original: Note that different from a usual GP, which is a random function deﬁned over a connected region of the Euclidean space, here z ( L ) is deﬁned over a discrete set of graph nodes.
Revised: Note that different from a usual GP, which is a random function deﬁned over a connected region of the Euclidean space, here z ( L ) is deﬁned over a discrete set of graph nodes.

Original: In the usual use of a graph in machine learning, this set is ﬁnite, such that the function distribution degenerates to a multivariate distribution.
Revised: In the usual use of a graph in machine learning, this set is ﬁnite, such that the function distribution degenerates to a multivariate distribution.

Original: In semi-supervised learning, the dimension of the distribution, N , is ﬁxed when one conducts transductive learning; but it will vary in the inductive setting because the graph will have new nodes and edges.
Revised: In semi-supervised learning, the dimension of the distribution, N , is ﬁxed when one conducts transductive learning; but it will vary in the inductive setting because the graph will have new nodes and edges.

Original: One special care of a graph-based GP over a usual GP is that the covariance matrix will need to be recomputed from scratch whenever the graph alters.
Revised: One special care of a graph-based GP over a usual GP is that the covariance matrix will need to be recomputed from scratch whenever the graph alters.

Original: Theorem 1 leaves out the base deﬁnition C (0) , whose entry denotes the covariance between two input nodes.
Revised: Theorem 1 leaves out the base deﬁnition C (0) , whose entry denotes the covariance between two input nodes.

Original: The traditional literature uses the inner product C (0) ( x, x (cid:48) )
Revised: The traditional literature uses the inner product C (0) ( x, x (cid:48) )

Original: = x · x (cid:48)
Revised: = x · x (cid:48)

Original: d 0 (Lee et al., 2018), but nothing prevents us using any positive-deﬁnite kernel alternatively.
Revised: d 0 (Lee et al., 2018), but nothing prevents us using any positive-deﬁnite kernel alternatively.

Original: For example, we could use the squared exponential kernel C (0) ( x, x (cid:48) )
Revised: For example, we could use the squared exponential kernel C (0) ( x, x (cid:48) )

Original: =
Revised: =

Original: exp (cid:18) − 12 (cid:80) d 0 j =1 (cid:16) x
Revised: exp (cid:18) − 12 (cid:80) d 0 j =1 (cid:16) x

Original: j − x (cid:48)
Revised: j − x (cid:48)

Original: j (cid:96) j (cid:17) 2 (cid:19) .
Revised: j (cid:96) j (cid:17) 2 (cid:19) .

Original: Such ﬂexibility in essence performs an implicit feature transformation as preprocessing.
Revised: Such ﬂexibility in essence performs an implicit feature transformation as preprocessing.

Original: 3. U NIVERSALITY
Revised: 3. U NIVERSALITY

Original: A covariance kernel is positive deﬁnite; hence, the Moore–Aronszajn theorem (Aronszajn, 1950) suggests that it deﬁnes a unique Hilbert space for which it is a reproducing kernel.
Revised: A covariance kernel is positive deﬁnite; hence, the Moore–Aronszajn theorem (Aronszajn, 1950) suggests that it deﬁnes a unique Hilbert space for which it is a reproducing kernel.

Original: If this space is dense, then the kernel is called universal .
Revised: If this space is dense, then the kernel is called universal .

Original: One can verify universality by checking if the kernel matrix is positive deﬁnite for any set of distinct points.
Revised: One can verify universality by checking if the kernel matrix is positive deﬁnite for any set of distinct points.

Original: For the case of graphs, it sufﬁces to verify if the covariance matrix for all nodes is positive deﬁnite.
Revised: For the case of graphs, it sufﬁces to verify if the covariance matrix for all nodes is positive deﬁnite.

Original: We do this job for the ReLU activation function.
Revised: We do this job for the ReLU activation function.

Original: It is known that the kernel E w ∼N (0 ,I d ) [ φ ( w · x ) φ ( w · x (cid:48) )] admits a closed-form expression as a function of the angle between x and x (cid:48) , hence named the arc-cosine kernel (Cho & Saul, 2009).
Revised: It is known that the kernel E w ∼N (0 ,I d ) [ φ ( w · x ) φ ( w · x (cid:48) )] admits a closed-form expression as a function of the angle between x and x (cid:48) , hence named the arc-cosine kernel (Cho & Saul, 2009).

Original: We ﬁrst establish the following lemma that states that the kernel is universal over a half-space.
Revised: We ﬁrst establish the following lemma that states that the kernel is universal over a half-space.

Original: Lemma 2.
Revised: Lemma 2.

Original: The arc-cosine kernel is universal on the upper-hemisphere
Revised: The arc-cosine kernel is universal on the upper-hemisphere

Original: [Equation]
Revised: S = (cid:8) x ∈ R

Revised: d : (cid:107) x (cid:107) 2 = 1 , x 1 > 0 (cid:9) for all d ≥ 2 .

Original: It is also known that the expectation in (3) is proportional to the arc-cosine kernel up to a factor (cid:112)
Revised: It is also known that the expectation in (3) is proportional to the arc-cosine kernel up to a factor (cid:112)

Original: K ( l ) ( x, x ) K ( l ) ( x (cid:48) , x (cid:48) )
Revised: K ( l ) ( x, x ) K ( l ) ( x (cid:48) , x (cid:48) )

Original: (Lee et al., 2018).
Revised: (Lee et al., 2018).

Original: Therefore, we iteratively work on the post-activation covariance (3) and the pre-activation covariance (4) and show that the covariance kernel resulting from the limiting GCN is universal, for any GCN with three or more layers.
Revised: Therefore, we iteratively work on the post-activation covariance (3) and the pre-activation covariance (4) and show that the covariance kernel resulting from the limiting GCN is universal, for any GCN with three or more layers.

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume A is irreducible and non-negative and C (0) does not contain two linearly dependent rows.
Revised: Assume A is irreducible and non-negative and C (0) does not contain two linearly dependent rows.

Original: Then, K ( l ) is positive deﬁnite for all l ≥ 3 .
Revised: Then, K ( l ) is positive deﬁnite for all l ≥ 3 .

Original: 3. L IMIT IN THE D EPTH
Revised: 3. L IMIT IN THE D EPTH

Original: The depth of a neural network exhibits interesting behaviors.
Revised: The depth of a neural network exhibits interesting behaviors.

Original: Deep learning tends to favor deep networks because of their empirically outstanding performance, exempliﬁed by generations of convolutional networks for the ImageNet classiﬁcation (Krizhevsky et al., 2012; Wortsman et al., 2022); while graph neural networks are instrumental to be shallow because of the over-smoothing and oversquashing properties (Li et al., 2018; Topping et al., 2022).
Revised: Deep learning tends to favor deep networks because of their empirically outstanding performance, exempliﬁed by generations of convolutional networks for the ImageNet classiﬁcation (Krizhevsky et al., 2012; Wortsman et al., 2022); while graph neural networks are instrumental to be shallow because of the over-smoothing and oversquashing properties (Li et al., 2018; Topping et al., 2022).

Original: For multi-layer perceptrons (networks with fully connected layers), several previous works have noted that the recurrence relation of the covariance kernel across layers leads to convergence to a ﬁxed-point kernel, when the depth L → ∞ (see, e.g., Lee et al.
Revised: For multi-layer perceptrons (networks with fully connected layers), several previous works have noted that the recurrence relation of the covariance kernel across layers leads to convergence to a ﬁxed-point kernel, when the depth L → ∞ (see, e.g., Lee et al.

Original: (2018); in Appendix A.5, we elaborate this limit).
Revised: (2018); in Appendix B.5, we elaborate this limit).

Original: In what follows, we offer the parallel analysis for GCN.
Revised: In what follows, we offer the parallel analysis for GCN.

Original: Theorem 4.
Revised: Theorem 4.

Original: Assume A is symmetric, irreducible, aperiodic, and non-negative with PerronFrobenius eigenvalue λ > 0 .
Revised: Assume A is symmetric, irreducible, aperiodic, and non-negative with PerronFrobenius eigenvalue λ > 0 .

Original: The following results hold as l → ∞.
Revised: The following results hold as l → ∞

Original: When σ 2 b = 0 , ρ min ( K ( l ) ) (cid:37) 1 , where ρ min denotes the minimum correlation between any two nodes x and x (cid:48) .
Revised: When σ 2 b = 0 , ρ min ( K ( l ) ) (cid:37) 1 , where ρ min denotes the minimum correlation between any two nodes x and x (cid:48) .

Original: When σ 2 w < 2 /λ 2 , a subsequence of K ( l ) converges to some matrix.
Revised: When σ 2 w < 2 /λ 2 , a subsequence of K ( l ) converges to some matrix.

Original: When σ 2 w > 2 /λ 2 , let c l = ( σ 2 w λ 2 / 2) l ; then, K ( l ) /c
Revised: When σ 2 w > 2 /λ 2 , let c l = ( σ 2 w λ 2 / 2) l ; then, K ( l ) /c

Original: l → vv T where v is an eigenvector corresponding to λ .
Revised: l → vv T where v is an eigenvector corresponding to λ .

Original: A few remarks follow.
Revised: A few remarks follow.

Original: The ﬁrst case implies that the correlation matrix converges monotonously to a matrix of all ones.
Revised: The ﬁrst case implies that the correlation matrix converges monotonously to a matrix of all ones.

Original: As a consequence, up to some scaling c (cid:48) l that may depend on l , the scaled covariance matrix K ( l ) /c
Revised: As a consequence, up to some scaling c (cid:48) l that may depend on l , the scaled covariance matrix K ( l ) /c

Original: (cid:48)
Revised: (cid:48)

Original: l converges to a rank-1 matrix.
Revised: l converges to a rank-1 matrix.

Original: The third case shares a similar result, with the limit explicitly spelled out, but note that the eigenvector v may not be normalized.
Revised: The third case shares a similar result, with the limit explicitly spelled out, but note that the eigenvector v may not be normalized.

Original: The second case is challenging to analyze.
Revised: The second case is challenging to analyze.

Original: According to empirical veriﬁcation, we speculate a stronger result— convergence of K ( l ) to a unique ﬁxed point—may hold. 
Revised: According to empirical veriﬁcation, we speculate a stronger result— convergence of K ( l ) to a unique ﬁxed point—may hold. [Table]

Original:  S CALABLE C OMPUTATION THROUGH L OW -R ANK A PPROXIMATION
Revised:  S CALABLE C OMPUTATION THROUGH L OW -R ANK A PPROXIMATION

Original: The computation of the covariance matrix K ( L ) through recursion (3)–(4) is the main computational bottleneck for GP posterior inference.
Revised: The computation of the covariance matrix K ( L ) through recursion (3)–(4) is the main computational bottleneck for GP posterior inference.

Original: We start the exposition with the mean prediction.
Revised: We start the exposition with the mean prediction.

Original: We compute Algorithm Require: Q (0) such that K (0) ≈ Q (0) Q (0) T 1: for l =0 , . .
Revised: We compute the posterior mean (cid:98) y ∗ =

Original: , L − 1 do 2:

Original: Compute (cid:98) K ( l ) : a = Q (0):: Q (0) a : T 3:

Original: Compute (cid:98) C ( l ) : a by (5), where C ( l ) (resp.

Original: K ( l ) )

Original: entries are replaced by (cid:98) C ( l ) (resp.

Original: (cid:98) K ( l ) ) entries 4:

Original: Compute Q ( l +1) = (cid:104) σ w A (cid:98) C ( l ) : a ( (cid:98) C ( l ) aa ) −σ b 1 N × 1 (cid:105) 5: end for the posterior mean (cid:98) y ∗ =

Original: K ( L ) ∗ b ( K ( L )
Revised: K ( L ) ∗ b ( K ( L )

Original: bb + (cid:15)I ) − 1 y b , where the subscripts b and ∗ denote the training set and the prediction set, respectively; and (cid:15) , called the nugget , is the noise variance of the training data.
Revised: bb + (cid:15)I ) − 1 y b , where the subscripts b and ∗ denote the training set and the prediction set, respectively; and (cid:15) , called the nugget , is the noise variance of the training data.

Original: Let there be N b training nodes and N ∗ prediction nodes.
Revised: Let there be N b training nodes and N ∗ prediction nodes.

Original: It is tempting to compute only the ( N b + N ∗ ) × N b submatrix of K ( L ) for the task, but the recursion (4) requires the full C ( L − 1) at the presence of A , and hence all the full C ( l ) ’s and K ( l ) ’
Revised: It is tempting to compute only the ( N b + N ∗ ) × N b submatrix of K ( L ) for the task, but the recursion (4) requires the full C ( L − 1) at the presence of A , and hence all the full C ( l ) ’s and K ( l ) ’

Original: s.
Revised: s.

Original: To reduce the computational costs, we resort to a low-rank approximation of C ( l ) , from which we easily see that K ( l +1) is also low-rank.
Revised: To reduce the computational costs, we resort to a low-rank approximation of C ( l ) , from which we easily see that K ( l +1) is also low-rank.

Original: Before deriving the approximation recursion, we note (again) that for the ReLU activation φ , C ( l ) in (3) is the arc-cosine kernel with a closed-form expression:
Revised: Before deriving the approximation recursion, we note (again) that for the ReLU activation φ , C ( l ) in (3) is the arc-cosine kernel with a closed-form expression:

Original: C ( l ) xx(cid:48)
Revised: C ( l ) xx

Original: = 1 2 π (cid:113) K ( l ) xx

Original: [Equation]
Revised: [Equation]

Original: where θ ( l ) xx (cid:48)
Revised: where θ ( l ) xx (cid:48)

Original: =
Revised: =

Original: arccos  
Revised: arccos  

Original: K ( l ) xx (cid:48) (cid:113) K 
Revised: K ( l ) xx (cid:48) (cid:113) K ( l

Original: ( l ) xx K ( l ) x (cid:48) x (cid:48) 
Revised: [Equation] xx K ( l ) x (cid:48) x (cid:48) 

Original:  .
Revised:  .

Original: Hence, the main idea is that starting with a low-rank approximation of K ( l ) , compute an approximation of C ( l ) by using (5), and then obtain an approximation of K ( l +1) based on (4); then, repeat.
Revised: Hence, the main idea is that starting with a low-rank approximation of K ( l ) , compute an approximation of C ( l ) by using (5), and then obtain an approximation of K ( l +1) based on (4); then, repeat.

Original: To derive the approximation, we use the subscript a to denote a set of landmark nodes with cardinality N a .
Revised: To derive the approximation, we use the subscript a to denote a set of landmark nodes with cardinality N a .

Original: The Nystr ¨ om approximation (Drineas & Mahoney, 2005) of K (0) is K (0): a ( K (0) aa ) − 1 K (0) a : , where the subscript : denotes retaining all rows/columns.
Revised: The Nystr ¨ om approximation (Drineas & Mahoney, 2005) of K (0) is K (0): a ( K (0) aa ) − 1 K (0) a : , where the subscript : denotes retaining all rows/columns.

Original: We rewrite this approximation in the
Revised: We rewrite this approximation in the

Original: Cholesky style as Q (0) Q (0) T , where Q (0) =
Revised: Cholesky style as Q (0) Q (0) T , where Q (0) =

Original: K (0): a ( K (0) aa ) − 12 has size N × N a .
Revised: K (0): a ( K (0) aa ) − 12 has size N × N a .

Original: Proceed with induction.
Revised: Proceed with induction.

Original: Let K ( l ) be approximated by (cid:98) K ( l ) =
Revised: Let K ( l ) be approximated by (cid:98) K ( l ) =

Original: Q ( l ) Q ( l ) T , where Q ( l ) has size N × ( N a + 1) .
Revised: Q ( l ) Q ( l ) T , where Q ( l ) has size N × ( N a + 1) .

Original: We apply (5) to compute an approximation to C ( l ) : a , namely (cid:98) C ( l ) : a , by using (cid:98) K ( l ) : a .
Revised: We apply (5) to compute an approximation to C ( l ) : a , namely (cid:98) C ( l ) : a , by using (cid:98) K ( l ) : a .

Original: Then, (4) leads to a
Revised: Then, (4) leads to a

Original: Cholesky style approximation of K ( l +1) :
Revised: Cholesky style approximation of K ( l +1) :

Original: [Equation]
Revised: [Equation]

Original:  where Q ( l +1) = (cid:104) [Equation] l ) [Equation]
Revised: , where Q ( l +1) = (cid:104) σ w A (cid:98) C ( l ) : a ( (cid:98) C ( l ) aa ) −σ b 1 N × 1 (cid:105) .

Original: l [Equation] σ b 1 N × 1 (cid:105) . Clearly, Q ( l +1) has size N × ( N a +1) , completing the induction.
Revised: Clearly, Q ( l +1) has size N × ( N a +1) , completing the induction.

Original: In summary, K ( L ) is approximated by a rank- ( N a +1) matrix (cid:98) K
Revised: In summary, K ( L ) is approximated by a rank- ( N a +1) matrix (cid:98) K

Original: ( L ) =
Revised: ( L ) =

Original: Q ( L ) Q ( L ) T .
Revised: Q ( L ) Q ( L ) T .

Original: The computation of Q ( L ) is summarized in Algorithm 1.
Revised: The computation of Q ( L ) is summarized in Algorithm 1.

Original: Once it is formed, the posterior mean is computed as
Revised: Once it is formed, the posterior mean is computed as

Original: [Equation]
Revised: [Equation]

Original: where note that the matrix to be inverted has size ( N a + 1) ×
Revised: where note that the matrix to be inverted has size ( N a + 1) ×

Original: ( N a + 1) , which is assumed to be signiﬁcantly smaller than N b × N b .
Revised: ( N a + 1) , which is assumed to be signiﬁcantly smaller than N b × N b .

Original: Similarly, the posterior variance is
Revised: Similarly, the posterior variance is

Original: [Equation]
Revised: [Equation]

Revised: [Equation]

Original: The computational costs of Q ( L ) and the posterior inference (6)–(7) are summarized in Table
Revised: The computational costs of Q ( L ) and the posterior inference (6)–(7) are summarized in Table

Original: C OMPOSING G RAPH N EURAL N ETWORK -I NSPIRED KERNELS
Revised: Algorithm K ( L ) ≈ (cid:98) K

Original: Theorem 1, together with its proof, suggests that the covariance matrix of the limiting GP can be computed in a composable manner.
Revised: ( L ) = Q ( L ) Q ( L ) T

Original: Moreover, the derivation of Algorithm 1 indicates that the lowrank approximation of the covariance matrix can be similarly composed.
Revised: Require: Q (0) such that K (0) ≈ Q (0) Q (0) T 1: for l = 0 , . . , L − 1 do 2: Compute (cid:98) K ( l ) : a = Q (0):: Q (0) a : T 3:

Original: Altogether, such a nice
Revised: Compute (cid:98) C ( l ) : a by (5), where C ( l ) (resp.

Original: [Table]
Revised: K ( l ) )

Original: [Table]
Revised: entries are replaced by (cid:98) C ( l ) (resp. (cid:98) K ( l ) ) entries 4: Compute Q ( l +1) = (cid:104) σ w A (cid:98) C ( l ) : a ( (cid:98) C ( l ) aa ) −σ b 1 N × 1 (cid:105) 5: end for

Original: [Table]
Revised: [Table]

Original: X ← [Table]
Revised: X ← X (0)

Original: K ← [Table]
Revised: K ← C (0)

Original: Q ← [Table] [Table]
Revised: Q ← Chol( C (0) )

Original: [Table]
Revised: [Table]

Original: X ← X + b
Revised: X ← X + b

Original: K ← K + σ 2 b 1 N × N
Revised: K ← K + σ 2 b 1 N × N

Original: Q ← [ Q σ b 1 N × 1 ]
Revised: Q ← [ Q σ b 1 N × 1 ]

Original: Weight

Original: [Table]
Revised: [Table]

Original: X ← XW
Revised: X ← XW

Original: K ← σ 2 w K
Revised: K ← σ 2 w K

Original: Q ← σ w Q
Revised: Q ← σ w Q

Original: Mixed

Original: [Table]
Revised: [Table]

Original: X ← X ( αI + βW )
Revised: X ← X ( αI + βW )

Original: K ← ( [Table] + β [Table]
Revised: K ← ( α 2 + β 2 σ 2 w ) K

Original: w ) K Q ← (cid:112) [Table] + 
Revised: Q ← (cid:112) α 2 + β 2 σ 2 w Q

Original: [Table]

Original: σ

Original: [Table]

Original: Q

Original: Graph convolution
Revised: Graph convolution

Original: X ← AX
Revised: X ← AX

Original: K ← AKA T
Revised: K ← AKA T

Original: Q ← AQ
Revised: Q ← AQ

Original: Activation
Revised: Activation

Original: X ← φ ( X )
Revised: X ← φ ( X )

Original: K ← g ( K )
Revised: K ← g ( K )

Original: Q ← Chol( g ( QQ T ))
Revised: Q ← Chol( g ( QQ T ))

Original: Independent addition
Revised: Independent addition

Original: X ← X 1 + X
Revised: X ← X 1 + X

Original: K ← K 1 + K
Revised: K ← K 1 + K

Original: Q ← [ Q
Revised: Q ← [ Q

Revised: Q 2 ]

Original: Q 2 ] property allows one to easily derive the corresponding covariance matrix and its approximation for a new GNN architecture, like writing a program and obtaining a transformation of it automatically through operator overloading (Novak et al., 2020): the covariance matrix is a transformation of the
Revised: [Equation] Theorem 1, together with its proof, suggests that the covariance matrix of the limiting GP can be computed in a composable manner. Moreover, the derivation of Algorithm 1 indicates that the lowrank approximation of the covariance matrix can be similarly composed. Altogether, such a nice property allows one to easily derive the corresponding covariance matrix and its approximation for a new GNN architecture, like writing a program and obtaining a transformation of it automatically through operator overloading (Novak et al., 2020): the covariance matrix is a transformation of the

Original: GNN and the composition of the former is in exactly the same manner and order as that of the latter.
Revised: GNN and the composition of the former is in exactly the same manner and order as that of the latter.

Original: We call the covariance matrices programmable .
Revised: We call the covariance matrices programmable .

Original: For example, we write a GCN layer as X ← Aφ ( X ) W + b , where for notational simplicity, X denotes pre-activation rather than post-activation as in the earlier sections.
Revised: For example, we write a GCN layer as X ← Aφ ( X ) W + b , where for notational simplicity, X denotes pre-activation rather than post-activation as in the earlier sections.

Original: The activation φ on X results in a transformation of the kernel matrix K into g ( K ) , deﬁned as:
Revised: The activation φ on X results in a transformation of the kernel matrix K into g ( K ) , deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: due to (3).
Revised: due to (3).

Original: Moreover, if K admits a low-rank approximation QQ T , then g ( K ) admits a low-rank approximation PP T where P = Chol( g ( K )) with 
Revised: Moreover, if K admits a low-rank approximation QQ T , then g ( K ) admits a low-rank approximation PP T where P = Chol( g ( K )) with Chol

Original: [Equation]
Revised: [Equation]

Original: The next operation—graph convolution—multiplies A to the left of the post-activation.
Revised: The next operation—graph convolution—multiplies A to the left of the post-activation.

Original: Correspondingly, the covariance matrix K is transformed to AKA T and the low-rank approximation factor Q is transformed to AQ .
Revised: Correspondingly, the covariance matrix K is transformed to AKA T and the low-rank approximation factor Q is transformed to AQ .

Original: Then, the operation—multiplying the weight matrix W to the right—will transform K to σ 2 w K and Q to σ w Q .
Revised: Then, the operation—multiplying the weight matrix W to the right—will transform K to σ 2 w K and Q to σ w Q .

Original: Finally, adding the bias b will transform K to K + σ 2 b 1 N × N and Q to [ Q σ b 1 N × 1 ] .
Revised: Finally, adding the bias b will transform K to K + σ 2 b 1 N × N and Q to [ Q σ b 1 N × 1 ] .

Original: Altogether, we have obtained the following updates per layer:
Revised: Altogether, we have obtained the following updates per layer:

Original: [Equation]
Revised: [Equation]

Original: One may verify the K update against (3)–(4) and the Q update against Algorithm 1.
Revised: One may verify the K update against (3)–(4) and the Q update against Algorithm 1.

Original: Both updates can be automatically derived based on the update of X .
Revised: Both updates can be automatically derived based on the update of X .

Original: We summarize the building blocks of a GNN and the corresponding kernel/low-rank operations in
Revised: We summarize the building blocks of a GNN and the corresponding kernel/low-rank operations in

Original: Table 2.
Revised: Table 2.

Original: The independent-addition building block is applicable to skip/residual connections.
Revised: The independent-addition building block is applicable to skip/residual connections.

Original: For example, here is the composition for the GCNII layer (Chen et al., 2020) without a bias term, where a skip connection with X (0) occurs:
Revised: For example, here is the composition for the GCNII layer (Chen et al., 2020) without a bias term, where a skip connection with X (0) occurs:

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table] For another example of the composability, we consider the popular GIN layer (Xu et al., 2019), which we assume uses a 2-layer MLP after the neighborhood aggregation:
Revised: For another example of the composability, we consider the popular GIN layer (Xu et al., 2019), which we assume uses a 2-layer MLP after the neighborhood aggregation:

Original: GIN:
Revised: GIN

Original: [Equation]
Revised: [Equation]

Original: where 
Revised: where B = σ 2 w Ag ( K ) A T + σ 2 b 1 N × N Q ← (cid:2) σ w Chol( g ( PP T )) σ b 1 N × 1 (cid:3)

Original: [Equation] where P = (cid:2) [Equation]
Revised: where P = (cid:2) σ w A Chol( g ( QQ T )) σ b 1 N × 1 (cid:3) .

Original: Additionally, the updates for a GraphSAGE layer (Hamilton et al., 2017) are given in Appendix B.
Revised: Additionally, the updates for a GraphSAGE layer (Hamilton et al., 2017) are given in Appendix C.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct a comprehensive set of experiments to evaluate the performance of the
Revised: In this section, we conduct a comprehensive set of experiments to evaluate the performance of the

Original: GP kernels derived by taking limits on the layer width of GCN and other GNNs.
Revised: GP kernels derived by taking limits on the layer width of GCN and other GNNs.

Original: We demonstrate that these GPs are comparable with GNNs in prediction performance, while being signiﬁcantly faster to compute.
Revised: We demonstrate that these GPs are comparable with GNNs in prediction performance, while being signiﬁcantly faster to compute.

Original: We also show that the low-rank version scales favorably, suitable for practical use.
Revised: We also show that the low-rank version scales favorably, suitable for practical use.

Original: Datasets.
Revised: Datasets.

Original: The experiments are conducted on several benchmark datasets of varying sizes, covering both classiﬁcation and regression.
Revised: The experiments are conducted on several benchmark datasets of varying sizes, covering both classiﬁcation and regression.

Original: They include predicting the topic of scientiﬁc papers organized in a citation network (Cora, Citeseer, PubMed, and ArXiv); predicting the community of online posts based on user comments (Reddit), and predicting the average daily trafﬁc of Wikipedia pages using hyperlinks among them (Chameleon, Squirrel, and Crocodile).
Revised: They include predicting the topic of scientiﬁc papers organized in a citation network (Cora, Citeseer, PubMed, and ArXiv); predicting the community of online posts based on user comments (Reddit), and predicting the average daily trafﬁc of Wikipedia pages using hyperlinks among them (Chameleon, Squirrel, and Crocodile).

Original: A summary of the datasetsis given in Table 3 and details (including sources and preprocessing) are given in Appendix C.
Revised: Details of the datasets (including sources and preprocessing) are given in Appendix D.

Original: Experiment environment, training details, and hyperparameters are given in Appendix C.
Revised: Experiment environment, training details, and hyperparameters are given in Appendix D.

Original: Prediction Performance: GCN-based comparison.
Revised: Prediction Performance: GCN-based comparison.

Original: We ﬁrst conduct the semi-supervised learning tasks on all datasets by using GCN and GPs with different kernels.
Revised: We ﬁrst conduct the semi-supervised learning tasks on all datasets by using GCN and GPs with different kernels.

Original: These kernels include the one equivalent to the limiting GCN (GCNGP), a usual squared-exponential kernel (RBF), and the GGP kernel proposed by Ng et al.
Revised: These kernels include the one equivalent to the limiting GCN (GCNGP), a usual squared-exponential kernel (RBF), and the GGP kernel proposed by Ng et al.

Original:  Each of these kernels has a low-rank version (sufﬁxed with -X).
Revised: 3 Each of these kernels has a low-rank version (sufﬁxed with -X).

Original: RBF-X and GGP-X 4 use the Nystr¨om approximation, consistent with GCNGP-X.
Revised: RBF-X and GGP-X 4 use the Nystr ¨ om approximation, consistent with GCNGP-X.

Original: GPs are by nature suitable for regression.
Revised: GPs are by nature suitable for regression.

Original: For classiﬁcation tasks, we use the one-hot representation of labels to set up a multi-output regression.
Revised: For classiﬁcation tasks, we use the one-hot representation of labels to set up a multi-output regression.

Original: Then, we take the coordinate with the largest output as the class prediction.
Revised: Then, we take the coordinate with the largest output as the class prediction.

Original: Such an ad hoc treatment is widely used in the literature, as other more principled approaches (such as using the Laplace approximation on the non-Gaussian posterior) are too time-consuming for large datasets, meanwhile producing no noticeable gain in accuracy.
Revised: Such an ad hoc treatment is widely used in the literature, as other more principled approaches (such as using the Laplace approximation on the non-Gaussian posterior) are too time-consuming for large datasets, meanwhile producing no noticeable gain in accuracy.

Original: Table 4 summarizes the accuracy for classiﬁcation and the coefﬁcient of determination, R 2 , for regression.
Revised: Table 3 summarizes the accuracy for classiﬁcation and the coefﬁcient of determination, R 2 , for regression.

Original: Whenever randomness is involved, the performance is reported as an average over ﬁve runs.
Revised: Whenever randomness is involved, the performance is reported as an average over ﬁve runs.

Original: The results of the two tasks show different patterns.
Revised: The results of the two tasks show different patterns.

Original: For classiﬁcation, GCNGP(-X) is sligtly better than GCN and GGP(-X), while RBF(-X) is signiﬁcantly worse than all others; moreover, the low-rank version is outperformed by using the full kernel matrix.
Revised: For classiﬁcation, GCNGP(-X) is sligtly better than GCN and GGP(-X), while RBF(-X) is signiﬁcantly worse than all others; moreover, the low-rank version is outperformed by using the full kernel matrix.

Original: On the other hand, for regression,
Revised: On the other hand, for regression,

Original: GCNGP(-X) signiﬁcantly outperforms GCN, RBF(-X), and GGP(-X); and the low-rank version[Table] [Table] becomes better.
Revised: GCNGP(-X) signiﬁcantly outperforms GCN, RBF(-X), and GGP(-X); and the low-rank version becomes better.

Original: The less competitive performance of RBF(-X) is expected, as it does not leverage the graph inductive bias.
Revised: The less competitive performance of RBF(-X) is expected, as it does not leverage the graph inductive bias.

Original: It is attractive that GCNGP(-X) is competitive with GCN.
Revised: It is attractive that GCNGP(-X) is competitive with GCN.

Original: Prediction Performance:
Revised: Prediction Performance:

Original: Comparison with other GNNs.
Revised: Comparison with other GNNs.

Original: In addition to GCN, we conduct experiments with several popularly used GNN architectures (GCNII, GIN, and GraphSAGE) and GPs  with the corresponding kernels.
Revised: In addition to GCN, we conduct experiments with several popularly used GNN architectures (GCNII, GIN, and GraphSAGE) and GPs [Table] [Table] with the corresponding kernels.

Original: We test with the three largest datasets: PubMed, ArXiv, and Reddit, for the latter two of which a low-rank version of the GPs is used for computational feasibility.
Revised: We test with the three largest datasets: PubMed, ArXiv, and Reddit, for the latter two of which a low-rank version of the GPs is used for computational feasibility.

Original: Table 5 summarizes the results.
Revised: Table 4 summarizes the results.

Original: The observations on other GNNs extend similarly those on the
Revised: The observations on other GNNs extend similarly those on the

Original: GCN.
Revised: GCN.

Original: In particular, on PubMed the GPs noticeably improve over the corresponding GNNs, while on ArXiv and Reddit the two families perform rather similarly.
Revised: In particular, on PubMed the GPs noticeably improve over the corresponding GNNs, while on ArXiv and Reddit the two families perform rather similarly.

Original: An exception is GIN for ArXiv, which signiﬁcantly underperforms the GP counterpart, as well as other GNNs.
Revised: An exception is GIN for ArXiv, which signiﬁcantly underperforms the GP counterpart, as well as other GNNs.

Original: It may improve with an extensive hyperparameter tuning.
Revised: It may improve with an extensive hyperparameter tuning.

Original: Running time.
Revised: Running time.

Original: We compare the running time of the methods covered by Table 4.
Revised: We compare the running time of the methods covered by Table 3.

Original: Different from usual neural networks, the training and inference of GNNs do not decouple in full-batch training.
Revised: Different from usual neural networks, the training and inference of GNNs do not decouple in full-batch training.

Original: Moreover, there is not a universally agreed split between the training and the inference steps in GPs.
Revised: Moreover, there is not a universally agreed split between the training and the inference steps in GPs.

Original: Hence, we compare the total time for each method.
Revised: Hence, we compare the total time for each method.

Original: Figure 1 plots the timing results, normalized against the GCN time for ease of comparison.
Revised: Figure 1 plots the timing results, normalized against the GCN time for ease of comparison.

Original: It suggests that GCNGP(-X) is generally faster than GCN.
Revised: It suggests that GCNGP(-X) is generally faster than GCN.

Original: Note that the vertical axis is in the logarithmic scale.
Revised: Note that the vertical axis is in the logarithmic scale.

Original: Hence, for some of the datasets, the speedup is even one to two orders of magnitude.
Revised: Hence, for some of the datasets, the speedup is even one to two orders of magnitude.

Original: Scalability.
Revised: Scalability.

Original: For graphs, especially under the semi-supervised learning setting, the computational cost of a GP is much more complex than that of a usual one (which can be simply described as “cubic in the training set size”).
Revised: For graphs, especially under the semi-supervised learning setting, the computational cost of a GP is much more complex than that of a usual one (which can be simply described as “cubic in the training set size”).

Original: One sees in Table 1 the many factors that determine the cost of our graph-based low-rank kernels.
Revised: One sees in Table 1 the many factors that determine the cost of our graph-based low-rank kernels.

Original: To explore the practicality of the proposed method, we use the timings gathered for Figure 1 to obtain an empirical scaling with respect to the graph size, M + N .
Revised: To explore the practicality of the proposed method, we use the timings gathered for Figure 1 to obtain an empirical scaling with respect to the graph size, M + N .

Original: Figure 2 ﬁts the running times, plotted in the log-log scale, by using a straight line.
Revised: Figure 2 ﬁts the running times, plotted in the log-log scale, by using a straight line.

Original: We see that for neither GCN nor GCNGP(-X), the actual running time closely follows a polynomial complexity.
Revised: We see that for neither GCN nor GCNGP(-X), the actual running time closely follows a polynomial complexity.

Original: However, interestingly, the least-squares ﬁttings all lead to a slope approximately 1, which agrees with a linear cost.
Revised: However, interestingly, the least-squares ﬁttings all lead to a slope approximately 1, which agrees with a linear cost.

Original: Theoretically, only GCNGP-X and GCN are approximately linear with respect to
Revised: Theoretically, only GCNGP-X and GCN are approximately linear with respect to

Original: M + N , while GCNGP is cubic.
Revised: M + N , while GCNGP is cubic.

Original: Analysis on the depth.
Revised: Analysis on the depth.

Original: The performance of GCN deteriorates with more layers, known as the oversmoothing phenomenon.
Revised: The performance of GCN deteriorates with more layers, known as the oversmoothing phenomenon.

Original: Adding residual/skip connections mitigates the problem, such as in
Revised: Adding residual/skip connections mitigates the problem, such as in

Original: GCNII.
Revised: GCNII.

Original: A natural question asks if the corresponding GP kernels behave similarly.
Revised: A natural question asks if the corresponding GP kernels behave similarly.

Original: Figure 3 shows that the trends of GCN and GCNII are indeed as expected.
Revised: Figure 3 shows that the trends of GCN and GCNII are indeed as expected.

Original: Interestingly, their GP counterparts both remain stable for depth L as large as 12.
Revised: Interestingly, their GP counterparts both remain stable for depth L as large as 12.

Original: Our depth analysis (Theorem 4) suggests that in the limit, the GPs may perform less well because the kernel matrix may degenerate to rank 1.
Revised: Our depth analysis (Theorem 4) suggests that in the limit, the GPs may perform less well because the kernel matrix may degenerate to rank 1.

Original: This empirical result indicates that the drop in performance may have not started yet.
Revised: This empirical result indicates that the drop in performance may have not started yet.

Original: Analysis on the landmark set.
Revised: Analysis on the landmark set.

Original: The number of landmark nodes, N a , controls the approximation quality of the low-rank kernels and hence the prediciton accuracy.
Revised: The number of landmark nodes, N a , controls the approximation quality of the low-rank kernels and hence the prediciton accuracy.

Original: On the other hand, the computational costs summarized in Table 1 indicate a dependency on N a as high as the third power.
Revised: On the other hand, the computational costs summarized in Table 1 indicate a dependency on N a as high as the third power.

Original: It is crucial to develop an empirical understanding of the accuracy-time trade-off it incurs.
Revised: It is crucial to develop an empirical understanding of the accuracy-time trade-off it incurs.

Original: Figure 4 clearly shows that as N a becomes larger, the running time increase is not linear, while the increase of accuracy diminishes as the landmark set approaches the training set.
Revised: Figure 4 clearly shows that as N a becomes larger, the running time increase is not linear, while the increase of accuracy diminishes as the landmark set approaches the training set.

Original: It is remarkable that using only 1/800 of the training set as landmark nodes already achieves an accuracy surpassing that of GCN, by using time that is only a tiny fraction of the time otherwise needed to gain an additional 1% increase in the accuracy.
Revised: It is remarkable that using only 1/800 of the training set as landmark nodes already achieves an accuracy surpassing that of GCN, by using time that is only a tiny fraction of the time otherwise needed to gain an additional 1% increase in the accuracy.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have presented a GP approach for semi-supervised learning on graph-structured data, where the covariance kernel incorporates a graph inductive bias by exploiting the relationship between a
Revised: We have presented a GP approach for semi-supervised learning on graph-structured data, where the covariance kernel incorporates a graph inductive bias by exploiting the relationship between a

Original: GP and a GNN with inﬁnitely wide layers.
Revised: GP and a GNN with inﬁnitely wide layers.

Original: Similar to other neural networks priorly investigated, one can work out the equivalent GP (in particular, the covariance kernel) for GCN; and inspired by this equivalence, we formulate a procedure to compose covariance kernels corresponding to many other members of the GNN family.
Revised: Similar to other neural networks priorly investigated, one can work out the equivalent GP (in particular, the covariance kernel) for GCN; and inspired by this equivalence, we formulate a procedure to compose covariance kernels corresponding to many other members of the GNN family.

Original: Moreover, every building block in the procedure has a lowrank counterpart, which allows for building a low-rank approximation of the covariance matrix that facilitates scalable posterior inference.
Revised: Moreover, every building block in the procedure has a lowrank counterpart, which allows for building a low-rank approximation of the covariance matrix that facilitates scalable posterior inference.

Original: We demonstrate the effectiveness of the derived kernels used for semi-supervised learning and show their advantages in computation time over GNNs. 
Revised: We demonstrate the effectiveness of the derived kernels used for semi-supervised learning and show their advantages in computation time over GNNs. A CKNOWLEDGMENTS

Revised: Mihai Anitescu was supported by the U

Revised: S. Department of Energy, Ofﬁce of Science, Ofﬁce of Advanced Scientiﬁc Computing Research (ASCR) under Contract DE-AC02-06CH11347.

Revised: Jie Chen acknowledges supports from the MIT-IBM Watson AI Lab

Original: A P ROOFS AND A DDITIONAL T HEOREMS A. P ROOF OF T HEOREM
Revised: A C ODE

Revised: Code is available at https://github.com/niuzehao/gnn-gp . B P ROOFS AND A DDITIONAL T HEOREMS B. P ROOF OF T HEOREM

Original: We use mathematical induction.
Revised: We use mathematical induction.

Original: First, write the matrix form of (2) as follows:
Revised: First, write the matrix form of (2) as follows:

Original: [Equation]
Revised: [Equation]

Original: Assume each column of Z ( l − 1) is iid, then X ( l − 1) also has iid columns.
Revised: Assume each column of Z ( l − 1) is iid, then X ( l − 1) also has iid columns.

Original: Hence, y ( l ) i ( x ) is a sum of iid terms and thus normal.
Revised: Hence, y ( l ) i ( x ) is a sum of iid terms and thus normal.

Original: Also, ( y ( l ) i ( x 1 ) , · · · , y ( l ) i ( x N )) are jointly multivariate normal, and identically distributed for different i ; so they form a GP.
Revised: Also, ( y ( l ) i ( x 1 ) , · · · , y ( l ) i ( x N )) are jointly multivariate normal, and identically distributed for different i ; so they form a GP.

Original: Moreover, terms like y ( l ) i ( x ) and y ( l ) i (cid:48)
Revised: Moreover, terms like y ( l ) i ( x ) and y ( l ) i (cid:48)

Original: ( x (cid:48) ) with i (cid:54) =
Revised: ( x (cid:48) ) with i (cid:54) =

Original: i (cid:48) are jointly Gaussian, with zero covariance:
Revised: i (cid:48) are jointly Gaussian, with zero covariance:

Original: [Equation]
Revised: Cov

Original: [Equation]
Revised: [Equation]

Original: Thus, they are independent, despite the fact that they may share the same X ( l − 1) terms.
Revised: Thus, they are independent, despite the fact that they may share the same X ( l − 1) terms.

Original: In conclusion, each column of Y ( l ) is an iid GP.
Revised: In conclusion, each column of Y ( l ) is an iid GP.

Original: Hence, each column of Z ( l ) is also an iid GP N (0 , K ( l ) ) .
Revised: Hence, each column of Z ( l ) is also an iid GP N (0 , K ( l ) ) .

Original: The covariance K ( l ) can be computed recursively.
Revised: The covariance K ( l ) can be computed recursively.

Original: Similar to the case of a fully connected network, the covariance of Y ( l ) is
Revised: Similar to the case of a fully connected network, the covariance of Y ( l ) is

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for any j.
Revised: any j

Revised: [Equation]

Original: In the matrix form, this is to say Cov( Y ( l ) j )
Revised: In the matrix form, this is to say Cov( Y ( l ) j )

Original: =
Revised: =

Original: σ 2 w E[ X ( l − 1)
Revised: σ 2 w E[ X ( l − 1)

Original: j
Revised: j

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original:  = [Equation] C ( l [Equation]
Revised: ] = σ 2 w C ( l − 1) .

Original: Then, because
Revised: Then, because

Original: Z ( l ) i =
Revised: Z ( l ) i =

Original: AY ( l ) i + 1 N × 1 b ( l ) i , we obtain
Revised: AY ( l ) i + 1 N × 1 b ( l ) i , we obtain

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: A. P ROOF OF L EMMA
Revised: B. P ROOF OF L EMMA

Original: We know that φ ( w · x ) is the feature mapping of the arc-cosine kernel.
Revised: We know that φ ( w · x ) is the feature mapping of the arc-cosine kernel.

Original: It sufﬁces to show for different x 1 , . . .
Revised: It sufﬁces to show for different x 1 , . . .

Original: , x m ∈ S , the φ ( w, x i ) ’s are linearly independent.
Revised: , x m ∈ S , the φ ( w, x i ) ’s are linearly independent.

Original: We prove this by contradiction.
Revised: We prove this by contradiction.

Original: Assume there exist c 1 , . . .
Revised: Assume there exist c 1 , . . .

Original: , c m not simultaneously zero, such that
Revised: , c m not simultaneously zero, such that

Original: [Equation]
Revised: [Equation]

Original: WLOG, we further assume that m is the smallest integer that satisﬁes (9).
Revised: WLOG, we further assume that m is the smallest integer that satisﬁes (9).

Original: Let e 

Revised: Let e

Original: =
Revised: =

Original: (1 , 0 , · · · , 0) ; then x · e 1 > 0 , ∀ x ∈ S .
Revised: (1 , 0 , · · · , 0) ; then x · e 1 > 0 , ∀ x ∈ S .

Original: Assume x m · e 1 = min 1 ≤ i ≤ m { x i · e 1 } > 0 .
Revised: Assume x m · e 1 = min 1 ≤ i ≤ m { x i · e 1 } > 0 .

Original: Then, for 1 ≤ i ≤ m − 1 , x i · x m 
Revised: Then, for 1 ≤ i ≤ m − 1 , x i · x m x i · e 1 < x m · e 1 = x m · x m x m · e 1 . Therefore, there exists t such that x i · x m x i · e

Original: [Equation]
Revised: [Equation]

Original: Therefore, there exists t such that
Revised: <

Original: [Equation]
Revised: [Equation]

Original: ≤ m −
Revised: < x [Equation] x

Original: [Equation]
Revised: [Equation]

Original:  Let w = x m − te 1 ; then w · x m > 0 , w · x i < 0 , 1 ≤ i ≤ m − 1 .
Revised: · e 1 , 1 ≤ i ≤ m − 1 . Let w = x m − te 1 ; then w · x m > 0 , w · x i < 0 , 1 ≤ i ≤ m − 1 .

Original: Using (9) we know c m ( w · x m )
Revised: Using (9) we know c m ( w · x m )

Original: = 

Revised: =

Original: ; so
Revised: ; so

Original: c m = 0 .
Revised: c m = 0 .

Original: Thus, c 1 , · · · , c m − 1 and φ ( w · x 1 ) , · · · , φ ( w · x m − 1 ) also satisfy (9), with a smaller number m , which forms a contradiction.
Revised: Thus, c 1 , · · · , c m − 1 and φ ( w · x 1 ) , · · · , φ ( w · x m − 1 ) also satisfy (9), with a smaller number m , which forms a contradiction.

Original: A. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: From the recursive relation (3)–(4), we know C (0) is element-wise non-negative.
Revised: From the recursive relation (3)–(4), we know C (0) is element-wise non-negative.

Original: Hence, K (1) is element-wise non-negative, C (1) is element-wise positive, and K (2) is element-wise positive.
Revised: Hence, K (1) is element-wise non-negative, C (1) is element-wise positive, and K (2) is element-wise positive.

Original: Assume the Cholesky decomposition K (2) = LL T and denote L =
Revised: Assume the Cholesky decomposition K (2) = LL T and denote L =

Original: ( l 1 , · · · , l n ) T .
Revised: ( l 1 , · · · , l n ) T .

Original: Let r i = (cid:107) l i (cid:107) 2 > 0 and x i = l i r i .
Revised: Let r i = (cid:107) l i (cid:107) 2 > 0 and x i = l i r i .

Original: Then, x 1 = ± e 1 , and WLOG we assume x 1 = e 1 .
Revised: Then, x 1 = ± e 1 , and WLOG we assume x 1 = e 1 .

Original: Then for each x i , we know
Revised: Then for each x i , we know

Original: [Equation]
Revised: [Equation]

Original: and each (cid:107) x i (cid:107) 2 = 1 .
Revised: and each (cid:107) x i (cid:107) 2 = 1 .

Original: Therefore, x i ∈ S is deﬁned in Lemma 2.
Revised: Therefore, x i ∈ S is deﬁned in Lemma 2.

Original: By the assumption that C (0) does not contain two linearly dependent rows, we know so are K (1) and K (2) .
Revised: By the assumption that C (0) does not contain two linearly dependent rows, we know so are K (1) and K (2) .

Original: Thus, no two l i ’s are linearly dependent and hence x i ’s are pairwise distinct.
Revised: Thus, no two l i ’s are linearly dependent and hence x i ’s are pairwise distinct.

Original: Using the universality of the arc-cosine kernel (Lemma 2), we know C (2) =
Revised: Using the universality of the arc-cosine kernel (Lemma 2), we know C (2) =

Original: g ( K (2) ) is positive deﬁnite.
Revised: g ( K (2) ) is positive deﬁnite.

Original: Hence, K (3) is positive deﬁnite.
Revised: Hence, K (3) is positive deﬁnite.

Original: An induction argument on the layer index completes the proof.
Revised: An induction argument on the layer index completes the proof.

Original: A. P ROOF OF T HEOREM
Revised: B. P ROOF OF T HEOREM

Original: For a covariance matrix K , denote its standard deviation and correlation by
Revised: For a covariance matrix K , denote its standard deviation and correlation by

Original: [Equation]
Revised: [Equation]

Original: From the closed-form formula (5) for the arc-cosine kernel, we have
Revised: From the closed-form formula (5) for the arc-cosine kernel, we have

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the correlation mapping from ρ K ( l ) to ρ C ( l ) :
Revised: We deﬁne the correlation mapping from ρ K ( l ) to ρ C ( l ) :

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst establish a few properties of f .
Revised: We ﬁrst establish a few properties of f .

Original: A pictorial illustration of f is given in Figure 5.
Revised: A pictorial illustration of f is given in Figure 5.

Original: Lemma 5.
Revised: Lemma 5.

Original: The following facts hold.
Revised: The following facts hold.

Original: f is increasing and it is a contraction mapping on [ − 1 , 1] .
Revised: f is increasing and it is a contraction mapping on [ − 1 , 1] .

Original: f ( ρ ) ≥ ρand
Revised: f ( ρ ) ≥ ρ

Original:  the equality holds only when ρ = 1 .
Revised: and the equality holds only when ρ = 1 .

Original: Proof.
Revised: Proof.

Original: For part 1, denote ρ = cos θ ∈ [ − 1 , 1] .
Revised: For part 1, denote ρ = cos θ ∈ [ − 1 , 1] .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Thus, f is increasing and it is a contraction mapping on [ − 1 , 1] .
Revised: Thus, f is increasing and it is a contraction mapping on [ − 1 , 1] .

Original: For part 2, we know that f (1) = 1 .
Revised: For part 2, we know that f (1) = 1 .

Original: Using the contraction mapping, we know f ( ρ ) ≥ ρ for ρ ∈ [ − 1 , 1] .
Revised: Using the contraction mapping, we know f ( ρ ) ≥ ρ for ρ ∈ [ − 1 , 1] .

Original: Clearly, the equality holds only when ρ = 1 .
Revised: Clearly, the equality holds only when ρ = 1 .

Original: After the correlation mapping f , we further consider the covariance mapping g : K ( l ) (cid:55)→ C ( l ) deﬁned in (3) (and equivalently in (8)).
Revised: After the correlation mapping f , we further consider the covariance mapping g : K ( l ) (cid:55)→ C ( l ) deﬁned in (3) (and equivalently in (8)).

Original: We establish a few properties of g below.
Revised: We establish a few properties of g below.

Original: Lemma 6.
Revised: Lemma 6.

Original: For positive semi-deﬁnite matrices B , the following facts hold.
Revised: For positive semi-deﬁnite matrices B , the following facts hold.

Original: g ( B ) =
Revised: g ( B ) =

Original: 12 D 1 / 2 f
Revised: 12 D 1 / 2 f

Original: ( D − 1 / 2 BD − 1 / 2 ) D 1 / 2 , where D = diag { B } ∈ R N × N and f is elementwise.
Revised: ( D − 1 / 2 BD − 1 / 2 ) D 1 / 2 , where D = diag { B } ∈ R N × N and f is elementwise.

Original: g ( B ) (cid:31) 12 B and diag { g ( B ) } = 12 diag { B } , where (cid:31) means elementwise greater. 3.
Revised: g ( B ) (cid:31) 12 B and diag { g ( B ) } = 12 diag { B } , where (cid:31) means elementwise greater. 3.

Original: tr (cid:0) Ag ( B ) A T (cid:1) ≤ 12 (cid:107) A (cid:107) 22 tr ( B ) , where (cid:107) A (cid:107) 2 is the spectral norm of matrix A .
Revised: tr (cid:0) Ag ( B ) A T (cid:1) ≤ 12 (cid:107) A (cid:107) 22 tr ( B ) , where (cid:107) A (cid:107) 2 is the spectral norm of matrix A .

Original: Proof.
Revised: Proof.

Original: For part 1, it is obvious according to (5).
Revised: For part 1, it is obvious according to (5).

Original: For part 2, using the properties of f we straightforwardly obtain
Revised: For part 2, using the properties of f we straightforwardly obtain

Original: [Equation]
Revised: [Equation]

Original: For part 3, note that g ( B ) is positive semi-deﬁnite.
Revised: For part 3, note that g ( B ) is positive semi-deﬁnite.

Original: Using von Neumann’s trace inequality,
Revised: Using von Neumann’s trace inequality,

Original: [Equation]
Revised: [Equation]

Original: where the last term is equal to (cid:107) A (cid:107) 22 tr ( g ( B ))
Revised: where the last term is equal to (cid:107) A (cid:107) 22 tr ( g ( B ))

Original: =
Revised: =

Original: 12 (cid:107) A (cid:107) 22 tr ( B ) .
Revised: 12 (cid:107) A (cid:107) 22 tr ( B ) .

Original: Now, we are ready to prove Theorem
Revised: Now, we are ready to prove Theorem

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 4, part 1.
Revised: Proof of Theorem 4, part 1.

Original: We focus on uniform convergence over all pairwise correlations.
Revised: We focus on uniform convergence over all pairwise correlations.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: To simplify notation [Equation] we temporarily let C =
Revised: To simplify notation, we temporarily let C =

Original: C
Revised: C

Original: ( l ) .
Revised: ( l ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:33) .
Revised: (cid:33) .

Original: Meanwhile,
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Original: and similarly
Revised: and similarly

Original: [Equation]
Revised: [Equation]

Original: Combining the above inequalities, we obtain
Revised: Combining the above inequalities, we obtain

Original: [Equation]
Revised: [Equation]

Original: and thusρ
Revised: and thus

Original: [Equation]

Original: K

Original: [Equation]
Revised: [Equation]

Original: From the correlation mapping, we know
Revised: From the correlation mapping, we know

Original: [Equation]
Revised: [Equation]

Original: Therefore, by the properties of f ,ρ min
Revised: Therefore, by the properties of f ,

Original: [Equation]
Revised: [Equation]

Original: Thus, ρ min ( K ( l ) ) (cid:37) some ρ ∞ ∈ [ − 1 , 1] .
Revised: Thus, ρ min ( K ( l ) ) (cid:37) some ρ ∞ ∈ [ − 1 , 1] .

Original: Because ρ ∞ = f ( ρ ∞ ) , we conclude that ρ ∞ = 1 .
Revised: Because ρ ∞ = f ( ρ ∞ ) , we conclude that ρ ∞ = 1 .

Original: Proof of Theorem 4, part 2. Consider each h : K ( l ) (cid:55)→ K ( l +1) :
Revised: Proof of Theorem 4, part 2. Consider each h : K ( l ) (cid:55)→ K ( l +1) :

Original: [Equation]
Revised: [Equation]

Original: Let δ = 12 σ 2 w λ 2 < 1 .
Revised: Let δ = 12 σ 2 w λ 2 < 1 .

Original: Using part 3 of Lemma 6,
Revised: Using part 3 of Lemma 6,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for sufﬁciently large l ,
Revised: Therefore, for sufﬁciently large l ,

Original: [Equation]
Revised: [Equation]

Original: Because each K ( l ) is positive semi-deﬁnite, we know K ( l ) is bounded.
Revised: Because each K ( l ) is positive semi-deﬁnite, we know K ( l ) is bounded.

Original: Thus, a subsequence of
Revised: Thus, a subsequence of

Original: K ( l ) → some matrix.
Revised: K ( l ) → some matrix.

Original: Proof of Theorem
Revised: Proof of Theorem

Original: [Equation]
Revised: [Equation]

Original: part
Revised: part

Original: [Equation]
Revised: [Equation]

Original: Deﬁne κ ( l ) :=
Revised: Deﬁne κ ( l ) :=

Original: K ( l ) /c l .
Revised: K ( l ) /c l .

Original: We have the following recursive relationship between κ ( l ) and κ ( l +1) :
Revised: We have the following recursive relationship between κ ( l ) and κ ( l +1) :

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where the inequality results from part 3 of Lemma 8.
Revised: where the inequality results from part 3 of Lemma 8.

Original: Therefore, κ ( l ) is bounded.
Revised: Therefore, κ ( l ) is bounded.

Original: Using the Perron-Frobenius theorem, we know the eigenspace V corresponding to λ is onedimensional, and there exists v ∈ V such that (cid:107) v (cid:107) 2 = 1 and v (cid:31) 0 .
Revised: Using the Perron-Frobenius theorem, we know the eigenspace V corresponding to λ is onedimensional, and there exists v ∈ V such that (cid:107) v (cid:107) 2 = 1 and v (cid:31) 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Therefore, v T κ ( l ) v has a limit c > 0 .
Revised: Therefore, v T κ ( l ) v has a limit c > 0 .

Original: Let P =
Revised: Let P =

Original: Proj V ⊥
Revised: Proj V ⊥

Original: =
Revised: =

Original: I − Proj V
Revised: I − Proj V

Original: = I − vv T .
Revised: = I − vv T .

Original: Then PA = AP , (cid:107) PA (cid:107) 2 = λ (cid:48) <
Revised: Then PA = AP , (cid:107) PA (cid:107) 2 = λ (cid:48) <

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: and tr
Revised: and tr

Original: [Equation]
Revised: [Equation]

Original: Pg

Original: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Thus, Pκ ( l ) P T → 0 .
Revised: Thus, Pκ ( l ) P T → 0 .

Original: Combining the results, we have κ ( l ) → cvv T .
Revised: Combining the results, we have κ ( l ) → cvv T .

Original: A. R ESULTS FOR M ULTI -L AYER P ERCEPTRONS
Revised: B. R ESULTS FOR M ULTI -L AYER P ERCEPTRONS

Original: Theorem 4 has parallel results for fully connected layers.
Revised: Theorem 4 has parallel results for fully connected layers.

Original: Theorem 7.
Revised: Theorem 7.

Original: Let A be the identity matrix.
Revised: Let A be the identity matrix.

Original: The following results hold as l → ∞ .
Revised: The following results hold as l → ∞ .

Original: When σ 2 w < 2 and σ  b > 
Revised: When σ 2 w < 2 and σ 2 b > 0 ,

Original: , then K ( l )
Revised: then K ( l )

Original: →
Revised: →

Original: q
Revised: q

Original: 1 N × N where q = σ 2 b 1 − σ  w / 2 .
Revised: N × N where q = σ 2 b 1 − σ 2 w / 2 .

Original: When σ 2 w > 2 , let c l = ( σ 2 w / 2) l ; then K ( l)
Revised: When σ 2 w > 2 , let c l = ( σ 2 w / 2) l ; then K ( l

Original: c l →vv T
Revised: ) /c l →

Original:  where v x = (cid:20)
Revised: vv T where v x = (cid:20)

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem [Equation] part 1.
Revised: Proof of Theorem 7, part 1.

Original: Starting with the diagonal elements [Equation] we have
Revised: Starting with the diagonal elements, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore [Equation] K ( l ) ( x, x ) → q =
Revised: Therefore, K ( l ) ( x, x ) → q =

Original: [Equation]
Revised: [Equation]

Original: For the off-diagonal elements [Equation] we have
Revised: For the off-diagonal elements, we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Original: Thus, K ( l ) ( x, x (cid:48) )
Revised: Thus, K ( l ) ( x, x (cid:48) )

Original: → q .
Revised: → q .

Original: In conclustion, K ( l ) → q 1 N × N .
Revised: In conclustion, K ( l ) → q 1 N × N .

Original: Proof of Theorem 7, part 2.
Revised: Proof of Theorem 7, part 2.

Original: Again, we start with diagonal elements.
Revised: Again, we start with diagonal elements.

Original: Deﬁne κ ( l ) :
Revised: Deﬁne κ ( l ) :

Original: = K ( l ) /c
Revised: = K ( l ) /c

Original: l .
Revised: l .

Original: When σ 2 w > 2 , we have
Revised: When σ 2 w > 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: For the off-diagonal elements, we have
Revised: For the off-diagonal elements [Equation] we have

Original: [Equation]
Revised: [Equation]

Original: Thus, κ ( l ) ( x, x (cid:48) ) has a limit c .
Revised: Thus, κ ( l ) ( x, x (cid:48) ) has a limit c .

Original: Taking limits on both sides, we have
Revised: Taking limits on both sides, we have

Original: [Equation]
Revised: [Equation]

Original: which suggests that c
Revised: which suggests that c

Original: = v x v x (cid:48) .
Revised: = v x v x (cid:48) .

Original: In conclustion, κ ( l ) → vv T .
Revised: In conclustion, κ ( l ) → vv T .

Original: B C OMPOSING G RAPH SAGE
Revised: C C OMPOSING G RAPH SAGE

Original: As before, X denotes pre-activation rather post activation.
Revised: As before, X denotes pre-activation rather post activation.

Original: A GraphSAGE layer has two weight matrices, W 1 and W 2 .
Revised: A GraphSAGE layer has two weight matrices, W 1 and W 2 .

Original: Hence, we use two variance parameters σ w 1 and σ w 2 for them, respectively.
Revised: Hence, we use two variance parameters σ w 1 and σ w 2 for them, respectively.

Original: For simplicity, we focus on the mean-aggregation version of GraphSAGE, where A is effectively the random-walk matrix.
Revised: For simplicity, we focus on the mean-aggregation version of GraphSAGE, where A is effectively the random-walk matrix.

Original: The updates are:
Revised: The updates are:

Original: GraphSAGE :
Revised: GraphSAGE :

Original: [Equation]
Revised: [Equation]

Revised: σ 2 w 1 g ( K ) + σ 2 w 2 Ag ( K ) A T

Revised: Q ← [ σ w 1 P σ w 2 AP ]

Original: ] where P = Chol( g ( QQ T )) .
Revised: where P = Chol( g ( QQ T )) .

Original: Note that we do not use a bias term.
Revised: Note that we do not use a bias term.

Original: If it is desired, one can easily add it to the updates following the example of GCN.
Revised: If it is desired, one can easily add it to the updates following the example of GCN.

Original: C E XPERIMENT D ETAILS
Revised: D E XPERIMENT D ETAILS

Original: Datasets.
Revised: Datasets.

Original:  The datasets Cora/Citeseer/PubMed/Reddit, with predeﬁned training/validation/test splits, are downloaded from the PyTorch Geometric library (Fey & Lenssen, 2019) and used as is.
Revised: A summary is given in Table 5. The datasets Cora/Citeseer/PubMed/Reddit, with predeﬁned training/validation/test splits, are downloaded from the PyTorch Geometric library (Fey & Lenssen, 2019) and used as is.

Original: The dataset ArXiv comes from the Open Graph Benchmark (Hu et al., 2020b).
Revised: The dataset ArXiv comes from the Open Graph Benchmark (Hu et al., 2020b).

Original: The datasets Chameleon/Squirrel/Crocodile come from MUSAE (Rozemberczki et al., 2021).
Revised: The datasets Chameleon/Squirrel/Crocodile come from MUSAE (Rozemberczki et al., 2021).

Original: The training/validation/test splits of the former two sets of datasets come from Geom-GCN (Pei et al., 2020), in accordance with the PyTorch Geometric library.
Revised: The training/validation/test splits of the former two sets of datasets come from GeomGCN (Pei et al., 2020), in accordance with the PyTorch Geometric library.

Original: The split for Crocodile  is not available, so we conduct a random split with the same 0.48/0.32/0.20 proportion as that used for Chameleon and Squirrel (Rozemberczki et al., 2021).
Revised: The split for Crocodile [Table] is not available, so we conduct a random split with the same 0.48/0.32/0.20 proportion as that used for Chameleon and Squirrel (Rozemberczki et al., 2021).

Original: Dataset preprocessing.
Revised: Dataset preprocessing.

Original: For each graph, we treat the edges as undirected and construct a binary, symmetric adjacent matrix A .
Revised: For each graph, we treat the edges as undirected and construct a binary, symmetric adjacent matrix A .

Original: Then, we apply the normalization proposed by Kipf & Welling (2017) to deﬁne A used in (1).
Revised: Then, we apply the normalization proposed by Kipf & Welling (2017) to deﬁne A used in (1).

Original: Speciﬁcally, A
Revised: Speciﬁcally, A

Original: = ( I N + D ) − 1 / 2 ( I + A )
Revised: = ( I N + D ) − 1 / 2 ( I + A )

Original: ( I N + D ) − 1 / 2 , where D = diag { (cid:80)
Revised: ( I N + D ) − 1 / 2 , where D = diag { (cid:80)

Original: j A ij } .
Revised: j A ij } .

Original: For GraphSAGE and GGP, we instead use the row-normalized version A
Revised: For GraphSAGE and GGP, we instead use the row-normalized version A

Original: = ( I N +
Revised: = ( I N +

Original: D ) − 1 ( I N + A ) .
Revised: D ) − 1 ( I N + A ) .

Original: No feature preprocessing is done except running the GPs on ArXiv, for which we apply a centering on the input features.
Revised: No feature preprocessing is done except running the GPs on ArXiv, for which we apply a centering on the input features.

Original: Environment.
Revised: Environment.

Original: All experiments are conducted on a Nvidia Quadro
Revised: All experiments are conducted on a Nvidia Quadro

Original: GV

Revised: GV

Original: GPU with 32GB of
Revised: GPU with 32GB of

Original: HBM2 memory.
Revised: HBM2 memory.

Original: The code is written in Python 3.10.4 as distributed 
Revised: The code is written in Python 3.10.4 as distributed with Ubuntu 22.04 LTS.

Original: We 
Revised: We use

Revised: PyTorch 1.11.0 and PyTorch Geometric 2.1.0 with CUDA 11.3.

Original: Training.
Revised: Training.

Original: For all datasets except Reddit, we train GCN using the Adam optimizer in full batch for 100 epochs.
Revised: For all datasets except Reddit, we train GCN using the Adam optimizer in full batch for 100 epochs.

Original: Reddit is too large for GPU computation and hence we conduct mini-batch training (with a batch size 10240) for 10 epochs.
Revised: Reddit is too large for GPU computation and hence we conduct mini-batch training (with a batch size 10240) for 10 epochs.

Original: Hyperparameters.
Revised: Hyperparameters.

Original: For classiﬁcation tasks in Table 4, the hyperparameters are set to σ b = 0 . 0 , σ w = 1 . 0 , L = 2 , hidden = 256 , and dropput = 0 .
Revised: For classiﬁcation tasks in Table 3, the hyperparameters are set to σ b = 0 . 0 , σ w = 1 . 0 , L = 2 , hidden = 256 , and dropput = 0 .

Original: GCN is trained with learning rate 0 .
Revised: GCN is trained with learning rate 0 .

Original: For regression tasks, they are set to σ b = √ 0 .
Revised: For regression tasks, they are set to σ b = √ 0 .

Original: 1 , σ w = 1 . 0 , L = 2 , hidden = 256 , and dropput = 0 .
Revised: 1 , σ w = 1 . 0 , L = 2 , hidden = 256 , and dropput = 0 .

Original: GCN is trained with learning rate √ 0 .
Revised: GCN is trained with learning rate √ 0 .

Original: We choose C (0) to be the inner product kernel.
Revised: We choose C (0) to be the inner product kernel.

Original: The exception is when working with the low-rank versions, we apply PCA on the input features to ensure the number of features is smaller than the number of landmark nodes.
Revised: The exception is when working with the low-rank versions, we apply PCA on the input features to ensure the number of features is smaller than the number of landmark nodes.

Original: For Table 5: GCNII uses a 2-layer architecture and sets α = 0 .
Revised: For Table 4: GCNII uses a 2-layer architecture and sets α = 0 .

Original: 1 , λ = 0 .
Revised: 1 , λ = 0 .

Original: 5 , βl = log( λl +1) .
Revised: 5 , β = log( λl +

Original: GIN uses L = 2 and(cid:15) =0 . 0 .
Revised: GIN uses L = 2 and =

Original: GraphSAGE uses a 2-layer architecture and sets σ w 1 = √ 0 .
Revised: GraphSAGE uses layer

Original: 1 for Reddit and σ [-w 1 = 0 . 
-]
Revised: Reddit and σ [-w 1 = 0 . 
-]for other and σ

Original: for other datasets, and σ w 2 = 1 . 0 .

Original: The nugget (cid:15) is chosen using a grid search over [10 − 3 , 10 1 ] .
Revised: The nugget (cid:15) is chosen using a grid search over [10 − 3 , 10 1 ] .

Original: For GCNGP-X, GNNGP-X and RBF-X, the landmark points are chosen to be the training set for small datasets, while for large datasets (ArXiv and Reddit)
Revised: For GCNGP-X, GNNGP-X and RBF-X, the landmark points are chosen to be the training set for small datasets, while for large datasets (ArXiv and Reddit)

Original: the landmark points are a random 1/ of the training set.
Revised: the landmark points are a random 1/ of the training set.

Original: Kernels.
Revised: Kernels.

Original: For the RBF kernel,
Revised: For the RBF kernel,

Original: [Equation]
Revised: [Equation]

Original: where γ is chosen using a grid search over [10 − 2 , 10 2 ] .
Revised: where γ is chosen using a grid search over [10 − 2 , 10 2 ] .

Original: For the GGP kernel,
Revised: For the GGP kernel,

Original: [Equation]
Revised: [Equation]

Original: where c = 5 . 0 , d = 3 . 0 .
Revised: where c = 5 . 0 , d = 3 . 0 .

Original: Note that we use only the kernel but not the robust-max likelihood nor the
Revised: Note that we use only the kernel but not the robust-max likelihood nor the

Original: ELBO training proposed by Ng et al.
Revised: ELBO training proposed by Ng et al.

Original: (2018), because their code written in Python 2 is outdated and cannot be used.
Revised: (2018), because their code written in Python 2 is outdated and cannot be used.

Original: Note also that GGP-X in Ng et al.
Revised: Note also that GGP-X in Ng et al.

Original: (2018) has a different meaning—training GGP by using additionally the 500 nodes in the validation set—while we use GGP-X to denote the Nystr¨om approximation of GGP.
Revised: (2018) has a different meaning—training GGP by using additionally the 500 nodes in the validation set—while we use GGP-X to denote the Nystr¨om approximation of GGP.