Original: G RAPH N EURAL N ETWORK -I NSPIRED K ERNELS FOR G AUSSIAN P ROCESSES IN S EMI -S UPERVISED L EARNING
Revised: G RAPH N EURAL N ETWORK -I NSPIRED K ERNELS FOR G AUSSIAN P ROCESSES IN S EMI -S UPERVISED L EARNING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Gaussian processes (GPs) are an attractive class of machine learning models because of their simplicity and ﬂexibility as building blocks of more complex
Revised: Gaussian processes (GPs) are an attractive class of machine learning models because of their simplicity and ﬂexibility as building blocks of more complex

Original: Bayesian models.
Revised: Bayesian models.

Original: Meanwhile, graph neural networks (GNNs) emerged recently as a promising class of models for graph-structured data in semi-supervised learning and beyond.
Revised: Meanwhile, graph neural networks (GNNs) emerged recently as a promising class of models for graph-structured data in semi-supervised learning and beyond.

Original: Their competitive performance is often attributed to a proper capturing of the graph inductive bias.
Revised: Their competitive performance is often attributed to a proper capturing of the graph inductive bias.

Original: In this work, we introduce this inductive bias into GPs to improve their predictive performance for graph-structured data.
Revised: In this work, we introduce this inductive bias into GPs to improve their predictive performance for graph-structured data.

Original: We show that a prominent example of GNNs, the graph convolutional network, is equivalent to some GP when its layers are inﬁnitely wide; and we analyze the kernel universality and the limiting behavior in depth.
Revised: We show that a prominent example of GNNs, the graph convolutional network, is equivalent to some GP when its layers are inﬁnitely wide; and we analyze the kernel universality and the limiting behavior in depth.

Original: We further present a programmable procedure to compose covariance kernels inspired by this equivalence and derive example kernels corresponding to several interesting members of the GNN family.
Revised: We further present a programmable procedure to compose covariance kernels inspired by this equivalence and derive example kernels corresponding to several interesting members of the GNN family.

Original: We also propose a computationally efﬁcient approximation of the covariance matrix for scalable posterior inference with large-scale data.
Revised: We also propose a computationally efﬁcient approximation of the covariance matrix for scalable posterior inference with large-scale data.

Original: We demonstrate that these graph-based kernels lead to competitive classiﬁcation and regression performance, as well as advantages in computation time, compared with the respective GNNs.
Revised: We demonstrate that these graph-based kernels lead to competitive classiﬁcation and regression performance, as well as advantages in computation time, compared with the respective GNNs.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Gaussian processes (GPs) (Rasmussen & Williams, 2006) are widely used in machine learning, uncertainty quantiﬁcation, and global optimization.
Revised: Gaussian processes (GPs) (Rasmussen & Williams, 2006) are widely used in machine learning, uncertainty quantiﬁcation, and global optimization.

Original: In the Bayesian setting, a GP serves as a prior probability distribution over functions, characterized by a mean (often treated as zero for simplicity) and a covariance.
Revised: In the Bayesian setting, a GP serves as a prior probability distribution over functions, characterized by a mean (often treated as zero for simplicity) and a covariance.

Original: Conditioned on observed data with a Gaussian likelihood, the random function admits a posterior distribution that is also Gaussian, whose mean is used for prediction and the variance serves as an uncertainty measure.
Revised: Conditioned on observed data with a Gaussian likelihood, the random function admits a posterior distribution that is also Gaussian, whose mean is used for prediction and the variance serves as an uncertainty measure.

Original: The closed-form posterior allows for exact Bayesian inference, resulting in great attractiveness and wide usage of GPs.
Revised: The closed-form posterior allows for exact Bayesian inference, resulting in great attractiveness and wide usage of GPs.

Original: The success of GPs in practice depends on two factors: the observations (training data) and the covariance kernel.
Revised: The success of GPs in practice depends on two factors: the observations (training data) and the covariance kernel.

Original: We are interested in semi-supervised learning, where only a small amount of data is labeled while a large amount of unlabeled data can be used together for training (Zhu, 2008).
Revised: We are interested in semi-supervised learning, where only a small amount of data is labeled while a large amount of unlabeled data can be used together for training (Zhu, 2008).

Original: In recent years, graph neural networks (GNNs)
Revised: In recent years, graph neural networks (GNNs)

Original: (Zhou et al., 2020; Wu et al., 2021) emerged as a promising class of models for this problem, when the labeled and unlabeled data are connected by a graph.
Revised: (Zhou et al., 2020; Wu et al., 2021) emerged as a promising class of models for this problem, when the labeled and unlabeled data are connected by a graph.

Original: The graph structure becomes an important inductive bias that leads to the success of GNNs.
Revised: The graph structure becomes an important inductive bias that leads to the success of GNNs.

Original: This inductive bias inspires us to design a GP model under limited observations, by building the graph structure into the covariance kernel.
Revised: This inductive bias inspires us to design a GP model under limited observations, by building the graph structure into the covariance kernel.

Original: An intimate relationship between neural networks and GPs is known: a neural network with fully connected layers, equipped with a prior probability distribution on the weights and biases, converges to a GP when each of its layers is inﬁnitely wide (Lee et al., 2018; de G. Matthews et al., 2018).
Revised: An intimate relationship between neural networks and GPs is known: a neural network with fully connected layers, equipped with a prior probability distribution on the weights and biases, converges to a GP when each of its layers is inﬁnitely wide (Lee et al., 2018; de G. Matthews et al., 2018).

Original: Such a result is owing to the central limit theorem (Neal, 1994; Williams, 1996) and the GP covariance can be recursively computed if the weights (and biases) in each layer are iid Gaussian.
Revised: Such a result is owing to the central limit theorem (Neal, 1994; Williams, 1996) and the GP covariance can be recursively computed if the weights (and biases) in each layer are iid Gaussian.

Original: Similar results for other architectures, such as convolution layers and residual connections, were subsequently established in the literature (Novak et al., 2019; Garriga-Alonso et al., 2019).
Revised: Similar results for other architectures, such as convolution layers and residual connections, were subsequently established in the literature (Novak et al., 2019; Garriga-Alonso et al., 2019).

Original: One focus of this work is to establish a similar relationship between GNNs and the limiting GPs.
Revised: One focus of this work is to establish a similar relationship between GNNs and the limiting GPs.

Original: We will derive the covariance kernel that incorporates the graph inductive bias as GNNs do.
Revised: We will derive the covariance kernel that incorporates the graph inductive bias as GNNs do.

Original: We start with one of the most widely studied GNNs, the graph convolutional network (GCN) (Kipf &
Revised: We start with one of the most widely studied GNNs, the graph convolutional network (GCN) (Kipf &

Original: Welling, 2017), and analyze the kernel universality as well as the limiting behavior when the depth also tends to inﬁnity.
Revised: Welling, 2017), and analyze the kernel universality as well as the limiting behavior when the depth also tends to inﬁnity.

Original: We then derive covariance kernels from other GNNs by using a programmable procedure that corresponds every building block of a neural network to a kernel operation.
Revised: We then derive covariance kernels from other GNNs by using a programmable procedure that corresponds every building block of a neural network to a kernel operation.

Original: Meanwhile, we design efﬁcient computational procedures for posterior inference (i.e., regression and classiﬁcation).
Revised: Meanwhile, we design efﬁcient computational procedures for posterior inference (i.e., regression and classiﬁcation).

Original: GPs are notoriously difﬁcult to scale because of the cubic complexity with respect to the number of training data.
Revised: GPs are notoriously difﬁcult to scale because of the cubic complexity with respect to the number of training data.

Original: Benchmark graph datasets used by the GNN literature may contain thousands or even millions of labeled nodes (Hu et al., 2020).
Revised: Benchmark graph datasets used by the GNN literature may contain thousands or even millions of labeled nodes (Hu et al., 2020b).

Original: The semi-supervised setting worsens the scenario, as the covariance matrix needs to be (recursively) evaluated in full because of the graph convolution operation.
Revised: The semi-supervised setting worsens the scenario, as the covariance matrix needs to be (recursively) evaluated in full because of the graph convolution operation.

Original: We propose a Nystr¨om-like scheme to perform low-rank approximations and apply the approximation recursively on each layer, to yield a low-rank kernel matrix.
Revised: We propose a Nystr¨om-like scheme to perform low-rank approximations and apply the approximation recursively on each layer, to yield a low-rank kernel matrix.

Original: Such a matrix can be computed scalably.
Revised: Such a matrix can be computed scalably.

Original: We demonstrate through numerical experiments that
Revised: We demonstrate through numerical experiments that

Original: the
Revised: the

Original: GP posterior inference is much faster than training a GNN and subsequently performing predictions on the test set.
Revised: GP posterior inference is much faster than training a GNN and subsequently performing predictions on the test set.

Original: We summarize the contributions of this work as follows:
Revised: We summarize the contributions of this work as follows:

Original: We derive the GP as a limit of the GCN when the layer widths tend to inﬁnity and study the kernel universality and the limiting behavior in depth.
Revised: We derive the GP as a limit of the GCN when the layer widths tend to inﬁnity and study the kernel universality and the limiting behavior in depth.

Original: We propose a computational procedure to compute a low-rank approximation of the covariance matrix for practical and scalable posterior inference.
Revised: We propose a computational procedure to compute a low-rank approximation of the covariance matrix for practical and scalable posterior inference.

Original: We present a programmable procedure to compose covariance kernels and their approximations and show examples corresponding to several interesting members of the GNN family.
Revised: We present a programmable procedure to compose covariance kernels and their approximations and show examples corresponding to several interesting members of the GNN family.

Original: We conduct comprehensive experiments to demonstrate that the GP model performs favorably compared to GNNs in prediction accuracy while being signiﬁcantly faster in computation
Revised: We conduct comprehensive experiments to demonstrate that the GP model performs favorably compared to GNNs in prediction accuracy while being signiﬁcantly faster in computation.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: It has long been observed that GPs are limits of standard neural networks with one hidden layer when the layer width tends to inﬁnity (Neal, 1994; Williams, 1996).
Revised: It has long been observed that GPs are limits of standard neural networks with one hidden layer when the layer width tends to inﬁnity (Neal, 1994; Williams, 1996).

Original: Recently, renewed interests in the equivalence between GPs and neural networks were extended to deep neural networks (Lee et al., 2018; de G. Matthews et al., 2018) as well as modern neural network architectures, such as convolution layers (Novak et al., 2019) and residual connections (Garriga-Alonso et al., 2019).
Revised: Recently, renewed interests in the equivalence between GPs and neural networks were extended to deep neural networks (Lee et al., 2018; de G. Matthews et al., 2018) as well as modern neural network architectures, such as convolution layers (Novak et al., 2019), recurrent networks (Yang, 2019), and residual connections (Garriga-Alonso et al., 2019).

Original: The term

Original:  NNGP (neural network Gaussian process) henceforth emerged under the context of Bayesian deep learning.
Revised: The term NNGP (neural network Gaussian process) henceforth emerged under the context of Bayesian deep learning.

Original: Besides the fact that an inﬁnite neural network deﬁnes a kernel, the training of a neural network by using gradient descent also deﬁnes a kernel—the neural tangent kernel (NTK)—that describes the evolution of the network (Jacot et al., 2018; Lee et al., 2019).
Revised: Besides the fact that an inﬁnite neural network deﬁnes a kernel, the training of a neural network by using gradient descent also deﬁnes a kernel—the neural tangent kernel (NTK)—that describes the evolution of the network (Jacot et al., 2018; Lee et al., 2019).

Original: Library supports in Python were developed to automatically construct the NNGP and NTK kernels based on programming the corresponding neural networks (Novak et al., 2020).
Revised: Library supports in Python were developed to automatically construct the NNGP and NTK kernels based on programming the corresponding neural networks (Novak et al., 2020).

Original: GNNs are neural networks that handle graph-structured data (Zhou et al., 2020; Wu et al., 2021).
Revised: GNNs are neural networks that handle graph-structured data (Zhou et al., 2020; Wu et al., 2021).

Original: They are a promising class of models for semi-supervised learning.
Revised: They are a promising class of models for semi-supervised learning.

Original: Many GNNs use the messagepassing scheme (Gilmer et al., 2017), where neighborhood information is aggregated to update the representation of the center node.
Revised: Many GNNs use the messagepassing scheme (Gilmer et al., 2017), where neighborhood information is aggregated to update the representation of the center node.

Original: Representative examples include GCN (Kipf & Welling, 2017),
Revised: Representative examples include GCN (Kipf & Welling, 2017),

Original: GraphSAGE (Hamilton et al., 2017), GAT (Veliˇckovi´c et al., 2018), and GIN (Xu et al., 2019).
Revised: GraphSAGE (Hamilton et al., 2017), GAT (Veliˇckovi´c et al., 2018), and GIN (Xu et al., 2019).

Original: It is found that the performance of GNNs degrades as they become deep; one approach to mitigating the problem is to insert residual/skip connections, as done by JumpingKnowledge (Xu et al., 2018),
Revised: It is found that the performance of GNNs degrades as they become deep; one approach to mitigating the problem is to insert residual/skip connections, as done by JumpingKnowledge (Xu et al., 2018),

Original: APPNP (Gasteiger et al., 2019), and GCNII (Chen et al., 2020).
Revised: APPNP (Gasteiger et al., 2019), and GCNII (Chen et al., 2020).

Original: GP inference is notoriously expensive because it requires the inverse of the N × N dense kernel matrix.
Revised: GP inference is too costly, because it requires the inverse of the N × N dense kernel matrix.

Original: Scalable approaches include low-rank methods, such as Nystr¨om approximation (Drineas & Mahoney, 2005), random features (Rahimi & Recht, 2007), and KISS-GP (Wilson & Nickisch, 2015); as well as multi-resolution (Katzfuss, 2017) and hierarchical methods (Chen et al., 2017; 
Revised: Scalable approaches include low-rank methods, such as Nystr ¨ om approximation (Drineas & Mahoney, 2005), random features (Rahimi & Recht, 2007), and KISS-GP (Wilson & Nickisch, 2015); as well as multi-resolution (Katzfuss, 2017) and hierarchical methods (Chen et al., 2017; Chen & Stein, 2021).

Revised: Prior efforts on integrating graphs into GPs exist.

Original: Chen & Stein, 2021).
Revised: Ng et al. (2018) deﬁne a GP kernel by combing a base kernel with the adjacency matrix; it is related to a special case of our kernels where the network has only one layer and the output admits a robust-max likelihood for classiﬁcation. Hu et al. (2020a) explore a similar route to us, by taking the limit of a GCN, but its exploration is less comprehensive because it does not generalize to other GNNs and does not tackle the scalability challenge.

Original:  G RAPH C ONVOLUTIONAL N ETWORK AS A G AUSSIAN P ROCESS
Revised:  G RAPH C ONVOLUTIONAL N ETWORK AS A G AUSSIAN P ROCESS

Original: We start with a few notations used throughout this paper.
Revised: We start with a few notations used throughout this paper.

Original: Let an undirected graph be denoted by
Revised: Let an undirected graph be denoted by

Original: G =
Revised: G =

Original: ( V , E ) with N = |V| nodes and M = |E| edges.
Revised: ( V , E ) with N = |V| nodes and M = |E| edges.

Original: For notational simplicity, we use A ∈ R N × N to denote the original graph adjacency matrix or any modiﬁed/normalized version of it.
Revised: For notational simplicity, we use A ∈ R N × N to denote the original graph adjacency matrix or any modiﬁed/normalized version of it.

Original: Using d l to denote the width of the l -th layer, the layer architecture of GCN reads
Revised: Using d l to denote the width of the l -th layer, the layer architecture of GCN reads

Original: [Equation]
Revised: [Equation]

Original: where X ( l − 1) ∈ R N × d l − 1 and X ( l ) ∈
Revised: where X ( l − 1) ∈ R N × d l − 1 and X ( l ) ∈

Original: R N × d l are layer inputs and outputs, respectively; W ( l ) ∈
Revised: R N × d l are layer inputs and outputs, respectively; W ( l ) ∈

Original: R d l − 1 × d l and b ( l ) ∈ R 1 × d l are the weights and the biases, respectively; and φ is the ReLU activation function.
Revised: R d l − 1 × d l and b ( l ) ∈ R 1 × d l are the weights and the biases, respectively; and φ is the ReLU activation function.

Original: The graph convolutional operator A is a symmetric normalization of the graph adjacency matrix with self-loops added (Kipf & Welling, 2017).
Revised: The graph convolutional operator A is a symmetric normalization of the graph adjacency matrix with self-loops added (Kipf & Welling, 2017).

Original: For ease of exposition, it will be useful to rewrite the matrix notation (1) in element-sums and products.
Revised: For ease of exposition, it will be useful to rewrite the matrix notation (1) in element-sums and products.

Original: To this end, for a node x , let z ( l ) i ( x ) and x ( l ) i ( x ) denote the pre- and post-activation value at the i -th coordinate in the l -th layer, respectively.
Revised: To this end, for a node x , let z ( l ) i ( x ) and x ( l ) i ( x ) denote the pre- and post-activation value at the i -th coordinate in the l -th layer, respectively.

Original: Particularly, in an L -layer GCN, x (0) ( x ) is the input feature vector and z ( L ) ( x ) is the output vector.
Revised: Particularly, in an L -layer GCN, x (0) ( x ) is the input feature vector and z ( L ) ( x ) is the output vector.

Original: The layer architecture of GCN reads
Revised: The layer architecture of GCN reads

Original: [Equation]
Revised: [Equation]

Original: 3. L IMIT IN THE W IDTH
Revised: 3. L IMIT IN THE W IDTH

Original: The following theorem states that when the weights and biases in each layer are iid zero-mean
Revised: The following theorem states that when the weights and biases in each layer are iid zero-mean

Original: Gaussians, in the limit on the layer width, the GCN output z ( L ) ( x ) is a multi-output GP over the index x .
Revised: Gaussians, in the limit on the layer width, the GCN output z ( L ) ( x ) is a multi-output GP over the index x .

Original: Theorem 1.
Revised: Theorem 1.

Original: Assume d 1 , . . .
Revised: Assume d 1 , . . .

Original: , d L − 1 to be inﬁnite in succession and let the bias and weight terms be independent with distributions
Revised: , d L − 1 to be inﬁnite in succession and let the bias and weight terms be independent with distributions

Original: [Equation]
Revised: [Equation]

Original: Then, for each i , the collection { z ( l ) i ( x ) } over all graph nodes x follows the normal distribution
Revised: Then, for each i , the collection { z ( l ) i ( x ) } over all graph nodes x follows the normal distribution

Original: N (0 , K ( l ) ) , where the covariance matrix K ( l ) can be computed recursively by
Revised: N (0 , K ( l ) ) , where the covariance matrix K ( l ) can be computed recursively by

Original: [Equation]
Revised: [Equation]

Original: All proofs of this paper are given in the appendix.
Revised: All proofs of this paper are given in the appendix.

Original: Note that different from a usual GP, which is a random function deﬁned over a connected region of the Euclidean space, here z ( L ) is deﬁned over a discrete set of graph nodes.
Revised: Note that different from a usual GP, which is a random function deﬁned over a connected region of the Euclidean space, here z ( L ) is deﬁned over a discrete set of graph nodes.

Original: In the usual use of a graph in machine learning, this set is ﬁnite, such that the function distribution degenerates to a multivariate distribution.
Revised: In the usual use of a graph in machine learning, this set is ﬁnite, such that the function distribution degenerates to a multivariate distribution.

Original: In semi-supervised learning, the dimension of the distribution, N , is ﬁxed when one conducts transductive learning; but it will vary in the inductive setting because the graph will have new nodes and edges.
Revised: In semi-supervised learning, the dimension of the distribution, N , is ﬁxed when one conducts transductive learning; but it will vary in the inductive setting because the graph will have new nodes and edges.

Original: One special care of a graph-based GP over a usual GP is that the covariance matrix will need to be recomputed from scratch whenever the graph alters.
Revised: One special care of a graph-based GP over a usual GP is that the covariance matrix will need to be recomputed from scratch whenever the graph alters.

Original: Theorem 1 leaves out the base deﬁnition C (0) , whose entry denotes the covariance between two input nodes.
Revised: Theorem 1 leaves out the base deﬁnition C (0) , whose entry denotes the covariance between two input nodes.

Original: The traditional literature uses the inner product C (0) ( x, x (cid:48) )
Revised: The traditional literature uses the inner product C (0) ( x, x (cid:48) )

Original: = x · x (cid:48)
Revised: = x · x (cid:48)

Original: d 0 (Lee et al., 2018), but nothing prevents us using any positive-deﬁnite kernel alternatively.
Revised: d 0 (Lee et al., 2018), but nothing prevents us using any positive-deﬁnite kernel alternatively.

Original: For example, we could use the squared exponential kernel C (0) ( x, x (cid:48) )
Revised: For example, we could use the squared exponential kernel C (0) ( x, x (cid:48) )

Original: =
Revised: =

Original: exp (cid:18) − 12 (cid:80) d 0 j =1 (cid:16) x
Revised: exp (cid:18) − 12 (cid:80) d 0 j =1 (cid:16) x

Original: j − x (cid:48)
Revised: j − x (cid:48)

Original: j (cid:96) j (cid:17) 2 (cid:19) .
Revised: j (cid:96) j (cid:17) 2 (cid:19) .

Original: Such ﬂexibility in essence performs an implicit feature transformation as preprocessing.
Revised: Such ﬂexibility in essence performs an implicit feature transformation as preprocessing.

Original: 3. U NIVERSALITY
Revised: 3. U NIVERSALITY

Original: A covariance kernel is positive deﬁnite; hence, the Moore–Aronszajn theorem (Aronszajn, 1950) suggests that it deﬁnes a unique Hilbert space for which it is a reproducing kernel.
Revised: A covariance kernel is positive deﬁnite; hence, the Moore–Aronszajn theorem (Aronszajn, 1950) suggests that it deﬁnes a unique Hilbert space for which it is a reproducing kernel.

Original: If this space is dense, then the kernel is called universal .
Revised: If this space is dense, then the kernel is called universal .

Original: One can verify universality by checking if the kernel matrix is positive deﬁnite for any set of distinct points.
Revised: One can verify universality by checking if the kernel matrix is positive deﬁnite for any set of distinct points.

Original: For the case of graphs, it sufﬁces to verify if the covariance matrix for all nodes is positive deﬁnite.
Revised: For the case of graphs, it sufﬁces to verify if the covariance matrix for all nodes is positive deﬁnite.

Original: We do this job for the ReLU activation function.
Revised: We do this job for the ReLU activation function.

Original: It is known that the kernel E w ∼N (0 ,I d ) [ φ ( w · x ) φ ( w · x (cid:48) )] admits a closed-form expression as a function of the angle between x and x (cid:48) , hence named the arc-cosine kernel (Cho & Saul, 2009).
Revised: It is known that the kernel E w ∼N (0 ,I d ) [ φ ( w · x ) φ ( w · x (cid:48) )] admits a closed-form expression as a function of the angle between x and x (cid:48) , hence named the arc-cosine kernel (Cho & Saul, 2009).

Original: We ﬁrst establish the following lemma that states that the kernel is universal over a half-space.
Revised: We ﬁrst establish the following lemma that states that the kernel is universal over a half-space.

Original: Lemma 2.
Revised: Lemma 2.

Original: The arc-cosine kernel is universal on the upper-hemisphere
Revised: The arc-cosine kernel is universal on the upper-hemisphere

Original: [Equation]
Revised: [Equation]

Original: It is also known that the expectation in (3) is proportional to the arc-cosine kernel up to a factor (cid:112)
Revised: It is also known that the expectation in (3) is proportional to the arc-cosine kernel up to a factor (cid:112)

Original: K ( l ) ( x, x ) K ( l ) ( x (cid:48) , x (cid:48) )
Revised: K ( l ) ( x, x ) K ( l ) ( x (cid:48) , x (cid:48) )

Original: (Lee et al., 2018).
Revised: (Lee et al., 2018).

Original: Therefore, we iteratively work on the post-activation covariance (3) and the pre-activation covariance (4) and show that the covariance kernel resulting from the limiting GCN is universal, for any GCN with three or more layers.
Revised: Therefore, we iteratively work on the post-activation covariance (3) and the pre-activation covariance (4) and show that the covariance kernel resulting from the limiting GCN is universal, for any GCN with three or more layers.

Original: Theorem 3.
Revised: Theorem 3.

Original: Assume A is irreducible and non-negative and C (0) does not contain two linearly dependent rows.
Revised: Assume A is irreducible and non-negative and C (0) does not contain two linearly dependent rows.

Original: Then, K ( l ) is positive deﬁnite for all l ≥ 3 .
Revised: Then, K ( l ) is positive deﬁnite for all l ≥ 3 .

Original: 3. L IMIT IN THE D EPTH
Revised: 3. L IMIT IN THE D EPTH

Original: The depth of a neural network exhibits interesting behaviors.
Revised: The depth of a neural network exhibits interesting behaviors.

Original: Deep learning tends to favor deep networks because of their empirically outstanding performance, exempliﬁed by generations of convolutional networks for the ImageNet classiﬁcation (Krizhevsky et al., 2012; Wortsman et al., 2022); while graph neural networks are instrumental to be shallow because of the over-smoothing and oversquashing properties (Li et al., 2018; Topping et al., 2022).
Revised: Deep learning tends to favor deep networks because of their empirically outstanding performance, exempliﬁed by generations of convolutional networks for the ImageNet classiﬁcation (Krizhevsky et al., 2012; Wortsman et al., 2022); while graph neural networks are instrumental to be shallow because of the over-smoothing and oversquashing properties (Li et al., 2018; Topping et al., 2022).

Original: For multi-layer perceptrons (networks with fully connected layers), several previous works have noted that the recurrence relation of the covariance kernel across layers leads to convergence to a ﬁxed-point kernel, when the depth L → ∞ (see, e.g., Lee et al.
Revised: For multi-layer perceptrons (networks with fully connected layers), several previous works have noted that the recurrence relation of the covariance kernel across layers leads to convergence to a ﬁxed-point kernel, when the depth L → ∞ (see, e.g., Lee et al.

Original: (2018); in Appendix A.5, we elaborate this limit).
Revised: (2018); in Appendix A.5, we elaborate this limit).

Original: In what follows, we offer the parallel analysis for GCN.
Revised: In what follows, we offer the parallel analysis for GCN.

Original: Theorem 4.
Revised: Theorem 4.

Original: Assume A is symmetric, irreducible, aperiodic, and non-negative with PerronFrobenius eigenvalue λ > 0 .
Revised: Assume A is symmetric, irreducible, aperiodic, and non-negative with PerronFrobenius eigenvalue λ > 0 .

Original: The following results hold as l → ∞ .
Revised: The following results hold as l → ∞ .

Original: When σ 2 b = 0 , ρ min ( K ( l ) ) (cid:37) 1 , where ρ min denotes the minimum correlation between any two nodes x and x (cid:48) .
Revised: When σ 2 b = 0 , ρ min ( K ( l ) ) (cid:37) 1 , where ρ min denotes the minimum correlation between any two nodes x and x (cid:48) .

Original: When σ 2 w < 2 /λ 2 , a subsequence of K ( l ) converges to some matrix.
Revised: When σ 2 w < 2 /λ 2 , a subsequence of K ( l ) converges to some matrix.

Original: When σ 2 w > 2 /λ 2 , let c l = ( σ 2 w λ 2 / 2) l ; then, K ( l ) /c
Revised: When σ 2 w > 2 /λ 2 , let c l = ( σ 2 w λ 2 / 2) l ; then, K ( l ) /c

Original: l → vv T where v is an eigenvector corresponding to λ .
Revised: l → vv T where v is an eigenvector corresponding to λ .

Original: A few remarks follow.
Revised: A few remarks follow.

Original: The ﬁrst case implies that the correlation matrix converges monotonously to a matrix of all ones.
Revised: The ﬁrst case implies that the correlation matrix converges monotonously to a matrix of all ones.

Original: As a consequence, up to some scaling c (cid:48) l that may depend on l , the scaled covariance matrix K ( l ) /c
Revised: As a consequence, up to some scaling c (cid:48) l that may depend on l , the scaled covariance matrix K ( l ) /c

Original: (cid:48)
Revised: (cid:48)

Original: l converges to a rank-1 matrix.
Revised: l converges to a rank-1 matrix.

Original: The third case shares a similar result, with the limit explicitly spelled out, but note that the eigenvector v may not be normalized.
Revised: The third case shares a similar result, with the limit explicitly spelled out, but note that the eigenvector v may not be normalized.

Original: The second case is challenging to analyze.
Revised: The second case is challenging to analyze.

Original: According to empirical veriﬁcation, we speculate a stronger result— convergence of K ( l ) to a unique ﬁxed point—may hold.
Revised: According to empirical veriﬁcation, we speculate a stronger result— convergence of K ( l ) to a unique ﬁxed point—may hold.

Original:  S CALABLE C OMPUTATION THROUGH L OW -R ANK A PPROXIMATION
Revised:  S CALABLE C OMPUTATION THROUGH L OW -R ANK A PPROXIMATION

Original: The computation of the covariance matrix K ( L ) through recursion (3)–(4) is the main computational bottleneck for GP posterior inference.
Revised: The computation of the covariance matrix K ( L ) through recursion (3)–(4) is the main computational bottleneck for GP posterior inference.

Original: We start the exposition with the mean prediction.
Revised: We start the exposition with the mean prediction.

Original: We compute
Revised: We compute

Original: Algorithm
Revised: Algorithm

Original: Require: Q (0) such that K (0)
Revised: Require: Q (0) such that K (0)

Original: ≈ Q (0) Q (0) T 1: for l = 0 , . .
Revised: ≈ Q (0) Q (0) T 1: for l = 0 , . .

Original: , L − 1 do 2:
Revised: , L − 1 do 2:

Original: Compute (cid:98) K ( l ) : a = Q (0):: Q (0) a : T 3:
Revised: Compute (cid:98) K ( l ) : a = Q (0):: Q (0) a : T 3:

Original: Compute (cid:98) C ( l ) : a by (5), where C ( l ) (resp.
Revised: Compute (cid:98) C ( l ) : a by (5), where C ( l ) (resp.

Original: K ( l ) )
Revised: K ( l ) )

Original: entries are replaced by (cid:98) C ( l ) (resp.
Revised: entries are replaced by (cid:98) C ( l ) (resp.

Original: (cid:98) K ( l ) ) entries 4:
Revised: (cid:98) K ( l ) ) entries 4:

Original: Compute Q ( l +1) = (cid:104) σ w A (cid:98) C ( l ) : a ( (cid:98) C ( l ) aa ) −σ b 1 N × 1 (cid:105) 5: end for the posterior mean (cid:98) y ∗ =
Revised: Compute Q ( l +1) = (cid:104) σ w A (cid:98) C ( l ) : a ( (cid:98) C ( l ) aa ) −σ b 1 N × 1 (cid:105) 5: end for the posterior mean (cid:98) y ∗ =

Original: K ( L ) ∗ b ( K ( L )
Revised: K ( L ) ∗ b ( K ( L )

Original: bb + (cid:15)I ) − 1 y b , where the subscripts b and ∗ denote the training set and the prediction set, respectively; and (cid:15) , called the nugget , is the noise variance of the training data.
Revised: bb + (cid:15)I ) − 1 y b , where the subscripts b and ∗ denote the training set and the prediction set, respectively; and (cid:15) , called the nugget , is the noise variance of the training data.

Original: Let there be N b training nodes and N ∗ prediction nodes.
Revised: Let there be N b training nodes and N ∗ prediction nodes.

Original: It is tempting to compute only the ( N b + N ∗ ) × N b submatrix of K ( L ) for the task, but the recursion (4) requires the full C ( L − 1) at the presence of A , and hence all the full C ( l ) ’s and K ( l ) ’
Revised: It is tempting to compute only the ( N b + N ∗ ) × N b submatrix of K ( L ) for the task, but the recursion (4) requires the full C ( L − 1) at the presence of A , and hence all the full C ( l ) ’s and K ( l ) ’

Original: s.
Revised: s.

Original: To reduce the computational costs, we resort to a low-rank approximation of C ( l ) , from which we easily see that K ( l +1) is also low-rank.
Revised: To reduce the computational costs, we resort to a low-rank approximation of C ( l ) , from which we easily see that K ( l +1) is also low-rank.

Original: Before deriving the approximation recursion, we note (again) that for the ReLU activation φ , C ( l ) in (3) is the arc-cosine kernel with a closed-form expression:
Revised: Before deriving the approximation recursion, we note (again) that for the ReLU activation φ , C ( l ) in (3) is the arc-cosine kernel with a closed-form expression:

Original: C ( l ) xx (cid:48)
Revised: C ( l ) xx (cid:48)

Original: = 1 2 π (cid:113) K ( l ) xx
Revised: = 1 2 π (cid:113) K ( l ) xx

Original: [Equation]
Revised: [Equation]

Original: where θ ( l ) xx (cid:48)
Revised: where θ ( l ) xx (cid:48)

Original: =
Revised: =

Original: arccos  
Revised: arccos  

Original: K ( l ) xx (cid:48) (cid:113) K
Revised: K ( l ) xx (cid:48) (cid:113) K

Original: ( l ) xx K ( l ) x (cid:48) x (cid:48) 
Revised: ( l ) xx K ( l ) x (cid:48) x (cid:48) 

Original:  .
Revised:  .

Original: Hence, the main idea is that starting with a low-rank approximation of K ( l ) , compute an approximation of C ( l ) by using (5), and then obtain an approximation of K ( l +1) based on (4); then, repeat.
Revised: Hence, the main idea is that starting with a low-rank approximation of K ( l ) , compute an approximation of C ( l ) by using (5), and then obtain an approximation of K ( l +1) based on (4); then, repeat.

Original: To derive the approximation, we use the subscript a to denote a set of landmark nodes with cardinality N a .
Revised: To derive the approximation, we use the subscript a to denote a set of landmark nodes with cardinality N a .

Original: The Nystr ¨ om approximation (Drineas & Mahoney, 2005) of K (0) is K (0): a ( K (0) aa ) − 1 K (0) a : , where the subscript : denotes retaining all rows/columns.
Revised: The Nystr ¨ om approximation (Drineas & Mahoney, 2005) of K (0) is K (0): a ( K (0) aa ) − 1 K (0) a : , where the subscript : denotes retaining all rows/columns.

Original: We rewrite this approximation in the
Revised: We rewrite this approximation in the

Original: Cholesky style as Q (0) Q (0) T , where Q (0) =
Revised: Cholesky style as Q (0) Q (0) T , where Q (0) =

Original: K (0): a ( K (0) aa ) − 12 has size N × N a .
Revised: K (0): a ( K (0) aa ) − 12 has size N × N a .

Original: Proceed with induction.
Revised: Proceed with induction.

Original: Let K ( l ) be approximated by (cid:98) K ( l ) =
Revised: Let K ( l ) be approximated by (cid:98) K ( l ) =

Original: Q ( l ) Q ( l ) T , where Q ( l ) has size N × ( N a + 1) .
Revised: Q ( l ) Q ( l ) T , where Q ( l ) has size N × ( N a + 1) .

Original: We apply (5) to compute an approximation to C ( l ) : a , namely (cid:98) C ( l ) : a , by using (cid:98) K ( l ) : a .
Revised: We apply (5) to compute an approximation to C ( l ) : a , namely (cid:98) C ( l ) : a , by using (cid:98) K ( l ) : a .

Original: Then, (4) leads to a
Revised: Then, (4) leads to a

Original: Cholesky style approximation of K ( l +1) :
Revised: Cholesky style approximation of K ( l +1) :

Original: [Equation]
Revised: [Equation]

Original: where Q ( l +1) = (cid:104)
Revised: where Q ( l +1) = (cid:104)

Original: [Equation]
Revised: [Equation]

Original: l )
Revised: l )

Original: [Equation]
Revised: [Equation]

Original: l
Revised: l

Original: [Equation]
Revised: [Equation]

Original: σ b 1 N × 1 (cid:105) .
Revised: σ b 1 N × 1 (cid:105) .

Original: Clearly, Q ( l +1) has size N × ( N a +1) , completing the induction.
Revised: Clearly, Q ( l +1) has size N × ( N a +1) , completing the induction.

Original: In summary, K ( L ) is approximated by a rank- ( N a +1) matrix (cid:98) K
Revised: In summary, K ( L ) is approximated by a rank- ( N a +1) matrix (cid:98) K

Original: ( L ) =
Revised: ( L ) =

Original: Q ( L ) Q ( L ) T .
Revised: Q ( L ) Q ( L ) T .

Original: The computation of Q ( L ) is summarized in Algorithm 1.
Revised: The computation of Q ( L ) is summarized in Algorithm 1.

Original: Once it is formed, the posterior mean is computed as
Revised: Once it is formed, the posterior mean is computed as

Original: [Equation]
Revised: [Equation]

Original: where note that the matrix to be inverted has size ( N a + 1) ×
Revised: where note that the matrix to be inverted has size ( N a + 1) ×

Original: ( N a + 1) , which is assumed to be signiﬁcantly smaller than N b × N b .
Revised: ( N a + 1) , which is assumed to be signiﬁcantly smaller than N b × N b .

Original: Similarly, the posterior variance is
Revised: Similarly, the posterior variance is

Original: [Equation]
Revised: [Equation]

Original: The computational costs of Q ( L ) and the posterior inference (6)–(7) are summarized in Table
Revised: The computational costs of Q ( L ) and the posterior inference (6)–(7) are summarized in Table

Original:  C OMPOSING G RAPH N EURAL N ETWORK -I NSPIRED K ERNELS
Revised:  C OMPOSING G RAPH N EURAL N ETWORK -I NSPIRED K ERNELS

Original: Theorem 1, together with its proof, suggests that the covariance matrix of the limiting GP can be computed in a composable manner.
Revised: Theorem 1, together with its proof, suggests that the covariance matrix of the limiting GP can be computed in a composable manner.

Original: Moreover, the derivation of Algorithm 1 indicates that the lowrank approximation of the covariance matrix can be similarly composed.
Revised: Moreover, the derivation of Algorithm 1 indicates that the lowrank approximation of the covariance matrix can be similarly composed.

Original: Altogether, such a nice
Revised: Altogether, such a nice

Original: [Table]
Revised: [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: X ← X + b
Revised: X ← [Table]

Original: K ← K + σ 2 b 1 N
Revised: K ← [Table] Q ← [Table]

Revised: [Table]

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: X ← X + b K ← K + σ 2 b 1 N × N Q ← [ Q σ b 1 N × 1 ] Weight

Original: [Table]
Revised: [Table]

Original:  ← XW
Revised: X ← XW

Revised: K ← σ 2 w K

Revised: Q ← σ w Q

Revised: Mixed

Original: [Table]
Revised: [Table]

Original:  ← 
Revised: X ← X ( αI + βW ) K ← ( [Table] + β

Original: [Table]
Revised: [Table]

Original: Graph convolution
Revised: w ) K Q ← (cid:112) [Table] +

Original: [Table]
Revised: [Table]

Original: ←
Revised: σ

Original: [Table]
Revised: [Table]

Revised: Q

Revised: Graph convolution

Revised: X ← AX

Revised: K ← AKA T

Revised: Q ← AQ

Original: Activation
Revised: Activation

Original: X ← φ ( X )
Revised: X ← φ ( X )

Original: K ← g ( K )
Revised: K ← g ( K )

Original: Q [Table] ( g ( QQ T ))
Revised: Q ← Chol( g ( QQ T ))

Original: [Table]
Revised: Independent addition

Original: X ← X 1 + X
Revised: X ← X 1 + X

Original: K ← K 1 + K
Revised: K ← K 1 + K

Original: Q ← [ Q
Revised: Q ← [ Q

Original: Q 2 ] property allows one to easily derive the corresponding covariance matrix and its approximation for a new GNN architecture, like writing a program and obtaining a transformation of it automatically through operator overloading (Novak et al., 2020): the covariance matrix is a transformation of the
Revised: Q 2 ] property allows one to easily derive the corresponding covariance matrix and its approximation for a new GNN architecture, like writing a program and obtaining a transformation of it automatically through operator overloading (Novak et al., 2020): the covariance matrix is a transformation of the

Original: GNN and the composition of the former is in exactly the same manner and order as that of the latter.
Revised: GNN and the composition of the former is in exactly the same manner and order as that of the latter.

Original: We call the covariance matrices programmable .
Revised: We call the covariance matrices programmable .

Original: For example, we write a GCN layer as X ← Aφ ( X ) W + b , where for notational simplicity, X denotes pre-activation rather than post-activation as in the earlier sections.
Revised: For example, we write a GCN layer as X ← Aφ ( X ) W + b , where for notational simplicity, X denotes pre-activation rather than post-activation as in the earlier sections.

Original: The activation φ on X results in a transformation of the kernel matrix K into g ( K ) , deﬁned as:
Revised: The activation φ on X results in a transformation of the kernel matrix K into g ( K ) , deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: due to (3).
Revised: due to (3).

Original: Moreover, if K admits a low-rank approximation QQ T , then g ( K ) admits a low-rank approximation PP T where P = Chol( g ( K )) with
Revised: Moreover, if K admits a low-rank approximation QQ T , then g ( K ) admits a low-rank approximation PP T where P = Chol( g ( K )) with

Original: [Equation]
Revised: [Equation]

Original: The next operation—graph convolution—multiplies A to the left of the post-activation.
Revised: The next operation—graph convolution—multiplies A to the left of the post-activation.

Original: Correspondingly, the covariance matrix K is transformed to AKA T and the low-rank approximation factor Q is transformed to AQ .
Revised: Correspondingly, the covariance matrix K is transformed to AKA T and the low-rank approximation factor Q is transformed to AQ .

Original: Then, the operation—multiplying the weight matrix W to the right—will transform K to σ 2 w K and Q to σ w Q .
Revised: Then, the operation—multiplying the weight matrix W to the right—will transform K to σ 2 w K and Q to σ w Q .

Original: Finally, adding the bias b will transform K to K + σ 2 b 1 N × N and Q to [ Q σ b 1 N × 1 ] .
Revised: Finally, adding the bias b will transform K to K + σ 2 b 1 N × N and Q to [ Q σ b 1 N × 1 ] .

Original: Altogether, we have obtained the following updates per layer:
Revised: Altogether, we have obtained the following updates per layer:

Original: [Equation]
Revised: [Equation]

Original: One may verify the K update against (3)–(4) and the Q update against Algorithm 1.
Revised: One may verify the K update against (3)–(4) and the Q update against Algorithm 1.

Original: Both updates can be automatically derived based on the update of X .
Revised: Both updates can be automatically derived based on the update of X .

Original: We summarize the building blocks of a GNN and the corresponding kernel/low-rank operations in
Revised: We summarize the building blocks of a GNN and the corresponding kernel/low-rank operations in

Original: Table 2.
Revised: Table 2.

Original: The independent-addition building block is applicable to skip/residual connections.
Revised: The independent-addition building block is applicable to skip/residual connections.

Original: For example, here is the composition for the GCNII layer (Chen et al., 2020) without a bias term, where a skip connection with X (0) occurs:
Revised: For example, here is the composition for the GCNII layer (Chen et al., 2020) without a bias term, where a skip connection with X (0) occurs:

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: For another example of the composability, we consider the popular GIN layer (Xu et al., 2019), which we assume uses a 2-layer MLP after the neighborhood aggregation:
Revised: For another example of the composability, we consider the popular GIN layer (Xu et al., 2019), which we assume uses a 2-layer MLP after the neighborhood aggregation:

Original: GIN :
Revised: GIN :

Original: [Equation]
Revised: [Equation]

Original: whereB =
Revised: where

Original: [Equation]
Revised: [Equation]

Original: where P = (cid:2)
Revised: where P = (cid:2)

Original: [Equation]
Revised: [Equation]

Original: A

Original: [Equation]

Original: ( g ( QQ [Equation] [Equation] (cid:3) .
Revised: Additionally, the updates for a GraphSAGE layer (Hamilton et al., 2017) are given in Appendix B.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we conduct a comprehensive set of experiments to evaluate the performance of the
Revised: In this section, we conduct a comprehensive set of experiments to evaluate the performance of the

Original: GP kernels derived by taking limits on the layer width of GCN and other GNNs.
Revised: GP kernels derived by taking limits on the layer width of GCN and other GNNs.

Original: We demonstrate that these GPs are comparable with GNNs in prediction performance, while being signiﬁcantly faster to compute.
Revised: We demonstrate that these GPs are comparable with GNNs in prediction performance, while being signiﬁcantly faster to compute.

Original: We also show that the low-rank version scales favorably, suitable for practical use.
Revised: We also show that the low-rank version scales favorably, suitable for practical use.

Original: Datasets.
Revised: Datasets.

Original: The experiments are conducted on several benchmark datasets of varying sizes, covering both classiﬁcation and regression.
Revised: The experiments are conducted on several benchmark datasets of varying sizes, covering both classiﬁcation and regression.

Original: They include predicting the topic of scientiﬁc papers organized in a citation network (Cora, Citeseer, PubMed, and ArXiv); predicting the community of online posts based on user comments (Reddit), and predicting the average daily trafﬁc of Wikipedia pages using hyperlinks among them (Chameleon, Squirrel, and Crocodile).
Revised: They include predicting the topic of scientiﬁc papers organized in a citation network (Cora, Citeseer, PubMed, and ArXiv); predicting the community of online posts based on user comments (Reddit), and predicting the average daily trafﬁc of Wikipedia pages using hyperlinks among them (Chameleon, Squirrel, and Crocodile).

Original: All datasets, with predeﬁned training/validation/test splits, are downloaded from the PyTorch-Geometric library (Fey & Lenssen, 2019), except for ArXiv, which comes from the Open Graph Benchmark (Hu et al., 2020).

Original: A summary of the datasets is given in Table 3 andadditional preprocessing details  are given in Appendix B.
Revised: A summary of the datasets is given in Table 3 and details (including sources and preprocessing) are given in Appendix C.

Revised: Experiment environment, training details, and hyperparameters are given in Appendix C.

Original: Prediction Performance: GCN-based comparison.
Revised: Prediction Performance: GCN-based comparison.

Original: We ﬁrst conduct the semi-supervised learning tasks on all datasets by using GCN and GPs with different kernels.
Revised: We ﬁrst conduct the semi-supervised learning tasks on all datasets by using GCN and GPs with different kernels.

Original: These kernels include the one equivalent to the limiting GCN (GCNGP) and a usual squared-exponential kernel (RBF).
Revised: These kernels include the one equivalent to the limiting GCN (GCNGP), a usual squared-exponential kernel (RBF), and the GGP kernel proposed by Ng et al.

Original: Each of these kernels has a low-rank version (sufﬁxed with -X).
Revised: Each of these kernels has a low-rank version (sufﬁxed with -X).

Original: RBF-X uses the Nystr ¨ om approximation, consistent with GCNGP-X.
Revised: RBF-X and GGP-X 4 use the Nystr¨om approximation, consistent with GCNGP-X.

Original: For all datasets except Reddit, we train GCN using the Adam optimizer in full batch for 100 epochs.

Original: Reddit is too large for GPU computation and hence we conduct mini-batch training.

Original: Hyperparameter tuning is done on the validation set.

Original: For additional hyperparameter details, see Appendix B.

Original: GPs are by nature suitable for regression.
Revised: GPs are by nature suitable for regression.

Original: For classiﬁcation tasks, we use the one-hot representation of labels to set up a multi-output regression.
Revised: For classiﬁcation tasks, we use the one-hot representation of labels to set up a multi-output regression.

Original: Then, we take the coordinate with the largest output as the class prediction.
Revised: Then, we take the coordinate with the largest output as the class prediction.

Original: Such an ad hoc treatment is widely used in the literature, as other more principled approaches (such as using  Laplace approximation on the non-Gaussian posterior) are too time-consuming for large datasets, meanwhile producing no noticeable gain in accuracy.
Revised: Such an ad hoc treatment is widely used in the literature, as other more principled approaches (such as using the Laplace approximation on the non-Gaussian posterior) are too time-consuming for large datasets, meanwhile producing no noticeable gain in accuracy.

Original: Table 4 summarizes the accuracy for classiﬁcation and the coefﬁcient of determination, R 2 , for regression.
Revised: Table 4 summarizes the accuracy for classiﬁcation and the coefﬁcient of determination, R 2 , for regression.

Original: Whenever randomness is involved, the performance is reported as an average over ﬁve runs.
Revised: Whenever randomness is involved, the performance is reported as an average over ﬁve runs.

Original: The results of the two tasks show different patterns.
Revised: The results of the two tasks show different patterns.

Original: For classiﬁcation, GCNGP(-X) is sligtly better than GCN, while RBF(-X) is signiﬁcantly worse than both; moreover, the low-rank version is outperformed by using the full kernel matrix.
Revised: For classiﬁcation, GCNGP(-X) is sligtly better than GCN and GGP(-X), while RBF(-X) is signiﬁcantly worse than all others; moreover, the low-rank version is outperformed by using the full kernel matrix.

Original: On the other hand, for regression, GCNGP(-X) signiﬁcantly outperforms GCN and RBF(-X); and the low-rank version becomes better.
Revised: On the other hand, for regression, GCNGP(-X) signiﬁcantly outperforms GCN, RBF(-X), and GGP(-X); and the low-rank version [Table] [Table]

Original:  The poor performance of RBF(-X) is expected, as it does not leverage the graph inductive bias.
Revised: becomes better. The less competitive performance of RBF(-X) is expected, as it does not leverage the graph inductive bias.

Original: It is attractive that GCNGP(-X) is competitive with GCN.
Revised: It is attractive that GCNGP(-X) is competitive with GCN.

Original: [Table]
Revised: Prediction Performance:

Original: [Table] In addition to GCN, we conduct experiments with several popularly used GNN architectures (GCNII, GIN, and GraphSAGE) and GPs with the corresponding kernels.
Revised: Comparison with other GNNs. In addition to GCN, we conduct experiments with several popularly used GNN architectures (GCNII, GIN, and GraphSAGE) and GPs with the corresponding kernels.

Original: We test with ArXiv and use the  low-rank version of the GPs  for computational feasibility.
Revised: We test with the three largest datasets: PubMed, ArXiv, and Reddit, for the latter two of which a low-rank version of the GPs is used for computational feasibility.

Original: Table 5 summarizes the results.
Revised: Table 5 summarizes the results.

Original: Similar to the GCN case, in other GNN cases we see generally a similar performance between the GNN and its GP counterpart.
Revised: The observations on other GNNs extend similarly those on the GCN. In particular, on PubMed the GPs noticeably improve over the corresponding GNNs, while on ArXiv and Reddit the two families perform rather similarly.

Original: A notable difference does exist (GIN), where the GIN performance is subpar, even compared with other GNNs.
Revised: An exception is GIN for ArXiv, which signiﬁcantly underperforms the GP counterpart, as well as other GNNs.

Original: It may improve with an extensive hyperparameter tuning.
Revised: It may improve with an extensive hyperparameter tuning.

Original: Running time.
Revised: Running time.

Original: We compare the running time of the methods covered by Table 4.
Revised: We compare the running time of the methods covered by Table 4.

Original: Different from usual neural networks, the training and inference of GNNs do not decouple in full-batch training.
Revised: Different from usual neural networks, the training and inference of GNNs do not decouple in full-batch training.

Original: Moreover, there is not a universally agreed split between the training and the inference steps in GPs.
Revised: Moreover, there is not a universally agreed split between the training and the inference steps in GPs.

Original: Hence, we compare the total time for each method.
Revised: Hence, we compare the total time for each method.

Original: Figure 1 plots the timing results, normalized against the GCN time for ease of comparison.
Revised: Figure 1 plots the timing results, normalized against the GCN time for ease of comparison.

Original: It suggests that GCNGP(-X) is generally faster than GCN.
Revised: It suggests that GCNGP(-X) is generally faster than GCN.

Original: Note that the vertical axis is in the logarithmic scale.
Revised: Note that the vertical axis is in the logarithmic scale.

Original: Hence, for some of the datasets, the speedup is even one to two orders of magnitude.
Revised: Hence, for some of the datasets, the speedup is even one to two orders of magnitude.

Original: Scalability.
Revised: Scalability.

Original: For graphs, especially under the semi-supervised learning setting, the computational cost of a GP is much more complex than that of a usual one (which can be simply described as “cubic in the training set size”).
Revised: For graphs, especially under the semi-supervised learning setting, the computational cost of a GP is much more complex than that of a usual one (which can be simply described as “cubic in the training set size”).

Original: One sees in Table 1 the many factors that determine the cost of our graph-based low-rank kernels.
Revised: One sees in Table 1 the many factors that determine the cost of our graph-based low-rank kernels.

Original: To explore the practicality of the proposed method, we use the timings gathered for Figure 1 to obtain an empirical scaling with respect to the graph size, M + N .
Revised: To explore the practicality of the proposed method, we use the timings gathered for Figure 1 to obtain an empirical scaling with respect to the graph size, M + N .

Original: Figure 2 ﬁts the running times, plotted in the log-log scale, by using a straight line.
Revised: Figure 2 ﬁts the running times, plotted in the log-log scale, by using a straight line.

Original: We see that for neither GCN nor GCNGP(-X), the actual running time closely follows a polynomial complexity.
Revised: We see that for neither GCN nor GCNGP(-X), the actual running time closely follows a polynomial complexity.

Original: However, interestingly, the least-squares ﬁttings all lead to a slope approximately 1, which agrees with a linear cost.
Revised: However, interestingly, the least-squares ﬁttings all lead to a slope approximately 1, which agrees with a linear cost.

Original: Theoretically, only GCNGP-X and GCN are approximately linear with respect to
Revised: Theoretically, only GCNGP-X and GCN are approximately linear with respect to

Original: M + N , while GCNGP is cubic.
Revised: M + N , while GCNGP is cubic.

Original: Analysis on the depth.
Revised: Analysis on the depth.

Original: The performance of GCN deteriorates with more layers, known as the oversmoothing phenomenon.
Revised: The performance of GCN deteriorates with more layers, known as the oversmoothing phenomenon.

Original: Adding residual/skip connections mitigates the problem, such as in
Revised: Adding residual/skip connections mitigates the problem, such as in

Original: GCNII.
Revised: GCNII.

Original: A natural question asks if the corresponding GP kernels behave similarly.
Revised: A natural question asks if the corresponding GP kernels behave similarly.

Original: Figure 3 shows that the trends of GCN and GCNII are indeed as expected.
Revised: Figure 3 shows that the trends of GCN and GCNII are indeed as expected.

Original: Interestingly, their GP counterparts both remain stable for depth L as large as 12.
Revised: Interestingly, their GP counterparts both remain stable for depth L as large as 12.

Original: Our depth analysis (Theorem 4) suggests that in the limit, the GPs may perform less well because the kernel matrix may degenerate to rank 1.
Revised: Our depth analysis (Theorem 4) suggests that in the limit, the GPs may perform less well because the kernel matrix may degenerate to rank 1.

Original: This empirical result indicates that the drop in performance may have not started yet.
Revised: This empirical result indicates that the drop in performance may have not started yet.

Original: Analysis on the landmark set.
Revised: Analysis on the landmark set.

Original: The number of landmark nodes, N a , controls the approximation quality of the low-rank kernels and hence the prediciton accuracy.
Revised: The number of landmark nodes, N a , controls the approximation quality of the low-rank kernels and hence the prediciton accuracy.

Original: On the other hand, the computational costs summarized in Table 1 indicate a dependency on N a as high as the third power.
Revised: On the other hand, the computational costs summarized in Table 1 indicate a dependency on N a as high as the third power.

Original: It is crucial to develop an empirical understanding of the accuracy-time trade-off it incurs.
Revised: It is crucial to develop an empirical understanding of the accuracy-time trade-off it incurs.

Original: Figure 4 clearly shows that as N a becomes larger, the running time increase is not linear, while the increase of accuracy diminishes as the landmark set approaches the training set.
Revised: Figure 4 clearly shows that as N a becomes larger, the running time increase is not linear, while the increase of accuracy diminishes as the landmark set approaches the training set.

Original: It is remarkable that using only 1/800 of the training set as landmark nodes already achieves an accuracy surpassing that of GCN, by using time that is only a tiny fraction of the time otherwise needed to gain an additional 1% increase in the accuracy.
Revised: It is remarkable that using only 1/800 of the training set as landmark nodes already achieves an accuracy surpassing that of GCN, by using time that is only a tiny fraction of the time otherwise needed to gain an additional 1% increase in the accuracy.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have presented a GP approach for semi-supervised learning on graph-structured data, where the covariance kernel incorporates a graph inductive bias by exploiting the relationship between a
Revised: We have presented a GP approach for semi-supervised learning on graph-structured data, where the covariance kernel incorporates a graph inductive bias by exploiting the relationship between a

Original: GP and a GNN with inﬁnitely wide layers.
Revised: GP and a GNN with inﬁnitely wide layers.

Original: Similar to other neural networks priorly investigated, one can work out the equivalent GP (in particular, the covariance kernel) for GCN; and inspired by this equivalence, we formulate a procedure to compose covariance kernels corresponding to many other members of the GNN family.
Revised: Similar to other neural networks priorly investigated, one can work out the equivalent GP (in particular, the covariance kernel) for GCN; and inspired by this equivalence, we formulate a procedure to compose covariance kernels corresponding to many other members of the GNN family.

Original: Moreover, every building block in the procedure has a lowrank counterpart, which allows for building a low-rank approximation of the covariance matrix that facilitates scalable posterior inference.
Revised: Moreover, every building block in the procedure has a lowrank counterpart, which allows for building a low-rank approximation of the covariance matrix that facilitates scalable posterior inference.

Original: We demonstrate the effectiveness of the derived kernels used for semi-supervised learning and show their advantages in computation time over GNNs.
Revised: We demonstrate the effectiveness of the derived kernels used for semi-supervised learning and show their advantages in computation time over GNNs.

Original: A P ROOFS AND A DDITIONAL T HEOREMS A. P ROOF OF T HEOREM
Revised: A P ROOFS AND A DDITIONAL T HEOREMS A. P ROOF OF T HEOREM

Original: We use mathematical induction.
Revised: We use mathematical induction.

Original: First, write the matrix form of (2) as follows:
Revised: First, write the matrix form of (2) as follows:

Original: [Equation]
Revised: [Equation]

Original: Assume each column of Z ( l − 1) is iid, then X ( l − 1) also has iid columns.
Revised: Assume each column of Z ( l − 1) is iid, then X ( l − 1) also has iid columns.

Original: Hence, y ( l ) i ( x ) is a sum of iid terms and thus normal.
Revised: Hence, y ( l ) i ( x ) is a sum of iid terms and thus normal.

Original: Also, ( y ( l ) i ( x 1 ) , · · · , y ( l ) i ( x N )) are jointly multivariate normal, and identically distributed for different i ; so they form a GP.
Revised: Also, ( y ( l ) i ( x 1 ) , · · · , y ( l ) i ( x N )) are jointly multivariate normal, and identically distributed for different i ; so they form a GP.

Original: Moreover, terms like y ( l ) i ( x ) and y ( l ) i (cid:48)
Revised: Moreover, terms like y ( l ) i ( x ) and y ( l ) i (cid:48)

Original: ( x (cid:48) ) with i (cid:54) =
Revised: ( x (cid:48) ) with i (cid:54) =

Original: i (cid:48) are jointly Gaussian, with zero covariance:
Revised: i (cid:48) are jointly Gaussian, with zero covariance:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: Thus, they are independent, despite the fact that they may share the same X ( l − 1) terms.
Revised: Thus, they are independent, despite the fact that they may share the same X ( l − 1) terms.

Original: In conclusion, each column of Y ( l ) is an iid GP.
Revised: In conclusion, each column of Y ( l ) is an iid GP.

Original: Hence, each column of Z ( l ) is also an iid GP N (0 , K ( l ) ) .
Revised: Hence, each column of Z ( l ) is also an iid GP N (0 , K ( l ) ) .

Original: The covariance K ( l ) can be computed recursively.
Revised: The covariance K ( l ) can be computed recursively.

Original: Similar to the case of a fully connected network, the covariance of Y ( l ) is
Revised: Similar to the case of a fully connected network, the covariance of Y ( l ) is

Original: [Equation]
Revised: [Equation]

Original: where
Revised: where

Original: [Equation]
Revised: [Equation]

Original: for any j.
Revised: for any j.

Original: In the matrix form, this is to say Cov( Y ( l ) j )
Revised: In the matrix form, this is to say Cov( Y ( l ) j )

Original: =
Revised: =

Original: σ 2 w E[ X ( l − 1)
Revised: σ 2 w E[ X ( l − 1)

Original: j
Revised: j

Original: X
Revised: X

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: C ( l
Revised: C ( l

Original: [Equation]
Revised: [Equation]

Original: Then, because
Revised: Then, because

Original: Z ( l ) i =
Revised: Z ( l ) i =

Original: AY ( l ) i + 1 N × 1 b ( l ) i , we obtain
Revised: AY ( l ) i + 1 N × 1 b ( l ) i , we obtain

Original: [Equation]
Revised: [Equation]

Original: which concludes the proof.
Revised: which concludes the proof.

Original: A. P ROOF OF L EMMA
Revised: A. P ROOF OF L EMMA

Original: We know that φ ( w · x ) is the feature mapping of the arc-cosine kernel.
Revised: We know that φ ( w · x ) is the feature mapping of the arc-cosine kernel.

Original: It sufﬁces to show for different x 1 , . . .
Revised: It sufﬁces to show for different x 1 , . . .

Original: , x m ∈ S , the φ ( w, x i ) ’s are linearly independent.
Revised: , x m ∈ S , the φ ( w, x i ) ’s are linearly independent.

Original: We prove this by contradiction.
Revised: We prove this by contradiction.

Original: Assume there exist c 1 , . . .
Revised: Assume there exist c 1 , . . .

Original: , c m not simultaneously zero, such that
Revised: , c m not simultaneously zero, such that

Original: [Equation]
Revised: [Equation]

Original: WLOG, we further assume that m is the smallest integer that satisﬁes (9).
Revised: WLOG, we further assume that m is the smallest integer that satisﬁes (9).

Original: Let e 

Revised: Let e

Original: =
Revised: =

Original: (1 , 0 , · · · , 0) ; then x · e 1 > 0 , ∀ x ∈ S .
Revised: (1 , 0 , · · · , 0) ; then x · e 1 > 0 , ∀ x ∈ S .

Original: Assume x m · e 1 = min 1 ≤ i ≤ m { x i · e 1 } > 0 .
Revised: Assume x m · e 1 = min 1 ≤ i ≤ m { x i · e 1 } > 0 .

Original: Then, for 1 ≤ i ≤ m − 1 , x i · x m
Revised: Then, for 1 ≤ i ≤ m − 1 , x i · x m

Original: [Equation]
Revised: [Equation]

Original: Therefore, there exists t such that
Revised: Therefore, there exists t such that

Original: [Equation]
Revised: [Equation]

Original: ≤ m −
Revised: ≤ m −

Original: [Equation]
Revised: [Equation]

Original: Let w = x m − te 1 ; then w · x m > 0 , w · x i < 0 , 1 ≤ i ≤ m − 1 .
Revised: Let w = x m − te 1 ; then w · x m > 0 , w · x i < 0 , 1 ≤ i ≤ m − 1 .

Original: Using (9) we know c m ( w · x m )
Revised: Using (9) we know c m ( w · x m )

Original: = 

Revised: =

Original: ; so
Revised: ; so

Original: c m = 0 .
Revised: c m = 0 .

Original: Thus, c 1 , · · · , c m − 1 and φ ( w · x 1 ) , · · · , φ ( w · x m − 1 ) also satisfy (9), with a smaller number m , which forms a contradiction.
Revised: Thus, c 1 , · · · , c m − 1 and φ ( w · x 1 ) , · · · , φ ( w · x m − 1 ) also satisfy (9), with a smaller number m , which forms a contradiction.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: From the recursive relation (3)–(4), we know C (0) is element-wise non-negative.
Revised: From the recursive relation (3)–(4), we know C (0) is element-wise non-negative.

Original: Hence, K (1) is element-wise non-negative, C (1) is element-wise positive, and K (2) is element-wise positive.
Revised: Hence, K (1) is element-wise non-negative, C (1) is element-wise positive, and K (2) is element-wise positive.

Original: Assume the Cholesky decomposition K (2) = LL T and denote L =
Revised: Assume the Cholesky decomposition K (2) = LL T and denote L =

Original: ( l 1 , · · · , l n ) T .
Revised: ( l 1 , · · · , l n ) T .

Original: Let r i = (cid:107) l i (cid:107) 2 > 0 and x i = l i r i .
Revised: Let r i = (cid:107) l i (cid:107) 2 > 0 and x i = l i r i .

Original: Then, x 1 = ± e 1 , and WLOG we assume x 1 = e 1 .
Revised: Then, x 1 = ± e 1 , and WLOG we assume x 1 = e 1 .

Original: Then for each x i , we know
Revised: Then for each x i , we know

Original: [Equation]
Revised: [Equation]

Original: and each (cid:107) x i (cid:107) 2 = 1 .
Revised: and each (cid:107) x i (cid:107) 2 = 1 .

Original: Therefore, x i ∈ S is deﬁned in Lemma 2.
Revised: Therefore, x i ∈ S is deﬁned in Lemma 2.

Original: By the assumption that C (0) does not contain two linearly dependent rows, we know so are K (1) and K (2) .
Revised: By the assumption that C (0) does not contain two linearly dependent rows, we know so are K (1) and K (2) .

Original: Thus, no two l i ’s are linearly dependent and hence x i ’s are pairwise distinct.
Revised: Thus, no two l i ’s are linearly dependent and hence x i ’s are pairwise distinct.

Original: Using the universality of the arc-cosine kernel (Lemma 2), we know C (2) =
Revised: Using the universality of the arc-cosine kernel (Lemma 2), we know C (2) =

Original: g ( K (2) ) is positive deﬁnite.
Revised: g ( K (2) ) is positive deﬁnite.

Original: Hence, K (3) is positive deﬁnite.
Revised: Hence, K (3) is positive deﬁnite.

Original: An induction argument on the layer index completes the proof.
Revised: An induction argument on the layer index completes the proof.

Original: A. P ROOF OF T HEOREM
Revised: A. P ROOF OF T HEOREM

Original: For a covariance matrix K , denote its standard deviation and correlation by
Revised: For a covariance matrix K , denote its standard deviation and correlation by

Original: [Equation]
Revised: [Equation]

Original: From the closed-form formula (5) for the arc-cosine kernel, we have
Revised: From the closed-form formula (5) for the arc-cosine kernel, we have

Original: [Equation]
Revised: [Equation]

Original: We deﬁne the correlation mapping from ρ K ( l ) to ρ C ( l ) :
Revised: We deﬁne the correlation mapping from ρ K ( l ) to ρ C ( l ) :

Original: [Equation]
Revised: [Equation]

Original: We ﬁrst establish a few properties of f .
Revised: We ﬁrst establish a few properties of f .

Original: A pictorial illustration of f is given in Figure 5.
Revised: A pictorial illustration of f is given in Figure 5.

Original: Lemma 5.
Revised: Lemma 5.

Original: The following facts hold.
Revised: The following facts hold.

Original: f is increasing and it is a contraction mapping on [ − 1 , 1] .
Revised: f is increasing and it is a contraction mapping on [ − 1 , 1] .

Original: f ( ρ ) ≥ ρ and
Revised: f ( ρ ) ≥ ρ and

Original: the equality holds only
Revised: the equality holds only

Original: when ρ = 1 .
Revised: when ρ = 1 .

Original: Proof.
Revised: Proof.

Original: For part 1, denote ρ = cos θ ∈ [ − 1 , 1] .
Revised: For part 1, denote ρ = cos θ ∈ [ − 1 , 1] .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Thus, f is increasing and it is a contraction mapping on [ − 1 , 1] .
Revised: Thus, f is increasing and it is a contraction mapping on [ − 1 , 1] .

Original: For part 2, we know that f (1) = 1 .
Revised: For part 2, we know that f (1) = 1 .

Original: Using the contraction mapping, we know f ( ρ ) ≥ ρ for ρ ∈ [ − 1 , 1] .
Revised: Using the contraction mapping, we know f ( ρ ) ≥ ρ for ρ ∈ [ − 1 , 1] .

Original: Clearly, the equality holds only when ρ = 1 .
Revised: Clearly, the equality holds only when ρ = 1 .

Original: After the correlation mapping f , we further consider the covariance mapping g : K ( l ) (cid:55)→ C ( l ) deﬁned in (3) (and equivalently in (8)).
Revised: After the correlation mapping f , we further consider the covariance mapping g : K ( l ) (cid:55)→ C ( l ) deﬁned in (3) (and equivalently in (8)).

Original: We establish a few properties of g below.
Revised: We establish a few properties of g below.

Original: Lemma 6.
Revised: Lemma 6.

Original: For positive semi-deﬁnite matrices B , the following facts hold.
Revised: For positive semi-deﬁnite matrices B , the following facts hold.

Original: g ( B ) =
Revised: g ( B ) =

Original: 12 D 1 / 2 f
Revised: 12 D 1 / 2 f

Original: ( D − 1 / 2 BD − 1 / 2 ) D 1 / 2 , where D = diag { B } ∈ R N × N and f is elementwise.
Revised: ( D − 1 / 2 BD − 1 / 2 ) D 1 / 2 , where D = diag { B } ∈ R N × N and f is elementwise.

Original: g ( B ) (cid:31) 12 B and diag { g ( B ) } = 12 diag { B } , where (cid:31) means elementwise greater. 3.
Revised: g ( B ) (cid:31) 12 B and diag { g ( B ) } = 12 diag { B } , where (cid:31) means elementwise greater. 3.

Original: tr (cid:0) Ag ( B ) A T (cid:1) ≤ 12 (cid:107) A (cid:107) 22 tr ( B ) , where (cid:107) A (cid:107) 2 is the spectral norm of matrix A .
Revised: tr (cid:0) Ag ( B ) A T (cid:1) ≤ 12 (cid:107) A (cid:107) 22 tr ( B ) , where (cid:107) A (cid:107) 2 is the spectral norm of matrix A .

Original: Proof.
Revised: Proof.

Original: For part 1, it is obvious according to (5).
Revised: For part 1, it is obvious according to (5).

Original: For part 2, using the properties of f we straightforwardly obtain
Revised: For part 2, using the properties of f we straightforwardly obtain

Original: [Equation]
Revised: [Equation]

Original: For part 3, note that g ( B ) is positive semi-deﬁnite.
Revised: For part 3, note that g ( B ) is positive semi-deﬁnite.

Original: Using von Neumann’s trace inequality,
Revised: Using von Neumann’s trace inequality,

Original: [Equation]
Revised: [Equation]

Original: where the last term is equal to (cid:107) A (cid:107) 22 tr ( g ( B ))
Revised: where the last term is equal to (cid:107) A (cid:107) 22 tr ( g ( B ))

Original: =
Revised: =

Original: 12 (cid:107) A (cid:107) 22 tr ( B ) .
Revised: 12 (cid:107) A (cid:107) 22 tr ( B ) .

Original: Now, we are ready to prove Theorem
Revised: Now, we are ready to prove Theorem

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem 4, part 1.
Revised: Proof of Theorem 4, part 1.

Original: We focus on uniform convergence over all pairwise correlations.
Revised: We focus on uniform convergence over all pairwise correlations.

Original: Denote
Revised: Denote

Original: [Equation]
Revised: [Equation]

Original: To simplify notation
Revised: To simplify notation

Original: [Equation]
Revised: [Equation]

Original: we temporarily let C =
Revised: we temporarily let C =

Original: C
Revised: C

Original: ( l ) .
Revised: ( l ) .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: ) (cid:33) .
Revised: ) (cid:33) .

Original: Meanwhile,
Revised: Meanwhile,

Original: [Equation]
Revised: [Equation]

Original: and similarly
Revised: and similarly

Original: [Equation]
Revised: [Equation]

Original: Combining the above inequalities, we obtain
Revised: Combining the above inequalities, we obtain

Original: [Equation]
Revised: [Equation]

Original: and thus ρ
Revised: and thus ρ

Original: [Equation]
Revised: [Equation]

Original: K
Revised: K

Original: [Equation]
Revised: [Equation]

Original: From the correlation mapping, we know
Revised: From the correlation mapping, we know

Original: [Equation]
Revised: [Equation]

Original: Therefore, by the properties of f , ρ min
Revised: Therefore, by the properties of f , ρ min

Original: [Equation]
Revised: [Equation]

Original: Thus, ρ min ( K ( l ) ) (cid:37) some ρ ∞ ∈ [ − 1 , 1] .
Revised: Thus, ρ min ( K ( l ) ) (cid:37) some ρ ∞ ∈ [ − 1 , 1] .

Original: Because ρ ∞ = f ( ρ ∞ ) , we conclude that ρ ∞ = 1 .
Revised: Because ρ ∞ = f ( ρ ∞ ) , we conclude that ρ ∞ = 1 .

Original: Proof of Theorem 4, part 2. Consider each h : K ( l ) (cid:55)→ K ( l +1) :
Revised: Proof of Theorem 4, part 2. Consider each h : K ( l ) (cid:55)→ K ( l +1) :

Original: [Equation]
Revised: [Equation]

Original: Let δ = 12 σ 2 w λ 2 < 1 .
Revised: Let δ = 12 σ 2 w λ 2 < 1 .

Original: Using part 3 of Lemma 6,
Revised: Using part 3 of Lemma 6,

Original: [Equation]
Revised: [Equation]

Original: Therefore, for sufﬁciently large l ,
Revised: Therefore, for sufﬁciently large l ,

Original: [Equation]
Revised: [Equation]

Original: Because each K ( l ) is positive semi-deﬁnite, we know K ( l ) is bounded.
Revised: Because each K ( l ) is positive semi-deﬁnite, we know K ( l ) is bounded.

Original: Thus, a subsequence of
Revised: Thus, a subsequence of

Original: K ( l ) → some matrix.
Revised: K ( l ) → some matrix.

Original: Proof of Theorem
Revised: Proof of Theorem

Original: [Equation]
Revised: [Equation]

Original: part
Revised: part

Original: [Equation]
Revised: [Equation]

Original: Deﬁne κ ( l ) :=
Revised: Deﬁne κ ( l ) :=

Original: K ( l ) /c l .
Revised: K ( l ) /c l .

Original: We have the following recursive relationship between κ ( l ) and κ ( l +1) :
Revised: We have the following recursive relationship between κ ( l ) and κ ( l +1) :

Original: [Equation]
Revised: [Equation]

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: where the inequality results from part 3 of Lemma 8.
Revised: where the inequality results from part 3 of Lemma 8.

Original: Therefore, κ ( l ) is bounded.
Revised: Therefore, κ ( l ) is bounded.

Original: Using the Perron-Frobenius theorem, we know the eigenspace V corresponding to λ is onedimensional, and there exists v ∈ V such that (cid:107) v (cid:107) 2 = 1 and v (cid:31) 0 .
Revised: Using the Perron-Frobenius theorem, we know the eigenspace V corresponding to λ is onedimensional, and there exists v ∈ V such that (cid:107) v (cid:107) 2 = 1 and v (cid:31) 0 .

Original: Then,
Revised: Then,

Original: [Equation]
Revised: [Equation]

Original: Therefore, v T κ ( l ) v has a limit c > 0 .
Revised: Therefore, v T κ ( l ) v has a limit c > 0 .

Original: Let P =
Revised: Let P =

Original: Proj V ⊥
Revised: Proj V ⊥

Original: =
Revised: =

Original: I − Proj V
Revised: I − Proj V

Original: = I − vv T .
Revised: = I − vv T .

Original: Then PA = AP , (cid:107) PA (cid:107) 2 = λ (cid:48) <
Revised: Then PA = AP , (cid:107) PA (cid:107) 2 = λ (cid:48) <

Original: λ
Revised: λ

Original: [Equation]
Revised: [Equation]

Original: and tr
Revised: and tr

Original: [Equation]
Revised: [Equation]

Original: Pg
Revised: Pg

Original: [Equation]
Revised: [Equation]

Original: =
Revised: =

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Thus, Pκ ( l ) P T → 0 .
Revised: Thus, Pκ ( l ) P T → 0 .

Original: Combining the results, we have κ ( l ) → cvv T .
Revised: Combining the results, we have κ ( l ) → cvv T .

Original: A. R ESULTS FOR M ULTI -L AYER P ERCEPTRONS
Revised: A. R ESULTS FOR M ULTI -L AYER P ERCEPTRONS

Original: Theorem 4 has parallel results for fully connected layers.
Revised: Theorem 4 has parallel results for fully connected layers.

Original: Theorem 7.
Revised: Theorem 7.

Original: Let A be the identity matrix.
Revised: Let A be the identity matrix.

Original: The following results hold as l → ∞ .
Revised: The following results hold as l → ∞ .

Original: When
Revised: When

Original: σ
Revised: σ

Original: 2 w
Revised: 2 w

Original: < 2 and σ b >
Revised: < 2 and σ b >

Original: , then K (
Revised: , then K (

Original: l )
Revised: l )

Original: →
Revised: →

Original: q
Revised: q

Original: 1 N × N where q =
Revised: 1 N × N where q =

Original: σ 2 b 1 − σ w / 2 .
Revised: σ 2 b 1 − σ w / 2 .

Original: When σ 2 w > 2 , let c l = ( σ 2 w / 2) l ; then K ( l )
Revised: When σ 2 w > 2 , let c l = ( σ 2 w / 2) l ; then K ( l )

Original: c l → vv
Revised: c l → vv

Original: T
Revised: T

Original: where
Revised: where

Original: v
Revised: v

Original: x
Revised: x

Original: = (cid:20)
Revised: = (cid:20)

Original: [Equation]
Revised: [Equation]

Original: Proof of Theorem
Revised: Proof of Theorem

Original: [Equation]
Revised: [Equation]

Original: part 1.
Revised: part 1.

Original: Starting with the diagonal elements
Revised: Starting with the diagonal elements

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Therefore
Revised: Therefore

Original: [Equation]
Revised: [Equation]

Original: K ( l ) ( x, x ) → q =
Revised: K ( l ) ( x, x ) → q =

Original: [Equation]
Revised: [Equation]

Original: For the off-diagonal elements
Revised: For the off-diagonal elements

Original: [Equation]
Revised: [Equation]

Original: we have
Revised: we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: Meanwhile
Revised: Meanwhile

Original: [Equation]
Revised: [Equation]

Original: Thus, K ( l ) ( x, x (cid:48) )
Revised: Thus, K ( l ) ( x, x (cid:48) )

Original: → q .
Revised: → q .

Original: In conclustion, K ( l ) → q 1 N × N .
Revised: In conclustion, K ( l ) → q 1 N × N .

Original: Proof of Theorem 7, part 2.
Revised: Proof of Theorem 7, part 2.

Original: Again, we start with diagonal elements.
Revised: Again, we start with diagonal elements.

Original: Deﬁne κ ( l ) :
Revised: Deﬁne κ ( l ) :

Original: = K ( l ) /c
Revised: = K ( l ) /c

Original: l .
Revised: l .

Original: When σ 2 w > 2 , we have
Revised: When σ 2 w > 2 , we have

Original: [Equation]
Revised: [Equation]

Original: Therefore,
Revised: Therefore,

Original: [Equation]
Revised: [Equation]

Original: For the off-diagonal elements, we have
Revised: For the off-diagonal elements, we have

Original: [Equation]
Revised: [Equation]

Original: Thus, κ ( l ) ( x, x (cid:48) ) has a limit c .
Revised: Thus, κ ( l ) ( x, x (cid:48) ) has a limit c .

Original: Taking limits on both sides, we have
Revised: Taking limits on both sides, we have

Original: [Equation]
Revised: [Equation]

Original: which suggests that c
Revised: which suggests that c

Original: = v x v x (cid:48) .
Revised: = v x v x (cid:48) .

Original: In conclustion, κ ( l ) → vv T .
Revised: In conclustion, κ ( l ) → vv T .

Original: B E XPERIMENT D ETAILS
Revised: B C OMPOSING G RAPH SAGE

Revised: As before, X denotes pre-activation rather post activation.

Revised: A GraphSAGE layer has two weight matrices, W 1 and W 2 .

Revised: Hence, we use two variance parameters σ w 1 and σ w 2 for them, respectively.

Revised: For simplicity, we focus on the mean-aggregation version of GraphSAGE, where A is effectively the random-walk matrix.

Original: Dataset preprocessing.
Revised: The updates are: GraphSAGE : [Equation] ] where P = Chol( g ( QQ T )) .

Revised: Note that we do not use a bias term.

Revised: If it is desired, one can easily add it to the updates following the example of GCN.

Revised: C E XPERIMENT D ETAILS

Revised: Datasets.

Revised: The datasets Cora/Citeseer/PubMed/Reddit, with predeﬁned training/validation/test splits, are downloaded from the PyTorch Geometric library (Fey & Lenssen, 2019) and used as is.

Revised: The dataset ArXiv comes from the Open Graph Benchmark (Hu et al., 2020b).

Revised: The datasets Chameleon/Squirrel/Crocodile come from MUSAE (Rozemberczki et al., 2021).

Revised: The training/validation/test splits of the former two sets of datasets come from Geom-GCN (Pei et al., 2020), in accordance with the PyTorch Geometric library.

Revised: The split for Crocodile is not available, so we conduct a random split with the same 0.48/0.32/0.20 proportion as that used for Chameleon and

Revised: Squirrel (Rozemberczki et al., 2021).

Revised: Dataset preprocessing.

Original: For each graph, we treat the edges as undirected and construct a binary, symmetric adjacent matrix A .
Revised: For each graph, we treat the edges as undirected and construct a binary, symmetric adjacent matrix A .

Original: Then, we apply the normalization proposed by Kipf & Welling (2017) to deﬁne A used in (1).
Revised: Then, we apply the normalization proposed by Kipf & Welling (2017) to deﬁne A used in (1).

Original: Speciﬁcally, A
Revised: Speciﬁcally, A

Original: = ( I N + D ) − 1 / 2 ( I + A )
Revised: = ( I N + D ) − 1 / 2 ( I + A )

Original: ( I N + D ) − 1 / 2 , where D = diag { (cid:80)
Revised: ( I N + D ) − 1 / 2 , where D = diag { (cid:80)

Original: j A ij } .
Revised: j A ij } .

Revised: For GraphSAGE and GGP, we instead use the row-normalized version A

Revised: = ( I N +

Revised: D ) − 1 ( I N + A ) .

Original: No feature preprocessing is done except for ArXiv, for which we apply a centering on the input features.
Revised: No feature preprocessing is done except running the GPs on ArXiv, for which we apply a centering on the input features.

Revised: Environment.

Revised: All experiments are conducted on a Nvidia Quadro

Original: Hyperparameters.{+GV
 GPU with 32GB of HBM2 memory. The code is written in Python 3.10.4 as distributed+}
Revised: {+GV
 GPU with 32GB of HBM2 memory. The code is written in Python 3.10.4 as distributed+}

Revised: We

Revised: Training.

Revised: For all datasets except Reddit, we train GCN using the Adam optimizer in full batch for 100 epochs.

Revised: Reddit is too large for GPU computation and hence we conduct mini-batch training (with a batch size 10240) for 10 epochs.

Original:  For classiﬁcation tasks in Table 4, the hyperparameters are set to σ b = 0 . 0 , σ w = 1 . 0 ,and L = 2  .
Revised: Hyperparameters. For classiﬁcation tasks in Table 4, the hyperparameters are set to σ b = 0 . 0 , σ w = 1 . 0 , L = 2 , hidden = 256 , and dropput = 0 .

Original: GCN is trained with learning rate 0 .
Revised: GCN is trained with learning rate 0 .

Original: For regression tasks, they are set to σ b = √ 0 .
Revised: For regression tasks, they are set to σ b = √ 0 .

Original: 1 , σ w = 1 . 0 ,and L = 2  .
Revised: 1 , σ w = 1 . 0 , L = 2 , hidden = 256 , and dropput = 0 .

Original: GCN is trained with learning rate √ 0 .
Revised: GCN is trained with learning rate √ 0 .

Original: We choose C (0) to be the inner product kernel.
Revised: We choose C (0) to be the inner product kernel.

Original: The exception is when working with the low-rank versions, we apply PCA on the input features to ensure the number of features is smaller than the number of landmark nodes.
Revised: The exception is when working with the low-rank versions, we apply PCA on the input features to ensure the number of features is smaller than the number of landmark nodes.

Revised: For Table 5: GCNII uses a 2-layer architecture and sets α = 0 .

Revised: 1 , λ = 0 .

Revised: 5 , β l = log( λl + 1) .

Revised: GIN uses L = 2 and (cid:15) = 0 . 0 .

Original: {+GraphSAGE uses a 2-layer architecture and sets σ w 1 = √ 0 . 1 for Reddit and σ w 1 = 0 . 
 for other datasets, and σ w 2 = 1 . 0 . The nugget (cid:15) is+} chosen using a grid search over [
Revised: {+GraphSAGE uses a 2-layer architecture and sets σ w 1 = √ 0 . 1 for Reddit and σ w 1 = 0 . 
 for other datasets, and σ w 2 = 1 . 0 . The nugget (cid:15) is+} chosen using a grid search over [10 − 3 , 10 1 ] .

Original: For GCNGP-X, GNNGP-X and RBF-X, the landmark points are chosen to be the training set for small datasets, while for large datasets (ArXiv and Reddit)
Revised: For GCNGP-X, GNNGP-X and RBF-X, the landmark points are chosen to be the training set for small datasets, while for large datasets (ArXiv and Reddit)

Original: the landmark points are a random 1/ of the training set. 
Revised: the landmark points are a random 1/ of the training set. Kernels.

Revised: For the RBF kernel,

Revised: [Equation]

Revised: where γ is chosen using a grid search over [10 − 2 , 10 2 ] .

Revised: For the GGP kernel,

Revised: [Equation]

Revised: where c = 5 . 0 , d = 3 . 0 .

Revised: Note that we use only the kernel but not the robust-max likelihood nor the

Revised: ELBO training proposed by Ng et al.

Revised: (2018), because their code written in Python 2 is outdated and cannot be used.

Revised: Note also that GGP-X in Ng et al.

Revised: (2018) has a different meaning—training GGP by using additionally the 500 nodes in the validation set—while we use GGP-X to denote the Nystr¨om approximation of GGP.