Original: S IMPLE S PECTRAL G RAPH C ONVOLUTION
Revised: S IMPLE S PECTRAL G RAPH C ONVOLUTION

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Convolutional Networks (GCNs) are leading methods for learning graph representations.
Revised: Graph Convolutional Networks (GCNs) are leading methods for learning graph representations.

Original: However, without specially designed architectures, the performance of GCNs degrades quickly with increased depth.
Revised: However, without specially designed architectures, the performance of GCNs degrades quickly with increased depth.

Original: As the aggregated neighborhood size and neural network depth are two completely orthogonal aspects of graph representation, several methods focus on summarizing the neighborhood by aggregating K -hop neighborhoods of nodes while using shallow neural networks.
Revised: As the aggregated neighborhood size and neural network depth are two completely orthogonal aspects of graph representation, several methods focus on summarizing the neighborhood by aggregating K -hop neighborhoods of nodes while using shallow neural networks.

Original: However, these methods still encounter oversmoothing, and suffer from high computation and storage costs.
Revised: However, these methods still encounter oversmoothing, and suffer from high computation and storage costs.

Original: In this paper, we use a modiﬁed Markov Diffusion Kernel to derive a variant of GCN called Simple Spectral Graph Convolution (S 2 GC)
Revised: In this paper, we use a modiﬁed Markov Diffusion Kernel to derive a variant of GCN called Simple Spectral Graph Convolution (S 2 GC)

Original: Our spectral analysis shows that our simple spectral graph convolution used in
Revised: Our spectral analysis shows that our simple spectral graph convolution used in

Original: S 2 GC is a trade-off of low- and high-pass ﬁlter bands which capture the global and local contexts of each node.
Revised: S 2 GC is a trade-off of low- and high-pass ﬁlter bands which capture the global and local contexts of each node.

Original: We provide two theoretical claims which demonstrate that we can aggregate over a sequence of increasingly larger neighborhoods compared to competitors while limiting severe oversmoothing.
Revised: We provide two theoretical claims which demonstrate that we can aggregate over a sequence of increasingly larger neighborhoods compared to competitors while limiting severe oversmoothing.

Original: Our experimental evaluations show that S 2 GC with a linear learner is competitive in text and node classiﬁcation tasks.
Revised: Our experimental evaluations show that S 2 GC with a linear learner is competitive in text and node classiﬁcation tasks.

Original: Moreover, S 2 GC is comparable to other state-of-the-art methods for node clustering and community prediction tasks.
Revised: Moreover, S 2 GC is comparable to other state-of-the-art methods for node clustering and community prediction tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the past decade, deep learning has become mainstream in computer vision and machine learning.
Revised: In the past decade, deep learning has become mainstream in computer vision and machine learning.

Original: Although deep learning has been applied for extraction of features on the Euclidean lattice (Euclidean grid-structured data) with great success, the data in many practical scenarios lies on non-Euclidean structures, whose processing poses a challenge for deep learning.
Revised: Although deep learning has been applied for extraction of features on the Euclidean lattice (Euclidean grid-structured data) with great success, the data in many practical scenarios lies on non-Euclidean structures, whose processing poses a challenge for deep learning.

Original: By deﬁning a convolution operator between the graph and signal, Graph Convolutional Networks (GCNs) generalize
Revised: By deﬁning a convolution operator between the graph and signal, Graph Convolutional Networks (GCNs) generalize

Original: Convolutional Neural Networks (CNNs) to graph-structured inputs which contain attributes.
Revised: Convolutional Neural Networks (CNNs) to graph-structured inputs which contain attributes.

Original: Message Passing Neural Networks (MPNNs)
Revised: Message Passing Neural Networks (MPNNs)

Original: (Gilmer et al., 2017) unify the graph convolution as two functions: the transformation function and the aggregation function.
Revised: (Gilmer et al., 2017) unify the graph convolution as two functions: the transformation function and the aggregation function.

Original: MPNN iteratively propagates node features based on the adjacency of the graph in a number of rounds.
Revised: MPNN iteratively propagates node features based on the adjacency of the graph in a number of rounds.

Original: Despite their enormous success in many applications like social media, trafﬁc analysis, biology, recommendation systems and even computer vision, many of the current GCN models use fairly shallow setting as many of the recent models such as GCN (Kipf & Welling, 2016) achieve their best performance given 2 layers.
Revised: Despite their enormous success in many applications like social media, trafﬁc analysis, biology, recommendation systems and even computer vision, many of the current GCN models use fairly shallow setting as many of the recent models such as GCN (Kipf & Welling, 2016) achieve their best performance given 2 layers.

Original: In other words, 2-layer GCN models aggregate nodes in two-hops neighborhood and thus have no ability to extract information in K -hops neighborhoods for K > 2 .
Revised: In other words, 2-layer GCN models aggregate nodes in two-hops neighborhood and thus have no ability to extract information in K -hops neighborhoods for K > 2 .

Original: Moreover, stacking more layers and adding a non-linearity tend to degrade the performance of these models.
Revised: Moreover, stacking more layers and adding a non-linearity tend to degrade the performance of these models.

Original: Such a phenomenon is called oversmoothing (Li et al., 2018a), characterized by the effect that as the number of layers increases, the representations of the nodes in GCNs tend to converge to a similar, non-distinctive from one another value.
Revised: Such a phenomenon is called oversmoothing (Li et al., 2018a), characterized by the effect that as the number of layers increases, the representations of the nodes in GCNs tend to converge to a similar, non-distinctive from one another value.

Original: Even adding residual connections, an effective trick for training very deep CNNs, merely slows down the oversmoothing issue (Kipf & Welling, 2016) in GCNs.
Revised: Even adding residual connections, an effective trick for training very deep CNNs, merely slows down the oversmoothing issue (Kipf & Welling, 2016) in GCNs.

Original: It appears that deep GCN models gain nothing but the performance degradation from the deep architecture.
Revised: It appears that deep GCN models gain nothing but the performance degradation from the deep architecture.

Original: One solution for that is to widen the receptive ﬁeld of aggregation function while limiting the depth of network because the required neighborhood size and neural network depth can be regarded as two separate aspects of design.
Revised: One solution for that is to widen the receptive ﬁeld of aggregation function while limiting the depth of network because the required neighborhood size and neural network depth can be regarded as two separate aspects of design.

Original: To this end, SGC (Wu et al., 2019) captures the context from Khops neighbours in the graph by applying the K -th power of the normalized adjacency matrix in a single layer of neural network.
Revised: To this end, SGC (Wu et al., 2019) captures the context from Khops neighbours in the graph by applying the K -th power of the normalized adjacency matrix in a single layer of neural network.

Original: This scheme is also used for attributed graph clustering (Zhang et al., 2019).
Revised: This scheme is also used for attributed graph clustering (Zhang et al., 2019).

Original: However, SGC also suffers from oversmoothing as K → ∞ , as shown in Theorem 1.
Revised: However, SGC also suffers from oversmoothing as K → ∞ , as shown in Theorem 1.

Original: PPNP and APPNP (Klicpera et al., 2019a) replace the power of the normalized adjacency matrix with the Personalized PageRank matrix to solve the oversmoothing problem.
Revised: PPNP and APPNP (Klicpera et al., 2019a) replace the power of the normalized adjacency matrix with the Personalized PageRank matrix to solve the oversmoothing problem.

Original: Although APPNP relieves the oversmoothing problem, it employs a non-linear operation which requires costly computation of the derivative of the ﬁlter due to the non-linearity over the multiplication of feature matrix with learnable weights.
Revised: Although APPNP relieves the oversmoothing problem, it employs a non-linear operation which requires costly computation of the derivative of the ﬁlter due to the non-linearity over the multiplication of feature matrix with learnable weights.

Original: In contrast, we show that our approach enjoys a free derivative computed in the feed-forward step due to the use of a linear model.
Revised: In contrast, we show that our approach enjoys a free derivative computed in the feed-forward step due to the use of a linear model.

Original: Furthermore, APPNP aggregates over multiple k -hop neighborhoods ( k =0 , · · · , K ) but the weighting scheme favors either global or local context making it difﬁcult if not impossible to ﬁnd a good value of balancing parameter.
Revised: Furthermore, APPNP aggregates over multiple k -hop neighborhoods ( k =0 , · · · , K ) but the weighting scheme favors either global or local context making it difﬁcult if not impossible to ﬁnd a good value of balancing parameter.

Original: In contrast, our approach aggregates over k -hop neighborhoods in a well-balanced manner.
Revised: In contrast, our approach aggregates over k -hop neighborhoods in a well-balanced manner.

Original: GDC (Klicpera et al., 2019b) further extends APPNP by generalizing Personalized PageRank (Page et al., 1999) to an arbitrary graph diffusion process.
Revised: GDC (Klicpera et al., 2019b) further extends APPNP by generalizing Personalized PageRank (Page et al., 1999) to an arbitrary graph diffusion process.

Original: GDC has more expressive power than SGC (Wu et al., 2019), PPNP and APPNP (Klicpera et al., 2019a) but it leads to a dense transition matrix which makes the computation and space storage intractable for large graphs, although authors suggest that the shrinkage method can be used to sparsify the generated transition matrix.
Revised: GDC has more expressive power than SGC (Wu et al., 2019), PPNP and APPNP (Klicpera et al., 2019a) but it leads to a dense transition matrix which makes the computation and space storage intractable for large graphs, although authors suggest that the shrinkage method can be used to sparsify the generated transition matrix.

Original: Noteworthy are also orthogonal research directions of Sun et al.
Revised: Noteworthy are also orthogonal research directions of Sun et al.

Original: ; Koniusz & Zhang (2020); Elinas et al.
Revised: ; Koniusz & Zhang (2020); Elinas et al.

Original: (2020) which improve the performance of GCNs by the perturbation of graph, high-order aggregation of features, and the variational inference, respectively.
Revised: (2020) which improve the performance of GCNs by the perturbation of graph, high-order aggregation of features, and the variational inference, respectively.

Original: To tackle the above issues, we propose a
Revised: To tackle the above issues, we propose a

Original: Simple
Revised: Simple

Original: Spectral Graph Convolution
Revised: Spectral Graph Convolution

Original: S 2 GC) network for node clustering and node classiﬁcation in semi-supervised and unsupervised settings.
Revised: S 2 GC) network for node clustering and node classiﬁcation in semi-supervised and unsupervised settings.

Original: By analyzing the
Revised: By analyzing the

Original: Markov Diffusion Kernel (Fouss et al., 2012)
Revised: Markov Diffusion Kernel (Fouss et al., 2012)

Original: , we obtain a very simple and effective spectral ﬁlter: we aggregate k -step diffusion matrices over k =
Revised: , we obtain a very simple and effective spectral ﬁlter: we aggregate k -step diffusion matrices over k =

Original: K steps, which is equivalent to aggregating over neighborhoods of gradually increasing sizes.
Revised: K steps, which is equivalent to aggregating over neighborhoods of gradually increasing sizes.

Original: Moreover we show that our design incorporates larger neighborhoods compared to SGC and copes better with oversmoothing.
Revised: Moreover we show that our design incorporates larger neighborhoods compared to SGC and copes better with oversmoothing.

Original: We explain that limiting overdominance of the largest neighborhoods in the aggregation step limits oversmoothing while preserving the large context of each node.
Revised: We explain that limiting overdominance of the largest neighborhoods in the aggregation step limits oversmoothing while preserving the large context of each node.

Original: We also show via the spectral analysis that S 2 GC is a trade-off between the low- and high-pass ﬁlter bands which leads to capturing the global and local contexts of each node.
Revised: We also show via the spectral analysis that S 2 GC is a trade-off between the low- and high-pass ﬁlter bands which leads to capturing the global and local contexts of each node.

Original: Moreover, we show how S 2 GC and APPNP (Klicpera et al., 2019a) are related and explain why S 2 GC captures a range of neighborhoods better than
Revised: Moreover, we show how S 2 GC and APPNP (Klicpera et al., 2019a) are related and explain why S 2 GC captures a range of neighborhoods better than

Original: APPNP.
Revised: APPNP.

Original: Our experimental results include node clustering, unsupervised and semi-supervised node classiﬁcation node property prediction and supervised text classiﬁcation.
Revised: Our experimental results include node clustering, unsupervised and semi-supervised node classiﬁcation node property prediction and supervised text classiﬁcation.

Original: We show that S 2 GC is highly competitive, often signiﬁcantly outperforming state-of-the-art methods.
Revised: We show that S 2 GC is highly competitive, often signiﬁcantly outperforming state-of-the-art methods.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Notations.
Revised: Notations.

Original: Let G =
Revised: Let G =

Original: ( V, E ) be a simple and connected undirected graph with n nodes and m edges.
Revised: ( V, E ) be a simple and connected undirected graph with n nodes and m edges.

Original: We use { 1 , · · · , n } to denote the node index of G , whereas d j denotes the degree of node j in G .
Revised: We use { 1 , · · · , n } to denote the node index of G , whereas d j denotes the degree of node j in G .

Original: Let A be the adjacency matrix and D be the diagonal degree matrix.
Revised: Let A be the adjacency matrix and D be the diagonal degree matrix.

Original: Let (cid:101)
Revised: Let (cid:101)

Original: A =
Revised: A =

Original: A + I n denote the adjacency matrix with added self-loops and the corresponding diagonal degree matrix (cid:101) D , where I n ∈ S n ++ is an identity matrix.
Revised: A + I n denote the adjacency matrix with added self-loops and the corresponding diagonal degree matrix (cid:101) D , where I n ∈ S n ++ is an identity matrix.

Original: Finally, let X ∈ R n × d denote the node feature matrix, where each node v is associated with a d -dimensional feature vector X v .
Revised: Finally, let X ∈ R n × d denote the node feature matrix, where each node v is associated with a d -dimensional feature vector X v .

Original: The normalized graph Laplacian matrix is deﬁned as L =
Revised: The normalized graph Laplacian matrix is deﬁned as L =

Original: I n − D − 1 / 2 AD − 1 / 2 ∈ S n + , that is, a symmetric positive semideﬁnite matrix with eigendecomposition UΛU (cid:62) , where Λ is a diagonal matrix with eigenvalues of L , and
Revised: I n − D − 1 / 2 AD − 1 / 2 ∈ S n + , that is, a symmetric positive semideﬁnite matrix with eigendecomposition UΛU (cid:62) , where Λ is a diagonal matrix with eigenvalues of L , and

Original: U ∈ R n × n is a unitary matrix that consists of the eigenvectors of L .
Revised: U ∈ R n × n is a unitary matrix that consists of the eigenvectors of L .

Original: Spectral Graph Convolution (Defferrard et al., 2016).
Revised: Spectral Graph Convolution (Defferrard et al., 2016).

Original: We consider spectral convolutions on graphs deﬁned as the multiplication of signal x ∈ R n with a ﬁlter g θ parameterized by θ ∈ R n in the Fourier domain:
Revised: We consider spectral convolutions on graphs deﬁned as the multiplication of signal x ∈ R n with a ﬁlter g θ parameterized by θ ∈ R n in the Fourier domain:

Original: [Equation]
Revised: [Equation]

Original: where the parameter θ ∈ R n is a vector of spectral ﬁlter coefﬁcients.
Revised: where the parameter θ ∈ R n is a vector of spectral ﬁlter coefﬁcients.

Original: One can understand g θ as a function operating on eigenvalues of L , that is, g ∗ θ ( Λ ) .
Revised: One can understand g θ as a function operating on eigenvalues of L , that is, g ∗ θ ( Λ ) .

Original: To avoid eigendecomposition, g θ ( Λ ) can be approximated by a truncated expansion in terms of Chebyshev polynomials T k ( Λ ) up to the K -th order (Defferrard et al., 2016):
Revised: To avoid eigendecomposition, g θ ( Λ ) can be approximated by a truncated expansion in terms of Chebyshev polynomials T k ( Λ ) up to the K -th order (Defferrard et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: with a rescaled ˜ Λ = 1 2 λ max Λ − I n , where λ max denotes the largest eigenvalue of L and θ ∈ R K is now a vector of Chebyshev coefﬁcients.
Revised: with a rescaled ˜ Λ = 1 2 λ max Λ − I n , where λ max denotes the largest eigenvalue of L and θ ∈ R K is now a vector of Chebyshev coefﬁcients.

Original: Vanila Graph Convolutional Network (GCN) (Kipf & Welling, 2016).
Revised: Vanila Graph Convolutional Network (GCN) (Kipf & Welling, 2016).

Original: The vanilla GCN is a ﬁrst-order approximation of spectral graph convolutions.
Revised: The vanilla GCN is a ﬁrst-order approximation of spectral graph convolutions.

Original: If one sets K = 1 , θ 0 = 2 , and θ 1 = −for Eq.
Revised: If one sets K = 1 , θ 0 = 2 , and θ 1 = −for Eq.

Original: 2, they obtain the convolution operation g θ ( L ) ∗ x
Revised: 2, they obtain the convolution operation g θ ( L ) ∗ x

Original: = ( I + D − 1 / 2 AD − 1 / 2 ) x .
Revised: = ( I + D − 1 / 2 AD − 1 / 2 ) x .

Original: Finally, by the renormalization trick, replacing matrix
Revised: Finally, by the renormalization trick, replacing matrix

Original: I + D − 1 / 2 AD − 1 / 2 by a normalized version (cid:101) T = (cid:101) D −
Revised: I + D − 1 / 2 AD − 1 / 2 by a normalized version (cid:101) T = (cid:101) D −

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: (cid:101) A (cid:101)
Revised: (cid:101) A (cid:101)

Original: [Equation]
Revised: [Equation]

Original: D
Revised: D

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: 1 / 2 leads to the GCN layer with a non-linear function σ :
Revised: 1 / 2 leads to the GCN layer with a non-linear function σ :

Original: [Equation]
Revised: [Equation]

Original: Graph Diffusion Convolution (GDC) (Klicpera et al., 2019b).
Revised: Graph Diffusion Convolution (GDC) (Klicpera et al., 2019b).

Original: A generalized graph diffusion is given by the diffusion matrix:
Revised: A generalized graph diffusion is given by the diffusion matrix:

Original: [Equation]
Revised: [Equation]

Original: with the weighting coefﬁcients θ k and the generalized transition matrix T .
Revised: with the weighting coefﬁcients θ k and the generalized transition matrix T .

Original: Eq.
Revised: Eq.

Original: 4 can be regarded as related to the Taylor expansion of matrix-valued functions.
Revised: 4 can be regarded as related to the Taylor expansion of matrix-valued functions.

Original: Thus, the choice of θ k and T k must at least ensure that Eq.
Revised: Thus, the choice of θ k and T k must at least ensure that Eq.

Original: 4 converges.
Revised: 4 converges.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019b) provide two special cases as low-pass ﬁlters ie .
Revised: (2019b) provide two special cases as low-pass ﬁlters ie .

Original: , the heat kernel and the kernel based on random walk with restarts.
Revised: , the heat kernel and the kernel based on random walk with restarts.

Original: If S denotes the adjacency matrix and D is the diagonal degree matrix of S , the corresponding graph diffusion convolution is then deﬁned as D − 1 / 2 SD − 1 / 2 x .
Revised: If S denotes the adjacency matrix and D is the diagonal degree matrix of S , the corresponding graph diffusion convolution is then deﬁned as D − 1 / 2 SD − 1 / 2 x .

Original: Note that θ k can be a learnable parameter, or it can be chosen in some other way.
Revised: Note that θ k can be a learnable parameter, or it can be chosen in some other way.

Original: Many works use the expansion in Eq.
Revised: Many works use the expansion in Eq.

Original: 4 but different choices of θ k realise very different ﬁlters, making each method unique.
Revised: 4 but different choices of θ k realise very different ﬁlters, making each method unique.

Original: Simple Graph Convolution (SGC) (Wu et al., 2019).
Revised: Simple Graph Convolution (SGC) (Wu et al., 2019).

Original: A classical MPNN (Gilmer et al., 2017) averages (in each layer) the hidden representations among 1 -hop neighbors.
Revised: A classical MPNN (Gilmer et al., 2017) averages (in each layer) the hidden representations among 1 -hop neighbors.

Original: This implies that each node in the K -th layer obtains feature information from all nodes that are K -hops away in the graph.
Revised: This implies that each node in the K -th layer obtains feature information from all nodes that are K -hops away in the graph.

Original: By hypothesizing that the non-linearity between GCN layers is not critical, SGC captures information from K -hops neighborhood in the graph by applying the K -th power of the transition matrix in a single neural network layer.
Revised: By hypothesizing that the non-linearity between GCN layers is not critical, SGC captures information from K -hops neighborhood in the graph by applying the K -th power of the transition matrix in a single neural network layer.

Original: The SGC can be regarded as a special case of GDC without non-linearity and without the normalization by D − 1 / 2 if we set θ K = 1 and θ i<K = 0 in Eq. 4, and
Revised: The SGC can be regarded as a special case of GDC without non-linearity and without the normalization by D − 1 / 2 if we set θ K = 1 and θ i<K = 0 in Eq. 4, and

Original: T = ˜ T , which yields:
Revised: T = ˜ T , which yields:

Original: [Equation]
Revised: [Equation]

Original: Although SGC is an efﬁcient and effective method, increasing K leads to oversmoothing.
Revised: Although SGC is an efﬁcient and effective method, increasing K leads to oversmoothing.

Original: Thus,
Revised: Thus,

Original: SGC uses a small K number of layers.
Revised: SGC uses a small K number of layers.

Original: Theorem 1 shows that oversmoothing is a result of convergence to the stationary distribution in the graph diffusion process when time t → ∞ .
Revised: Theorem 1 shows that oversmoothing is a result of convergence to the stationary distribution in the graph diffusion process when time t → ∞ .

Original: Theorem 1.
Revised: Theorem 1.

Original: (Chung & Graham, 1997) Let λ 2 denote the second largest eigenvalue of transition matrix (cid:101) T
Revised: (Chung & Graham, 1997) Let λ 2 denote the second largest eigenvalue of transition matrix (cid:101) T

Original: =
Revised: =

Original: D − 1 A of a non-bipartite graph, p ( t ) be the probability distribution vector and π the stationary distribution.
Revised: D − 1 A of a non-bipartite graph, p ( t ) be the probability distribution vector and π the stationary distribution.

Original: If walk starts from the vertex i , p i (0) = 1 , then after t steps for every vertex, we have:
Revised: If walk starts from the vertex i , p i (0) = 1 , then after t steps for every vertex, we have:

Original: [Equation]
Revised: [Equation]

Original: APPNP.
Revised: APPNP.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019a) proposed to use the Personalized PageRank to derive a ﬁxed ﬁlter of order K .
Revised: (2019a) proposed to use the Personalized PageRank to derive a ﬁxed ﬁlter of order K .

Original: Let f θ ( X ) denote the output of a two-layer fully connected neural network on the feature matrix X , then the PPNP model is deﬁned as H =
Revised: Let f θ ( X ) denote the output of a two-layer fully connected neural network on the feature matrix X , then the PPNP model is deﬁned as H =

Original: α I n − (1 − α ) (cid:101) T − 1 f θ ( X ) .
Revised: α I n − (1 − α ) (cid:101) T − 1 f θ ( X ) .

Original: To avoid calculating the inverse of matrix (cid:101) T , Klicpera et al. (2019a) also propose the Approximate PPNP (APPNP), which replaces the costly inverse with an approximation by the truncated power iteration:
Revised: To avoid calculating the inverse of matrix (cid:101) T , Klicpera et al. (2019a) also propose the Approximate PPNP (APPNP), which replaces the costly inverse with an approximation by the truncated power iteration:

Original: [Equation]
Revised: [Equation]

Original: where H (0) =
Revised: where H (0) =

Original: f θ ( X )=
Revised: f θ ( X )=

Original: ReLU ( XW ) or H (0) =
Revised: ReLU ( XW ) or H (0) =

Original: f θ ( X )=
Revised: f θ ( X )=

Original: MLP ( X ) .
Revised: MLP ( X ) .

Original: By decoupling feature transformation and propagation steps, PPNP and APPNP aggregate information from multi-hop neighbors.
Revised: By decoupling feature transformation and propagation steps, PPNP and APPNP aggregate information from multi-hop neighbors.

Original: [Figure]
Revised: [Figure]

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: Below, we ﬁrstly outline two claims which underlie the design of our network, with the goal of mitigating oversmoothing.
Revised: Below, we ﬁrstly outline two claims which underlie the design of our network, with the goal of mitigating oversmoothing.

Original: Moreover, we analyze the Markov Diffusion Kernel (Fouss et al., 2012) and note that it acts as a low-pass spectral ﬁlter of various degree.
Revised: Moreover, we analyze the Markov Diffusion Kernel (Fouss et al., 2012) and note that it acts as a low-pass spectral ﬁlter of various degree.

Original: Based on the feature mapping function underlying this kernel, we present our Simple Spectral Graph Convolution network and discuss its relation with other models.
Revised: Based on the feature mapping function underlying this kernel, we present our Simple Spectral Graph Convolution network and discuss its relation with other models.

Original: Finally, we provide the comparison of computational and storage complexity requirements.
Revised: Finally, we provide the comparison of computational and storage complexity requirements.

Original: 3. M OTIVATION
Revised: 3. M OTIVATION

Original: Our design follows Claims I and II described in Section A.3, which includes their detailed proofs.
Revised: Our design follows Claims I and II described in Section A.3, which includes their detailed proofs.

Original: Claim I. By design, our ﬁlter gives the highest weight to the closest neighborhood of a node, as neighborhoods N of diffusion steps k = 0 , · · · , K obey N ( (cid:101) T 0 ) ⊆
Revised: Claim I. By design, our ﬁlter gives the highest weight to the closest neighborhood of a node, as neighborhoods N of diffusion steps k = 0 , · · · , K obey N ( (cid:101) T 0 ) ⊆

Original: N ( (cid:101) T 1 ) ⊆ · · · ⊆ N ( (cid:101) T K ) ⊆
Revised: N ( (cid:101) T 1 ) ⊆ · · · ⊆ N ( (cid:101) T K ) ⊆

Original: N ( (cid:101) T ∞ ) .
Revised: N ( (cid:101) T ∞ ) .

Original: That is, smaller neighborhoods belong to larger neighborhoods too.
Revised: That is, smaller neighborhoods belong to larger neighborhoods too.

Original: Claim II.
Revised: Claim II.

Original: As K → ∞ , the ratio of energies contributed by S 2 GC to SGC is 0.
Revised: As K → ∞ , the ratio of energies contributed by S 2 GC to SGC is 0.

Original: Thus, the energy of inﬁnite-dimensional receptive ﬁeld (largest k ) will not dominate the sum energy of our ﬁlter.
Revised: Thus, the energy of inﬁnite-dimensional receptive ﬁeld (largest k ) will not dominate the sum energy of our ﬁlter.

Original: Thus,
Revised: Thus,

Original: S 2 GC can incorporate larger receptive ﬁelds without undermining contributions of smaller receptive ﬁelds.
Revised: S 2 GC can incorporate larger receptive ﬁelds without undermining contributions of smaller receptive ﬁelds.

Original: This is substantiated by Table 8, where S 2 GC achieves the best results for K = 16 , whereas
Revised: This is substantiated by Table 8, where S 2 GC achieves the best results for K = 16 , whereas

Original: SGC achieves poorer results by comparison, whose peak is at K =4 (note that larger K is better).
Revised: SGC achieves poorer results by comparison, whose peak is at K =4 (note that larger K is better).

Original: 3. M ARKOV D IFFUSION K ERNEL
Revised: 3. M ARKOV D IFFUSION K ERNEL

Original: Two nodes are considered similar when they are diffused in a similar way through the graph, as then they inﬂuence the other nodes in a similar manner (Fouss et al., 2012).
Revised: Two nodes are considered similar when they are diffused in a similar way through the graph, as then they inﬂuence the other nodes in a similar manner (Fouss et al., 2012).

Original: Moreover, two nodes are close neighbors if they are in the same distinct cluster.
Revised: Moreover, two nodes are close neighbors if they are in the same distinct cluster.

Original: The Markov Diffusion distance between nodes i and j at time K is deﬁned as:
Revised: The Markov Diffusion distance between nodes i and j at time K is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where the average visiting rate x i ( K ) after K steps for a process that started at time k = 0 is computed as follows:
Revised: where the average visiting rate x i ( K ) after K steps for a process that started at time k = 0 is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: By deﬁning Z ( K ) = 1 K (cid:80)
Revised: By deﬁning Z ( K ) = 1 K (cid:80)

Original: Kk =1 T k , we reformulate Eq.
Revised: Kk =1 T k , we reformulate Eq.

Original: 8 as the following metric:
Revised: 8 as the following metric:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: The underlying feature map of Markov Diffusion Kernel (MDK) is given as Z ( K ) x
Revised: The underlying feature map of Markov Diffusion Kernel (MDK) is given as Z ( K ) x

Original: i (0) for node i .
Revised: i (0) for node i .

Original: The effect of the linear projection Z ( K ) (ﬁlter) acting on spectrum as f ( λ ) = 1 K (cid:80)
Revised: The effect of the linear projection Z ( K ) (ﬁlter) acting on spectrum as f ( λ ) = 1 K (cid:80)

Original: Kk =

Revised: Kk =

Original: λ k
Revised: λ k

Original: (we sum from 0 to include self-loops) is plotted in Figure 1, from which we observe the following properties: (i) Z ( K ) preserves leading (large) eigenvalues of T and (ii) the higher K is the stricter the low-pass ﬁlter becomes but the ﬁlter also preserves the high frequency.
Revised: (we sum from 0 to include self-loops) is plotted in Figure 1, from which we observe the following properties: (i) Z ( K ) preserves leading (large) eigenvalues of T and (ii) the higher K is the stricter the low-pass ﬁlter becomes but the ﬁlter also preserves the high frequency.

Original: In other words, as K grows, this ﬁlter includes larger and larger neighborhood but also maintains the closest locality of nodes.
Revised: In other words, as K grows, this ﬁlter includes larger and larger neighborhood but also maintains the closest locality of nodes.

Original: Note that
Revised: Note that

Original: L = I − T , where L is the normalized Laplacian matrix and T is the normalized adjacency matrix.
Revised: L = I − T , where L is the normalized Laplacian matrix and T is the normalized adjacency matrix.

Original: Thus, keeping large positive eigenvalues for T equals keeping small eigenvalues for L .
Revised: Thus, keeping large positive eigenvalues for T equals keeping small eigenvalues for L .

Original: 3. S IMPLE S PECTRAL G RAPH C ONVOLUTION
Revised: 3. S IMPLE S PECTRAL G RAPH C ONVOLUTION

Original: Based on the aforementioned Markov Diffusion Kernel, we include self-loops and propose the Simple Spectral Graph Convolution (S 2 GC) network with the softmax classiﬁer after the linear layer:
Revised: Based on the aforementioned Markov Diffusion Kernel, we include self-loops and propose the Simple Spectral Graph Convolution (S 2 GC) network with the softmax classiﬁer after the linear layer:

Original: [Equation]
Revised: [Equation]

Original: Let (cid:107) x
Revised: Let (cid:107) x

Original: i (cid:107) 2 = 1 , ∀ i (each x
Revised: i (cid:107) 2 = 1 , ∀ i (each x

Original: i is a row in X ).
Revised: i is a row in X ).

Original: If K → ∞ then H = (cid:80) ∞ k =0 (cid:101)
Revised: If K → ∞ then H = (cid:80) ∞ k =0 (cid:101)

Original: T k X is the optimal diffused representation of the normalized Laplacian Regularization problem given below:
Revised: T k X is the optimal diffused representation of the normalized Laplacian Regularization problem given below:

Original: [Equation]
Revised: [Equation]

Original: q ( H ) , where q[-( H )= 
-]
Revised: q ( H ) , where q[-( H )= 
-]

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: H

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: (12) and each vector h i denotes the i -th row of H .
Revised: (12) and each vector h i denotes the i -th row of H .

Original: Compared with the more common form in (Zhou et al., 2004), we impose (cid:107) h
Revised: Compared with the more common form in (Zhou et al., 2004), we impose (cid:107) h

Original: i (cid:107) 

Revised: i (cid:107)

Original: = (cid:107) x
Revised: = (cid:107) x

Original: i (cid:107) 22 = 1 , which tend to minimize the difference between h i and x i with cosine distance (a special case of the Euclidean distance).
Revised: i (cid:107) 22 = 1 , which tend to minimize the difference between h i and x i with cosine distance (a special case of the Euclidean distance).

Original: Differentiating q ( H ) with respect to H , we have (cid:101) LH − X = 0 .
Revised: Differentiating q ( H ) with respect to H , we have (cid:101) LH − X = 0 .

Original: Thus, the optimal representation H =
Revised: Thus, the optimal representation H =

Original: ( I − (cid:101) T ) − 1 X , where ( I − (cid:101) T ) − 1 = (cid:80) ∞
Revised: ( I − (cid:101) T ) − 1 X , where ( I − (cid:101) T ) − 1 = (cid:80) ∞

Original: k
Revised: k

Original: =0 (cid:101) T k .
Revised: =0 (cid:101) T k .

Original: However, the inﬁnite expansion resulting from Eq.
Revised: However, the inﬁnite expansion resulting from Eq.

Original: 12 is in fact suboptimal due to oversmoothing.
Revised: 12 is in fact suboptimal due to oversmoothing.

Original: Thus, we include in Eq.
Revised: Thus, we include in Eq.

Original: 11 a self-loop (cid:101) T 

Revised: 11 a self-loop (cid:101) T

Original: =
Revised: =

Original: I , the α ∈ [0 , 1] parameter (Table 9 evaluates its impact) to balance the self-information of node vs .
Revised: I , the α ∈ [0 , 1] parameter (Table 9 evaluates its impact) to balance the self-information of node vs .

Original: consecutive neighborhoods, and we consider ﬁnite K .
Revised: consecutive neighborhoods, and we consider ﬁnite K .

Original: We generalize the Eq.
Revised: We generalize the Eq.

Original: 11 as:
Revised: 11 as:

Original: [Equation]
Revised: [Equation]

Original: Relation of S 2 GC to GDC.
Revised: Relation of S 2 GC to GDC.

Original: GDC uses the entire ﬁlter matrix S of size n × n as S is re-normalized numerous times by its degree.
Revised: GDC uses the entire ﬁlter matrix S of size n × n as S is re-normalized numerous times by its degree.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019b) explain that ‘most graph diffusions result in a dense matrix S ’.
Revised: (2019b) explain that ‘most graph diffusions result in a dense matrix S ’.

Original: In contrast, our approach is simply computed as ( (cid:80)
Revised: In contrast, our approach is simply computed as ( (cid:80)

Original: Kk =1 (cid:101) T k X ) W (plus the self-loop), where X is of size n × d , and d (cid:28) n , where n and d are the number of nodes and features, respectively.
Revised: Kk =1 (cid:101) T k X ) W (plus the self-loop), where X is of size n × d , and d (cid:28) n , where n and d are the number of nodes and features, respectively.

Original: The (cid:101) T k X step is computed as (cid:101) T · ( (cid:101) T · ( · · · ( (cid:101) TX ) · · · )) , which requires K sparse matrix-matrix multiplications between a sparse matrices of size n × n and a dense matrix of size n × d .
Revised: The (cid:101) T k X step is computed as (cid:101) T · ( (cid:101) T · ( · · · ( (cid:101) TX ) · · · )) , which requires K sparse matrix-matrix multiplications between a sparse matrices of size n × n and a dense matrix of size n × d .

Original: Thus, S 2 GC can handle extremely large graphs as S 2 GC does not need to sparsify dense ﬁlter matrices (in contrast to GDC).
Revised: Thus, S 2 GC can handle extremely large graphs as S 2 GC does not need to sparsify dense ﬁlter matrices (in contrast to GDC).

Original: Relation of S 2 GC to APPNP.
Revised: Relation of S 2 GC to APPNP.

Original: Let H 

Revised: Let H

Original: =
Revised: =

Original: XW as we use the linear step in our S 2 GC.
Revised: XW as we use the linear step in our S 2 GC.

Original: Then and only then, for l = 0 and H 0 = XW , APPNP expansion yields H 1 =
Revised: Then and only then, for l = 0 and H 0 = XW , APPNP expansion yields H 1 =

Original: (1 − α )
Revised: (1 − α )

Original: (cid:101) TXW + α XW =
Revised: (cid:101) TXW + α XW =

Original: ((1 − α ) (cid:101) T + α I ) XW , which is equal to our Z (1) XW
Revised: ((1 − α ) (cid:101) T + α I ) XW , which is equal to our Z (1) XW

Original: =
Revised: =

Original: ( (cid:80) Kk =

Revised: ( (cid:80) Kk =

Original: (cid:101) T k ) XW
Revised: (cid:101) T k ) XW

Original: = (cid:101) TX + X = ( (cid:101) T +
Revised: = (cid:101) TX + X = ( (cid:101) T +

Original: I ) XW if α = 0 .
Revised: I ) XW if α = 0 .

Original: 5 , K =1 , except for scaling (constant) of H 1 .
Revised: 5 , K =1 , except for scaling (constant) of H 1 .

Original: In contrast, for l = 1 and general case H 

Revised: In contrast, for l = 1 and general case H

Original: = f
Revised: = f

Original: ( X ; W ) , APPNP yields H 2 =
Revised: ( X ; W ) , APPNP yields H 2 =

Original: (1 − α ) 2 (cid:101) T 2 f
Revised: (1 − α ) 2 (cid:101) T 2 f

Original: ( X ; W )
Revised: ( X ; W )

Original: + (1 − α )
Revised: + (1 − α )

Original: α (cid:101) T f ( X ; W )+ αf
Revised: α (cid:101) T f ( X ; W )+ αf

Original: ( X ; W ) from which it is easy to note speciﬁc weight coefﬁcients (1 − α ) 2 , (1 − α )
Revised: ( X ; W ) from which it is easy to note speciﬁc weight coefﬁcients (1 − α ) 2 , (1 − α )

Original: α and α associated with 2 -, 1 -, and 0 -hops.
Revised: α and α associated with 2 -, 1 -, and 0 -hops.

Original: This shows that the APPNP expansion is very different to the S 2 GC expansion in Eq.
Revised: This shows that the APPNP expansion is very different to the S 2 GC expansion in Eq.

Original: In fact, S 2 GC and APPNP are only equivalent if α = 0 .
Revised: In fact, S 2 GC and APPNP are only equivalent if α = 0 .

Original: K = 1 and f is the linear transformation.
Revised: K = 1 and f is the linear transformation.

Original: [Table]
Revised: [Table]

Original: GDC
Revised: GDC

Original: SGC
Revised: SGC

Original: S 2 GC
Revised: S 2 GC

Original: [Table]
Revised: [Table]

Original: K | E | d + Knd ≈ K | E | n
Revised: K | E | d + Knd ≈ K | E | n

Original: K | E | d
Revised: K | E | d

Original: K | E | d + Knd
Revised: K | E | d + Knd

Original: [Table]
Revised: [Table]

Original: nd + | E |
Revised: nd + | E |

Original: ≈ nnd + | E | nd + | E |
Revised: ≈ nnd + | E | nd + | E |

Original: [Table]
Revised: [Table]

Original: K | E | d
Revised: K | E | d

Original: [Table]
Revised: [Table]

Original: nd + | E |
Revised: nd + | E |

Original: Moreover, as APPNP assumes H 

Revised: Moreover, as APPNP assumes H

Original: = f
Revised: = f

Original: ( X ; W ) =
Revised: ( X ; W ) =

Original: ReLU ( XW ) (or MLP in place of ReLU), their optimizer has to backpropagate through f ( X ; W ) to obtain ∂f∂ W and multiply this result with the above expansion e.g .,
Revised: ReLU ( XW ) (or MLP in place of ReLU), their optimizer has to backpropagate through f ( X ; W ) to obtain ∂f∂ W and multiply this result with the above expansion e.g .,

Original: [Equation]
Revised: [Equation]

Original: In contrast, we use the linear function XW .
Revised: In contrast, we use the linear function XW .

Original: Thus, ∂ XW ∂ W yields X .
Revised: Thus, ∂ XW ∂ W yields X .

Original: Thus, the multiplication of our expansion with X for the backpropagation step is in fact obtained in the forward pass which makes our approach very fast for large graphs.
Revised: Thus, the multiplication of our expansion with X for the backpropagation step is in fact obtained in the forward pass which makes our approach very fast for large graphs.

Original: Relation of S 2 GC to AR.
Revised: Relation of S 2 GC to AR.

Original: The AR ﬁlter (Li et al., 2019) uses the regularized Laplacian kernel (Smola & Kondor, 2003) which differs from the (modiﬁed) Markov Diffusion Kernel used by us.
Revised: The AR ﬁlter (Li et al., 2019) uses the regularized Laplacian kernel (Smola & Kondor, 2003) which differs from the (modiﬁed) Markov Diffusion Kernel used by us.

Original: Speciﬁcally, the regularized Laplacian kernel uses the negated Laplacian matrix − L yielding K L = (cid:80) ∞ k =

Revised: Speciﬁcally, the regularized Laplacian kernel uses the negated Laplacian matrix − L yielding K L = (cid:80) ∞ k =

Original: α k ( − L ) k
Revised: α k ( − L ) k

Original: = ( I + α L ) − 1 , where L = I − (cid:101) T , which is related to the von Neumann diffusion kernel K vN = (cid:80) ∞
Revised: = ( I + α L ) − 1 , where L = I − (cid:101) T , which is related to the von Neumann diffusion kernel K vN = (cid:80) ∞

Original: k
Revised: k

Original: =

Revised: =

Original: α k A k .
Revised: α k A k .

Original: In contrast, the Markov Diffusion Kernel is deﬁned as K MD ( K ) =
Revised: In contrast, the Markov Diffusion Kernel is deﬁned as K MD ( K ) =

Original: Z ( K ) Z T ( K ) , where Z ( K )=
Revised: Z ( K ) Z T ( K ) , where Z ( K )=

Original: 1 K (cid:80)
Revised: 1 K (cid:80)

Original: Kk =1 (cid:101) T k and (cid:101) T
Revised: Kk =1 (cid:101) T k and (cid:101) T

Original: =
Revised: =

Original: D − 1 / 2 AD − 1 / 2 .
Revised: D − 1 / 2 AD − 1 / 2 .

Original: Relation of S 2 GC to Jumping Knowledge Network (JKN).
Revised: Relation of S 2 GC to Jumping Knowledge Network (JKN).

Original: Xu et al.
Revised: Xu et al.

Original: (2018b) combine intermediate node representations from each layer by concatenating them in the ﬁnal layer.
Revised: (2018b) combine intermediate node representations from each layer by concatenating them in the ﬁnal layer.

Original: However, (Xu et al., 2018b) use non-linear layers, which results in a completely different network architecture and the usual slower processing time due to the complex backpropagation chain.
Revised: However, (Xu et al., 2018b) use non-linear layers, which results in a completely different network architecture and the usual slower processing time due to the complex backpropagation chain.

Original: 3. C OMPLEXITY A NALYSIS
Revised: 3. C OMPLEXITY A NALYSIS

Original: For S 2 GC, the storage costs is O ( | E | + nd ) , where | E | is the total edge count, nd relates to saving the (cid:101)
Revised: For S 2 GC, the storage costs is O ( | E | + nd ) , where | E | is the total edge count, nd relates to saving the (cid:101)

Original: T k X during intermediate multiplications (cid:101) T · ( (cid:101) T · ( · · · ( (cid:101) TX ) · · · )) .
Revised: T k X during intermediate multiplications (cid:101) T · ( (cid:101) T · ( · · · ( (cid:101) TX ) · · · )) .

Original: The computational cost is
Revised: The computational cost is

Original: O ( K | E | d + Knd ) .
Revised: O ( K | E | d + Knd ) .

Original: Each sparse matrix-matrix multiplication (cid:101) TX costs | E | d .
Revised: Each sparse matrix-matrix multiplication (cid:101) TX costs | E | d .

Original: We need K such multiplications, where Knd and nd are costs of summation over ﬁlters and adding features X .
Revised: We need K such multiplications, where Knd and nd are costs of summation over ﬁlters and adding features X .

Original: In contrast, the storage cost of GDC is approximately O ( n 2 ) , and the computational cost is approximately O
Revised: In contrast, the storage cost of GDC is approximately O ( n 2 ) , and the computational cost is approximately O

Original: ( K | E | n ) , where n is the node numbers, K is the order of terms and | E | is the number of graph edges.
Revised: ( K | E | n ) , where n is the node numbers, K is the order of terms and | E | is the number of graph edges.

Original: APPNP, SGC and S 2 GC have much lower cost than GDC.
Revised: APPNP, SGC and S 2 GC have much lower cost than GDC.

Original: Note that K | E | d (cid:29) Knd and n (cid:29) d .
Revised: Note that K | E | d (cid:29) Knd and n (cid:29) d .

Original: We found that APPNP, SGC and S 2 GC have similar computational and storage costs in the forward stage.
Revised: We found that APPNP, SGC and S 2 GC have similar computational and storage costs in the forward stage.

Original: We note that symbol d in APPNP is not the dimension of features X but dimension of f ( X ) , which is the number of categories.
Revised: We note that symbol d in APPNP is not the dimension of features X but dimension of f ( X ) , which is the number of categories.

Original: For the backward stage including computations of the gradient of the classiﬁcation step, the computational costs of GDC, SGC and S 2 GC are independent of K and | E | because the graph convolution for these methods does not require backpropagation (the gradients is computed in the forward step).
Revised: For the backward stage including computations of the gradient of the classiﬁcation step, the computational costs of GDC, SGC and S 2 GC are independent of K and | E | because the graph convolution for these methods does not require backpropagation (the gradients is computed in the forward step).

Original: In contrast, APPNP requires backprop as explained earlier.
Revised: In contrast, APPNP requires backprop as explained earlier.

Original: Table 1 summarizes the computational and storage costs of several methods.
Revised: Table 1 summarizes the computational and storage costs of several methods.

Original: Table 2 demonstrates that APPNP is over 66 × slower than S 2 GC on the large scale Products dataset (OGB benchmark) despite, for fairness, we use the same basic building blocks of PyTorch among compared methods.
Revised: Table 2 demonstrates that APPNP is over 66 × slower than S 2 GC on the large scale Products dataset (OGB benchmark) despite, for fairness, we use the same basic building blocks of PyTorch among compared methods.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the proposed method on four different tasks: node clustering, community prediction, semi-supervised node classiﬁcation and text classiﬁcation.
Revised: In this section, we evaluate the proposed method on four different tasks: node clustering, community prediction, semi-supervised node classiﬁcation and text classiﬁcation.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: 4. N ODE C LUSTERING
Revised: 4. N ODE C LUSTERING

Original: We compare S 2 GC with three variants of clustering: (i) Methods that only use node features ie ., k-means and spectral clustering (spectral-f) that construct a similarity matrix with the node features by a linear kernel.
Revised: We compare S 2 GC with three variants of clustering: (i) Methods that only use node features ie ., k-means and spectral clustering (spectral-f) that construct a similarity matrix with the node features by a linear kernel.

Original: (ii) Structural clustering methods that only use graph structures ie ., spectral clustering (spectral-g) that takes the node adjacency matrix as the similarity matrix, DeepWalk (Perozzi et al., 2014), and (iii) Attributed graph clustering methods that utilize both node features and graph structures: Graph Autoencoder (GAE) and Graph Variational Autoencoder (VGAE) (Kipf & Welling, 2016), and Adversarially Regularized Graph Autoencoder (ARGE), Variational Graph
Revised: (ii) Structural clustering methods that only use graph structures ie ., spectral clustering (spectral-g) that takes the node adjacency matrix as the similarity matrix, DeepWalk (Perozzi et al., 2014), and (iii) Attributed graph clustering methods that utilize both node features and graph structures: Graph Autoencoder (GAE) and Graph Variational Autoencoder (VGAE) (Kipf & Welling, 2016), and Adversarially Regularized Graph Autoencoder (ARGE), Variational Graph

Original: Autoencoder (ARVGE) (Pan et al., 2018) and AGC (Zhang et al., 2019).
Revised: Autoencoder (ARVGE) (Pan et al., 2018) and AGC (Zhang et al., 2019).

Original: To evaluate the clustering performance, three performance measures are adopted: clustering Accuracy (Acc), Normalized Mutual Information (NMI) and macro F1-score (F1).
Revised: To evaluate the clustering performance, three performance measures are adopted: clustering Accuracy (Acc), Normalized Mutual Information (NMI) and macro F1-score (F1).

Original: We run each method 10 times on four datasets:
Revised: We run each method 10 times on four datasets:

Original: Cora, CiteSeer, PubMed, and Wiki, and we report the average clustering results in Table 3, where top-1 results are highlighted in bold.
Revised: Cora, CiteSeer, PubMed, and Wiki, and we report the average clustering results in Table 3, where top-1 results are highlighted in bold.

Original: To adaptively select the order K , we use the clustering performance metric: internal criteria based on the information intrinsic to the data alone Zhang et al.
Revised: To adaptively select the order K , we use the clustering performance metric: internal criteria based on the information intrinsic to the data alone Zhang et al.

Original: 4. C OMMUNITY P REDICTION
Revised: 4. C OMMUNITY P REDICTION

Original: We supplement our social network analysis by using S 2 GC to inductively predict the community structure on Reddit, a large scale dataset, as shown in Table 10, which cannot be processed by the vanilla GCN Kipf & Welling (2016) and GDC (Klicpera et al., 2019b)
Revised: We supplement our social network analysis by using S 2 GC to inductively predict the community structure on Reddit, a large scale dataset, as shown in Table 10, which cannot be processed by the vanilla GCN Kipf & Welling (2016) and GDC (Klicpera et al., 2019b)

Original: due to the memory issues.
Revised: due to the memory issues.

Original: On the Reddit dataset, we train S 2 GC with L-BFGS using no regularization, and we set K = 5 and α = 0 .
Revised: On the Reddit dataset, we train S 2 GC with L-BFGS using no regularization, and we set K = 5 and α = 0 .

Original: We evaluate S 2 GC inductively according to protocol (Chen et al., 2018).
Revised: We evaluate S 2 GC inductively according to protocol (Chen et al., 2018).

Original: We train S 2 GC on a subgraph comprising only training nodes and test on the original graph.
Revised: We train S 2 GC on a subgraph comprising only training nodes and test on the original graph.

Original: On all datasets, we tune the number of epochs based on both the convergence behavior and the obtained validation accuracy.
Revised: On all datasets, we tune the number of epochs based on both the convergence behavior and the obtained validation accuracy.

Original: For Reddit, we compare S 2 GC to the reported performance of supervised and unsupervised variants of GraphSAGE (Hamilton et al., 2017), FastGCN (Chen et al., 2018), SGC (Wu et al., 2019) and
Revised: For Reddit, we compare S 2 GC to the reported performance of supervised and unsupervised variants of GraphSAGE (Hamilton et al., 2017), FastGCN (Chen et al., 2018), SGC (Wu et al., 2019) and

Original: DGI (Velickovic et al., 2019).
Revised: DGI (Velickovic et al., 2019).

Original: Table 4 also highlights the setting of the feature extraction step for each method.
Revised: Table 4 also highlights the setting of the feature extraction step for each method.

Original: Note that S 2 GC and SGC involve no learning because they do not learn any parameters to extract features.
Revised: Note that S 2 GC and SGC involve no learning because they do not learn any parameters to extract features.

Original: The logistic regression is used as the classiﬁer for both unsupervised and no-learning approaches to train with labels afterward.
Revised: The logistic regression is used as the classiﬁer for both unsupervised and no-learning approaches to train with labels afterward.

Original: 4. N ODE C LASSIFICATION
Revised: 4. N ODE C LASSIFICATION

Original: For the semi-supervised node classiﬁcation task, we apply the standard ﬁxed training, validation and testing splits (Yang et al., 2016) on the Cora, Citeseer, and Pubmed datasets, with 20 nodes per class for training, 500 nodes for validation and 1,000 nodes for testing.
Revised: For the semi-supervised node classiﬁcation task, we apply the standard ﬁxed training, validation and testing splits (Yang et al., 2016) on the Cora, Citeseer, and Pubmed datasets, with 20 nodes per class for training, 500 nodes for validation and 1,000 nodes for testing.

Original: For baselines, We include
Revised: For baselines, We include

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: three state-of-the-art shallow models: GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´
Revised: three state-of-the-art shallow models: GCN (Kipf & Welling, 2016), GAT (Veli ˇ ckovi ´

Original: c et al., 2017),
Revised: c et al., 2017),

Original: FastGCN
Revised: FastGCN

Original: (Chen et al., 2018), APPNP (Klicpera et al., 2019a), Mixhop (Abu-El-Haija et al., 2019),
Revised: (Chen et al., 2018), APPNP (Klicpera et al., 2019a), Mixhop (Abu-El-Haija et al., 2019),

Original: SGC (Wu et al., 2019), DGI (Velickovic et al., 2019) and GIN (Xu et al., 2018a).
Revised: SGC (Wu et al., 2019), DGI (Velickovic et al., 2019) and GIN (Xu et al., 2018a).

Original: We use the Adam
Revised: We use the Adam

Original: SGD optimizer (Kingma & Ba, 2014) with a learning rate of 0.02 to train S 2 GC.
Revised: SGD optimizer (Kingma & Ba, 2014) with a learning rate of 0.02 to train S 2 GC.

Original: We set α = 0 .and
Revised: We set α = 0 .and

Original: K = 16 on all datasets.
Revised: K = 16 on all datasets.

Original: To determine K and α , we used the MetaOpt package Bergstra et al.
Revised: To determine K and α , we used the MetaOpt package Bergstra et al.

Original: (2015) with 20 steps to meta-optimize hyperparameters on the validation set of Cora.
Revised: (2015) with 20 steps to meta-optimize hyperparameters on the validation set of Cora.

Original: Following that, we ﬁxed K = 

Revised: Following that, we ﬁxed K =

Original: and α = 0 .
Revised: and α = 0 .

Original: across all datasets so K and α are not tuned to individual datasets at all.
Revised: across all datasets so K and α are not tuned to individual datasets at all.

Original: We will discuss the inﬂuence of α and K later.
Revised: We will discuss the inﬂuence of α and K later.

Original: To evaluate the proposed method on large scale benchmarks (see Table 6), we use Arxiv, Mag and
Revised: To evaluate the proposed method on large scale benchmarks (see Table 6), we use Arxiv, Mag and

Original: Products datasets to compare the proposed method with SGC, GraphSage, GCN, MLP and Softmax (multinomial Regression).
Revised: Products datasets to compare the proposed method with SGC, GraphSage, GCN, MLP and Softmax (multinomial Regression).

Original: On these three datasets, our method consistently outperforms SGC.
Revised: On these three datasets, our method consistently outperforms SGC.

Original: On
Revised: On

Original: Arxiv and Products, our method cannot outperform GCN and GraphSage while MLP outperforms softmax classiﬁer signiﬁcantly.
Revised: Arxiv and Products, our method cannot outperform GCN and GraphSage while MLP outperforms softmax classiﬁer signiﬁcantly.

Original: Thus, we argue that MLP plays a more important role here than the graph convolution.
Revised: Thus, we argue that MLP plays a more important role here than the graph convolution.

Original: To prove this point, we also conduct an experiment (S 2 GC+MLP) for which we use MLP in place of the linear classiﬁer, and we obtain a more powerful variant of S 2 GC.
Revised: To prove this point, we also conduct an experiment (S 2 GC+MLP) for which we use MLP in place of the linear classiﬁer, and we obtain a more powerful variant of S 2 GC.

Original: On Mag,
Revised: On Mag,

Original: S 2 GC+MLP outperforms S 2 GC by a tiny margin because the performance of MLP is close to the one of softmax.
Revised: S 2 GC+MLP outperforms S 2 GC by a tiny margin because the performance of MLP is close to the one of softmax.

Original: On other two datasets, S 2 GC+MLP is a very strong performer.
Revised: On other two datasets, S 2 GC+MLP is a very strong performer.

Original: Our S 2 GC+MLP is the best performer on Mag and Arxiv.
Revised: Our S 2 GC+MLP is the best performer on Mag and Arxiv.

Original: 4. T EXT C LASSIFICATION
Revised: 4. T EXT C LASSIFICATION

Original: Text classiﬁcation predicts the labels of documents.
Revised: Text classiﬁcation predicts the labels of documents.

Original: Yao et al.
Revised: Yao et al.

Original: (2019) use a 2-layer GCN to achieve state-of-the-art results by creating a corpus-level graph, which treats both documents and words as nodes in a graph.
Revised: (2019) use a 2-layer GCN to achieve state-of-the-art results by creating a corpus-level graph, which treats both documents and words as nodes in a graph.

Original: Word-to-word edge weights are given by Point-wise Mutual Information (PMI) and word-document edge weights are given by the normalized TF-IDF scores.
Revised: Word-to-word edge weights are given by Point-wise Mutual Information (PMI) and word-document edge weights are given by the normalized TF-IDF scores.

Original: We ran our experiments on ﬁve widely used benchmark corpora including the Movie Review (MR), 20-Newsgroups (20NG), Ohsumed, R52 and R8 of Reuters 21578.
Revised: We ran our experiments on ﬁve widely used benchmark corpora including the Movie Review (MR), 20-Newsgroups (20NG), Ohsumed, R52 and R8 of Reuters 21578.

Original: We ﬁrst preprocessed all datasets by cleaning and tokenizing text as Kim (2014).
Revised: We ﬁrst preprocessed all datasets by cleaning and tokenizing text as Kim (2014).

Original: We then removed stop words deﬁned in NLTK6 and low-frequency words appearing less than 5 times for 20NG, R8, R52 and Ohsumed.
Revised: We then removed stop words deﬁned in NLTK6 and low-frequency words appearing less than 5 times for 20NG, R8, R52 and Ohsumed.

Original: We compare our method with GCN (Kipf & Welling, 2016) and SGC (Wu et al., 2019).
Revised: We compare our method with GCN (Kipf & Welling, 2016) and SGC (Wu et al., 2019).

Original: The statistics of the
Revised: The statistics of the

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original: preprocessed datasets are summarized in Table 11.
Revised: preprocessed datasets are summarized in Table 11.

Original: Table 7 shows that S 2 GC rivals their models on 5 benchmark datasets.
Revised: Table 7 shows that S 2 GC rivals their models on 5 benchmark datasets.

Original: We provide the parameters setting in the supplementary material.
Revised: We provide the parameters setting in the supplementary material.

Original: 4. A DETAILED COMPARISON WITH VARIOUS NUMBERS OF LAYERS AND α
Revised: 4. A DETAILED COMPARISON WITH VARIOUS NUMBERS OF LAYERS AND α

Original: Table 8 summaries the results for models with various numbers of layers ( K is the number of layers and it coincides with the number of aggregated ﬁlters in S 2 GC).
Revised: Table 8 summaries the results for models with various numbers of layers ( K is the number of layers and it coincides with the number of aggregated ﬁlters in S 2 GC).

Original: We observe that on Cora, Citeseer and Pubmed, our method consistently obtains the best performance with K = 16 , equivalent oflayers.
Revised: We observe that on Cora, Citeseer and Pubmed, our method consistently obtains the best performance with K = 16 , equivalent oflayers.

Original: Overall, the results suggest that S 2 GC can aggregate over larger neighborhoods better than
Revised: Overall, the results suggest that S 2 GC can aggregate over larger neighborhoods better than

Original: SGC while suffering less from oversmoothing.
Revised: SGC while suffering less from oversmoothing.

Original: In contrast to S 2 GC, the performance of GCN and
Revised: In contrast to S 2 GC, the performance of GCN and

Original: SGC drops rapidly as the number of layers exceeds 32 due to oversmoothing.
Revised: SGC drops rapidly as the number of layers exceeds 32 due to oversmoothing.

Original: Table 9 summaries the results for the proposed method for various α ranging from 0 to 0.15.
Revised: Table 9 summaries the results for the proposed method for various α ranging from 0 to 0.15.

Original: The table shows that α slightly improves the performance of S 2 GC.
Revised: The table shows that α slightly improves the performance of S 2 GC.

Original: Thus, balancing the impact of selfloop by α w.r.t.
Revised: Thus, balancing the impact of selfloop by α w.r.t.

Original: other ﬁlters of consecutively larger receptive ﬁelds is useful but the self-loop is not mandatory.
Revised: other ﬁlters of consecutively larger receptive ﬁelds is useful but the self-loop is not mandatory.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have proposed Simple Spectral Graph Convolution (S 2 GC), a method extending the Markov
Revised: We have proposed Simple Spectral Graph Convolution (S 2 GC), a method extending the Markov

Original: Diffusion Kernel (Section 3.2), whose feature maps emerge from the normalized Laplacian Regularization problem (Section 3.3) if K → ∞ .
Revised: Diffusion Kernel (Section 3.2), whose feature maps emerge from the normalized Laplacian Regularization problem (Section 3.3) if K → ∞ .

Original: Our theoretical analysis shows that S 2 GC obtains the right level of balance during the aggregation of consecutively larger receptive ﬁelds.
Revised: Our theoretical analysis shows that S 2 GC obtains the right level of balance during the aggregation of consecutively larger receptive ﬁelds.

Original: We have shown there exists a connection between S 2 GC and SGC, APPNP and JKN by analyzing spectral properties and implementation of each model.
Revised: We have shown there exists a connection between S 2 GC and SGC, APPNP and JKN by analyzing spectral properties and implementation of each model.

Original: However, as our Claims I and II show that we have designed a ﬁlter with unique properties to capture a cascade of gradually increasing contexts while limiting oversmoothing by giving proportionally larger weights to the closest neighborhoods of each node.
Revised: However, as our Claims I and II show that we have designed a ﬁlter with unique properties to capture a cascade of gradually increasing contexts while limiting oversmoothing by giving proportionally larger weights to the closest neighborhoods of each node.

Original: We have conducted extensive and rigorous experiments which show that S 2 GC is competitive frequently outperforming many state-of-the-art methods on unsupervised, semi-supervised and supervised tasks given several popular dataset benchmarks.
Revised: We have conducted extensive and rigorous experiments which show that S 2 GC is competitive frequently outperforming many state-of-the-art methods on unsupervised, semi-supervised and supervised tasks given several popular dataset benchmarks.

Original: A CKNOWLEDGMENTS
Revised: A CKNOWLEDGMENTS

Original: This research is supported by an Australian Government Research Training Program (RTP) Scholarship.
Revised: This research is supported by an Australian Government Research Training Program (RTP) Scholarship.

Original: A S UPPLEMENTARY M ATERIAL A. N ODE C LUSTERING
Revised: A S UPPLEMENTARY M ATERIAL A. N ODE C LUSTERING

Original: For S 2 GC and AGC, we set max iterations to 60 .
Revised: For S 2 GC and AGC, we set max iterations to 60 .

Original: For other baselines, we follow the parameter settings in the original papers.
Revised: For other baselines, we follow the parameter settings in the original papers.

Original: In particular, for DeepWalk, the number of random walks is 10, the number of latent dimensions for each node is 128, and the path length of each random walk is 80.
Revised: In particular, for DeepWalk, the number of random walks is 10, the number of latent dimensions for each node is 128, and the path length of each random walk is 80.

Original: For DNGR, the autoencoder is of three layers with 512 neurons and 256 neurons in the hidden layers respectively.
Revised: For DNGR, the autoencoder is of three layers with 512 neurons and 256 neurons in the hidden layers respectively.

Original: For GAE and VGAE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer, and train the encoders for 200 iterations using the Adam optimizer with learning rate equal 0.01.
Revised: For GAE and VGAE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer, and train the encoders for 200 iterations using the Adam optimizer with learning rate equal 0.01.

Original: For ARGE and ARVGE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer.
Revised: For ARGE and ARVGE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer.

Original: The discriminators are built by two hidden layers with 16 and 64 neurons respectively.
Revised: The discriminators are built by two hidden layers with 16 and 64 neurons respectively.

Original: On Cora, Citeseer and Wiki, we train the autoencoder-related models of ARGE and ARVGE for 200 iterations with the Adam optimizer, with the encoder and discriminator learning rates both set as 0.001; on Pubmed, we train them for 2000 iterations with the encoder learning rate 0.001 and the discriminator learning rate 0.008.
Revised: On Cora, Citeseer and Wiki, we train the autoencoder-related models of ARGE and ARVGE for 200 iterations with the Adam optimizer, with the encoder and discriminator learning rates both set as 0.001; on Pubmed, we train them for 2000 iterations with the encoder learning rate 0.001 and the discriminator learning rate 0.008.

Original: A. T EXT C LASSIFICATION
Revised: A. T EXT C LASSIFICATION

Original: The 20NG dataset1 (bydate version) contains 18,846 documents evenly categorized into 20 different categories.
Revised: The 20NG dataset1 (bydate version) contains 18,846 documents evenly categorized into 20 different categories.

Original: In total, 11,314 documents are in the training set and 7,532 documents are in the test set.
Revised: In total, 11,314 documents are in the training set and 7,532 documents are in the test set.

Original: The Ohsumed corpus2 is from the MEDLINE database, which is a bibliographic database of important medical literature maintained by the National Library of Medicine.
Revised: The Ohsumed corpus2 is from the MEDLINE database, which is a bibliographic database of important medical literature maintained by the National Library of Medicine.

Original: In this work, we used the 13,929 unique cardiovascular diseases abstracts in the ﬁrst 20,000 abstracts of the year 1991.
Revised: In this work, we used the 13,929 unique cardiovascular diseases abstracts in the ﬁrst 20,000 abstracts of the year 1991.

Original: Each document in the set has one or more associated categories from the 23 disease categories.
Revised: Each document in the set has one or more associated categories from the 23 disease categories.

Original: As we focus on single-label text classiﬁcation, the documents belonging to multiple categories are excluded so that 7,400 documents belonging to only one category remain.
Revised: As we focus on single-label text classiﬁcation, the documents belonging to multiple categories are excluded so that 7,400 documents belonging to only one category remain.

Original: 3,357 documents are in the training set and 4,043 documents are in the test set.
Revised: 3,357 documents are in the training set and 4,043 documents are in the test set.

Original: R52 and R8 (all-terms version) are two subsets of the Reuters 21578 dataset.
Revised: R52 and R8 (all-terms version) are two subsets of the Reuters 21578 dataset.

Original: R8 has 8 categories, and was split to 5,485 training and 2,189 test documents.
Revised: R8 has 8 categories, and was split to 5,485 training and 2,189 test documents.

Original: R52 has 52 categories, and was split to 6,532 training and 2,568 test documents.
Revised: R52 has 52 categories, and was split to 6,532 training and 2,568 test documents.

Original: MR is a movie review dataset for binary sentiment classiﬁcation, in which each review only contains one sentence (Pang & Lee, 2005)
Revised: MR is a movie review dataset for binary sentiment classiﬁcation, in which each review only contains one sentence (Pang & Lee, 2005)

Original: The corpus has 5,331 positive and 5,331 negative reviews.
Revised: The corpus has 5,331 positive and 5,331 negative reviews.

Original: We used the training/test split in (Tang et al., 2015).
Revised: We used the training/test split in (Tang et al., 2015).

Original: A.2. T EXT C LASSIFICATION
Revised: A.2. T EXT C LASSIFICATION

Original: Parameters.
Revised: Parameters.

Original: We follow the setting of Text GCN (Yao et al., 2019) that includes experiments on four widely used benchmark corpora such as 20-Newsgroups (20NG), Ohsumed, R52 and R8 of
Revised: We follow the setting of Text GCN (Yao et al., 2019) that includes experiments on four widely used benchmark corpora such as 20-Newsgroups (20NG), Ohsumed, R52 and R8 of

Original: Reuters 21578.
Revised: Reuters 21578.

Original: For Text GCN, SGC, and our approach, the embedding size of the ﬁrst convolution
Revised: For Text GCN, SGC, and our approach, the embedding size of the ﬁrst convolution

Original: [Table]
Revised: [Table]

Original: layer is 200 and the window size is 20.
Revised: layer is 200 and the window size is 20.

Original: We set the learning rate to 0.02, dropout rate to 0.5 and the decay rate to 0.
Revised: We set the learning rate to 0.02, dropout rate to 0.5 and the decay rate to 0.

Original: The 10% of training set is randomly selected for validation.
Revised: The 10% of training set is randomly selected for validation.

Original: Following (Kipf & Welling, 2016), we trained our method and Text GCN for a maximum of 200 epochs using the
Revised: Following (Kipf & Welling, 2016), we trained our method and Text GCN for a maximum of 200 epochs using the

Original: Adam (Kingma & Ba, 2014) optimizer, and we stop training if the validation loss does not decrease for 10 consecutive epochs.
Revised: Adam (Kingma & Ba, 2014) optimizer, and we stop training if the validation loss does not decrease for 10 consecutive epochs.

Original: The text graph was built according to steps detailed in the supplementary material.
Revised: The text graph was built according to steps detailed in the supplementary material.

Original: To convert text classiﬁcation into the node classiﬁcation on graph, there are two relationships considered when forming graphs: (i) the relation between documents and words and (ii) the connection between words.
Revised: To convert text classiﬁcation into the node classiﬁcation on graph, there are two relationships considered when forming graphs: (i) the relation between documents and words and (ii) the connection between words.

Original: For the ﬁrst type of relations, we build edges among word nodes and document nodes based on the word occurrence in documents.
Revised: For the ﬁrst type of relations, we build edges among word nodes and document nodes based on the word occurrence in documents.

Original: The weight of the edge between a document node and a word node is the Term Frequency-Inverse Document Frequency (Rajaraman & Ullman, 2011) (TF-IDF) of the word in the document applied to build the Docs-words graph.
Revised: The weight of the edge between a document node and a word node is the Term Frequency-Inverse Document Frequency (Rajaraman & Ullman, 2011) (TF-IDF) of the word in the document applied to build the Docs-words graph.

Original: For the second type of relations, we build edges in graph among word co-occurrences across the whole corpus.
Revised: For the second type of relations, we build edges in graph among word co-occurrences across the whole corpus.

Original: To utilize the global word co-occurrence information, we use a ﬁxed-size sliding window on all documents in the corpus to gather co-occurrence statistics.
Revised: To utilize the global word co-occurrence information, we use a ﬁxed-size sliding window on all documents in the corpus to gather co-occurrence statistics.

Original: Point-wise Mutual Information (Church &
Revised: Point-wise Mutual Information (Church &

Original: Hanks, 1990) (PMI), a popular measure for word associations, is used to calculate weights between two word nodes according to the following deﬁnition:
Revised: Hanks, 1990) (PMI), a popular measure for word associations, is used to calculate weights between two word nodes according to the following deﬁnition:

Original: [Equation]
Revised: [Equation]

Original: ) where p ( i, j ) =
Revised: ) where p ( i, j ) =

Original: W ( i,j ) W , p ( i ) =
Revised: W ( i,j ) W , p ( i ) =

Original: W ( i ) W .
Revised: W ( i ) W .

Original: # W ( i ) is the number of sliding windows in a corpus that contain word i , # W ( i, j ) is the number of sliding windows that contain both word i and word j , and # W is the total number of sliding windows in the corpus.
Revised: # W ( i ) is the number of sliding windows in a corpus that contain word i , # W ( i, j ) is the number of sliding windows that contain both word i and word j , and # W is the total number of sliding windows in the corpus.

Original: A positive PMI value implies a high semantic correlation of words in a corpus, while a negative PMI value indicates little or no semantic correlation in the corpus.
Revised: A positive PMI value implies a high semantic correlation of words in a corpus, while a negative PMI value indicates little or no semantic correlation in the corpus.

Original: Therefore, we only add edges between word pairs with positive
Revised: Therefore, we only add edges between word pairs with positive

Original: PMI values:
Revised: PMI values:

Original: [Equation]
Revised: [Equation]

Original: A. G RAPH C LASSIFICATION
Revised: A. G RAPH C LASSIFICATION

Original: We report the average accuracy of 10-fold cross validation on a number of common benchmark datasets, shown in Table 12, where we randomly sample a training fold to serve as a validation set.
Revised: We report the average accuracy of 10-fold cross validation on a number of common benchmark datasets, shown in Table 12, where we randomly sample a training fold to serve as a validation set.

Original: We only make use of discrete node features.
Revised: We only make use of discrete node features.

Original: In case they are not given, we use one-hot encodings of node degrees as the feature input.
Revised: In case they are not given, we use one-hot encodings of node degrees as the feature input.

Original: We note that graph classiﬁcation is a task highly dependent on the global pooling strategy.
Revised: We note that graph classiﬁcation is a task highly dependent on the global pooling strategy.

Original: There exist methods that apply sophisticated mechanisms for this step.
Revised: There exist methods that apply sophisticated mechanisms for this step.

Original: However, with a readout function and a highly scalable S 2 GC model, we comfortably outperform all methods on MUTAG, Proteins and IMDB-Binary, even DiffPool which has a differentiable graph pooling module to gather information across different scales.
Revised: However, with a readout function and a highly scalable S 2 GC model, we comfortably outperform all methods on MUTAG, Proteins and IMDB-Binary, even DiffPool which has a differentiable graph pooling module to gather information across different scales.

Original: A stronger performer (Koniusz &
Revised: A stronger performer (Koniusz &

Original: Zhang, 2020) uses the GIN-0 backbone and second-order pooling with the so-called spectral power normalization, referred to as MaxExp(F).
Revised: Zhang, 2020) uses the GIN-0 backbone and second-order pooling with the so-called spectral power normalization, referred to as MaxExp(F).

Original: In contrast, we use a simple readout feature aggregation.
Revised: In contrast, we use a simple readout feature aggregation.

Original: [Table]
Revised: [Table]

Original: A. T HEORETICAL ANALYSIS
Revised: A. T HEORETICAL ANALYSIS

Original: Below we show that we can reduce oversmoothing compared to SGC while incorporating larger receptive ﬁelds.
Revised: Below we show that we can reduce oversmoothing compared to SGC while incorporating larger receptive ﬁelds.

Original: Our design contains a sum of consecutive diffusion matrices (cid:101) T k , k = 0 , · · · , K .
Revised: Our design contains a sum of consecutive diffusion matrices (cid:101) T k , k = 0 , · · · , K .

Original: As k increases, so does the neighborhood of each node visited during diffusion (cid:101) T k (analogy to random walks).
Revised: As k increases, so does the neighborhood of each node visited during diffusion (cid:101) T k (analogy to random walks).

Original: This means that:
Revised: This means that:

Original: Claim I.
Revised: Claim I.

Original: Our ﬁlter, by design, will give the highest weight to the closest neighborhood of a node as neighborhoods N of diffusion steps k = 0 , · · · , K obey N ( (cid:101) T 0 ) ⊆
Revised: Our ﬁlter, by design, will give the highest weight to the closest neighborhood of a node as neighborhoods N of diffusion steps k = 0 , · · · , K obey N ( (cid:101) T 0 ) ⊆

Original: N ( (cid:101) T 1 ) ⊆ · · · ⊆ N ( (cid:101) T K ) ⊆
Revised: N ( (cid:101) T 1 ) ⊆ · · · ⊆ N ( (cid:101) T K ) ⊆

Original: N ( (cid:101) T ∞ ) .
Revised: N ( (cid:101) T ∞ ) .

Original: That is, smaller neighborhoods belong to larger neighborhoods too.
Revised: That is, smaller neighborhoods belong to larger neighborhoods too.

Original: To see this clearer, for the q -dimensional Euclidean lattice graph with inﬁnite number of nodes, after t steps of random walk, the estimate of absolute distance the walk moves from the source to its current position is given as:
Revised: To see this clearer, for the q -dimensional Euclidean lattice graph with inﬁnite number of nodes, after t steps of random walk, the estimate of absolute distance the walk moves from the source to its current position is given as:

Original: [Equation]
Revised: [Equation]

Original: where r ( t, q ) is the absolute distance walked from the source to the current point and Γ( · ) is the
Revised: where r ( t, q ) is the absolute distance walked from the source to the current point and Γ( · ) is the

Original: Gamma function.
Revised: Gamma function.

Original: Moreover, if the number of dimensions q → ∞ , we have r ( t, q ) ≤ √ t .
Revised: Moreover, if the number of dimensions q → ∞ , we have r ( t, q ) ≤ √ t .

Original: It is clear then that the receptive ﬁeld associated with the random walk (and thus diffusion at time t ) obeys the monotonically increasing radius r , that is r (0)
Revised: It is clear then that the receptive ﬁeld associated with the random walk (and thus diffusion at time t ) obeys the monotonically increasing radius r , that is r (0)

Original: ≤ r (1) ≤ · · · ≤ r ( K ) ≤ · · · ≤ r ( ∞ ) .
Revised: ≤ r (1) ≤ · · · ≤ r ( K ) ≤ · · · ≤ r ( ∞ ) .

Original: To see that, simply plot √ t (and/or the more complicated expression that includes the Gamma function).
Revised: To see that, simply plot √ t (and/or the more complicated expression that includes the Gamma function).

Original: This proves Claim I for the Euclidean lattice graph.
Revised: This proves Claim I for the Euclidean lattice graph.

Original: That is, for consecutive diffusion steps (cid:101) T k , k = 0 , · · · , K , our receptive ﬁeld grows.
Revised: That is, for consecutive diffusion steps (cid:101) T k , k = 0 , · · · , K , our receptive ﬁeld grows.

Original: Moreover, note that our ﬁlter is realized as the sum of consecutive diffusion steps, that ist
Revised: Moreover, note that our ﬁlter is realized as the sum of consecutive diffusion steps, that ist

Original: [Equation]
Revised: [Equation]

Original: =0 diff ( s, τ ) where s is the source of walk.
Revised: =0 diff ( s, τ ) where s is the source of walk.

Original: It is easy to see then that even if each walked distance was to contribute the energy proportional with r ( t ) to the summation term, we have:
Revised: It is easy to see then that even if each walked distance was to contribute the energy proportional with r ( t ) to the summation term, we have:

Original: [Equation]
Revised: [Equation]

Original: (17) where the enumerator is the model of the total energy when aggregating over receptive ﬁelds from size 0 to ∞ in S 2 GC while the denominator is the total energy of SGC (ﬁlter is given by (cid:101) T K , that is by diff ( s, t ) ).
Revised: (17) where the enumerator is the model of the total energy when aggregating over receptive ﬁelds from size 0 to ∞ in S 2 GC while the denominator is the total energy of SGC (ﬁlter is given by (cid:101) T K , that is by diff ( s, t ) ).

Original: The above proof shows that the above ratio of energies is 0, which means that:
Revised: The above proof shows that the above ratio of energies is 0, which means that:

Original: Claim II.
Revised: Claim II.

Original: When the ratio of energies of two models is 0, the energy of the inﬁnite-dimensional receptive ﬁeld (when t → ∞ ) in S 2 GC is not going to dominate the sum energy of our ﬁlter.
Revised: When the ratio of energies of two models is 0, the energy of the inﬁnite-dimensional receptive ﬁeld (when t → ∞ ) in S 2 GC is not going to dominate the sum energy of our ﬁlter.

Original: Thus,
Revised: Thus,

Original: S 2 GC can incorporate larger receptive ﬁelds than SGC without eclipsing the contributions from smaller receptive ﬁelds as t → ∞ on the Euclidean lattice graph.
Revised: S 2 GC can incorporate larger receptive ﬁelds than SGC without eclipsing the contributions from smaller receptive ﬁelds as t → ∞ on the Euclidean lattice graph.

Original: However, in practice, we work with ﬁnite-dimensional non-Euclidean graphs.
Revised: However, in practice, we work with ﬁnite-dimensional non-Euclidean graphs.

Original: Obtaining the absolute distance r ( t ) walked from the source is a difﬁcult topic.
Revised: Obtaining the absolute distance r ( t ) walked from the source is a difﬁcult topic.

Original: As an example, see Eq.
Revised: As an example, see Eq.

Original: 184 in Masuda et al.
Revised: 184 in Masuda et al.

Original: For this reason, below we use a simple approximation.
Revised: For this reason, below we use a simple approximation.

Original: We use Theorem 1 as the proxy for the walked radius.
Revised: We use Theorem 1 as the proxy for the walked radius.

Original: That is to say the error of convergence to the stationary distribution is indicative of the absolute distance walked from the source/node.
Revised: That is to say the error of convergence to the stationary distribution is indicative of the absolute distance walked from the source/node.

Original: Speciﬁcally, we have:
Revised: Speciﬁcally, we have:

Original: Recall Theorem 1, that is let λ 2 denote second largest eigenvalue of transition matrix (cid:101) T
Revised: Recall Theorem 1, that is let λ 2 denote second largest eigenvalue of transition matrix (cid:101) T

Original: =
Revised: =

Original: D − 1 A , p ( t ) be the probability distribution vector and π the stationary distribution.
Revised: D − 1 A , p ( t ) be the probability distribution vector and π the stationary distribution.

Original: If walk starts from the vertex i , p i (0) = 1 , then after t steps for every vertex:
Revised: If walk starts from the vertex i , p i (0) = 1 , then after t steps for every vertex:

Original: [Equation]
Revised: [Equation]

Original: Then, the average walked distance r from node i over t steps in a graph with n nodes and connectivity given by the second largest eigenvalue λ 2 , denoted by r ( i, t, n ) is lower-bounded by ¯ r ( i, t, n ) as follows:
Revised: Then, the average walked distance r from node i over t steps in a graph with n nodes and connectivity given by the second largest eigenvalue λ 2 , denoted by r ( i, t, n ) is lower-bounded by ¯ r ( i, t, n ) as follows:

Original: [Equation]
Revised: [Equation]

Original: where n is the number of nodes, t is the number of diffusion steps (think (cid:101) T k ), d i and d j are degrees of nodes i and j ,
Revised: where n is the number of nodes, t is the number of diffusion steps (think (cid:101) T k ), d i and d j are degrees of nodes i and j ,

Original: λ 2 being the second largest eigenvalue intuitively denotes the graph connectivity (large λ 2 ≤ 1 indicates low connectivity while low λ 2 indicates high connectivity in graph), (cid:101) E is the sum of square roots of node degrees and ρ =
Revised: λ 2 being the second largest eigenvalue intuitively denotes the graph connectivity (large λ 2 ≤ 1 indicates low connectivity while low λ 2 indicates high connectivity in graph), (cid:101) E is the sum of square roots of node degrees and ρ =

Original: ( n − 1) √ d i (cid:101) E −√ d i is in fact a constant for a given graph.
Revised: ( n − 1) √ d i (cid:101) E −√ d i is in fact a constant for a given graph.

Original: While the above approximations may be loose for very small/large t , the important property to note is that r
Revised: While the above approximations may be loose for very small/large t , the important property to note is that r

Original: ( i, 0 , n ) ≤ r ( i, 1 , n ) ≤ · · · ≤ r ( i, t, n ) which indicates that our ﬁlter indeed realises the sum over increasingly larger receptive ﬁelds.
Revised: ( i, 0 , n ) ≤ r ( i, 1 , n ) ≤ · · · ≤ r ( i, t, n ) which indicates that our ﬁlter indeed realises the sum over increasingly larger receptive ﬁelds.

Original: As smaller receptive ﬁelds are a subset of larger receptive ﬁelds given node i , that is N ( (cid:101) T 0 ) ⊆
Revised: As smaller receptive ﬁelds are a subset of larger receptive ﬁelds given node i , that is N ( (cid:101) T 0 ) ⊆

Original: N ( (cid:101) T 1 ) ⊆ · · · ⊆ N ( (cid:101) T K ) ⊆
Revised: N ( (cid:101) T 1 ) ⊆ · · · ⊆ N ( (cid:101) T K ) ⊆

Original: N ( (cid:101) T ∞ ) , this proves our Claim I.
Revised: N ( (cid:101) T ∞ ) , this proves our Claim I.

Original: To prove Claim II for a general connected non-bipartite graph, we have:
Revised: To prove Claim II for a general connected non-bipartite graph, we have:

Original: [Equation]
Revised: [Equation]

Original: Similar ﬁndings can be noted by carefully considering the meaning of so-called Cheeger constant introduced in Section A.5.
Revised: Similar ﬁndings can be noted by carefully considering the meaning of so-called Cheeger constant introduced in Section A.5.

Original: More details on spectral analysis of ﬁlters in GCNs can be found in studies of Li et al.
Revised: More details on spectral analysis of ﬁlters in GCNs can be found in studies of Li et al.

Original: (2018a)
Revised: (2018a)

Original: and
Revised: and

Original: Li et al.
Revised: Li et al.

Original: (2018b).
Revised: (2018b).

Original: A. G RAPH PARTITIONING
Revised: A. G RAPH PARTITIONING

Original: Below we introduce the deﬁnitions of expansion and k -way Cheeger constant.
Revised: Below we introduce the deﬁnitions of expansion and k -way Cheeger constant.

Original: Deﬁnition A.1.
Revised: Deﬁnition A.1.

Original: For a node subset S ⊆ V , so-called expansion φ ( S ) =
Revised: For a node subset S ⊆ V , so-called expansion φ ( S ) =

Original: | E ( S ) | min { vol ( S ) , vol ( V \ S ) } , where
Revised: | E ( S ) | min { vol ( S ) , vol ( V \ S ) } , where

Original: E ( S ) is the set of edges with one node in S and vol ( S ) is the sum of degree of nodes in set S.
Revised: E ( S ) is the set of edges with one node in S and vol ( S ) is the sum of degree of nodes in set S.

Original: Deﬁnition A.2.
Revised: Deﬁnition A.2.

Original: The k -way Cheeger constant is given as: ρ G ( k ) =
Revised: The k -way Cheeger constant is given as: ρ G ( k ) =

Original: min S 1 ,S 2 , ··· ,S k max { φ ( S i ) : i = { 1 , · · · , k }} where the minimum is over all collections of k non-empty disjoint subsets
Revised: min S 1 ,S 2 , ··· ,S k max { φ ( S i ) : i = { 1 , · · · , k }} where the minimum is over all collections of k non-empty disjoint subsets

Original: S 1 , S 2 , · · · , S k ⊆ V .
Revised: S 1 , S 2 , · · · , S k ⊆ V .

Original: According to the deﬁnitions, the expansion in Def.
Revised: According to the deﬁnitions, the expansion in Def.

Original: A.1 describes the effect of graph partitioning according to subset S while the k -way Cheeger constant reﬂects the effect of the graph partitioning into k parts–the smaller the value the better the partitioning is.
Revised: A.1 describes the effect of graph partitioning according to subset S while the k -way Cheeger constant reﬂects the effect of the graph partitioning into k parts–the smaller the value the better the partitioning is.

Original: Higher-order Cheeger’s inequality (Bandeira et al., 2013; Lee et al., 2014) bridges the gap between the network spectral analysis and graph partitioning by controlling the bounds of k -way Cheeger constant:
Revised: Higher-order Cheeger’s inequality (Bandeira et al., 2013; Lee et al., 2014) bridges the gap between the network spectral analysis and graph partitioning by controlling the bounds of k -way Cheeger constant:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ) where λ k is the k-th eigenvalue of the normalized Laplacian matrix and 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n .
Revised: ) where λ k is the k-th eigenvalue of the normalized Laplacian matrix and 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n .

Original: From inequality 21, we can conclude that small (large) eigenvalues control global clustering (local smoothing) effect of the graph partitioned into a few large parts (many small parts).
Revised: From inequality 21, we can conclude that small (large) eigenvalues control global clustering (local smoothing) effect of the graph partitioned into a few large parts (many small parts).

Original: Thus, speciﬁc combination of low- and high-pass ﬁltering of our design (see Figure 1) a indicates the weight tradeoff between large and small partitions contained by the node.
Revised: Thus, speciﬁc combination of low- and high-pass ﬁltering of our design (see Figure 1) a indicates the weight tradeoff between large and small partitions contained by the node.