Original: S IMPLE S PECTRAL G RAPH C ONVOLUTION
Revised: S IMPLE S PECTRAL G RAPH C ONVOLUTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Convolutional Networks (GCNs) have drawn signiﬁcant attention and become promising methods for learning graph representations.
Revised: Graph Convolutional Networks (GCNs) have drawn signiﬁcant attention and become promising methods for learning graph representations.

Original: The most GCNs suffer the performance loss when the depth of the model increases.
Revised: The most GCNs suffer the performance loss when the depth of the model increases.

Original: Similarly to
Revised: Similarly to

Original: CNNs, without specially designed architectures, the performance of a network degrades quickly.
Revised: CNNs, without specially designed architectures, the performance of a network degrades quickly.

Original: Some researchers argues the required neighbourhood size and neural network depth are two completely orthogonal aspects of graph representation.
Revised: Some researchers argue that the required neighbourhood size and neural network depth are two completely orthogonal aspects of graph representation.

Original: Thus, several methods extend the neighbourhood by aggregating k -hop neighbourhoods of nodes while using shallow neural networks.
Revised: Thus, several methods extend the neighbourhood by aggregating k -hop neighbourhoods of nodes while using shallow neural networks.

Original: However, these methods still encounter oversmoothing, high computation and storage costs.
Revised: However, these methods still encounter oversmoothing, high computation and storage costs.

Original: In this paper, we use the Markov diffusion kernel to derive a variant of GCN called
Revised: In this paper, we use the Markov Diffusion Kernel to derive a variant of GCN called

Original: Simple Spectral Graph Convolution (S 2 GC) which is closely related to spectral models and combines strengths of both spatial and spectral methods.
Revised: Simple Spectral Graph Convolution (S 2 GC) which is closely related to spectral models and combines strengths of both spatial and spectral methods.

Original: Our spectral analysis shows that our simple spectral graph convolution used in S 2 GC is a low-pass ﬁlter which partitions networks into a few large parts.
Revised: Our spectral analysis shows that our simple spectral graph convolution used in S 2 GC is a low-pass ﬁlter which partitions networks into a few large parts.

Original: Our experimental evaluation demonstrates that S 2 GC with a linear learner is competitive in text and node classiﬁcation tasks.
Revised: Our experimental evaluation demonstrates that S 2 GC with a linear learner is competitive in text and node classiﬁcation tasks.

Original: Moreover, S 2 GC is comparable to other state-of-the-art methods for node clustering and community prediction tasks.
Revised: Moreover, S 2 GC is comparable to other state-of-the-art methods for node clustering and community prediction tasks.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: In the past few years, the rise and application of deep learning have successfully promoted the research of computer vision and data mining.
Revised: In the past few years, the rise and application of deep learning have successfully promoted the research of computer vision and data mining.

Original: Although these deep learning methods have been applied to extract the features of on the Euclidean grid (spatial data) with great success, the data in many practical scenarios lies on non-Euclidean spaces.
Revised: Although these deep learning methods have been applied to extract the features of on the Euclidean grid (spatial data) with great success, the data in many practical scenarios lies on non-Euclidean spaces.

Original: Processing non-Euclidean spatial data is a challenge for deep learning methods.
Revised: Processing non-Euclidean spatial data is a challenge for deep learning methods.

Original: By deﬁning a convolution operator between the graph and signal, Graph Convolutional Networks (GCNs) generalize Convolutional Neural Networks (CNNs) to graph-structured inputs which contain attributes.
Revised: By deﬁning a convolution operator between the graph and signal, Graph Convolutional Networks (GCNs) generalize Convolutional Neural Networks (CNNs) to graph-structured inputs which contain attributes.

Original: Message Passing Neural Networks (MPNNs)
Revised: Message Passing Neural Networks (MPNNs)

Original: (Gilmer et al., 2017) unify the graph convolution as two functions: the transformation function and the aggregation function.
Revised: (Gilmer et al., 2017) unify the graph convolution as two functions: the transformation function and the aggregation function.

Original: MPNN iteratively propagates node features based on the adjacency structure of graph in a number of rounds.
Revised: MPNN iteratively propagates node features based on the adjacency structure of graph in a number of rounds.

Original: Despite their enormous success in many applications like social media, trafﬁc analysis, biology, recommendation systems and even computer vision, most of the current GCN models use shallow architectures because many of the recent models such as GCN (Kipf & Welling, 2016) achieve their best performance with 2-layer models.
Revised: Despite their enormous success in many applications like social media, trafﬁc analysis, biology, recommendation systems and even computer vision, most of the current GCN models use shallow architectures because many of the recent models such as GCN (Kipf & Welling, 2016) achieve their best performance with 2-layer models.

Original: In other words, 2-layer GCN models aggregate nodes in twohops neighbourhood and thus have no ability to extract information in k -hops neighbourhoods for k > 2 .
Revised: In other words, 2-layer GCN models aggregate nodes in twohops neighbourhood and thus have no ability to extract information in k -hops neighbourhoods for k > 2 .

Original: Moreover, stacking more layers and adding a non-linearity tends to degrade the performance of these models.
Revised: Moreover, stacking more layers and adding a non-linearity tends to degrade the performance of these models.

Original: Such a phenomenon is called oversmoothing (Li et al., 2018), which suggests that as the number of layers increases, the representations of the nodes in GCNs are inclined to converge to a certain value and become less distinct from one another.
Revised: Such a phenomenon is called oversmoothing (Li et al., 2018), which suggests that as the number of layers increases, the representations of the nodes in GCNs are inclined to converge to a certain value and become less distinct from one another.

Original: Even adding residual connections, an effective trick for training very deep neural networks in computer vision, merely slows down the oversmoothing issue (Kipf & Welling, 2016) in GCNs.
Revised: Even adding residual connections, an effective trick for training very deep neural networks in computer vision, merely slows down the oversmoothing issue (Kipf & Welling, 2016) in GCNs.

Original: It appears that deep GCN models gain nothing but the performance degradation from the deep architecture.
Revised: It appears that deep GCN models gain nothing but the performance degradation from the deep architecture.

Original: One solution for that is only deepening the aggregation function but keeping the same transformation function because the required neighbourhood size and neural network depth can be regarded as two orthogonal aspects of design.
Revised: One solution for that is only deepening the aggregation function but keeping the same transformation function because the required neighbourhood size and neural network depth can be regarded as two orthogonal aspects of design.

Original: SGC (Wu et al., 2019) attempts to capture information from k -hops neighbours in the graph by applying the K -th power of the graph convolution matrix in a single neural network layer.
Revised: SGC (Wu et al., 2019) attempts to capture information from k -hops neighbours in the graph by applying the K -th power of the graph convolution matrix in a single neural network layer.

Original: This scheme is also used for attributed graph clustering (Zhang et al.,
Revised: This scheme is also used for attributed graph clustering (Zhang et al.,

Original: However, SGC is also suffering from oversmoothing as shown in Theorem 1.
Revised: However, SGC is also suffering from oversmoothing as shown in Theorem 1.

Original: PPNP and
Revised: PPNP and

Original: APPNP (Klicpera et al., 2019a) replace the power of the graph convolution matrix with the Personalized PageRank matrix to solve the oversmoothing problem.
Revised: APPNP (Klicpera et al., 2019a) replace the power of the graph convolution matrix with the Personalized PageRank matrix to solve the oversmoothing problem.

Original: Although APPNP tackles the oversmoothing problem, it exchanges the order of the aggregation function and the transformation function.
Revised: Although APPNP tackles the oversmoothing problem, it exchanges the order of the aggregation function and the transformation function.

Original: It thus needs to learn a transformation function for the input before starting aggregating which limits the applications in unsupervised learning.
Revised: It thus needs to learn a transformation function for the input before starting aggregating which limits the applications in unsupervised learning.

Original: GDC (Klicpera et al., 2019b) further extends
Revised: GDC (Klicpera et al., 2019b) further extends

Original: APPNP by generalizing Personalized PageRank (Page et al., 1999) to an arbitrary graph diffusion process.
Revised: APPNP by generalizing Personalized PageRank (Page et al., 1999) to an arbitrary graph diffusion process.

Original: GDC, technically, has stronger expressive power than SGC (Wu et al., 2019), PPNP and
Revised: GDC, technically, has stronger expressive power than SGC (Wu et al., 2019), PPNP and

Original: APPNP (Klicpera et al., 2019a)
Revised: APPNP (Klicpera et al., 2019a)

Original: but it leads to a dense transition matrix which makes the computation and space storage intractable for large graphs (although authors suggest that the shrinkage method can be used to sparsify the generated transition matrix by ignoring small values).
Revised: but it leads to a dense transition matrix which makes the computation and space storage intractable for large graphs (although authors suggest that the shrinkage method can be used to sparsify the generated transition matrix by ignoring small values).

Original: To solve aforementioned issues, we propose a Simple Spectral Graph Convolution (S 2 GC) network for node representation for different tasks such as: node clustering and node classiﬁcation under semi-supervised and unsupervised setting.
Revised: To solve aforementioned issues, we propose a Simple Spectral Graph Convolution (S 2 GC) network for node representation for different tasks such as: node clustering and node classiﬁcation under semi-supervised and unsupervised setting.

Original: By analyzing Markov diffusion kernel (Fouss et al., 2012), we obtain a simple way to compute graph convolution: averaging the result of spatial graph convolutions with different power versions of the normalized adjacency matrix, which can be seen as aggregating neighbourhood information in the fashion of k -steps random walk.
Revised: By analyzing Markov Diffusion Kernel (Fouss et al., 2012), we obtain a simple way to compute graph convolution: averaging the result of spatial graph convolutions with different power versions of the normalized adjacency matrix, which can be seen as aggregating neighbourhood information in the fashion of k -steps random walk.

Original: Thus, S 2 GC is closely related to spectral models while it is still can be viewed as operating in the spatial domain.
Revised: Thus, S 2 GC is closely related to spectral models while it is still can be viewed as operating in the spatial domain.

Original: We also show that in spectral analysis the proposed method is a low-pass ﬁlter which leads graph partition into a few large parts.
Revised: We also show that in spectral analysis the proposed method is a low-pass ﬁlter which leads graph partition into a few large parts.

Original: Moreover, we show a close connection between S 2 GC and
Revised: Moreover, we show a close connection between S 2 GC and

Original: APPNP (Klicpera et al., 2019a), and provide a generalized form of S 2 GC.
Revised: APPNP (Klicpera et al., 2019a), and provide a generalized form of S 2 GC.

Original: Our experimental results include node clustering, unsupervised node classiﬁcation, semi-supervised node classiﬁcation and supervised text classiﬁcation.
Revised: Our experimental results include node clustering, unsupervised node classiﬁcation, semi-supervised node classiﬁcation and supervised text classiﬁcation.

Original: We show that S 2 GC is highly competitive and often can signiﬁcantly outperform state-of-the-art methods.
Revised: We show that S 2 GC is highly competitive and often can signiﬁcantly outperform state-of-the-art methods.

Original:  P RELIMINARIES
Revised:  P RELIMINARIES

Original: Notations.
Revised: Notations.

Original: Let G =
Revised: Let G =

Original: ( V, E ) be a simple and connected undirected graph with n nodes and m edges.
Revised: ( V, E ) be a simple and connected undirected graph with n nodes and m edges.

Original: We use { 1 , · · · , n } to denote the node index of G , and d j denote the degree of node j in
Revised: We use { 1 , · · · , n } to denote the node index of G , and d j denote the degree of node j in

Original: G .
Revised: G .

Original: Let A be the adjacency matrix and D the diagonal degree matrix.
Revised: Let A be the adjacency matrix and D the diagonal degree matrix.

Original: Let (cid:101)
Revised: Let (cid:101)

Original: A =
Revised: A =

Original: A + I n denote the adjacency matrix with added self-loops and the corresponding diagonal degree matrix (cid:101) D where
Revised: A + I n denote the adjacency matrix with added self-loops and the corresponding diagonal degree matrix (cid:101) D where

Original: I n ∈ S n ++ is an identity matrix.
Revised: I n ∈ S n ++ is an identity matrix.

Original: Finally, let X ∈ R n × d denote the node feature matrix and each node v is associated with a d -dimensional feature vector X v .
Revised: Finally, let X ∈ R n × d denote the node feature matrix and each node v is associated with a d -dimensional feature vector X v .

Original: The normalized graph Laplacian matrix is deﬁned as L =
Revised: The normalized graph Laplacian matrix is deﬁned as L =

Original: I n − D − 1 / 2 AD − 1 / 2 ∈ S n + , a symmetric positive semideﬁnite matrix with eigendecomposition UΛU (cid:62) .
Revised: I n − D − 1 / 2 AD − 1 / 2 ∈ S n + , a symmetric positive semideﬁnite matrix with eigendecomposition UΛU (cid:62) .

Original: Here Λ is a diagonal matrix of the eigenvalues of L , and U ∈ R n × n is a unitary matrix that consists of the eigenvectors of L .
Revised: Here Λ is a diagonal matrix of the eigenvalues of L , and U ∈ R n × n is a unitary matrix that consists of the eigenvectors of L .

Original: Spectral Graph Convolution (Defferrard et al., 2016).
Revised: Spectral Graph Convolution (Defferrard et al., 2016).

Original: We consider spectral convolutions on graphs deﬁned as the multiplication of a signal x ∈ R n (a scalar for every node) with a ﬁlter g θ parameterized by θ ∈ R n in the Fourier domain, ie .:
Revised: We consider spectral convolutions on graphs deﬁned as the multiplication of a signal x ∈ R n (a scalar for every node) with a ﬁlter g θ parameterized by θ ∈ R n in the Fourier domain, ie .:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the parameter θ ∈ R n corresponds to a vector of spectral ﬁlter coefﬁcients.
Revised: where the parameter θ ∈ R n corresponds to a vector of spectral ﬁlter coefﬁcients.

Original: We can understand g θ as a function operating on eigenvalues of L , that is g ∗ θ ( Λ ) .
Revised: We can understand g θ as a function operating on eigenvalues of L , that is g ∗ θ ( Λ ) .

Original: To avoid eigendecomposition, g θ ( Λ ) can be approximated by a truncated expansion in terms of Chebyshev polynomials T k ( Λ ) up to the K -th order (Defferrard et al., 2016):
Revised: To avoid eigendecomposition, g θ ( Λ ) can be approximated by a truncated expansion in terms of Chebyshev polynomials T k ( Λ ) up to the K -th order (Defferrard et al., 2016):

Original: [Equation]
Revised: [Equation]

Original: with a rescaled ˜ Λ = 1 2 λ max Λ − I n where λ max denotes the largest eigenvalue of L and θ ∈ R K is now a vector of Chebyshev coefﬁcients.
Revised: with a rescaled ˜ Λ = 1 2 λ max Λ − I n where λ max denotes the largest eigenvalue of L and θ ∈ R K is now a vector of Chebyshev coefﬁcients.

Original: Vanila Graph Convolutional Network (GCN) (Kipf & Welling, 2016).
Revised: Vanila Graph Convolutional Network (GCN) (Kipf & Welling, 2016).

Original: The vanilla GCN can be regarded as a ﬁrst-order approximation of spectral graph convolutions.
Revised: The vanilla GCN can be regarded as a ﬁrst-order approximation of spectral graph convolutions.

Original: If one sets K = 1 , θ 0 = 2 , and θ 1 = − 1 for Eq.
Revised: If one sets K = 1 , θ 0 = 2 , and θ 1 = − 1 for Eq.

Original: they obtain the convolution operation g θ ( L ) ∗ x
Revised: they obtain the convolution operation g θ ( L ) ∗ x

Original: = ( I + D − 1 / 2 AD − 1 / 2 ) x .
Revised: = ( I + D − 1 / 2 AD − 1 / 2 ) x .

Original: Finally, by the renormalization trick, replacing the matrix
Revised: Finally, by the renormalization trick, replacing the matrix

Original: I + D − 1 / 2 AD − 1 / 2 by a normalized version (cid:101) T = (cid:101)
Revised: I + D − 1 / 2 AD − 1 / 2 by a normalized version (cid:101) T = (cid:101)

Original: D − 1 / 2 (cid:101)
Revised: D − 1 / 2 (cid:101)

Original: A (cid:101) D − 1 / 2 =
Revised: A (cid:101) D − 1 / 2 =

Original: ( D + I n ) − 1 / 2 ( A + I n )( D + I n ) − 1 / 2 leads to the Graph Convolutional
Revised: ( D + I n ) − 1 / 2 ( A + I n )( D + I n ) − 1 / 2 leads to the Graph Convolutional

Original: Layer:
Revised: Layer:

Original: [Equation]
Revised: [Equation]

Original: (3) where σ is a non-linear function, e.g . ReLU.
Revised: (3) where σ is a non-linear function, e.g . ReLU.

Original: Graph Diffusion Convolution (GDC) (Klicpera et al., 2019b).
Revised: Graph Diffusion Convolution (GDC) (Klicpera et al., 2019b).

Original: A generalized graph diffusion is given by the diffusion matrix
Revised: A generalized graph diffusion is given by the diffusion matrix

Original: [Equation]
Revised: [Equation]

Original: with the weighting coefﬁcients θ k and the generalized transition matrix T .
Revised: with the weighting coefﬁcients θ k and the generalized transition matrix T .

Original: Eq.
Revised: Eq.

Original: 4 can be regarded as related to the Taylor expansion of matrix-valued functions.
Revised: 4 can be regarded as related to the Taylor expansion of matrix-valued functions.

Original: Thus, the choice of θ k and T k must at least ensure that Eq.
Revised: Thus, the choice of θ k and T k must at least ensure that Eq.

Original: 4 converges.
Revised: 4 converges.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019b) provide two special cases as low-pass ﬁlters, ie .
Revised: (2019b) provide two special cases as low-pass ﬁlters, ie .

Original: heat kernel and the kernel based on random walk with restarts.
Revised: heat kernel and the kernel based on random walk with restarts.

Original: If S denote the adjacency matrix and D be the diagonal degree matrix of S then the corresponding graph diffusion convolution is deﬁned as D − 1 / 2 SD − 1 / 2 x .
Revised: If S denote the adjacency matrix and D be the diagonal degree matrix of S then the corresponding graph diffusion convolution is deﬁned as D − 1 / 2 SD − 1 / 2 x .

Original: Simple Graph Convolution (SGC) (Wu et al., 2019).
Revised: Simple Graph Convolution (SGC) (Wu et al., 2019).

Original: For a classical MPNN (Gilmer et al., 2017), in each layer the hidden representations are averaged among neighbors that are 1 -hop away.
Revised: For a classical MPNN (Gilmer et al., 2017), in each layer the hidden representations are averaged among neighbors that are 1 -hop away.

Original: This implies that each node in the k -th layer obtains feature information from all nodes that are k -hops away in the graph.
Revised: This implies that each node in the k -th layer obtains feature information from all nodes that are k -hops away in the graph.

Original: By hypothesizing that the non-linearity between GCN layers is not critical,
Revised: By hypothesizing that the non-linearity between GCN layers is not critical,

Original: SGC attempts to capture information from k -hops neighbourhood in the graph by applying the K -th power of the transition matrix in a single neural network layer.
Revised: SGC attempts to capture information from k -hops neighbourhood in the graph by applying the K -th power of the transition matrix in a single neural network layer.

Original: The SGC can be regarded as a special case of GDC without normalization by D − 1 / 2 if we set θ k = 1 and θ i<K = 0 for Eq. 5, and T =
Revised: The SGC can be regarded as a special case of GDC without normalization by D − 1 / 2 if we set θ k = 1 and θ i<K = 0 for Eq. 5, and T =

Original: ˜ T :
Revised: ˜ T :

Original: [Equation]
Revised: [Equation]

Original: Although SGC is an efﬁcient and effective method, increasing K leads to oversmoothing.
Revised: Although SGC is an efﬁcient and effective method, increasing K leads to oversmoothing.

Original: Thus, SGC uses similar K number of layers as GCN.
Revised: Thus, SGC uses similar K number of layers as GCN.

Original: We provide Theorem 1 to demonstrate that oversmoothing is a result of convergence to the stationary distribution in graph diffusion as time t → ∞ .
Revised: We provide Theorem 1 to demonstrate that oversmoothing is a result of convergence to the stationary distribution in graph diffusion as time t → ∞ .

Original: Theorem 1.
Revised: Theorem 1.

Original: (Chung & Graham, 1997) Let λ 2 denote second largest eigenvalue of transition matrix
Revised: (Chung & Graham, 1997) Let λ 2 denote second largest eigenvalue of transition matrix

Original: P = D − 1 A , p ( t ) be the probability distribution vector and π the stationary distribution.
Revised: P = D − 1 A , p ( t ) be the probability distribution vector and π the stationary distribution.

Original: If walk starts from the vertex i , p i (0) = 1 , then after t steps for every vertex:
Revised: If walk starts from the vertex i , p i (0) = 1 , then after t steps for every vertex:

Original: [Equation]
Revised: [Equation]

Original: Note that Eq.
Revised: Note that Eq.

Original: 6 converges to the stationary distribution only if the graph is non-bipartite.
Revised: 6 converges to the stationary distribution only if the graph is non-bipartite.

Original: APPNA.
Revised: APPNA.

Original: Klicpera et al.
Revised: Klicpera et al.

Original: (2019a)
Revised: (2019a)

Original: proposed APPNP which uses Personalized PageRank to derive a ﬁxed ﬁlter of order K .
Revised: proposed APPNP which uses Personalized PageRank to derive a ﬁxed ﬁlter of order K .

Original: Let f θ ( X ) denote the output of a two-layer fully connected neural network on the feature matrix X , the PPNP model is deﬁned as:
Revised: Let f θ ( X ) denote the output of a two-layer fully connected neural network on the feature matrix X , the PPNP model is deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: To avoid calculating the inverse of matrix (cid:101) T , Klicpera et al. (2019a) also proposes APPNP which replaces Eq.
Revised: To avoid calculating the inverse of matrix (cid:101) T , Klicpera et al. (2019a) also proposes APPNP which replaces Eq.

Original: 7 with an approximation derived by the truncated power iteration, deﬁned as:
Revised: 7 with an approximation derived by the truncated power iteration, deﬁned as:

Original: [Equation]
Revised: [Equation]

Original: where H (0) =
Revised: where H (0) =

Original: f θ ( X ) =
Revised: f θ ( X ) =

Original: ReLU ( XW ) .
Revised: ReLU ( XW ) .

Original: By decoupling the feature transformation and propagation steps, PPNP and APPNP aggregate information from multi-hop neighbors without increasing the number of layers in the neural network.
Revised: By decoupling the feature transformation and propagation steps, PPNP and APPNP aggregate information from multi-hop neighbors without increasing the number of layers in the neural network.

Original: [Figure]
Revised: [Figure]

Original:  M ETHODOLOGY
Revised:  M ETHODOLOGY

Original: In this section, ﬁrstly we brieﬂy discuss the spectral analysis and graph partitioning, and conclude the small eigenvalues of a Laplacian matrix control global clustering which partitions the graph into a few of large clusters.
Revised: In this section, ﬁrstly we brieﬂy discuss the spectral analysis and graph partitioning, and conclude the small eigenvalues of a Laplacian matrix control global clustering which partitions the graph into a few of large clusters.

Original: Secondly, we analyze the Markov diffusion kernel (Fouss et al., 2012) and note that the corresponding feature mapping function acts as a low-pass ﬁlter.
Revised: Secondly, we analyze the Markov Diffusion Kernel (Fouss et al., 2012) and note that the corresponding feature mapping function acts as a low-pass ﬁlter.

Original: Finally, based on the feature mapping function we present our Simple Spectral Graph Convolution network.
Revised: Finally, based on the feature mapping function we present our Simple Spectral Graph Convolution network.

Original: 3. S PECTRAL ANALYSIS AND GRAPH PARTITIONING
Revised: 3. S PECTRAL ANALYSIS AND GRAPH PARTITIONING

Original: Firstly, we introduce the deﬁnitions of expansion and k -way Cheeger constant.
Revised: Firstly, we introduce the deﬁnitions of expansion and k -way Cheeger constant.

Original: Deﬁnition 3.1.
Revised: Deﬁnition 3.1.

Original: Expansion:
Revised: Expansion:

Original: For a node subset S ⊆ V, φ ( S ) =
Revised: For a node subset S ⊆ V, φ ( S ) =

Original: [Equation]
Revised: [Equation]

Original: where E ( S ) is the set of edges with one node in S and vol ( S ) is the sum of degree of nodes in set S.
Revised: where E ( S ) is the set of edges with one node in S and vol ( S ) is the sum of degree of nodes in set S.

Original: Deﬁnition 3.2.
Revised: Deﬁnition 3.2.

Original: The k -way Cheeger constant is given as: ρ G ( k ) =
Revised: The k -way Cheeger constant is given as: ρ G ( k ) =

Original: min S 1 ,S 2 , ··· ,S k max { φ ( S i ) : i = { 1 , · · · , k }} where the minimum is over all collections of k non-empty disjoint subsets
Revised: min S 1 ,S 2 , ··· ,S k max { φ ( S i ) : i = { 1 , · · · , k }} where the minimum is over all collections of k non-empty disjoint subsets

Original: S 1 , S 2 , · · · , S k ⊆ V .
Revised: S 1 , S 2 , · · · , S k ⊆ V .

Original: According to the deﬁnitions, the expansion in Def.
Revised: According to the deﬁnitions, the expansion in Def.

Original: 3.1 describes the effect of graph partitioning according to subset S while the k -way Cheeger constant reﬂects the effect of the graph partitioning into k parts–the smaller the value the better the partitioning is.
Revised: 3.1 describes the effect of graph partitioning according to subset S while the k -way Cheeger constant reﬂects the effect of the graph partitioning into k parts–the smaller the value the better the partitioning is.

Original: Higher-order Cheeger’s inequality (Bandeira et al., 2013; Lee et al., 2014) bridges the gap between the network spectral analysis and graph partitioning by controlling the bounds of k -way Cheeger constant:
Revised: Higher-order Cheeger’s inequality (Bandeira et al., 2013; Lee et al., 2014) bridges the gap between the network spectral analysis and graph partitioning by controlling the bounds of k -way Cheeger constant:

Original: [Equation]
Revised: [Equation]

Original: where λ k is the k-th eigenvalue of the normalized Laplacian matrix and
Revised: where λ k is the k-th eigenvalue of the normalized Laplacian matrix and

Original: 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n .
Revised: 0 = λ 1 ≤ λ 2 ≤ · · · ≤ λ n .

Original: From inequality 9, we can conclude that small (large) eigenvalues control global clustering (local smoothing) effect of the graph partitioned into a few large parts (many small parts).
Revised: From inequality 9, we can conclude that small (large) eigenvalues control global clustering (local smoothing) effect of the graph partitioned into a few large parts (many small parts).

Original: 3. M ARKOV D IFFUSION K ERNEL
Revised: 3. M ARKOV D IFFUSION K ERNEL

Original: Before we deﬁne the diffusion distance, we brieﬂy introduce the intuition behind it: two nodes are considered similar when they diffuse in a similar way through the graph, and therefore when they inﬂuence the other nodes in a similar manner (Fouss et al., 2012).
Revised: Before we deﬁne the diffusion distance, we brieﬂy introduce the intuition behind it: two nodes are considered similar when they diffuse in a similar way through the graph, and therefore when they inﬂuence the other nodes in a similar manner (Fouss et al., 2012).

Original: In other words, two nodes are close if they are in the same cluster which has a consistent local structure.
Revised: In other words, two nodes are close if they are in the same cluster which has a consistent local structure.

Original: More precisely, the diffusion distance at time K between nodes i and j is deﬁned as follows:
Revised: More precisely, the diffusion distance at time K between nodes i and j is deﬁned as follows:

Original: [Equation]
Revised: [Equation]

Original: where the average visiting rate x i ( K ) after K steps for a process that started at time k = 0 is computed as follows:
Revised: where the average visiting rate x i ( K ) after K steps for a process that started at time k = 0 is computed as follows:

Original: [Equation]
Revised: [Equation]

Original: By deﬁning Z ( K ) = 1 K (cid:80) Kk
Revised: By deﬁning Z ( K ) = 1 K (cid:80) Kk

Original: =
Revised: =

Original: T k , we reformulate Eq.
Revised: T k , we reformulate Eq.

Original: 10 as:
Revised: 10 as:

Original: [Equation]
Revised: [Equation]

Original: The underlying feature map of Markov Diffusion Kernel (MDK) is given as Z ( K ) x
Revised: The underlying feature map of Markov Diffusion Kernel (MDK) is given as Z ( K ) x

Original: i (0) for node i .
Revised: i (0) for node i .

Original: The effect of the linear projection Z ( K ) (ﬁlter) acting on spectrum as f ( λ ) = 1 K (cid:80)
Revised: The effect of the linear projection Z ( K ) (ﬁlter) acting on spectrum as f ( λ ) = 1 K (cid:80)

Original: Kk =

Revised: Kk =

Original: λ k is plotted in Figure 1, from which we observe the following two properties: (i) Z ( K ) preserves leading (large) eigenvalues of T and (ii) the higher K is the stricter the low-pass ﬁlter becomes.
Revised: λ k is plotted in Figure 1, from which we observe the following two properties: (i) Z ( K ) preserves leading (large) eigenvalues of T and (ii) the higher K is the stricter the low-pass ﬁlter becomes.

Original: In other words, as K grows, this ﬁlter tries to approximate a low-rank positive semi-deﬁnite matrix by keeping the largest positive eigenvalues.
Revised: In other words, as K grows, this ﬁlter tries to approximate a low-rank positive semi-deﬁnite matrix by keeping the largest positive eigenvalues.

Original: Note the relationship between the normalized Laplacian matrix L and the corresponding normalized adjacent matrix T which is L =
Revised: Note the relationship between the normalized Laplacian matrix L and the corresponding normalized adjacent matrix T which is L =

Original: I − T , and thus keeping large positive eigenvalues for T equals keeping small eigenvalues for L .
Revised: I − T , and thus keeping large positive eigenvalues for T equals keeping small eigenvalues for L .

Original: 3. S IMPLE S PECTRAL G RAPH C ONVOLUTION
Revised: 3. S IMPLE S PECTRAL G RAPH C ONVOLUTION

Original: Based on the aforementioned Markov Diffusion Kernel, we propose the Simple Spectral Graph
Revised: Based on the aforementioned Markov Diffusion Kernel, we propose the Simple Spectral Graph

Original: Convolution (S 2 GC) network with a softmax after a linear layer:
Revised: Convolution (S 2 GC) network with a softmax after a linear layer:

Original: [Equation]
Revised: [Equation]

Original: As K → ∞ , H = (cid:80) ∞ k =

Revised: As K → ∞ , H = (cid:80) ∞ k =

Original: (cid:101) T k is the optimal diffused representation of the normalized Laplacian
Revised: (cid:101) T k is the optimal diffused representation of the normalized Laplacian

Original: Regularization problem (Chapelle et al., 2006):
Revised: Regularization problem (Chapelle et al., 2006):

Original: [Equation]
Revised: [Equation]

Original: where each vector h i denotes the i -th row of H .
Revised: where each vector h i denotes the i -th row of H .

Original: The S 2 GC can be regarded as a special case of GDC without normalization by D − 1 / 2 they use if we set θ i
Revised: The S 2 GC can be regarded as a special case of GDC without normalization by D − 1 / 2 they use if we set θ i

Original: = 1 /K and for Eq.
Revised: = 1 /K and for Eq.

Original: 5 to obtain the convolution operation g θ ( (cid:101) T ) ∗ x = 1 K
Revised: 5 to obtain the convolution operation g θ ( (cid:101) T ) ∗ x = 1 K

Original: (cid:80)
Revised: (cid:80)

Original: Kk =

Revised: Kk =

Original: (cid:101) T k x which can be decomposed into two terms: 1 K (cid:80)
Revised: (cid:101) T k x which can be decomposed into two terms: 1 K (cid:80)

Original: Kk =1 (cid:101) T k X and 1 K X .
Revised: Kk =1 (cid:101) T k X and 1 K X .

Original: When setting K = 2 and (cid:101) P = (cid:101) T + (cid:101) T 2for Eq.
Revised: When setting K = 2 and (cid:101) P = (cid:101) T + (cid:101) T 2for Eq.

Original: 13, we obtain:
Revised: 13, we obtain:

Original: [Equation]
Revised: [Equation]

Original: where H (0) =
Revised: where H (0) =

Original: XW .
Revised: XW .

Original: It is easy to observe that it is a special case of APPNA (Klicpera et al., 2019a).
Revised: It is easy to observe that it is a special case of APPNA (Klicpera et al., 2019a).

Original: Speciﬁcally, it is an expansion containing the ﬁrst step of APPNA without ReLU for f θ ( X ) , and the propagation matrix is a 2 -hops matrix (cid:101) P
Revised: Speciﬁcally, it is an expansion containing the ﬁrst step of APPNA without ReLU for f θ ( X ) , and the propagation matrix is a 2 -hops matrix (cid:101) P

Original: = (cid:101) T
Revised: = (cid:101) T

Original: + (cid:101) T 2.
Revised: + (cid:101) T 2.

Original: We have shown Eq.
Revised: We have shown Eq.

Original: 13 implicitly consider that attributes can be propagated back to themselves due to (cid:101) T 

Revised: 13 implicitly consider that attributes can be propagated back to themselves due to (cid:101) T

Original: = I which prevents oversmoothing (Kipf & Welling, 2016).
Revised: = I which prevents oversmoothing (Kipf & Welling, 2016).

Original: Furthermore, by introducing the α parameter to preserve the node-self information as in PPNP and APPNP, we can generalize the
Revised: Furthermore, by introducing the α parameter to preserve the node-self information as in PPNP and APPNP, we can generalize the

Original: Eq. 13 and explicitly consider the following form:
Revised: Eq. 13 and explicitly consider the following form:

Original: [Equation]
Revised: [Equation]

Original: where α ∈ (0 , 1] .
Revised: where α ∈ (0 , 1] .

Original: Relation with APPNP.
Revised: Relation with APPNP.

Original: If we deﬁne H 0 =
Revised: If we deﬁne H 0 =

Original: XW then Eq.
Revised: XW then Eq.

Original: 16 can be reformulated as:
Revised: 16 can be reformulated as:

Original: [Equation]
Revised: [Equation]

Original: Thus, the Eq.
Revised: Thus, the Eq.

Original: 16 can be regarded as an aggregation on the ﬁrst step of APPNP with a propagation matrix of different order.
Revised: 16 can be regarded as an aggregation on the ﬁrst step of APPNP with a propagation matrix of different order.

Original: In spectral analysis, the diffusion projection matrix of APPNP is given as
Revised: In spectral analysis, the diffusion projection matrix of APPNP is given as

Original: [Equation]
Revised: [Equation]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: Relation to Jumping Knowledge Network.
Revised: Relation to Jumping Knowledge Network.

Original: In (Xu et al., 2018b), Jumping Knowledge Networks (JKN) combine intermediate node representations from each layer by forwarding them to the output of the ﬁnal layer.
Revised: In (Xu et al., 2018b), Jumping Knowledge Networks (JKN) combine intermediate node representations from each layer by forwarding them to the output of the ﬁnal layer.

Original: Eq.
Revised: Eq.

Original: 13 can be regarded as a variant of JKN with the average aggregation function.
Revised: 13 can be regarded as a variant of JKN with the average aggregation function.

Original: However, the mean-pooling aggregation is not mentioned in (Xu et al., 2018b) which use the max-pooling, concatenation- and attention-based aggregation layers.
Revised: However, the mean-pooling aggregation is not mentioned in (Xu et al., 2018b) which use the max-pooling, concatenation- and attention-based aggregation layers.

Original: For Eq.
Revised: For Eq.

Original: 17, the intermediate representation for the mean-pooling aggregation function is not the same as in the next layers.
Revised: 17, the intermediate representation for the mean-pooling aggregation function is not the same as in the next layers.

Original: Speciﬁcally, for the k -th layer, the input is (cid:101) T k − 1 X while the intermediate representation of the (k-1)-th layer is (1 − α ) (cid:101) T k − 1 X + α X .
Revised: Speciﬁcally, for the k -th layer, the input is (cid:101) T k − 1 X while the intermediate representation of the (k-1)-th layer is (1 − α ) (cid:101) T k − 1 X + α X .

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: In this section, we evaluate the proposed method on four different tasks: node clustering, community prediction, semi-supervised node classiﬁcation and text classiﬁcation.
Revised: In this section, we evaluate the proposed method on four different tasks: node clustering, community prediction, semi-supervised node classiﬁcation and text classiﬁcation.

Original: 4. N ODE C LUSTERING
Revised: 4. N ODE C LUSTERING

Original: We compare S 2 GC with three kinds of clustering methods: (i) Methods that only use node features: k-means and spectral clustering (spectral-f) that constructs a similarity matrix with the node features by a linear kernel.
Revised: We compare S 2 GC with three kinds of clustering methods: (i) Methods that only use node features: k-means and spectral clustering (spectral-f) that constructs a similarity matrix with the node features by a linear kernel.

Original: (ii) Structural clustering methods that only use graph structures: spectral clustering (spectral-g) that takes the node adjacency matrix as the similarity matrix, DeepWalk (Perozzi et al., 2014), and Deep Neural Networks for Graph Representations (DNGR)
Revised: (ii) Structural clustering methods that only use graph structures: spectral clustering (spectral-g) that takes the node adjacency matrix as the similarity matrix, DeepWalk (Perozzi et al., 2014), and Deep Neural Networks for Graph Representations (DNGR)

Original: (Cao et al., 2016), and (iii) Attributed graph clustering methods that utilize both node features and graph structures: Graph Autoencoder (GAE) and Graph Variational Autoencoder (VGAE)
Revised: (Cao et al., 2016), and (iii) Attributed graph clustering methods that utilize both node features and graph structures: Graph Autoencoder (GAE) and Graph Variational Autoencoder (VGAE)

Original: (Kipf & Welling, 2016), and Adversarially Regularized Graph Autoencoder (ARGE), Variational Graph Autoencoder (ARVGE) (Pan et al., 2018) and AGC (Zhang et al., 2019).
Revised: (Kipf & Welling, 2016), and Adversarially Regularized Graph Autoencoder (ARGE), Variational Graph Autoencoder (ARVGE) (Pan et al., 2018) and AGC (Zhang et al., 2019).

Original: To evaluate the clustering performance, three performance measures are adopted: clustering Accuracy (Acc), Normalized Mutual Information (NMI) and macro F1-score (F1).
Revised: To evaluate the clustering performance, three performance measures are adopted: clustering Accuracy (Acc), Normalized Mutual Information (NMI) and macro F1-score (F1).

Original: We run each method 10 times on four datasets: Cora, CiteSeer,
Revised: We run each method 10 times on four datasets: Cora, CiteSeer,

Original: PubMed, and Wiki, and we report the average clustering results in Table 2, where top-1 results are highlighted in bold.
Revised: PubMed, and Wiki, and we report the average clustering results in Table 2, where top-1 results are highlighted in bold.

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table]
Revised: [Table]

Original:  C OMMUNITY P REDICTION
Revised:  C OMMUNITY P REDICTION

Original: We supplement our social network analysis by using S 2 GC to inductively predict the community structure on Reddit, a large scale dataset as shown in Table 3 which cannot be processed by the vanilla GCN Kipf & Welling (2016) and GDC (Klicpera et al., 2019b) due to the memory issues.
Revised: We supplement our social network analysis by using S 2 GC to inductively predict the community structure on Reddit, a large scale dataset as shown in Table 3 which cannot be processed by the vanilla GCN Kipf & Welling (2016) and GDC (Klicpera et al., 2019b) due to the memory issues.

Original: On the Reddit dataset, we train S 2 GC with L-BFGS using no regularization, and we set K = 5 and α = 0 .
Revised: On the Reddit dataset, we train S 2 GC with L-BFGS using no regularization, and we set K = 5 and α = 0 .

Original: We evaluate S 2 GC inductively according to protocol (Chen et al., 2018).
Revised: We evaluate S 2 GC inductively according to protocol (Chen et al., 2018).

Original: We train S 2 GC on a subgraph comprising only training nodes and test on the original graph.
Revised: We train S 2 GC on a subgraph comprising only training nodes and test on the original graph.

Original: On all datasets, we tune the number of epochs based on both convergence behavior and the obtained validation accuracy.
Revised: On all datasets, we tune the number of epochs based on both convergence behavior and the obtained validation accuracy.

Original: For Reddit, we compare S 2 GC to the reported performance of supervised and unsupervised variants of GraphSAGE (Hamilton et al., 2017), FastGCN (Chen et al., 2018), SGC (Wu et al., 2019) and
Revised: For Reddit, we compare S 2 GC to the reported performance of supervised and unsupervised variants of GraphSAGE (Hamilton et al., 2017), FastGCN (Chen et al., 2018), SGC (Wu et al., 2019) and

Original: DGI (Velickovic et al., 2019).
Revised: DGI (Velickovic et al., 2019).

Original: Table 1 also highlights the setting of the feature extraction step for each method.
Revised: Table 1 also highlights the setting of the feature extraction step for each method.

Original: Note that S 2 GC and SGC involve no learning because they do not learn any parameters to extract features.
Revised: Note that S 2 GC and SGC involve no learning because they do not learn any parameters to extract features.

Original: The logistic regression is used as a classiﬁer for both unsupervised and no-learning approaches to train with labels afterward.
Revised: The logistic regression is used as a classiﬁer for both unsupervised and no-learning approaches to train with labels afterward.

Original: 4. S EMI -S UPERVISED N ODE C LASSIFICATION
Revised: 4. S EMI -S UPERVISED N ODE C LASSIFICATION

Original: For the semi-supervised node classiﬁcation task, we apply the standard ﬁxed training/validation/testing split (Yang et al., 2016) on the Cora, Citeseer, and Pubmed datasets, with 20 nodes per class for training, 500 nodes for validation and 1,000 nodes for testing.
Revised: For the semi-supervised node classiﬁcation task, we apply the standard ﬁxed training/validation/testing split (Yang et al., 2016) on the Cora, Citeseer, and Pubmed datasets, with 20 nodes per class for training, 500 nodes for validation and 1,000 nodes for testing.

Original: For baselines,
Revised: For baselines,

Original: We include three state-of-the-art shallow models: GCN (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2017), FastGCN (Chen et al., 2018), APPNP (Klicpera et al., 2019a), Mixhop (Abu-El-Haija et al., 2019), SGC (Wu et al., 2019), DGI (Velickovic et al., 2019) and GIN (Xu et al., 2018a).
Revised: We include three state-of-the-art shallow models: GCN (Kipf & Welling, 2016), GAT (Veliˇckovi´c et al., 2017), FastGCN (Chen et al., 2018), APPNP (Klicpera et al., 2019a), Mixhop (Abu-El-Haija et al., 2019), SGC (Wu et al., 2019), DGI (Velickovic et al., 2019) and GIN (Xu et al., 2018a).

Original: We use the Adam SGD optimizer (Kingma & Ba, 2014) with a learning rate of 0.02 to train S 2 GC.
Revised: We use the Adam SGD optimizer (Kingma & Ba, 2014) with a learning rate of 0.02 to train S 2 GC.

Original: We set α = 0 .
Revised: We set α = 0 .

Original: 05 and K
Revised: 05 and K

Original: = 16 on all datasets.
Revised: = 16 on all datasets.

Original: We perform a grid search to tune the weight decay for models with different depths based on the accuracy and loss of the validation set.
Revised: We perform a grid search to tune the weight decay for models with different depths based on the accuracy and loss of the validation set.

Original: More details of hyperparameters are listed in the supplementary material.
Revised: More details of hyperparameters are listed in the supplementary material.

Original: 4. T EXT C LASSIFICATION
Revised: 4. T EXT C LASSIFICATION

Original: Text classiﬁcation predicts the labels of documents.
Revised: Text classiﬁcation predicts the labels of documents.

Original: Yao et al.
Revised: Yao et al.

Original: (2019) use a 2-layer GCN to achieve state-of-the-art results by creating a corpus-level graph which treats both documents and words as nodes in a graph.
Revised: (2019) use a 2-layer GCN to achieve state-of-the-art results by creating a corpus-level graph which treats both documents and words as nodes in a graph.

Original: Word-to-word edge weights are given by Point-wise Mutual Information (PMI) and word-document edge weights are given by normalized TF-IDF scores.
Revised: Word-to-word edge weights are given by Point-wise Mutual Information (PMI) and word-document edge weights are given by normalized TF-IDF scores.

Original: [Table]
Revised: [Table]

Original: [Table]
Revised: [Table]

Original: Setting and baseline We ran our experiments on ﬁve widely used benchmark corpora including movie review (MR), 20-Newsgroups (20NG), Ohsumed, R52 and R8 of Reuters 21578.
Revised: Setting and baseline We ran our experiments on ﬁve widely used benchmark corpora including movie review (MR), 20-Newsgroups (20NG), Ohsumed, R52 and R8 of Reuters 21578.

Original: We ﬁrst preprocessed all the datasets by cleaning and tokenizing text as (Kim, 2014).
Revised: We ﬁrst preprocessed all the datasets by cleaning and tokenizing text as (Kim, 2014).

Original: We then removed stop words deﬁned in NLTK6 and low frequency words appearing less than 5 times for 20NG, R8, R and Ohsumed.
Revised: We then removed stop words deﬁned in NLTK6 and low frequency words appearing less than 5 times for 20NG, R8, R and Ohsumed.

Original: We compare two state-of-the-art models with our method: GCN (Kipf & Welling, 2016) and SGC (Wu et al., 2019).
Revised: We compare two state-of-the-art models with our method: GCN (Kipf & Welling, 2016) and SGC (Wu et al., 2019).

Original: The statistics of the preprocessed datasets are summarized in
Revised: The statistics of the preprocessed datasets are summarized in

Original: Table 5.
Revised: Table 5.

Original: Table 6 shows that an S 2 GC rivals their models on 5 benchmark datasets.
Revised: Table 6 shows that an S 2 GC rivals their models on 5 benchmark datasets.

Original: Parameters.
Revised: Parameters.

Original: We follow the setting of Text GCN (Yao et al., 2019) that includes experiments on four widely used benchmark corpora such as 20-Newsgroups (20NG), Ohsumed, R52 and R8 of
Revised: We follow the setting of Text GCN (Yao et al., 2019) that includes experiments on four widely used benchmark corpora such as 20-Newsgroups (20NG), Ohsumed, R52 and R8 of

Original: Reuters 21578.
Revised: Reuters 21578.

Original: For Text GCN, SGC, and our approach, the embedding size of the ﬁrst convolution layer is 200 and the window size is 20.
Revised: For Text GCN, SGC, and our approach, the embedding size of the ﬁrst convolution layer is 200 and the window size is 20.

Original: We set the learning rate to 0.02, dropout rate to 0.5 and the decay rate to 0.
Revised: We set the learning rate to 0.02, dropout rate to 0.5 and the decay rate to 0.

Original: The 10% of training set is randomly selected for validation.
Revised: The 10% of training set is randomly selected for validation.

Original: Following (Kipf & Welling, 2016), we trained our method and Text GCN for a maximum of 200 epochs using the
Revised: Following (Kipf & Welling, 2016), we trained our method and Text GCN for a maximum of 200 epochs using the

Original: Adam (Kingma & Ba, 2014) optimizer, and we stop training if the validation loss does not decrease for 10 consecutive epochs.
Revised: Adam (Kingma & Ba, 2014) optimizer, and we stop training if the validation loss does not decrease for 10 consecutive epochs.

Original: The text graph was built according to steps detailed in the supplementary material.
Revised: The text graph was built according to steps detailed in the supplementary material.

Original:  C ONCLUSIONS
Revised:  C ONCLUSIONS

Original: We have proposed Simple Spectral Graph Convolution (S 2 GC), a method based on the Markov
Revised: We have proposed Simple Spectral Graph Convolution (S 2 GC), a method based on the Markov

Original: Diffusion Kernel (Section 3.2) whose feature maps emerge from the normalized Laplacian Regularization problem (Section 3.3).
Revised: Diffusion Kernel (Section 3.2) whose feature maps emerge from the normalized Laplacian Regularization problem (Section 3.3).

Original: The theoretical analysis shows that S 2 GC is able to express a special case of the generalized graph diffusion (as shown in Eq. 4) with the same ﬁxed coefﬁcient ( θ i = 1 /K ) for different order polynomial ﬁlters, and evaluations show our model is powerful.
Revised: The theoretical analysis shows that S 2 GC is able to express a special case of the generalized graph diffusion (as shown in Eq. 4) with the same ﬁxed coefﬁcient ( θ i = 1 /K ) for different order polynomial ﬁlters, and evaluations show our model is powerful.

Original: We have shown there exist a close connection between S 2 GC and other methods such as SGC, APPNP and JKN by analyzing spectral properties and implementation of each model.
Revised: We have shown there exist a close connection between S 2 GC and other methods such as SGC, APPNP and JKN by analyzing spectral properties and implementation of each model.

Original: S 2 GC inherits many of the strengths of spectral methods and none of their weaknesses, e.g . oversmoothing.
Revised: S 2 GC inherits many of the strengths of spectral methods and none of their weaknesses, e.g . oversmoothing.

Original: We have conducted extensive and rigorous experiments that have shown that S 2 GC is competitive and frequently outperforms many state-of-the-art methods on unsupervised, semi-supervised and supervised tasks and several popular datasets
Revised: We have conducted extensive and rigorous experiments that have shown that S 2 GC is competitive and frequently outperforms many state-of-the-art methods on unsupervised, semi-supervised and supervised tasks and several popular datasets

Original: A A PPENDIX A. N ODE C LUSTERING
Revised: A S UPPLEMENTARY M ATERIAL A. N ODE C LUSTERING

Original: For S 2 GC and AGC, we set max iterations to 60 .
Revised: For S 2 GC and AGC, we set max iterations to 60 .

Original: For other baselines, we follow the parameter settings in the original papers.
Revised: For other baselines, we follow the parameter settings in the original papers.

Original: In particular, for DeepWalk, the number of random walks is 10, the number of latent dimensions for each node is 128, and the path length of each random walk is 80.
Revised: In particular, for DeepWalk, the number of random walks is 10, the number of latent dimensions for each node is 128, and the path length of each random walk is 80.

Original: For DNGR, the autoencoder is of three layers with 512 neurons and 256 neurons in the hidden layers respectively.
Revised: For DNGR, the autoencoder is of three layers with 512 neurons and 256 neurons in the hidden layers respectively.

Original: For GAE and VGAE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer, and train the encoders for 200 iterations using the Adam optimizer with learning rate equal 0.01.
Revised: For GAE and VGAE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer, and train the encoders for 200 iterations using the Adam optimizer with learning rate equal 0.01.

Original: For ARGE and ARVGE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer.
Revised: For ARGE and ARVGE, we construct encoders with a 32-neuron hidden layer and a 16-neuron embedding layer.

Original: The discriminators are built by two hidden layers with 16 and 64 neurons respectively.
Revised: The discriminators are built by two hidden layers with 16 and 64 neurons respectively.

Original: On Cora, Citeseer and Wiki, we train the autoencoder-related models of ARGE and ARVGE for 200 iterations with the Adam optimizer, with the encoder and discriminator learning rates both set as 0.001; on Pubmed, we train them for 2000 iterations with the encoder learning rate 0.001 and the discriminator learning rate 0.008.
Revised: On Cora, Citeseer and Wiki, we train the autoencoder-related models of ARGE and ARVGE for 200 iterations with the Adam optimizer, with the encoder and discriminator learning rates both set as 0.001; on Pubmed, we train them for 2000 iterations with the encoder learning rate 0.001 and the discriminator learning rate 0.008.

Original: A. T EXT C LASSIFICATION
Revised: A. T EXT C LASSIFICATION

Original: The 20NG dataset1 (bydate version) contains 18,846 documents evenly categorized into 20 different categories.
Revised: The 20NG dataset1 (bydate version) contains 18,846 documents evenly categorized into 20 different categories.

Original: In total, 11,314 documents are in the training set and 7,532 documents are in the test set.
Revised: In total, 11,314 documents are in the training set and 7,532 documents are in the test set.

Original: The Ohsumed corpus2 is from the MEDLINE database, which is a bibliographic database of important medical literature maintained by the National Library of Medicine.
Revised: The Ohsumed corpus2 is from the MEDLINE database, which is a bibliographic database of important medical literature maintained by the National Library of Medicine.

Original: In this work, we used the 13,929 unique cardiovascular diseases abstracts in the ﬁrst 20,000 abstracts of the year 1991.
Revised: In this work, we used the 13,929 unique cardiovascular diseases abstracts in the ﬁrst 20,000 abstracts of the year 1991.

Original: Each document in the set has one or more associated categories from the 23 disease categories.
Revised: Each document in the set has one or more associated categories from the 23 disease categories.

Original: As we focus on single-label text classiﬁcation, the documents belonging to multiple categories are excluded so that 7,400 documents belonging to only one category remain.
Revised: As we focus on single-label text classiﬁcation, the documents belonging to multiple categories are excluded so that 7,400 documents belonging to only one category remain.

Original: 3,357 documents are in the training set and 4,043 documents are in the test set.
Revised: 3,357 documents are in the training set and 4,043 documents are in the test set.

Original: R52 and R8 (all-terms version) are two subsets of the Reuters 21578 dataset.
Revised: R52 and R8 (all-terms version) are two subsets of the Reuters 21578 dataset.

Original: R8 has 8 categories, and was split to 5,485 training and 2,189 test documents.
Revised: R8 has 8 categories, and was split to 5,485 training and 2,189 test documents.

Original: R52 has 52 categories, and was split to 6,532 training and 2,568 test documents.
Revised: R52 has 52 categories, and was split to 6,532 training and 2,568 test documents.

Original: MR is a movie review dataset for binary sentiment classiﬁcation, in which each review only contains one sentence (Pang & Lee, 2005)
Revised: MR is a movie review dataset for binary sentiment classiﬁcation, in which each review only contains one sentence (Pang & Lee, 2005)

Original: The corpus has 5,331 positive and 5,331 negative reviews.
Revised: The corpus has 5,331 positive and 5,331 negative reviews.

Original: We used the training/test split in (Tang et al., 2015).
Revised: We used the training/test split in (Tang et al., 2015).

Original: A.2. T EXT G RAPH
Revised: A.2. T EXT G RAPH

Original: To convert text classiﬁcation into the node classiﬁcation on graph, there are two relationships considered when forming graphs: (i) the relation between documents and words and (ii) the connection between words.
Revised: To convert text classiﬁcation into the node classiﬁcation on graph, there are two relationships considered when forming graphs: (i) the relation between documents and words and (ii) the connection between words.

Original: For the ﬁrst type of relations, we build edges among word nodes and document nodes based on the word occurrence in documents.
Revised: For the ﬁrst type of relations, we build edges among word nodes and document nodes based on the word occurrence in documents.

Original: The weight of the edge between a document node and a word node is the Term Frequency-Inverse Document Frequency (Rajaraman & Ullman, 2011) (TF-IDF) of the word in the document applied to build the Docs-words graph.
Revised: The weight of the edge between a document node and a word node is the Term Frequency-Inverse Document Frequency (Rajaraman & Ullman, 2011) (TF-IDF) of the word in the document applied to build the Docs-words graph.

Original: For the second type of relations, we build edges in graph among word co-occurrences across the whole corpus.
Revised: For the second type of relations, we build edges in graph among word co-occurrences across the whole corpus.

Original: To utilize the global word co-occurrence information, we use a ﬁxed-size sliding window on all documents in the corpus to gather co-occurrence statistics.
Revised: To utilize the global word co-occurrence information, we use a ﬁxed-size sliding window on all documents in the corpus to gather co-occurrence statistics.

Original: Point-wise Mutual Information (Church &
Revised: Point-wise Mutual Information (Church &

Original: Hanks, 1990) (PMI), a popular measure for word associations, is used to calculate weights between two word nodes according to the following deﬁnition:
Revised: Hanks, 1990) (PMI), a popular measure for word associations, is used to calculate weights between two word nodes according to the following deﬁnition:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where p ( i, j ) =
Revised: where p ( i, j ) =

Original: W ( i,j ) W , p ( i ) =
Revised: W ( i,j ) W , p ( i ) =

Original: W ( i ) W .
Revised: W ( i ) W .

Original: # W ( i ) is the number of sliding windows in a corpus that contain word i , # W ( i, j ) is the number of sliding windows that contain both word i and word j , and # W is the total number of sliding windows in the corpus.
Revised: # W ( i ) is the number of sliding windows in a corpus that contain word i , # W ( i, j ) is the number of sliding windows that contain both word i and word j , and # W is the total number of sliding windows in the corpus.

Original: A positive PMI value implies a high semantic correlation of words in a corpus, while a negative PMI value indicates little or no semantic correlation in the corpus.
Revised: A positive PMI value implies a high semantic correlation of words in a corpus, while a negative PMI value indicates little or no semantic correlation in the corpus.

Original: Therefore, we only add edges between word pairs with positive
Revised: Therefore, we only add edges between word pairs with positive

Original: PMI values:
Revised: PMI values:

Original: [Equation]
Revised: [Equation]

Original: or
Revised: or

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation]
Revised: [Equation]

Original: are words
Revised: are words

Original: [Equation]
Revised: [Equation]

Original: PMI
Revised: PMI

Original: [Equation]
Revised: [Equation]

Original: i
Revised: i

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: ij
Revised: ij

Original: [Equation]
Revised: [Equation]

Original: i is document
Revised: i is document

Original: [Equation]
Revised: [Equation]

Original: j is word
Revised: j is word

Original: [Equation]
Revised: [Equation]

Original: if
Revised: if

Original: [Equation]
Revised: [Equation]