Original: N ETWORK OF G RAPH C ONVOLUTIONAL N ETWORKS T RAINED ON R ANDOM W ALKS
Revised: N ETWORK OF G RAPH C ONVOLUTIONAL N ETWORKS T RAINED ON R ANDOM W ALKS

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph Convolutional Networks (GCNs) are a recently proposed architecture which has had success in semi-supervised learning on graph-structured data.
Revised: Graph Convolutional Networks (GCNs) are a recently proposed architecture which has had success in semi-supervised learning on graph-structured data.

Original: At the same time, unsupervised learning of graph embeddings has beneﬁted from the information contained in random walks.
Revised: At the same time, unsupervised learning of graph embeddings has beneﬁted from the information contained in random walks.

Original: In this paper we propose a model,
Revised: In this paper we propose a model,

Original: Network of GCNs (N-GCN), which marries these two lines of work.
Revised: Network of GCNs (N-GCN), which marries these two lines of work.

Original: At its core,
Revised: At its core,

Original: N-GCN trains multiple instances of GCNs over node pairs discovered at different distances in random walks and learns a combination of the instance outputs which optimizes the classiﬁcation objective.
Revised: N-GCN trains multiple instances of GCNs over node pairs discovered at different distances in random walks and learns a combination of the instance outputs which optimizes the classiﬁcation objective.

Original: Our experiments show that our proposed NGCN model achieves state-of-the-art performance on all of the challenging node classiﬁcation tasks we consider: Cora, Citeseer, Pubmed and PPI.
Revised: Our experiments show that our proposed NGCN model achieves state-of-the-art performance on all of the challenging node classiﬁcation tasks we consider: Cora, Citeseer, Pubmed and PPI.

Original: In addition, our proposed method has other desirable properties, including generalization to recently proposed semi-supervised learning methods such as GraphSAGE, allowing us to propose N-SAGE, and resilience to adversarial input perturbations.
Revised: In addition, our proposed method has other desirable properties, including generalization to recently proposed semi-supervised learning methods such as GraphSAGE, allowing us to propose N-SAGE, and resilience to adversarial input perturbations.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Semi-supervised learning on graphs is important in many real-world applications, where the goal is to recover labels for all nodes given only a fraction of labeled ones.
Revised: Semi-supervised learning on graphs is important in many real-world applications, where the goal is to recover labels for all nodes given only a fraction of labeled ones.

Original: Some applications include social networks, where one wishes to predict user interests, or in health care, where one wishes to predict whether a patient should be screened for cancer.
Revised: Some applications include social networks, where one wishes to predict user interests, or in health care, where one wishes to predict whether a patient should be screened for cancer.

Original: In many such cases, collecting node labels can be prohibitive.
Revised: In many such cases, collecting node labels can be prohibitive.

Original: However, edges between nodes can be easier to obtain, either using an explicit graph (e.g. social network) or implicitly by calculating pairwise similarities (e.g. using a patient-patient similarity kernel, Merdan et al., 2017).
Revised: However, edges between nodes can be easier to obtain, either using an explicit graph (e.g. social network) or implicitly by calculating pairwise similarities (e.g. using a patient-patient similarity kernel, Merdan et al., 2017).

Original: Convolutional Neural Networks (LeCun et al., 1998) learn location-invariant hierarchical ﬁlters, enabling signiﬁcant improvements on Computer Vision tasks (Krizhevsky et al., 2012;
Revised: Convolutional Neural Networks (LeCun et al., 1998) learn location-invariant hierarchical ﬁlters, enabling signiﬁcant improvements on Computer Vision tasks (Krizhevsky et al., 2012;

Original: Szegedy et al., 2015; He et al., 2016).
Revised: Szegedy et al., 2015; He et al., 2016).

Original: This success has motivated researchers (Bruna et al., 2014) to extend convolutions from spatial (i.e. regular lattice) domains to graph-structured (i.e. irregular) domains, yielding a class of algorithms known as Graph Convolutional Networks (GCNs).
Revised: This success has motivated researchers (Bruna et al., 2014) to extend convolutions from spatial (i.e. regular lattice) domains to graph-structured (i.e. irregular) domains, yielding a class of algorithms known as Graph Convolutional Networks (GCNs).

Original: Formally, we are interested in semi-supervised learning where we are given a graph G =
Revised: Formally, we are interested in semi-supervised learning where we are given a graph G =

Original: ( V , E ) with N = |V| nodes; adjacency matrix A ; and matrix X ∈ R N × F of node features.
Revised: ( V , E ) with N = |V| nodes; adjacency matrix A ; and matrix X ∈ R N × F of node features.

Original: Labels for only a subset of nodes V L ⊂ V observed.
Revised: Labels for only a subset of nodes V L ⊂ V observed.

Original: In general, |V L | (cid:28) |V| .
Revised: In general, |V L | (cid:28) |V| .

Original: Our goal is to recover labels for all unlabeled nodes V U
Revised: Our goal is to recover labels for all unlabeled nodes V U

Original: =
Revised: =

Original: V − V L , using the feature matrix X , the known labels for nodes in V L , and the graph G .
Revised: V − V L , using the feature matrix X , the known labels for nodes in V L , and the graph G .

Original: In this setting, one treats the graph as the “unsupervised” and labels of V L as the “supervised” portions of the data.
Revised: In this setting, one treats the graph as the “unsupervised” and labels of V L as the “supervised” portions of the data.

Original: Depicted in Figure 1, our model for semi-supervised node classiﬁcation builds on the GCN module proposed by Kipf & Welling (2017), which operates on the normalized adjacency matrix ˆ A , as in GCN ( ˆ A ) , where ˆ A
Revised: Depicted in Figure 1, our model for semi-supervised node classiﬁcation builds on the GCN module proposed by Kipf & Welling (2017), which operates on the normalized adjacency matrix ˆ A , as in GCN ( ˆ A ) , where ˆ A

Original: = D − 12 AD − 12 , and D is diagonal matrix of node degrees.
Revised: = D − 12 AD − 12 , and D is diagonal matrix of node degrees.

Original: Our proposed extension of GCNs is inspired by the recent advancements in random walk based graph embeddings (e.g. Perozzi et al., 2014; Grover & Leskovec, 2016; Abu-El-Haija et al., 2017).
Revised: Our proposed extension of GCNs is inspired by the recent advancements in random walk based graph embeddings (e.g. Perozzi et al., 2014; Grover & Leskovec, 2016; Abu-El-Haija et al., 2017).

Original: We make a Network of GCN modules (N-GCN), feeding each module a different power of ˆ A , as in { GCN ( ˆ A 0 ) , GCN ( ˆ A 1 ) , GCN ( ˆ A 2 ) , . . . } .
Revised: We make a Network of GCN modules (N-GCN), feeding each module a different power of ˆ A , as in { GCN ( ˆ A 0 ) , GCN ( ˆ A 1 ) , GCN ( ˆ A 2 ) , . . . } .

Original: The k -th power contains statistics from the k -th step of a random walk on the graph.
Revised: The k -th power contains statistics from the k -th step of a random walk on the graph.

Original: Therefore, our N-GCN model is able to combine information from various step-sizes.
Revised: Therefore, our N-GCN model is able to combine information from various step-sizes.

Original: We then combine the output of all GCN modules into a classiﬁcation sub-network, and we jointly train all GCN modules and the classiﬁcation sub-network on the upstream objective,
Revised: We then combine the output of all GCN modules into a classiﬁcation sub-network, and we jointly train all GCN modules and the classiﬁcation sub-network on the upstream objective,

Original: I
Revised: I

Original: X ˆ A
Revised: X ˆ A

Original: GCN •
Revised: GCN •

Original: GCN × • GCN × • ...
Revised: GCN × • GCN × • ...

Original: ... GCNGCN
Revised: ... GCNGCN

Original: GCNGCN GCNGCN
Revised: GCNGCN GCNGCN

Original: GCN × •
Revised: GCN × •

Original: GCNGCN ...
Revised: GCNGCN ...

Original: N
Revised: N

Original: C C
Revised: C C

Original: C
Revised: C

Original: C K − 1 r concatenate
Revised: C K − 1 r concatenate

Original: N (cid:80) K − 1 k =0 rC k fully-connected
Revised: N (cid:80) K − 1 k =0 rC k fully-connected

Original: N
Revised: N

Original: C
Revised: C

Original: [Figure]
Revised: [Figure]

Original: enhance
Revised: enhance

Original: The rest of this paper is organized as follows.
Revised: The rest of this paper is organized as follows.

Original: Section 2 reviews background work that provides the foundation for this paper.
Revised: Section 2 reviews background work that provides the foundation for this paper.

Original: In Section 3, we describe our proposed method, followed by experimental evaluation in Section 4.
Revised: In Section 3, we describe our proposed method, followed by experimental evaluation in Section 4.

Original: We compare our work with recent closely-related methods in Section 5.
Revised: We compare our work with recent closely-related methods in Section 5.

Original: Finally, we conclude with our contributions and future work in Section 6.
Revised: Finally, we conclude with our contributions and future work in Section 6.

Original:  B ACKGROUND 2. S EMI -S UPERVISED N ODE C LASSIFICATION
Revised:  B ACKGROUND 2. S EMI -S UPERVISED N ODE C LASSIFICATION

Original: Traditional label propagation algorithms (Weston et al., 2012; Belkin et al., 2006a) learn a model that transforms node features into node labels and uses the graph to add a regularizer term:
Revised: Traditional label propagation algorithms (Weston et al., 2012; Belkin et al., 2006a) learn a model that transforms node features into node labels and uses the graph to add a regularizer term:

Original: [Equation]
Revised: [Equation]

Original: where f : R N × d 0 → R N × C is the model, ∆ is the graph Laplacian, and λ ∈ R is the regularization coefﬁcient hyperparameter.
Revised: where f : R N × d 0 → R N × C is the model, ∆ is the graph Laplacian, and λ ∈ R is the regularization coefﬁcient hyperparameter.

Original: 2. G RAPH C ONVOLUTIONAL N ETWORKS
Revised: 2. G RAPH C ONVOLUTIONAL N ETWORKS

Original: Graph Convolution (Bruna et al., 2014) generalizes convolution from Euclidean domains to graphstructured data.
Revised: Graph Convolution (Bruna et al., 2014) generalizes convolution from Euclidean domains to graphstructured data.

Original: Convolving a “ﬁlter” over a signal on graph nodes can be calculated by transforming both the ﬁlter and the signal to the Fourier domain, multiplying them, and then transforming the result back into the discrete domain.
Revised: Convolving a “ﬁlter” over a signal on graph nodes can be calculated by transforming both the ﬁlter and the signal to the Fourier domain, multiplying them, and then transforming the result back into the discrete domain.

Original: The signal transform is achieved by multiplying with the eigenvectors of the graph Laplacian.
Revised: The signal transform is achieved by multiplying with the eigenvectors of the graph Laplacian.

Original: The transformation requires a quadratic eigendecomposition of the symmetric Laplacian; however, the low-rank approximation of the eigendecomposition can be calculated using truncated Chebyshev polynomials (Hammond et al., 2011).
Revised: The transformation requires a quadratic eigendecomposition of the symmetric Laplacian; however, the low-rank approximation of the eigendecomposition can be calculated using truncated Chebyshev polynomials (Hammond et al., 2011).

Original: For instance, Kipf &
Revised: For instance, Kipf &

Original: Welling (2017) calculates a rank-1 approximation of the decomposition.
Revised: Welling (2017) calculates a rank-1 approximation of the decomposition.

Original: They propose a multi-layer
Revised: They propose a multi-layer

Original: Graph Convolutional Networks (GCNs) for semi-supervised graph learning.
Revised: Graph Convolutional Networks (GCNs) for semi-supervised graph learning.

Original: Every layer computes the transformation:
Revised: Every layer computes the transformation:

Original: [Equation]
Revised: [Equation]

Original: where H ( l ) ∈ R N × d l is the input activation matrix to the l -th hidden layer with row H ( l ) i containing a d l -dimensional feature vector for vertex i ∈ V , and
Revised: where H ( l ) ∈ R N × d l is the input activation matrix to the l -th hidden layer with row H ( l ) i containing a d l -dimensional feature vector for vertex i ∈ V , and

Original: W ( l ) ∈ R
Revised: W ( l ) ∈ R

Original: d l × d l +1 is the layer’s trainable weights.
Revised: d l × d l +1 is the layer’s trainable weights.

Original: The ﬁrst hidden layer H (0) is set to the input features X .
Revised: The ﬁrst hidden layer H (0) is set to the input features X .

Original: A softmax on the last layer is used to classify labels.
Revised: A softmax on the last layer is used to classify labels.

Original: All layers use the same “normalized adjacency” ˆ A , obtained by the “renormalization trick” utilized by Kipf & Welling (2017), as ˆ A
Revised: All layers use the same “normalized adjacency” ˆ A , obtained by the “renormalization trick” utilized by Kipf & Welling (2017), as ˆ A

Original: = D − 12 AD − 12 .
Revised: = D − 12 AD − 12 .

Original: Eq.
Revised: Eq.

Original: (2) is a ﬁrst order approximation of convolving ﬁlter W ( l ) over signal H ( l ) (Hammond et al., 2011; Kipf & Welling, 2017).
Revised: (2) is a ﬁrst order approximation of convolving ﬁlter W ( l ) over signal H ( l ) (Hammond et al., 2011; Kipf & Welling, 2017).

Original: The left-multiplication with ˆ A averages node features with their direct neighbors; this signal is then passed through a non-linearity function σ ( · ) (e.g, ReLU ( z )
Revised: The left-multiplication with ˆ A averages node features with their direct neighbors; this signal is then passed through a non-linearity function σ ( · ) (e.g, ReLU ( z )

Original: = max(0 , z ) ).
Revised: = max(0 , z ) ).

Original: Successive layers effectively diffuse signals from nodes to neighbors.
Revised: Successive layers effectively diffuse signals from nodes to neighbors.

Original: Two-layer GCN model can be deﬁned in terms of vertex features X and normalized adjacency ˆ A as:
Revised: Two-layer GCN model can be deﬁned in terms of vertex features X and normalized adjacency ˆ A as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where the GCN parameters θ = (cid:8) W (0) , W (1) (cid:9) are trained to minimize the cross-entropy error over labeled examples.
Revised: where the GCN parameters θ = (cid:8) W (0) , W (1) (cid:9) are trained to minimize the cross-entropy error over labeled examples.

Original: The output of the GCN model is a matrix R N × C , where N is the number of nodes and C is the number of labels.
Revised: The output of the GCN model is a matrix R N × C , where N is the number of nodes and C is the number of labels.

Original: Each row contains the label scores for one node, assuming there are
Revised: Each row contains the label scores for one node, assuming there are

Original: C classes.
Revised: C classes.

Original: 2. G RAPH E MBEDDINGS
Revised: 2. G RAPH E MBEDDINGS

Original: Node Embedding methods represent graph nodes in a continuous vector space.
Revised: Node Embedding methods represent graph nodes in a continuous vector space.

Original: They learn a dictionary Z ∈ R N × d , with one d -dimensional embedding per node.
Revised: They learn a dictionary Z ∈ R N × d , with one d -dimensional embedding per node.

Original: Traditional methods use the adjacency matrix to learn embeddings.
Revised: Traditional methods use the adjacency matrix to learn embeddings.

Original: For example, Eigenmaps (Belkin & Niyogi, 2003) calculates the following constrained optimization:
Revised: For example, Eigenmaps (Belkin & Niyogi, 2003) calculates the following constrained optimization:

Original: [Equation]
Revised: [Equation]

Original: where I is identity vector.
Revised: where I is identity vector.

Original: Skipgram models on text corpora (Mikolov et al., 2013) inspired modern graph embedding methods, which simulate random walks to learn node embeddings (Perozzi et al., 2014; Grover & Leskovec, 2016).
Revised: Skipgram models on text corpora (Mikolov et al., 2013) inspired modern graph embedding methods, which simulate random walks to learn node embeddings (Perozzi et al., 2014; Grover & Leskovec, 2016).

Original: Each random walk generates a sequence of nodes.
Revised: Each random walk generates a sequence of nodes.

Original: Sequences are converted to textual paragraphs, and are passed to a word2vec-style embedding learning algorithm (Mikolov et al., 2013).
Revised: Sequences are converted to textual paragraphs, and are passed to a word2vec-style embedding learning algorithm (Mikolov et al., 2013).

Original: As shown in Abu-El-Haija et al.
Revised: As shown in Abu-El-Haija et al.

Original: (2017), this learning-by-simulation is equivalent, in expectation, to the decomposition of a random walk co-occurrence statistics matrix D .
Revised: (2017), this learning-by-simulation is equivalent, in expectation, to the decomposition of a random walk co-occurrence statistics matrix D .

Original: The expectation on D can be written as:
Revised: The expectation on D can be written as:

Original: [Equation]
Revised: [Equation]

Original: where T = D − 1 A is the row-normalized transition matrix (a.k.a right-stochastic adjacency matrix), and Q is a “context distribution” that is determined by random walk hyperparameters, such as the length of the random walk.
Revised: where T = D − 1 A is the row-normalized transition matrix (a.k.a right-stochastic adjacency matrix), and Q is a “context distribution” that is determined by random walk hyperparameters, such as the length of the random walk.

Original: The expectation therefore weights the importance of one node on another as a function of how well-connected they are, and the distance between them.
Revised: The expectation therefore weights the importance of one node on another as a function of how well-connected they are, and the distance between them.

Original: The main difference between traditional node embedding methods and random walk methods is the optimization criteria: the former minimizes a loss on representing the adjacency matrix A (see Eq. 4), while the latter minimizes a loss on representing random walk co-occurrence statistics D .
Revised: The main difference between traditional node embedding methods and random walk methods is the optimization criteria: the former minimizes a loss on representing the adjacency matrix A (see Eq. 4), while the latter minimizes a loss on representing random walk co-occurrence statistics D .

Original:  O UR M ETHOD 3. M OTIVATION
Revised:  O UR M ETHOD 3. M OTIVATION

Original: Graph Convolutional Networks and random walk graph embeddings are individually powerful.
Revised: Graph Convolutional Networks and random walk graph embeddings are individually powerful.

Original: Kipf & Welling (2017) uses GCNs for semi-supervised node classiﬁcation.
Revised: Kipf & Welling (2017) uses GCNs for semi-supervised node classiﬁcation.

Original: Instead of following tradi- tional methods that use the graph for regularization (e.g. Eq. 4), Kipf & Welling (2017) use the adjacency matrix for training and inference, effectively diffusing information across edges at all
Revised: Instead of following tradi- tional methods that use the graph for regularization (e.g. Eq. 4), Kipf & Welling (2017) use the adjacency matrix for training and inference, effectively diffusing information across edges at all

Original: GCN layers (see Eq. 6).
Revised: GCN layers (see Eq. 6).

Original: Separately, recent work has showed that random walk statistics can be very powerful for learning an unsupervised representation of nodes that can preserve the structure of the graph (Perozzi et al., 2014; Grover & Leskovec, 2016; Abu-El-Haija et al., 2017).
Revised: Separately, recent work has showed that random walk statistics can be very powerful for learning an unsupervised representation of nodes that can preserve the structure of the graph (Perozzi et al., 2014; Grover & Leskovec, 2016; Abu-El-Haija et al., 2017).

Original: Under special conditions, it is possible for the GCN model to learn random walks.
Revised: Under special conditions, it is possible for the GCN model to learn random walks.

Original: In particular, consider a two-layer GCN deﬁned in Eq.
Revised: In particular, consider a two-layer GCN deﬁned in Eq.

Original: 6 with the assumption that ﬁrst-layer activation is identity as σ ( z )
Revised: 6 with the assumption that ﬁrst-layer activation is identity as σ ( z )

Original: =
Revised: =

Original: z , and weight W (0) is an identity matrix (either explicitly set or learned to satisfy the upstream objective).
Revised: z , and weight W (0) is an identity matrix (either explicitly set or learned to satisfy the upstream objective).

Original: Under these two identity conditions, the model reduces to:
Revised: Under these two identity conditions, the model reduces to:

Original: [Equation]
Revised: [Equation]

Original: where ˆ A 2 can be expanded as:
Revised: where ˆ A 2 can be expanded as:

Original: [Equation]
Revised: [Equation]

Original: By multiplying the adjacency A with the transition matrix T before normalization, the GCN is effectively doing a one-step random walk.
Revised: By multiplying the adjacency A with the transition matrix T before normalization, the GCN is effectively doing a one-step random walk.

Original: 3. E XPLICIT R ANDOM W ALKS
Revised: 3. E XPLICIT R ANDOM W ALKS

Original: The special conditions described above are not true in practice.
Revised: The special conditions described above are not true in practice.

Original: Although stacking hidden GCN layers allows information to ﬂow through graph edges, this ﬂow is indirect as the information goes through feature reduction (matrix multiplication) and a non-linearity (activation function σ ( · ) ).
Revised: Although stacking hidden GCN layers allows information to ﬂow through graph edges, this ﬂow is indirect as the information goes through feature reduction (matrix multiplication) and a non-linearity (activation function σ ( · ) ).

Original: Therefore, the vanilla GCN cannot directly learn high powers of ˆ A , and could struggle with modeling information across distant nodes.
Revised: Therefore, the vanilla GCN cannot directly learn high powers of ˆ A , and could struggle with modeling information across distant nodes.

Original: We hypothesize that making the GCN directly operate on random walk statistics will allow the network to better utilize information across distant nodes, in the same way that node embedding methods (e.g. DeepWalk, Perozzi et al.
Revised: We hypothesize that making the GCN directly operate on random walk statistics will allow the network to better utilize information across distant nodes, in the same way that node embedding methods (e.g. DeepWalk, Perozzi et al.

Original: operating on
Revised: operating on

Original: D are superior to traditional embedding methods operating on the adjacency matrix (e.g. Eigenmaps, Belkin & Niyogi (2003)).
Revised: D are superior to traditional embedding methods operating on the adjacency matrix (e.g. Eigenmaps, Belkin & Niyogi (2003)).

Original: Therefore, in addition to feeding only ˆ A to the GCN model as proposed by Kipf & Welling (2017) (see Eq. 6), we propose to feed a K -degree polynomial of ˆ A to
Revised: Therefore, in addition to feeding only ˆ A to the GCN model as proposed by Kipf & Welling (2017) (see Eq. 6), we propose to feed a K -degree polynomial of ˆ A to

Original: K instantiations of GCN.
Revised: K instantiations of GCN.

Original: Generalizing Eq.
Revised: Generalizing Eq.

Original: (7) gives:
Revised: (7) gives:

Original: [Equation]
Revised: [Equation]

Original: We also deﬁne ˆ A 0 to be the identity matrix.
Revised: We also deﬁne ˆ A 0 to be the identity matrix.

Original: Similar to Kipf & Welling (2017), we add selfconnections and convert directed graphs to undirected ones, making ˆ A and hence ˆ A k symmetric matrices.
Revised: Similar to Kipf & Welling (2017), we add selfconnections and convert directed graphs to undirected ones, making ˆ A and hence ˆ A k symmetric matrices.

Original: The eigendecomposition of symmetric matrices is real.
Revised: The eigendecomposition of symmetric matrices is real.

Original: Therefore, the low-rank approximation of the eigendecomposition Hammond et al.
Revised: Therefore, the low-rank approximation of the eigendecomposition Hammond et al.

Original: (2011) is still valid, and a one layer of Kipf &
Revised: (2011) is still valid, and a one layer of Kipf &

Original: Welling (2017) utilizing ˆ A k should still approximate multiplication in the Fourier domain.
Revised: Welling (2017) utilizing ˆ A k should still approximate multiplication in the Fourier domain.

Original: 3. N ETWORK OF GCN S
Revised: 3. N ETWORK OF GCN S

Original: Consider K instantiations of { GCN ( ˆ A 0 , X ) , GCN ( ˆ A 1 , X ) , . .
Revised: Consider K instantiations of { GCN ( ˆ A 0 , X ) , GCN ( ˆ A 1 , X ) , . .

Original: . , GCN ( ˆ A K − 1 , X ) } .
Revised: . , GCN ( ˆ A K − 1 , X ) } .

Original: Each GCN outputs a matrix R N × C k , where the v -th row describes a latent representation of that particular
Revised: Each GCN outputs a matrix R N × C k , where the v -th row describes a latent representation of that particular

Original: GCN for node v ∈ V , and where C k is the latent dimensionality.
Revised: GCN for node v ∈ V , and where C k is the latent dimensionality.

Original: Though C k can be different for each GCN, we set all C k to be the same for simplicity.
Revised: Though C k can be different for each GCN, we set all C k to be the same for simplicity.

Original: We then combine the output of all K
Revised: We then combine the output of all K

Original: GCN and feed them into a classiﬁcation sub-network, allowing us to jointly train all GCNs and the classiﬁcation sub-network via backpropagation.
Revised: GCN and feed them into a classiﬁcation sub-network, allowing us to jointly train all GCNs and the classiﬁcation sub-network via backpropagation.

Original: This should allow the classiﬁcation sub-network to choose features from the various GCNs, effectively allowing the overall model to learn a combination of features using the raw (normalized) adjacency, different steps of random walks, and the input features X (as they are multiplied by identity ˆ A 0 ).
Revised: This should allow the classiﬁcation sub-network to choose features from the various GCNs, effectively allowing the overall model to learn a combination of features using the raw (normalized) adjacency, different steps of random walks, and the input features X (as they are multiplied by identity ˆ A 0 ).

Original: 3.3. F ULLY -C ONNECTED C LASSIFICATION N ETWORK
Revised: 3.3. F ULLY -C ONNECTED C LASSIFICATION N ETWORK

Original: From a deep learning prospective, it is intuitive to represent the classiﬁcation network as a fullyconnected layer.
Revised: From a deep learning prospective, it is intuitive to represent the classiﬁcation network as a fullyconnected layer.

Original: We can concatenate the output of the K GCNs along the column dimension, i.e. concatenating all GCN ( X, ˆ A k ) , each ∈ R N × C k into matrix ∈ R N × C K where C K =
Revised: We can concatenate the output of the K GCNs along the column dimension, i.e. concatenating all GCN ( X, ˆ A k ) , each ∈ R N × C k into matrix ∈ R N × C K where C K =

Original: (cid:80) k C k .
Revised: (cid:80) k C k .

Original: We add a fully-connected layer f fc : R N × C K → R N × C , with trainable parameter matrix W fc ∈ R C K × C , written as:
Revised: We add a fully-connected layer f fc : R N × C K → R N × C , with trainable parameter matrix W fc ∈ R C K × C , written as:

Original: [Equation]
Revised: [Equation]

Original: The classiﬁer parameters W fc are jointly trained with GCN parameters θ = { θ (0) , θ (1) , . . . } .
Revised: The classiﬁer parameters W fc are jointly trained with GCN parameters θ = { θ (0) , θ (1) , . . . } .

Original: We use subscript fc on N-GCN to indicate the classiﬁcation network is a fully-connected layer.
Revised: We use subscript fc on N-GCN to indicate the classiﬁcation network is a fully-connected layer.

Original: 3.3. A TTENTION C LASSIFICATION N ETWORK
Revised: 3.3. A TTENTION C LASSIFICATION N ETWORK

Original: We also propose a classiﬁcation network based on “softmax attention”, which learns a convex combination of the GCN instantiations.
Revised: We also propose a classiﬁcation network based on “softmax attention”, which learns a convex combination of the GCN instantiations.

Original: Our attention model (N-GCN a ) is parametrized by vector (cid:101) m ∈ R K , one scalar for each GCN.
Revised: Our attention model (N-GCN a ) is parametrized by vector (cid:101) m ∈ R K , one scalar for each GCN.

Original: It can be written as:
Revised: It can be written as:

Original: [Equation]
Revised: [Equation]

Original: where m is output of a softmax: m = softmax ( (cid:101) m ) .
Revised: where m is output of a softmax: m = softmax ( (cid:101) m ) .

Original: This softmax attention is similar to “Mixture of Experts” model, especially if we set the number of output channels for all GCNs equal to the number of classes, as in C
Revised: This softmax attention is similar to “Mixture of Experts” model, especially if we set the number of output channels for all GCNs equal to the number of classes, as in C

Original: =
Revised: =

Original: C 1 = · · · = C .
Revised: C 1 = · · · = C .

Original: This allows us to add cross entropy loss terms on all GCN outputs in addition to the loss applied at the output NGCN, forcing all GCN’s to be independently useful.
Revised: This allows us to add cross entropy loss terms on all GCN outputs in addition to the loss applied at the output NGCN, forcing all GCN’s to be independently useful.

Original: It is possible to set the m ∈ R K parameter vector “by hand” using the validation split, especially for reasonable K such as K ≤ 6 .
Revised: It is possible to set the m ∈ R K parameter vector “by hand” using the validation split, especially for reasonable K such as K ≤ 6 .

Original: One possible choice might be setting m 0 to some small value and remaining m 1 , . . .
Revised: One possible choice might be setting m 0 to some small value and remaining m 1 , . . .

Original: , m K − 1 to the harmonic series 1 k ; another choice may be linear decay K − k K − 1 .
Revised: , m K − 1 to the harmonic series 1 k ; another choice may be linear decay K − k K − 1 .

Original: These are respectively similar to the context distributions of GloVe (Pennington et al., 2014) and word2vec (Mikolov et al., 2013; Levy et al., 2015).
Revised: These are respectively similar to the context distributions of GloVe (Pennington et al., 2014) and word2vec (Mikolov et al., 2013; Levy et al., 2015).

Original: We note that if on average a node’s information is captured by its direct or nearby neighbors, then the output of GCNs consuming lower powers of ˆ A should be weighted highly.
Revised: We note that if on average a node’s information is captured by its direct or nearby neighbors, then the output of GCNs consuming lower powers of ˆ A should be weighted highly.

Original: 3. T RAINING
Revised: 3. T RAINING

Original: We minimize the cross entropy between our model output and the known training labels Y as:
Revised: We minimize the cross entropy between our model output and the known training labels Y as:

Original: [Equation]
Revised: [Equation]

Original: where ◦ is Hadamard product, and diag ( V L ) denotes a diagonal matrix, with entry at ( i, i )
Revised: where ◦ is Hadamard product, and diag ( V L ) denotes a diagonal matrix, with entry at ( i, i )

Original: set toif
Revised: set toif

Original: i ∈ V L and 0 otherwise.
Revised: i ∈ V L and 0 otherwise.

Original: In addition, we can apply intermediate supervision for the NGCN a to attempt make all GCN become independently useful, yielding minimization objective:
Revised: In addition, we can apply intermediate supervision for the NGCN a to attempt make all GCN become independently useful, yielding minimization objective:

Original: [Equation]
Revised: [Equation]

Original: 3. GCN R EPLICATION
Revised: 3. GCN R EPLICATION

Original: To simplify notation, our N-GCN derivations (e.g. Eq. 9) assume that there is one GCN per ˆ A power.
Revised: To simplify notation, our N-GCN derivations (e.g. Eq. 9) assume that there is one GCN per ˆ A power.

Original: However, our implementation feeds every ˆ A to r GCN modules, as shown in Fig.
Revised: However, our implementation feeds every ˆ A to r GCN modules, as shown in Fig.

Original: 3. G ENERALIZATION TO OTHER G RAPH M ODELS
Revised: 3. G ENERALIZATION TO OTHER G RAPH M ODELS

Original: In addition to vanilla GCNs (e.g. Kipf & Welling, 2017), our derivation also applies to other graph models including GraphSAGE (SAGE, Hamilton et al., 2017).
Revised: In addition to vanilla GCNs (e.g. Kipf & Welling, 2017), our derivation also applies to other graph models including GraphSAGE (SAGE, Hamilton et al., 2017).

Original: Algorithm 1 shows a generalization that allows us to make a network of arbitrary graph models (e.g. GCN, SAGE, or others).
Revised: Algorithm 1 shows a generalization that allows us to make a network of arbitrary graph models (e.g. GCN, SAGE, or others).

Original: Algorithm 2 shows pseudo-code for the vanilla GCN.
Revised: Algorithm 2 shows pseudo-code for the vanilla GCN.

Original: Finally, Algorithm 3 deﬁnes our full Network of GCN model (N-GCN) by plugging Algorithm 2 into Algorithm 1.
Revised: Finally, Algorithm 3 deﬁnes our full Network of GCN model (N-GCN) by plugging Algorithm 2 into Algorithm 1.

Original: Similarly, we list the algorithms for
Revised: Similarly, we list the algorithms for

Original: SAGE and Network of SAGE (N-SAGE) in the Appendix.
Revised: SAGE and Network of SAGE (N-SAGE) in the Appendix.

Original: We can recover the original algorithms GCN (Kipf & Welling, 2017) and SAGE (Hamilton et al., 2017), respectively, by using Algorithms 3 (N-GCN) and 5 (N-SAGE, listed in Appendix) with r = 1 , K = 1 , identity C LASSIFIER F N , and modifying line 2 in Algorithm 1 to P ← ˆ A .
Revised: We can recover the original algorithms GCN (Kipf & Welling, 2017) and SAGE (Hamilton et al., 2017), respectively, by using Algorithms 3 (N-GCN) and 5 (N-SAGE, listed in Appendix) with r = 1 , K = 1 , identity C LASSIFIER F N , and modifying line 2 in Algorithm 1 to P ← ˆ A .

Original: Moreover, we can recover original DCNN (Atwood & Towsley, 2016) by calling Algorithm 3 with L = 1 , r = 1 , modifying line 3 to ˆ A ← D − 1 A , and keeping K > 1 as their proposed model operates on the power series of the transition matrix i.e. unmodiﬁed random walks, like ours.
Revised: Moreover, we can recover original DCNN (Atwood & Towsley, 2016) by calling Algorithm 3 with L = 1 , r = 1 , modifying line 3 to ˆ A ← D − 1 A , and keeping K > 1 as their proposed model operates on the power series of the transition matrix i.e. unmodiﬁed random walks, like ours.

Original: Require:
Revised: Require:

Original: ˆ A is a normalization of A 1: function N ETWORK (G RAPH M ODEL F N , ˆ A , X , L , r = 4 , K = 6 , C LASSIFIER F N =F C L AYER )
Revised: ˆ A is a normalization of A 1: function N ETWORK (G RAPH M ODEL F N , ˆ A , X , L , r = 4 , K = 6 , C LASSIFIER F N =F C L AYER )

Original: P ← I 3:
Revised: P ← I 3:

Original: GraphModels ← [] 4: for k
Revised: GraphModels ← [] 4: for k

Original: = 1 to K do 5: for i = 1 to r do 6:
Revised: = 1 to K do 5: for i = 1 to r do 6:

Original: GraphModels.append(G RAPH M ODEL F N ( P, X, L ) ) 7:
Revised: GraphModels.append(G RAPH M ODEL F N ( P, X, L ) ) 7:

Original: P ← ˆ AP 8: return C LASSIFIER F N (GraphModels)
Revised: P ← ˆ AP 8: return C LASSIFIER F N (GraphModels)

Original: Require:
Revised: Require:

Original: ˆ A is a normalization of A 1: function G CN M ODEL ( ˆ A , X , L ) 2:
Revised: ˆ A is a normalization of A 1: function G CN M ODEL ( ˆ A , X , L ) 2:

Original: Z ← X 3: for i = 1 to L do 4:
Revised: Z ← X 3: for i = 1 to L do 4:

Original: Z ← σ ( ˆ AZW ( i ) ) 5: return Z
Revised: Z ← σ ( ˆ AZW ( i ) ) 5: return Z

Original: Algorithm
Revised: Algorithm

Original: 1: function N GCN ( A , X , L = 2 ) 2:
Revised: 1: function N GCN ( A , X , L = 2 ) 2:

Original: D ← diag ( A 1 ) (cid:46)
Revised: D ← diag ( A 1 ) (cid:46)

Original: Sum rows 3: ˆ A ← D − 1 / 2 AD − 1 /4: return N ETWORK ( G CN M ODEL , ˆ A, X, L )
Revised: Sum rows 3: ˆ A ← D − 1 / 2 AD − 1 /4: return N ETWORK ( G CN M ODEL , ˆ A, X, L )

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We follow the experimental setup by Kipf & Welling (2017) and Yang et al.
Revised: We follow the experimental setup by Kipf & Welling (2017) and Yang et al.

Original: (2016), including the provided dataset splits (train, validation, test) produced by Yang et al.
Revised: (2016), including the provided dataset splits (train, validation, test) produced by Yang et al.

Original: 4. D ATASETS
Revised: 4. D ATASETS

Original: We experiment on three citation graph datasets: Pubmed, Citeseer, Cora, and a biological graph:
Revised: We experiment on three citation graph datasets: Pubmed, Citeseer, Cora, and a biological graph:

Original: Protein-Protein Interactions (PPI).
Revised: Protein-Protein Interactions (PPI).

Original: We choose the aforementioned datasets because they are available online and are used by our baselines.
Revised: We choose the aforementioned datasets because they are available online and are used by our baselines.

Original: The citation datasets are prepared by Yang et al.
Revised: The citation datasets are prepared by Yang et al.

Original: and the PPI dataset is prepared by Hamilton et al.
Revised: and the PPI dataset is prepared by Hamilton et al.

Original: Table 1 summarizes dataset statistics.
Revised: Table 1 summarizes dataset statistics.

Original: Each node in the citation datasets represents an article published in the corresponding journal.
Revised: Each node in the citation datasets represents an article published in the corresponding journal.

Original: An edge between two nodes represents a citation from one article to another, and a label represents the subject of the article.
Revised: An edge between two nodes represents a citation from one article to another, and a label represents the subject of the article.

Original: Each dataset contains a binary Bag-of-Words (BoW) feature vector for each node.
Revised: Each dataset contains a binary Bag-of-Words (BoW) feature vector for each node.

Original: The BoW are extracted from the article abstract.
Revised: The BoW are extracted from the article abstract.

Original: Therefore, the task is to predict the subject of articles, given the BoW of their abstract and the citations to other (possibly labeled) articles.
Revised: Therefore, the task is to predict the subject of articles, given the BoW of their abstract and the citations to other (possibly labeled) articles.

Original: Following Yang et al.
Revised: Following Yang et al.

Original: and Kipf & Welling (2017), we use 20 nodes per class for training, 500 (overall) nodes for validation, and 1000 nodes for evaluation.
Revised: and Kipf & Welling (2017), we use 20 nodes per class for training, 500 (overall) nodes for validation, and 1000 nodes for evaluation.

Original: We note that the validation set is larger than training |V L | for these datasets!
Revised: We note that the validation set is larger than training |V L | for these datasets!

Original: The PPI graph, as processed and described by Hamilton et al.
Revised: The PPI graph, as processed and described by Hamilton et al.

Original: (2017), consists of 24 disjoint subgraphs, each corresponding to a different human tissue.
Revised: (2017), consists of 24 disjoint subgraphs, each corresponding to a different human tissue.

Original: 20 of those subgraphs are used for training, 2 for validation, and 2 for testing, as partitioned by Hamilton et al.
Revised: 20 of those subgraphs are used for training, 2 for validation, and 2 for testing, as partitioned by Hamilton et al.

Original: 4. B ASELINE M ETHODS
Revised: 4. B ASELINE M ETHODS

Original: For the citation datasets, we copy baseline numbers from Kipf & Welling (2017).
Revised: For the citation datasets, we copy baseline numbers from Kipf & Welling (2017).

Original: These include label propagation (LP, Zhu et al.
Revised: These include label propagation (LP, Zhu et al.

Original: ; semi-supervised embedding (SemiEmb, Weston et al.
Revised: ; semi-supervised embedding (SemiEmb, Weston et al.

Original: ; manifold regularization (ManiReg, Belkin et al. (2006b)); skip-gram graph embeddings (DeepWalk Perozzi et al., 2014); Iterative Classiﬁcation Algorithm (ICA, Lu & Getoor, 2003);
Revised: ; manifold regularization (ManiReg, Belkin et al. (2006b)); skip-gram graph embeddings (DeepWalk Perozzi et al., 2014); Iterative Classiﬁcation Algorithm (ICA, Lu & Getoor, 2003);

Original: Planetoid (Yang et al., 2016); vanilla GCN (Kipf & Welling, 2017).
Revised: Planetoid (Yang et al., 2016); vanilla GCN (Kipf & Welling, 2017).

Original: For PPI, we copy baseline numbers from (Hamilton et al., 2017), which include GraphSAGE with LSTM aggregation (SAGELSTM) and GraphSAGE with pooling aggregation (SAGE).
Revised: For PPI, we copy baseline numbers from (Hamilton et al., 2017), which include GraphSAGE with LSTM aggregation (SAGELSTM) and GraphSAGE with pooling aggregation (SAGE).

Original: Further, for all datasets, we use our implementation to run baselines DCNN (Atwood & Towsley, 2016), GCN (Kipf & Welling, 2017),
Revised: Further, for all datasets, we use our implementation to run baselines DCNN (Atwood & Towsley, 2016), GCN (Kipf & Welling, 2017),

Original: [Table] [Table]
Revised: [Table] [Table]

Original: and SAGE (with pooling aggregation, Hamilton et al., 2017), as these baselines can be recovered as special cases of our algorithm, as explained in Section 3.6.
Revised: and SAGE (with pooling aggregation, Hamilton et al., 2017), as these baselines can be recovered as special cases of our algorithm, as explained in Section 3.6.

Original: 4. I MPLEMENTATION
Revised: 4. I MPLEMENTATION

Original: We use TensorFlow(Abadi et al., 2015) to implement our methods, which we use to also measure the performance of baselines GCN, SAGE
Revised: We use TensorFlow(Abadi et al., 2015) to implement our methods, which we use to also measure the performance of baselines GCN, SAGE

Original: [Table]
Revised: [Table]

Original: and DCNN.
Revised: and DCNN.

Original: For our methods and baselines, all GCN and
Revised: For our methods and baselines, all GCN and

Original: SAGE modules that we train are 2 layers, where the ﬁrst outputs 16 dimensions per node and the second outputs the number of classes (dataset-dependent).
Revised: SAGE modules that we train are 2 layers, where the ﬁrst outputs 16 dimensions per node and the second outputs the number of classes (dataset-dependent).

Original: DCNN baseline has one layer and outputs 16 dimensions per node, and its channels (one per transition matrix power) are concatenated into a fully-connected layer that outputs the number of classes.
Revised: DCNN baseline has one layer and outputs 16 dimensions per node, and its channels (one per transition matrix power) are concatenated into a fully-connected layer that outputs the number of classes.

Original: We use 50% dropout and L2 regularization of 10 − 5 for all of the aforementioned models.
Revised: We use 50% dropout and L2 regularization of 10 − 5 for all of the aforementioned models.

Original: [Equation]
Revised: [Equation]

Original: LASSIFICATION A CCURACY
Revised: LASSIFICATION A CCURACY

Original: Table 2 shows node classiﬁcation accuracy results.
Revised: Table 2 shows node classiﬁcation accuracy results.

Original: We run 20 different random initializations for every model (baselines and ours), train using Adam optimizer (Ba & Kingma, 2015) with learning rate of 0.01 for 600 steps, capturing the model parameters at peak validation accuracy to avoid overﬁtting.
Revised: We run 20 different random initializations for every model (baselines and ours), train using Adam optimizer (Ba & Kingma, 2015) with learning rate of 0.01 for 600 steps, capturing the model parameters at peak validation accuracy to avoid overﬁtting.

Original: For our models, we sweep our hyperparameters r , K , and choice of classiﬁcation sub-
Revised: For our models, we sweep our hyperparameters r , K , and choice of classiﬁcation subnetwork ∈ { fc , a } .

Revised: For baselines and our models, we choose the model with the highest accuracy on validation set, and use it to record metrics on the test set in Table 2.

Original: r4 K 1 2 3 469.2 69.4 69.
Revised: r 1 24 K    69.2 69.4 69.

Original: (a) N-GCN fc on Citeseer r4 K 1 2 3 471.1 71.
Revised: (a) N-GCN fc on Citeseer r 1 24 K    71.1 71.

Original: (b) N-GCN a on Citeseer r  4 K 1 2 3 480.680.881.081.
Revised: (b) N-GCN a on Citeseer r  4 K 1 2 3 480.680.881.081.

Original: (c) N-GCN fc on Cora r  4 K 1 2 3 482.2 82.3 82.
Revised: (c) N-GCN fc on Cora r  4 K 1 2 3 482.2 82.3 82.

Original: (d) N-GCN a on Cora r  4 K 1 2 3 476.5 77.
Revised: (d) N-GCN a on Cora r  4 K 1 2 3 476.5 77.

Original: (e) N-GCN fc on Pubmed r  4 K 1 2 3 478.278.478.678.879.
Revised: (e) N-GCN fc on Pubmed r  4 K 1 2 3 478.278.478.678.879.

Original: (f) N-GCN a on Pubmed
Revised: (f) N-GCN a on Pubmed

Original: Nodes per class
Revised: Nodes per class

Original: DCNN (our implementation) 63 . 0 ± 1 .72 .
Revised: DCNN (our implementation) 63 . 0 ± 1 .72 .

Original: 3 ± 0 .79 .
Revised: 3 ± 0 .79 .

Original: 2 ± 0 .82 .
Revised: 2 ± 0 .82 .

Original: 6 ± 0 .
Revised: 6 ± 0 .

Original: GCN (our implementation) 64 .
Revised: GCN (our implementation) 64 .

Original: 6 ± 0 .70 . 0 ± 3 .79 .
Revised: 6 ± 0 .70 . 0 ± 3 .79 .

Original: 1 ± 0 .81 .
Revised: 1 ± 0 .81 .

Original: 8 ± 0 .
Revised: 8 ± 0 .

Original: SAGE (our implementation) 69 . 0 ± 1 .72 .0 ±
Revised: SAGE (our implementation) 69 . 0 ± 1 .72 .

Original: [Table]
Revised: [Table]

Original: [Table][Table] [Table] network ∈ { fc , a } .
Revised: [Table]

Original: For baselines and our models, we choose the model with the highest accuracy on validation set

Original: [Table]
Revised: [Table]

Original: and use it to record metrics on the test set in Table 2.

Original: Table 2 shows that N-GCN outperforms GCN (Kipf & Welling, 2017) and N-SAGE improves on
Revised: Table 2 shows that N-GCN outperforms GCN (Kipf & Welling, 2017) and N-SAGE improves on

Original: SAGE for all datasets, showing that unmodiﬁed random walks indeed help in semi-supervised node classiﬁcation.
Revised: SAGE for all datasets, showing that unmodiﬁed random walks indeed help in semi-supervised node classiﬁcation.

Original: Finally, our proposed models acheive state-of-the-art on all datasets.
Revised: Finally, our proposed models acheive state-of-the-art on all datasets [Table]

Original: 4. S ENSITIVITY A NALYSIS
Revised: 4. S ENSITIVITY A NALYSIS

Original: We analyze the impact of K and r on classiﬁcation accuracy in Figure 2.
Revised: We analyze the impact of K and r on classiﬁcation accuracy in Figure 2.

Original: We note that adding random walks by speciﬁcally setting K > 1 improves model accuracy due to the additional information, not due to increased model capacity.
Revised: We note that adding random walks by speciﬁcally setting K > 1 improves model accuracy due to the additional information, not due to increased model capacity.

Original: Contrast K = 1 , r > 1 (i.e. mixture of GCNs, no random walks) with K > 1 , r = 1 (i.e. N-GCN on random walks): in both scenarios, the model has more capacity, but the latter shows better performance.
Revised: Contrast K = 1 , r > 1 (i [Table] e. mixture of GCNs, no random walks) with K > 1 , r = 1 (i.e. N-GCN on random walks): in both scenarios, the model has more capacity, but the latter shows better performance.

Original: The same holds for SAGE, as shown in Appendix.
Revised: The same holds for SAGE, as shown in Appendix.

Original: 4. T OLERANCE TO FEATURE NOISE
Revised: 4. T OLERANCE TO FEATURE NOISE

Original: We test our method under feature noise perturbations by removing node features at random.
Revised: We test our method under feature noise perturbations by removing node features at random.

Original: This is practical, as article authors might forget to include relevant terms in the article abstract, and more[Figure] [Figure] generally not all nodes will have the same amount of detailed information.
Revised: This is practical, as article authors might forget to include relevant terms in the article abstract, and more generally not all nodes will have the same amount of detailed information.

Original: Figure 3 shows that when features are removed, methods utilizing unmodiﬁed random walks: N-GCN, N-SAGE, and
Revised: Figure 3 shows that when features are removed, methods utilizing unmodiﬁed random walks: N-GCN, N-SAGE, and

Original: DCNN, outperform convolutional methods including GCN and SAGE.
Revised: DCNN, outperform convolutional methods including GCN and SAGE.

Original: Moreover, the performance  gap widens as we remove more features.
Revised: Moreover, the performance [Figure] [Figure] gap widens as we remove more features.

Original: This suggests that our methods can somewhat recover removed features by directly pulling-in features from nearby and distant neighbors.
Revised: This suggests that our methods can somewhat recover removed features by directly pulling-in features from nearby and distant neighbors.

Original: We visualize in Figure 4 the attention weights as a function of % features removed.
Revised: We visualize in Figure 4 the attention weights as a function of % features removed.

Original: With little feature removal, there is some weight on ˆ A 0 , and the attention weights for ˆ A 1 , ˆ A 2 , . . .
Revised: With little feature removal, there is some weight on ˆ A 0 , and the attention weights for ˆ A 1 , ˆ A 2 , . . .

Original: follow some decay function.
Revised: follow some decay function.

Original: Maliciously dropping features causes our model to shift its attention weights towards higher powers of ˆ A .
Revised: Maliciously dropping features causes our model to shift its attention weights towards higher powers of ˆ A .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: The ﬁeld of graph learning algorithms is quickly evolving.
Revised: The ﬁeld of graph learning algorithms is quickly evolving.

Original: We review work most similar to ours.
Revised: We review work most similar to ours.

Original: Defferrard et al.
Revised: Defferrard et al.

Original: (2016) deﬁne graph convolutions as a K -degree polynomial of the Laplacian, where the polynomial coefﬁcients are learned.
Revised: (2016) deﬁne graph convolutions as a K -degree polynomial of the Laplacian, where the polynomial coefﬁcients are learned.

Original: In their setup, the K -th degree Laplacian is a sparse square matrix where entry at ( i, j ) will be zero if nodes i and j are more than K hops apart.
Revised: In their setup, the K -th degree Laplacian is a sparse square matrix where entry at ( i, j ) will be zero if nodes i and j are more than K hops apart.

Original: Their sparsity analysis also applies here.
Revised: Their sparsity analysis also applies here.

Original: A minor difference is the adjacency normalization.
Revised: A minor difference is the adjacency normalization.

Original: We use ˆ A whereas they use the Laplacian deﬁned as I − ˆ A .
Revised: We use ˆ A whereas they use the Laplacian deﬁned as I − ˆ A .

Original: Raising ˆ A to power K will produce a square matrix with entry ( i, j ) being the probability of random walker ending at node i after K steps from node j .
Revised: Raising ˆ A to power K will produce a square matrix with entry ( i, j ) being the probability of random walker ending at node i after K steps from node j .

Original: The major difference is the order of random walk versus non-linearity.
Revised: The major difference is the order of random walk versus non-linearity.

Original: In particular, their model calculates learns a linear combination of K -degree polynomial and pass through classiﬁer function g , as in g ( (cid:80) k q k (cid:101)
Revised: In particular, their model calculates learns a linear combination of K -degree polynomial and pass through classiﬁer function g , as in g ( (cid:80) k q k (cid:101)

Original: A k ) , while our (e.g. N-GCN) model calculates (cid:80)
Revised: A k ) , while our (e.g. N-GCN) model calculates (cid:80)

Original: k q k g ( (cid:101)
Revised: k q k g ( (cid:101)

Original: A k ) , where (cid:101)
Revised: A k ) , where (cid:101)

Original: A is ˆ A in our model
Revised: A is ˆ A in our model

Original: and I − ˆ A in theirs, and our g can be a GCN module.
Revised: and I − ˆ A in theirs, and our g can be a GCN module.

Original: In fact, Defferrard et al.
Revised: In fact, Defferrard et al.

Original: (2016) is also similar to work by Abu-El-Haija et al.
Revised: (2016) is also similar to work by Abu-El-Haija et al.

Original: (2017), as they both learn polynomial coefﬁcients to some normalized adjacency matrix.
Revised: (2017), as they both learn polynomial coefﬁcients to some normalized adjacency matrix.

Original: Atwood & Towsley (2016) propose DCNN, which calculates powers of the transition matrix and keeps each power in a separate channel until the classiﬁcation sub-network at the end.
Revised: Atwood & Towsley (2016) propose DCNN, which calculates powers of the transition matrix and keeps each power in a separate channel until the classiﬁcation sub-network at the end.

Original: Their model is therefore similar to our work in that it also falls under (cid:80)
Revised: Their model is therefore similar to our work in that it also falls under (cid:80)

Original: k q k g ( (cid:101)
Revised: k q k g ( (cid:101)

Original: A k ) .
Revised: A k ) .

Original: However, where their model multiplies features with each power (cid:101)
Revised: However, where their model multiplies features with each power (cid:101)

Original: A k once, our model makes use of GCN’s
Revised: A k once, our model makes use of GCN’s

Original: (Kipf & Welling, 2017) that multiply by (cid:101)
Revised: (Kipf & Welling, 2017) that multiply by (cid:101)

Original: A k at every GCN layer (see Eq. 2).
Revised: A k at every GCN layer (see Eq. 2).

Original: Thus, DCNN model (Atwood & Towsley, 2016) is a special case of ours, when GCN module contains only one layer, as explained in Section 3.6.
Revised: Thus, DCNN model (Atwood & Towsley, 2016) is a special case of ours, when GCN module contains only one layer, as explained in Section 3.6.

Original:  C ONCLUSIONS AND F UTURE W ORK
Revised:  C ONCLUSIONS AND F UTURE W ORK

Original: In this paper, we propose a meta-model that can run arbitrary Graph Convolution models, such as
Revised: In this paper, we propose a meta-model that can run arbitrary Graph Convolution models, such as

Original: GCN (Kipf & Welling, 2017) and SAGE (Hamilton et al., 2017), on the output of random walks.
Revised: GCN (Kipf & Welling, 2017) and SAGE (Hamilton et al., 2017), on the output of random walks.

Original: Traditional Graph Convolution models operate on the normalized adjacency matrix.
Revised: Traditional Graph Convolution models operate on the normalized adjacency matrix.

Original: We make multiple instantiations of such models, feeding each instantiation a power of the adjacency matrix, and then concatenating the output of all instances into a classiﬁcation sub-network.
Revised: We make multiple instantiations of such models, feeding each instantiation a power of the adjacency matrix and then concatenating the output of all instances into a classiﬁcation sub-network.

Original: Our model, Network of GCNs (and similarly, Network of SAGE), is end-to-end trainable, and is able to directly learn information across near or distant neighbors.
Revised: Our model, Network of GCNs (and similarly, Network of SAGE), is end-to-end trainable, and is able to directly learn information across near or distant neighbors.

Original: We inspect the distribution of parameter weights in our classiﬁcation sub-network, which reveal to us that our model is effectively able to circumvent adversarial perturbations on the input by shifting weights towards model instances consuming higher powers of the adjacency matrix.
Revised: We inspect the distribution of parameter weights in our classiﬁcation sub-network, which reveal to us that our model is effectively able to circumvent adversarial perturbations on the input by shifting weights towards model instances consuming higher powers of the adjacency matrix.

Original: For future work, we plan to extend our methods to a stochastic implementation and tackle other (larger) graph datasets.
Revised: For future work, we plan to extend our methods to a stochastic implementation and tackle other (larger) graph datasets.

Original:  A PPENDIX 
Revised:  A PPENDIX 7. A LGORITHM FOR N ETWORK OF SAGE

Original: Algorithms 4 and 5, respectively, deﬁne SAGE Hamilton et al.
Revised: Algorithms 4 and 5, respectively, deﬁne SAGE Hamilton et al.

Original: (2017) and Network of SAGE (NSage).
Revised: (2017) and Network of SAGE (NSAGE).

Original: Algorithm 4 assumes mean-pool aggregation by Hamilton et al.
Revised: Algorithm 4 assumes mean-pool aggregation by Hamilton et al.

Original: (2017), which performs on-par to their top performer max-pool aggregation.
Revised: (2017), which performs on-par to their top performer max-pool aggregation.

Original: Further, Algorithm 4 operates in full-batch while Hamilton et al.
Revised: Further, Algorithm 4 operates in full-batch while Hamilton et al.

Original: (2017) offer a stochastic implementation with edge sampling.
Revised: (2017) offer a stochastic implementation with edge sampling.

Original: Nonetheless, their proposed stochastic implementation should be wrapped in a network, though we would need a way to approximate (e.g. sample entries) from dense ˆ A k as k increases.
Revised: Nonetheless, their proposed stochastic implementation should be wrapped in a network, though we would need a way to approximate (e.g. sample entries) from dense ˆ A k as k increases.

Original: We leave this as future work.
Revised: We leave this as future work.

Original: Require:
Revised: Require:

Original: ˆ A is a normalization of A 1: function S AGE M ODEL ( ˆ A , X , L ) 2:
Revised: ˆ A is a normalization of A 1: function S AGE M ODEL ( ˆ A , X , L ) 2:

Original: Z ← X 3: for i = 1 to L do 4:
Revised: Z ← X 3: for i = 1 to L do 4:

Original: Z ← σ ( (cid:2)
Revised: Z ← σ ( (cid:2)

Original: Z ˆ AZ (cid:3) W ( i ) ) 5:
Revised: Z ˆ AZ (cid:3) W ( i ) ) 5:

Original: Z ← L2N ORMALIZE R OWS ( Z ) 6: return Z
Revised: Z ← L2N ORMALIZE R OWS ( Z ) 6: return Z

Original: Algorithm
Revised: Algorithm

Original: 1: function N SAGE ( A , X ) 2:
Revised: 1: function N SAGE ( A , X ) 2:

Original: D ← diag ( A 1 ) (cid:46)
Revised: D ← diag ( A 1 ) (cid:46)

Original: Sum rows 3: ˆ A ← D − 1 A 4: return N ETWORK ( S AGE M ODEL , ˆ A, X, 2)
Revised: Sum rows 3: ˆ A ← D − 1 A 4: return N ETWORK ( S AGE M ODEL , ˆ A, X, 2)

Original: Using SAGE with mean-pooling aggregation is very similar to a vanilla GCN model but with three differences.
Revised: Using SAGE with mean-pooling aggregation is very similar to a vanilla GCN model but with three differences.

Original: First, the choice of adjacency normalization ( D − 1 A versus D − 12 AD − 12 ).
Revised: First, the choice of adjacency normalization ( D − 1 A versus D − 12 AD − 12 ).

Original: Second, the skip connections in line 4, which concatenates the features with the adjacency-multiplied (i.e. diffused) features.
Revised: Second, the skip connections in line 4, which concatenates the features with the adjacency-multiplied (i.e. diffused) features.

Original: We believe this is analogous in intuition of incorporating ˆ A 0 in our model, which keeps the original features.
Revised: We believe this is analogous in intuition of incorporating ˆ A 0 in our model, which keeps the original features.

Original: Third, the use of node-wise L2 feature normalization at line 5, which is equivalent to applying a layernorm transformation J. Ba (2016).
Revised: Third, the use of node-wise L2 feature normalization at line 5, which is equivalent to applying a layernorm transformation J. Ba (2016).

Original: Nonetheless, it is worth noting
Revised: Nonetheless, it is worth noting

Original: Hamilton et al.
Revised: Hamilton et al.

Original: (2017)’s formulation of SAGE is ﬂexible to allow different aggregations, such as max-pooling or LSTM, which further deviates SAGE from GCN.
Revised: (2017)’s formulation of SAGE is ﬂexible to allow different aggregations, such as max-pooling or LSTM, which further deviates SAGE from GCN.

Original: 6. S ENSITIVITY A NALYSIS
Revised: 7. S ENSITIVITY A NALYSIS

Original: Earlier, in Table 2, we showed the test performance corresponding to the model performing best on the validation split.
Revised: Earlier, in Table 2, we showed the test performance corresponding to the model performing best on the validation split.

Original: The number of labeled nodes are small, and such model selection is important to avoid overﬁtting.
Revised: The number of labeled nodes are small, and such model selection is important to avoid overﬁtting.

Original: For example, there can be up to 10% relative test accuracy difference when training the same model architecture but with different random seed.
Revised: For example, there can be up to 10% relative test accuracy difference when training the same model architecture but with different random seed.

Original: In this section, we programatically sweep hyperparameters r , K , choice of classiﬁcation network ( ∈ { fc , a } ), and whether or not we enable ˆ A 0 , for both N-GCN and N-SAGE models.
Revised: In this section, we programatically sweep hyperparameters r , K , choice of classiﬁcation network ( ∈ { fc , a } ), and whether or not we enable ˆ A 0 , for both N-GCN and N-SAGE models.

Original: The settings when ( K = 1 , r = 1 , and ˆ A 0 disabled), correspond to the vanilla base model.
Revised: The settings when ( K = 1 , r = 1 , and ˆ A 0 disabled), correspond to the vanilla base model.

Original: Further, the settings when ( K = 1 , r > 1 , and ˆ A 0 disabled), correspond to an ensemble of the base model.
Revised: Further, the settings when ( K = 1 , r > 1 , and ˆ A 0 disabled), correspond to an ensemble of the base model.

Original: These cases are outperformed when K > 1 , showing that unmodiﬁed random walks indeed help these convolutional methods perform better, by gathering information from nearby and distant nodes.
Revised: These cases are outperformed when K > 1 , showing that unmodiﬁed random walks indeed help these convolutional methods perform better, by gathering information from nearby and distant nodes.

Original: The automatically generated tables are shown below:
Revised: The automatically generated tables are shown below:

Original: K =
Revised: K =

Original: K = K =
Revised: K = K =

Original: K =
Revised: K =

Original: K =r =79 . [Table] ± 
Revised: K =r =79 . 0 ± 0 .79 .

Original: [Table]
Revised: 5 ± 0 .79 . 3 ± 0 .79 . 4 ± 0 .79 . 4 ± 0 .r

Original: [Table]
Revised: =

Original: r =79 . 1 ± 
Revised: 1 ± 0 .79 .

Original: [Table] [Table] [Table]
Revised: 3 ± 0 .79 . 4 ± 0 .79 .

Original: [Table]
Revised: 4 ± 0 .79 . 4 ± 0 .r

Original: r =78 .
Revised: =78 .

Original: 9 ± 0  .
Revised: 9 ± 0 .79 .

Original: [Table]
Revised: 4 ± 0 .79 . 3 ± 0 .79 . 5 ± 0 .79 . 5 ± 0 .

Original: [Table]
Revised: [Table]

Original: [Table] [Table][Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original: [Table] [Table]
Revised: [Table] [Table]