Original: A
Revised: A

Original: U NIVERSAL
Revised: U NIVERSAL

Original: S ELF -A TTENTION
Revised: S ELF -A TTENTION

Original: G
Revised: G

Original: RAPH
Revised: RAPH

Original: N EURAL
Revised: N EURAL

Original: N ETWORK
Revised: N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Existing graph embedding models often have weaknesses in exploiting graph structure similarities, potential dependencies among nodes and global network properties.
Revised: Existing graph embedding models often have weaknesses in exploiting graph structure similarities, potential dependencies among nodes and global network properties.

Original: To this end, we present U2GNN, a novel embedding model leveraging on the strength of the recently introduced universal self-attention network (Dehghani et al., 2019), to learn low-dimensional embeddings of graphs which can be used for graph classiﬁcation.
Revised: To this end, we present U2GNN, a novel embedding model leveraging on the strength of the recently introduced universal self-attention network (Dehghani et al., 2019), to learn low-dimensional embeddings of graphs which can be used for graph classiﬁcation.

Original: In particular, given an input graph, U2GNN ﬁrst applies a self-attention computation, which is then followed by a recurrent transition to iteratively memorize its attention on vector representations of each node and its neighbors across each iteration.
Revised: In particular, given an input graph, U2GNN ﬁrst applies a self-attention computation, which is then followed by a recurrent transition to iteratively memorize its attention on vector representations of each node and its neighbors across each iteration.

Original: Thus, U2GNN can address the weaknesses in the existing models in order to produce plausible node embeddings whose sum is the ﬁnal embedding of the whole graph.
Revised: Thus, U2GNN can address the weaknesses in the existing models in order to produce plausible node embeddings whose sum is the ﬁnal embedding of the whole graph.

Original: Experimental results in both supervised and unsupervised fashions show that our U2GNN produces new state-of-the-art performances on a range of well-known benchmark datasets for the graph classiﬁcation task.
Revised: Experimental results in both supervised and unsupervised fashions show that our U2GNN produces new state-of-the-art performances on a range of well-known benchmark datasets for the graph classiﬁcation task.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Many real-world and scientiﬁc data are represented in forms of graphs, e.g. data from knowledge graphs, recommender systems, social and citation networks as well as telecommunication and biological networks (Battaglia et al., 2018; Zhang et al., 2018c).
Revised: Many real-world and scientiﬁc data are represented in forms of graphs, e.g. data from knowledge graphs, recommender systems, social and citation networks as well as telecommunication and biological networks (Battaglia et al., 2018; Zhang et al., 2018c).

Original: In general, a graph can be viewed as a network of nodes and edges, where nodes correspond to individual objects and edges encode relationships among those objects.
Revised: In general, a graph can be viewed as a network of nodes and edges, where nodes correspond to individual objects and edges encode relationships among those objects.

Original: For example, in online forum, each discussion thread can be constructed as a graph where nodes represent users and edges represent commenting activities between users (Yanardag & Vishwanathan, 2015).
Revised: For example, in online forum, each discussion thread can be constructed as a graph where nodes represent users and edges represent commenting activities between users (Yanardag & Vishwanathan, 2015).

Original: Early approaches focus on computing the similarities among graphs to build a graph kernel for graph classiﬁcation (G ¨ artner et al., 2003;
Revised: Early approaches focus on computing the similarities among graphs to build a graph kernel for graph classiﬁcation (G ¨ artner et al., 2003;

Original: Kashima et al., 2003; Borgwardt & Kriegel, 2005; Shervashidze et al., 2009; Vishwanathan et al., 2010; Shervashidze et al., 2011; Yanardag & Vishwanathan, 2015;
Revised: Kashima et al., 2003; Borgwardt & Kriegel, 2005; Shervashidze et al., 2009; Vishwanathan et al., 2010; Shervashidze et al., 2011; Yanardag & Vishwanathan, 2015;

Original: Narayanan et al., 2017; Ivanov & Burnaev, 2018).
Revised: Narayanan et al., 2017; Ivanov & Burnaev, 2018).

Original: These graph kernel-based approaches treat each atomic substructure (e.g.,graphlet, subtree structure, random walk or shortest path) as an individual feature, and count their frequencies to construct a numerical vector to represent the entire graph, hence they ignore node attributes , substructure similarities and global network properties.
Revised: These graph kernel-based approaches treat each atomic substructure (e.g., subtree structure, random walk or shortest path) as an individual feature, and count their frequencies to construct a numerical vector to represent the entire graph, hence they ignore node attributes , substructure similarities and global network properties.

Original: One recent notable strand is to learn low-dimensional continuous embeddings of the whole graphs (Hamilton et al., 2017b;
Revised: One recent notable strand is to learn low-dimensional continuous embeddings of the whole graphs (Hamilton et al., 2017b;

Original: Zhang et al., 2018a;
Revised: Zhang et al., 2018a;

Original: Zhou et al., 2018), and then use these learned embeddings to train a classiﬁer to predict graph labels (Wu et al., 2019).
Revised: Zhou et al., 2018), and then use these learned embeddings to train a classiﬁer to predict graph labels (Wu et al., 2019).

Original: Advanced approaches in this direction have attempted to exploit graph neural network (Scarselli et al., 2009), capsule network (Sabour et al., 2017) or graph convolutional neural network (Kipf & Welling, 2017; Hamilton et al., 2017a) for supervised learning objectives (Li et al., 2016;
Revised: Advanced approaches in this direction have attempted to exploit graph neural network (Scarselli et al., 2009), capsule network (Sabour et al., 2017) or graph convolutional neural network (Kipf & Welling, 2017; Hamilton et al., 2017a) for supervised learning objectives (Li et al., 2016;

Original: Niepert et al., 2016; Zhang et al., 2018b;
Revised: Niepert et al., 2016; Zhang et al., 2018b;

Original: Ying et al., 2018; Verma & Zhang, 2018; Xu et al., 2019; Xinyi & Chen, 2019; Maron et al., 2019b;
Revised: Ying et al., 2018; Verma & Zhang, 2018; Xu et al., 2019; Xinyi & Chen, 2019; Maron et al., 2019b;

Original: Chen et al., 2019).
Revised: Chen et al., 2019).

Original: These graph neural network (GNN)-based approaches usually consist of two common phases: the propagating phase and the readout phase.
Revised: These graph neural network (GNN)-based approaches usually consist of two common phases: the propagating phase and the readout phase.

Original: The former phase aims to iteratively update vector representation of each node by recursively aggregating representations of its neighbors, and then the latter phase applies a pooling function (e.g., mean, max or sum pooling) on output node representations to produce an embedding of each entire graph; and this graph embedding is used to predict the graph label.
Revised: The former phase aims to iteratively update vector representation of each node by recursively aggregating representations of its neighbors, and then the latter phase applies a pooling function (e.g., mean, max or sum pooling) on output node representations to produce an embedding of each entire graph; and this graph embedding is used to predict the graph label.

Original: We ﬁnd that these approaches are currently showing very promis- ing performances, nonetheless the dependency aspect among nodes, which often exhibit strongly in many kinds of real-world networks, has not been exploited effectively.
Revised: We ﬁnd that these approaches are currently showing very promising performances, nonetheless the dependency aspect among nodes, which often exhibit strongly in many kinds of real-world networks, has not been exploited effectively due to the straightforward computations within the propagating phase.

Original: Very recently, the universal self-attention network (Dehghani et al., 2019) has been shown to be very powerful in NLP tasks such as question answering, machine translation and language modeling.
Revised: Very recently, the universal self-attention network (Dehghani et al., 2019) has been shown to be very powerful in NLP tasks such as question answering, machine translation and language modeling.

Original: Inspired by this new attention network, we propose U2GNN – a novel universal self-attention graph neural network to learn plausible node and graph embeddings.
Revised: Inspired by this new attention network, we propose U2GNN – a novel universal self-attention graph neural network to learn plausible node and graph embeddings.

Original: Our intuition comes from an observation that the recurrent attention process in the universal self-attention network can memorize implicit dependencies between each node and its neighbors from previous iterations, which can be then aggregated to further capture the dependencies among substructures into latent representations in subsequent iterations; this process, hence, can capture both local and global graph structures.
Revised: Our intuition comes from an observation that the recurrent attention process in the universal self-attention network can memorize implicit dependencies between each node and its neighbors from previous iterations, which can be then aggregated to further capture the dependencies among substructures into latent representations in subsequent iterations; this process, hence, can capture both local and global graph structures.

Original: Algorithmically, at each timestep, our proposed U2GNN iteratively exchanges a node representation with its neighborhood representations using a self-attention mechanism (Vaswani et al., 2017) followed by a recurrent transition to infer node embeddings.
Revised: Algorithmically, at each timestep, our proposed U2GNN iteratively exchanges a node representation with its neighborhood representations using a self-attention mechanism (Vaswani et al., 2017) followed by a recurrent transition to infer node embeddings.

Original: We ﬁnally take the sum of all learned node embeddings to obtain the embedding of the whole graph.
Revised: We ﬁnally take the sum of all learned node embeddings to obtain the embedding of the whole graph.

Original: Our main contributions are as follows:
Revised: Our main contributions are as follows:

Original: • In our proposed U2GNN, the novelty of memorizing the dependencies among nodes implies that U2GNN can explore the graph structure similarities locally and globally – an important feature that most of existing
Revised: • In our proposed U2GNN, the novelty of memorizing the dependencies among nodes implies that U2GNN can explore the graph structure similarities locally and globally – an important feature that most of existing

Original: approaches
Revised: approaches

Original: are unable to do.
Revised: are unable to do.

Original: • U2GNN can be seen as a general framework where we prove the powerfulness of our model in both supervised or unsupervised fashions.
Revised: • U2GNN can be seen as a general framework where we prove the powerfulness of our model in both supervised or unsupervised fashions.

Original: The experimental results on 9 well-known benchmark datasets  show that both our supervised and unsupervised U2GNN models produce new state-ofthe-art (SOTA) accuracies in most of benchmark cases.
Revised: The experimental results on 9 well-known benchmark datasets for the graph classiﬁcation task show that both our supervised and unsupervised U2GNN models produce new state-of-the-art (SOTA) accuracies in most of benchmark cases.

Original: • To our best of knowledge, our work is the ﬁrst to show that a unsupervised model can noticeably outperform up-to-date supervised approaches by a large margin.
Revised: • To our best of knowledge, our work is the ﬁrst to show that a unsupervised model can noticeably outperform up-to-date supervised approaches by a large margin.

Original: Therefore, our most important contribution is to suggest that future GNN works should pay more attention to the unsupervised fashion as
Revised: Therefore, we suggest that future GNN works should pay more attention to the unsupervised fashion as

Original: well as not comparing supervised models with unsupervised models together.
Revised: well as not comparing supervised models with unsupervised models together.

Original: This is important in both industry and academic applications in reality where expanding unsupervised GNN models is more suitable due to the limited availability of class labels.
Revised: This is important in both industry and academic applications in reality where expanding unsupervised GNN models is more suitable due to the limited availability of class labels.

Original:  R ELATED WORK
Revised:  R ELATED WORK

Original: Early popular approaches are based on “graph kernel” which aims to recursively decompose each graph into “atomic substructures” (e.g., graphlets, subtree structures, random walks or shortest paths) in order to measure the similarity between two graphs (G ¨ artner et al., 2003).
Revised: Early popular approaches are based on “graph kernel” which aims to recursively decompose each graph into “atomic substructures” (e.g., graphlets, subtree structures, random walks or shortest paths) in order to measure the similarity between two graphs (G ¨ artner et al., 2003).

Original: For this reason, we can view each atomic substructure as a word token and each graph as a text document, hence we represent a collection of graphs as a document-term matrix which describes the normalized frequency of terms in documents.
Revised: For this reason, we can view each atomic substructure as a word token and each graph as a text document, hence we represent a collection of graphs as a document-term matrix which describes the normalized frequency of terms in documents.

Original: Then, we can use a dot product to compute the similarities among graphs to derive a kernel matrix used to measure the classiﬁcation performance using a kernel-based learning algorithm such as Support Vector Machines (SVM) (Hofmann et al., 2008).
Revised: Then, we can use a dot product to compute the similarities among graphs to derive a kernel matrix used to measure the classiﬁcation performance using a kernel-based learning algorithm such as Support Vector Machines (SVM) (Hofmann et al., 2008).

Original: We refer to an overview of the graph kernel-based approaches in (Nikolentzos et al., 2019; Kriege et al., 2019).
Revised: We refer to an overview of the graph kernel-based approaches in (Nikolentzos et al., 2019; Kriege et al., 2019).

Original: Since the introduction of word embedding models i.e., Word2Vec (Mikolov et al., 2013) and
Revised: Since the introduction of word embedding models i.e., Word2Vec (Mikolov et al., 2013) and

Original: Doc2Vec (Le & Mikolov, 2014), there have been several efforts attempted to apply them for the graph classiﬁcation task.
Revised: Doc2Vec (Le & Mikolov, 2014), there have been several efforts attempted to apply them for the graph classiﬁcation task.

Original: Deep Graph Kernel (DGK) (Yanardag & Vishwanathan, 2015) applies
Revised: Deep Graph Kernel (DGK) (Yanardag & Vishwanathan, 2015) applies

Original: Word2Vec to learn embeddings of atomic substructures to create the kernel matrix.
Revised: Word2Vec to learn embeddings of atomic substructures to create the kernel matrix.

Original: Graph2Vec (Narayanan et al., 2017) employs Doc2Vec to obtain embeddings of entire graphs in order to train
Revised: Graph2Vec (Narayanan et al., 2017) employs Doc2Vec to obtain embeddings of entire graphs in order to train

Original: a
Revised: a

Original: SVM classiﬁer to perform classiﬁcation.
Revised: SVM classiﬁer to perform classiﬁcation.

Original: Anonymous Walk Embedding (AWE) (Ivanov & Burnaev, 2018) maps random walks into “anonymous walks” which are considered as word tokens, and then utilizes Doc2Vec to achieve the graph embeddings to produce the kernel matrix.
Revised: Anonymous Walk Embedding (AWE) (Ivanov & Burnaev, 2018) maps random walks into “anonymous walks” which are considered as word tokens, and then utilizes Doc2Vec to achieve the graph embeddings to produce the kernel matrix.

Original: In parallel, another recent line of work has focused on using deep neural networks to perform the graph classiﬁcation in a supervised manner.
Revised: In parallel, another recent line of work has focused on using deep neural networks to perform the graph classiﬁcation in a supervised manner.

Original: PATCHY-SAN (Niepert et al., 2016) adapts a graph labeling procedure to generate a ﬁxed-length sequence of nodes from an input graph, and orders k -hop neighbors for each node in the generated sequence according to their graph labelings; PATCHY-SAN then selects a ﬁxed number of ordered neighbors for each node and applies a convolutional neural network to classify the input graph.
Revised: PATCHY-SAN (Niepert et al., 2016) adapts a graph labeling procedure to generate a ﬁxed-length sequence of nodes from an input graph, and orders k -hop neighbors for each node in the generated sequence according to their graph labelings; PATCHY-SAN then selects a ﬁxed number of ordered neighbors for each node and applies a convolutional neural network to classify the input graph.

Original: MPNN (Gilmer et al., 2017), DGCNN (Zhang et al., 2018b) and D IFF P OOL (Ying et al., 2018) are end-to-end supervised models which share similar two-phase process by (i) using stacked multiple graph convolutional layers (e.g., GCN layer (Kipf & Welling,
Revised: MPNN (Gilmer et al., 2017), DGCNN (Zhang et al., 2018b) and D IFF P OOL (Ying et al., 2018) are end-to-end supervised models which share similar two-phase process by (i) using stacked multiple graph convolutional layers (e.g., GCN layer (Kipf & Welling,

Original: 2017) or GraphSAGE layer (Hamilton et al., 2017a)) to aggregate node feature vectors, and (ii) applying a graph-level pooling layer (e.g., mean, max or sum pooling, sort pooling or differentiable pooling) to obtain the graph embeddings which are then fed to a fully-connected layer followed by a softmax layer to predict the graph labels.
Revised: 2017) or GraphSAGE layer (Hamilton et al., 2017a)) to aggregate node feature vectors, and (ii) applying a graph-level pooling layer (e.g., mean, max or sum pooling, sort pooling or differentiable pooling) to obtain the graph embeddings which are then fed to a fully-connected layer followed by a softmax layer to predict the graph labels.

Original: Graph neural networks (GNNs) (Scarselli et al., 2009) aim to iteratively update the vector representation of each node by recursively propagating the representations of its neighbors using a recurrent function until convergence.
Revised: Graph neural networks (GNNs) (Scarselli et al., 2009) aim to iteratively update the vector representation of each node by recursively propagating the representations of its neighbors using a recurrent function until convergence.

Original: The recurrent function can be a neural network e.g., gated recurrent unit (GRU) (Li et al., 2016), or multi-layer perceptron (MLP) (Xu et al., 2019).
Revised: The recurrent function can be a neural network e.g., gated recurrent unit (GRU) (Li et al., 2016), or multi-layer perceptron (MLP) (Xu et al., 2019).

Original: Note that both stacked
Revised: Note that both stacked

Original: GCN and GraphSAGE multiple layers can be seen as variants of the recurrent function in GNNs.
Revised: GCN and GraphSAGE multiple layers can be seen as variants of the recurrent function in GNNs.

Original: Other graph embedding models are brieﬂy summarized in (Zhou et al., 2018;
Revised: Other graph embedding models are brieﬂy summarized in (Zhou et al., 2018;

Original: Zhang et al., 2018c;
Revised: Zhang et al., 2018c;

Original: Wu et al., 2019).
Revised: Wu et al., 2019).

Original:  T HE PROPOSED U2GNN
Revised:  T HE PROPOSED U2GNN

Original: In this section, we detail how to construct our U2GNN and then present how U2GNN learns model parameters to produce node and graph embeddings.
Revised: In this section, we detail how to construct our U2GNN and then present how U2GNN learns model parameters to produce node and graph embeddings.

Original: Graph classiﬁcation.
Revised: Graph classiﬁcation.

Original: Given a set of graphs {G 1 , G 2 , ..., G N } ⊆ G and their corresponding class labels { y 1 , y 2 , ..., y N } ⊆ Y , our U2GNN aims to learn a plausible embedding o G of each entire graph G in order to predict its label y .
Revised: Given a set of graphs {G 1 , G 2 , ..., G N } ⊆ G and their corresponding class labels { y 1 , y 2 , ..., y N } ⊆ Y , our U2GNN aims to learn a plausible embedding o G of each entire graph G in order to predict its label y .

Original: Each graph G is deﬁned as G =
Revised: Each graph G is deﬁned as G =

Original: ( V , E , X ) , where V is a set of nodes, E is a set of edges, and
Revised: ( V , E , X ) , where V is a set of nodes, E is a set of edges, and

Original: X ∈ R |V|× d represents feature vectors of nodes.
Revised: X ∈ R |V|× d represents feature vectors of nodes.

Original: In U2GNN, as illustrated in Figures 1 and 2, we use a universal self-attention network (Dehghani et al., 2019) to learn a node embedding o v of each node v ∈ V , and then o G is simply returned by summing all learned node embeddings as follows:
Revised: In U2GNN, as illustrated in Figures 1 and 2, we use a universal self-attention network (Dehghani et al., 2019) to learn a node embedding o v of each node v ∈ V , and then o G is simply returned by summing all learned node embeddings as follows:

Original: [Equation]
Revised: [Equation]

Original: Constructing U2GNN.
Revised: Constructing U2GNN.

Original: Formally, given an input graph G , we uniformly sample a set of N neighbors for each v ∈ V , and then use node v and its neighbors for the U2GNN learning process.
Revised: Formally, given an input graph G , we uniformly sample a set of N neighbors for each v ∈ V , and then use node v and its neighbors for the U2GNN learning process.

Original: For example, as illustrated in Figure 2, we generate a set of N = 3 neighbors { 1 , 2 , 4 } for node 3, and then consider { 3 , 1 , 2 , 4 } as an input to U2GNN where we leverage on the universal self-attention network (Dehghani et al., 2019) to learn an effective embedding of node 3.
Revised: For example, as illustrated in Figure 2, we generate a set of N = 3 neighbors { 1 , 2 , 4 } for node 3, and then consider { 3 , 1 , 2 , 4 } as an input to U2GNN where we leverage on the universal self-attention network (Dehghani et al., 2019) to learn an effective embedding of node 3.

Original: Intuitively, the universal self-attention network can help to better aggregate feature vectors from neighbors of a given node to produce its plausible embedding.
Revised: Intuitively, the universal self-attention network can help to better aggregate feature vectors from neighbors of a given node to produce its plausible embedding.

Original: In particular, each node and its neighbors is transformed into a sequence of feature vectors which are then iteratively reﬁned at each timestep – using a self-attention mechanism (Vaswani et al., 2017) followed by a recurrent transition along with adding residual connection (He et al., 2016) and layer normalization (L AYER N ORM )
Revised: In particular, each node and its neighbors is transformed into a sequence of feature vectors which are then iteratively reﬁned at each timestep – using a self-attention mechanism (Vaswani et al., 2017) followed by a recurrent transition along with adding residual connection (He et al., 2016) and layer normalization (L AYER N ORM )

Original: (Ba et al., 2016).
Revised: (Ba et al., 2016).

Original: Given a sampled sequence of ( N + 1 ) nodes { v i } N +

Revised: Given a sampled sequence of ( N + 1 ) nodes { v i } N +

Original: i
Revised: i

Original: =1 where { v 2 , v 3 , ..., v N +1 } are neighbors of v 1 , we obtain an input sequence of feature vectors { h 0 v i } N +1 i =1 for which h 0 v i
Revised: =1 where { v 2 , v 3 , ..., v N +1 } are neighbors of v 1 , we obtain an input sequence of feature vectors { h 0 v i } N +1 i =1 for which h 0 v i

Original: =
Revised: =

Original: X v i ∈ R d .
Revised: X v i ∈ R d .

Original: In U2GNN, at each step t , we consider { h t − 1 v i } N +

Revised: In U2GNN, at each step t , we consider { h t − 1 v i } N +

Original: i
Revised: i

Original: =1 as an input sequence and produce an output sequence { h t v i } N +

Revised: =1 as an input sequence and produce an output sequence { h t v i } N +

Original: i = for which h t v i ∈ R d as follows:
Revised: i = for which h t v i ∈ R d as follows:

Original: [Equation]
Revised: [Equation]

Original: where T RANSITION ( . ) and ATT ( . ) denote a feed-forward network and a self-attention network respectively as follows:
Revised: where T RANSITION ( . ) and ATT ( . ) denote a feed-forward network and a self-attention network respectively as follows:

Original: [Equation]
Revised: [Equation]

Original: where W 1 ∈ R k × d and W 2 ∈ R d × k are weight matrices, and b 1 and b 2 are bias parameters, and:
Revised: where W 1 ∈ R k × d and W 2 ∈ R d × k are weight matrices, and b 1 and b 2 are bias parameters, and:

Original: [Equation]
Revised: [Equation]

Original: where V ∈ R d × d is a value-projection weight matrix; α i,j is an attention weight which is computed using the softmax function over scaled dot products between the i -th and j -th input nodes:
Revised: where V ∈ R d × d is a value-projection weight matrix; α i,j is an attention weight which is computed using the softmax function over scaled dot products between the i -th and j -th input nodes:

Original: [Equation]
Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where Q ∈ R d × d and K ∈ R d × d are query-projection and key-projection matrices, respectively.
Revised: where Q ∈ R d × d and K ∈ R d × d are query-projection and key-projection matrices, respectively.

Original: 1 Input : A graph G =
Revised: 1 Input : A graph G =

Original: ( V , E , X ) .
Revised: ( V , E , X ) .

Original: 2 for v ∈ V do
Revised: 2 for v ∈ V do

Original: S AMPLE a set C v of N neighbors of v ∀ u ∈ { v } ∪ C v for t = 1, 2, ..., T do
Revised: S AMPLE a set C v of N neighbors of v ∀ u ∈ { v } ∪ C v for t = 1, 2, ..., T do

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: [Figure]
Revised: [Figure]

Original: [Equation]
Revised: [Equation]

Original: Learning parameters of U2GNN in “unsupervised” fashion: After T steps, we use the vector representation h T v 1 to infer node embeddings o v 1 .
Revised: Learning parameters of U2GNN in “unsupervised” fashion: After T steps, we use the vector representation h T v 1 to infer node embeddings o v 1 .

Original: For example, as shown in Figure 2, we have v 1 = 3 , v 2 = 1 , v 3 = 2 and v 4 = 4 , and then consider h T 3 to infer o 3 .
Revised: For example, as shown in Figure 2, we have v 1 = 3 , v 2 = 1 , v 3 = 2 and v 4 = 4 , and then consider h T 3 to infer o 3 .

Original: We learn our model parameters (including the weight matrices and biases as well as node embeddings o v ) by minimizing the sampled softmax loss function (Jean et al., 2015) applied to node v ∈ V as follows:
Revised: We learn our model parameters (including the weight matrices and biases as well as node embeddings o v ) by minimizing the sampled softmax loss function (Jean et al., 2015) applied to node v ∈ V as follows:

Original: [Equation]
Revised: [Equation]

Original: where V (cid:48) is a subset sampled from V .
Revised: where V (cid:48) is a subset sampled from V .

Original: We brieﬂy describe the general learning process of our proposed U2GNN model in Algorithm 1.
Revised: We brieﬂy describe the general learning process of our proposed U2GNN model in Algorithm 1.

Original: Here, the learned node embeddings o v are used as the ﬁnal representations of nodes v ∈ V .
Revised: Here, the learned node embeddings o v are used as the ﬁnal representations of nodes v ∈ V .

Original: After that, we obtain the plausible embedding o G of the graph G by summing all learned node embeddings as mentioned in Equation 1.
Revised: After that, we obtain the plausible embedding o G of the graph G by summing all learned node embeddings as mentioned in Equation 1.

Original: Finally, we employ a logistic regression classiﬁer from LIBLINEAR
Revised: Finally, we employ a logistic regression classiﬁer from LIBLINEAR

Original: (Fan et al., 2008) to predict the graph labels.
Revised: (Fan et al., 2008) to predict the graph labels.

Original: Learning parameters of U2GNN in “supervised” fashion: After T steps, we use h T v 1 as the ﬁnal embedding of node v 1 ∈ V (i.e., o v 1 =
Revised: Learning parameters of U2GNN in “supervised” fashion: After T steps, we use h T v 1 as the ﬁnal embedding of node v 1 ∈ V (i.e., o v 1 =

Original: h T v 1 ).
Revised: h T v 1 ).

Original: We then feed o G to a fully connected layer to predict the graph labels.
Revised: We then feed o G to a fully connected layer to predict the graph labels.

Original: We learn the model parameters by minimizing thesoftmax cross entropy loss function.
Revised: We learn the model parameters by minimizing the cross entropy loss function.

Original: Intuition :
Revised: Intuition :

Original: In general, on the node level, each node and its neighbors are iteratively attended in the recurrent process with weight matrices shared across timesteps and iterations, thus U2GNN can memorize the potential dependencies among nodes within substructures.
Revised: In general, on the node level, each node and its neighbors are iteratively attended in the recurrent process with weight matrices shared across timesteps and iterations, thus U2GNN can memorize the potential dependencies among nodes within substructures.

Original: On the graph level,
Revised: On the graph level,

Original: U2GNN views the shared weight matrices as memories to access the updated node-level information from previous iterations to further aggregate broader dependencies among substructures into implicit graph representations in subsequent iterations.
Revised: U2GNN views the shared weight matrices as memories to access the updated node-level information from previous iterations to further aggregate broader dependencies among substructures into implicit graph representations in subsequent iterations.

Original: Therefore, U2GNN is advantageous to capture both global and local graph structures to learn effective node and graph embeddings, leading to state-ofthe-art performances for the graph classiﬁcation task.
Revised: Therefore, U2GNN is advantageous to capture both global and local graph structures to learn effective node and graph embeddings, leading to state-ofthe-art performances for the graph classiﬁcation task.

Original:  E XPERIMENTAL SETUP
Revised:  E XPERIMENTAL SETUP

Original: We prove the effectiveness of our U2GNN on the graph classiﬁcation task using a range of wellknown benchmark datasets for both supervised and unsupervised fashions.
Revised: We prove the effectiveness of our U2GNN on the graph classiﬁcation task using a range of wellknown benchmark datasets for both supervised and unsupervised fashions

Original:  4. D ATASETS
Revised: . 4. D ATASETS

Original: We use 9 well-known datasets consisting of 5 social network datasets (COLLAB, IMDB-B, IMDBM, RDT-B and RDT-M5K)
Revised: We use 9 well-known datasets consisting of 5 social network datasets (COLLAB, IMDB-B, IMDBM, RDT-B and RDT-M5K)

Original: (Yanardag & Vishwanathan, 2015) and 4 bioinformatics datasets (DD,
Revised: (Yanardag & Vishwanathan, 2015) and 4 bioinformatics datasets (DD,

Original: MUTAG, PROTEINS and PTC).
Revised: MUTAG, PROTEINS and PTC).

Original: We follow (Niepert et al., 2016; Zhang et al., 2018b) to use node degrees as features on all social network datasets as these datasets do not have available node features.
Revised: We follow (Niepert et al., 2016; Zhang et al., 2018b) to use node degrees as features on all social network datasets as these datasets do not have available node features.

Original: Table 1 reports the statistics of these datasets.
Revised: Table 1 reports the statistics of these datasets.

Original: [Table]
Revised: [Table]

Original: Social networks datasets.
Revised: Social networks datasets.

Original: COLLAB is a scientiﬁc dataset where each graph represents a collaboration network of a corresponding researcher with other researchers from each of 3 physics ﬁelds; and each graph is labeled to a physics ﬁeld the researcher belong to.
Revised: COLLAB is a scientiﬁc dataset where each graph represents a collaboration network of a corresponding researcher with other researchers from each of 3 physics ﬁelds; and each graph is labeled to a physics ﬁeld the researcher belong to.

Original: IMDB-B and IMDB-M are movie collaboration datasets where each graph is derived from actor/actress and genre information of different movies on IMDB, in which nodes correspond to actors/actresses, and each edge represents a co-appearance of two actors/actresses in the same movie; and each graph is assigned to a genre.
Revised: IMDB-B and IMDB-M are movie collaboration datasets where each graph is derived from actor/actress and genre information of different movies on IMDB, in which nodes correspond to actors/actresses, and each edge represents a co-appearance of two actors/actresses in the same movie; and each graph is assigned to a genre.

Original: RDTB and RDT-M5K are datasets derived from
Revised: RDTB and RDT-M5K are datasets derived from

Original: Reddit community, in which each online discussion thread is viewed as a graph where nodes correspond to users, two users are linked if at least one of them replied to another’s comment; and each graph is labeled to a sub-community the corresponding thread belongs to.
Revised: Reddit community, in which each online discussion thread is viewed as a graph where nodes correspond to users, two users are linked if at least one of them replied to another’s comment; and each graph is labeled to a sub-community the corresponding thread belongs to.

Original: Bioinformatics datasets.
Revised: Bioinformatics datasets.

Original: DD (Dobson & Doig, 2003) is a collection of 1,178 protein network structures with 82 discrete node labels, where each graph is classiﬁed into enzyme or non-enzyme class.
Revised: DD (Dobson & Doig, 2003) is a collection of 1,178 protein network structures with 82 discrete node labels, where each graph is classiﬁed into enzyme or non-enzyme class.

Original: PROTEINS comprises 1,113 graphs obtained from (Borgwardt et al., 2005) to present secondary structure elements (SSEs).
Revised: PROTEINS comprises 1,113 graphs obtained from (Borgwardt et al., 2005) to present secondary structure elements (SSEs).

Original: MUTAG (Debnath et al., 1991) is a collection of 188 nitro compound networks with 7 discrete node labels, where classes indicate a mutagenic effect on a bacterium.
Revised: MUTAG (Debnath et al., 1991) is a collection of 188 nitro compound networks with 7 discrete node labels, where classes indicate a mutagenic effect on a bacterium.

Original: PTC (Toivonen et al., 2003) consists of 344 chemical compound networks with 19 discrete node labels where classes show carcinogenicity for male and female rats.
Revised: PTC (Toivonen et al., 2003) consists of 344 chemical compound networks with 19 discrete node labels where classes show carcinogenicity for male and female rats.

Original: 4. T RAINING PROTOCOL TO LEARN GRAPH EMBEDDINGS
Revised: 4. T RAINING PROTOCOL TO LEARN GRAPH EMBEDDINGS

Original: Coordinate embedding: The relative coordination among nodes might provide meaningful information about graph structure.
Revised: Coordinate embedding: The relative coordination among nodes might provide meaningful information about graph structure.

Original: We follow Dehghani et al.
Revised: We follow Dehghani et al.

Original: (2019) to associate each position i at step t a pre-deﬁned coordinate embedding p ti using the sinusoidal functions (Vaswani et al., 2017), thus we can change Equation 3 in Section 3 to:
Revised: (2019) to associate each position i at step t a pre-deﬁned coordinate embedding p ti using the sinusoidal functions (Vaswani et al., 2017), thus we can change Equation 3 in Section 3 to:

Original: [Equation]
Revised: [Equation]

Original: with p ti, 2 j = sin [Equation]
Revised: with p ti, 2 j = sin( i/ 10000 2 j/d ) + sin( t/ 10000 2 j/d ) p

Original: [Equation]
Revised: [Equation]

Original: From the preliminary experiments, adding coordinate embeddings enhances classiﬁcation results on
Revised: From the preliminary experiments, adding coordinate embeddings enhances classiﬁcation results on

Original: MUTAG and PROTEINS in the unsupervised fashion, hence we use the coordinate embeddings only for these two datasets.
Revised: MUTAG and PROTEINS in the unsupervised fashion, hence we use the coordinate embeddings only for these two datasets.

Original: Hyper-parameter setting to learn our model parameters for all experimental datasets: (i) Regarding the unsupervised fashion, we ﬁx the hidden size of the feed-forward network in Equation 4 to 1024 ( k = 1024 ), and the number of samples in the sampled loss function L U2GNN to( |V (cid:48)
Revised: Hyper-parameter setting to learn our model parameters for all experimental datasets: (i) Regarding the unsupervised fashion, we ﬁx the hidden size of the feed-forward network in Equation 4 to 1024 ( k = 1024 ), and the number of samples in the sampled loss function L U2GNN to( |V (cid:48)

Original: | = 512 ) in Equation 7.
Revised: | = 512 ) in Equation 7.

Original: We set the batch size to 512 for COLLAB, DD and RDT-B; 1024 for
Revised: We set the batch size to 512 for COLLAB, DD and RDT-B; 1024 for

Original: RDT-M5K; and 128 for remaining datasets.
Revised: RDT-M5K; and 128 for remaining datasets.

Original: (ii)
Revised: (ii)

Original: Regarding the supervised fashion, we vary the the hidden size k in { 32, 128, 512, 1024 } and the batch size in {  2, 4 } .
Revised: Regarding the supervised fashion, we vary the the hidden size k in { 32, 128, 512, 1024 } and the batch size in { 1, 2, 4 } .

Original: (iii)
Revised: (iii)

Original: For both the unsupervised and supervised fashions, we use the number N of neighbors sampled for each node from { 4, 8, 16 } and the number T of steps from { 1, 2, 3, 4, 5, 6 } .
Revised: For both the unsupervised and supervised fashions, we use the number N of neighbors sampled for each node from { 4, 8, 16 } and the number T of steps from { 1, 2, 3, 4, 5, 6 } .

Original: We apply the Adam optimizer (Kingma & Ba, 2014) to train our U2GNN model and apply a grid search to select the Adam initial learning rate lr ∈ { 5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3 } .
Revised: We apply the Adam optimizer (Kingma & Ba, 2014) to train our U2GNN model and apply a grid search to select the Adam initial learning rate lr ∈ { 5 e − 5 , 1 e − 4 , 5 e − 4 , 1 e − 3 } .

Original: We run up to 50 epochs and evaluate the model as in what follows.
Revised: We run up to 50 epochs and evaluate the model as in what follows.

Original: 4. E VALUATION PROTOCOL
Revised: 4. E VALUATION PROTOCOL

Original: For each dataset, after obtaining the graph embeddings, we perform the same evaluation process from (Yanardag & Vishwanathan, 2015; Niepert et al., 2016; Zhang et al., 2018b; Xu et al., 2019;
Revised: For each dataset, after obtaining the graph embeddings, we perform the same evaluation process from (Yanardag & Vishwanathan, 2015; Niepert et al., 2016; Zhang et al., 2018b; Xu et al., 2019;

Original: Xinyi & Chen, 2019), which is using 10-fold cross-validation scheme to calculate the classiﬁcation performance for a fair comparison.
Revised: Xinyi & Chen, 2019), which is using 10-fold cross-validation scheme to calculate the classiﬁcation performance for a fair comparison.

Original: Baseline models: We compare our U2GNN with up-to-date strong baselines as follows:
Revised: Baseline models: We compare our U2GNN with up-to-date strong baselines as follows:

Original: • Unsupervised approaches: Graphlet Kernel (GK) (Shervashidze et al., 2009), WeisfeilerLehman kernel (WL) (Shervashidze et al., 2011), Deep Graph Kernel (DGK) (Yanardag & Vishwanathan, 2015) and Anonymous Walk Embedding (AWE) (Ivanov & Burnaev, 2018).
Revised: • Unsupervised approaches: Graphlet Kernel (GK) (Shervashidze et al., 2009), WeisfeilerLehman kernel (WL) (Shervashidze et al., 2011), Deep Graph Kernel (DGK) (Yanardag & Vishwanathan, 2015) and Anonymous Walk Embedding (AWE) (Ivanov & Burnaev, 2018).

Original: • Supervised approaches: PATCHY-SAN (PSCN)
Revised: • Supervised approaches: PATCHY-SAN (PSCN)

Original: (Niepert et al., 2016), Graph Convolutional Network (GCN) (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017a), Deep Graph CNN (DGCNN) (Zhang et al., 2018b), Graph Capsule Convolution Neural Network (GCAPS) (Verma & Zhang, 2018), Capsule Graph Neural Network (CapsGNN) (Xinyi & Chen, 2019),
Revised: (Niepert et al., 2016), Graph Convolutional Network (GCN) (Kipf & Welling, 2017), GraphSAGE (Hamilton et al., 2017a), Deep Graph CNN (DGCNN) (Zhang et al., 2018b), Graph Capsule Convolution Neural Network (GCAPS) (Verma & Zhang, 2018), Capsule Graph Neural Network (CapsGNN) (Xinyi & Chen, 2019),

Original: Graph Isomorphism Network (GIN) (Xu et al., 2019), Graph Feature Network (GFN) (Chen et al., 2019), Invariant-Equivariant Graph Network (IEGN)
Revised: Graph Isomorphism Network (GIN) (Xu et al., 2019), Graph Feature Network (GFN) (Chen et al., 2019), Invariant-Equivariant Graph Network (IEGN)

Original: (Maron et al., 2019b), Provably Powerful
Revised: (Maron et al., 2019b), Provably Powerful

Original: Graph Network (PPGN) (Maron et al., 2019a) and Discriminative Structural Graph Classiﬁcation (DSGC) (Seo et al., 2019).
Revised: Graph Network (PPGN) (Maron et al., 2019a) and Discriminative Structural Graph Classiﬁcation (DSGC) (Seo et al., 2019).

Original: We report the baseline results taken from the original papers or published in (Ivanov & Burnaev, 2018; Verma & Zhang, 2018; Xinyi & Chen, 2019; Chen et al., 2019; Seo et al., 2019).
Revised: We report the baseline results taken from the original papers or published in (Ivanov & Burnaev, 2018; Verma & Zhang, 2018; Xinyi & Chen, 2019; Chen et al., 2019; Seo et al., 2019).

Original:  E XPERIMENTAL RESULTS
Revised:  E XPERIMENTAL RESULTS

Original: Table 2 presents the experimental results on the 9 benchmark datasets.
Revised: Table 2 presents the experimental results on the 9 benchmark datasets.

Original: We use “ Un-sup. ” to denote the unsupervised graph embedding models that can access all nodes from the whole dataset, but not  graph labels; and we use “ Sup. ” to denote the supervised models that can use the graph labels during training the models.
Revised: We use “ Unsupervised ” to denote the unsupervised graph embedding models that can access all nodes from the whole dataset, but not the graph labels; and we use “ Supervised ” to denote the supervised models that can use the graph labels of training graphs during training.

Original: In general, our unsupervised U2GNN achieves state-of-the-art performances on a range of benchmarks for the graph classiﬁcation task, hence this demonstrates a high impact of the unsupervised
Revised: In general, our unsupervised U2GNN achieves state-of-the-art performances on a range of benchmarks for the graph classiﬁcation task, hence this demonstrates a high impact of the unsupervised

Original: U2GNN in inferring  plausible node and graph embeddings.
Revised: U2GNN in inferring the plausible node and graph embeddings.

Original: It is to note that several existing works (Xinyi & Chen, 2019; Xu et al., 2019; Chen et al., 2019; Maron et al., 2019b; Seo et al., 2019) compared the supervised models with the unsupervised models together due to the use of the same 10-fold cross-validation scheme.
Revised: It is to note that several existing works (Xinyi & Chen, 2019; Xu et al., 2019; Chen et al., 2019; Maron et al., 2019b; Seo et al., 2019) compared the supervised models with the unsupervised models together due to the use of the same 10-fold cross-validation scheme.

Original: However, our unsupervised U2GNN signiﬁcantly outperforms the supervised methods in most of benchmark cases by a large margin of 18+%, thus we suggest that future GNN works should separate two training fashions.
Revised: However, our unsupervised U2GNN signiﬁcantly outperforms the supervised methods in most of benchmark cases by a large margin of 18+%, thus we suggest that future GNN works should separate two training fashions.

Original: [Table]
Revised: [Table]

Original: Regarding our supervised U2GNN, on the social network datasets
Revised: Regarding our supervised U2GNN, on the social network datasets

Original: [Table]
Revised: [Table]

Original: our model produces new stateof-the-art performances on IMDB-B and IMDB-M, especially U2GNN signiﬁcantly achieves 4+% absolute higher accuracies than all supervised baselines.
Revised: our model produces new stateof-the-art performances on IMDB-B and IMDB-M, especially U2GNN signiﬁcantly achieves 4+% absolute higher accuracies than all supervised baselines.

Original: In addition, U2GNN obtains promising scores on COLLAB, RDT-B and RDT-M5K.
Revised: In addition, U2GNN obtains promising scores on COLLAB, RDT-B and RDT-M5K.

Original: On the bioinformatics datasets, our U2GNN obtains new highest accuracies on DD, PROTEINS and PTC.
Revised: On the bioinformatics datasets, our U2GNN obtains new highest accuracies on DD, PROTEINS and PTC.

Original: Moreover, U2GNN produces a competitive accuracy compared with those of baseline models on MUTAG, for which there are no signiﬁcant differences between our supervised U2GNN and some supervised baselines (e.g., GFN, GIN and
Revised: Moreover, U2GNN produces a competitive accuracy compared with those of baseline models on MUTAG, for which there are no signiﬁcant differences between our supervised U2GNN and some supervised baselines (e.g., GFN, GIN and

Original: PPGN).
Revised: PPGN).

Original: Note that there are only 188 graphs in the MUTAG dataset, which explains the high variance in the results.
Revised: Note that there are only 188 graphs in the MUTAG dataset, which explains the high variance in the results.

Original: [Figure]
Revised: [Figure]

Original: other hyper-
Revised: other hyper-

Original: parameters.
Revised: parameters.

Original: Next we investigate the effects of hyper-parameters on the experimental datasets in the unsupervised fashion in Figure 3. 

Revised: Next we investigate the effects of hyper-parameters on the experimental datasets in the unsupervised fashion in Figure 3.

Original: In general, our unsupervised U2GNN could consistently obtain better results than those of baselines with any value of T and N , as long as the training process is stopped precisely for all datasets.
Revised: In general, our unsupervised U2GNN could consistently obtain better results than those of baselines with any value of T and N , as long as the training process is stopped precisely for all datasets.

Original: In particular, we ﬁnd that higher T helps on most of the datasets, and especially boosts the performance on bioinformatics data.
Revised: In particular, we ﬁnd that higher T helps on most of the datasets, and especially boosts the performance on bioinformatics data.

Original: A possible reason is that the bioinformatics datasets comprise sparse networks where the average number of neighbors per node is below 5 as shown in Table 1, hence we need to use more steps to learn graph properties.
Revised: A possible reason is that the bioinformatics datasets comprise sparse networks where the average number of neighbors per node is below 5 as shown in Table 1, hence we need to use more steps to learn graph properties.

Original: In addition, using small N generally produces higher performances on the bioinformatics datasets, while using higher values of N is more suitable for the social network datasets.
Revised: In addition, using small N generally produces higher performances on the bioinformatics datasets, while using higher values of N is more suitable for the social network datasets.

Original: Besides, the social network datasets are much denser than the bioinformatics datasets, thus this is reason why we should use more sampled neighbors on the social networks rather than the bioinformatics ones.
Revised: Besides, the social network datasets are much denser than the bioinformatics datasets, thus this is reason why we should use more sampled neighbors on the social networks rather than the bioinformatics ones.

Original: Note that the similar ﬁndings also occur in the supervised fashion.
Revised: Note that the similar ﬁndings also occur in the supervised fashion.

Original: To qualitatively demonstrate the effectiveness of capturing the local and global graph properties, we use t-SNE (Maaten & Hinton, 2008) to visualize the learned node and graph embeddings in the unsupervised fashion on the DD dataset where the node labels are available.
Revised: To qualitatively demonstrate the effectiveness of capturing the local and global graph properties, we use t-SNE (Maaten & Hinton, 2008) to visualize the learned node and graph embeddings in the unsupervised fashion on the DD dataset where the node labels are available.

Original: It can be seen from
Revised: It can be seen from

Original: Figure 4 that our unsupervised U2GNN can effectively capture the local structure wherein the nodes are clustered according to the node labels, and the global structure wherein the graph embeddings are well-separated from each other; verifying the plausibility of the learned node and graph embeddings.
Revised: Figure 4 that our unsupervised U2GNN can effectively capture the local structure wherein the nodes are clustered according to the node labels, and the global structure wherein the graph embeddings are well-separated from each other; verifying the plausibility of the learned node and graph embeddings.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this paper, we introduce a novel graph neural network model U2GNN for the graph classiﬁcation task.
Revised: In this paper, we introduce a novel graph neural network model U2GNN for the graph classiﬁcation task.

Original: Given an input graph, U2GNN applies a self-attention mechanism followed by a recurrent transition to learn the node embeddings and then sums all the learned node embeddings to obtain the embedding of the entire graph.
Revised: Given an input graph, U2GNN applies a self-attention mechanism followed by a recurrent transition to learn the node embeddings and then sums all the learned node embeddings to obtain the embedding of the entire graph.

Original: Our U2GNN achieves new highest accuracies on most of
Revised: Our U2GNN achieves new highest accuracies on most of

Original: [Figure]
Revised: [Figure]

Original: well-known benchmark datasets in both the supervised and unsupervised fashions, using the same 10-fold cross-validation scheme, against the up-to-date unsupervised and supervised baselines.
Revised: well-known benchmark datasets in both the supervised and unsupervised fashions, using the same 10-fold cross-validation scheme, against the up-to-date unsupervised and supervised baselines.

Original: In future works, we plan to investigate the effectiveness of U2GNN on other important tasks such as node classiﬁcation and link prediction.
Revised: In future works, we plan to investigate the effectiveness of U2GNN on other important tasks such as node classiﬁcation and link prediction.

Original: Our code is available at: https://anonymous-url/.
Revised: Our code is available at: https://anonymous-url/.

Original: A GNN VARIANTS
Revised: A GNN VARIANTS

Original:  GCN (Kipf & Welling, 2017)is a semi-supervised learning model using a variant of convolutional neural networks on the graph-structured data. More formally, each node v ∈ V has a feature vector x v ∈ R d1 , and GCN infers a new vector representation h v ∈ R d 2 for node v from its neighbors as follows:
Revised: Regarding GCN (Kipf & Welling, 2017) each node v ∈ V has a feature vector x v ∈ R d and GCN infers a new vector representation h v ∈ R d 2 for node v from its neighbors as follows:

Original: [Equation]
Revised: [Equation]

Original: where W ∈ R
Revised: where W ∈ R

Original: d 2 × d 1 and b ∈ R d 2 are a weight matrix and a bias respectively, g is a non-linear activation function ReLU (Glorot et al., 2011), and N v is the set of neighbors of node v .
Revised: d 2 × d 1 and b ∈ R d 2 are a weight matrix and a bias respectively, g is a non-linear activation function, and N v is the set of neighbors of node v .

Original: GCN can indirectly capture nodes at many hops away by using multiple GCN layers stacked on top of each other as follows:
Revised: GCN can indirectly capture nodes at many hops away by using multiple GCN layers stacked on top of each other as follows:

Original: [Equation]
Revised: [Equation]

Original: where k is the layer index, and h (0)
Revised: where k is the layer index, and h (0)

Original: u
Revised: u

Original: = x u .
Revised: = x u .

Original: The vector outputs of the last GCN layer are used as the ﬁnal vector representations of nodes.
Revised: The vector outputs of the last GCN layer are used as the ﬁnal vector representations of nodes.

Original: GCN can be jointly trained with a classiﬁcation classiﬁer by minimizing the cross-entropy loss function.
Revised: GCN can be jointly trained with a classiﬁcation classiﬁer by minimizing the cross-entropy loss function.

Original: GraphSAGE (Hamilton et al., 2017a) is an extension of GCN, in which the representation h ( k +1) v of node v after k -th GraphSAGE layer is produced as follows: 
Revised: GraphSAGE (Hamilton et al., 2017a) is an extension of GCN, in which the representation h ( k +1) v of node v after k -th GraphSAGE layer is produced as follows: [Equation]

Original: [Equation]
Revised: [Equation]

Original: where [ ⊕ ] denotes a vector concatenation operation, and h ( k +1)
Revised: where [ ⊕ ] denotes a vector concatenation operation, and h ( k +1)

Original: N v can be computed in several ways such as using the GCN operation as follows:
Revised: N v can be computed in several ways such as using the GCN operation as follows:

Revised: [Equation]

Original: [Equation]
Revised: [Equation]

Original: or applying an element-wise max-pooling operation as follows:
Revised: or applying an element-wise max-pooling operation as follows:

Original: [Equation]
Revised: [Equation]

Original: In addition, h ( k +1) N v can be computed by taking the mean (or sum) of all vectors in { h ( k ) u , ∀ u ∈ N v } .
Revised: In addition, h ( k +1) N v can be computed by taking the mean (or sum) of all vectors in { h ( k ) u , ∀ u ∈ N v } .

Original: N v is deﬁned as a ﬁxed-size, uniformly drawn from the set of all neighbor nodes of v , and uniformly sampled differently through each layer.
Revised: N v is deﬁned as a ﬁxed-size, uniformly drawn from the set of all neighbor nodes of v , and uniformly sampled differently through each layer.

Original: The vector outputs of the last GraphSAGE layer are used to infer node embeddings.
Revised: The vector outputs of the last GraphSAGE layer are used to infer node embeddings.

Original: GraphSAGE employs the random walk algorithm (Perozzi et al., 2014) to generate context nodes for each target node and then adapts the negative sampling technique (Mikolov et al., 2013) to learn the model parameters.
Revised: GraphSAGE employs the random walk algorithm (Perozzi et al., 2014) to generate context nodes for each target node and then adapts the negative sampling technique (Mikolov et al., 2013) to learn the model parameters.

Original: GAT (Veli ˇ ckovi ´ c et al., 2018) extends GCN in assigning importance weights to neighbors of a given node by exploring the standard attention technique (Bahdanau et al., 2015).
Revised: GAT (Veli ˇ ckovi ´ c et al., 2018) extends GCN in assigning importance weights to neighbors of a given node by exploring the standard attention technique (Bahdanau et al., 2015).

Original: GAT induces a new vector representation h v of node v from its neighbors as follows:
Revised: GAT induces a new vector representation h v of node v from its neighbors as follows:

Original: [Equation]
Revised: [Equation]

Original:  where τ v , u is an importance weight which is computed as follows:
Revised: 14) where τ v , u is an importance weight which is computed as follows:

Original: [Equation]

Original: GAT can employ multi-head attention (Vaswani et al., 2017) as follows:

Original: [Equation]
Revised: [Equation]

Original: where K is the number of independent attention heads to execute Equation

Original: [Figure]

Original: B E FFECTS OF HYPER - PARAMETERS
Revised: B E FFECTS OF HYPER - PARAMETERS

Original: [Figure]
Revised: [Figure]

Original: of steps ( T ) in the unsupervised fashion.

Original: Regarding each dataset, for all 10 folds, we vary the value of T while using the same ﬁxed values of other hyper-parameters.

Original: [Figure]
Revised: [Figure]

Original: Regarding each dataset, for all 10 folds, we vary the value of N while using the same ﬁxed values of other hyper-parameters.