Original: G RAPH N EIGHBORHOOD A TTENTIVE P OOLING
Revised: G RAPH N EIGHBORHOOD A TTENTIVE P OOLING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Network representation learning (NRL) is a powerful technique for learning lowdimensional vector representation of high-dimensional and sparse graphs.
Revised: Network representation learning (NRL) is a powerful technique for learning lowdimensional vector representation of high-dimensional and sparse graphs.

Original: Most studies explore the structure and meta data associated with the graph using random walks and employ a unsupervised or semi-supervised learning schemes.
Revised: Most studies explore the structure and metadata associated with the graph using random walks and employ an unsupervised or semi-supervised learning schemes.

Original: Learning in these methods is context-free, because only a single representation per node is learned.
Revised: Learning in these methods is context-free, because only a single representation per node is learned.

Original: Recently studies have argued on the sufﬁciency of a single representation and proposed a context-sensitive approach that proved to be highly effective in applications such as link prediction and ranking.
Revised: Recently studies have argued on the sufﬁciency of a single representation and proposed a context-sensitive approach that proved to be highly effective in applications such as link prediction and ranking.

Original: However, most of these methods rely on additional textual features that require
Revised: However, most of these methods rely on additional textual features that require

Original: RNNs or CNNs to capture high-level features or rely on a community detection algorithm to identifying multiple contexts of a node.
Revised: RNNs or CNNs to capture high-level features or rely on a community detection algorithm to identify multiple contexts of a node.

Original: In this study, without requiring additional features nor a community detection algorithm, we propose a novel context-sensitive algorithm called G AP that learns to attend on different part of a node’s neighborhood using attentive pooling networks.
Revised: In this study, without requiring additional features nor a community detection algorithm, we propose a novel context-sensitive algorithm called G AP that learns to attend on different parts of a node’s neighborhood using attentive pooling networks.

Original: We show the efﬁcacy of G AP using three real-world datasets on link prediction and node clustering tasks and compare it against 10 popular and state-of-the-art (SOTA) baselines.
Revised: We show the efﬁcacy of G AP using three real-world datasets on link prediction and node clustering tasks and compare it against 10 popular and state-ofthe-art (SOTA) baselines.

Original: G AP consistently outperforms them and achieves up to ≈ 9% and ≈ 20% gain over the best performing methods on link prediction and clustering tasks, respectively.
Revised: G AP consistently outperforms them and achieves up to ≈ 9% and ≈ 20% gain over the best performing methods on link prediction and clustering tasks, respectively.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: NRL is a powerful technique to learn representation of graphs.
Revised: NRL is a powerful technique to learn representation of a graph.

Original: Such a representation gracefully lends itself to a wide variety of network analysis tasks, such as link prediction, node clustering, node classiﬁcation, recommendation, and so forth.
Revised: Such a representation gracefully lends itself to a wide variety of network analysis tasks, such as link prediction, node clustering, node classiﬁcation, recommendation, and so forth.

Original: In most studies, the learning is done in a context-free fashion.
Revised: In most studies, the learning is done in a context-free fashion.

Original: That is, the representation of a node characterizes just a single aspect of the node, for instance, the local or global neighborhood of a node.
Revised: That is, the representation of a node characterizes just a single aspect of the node, for instance, the local or global neighborhood of a node.

Original: Recently, a complementary line of research has questioned the sufﬁciency of single representations and considered a context-sensitive approach.
Revised: Recently, a complementary line of research has questioned the sufﬁciency of single representations and considered a context-sensitive approach.

Original: Given a node, this approach projects it to different points in a space depending on other contexts its coupled with.
Revised: Given a node, this approach projects it to different points in a space depending on other contexts it is coupled with.

Original: A context node can be sampled from a neighborhood (Tu et al., 2017; Zhang et al., 2018), random walk, and so on.
Revised: A context node can be sampled from a neighborhood (Tu et al., 2017; Zhang et al., 2018), random walk (Ying et al., 2018), and so on.

Original: In this study we sample from a node neighborhood (nodes connected by an edge).
Revised: In this study we sample from a node neighborhood (nodes connected by an edge).

Original: Thus, in the learning process of our approach a source node’s representation changes depending on the target (context) node it is accompanied by.
Revised: Thus, in the learning process of our approach a source node’s representation changes depending on the target (context) node it is accompanied by.

Original:  A related notion (Peters et al., 2018; Devlin et al., 2018) in NLP has signiﬁcantly improved SOTA across several NLP tasks.NRL studies have shown a context-sensitive approach signiﬁcantly outperform previous context-free SOTA methods in link-prediction task.
Revised: Studies have shown that context-sensitive approaches signiﬁcantly outperform previous context-free SOTA methods in link-prediction task. A related notion (Peters et al., 2018; Devlin et al., 2018) in NLP has signiﬁcantly improved SOTA across several NLP tasks.

Original: In this paper we propose G AP ( G raph neighborhood a ttentive p ooling), which is inspired by attentive pooling networks ( APN ) (dos Santos et al., 2016), originally proposed for solving the problem of pair ranking in NLP.
Revised: In this paper we propose G AP ( G raph neighborhood a ttentive p ooling), which is inspired by attentive pooling networks ( APN ) (dos Santos et al., 2016), originally proposed for solving the problem of pair ranking in NLP.

Original: For instance, given a question q , and a set of answers A
Revised: For instance, given a question q , and a set of answers A

Original: = { a 1 , . .
Revised: = { a 1 , . .

Original: . , a z } , an
Revised: . , a z } , an

Original: APN can be trained to rank the answers in A with respect to q by using a two-way attention mechanism.
Revised: APN can be trained to rank the answers in A with respect to q by using a two-way attention mechanism.

Original: APN is based on the prevalent deep learning formula for SOTA NLP, that is, embed , encode , attend , predict (Honnibal, 2018).
Revised: APN is based on the prevalent deep learning formula for SOTA NLP, that is, embed , encode , attend , predict (Honnibal, 2018).

Original: Given a question-answer pair ( q, a i ) , the APN model ﬁrst projects the embedding of the pairs using two separate encoders, and the encoder can be a CNN or LSTM .
Revised: Given a question-answer pair ( q, a i ) , the APN model ﬁrst projects the embedding of the pairs using two separate encoders, and the encoder can be a CNN or LSTM .

Original: The projection helps to capture n-gram context information and/or long-term dependencies in the input sequence.
Revised: The projection helps to capture n-gram context information and/or long-term dependencies in the input sequence.

Original: Next, a soft-alignment matrix is obtained by a mutual-attention mechanism that transforms these projections using a parameter matrix.
Revised: Next, a soft-alignment matrix is obtained by a mutual-attention mechanism that transforms these projections using a parameter matrix.

Original: Attention vectors are then computed through a column-wise and row-wise pooling operations on the alignment matrix.
Revised: Attention vectors are then computed through a column-wise and row-wise pooling operations on the alignment matrix.

Original: Finally, the weighted sum of each of the above projections by its respective attention vector is computed to obtain the representations of the question and answer.
Revised: Finally, the weighted sum of each of the above projections by its respective attention vector is computed to obtain the representations of the question and answer.

Original: Each candidate answer a i is then ranked according to its similarity with the question q computed using the representations of q and a i .
Revised: Each candidate answer a i is then ranked according to its similarity with the question q computed using the representations of q and a i .

Original: Recently, APN have been applied to context-sensitive NRL by Tu et al.
Revised: Recently, APN have been applied to context-sensitive NRL by Tu et al.

Original: (2017), and the inputs are textual information attached with a pair of incident nodes of edges in a graph.
Revised: (2017), and the inputs are textual information attached with a pair of incident nodes of edges in a graph.

Original: Such information, however, has the added overhead of encoding textual information.
Revised: Such information, however, has the added overhead of encoding textual information.

Original: Though we adopt APN in G AP , we capitalize on the graph neighborhood of nodes to avoid the need for textual documents without compromising the quality of the learned representations.
Revised: Though we adopt APN in G AP , we capitalize on the graph neighborhood of nodes to avoid the need for textual documents without compromising the quality of the learned representations.

Original: Our hypothesis is that one can learn high-quality context-sensitive node representations just by mutually attending to the graph neighborhood of a node and its context node.
Revised: Our hypothesis is that one can learn high-quality context-sensitive node representations just by mutually attending to the graph neighborhood of a node and its context node.

Original: To achieve this, we naturally assume that the order of nodes in the graph neighborhood of a node is arbitrary.
Revised: To achieve this, we naturally assume that the order of nodes in the graph neighborhood of a node is arbitrary.

Original: Moreover, we exploit this assumption to simplify the APN model by removing the expensive encode phase.
Revised: Moreover, we exploit this assumption to simplify the APN model by removing the expensive encode phase.

Original: Akin to textual features in APN , G AP simply uses graph neighborhood of nodes.
Revised: Akin to textual features in APN , G AP simply uses graph neighborhood of nodes.

Original: That is, for every node in the graph we deﬁne a graph neighborhood function to build a ﬁxed size neighborhood sequence, which speciﬁes the input of G AP .
Revised: That is, for every node in the graph we deﬁne a graph neighborhood function to build a ﬁxed size neighborhood sequence, which speciﬁes the input of G AP .

Original: In the APN model, the encoder phase is usually required to capture high-level features such as n-grams, and long term and order dependencies in textual inputs.
Revised: In the APN model, the encoder phase is usually required to capture high-level features such as n-grams, and long term and order dependencies in textual inputs.

Original: As we have no textual features and due to our assumption that there is no ordering in the graph neighborhood of nodes we can effectively strip off the encoder.
Revised: As we have no textual features and due to our assumption that there is no ordering in the graph neighborhood of nodes, we can effectively strip off the encoder.

Original: The encoder is the expensive part of APN as it involves a RNN or CNN , and hence G AP can be trained faster than APN .
Revised: The encoder is the expensive part of APN as it involves a RNN or CNN , and hence G AP can be trained faster than APN .

Original: This simple yet empirically fruitful modiﬁcation of the APN model enables G AP to achieve SOTA performance on link prediction and node clustering tasks using three real world datasets.
Revised: This simple yet empirically fruitful modiﬁcation of the APN model enables G AP to achieve SOTA performance on link prediction and node clustering tasks using three real world datasets.

Original: Furthermore, we have empirically shown that G AP is more than 2 times faster than an APN like NRL algorithm based on text input.
Revised: Furthermore, we have empirically shown that G AP is more than 2 times faster than an APN like NRL algorithm based on text input.

Original: In addition, the simpliﬁcation in G AP does not introduce new hyperparameters other than the usual ones, such as the learning rate and sequence length in APN .
Revised: In addition, the simpliﬁcation in G AP does not introduce new hyperparameters other than the usual ones, such as the learning rate and sequence length in APN .

Original:  APN A RCHITECTURE
Revised:  APN A RCHITECTURE

Original: For the sake of being self-contained, here we brieﬂy describe the original APN architecture.
Revised: For the sake of being self-contained, here we brieﬂy describe the original APN architecture.

Original: We are given a pair of natural language texts ( x, y ) as input, where x = ( w 1 , . . .
Revised: We are given a pair of natural language texts ( x, y ) as input, where x = ( w 1 , . . .

Original: w X ) and
Revised: w X ) and

Original: y =
Revised: y =

Original: ( w 1 , . . .
Revised: ( w 1 , . . .

Original: , w Y ) are a sequence of words of variable lengths, and each word w i is a sample from a vocabulary V ,
Revised: , w Y ) are a sequence of words of variable lengths, and each word w i is a sample from a vocabulary V ,

Original: X = | x | , Y = | y | , and X and Y could be different.
Revised: X = | x | , Y = | y | , and X and Y could be different.

Original: The APN ’s forward execution is shown in
Revised: The APN ’s forward execution is shown in

Original: Fig.
Revised: Fig.

Original: 1(A)
Revised: 1(A)

Original: and in the following we describe each component.
Revised: and in the following we describe each component.

Original: Embed: First embedding matrices of x and y are constructed through a lookup operation on an embedding matrix E ∈ R d ×| V | of words, where d is the embedding dimension.
Revised: Embed: First embedding matrices of x and y are constructed through a lookup operation on an embedding matrix E ∈ R d ×| V | of words, where d is the embedding dimension.

Original: That is, for both x and y , respectively, embedding matrices X =
Revised: That is, for both x and y , respectively, embedding matrices X =

Original: [ w 1 , . . . , w X ] and Y =
Revised: [ w 1 , . . . , w X ] and Y =

Original: [ w 1 , . . .
Revised: [ w 1 , . . .

Original: , w Y ] are constructed by concatenating embeddings w i of each word w i in x and y , the Embed box in Fig.
Revised: , w Y ] are constructed by concatenating embeddings w i of each word w i in x and y , the Embed box in Fig.

Original: 1(A).
Revised: 1(A).

Original: Encode: Each embedding matrix is then projected using a CNN or BI -LSTM encoder to capture inherent high-level features, the Encode box in Fig.
Revised: Encode: Each embedding matrix is then projected using a CNN or BI -LSTM encoder to capture inherent high-level features, the Encode box in Fig.

Original: 1(A).
Revised: 1(A).

Original: More formally, the embedded texts X ∈ R d × X , Y ∈ R d × Y are projected as X enc = f
Revised: More formally, the embedded texts X ∈ R d × X , Y ∈ R d × Y are projected as X enc = f

Original: ( X , Θ) and Y enc =
Revised: ( X , Θ) and Y enc =

Original: f ( Y , Θ) where f is the encoder,
Revised: f ( Y , Θ) where f is the encoder,

Original: CNN or BI -LSTM, Θ is the set of parameters of the encoder, and X enc ∈ R c × X and Y enc ∈ R c
Revised: CNN or BI -LSTM, Θ is the set of parameters of the encoder, and X enc ∈ R c × X and Y enc ∈ R c

Original: × Y , where c is the number of ﬁlters or hidden features of the CNN and BI -LSTM, respectively.
Revised: × Y , where c is the number of ﬁlters or hidden features of the CNN and BI -LSTM, respectively.

Original: Attend:
Revised: Attend:

Original: In the third step, a parameter matrix P ∈ R c × c is introduced so as to learn a similarity or soft-alignment matrix A ∈ R X × Y between the sequence projections X enc and Y enc as:
Revised: In the third step, a parameter matrix P ∈ R c × c is introduced so as to learn a similarity or soft-alignment matrix A ∈ R X × Y between the sequence projections X enc and Y enc as:

Original: [Equation]
Revised: [Equation]

Original: Then unnormalized attention weight vectors x (cid:48) ∈ R X and y (cid:

Revised: Then unnormalized attention weight vectors x (cid:48) ∈ R X and y (cid:

Original: ) ∈ R Y are obtained through a column-wise and row-wise max-pooling operations on A , respectively as x (cid:48)
Revised: ) ∈ R Y are obtained through a column-wise and row-wise max-pooling operations on A , respectively as x (cid:48)

Original: i = max
Revised: i = max

Original: ( A i, : ) , y (cid:48)
Revised: ( A i, : ) , y (cid:48)

Original: j = max( A : ,j ) , where 0 ≤ i < X , 0 ≤ j < Y and A
Revised: j = max( A : ,j ) , where 0 ≤ i < X , 0 ≤ j < Y and A

Original: i, : and A : ,j are the i -th and j -th row and column of A , respectively.
Revised: i, : and A : ,j are the i -th and j -th row and column of A , respectively.

Original: Next, the attention vectors are normalized using softmax, x = softmax ( x (cid:48) )and y = softmax ( y (cid:48) ) . Finally, the normalized attention vectors are used to compute the ﬁnal representations as r x = X enc · x T and r y = Y enc · y T . [Figure]
Revised: Next, the attention vectors are normalized using softmax, x = softmax ( x (cid:48) )

Original: [Figure]

Original:  ]

Original: [Figure]

Original: =

Original:  (  ,Θ)

Original: [Figure]
Revised: [Figure]

Original: 
Revised:

Original: [Figure]

Original:  (  ,Θ)

Original: [Figure]

Original: Embed  = [ : ∈  ]       =

Original: [Figure]

Original:  = =

Original:   =   =

Original: =

Original: Embed

Original: [Figure]

Original: [ : ∈ ]

Original: Embed  = [ : ∈ ]         =

Original: [Figure]
Revised: [Figure]

Original: 
Revised:

Original: [Figure]

Original:  =

Original: [Figure]
Revised: [Figure]

Original: 
Revised:

Original: [Figure]

Original: [Figure]

Original: Attend  = tanh (  )

Original: [Figure]
Revised: [Figure]

Original: 
Revised:

Original: [Figure]
Revised: [Figure]

Original:   
Revised: =   =

Original: [Figure]
Revised: [Figure]

Original: Row

Original: [Figure]  ′ [Figure]  ′ = 
Revised: and y = softmax ( y (cid:48) ) .

Original:   =  ⋅ [Figure] (  ′)  = ⋅    (  ′)
Revised: Finally, the normalized attention vectors are used to compute the ﬁnal representations as r x = X enc · x T and r y =

Original:  =   =  [Figure]
Revised: Y enc · y T .

Original: Predict:
Revised: Predict:

Original: In the last step, the representations r x and r y will be used for ranking depending on the task on hand.
Revised: In the last step, the representations r x and r y will be used for ranking depending on the task on hand.

Original: For instance, in a question and answer setting, each candidate answer’s representation r y will be ranked based on its similarity score with the question’s representation r x .
Revised: For instance, in a question and answer setting, each candidate answer’s representation r y will be ranked based on its similarity score with the question’s representation r x .

Original:  G AP
Revised:  G AP

Original: G AP adopts the APN model for learning the representations of node pairs in a graph G =
Revised: G AP adopts the APN model for learning the representations of node pairs in a graph G =

Original: ( V, E ) with a set of
Revised: ( V, E ) with a set of

Original: n nodes V and m edges E .
Revised: n nodes V and m edges E .

Original: G can be a directed or undirected and weighted or unweighted graph.
Revised: G can be a directed or undirected and weighted or unweighted graph.

Original: Without loss of generality we assume that G is an unweighted directed graph.
Revised: Without loss of generality we assume that G is an unweighted directed graph.

Original: We deﬁne a neighborhood function N : V → 2 V , which maps each node u ∈ V to a set of nodes
Revised: We deﬁne a neighborhood function N : V → 2 V , which maps each node u ∈ V to a set of nodes

Original: N u ⊆ V .
Revised: N u ⊆ V .

Original: A simple way of materializing N u is to consider the ﬁrst-order neighbors of u , that is, N u
Revised: A simple way of materializing N u is to consider the ﬁrst-order neighbors of u , that is, N u

Original: = [ v : ( u, v ) ∈
Revised: = [ v : ( u, v ) ∈

Original: E ∨ ( v, u ) ∈ E ] .
Revised: E ∨ ( v, u ) ∈ E ] .

Original: An important assumption that we have on N u is that the ordering of the nodes in N u is not important.
Revised: An important assumption that we have on N u is that the ordering of the nodes in N u is not important.

Original: G AP capitalizes on this assumption to simplify the APN model and achieve SOTA performance.
Revised: G AP capitalizes on this assumption to simplify the APN model and achieve SOTA performance.

Original: Even though one can explorea more sophisticated neighborhood functions, in this study we simply consider the ﬁrst order neighborhood.
Revised: Even though one can explore more sophisticated neighborhood functions, in this study we simply consider the ﬁrst order neighborhood.

Original: Our goal is to learn node representations using the simpliﬁed APN based on the node neighborhood function N .
Revised: Our goal is to learn node representations using the simpliﬁed APN based on the node neighborhood function N .

Original: Hence, akin to the input text pairs in APN , we consider a pair of neighborhood sequences
Revised: Hence, akin to the input text pairs in APN , we consider a pair of neighborhood sequences

Original: N s =
Revised: N s =

Original: ( u 1 , . . .
Revised: ( u 1 , . . .

Original: , u S ) and N t
Revised: , u S ) and N t

Original: = ( v 1 , . .
Revised: = ( v 1 , . .

Original: . , v T )
Revised: . , v T )

Original: associated with a pair ( s, t )
Revised: associated with a pair ( s, t )

Original: ∈ E of nodes s and t , and
Revised: ∈ E of nodes s and t , and

Original: S = | N s | and T = | N t | .
Revised: S = | N s | and T = | N t | .

Original: Without loss of generality we consider
Revised: Without loss of generality we consider

Original: S = T .
Revised: S = T .

Original: Recall that we assume the order of nodes in N s and N t is arbitrary.
Revised: Recall that we assume the order of nodes in N s and N t is arbitrary.

Original: Dataset # Nodes # Edges
Revised: Dataset # Nodes # Edges

Original: Features
Revised: Features

Original: Cora
Revised: Cora

Original: Paper Abstract
Revised: Paper Abstract

Original: Zhihu
Revised: Zhihu

Original: User post
Revised: User post

Original: Email
Revised: Email

Original: NA
Revised: NA

Original: G AP ’s forward execution model is shown in Fig 1(B), and learning starts by embedding N s and
Revised: G AP ’s forward execution model is shown in Fig 1(B), and learning starts by embedding N s and

Original: N t , respectively, as S
Revised: N t , respectively, as S

Original: =
Revised: =

Original: ( u 1 , . . .
Revised: ( u 1 , . . .

Original: , u S ) and T = ( v 1 , . .
Revised: , u S ) and T = ( v 1 , . .

Original: . , v T ) .
Revised: . , v T ) .

Original: Since there is no order dependency between the nodes in N s or N t , besides being a neighbor of the respective node, we leave out the
Revised: Since there is no order dependency between the nodes in N s or N t , besides being a neighbor of the respective node, we leave out the

Original: CNN or BI -LSTM based projections of S and T that could capture the dependencies.
Revised: CNN or BI -LSTM based projections of S and T that could capture the dependencies.

Original: No encoder!
Revised: No encoder!

Original: Thus, the next step of G AP is mutually attending on the embeddings, S and T , of the graph neighborhood of the node pairs; the Attend box of 1(B).
Revised: Thus, the next step of G AP is mutually attending on the embeddings, S and T , of the graph neighborhood of the node pairs; the Attend box of 1(B).

Original: That is, we employ the trainable parameter matrix
Revised: That is, we employ the trainable parameter matrix

Original: P ∈ R d × d and compute the soft-alignment matrix, A , between the neighbors of s and t .
Revised: P ∈ R d × d and compute the soft-alignment matrix, A , between the neighbors of s and t .

Original: [Equation]
Revised: [Equation]

Original: Here A is a soft-alignment matrix between every pair of nodes, ( u, v ) ∈ N s × N t .
Revised: Here A is a soft-alignment matrix between every pair of nodes, ( u, v ) ∈ N s × N t .

Original: Therefore, for each axis of A , we proceed by pooling the maximum alignment score for each node to obtain the unnormalized attention vectors s (cid:48)
Revised: Therefore, for each axis of A , we proceed by pooling the maximum alignment score for each node to obtain the unnormalized attention vectors s (cid:48)

Original: i = max
Revised: i = max

Original: ( A i, : ) and t (cid:48)
Revised: ( A i, : ) and t (cid:48)

Original: j = max( A : ,j ) .
Revised: j = max( A : ,j ) .

Original: As a result of the pooling operations, each neighbor u ∈ N s of the source node, s , selects a neighbor v ∈ N t of the target node, t , with the maximum alginment or similarity score.
Revised: As a result of the pooling operations, each neighbor u ∈ N s of the source node, s , selects a neighbor v ∈ N t of the target node, t , with the maximum alignment or similarity score.

Original: A similar selection is done for v ∈ N t .
Revised: A similar selection is done for v ∈ N t .

Original: This enables the source and target neighborhood sequences of the pair to inﬂuence each other in order to learn a context-sensitive representation of s and t .
Revised: This enables the source and target neighborhood sequences of the pair to inﬂuence each other in order to learn a context-sensitive representation of s and t .

Original: The normalized attention vectors are then obtained by applying softmax as s = softmax ( s (cid:48) ) and t = softmax ( t (cid:48) ) .
Revised: The normalized attention vectors are then obtained by applying softmax as s = softmax ( s (cid:48) ) and t = softmax ( t (cid:48) ) .

Original: Ultimately, we compute the context-sensitive representations r s and r t of the source and target nodes s and t , respectively as r s = S · s T and r t = T · t T .
Revised: Ultimately, we compute the context-sensitive representations r s and r t of the source and target nodes s and t , respectively as r s = S · s T and r t = T · t T .

Original: Optimization: The objective of G AP is to maximize the likelihood of the graph (edges) by predicting edge similarities using the dot product of the source and target representations as r s · r t ;
Revised: Optimization: The objective of G AP is to maximize the likelihood of the graph (edges) by predicting edge similarities using the dot product of the source and target representations as r s · r t ;

Original: the
Revised: the

Original: Predict box of Fig 1(B).
Revised: Predict box of Fig 1(B).

Original: Hence we employ a hard-margin loss given in Eq. 2.
Revised: Hence, we employ a hard-margin loss given in Eq. 2.

Original: [Equation]
Revised: [Equation]

Original: where r − t is the representation of a negative target node t − , that is ( s, t − ) / ∈ E .
Revised: where r − t is the representation of a negative target node t − , that is ( s, t − ) / ∈ E .

Original: The goal is to learn, in an unsupervised fashion, a context-sensitive embedding of nodes that enable us to rank the positive edges ( s,
Revised: The goal is to learn, in an unsupervised fashion, a context-sensitive embedding of nodes that enable us to rank the positive edges ( s,

Original: t ) ∈
Revised: t ) ∈

Original: E higher than the negative pairs ( s, t − ) .
Revised: E higher than the negative pairs ( s, t − ) .

Original: Finally a word on the computational complexity of G AP that is proportional to the the number of edges, as we are considering each edge as an input pair.
Revised: Finally a word on the computational complexity of G AP that is proportional to the the number of edges, as we are considering each edge as an input pair.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: In this section we provide an empirical evaluation of G AP .
Revised: In this section we provide an empirical evaluation of G AP .

Original: To this end, experiments are carried out using the following datasets, and a basic summary is given in Table 1.
Revised: To this end, experiments are carried out using the following datasets, and a basic summary is given in Table 1.

Original: Cora Tu et al.
Revised: Cora Tu et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018): is a citation network dataset, where a node represents a paper and an edge ( u, v ) ∈
Revised: (2018): is a citation network dataset, where a node represents a paper and an edge ( u, v ) ∈

Original: E represents that paper u has cited paper v . 2.
Revised: E represents that paper u has cited paper v . 2.

Original: Zhihu Tu et al.
Revised: Zhihu Tu et al.

Original: ; Zhang et al.
Revised: ; Zhang et al.

Original: (2018): is the biggest social network for Q&A and it is based in China.
Revised: (2018): is the biggest social network for Q&A and it is based in China.

Original: Nodes are the users and the edges are follower relations between the users.
Revised: Nodes are the users and the edges are follower relations between the users.

Original: Email Leskovec et al.
Revised: Email Leskovec et al.

Original: (2007): is an email communication network between the largest European 
Revised: (2007): is an email communication network between the largest European research

Original: research institutes.
Revised: institutes.

Original: A node represents a person and an edge ( u
Revised: A node represents a person and an edge ( u

Original: , v ) ∈ G denotes that person u has sent an email to v .
Revised: , v ) ∈ G denotes that person u has sent an email to v .

Original: [Table]
Revised: [Table]

Original: Cora
Revised: Cora

Original: [Table]
Revised: [Table]

Original: Zhihu
Revised: Zhihu

Original: [Table]
Revised: [Table]

Original: Email
Revised: Email

Original: [Table]
Revised: [Table]

Original: The ﬁrst two datasets have features (documents) associated to nodes.
Revised: The ﬁrst two datasets have features (documents) associated to nodes.

Original: For Cora, abstract of papers and Zhihu user posts.
Revised: For Cora, abstract of papers and Zhihu user posts.

Original: Some of the baselines, discussed bellow, they require textual information, and hence they consume the aforementioned features.
Revised: Some of the baselines, discussed beneath, require textual information, and hence they consume the aforementioned features.

Original: The Email dataset has ground-truth community assignment for nodes based on a persons afﬁliation to one of the 42 departments.
Revised: The Email dataset has ground-truth community assignment for nodes based on a person’s afﬁliation to one of the 42 departments.

Original: We compare our method against the following 10 popular and SOTA baselines grouped as:
Revised: We compare our method against the following 10 popular and SOTA baselines grouped as:

Original: • Structure based methods : D EEP W ALK Perozzi et al.
Revised: • Structure based methods : D EEP W ALK Perozzi et al.

Original: (2014), N ODE 2V EC Grover & Leskovec (2016),
Revised: (2014), N ODE 2V EC Grover & Leskovec (2016),

Original: W ALK L ETS Perozzi et al.
Revised: W ALK L ETS Perozzi et al.

Original: (2016), A TTENTIVE W ALK Abu-El-Haija et al.
Revised: (2016), A TTENTIVE W ALK Abu-El-Haija et al.

Original: L INE Tang et al.
Revised: L INE Tang et al.

Original: (2015): • Structure and attribute based methods : T RI D NR Pan et al.
Revised: (2015): • Structure and attribute based methods : T RI D NR Pan et al.

Original: , TADW Yang et al.
Revised: , TADW Yang et al.

Original: (2015), CENE Sun et al.
Revised: (2015), CENE Sun et al.

Original: (2016) • Context-sensitive: CANE Tu et al.
Revised: (2016) • Context-sensitive: CANE Tu et al.

Original: DMTE Zhang et al.
Revised: DMTE Zhang et al.

Original: We are aware of another SOTA context-sensitive algorithm (Epasto & Perozzi, 2019) that do not require textual information, however we have not included it in our experiments because the source code is not available.
Revised: We are aware of another SOTA context-sensitive algorithm (Epasto & Perozzi, 2019) that do not require textual information, however we have not included it in our experiments because the source code is not available.

Original: 1 Now we report the experimental results carried out in two tasks, which are link prediction and node clustering.
Revised: 1 Now we report the experimental results carried out in two tasks, which are link prediction and node clustering.

Original: All experiments are performed using a 24-Core CPU and 125GB
Revised: All experiments are performed using a 24-Core CPU and 125GB

Original: RAM Ubuntu 18.04 machine.
Revised: RAM Ubuntu 18.04 machine.

Original: 4. L INK P REDICTION
Revised: 4. L INK P REDICTION

Original: Link prediction is an important task that graph embedding algorithms are applied to.
Revised: Link prediction is an important task that graph embedding algorithms are applied to.

Original: Particularly context-sensitive embedding techniques have proved to be well suited for this task.
Revised: Particularly context-sensitive embedding techniques have proved to be well suited for this task.

Original: Similar to existing studies we perform this experiment using a fraction of the edges as a training set.
Revised: Similar to existing studies we perform this experiment using a fraction of the edges as a training set.

Original: We hold out the remaining fraction of the edges from the training phase and we will only reveal them during the test phase, results are reported using this set.
Revised: We hold out the remaining fraction of the edges from the training phase and we will only reveal them during the test phase, results are reported using this set.

Original: All hyper-parameter tuning is performed by taking a small fraction of the training set as a validation set.
Revised: All hyper-parameter tuning is performed by taking a small fraction of the training set as a validation set.

Original: Setup: In-line with existingexisting techniques (Tu et al., 2017; Zhang et al., 2018), the percentage of training edges ranges from 15% to 95% by a step of 10.
Revised: Setup: In-line with existing techniques (Tu et al., 2017; Zhang et al., 2018), the percentage of training edges ranges from 15% to 95% by a step of 10.

Original: The hyper-parameters of all algorithms are tuned using random-search.
Revised: The hyper-parameters of all algorithms are tuned using random-search.

Original: For some of the baselines, our results are consistent with what is reported in previous studies, and hence for Cora and Zhihu we simply report these results.
Revised: For some of the baselines, our results are consistent with what is reported in previous studies, and hence for Cora and Zhihu we simply report these results.

Original: Except the “unavoidable” hyper-parameters (Eg. learning rate, regularization/dropout rate) that are common in allof the algorithms, our model has just one hyper-parameter which is the neighborhood sequence length (#Neighborhood = S = T ), for nodes with smaller neighbohood size we use zero padding.
Revised: Except the “unavoidable” hyper-parameters (eg. learning rate, regularization/dropout rate) that are common in all the algorithms, our model has just one hyper-parameter which is the neighborhood sequence length (#Neighborhood = S = T ), for nodes with smaller neighborhood size we use zero padding.

Original: As we shall verify later, G AP is not signiﬁcantly affected by the choice of this parameter.
Revised: As we shall verify later, G AP is not signiﬁcantly affected by the choice of this parameter.

Original: The quality of the prediction task is measured using the AUC score.
Revised: The quality of the prediction task is measured using the AUC score.

Original: AUC indicates the probability that a randomly selected pair ( u, w ) / ∈ E will have a higher similarity score than an edge ( u, v ) ∈ E .
Revised: AUC indicates the probability that a randomly selected pair ( u, w ) / ∈ E will have a higher similarity score than an edge ( u, v ) ∈ E .

Original: Similarity between a pair of nodes is computed as the dot product of their representation.
Revised: Similarity between a pair of nodes is computed as the dot product of their representation.

Original: For all the algorithms the representation size – d is 200 and G AP ’s conﬁguration is shown in Table 2.
Revised: For all the algorithms the representation size – d is 200 and G AP ’s conﬁguration is shown in Table 2.

Original: Results: The results of the empirical evaluations on the Cora, Zhihu, and Email datasets are reporeted in Tables 3, 4, and 5.
Revised: Results: The results of the empirical evaluations on the Cora, Zhihu, and Email datasets are reported in Tables 3, 4, and 5.

Original: G AP outperforms the SOTA baselines in all cases for Zhihu and Email, and in almost all cases for Cora.
Revised: G AP outperforms the SOTA baselines in all cases for Zhihu and Email, and in almost all cases for Cora.

Original: One can see that as we increase the percentage of training edges, performance increases for all the algorithms.
Revised: One can see that as we increase the percentage of training edges, performance increases for all the algorithms.

Original: As indicated by the “Gain” row, G AP achieves upto 9% improvement over SOTA context-sensitive techniques.
Revised: As indicated by the “Gain” row, G AP achieves up to 9% improvement over SOTA context-sensitive techniques.

Original: Notably the gain is pronounced forsmaller
Revised: Notably the gain is pronounced for

Original: [Table]
Revised: [Table]

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  values of percentage of edges used for training.
Revised: smaller values of percentage of edges used for training.

Original: This is shows that G AP is suitable both in cases where there are several missing links and most of the links are present.[Table]
Revised: This is shows that G AP is suitable both in cases where there are several missing links and most of the links are present.

Original:  ODE 
Revised: 4. N ODE C LUSTERING

Original: [Table] LUSTERING

Original: Nodes in a network has the tendency to form cohesive structures based on some kinds of shared aspects.
Revised: Nodes in a network has the tendency to form cohesive structures based on some kinds of shared aspects.

Original: These structures are usually referred to as groups, clusters or communities and identifying them is an important task in network analysis.
Revised: These structures are usually referred to as groups, clusters or communities and identifying them is an important task in network analysis.

Original: In this section we use the Email dataset that has ground truth communities, and there are 42 of them.
Revised: In this section we use the Email dataset that has ground truth communities, and there are 42 of them.

Original: Since this dataset has only structural information, we have excluded the baselines that require textual information.
Revised: Since this dataset has only structural information, we have excluded the baselines that require textual information.

Original: Algorithm %of training edges 25% 55% 75% 95%
Revised: Algorithm %of training edges 25% 55% 75% 95%

Original: NMI
Revised: NMI

Original: AMI
Revised: AMI

Original: NMI
Revised: NMI

Original: AMI
Revised: AMI

Original: NMI
Revised: NMI

Original: AMI
Revised: AMI

Original: NMI
Revised: NMI

Original: AMI
Revised: AMI

Original: D
Revised: D

Original: [Table]
Revised: [Table]

Original: W ALK 41.
Revised: W ALK 41.

Original: L INE 44.
Revised: L INE 44.

Original: N ODE 2V EC 46.
Revised: N ODE 2V EC 46.

Original: W ALK L ETS 47.
Revised: W ALK L ETS 47.

Original: [Figure]
Revised: [Figure]

Original: TTENTIVE W ALK 42.
Revised: TTENTIVE W ALK 42.

Original: G AP 67.
Revised: G AP 67.

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: email zhihu
Revised: email zhihu

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure] [Figure]
Revised: [Figure] [Figure]

Original: [Figure]
Revised: [Figure]

Original: Setup: Since each node belongs to exactly one cluster, we employ the Spectral Clustering algorithm to identify clusters.
Revised: Setup: Since each node belongs to exactly one cluster, we employ the Spectral Clustering algorithm to identify clusters.

Original: The learned representations of nodes by a certain algorithm are the input features of the clustering algorithm.
Revised: The learned representations of nodes by a certain algorithm are the input features of the clustering algorithm.

Original: In this experiment the percentage of training edges varies from 25% to 95% by a step of 20%, for the rest we use the same conﬁguration as in the above experiment.
Revised: In this experiment the percentage of training edges varies from 25% to 95% by a step of 20%, for the rest we use the same conﬁguration as in the above experiment.

Original: Given the ground truth community assignment y of nodes and the predicted community assignments ˆ y , usually the agreement between y and ˆ y are measured using mutual information
Revised: Given the ground truth community assignment y of nodes and the predicted community assignments ˆ y , usually the agreement between y and ˆ y are measured using mutual information

Original: I ( y, ˆ y ) .
Revised: I ( y, ˆ y ) .

Original: However,
Revised: However,

Original: I is not bounded and difﬁcult for comparing methods, hence we use two other variant of I (Vinh et al., 2010).
Revised: I is not bounded and difﬁcult for comparing methods, hence we use two other variants of I (Vinh et al., 2010).

Original: Which are, the normalized mutual information NMI ( y, ˆ y ) , which simply normalizes
Revised: Which are, the normalized mutual information NMI ( y, ˆ y ) , which simply normalizes

Original: I and adjusted mutual information AMI ( y, ˆ y ) , which adjusts or normalizes I to random chances.
Revised: I and adjusted mutual information AMI ( y, ˆ y ) , which adjusts or normalizes I to random chances.

Original: Results: The results of this experiment are reported in Table 6, and G AP signiﬁcantly outperforms all the baselines by upto 20% with respect to AMI score.
Revised: Results: The results of this experiment are reported in Table 6, and G AP signiﬁcantly outperforms all the baselines by up to 20% with respect to AMI score.

Original: Consistent to our previous experiment
Revised: Consistent to our previous experiment

Original: G AP performs well in both extremes for the value of the percentage of the training edges.
Revised: G AP performs well in both extremes for the value of the percentage of the training edges.

Original: Similar improvements are achieved for AMI score.
Revised: Similar improvements are achieved for AMI score.

Original: 4. P ARAMETER S ENSITIVITY A NALYSIS
Revised: 4. P ARAMETER S ENSITIVITY A NALYSIS

Original: Here we ﬁrst show the main hyper-parameter of G AP , which is the size of the neighborhood, #Neighborhood = S = T .
Revised: Here we ﬁrst show the main hyper-parameter of G AP , which is the size of the neighborhood, #Neighborhood = S = T .

Original: Figures 2 and 3(A) show the effect of this paramter both for link prediction and node clustering tasks.
Revised: Figures 2 and 3(A) show the effect of this parameter both for link prediction and node clustering tasks.

Original: In both cases we notice that G AP is not signiﬁcantly affected by the change of values.
Revised: In both cases we notice that G AP is not signiﬁcantly affected by the change of values.

Original: We show the effect across different values of percentage and ﬁxed (55%) of training edges for link prediction and node clustering tasks, respectively.
Revised: We show the effect across different values of percentage and ﬁxed (55%) of training edges for link prediction and node clustering tasks, respectively.

Original: Regardless of the percentage of training edges, we only see a small change of AUC (Fig 2), and NMI and AMI (Fig 3-a) across different values of #Neighborhood.
Revised: Regardless of the percentage of training edges, we only see a small change of AUC (Fig 2), and NMI and AMI (Fig 3-a) across different values of #Neighborhood.

Original: Next we analyze the run time of training G AP and our goal is to show the beneﬁt of removing the encoder of APN and we do this by comparing G AP against CANE , which employs the exact APN ar-
Revised: Next, we analyze the run time of training G AP and our goal is to show the beneﬁt of removing the encoder of APN and we do this by comparing G AP against CANE , which employs the exact APN ar-

Original: [Figure]
Revised: [Figure]

Original: ea
Revised: ea

Original: [Figure]
Revised: [Figure]

Original: u r e m en t
Revised: u r e m en t

Original: [Figure]
Revised: [Figure]

Original: chitecture.
Revised: chitecture.

Original: For this experiment we include two randomly generated graphs, using Erd ˝
Revised: For this experiment we include two randomly generated graphs, using Erd ˝

Original: osR ´ enyi (ERG) and Barab ´
Revised: osR ´ enyi (ERG) and Barab ´

Original: asiAlbert (BAG) models.
Revised: asiAlbert (BAG) models.

Original: ERG has 200K edges, and BAG has 1.5M edges.
Revised: ERG has 200K edges, and BAG has 1.5M edges.

Original: Figure 3(B) clearly shows that G AP is at least 2 times faster than CANE in all the graphs.
Revised: Figure 3(B) clearly shows that G AP is at least 2 times faster than CANE in all the graphs.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: NRL is usually carried out by exploring the structure of the graph and meta data, such as node attributes, attached to the graph (Perozzi et al., 2014; Grover & Leskovec, 2016; Tang et al., 2015;
Revised: NRL is usually carried out by exploring the structure of the graph and meta data, such as node attributes, attached to the graph (Perozzi et al., 2014; Grover & Leskovec, 2016; Tang et al., 2015;

Original: Perozzi et al., 2016; Wang et al., 2016;
Revised: Perozzi et al., 2016; Wang et al., 2016;

Original: Yang et al., 2015; Pan et al., 2016; Sheikh et al., 2019).
Revised: Yang et al., 2015; Pan et al., 2016; Sheikh et al., 2019).

Original: Random walks are widely used to explore local/global neighborhood structures, which are then fed into a learning algorithm.
Revised: Random walks are widely used to explore local/global neighborhood structures, which are then fed into a learning algorithm.

Original: The learning is carried out in unsupervised manner by maximizing the likelihood of observing the neighbor nodes and/or attributes of a center node.
Revised: The learning is carried out in unsupervised manner by maximizing the likelihood of observing the neighbor nodes and/or attributes of a center node.

Original: Recently graph convolution networks have also been proposed for semi-supervised network analysis tasks (Kipf & Welling, 2017; Hamilton et al., 2017; Wu et al., 2019; Velickovic et al., 2017;
Revised: Recently graph convolution networks have also been proposed for semi-supervised network analysis tasks (Kipf & Welling, 2017; Hamilton et al., 2017; Wu et al., 2019; Velickovic et al., 2017;

Original: Abu-El-Haija et al., 2019).
Revised: Abu-El-Haija et al., 2019).

Original: These algorithms work by way of aggregating neighborhood features, with a down-stream objective based on partial labels of nodes, for example.
Revised: These algorithms work by way of aggregating neighborhood features, with a down-stream objective based on partial labels of nodes, for example.

Original: All these methods are essentially different from our approach because they are context-free.
Revised: All these methods are essentially different from our approach because they are context-free.

Original: Context-sensitive learning is another paradigm to NRL that challenges the sufﬁciency of a single representation of a node for applications such as, link prediction, product recommendation, ranking.
Revised: Context-sensitive learning is another paradigm to NRL that challenges the sufﬁciency of a single representation of a node for applications such as, link prediction, product recommendation, ranking.

Original: While some of these methods (Tu et al., 2017; Zhang et al., 2018) rely on textual information, others have also shown that a similar goal can be achieved using just the structure of the graph (Epasto & Perozzi, 2019).
Revised: While some of these methods (Tu et al., 2017; Zhang et al., 2018) rely on textual information, others have also shown that a similar goal can be achieved using just the structure of the graph (Epasto & Perozzi, 2019).

Original: However, they require an extra step of persona decomposition that is based on microscopic level community detection algorithms to identify multiple contexts of a node.
Revised: However, they require an extra step of persona decomposition that is based on microscopic level community detection algorithms to identify multiple contexts of a node.

Original: Unlike the ﬁrst approaches our algorithm does not require extra textual information and with respect to the second ones our approach does not require any sort of community detection algorithm.
Revised: Unlike the ﬁrst approaches our algorithm does not require extra textual information and with respect to the second ones our approach does not require any sort of community detection algorithm.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this study we present a novel context-sensitive graph embedding algorithm called G AP .
Revised: In this study we present a novel context-sensitive graph embedding algorithm called G AP .

Original: It consumes node neighborhood as input feature, which are constructed based on an important assumption that their ordering is arbitrary.
Revised: It consumes node neighborhood as input feature, which are constructed based on an important assumption that their ordering is arbitrary.

Original: To learn representations of nodes G AP employs attentive pooling networks ( APN ).
Revised: To learn representations of nodes G AP employs attentive pooling networks ( APN ).

Original: By exploiting the above assumption it makes an important simpliﬁcation of APN and gain more than 2X speed up over another SOTA method, which employs the exact APN .
Revised: By exploiting the above assumption, it makes an important simpliﬁcation of APN and gains more than 2X speed up over another SOTA method, which employs the exact APN .

Original: Furthermore, G AP consistently outperforms all the baselines and achieves upto ≈ 9% and ≈ 20% improvement over the best performing ones on the link prediction and node clustering tasks, respectively.
Revised: Furthermore, G AP consistently outperforms all the baselines and achieves up to ≈ 9% and ≈ 20% improvement over the best performing ones on the link prediction and node clustering tasks, respectively.

Original: In future we will investigate how node attributes can be incorporated and provide a theoretical framework on the relation between the neighborhood sampling and topological properties. 
Revised: In future we will investigate how node attributes can be incorporated and provide a theoretical framework on the relation between the neighborhood sampling and topological properties. A A PPENDIX

Revised: Here we give a brief ablation experiment to strengthen the case for G AP .

Revised: Concretely, we seek to compare G AP with different baselines to further motivate our choice of (i) the way we model nodes using their neighborhood, (ii) the assumption that order in this neighborhood is not important, and (iii) the choice of the APN algorithm without the CNN or BI -LSTM encoder.

Revised: To this end, we introduce the following baselines and apart from the indicated difference everything will be the same as G AP .

Revised: Datasets

Revised: Cora

Revised: Email

Revised: Baselines

Revised: Training ratio

Revised: Training ratio 25% 55% 75% 95% 25% 55% 75% 95%

Revised: G AP C N

Revised: G AP A PN

Revised: [Table]

Revised: First we assume order is important in the neighborhood of nodes based on neighbors similarity with the current node.

Revised: We use two types a topological feature to induce order, which is common neighbors, referred as G AP C N .

Revised: This baseline uses the exact APN model to capture “order”.

Revised: Second, we use the same input as in G AP , but nodes’ neighborhood is now randomly permuted, and the exact APN model is employed; this baseline is referred to as G AP A PN 3.

Revised: Finally we replace the G AP model with a standard feed-forward neural network that consumes the same input (Embedding matrices S and T ) and also has the same learning objective speciﬁed in Eq. 2; the baseline is referred to as G AP M LP .

Revised: In Table 7 we report the results of the ablation experiment.

Revised: This sheds some light on the assumptions and the design choices of G AP .

Revised: For the reported results G AP C N and G AP

Revised: A PN we use a CNN

Revised: Encoder.

Revised: In both cases, they quickly over-ﬁt the data, and we found out that we have to employ aggressive regularization using a dropout rate of 0.95.

Revised: In addition, for G AP C N we have also observed that as we increase the kernel size to greater values than 1 (up to 5) the results keep getting worse, and hence, what we reported is the best one which is obtained using kernel size of 1.

Revised: For example, with a kernel size of three we have obtained 54, 55, 56, and 58 percent AUC, respectively, for 25, 55, 75, and 95 percent training ratios for the Cora dataset.

Revised: For the Email dataset 66, 69,and 77 percent AUC, respectively, for 25, 55, 75, and 95 percent training ratios.

Revised: We conjecture that this is due the models attempt to enforce high-level neighborhood patterns that are not the intrinsic behavior of the underlying edge formation phenomena.

Revised: Rather what is important is to effectively pay attention to the presence of neighbors both in N s and N t regardless of their order.

Revised: Apparently, training these models is at least twice slower than G AP as it is also illustrated in Section 4.3.

Revised: In the case of G AP A PN though we have not observed important variations in AUC with respect to the change in the kernel size the training time of these model has increased almost by an order of magnitude.

Revised: Finally we see that the mutual attention mechanism plays an important role by observing the result from the G AP M LP .