Original: M ULTI - SCALE A TTRIBUTED N ODE E MBEDDING
Revised: M ULTI - SCALE A TTRIBUTED N ODE E MBEDDING

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: We present network embedding algorithms that capture information about a node from the local distribution over node attributes around it, as observed over random walks following an approach similar to Skip-gram.
Revised: We present network embedding algorithms that capture information about a node from the local distribution over node attributes around it, as observed over random walks following an approach similar to Skip-gram.

Original: Observations from neighborhoods of different sizes are either pooled (AE) or encoded distinctly in a multiscale approach (MUSAE).
Revised: Observations from neighborhoods of different sizes are either pooled (AE) or encoded distinctly in a multiscale approach (MUSAE).

Original: Capturing attribute-neighborhood relationships over multiple scales is useful for a diverse range of applications, including latent feature identiﬁcation across disconnected networks with similar attributes.
Revised: Capturing attribute-neighborhood relationships over multiple scales is useful for a diverse range of applications, including latent feature identiﬁcation across disconnected networks with similar attributes.

Original: We prove theoretically that matrices of node-feature pointwise mutual information are implicitly factorized by the embeddings.
Revised: We prove theoretically that matrices of node-feature pointwise mutual information are implicitly factorized by the embeddings.

Original: Experiments show that our algorithms are robust, computationally efﬁcient and outperform comparable models on social, web and citation network datasets.
Revised: Experiments show that our algorithms are robust, computationally efﬁcient and outperform comparable models on social, web and citation network datasets.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Node embedding is a fundamental technique in network analysis that serves as a precursor to numerous downstream machine learning and optimisation tasks, e.g. community detection, network visualization and link prediction (Perozzi et al., 2014; Grover & Leskovec, 2016; Tang et al., 2015).
Revised: Node embedding is a fundamental technique in network analysis that serves as a precursor to numerous downstream machine learning and optimisation tasks, e.g. community detection, network visualization and link prediction (Perozzi et al., 2014; Grover & Leskovec, 2016; Tang et al., 2015).

Original: Several recent network embedding methods, such as Deepwalk (Perozzi et al., 2014), Node2Vec (Grover & Leskovec, 2016) and Walklets (Perozzi et al., 2017), achieve impressive performance by learning the network structure following an approach similar to Word2Vec Skip-gram (Mikolov et al., 2013b), originally designed for word embedding.
Revised: Several recent network embedding methods, such as Deepwalk (Perozzi et al., 2014), Node2Vec (Grover & Leskovec, 2016) and Walklets (Perozzi et al., 2017), achieve impressive performance by learning the network structure following an approach similar to Word2Vec Skip-gram (Mikolov et al., 2013b), originally designed for word embedding.

Original: In these works, sequences of neighboring nodes are generated from random walks over a network, and representations are distilled from extracted node-node proximity statistics that capture local neighbourhood information.
Revised: In these works, sequences of neighboring nodes are generated from random walks over a network, and representations are distilled from extracted node-node proximity statistics that capture local neighbourhood information.

Original: When the nodes of a network have attributes (or features), their embeddings can be used to capture information about the attributes in their local neighbourhood.
Revised: When the nodes of a network have attributes (or features), their embeddings can be used to capture information about the attributes in their local neighbourhood.

Original: For a social network, attributes might represent a person’s interests, habits, history or preferences.
Revised: For a social network, attributes might represent a person’s interests, habits, history or preferences.

Original: The pattern of node attributes are often similar in a neighborhood, and conversely, nodes with similar attributes are more likely to be connected.
Revised: The pattern of node attributes are often similar in a neighborhood, and conversely, nodes with similar attributes are more likely to be connected.

Original: This property is known as homophily .
Revised: This property is known as homophily .

Original: Attributed network embedding methods (Yang et al., 2015; Huang et al., 2017; Liao et al., 2018) leverage this additional information to supplement that of node neighbourhood structure, beneﬁting many applications, e.g. recommender systems, node classiﬁcation and link prediction (Yang et al., 2018; Yang & Yang, 2018; Zhang et al., 2018).
Revised: Attributed network embedding methods (Yang et al., 2015; Huang et al., 2017; Liao et al., 2018) leverage this additional information to supplement that of node neighbourhood structure, beneﬁting many applications, e.g. recommender systems, node classiﬁcation and link prediction (Yang et al., 2018; Yang & Yang, 2018; Zhang et al., 2018).

Original: The neighborhood of a node can be considered at different path lengths, or scales .
Revised: The neighborhood of a node can be considered at different path lengths, or scales .

Original: In a social network, near neighbors may correspond to classmates, whereas nodes separated by greater scales may be in different cities or countries.
Revised: In a social network, near neighbors may correspond to classmates, whereas nodes separated by greater scales may be in different cities or countries.

Original: Attributes of neighbors at different scales can be considered separately ( multi-scale ) or pooled in some way (e.g. weighted average).
Revised: Attributes of neighbors at different scales can be considered separately ( multi-scale ) or pooled in some way (e.g. weighted average).

Original: Figure 1a shows how the attribute distribution over neighbourhoods at different scales can indicate nodes with similar network roles even if they are distant in the network, or even in different networks.
Revised: Figure 1a shows how the attribute distribution over neighbourhoods at different scales can indicate nodes with similar network roles even if they are distant in the network, or even in different networks.

Original: Methods that take attributes of nearby nodes into account generalizes those that do not, e.g. Perozzi et al.
Revised: Methods that take attributes of nearby nodes into account generalizes those that do not, e.g. Perozzi et al.

Original: (2017), for which feature vectors can be considered standard basis vectors.
Revised: (2017), for which feature vectors can be considered standard basis vectors.

Original: Many embedding methods correspond to matrix factorization, indeed some attributed embedding methods (e.g. Yang et al.
Revised: Many embedding methods correspond to matrix factorization, indeed some attributed embedding methods (e.g. Yang et al.

Original: (2018)) explicitly factorize a matrix of link-attribute information.
Revised: (2018)) explicitly factorize a matrix of link-attribute information.

Original: Embeddings learned using Skip-gram are known to factorize a matrix of pointwise mutual information (PMI) of co-occurrences between each word and local context words (Levy & Goldberg, 2014).
Revised: Embeddings learned using Skip-gram are known to factorize a matrix of pointwise mutual information (PMI) of co-occurrences between each word and local context words (Levy & Goldberg, 2014).

Original: Related network embedding methods (Perozzi et al., 2014; Grover & Leskovec, 2016; Tang et al., 2015; Qiu et al., 2018) also implicitly factorize PMI matrices based on the probability of encountering each (context) node on a random walk from each starting node (Qiu et al., 2018).
Revised: Related network embedding methods (Perozzi et al., 2014; Grover & Leskovec, 2016; Tang et al., 2015; Qiu et al., 2018) also implicitly factorize PMI matrices based on the probability of encountering each (context) node on a random walk from each starting node (Qiu et al., 2018).

Original: {♣ ; ♥ ; ♦} {♥} {♣ ; ♦} {♥ ; ♠} {♣ ; ♥ ; ♦ ; ♠} {♦ ; ♠} {♠} {♣} {♥ ; ♠} {♣ ; ♦} {♥}
Revised: {♣ ; ♥ ; ♦} {♥} {♣ ; ♦} {♥ ; ♠} {♣ ; ♥ ; ♦ ; ♠} {♦ ; ♠} {♠} {♣} {♥ ; ♠} {♣ ; ♦} {♥}

Original: A
Revised: A

Original: B
Revised: B

Original: C
Revised: C

Original: D
Revised: D

Original: E
Revised: E

Original: F
Revised: F

Original: G
Revised: G

Original: H
Revised: H

Original: I
Revised: I

Original: J
Revised: J

Original: K (a) Attributed example graph.
Revised: K (a) Attributed example graph.

Original: r
Revised: r

Original: D e n s it y
Revised: D e n s it y

Original: o
Revised: o

Original: f A r
Revised: f A r

Original: X
Revised: X

Original: FacebookGitWebMLTwitchESTwitchRUChameleonsSquirrels
Revised: FacebookGitWebMLTwitchESTwitchRUChameleonsSquirrels

Original: [Figure]
Revised: [Figure]

Original: Our key contributions are:
Revised: Our key contributions are:

Original: 1. to introduce the ﬁrst Skip-gram style embedding algorithms that consider attribute distributions over local neighborhoods, both pooled ( AE ) and multi-scale ( MUSAE ), and their counterparts that attribute distinct features to each node ( AE-EGO and MUSAE-EGO ); 2. to theoretically prove that their embeddings approximately factorize PMI matrices based on the product of an adjacency matrix power and node-feature matrix; 3. to show that popular network embedding methods DeepWalk (Perozzi et al., 2014) and Walklets (Perozzi et al., 2017) are special cases of our AE and MUSAE ; 4.
Revised: 1. to introduce the ﬁrst Skip-gram style embedding algorithms that consider attribute distributions over local neighborhoods, both pooled ( AE ) and multi-scale ( MUSAE ), and their counterparts that attribute distinct features to each node ( AE-EGO and MUSAE-EGO ); 2. to theoretically prove that their embeddings approximately factorize PMI matrices based on the product of an adjacency matrix power and node-feature matrix; 3. to show that popular network embedding methods DeepWalk (Perozzi et al., 2014) and Walklets (Perozzi et al., 2017) are special cases of our AE and MUSAE ; 4.

Original: we show empirically that AE and MUSAE embeddings enable strong performance at regression, classiﬁcation, and link prediction tasks for real-world networks (e.g. Wikipedia and Facebook), are computationally scalable and enable transfer learning between networks.
Revised: we show empirically that AE and MUSAE embeddings enable strong performance at regression, classiﬁcation, and link prediction tasks for real-world networks (e.g. Wikipedia and Facebook), are computationally scalable and enable transfer learning between networks.

Original: We provide reference implementations of AE and MUSAE , together with the datasets used for evaluation at https://github.com/iclr2020/MUSAE .
Revised: We provide reference implementations of AE and MUSAE , together with the datasets used for evaluation at https://github.com/iclr2020/MUSAE .

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: Efﬁcient unsupervised learning of node embeddings for large networks has seen unprecedented development in recent years.
Revised: Efﬁcient unsupervised learning of node embeddings for large networks has seen unprecedented development in recent years.

Original: The current paradigm focuses on learning latent space representations of nodes such that those that share neighbors (Perozzi et al., 2014; Tang et al., 2015; Grover &
Revised: The current paradigm focuses on learning latent space representations of nodes such that those that share neighbors (Perozzi et al., 2014; Tang et al., 2015; Grover &

Original: Leskovec, 2016; Perozzi et al., 2017), structural roles (Ribeiro et al., 2017; Ahmed et al., 2018) or attributes are located close together in the embedding space.
Revised: Leskovec, 2016; Perozzi et al., 2017), structural roles (Ribeiro et al., 2017; Ahmed et al., 2018) or attributes are located close together in the embedding space.

Original: Our work falls under the last of these categories as our goal is to learn similar latent representations for nodes with similar sets of features in their neighborhoods, both on a pooled and multi-scale basis.
Revised: Our work falls under the last of these categories as our goal is to learn similar latent representations for nodes with similar sets of features in their neighborhoods, both on a pooled and multi-scale basis.

Original: Neighborhood preserving node embedding procedures place nodes with common ﬁrst, second and higher order neighbors within close proximity in the embedding space.
Revised: Neighborhood preserving node embedding procedures place nodes with common ﬁrst, second and higher order neighbors within close proximity in the embedding space.

Original: Recent works in the neighborhood preserving node embedding literature were inspired by the Skip-gram model (Mikolov et al., 2013a;b), which generates word embeddings by implicitly factorizing a shifted pointwise mutual information (PMI) matrix (Levy & Goldberg, 2014) obtained from a text corpus.
Revised: Recent works in the neighborhood preserving node embedding literature were inspired by the Skip-gram model (Mikolov et al., 2013a;b), which generates word embeddings by implicitly factorizing a shifted pointwise mutual information (PMI) matrix (Levy & Goldberg, 2014) obtained from a text corpus.

Original: This procedure inspired DeepWalk (Perozzi et al., 2014), a method which generates truncated random walks over a graph to obtain a “corpus” from which the Skip-gram model generates neighborhood preserving node embeddings.
Revised: This procedure inspired DeepWalk (Perozzi et al., 2014), a method which generates truncated random walks over a graph to obtain a “corpus” from which the Skip-gram model generates neighborhood preserving node embeddings.

Original: In doing so, DeepWalk implicitly factorizes a PMI matrix, which can be shown, based on the underlying ﬁrst-order Markov process, to correspond to the mean of a set of normalized adjacency matrix powers up to a given order (Qiu et al., 2018).
Revised: In doing so, DeepWalk implicitly factorizes a PMI matrix, which can be shown, based on the underlying ﬁrst-order Markov process, to correspond to the mean of a set of normalized adjacency matrix powers up to a given order (Qiu et al., 2018).

Original: Such pooling of matrices can be suboptimal since neighbors over increasing path lengths (or scales) are treated equally or according to ﬁxed weightings (Mikolov et al., 2013a; Grover & Leskovec, 2016); whereas it has been found that an optimal weighting may be task or dataset speciﬁc (Abu-El-Haija et al., 2018).
Revised: Such pooling of matrices can be suboptimal since neighbors over increasing path lengths (or scales) are treated equally or according to ﬁxed weightings (Mikolov et al., 2013a; Grover & Leskovec, 2016); whereas it has been found that an optimal weighting may be task or dataset speciﬁc (Abu-El-Haija et al., 2018).

Original: In contrast, multi-scale node embedding methods such as LINE (Tang et al., 2015), GraRep (Cao et al., 2015) and Walklets (Perozzi et al., 2017) separately learn lower-dimensional node embedding components from each adjacency matrix power and concatenate them to form the full node representation.
Revised: In contrast, multi-scale node embedding methods such as LINE (Tang et al., 2015), GraRep (Cao et al., 2015) and Walklets (Perozzi et al., 2017) separately learn lower-dimensional node embedding components from each adjacency matrix power and concatenate them to form the full node representation.

Original: Such un-pooled representations, comprising distinct but less information at each scale, are found to give higher performance in a number of downstream settings, without increasing the overall number of free parameters (Perozzi et al., 2017).
Revised: Such un-pooled representations, comprising distinct but less information at each scale, are found to give higher performance in a number of downstream settings, without increasing the overall number of free parameters (Perozzi et al., 2017).

Original: Attributed node embedding procedures reﬁne ideas from neighborhood based node embeddings to also incorporate node attributes (equivalently, features or labels)
Revised: Attributed node embedding procedures reﬁne ideas from neighborhood based node embeddings to also incorporate node attributes (equivalently, features or labels)

Original: (Yang et al., 2015; Liao et al., 2018;
Revised: (Yang et al., 2015; Liao et al., 2018;

Original: Huang et al., 2017; Yang et al., 2018; Yang & Yang, 2018).
Revised: Huang et al., 2017; Yang et al., 2018; Yang & Yang, 2018).

Original: Similarities between both a node’s neighborhood structure and features contribute to determining pairwise proximity in the node embedding space.
Revised: Similarities between both a node’s neighborhood structure and features contribute to determining pairwise proximity in the node embedding space.

Original: These models follow quite different strategies to obtain such representations.
Revised: These models follow quite different strategies to obtain such representations.

Original: The most elemental procedure, TADW (Yang et al., 2015), decomposes a convex combination of normalized adjacency matrix powers into a matrix product that includes the feature matrix.
Revised: The most elemental procedure, TADW (Yang et al., 2015), decomposes a convex combination of normalized adjacency matrix powers into a matrix product that includes the feature matrix.

Original: Several other models, such as SINE (Zhang et al., 2018) and ASNE (Liao et al., 2018), implicitly factorize a matrix formed by concatenating the feature and adjacency matrices.
Revised: Several other models, such as SINE (Zhang et al., 2018) and ASNE (Liao et al., 2018), implicitly factorize a matrix formed by concatenating the feature and adjacency matrices.

Original: Other approaches such as TENE (Yang & Yang, 2018), formulate the attributed node embedding task as a joint non-negative matrix factorization problem in which node representations obtained from sub-tasks are used to regularize one another.
Revised: Other approaches such as TENE (Yang & Yang, 2018), formulate the attributed node embedding task as a joint non-negative matrix factorization problem in which node representations obtained from sub-tasks are used to regularize one another.

Original: AANE (Huang et al., 2017) uses a similar network structure based regularization approach, in which a node feature similarity matrix is decomposed using the alternating direction method of multipliers.
Revised: AANE (Huang et al., 2017) uses a similar network structure based regularization approach, in which a node feature similarity matrix is decomposed using the alternating direction method of multipliers.

Original: The method most similar to our own is BANE (Yang et al., 2018), in which the product of a normalized adjacency matrix power and a feature matrix is explicitly factorized to obtain attributed node embeddings.
Revised: The method most similar to our own is BANE (Yang et al., 2018), in which the product of a normalized adjacency matrix power and a feature matrix is explicitly factorized to obtain attributed node embeddings.

Original: Many other methods exist, but do not consider the attributes of higher order neighborhoods (Yang et al., 2015;
Revised: Many other methods exist, but do not consider the attributes of higher order neighborhoods (Yang et al., 2015;

Original: Liao et al., 2018; Huang et al., 2017; Zhang et al., 2018;
Revised: Liao et al., 2018; Huang et al., 2017; Zhang et al., 2018;

Original: Yang & Yang, 2018).
Revised: Yang & Yang, 2018).

Original: The relationship between our pooled ( AE ) and multi-scale ( MUSAE ) attributed node embedding methods mirrors that between graph convolutional neural networks (GCNNs) and multi-scale GCNNs.
Revised: The relationship between our pooled ( AE ) and multi-scale ( MUSAE ) attributed node embedding methods mirrors that between graph convolutional neural networks (GCNNs) and multi-scale GCNNs.

Original: Widely used graph convolutional layers, such as GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2018), APPNP (Klicpera et al., 2019), SGCONV (Wu et al., 2019) and ClusterGCN (Chiang et al., 2019), create latent node representations that pool node attributes from arbitrary order neighborhoods, which are then inseparable and unrecoverable.
Revised: Widely used graph convolutional layers, such as GCN (Kipf & Welling, 2017), GraphSage (Hamilton et al., 2017), GAT (Veli ˇ ckovi ´ c et al., 2018), APPNP (Klicpera et al., 2019), SGCONV (Wu et al., 2019) and ClusterGCN (Chiang et al., 2019), create latent node representations that pool node attributes from arbitrary order neighborhoods, which are then inseparable and unrecoverable.

Original: In contrast, MixHop (Abu-El-Haija et al., 2019) learns latent features for each proximity.
Revised: In contrast, MixHop (Abu-El-Haija et al., 2019) learns latent features for each proximity.

Original:  A TTRIBUTED E MBEDDING M ODELS
Revised:  A TTRIBUTED E MBEDDING M ODELS

Original: We now deﬁne algorithms to learn node embeddings using the attributes of nearby nodes, that allows both node and attribute embeddings to be learned jointly.
Revised: We now deﬁne algorithms to learn node embeddings using the attributes of nearby nodes, that allows both node and attribute embeddings to be learned jointly.

Original: The aim is to learn similar embeddings for nodes that occur in neighbourhoods of similar attributes; and similar embeddings for attributes that often occur in similar neighbourhoods of nodes.
Revised: The aim is to learn similar embeddings for nodes that occur in neighbourhoods of similar attributes; and similar embeddings for attributes that often occur in similar neighbourhoods of nodes.

Original: Let G =
Revised: Let G =

Original: ( V , L ) be an undirected graph of interest where V and L are the sets of vertices and edges (or links) respectively; and let F be the set of all possible node features (i.e. attributes).
Revised: ( V , L ) be an undirected graph of interest where V and L are the sets of vertices and edges (or links) respectively; and let F be the set of all possible node features (i.e. attributes).

Original: We deﬁne F v ⊆ F as the subset of features belonging to each node v ∈ V .
Revised: We deﬁne F v ⊆ F as the subset of features belonging to each node v ∈ V .

Original: An embedding of nodes is a mapping g : V → R d that assigns a d -dimensional representation g ( v ) (or simply g v ) to each node v and is fully described by a matrix G ∈ R | V |× d .
Revised: An embedding of nodes is a mapping g : V → R d that assigns a d -dimensional representation g ( v ) (or simply g v ) to each node v and is fully described by a matrix G ∈ R | V |× d .

Original: Similarly, an embedding of the features (to the same latent space) is a mapping h : F → R d with embeddings denoted h ( f ) (or simply h f ), and is fully described by a matrix H ∈ R | F |× d .
Revised: Similarly, an embedding of the features (to the same latent space) is a mapping h : F → R d with embeddings denoted h ( f ) (or simply h f ), and is fully described by a matrix H ∈ R | F |× d .

Original: 3. A TTRIBUTED E MBEDDING
Revised: 3. A TTRIBUTED E MBEDDING

Original: The Attributed Embedding ( AE ) procedure is described by Algorithm 1.
Revised: The Attributed Embedding ( AE ) procedure is described by Algorithm 1.

Original: We sample n nodes w 1 , from which to start attributed random walks on G , with probability proportional to their degree (Line 2).
Revised: We sample n nodes w 1 , from which to start attributed random walks on G , with probability proportional to their degree (Line 2).

Original: From each starting node, a node sequence of length l is sampled over G (Line 3), where sampling follows a ﬁrst order random walk.
Revised: From each starting node, a node sequence of length l is sampled over G (Line 3), where sampling follows a ﬁrst order random walk.

Original: For a given window size t , we iterate over each of the ﬁrst l − t nodes of the sequence termed source nodes w j (Line 4).
Revised: For a given window size t , we iterate over each of the ﬁrst l − t nodes of the sequence termed source nodes w j (Line 4).

Original: For each source node, we consider the following t nodes as target nodes (Line 5).
Revised: For each source node, we consider the following t nodes as target nodes (Line 5).

Original: For each target node w j + r , we add the tuple ( w j , f ) to the corpus D for each target feature f ∈ F w j + r (Lines 6 and 7).
Revised: For each target node w j + r , we add the tuple ( w j , f ) to the corpus D for each target feature f ∈ F w j + r (Lines 6 and 7).

Original: We also consider features of the 
Revised: We also consider features of the source node f ∈ F w j , adding each ( w j + r , f ) tuple to D (Lines 9 and 10). Running Skip-gram on D with b negative samples (Line 15) generates the d -dimensional node and feature embeddings.

Original: [Table] [Table]
Revised: [Table] [Table]

Original:  ( w j ,f ) tomultiset D
Revised: Addthetuple ( w j ,f ) tomultiset D

Original: → r .endfor f in F w j do
Revised: → r .endfor f in F w j do

Original: Addthetuple ( w j + r ,f ) tomultiset D ← r .endendend 14 end 15 for r in 1 : t do
Revised: Addthetuple ( w j + r ,f ) tomultiset D ← r .endendend 14 end 15 for r in 1 : t do

Original: Create D r byuniﬁcationof D → r and D ← r .
Revised: Create D r byuniﬁcationof D → r and D ← r .

Original: RunSGNSon D r with b negativesamplesand dt dimensions.
Revised: RunSGNSon D r with b negativesamplesand dt dimensions.

Original: Output g rv , ∀ v ∈ V ,and h rf , ∀ f ∈ F = ∪ V F v .
Revised: Output g rv , ∀ v ∈ V ,and h rf , ∀ f ∈ F = ∪ V F v .

Original: 19 end
Revised: 19 end

Original: Algorithm 2: MUSAE sampling and training proceduresource node f ∈ F w j , adding each ( w j + r , f ) tuple to D (Lines 9 and 10).
Revised: Algorithm 2: MUSAE sampling and training procedure

Original: Running Skip-gram on D with b negative samples (Line 15) generates the d -dimensional node and feature embeddings.

Original: 3. M ULTI - SCALE A TTRIBUTED E MBEDDING
Revised: 3. M ULTI - SCALE A TTRIBUTED E MBEDDING

Original: The AE method (Algorithm 1) pools feature sets of neighborhoods at different proximities.
Revised: The AE method (Algorithm 1) pools feature sets of neighborhoods at different proximities.

Original: Inspired by the performance of (unattributed) multi-scale node embeddings, we adapt the AE algorithm to give multi-scale attributed node embeddings ( MUSAE ).
Revised: Inspired by the performance of (unattributed) multi-scale node embeddings, we adapt the AE algorithm to give multi-scale attributed node embeddings ( MUSAE ).

Original: The embedding component of a node v ∈ V for a speciﬁc proximity r ∈ { 1 , ..., t } is given by a mapping g r : V → R d/t (assuming t divides d ).
Revised: The embedding component of a node v ∈ V for a speciﬁc proximity r ∈ { 1 , ..., t } is given by a mapping g r : V → R d/t (assuming t divides d ).

Original: Similarly, the embedding component of feature f ∈ F at proximity r is given by a mapping h r : F → R d/t .
Revised: Similarly, the embedding component of feature f ∈ F at proximity r is given by a mapping h r : F → R d/t .

Original: Concatenating gives a d -dimensional embedding for each node and feature.
Revised: Concatenating gives a d -dimensional embedding for each node and feature.

Original: The Multi-Scale Attributed Embedding procedure is described by Algorithm 2.
Revised: The Multi-Scale Attributed Embedding procedure is described by Algorithm 2.

Original: We again sample n starting nodes w 1 with a probability proportional to node degree (Line 2) and, for each, sample a node sequence of length l over G (Line 3) according to either a ﬁrst or second order random walk.
Revised: We again sample n starting nodes w 1 with a probability proportional to node degree (Line 2) and, for each, sample a node sequence of length l over G (Line 3) according to either a ﬁrst or second order random walk.

Original: For a given window size t , we iterate over the ﬁrst l − t (source) nodes w j of the sequence (Line 4) and for each source node we iterate through the t (target) nodes w j + r that follow (Line 5).
Revised: For a given window size t , we iterate over the ﬁrst l − t (source) nodes w j of the sequence (Line 4) and for each source node we iterate through the t (target) nodes w j + r that follow (Line 5).

Original: We again consider each target node feature f ∈ F w j + r , but now add tuples ( w j , f ) to a sub-corpus D r
Revised: We again consider each target node feature f ∈ F w j + r , but now add tuples ( w j , f ) to a sub-corpus D r

Original: → (Lines 6 and 7).
Revised: → (Lines 6 and 7).

Original: We add tuples ( w j + r , f ) to another sub-corpus D r ← for each source node feature f ∈ F w j (Lines 9 and 10).
Revised: We add tuples ( w j + r , f ) to another sub-corpus D r ← for each source node feature f ∈ F w j (Lines 9 and 10).

Original: Running Skip-gram on each sub-corpus D r = D r → ∪ D r ← with b negative samples (Line 16) output t ( d t ) -dimensional
Revised: Running Skip-gram on each sub-corpus D r = D r → ∪ D r ← with b negative samples (Line 16) output t ( dt ) -dimensional

Original: node and feature embeddings that are concatenated.
Revised: node and feature embeddings that are concatenated.

Original:  A TTRIBUTED EMBEDDING AS IMPLICIT MATRIX FACTORIZATION
Revised:  A TTRIBUTED EMBEDDING AS IMPLICIT MATRIX FACTORIZATION

Original: Levy & Goldberg (2014) showed that the loss function of Skip-gram with negative sampling (SGNS) is minimized if the embedding matrices factorize a matrix of pointwise mutual information (PMI) of word co-occurrence statistics.
Revised: Levy & Goldberg (2014) showed that the loss function of Skip-gram with negative sampling (SGNS) is minimized if the embedding matrices factorize a matrix of pointwise mutual information (PMI) of word co-occurrence statistics.

Original: Speciﬁcally, for a word dictionary V with | V | = n , SGNS (with b negative samples) outputs two embedding matrices W , C ∈ R d × n such that ∀ w, c ∈ V :
Revised: Speciﬁcally, for a word dictionary V with | V | = n , SGNS (with b negative samples) outputs two embedding matrices W , C ∈ R d × n such that ∀ w, c ∈ V :

Original: [Equation]
Revised: [Equation]

Original: where #( w, c ) , #( w ) , #( c ) denote counts of word-context pair ( w, c ) , w and c over a corpus D ; and word embeddings w w , c c ∈ R d are columns of W and C corresponding to w and c respectively.
Revised: where #( w, c ) , #( w ) , #( c ) denote counts of word-context pair ( w, c ) , w and c over a corpus D ; and word embeddings w w , c c ∈ R d are columns of W and C corresponding to w and c respectively.

Original: Considering #( w ) | D | , #( c ) | D | , #( w,c )
Revised: Considering #( w ) | D | , #( c ) | D | , #( w,c )

Original: | D | as empirical estimates of p ( w ) , p ( c ) and p ( w, c ) respectively shows:
Revised: | D | as empirical estimates of p ( w ) , p ( c ) and p ( w, c ) respectively shows:

Original: [Equation]
Revised: [Equation]

Original: i.e. an approximate low-rank factorization of a shifted PMI matrix (low rank since typically d (cid:28) n ).
Revised: i.e. an approximate low-rank factorization of a shifted PMI matrix (low rank since typically d (cid:28) n ).

Original: Qiu et al.
Revised: Qiu et al.

Original: (2018) extended this result to node embedding models that apply SGNS to a “corpus” generated from random walks over the graph.
Revised: (2018) extended this result to node embedding models that apply SGNS to a “corpus” generated from random walks over the graph.

Original: In the case of DeepWalk where random walks are ﬁrst-order Markov, the joint probability distributions over nodes at different stages of a random walk can be expressed in closed form.
Revised: In the case of DeepWalk where random walks are ﬁrst-order Markov, the joint probability distributions over nodes at different stages of a random walk can be expressed in closed form.

Original: A closed form then follows for the factorized PMI matrix.
Revised: A closed form then follows for the factorized PMI matrix.

Original: We show that AE and MUSAE implicitly perform analogous matrix factorizations.
Revised: We show that AE and MUSAE implicitly perform analogous matrix factorizations.

Original: Notation:
Revised: Notation:

Original: A ∈ R n × n denotes the adjacency matrix and D ∈ R n × n the diagonal degree matrix of a graph G , i.e. D w,w = deg ( w ) =
Revised: A ∈ R n × n denotes the adjacency matrix and D ∈ R n × n the diagonal degree matrix of a graph G , i.e. D w,w = deg ( w ) =

Original: (cid:80) v A w,v .
Revised: (cid:80) v A w,v .

Original: We denote the volume of G by c = (cid:80) v,w A v,w .
Revised: We denote the volume of G by c = (cid:80) v,w A v,w .

Original: We deﬁne the binary attribute matrix F ∈ { 0 , 1 } | V |×| F | by F w,f = 1 f ∈ F w , ∀ w ∈ V , f ∈ F .
Revised: We deﬁne the binary attribute matrix F ∈ { 0 , 1 } | V |×| F | by F w,f = 1 f ∈ F w , ∀ w ∈ V , f ∈ F .

Original: For ease of notation, we let P
Revised: For ease of notation, we let P

Original: =
Revised: =

Original: D − 1 A and E
Revised: D − 1 A and E

Original: = diag ( 1 (cid:62) DF ) , where diag indicates a diagonal matrix.
Revised: = diag ( 1 (cid:62) DF ) , where diag indicates a diagonal matrix.

Original: Interpretation:
Revised: Interpretation:

Original: Assuming G is ergodic: p ( w )
Revised: Assuming G is ergodic: p ( w )

Original: =
Revised: =

Original: deg ( w ) c , w ∈ V is the stationary distribution over nodes, i.e. c − 1 D = diag ( p ( w )) ; and c − 1 A is the stationary joint distribution over consecutive nodes p ( w j , w j +1 ) .
Revised: deg ( w ) c , w ∈ V is the stationary distribution over nodes, i.e. c − 1 D = diag ( p ( w )) ; and c − 1 A is the stationary joint distribution over consecutive nodes p ( w j , w j +1 ) .

Original: F w,f can be considered a Bernoulli parameter describing the probability p ( f | w ) of observing a feature f at a node w and so c − 1 DF describes the stationary joint distribution p ( f, w j ) over nodes and features.
Revised: F w,f can be considered a Bernoulli parameter describing the probability p ( f | w ) of observing a feature f at a node w and so c − 1 DF describes the stationary joint distribution p ( f, w j ) over nodes and features.

Original: Accordingly, P is the matrix of conditional distributions p ( w j +1 | w j ) ; and E is a diagonal matrix proportional to the probability of observing each feature at the stationary distribution p ( f ) (note that p ( f ) need not sum to 1, whereas p ( w ) necessarily must).
Revised: Accordingly, P is the matrix of conditional distributions p ( w j +1 | w j ) ; and E is a diagonal matrix proportional to the probability of observing each feature at the stationary distribution p ( f ) (note that p ( f ) need not sum to 1, whereas p ( w ) necessarily must).

Original: 4. M ULTI - SCALE CASE (MUSAE)
Revised: 4. M ULTI - SCALE CASE (MUSAE)

Original: We know that the SGNS aspect of MUSAE (Algorithm 2, Line 17) is minimized when the learned embeddings g rv , h rf satisfy g rw (cid:62)
Revised: We know that the SGNS aspect of MUSAE (Algorithm 2, Line 17) is minimized when the learned embeddings g rv , h rf satisfy g rw (cid:62)

Original: h rf
Revised: h rf

Original: ≈ log (cid:0) #( w,f ) r | D r | #( w )
Revised: ≈ log (cid:0) #( w,f ) r | D r | #( w )

Original: r
Revised: r

Original: #( f ) r (cid:1) − log b ∀ w ∈ V , f ∈ F .
Revised: #( f ) r (cid:1) − log b ∀ w ∈ V , f ∈ F .

Original: Our aim is to express this factorization in terms of known properties of the graph G and its features.
Revised: Our aim is to express this factorization in terms of known properties of the graph G and its features.

Original: Lemma 1.
Revised: Lemma 1.

Original: The empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps (i) after; or (ii) before node v ∈ V , as given by:
Revised: The empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps (i) after; or (ii) before node v ∈ V , as given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix.
Revised: See Appendix.

Original: Lemma 2.
Revised: Lemma 2.

Original: Empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps either side of node v ∈ V , given by:
Revised: Empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps either side of node v ∈ V , given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: See Appendix.
Revised: See Appendix.

Original: Marginalizing gives unbiased estimates of stationary probability distributions of nodes and features:
Revised: Marginalizing gives unbiased estimates of stationary probability distributions of nodes and features:

Original: [Equation]
Revised: [Equation]

Original: andplim
Revised: and

Original: [Equation]
Revised: [Equation]

Original: Theorem 1.
Revised: Theorem 1.

Original: MUSAE embeddings approximately factorize the node-feature PMI matrix: 
Revised: MUSAE embeddings approximately factorize the node-feature PMI matrix: log

Original: [Equation]
Revised: [Equation]

Original: , for r = 1 , ... , t.
Revised: − log b, for r = 1 , ... , t.

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: 4. P OOLED CASE (AE)
Revised: 4. P OOLED CASE (AE)

Original: Lemma 3.
Revised: Lemma 3.

Original: The empirical statistics of node-feature pairs learned by the AE algorithm give unbiased estimates of mean joint probabilities over different path lengths as follows:
Revised: The empirical statistics of node-feature pairs learned by the AE algorithm give unbiased estimates of mean joint probabilities over different path lengths as follows:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: By construction, | D | = (cid:80) r | D r | , #( w, f )= (cid:80) r #( w, f ) r , | D r | = | D s | ∀ r, s ∈{ 1 , . .
Revised: By construction, | D | = (cid:80) r | D r | , #( w, f )= (cid:80) r #( w, f ) r , | D r | = | D s | ∀ r, s ∈{ 1 , . .

Original: . , t } and so | D s | = t − 1 | D | .
Revised: . , t } and so | D s | = t − 1 | D | .

Original: Combining with Lemma 2, the result follows.
Revised: Combining with Lemma 2, the result follows.

Original: Theorem 2.
Revised: Theorem 2.

Original: AE embeddings approximately factorize the pooled node-feature matrix:
Revised: AE embeddings approximately factorize the pooled node-feature matrix:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is analogous to the proof of Theorem 1.
Revised: The proof is analogous to the proof of Theorem 1.

Original: Remark 1.
Revised: Remark 1.

Original: DeepWalk is a corner case of AE with F = I | V | .
Revised: DeepWalk is a corner case of AE with F = I | V | .

Original: That is, DeepWalk is equivalent to AE if each node has a single unique feature.
Revised: That is, DeepWalk is equivalent to AE if each node has a single unique feature.

Original: Thus E = diag ( 1 (cid:62) DI )=
Revised: Thus E = diag ( 1 (cid:62) DI )=

Original: D and, by Theorem 2, DeepWalk ’s embeddings factorize log (cid:0) ct ( (cid:80) tr =1 P r ) D − 1 (cid:1) − log b , as previously noted by Qiu et al.
Revised: D and, by Theorem 2, DeepWalk ’s embeddings factorize log (cid:0) ct ( (cid:80) tr =1 P r ) D − 1 (cid:1) − log b , as previously noted by Qiu et al.

Original: Remark 2.
Revised: Remark 2.

Original: Walklets is a corner case of MUSAE with F = I | V | .
Revised: Walklets is a corner case of MUSAE with F = I | V | .

Original: Thus, for r = 1 , . .
Revised: Thus, for r = 1 , . .

Original: , t , the embeddings of Walklets factorise log (cid:0)
Revised: , t , the embeddings of Walklets factorise log (cid:0)

Original: c P r D − 1 (cid:1) −
Revised: c P r D − 1 (cid:1) −

Original: log b.
Revised: log b.

Original: Remark 3.
Revised: Remark 3.

Original: Appending an identity matrix I to the feature matrices F of AE and MUSAE (denoted [ F ; I ] ) adds a unique feature to each node.
Revised: Appending an identity matrix I to the feature matrices F of AE and MUSAE (denoted [ F ; I ] ) adds a unique feature to each node.

Original: The resulting algorithms, named AE-EGO and MUSAEEGO, learn
Revised: The resulting algorithms, named AE-EGO and MUSAEEGO, learn

Original: embeddings that, respectively, approximately factorize the node-feature PMI matrices: log (cid:0)
Revised: embeddings that, respectively, approximately factorize the node-feature PMI matrices: log (cid:0)

Original: c P r [ F ; I ]
Revised: c P r [ F ; I ]

Original: E − 1 (cid:1) − log b, ∀ r ∈{ 1 , ..., t } ; and log (cid:0) ct ( t (cid:88) r
Revised: E − 1 (cid:1) − log b, ∀ r ∈{ 1 , ..., t } ; and log (cid:0) ct ( t (cid:88) r

Original: = P r ) [ F ; I ] E − 1 (cid:1) − log b .
Revised: = P r ) [ F ; I ] E − 1 (cid:1) − log b .

Original: C OMPLEXITY ANALYSIS
Revised: C OMPLEXITY ANALYSIS

Original: Under the assumption of a constant number of features per source node and ﬁrst-order attributed random walk sampling, the corpus generation has a runtime complexity of O ( n l t x/y ) , where x = (cid:80) v ∈ V | F v | the total number of features across all nodes (including repetition) and y =
Revised: Under the assumption of a constant number of features per source node and ﬁrst-order attributed random walk sampling, the corpus generation has a runtime complexity of O ( n l t x/y ) , where x = (cid:80) v ∈ V | F v | the total number of features across all nodes (including repetition) and y =

Original: | V | the number of nodes.
Revised: | V | the number of nodes.

Original: Using negative sampling, the optimization runtime of a single asynchronous gradient descent epoch on AE and the joint optimization runtime of MUSAE embeddings is described by O ( b d n l t x/y ) .
Revised: Using negative sampling, the optimization runtime of a single asynchronous gradient descent epoch on AE and the joint optimization runtime of MUSAE embeddings is described by O ( b d n l t x/y ) .

Original: If one does p truncated walks from each source node, the corpus generation complexity is O
Revised: If one does p truncated walks from each source node, the corpus generation complexity is O

Original: ( p y l t x ) and the model optimization runtime is O ( b d p y l t x ) .
Revised: ( p y l t x ) and the model optimization runtime is O ( b d p y l t x ) .

Original: Our later runtime experiments in Section 5 will underpin optimization runtime complexity discussed above.
Revised: Our later runtime experiments in Section 5 will underpin optimization runtime complexity discussed above.

Original: Corpus generation has a memory complexity of O ( n l t x/y ) while the same when generating p truncated walks per node has a memory complexity of O ( p y l t x ) .
Revised: Corpus generation has a memory complexity of O ( n l t x/y ) while the same when generating p truncated walks per node has a memory complexity of O ( p y l t x ) .

Original: Storing the parameters of an AE embedding has a memory complexity of O ( y d ) and MUSAE embeddings also use O ( y d ) memory.
Revised: Storing the parameters of an AE embedding has a memory complexity of O ( y d ) and MUSAE embeddings also use O ( y d ) memory.

Original:  E XPERIMENTAL E VALUATION
Revised:  E XPERIMENTAL E VALUATION

Original: In order to evaluate the quality of created representations we test the embeddings on supervised downstream tasks such as node classiﬁcation, transfer learning across networks, regression, and link prediction.
Revised: In order to evaluate the quality of created representations we test the embeddings on supervised downstream tasks such as node classiﬁcation, transfer learning across networks, regression, and link prediction.

Original: Finally, we investigate how changes in the input size affect the runtime.
Revised: Finally, we investigate how changes in the input size affect the runtime.

Original: For doing so we utilize social networks and web graphs that we collected from Facebook, Github, Twitch and
Revised: For doing so we utilize social networks and web graphs that we collected from Facebook, Github, Twitch and

Original: Wikipedia.
Revised: Wikipedia.

Original: The data sources, collection procedures and the datasets themselves are described with great detail in Appendix B.
Revised: The data sources, collection procedures and the datasets themselves are described with great detail in Appendix B.

Original: In addition we tested our methods on citation networks widely used for model evaluation (Shchur et al., 2018).
Revised: In addition we tested our methods on citation networks widely used for model evaluation (Shchur et al., 2018).

Original: Across all experiments we use the same hyperparameter settings of our own model, competing unsupervised methods and graph neural networks – these are respectively listed in Appendices C, E and F.
Revised: Across all experiments we use the same hyperparameter settings of our own model, competing unsupervised methods and graph neural networks – these are respectively listed in Appendices C, E and F.

Original: 5. N ODE C LASSIFICATION
Revised: 5. N ODE C LASSIFICATION

Original: We evaluate the node classiﬁcation performance in two separate scenarios.
Revised: We evaluate the node classiﬁcation performance in two separate scenarios.

Original: In the ﬁrstone we do k -shot learning by using the attributed embedding vectors with logistic regression to predict labels on the Facebook, Github and Twitch Portugal graphs.
Revised: In the ﬁrst we do k -shot learning by using the attributed embedding vectors with logistic regression to predict labels on the Facebook, Github and Twitch Portugal graphs.

Original: In the secondone we test the predictive performance under a ﬁxed size train-test split to compare against various embedding methods and competitive neural network architectures.
Revised: In the second we test the predictive performance under a ﬁxed size train-test split to compare against various embedding methods and competitive neural network architectures.

Original: 5.1. K - SHOT LEARNING
Revised: 5.1. K - SHOT LEARNING

Original: In this experiment we take k randomly selected samples per class, and use the attributed node embeddings to train a logistic regression model with l 2 regularization and predict the labels on the remaining vertices.
Revised: In this experiment we take k randomly selected samples per class, and use the attributed node embeddings to train a logistic regression model with l 2 regularization and predict the labels on the remaining vertices.

Original: We repeated the above procedure with seeded splits 100 times to obtain robust comparable results (Shchur et al., 2018).
Revised: We repeated the above procedure with seeded splits 100 times to obtain robust comparable results (Shchur et al., 2018).

Original: From these we calculated the average of micro averaged
Revised: From these we calculated the average of micro averaged

Original: F 1 scores to compare our own methods with other unsupervised node embedding procedures.
Revised: F 1 scores to compare our own methods with other unsupervised node embedding procedures.

Original: We varied k in order to show the efﬁcacy of the methods – what are the gains when the training set size is increased.
Revised: We varied k in order to show the efﬁcacy of the methods – what are the gains when the training set size is increased.

Original: These results are plotted on subplots of Figure 2 forthe Facebook, Github and Twitch Portugal networks.
Revised: These results are plotted in Figure 2 for Facebook, Github and Twitch Portugal networks.

Original: Based on these plots it is evident that MUSAE and AE embeddings have little gains in terms of micro F 1 score when additional data points are added to the training set when k is larger than 12.
Revised: Based on these plots it is evident that MUSAE and AE embeddings have little gains in terms of micro F 1 score when additional data points are added to the training set when k is larger than 12.

Original: This implies that our method is data efﬁcient.
Revised: This implies that our method is data efﬁcient.

Original: Moreover, MUSAE-EGO and AE-EGO have a slight performance advantage, which means that including the nodes in the attributed random walks helps when a small amount of labeled data is available in the downstream task.[Figure]
Revised: Moreover, MUSAE-EGO and AE-EGO have a slight performance advantage, which means that including the nodes in the attributed random walks helps when a small amount of labeled data is available in the downstream task.

Original: [Figure]
Revised: [Figure]

Original: MachineLearning     0 .0 .0 .0 .0 .

Original: Trainingsamplesperclass

Original: T e s t

Original: m i c r o F 1 s c o r e

Original: TwitchPortugal

Original: TADW

Original: AANE

Original: ASNE

Original: BANE

Original: TENE

Original: AE

Original: MUSAE AEEGO MUSAEEGO 5.1. F IXED RATIO TRAIN - TEST SPLITS
Revised: 5.1. F IXED RATIO TRAIN - TEST SPLITS

Original: In this series of experiments we created a 100 seeded train test splits of nodes (80% train - 20% test) and calculated weighted, micro and macro averaged F 1 scores on the test set to compare our methods to various embedding and graph neural network methods.
Revised: In this series of experiments we created a 100 seeded train test splits of nodes (80% train - 20% test) and calculated weighted, micro and macro averaged F 1 scores on the test set to compare our methods to various embedding and graph neural network methods.

Original: Across procedures the same random seeds were used to obtain the train-test split this way the performances are directly comparable.
Revised: Across procedures the same random seeds were used to obtain the train-test split this way the performances are directly comparable.

Original: We attached these results on the Facebook, Github and Twitch Portugal graphs as Table 5 of
Revised: We attached these results on the Facebook, Github and Twitch Portugal graphs as Table 6 of

Original: Appendix G.
Revised: Appendix G.

Original: In each column red denotes the best performing unsupervised embedding model and blue corresponds to the strongest supervised neural model.
Revised: In each column red denotes the best performing unsupervised embedding model and blue corresponds to the strongest supervised neural model.

Original: We also attached additional supporting results using the same experimental setting with the unsupervised methods on the Cora, Citeseer, and Pubmed graphs as Table 6 of Appendix G.
Revised: We also attached additional supporting results using the same experimental setting with the unsupervised methods on the Cora, Citeseer, and Pubmed graphs as Table 5 of Appendix G.

Original: In terms of micro F 1 score our strongest method outperforms on the Facebook and GitHub networks the best unsupervised method by 1.01% and 0.47% respectively.
Revised: In terms of micro F 1 score our strongest method outperforms on the Facebook and GitHub networks the best unsupervised method by 1.01% and 0.47% respectively.

Original: On the Twitch Portugal network the relative micro F 1 advantage of ASNE over our best method is 1.02%.
Revised: On the Twitch Portugal network the relative micro F 1 advantage of ASNE over our best method is 1.02%.

Original: Supervised node embedding methods outperform our and other unsupervised methods on every dataset for most metrics.
Revised: Supervised node embedding methods outperform our and other unsupervised methods on every dataset for most metrics.

Original: In terms of micro F 1 this relative advantage over our best performing model variant is the largest with 4.67% on the Facebook network, and only 0.11% on Twitch Portugal.
Revised: In terms of micro F 1 this relative advantage over our best performing model variant is the largest with 4.67% on the Facebook network, and only 0.11% on Twitch Portugal.

Original: One can make four general observations based on our results (i) multi-scale representations can help with the classiﬁcation tasks compared to pooled ones; (ii) the addition of the nodes in the ego augmented models to the feature sets does not help the performance when a large amount of labeled training data is available; (iii) based on the standard errors supervised neural models do not necessarily have a signiﬁcant advantage over unsupervised methods (see the results on the Github and
Revised: One can make four general observations based on our results (i) multi-scale representations can help with the classiﬁcation tasks compared to pooled ones; (ii) the addition of the nodes in the ego augmented models to the feature sets does not help the performance when a large amount of labeled training data is available; (iii) based on the standard errors supervised neural models do not necessarily have a signiﬁcant advantage over unsupervised methods (see the results on the Github and

Original: Twitch datasets); (iv) attributed node embedding methods that only consider ﬁrst-order neighbourhoods have a poor performance.
Revised: Twitch datasets); (iv) attributed node embedding methods that only consider ﬁrst-order neighbourhoods have a poor performance.

Original: 5. T RANSFER L EARNING ON TWITCH SOCIAL NETWORKS
Revised: 5. T RANSFER L EARNING ON TWITCH SOCIAL NETWORKS

Original: Neighbourhood based methods such as DeepWalk (Perozzi et al., 2014) are transductive and the function used to create the embedding cannot map nodes that are not connected to the original graph to the latent space.
Revised: Neighbourhood based methods such as DeepWalk (Perozzi et al., 2014) are transductive and the function used to create the embedding cannot map nodes that are not connected to the original graph to the latent space.

Original: However, vanilla MUSAE and AE are inductive and can easily map nodes to the embedding space if the attributes across the source and target graph are shared.
Revised: However, vanilla MUSAE and AE are inductive and can easily map nodes to the embedding space if the attributes across the source and target graph are shared.

Original: This also means that supervised models trained on the embedding of a source graph are transferable.
Revised: This also means that supervised models trained on the embedding of a source graph are transferable.

Original: Importantly those attributed embedding methods such as AANE or ASNE that explicitly use the graph are unable to do this transfer. 
Revised: Importantly those attributed embedding methods such as AANE or ASNE that explicitly use the graph are unable to do this transfer. [Table] [Figure] o

Revised: [Figure]

Original: Using the disjoint Twitch country level social networks (inter country edges are not present)
Revised: Using the disjoint Twitch country level social networks (inter country edges are not present)

Original: we did a transfer learning experiment.
Revised: we did a transfer learning experiment.

Original: First, we learn an embedding function given the social network from a country with the standard parameter settings.
Revised: First, we learn an embedding function given the social network from a country with the standard parameter settings.

Original: Second, we train regularized logistic regression on the embedding to predict whether the Twitch user streams explicit content.
Revised: Second, we train regularized logistic regression on the embedding to predict whether the Twitch user streams explicit content.

Original: Third, using the embedding function we map the target graph to the embedding space.
Revised: Third, using the embedding function we map the target graph to the embedding space.

Original: Fourth, we use the logistic model to predict the node labels on the target graph.
Revised: Fourth, we use the logistic model to predict the node labels on the target graph.

Original: We evaluate the performance by the micro F 1 score based on 10 experimental repetitions.
Revised: We evaluate the performance by the micro F 1 score based on 10 experimental repetitions.

Original: These averages with standard error bars are plotted for the
Revised: These averages with standard error bars are plotted for the

Original: Twitch Germany, England and Spain datasets as target graphs on Figure 3.
Revised: Twitch Germany, England and Spain datasets as target graphs on Figure 3.

Original: We added additional results with France, Portugal and Russia being the target country in Appendix H as Table 5.[Figure] M i c r o F 1 S c
Revised: We added additional results with France, Portugal and Russia being the target country in Appendix H as Table 5.

Original: o r e

Original:  MUSAE [Figure] to DE users DE ES
Revised: These results support that MUSAE and AE create features that are transferable across graphs that share vertex features.

Original: FR

Original: PT RU

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: M i c r o

Original: F 1 S c

Original: o r e

Original: [Figure]

Original: EN

Original: [Figure]

Original: DE EN FR

Original: PT RU

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: [Figure]

Original: M i c r o

Original: F 1 S c

Original: o r e

Original: [Figure]

Original: ES users

Original: EN ES

Original: FR

Original: PT RU0 .0 .

Original: [Figure] M i c r o F 1 S c o r e
Revised: For example, based on a comparison to non transfer-learning results we ﬁnd that the transfer between the German and English user graphs is effective in terms of micro F 1 score.

Original: AE transfer [Figure] DE users [Figure]
Revised: Transfer from English users to German ones considerably improves performance, and the other way around there is a little gain.

Original: [Figure]

Original: AE

Original: [Figure]

Original: AE [Figure] These results support that MUSAE and AE create features that are transferable across disjoint graphs that share vertex features. Moreover, the transfer of the downstream model is also possible across datasets.
Revised: We also see that the upstream and downstream models that we trained on graphs with more vertices transfer well while transfer to the small ones is generally poor – most of the times worse than random guessing.

Original: There is no clear evidence that either MUSAE or AE gives better results on this speciﬁc problem.
Revised: There is no clear evidence that either MUSAE or AE gives better results on this speciﬁc problem.

Original: We also see some evidence that upstream and downstream models that we trained on graphs with more vertices transfer better.

Original: 5. R EGRESSION ON W IKIPEDIA G RAPHS
Revised: 5. R EGRESSION ON W IKIPEDIA G RAPHS

Original: We created embeddings of the Wikipedia webgraphs with all of our methods and the unsupervised baselines.
Revised: We created embeddings of the Wikipedia webgraphs with all of our methods and the unsupervised baselines.

Original: Using a 80% train - 20% test split we predict the log of average trafﬁc for each pageusing an elastic net model.
Revised: Using a 80% train - 20% test split we predict the log of average trafﬁc for each page an elastic net model.

Original: The hyperparameters of the downstream model are available in Appendix D.
Revised: The hyperparameters of the downstream model are available in Appendix D.

Original: In Table 7 of Appendix I we report average test R 2 and standard error of the predictive performance over 100 seeded train-test splits.
Revised: In Table 7 of Appendix I we report average test R 2 and standard error of the predictive performance over 100 seeded train-test splits.

Original: Our key observation are: (i) that MUSAE outperforms all
Revised: Our key observation are: (i) that MUSAE outperforms all

Original: benchmark neighbourhood preserving and attributed node embedding methods, with the strongest MUSAE variant outperforming the best baseline between 2.05% and 10.03% (test R 2 ); (ii) that MUSAE signiﬁcantly outperforms AE by between 2.49% and 21.64% (test R 2 ); and (iii) the beneﬁt of using the vertices as features (ego augmented model) can improve the performance of embeddings, but appears to be dataset speciﬁc phenomenon.
Revised: benchmark neighbourhood preserving and attributed node embedding methods, with the strongest MUSAE variant outperforming the best baseline between 2.05% and 10.03% (test R 2 ); (ii) that MUSAE signiﬁcantly outperforms AE by between 2.49% and 21.64% (test R 2 ); and (iii) the beneﬁt of using the vertices as features (ego augmented model) can improve the performance of embeddings, but appears to be dataset speciﬁc phenomenon.

Original: 5. L INK PREDICTION ON WEB GRAPHS AND SOCIAL NETWORKS
Revised: 5. L INK PREDICTION ON WEB GRAPHS AND SOCIAL NETWORKS

Original: The ﬁnal series of experiments dedicated to the representation quality is about link prediction.
Revised: The ﬁnal series of experiments dedicated to the representation quality is about link prediction.

Original: We carried out an attenuated graph embedding trial to predict the removed edges from the graph.
Revised: We carried out an attenuated graph embedding trial to predict the removed edges from the graph.

Original: First, we randomly removed 50% of edges while the connectivity of the graph was not changed.
Revised: First, we randomly removed 50% of edges while the connectivity of the graph was not changed.

Original: Second, an embedding is created from the attenuated graph.
Revised: Second, an embedding is created from the attenuated graph.

Original: Third, we calculate features for the removed edges and the same number of randomly selected pairs of nodes (negative candidates) with binary operators to create d -dimensional edge features.
Revised: Third, we calculate features for the removed edges and the same number of randomly selected pairs of nodes (negative candidates) with binary operators to create d -dimensional edge features.

Original: We use the binary operators applied by Grover &
Revised: We use the binary operators applied by Grover &

Original: Leskovec (2016).
Revised: Leskovec (2016).

Original: Speciﬁcally, we calculated the average, element-wise product, element-wise lnorm and the element-wise l 2 norm of vectors.
Revised: Speciﬁcally, we calculated the average, element-wise product, element-wise lnorm and the element-wise l 2 norm of vectors.

Original: Finally, we created a 100 seeded 80% train - 20% test splits and used logistic regression to predict whether an edge exists.
Revised: Finally, we created a 100 seeded 80% train - 20% test splits and used logistic regression to predict whether an edge exists.

Original: We compared to attributed and neighbourhood based embedding methods and average AUC scores are presented in Tables 8 and 9 of Appendix J. Our results show that Walklets (Perozzi et al., 2017) the multi-scale neighbourhood based embedding method materially outperforms every other method on most of the datasets and attributed embedding methods generally do poorly in terms of AUC compared to neighbourhood based ones.
Revised: We compared to attributed and neighbourhood based embedding methods and average AUC scores are presented in Tables 8 and 9 of Appendix J. Our results show that Walklets (Perozzi et al., 2017) the multi-scale neighbourhood based embedding method materially outperforms every other method on most of the datasets and attributed embedding methods generally do poorly in terms of AUC compared to neighbourhood based ones.

Original: 5. S CALABILITY
Revised: 5. S CALABILITY

Original: In order to show the efﬁcacy of our algorithms we run a series of experiments on synthetic graphs where we are able to manipulate the input size.
Revised: In order to show the efﬁcacy of our algorithms we run a series of experiments on synthetic graphs where we are able to manipulate the input size.

Original: Speciﬁcally, we look at the effect of changing the number of vertices and features per vertex.
Revised: Speciﬁcally, we look at the effect of changing the number of vertices and features per vertex.

Original: Our detailed experimental setup was as follows.
Revised: Our detailed experimental setup was as follows.

Original: Each point in Figure 4 is the mean runtime obtained from 100 experimental runs on Erdos-Renyi graphs.
Revised: Each point in Figure 4 is the mean runtime obtained from 100 experimental runs on Erdos-Renyi graphs.

Original: The base graph that we manipulated had 2 11 nodes, 2 3 edges and the same number of unique features per node uniformly selected from a feature set of 2 11 .
Revised: The base graph that we manipulated had 2 11 nodes, 2 3 edges and the same number of unique features per node uniformly selected from a feature set of 2 11 .

Original: Our experimental settings were the same as the ones described in Appendix C except for the number of epochs.
Revised: Our experimental settings were the same as the ones described in Appendix C except for the number of epochs.

Original: We only did a single training epoch with asynchronous gradient descent on each graph.
Revised: We only did a single training epoch with asynchronous gradient descent on each graph.

Original: We tested the runtime with 1, 2 andcores and included a dashed line as the linear runtime reference in each subﬁgure.
Revised: We tested the runtime with 1, 2 andcores and included a dashed line as the linear runtime reference in each subﬁgure.

Original: [Figure]
Revised: [Figure]

Original: We observe that doubling the average number of features per vertex doubles the runtime of AE and
Revised: We observe that doubling the average number of features per vertex doubles the runtime of AE and

Original: MUSAE .
Revised: MUSAE .

Original: Moreover, the number of cores used during the optimization does not decrease the runtime when the number of unique features per vertex compared to the cardinality of the feature set is large.
Revised: Moreover, the number of cores used during the optimization does not decrease the runtime when the number of unique features per vertex compared to the cardinality of the feature set is large.

Original: When we look at the change in the vertex set size we also see a linear behaviour.
Revised: When we look at the change in the vertex set size we also see a linear behaviour.

Original: Doubling the input size simply results in a doubled optimization runtime.
Revised: Doubling the input size simply results in a doubled optimization runtime.

Original: In addition, if one interpolates linearly from these results it comes that a network with 1 million nodes, 8 edges per node, 8 unique features per node can be embedded with MUSAE on commodity hardware in less than 5 hours.
Revised: In addition, if one interpolates linearly from these results it comes that a network with 1 million nodes, 8 edges per node, 8 unique features per node can be embedded with MUSAE on commodity hardware in less than 5 hours.

Original: This interpolation assumes that the standard parameter settings proposed in Appendix C and 4 cores were used for optimization.
Revised: This interpolation assumes that the standard parameter settings proposed in Appendix C and 4 cores were used for optimization.

Original:  D ISCUSSION AND C ONCLUSION
Revised:  D ISCUSSION AND C ONCLUSION

Original: We investigated attributed node embedding and proposes efﬁcient pooled ( AE ) and multi-scale ( MUSAE ) attributed node embedding algorithms with linear runtime.
Revised: We investigated attributed node embedding and proposes efﬁcient pooled ( AE ) and multi-scale ( MUSAE ) attributed node embedding algorithms with linear runtime.

Original: We proved that these algorithms implicitly factorize probability matrices of features appearing in the neighbourhood of nodes.
Revised: We proved that these algorithms implicitly factorize probability matrices of features appearing in the neighbourhood of nodes.

Original: Two widely used neighbourhood preserving node embedding methods Perozzi et al.
Revised: Two widely used neighbourhood preserving node embedding methods Perozzi et al.

Original: (2014; 2017) are in fact simpliﬁed cases of our models.
Revised: (2014; 2017) are in fact simpliﬁed cases of our models.

Original: On several datasets (Wikipedia, Facebook, Github, and citation networks) we found that representations learned by our methods, in particular MUSAE , outperform neighbourhood based node embedding methods (Perozzi et al.
Revised: On several datasets (Wikipedia, Facebook, Github, and citation networks) we found that representations learned by our methods, in particular MUSAE , outperform neighbourhood based node embedding methods (Perozzi et al.

Original: Grover & Leskovec (2016)), multi-scale algorithms (Tang et al.
Revised: Grover & Leskovec (2016)), multi-scale algorithms (Tang et al.

Original: Perozzi et al.
Revised: Perozzi et al.

Original: and recently proposed attributed node embedding procedures (Yang et al.
Revised: and recently proposed attributed node embedding procedures (Yang et al.

Original: ; Liao et al.
Revised: ; Liao et al.

Original: ; Huang et al.
Revised: ; Huang et al.

Original: Yang et al.
Revised: Yang et al.

Original: ; Yang & Yang (2018)).
Revised: ; Yang & Yang (2018)).

Original: Our proposed embedding models are differentiated from other methods in that they encode feature information from higher order neighborhoods.
Revised: Our proposed embedding models are differentiated from other methods in that they encode feature information from higher order neighborhoods.

Original: The most similar previous model BANE (Yang et al., 2018)
Revised: The most similar previous model BANE (Yang et al., 2018)

Original: encodes node attributes from higher order neighbourhoods but has non-linear runtime complexity and the product of adjacency matrix power and feature matrix is decomposed explicitly.
Revised: encodes node attributes from higher order neighbourhoods but has non-linear runtime complexity and the product of adjacency matrix power and feature matrix is decomposed explicitly.

Original: A P ROOFS
Revised: A P ROOFS

Original: Lemma 1.
Revised: Lemma 1.

Original: The empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps (i) after; or (ii) before node v ∈ V , as given by:
Revised: The empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps (i) after; or (ii) before node v ∈ V , as given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: The proof is analogous to that given for Theorem 2.1 in Qiu et al.
Revised: The proof is analogous to that given for Theorem 2.1 in Qiu et al.

Original: We show that the computed statistics correspond to sequences of random variables with ﬁnite expectation, bounded variance and covariances that tend to zero as the separation between variables within the sequence tends to inﬁnity.
Revised: We show that the computed statistics correspond to sequences of random variables with ﬁnite expectation, bounded variance and covariances that tend to zero as the separation between variables within the sequence tends to inﬁnity.

Original: The Weak Law of Large Numbers (S.N.Bernstein) then guarantees that the sample mean converges to the expectation of the random variable.
Revised: The Weak Law of Large Numbers (S.N.Bernstein) then guarantees that the sample mean converges to the expectation of the random variable.

Original: We ﬁrst consider the special case n = 1 , i e. we have a single sequence w 1 , ..., w l generated by a random walk (see Algorithm 1).
Revised: We ﬁrst consider the special case n = 1 , i.e. we have a single sequence w 1 , ..., w l generated by a random walk (see Algorithm 1).

Original: For a particular node-feature pair ( w, f ), we let Y i , i ∈ { 1 , ..., l − t } , be the indicator function for the event w i = w and f ∈ F i + r .
Revised: For a particular node-feature pair ( w, f ), we let Y i , i ∈ { 1 , ..., l − t } , be the indicator function for the event w i = w and f ∈ F i + r .

Original: Thus, we have:
Revised: Thus, we have:

Original: [Equation]
Revised: [Equation]

Original: the sample average of the Y i s. We also have:
Revised: the sample average of the Y i s. We also have:

Original: [Equation]
Revised: [Equation]

Original: for j > i + r .
Revised: for j > i + r .

Original: This allows us to compute the covariance:
Revised: This allows us to compute the covariance:

Original: [Equation]
Revised: [Equation]

Original: where 1 is a vector of ones.
Revised: where 1 is a vector of ones.

Original: The difference term (indicated) tends to zero as j − i → ∞ since then p ( w j = w | w i + r ) tends to the stationary distribution p ( w ) =
Revised: The difference term (indicated) tends to zero as j − i → ∞ since then p ( w j = w | w i + r ) tends to the stationary distribution p ( w ) =

Original: deg ( w ) c , regardless of w i + r .
Revised: deg ( w ) c , regardless of w i + r .

Original: Thus, applying the Weak Law of Large Numbers, the sample average converges in probability to the expected value, i.e.:
Revised: Thus, applying the Weak Law of Large Numbers, the sample average converges in probability to the expected value, i.e.:

Original: [Equation]
Revised: [Equation]

Original: A similar argument applies to #( w,f ) ← r | D ← r | , with expectation term 1 c ( F (cid:62) DP r ) f,w .
Revised: A similar argument applies to #( w,f ) ← r | D ← r | , with expectation term 1 c ( F (cid:62) DP r ) f,w .

Original: In both cases, the argument readily extends to the general setting where n > 1 with suitably deﬁned indicator functions for each of the n random walks (see Qiu et al.
Revised: In both cases, the argument readily extends to the general setting where n > 1 with suitably deﬁned indicator functions for each of the n random walks (see Qiu et al.

Original: Lemma 2.
Revised: Lemma 2.

Original: Empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps either side of node v ∈ V , given by:plim
Revised: Empirical statistics of node-feature pairs obtained from random walks give unbiased estimates of joint probabilities of observing feature f ∈ F r steps either side of node v ∈ V , given by:

Original: [Equation]
Revised: [Equation]

Original: Proof.
Revised: Proof.

Original: [Equation]
Revised: [Equation]

Original: The ﬁnal step follows by symmetry of A , indicating how the Lemma can be extended to directed graphs.
Revised: The ﬁnal step follows by symmetry of A , indicating how the Lemma can be extended to directed graphs.

Original: B D ATASETS AND DESCRIPTIVE STATISTICS
Revised: B D ATASETS AND DESCRIPTIVE STATISTICS

Original: Our method was evaluated on a variety of social networks and web page-page graphs that we collected from openly available API services.
Revised: Our method was evaluated on a variety of social networks and web page-page graphs that we collected from openly available API services.

Original: In Table 1 we described the graphs with widely used statistics with respect to size, diameter, and level of clustering.
Revised: In Table 1 we described the graphs with widely used statistics with respect to size, diameter, and level of clustering.

Original: We also included the average number of features per vertex and unique feature count in the last columns.
Revised: We also included the average number of features per vertex and unique feature count in the last columns.

Original: These datasets are available with the source code of MUSAE and AE at https://github.com/iclr2020/MUSAE .
Revised: These datasets are available with the source code of MUSAE and AE at https://github.com/iclr2020/MUSAE .

Original: [Table]
Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Revised: [Table]

Original: B. F ACEBOOK P AGE -P AGE DATASET
Revised: B. F ACEBOOK P AGE -P AGE DATASET

Original: This webgraph is a page-page graph of veriﬁed Facebook sites.
Revised: This webgraph is a page-page graph of veriﬁed Facebook sites.

Original: Nodes represent ofﬁcial Facebook pages while the links are mutual likes between sites.
Revised: Nodes represent ofﬁcial Facebook pages while the links are mutual likes between sites.

Original: Node features are extracted from the site descriptions that the page owners created to summarize the purpose of the site.
Revised: Node features are extracted from the site descriptions that the page owners created to summarize the purpose of the site.

Original: This graph was collected through the Facebook Graph API in November 2017 and restricted to pages from 4 categories which are deﬁned by Facebook.
Revised: This graph was collected through the Facebook Graph API in November 2017 and restricted to pages from 4 categories which are deﬁned by Facebook.

Original: These categories are: politicians , governmental organizations , television shows and companies .
Revised: These categories are: politicians , governmental organizations , television shows and companies .

Original: As one can see in Table 1 it is a highly clustered graph with a large diameter.
Revised: As one can see in Table 1 it is a highly clustered graph with a large diameter.

Original: The task related to this dataset is multi-class node classiﬁcation for the 4 site categories.
Revised: The task related to this dataset is multi-class node classiﬁcation for the 4 site categories.

Original: B. G IT H UB WEB AND MACHINE LEARNING DEVELOPERS DATASET
Revised: B. G IT H UB WEB AND MACHINE LEARNING DEVELOPERS DATASET

Original: The largest graph used for evaluation is a social network of GitHub developers which we collected from the public API in June 2019.
Revised: The largest graph used for evaluation is a social network of GitHub developers which we collected from the public API in June 2019.

Original: Nodes are developers who have starred at least 10 repositories and edges are mutual follower relationships between them.
Revised: Nodes are developers who have starred at least 10 repositories and edges are mutual follower relationships between them.

Original: The vertex features are extracted based on the location, repositories starred, employer and e-mail address.
Revised: The vertex features are extracted based on the location, repositories starred, employer and e-mail address.

Original: The task related to the graph is binary node classiﬁcation – one has to predict whether the GitHub user is a web or a machine learning developer.
Revised: The task related to the graph is binary node classiﬁcation – one has to predict whether the GitHub user is a web or a machine learning developer.

Original: This target feature was derived from the job title of each user.
Revised: This target feature was derived from the job title of each user.

Original: As the descriptive statistics show in Table 1 this is the largest graph that we use for evaluation with the highest sparsity.
Revised: As the descriptive statistics show in Table 1 this is the largest graph that we use for evaluation with the highest sparsity.

Original: B. W IKIPEDIA DATASETS
Revised: B. W IKIPEDIA DATASETS

Original: The datasets that we use to perform node level regression are Wikipedia page-page networks collected on three speciﬁc topics: chameleons, crocodiles and squirrels.
Revised: The datasets that we use to perform node level regression are Wikipedia page-page networks collected on three speciﬁc topics: chameleons, crocodiles and squirrels.

Original: In these networks nodes are articles from the English Wikipedia collected in December 2018, edges are mutual links that exist between pairs of sites.
Revised: In these networks nodes are articles from the English Wikipedia collected in December 2018, edges are mutual links that exist between pairs of sites.

Original: Node features describe the presence of nouns appearing in the articles.
Revised: Node features describe the presence of nouns appearing in the articles.

Original: For each node we also have the average monthly trafﬁc between October 2017 and November 2018.
Revised: For each node we also have the average monthly trafﬁc between October 2017 and November 2018.

Original: In the regression tasks used for embedding evaluation the logarithm of average trafﬁc is the target variable.
Revised: In the regression tasks used for embedding evaluation the logarithm of average trafﬁc is the target variable.

Original: Table 1 shows that these networks are heterogeneous in terms of size, density, and clustering.
Revised: Table 1 shows that these networks are heterogeneous in terms of size, density, and clustering.

Original: B. T WITCH DATASETS
Revised: B. T WITCH DATASETS

Original: These datasets used for node classiﬁcation and transfer learning are Twitch user-user networks of gamers who stream in a certain language.
Revised: These datasets used for node classiﬁcation and transfer learning are Twitch user-user networks of gamers who stream in a certain language.

Original: Nodes are the users themselves and the links are mutual friendships between them.
Revised: Nodes are the users themselves and the links are mutual friendships between them.

Original: Vertex features are extracted based on the games played and liked, location and streaming habits.
Revised: Vertex features are extracted based on the games played and liked, location and streaming habits.

Original: Datasets share the same set of node features, this makes transfer learning across networks possible.
Revised: Datasets share the same set of node features, this makes transfer learning across networks possible.

Original: These social networks were collected in May 2018.
Revised: These social networks were collected in May 2018.

Original: The supervised task related to these networks is binary node classiﬁcation – one has to predict whether a streamer uses explicit language.
Revised: The supervised task related to these networks is binary node classiﬁcation – one has to predict whether a streamer uses explicit language.

Original: C S TANDARD HYPERPARAMETER SETTINGS OF OUR EMBEDDING MODELS
Revised: C S TANDARD HYPERPARAMETER SETTINGS OF OUR EMBEDDING MODELS

Original: In MUSAE and AE models we have a set of parameters that we use for model evaluation.
Revised: In MUSAE and AE models we have a set of parameters that we use for model evaluation.

Original: Our parameter settings listed in Table 2 are quite similar to the widely used general settings of random walk sampled implicit factorization machines (Perozzi et al., 2014; Grover & Leskovec, 2016;
Revised: Our parameter settings listed in Table 2 are quite similar to the widely used general settings of random walk sampled implicit factorization machines (Perozzi et al., 2014; Grover & Leskovec, 2016;

Original: Ribeiro et al., 2017; Perozzi et al., 2017).
Revised: Ribeiro et al., 2017; Perozzi et al., 2017).

Original: Each of our models is augmented with a Doc2Vec (Mikolov et al., 2013a;b) embedding of node features – this is done such way that the overall dimension is still 128.
Revised: Each of our models is augmented with a Doc2Vec (Mikolov et al., 2013a;b) embedding of node features – this is done such way that the overall dimension is still 128.

Original: [Table]
Revised: [Table]

Original:   D H YPERPARAMETER SETTINGS OF THE DOWNSTREAM MODELS
Revised:  b D H YPERPARAMETER SETTINGS OF THE DOWNSTREAM MODELS

Original: The downstream tasks uses logistic and elastic net regression from Scikit-learn (Pedregosa et al., 2011) for node level classiﬁcation, regression and link prediction.
Revised: The downstream tasks uses logistic and elastic net regression from Scikit-learn (Pedregosa et al., 2011) for node level classiﬁcation, regression and link prediction.

Original: For the evaluation of every embedding model we use the standard settings of the library except for the regularization and norm mixing parameters.
Revised: For the evaluation of every embedding model we use the standard settings of the library except for the regularization and norm mixing parameters.

Original: These are described in Table 3.[Table]
Revised: These are described in Table 3.

Original: E H YPERPARAMETER SETTINGS OF COMPETING UNSUPERVISED EMBEDDING METHODS
Revised: E H YPERPARAMETER SETTINGS OF COMPETING UNSUPERVISED EMBEDDING METHODS

Original: Our purpose was a fair evaluation compared to otherunsupervised neighbourhood based and attributed node embedding procedures.
Revised: Our purpose was a fair evaluation compared to other node embedding procedures.

Original: Because of this each we tried to use hyperparameter settings that give similar expressive power to the competing  methods with respect to target matrix approximation (Perozzi et al., 2014; Grover & Leskovec, 2016; Perozzi et al., 2017) and number of dimensions.
Revised: Because of this each we tried to use hyperparameter settings that give similar expressive power to the competing [Table] methods with respect to target matrix approximation (Perozzi et al., 2014; Grover & Leskovec, 2016; Perozzi et al., 2017) and number of dimensions.

Original: • DeepWalk (Perozzi et al., 2014):
Revised: • DeepWalk (Perozzi et al., 2014):

Original: We used the hyperparameter settings described in Table 2.
Revised: We used the hyperparameter settings described in Table 2.

Original: While the original DeepWalk model uses hierarchical softmax to speed up calculations we used a negative sampling based implementation.
Revised: While the original DeepWalk model uses hierarchical softmax to speed up calculations we used a negative sampling based implementation.

Original: This way DeepWalk can be seen as a special case of Node2Vec (Grover & Leskovec, 2016) when the second-order random walks are equivalent to the ﬁrs-order walks.
Revised: This way DeepWalk can be seen as a special case of Node2Vec (Grover & Leskovec, 2016) when the second-order random walks are equivalent to the ﬁrs-order walks.

Original: • LINE 2 (Tang et al., 2015):
Revised: • LINE 2 (Tang et al., 2015):

Original: We created 64 dimensional embeddings based on ﬁrst and second order proximity and concatenated these together for the downstream tasks.
Revised: We created 64 dimensional embeddings based on ﬁrst and second order proximity and concatenated these together for the downstream tasks.

Original: Other hyperparameters are taken from the original work.
Revised: Other hyperparameters are taken from the original work.

Original: • Node2Vec
Revised: • Node2Vec

Original: (Grover & Leskovec, 2016): Except for the in-out and return parameters that control the second-order random walk behavior we used the hyperparameter settings described in Table 2.
Revised: (Grover & Leskovec, 2016): Except for the in-out and return parameters that control the second-order random walk behavior we used the hyperparameter settings described in Table 2.

Original: These behavior control parameters were tuned with grid search from the { 4 , 2 , 1 , 0 . 5 , 0 .
Revised: These behavior control parameters were tuned with grid search from the { 4 , 2 , 1 , 0 . 5 , 0 .

Original: 25 } set using a train-validation split of 80% − 20% within the training set itself.
Revised: 25 } set using a train-validation split of 80% − 20% within the training set itself.

Original: • Walklets (Perozzi et al., 2017):
Revised: • Walklets (Perozzi et al., 2017):

Original: We used the hyperparameters described in Table 2 except for window size.
Revised: We used the hyperparameters described in Table 2 except for window size.

Original: We set a window size of 4 with individual embedding sizes of 32.
Revised: We set a window size of 4 with individual embedding sizes of 32.

Original: This way the overall number of dimensions of the representation remained the same.
Revised: This way the overall number of dimensions of the representation remained the same.

Original: • The attributed node embedding methods AANE , ASNE , BANE , TADW , TENE all use the hyperparameters described in the respective papers except for the dimension.
Revised: • The attributed node embedding methods AANE , ASNE , BANE , TADW , TENE all use the hyperparameters described in the respective papers except for the dimension.

Original: We parametrized these methods such way that each of the ﬁnal embeddings used in the downstream tasks is 128 dimensional.
Revised: We parametrized these methods such way that each of the ﬁnal embeddings used in the downstream tasks is 128 dimensional.

Original: F H YPERPARAMETER SETTINGS OF COMPETING GRAPH NEURAL NETWORKS
Revised: F H YPERPARAMETER SETTINGS OF COMPETING GRAPH NEURAL NETWORKS

Original: Each model was optimized with the Adam optimizer (Kingma & Ba, 2015) with the standard moving average parameters and the model implementations are sparsity aware modiﬁcations based on
Revised: Each model was optimized with the Adam optimizer (Kingma & Ba, 2015) with the standard moving average parameters and the model implementations are sparsity aware modiﬁcations based on

Original: PyTorch Geometric (Fey & Lenssen, 2019).
Revised: PyTorch Geometric (Fey & Lenssen, 2019).

Original: We needed these modiﬁcations in order to accommodate the large number of vertex features – see the last column in Table 1.
Revised: We needed these modiﬁcations in order to accommodate the large number of vertex features – see the last column in Table 1.

Original: Except for the GAT model (Veliˇckovi´c et al., 2018)
Revised: Except for the GAT model (Veli ˇ ckovi ´ c et al., 2018)

Original: we used ReLU intermediate activation functions (Nair & Hinton, 2010) with a softmax unit in the ﬁnal layer for classiﬁcation.
Revised: we used ReLU intermediate activation functions (Nair & Hinton, 2010) with a softmax unit in the ﬁnal layer for classiﬁcation.

Original: The hyperparameters used for the training and regularization of the neural models are listed in Table 4.
Revised: The hyperparameters used for the training and regularization of the neural models are listed in Table 4.

Original: [Table]
Revised: [Table]

Original: Except for the APPNP model each baseline uses information up to 2-hop neighbourhoods.
Revised: Except for the APPNP model each baseline uses information up to 2-hop neighbourhoods.

Original: The model speciﬁc settings when we needed to deviate from the basic settings which are listed in Table 4 were as follows:
Revised: The model speciﬁc settings when we needed to deviate from the basic settings which are listed in Table 4 were as follows:

Original: • Classical GCN (Kipf & Welling, 2017):
Revised: • Classical GCN (Kipf & Welling, 2017):

Original: We used the standard parameter settings described in this section
Revised: We used the standard parameter settings described in this section.

Original: • GraphSAGE (Hamilton et al., 2017):
Revised: • GraphSAGE (Hamilton et al., 2017):

Original: We utilized a graph convolutional aggregator on the sampled neighbourhoods, samples of 40 nodes per source, andthe standardhyperparameter settings.
Revised: We utilized a graph convolutional aggregator on the sampled neighbourhoods, samples of 40 nodes per source, and standard settings.

Original: • GAT (Veli ˇ ckovi ´ c et al., 2018):
Revised: • GAT (Veliˇckovi´c et al., 2018):

Original: The negative slope parameter of the leaky ReLU function was [-0.
 and-] we applied a single attention head. Besides these, we used the standard hyperparameter settings.
Revised: The negative slope parameter of the leaky ReLU function was [-0.
 and-]0.2, we applied a single attention head, and used the standard hyperparameter settings.

Original: • MixHop (Abu-El-Haija et al., 2019):
Revised: • MixHop (Abu-El-Haija et al., 2019):

Original: We took advantage of the 0 th , 1 st and 2 nd powers of the normalized adjacency matrix with 32 dimensional convolutional ﬁlters for creating the ﬁrst hidden representations.
Revised: We took advantage of the 0 th , 1 st and 2 nd powers of the normalized adjacency matrix with 32 dimensional convolutional ﬁlters for creating the ﬁrst hidden representations.

Original: This was fed to a feed-forward layer to classify the nodes.
Revised: This was fed to a feed-forward layer to classify the nodes.

Original: • ClusterGCN (Chiang et al., 2019): Just as Chiang et al.
Revised: • ClusterGCN (Chiang et al., 2019): Just as Chiang et al.

Original: (2019) did, we used the METIS procedure (Karypis & Kumar, 1998).
Revised: (2019) did, we used the METIS procedure (Karypis & Kumar, 1998).

Original: We clustered the graphs into disjoint clusters, and the number of clusters was the same as the number of node classes (e.g. in case of the Facebook page-page network we created 4 clusters).
Revised: We clustered the graphs into disjoint clusters, and the number of clusters was the same as the number of node classes (e.g. in case of the Facebook page-page network we created 4 clusters).

Original: For training we used the earlier described setup.
Revised: For training we used the earlier described setup.

Original: • APPNP (Klicpera et al., 2019):
Revised: • APPNP (Klicpera et al., 2019):

Original: The top level feed-forward layer had 32 hidden neurons, the teleport probability was set as 0.2 and we used 20 steps for approximate personalized pagerank calculation.
Revised: The top level feed-forward layer had 32 hidden neurons, the teleport probability was set as 0.2 and we used 20 steps for approximate personalized pagerank calculation.

Original: • SGCONV
Revised: • SGCONV

Original: (Wu et al., 2019):
Revised: (Wu et al., 2019):

Original: We used the 2 nd power of the normalized adjacency matrix for training the 
Revised: We used the 2 nd power of the normalized adjacency matrix for training the classiﬁer.

Original: G C LASSIFICATION PERFORMANCE
Revised: G C LASSIFICATION PERFORMANCE

Original: [Table] [Table]
Revised: [Table] [Table]

Original: H A DDITIONAL T RANSFER L EARNING R ESULTS ON THE T WITCH G RAPHS
Revised: H A DDITIONAL T RANSFER L EARNING R ESULTS ON THE T WITCH G RAPHS

Revised: DE EN ES

Original: [Figure]
Revised: PT RU0 .0 .0 .0 . M i c r o F 1 S c o r e

Revised: MUSAE transfer to FR users

Revised: DE EN ES

Revised: FR RU0 .0 .0 .0 .

Revised: M i c r o

Revised: F 1 S c

Revised: o r e

Revised: [Table]

Revised: transfer to PT users

Revised: DE EN ES

Revised: FR

Revised: PT0 .0 .0 .0 .

Revised: M i c r o

Revised: F 1 S c

Revised: o r e

Revised: [Table]

Revised: transfer to RU users

Revised: DE EN

Revised: [Figure]

Revised: PT

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: M i c r o

Revised: F 1 S c

Revised: o r e

Revised: AE transfer to FR users

Revised: DE EN ES

Revised: FR RU

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: [Figure]

Revised: M i c r o

Revised: F 1 S c

Revised: o r e

Revised: AE transfer to PT users

Revised: DE EN ES

Revised: FR

Revised: PT

Revised: [Figure]

Revised: [Figure]

Original: I R EGRESSION RESULTS ON W IKIPEDIA P AGE - PAGE GRAPHS
Revised: M i c r o F 1 S c o r e AE transfer to RU users

Original: [Figure]
Revised: [Figure]

Revised: I R EGRESSION RESULTS ON W IKIPEDIA P AGE - PAGE GRAPHS

Original: [Table]
Revised: [Table]

Original: J L INK PREDICTION RESULTS ON S OCIAL AND WEB NETWORKS
Revised: J L INK PREDICTION RESULTS ON S OCIAL AND WEB NETWORKS

Original: [Table] [Table]
Revised: [Table] [Table]