Original: G RAPH A UTO -E NCODER V IA N EIGHBORHOOD W ASSERSTEIN R ECONSTRUCTION
Revised: G RAPH A UTO -E NCODER V IA N EIGHBORHOOD W ASSERSTEIN R ECONSTRUCTION

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph neural networks (GNNs) have drawn signiﬁcant research attention recently, mostly under the setting of semi-supervised learning.
Revised: Graph neural networks (GNNs) have drawn signiﬁcant research attention recently, mostly under the setting of semi-supervised learning.

Original: When task-agnostic representations are preferred or supervision is simply unavailable, the auto-encoder framework comes in handy with a natural graph reconstruction objective for unsupervised GNN training.
Revised: When task-agnostic representations are preferred or supervision is simply unavailable, the auto-encoder framework comes in handy with a natural graph reconstruction objective for unsupervised GNN training.

Original: However, existing graph auto-encoders are designed to reconstruct the direct links, so GNNs trained in this way are only optimized towards proximity-oriented graph mining tasks, and will fall short when the topological structures matter.
Revised: However, existing graph auto-encoders are designed to reconstruct the direct links, so GNNs trained in this way are only optimized towards proximity-oriented graph mining tasks, and will fall short when the topological structures matter.

Original: In this work, we revisit the graph encoding process of GNNs which essentially learns to encode the neighborhood information of each node into an embedding vector, and propose a novel graph decoder to reconstruct the entire neighborhood information regarding both proximity and structure via Neighborhood Wasserstein Reconstruction (NWR).
Revised: In this work, we revisit the graph encoding process of GNNs which essentially learns to encode the neighborhood information of each node into an embedding vector, and propose a novel graph decoder to reconstruct the entire neighborhood information regarding both proximity and structure via Neighborhood Wasserstein Reconstruction (NWR).

Original: Speciﬁcally, from the GNN embedding of each node, NWR jointly predicts its node degree and neighbor feature distribution, where the distribution prediction adopts optimal-transport loss based on the Wasserstein distance.
Revised: Speciﬁcally, from the GNN embedding of each node, NWR jointly predicts its node degree and neighbor feature distribution, where the distribution prediction adopts optimal-transport loss based on the Wasserstein distance.

Original: Extensive experiments on both synthetic and real-world network datasets show that the unsupervised node representations learned with
Revised: Extensive experiments on both synthetic and real-world network datasets show that the unsupervised node representations learned with

Original: NWR have much more advantageous in structure-oriented graph mining tasks, while also achieving competitive performance in proximity-oriented ones.
Revised: NWR have much more advantageous in structure-oriented graph mining tasks, while also achieving competitive performance in proximity-oriented ones.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Network/Graph representation learning (a.k.a.
Revised: Network/Graph representation learning (a.k.a.

Original: embedding) aims to preserve the high-dimensional complex graph information involving node features and link structures in a low-dimensional embedding space, which requires effective feature selection and dimension reduction (Hamilton et al., 2017b).
Revised: embedding) aims to preserve the high-dimensional complex graph information involving node features and link structures in a low-dimensional embedding space, which requires effective feature selection and dimension reduction (Hamilton et al., 2017b).

Original: Graph neural networks (GNNs) have done great jobs to this end, but most of them rely on node labels from speciﬁc downstream tasks to be trained in a semi-supervised fashion (Kipf &
Revised: Graph neural networks (GNNs) have done great jobs to this end, but most of them rely on node labels from speciﬁc downstream tasks to be trained in a semi-supervised fashion (Kipf &

Original: Welling, 2017; Hamilton et al., 2017a; Wu et al., 2019;
Revised: Welling, 2017; Hamilton et al., 2017a; Wu et al., 2019;

Original: Veli ˇ ckovi ´ c et al., 2018a; Klicpera et al., 2019).
Revised: Veli ˇ ckovi ´ c et al., 2018a; Klicpera et al., 2019).

Original: However, similar to other domains, unsupervised representation learning is preferred in many cases, not only because labeled data is not always available (Hu et al., 2020; Xie et al., 2021), but also task-agnostic representations can better transfer and generalize among different scenarios (Erhan et al., 2010; Bengio, 2012;
Revised: However, similar to other domains, unsupervised representation learning is preferred in many cases, not only because labeled data is not always available (Hu et al., 2020; Xie et al., 2021), but also task-agnostic representations can better transfer and generalize among different scenarios (Erhan et al., 2010; Bengio, 2012;

Original: Radford et al., 2016).
Revised: Radford et al., 2016).

Original: To train GNNs in an unsupervised fashion, the classic auto-encoder framework (Baldi, 2012; Goodfellow et al., 2016) provides a natural solution and has been widely explored such as the prominent work (V)GAE (Kipf & Welling, 2016).
Revised: To train GNNs in an unsupervised fashion, the classic auto-encoder framework (Baldi, 2012; Goodfellow et al., 2016) provides a natural solution and has been widely explored such as the prominent work (V)GAE (Kipf & Welling, 2016).

Original: Speciﬁcally, classic auto-encoders aim to decode from the low-dimensional representations information in the entire receptive ﬁeld of the neural networks.
Revised: Speciﬁcally, classic auto-encoders aim to decode from the low-dimensional representations information in the entire receptive ﬁeld of the neural networks.

Original: For
Revised: For

Original: GNNs, the receptive ﬁeld of a node representation is its entire neighborhood.
Revised: GNNs, the receptive ﬁeld of a node representation is its entire neighborhood.

Original: However, existing graph auto-encoders appear away from such a motivation and are designed to merely decode the direct links between the node pairs by minimizing a link reconstruction loss.
Revised: However, existing graph auto-encoders appear away from such a motivation and are designed to merely decode the direct links between the node pairs by minimizing a link reconstruction loss.

Original: The fundamental difﬁculty to reconstruct the entire receptive ﬁelds of GNNs is due to the non-trivial design of a reconstruction loss on the irregular graph structures.
Revised: The fundamental difﬁculty to reconstruct the entire receptive ﬁelds of GNNs is due to the non-trivial design of a reconstruction loss on the irregular graph structures.

Original: Unfortunately, the over-simpliﬁcation into link reconstruction makes the learned node representations drop much information and thus provides undesired performance in many downstream tasks.
Revised: Unfortunately, the over-simpliﬁcation into link reconstruction makes the learned node representations drop much information and thus provides undesired performance in many downstream tasks.

Original: Take Figure 1 as an example, where different types of information are mixed in a graph ( e.g. , proximity and structure information as illustrated in Figure 5 in Appendix B).
Revised: Take Figure 1 as an example, where different types of information are mixed in a graph ( e.g. , proximity and structure information as illustrated in Figure 5 in Appendix B).

Original: The node representations learned by existing graph auto-encoders such as GAE (Kipf & Welling, 2016) are driven too much to be similar on linked nodes due to their simple link reconstruction objective, and thus fail to distinguish node pairs like (2, 4) and (3, 5) in the cliques, though they clearly have different structural roles and node features.
Revised: The node representations learned by existing graph auto-encoders such as GAE (Kipf & Welling, 2016) are driven too much to be similar on linked nodes due to their simple link reconstruction objective, and thus fail to distinguish node pairs like (2, 4) and (3, 5) in the cliques, though they clearly have different structural roles and node features.

Original: On the other hand, structure-oriented embedding models like GraphWave (Donnat et al., 2018) cannot consider node features and spatial proximity, and thus fail to distinguish node pairs like (0, 1), (2, 4) and (3, 5) though they have different features, as well as (2, 5) and (3, 4) though they are further apart.
Revised: On the other hand, structure-oriented embedding models like GraphWave (Donnat et al., 2018) cannot consider node features and spatial proximity, and thus fail to distinguish node pairs like (0, 1), (2, 4) and (3, 5) though they have different features, as well as (2, 5) and (3, 4) though they are further apart.

Original: An ideal unsupervised node representation learning model as we advocate in this work is expected to be task-agnostic and encode as much information as possible of all types in a low-dimensional embedding space.
Revised: An ideal unsupervised node representation learning model as we advocate in this work is expected to be task-agnostic and encode as much information as possible of all types in a low-dimensional embedding space.

Original: In this work, we aim to fundamentally address the above limitations of existing unsupervised node representation learning models by proposing a novel graph auto-encoder framework for unsupervised
Revised: In this work, we aim to fundamentally address the above limitations of existing unsupervised node representation learning models by proposing a novel graph auto-encoder framework for unsupervised

Original: GNN training.
Revised: GNN training.

Original: The new framework is equipped with a powerful decoder that fully reconstructs the information from the entire receptive ﬁeld of a node representation.
Revised: The new framework is equipped with a powerful decoder that fully reconstructs the information from the entire receptive ﬁeld of a node representation.

Original: Our key technical contribution lies in designing a principled and easy-to-compute loss to reconstruct the entire irregular structures of the node neighborhood.
Revised: Our key technical contribution lies in designing a principled and easy-to-compute loss to reconstruct the entire irregular structures of the node neighborhood.

Original: Speciﬁcally, we characterize the decoding procedure as iteratively sampling from a series of probability distributions deﬁned over multi-hop neighbors’ representations obtained through the GNN encoder.
Revised: Speciﬁcally, we characterize the decoding procedure as iteratively sampling from a series of probability distributions deﬁned over multi-hop neighbors’ representations obtained through the GNN encoder.

Original: Then, the reconstruction loss can be decomposed into three parts, for sampling numbers (node degrees), neighbor-representation distributions and node features.
Revised: Then, the reconstruction loss can be decomposed into three parts, for sampling numbers (node degrees), neighbor-representation distributions and node features.

Original: All of these terms are easy to compute but may represent the entire receptive ﬁeld of a node instead of just the linkage information to its direct neighbors.
Revised: All of these terms are easy to compute but may represent the entire receptive ﬁeld of a node instead of just the linkage information to its direct neighbors.

Original: For the most novel and important term, neighborrepresentation distribution reconstruction, we adopt an optimal-transport loss based on Wasserstein distance (Frogner et al., 2015) and thus name this new framework as Neighborhood Wasserstein
Revised: For the most novel and important term, neighborrepresentation distribution reconstruction, we adopt an optimal-transport loss based on Wasserstein distance (Frogner et al., 2015) and thus name this new framework as Neighborhood Wasserstein

Original: Reconstruction Graph Auto-Encoder ( NWR-GAE ).
Revised: Reconstruction Graph Auto-Encoder ( NWR-GAE ).

Original: As also illustrated in Figure 1, NWR-GAE can effectively distinguish all pairs of nodes dissimilar in different perspectives, and concisely reﬂect their similarities in the low-dimensional embedding space.
Revised: As also illustrated in Figure 1, NWR-GAE can effectively distinguish all pairs of nodes dissimilar in different perspectives, and concisely reﬂect their similarities in the low-dimensional embedding space.

Original: We have conducted extensive experiments on four synthetic datasets and nine real-world datasets.
Revised: We have conducted extensive experiments on four synthetic datasets and nine real-world datasets.

Original: Among the real-world datasets, three have proximity-oriented tasks, three have structure-oriented tasks, and three have proximity-structure-mixed tasks.
Revised: Among the real-world datasets, three have proximity-oriented tasks, three have structure-oriented tasks, and three have proximity-structure-mixed tasks.

Original: We can observe signiﬁcant improvements brought by NWR-GAE over the best method among the state-of-the-art baselines on all structureoriented tasks (8.74% to 18.48%) and proximity-structure-mixed tasks (-2.98% to 8.62%), and competitive performance on proximity-oriented tasks (-3.21% to -0.32%).
Revised: We can observe signiﬁcant improvements brought by NWR-GAE over the best method among the state-of-the-art baselines on all structureoriented tasks (8.74% to 18.48%) and proximity-structure-mixed tasks (-2.98% to 8.62%), and competitive performance on proximity-oriented tasks (-3.21% to -0.32%).

Original: In-depth ablation and hyper-parameter studies further consolidate the claimed advantages of NWR-GAE .
Revised: In-depth ablation and hyper-parameter studies further consolidate the claimed advantages of NWR-GAE .

Original:  P RELIMINARIES , M OTIVATIONS & O THER R ELATED W ORKS
Revised:  P RELIMINARIES , M OTIVATIONS & O THER R ELATED W ORKS

Original: In this work, we focus on the auto-encoder framework for unsupervised task-agnostic graph representation learning.
Revised: In this work, we focus on the auto-encoder framework for unsupervised task-agnostic graph representation learning.

Original: The original motivation of auto-encoders is to perform neural-network-based dimension reduction of the data that originally lies in a high-dimensional space (Hinton & Salakhutdinov, 2006).
Revised: The original motivation of auto-encoders is to perform neural-network-based dimension reduction of the data that originally lies in a high-dimensional space (Hinton & Salakhutdinov, 2006).

Original: Speciﬁcally, an auto-encoder consists of two components, an encoder and a decoder.
Revised: Speciﬁcally, an auto-encoder consists of two components, an encoder and a decoder.

Original: The encoder works to compress each data point into a low-dimensional vector representation, while the decoder works to reconstruct the original information from this vector.
Revised: The encoder works to compress each data point into a low-dimensional vector representation, while the decoder works to reconstruct the original information from this vector.

Original: By minimizing the reconstruction error, the encoder automatically converges to a good compressor that allows the low-dimensional representations to capture as much information as possible from the original data.
Revised: By minimizing the reconstruction error, the encoder automatically converges to a good compressor that allows the low-dimensional representations to capture as much information as possible from the original data.

Original: Although the above high-level idea of auto-encoders is clear, when it is applied to graph structured data, the problem becomes challenging.
Revised: Although the above high-level idea of auto-encoders is clear, when it is applied to graph structured data, the problem becomes challenging.

Original: This is because in graph-structured data, information of data 𝑣 𝑎 𝑏 𝑐 𝑒 𝑓 2-hop neighborhood 𝑣 𝑎 𝑏 𝑐 𝑣 𝑐 𝑒 𝑣 𝑎 𝑓 𝑣 𝑒 ℎ (cid:3049)((cid:2870))
Revised: This is because in graph-structured data, information of data 𝑣 𝑎 𝑏 𝑐 𝑒 𝑓 2-hop neighborhood 𝑣 𝑎 𝑏 𝑐 𝑣 𝑐 𝑒 𝑣 𝑎 𝑓 𝑣 𝑒 ℎ (cid:3049)((cid:2870))

Original: Two GNN layers 1-hop neighborhood ℎ (cid:3049)((cid:2870)) obtained via two
Revised: Two GNN layers 1-hop neighborhood ℎ (cid:3049)((cid:2870)) obtained via two

Original: GNN layers has the information from 2-hop neighborhood of 𝑣 . points (nodes to be speciﬁc as most widely studied) correlated
Revised: GNN layers has the information from 2-hop neighborhood of 𝑣 . points (nodes to be speciﬁc as most widely studied) correlated

Original: Without a speciﬁc task need in a priori, the learned low-dimensional representation of a node should carry as much information as possible from not only its own features but also the features of the nodes it connects to (both directly and indirectly).
Revised: Without a speciﬁc task need in a priori, the learned low-dimensional representation of a node should carry as much information as possible from not only its own features but also the features of the nodes it connects to (both directly and indirectly).

Original: This implies that when building auto-encoders for graph-structure data, we expect the node representations to be able to reconstruct all correlated node features.
Revised: This implies that when building auto-encoders for graph-structure data, we expect the node representations to be able to reconstruct all correlated node features.

Original: However, existing graph auto-encoders seem to be away from this motivation.
Revised: However, existing graph auto-encoders seem to be away from this motivation.

Original: Previous prominent works such as unsupervised GraphSAGE (Hamilton et al., 2017a), GAE (Kipf & Welling, 2016), their generative variants such as
Revised: Previous prominent works such as unsupervised GraphSAGE (Hamilton et al., 2017a), GAE (Kipf & Welling, 2016), their generative variants such as

Original: VGAE (Kipf & Welling, 2016), Graphite (Grover et al., 2019), and many others (Pan et al., 2018;
Revised: VGAE (Kipf & Welling, 2016), Graphite (Grover et al., 2019), and many others (Pan et al., 2018;

Original: Shi et al., 2020), use GNNs to encode graph structured data into node representations.
Revised: Shi et al., 2020), use GNNs to encode graph structured data into node representations.

Original: Without exception, they follow the rationale of traditional network embedding techniques (Perozzi et al., 2014; Qiu et al., 2018; Grover & Leskovec, 2016) and adopt link reconstruction in the decoder as the main drive to optimize their GNN encoders.
Revised: Without exception, they follow the rationale of traditional network embedding techniques (Perozzi et al., 2014; Qiu et al., 2018; Grover & Leskovec, 2016) and adopt link reconstruction in the decoder as the main drive to optimize their GNN encoders.

Original: The obtained node representations best record the network linkage information but lose much of other important information, such as local structures, neighbors’ features, etc.
Revised: The obtained node representations best record the network linkage information but lose much of other important information, such as local structures, neighbors’ features, etc.

Original: Hence, these auto-encoders will most likely fail in other tasks such as node classiﬁcations (especially structure-oriented ones as manifested in Figure 1).
Revised: Hence, these auto-encoders will most likely fail in other tasks such as node classiﬁcations (especially structure-oriented ones as manifested in Figure 1).

Original: To better understand this point, we carefully analyze the source of information encoded in each node representation via a GNN.
Revised: To better understand this point, we carefully analyze the source of information encoded in each node representation via a GNN.

Original: Suppose a standard message-passing GNN (Gilmer et al., 2017) is adopted as the encoder, which is a general framework that includes GCN (Kipf & Welling, 2017),
Revised: Suppose a standard message-passing GNN (Gilmer et al., 2017) is adopted as the encoder, which is a general framework that includes GCN (Kipf & Welling, 2017),

Original: GraphSAGE (Hamilton et al., 2017a), GAT (Veli ˇ ckovi ´ c et al., 2018a), GIN (Xu et al., 2019c) and so on.
Revised: GraphSAGE (Hamilton et al., 2017a), GAT (Veli ˇ ckovi ´ c et al., 2018a), GIN (Xu et al., 2019c) and so on.

Original: After k -hop message passing, the source of information encoded in the representation of a node v essentially comes from the k -hop neighborhood of v (Fig. 2).
Revised: After k -hop message passing, the source of information encoded in the representation of a node v essentially comes from the k -hop neighborhood of v (Fig. 2).

Original: Therefore, a good representation of node v should capture the information of features from all nodes in its k -hop neighborhood, which is agnostic to downstream tasks.
Revised: Therefore, a good representation of node v should capture the information of features from all nodes in its k -hop neighborhood, which is agnostic to downstream tasks.

Original: Note that this may not be ideal as nodes out of k -hop neighborhood may also provide useful information, but this is what GNN-based graph auto-encoders can be expected to do due to the architectures of GNN encoders.
Revised: Note that this may not be ideal as nodes out of k -hop neighborhood may also provide useful information, but this is what GNN-based graph auto-encoders can be expected to do due to the architectures of GNN encoders.

Original: This observation motivates our study on a novel graph decoder that can better facilitate the goal of GNN-based graph auto-encoders, based on the neighborhood reconstruction principle that will be formalized in Sec. 3.
Revised: This observation motivates our study on a novel graph decoder that can better facilitate the goal of GNN-based graph auto-encoders, based on the neighborhood reconstruction principle that will be formalized in Sec. 3.

Original: Relation to the InfoMax principle.
Revised: Relation to the InfoMax principle.

Original: Recently, DGI (Veliˇckovi´c et al., 2018b), InfoGraph (Sun et al., 2020) and others (Hu et al., 2020; You et al., 2020; Hassani & Khasahmadi, 2020; Suresh et al., 2021) have used constrasive learning for unsupervised GNN training methods and may capture information beyond the directed links.
Revised: Recently, DGI (Veliˇckovi´c et al., 2018b), InfoGraph (Sun et al., 2020) and others (Hu et al., 2020; You et al., 2020; Hassani & Khasahmadi, 2020; Suresh et al., 2021) have used constrasive learning for unsupervised GNN training methods and may capture information beyond the directed links.

Original: They adopt the rule of mutual information maximization (InfoMax), which essentially works to maximize certain correspondence between the learned representations and the original data.
Revised: They adopt the rule of mutual information maximization (InfoMax), which essentially works to maximize certain correspondence between the learned representations and the original data.

Original: For example, DGI (Veliˇckovi´c et al., 2018b) maximizes the correspondence between a node representation and which graph the node belongs to, but this has no guarantee to reconstruct the structural information of node neighborhoods.
Revised: For example, DGI (Veliˇckovi´c et al., 2018b) maximizes the correspondence between a node representation and which graph the node belongs to, but this has no guarantee to reconstruct the structural information of node neighborhoods.

Original: Recent works even demonstrate that maximizing such correspondence risks capturing only the noisy information that is irrelevant to the downsteam tasks because noisy information itself is sufﬁcient for the models to achieve InfoMax (Tschannen et al., 2020;
Revised: Recent works even demonstrate that maximizing such correspondence risks capturing only the noisy information that is irrelevant to the downsteam tasks because noisy information itself is sufﬁcient for the models to achieve InfoMax (Tschannen et al., 2020;

Original: Suresh et al., 2021), which gets demonstrated again by our experiments.
Revised: Suresh et al., 2021), which gets demonstrated again by our experiments.

Original: Our goal instead is to let node representations not just capture enough information to distinguish nodes but capture as much information as possible to reconstruct the features and structure of the neighborhood.
Revised: Our goal instead is to let node representations not just capture enough information to distinguish nodes but capture as much information as possible to reconstruct the features and structure of the neighborhood.

Original: Optimal-transport (OT) losses.
Revised: Optimal-transport (OT) losses.

Original: Many machine learning problems depend on the characterization of the distance between two probability measures.
Revised: Many machine learning problems depend on the characterization of the distance between two probability measures.

Original: The family f -divergence has the non-continuous issue when the two measures of interest have non-overlapped support (Ali & Silvey, 1966).
Revised: The family f -divergence has the non-continuous issue when the two measures of interest have non-overlapped support (Ali & Silvey, 1966).

Original: Therefore,
Revised: Therefore,

Original: OT losses are often adopted and have shown great success in generative models (Gulrajani et al., 2017) and domain adaptation (Courty et al., 2016).
Revised: OT losses are often adopted and have shown great success in generative models (Gulrajani et al., 2017) and domain adaptation (Courty et al., 2016).

Original: OT-losses have been used to build variational auto-encoders for non-graph data (Tolstikhin et al., 2018; Kolouri et al., 2018;
Revised: OT-losses have been used to build variational auto-encoders for non-graph data (Tolstikhin et al., 2018; Kolouri et al., 2018;

Original: Patrini et al., 2020).
Revised: Patrini et al., 2020).

Original: But one should note the our work is not a graph-data-oriented generalization of these works:
Revised: But one should note the our work is not a graph-data-oriented generalization of these works:

Original: They
Revised: They

Original: [Table]
Revised: [Table]

Original: [Figure]
Revised: [Figure]

Original: use OT losses to characterize the distance between the variational distribution and the data empirical distribution while our model even does not have a variational distribution.
Revised: use OT losses to characterize the distance between the variational distribution and the data empirical distribution while our model even does not have a variational distribution.

Original: Our model may be further improved by being reformulated as a variational autoencoder but we leave it as a future direction.
Revised: Our model may be further improved by being reformulated as a variational autoencoder but we leave it as a future direction.

Original: Here, we give a frequently-used OT loss based on 2-Wasserstein distance that will be used later.
Revised: Here, we give a frequently-used OT loss based on 2-Wasserstein distance that will be used later.

Original: Deﬁnition 2.1.
Revised: Deﬁnition 2.1.

Original: Let P , Q denote two probability distributions with ﬁnite second moment deﬁned on
Revised: Let P , Q denote two probability distributions with ﬁnite second moment deﬁned on

Original: Z ⊆ R m .
Revised: Z ⊆ R m .

Original: The 2 -Wasserstein distance between P and Q deﬁned on Z , Z (cid:48) ⊆
Revised: The 2 -Wasserstein distance between P and Q deﬁned on Z , Z (cid:48) ⊆

Original: R m is the solution to the optimal mass transportation problem with (cid:96) 2 transport cost (Villani, 2008):
Revised: R m is the solution to the optimal mass transportation problem with (cid:96) 2 transport cost (Villani, 2008):

Original: [Equation]
Revised: [Equation]

Original: where Γ( P , Q ) contains all joint distributions of ( Z, Z (cid:48) ) with marginals P and Q respectively.
Revised: where Γ( P , Q ) contains all joint distributions of ( Z, Z (cid:48) ) with marginals P and Q respectively.

Original:  M ETHODS 3. N EIGHBORHOOD R ECONSTRUCTION P RINCIPLE (NRP)
Revised:  M ETHODS 3. N EIGHBORHOOD R ECONSTRUCTION P RINCIPLE (NRP)

Original: Let G =
Revised: Let G =

Original: ( V, E, X ) denote the input graph where V is the node set, E is the edge set and X
Revised: ( V, E, X ) denote the input graph where V is the node set, E is the edge set and X

Original: = { x v | v ∈ V } includes the node features.
Revised: = { x v | v ∈ V } includes the node features.

Original: Given a node v ∈ V , we deﬁne its k -hop neighborhood as the nodes which have shortest paths of length no greater than k from v .
Revised: Given a node v ∈ V , we deﬁne its k -hop neighborhood as the nodes which have shortest paths of length no greater than k from v .

Original: Let N v denote the 1-hop neighborhood of v , and d v denote the degree of node v .
Revised: Let N v denote the 1-hop neighborhood of v , and d v denote the degree of node v .

Original: We are to build an auto-encoder to learn a low-dimensional representation of each node.
Revised: We are to build an auto-encoder to learn a low-dimensional representation of each node.

Original: Speciﬁcally, the auto-encoder will have an encoder φ and a decoder ψ .
Revised: Speciﬁcally, the auto-encoder will have an encoder φ and a decoder ψ .

Original: φ could be any message-passing
Revised: φ could be any message-passing

Original: GNNs (Gilmer et al., 2017).
Revised: GNNs (Gilmer et al., 2017).

Original: The decoder further contains three parts ψ = ( ψ s , ψ p , ψ d ) .
Revised: The decoder further contains three parts ψ = ( ψ s , ψ p , ψ d ) .

Original: The physical meaning of each part will be clear as we introduce them in the later subsections.
Revised: The physical meaning of each part will be clear as we introduce them in the later subsections.

Original: First-hop neighborhood reconstruction.
Revised: First-hop neighborhood reconstruction.

Original: To simplify the exposition, we start from 1 -hop neighborhood reconstruction.
Revised: To simplify the exposition, we start from 1 -hop neighborhood reconstruction.

Original: We initialize the node representation by H (0) based on X .
Revised: We initialize the node representation by H (0) based on X .

Original: For every node v ∈ V , after being encoded by one GNN layer, its node representation h (1) v collects information from h (0) v and its neighbors’ representations H (0) N v = { h (0) u | u ∈ N v } .
Revised: For every node v ∈ V , after being encoded by one GNN layer, its node representation h (1) v collects information from h (0) v and its neighbors’ representations H (0) N v = { h (0) u | u ∈ N v } .

Original: Hence, we consider the following principle that reconstructs the information from both h (0) v and H (0) N v .
Revised: Hence, we consider the following principle that reconstructs the information from both h (0) v and H (0) N v .

Original: Therefore, we have
Revised: Therefore, we have

Original: [Equation]
Revised: [Equation]

Original: where M ( · , · ) deﬁnes the reconstruction loss.
Revised: where M ( · , · ) deﬁnes the reconstruction loss.

Original: M can be split into two parts that measure self-feature reconstruction and neighborhood reconstruction respectively:
Revised: M can be split into two parts that measure self-feature reconstruction and neighborhood reconstruction respectively:

Original: [Equation]
Revised: [Equation]

Original: Note that M s works just as the reconstruction loss of a standard feedforward neural network (FNN)- based auto-encoder, so we adopt
Revised: Note that M s works just as the reconstruction loss of a standard feedforward neural network (FNN)- based auto-encoder, so we adopt

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: M n is much harder to characterize, as it measures the loss to reconstruct a set of features H (0) N v .
Revised: M n is much harder to characterize, as it measures the loss to reconstruct a set of features H (0) N v .

Original: There are two fundamental challenges: First, as in real-world networks the distribution of node degrees is often long-tailed, the sets of node neighbors may have very different sizes.
Revised: There are two fundamental challenges: First, as in real-world networks the distribution of node degrees is often long-tailed, the sets of node neighbors may have very different sizes.

Original: Second, to compare even two equal-sized sets, a matching problem has to be solved, as no ﬁxed orders of elements in the sets can be assumed.
Revised: Second, to compare even two equal-sized sets, a matching problem has to be solved, as no ﬁxed orders of elements in the sets can be assumed.

Original: The complexity to solve a matching problem is high if the size of the set is large.
Revised: The complexity to solve a matching problem is high if the size of the set is large.

Original: We solve the above challenges by decoupling the neighborhood information into a probability distribution and a sampling number.
Revised: We solve the above challenges by decoupling the neighborhood information into a probability distribution and a sampling number.

Original: Speciﬁcally, for node v the neighborhood information is represented as an empirical realization of i.i.d.
Revised: Speciﬁcally, for node v the neighborhood information is represented as an empirical realization of i.i.d.

Original: sampling d v elements from P (0) v , where P (0) v
Revised: sampling d v elements from P (0) v , where P (0) v

Original: [Equation]
Revised: [Equation]

Original: Based on this view, we are able to decompose the reconstruction into the part for the node degree and that for the distribution respectively, where the various sizes of node neighborhoods are handled properly.
Revised: Based on this view, we are able to decompose the reconstruction into the part for the node degree and that for the distribution respectively, where the various sizes of node neighborhoods are handled properly.

Original: Speciﬁcally, we adopt
Revised: Speciﬁcally, we adopt

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: where W 2 ( · , · ) is the 2-Wasserstein distance as deﬁned in Eq.1.
Revised: where W 2 ( · , · ) is the 2-Wasserstein distance as deﬁned in Eq.1.

Original: Plug Eqs.4,5 into Eq.3, we obtain the objective for ﬁrst-hop neighborhood reconstruction.
Revised: Plug Eqs.4,5 into Eq.3, we obtain the objective for ﬁrst-hop neighborhood reconstruction.

Original: Generalizing to k -hop neighborhood reconstruction.
Revised: Generalizing to k -hop neighborhood reconstruction.

Original: Following the above derivation for the ﬁrsthop case, one can similarly generalize the loss for reconstructing ( h ( i − 1) v , H
Revised: Following the above derivation for the ﬁrsthop case, one can similarly generalize the loss for reconstructing ( h ( i − 1) v , H

Original: ( i − 1) N v ) based on h ( i ) v for all 1 ≤ i ≤ k .
Revised: ( i − 1) N v ) based on h ( i ) v for all 1 ≤ i ≤ k .

Original: Then, if we sum such losses over all nodes v ∈ V and all hops 1 ≤ i ≤ k , we may achieve the objective for k -hop neighborhood reconstruction.
Revised: Then, if we sum such losses over all nodes v ∈ V and all hops 1 ≤ i ≤ k , we may achieve the objective for k -hop neighborhood reconstruction.

Original: We may further simplify such an objective.
Revised: We may further simplify such an objective.

Original: Typically, only the ﬁnal layer representation H ( k ) is used as the output dimension-reduced representations.
Revised: Typically, only the ﬁnal layer representation H ( k ) is used as the output dimension-reduced representations.

Original: Too many intermediate hops make the model hard to train and slow to converge.
Revised: Too many intermediate hops make the model hard to train and slow to converge.

Original: Therefore, we adopt a more economic way by merging the multi-step reconstruction:
Revised: Therefore, we adopt a more economic way by merging the multi-step reconstruction:

Original: [Equation]
Revised: [Equation]

Original: That is, we expect h ( k ) v to directly reconstruct H ( i ) N v even if i < k − 1 .
Revised: That is, we expect h ( k ) v to directly reconstruct H ( i ) N v even if i < k − 1 .

Original: Speciﬁcally, for each node v ∈ V , we use the reconstruction loss
Revised: Speciﬁcally, for each node v ∈ V , we use the reconstruction loss

Original: [Equation]
Revised: [Equation]

Original: where ψ s is to decode the initial features, ψ d is to decode the degree and ψ ( i ) p , 0 ≤ i ≤ k − 1 is to decode the i -layer neighbor representation distribution P ( i ) v (: (cid:44) 

Revised: where ψ s is to decode the initial features, ψ d is to decode the degree and ψ ( i ) p , 0 ≤ i ≤ k − 1 is to decode the i -layer neighbor representation distribution P ( i ) v (: (cid:44)

Original: d v (cid:80) u ∈N v δ h
Revised: d v (cid:80) u ∈N v δ h

Original: ( i ) u ) .
Revised: ( i ) u ) .

Original: λ
Revised: λ

Original: s and λ d are non-negative hyperparameters.
Revised: s and λ d are non-negative hyperparameters.

Original: Hence, the entire objective for k -hop neighborhood reconstruction is min φ,ψ (cid:88)
Revised: Hence, the entire objective for k -hop neighborhood reconstruction is min φ,ψ (cid:88)

Original: [Equation]
Revised: [Equation]

Original: where φ = { φ ( i ) | 1 ≤ i ≤ k } include k GNN layers and M (cid:48) is deﬁned in Eq.6.
Revised: where φ = { φ ( i ) | 1 ≤ i ≤ k } include k GNN layers and M (cid:48) is deﬁned in Eq.6.

Original: Remark 3.1 .
Revised: Remark 3.1 .

Original: The loss in Eq.
Revised: The loss in Eq.

Original: 6 does not directly push the k -layer representation h ( k ) v to reconstruct the features of all nodes that are direct neighbors but lie in k -hop neighborhood of v .
Revised: 6 does not directly push the k -layer representation h ( k ) v to reconstruct the features of all nodes that are direct neighbors but lie in k -hop neighborhood of v .

Original: However, by deﬁnition, there exists a shortest path of length no greater than k from v to every node in this k -hop neighborhood.
Revised: However, by deﬁnition, there exists a shortest path of length no greater than k from v to every node in this k -hop neighborhood.

Original: As every two adjacent nodes along the path will introduce at least one term of reconstruction loss in the sum of Eq.
Revised: As every two adjacent nodes along the path will introduce at least one term of reconstruction loss in the sum of Eq.

Original: 6, i.e., the objective in Eq.
Revised: 6, i.e., the objective in Eq.

Original: This guarantees that the optimization in Eq.
Revised: This guarantees that the optimization in Eq.

Original: 7 pushes h ( k ) v to reconstruct the entire k -hop neighborhood of v .
Revised: 7 pushes h ( k ) v to reconstruct the entire k -hop neighborhood of v .

Original: 3. D ECODING D ISTRIBUTIONS — D ECODERS ψ ( i ) p , 0 ≤ i ≤ k −
Revised: 3. D ECODING D ISTRIBUTIONS — D ECODERS ψ ( i ) p , 0 ≤ i ≤ k −

Original: Our NRP essentially represent the node neighborhood H ( i ) N v as a sampling number (node degree d v ) plus a distribution of neighbors’ representations P ( i ) v (Eqs.5,6).
Revised: Our NRP essentially represent the node neighborhood H ( i ) N v as a sampling number (node degree d v ) plus a distribution of neighbors’ representations P ( i ) v (Eqs.5,6).

Original: We adopt Wasserstein distance to characterize the distribution reconstruction loss because P ( i ) v has atomic non-zero measure supports in a continuous space, where the family of f -divergences such as KL-divergence cannot be applied.
Revised: We adopt Wasserstein distance to characterize the distribution reconstruction loss because P ( i ) v has atomic non-zero measure supports in a continuous space, where the family of f -divergences such as KL-divergence cannot be applied.

Original: Maximum mean discrepancy may be applied but it needs to specify a kernel function.
Revised: Maximum mean discrepancy may be applied but it needs to specify a kernel function.

Original: We set the decoded distribution ψ ( i ) p ( h ( k ) v ) as an FNN-based transformation of a Gaussian distribution parameterized by h ( k ) v .
Revised: We set the decoded distribution ψ ( i ) p ( h ( k ) v ) as an FNN-based transformation of a Gaussian distribution parameterized by h ( k ) v .

Original: The rationale of this setting is due to the universal approximation capability of
Revised: The rationale of this setting is due to the universal approximation capability of

Original: FNNs to (approximately) reconstruct any distributions in 1-Wasserstein distance, as formally stated in Theorem 3.1 (See the proof in Appendix A, reproduced by Theorem 2.1 in (Lu & Lu, 2020)).
Revised: FNNs to (approximately) reconstruct any distributions in 1-Wasserstein distance, as formally stated in Theorem 3.1 (See the proof in Appendix A, reproduced by Theorem 2.1 in (Lu & Lu, 2020)).

Original: For a better empirical performance, our case adopts the 2-Wasserstein distance and an FNN with m -dim output instead of the gradient of an FNN with 1-dim outout.
Revised: For a better empirical performance, our case adopts the 2-Wasserstein distance and an FNN with m -dim output instead of the gradient of an FNN with 1-dim outout.

Original: Here, the reparameterization trick needs to be used (Kingma & Welling, 2014), i.e., ψ ( i ) p ( h ( k ) v ) =
Revised: Here, the reparameterization trick needs to be used (Kingma & Welling, 2014), i.e., ψ ( i ) p ( h ( k ) v ) =

Original: FNN ( i ) p ( ξ ) , ξ ∼ N ( µ v , Σ v ) , where µ v = FNN µ ( h ( k ) v ) , Σ v = diag (exp( FNN σ ( h ( k ) v ))) .
Revised: FNN ( i ) p ( ξ ) , ξ ∼ N ( µ v , Σ v ) , where µ v = FNN µ ( h ( k ) v ) , Σ v = diag (exp( FNN σ ( h ( k ) v ))) .

Original: Theorem 3.1.
Revised: Theorem 3.1.

Original: For any (cid:15) > 0 , if the support of the distribution P ( i ) v lies in a bounded space of R m , there exists a FNN u ( · ) :
Revised: For any (cid:15) > 0 , if the support of the distribution P ( i ) v lies in a bounded space of R m , there exists a FNN u ( · ) :

Original: R m → R (and thus its gradient ∇ u ( · ) : R m → R m ) with large enough width and depth (depending on (cid:15) ) such that W 22 ( P ( i ) v , ∇ u ( G )) < (cid:15) where ∇ u ( G ) is the distribution generated via the mapping ∇ u ( ξ ) , ξ ∼ a m -dim non-degenerate Gaussian distribution.
Revised: R m → R (and thus its gradient ∇ u ( · ) : R m → R m ) with large enough width and depth (depending on (cid:15) ) such that W 22 ( P ( i ) v , ∇ u ( G )) < (cid:15) where ∇ u ( G ) is the distribution generated via the mapping ∇ u ( ξ ) , ξ ∼ a m -dim non-degenerate Gaussian distribution.

Original: A further challenge is that Wasserstein distance between P ( i ) v and ψ ( i ) p ( h ( k ) v ) does not have a closed form.
Revised: A further challenge is that Wasserstein distance between P ( i ) v and ψ ( i ) p ( h ( k ) v ) does not have a closed form.

Original: Therefore, we adopt the empirical Wasserstein distance that can provably approximate the population one as stated in Sec.8.4.1 in (Peyré et al., 2019).
Revised: Therefore, we adopt the empirical Wasserstein distance that can provably approximate the population one as stated in Sec.8.4.1 in (Peyré et al., 2019).

Original: For every forward pass, the model will get q sampled nodes N v , denoted by v 1 , v 2 , ..., v q
Revised: For every forward pass, the model will get q sampled nodes N v , denoted by v 1 , v 2 , ..., v q

Original: and thus { h ( i ) v j | 1 ≤ j ≤ q } are q samples from P ( i ) v ; Next, get q samples from N ( µ v , Σ v ) , denoted by ξ 1 , ξ 2 , ..., ξ q , and thus { ˆ h ( i,j ) v =
Revised: and thus { h ( i ) v j | 1 ≤ j ≤ q } are q samples from P ( i ) v ; Next, get q samples from N ( µ v , Σ v ) , denoted by ξ 1 , ξ 2 , ..., ξ q , and thus { ˆ h ( i,j ) v =

Original: FNN ( i ) p ( ξ j ) | 1 ≤ j ≤ q } are q samples from ψ ( i ) p ( h ( k ) v ) ; Adopt the following empirical surrogated loss of (cid:80) k − 1 i =

Revised: FNN ( i ) p ( ξ j ) | 1 ≤ j ≤ q } are q samples from ψ ( i ) p ( h ( k ) v ) ; Adopt the following empirical surrogated loss of (cid:80) k − 1 i =

Original: W 22 ( P ( i ) v , ψ ( i ) p ( h ( k ) v )) in Eq.6,
Revised: W 22 ( P ( i ) v , ψ ( i ) p ( h ( k ) v )) in Eq.6,

Original: [Equation]
Revised: [Equation]

Original: bijective
Revised: bijective

Original: [Equation]
Revised: [Equation]

Original: [ q
Revised: [ q

Original: [Equation]
Revised: [Equation]

Original: q ] .
Revised: q ] .

Original: [Equation]
Revised: [Equation]

Original: Computing the above loss is based on solving a matching problem and needs the Hungarian algorithm with O ( q 3 ) complexity (Jonker & Volgenant, 1987).
Revised: Computing the above loss is based on solving a matching problem and needs the Hungarian algorithm with O ( q 3 ) complexity (Jonker & Volgenant, 1987).

Original: More efﬁcient types of surrogate loss may be adopted, such as Chamfer loss based on greedy approximation (Fan et al., 2017) or Sinkhorn loss based on continuous relaxation (Cuturi, 2013), whose complexities are O ( q 2 ) .
Revised: More efﬁcient types of surrogate loss may be adopted, such as Chamfer loss based on greedy approximation (Fan et al., 2017) or Sinkhorn loss based on continuous relaxation (Cuturi, 2013), whose complexities are O ( q 2 ) .

Original: In our experiments, as q is ﬁxed as a small constant, say 5 , we use Eq.8 based on a Hungarian matching and do not introduce much computational overhead.
Revised: In our experiments, as q is ﬁxed as a small constant, say 5 , we use Eq.8 based on a Hungarian matching and do not introduce much computational overhead.

Original: Although not directly related, we would like to highlight some recent works that use OT losses as distance between two graphs, where Wasserstein distance between the two sets of node embeddings of these two graphs is adopted (Xu et al., 2019b;a).
Revised: Although not directly related, we would like to highlight some recent works that use OT losses as distance between two graphs, where Wasserstein distance between the two sets of node embeddings of these two graphs is adopted (Xu et al., 2019b;a).

Original: Borrowing such a concept, one can view our OT loss as also to measure the distance between the original graph and the decoded graph.
Revised: Borrowing such a concept, one can view our OT loss as also to measure the distance between the original graph and the decoded graph.

Original: 3. F URTHER D ISCUSSION — D ECODERS ψ s AND ψ d
Revised: 3. F URTHER D ISCUSSION — D ECODERS ψ s AND ψ d

Original: The decoder ψ s to reconstruct the initial features h (0) v is an FNN.
Revised: The decoder ψ s to reconstruct the initial features h (0) v is an FNN.

Original: The decoder ψ d to reconstruct the node degree is an FNN plus exponential neuron to make the value non-negative.
Revised: The decoder ψ d to reconstruct the node degree is an FNN plus exponential neuron to make the value non-negative.

Original: [Equation]
Revised: [Equation]

Original: In practice, the original node features X could be very high-dimensional and reconstructing them directly may introduce a lot of noise into the node representations.
Revised: In practice, the original node features X could be very high-dimensional and reconstructing them directly may introduce a lot of noise into the node representations.

Original: Instead, we may ﬁrst map X into a latent space to initialize H (0) .
Revised: Instead, we may ﬁrst map X into a latent space to initialize H (0) .

Original: However, if H (0) is used in both representation learning and reconstruction, it has the risk to collapse to trivial points.
Revised: However, if H (0) is used in both representation learning and reconstruction, it has the risk to collapse to trivial points.

Original: Hence, we normalize H (0) =
Revised: Hence, we normalize H (0) =

Original: { h (0) v | v ∈ V } properly via pair-norm (Zhao & Akoglu, 2020) to avoid this trap as follows
Revised: { h (0) v | v ∈ V } properly via pair-norm (Zhao & Akoglu, 2020) to avoid this trap as follows

Original: [Equation]
Revised: [Equation]

Original: is a learnable parameter matrix
Revised: is a learnable parameter matrix

Original: [Equation]
Revised: [Equation]

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We design our experiments to evaluate NWR-GAE , focusing on the following research questions:
Revised: We design our experiments to evaluate NWR-GAE , focusing on the following research questions:

Original: • RQ1 : How does NWR-GAE perform on structure role based synthetic datasets in comparison to state-of-the-art unsupervised graph embedding baselines?
Revised: • RQ1 : How does NWR-GAE perform on structure role based synthetic datasets in comparison to state-of-the-art unsupervised graph embedding baselines?

Original: • RQ2 : How do NWR-GAE and its ablations compare to the baselines on different types of real-world graph datasets?
Revised: • RQ2 : How do NWR-GAE and its ablations compare to the baselines on different types of real-world graph datasets?

Original: • RQ3 : What are the impacts of the major model parameters including embedding size d and sampling size q on NWR-GAE ?
Revised: • RQ3 : What are the impacts of the major model parameters including embedding size d and sampling size q on NWR-GAE ?

Original: 4. E XPERIMENTAL S ETUP 4.1. D ATASETS
Revised: 4. E XPERIMENTAL S ETUP 4.1. D ATASETS

Original: Synthetic datasets.
Revised: Synthetic datasets.

Original: Following existing works on structure role oriented graph embedding (Henderson et al., 2012; Ribeiro et al., 2017; Donnat et al., 2018), we construct multiple sets of planted structural equivalences graphs , which are generated by regularly placing different basic shapes ( House , Fan ,
Revised: Following existing works on structure role oriented graph embedding (Henderson et al., 2012; Ribeiro et al., 2017; Donnat et al., 2018), we construct multiple sets of planted structural equivalences graphs , which are generated by regularly placing different basic shapes ( House , Fan ,

Original: Star ) along a cycle of certain length.
Revised: Star ) along a cycle of certain length.

Original: We use node degrees as the node features, and generate the structural role labels for nodes in the graphs in the same way as (Donnat et al., 2018).
Revised: We use node degrees as the node features, and generate the structural role labels for nodes in the graphs in the same way as (Donnat et al., 2018).

Original: Real-world graph Datasets.
Revised: Real-world graph Datasets.

Original: We use a total of nine public real-world graph datasets, which roughly belong to three types, one with proximity-oriented (assortative (Liu et al., 2020)) labels, one with structureoriented (disassortative) labels, and one with proximity-structure-mixed labels.
Revised: We use a total of nine public real-world graph datasets, which roughly belong to three types, one with proximity-oriented (assortative (Liu et al., 2020)) labels, one with structureoriented (disassortative) labels, and one with proximity-structure-mixed labels.

Original: Among them, Cora,
Revised: Among them, Cora,

Original: Citeseer, Pubmed are publication networks (Namata et al., 2012); Cornell, Texas, and Wisconsin are school department webpage networks (Pei et al., 2020); Chameleon, Squirrel are page-page networks in Wikipedia (Rozemberczki et al., 2021); Actor is an actor co-ﬁlming network (Tang et al., 2009).
Revised: Citeseer, Pubmed are publication networks (Namata et al., 2012); Cornell, Texas, and Wisconsin are school department webpage networks (Pei et al., 2020); Chameleon, Squirrel are page-page networks in Wikipedia (Rozemberczki et al., 2021); Actor is an actor co-ﬁlming network (Tang et al., 2009).

Original: More details of the datasets are put in Appendix B.
Revised: More details of the datasets are put in Appendix B.

Original: 4.1. B ASELINES
Revised: 4.1. B ASELINES

Original: We compare NWR-GAE with a total of 11 baselines, representing four types of state-of-the-art on unsupervised graph embedding, 1) Random walk based (DeepWalk, node2vec), 2) Structural role based (RoleX, struc2vec, GraphWave), 3) Graph auto-encoder based (GAE, VGAE, ARGVA), 4)
Revised: We compare NWR-GAE with a total of 11 baselines, representing four types of state-of-the-art on unsupervised graph embedding, 1) Random walk based (DeepWalk, node2vec), 2) Structural role based (RoleX, struc2vec, GraphWave), 3) Graph auto-encoder based (GAE, VGAE, ARGVA), 4)

Original: Contrastive
Revised: Contrastive

Original: learning based (DGI, GraphCL, MVGRL).
Revised: learning based (DGI, GraphCL, MVGRL).

Original: • DeepWalk (Perozzi et al., 2014): a two-phase method which generates random walks from all nodes in the graph and then treats all walks as input to a Skipgram language model.
Revised: • DeepWalk (Perozzi et al., 2014): a two-phase method which generates random walks from all nodes in the graph and then treats all walks as input to a Skipgram language model.

Original: • node2vec (Grover & Leskovec, 2016): a method similar to DeepWalk but uses two hyper-paramters p and q to control the behavior of the random walker.
Revised: • node2vec (Grover & Leskovec, 2016): a method similar to DeepWalk but uses two hyper-paramters p and q to control the behavior of the random walker.

Original: • RolX
Revised: • RolX

Original: (Henderson et al., 2012): a structure role discovery method which extracts features and group features of each node, and then generates structure roles via matrix factorization.
Revised: (Henderson et al., 2012): a structure role discovery method which extracts features and group features of each node, and then generates structure roles via matrix factorization.

Original: • struc2vec
Revised: • struc2vec

Original: (Ribeiro et al., 2017): a graph structural embedding method that encodes the structural similarities of nodes and generates structural context for each node.
Revised: (Ribeiro et al., 2017): a graph structural embedding method that encodes the structural similarities of nodes and generates structural context for each node.

Original: • GraphWave (Donnat et al., 2018): a graph structural embedding method that models the diffusion of a spectral graph wavelet and learns embeddings from network diffusion spreads.
Revised: • GraphWave (Donnat et al., 2018): a graph structural embedding method that models the diffusion of a spectral graph wavelet and learns embeddings from network diffusion spreads.

Original: • GAE (Kipf & Welling, 2016): a GCN encoder trained with a link reconstruction decoder.
Revised: • GAE (Kipf & Welling, 2016): a GCN encoder trained with a link reconstruction decoder.

Original: • VGAE (Kipf & Welling, 2016): the variational version of GAE.
Revised: • VGAE (Kipf & Welling, 2016): the variational version of GAE.

Original: • ARGVA
Revised: • ARGVA

Original: Pan et al., 2018): a method which considers topological structure and node content, trained by reconstructing the graph structure.
Revised: Pan et al., 2018): a method which considers topological structure and node content, trained by reconstructing the graph structure.

Original: • DGI (Veliˇckovi´c et al., 2018b): a GCN encoder trained via graph mutual information maximization.
Revised: • DGI (Veliˇckovi´c et al., 2018b): a GCN encoder trained via graph mutual information maximization.

Original: • GraphCL
Revised: • GraphCL

Original: (You et al., 2020): a contrastive learning method via four types of graph augmentation.
Revised: (You et al., 2020): a contrastive learning method via four types of graph augmentation.

Original: • MVGRL (Hassani & Khasahmadi, 2020): a constastive learning method by contrasting encodings from ﬁrst-order neighbors and a graph diffusion.
Revised: • MVGRL (Hassani & Khasahmadi, 2020): a constastive learning method by contrasting encodings from ﬁrst-order neighbors and a graph diffusion.

Original: 4.1. E VALUATION METRICS
Revised: 4.1. E VALUATION METRICS

Original: Following (Donnat et al., 2018)
Revised: Following (Donnat et al., 2018)

Original: we measure the performance of compared algorithms on the synthetic datasets based on agglomerative clustering (with single linkage) with the following metrics
Revised: we measure the performance of compared algorithms on the synthetic datasets based on agglomerative clustering (with single linkage) with the following metrics

Original: • Homogeneity : conditional entropy of ground-truth among predicting clusters.
Revised: • Homogeneity : conditional entropy of ground-truth among predicting clusters.

Original: • Completeness : ratio of nodes with the same ground-truth labels assigned to the same cluster.
Revised: • Completeness : ratio of nodes with the same ground-truth labels assigned to the same cluster.

Original: • Silhouette score : intra-cluster distance vs. inter-cluster distance
Revised: • Silhouette score : intra-cluster distance vs. inter-cluster distance

Original: For the experiments on real-world graph datasets, we learn the graph embeddings with different algorithms and then follow the standard setting of supervised classiﬁcation.
Revised: For the experiments on real-world graph datasets, we learn the graph embeddings with different algorithms and then follow the standard setting of supervised classiﬁcation.

Original: To be consistent across
Revised: To be consistent across

Original: [Table] [Table]
Revised: [Table] [Table]

Original: all datasets in our experiments, we did not follow the standard semi-supervised setting (20 labels per class for training) on Cora, Citeseer and Pubmed, but rather randomly split all datasets with 60% training set, 20% validation set, and 20% testing set, which is a common practice on WebKB and
Revised: all datasets in our experiments, we did not follow the standard semi-supervised setting (20 labels per class for training) on Cora, Citeseer and Pubmed, but rather randomly split all datasets with 60% training set, 20% validation set, and 20% testing set, which is a common practice on WebKB and

Original: Wikipedia network datasets (i.e. Cornell, Texas, Chameleon, etc.)
Revised: Wikipedia network datasets (i.e. Cornell, Texas, Chameleon, etc.)

Original: (Liu et al., 2020; Pei et al., 2020;
Revised: (Liu et al., 2020; Pei et al., 2020;

Original: Ma et al., 2021).
Revised: Ma et al., 2021).

Original: The overall performance is the average performance over 10 random splits.
Revised: The overall performance is the average performance over 10 random splits.

Original: Due to space limitation, we put details of the compared models and hyper-parameter settings in Appendix C.
Revised: Due to space limitation, we put details of the compared models and hyper-parameter settings in Appendix C.

Original: P ERFORMANCE ON S YNTHETIC D ATASETS (RQ1)
Revised: P ERFORMANCE ON S YNTHETIC D ATASETS (RQ1)

Original: We ﬁrst compare NWR-GAE with all baselines on the synthetic datasets.
Revised: We ﬁrst compare NWR-GAE with all baselines on the synthetic datasets.

Original: As can be observed in Table 1, NWR-GAE has the best performance over all compared algorithms regarding all evaluation metrics, corroborating its claimed advantages in capturing the neighborhood structures of nodes on graphs.
Revised: As can be observed in Table 1, NWR-GAE has the best performance over all compared algorithms regarding all evaluation metrics, corroborating its claimed advantages in capturing the neighborhood structures of nodes on graphs.

Original: Speciﬁcally, (1) on the purely symmetrical graphs of House , NWR-GAE achieves almost perfect predictions on par with RolX, GraphWave and GAE; (2) on the asymmetrical graphs of House
Revised: Speciﬁcally, (1) on the purely symmetrical graphs of House , NWR-GAE achieves almost perfect predictions on par with RolX, GraphWave and GAE; (2) on the asymmetrical graphs of House

Original: Perturbed (by randomly adding or deleting edges between nodes), Varied (by placing varied shapes around the base circle such as Fans and Stars ) and Varied perturbed , the performances of all algorithms degrade signiﬁcantly, while NWR-GAE is most robust to such perturbation and variance; (3) among the baselines, proximity oriented methods of DeepWalk and node2vec perform extremely poorly due to the ignorance of structural roles, while structural role oriented methods of RolX, strc2vec and GraphWave are much more competitive; (4) GNN-based baselines of GAE, VGAE and
Revised: Perturbed (by randomly adding or deleting edges between nodes), Varied (by placing varied shapes around the base circle such as Fans and Stars ) and Varied perturbed , the performances of all algorithms degrade signiﬁcantly, while NWR-GAE is most robust to such perturbation and variance; (3) among the baselines, proximity oriented methods of DeepWalk and node2vec perform extremely poorly due to the ignorance of structural roles, while structural role oriented methods of RolX, strc2vec and GraphWave are much more competitive; (4) GNN-based baselines of GAE, VGAE and

Original: DGI still enjoy certain predictive power, due to the structure representativeness of their GIN encoder, but such representativeness is actually impeded by their improper loss functions for structural role identiﬁcation, thus still leading to rather inferior performance.
Revised: DGI still enjoy certain predictive power, due to the structure representativeness of their GIN encoder, but such representativeness is actually impeded by their improper loss functions for structural role identiﬁcation, thus still leading to rather inferior performance.

Original: 4. P ERFORMANCE ON R EAL -W ORLD D ATASETS (RQ2)
Revised: 4. P ERFORMANCE ON R EAL -W ORLD D ATASETS (RQ2)

Original: All 11 baselines are included in the real-world datasets performance comparison.
Revised: All 11 baselines are included in the real-world datasets performance comparison.

Original: As can be observed in Table 2, NWR-GAE achieves signiﬁcantly better results compared to all baselines on all structureoriented and proximity-structure-mixed graph datasets, and maintains close performance to the best baselines of DGI, GraphCL and MVGRL on all proximity-oriented graph datasets.
Revised: As can be observed in Table 2, NWR-GAE achieves signiﬁcantly better results compared to all baselines on all structureoriented and proximity-structure-mixed graph datasets, and maintains close performance to the best baselines of DGI, GraphCL and MVGRL on all proximity-oriented graph datasets.

Original: Note that, most of the strong baselines can only perform well on either of the proximity-oriented or structure-oriented datasets, while NWR-GAE is the only one that performs consistently well on all three types.
Revised: Note that, most of the strong baselines can only perform well on either of the proximity-oriented or structure-oriented datasets, while NWR-GAE is the only one that performs consistently well on all three types.

Original: Such results again indicate the superiority of NWR-GAE in capturing the structural information on graphs, yet without signiﬁcantly sacriﬁcing the capturing of node proximity.
Revised: Such results again indicate the superiority of NWR-GAE in capturing the structural information on graphs, yet without signiﬁcantly sacriﬁcing the capturing of node proximity.

Original: Speciﬁcally, (1) Most GNN-based methods (both auto-encoder based, and contrastive learning based), like GAE, VGAE, ARGVA, DGI and GraphCL only perform well on graphs with proximity-oriented labels due to their strongly proximity-preserving loss functions, and perform rather poorly on most graphs with structure-oriented and mixed labels; MVGRL achieves relatively better but still inferior performance to ours on structure oriented datasets due to its structure aware contrastive training.
Revised: Speciﬁcally, (1) Most GNN-based methods (both auto-encoder based, and contrastive learning based), like GAE, VGAE, ARGVA, DGI and GraphCL only perform well on graphs with proximity-oriented labels due to their strongly proximity-preserving loss functions, and perform rather poorly on most graphs with structure-oriented and mixed labels; MVGRL achieves relatively better but still inferior performance to ours on structure oriented datasets due to its structure aware contrastive training.

Original: (2) the random walk based methods of DeepWalk and node2vec perform well mostly on graphs with proximity-oriented and mixed labels; (3) the structural role oriented methods of RolX, struc2vec and GraphWave perform rather poorly on graphs with proximity-oriented and mixed labels, and sometimes also the structure-oriented ones, due to their incapability of modeling node features.
Revised: (2) the random walk based methods of DeepWalk and node2vec perform well mostly on graphs with proximity-oriented and mixed labels; (3) the structural role oriented methods of RolX, struc2vec and GraphWave perform rather poorly on graphs with proximity-oriented and mixed labels, and sometimes also the structure-oriented ones, due to their incapability of modeling node features.

Original: 4. I N -D EPTH A NALYSIS OF NWR-GAE (RQ3)
Revised: 4. I N -D EPTH A NALYSIS OF NWR-GAE (RQ3)

Original: In Table 2, we also compare several important ablations of NWR-GAE , i.e., the one without degree predictor, the one without feature distribution reconstructor, and the one without Hungarian loss (replaced by a greedy-matching based loss), which all lead to inferior performance on all datasets, demonstrating the effectiveness of our novel model designs.
Revised: In Table 2, we also compare several important ablations of NWR-GAE , i.e., the one without degree predictor, the one without feature distribution reconstructor, and the one without Hungarian loss (replaced by a greedy-matching based loss), which all lead to inferior performance on all datasets, demonstrating the effectiveness of our novel model designs.

Original: In Figure 4, we present results on hyper-parameter and efﬁciency studies of NWR-GAE .
Revised: In Figure 4, we present results on hyper-parameter and efﬁciency studies of NWR-GAE .

Original: Embedding size efﬁciency is an important metric for unsupervised representation learning, which reﬂects the effectiveness in information compression.
Revised: Embedding size efﬁciency is an important metric for unsupervised representation learning, which reﬂects the effectiveness in information compression.

Original: As can be seen in Figure 4 (a), NWR-GAE quickly converges when the embedding size reaches 200-300 on the mid-scale datasets of Chameleon and Squirrel, and maintains good performance as the sizes further increase without overﬁtting.
Revised: As can be seen in Figure 4 (a), NWR-GAE quickly converges when the embedding size reaches 200-300 on the mid-scale datasets of Chameleon and Squirrel, and maintains good performance as the sizes further increase without overﬁtting.

Original: Moreover, we conjecture the neighborhood sampling process might signiﬁcantly impact the performance of NWR-GAE , and such sampling processes and the computations over sampled neighbors incur the most computational time.
Revised: Moreover, we conjecture the neighborhood sampling process might signiﬁcantly impact the performance of NWR-GAE , and such sampling processes and the computations over sampled neighbors incur the most computational time.

Original: However, as we can observe from the training curves of NWR-GAE on Chameleon in Figure 4 (b), the performance of NWR-GAE is robust to the neighborhood sampling sizes, where the larger size of 15 only leads to slightly better performance than the smaller size of 5.
Revised: However, as we can observe from the training curves of NWR-GAE on Chameleon in Figure 4 (b), the performance of NWR-GAE is robust to the neighborhood sampling sizes, where the larger size of 15 only leads to slightly better performance than the smaller size of 5.

Original: NWR-GAE also converges rapidly after 1-2 training epochs, achieving signiﬁcantly better performance compared with the best baseline of DeepWalk and typical baselines like DGI.
Revised: NWR-GAE also converges rapidly after 1-2 training epochs, achieving signiﬁcantly better performance compared with the best baseline of DeepWalk and typical baselines like DGI.

Original: As we can observe from the runtimes of NWR-GAE on Chameleon in Figure 4 (c), smaller neighborhood sampling sizes do lead to signiﬁcantly shorter runtimes, while our runtimes are much better compared with typical baselines like DeepWalk and DGI, especially under the fact that we can converge rather rapidly observed from Figure 4 (b).
Revised: As we can observe from the runtimes of NWR-GAE on Chameleon in Figure 4 (c), smaller neighborhood sampling sizes do lead to signiﬁcantly shorter runtimes, while our runtimes are much better compared with typical baselines like DeepWalk and DGI, especially under the fact that we can converge rather rapidly observed from Figure 4 (b).

Original: Finally, to further understand the effectiveness of our proposed neighborhood Wasserstein reconstruction method, we provide detailed results on how well the generated neighbor features can approximate the real ones in Appendix D.
Revised: Finally, to further understand the effectiveness of our proposed neighborhood Wasserstein reconstruction method, we provide detailed results on how well the generated neighbor features can approximate the real ones in Appendix D.

Original: The results show that the method can indeed recover the entire real neighbor features, and the approximation gets better as we use a larger sampling factor q especially on larger datasets, while the performances on downstream tasks are already satisfactory even with smaller q values, allowing the training to be rather efﬁcient.
Revised: The results show that the method can indeed recover the entire real neighbor features, and the approximation gets better as we use a larger sampling factor q especially on larger datasets, while the performances on downstream tasks are already satisfactory even with smaller q values, allowing the training to be rather efﬁcient.

Original: [Figure]
Revised: [Figure]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: In this work, we address the limitations of existing unsupervised graph representation methods and propose the ﬁrst model that can properly capture the proximity, structure and feature information of nodes in graphs, and discriminatively encode them in the low-dimensional embedding space.
Revised: In this work, we address the limitations of existing unsupervised graph representation methods and propose the ﬁrst model that can properly capture the proximity, structure and feature information of nodes in graphs, and discriminatively encode them in the low-dimensional embedding space.

Original: The model is extensively tested on both synthetic and real-world benchmark datasets, and the results strongly support its claimed advantages.
Revised: The model is extensively tested on both synthetic and real-world benchmark datasets, and the results strongly support its claimed advantages.

Original: Since it is general, efﬁcient, and also conceptually wellunderstood, we believe it to have the potential to serve as the de facto method for unsupervised graph representation learning.
Revised: Since it is general, efﬁcient, and also conceptually wellunderstood, we believe it to have the potential to serve as the de facto method for unsupervised graph representation learning.

Original: In the future, it will be promising to see its applications studied in different domains, as well as its potential issues such as robustness and privacy carefully analyzed.
Revised: In the future, it will be promising to see its applications studied in different domains, as well as its potential issues such as robustness and privacy carefully analyzed.

Original: A P ROOF OF T HEOREM 3. 
Revised: A P ROOF OF T HEOREM 3.

Original: Theorem 3.1 can be obtained by properly revising Theorem 2.1 in (Lu & Lu, 2020).
Revised: Theorem 3.1 can be obtained by properly revising Theorem 2.1 in (Lu & Lu, 2020).

Original: We ﬁrst state
Revised: We ﬁrst state

Original: Theorem 2.1 in (Lu & Lu, 2020).
Revised: Theorem 2.1 in (Lu & Lu, 2020).

Original: Theorem A.1 (Theorem 2.1 in Lu & Lu (2020)) .
Revised: Theorem A.1 (Theorem 2.1 in Lu & Lu (2020)) .

Original: Let P and Q be the target and the source distributions respectively, both deﬁned on R m .
Revised: Let P and Q be the target and the source distributions respectively, both deﬁned on R m .

Original: Assume that E x ∼ P (cid:107) x (cid:107) 3 is bounded and Q is absolutely continuous with respect to the Lebesgue measure.
Revised: Assume that E x ∼ P (cid:107) x (cid:107) 3 is bounded and Q is absolutely continuous with respect to the Lebesgue measure.

Original: It holds that for any given approximation error (cid:15) , setting n =
Revised: It holds that for any given approximation error (cid:15) , setting n =

Original: O ( 1 (cid:15) m ) , there is a fully connected and feed-forward deep neural network u ( · ) of depth
Revised: O ( 1 (cid:15) m ) , there is a fully connected and feed-forward deep neural network u ( · ) of depth

Original: L = (cid:100) log 2 n (cid:101) and width N = 2 L , with d inputs and a single output and with ReLU activation such that W 1 ( P, ∇ u ( Q )) < (cid:15) .
Revised: L = (cid:100) log 2 n (cid:101) and width N = 2 L , with d inputs and a single output and with ReLU activation such that W 1 ( P, ∇ u ( Q )) < (cid:15) .

Original: Here, ∇ u ( · ) is the function R d → R d induced by the gradient of u while ∇ u ( Q ) is the distribution that is generated from the distribution Q through the mapping ∇ u ( · ) .
Revised: Here, ∇ u ( · ) is the function R d → R d induced by the gradient of u while ∇ u ( Q ) is the distribution that is generated from the distribution Q through the mapping ∇ u ( · ) .

Original: To prove Theorem 3.1, we just need to verify the conditions in the above Theorem.
Revised: To prove Theorem 3.1, we just need to verify the conditions in the above Theorem.

Original: We need to show that P = P ( i ) v has a bounded E x ∼ P (cid:107) x (cid:107) 3 , and Q as a non-degenerate m -dim
Revised: We need to show that P = P ( i ) v has a bounded E x ∼ P (cid:107) x (cid:107) 3 , and Q as a non-degenerate m -dim

Original: Gaussian distribution is absolutely continuous with respect to the Lebesgue measure.
Revised: Gaussian distribution is absolutely continuous with respect to the Lebesgue measure.

Original: Also, the original statement is for
Revised: Also, the original statement is for

Original: W 1 ( · , · ) while we use W 2 ( · , · ) .
Revised: W 1 ( · , · ) while we use W 2 ( · , · ) .

Original: We need to show the connection between W 1 ( · , · ) and W 2 ( · , · ) .
Revised: We need to show the connection between W 1 ( · , · ) and W 2 ( · , · ) .

Original: Fortunately, all these aspects are easy to be veriﬁed.
Revised: Fortunately, all these aspects are easy to be veriﬁed.

Original: P = P ( i ) v has a bounded 3-order moment because the support P ( i ) v are in a bounded space of R m .
Revised: P = P ( i ) v has a bounded 3-order moment because the support P ( i ) v are in a bounded space of R m .

Original: Q as a Gaussian distribution is absolutely continuous with respect to the Lebesgue measure, as long as Q is not degenerate.
Revised: Q as a Gaussian distribution is absolutely continuous with respect to the Lebesgue measure, as long as Q is not degenerate.

Original: Lastly we show the connection between W 1 ( · , · ) and W 2 ( · , · ) .
Revised: Lastly we show the connection between W 1 ( · , · ) and W 2 ( · , · ) .

Original: The support P = P ( i ) v is bounded, i.e., δ ∈ supp ( P ) , (cid:107) δ (cid:107) 2 ≤ B .
Revised: The support P = P ( i ) v is bounded, i.e., δ ∈ supp ( P ) , (cid:107) δ (cid:107) 2 ≤ B .

Original: According to Lu & Lu (2020), the Q = ∇ u ( G ) also has bounded support.
Revised: According to Lu & Lu (2020), the Q = ∇ u ( G ) also has bounded support.

Original: Without loss of generality, δ ∈ supp ( Q ) , (cid:107) δ (cid:107) 2 ≤ B .
Revised: Without loss of generality, δ ∈ supp ( Q ) , (cid:107) δ (cid:107) 2 ≤ B .

Original: Then, we may show that
Revised: Then, we may show that

Original: [Equation]
Revised: [Equation]

Original: As B and m are just constant, so we may have W 22 ( P, Q ) =
Revised: As B and m are just constant, so we may have W 22 ( P, Q ) =

Original: O ( (cid:15) ) .
Revised: O ( (cid:15) ) .

Revised: Note that the above ﬁrst inequality is because (cid:107)

Revised: Z − Z (cid:48)

Revised: [Equation]

Original: B A DDITIONAL D ATASET D ETAILS
Revised: 2 ≤ [Equation] Z [Equation]

Revised: 2 +

Revised: [Equation]

Revised: Z (cid:48) (cid:107) 2 = 2 B .

Revised: The second inequality is because

Revised: [Equation]

Revised: (There exists a sequence of measures { γ i } ∞ i =

Revised: 1 achieving the inﬁmum)

Revised: [Equation]

Revised: B A DDITIONAL D ATASET D ETAILS

Original: In this section, we provide some additional important details about the synthetic and real-world network datasets we use in the node classiﬁcation experiments.
Revised: In this section [Equation] we provide some additional important details about the synthetic and real-world network datasets we use in the node classiﬁcation experiments

Original: since they are relatively close in terms of node distances in the global network, whereas A and C are "close in structure" since they have relatively similar local neighborhood structures
Revised: [Equation]

Original: From table B, the node labels shows how the node are classiﬁed.
Revised: From table B [Equation] the node labels shows how the node are classiﬁed.

Original: For example, the node labels of citation networks are academic topics, which is highly related to proximity information.
Revised: For example, the node labels of citation networks are academic topics, which is highly related to proximity information.

Original: But the identity roles of website is more structural oriented since similar identity have similar hyperlinks structure.
Revised: But the identity roles of website is more structural oriented since similar identity have similar hyperlinks structure.

Original: Detailed statistics of the real-world graph datasets are provided in Table4.
Revised: Detailed statistics of the real-world graph datasets are provided in Table

Original:  [Figure][Table]
Revised: [Equation] [Figure]

Original: C D ETAILED SETTINGS OF THE COMPARED MODELS
Revised: C D ETAILED SETTINGS OF THE COMPARED MODELS

Original: DeepWalk : for all datasets, we ran 10 random walks of length 40 and input these walks into a skipgram langauge model.
Revised: DeepWalk : for all datasets, we ran 10 random walks of length 40 and input these walks into a skipgram langauge model.

Original: We implement the DeepWalk method based on Python code (GPLv license) provided by Perozzi et al.
Revised: We implement the DeepWalk method based on Python code (GPLv license) provided by Perozzi et al.

Original: node2vec : similar to DeepWalk, we also ran 10 random walks of length 40 with p and q equal to 0.25.
Revised: node2vec : similar to DeepWalk, we also ran 10 random walks of length 40 with p and q equal to 0.25.

Original: The implementation is provided by Grover & Leskovec (2016).
Revised: The implementation is provided by Grover & Leskovec (2016).

Original: RolX : by using Henderson et al.
Revised: RolX : by using Henderson et al.

Original: (2012)’s code, we set the initial learning rate as 0.01 and ran the algorithm with batch size 32.
Revised: (2012)’s code, we set the initial learning rate as 0.01 and ran the algorithm with batch size 32.

Original: struc2vec : similar to DeepWalk and node2vec, we ran the same number of random walks with walk length 40.
Revised: struc2vec : similar to DeepWalk and node2vec, we ran the same number of random walks with walk length 40.

Original: We set window size 5 as context size for optimization.
Revised: We set window size 5 as context size for optimization.

Original: The reference implementation is provided by Ribeiro et al.
Revised: The reference implementation is provided by Ribeiro et al.

Original: GraphWave : the implementation is under the MIT license and provided by Stanford SNAP lab (Donnat et al., 2018).
Revised: GraphWave : the implementation is under the MIT license and provided by Stanford SNAP lab (Donnat et al., 2018).

Original: All the parameters are automatically determined by the algorithm itself.
Revised: All the parameters are automatically determined by the algorithm itself.

Original: GAE : we use a two layer GCN as graph encoder for node classiﬁcation task and a three layer GIN for synthetic structural identiﬁcation task.
Revised: GAE : we use a two layer GCN as graph encoder for node classiﬁcation task and a three layer GIN for synthetic structural identiﬁcation task.

Original: The decoder hyperparameters are selected by cross-validation, the implementation is based on the code provided by Kipf & Welling (2016)
Revised: The decoder hyperparameters are selected by cross-validation, the implementation is based on the code provided by Kipf & Welling (2016)

Original: VGAE : same encoder structure and similar decoder as GAE, the implementation is also based on Kipf & Welling (2016).
Revised: VGAE : same encoder structure and similar decoder as GAE, the implementation is also based on Kipf & Welling (2016).

Original: ARGVA : same encoder structure as GAE and VGAE.
Revised: ARGVA : same encoder structure as GAE and VGAE.

Original: We use the Tensorﬂow implementation based on the implementation of Pan et al.
Revised: We use the Tensorﬂow implementation based on the implementation of Pan et al.

Original: DGI : same encoder structure as GAE and VGAE, the initial number of epochs is 10000 but may early stop when the algorithm is converged.
Revised: DGI : same encoder structure as GAE and VGAE, the initial number of epochs is 10000 but may early stop when the algorithm is converged.

Original: We use the DGI method based on the implementation of Veliˇckovi´c et al.
Revised: We use the DGI method based on the implementation of Veliˇckovi´c et al.

Original: (2018b).
Revised: (2018b).

Original:  GraphCL : use the default encoder GraphSage and all the four different types of augumentation.
Revised: [Table] [Table] GraphCL : use the default encoder GraphSage and all the four different types of augumentation.

Original: The implementation is based on You et al.
Revised: The implementation is based on You et al.

Original: MVGRL : the same default GCN encoder as GAE, We use the PyTorch implementation of Hassani & Khasahmadi (2020).
Revised: MVGRL : the same default GCN encoder as GAE, We use the PyTorch implementation of Hassani & Khasahmadi (2020).

Original: NWR-GAE : same encoder structure as the above GNN-based methods, learning rate, trade-off weight between degree loss and neighborhood reconstruction loss and sample size q are tuned using grid search.
Revised: NWR-GAE : same encoder structure as the above GNN-based methods, learning rate, trade-off weight between degree loss and neighborhood reconstruction loss and sample size q are tuned using grid search.

Original: Encoder setting.
Revised: Encoder setting.

Original: We used the same GCN encoders for almost all baselines because our contribution in this work is on the decoder and we wanted to be fair and focus on the comparison of different decoders.
Revised: We used the same GCN encoders for almost all baselines because our contribution in this work is on the decoder and we wanted to be fair and focus on the comparison of different decoders.

Original: Speciﬁcally, we use the default encoders in their original work for all baselines that[Table] involve GNN encoders (GraphSAGE (Hamilton et al., 2017a) for GraphCL (You et al., 2020), GCN (Kipf & Welling, 2017) for GAE (Kipf & Welling, 2016), VGAE (Kipf & Welling, 2016), DGI (Veliˇckovi´c et al., 2018b), MVGRL (You et al., 2020)).
Revised: Speciﬁcally, we use the default encoders in their original work for all baselines that involve GNN encoders (GraphSAGE (Hamilton et al., 2017a) for GraphCL (You et al., 2020), GCN (Kipf & Welling, 2017) for GAE (Kipf & Welling, 2016), VGAE (Kipf & Welling, 2016), DGI (Veli ˇ ckovi ´ c et al., 2018b), MVGRL (You et al., 2020)).

Original: The GCN encoders all have the same twolayer architecture, which is the same as we used for NWR-GAE .
Revised: The GCN encoders all have the same twolayer architecture, which is the same as we used for NWR-GAE .

Original: In our experiments, we have actually tried to use other GNNs as the encoders for all methods including NWR-GAE and the baselines, but found GCN to have the best performance.
Revised: In our experiments, we have actually tried to use other GNNs as the encoders for all methods including NWR-GAE and the baselines, but found GCN to have the best performance.

Original: For example, in the following Table C, we provide the results of GAE with the GIN (Xu et al., 2019c) and GraphSage (Hamilton et al., 2017a)
Revised: For example, in the following Table C, we provide the results of GAE with the GIN (Xu et al., 2019c) and GraphSage (Hamilton et al., 2017a)

Original: encoders, which are clearly worse than that with GCN.
Revised: encoders, which are clearly worse than that with GCN.

Original: Hyper-parameter tuning.
Revised: Hyper-parameter tuning.

Original: For all compared models, we performed hyper-parameter selection on learning rate {5e-3, 5e-4, 5e-5, 5e-6, 5e-7} and epoch size {100, 200, 300, 400, 500, 600}.
Revised: For all compared models, we performed hyper-parameter selection on learning rate {5e-3, 5e-4, 5e-5, 5e-6, 5e-7} and epoch size {100, 200, 300, 400, 500, 600}.

Original: For
Revised: For

Original: NWR-GAE , we selected the sample size q from {3, 5, 8, 10}, and the trade-off weight parameters
Revised: NWR-GAE , we selected the sample size q from {3, 5, 8, 10}, and the trade-off weight parameters

Original: λ d , λ s from {10, 1, 1e-1, 1e-2, 1e-3, 1e-4, 1e-5}.
Revised: λ d , λ s from {10, 1, 1e-1, 1e-2, 1e-3, 1e-4, 1e-5}.

Original: Notes on fairness.
Revised: Notes on fairness.

Original: We set all random walk based method (DeepWalk, node2vec, struc2vec) with same number of random walks and walk lengths.
Revised: We set all random walk based method (DeepWalk, node2vec, struc2vec) with same number of random walks and walk lengths.

Original: For all graph neural network based methods (GAE,
Revised: For all graph neural network based methods (GAE,

Original: VGAE, DGI, NWR-GAE ), we use the same GCN encoder with same number of encoder layer from
Revised: VGAE, DGI, NWR-GAE ), we use the same GCN encoder with same number of encoder layer from

Original: DGL library (Wang et al., 2019).
Revised: DGL library (Wang et al., 2019).

Original: For the node classiﬁcation task, all methods use the same 4-layer
Revised: For the node classiﬁcation task, all methods use the same 4-layer

Original: MLP predictor with learning rate 0.01.
Revised: MLP predictor with learning rate 0.01.

Original: For all settings, we use Adam optimizer and backward propagation from PyTorch Python package, and a ﬁx dimension size as same as the graph node feature size.
Revised: For all settings, we use Adam optimizer and backward propagation from PyTorch Python package, and a ﬁx dimension size as same as the graph node feature size.

Original: All experiments are performed on a 8GB NVIDIA GeForce RTX 3070 GPU.
Revised: All experiments are performed on a 8GB NVIDIA GeForce RTX 3070 GPU.

Original: [Table]
Revised: [Table]

Original: Notes on neighborhood sampling.
Revised: Notes on neighborhood sampling.

Original: due to the heavy tailed nature of node distribution, we down sample the number of neighbors in NWR-GAE method.
Revised: due to the heavy tailed nature of node distribution, we down sample the number of neighbors in NWR-GAE method.

Original: We only sampled at most 10 neighbors per node.
Revised: We only sampled at most 10 neighbors per node.

Original: This provides a good trade-off between efﬁciency and performance, which makes the method have reasonable running time on one GPU machine and also achieve great enough accuracy.
Revised: This provides a good trade-off between efﬁciency and performance, which makes the method have reasonable running time on one GPU machine and also achieve great enough accuracy.

Original: D A DDITIONAL E XPERIMENTAL R ESULTS
Revised: D A DDITIONAL E XPERIMENTAL R ESULTS

Original: To further understand the effectiveness of our proposed neighborhood Wasserstein reconstruction method, we provide detailed results on how well the generated neighbor features can approximate the real ones.
Revised: To further understand the effectiveness of our proposed neighborhood Wasserstein reconstruction method, we provide detailed results on how well the generated neighbor features can approximate the real ones.

Original: Speciﬁcally, for every node v, to reconstruct k-hops neighborhood, the model will sample q neighbor nodes from N v in layer i, where 0 ≤ i ≤ k , thus { h ( i ) v j | 1 ≤ j ≤ q } denotes the q ground truth embeddings of v’s neighbor.
Revised: Speciﬁcally, for every node v, to reconstruct k-hops neighborhood, the model will sample q neighbor nodes from N v in layer i, where 0 ≤ i ≤ k , thus { h ( i ) v j | 1 ≤ j ≤ q } denotes the q ground truth embeddings of v’s neighbor.

Original: Next the model gets q samples { ξ j , 1 ≤ j ≤ q } from Gaussian distribution parameterized by GNN encoder, and uses an FNN-based transformation { ˆ h ( i,j ) v =
Revised: Next the model gets q samples { ξ j , 1 ≤ j ≤ q } from Gaussian distribution parameterized by GNN encoder, and uses an FNN-based transformation { ˆ h ( i,j ) v =

Original: FNN ( i ) ( ξ j ) } to reconstruct h ( i ) v j .
Revised: FNN ( i ) ( ξ j ) } to reconstruct h ( i ) v j .

Original: Based on above, we create x-y pairs with 
Revised: Based on above, we create x-y pairs with x

Original: [Equation]
Revised: [Equation]

Revised: |

Revised: [Equation]

Revised: [Equation]

Original: , π is any permutation
Revised: is any permutation

Original:  In Table 6, we ranked all x-y pairs by the ratio , and presented the ratio at 5%, 25%, 50%, 75%, 95% (like a box-plot).
Revised: [Table] In Table 6, we ranked all x-y pairs by the ratio , and presented the ratio at 5%, 25%, 50%, 75%, 95% (like a box-plot).

Original: As we can observe, our model can get a reasonably good approximation to the ground-truth neighborhood information (the smaller the better approximation).
Revised: As we can observe, our model can get a reasonably good approximation to the ground-truth neighborhood information (the smaller the better approximation).

Original: We tried different sample sizes q such as 5, 10, 20, which do not make a large difference in the ﬁrst three datasets[Table] since most nodes in the graphs have rather small numbers of neighbors.
Revised: We tried different sample sizes q such as 5, 10, 20, which do not make a large difference in the ﬁrst three datasets since most nodes in the graphs have rather small numbers of neighbors.

Original: Our model shows better approximation power on the neighborhood information with larger sample sizes such as Chameleon with denser links.
Revised: Our model shows better approximation power on the neighborhood information with larger sample sizes such as Chameleon with denser links.

Original: However, the performances on downstream tasks are already satisfactory even with smaller q values (the results we show in Table 2 are with q = 5 , and the results we show in
Revised: However, the performances on downstream tasks are already satisfactory even with smaller q values (the results we show in Table 2 are with q = 5 , and the results we show in

Original: Figure 4 (b) are with q = 5 , 10 , 15 ).
Revised: Figure 4 (b) are with q = 5 , 10 , 15 ).

Original: This indicates that while our model has the ability to more ideally approximate the entire neighborhood information, it often may not be necessary to guarantee satisfactory performance in downstream tasks, which allows us to train the model with small sample sizes and achieve good efﬁciency.
Revised: This indicates that while our model has the ability to more ideally approximate the entire neighborhood information, it often may not be necessary to guarantee satisfactory performance in downstream tasks, which allows us to train the model with small sample sizes and achieve good efﬁciency.

Original: To better understand the effect of number of GNN layers k in our proposed model, we vary k fromto 5 on the synthetic dataset of “House”.
Revised: To better understand the effect of number of GNN layers k in our proposed model, we vary k fromto 5 on the synthetic dataset of “House”.

Original: The model performance only gets better when k grows over 3, because the important graph structures there cannot be captured by GNNs with less than 3 layers.
Revised: The model performance only gets better when k grows over 3, because the important graph structures there cannot be captured by GNNs with less than 3 layers.

Original: Motivated by such observations and understandings, we set k to 4 for all real-world experiments, which can empirically capture most important local graph structures, to achieve a good trade-off between model effectiveness and efﬁciency.
Revised: Motivated by such observations and understandings, we set k to 4 for all real-world experiments, which can empirically capture most important local graph structures, to achieve a good trade-off between model effectiveness and efﬁciency.

Original: [Table]
Revised: [Table]

Original: To gain more insight into the behavior of different unsupervised graph embedding methods, we visualize the embedding space learned by several representative methods on Chameleon in Figure(reduced to two-dimensional via PCA).
Revised: To gain more insight into the behavior of different unsupervised graph embedding methods, we visualize the embedding space learned by several representative methods on Chameleon in Figure(reduced to two-dimensional via PCA).

Original: As we can observe, in these unsupervised two-dimensional embedding spaces, our baselines of node2vec, struc2vec, GraphWave, GAE and DGI can hardly distinguish the actual node classes, whereas NWR-GAE can achieve relatively better class separation.
Revised: As we can observe, in these unsupervised two-dimensional embedding spaces, our baselines of node2vec, struc2vec, GraphWave, GAE and DGI can hardly distinguish the actual node classes, whereas NWR-GAE can achieve relatively better class separation.

Original: Note that since all algorithms are unsupervised and the visualized embedding space is very lowdimensional, it is hard for any algorithm to achieve perfectly clear class separation, but NWR-GAE can capture more discriminative information and deliver less uniform node distribution, which is more useful for the node classiﬁcation task as consistent with our results in Table 2.
Revised: Note that since all algorithms are unsupervised and the visualized embedding space is very lowdimensional, it is hard for any algorithm to achieve perfectly clear class separation, but NWR-GAE can capture more discriminative information and deliver less uniform node distribution, which is more useful for the node classiﬁcation task as consistent with our results in Table 2.

Original: [Figure]
Revised: [Figure]