Original: C URVATURE G RAPH N ETWORK
Revised: C URVATURE G RAPH N ETWORK

Original: Anonymous authors
Revised: Anonymous authors

Original: Paper under double-blind review
Revised: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Graph-structured data is prevalent in many domains.
Revised: Graph-structured data is prevalent in many domains.

Original: Despite the widely celebrated success of deep neural networks, their power in graph-structured data is yet to be fully explored.
Revised: Despite the widely celebrated success of deep neural networks, their power in graph-structured data is yet to be fully explored.

Original: We propose a novel network architecture that incorporates advanced graph structural information.
Revised: We propose a novel network architecture that incorporates advanced graph structural information.

Original: In particular, we leverage discrete graph curvature, which measures how the neighborhoods of a pair of nodes are structurally related.
Revised: In particular, we leverage discrete graph curvature, which measures how the neighborhoods of a pair of nodes are structurally related.

Original: The curvature of an edge ( x, y ) deﬁnes the distance taken to travel from neighbors of x to neighbors of y , compared with the length of edge ( x, y ) .
Revised: The curvature of an edge ( x, y ) deﬁnes the distance taken to travel from neighbors of x to neighbors of y , compared with the length of edge ( x, y ) .

Original: It is a much more descriptive structural measure compared to previously ones that only focus on node speciﬁc attributes or limited topological information such as degree.
Revised: It is a much more descriptive structural measure compared to previously ones that only focus on node speciﬁc attributes or limited topological information such as degree.

Original: Our curvature graph convolution network outperforms state-of-the-art on various synthetic and real-world graphs, especially the larger and denser ones.
Revised: Our curvature graph convolution network outperforms state-of-the-art on various synthetic and real-world graphs, especially the larger and denser ones.

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: Despite the huge success of deep neural networks, it remains challenging to fully exploit their power on graph-structured data , i.e., data whose underlying structure is a graph, e.g., a social network, a telecommunication network, a biological network and a brain connectome.
Revised: Despite the huge success of deep neural networks, it remains challenging to fully exploit their power on graph-structured data , i.e., data whose underlying structure is a graph, e.g., a social network, a telecommunication network, a biological network and a brain connectome.

Original: Inspired by the power of convolution on image data, convolutional networks have been proposed for graph-structured data.
Revised: Inspired by the power of convolution on image data, convolutional networks have been proposed for graph-structured data.

Original: Existing works can be roughly divided into two categories, depending on whether convolution is applied to the spectral or spatial domain.
Revised: Existing works can be roughly divided into two categories, depending on whether convolution is applied to the spectral or spatial domain.

Original: Spectral approaches (Bruna et al., 2013; Defferrard et al., 2016; Henaff et al., 2015;
Revised: Spectral approaches (Bruna et al., 2013; Defferrard et al., 2016; Henaff et al., 2015;

Original: Veli ˇ ckovi ´ c et al., 2017) apply convolution to eigen-decomposed graph Laplacians and are generally efﬁcient in both computation and memory.
Revised: Veli ˇ ckovi ´ c et al., 2017) apply convolution to eigen-decomposed graph Laplacians and are generally efﬁcient in both computation and memory.

Original: However, the learned convolution ﬁlters are graph-speciﬁc and cannot generalize to different graphs.
Revised: However, the learned convolution ﬁlters are graph-speciﬁc and cannot generalize to different graphs.

Original: Spatial approaches execute “convolution” directly on the graph and operate on the neighborhood as deﬁned by the graph topology.
Revised: Spatial approaches execute “convolution” directly on the graph and operate on the neighborhood as deﬁned by the graph topology.

Original: A spatial method iteratively updates the representation of each graph node by aggregating representations from its neighbors, i.e., adjacent nodes (Xu et al., 2018).
Revised: A spatial method iteratively updates the representation of each graph node by aggregating representations from its neighbors, i.e., adjacent nodes (Xu et al., 2018).

Original: Nonlinear transformations are applied to the representation passed from one node to another, called a message .
Revised: Nonlinear transformations are applied to the representation passed from one node to another, called a message .

Original: These transformations have the same input/output dimension, i.e., the dimension of the node representation.
Revised: These transformations have the same input/output dimension, i.e., the dimension of the node representation.

Original: They can be shared and learned across different nodes and even different graphs.
Revised: They can be shared and learned across different nodes and even different graphs.

Original: For spatial approaches, it is important to incorporate local structural information of the graph.
Revised: For spatial approaches, it is important to incorporate local structural information of the graph.

Original: Node degree has been used to reparametrize the nonlinear transformation of messages (Monti et al., 2017) or as an additional node feature (Hamilton et al., 2017).
Revised: Node degree has been used to reparametrize the nonlinear transformation of messages (Monti et al., 2017) or as an additional node feature (Hamilton et al., 2017).

Original: However, node degree is fairly limited; there can be different graph topologies with the same degree distribution.
Revised: However, node degree is fairly limited; there can be different graph topologies with the same degree distribution.

Original: The limitation is illustrated in
Revised: The limitation is illustrated in

Original: Figure 1.
Revised: Figure 1.

Original: Nodes x and y have the same degree in three signiﬁcantly different graphs: a tree, a grid graph and a clique.
Revised: Nodes x and y have the same degree in three signiﬁcantly different graphs: a tree, a grid graph and a clique.

Original: To effectively make use of graph structural knowledge, one would need a feature with more discriminative power; one that can distinguish these three scenarios in Figure 1.
Revised: To effectively make use of graph structural knowledge, one would need a feature with more discriminative power; one that can distinguish these three scenarios in Figure 1.

Original: In this paper, we propose a novel graph neural network that exploits advanced structural information.
Revised: In this paper, we propose a novel graph neural network that exploits advanced structural information.

Original: Notice that node degree only describes the number of neighbors of each node, but does not say how these neighbors are connected among themselves.
Revised: Notice that node degree only describes the number of neighbors of each node, but does not say how these neighbors are connected among themselves.

Original: We seek to use structural information characterizing how neighborhoods of a pair of nodes relate to each other.
Revised: We seek to use structural information characterizing how neighborhoods of a pair of nodes relate to each other.

Original: In Figure 1, the neighborhoods of x and y are well separated in a tree.
Revised: In Figure 1, the neighborhoods of x and y are well separated in a tree.

Original: In a grid graph, the two neighborhoods are within a parallel shift of each other.
Revised: In a grid graph, the two neighborhoods are within a parallel shift of each other.

Original: In a clique, they completely overlap.
Revised: In a clique, they completely overlap.

Original: To quantify such pairwise structural information, we draw inspiration from recent study of graph curvature (Ollivier, 2009;
Revised: To quantify such pairwise structural information, we draw inspiration from recent study of graph curvature (Ollivier, 2009;

Original: Lin et al., 2011; Weber et al., 2016).
Revised: Lin et al., 2011; Weber et al., 2016).

Original: Similar to the curvature in the continuous domain, e.g., the Ricci curvature of a Riemannian manifold, the discrete graph curvature measures how the geometry of a pair of neighborhoods deviates from a “ﬂat” case, namely, the case of a grid graph.
Revised: Similar to the curvature in the continuous domain, e.g., the Ricci curvature of a Riemannian manifold, the discrete graph curvature measures how the geometry of a pair of neighborhoods deviates from a “ﬂat” case, namely, the case of a grid graph.

Original: There are several deﬁnitions of discrete curvature for graphs.
Revised: There are several deﬁnitions of discrete curvature for graphs.

Original: The most notable one is Ollivier’s Ricci curvature (Ollivier, 2009).
Revised: The most notable one is Ollivier’s Ricci curvature (Ollivier, 2009).

Original: The edges of a (inﬁnite)
Revised: The edges of a (inﬁnite)

Original: [Figure]
Revised: [Figure]

Original: grid graph have zero curvature.
Revised: grid graph have zero curvature.

Original: The curvature of an edge ( x, y ) in a tree is negative and is positive in a complete graph.
Revised: The curvature of an edge ( x, y ) in a tree is negative and is positive in a complete graph.

Original: Intuitively, the graph curvature measures how well two neighborhoods are connected and/or overlap with each other.
Revised: Intuitively, the graph curvature measures how well two neighborhoods are connected and/or overlap with each other.

Original: Such information is related to how information propagates in the neighborhood, and should be leveraged by a graph convolutional network.
Revised: Such information is related to how information propagates in the neighborhood, and should be leveraged by a graph convolutional network.

Original: We propose Curvature Graph Network (CurvGN) , the ﬁrst graph convolutional network built on advanced graph curvature information.
Revised: We propose Curvature Graph Network (CurvGN) , the ﬁrst graph convolutional network built on advanced graph curvature information.

Original: In particular, we propose a novel network architecture that efﬁciently computes graph curvature and fully leverages such information in graph convolution.
Revised: In particular, we propose a novel network architecture that efﬁciently computes graph curvature and fully leverages such information in graph convolution.

Original: Using curvature information, CurvGN better adapts to different local structural scenarios and ﬁlter messages passed between nodes differently.
Revised: Using curvature information, CurvGN better adapts to different local structural scenarios and ﬁlter messages passed between nodes differently.

Original: Notice that the curvature information captures how easy information ﬂows between the nodes.
Revised: Notice that the curvature information captures how easy information ﬂows between the nodes.

Original: Within a well-connected community, neighborhoods of adjacent nodes have large overlap and many shortcuts.
Revised: Within a well-connected community, neighborhoods of adjacent nodes have large overlap and many shortcuts.

Original: The corresponding curvature is positive and passing information between the nodes is easy.
Revised: The corresponding curvature is positive and passing information between the nodes is easy.

Original: For edges bridging two clusters/cliques, the curvature is negative and information is hard to pass.
Revised: For edges bridging two clusters/cliques, the curvature is negative and information is hard to pass.

Original: A key to our success is that we choose to be agnostic on whether the curvature information should be used to block or accelerate the messages in graph convolution.
Revised: A key to our success is that we choose to be agnostic on whether the curvature information should be used to block or accelerate the messages in graph convolution.

Original: We exploit the curvature in a data-driven manner and learn how to use it to reweigh different channels of the message.
Revised: We exploit the curvature in a data-driven manner and learn how to use it to reweigh different channels of the message.

Original: To further investigate how curvature information affects graph convolution, we carried out extensive experiments with various synthetic graphs and real-world graphs.
Revised: To further investigate how curvature information affects graph convolution, we carried out extensive experiments with various synthetic graphs and real-world graphs.

Original: Our synthetic data are generated according to various well-established graph models, e.g., stochastic block model (Decelle et al., 2011),
Revised: Our synthetic data are generated according to various well-established graph models, e.g., stochastic block model (Decelle et al., 2011),

Original: Watts–Strogatz network (Watts & Strogatz, 1998), Newman–Watts network (Newman & Watts, 1999) and Kleinberg’s navigable small world graph (Kleinberg, 2000).
Revised: Watts–Strogatz network (Watts & Strogatz, 1998), Newman–Watts network (Newman & Watts, 1999) and Kleinberg’s navigable small world graph (Kleinberg, 2000).

Original: On these data, CurvGN outperforms vanilla graph network and networks using node degree information and self attention, demonstrating the beneﬁt of curvature information in graph convolution.
Revised: On these data, CurvGN outperforms vanilla graph network and networks using node degree information and self attention, demonstrating the beneﬁt of curvature information in graph convolution.

Original: Such beneﬁt is more apparent as the graph size increases.
Revised: Such beneﬁt is more apparent as the graph size increases.

Original: We hypothesize that graph convolution alone can adapt to any graph topology, at the cost of more convolutional layers and more training data.
Revised: We hypothesize that graph convolution alone can adapt to any graph topology, at the cost of more convolutional layers and more training data.

Original: This is corroborated by our experiments on real-world graph.
Revised: This is corroborated by our experiments on real-world graph.

Original: CurvGN outperforms state-of-the-art graph neural networks, especially on larger and denser graphs, which tend to have a large variation of local structures.
Revised: CurvGN outperforms state-of-the-art graph neural networks, especially on larger and denser graphs, which tend to have a large variation of local structures.

Original: The success of CurvCN demonstrates how theoretical insights inspire better practical solutions.
Revised: The success of CurvCN demonstrates how theoretical insights inspire better practical solutions.

Original: It encourages us to continue the endeavor in applying principles mathematics and theory in successful deployment of deep learning.
Revised: It encourages us to continue the endeavor in applying principles mathematics and theory in successful deployment of deep learning.

Original:  R ELATED W ORK
Revised:  R ELATED W ORK

Original: We brieﬂy summarize previous works on graph convolution.
Revised: We brieﬂy summarize previous works on graph convolution.

Original: In early works (Frasconi et al., 1998;
Revised: In early works (Frasconi et al., 1998;

Original: Sperduti & Starita, 1997), recursive neural networks were applied on data whose underlying structures are directed acyclic graphs.
Revised: Sperduti & Starita, 1997), recursive neural networks were applied on data whose underlying structures are directed acyclic graphs.

Original: In Graph Neural Networks (GNNs) (Gori et al., 2005; Scarselli et al., 2009), the recursive network framework was extended to general graphs.
Revised: In Graph Neural Networks (GNNs) (Gori et al., 2005; Scarselli et al., 2009), the recursive network framework was extended to general graphs.

Original: Li et al.
Revised: Li et al.

Original: introduced gated recurrent units into the framework in order to improve the performance.
Revised: introduced gated recurrent units into the framework in order to improve the performance.

Original: Since Convolutional
Revised: Since Convolutional

Original: Neural Networks (CNNs) have demonstrated strong performance in grid-like-structured data, various methods have been proposed to implement “convolution” on graph-structured data.
Revised: Neural Networks (CNNs) have demonstrated strong performance in grid-like-structured data, various methods have been proposed to implement “convolution” on graph-structured data.

Original: The efforts can be roughly divided into spectral approaches and spatial approaches.
Revised: The efforts can be roughly divided into spectral approaches and spatial approaches.

Original: Below we will review both categories in details.
Revised: Below we will review both categories in details.

Original: Spectral approaches.
Revised: Spectral approaches.

Original: Bruna et al.
Revised: Bruna et al.

Original: (2013) transformed the graph convolution into spectral domain multiplication by graph Fourier transform.
Revised: (2013) transformed the graph convolution into spectral domain multiplication by graph Fourier transform.

Original: This method is expensive due to the matrix eigendecomposition.
Revised: This method is expensive due to the matrix eigendecomposition.

Original: Furthermore, it cannot create spatially localized ﬁlters as in CNNs.
Revised: Furthermore, it cannot create spatially localized ﬁlters as in CNNs.

Original: Henaff et al.
Revised: Henaff et al.

Original: (2015) applied smooth coefﬁcients on spectral ﬁlters to make them spatially localized.
Revised: (2015) applied smooth coefﬁcients on spectral ﬁlters to make them spatially localized.

Original: Defferrard et al.
Revised: Defferrard et al.

Original: (2016) used Chebyshev expansion of the graph Laplacian to approximate the ﬁlters as a k -polynomial function.
Revised: (2016) used Chebyshev expansion of the graph Laplacian to approximate the ﬁlters as a k -polynomial function.

Original: Kipf & Welling (2016) simpliﬁed those methods by reducing the polynomials to degree 1.
Revised: Kipf & Welling (2016) simpliﬁed those methods by reducing the polynomials to degree 1.

Original: Their method is essentially ﬁltering the graph with a kernel whose receptive ﬁeld is the 1-hop neighborhood of each node.
Revised: Their method is essentially ﬁltering the graph with a kernel whose receptive ﬁeld is the 1-hop neighborhood of each node.

Original: The common limitation of spectral approaches is that the convolution ﬁlters depend on the Laplacian of each speciﬁc graph.
Revised: The common limitation of spectral approaches is that the convolution ﬁlters depend on the Laplacian of each speciﬁc graph.

Original: Spatial approaches.
Revised: Spatial approaches.

Original: The main challenge of spatial approaches is to design an operator which applies to neighborhoods with different topology and still maintains shared ﬁlters.
Revised: The main challenge of spatial approaches is to design an operator which applies to neighborhoods with different topology and still maintains shared ﬁlters.

Original: Monti et al.
Revised: Monti et al.

Original: (2017) introduced a mixture model CNN (MoNet) that maps graph neighborhood into spatial neighborhood (with pseudo-coordinates) for spatial convolution.
Revised: (2017) introduced a mixture model CNN (MoNet) that maps graph neighborhood into spatial neighborhood (with pseudo-coordinates) for spatial convolution.

Original: Hamilton et al.
Revised: Hamilton et al.

Original: (2017) proposed GraphSAGE that samples ﬁxed size neighbors and aggregates their representations.
Revised: (2017) proposed GraphSAGE that samples ﬁxed size neighbors and aggregates their representations.

Original: Veliˇckovi´c et al.
Revised: Veliˇckovi´c et al.

Original: (2017) proposed Graph Attention Network (GAT), which uses self-attention mechanism to reweigh graph convolution.
Revised: (2017) proposed Graph Attention Network (GAT), which uses self-attention mechanism to reweigh graph convolution.

Original: Recently, there have been other methods which studied graph neural networks from different perspectives, such as pooling (Gao & Ji, 2019;
Revised: Recently, there have been other methods which studied graph neural networks from different perspectives, such as pooling (Gao & Ji, 2019;

Original: Ying et al., 2018).
Revised: Ying et al., 2018).

Original: Discrete graph curvature.
Revised: Discrete graph curvature.

Original: Different proposals for discrete graph curvature have been introduced in recent years, including Ollivier’s Ricci curvature and Forman curvature (Ollivier, 2009; Lin et al., 2011; Forman, 2003).
Revised: Different proposals for discrete graph curvature have been introduced in recent years, including Ollivier’s Ricci curvature and Forman curvature (Ollivier, 2009; Lin et al., 2011; Forman, 2003).

Original: We focus on Ollivier’s Ricci curvature as it is more geometric in nature.
Revised: We focus on Ollivier’s Ricci curvature as it is more geometric in nature.

Original: Meanwhile, Forman’s deﬁnition of discrete curvature (Forman, 2003) is more combinatorial and is faster to compute.
Revised: Meanwhile, Forman’s deﬁnition of discrete curvature (Forman, 2003) is more combinatorial and is faster to compute.

Original: Both curvatures have been applied to real-world graphs.
Revised: Both curvatures have been applied to real-world graphs.

Original: Ni et al.
Revised: Ni et al.

Original: (2015) showed that Ollivier’s Ricci curvature can be used to identify backbone edges of an Internet AS graph and is closely related to network vulnerability.
Revised: (2015) showed that Ollivier’s Ricci curvature can be used to identify backbone edges of an Internet AS graph and is closely related to network vulnerability.

Original: In Ni et al.
Revised: In Ni et al.

Original: it was shown that using Ollivier’s Ricci ﬂow, one can deﬁne a new metric that is more robust for network matching.
Revised: it was shown that using Ollivier’s Ricci ﬂow, one can deﬁne a new metric that is more robust for network matching.

Original: Forman curvature is shown to have similar effect (Weber et al., 2017; 2016).
Revised: Forman curvature is shown to have similar effect (Weber et al., 2017; 2016).

Original: To the best of our knowledge, graph curvature has not been used in graph neural networks.
Revised: To the best of our knowledge, graph curvature has not been used in graph neural networks.

Original:  C URVATURE G RAPH N ETWORK
Revised:  C URVATURE G RAPH N ETWORK

Original: We ﬁrst formulate the node label prediction problem of a graph, and explain the mechanism of a
Revised: We ﬁrst formulate the node label prediction problem of a graph, and explain the mechanism of a

Original: Graph Neural Network (GNN).
Revised: Graph Neural Network (GNN).

Original: Suppose we have an undirected graph G =
Revised: Suppose we have an undirected graph G =

Original: ( V, E ) with features on the vertices
Revised: ( V, E ) with features on the vertices

Original: H
Revised: H

Original: =
Revised: =

Original: ( h 1 , h 2 , · · · , h n ) , h i ∈ R F .
Revised: ( h 1 , h 2 , · · · , h n ) , h i ∈ R F .

Original: Here n = | V | is the number of nodes in the graph and F is the feature dimension of each node.
Revised: Here n = | V | is the number of nodes in the graph and F is the feature dimension of each node.

Original: Given labels of some nodes in V , we would like to predict the labels of the remaining nodes.
Revised: Given labels of some nodes in V , we would like to predict the labels of the remaining nodes.

Original: A GNN iteratively updates the graph G ’s node representation and eventually predicts node labels.
Revised: A GNN iteratively updates the graph G ’s node representation and eventually predicts node labels.

Original: A GNN consists of multiple hidden layers that update node representation from lower level node representation H t ∈ R n × F t to high level representation H t +1 ∈ R n × F t +1 .
Revised: A GNN consists of multiple hidden layers that update node representation from lower level node representation H t ∈ R n × F t to high level representation H t +1 ∈ R n × F t +1 .

Original: In particular, H 0 is the input feature, H .
Revised: In particular, H 0 is the input feature, H .

Original: Node representations of the last layer, H T , are fed to a fully connected layer or a linear classiﬁer to predict node labels.
Revised: Node representations of the last layer, H T , are fed to a fully connected layer or a linear classiﬁer to predict node labels.

Original: The layers and their representations are illustrated in the top of Figure 2.
Revised: The layers and their representations are illustrated in the top of Figure 2.

Original: Now we explain how to construct hidden layers that update node representations from H t to H t +1 .
Revised: Now we explain how to construct hidden layers that update node representations from H t to H t +1 .

Original: We focus on spatial approaches and treat the convolution as a message passing scheme.
Revised: We focus on spatial approaches and treat the convolution as a message passing scheme.

Original: The ( t +1) -th representation of node x is computed by aggregating messages passed from x ’s neighbors.
Revised: The ( t +1) -th representation of node x is computed by aggregating messages passed from x ’s neighbors.

Original: We also include the message from x to itself.
Revised: We also include the message from x to itself.

Original: There are several aggregation methods, such as mean, max and sum.
Revised: There are several aggregation methods, such as mean, max and sum.

Original: We choose summation as it is a commonly used aggregation method (Kipf & Welling, 2016;
Revised: We choose summation as it is a commonly used aggregation method (Kipf & Welling, 2016;

Original: Veli ˇ ckovi ´ c et al., 2017; Xu et al., 2018).
Revised: Veli ˇ ckovi ´ c et al., 2017; Xu et al., 2018).

Original: Denote by N ( x ) = N ( x ) ∪ { x } the neighborhood of x including itself.
Revised: Denote by N ( x ) = N ( x ) ∪ { x } the neighborhood of x including itself.

Original: We have h t +1 x = σ t (cid:16)(cid:80)
Revised: We have h t +1 x = σ t (cid:16)(cid:80)

Original: y ∈N ( x ) M ty → x (cid:17) , in which σ t is the non-linear transformation.
Revised: y ∈N ( x ) M ty → x (cid:17) , in which σ t is the non-linear transformation.

Original: A message passed from y to x is a linear transformation of y ’s representation.
Revised: A message passed from y to x is a linear transformation of y ’s representation.

Original: We also introduce a weight τ txy whose purpose will be clear later.
Revised: We also introduce a weight τ txy whose purpose will be clear later.

Original: Formally, we have M ty → x = τ txy W t h ty , in which W t is the linear transformation matrix learned in training.
Revised: Formally, we have M ty → x = τ txy W t h ty , in which W t is the linear transformation matrix learned in training.

Original: Formally, we have the representation updating equation
Revised: Formally, we have the representation updating equation

Original: [Equation]
Revised: [Equation]

Original: It is crucial to obtain suitable reweighting parameter τ txy since it is directly affecting how neighboring node information are passed to the node x .
Revised: It is crucial to obtain suitable reweighting parameter τ txy since it is directly affecting how neighboring node information are passed to the node x .

Original: Some papers use node degree information as τ txy (Kipf & Welling, 2016; Monti et al., 2017) and other work uses joint node features to compute the self attention as τ txy (Veliˇckovi´c et al., 2017).
Revised: Some papers use node degree information as τ txy (Kipf & Welling, 2016; Monti et al., 2017) and other work uses joint node features to compute the self attention as τ txy (Veliˇckovi´c et al., 2017).

Original: We propose to use more advanced structural information, i.e., the Ricci curvature, to compute τ txy .
Revised: We propose to use more advanced structural information, i.e., the Ricci curvature, to compute τ txy .

Original: It is also known that the reweighting parameter τ txy is not necessarily a scalar.
Revised: It is also known that the reweighting parameter τ txy is not necessarily a scalar.

Original: It can also be anything between a scalar and a F t × F t matrix.
Revised: It can also be anything between a scalar and a F t × F t matrix.

Original: In fact, we choose
Revised: In fact, we choose

Original: F t later on because it has more expressive power than a scalar and it is easier to train than a matrix.
Revised: F t later on because it has more expressive power than a scalar and it is easier to train than a matrix.

Original: To illustrate how we build curvature convolution layer in Equation (3.1), we deﬁne Ricci Curvature in the context of graph (Section 3.1).
Revised: To illustrate how we build curvature convolution layer in Equation (3.1), we deﬁne Ricci Curvature in the context of graph (Section 3.1).

Original: We explain how to compute τ txy from the curvature in Section 3.2.
Revised: We explain how to compute τ txy from the curvature in Section 3.2.

Original: 3. R ICCI C URVATURE
Revised: 3. R ICCI C URVATURE

Original: In Riemannian geometry, curvature measures how a smooth object deviates from being ﬂat, or being straight in the case of a line.
Revised: In Riemannian geometry, curvature measures how a smooth object deviates from being ﬂat, or being straight in the case of a line.

Original: Similar concepts can be extended to non-smooth setting for discrete objects.
Revised: Similar concepts can be extended to non-smooth setting for discrete objects.

Original: In particular, curvature has been studied for metric-measure space (Bonciocat, 2014;
Revised: In particular, curvature has been studied for metric-measure space (Bonciocat, 2014;

Original: Bonciocat & Sturm, 2009; Lott & Villani, 2009; Sturm et al., 2006), Markov chain (Ollivier, 2009) and general graphs (Lin et al., 2011).
Revised: Bonciocat & Sturm, 2009; Lott & Villani, 2009; Sturm et al., 2006), Markov chain (Ollivier, 2009) and general graphs (Lin et al., 2011).

Original: The deﬁnitions of curvatures that are easier to generalize in a discrete graph setting are sectional curvature and Ricci curvature.
Revised: The deﬁnitions of curvatures that are easier to generalize in a discrete graph setting are sectional curvature and Ricci curvature.

Original: For a point x on the surface, considers two tangent vectors v and w x .
Revised: For a point x on the surface, considers two tangent vectors v and w x .

Original: Let y be the end point of the tangent vector v at x .
Revised: Let y be the end point of the tangent vector v at x .

Original: Imagine transporting in parallel w x along v to a tangent vector w y at y .
Revised: Imagine transporting in parallel w x along v to a tangent vector w y at y .

Original: If the surface is ﬂat, any pair of points x (cid:48) and y (cid:48) which is (cid:15) away from x and y along w x and w y will have the same distance as x and y .
Revised: If the surface is ﬂat, any pair of points x (cid:48) and y (cid:48) which is (cid:15) away from x and y along w x and w y will have the same distance as x and y .

Original: Its deviation from | v | deﬁnes the sectional curvature.
Revised: Its deviation from | v | deﬁnes the sectional curvature.

Original: Then averaging it over all directions of w x gives the Ricci curvature which only depends on the tangent vector v .
Revised: Then averaging it over all directions of w x gives the Ricci curvature which only depends on the tangent vector v .

Original: Intuitively, instead of w x , we can think of S x be the set of end points of all tangent vectors at x with length (cid:15) .
Revised: Intuitively, instead of w x , we can think of S x be the set of end points of all tangent vectors at x with length (cid:15) .

Original: Again, if we map S x to S y using parallel transport along v , the distance between a point at
Revised: Again, if we map S x to S y using parallel transport along v , the distance between a point at

Original: S x and its corresponding image at S y can be different from | v | if the surface is not ﬂat.
Revised: S x and its corresponding image at S y can be different from | v | if the surface is not ﬂat.

Original: To generalize Ricci curvature to discrete spaces, Ollivier (2009) takes a coarse approach that represent
Revised: To generalize Ricci curvature to discrete spaces, Ollivier (2009) takes a coarse approach that represent

Original: S x as a probability measure m x of mass 1 around x .
Revised: S x as a probability measure m x of mass 1 around x .

Original: Thus the distance can be measured by
Revised: Thus the distance can be measured by

Original: Wasserstein distance (or Earth Mover distance) which ﬁnds the optimal mass-preserving transportation plan between two probability measures.
Revised: Wasserstein distance (or Earth Mover distance) which ﬁnds the optimal mass-preserving transportation plan between two probability measures.

Original: Then the coarse Ricci curvature κ ( x, y ) on edge ( x, y ) is deﬁned by comparing the Wasserstein distance W ( m x , m y ) to the distance d ( x, y ) , formally, κ xy = 1 − W ( m x , m y ) /d
Revised: Then the coarse Ricci curvature κ ( x, y ) on edge ( x, y ) is deﬁned by comparing the Wasserstein distance W ( m x , m y ) to the distance d ( x, y ) , formally, κ xy = 1 − W ( m x , m y ) /d

Original: ( x, y ) .
Revised: ( x, y ) .

Original: The natural analogy of a small ball S x at point x in the metric space is the 1-hop neighborhood N ( x ) of node x in a graph.
Revised: The natural analogy of a small ball S x at point x in the metric space is the 1-hop neighborhood N ( x ) of node x in a graph.

Original: This motivates the Ollivier-Ricci curvature on graph edges.
Revised: This motivates the Ollivier-Ricci curvature on graph edges.

Original: For an undirected graph
Revised: For an undirected graph

Original: G =
Revised: G =

Original: ( V, E ) , denotes the set of neighboring nodes of a node x ∈ V as N ( x ) = { x 1 , x 2 , . . .
Revised: ( V, E ) , denotes the set of neighboring nodes of a node x ∈ V as N ( x ) = { x 1 , x 2 , . . .

Original: , x k } .
Revised: , x k } .

Original: Then we can deﬁne a probability measure m αx at x :
Revised: Then we can deﬁne a probability measure m αx at x :

Original: [Equation]
Revised: [Equation]

Original: otherwise where α is a parameter within [0 , 1] .
Revised: otherwise where α is a parameter within [0 , 1] .

Original: It is to keep probability mass of α at node x itself and distribute the rest uniformly over the neighborhood.
Revised: It is to keep probability mass of α at node x itself and distribute the rest uniformly over the neighborhood.

Original: To compute the Wasserstein distance W ( m αx , m αy ) between the probability measures around two end points x, y of the edge ( x, y ) , the optimal transportation plan can be solved by the following linear programming:
Revised: To compute the Wasserstein distance W ( m αx , m αy ) between the probability measures around two end points x, y of the edge ( x, y ) , the optimal transportation plan can be solved by the following linear programming:

Original: [Equation]
Revised: [Equation]

Original: where M ( x i , y j ) is the amount of probability mass transported from node x i to y i along the shortest path with length d ( x i , y j ) .
Revised: where M ( x i , y j ) is the amount of probability mass transported from node x i to y i along the shortest path with length d ( x i , y j ) .

Original: In the language of graph theory, if the Ollivier-Ricci curvature is negative
Revised: In the language of graph theory, if the Ollivier-Ricci curvature is negative

Original: ( W ( m αx , m αy ) >
Revised: ( W ( m αx , m αy ) >

Original: d ( x, y ) ), the edge ( x, y ) serves as the bridge connecting the neighborhood around x and y .
Revised: d ( x, y ) ), the edge ( x, y ) serves as the bridge connecting the neighborhood around x and y .

Original: This aligns with the intuition in the smooth setting where the small balls S x and S y are closer to each other than their centers.
Revised: This aligns with the intuition in the smooth setting where the small balls S x and S y are closer to each other than their centers.

Original: Following existing work (Ni et al., 2018), we set α = 0 .
Revised: Following existing work (Ni et al., 2018), we set α = 0 .

Original: 3. C URVATURE -D RIVEN G RAPH C ONVOLUTION
Revised: 3. C URVATURE -D RIVEN G RAPH C ONVOLUTION

Original: Next we present how Ricci curvature is used in our graph convolutional network.
Revised: Next we present how Ricci curvature is used in our graph convolutional network.

Original: The usage of curvature should depend on the problem and the data.
Revised: The usage of curvature should depend on the problem and the data.

Original: Intuitively, curvature measures how easy a message ﬂows through an edge, and should be used to control messages in convolution.
Revised: Intuitively, curvature measures how easy a message ﬂows through an edge, and should be used to control messages in convolution.

Original: For example, an edge with negative curvature is likely to be a bridge connecting two different communities.
Revised: For example, an edge with negative curvature is likely to be a bridge connecting two different communities.

Original: If we assume different communities tend to have different representations/labels, a message should be blocked on this edge.
Revised: If we assume different communities tend to have different representations/labels, a message should be blocked on this edge.

Original: Meanwhile, an edge with positive curvature tends to be intra-community and thus should have accelerated message ﬂow.
Revised: Meanwhile, an edge with positive curvature tends to be intra-community and thus should have accelerated message ﬂow.

Original: However, the intuition may be invalid if the community structure is not correlated with node representation/labels.
Revised: However, the intuition may be invalid if the community structure is not correlated with node representation/labels.

Original: We choose to be agnostic on how the knowledge of edge curvature should be used.
Revised: We choose to be agnostic on how the knowledge of edge curvature should be used.

Original: We resort to a data-driven strategy and learn a mapping function that maps Ricci curvature κ xy to the weight of messages, i.e., τ txy in Equation (3.1).
Revised: We resort to a data-driven strategy and learn a mapping function that maps Ricci curvature κ xy to the weight of messages, i.e., τ txy in Equation (3.1).

Original: We ﬁrst explain how the mapping is learned end-to-end (CurvGN-1).
Revised: We ﬁrst explain how the mapping is learned end-to-end (CurvGN-1).

Original: Next we expand the mapping to a multi-valued version, to incorporate more ﬂexibility in the model (CurvGN-n).
Revised: Next we expand the mapping to a multi-valued version, to incorporate more ﬂexibility in the model (CurvGN-n).

Original: CurvGN-1.
Revised: CurvGN-1.

Original: As mentioned before, τ txy can be anything between a scalar and a F t × F t matrix.
Revised: As mentioned before, τ txy can be anything between a scalar and a F t × F t matrix.

Original: We ﬁrst assume τ txy is a scalar.
Revised: We ﬁrst assume τ txy is a scalar.

Original: Then the mapping function can be deﬁned as:
Revised: Then the mapping function can be deﬁned as:

Original: [Equation] [Equation]
Revised: [Equation] [Equation]

Original: We create a multi-layer perceptron (MLP) to approximate the mapping function f t since MLP is proved to be a universal approximation machine and can be easily incorporated into our GNN model for end-to-end training.
Revised: We create a multi-layer perceptron (MLP) to approximate the mapping function f t since MLP is proved to be a universal approximation machine and can be easily incorporated into our GNN model for end-to-end training.

Original: Denote the MLP at the t -th layer as MLP t .
Revised: Denote the MLP at the t -th layer as MLP t .

Original: As summation is used as the aggregation function in Equation (3.1), the messages may accumulate to an arbitrarily large value.
Revised: As summation is used as the aggregation function in Equation (3.1), the messages may accumulate to an arbitrarily large value.

Original: To prevent a numerical explosion, we apply a softmax function, S t , to MLP t ( κ xy ) of all neighbors of x including itself, y ∈ N ( x ) nodes.
Revised: To prevent a numerical explosion, we apply a softmax function, S t , to MLP t ( κ xy ) of all neighbors of x including itself, y ∈ N ( x ) nodes.

Original: [Equation]
Revised: [Equation]

Original: Figure 2 bottom shows how the MLP transforms a curvature and uses it to reweigh messages.
Revised: Figure 2 bottom shows how the MLP transforms a curvature and uses it to reweigh messages.

Original: CurvGN-n. Messages M ty → x are usually multi-channeled.
Revised: CurvGN-n. Messages M ty → x are usually multi-channeled.

Original: In particular, they are F t +1 -dimensional.
Revised: In particular, they are F t +1 -dimensional.

Original: The scalar weight generated using curvature is not necessarily the same for different channels.
Revised: The scalar weight generated using curvature is not necessarily the same for different channels.

Original: To improve the expressing power of τ txy , we create a similar mapping function as f t in Equation (3.3).
Revised: To improve the expressing power of τ txy , we create a similar mapping function as f t in Equation (3.3).

Original: But the new mapping generates a reweighing vector T txy ∈ R F t +1 .
Revised: But the new mapping generates a reweighing vector T txy ∈ R F t +1 .

Original: In other words, we learn to reweigh different message channels differently.
Revised: In other words, we learn to reweigh different message channels differently.

Original: In principle, we can even learn an R F t +1 × R F t +1 to reweigh the message.
Revised: In principle, we can even learn an R F t +1 × R F t +1 to reweigh the message.

Original: However, a vector has signiﬁcantly less parameters to train and is found to be sufﬁcient in practice.
Revised: However, a vector has signiﬁcantly less parameters to train and is found to be sufﬁcient in practice.

Original: Using the same strategy as CurvGN-1, the vector T txy is calculated by applying a MLP t with F t + outputs.
Revised: Using the same strategy as CurvGN-1, the vector T txy is calculated by applying a MLP t with F t + outputs.

Original: Then, we apply a channel-wise softmax function, S t , that normalizes the MLP outputs separately on each message channel: T txy =
Revised: Then, we apply a channel-wise softmax function, S t , that normalizes the MLP outputs separately on each message channel: T txy =

Original: S t ( MLP t ( κ xy ))
Revised: S t ( MLP t ( κ xy ))

Original: Substituting T xy into Equation (3.1), we have the convolution of CurvGN-n:
Revised: Substituting T xy into Equation (3.1), we have the convolution of CurvGN-n:

Original: [Equation]
Revised: [Equation]

Original: Here diag ( T txy ) is a matrix whose diagonal entries are entries of T txy .
Revised: Here diag ( T txy ) is a matrix whose diagonal entries are entries of T txy .

Original: For details of MLP t , please refer to Appendix A.1.
Revised: For details of MLP t , please refer to Appendix A.1.

Original: Design details of the network.
Revised: Design details of the network.

Original: In practice, we use a two-convolutional-layer CurvGN model.
Revised: In practice, we use a two-convolutional-layer CurvGN model.

Original: The ﬁrst layer is a linear transform layer that produces an output feature vector paired with a three layer
Revised: The ﬁrst layer is a linear transform layer that produces an output feature vector paired with a three layer

Original: MLP that computes reweighing vector.
Revised: MLP that computes reweighing vector.

Original: The output feature is pushed into an exponential linear unit layer to add non-linearity.
Revised: The output feature is pushed into an exponential linear unit layer to add non-linearity.

Original: The second layer is for classiﬁcation, with the same structure as the ﬁrst layer except that the output feature is now at length of class number.
Revised: The second layer is for classiﬁcation, with the same structure as the ﬁrst layer except that the output feature is now at length of class number.

Original: The hyperparameters are similar to GAT implemented in Veliˇckovi´c et al.
Revised: The hyperparameters are similar to GAT implemented in Veliˇckovi´c et al.

Original: For synthetic experiments, the hidden layer output is reduced to 8 dimensions.
Revised: For synthetic experiments, the hidden layer output is reduced to 8 dimensions.

Original:  E XPERIMENTS
Revised:  E XPERIMENTS

Original: We evaluate our method on both synthetic and real-world graphs.
Revised: We evaluate our method on both synthetic and real-world graphs.

Original: Our method outperforms the state-of-the-art methods, especially on larger and denser graphs, which tend to have heterogeneous topology.
Revised: Our method outperforms the state-of-the-art methods, especially on larger and denser graphs, which tend to have heterogeneous topology.

Original: In addition to proving the prediction power, we use different graph theoretical models in synthetic experiments and different parameter settings to gain insights of how curvature information helps graph convolution.
Revised: In addition to proving the prediction power, we use different graph theoretical models in synthetic experiments and different parameter settings to gain insights of how curvature information helps graph convolution.

Original: We focus on node classiﬁcation task, while our method easily generalizes to graph classiﬁcation task.
Revised: We focus on node classiﬁcation task, while our method easily generalizes to graph classiﬁcation task.

Original: 4. S YNTHETIC E XPERIMENTS ON D IFFERENT G RAPH T HEORETICAL M ODELS
Revised: 4. S YNTHETIC E XPERIMENTS ON D IFFERENT G RAPH T HEORETICAL M ODELS

Original: We generate synthetic data using different graph theoretical models.
Revised: We generate synthetic data using different graph theoretical models.

Original: We start with the Stochastic
Revised: We start with the Stochastic

Original: Block Model (SBM) (Holland et al., 1983), which assumes a partition of the graph into communities.
Revised: Block Model (SBM) (Holland et al., 1983), which assumes a partition of the graph into communities.

Original: We create random graphs, each with 1000 nodes and equally partition the node set into ﬁve disjoint communities.
Revised: We create random graphs, each with 1000 nodes and equally partition the node set into ﬁve disjoint communities.

Original: Nodes in the same community have the same class label.
Revised: Nodes in the same community have the same class label.

Original: Edges are randomly sampled with an intra-community probability, p , if they are within the same community.
Revised: Edges are randomly sampled with an intra-community probability, p , if they are within the same community.

Original: They are sampled with an inter-community probability, q , if they are cross-community, e.g., bridges connecting different communities.
Revised: They are sampled with an inter-community probability, q , if they are cross-community, e.g., bridges connecting different communities.

Original: We randomly create 100 graphs with p ranging in { 0 . 05 , 0 . 07 , · · · , 0 . 23 } and q ranging in { 0 . 0 , 0 . 005 , · · · , 0 . 045 } .
Revised: We randomly create 100 graphs with p ranging in { 0 . 05 , 0 . 07 , · · · , 0 . 23 } and q ranging in { 0 . 0 , 0 . 005 , · · · , 0 . 045 } .

Original: For each generated graph, we randomly select 400 nodes as training set, another 400 nodes as validation set and the remaining 200 nodes as test set.
Revised: For each generated graph, we randomly select 400 nodes as training set, another 400 nodes as validation set and the remaining 200 nodes as test set.

Original: We assign each node with a randomly generated feature of dimension 10 and use them as uninformative input of our CurvGN.
Revised: We assign each node with a randomly generated feature of dimension 10 and use them as uninformative input of our CurvGN.

Original: Baselines.
Revised: Baselines.

Original: We use ﬁve different methods.
Revised: We use ﬁve different methods.

Original: They include two popular state-of-the-arts: GCN (Kipf & Welling, 2016) and GAT (Veliˇckovi´c et al., 2017).
Revised: They include two popular state-of-the-arts: GCN (Kipf & Welling, 2016) and GAT (Veliˇckovi´c et al., 2017).

Original: For these methods, we use the exact same setting as for Cora dataset mentioned in Veliˇckovi´c et al.
Revised: For these methods, we use the exact same setting as for Cora dataset mentioned in Veliˇckovi´c et al.

Original: (2017); Kipf & Welling (2016), except that the output of hidden layer is a vector of length 8.
Revised: (2017); Kipf & Welling (2016), except that the output of hidden layer is a vector of length 8.

Original: We also use a baseline method which aggregates messages without reweighing Vanilla GN .
Revised: We also use a baseline method which aggregates messages without reweighing Vanilla GN .

Original: We also apply the two proposed networks, CurvGNand CurvGN-n .
Revised: We also apply the two proposed networks, CurvGNand CurvGN-n .

Original: Compared with the Vanilla GN, GCN reweighs messages using node degrees.
Revised: Compared with the Vanilla GN, GCN reweighs messages using node degrees.

Original: GAT reweighs messages using self attention map computed using node representations.
Revised: GAT reweighs messages using self attention map computed using node representations.

Original: CurvGN-1 and
Revised: CurvGN-1 and

Original: CurvGN-n reweigh messages using scalar and vector computed by Ricci curvature.
Revised: CurvGN-n reweigh messages using scalar and vector computed by Ricci curvature.

Original: We run all the methods on 100 random graphs.
Revised: We run all the methods on 100 random graphs.

Original: For each graph, we run the training and inference task for 10 times and take the average accuracy.
Revised: For each graph, we run the training and inference task for 10 times and take the average accuracy.

Original: For each training, we run 200 epochs and use validation set for early stopping.
Revised: For each training, we run 200 epochs and use validation set for early stopping.

Original: Figure 3 shows the heat maps for all methods.
Revised: Figure 3 shows the heat maps for all methods.

Original: The title of each heatmap also includes the max and average performance over all parameter settings.
Revised: The title of each heatmap also includes the max and average performance over all parameter settings.

Original: In Figure 3(c), we run the same experiments on graphs with different sizes and report the average accuracy.
Revised: In Figure 3(c), we run the same experiments on graphs with different sizes and report the average accuracy.

Original: Discussion.
Revised: Discussion.

Original: Looking at the heatmaps, we observe that Vanilla GN, GCN and GAT are not better than random guessing.
Revised: Looking at the heatmaps, we observe that Vanilla GN, GCN and GAT are not better than random guessing.

Original: This implies that using node degree information is not enough.
Revised: This implies that using node degree information is not enough.

Original: Meanwhile, we observe improved performance by CurvGN-1, conﬁrming the power of reweighing with curvature in graph convolution.
Revised: Meanwhile, we observe improved performance by CurvGN-1, conﬁrming the power of reweighing with curvature in graph convolution.

Original: In addition, CurvGN-n outperforms CurvGN-1, suggesting that the multi-channel reweighing based on curvature is beneﬁcial.
Revised: In addition, CurvGN-n outperforms CurvGN-1, suggesting that the multi-channel reweighing based on curvature is beneﬁcial.

Original: Furthermore, looking at the results with different graph sizes, we observe that the beneﬁt of curvature increases as the graph size increases.
Revised: Furthermore, looking at the results with different graph sizes, we observe that the beneﬁt of curvature increases as the graph size increases.

Original: We hypothesize that the graph convolution is sufﬁcient in small graph setting to fully explore the graph structure.
Revised: We hypothesize that the graph convolution is sufﬁcient in small graph setting to fully explore the graph structure.

Original: Only with larger graph and more diverse topology, advanced structural information becomes signiﬁcant.
Revised: Only with larger graph and more diverse topology, advanced structural information becomes signiﬁcant.

Original: We also visualize the prediction results on one particular graph generated at ( p, q ) =
Revised: We also visualize the prediction results on one particular graph generated at ( p, q ) =

Original: 025) in Figure 4.
Revised: 025) in Figure 4.

Original: We observe that CurvGN-1 and CurvGN-n make high quality predictions except for a small portion of data in a few communities.
Revised: We observe that CurvGN-1 and CurvGN-n make high quality predictions except for a small portion of data in a few communities.

Original: Meanwhile, other baselines can completely mix different communities and results are unsatisfactory.
Revised: Meanwhile, other baselines can completely mix different communities and results are unsatisfactory.

Original: Further studies on different graph theoretical models .
Revised: Further studies on different graph theoretical models .

Original: To further examine the expressive power of Ricci curvature, we run synthetic experiments using several well-accepted graph theoretical models:
Revised: To further examine the expressive power of Ricci curvature, we run synthetic experiments using several well-accepted graph theoretical models:

Original: Watts–Strogatz network (Watts & Strogatz, 1998), Newman–Watts network (Newman & Watts, 1999) and Kleinberg’s navigable small world graph (Kleinberg, 2000).
Revised: Watts–Strogatz network (Watts & Strogatz, 1998), Newman–Watts network (Newman & Watts, 1999) and Kleinberg’s navigable small world graph (Kleinberg, 2000).

Original: Watts-Strogatz network randomly rewires edges of a ring graph.
Revised: Watts-Strogatz network randomly rewires edges of a ring graph.

Original: Newman-Watts network randomly adds new edges to the ring.
Revised: Newman-Watts network randomly adds new edges to the ring.

Original: The
Revised: The

Original: Kleinberg’s model also adds random edges, but the probability of a new edge is inversely proportional to the geodesic distance between the nodes.
Revised: Kleinberg’s model also adds random edges, but the probability of a new edge is inversely proportional to the geodesic distance between the nodes.

Original: In all these settings, we partition the ring into different communities and design experiments similar to the Stochastic Block Model.
Revised: In all these settings, we partition the ring into different communities and design experiments similar to the Stochastic Block Model.

Original: More details can be found in the Appendix A.2.
Revised: More details can be found in the Appendix A.2.

Original: We compare all ﬁve methods on these different graph models and report the average accuracy over different parameter settings in Table 1.
Revised: We compare all ﬁve methods on these different graph models and report the average accuracy over different parameter settings in Table 1.

Original: The standard deviations on these results are generally large as we are averaging over all different parameter settings.
Revised: The standard deviations on these results are generally large as we are averaging over all different parameter settings.

Original: We observe consistently better performance of CurvGN-n than other methods.
Revised: We observe consistently better performance of CurvGN-n than other methods.

Original: This conﬁrms that curvature information is beneﬁcial in a broad spectrum of graphs.
Revised: This conﬁrms that curvature information is beneﬁcial in a broad spectrum of graphs.

Original: We also note that GCN and GAT are indeed doing well for Watts-Strogatz and Newman-Watts models.
Revised: We also note that GCN and GAT are indeed doing well for Watts-Strogatz and Newman-Watts models.

Original: This is because, in these networks, edge rewiring and addition create difference in node degrees.
Revised: This is because, in these networks, edge rewiring and addition create difference in node degrees.

Original: Bridges crossing different communities tend to have higher node degree.
Revised: Bridges crossing different communities tend to have higher node degree.

Original: Therefore
Revised: Therefore

Original: [Table]
Revised: [Table]

Original: node degrees carry useful structural information and can help with graph convolution.
Revised: node degrees carry useful structural information and can help with graph convolution.

Original: We do not observe the same beneﬁt of node degree information in SBM and Kleinberg’s model as in
Revised: We do not observe the same beneﬁt of node degree information in SBM and Kleinberg’s model as in

Original: [Table]
Revised: [Table]

Original: these models, node degree is not correlated with the locations of bridges.
Revised: these models, node degree is not correlated with the locations of bridges.

Original: More heatmap results can be found in the Appendix A.2.
Revised: More heatmap results can be found in the Appendix A.2.

Original: 4. R EAL -W ORLD B ENCHMARKS
Revised: 4. R EAL -W ORLD B ENCHMARKS

Original: Our real-world benchmarks include two families of datasets: small sparse graphs and large dense graphs.
Revised: Our real-world benchmarks include two families of datasets: small sparse graphs and large dense graphs.

Original: We compare our networks CurvGN-1 and CurvGN-n with several strong baselines.
Revised: We compare our networks CurvGN-1 and CurvGN-n with several strong baselines.

Original: Aside from GCN and GAT that have been used in the synthetic experiments, we also compare CurvGNand CurvGN-n with multilayer perceptron (MLP), MoNet (Monti et al., 2017), WSCN (Morris et al., 2019) and GraphSAGE with mean aggregation (GS-mean) (Hamilton et al., 2017).
Revised: Aside from GCN and GAT that have been used in the synthetic experiments, we also compare CurvGNand CurvGN-n with multilayer perceptron (MLP), MoNet (Monti et al., 2017), WSCN (Morris et al., 2019) and GraphSAGE with mean aggregation (GS-mean) (Hamilton et al., 2017).

Original: Our method is on par with state-of-the-art methods on relatively small graphs and greatly outperforms state-of-the-art methods on large and dense graphs, which tend to have heterogeneous topology.
Revised: Our method is on par with state-of-the-art methods on relatively small graphs and greatly outperforms state-of-the-art methods on large and dense graphs, which tend to have heterogeneous topology.

Original: Datasets.
Revised: Datasets.

Original: We use three popular citation network benchmark datasets: Cora, citetseer and PubMed (Sen et al., 2008).
Revised: We use three popular citation network benchmark datasets: Cora, citetseer and PubMed (Sen et al., 2008).

Original: We categorize Cora and citetseer into the ﬁrst family since both Cora and citetseer graphs are relatively small and sparse.
Revised: We categorize Cora and citetseer into the ﬁrst family since both Cora and citetseer graphs are relatively small and sparse.

Original: They have thousands of nodes and edges with an average node degree below 2 .
Revised: They have thousands of nodes and edges with an average node degree below 2 .

Original: We also use four extra datasets: Coauthor CS and Coauthor Physics which are co-authorship graphs based on the Microsoft Academic Graph from the KDD Cup 2016 challenge;
Revised: We also use four extra datasets: Coauthor CS and Coauthor Physics which are co-authorship graphs based on the Microsoft Academic Graph from the KDD Cup 2016 challenge;

Original: Amazon Computers and Amazon Photos which are segments of the Amazon co-purchase graph in
Revised: Amazon Computers and Amazon Photos which are segments of the Amazon co-purchase graph in

Original: McAuley et al.
Revised: McAuley et al.

Original: These graphs, together with PubMed, are large and dense graphs.
Revised: These graphs, together with PubMed, are large and dense graphs.

Original: Those graphs have more than 10 thousands node and 200 thousands edges with an average node degree as high as 20 .
Revised: Those graphs have more than 10 thousands node and 200 thousands edges with an average node degree as high as 20 .

Original: We use the exact data splitting as in semi-supervised learning setting used in Kipf & Welling (2016);
Revised: We use the exact data splitting as in semi-supervised learning setting used in Kipf & Welling (2016);

Original: Veli ˇ ckovi ´ c et al.
Revised: Veli ˇ ckovi ´ c et al.

Original: (2017): using 20 nodes per class for training, 500 nodes for validation and 1000 nodes for testing.
Revised: (2017): using 20 nodes per class for training, 500 nodes for validation and 1000 nodes for testing.

Original: Descriptions and statistics for all datasets in our experiments can be found in the Appendix A.3.
Revised: Descriptions and statistics for all datasets in our experiments can be found in the Appendix A.3.

Original: During training stage, we set L 2 regularization with λ = 0 .
Revised: During training stage, we set L 2 regularization with λ = 0 .

Original: for all datasets.
Revised: for all datasets.

Original: Also, all the models are initialized by Glorot initialization and trained by minimizing cross-entropy loss using Adam SGD optimizer with learning rate r = 0 .
Revised: Also, all the models are initialized by Glorot initialization and trained by minimizing cross-entropy loss using Adam SGD optimizer with learning rate r = 0 .

Original: We apply an early stopping strategy with the help of the validation set based on the validation set’s accuracy with a patience of 100 epochs.
Revised: We apply an early stopping strategy with the help of the validation set based on the validation set’s accuracy with a patience of 100 epochs.

Original: We compute curvature exactly following Eq.
Revised: We compute curvature exactly following Eq.

Original: (3.2) for all datasets but one.
Revised: (3.2) for all datasets but one.

Original: For the Amazon Computer dataset, we use an approximation scheme for efﬁciency (Ni et al., 2018).
Revised: For the Amazon Computer dataset, we use an approximation scheme for efﬁciency (Ni et al., 2018).

Original: The linear programming problem is solved using an interior point solver (ECOS).
Revised: The linear programming problem is solved using an interior point solver (ECOS).

Original: We report the mean and standard deviation of classiﬁcation accuracy on test nodes on 100 runs and re-use the metrics reported by Monti et al.
Revised: We report the mean and standard deviation of classiﬁcation accuracy on test nodes on 100 runs and re-use the metrics reported by Monti et al.

Original: ; Shchur et al.
Revised: ; Shchur et al.

Original: (2018); Veli ˇ ckovi ´ c et al.
Revised: (2018); Veli ˇ ckovi ´ c et al.

Original: (2017) for other state-of-the-art methods.
Revised: (2017) for other state-of-the-art methods.

Original: The results are reported in Table 2.
Revised: The results are reported in Table 2.

Original: Our method is on par with state-of-the-art performance for relatively small graph and achieves superior performance on large and dense graphs.
Revised: Our method is on par with state-of-the-art performance for relatively small graph and achieves superior performance on large and dense graphs.

Original: This is consistent with our conclusion from synthetic experiments: when graph is large and has heterogeneous topology, advanced structural information becomes critical in graph convolution.
Revised: This is consistent with our conclusion from synthetic experiments: when graph is large and has heterogeneous topology, advanced structural information becomes critical in graph convolution.

Original: [Table]
Revised: [Table]

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: We introduce a novel graph convolution network to leverage advanced graph structural information, namely, the graph curvature.
Revised: We introduce a novel graph convolution network to leverage advanced graph structural information, namely, the graph curvature.

Original: The curvature information effectively helps achieve superior performance on synthetic and real-world datasets, especially on larger and denser graphs.
Revised: The curvature information effectively helps achieve superior performance on synthetic and real-world datasets, especially on larger and denser graphs.

Original: This shows how principled mathematics and theory help the deployment of deep learning and encourages us to continue the endeavor in bridging the gap between graph theoretical foundation and neural networks
Revised: This shows how principled mathematics and theory help the deployment of deep learning and encourages us to continue the endeavor in bridging the gap between graph theoretical foundation and neural networks

Original: [Table]
Revised: [Table]

Original: A A PPENDIX A. T HE D ETAILS OF MLP T
Revised: A A PPENDIX A. T HE D ETAILS OF MLP T

Original: We describe the details of MLP t for the t -th convolutional layer, which maps a curvature κ xy to the weight vector T txy ∈ R F t +1 .
Revised: We describe the details of MLP t for the t -th convolutional layer, which maps a curvature κ xy to the weight vector T txy ∈ R F t +1 .

Original: MLP t has three layers: an input layer, followed by a non-linear transformation layer and an output layer.
Revised: MLP t has three layers: an input layer, followed by a non-linear transformation layer and an output layer.

Original: The input layer In linearly transforms the Ricci curvature κ xy into an output vector with same size of message M ty → x ∈ R F t +1 .
Revised: The input layer In linearly transforms the Ricci curvature κ xy into an output vector with same size of message M ty → x ∈ R F t +1 .

Original: We use LeakyReLU function in our non-linear layer.
Revised: We use LeakyReLU function in our non-linear layer.

Original: For output layer Out , we use a transformation matrix with size F t +1 × F t + to compute reweighing vector T txy .
Revised: For output layer Out , we use a transformation matrix with size F t +1 × F t + to compute reweighing vector T txy .

Original: Formally,
Revised: Formally,

Original: [Equation]
Revised: [Equation]

Original: Recall a node also passes a message to itself.
Revised: Recall a node also passes a message to itself.

Original: To generate its weight vector T txy , we set κ xx = 0 , as if the edge ( x, x ) is a grid edge.
Revised: To generate its weight vector T txy , we set κ xx = 0 , as if the edge ( x, x ) is a grid edge.

Original: For the case when we reweigh the message using a single scalar τ xy (e.g., CurvGN-1 network), we change the size of transformation matrix of output layer into F t +1 × 1 .
Revised: For the case when we reweigh the message using a single scalar τ xy (e.g., CurvGN-1 network), we change the size of transformation matrix of output layer into F t +1 × 1 .

Original: A. D IFFERENT N ETWORK M ODELS FOR S YNTHETIC E XPERIMENTS
Revised: A. D IFFERENT N ETWORK M ODELS FOR S YNTHETIC E XPERIMENTS

Original: All three models, Watts-Strogatz, Newman-Watts and Kleinberg’s model, are created by randomly modifying/adding edges to a ring graph.
Revised: All three models, Watts-Strogatz, Newman-Watts and Kleinberg’s model, are created by randomly modifying/adding edges to a ring graph.

Original: 1 A ring graph has n nodes embedded on a circle, with each node connected to its k nearest neighbors.
Revised: 1 A ring graph has n nodes embedded on a circle, with each node connected to its k nearest neighbors.

Original: Figure 5(a) is an example ring graph with n =and k = 4 .
Revised: Figure 5(a) is an example ring graph with n =and k = 4 .

Original: To create communities, we partition the nodes into 5 equal-size sets according to their locations on the circle.
Revised: To create communities, we partition the nodes into 5 equal-size sets according to their locations on the circle.

Original: In addition, we remove the edges cross different communities.
Revised: In addition, we remove the edges cross different communities.

Original: Next we explain how edges of the ring graph are randomly changed for Watts-Strogatz, Newman-Watts and
Revised: Next we explain how edges of the ring graph are randomly changed for Watts-Strogatz, Newman-Watts and

Original: Kleinberg’s model, respectively.
Revised: Kleinberg’s model, respectively.

Original: Watts-Strogatz Network.
Revised: Watts-Strogatz Network.

Original: Watts-Strogatz Network (Watts & Strogatz, 1998) is created by randomly rewiring edges of the ring graph with a predeﬁned probability, p .
Revised: Watts-Strogatz Network (Watts & Strogatz, 1998) is created by randomly rewiring edges of the ring graph with a predeﬁned probability, p .

Original: See Figure 5(b) for an example of
Revised: See Figure 5(b) for an example of

Original: Watts-Strogatz network.
Revised: Watts-Strogatz network.

Original: In our experiments, we generate 100 random Watts-Strogatz graphs of size n = 1000 using different parameter combinations of k and p : k ∈ { 5 , 10 , · · · , 50 } and p ∈ { 0 . 02 , 0 . 04 , · · · , 0 . 2 } .
Revised: In our experiments, we generate 100 random Watts-Strogatz graphs of size n = 1000 using different parameter combinations of k and p : k ∈ { 5 , 10 , · · · , 50 } and p ∈ { 0 . 02 , 0 . 04 , · · · , 0 . 2 } .

Original: For each graph, the 5 communities correspond to nodes with 5 different labels.
Revised: For each graph, the 5 communities correspond to nodes with 5 different labels.

Original: We randomly generate a 10 -dimensional feature for each node, as in Stochastic Block Model experiments.
Revised: We randomly generate a 10 -dimensional feature for each node, as in Stochastic Block Model experiments.

Original: The training set is created by randomly sampling 400 nodes in one graph.
Revised: The training set is created by randomly sampling 400 nodes in one graph.

Original: The validation set and testing set are create in the same way with size 400 and 200 , respectively.
Revised: The validation set and testing set are create in the same way with size 400 and 200 , respectively.

Original: For each graph, we run the experiment 10 times with 200 epochs each time and report the average.
Revised: For each graph, we run the experiment 10 times with 200 epochs each time and report the average.

Original: Figure 6(a) shows the results of all ﬁve methods (GCN, GAT, Vanilla GN, CurvGN-1 and CurvGN-n).
Revised: Figure 6(a) shows the results of all ﬁve methods (GCN, GAT, Vanilla GN, CurvGN-1 and CurvGN-n).

Original: We observe that CurvGN-n has the best performance compared with others.
Revised: We observe that CurvGN-n has the best performance compared with others.

Original: It suggests that edge curvature information is crucial in prediction: a rewired edge has a high probability to be a bridge with negative curvature.
Revised: It suggests that edge curvature information is crucial in prediction: a rewired edge has a high probability to be a bridge with negative curvature.

Original: Curvature information can effectively distinguish bridges and intra-community edges, and therefore help graph convolution.
Revised: Curvature information can effectively distinguish bridges and intra-community edges, and therefore help graph convolution.

Original: It is also worth mentioning that GCN also has good performance.
Revised: It is also worth mentioning that GCN also has good performance.

Original: We hypothesize that this is because rewired edges (likely bridges) tend to have higher degrees on adjacent nodes, and thus can be distinguished using degree information alone.
Revised: We hypothesize that this is because rewired edges (likely bridges) tend to have higher degrees on adjacent nodes, and thus can be distinguished using degree information alone.

Original: Newman-Watts Network.
Revised: Newman-Watts Network.

Original: The Newman-Watts network (Newman & Watts, 1999) is similar to the
Revised: The Newman-Watts network (Newman & Watts, 1999) is similar to the

Original: Watts-Strogatz model except that it adds random edges on the ring graph with probability p , instead of rewiring existing edges.
Revised: Watts-Strogatz model except that it adds random edges on the ring graph with probability p , instead of rewiring existing edges.

Original: We run the experiments in the same setting as Watts-Strogatz model.
Revised: We run the experiments in the same setting as Watts-Strogatz model.

Original: The results are shown in Figure 6(b).
Revised: The results are shown in Figure 6(b).

Original: We observe similar effects as Watts-Strogatz.
Revised: We observe similar effects as Watts-Strogatz.

Original: Kleinberg’s Navigable Small World Graph.
Revised: Kleinberg’s Navigable Small World Graph.

Original: Instead of randomly generating edges with a ﬁxed probability p , Kleinberg’s model (Kleinberg, 2000) adds a ﬁxed number of additional long-range edges to the ring graph.
Revised: Instead of randomly generating edges with a ﬁxed probability p , Kleinberg’s model (Kleinberg, 2000) adds a ﬁxed number of additional long-range edges to the ring graph.

Original: For each node u , add e l random edges ( u, v ) with v picked with a probability proportional to 1 /d ( u, v ) , in which d ( u, v ) is the distance between u and v in the circle.
Revised: For each node u , add e l random edges ( u, v ) with v picked with a probability proportional to 1 /d ( u, v ) , in which d ( u, v ) is the distance between u and v in the circle.

Original: 2 We slightly modify the original deﬁnition by making all edges undirected and removing self-loops.
Revised: 2 We slightly modify the original deﬁnition by making all edges undirected and removing self-loops.

Original: Figure 5(c) shows an example graph of Kleinberg’s model with 100 nodes.
Revised: Figure 5(c) shows an example graph of Kleinberg’s model with 100 nodes.

Original: We observe much fewer long range (cross-community) connections and more intra-community connections than the other models.
Revised: We observe much fewer long range (cross-community) connections and more intra-community connections than the other models.

Original: [Figure]
Revised: [Figure]

Original: [Figure]
Revised: [Figure]

Original: We generate 100 different graphs using different combinations of parameters e s and e l .
Revised: We generate 100 different graphs using different combinations of parameters e s and e l .

Original: Here e s ∈ { ﬂoor (2 . 5) , ﬂoor (5) , · · · ﬂoor (25) } controls the distance upperbound for short-range neighbors; any nodes within distance e s of x is connected with x .
Revised: Here e s ∈ { ﬂoor (2 . 5) , ﬂoor (5) , · · · ﬂoor (25) } controls the distance upperbound for short-range neighbors; any nodes within distance e s of x is connected with x .

Original: e s is very similar to k in Wattz-Strogatz graph.
Revised: e s is very similar to k in Wattz-Strogatz graph.

Original: Similar to previous models, we run experiments on each graph 10 times and use the validation set for early stopping.
Revised: Similar to previous models, we run experiments on each graph 10 times and use the validation set for early stopping.

Original: Since the added edge has a probability proportional to 1 /d ( u, v ) , there can be long distance intercommunity edges.
Revised: Since the added edge has a probability proportional to 1 /d ( u, v ) , there can be long distance intercommunity edges.

Original: And the node degree can no longer capture the structural information introduced by the added edges.
Revised: And the node degree can no longer capture the structural information introduced by the added edges.

Original: The GCN behaves similar to random guessing in this case.
Revised: The GCN behaves similar to random guessing in this case.

Original: However, the Ricci
Revised: However, the Ricci

Original: Curvature is still negative on those inter community edges and it can still predict communities.
Revised: Curvature is still negative on those inter community edges and it can still predict communities.

Original: Figure 6(c) shows the heat maps of ﬁve different algorithms.
Revised: Figure 6(c) shows the heat maps of ﬁve different algorithms.

Original: CurvGN-n outperforms other methods by a large margin.
Revised: CurvGN-n outperforms other methods by a large margin.

Original: A. S TATISTICAL DETAIL OF BENCHMARKS
Revised: A. S TATISTICAL DETAIL OF BENCHMARKS

Original: We describe the statistical details of all datasets in Table 3.
Revised: We describe the statistical details of all datasets in Table 3.

Original: Cora and Citeseer are considered as small and sparse graphs while PubMed, Coauthors and Amazons are considered as large and dense graphs.
Revised: Cora and Citeseer are considered as small and sparse graphs while PubMed, Coauthors and Amazons are considered as large and dense graphs.

Original: [Table]
Revised: [Table]

Original: A. C OMPLEXITY OF C URVATURE C OMPUTATION
Revised: A. C OMPLEXITY OF C URVATURE C OMPUTATION

Original: Exactly calculating the curvatures is somewhat time-consuming.
Revised: Exactly calculating the curvatures is somewhat time-consuming.

Original: We need to solve a linear programming (LP) problem for each edge.
Revised: We need to solve a linear programming (LP) problem for each edge.

Original: For an edge ( u, v ) , the LP problem has d u × d v variables and d u + d v linear constraints, in which d u and d v are the degrees of the nodes u and v .
Revised: For an edge ( u, v ) , the LP problem has d u × d v variables and d u + d v linear constraints, in which d u and d v are the degrees of the nodes u and v .

Original: Take the interior point solver (ECOS) as an example, the complexity is O (( d u × d v ) w ) , in which w is the exponent of matrix multiplication (the best known is 2 . 373 ).
Revised: Take the interior point solver (ECOS) as an example, the complexity is O (( d u × d v ) w ) , in which w is the exponent of matrix multiplication (the best known is 2 . 373 ).

Original: However, in practice, there are many approximation methods to accelerate the computation of optimal transport distance, among which the best known is the Sinkhorn algorithm (Cuturi, 2013).
Revised: However, in practice, there are many approximation methods to accelerate the computation of optimal transport distance, among which the best known is the Sinkhorn algorithm (Cuturi, 2013).

Original: It is known to have near-linear time complexity (Altschuler et al., 2017).
Revised: It is known to have near-linear time complexity (Altschuler et al., 2017).

Original: So for small datasets we kept using the ECOS to get exact solution while for large datasets we may use advanced approximation methods.
Revised: So for small datasets we can keep using the ECOS to get exact solution while for large datasets we may use advanced approximation methods.

Original: In Table 4, we show the computation time for the curvatures with two 18-core CPUs.
Revised: In Table 4, we show the computation time for the curvatures with two 18-core CPUs.

Original: Note that this computation is only needed once for data pre-processing, and the values can be reused in various downstream tasks, so it is feasible and practical.
Revised: Note that this computation is only needed once for data pre-processing, and the values can be reused in various downstream tasks, so it is feasible and practical.

Original: [Table]
Revised: [Table]

Original: A. N ECESSITY OF M ULTI - CHANNEL MLP
Revised: A. N ECESSITY OF M ULTI - CHANNEL MLP

Original: The usage of multi-channel MLP is essential in the success of using the curvature information.
Revised: The usage of multi-channel MLP is essential in the success of using the curvature information.

Original: To illustrate this, in Table 5, we report the performance of our model if we directly use the curvature to weight messages, i.e., τ xy
Revised: To illustrate this, in Table 5, we report the performance of our model if we directly use the curvature to weight messages, i.e., τ xy

Original: = κ xy for all layers.
Revised: = κ xy for all layers.

Original: We call this baseline CurvGN- κ .
Revised: We call this baseline CurvGN- κ .

Original: We observe that curvature alone cannot provide the best reweighting scheme for graph convolution.
Revised: We observe that curvature alone cannot provide the best reweighting scheme for graph convolution.

Original: Only learning from the data helps us determine how weights on different channels should be dependent on kappa.
Revised: Only learning from the data helps us determine how weights on different channels should be dependent on kappa.

Original: In Figure 7, we show for one example graph in CurvGN-n how τ xy at different channels
Revised: In Figure 7, we show for one example graph in CurvGN-n how τ xy at different channels

Original: [Table]
Revised: [Table]

Original: change according to κ xy .
Revised: change according to κ xy .

Original: We observe weights of different channels are changing very differently w.r.t.
Revised: We observe weights of different channels are changing very differently w.r.t.

Original: κ xy ; some are increasing w.r.t.
Revised: κ xy ; some are increasing w.r.t.

Original: κ xy , others are decreasing.
Revised: κ xy , others are decreasing.

Original: MLP is necessary in learning such dependence from data.
Revised: MLP is necessary in learning such dependence from data.

Original: Using MLP enhances the model ﬂexibility and model capacity.
Revised: Using MLP enhances the model ﬂexibility and model capacity.

Original: But also note that curvature κ xy is the essential input that the MLP relies on.
Revised: But also note that curvature κ xy is the essential input that the MLP relies on.